KeyError when scoring interactions in CellPhoneDB

Description
I encountered a KeyError when running the statistical analysis method in CellPhoneDB with interaction scoring enabled. I want to gracefully handle missing keys by returning np.nan instead of raising an error.
Steps to Reproduce
Load the anndata object with normalized counts and cell type metadata.
Generate the metadata file for CellPhoneDB.
Run the cpdb_statistical_analysis_method.call with the following parameters:

Load the anndata object with normalized counts and cell type metadata.
Generate the metadata file for CellPhoneDB.
Run the cpdb_statistical_analysis_method.call with the following parameters:

prepare data
import numpy as np
import pandas as pd
import scanpy as sc
import anndata
import glob
import os
import sys
from scipy import sparse

sc.settings.verbosity = 1  # verbosity: errors (0), warnings (1), info (2), hints (3)
sys.executable

# 1. Load anndata
adata = sc.read_h5ad('./write/adata.h5ad')
# 2. Generate meta
adata.obs['subtype'].values.describe()
df_meta = pd.DataFrame(data={'Cell':list(adata.obs.index),
                             'celltype':[ i for i in adata.obs['subtype']]
                            })
df_meta.set_index('Cell', inplace=True)
df_meta.to_csv('./write/adata_meta.tsv', sep = '\t')

cpdb_statistical_analysis_method.call
from IPython.display import HTML, display
from cellphonedb.utils import db_releases_utils

display(HTML(db_releases_utils.get_remote_database_versions_html()['db_releases_html_table']))

Version	Release date
v5.0.0	2023-10-31
v4.1.0	2023-03-09
cpdb_version = 'v5.0.0'
cpdb_target_dir = os.path.join('./write/cellphoneDB_database', cpdb_version)
from cellphonedb.utils import db_utils
db_utils.download_database(cpdb_target_dir, cpdb_version)

meta_file_path = './write/adata_meta.tsv'
counts_file_path = './write/adata.h5ad'
cpdb_file_path = './write/cellphoneDB_database/v5.0.0/cellphonedb.zip'

following is the keyerror code

First Attempt
cpdb_results = cpdb_statistical_analysis_method.call(
    cpdb_file_path = cpdb_file_path,                 # mandatory: CellphoneDB database zip file.
    meta_file_path = meta_file_path,                 # mandatory: tsv file defining barcodes to cell label.
    counts_file_path = counts_file_path,             # mandatory: normalized count matrix - a path to the counts file, or an in-memory AnnData object
    counts_data = 'hgnc_symbol',                     # defines the gene annotation in counts matrix.
   # active_tfs_file_path = active_tf_path,          # optional: defines cell types and their active TFs.
   # microenvs_file_path = microenvs_file_path,      # optional (default: None): defines cells per microenvironment.
    score_interactions = True,                       # optional: whether to score interactions or not.
    iterations = 1000,                               # denotes the number of shufflings performed in the analysis.
    threshold = 0.1,                                 # defines the min % of cells expressing a gene for this to be employed in the analysis.
    threads = 5,                                     # number of threads to use in the analysis.
    debug_seed = 42,                                 # debug randome seed. To disable >=0.
    result_precision = 3,                            # Sets the rounding for the mean values in significan_means.
    pvalue = 0.05,                                   # P-value threshold to employ for significance.
    subsampling = False,                             # To enable subsampling the data (geometri sketching).
    subsampling_log = False,                         # (mandatory) enable subsampling log1p for non log-transformed data inputs.
    subsampling_num_pc = 100,                        # Number of componets to subsample via geometric skectching (dafault: 100).
    subsampling_num_cells = 1000,                    # Number of cells to subsample (integer) (default: 1/3 of the dataset).
    separator = '|',                                 # Sets the string to employ to separate cells in the results dataframes "cellA|CellB".
    debug = False,                                   # Saves all intermediate tables employed during the analysis in pkl format.
    output_path =  "./write/cellphonedb_out",                          # Path to save results.
    output_suffix = "SMGs"                             # Replaces the timestamp in the output files by a user defined string in the  (default: None).
    )

[CORE][22/05/24-17:42:04][INFO] Scoring interactions: Calculating scores for all interactions and cell types..
100%|██████████| 1444/1444 [05:53<00:00,  4.08it/s]
KeyError                                  Traceback (most recent call last)
Cell In[15], line 1
----> 1 cpdb_results = cpdb_statistical_analysis_method.call(
2     cpdb_file_path = cpdb_file_path,                 # mandatory: CellphoneDB database zip file.
3     meta_file_path = meta_file_path,                 # mandatory: tsv file defining barcodes to cell label.
4     counts_file_path = counts_file_path,             # mandatory: normalized count matrix - a path to the counts file, or an in-memory AnnData object
5     counts_data = 'hgnc_symbol',                     # defines the gene annotation in counts matrix.
6    # active_tfs_file_path = active_tf_path,          # optional: defines cell types and their active TFs.
7    # microenvs_file_path = microenvs_file_path,      # optional (default: None): defines cells per microenvironment.
8     score_interactions = True,                       # optional: whether to score interactions or not.
9     iterations = 1000,                               # denotes the number of shufflings performed in the analysis.
10     threshold = 0.1,                                 # defines the min % of cells expressing a gene for this to be employed in the analysis.
11     threads = 5,                                     # number of threads to use in the analysis.
12     debug_seed = 42,                                 # debug randome seed. To disable >=0.
13     result_precision = 3,                            # Sets the rounding for the mean values in significan_means.
14     pvalue = 0.05,                                   # P-value threshold to employ for significance.
15     subsampling = False,                             # To enable subsampling the data (geometri sketching).
16     subsampling_log = False,                         # (mandatory) enable subsampling log1p for non log-transformed data inputs.
17     subsampling_num_pc = 100,                        # Number of componets to subsample via geometric skectching (dafault: 100).
18     subsampling_num_cells = 1000,                    # Number of cells to subsample (integer) (default: 1/3 of the dataset).
19     separator = '|',                                 # Sets the string to employ to separate cells in the results dataframes "cellA|CellB".
20     debug = False,                                   # Saves all intermediate tables employed during the analysis in pkl format.
21     output_path =  "./write/cellphonedb_out",                          # Path to save results.
22     output_suffix = "SMGs"                             # Replaces the timestamp in the output files by a user defined string in the  (default: None).
23     )
File ~/miniconda3/envs/cpdb/lib/python3.8/site-packages/cellphonedb/src/core/methods/cpdb_statistical_analysis_method.py:157, in call(cpdb_file_path, meta_file_path, counts_file_path, counts_data, output_path, microenvs_file_path, active_tfs_file_path, iterations, threshold, threads, debug_seed, result_precision, pvalue, subsampling, subsampling_log, subsampling_num_pc, subsampling_num_cells, separator, debug, output_suffix, score_interactions)
154 if score_interactions:
155     # Make sure all cell types are strings
156     meta['cell_type'] = meta['cell_type'].apply(str)
--> 157     interaction_scores = scoring_utils.score_interactions_based_on_participant_expressions_product(
158         cpdb_file_path, counts4scoring, means_result.copy(), separator, meta, threshold, "cell_type", threads)
159     analysis_result['interaction_scores'] = interaction_scores
161 file_utils.save_dfs_as_tsv(output_path, output_suffix, "statistical_analysis", analysis_result)
File ~/miniconda3/envs/cpdb/lib/python3.8/site-packages/cellphonedb/utils/scoring_utils.py:344, in score_interactions_based_on_participant_expressions_product(cpdb_file_path, counts, means, separator, metadata, threshold, cell_type_col_name, threads)
340 cpdb_fms = scale_expression(cpdb_fmsh,
341                             upper_range=10)
343 # Step 5: calculate the ligand-receptor score.
--> 344 interaction_scores = score_product(matrix=cpdb_fms,
345                                    means=means,
346                                    separator=separator,
347                                    interactions=interactions,
348                                    id2name=id2name,
349                                    threads=threads)
350 return interaction_scores
File ~/miniconda3/envs/cpdb/lib/python3.8/site-packages/cellphonedb/utils/scoring_utils.py:290, in score_product(matrix, interactions, means, separator, id2name, threads)
288 for ct_pair, lr_scores_filtered in results:
289     interacting_pair2score = dict(zip(lr_scores_filtered['interacting_pair'], lr_scores_filtered['score']))
--> 290     interaction_scores[ct_pair] = [interacting_pair2score[id] for id in interaction_scores['interacting_pair']]
292 return interaction_scores
File ~/miniconda3/envs/cpdb/lib/python3.8/site-packages/cellphonedb/utils/scoring_utils.py:290, in (.0)
288 for ct_pair, lr_scores_filtered in results:
289     interacting_pair2score = dict(zip(lr_scores_filtered['interacting_pair'], lr_scores_filtered['score']))
--> 290     interaction_scores[ct_pair] = [interacting_pair2score[id] for id in interaction_scores['interacting_pair']]
292 return interaction_scores
KeyError: 'COL11A1_integrin_a11b1_complex'
Proposed Solution:
I propose to modify the score_product function to handle missing keys gracefully by returning np.nan. Here is the modified score_product function:
from joblib import Parallel, delayed
import numpy as np
import pandas as pd

def score_interaction(ct_pair, matrix, interactions, id2name, separator):
    sub_matrix = matrix[[ct_pair]].copy()
    sub_matrix = sub_matrix.dropna()
    sub_matrix = sub_matrix.T
    scores = []

    for idx, row in sub_matrix.iterrows():
        interaction_id = row.name
        sub_prod = row.product()
        try:
            geom = np.power(sub_prod, 1 / len(row))
        except ValueError:
            geom = np.nan
        scores.append((interaction_id, geom))

    scores_filtered = pd.DataFrame(scores, columns=['interacting_pair', 'score'])
    return ct_pair, scores_filtered

def score_product(matrix, interactions, means, separator, id2name, threads):
    interaction_scores = means[['interacting_pair']].copy()
    interaction_scores = interaction_scores.assign(**{col: np.nan for col in matrix.columns})

    results = Parallel(n_jobs=threads, backend='multiprocessing')(
        delayed(score_interaction)(
            ct_pair, matrix, interactions, id2name, separator
        ) for ct_pair in matrix.columns
    )

    for ct_pair, lr_scores_filtered in results:
        interacting_pair2score = dict(zip(lr_scores_filtered['interacting_pair'], lr_scores_filtered['score']))
        scores = []
        for id in interaction_scores['interacting_pair']:
            try:
                scores.append(interacting_pair2score[id])
            except KeyError:
                scores.append(np.nan)
        interaction_scores[ct_pair] = scores

    return interaction_scores

KeyError: 'COL11A1_integrin_a11b1_complex'
Environment

Python version: 3.8
CellPhoneDB version: [specific version]
Operating System: [your operating system]

I also found an issue where score_interactions function works successfully in v4.1.0 and v5.0.0, but in v4.0.0 it also gives an error (keyerror), and when set to false, it runs successfully.

Probably the problems count for your adata count, which is the count data nor the log normalized data From: XiongSQ @.> Sent: Monday, June 3, 2024 11:06 AM To: ventolab/CellphoneDB @.> Cc: Jason GUO @.>; Mention @.> Subject: Re: [ventolab/CellphoneDB] KeyError when scoring interactions in CellPhoneDB (Issue #190) I also found an issue where score_interactions function works successfully in v4.1.0 and v5.0.0, but in v4.0.0 it also gives an error (keyerror), and when set to false, it runs successfully. — Reply to this email directly, view it on GitHub<#190 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZNZINDQ3MH7BNTV4M5MMHDZFPMQRAVCNFSM6AAAAABIEXUIJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBUGE4TGMRYHA. You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>
Sorry for the delayed response, and thank you very much for your reply. However, my adata X is indeed a normalized count matrix. And running in two other versions of db is normal. QAQ

Hi @GUOYF0412,
Just to double-check, are you still having this issue since I couldn't be sure because the issue was closed and reopened. If you're still having the issues, could you make sure that your data was normalised without z-scaling so the zeros in the matrix are not transformed?
Best, Batu

Hi@

Hi @GUOYF0412,
Just to double-check, are you still having this issue since I couldn't be sure because the issue was closed and reopened. If you're still having the issues, could you make sure that your data was normalised without z-scaling so the zeros in the matrix are not transformed?
Best, Batu

Hi @cakirb,
I have similar issue with same iterate RuntimeWarning: invalid value encountered in power
geom = np.power(sub_prod, 1 / len(sub_values)) and a KeyError.


Revert "Add missing package bigchaindb.web to setup.py"

Reverts bigchaindb/bigchaindb#120

Current coverage is 84.69%

Merging #144 into develop will increase coverage by +84.69% as of 1ebba35

@@            develop    #144   diff @@
=======================================
  Files             0      18    +18
  Stmts             0     869   +869
  Branches          0       0
  Methods           0       0
=======================================
+ Hit               0     736   +736
  Partial           0       0
- Missed            0     133   +133

Review entire Coverage Diff as of 1ebba35

Powered by Codecov. Updated on successful CI builds.

:+1:

Note: This PR fixes a bug that @crazyquark found, using a more general solution than the one in PR #120
Thanks @crazyquark for pointing it out!


Lifecycle / hook for when the_post is available

Hi - currently we have 2 hooks for __before and __after however it would be good to have a hook available once the post was ready.
For example it means you could do something like
public function __content_ready()
{
    $indexer = new Indexer(get_the_content());
}

Normally you'd do something like this in a __construct but as this fires before everything is booted.
Does this make sense?

I can look into getting a __ready hook added for the next release.

@OwenMelbz I'm looking at getting this added, but before I do, can you give me a use-case for this? Could the same not be achieved with __after() as all the data is loaded then?

This might be achievable with the __after() but we couldnt see how :D
So the situation we had was that we needed to access the HTML from the_content THEN set a variable for the template. e.g. so it was something like
PageController {

public function __construct(ContentManipulator $manipulator)
    $this->manipulator = $manipulator;
}

public function __ready()
{
    $this->manipulator->setOriginalContent(get_the_content());
}

public function manipulatedContent()
{
    return $this->manipulator->getManipulatedContent();
}

public function extras()
{
    return $this->manipulator->getExtrasFromContent();
}

}

So in our blade template we can then use $manipulated_content rather than the_content and we also get $extras.
The reason we had to do this was because in the CMS there it the WordPress WYSIWYG - the client adds various content such as:

Section 1

Para

Section 2

Para

Section 3

Para

The content manipulator modifies the markup into

Section 1

Para

Section 2

Para

Section 3

Para

it also returns an array of those sections e.g.
$extras = [
[
    "label" => "Section 1",
    "anchor" => "section-1"
],
[
    "label" => "Section 2",
    "anchor" => "section-2"
],
[
    "label" => "Section 3",
    "anchor" => "section-3"
]
];

So then we have a sidebar with all the sections with jump links, and the main content in the middle.
Tadaaa?!

You could use after(), for example;
    public function __after()
    {
        $content = $this->data['post']->post_content;
        $this->manipulator->setOriginalContent($content);
        $this->data['post']->post_content = $this->manipulator->getManipulatedContent();
    }

This would overwrite the existing post_content content.
I could include __ready() in which case, $this->data['post'] would be available in Controller, but I'm not sure how widely it would be used.

I think it's raw, but you can apparently do this to apply filters;
$post   = get_post();
$output =  apply_filters( 'the_content', $post->post_content );

Right lol - I think this is why I came to the conclusion that it started to get quite messy.
Personally I feel the example I provided is clean and descriptive which is why I was hoping we could use the hooks more effectively

So there's one of two improvements I could make.
I could either create $this->post with all $post object, which would be easy to get title, permalink, content, etc. (eg. $this->post->post_title)
Or I could use setup_postdata($post) to make the hooks work. I guess this would be more inline with what users are used too, and would make your example above work. It just feels nasty, but that's WP I guess.

Having testing more, get_the_content() is the only function that has an issue running in a controller. get_the_title(), get_permalink(), etc all work as expected. So this is looking pretty edge-case.


Create docs for consolidate example

Description
Add consolidate example to Account and Addresses section?
Motivation
Guess consolidating is quite important ;)
Are you planning to do it yourself in a pull request?
Yes

Would be nice to have it for 1.0 indeed


Better Post-Login experience

Once a user logs in, it might be a good idea to remove the login form and replace the login button with a logout one, would seriously improve the UX.

The next update will be integrating song requesting for supporters - as such, that already has a planned UI overhaul for the little popup window in the pipeline 🙂

Nice, can't wait


Proposal: simple context menu entries in app manifest

Proposal: simple context menu entries in app manifest

Summary
Support declaring a context menu entry with a specific icon, display name and executable in app manifest instead of requiring to provide an IExplorerCommand implementation.
Rationale

Currently, to have an entry in the new Windows 11 context menu (instead of the legacy Windows 10 menu, which shows up under "Show more options"), one needs to implement IExplorerCommand
This is needlessly complicated if the context menu entry is just a simple entry which launches the app executable with a certain command line argument (which I'd wager is most context menu entries out there, people who need more already have a shell extension).
It also requires implementing it in C++, as C# and other managed languages are heavily discouraged by Microsoft for in-process shell extensions (and IExplorerCommand indeed only supports in-process activation).
This will cause the DLL to be loaded in any process with a save as/browse dialog, causing potential runtime version conflicts. So, even if one disregards the MS recommendations and goes ahead with a managed implementation, this will cause problems in the real world.
This means that, in practice, many projects might skip this altogether because of implementation complexity.

Scope

Capability
Priority

This proposal will allow developers to register context menu entries that launch an app when clicked
Must

This proposal will allow developers to pass command line arguments including the selected file, if any, to the Win32 app
Must

This proposal will allow developers to activate UWP apps in response to a context menu entry click
Should

This proposal will allow end users to show/hide undesired context menu entries from the Settings app
Could

This proposal will allow end users to register their own context menu entries
Won't

Important Notes
The app manifest entry could look something like this:
<desktop4:Extension Category="windows.fileExplorerContextMenus">
    <desktop4:FileExplorerContextMenus>
        <!-- add a context menu entry on a folder, with no icon -->
        <desktop5:ItemType Type="Directory">
            <desktop7:SimpleVerb Id="OpenFolderWin32" Executable="myappwin32.exe" EntryPoint="Windows.FullTrustApplication" Parameters='--folder "%1"'>
                <uap:DisplayName>Open folder in App</uap:DisplayName>
            </desktop7:SimpleVerb>
        </desktop5:ItemType>

        <!-- add a context menu entry on the desktop wallpaper, activating a UWP app -->
        <desktop5:ItemType Type="Directory\Background">
            <desktop7:SimpleVerb Id="OpenBackgroundUWP" Executable="myappuwp.exe" EntryPoint="Contoso.App">
                <uap:DisplayName>Do something</uap:DisplayName>
            </desktop7:SimpleVerb>
        </desktop5:ItemType>

        <!-- add a context menu entry on the desktop wallpaper, opening a Win32 app without command line parameters -->
        <desktop5:ItemType Type="Directory\Background">
            <desktop7:SimpleVerb Id="OpenBackgroundWin32" Executable="myappwin32.exe" EntryPoint="Windows.FullTrustApplication">
                <uap:DisplayName>Do something but with Win32</uap:DisplayName>
            </desktop7:SimpleVerb>
        </desktop5:ItemType>

        <!-- add a context menu entry on the desktop wallpaper, opening a Win32 app with command line parameters -->
        <desktop5:ItemType Type="Directory\Background">
            <desktop7:SimpleVerb Id="OpenBackgroundWin32" Executable="myappwin32.exe" EntryPoint="Windows.FullTrustApplication" Parameters='--background'>
                <uap:DisplayName>Do something but with Win32</uap:DisplayName>
            </desktop7:SimpleVerb>
        </desktop5:ItemType>

        <!-- add a context menu entry on all files, using a package resource as display name -->
        <desktop5:ItemType Type="*">
            <desktop7:SimpleVerb Id="OpenFileWin32" Executable="myappwin32.exe" EntryPoint="Windows.FullTrustApplication" Parameters='--file "%1"'>
                <uap:DisplayName>ms-resource:OpenFileVerbDescription</uap:DisplayName>
            </desktop7:SimpleVerb>
        </desktop5:ItemType>

        <!-- add a context menu entry on text files, with an icon -->
        <desktop5:ItemType Type=".txt">
            <desktop7:SimpleVerb Id="OpenTxtWin32" Executable="myappwin32.exe" EntryPoint="Windows.FullTrustApplication" Parameters='--text-file "%1"'>
                <uap:DisplayName>Edit file in App</uap:DisplayName>
                <uap:Logo>Assets\EditFile.ico</uap:Logo>
            </Verb>
        </desktop5:ItemType>

        <!-- add a context menu entry on log files, with a PNG file as icon -->
        <desktop5:ItemType Type=".log">
            <desktop7:SimpleVerb Id="OpenLogWin32" Executable="myappwin32.exe" EntryPoint="Windows.FullTrustApplication" Parameters='--log-file "%1"'>
                <uap:DisplayName>View log file in App</uap:DisplayName>
                <uap:Logo>Assets\ViewLog.png</uap:Logo>
            </desktop7:SimpleVerb>
        </desktop5:ItemType>

        <!-- add a context menu entry on C++ files, activating a UWP app -->
        <desktop5:ItemType Type=".cpp">
            <desktop7:SimpleVerb Id="OpenCodeUWP" Executable="myappuwp.exe" EntryPoint="Contoso.App">
                <uap:DisplayName>Edit C++ in App</uap:DisplayName>
            </desktop7:SimpleVerb>
        </desktop5:ItemType>
    </desktop4:FileExplorerContextMenus>
</desktop4:Extension>

In command line arguments, %1 is a placeholder for the file/folder name, if any.
One could then use activation APIs to determine if the app was launched in response to a context menu entry (or for Win32 apps, they can read command line arguments). It wouldn't be unlike the File activation API:
 namespace Windows.ApplicationModel.Activation
 {
     public enum ActivationKind
     {
+        ContextMenu = 1023
     }

+    public sealed class ContextMenuTargetActivatedEventArgs : IActivatedEventArgsWithUser
+    {
+        public string ItemType { get; }
+        public string Verb { get; }
+        public IStorageItem? File { get; } // null if something like Directory\Background was activated
+    }
 }

 namespace Windows.UI.Xaml
 {
     public class Application
     {
+        protected virtual void OnContextMenuTargetActivated(ContextMenuTargetActivatedEventArgs args);
     }
 }

Command line arguments are supported to simplify the migration of an existing Win32 app to MSIX, as the app might already have command line arguments for context menu entries and adding code checking for the activation kind is not possible or too costly. It's not needed for UWP/Windows App SDK apps, which use activation from the start.
Open Questions
None that I can think of.

FYI @chrisguzak. Agree that the current approach is unnecessarily complicated.

There are some bigger questions to ask in relation to this.

Assuming there was a reason for the change in the Windows 11 context menu, it would be good to understand why it was made before trying to work around it.
When there are primary and secondary lists, it's human nature to want the thing you've made to be on the primary list. What would stop every developer from trying to put their entry on the main list? Of course, to them, it's the most important thing but it would easily become a commons and result in everybody losing out in the long run.
Are there any metrics available on how many apps actually extend the context menu? (how many people would benefit if this was made easier?)
Is there any other feedback from other developers/apps that would like to do this but haven't/can't because it is so (too?) complicated?
Rather than changing the way the functionality is implemented, would it be suitable to instead make it quicker and easier to implement? I'm thinking of having a template (possibly through template studio--this is the kind of thing it was created for) that would produce the basic structure of the project and files needed and then only require the developer to provide the (for example) path/filename, icon, and details of the parameters to pass.

Assuming there was a reason for the change in the Windows 11 context menu, it would be good to understand why it was made before trying to work around it.

The change was made mostly to bring organization to the menu (for example when an app has more than 1 entry, it's automatically put in a submenu, and all app entries are in the same region of the menu). We are not working around it, there is a documented way to add your own entries in there, it's just needlessly complicated.

When there are primary and secondary lists, it's human nature to want the thing you've made to be on the primary list. What would stop every developer from trying to put their entry on the main list? Of course, to them, it's the most important thing but it would easily become a commons and result in everybody losing out in the long run.

If there is more than 1 entry from a single app, Windows 11 automatically puts them in a submenu. And I don't see this as a valid point, seeing as there's already a way to add entries there.
Allowing users to hide entries is also a could in the scope table.

Are there any metrics available on how many apps actually extend the context menu? (how many people would benefit if this was made easier?)

To name a few:

Visual Studio (when right clicking a folder)
Visual Studio Code (when right clicking a folder)
Windows Terminal (when right clicking a folder or the desktop background)
They currently implement IExplorerCommand for this because they are packaged but could completely remove that if this feature was implemented.
ShareX (when right clicking a file)
The packaged versions of ShareX are completely missing this because it used not to be possible and they do not want to add C++ code for IExplorerCommand.
Intel/AMD/Nvidia graphics (when right clicking the desktop background)
x64dbg (when right clicking a .exe/.dll file)
Jetbrains dotPeek (when right clicking a .exe/.dll file)

These are the ones on my system, there are most likely many more out there.

Is there any other feedback from other developers/apps that would like to do this but haven't/can't because it is so (too?) complicated?

See ShareX above.

Rather than changing the way the functionality is implemented, would it be suitable to instead make it quicker and easier to implement? I'm thinking of having a template (possibly through template studio--this is the kind of thing it was created for) that would produce the basic structure of the project and files needed and then only require the developer to provide the (for example) path/filename, icon, and details of the parameters to pass.

This still adds deployment and build complexity. You need to ship a different binary for every architecture, even though your app would otherwise be architecture neutral. You also need the C++ build tools installed, and to have your build system capable of building C++ (e.g., packaged Electron apps can't as easily add C++ stuff to their build). It's a maintenance burden for people who do not know C++ well, as well.

The change was made mostly to bring organization to the menu (for example when an app has more than 1 entry, it's automatically put in a submenu, and all app entries are in the same region of the menu).

What's the source for this explanation?
This is also different--as I understand it--from the issue defined in the OP, which was about the "show more options" menu added in Windows 11.

If there is more than 1 entry from a single app, Windows 11 automatically puts them in a submenu. And I don't see this as a valid point, seeing as there's already a way to add entries there.

My point was about this being a request to make it easier to add entries to the context menu. If such a change was made and it was then promoted on the grounds of "we've made it easier to add windows explorer context menu entries." Many developers would see this as an excuse to add such an entry for their app too, regardless of whether it makes sense or adds value to the people using the app. (I make this claim based on years of watching developers add pointless and irrelevant features to their apps because they're new and/or easy to do. "What's the harm they feel?" And then users end up with a context menu full of irrelevant entries and the Windows team has to come up with another way to reorganize and prioritize all the entries.
My request for metrics was meant for the Windows team.
Just listing 7 apps (3 from Microsoft) is unlikely to be enough to raise the priority of this.
Knowing how many people/apps/developers are affected by the issue will help understand how this request might be prioritized.

Template Studio is VS-only (and an optional extension install), so a template added there would not reach a significant part of the audience.

Is being VS-only a big blocker for this given that it's a Windows-only feature?
My reason for suggesting this as something that could be added to Template Studio is that this feels like a very niche request and so is unlikely to be something that would quickly get up the priority queue for a change in the build system. Adding templates would be a couple of days' effort at most and would then at least make the process simpler. It's also something that could be community-contributed rather than waiting for a space in the WindowsAppsSDK backlog. I suspect (and hope) there are a lot of other things that are a higher priority and would be done first.
Still, this is all good extra context for those who make the decisions.

What's the source for this explanation?
This is also different--as I understand it--from the issue defined in the OP, which was about the "show more options" menu added in Windows 11.

Right here. All of the points listed are organizational changes. The OP is about being able to show up in the new Windows 11 menu, not the old one under "show more options", without needing to implement IExplorerCommand in a shell extension, which is needlessly complicated.

And then users end up with a context menu full of irrelevant entries and the Windows team has to come up with another way to reorganize and prioritize all the entries.

That's why "This proposal will allow end users to show/hide undesired context menu entries from the Settings app" is a could. Collapsing apps with multiple verbs under a dropdown also helps it from being invasive.

My request for metrics was meant for the Windows team.
Just listing 7 apps (3 from Microsoft) is unlikely to be enough to raise the priority of this.
Knowing how many people/apps/developers are affected by the issue will help understand how this request might be prioritized.

Then it's not possible for me to answer that in an appropriate manner, since I am not a MS employee with access to Windows telemetry that could provide this data. I can only speak of personal experience.

Is being VS-only a big blocker for this given that it's a Windows-only feature?

Yes, as there are many Windows apps which are being developed without Visual Studio. For example Electron, React Native, Rust, MSYS2/mingw, etc. In all of those except one, integrating C++ code is a pain.

Please provide an option to completely disable all third-party context menu items and just have the default ones.

Allowing users to hide entries is also a could in the scope table.

Thank you for the thorough suggestion! One thing I want to quickly clarify is that IExplorerCommand will run Out of Proc when invoked in the way described in the blog post and sample.

Thanks for the info! I tried to find information of whether IExplorerCommand is in-proc or out-of-proc, and I've only found documentation mentioning in-proc. I imagine out-of-proc support is something new for Windows 11?
This does remove concerns with regards to managed implementations (which is awesome), but I think there is still value to simplifying it, as this is currently additional complexity to adopting packaged support for some apps (where the unpackaged version just directly registers context menu verbs in the registry instead of using a shell extension). Or just additional complexity in general compared to writing a few entries in XML.

Out-of-proc support extends back into Windows 10 for some time but the documentation needs work.
I agree we should consider making this easier. We're discussing options internally.

We're chatting in another thread already but just want to mirror here.
There doesn't appear to be a good story for folks that were previously just doing:
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\exefile\shell\symchk]

[HKEY_CLASSES_ROOT\exefile\shell\symchk\command]
@=hex(2):63,00,6d,00,64,00,20,00,2f,00,63,00,20,00,22,00,20,00,22,00,43,00,3a,\
  00,5c,00,50,00,72,00,6f,00,67,00,72,00,61,00,6d,00,20,00,46,00,69,00,6c,00,\
  65,00,73,00,20,00,28,00,78,00,38,00,36,00,29,00,5c,00,57,00,69,00,6e,00,64,\
  00,6f,00,77,00,73,00,20,00,4b,00,69,00,74,00,73,00,5c,00,31,00,30,00,5c,00,\
  44,00,65,00,62,00,75,00,67,00,67,00,65,00,72,00,73,00,5c,00,78,00,36,00,34,\
  00,5c,00,73,00,79,00,6d,00,63,00,68,00,6b,00,2e,00,65,00,78,00,65,00,22,00,\
  20,00,2f,00,69,00,66,00,20,00,22,00,25,00,31,00,22,00,20,00,2f,00,73,00,75,\
  00,20,00,22,00,25,00,5f,00,4e,00,54,00,5f,00,53,00,59,00,4d,00,42,00,4f,00,\
  4c,00,5f,00,50,00,41,00,54,00,48,00,25,00,22,00,20,00,22,00,00,00

Right now, it appears I have to write an app, package it (or sparse package it), sign it, deploy it, all to simply shell out to an executable.

This is why we have "Show more options" - it's important to recognize that it's still possible to add a Shell extension in this way, but it's no longer easy to put it at the top level of the context menu.

I'm going to imagine at some point in the future there'll be an app that registers its own shell extension under * and allows you to add custom verbs to it. IExplorerCommand supports submenus (EnumSubCommands) so it's possible to have multiple entries in there.

yeah I was planning on doing something like that next week :) LMK if you want to collab :)

Any news?

Any progress?


Index controls

Simple index controls

add
delete

Compound index controls

add
delete

Also an "ensure" would be good (create or update as needed).


Add cabal.project to allow building with cabal

The posix-waitpid and linux-ptrace dependendencies on Hackage are quite
outdated which is why stack.yaml has them as git dependencies instead.
When trying to build with cabal we just get a bunch of solver errors. So I
fix this by adding the cabal equivalent of stack.yaml, cabal.project, with
the git dependencies.

This change is

OK! I added 2 comments as reminders to keep things in sync.


README: Links to license files go nowhere

Hi folks,
Thank you very much for providing this repository. I observed that the links to license files at the end of the README.md file does not go where they ought to go. Cheers!

Thank you for the notice.   We are in process of updating with a newer version, with many new features.


MAINT: Add dependabot yaml

Closes #127
@Carreau I think this should already be enough to get dependabot to open PRs. At least for other projects in orgs without Dependabot added I think it has made it start opening PRs.
Also removes some cruft from the README and adds a release.yml in case we want to use autogenerated release notes at some point.

Good call, pushed that change @matthewfeickert !

Thanks @larsoner!


Maybe use bash-prexec for bash (still DEBUG trap though)

https://github.com/rcaloras/bash-preexec

Not really into this, as bash-preexec modifies HISTCONTROL and remove ignorespace. Its a privacy issue, even if this app is (for now) only designed for personal use. Someone used to hide commands by prepending a space would be annoyed to see these very commands appended into his extended history.


libmodsecurity bump version and fix CVE-2020-15598

Motivation for this change
I am using this software and there are patches.
There is a dispute whether this is a vuln or not:

https://coreruleset.org/20200914/cve-2020-15598/
https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/modsecurity-regular-expressions-and-disputed-cve-2020-15598/
NIST has not published anything yet: https://nvd.nist.gov/vuln/detail/CVE-2020-15598
related upstream issue: https://github.com/SpiderLabs/ModSecurity/issues/2401

Things done
Bumped the version. Applied the patch. Rebuilt the server. The website is still working and modsecurity still blocks shady requests.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

CC maintainer: @Izorkin

If upstream doesn't consider it an issue where does this patch come from and what exactly does it do?

Patch was taken from the corerulesset blog post and the PR should reproduce the recommended build instructions.

[...] Since Trustwave does not want to release a new version, we are providing you with a backported patch for the vanilla ModSecurity 3.0.4. Please note that this is the patch that the Debian packaging team as well as NGINX will be using for their security updates.
cve-2020-15598.patch
This patch is a backport of the git commit, that brings back the former behavior of the regular expression matching. [...]

As I understand the issue: Matching behaviour changed from version 2.x to 3.x, and some rules (in the coreruleset) can now trigger resource exhaustion. CRS team wants modsecurity to fix it (that is what the patch should do). Modsecurity team wants CRS to fix their rules.

CRS project here: Vendor has reverted behavior but refuses to release an update. The patch in question is derived from the official tree of the vendor. It's also the patch that NGINX Plus uses and the one that Debian is going to use (but has not pushed to the repo yet).


Fix: #274

Problem
Issue with saving a custom store. Would get an error thrown.
Solution
Needed to rename custom store save data variable to improve readability. Also needed to initialize the custom store in the original saveData variable. Created a condition to limit the creation to only once and only if there are custom stores with save functionality.

Doesn't look like it passed the CI build. Are you able to see what's breaking?

Doesn't look like it passed the CI build. Are you able to see what's breaking?

just merged when you said that lol. Yeah I looked at it and it's a problem with pnpm versions in CI, I need to change something in config files because it breaks every time there's an update, so for now I just merged this PR anyway

Thanks Liana. This is the first time I contribute to a public repo, so still learning how it all works.

I tried downloading the latest narrat and it doesn't look like the fix is there yet. I'm not sure if I'm doing something wrong or you just haven't published yet.


🕵 Investigate the Usage of an Alternative CNI for EKS cluster to be built in MP

As a DevOps Engineer, I would like to investigate which alternative container network interface implementation I can use in preference to deault vpc-cni which ships with EKS.
vpc-cni while simple to use, assigns pod IP addresses out of the general subnet CIDR ranges. Since we're sharing networking with the rest of Modernisation Platform, this is not a viable strategy as there are not enough IP addresses available in those CIDR ranges for our workloads.
Definition of Done:

[ ] Identify an appropriate replacement for the CNI plugin to use instead of vpc-cni
[ ] Determine the best way to deploy
[ ] Capture the decision in our ADR

VPC CNI has been chosen


Development Server not Starting (craco start)

What's happening
When using craco start instead of react-scripts start, the development server gets stuck at the console output Starting the development server... but never starting
CRACO version
"@craco/craco": "^7.1.0",
"react-scripts": "5.0.1",
on nodejs 18.16.1
CRACO config
module.exports = {
  webpack: {
    configure: (webpackConfig) => {
      webpackConfig.optimization.splitChunks = {
        chunks: "all",
      };

      return webpackConfig;
    },
  },
};

package.json
  "dependencies": {
    "@azure/msal-browser": "2.28.3",
    "@azure/msal-react": "1.4.7",
    "@emotion/react": "11.10.0",
    "@emotion/styled": "11.10.0",
    "@hookform/error-message": "^2.0.0",
    "@hookform/resolvers": "^2.9.7",
    "@iconify/react": "^3.2.2",
    "@mui/icons-material": "5.8.4",
    "@mui/lab": "5.0.0-alpha.93",
    "@mui/material": "5.9.3",
    "@mui/x-date-pickers": "5.0.0-beta.5",
    "@reduxjs/toolkit": "^1.8.3",
    "axios": "^0.27.2",
    "classnames": "^2.3.2",
    "date-fns": "^2.29.1",
    "dayjs": "^1.11.7",
    "env-cmd": "^10.1.0",
    "i18next": "^21.8.16",
    "i18next-browser-languagedetector": "^6.1.4",
    "i18next-http-backend": "^1.4.1",
    "include-media": "^2.0.0",
    "js-cookie": "^3.0.1",
    "js-file-download": "^0.4.12",
    "jwt-decode": "^3.1.2",
    "lodash": "^4.17.21",
    "moment": "^2.29.4",
    "mui-tel-input": "^2.1.0",
    "prop-types": "^15.8.1",
    "react": "^18.2.0",
    "react-background-video-player": "^1.1.8",
    "react-dom": "^18.2.0",
    "react-dropzone": "^14.2.2",
    "react-helmet-async": "^1.3.0",
    "react-hook-form": "^7.34.0",
    "react-i18next": "^11.18.3",
    "react-player": "2.11.0",
    "react-query": "^3.39.3",
    "react-redux": "^8.0.2",
    "react-router-dom": "^6.3.0",
    "react-scripts": "5.0.1",
    "react-toastify": "^9.0.8",
    "react-transition-group": "^4.4.5",
    "simplebar": "^5.3.8",
    "simplebar-react": "^2.4.1",
    "store2": "^2.14.2",
    "tinycolor2": "^1.4.2",
    "use-debounce": "^8.0.4",
    "uuid": "^9.0.0",
    "web-vitals": "^2.1.4",
    "workbox-precaching": "^6.6.0",
    "workbox-routing": "^6.6.0",
    "workbox-strategies": "^6.6.0",
    "workbox-window": "^7.0.0",
    "xlsx": "^0.18.5",
    "yup": "^0.32.11"

Additional information
Using eslint
If I change *only" craco start back to react-scripts start, the server starts again, however eslint warnings don't appear in the terminal.
If I also change craco test back to react-scripts test, eslint warnings get logged in the terminal again, even when only running  react-scripts start.
Console output:
> craco start --verbose

Project root path resolved to:  C:\Users\emkayy\FrontEnd
Override started with arguments:  [
  'C:\\Program Files\\nodejs\\node.exe',
<EMAIL_ADDRESS>  '--verbose'
]
For environment:  development
Config file path resolved to:  C:\Users\emkayy\FrontEnd\craco.config.js
Applied craco config plugins.

Logging the webpack configuration in the craco.config.js looks ok.

All craco is doing here is injecting the webpack config you specified into create react app. This would be a webpack issue rather than a craco one.

I also encountered the same problem, you can consider the development of webpack configuration notes, to be completed after the development of the build, uncomment, often webpack configuration is online some of the details of optimization, the development process can be temporarily ignored


Code refactoring?

It would be nice to have the code more organized with something like Browserify for example.
Would you be open to this?

It isn't a large enough project to need Browserify.  We use all the scripts we're loading, and minification will bring them in a single file.
At the moment, we don't even have a task runner.

I didn't suggest it for minification purposes (since that's unnecessary), but I think we would benefit from the CommonJS require system.
Currently the main.js file has over 800 LOC in it. It's a bit hard to read. Even if we split it into 5 file we would have to keep including them in the index.html page. By using something like Browserify we can easily split it up and even possibly use modules from NPM.
It would indeed require a bundling step, but that wouldn't be so hard in my opinion.

We've got a fork to move it to Angular.  https://github.com/SunburnedGoose/DIM
My goal atm is to break up main.js into directives and services.  I feel your pain looking at main.js.

That's a good effort, however this will eliminate the need to including a lot of files by hand: https://github.com/SunburnedGoose/DIM/blob/master/window.html#L72-L89
Also (this is just my opinion) I think it would be better to keep this with just raw JavaScript, since Angular may be a barrier to other contributions (such as it would be for me).
I don't mean to be rude, I really appreciate your work.

Just found this repo today, so hopefully I'll be making some PRs soon...I agree with @alessioalex  that Angular is probably a little heavy-handed for a project like this. That said, a basic AMD approach using something like RequireJS might make a bit more sense, and would introduce the desired modularity. I haven't used CommonJS personally, but from some initial digging, it seems like it might be a bit much as well.

Either you use  or require(..), gotta refer to the code somewhere by hand.
Raw javascript was building up friction the larger it was growing.  Sometimes the HTML didn't match the data model because there was an error.  Looking at main.js, you see that about 2/3rds of it is manipulating HTML on load and while responding to events.  We're going to cut that way down and make the view model logic clearer.

Either you use scripts**

angularjs port is in development.
https://github.com/SunburnedGoose/DIM
will allow for a more organized workflow.


0.6.1 update

Jira issue
Upstream repo
Target channel: Snowflake ❄

First build for Windows
v0.6.1 requires pytorch >=2.0, which is unavailable for ppc64le

linux-ppc64le failed:
ModuleNotFoundError: No module named 'torch.utils.data.datapipes.iter.sharding'
Looks like pytorch v1.13.1 can be incompatible


Expose entity tags

SpongeCommon | SpongeAPI
Hey,
Here is a small PR trying to expose entity tags which can be used as selector arguments in commands.
Until now they were not exposed in the API.
Learn more about tags: https://minecraft.gamepedia.com/Scoreboard#Tags
Thanks for reading this PR.

Manually merged (with minor updates), thanks!


[Feature] Wuthering waves (wuwa) support

Checklist

[X] I have already read docs/Tips.md, but my feature is not implemented.
[X] My suggested feature was not mentioned by others, and it is not a duplicate feature.

Summary
It should be okay, right?
Solution or Design
No idea, i am not a designer, but following the same one for the others should be okay.

I think it shuld be better to make rename the luncher to Multi Game-Luncher and add another games like ToF and other, the design is here and i dont think that need be much redisign for it.

绝了

Same as https://github.com/Scighost/Starward/issues/793#issuecomment-2131164740


App crashes after connect attempt

Since some days the App crashes after I try to connect to my Raspberry pi. I get the Message "App angehalten", since it's a german Android Phone (Samsung Galaxy S6 with Android 6). I connect via username and password. I don't know if this is a Marshmallow Issue.
I have no problem in the log:
22:39:31.853 [AsyncTask #7] INFO  d.e.r.ssh.impl.queries.MemoryQuery - Querying memory information...
22:39:31.933 [AsyncTask #7] INFO  d.e.rpicheck.ssh.impl.RaspiQuery - Disconnecting from host <IP_ADDRESS>.
22:39:58.299 [main] WARN  d.e.r.activity.SettingsActivity - Enabling debug logging. Be warned that the log file can get huge because of this.
22:39:58.305 [main] DEBUG d.e.r.activity.helper.LoggingHelper - Logging was configured, debug logging enabled: true
22:40:02.425 [main] DEBUG d.e.rpicheck.activity.MainActivity - Load average preference: FIVE_MINUTES
22:40:02.479 [AsyncTask #4] INFO  d.e.rpicheck.ssh.impl.RaspiQuery - New RaspiQuery for host: <IP_ADDRESS>
22:40:02.479 [AsyncTask #4] INFO  d.e.rpicheck.ssh.impl.RaspiQuery - Connecting to host: <IP_ADDRESS> on port 22.
22:40:02.509 [AsyncTask #4] INFO  d.e.rpicheck.ssh.impl.RaspiQuery - Using no host key verification.
22:40:03.260 [AsyncTask #4] DEBUG d.e.rpicheck.ssh.impl.RaspiQuery - Querying vcgencmd...
22:40:03.276 [AsyncTask #4] DEBUG d.e.rpicheck.ssh.impl.RaspiQuery - Checking vcgencmd location: vcgencmd
22:40:03.327 [AsyncTask #4] DEBUG d.e.rpicheck.ssh.impl.RaspiQuery - Path check output:  22:40:04 up 1 day,  5:04,  1 user,  load average: 0,09, 0,07, 0,05
22:40:03.328 [AsyncTask #4] INFO  d.e.rpicheck.ssh.impl.RaspiQuery - Found vcgencmd in path: vcgencmd.
22:40:03.505 [AsyncTask #4] INFO  d.e.rpicheck.ssh.impl.RaspiQuery - Querying core volts...
22:40:03.686 [AsyncTask #4] DEBUG d.e.r.ssh.impl.queries.FirmwareQuery - Querying firmware version, vcgencmd path=vcgencmd
22:40:03.775 [AsyncTask #4] DEBUG d.e.r.ssh.impl.queries.FirmwareQuery - Firmware version: Copyright (c) 2012 Broadcom
22:40:03.776 [AsyncTask #4] INFO  d.e.r.s.i.queries.LoadAverageQuery - Querying load average for time period FIVE_MINUTES
22:40:03.898 [AsyncTask #4] DEBUG d.e.r.s.i.queries.LoadAverageQuery - Checking line:  22:40:04 up 1 day,  5:04,  1 user,  load average: 0,09, 0,07, 0,05
22:40:03.899 [AsyncTask #4] DEBUG d.e.r.s.i.queries.LoadAverageQuery - Skipping line:  22:40:04 up 1 day,  5:04,  1 user,  load average: 0,09, 0,07, 0,05
22:40:03.900 [AsyncTask #4] DEBUG d.e.r.s.i.queries.LoadAverageQuery - Checking line: 0.09 0.07 0.05 1/147 24739
22:40:03.901 [AsyncTask #4] DEBUG d.e.r.s.i.queries.LoadAverageQuery - Checking line: 5
22:40:03.903 [AsyncTask #4] INFO  d.e.r.ssh.impl.queries.UptimeQuery - Querying uptime...
22:40:04.003 [AsyncTask #4] DEBUG d.e.r.ssh.impl.queries.UptimeQuery - Skipping line:  22:40:04 up 1 day,  5:04,  0 users,  load average: 0,09, 0,07, 0,05
22:40:04.005 [AsyncTask #4] INFO  d.e.r.ssh.impl.queries.MemoryQuery - Querying memory information...
22:40:04.084 [AsyncTask #4] INFO  d.e.rpicheck.ssh.impl.RaspiQuery - Disconnecting from host <IP_ADDRESS>.
22:40:13.650 [main] DEBUG d.e.r.activity.helper.LoggingHelper - Logging was configured, debug logging enabled: true
22:40:13.668 [AsyncTask #2] DEBUG d.e.rpicheck.activity.MainActivity - Device cursor rows: 1
22:40:13.814 [main] DEBUG d.e.rpicheck.activity.MainActivity - Spinner item selected: pos=0, id=1
22:40:15.667 [main] DEBUG d.e.rpicheck.activity.MainActivity - Saving instance state (current device)
22:40:15.667 [main] DEBUG d.e.rpicheck.activity.MainActivity - Saving new instance of all devices.

Because the app logfile looks normal, I can't really figure out whats going on.
Could you try reinstalling the app?

I have done deinstalling and reinstalling the app before i made this post :-) For me it looks like, that the App could successfully pull all the required infos from the Server. But cant parse the Data. I have seen that there was some updates in the last time for some Packages on Debian/Raspbian. Maybe this updates break something in the returned data format what the app cant read.
Is there a way, where i can pull the needed information, module for module? Maybe only one information is in the wrong format.

You could try to get a full Logcat while the app is crashing, so that we get the stacktrace for the crash.


Convert empty string to float error

Hi Robert,
First thanks for maintaining this integration! I just installed it and I seem to be running into an issue where it is trying to convert an empty string to a float specifically in this line: https://github.com/RobertD502/home-assistant-iocare/blob/f90d1037e4a79ea6471fd6295cedf8c98bd09cbf/custom_components/coway/sensor.py#L241
I can reproduce the error seen in the log if I just switch the Preset Mode to auto  or even toggle the light on/off which the commands end up not executing because of the error.  Seems to be related to AQI and I do notice for some reason my sensor.airmega_aqi is showing unavailable.
Is this a bug or is there something perhaps I'm missing regarding the setup? Thanks!

Please post any related logs here. The preset mode and light switch are separate from the AQI sensor, so this line shouldn't be having any impact on those function. Regarding the sensor being unavailable - this isn't a bug. If the coway servers are reporting your purifier as not connected to their servers then the entities related to the purifier become unavailable. They become available again once Coway's servers report the purifier as being connected to their servers again.
To further narrow down your issue, as said before, please post any related logs (including anything regarding changing the preset mode, turning light on/off).
Also, just to make sure - are you using either a 300S or 400s purifier? If you are using their new purifier (I believe it is the 250S?), I can't confirm that this integration will work as I don't have access to one/wrote the code for the 300S and 400s.

I have the AirMega 400S White. Software Version : MCU: V<IP_ADDRESS>( 2022.03.23 16:31)
I don't believe I saw anything in the logs regarding being disconnected but I can't rule it out.
Here' a complete log after startup and trying to toggle the light entity

2022-03-24 00:13:04 ERROR (MainThread) [homeassistant.components.sensor] Error adding entities for domain sensor with platform coway
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 382, in async_add_entities
await asyncio.gather(*tasks)
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 614, in _async_add_entity
await entity.add_to_platform_finish()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 801, in add_to_platform_finish
self.async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 553, in async_write_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 590, in _async_write_ha_state
state = self._stringify_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 559, in _stringify_state
if (state := self.state) is None:
File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 372, in state
value = self.native_value
File "/config/custom_components/coway/sensor.py", line 241, in native_value
return round(float(self.coordinator.data[self._device].quality["air_quality_index"]), 1)
ValueError: could not convert string to float: ''
2022-03-24 00:13:04 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up coway platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 257, in _async_setup_platform
await asyncio.gather(*pending)
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 382, in async_add_entities
await asyncio.gather(*tasks)
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 614, in _async_add_entity
await entity.add_to_platform_finish()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 801, in add_to_platform_finish
self.async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 553, in async_write_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 590, in _async_write_ha_state
state = self._stringify_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 559, in _stringify_state
if (state := self.state) is None:
File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 372, in state
value = self.native_value
File "/config/custom_components/coway/sensor.py", line 241, in native_value
return round(float(self.coordinator.data[self._device].quality["air_quality_index"]), 1)
ValueError: could not convert string to float: ''
2022-03-24 00:13:36 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 134, in _handle_refresh_interval
await self._async_refresh(log_failures=True, scheduled=True)
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 265, in _async_refresh
update_callback()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 325, in _handle_coordinator_update
self.async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 553, in async_write_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 590, in _async_write_ha_state
state = self._stringify_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 559, in _stringify_state
if (state := self.state) is None:
File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 372, in state
value = self.native_value
File "/config/custom_components/coway/sensor.py", line 241, in native_value
return round(float(self.coordinator.data[self._device].quality["air_quality_index"]), 1)
ValueError: could not convert string to float: ''
2022-03-24 00:13:40 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139928036300496] could not convert string to float: ''
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 192, in handle_call_service
await hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 1636, in async_call
task.result()
File "/usr/src/homeassistant/homeassistant/core.py", line 1673, in _execute_service
await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 204, in handle_service
await self.hass.helpers.service.entity_service_call(
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 671, in entity_service_call
future.result() # pop exception if have
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 949, in async_request_call
await coro
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 708, in _handle_entity_call
await result
File "/config/custom_components/coway/switch.py", line 76, in async_turn_off
await self.coordinator.async_request_refresh()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 141, in async_request_refresh
await self._debounced_refresh.async_call()
File "/usr/src/homeassistant/homeassistant/helpers/debounce.py", line 78, in async_call
await task
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 165, in async_refresh
await self._async_refresh(log_failures=True)
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 265, in _async_refresh
update_callback()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 325, in _handle_coordinator_update
self.async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 553, in async_write_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 590, in _async_write_ha_state
state = self._stringify_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 559, in _stringify_state
if (state := self.state) is None:
File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 372, in state
value = self.native_value
File "/config/custom_components/coway/sensor.py", line 241, in native_value
return round(float(self.coordinator.data[self._device].quality["air_quality_index"]), 1)
ValueError: could not convert string to float: ''

Ok, so based on the logs it is unavailable because the coway server is returning an empty AQI - this is odd behavior as it should never be empty.
Can you press and hold the WiFi button on your Airmega 400S to turn off WiFi, press the button again to turn WiFi on and wait for it to reconnect to your network. After it reconnects, restart Home Assistant and let me know if the problem persists. Forcing the purifier to reconnect to Coway's servers may help getting the AQI to update properly.

Still no luck :( It's getting late here but let me keep playing with it. I even tried commenting out adding the AQI entity in sensor.py which prevents the error but toggling fan/light don't seem to work. I'll keep poking at it and I'll let you know what I find. Thanks!

It was around 3 AM when I saw your initial post and I misread thinking all of the entities related to a purifier appeared as unavailable. Just to clarify, if the coway servers are reporting your purifier(s) as not connected then all entities related to the purifier(s) will be unavailable - this isn't a bug. However, in your case, just the AQI sensor entity is appearing as unavailable, which isn't normal behavior.
Where did you obtain your Software Version from? I checked all the server endpoints but wasn't able to find a key related to software version for either of my two purifiers. I think for starters we need to determine if your purifier is using a more recent version and, if so, if it uses different endpoints.

I just grabbed the version info from the ioCare app (using Android app) there was a tab called Device Management and under Product Software Version it listed that MCU information. On the back of the device it reads 400S (AP-2015E)
I was hoping it was some connection problem or it just needed more data before it started displaying the AQI information. I hoped I would wake up today seeing it magically work after a restart of HA but no dice :(.
I tried toggling WiFi, rejoining my WiFi to see if it would change behavior but it didn't seem to do much. The controls seem to work alright through the app so it could perhaps be a version change as you suspect?

Ahhhhh this info isn't listed in the iOS app. I'll have to dig out an android tablet I have laying around and check the version there. In the meantime, would you feel comfortable sharing your account with me via email (changing the password before I use it and then changing it back to your original password when I am done)? I'll need it to sniff out what endpoints the app is using for your purifier to determine if there is a difference.

Went to the Device Management tab (on the Android app) which shows the Max2 and Prefilter life. On that page I clicked on product settings. No Product Software Version listed on either the main Device Management tab or the product settings page.

My Airmega 400S is generating a similar error:
Logger: homeassistant.components.websocket_api.http.connection
Source: custom_components/coway/sensor.py:241
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 3:45:20 PM (1 occurrences)
Last logged: 3:45:20 PM

[547932243472] could not convert string to float: ''
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 192, in handle_call_service
    await hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1636, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1673, in _execute_service
    await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 204, in handle_service
    await self.hass.helpers.service.entity_service_call(
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 668, in entity_service_call
    future.result()  # pop exception if have
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 949, in async_request_call
    await coro
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 705, in _handle_entity_call
    await result
  File "/config/custom_components/coway/fan.py", line 151, in async_turn_off
    await self.coordinator.async_request_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 141, in async_request_refresh
    await self._debounced_refresh.async_call()
  File "/usr/src/homeassistant/homeassistant/helpers/debounce.py", line 78, in async_call
    await task
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 165, in async_refresh
    await self._async_refresh(log_failures=True)
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 265, in _async_refresh
    update_callback()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 325, in _handle_coordinator_update
    self.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 553, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 590, in _async_write_ha_state
    state = self._stringify_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 559, in _stringify_state
    if (state := self.state) is None:
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 372, in state
    value = self.native_value
  File "/config/custom_components/coway/sensor.py", line 241, in native_value
    return round(float(self.coordinator.data[self._device].quality["air_quality_index"]), 1)
ValueError: could not convert string to float: ''

Unfortunately, I won't be able to track down what is causing the issue without access to an affected account. I have tested it with a 400S that I have access to, but it isn't experiencing the reported issue. This leads me to believe that some 400s purifiers may be running a different software version and I will need to investigate the communication between purifier and coway servers in order resolve the problem.

Hi @RobertD502,
Sorry for the late reply. I am unsure if I am able to share my account but I was able to sniff some of the information which I think could prove useful.
I think there's a new endpoint CWIG0009.  Here's a sample JSON

{
    "body": {
      "updateState": "NONE",
      "newestMcuVer": null,
      "newestWifiVer": null,
      "isShowResveUpdateMessage": false,
      "curUpdateDate":<PHONE_NUMBER>868,
      "curMcuVer": "V<IP_ADDRESS>",
      "curWifiVer": "4889",
      "resveUpdateDate": 0
    },
    "header": {
      "accessToken": "REMOVED"
      "error_code": "",
      "error_text": "",
      "info_text": "",
      "login_session_id": "",
      "message_version": "",
      "refreshToken": "REMOVED",
      "result": true,
      "trcode": "CWIG0615"
    }
  }

Hope this helps! I'm guessing there will need to be a way to figure out if the 400S version is the newer ones with different endpoints? Hopefully that second JSON or there's some way to figure that out. Let me know if I can help more!

I have the CWIG0009 endpoint, but that isn't where AQI for the unit is obtained from. That endpoint serves outdoor air quality measurements and not those obtained by the purifier itself.
I can confirm that I do not have the CWIG0615 endpoint.

Oh, I see. odd then, perhaps these new 400S units need some update because all of the IAQ section definitely just comes in blank :(

What does your CWIA0120 endpoint json look like? In particular, the IAQ section:

The keys are there but the values except for dustpm10 are blank
"IAQ": [
      {
        "co2": "",
        "inairquality": "",
        "humidity": "",
        "dustpm25": "",
        "temperature": "",
        "dustpm10": "25",
        "dustpm1": "",
        "vocs": ""
      }
    ],
    "OAQ": []

just invoking iocareapi.py on its own, I also noticed the control commands don't work unless I add mqttRequest: True  in the request body.
def control(self, device, command, value):
        response = self._request(CONTROL, {
            'barcode': device.device_id,
            'dvcBrandCd': device.device_brand,
            'prodName': device.product_name,
            'dvcTypeCd': device.device_type,
            'funcList': [{
              'comdVal': value,
              'funcId': command
            }],
            'mqttDevice': True
        })

So with the info above, I kind of got it partially working on Home Assistant with some kludging:

I removed the AQI sensor (commenting out this entry https://github.com/RobertD502/home-assistant-iocare/blob/f90d1037e4a79ea6471fd6295cedf8c98bd09cbf/custom_components/coway/sensor.py#L36)
I added mqttDevice: True in iocareapi.py (appended to this line: https://github.com/RobertD502/pyiocare/blob/4db37d46de23aab30ebe9d408332c76585527710/iocare/iocareapi.py#L215)

The functionality seems to work sans the AQI information.

It should work without just the AQI sensor. Home Assistant tried to update info for all sensors when any control functions are called which is why yours hangs up when it fails to update the AQI in the process.
The MQTT portion shouldn't be necessary - it is also sent for my purifiers, but still works without it (the library was created with a lower iOS version app before they started using that option).

Does your report section look like this?:

If so, that means the purifier has to be reporting AQI to the Coway servers...the issue is where.

Yea, I have those too

But the only thing populated in the CWIA0120 section which has some semblance of AQI is that pm10Graph section, here's a snippet.

"pm10Graph":[
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290320",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290330",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290340",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"None",
            "place":"in",
            "msrdt":"202203290350",
            "graphvalue":"None"
         },
         {
            "graphhighvalue":"10",
            "place":"in",
            "msrdt":"202203290400",
            "graphvalue":"1"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290410",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290420",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290430",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290440",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290450",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"10",
            "place":"in",
            "msrdt":"202203290500",
            "graphvalue":"1"
         },
         {
            "graphhighvalue":"9",
            "place":"in",
            "msrdt":"202203290510",
            "graphvalue":"1"
         },
         {
            "graphhighvalue":"17",
            "place":"in",
            "msrdt":"202203290520",
            "graphvalue":"2"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290530",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290540",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"13",
            "place":"in",
            "msrdt":"202203290550",
            "graphvalue":"1"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290600",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290610",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290620",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290630",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290640",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290650",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290700",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"7",
            "place":"in",
            "msrdt":"202203290710",
            "graphvalue":"1"
         },
         {
            "graphhighvalue":"12",
            "place":"in",
            "msrdt":"202203290720",
            "graphvalue":"2"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290730",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290740",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"None",
            "place":"in",
            "msrdt":"202203290750",
            "graphvalue":"None"
         },
         {
            "graphhighvalue":"7",
            "place":"in",
            "msrdt":"202203290800",
            "graphvalue":"1"
         },
         {
            "graphhighvalue":"4",
            "place":"in",
            "msrdt":"202203290810",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"15",
            "place":"in",
            "msrdt":"202203290820",
            "graphvalue":"2"
         },
         {
            "graphhighvalue":"6",
            "place":"in",
            "msrdt":"202203290830",
            "graphvalue":"1"
         },
         {
            "graphhighvalue":"18",
            "place":"in",
            "msrdt":"202203290840",
            "graphvalue":"3"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290850",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"46",
            "place":"in",
            "msrdt":"202203290900",
            "graphvalue":"8"
         },
         {
            "graphhighvalue":"0",
            "place":"in",
            "msrdt":"202203290910",
            "graphvalue":"0"
         },
         {
            "graphhighvalue":"25",
            "place":"in",
            "msrdt":"202203290920",
            "graphvalue":"4"
         }
      ],

If I'm understanding it right, is "place":"in" here meaning the indoor AQI value? This might be the only thing provided now for these newer models perhaps? The values I have here seem to match this graph in the Home tab. At least comparing that fairly high value between 8 and 9 which looks to have that graphhighvalue of 46?

Maybe they'll update it down the line. For now it seems the newer models won't have AQI sensors - It is a bit puzzling why they chose to report PM10, but not anything else.

Just an update:
Even though my purifiers don't call the CWIG0615 endpoint, I made a manual call using the backend library and it returns:
{'updateState': '', 'newestMcuVer': None, 'newestWifiVer': None, 'isShowResveUpdateMessage': False, 'curUpdateDate': 0, 'curMcuVer': None, 'curWifiVer': None, 'resveUpdateDate': 0}
So, I can use this endpoint to determine if a user has an older model or the updated model.

@romedtino @theglus
I just released a <IP_ADDRESS> pre-release and need to have it tested with your new units before I release it in the wild.
First of all, I need to test if everything works out from onboarding, so, I will need for you all to
please delete the integration (from the Home Assistant integration page) if you currently have it set up.
After doing so, go to HACS, select Integrations and find the Coway integration card. On the coway card, select the overflow menu (three vertical dots) and click on redownload. After doing so be sure Show beta versions is selected. You should then be able to select version <IP_ADDRESS>. Install it, restart Home Assistant, and set the integration up through the Home Assistant integrations page.
Things I need tested:

Confirm that an AQI sensor is not created for your purifier
Test turning purifier on and off
Test turning purifier light on and off
Test setting purifier preset mode to auto and to night
Test setting purifier fan speed to each of the three levels as well as 0 (this should turn the purifier off)
Test setting the timer to 1 hour, 2 hours, 4 hours, and 8 hours. Also setting the timer to off.

I appreciate the help!

@RobertD502 Awesome, thanks for the quick turn-around time. I'm out of the house now, but I'll put together a testing report with my results when I get home.

I verified the list you mentioned

[x] Confirm that an AQI sensor is not created for your purifier
[x] Test turning purifier on and off
[x] Test turning purifier light on and off
[x] Test setting purifier preset mode to auto and to night
[x] Test setting purifier fan speed to each of the three levels as well as 0 (this should turn the purifier off)
[x] Test setting the timer to 1 hour, 2 hours, 4 hours, and 8 hours. Also setting the timer to off.

While the control of it worked, I would get this as an error in the HA logs for every command
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 134, in _handle_refresh_interval
await self._async_refresh(log_failures=True, scheduled=True)
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 265, in _async_refresh
update_callback()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 325, in _handle_coordinator_update
self.async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 553, in async_write_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 590, in _async_write_ha_state
state = self._stringify_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 559, in _stringify_state
if (state := self.state) is None:
File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 372, in state
value = self.native_value
File "/config/custom_components/coway/sensor.py", line 225, in native_value
return round(float(self.coordinator.data[self._device].quality["air_quality_index"]), 1)
ValueError: could not convert string to float: ''

I had my smart mode sensitivity to Normal, are you thinking that affects the info we get back?

Just to confirm, you deleted the integration from the Home Assistant integrations page (not from your custom_components folder) prior to updating to <IP_ADDRESS> and restarting Home Assistant?
Regarding the smart mode sensitivity, can you set it to sensitive and then sniff the traffic to see if AQI is reported?

@romedtino
Make these changes to the sensor.py file starting at line 26, restart Home Assistant, and let me know what is logged:
    for idx, ent in enumerate(coordinator.data):
        _LOGGER.error(f'{coordinator.data[idx].name} is new model: {coordinator.data[idx].new_model}')
        if not coordinator.data[idx].new_model:
            _LOGGER.error(f'AQI sensor added for {coordinator.data[idx].name}')
            sensors.append(AirQualityIndex(coordinator, idx))
        sensors.append(PreFilter(coordinator, idx))
        sensors.append(MAX2Filter(coordinator, idx))
        sensors.append(TimerRemaining(coordinator, idx))
        sensors.append(ParticulateMatter25(coordinator, idx))
        sensors.append(ParticulateMatter10(coordinator, idx))
        sensors.append(CarbonDioxide(coordinator, idx))
        sensors.append(VolatileOrganicCompounds(coordinator, idx))

Yeah, I see the new changes in my copies of custom_components/coway/* and iocareapi.py
Pulling the newer copy of iocareapi.py, trying out the new mcu_version it seems that the body key/value 'barcode': device.device_id is now 'deviceId': device_id. I think that might be why it's thinking the AQI sensor needs to be added because otherwise, the curMcuVer returns None

What all does the request body include for the 0615 endpoint based on your sniffing?
Are you saying that the current iocareapi backend returns None for curMcuVer, but a value is returned when using 'deviceId': device_id?

Sure the body for 0615

```json
{
    "header":{
       "result":false,
       "error_code":"",
       "error_text":"",
       "info_text":"",
       "message_version":"",
       "login_session_id":"",
       "trcode":"CWIG0615",
       "accessToken": "REMOVED",
       "refreshToken": "REMOVED",
    },
    "body":{
       "deviceId":"15102DNO2191000274",
       "timezone":"America/Los_Angeles"
    }
 }
```

and yes precisely. I guess they changed it for the newer models
OLD
'barcode': device.device_id,
NEW
'deviceId': device.device_id,
otherwise, curMcuVer returns None

@theglus @romedtino
Out of curiosity, what do you all have your sensitivity set to (located on the product settings page)?
I'm curious to know if perhaps settings aside from sensitive for newer models don't report AQI values to Coway's servers.

I have mine set to sensitive.

Alright, I can confirm that curMcuVer returns None for me when using device_id.  I also tested to see if the quality endpoint works when using deviceId, but not luck.
I'll go ahead and push the deviceId change to pypi and then create a new release

@theglus @romedtino
Just released <IP_ADDRESS>. Let me know if that prevents the AQI sensor from being added.

Can confirm no AQI sensor and no more errors for all the commands 🎉 thanks for the quick turnaround!

Testing report for 0.0.33 pre-release

[x] Deleted integration from the HA Integrations page.
[x] Rebooted Home Assistant.
[x] Redownloaded Coway integration, installing beta version <IP_ADDRESS>.
[x] Rebooted Home Assistant.
[x] Confirmed that an AQI sensor is not created for your purifier
[x] Tested turning purifier on and off
[x] Tested turning purifier light on and off
[x] Tested setting purifier preset mode to auto and to night
[x] Tested setting purifier fan speed to each of the three levels as well as 0 (this should turn the purifier off)
[x] Tested setting the timer to 1 hour, 2 hours, 4 hours, and 8 hours. Also setting the timer to off.

Perfect!
Thanks to both of you for helping out!
@romedtino If you don't mind, can you check the quality endpoint from time to time and report back here if an AQI ever gets populated?

No problem, can do 🥳


updated variable-hoisting.md

This will close #2021

Please review the PR, and provide necessary feedback.


create and test fromHexString for SszBitList

As a developer using the SszBitList class,
it would be convenient to have fromHexString and fromBytes defined,
So that I can create an SszBitList from a hex or bytes payload.
Basically, fromBytes would do the heavy lifting, fromHexString would convert to bytes then call fromBytes.
I'd suggest following how ofBits is done basically.
The implementation will likely be in SszBitListImpl where ofBits is defined currently. and would be on the SszBitListSchema
Testing will be in SszBitListTest.

hey @rolfyone! you can assign this to me

Solved by PR #8611

closing - fixed.


UnboundLocalError: local variable 'tuple_3'

Hello,
I've just installed the software. When running:
conda activate LongQC_env
export PATH=/home/m.sevi/software/LongQC/minimap2_mod/:$PATH
python /home/m.sevi/software/LongQC/longQC.py sampleqc -p 4 -x ont-rapid -o fne_qc_out_dir /scratch/m.sevi/processing/WW_HRSD/data/long/basecalled_fq/fne/fne_fastq/fne.fastq

The run fails with standard output:

longQC:2020-06-18 15:11:19,615:166:INFO:Cmd: /home/m.sevi/software/LongQC/longQC.py sampleqc -p 4 -x ont-rapid -o fne_qc_out_dir /scratch/m.sevi/processing/WW_HRSD/data/long/basecalled_fq/fne/fne_fastq/fne.fastq
longQC:2020-06-18 15:11:19,615:216:INFO:Preset "ont-rapid" was applied. Options --pb(--ont) is overwritten.
longQC:2020-06-18 15:11:21,621:288:INFO:Computation of the low complexity region started for a chunk 0
lq_mask:2020-06-18 15:11:22,791:111:INFO:New job was submitted: in->fne_qc_out_dir/analysis/tmp_0.fastq, out->fne_qc_out_dir/analysis/tmp_0.txt
longQC:2020-06-18 15:11:22,792:293:INFO:Adapter search is starting for a chunk 0.
longQC:2020-06-18 15:11:22,792:309:INFO:Computation of the GC fraction started for a chunk 0
lq_utils:2020-06-18 15:11:25,979:380:INFO:list for subsample is not initialized. Initializing now.
lq_adapt:2020-06-18 15:11:30,810:76:INFO:7038 reads were skipped due to their short lengths.
lq_adapt:2020-06-18 15:11:30,811:96:INFO:Adapter Sequence: GTTTTCGCATTTATCGTGAAACGCTTTCGCGTTTTTCGTGCGCCGCTTCA, max identity:-1.000000 and the number of trimmed reads: 0
longQC:2020-06-18 15:11:45,124:320:INFO:Adapter search has done for a chunk 0.
longQC:2020-06-18 15:11:45,125:324:INFO:subsample finished for chunk 0.
Traceback (most recent call last):
File "/home/m.sevi/software/LongQC/longQC.py", line 920, in
main(args)
File "/home/m.sevi/software/LongQC/longQC.py", line 63, in main
args.handler(args)
File "/home/m.sevi/software/LongQC/longQC.py", line 335, in command_sample
if tuple_3:
UnboundLocalError: local variable 'tuple_3' referenced before assignment

Below find information about my environment:
_libgcc_mutex             0.1                        main
blas                      1.0                         mkl
bzip2                     1.0.8                h7b6447c_0
ca-certificates           2020.1.1                      0    anaconda
certifi                   2<IP_ADDRESS>               py37_0    anaconda
cycler                    0.10.0                     py_2    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
edlib                     1.2.3                h2d50403_1    bioconda
expat                     2.2.9                he1b5a44_2    conda-forge
fontconfig                2.13.1            he4413a7_1000    conda-forge
freetype                  2.10.2               he06d7ca_0    conda-forge
glib                      2.63.1               h3eb4bd4_1
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb31296c_0
h5py                      2.10.0           py37h7918eee_0
hdf5                      1.10.4               hb1b8bf9_0
icu                       58.2              hf484d3e_1000    conda-forge
intel-openmp              2019.4                      243
jinja2                    2.11.2                     py_0    anaconda
joblib                    0.15.1                     py_0    anaconda
jpeg                      9d                   h516909a_0    conda-forge
kiwisolver                1.2.0            py37h99015e2_0    conda-forge
krb5                      1.17.1               h173b8e3_0
ld_impl_linux-64          2.33.1               h53a641e_7
libcurl                   7.69.1               h20c2e04_0
libdeflate                1.6                  h516909a_0    conda-forge
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.3                  he6710b0_1
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
libpng                    1.6.37               hed695b0_1    conda-forge
libssh2                   1.9.0                h1ba5d50_1
libstdcxx-ng              9.1.0                hdf63c60_0
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.10               he19cac6_1
markupsafe                1.1.1            py37h7b6447c_0    anaconda
matplotlib                3.2.1                         0    conda-forge
matplotlib-base           3.2.1            py37hef1b27d_0
mkl                       2019.4                      243
mkl-service               2.3.0            py37he904b0f_0
mkl_fft                   1.0.14           py37hd81dba3_0    r
mkl_random                1.0.4            py37hd81dba3_0    r
ncurses                   6.2                  he6710b0_1
numpy                     1.17.0           py37h7e9f1db_0    r
numpy-base                1.17.0           py37hde5b4d6_0    r
openssl                   1.1.1g               h7b6447c_0    anaconda
pandas                    1.0.4            py37h0573a6f_0    anaconda
pcre                      8.44                 he1b5a44_0    conda-forge
pip                       20.1.1                   py37_1
pthread-stubs             0.4               h14c3975_1001    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyqt                      5.9.2            py37hcca6a23_4    conda-forge
pysam                     <IP_ADDRESS>         py37hc501bad_0    bioconda
python                    3.7.7                hcff3b4d_5
python-dateutil           2.8.1                      py_0    anaconda
python-edlib              1.3.8.post1      py37h99015e2_1    bioconda
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2020.1                     py_0    anaconda
qt                        5.9.7                h5867ecd_1
readline                  8.0                  h7b6447c_0
scikit-learn              0.22.1           py37hd81dba3_0    anaconda
scipy                     1.4.1            py37h0b6359f_0    anaconda
setuptools                47.3.0                   py37_0
sip                       4.19.8           py37hf484d3e_0
six                       1.15.0                     py_0
sqlite                    3.31.1               h62c20be_1
tk                        8.6.8                hbc83047_0
tornado                   6.0.4            py37h8f50634_1    conda-forge
wheel                     0.34.2                   py37_0
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xz                        5.2.5                h7b6447c_0
zlib                      1.2.11               h7b6447c_3
I'd appreciate any feedback.
Thank you,
Maria

Hello @msevi,
Sorry for the issue, but thank you for your interest and raising this point!
There was a bug for adapter search part, and I fixed and pushed a new version.
Could you please try new one, version 1.1.1?
Yoshi

Hello Yoshi,
Thanks for your reply. I cloned the newest version and re ran, but the same error persists.
Cheers,
Maria

Hello @msevi,
Thank you for trying, and I'm sorry for the issue. I forgot to upload the most important file...
Now, it should be working, and could you please try again?
Yoshi

Hi Yoshi,
Thanks for your help, processes finished without error!
Cheers,
Maria

Hi Maria,
I'm glad to hear that it worked, and thank you for your kind update!
Best,
Yoshi


troubleshooting: Local PV hostpath fails due to secure cluster policy

From the slack conversation:
an user reported that provisioning local pv hostpath volumes is throwing the following error:
' failed to provision volum ││ e with StorageClass "local-hostpath": Pod "init-pvc-86c9d288-262b-
4b34-b8d7-9299f868a749" is invalid: spec.containers[0].securityContext.privileged: Forbidden:
disallowed  ││ by cluster policy

Solution/Workaround: User got this working by setting --allow-privileged=true on the kube-apiserver.

The issue was seen on:
microk8s on ubuntu 18.04
k8s version: 1.18.2


pinscan usage

On my motor switch I have two relays and three touch buttons. I found pins of relays, but I can't find pins of buttons.
I think in theory itis possible to find them, but I don't see anything in pinscan, when I push buttons

Right, PinScan doesn't enable internal pull-up/pull-down when doing manual scan.
If you verify that the button has no hardware pull (no resistor to GND or VCC) you can run manual scan, and apply voltage (0V or 3.3V) to the button terminal, to see which pin changes.


Sport UV dataset

Hi,
Thanks for your great work,
Can you provide the link or the sport basketball dataset you used in your paper?
You also mentioned that you focused on the PnR instances of the game. How to find these instances?
Best,

Hi, I though that you process the dataset from the following repo:
https://github.com/linouk23/NBA-Player-Movements
Best

Might be based on similar data, but we did not use that repo

@os1a Did you find the basketball dataset used here?


"raised by" should be clickable

and send to corresponding file and line on GitHub

This is not easy to be exact since FixMyDjango doesn't considers minor Django versions yet. Setting as wontfix for now.


Use TPM key only for vault protection

Starting this commit a new install of EVE-OS will create vault config file on systems with TPM support. That file will be used to determine whether to use only the TPM key or merge the TPM and controller key.  This applies to both ext4 and zfs filesystems.
On systems with TPM enabled

All existing installs as part of upgrade will create vault config with {"TpmKeyOnly":false}
New installs will create create vault config with {"TpmKeyOnly":true}

On non-TPM systems, its a no-op
Testing

New installs on both ext4 and zfs created global.json with content {"TpmKeyOnly":true} and only 32byte TPM key is used to encrypt the vault.
Upgrades from earlier versions did not create the vaultconfig file and hence merged key is used.
Restart/reboot of devices always read vaultconfig file and use keys according to the config set.

Signed-off-by: Pramodh Pallapothu<EMAIL_ADDRESS>

@giggsoff there are two scenarios for downgrade.

start at older version -> upgrade to this version -> downgrade to older version ==> this should work
start at this version -> downgrade to older version ===> this would not work.

I believe #1 is the common scenario. #2 is rare because I do not see customer installing a version and then later downgrading.

I believe #1 is the common scenario. #2 is rare because I do not see customer installing a version and then later downgrading.

Agreed.


fix(setup.py): compile when using Anaconda

Anaconda adds another compilation step against the statically
compiled library libdeflate.a and dies because libdeflate.a
was not compiled with position independent code. This adds
an environment variable tweak that ensures -fPIC is added to
the compilation of libdeflate.a.
Resolves #4

I added this flag as part of some other changes (and wheels will be published shortly as well). Thank you!


Build an SAP Fiori App with SAP API Management

Mission URL: https://developers.sap.com/mission.apim-fiori-app.html
I've concluded all the mission but complete percent still in 88%.
I've clean the browser cache, restarted the S.O. and still 88%.
Could you verify, please?

Can you let me know which tutorial is not complete? Screenshot would help of the mission page, and then the tutorial that does not complete.

Hi thecodester,
Sorry for being late. I tried a few days ago and got it done.
Thank you.


think about task id

At now task id is serial in db and string in api. It ways to convert number to string and vice versa.
How about to use string id in db?

The problem is that taskID passed as a string to work with DB. If those converts become a business logic part (after some refactoring) the problem will disappear.


add api_version parameter to azure LLM service config

Resolves https://github.com/dimagi/open-chat-studio/issues/117

Awesome. Were you able to test it?

no, but it should work - not sure why I didn't include this in the original version. It's mentioned in the docs.