Remove references to materialized views in dbt-bigquery

Contributions

[X] I have read the contribution docs, and understand what's expected of me.

Link to the page on docs.getdbt.com requiring updates
https://docs.getdbt.com/reference/resource-configs/bigquery-configs#materialized-view
What part(s) of the page would you like to see updated?
dbt-bigquery will not support materialized views in v1.6. We will release this functionality early in the v1.7 lifecycle, likely in v1.7.0b1. We need to remove references to materialized view support for dbt-bigquery in the docs.
Additional information
No response

resolved by: #3779


Build failure: netsurf.browser

Steps To Reproduce
Steps to reproduce the behavior:

build netsurf.browser

Build log
netsurf-log.txt
Additional context
Likely due to #308976
Notify maintainers

@vrthra @AndersonTorres
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

Add a :+1: reaction to issues you find important.

oops nevermind, #308976 did not land on master yet. in fact it probably fixes this issue.


add missing cors config

Bugfix: Add missing CORS config
The graph, userlog and ocdav services had no CORS config options.
Reva Part: https://github.com/cs3org/reva/pull/3764

@LukasHirt I think that unblocks the filepicker integration.

@LukasHirt I think that unblocks the filepicker integration.

Awesome, thanks! I'll give it a shot once it's ready to confirm it


Accessibility:  removing outline: none

Removing the outline: none param allows for users to see where they
are when navigating the dashboard via the keyboard.
https://www.w3.org/WAI/WCAG21/quickref/#focus-visible

@xirclebox Thanks for submitting this. A couple of follow-up comments and questions:

It might be nice to style the color of the outline to something from the palette. e.g., outline-color: $purple-light; What do you think? Are there any a11y best practices for outline styles?
I noticed that the QuillJS WYSIWYG editor behaves oddly with outline removed, when there's only a few lines of text. (see below).  Do you have any experience with this editor, or suggestion on what to do when a textarea is embedded within a WYSIWYG?
Are there any other accessibility improvements you can think of? I'd really appreciate your help making Vapid more accessible.


16khz for inference vs 22khz?

I notice 16khz seems to be hard programmed into the code for the downsampling phase. I think supporting more modular sample rates for inference audio would make sense. With librosa the standard for handling audio is more like 22khz instead of 16khz. Migrating to this standard may reduce inference time but there are potential gains, especially with pronouncing words quickly. 16khz, to me, seems slightly too low to support the entire range of human speech.
https://analyticsindiamag.com/hands-on-guide-to-librosa-for-handling-audio-files/#:~:text=The sampling rate is nothing,by your desired sampling rate

It's because the pretrained feature encoder we used only support 16khz.
In Automatic Speech Recognition (ASR) task, there is no pretrained feature encoder use audio exceeding 16khz as input. Maybe increasing input audio sample rate can't increase the accurary of ASR.

Hi RVC-Boss,
If the pretrained feature encoder only supports 16kHz, why didn't you use the vocoder (HiFi GAN) to generate 16kHz audio?
On the main branch, I only found vocoder models that support audio generation in the 40kHz and 48kHz formats.
In my understanding, since encoder only supports 16kHz, this method will block or filter audio component above 8kHz.
And the vocder used the encoder feaure which olny contained 8kHz information.
My question is why you used the vocoder (HiFi GAN) to generate audio at 40kHz/48kHz instead of using 16kHz.
BTW, I notice that outcome audio from  40kHz/48kHz vocoders only contaion audio information below 8kHz.


Reference SecurityGroups from SecurityGroups without hardcoding IDs

What problem are you facing?
When creating SecurityGroups with Crossplane, we are not able to reference other SecurityGroups without hardcoding the SecurityGroupIDs.
For every application, we want a SecurityGroup for only that app. This should be referenced when creating a new AWS service, and service SecurityGroup, for example a new CacheCluster.
Reason Zero Trust Networking.
How could Crossplane help solve your problem?
We would like to be able to reference other SecurityGroups in userIdGroupPairs in both Ingress and Egress.
An example where we create a app SG and a CacheCluster SG and want to link them together. This is not necessary in the same composition.
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
  name: applicationname
  labels:
    app: appname
spec:
  forProvider:
    description: application security group attached to pod
    groupName: applicationname
    egress:
    - ipProtocol: tcp
      securityGroupObjectRef:
      - groupIDSelector:
          matchLabels:
            appservices: applicationname
---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
  name: cacheClusterSecurityGroup
  labels:
    appservices: applicationname
spec:
  forProvider:
    description: CacheCluster security group for inbound access from application
    groupName: appCacheCluster
    ingress:
      - ipProtocol: tcp
        userIdGroupPairs:
          - groupIDSelector:
              matchLabels:
                app: appname
               #matchControllerRef: [true|false]
            # or a new object ref would be nice (since we only want one without labels in this specific use-case)
            securityGroupObjectRef:
              namespace: app
              name: applicationname
              kind: SecurityGroup

@larhauga It should be possible to add groupIdRef and groupIdSelector that will resolve to SecurityGroup. But there is also userId field which expects ID of an AWS account. Does this ID is the one that the security group whose name is given via groupId is created? In other words, can we assume that user will always enter the account ID of the security group that they specified in groupId?

@larhauga It should be possible to add groupIdRef and groupIdSelector that will resolve to SecurityGroup. But there is also userId field which expects ID of an AWS account. Does this ID is the one that the security group whose name is given via groupId is created? In other words, can we assume that user will always enter the account ID of the security group that they specified in groupId?

@muvaf that would be very nice indeed!
I think it would make sense that the UserID should be that of the given via groupID if that is possible. I think the userID should be that of the provider if not specified.

Related issue: https://github.com/crossplane/crossplane-runtime/issues/250

@chlunde let me know when this is ready for another review after manual test.

I will have to find out what happens here, it always produces a diff:
  Conditions:
    Last Transition Time:  2021-04-08T12:20:03Z
    Reason:                Creating
    Status:                False
    Type:                  Ready
    Last Transition Time:  2021-04-08T12:20:04Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced

It would be nice if --debug printed cmp.Diff in the log
Originally posted by @chlunde in https://github.com/crossplane/provider-aws/issues/614#issuecomment-815765792
think i also see this behavior when using self-reference via ingress - @chlunde would be cool to see your suggestion for fix the diff issue merged ;)
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
  name: self-referential-glue
spec:
  forProvider:
    region: eu-central-1
    vpcIdRef:
      name: sample-vpc
    groupName: self-referential-glue
    description: self-referential glue security group
    ingress:
      - fromPort: 0
        toPort: 65535
        ipProtocol: tcp
        userIdGroupPairs:
          - groupName: self-referential-glue
  providerConfigRef:
    name: default

  Conditions:
    Last Transition Time:  2021-10-30T19:44:33Z
    Reason:                Creating
    Status:                False
    Type:                  Ready
    Last Transition Time:  2021-10-30T19:44:34Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
Events:
  Type    Reason                   Age                  From                                         Message
  ----    ------                   ----                 ----                                         -------
  Normal  CreatedExternalResource  4m22s                managed/securitygroup.ec2.aws.crossplane.io  Successfully requested creation of external resource
  Normal  UpdatedExternalResource  20s (x8 over 4m22s)  managed/securitygroup.ec2.aws.crossplane.io  Successfully requested update of external resource

I think this was fixed by https://github.com/crossplane/provider-aws/pull/631.


RFC0334: added clarifications to unresolved questions

including clarifications in the JWE format around key types and base64URL encoding and minor typo fixes
Signed-off-by: Baha Shaaban<EMAIL_ADDRESS>

@swcurran I meant to amend my previous pr with these updates, but no worries, a new PR makes changes to this RFC clearer, especially for answers around unresolved questions (which I believe is what we're aiming for in aries-framework-go and hoping the community agrees).
I know there's a been a long discussion about KIDs in the past, but for the sake of moving forward with JWE and finalize implementation of this RFC, each Aries implementation can have their own definition of kid format (it could be a did key or any string value, we chose to use the JWK thumbprint in Aries-framework-go to be consistent with the public key the kid represents).
tagging @kdenhartog @dhh1128 @llorllale @troyronda @andrewwhitehead  as well, I sincerely hope this pr doesn't translate into another argument debacle around kids


Python 3.11 support

Request description
IREE's third_party/pybind11 is a bit old (2.9.0.dev1) to support Python 3.11.
Python 3.11 support is added in pybind11 v2.10.1.
Bumping up pybind11 version should fix the issue.
What component(s) does this issue relate to?
Python
Additional context
Here's the error message during build:
[2217/4422] Building CXX object runtime/bindings/python/CMakeFiles/iree_runtime_bindings_python_PyExtRt.dir/vm.cc.o
FAILED: runtime/bindings/python/CMakeFiles/iree_runtime_bindings_python_PyExtRt.dir/vm.cc.o
/usr/bin/clang++ -DCPUINFO_SUPPORTED_PLATFORM=1 -DIREE_HAVE_HAL_EXECUTABLE_LOADER_EMBEDDED_ELF=1 -DIREE_HAVE_HAL_EXECUTABLE_LOADER_SYSTEM_LIBRARY=1 -DIREE_HAVE_HAL_EXECUTABLE_LOADER_VMVX_MODULE=1 -DIREE_HAVE_HAL_LOCAL_SYNC_DRIVER_MODULE=1 -DIREE_HAVE_HAL_LOCAL_TASK_DRIVER_MODULE=1 -DIREE_HAVE_HAL_VULKAN_DRIVER_MODULE=1 -DIREE_HAVE_VMVX_MODULE -DIREE_TRACING_HOOK_CPP_NEW_DELETE=0 -Diree_runtime_bindings_python_PyExtRt_EXPORTS -I/home/ryuta/packages/iree/iree/third_party/cpuinfo/include -I/home/ryuta/packages/iree/iree/third_party/vulkan_headers/include -isystem /usr/include/python3.11 -isystem /home/ryuta/packages/iree/iree/third_party/pybind11/include -isystem /home/ryuta/packages/iree/iree -isystem /home/ryuta/packages/iree/build -isystem /home/ryuta/packages/iree/iree/runtime/src -isystem /home/ryuta/packages/iree/build/runtime/src -isystem /home/ryuta/packages/iree/build/runtime/src/iree/base/internal/flatcc -isystem /home/ryuta/packages/iree/iree/third_party/flatcc/include -isystem /home/ryuta/packages/iree/build/runtime/src/iree/schemas -isystem /home/ryuta/packages/iree/iree/third_party/vulkan_memory_allocator/include -stdlib=libc++ -O2 -g -DNDEBUG -fPIC -fvisibility=hidden -fvisibility=hidden -fno-rtti -fno-exceptions -Werror -Wall -Wno-ambiguous-member-template -Wno-char-subscripts -Wno-extern-c-compat -Wno-gnu-alignof-expression -Wno-gnu-variable-sized-type-not-at-end -Wno-ignored-optimization-argument -Wno-invalid-offsetof -Wno-invalid-source-encoding -Wno-mismatched-tags -Wno-pointer-sign -Wno-reserved-user-defined-literal -Wno-return-type-c-linkage -Wno-self-assign-overloaded -Wno-sign-compare -Wno-signed-unsigned-wchar -Wno-strict-overflow -Wno-trigraphs -Wno-unknown-pragmas -Wno-unknown-warning-option -Wno-unused-command-line-argument -Wno-unused-const-variable -Wno-unused-function -Wno-unused-local-typedef -Wno-unused-private-field -Wno-user-defined-warnings -Wctad-maybe-unsupported -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wfor-loop-analysis -Wformat-security -Wgnu-redeclared-enum -Wimplicit-fallthrough -Winfinite-recursion -Wliteral-conversion -Wnon-virtual-dtor -Woverloaded-virtual -Wpointer-arith -Wself-assign -Wstring-conversion -Wtautological-overlap-compare -Wthread-safety -Wthread-safety-beta -Wunused-comparison -Wvla -fno-lax-vector-conversions -fmacro-prefix-map=/home/ryuta/packages/iree/iree=iree -frtti -fexceptions -I/home/ryuta/packages/iree/iree/third_party/flatcc/include/ -I/home/ryuta/packages/iree/iree/third_party/flatcc/include/flatcc/reflection/ -std=gnu++17 -MD -MT runtime/bindings/python/CMakeFiles/iree_runtime_bindings_python_PyExtRt.dir/vm.cc.o -MF runtime/bindings/python/CMakeFiles/iree_runtime_bindings_python_PyExtRt.dir/vm.cc.o.d -o runtime/bindings/python/CMakeFiles/iree_runtime_bindings_python_PyExtRt.dir/vm.cc.o -c /home/ryuta/packages/iree/iree/runtime/bindings/python/vm.cc
In file included from /home/ryuta/packages/iree/iree/runtime/bindings/python/vm.cc:7:
In file included from /home/ryuta/packages/iree/iree/runtime/bindings/python/./vm.h:12:
In file included from /home/ryuta/packages/iree/iree/runtime/bindings/python/./binding.h:14:
In file included from /home/ryuta/packages/iree/iree/third_party/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/ryuta/packages/iree/iree/third_party/pybind11/include/pybind11/attr.h:13:
In file included from /home/ryuta/packages/iree/iree/third_party/pybind11/include/pybind11/cast.h:16:
/home/ryuta/packages/iree/iree/third_party/pybind11/include/pybind11/detail/type_caster_base.h:482:26: error: member access into incomplete type 'PyFrameObject' (aka '_frame')
            frame = frame->f_back;
                         ^
/usr/include/python3.11/pytypedefs.h:22:16: note: forward declaration of '_frame'
typedef struct _frame PyFrameObject;
               ^
1 error generated.

It usually isn't very disruptive to bump the pybind module. Shouldn't be a problem...

https://github.com/iree-org/iree/commit/beaa511a5807a4aee749c2acd381d791e577990a bumped the pybind submodule to https://github.com/pybind/pybind11/commit/59aa99860c60bd171b9565e9920f125fdb749267. I'm not sure why we stopped at 2.9. @ScottTodd ?

(Progress here is likely to be a bit slow over the holiday though)

I tried to update pybind for this in https://github.com/iree-org/iree/pull/11564, but many things broke... The errors seemed unrelated to Python, so I suspected that something else in our Docker images was unstable.

beaa511 bumped the pybind submodule to pybind/pybind11@59aa998. I'm not sure why we stopped at 2.9. @ScottTodd ?

That was the latest at the time (December 2011, last year)

Ah, I mixed that up with another recent third_party bump you did: https://github.com/iree-org/iree/commit/4293848f0e6d2ed49cdedee458acbdfc14aa5896

Okay, I updated our version of pybind and was able to build the Python bindings on my Windows machine using Python 3.11. I think that fixes this issue as reported, but we should also start building and distributing Python 3.11 wheels for installation through pip.

I confirmed the update. IREE can be built on my Linux (Fedora 37).
Thank you!


Fix minor pytest deprecation

/_pytest/fixtures.py:900: PytestRemovedIn8Warning: Support for nose tests is deprecated and will be removed in a future release.
  project/tests/test_logging.py::TestLogBehaviorClass::test_set_stream_handler_format is using nose-specific method: `setup(self)`
  To remove this warning, rename it to `setup_method(self)`
  See docs: https://docs.pytest.org/en/stable/deprecations.html#support-for-tests-written-for-nose

Codecov Report

Merging #239 (75b72df) into master (70442c4) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #239   +/-   ##
=======================================
  Coverage   89.89%   89.89%
=======================================
  Files          10       10
  Lines        1187     1187
=======================================
  Hits         1067     1067
  Misses        120      120

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


can't use IndexOptions in BsonIndex attribute

Hi! I am trying to set BsonIndex attribute with parameters, like in example but it seems that it's impossible. Compiler says "An attribute argument must be a constant expression, typeof expression or array creation expression of an attribute parameter type".
Am I missing something obvious or this is wrong attribute implementation?
public class Customer
{
   public int Id { get; set; }

   [BsonIndex]
   public string Name { get; set; }

   // written below doesn't compiles
   [BsonIndex(new IndexOptions { Unique = true, IgnoreCase = false, RemoveAccents = false })]
   public string Email { get; set; }
}

Hi @VoronFX,thanks for find this bug. I fixed in master branch. Please, use:
   [BsonIndex(unique: true, ignoreCase: false, removeAccents: false)]
   public string Email { get; set; }

I will update nuget later.


SMILES to name mapping, missing info?

I want to obtain the chemical name for each metabolite in the EnzymeMap dataset. Specifically I'm interested in the balanced reactions. However, with the files provided it seems that there are about 20 % of smiles for which there is no name mapping. The parsing is based on the assumption that the "non-bond" symbol "." is used to separate substrates and products.
The relevant files here are processed_reactions.csv and compound_to_smiles.json in the data folder. The code below prints out the following messages:

There are 43903 unique smiles in the reaction dataset.
There are 52498 unique smiles in the name to smile mapping file.
There are 8009 smiles in the enzymemap dataset that are not in the name to smile mapping file.

Am I doing something wrong here? Is there a good reason for the missing smile to name mappings? Or is it perhaps a bug?
from os.path import join
import pandas as pd
import json

### get all the smiles ###

# load the data
df = pd.read_csv(join('data', 'processed_reactions.csv'),
                 sep=',')

# now iterate through each row
enzymemap_smiles = set()
for reaction in df.unmapped.values:

    # get the substrates and products
    substrates, products = reaction.split('>>')
    for smiles in [*substrates.split('.'), *products.split('.')]:
        enzymemap_smiles.add(smiles)

print(f'There are {len(enzymemap_smiles)} unique smiles in the reaction dataset.')

### get the names to smiles mapping ###

# obtain smile to name mapping
with open(join('data', 'compound_to_smiles.json'), 'r') as f:
    name_to_smiles = json.loads(f.read())

# invert to get smile to name mapping
smile_to_name = {}
for name, smiles in name_to_smiles.items():
    if smiles == []:
        continue

    for s in smiles:
        if smile_to_name.get(s) is None:
            smile_to_name[s] = []

        smile_to_name[s].append(name)

print(f'There are {len(smile_to_name)} unique smiles in the name to smile mapping file.')

### ensure that all the smiles in the enzymemap dataset have a name ###

# cross-reference with the enzymemap dataset
missing = enzymemap_smiles - set(smile_to_name.keys())

print(f'There are {len(missing)} smiles in the enzymemap dataset that are not in the name to smile mapping file.')

Hi,
Yes, it is possible that there are SMILES in EnzymeMap that are not in the compound_to_smiles.json, namely in the following cases:

the reaction was corrected for stereochemistry
the reaction outcome was suggested due to an imbalanced or non-mapped reaction

So your code and observation is correct, but a feature, not a bug :)
Also (just a note, I think you had consistently used version 1 above), I just updated GitHub to version 2 of EnzymeMap and will update Zenodo later today (so make sure to not accidentally mix up files today)

Thanks for the quick response! :) That makes a lot of sense. So essentially these are potential errors in the BRENDA database, which makes them even more interesting.
How should I go about getting a name / identifier for these that would link them to an actual chemical? Any fixed identifier would do. pubchempy resolves some of them, but far from all.

Running the mapping now, will attach to a comment later if you're interested.

Mmh, I would have said Pubchempy, too. You can try different options, e.g. send the SMILES or convert to INCHI first (sometimes that helps). You could also try the RCI resolver https://cactus.nci.nih.gov/chemical/structure (there's an API), it's a bit slow, but you only have to run it once I guess

For name to SMILES, opsin is also a good option, but I just checked, it doesn't work the other way around.
Yes, happy to see the new outcome

In the new file there are ~8900 SMILES without names. About 2000 of those could be mapped with pubchempy.
Now trying the strategy of converting the SMILES to INCHI using Rdkit and then mapping with pubchempy. Only have limited experience with Rdkit, but it seems to not like the SMILES very much (see below).  Then I think that is not too bad. Sometimes stereo information is deleted in the products when it is not defined in the reactants for atoms far away from the reactive center, and I simply remove the stereochemistry flag. So I guess the "Omitted undefined stereo" comes from there. I have never seen the "Charges were rearranged", have to dig into this to see what it means.
Also, did it take so long to resolve via PubchemPy (or were you just writing only now) - if this takes so long, I can send you a script that will resolve ~10000k molecules via PubchemPy in a few minutes

No, that's just a very small subset. Lots and lots of warnings. I'd say for most of them. If the stereochemistry flag is removed that could explain why the SMILES don't match.
Yes, it took that long. About 1.5 s per compound. Would love to get the script! :)

Mmh maybe we would have needed another round of standardization after removing the flag, I have to look into this.
I'll post the script in a second

mmh okay adapting the script takes a bit longer than I though (I only have name to smiles, thought it would be trivial to reverse). I can continue tomorrow - I guess you are using pubchempy at the moment (which is very slow) - I would recommend switching to PUG (although this involves some unpleasant XML), you can look into query_pubchem() in helpers_resolve_smiles.py if you want to get started already. The query needs to be changed + the desired output according to https://pubchem.ncbi.nlm.nih.gov/docs/power-user-gateway#section=FAQs

The Rdkit SMILES to INCHI transformation followed by PubChemPy only gave me two more molecules resolved. So sadly not a solution. :(
Will have a look at PUG (how can you go wrong with something called "Power User Gateway"?), but based on what I saw with Rdkit my expectations are low.
What are the relative merits of:

Taking the name from the BRENDA raw text field (I know they are not in the same order as the smiles, but I'm sure this can be resolved somehow).
Not editing the SMILES by removing the stereochemistry flag.
Mapping the SMILES to names before stereochemistry editing and using that name also for the edited SMILE vector?

Okay then don’t spend time on Pubchem PUG, it will give you the same results only faster. I think I need to fix the underlying issue instead.

would be possible bit doesn’t work for suggested reactions

simply not removing/editing them will create lots of lost reactions, since BRENDA has a lot of stereochemistry problems due to the ambiguity in resolving names to SMILES. However, there might be a better way to do this, I will need to check why the SMILES are weird, and can possibly fix it.

Possible, but then we might have different SMILES for the same name (which we do have anyways, so not a big problem I guess). Will also not work for suggested reactions.

So I need to look into whether the main contributor to the non-standardized SMILES is the suggestion or stereochemistry correction (or something else). If the suggestion is not a problem, we can simply do 1 or 3. In any case, we shouldn’t get SMILES that are not resolvable via pubchempy. I also need to check whether this affects small, large and „exotic“ compounds alike.

Sounds like a plan! Agree that all smiles should be resolvable. Still hoping that there is some error on my end and that more of the 8.9 K are actually resolvable. Attached is a json of the 2.3 K that I could get.
smiles_to_missing.zip

Thanks! Since I encountered the same problem once: your pubchempy requests do really go through for all compounds and just return an empty list, right? Because depending on how you set up your scripts, pubchem might block your IP address for a few minutes if you send a lot of requests and if you had just done eg a try/except you will never see the http error you get and it seems like compounds don’t resolve.

Okay so here's the PUG solution. This resolved the 2000 compounds in 15 seconds, yay (I also made it work for inchi, and different output types (e.g. 'iupac' for the name, 'synonyms' for all names, etc):
import os
from lxml import etree as ET
import requests
import shutil
import urllib.request as request
from contextlib import closing
import time
from collections import defaultdict
from rdkit import Chem

def download_ftp(ftp_link: str, outfile: str):
    """download_ftp.

    Args:
        ftp_link (str): ftp_link
        outfile (str): outfile
    """

    ftp_link=ftp_link.replace('ftp','https')
    with closing(request.urlopen(ftp_link)) as r:
        with open(outfile, 'wb') as f:
            shutil.copyfileobj(r, f)

def query_pubchem_name(ids: list,
                   query_type: str = "smiles",
                   output_type: str = "iupac",
                   save_file: str = "pubchem_save.txt",
                   rm_save: bool = True,
                   encoding : str ="utf8",
                   return_single : bool = False) -> dict:
    """query_pubchem.

    Args:
        ids (list):
        query_type (str): inchi, chebi, or synonym
        save_file (str):
        rm_save (bool): If true, delete the file saved afterward
        encoding (str): Encoding to send request, defaults to utf-8
        return_single (bool): If true, only return the top hit for smiles

    Return:
        dict mapping ids to smiles lists
    """

    WAIT_TIME = 10
    DOCHEADER = "<!DOCTYPE PCT-Data PUBLIC \"-//NCBI//NCBI PCTools/EN\" \"NCBI_PCTools.dtd\">"
    URL = "https://pubchem.ncbi.nlm.nih.gov/pug/pug.cgi"
    REQUIRED_HEADINGS = [
        "PCT-Data_input", "PCT-InputData", "PCT-InputData_query", "PCT-Query",
        "PCT-Query_type", "PCT-QueryType", "PCT-QueryType_id-exchange",
        "PCT-QueryIDExchange"
    ]

    QUERY_SUBTREE_NAMES = ["PCT-QueryIDExchange_input", "PCT-QueryUids"]
    OUTPUT_HEADERS = [
        "PCT-QueryIDExchange_operation-type", "PCT-QueryIDExchange_output-type",
        "PCT-QueryIDExchange_output-method", "PCT-QueryIDExchange_compression"
    ]
    OUTPUT_VALUES = ["same", output_type, "file-pair", "none"]

    # Start building the query tree
    root = ET.Element("PCT-Data")
    cur_pos = root

    for new_heading in REQUIRED_HEADINGS:
        cur_pos = ET.SubElement(cur_pos, new_heading)

    # Navigate down to where we add the inchis
    query_subtree = cur_pos
    for query_subtree_name in QUERY_SUBTREE_NAMES:
        query_subtree = ET.SubElement(query_subtree, query_subtree_name)

    if query_type == "smiles":
        query_root, query_name = "PCT-QueryUids_smiles", "PCT-QueryUids_smiles_E"
        query_subtree = ET.SubElement(query_subtree, query_root)
        for id_ in ids:
            new_id = ET.SubElement(query_subtree, query_name)
            # give this the id text
            try:
                new_id.text = id_
            except ValueError:
                pass
    elif query_type == "inchi":
        query_root, query_name = "PCT-QueryUids_inchi", "PCT-QueryUids_inchi_E"
        query_subtree = ET.SubElement(query_subtree, query_root)
        for id_ in ids:
            new_id = ET.SubElement(query_subtree, query_name)
            # give this the id text
            try:
                new_id.text = id_
            except ValueError:
                pass

    # Go back up to to current position holder
    # Add the output specification
    for output_header, output_value in zip(OUTPUT_HEADERS, OUTPUT_VALUES):
        output_xml = ET.SubElement(cur_pos, output_header)
        output_xml.set("value", output_value)

    out_xml = ET.tostring(root,
                          encoding=encoding,
                          method="xml",
                          xml_declaration=True,
                          doctype=DOCHEADER).decode()

    # Post the request!
    resp = requests.post(URL, data=out_xml.encode('utf-8'))

    # Handle response and build a request to check on status
    resp_tree = ET.fromstring(resp.text)
    waiting_id = resp_tree.xpath("//PCT-Waiting_reqid")
    waiting_id = waiting_id[0].text if waiting_id else None

    STATUS_CHECK_HEADERS = [
        "PCT-Data_input", "PCT-InputData", "PCT-InputData_request",
        "PCT-Request"
    ]

    root = ET.Element("PCT-Data")
    cur_pos = root
    for header in STATUS_CHECK_HEADERS:
        cur_pos = ET.SubElement(cur_pos, header)
    req_id = ET.SubElement(cur_pos, "PCT-Request_reqid")
    req_id.text = waiting_id
    req_type = ET.SubElement(cur_pos, "PCT-Request_type")
    req_type.set("value", "status")
    query_xml = ET.tostring(root,
                            encoding=encoding,
                            method="xml",
                            xml_declaration=True,
                            doctype=DOCHEADER).decode()

    download_link = None
    waiting_time = 0

    # TODO: Add stop timeout condition?
    # Repeatedly query to see if the results are done, then sleep for WAITIME
    # in case they aren't
    while not download_link:
        resp = requests.post(URL, data=query_xml.encode('utf-8'))
        resp_tree = ET.fromstring(resp.text)
        download_link = resp_tree.xpath("//PCT-Download-URL_url")
        download_link = download_link[0].text if download_link else None
        time.sleep(WAIT_TIME)
        waiting_time += WAIT_TIME

    # At conclusion, download the ftp file
    download_ftp(download_link, save_file)

    # Also parse this
    ret_dict = defaultdict(lambda: [])
    with open(save_file, "r") as fp:
        for linenum, line in enumerate(fp):
            line = line.strip()
            split_line = line.split("\t")
            if len(split_line) == 2:
                mol, smiles = split_line
                mol = mol.strip()
                smiles = smiles.strip()
                ret_dict[mol].append(smiles)

    # If we should only return a single item and not a list
    if return_single:
        ret_dict = {k : v[0] for k,v in ret_dict.items() if len(v) > 0}

    # Remove temp file
    if os.path.exists(save_file) and rm_save:
        os.remove(save_file)

    return ret_dict

# Input SMILES, output iupac
d = query_pubchem_name(['OCC','C(O)C','CCO','C(CCN)C[C@@H](C(=O)O)N'],query_type='smiles',output_type='iupac')
for k in d.keys():
    print(k, d[k])

# Input SMILES, output synonyms
#d = query_pubchem_name(['OCC','C(O)C','CCO','C(CCN)C[C@@H](C(=O)O)N'],query_type='smiles',output_type='synonyms')
#for k in d.keys():
#    print(k, d[k])

# Input Inchi, output iupac
#l = [Chem.MolToInchiKey(Chem.MolFromSmiles(x)) for x in ['OCC','C(O)C','CCO','C(CCN)C[C@@H](C(=O)O)N']]
#d = query_pubchem_name(l,query_type='inchi',output_type='iupac')
#for k in d.keys():
#    print(k, d[k])

import json
with open("smiles_to_missing.json") as f:
    ll=json.load(f)
print(list(ll.keys())[:10])
d = query_pubchem_name(list(ll.keys()),query_type='smiles',output_type='iupac')
for k in d.keys():
    print(k, d[k])

Can you try to query again all 8.9k compounds and report back how many are truly unresolvable? I will then take a look where they origin from (suggested, stereochemistry correction, etc)

Okay I also cannot get more resolved, seems like your queries actually went through (maybe pubchempy improved on this in the last two years!)

So, I have now checked which reactions are affected. From a list of unmapped reactions (but processed) not containing duplicates, there are about 10k reactions with compounds not in the json file:

4985 are suggested reactions, and thus have products which are likely not in the json file. This is expected and correct
1411 are single step from multi step reactions, which can have reactants or products not in the json file for all intermediate steps. This is also expected and correct
3906 are reactions obtained directly from the entry (or the reversed entry). Out of those:
-- 3668 have corrected stereochemistry, and manually checking a few dozen showed that they are ok. So, the stereochemistry checking did not introduce a weird error (like I feared for a hot second), and the SMILES are valid. By the way, I googled a bit about RDKits Inchi functionality and found a lot of people reporting these warnings. Looking manually at SMILES that produced warnings, it seemed like the warnings are caused by e.g. double bonds in rings that don't have a specified stereochemistry, but of course can only be cis since they are in a ring.
-- 238 have undergone an unexpected change, and might be the cause of a bug, that I will now hunt down. I guess 200 for a dataset of that size is not a big problem, but for the sake of completeness, I will fix it nevertheless.

So, for nearly all of the missing SMILES strings, there is a good reason why they are missing. That doesn't solve your initial problem though, which is to resolve them. Apart from trying a few different resolvers, I don't have a good answer here. Maybe you also don't want to use "suggested" and "single from multi" entries, since they are putative anyways. Then you would only be left with the 3.9k changed stereochemistry ones, which we could possibly link back to the name with unchanged stereochemistry.

Thanks! Looks like I messed up when fixing the isomerase bug (thats the -1 ids), thanks for reporting, will try to get this fixed asap. I fixed the issue for the 200 wrong ones today, and will try to investigate what you were reporting

Regarding reactions with rule_id=-1: In general, that rule_id is assigned if the achiral reactants equal the achiral products. In general, this is necessary to correctly map cis/trans and D/L isomerase, because there is no according rule in the Broadbelt reaction set (which does not account for chirality). So, rule_id=-1 doesn't mean that the reaction is wrong per-se. Now, regarding redox-reactions, I have filtered all EC=1...* with rule_id=-1 and find 17 unique reactions:

8 are the same on reactant and product side because they are the same on both sides in BRENDA. In some of those cases, e.g. "NAD+ + NADH = NADH + NAD+", there obviously is a reaction which we are not detecting. However, these reactions might not be very interesting for a synthesis perspective anyways.
5 correspond to cis/trans isomerase reactions, and correspond to the respective BRENDA entry (although that sounds like a misclassification to me)
4 are the same on reactant/product, although the BRENDA entry is not the same, because sulfur and H2S, as well as sulfide all gets resolved to "H2S" when changing trivial names to SMILES, and another one resolves ferricytochrome c and ferrocytochrome c to the same SMILES.
So for redox-reactions (EC=1...*) I didn't find anything unusual (although I agree they could have been filtered out). Did you mean other reactions? Since you mentioned explicit Hs, I somehow think that you are referring to other reactions? It should not matter whether the hydrogens are explicit or implicit, since a reduction/oxidation changes the number of hydrogens, and that is reflected in both version of SMILES (explicit/implicit Hs). We just do not track with hydrogen is which.

For isomerase reactions (filtered all EC=5*.. with rule_id=-1) there are 1271 unique reactions. For 1163 of them, the SMILES is not the same on reactant and product side, but differs in stereochemistry, e.g. cis/trans or D/L, so that is correct from my point of view (maybe you did not mean those, or maybe you looked at achiral educts/products?). Many of those were wrong in version 1, because we didn't have the rule_id=-1 and then we got rules that changed some atom numbers which did not reflect reality. For the 108 remaining ones, the reactant and product side is the same. For some of them, the BRENDA entry also has the same educts/products. However, most of them actually do undergo some isomerisation in the BRENDA entry, but the names get resolved to the same SMILES. I am not sure how to deal with this in an automated way (apart from removing reactions were the educts equal the products altogether).  Since these are not many reactions, it would also be doable to manually correct the SMILES in the next version.
So, I am glad that there is no problem in the mapping itself (so no bug introduced when I corrected the isomerase mappings), but the issues arise from resolving names to SMILES, or are already in BRENDA. Nevertheless, these should be corrected in the next version.
I will look at the other problems later today.

For the second question: The reaction could not be mapped via rule application. It therefore was assigned suggested products (the source='suggested' flag is set), meaning that the products do not reflect the original reaction text. Suggested products get assigned based on rules that are already in the same EC class. Here, for example reaction 319574 (rxn_idx=297591) says that 'S-(methylthiomethyl)cysteine 4-oxide = (methylthio)methane-sulfenic acid + 2-aminoacrylate'  (nearly the same reactants apart from a side-chain far away from the reactants) - based on these and other examples the new products are suggested. Since there are many examples where only the S-C bond is cleaved, this reaction seems more likely than the one recorded in BRENDA. So I think this one is treated correctly. However, if you do not want suggested products, you can filter for them (in the 'source' tab).

For the third question. The mapping is also flagged as 'suggested', meaning that unfortunately the native BRENDA entry could not be mapped. So if you filter for suggested reactions, you would not see this one. I think I have to add a filter for suggested reactions if they produce the exact same product to not use them. So thanks a lot for bringing this up.
The reason that the reaction could not be mapped is that there is no balanced reaction (the product side is missing a water molecule). I.e., the BRENDA entry is 'D-glucose + kojibiose = 4-O-alpha-kojibiosyl-D-glucose {r}', but should have been 'D-glucose + kojibiose = 4-O-alpha-kojibiosyl-D-glucose + water {r}'. If I manually add the water, it resolved fine to the expected reaction. I will have to check whether other reactions are also affected by missing water molecules, maybe that could be added to the preprocessing.

In all cases, thank you very much for reporting! These are definitely things that can be improved in the next version, and I will happily add them. From what I have seen, these should not occur frequently, and not impact the overall usability/performance of EnzymeMap a lot, so for anyone using Version 2, they should not be a big problem.
I think you wanted to filter for suggested reactions anyways (and maybe it would also be good to filter everything with reactants=products for now), so most of these should not affect your work hopefully.

Regarding reactions with rule_id=-1: In general, that rule_id is assigned if the achiral reactants equal the achiral products. In general, this is necessary to correctly map cis/trans and D/L isomerase, because there is no according rule in the Broadbelt reaction set (which does not account for chirality). So, rule_id=-1 doesn't mean that the reaction is wrong per-se. Now, regarding redox-reactions, I have filtered all EC=1...* with rule_id=-1 and find 17 unique reactions:
* 8 are the same on reactant and product side because they are the same on both sides in BRENDA. In some of those cases, e.g. "NAD+ + NADH = NADH + NAD+", there obviously is a reaction which we are not detecting. However, these reactions might not be very interesting for a synthesis perspective anyways.

* 5 correspond to cis/trans isomerase reactions, and correspond to the respective BRENDA entry (although that sounds like a misclassification to me)

* 4 are the same on reactant/product, although the BRENDA entry is not the same, because sulfur and H2S, as well as sulfide all gets resolved to "H2S" when changing trivial names to SMILES, and another one resolves ferricytochrome c and ferrocytochrome c to the same SMILES.
  So for redox-reactions (EC=1._._.*) I didn't find anything unusual (although I agree they could have been filtered out). Did you mean other reactions? Since you mentioned explicit Hs, I somehow think that you are referring to other reactions? It should not matter whether the hydrogens are explicit or implicit, since a reduction/oxidation changes the number of hydrogens, and that is reflected in both version of SMILES (explicit/implicit Hs). We just do not track with hydrogen is which.

For isomerase reactions (filtered all EC=5*.. with rule_id=-1) there are 1271 unique reactions. For 1163 of them, the SMILES is not the same on reactant and product side, but differs in stereochemistry, e.g. cis/trans or D/L, so that is correct from my point of view (maybe you did not mean those, or maybe you looked at achiral educts/products?). Many of those were wrong in version 1, because we didn't have the rule_id=-1 and then we got rules that changed some atom numbers which did not reflect reality. For the 108 remaining ones, the reactant and product side is the same. For some of them, the BRENDA entry also has the same educts/products. However, most of them actually do undergo some isomerisation in the BRENDA entry, but the names get resolved to the same SMILES. I am not sure how to deal with this in an automated way (apart from removing reactions were the educts equal the products altogether). Since these are not many reactions, it would also be doable to manually correct the SMILES in the next version.
So, I am glad that there is no problem in the mapping itself (so no bug introduced when I corrected the isomerase mappings), but the issues arise from resolving names to SMILES, or are already in BRENDA. Nevertheless, these should be corrected in the next version.
I will look at the other problems later today.

Yes the redox things that looked are related to what you mentioned. A few examples below, all with reaction rule -1:
S>>S
NC@@HC(=O)O.S>>NC@@HC(=O)O.S
COC1=C(OC)C(=O)C(C/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CCC=C(C)C)=C(C)C1=O.S>>COC1=C(OC)C(=O)C(C/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CC/C=C(\C)CCC=C(C)C)=C(C)C1=O.S

Many thanks for following up on all these different comments! Seems to me that the impact on the dataset itself is indeed quite small, and that these are edge cases. Suggest we close the issue.


Native blob disposal from finalizer thread

Describe the bug
Native blob disposal from finalizer thread
Backtrace:    at System.Environment.get_StackTrace()
   at Avalonia.Platform.Internal.UnmanagedBlob.Dispose() in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Base/Platform/Internal/UnmanagedBlob.cs:line 87
   at Avalonia.Skia.WriteableBitmapImpl.ReleaseProc(IntPtr address, Object ctx) in /home/spaccabit/GitHub/Avalonia/src/Skia/Avalonia.Skia/WriteableBitmapImpl.cs:line 177
   at SkiaSharp.SKBitmap.<>c__DisplayClass108_0.<InstallPixels>b__0(IntPtr addr, Object _)
   at SkiaSharp.DelegateProxies.SKBitmapReleaseDelegateProxyImplementation(Void* address, Void* context)
   at SkiaSharp.SkiaApi.sk_bitmap_destructor(IntPtr cbitmap)
   at SkiaSharp.SkiaApi.sk_bitmap_destructor(IntPtr cbitmap)
   at SkiaSharp.SKBitmap.DisposeNative()
   at SkiaSharp.SKNativeObject.Dispose(Boolean disposing)
   at SkiaSharp.SKBitmap.Dispose(Boolean disposing)
   at SkiaSharp.SKNativeObject.Finalize()

Blob created by    at System.Environment.get_StackTrace()
   at Avalonia.Platform.Internal.UnmanagedBlob..ctor(Int32 size) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Base/Platform/Internal/UnmanagedBlob.cs:line 53
   at Avalonia.Platform.StandardRuntimePlatform.AllocBlob(Int32 size) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Base/Platform/StandardRuntimePlatform.cs:line 15
   at Avalonia.Skia.WriteableBitmapImpl..ctor(PixelSize size, Vector dpi, PixelFormat format, AlphaFormat alphaFormat) in /home/spaccabit/GitHub/Avalonia/src/Skia/Avalonia.Skia/WriteableBitmapImpl.cs:line 107
   at Avalonia.Skia.PlatformRenderInterface.CreateWriteableBitmap(PixelSize size, Vector dpi, PixelFormat format, AlphaFormat alphaFormat) in /home/spaccabit/GitHub/Avalonia/src/Skia/Avalonia.Skia/PlatformRenderInterface.cs:line 228
   at Avalonia.Media.Imaging.WriteableBitmap.CreatePlatformImpl(PixelSize size, Vector& dpi, Nullable`1 format, Nullable`1 alphaFormat) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Base/Media/Imaging/WriteableBitmap.cs:line 77
   at Avalonia.Media.Imaging.WriteableBitmap..ctor(PixelSize size, Vector dpi, Nullable`1 format, Nullable`1 alphaFormat) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Base/Media/Imaging/WriteableBitmap.cs:line 22
   at Avalonia.Controls.Primitives.ColorPickerHelpers.CreateBitmapFromPixelData(IList`1 bgraPixelData, Int32 pixelWidth, Int32 pixelHeight) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Controls.ColorPicker/Helpers/ColorPickerHelpers.cs:line 614
   at Avalonia.Controls.Primitives.ColorSlider.UpdateBackground() in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Controls.ColorPicker/ColorSlider/ColorSlider.cs:line 113
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.<>c.<.cctor>b__17_0(Object state)
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask)
   at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(Task task, Boolean canInlineContinuationTask)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Avalonia.Controls.Primitives.ColorPickerHelpers.CreateComponentBitmapAsync(Int32 width, Int32 height, Orientation orientation, ColorModel colorModel, ColorComponent component, HsvColor baseHsvColor, Boolean isAlphaMaxForced, Boolean isSaturationValueMaxForced) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Controls.ColorPicker/Helpers/ColorPickerHelpers.cs:line 307
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.<>c.<.cctor>b__8_0(Object state)
   at Avalonia.Threading.JobRunner.JobWithArg.Avalonia.Threading.JobRunner.IJob.Run() in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Base/Threading/JobRunner.cs:line 241
   at Avalonia.Threading.JobRunner.RunJobs(Nullable`1 priority) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Base/Threading/JobRunner.cs:line 38
   at Avalonia.Controls.Platform.InternalPlatformThreadingInterface.RunLoop(CancellationToken cancellationToken) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Controls/Platform/InternalPlatformThreadingInterface.cs:line 27
   at Avalonia.Threading.Dispatcher.MainLoop(CancellationToken cancellationToken) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Base/Threading/Dispatcher.cs:line 61
   at Avalonia.Controls.DesktopApplicationExtensions.Run(Application app, CancellationToken token) in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Controls/DesktopApplicationExtensions.cs:line 55
   at LinuxFramebufferPlatformExtensions.StartLinuxDirect[T](T builder, String[] args, IOutputBackend outputBackend, IInputBackend inputBackend) in /home/spaccabit/GitHub/Avalonia/src/Linux/Avalonia.LinuxFramebuffer/LinuxFramebufferPlatform.cs:line 162
   at LinuxFramebufferPlatformExtensions.StartLinuxFbDev[T](T builder, String[] args, String fbdev, Double scaling, IInputBackend inputBackend) in /home/spaccabit/GitHub/Avalonia/src/Linux/Avalonia.LinuxFramebuffer/LinuxFramebufferPlatform.cs:line 146
   at ControlCatalog.NetCore.Program.Main(String[] args) in /home/spaccabit/GitHub/Avalonia/samples/ControlCatalog.NetCore/Program.cs:line 47
Native blob disposal from finalizer thread
Backtrace:    at System.Environment.get_StackTrace()
   at Avalonia.Platform.Internal.UnmanagedBlob.Dispose() in /home/spaccabit/GitHub/Avalonia/src/Avalonia.Base/Platform/Internal/UnmanagedBlob.cs:line 87
   at Avalonia.Skia.WriteableBitmapImpl.ReleaseProc(IntPtr address, Object ctx) in /home/spaccabit/GitHub/Avalonia/src/Skia/Avalonia.Skia/WriteableBitmapImpl.cs:line 177
   at SkiaSharp.SKBitmap.<>c__DisplayClass108_0.<InstallPixels>b__0(IntPtr addr, Object _)
   at SkiaSharp.DelegateProxies.SKBitmapReleaseDelegateProxyImplementation(Void* address, Void* context)
   at SkiaSharp.SkiaApi.sk_bitmap_destructor(IntPtr cbitmap)
   at SkiaSharp.SkiaApi.sk_bitmap_destructor(IntPtr cbitmap)
   at SkiaSharp.SKBitmap.DisposeNative()
   at SkiaSharp.SKNativeObject.Dispose(Boolean disposing)
   at SkiaSharp.SKBitmap.Dispose(Boolean disposing)
   at SkiaSharp.SKNativeObject.Finalize()

To Reproduce
Steps to reproduce the behavior:

Build Avalonia
Go to Linux Terminal
Go to samples/ControlCatalog.NetCore/bin/Debug/net6.0
Write dotnet ControlCatalog.NetCore.dll
Go ColorPicker example
See error in console

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: Linux Ubuntu 22.04.1 64bit
Version: 84791dbf2e772cad889d575ef64000b5ba3d637a

Additional context
Add any other context about the problem here.

There are couple of places where writeable bitmap should be reused in ColorPicker code. But the most of these finalizer thread disposal comes from this slider part
https://github.com/AvaloniaUI/Avalonia/blob/master/src/Avalonia.Controls.ColorPicker/ColorSlider/ColorSlider.cs#L113

Controls should dispose their bitmaps either when they no longer used or when such control is detached from the visual tree. If there are undisposed bitmaps being finalized it's a bug in the control code.

Note that releasing WriteableBitmaps when detaching from the visual tree negatively affects performance especially on the ColorSpectrum. This is because those bitmaps take some time to build and now have to be re-built every single time the control comes in and out of view. @kekekeks If there is a better way to do this without a performance impact using OnDetachedFromVisualTree() I would like to pursue it.

This is because those bitmaps take some time to build and now have to be re-built every single time the control comes in and out of view

Could you just reuse the same bitmap for all ColorSpectrum instances?

should be responsible for it's own lifecycle management as in WinUI

Yes, we have the same "lifecycle management" as WinUI which is "automatically release memory when GC collects WritableBitmap". We just print a warning in debug build when that happens.

This is because those bitmaps take some time to build and now have to be re-built every single time the control comes in and out of view

Could you just reuse the same bitmap for all ColorSpectrum instances?

Interesting idea but:

It would be a pretty annoying tear-up to the code
Would probably be limited benefit because the majority of the delay is calculating the actual bytes of the spectrum bitmap itself -- not instantiating the writeable bitmap and copying in the bytes. The bitmaps also do change between controls depending on selected color and certain parameters. Only the default spectrum bitmap which varies Hue and Saturation appears the same on all controls. It's possible to vary other parameters.

Yes, we have the same "lifecycle management" as WinUI which is "automatically release memory when GC collects WritableBitmap". We just print a warning in debug build when that happens.

If so that seems to contradict "Controls should dispose their bitmaps either when they no longer used or when such control is detached from the visual tree. If there are undisposed bitmaps being finalized it's a bug in the control code." In that if controls don't release these resources it's not an actual bug and the GC will handle it correctly freeing memory and resources.
With that in mind, it might be better to NEVER dispose when detached from the visual tree and only dispose instances when a new bitmap needs to be generated.

Calling .Dispose() on WriteableBitmap or attempting to re-use it causes intermittent crashes that usually don't throw exceptions. It's a hard, silent crash. Therefore, I'm probably not going to really fix this one for now... It sounds like it wasn't really a bug to begin with and is just a warning in debug mode.

.Dispose() on WriteableBitmap or attempting to re-use it causes intermittent crashes

Please, file a bug report. We have reference tracking code specifically to prevent the actual bitmap from being disposed while it's still in use by the renderer.

@kekekeks Created a bug report #9092


Submit Queue UI should have direct links to each tab

Right now I can't share a link to the 'Queue' tab. It also means that when you reload the page it always sticks you back on the first tab, which is annoying.
This should be trivial to do with html fragments.
/cc @brendandburns

Fixed by #463


Use instance ID throughout dashboard UI for disambiguation

Resource names on resource and telemetry pages:

Display the resource name if there is one resource with a name
Display the resource name + 7 chars of UID if there are multiple resources with a name. I changed the UI to display the old format, e.g. frontend-sdfgdg, because there are existing places that have trailing parentheses and it doesn't look good. e.g. frontend (sdfsdf) (1)

The decision for adding the UID is based on checking if there is more than one resource with a name. This feels a little hacky. It could be improved by the dashboard having knowledge about whether a resource is part of a replicaset and the replicaset count.

Resources and console logs could check to see if the resource belongs to a replicaset with multiple instances.
OpenTelemetry pages have to look at instance id vs application name. An improvement could come from linking the OTEL instance id to the hosting UID, then looking at replicaset.

Before:

After:

Filed https://github.com/dotnet/aspire/issues/1233 for replica sets. I think that data can be pulled out of dcp.