Support certificate identifier when loading from azure

This commit adds support for loading a certificate identifier. It does
so by checking if the identifier is a 'secret' or 'certificate' in
the same manner that the 'SecretIdentifier' and 'CertificateIdentifier'
APIs in core do. By checking for a substring.
This commit does not remove support for fetching the certificates from
the 'secrets' in keyvault to avoid a breaking change.
Ref ActiveLogin/ActiveLogin.Authentication#127
The url checking seems to work fine but I would like to spend a bit more time testing.

I've been trying this out now, and now I do remember the issue I ran into. I'll explain it below, and would be glad if we find a solution. I have one myself.
When I run this, I do get an exception because the cert is not exported with a private key. It fails here (line 119): exportedCertCollection.Cast<X509Certificate2>().First(x => x.HasPrivateKey)
According to this SO-answer, you need to access it as a secret to get it with the private key. Tried to send in the cert without private key, and then the SSL connection to BankID can't be established.
Did you get it working?
The URL:s are convention bases, so if we get /certificates/ we could replace that with /secrets/. But optimally we would find a way to export it with the private key, but seems to be designed in a way so we can't do that. Would love to discuss the best way :)

Wow, I had no idea. I read up on the link from the SO-answer and then also the very extensive description on keyvault secrets and certificates.
It seems to me we could do a few things.

Scrap this whole PR and clarify in documentation on why we need to use /secrets/.
Be convenient and replace /certificates/ with secrets.
Be friendly and add a descriptive exception if the user provides /certificates/ to make them change the url

Number 2 does hidden magic so I do not think I like it. 1 and 3 would be most user friendly I suppose.

Also see link https://azidentity.azurewebsites.net/post/2018/07/03/azure-key-vault-certificates-are-secretsDen 2 nov. 2019 21:33 skrev Peter Örneholm notifications@github.com:I've been trying this out now, and now I do remember the issue I ran into. I'll explain it below, and would be glad if we find a solution. I have one myself.
When I run this, I do get an exception because the cert is not exported with a private key. It fails here (line 119): exportedCertCollection.Cast().First(x => x.HasPrivateKey)
According to this SO-answer, you need to access it as a secret to get it with the private key. Tried to send in the cert without private key, and then the SSL connection to BankID can't be established.
Did you get it working?
The URL:s are convention bases, so if we get /certificates/ we could replace that with /secrets/. But optimally we would find a way to export it with the private key, but seems to be designed in a way so we can't do that. Would love to discuss the best way :)
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.

Number 2 does hidden magic so I do not think I like it. 1 and 3 would be most user friendly I suppose.

I just relized checking for /certificate/ could go wrong as well. If you call your secret certificate part of the url will be /certificate/, so if we throw an error, we would need to search for something more specific.
Maybe start with alt. 1 for now?

Yes. Will abondon this now and create a new issue for the documentation.


fixed dsn connection string format for SQL Server

Using MS SQL server was not possible due to incorrect parameter names being used in the dsn connection string. Fixed those, can now connect to MS SQL Server successfully.

Thanks! I will look at it soon!


Compile fails with tonic 0.8

Bug Report
Version
0.8
Platform
5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Crates
Description
After upgrading from 0.7 to 0.8, build started to fail with this message.
  --- stdout
  cargo:rerun-if-changed=proto/reflection.proto
  cargo:rerun-if-changed=proto/

  --- stderr
  thread 'main' panicked at '
  Could not find `protoc` installation and this build crate cannot proceed without
  this knowledge. If `protoc` is installed and this crate had trouble finding
  it, you can set the `PROTOC` environment variable with the specific path to your
  installed `protoc` binary.

This seems to me the change https://github.com/hyperium/tonic/pull/1021 makes it require protoc for some reason.
Because my project was ok with tonic 0.7, this is kind of a bug.

I am also experiencing this behavior. To get past it I installed protoc as the error says to get it to work. Would like to know what caused this and if it can be fixed

Related to this, I expect that some people will also be seeing something similar to the following:
error: failed to run custom build command for `tonic-types v0.6.0 (/tonic/tonic-types)`

Caused by:
  process didn't exit successfully: `/tonic/target/debug/build/tonic-types-124bdc186e7641a9/build-script-build` (exit status: 101)
  --- stdout
  Running: "/usr/bin/protoc" "--include_imports" "--include_source_info" "-o" "/tmp/prost-buildaQyv1g/prost-descriptor-set" "-I" "proto/" "proto/status.proto"

  --- stderr
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "protoc failed: google/protobuf/any.proto: File not found.\nstatus.proto:19:1: Import \"google/protobuf/any.proto\" was not found or had errors.\nstatus.proto:46:12: \"google.protobuf.Any\" is not defined.\n" }', tonic-types/build.rs:2:71
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

In the case of said related error, as per https://stackoverflow.com/a/61368354 it means that one has protoc installed but not the protobuf resource files.
On Ubuntu, if you are installing via the official repos, make sure both are installed:
apt update && apt upgrade -y
apt install -y protobuf-compiler libprotobuf-dev

On Alpine Linux, if you are installing via the official repos, make sure both are installed:
apk add protoc protobuf-dev

Or alternatively, if the version of protobuf in the repositories is too old then you could instead fetch a specific newer version from GitHub releases page of protobuf, and extract it into /usr. This is what we are doing currently in my company for a CI pipeline where we want a specific version of protobuf.
PROTOC_VERSION=3.20.1
PB_REL="https://github.com/protocolbuffers/protobuf/releases"

which zip || apt update && apt upgrade -y && apt install zip -y

curl -sSLO "$PB_REL/download/v$PROTOC_VERSION/protoc-$PROTOC_VERSION-linux-x86_64.zip"

unzip -o protoc-$PROTOC_VERSION-linux-x86_64.zip -d "/usr"

(On an ordinary end-user machine, or on a long living server, or similar, you would not want to extract stuff into /usr like that but in this case it's an ephemeral Docker container so it does no harm.)

Generating protobuf via prost now requires protoc in the path (or you can pass in a FileDescriptorSet). The related change is here https://github.com/tokio-rs/prost/pull/657. I believe though that tonic didn't get its docs as updated as prost's so I've opened this issue https://github.com/hyperium/tonic/issues/1049 that we can follow up on.
If you need help with how to get CI working you can also take a look at how tonic and prost both do it. I am going to close this issue, please feel free to reopen if you have any more questions.

It feels unfortunate and heavy to require protoc as a dep with tools such as buf[1] around with rust support. I have designed my projects to intentionally NOT require protoc as a dep. Now it must be added to support a single proto required for reflection?
[1] https://github.com/bufbuild/buf

@hexfusion protoc has always been a requirement, before it was just included as a binary in the published crate. But thi is a big security issue and was removed. There is support for buf at https://docs.rs/protoc-gen-tonic/0.2.0/protoc_gen_tonic/

Thanks for the clarification.


[DataGrid] NRE when deleting item right after editing

Describe the bug
Editing an item and triggering it's removal from the data grid items list results in an NRE when committing the edit.
To Reproduce
Steps to reproduce the behavior:

Start from the Avalonia MVVM template, and install data grid

On MainWindow.xaml:
<DataGrid AutoGenerateColumns="True" Items="{Binding Items}" />

On MainWindowViewModel.cs:
public class MainWindowViewModel : ViewModelBase
{
    public sealed class Item : ViewModelBase
    {
        private string _value;

        public Item(MainWindowViewModel owner, string value)
        {
            _value = value;

            this.WhenAnyValue(x => x.Value).Subscribe(_ => owner.Items.Remove(this));
        }

        public string Value
        {
            get => _value;
            set => this.RaiseAndSetIfChanged(ref _value, value);
        }
    }

    public MainWindowViewModel()
    {
        Items.Add(new Item(this, "A"));
        Items.Add(new Item(this, "B"));
    }

    public ObservableCollection<Item> Items { get; } = new ObservableCollection<Item>();
}

Run the app, and edit one of the values, then commit the edit, and it throws an NRE.

Expected behavior
The item should be successfully removed from the list.
Desktop (please complete the following information):

OS: Windows 11
Version: 11.0.0-preview2

I think it works if you use the Dispatcher to remove the items. That way you can wait for Avalonia to commit the changes before removing the item. I've used the following item view model for my tests:
public sealed class Item : ViewModelBase
{
    private string _value;

    // Added two private fields which we need to access later.
    private MainWindowViewModel owner;
    private bool _init;

    public Item(MainWindowViewModel owner, string value)
    {
        this.owner = owner;
        _value = value;

        this.WhenAnyValue(x => x.Value)
            .Subscribe(_ => RemoveMe());

        // RxUI is calling WhenAnyValue in the constructor for some reason. We don't want this, so we wait for the model to be ready
        _init = true;
    }

    // I made a function of it. Feel free to move it to your lambda if you want to.
    private void RemoveMe()
    {
        if (!_init) return;
        Dispatcher.UIThread.Post(() => owner.Items.Remove(this));
    }

    public string Value
    {
        get => _value;
        set => this.RaiseAndSetIfChanged(ref _value, value);
    }
}

tbh I don't consider this a bug. Therefore I move this into a discussion. If you are okay with the solution showed above, please mark it as the answer for others.


GA Sunsetting

Hi, with the sunsetting of the GoogleAnalytics SDK, are there plans to remove it as a dependency for GTM for SPM?
Thanks!

GoogleTagManager v8.0.0 has been published and no longer depends on GoogleAnalytics.


Link to Adoption Kit loads external non-related website

PROVISIONING NOTE: If you are having issues with the SharePoint look book while trying to provision M365 learning pathways please do not post that issue to this repo as we cannot help you here. Please post all questions and issues for the SharePoint look book to SharePoint/sp-provisioning-service GitHub Issues List.

WEB PART NOT LOADING: If you are getting the error: "Microsoft 365 learning pathways has a configuration issue. Ask your administrator for assistance". Please review the Diagnosing Pre-Check guidance before submitting your bug.

Describe the bug
Clicking on links to the "downloadable Adoption Kit" - https://teamworktools.azurewebsites.net/m365lp/m365lpadoptionkit.zip loads a non-related website: https://tempmailo.com/bl/
To Reproduce
Go to the Admin Success Centre > Drive Adoption > Drive adoption of learning pathways > Use the downloadable Adoption Kit section > in the first sentence, click on the link to the "downloadable adoption kit".
Expected behavior
I would expect the link to download the zip file, not open a website.
Screenshots

Learning Pathways version number
<IP_ADDRESS>
Additional context

Add any other context about the problem here.

Hi @MandyMcPhail, Thank you for bringing this issue to our attention. However this is not a Learning Pathways webpart issue but a content issue as we are just pulling the underlying content and displaying it. We will try to work with content team to sort this out.

Thanks for the response @amitkh22

@MandyMcPhail The link in the document has been fixed. In testing it I realized that there is also a link in the right side quick links to the adoption kit in the Success center. Make sure you change it there as well. We will need to make an update of the template to make it available in the LookBook but here is the url of the toolkit.
https://adoption.microsoft.com/files/m365lp/m365lpadoptionkit.zip

That's great, thanks very much!

@MandyMcPhail It took a while but we updated the template on the LookBook and this issue was resolved. Thanks for bringing this to our attention.

Putting the correct links here in case others need to reference this.
Learning pathways documentation | https://aka.ms/M365LPDocs
Feedback and support (issues list) | https://aka.ms/M365LPIssues
Adoption resources | https://aka.ms/M365LP
Downloadable adoption kit | https://aka.ms/M365LPAdoptionKit
Link	Target URL	AKA
Learning pathways documentation 	https://docs.microsoft.com/office365/customlearning/  https://aka.ms/M365LPDocs
Feedback and support (issues list)	https://github.com/pnp/custom-learning-office-365/issues https://aka.ms/M365LPIssues
Adoption resources 	https://adoption.microsoft.com/microsoft-365-learning-pathways/ https://aka.ms/M365LP
Downloadable adoption kit 	https://adoption.microsoft.com/files/m365lp/m365lpadoptionkit.zip https://aka.ms/M365LPAdoptionKit


Support for Pre-Tokenized Datasets in SFTTrainer

Problem:
I have a dataset in the following format:
Dataset({
    features: ['input_ids', 'attention_mask'],
    num_rows: 43471
})

The dataset is already tokenized, and I would like to skip the tokenization step in SFTTrainer, as it takes a considerable amount of time (approximately 1 hour on my dataset) to encode each time.
Feature Request:
I would like to request a feature that allows users to input pre-tokenized datasets directly into the SFTTrainer, bypassing the need for the 'text' column and the subsequent tokenization step. This would significantly improve the training efficiency for users with large datasets.
Note:
I am unsure if this feature already exists, so I am reaching out to the community to clarify and request support for pre-tokenized datasets in SFTTrainer. Thank you!

I think this should work out of the box, or you could just use the transformers.Trainer instead of the SFTTrainer.

The goal of the SFTTrainer as compared to the transformers.Trainer is to make that step obsolete. Otherwise you could use as @tcapelle proposed the original trainer.

Thank you for the suggestion. I am now using the 'text' column from the dataset, and I fixed the slow tokenization by setting dataset_num_proc to 5. Now, it's working fine. Thanks.


3/jwt plugin

https://synapse.leankit.com/Boards/View/285322106/334869852

Would you mind adding a "Plugins" section to the README that will function as a place to list the plugins made available by this package with simple usage examples.


Offerings policy bug when teacher is project admin, but has classes outside that project

see PT Story
We first noticed this Bug when Cynthia wasn't able to get a clue dashboard for her class when a second teacher (Leslie) was.
If the user_id is for the current user and the current_user is a project_admin or a project_researcher for any project, but is not in a cohort associated with one of those projects, the api endpoint will not return any offerings.
see: https://github.com/concord-consortium/rigse/blob/master/app/policies/portal/offering_policy.rb#L15-L32
which uses: https://github.com/concord-consortium/rigse/blob/master/app/policies/portal/teacher_policy.rb#L9-L15
This PR adds failing tests, and then gets the tests passing with a new raw SQL query which finds id selection to apply to the portal_teacher scope.

@scytacki  -- I think this is good for a final review.

Thanks for select_values @scytacki  I added it in c2c114227
Local specs seemed fine, just waiting on travis.


Maya: Implementation of Ornatrix product type

Changelog Description
Implementation of Ornatrix Exports in Maya

Ornatrix Rig: Publishing Ornatrix or any Ornatrix related objects. If there are any painted map(s) by user, they are published too and the published maps would be located in the {your publish folder}/resources
Ornatrix Cache: Publishing Ornatrix fur simulation as Ornatrix Alembic cache. It can be loaded as Arnold StandIn, Vray Proxy and Alembic reference while it can be loaded as the file cache in the Ornatrix-related GuideFromMeshNode.

Additional info
Although Ornatrix cache loader can add HairfromGuideNode into the operator stack, it can only be shown once into the stack editor (due to some api limitation). Users need to select the objects they want to load the file cache in order to add the cached nodes "safely" into the operator stack. (But users can still find the guide node in the container set)
Requires ayon-core update
Please test along with the branch from this PR https://github.com/ynput/ayon-core/pull/734, otherwise it will error out the collect resources path when publishing
This error hints at that the ayon-core PR branch is not used:
Traceback (most recent call last):
  File "C:\Users\lbate\AppData\Local\Ynput\AYON\dependency_packages\ayon_2406251801_windows.zip\dependencies\pyblish\plugin.py", line 528, in __explicit_process
    runner(*args)
  File "C:\Users\lbate\AppData\Local\Ynput\AYON\addons\maya_0.2.9\ayon_maya\plugins\publish\extract_ornatrix_rig.py", line 43, in process
    image_search_path = instance.data["resourcesDir"]
KeyError: 'resourcesDir

Testing notes:
Ornatrix Rig

Create Ornatrix Rig with your geometry which is assigned with ornatrix fur
Publish
If the content doesn't include the Ornatrix guide, it can't pass the validator.
Texture maps and ma scene would be published along with rigsettings which stores all the ornatrix related parameters.
Start new scene and load the reference scene of the published ornatrix rig
Once you start publisher, the ornatrix cache will be automatically created with the references of the ornatrix rig.
If you load the ornatrix cache, it will create group with a mesh shape which includes the GuideFromMeshNode storing the abc file cache.
You can also load the ornatrix cache with Arnold StandIn, Vray Proxy and Ornatrix Alembic
You can also connect your ornatrix rig with the same meshes located in pointcache and animation product type (select your mesh from pointcache and animation product type first and then your ornatrix product type)

Im getting error when trying publish oxrigMain because of my source file textures
Traceback (most recent call last):
  File "C:\Users\lbate\AppData\Local\Ynput\AYON\dependency_packages\ayon_2406251801_windows.zip\dependencies\pyblish\plugin.py", line 528, in __explicit_process
    runner(*args)
  File "C:\Users\lbate\AppData\Local\Ynput\AYON\addons\maya_0.2.9\ayon_maya\plugins\publish\extract_ornatrix_rig.py", line 43, in process
    image_search_path = instance.data["resourcesDir"]
KeyError: 'resourcesDir'

Im getting error when trying publish oxrigMain because of my source file textures
my files sits within the work folder in ../ornatrix_maps subfolder
C:\Projects\Ayon_Tests\maya\characters\ray\work\fx\ornatrix_maps\densityHair.png C:\Projects\Ayon_Tests\maya\characters\ray\work\fx\ornatrix_maps\distributionMap.png
Traceback (most recent call last):
  File "C:\Users\lbate\AppData\Local\Ynput\AYON\dependency_packages\ayon_2406251801_windows.zip\dependencies\pyblish\plugin.py", line 528, in __explicit_process
    runner(*args)
  File "C:\Users\lbate\AppData\Local\Ynput\AYON\addons\maya_0.2.9\ayon_maya\plugins\publish\extract_ornatrix_rig.py", line 43, in process
    image_search_path = instance.data["resourcesDir"]
KeyError: 'resourcesDir'

@LiborBatek  have you installed the addon with this branch? https://github.com/ynput/ayon-core/pull/734

@moonyuet ouch I have overlooked that needed branch! my bad...sry will do another testing round!

Please test along with the branch from this PR https://github.com/ynput/ayon-core/pull/734, otherwise it will error out the collect resources path when publishing

If that PR is required for this PR to work, then you have to raise required version of ayon-core to >0.4.0 in package.py.

so I was able to produce publish of oxrig this time and load it as oxGroom into new workfile but I was not able to use connect action...

it didnt print any log into script editor nor console tho
Im not sure with published .ma representation as it behaves as before bringing in the full character asset with ornatrix setup - do we want that? maybe by first using the locally duplicated meshShape for setting up the Ornatrix rig might do? if we want to follow such workflow...
I wasnt able to perform Load OxCache from the published Ornatrix setup (as a side product when published)? whats the supposed workflow for it?

as I wasnt able to reconnect to my pointcache animated character I didnt test to produce oxCache for particular shot animation yet... I will try at least on manually connected oxRig to see how it behaves for publishing process...

as I wasnt able to reconnect to my pointcache animated character I didnt test to produce oxCache for particular shot animation yet... I will try at least on manually connected oxRig to see how it behaves for publishing process...

Once you loaded the Oxcache family, you can find the guide node with cached abc in the container set.

so I was also able to produce oxCache publish on manually setuped oxRig> pointcache animation for particular shot but there is no currently any Action in manage available to the user atm?!

so I was able to produce publish of oxrig this time and load it as oxGroom into new workfile but I was not able to use

so I was also able to produce oxCache publish on manually setuped oxRig> pointcache animation for particular shot but there is no currently any Action in manage available to the user atm?!

there is a bug for this, will fix it as soon as possible.

when manually bringing the published oxCache into the workfile by performing File>Import>OrnatrixAlembic there is a cache loaded into my active workfile but it is not sequenceaka lacks animation and also doesnt look as the oxRig setup ...the hair strands are ackwardly scaled aka thick?!

[ ]  use OxAlembicExport in Ornatrix Alembic Extractor
[ ]  connect Yeti Nodes issue.

@antirotor @LiborBatek can you take a look at it again?

Im getting error when trying to publish oxrig product when extract oxcache happens

P.S. I m using ayon-core develop as that needed ayon-core PR is already merged into it

It could be that the melscript doesn't work quite well with the fstring, I have made the change to the format string and test it, it works in my side(and should be working for yours)

hmm did pull that your commit but still no luck:
DEBUG: ['hairdoShape']
Traceback (most recent call last):
  File "C:\Users\lbate\AppData\Local\Ynput\AYON\dependency_packages\ayon_2406251801_windows.zip\dependencies\pyblish\plugin.py", line 528, in __explicit_process
    runner(*args)
  File "c:\Work\REPO\ayon-maya\client\ayon_maya\plugins\publish\extract_ornatrix_cache.py", line 29, in process
    self._extract(ox_abc_path, attr_values)
  File "c:\Work\REPO\ayon-maya\client\ayon_maya\plugins\publish\extract_ornatrix_cache.py", line 87, in _extract
    return mel.eval("{0};".format(ox_export_cmd))
RuntimeError: Error occurred during execution of MEL script
line 1: Invalid syntax

hmm did pull that your commit but still no luck:
DEBUG: ['hairdoShape']
Traceback (most recent call last):
  File "C:\Users\lbate\AppData\Local\Ynput\AYON\dependency_packages\ayon_2406251801_windows.zip\dependencies\pyblish\plugin.py", line 528, in __explicit_process
    runner(*args)
  File "c:\Work\REPO\ayon-maya\client\ayon_maya\plugins\publish\extract_ornatrix_cache.py", line 29, in process
    self._extract(ox_abc_path, attr_values)
  File "c:\Work\REPO\ayon-maya\client\ayon_maya\plugins\publish\extract_ornatrix_cache.py", line 87, in _extract
    return mel.eval("{0};".format(ox_export_cmd))
RuntimeError: Error occurred during execution of MEL script
line 1: Invalid syntax

Updated commitbe63eb6 Won't have any issues when you are publishing the oxrig family due to the missing data

Now finally publish of oxRig works and without any errors finnishes:

But Im not able to connect loaded oxRig and pointcache or animation ....always getting following pop up msg:

small note:
I would also like to ask, if you are checking for the particular groom object by name or by other method...as I can imagine it would be best to strip namespace for such operation as they can differ... something like filenamespace:groomShape and instead using just groomShape as namespaces can cause  issues.
this being true if I decide to setup my ornatrix rig on local duplicate of groomShape so it wont have such namespace then (btw I was able to publish from setup like this) and I think it should be the preffered one as it wont bring any other objects from rig asset but only that groom mesh + ornatrix node

btw I was rather testing both ways (to setup Ornatrix directly using referenced char rig and its groom geo) and also my separate duplicate setup....however both werent working for connect oxrig to geo ...so it wasnt caused by that.

btw I was rather testing both ways (to setup Ornatrix directly using referenced char rig and its groom geo) and also my separate duplicate setup....however both werent working for connect oxrig to geo ...so it wasnt caused by that.

are you at the updated version as I can connect with my ornatrix rig in my side.(Very simple setup)
https://github.com/user-attachments/assets/700defb5-fa90-4b2c-90c4-9a84f97ef549

hmm you know what seeing your vid...it seems that Im trying the connect action with oxRig .ma not the oxg.zip is that an issue?? as I have loaded that maya oxrig product in precedent step...

hmm you know what seeing your vid...it seems that Im trying the connect action with oxRig .ma not the oxg.zip is that an issue?? as I have loaded that maya oxrig product in precedent step...

I also have tried to load with oxrig. Guess we can also support current selection if it is match to the node name in the data.
Will update you after adding it into the connector.
https://github.com/user-attachments/assets/d53244b9-96e0-4f28-84c2-2002296c5f13

@LiborBatek I have updated the PR to support current selection and the node without the namespace(the node name is grabbed from the data stored in the .rigsetting file btw)

@moonyuet I'm hitting this error:

DEBUG: Looking in settings for scene type ...
DEBUG: Writing metadata file: C:\Users\User\AppData\Local\Temp\pyblish_tmp_b6duu4d8\ornatrix.rigsettings
Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403061937_windows.zip\dependencies\pyblish\plugin.py", line 527, in __explicit_process
    runner(*args)
  File "E:\dev\ayon-maya\client\ayon_maya\plugins\publish\extract_ornatrix_rig.py", line 87, in process
    mel.eval(f'OxSaveGroom -path "{ox_groom_path}" -optional;')
RuntimeError: Error occurred during execution of MEL script
line 1: Please select a single hair shape from which to generate the groom

@moonyuet I'm hitting this error:

DEBUG: Looking in settings for scene type ...
DEBUG: Writing metadata file: C:\Users\User\AppData\Local\Temp\pyblish_tmp_b6duu4d8\ornatrix.rigsettings
Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403061937_windows.zip\dependencies\pyblish\plugin.py", line 527, in __explicit_process
    runner(*args)
  File "E:\dev\ayon-maya\client\ayon_maya\plugins\publish\extract_ornatrix_rig.py", line 87, in process
    mel.eval(f'OxSaveGroom -path "{ox_groom_path}" -optional;')
RuntimeError: Error occurred during execution of MEL script
line 1: Please select a single hair shape from which to generate the groom

you need to put the geometry instead of hairshape.

you need to put the geometry instead of hairshape.

Tried that as well..

But then I figured, maybe you meant the actual mesh shape.
Which got me to:

@BigRoy It works in my side.
[
https://github.com/user-attachments/assets/a62d33eb-2b4b-4c59-806e-7d0c345954fb
](url)

@BigRoy Continuing the conversation in discord about the connect ornatrix rig , we can connect the outmesh of target geometry shape node into the inmesh of the GuideFromMesh node. e.g. # connectAttr -f pSphereShape2.outMesh GuidesFromMesh1.inputMesh;
But after the connection, we need to parent the FurShape node into the target geometry to show the fur.

@BigRoy Connect Rigs work as expected and the order of the ox groom loader also looks good.
https://github.com/user-attachments/assets/cfd3cd75-aab1-4608-ad7f-794f50991965
@antirotor I think the implementation is finished. Can we merge this ticket?

@BigRoy I have implemented some changes on load_ornatrix_cache.py as the hair guide node needed to be select to show on the operator stack dialog.

TODO

[ ] Add validation for when user adds multiple meshes into the Ornatrix Rig instance

I guess there is a case which geometry is with more than one hair shape and it can still be published?

I can still publish without the validator btw.

Based on the discussion here me and @moonyuet have decided to:

Lift the single hair shape validation/restriction.
Remove the loader for Ornatrix Groom .ZIP (which with that may also remove the Extraction of that actually because then it's unused?)
Remove the loader for "Load Ornatrix Cache with Hair Guide" because it's complicated to get it working from an artist's perspective and it seems to suffer from the same bugs.

That brings this PR back to being able to generate Ornatrix RIGs and Alembic caches.
I'll port over the things I remove to another PR but turn that into a draft just so we can release the first feature set now @m-u-r-p-h-y @tatiana-ynput. @moonyuet also mentioned that most likely the client will be using the Extracted alembics anyway by loading into e.g. proxies or alike.

So it now works.
HOWEVER, please be aware that the Ornatrix Alembic Loader does NOT import the animation of the cache.

I spent way too much time thinking the export didn't include the animation - just to figure out that importer doesn't load it. Arnold Standins do load them and render fine.
@moonyuet it'd be good if you can do review.
We've reverted some logic closer to what @LiborBatek reported issues with - but I was unable whatsoever to reproduce those issues (in any way!), so let's take them with a grain of salt and assume it was user error at the time.
I'll mark it approved so you can check it one more @moonyuet and then can merge  it.

So it now works.
HOWEVER, please be aware that the Ornatrix Alembic Loader does NOT import the animation of the cache.

I spent way too much time thinking the export didn't include the animation - just to figure out that importer doesn't load it. Arnold Standins do load them and render fine.
@moonyuet it'd be good if you can do review.
We've reverted some logic closer to what @LiborBatek reported issues with - but I was unable whatsoever to reproduce those issues (in any way!), so let's take them with a grain of salt and assume it was user error at the time.
I'll mark it approved so you can check it one more @moonyuet and then can merge  it.

Okay good. I will check tomorrow and merge it. Thank you so much for the help!

@BigRoy Looks good and we would merge this.
https://github.com/user-attachments/assets/b6b8744b-ecee-444d-82f5-e1ac36bf000d


AadHttpClient - Timeout / Retry Policy

What type of issue is this?
Question
What SharePoint development model, framework, SDK or API is this about?
💥 SharePoint Framework
Target SharePoint environment
SharePoint Online
What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[ ] 💥 Microsoft Edge
[X] 💥 Google Chrome
[ ] 💥 FireFox
[ ] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[ ] not applicable
[ ] other (enter in the "Additional environment details" area below)

Additional environment details
No response
Issue description
I have a question around the AadHttpClient. We send a post request to a custom Web API, which processes documents, and can take a long time to complete. I with the smaller requests it works fine, but on the larger requests I notice two things:

The request times out returning a 503 - even though the server is still processing
It retries the request once, so two requests are received at the server.

My question is, is it possible to set timeout limits with the http client and also set retry requests to 0?

Just to confirm, the client only retries after timeout occurs. But there is no retry request in the network traffic, so I assume this may be server related?

I have found someone with a similar issue, but they are able to configure the retry policy using the Volly Http Client:
https://stackoverflow.com/questions/26672015/web-api-controller-action-called-twice
My problem is I need the authenticated bearer token to authorise the function on the API, if the AadHttpClient retry policy can't be modified, then is it possible to get that token another way?

Any update?

You have full control over the headers, so if it can be set that way, you could try.  Additionally, the back end server (say Azure) can have their own timeouts that are not controllable by either side.
If you have an extremely long running server task, you'll probably need to take a different approach.  For example, the initial call can return a token that can be used to poll the service for a response when complete, you could use something like socket I/O to create a persistent channel, etc.


Invalid configuration object: mangleExports using 5.0.0-beta.14

Bug report
What is the current behavior?
When I use the example configuration for mangleExports:
Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.
 - configuration.optimization has an unknown property 'mangleExports'. These properties are valid:
   object { checkWasmTypes?, chunkIds?, concatenateModules?, flagIncludedChunks?, hashedModuleIds?, mangleWasmImports?, mergeDuplicateChunks?, minimize?, minimizer?, moduleIds?, namedChunks?, namedModules?, noEmitOnErrors?, nodeEnv?, occurrenceOrder?, portableRecords?, providedExports?, removeAvailableModules?, removeEmptyChunks?, runtimeChunk?, sideEffects?, splitChunks?, usedExports? }
   -> Enables/Disables integrated optimizations

If the current behavior is a bug, please provide the steps to reproduce.
create webpack.config.js as example profided in documentation:
module.exports = {
  //...
  optimization: {
    mangleExports: false
  }
};

What is the expected behavior?
Webpack should not mangle exports like the documentation provides
Other relevant information:
webpack version:  5.0.0-beta-14
Node.js version: v13.12.0
Operating System: linux
Additional tools<EMAIL_ADDRESS>

The error message is coming from webpack 4. Check that you are really using webpack 5.

$ webpack --version
5.0.0-beta.14
[webpack-cli] Promise rejection: Error: Cannot find module '../../../../package.json'
Require stack:
- /usr/lib/node_modules/webpack-cli/lib/groups/HelpGroup.js
- /usr/lib/node_modules/webpack-cli/lib/webpack-cli.js
- /usr/lib/node_modules/webpack-cli/lib/bootstrap.js
[webpack-cli] Error: Cannot find module '../../../../package.json'
Require stack:
- /usr/lib/node_modules/webpack-cli/lib/groups/HelpGroup.js
- /usr/lib/node_modules/webpack-cli/lib/webpack-cli.js
- /usr/lib/node_modules/webpack-cli/lib/bootstrap.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:976:15)
    at Function.Module._load (internal/modules/cjs/loader.js:859:27)
    at Module.require (internal/modules/cjs/loader.js:1036:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at HelpGroup.outputVersion (/usr/lib/node_modules/webpack-cli/lib/groups/HelpGroup.js:43:25)
    at WebpackCLI.runVersion (/usr/lib/node_modules/webpack-cli/lib/webpack-cli.js:303:32)
    at runCLI (/usr/lib/node_modules/webpack-cli/lib/bootstrap.js:54:13)
    at Object.<anonymous> (/usr/lib/node_modules/webpack-cli/lib/bootstrap.js:117:1)
    at Module._compile (internal/modules/cjs/loader.js:1147:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1167:10)

$ cat webpack.config.js
module.exports = {
	optimization: {
		mangleExports: true
	}
};
$ webpack js/main.js -o dist/main.js
Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.
 - configuration.optimization has an unknown property 'mangleExports'. These properties are valid:
   object { checkWasmTypes?, chunkIds?, concatenateModules?, flagIncludedChunks?, hashedModuleIds?, mangleWasmImports?, mergeDuplicateChunks?, minimize?, minimizer?, moduleIds?, namedChunks?, namedModules?, noEmitOnErrors?, nodeEnv?, occurrenceOrder?, portableRecords?, providedExports?, removeAvailableModules?, removeEmptyChunks?, runtimeChunk?, sideEffects?, splitChunks?, usedExports? }
   -> Enables/Disables integrated optimizations

@JAicewizard Can you create reproducible test repo?

https://github.com/JAicewizard/webpack_issue
I have installed webpack and webpack cli using:
# npm install -g<EMAIL_ADDRESS># npm install -g<EMAIL_ADDRESS>


Please provide the weight pruned InceptionV3 and MobileNetV1 models

I want to try the compression performance of weight pruning of TensorFlow. Is it OK for you to provide the model after pruning? e.g., the 50% sparsity MobileNetV1?

@lee-bin Please have a look on this link. Let us know if that helps. Thanks!

Thanks! I had tried that before, but it's MNIST model, not InceptionV3 and MobileNetV1.
Could you please share the codes or the pruned model for these two models?
I know that you have tested it from here.
I'm very new to this, any help would be appreciated.

@lee-bin This is not Build/Installation or Bug/Performance issue. Please post this kind of support questions at StackOverflow. There is a big community to support and learn from your questions. GitHub is mainly for addressing bugs in installation and performance. Thanks!

Thanks! I asked it at https://stackoverflow.com/questions/56470335/is-there-any-keras-code-to-reproduce-the-weight-pruning-of-mobilenet.


[v1.10] Fix race condition in DNS proxy when multiple DNS requests for the same name end up with policy drops

:lgtm:


chore: add 'lint:deps' script to check for unused and unlisted deps

This uses knip (https://knip.dev) to check dependencies, in hopes of avoiding
issues like #2473. This adds 'npm run lint:deps' and adds that to the
existing 'npm run lint'.
This change includes fixes for a handful of unused and unlisted deps.
For now knip is configured to only check 'production' deps. Checking non-prod
deps results in way too many false positives.
before the fixes in this commit
This is the output of knip --dependencies --production before the fixes included in this commit:
Unused dependencies (10)
@opentelemetry/sdk-metrics     packages/opentelemetry-host-metrics/package.json
@opentelemetry/resources       plugins/node/opentelemetry-instrumentation-aws-lambda/package.json
semver                         plugins/node/opentelemetry-instrumentation-dns/package.json
@opentelemetry/sdk-metrics     plugins/node/opentelemetry-instrumentation-mongodb/package.json
@opentelemetry/sdk-trace-base  plugins/web/opentelemetry-instrumentation-document-load/package.json
@opentelemetry/sdk-trace-web   plugins/web/opentelemetry-instrumentation-long-task/package.json
@opentelemetry/context-zone    plugins/web/opentelemetry-plugin-react-load/package.json
@opentelemetry/sdk-trace-base  plugins/web/opentelemetry-plugin-react-load/package.json
@opentelemetry/sdk-trace-web   plugins/web/opentelemetry-plugin-react-load/package.json
Unlisted dependencies (3)
@opentelemetry/otlp-transformer  packages/opentelemetry-test-utils/src/test-fixtures.ts
@opentelemetry/sdk-metrics       packages/opentelemetry-test-utils/src/test-utils.ts
@cucumber/messages               plugins/node/instrumentation-cucumber/src/instrumentation.ts

The cucumber one was about imported types, so would not have broken users of the published package.

On the lint test failure: It borks if the "version.ts" files haven't been generated:
> knip --dependencies --production

Unresolved imports (46)
./version  packages/opentelemetry-host-metrics/src/BaseMetrics.ts:19:9
./version  packages/winston-transport/src/OpenTelemetryTransportV3.ts:19:9
./version  plugins/node/instrumentation-amqplib/src/amqplib.ts:77:9
./version  plugins/node/instrumentation-cucumber/src/instrumentation.ts:41:9
./version  plugins/node/instrumentation-dataloader/src/instrumentation.ts:30:9
./version  plugins/node/instrumentation-fs/src/instrumentation.ts:24:9
...

On the lint test failure: It borks if the "version.ts" files haven't been generated

🤔  I wonder if we can specifically exclude by name

I wonder if we can specifically exclude by name

Got an answer in commit 47ee30b5c2f1d42690c87ff5759dc865d2ada098.

Ugh, perhaps the following is fine, but it is annoying. The addition of the knip dep at the top-level resulted in its<EMAIL_ADDRESS>dep being placed at the top-level ./node_modules/typescript... and the<EMAIL_ADDRESS>dep of every other package now being separately installed in every other package's own .../node_modules dir. Boo.

Ugh, perhaps the following is fine, but it is annoying.

Actually I think this is unacceptable. The result is a MUCH larger working copy:
Before:
$ npm ci
$ du -sh .
1.2G	.

After (i.e. this PR branch):
% du -sh .
4.9G	.

Ew.

(Profanity.)
I don't know how to solve this well. knip has a peerDep on typescript >=5.0.something.  I tried to install<EMAIL_ADDRESS>in opentelemetry-js-contrib's top-level ... to force us having the typescript that most packages will use installed at the top-level node_modules/typescript.  But the installing knip fails:
% npm install --save-dev knip
npm error code ERESOLVE
npm error ERESOLVE unable to resolve dependency tree
npm error
npm error While resolving<EMAIL_ADDRESS>npm error Found<EMAIL_ADDRESS>npm error node_modules/typescript
npm error   dev<EMAIL_ADDRESS>from the root project
npm error
npm error Could not resolve dependency:
npm error peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm error node_modules/knip
npm error   dev knip@"*" from the root project
npm error
npm error Fix the upstream dependency conflict, or retry
npm error this command with --force or --legacy-peer-deps
npm error to accept an incorrect (and potentially broken) dependency resolution.
npm error
npm error
npm error For a full report see:
npm error /Users/trentm/.npm/_logs/2024-10-10T22_06_05_024Z-eresolve-report.txt
npm error A complete log of this run can be found in: /Users/trentm/.npm/_logs/2024-10-10T22_06_05_024Z-debug-0.log

Using --legacy-peer-deps (on the CLI or in .npmrc) isn't a satisfatory answer.
options

Only ever use knip via npx knip ....  This installs it separately... somewhere, I'm not exactly sure where. I gather it is outside of the git clone tree, which means it perhaps is not acceptable for developers; it wouild be fine in CI.   I think it will prompt if it is okay to install knip. That will be a WTF moment for someone running npm run lint.

Make an esbuild-bundled version of knip and commit that? Or publish it as @trentm/knip or whatever and use that. That isn't very satisfying.

Use pnpm on the (naive) assumption it can help here.

Avoid knip, at least until we've upgrade to typescript@5. (I had tried depcheck earlier and didn't get satisfactory results. I think it missed some things.)

I can't think of other options.

(Aside: interesting earlier issue related to @cucumber/messages. I'm not sure if having added the @cucumber/messages dep would have solved that older issue.)

After reading more about how npx works (https://docs.npmjs.com/cli/v8/commands/npm-exec) I'm more confident that running knip via npx (to workaround the install-tree issue) is fine for dev and for CI.
This is Option (1.) from above.
@open-telemetry/javascript-approvers This PR adds this script to package.json:
"lint:deps": "npx --yes<EMAIL_ADDRESS>--dependencies --production --tags=-knipignore",

I think the usage of npx ... here for running knip -- rather than npm install --save-deving it -- is worth an explanation, so that a future maintainer doesn't naively just npm install it.  Do you think this explanation should live in a .md file in the repo somewhere? If so, where? I could start a new DEVELOPMENT.md or a doc/contributing/something.md (similar to what is in the core repo). Or we could just leave it to this PR (I'd update the PR description with notes).  Thoughts?

I think the usage of npx ... here for running knip -- rather than npm install --save-deving it -- is worth an explanation, so that a future maintainer doesn't naively just npm install it. Do you think this explanation should live in a .md file in the repo somewhere? If so, where? I could start a new DEVELOPMENT.md or a doc/contributing/something.md (similar to what is in the core repo). Or we could just leave it to this PR (I'd update the PR description with notes). Thoughts?

I think that if we can make it a dev dependency once we upgrade to typescript 5, then it's scoped in time (hopefully short time) and we can skip the documentation. If you think it's important, I would consider maybe having it in the knip configuration file? hoping one will take a look there before making any changes to knip setup
I am also ok with DEVELOPMENT.md if that makes most sense to you

Ugh, perhaps the following is fine, but it is annoying.

Actually I think this is unacceptable. The result is a MUCH larger working copy:
Before:
$ npm ci
$ du -sh .
1.2G	.

After (i.e. this PR branch):
% du -sh .
4.9G	.

Ew.

Do you think it makes sense to add a CI check for that? that will just run npm ci and make sure a PR will not blow up the installation size?
I don't have time to work on it myself at the moment, just sharing thoughts and potential alternatives for someone to pick up or not :)

Do you think it makes sense to add a CI check for that? that will just run npm ci and make sure a PR will not blow up the installation size?

Perhaps, yes. :)  If we had some infra for collecting dev stats per commit, that would be nice. We'd either have to set a static size threshold or have it only check in CI and compare against previous values.   Also something I won't have time for right now.

I think that if we can make it a dev dependency once we upgrade to typescript 5, then it's scoped in time (hopefully short time) and we can skip the documentation.

I added a note to follow-up in https://github.com/open-telemetry/opentelemetry-js/issues/4870#issuecomment-2433488407

This uses knip (https://knip.dev) to check dependencies, in hopes of avoiding
issues like #2473. This adds 'npm run lint:deps' and adds that to the
existing 'npm run lint'.
This change includes fixes for a handful of unused and unlisted deps.
For now knip is configured to only check 'production' deps. Checking non-prod
deps results in way too many false positives.
Note that knip is being run via npx rather than installing as a devDep because
there is a conflict: knip deps on typescript@5 and all packaegs in this repo
currently use<EMAIL_ADDRESS>Installing knip at the top-level results in a
ballooning of the node_modules install size as<EMAIL_ADDRESS>is installed
40+ types in all separate workspaces.


Use MIToS v3

Hi @kool7d
This is a PR to make BioMakie support MIToS v3. I was not able to test the changes, as BioMakie fails to load on my computer due to an error with its dependencies, in particular with AdaptStaticArraysCoreExt. I hope this helps anyway :) Please let me know if you need further help with the migration.
Cheers

Yeah I am getting a little annoyed by that dependency error. I am having a hard time with it.

Ok, the problem was the Meshes compat. Fixed...


Remove --always-copy from default options

--always-copy breaks on RHEL boxes. And also this shouldn't be enabled by default because old packages don't require them.

Travis isn't running again. I'll merge.


Publishing Azure DevOps extension to the marketplace

Closing this for now. I think I need to start over, as my marketplace schema is corrupt and won't upload anymore.


refactor(coral): improve MSW response mocking

The mocked API payload shape should defined closer where it is relevant. If we would use the scenario model, we would have ever growing list of hard coded mocks whereas we could have it more "dynamic".

I like this approach!
I'm going back and fourth between different thoughts about the separation since a while (written and deleted a lot of text here already 😅 ) and I think I landed on this one:

I would not pass responses as data, but have a response property required, that defaults to 200.
That way, we can use it in the tests same way as in your PR, but we can do the same while prototyping.
We create the response scenarios (e.g. single-page etc) all in one file, with very distinct naming of topics etc and as small as possible. That is not ideal for prototyping, because the responses won't look a lot like in the real world (e.g.  have different title length etc), but our prio use-case is testing anyway.

I feel that way we have a stronger connection between the "real" api call and the tests for it. We avoid getting to a point where for some reason data may diverge.
WDYT?

I would not pass responses as data, but have a response property required, that defaults to 200.

Could you elaborate a bit. If you could provide a dummy function definition with types, that would be awesome. I think the best way to provide the mock would be a Response object, that but is not yet supported [REF]. In this PR I did not yet deprecate the scenario property and make response as mandatory, but that would follow soon after if we take this route.

That way, we can use it in the tests same way as in your PR, but we can do the same while prototyping.

Agreed!

We create the response scenarios (e.g. single-page etc) all in one file, with very distinct naming of topics etc and as small as possible. That is not ideal for prototyping, because the responses won't look a lot like in the real world (e.g. have different title length etc), but our prio use-case is testing anyway.

If we choose to go with response definitions that are not defined right next to the component or unittest, we could look into extracting those from OpenAPI examples. We just need to check how feasible it would be to parse YAML.

Could you elaborate a bit.

Basically what you did, but provide a default for  response argument.
Something like

type ResponseObject = {
    status: HTTPStatus,
    payload: JSON
}

const successResponse = {
    status: 200,
    json: {
        ...createMockEnvironmentDTO("DEV"),
        ....createMockEnvironmentDTO("TST")
}

function mockGetEnvironment( {mswInstance, response = defaultResponse } : { mswInstance: MswInstance, response?: ResponseObject } {
    return res(
        ctx.status(response.status),
        ctx.json(response.json)
        )
      );
}

The, in the component, we would use
mockGetEnvironment({ mswInstance: browserEnvWorker })
and in tests we could use
import { mockedErrorResponseEnvironment } from "src/some/path/test-mocks-environment-api"

mockGetEnvironment({ mswInstance: browserEnvWorker,  response: mockedErrorResponseEnvironment })
```

> If we choose to go with response definitions that are not defined right next to the component or unittest (...)
Yeah I think they should colocate with the api mocks, makes the most sense.

I'm a bit conflicted about this in general now, because we're generating a whole lot of code to basically test the api in all our unit-tests. It starts to feel this is almost more testing that our msw mocks are set up correctly 🤔  Also, wether we have the responses autogenerated somehow or we do that manually, if something in the API changes (that could be something small like a property not being optional anymore), it will be a lot of maintenance of tests to keep everything up to date, since these responses will affect a lot of tests.

Maybe we could also think about a complete different approach - go the direction Lorna mentioned (prism), use that for integration tests and mock the api out in our unit-like tests. That would make sure we have the API site tested, but without that tight coupling between a lot of smallish to big jest tests relying on the API mocks.

Lets not yet implement those fixtures as either own json files or parsed from openapi.yaml. We need to make sure, that TS compiler is able to validate the response data.I suggest that we put this is as is and then continue the refactoring so that we can later drop the scenario argument and replace that with the response with a nice default as you described above.

So add this PR as is and in following PRs add the same approach + defaults to the rest?
WDYT about the idea to separating jest / integration tests for us and have a fully mocked response (prism) for integration tests. That way we can make sure everything with the API is fine but have less load in the Jest tests? (it's also something we can revisit later, does not need to be a decision right now)

WDYT about the idea to separating jest / integration tests for us and have a fully mocked response (prism) for integration tests.

I agree on separation and the need, but I do not think we need them yet. At the moment I have strong confidence from TS type system and component unittests. I would revisit the need after we have completed the first end2end feature.


Font styles are not applied

I have tested this with the preview from PR #6891 as that makes the PDF render properly.
Opening the file below with the preview at http://<IP_ADDRESS>:8877/20a6d5dd16fdaa6/web/viewer.html makes all characters display properly, but the font styles (bold and italic) are not applied. Adobe Acrobat Reader DC has no problem with this. The PDF is generated by Scribus 1.5.0svn.
Below is a reduced test case for inclusion in a PR that fixes this issue. It might be possible to reduce it even further.
reduced-font-styles.pdf

This could possibly be related to the remaining issue (i.e. the slightly incorrect glyph rendering) observed in bug 1132849, since the bold/italic styles in this issue were correct prior to commit https://github.com/mozilla/pdf.js/commit/b5b94a4af389ef387570cf22662d9fc6bd41417a (i.e. PR #4259). See also https://github.com/mozilla/pdf.js/pull/6090#issuecomment-111081768.
@brendandahl Any ideas here?


added miniproject

Metric
Value

alert_status
:white_check_mark: OK

bugs
1

code_smells
55

reliability_rating
C (at least 1 Major Bug)Note: A being best, E being worst

security_rating
A (0 Vulnerabilities)Note: A being best, E being worst

vulnerabilities
0


fix: fixing broken backward hooks change

Description
This PR fixes a bug in v2.x.x of TransformerLens where backwards hooks were changed to use the deprecated and broken register_backward_hook() method instead of the correct register_full_backward_hook(). Before v2.x.x the correct method was used. The breaking change happened in #607, but it's not clear why this change was made.
This PR also adds a test which will not allow this regression to happen again.
Fixes #672
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] I have not rewritten tests relating to key interfaces which would affect backward compatibility

Thanks for getting this in! The PR in question was done over a long period of time, and I think the wrong line was grabbed during a merge conflict. I will get this into PIP within the next hour.


Amcrest version bump to 1.1.0

Description:
Amcrest version bump to 1.1.0
Checklist:
If the code communicates with devices, web services, or third-party tools:

[x] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[x] New dependencies have been added to the REQUIREMENTS variable ([example][ex-requir]).
[x] New dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.

Closing this PR in regards of #5310


Announcement: new moderation features

Not all features are of any interest for regular users, but some of them would be worth to be announced, so users know that some specific issue can be addressed in general:

[...]

Let mods hide solution group from the list (audited)
Let mods invalidate solution group without verifying (audited)
Let mods recalculate buggy beta votes (audited)
Let mods unpublish beta kata (audited)
Let mods remove rank assessments and votes (audited)

[...]

inform users that some issues are now possible to be handled
ask users to report problematic instances, so mods could handle them
say "thanks" to maintainers who did the hard work of implementing these

Additional point: link to docs on moderation:

https://docs.codewars.com/community/moderation/
the result of codewars/docs#250

Additional point: link to docs on moderation:

https://docs.codewars.com/community/moderation/
the result of codewars/docs#250

My initial idea behind this post was that #18 could be published before https://github.com/codewars/docs/issues/250 gets realized, leaving community without details. But now, since #18 has not been published yet, and it already links to the result of codewars/docs#251, separate post with the announcement of specific features might be not necessary anymore (until new ones will be provided, that is).

My initial idea behind this post was that #18 could be published before https://github.com/codewars/docs/issues/250 gets realized, leaving community without details. But now, since #18 has not been published yet, and it already links to the result of codewars/docs#251, separate post with the announcement of specific features might be not necessary anymore (until new ones will be provided, that is).


Fixed Siemens private tag parsing for non UN VRs

Ended up doing pretty much what you suggested.

Still missing. I wasn't sure whether I should start trying to integrate Pooch or not.
Might be best to keep this PR simple and only add one of your files to the tests module for now. Moving everything to Pooch should probably be done in a different PR.


ListViewAnimations Setup inside of Android Studio

I'm trying to setup "nhaarman/ListViewAnimations" inside of Android Studio 1.1.0.
From Setup:
Add the following to your build.gradle:
repositories {
    mavenCentral()
}

dependencies {
    compile 'com.nhaarman.listviewanimations:lib-core:3.1.0@aar'
    compile 'com.nhaarman.listviewanimations:lib-manipulation:3.1.0@aar'
    compile 'com.nhaarman.listviewanimations:lib-core-slh:3.1.0@aar'
}

yet Android Studio says:
Gradle project sync failed. Basic functionality (e.g. editing, debugging) will not work properly.
Error:(26, 0) Gradle DSL method not found: 'compile()'
Possible causes:

The project 'X' may be using a version of Gradle that does not contain the method.
Gradle settings

The build file may be missing a Gradle plugin.
Apply Gradle plugin

This looks more like an issue in your project than in this library.

It could very well be. I created brand new project and added those lines to build.gradle.
I'm using Android Studio 1.1 with Gradle 2.2.1

after adding those lines to:
build.gradle (Module: app)
instead of
build.gradle (Project: app)
no errors)


ImageOptim 1.6 changes

Heads up: pngquant bundled with ImageOptim is going to move from ImageOptim.app/Contents/MacOS/pngquant to ImageOptim.app/Contents/Frameworks/ImageOptimGPL.framework/Resources/pngquant. I could add a symlink in the old location, but I see you're using your own version anyway.
The new embedded framework finally makes it possible to run optimizations without the main ImageOptim UI!
So far I've only used it for Apple Mail extension, but of course it could be used to improve CLI support. Let me know if you have any wishes for the interface.

Appreciate this info, thanks @pornel.


Jupyter server crashed. Unable to connect. Cannot assign requested address

Environment data

VS Code version:

Version: 1.36.1
Commit: 2213894ea0415ee8c85c5eea0d0ff81ecc191529
Date: 2019-07-22T07:38:21.043Z
Electron: 4.2.8
Chrome: 69.0.3497.128
Node.js: 10.11.0
V8: 6.9.427.31-electron.0
OS: Linux x64 4.19.61-1-lts

Extension version (available under the Extensions sidebar): 2019.6.24221 (9 July 2019)
OS and version: Archlinux
Python version (& distribution if applicable, e.g. Anaconda): py3.7
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977):

User belongs to experiment group 'ShowPlayIcon - runFile'
Starting Jedi Python language engine.
##########Linting Output - flake8##########

Steps
In a python file, having:
#%%
print("hi")

CTRL+enter to run
I'm not sure, but the issue may be that the jupter server wants to bind to something other than <IP_ADDRESS>, which it would fail. I do have a global option set in ~/.jupyter/jupyter_notebook_config.py: c.NotebookApp.ip = '<IP_ADDRESS>', and w/o it (or without setting --ip, the jupyter notebook would fail on my machine. It may be the case that this estension is not reading that config file, but not sure.
Expected behaviour
CTRL+enter should  start the jupyter server, and run the cell.
Actual behaviour
A notification shows: Jupyter server crashed. Unable to connect. Error code from jupyter: 1.
Logs
Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)
[Extension Host] Info Python Extension: 2019-07-30 12:48:14: Cached data exists getEnvironmentVariables, <No Resource>
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:14: Getting jupyter server options ...
webviewElement.ts:363 [Embedded Page] %cElectron Security Warning (Insecure Content-Security-Policy)
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Request for onigasm file at /home/adrin/.vscode-oss/extensions/ms-python.python-2019.6.24221/out/client/node_modules/onigasm/lib/onigasm.wasm
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Searching for token colors ...
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Attempting search for colors ...
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Connecting to jupyter server ...
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Searching for token colors ...
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Attempting search for colors ...
2console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Loading colors from /usr/lib/code/extensions/theme-defaults/themes/dark_plus.json ...
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Connecting to history server
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Cached data exists getEnvironmentVariables, <No Resource>
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Launching history server
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Generating custom default config at /tmp/0beef754-8aaa-420c-8620-27143cf83436/jupyter_notebook_config.py
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Attempting search for colors ...
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Loading base colors from /usr/lib/code/extensions/theme-defaults/themes/dark_plus.json ...
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Attempting search for colors ...
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Loading base colors from /usr/lib/code/extensions/theme-defaults/themes/dark_plus.json ...
2console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Using colors to generate CSS ...
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:17: Cached data exists getEnvironmentVariables, <No Resource>
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:19: Request for tmlanguage file.
webviewElement.ts:363 [Embedded Page] Could not create web worker(s). Falling back to loading web worker code in main thread, which might cause UI freezes. Please see https://github.com/Microsoft/monaco-editor#faq
webviewElement.ts:363 [Embedded Page] Failed to construct 'Worker': Script at 'vscode-resource:/home/adrin/.vscode-oss/extensions/ms-python.python-2019.6.24221/out/datascience-ui/history-react/editor.worker.js' cannot be accessed from origin 'null'.
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:20: Traceback (most recent call last):  File "/home/adrin/miniconda3/envs/sklearn/bin/jupyter-notebook", line 10, in <module>
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:20:     sys.exit(main())  File "/home/adrin/miniconda3/envs/sklearn/lib/python3.7/site-packages/jupyter_core/application.py", line 267, in launch_instance    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)  File "/home/adrin/miniconda3/envs/sklearn/lib/python3.7/site-packages/traitlets/config/application.py", line 657, in launch_instance    app.initialize(argv)  File "</home/adrin/miniconda3/envs/sklearn/lib/python3.7/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:20:   File "/home/adrin/miniconda3/envs/sklearn/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error    return method(app, *args, **kwargs)  File "/home/adrin/miniconda3/envs/sklearn/lib/python3.7/site-packages/notebook/notebookapp.py", line 1679, in initialize
console.ts:134 [Extension Host] Info Python Extension: 2019-07-30 12:48:20:     self.init_webapp()  File "/home/adrin/miniconda3/envs/sklearn/lib/python3.7/site-packages/notebook/notebookapp.py", line 1442, in init_webapp    self.http_server.listen(port, self.ip)  File "/home/adrin/miniconda3/envs/sklearn/lib/python3.7/site-packages/tornado/tcpserver.py", line 151, in listen    sockets = bind_sockets(port, address=address)  File "/home/adrin/miniconda3/envs/sklearn/lib/python3.7/site-packages/tornado/netutil.py", line 174, in bind_sockets    sock.bind(sockaddr)OSError: [Errno 99] Cannot assign requested address
notificationsAlerts.ts:40 Jupyter server crashed. Unable to connect.   Unable to connect.

We do force our own config file, but you can disable this with this setting:

I think it's because user processes cannot bind to localhost, or the external interface, which I think is a very sane option for personal PCs and laptops. I don't want my jupyter to be exposed in the network anyway. It may have something to do with the default settings in archlinux, jupyter has never worked for me without this option.

Does it work if you turn off 'Use default config for Jupyter'?

Yes it does :)

I think I'm going to close this then. Your jupyter is setup to require the ip setting. Not sure we can detect that's the case, other than to show the error that comes up.
Maybe we could try it without the config setting if we hit this scenario though.

You could also expose the jupyter settings to the user or let the user set them in the json file and also document it.

This failed for us as well (suddenly on the july update), as we also have jupyter running on 8888 on a separate process. Using default configs are good for other cases, but it would be great to be able to override them in settings, as @adrinjalali suggests.

There is a setting already. The one I mentioned above:
https://user-images.githubusercontent.com/19672699/62153040-9a105900-b2b8-11e9-8e4e-0b1f5b1a5f26.png
If you turn that setting on, your global jupyter config will still apply

yes, but as I have jupyter lab running in a separate process vscode seems to fail to launch its own jupyter. When I tick this off it works, but then I need to enter the jupyter psw in vscode to run. Not critical, but a bit annoying. :)

If it is any consolation, the interactive windows worked for me when I changed the setting "Python>Testing: Debug Port" to 8888.

Same issue with Win10 Python 3.8.0
Jupyter server crashed. Unable to connect.  Error code from jupyter: 1

@Karlheinzniebuhr that's unlikely to be the same root cause. Jupyter is just plain crashing with Python 3.8 (at least in the other cases we've seen).
This bug is really about documenting and exposing what options we pass on the command line to jupyter when starting up.

Same symptoms with Windows 10 Python 3.8.1

I have the same problem! after to install pytorch GPU, any one find a solution?
run Jupyter outside to VScode it work but with VSCODE it doens't run

Why does the jupyter server fail without c.NotebookApp.ip = '<IP_ADDRESS>'?

Here is why and an easy fix. https://github.com/ipython/ipython/issues/6193#issuecomment-466082026
And, somehow on my computer, the jupyter notebook started by VS Code's python extension does not read my configuration file ~/.jupyter/jupyter_notebook_config.py (especially the c.NotebookApp.ip settiing), although I have turned off "turn off 'Use default config for Jupyter' "  in VS Code.

I was running into the same problem with an environment. I was able to resolve by setting the python.condaPath setting to the full path to conda.exe in my user settings. I have miniconda installed in c:\users[username]\miniconda3
ctrl->shift->P Prefrences: Open Settings (JSON)
"python.condaPath": "C:\Users[username]\miniconda3\Scripts\conda.exe"
This resolved the issue with opening jupyter in vscode and it now activates my virtual environment when opening a python terminal.

This worked for me. Thanks a lot, you saved me.

I got same error as @Karlheinzniebuhr, but fixed after restart VSCode.


test default due fails with assertionerror

Hi,
In the last update in Archlinux (updating from 2.0.2-1 to 2.1.0-1) I got the following failure while performing the test test_default_due2
________________________________________________ test_default_due2 _________________________________________________

tmpdir = local('/tmp/pytest-of-XXXXXX/pytest-9/test_default_due20')
runner = <click.testing.CliRunner object at 0x7f9b9d3f4f60>
create = <function create.<locals>.inner at 0x7f9ba0de31e0>
default_database = <todoman.model.Database object at 0x7f9b9d3f4b00>

    @freeze_time(datetime.datetime.now())
    def test_default_due2(tmpdir, runner, create, default_database):
        cfg = tmpdir.join('config')
        cfg.write('default_due = 24\n', 'a')

        r = runner.invoke(cli, ['new', '-ldefault', '-dtomorrow', 'aaa'])
>       assert not r.exception
E       assert not InvalidArgsError(<class 'icu.SimpleDateFormat'>, 'format', (FakeDatetime(2003, 10, 30, 11, 45),))
E        +  where InvalidArgsError(<class 'icu.SimpleDateFormat'>, 'format', (FakeDatetime(2003, 10, 30, 11, 45),)) = <Result InvalidArgsError(<class 'icu.SimpleDateFormat'>, 'format', (FakeDatetime(2003, 10, 30, 11, 45),))>.exception

tests/test_basic.py:225: AssertionError
================================== 1 failed, 45 passed, 3 xfailed in 2.71 seconds ==================================

I'm not sure what's going wrong, maybe this can be helpful for you.

I can't reproduce this failure when installing todoman from AUR.

Nor can I (I also use arch and actually maintain the AUR package). Can you provide the output of pip freeze. Is this the output of makepkg, or some helper? Are you using a virtualenv?

Thanks for the fast reply... I'm installig todoman with pacaur so yes, it should be the makepkg output and no, I'm not using any virtualenv.
Just tried to download the PKGBUILD and makepkg give me the same failure.
Here I append the output of pip freeze:
alabaster==0.7.9
ansi==0.1.3
appdirs==1.4.0
arrow==0.10.0
atomicwrites==1.1.5
Babel==2.3.4
Beaker==1.8.1
beautifulsoup4==4.5.3
cffi==1.9.1
chardet==2.3.0
click==6.7
click-log==0.1.8
click-repl==0.1.1
click-threading==0.4.3
colorlog==2.10.0
colour==0.1.2
configobj==5.0.6
cryptography==1.7.2
cycler==0.10.0
Cython==0.25.2
decorator==4.0.11
docopt==0.6.2
docutils==0.13.1
easygui===0.98.0-UNRELEASED
file-magic==0.3.0
flake8==3.3.0
freezegun==0.3.8
future==0.16.0
greenlet==0.4.12
httplib2==0.10.3
hypothesis==3.6.1
icalendar==3.11.3
idna==2.2
imagesize==0.7.1
Jinja2==2.9.5
jsonschema==2.6.0
keyutils==0.4
khal==0.9.3.dev24+ngaa986b4
khard==0.11.4
lensfun==0.3.2
louis==3.0.0
lxml==3.7.3
Mako==1.0.6
Markdown==2.6.8
Markups==2.0.0
MarkupSafe==0.23
mccabe==0.6.1
mistune==0.7.3
msgpack-python==0.4.8
nose==1.3.7
notebook==4.3.1
oauthlib==2.0.1
olefile==0.43
opendht==1.2.1
packaging==16.8
pafy==<IP_ADDRESS>
parsedatetime==2.2
pew==0.1.26
pexpect==4.2.1
pickleshare==0.7.4
Pillow==4.0.0
pkginfo==1.4.1
ply==3.10
prompt-toolkit==1.0.13
psutil==5.1.3
ptyprocess==0.5.1
py==1.4.32
pyasn1==0.2.2
pycodestyle==2.3.1
pycparser==2.17
pycrypto==2.6.1
pyenchant==1.6.8
pyflakes==1.5.0
Pygments==2.2.0
pygobject==3.22.0
PyICU==1.9.5
pymediainfo==2.1.5
pyparsing==2.1.10
PyPDF2==1.26.0
PyPrind==2.9.8
pyshp==1.2.10
pytest==3.0.6
pytest-runner==2.11.1
python-dateutil==2.6.0
python-distutils-extra==2.39
python-libtorrent==1.1.1
python-sane==2.8.3
pythonz-bd==1.10.2
pytz==2016.10
pyxdg==0.25
PyYAML==3.12
pyzmq==16.0.2
requests==2.13.0
requests-oauthlib==0.8.0
requests-toolbelt==0.7.1
resumable-urlretrieve==0.1.5
setuptools-scm==1.15.1.dev17+ngb0d17f1
simplegeneric==0.8.1
six==1.10.0
snowballstemmer==1.2.1
sortedcontainers==1.5.7
Sphinx==1.5.2
sphinx-rtd-theme==0.1.9
sphinxcontrib-newsfeed==0.1.4
svgwrite==1.1.10
team==1.0
terminado==0.6
todoman==2.0.2
tornado==4.4.2
traitlets==4.3.1
tzlocal==1.3
urllib3==1.20
urlscan==0.8.3
urwid==1.3.1
vdirsyncer==0.14.1
virtualenv==15.1.0
virtualenv-clone==0.2.6
vobject==<IP_ADDRESS>
wcwidth==0.1.7
widgetsnbextension==1.2.6
yapf==0.16.0

I don't have PyICU installed and I wouldn't know how todoman would depend on it, but it somehow appears in the output of check()?

Indeed I have python-pyicu installed which is required by python-vobject
which is required by khard, I don't know how this affects todoman.
Tomorrow I'll try to disintall it temporarily.

Actually that was the problem. In fact I:

removed khard and its python-pyicu deps
updated todoman with no issues
reinstalled khard and its deps

This worked fine but I don't know if any bug is hidden somewhere. If I'll experience some problem I'll get back.
Thanks for the advice.

I could reproduce this by installing pyicu. That's pretty nasty.

Looks like parsedatetime uses pyicu only if avaiable:
https://github.com/bear/parsedatetime/blob/master/parsedatetime/pdt_locales/__init__.py#L10
https://github.com/bear/parsedatetime/blob/master/parsedatetime/pdt_locales/icu.py#L16
The problem is, it crashes, while not using it does not.

This seems a bit crazy...
But there seems to be no side effects on the execution... I haven't tested extensively though.

I might fail when trying to create events with something like todo new --due tomorrow, but I'm not 100% certain.

Just tryed: todo new --due tomorrow buy bread worked fine

Cool. So we just need to fix the test, probably (or rather, see why it crashes). I'll keep this open until we figure out the crash.

For anyone wanting to look into this: I'd add a tox environment, that runs the entire test suite with pyicu as an extra requirement, so we're sure we never regress on this again.

This is an upstream issue, but I'm not sure if it's freezegun or parsedatetime that's failing here. Here's a minimal example reproducing the crash:
import parsedatetime
from freezegun import freeze_time

with freeze_time('2017-03-17 20:22:19'):
    parsedatetime.Calendar()

I've reported this upstream here.


Feedback

@JLockleyASC it looks like you are just missing part of the final step:

[x] ~Add a page heading by creating an <h1> element inside of the <body> element~
[x] ~Add a sentence about what you are most excited about in this course by creating a <p> element below the <h1> element~
[ ] Enclose the <h1> and <p> elements by wrapping them inside a <div> element and add a class named "content-container" to the <div> element with the class attribute

You have added class="content-container" to your <h1> and <p> elements instead of creating a parent <div> element with that class. Once you make that change, you should be finished 🙂
Let me know if you have any questions about this!

Wow just saw this I’ll look over it and fix it thank you

Hey I’m having trouble pushing my changed file to GitHub . Could u help ?
How do I get my terminal to pull up my file ? And set it as the repo.


Cookbooks >> Reference Architectures gets me 404 error

[Enter feedback here]
When I select Cookbooks >> Reference Architectures, I get
404-Page not found

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c13f5f59-c119-40e6-1396-d6e65b737693
Version Independent ID: eb8a9ad0-44fc-6e06-adb0-29df32aee19a
Content: Create a Java web app on Linux - Azure App Service
Content Source: articles/app-service/containers/quickstart-java.md
Service: app-service-web
GitHub Login: @msangapu
Microsoft Alias: msangapu

@kumarallamraju Thanks for the feedback!  I have created pull request it will be visible in some time.

This has been fixed. Thank you for the feedback, @kumarallamraju.
#please-close


Restrict matplotlib

matplotlib>=3.5 with pygimli=1.2.1 results in a TypeError: int() argument must be a string, a bytes-like object or a number, not 'slice' for the command ax.plot(pg.x(data1), pg.y(data2), "ro");. Restricting therefore matplotlib.
I also changed empymod to be installed with conda, as it is on conda-forge. This also allows to remove the pip.

Thanks Dieter! Much appreciated.

Just tested @halbmy, and yes indeed. Opened therefore https://github.com/gimli-org/transform2022/pull/1.
There is also this warning
MatplotlibDeprecationWarning: The seaborn styles shipped by Matplotlib are deprecated since 3.6, as they no longer correspond to the styles shipped by seaborn. However, they will remain available as 'seaborn-v0_8-<style>'. Alternatively, directly use the seaborn API instead.
  plt.style.use("seaborn-notebook")

which is "fixed" by setting "matplotlib<3.5".


GraalVM EE actually has issues with some mods.

I can confirm Immersive Portals has issues with EE that don't present on CE. I have also seen people say other mods that touch the technicals of Minecraft are not well compatible. This is something that should be pointed out, after more information is acquired.

Ooh, in the same configuration CE results in unbearable log spam with GL errors. Oofie.

Ooh, in the same configuration CE results in unbearable log spam with GL errors. Oofie.

Oh interesting. CE is basicaly OpenJDK, so I'm surprised it has any issues.
The immersive portals issue still happen with no arguments?
I will test immersive, but can you post a log for the GL thing?

@HyperSoop you can fix this for now by rolling back to GraalVM  22.1.0 and removing the line -Dgraal.EarlyGVN=true -Dgraal.StripMineCountedLoops=true

Ooh, in the same configuration CE results in unbearable log spam with GL errors. Oofie.

Oh interesting. CE is basicaly OpenJDK, so I'm surprised it has any issues.
The immersive portals issue still happen with no arguments?
I will test immersive, but can you post a log for the GL thing?

The log i'll send now, and also i'll try without flags in a bit.
[20:30:20] [Render thread/INFO]: OpenGL debug message: id=1282, source=API, type=ERROR, severity=HIGH, message='GL_INVALID_OPERATION error generated. The source and destination internal formats are not compatible.' - like 20 of these per second.

I see you have already figured out the problem with EE, i'll now also try to reproduce the CE log spam.
(Yes, i got it. It's only happening with the flags)

I see you have already figured out the problem with EE.

Not exactly :(. I know its some kind of entperprise compiler regression between 22.2.0 and, 22.1.0, but I would prefer to fix it in 22.2.0 without rolling back or disabling the whole enterprise compiler.
And yeah, I ran CE with the OpenJDK 17 flags, and I'm not getting that GL error... but I have seen that error before, when running the Iris mod.
Hence I'd like to see your whole log if possible, to know your hardware, environment and so on. But if I can't replicate it, you may have to take half the flags out at a time to narrow down which mod is bugging immportals out.

I actually have iris on that instance, so it's just poor compatibility with immptl and iris which is somewhat known.

I actually have iris on that instance, so it's just poor compatibility with immptl and iris which is somewhat known.

Oh actually I get that as a standalone Iris issue on my RTX 2060, even with shaders disabled.
Its not a GraalVM-specific issue either, I don't think, but I will double check that. I've been meaning to report this as well, I will ping you if I do.

Oh I've confirmed this breaks Iris too, you were right.
I've updated the args to remove the offending flag.

So to summarize, one bug I thought was an immersive portals bug was actually a vanilla crash, the other (the VectorizeSIMD flag) only seems to break Iris.
You have any more details about what's breaking immersive portals?

You have any more details about what's breaking immersive portals?

I don't know. My guess is immersive portals makes big changes to the area of rendering code affected, and that's why it makes the issue so noticable.

But whats the issue? I tried the dimension stack feature and building a portal in Graal, and both seem to work/look fine.

You have any more details about what's breaking immersive portals?

I don't know. My guess is immersive portals makes big changes to the area of rendering code affected, and that's why it makes the issue so noticable.

But whats the issue? I tried the dimension stack feature and building a portal in Graal EE, and both seem to work/look fine.

Maybe it was related to your arguments. I think we ahve already decided what the problem is...?

I was mistaken before. I thought I was running into a startup crash with immersive portals, but turns out that was a vanilla issue.

Astral Sorcery does not display constellations in the book and in the telescope when the JDK from GraalVM is installed. Happened to me and my friend on Enigmatica 6 Expert modpack. Or displays for a while and then disappears.

I tried removing all the flags and some mods, but the only thing that helped was changing java to adoptium

Astral Sorcery does not display constellations in the book and in the telescope when the JDK from GraalVM is installed. Happened to me and my friend on Enigmatica 6 Expert modpack. Or displays for a while and then disappears.

I will try to reproduce this. But if you wish, you can disable these flags like so, as most are enabled by default: -Dgraal.UsePriorityInlining=false -Dgraal.Vectorization=false -Dgraal.OptDuplication=false -Dgraal.DetectInvertedLoopsAsCounted=false -Dgraal.LoopInversion=false -Dgraal.VectorizeHashes=false -Dgraal.EnterprisePartialUnroll=false -Dgraal.VectorizeSIMD=false -Dgraal.StripMineNonCountedLoops=false -Dgraal.SpeculativeGuardMovement=false -Dgraal.InfeasiblePathCorrelation=false -Dgraal.LoopRotation=false

Oh, and I have rolled back to GraalVM 22.1.0 EE myself. 22.2 seems like a funky release.

The book and telescope work for me with the following block: -server -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -Dsun.rmi.dgc.server.gcInterval=2147483646 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 -XX:+EnableJVMCI -XX:+UseJVMCICompiler -XX:+EagerJVMCI -XX:+UseStringDeduplication -XX:+UseFastUnorderedTimeStamps -XX:+UseAES -XX:+UseAESIntrinsics -XX:AllocatePrefetchStyle=1 -XX:+UseLoopPredicate -XX:+RangeCheckElimination -XX:+EliminateLocks -XX:+DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:+UseFastJNIAccessors -XX:+OptimizeStringConcat -XX:+UseCompressedOops -XX:+UseThreadPriorities -XX:+OmitStackTraceInFastThrow -XX:+TrustFinalNonStaticFields -XX:ThreadPriorityPolicy=1 -XX:+UseInlineCaches -XX:+RewriteBytecodes -XX:+RewriteFrequentPairs -XX:+UseNUMA -XX:-DontCompileHugeMethods -XX:+UseFPUForSpilling -XX:+UseXMMForArrayCopy -Djdk.nio.maxCachedBufferSize=262144 -Dgraal.TuneInlinerExploration=1 -Dgraal.CompilerConfiguration=enterprise -Dgraal.UsePriorityInlining=false -Dgraal.Vectorization=false -Dgraal.OptDuplication=false -Dgraal.DetectInvertedLoopsAsCounted=false -Dgraal.LoopInversion=false -Dgraal.VectorizeHashes=false -Dgraal.EnterprisePartialUnroll=false -Dgraal.VectorizeSIMD=false -Dgraal.StripMineNonCountedLoops=false -Dgraal.SpeculativeGuardMovement=false -Dgraal.InfeasiblePathCorrelation=false -XX:+UseLargePages -XX:LargePageSizeInBytes=2m
Gonna try to enable some of the graal options to narrow it down.

@mllightitup
I jumped the gun in my previous post, as I can't get this to actually happen in my E6E instance.
What version of GraalVM are yall using? And how long does it usually take to happen?

I tried in java 17 graalvm ee 22.2.0
If you run an enigmatic server for example, there are flags in the default settings to make everything work on java 16+. And as I understood from the official discord, everything should work fine on java 16+ on the client too

I just now noticed the first sentence. Maybe this is just one of those bugs.)

Can you try to replicate this on java 17 graalvm 22.2.0? Maybe the problem is with the java 17 from graal

I tried on java 17 graalvm ee 22.2.0 If you run an enigmatic server for example, there are flags in the default settings to make everything work on java 16+. And as I understood from the official discord, everything should work fine on java 16+ on the client too

It just errors out for me when I try to run Java 17, with a message saying I have to run Java 8, so I havent tried 17 yet.
Does this bug happen in Adoptium 17 too?

If you use multimc or polymc there is a checkbox "skip java compatibility check" or something like that

If the checkbox is active it allows you to use java 17

Does this bug happen in Adoptium 17 too?

Nope. Adoptium 17 fixes this bug with astral sorcery

Does this bug happen in Adoptium 17 too?

Nope. Adoptium 17 fixes this bug with astral sorcery

I have confirmed this. It even happens in GraalVM Community Edition!
And it doesn't happen in Graal EE for Java 8.
This is a weird one.

I want to document this here for anyone having the same issue - on 1.18.2, TerraBlender seems to break with Java 17 EE 22.2.0,  and for whatever reason, the same is not true with EE 22.1.0
Error:
Failed to load datapacks, can't proceed with server load
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: SubTree needs at least one child

Something is definitely wrong with 22.2.0, at least for Minecraft.

I want to document this here for anyone having the same issue - on 1.18.2, TerraBlender seems to break with Java 17 EE 22.2.0, and for whatever reason, the same is not true with EE 22.1.0
Error:
Failed to load datapacks, can't proceed with server load
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: SubTree needs at least one child

Something is definitely wrong with 22.2.0, at least for Minecraft.

22.2.0 breaks vanilla with the default flags. It is indeed fishy.


Couldn't train rasa core with custom model_architecture in keras policy

I have to use a custom keras policy for dialog management using rasa core. I have tried to implement it using a sequential model in as per documentation in keras. I have trained the model using my custom dataset externally without using rasa. But when I have tried to integrate this architecture in model_architecture method of KerasPolicy of rasa. I have attached the code below,
from __future__ import print_function

from keras.models import Model
from keras.layers import Input, LSTM, Dense
import numpy as np

batch_size = 64  # Batch size for training.
epochs = 100  # Number of epochs to train for.
latent_dim = 256  # Latent dimensionality of the encoding space.
num_samples = 10000  # Number of samples to train on.
# Path to the data txt file on disk.
data_path = 'fra-eng/fra.txt'

class RestaurantPolicy(Policy):
    def model_architecture(self, input_shape, output_shape):
        input_texts = []
        target_texts = []
        flatten_input_texts = []
        flatten_target_texts = []
        input_intent_sv = set()
        target_intent_sv = set()
        with open(data_path, 'r', encoding='utf-8') as f:
             lines = f.read().split('\n')
        for line in lines[: min(num_samples, len(lines) - 1)]:
            input_text, target_text = line.split('\t')
            # We use "tab" as the "start sequence" character
            # for the targets, and "\n" as "end sequence" character.
            target_text = '\t' + target_text + '\n'
           input_texts.append(input_text)
            target_texts.append(target_text)
            for char in input_text:
                if char not in input_characters:
                   input_characters.add(char)
            for char in target_text:
                if char not in target_characters:
                    target_characters.add(char)

        input_intent_sv = sorted(list(input_intent_sv))
        target_intent_sv = sorted(list(target_intent_sv))
        num_encoder_tokens = len(input_intent_sv)
        num_decoder_tokens = len(target_intent_sv)
        max_encoder_seq_length = max([len(txt) for txt in flatten_input_texts])
        max_decoder_seq_length = max([len(txt) for txt in flatten_target_texts])

        print('Number of samples:', len(input_texts))
        print('Number of unique input tokens:', num_encoder_tokens)
        print('Number of unique output tokens:', num_decoder_tokens)
        print('Max sequence length for inputs:', max_encoder_seq_length)
        print('Max sequence length for outputs:', max_decoder_seq_length)

        input_token_index = dict(
            [(intent_sv, i) for i, intent_sv in enumerate(input_intent_sv)])
        target_token_index = dict(
            [(intent_sv, i) for i, intent_sv in enumerate(target_intent_sv)])

        encoder_input_data = np.zeros(
            (len(flatten_input_texts), max_encoder_seq_length, num_encoder_tokens),
            dtype='float32')
        decoder_input_data = np.zeros(
            (len(flatten_input_texts), max_decoder_seq_length, num_decoder_tokens),
            dtype='float32')
        decoder_target_data = np.zeros(
            (len(flatten_input_texts), max_decoder_seq_length, num_decoder_tokens),
            dtype='float32')

        for i, (input_text, target_text) in enumerate(zip(flatten_input_texts, flatten_target_texts)):
            for t, char in enumerate(input_text):
                encoder_input_data[i, t, input_token_index[char]] = 1.
            encoder_input_data[i, t + 1:, input_token_index[' ']] = 1.
            for t, char in enumerate(target_text):
                # decoder_target_data is ahead of decoder_input_data by one timestep
                decoder_input_data[i, t, target_token_index[char]] = 1.
                if t > 0:
                    # decoder_target_data will be ahead by one timestep
                    # and will not include the start character.
                    decoder_target_data[i, t - 1, target_token_index[char]] = 1.
            decoder_input_data[i, t + 1:, target_token_index[' ']] = 1.
            decoder_target_data[i, t:, target_token_index[' ']] = 1.

        # Define an input sequence and process it.
        encoder_inputs = Input(shape=(None, num_encoder_tokens))
        encoder =  LSTM(latent_dim, return_state=True)
        encoder_outputs, state_h, state_c = encoder(encoder_inputs)
        # We discard `encoder_outputs` and only keep the states.
        encoder_states = [state_h, state_c]

        # Set up the decoder, using `encoder_states` as initial state.
        decoder_inputs = Input(shape=(None, num_decoder_tokens))
        # We set up our decoder to return full output sequences,
        # and to return internal states as well. We don't use the
        # return states in the training model, but we will use them in inference.
        decoder = LSTM(latent_dim, return_sequences=True, return_state=True)
        decoder_outputs, _, _ = decoder(decoder_inputs,
                                                 initial_state=encoder_states)
        decoder_dense = Dense(num_decoder_tokens, activation='softmax')
        decoder_outputs = decoder_dense(decoder_outputs)

        # Define the model that will turn
        # `encoder_input_data` & `decoder_input_data` into `decoder_target_data`
        model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

        # Run training
        model.compile(optimizer='rmsprop', loss='categorical_crossentropy',
                    metrics=['accuracy'])
        logger.debug(model.summary())
        return model

I get an error, which is as below:
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to         your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays

I have searched regarding this issue, but there were no answers for it (https://datascience.stackexchange.com/questions/39648/valueerror-numpy-arrays-that-you-are-passing-to-your-model-is-not-the-size-the)
Hence, I tried to customize the train method in the KerasPolicy as shown below,
def train(self,
    training_trackers: List[DialogueStateTracker],
    domain: Domain,
    **kwargs: Any
) -> None:

    # set numpy random seed
    np.random.seed(self.random_seed)

    training_data = self.featurize_for_training(training_trackers, domain, **kwargs)
    # noinspection PyPep8Naming
    shuffled_X, shuffled_y = training_data.shuffled_X_y()
    self.graph = tf.Graph()

    with self.graph.as_default():
        # set random seed in tf
        tf.set_random_seed(self.random_seed)
        self.session = tf.Session(config=self._tf_config)

    with self.session.as_default():
        if self.model is None:
            model, encoder_input_data, decoder_input_data, decoder_target_data = self.model_architecture(
                shuffled_X.shape[1:], shuffled_y.shape[1:]
            )
        logger.info(
                "Fitting model with {} total samples and a "
                "validation split of {}"
                "".format(training_data.num_examples(), self.validation_split)
            )

        # filter out kwargs that cannot be passed to fit
        #self._train_params = self._get_valid_params(
            #self.model.fit, **self._train_params
        #)

        self.model.fit(
            np.array([encoder_input_data, decoder_input_data]),
            np.array(decoder_target_data),
            epochs=self.epochs,
            batch_size=self.batch_size,
            shuffle=False,
            verbose=obtain_verbosity(),
            #**self._train_params
        )
            # the default parameter for epochs in keras fit is 1
        self.current_epoch = self.defaults.get("epochs", 1)
        logger.info("Done fitting keras policy model")

This throws an error as below,
custom_policy  - Fitting model with 21562 total samples and a validation split of 0.2

Traceback (most recent call last):
File "C:\Apps\sa\Scripts\rasa-script.py", line 11, in
load_entry_point('rasa', 'console_scripts', 'rasa')()
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa_main_.py", line 76, in main
cmdline_arguments.func(cmdline_arguments)
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa\cli\train.py", line 77, in train
kwargs=extract_additional_arguments(args),
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa\train.py", line 40, in train
kwargs=kwargs,
File "c:\apps\sa\lib\asyncio\base_events.py", line 579, in run_until_complete
return future.result()
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa\train.py", line 87, in train_async
kwargs,
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa\train.py", line 169, in _train_async_internal
kwargs=kwargs,
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa\train.py", line 203, in _do_training
kwargs=kwargs,
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa\train.py", line 331, in _train_core_with_validated_data
kwargs=kwargs,
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa\core\train.py", line 66, in train
agent.train(training_data, **kwargs)
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa\core\agent.py", line 713, in train
self.policy_ensemble.train(training_trackers, self.domain, **kwargs)
File "C:\Users\SA\AppData\Roaming\Python\Python37\site-packages\rasa\core\policies\ensemble.py", line 90, in train
policy.train(training_trackers, domain, **kwargs)
File "C:\ICA\restaurantbot\custom_policy.py", line 286, in train
self.model.fit(
AttributeError: 'NoneType' object has no attribute 'fit
Could I get help to solve this issue?

Thanks for the question, our Github issue board is for bugs and feature requests only, though 😕
Please ask this question in the forum✨
If you were describing a bug, please create a report and fill out the template