Twister: additional scripts

Add the execution of external scripts at precise moments. These scripts can be strategically deployed in three distinct phases: pre-script, post-flash-script and post-script. This functionality could help configuring the environment optimally before testing.
Script phases are already implemented solution, this PR serves as an extension to broaden their application and utility.
Usage is similar to quarantine file :
twister -p <BOARD> -T <TEST_SCENARIO> --scripting-list <PATH_TO_SCRIPTING_YAML>
An example of entries in a scripting list yaml:
    - scenarios:
    - sample.basic.helloworld
    platforms:
    - frdm_k64f
    pre_script:
        path: <PATH_TO_PRE_SCRIPT>
        timeout: <TIMEOUT TIME>
        override_script: <BOOLEAN>
    post_flash_script:
        path: <PATH_TO_PRE_SCRIPT>
        timeout: <TIMEOUT TIME>
        override_script: <BOOLEAN>
    post_script:
        path: <PATH_TO_PRE_SCRIPT>
        timeout: <TIMEOUT TIME>
        override_script: <BOOLEAN>
    comment:
        Testing extra scripts feature

Usage of all (pre_script, post_flash_script, post_script) is not mandatory, it could be only one parameter.
One of the primary benefits of this approach is the ability to execute scripts tailored to individual scenarios.

why no just put to the map.yaml file? along with --device-testing?

why no just put to the map.yaml file? along with --device-testing?

Currently, we have the capability to run scripts based on specified runner parameters; however, this approach limits us to the execution of pre scripts only. An alternative method would involve enhancing the map file to accommodate script execution, but this leads to redundancy when dealing with multiple identical boards, as we would need to replicate the script type, scenario, and path information for each entry.

Currently, we have the capability to run scripts based on specified runner parameters; however, this approach limits us to the execution of pre scripts only. An alternative method would involve enhancing the map file to accommodate script execution, but this leads to redundancy when dealing with multiple identical boards, as we would need to replicate the script type, scenario, and path information for each entry.
By this approach its easy to execute scripts tailored to individual scenarios.

This could be easily out of control as no one knows what you do in the rep/post, why we need customer for each case? why not just use pytest framework to do so?

You will have the flexibility to manage scripts within your own environment. This file is not mandatory, it serves to enhance the functionality of Twister for use in personalized settings.

this thing is if go this way, the twister will be diverged in its behavour, quaratine is to skip test cases. but addin gpre/post staff will impact the way test runs. it is better in a controlable way. unless you have strong user case to supporting doing so.

You will have the flexibility to manage scripts within your own environment. This file is not mandatory, it serves to enhance the functionality of Twister for use in personalized settings.

this thing is if go this way, the twister will be diverged in its behavour, quaratine is to skip test cases. but adding pre/post staff will impact the way test runs. it is better in a controlable way. unless you have strong user case to supporting doing so.

There are many ways of usage, i could take an example power management tests, where we can set up the necessary power conditions, calibrate or reset power measurement tools, perform initial checks to validate that the device is in the right state for power measurements and gather power usage data

@evgeniy-paltsev could you take a look?

why no just put to the map.yaml file? along with --device-testing?

Currently, we have the capability to run scripts based on specified runner parameters; however, this approach limits us to the execution of pre scripts only. An alternative method would involve enhancing the map file to accommodate script execution, but this leads to redundancy when dealing with multiple identical boards, as we would need to replicate the script type, scenario, and path information for each entry. By this approach its easy to execute scripts tailored to individual scenarios.

@majunkier what about these alternative improvements then:

extend test-config-schema with additional attributes instead of introducing the separate scripting-schema ?
run twister several times with different command line options for a test scope, platforms and scripts (+ missed options)  ?

@majunkier do you have time to follow up with @golowanow comments? besides that this PR looks ok.

why no just put to the map.yaml file? along with --device-testing?

Currently, we have the capability to run scripts based on specified runner parameters; however, this approach limits us to the execution of pre scripts only. An alternative method would involve enhancing the map file to accommodate script execution, but this leads to redundancy when dealing with multiple identical boards, as we would need to replicate the script type, scenario, and path information for each entry. By this approach its easy to execute scripts tailored to individual scenarios.

@majunkier what about these alternative improvements then:

extend test-config-schema with additional attributes instead of introducing the separate scripting-schema ?
run twister several times with different command line options for a test scope, platforms and scripts (+ missed options)  ?

This works as quarantine list file - reduce complexity of test-config-schema, easier to manage, test-config-schema contains the structure and rules for how tests should be written and this is extra action without rules
There are few tests that check functionality

I saw @hakehuang asking about it, but haven't seen an answer: isn't using pytest for such scripting already a more general solution with the ease of expandability with fixtures?

I saw @hakehuang asking about it, but haven't seen an answer: isn't using pytest for such scripting already a more general solution with the ease of expandability with fixtures?

Indeed, there are numerous methods to attain the same or comparable solutions. However, this solution emphasizes enhancing the core functionality of Twister, making it more apt for testing within the Zephyr ecosystem.

Indeed, there are numerous methods to attain the same or comparable solutions. However, this solution emphasizes enhancing the core functionality of Twister, making it more apt for testing within the Zephyr ecosystem.

I do support to add such case by case control method, however, as @PerMac and @golowanow suggested, maybe we need consider to use pytest instead of on the fly shell scripts, as those scripts are not likely be maintained in tree, this diverge may introduce inconsistency to test result. so for your case, would you try use pytest to organize your logic and which will be in-tree and benefit for all other developers. or you find it is impossible to do in pytest?

Indeed, there are numerous methods to attain the same or comparable solutions. However, this solution emphasizes enhancing the core functionality of Twister, making it more apt for testing within the Zephyr ecosystem.

I do support to add such case by case control method, however, as @PerMac and @golowanow suggested, maybe we need consider to use pytest instead of on the fly shell scripts, as those scripts are not likely be maintained in tree, this diverge may introduce inconsistency to test result. so for your case, would you try use pytest to organize your logic and which will be in-tree and benefit for all other developers. or you find it is impossible to do in pytest?

This change does not affect pytest as a plugin, it is an extension to the basic twister implementation. The functionality already exists, this changes are adding an additional extension for the currently working solution.

Apart from review changes I have implemented additional parameters to the scripting schema:
'timeout': parameter to specify the duration for script execution. @hakehuang  based on your changes
'override_script': a parameter that requires confirmation before proceeding to override and execute the existing script.

@majunkier would update this PR, I think there are only some code snippet changes by now. Thanks. Am I right? @golowanow

@majunkier please rebase

@majunkier can you add/update a testcase to example the usage?


Cake Icon in Solution Explorer

Within the IgnoreFiles VS Extension from @madskristensen I "think" I am right in saying that he is associating a, icon with the different ignore files, so that these icons appear in the solution explorer:
https://github.com/madskristensen/IgnoreFiles/blob/master/src/ContentType/icons.pkgdef
I wonder whether something like this would be possible for the build.cake file?  Can we associated the cake icon with the build.cake file, so that it appears in the solution explorer?

@hmemcpy do you know if this is possible?

The KnownMonikers, as the name suggests, are icons that are known to Visual Studio (you can see a list of them here: http://glyphlist.azurewebsites.net/knownmonikers/)
I'm fairly certain it's possible to register a custom icon and give it a name, so it can be assigned. But I haven't done this, so I don't know for sure.
Another option is to assign .cake files to an icon in Windows, and it be picked up in Visual Studio too.
I need to look into it some more (perhaps the folks in extendvs have better ideas...

@hmemcpy said...
I need to look into it some more (perhaps the folks in extendvs have better ideas..

Perfect, I will reach out to them, thanks, didn't know about that gitter room :+1:

Oh yeah, that room is a goldmine for anything extension related. All the cool kids hang out there :)

Also, I completely forgot that I've done something like this before in code: using some goodies from the VS Project System (aka CPS, Common Project System)!
Totally doable: https://github.com/Microsoft/VSProjectSystem/blob/master/doc/scenario/provide_custom_icons_for_the_project_or_item_type.md

Aw sweet! I will take a look at this as well. Cheers!

@hmemcpy ok, I am reading through that article, but I am clearly missing something.  Do you know if there is an example somewhere that makes use of it?

Oof, I think I misled you about CPS - this would only work for CPS-based projects, and unfortunately, the existing C# project system is not based on CPS (yet, they are rewriting it for VS"15")
I'll have another look over the weekend...

@hmemcpy ah, ok. Not a problem.

Replaced with: https://github.com/agc93/Cake.VisualStudio


[vtk] Fix building with MPI and Python enabled

This fixes the configure step of VTK when both MPI and Python are enabled.
To build with MPI, we need to pass -DVTK_USE_MPI=ON.
When both MPI and Python are enabled, VTK needs mpi4py. This PR enabled VTK to build its own version of mpi4py instead of looking for an external version of it.
I am not 100% sure whether this is the best approach. Maybe having mpi4py as separate port is the better solution. Please note: even though mpi4py is a Python package, usually it is required that mpi4py and VTK are linking to the same MPI implementation to function correctly. Additionally installing mpi4py comes with a header mpi4py.h that is required by VTK, so in some sense it is both, a Python package and a C library, just like VTK itself. Let me know what you think.

mpi4py should probably be a freestandig port on its own.

Please update the version info. See documentation.

Sorry, forgot to update it. I've pushed a change to increase to port version.

@albertziegenhagel Sorry, I've no idea, I don't use that so I have no authority.

@JackBoosY: I am also not using it and also don't have any authority but I have an opinion and you should have one too ;)
Just consider the fact that https://github.com/mpi4py/mpi4py is a completly separate from VTK and the VTK option is named VTK_MODULE_USE_EXTERNAL_VTK_mpi4py

mpi4py should indeed be a separate port, but we have not added it yet. So i think it's ok.

When building vtk-m[core,mpi]:x86-windows:
G:\12971\vcpkg\buildtrees\vtk-m\src\76c3434ac0-e3aceb77b8.clean\vtkm/Math.h(2665): error C3861: '_BitScanForward64': identifier not found
G:\12971\vcpkg\buildtrees\vtk-m\src\76c3434ac0-e3aceb77b8.clean\vtkm/Math.h(2748): error C3861: '__popcnt64': identifier not found

@JackBoosY: That means the feature does not support x86 since that are x64 intrinsics. See https://docs.microsoft.com/de-de/cpp/intrinsics/bitscanforward-bitscanforward64?view=vs-2019

Is this a regression?
This PR is only about VTK, not VTK-m. I have not tried building VTK-m with MPI yet, but I can't see how changes to the VTK port should affect VTK-m. Could you elaborate on where those errors come from?

Well, ignore feature mpi when building x86-windows, I still got some errors:
Building vtk[python]:x86-windows:
CMake Error at ThirdParty/hdf5/CMakeLists.txt:26 (message):
  An external MPI-aware HDF5 requires that VTK be built with MPI support as
  well.

@JackBoosY: That happens if you have hdf5[parallel] installed

@Neumann-A We may need to find a way to warn users in portfile.cmake.

@JackBoosY: Shouldn't block this PR since this error is also currently in master. Currently vcpkg does not provide a way to inspect/retrieve the installed feature set & build dependencies (both are required) of a installed port. You can try to invent vcpkg_get_installed_port_info which returns that information.

Test results:

x64-windows pass
x64-windows-static pass
x86-windows because of the existing regression, ignore

Thanks for your contribution!


Concurrency management

As discussed #17
It seems that we are not correctly implementing the "secure" routine for the TCP connection/request handler.
https://crystal-lang.org/docs/guides/concurrency.html
Currently we only spawn new connections and new requests, but we are not entirely sure if they do run, when do they run, or even how do they run. If one of those raises an error we are just going to ignore it probably. (before we were crashing everything for any sort of error, now we just ignore as an Unhandled exception). Also if the program finishes or crashes we don't make sure to handle or close incoming requests.
The next step should be implementing something with Fibers and Channels, to achieve a safe-way of handling the concurrency.
One or more fibers will handle the BoJack::Memory and the new connections, as also a channel will handle the requests itself, since we don't need to share the memory between then but only access the memory instance.
Crystal's guide already has a very nice example, with TCP Sockets:
require "socket"

channel = Channel(String).new
server = TCPServer.new
spawn do
  socket = server.accept

 while request = socket.gets
    channel.send(request)
  end
end

loop do
   request = channel.receive
   # handle the request
   ...
end

I believe this way we can achive something more reliable, because we will create something similar to a queue to handle the requests, even though we may hold a connection open for a long time on very huge concurrent access, at least we ensure its execution.

Implemented on 5b721d1


Filter out default pages

Many people never changed their homepage and don't have any interesting content. These should be filtered out!

We can use hashes to detect default pages.
The hahses of ~club are already available:
https://raw.githubusercontent.com/tildeclub/site/master/index.php


Calculate generator polynom once

Improves performance by calculating generatorPolynom only a single time.  Mostly affects larger QR codes, as they contain more than a very few blocks of data.
Before

Method
Mean
Error
StdDev
Gen0
Allocated

CreateQRCode
107.2 us
0.66 us
0.62 us
1.2207
23.18 KB

CreateQRCodeLong
1,878.4 us
4.68 us
4.15 us
23.4375
433.53 KB

After

Method
Mean
Error
StdDev
Gen0
Allocated

CreateQRCode
104.3 us
1.20 us
1.06 us
1.2207
23.19 KB

CreateQRCodeLong
1,687.0 us
9.91 us
9.27 us
7.8125
156.82 KB

I don't know why these tests pass.  CalculateECCWords mutates generatorPolynom.  I have to rework this PR.

Just for fun, I've added another benchmark for a 2600-character QR code and compared this PR against v1.5.1.  For these unreasonably-long QR codes, memory requirements have been reduced from 110 MB to 114 KB, a 99.9% reduction.  And there's still room for improvement.
  I'll merge in your suggestions here later tonight.


Integrate depacketizer in ExternalOuput adding h264

Description
This PR integrates the new Depacketizer into ExternalOutput replacing all the old vp8 depacketizer login. As part of the process of integrating it, this PR updates the Depacketizer to better suit the rest of Erizo naming conventions and removing the throw code.
This PR also adds support for H264 in recording new that we have a compatible depacketizer, that required a minor change in LayerDetectorHandler to start marking the spatial layer of h264 packets (for now it's always 0).
[] It needs and includes Unit Tests
Changes in Client or Server public APIs

[] It includes documentation for these changes in /doc.

@yannistseng that did the trick. Thank you very much! :+1:


gitlens.view.stageFile bind hotkey doesnt' work

Logs:
There was nothing output to the output panel when press key that bind into stageField command
Reproduce:
1 Bind command with desire key

2 Focus onto repositories view and changed file section and press key that bind into the stageFile command and nothing happend

Unfortunately this is a bug in vscode itself. I have open this issue: https://github.com/Microsoft/vscode/issues/72442


api.tracker.watchdog not initialized?

I'm assuming this is related to the new gmail interface.
I'm not much of a JS developer but I was just tasked with creating a chrome extension for gmail. I've been debugging the issue where I'm trying to observe.before("send_message") but it looks like XHR events aren't working correctly after the change.
I'm seeing an issue at line 1757 where api.tracker.watchdog is undefined.
I see this is setup under the api.observe.bind function, but it seems this is never called so api.tracker.watchdog is never set.
How does this get initialized? I'd love to help getting this working right again so any maintainers feel free to reach out and point me in the right direction.

I don’t think before message changed works on New Gmail at all.
After should be good though.

Ha, but I need before. Does observe.on work for send_message?
Is there any documentation on what's been found with the google xhr requests? I believe I saw somewhere else that you (maybe someone else) have an NDA but is there anything as part of this project?
What I think I'm seeing is that the bind function is called after the xhr request?
Sorry, still wrapping my head around how this works. I'm basically just trying to find a way to check a few things right before an email is sent and then either cancel sending the email or continue based on that check. Starting to think I should drop using this library for now and just attempt to override the XMLHttpRequests open and send methods myself since this is a pretty narrow use case I need it for.

If you want to change the content of an email before sending, I believe right now your only choice is intercepting clicks to the DOM Send-button, alter the compose content and manually trigger a real (unintercepted) send afterwards.
I know: not really smooth, but that’s the only thing I can think of.

Yeah, I took that approach before but it's not exactly what I'm looking for. Thanks for the help.

Old issue is old. Closing.


[TIR, Relay] improve bfloat16 support

Motivation:
We are enabling bfloat16 in BYOC-oneDNN following the path: [float32 graph] --> <AMP> --> [bfloat16 graph] --> <BYOC> --> [TVM + oneDNN module]. While some of the Passes like FoldConstant can not work for bfloat16 before the improvements below.
Changes:

Add runtime datatype dispatch and skip asserts for uint16 for bfloat16 compatibility.
Add bfloat16 casting for unary intrinsic operators to enable the graph optimization.
Improve the bf16_legalize module to enable bfloat16 lowering.

With those improvements, a float32 graph could be converted to bfloat16 through AMP, and then be lowered to inference in bfloat16 mode now.
Tested Models (gluoncv):

ResNet<18/34/50/101/152>_v1b
VGG<11/13/16/19>
VGG<11/13/16/19>_bn
DenseNet121
InceptionV3

By tested I mean I confirm it did some transformation on the graph and a forward pass could be run on CPU and matches the fp32 output somewhat. I have nothing on performance metrics or other devices yet.

As @AndrewZhaoLuo said at https://github.com/apache/tvm/pull/8069
Pending:
The support for bfloat16 in BYOC-oneDNN is based on multi-blocking layout transform and the extensions on BYOC-oneDNN and pending.

Tested Models (gluoncv):
ResNet<18/34/50/101/152>_v1b
VGG<11/13/16/19>
VGG<11/13/16/19>_bn
DenseNet121
InceptionV3

By tested I mean I confirm it did some transformation on the graph and a forward pass could be run on CPU and matches the fp32 output somewhat. I have nothing on performance metrics or other devices yet.

Hi @yangulei , Thanks for the patch. Could you share some details about the work you had done on these graphs?

Hi @billishyahao ,
You can checkout my fork. The readme file include a brief introduction about how to setup the environment and do the benchmarking. Be aware that this is not the final branch for upstreaming. You need a CPU with AVX-512 enabled to support bfloat16 functionally, and a CPU with AMX enabled to support bfloat16 natively, take a look at oneDNN document for more details.
FYI: we plan to upstream a tutorial about BYOC-oneDNN after the PRs about this are merged.


[T2A2] [T01-T3] Teo Shu Qi

Ready For Review

@teoshuqi Some comments added. Please close the PR after reading comments.


data leakage in webis/conclugen dataset

Describe the bug
Some samples (argument-conclusion pairs) in the training split of the webis/conclugen dataset are present in both the validation and test splits, creating data leakage and distorting model results.
Furthermore, all splits contain duplicate samples.
Steps to reproduce the bug
from datasets import load_dataset

training = load_dataset("webis/conclugen", "base", split="train")
validation = load_dataset("webis/conclugen", "base", split="validation")
testing = load_dataset("webis/conclugen", "base", split="test")

# collect which sample id's are present in the training split
ids_validation = list()
ids_testing = list()

for train_sample in training:
    train_argument = train_sample["argument"]
    train_conclusion = train_sample["conclusion"]
    train_id = train_sample["id"]

    # test if current sample is in validation split
    if train_argument in validation["argument"]:
        for validation_sample in validation:
            validation_argument = validation_sample["argument"]
            validation_conclusion = validation_sample["conclusion"]
            validation_id = validation_sample["id"]
            if train_argument == validation_argument and train_conclusion == validation_conclusion:
                ids_validation.append(validation_id)

    # test if current sample is in test split
    if train_argument in testing["argument"]:
        for testing_sample in testing:
            testing_argument = testing_sample["argument"]
            testing_conclusion = testing_sample["conclusion"]
            testing_id = testing_sample["id"]
            if train_argument == testing_argument and train_conclusion == testing_conclusion:
                ids_testing.append(testing_id)

Expected results
Length of both lists ids_validation and ids_testing should be zero.
Actual results
Length of ids_validation = 2556
Length of ids_testing = 287
Furthermore, there seems to be duplicate samples in (at least) the training split, since:
print(len(set(ids_validation))) = 950
print(len(set(ids_testing))) = 101
All in all, around 7% of the samples of each the validation and test split seems to be present in the training split.
Environment info

datasets version: 1.18.4
Platform: macOS-12.3.1-arm64-arm-64bit
Python version: 3.9.10
PyArrow version: 7.0.0

Hi @xflashxx, thanks for reporting.
Please note that this dataset was generated and shared by Webis Group: https://huggingface.co/webis
We are contacting the dataset owners to inform them about the issue you found. We'll keep you updated of their reply.

i'd suggest just pinging the authors here in the issue if possible?

Thanks for reporting this @xflashxx. I'll have a look and get back to you on this.

Hi @xflashxx and @albertvillanova,
I have updated the files with de-duplicated splits. Apparently the debate portals from which part of the examples were sourced had unique timestamps for some examples (up to 6%; updated counts in the README) without any actual content updated that lead to "new" items. The length of ids_validation and ids_testing is zero.
Regarding impact on scores:

We employed automatic evaluation (on a separate set of 1000 examples) only to justify the exclusion of the smaller models for manual evaluation (due to budget constraints). I am confident the ranking still stands (unsurprisingly, the bigger models doing better than those trained on the smaller splits). We also highlight this in the paper.

The examples used for manual evaluation have no overlap with any splits (also because they do not have any ground truth as we applied the trained models on an unlabeled sample to test its practical usage). I've added these two files to the dataset repository.

Hope this helps!

Thanks @shahbazsyed for your fast fix.
As a side note:

Your email appearing as Point of Contact in the dataset README has a typo: @uni.leipzig.de instead of @uni-leipzig.de
Your commits on the Hub are not linked to your profile on the Hub: this is because we use the email address to make this link; the email address used in your commit author and the email address set on your Hub account settings.


[Youtube] Download of all playlists on a /c/ link not working as on /user/

Checklist

[X] I'm reporting a broken site support issue
[X] I've verified that I'm running youtube-dl version 2<IP_ADDRESS>
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar bug reports including closed ones
[X] I've read bugs section in FAQ

Version output
> youtube-dl --version
2<IP_ADDRESS>

Verbose log
> youtube-dl --skip-download --verbose https://www.youtube.com/c/CodingGardenwithCJ/playlists
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--skip-download', u'--verbose', u'https://www.youtube.com/c/CodingGardenwithCJ/playlists']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2<IP_ADDRESS>
[debug] Python version 2.7.17 (CPython) - Linux-4.19.104-microsoft-standard-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6
[debug] Proxy map: {}
[youtube:user] CodingGardenwithCJ: Downloading channel page
[youtube:playlist] UULNgu_OupwoeESgtab33CCw: Downloading webpage
[download] Downloading playlist: Uploads from Coding Garden with CJ
[youtube:playlist] UULNgu_OupwoeESgtab33CCw: Downloading page #1
[youtube:playlist] UULNgu_OupwoeESgtab33CCw: Downloading page #2
^C
ERROR: Interrupted by user

Description
I noticed a problem: If I want to download all playlist of a channel which has a /c/ link, youtube-dl just finds one playlist (Uploads of <Channellist>) but not all other playlists like with a /user/ link.
2 links where it is not working: https://www.youtube.com/c/GotoConferences/playlists, https://www.youtube.com/c/CodingGardenwithCJ/playlists
link where it works: https://www.youtube.com/user/TheDeepForces/playlists,

Came here to report the same issue.  I am however not sure that it is related to /c/ or /user/ link.
OP: as a workaround you can try https://www.youtube.com/channel/UCLNgu_OupwoeESgtab33CCw/playlists and https://www.youtube.com/user/GotoConferences/playlists  but it seems to depend on the channel what link you should follow.

Hey @stynoo :) Thanks for the workaround. How did you found the channel id for the channel?

I use the youtube search box to search for 'CodingGardenwithCJ' and when you follow the first result it lands you right on that page.  I am once again not sure that that will work in every case, but I got good results so far.

This was fixed by e942cfd1a74cea3a3a53231800bbf5a002eed85e

So it will be in the next Release?


Adding annotations fails with ID and Contact Annotations

Problem description

Steps to reproduce the issue

Start up a 1.1.1 Data-node
Run my custom program to insert data from mcw into this node after inserting the data . I pretty much followed the same process as : push_dataset.py
Received the error

datanode.exceptions.ApiAttributeError: AnnotationCreateRequest has no attribute 'text_id_annotations' at ['annotation_create_request']['text_id_annotations']

The program worked up to this point. I am just trying to upload an empty set , but this worked for the other annotation types like date , person_name ,and physical_address.
Expected behavior
-Successful uploading of data
Extra information
I tried to get data out of the node after successfully uploading data ( without the text_id_annotations and text_contact_annotations elements with :
nlp-cli datanode list-annotations --data_node_host http://myhost.mcw.edu:8080/api/v1 --dataset_id mcwtest --annotation_store_id annotation2

and get the error :
 nlp-cli datanode list-annotations --data_node_host http://myhost.mcw.edu:8080/api/v1 --dataset_id mcwtest --annotation_store_id annotation2
Traceback (most recent call last):
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1599, in get_allof_instances
    allof_instance = allof_class(**kwargs)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1567, in wrapped_init
    return fn(self, *args, **kwargs)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model/page_of_annotations_all_of.py", line 172, in __init__
    setattr(self, var_name, var_value)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 168, in __setattr__
    self[attr] = value
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 353, in __setitem__
    self.set_attribute(name, value)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 142, in set_attribute
    self._check_type, configuration=self._configuration)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1451, in validate_and_convert_types
    configuration=configuration
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1403, in validate_and_convert_types
    check_type=_check_type
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1284, in attempt_convert_item
    configuration, spec_property_naming)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1202, in deserialize_model
    return model_class(**kw_args)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1567, in wrapped_init
    return fn(self, *args, **kwargs)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model/annotation.py", line 196, in __init__
    setattr(self, var_name, var_value)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 168, in __setattr__
    self[attr] = value
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 353, in __setitem__
    self.set_attribute(name, value)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 120, in set_attribute
    path_to_item
datanode.exceptions.ApiAttributeError: Annotation has no attribute 'textContactAnnotations' at ['received_data']['annotations'][0]['textContactAnnotations']

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/bin/nlp-cli", line 33, in <module>
    sys.exit(load_entry_point('nlpsandbox-client==1.0.1', 'console_scripts', 'nlp-cli')())
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/nlpsandboxclient/cli/__main__.py", line 19, in main
    cli()
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/nlpsandboxclient/cli/datanode_cli.py", line 106, in list_annotations
    utils.stdout_or_json(list(annotations), output)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/nlpsandboxclient/client.py", line 230, in list_annotations
    offset=offset, limit=limit
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/api_client.py", line 772, in __call__
    return self.callable(self, *args, **kwargs)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/api/annotation_api.py", line 538, in __list_annotations
    return self.call_with_http_info(**kwargs)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/api_client.py", line 850, in call_with_http_info
    collection_formats=params['collection_format'])
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/api_client.py", line 415, in call_api
    _check_type)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/api_client.py", line 227, in __call_api
    _check_type
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/api_client.py", line 331, in deserialize
    configuration=self.configuration
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1403, in validate_and_convert_types
    check_type=_check_type
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1291, in attempt_convert_item
    raise conversion_exc
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1284, in attempt_convert_item
    configuration, spec_property_naming)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1202, in deserialize_model
    return model_class(**kw_args)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1567, in wrapped_init
    return fn(self, *args, **kwargs)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model/page_of_annotations.py", line 215, in __init__
    constant_args, model_args, self)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1847, in validate_get_composed_info
    allof_instances = get_allof_instances(self, model_args, constant_args)
  File "/Users/gkowalsk/PycharmProjects/prodigy/venv/lib/python3.7/site-packages/datanode/model_utils.py", line 1611, in get_allof_instances
    ) from ex
datanode.exceptions.ApiValueError: Invalid inputs given to generate an instance of 'PageOfAnnotationsAllOf'. The input data was invalid for the allOf schema 'PageOfAnnotationsAllOf' in the composed schema 'PageOfAnnotations'. Error=Annotation has no attribute 'textContactAnnotations' at ['received_data']['annotations'][0]['textContactAnnotations']

@gkowalski this is because the client doesn't support it currently.

@gkowalski Meanwhile you should aim at fully supporting date, person nanme and physical address, i.e. have the entire dataset annotated and populated in the date node. I'll work this week on formatting the i2b2 data to the NLP Sandbox schemas (maybe tonight?) and then we will update the client tomorrow/Thursday. At least that's the plan.

@gkowalski I think this issue is fixed, but to make sure, here are the steps:

Update the datanode (there should be version 1.1.2 now)
Install the nlpsandbox-client from source: https://github.com/nlpsandbox/nlpsandbox-client (There will be v2.0 coming out soon, but I just wanted to make sure everything was tested)
Push the example (should include id, contact and covid_symptom annotations.  See the examples_bundles.jsoncd nlpsandbox-client
python examples/push_dataset.py

(venv) [crdw@darvin:~/nlpsandbox-client]$ python examples/push_dataset.py
Traceback (most recent call last):
File "examples/push_dataset.py", line 16, in
import nlpsandboxsdk
ModuleNotFoundError: No module named 'nlpsandboxsdk'
(venv) [crdw@darvin:~/nlpsandbox-client]$
hmm, not in the requirements.txt file.

pip install nlpsandbox-client --upgrade was also tried and didn't help

@gkowalski Tom is asking to install the python client from source (not from PyPi)

worked
python push_dataset.py
Deleting exist dataset: test-dataset
Creating dataset: test-dataset
Creating Fhir Store: evaluation
Creating Annotation Store: goldstandard
Creating patient: 110
Creating note (110-01) for patient (110)
Creating annotation for note: 110-01
Creating note (110-02) for patient (110)
Creating annotation for note: 110-02
Creating note (110-03) for patient (110)
Creating annotation for note: 110-03
Creating note (110-04) for patient (110)
Creating annotation for note: 110-04
Creating note (110-05) for patient (110)
Creating annotation for note: 110-05


move notebooks to root

the notebook directory should be at the root of the repository

Closing per #18


Request Site javbangers.com - exits with 403

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a new site support request
[X] I've verified that I'm running yt-dlp version 2022.10.04 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that none of provided URLs violate any copyrights or contain any DRM to the best of my knowledge
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and am willing to share it if required

Region
Switzerland
Example URLs
https://www.javbangers.com/video/59569/new-angel-42
Provide a description that is worded well enough to be understood
javbangers.com exits with 403.
Porntrex.com uses the same mechanisms since they are part of a network, they share the same login credentials and "KWS Player".
My command works for porntrex but not for javbangers, using "--cookies "javbangers.com_cookies.txt"" or "--cookies-from-browser chrome" when logged in didnt change the output.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['-vU', '-i', '-o', 'D:\\yt-dlp-downloads\\%(title)s.%(ext)s', 'https://www.javbangers.com/video/59569/new-angel-42']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.10.04 [4e0511f] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg N-105029-gf3a949dc4c-20211225 (setts)
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1690 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.10.04, Current version: 2022.10.04
yt-dlp is up to date (2022.10.04)
[debug] [generic] Extracting URL: https://www.javbangers.com/video/59569/new-angel-42
[generic] new-angel-42: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] new-angel-42: Extracting information
[debug] Looking for Brightcove embeds
[debug] Looking for embeds
[debug] Identified a KWS Player
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 59569: Downloading 1 format(s): video_url
[debug] Invoking http downloader on "https://www.javbangers.com/get_file/4/de37a6f2d59230cd893c918aef2e7032f9214dc1d1/59000/59569/59569.mp4/?br=591"
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 3196, in process_info
  File "yt_dlp\YoutubeDL.py", line 2920, in dl
  File "yt_dlp\downloader\common.py", line 446, in download
  File "yt_dlp\downloader\http.py", line 371, in real_download
  File "yt_dlp\downloader\http.py", line 129, in establish_connection
  File "yt_dlp\YoutubeDL.py", line 3682, in urlopen
  File "urllib\request.py", line 531, in open
  File "urllib\request.py", line 640, in http_response
  File "urllib\request.py", line 563, in error
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 755, in http_error_302
  File "urllib\request.py", line 531, in open
  File "urllib\request.py", line 640, in http_response
  File "urllib\request.py", line 569, in error
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 649, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

This doesn't seem to be an legal website. Closing


Android - Showing notification from inside Foreground Service while app is in Background

Hello,
I would like to know if it is possible to show a notification from a Foreground Service while the app is in background
My use case is an Android application polling an API from inside a Foreground Service and at a certain point showing a FullScreen notification.
If the user presses the "accept" action button, the app should rise up and go to the right Screen.
When I fire a notification from my Foreground Service while app is in Background, I get a notification "DELIVERED" to my onBackgroundEvent handler, but no notification is shown.
So I tried showing the notification from inside the onBackgroundEvent handler (accoridng to this documentation) but I cannot do it anyway.
If I try to show a notification from the onBackgroundEvent handler i create a loop in the app (each notification triggers another background event)
By the way, when I run the foreground service while keeping the app in Foreground, I see the FullScreen notification.
Any help is appreciated,
thank you.

I found the problem on my side.
In order to help someone else in the future having trouble in settng up this scenario, I created a simple demo repo accomplishing the task.
Thank you for this great library!


fix(SelectMenu): use by prop to compare objects for selected values & add support for dot notation in value-attribute

🔗 Linked issue
fixes https://github.com/nuxt/ui/issues/2028
❓ Type of change

[ ] 📖 Documentation (updates to the documentation or readme)
[x] 🐞 Bug fix (a non-breaking change that fixes an issue)
[x] 👌 Enhancement (improving an existing functionality)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

This PR fixes a bug introduced in https://github.com/nuxt/ui/pull/1349, where the by prop was not being consiederd when computing the selected values.
It also adds the possibility to combine the by prop with the value-attribute prop to compare nested objects like in the following example:
<template>
  <USelectMenu
    v-model="selected"
    :options="options"
    multiple
    value-attribute="object"
    by="id"
  />
</template>

<script lang="ts" setup>
const selected = ref([{
  id: 1,
  title: 'Hello'
}])

const options = [
  {
    key: 'id',
    label: 'ID',
    object: {         // 👈 this will be the selected value
      id: 1,             // 👈 the objects will be compared by this attribute
      title: 'Hello'
    }
  },
  {
    key: 'title',
    label: 'Title',
    object: {
      id: 2,             // 👈 the objects will be compared by this attribute
      title: 'Hi'
    }
  }
]
</script>

📝 Checklist

[x] I have linked an issue or discussion.
[ ] I have updated the documentation accordingly.

I also added the possibility to compare nested objects using the by prop when the value-attribute prop is set
Consider the following example:
<template>
  <USelectMenu
    v-model="selected"
    :options="options"
    multiple
    value-attribute="object"
    by="id"
  />
</template>

<script lang="ts" setup>
const selected = ref([{
  id: 1,
  title: 'Hello'
}])

const options = [
  {
    key: 'id',
    label: 'ID',
    object: {
      id: 1,
      title: 'Hello'
    }
  },
  {
    key: 'title',
    label: 'Title',
    object: {
      id: 2,
      title: 'Hi'
    }
  },
  {
    key: 'status',
    label: 'Status',
    object: {
      id: 3,
      title: 'Hey'
    }
  }
]
</script>

Previously (even before the introduction of the selected computed) it was not possible to combine these 2 props. This approach made sense to me, but I’m happy to hear any feedback on this.

Would you mind fixing the same thing on the InputMenu?  Both have the same code.

@benjamincanac sure! However, I'll have to postpone it to tomorrow if that's ok

I've updated the code to support dot notation for the value-attribute prop in <SelectMenu>. I also wanted to add dot notation support for the by prop, but ran into a limitation with HeadlessUI: the HCombobox and HListbox components don't support it. The by prop is passed as a string to these components, so there's currently no way to support it in NuxtUI unless HeadlessUI adds support.
It should also be noted that the by prop should only be used when the final value is an object. Using it with other types results in strange behavior, which we cannot reasonably fix due to limitations with HCombobox, which is the source of these issues. This has been the case before as well. We should maybe document that more explicitly.

Closing in favor of #2566.


Unhandled Exception

Hi,
Running the command forge by itself in the command line produces the following
Unhandled Exception:
System.IndexOutOfRangeException: Index was outside the bounds of the array.
  at Forge.Commands.applyAlias (System.String[] args) [0x00007] in <59b9172273406ffea74503832217b959>:0
  at Forge.Commands.runForge (System.String[] args) [0x00001] in <59b9172273406ffea74503832217b959>:0
  at Forge.App.main (System.String[] argv) [0x0007b] in <59b9172273406ffea74503832217b959>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.IndexOutOfRangeException: Index was outside the bounds of the array.
  at Forge.Commands.applyAlias (System.String[] args) [0x00007] in <59b9172273406ffea74503832217b959>:0
  at Forge.Commands.runForge (System.String[] args) [0x00001] in <59b9172273406ffea74503832217b959>:0
  at Forge.App.main (System.String[] argv) [0x0007b] in <59b9172273406ffea74503832217b959>:0

Fixed in 36c9f5e8decbc7bf6e5b569a1feaa7db2aa800e0, will be available in next release

Hi! Maybe it's just me, because 2.0.0 is not yet available on scoop, so just overwritten the binaries, but without arguments, it does blow up:
>forge

Unhandled Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Forge.Commands.applyAlias(String[] args)
   at Forge.Commands.runForge(String[] args)
   at Forge.App.main(String[] argv)

Win 10, Forge 2.0.0 downloaded from the 2.0.0 release zip.


Can you publish this to Hackage?

I've gotten to the point in my work where I've defined my attoparsec parser, and now need to stream with it. I suppose I'll use streaming-utils until this is up?

https://github.com/haskell-streaming/streaming-attoparsec


Remove probably obsolete requirement for ruamel.yaml

See issue #134

@kennethreitz why are we using ruamel.yaml ? I've noticed that dateparser is using ruamel.yaml but when is this warning raised which we are ignoring?

Awesome - this has been bugging me for a while as well. Any idea when the sans-ruamel.yaml version will be released to PyPI?

ping! any word on a new release?

I've just released v0.4.1 which you can use :tada:

many thanks


Key not working with functional component

Version
2.5.16
Reproduction link
https://codepen.io/dododavid006/pen/vrKgEv
Steps to reproduce

Type something into input so you can distinguish input
Click the third remove button

gif:  https://imgur.com/I2Hqu6K
What is expected?
It should keep the fourth input.
What is actually happening?
The third input is keep.

functional components don't have an instance,, so you can't key the component itself. rather, the functional component has to key its element(s) itself - because it could potentially return multiple elements, so Vue can't reliably decide on which to apply the key, if one was provided.
render (h, {data, listeners: {remove}}) {
    return h(
      'div',
      { key: data.key },
      [
        h(
          'input',


fix: incorrect payload and context ordering on track

fix: incorrect payload and context ordering on track
Fixing a regression introduced in https://github.com/dendronhq/dendron/pull/1515 where the ordering of payload and context were flipped. Also fixes a typo introduced with the nodeVersion property.

Pull Request Checklist
You can go to dendron pull requests to see full details for items in this checklist.
General
Quality Assurance

[ ] add a test for the new feature
[ ] make sure all the existing tests pass
[ ] do a spot check by running your feature with our test workspace
[ ] after you submit your pull request, check the output of our integration test and make sure all tests pass

NOTE: if you running MacOS or Linux, pay special attention to the Windows output and vice versa if you are developing on Windows

Special Cases

[ ] if your tests changes an existing snapshot, make sure that snapshots are updated
[ ] if you are adding a new language feature (graphically visible in VS Code/preview/publishing), make sure that it is included in test-workspace. We use this to manually inspect new changes and for auto regression testing

Docs

[ ] Make sure that the PR title follows our commit style
[ ] Please summarize the feature or impact in 1-2 lines in the PR description
[ ] If your change reflects documentation changes, also submit a PR to dendron-site and mention the doc PR link in your current PR

Example PR Description
# feat: capitalize all foos

This changes capitalizes all occurrences of `foo` to `Foo`

Docs PR: <URL_TO_DOCS_PR>

Special Cases
First Time PR

[ ] sign the CLA which will be prompted by our github bot after you submit the PR
[ ] add your discord alias in the review so that we can give you the horticulturalist badge in our community

Analytics

[ ] if you are adding analytics related changes, make sure the Telemetry docs are updated

nice catch on the typo. I believe that's on me.
this regression could be a reminder that we should prefer passing objects as arguments and destructuring them to access them rather than passing in individual arguments. [[prefer using object notation for methods ith multiple arguments|dendron.dev.style#prefer-using-object-notation-for-methods-ith-multiple-arguments]]
I've been reminding myself to follow this as well these past couple weeks.

Yeah that's a good point.


Dual-publish our images to Docker Hub and ECR

To enable external users to get copies of the storage service code.
Include changelogs/versions.

We did this, albeit to ECR Public instead of Docker Hub.


Remove FeatureType enum, replace with np.dtype.

[x] Forked repo is synced with upstream -> github shows no code delta outside of the desired.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
[x] Tests are passing? https://github.com/microsoft/DeepGNN/blob/main/CONTRIBUTING.md#run-tests
[x] Documentation is added or updated to reflect new code in the same format as the rest of the repo.
[x] PR is labeled using the label menu on the right side.

New Behavior

Remove FeatureType enum, replace with np.dtype.
Fetch features parameter feature_type -> dtype.

Remake #46


feat: remove the text-based linters for module docstring, copyright and author lines

They are superseded by the header linter (#16574).

I suggest an experiment: add a file that breaks the rules and see that the new linter actually catches the errors. But probably people who are more involved in CI and metaprogramming have better ideas.

I just added such a file, the linter works locally. Does that look sufficient to you? (If CI fails, I'll remove it again.)

The internal logic of the headerLinter is tested in test/HeaderLinter.lean.  The linter picked up non-complying copyright headers in development and I also think that I did a test similar to the ones that Michael just did while developing the linter.
The test here also seems to work: besides the fact that the "new" definition does not have a doc-string, there are also build errors coming from the malformed copyright header.
My main reason for not being immediately enthusiastic about removing these "text-based" linters is that the syntax linter will typically lint Mathlib.Init and whatever imports that, but the text-based ones also cover the "early" files.

Wasn't the point of Mathlib.Init that every file imports it (or is imported in it)?
Checking the copyright headers of all these syntax linters is nice, I agree - but I don't think keeping a separate linter implementation around just for them makes too much sense.

Right, I simply have not yet accepted that the style-linters will not necessarily by style-compliant, that's all!  I agree that having as much as possible enforced in Lean is a good aim: once CI is happy, I am happy to maintainer merge this PR!

@adomani I just wrote #18725, importing the header linter in every file that is imported in Mathlib.Init. This should ensure all files are checked for missing copyright (or module doc-string). Does that assugae your concerns on this PR?

This is what was going to happen once the linter had been rewritten as a syntax linter, isn't it?  :wink:
I'll
maintainer merge
and also ping @bryangingechen, in case there are any last-minute comments!

LGTM, thanks for the ping!
bors r+


Finish StudySession Feature

New Session
1.1. Countdown timer
1.2. Termination and success conditions for sessions
View Session History
2.1. Insertion of session data in supabase
2.2.Retrieval and rendering of session data from supabase

TODO later:

Ensure users can't enter decimals for hours/minutes


Add support center link to "Help"

Expected Behavior
Replace "Forum" with "Support Center" linking to valoraapp.com/support
Current Behavior
Forum is outdated

support will now be located at https://support.valoraapp.com/
This site is not live yet, but will be soon (ideally by mid-july)


lslint: add checker for lsl

Adding a syntax checker for the Linden Scripting Language, used in Second Life and other similar games.
You can get lslint here.

This seems a little too specialized to be added to syntastic.  Perhaps you can maintain it as an external checker?


Update to version 2.0.1

Checklist

[ ] Used a fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Requires https://github.com/conda-forge/staged-recipes/pull/6924

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Requires https://github.com/conda-forge/r-pkgbuild-feedstock/pull/1


documentation: update operator documentation

In the documentation the catalog source's yaml file references an outdated operator that can't be downloaded from the quay repository:
spec:
  sourceType: grpc
  image: quay.io/hgao/operator-catalog:v0.0.1

Proposed fix: The private repo quay.io/hgao should be removed by the public doc

Could make sense to use a floating tag like latest to avoid outdated docs in future.

After taking a deeper look to the current bundle.md, I think it require a deeper refactoring:

The steps to build the bundle image should be removed because now the CI workflow push the official bundle images to quay.io/artemiscloud/activemq-artemis-operator-bundle
The v prefix should be removed from the tag versions
The private repo quay.io/hgao should be removed by the public doc
The catalog image tag should not match the operator version because a catalog image usually hosts multiple operators

Closed as completed.


early exit from telogen phase

Add new term:
early exit from telogen phase (a child of abnormal hair cycle telogen phase MP:0008860)
DEF: anomaly in the resting or quiescent phase of the hair growth cycle resulting a shortened telogen phase
SYN: shortened telogen phase
SYN: short telogen
SYN: decreased telogen phase duration
SYN: premature end of telogen
SYN: precocious exit from telogen
Annotation in J:189569

Added new term:
early exit from telogen phase MP:0030924


Romanlutz/cleanup charts

Description
This PR contains a few smaller cleanup tasks that I've been meaning to do:

Delete unused DisaggregatedMetricsTable. It was meant to be moved out of fairness to be generally available, but I decided not to go with this for technical reasons. It's best if it stays within fairness where there is already a differently named version of it.
Remove showMetricSummary prop from OverallMetricChart since it's always set to true. model-assessment is moving away from OverallMetricChart anyway so this will not be relevant in the future.

Areas changed

npm packages changed:

[ ] responsibleai/causality
[x] responsibleai/core-ui
[ ] responsibleai/counterfactuals
[ ] responsibleai/dataset-explorer
[ ] responsibleai/fairness
[ ] responsibleai/interpret
[ ] responsibleai/localization
[ ] responsibleai/mlchartlib
[ ] responsibleai/model-assessment

Python packages changed:

[ ] erroranalysis
[ ] raiutils
[ ] raiwidgets
[ ] rai_core_flask
[ ] responsibleai

Tests

[x] No new tests required.
[ ] New tests for the added feature are part of this PR.
[x] I validated the changes manually.

Screenshots (if appropriate):
Documentation:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

Codecov Report

Merging #1313 (bb760f2) into main (3c37a7c) will decrease coverage by 5.87%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1313      +/-   ##
==========================================
- Coverage   86.30%   80.43%   -5.88%
==========================================
  Files          99       18      -81
  Lines        4615      322    -4293
==========================================
- Hits         3983      259    -3724
+ Misses        632       63     -569

Flag
Coverage Δ

unittests
80.43% <ø> (-5.88%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

responsibleai/responsibleai/_config/__init__.py

...nsibleai/responsibleai/_managers/causal_manager.py

...leai/responsibleai/modelanalysis/model_analysis.py

responsibleai/responsibleai/_interfaces.py

erroranalysis/erroranalysis/report/__init__.py

...ibleai/responsibleai/managers/explainer_manager.py

responsibleai/responsibleai/_tools/__init__.py

...bleai/responsibleai/_tools/causal/causal_result.py

erroranalysis/erroranalysis/_internal/metrics.py

responsibleai/responsibleai/exceptions.py

... and 71 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c37a7c...bb760f2. Read the comment docs.


responsive/breakpoint unslick problem

I have set 'unslick' on my widest breakpoint (as per docs) with a view to display my items in a grid at larger device sizes. this works as expected when 'entering' the breakpoint that unslick is set on (i.e. the carousel gets destroyed/deactivated), but when resizing the window back down into the narrower breakpoints the carousel never reinitializes.
is this intended functionality, a bug or do i have to add a specific argument to my settings in the other breakpoints in order to reinitialize it somehow?

Its the design of the method :)

@simeydotme do you consider extending the features to provide reslick behavior out of the box? As I understand, to make it behave that way we would have to set a listener on window.resize and then reinitiate when appropriate. However, when we already have the listener in place then the unslick in the responsive settings isn't of that much use. Not that it's so big a deal, just curious.

I second this - I'm working on a project now where I need to unslick and reslick without having to refresh the page to reinitiate it.

Old issue, yet important, at least to me. Would be a nice feature the possibility to unslick and reslick responsively.


[Request for External] Codeunit 12101 - Withholding - Contribution change scope value - SocSecLineFilter

Please make external function "SocSecLineFilter" in the codeunit 12101 "Withholding - Contribution".
Localization: Italian

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Residual quantization - VRAM  bottleneck

This might be a issue related to the vector-quantize-pytorch but I'll bring it up here.
During some testing I discovered that the vector quantizer took up most VRAM. I'm not sure if this is a bug or the requirements of the quantizer.
After calling Encode() it used up about 500MB and decode uses about 300MB. After calling the quantize() it used about 4 116MB, if this could be resolved, the VRAM issues might be resolved as well.
I saw no signifiant difference using ResidualLFQ or ResidualVQ.
I tested with a big 3D model with 5956 vertices and 3k faces with batch size 1, the issue remains if you use a smaller 3d model.
I setup a bunch of debug prints to monitor VRAM, here is the results after calling autoendcoder.forward()

Autoencoder - forward start: 3064.0 MB
After encode() call : 3564.0 MB
Calling quantize():
ResidualLFQ - Forward: : 3564.0 MB
ResidualLFQ - for quantizer_index, layer in enumerate(self.layers)
LFQ layer 1:
Start VRAM : 3564.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
4312.0 MB    distance : torch.Size([1, 5956, 1, 16384])  original_input: torch.Size([1, 5956, 1, 14])  self.codebook: torch.Size([16384, 14])
prob = (-distance * inv_temperature).softmax(dim = -1)
4686.0 MB    prob: torch.Size([1, 5956, 1, 16384])
per_sample_entropy = entropy(prob).mean()
5808.0 MB  per_sample_entropy: 1.1416213512420654
entropy_aux_loss = per_sample_entropy - self.diversity_gamma * codebook_entropy
5856.0 MB    entropy_aux_loss: -0.8769185543060303
LFQ layer 2:
Start VRAM : 5808.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
6184.0 MB   distance : torch.Size([1, 5956, 1, 16384])  original_input: torch.Size([1, 5956, 1, 14])  self.codebook: torch.Size([16384, 14])
prob = (-distance * inv_temperature).softmax(dim = -1)
6558.0 MB    prob: torch.Size([1, 5956, 1, 16384])
per_sample_entropy = entropy(prob).mean()
7680.0 MB  per_sample_entropy: 0.0
entropy_aux_loss = per_sample_entropy - self.diversity_gamma * codebook_entropy
7728.0 MB    entropy_aux_loss: -1.3798797130584717
After calling quantize(): 7680.0 MB
After calling decoded() : 7937.0 MB
Step #2:
Autoencoder - forward start:  9095.0 MB

I don't think MeshDiscretizer can be used as a solution since I think they already tried it in paper and got quite bad results.

@MarcusLoppe ohh thanks for the analysis
yea let us look into gradient checkpointing after the holidays
have a great new years!

@MarcusLoppe ohh thanks for the analysis
yea let us look into gradient checkpointing after the holidays
have a great new years!
ps: i missed that ablation; it looks good. i'll remove the mesh discretizer in january

Oh, sorry for interrupting the vacation. 😢
I posted since didn't want to need to remember this info till next year.
Likewise, happy new year 🎉

@MarcusLoppe hey circling back to this. will try to get recomputed gradients working for the quantization module by tonight

@MarcusLoppe i added memory checkpointing in the latest version, which can be turned on by setting checkpoint_quantizer = True on the autoencoder
do you want to check to see if this reduces the memory usage you observe?

@MarcusLoppe i added memory checkpointing in the latest version, which can be turned on by setting checkpoint_quantizer = True on the autoencoder
do you want to check to see if this reduces the memory usage you observe?

@lucidrains
Great, I'll check this out tonight.
Btw, do you have good ideas to replace the Resnet34? I've tried using Resnet50 but since it's downsampling the output shape isn't the same, even if I use a upsampling (since the downsampling takes a e.g uneven number of faces 239 and reduces it to 120, then the upsampling will increase to 240 and make the output invalid).
Any idea for improving the ConvSage? Maybe through in a flash attention layer? :)
I'm asking this due to the ConvSage, Resnet and the codebook needs to get a lot better, the models and codebook aren't big to handle generalized mesh.
In the paper they pre-trained it on all meshes, then fine-tuned it on specific category's e.g chair.
This means that the models and codebook doesn't need to be that big since it only needs to be able to store one type of model pattern.
When I trained the autoencoder on 4 different categories (chair, table, sofa and display) using 50 models from each and augmented them 50 times (as per Polygen paper), I noticed that the loss got stuck at 1.17 after 200+ epochs and it would take additional 1-2k epoch to reach 0.7 loss.
But when i upped the codebook 32 768 (x2 increase), it managed to sail though the 1.17 with a breeze.
I then increased the dim_codebook to 384 (x2) and increased the model sizes so the decoder has 48.1M (from 6.8M) and the encoder had 6.1M (from 660k) parameters.
This made it so it reached 0.7 loss in 68 epochs.
I'm currently testing using  43.9M (decoder) and 8.0M (encoder) with a codebook size of 65 536 using 384 dim_codebook.
I'll let you know what I find, please let me know if you have any other suggestions for how to improve the models the most efficient way rather then just throwing lots of parameters on it 😄

It seems to save about 700MB using the 6200 face mesh model @ batch size 1, the curve is more linear so it did something :)
But it seems like "per_sample_entropy = entropy(prob).mean()" has the most effect on the VRAM, is there any optimization for that?

Start of loop =
for quantizer_index, layer in enumerate(self.layers):
checkpoint_quantize = False
Start of loop: 3405.0 MB
Step 1:
loop 0  3404.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
4188.0 MB
prob = (-distance * inv_temperature).softmax(dim = -1)
4588.0 MB
per_sample_entropy = entropy(prob).mean()
5788.0 MB
loop 1  5788.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
6192.0 MB
prob = (-distance * inv_temperature).softmax(dim = -1)
6593.0 MB
per_sample_entropy = entropy(prob).mean()
7793.0 MB
End of loop: 7793.0 MB
Step 2:
1 VRAM Used: 9776.0 MB
loop 0  9776.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
9776.0 MB
prob = (-distance * inv_temperature).softmax(dim = -1)
9776.0 MB
per_sample_entropy = entropy(prob).mean()
9776.0 MB
loop 1  9776.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
9787.0 MB
prob = (-distance * inv_temperature).softmax(dim = -1)
9787.0 MB
per_sample_entropy = entropy(prob).mean()
9787.0 MB
4 VRAM Used: 9787.0 MB
checkpoint_quantize = True
Step 1:
for quantizer_index, layer in enumerate(self.layers):
Start of loop: 3387.0 MB
loop 0  3387.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
4187.0 MB
prob = (-distance * inv_temperature).softmax(dim = -1)
4587.0 MB
per_sample_entropy = entropy(prob).mean()
5387.0 MB
loop 1  5387.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
5389.0 MB
prob = (-distance * inv_temperature).softmax(dim = -1)
5399.0 MB
per_sample_entropy = entropy(prob).mean()
5399.0 MB
End of loop: 5399.0 MB
Step 2:
1 VRAM Used: 5784.0 MB
loop 0  5784.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
5786.0 MB
prob = (-distance * inv_temperature).softmax(dim = -1)
5786.0 MB
per_sample_entropy = entropy(prob).mean()
6586.0 MB
Start of loop: 6586.0 MB
distance = -2 * einsum('... i d, j d -> ... i j', original_input, self.codebook)
6986.0 MB
prob = (-distance * inv_temperature).softmax(dim = -1)
7396.0 MB
per_sample_entropy = entropy(prob).mean()
8596.0 MB
End of loop: 8596.0 MB

@MarcusLoppe thanks for the analysis! could you try reducing this hyperparameter (to say 0.1) and see if that helps?

@MarcusLoppe thanks for the analysis! could you try reducing this hyperparameter (to say 0.1) and see if that helps?

Ah very nice! It managed to drop down to 3299MB from 6731MB, using 0.5 made it use 5133MB so the relationship is not quite linear.
I did the same test using fp16 and it went from 4029MB to 2636MB, so training a fp16 model using 6.2k triangles used only 2.6-4GB @ batch size 1.
Other then that test when I've trained the model using fp16 I've noticed a x2 in performance and less memory usage.
The only modification I did was to convert the LFQ codebook and the gaussian_blur_1d function so the distance had the dtype float16.
This might be a downside of fp16 but I got stuck at 0.35 loss, I'm currently checking and using float32 to verify that fp16 was the actual problem or not.
Is it possible to have the LFQ codebook or gaussian_blur_1d using the float32 while the models tensor are in fp16 or is this not possible? I think some of the details might have been lost due to the fp16 type but I don't think it's a overall problem but might just be due to the coodebook and gaussian blur.
frac_per_sample_entropy = 0.1
3921.0MB @  Starting loop
4721.0MB @  distance = -2 * einsum(
5121.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
5121.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
5112.0MB @  distance = -2 * einsum(
5074.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
5075.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
5076.0MB @  End of loop
5579.0MB @  Starting loop
5579.0MB @  distance = -2 * einsum(
5979.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
5979.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
6381.0MB @  distance = -2 * einsum(
6781.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
6781.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
6781.0MB @  End of loop
Epoch 1/280:   2%|▏         | 1/50 [00:03<02:41,  3.30s/it, commit_loss=1.04, loss=5.02, recon_loss=4.92]
7210.0MB @  Starting loop
7210.0MB @  distance = -2 * einsum(
7210.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7210.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7210.0MB @  distance = -2 * einsum(
7210.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7210.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7210.0MB @  End of loop
7220.0MB @  Starting loop
7220.0MB @  distance = -2 * einsum(
7220.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7220.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7220.0MB @  distance = -2 * einsum(
7220.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7220.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7220.0MB @  End of loop

frac_per_sample_entropy = 0.5
3901.0MB @  Starting loop
4701.0MB @  distance = -2 * einsum(
5101.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
5501.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
5501.0MB @  distance = -2 * einsum(
5501.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
5485.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
5485.0MB @  End of loop
5785.0MB @  Starting loop
5785.0MB @  distance = -2 * einsum(
6185.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
6985.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7387.0MB @  distance = -2 * einsum(
7787.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
8587.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
8587.0MB @  End of loop
Epoch 1/280:   2%|▏         | 1/50 [00:03<02:43,  3.34s/it, commit_loss=1.08, loss=5.04, recon_loss=4.93]
9034.0MB @  Starting loop
9034.0MB @  distance = -2 * einsum(
9034.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
9034.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
9034.0MB @  distance = -2 * einsum(
9034.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
9034.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
9034.0MB @  End of loop
9034.0MB @  Starting loop
9034.0MB @  distance = -2 * einsum(
9034.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
9034.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
9034.0MB @  distance = -2 * einsum(
9034.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
9034.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
9034.0MB @  End of loop

frac_per_sample_entropy = 1.0
3856.0MB @  Starting loop
4656.0MB @  distance = -2 * einsum(
5056.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
5856.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
5856.0MB @  distance = -2 * einsum(
5856.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
5855.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
5829.0MB @  End of loop
6183.0MB @  Starting loop
6183.0MB @  distance = -2 * einsum(
6183.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7783.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7785.0MB @  distance = -2 * einsum(
8185.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
9785.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
9785.0MB @  End of loop
Epoch 1/280:   2%|▏         | 1/50 [00:03<02:46,  3.40s/it, commit_loss=1.09, loss=5.13, recon_loss=5.02]
10235.0MB @  Starting loop
10235.0MB @  distance = -2 * einsum(
10235.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
10235.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
10235.0MB @  distance = -2 * einsum(
10235.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
10235.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
10235.0MB @  End of loop
10227.0MB @  Starting loop
10227.0MB @  distance = -2 * einsum(
10227.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
10227.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
10227.0MB @  distance = -2 * einsum(
10227.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
10227.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
10227.0MB @  End of loop

Using fp16 training
frac_per_sample_entropy = 0.1
3599.0MB @  Starting loop
3999.0MB @  distance = -2 * einsum(
4199.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
4199.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
4199.0MB @  distance = -2 * einsum(
4199.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
4199.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
4190.0MB @  End of loop
5069.0MB @  Starting loop
5451.0MB @  distance = -2 * einsum(
5651.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
5632.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
5828.0MB @  distance = -2 * einsum(
6028.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
6028.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
6028.0MB @  End of loop
Epoch 1/280:   2%|▏         | 1/50 [00:03<02:38,  3.23s/it, commit_loss=1.13, loss=5.1, recon_loss=4.98]
6228.0MB @  Starting loop
6228.0MB @  distance = -2 * einsum(
6228.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
6228.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
6228.0MB @  distance = -2 * einsum(
6228.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
6228.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
6228.0MB @  End of loop
6235.0MB @  Starting loop
6235.0MB @  distance = -2 * einsum(
6235.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
6235.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
6235.0MB @  distance = -2 * einsum(
6235.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
6235.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
6235.0MB @  End of loop

frac_per_sample_entropy = 1.0
Epoch 1/280:   0%|          | 0/50 [00:00<?, ?it/s]
3756.0MB @  Starting loop
4156.0MB @  distance = -2 * einsum(
4354.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
4748.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
4747.0MB @  distance = -2 * einsum(
4747.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
4747.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
4747.0MB @  End of loop
5193.0MB @  Starting loop
5593.0MB @  distance = -2 * einsum(
5793.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
6592.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
6583.0MB @  distance = -2 * einsum(
6783.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7583.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7583.0MB @  End of loop
Epoch 1/280:   2%|▏         | 1/50 [00:03<02:39,  3.26s/it, commit_loss=1.09, loss=5.09, recon_loss=4.98]
7777.0MB @  Starting loop
7777.0MB @  distance = -2 * einsum(
7777.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7777.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7777.0MB @  distance = -2 * einsum(
7777.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7777.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7777.0MB @  End of loop
7785.0MB @  Starting loop
7785.0MB @  distance = -2 * einsum(
7785.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7785.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7785.0MB @  distance = -2 * einsum(
7785.0MB @  prob = (-distance * inv_temperature).softmax(dim = -1
7785.0MB @  per_sample_entropy = entropy(per_sample_probs).mean()
7785.0MB @  End of loop

@MarcusLoppe nice re: memory usage drop! i disabled autocast for LFQ and gaussian blur as well

@MarcusLoppe nice re: memory usage drop! i disabled autocast for LFQ and gaussian blur as well

Ah I think i didn't explained it right.
The autocast might have resolved the codebook where I needed to call ".to(dtype=torch.float16)":
codebook = self.bits_to_codes(bits).to(dtype=torch.float16)
But the gaussian_blur had the dtype value declared in the arange, since it's explicit declared I'm not sure if the autocast will work(maybe?).
Old:
distance = torch.arange(-half_width, half_width + 1, dtype = torch.float, device = device)
My 'fix'
distance = torch.arange(-half_width, half_width + 1, dtype = torch.float16, device = device)

@MarcusLoppe oh got it, reverted the gaussian blur
for quantization, i think it should be kept at f32 to be safe

@MarcusLoppe oh got it, reverted the gaussian blur
for quantization, i think it should be kept at f32 to be safe

Probably true, if you smooth the quantized values too much it will be very lossy compression.
It might not be worth it to use fp16 outside of the encoder & decoder, but I'm guessing the data needs to be kept in the same format or else some bad unexpected things might happens.
However the transformer can probably be trained in fp16 since the reduced probability in the next token doesn't seem to matter went 8/16 bits is used in LLMs.

@lucidrains
But this seems like a good solution, I think I'll close the issue since the checkpoint & reducing sample size seems to be working.
However I'm bit unsure about the reducing of the sample size since it just reduces the data it process and not a better way of processing it, but If you think that a smaller sample size if enough to build the codebook I believe you 😄 If not re-open this issue.

this isn't actually my idea
it is one used all the way back in StyleGAN2
they even have a scheme where for expensive auxiliary losses, they only apply it once every four iterations


vtk: unvendor some of the dependencies, fix on darwin

Description of changes
This prevents vtk from building its own dependencies from scratch, which allows to fix it on darwin.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[x] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Nice, I tried the same in https://github.com/NixOS/nixpkgs/pull/171934 but never managed to make it build.

Nice, I tried the same in #171934 but never managed to make it build.

Oh, would be nice to sometime rebase your work on top of this. I'd prefer to first review this as is first.

@GrahamcOfBorg build vtk_7
@GrahamcOfBorg build vtk_9
@GrahamcOfBorg build python3Packages.vtk

@GrahamcOfBorg f3d

@GrahamcOfBorg build f3d

The f3d build fails during configuring due to missing libraries but perhaps as with #178367 propagating some of these additional libraries from vtk_9 is the correct solution.

The f3d build fails during configuring due to missing libraries but perhaps as with #178367 propagating some of these additional libraries from vtk_9 is the correct solution.

Thank you for this summary. I agree, propagating the libraries is a solution. Although, it's a bit unusual that they are needed in runtime. One would think that VTK developers how to best use CMake, but it could be still a bug caused by lack of testing in environments with separated dependencies.

Sorry, I meant configurePhase -- this is a build time, not runtime, failure.

Maybe time to rebase this now that some other VTK-related issues are fixed?

Maybe time to rebase this now that some other VTK-related issues are fixed?

Would you like to give it a try?


Sorted Arrays

Hi, I am a contributor under the GGSOC 2023. I would like to share how to solve sorted arrays problem using different approaches like binary search,        two pointer method ,        cyclic sort ,      merge sort,        dynamic programing in  c++ . Please assign it to me.
please review it @aaryahjolia

I will provide the detailed explanation with different time complexities.

Hi @aaryahjolia , I am a GSSoC'23 contributor and would love to work on this issue.I have solved around 900 questions on leetcode(https://leetcode.com/code_hobby_2404/).I will provide all sorted array algorithms with several aproaches.

I would like to contribute to this issue. Please assign me.


Remove unnecessary initialized annotation.

The pod.alpha.kubernetes.io/initialized annotation is deprecated and the default in StatefulSet is true so it is unnecessary.
ref https://github.com/kubernetes/kubernetes/issues/41605

/lgtm

I'll just go ahead and merge this.


get_file behaves differently using SSHFileSystem vs LocalFileSystem

I've implemented the AbstractFileSystem in my code in order to direct my application either to the local file system or file system over SSH. Only I noticed that the behavior of get_file is different in both. I wrote this little script to test and demonstrate.
from fsspec.implementations.local import LocalFileSystem
from sshfs import SSHFileSystem

ssh_fs = SSHFileSystem(
    "localhost",
    username="foobar",
    password="foobar",
)
ssh_fs.get_file("/tmp/foobar", ".")

local_fs = LocalFileSystem()
local_fs.get_file("/tmp/foobar2", ".")

I would expect that calling get_file on either with similar parameters would result the copying of the requested file to the local folder. Only the LocalFileSystem implementation results in an error:
IsADirectoryError: [Errno 21] Is a directory: '/home/west/Research/sshfs/.'

The LocalFileSystem implementation requires a full file path:
local_fs.get_file("/tmp/foobar2", "./foobar2")

I seems to me that the implemenation of get_file in SSHFileSystem does not follow the fsspec API. Or am I missing something?

Hi @sguldemond !
Good catch! Indeed, it is an inconsistecy with LocalFileSystem. I'm not 100% sure how all other implementations behave though. My guess is that it probably depends on the underlying library that is used in particular implementation. One could possibly consider this an extended feature support, so I'm not sure if this is really worth going around and unifying across all implementations. This needs a bit more research to see if there is a more general solution that is feasible.


Update all db examples to ensure ts compiles

This should complete the list of examples. They should now all compile/build correctly. Thanks for the huge effort to get this going @andrewxhill!

In addition to being super handy for keeping docs/code in sync, I think this actually leads to better organization of docs and explanations. You need to discuss things when they are needed in code, which forces you to think seriously about how you present information.

Ship it!


Disabling particle collisions with leaf blocks

The falling leaf particles actually spawn inside the block they come from, which is no issue if they are going downwards, but if they are going sideways, they can collide with other leaf blocks, resulting in stuck leaves.
It's currently not a big issue since it happens rarely, but when wind is implemented, it makes a lot of sense to me that leaves should flow through leaf blocks.

I've got some partial success with this by slightly changing Entity.adjustMovementForCollisions().
It uses either Entity.adjustMovementForCollisions() or Entity.adjustSingleAxisMovementForCollisions().
adjustMovementForCollisions() is taking a stream of block collisions from world.getBlockCollisions(), which has a version taking a predicate, so I put in (blockState, blockPos) -> !LeafUtil.isLeafBlock(blockState.getBlock(), true), which did work!
However, adjustSingleAxisMovementForCollisions() has 0 configurability.
It calls calculatePushVelocity() which walks through the blocks(tates) in a loop that is truly hard to understand.
Interestingly enough, I can't figue out the real difference between the two methods, using either of them alone seems to work, but there has to be a reason they both exist.

Hey, I've been trying to improve my collision handling in my mod recently and have come across the same methods. I've been googling to try to understand what the loop in calculatePushVelocity does but I'm partially stumped too.
Possibly this gist I made can help you understand it.
I noticed that adjustMovementForCollisions (aka collideBoundingBoxLegacy) doesn't work properly when the entity is standing still (only adjustSingleAxisMovementForCollisions does) and visa versa.

Oh, thanks for the input!
It's been a while since I looked into this, I remember reading some vague descriptions of the algo and about how clever it is.
Not only is there the loop in calculatePushVelocity/collide but it is using calculateMaxDistance/collideX which is another non-trivial loop.
My guess it is using some optimization technique like sweep and prune (aka sort and sweep), which I vaguely remember as something like:
Looking into a fixed direction, sort the objects by how close their front face and back faces are and then walk through all objects in order - there then is a point where either the front or back faces are too far away to allow collisions, at which the iteration stops, skipping most collision tests.
I can't remember the exact details though.
For Falling Leaves I thought about the idea of temporarily modifying collision boxes (just before and after the particle's move method), though this sounds equally as terrible as modifying the collision code, considering it's just mod which just adds some particle effects. 😅

It's been a while since I looked into this, I remember reading some vague descriptions of the algo and about how clever it is.

Do you remember where you read these? I'd be interested in reading it

Do you remember where you read these? I'd be interested in reading it

No, and a quick search turned up nothing, not sure what search terms I used.
I remember having seen it in some forum and I think there was code too, but it wasn't yarn mapped.
I'll see if I can find it (or something) again - maybe looking up the MCP mappings could help?
I did have a look at the Spigot mappings but I couldn't find the calculatePushVelocity or calculateMaxDistance equivalents (they don't seem to be mapped).

Hm... can't seem to find anything, tried searching for the method names (Confusingly, MCP seems to have the exact same name for Entity.collideBoundingBox?), general "minecraft collision code", "how do minecraft collisions work" etc, searched the Fabric and SpongePowered discord but alas...
Anyway, I think that the stuff I read wasn't really that helpful, it was basically saying something along the lines of "there's 2 collision codes", "actually very clever" but nothing indepth to understand what's actually going on.
It probably makes more sense looking up (AABB) collision code optimizations and see if they match what Minecraft is using.
Searching for "minecraft like collision code" does give a few results (even video) about possible implementations.
Is the reason you're looking into the collision code related to https://github.com/Cadiboo/NoCubes?
Because if you want collisions to be as smooth as the terrain you likely have to rewrite it from scratch, because the basic assumption of only having AABB collisions is broken (and the outline boxes probably don't fully contain the collision mesh too).

Is the reason you're looking into the collision code related to https://github.com/Cadiboo/NoCubes?
Because if you want collisions to be as smooth as the terrain you likely have to rewrite it from scratch, because the basic assumption of only having AABB collisions is broken (and the outline boxes probably don't fully contain the collision mesh too).

Sorry for the late response
Yeah, that's why I'm doing it - I've been able to get the vanilla collisions system working work my smooth terrain but I've done it rather inefficiently (by generating a smooth mesh for every single block) and I'm now trying to optimise it (by replacing the vanilla algorithm with a hybrid algorithm that generates the collisions for an entire area).

I encountered the same bug, using the latest version of mod, the game version is 1.21fabric


[feature request] Vision / VNRecognizeTextRequest / recognitionLanguages

https://developer.apple.com/documentation/vision/vnrecognizetextrequest/recognitionlanguages?language=objc

Hello @louis030195,
Pushed missing api for recognize text request. Please, take a look.

Closing as completed. Feel free to reopen if needed.


UI - Search domain by attributes

Today Athenz UI only allows searching for the domain by its name.

Enhance the current search functionality to support search by - aws account, azure subscription id, product id, tag key, tag value, business service.
Existing ZMS API supports look up domain by different parameters - https://github.com/AthenZ/athenz/blob/master/core/zms/src/main/rdl/Domain.rdli#L34
e.g. https://:4443/zms/v1/domain?azure=mysubscription
Guide to start a local UI server - https://athenz.github.io/athenz/local_ui_development/
Expectations:

New code and corresponding unit tests with 100% code coverage

@abvaidya Can I claim this?


Docker install and mmlabs fail on git clone

Something seems to be off with "git clone" not working during the docker build phase. I'm told not to expect to be able to depend on that since docker builds don't come with ssh keys and github doesn't like anonymity.
I tried alternative installs for OpenMMLabs, like OpenMim and "mim install" commands, but that fails as well with a timeout error below:
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fe28d672910>: Failed to establish a new connection: [Errno 111] Connection refused'))': /openmim/

Is this something that someone else has experienced and solved?

A few months ago github stopped accepting clone requests without keys, so git clone won't work anymore in a docker repo.
I got this to eventually work by using "sudo pip install openmim" and then "sudo mim install [packages]" for the OpenMMLabs pieces. The "sudo" is important here, which is odd since dockers are running as root in the first place, but I disgress.


Fix R scanner behavior for array values

Fixes #1247

TreeScanner api visits all values at once when calling accept(Tree) so aggregate the symbols within the scanner itself
Added new unit tests

Seems like the connectedCheck task is failing (Its passing locally for me on sdk 26 emulator)

Instrumentation tests worked on an api 19 emulator as well :| . Maybe travis ci has some flakiness?

rekicking travis ci


Use vite JS API to build svelte app

This PR imports the build function from vite and uses it directly instead of executing a shell command.
It's the second step to solve #20 after #35 (Use npm workspaces PR)
Why do #35 first?
The requirement was to be able to run vite JS functions from generator.js, to import vite from there, we needed to have one node_modules file in the same place.
Svelte is just a plugin for vite, running vite build is the same as building the svelte app, by using process.chdir to the svelte directory, it picks the correct vite.config.ts file and the correct app.html input file which is used by rollup to bundle our output app.

@hammady I think it needs some changes. Please copy the necessary directories if they do not exist


[base-ui] Fix React spread key warning in test

Fixes a React 18.3 warning in a useAutocomplete test:
A props object containing a "key" prop is being spread into JSX

@DiegoAndai same!


Need to provide CURIE list for NIFSTD

The web service call getCuriePrefixes returns an empty list:
http://matrix.neuinfo.org:9000/scigraph/vocabulary/prefixes?callback=fn
http://matrix.neuinfo.org:9000/scigraph/docs/#!/vocabulary/getCuriePrefixes

The list is empty because NIF hasn't defined any CURIE prefixes (probably because NIF hasn't ever used CURIES). This can be configured by adding appropriate mappings to the application configuration.

The prefixes were specified and a request to re-load NIFSTD was sent before the break. Let's leave this item open until we can confirm this web service is returning the correct content after re-loading NIFSTD.

Wait - there are two separate issues here. One is the categories and one is the CURIEs. Which is this ticket about?

Sorry, this ticket is for the "getCuriePrefixes" call. Is this a different mapping/configuration file than what is needed for "getCategories"? If yes, can you point me to details of what is needed for the "getCuriePrefixes" call?

Yes - the CURIEs are separate from the categories. Monarch's CURIE configuration looks something like this:
curies:
  http://purl.obolibrary.org/obo/DOID_ : DOID
  http://purl.obolibrary.org/obo/ORPHANET_ : ORPHANET
  ...

Refer to Jira NIF-9399 for discussion about SOLR and format using commas.

How is NIF-9399 related to this ticket?

I believe Chris M. mentioned ticket NIF-9399 in relation to this one.

Not this one. NIF-9399 is about how lists are modeled in the federation solr views, which may have an impact on how closure indexing is done for views such as the one mentioned on the arch call, I think the resources view.


Write mode fixes

Section labels are now passed through "edit this comment" from review page (bugfix in eregs/notice-and-comment#308)
Write mode page loads at top of screen (setting scrollTop(0)). It was attempting to scroll down to a lower section before.

Looks good!