WIP: Move install routes to chi

As title.

Codecov Report

Merging #13558 (54edc60) into master (75ebf7c) will decrease coverage by 0.03%.
The diff coverage is 9.55%.

@@            Coverage Diff             @@
##           master   #13558      +/-   ##
==========================================
- Coverage   42.20%   42.17%   -0.04%
==========================================
  Files         696      697       +1
  Lines       76541    76637      +96
==========================================
+ Hits        32307    32319      +12
- Misses      38932    39027      +95
+ Partials     5302     5291      -11

Impacted Files
Coverage Δ

cmd/web.go
0.00% <0.00%> (ø)

modules/context/context.go
47.29% <0.00%> (-9.77%)
:arrow_down:

routers/install.go
0.00% <0.00%> (ø)

routers/routes/chi.go
43.26% <ø> (+2.59%)
:arrow_up:

routers/routes/macaron.go
92.66% <ø> (+0.72%)
:arrow_up:

services/mailer/mail.go
60.21% <0.00%> (-1.08%)
:arrow_down:

routers/init.go
47.57% <50.00%> (-4.50%)
:arrow_down:

modules/translation/translation.go
51.85% <51.85%> (ø)

modules/util/timer.go
42.85% <0.00%> (-42.86%)
:arrow_down:

modules/queue/workerpool.go
58.77% <0.00%> (-1.23%)
:arrow_down:

... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 75ebf7c...54edc60. Read the comment docs.

Blocked by #13857


Option to hide dynisland without killing daemon

Hey,
it would be cool to allow toggling the visibility of dynisland either either via an command or by specific Signals that may be sent to dynisland.
My current solution is to always kill the daemon and that's... meh
What do you think? Can this be easily implemented?

Wow that was fast! But this still makes dynisland consume all clicks/buttons/...
I think eww had the exact same issue, but i can't find the issue right now.

I added dynisland layout hide and dynisland layout show.
When a window is hidden it gets unregistered from the compositor, so it doesn't consume events.

 Works like a charm. Thank you and keep going!!!


Use correct lint function

There were places where we were using compileAndLint that we should use compileAndLintWithContext, other that use lint that could use compileAndLint and similar cases. This PR fixes a lot of them (probably this is not exhaustive but at least it's better than before).
This PR will also help to move from @RequireTypeSolving as an annotation to an interface.

Thanks for doing this @BraisGabin


Error starting stopped VM: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: Could not read keymap file: 'en-us'')

General information

OS: Linux
Hypervisor: KVM
Did you run crc setup before starting it (Yes)?
Running CRC on: VM

CRC version
[crcuser@ecs-openshift ~]$ crc version
CodeReady Containers version: 1.19.0+94b0362
OpenShift version: 4.6.3 (embedded in executable)

CRC status
# Put `crc status --log-level debug` output here

CRC config
[crcuser@ecs-openshift ~]$ crc status --log-level debug
DEBU CodeReady Containers version: 1.19.0+94b0362
DEBU OpenShift version: 4.6.3 (embedded in executable)
Checking file: /home/crcuser/.crc/machines/crc/.crc-exist
CRC VM:          Stopped
OpenShift:       Stopped
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     15.32GB
Cache Directory: /home/crcuser/.crc/cache

Host Operating System
[crcuser@ecs-openshift ~]$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Steps to reproduce

crc delete
crc setup
crc start

Expected
crc start successfully
Actual
Error starting stopped VM: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: Could not read keymap file: 'en-us'')
Logs
Before gather the logs try following if that fix your issue
DEBU CodeReady Containers version: 1.19.0+94b0362
DEBU OpenShift version: 4.6.3 (embedded in executable)
DEBU Total memory of system is<PHONE_NUMBER>2 bytes
DEBU No new version available. The latest version is 1.19.0
DEBU Couldn't set key CPE_NAME, no corresponding struct field found
DEBU Couldn't set key , no corresponding struct field found
DEBU Couldn't set key CENTOS_MANTISBT_PROJECT, no corresponding struct field found
DEBU Couldn't set key CENTOS_MANTISBT_PROJECT_VERSION, no corresponding struct field found
DEBU Couldn't set key REDHAT_SUPPORT_PRODUCT, no corresponding struct field found
DEBU Couldn't set key REDHAT_SUPPORT_PRODUCT_VERSION, no corresponding struct field found
DEBU Couldn't set key , no corresponding struct field found
INFO Checking if podman remote executable is cached
DEBU Currently podman remote is not supported
INFO Checking if goodhosts executable is cached
DEBU goodhost executable already cached
INFO Checking minimum RAM requirements
DEBU Total memory of system is<PHONE_NUMBER>2 bytes
INFO Checking if running as non-root
INFO Checking if Virtualization is enabled
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo
DEBU CPU virtualization flags are good
INFO Checking if KVM is enabled
DEBU Checking if /dev/kvm exists
DEBU /dev/kvm was found
INFO Checking if libvirt is installed
DEBU Checking if 'virsh' is available
DEBU 'virsh' was found in /usr/bin/virsh
DEBU Checking 'virsh capabilities' for libvirtd/qemu availability
DEBU Running 'virsh capabilities'
DEBU Found x86_64 hypervisor with 'hvm' capabilities
INFO Checking if user is part of libvirt group
DEBU Checking if current user is part of the libvirt group
DEBU Running '/usr/bin/groups crcuser'
DEBU Current user is already in the libvirt group
INFO Checking if libvirt daemon is running
DEBU Checking if libvirtd service is running
DEBU Running 'systemctl status virtqemud.socket'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit virtqemud.socket could not be found.
DEBU virtqemud.socket is not running
DEBU Running 'systemctl status libvirtd.socket'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit libvirtd.socket could not be found.
DEBU libvirtd.socket is not running
DEBU Running 'systemctl status virtqemud.service'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit virtqemud.service could not be found.
DEBU virtqemud.service is not running
DEBU Running 'systemctl status libvirtd.service'
DEBU libvirtd.service is running
INFO Checking if a supported libvirt version is installed
DEBU Checking if libvirt version is >=3.4.0
DEBU Running 'virsh -v'
INFO Checking if crc-driver-libvirt is installed
DEBU Checking if crc-driver-libvirt is installed
DEBU Running '/home/crcuser/.crc/bin/crc-driver-libvirt version'
DEBU Found crc-driver-libvirt version 0.12.12
DEBU crc-driver-libvirt is already installed
INFO Checking if libvirt 'crc' network is available
DEBU Checking if libvirt 'crc' network exists
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU Checking if libvirt 'crc' definition is up to date
DEBU Running 'virsh --connect qemu:///system net-dumpxml --inactive crc'
DEBU libvirt 'crc' network has the expected value
INFO Checking if libvirt 'crc' network is active
DEBU Checking if libvirt 'crc' network is active
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU libvirt 'crc' network is already active
INFO Checking if NetworkManager is installed
DEBU Checking if 'nmcli' is available
DEBU 'nmcli' was found in /usr/bin/nmcli
INFO Checking if NetworkManager service is running
DEBU Checking if NetworkManager.service is running
DEBU Running 'systemctl status NetworkManager'
DEBU NetworkManager.service is already running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
DEBU Checking NetworkManager configuration
DEBU NetworkManager configuration is good
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
DEBU Checking dnsmasq configuration
DEBU dnsmasq configuration is good
Checking file: /home/crcuser/.crc/machines/crc/.crc-exist
Found binary path at /home/crcuser/.crc/bin/crc-driver-libvirt
Launching plugin server for driver libvirt
Plugin server listening at address <IP_ADDRESS>:41287
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(crc) Calling .GetBundleName
(crc) Calling .GetState
(crc) DBG | Getting current state...
(crc) DBG | Fetching VM...
INFO Starting CodeReady Containers VM for OpenShift 4.6.3...
DEBU Updating CRC VM configuration
(crc) Calling .UpdateConfigRaw
(crc) DBG | Updating memory size to 13312 kiB
(crc) DBG | Setting memory to 13312 MB
(crc) DBG | disk image capacity is already<PHONE_NUMBER>4 bytes
(crc) Calling .GetConfigRaw
(crc) Calling .Start
(crc) DBG | Starting VM crc
(crc) DBG | Validating network
(crc) DBG | Validating storage pool
(crc) Failed to start: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: Could not read keymap file: 'en-us'')
Making call to close driver server
(crc) Calling .Close
(crc) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Error starting stopped VM: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: Could not read keymap file: 'en-us'')

/var/log/libvirt/qemu/crc.log should give more details as to what's happening.  You can use subscription-manager to register.

Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
qemu-firmware-jenkins                                                                                | 2.9 kB  00:00:00
Package 10:qemu-kvm-ev-2.12.0-44.1.el7_8.1.x86_64 already installed and latest version
Nothing to do

I delete qemu-kvm-ev and reinstall, It's works now :)


Inconsistent Results During Testing of DeformableFeatureAggregation Operator

We are experiencing inconsistent results during the testing phase of the DeformableFeatureAggregation (DFA) operator. Each test run yields different results.
multi-gpu test
multi-gpu first test

multi-gpu second test

single-gpu test
single-gpu first test

single-gpu second test

We have also noticed this phenomenon. There is indeed an issue with the evaluation results not being accurately reproducible, but we have not yet analyzed it.

Hi @linxuewu , is there any update on this issue?


[Frontend][PyTorch][Bugfix] Ignore Cuda in PyTorch version number when comparing versions

I encountered a quirk today when using the PyTorch importer. My local PyTorch version is 1.5.1 and it uses Cuda (10.1 on my system), so torch.__version__ returns 1.5.1+cu101.
When I used the importer I encountered the following error:
  File "/home/sslyu/tvm/python/tvm/relay/frontend/pytorch.py", line 4064, in from_pytorch
    _run_jit_passes(graph, enable_lower_all_tuples)
  File "/home/sslyu/tvm/python/tvm/relay/frontend/pytorch.py", line 3637, in _run_jit_passes
    torch._C._jit_pass_onnx_function_substitution(graph)
AttributeError: module 'torch._C' has no attribute '_jit_pass_onnx_function_substitution'

That is because PT version 1.5.1 doesn't have that function. In principle, the importer should check for that and not use that function for versions that don't have it, which is the logic here: https://github.com/apache/tvm/blob/119afda6344785aee5cf1729eec30624ac068f33/python/tvm/relay/frontend/pytorch.py#L3628-L3642
However, the "+cu101" in the version number causes is_version_greater_than to return true. This change disregards the "+cu" code in the version number for that comparison.

Requesting review from @masahi because he has edited the pytorch_utils file before.


iOS App: TestFlight 1.3(39)

Issues to test:

[ ] [phetsims/phet-ios-app#497](Convert Filter from Delegation to Notification)
[ ] [phetsims/phet-ios-app#492](Convert Heart and Chevron assets/elements to Templates)
[ ] [phetsims/phet-ios-app#487](Improve Update System: Localization)
[ ] [phetsims/phet-ios-app#475](Table View Based Filtering)
[ ] [phetsims/phet-ios-app#473](Localize Category Labels)
[ ] [phetsims/phet-ios-app#344](Inconsistent collection view scroll position)

@KatieWoe build 39 should be available in TestFlight now or within the hour.

@ariel-phet should this be the same priority?

[x] https://github.com/phetsims/phet-ios-app/issues/506
[x] https://github.com/phetsims/phet-ios-app/issues/505
[ ] https://github.com/phetsims/phet-ios-app/issues/504
[ ] https://github.com/phetsims/phet-ios-app/issues/503
[ ] https://github.com/phetsims/phet-ios-app/issues/502

These were given the label later

QA is done with (39).  Issues either added, altered due to continued issues, left due to other issues preventing testing, or closed.

QA is done with (39).


Sort github list

This issue was made by minemobs#6904<PHONE_NUMBER>88721930) on the Pépite d'or Bot server.
Issue:
846048803554852904

Done https://github.com/TheSimpleTeam/SimpleBot/commit/e0b6fb51445585c677b9de6335b1e4c8c290fc64


Call the connect() callback if reconnecting

great catch! Thanks!

Actually... I've reverted this via https://github.com/taskrabbit/node-resque/pull/148.
Can you explain more about how this is valuable?  With your change, if you have code like the below, every time you reconnect "HERE" would be logged.  That means if you have any code following after your connect line, it would be executed again and again.  Adding an event here reconnect would be a safer pattern I think.
var queue = new NR.queue({connection: connectionDetails}, jobs);
queue.connect(function(){
  console.log("HERE");
});

Sorry for the false alarm. While writing a reduced test case for this reply, I discovered that I was erroneously keeping the connection open, and so my callback to queue.connect was never being called. A reconnect event certainly would be useful.

feel free to add one via a new PR!


I used draco(-d) compression, why put my custom data to the loss. There are two files in the attachment that have not been compressed. The customized data in 100.glb still exists after compression, but the customized data in 102.glb is lost.

testfile.zip
I used draco(-d) compression, why put my custom data to the loss. There are two files in the attachment that have not been compressed. The customized data in 100.glb still exists after compression, but the customized data in 102.glb is lost.
The custom data is placed in the first and second elements of the bufferview array.

@sui0312feng there are some validation issues with 102.glb. Is it possible to fix those and then try again?  Initial: 'IndexBuffer', new: 'VertexBuffer'.",
                "severity": 0,
                "pointer": "/meshes/0/primitives/0/attributes/_FEATURE_ID_0"
            },
            {
                "code": "MESH_PRIMITIVE_ATTRIBUTES_ACCESSOR_UNSIGNED_INT",
                "message": "Mesh attributes cannot use UNSIGNED_INT component type.",
                "severity": 0,
                "pointer": "/meshes/0/primitives/0/attributes/_FEATURE_ID_0"
            },
            {
                "code": "BUFFER_VIEW_TARGET_OVERRIDE",
                "message": "Override of previously set bufferView target or usage. Initial: 'IndexBuffer', new: 'VertexBuffer'.",
                "severity": 0,
                "pointer": "/meshes/0/primitives/1/attributes/_FEATURE_ID_0"
            },
            {
                "code": "MESH_PRIMITIVE_ATTRIBUTES_ACCESSOR_UNSIGNED_INT",
                "message": "Mesh attributes cannot use UNSIGNED_INT component type.",
                "severity": 0,
                "pointer": "/meshes/0/primitives/1/attributes/_FEATURE_ID_0"
            },

Modified the above problem, the compression still lost the custom data
No_0.zip

@sui0312feng there are some validation issues with 102.glb. Is it possible to fix those and then try again?  Initial: 'IndexBuffer', new: 'VertexBuffer'.",
                "severity": 0,
                "pointer": "/meshes/0/primitives/0/attributes/_FEATURE_ID_0"
            },
            {
                "code": "MESH_PRIMITIVE_ATTRIBUTES_ACCESSOR_UNSIGNED_INT",
                "message": "Mesh attributes cannot use UNSIGNED_INT component type.",
                "severity": 0,
                "pointer": "/meshes/0/primitives/0/attributes/_FEATURE_ID_0"
            },
            {
                "code": "BUFFER_VIEW_TARGET_OVERRIDE",
                "message": "Override of previously set bufferView target or usage. Initial: 'IndexBuffer', new: 'VertexBuffer'.",
                "severity": 0,
                "pointer": "/meshes/0/primitives/1/attributes/_FEATURE_ID_0"
            },
            {
                "code": "MESH_PRIMITIVE_ATTRIBUTES_ACCESSOR_UNSIGNED_INT",
                "message": "Mesh attributes cannot use UNSIGNED_INT component type.",
                "severity": 0,
                "pointer": "/meshes/0/primitives/1/attributes/_FEATURE_ID_0"
            },

Modified the above problem, the compression still lost the custom data。
No_0.zip

@sui0312feng thanks for sending over the new file.
It still looks like there are problems. No_0.glb is on the left, 100.glb is on the right.

properties should be an object (I think this is the main problem)
_class should be class
_name should be name

thank you!  you are right!  issue was settled !


Fixed width of CapsuleShape2D::get_rect + set center to center of shape

The width of the rect was only half of the width of the shape, and the 0;0 coord was at the top left of the rect.
Now the width properly matches the width of the shape, and the 0;0 coord is at the center of the shape. It should match the behavior of Godot 3.X (https://github.com/godotengine/godot/blob/3.x/scene/resources/capsule_shape_2d.cpp#L93-L99).
I haven't found a test file for CapsuleShape2D, I don't know if I have to create one or if it should be written in an existing file, or something else.

Fixed it, thanks for notifying me.
I started using godot last week and it seems to be a really great product, I hope I'll be able to contribute more in the future!


Split server services

1.split the img api in the ts server to two apis (one for detect hands,one for detect sign +hands cus detect hands is cheap anyway)
2.updated the signifyService.py ==> now its recive request (hands, hands+sign) and the img as input
if request = hands its return only the rectangle of the hands
if request = hands+sign its return the reactangle of the hands + detecing the sign
3. updated the reactNative  to support the two apis
now the frameProcessorFps property of signifyCamera works like that:
the first and the last frame it will detect hands + sign (which is expensive)
and in the middle frames it will detect only the reactangle/
example: frameProcessorFps = 5
first frame it will detect hands + sign , 3 frames letter only hands , last frame hands + sign again
**** in the future i will allow to send more hands + sign requests by paremter and not fix to the first and last frame.
fix #65

Maybe marge only the camera commit for now, and we can decide which way is better after we see how the 2 seperate services work

Maybe only marge the camera commit for now, and we can decide which way is better after we see how the 2 seperate services work

hmm. i dont think its that big deal we can do it later seperate to two pythons services
anyway you can use two pools now
just for the first pool you will send the request DetectType.Hands
and the second one DetectType.signHands.
so the only diffrence is that now the pyProcess assume the the python file he runs
recive request and data (which i think is more maintablity ),
anyway the changes in the app are counted on the changes in the server (two new apis and so on).
so i think its better to merge this one (the two pools is possible anyway)
and if you want to make two python files one for hands recognize and one for sign recognize
with out request we can just send request = "",
or remove the request when we create the two pools
anyway i thogut of it and i am not sure if running one pool on the cpu is such good idea
cus on the cpu hand recognize can take about 100ms or even more and we want to use it much faster
so we can do atlist 12 of it in a second.

Maybe only marge the camera commit for now, and we can decide which way is better after we see how the 2 seperate services work

hmm. i dont think its that big deal we can do it later seperate to two pythons services anyway you can use two pools now just for the first pool you will send the request DetectType.Hands and the second one DetectType.signHands.
so the only diffrence is that now the pyProcess assume the the python file he runs recive request and data (which i think is more maintablity ), anyway the changes in the app are counted on the changes in the server (two new apis and so on). so i think its better to merge this one (the two pools is possible anyway) and if you want to make two python files one for hands recognize and one for sign recognize with out request we can just send request = "", or remove the request when we create the two pools anyway i thogut of it and i am not sure if running one pool on the cpu is such good idea cus on the cpu hand recognize can take about 100ms or even more and we want to use it much faster so we can do atlist 12 of it in a second.

I built my commit on your commit (the commit that changed the camera-app) so it works good I think, ( I copied the structure that you had for the signifyApi too ).
Anyway - I tried it on my computer (one service running on gpu and 5 on cpu) and it works great for me - much faster than alot of requests for the gpu - check my PR out and tell me what you think (tried it with the fetch_debug.js script - not with the phone)
I'm not happy with the signifyService in general - I think its a bit messy - for now I just use the useCuda to decide if I want to detect the sign or not.. but I think that neither this solution nor the request string are particularly clean.

this pull request is dead
and #71  replace it


Some aggregations do not work on Store-Api /product/{productId}

PHP Version
8.2
Shopware Version
<IP_ADDRESS>
Affected area / extension
Platform(Default)
Expected behaviour
When i POST this request to /product/01912c3fb71e78b6a31711fd18523d14
{ 	"associations": { 		"unit":  {} 	}, 	"includes": { 		"product": ["unit"] 	} }
I would expect the unit object to be included.
Actual behaviour
unit is null
How to reproduce
Create a product (in my case with variations). Add configure a unit to it.
Then do the request above.
Interestingly the exact same request to POST /products
{ 	"ids": ["01912c3fb71e78b6a31711fd18523d14"], 	"associations": { 		"unit": {} 	}, 	"includes": { 		"product": ["unit"] 	} }
Works and i get the unit object back. The same as above is also true for "deliveryTime".
Note: I use the docker setup from here: https://github.com/shopware/docker

[public] Automated response: This issue is linked to the internal issue https://shopware.atlassian.net/browse/NEXT-37717.
[created from NEXT-37717, comment 487133]


feat: override platform wsl with linux during update command

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

v1.1.0
npm package (@latest dist-tag)
GitHub release
v1.1.0
v1.1.0

Your semantic-release bot :package::rocket:


Introduce a setting to hide embeddings

Related product team resources: PRD (internal only)
Related product discussion: https://github.com/orgs/meilisearch/discussions/677
Motivation

Improve AI search performance
Usage

Introduce a setting to hide embeddings
https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697?pvs=74
TODO

Reminders when modifying the Setting API

[ ] Ensure the new setting route is at least tested by the test_setting_routes macro
[ ] Ensure Analytics are fully implemented

[ ] /settings/my-new-setting configurated in the make_setting_routes macro
[ ] global /settings route configurated in the update_all function

[ ] Ensure the dump serializing is consistent with the /settings route serializing, e.g., enums case can be different (camelCase in route and PascalCase in the dump)

Special cases when adding a setting for an experimental feature

[ ] ⚠ API stability: The setting does not appear on the main settings route when the feature has never been enabled (e.g. mark it Unset when returned from the index in this situation. See an example)
[ ] The setting cannot be set when the feature is disabled, either by the main settings route or the subroute (see validate_settings function)
[ ] If possible, the setting is reset when the feature is disabled (hard if it requires reindexing)

Impacted teams

@meilisearch/docs-team @meilisearch/integration-team

After discussing with the team, we finally choose to hide by default, so to implement only https://github.com/meilisearch/meilisearch/issues/4607
Of course, we will implement a setting if needed for users at some point.


out of gpu memory at the end closed to converge...

I used transformer in Chinese to English translation task..
all the time it was running normally.. but after about 300000 iters, it always ran out of gpu memory when running several iters.
my configuration was all the same..
except this, all is perfect..
Does anybody has some ideas?
the machine is 4 * K80..
and tensor2tensor version is 1.4.1..
my training configuration is like below:
@registry.register_hparams
def translate_any_c2e_hparams():
hparams = transformer.transformer_big_single_gpu()
hparams.optimizer_adam_beta2 = 0.999
hparams.batch_size = 2000
hparams.learning_rate_warmup_steps = 2000
hparams.learning_rate = 0.05
return hparams

I think this is a duplicate of #581

I think there's difference that, when after 300,000 iterators training, it will crash soon (maybe no more than 20,000 iterators) once I restart it. But it was running normally before 300,000 iterators, like 7 days, and during this process anytime you restart it, it would run normally until 300,000 iterators..
Is there any special operation when training process close to converge ?

I agree 300k steps (7 days) is quite a long time before OOM crash, but I still think this is exactly the same issue as #581.
So far my longest time before OOM was two and half days training (150k steps, with batch_size=2300 and 4 GPUs, thus after 150k2.3k4=1.38 gigatokens). I restarted the training from the last checkpoint and it crashed after 11k steps (282 minutes). I restarted again (from a checkpoint saved 42 before the crash) and it crashed again after 11k steps (282 minutes).  I think this is caused by a particular datapoint (mischievous batch) in the training data or a sequence of mischievous batches. I don't think the OOM depends on whether the training is "close to converged", but maybe there is a relationship to the training time (the initial 150k steps is more than one epoch, so the mischievous batch has been processed for the first time without any OOM). My wild guess is that this may be related to the LazyAdam, which does not allocate momentum weights for tokens which were not seen so far.
After the third crash, I decreased the batch size to 2250 and resumed training, and it crashed after 311 minutes. Then I decreased the batch size to 2200, resumed and it finally trained without any OOM.

I experienced both cases @annisamansa and @martinpopel mentioned: (1) OOM shortly or 2.5 days of training ans (2) OOM after a week. Setting hparams.symbol_modality_num_shards = 1 helped me with OOMs where training actually works but then unexpectedly crashes.

Closing in favor of #581


Use optional client output location when it exists

The import path in the generated output assumes the Prisma client will always exist in the default location. But the Prisma client config allows users to specify a different output location:
generator client {
  provider = "prisma-client-js"
  output   = "../src/generated/client"
}

This change checks for the optional output value and uses it in the generated import path instead if it exists. Otherwise it defaults to the current behavior.

Thanks for the PR! Released in 0.13.0


TTNN Generic MLIR Op

TTIR Generic
We want to start building out the compiler side of lowering to a generic op.  As a primer please refer to the linalg generic op, ttir follows the same structure: https://mlir.llvm.org/docs/Dialects/Linalg/#linalggeneric-linalggenericop
./build/bin/ttmlir-opt --ttir-generic test/ttmlir/Dialect/TTIR/test_generic.mlir:
#any_device = #tt.operand_constraint<dram|l1|scalar|tile|any_device|any_device_tile>
#map = affine_map<(d0, d1) -> (d0, d1)>
#parallel = #tt.iterator_type<parallel>
module attributes {torch.debug_module_name = "_lambda", tt.system_desc = #tt.system_desc<[{arch = <wormhole_b0>, grid = 8x8, l1_size = 1048576, num_dram_channels = 12, dram_channel_size = 1048576}], [0], [<pcie|host_mmio>], [<0, 0, 0, 0>]>} {
  func.func @forward(%arg0: tensor<64x128xf32>, %arg1: tensor<64x128xf32>) -> tensor<64x128xf32> {
    %0 = tensor.empty() : tensor<64x128xf32>
    %1 = "ttir.generic"(%arg0, %arg1, %0) <{grid = #tt.grid<1x1>, indexing_maps = [#map, #map, #map], iterator_types = [#parallel, #parallel], operandSegmentSizes = array<i32: 2, 1>, operand_constraints = [#any_device, #any_device, #any_device]}> ({
    ^bb0(%arg2: tensor<64x128xf32>, %arg3: tensor<64x128xf32>, %arg4: tensor<64x128xf32>):
      %2 = "ttir.kernel"(%arg2, %arg3, %arg4) <{kind = @eltwise, op = @mulitply, operandSegmentSizes = array<i32: 2, 1>}> : (tensor<64x128xf32>, tensor<64x128xf32>, tensor<64x128xf32>) -> tensor<64x128xf32>
      "ttir.yield"(%2) : (tensor<64x128xf32>) -> ()
    }) : (tensor<64x128xf32>, tensor<64x128xf32>, tensor<64x128xf32>) -> tensor<64x128xf32>
    return %1 : tensor<64x128xf32>
  }
}

This op is composed of 2 primary structural components:

The top level ttir.generic this has attributes:

indexing_maps: a list of AffineMapAttr, one AffineMapAttr per each input and output view.
iterator_types:  an ArrayAttr specifying the type of the enclosing loops. Each element of the list represents and iterator of one of the following types: parallel, reduction, window.
These 2 attributes together describe how the operation over the tensor can legally be parallelized / divided.  For this issue, let's only consider iterator_type, "parallel", which mean we trivially visit each tensor element in isolation, i.e. element wise.  From these attributes alone I think we can completely infer the data movement for the op, in the case of element wise there is none, but it'll also tell the optimizer how the tensors are legally allowed to be parallelized on a grid.

The inner region, we have a lambda-like body with a basic block named ^bb0.  I like to think of this as the high level implementation of a kernel, and is effectively what each core is executing in parallel.

"ttir.kernel": This is an opaque op, can be thought of like an FFI call to some kernel that's hand written somewhere and will be linked in.  I think this should be the first step, we'll have kernels written by hand in C++ that conform to an interface such that the compiler can autogenerate the data movement to bring the data to it.
"ttir.yield": Nothing super interesting here, mlir requires us to define terminator ops to end a basic block.

Lowering to TTNN Generic
Ok, so there actually has already been a decent amount of POC done demonstrating the lowering of a ttir.generic op, it's just that it was done inside of the TTMetal dialect, but I think many of the same concepts will carry over to the ttnn version.
Again let's look at a simple test, we can run a TTMetal dialect test:
./build/bin/ttmlir-opt --ttir-generic --ttir-layout --ttir-generic-region-operands-to-memref --ttir-allocate --convert-ttir-to-ttmetal test/ttmlir/Dialect/TTMetal/simple_multiply.mlir

This result is quite verbose, so I only stripped out the "generic" op that's called "ttmetal.dispatch" in the TTMetal dialect.
    %5 = "ttmetal.dispatch"(%1, %3, %4) <{core_ranges = [#ttmetal.core_range<0x0, 1x1>, #ttmetal.core_range<0x0, 1x1>, #ttmetal.core_range<0x0, 1x1>], operandSegmentSizes = array<i32: 2, 1>, operand_cb_port_mapping = [0, 1, 2], threadTypes = [#ttkernel.thread<noc0>, #ttkernel.thread<noc1>, #ttkernel.thread<tensix>]}> ({
    ^bb0(%arg2: !ttkernel.cb<0, 0, memref<64x128xf32, #l1_>>, %arg3: !ttkernel.cb<0, 1, memref<64x128xf32, #l1_>>, %arg4: !ttkernel.cb<0, 2, memref<64x128xf32, #l1_>>):
      "ttkernel.cb_push_back"(%arg2) : (!ttkernel.cb<0, 0, memref<64x128xf32, #l1_>>) -> ()
      "ttkernel.return"() : () -> ()
    }, {
    ^bb0(%arg2: !ttkernel.cb<0, 0, memref<64x128xf32, #l1_>>, %arg3: !ttkernel.cb<0, 1, memref<64x128xf32, #l1_>>, %arg4: !ttkernel.cb<0, 2, memref<64x128xf32, #l1_>>):
      "ttkernel.cb_push_back"(%arg3) : (!ttkernel.cb<0, 1, memref<64x128xf32, #l1_>>) -> ()
      "ttkernel.return"() : () -> ()
    }, {
    ^bb0(%arg2: !ttkernel.cb<0, 0, memref<64x128xf32, #l1_>>, %arg3: !ttkernel.cb<0, 1, memref<64x128xf32, #l1_>>, %arg4: !ttkernel.cb<0, 2, memref<64x128xf32, #l1_>>):
      "ttkernel.builtin"(%arg2, %arg3, %arg4) <{kind = @eltwise, op = @mulitply}> : (!ttkernel.cb<0, 0, memref<64x128xf32, #l1_>>, !ttkernel.cb<0, 1, memref<64x128xf32, #l1_>>, !ttkernel.cb<0, 2, memref<64x128xf32, #l1_>>) -> ()
      "ttkernel.return"() : () -> ()
    }) : (tensor<64x128xf32, #layout1>, tensor<64x128xf32, #layout1>, tensor<64x128xf32, #layout1>) -> tensor<64x128xf32, #layout1>

So breaking this down again, it has a fairly similar structure to ttir.generic except now it's been decomposed into 3 regions, corresponding to 3 threads, noc0, noc1 and tensix.  Notice also these regions have been decomposed into the TTKernel dialect which strives to model as closely as possible the metal compute + dataflow programming model.
TTKernel also supports going through the EmitC dialect, so it is expected that they will be converted to C++ and then passed into the TTNN runtime interface to be compiled by the GCC toolchain.
Some things to call out:

Ops from the TTKernel dialect should be enforced to only be legal inside of regions of ttmetal.dispatch and ttnn.generic.
ttkernel.cb<0, 0, memref<64x128xf32, #l1_>>: Metal uses circular buffers (CBs) for sharing L1 data between risc threads.  Here CBs replace all the tensor arguments and represent the working shard of the tensor for that core.  The embedded memref defines the shape of the tensor shard for each core.  Generally noc threads will be pushing data to inputs tensor's CB, and tensix thread will be popping from it, doing work, and then pushing to the output CB.
"ttkernel.builtin": Just like ttir.kernel, this is an FFI function call to some kernel loop that's been hand written.  For now the compiler could just pattern match from a set of hand written kernel functions based on the kind and op attributes and then inline the function during C++ generation.

Tasks
So breaking down the above into tasks:

#202
#203
#204
Coordinate with issue #111

Writing an update on how far we have come with this.
ttnn progress
As a starting point, @mbezuljTT devised and @pjanevskiTT continued to work on https://github.com/tenstorrent/tt-mlir/issues/111, where they implemented
struct GenericOp {
    static Tensor operator()(
        const std::vector<Tensor> &input_tensors,
        Tensor& output_tensor,
        const GenericOpDeviceOperation::operation_attributes_t& operation_attributes);
};

where
struct operation_attributes_t {
    std::unordered_map<uint8_t, circular_buffer_attributes_t> circular_buffer_attributes;
    std::vector<data_movement_attributes_t> data_movement_attributes;
    std::vector<compute_attributes_t> compute_attributes;
};

and so on, not going into details here.
Caller of ttnn::generic_op has to provide input and output tensors (respecting Destination Passing Style we use in MLIR) as well as a struct which contains various settings (like settings for circular buffers, paths to data movement and compute kernels, etc). The idea is that user/caller will implement their own kernels (maybe even just compute kernels, if data movement can be automatically generated, which is an idea for future work) and some amount of configuration (which is currently in this POC not insignificant, all of it is hidden behind a concrete function call in ttnn for example ttnn::add) which will be sufficient to run their programs on our HW.
ttmlir progress
On my branch I implemented structs contained in operation_attributes_t in TT dialect, a ttnn.generic op and ttir.external_generic op (since ttir.generic was already in use for Nicks POC for lowering to ttmetal), and integrated a pass in --ttir-to-ttnn-backend-pipeline which enables us to lower
#any_device = #tt.operand_constraint<dram|l1|scalar|tile|any_device|any_device_tile>
module attributes {} {
  func.func @forward(%arg0: tensor<64x128xf32>, %arg1: tensor<64x128xf32>) -> tensor<64x128xf32> {
    %0 = tensor.empty() : tensor<64x128xf32>

    %1= "ttir.generic"(%2, %4, %5) <{
      circular_buffer_attributes = [
        #tt.circular_buffer_attributes<c_in0, <0x0, 6x6>, 4096, 2048, bf16>,
        #tt.circular_buffer_attributes<c_in1, <0x0, 6x6>, 4096, 2048, bf16>,
        #tt.circular_buffer_attributes<c_out0, <0x0, 6x6>, 4096, 2048, bf16>
      ],
      compute_attributes = [
        #tt.compute_attributes<
          <0x0, 6x6>,
          "ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/compute/eltwise_binary_kernel.cpp",
          <hifi4, false, false, false, <1>,
          {ELTWISE_OP = "add_tiles", ELTWISE_OP_TYPE = "EltwiseBinaryType::ELWADD"}>>
      ],
      data_movement_attributes = [
        #tt.data_movement_attributes<
          <0x0, 6x6>,
          "ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/dataflow/reader_binary_interleaved_start_id.cpp",
          <reader, <1, 1>>>,
        #tt.data_movement_attributes<
          <0x0, 6x6>,
          "ttnn/cpp/ttnn/operations/eltwise/unary/device/kernels/dataflow/writer_unary_interleaved_start_id.cpp",
          <writer, <1, 1>>>
      ],
      operandSegmentSizes = array<i32: 2, 1>
    }> : (tensor<64x128xf32>, tensor<64x128xf32>, tensor<64x128xf32) -> tensor<64x128xf32>

    return %1 : tensor<64x128xf32>
  }
}

to
#l1_ = #tt.memory_space<l1>
#system = #tt.memory_space<system>
#layout = #tt.layout<(d0, d1) -> (d0, d1), undef, <1x1>, memref<64x128xf32, #system>>
#layout1 = #tt.layout<(d0, d1) -> (d0, d1), undef, <8x8>, memref<8x16xf32, #system>>
#layout2 = #tt.layout<(d0, d1) -> (d0, d1), undef, <8x8>, memref<8x16xf32, #l1_>>
module attributes {tt.device = #tt.device<#tt.grid<8x8, (d0, d1) -> (0, d0, d1)>, [0]>, tt.system_desc = #tt.system_desc<[{arch = <wormhole_b0>, grid = 8x8, l1_size = 1499136, num_dram_channels = 12, dram_channel_size =<PHONE_NUMBER>, noc_l1_address_align_bytes = 16, pcie_address_align_bytes = 32, noc_dram_address_align_bytes = 32}], [0], [<pcie|host_mmio>], [<0, 0, 0, 0>]>} {
  func.func @forward(%arg0: tensor<64x128xf32, #layout>, %arg1: tensor<64x128xf32, #layout>) -> tensor<64x128xf32, #layout1> {
    %0 = "ttnn.open_device"() : () -> !tt.device<<#tt.grid<8x8, (d0, d1) -> (0, d0, d1)>, [0]>>
    %1 = "ttnn.full"(%0) <{fillValue = 0.000000e+00 : f32}> : (!tt.device<<#tt.grid<8x8, (d0, d1) -> (0, d0, d1)>, [0]>>) -> tensor<64x128xf32, #layout2>
    %2 = "ttnn.to_memory_config"(%arg0, %1) : (tensor<64x128xf32, #layout>, tensor<64x128xf32, #layout2>) -> tensor<64x128xf32, #layout2>
    %3 = "ttnn.full"(%0) <{fillValue = 0.000000e+00 : f32}> : (!tt.device<<#tt.grid<8x8, (d0, d1) -> (0, d0, d1)>, [0]>>) -> tensor<64x128xf32, #layout2>
    %4 = "ttnn.to_memory_config"(%arg1, %3) : (tensor<64x128xf32, #layout>, tensor<64x128xf32, #layout2>) -> tensor<64x128xf32, #layout2>
    %5 = "ttnn.full"(%0) <{fillValue = 0.000000e+00 : f32}> : (!tt.device<<#tt.grid<8x8, (d0, d1) -> (0, d0, d1)>, [0]>>) -> tensor<64x128xf32, #layout2>

    %6= "ttnn.generic"(%2, %4, %5) <{
      circular_buffer_attributes = [
        #tt.circular_buffer_attributes<c_in0, <0x0, 6x6>, 4096, 2048, bf16>,
        #tt.circular_buffer_attributes<c_in1, <0x0, 6x6>, 4096, 2048, bf16>,
        #tt.circular_buffer_attributes<c_out0, <0x0, 6x6>, 4096, 2048, bf16>
      ],
      compute_attributes = [
        #tt.compute_attributes<
          <0x0, 6x6>,
          "ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/compute/eltwise_binary_kernel.cpp",
          <hifi4, false, false, false, <1>,
          {ELTWISE_OP = "add_tiles", ELTWISE_OP_TYPE = "EltwiseBinaryType::ELWADD"}>>
      ],
      data_movement_attributes = [
        #tt.data_movement_attributes<
          <0x0, 6x6>,
          "ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/dataflow/reader_binary_interleaved_start_id.cpp",
          <reader, <1, 1>>>,
        #tt.data_movement_attributes<
          <0x0, 6x6>,
          "ttnn/cpp/ttnn/operations/eltwise/unary/device/kernels/dataflow/writer_unary_interleaved_start_id.cpp",
          <writer, <1, 1>>>
      ],
      operandSegmentSizes = array<i32: 2, 1>
    }> : (tensor<64x128xf32, #layout2>, tensor<64x128xf32, #layout2>, tensor<64x128xf32, #layout2) -> tensor<64x128xf32, #layout2>

    %7 = "ttnn.full"(%0) <{fillValue = 0.000000e+00 : f32}> : (!tt.device<<#tt.grid<8x8, (d0, d1) -> (0, d0, d1)>, [0]>>) -> tensor<64x128xf32, #layout1>
    %8 = "ttnn.to_memory_config"(%6, %7) : (tensor<64x128xf32, #layout2>, tensor<64x128xf32, #layout1>) -> tensor<64x128xf32, #layout1>
    "ttnn.close_device"(%0) : (!tt.device<<#tt.grid<8x8, (d0, d1) -> (0, d0, d1)>, [0]>>) -> ()
    return %8 : tensor<64x128xf32, #layout1>
  }
}

After this I went on to write a simple Python test which was supposed to mock a frontent (tt-forge for example) user of our compiler, who wrote their own kernels and settings for them and wanted to run it.
I managed to create pybindings for devised mlir compute, data movement and circular buffer attributes and filled them with dummy data. Printing them gives
[Attribute(#tt.circular_buffer_attributes<c_in0, <0x0, 6x6>, 4096, 2048, bf16>), Attribute(#tt.circular_buffer_attributes<c_in1, <0x0, 6x6>, 4096, 2048, bf16>), Attribute(#tt.circular_buffer_attributes<c_out0, <0x0, 6x6>, 4096, 2048, bf16>)]
[Attribute(#tt.data_movement_attributes<<0x0, 6x6>, "ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/dataflow/reader_binary_interleaved_start_id.cpp", <reader, <1, 1>>>), Attribute(#tt.data_movement_attributes<<0x0, 6x6>, "ttnn/cpp/ttnn/operations/eltwise/unary/device/kernels/dataflow/writer_unary_interleaved_start_id.cpp", <writer, <1, 1>>>)]
[Attribute(#tt.compute_attributes<<0x0, 6x6>, "ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/compute/eltwise_binary_kernel.cpp", <hifi4, false, false, false, <1, 1>, {ELTWISE_OP = "add_tiles", ELTWISE_OP_TYPE = "EltwiseBinaryType::ELWADD"}>>)]

which exactly resembles the mlir form above. Thus, user can use those attributes as objects in Python, fill them with data and crate a .mlir file from them.
However, I got stuck at creating ExternalOp. The automatically generated API was too confusing and cumbersome, I could not wrap my head around it for two days. I managed to get
%0 = "ttir.external_generic"(<<UNKNOWN SSA VALUE>>, <<UNKNOWN SSA VALUE>>, <<UNKNOWN SSA VALUE>>) <{operandSegmentSizes = array<i32: 2, 1>}> : (tensor<64x128xf32>, tensor<64x128xf32>, tensor<64x128xf32>) -> tensor<64x128xf32>

which is close, but is missing something obviously. I left a TODO in test/python/generic_op_test.py. Someone should take a look at it.


Replace babel-preset-env with @babel/preset-env

https://github.com/babel/babel/tree/master/packages/babel-preset-env#forcealltransforms


feat: Remove stacks automation

I'm wondering what happens if one of those commands fails thinking Will it impact the other below commands of it as well, so the deletion process will be in a sort of "broken" state?

yeah good question, it just keeps going. At least when the stack or the bucket is missing. Strangely most of the hanging stacks were only deleted in half so either crn or gp2 was still hanging and the script handled that


Archetype 12: autoInstallPackage does not install/update the bundle

I am unable to install/update the bundle using the command : mvn clean install -PautoInstallPackage.
I am using :

AEM 6.3 SP1 with AEM Forms 6.3 SP1 Add-on.
Eclipse Oxygen (4.7.2)
AEM developer tools (1.1) + Sling IDE Tool 1.2

Steps:

Create AEM project using Maven Archetype 12.
Updated uber.jar dependency to use version 6.3.1.
Run mvn clean install -PautoInstallPackage. It install the package and the bundle first time successfully.
Note: I do see some white listing errors when it install the package.
Change some component code from crx/de. And then import the content using the AEM Developer Tool for Eclipse.
Make additional component code change in ui.apps project.
Update code in the core project (update any java class).
Run mvn clean install -PautoInstallPackage.
No error reported. But I only see component code updates. Bundle is not updated. I checked the bundle in /system/console/bundles. Timestamp for bundle modification is not updated.
But if I use mvn install -PautoInstallBundle command. It deploys the bundle just fine.

Additional Notes:
If I create a project using the traditional Archetype 10 (for AEM 6.2), I don't see this issue.  But it still uses the maven-scr-plugin and scr annotations. I would prefer to use the latest archetype for my 6.3 project.

Hi @priyankp3005, meanwhile could you find a solution to this problem?

Yes. I started using lazybones and that is working as expected..

I was not able to reproduce the issue with the latest archetype version and AEM 6.4.


Unit test coverage report dashboard

Right now, we execute unit tests in Travis with -cover, and you can see the results on each line:
ok  	github.com/GoogleCloudPlatform/kubernetes/pkg/api	19.821s	coverage: 41.7% of statements

but we don't have an easy place to go browse our code and figure out what's covered and what isn't. Is there some awesome way to do this? At the very least, can we get the actual line-by-line coverage results integrated into our CI build process so we can poke at these somehow?

cc @roberthbailey

I'm willing to take a look into this to at least see if something is possible.

Looking into using Coveralls; there are a few other teams at Google already using this.
https://coveralls.io/r/Google
https://coveralls.io/r/GoogleCloudPlatform
Integration with Travis and Go are supported with the goveralls package: https://github.com/mattn/goveralls

Unit test coverage can now be viewed at https://coveralls.io/r/GoogleCloudPlatform/kubernetes - both master push builds and PRs will be shown there; additionally, Coveralls will update the status and post comments on PRs with coverage stats. (If we find these too spammy, we can disable either or both.)
The top-level README.md also has a badge/link to the coverage dashbaord.


Design and Implement Login/Signup Functionality using localStorage

Description:
This PR implements the login and signup designing and  functionality using localStorage. The focus is on managing user sessions without a backend setup, as the project is front-end-only.
Summary of Changes:

Login/Signup Page: Created forms for users to sign up and log in.
Homepage Button: The "Login/SignUp" button changes to display the username after a successful login.
LocalStorage: User data is stored in localStorage, and the session is maintained until logout.
Logout: Users can log out with a confirmation prompt, and the session is cleared.

Related Issues:
Closes #6
ScreenShot

Hey @grasyPatel, really appreciate your work! Please add an OTP feature when the user enters their email and presses continue. Then, an OTP option should be shown, and after entering the OTP, the password and confirm password options should appear. Hope you understand. If you can not do OTP thing then no problem add alteast Confirm password option.

Thanks so much for the appreciation! 😊 I’ll definitely work on adding the confirm password option right away. Regarding the OTP feature, I’ll check if I can implement it within the current setup. If not, I’ll ensure the confirm password functionality is fully integrated. Will keep you updated!

@thakuratul2  Could you please add the following labels to this PR?
GSSoC'24 extd
Level (based on the complexity)
Status: Completed or Closes
This will help ensure proper tracking under GSSoC'23.


new(userspace/falco): add support for kernel side simple consumer

Signed-off-by: Federico Di Pierro<EMAIL_ADDRESS>What type of PR is this?
Uncomment one (or more) /kind <> lines:

/kind feature
Any specific area of the project related to this PR?
/area engine
What this PR does / why we need it:
Updates libs version and adds support for kernel side simple consumer.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
new: simpleconsumer filtering is now being done at kernel level

@FedeDP: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: FedeDP
To complete the pull request process, please assign kris-nova after the PR has been reviewed.
You can assign the PR to them by writing /assign @kris-nova in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 463fad471c1448513fcd4e30a7e051c0c57546db

/milestone 0.31.0

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, jasondellaluce, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jasondellaluce,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Customized JIRA fields

We would like to following fields to be configured via the cfg.yaml:
Project-ID
Project-Name
Board
Issue-Type
Priority
Jira-Labels=
Assignee
(custom fields)
Custom.Key=Value
Custom.Key=Value

@jerbia I transferred my questions to this ticket.
There are a few moments:

Can we use a projectKey instead of project id? I think that Project id is Jira's inner data.

What is mean a project name? Now it uses as project key. There is an error if project name doesn't match project id

IsuueType is empty, but issue type is required. It should set up, or need default value. Bug?

Now priority and assignee are required params. Is it correct? Maybe I need default values?
High and current user?

Actions with description param. What I have to do with it? this param overlays rendered data. is it correct?

Yes. Let's remove the project ID from the configuration file.
Let's remove Project Name and just use Project Key
Let's put "Bug" as default value.
Yes, default values will be good.
Description can be removed.

Basically can you check that all the fields are easy to get from JIRA UI and that we don't have something that requires an internal ID?

@afdesk I've updated the issue details. Seems like there is already support to custom fields in the cfg.yaml. Can you please test that it is working?
custom:
custom-field:  e.g., hello world
custom-field-numeric-field: 1337
custom-field-multiple-value: ,  e.g., 1,2,3 (must be separated by commas)
custom-field-multiple-line-text-field: "text \n moretext" (quotes are mandatory for this field)
custom-field-date-time-picker: <date && time> e.g., 2014-04-11T12:14:26.880+0400
custom-field-url:  e.g., https://tour.golang.org/moretypes/7

OK, I've checked the "unknown" and it seems to work (custom fields are created).
Please merge the pull request with master and I will approve it.

#15


Add an option to initialize pycondor jobs with arguments.

Description

Currently arguments must be added to a Job via add_args().  Enabling a single set of arguments to be passed during instantiation would provide an easier way to mimic the original dagman workflow.

Good idea @zdgriffith, I totally agree! I think for cases where only a single argument is being added to a Job object that
job = Job('myjob', ex=ex, argument='some command line argument')

is much cleaner that
job = Job('myjob', ex=ex)
job.add_arg('some command line argument')

Do you have an interest in submitting a PR?

Sure, I can take it.

Closed in  #90


[Generated Code] Security API

Added all Security API actions as listed on OpenSearch API Docs using api_generator.
Fixed API Generator bugs:

Spec failed when an API has no query_params
Wrong method names for perform_request_*_ignore404

closes https://github.com/opensearch-project/opensearch-ruby/issues/171
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@VachaShah the core Ruby module uses RSpec test framework, which is usually stored in ./spec folder while dsl and transport use Minitest framework, which is stored in ./test folder.

@VachaShah the core Ruby module uses RSpec test framework, which is usually stored in ./spec folder while dsl and transport use Minitest framework, which is stored in ./test folder.

We should probably move everything to RSpec.


Migrate to the Latest IDC

Summary
This PR attempts to migrate the scripts to be compatible with the latest idc for better compatibility with the latest IDA Pro versions.

Looks great. Thanks for the contribution


API error: Cannot read property 'trace' of undefined

I'm unsure if this is an issue with this package or something within Rollbar.  I just added this to my Laravel 5 project using https://github.com/jenssegers/laravel-rollbar and am testing things out and Rollbar is seeing an exception, but it's not the one I'm throwing.
I've verified that the exception and appropriate context is being passed to the rollbar-php package.

@dlsteuer began investigating this issue with me and we've narrowed it down to https://github.com/rollbar/rollbar-php/blob/master/src/rollbar.php#L342-L343 being the cause of the issue.  I've created this ticket for tracking purposes.

My app needs to go in production in the next month and this is a blocker for me.  I'm happy to Fork the project and disable the conversion that appears to be breaking this, but would prefer to only take that route if this isn't something that is on the radar for a fix.

Hey @bkuhl we're working towards the 1.0 release, which wouldn't have this problem because it's been removed in that branch, I need to have a discussion with some other members of the team about that. The only thing holding back the 1.0 release right now is api compatibility with the current version.

Any idea on what the anticipated date of that is?

Hey Ben, unfortunately I'm not sure on a timeline, so I added a config switch for this. If you add
$configArray = array(
    ...
    enable_utf8_sanitization => false,
    ...
)

This will disable the sanitization for you. Let me know if you have any further problems. It should be available in composer shortly as v0.18.1


Graphs don't show up

Hi, I was following a guide to setup promdash with node_exporter:
https://www.digitalocean.com/community/tutorials/how-to-use-prometheus-to-monitor-your-ubuntu-14-04-server
This worked out fine (even on Fedora 22) and I got Promdash running. Now I'm trying to create a graph. The problem here is that the metrics stored in Prometheus are recognized (visible due to the 'auto-completions' popping up) but the graph is just not showing up. I also noticed that the "Link to Graph in Prometheus" option is also not working because "No datapoints found.". When I simply render the graph using the default frontend of Prometheus there are no problems.
Another less important question: where can I reconfigure the port for promdash?
Hope someone can help me :)
EDIT: yes, I used the 'refresh' buttons

can you open your console and see if there is any error output?

Another less important question: where can I reconfigure the port for promdash?

from the docs:
bin/env bin/bundle exec bin/thin -p $PORT start

The console output shows no errors.
Something else: As mentioned the "Link to Graph in Prometheus" option is not working but when I just add a new graph and put in the same expression everything is fine.
Thanks for the hint with the port!


Dynamically add tooltip and trigger it on hover

Hi,
I can't get the right options or implementation to trigger a tooltip if a condition is met with a hover event. This is the code:
<select
    v-tooltip.top-start="(disabled === true) ? 'TOOLTIPTEXT' : false"
    :disabled="disabled">
    <option></option>
</select>

It looks like it is not reactive to changes, what I need is whenever disabled variable is true the user might be able to open the tooltip by hovering on the select.
I also tried this:
<select
    v-tooltip.top-start="{content: 'TOOLTIPTEXT', show: disabled : trigger: 'manual'}"
    :disabled="disabled">
    <option></option>
</select>

This variation will display the tooltip when the condition is met (without the need of hovering on it, that's the problem)
Is this possible with the current options?
Thanks

v-tooltip.top-start="!disabled && 'TOOLTIPTEXT'"

Thanks for the quick response @Akryum.
It works if the condition is met when the page renders at the beginning and then when the variable changes it works as expected. However if the condition is not met at the beginning and then the variable disabled changes it does not work trigger.
My guess is as the content is empty at the beginning the tooltip does not get created and then changes won't be reflected but I am not sure.

I think the problem is the combination of disabled attribute on a select. This is the fiddle: https://jsfiddle.net/nass600/z1cfmrvb/7/

Your fiddle seems to be working. What is the issue it has exactly?


Implement class gui::items::Picture (cpp)

This class will first be used to load controls pictures in the control panel.

Completed; Compiles; Still to be tested.


feat: add ability to output bundles in separate directory

Implements #4788
Add bundlesOutDir option to .angular-cli.json
.
..
bundles/
    main.bundle.js
index.html

There are errors with deploy test.
Can I run only my test with some command?
Manual testing is fine.

@filipesilva could you see this css-urls fail? It is from #4803
It seems, that EOT files really should named with hash.

Heya @ValeryVS, I'm not sure we want to add this option... It's very similar to the existing --deploy-url and I'm not sure how they should interact together. In fact, that's why the css-urls test is failing.
I think you raise a good point about EOT files. You can do it currently via ng build --output-hashing=media, but now that more stuff is being extracted then we should default to it.

I will research about deployUrl options more, and, may be, recreate this PR later.

I see...
For same purpose, I can use "deployUrl" setting and, then move bundle files and assets dir into "bundles/" dir.

@ValeryVS how were you able to use deployUrl with moving the bundle files? I'm attempting to achieve this with
mkdir dist/static && find dist -type f -maxdepth 1 -mindepth 1 -not -name index.html -exec mv '{}' dist/static \;

which seems to mostly work, however some of the links are broken as I have a separate asset directory, but setting the deployUrl to static causes the path /static to be prepended to those asset urls

@zakhenry
.angular-cli.sjon
{
  "deployUrl": "app/"
}

makefile or shell script
mv dist dist-tmp
mkdir dist
mv dist-tmp dist/app
mv dist/app/index.html dist/

Result
./
    dist/
        app/
            main.bundle.js
        index.html
    src/

@ValeryVS thanks, I was able to work it out, turns out I was having issue with the base root definitions in my scss files, eg I had to change url(/assets/path/to/file) to url(~assets/path/to/file so the webpack asset flattening still worked

@zakhenry
Thanks for hint with tilda.


vp9 hw acceleration

I've implemented vp9 encoding process with ffmpeg + libvpx.   Would like to further accelerate the encoding process of vp9 (realtime encoding).
Will using Intel Media SDK require me to redesign the system that I currently have? Or would it be easier to use intel quick sync video?

Media SDK is SDK which expose Quick Sync Sync video HW features to programmer. I anticipate that at most you'll need to rebuild ffmpeg with qsv support and probably adjust some command-line parameters. But you'll need Icelake system to get VP9e support. see readme in https://github.com/intel/media-driver.

@onabiull Thanks, looking into it.

ICL support VP9 encode

ICL support VP9 encode

Does Ice Lake support the hardware acceleration of vp9 encoder?

Hi @syjml ,
yes

Hi @syjml ,
yes

Does the processor code-named Comet Lake support VP9 encoder hardware acceleration?

No. It's Gen9. VP9 VDEnc (aka LowPower) is not supported there.

No. It's Gen9. VP9 VDEnc (aka LowPower) is not supported there.

Are you saying that the processor code-named Comet Lake does not support VP9 encoder hardware acceleration?


armhf?

Hi, this project looks great,
do you know if dynamsoft provides binaries for armhf (arm32) to run on raspbian?

@arogozhnikov No more arm32 support as I know. But you can flash arm64 raspbian for Raspberry Pi.

But you can flash arm64 raspbian for Raspberry Pi.

True. Except I am physically not in the same place as raspberry 😅  . Nevermind, thanks!


Maintenance: Refresh SAM examples

Summary
The repo contains SAM examples of how to use the libraries. These examples are one of the most visited pages in the repository and since they were released the project and its documentation has evolved. The examples should be updated to reflect that.
Why is this needed?
The SAM samples were first created several months ago and in the meantime several updates have happened, so it's time to refresh them.
Which area does this relate to?
Other
Solution
Below a list of changes in no particular order:
Updates to README

[ ] Change the section that reads "You will need to have a valid AWS Account in order to deploy these resources" to a "Note" with GitHub specific markdown (see here), so that it's better highlighted.
[ ] Link documentation about Layers in Layers section of the README
[ ] Modify X-Ray section of docs to point to CloudWatch Traces section instead
[ ] Remove the section that talks/lists all IDEs (see image in details below)

![Screenshot 2022-11-03 at 11 26 03](https://user-images.githubusercontent.com/7353869/199697804-a53d6c01-fbbf-4b42-96f3-c4df42f82c35.png)

Maintenance

[ ] Update Powertools dependencies to latest available at time of PR
[ ] Update all devDependencies to latest available & test that everything works
[ ] Move back code to examples/lambda-functions (where it was originally) & symlink (how it was originally)

Code

[ ] Update one Lambda function to use Middy middleware usage (see docs)
[ ] Update second Lambda to use Class-based/decorators usage (see docs)
[ ] Update third Lambda (manual usage - like it is now) and make sure all features are represented (see docs)
[ ] Move Powertools instances creation in shared file & import it throughout other files (see example here)
[ ] Make sure all functions are minified, have source maps, use NODE_OPTIONS: --enable-source-maps' ([here](https://docs.aws.amazon.com/lambda/latest/dg/typescript-exceptions.html)), and use AWS_NODEJS_CONNECTION_REUSE_ENABLED = 1` (here)
[ ] Move DynamoDB client instantiation in separate shared file, reuse across functions (see example here) & apply tracing to it
[ ] Move from aws-sdk (v2) to @aws-sdk/client-dynamodb (v3 - docs) - keep DocumentClient
[ ] Add an example of HTTP call (i.e. fake retrieve data at some point) in each function at the beginning, use phin to make requests, make a request to https://httpbin.org/#/Dynamic_data/get_uuid and add the returned uuid to logs (with this), to main segment as annotation (see docs), and as metadata to metrics (using addMetadata).
[ ] All functions should log the incoming event: middleware/decorator via parameter, manual via a logger.info(...)

Acknowledgment

[X] This request meets Lambda Powertools Tenets
[ ] Should this be considered in other Lambda Powertools languages? i.e. TypeScript, Java

@dreamorosi as discussed you can assign this to me, I will start working on this

@dreamorosi I have just created a PR covering the mentioned changes


[shaderc] Use build type to build targets

Currently, shaderc has generated dynamic libraries, static libraries and static libraries containing all dependent binaries at the same time.
This PR will:

Determine whether to generate a dynamic library or a static library according to BUILD_SHARED_LIBS.
Disable generating static libraries containing all dependent binaries.

Partly solved #15452.
Fixes #16122.

Waiting for merge #16138.

/azp run

This LGTM

Hi @JackBoosY I updated vcpkg to the latest version.  It seems that only static shaderc was built.
How do I build the shared lib?
Am I supposed to set BUILD_SHARED_LIBS environment variable or something?

Seems that this PR disables shaderc shared library build?
#Note: glslang and spir tools doesn't export symbol and need to be build as static lib for cmake to work
vcpkg_check_linkage(ONLY_STATIC_LIBRARY)
This part doesn't make sense.  If glslang and spir-tools require static shaderc then they can link against the static shaderc, that doesn't mean that shaderc doesn't support shared library build?

@jmoguillansky-gpsw For building shared, use command ./vcpkg install shaderc.
You should use the right triplet to build it.

@jmoguillansky-gpsw For building shared, use command ./vcpkg install shaderc.
You should use the right triplet to build it.

Hi @JackBoosY ,
Yes I tried that: vcpkg install shaderc:x64-windows, but it still only built the static lib?

@jmoguillansky-gpsw  Ops, sorry, I forgot we force it to build as a static library.
I will investigate later.

The issue is not that glslang and spir-tools require static shaderc -- rather, the problem is that glslang and spir-tools can only be built statically, which means building a shared library for shaderc is extremely suspect (it will contain a duplicate copy of the glslang/spir-tools code along with the main application binary as well as any other consumers). Shaderc consumes glslang, not the other way around.
Therefore, until glslang and spir-tools can be built dynamically, we do not want to build shaderc dynamically.

Shaderc upstream supports shared lib build, and its shipped as part of vulkan sdk.  There's no issue with shaderc , the tool just decides to use the static lib, thats all

I think there's a broader discussion with regards to vcpkg architecture.  Each module doesn't know how it's used, only what it can build.  Some executables may choose to always use static lib, some shared lib, etc.  From my perspective the vcpkg architecture should accomodate these cases.
One option:
The module should build all supported configurations (static, shared, etc).
Vcpkg integrate install can have a param of default lib type (e.g. shared or static) the same way that a linker (ld on linux) has a param to specify default lib type.

Regarding vcpkg integrate install:
Maybe one option to consider is:
let the user navigate to a project and type something like "vcpkg integrate " and then vcpkg will integrate the module into the project.
This gives the user convenience, but perhaps safer than making all libs globally available for all msvc projects,etc.

glslang and spir-tools only depend on the static library, not dynamic.


More flexible update sources configuration

Is your feature request related to a problem? Please describe.
The current update sources configuration is quite limited to files from static URLs and git repositories, and it's not always enough if data is not distributed in such a way. There is also no way to set some configuration per update source other than related to the connection.
There are two examples:

NSRL dataset used in the official safelist service

the service uses some hack to make the url dynamic... but the logic fails as NIST does not publish full db dump every quarter, and the service does not support delta updates. In addition, source-related config about trusted distributors is configured on the service level, even if it's not used for any other sources than NSRL.

Freshclam configuration for ClamAV (not published service)

it requires some special treatment due to rate limits and additional configuration possible, currently I also use some hacks to handle it (e.g. headers as configuration).

Describe the solution you'd like
Let's define sources types - keep the standard ones static URL & git repos, and let the services define their own additional types with basic general fields (like URL, auth, CA etc.) and additional configuration as in normal service config. Refactor update service skeleton to choose source handler based on the type - either one of globals, or a specific one registered by the service. The downloading method (with handling of all auth and git config) should be accessible to the service, but not used by default for custom types.
Then, for example, the safelist service could have a specialized NSRL service taking as config base url (e.g.  https://s3.amazonaws.com/rds.nsrl.nist.gov/RDS/), the type (e.q. modern) and base full dump (e.g. 2023.06). The service would be then responsible for handling the inital db download, and the deltas later. It could also have the trusted_distributors allowing to differ the list between types.
Describe alternatives you've considered
Services currently can still handle special cases by doing a little hacking, or it's always possible to have some additional services pre-processing data and sharing it internally in a way easier to handle by AL.
Additional context

A variation of this exists where a lot of the configuration is in the service details but is organized in a way the configuration is meant for a particular dependency container (typically an updater).
See NetRep's configuration as an example
We've discussed the idea of having metadata attached to update sources to instruct the updater that the sources should be treated differently from one another and that they have different ways of getting/parsing the data of interest.
As for Safelist, it could be configured in a similar way where you can specify the base DB and it can start looking for deltas from that initial point (that particular configuration would require persistence of the database otherwise it'll waste a lot of time if the container reboots).
As for ClamAV, I think having different update periods per source rather than a global could be useful to handle the rate-limiting use-case.

As for ClamAV, I think having different update periods per source rather than a global could be useful to handle the rate-limiting use-case.

It can be helpful, but probably more important is that for unofficial signatures, you can just download them. For the official one, the high traffic control is in place, and you should use DNS check to download just the diff. Basically, not a thing to be just downloaded via URL. I'm delegating this to the official updater software

Adding just a little new use case: I'm considering adding a new source to the badlist service, what currently requires editing the service configuration to instruct the updater. However, if I did that, then this configuration would be marked as changed, so any further upstream changes from your side won't be automatically applied. It's probably not a big deal, it doesn't change often I think, but if we could decouple it from service config and attach to each source independently, it would be more flexible.

Yup, this was something I was considering to do if there were more reason to make adding metadata/configuration to update sources rather than as something disjoint that you have to configuration as a service configuration

This should be resolved in the <IP_ADDRESS> release and all services built under this release.

OMG, now I need to rewrite all the special treatments I wrote, and I don't have time for that, why do you do it to me? :scream:

This should help save you from rewriting until the appropriate time you want to transition over, sorry if this broke anything! 😭
https://github.com/kam193/assemblyline-services/pull/1

Haha, thank you a lot <3 I really love the change, it was a big missing feature ;)
BTW, hove you already updated all services to respect changes? I've notices that ConfigExtractor updater was trying to pull a disabled source (but maybe my service version <IP_ADDRESS> isn't the newest?)

Yup, all services with updater dependencies should've been updated with respect to these changes. The latest version of ConfigExtractor should be <IP_ADDRESS>.

Ah, okay, than my caching proxy didn't get it yet :D


when renaming file (using extension), it still exist in recently opened list.

I'm using the last version that exists today

Which extension are you using?

And which OS? And is there any way you can take a gif or some screenshots?

fyi @bpasero


TerraformBoot callback APIs of Xpanse retruns unexpected authentication exception.

TerraformBoot callback APIs of Xpanse retruns unexpected authentication exception.

Caused by: In the callback the method identityProviderManager.getCurrentLoginUserId() but the TerraformBoot client provided any authentication information

Please add integration testing for the WebhookApi to avoid the same problem

@swaroopar When the TerraformBoot is enabled, the UUID of  taskId 、the workspace of terraform scripts and the UUID in the output log  in the TerrraformBoot are all different. Why don't we use the same ID as TerraformLocal

@swaroopar When the TerraformBoot is enabled, the UUID of taskId 、the workspace of terraform scripts and the UUID in the output log in the TerrraformBoot are all different. Why don't we use the same ID as TerraformLocal

@baixinsui it is a different topic. Let us discuss it separately.

@WangLiNaruto looks like one more usecase that is broken after our recent changes.

@swaroopar When the TerraformBoot is enabled, the UUID of taskId 、the workspace of terraform scripts and the UUID in the output log in the TerrraformBoot are all different. Why don't we use the same ID as TerraformLocal

@baixinsui it is a different topic. Let us discuss it separately.

ok


How to convert the uri string to the file or absolute path?

Thanks for the awesome package, I'm using it and getting issue like:
how to get real path of the notification/ringtone we selected, we have to play sound also, How to archive it.


Make app code follow clean coding

The changes in this PR attempt to make the app code more self-explanatory. Instead of comments we're using docstrings, functions do "only one thing", we clean up all unneeded code (unused imports).
BeeWare users should be inspired by self-explanatory, clean code. The rest is in the documentation.
Note: After this is merged, the related documentation needs be aligned again, too.

Thanks for the contribution, but I disagree that these changes make anything "cleaner".
The purpose of the Briefcase template is to bootstrap a project as quickly as possible. It gives the end-user a minimal working project, including imports that almost every project will need. Including additional complexity in a bootstrap gives the impression that there is a coding convention that all projects need to follow, when no such convention exists. If every novice user gets "make_main_window()" and "make_main_content()" in their stub, at least some of them are going to ascribe significance to those names, in those places, and then ask why those methods need to exist, and whether they can add their own, and so on.
Don't get me wrong - refactoring submethods in this way will often be a good idea - but when we provide a template, we need to be careful about the message we're communicating with that template.

including imports that almost every project will need

That is fine, I can bring that back in.

Including additional complexity in a bootstrap

I believe, it's the contrary. These changes increase readability. It's an implementation of PEP 20, code must be beautiful and self-explanatory (Clean code is a perfect match of what PEP 20 says).
We need to provide developers with a good place to start software development from, not just a template to hack a solution. The Python community deserves this!
Maybe I should have opened an issue first to discuss matters?

I'm sorry, but I disagree. Adding submethods in this way adds 2 levels of indirection for something that is only 4 instructions to start with.
Templates like this one serve two purposes - they provide a basis for future development, but they also communicate things that we, as the library maintainers, think the user should know about. Adding a method like "make_main_content" imposes a cognitive burden on new users. They need to ask "Why is it there?" What is the significance of this name? Do I have to provide this method?
It is not even a slight stretch in my mind to imagine a world where developers end up defining empty "make_main_content()" methods because they think the method is needed by the framework, because the initial template explicitly told them it was needed.
We could certainly do a better job of encouraging good code use, reuse and structure - but IMHO the place for that is the code examples of full projects, not the initial project template.
In terms of project procedure: there's no problem opening a PR for this sort of thing. You probably want to start with an issue and discussion if the implementation is going to be complex; but in the case of a relatively straightforward change like this one, a PR serves as a good starting point for describing exactly what you have in mind.

Sorry, Russel, that my PR caused you personal anger. I should have avoided that and discuss this in an issue first. I hoped that "being a welcoming community" also stretched to contributions, and that we would discuss the PR before it would be closed unmerged (in a mutual agreement). I should have known better and I apologize.
Let me just stress that, specifically, I'm not happy with the make_foo functions at all. I had hoped we'd come up with either a better naming or not using them at all, while the PR would have gone forward.
And while I see value in the explanations you provide I believe we should treat each other with respect, help each other along with contributions, strive to understand each other's motivations, and show the world that the Python community is a welcoming community. -- I could have been a first-time contributor. (Well, true, I am not -- that's why I apologized.)

You certainly haven't caused me "personal anger", and if my response gave you that impression, I sincerely apologize. I aspire to BeeWare (and the Python community) being welcoming to people of all backgrounds and experience levels, both as contributors and as users. If that hasn't been your experience, then that's my failing, not yours. If you have any suggestions about how I could have managed this discussion better, please let me know.
I'm very keen to discuss any suggestion. However, that doesn't mean every suggestion will be accepted. In this case, I closed the PR because I couldn't see any obvious path forward. I couldn't see a case where "if only this one detail was slightly different, I would merge". I can see how closing the ticket perhaps comes off as "anger". I can only say that from my perspective, it's only an indicator that at least for now, I've made a decision as a project lead. However, nothing in Github (or most other ticket systems for that matter) is irreversible. You should feel free to continue to make your case if you feel there's some aspect of the problem that I've missed.
For the benefit of discussion (and, to demonstrate in some small way that there's no ill-will on my part), I'm happy to reopen this ticket. However, I will state in advance that I'm still not sure I see what the path forward is. Either you believe that factoring the template into submethods is "cleaner", or you don't; and I don't. It's not just a matter of naming; it's a pedagogical issue (which I've already detailed).
I'm intrigued by your suggestion that your aims could be met by not using these functions at all. I'm having imagining out what this patch would look like without adding the extra methods. If you've got other ideas - please, by all means let me know what they are.

Russell, I've taken your thoughts into account, I think. The code doesn't need comments this way. I hope this is a better ground for discussion, now.

Thanks, Russell! :+1:


Details of 'http : HttpClient' object is visible to the public through Ballerina api-docs

Description:
The 'http: HttpClient' object details (which should be private) can still be seen in the ballerina api-docs [1].
[1] https://ballerina.io/learn/api-docs/ballerina/http/clients/HttpClient.html

Possible solution:
Public client classes
  http:Client
  http:LoadBalanceClient
  http:FailoverClient

Internal Client classes
  HttpClient - which is native client connector
  HttpSecureClient
  RetryClient
  RedirectClient
  HttpCachingClient
  CircuitBreakerClient
  CookieClient

All these internal clients are structurally equal. So whatever we do, should be done to all of them. When trying to make the private, it blocks at CircuitBreakerClient level. because CircuitBreakerClient some additional APIs also. Eg: forceClose(), forceOpen().
So as a solution, we can make CircuitBreakerClient a public client and hide all other clients.
Public client classes
 http:Client
 http:LoadBalanceClient
 http:FailoverClient
 http:CircuitBreakerClient

Possible solution:
Public client classes
  http:Client
  http:LoadBalanceClient
  http:FailoverClient

Internal Client classes
  HttpClient - which is native client connector
  HttpSecureClient
  RetryClient
  RedirectClient
  HttpCachingClient
  CircuitBreakerClient
  CookieClient

All these internal clients are structurally equal. So whatever we do, should be done to all of them. When trying to make the private, it blocks at CircuitBreakerClient level. because CircuitBreakerClient some additional APIs also. Eg: forceClose(), forceOpen().
So as a solution, we can make CircuitBreakerClient a public client and hide all other clients.
Public client classes
 http:Client
 http:LoadBalanceClient
 http:FailoverClient
 http:CircuitBreakerClient

This API change is not backward compatible and can affect the users

When checking the implementation, it was found that http:HttpClient was visible for circuit breaker to execute 3 methods.

forceClose()
forceOpen()
getCurrentState()

So moved them to http:Client and hide all the subtypes of http:Client via https://github.com/ballerina-platform/module-ballerina-http/pull/994
Hence only the following clients will be public
Public client classes
 http:Client
 http:LoadBalanceClient
 http:FailoverClient


Improve filters with request and response filters

Purpose

Now there are two types of filters independently for the request and the response path namely RequestFilter and ResponseFilter.

Approach

Change filter object definition and its calling at Desugar level

Samples

N/A

Remarks

N/A

Check List

[ ] Read the Contributing Guide
[ ] Required Balo version update
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[x] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[x] Ballerina By Examples

Codecov Report

Merging #17179 into master will decrease coverage by 54.66%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #17179       +/-   ##
===========================================
- Coverage   68.85%   14.18%   -54.67%
===========================================
  Files           5       51       +46
  Lines         122     1600     +1478
  Branches       21      260      +239
===========================================
+ Hits           84      227      +143
- Misses         38     1360     +1322
- Partials        0       13       +13

Impacted Files
Coverage Δ

tool-plugins/vscode/src/utils/logger.ts
52.63% <0%> (ø)

tool-plugins/vscode/src/core/index.ts
100% <0%> (ø)

tool-plugins/vscode/src/utils/rpc/handler.ts
5.88% <0%> (ø)

tool-plugins/vscode/src/debugger/index.ts
0% <0%> (ø)

tool-plugins/vscode/src/debugger/manager.ts
0% <0%> (ø)

...ugins/vscode/src/project-tree-view/project-tree.ts
0% <0%> (ø)

tool-plugins/vscode/src/test-runner/activator.ts
0% <0%> (ø)

...lugins/vscode/src/core/extended-language-client.ts
24% <0%> (ø)

tool-plugins/vscode/src/debugger/session.ts
0% <0%> (ø)

tool-plugins/vscode/src/extension.ts
0% <0%> (ø)

... and 36 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e286a4c...2cd5f5f. Read the comment docs.


allow only issuer with SSL (https)

this is the work of the administrator or software, not the library


Feature: abstract agent design and an example DQN implementation

This PR proposes an abstract BaseAgent design and an implementation of DQN based on this design. More details will be revealed with docstring format soon. Currently the TODO list contains:

[ ] A stable-baselines-style learn method to implement the run_loop function as currently examples show.
[ ] Return intermediate variables for the update method (e.g., chosen Q-values, target Q-values, different portions of the total loss).
[ ] Logging support for learn method such as TensorBoard and WandB.

Some minor updates to file structures and dependencies:

rename the agent submodule to agents.
add stable-baselines3 requirement as a temporary solution to VecEnv (which should be replaced as its heavy dependency on PyTorch).
modify dm-haiku package source from GitHub repo to PyPI.

As current agent-design branch contains many changes to the core package (dependencies, versions, etc.). We decide to merge this PR and leave support of the rest of features support as further developments.


Which func to use to get all networks/cidr?

what is the function to use to get all networks?

Hi @tmudalagiri , there isn't function for this for now. We will add that feature to get the networks, but there are two conditions to get IPv4/v6 networks, and we can add the condition for checking all networks in a specific network_view.

Hi @tmudalagiri  , there is an go client generator which will add the function that supports  and includes this get all networks feature which is under review, so once that gets into the go client, then this function will work as per wapi api documentation.

hi @hemanthKa677 any information about this feature?
For the moment, I'm statically declaring the references of my networks in my conf. I don't know if there's a better way without this function available.

Hi @Awirbl,
You can retrieve data for any object using the following method.
netview2 := "default"
  sf2 := map[string]string{
    "network_view": netview2,
    "comment":      "test comment",
  }

queryParams2 := ibclient.NewQueryParams(false, sf2)

var res2 []ibclient.Network
var nw ibclient.Network
nw.SetReturnFields([]string{"extattrs", "network", "network_view", "comment"})

err = conn.GetObject(&nw, "network", queryParams2, &res2)
if err != nil {
    fmt.Println("Error: ", err)
} else {
    fmt.Println("res:", res2)
}

Outcome for the provided code snippet.
res: [{{[] map[]}  network/ZG5zLm5ldHdvcmskMTIuMS4xLjAvMjYvMA:<IP_ADDRESS>/26/default default <IP_ADDRESS>/26 map[Site:Uzbekistan] test comment}]
I believe this solution should resolve your concern.

Hello @JkhatriInfobox
Super cool, it actually works on the first try! I'm digging a bit deeper because I was missing some elements that weren't very clear in my documentation.
Thank you


[Android] API build script

Set sdk path and build options, build Android lib using the script in nnstreamer.
Signed-off-by: Jaeyun<EMAIL_ADDRESS>

:memo: TAOS-CI Version: 1.5.20200324. Thank you for submitting PR #626. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://nnsuite.mooo.com/.

INFO: You can read if there are syntax errors in the our shellcheck report. Please read http://nnsuite.mooo.com/TAOS-CI/ci/repo-workers/pr-checker/626-202004071616230.030150890350342-6fd17c32f6089125b2aa3fe2487af34833115bd7/report/shellcheck_syntax_result.txt.

Waiting for other PR (https://github.com/nnstreamer/nnstreamer/pull/2213)
This requires android build script in nnstreamer.

INFO: You can read if there are syntax errors in the our shellcheck report. Please read http://nnsuite.mooo.com/TAOS-CI/ci/repo-workers/pr-checker/626-202004081940320.95204591751099-13f20c8ea6e62ca94cb5d171a9c5d2d1c80c1a93/report/shellcheck_syntax_result.txt.

After the merge, you will need to update the server's scripts manually.


Implement Debugging

There needs to be a structured mechanism for logging and a method of triggering log levels.
I'm thinking a debug flag qaze --debug
To that end, I need to identify a good Go logging library that takes care of the log level mechanisms.

This has now been implemented with version v0.30.0-alpha.

Debug Flag Added
Implemented Logging via the Logrus Go Library.
Enriched logging and verbosity for better troubleshooting.


Add dethy

Add's the dethy setup along with the dethy cops

oops I forgot a description

Additionally, Goons shouldn't be able to kill N1 on Dethy. You should add an appropriate check there in Goon#canUseAction

Where would I add this?

I found a decent amount of bugs so I'm closing this for now


VxWorks platform for Cavium Octeon MIPS64

What is the feature you'd like to have?
vxworks-cavium-mips64 platform with Cavium MIPS calling conventions and VxWorks platform types
Are any alternative solutions acceptable?
The platform can be manually set to cavium-mips64 on load of the VxWorks image for now. But, this won't allow platform types to load automatically.

Added vxworks-cavium-mips64 and vxworks-mipsel platforms in 4.3.6594-dev


Fix issue #2 | by Nihal

Solves issue #2
Describe the changes you made briefly

[ ] Modified C to C++

This has been resolved #88


Fix destination path to be relative to current directory.

Issue #, if available:
Fixes issue #45
Description of changes:
Converts to/from files to relative paths before passing them into path_to_file, which expects two paths relative to the same root.
I don't believe this should cause issues with existing CBMC-viewer usage:

If both paths are already relative (even if not relative to current directory) , then os.path.relpath won't change either of them, so there is no issue.
If both paths are absolute, then they still have the same relative difference, and so path_to_file should return the same value.
If one of the paths is absolute and the other is relative to the current directory, then os.path.relpath will not change the relative difference between the two paths.
If one of the paths is absolute and the other is relative to a directory other than the current directory, then this change will lead to invalid links, however this scenario would already lead to invalid links prior to this PR.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I prefer this solution https://github.com/awslabs/aws-viewer-for-cbmc/pull/47

Closing in favor of https://github.com/awslabs/aws-viewer-for-cbmc/pull/47


Window progress should show title and increment message

Follow up from https://github.com/Microsoft/vscode/issues/50932, we should show title and message also in the status bar when both are provided.


VM with same name on 2 separate vcenter sources

I have 2 vCenter instances (one 6.5, one 6.7) that have some VMs with the same name.  Each time the sync job runs, the VM item in Netbox is overwritten by whichever source runs second.
I tested, and Netbox allows multiple VMs with the same name.  It seems like it might be a bug based on netbox-sync searching based off of name.  It may be prudent to store a uuid to search for instead of testing the name of the VM against the name in Netbox.  I would instinctually say instanceUuid (vim.vm.ConfigInfo.instanceuuid), but it is marked as potentially not set in the docs.  vim.vm.ConfigInfo.uuid is set, but could be identical if there are cloned VMs.
Perhaps the search checks if the name exists and the uuid matches the current vm data, then its a match.  Otherwise, its a new VM entry.  Or, simply adding an option in the settings.ini to match name AND "source: sourcename" tag.
I apologize if I've missed anything in the source where you're already checking for unique instances (other than Netbox > Virtual Machine > name)

Here is my output from the docker container running at DEBUG2:
vcsync.log

This is not an issue with the script.  I believe I misinterpreted the issue


Make hook work with actual versions of dehydrated

Hello,
as some functions are missing, this hook does not work with dehydrated.
By adding dummy functions, it works again.

Sorry about that, I haven't updated my version of dehydrated in a while.


eventshub.Install docs

Signed-off-by: Francesco Guardiani<EMAIL_ADDRESS>
Changes

Just a little doc on the expectations of eventshub.Install

/kind

Fixes #

Release Note

Docs

/lgtm

/lgtm

installing ko is failing

installing ko is failing

/lgtm

/lgtm

/lgtm
/approve

/lgtm
/approve


chore(db): add pgbouncer to our dev setup

This PR adds pgbouncer to our dev setup. It moves the "real" postgres instances to 7432 (si) and 8432 (si-test) respectively, replacing them with two different pgbouncers. This change is primarily to ensure that we avoid any threat of pool exhaustion in the test suite, but has the added benefit of making the dev stack closer to our production deployment (which would also include pgbouncer.)
PgBouncer configures a pool per hostname+database. Since each of our dal integration tests creates its own database, it thereby also creates its own pool. To manage this, we create a pool size of 10 per hostname+database tuple, which we will then reap after 10 idle seconds. The result is that the current test suite peaks somewhere around 300 tests, but very rapidly shrinks. We should have plenty of headroom.
The only tricky thing here is that we alter the migration paths for tests, as we migrate with the underlying database connection directly. When we do, we must forcibly disconnect the bouncer - so if you happen to be logged in to postgres, get ready to be logged out when we need to make the original template database for tests.

bors try

bors try

bors try

bors try

bors try

bors try

bors try

bors try

bors try

bors merge


fix: skipping checking active volumes with the same number of files at the moment

What problem are we solving?
https://github.com/seaweedfs/seaweedfs/issues/4140
How are we solving the problem?
If the volume modification time is close to the time of the last statistics collection pulse in the topology, then additionally check the number of vils at the moment for each such volume
How is the PR tested?
localy
1. weed -v 4 server -s3 -filer -volume.max=0 -master.defaultReplication 001 -master.volumeSizeLimitMB=1024 -volume.preStopSeconds=1 -s3.port=8000 -s3.allowEmptyFolder=false -s3.allowDeleteBucketNotEmpty=false -s3.config=./docker/compose/s3.json
2. weed -v 4 volume -port 8081 -max=0 -preStopSeconds=1 -dir=/tmp
3. for i in {1..1000}; do s3cmd --access_key=some_access_key1 --secret_key=some_secret_key1 --no-ssl --host=<IP_ADDRESS>:8000 --host-bucket=http://<IP_ADDRESS>:8000 put ~/Downloads/any_doc.pdf "s3://test/any_doc.$i.pdf"; done
4. echo "volume.check.disk -v -force -syncDeleted" | weed shell
skipping active volumes 37 with the same file counts on <IP_ADDRESS>:8080 and <IP_ADDRESS>:8081
skipping active volumes 34 with the same file counts on <IP_ADDRESS>:8080 and <IP_ADDRESS>:8081
skipping active volumes 38 with the same file counts on <IP_ADDRESS>:8080 and <IP_ADDRESS>:8081
skipping active volumes 35 with the same file counts on <IP_ADDRESS>:8080 and <IP_ADDRESS>:8081
skipping active volumes 39 with the same file counts on <IP_ADDRESS>:8080 and <IP_ADDRESS>:8081
skipping active volumes 36 with the same file counts on <IP_ADDRESS>:8080 and <IP_ADDRESS>:8081

Checks

[ yes ] I have added unit tests if possible.
[ no ] I will add related wiki document changes and link to this PR after merging.

@chrislusf corrected


A Scaler for Selenium Grid

Use a KEDA scalar to scale selenium grid automatically based on the number of requests on the Selenium hub pod.
Use-Case
There is currently no easy way to reliablity scale Selenium Grid chrome-node pods. We have tried HPA but they make our tests more unreliable and the hub gets exhausted eventually.
We know that the Hub is a single point of failure and maybe SeleniumGrid is not an appropriate tool.
Specification

Scale Selenium browser nodes based on the instance requests on the hub
Auto scale up/down based on demand.
Can the hub be scaled up?

Regards,
Nas

Thanks for the suggestion, is this something you are willing to contribute?

@tomkerkhove Apologies, it seems like it requires GO and I have no experience with GO.
I'll try when I can find time.

No worries then! We'll get there at some point.

I would like to contribute for this. I already some poc to make this work using metrics api scaler but since i know Golang i thought i can implement a new scaler for selenium. Let me how to proceed

That would be great, thank you!

Also, if you are already a KEDA user and want to be listed - Please let me know!

We are experimenting with KEDA in our local setup to see how it suits our needs. Once we push it to production, we would be happy to be listed as an user. I will keep you posted on that.

Regarding the Scaler for selenium grid. I just have quick question. How is it going to be. Is it going to be a external scaler or is it going to be a inbuilt scaler