[Task]: Enable the Google Analytics transform to connect the GA4 API

What needs to happen?
Migration: https://issues.apache.org/jira/browse/HOP-3982
The Google Analytics transformation currently connects Google Analytics Universal version 3, however there is a new version of GA, GA4 (4 does not mean version 4, in fact this new API is in version 1) that the transformation does not read and that's why it needs a refactoring to allow reading this new version of GA.
Below the error when you try to connect a GA4
IOException: User does not have sufficient permissions for this profile.
Root cause: GoogleJsonResponseException: 403 Forbidden
GET https://www.googleapis.com/analytics/v3/data/ga?dimensions=ga:browser&end-date=2022-05-04&ids=ga:211279119&max-results=1&metrics=ga:visits&samplingLevel=FASTER&sort=ga:visits&start-date=2022-05-04
{
  "code" : 403,
  "errors" : [
{     "domain" : "global",     "message" : "User does not have sufficient permissions for this profile.",     "reason" : "insufficientPermissions"   }
],
  "message" : "User does not have sufficient permissions for this profile."
}
Here is a reference to the documentation of this new API
https://developers.google.com/analytics/devguides/reporting/data/v1/rest
ATTENTION: Your GA account needs to be migrated to GA4
Issue Priority
Priority: 3
Issue Component
Component: Transforms

the GA4 is a complete rewrite, there is no upgrade path from GA3. this will be a new transform.


Update all ExprKinds to wrap structs

Codecov Report
Attention: Patch coverage is 77.31959% with 22 lines in your changes missing coverage. Please review.

Project coverage is 84.04%. Comparing base (685f22a) to head (9669405).

Files
Patch %
Lines

src/Escalier.Codegen/Codegen.fs
31.25%
11 Missing :warning:

src/Escalier.TypeChecker/Mutability.fs
57.14%
3 Missing :warning:

src/Escalier.TypeChecker/Visitor.fs
50.00%
3 Missing :warning:

src/Escalier.Data/Library.fs
60.00%
2 Missing :warning:

src/Escalier.Parser/Parser.fs
95.23%
2 Missing :warning:

src/Escalier.TypeChecker/Helpers.fs
50.00%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #356      +/-   ##
==========================================
+ Coverage   84.00%   84.04%   +0.04%
==========================================
  Files          22       22
  Lines       12316    12347      +31
  Branches     1757     1757
==========================================
+ Hits        10346    10377      +31
  Misses       1970     1970

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


UPDATE and DELETE have no effect

Hey there, thanks for the amazing work on chdb!
I noticed that ALTER TABLE ... UPDATE and ALTER TABLE ... DELETE statements don't seem to have any effect in chdb. Data simply stays unchanged.
The same does work in clickhouse-local.
How to reproduce
The below examples can also be seen in this Colab notebook
UPDATE
from chdb import session as chs

sess = chs.Session()

sess.query("CREATE DATABASE test_db ENGINE = Atomic;")
sess.query("CREATE TABLE test_db.test_table (x String, y String) ENGINE = MergeTree ORDER BY tuple()")
sess.query("INSERT INTO test_db.test_table (x, y) VALUES ('A', 'B')")

print("Original values:")
print(sess.query("SELECT * FROM test_db.test_table"))

sess.query("ALTER TABLE test_db.test_table UPDATE y = 'updated' WHERE x = 'A' AND y = 'B'")

print('Values after UPDATE (expected "A", "updated"):')
print(sess.query("SELECT * FROM test_db.test_table"))

This prints:
Original values:
"A","B"

Values after UPDATE (expected "A", "updated"):
"A","B"

DELETE
from chdb import session as chs

sess = chs.Session()

sess.query("CREATE DATABASE test_db ENGINE = Atomic;")
sess.query("CREATE TABLE test_db.test_table (x String, y String) ENGINE = MergeTree ORDER BY tuple()")
sess.query("INSERT INTO test_db.test_table (x, y) VALUES ('A', 'B')")

print("Original values:")
print(sess.query("SELECT * FROM test_db.test_table"))

sess.query("ALTER TABLE test_db.test_table DELETE WHERE x = 'A' AND y = 'B'")

print("Values after DELETE (expected is no rows):")
print(sess.query("SELECT * FROM test_db.test_table"))

This prints:
Original values:
"A","B"

Values after DELETE (expected is no rows):
"A","B"

Expected behavior
Data is updated / deleted, the same way as in ClickHouse / clickhouse-local.
Additional context
I tested the same queries as shown above in a clickhouse-local session, started with clickhouse local --path .. There it works as expected.
I can reproduce the issue with chdb on my M1 Mac and on Google Colab

sess.query("delete from test_db.test_table WHERE x = 'A' AND y = 'B'") hangs

same  problem with create or replace, it seems the issue is with the engine ENGINE = MergeTree
Only Engine=Log seems to be working

Looking into that.

Is this report validated with chdb==1.0.0rc2?

Yes

I confirm that this is an issue caused by the delayed execution of DELETE and ALTER SQL in MergeTree family. I am currently finding the best way to handle it for chdb.
Give me some time, no need to do further tests until 1.0.0rc3 released

pypi.org isn't updated to 1.0rc3 now

Please wait for the build pipeline

Please wait for the build pipeline

seems failed https://github.com/chdb-io/chdb/actions/runs/7018226681

it's ok now
(base) root@e1a427840adf:/# pip install chdb==1.0.0.rc3 -i https://pypi.tuna.tsinghua.edu.cn/simple
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting chdb==1.0.0.rc3
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e5/ae/710ed24e1118e91d47431c5ebcf47afbaeb1b869046b10391f3d3348e81a/chdb-1.0.0rc3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (99.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.6/99.6 MB 5.5 MB/s eta 0:00:00
Installing collected packages: chdb
  Attempting uninstall: chdb
    Found existing installation: chdb 1.0.0rc2
    Uninstalling chdb-1.0.0rc2:
      Successfully uninstalled chdb-1.0.0rc2
Successfully installed chdb-1.0.0rc3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(base) root@e1a427840adf:/#

(base) root@e1a427840adf:/# python
Python 3.11.4 (main, Jul  5 2023, 13:36:48) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from chdb import session as chs
Lowered mark cache size to 3.76 GiB because the system has limited RAM
Lowered index mark cache size to 0.00 B because the system has limited RAM
>>>
>>> sess = chs.Session()
>>>
>>> sess.query("CREATE DATABASE test_db ENGINE = Atomic;")
Lowered mark cache size to 3.76 GiB because the system has limited RAM
Lowered index mark cache size to 0.00 B because the system has limited RAM

>>> sess.query("CREATE TABLE test_db.test_table (x String, y String) ENGINE = MergeTree ORDER BY tuple()")
Lowered mark cache size to 3.76 GiB because the system has limited RAM
Lowered index mark cache size to 0.00 B because the system has limited RAM

>>> sess.query("INSERT INTO test_db.test_table (x, y) VALUES ('A', 'B')")
Lowered mark cache size to 3.76 GiB because the system has limited RAM
Lowered index mark cache size to 0.00 B because the system has limited RAM

>>>
>>> print("Original values:")
Original values:
>>> print(sess.query("SELECT * FROM test_db.test_table"))
Lowered mark cache size to 3.76 GiB because the system has limited RAM
Lowered index mark cache size to 0.00 B because the system has limited RAM
"A","B"

how to avoid output following message?
Lowered mark cache size to 3.76 GiB because the system has limited RAM
Lowered index mark cache size to 0.00 B because the system has limited RAM

how to avoid output following message?

we're aware of this issue - those warnings come alongside results from the clickhouse engine itself

version 1.0.0 still has the RAM messages


Add higress lua canary script

Embed the default higress lua canary script.

Codecov Report

Merging #116 (6ff6a9a) into master (843e8b8) will not change coverage.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@           Coverage Diff           @@
##           master     #116   +/-   ##
=======================================
  Coverage   44.74%   44.74%
=======================================
  Files          44       44
  Lines        4519     4519
=======================================
  Hits         2022     2022
  Misses       2158     2158
  Partials      339      339

Flag
Coverage Δ

unittests
44.74% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

/lgtm

/approve


Drop your email

Add an API to send email to the user when he drops a email on the website
Reference : https://www.neuebel.com/

I want to work on this issue.

Currently, I've not opened it for everyone, I would be working on this later on. Not on priority

Currently, I've not opened it for everyone, I would be working on this later on. Not on priority

Okay! Cool.


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/48677

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/100679


Theme change button not work at first click

localStorage.getItem("theme") checking is null or undefined in first render
that's why i set default dark at render

[LEVEL]/[COMPONENT]
TASK:
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Only
[x] Refactor

Description, Motivation and Context

Screenshots - Animations

Hey @techhjork thanks for your contribution, however the problem was fixed here -> https://github.com/nextui-org/nextui/pull/209 and it's pending releasing


SSML Markdown Language

Speech Synthesis Markup Language (SSML) is designed to provide a rich, XML-based markup language
for assisting the generation of synthetic speech in Web and other applications. The essential role of the
markup language is to give authors of synthesizable content a standard way to control aspects of speech
output such as pronunciation, volume, pitch, rate, etc. across different synthesis-capable platforms.

There are good references on Speech Synthesis Markup Language at these links:

https://developer.amazon.com/en-US/docs/alexa/custom-skills/speech-synthesis-markup-language-ssml-reference.html

https://www.w3.org/TR/speech-synthesis11/
https://cloud.google.com/text-to-speech/docs/ssml

Are there any specific features we're looking to document?


Plot brushing API in Express

Currently the brush options are defined in ui.output_plot() so it doesn't seem like plot brushing can be used in Express.
 ui.output_plot("plot1", click=True, dblclick=True, hover=True, brush=True)

Maybe we should pass these options through @render.plot() instead so they work in express.
Related to #975.

In case anyone is interesting in doing this today, it's possible with shiny.express.output_args, but it seems likely we'll come up with a better API. Here's some examples https://github.com/posit-dev/shinylive/pull/108


Allow to destroy a workflow session when pwd is not in the workflow config folder

When creating a workflow session, we do -apply from a folder that holds the config.fab for this workflow.
Currently we have to stay in the same folder to -destroy this session. However, when we work with multiple workflows, it is inconsistent to always return the the original folder for the destroy op. It would work better if the session context can remember the original folder path and use that wherever we try to destroy the session.

Resolved now the directory is associated with the session. The mapping is stored in a meta data file on disk.


RHODS: update the robot test

toolbox: _common.py: do not generate artifact_extra_logs_dir/exit_code

This file is redundant with FAILURE generated in case of failure.

rhods_test_jupyterlab: do not keep our canary file 'os-release' in the artifacts

This file is added to the bucket to avoid it being empty and making
some copy fail because of it.
We do not want to keep it in the artifacts.

rhods_test_jupyterlab: robot test: refactor to use variables and do not call 'Begin/End Web Test'

Variables make a cleaner separation between configuration and
execution
Being/End Web Test ROBOT keywords were doing things out of our view,
like connecting to RHODS/JupyterHub, and deleting the Notebook.

rhods_test_jupyterlab: robot test: remove the call to 'Can Close Notebook when done'

We want to keep the notebook running after the robot test.
They are deleted by the rhods_test_jupyterlab ansible tasks.

/cc @fcami

/test ods-jh-on-ocp

test passed with 15/15 \o/
==============================================================================
Test-Jupyterlab-Psap-Simplenotebook
==============================================================================
Begin and Authenticate                                                | PASS |
------------------------------------------------------------------------------
Login to Jupyterhub                                                   | PASS |
------------------------------------------------------------------------------
Spawn a Notebook                                                      | PASS |
------------------------------------------------------------------------------
Git Clone the PSAP notebooks                                          | PASS |
------------------------------------------------------------------------------
Run the PSAP jh-at-scale Simple Notebook                              | PASS |
------------------------------------------------------------------------------
Test-Jupyterlab-Psap-Simplenotebook                                   | PASS |
5 tests, 5 passed, 0 failed

/test ods-plot-jh-on-ocp

@fcami PR looks good to me, if you can have a look
/approve

thanks Kevin, looks good !
/approve
/lgtm


Update to version that moves audio definitions.

Closes #317

Interestingly this has regressed completions in microbit.audio. Importing audio itself is fine.
The pin0, MicroBitDigitalPin and Sound imports that aren't re-exported from audio appear as if they are available on microbit.audio. This is unexpected to me at least. The reverse wasn't an issue previously (i.e. via import audio has no audio.pin0).
This is a preexisting issue for all the modules already similarly structured to audio so I think it makes sense to merge this and fix separately. I'm not clear on whether it's a Pyright issue or a stubs issue so it'll need some research.
It's the reason there's display.Image which is probably the easiest issue to encounter.

Raised #410 for reexport issue. Merging.


restrict files included in npm package

Currently, the npm package contains the following:
benchmarks/
bin/
examples/
lib/
node_modules/
test/
.eslintrc
.npmignore
.travis.yml
API.md
Changelog.md
index.js
LICENSE
package.json
README.md
test_sentinel.js

Would it make sense to restrict it to the following?
lib/
Changelog.md
index.js
LICENSE
package.json
README.md

Edit: it should be easier than tracking each file :innocent: (https://github.com/luin/ioredis/pull/378/commits/1df3da92f72f252a0b0e70ba3983ead18027b985)

Good point! Thank you for the patch.


Shakapacker gem and node package versions do not match

If, by mistake, both shakapacker and webpacker gems are installed, the following error will occur:
$ bundle exec rake shakapacker:compile
...
**ERROR** Shakapacker: Shakapacker gem and node package versions do not match
Detected: 7.0.3
     gem: 5.4.4
Ensure the installed version of the gem is the same as the version of
your installed node package.
Do not use >= or ~> in your Gemfile for shakapacker without a lockfile.
Do not use ^ or ~ in your package.json for shakapacker without a lockfile.

Expected behavior:
a better error message explanation (related to webpacker gem possible installation)
Actual behavior:
the error message is misleading (gem: 5.4.4 is the webpacker gem version, not shakapacker)
Small, reproducible repo:
Setup environment:

Ruby version: 3.2.2
Rails version: <IP_ADDRESS>
Shakapacker version: 7.0.3

"Better" is not a bug, it's an  "Enhancement."
Instead, Shakapacker should detect that webpacker is installed and raise an error during initialization, halting the program.

yes, it is an "enhancement", but I couldn't change the label, "bug" or "feature request" only are available.

The issue is always here, the error occurs when I have both gems in Gemfile:
...
gem 'webpacker'
gem 'shakapacker'
...

Here the whole output, maybe it helps:
$ bundle exec rake shakapacker:compile --trace
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/shakapacker-7.0.3/lib/shakapacker.rb:51: warning: already initialized constant Webpacker
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/webpacker-5.4.4/lib/webpacker.rb:6: warning: previous definition of Webpacker was here
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/webpacker-5.4.4/lib/webpacker/version.rb:3: warning: already initialized constant Shakapacker::VERSION
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/shakapacker-7.0.3/lib/shakapacker/version.rb:3: warning: previous definition of VERSION was here
** Invoke shakapacker:compile (first_time)
** Invoke shakapacker:verify_install (first_time)
** Invoke shakapacker:verify_config (first_time)
** Execute shakapacker:verify_config
** Invoke shakapacker:check_node (first_time)
** Execute shakapacker:check_node
** Invoke shakapacker:check_yarn (first_time)
** Execute shakapacker:check_yarn
** Invoke shakapacker:check_binstubs (first_time)
** Execute shakapacker:check_binstubs
** Execute shakapacker:verify_install
** Invoke environment (first_time)
** Execute environment
rake aborted!
**ERROR** Shakapacker: Shakapacker gem and node package versions do not match
Detected: 7.0.3
     gem: 5.4.4
Ensure the installed version of the gem is the same as the version of
your installed node package.
Do not use >= or ~> in your Gemfile for shakapacker without a lockfile.
Do not use ^ or ~ in your package.json for shakapacker without a lockfile.
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/shakapacker-7.0.3/lib/shakapacker/version_checker.rb:72:in `raise_differing_versions_warning'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/shakapacker-7.0.3/lib/shakapacker/version_checker.rb:52:in `raise_if_gem_and_node_package_versions_differ'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/shakapacker-7.0.3/lib/shakapacker/railtie.rb:13:in `block in <class:Engine>'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/railties-<IP_ADDRESS>/lib/rails/initializable.rb:32:in `instance_exec'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/railties-<IP_ADDRESS>/lib/rails/initializable.rb:32:in `run'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/railties-<IP_ADDRESS>/lib/rails/initializable.rb:61:in `block in run_initializers'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/3.2.0/tsort.rb:228:in `block in tsort_each'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/3.2.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/3.2.0/tsort.rb:431:in `each_strongly_connected_component_from'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/3.2.0/tsort.rb:349:in `block in each_strongly_connected_component'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/3.2.0/tsort.rb:347:in `each'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/3.2.0/tsort.rb:347:in `call'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/3.2.0/tsort.rb:347:in `each_strongly_connected_component'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/3.2.0/tsort.rb:226:in `tsort_each'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/3.2.0/tsort.rb:205:in `tsort_each'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/railties-<IP_ADDRESS>/lib/rails/initializable.rb:60:in `run_initializers'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/railties-<IP_ADDRESS>/lib/rails/application.rb:391:in `initialize!'
/home/ror/Documents/AptanaStudio3Workspace/konsys/config/environment.rb:5:in `<main>'
<internal:/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/zeitwerk-2.6.11/lib/zeitwerk/kernel.rb:38:in `require'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-<IP_ADDRESS>/lib/active_support/dependencies.rb:332:in `block in require'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-<IP_ADDRESS>/lib/active_support/dependencies.rb:299:in `load_dependency'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-<IP_ADDRESS>/lib/active_support/dependencies.rb:332:in `require'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/railties-<IP_ADDRESS>/lib/rails/application.rb:367:in `require_environment!'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/railties-<IP_ADDRESS>/lib/rails/application.rb:533:in `block in run_tasks_blocks'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/home/ror/.rbenv/versions/3.2.2/bin/rake:25:in `load'
/home/ror/.rbenv/versions/3.2.2/bin/rake:25:in `<top (required)>'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/cli/exec.rb:58:in `load'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/cli/exec.rb:58:in `kernel_load'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/cli/exec.rb:23:in `run'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/cli.rb:492:in `exec'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/cli.rb:34:in `dispatch'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/cli.rb:28:in `start'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.19/exe/bundle:37:in `block in <top (required)>'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/home/ror/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.19/exe/bundle:29:in `<top (required)>'
/home/ror/.rbenv/versions/3.2.2/bin/bundle:25:in `load'
/home/ror/.rbenv/versions/3.2.2/bin/bundle:25:in `<main>'
Tasks: TOP => shakapacker:compile => environment

Everything if fine after removing the 'webpacker' gem.
Unfortunately I cannot share the repo. If you're not able to reproduce it, please close the issue.

I tried to make a repo with both gems and npm packages installed. I didn't get such an error.
I closed the issue since it is not reproducible. If you could make a simple repo to demonstrate the issue, please let us know.
Thanks for reporting.


Ubuntu22-CIS can't logon after remediation

Question
I can;t logon anymore after I applied the UBUNTU22-CIS on an Ubuntu 22.04 running in Hyper-V. I'm trying to logon using the console.
I'm getting the error, sorry password authentication didn't work.

Hi @Acenl12
Thank you for raising the this issue and interaction on the discord community. I feel the biggest change we could make would be to make the default for the following settings
ubtu22cis_disruption_high: false

I will raise this work to take place in the next PR
Many thanks
uk-bolly

I ran into the same issue. A solution for us (as we want to use a central directory service which takes care of this) was to deactivate Ruleset 5.4.2
ubtu22cis_rule_5_4_2: false
I was unable to dig further but could identify this as the source of disruption.
Hope this helps.

Running it on ubuntu-server instead ubuntu-desktop then it worked fine. Thanks for the help so far.


kconfig: Correct format of KERNELVERSION

Change the format of KERNELVERSION as it exists in Kconfig to match
it's format elsewhere. Specifically, change from a string format
"1.10.3" to a numerical format.
This format change is a backwards-incompatible change. Any out-of-tree
Kconfig sources that were accessing KERNELVERSION with
$(KERNELVERSION) will need to be ported.
This change is to fix a bug. KERNELVERSION should have always been in
a numerical format. It should have been in a numerical format because;
KERNELVERSION is in a numerical format as a CMake variable and as a C
macro.
The purpose of having KERNELVERSION available in Kconfig is to have
configuration depend on the version, but this is only possible if it
exists in a numerical format[0].
[0] A string comparison of two semver strings does not always result
in a correct version comparison.

Codecov Report

Merging #10285 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #10285   +/-   ##
=======================================
  Coverage   53.05%   53.05%
=======================================
  Files         214      214
  Lines       26201    26201
  Branches     5645     5645
=======================================
  Hits        13901    13901
  Misses      10028    10028
  Partials     2272     2272

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7bf9de...c131d0a. Read the comment docs.

The environment variable 'KERNELVERSION' is set when running Kconfig,
but it is never read so we remove it.

How do you know it's never read? (Do we have a registry of all 3rd-party software written for Zephyr?) Even from quick look, someone might use it for doing conditionals in (3rd-party) Makefile (which someone trying to use Zephyr in production for sure would have to do, given all the churn we have.)

How do you know it's never read? (Do we have a registry of all 3rd-party software written for Zephyr?) Even from quick look, someone might use it for doing conditionals in (3rd-party) Makefile (which someone trying to use Zephyr in production for sure would have to do, given all the churn we have.)
-- @pfalcon

My main gripe with KERNELVERSION was that it had the wrong format. This has now been corrected,
(instead of removing the feature) . See the PR description.
Note that external build scripts would not be detecting the kernel version by checking a Kconfig environment variable, but would instead use another mechanism. The only users that would be checking the Kconfig environment variable are out-of-tree Kconfig sources, but they need a numerical format, not a string format as is used today.

Note that external build scripts would not be detecting the kernel version by checking a Kconfig environment variable, but would instead use another mechanism.

Ok, thanks for the reply. I personally don't remember a need to have Makefiles conditional on kernel version (but definitely had/have a need to have a code conditional on kernel version for example), so wanted to make sure such usecases considered.

@nashif : All review comments have been addressed. Please re-asses the review when convenient.


Support passing custom script to start_with_cloudinit

Currently, the cloud-init integration supports only a sub-set of
cloud-init commands. With custom_script support, one can pass
arbitrary cloud-init yaml to the managed host and make sure
the provisioned host will get it unchanged (which is not the case
when passing the user data currently, as the rbovirt parses into
ovirt-specific keys)
Example:
cloudinit = <<EOF
yum_repos:
    zoo:
        baseurl: https://example.com/yumrepo
        name: example-repo
        enabled: true
EOF

service.vm_start_with_cloudinit(:id =>id,
  :user_data => { :custom_script => cloudinit }
)

There is no way to achieve this without this patch.

Ok I can only merge without testing, looks good to me other than the nitpick. Please prepare the next rebase already tested and I will merge right away!

It was very useful nitpick, thanks @lzap for review, updated.

@lzap mind merging now?

Done.


feat: remove asset from create note

Closes https://github.com/0xPolygonMiden/miden-base/issues/596
Supersedes https://github.com/0xPolygonMiden/miden-base/pull/614
I remove ASSET from the create_note procedure and the subsequent procedures. I also needed to adjust the event handler.

One other thing I think we should do is add a comment here to say that if there are no assets, [ZERO; 4] is returned as the asset hash.


Android API 21,22  CRASH  on simulator

crash after re dlopen libunity.so, test on simulator;

another app run ok on these simulator

(ps: fixed by preload in native c++ code)


Digital shadows solution v2.0.0

Change(s):

package creation

Reason for Change(s):

New package creation script used
package v2.0.0

Version Updated:

Yes

Testing Completed:

Yes

Checked that the validations are passing and have addressed any issues that are present:

Yes

@microsoft-github-policy-service agree [company="Digital Shadows Searchlight"]

@microsoft-github-policy-service agree [company="Digital Shadows Searchlight"]

Hey @yashwanth-metron, I can see that just version is updated but the package is still created with Version 1 of solution tool.

@yashwanth-metron, please refer to this document, to create it with version2 od solution tool.

Hey @v-sabiraj, The package was created using Github/Azure-Sentinel/Tools/Create-Azure-Sentinel-Solution/V2/createSolutionV2.ps1 solution tool.

Hey @yashwanth-metron, we will check on this.

Hey @yashwanth-metron, can you please update the latest zip package, thanks.

Hey @v-sabiraj PR has been created and is under review.

Hey @v-sabiraj PR has been created and is under review.

I don't see the changes are made here, is there any other PR raised for this?

Hey @v-sabiraj, The PR is raised in the forked repo and it is still under review.

@v-sabiraj will update you once it is approved and merged.

@v-sabiraj will update you once it is approved and merged.

Ok, thanks

Hey @v-sabiraj, merged the changes.

Hey @v-sabiraj, Please check now.

Let me check and merge this, thanks.


reuse existing client

This PR is an attempt to reuse the same client across S3 requests.

Coverage decreased (-0.3%) to 91.239% when pulling d9e03a867bf1c391160d28ca1397c8bbb1ae718f on reuse-existing-client into 373cbe450006813595c973cba44e02478d5b215e on main.

Does it matter that we restart the client? From what I understood from the docs the client should be re-created every time (it does not hold any connections, does it?).

Does it matter that we restart the client? From what I understood from the docs the client should be re-created every time (it does not hold any connections, does it?).

Based on this comment, it seems that the client can reuse the connections https://github.com/aio-libs/aiobotocore/issues/928


Support Playwright 1.40 (TimeoutError became Error)

Playwright 1.40 is out and the playwright team again changed slightly their error messages.
Unfortunately TimeoutError are now just Error .
Still in the error message you can get that it is a timeout error.
So that may have some backwards incompatibilities if you expect a TimeoutError.

will be fix #3250


Does the demo code runs on windows? + some questions about how to train in a new dataset

Hi, Thank you for sharing the code.
I would like to try your code on my machine. However, I am having problems trying to install pytorch in my windows PC. Actually, conda only has pytorch 0.1.12, not 0.2.0, torchvision as you said it was required. Any advice for me about this.
I am highly interesed on apply machine learning and cnn to solve traditional pattern recognition problems which were solved using generalized hough transform, ranscac and others. If the input image and dataset are easier samples such as brand logos in whiter or black background but with a  little bit of noise for example, do you think the accuracy will be higher than the results obtained in your papers using images of clustered environments?
What will be the best way to create the annotations of the transformation matrix (train and test csv files) feature points (test_pairs_pf csv file) for a new dataset?
Also, it will be nice to see this code running on tensorflow. Do you think it will be possible to convert this code into tensorflow and get the same accuracy?
Thanks in advance.
Update:
I asked directly to pytorch support and they replied really fast and helped me to install everything ok in windows and finally demo code runs without problem. I hope you still can reply other questions.

Hi, sorry for the late reply.
It's great you could get PyTorch running, and that the code is working. I'm not sure about the other questions and they are not issues of this repo, so I'm closing this thread.


Fail on invalid yaml

This pr attempts to fail on invalid yaml. Prior to this commit invalid yaml would be silently ignored and would not render in the base.
Including the following yaml file in a release would render one file base/myyaml-1.yaml. There would be no messaging to the user and the installation would proceed.
With the addition of this commit the user will clearly see an error failed to unmarshal... in the ui when installing the release.
myyaml.yaml
{this is not valid yaml}
---
this: is valid
---
(nor is this}

@emosbaugh I'm trying to fully understand the implications of this:
We'd silently ignore all files that don't have a .yml or .yaml extension. But kots would fail if a file did have a valid yaml extension but wasn't unmarshallable?
How does KOTS fail in the second scenario? Would you be able to include an example of the user experience here?

@marccampbell description updated

I think a more compatible approach here could be to change this to use the old behavior, store the unmarshallable files, allow the deployment to continue, and show the warnings in the Admin Console / CLI?
Something like
kots pull ...
...
Completed with warnings. The following files were not able to be parsed and were not included in the application base:

file1.yaml
file2.yaml

@marccampbell I will have to return warnings from the base package along with errors for functions base.RenderUpstream, base.RenderHelm and base.Base.WriteBase. The return signature would look something like (Warnings, error). That is the only way i can achieve the style of warnings you are asking for. This will affect the public function signatures. Is that a direction you would be ok with? If so I will write up a proposal.


crypto/tls: timeout tests

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

tip

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

run the crypto/tls tests.  Occasionally, I get the error:
--- FAIL: TestDialTimeout (0.01s)
tls_test.go:192: resulting error not a timeout: dial tcp <IP_ADDRESS>:49575: i/o timeout
The test used to detect whether an error is a timeout is to look for the string "timed out" in the error string.  Maybe we should include "timeout" also?

I can't reproduce but I sent https://go-review.googlesource.com/20403 which at least also still passes for me and will fail with more info if it continues to flake for you.


No support for $id which are not directly absolute URIs / breaking at validation time

Describe the bug
If the value of $id is not an absolute URL, resolving $ref-s fails. This seems to have worked with ManateeJson; if removing this feature is intentional, we should probably still fail on de-serialization (i.e. validation) of the schema, instead of getting an exception when attempting to resolve the $ref when using the schema to validate another json.
To Reproduce
using System;
using System.Text.Json;
using Json.Schema;
using Manatee.Json;

using ManateeJsonSerialiazer = Manatee.Json.Serialization.JsonSerializer;
using ManateeJsonSchema = Manatee.Json.Schema.JsonSchema;

namespace test
{
    class Program
    {
        static void Main(string[] args)
        {
            var schemaStr = @"
{
  ""$schema"": ""http://json-schema.org/draft-07/schema#"",
  ""$id"": ""mySchema.json"",
  ""properties"": {
      ""abc"": { ""$ref"": ""mySchema.json"" }
  },
  ""additionalProperties"": false
}";
            var jsonStr = @"
{
  ""abc"": {
    ""abc"": {
        ""abc"": ""abc""
    }
  }
}";
            var serializer = new ManateeJsonSerialiazer();
            var manateeSchema = serializer.Deserialize<ManateeJsonSchema>(JsonValue.Parse(schemaStr));
            var manateeJson = JsonValue.Parse(jsonStr);
            var manateeValidation = manateeSchema.Validate(manateeJson);
            Console.WriteLine("Manatee: {0}", serializer.Serialize(manateeValidation));
            var schema = JsonSerializer.Deserialize<JsonSchema>(schemaStr);
            var json = JsonDocument.Parse(jsonStr).RootElement;
            var validation = schema.Validate(json);
            validation.ToDetailed();
            Console.WriteLine("JsonSchema.NET: {0}", JsonSerializer.Serialize(validation));
        }
    }
}

This throws an exception on the Validate line - instead of throwing it at the Deserialize one, or accepting this.
Expected behavior
I would expect partial URIs to be valid in $id position, valid for use within the same document, and ideally also resolving to the base URI where the schema was obtained (allowing support across documents which share a base URI, even if it isn't specified inside the schema).
-OR-
If using partial URIs is strictly forbidden, I would expect this to be validated on schema de-serialization.
Also, as a side issue - I think it would also be better for $ref-s to be resolved on de-serialization whenever possible. I'm not sure if this is possible for $recursiveRef from draft2019-09, as the base URI can change there; but for $ref it is fixed to the definition site, so this should be possible.

It seems I need to bring over the "default base URI" option.  I thought I had.  Not sure why the $ref isn't resolving.  The $id should register that URI which the $ref then looks up.  I'll add a test.
Regarding throwing on deserialization, this is not possible. You've provided a valid URI.  I can't know at deserialization time whether I'm at the root schema to try to enforce an absolute URI.
Also, $ref resolution at deserialization isn't possible because I haven't fully deserialize the JSON structure yet, and I may not have those locations yet.
I'll look at adding the default base URI option.

Sorry for not providing the (informative) stack trace. It fails at the UriExtensions method of GetParent, because it attempts to get the property AbsoluteUri on a non-absolute URI.
About errors at deserialization time, I understand your points. I guess these things then could be done afterwards as some kind of internal validation step. This could be a useful feature, for guaranteeing that validating a json can't throw an exception; but this is then a completely new feature, and since $recursiveRef couldn't probably be supported in the general case, it's probably not worth the trouble.
Thanks a lot for your work on this library and the quick responses!

I checked this, and this still seems not to work with cross-file references. You can see a failing test in my branch: https://github.com/amosonn/json-everything/tree/test-1
I didn't open a PR because I wasn't sure what's the right way to resolve this; I'm not sure what's the point of the logic regarding newUri vs newBaseUri in RefKeyword, but the bottom line problem is that the newly joined URI is only used in SchemaRegistry.Get, and this doesn't attempt to fetch new resources.

The link you provided isn't leading me anywhere specific, just to the fork.
A non-absolute URI isn't going to be able to have references outside of the file because it has no knowledge of where the file is.  You can set the base URI or provide a download func in the validation options.

It takes to the branch with the new test, which is one commit ahead of your master. Here's the link to the diff directly: https://github.com/gregsdennis/json-everything/compare/master...amosonn:test-1
After your previous changes relevant to this topic, if the schema is created from a file, the path to this file is maintained as the base URI for this schema, and I would expect the reference to look under this path. I checked using debug prints, and the code in RefKeyword does calculate the correct absolute URI from the relative one in this case; it just doesn't attempt to fetch this resource.
I guess I haven't seen the code which fetches schemas with http-based URI-s, either. Does this mean that they need to be preloaded as well?

They either need to be preloaded or set up the download function in the options to retrieve them.

Ok, thanks! I created a PR making setting up this function more ergonomic.

My apologies. The download method isn't directly on the options. It's on the registry, which the options carry an instance.

My apologies. The download method isn't directly on the options. It's on the registry, which the options carry an instance.

I continued hacking at this, now I have trouble with relative $ref-s where the $id is not absolute and the Schema is received from a web server instead of a file. Whereas FromFile registered the given filename as BaseUri, there's no similar way to register a web URI as the base URI of the schema.
The obvious option is to add another ctor for this, or to make this property publicly-settable. But another possibility, which also provides some nice ergonomics, is to have the SchemaRegistry set this up on registration; I would definitely expect a JsonSchema which was registered under a given Uri to use it as the base for resolving others.

There's an option that allows you to set a default URI.  Change this to your web server.

I would definitely expect a JsonSchema which was registered under a given Uri to use it as the base for resolving others.

It does this.

By "registered under", I meant something along the lines of:
var schema1Str = @"
{
  ""$schema"": ""http://json-schema.org/draft-07/schema#"",
  ""$id"": ""schema1.json"",
  ""properties"": {
    ""abc"": { ""type"": ""string"" }
  }
}";
var schema2Str = @"
{
  ""$schema"": ""http://json-schema.org/draft-07/schema#"",
  ""$id"": ""schema2.json"",
  ""$ref"": ""schema1.json""
}";
var uri1 = new Uri("http://example.com/schema1.json");
var uri2 = new Uri("http://example.com/schema2.json");
var jsonStr = @"{ ""abc"": ""s"" }";

var schema1 = JsonSerializer.Deserialize<JsonSchema>(schema1Str);
var schema2 = JsonSerializer.Deserialize<JsonSchema>(schema2Str);
var json = JsonDocument.Parse(jsonStr).RootElement;

var options = new ValidationOptions { OutputFormat = OutputFormat.Verbose };
options.SchemaRegistry.Register(uri1, schema1);
options.SchemaRegistry.Register(uri2, schema2);

var schema = options.SchemaRegistry.Get(uri2);
var validation = schema.Validate(json, options);
Console.WriteLine("JsonSchema.NET: {0}", JsonSerializer.Serialize(validation));

With or without $id. The schema only knows its relative path, but I know the absolute one when I'm fetching it.
Setting the default URI is a workaround. What if I have two web servers I'm using, each of them with relative schema-s pointing inside it?
The JsonSchema draft specifies:

RFC3986 Section 5.1 defines how to determine the default base URI of a document.
Informatively, the initial base URI of a schema is the URI at which it was found, whether that was a network location, a local filesystem, or any other situation identifiable by a URI of any known scheme.
If a schema document defines no explicit base URI with "$id" (embedded in content), the base URI is that determined per RFC 3986 section 5.

But for this to work, I should be able to tell the library the URI "at which it was found", which I currently can't (unless this URI is a file). It seems to me that using the SchemaRegistry.Register method (directly or by Get calling Fetch) should be a good way to tell the library this URI, because it will continue being found under this URI in the future.

Bumping this, in case it got lost. Should I open a new issue for this? This is still somewhat related to the original issue, but is also somewhat different.

As I've already stated, the combination of the default base URI (this is per validation pass, not global) and being able to define a fetching method covers this.  I'm not making further changes in this regard unless you can devise a test that this library cannot be configured to pass.

Alright, I wrote the more complicated test I had in mind, and you're right, it works :) (though the diagnostics are slightly off - in some places the absoluteKeywordLocation does not point to the schema but rather to its parent. I'll try to make a PR later).
I still feel that the ergonomics around having to keep both the JsonSchema and the ValidationOptions around are a bit awkward, but I'll open a new issue for that sometime later.
Thanks for your replies!


Ih: new project exploration

{
"step_name": "Project Exploration",
"isRequired": true,
"prompt": "Using the provided JSON format as a reference, analyze the user's earlier input about the project to identify key objectives, desired functionalities, and any constraints or limitations. Ensure that the output follows the JSON structure: {  "project_name": "",  "objectives": [    ""  ],  "functionalities": [    ""  ],  "constraints": {    "resources": [      ""    ],    "technology": [      ""    ],    "other": [      ""    ]  }}. Fill in any missing information to complete the task."
}

With user input:
{
"step_name": "Project Exploration",
"isRequired": true,
"prompt": "User's Input: [Insert User's Input Here] \n\nAnalyze the user's earlier input about the project and provide information in JSON format. Ensure that the output follows the JSON structure: {  "project_name": "",  "objectives": [    ""  ],  "functionalities": [    ""  ],  "constraints": {    "resources": [      ""    ],    "technology": [      ""    ],    "other": [      ""    ]  }}. Fill in any missing information to complete the task."
}


Initial MR4 compiler

Initial MR4 compiler with that only implements existing features from MR3.

@Nurdok, please review this.


RSpec/EmptyLineAfterSubject triggers on factories

I have the following FactoryBot factory:
factory :announcement do
  subject { 'Subject' }
  message { 'Message' }
end

And of course I get:
Add empty line after `subject`

I think the rule can be applied by default only to files matching the following name pattern: spec/**/*_spec.rb which of course would exclude spec/factories.rb.
That is also the default rspec filename filter AFAIK.

Nice catch. I think there was a similar problem in the past, or there's still an open ticket.
Would you like to tackle this?
I imagine you'd have to tweak this logic https://github.com/rubocop-hq/rubocop-rspec/blob/master/lib/rubocop/cop/rspec/cop.rb#L43 or maybe take a look at the other RuboCop extensions how they manage to do this.
Last time I've checked, https://github.com/rubocop-hq/rubocop-minitest didn't have anything related.
Another simpler option would be to tweak the pattern here https://github.com/rubocop-hq/rubocop-rspec/blob/master/config/default.yml#L6

For the reference https://github.com/rubocop-hq/rubocop-rspec/issues/703


Messaging doc updates

Description

combine push and iam docs for better clarity
add release notes for iOS Messaging 1.1.1 and Android Messaging 2.0.2

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have signed the Adobe Open Source CLA.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Shall we go ahead and merge this?

realized i forgot to push my review update for steve's comment. i'll get it pushed and resolve the merge conflict.


[Question] High Load Issue

Question
Hi,
I am experiencing an issue and this issue is frequently happened.
I get High load on my CPU and Memory and to fix this issue i need to restart my container ( i run hertzbeat as docker) and always the service which has high load is java -server -XX:SurvivorRatio=6 -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/hertzbeat/logs -Dlogging.path+
I also found that i get this logs from my logs file,you can refer to the link below :
https://pastebin.com/emMCxJn6
Please Advice
Thanks in advance

thanks for feedback.

what is the hertzbeat version, can you give us the OOM HeapDump file xxx.hprof to analyze

Hi,
Thanks for your reply
We are using HertzBeat v1.2.5 with latest image docker
As per i check the file is quite big if you need to check it i will try to upload the file from the cloud like google drive
-rw------- 1 root root 1.1G Mar  4 00:53 java_pid10.hprof
-rw------- 1 root root 902M Feb 25 19:56 java_pid9.hprof
Best Regards,

Hi @tomsun28,
You can download the .hprof from the link below :
Download
Thanks in advance

hi please upload java_pid9.hprof. we need it thanks.

Hi @tomsun28,
You can download the .hprof from the link below :
Download
Thanks in advance

thanks

Hi @tomsun28,
You can download the .hprof from the link below :
Download
Thanks in advance

thanks

You're welcome

@muslimcodeid hi, thanks for feedback and your hprof file. This is an oom bug due users not config tdengine database or iotdb database to store metrics data. We have fix it in latest docker image now. You can pull latest docker images from https://hub.docker.com/r/tancloud/hertzbeat/tags
Thanks 👍

Hi @tomsun28,
I have pulled the latest image
Thanks for your help


Add PropTypes to components.

Thought this might be helpful during development :)

@leebyron i think that'll do it.

Dope


Cannot search for multiple words in an elasticsearch object

package main

type Opts struct {
    String string  `short:"s" long:"string" description:"String to search for" required:"true"`
    Rate   float64 `short:"r" long:"rate" description:"Maximum rate before going critical" required:"true"`
}

var opts *Opts

func main() {
    opts = &Opts{}
    _, err := flags.Parse(opts)
    if err != nil {
        log.Fatal(err)
    }

    index := "syslog-2016-02-04"

    // Look for all of the relevant elasticsearch documents
    termQuery := elastic.NewTermQuery("message", opts.String)
    res, err := client.Search().
        Index(idx).
        Query(&termQuery).
        Size(50).
        Type("events").
        Do()
    if err != nil {
        log.Fatal("Could not fetch from Elasticsearch: %#v", err)
    }

    if len(res.Hits.Hits) == 0 {
        log.Fatal("No results")
    }

    for count, hit := range res.Hits.Hits {
        log.Printf("%s", *hit.Source)
    }
}

When I run this program such as:
$ ./foo --string "found preference" -r 0

I expect it to match any document in the syslog index with the string "found preference" in it. However this does match if I change the input string to "preference". How can I search for multiple words?

@Xe Hi there. Well, this is more of an Elasticsearch issue, but let me explain nevertheless.
ES is not a database. ES and the underlying Lucene technology deals more with human language which is somewhat subtle and ambiguous, and very often not about exact matches. ES will happily find near matches, documents that sound like those your original query, or words similar to those specified in your query. It will apply a score to the documents and return them in order. E.g. if "found preference" is an exact match with a perfect score of 1.0, what score would "preference" get, and will it be the same or different than a search for "found"? And what about "find": It's the same word as "found", only in present tense, so should it get the same score? ES/Lucene can handle these things.
So if you have specific requirements on how things should work, you must help ES to correctly deal with it. The key here is to provide a so-called mapping for your index. Such a mapping describes the fields of your documents, how to analyze and tokenize them, and how to index and search them.
That's such a broad topic that it will not fit into a GitHub issue. But don't worry: Core members of Elasticsearch have written a free book that has all the details: Elasticsearch - The Definite Guide. Go check it out, then take a look at the documentation, then apply your knowledge to Elastic (read the tests to illustrate how to make Go code out of the JSON structures). I'm sorry: There is no shortcut :-)
HTH.


[WIP #636] OSX - Shell script to open oni from command line

This change implements an OSX shell script to be symlinked via the Add to Path gesture. This is the first step, but now allows passing in the working directory via the ONI_CWD variable, as well as handling arguments correctly in this case (and not swallowing input from the terminal).
There are still some issues to address, as the current makeSingleInstance lets us get new arguments, but doesn't let us see the working directory / environment variable for the incoming process.

May need to combine with a fix for #679 , so that the working directory can be passed around.


microsoft.insights registration issue

Describe the bug
unable to register the Resource Provider microsoft.insights on one of our client tenants. Stuck on Registering status indefinitely. Subscription ID = d39e851a-2331-441b-a05f-08b8f7951e9a
Related command
Register-AzResourceProvider -ProviderNamespace microsoft.insights
Also tried the same command using AzureRM
Also tried in the Azure portal
Errors
Subscription Registering Resource Provider
'Register' operation check timed out on Resource Provider 'microsoft.insights', please refresh resource providers list to check for registration status
20 hours ago
Subscription Unregistering Resource Provider
'Unregister' operation check timed out on Resource Provider 'microsoft.insights', please refresh resource providers list to check for registration status
20 hours ago
Issue script & Debug output
az-cli-log.txt
ProviderNamespace : microsoft.insights
RegistrationState : Registering
ResourceTypes     : {components, components/query, components/metadata, components/metrics...}
Locations         : {East US, South Central US, North Europe, West Europe...}
DEBUG: 8:14:41 AM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning
default value [True].
DEBUG: AzureQoSEvent:  Module: Az.Resources:6.7.0; CommandName: Register-AzResourceProvider; PSVersion: 5.1.19041.3031;
IsSuccess: True; Duration: 00:00:10.9662072
DEBUG: 8:14:41 AM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning
default value [True].
DEBUG: 8:14:41 AM - RegisterAzureProviderCmdlet end processing.
Expected behavior
I expect the Resource Provider to register like it should
Environment Summary
PS C:\Windows\system32> Get-InstalledModule -Name Az
Version              Name                                Repository           Description

10.0.0               Az                                  PSGallery            Microsoft Azure PowerShel
Additional context
No response

Thank you for opening this issue, we will look into it.

Please open this issue in azure-powershell repo.

Please open this issue in azure-powershell repo.

Hi @yonzhan , actually this is also happening on azure-cli:
`az provider show --namespace microsoft.insights -o table
Namespace           RegistrationPolicy    RegistrationState

microsoft.insights  RegistrationRequired  Registering`
Same behaviour as described by @resilientit for nearly 24 hours stuck in Register state.

I am also having this problem. It has prevented our first release unfortunately


sys: Update lazy_static to v1

Coverage remained the same at 52.125% when pulling 8af7a96e510b4fa731b758608944ebef99131f52 on chetgurevitch:master into 65ee31202d22cd2849752956a99a243f1976ad39 on Smithay:master.

Thanks!


Honour new relative git URL behaviour controlled by CMP0150

I added support for relative git URLs to FetchContent/ExternalProject which was introduced in cmake v3.27 under policy CMP0150. Unfortunately, this doesn't seem to work when wrapped with CPM and it looks like this is because CPM minimum cmake version is set to 3.14.
I'm not sure on the best strategy here, but one option would be for CPM to inherit the policies from the parent project, and just assert that they're set at least to a minimum of 3.14 as CPM needs, rather than actually explicitly setting them to 3.14 as is currently the case. Alternatively, it should at least honour policy 150 and pass it through if it can.
Happy to submit a PR for this if there's a preferred solution.

Hey, thanks for raising the issue, I agree that it would make sense for CPM to honour the new policy. Currently we handle overriding policies inside a dedicated macro (see here for discussion), which would be the preferred location to enable CMP0150. Would be very happy to review a PR for this!

Sure thing, I'll submit a PR!


Added more nginx commands, since my server needed a 'nginx:start'

The following commands are now available for nginx:
cap ENV nginx:start
cap ENV nginx:stop
cap ENV nginx:reload
cap ENV nginx:configtest
cap ENV nginx:status
cap ENV nginx:force-reload
cap ENV nginx:upgrade
cap ENV nginx:restart
cap ENV nginx:reopen_logs

@erwald I chose to target your branch, because your changes work for an Amazon Linux.

@jesperronn Sure, looks good. I'll also take the opportunity to mention that my changes were very much based on those made in this fork.

Ok thanks. Have made own fork now since I think it makes sense to release separately.
Btw ran into problem with unicorn in my setup using rvm1-capistrano3.
But I better document that separately.
Sent from my iPhone

On 30. jan. 2016, at 13.08, erwald<EMAIL_ADDRESS>wrote:
@jesperronn Sure, looks good. I'll also take the opportunity to mention that my changes were very much based on those made in this fork.
—
Reply to this email directly or view it on GitHub.


ReplacePointerBitcastPass introduces invalid IR with struct field of a struct

This one is quite complex!  The original code is a recursive BiQuad filter running on custom vectors and parallel distributors, with 4 passes atomically synced in one kernel.  Everything runs fine, and at top speed!
https://godbolt.org/z/ca81s5Yer
Clspv's  ReplacePointerBitcastPass  gets confused when a new field of "struct.beyond::SS" type is introduced to the custom image "class.beyond::Samples2D".
%"class.beyond::A" = type { i64 }
%"class.beyond::Samples2D" = type <{ %"class.beyond::A", %"struct.beyond::Vector2D", %"struct.beyond::Vector2D.0", %"struct.beyond::Vector2D.0", %"struct.beyond::SS", [4 x i8] }>
%"struct.beyond::Vector2D" = type { i32, i32 }
%"struct.beyond::Vector2D.0" = type { float, float }
%"struct.beyond::SS" = type { i32 }

Now this field is not yet used anywhere in the code, but its mere existence triggers ReplacePointerBitcastPass to emit invalid bitcasts;
Invalid bitcast
%5185 = bitcast <2 x %"struct.beyond::SS"> %5184 to i64

https://godbolt.org/z/5e1EKT3cM
This is emitted from ReplacePointerBitcastPass.cpp ReduceType()   if (!IsGEPUser) {...}
And subsequently Clspv crashes; "LLVM ERROR: Broken module found, compilation aborted!"
Strangely, the problem goes away introducing an i8 field at the end of "class.beyond::Samples2D" with automatic padding;
%"class.beyond::Samples2D" = type <{ %"class.beyond::A", %"struct.beyond::Vector2D", %"struct.beyond::Vector2D.0", %"struct.beyond::Vector2D.0", %"struct.beyond::SS", i8, [3 x i8] }>

https://godbolt.org/z/9Kjfb81Pc
another location works too, and no padding;
%"class.beyond::Samples2D" = type { %"class.beyond::A", %"struct.beyond::Vector2D", %"struct.beyond::Vector2D.0", %"struct.beyond::Vector2D.0", i8, %"struct.beyond::SS" }

https://godbolt.org/z/P96TbETKr
Also works, adding a second field inside SS;
%"struct.beyond::SS" = type { i32, i32 }
https://godbolt.org/z/M99b8P9vs
but not 3 fields, which Clspv compiles, but it crashes Vulkan;
%"struct.beyond::SS" = type { i32, i32, i32 }
https://godbolt.org/z/dYK1KM8qE
but 4 fields works again;
%"struct.beyond::SS" = type { i32, i32, i32, i32 }
https://godbolt.org/z/9M6a1edoz
Again, none of these fields, or the struct SS are yet used in the program, yet they bring out the dormant bugs from ReplacePointerBitcastPass.
I hope you like a good challenge! Good luck!

This first godbolt link is compiling fine now. Should we close this issue?

Well, the first one was always compiling as a baseline.  It is the second one crashing Clspv, and the second from the last one that was crashing Vulkan.
After we applied the i8 Canonicalization of GEPs in our custom branch, Clang and LLVM became able to inline and eliminate more of these structure definitions which were triggering bugs in ReplacePointerBitcastPass.
A bit more on that here: https://github.com/google/clspv/issues/1292#issuecomment-2002059632


Capturing things outside the env

Just listened to your talk :)
I asked about the captured bound identifiers which are not part of the env because I cannot make this snippet (scala-cli) fail at compile time:
//> using lib "com.phaller::spores3::0.1.0"

import com.phaller.spores.Spore
import com.phaller.spores.{Spore, SporeData, PackedSporeData}
import com.phaller.spores.upickle.given

val str = "anonymous function"
val BLA = "hello"
val s = Spore(str) { // `str` is the environment of the spore
  env => (x: Int) => x + env.length + BLA.length
}

val z = 25
object MySpore
    extends Spore.Builder[Int, Int, Int](env => (x: Int) => env + x + 1 + z)

import upickle.default._

@main def hello =
  val x = 12
  val data =
    SporeData(MySpore, Some(x)) // `x` is the environment, as before
  val pickledData = write(data)
  val unpickledData = read[PackedSporeData](pickledData)
  val unpickledSpore = unpickledData.toSpore[Int, Int]
  assert(unpickledSpore(3) == 16)

I think this is hard to impossible to do in a macro at the moment: https://github.com/scala/scala3/discussions/19349


Schema Redesign

Updated JSON schema to remove ambiguity around what constitutes a "fingerprint," and allow for us to more easily add more attributes:
Single "Fingerprint" Object
{
    "name": "ASUS RT-N66U",
    "created": "2015-09-07 01:02:10.229153",
    "resources": ["/images/New_ui/asustitle.png","/images/loading.gif","/images/alertImg.png","/images/New_ui/networkmap/line_one.png","/images/New_ui/networkmap/lock.png","/images/New_ui/networkmap/line_two.png","/index_style.css","/form_style.css","/NM_style.css","/other.css"],
}

name is a string
created is a UTC timestamp when the fingerprint was created
resources is a list of strings, which are the paths of web resources we're going to attempt to scan for

List of Fingerprints
 [
    {
        "name": "ASUS RT-N66U",
        "created": "2015-09-07 01:02:10.229153",
        "resources": ["/images/New_ui/asustitle.png","/images/loading.gif","/images/alertImg.png","/images/New_ui/networkmap/line_one.png","/images/New_ui/networkmap/lock.png","/images/New_ui/networkmap/line_two.png","/index_style.css","/form_style.css","/NM_style.css","/other.css"],

    },
    {
        "name": "Linksys WRT54G",
        "created": "2015-08-07 01:02:10.229153",
        "resources": ["/UILinksys.gif","/UI_10.gif","/UI_07.gif","/UI_06.gif","/UI_03.gif","/UI_02.gif","/UI_Cisco.gif","/style.css"],

    },
]

+1 for the new schema. An fingerprint object would allow the fingerprint to be more flexible like specifying IP addresses or ranges for a fingerprint, this could be useful for #10. It could also by used to specify the device is only accessible over https or to add a separate version field to help automate post discovery actions like exploitation.
 [
    {
        "name": "Some device",
        "version": "1.2.3",
        "created": "2015-09-07 01:02:10.229153",
        "resources": ["/someresource.css"],
        "address": ["<IP_ADDRESS>"],
        "protocol": "https",
    },
]

Yep this is a good idea - I'll start coding on it today (thanks @moloch-- and @x0e-foobar)

Changed scheme to reflect what @moloch-- suggested, will add more functionality for things such as protocol and address soon :smile:
Google Chrome fingerprint generator update should be approval in an hour or two.

Adding a version is also a very good idea


Scalability of large subsets

In the current BOLD database, nearly all taxonomic families are small enough to be tractable for the tree inference engine, raxml. The rule of thumb that is followed, is that raxml ought to be able to handle up to ±10k input sequences. Since the tree searches are topologically constrained, the overall search space is significantly reduced, meaning that this may indeed be a realistic rule. Nevertheless, a small number of families (<10) exceed this size. These families have not been attempted yet. As a first step, this needs to be tested. If there are problems, these families will need to be partitioned further, e.g. down to the subfamily or genus level. This will have implications for the parallelization strategy (see #99).
This issue is considered 'done' when the BOLD 10M data set has been processed successfully.

@AnnemiekeSchonthaler, could you attach the plots you generated for the Lepidoptera, showing the running times?

@rvosa I attached the plots in the doc folder on the lepidoptera branch.


Update Comparison with Other Frameworks

As promised in #2685 I have updated the framework comparison table with (almost?) every actively developed deep learning framework and several new axes of comparison. Let me know if anything seems inaccurate or irrelevant.

The errors that occurred in the Travis CI are now fixed in the master branch (#2716)

I am sorry to make you wait. We are now working on the tasks related v2, which will be released 30th May. and could not take time to review it. Is it OK to review it after the release of v2?

No problem. Looking forward to v2 release!

Sorry for making you wait. Now I can work on this PR.

I was wondering if we might want to include another row in the table for acceleration using anything other than cuDNN (such as OpenCL and/or other hardware platform support), but I am not aware of any frameworks with actively-developed support.

Framework
assignee

Chainer

PyTorch

TensorFlow

Theano-based

Caffe1/2

Torch7

MXNet

DyNet

PaddlePaddle

DL4J

CNTK

neon

Knet.jl

Darknet

Thinc

@jekbradbury Could you let us know it if you had some idea in mind which framework we should pick up or not in making the table? (I should have discussed the matter before assigning reviewers and starting to review each column.).

I tried to include every actively developed full-featured deep learning framework that isn't just a wrapper or frontend around another framework. All of them are open source and either used fairly widely by researchers or supported by a major company, with the exception of Darknet, which is prominent in the area of visual object detection and useful if you want to use pure C.
BTW I'd describe non-CUDA GPU support as:
Chainer, Theano, DyNet, PaddlePaddle, DL4J, CNTK, Knet, Darknet, Thinc: CUDA-only for the foreseeable future
TensorFlow: Codeplay has a port to ComputeCPP, their proprietary SYCL-based OpenCL frontend; some subset of that has been merged into core
Torch7, TensorFlow, eventually PyTorch: Hugh Perkins has Coriander-based ports to OpenCL 1.2 but Coriander has performance drawbacks
PyTorch, Caffe1/2, Torch7, MXNet, eventually TensorFlow: AMD has in-progress HIP ports (this is not OpenCL, it's a CUDA-like framework that can cross-compile to both NVidia CUDA and AMD's ROCm API); these use MIOpen, AMD's upcoming cuDNN clone, and are likely to be performance-competitive but rely on AMD developers doing significant porting work and then either maintaining forks or getting their PRs accepted.

Thank you for your comment on the choice of DL frameworks. It seems reasonable for me.
For the support of non-CUDA GPU, I have no idea for now to what extent we should mention non-official forked repositories as we can expect there are many forks that widen the non-CUDA GPU support and it could be difficult to determine where to draw a line.

Could you let me know what's the difference between "full" and "partial" in CNNs/RNNs rows?

It's not particularly precise, but essentially I wanted to distinguish between frameworks that aim to support all major variants/uses of CNNs/RNNs (it may not be easy to write them in the framework, but it is at least possible) and frameworks that have more limited support. For example, Caffe was not designed with NLP in mind and their RNN support is not flexible or customizable; DyNet is an NLP-focused framework that recently added basic convolution and pooling layers but wouldn't be a good choice if you want to write a complex computer vision model. A few frameworks don't intend to support certain use cases at all (e.g. Thinc is only for NLP and Darknet is only for computer vision).

Also, I don't think we need to mention non-CUDA at least until AMD officially announces their ports (which will be the first performance-competitive, well-supported deep learning frameworks for non-NVidia GPUs) -- right now they're still in progress on GitHub.

I checked the table for MXNet, Torch7, and Thinc and think it looks fine. I did not see any inaccurate information.
I also agree that there is no need to mention non-CUDA for now, since the AMD ports are apparently still in progress.

Thank you for the PR! Could you tell me what "Per-batch architectures" means?

That row was in the original version of the comparison table. It means that the framework is capable of building a totally different network structure for each batch; that's essentially the same thing as define-by-run but it emphasizes what you can do with it.

Oh, sorry, I didn't notice that. OK, now I understand it! Thank you for the kind explanation :)

So, how about changing the row title to "Different architectures per-batch"?

That works. I believe (grammatically) it should be "Different architectures per batch" without the hyphen.

Hi, these are my comments for my assignment. Please point me out if anything is incorrect.
General:

What are the differences among "Multi-GPU ~ parallelism", "Multiprocessing", and "Distributed training"?
In "CPU/GPU backend", "custom" could be misunderstood as that "users can use their own custom backend". How about writing as "native" instead?
It's better to put a link to the source if any.

Theano-based:

In "Higher-order grads", it seems to only support Hessian (http://deeplearning.net/software/theano/tutorial/gradients.html). How abount writing as "Only Hessian"?

"Multi-GPU dataparallelism seems to be experimental" (https://github.com/Theano/Theano/wiki/Using-Multiple-GPUs). How about writing as "Experimental"?

It seems to have native trainer. (https://github.com/kirk86/theano/blob/master/trainer.py)

Caffe1/2:

It also has MATLAB binding (only Caffe1, though). (http://caffe.berkeleyvision.org/tutorial/interfaces.html)
I couldn't find a source about Multi-GPU model paralellism.
I couldn't find a source about graph computation engine.

Darknet:

There's an RNN example on web site. Shouldn't "RNNs" be "full"?
It doesn't seem to have CPU/GPU generic backend.
It seems to have Multi-GPU data parallelism mechanism (https://github.com/pjreddie/darknet/blob/master/examples/classifier.c#L102, https://github.com/pjreddie/darknet/blob/master/src/network_kernels.cu#L375).
It has "train_network" function, so can't we say it has a native trainer?

What are the differences among "Multi-GPU ~ parallelism", "Multiprocessing", and "Distributed training"?

I came up a same question when I was checking PyTorch. Why do you think Multiprocessing support of PyTorch is partial?

Thanks for the detailed feedback, and for catching a bunch of mistakes! I'll fix the cells I was wrong about soon. Here are some clarifications:
What are the differences among "Multi-GPU ~ parallelism", "Multiprocessing", and "Distributed training"?
Multi-GPU model parallelism: on one machine, placing different parts of a model on different GPUs
Multi-GPU data parallelism: on one machine, replicating the model across GPUs with synchronous data-parallel training (e.g. ParallelUpdater)
Distributed training: training a single model across multiple machines (e.g. ChainerMN)
Multiprocessing: training across multiple OS processes on the same machine (e.g., MultiprocessParallelUpdater) -- this is important for frameworks that run lots of Python code at runtime because Python can only use multiple CPU cores with multiprocessing
In "CPU/GPU backend", "custom" could be misunderstood as that "users can use their own custom backend". How about writing as "native" instead?
"Native" works, but the point I was trying to get across is that those frameworks built their own array types from scratch but don't expose them with separate APIs, meaning that the array backend is less modular/extensible and can't be used on its own.
It's better to put a link to the source if any.
Yes, I'll do that.
Theano-based:
In "Higher-order grads", it seems to only support Hessian (http://deeplearning.net/software/theano/tutorial/gradients.html). How about writing as "Only Hessian"?
No, Theano supports arbitrarily nesting theano.gradients.grad calls (I've used 4+, although it gets very slow). Higher-order grads are most useful for calculating the Hessian, though, so Theano also offers a convenience function for that use case.
"Multi-GPU dataparallelism seems to be experimental" (https://github.com/Theano/Theano/wiki/Using-Multiple-GPUs). How about writing as "Experimental"?
Theano has a very slow development process now, so even things that are several years old and work pretty well are described as "new" and "experimental." Not sure what the best way to describe it in the table here is -- I definitely wouldn't want to use Theano for multi-GPU projects, but that's because I think it would be unnecessarily complicated, not because it would be broken.
It seems to have native trainer. (https://github.com/kirk86/theano/blob/master/trainer.py)
That's a fairly old module in someone's stale branch; Theano currently (by design) leaves things like trainers/iterators/datasets to wrapper packages including Blocks, Lasagne, and Keras.
Caffe1/2:
It also has MATLAB binding (only Caffe1, though). (http://caffe.berkeleyvision.org/tutorial/interfaces.html)
That's true, thanks.
I couldn't find a source about Multi-GPU model paralellism.
It looks like Caffe1 never did implement model parallelism, except in some specific forks where people implemented Alex Krizhevsky's model-parallel AlexNet variant.
On the other hand "Caffe2 also supports model parallelism, but pretty manually. You can assign each operator to different GPU by using DeviceScope." (https://github.com/caffe2/caffe2/issues/371) This is similar to most modern frameworks, including Chainer.
Darknet:
There's an RNN example on web site. Shouldn't "RNNs" be "full"?
Those are fairly new -- the LSTM was added three weeks ago. I think that means RNNs should be listed as "partial" since the built-in modules only support vanilla RNNs and classic LSTMs, and they aren't very customizable / the user can't easily add their own.
It doesn't seem to have CPU/GPU generic backend.
You're right; it uses C macros to switch between compiling exclusively for CPU and exclusively for GPU. So that should be "no."
It seems to have Multi-GPU data parallelism mechanism (https://github.com/pjreddie/darknet/blob/master/examples/classifier.c#L102, https://github.com/pjreddie/darknet/blob/master/src/network_kernels.cu#L375).
Yes, it does.
It has "train_network" function, so can't we say it has a native trainer?
I think what I mean by a native trainer is functionality that means the user doesn't have to write a custom training loop for each new model; darknet's examples all have their own training loops while the train_network function only does an SGD update.

I described multiprocessing support in PyTorch as partial because it's very difficult (I don't think anyone's made it work yet) to use the torch multiprocessing module to build synchronous multi-GPU training similar to MultiprocessParallelUpdater. Instead it's mostly been used for asynchronous training (Hogwild) on CPU; users are supposed to wait for Distributed PyTorch (which has been merged to master but not released) if they want multi-process multi-GPU training.

On DL4J:

it has RNN tutorial. I thought its "RNN support" can be "full".
I thought basically at least "Reverse-mode autograd" or "Forward-mode autograd" should be Y, because I thought otherwise we could not train models (same is true of Caffe 1/2 and Darknet). Is my understanding correct?
Why did you consider "cuDNN support" of DL4J is partial?
Can't we use usual debuggers for Java for runtime debugging?

I called DL4J's RNN support "partial" because it offers three kinds of RNNs (BaseRecurrent and uni- and bidirectional LSTMs) that are not intended to be modified/customized by the user. In order to add another one, a user would have to implement both the forward and backward passes as raw array operations.
It isn't an autograd-based framework: like darknet and Caffe it's restricted in the ways you can put together network layers -- just a single linear stack, with limited exceptions. You don't need any autograd (i.e., toposort + traversing the graph backwards) to implement these layer-based frameworks and you won't find those capabilities in their code.
I called the cuDNN support partial because it doesn't support cuDNN RNNs.
I will fix the debugging cell; runtime debugging should work.

@jekbradbury the cudnn support for RNNs has a bit more work to finish: https://github.com/deeplearning4j/deeplearning4j/pull/3339 (mainly just lack of bandwidth)
As for the autodiff component: https://github.com/deeplearning4j/nd4j/pull/1750 You can find more on that here. I am intending nd4j to be the "chainer/torch" equivalent. DL4j is likely going to stay higher level closer to keras.
As for runtime debugging, yes it's equivalent if not better than python in this department. The JVM actually supports remote debugging via intellij/eclipse etc and your favorite tools.
You're usually using an equivalent of:
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8050
to do runtime debugging. You have to explicitly expose it on a port though. Then it's just like your local debugger.
There's also runtime profiling.
The linear stack is just plain wrong though. http://deeplearning4j.org/compgraph allows anything you want.
We will be combining this with the autodiff support in there to be flexible just like the other frameworks in this case.
Scrolling up seeing some of the other comparisons, I'll also just briefly touch on multi gpu (most folks never get this right).
For distributed training, we have spark and a parameter server based approach. You can see more on that here:
https://deeplearning4j.org/distributed
http://deeplearning4j.org/spark
For single node training we support parallelwrapper:
https://github.com/deeplearning4j/dl4j-examples/blob/master/dl4j-cuda-specific-examples/src/main/java/org/deeplearning4j/examples/multigpu/MultiGpuLenetMnistExample.java
which is basically a  data parallel implementation that supports the same knobs our spark implementation does (it makes some assumptions about single node and the like though) Both of these support any arbitrary neural net config.

Adding to @agibsonccc's earlier comment

I called DL4J's RNN support "partial" ... not intended to be modified/customized by the user.

Depends on your definitions of "partial" and "modified/customized". :)
You are correct that adding a new RNN layer (or, new unit, such as GRU) requires manual backprop implementations (for now).
The usual customizations (activation functions, weight inits, TBPTT) are all in there.
DL4J also supports stateful RNNs - i.e., users can do partial forward pass based on the next step/s in a sequence; masking functionality/variable length sequences; global pooling over (variable length) time series etc.
Happy to answer questions if you want more info on any of that.

Thanks for the clarifications, Adam and Alex!

About Knet

Knet relies on KnetArray as GPU backend
Knet.conv4 seems to support cuDNN, but I found most of cuDNN supports are deprecated. So I am wondering if we should mark cuDNN support as "partial" or not.

@jekbradbury Thank you for updating the table. Also thank you @agibsonccc and @AlexDBlack for your invaluable comments.

About PaddlePaddle

Could you let me know why you think of CNN support as partial?
You put "full" to cuDNN support. But it seems PaddlePaddle does not use cuDNN for RNN. I judged so because I could not find RNN stuffs here and I found for LSTM ) and I found they directly calls kernels in the the forward propagation of LSTM here. Therefore I think "partial" can be better.

For CPU/GPU backend, it could be better to fill something in if the frameworks implements array libraries by themselves but is not named (like CuPy). For example "native" or the same name as the library (as is done in neon column).

About neon

Although it has its CPU tensor class, it seems a wrapper of NumPy ndarray class (see here). So, how about writing "CPU backend package" as "Wrapper of NumPy", or simply "NumPy"?
GPU array backend, it depends on PyCUDA for at least default GPU memory allocation, and custom GPU kernels. So same as CPU, "Wrapper of PyCUDA" or "PyCUDA" could be better as "GPU backend package".

The reason I listed PaddlePaddle's cuDNN support as "full" is because they wrap everything in cuDNN except the RNN functions, but they implement their own time-fused, cuDNN-like RNN kernels instead (I believe this is because they wrote them before cuDNN RNNs were available). So those are likely to be competitive with cuDNN in performance, which is not the case with most other frameworks' non-cuDNN RNN kernels.

Hey folks - Just watching this thread here. http://nd4j.org/backend.html Our equivalent for dl4j is a tensor lib called nd4j. CPU and gpu are supported Basic pitch is "hardware as a jar file" rather than compile/link.
The c++ internals are: https://github.com/deeplearning4j/libnd4j - 1 code base for cpu/gpu (mostly shared business logic for tensor primitives)

LGTM for the frameworks I reviewed.

@jekbradbury
I'm sorry for delayed reply. Thank you for comments to my review.
Can you add "MATLAB" to "Caffe1/Caffe2" / "Language" cell, and put "Y" to "Darknet" / "Multi-GPU data parallelism" cell, as I wrote?
LGTM otherwise.

Thank you for fix!
LGTM as for my assignments.

jenkins, test this please.

test passed


Add support for playwright  global exceptions to console reporter

according to #2244

Hey @In1th, I had a look at how onError behaves, and I'd suggest a slightly different approach.
When Playwright Test encounters an unhandled error, it invokes onError, and then onEnd as usual.
What I'd suggest is to enhance SerenityReporterForPlaywrightTest to introduce an optional field error:
export class SerenityReporterForPlaywrightTest implements Reporter {

    private error?: Error;

and a simple onError handler that assigns any uncaught error to that field:
    onError(error: Error): void {
        this.error = error;
    }

With this capture mechanism, we could then modify onEnd like this:
    async onEnd(): Promise<void> {
        this.serenity.announce(new TestRunFinishes(this.serenity.currentTime()));

        try {
            await this.serenity.waitForNextCue();

            const outcome = this.error
                ? new ExecutionFailedWithError(this.errorParser.errorFrom(this.error))
                : new ExecutionSuccessful()

            this.serenity.announce(new TestRunFinished(outcome, this.serenity.currentTime()));
        }
        catch (error) {
            this.serenity.announce(new TestRunFinished(new ExecutionFailedWithError(error), this.serenity.currentTime()));
            throw error;
        }
    }

Since the uncaught error is now correctly emitted as part of the TestRunFinished event, ConsoleReporter reports it as follows:
[test:acceptance] ================================================================================
[test:acceptance] Execution Summary
[test:acceptance]
[test:acceptance]
[test:acceptance] Total time: 0ms
[test:acceptance] Real time: 41ms
[test:acceptance] Scenarios:  0
[test:acceptance] ================================================================================
[test:acceptance] ReferenceError: asd is not defined
[test:acceptance]     at /Users/jan/Projects/serenity-js/serenity-js/examples/playwright-test-todomvc/spec/bulk_operations.spec.ts:11:5
[test:acceptance]     at Object.<anonymous> (/Users/jan/Projects/serenity-js/serenity-js/examples/playwright-test-todomvc/spec/bulk_operations.spec.ts:10:9)

This is of course better than not having any information about the unhandled error, but not as nice as what Playwright Test line reporter outputs:
[test:acceptance] ReferenceError: asd is not defined
[test:acceptance]
[test:acceptance]    at bulk_operations.spec.ts:11
[test:acceptance]
[test:acceptance]    9 |
[test:acceptance]   10 | describe('Bulk operations', () => {
[test:acceptance] > 11 |     asd
[test:acceptance]      |     ^
[test:acceptance]   12 |     describe('Todo List App', () => {
[test:acceptance]   13 |
[test:acceptance]   14 |         beforeEach(async ({ actor }) => {
[test:acceptance]
[test:acceptance]     at /Users/jan/Projects/serenity-js/serenity-js/examples/playwright-test-todomvc/spec/bulk_operations.spec.ts:11:5
[test:acceptance]     at Object.<anonymous> (/Users/jan/Projects/serenity-js/serenity-js/examples/playwright-test-todomvc/spec/bulk_operations.spec.ts:10:9)

So if you'd be up for that, perhaps "phase 1" of this PR could be to capture and emit the unhandled error, and "phase 2" could be to offer more in-depth feedback to help developers identify what caused the error in the first place?

@jan-molak, definitely! I'll go and change this to your proposal and try to run tests locally. I suppose I may have an idea of how to get more info, but let me try to run it once 😄

Ok, I did the version you suggested @jan-molak. I even figured out how to include a code snippet, turns out playwright passes it through its own object.
-Error: Trigger failure for worker 0
      -   7 |         beforeAll(async ({ browserName }, workerInfo) => {
      -   8 |             if (workerInfo.workerIndex < 2) {
      ->  9 |                 throw new Error(`Trigger failure for worker ${ workerInfo.workerIndex }`);
      -     |                       ^
      -  10 |             }
      -  11 |         });
      -  12 |

I copied it from integration test, since I modified playwright error parser to include code snippet... What means every error has this. That's why I have to ask you the approach, since just adding this functionality on by default would mean changing all failing tests to have this snippet and that sounds brittle.
So I thought about having a reporter config option to enable code snippets in errors.

@jan-molak I didn't know if I got it right, you want me to implement our own code snippet functionality and go to every console reporter implementation for different frameworks or just do it in the playwright reporter?
For now I added a new flag called includeSnippetOnError to include them. Also I added new functionality to playwright tests to be able to manually change ENV variables, for this case a variable to enable snippets just for one test. 😄

you want me to implement our own code snippet functionality and go to every console reporter implementation for different frameworks or just do it in the playwright reporter?

ConsoleReporter is generic, so there's only one implementation that works with all the reporters.
To keep it simple, though, I'd suggest splitting this PR into 2:

one to add the global error handler, as per our conversation
another to do the formatting

Adding the information about a global unhandled error is already valuable, even if it doesn't have the snippet yet.
WHat do you think?

Alright, I did the revert to playwright error handling, so I guess for this PR it finally has what it advertises 😆

Merged, will be released in 3.19.0 🚀
Thanks!


test-network envVar.sh script improvement

Change shell script to use single equals.
In my shell environment single bracket with double equals did not work. It caused failure when running the chaincode-external tutorial README.
It looks like for maximum portability, it is best to use single bracket with single equals.

@mergifyio rebase

@satota2 I found this while testing your chaincode-external updates...


Add support for Xcode

Search before asking

[X] I had searched in the issues and found no similar issues.

Motivation
In CMakeLists.txt, if(CMAKE_GENERATOR STREQUAL "Ninja") is applicable to Xcode as well.
Solution
if(CMAKE_GENERATOR STREQUAL Ninja OR CMAKE_GENERATOR STREQUAL Xcode)
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

@bijanMC Would you like to submit a PR to resolve this?

Is this really necessary? In fact, existing systems can also be used to build on macOS.


Question: Can you preserve position using indentation?

Is there a flag or something?
I am thinking something like a Ditto Mark perhaps?
https://en.wikipedia.org/wiki/Ditto_mark

Sure. Initially I was thinking of some kind of dito mark, but that would add to the character count. I sacrificed some flexibility there.
But I love the idea of adding a flag for a csv file that has a dito marks.
I would be happy to add that.

What do you think a good mark would be.

No idea TBH.. I mean " is kind of universal, but even just spacing it out would be good.
Unicode / emoticons would be very distinct. I can't imagine not making this configurable.

For sure, I just wanted to make sure that if you had some thoughts you had a chance to lay them out.
I can add a flag for ditto that will default to ", but users to also specify their own mark.
Thanks again for the suggestion.

I merged #4 let me know if that is what you were thinking @arran4. I used . as opposed to " because sometimes and open quote expects a closing quote.


Switch from axios to fetch?

Hi,
Have you considered updating this package to use fetch instead of Axios?
We are using @workos-inc/node with Cloudflare Workers/Pages, which do not support all the Node APIs that Axios requires.
Older versions of @workos-inc/node were using using<EMAIL_ADDRESS>and there was a fetch adapter that was compatible with that version of Axios (https://github.com/vespaiach/axios-fetch-adapter). Unfortunately, the most recent version of @workos-inc/node depends on<EMAIL_ADDRESS>and there doesn't seem to be a good fetch adapter for it.
Node v18 supports the fetch API (and you may be able to inject node-fetch or similar in earlier versions).
Alternatively, you could add a new workos-fetch package. But that may be too much work since I don't think the SDKs are auto-generated.
Thanks.

This https://security.snyk.io/vuln/SNYK-JS-AXIOS-6032459 issue probably landed in a lot of inboxes today 🧐

This https://security.snyk.io/vuln/SNYK-JS-AXIOS-6032459 issue probably landed in a lot of inboxes today 🧐

it's been 2 weeks, would appreciate an update from the workos team on eta of this fix.

Replacing Axios with fetch is on the roadmap! Don't have exact timelines to share right now but we're actively working on it.

Hey @BrandonNoad, we are working on this as we speak, we will have an update soon.

Switching to fetch will have a performance and size impact on Bun/Deno and newer versions of node.

Hey all, I wanted to give you a quick update on this.
It's coming along nicely and I have just opened a PR: https://github.com/workos/workos-node/pull/962

I have published a release candidate of the SDK on npm (6.0.0-rc.2 tagged next)
It can be installed with npm install @workos-inc/node@next

v6.0.0 has now been released and adds support for cloudflare workers / edge runtimes.


Fix statefulMap may call onComplete twice if it throws

#596

LGTM only two comments.

@GreyPlane could we get you to fill in an Apache CLA for this? https://www.apache.org/licenses/contributor-agreements.html
For a comparatively small change like this, we might not always require the CLA but this issue/PR has become a potential legal issue since @kerr has looked at the Akka solution (and it is not licensed under a license that is compatible with use by us). We don't want any risk that anyone could claim that Category X code has been copied into our project.
https://www.apache.org/legal/resolved.html#category-x

@pjfanning The fix of my came out independently,  but the tests were what I copied for quick validation, let's make that clear, and then i checked the Akka fix which is not the same as mine.
So it's the tests I took not the fix.
Many ones in Wechat group can prove this.
I was want to make sure the onComplete to be called and then mark the boolean to false, but that will need the onComplete be idempotent and not easy to use.

@kerr Let us do a clean room fix.

And you may find someone not in tech to check and validation those fix is not the same

@GreyPlane Sorry, I think this fix should be closed, even I knew you come up this independently, but as you are involved/leved comments in the akka pr, so this fix should be closed and come out with someone else, thank you very much.


Zotero ribbon doesn't add to tab bars in Word

Hello,
Is any settings that needs to be changed to allow the ribbon tab to be executed correctly and display in word ?
Zotero says that's not a Zotero issue.

I do not know what Zotero is. I provide a script to download the latest  geniune Microsoft Office. I receive a feedback regarding the script work, not any 3rd party add-ins.


Bulk deletion of templates  from multiple zones is not working

ISSUE TYPE

Bug

COMPONENT NAME
Component: UI
CLOUDSTACK VERSION
Cloudstack version 4.18.1
SUMMARY
Bulk deletion of templates from multiple zones is not working
Steps to reproduce the issue

Create 2 zones Cloudstack environment

Register a template/ISO in alll zone

Select  the Bulk delete template  option and select both the zone

The template gets deleted in only one zone

https://cloudstack.apache.org/api/apidocs-4.18/apis/deleteTemplate.html
Only one zone id is getting passed in the api call
http://<IP_ADDRESS>:8080/client/api/?id=6897f3b3-9030-45c7-bebe-216e1c799393&forced=true&zoneid=90085d02-1fa4-4511-838f-c431ee7e2b27&command=deleteTemplate&response=json
The issue is not observed for the bulk deletion of  ISO's
Expected behavior
The template should get deleted in both the zones
Actual behavior
The template should is getting deleted in a single zone

Fixed by #8118


Add Match to Option

When working with Options, you'll eventually need to eventually leave the Option context near the boundary and work on the value directly.
As it stands now, the only way to get the value is to use Do and a mutable variable.
Suggested feature is to add a method called Match that satisfies the following
var some = Option.Some(4)
var none = Option.No<int>()
int result = some.Match(() => return 0, x => return x); // would yield 4
int result = none.Match(() => return 0, x => return x); // would yield 0

#26 Pull request sent

Awesome work @zaudtke, thanks!