Implement a Compatibility Decomposition Normalizer

Meilisearch is unable to find Canonical and Compatibility equivalences, for instance, ｶﾞｷﾞｸﾞｹﾞｺﾞ can't be found with a query ガギグゲゴ.
Technical approach
Implement a new Normalizer CompatibilityDecompositionNormalizer using the method nfkd of the unicode-normalization crate.
Files expected to be modified

create  /src/normalizer/compatibility_decompostion.rs
/src/normalizer/mod.rs

Misc
related to product#532

Hey! 👋
Before starting any implementation, make sure that you read the CONTRIBUTING.md file.
In addition to the recurrent rules, you can find some guides to easily implement a Segmenter or a Normalizer.
Thanks a lot for your Contribution! 🤝

@ManyTheFish
Thank you for the creating good issue!
This character normalization seems to be performed after tokenization, but in Japanese, some cases it is better to performe character normaliz before tokenization.
For example, this is a case where there is no problem even after tokenization:
$ echo "私はメガネを買いました。" | lindera
私	名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ
は	助詞,係助詞,*,*,*,*,は,ハ,ワ
メガネ	名詞,一般,*,*,*,*,メガネ,メガネ,メガネ
を	助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
買い	動詞,自立,*,*,五段・ワ行促音便,連用形,買う,カイ,カイ
まし	助動詞,*,*,*,特殊・マス,連用形,ます,マシ,マシ
た	助動詞,*,*,*,特殊・タ,基本形,た,タ,タ
。	記号,句点,*,*,*,*,。,。,。
EOS

$ echo "私はﾒｶﾞﾈを買いました。" | lindera
私	名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ
は	助詞,係助詞,*,*,*,*,は,ハ,ワ
ﾒｶﾞﾈ	UNK
を	助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
買い	動詞,自立,*,*,五段・ワ行促音便,連用形,買う,カイ,カイ
まし	助動詞,*,*,*,特殊・マス,連用形,ます,マシ,マシ
た	助動詞,*,*,*,特殊・タ,基本形,た,タ,タ
。	記号,句点,*,*,*,*,。,。,。
EOS

Half-width ﾒｶﾞﾈ is not a problem because it is tokenized as a single unknown word, although it is not exist in the Japanese morphological dictionary (IPADIC).
But the following cases can be problematic.
$ echo "私は時給1000円です。" | lindera
私	名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ
は	助詞,係助詞,*,*,*,*,は,ハ,ワ
時給	名詞,一般,*,*,*,*,時給,ジキュウ,ジキュー
1000	UNK
円	名詞,接尾,助数詞,*,*,*,円,エン,エン
です	助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
。	記号,句点,*,*,*,*,。,。,。
EOS

$ echo "私は時給１０００円です。" | lindera
私	名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ
は	助詞,係助詞,*,*,*,*,は,ハ,ワ
時給	名詞,一般,*,*,*,*,時給,ジキュウ,ジキュー
１	名詞,数,*,*,*,*,１,イチ,イチ
０	名詞,数,*,*,*,*,０,ゼロ,ゼロ
０	名詞,数,*,*,*,*,０,ゼロ,ゼロ
０	名詞,数,*,*,*,*,０,ゼロ,ゼロ
円	名詞,接尾,助数詞,*,*,*,円,エン,エン
です	助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
。	記号,句点,*,*,*,*,。,。,。
EOS

Since full-width numbers are registered in the morphological dictionary (IPADIC), each number becomes a separate token, so that a full-width １０００ cannot be searched for 1000.
For this reason, it is common for search engines that handle Japanese to perform character normalization before tokenization. Is there a way for Meilisearch to perform character normalization before tokenization?

Hello @mosuka,
Thanks a lot for those precisions!
Unfortunately, it would be a lot of work to invert the process.
For now, we will keep it like this and stick with this issue.
However, I have a similar issue with the Malayalam Language and I'll have to find a way to have both pre-post-normalizers. 🤔

Hey folks, I started trying to implement this issue still during the Hacktober fest. But I couldn't focus so much on it. Then I didn't ever commented here to avoid to "block" an issue that someone could eventually implement faster....
But I kept trying to implement it anyway, just to learn more about rust and charabia engine.
Well, talking now about the implementation.... I started working on it when we still had to implement normalize(). So, I had to change some pieces after the consolidation the team made weeks ago, to use the CharNormalizer trait. I think it is ok now, but I'm experiencing some issues when we have the global normalization pipeline (and that's the reason I haven't opened a PR yet - please let me know if you prefer that I open it).
The first one is to understand which Scripts should be normalized with nfkd(). I tried to follow  some other implementations, but I'm not 100% it is ok - so if someone can give me some guidance it would be great :-)
Other and maybe more complex, is that the LatinNormalizer uses deunicode, which will try to remove the accent marks and remove the characters that it doesn't recognize. So... should we avoid to use nfkd() side-by-side with LatinNormalizer?
Thanks for your help :-)

Hello @charlesschaefer,
first, don't hesitate to create a PR on this repo, this would be easier for me to guide you in the implementation.
Then, UnicodeNormalization trait is implemented for any type that implements IntoIter, this way Some('a').nfkd() would give you an iterator over chars, then, as the Latin normalizer a match would finally do the job to convert your Iterator into the expected type.


node_delete_nic and headnode_delete_hnic don't check for network attachment

This causes the following behavior:
-Attach hnic to network
-Delete hnic
-Create new hnic with same name
-New hnic is /already/ attached to a network
I haven't gone through and debugged exactly what's happening, but this is clearly not the right behavior.

weird. probably easy enough to transliterate this big report into a test case.


feature: Capability Discovery API

Is your feature request related to a problem? Please describe
A Relying Party is assumed to have implicit knowledge of the capabilities of a Verifier e.g. what tokens are supported and what constraints are applied to that token support e.g. the supported signature algms.
If possible, describe a solution
Add a discovery API that would describe the configured tokens for this deployment. Schema is TBD but probably related to the configuration of the Verifier.
Advanced uses: links to peer deployments, potentially with different tokens supported; conveyance of additional trust data for the RP to evaluate.
Describe alternatives you have considered
Alternative: a probe-and-fail pattern for an agreed template form of a stub token submitted to the Verification API.
Additional context
NA

issue now tracked under services repo:
https://github.com/veraison/services/issues/80


Is this project DEAD?!

Its primary maintainer has not contributed code to the project in over a year. You may draw your own conclusions from there, although they have also repeatedly stated that they have no intention of discontinuing the project.
X-Ref: #607.

It's a pity to see this project not actively developed. I just forked this project to start my own https://github.com/mazong1123/ratchet-pro
Note: I'll add more framework oriented features like dynamic script loading and executing, caching, etc.

@mazong1123 I agree I think this project is great

Hey,
I think project is doing great and I'm in for helping, any roadmaps?

I am sad this project seems abandoned. Its one of the few mobile ui framework's I've found that aren't absurdly over-complicated or bundled with some kind of terminal executables.

@jgog @blamarche I forked this project and adding new features. I'm using ratchet.js in my hybrid apps in Production so I'll maintain this new fork continues. Feel free to file issue or send pull request to this new repo: https://github.com/mazong1123/ratchet-pro

Really great to hear about Rachet Pro. Any thoughts on synching w/ B4? Are you getting help?

@snigglet Would you please give me a link to "w/ B4"? I don't have any knowledge about that.

Sorry ... Bootstrap 4
Sent from my iPhone

On Oct 25, 2016, at 8:36 PM, Jim Ma<EMAIL_ADDRESS>wrote:
@snigglet Would you please give me a link to "w/ B4"? I don't have any knowledge about that.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@snigglet I don't have plan on syncing Bootstrap 4 by now. But if anyone can help on this is definitely welcome :)

Don't have capacity to drive much right now but we would enjoy being
involved. Has there been any analysis on whether anyone else is doing
something like this too?
On Friday, October 28, 2016, Jim Ma<EMAIL_ADDRESS>wrote:

@snigglet https://github.com/snigglet I don't have plan on syncing
Bootstrap 4 by now. But if anyone can help on this is definitely welcome :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/twbs/ratchet/issues/792#issuecomment-256904030, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABvaXLXFGoMlwr-7bToBoaCCuTJf2f7lks5q4eV1gaJpZM4FSIyM
.

--
Jean Holt
CEO, Gatelets
415.999.2789
gatelets.com


Nested archetypes

I have the following structure in content:
content
├── techtalk
│   ├── algorithms
│   │   ├── algorithm_binary_tree.md
│   │   ├── algorithm_fastman.md
│   │   ├── algorithm_stream_merger.md

So in my archetype directory I have:
archetypes
├── jmetadata.md
└── techtalk.md

There is no way to specify an archetype to algorithms pages directory?

Please use http://discuss.gohugo.io/ for questions.


Tokens with same symbol not displaying properly

Discussion: https://forums.eoscommunity.org/t/anchor-wallet-tracked-tokens/4176

Notes on this: the balances reducer is currently using a key/value pair of symbol/value for tracking.
We will need to update that structure to be smarter and potentially use contract::symbol for the keys to make it more specific and avoid issues like this.
The keys from the balance reducer are used in a number of places, so this change will take some effort to implement.
For testing:
account: developjesta
token1: soonsoonsoon:soon
token2: atokenmancan:soon


Hotfix twitter login

About the bug
I found a bug with Twitter login with external module reported at the issue #800
How to fix
I tried to fix it by myself, then I add username property to and remove not null constraint of email from users in lib/generators/sorcery/templates/migration/core.rb.
Username is needed when you login with Twitter, but email isn't needed.
Without username property, it raises error(mentioned in issue #800).
Thinking about email, not null constraint in a database layer raises an error
at user.sorcery_adapter.save(:validate => false) in #create_from_provider in  lib/sorcery/model/submodules/external.rb.
The error is:
Mysql2::Error: Field 'email' doesn't have a default value: INSERT INTO `users` (`username`, `created_at`, `updated_at`) VALUES ('blablabla', '2016-09-22 15:56:22', '2016-09-22 15:56:22')

Using #save with validate option, expected that a validation of email is ignored. The validation should be imposed only in model.
I wonder why tests don't fail, looking into specs.
In spec/rails_app/db/migrate/core/20101224223620_create_users.rb, different from the migration template mentioned above, defined are columns username and email without not null constraint.
I think users defined in tests is expected, and I fix in this way.
How to test
I run all tests by bundle exec rpsec.
I also confirmed expected migration template as follows:
class SorceryCore < ActiveRecord::Migration
  def change
    create_table :users do |t|
      t.string :email
      t.string :username
      t.string :crypted_password
      t.string :salt

      t.timestamps
    end

    add_index :users, :email, unique: true
  end
end

I'm looking forward to a comment. Thank you for reading.

@ebihara99999 Can you move this over to https://github.com/Sorcery/sorcery/?

Do you mean you ask me to create the same new pull request to the repo?
Then yes, I will do that.

@ebihara99999 Yeah, this repo is going to be deprecated soon in favor of the Sorcery org.

Once this is moved over to the Sorcery org repo, I'll take a look at this.

I moved this to Sorcery/sorcery pull requests. I hope you read.


IScroll 5 demo lags in browser

Hi,
A few days ago I posted a question on a phonegap forum (https://groups.google.com/forum/#!topic/phonegap/G3-69UMw_2k). So I want to report a bug, because we think it's a issue in the IScroll library.
Problem:
In my standard mobile browser (on my HTC one S phone) scrolling is really lagging. It's bouncing, but very fast. So I see text multiple times. Conclusion it's not smooth, but when I test it in the google chrome mobile browser it's fine. The Iscroll 4 library is also in the standard mobile browser fine.
My problem is that phonegap used the standard mobile browser engine, so in my application I got also the laggy scrolling.
My temporary workaround is to use the probe library and update an almost invisible text. with the workaround scrolling is less laggy, but it's still not very smooth like a native app. Is this issue know? I tested the demo on a:
HTC one S and a HTC desire X both have the same problem.

useTransition: true forces teribble visual effects at least on ios


Make a plan for the twitter wall

A tradition started at BTPDF2 was a twitter wall.
This requires a few things:
projector that is not conflicting with the main goings on of presentations etc.
someone to monitor for spam/malicious stuff
someone to actively tweet and promote conference activities

We need to look at the venue. Maybe multiple monitors around the venue may be necessary. I don't know if we need to do the monitoring and promotion - this happens spontaneously


How can I solve this

Help please
lib/analytics/analytics_service.dart:4:37: Error: Couldn't find constructor 'FirebaseAnalytics'.
final FirebaseAnalytics analytics = FirebaseAnalytics();

check Firebase Analytics package is properly installed and imported in your project

The correct line should be:
final FirebaseAnalytics analytics = FirebaseAnalytics.instance;
That should solve the problem


Prepare project brief for the EFL

See last year's project description and space needs spreadsheet.
Include projects from all sections (including Steve's).
Items:

Size of project, i.e. space needs
Fab needs
Simple title and description of project
Does it need storage space on campus? Does it matter where it is?

This is done.


Arkane connect: allow secretType parameter to pick the chain

Right now connecting to Arkane (or Venly) connects directly to ETHEREUM. If the Arkane connector could support the secretType parameter, we can pick which chain to use. In my case I would like to connect to MATIC.
Just providing rpcUrl still defaults to ethereum for some reason.
src/providers/connectors/arkane.ts
        const options = {
          clientId: opts.clientId,
          rpcUrl: opts.nodeUrl,
          environment: opts.environment,
          secretType: opts.secretType,   // Suggestion
          signMethod: "POPUP"
        };

With stable version 2.0.0 of Web3Modal now released, we are officially dropping support for version 1.x
Due to this this issue/pr was marked for closing. It is highly recommended to upgrade as 2.x will be receiving further updates that will enable functionality for some of our newer sdks like auth and push as well as support for WalletConnect v2 (See this post about WalletConnect v1 being deprecated https://medium.com/walletconnect/walletconnect-v1-0-sunset-notice-and-migration-schedule-8af9d3720d2e)
If you need to continue using Web3Modal 1.x and require this feature/fix implemented, we suggest adding it via forking V1 branch.


Fix energy equation in Notebook 5

As noted by @eric-wieser , using np.sum(X**2) to calculate the energy of the image X gives an incorrect result because the elements of X are of type np.uint8, so they overflow:

This can be quite confusing when comparing the energy of the original image and the filtered versions, as the filtered images have elements of type np.float64 so they don't overflow, which makes the comparison useless.
It might be useful to fix this by using np.sum(X**2.0) so that students don't get confused.

Thanks for the PR. Unfortunately our CI system that hides the cribs from students means we can't actually merge this through github, but I went ahead and deployed a fix in 662381d3b4ebd2c574c96bc54a87c504e9bbdb55


Cache app charts in the server pod

Fix #2532

Tested on branch with Epinio Server Version: v1.10.0-rc2-11-g98505791 and latest Epinio ui vs v1.10.0-rc2 without the fix and works well for me. Besides checking no errors I also took a look at performance improvement and looks good to me.
CLI  TEST

 Upscale from 1 to 25 instances and back to 1 one by one
# Deploy sample app
epinio push -n sample -p /home/mmartin/suse/epinio-repos/epinio/assets/sample-app

# Upscale to 25 instances and downscale to 1
epinio app update sampleapp -i 1
epinio app update sampleapp -i 2
epinio app update sampleapp -i 3
epinio app update sampleapp -i 4
epinio app update sampleapp -i 5
epinio app update sampleapp -i 6
epinio app update sampleapp -i 7
epinio app update sampleapp -i 8
epinio app update sampleapp -i 9
epinio app update sampleapp -i 10
epinio app update sampleapp -i 11
epinio app update sampleapp -i 12
epinio app update sampleapp -i 13
epinio app update sampleapp -i 14
epinio app update sampleapp -i 15
epinio app update sampleapp -i 16
epinio app update sampleapp -i 17
epinio app update sampleapp -i 18
epinio app update sampleapp -i 19
epinio app update sampleapp -i 20
epinio app update sampleapp -i 21
epinio app update sampleapp -i 22
epinio app update sampleapp -i 23
epinio app update sampleapp -i 24
epinio app update sampleapp -i 25
epinio app update sampleapp -i 24
epinio app update sampleapp -i 23
epinio app update sampleapp -i 22
epinio app update sampleapp -i 21
epinio app update sampleapp -i 20
epinio app update sampleapp -i 19
epinio app update sampleapp -i 18
epinio app update sampleapp -i 17
epinio app update sampleapp -i 16
epinio app update sampleapp -i 15
epinio app update sampleapp -i 14
epinio app update sampleapp -i 13
epinio app update sampleapp -i 12
epinio app update sampleapp -i 11
epinio app update sampleapp -i 10
epinio app update sampleapp -i 9
epinio app update sampleapp -i 8
epinio app update sampleapp -i 7
epinio app update sampleapp -i 6
epinio app update sampleapp -i 5
epinio app update sampleapp -i 4
epinio app update sampleapp -i 3
epinio app update sampleapp -i 2
epinio app update sampleapp -i 1

Results AFTER FIX:

Task completed done in less than in aprox 40 seconds.

No errors logged this time, but were observed in the past. I could have forced them if playing more with upscale / downscale, but this point was observed in the CLI test before. The enhancement is clear after fix

Comparison video on UI:
https://github.com/epinio/epinio/assets/37271841/c638332b-0cb4-41be-ac86-4b846d9d7935

This package could be even unit tested. That will be nice. :slightly_smiling_face:

Unit tests added - commit<PHONE_NUMBER>628abf152a09dae2ee5a2a7d9bf81d and preceding commit, with some prep and fixes found during unit testing

Some comments regarding logs and the statusCode.
Also we have the make appchart command that was used to test locally and in a quick way the appchart. It will package the chart in a local nginx. In this way it's possible to test this. Now that is cached I assume that the chart, having always the same URL, it won't be downloaded anymore, so any edits won't be reflected.
It's probably enough to add a timestamp now() to the url to skip the caching and changing the url here (something like ?t=3432432421)
or force a restart of the Epinio server.

Commit c4a28a0a300d6259630463617e9afbaf1d35d652 uses timestamped urls to force a miss and fetch.


Basic Snake Game using python

@KIET7UKE
Tried to replicate 90's snake game in python language. Can I add that?

Sure, go for it @HarshChhikara

@KIET7UKE I have raised the pr. kindly review it.

it shows excluded on hacktoberfest website. any particular reason?


Proposal: Create Tokens Scooper DApp

I want to create Tokens scooper DApp.
A DApp that allows a user to scoop all their low quantity KIP7 tokens.
It allows the user to swap the scooped tokens for KLAY.

Sounds good. I like to clean out all my dust too. You can make some fake tokens on testnet (with same token ticker on mainnet) for the demo app.


feat: introduce generate_modulemap to support @import in Objective-C modules

Related to #231.

Related to #208.


Add bundlesize

Follow-up from the https://github.com/limonte/sweetalert2/issues/728#issuecomment-346040779
Thanks @StephanBijzitter for the suggestion :beers:

🕺 I especially love being able to compare pull requests against master. If you make a few changes and then suddenly the size of your output doubles, you know you've done something wrong. And it's also very satisfying to see the size go down as you trim away more and more!


Feature Request: A section that gives a high-level overview of the Kubernetes project (targeted towards new contributors)

For new contributors (esp. if they are new to open source projects), understanding such a big codebase may seem daunting. There should be a section that walks us through the codebase and gives us a high-level overview.
The section should go over the following topics:

Codebase Tour: Goes over the project repo and explains how the project is structured
Architecture: Explains the overall architecture of the project (i.e. why things are the way they are)

This section doesn't have to go too deep into the details. Rather, the goal is to give new contributors a high-level view of the project.

/kind feature

/remove-lifecycle rotten
This is actually a planned feature for the site, so no reason to close the issue.

/remove-lifecycle stale

@mrbobbytables  could you help me with this? I know you are creating the new contributor workshop material and I would be happy to add the content here for new contributors.

This is actually a planned feature for the site, so no reason to close the issue.

/lifecycle frozen

Hi Folks, any help or pointers for this task would be great for me to kick-start. I am still interested in finishing this task.

Update: I got details to proceed with the architecture. It's a seemingly big issue with a chunk of tasks. I will try to create a task list as I move on. Thanks, @mrbobbytables for the heads up.

FYI, for folks interested in reviewing this check this rendered content

/close


[BUG] Item Containers weight calculation with backpacks becomes reverse of intended

Module Version: v0.6.8
Describe the bug
It seems that this module combined with the newest version of Item Containers (1.8.9) "reverses" when the carrying weight of a bag is calculated. Without this module enabled, I can equip a bag from Item Containers, and the weight is then added and calculated, and when I unequip it, the weight is removed. With the module on, it seems to reverse this?
To Reproduce
Steps to reproduce the behavior:

Use the Backpack item from the DnD 5e compendium, have the aforementioned modules enabled.
Equip/unequip the bag, notice the weight added doesn't correlate right depending on whether it is equipped or not - for me it added the weight when it was unequipped, rather than equipped.

Foundry Version:
0.8.9
Game System:
DnD 5e 1.5.3

Should be fixed with 0.6.2

I'm still seeing the same behavior, this is with Find the Culprit and only Midi QOL, ItemContainers and VariantEncumbrance on (with dependencies socketlib and the other lib). Backpack goes on, weight goes off.

i think is a retrocompatibility issue is the old backpack you show me? can you try to add a new backpack and tell me if the issue persist ?

All the backpacks were from the SRD where I changed the sheet, tried and created a new backpack from scratch, issue persists unfortunately

also happens on other characters

Add some more check on 0.6.3  if the error persist export the charater sheet and pass to me for better analyze the use case.

Thanks for your effort!
The new update made it so the bag's effectively functionless, the weight remains the same whether it is equipped or unequipped, unless I tick on "weightless contents".  Here's the actor data (json saved as a .txt file).
If you want hands on debugging in my world or want more direct contact, you're also free to contact me on Discord at Krigsmaskine#8165.
fvtt-Actor-jack.txt

I don't know what to say with me it works, added more checks and cleaned up the code on version 0.6.4. If that doesn't work for you, pass me the reference to your world on private channel on discord here the username "4535992#1766".


Add old way to check unapplied migration

Hello @tyzhnenko. This is a nice addition, but i think it should be named differently, because check-unapplied-migrations supposed to check if we have migration file which is not applied on a DB, while makemigrations --check will return non-zero when we have changes in the models but we don't have a migration file for a such change.

lgmt and make a new release :)

@tyzhnenko thanks for the contribution. I'll release this soon.


update dev dependencies and follow types

No longer uses FC.

I think I am in need of this PR, as the type definition of FC no longer includes children in it.

Okay, let's release this.

Published: https://www.npmjs.com/package/use-context-selector/v/1.3.10

Oh! Thank you so much for your responsiveness @dai-shi !! I am downloading now.
This is a big help as I was just about to go through my codebase and put Provider as any everywhere


Whitespace

Optional pull request but a fix for SilverStripe code compliance using the node based editorconfig-tool check and fix commands.
As an aside could you also enable issues on this repository, it is off by default for forks.  The language files have some merge markers in them.

Thx! I've enabled issues (no idea why it wasn't in the first place), will merge this PR as soon as I'm working on a site containing this module (needs command line).

Let me know when you consider the codebase stable and I can run my command line tools over the codebase and manually fix other issues.  Doesn't take too long


Move container labels from Latest

As noted at https://github.com/weaveworks/scope/issues/1201#issuecomment-233545367, Docker container labels and environment variables are sent in the Latest structure even though they never change.
We could put them in Sets instead.

Endpoints host_node_id and pid are in the same boat, though it is less obvious that wrapping in Sets will be a win for just two values.


Fix GT and LTE for type date

A follow-up to #113

@Ibrahemkhalil nice, much appreciated.  I'll probably decrement back down to 1.3.1 before I release a new package, I'm waiting on some Mongo tests this weekend from another committer who is adding some integration tests around that, but I can handle that change when it's ready to release.  Thanks for the PR!


fix: Trigger rebuild on changes

It's much easier to develop than cleaning the target dir before every build.
Thanks

ping @vlad17


make analysis doesn't work

Your targets for the analysis (all-analysis, and the individual analyses themselves) don't work. If you are planning on fixing this, please push the changes soon so we can evaluate the project. If not, apply the wontfix label and we will try to run your code by hand

@rossbar @soazig  please give us a second, we'll fix them now

Hello @rossbar
It seems like one of our plotting function got lost between the updates, I am uploading it now.
However, if this is not to much to ask, where you able to download the data ?
Thank you
@mingujo

Yes I was able to successfully download the data (though not from the Makefile in the top-level project directory, I had to use the one in the data/ directory). Thanks for taking a look!

Thank you @Ross,
Yes, we realized we forgot to add 'data' in the .PHONY which is why it was
saying that was already up to date.
Thanks !
On Dec 15, 2015 10:58 PM, "Ross Barnowski"<EMAIL_ADDRESS>wrote:

Yes I was able to successfully download the data (though not from the
Makefile in the top-level project directory, I had to use the one in the
data/ directory). Thanks for taking a look!
—
Reply to this email directly or view it on GitHub
https://github.com/berkeley-stat159/project-epsilon/issues/315#issuecomment-165017477
.

@rossbar we're running into couple issues so can you give us some couple of minutes to fix and comment when we're done?

That's fine, you don't have to get it done immediately, although the sooner the better. I just posted a piazza announcement about this as many groups had build problems. Just notify me when you are done with these modifications and have verified a successful build from scratch on at least one system (or apply the "wontfix" tag, though it sounds like you're working through the issues)

@rossbar
Actually, we fixed the bug. Navigation from the original repo should be fine now.
However, just to make a notice, 'make glm' and 'make multi-comparison' will run for long time.
For multi-comparison, it will run about an hour

Thanks for the heads-up, long runtimes are okay, but that's why I want to verify that the analysis will run without any errors so that I don't have to keep rerunning a very long analysis only to have it fail multiple times. I will pull and attempt to rerun make all-analysis

Still getting IOErrors from make eda. Let me know when you have verified a successful run of make all-analysis from a fresh clone

Hello Ross,
I used a fresh clone and ran into error too.
I have a final at 8am tomorrow, so if that's ok, I'll take a look in the
morning after my final.
Thank you,
On Tue, Dec 15, 2015 at 11:58 PM, Ross Barnowski<EMAIL_ADDRESS>wrote:

Still getting IOErrors from make eda. Let me know when you have verified
a successful run of make all-analysis from a fresh clone
—
Reply to this email directly or view it on GitHub
https://github.com/berkeley-stat159/project-epsilon/issues/315#issuecomment-165026431
.

--
Soazig Kaam
PhD student | UC Berkeley College of Environmental Design
http://ced.berkeley.edu/
Graduate Student Researcher | Center for the Built Environment (CBE)
http://www.cbe.berkeley.edu/

Yes definitely,  good luck on your final!

Thank you @Ross,

Don't think I'm the Ross you were looking for. Apparently you wanted the other Ross in Berkeley, CA :smile:

Any luck on this guys? I'd like to re-run the analysis ASAP so I can grade the project. Hope all is well with regards to finals and other projects! Let me know how much more time you think you'll need.

Hello Ross,
Sorry, I think I was able to debug a couple of things.

The data type was the most critical, I made sure all data were in float
type.

Some of our code were still running with a limited number of subjects
instead of the entire list.

I also added some verbose to know where we are in the code.

I changed the README and Makefile to distinguish the analysis for 2
subjects than for the entire subject list.

My tests, coverage and run were successful for the 'make analysis except
multi". I am finishing up the multi comparison thing.

As I am running on the stats department server, I am only able to test
the multi comparison scripts on 2 subjects. So far so good.

I will update asap, thank you for understanding !
On Dec 17, 2015 3:53 PM, "Ross Barnowski"<EMAIL_ADDRESS>wrote:

Any luck on this guys? I'd like to re-run the analysis ASAP so I can grade
the project. Hope all is well with regards to finals and other projects!
Let me know how much more time you think you'll need.
—
Reply to this email directly or view it on GitHub
https://github.com/berkeley-stat159/project-epsilon/issues/315#issuecomment-165618661
.

Hello Ross,
Hopefully the master works now.
 I have a couple of things to add.

There was an error in the .bh  script to download the filtered data, you
only got subject
s
1 and 5. I added a new entry in the Makefile in order to download only the
filtered data. From the main project-epsilon directory, please do make filtered_data_only.
 I added a note for you in the README.

I also separated the fast analysis involving only subject 1 and 5 from
the longer analysis that compare
s
all subjects.
I updated the README in consequence.

There are now 2 steps to make the full analysis, To make all
analyses make analysis-except-multi and then make multi-comparison
 (see README)

I hope this makes sense, I am sorry about the confusion
 and delay
Thank you!
Soazig
On Dec 17, 2015 4:45 PM, "Soazig Kaam"<EMAIL_ADDRESS>wrote:

Hello Ross,
Sorry, I think I was able to debug a couple of things.

The data type was the most critical, I made sure all data were in float
type.

Some of our code were still running with a limited number of subjects
instead of the entire list.

I also added some verbose to know where we are in the code.

I changed the README and Makefile to distinguish the analysis for 2
subjects than for the entire subject list.

My tests, coverage and run were successful for the 'make analysis
except multi". I am finishing up the multi comparison thing.

As I am running on the stats department server, I am only able to test
the multi comparison scripts on 2 subjects. So far so good.

I will update asap, thank you for understanding !
On Dec 17, 2015 3:53 PM, "Ross Barnowski"<EMAIL_ADDRESS>wrote:

Any luck on this guys? I'd like to re-run the analysis ASAP so I can
grade the project. Hope all is well with regards to finals and other
projects! Let me know how much more time you think you'll need.
—
Reply to this email directly or view it on GitHub
https://github.com/berkeley-stat159/project-epsilon/issues/315#issuecomment-165618661
.

Hello @rossbar
I don't know if you saw my messages above.
Thank you.

Yup, I got them. Just finished successfully running all the analysis :+1:

OMG, really ??
I am sorry, it has been a very long week and I was very worried it would not work.
Thank you very much for your time and for being understanding !

Sorry I closed the issue

There was one remaining syntax error in the multicomparison part but that was a very simple fix

Oh, ok, thank you !


add jdk 17 with jenkins version

Adding JDK version 17 with specific jenkins version to test if it runs on CI with no issues

[x] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
[x] Ensure that the pull request title represents the desired changelog entry
[x] Please describe what you did
[x] Link to relevant issues in GitHub or Jira (none)
[x] Link to relevant pull requests, esp. upstream and downstream changes (none)
[x] Ensure you have provided tests - that demonstrates feature works or fixes the issue

I don't understand the motivation for the change.  JDK 17 is already being tested on Windows.  We intentionally chose to keep CI costs down.  Is there a compelling reason that we need Java 17 testing on both Windows and Linux?

@MarkEWaite, We are primarily interested in linux support with java 11 and 17, no objection to test windows as well. Would you say testing win+j17 provides good enough guarantee java 17 is covered on linux as well?
One thing that comes to mind is: https://github.com/jenkinsci/cloud-stats-plugin/blob/9294e95bce45bfa0779c94a824d79f91162df894/src/test/java/org/jenkinsci/plugins/cloudstats/CloudStatisticsTest.java#L209

@MarkEWaite, We are primarily interested in linux support with java 11 and 17, no objection to test windows as well. Would you say testing win+j17 provides good enough guarantee java 17 is covered on linux as well?
One thing that comes to mind is:
https://github.com/jenkinsci/cloud-stats-plugin/blob/9294e95bce45bfa0779c94a824d79f91162df894/src/test/java/org/jenkinsci/plugins/cloudstats/CloudStatisticsTest.java#L209

I don't recall ever detecting a problem on Java 17 Linux that was not also detected on Java 17 Windows, but I also understand your strong desire to be absolutely certain that the code is well behaved on Java 17 Linux.  I have felt that same strong desire with a few of the plugins that I maintain.  You and @npotluri-rh have my full support for the addition of Java 17 Linux as a test configuration.
Thanks for explaining!


Please update "Introduction"

Hey @spf13 :) I've made some significant responsiveness upgrades and updated the screenshot and readme. Please update my theme!
https://github.com/vickylaixy/hugo-theme-introduction
Thanks!

Will do.


Update some places from rust-vyper to fe

What was wrong
Some places where still using rust-vyper that should use Fe

@g-r-a-n-t can't ping you as a reviewer. Can you add me to the project?

Thanks! Added you as admin


(WIP) Build: Use ESLint instead of JSHint/JSCS

Work in progress: Still need to fix Travis configuration for ESLint 3 (which won't run on Node versions older than 4). Do not merge!
Open questions:

[x] What is the preferred IIFE-wrapping style? Wrap the call expression ("wrap the outside"), or wrap the function expression ("wrap the inside")? I've seen the former for parameterless IIFEs and the latter for IIFEs with parameters (so far).

Allow both for now, unless someone objects.

[ ] Did I disable and leave enabled the right set of ESLint rules for test/main/deepEqual.js?
[ ] Best way to configure Travis/Grunt since ESLint 3 won't run on Node older than 4?

Cripes, this will fail on Node <4.x due to ESLint no longer supporting that environment. Should I move the ESLint Grunt target into the build section so it's run on latest Node?

Should I move the ESLint Grunt target into the build section so it's run on latest Node?

Yes. we can run ESLint only in a single version of Node within Travis.
There's a package.json script called 'ci' calling grunt. We should redefine those to exclude linting tasks and make a special call for it on travis latest Node.

Okay, I've pushed a commit which fixes the ESLint issues in tests.
test/main/deepEqual.js proved particularly problematic, probably because it is testing some extreme cases. So here is what I did in that file:

Fixed space-before-function-paren, space-in-parens, and one semi-spacing issue
Replaced a /* jshint asi:true */ with equivalent /* eslint-disable-next-line: semi */
Disabled max-len (no more than 100 characters/line), array-bracket-spacing (require space just inside array brackets), and comma-spacing (require space after commas) for that whole file.

If you want me to do something different for that file (e.g., fix the comma-spacing issues), let me know and I can push another commit.
Everything else was pretty straightforward, but let me know if I missed anything important.

@leobalter I could use some help configuring Travis as you suggested. Also, can you confirm that AppVeyor intentionally only runs on latest Node and so no reconfiguration there will be necessary? Thanks!

What is the preferred IIFE-wrapping style?

Looks like both have examples in our JS style guide. I like the AirBnb approach to this. Additionally, I have heard, that using the (function() {}()) form is better optimized in V8 (but that shouldn't matter much to us).

Okay- their style guide says either way is fine, though their ESLint
configuration prefers wrapping the CallExpression. Wrapping the
CallExpression is also the mode that you said might be optimized better in
v8.
Here's what I propose. Since there is inconsistency right now, I will
ensure IIFEs are wrapped but allow both styles. Then in a future issue, if
someone wants to make the case for bringing consistency into the codebase
by switching to CallExpression/"outer" wrapping everywhere, we can evaluate
then.
@trentmwillis How does that sound?
On Aug 23, 2016 10:14 PM, "Trent Willis"<EMAIL_ADDRESS>wrote:

What is the preferred IIFE-wrapping style?
Looks like both have examples in our JS style guide. I like the AirBnb
approach https://github.com/airbnb/javascript#functions--iife to this.
Additionally, I have heard, that using the (function() {}()) form is
better optimized in V8 (but that shouldn't matter much to us).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/jquery/qunit/pull/1028#issuecomment-241946153, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AARWeqglcWZY4PLH0XbapMoYAh7q43_9ks5qi7cVgaJpZM4JfuM0
.

It sounds surprising that one IIFE form may be better optimized (whatever
that means) in v8 - it all disappears anyway when the code is JIT'ted.
Also, you'd have to create IIFEs in loops to see any difference, if it's
just for wrapping a few files then it shouldn't matter in the slightest.
On Wednesday, 24 August 2016, Kevin Partington<EMAIL_ADDRESS>wrote:

Okay- their style guide says either way is fine, though their ESLint
configuration prefers wrapping the CallExpression. Wrapping the
CallExpression is also the mode that you said might be optimized better in
v8.
Here's what I propose. Since there is inconsistency right now, I will
ensure IIFEs are wrapped but allow both styles. Then in a future issue, if
someone wants to make the case for bringing consistency into the codebase
by switching to CallExpression/"outer" wrapping everywhere, we can evaluate
then.
@trentmwillis How does that sound?
On Aug 23, 2016 10:14 PM, "Trent Willis"<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','notifications@github.com');>> wrote:

What is the preferred IIFE-wrapping style?
Looks like both have examples in our JS style guide. I like the AirBnb
approach https://github.com/airbnb/javascript#functions--iife to this.
Additionally, I have heard, that using the (function() {}()) form is
better optimized in V8 (but that shouldn't matter much to us).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/jquery/qunit/pull/1028#issuecomment-241946153, or
mute
the thread
<https://github.com/notifications/unsubscribe-auth/
AARWeqglcWZY4PLH0XbapMoYAh7q43_9ks5qi7cVgaJpZM4JfuM0>
.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jquery/qunit/pull/1028#issuecomment-241947186, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABrUnu6W6f1mpPsRWGL4XPmYdZWe0bX-ks5qi7k8gaJpZM4JfuM0
.

--
Michał Gołębiowski

@mgol Apologies, I thought I had responded to your comment. I'm going to interpret your comment as "IIFE wrapping style shouldn't matter" and thus continue with my current proposal. Let me know if you object.

@leobalter Okay, I think I've figured things out. I just pushed a commit which adds a package.json script "ci-include-lint", which runs grunt eslint && grunt test && grunt coveralls. In .travis.yml, I've added an extra build node to the matrix which runs that script. An example of how that job would look can be found here.
I think this gets us 95% of the way there. I think that maybe instead of "ci-include-lint" (which also runs the same CI job that is run on the default Node 6 build node), I should create a "lint" script which would only run ESLint on Node 6, that way we don't run more CI jobs than we need to. If you see this before I get a chance to try that experiment, please do let me know if you think that's a good direction.

I should create a "lint" script which would only run ESLint on Node 6, that way we don't run more CI jobs than we need to.

That's the first thing I noticed. Also the current way repeats the other tasks on ci-include-lint.
This is looking good, almost ready to go! ++

@leobalter Not sure I fully understood your reply. Do you want me to avoid creating an extra build node altogether? Or can I create that node (as I am currently doing), as long as it only lints rather than running the CI scripts as well?
I was proposing the latter, but if you would prefer I find another way, please tell me what you have in mind.

@platinumazure Currently you're invoking both ci and ci-include-lint on Node 6 which duplicates most stuff. Since Node 6 already runs the ci-include-lint task it shouldn't run the ci one. That's, I think, what @leobalter says.

Oh. I was going to go the other way and create a "lint" task (and remove
"ci-include-lint") which the extra node would run, plus have a Node 6 box
that still runs CI. Those seem equivalent to me except for one more build
node (and therefore one more npm install at least) in my proposal.
If you can advise me on how I can conditionally remove/set the existing
Node 6 build node to use ci-include-lint instead of ci in Travis config,
then I'm certainly okay with doing that too. This is the first time I've
had to go build matrix configuration with Travis, so I'm definitely taking
shots in the dark here.
I won't be able to make any modifications until tonight (US time) at
earliest anyway, so no huge rush. Thanks in advance!
On Sep 5, 2016 7:15 AM, "Michał Gołębiowski"<EMAIL_ADDRESS>wrote:

@platinumazure https://github.com/platinumazure Currently you're
invoking both ci and ci-include-lint on Node 6 which duplicates most
stuff. Since Node 6 already runs the ci-include-lint task it shouldn't
run the ci one. That's, I think, what @leobalter
https://github.com/leobalter says.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jquery/qunit/pull/1028#issuecomment-244733047, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AARWeoQkwwUjh3yfmBx_tnzirc0xmOmFks5qnAfbgaJpZM4JfuM0
.

If you can advise me on how I can conditionally remove/set the existing
Node 6 build node to use ci-include-lint instead of ci in Travis config,
then I'm certainly okay with doing that too.

The following URL should explain it a little:
https://docs.travis-ci.com/user/customizing-the-build#Build-Matrix

@mgol Thanks for the link! I already skimmed that to get as far as I have, but I guess I had to dig deeper. :smile:
I think this latest commit should be pretty much good enough to merge, unless I've missed something. By all means, please review and let me know if I should do something different (either in Travis config or elsewhere). I can squash if necessary, as well (just leaving commit history visible for now).
@leobalter Think I've figured out the Travis config now, please review at your convenience. Latest PR build can be found here: https://travis-ci.org/jquery/qunit/builds/157738305

Oh, and I could still use some feedback on my second big question: In tests/main/deepEqual.js, does the balance of rules enabled (with violations fixed) vs rules selectively disabled/ignored feel okay? Thanks!

In tests/main/deepEqual.js, does the balance of rules enabled (with violations fixed) vs rules selectively disabled/ignored feel okay?

Well, that file is a nightmare for linting tools. It feels fine for me in any case as I've passed some long moments trying to get it better and I know how hard it is to keep it ok. I would say let's keep the current way it is and see if we get any feedback in the future.

Okay, I think all open questions have been addressed at this point. Let me know if I should squash. Thanks!

LGTM, we've only got some remaining whitespace issues but those can be fixed during merge.
The Gruntfile.js file still has some indentation issues, and there are these inline rules which I would use the comment above form. Do you prefer sticking with the inline style?

@leobalter Left a response to your comment on eslint-disable-line.
Also, regarding indentation: I can try to find remaining indentation issues by using the ESLint "indent" rule. It's not enabled in eslint-config-jquery because the rule does not elegantly handle the top-level IIFE requirement in the jQuery style guide, but I can at least go through and try to fix any real errors. Sound okay?

On the indentation, I'm ok with that but the current issues are not many so we can merge this PR before and open a follow up. As you prefer.

I found a few files that had two-space indentation (these are files I
haven't yet touched). Shall I fix those up as well?
On Sep 6, 2016 8:23 AM, "Leo Balter"<EMAIL_ADDRESS>wrote:

On the indentation, I'm ok with that but the current issues are not many
so we can merge this PR before and open a follow up. As you prefer.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jquery/qunit/pull/1028#issuecomment-244948125, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AARWeu0zdbZosnoqviMs1bv7X7E30K8_ks5qnWkugaJpZM4JfuM0
.

@leobalter I've pushed a commit replacing eslint-disable-line with eslint-disable-next-line.
Regarding the indentation issues: I've been looking and I see a number of indentation problems throughout the repository (about 200 or so violations), many in files I didn't touch. So I guess I'd like to treat those as a separate issue. I am happy to fix any indentation problems I've introduced in this PR, if you can help me find them. Otherwise, can we merge? (Assuming there aren't other issues, of course.)
Let me know if you need me to rebase and/or squash.

Pinging @leobalter. Can you please let me know which indent errors you want me to fix?
Also, I'm happy to report that a long-standing bug with the indent rule has been fixed and will be released with ESLint 3.6.0: If tabs are required but spaces are present, it would report "0 tabs" instead of "n spaces", or vice versa. So we could take a serious look at indentation issues at that time, if desired.

I'm sorry I didn't answer this before, let get this done and merged

@leobalter No worries. As far as I'm concerned this is ready except for "some remaining whitespace issues". I fixed the Gruntfile.js indentation and any other mistakes I had introduced, as far as I know. Are there any other indentation problems that I introduced that I should fix, or can we merge?

I was already merging this, just runned a final batch of tests to confirm.
I've got it squashed on 89731f9.
Thanks, that's a great work!

Thanks! Sorry for pestering you when you were merging, I wasn't sure if you were still waiting on me.

No worries at all, you can ping anytime!


Agent streaming final response

I have been streaming the final response of an agent by using the callback system (handleLLMNewToken). This works just fine. I recently started using LangSmith in my porject and I started facing some issues. It seems that sometimes, the request to LangSmith takes a bit longer and results in the callback not being able to stream the new tokens. I assume that this is because the callbacks are awaited, and therefore my custom callback has to wait for the other to finish.
Looking for a solution, I ended up implementing the one proposed in this article: https://js.langchain.com/docs/modules/agents/how_to/streaming#streaming-tokens. The recommended solution is to use the streamLog and check the operations and values. This solution works fine too, but it is a bit cumbersome and difficult to read and understand imho.
In the same page, there is another section that talks about the stream method of the AgentExecutor. However, such method only returns the chunks after they have been completely received, that is, completely generated. This sounds a bit counterintuitive to me, since other stream methods, such as in the chat models (https://js.langchain.com/docs/modules/model_io/chat/streaming#using-stream) and the LLMs (https://js.langchain.com/docs/modules/model_io/llms/streaming_llm#using-stream) seem to indeed provide the tokens as soon as they come.
Wouldn't it make more sense that the stream method in the AgentExecutor yielded, at least, the tokens that the underlying model generates? It could even be generalized to stream both tokens and intermediate steps, and it would be up to the reader of the stream to decide what to do in each case: do something about the AgentStep or process the new token.
What do you think?
Thanks in advance!

Yes I agree that the default behavior of .stream() isn't super intuitive - I'll be lobbying to change that in a future release.
In the meantime, we're planning to add a more accessible version of streamLog to LangChain.js. This is what it looks like in Python: https://github.com/langchain-ai/langchain/pull/16172
Does that look like a reasonable middle ground?

I gave a glance at the changes in python and it looks sensible. I think it's a reasonable compromise for the moment.
IMO, a more unified version of stream would indeed be beneficial to everyone. So thanks for keeping an eye on that in the future.
Closing the issue now.
Cheers!


Can't run ui on linux

when running from terminal in linux got a window with a throw exception

SO: Archlinux
Kernel: 5.15.6-arch2-1
comictagger version: 1.3.0a4
python version: 3.10.1
installed with: pip install comictagger
command to start: comictagger

Hello @rurickdev, this appears to be an incompatibility with newer versions of PyQt. PyQt should be limited to versions <= 5.15.3 through the requirements file and I know that there are no wheels for Python 3.10 and PyQt5 5.13.3 so it is probably an issue with Arch using the latest PyQt.
In any case the next alpha release will fix this issue.

Awesome, I'll wait till that.
keep the good work!


Position the conpty cursor correctly when wrappedRow is set

Summary of the Pull Request
If the VT render engine is moving the cursor to the start of a row, and
the previous row was marked as wrapped, it will assume that it doesn't
need to do anything, because the next output should automatically move
the cursor to the correct position anyway.
However, if that cursor movement is coming from the final PaintCursor
call for the frame, there isn't going to be any more output, so the
cursor will be left in the wrong position.
This PR fixes that issue by clearing the _wrappedRow field before the
_MoveCursor call in the PaintCursor method.
Validation Steps Performed
I've confirmed that this fixes all the test cases mentioned in issue
#17270, and I've added a unit test to check the new behavior is working
as expected.
However, this change does break a couple of ConptyRoundtripTests that
were expecting the terminal row to be marked as wrapped when writing a
wrapped line in two parts using WriteCharsLegacy. This is because the
legacy way of wrapping a line isn't the same as a VT delayed wrap, so it
has to be emulated with cursor movement, and that can end up resetting
the wrap flag.
It's possible that could be fixed, but it's already broken in a number
of other ways, so I don't think this makes things much worse. For now,
I've just made the affected test cases skip the wrapping check.
PR Checklist

[x] Closes #17270
[x] Tests added/passed

In an ideal world the wrapping code needs a major rewrite, so if that's something someone is already planning to do, you may want to ignore this and wait for a proper fix. But if you want a quick resolution to #17270, this may be the least worst option for now. And I think having the cursor correctly positioned is way more important than regressing a few reflow wrapping cases that were never fully working to begin with.

Before I say anything else: This is great! I think we should merge this because alternatives won't arrive any time soon.

In an ideal world the wrapping code needs a major rewrite, so if that's something someone is already planning to do, you may want to ignore this and wait for a proper fix.

FYI coincidentally I did start writing a ConPTY variant today that does straight conversions to VT and has no VtEngine at all. Basic usage of pwsh and wsl already works. I believe the main complexity of the work will be the rewrite of cooked read in terms of VT sequences without direct TextBuffer access, removing all the "passthrough" logic out of the VT parser, as well as finding the numerous edge cases, like how we need to restore win32-input-mode when encountering reset sequences. I'm confident I can get a version out in ~a month that certainly will have bugs but should already be useable. 🙂
I'm sure the initial bugs will not be great, but working on it today I found that resolving these is significantly easier when the Renderer code isn't in the way. At least in my opinion. 😅


Add vis-2-shelly to latest

RE-CHECK!

Thanks for spending your time and providing a new adapter for ioBroker.
Your adapter will get a manual review as soon as possible. Please stand by - this might last one or two weeks. Feel free to continue your work and create new releases. You do NOT need to close or update this PR in case of new releases.
You will find the results of the review and evntually issues / suggestings as a communt to this PR. So please keep this PR watched.
If you have any urgent questions feel free to ask.
mcm1957

First of all - THANK YOU for the time and effort you spend to maintain this adapter.
I would like to give some feedback based on my personal oppinion. @Apollon77 might have additional suggestions or even a different oppinion to one or the other statement. Please feel free to contact him if you cannot follow my suggestions or want to discuss some special aspects. As this adapter is related to vis-2 a review from Aüpollon77 or Foxriver might be required too.

[ ] errors reported by adapter checker must be fixed
Especially error  [E201] Bluefox was not found in the collaborators on NPM!. must be fixed. If you just invited Bluefox its ignore to wait for his reaction. Please state that the invitation has been sent in this case.

[ ] directory readme
Documentation should be stored insiede directory doc. So please rename directory readme to doc.

[ ] please verify that all dependencies are up to date
Some dependencies are outdated. Please update to current releases if no special reason to keep an older version exists. There are several open dependabot PRs indicating outdated dependencies. Please merge unless theres a technical reason to delay an upgrade:

[ ] check 'singleton' flag
Is it possible to run multiple instances of this adapter? Id not you should set attribute 'singleton=true'

[ ] some UI files seem to be templates

i18n files are templates, please sync with words.js (use npm run translate ...) or remove them
custom config is likely misconfigured. Please check.

[ ] remove license entyr from io-package.json
license entry is deprected and has been replyed by licenseInformation. Only one of those entries must be present.

[ ] invalid characters should be filtered from object ids
Some characters are not allowed to be used as part of an object id. If an object id is not hardcoded or even depending on user input you should ensure that such an object id does not contain an invalid character.  Iobroker provides the constant adapter.FORBIDDEN_CHARS, i.e. by using the following snippet:

function name2id(pName) {
    return (pName || '').replace(adapter.FORBIDDEN_CHARS, '_');
}

[ ] avoid usage of setObject
setObject(Async) creates a new object whenever it is called. You should avoid using setObject unless you really want to create new objects. Creating a new object deletes any userConfig, i.e. configures history logging which is most likely not desired. Most likely either setObjectNotExists oder (even better in most cases) extendObject should be used. Extend Object ensures that attributes like role, read/write flag etc. are set as specified and hence extendObjet shul dbe preferred when called onece during startup of the adapter.

[ ] checkfor missing  await
at some locations await seems to be missing. I.e. here https://github.com/Kollos2k/ioBroker.vis-2-shelly/blob/e195af3a67a7b115268f3a1ba67122f05eed656d/main.js#L232 you use setObject to create a state without await and assign a value 3 lines below. This state might not be created when trying to set it. Please check all your async operations.

[ ] unused routine ?
Routine createAndSetState seems to be unused. Please remove if this is correct.
https://github.com/Kollos2k/ioBroker.vis-2-shelly/blob/e195af3a67a7b115268f3a1ba67122f05eed656d/main.js#L316

[ ] onStateChange handler ignores ack flag
The adapter must honor the ack flag when processing onStateChange events. Whenever the onStateChange handler is called for an own state (a state owned by the processing instance of the adapter) it must ignore this event if the ack flags is set. Only stateChanges with ack==false are allowed to be processed.
When processing foreign states which are output of another adapter  processing must only occure if the ack flag is true.

[ ] reevaluate state roles
Only the values specified here (https://github.com/ioBroker/ioBroker.docs/blob/master/docs/en/dev/stateroles.md) may be used as state roles. Do not invent new names as this might disturb the functionalyity of other adapters or vis.
In addition the roles MUST match the read/write functionality. As an example a role value.* requires that the state is a readable state. Input states (write only) should have roles like level.*. Please read the description carefully. States with role 'button' should (must) have attribute 'common.read=false' set. A button ( "Taster" in german only triggers when you press but else has no state), so it should also be not readable. This is also like HomeMatic does it. A switch has clear states true or false and so can be read.
Please avoid using general roles (state, value) whnever a dedicated role exists.
Roles like role: "shelly.deviceList" are prohibited.
Ensure that write:true is ONLY set if this state is to be written by other adapters or userscripts.

[ ] some labels seem to have no translation into all languages
It seems that some labels are not translated into all supported languages, i.e. ...
Please consider adding missing translations

[ ] check and adapt testing
Standard iobroker testing is required but seems to be missing. Please add and setup ./gizthub/workflows/test-and-release.yml. Please add standard workflow. Please enable linter - should be possible with new code.

[ ] adapt testing to supported node releases
As node 14 is end of life and will no longer be supported by js-controller 5.x, consider removing node 14.x tests from matrix id still present.
Tests for node 16 is mandatory unless you require node 18 and higher (at package.json).
Tests for node 18 is mandatory as this is the recommended node version and node 16 will be eol and deprected Q3/2023.
Add node 20 tests unless you already know incompatibilities. In this case, try to fix it as soon as possible and definitly add a issue describing the problem.
So the recommended testmatrix is [16.x, 18.x, 20.x] or [18.x, 20.x] depending on engines requirments setting at package.json

[ ] Testing must be performed at all supported platforms (linux, windows, mac-os).
Unless there are any technical reasons to support only a subset of platforms, adapters must work at all platforms supported by ioBroker.

Thanks for reading and evaluating this suggestions.
Please keep in mind that this a quick first review so that you get a feedback as soon as possible. There might be additional remarks later.
McM1957
Please add a comment when you have reviewed and fixed the suggestionsor at least commented the suggestions and you think the adapter is ready for a re-review!
@Apollon77:

[ ] please indicate whether the NAME of this adapter is acceptable or it should include "widget" or so

@Kollos2k
Did you close this PR intentionally ?
May I ask why?

I try to figure out how all this works..... I have reopen it.

Frontend code looks good to me, nothing critical @mcm1957

I try to figure out how all this works..... I have reopen it.

No problem - I'll recheck as soon as I have time.

Severel issues still open. (NO problem).
Please add a comment when you have reviewed and fixed the suggestionsor at least commented the suggestions and you think the adapter is ready for a re-review!
Please note that I will not be really reachable the rest of this week.
reminder 25.3.2024

@Lokkos2k
How shall we continue?
Do you need any help?
Did you notice the last comment and the provided PR?
Please check andfix or at least comment the issues listed earlier (https://github.com/ioBroker/ioBroker.repositories/pull/3384#issuecomment-1979597075)
reminder 8.4.2024

I have no idea what I have to do. Everything seems good for me and there are no errors I have to fix?...

I think @mcm1957 refers to this todo list, where some entries are still open https://github.com/ioBroker/ioBroker.repositories/pull/3384#issuecomment-1979597075

I have no idea what I have to do. Everything seems good for me and there are no errors I have to fix?...

Did you check / process the list of issues I posted earlier (https://github.com/ioBroker/ioBroker.repositories/pull/3384#issuecomment-1979597075) and think those points are solved?
Then If yes I will do a recheck. Please indicate.
I did not see a comment stating that the adapter is ready for review yet so I posted this reminder.

Ok. I have to check it at evening.

Ok. I have to check it at evening.

Thanks. And please feel free to ask if you have any questions and need support.

I hope I have fixed the most issues.

RE-CHECK!

RE-CHECK!

@Kollos2k
Thanks for all the fixes already performed. Please see my notes from the last review below.

Testing must be fixed.
Updates of dependencies read as a note / suggestion only.
ack Handling at onStateChange must be fixed / discussed. Might be that I do not understand the purpose of your onStateChange handler. Please respond if something should be clearified.

In general 90% are done - good work and THANK you.

[ ] please verify that all dependencies are up to date (NON-blocking !)

Some dependencies are outdated. Please update to current releases if no special reason to keep an older version exists. There are several open dependabot PRs indicating outdated dependencies. Please merge PRs unless theres a technical reason to delay an upgrade

[ ] onStateChange handler ignores ack flag

The adapter must honor the ack flag when processing onStateChange events. Whenever the onStateChange handler is called for an own state (a state owned by the processing instance of the adapter) it must ignore this event if the ack flags is set. Only stateChanges with ack==false are allowed to be processed.
When processing foreign states which are output of another adapter processing must only occure if the ack flag is true.
You subscribe to OWN states. But you do not check for ACK flag to be false when getting notified about a state change occuring at own states.
I think it will be necessary to distuingish whether onStateChange occures for own or for foreign states as this will likely require different actions. In any case ack handling should be different.

[ ] adapt testing to supported node releases

Tests for node 18 are mandatory as this is the recommended node version and node 16 will be eol and deprected Q3/2023.
Tests for node 20 are mandatory unless you already know incompatibilities which cannot be fixed immidiatly. In this case, try to fix it as soon as possible and definitly add a issue describing the problem.
So the recommended testmatrix is [18.x, 20.x] depending on engines requirments setting at package.json
Please add a comment when you have reviewed and fixed the suggestionsor at least commented the suggestions and you think the adapter is ready for a re-review!
reminder 12.4.2024

🔸I don't know how to fix the tests. I use the GitHub automation tests like showed in the video from Haus-Automatisierung.com....

Script is ok, but node 20 had been commented out. I readded it to test matrix. See PR:
https://github.com/Kollos2k/ioBroker.vis-2-shelly/pull/18

🔸 I'll fix the depencies.

Thanks

🔸I thought I have fixed the onStateChange issue. In the main.js the onStateChange will fired but only an function is called if the ack is true..... Or I have misunderstood the issue?

I will reposnd to this a little bit later

@Kollos2k

🔸I don't know how to fix the tests. I use the GitHub automation tests like showed in the video from Haus-Automatisierung.com....

I readded it to test matrix and provided a PR. Hope this explains everything for this point. Kollos2k/ioBroker.vis-2-shelly#18

I c reated the mentioned PR. Merging this PR should correct test-and-release.yml.
Please check and merge the PR.

🔸I thought I have fixed the onStateChange issue. In the main.js the onStateChange will fired but only an function is called if the ack is true..... Or I have misunderstood the issue?

I will reposnd to this a little bit later

Sorry for the very late answer. Was busy the last days.
onStateChange is called whenever a subscribed state is changed.
You subscribe to i.e. "shelly.*" which are foreign states. Whenever an onStateChange from those foreign states is received, your adapter should (most likely) only react on changes with ack=true. This rule is typical and there might be reasons to also process events with ack=false.
You also subscribe to OWN states (= states created by this adapter) as far as I read the code. Any write to a state of your adapter must be ignored if ack=true. Users (vis, java-script) must write changes with ack=false.
So you must most likely filter ack value depending an "class" of state.
Maybe it wouldbe goog to explain what state changes you expect done and what reaction you will perfrom on them,
Please comment / discuss if any more questions,
reminder 17.4.2024

@Kollos2k
Do you need any help or have any questions to discuss?
reminder 27.4.2024

RE-CHECK!

New PR to update tests to nodejs 18/20
https://github.com/Kollos2k/ioBroker.vis-2-shelly/pull/22

@Kollos2k
How should we continue?
New PR to update tests to nodejs 18/20 has been proivided but not yet merged.
https://github.com/Kollos2k/ioBroker.vis-2-shelly/pull/22
Do you still work on this adapter oder did you stop working on it?
I do not want to recheck if you lost interest. But according to last comments most likely missing dependendy updates and especially missing testing with node 20 are blocking.
So please check and merge the PR and update testing and adapter-core to current versions (otherwise the tests will most likely fail). If you need an PR to update dependencies, please let me know.
reminder 28.5.2024

@Kollos2k
How should we continue?
New PR to update tests to nodejs 18/20 has been proivided but not yet merged.
https://github.com/Kollos2k/ioBroker.vis-2-shelly/pull/22
Do you still work on this adapter oder did you stop working on it?
I do not want to recheck if you lost interest. But according to last comments most likely missing dependendy updates and especially missing testing with node 20 are blocking.
So please check and merge the PR and update testing and adapter-core to current versions (otherwise the tests will most likely fail). If you need an PR to update dependencies, please let me know.
If there is no reaction at all until 30.6.2024 this PR will be closed
reminder 15.6.2024

@Kollos2k
How should we continue?
New PR to update tests to nodejs 18/20 has been proivided but not yet merged.
https://github.com/Kollos2k/ioBroker.vis-2-shelly/pull/22
Do you still work on this adapter oder did you stop working on it?
I do not want to recheck if you lost interest. But according to last comments most likely missing dependendy updates and especially missing testing with node 20 are blocking.
So please check and merge the PR and update testing and adapter-core to current versions (otherwise the tests will most likely fail). If you need an PR to update dependencies, please let me know.
If there is no reaction at all until 30.6.2024 this PR will be closed
reminder 01.07.2024

As there was no reaction until now I will close this PR.
Please feel vre to create a new PR at any time.


Run CI using image built in incubator-nuttx-testing and enable esp32 build

This updates the GitHub action to use the container that is built by incubator-nuttx-testing.  It also update the ESP32 postbuild step to have the binary blob path be configurable making it easier to use pre-built files rather than expecting the whole esp-idf project to be available.
This does require this PR to be merged first apache/incubator-nuttx-testing#20

@btashton the path need rebase the latest mainline, because PR:
https://github.com/apache/incubator-nuttx/pull/635
https://github.com/apache/incubator-nuttx-apps/pull/145

@xiaoxiang781216 Should be good now.  Let me know if there is anything else.


[BUG] Does not work with chrome extensions

Environment

chromium Version: 117.0.0
puppeteer-core Version:
Node.js Version: 18.16.0
Lambda Runtime: nodejs18.x

Expected Behavior
Hello there, thanks for maintaining this library!
I don't exactly know if it's a bug report, a question, or a feature request, but here's what I have a problem with: I want to start chromium with --headless=new and start it with an extension that uses the tabCapture API: puppeteer-stream. This library adds the extension with the launch args:

--load-extension
--disable-extensions-except
--allowlisted-extension-id

When I test it locally with the exact same launch options and the same chrome version, it works without a problem. On AWS Lambda, this does not work sadly.
Current Behavior
The browser starts successfully and does not produce any warning or error logs (with dumpio and a --log-level arg) and I can interact with it as usual. But as soon as the puppeteer-stream library executes browser.waitForTarget to wait for the background service worker of the chrome extenion, it fails with the TimeoutError.
I added event listeners to log created and destroyed targets, and the extension does not pop up.
Maybe the custom compiled chromium version does not support extensions?
Steps to Reproduce
const { getStream, launch } = require('puppeteer-stream');
const puppeteer = require('puppeteer-core');
const chromium = require('@sparticuz/chromium');
const fs = require('node:fs');

chromium.setHeadlessMode = 'new'
chromium.setGraphisMode = true

/**
 * Args that are forbidden for chromium for video and audio capturing to work properly.
 * Some of these args are set by us and some are set by "puppeteer-stream".
 */
const FORBIDDEN_PUPPETEER_ARGS = [
  '--disable-extensions',
  '--mute-audio',
  '--window-size',
  '--disable-component-extensions-with-background-pages',
  '--disable-default-apps',
  '--headless',
]

exports.handler = async (event, context, callback) => {
  let result = null;
  let browser = null;

  try {
    browser = await launch(
      {
        launch: async (opts: PuppeteerLaunchOptions) => {
          const options: PuppeteerLaunchOptions = { ...opts, headless: 'new' }

          const extensionPath = options.args!.find((arg) => arg.startsWith('--load-extension='))!.split('=')[1]
          console.log('debug', 'Loading extension', { extensionPath, contents: fs.readdirSync(extensionPath) })

          console.log('debug', 'Launching browser', options)
          return puppeteer.launch(options)
        },
      },
      {
        executablePath: await chromium.executablePath(),
        args: [
          chromium.args.filter((arg) => FORBIDDEN_PUPPETEER_ARGS.every((forbidden) => !arg.startsWith(forbidden))),
          `--window-size=1920,1080`,
          '--start-fullscreen',
          `--ozone-override-screen-size=1920,1080`,
          '--headless=new',
        ],
        ignoreDefaultArgs: true,
        waitForInitialPage: false,
        defaultViewport: null,
        ignoreHTTPErrors: true,
      },
    )

    let page = await browser.newPage();

    await page.goto(event.url || 'https://example.com');

    const stream = await getStream(page, { audio: true, video: true });
    const output = '/tmp/testfile.webm';
    const file = fs.createWriteStream(output);
    stream.pipe(file);
    await new Promise((resolve) => {
      setTimeout(async () => {
        await stream.destroy()
        file.close()
        console.log('debug', 'Destroyed stream')
        resolve(true)
      }, 1000 * 10); // 10 seconds
    });

    const fileSize = fs.statSync(output).size
    return callback(null, JSON.stringify({ fileSize }))
  } catch (error) {
    return callback(error);
  } finally {
    if (browser !== null) {
      await browser.close();
    }
  }

  return callback(null, result);
};

-->

Did you end up finding a fix for this?

No, we shifted to Google Cloud Platforms and ended up using Google Cloud Run with @google-cloud/functions-framework and custom Docker images.
On GCP, you can even run headful chromium with puppeteer. So yeah 🤷🏽♂

No, we shifted to Google Cloud Platforms and ended up using Google Cloud Run with @google-cloud/functions-framework and custom Docker images.
On GCP, you can even run headful chromium with puppeteer. So yeah 🤷🏽♂

Thanks, will check it out :)

Happy to help :) Just for clarification: I don't use this package anymore but install the default puppeteer chromium or I install it via a system package.

No, we shifted to Google Cloud Platforms and ended up using Google Cloud Run with @google-cloud/functions-framework and custom Docker images.
On GCP, you can even run headful chromium with puppeteer. So yeah 🤷🏽♂

Having the same problem as you @pantajoe  and @ashwwwin. Before I port my project to GCP I'd like to ask.. Is there a reason you opted for GCP? With AWS Lambda, you can also use docker.

No, we shifted to Google Cloud Platforms and ended up using Google Cloud Run with @google-cloud/functions-framework and custom Docker images.
On GCP, you can even run headful chromium with puppeteer. So yeah 🤷🏽♂

Having the same problem as you @pantajoe and @ashwwwin. Before I port my project to GCP I'd like to ask.. Is there a reason you opted for GCP? With AWS Lambda, you can also use docker.

Sure, I get wanting to make sure 😄 The reason is plain and simple: We tried using AWS Lambda with customer docker images before porting our serverless functions to GCP. Unfortunately, we found that the same restrictions apply 😬 We used alpine and tried installing chromium both via the default alpine repository as well as with this package and both approaches did not work at all. Chromium installed via alpine did not even start which is why this package here exists. And using this package's custom chromium had exactly this limitation that extensions do not work.

@pantajoe Went down the rabbit hole of setting the sparticuz/chromium and running custom docker images :D
Perhaps interesting: default chromium.args provide by this package have a flag --disable-extensions so it is impossible to get it to work out of the box. Unfortunately, even with this flag omitted, extensions don't load. Perhaps someone with more knowledge of chromium internals could get it to work if they knew that extensions are disabled via flag in the first place.
Regarding docker images, I had great success just installing google chrome from the debian repository. Here is image example: https://dev.to/cloudx/how-to-use-puppeteer-inside-a-docker-container-568c. Extensions work as expected.
The problem is, that it does not pass cloudflare bot detection if the script is run inside docker image, but works perfectly fine if run from my machine directly.

@mittster Thanks for the update! Interesting, I certainly encountered the very same article you linked during my work 😄 Did you run Chromium in headless or headful mode? Because for headless mode I found that customer docker images and downloading chromium from a repository work like you described (for me, my extension with puppeteer-stream didn't work, maybe others do). However, it didn't work with headful Chromium at all for me.

@pantajoe All the tests I've done were in headless mode. Never succeeded running in headful.
I've ported to GCP Functions and bot detection issues are the same. I suppose I shouldn't be surprised, because GCP Functions use docker internally. And thats not because of puppeteer(I don't use puppeteer, but custom implementation to avoid detection).
You said in one of the posts above: On GCP, you can even run headful chromium with puppeteer. May I ask how did you do it?

I haven't kept up with this whole conversation, but puppeteer's docs state you can do this
const browser = await puppeteer.launch({
  ignoreDefaultArgs: ['--disable-extensions'],
});

I haven't kept up with this whole conversation, but puppeteer's docs state you can do this
const browser = await puppeteer.launch({
  ignoreDefaultArgs: ['--disable-extensions'],
});

Yes, that's true, I even did ignoreDefaultArgs: true (see original issue description) to ignore all default args, and set every flag in headless mode myself, but that didn't work either 😅
Since that's no longer relevant for me, feel free to close the issue for now.

I have the same issue right now. Starting Chromium with an extension works locally, but it doesn't work on Lambda. I tried every possible combination of args and investigated the logs, but it just doesn't work. I don't really know if it is an Chromium or Puppeteer issue, but currently leaning towards to think that the Chromium Linux build is not working properly.
I assume this issue will pop up even more over time since it's possible to run extension in the new headless mode of Chromium.

@zirkelc did you work it out?

@cantonalex i stopped trying, it may work in newer versions of Chrome/Puppeteer