[BUG] [Potential Issue] Cluster state reponse handling thread blocked

Describe the bug
Observed this during gradle-check run for https://github.com/opensearch-project/OpenSearch/pull/12813#issuecomment-2011604246
https://build.ci.opensearch.org/job/gradle-check/35542/console
3 generic threads were blocked for processing publication response (it was 3 node cluster in test)
Thread[id=5851, name=opensearch[node_t0][generic][T#2], state=BLOCKED, group=TGRP-SearchWeightedRoutingIT]
  2>         at org.opensearch.cluster.coordination.Coordinator$5.onResponse(Coordinator.java:1381)
  2>         at org.opensearch.cluster.coordination.PublicationTransportHandler$PublicationContext$3.handleResponse(PublicationTransportHandler.java:442)
  2>         at org.opensearch.cluster.coordination.PublicationTransportHandler$PublicationContext$3.handleResponse(PublicationTransportHandler.java:433)
  2>         at org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleResponse(TraceableTransportResponseHandler.java:72)
  2>         at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1501)
  2>         at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:420)
  2>         at org.opensearch.transport.InboundHandler.lambda$handleResponse$3(InboundHandler.java:414)
  2>         at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
  2>         at java.****/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  2>         at java.****/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)

All the threads were probably waiting on below mutex :
https://github.com/opensearch-project/OpenSearch/blob/f3d2beee637f63e38c8f26dbcee9f2a82f9c87b6/server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java#L1376-L1392
Related component
Cluster Manager
To Reproduce

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
Investigate which code path was holding mutex and if it can be optimized (lock duration for code path which was holding it). Right now, it is not clear for how long the threads were blocked.
Additional Details
No response

[Triage - attendees 1 2 3 4 5 6 7]
@shwetathareja Thanks for creating this issue

[Attendees - 1, 2, 3]
Next Steps

Verify if the test is flaky
Understand as to why the state publication is performed from generic threadpool


More testcases

https://github.com/gfxmonk/indent-finder/tree/master/test_files
?

Need more test-cases, especially failing ones, that proves it works ok in the wild.

There is a $40.00 open bounty on this issue. Add more on Issuehunt.

Checkout the Issuehunt explorer to discover more funded issues.
Need some help from other developers? Add your repositories on Issuehunt to raise funds.

@issuehunt has funded $40.00 to this issue.

Submit pull request via IssueHunt to receive this reward.
Want to contribute? Chip in to this issue via IssueHunt.
Checkout the IssueHunt Issue Explorer to see more funded issues.
Need help from developers? Add your repository on IssueHunt to raise funds.


Call log modal broken by JS

Sad to say, the modal JS stopped working when the search bar went in.
Need to fix so we can continue to log calls, but not a huge deal.

PR #129 addresses this, but it seems strange that it broke when the search bar went in. Makes me wonder if there's something else wrong...

There was some hardcore wrenching on the partials in my integration tests PR to dry everything up, so I think likely that's what screwed it up. At any rate, really appreciate your fixing it.

Sure thing!

On Mar 10, 2016, at 11:29 PM, Colin<EMAIL_ADDRESS>wrote:
There was some hardcore wrenching on the partials in my integration tests PR to dry everything up, so I think likely that's what screwed it up. At any rate, really appreciate your fixing it.
—
Reply to this email directly or view it on GitHub https://github.com/colinxfleming/dcaf_case_management/issues/121#issuecomment-195173665.


Added Full stack open Course

Added Web-dev Course by Full stack open.

will they provide after completion of the course

Yes they have mentioned it in the FAQ section .
https://fullstackopen.com/en/faq


Handle additional folder draft operations

Proposed solution

After refactoring the folders to include drafts in the backend in #144, the frontend needs to take this into account. This means, when drafts are created, the folder where the draft belongs in should be patched accordingly. Also for publishing a folder, there is a new endpoint /publish/{folderId}. This endpoint discards the unfinished drafts within the folder and deletes them from the draft database collection. Therefore, upon publishing the user should be alerted that the drafts will be discarded.
DoD (Definition of Done)

-Additional PATCH operations are added for adding and removing drafts in folders
-Alerts are added for when folders are being published and drafts within the folders will be removed

dependent also on https://github.com/CSCfi/metadata-submitter/issues/153

Pending on #67


bsp/boards/stm32f1/sctimer: fix time division

This PR fixes a bug in the time division implementation for stm32f1 boards,  sctimer_vars.convert is never set to false, this error would only show up every 2^31/16384.

@changtengfei could you take a look? Should be straight forward since we discussed this offline.

@fjmolinas thanks for fixing this issue. Have you tested on one of the boards that is able to form a network? Just trying to make sure it doesn't break anything.

@fjmolinas thanks for fixing this issue. Have you tested on one of the boards that is able to form a network? Just trying to make sure it doesn't break anything.

I tested for a short time, I'll lunch an experiment running for a long time report back.

I tested for a short time, I'll lunch an experiment running for a long time and report back.

I also wasn't testing on a pure OpenWSN network, so I'll do that as well.

Actually seems like although the change makes sense code wise, it breaks something since I lost proper operation of the network... I need to revisit.


uPrint EEPROM uid starts with 00 and keeps changing

I'm trying to use this tool with a uprint eeprom using the one pin data method on a raspberry pi. The uid seems to keep changing every couple of seconds and it starts with 00- rather than the 23- that the guide says it should have. Anyone have any insight on this problem?

uprint is a bit special. You are definitly unable to write to this chip. It is possible to decrypt the data with bvanheus script but I have not had success to read it with this script. I found a script (tool.py) do dump the uprint chip and drcrypt it with bvanheus script. Take a look at my github and at @mayrthom. There is help for uprint .... to read, decrypt, recode and re-write it.

Im currently waiting for a null modem cable to read and write with the daig port to arrive. I'll definitely try that when I get it. Where is this tool.py script?

You can find the several scripts here: https://github.com/256FF/Refill-uPrint-EEPROMs
For communication over the DIAG Port you can use any Serial Terminal program.
I think it may be easier with my rewrite of the cartridgewriter, because it is an all in one solution for the Stratasys uPrint. It can read, decrypt and rewrite the Cartridges over the DIAG Port in one go.
You can download it here: https://github.com/mayrthom/CartridgeWriter_uPrint
If you have any questions feel free to ask.

Someone worte, that Machine ID for SE is 6B2A268B5ED3374A (same as "Ktype") but it was not confirmed yet. Furthermore mayrthom also has this ID in his code (machine.cs) but outcommented:
//new Machine {Number = new byte[8] {0x6B, 0x2A, 0x26, 0x8B, 0x5E, 0xD3, 0x37, 0x4A}, Type = "uprintse"}

has anyone located the correct machine ID for the uprint SE yet?

@cdnreprap see this commit https://github.com/bvanheu/stratatools/commit/41accb5cc35dfcbb94298694bc5f60aaaa82307c


Oops ...  Unintentional push.

I didn't realise I had push privileges to this repo.?  Looks like I pushed directly rather than a fork.  Sorry.

I am going to undo this push to master.


Adds support for generating Mermaid diagrams from APIs

Added a show command to the hidi command line and also added a WriteMermaid function to the OpenApiUriTreeNode class.
Examples of the output are here https://github.com/darrelmiller/openapi-collection/tree/main/ApiDiagrams
The show command wraps the mermaid in a markdown file with a legend to explain the coloring of the nodes.  Each node is styled with a color that maps to the set of HTTP methods that are supported.  This mapping is defined by the MermaidColorScheme dictionary on OpenApiUriTreeNode.  That dictionary can be customized to change the colors or add new combinations of HTTP methods.
GitHub markdown does not render the HTML legend that the show command emits.  However, the VS code markdown preview does.
This closes #820

@baywet   You tricked me with that .Order() method.  It doesn't exist.  At least my version of .NET doesn't know about it.  I put the .OrderBy( o=>o) back.

@baywet You tricked me with that .Order() method. It doesn't exist. At least my version of .NET doesn't know about it. I put the .OrderBy( o=>o) back.

@darrelmiller ah right! it's an addition of .net 7, sorry about that, I'm in the future.

OMG. I rebased successfully for the first time ever.

@baywet The git CLI, yes.  I was lucky because there were no merge conflicts.

Thank you @darrelmiller for taking this up. ☺

I'm still working on getting the test coverage up.  I'll try and get that done today or tomorrow.

Code coverage metric achieved.  :-)

good job on the coverage. I encourage you to check the code smells to wrap this up (+ the comment I left) https://sonarcloud.io/project/issues?id=microsoft_OpenAPI.NET&pullRequest=1113&resolved=false&types=CODE_SMELL

This PR now also fixes #1124 and #1127


Experimental: Reinstance messages

Considering that Youtube is using Webcomponent / Polymer, it may be possible to reinstance it and get access to the block/report function...

Experiment ended with reusing the menu from Youtube in ytcFilter 2.1.0

Experiment ended with reusing the menu from Youtube in ytcFilter 2.1.0


SuperEditor doesn't build with current flutter master

Text Editing Model refactor has landed (https://github.com/flutter/flutter/commit/2382b4c046623da0cfe9cc1cafc4ad8ea1b161e7) and removed RenderEditable.nextCharacter (used here). Same goes for previous character.

This should be a trivial fix moving the prevCharacter and nextCharacter methods into a utility class. I'm working on a fix locally, should be done soon.
NB: This is why using methods visible for testing is bad - and why there's a TODO to fix this :)
https://github.com/superlistapp/super_editor/blob/191410aba2ef54135d97616b30a17a6d2b5b0522/super_editor/lib/src/infrastructure/text_layout.dart#L115-L120


After upgrading from 0.66 to 0.73.6 i am getting build error on tvos 'React/RCTTVView.h' file not found

New Version
0.73.6-3
Old Version
0.64.2-2
Build Target(s)
debug tvos target
Output of react-native info
System:
OS: macOS 14.3
CPU: (10) x64 Apple M1 Pro
Memory: 195.58 MB / 16.00 GB
Shell:
version: "5.9"
path: /bin/zsh
Binaries:
Node:
version: 18.19.1
path: ~/.nvm/versions/node/v18.19.1/bin/node
Yarn:
version: 1.22.22
path: ~/.nvm/versions/node/v18.19.1/bin/yarn
npm:
version: 10.2.4
path: ~/.nvm/versions/node/v18.19.1/bin/npm
Watchman: Not Found
Managers:
CocoaPods:
version: 1.14.3
path: /Users/chadyayoub/.rvm/gems/ruby-2.7.6/bin/pod
SDKs:
iOS SDK:
Platforms:
- DriverKit 23.2
- iOS 17.2
- macOS 14.2
- tvOS 17.2
- visionOS 1.0
- watchOS 10.2
Android SDK: Not Found
IDEs:
Android Studio: 2023.2 AI-232.10300.40.2321.11567975
Xcode:
version: 15.2/15C500b
path: /usr/bin/xcodebuild
Languages:
Java:
version: 17.0.10
path: /usr/bin/javac
Ruby:
version: 2.7.6
path: /Users/chadyayoub/.rvm/rubies/ruby-2.7.6/bin/ruby
npmPackages:
"@react-native-community/cli": Not Found
react:
installed: 18.2.0
wanted: 18.2.0
react-native: Not Found
react-native-macos: Not Found
react-native-tvos:
installed: 0.73.6-3
npmGlobalPackages:
"react-native": Not Found
Android:
hermesEnabled: Not found
newArchEnabled: Not found
iOS:
hermesEnabled: Not found
newArchEnabled: false
Issue and Reproduction Steps
after upgrading i just run the project from xcode and i get build error 'React/RCTTVView.h' file not found. also xcode does not seem to find the reference to this import as when i cmd + click on #import <React/RCTTVView.h>, it does not take me to its reference.

There are many things that might have gone wrong when upgrading from such an old version.
I would start by checking your project's Podfile against the one in the current template (packages/react-native/template/ios/Podfile) to make sure that it has been updated, and do a clean pod install.

There are many things that might have gone wrong when upgrading from such an old version.
I would start by checking your project's Podfile against the one in the current template (packages/react-native/template/ios/Podfile) to make sure that it has been updated, and do a clean pod install.

Thank you, i was using the podfile that was in the react native main repo instead of the one in this repo after the upgrade, however now i am facing an issue with some pod installations, react-native-render-html used to work normally before the upgrade now whenever i try to do pod install i get this error: The platform of the target appName-tvOS (tvOS 12.0) is not compatible with react-native-render-html (5.1.1), which does not support tvOS.

Thank you

@ChadyAyoub4 I'm going to go ahead and close this issue... if you are still having problems, or have a minimal repo that reproduces the problem, please post here and I can reopen.

@douglowder The issue was as you mentionned i was using the podfile from upgrade helper which had 2 missing lines and installs pods for ios not tvos. it works now


Illegal string offset 'table'

This plugin is great! However I am having a random error appear.
I am occasionally getting the following error from the plugin:
Illegal string offset 'table' this is coming from the following line in the prepValue function: $value['table'] = TemplateHelper::getRaw($html);
It would appear the key can't be identified in the array, any ideas why this would be happening?
I should point out I am using this field inside a matrix, if that would make a difference.
Thanks,
Scott

Hi Scott,
I'm not totally sure why that would be happening - what version of Craft and PHP are you running?
Josh

Hi Josh,
Thanks for getting back.
Craft version is  2.3.2644
PHP version is 5.5.17
I did just find something else....as the error stopped me getting to the entry in the CMS is added a check around the line:
	if (isset($value['table'])) {
		$value['table'] = TemplateHelper::getRaw($html);
	}

I then deleted the table, created another and saved the entry. The error then no longer appeared and I could view the table on my front end.
Sorry I'm not too familiar with the craft plugin architecture, when does the 'table' key get created? It seems very random as I have multiple entries with the same fields and some work and others don't.
Cheers,
Scott

Ok - is it only happening with empty tables?
For example: create a block with a table field, make sure there are no rows then save it.

It worked fine with an empty table, I just get an empty table on the front end.
I've got around 40 'product' entries which use this and only 2 had the problem which I fixed by removing the table and adding a new one.
I just had one idea, maybe the 2 broken entries were created before a craft update and all others were created after. I can't be sure I'll have a quick look at the creation dates.
Cheers,
Scott

Good plan. I should probably add some more checks though so I'll do that anyway at some point.

I am getting this error too on a Craft 2 site, did anyone every figure it out?

Closing due to age, and we no longer support the Craft 2 version.


Install error

npm install rst2html results in:
npm ERR! Unexpected token < in JSON at position 25981

Actually the issues was with package-lock.json (https://github.com/npm/npm/issues/17340)


Expensive formatting calls in two Debug Logs

calls to the Gorep helper function are expensive (as noted in its comment).  While profiling some performance tests, I found two culprits which called it very often, accounting for roughly 9 seconds of CPU time in a 120 second profile.  These numbers will vary quite a bit based on the rules that are executed.
Would you consider the following two debug lines eligible for change or deletion?

logging all the bindings in RunJavascript accounted for 7 seconds
logging the output of RunJavascript accounted for an additional 2 seconds

The problem isn't in the logging, unfortunately it's in the call to Gorep that's passed to the logger. So omitting the call to Gorep would make it cheaper.  Internally I simply deleted the log lines as we don't consume debug-level messages.

I'll not give in to the urge to lament the lack of Go macros for these situations.
Is there a clever, effective optimization for Gorep? (It got its own function out of optimism for such a possibility.) Alternately -- and perhaps preferably -- perhaps Log needs a better implementation. Something that defers more computation until after the "should I log anything" check? Alternatively, maybe doing an explicit conditional ("Are we debugging?") before the Log call in these sensitive locations?

What's the desired functionality of Gorep?  The current implementation is "if it's a string, use the string. otherwise print the fully type-annotated representation".  It's used exclusively in logging, so I'm tempted to believe that changes away from "print the fully type-annotated representation" are okay.
I can see a few options here:

wrap any log statement using Gorep with a log level check.  Ugly and lazy, but a viable fallback
drop the type data, e.g. %#v -> %+v.  I have no idea what the performance impact of this is and will need to do some benchmarks
add a check to see if the interface given to Gorep implements the fmt.Stringer interface, and potentially use that.  I'll also need to do some digging to see what the interface{} types are under the hood at runtime, I don't have any intuition for what the binding types actually are.

What's the desired functionality of Gorep?

Historically it was used to do logging in a way that could be processed (by Rulio!) relatively easily while being flexible and robust. At least that was its aspiration -- at great cost. That said, I don't know of any real dependency today on the structure of the log messages.
The "Are we debugging" conditional sounds like an expedient solution. Maybe overhaul logging itself later?

The "Are we debugging" conditional sounds like an expedient solution. Maybe overhaul logging itself later?

Agreed.  I'll have a tiny PR within a couple hours.


Allow sorting of volunteer prize table

Great suggestions! I was able to really clean up the sorting function. Thanks!


[Flake] Bigtable integration test fails to connect to emulator with DEADLINE_EXCEEDED

Bigtable AsyncSampleRows test failed to connect to emulator. It failed on all 3 tries of the Bigtable integration tests with emulator. This was with the asan-pr build. I was able to reproduce the flake locally once.
Log: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/6847/74e19639dfb19e1eb44001086b073d3caea93381/fedora-34-asan/log-3635dbcd-1d00-4efa-b093-02dc6471ff43.txt
==================== Test output for //google/cloud/bigtable/tests:table_sample_rows_integration_test:
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from SampleRowsIntegrationTest
[ RUN      ] SampleRowsIntegrationTest.Synchronous
[       OK ] SampleRowsIntegrationTest.Synchronous (66 ms)
[ RUN      ] SampleRowsIntegrationTest.Asynchronous
2021-06-23T05:42:05.915530948Z [INFO] <140483371157056> Enabled logging for gRPC calls (google/cloud/bigtable/admin_client.cc:454)
2021-06-23T05:42:05.934103695Z [DEBUG] <140483371157056> CreateTable() << parent: "projects/cloud-cpp-testing-resources/instances/test-instance" table_id: "tbl-2021-06-23-42e7ch8kb3v8kot5908de6rsvv4y24991a" table { column_families { key: "family1" value { gc_rule { max_num_versions: 10 } } } column_families { key: "family2" value { gc_rule { max_num_versions: 10 } } } column_families { key: "family3" value { gc_rule { max_num_versions: 10 } } } column_families { key: "family4" value { gc_rule { max_num_versions: 10 } } } }  (./google/cloud/internal/log_wrapper.h:238)
2021-06-23T05:42:05.985121646Z [DEBUG] <140483371157056> CreateTable() << name: "projects/cloud-cpp-testing-resources/instances/test-instance/tables/tbl-2021-06-23-42e7ch8kb3v8kot5908de6rsvv4y24991a" column_families { key: "family1" value { gc_rule { max_num_versions: 10 } } } column_families { key: "family2" value { gc_rule { max_num_versions: 10 } } } column_families { key: "family3" value { gc_rule { max_num_versions: 10 } } } column_families { key: "family4" value { gc_rule { max_num_versions: 10 } } } granularity: MILLIS  (./google/cloud/internal/log_wrapper.h:243)
2021-06-23T05:42:11.032681993Z [INFO] <140483371157056> Enabled logging for gRPC calls (google/cloud/bigtable/data_client.cc:207)
2021-06-23T05:42:11.035126929Z [DEBUG] <140483371157056> SampleRowKeys() << table_name: "projects/cloud-cpp-testing-resources/instances/test-instance/tables/tbl-2021-06-23-42e7ch8kb3v8kot5908de6rsvv4y24991a"  (./google/cloud/internal/log_wrapper.h:255)
2021-06-23T05:42:11.056585779Z [DEBUG] <140483371157056> SampleRowKeys() >> not null stream (./google/cloud/internal/log_wrapper.h:257)
2021-06-23T05:42:11.099311762Z [INFO] <140483371157056> Enabled logging for gRPC calls (google/cloud/bigtable/data_client.cc:207)
2021-06-23T05:43:21.252844523Z [WARNING] <140483371157056> Failed to refresh connection. Error: Connection couldn't connect before requested deadline [DEADLINE_EXCEEDED] (google/cloud/bigtable/internal/common_client.cc:66)
....................................... this like 1000 times .......................................
2021-06-23T05:52:48.391821112Z [WARNING] <140483371157056> Failed to refresh connection. Error: Connection couldn't connect before requested deadline [DEADLINE_EXCEEDED] (google/cloud/bigtable/internal/common_client.cc:66)
google/cloud/bigtable/tests/table_sample_rows_integration_test.cc:48: Failure
Value of: samples
Expected: code is equal to OK and message is anything
  Actual: 64-byte object <04-00 00-00 FE-7F 00-00 A0-DA BD-07 30-60 00-00 11-00 00-00 00-00 00-00 11-00 00-00 00-00 00-00 F8-F2 F2-F2 F8-F2 F2-F2 60-D2 E4-91 FE-7F 00-00 F0-D0 E4-91 FE-7F 00-00 DD-21 2F-DC C4-7F 00-00>, whose status is Deadline Exceeded [DEADLINE_EXCEEDED], with a code that isn't equal to OK, but a message that is anything
2021-06-23T05:52:51.330837174Z [WARNING] <140483371157056> Failed to refresh connection. Error: Connection couldn't connect before requested deadline [DEADLINE_EXCEEDED] (google/cloud/bigtable/internal/common_client.cc:66)
2021-06-23T05:52:51.699781599Z [WARNING] <140483371157056> Failed to refresh connection. Error: Connection couldn't connect before requested deadline [DEADLINE_EXCEEDED] (google/cloud/bigtable/internal/common_client.cc:66)
2021-06-23T05:52:55.462748486Z [WARNING] <140483371157056> Failed to refresh connection. Error: Connection couldn't connect before requested deadline [DEADLINE_EXCEEDED] (google/cloud/bigtable/internal/common_client.cc:66)
[  FAILED  ] SampleRowsIntegrationTest.Asynchronous (644383 ms)
[----------] 2 tests from SampleRowsIntegrationTest (644450 ms total)

I blame #441 for these.  If the emulator crashes the first request fails (with kUnavailable) and all future retries fail until the test times out.

Closing issue because I agree that it is a dupe of #441


stargazer: 1.2.2 -> 1.2.3

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

There is 1.3.1  release.

That's a feature release ahead. 1.2.3 is the patched version of the release already in 24.05. I have a seperate PR updating unstable to 1.3.1.

That's a feature release ahead. 1.2.3 is the patched version of the release already in 24.05. I have a seperate PR updating unstable to 1.3.1.

Oh, I see.


Creating Brackets in quotes creates extra brackets

So when I create twig brackets ({% %} and {{ }}) inside single or double quotes, it creates a second set of brackets ({{ and {% ) inside the quotes.
I've been able to reproduce pretty consistently.

Hello. Fixed in v0.7.5.

You're amazing! Thank you!


modify handlers/main.yml to detect the boot mode

The code for regerating the grub2 config should be able to detect the boot mode.

Solved by pull request #31.


fix(onboarding): improved connection error messages

Description:
Improved connection error messages.
This PR ports over some error handling from master that got lost in the gRPC refactor and adds some additional detail to the Failed to connect before the deadline message to help users diagnose the issue.
Error messages are not i18n aware as is the case for many errors message throughout the app - adding i18n support to the error messages can be handled separately.
Motivation and Context:
Fix #1564
How Has This Been Tested?
Try connecting with an invalid cert.
Screenshots (if appropriate):

Types of changes:
Enhancement
Checklist:

[x] My code follows the code style of this project.
[x] I have reviewed and updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes where needed.
[ ] All new and existing tests passed.
[x] My commits have been squashed into a concise set of changes.

Coverage decreased (-0.07%) to 20.115% when pulling 253ae63631c243c6c9ea1dec6864c00cfdd899e4 on mrfelton:fix/1564-better-cert-errors into ed8bda52b149a98664c1a75071c736563a81157f on LN-Zap:next.


fix(rendering): set auth_backend default

PR progress checklist (to be filled in by reviewers)

[x] Changes to documentation are appropriate (or tick if not required)
[x] Changes to tests are appropriate (or tick if not required)
[ ] Reviews completed

What type of PR is this?

Primary type

[ ] [build]    Changes related to the build system
[ ] [chore]    Changes to the build process or auxiliary tools and libraries such as documentation generation
[ ] [ci]       Changes to the continuous integration configuration
[ ] [feat]     A new feature
[x] [fix]      A bug fix
[ ] [perf]     A code change that improves performance
[ ] [refactor] A code change that neither fixes a bug nor adds a feature
[ ] [revert]   A change used to revert a previous commit
[ ] [style]    Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)

Secondary type

[ ] [docs]     Documentation changes
[ ] [test]     Adding missing or correcting existing tests

Does this PR introduce a BREAKING CHANGE?

No.
Related issues and/or pull requests

Describe the changes you're proposing

This PR fixes rendering no-pillars rendering error by defining auth_backend
[CRITICAL] Rendering SLS 'base:alcali.package.install' failed: Jinja variable 'dict object' has no attribute 'auth_bac
kend'
local:
    Data failed to compile:
----------
    Rendering SLS 'base:alcali.package.install' failed: Jinja variable 'dict object' has no attribute 'auth_backend'

Pillar / config required to test the proposed changes

Debug log showing how the proposed changes work

Documentation checklist

[ ] Updated the README (e.g. Available states).
[ ] Updated pillar.example.

Testing checklist

[ ] Included in Kitchen (i.e. under state_top).
[ ] Covered by new/existing tests (e.g. InSpec, Serverspec, etc.).
[ ] Updated the relevant test pillar.

Additional context

Thanks @noelmcloughlin


Remove LocalRPCDebug flags from keybase loopback

I'm afraid we'll forget about these things and put things in logs we shouldn't when we start shipping this.
@gabriel

Please rebase

👍


Swap Gui.Builder.border parameter order

Arguments For

More consistent
More intuitive

Arguments Against

Can't use varargs

This is a breaking change, but not a major change. Gui.Builder.border is @ApiStatus.Experimental.

Deciding for this change. I don't think the need for multiple borders is common enough to be an argument for creating inconsistencies.


require / node version of comicgen

Enable comicgen to used in require('comicgen') and import('comicgen') environments. Probably default document.currentScript.src to http://gramener.com or /
@sanand0 -- is this something you'd like in this project?

@pratapvardhan What would be some uses of it? Is it to generate the SVG from the server-side? Based on how this would be used, we can prioritize it, and figure out how to structure it.

Yes. For now, I see these two.

SVG from the server-side
observablehq notebook compatible

Later is my primary motivation.

Got it. I'm not familiar with Observable notebooks -- but that'd be a powerful alternative to what we're doing right now (with Codepen).
Would you like to send a pull request for this, @pratapvardhan? That'll help me understand what code will work. Thanks!

A good use case can be to something like this import { Dee } from  'comicgen/react');
Similar same can be implement for various framework. (Angular, Vue etc)


4962 cleanup timestamp columns

This branch removes the dependency of the editor upon the created_at and updated_at columns traditionally added to user tables during cartodbfy.
Upcoming version of cartodbfy will not add them so the editor's code should be compatible with both scenarios:

existing tables have the created_at and updated_at columns and they should not be removed because they have associated triggers.
new tables won't have those columns but nevertheless they should be added to the dashboard.

See https://github.com/CartoDB/cartodb/issues/4962 for details
@Kartones please review (the tricky question is not what's wrong but what I might have missed).

retest this, please

:+1: Looks simple regarding the editor, as I see mostly specs so go ahead!


Providing support for SMART Authorization

If configured, will check scopes against resources served. See smart_auth.go for
details.
Created a Config struct. This can hold additional configuration information as
we add more features.
This will likely fail since we are pointed to master on the echo GitHub repository and that has moved to v2 which has API breaking changes.

Will be rebasing and making some small changes.


[GLUTEN-5880][CORE] Ignore fallback for ColumnarWriteFilesExec children

What changes were proposed in this pull request?
As title
Fixes: #5880
How was this patch tested?
added unit test and manual test:
create table wz_test_20240904 (a int) STORED AS PARQUET;
insert into wz_test_20240904 values (1);

before:

after this:

@Yohahaha @zhztheplayer could you please take a look?


Update Readme

#Update ReadMe for local envirnoment setup
In readme it states to run lerna clean and lerna bootstrap
which does not works fine
rather npm run repoclean which correspondes to lerna clean and
npm run bootstrap which corresponds to lerna bootstrap works fine
@irmerk  This is my first time in open source please guide me if something to correct and how to procede

Congratulations on opening your first issue @HarshJaiswani !
The scripts in package.json mean that npm run repoclean and lerna clean are equivalent, providing that you have lerna installed globally (which is part of the installation instructions).
It appears that you have some other issue related to running lerna.


Create input mangment subsystem

Create subsystem that would manage all input from:

mouse
keyboard

Key features:

multiplatform
event-based
easy to expand
easy to port on other under-the-hood-libraries

Input modules now suports window-based input events:

key strokes
mouse buttons
mouse movement
(mouse) scrolling

Input example shows basic set-up and usage of this modules.


chore(RingTheory): split up Algebraic.lean

Split up the file RingTheory/Algebraic.lean into a few subfiles:

Algebraic/Defs.lean: definition of algebraic and transcendental elements and (sub)algebras
Algebraic/Integral.lean: relate IsAlgebraic and IsIntegral
Algebraic/Basic.lean: most results on showing certain elements are algebraic (this is probably the file you want to import afterwards)
Algebraic/LinearIndependent.lean: showing certain families are linear independent using transcendentality
Algebraic/MvPolynomial.lean: results on multivariate polynomials
Algebraic/Pi.lean: a definition of algebraic function (which seems to be unused otherwise). See also mathlib3#11156.

I don't expect a particularly exciting import reduction, but it does clean up the file structure somewhat, in particular by moving the relation between algebraic and integral elements to one dedicated place (which had been irritating me for years!).

Great work! Do you have plan to also split AlgebraicIndependent (in another PR)?

Maybe we should swap the include order of Basic and Integral?

Ooh, good idea! Looks like we only need to move a few lemmas in order to do that, and it should clean up the dependencies quite a bit.

Thanks!
maintainer merge

Good find! It seems AlgebraicCard has shorter proofs but the namespace should be IsAlgebraic, not Algebraic. I'll dig history to find how both got merged in 2022 ...

I'll dig history to find how both got merged in 2022 ...

No, Mathlib.RingTheory.Algebraic.Cardinality is recently split from Mathlib.FieldTheory.IsAlgClosed.Classification by me 😅 I just copied the copyright header.

The content of Algebraic.Cardinality was added by @ChrisHughes24 in mathlib3#9370 on Feb 3, 2022 and the content of AlgebraicCard was added later by @vihdzp in #12869 on Apr 12, 2022. But @vihdzp should have become aware of the latter from her work in #14467 which touched the file.
I'm tagging the people to collect thoughts about merging of the files.

I see no problem with the merge, both are files about cardinalities and algebraic numbers.
I'm far from an expert on this part of Mathlib anyways, I just wanted to provide a simple proof of the existence of transcendental numbers.


Operation Abyss: New Tokyo Legacy [PCSE00579]

App summary

App name: Operation Abyss: New Tokyo Legacy
App serial: PCSE00579
App version: 1.01

Vita3K summary

Version: v0.1.5
Build number: 2663
Commit hash: https://github.com/vita3k/vita3k/commit/e8151664
CPU backend: Dynarmic

Test environment summary

Tested by: gymzatan
OS: Windows 11
CPU: AMD Ryzen 7 5800H
GPU: AMD Radeon(TM) Graphics
RAM: 16 GB

Issues
Distorted graphics.
Screenshots

Log
vita3k.log
Recommended labels

Ingame -
graphics bug

Vita3K summary

Version: v0.1.5
Build number: 2671
Commit hash: https://github.com/vita3k/vita3k/commit/bb7c843e
CPU backend: Dynamic

Test environment summary

Tested by: othmanead
OS: Windows 11
CPU: Intel i5-10300H
GPU: RTX 3060 mobile
RAM: 16 GB

Issue:

Game plays with minor graphical issues

Screenshots:

Log
vita3k.log
Recommended labels

Playable
Graphics bug


Ungit is not loading in the browser

Describe the bug:
The Ungit link, from the workspace "Open Tool", is not loading the web interface for the Ungit.

Expected behaviour:
Open the web interface for Ungit

Steps to reproduce the issue:
Open the ML-Hub login to the server.
2. Create a new server with a given name
3. After the server spawn, access the server.
4. Open Tool -> Ungit

Technical details:

Workspace version : 0.8.7
Docker version :
Host Machine OS (Windows/Linux/Mac): Linux
Command used to start the workspace :  ML-Hub
Browser (Chrome/Firefox/Safari): Chrome, Firefox

Possible Fix:

Additional context:

The problem was the port mapping between Ungit port and the Nginx port.

@LukasMasuch I have tested the issue further during the last couple of day. It is not related to ML Hub, it is an issue with traefik.
It is not able to handle the URL of Ungit. I was not able to understand why or how to fix it.

@IslamAlam Thanks for investigating 👍 I think the issue either needs to be fixed with traefik or with Ungit, we probably cannot really do anything to fix this issue within the workspace. If you find out more information why this happens, feel free to post it here.

@IslamAlam did you find a solution? I have the same issue but it's more widespread (all apps have the same response and kernel's won't connect. This is all via traefik over the public internet. If I connect via local IP to my server at home it all works just fine.

@alexjj what is the traefik' labels which you are using for the container of the ml-workspace?
This is my currnet setup which works with all apps except Ungit. My feeling is the # is the problematic character in the Ungit url path.
labels:
    traefik.docker.network: web
    traefik.enable: "true"
    traefik.http.middlewares.ml-workspace-https-redirect.redirectscheme.scheme: https
    traefik.http.routers.ml-workspace-secure.entrypoints: web-secure
    traefik.http.routers.ml-workspace-secure.rule: Host(`***.***.***`)
    traefik.http.routers.ml-workspace-secure.service: ml-workspace
    traefik.http.routers.ml-workspace-secure.tls: "true"
    traefik.http.routers.ml-workspace-secure.tls.certresolver: default
    traefik.http.routers.ml-workspace.entrypoints: web
    traefik.http.routers.ml-workspace.middlewares: ml-workspace-https-redirect
    traefik.http.routers.ml-workspace.rule: Host(`***.***.***``)
    traefik.http.services.ml-workspace.loadbalancer.server.port: '8000'

I'm still using traefik 1.7 so the old style. I had to add the passHostHeader and make a websockets entry point (wss) in my traefik.toml otherwise jupyter couldn't connect to the kernel. Ungit still doesn't work. Most other things do.
    labels:
      - "traefik.backend=mlworkspace"
      - "traefik.enable=true"
      - "traefik.port=8080"
      - "traefik.frontend.rule=Host:ml.domain.com"
      - "traefik.docker.network=web"
      - "traefik.frontend.passHostHeader=true"
      - "traefik.frontend.entryPoints=http,https,wss"


Move data plane libraries out of ResourceManagement folder

We want to move to a structure that has management plane libraries under ResourceManagement folder. Data plane libraries will exist at the same level as ResourceManagement folder. The following should be moved out

DataLake.Store
DataLake.StoreUploader
Graph.RBAC

@twitchax Please decide how we should do this

Is this still an issue?


[bugfix][WIP] Fix the bug of Chinese tokenization in 2.0.0rc1

Proposed changes:

Add a character-based tokenizer for Chinese and other languages not suit for whitespace tokenizer
Conditionally use character-based tokenizer in HFTransformersNLP

fix #6752
Status (please check what you already did):

[x] added some tests for the functionality
[ ] updated the documentation
[x] updated the changelog (please check changelog for instructions)
[ ] reformat files using black (please check Readme for instructions)

Thanks for submitting a pull request 🚀 @b-quachtran will take a look at it as soon as possible ✨

@tabergma Please take a look.

@howl-anderson How is it going? Would be awesome to get this in the Rasa Open Source 2.1 release on November 17th.

@tabergma @wochinge Sorry for the late reply, I recently working on another project intensively. I will concentrate a lot of time on this issue in these two days, hoping to finish this work before the 2.1 release.

This problem should also be solved once https://github.com/RasaHQ/rasa/pull/7027 is merged, which should happen soonish. It allows you to use any kind of tokenizer together with the LanguageModelFeaturizer. However, adding the CharacterTokenizer might still be relevant.

Thank you @tabergma, I will take a closer look.

@tabergma @wochinge progress update: I am working on a  PreTrainedTokenizerFast based solution, it's return_offsets_mapping option can greatly help me to resolve this issue:

additional methods to map between the original string (character and words) and the token space (e.g. getting the index of the token comprising a given character or the span of characters corresponding to a given token).

but it comes with a limitation:

Currently no “Fast” implementation is available for the SentencePiece-based tokenizers (for T5, ALBERT, CamemBERT, XLMRoBERTa and XLNet models).

I working on estimating how the limitation will influence the user, I will submit a detailed report about this.

@howl-anderson Just to clarify: You are discarding the changes you already did and trying to solve the problem with a completely new idea? If that is the case it might make sense to close this PR and open a new one once your solution is ready. Would also be great if you could share more details about the new idea on the original issue. Thanks!

@tabergma Yes, I will open a new PR If it proves to be feasible. I will provide details when it's ready, then we will discuss it to make sure its correctness. Right now, I am just doing some POC work.

Report on  PreTrainedTokenizerFast based solution
Status: draft
Test environment
Rasa 2.0.4
configure:
pipeline:
  - name: HFTransformersNLP
    model_name: bert
    model_weights: bert-base-chinese
  - name: LanguageModelTokenizer
  - name: LanguageModelFeaturizer
  - name: RegexFeaturizer
  - name: DIETClassifier
    epochs: 1000
    learning_rate: 0.001
    tensorboard_log_directory: ./tensorboard_log
  - name: ResponseSelector
    epochs: 1000
    learning_rate: 0.001
  - name: EntitySynonymMapper

Current issue
The training sample is 从[东畈村]("city")走了。. After tokenization by whitespace_tokenizer in rasa/nlu/utils/hugging_face/hf_transformers.py::_tokenize_example, its tokens_out will be Token(data={"number_of_sub_tokens": 7}, start=0, end=7, text="从东畈村走了。") and token_ids_out would be  [794, 691, 100, 3333, 6624, 749, 511].
Rasa will get token features from the sample (in rasa/nlu/utils/hugging_face/hf_transformers.py::_get_model_features_for_batch), during this process, it will merge the features of sub-tokens to the token (in rasa/utils/train_utils.py::align_token_features),
it will average the features of sub-tokens. where there has a BIG issue: since in most case of Chinese, it just similar to the 从东畈村走了。,
it can not be whitespace splittable, so lots of characters will merged into one token, in the worst case, whole text will be one token.
in the later BILOU label generation process (rasa/nlu/utils/bilou_utils.py::_add_bilou_tags_to_entities), it will discard the entity info when tokenization is incorrect. So in our case (从东畈村走了。), the BILUO will be ['O'], entity info are completely lost. And this situation is not rare in Chinese.
Proposal
change the implementation of rasa/nlu/utils/hugging_face/hf_transformers.py::_tokenize_example from
    tokens_in = self.whitespace_tokenizer.tokenize(message, attribute)

    tokens_out = []

    token_ids_out = []

    for token in tokens_in:
        # use lm specific tokenizer to further tokenize the text
        split_token_ids, split_token_strings = self._lm_tokenize(token.text)

        split_token_ids, split_token_strings = self._lm_specific_token_cleanup(
            split_token_ids, split_token_strings
        )

        token_ids_out += split_token_ids

        token.set(NUMBER_OF_SUB_TOKENS, len(split_token_strings))

        tokens_out.append(token)

    return tokens_out, token_ids_out

to
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(self.model_weights, use_fast=True)

    text = message.get(attribute)
    encoded_input = tokenizer(text, return_offsets_mapping=True, add_special_tokens=False)

    tokens_text_in = tokenizer.convert_ids_to_tokens(encoded_input["input_ids"])

    tokens_out = []

    token_ids_out = encoded_input["input_ids"]

    for token_text, position in zip(tokens_text_in, encoded_input["offset_mapping"]):
        token = Token(token_text, position[0], position[1])
        tokens_out.append(token)

    return tokens_out, token_ids_out

in our case (从东畈村走了。), the original implementation has such incorrect result:
tokens_out: [Token(data={"number_of_sub_tokens": 7}, start=0, end=7, text="从东畈村走了。")]
token_ids_out: [794, 691, 100, 3333, 6624, 749, 511]

but the proposal implementation has a correct result:
# tokens_out:
[
    Token(start=0, end=1, text="从"),
    Token(start=1, end=2, text="东"),
    Token(start=2, end=3, text="[UNK]"),  # 畈 is a UNK to model, the length of UNK token is correct and independent from how the UNK token is represent (`[UNK]` in our case)
    Token(start=3, end=4, text="村"),
    Token(start=4, end=5, text="走"),
    Token(start=5, end=6, text="了"),
    Token(start=6, end=7, text="。")
]
# token_ids_out:
[794, 691, 100, 3333, 6624, 749, 511]

Pros
The correctness of new implementation is guaranteed and maintained by transformers developer team, since they officially provide the offset_mapping feature.
A key advantage of this implementation is it is language-agnostic, this implementation can use to any language that transformers supported.
Cons
upgrade transformers is needed
Currently, the transformers in Rasa is 2.x version, but we need upgrade to 3.x so we can use the new tokenizer API. We need some tests to make sure the upgrade will not cause bugs.
some unusual cases need to care
some (so far only one) model family has unusual tokenization schema, such as xlnet,
# raw text:
"Hello, I'm a single sentence!"

# offset_mapping:
[(0, 1), (0, 5), (5, 6), (7, 8), (8, 9), (9, 10), (11, 12), (13, 19), (20, 28), (28, 29)]

# tokens:
['▁', 'Hello', ',', '▁I', "'", 'm', '▁a', '▁single', '▁sentence', '!']

Note that there are two token ('▁' and 'Hello') start at same location ((0, 1) and (0, 5)).
We need make sure our code can work correctly to such schema.
some models are not supported yet
Because

Currently no “Fast” implementation is available for the SentencePiece-based tokenizers (for T5, ALBERT, CamemBERT, XLMRoBERTa and XLNet models).

(from https://huggingface.co/transformers/main_classes/tokenizer.html#tokenizer)
so, some models are not supported yet (maybe yes in future),
Here is a supporting matrix for all the model families currently we supported: "bert", "gpt", "gpt2", "xlnet", "distilbert" and "roberta" (source: rasa/nlu/utils/hugging_face/registry.py::model_class_dict):
+----------+----------------------------------------------------------+---------------+----------------+----------------------+--------------------------------------------+----------------------------------+
|          | BERT                                                     | GPT           | GPT-2          | XLNet                | DistilBERT                                 | RoBERTa                          |
+==========+==========================================================+===============+================+======================+============================================+==================================+
| English  | bert-base-uncased: ✔                                     | openai-gpt: ✔ | gpt2: ✔        | xlnet-base-cased: ✔  | distilbert-base-uncased: ✔                 | roberta-base: ✔                  |
|          | bert-large-uncased: ✔                                    |               | gpt2-medium: ✔ | xlnet-large-cased: ✔ | distilbert-base-uncased-distilled-squad: ✔ | roberta-large: ✔                 |
|          | bert-base-cased: ✔                                       |               | gpt2-large: ✔  |                      | distilbert-base-cased: ✔                   | roberta-large-mnli: ✔            |
|          | bert-large-cased: ✔                                      |               | gpt2-xl: ✔     |                      | distilbert-base-cased-distilled-squad: ✔   | distilroberta-base: ✔            |
|          | bert-base-multilingual-uncased: ✔                        |               |                |                      | distilgpt2: ✔                              | roberta-base-openai-detector: ✔  |
|          | bert-base-multilingual-cased: ✔                          |               |                |                      | distilbert-base-multilingual-cased: ✔      | roberta-large-openai-detector: ✔ |
|          | bert-large-uncased-whole-word-masking: ✔                 |               |                |                      |                                            |                                  |
|          | bert-large-uncased-whole-word-masking-finetuned-squad: ✔ |               |                |                      |                                            |                                  |
|          | bert-large-cased-whole-word-masking-finetuned-squad: ✔   |               |                |                      |                                            |                                  |
|          | bert-base-cased-finetuned-mrpc: ✔                        |               |                |                      |                                            |                                  |
+----------+----------------------------------------------------------+---------------+----------------+----------------------+--------------------------------------------+----------------------------------+
| Chinese  | bert-base-chinese: ✔                                     |               |                |                      | distilbert-base-multilingual-cased: ✔      |                                  |
|          | bert-base-multilingual-uncased: ✔                        |               |                |                      |                                            |                                  |
|          | bert-base-multilingual-cased: ✔                          |               |                |                      |                                            |                                  |
+----------+----------------------------------------------------------+---------------+----------------+----------------------+--------------------------------------------+----------------------------------+
| Japanese | bert-base-multilingual-uncased: ✔                        |               |                |                      | distilbert-base-multilingual-cased: ✔      |                                  |
|          | bert-base-multilingual-cased: ✔                          |               |                |                      |                                            |                                  |
|          | cl-tohoku/bert-base-japanese: ✗                          |               |                |                      |                                            |                                  |
|          | cl-tohoku/bert-base-japanese-whole-word-masking: ✗       |               |                |                      |                                            |                                  |
|          | cl-tohoku/bert-base-japanese-char: ✗                     |               |                |                      |                                            |                                  |
|          | cl-tohoku/bert-base-japanese-char-whole-word-masking: ✗  |               |                |                      |                                            |                                  |
+----------+----------------------------------------------------------+---------------+----------------+----------------------+--------------------------------------------+----------------------------------+
| Thai     | bert-base-multilingual-cased: ✔                          |               |                |                      | distilbert-base-multilingual-cased: ✔      |                                  |
+----------+----------------------------------------------------------+---------------+----------------+----------------------+--------------------------------------------+----------------------------------+
| Korean   | bert-base-multilingual-uncased: ✔                        |               |                |                      | distilbert-base-multilingual-cased: ✔      |                                  |
|          | bert-base-multilingual-cased: ✔                          |               |                |                      |                                            |                                  |
+----------+----------------------------------------------------------+---------------+----------------+----------------------+--------------------------------------------+----------------------------------+

(source code at https://colab.research.google.com/drive/13MDUBDijAbsk4YhyRrro76xPCBqTn-XK?usp=sharing)
except the cl-tohoku/* (third-party models from https://github.com/cl-tohoku/) models are not supported, all other models are compatible now.

@tabergma @wochinge Please take a look at my proposal, we can discuss further to make sure its correctness and feasible.

@dakshvar22 Can you please also take a look. Thanks.

Thanks for the proposal!
With the latest changes from master (after https://github.com/RasaHQ/rasa/pull/7027 being merged), your immediate problem should be solved. We are going to release the update probably by the end of next week. You can now use, for example, the following pipeline
pipeline:
  - name: JiebaTokenizer
  - name: LanguageModelFeaturizer
    model_name: bert
    model_weights: bert-base-chinese
  - name: RegexFeaturizer
  - name: DIETClassifier
    epochs: 100
  - name: ResponseSelector
    epochs: 100
  - name: EntitySynonymMapper

The LanguageModelFeaturizer does not depend on the LangaugeModelTokenizer anymore. Using this pipeline we would not use any kind of whitespace tokenization but instead we would use the JiebaTokenizer which should be able to tokenize your Chinese example correctly. It would be great if you could check this out and let us know how it works for you.
Regarding your proposal to introduce fast tokenizers from HuggingFace: We briefly looked into them a few weeks back. However, back then some library dependencies were blocking us. We are planning to check if those dependencies are solved so that we can take a closer look. If we want to introduce the fast tokenizers from HuggingFace, we would most likely want to create a new tokenizer component for it. This could then be used together with the LanguageModelFeaturizer. However, before we would introduce them, we would need to check if they give us an actual benefit. E.g. does the fast tokenizer from HuggingFace do a better job on tokenizing a Chinese text compared to our JiebaTokenizer or MitieTokenizer? Do you already have some input on that?
Thanks for the input!

@howl-anderson Thanks for the amazingly detailed proposal! In addition to what @tabergma mentioned, I would be interested to know what would JiebaTokenizer and MitieTokenizer output as tokens if you used the same example input - "从东畈村走了。". Would the tokens be valid/different from what the BERT based tokenizer gave above in your proposed code snippet? If yes, it would be helpful if you can give some details on what Jieba/Mitie might be missing.

@tabergma @dakshvar22 Sure, I will work on the comparison, when it's ready I will let you know.

['从', '东', '[UNK]', '村', '走', '了', '。']
['从', '东', '畈', '村', '走', '了', '。']
["从东" ('从', '东'),"畈" ('[UNK]',),"村" ('村',),"走" ('走',),"了" ('了',),"。" ('。',)]
["从东" ('从', '东'),"畈" ('畈',),"村" ('村',),"走" ('走',),"了" ('了',),"。" ('。',)]

我要听 Michael Jackson 的歌
['我', '要', '听', '[UNK]', '[UNK]', '的', '歌']
['我', '要', '听', 'Michael', 'Jackson', '的', '歌']
["我要" ('我', '要'),"听" ('听',),"Michael" ('[UNK]',),"Jackson" ('[UNK]',),"的" ('的',),"歌" ('歌',)]
["我要" ('我', '要'),"听" ('听',),"Michael" ('Michael',),"Jackson" ('Jackson',),"的" ('的',),"歌" ('歌',)]

XIAOMI的5号电池多少钱？
['[UNK]', '的', '5', '号', '电', '池', '多', '少', '钱', '？']
['XI', '##A', '##OM', '##I', '的', '5', '号', '电', '池', '多', '少', '钱', '？']
["XIAOMI" ('[UNK]',),"的" ('的',),"5" ('5',),"号" ('号',),"电池" ('电', '池'),"多少" ('多', '少'),"钱" ('钱',),"？" ('？',)]
["XIAOMI" ('XI', 'A', 'OM', 'I'),"的" ('的',),"5" ('5',),"号" ('号',),"电池" ('电', '池'),"多少" ('多', '少'),"钱" ('钱',),"？" ('？',)]

Hello, I'm a single sentence!
['[UNK]', ',', '[UNK]', "'", 'm', 'a', 'single', 'se', '##nt', '##ence', '!']
['Hello', ',', 'I', "'", 'm', 'a', 'single', 'sentence', '!']
["Hello" ('[UNK]',),"," (',',),"I" ('[UNK]',),"'" ("'",),"m" ('m',),"a" ('a',),"single" ('single',),"sentence" ('se', 'nt', 'ence'),"!" ('!',)]
["Hello" ('Hello',),"," (',',),"I" ('I',),"'" ("'",),"m" ('m',),"a" ('a',),"single" ('single',),"sentence" ('sentence',),"!" ('!',)]

Note that the result of Jieba+BERT tokenizer almost equal to the result of pure BERT based tokenizers with one exception: "从东畈村走了。". Jieba incorrectly split the text into ["从东", '畈', '村', '走', '了', '。'], the correct one should be ["从", '东畈村', '走', '了', '。'], the consequence of this mistake is that the entity (从[东畈村]("city")走了。) misaligned and will be dropped. Chinese tokenization is considered as a hard question, currently SOTA can arrchive scores from 96.5 to 98.4 depends on corpura (see https://chinesenlp.xyz/docs/word_segmentation.html for details).
Also in Chinese and Japanese, there are exist some words they share the same characters (with different order) but not the same meaning: such like "蜂蜜" (honey) VS "蜜蜂" (bee). In such situation, average the feature of sub-tokens may have issue.

@howl-anderson Thanks for the report. Just so that I understand this better, a couple of questions -

Results under bert-base-chinese with JiebaTokenizer are produced by first applying JiebaTokenizer and then applying BERT Tokenizer with bert-base-chinese weights on each token individually?

Slightly related to (1) but, is it possible to load a BERT tokenizer inside the JiebaTokenizer itself?

If it's easier to answer the above questions with a script which generated these results, I am happy to look at that two.

For bert-xxx with JiebaTokenizer
I use the code from master (commit id: 27a38fe07c6153324f5aec3878eb928f1acd075f, which already include the code from #7027) for bert-xxx with JiebaTokenizer, I build it as a python pkg, install it, and modified a few codes to bring out the output:
The few codes that I changed (in rasa/nlu/featurizers/dense_featurizer/lm_featurizer.py::_tokenize_example):
        tokens_in = message.get(TOKENS_NAMES[attribute])
        tokens_out = []

        token_ids_out = []

        for token in tokens_in:
            # use lm specific tokenizer to further tokenize the text
            split_token_ids, split_token_strings = self._lm_tokenize(token.text)

            if not split_token_ids:
                # This is a bugfix from me to fix the situation that token.text is a whitespace character, which will cause self._lm_specific_token_cleanup() to raise an exception
                continue

            (split_token_ids, split_token_strings) = self._lm_specific_token_cleanup(
                split_token_ids, split_token_strings
            )

            token_ids_out += split_token_ids

            token.set(NUMBER_OF_SUB_TOKENS, len(split_token_strings))

            tokens_out.append(token)

            print(token.text, split_token_strings)

        print("=" *10)
        return tokens_out, token_ids_out

here is the input (nlu.yaml):
version: "2.0"

nlu:
- intent: greet
  examples: |
    - 知识就是力量
    - 我的手机是<PHONE_NUMBER>
    - 我的手机是1234567890
    - 请帮我导航去 KFC
    - 请帮我导航去KFC

- intent: mood_great
  examples: |
    - 从[东畈村]("city")走了。
    - 我要听 Michael Jackson 的歌
    - XIAOMI的5号电池多少钱？
    - Hello, I'm a single sentence!

here is the configure (config.yml):
language: zh

pipeline:
  - name: JiebaTokenizer
  - name: LanguageModelFeaturizer
    model_name: bert
    model_weights: bert-base-multilingual-cased  # or bert-base-chinese

the command is
rasa train nlu
here is the output:
for bert-base-multilingual-cased
知识 ('知', '识')
就是 ('就', '是')
力量 ('力', '量')
==========
我 ('我',)
的 ('的',)
手机 ('手', '机')
是 ('是',)
1234567890 ('1234', '5', '6', '7', '8', '90')
==========
我 ('我',)
的 ('的',)
手机 ('手', '机')
是 ('是',)
1234567890 ('1234', '5', '6', '7', '8', '90')
==========
请 ('请',)
帮 ('帮',)
我 ('我',)
导航 ('导', '航')
去 ('去',)
KFC ('K', 'FC')
==========
请 ('请',)
帮 ('帮',)
我 ('我',)
导航 ('导', '航')
去 ('去',)
KFC ('K', 'FC')
==========
从东 ('从', '东')
畈 ('畈',)
村 ('村',)
走 ('走',)
了 ('了',)
。 ('。',)
==========
我要 ('我', '要')
听 ('听',)
Michael ('Michael',)
Jackson ('Jackson',)
的 ('的',)
歌 ('歌',)
==========
XIAOMI ('XI', 'A', 'OM', 'I')
的 ('的',)
5 ('5',)
号 ('号',)
电池 ('电', '池')
多少 ('多', '少')
钱 ('钱',)
？ ('？ ('。',)
==========
我要 ('我', '要')
听 ('听',)
Michael ('[UNK]',)
Jackson ('[UNK]',)
的 ('的',)
歌 ('歌',)
==========
XIAOMI ('[UNK]',)
的 ('的',)
5 ('5',)
号 ('号',)
电池 ('电', '池')
多少 ('多', '少')
钱 ('钱',)
？ ('？',)
==========
Hello ('[UNK]',)
, (',',)
I ('[UNK]',)
' ("'",)
m ('m',)
a ('a',)
single ('single',)
sentence ('se', 'nt', 'ence')
! ('!',)
==========

For bert-xxx
I use Rasa 2.0.4 with some code change for use Fast tokenizer and print output message:
(in rasa/nlu/utils/hugging_face/hf_transformers.py::_tokenize_example):
        from transformers import AutoTokenizer
        tokenizer = AutoTokenizer.from_pretrained(self.model_weights, use_fast=True)

        text = message.get(attribute)
        encoded_input = tokenizer(text, return_offsets_mapping=True, add_special_tokens=False)

        tokens_text_in = tokenizer.convert_ids_to_tokens(encoded_input["input_ids"])

        print(">>> ", "BERT", text, tokens_text_in)

        tokens_out = []

        token_ids_out = encoded_input["input_ids"]

        for token_text, position in zip(tokens_text_in, encoded_input["offset_mapping"]):
            token = Token(token_text, position[0], position[1])
            tokens_out.append(token)

        return tokens_out, token_ids_out

here is the input (nlu.yaml):
same with the bert-xxx with JiebaTokenizer
here is the configure (config.yml):
language: zh

pipeline:
  - name: HFTransformersNLP
    model_name: bert
    model_weights: bert-base-multilingual-cased

the command is
rasa train nlu
here is the output:
for bert-base-multilingual-cased
>>>  BERT 知识就是力量 ['知', '识', '就', '是', '力', '量']
>>>  BERT 我的手机是<PHONE_NUMBER> ['我', '的', '手', '机', '是', '1234', '##5', '##6', '##7', '##8', '##90']
>>>  BERT 我的手机是1234567890 ['我', '的', '手', '机', '是', '1234', '##5', '##6', '##7', '##8', '##90']
>>>  BERT 请帮我导航去 KFC ['请', '帮', '我', '导', '航', '去', 'K', '##FC']
>>>  BERT 请帮我导航去KFC ['请', '帮', '我', '导', '航', '去', 'K', '##FC']
>>>  BERT 从东畈村走了。 ['从', '东', '畈', '村', '走', '了', '。']
>>>  BERT 我要听 Michael Jackson 的歌 ['我', '要', '听', 'Michael', 'Jackson', '的', '歌']
>>>  BERT XIAOMI的5号电池多少钱？ ['XI', '##A', '##OM', '##I', '的', '5', '号', '电', '池', '多', '少', '钱', '？']
>>>  BERT Hello, I'm a single sentence! ['Hello', ',', 'I', "'", 'm', 'a', 'single', 'sentence', '!']

for bert-base-chinese
>>>  BERT 知识就是力量 ['知', '识', '就', '是', '力', '量']
>>>  BERT 我的手机是<PHONE_NUMBER> ['我', '的', '手', '机', '是', '123456', '##78', '##90']
>>>  BERT 我的手机是1234567890 ['我', '的', '手', '机', '是', '123456', '##78', '##90']
>>>  BERT 请帮我导航去 KFC ['请', '帮', '我', '导', '航', '去', '[UNK]']
>>>  BERT 请帮我导航去KFC ['请', '帮', '我', '导', '航', '去', '[UNK]']
>>>  BERT 从东畈村走了。 ['从', '东', '[UNK]', '村', '走', '了', '。']
>>>  BERT 我要听 Michael Jackson 的歌 ['我', '要', '听', '[UNK]', '[UNK]', '的', '歌']
>>>  BERT XIAOMI的5号电池多少钱？ ['[UNK]', '的', '5', '号', '电', '池', '多', '少', '钱', '？']
>>>  BERT Hello, I'm a single sentence! ['[UNK]', ',', '[UNK]', "'", 'm', 'a', 'single', 'se', '##nt', '##ence', '!']

is it possible to load a BERT tokenizer inside the JiebaTokenizer itself?

I don't understand what do you mean? Do you mean that is it possible to load&use the BERT tokenizer in the JiabaTokenzier?

@tabergma @dakshvar22 Please take a look, thank you!

@howl-anderson Thanks for the detailed report 💯
It seems like the JiebaTokenizer does a good job compared to BERT. Do you still think it would be needed to introduce another tokenizer for Chinese?

In Chinese and Japanese, there exist some words they share the same characters (in a different order) but not the same meaning: such as "蜂蜜" (honey) VS "蜜蜂" (bee). In such a situation, the average feature of sub-tokens may have issues.

The reason we introduced the sub-tokens and taking the average feature vector is because we saw a lot of misaligned entities in our predictions. E.g. if you have a sentence like I am going to Berlin and Berlin would be split into Ber and lin. The entity extractor might predict just Ber as entity. As cleaning up those "incorrect" entities after prediction let to more confusion/errors, we decided to simply not do sub-word tokenization. As I am not familiar with Chinese at all, do you suggest that we should remove this kind of logic for Chinese? Or do you have any other idea how we could handle cases like the one you mentioned?

@tabergma Sorry for the delay (I have to be busy with things in the company). Yes, I agree with you that the way that #7027 implemented is good enough for most case. It should be selected as the solution. The report I summited is an evidence for the decision. It also provides some useful information about issues that user may report in the future which we already know that it may happens. This report provide the detailed info about why issues happen, we can write documents base on this to tell user how to fix it in feature.


[Snyk] Upgrade @ledgerhq/hw-transport from 6.27.1 to 6.30.0

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to upgrade @ledgerhq/hw-transport from 6.27.1 to 6.30.0.
:information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project.

The recommended version is 89 versions ahead of your current version.
The recommended version was released a month ago, on 2023-11-24.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.
For more information:
🧐 View latest project report
🛠 Adjust upgrade PR settings
🔕 Ignore this dependency or unsubscribe from future upgrade PRs

Closing the PR, reason: https://github.com/near/wallet-selector/pull/1032#issuecomment-1847070643


[BUG(ish)] Magic command flaw.

So I have this fun little code snippet:
current = spotify.currently_playing()
album_url = current['item']['album']['images'][0]['url']

@show(album_url)

where I'm grabbing an image link from spotify's api based on what song I'm currently listening to.
was losing my mind over trying to make matplotlib handle just raw images rather than trying to display them in complex diagrams.
saw that this plugin supported "magic commands" which could essentially just let me plug the link straight in as a single argument
BEAUTY AT LAST!
except these magic commands are coded to only accept raw text, and not variables, since they're parsed before compile.
womp womp
back to mathplotlib again i guess..
also for some reason your mathplotlib images get displayed with these ugly horizontal whitespace bars on each side of the image.
i tested the python code locally and couldn't replicate this problem, so i'm guessing it's how you handle it on your side.

Hi, the magic commands are resolved before running the code, otherwise we would have to implement each magic command separately for each programming language (like we already do for the plot command). That would be annoying. I suggests you simply use python to grab the image from the url and save it to a fixed path, then you can use the magic command to display the image:
img_data = requests.get(image_url).content
with open('path/to/image.jpg', 'wb') as handler:
    handler.write(img_data)

@show("path/to/image.jpg")

Concerning the white bars with matplotlib: I never saw this in my own notes. Maybe its because of your version of matplotlib or python, or because of a theme.


MAUI support?

Is there a plan for this project to support MAUI, including the Windows platform?

I would also be interested in this

me 2


Library collision in v1.7 plugin.

The default metatdata for shared libraries in the plugin folder causes a collision when building for Android. This is because the metadata for all shared libraries in the x86 and x86_64 directories are marked for use on all platforms. They should probably only use the Editor and Standalone options.
Repro steps:

Start a clean project.
Import v1.7 OSVR plugin from the Asset Store.
Go to File -> Build Settings and swap target platform to Android.
Select any scene to add to the build.
Open player settings and change the Bundle ID of the application so that it will build.
Build and see the following compiler error.

Plugins colliding with each other.

Is this something that we'll have to ditch Unity 4 support (or even early 5.x) to fix?
(What was wrong with their tagged directories for the plugin platforms? Those worked perfectly fine... arg...)

This is an easy fix, just checking those boxes for Windows and Android libs. And yes, upgrading to Unity 5 was a prereq, plugins did not have those checkboxes in Unity 4. This replaced the directory structure method of determining the intended platform of plugins, although I think it still uses directory names when upgrading to make a best guess about the bitness.

Yeah library metadata is new. This shouldn't force a move away from 4, and can probably be addressed with an editor script that post processes package imports.
Don't know why they changed it, unless it's to improve scalability with new platforms.

We've already upgraded to 5, so I'll fix this today.

Does this mean that Unity 4 is no longer supported? If so, the plugin description on the Asset store should also be changed (this may need a separate ticket).
Snippet from the plugin description:

OSVR Plugin for Unity is compatible with Unity 4 and 5 Free and Pro versions (image distortion is disabled in Unity 4 Free version).


fix: Can strike-through pre code block.

Fixed Issues
$ https://github.com/Expensify/App/issues/24281
Tests

Open any report
Paste "~Code~"
Observe the code block text doesn't gets strike through line and code block creates line break before start and after end

QA

Open any report
Paste "~Code~"
Observe the code block text doesn't gets strike through line and code block creates line break before start and after end

Demo
https://github.com/Expensify/expensify-common/assets/85894871/a3eeae28-204b-4a1a-9c36-034113c4b2d6

@Krishna2323 I edited the "Fixed issues" link format for you which is important to make sure we get the proper person assigned. I'll assign @eVoloshchak from the issue for the C+ review and then Andrew will review after C+ approves.

@eVoloshchak I can't assign you but would you please review?

Looks good!
@Krishna2323, when submitting the second PR make sure that step 2 for Tests includes the correct message, it should be

Paste ~```Test```~


HADOOP-19224 (3.4.1)Upgrade esdk to the latest version 3.24.3

Description of PR
How was this patch tested?
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 44s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ branch-3.4.1 Compile Tests _

+1 :green_heart:
mvninstall
48m 28s

branch-3.4.1 passed

+1 :green_heart:
compile
0m 27s

branch-3.4.1 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
0m 25s

branch-3.4.1 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
mvnsite
0m 30s

branch-3.4.1 passed

+1 :green_heart:
javadoc
0m 29s

branch-3.4.1 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
0m 25s

branch-3.4.1 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
shadedclient
88m 42s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m 25s
/patch-mvninstall-hadoop-cloud-storage-project_hadoop-huaweicloud.txt
hadoop-huaweicloud in the patch failed.

+1 :green_heart:
compile
0m 19s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
0m 19s

the patch passed

+1 :green_heart:
compile
0m 17s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
0m 17s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 20s

the patch passed

+1 :green_heart:
javadoc
0m 18s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
0m 18s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
shadedclient
39m 34s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 21s

hadoop-huaweicloud in the patch passed.

+1 :green_heart:
asflicense
0m 34s

The patch does not generate ASF License warnings.

132m 40s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7004/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7004

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint

uname
Linux 8e3fe733d40b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.4.1 / c1962185d38b132614a3484e904847ee0cc88afa

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7004/1/testReport/

Max. process+thread count
597 (vs. ulimit of 5500)

modules
C: hadoop-cloud-storage-project/hadoop-huaweicloud U: hadoop-cloud-storage-project/hadoop-huaweicloud

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7004/1/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

3.4.1 is already released, no change can be merged over there, get it committed to trunk & if you want it in 3.x, raise a PR to branch-3 post getting it committed to trunk, & the change will be part of next 3.3.x release

@ayushtkn I can not see ? 3.4.1` anywhere? Where is this information?
https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-registry

I think the vote got vetoed
https://lists.apache.org/thread/hmhhxllq49q1y9z4hxn5oj22qb662gjz
But I don’t think we are accepting anything which ain’t blocker for 3.4.1, all goes to 3.4.2


Chore/340 validated changes to mobile layouts

This PR implements (some of) the changes made to try and improve the experience for job seekers on mobile:

jobseekers' index page: adding the ' n results' and 'refine search' anchor link UI at the top of the results page, and moving the filters to the very bottom
job show page: apply listing layout changes as previously deployed to testing, except place the map below the about school info (rather than above)

Browser detection is performed using the Browser gem. Different views are served to mobile devices using Rails' ActionPack Variants.
before - list view

after - list view

before - job view

after - job view

 Validated changes to mobile layouts


mac环境下

报错。。。。

did you install kaldi successfully or check prerequisite configuration such as path.sh before running this script? In the log, run.pl job failed while making mfcc, so please look into tmp/work_1/log_1/make_mfcc_trans_1.1.log then you might be able to find an error. I doubt that path.sh or other configuration might raise this error. Let me know if you found other errors.

I git pulled it again and tested in my server but it ran successfully.

install    extras/install_openblas.sh      err。

Looks like kaldi wasn't installed properly and it raised the issue. If you follow error logs then you could install the kaldi at the end. It might not be too difficult to install it as long as you are familiar with bash and linux.


Fixes #199 Adds startsWith(prefix) and tests to the InterledgerAddress interface

Add startsWith() to accept InterledgerAddressPrefix
Add corresponding tests for InterledgerAddress that startsWith(prefix)

Signed-off-by: sudheesh001<EMAIL_ADDRESS>

Codecov Report

Merging #211 into master will decrease coverage by 0.02%.
The diff coverage is 50%.

@@             Coverage Diff              @@
##             master     #211      +/-   ##
============================================
- Coverage     74.18%   74.15%   -0.03%
- Complexity      390      392       +2
============================================
  Files           112      112
  Lines          1743     1745       +2
  Branches        155      156       +1
============================================
+ Hits           1293     1294       +1
  Misses          381      381
- Partials         69       70       +1

Impacted Files
Coverage Δ
Complexity Δ

.../java/org/interledger/core/InterledgerAddress.java
96.51% <50%> (-1.11%)
9 <2> (+2)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77a05bb...6b8d312. Read the comment docs.

One final minor issue in the test, and then I think this is ready to merge. Nice work!


Fixed error in permission cache clearing

Purpose
There is an error in permission cache clearing when tenant ID is -1. To resolve this issue, we ensure that the tenant ID is not -1 before removing the permission cache.
Goals

Describe the solutions that this feature/fix will introduce to resolve the problems described above

Approach

Describe how you are implementing the solutions. Include an animated GIF or screenshot if the change affects the UI (email<EMAIL_ADDRESS>to review all UI text). Include a link to a Markdown file or Google doc if the feature write-up is too long to paste here.

User stories

Summary of user stories addressed by this change>

Release note

Brief description of the new feature or bug fix as it will appear in the release notes

Documentation

Link(s) to product documentation that addresses the changes of this PR. If no doc impact, enter “N/A” plus brief explanation of why there’s no doc impact

Training

Link to the PR for changes to the training content in https://github.com/wso2/WSO2-Training, if applicable

Certification

Type “Sent” when you have provided new/updated certification questions, plus four answers for each question (correct answer highlighted in bold), based on this change. Certification questions/answers should be sent to<EMAIL_ADDRESS>and NOT pasted in this PR. If there is no impact on certification exams, type “N/A” and explain why.

Marketing

Link to drafts of marketing content that will describe and promote this feature, including product page changes, technical articles, blog posts, videos, etc., if applicable

Automation tests

Unit tests

Code coverage information

Integration tests

Details about the test cases and coverage

Security checks

Followed secure coding standards in http://wso2.com/technical-reports/wso2-secure-engineering-guidelines? yes/no
Ran FindSecurityBugs plugin and verified report? yes/no
Confirmed that this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets? yes/no

Samples

Provide high-level details about the samples related to this feature

Related PRs

List any other related PRs

Migrations (if applicable)

Describe migration steps and platforms on which migration has been tested

Test environment

List all JDK versions, operating systems, databases, and browser/versions on which this feature/fix was tested

Learning

Describe the research phase and any blog posts, patterns, libraries, or add-ons you used to solve the problem.

Let's analyse why this method is been called with tenant id is -1. We need to find the initial place where this is initlized and fix there as well.

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2953524231

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2953524231
Status: success


add throwing star

About the PR
throwing star that embeds and does 18 damage total
not added to uplink or anything, for future use with ninja
Media
https://github.com/space-wizards/space-station-14/assets/39013340/a1020b04-8b39-4152-a95c-974b75130ecb

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog
no cl no fun

@metalgearsloth

pro

real


[Feature Request] Add interval option for Update DM

Is your feature request related to a problem? Please describe.
Whenever the bot DMs me about an update I often think it's someone asking for help rather than a bot sending a DM (As I only have DMs open for close friends and servers I act as support in). As I can't actually update it during the time I first get them, it's fairly inconvenient to get this same thing every day until the day I can actually update it as it takes my attention away from what I'm doing. I do however need to know when an update is out, so disabling the updatealerts option is not a solution, as I otherwise wouldn't know an update is out for possibly months.
What is your ideal solution to the problem?
Add an second option that allows you to specify an interval. I'd recommend hours or days. But at least something that allows us to specify how often we'll be reminded.
I did also see #983 where you mention valid points why not to add it. If you're still on par with that despite my own points, do prefer to add an option of when during the day the bot should DM. Currently I get the DMs at around 1pm. If I could instead choose something like 8pm that would also be acceptable.
How would this feature be used?
Let's say the option is called updatealerts.interval, here's a few examples of what different values would do, assuming hours:

updatealerts.interval=24 - Sends a every day (Current behaviour).
updatealerts.interval=48 - Sends a DM every 2 days.
updatealerts.interval=168 - Sends a DM every week.
updatealerts.interval=0 - Sends a DM only once.

As for the relation to #983 you'd possibly just have an option updatealerts.time which would be a 4 digit number specifying the time. 2000 would DM you at 8pm and 730 would DM you at 7:30am. This way you wouldn't need to change the 24 hour interval but could still allow us to get the DM when we're easily available.
Additional Info
I'm also very sensitive to notifications in general due to mental issues, mostly ones that aren't expected. If someone DMs me about needing help, that's fine. If my bot DM me about an update once, that's fine too. But once I get either spam (which I've disabled DMs from strangers for a reason) or if my bot starts DMing me on an interval when I need not to, it does stress me out heavily. So options like this really help me in the long run as it's basic accessibility settings at that point.
Checklist

[X] I have checked the documentation to make sure this feature doesn't already exist
[X] I have searched for similar feature requests
[X] I have confirmed that my requests is not on the list of things that will not be added
[X] I am running the latest version of the bot:

I second this suggestion, I'd love for the update alert dm to only be sent once after the update is released as sending it repeatedly is annoying and unnecessary imo.


[oneDNN]: Adding support for sigmoid activation as a fused op in quantized-conv2D

This PR adds support for sigmoid activation as a fused op in quantized-conv2D.

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @penpornk Can you please review this PR ? Thank you!

Hi @bhavani-subramanian Can you please confirm if this PR is still valid? Thank you!


Add missing required "Client-Id" header to Twitch Provider docs

The https://api.twitch.tv/helix/users endpoint requires you to pass your Twitch OAuth Application's "Client-Id as a header.
Not supplying the "Client-Id" header will result in this error:
{ error: 'Unauthorized', status: 401, message: 'Client ID is missing' }

This PR adds this header to the documentation

Thanks!


Add support for multiple filetypes in filetype aliasing

Now we can set a list of filetypes when aliasing new ones.
This is my first work on a plugin for vim, and my first contribution for a project on github. So, please, go easy on me ;)
All tests are passing so I think I didn't break anything. But the way that ale#linter#GetAll is handling failures during the load of linter files is kind of bothering me. I kept the same approach as before, if a failure occurs, an empty list of linters is returned. But, now that function is loading linters for multiple filetypes so returning nothing while at least one linter could be functional may be a little drastic, or may be not. What do you think about that?

Done. I went ahead and squased my commits. Hope that's ok.

Cheers! :beers:


Train my own dataset

Hi. I use this repro to train my own dataset. During the training process, the PSNR value of my test set was decreasing. What is the reason for this？
my low resolution dataset (down x2) is to scale the high-resolution dataset to half its original size. That is to say, my low resolution data sets are collected from high-resolution data sets. Thanks.

Hard to say, how many data points are you using for training and test? Is PSNR increasing for training set? What config are you using?

I had similar issues when trying to train a quite large set of product shots with a lot of white space around them.
Since the discriminator has a fixed flatness set, that used patches randomly containing no detail throughout the training process and when the flatness schedule of the model increased comparisons started to differ.
To remedy that problem I set the flatness increase parameter to 0.
flatness = {'min': 0.00, 'max': 0.15, 'increase': 0.0, 'increase_frequency': 5 }

I also think I encountered an issue with the adam optimizer used in the trainer.
There is a comment inside trainer.py linking to this article https://stackoverflow.com/questions/42327543/adam-optimizer-goes-haywire-after-200k-batches-training-loss-grows
I set the epsilon value to 0.1 like below.
optimizer = {'beta1': 0.9, 'beta2': 0.999, 'epsilon': 0.1 }

Training went fine after that. Hope it helps someone.


make-bootstrap-tools: fix test to include libstdc++ -rpath

After https://github.com/NixOS/nixpkgs/pull/209054 we started moving libstdc++.so out of default glibc's paths. This exposed bootstrap tools build failure as:
$ nix build --no-link -f ./pkgs/stdenv/linux/make-bootstrap-tools.nix
...
>
.../bin/bar: error while loading shared libraries: libstdc++.so.6:
   cannot open shared object file: No such file or directory

Note that bootstrap itself did not break. The change only expands handcrafted -rpath entries.
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Thank you for deduplicating those two branches of the code while you were doing this!


Blog: Blog post for scaling OpenTelemetry Collectors using Ansible

Preview: https://deploy-preview-4140--opentelemetry.netlify.app/blog/2024/scaling-opentelemetry-collectors/

Hey @svrnm The REFCACHE workflow tell me to run npm run test-and-fix but that doesnt seem to do anything for me

do not worry too much about that, we fix the checks at the end before merging the PR, there are some nuances with links that sometimes break the pipeline, and while updating the content you may break it again and again

also for the canonical_url This can be added later right as I would be publishing a similar blog (for Grafana Cloud usage as OSS is covered here) but on Grafana's Blog

So, here is my semi-professional SEO opinion: the content that comes first is the "canonical url", so the grafana blog should point to this blog as canoncial. If the grafana blog has a policy against that, we need to discuss that.

So, here is my semi-professional SEO opinion: the content that comes first is the "canonical url", so the grafana blog should point to this blog as canoncial. If the grafana blog has a policy against that, we need to discuss that.

Ahh, I don't think that is possible, Ill get that if possible too but would be nice to have it from here aswell, That is possible to update later or needs to be in this PR?

Do you think that it is not possible or do you know that it is not possible? Can you please check with the Grafana Blog Team first and then follow up with us?
Our current policy does not cover this use case, so we would need look into this (cc @open-telemetry/docs-approvers).
Personally I still think it doesn't feel right to update an existing blog post to point to a blog post that came later and by that downgrading it from a SEO perspective: whoever takes the blog post first has the major work of copyediting and reviewing.


ERROR: No video formats found; with Pluralsight

Checklist

[ ] I'm reporting a broken site support issue
[X] I've verified that I'm running youtube-dl version 2019.11.28
[X] I've checked that all provided URLs are alive and playable in a browser
[ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar bug reports including closed ones
[X] I've read bugs section in FAQ

Verbose log

PASTE VERBOSE LOG HERE
```[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--username', 'PRIVATE', '--password', 'PRIVATE', '--cookies', 'C:\\Users\\Alessio\\Desktop\\YouTubeDL\\cookies.txt', '--min-sleep-interval', '45', '--max-sleep-interval', '60', 'https://app.pluralsight.com/library/courses/object-oriented-programming-fundamentals-csharp/table-of-contents', '--playlist-start', '1', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2019.11.28
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17763
[debug] exe versions: none
[debug] Proxy map: {}
[pluralsight:course] object-oriented-programming-fundamentals-csharp: Downloading JSON metadata
[download] Downloading playlist: Object-Oriented Programming Fundamentals in C#
[pluralsight:course] playlist Object-Oriented Programming Fundamentals in C#: Collected 82 video ids (downloading 82 of them)
[download] Downloading video 1 of 82
[pluralsight] Downloading login page
[pluralsight] Logging in
[pluralsight] 964bfbf4-020f-4027-b383-14f21ab8a51e-0: Downloading JSON metadata
[pluralsight] 964bfbf4-020f-4027-b383-14f21ab8a51e-0: Downloading mp4-high-widescreen viewclip graphql
[pluralsight] 964bfbf4-020f-4027-b383-14f21ab8a51e-0: Waiting for 5 seconds to avoid throttling
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'https://vid5.pluralsight.com/expiretime=1576675325/82ee8cdf1f797a0da3b6caa9dab9da4a/clips/resolution/66d6f216-2fe1-4e8e-928f-d2957c030a5c/current/mp4/1280x720.mp4'
[download] Sleeping 46.75 seconds...
[download] Destination: Course Overview-66d6f216-2fe1-4e8e-928f-d2957c030a5c.mp4
[download] 100% of 2.73MiB in 00:02
[download] Downloading video 2 of 82
[pluralsight] oop-fundamentals-csharp-m1-introduction-0: Downloading JSON metadata
[pluralsight] oop-fundamentals-csharp-m1-introduction-0: Downloading mp4-high-widescreen viewclip graphql
[pluralsight] oop-fundamentals-csharp-m1-introduction-0: Waiting for 5 seconds to avoid throttling
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

## Description

<!--
Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

WRITE DESCRIPTION HERE
Hi! I'm trying to download a course from Pluralsight with this command line

youtube-dl.exe --username myAccount --password myPass --cookies C:\Users\Alessio\Desktop\YouTubeDL\cookies.txt --min-sleep-interval 45 --max-sleep-interval 60 https://app.pluralsight.com/library/courses/object-oriented-programming-fundamentals-csharp/table-of-contents --playlist-start 1 --verbose

but I'm getting this error

ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I can only download the first video but not the other 82

Yesterday I was able to Download only 1 full course. Today I'm getting that problem.

Can someone please help me?

I've solved the problemy by myself.

how did you solve it please ??


about dataset

In the code, I see that in the evaluation of the recognition results,use these two json files:
../../output/erfnet_baseline_tusimple.json & /home/data/Tusimple/test_label.json
I would like to ask you if I want to evaluate my dataset, do I also need to generate json files in this format. This file contains lanes, H_ samples,raw_ file.  I've been looking for many websites these two days, but I haven't found a way to generate this JSON file.
If I want to use your method to train and evaluate my data set, is there any other method？
thankYou !!

There are mainly 2 formats for lane evaluation on the market. One is the TuSimple format as you just described, the other would be the CULane format. with a directory of txt files, maybe you can try if that format is easier? CULane metric does not constrain a fixed set of H_samples.

You mean that it is recommended to label my dataset in the format of culane instead of tusimple. It will be simpler. Is my understanding correct?thank you!

Well, it depends on what you have. Do you already have labels? Mainly, if they are labeled with fixed set of heights, TuSimple is better, if they are labeled with free points, CULane is better.

The CULane metric and annotation format is also better accepted by the community as well. You can see the new LLAMAS benchmark adopting this standard.

okay，i got it . i will try it.thankyou very much!

It seems the problem is resolved, I'll close for now. Feel free to reopen!