Vim9: cannot use ":legacy" modifier in front of ":return" command

Describe the bug
In Vim9 script, we cannot use the :legacy modifier in front of the :return command.
To Reproduce
Run this shell command:
vim -Nu NONE -S <(cat <<'EOF'
    vim9script
    def Func(): number
        legacy return 123
    enddef
    defcompile
EOF
)

E1027 is raised:
E1027: Missing return statement

Expected behavior
No error is raised, because there is nothing at :h :legacy which specifies a limitation with regards to the :return command.
Environment

Vim version: 8.2 Included patches: 1-2854
OS: Ubuntu 20.04.2 LTS
Terminal: XTerm(353)

Additional context
N/A

Slightly less contrived example:
vim9script
def Func(): number
    legacy return range(3)->map('v:val + 1')
enddef
echo Func()

Expected:
[1, 2, 3]

Actual:
E1027: Missing return statement

Slightly less contrived example:

Ah no, it's still contrived, because eval strings do work in Vim9; no need of :legacy.

OK, this example is a little better:
vim9script
def Func(): number
    legacy return {-> 123}()
enddef
echo Func()

Actual:
E1027: Missing return statement

Expected:
123

Now that I think about it, I don't think it can be made to work.  For example, it would require Vim to parse the legacy lambda to check it returns a number.  It's not that important anyway.  Closing then.

Although, maybe this limitation needs to be documented:
diff --git a/runtime/doc/vim9.txt b/runtime/doc/vim9.txt
index a368af0a5..290840d1e 100644
--- a/runtime/doc/vim9.txt
+++ b/runtime/doc/vim9.txt
@@ -71,6 +71,7 @@ rewrite old scripts, they keep working as before.  You may want to use a few
 		useful in a Vim9 script or a :def function.
 		Note that {cmd} cannot use local variables, since it is parsed
 		with legacy expression syntax.
+                Also, {cmd} cannot be :return.

 ==============================================================================

Sorry, wrong indentation:
diff --git a/runtime/doc/vim9.txt b/runtime/doc/vim9.txt
index a368af0a5..70cc831b4 100644
--- a/runtime/doc/vim9.txt
+++ b/runtime/doc/vim9.txt
@@ -71,6 +71,7 @@ rewrite old scripts, they keep working as before.  You may want to use a few
 		useful in a Vim9 script or a :def function.
 		Note that {cmd} cannot use local variables, since it is parsed
 		with legacy expression syntax.
+		Also, {cmd} cannot be :return.

 ==============================================================================


Add Jetbrains sponsor

Thanks to Jetbrains provide me with a free os license for all products
@ReeceM please provide me with your email so I can send your license
Thanks for your contributions!

Hi there,
that's awesome that they doing that 👍 :)
email to follow:

email address is<EMAIL_ADDRESS>Will delete it after getting the mail 👍

Will delete it after getting the mail 👍

Invitation sent!

@Qoraiche 💯


Version 2.4 replication query problems on PG 10

The replication monitoring scripts for master and slave still used the old functions:
pg_whatever_xlog_whatever
instead of
pg_whatever_wal_whatever
Also, change all occurrences of location to lsn in those functions.
But that is pretty neat how I can go in there and modify the scripting to get past any errors.

Hi @MichaelDBA ,
Thanks for your feedback. We will create separate versions of the monitors. Just for the record, all default monitors are open source in this repository [1].
Also, for more information about the Monitoring Dashboard, please take a look at the documentation [2]. Any further feedback is very welcome :)
[1] https://github.com/OmniDB/monitors
[2] https://omnidb.org/en/documentation-en/14-monitoring-dashboard-en

Great!  Thanks!


[Test] Build hack and slash, based on google branch

DO NOT MERGE

#3874 landed and it looks like the changes you have here all made it back into that.


Unable to handle wildcard support

Consider the following example:
►facad *.ttf
Error: Unexpected argument 'NotoMono-Regular.ttf'

Here I ask to list all .ttf files. Since the wildcard expansion is a part of the shell itself, facad sees expanded list of files. And fails, because it expects only one argument.
The expected correct behavior is to have the same as in ls:
►ls *.ttf
NotoMono-Regular.ttf  NotoSans-BoldItalic.ttf  NotoSans-Bold.ttf  NotoSans-Italic.ttf  NotoSansMono-Regular.ttf  NotoSans-Regular.ttf

Thank you, @dmikushin  ! Wildcard support added

As of 62126dece16d3a5bde91b563f8f567a20e335f7b , there is an issue somewhere.
With '*' all files are correctly displayed:
~/f/f/build►./facad *                                                                                (main|?↑1) 19:41
💾 facad                      🏝 compile_commands.json         🏝 intro-tests.json       🧩 src_emoji_mappings.c.o
🔬 sanitycheckc.c             🏝 intro-benchmarks.json         🏝 meson-info.json        🧩 src_emoji_utils.c.o
❓ build.dat                  🏝 intro-buildoptions.json       🔒 meson.lock             🧩 src_file_card.c.o
❓ coredata.dat               🏝 intro-buildsystem_files.json  🥷 build.ninja            🧩 src_longlisting.c.o
❓ install.dat                🏝 intro-dependencies.json       🧩 src_args_parser.c.o    🧩 src_main.c.o
❓ meson_benchmark_setup.dat  🏝 intro-install_plan.json       🧩 src_dev_dir_utils.c.o  ❓ coredata.dat.prev
❓ meson_test_setup.dat       🏝 intro-installed.json          🧩 src_dir_analytics.c.o  📝 cmd_line.txt
💾 sanitycheckc.exe           🏝 intro-projectinfo.json        🧩 src_dir_config.c.o     📝 install-log.txt
🧢 config.h                   🏝 intro-targets.json            🧩 src_display_utils.c.o  📝 meson-log.txt

With *.json only one JSON file is listed, however there are many of them above:
~/f/f/build►./facad *.json                                                                                (main|?↑1) 19:41
🏝 compile_commands.json

But I'm not sure if it is related to wildcard, because -l * also misses a lot of files:
~/f/f/build►./facad -l *                                                                                (main|?↑1) 19:40
  172.0K  4m ago  👑: marcusmae 📜: 👀✏🚀  📁 facad.p          (0)
  119.2K  4m ago  👑: marcusmae 📜: 👀✏🚀  📁 meson-private    (0)
   11.2K  4m ago  👑: marcusmae 📜: 👀✏🚀  📁 meson-info       (0)
    3.0K  2d ago  👑: marcusmae 📜: 👀✏🚀  📁 meson-logs       (0)
  108.0K  4m ago  👑: marcusmae 📜: 👀✏🚀  💾 facad
    5.6K  4m ago  👑: marcusmae 📜: 👀✏❌  🥷 build.ninja
    4.0K  4m ago  👑: marcusmae 📜: 👀✏❌  🏝 compile_commands.json
  135.0B  4m ago  👑: marcusmae 📜: 👀✏❌  🧢 config.h

What do you think?

So negative: I cannot reproduce your output of multiple *go files, I always get only one.

@dmikushin , at least for fast checking can't reproduce....


Fix compilation of Recursive GADT schema derivation

/claim #547

@zio/zio-schema @jdegoes gentle reminder that this needs review, thanks!


Added command to restore an archived Microsoft Teams team

Added the command to restore an archived Microsoft Teams team. This restores users' ability to send messages and edit the team, abiding by tenant and team settings.
command: graph teams unarchive --teamId 6f6fd3f7-9ba5-4488-bbe6-a789004d0d55
teamId: The ID of the Microsoft Teams team to unarchive

Coverage remained the same at 100.0% when pulling 8789b0b752d5e04d297e5aef6bfe2bf63f606ac5 on sprider:teams-unarchive into 81f64dd32e812d5e380cd9bbfdc0211dcdbb73c5 on pnp:dev.

Thanks @sprider, we'll have a look shortly 👏

Hey guys, there are merge conflicts. @sprider, could you please pull and rebase the latest dev branch so the conflicts can be resolved.

@VelinGeorgiev,
Do you want me to run the following command on the dev branch?
git pull --rebase
git push

@sprider , that might work and resolve the conflicts. As you can see there are currently conflicts with the dev branch.

If you indeed rebase from upstream dev  and push to your branch might solve them.

@VelinGeorgiev, It says "Current branch teams-unarchive is up to date."

@VelinGeorgiev this isn't an issue per se. You see it typically the moment a commit was added to the upstream dev that isn't a part of the PR. 9/10 times, git can handle the issue just fine, so let's check it locally first.

Sounds like a plan, let's do that. :)

Merged manually. Thank you @sprider! Welcome to the crew 💪

Thank you @waldekmastykarz & @VelinGeorgiev


Add icon for preview build flavor

@FabioTacke I am not sure if this is the correct way to add the icon. Also the actual "preview" banner at the bottom is cut if when the icon is circular :/

Yes, could you please move the standard icons into the device subdirectories? This avoids confusion because there is no "default" case here.
Regarding the layout issue with the banner: Could you please check in with the design team on that matter?

Updated, now looks like this:


compile **julia **  failed

compile and run julia will result in segment fault.  _struct _ event structure is packed in the kernel. But as wgtcc do not support this extension, it compiles struct event into 16 bytes(while gcc will packed it to 12 bytes). It causes segment fault we trying to get fd when accepting request.

This is fixed by allowing wgtcc reduce alignment through the keyword alignas. Julia now use a event struct that has the same field layout as the struct epoll_event in the kernel. Fixing this, wgtcc compiles julia, and julia runs very well :)
I Tried to support the GNU exntensions( attribute((packed))), but being compatible with gcc, means a complete supporting of GNU extensions. There is much work to do. So, i made this workaround to making
wgtcc works for julia :)


Add glib (@ 2.82.2)

Limitations:

//glib:gregex.c is unavailable because it depends on pcre2 JIT, which isn't currently available in the BCR pcre2 module.
Builds for //gio, //gobject, //gthread, and //girepository are not implemented.
Windows builds are not implemented.

With //glib and //gmodule working, this provides enough of GLib to build pkg-config and QEMU.

Hello @bazelbuild/bcr-maintainers, modules without existing maintainers (glib) have been updated in this PR. Please review the changes.


Add ntsd.me

@ntsd Thanks, and welcome to the webring!

@ntsd Thanks, and welcome to the webring!


InBios Intl - Message Validation

Hello,
User states:
Hello,
We are beginning to conduct formal validations for the reporting feature on our app and wanted to validate the transmission going from our App to ReportStream is validated.
We’ve submitted 3 entries and needed help to validate these entries or conversely share a csv file for the three IDs so we can manually verify the information ReportStream has received is correct.
Test ID

25ffa2ff-de8b-43a1-b4cb-6d2fa0018cf7
8eb1d891-ab06-4549-b8a0-2397ee3eac1f
* 8a9d00df-4952-498d-b14a-d954daaf9862

User is based out of WA and their organization is InBios International Inc.
User's name is Vik Bohra.

@BerniXiongA6 @brandonnava
Did we schedule a Kickoff meeting for this sender? Should I request some times?

Hi @sliu1000 I can reach out to them since we’ve changed up our sender kickoff process slightly. Thanks for the reminder! cc: @brandonnava

Email from James:
Hi Sharon,
Yes, we already had a kickoff meeting some time ago with Anshul Kumar and Joel Biskie; we did some initial data verification with them as well and that was successful.  They subsequently provided the key and server information for the production server.
However, for this product, we had to go through clinical trials and this delayed implementation of the actual reporting feature.  As part of the formal software validation process, we need to verify that this “final product” data that we are sending to your servers is “matching” to the expected information.  (Not just receiving the successful ‘ping’ from the server.) Please let me know if there’s anything else we need to clarify??  Is it possible for Vik to get the information / data he requested as part of the validation process??
Thanks!
-James

@BerniXiongA6 Sounds like they need a kickoff? Or would James or Brick emailing them be enough?

Hey @sliu1000 @brandonnava I can send them new sender kickoff materials and then I'll propose times to schedule in Decembrer with them. Will udpate this ticket when I send out the email. Thanks!

BX sent email to sender on 11/25:

Hello Vik,
Thanks for your reply. Since we've updated our Programmer's Guide within the last few months, please review the guide to make sure your test file follows the list of field requirements for the COVID-19 schema. If your test files were created with the programmer's guide prior to Version August 2022, please generate a new test file and reply to all here with a new test file.
Our current backlog is about 3 to 4 weeks; therefore, the soonest we'll be able to accept another data sender into our queue is in mid-December. Please stand by and we'll let you know when your ticket has been assigned to an onboarding engineer. In the mean time, you can email your new test file (if needed) anytime before then.
Thank you so much.
Best,
Berni

Email received from sender on 11/23/2022:

From: Vik Bohra<EMAIL_ADDRESS>Sent: Wednesday, November 23, 2022 4:50 PM
Hello Berni,
Very nice to meet you! Your right, our team has engaged previously with Report Stream team (Anshul Kumar and Joel Biskie); where we did initial data verification with them, which was successful. They subsequently provided the key and server information for the production server.
However, for this product, we had to go through clinical trials and this delayed implementation of the actual reporting feature.  As part of the formal software validation process, we need to verify that this “final product” data that we are sending to your servers is “matching” to the expected information.  (Not just receiving the successful ‘201’ status from the server.)
At this point, we don’t require a kick-off meeting. We are just looking for you to provide us the information / data where we can validate what Report Stream received is indeed what we sent across as part of our validation process. The first attachment includes csv files of three patients with non-PII data which we sent on November 1st. Could you confirm what you received for these entries on your end and share the respective csv files so we can compare and confirm they are matching the values. Please let me know what you think.
Looking forward to working with you. Thanks!
Regards,
Vik

Email sent from BX to sender on 11/22/2022:

Hi Vik:
Thank you for your interest in getting started with ReportStream. It appears you may have worked with a previous team member a while back. I'm the Delivery Manager for our Onboarding Team at ReportStream, and I'll be happy to assist you at this time.
Since it has been a bit, some of our processes have changed. The next step is to get you acclimated to how ReportStream works and our current onboarding process for data senders. I'm sharing some useful onboarding resources in the following that you can preview ahead of scheduling a kick-off session with our technical team:

Watch this 5-minute video to see the kick-off presentation on ReportStream for data senders (click to download MP4)

Review the accompanying transcript, which includes mentioned resource links (click to download PDF)

Preview the Programmer’s Guide (which is also mentioned in the transcript above) ahead of scheduling a Sender Kick-Off session with our technical team.

Schedule your Sender Kick-Off session with our technical team by replying to this email. We’ll offer a few time slots in which our technical team member/s will be available to meet with you by video call for 30 minutes to answer specific questions you have about technical requirements, etc.

Please note that onboarding officially begins when ReportStream starts the “test file validation” process with your team (which is mentioned in the programmer’s guide above). With our current backlog, the soonest we’re able to onboard new data senders is mid-late December timeframe.
We look forward to assisting you.
Thank you.
Berni

Any engineer can assign themselves to validate this sender's files once the Sender has confirmed their test files are using the most current API fields list. cc: @brandonnava @brick-green @sliu1000

Hello,
User has come back with an updated inquiry.
"Since we are conducting csv implementation, do we need to include every field in our implementation which is marked as “(OTC)” in the csv column name?"

Additional Questions from Sender:
Hi James –
Thank you for the prompt response!
Here are a couple of additional questions:
As you indicated below, the guide does require a few updates so clarify any confusion. So, for the fields which are not required (i.e., patient_phone_number, patient_city, patient_street, etc.) but cannot have a blank value – is there a recommended value we can set them to? For ex. “Unknown / Not Provided”
And for all “order_provider_*” fields à we will just leave these fields as blank per your guidance below.
Can you confirm the recommended values would be okay for the following fields in the table?

“test_result_status” field indicates it to be a required field in the programmer’s guide but on the website: https://reportstream.cdc.gov/resources/csv-schema#ordering-facility it shows it to be an optional field. Could you help clarify which it is?
We’d be happy to partner with you on the self-validation service. I will keep you posted on our progress on getting you a sample test file (likely by end of next week) and similarly will continue to reach out for any other follow-up questions I may have. Your responses below helped tremendously. Thank you again
Regards,

Update from Sender:
Hello James,
Hope all is well.
[https://app.zenhub.com/files/304423150/51cf0b41-5ce2-436d-9ffc-c98cb2c39c90/download]
Apologies on the delay as it took a bit longer than we anticipated. We had to complete our implementation and validate the changes to ensure our csv format and data was all accurate before we were ready to share it for review. I’m happy to say that we appear to be at a good point based on all the feedback you provided below and would like to share our first patient csv file with dummy/simulated data, please see attached. Please review and let us know what changes or modifications we need to make to ensure we can send these results to ReportStream moving forward as OTC.
Also, happy to engage this week if you are available to discuss the self-validation service, and if not, we can touch base at the beginning of the year as well. Please let me know what best works.
Look forward to your updates on this.
Thanks & Happy Holidays!
Regards,
Vik

Hi James,
Great, thank you for the feedback here. We will get these addressed and share an updated csv file very soon.
Happy Holidays and Happy New Year!
Regards,

Provided feedback and asked for 2nd round of modifications

@GilmoreA6
Vik from Inbios has sent another sample for your review.  Please let me know if you'd like me to forward the email to you. However, I believe you should have been cc'd.
Best regards,
Sharon

Update from Sender:
@GilmoreA6
Hi ReportStream Team, James,
Following up here to see if there was any additional feedback you or the team had from the preliminary/draft report we shared. Please let me know. Thank you!
Regards,
Vik

Hi Vik,
Can you send another test file to staging (please make sure it is fake data)? I had to make an adjustment on our side to your schema in staging. Other than that the file looks good.
Thanks,
James


Support for Godot 4.0

I'm building a game for a tutorial series I am developing with Godot 4.0 and I am interested in adding godot mod loader to it. However, I found that it was not available for Godot 4.0. Is anyone planning on porting it?
Thanks,
Joshpinto6

there are plans for it already, but since no game release on steam that is currently interested in modding uses godot 4, it isn't a priority

I've created a branch for it, you can follow development:
https://github.com/GodotModding/godot-mod-loader/tree/4.x

I've created a branch for it, you can follow development: https://github.com/GodotModding/godot-mod-loader/tree/4.x

Thank you!

Looking forward the port!

base version is in, now it just needs extensive testing to see if everything is working as expected

I'm also interested in using this great plugin for my game in G4

I will also use this for a game I am developing with Godot 4.

So far the only Godot plugin that supports Godot 4 that i could find (outside of the branch that was created: https://github.com/GodotModding/godot-mod-loader/tree/4.x) is this:
https://github.com/KoBeWi/Godot-Universal-Mod-Manager

Update about the 4.x Loader

The 4.x branch is now on par with the main branch (v6.2.0).
We currently have one massive blocker, there is a bug that prevents extending a named class after it has been taken over with take_over_path.
We reported this bug and opened a PR with a possible fix:

https://github.com/godotengine/godot/issues/83542
https://github.com/godotengine/godot/pull/84148

There is hope that it will be fixed with version 4.3.

What does this mean for the current 4.x branch?

For most games, the ease of use of the 3.5 loader is gone.
The game dev or modder interested in using the current 4.x loader version has to ensure that a named class is only extended once.
This can be achieved by creating an "API Mod" that all other mods use.
Some experiments to make this process easier are here:

https://github.com/GodotModding/godot-mod-loader/pull/347
https://github.com/GodotModding/godot-mod-tool/pull/105
(I dropped the development on this to focus on finding the issue in the engine, but let me know if anything seems interesting)

So just free-form modding with 4.x will currently most likely end up crashing your game with more than 1 mod installed.

Anything else?

Yes, https://github.com/GodotModding/godot-mod-loader/issues/345
"Parser Error: Class "ClassName" hides a global script class." I think the only reason why this works in Godot 3.x is that there is no check for that there. So we have to scrap the script saving in 4.x or come up with a solution to save them without triggering this parse error.

renamed since the mod loader (4.x branch) can be used as normal as long as no class_names are used - as proven by Windowkill


Code reuse for exporters

The only difference between span exporter and metric exporter is an additional temporality property in metric exporter. Therefore, the MATLAB code for processing inputs and outputs in these exporters' implementation may be reused for better maintainence.

In metrics branch


feat: AS-297 Make refine header optional

This PR adds a new props to the FilterListDrilldown to allow for hiding the refine bar.
new props defaults to true and it's named showDrillDownHeader (Optional- boolean).

This package will no longer be a part of this repo. This will be handled in a separate repo where further fscomponents refactor will be completed.


Building ARM and x64 facing release build errors

I'm trying to build a UWP application for release using AWS SDK (Core, S3, Cognito), but I'm having these error pop up. Any thoughts an suggestions would be really helpful!
I'm building towards fall creators, and my min is set to 240.
1>C:\Users\jeffreyhaines\.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets(697,5): warning : ILTransform : warning ILT0005: Type 'Amazon.Runtime.Internal.Util.AESEncryptionPutObjectStream' was not included in compilation, but was referenced in method 'EncryptionUtils.EncryptRequestUsingInstruction(Stream, EncryptionInstructions)'. There may have been a missing assembly. 1>C:\Users\jeffreyhaines\.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets(697,5): warning : ILTransform : warning ILT0003: Method 'EncryptionUtils.EncryptRequestUsingInstruction(Stream, EncryptionInstructions)' will always throw an exception due to the missing method 'AESEncryptionPutObjectStream..ctor(Stream, byte[], byte[])'. There may have been a missing assembly. 1>C:\Users\jeffreyhaines\.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets(697,5): warning : ILTransform : warning ILT0005: Type 'Amazon.Runtime.Internal.Util.AESEncryptionUploadPartStream' was not included in compilation, but was referenced in method 'EncryptionUtils.EncryptUploadPartRequestUsingInstructions(Stream, EncryptionInstructions)'. There may have been a missing assembly. 1>C:\Users\jeffreyhaines\.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets(697,5): warning : ILTransform : warning ILT0003: Method 'EncryptionUtils.EncryptUploadPartRequestUsingInstructions(Stream, EncryptionInstructions)' will always throw an exception due to the missing method 'AESEncryptionUploadPartStream..ctor(Stream, byte[], byte[])'. There may have been a missing assembly. 1>C:\Users\jeffreyhaines\.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets(697,5): warning : ILTransform : warning ILT0005: Type 'Amazon.Runtime.Internal.Util.AESDecryptionStream' was not included in compilation, but was referenced in method 'EncryptionUtils.DecryptStream(Stream, EncryptionInstructions)'. There may have been a missing assembly. 1>C:\Users\jeffreyhaines\.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets(697,5): warning : ILTransform : warning ILT0003: Method 'EncryptionUtils.DecryptStream(Stream, EncryptionInstructions)' will always throw an exception due to the missing method 'AESDecryptionStream..ctor(Stream, byte[], byte[])'. There may have been a missing assembly. 1>C:\Users\jeffreyhaines\.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets(697,5): warning : ILTransform : warning ILT0005: Type 'Amazon.Runtime.Internal.Util.EncryptUploadPartStream' was not included in compilation, but was referenced in method 'SetupDecryptionHandler.PostInvokeSynchronous(IExecutionContext, byte[])'. There may have been a missing assembly. 1>C:\Users\jeffreyhaines\.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets(697,5): warning : ILTransform : warning ILT0003: Method 'SetupDecryptionHandler.PostInvokeSynchronous(IExecutionContext, byte[])' will always throw an exception due to the missing method 'EncryptUploadPartStream.get_InitializationVector()'. There may have been a missing assembly. 1>C:\Users\jeffreyhaines\.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets(697,5): warning : ILTransform : warning ILT0005: Type 'Amazon.Runtime.Internal.Util.DecryptStream' was not included in compilation, but was referenced in method 'GetObjectResponse.<>c.<ValidateWrittenStreamSize>b__114_0(Stream)'. There may have been a missing assembly.
Here are my dependencies:
Syncfusion Controls for UWP XAML, Version=<IP_ADDRESS>
AWSSDK.CognitoIdentity <IP_ADDRESS>
AWSSDK.Core 3.3.18
AWSSDK.S3 <IP_ADDRESS>
Microsoft.ApplicationInsights 2.4.0
Microsoft.ApplicationInsights.PersistenceChannel 1.2.3
Microsoft.ApplicationInsights.WindowsApps 1.1.1
Microsoft.EntityFrameworkCore.Sqlite 1.1.3
Microsoft.NETCore.UniversalWindowsPlatform 5.4.1
Microsoft.Toolkit.Uwp 1.5.1
Microsoft.Toolkit.Uwp.UI 1.5.1
Microsoft.Toolkit.Uwp.UI.Controls 1.5.1
Newtonsoft.Json 10.0.3
Win2D.uwp 1.21.0

It looks like we have an issue with our nuspec file for AWSSDK.S3 and AWSSDK.CognitoIdentity which is causing Visual Studio to pull in the .NET Standard version of the those packages but then for AWSSDK.Core it is pulling in the portable class library version.

@normj what is the timeline for a fix? we are still unable to build our projects because of this issue.

New versions of all of the packages was released today that should fix the issue.

I'm seeing a set of new errors pointing to amazon:
Severity	Code	Description	Project	File	Line	Suppression State
Error		ILT0005: 'C:\Users\jeffreyhaines.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\x64\ilc\Tools\nutc_driver.exe @"C:\Users\jeffreyhaines\Documents\git\StorySlab-Windows\SCHWING\obj\x64\Release\ilc\intermediate\MDIL\SCHWING.rsp"' returned exit code 1	SCHWING	C:\Users\jeffreyhaines.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets	697
Error		Type 'System.Net.IWebProxy' was not included in compilation, but was referenced in method 'AmazonS3HttpUtil.GetProxyIfAvailableAndConfigured(IClientConfig)'. There may have been a missing assembly.	SCHWING	C:\Users\jeffreyhaines.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets	697
Error		Method 'AmazonS3HttpUtil.GetHttpClient(IClientConfig)' will always throw an exception due to the missing method 'HttpClientHandler.set_Proxy(IWebProxy)'. There may have been a missing assembly.	SCHWING	C:\Users\jeffreyhaines.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets	697
Error		Method 'AmazonS3HttpUtil.GetHttpClient(IClientConfig)' will always throw an exception due to the missing method 'HttpClientHandler.set_Proxy(IWebProxy)'. There may have been a missing assembly.	SCHWING	C:\Users\jeffreyhaines.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets	697
Error		Type 'System.Net.IWebProxy' was not included in compilation, but was referenced in method 'AmazonS3HttpUtil.GetProxyIfAvailableAndConfigured(IClientConfig)'. There may have been a missing assembly.	SCHWING	C:\Users\jeffreyhaines.nuget\packages\microsoft.net.native.compiler\1.7.1\tools\Microsoft.NetNative.targets	697

Hello, just wanted to report we were able to successfully build after migrating our Syncfusion dependency, which we believe was precompiled and interfering with the ones AWS SDK required. After moving from the precompiled Syncfusion package installer to managing them under Nuget, the Amazon errors I reported resolved and we were able to compile for all target platforms.

Thanks for following up and letting us know what was causing the additional errors.


lovelace:0.3.0

I am submitting

[ ] a new package
[x] an update for a package

Lovelace is a package for producing pseudocode with great flexibility.
The new version contains a major rewrite of the package, updating it to the new Typst grid features.

Thank you :)


healthchecker is nil

Summary
A nil Healthchecker causes an abort.
Additional Details & Logs

Kong version ($ kong version) 1.4
Kong debug-level startup logs ($ kong start --vv)
Kong error logs (<KONG_PREFIX>/logs/error.log)

lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/runloop/balancer.lua:241: attempt to index local 'healthchecker' (a nil value)

Operating system: Alpine docker image

@hbagdi This seems related to #5189, with the balancer being in a bad state.


Being able to adjust number of decimals

Most of the time decimals are not required on weather related values.
E.g. air pressure in hPa and with one decimal, not even our national weather service do that and it requires high precision sensors of a totally different kind that normally not are in use in home assistant environments.
The same goes with temperature, if one could choose to show decimals or not. It will give a cleaner look.

Sorry about the air pressure example, nothing to complain over here :-),
but I hope to have made my point anyway. :-)

Card is just displaying sensor value, attributes are rounded and you can choose in the forecast settings to round the temperature for the forecast. For now I will leave it like this.


Add "Listed" permission to a book

Describe the feature you'd like
The idea is to add a "listed" permission to a book (or article, or shelve), that would tell whether a book can be seen on a shelve or not without certain permission (but not to access it).
If ticked, it would behave as it does now
If unticked, it would act pretty much like Youtube's unlisted videos: If you have the correct link, you can read them (and if the permission is set on the book level and you have access to it, you can see the list of articles in said book), but they cannot be reached only by clicking.
Describe the benefits this feature would bring to BookStack users
Example case: You organize an event and have to disclose informations to a lot of people (they are not "sensitive" but they should not be public). Having to ask them to register and then manually act on the account to change its role so that they can read is painful.
This solution would allow to just give them a link without fearing that any wiki user find them.

Thanks for creating this issue @LoosGuccreen.
This is an interesting idea, I actually thought of something similar, except I was thinking about chapters having this "Unlisted" functionality instead. My thinking can be seen here in second-to-last paragraph.

I think that, once we have #288 implemented, this will be somewhat possible by removing all default view permissions after creating a custom access link.


Update title and introduction of the Project Portfolio Page

Can someone help update title and introduction of the Project Portfolio Page 😄

I think we have to add that ourselves. See my pull request #35 under files changed as a reference.

I think so too. I think we each wrote something so this should be complete. We may need to standardise it across our pages later though, but that's probably a separate issue. I will close this for now.


update kubespray default version to v1.24.7

What type of PR is this?
/kind feature
What this PR does / why we need it:
update kubespray default version to v1.24.7
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

/lgtm


Cannot run app

import { getHWID } from 'hwid'
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:355:18)
    at wrapSafe (node:internal/modules/cjs/loader:1039:15)
    at Module._compile (node:internal/modules/cjs/loader:1073:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1138:10)
    at Module.load (node:internal/modules/cjs/loader:989:32)
    at Function.Module._load (node:internal/modules/cjs/loader:829:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
    at node:internal/main/run_main_module:17:47

This project only works with ESM imports. See this post for more information.
If you do not wish to migrate to ESM (which you should), v0.3.0 and older still support CommonJS.

Ok


Updated to dwm 6.4

Merged commits "bump version to 6.4".."manage: For isfloating/oldstate check/set, ensure trans client actually exists" of upstream.

Bump

Merge?


Basics: don't point users to bugs.swift.org with InternalError

Since https://bugs.swift.org redirects to https://github.com/apple/swift/issues/, we'd like to point users to the SwiftPM repository issues instead.

@swift-ci smoke test


Freebsd ci

Thanks @olevole! Strangely it looks like at the build step in YAML, the /etc/skel directory is not the same, therefore the npm install (which invokes clang) gets skipped on i386: https://cirrus-ci.com/task/5116249953533952
if test "$abi" = "freebsd:11:x86:32"; then
  $execPrefix cd /etc/skel
  $execPrefix ls
fi
.cshrc
.profile
bin
boot
compat
COPYRIGHT
dev
etc
home
lib
libexec
media
mnt
net
proc
rescue
root
sbin
sys
tmp
usr
var

Merged ef36dc270407b3f450a1aa262d5c7cec44b57316.
Still trying to figure out how to do the batching in YAML.

Since heredoc doesn't work in .cirrus.yml (yet?) as does in .travis.yml for example, we will need to create a separate file for execution.
Could cbsd support workdir with jexed? Something like: cbsd jexec workdir=/my/path jail=myjname <command-with-args>. More generally, it would be also cool to be able to do something like:
cbsd jexec jname=j11i386 sh -c "(cd /etc/skel && ls && npm install --unsafe-params)"

in a single line.

Turned out that both heredoc and herestring variants work in normal script, but not from the YAML.
I ended up echoing it like this:
  build_script:
    - |
      if test "$abi" = "freebsd:11:x86:32"; then
        changeDir="cd /etc/skel &&"
      fi
      echo "$changeDir npm install --unsafe-perm" | $execPrefix /bin/sh

Builds are green, trying to furnish desire of publishing binaries to GitHub release artifacts from CI now. 😄

@olevole, I saw cbsd 12.0.4 has been released, could you tell how long before it will show up at http://pkg.freebsd.org/freebsd:11:x86:64/latest/All/ and http://pkg.freebsd.org/freebsd:11:x86:32/latest/All/?

@am11 unfortunately, I don’t know how often these packages are build. Probably during the week. But CBSD 11.0.4 still not in ports tree: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235412
I'm just waiting ;) due to not have ports commit bit


Add Collection object

This fleshes out the collection object so it can be used in further pull requests.

This looks fine, though I don't have a compelling use-case to try this object out.

Ready to go.

:+1:

If someone else can review, this should be ready to be merged.


make a kazuma kiryu facecam typing plugin

a plugin that adds a facecam of kazuma kiryu in the top left or top right of your screen basically whenever you are typing kazuma moves his hands when you send the message kazuma nods, to understand what I mean refer to this gif of kazuma kiryu typing "BOOOOBS" or the second gif https://cdn.discordapp.com/attachments/809232676590059550/1190008938696032317/RDT_20231228_1746551194849672011862037.gif?ex=65a03d30&is=658dc830&hm=653392a2b7bb897df0bbc8ca91381402980f6c22d1a7eb8b30192f77c2c03434&
https://cdn.discordapp.com/attachments/1205207690352005243/1208913839257686097/yXOy4hYw.gif?ex=65e503bf&is=65d28ebf&hm=62ce39acb04a42e69baa25f367083692285d01680caf72ac6df88f1c1ba92ec9&
bonus : you can add this song to be playing when the plugin is active (song can be turned off) : https://youtu.be/eO2J-pwpU8Y

@ErronZak could you send the first gif again pretty please the link broke


AuthorizationFailed when switching between Learn Modules

Does this occur consistently? Yes
Repro steps:
I use Azure sandboxes for Azure learning paths. I don't see the new subscription even when I refresh in vscode. I have the following error:
Action: azureFunctions.createFunctionApp
Error type: AuthorizationFailed
Error Message: The client<EMAIL_ADDRESS>with object id 'cafee073-eba5-4c2a-8166-864643125a29' does not have authorization to perform action 'Microsoft.Resources/subscriptions/locations/read' over scope '/subscriptions/0cb1fffd-fe14-44ea-a686-6eff61f986fc' or the scope is invalid. If access was recently granted, please refresh your credentials.
Version: 0.22.1
OS: linux
OS Release: 4.15.0-106-generic
Product: Visual Studio Code
Product Version: 1.46.1
Language: en
Call Stack
extension.bundle.js:225:374996
retryCallback extension.bundle.js:225:114550
retryCallback extension.bundle.js:157:31063
extension.bundle.js:157:32676
handleRedirect extension.bundle.js:171:7752
extension.bundle.js:157:29848
Request._callback extension.bundle.js:1:670871
Request.n._callback.n.callback.n.callback extension.bundle.js:201:4483
Request.emit events.js:203:13
Request.<anonymous> extension.bundle.js:201:17592
Request.emit events.js:208:15
IncomingMessage.<anonymous> extension.bundle.js:201:16382
Object.onceWrapper events.js:291:20
IncomingMessage.emit events.js:208:15
endReadableNT _stream_readable.js:1168:12
processTicksAndRejections task_queues.js:77:11

@hamzahamidi can you link me to a specific learn module where this reproduces for you? I've tried a few learn modules in the past and they seemed to work

Hello @EricJizbaMSFT. Thank you for your response. I was following the create serverless application path  and then when moving from a course to another I had this problem. I believe it's because at the end of each course the sandbox is deleted & a new one is created when beginning a new course.

Ah so it only happened right after switching courses?

Yes. Reinstalling Vscode does force the force thus solving the issue though.

This issue has become stale and is at risk of being closed. The community has 60 days to upvote the issue. If it receives 5 upvotes we will keep it open and take another look. If not, we will close it. To learn more about how we handle issues, please see our documentation.\n\nHappy Coding!

:slightly_frowning_face: In the last 60 days, this issue has received less than 5 community upvotes and we closed it. Still a big Thank You to you for taking the time to create it! To learn more about how we handle issues, please see our documentation.
Happy Coding!


Storing user object in all APIs and enabling filter of response based on user

Signed-off-by: Ravi Thaluru<EMAIL_ADDRESS>Issue #, if available:
https://github.com/opensearch-project/index-management/issues/75
Description of changes:

Storing user information populated by security plugin in the APIs

For transforms, rollups, policies there is a high level attribute called user which stores the user information
For managed indices, the attached policies of the managed index will have the user information
None of the indexed job metadata has the user information

Using the stored user information to run the background jobs - rollups, transforms, ism jobs

The user information is loaded into thread context to do this and this is only done for specific transport actions called as part of the background job - e.g. when writing data to ism config index the threadcontext will not have user information but when reading data from the source index in rollup job will have the user information in threadcontext. This will ensure an less privileged user will not be able to read/write data from other indices they don't have permissions for by creating rollup/transform jobs. Similarly when executing the policy actions the user information is added to thread context
Any data that was written prior to this change will not have user information in them and they are regarded as jobs with full permissions

Enabling new setting to filter stored information based on the appropriate use permissions

There is a new setting which enables filtering of data returned by the IndexManagement plugin APIs, if this setting is enabled it is required that all users updating/creating jobs using IM plugin have backend roles. Other wise the request will fail
If the setting is enabled a user will only see data that matches their backend roles. All the existing data that is written without backend roles will not be visible to anyone (This is a limitation)

Call outs:

One can no longer create a managed index job with a non existent policy, add policy API will verify that the policy exists and is stored with the managed index at the time of creation
Filtering for managed indices is implemented yet based on search, i.e if user has a search permission on an index they will be able to manage the index - this check is implemented in AddPolicy API, ManageIndices API and the cluster event sweeper to determine the policy that should be applied to an index
User information stored in the jobs is not being returned as part of the REST API responses

CheckList:
[  ] Commits are signed per the DCO using --signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #115 (77b8159) into main (5f6dd66) will decrease coverage by 1.43%.
The diff coverage is 62.36%.

@@             Coverage Diff              @@
##               main     #115      +/-   ##
============================================
- Coverage     77.26%   75.83%   -1.44%
+ Complexity     1900     1888      -12
============================================
  Files           258      259       +1
  Lines         10523    10839     +316
  Branches       1676     1716      +40
============================================
+ Hits           8131     8220      +89
- Misses         1474     1675     +201
- Partials        918      944      +26

Impacted Files
Coverage Δ

...agement/indexstatemanagement/model/ChangePolicy.kt
80.39% <0.00%> (ø)

...temanagement/opensearchapi/OpenSearchExtensions.kt
79.26% <ø> (-2.06%)
:arrow_down:

...ent/transport/action/explain/ExplainAllResponse.kt
92.00% <0.00%> (ø)

...ment/indexstatemanagement/util/RestHandlerUtils.kt
100.00% <ø> (+6.25%)
:arrow_up:

...arch/indexmanagement/rollup/RollupSearchService.kt
57.40% <0.00%> (-3.38%)
:arrow_down:

...arch/indexmanagement/transform/TransformIndexer.kt
71.73% <0.00%> (-6.84%)
:arrow_down:

...ch/indexmanagement/transform/TransformValidator.kt
58.00% <0.00%> (-2.00%)
:arrow_down:

...g/opensearch/indexmanagement/util/SecurityUtils.kt
20.00% <9.67%> (-24.45%)
:arrow_down:

...arch/indexmanagement/rollup/RollupMapperService.kt
53.65% <10.00%> (-4.24%)
:arrow_down:

...ndexmanagement/transform/TransformSearchService.kt
75.00% <33.33%> (-0.31%)
:arrow_down:

... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f6dd66...77b8159. Read the comment docs.

Few top level comments:

Lots of changes in APIs, just confirm we are not introducing any breaking changes by removing or renaming fields or changing models etc.
Lots of new behavior added for security, but I know we have issues running security tests locally w/ the local integ Cluster that runs. Can we still add integration tests that get filtered out if remote cluster is not used? i.e. we can still write integration tests, but they only run if isRemoteCluster && isSecurityEnabled or something.

If an existing ISM job doesn't have user info saved, what are the ways for user to adopt the new security model?
By calling change policy API?
And same question for Rollup and Transform jobs without user.

If an existing ISM job doesn't have user info saved, what are the ways for user to adopt the new security model?
By calling change policy API?
And same question for Rollup and Transform jobs without user.

Yes, updating would be the option


Tevatron/wiki-ss-nq Dataset issue

Loading Tevatron/wiki-ss-nq dataset fails using the following code snippet  (classes are from examples/des/dataset.py):
data_args = DataArguments(
    dataset_name="Tevatron/wiki-ss-nq",
    corpus_name="Tevatron/wiki-ss-corpus",
    train_group_size=2,
    query_max_len=128,
    passage_max_len=4096,
)
dataset = TrainDataset(data_args)
print(dataset[0])

The error obtained is the same error that is also present in the corresponding HuggingFace dataset page:
All the data files must have the same columns, but at some point there are 1 new columns ({'answers'}) and 2 missing columns ({'positive_passages', 'negative_passages'}).
The issue is also mentioned here: https://huggingface.co/datasets/Tevatron/wiki-ss-nq/discussions/2

weird...let me take a look today

Thanks for looking into it.
In particular,
from datasets import load_dataset
ds = load_dataset("Tevatron/wiki-ss-nq")

fails when creating the test split, as it seems the test split doesn't contain pos/neg passages columns.

does it works for you now! I tried to fixed it

Dataset loading works, thanks for the quick fix!


kubectl apply doesn't update nodeSelector

BUG REPORT
Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Environment:

Hardware configuration: bare metal x86_64
OS: Ubuntu 16.04.5 LTS
Kernel: 4.4.0-131-generic
Install tools: Rancher 2.0

What happened:
When I apply a modified YAML file for an existing Deployment, all changes are received by the cluster except nodeSelector.
What you expected to happen:
nodeSelector should be updated.
How to reproduce it:

Write a deployment file with some node selector in the pod spec template
Apply the deployment: kubectl apply -f path/to/file.yaml (should create the deployment)
Modify the nodeSelector (eg. add new entry) and another parameter (eg. Replicas)
Again apply the deployment: kubectl apply -f path/to/file.yaml
Observe that other changes are reflected, but nodeSelector stays unchanged

/kind bug
/sig cli
/area kubectl
/priority P2

I think it's fixed in 1.12 or it was just a glitch, as now I'm unable to reproduce it.
Current version:
Client Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.0-rc.1", GitCommit:"3e4aee86dfaf933f03e052859c0a1f52704d4fef", GitTreeState:"clean", BuildDate:"2018-09-18T21:08:06Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

/remove-lifecycle stale

@seans3
I think @adampl is correct in his earlier statement. I tested just now and could not reproduce.
Tried a few scenerios including:

without and then with nodeSelector
changing nodeSelector

I think we can close.

@pswica Thanks for the help--closing.
/close

Hey, we've been observing this on GKE, with 1.13.7-gke.8 for master/nodes and v1.13.7 kubectl. I added a nodeSelector, which got applied, but then removed it from the YAML later on and any future kubectl apply operations (that could successfully change other things) didn't touch nodeSelector.


[C#, Speech] remove ssml from C# EchoBot samples, generators

Proposed Changes

Remove SSML from C# EchoBot samples and Generators, use plain text for the Activity.Speak properties.

Testing
Tested by deploying bot to Azure and used the Direct Line Speech Client.

Related to https://github.com/microsoft/BotBuilder-Samples/issues/1981.

This PR is ready to go, I tested the bot with multiple DLS Clients and everything worked as expected. I also removed the SSML and used MessageFactory.Text(), following the pattern used in the C# Core Bot.


Honor percore flag when using psutil in CPUCollector

Fixes #788

@kormoc Can I interest you in this simple fix?

This repo has moved to https://github.com/python-diamond/Diamond
Please open a new pull request there if this issue is not resolved with the current code there.
Thanks!


V2 Planning

High-level Goals

Produce a cleaner final set of files.

1 interface file.
1 client file.
1 type file.

Delete types not in use anymore.
Overwrite GeneratedCodeAttribute to include our tool along with the original tools.

Dry run capabilities.
???

#4


Limit the number of simultaneous thumbnails that are requested in the itemList.

Chrome limits the maximum connectors to a domain to 6.  Girder uses one for getting events, leaving five.  This uses no more than 3 for thumbnails, which leaves some overhead for navigation.  If you navigate between item lists quickly, the connections can still be saturated, as Chrome doesn't cancel the image requests.  We would have to load the image to a hidden element first, and then load it to the visible element to work around that, so that the hidden elements don't vanish on page transitions.
This resolves issue #89.

@dgutman This will substantially improve the browser getting stuck when navigating between folders.  If you find through rapid navigation that this still happens, we can fetch images in a hidden element before showing them so that we track what is happening between navigation.  I don't want to implement that unless it is necessary, as it adds complexity.

@zachmullen You are probably the best person to review this, when you get a chance.

I would expect that deleting the img element would automatically cause the request for its data to be canceled, is that not the case?

The request is still being processed by the server.  I can't tell if they are still counted by Chrome.

The request is still being processed by the server. I can't tell if they are still counted by Chrome.

I guess the server wouldn't know until it actually tried to send the bytes back down. This may be something the network tab in chrome can show, in particular if we introduce an artificial delay on the server side.

I just placed a 30 second delay on all large image thumbnails.   When I navigate between two folders that have large images, the previous pending images remain in Chrome's network list, and DO block things until they resolve.
I can either work around that as an extension to this PR, or only make a separate PR when we decide that it is a real-world problem and not just when I artificially introduce it.

I think it's fine, mostly just curiosity.


Closes https://github.com/smartcontractkit/foundry-starter-kit/issues/29

What

Why

Closes https://github.com/smartcontractkit/foundry-starter-kit/issues/29


corrupt cursor in contacts list

We were not able to use the python sdk to pull the full contact list. It would error in a bad iterator on the second page of results from the rest api.
We resolved the issue by adding a limit param to the list call and then running the request once and then removing the limit param and the full request succeeded.
It seem the cursor on your server was somehow corrupt and then when a new one was issued it resolved itself.
The error was persistent over multiple days until we added that limit param.

Thanks for the info and sorry you had problems. I will pass this on to the relevant team members for investigation

@dp-rufus has this been investigated? it is now happening even more frequently

Hi Alex
To help me find this in our logs and figure the problem out, please can you confirm my understanding:
When this problem occurs (intermittent but increasingly frequent),

A request GET /api/v2/contacts (page 1) will appear to complete successfully

http 200 status code
'items' array of contacts
plausible-looking 'cursor' value (a long string, not absent or null)
are you providing the optional owner_id parameter?

You then (quickly afterwards) request GET /api/v2/contacts?cursor=xxxxx for page 2

this request fails - in what manner?
what status code do you get?
do you get any error message in the json response?

By repeating request 1 with a limit value, you could 'trick' the API into returning a different cursor, and this cursor then worked successfully in request 2

what limit value are you using?

Thanks for your help

We are using the python sdk python-dialpad==2.1.0. We call contacts = dp_client.contact.list(owner_id=x)
We get the following error:
500 Server Error: Internal Server Error for url: https://dialpad.com/api/v2/contacts?limit=25&owner_id=4526224355819520'
Stack trace:
"/usr/local/lib/python3.9/site-packages/dialpad/resources/contact.py line 20 in list", "/usr/local/lib/python3.9/site-packages/dialpad/resources/resource.py line 11 in request", "/usr/local/lib/python3.9/site-packages/dialpad/client.py line 74 in request", "/usr/local/lib/python3.9/site-packages/requests/models.py line 953 in raise_for_status"
We started with limit 25 but then that stopped working so we went to random between 20 and 30. That now no longer works.  I think the cursors are getting corrupted somehow and then not being removed.

Thanks for the info, I've found your stacktraces on our side. Please bear with me while I figure out the root cause and how to address it (I don't think it's related to cursors, surprisingly)

Great. Thank you. Looking forward to seeing the solution.

We're tracking the issue internally as DP-45739
In the meantime please consider adding retry logic on your side.
If you subclass DialpadClient you can attach a custom urllib3 Retry policy with an adaptor:
from urllib3.util import Retry
from requests.adapters import HTTPAdapter

retry_policy = Retry(.............................)
retry_adapter = HTTPAdapter(max_retries=retry_policy)

class BetterDialpadClient(DialpadClient):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._session.mount("https://", retry_adapter)

I'll give that a try and let you know if the issue continues.
Thank you for creating the internal issue and tracking for us. Much
appreciated.
-Alex
On Mon, Jan 31, 2022 at 8:48 PM dp-rufus @.***> wrote:

We're tracking the issue internally as DP-45739
In the meantime please consider adding retry logic on your side.
If you subclass DialpadClient you can attach a custom urllib3 Retry policy
with an adaptor:
from urllib3.util import Retryfrom requests.adapters import HTTPAdapter
retry_policy = Retry(.............................)retry_adapter = HTTPAdapter(max_retries=retry_policy)
class BetterDialpadClient(DialpadClient):
def init(self, *args, **kwargs):
super().init(*args, **kwargs)
self._session.mount("https://", retry_adapter)
—
Reply to this email directly, view it on GitHub
https://github.com/dialpad/dialpad-python-sdk/issues/13#issuecomment-1026475051,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOP6VYHTGJT4JWRCKXK47LUY5QZ7ANCNFSM5KESZWOA
.
You are receiving this because you authored the thread.Message ID:
@.***>

any updates here?


Remove #scrapbook from channelKeywords

I noticed that scrappy was reacting to all the #scrapbook posts with the :scrappy: emoji, so I believe I initially misunderstood the purpose of channelKeywords.
I thought it was the same as emojiKeywords but with channel IDs. However it seems like Scrappy reacts with the respective emoji if the post is scrapbooked from the specified channel.
This PR fixes that, so scrappy stops reacting to every single post with :scrappy:!

thanks!


/api/nft/upload 404

Hi,
Thanks for sharing sample project for NFT. I'm trying to deploy it, but i got 404 not found message for /api/nft/upload.
I tried on your demo and it also got the same error.
Could you guide me to fix this error?
Thank you so much.

Hi @arrowhitech-gp
Have you added the .env as I mentioned on readme yet? https://github.com/jellydn/nft-app#usage
By the way, I will fix the deployment issue. Thanks.

Here is my setup on local.
.env
VITE_NFT_DEPLOYED_ADDRESS=0xA0011f0372F6E4E29C0ae6BC64F5C087E48BfEB7
# Create an account on https://nft.storage/ and get a key
NFT_STORAGE_KEY=
VITE_CHAIN_ID=4
API_URL=http://localhost:8080

and
server/.env
# Create an account on https://nft.storage/ and get a key
NFT_STORAGE_KEY=<YOUR NFT STORAGE KEY>
PORT=8080

Let me know if that works for you. @arrowhitech-gp

Already fixed the deployment issue. Thank you for your report.

Thank @jellydn, I will try and update you soon.

It is working fine now, thank you @jellydn


OstSdk & App change: Handling of apis lower than 22.

OstSdk change: Handling of apis lower than 22.
OstSdk App change: Application lower than 21 need multi-dex support, multi-dex file support added and code change for compatibility with ost-sdk

Closing the pull request as this is obsolete.


fixed scopes parsing in refresh token

fixed "The requested scope is invalid, unknown, or malformed: Check the [object Object] scope(s)" erro while generating refresh token without "scope" claims in the request

see #11


Design

Change or add design on the website 🍪

Can you specify what exactly do you want?

@iamshm I don't have any specific in mind. You can change or add almost anything ;)

cool then I'll try to enhance this website :)

I did enhance a bit. I'm a new webdev so I did my best.

Great work thanks :)

Challenge accepted :thinking:

Awesome work will close this for now


DOC: Update jnp.mean doc

Updates the doc of jnp.mean and also fixes a typo in jnp.sum doc.
Part of #21461.

Implemented the review changes and squashed into single commit.


Modifiers declared in the incorrect order.

What is the issue/code smell?
Java modifiers are declared the incorrect order.
Why is this issue/ code smell relevant?
This causes a minor impact as doesn't cause any technical problems but as most of the developers follow the standard order which says "static" should be before "final" that can reduce the readability of the code.
How is the issue resolved?
Reordering the modifiers to comply with the Java Language Specification i.e., changing the order from "public final static" to
" public static final".

Looks good.


Invalid encryption keys: Provided protocol not recognized

Hi,
We are trying to encrypt a docker image with a key from coco_keyprovider for example I used the following command to encrypt the docker image
OCICRYPT_KEYPROVIDER_CONFIG=newocicrypt.conf skopeo copy --encryption-key provider:attestation-agent:sample=true  oci:alpine oci:encrypted
But I get the following error:
FATA[0000] Invalid encryption keys: Provided protocol not recognized
Any idea why it might happens?

Hi @pegahnikbakht could you share more context to reproduce the error?
btw, the error seems from skopeo

You might double check that skopeo can find the ocicrypt config you specific and that the file is formatted correctly.

Hi, yeah the problem was skopeo version, I updated and it works fine now.


PLs help.

its just stuck on this :(

Wrong installer use this and run it  https://github.com/rauenzi/BetterDiscordApp/releases/ and if it's stuck on killing process then run it on the desktop as admin


Question: Add model phpdocs to method

When generating phpdocs for models they're added to the model class by default (top of the file, above the class).  Is it possible to add them above each method they're referring to within the model class instead?

What problem would this solve?

The phpdoc contains references to properties (i.e. database columns) which might / might not have a custom getter
and then it also contains references to "virtual" methods which don't exist in the first place, so to which other method should they be added?

Personally I don't think this is something the package should do.
@barryvdh personal opinion 😬, I don't think this should be supported and vote for closing :}


Use additional sources for better coverage

[x] http://unimorph.ethz.ch/languages
[x] https://github.com/lenakmeth/Wikinflection-Corpus
[x] https://github.com/TALP-UPC/FreeLing/tree/master/data/
[x] https://github.com/explosion/spacy-lookups-data/tree/master/spacy_lookups_data/data
[x] https://github.com/tatuylonen/wiktextract
[ ] http://pauillac.inria.fr/~sagot/index.html#udlexicons

kaikki.org recently published a number of parsed dumps of Wiktionary in other languages. Could be interesting to include as they may have more morphology data.


Dissenter doesn't work in brave

The login page fails to load. Look at what happens when I click the login button.

Wonder if this is related to this problem I am seeing on their website today:

The login page fails to load. Look at what happens when I click the login button.

A way to bypass this is right-click on the login in button and open in new tab, it should load "Gab Trends". Log into that website with your Gab account and then you'll be auto-logged into the extension.


Question Structure Review

-Review existing questions for YAML structure
-Review existing questions for multiple choice options

YAML Structure is fine as it will be converted to CSV


Crashes under high load

master process is writing to 1 nested dict1 (recurse=1) shared between 20-40 processes,  total dict1 size ~1500 keys with nested dict (as value, small)
processes created via multiprocessing.Process, and writing to other shared dict - dict2[process_id] once per second, dict2 size - same, but *num_processes
main process analyzing statistics from dict2: for process_id in dict2: dict2[process_id]: ...
and write changes to shared dict1 once per second: for change in changes: dict1['nested'][change] = {'time': 123, 'blah': '123'}
crashing appears if changes size is 300-2000 in 1 second, and read lookups is HUGE (>100k/sec) but i tried to cache it once per second to local dict using deepcopy and this doesnt help...
total memory usage not exceed 2-4GB i think (free ram is about 60GB), CPU usage up to 100%
dict1 size in bytes determined on local dict with same structure is less than 150kb
i tried:

copy.deepcopy(dict1) once per second to create a local copy in processes for cached lookups - doesn't help
shared_lock
with dict1.lock/etc
increasing buffer to huge values, increasing full dump size/etc

and nothing helps... on low speeds (or no/small changes from master to dict1) all is working, or using multiprocessing.manager().dict all is working too, but slow
Examples of exceptions:
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
	self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
	self._target(*self._args, **self._kwargs)
  File "zvshield.py", line 793, in zvshield.accept_connections
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 585, in __contains__
	self.apply_update()
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 511, in apply_update
	assert bytes(self.buffer.buf[pos:pos+1]) == b'\x00'
AssertionError

File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 248, in __init__
	self.buffer = self.get_memory(create=True, name=self.name + '_memory', size=buffer_size)
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 347, in get_memory
	full_dump = self.serializer.loads(bytes(buf[pos:pos+length]))
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 304, in __init__
	self.apply_update()
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 520, in apply_update
	memory = multiprocessing.shared_memory.SharedMemory(name=name)
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 114, in __init__
	mode, key, value = self.serializer.loads(bytes(self.buffer.buf[pos:pos+length]))
	self._mmap = mmap.mmap(self._fd, size)
OSError: [Errno 12] Cannot allocate memory

EOFError: Ran out of input
Exception ignored in: <function SharedMemory.__del__ at 0x7fb80639e820>

Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 184, in __del__
	self.close()
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 227, in close
Exception ignored in: <function SharedMemory.__del__ at 0x7fb80639e820>
	self._mmap.close()
Traceback (most recent call last):
BufferError: cannot close exported pointers exist
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 184, in __del__
	self.close()
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 227, in close
	self._mmap.close()
BufferError: cannot close exported pointers exist
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 184, in __del__
	self.close()
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 227, in close
	self._mmap.close()
BufferError: cannot close exported pointers exist
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
	self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
	self._target(*self._args, **self._kwargs)
  File "zvshield.py", line 793, in zvshield.accept_connections
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 585, in __contains__
	self.apply_update()
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 500, in apply_update
	self.load(force=True)
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 450, in load
	full_dump = self.serializer.loads(bytes(buf[pos:pos+length]))
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 304, in __init__
	self.apply_update()
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 520, in apply_update
	mode, key, value = self.serializer.loads(bytes(self.buffer.buf[pos:pos+length]))
EOFError: Ran out of input
Exception ignored in: <function SharedMemory.__del__ at 0x7fc48f4d4820>
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 184, in __del__
	self.close()
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 227, in close
	self._mmap.close()
BufferError: cannot close exported pointers exist
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
	self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
	self._target(*self._args, **self._kwargs)
  File "zvshield.py", line 793, in zvshield.accept_connections
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 585, in __contains__
	self.apply_update()
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 500, in apply_update
	self.load(force=True)
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 450, in load
	full_dump = self.serializer.loads(bytes(buf[pos:pos+length]))
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 304, in __init__
	self.apply_update()
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 520, in apply_update
	mode, key, value = self.serializer.loads(bytes(self.buffer.buf[pos:pos+length]))
EOFError: Ran out of input

Exception ignored in: <function SharedMemory.__del__ at 0x7fc48f4d4820>
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 184, in __del__
	self.close()
  File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 227, in close
	self._mmap.close()
BufferError: cannot close exported pointers exist

interesting that after exceptions like above python starts to crashing in other parts of code:
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f9f9ce0d370>
transport: <_SelectorSocketTransport fd=172 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
File "/usr/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
data = self._sock.recv(self.max_size)
MemoryError
Fatal read error on socket transport
but free memory is HUGE and ram usage is small

Traceback (most recent call last):
File "zvshield.py", line 598, in shared_dict_updater
File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 557, in setitem
self.apply_update()
File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 500, in apply_update
self.load(force=True)
File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 450, in load
full_dump = self.serializer.loads(bytes(buf[pos:pos+length]))
_pickle.UnpicklingError: invalid load key, '\x00'.

reducing "changes" updates from
"for change in changes: dict1['nested']['change'] = change"
to
"dict['nested'] = changes"
helping... not a fix but workaround

@rojamit Many thanks for your report. I'll look into it and try to analyze what's going on. In the meantime, feel free to test the dev branch. It possibly contains fixes for some of the errors you have seen.
There's also a new example examples/mccoydj1.py (from issue #6 ) in the dev branch where 2 processes write with maximum speed and one process reads with maximum speed. The fixes in the dev branch are mainly to stabilize this.

ooooo.... my terribly cruel use case earned itself its own example file! Sorry for causing so much pain... thanks for your help improving this library :)

@rojamit Regardig your points:

There already is dict1.data which is a local copy and you can use it to access the dict cached data. You could then run dict1.apply_update() manually to update the local copy once a sec or so.
If you fork your processes from a parent process, you can use the faster RLock with shared_lock=False.
Using the lock should help, but it depends on your situation. Reads are not protected by locks for performance. The dev branch contains some code to better protect reads.
You need to find the right balance for the buffer sizes. If your total data is like 150k, I wouldn't make the buffer much bigger.

Please let me know if the dev branch improves your situation.

@ronny-rentner thanks, i'll test dev branch today
my current buffer_size=500000
in my case (for now) there is 1 write/sec from master process, same for each child process, but child processes may read dict at 100k/s speed or even more
i have structure like dict[process_id] for each child process, do i still need locks?

@ronny-rentner dev branch is much more stable. but... after about 12 hours, master has crashed:
future: <Task finished name='Task-1' coro=<()> exception=OSError(12, 'Cannot allocate memory')>
Traceback (most recent call last):
File "zvshield.py", line 515, in main
File "zvshield.py", line 410, in zvshield.analyze_netstat
File "/usr/lib/python3.9/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.9/copy.py", line 230, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.9/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.9/copy.py", line 264, in _reconstruct
y = func(*args)
File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 369, in init
self.lock = multiprocessing.RLock()
File "/usr/lib/python3.9/multiprocessing/context.py", line 73, in RLock
return RLock(ctx=self.get_context())
File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 187, in init
SemLock.init(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 57, in init
sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 12] Cannot allocate memory
Exception ignored in: <function SharedMemory.del at 0x7f01051974c0>
Traceback (most recent call last):
File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 184, in del
self.close()
File "/usr/lib/python3.9/multiprocessing/shared_memory.py", line 227, in close
self._mmap.close()
BufferError: cannot close exported pointers exist

@rojamit Thanks for the update.
Regarding your question:

i have structure like dict[process_id] for each child process, do i still need locks?

No, in this case you don't need additional locks. UltraDict will use locks internally to protect writes. Though, you have to use shared_lock=true if you spawn the processes. If you fork them all from the same master process, that is not necessary.
Regarding the remaining exception(s):
Interesting. Can you check how much shared memory is left if you get such an exception? The error does not seem to happen directly in UltraDict but when Python tried to initialize an RLock.
File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 57, in init sl = self._semlock = _multiprocessing.SemLock( OSError: [Errno 12] Cannot allocate memory
Though, of course it could be caused by UltraDict not doing clean up correctly.
Over those 12 hours, are you using the same UltraDicts or are you creating new ones?
UltraDict tries to initialize its lock only when a new instance is created.
File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 369, in init self.lock = multiprocessing.RLock()
Are you maybe (accidentally?) keeping around lots of UltraDict instances until all memory is gone?

@ronny-rentner using dev branch, app is running fine for 2 days now (no restarts), definitely dev is much more stable: no errors from UltraDict during this period.
I'm creating these dicts once (at startup):
	HOST_SETTINGS = UltraDict({}, buffer_size=1000000, recurse=True) 	WORKERS = UltraDict({}, buffer_size=1000000) 	ADDR_INFO = UltraDict({ 		'banned': {}, 		'checked': UltraDict({}, buffer_size=1000000) 	}, buffer_size=1000000) 	EVENT_BUS = UltraDict({ 		'events': {}, 		'new_ssl': {} 	}, buffer_size=1000000, recurse=True)

@ronny-rentner yay just told all is ok and now after load spike app crashed again
40 processes, 200+k/s readings like "if x in dict1.data['key']"
dict1.data['key'] is a nested dict
crashed with KeyError: 'key', and then looks like ultradict is disconnected from process where error appears

@rojamit That's interesting because ADDR_INFO.data is an original Python dict. If I understand it correctly, KeyError simply means that 'banned' does not exist in the dict. If you access ADDR_INFO['banned'], does it work? If you can catch the error, can you run ADDR_INFO.print_status() so we get some more debug info?

@ronny-rentner i'll try print_status, but 'banned' is always-existing key... processes are crashing randomly on high load (not all in one time!), 'banned' is updating from master process using ADDR_INFO[banned] = x, other processes only reading, and gets updates in another asyncio coroutine using ADDR_INFO.apply_update() (of course in its process)
if this KeyError occurs in process, he never recovers, in simple words it looks like that process is disconnected from UltraDict, i dont know how exactly UltraDict works in details, but can 'banned' disappear from .data while UltraDict is updating (or doing something else)?

@ronny-rentner also using recurse=True increased errors alot

@rojamit

i dont know how exactly UltraDict works in details, but can 'banned' disappear from .data while UltraDict is updating (or doing something else)?

UltraDict works with a stream of updates. This includes the possibility of sending a delete message in the update stream. If you run del ADDR_INFO['banned'] in one of the processes, it will try to propagate this update. This is eventually just one byte in the update stream that tells UltraDict to either delete or update the key.
You could try to add some debug prints in the __delitem__() method to find out if really a delete message is being received. https://github.com/ronny-rentner/UltraDict/blob/42066b997b99d8882c27ef7bb8bf722ca648ac8a/UltraDict.py#L545
During my tests I had lowered the buffer_size to a really tiny value (like 100 or 1000 bytes). This has triggered the error much faster. Is this also the case with your program? It might make it easier to get a stack trace.

@rojamit I've found an error which only appears with full dumps and recurse=True which is most likely the cause for your OSError: [Errno 12] Cannot allocate memory and it might be the cause for other errors. It seems some file pointers are left behind and never closed and at some point you run out of memory or have too many files open. I'll let you know once I have a fix.

@ronny-rentner yes, looks like error only appears with recurse=True (with dev branch)
'banned' key is always-existing key as i mentioned, it never gets deleted.
In my case some processes crashes but others continue work, but when key is deleted i think its not possible way
Looks like dev is stable without recurse=True and with updating method i mentioned (for large updates)

@ronny-rentner some errors on high load today
ERROR:root:Exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 649, in apply_update
    assert bytes(self.buffer.buf[pos:pos+1]) == b'\xFF'
AssertionError

Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/selector_events.py", line 870, in _read_ready__data_received
    self._protocol.data_received(data)
  File "zvshield.py", line 247, in zvshield.HttpProtocol.data_received
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 753, in __contains__
    self.apply_update()
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 692, in apply_update
    raise e
  File "/usr/local/lib/python3.9/dist-packages/UltraDict/UltraDict.py", line 649, in apply_update
    assert bytes(self.buffer.buf[pos:pos+1]) == b'\xFF'
AssertionError

WARNING:root:Full dumps too fast full_dump_counter=20 full_dump_counter_remote=21. Consider increasing buffer_size.

@rojamit Unfortunately the line numbers are not matching the dev branch exactly anymore. It's a bit strange because I had added code to actually use a lock as a last resort to recover from such errors. You can find it here: https://github.com/ronny-rentner/UltraDict/blob/dev/UltraDict.py#L691 Either there are more edge cases that I didn't find yet or the locking is not working correctly on your computer or your're running older code that did not have the lock as a fallback.
The root cause, at least as far as I understand it, is too small buffers. I think the following is happening:

You have 2 processes.
Process 1 is writing changes to the shared memory dict.
Process 2 is reading changes from the shared memory dict.

Imagine Process 2 is in the middle of processing an streaming update from the stream buffer but your operating system then pauses this process and gives another process some CPU time.
While Process 2 is paused, Process 1 continues to write updates to the stream. It has so many updates that it needs to create a full dump and reset the stream. And then it will continue to write more updates to the stream until it reaches the position in the stream where Process 2 currently is (but paused). Process 2 is still in the middle of processing the old update from the stream. Process 1 is lapping Process 2 and thus Process 2 is not able to finish processing the streaming updates.
Which buffer size are you using? Does it help to increase the buffer size? Ideally increase it until you don't get warnings anymore about too fast full dumps.

@ronny-rentner my buffer size is buffer_size=1000000 (as you can see in my past messages)
yes, i think the same about r/w issue, i'll update UltraDict again and check if error still occurs (previously updated from dev about week ago)

@ronny-rentner looks like json.dumps doesnt work with UltraDict(recurse=True), even when accessing it through .data
from nested dict like {k:{k:v}} json.dumps returns only {k: {}} (empty dict), strange

@ronny-rentner i think dev is working stable for now, without recurse=True. No errors during week at all

@rojamit Thanks for the feedback. I'm glad it works in this mode. I'm still working on improvements and cleanup for the recursive mode.

@rojamit There are some new improvements in the dev branch that are supposed to help on the recurse mode. I'm not sure if I should actually keep it or remove this feature as it adds quite some complexity.
Regarding json: Yes, I've noticed there are different json packages and jsons seems to do what we need.
>>> ultra
{'nested': {1: 1}}
>>> json.dumps(ultra)
'{}'
>>> jsons.dumps(ultra)
'{"nested": {"1": 1}}'

@ronny-rentner looks like i missed something, or performance on latest dev branch decreased
for now i have large dict (1k-50k keys or more) with statistics per IP to hostname, for example:
{ <IP_ADDRESS>: {hostname1: 11, hostname2: 37}, ... }
40 worker-processses counting requests and store stats in local dict
i need to summarize stats from all processes
previously i used method like:
shared_dict_workers[worker_id] = stats in worker,
and then loop over shared_dict_workers in master process
but it has poor performance because of increased complexity O(workers_count=40), same about ram usage
so i tried to create shared_dict_stats which gets updated directly from workers with lock from local worker_stats
but ultradict is huge bottleneck here, and i cant even use dict.data & dict.apply_update() because it doesnt work
dirty sketch:
`import UltraDict, multiprocessing, time
stats_total = UltraDict.UltraDict(buffer_size=1_000_000, recurse=True)
def worker(pid):
worker_stats = {                                                                               # worker_stats isnt static irl
f'{i}.{i}.{i}.{i}': dict( (f'hostname-{x}', x) for x in range(10) ) for i in range(3000) } # 3000 = keys, 10 = hostnames
while 1:
with stats_total.lock:
for addr in worker_stats:
if addr not in stats_total.data:
stats_total[addr] = worker_stats[addr]
for hostname in worker_stats[addr]:
if hostname not in stats_total.data[addr]:
stats_total[addr][hostname] = worker_stats[addr][hostname]
stats_total[addr][hostname] += 1 # actually it must be 'worker_stats[addr][hostname]', not 1
#print('updated from worker', pid)
time.sleep(0.001)
for pid in range(10): # processes count
process = multiprocessing.Process(target=worker, args=(pid,))
process.daemon = True
process.start()
time.sleep(2)
last = 0
while 1:
print('updates/sec:', stats_total['<IP_ADDRESS>']['hostname-1'] - last,
'total:', stats_total['<IP_ADDRESS>']['hostname-1'])
last = stats_total['<IP_ADDRESS>']['hostname-1']
time.sleep(1)
`
i need to summarize these stats once per second, but this sketch have lower rate (because of performance)

this works alot faster (mostly because of recurse=False?)
import UltraDict, multiprocessing, time

stats_total = UltraDict.UltraDict(buffer_size=1_000_000, recurse=False, data = {                                                                               # worker_stats isnt static irl
    f'{i}.{i}.{i}.{i}': dict( (f'hostname-{x}', x) for x in range(10) ) for i in range(3000)})

def worker(pid):
    while 1:
        with stats_total.lock:
            new = stats_total['data'].copy()
            for addr in new:
                for hostname in new[addr]:
                    new[addr][hostname] += 1  # must be 'worker_stats[addr][hostname]', not 1
            stats_total['data'] = new
        #print('updated from worker', pid)
        time.sleep(0.01)

for pid in range(10): # processes count
    process = multiprocessing.Process(target=worker, args=(pid,))
    process.daemon = True
    process.start()
time.sleep(2)

last = 0
while 1:
    print('updates/sec:', stats_total['data']['<IP_ADDRESS>']['hostname-1'] - last,
          'total:', stats_total['data']['<IP_ADDRESS>']['hostname-1'])
    last = stats_total['data']['<IP_ADDRESS>']['hostname-1']
    time.sleep(1)

my super-naive realization of something like shared dict shows that looks like actual problem is not with UltraDict, but with slow shared memory io itself:
`import multiprocessing, time, ctypes
lock = multiprocessing.RLock()
shared_string = multiprocessing.RawArray(ctypes.c_char, b'\x00'* 500000)
shared_string.value = str({
f'{i}.{i}.{i}.{i}': dict( (f'hostname-{x}', x) for x in range(2) ) for i in range(5000)}).encode()
def worker(pid,shared_str):
while 1:
t1 = time.time()
with lock:
new = eval(shared_str.value)
for addr in new:
for hostname in new[addr]:
new[addr][hostname] += 1
shared_str.value = str(new).encode()
t2 = time.time()
print('updated from worker', pid, 'took', float(t2-t1), new['<IP_ADDRESS>']['hostname-1'])
time.sleep(0.5)
for pid in range(40): # processes count
process = multiprocessing.Process(target=worker, args=(pid,shared_string))
process.daemon = True
process.start()
time.sleep(2)
last = 0
while 1:
stats = eval(shared_string.value)
print(stats['<IP_ADDRESS>'])
print('updates/sec:', stats['<IP_ADDRESS>']['hostname-1'] - last,
'total:', stats['<IP_ADDRESS>']['hostname-1'])
last = stats['<IP_ADDRESS>']['hostname-1']
time.sleep(1)`
output:
updates/sec: 24 total: 115
updated from worker 35 took 1.7005925178527832 117
updated from worker 36 took 1.6961350440979004 118
updated from worker 37 took 1.6979949474334717 119
updated from worker 38 took 1.6976847648620605 120
updated from worker 39 took 1.6979420185089111 121
updated from worker 0 took 1.697028636932373 122
updated from worker 1 took 1.695997714996338 123

for now my best is:
(not related to ultradict, but for performance comparison)
`import multiprocessing, time
from multiprocessing import RLock
import msgpack
import mmap
lock = RLock()
_data = msgpack.packb({
f'{i}.{i}.{i}.{i}': dict( (f'hostname-{x}', x) for x in range(1) ) for i in range(50000)})
shared_string = mmap.mmap(-1, len(_data)+100000)
shared_cursor = multiprocessing.RawValue('i', 0)
shared_string.write(_data)
shared_cursor.value = shared_string.tell()
def worker(pid,shared_str):
while 1:
t1 = time.time()
with lock:
msg_len = shared_cursor.value
shared_str.seek(0)
_data = shared_str.read(msg_len)
try:
new = msgpack.unpackb(_data, max_map_len=50000)
except Exception as error:
assert msg_len == len(_data)
print(error, 'msg len', msg_len, [_data])
else:
for addr in new:
for hostname in new[addr]:
new[addr][hostname] += 1
shared_str.seek(0)
packed = msgpack.packb(new)
wrote = shared_str.write(packed)
#print('wrote', wrote, 'cursor', shared_str.tell(), 'packed', len(packed))
shared_cursor.value = shared_str.tell()
t2 = time.time()
print('updated from worker', pid, 'took', float(t2-t1), new['<IP_ADDRESS>']['hostname-0'])
time.sleep(0.5)
for pid in range(40): # processes count
process = multiprocessing.Process(target=worker, args=(pid,shared_string))
process.daemon = True
process.start()
time.sleep(2)
last = 0
while 1:
'''stats = eval(shared_string.value)
print(stats['<IP_ADDRESS>'])
print('updates/sec:', stats['<IP_ADDRESS>']['hostname-1'] - last,
'total:', stats['<IP_ADDRESS>']['hostname-1'])
last = stats['<IP_ADDRESS>']['hostname-1']'''
time.sleep(1)
`
this code solves my task (because runs faster) - 50k dict got updated from 40 processes within a second with 2% total CPU usage, code itself took only 50ms per process, but with lock its about 1sec... how to do it without locking and separate dicts/etc?

Thanks for sharing the code. If I may, I'll use it as a future example and I'll try to make UltraDict fast for this use case.

@ronny-rentner Yes, of course you can
However, I was not able to achieve acceptable performance when using RLock's on the number of processes = 40 updating a large dictionary more than once a second (then working directly with mmap/shared memory without UltraDict)
Therefore, I abandoned the locks, instead I synchronize updates from main process, sending a signal to the child processes (after which the process writes updated dictionary to mmap), then, after the update is completed, the child process sends a signal to the main process, and everything repeats with the remaining processes (locks are not needed here because at one time only one process works with mmap due to synchronization from the main process)
(main process itself does not update the dictionary, but tell childs to update it one by one and then fetchs result from mmap)
With this approach, it is possible to update a dictionary of 50 thousand keys in size from 40 processes in 300 milliseconds of time in the main process, spending less than 5% of the total CPU time (on my configuration)
(300ms because it takes 7.5ms for a child process to update on such a dictionary size, and the main process just waits for the completion signal to pass the turn to the next child process)
those results achieved using marshal and mmap, python built-in packages

@rojamit
I've taken your explanations and came up with the following approach:
import multiprocessing, threading, time
import sys; sys.path.insert(0, '..')
from UltraDict import UltraDict
from collections import defaultdict

number_of_items = 50_000
update_interval = number_of_items * 10
number_of_processes = 4

def worker(pid, ultra):

    update_signal = threading.Event()
    updater_data = {}
    def updater():
        while update_signal.wait():
            with ultra.lock:
                ultra.apply_update()
                for k, v in updater_data.items():
                        ultra.data[k] += v
                ultra.dump()
            update_signal.clear()

            # only print for first process
            #if pid == 0: print(f'{pid} : update done')

    update_thread = threading.Thread(target=updater)
    update_thread.start()

    data = defaultdict(int)

    updated = 0
    t1 = time.perf_counter()
    while True:

        ##
        # Do some work, count something fast
        ##

        for i in range(number_of_items):

            data[i] += 1

            # Count number of updates to local dict
            updated += 1

        # If we have more than `update_interval` updates, then we send a signal to the
        # update thread to dump all the changes to the shared memory. We could also do
        # this time based every N seconds or based on another signal from the parent process.
        if updated > update_interval:
            t2 = time.perf_counter()
            #print(pid, 'inc:', (t2 - t1)*1000)

            # only print for first process
            #if pid == 0: print(pid, ':', int((updated / (t2 - t1))/1000), 'kupds/s', 'len: ', len(data))

            if update_signal.is_set():
                # The updater is not done yet, we cannot do another update, let's try next time
                # TODO: There's a risk we never get the chance to apply our updates
                continue

            # The updater thread will work on this copy of our data
            updater_data = data.copy()
            data = defaultdict(int)
            updated = 0

            update_signal.set()

            t1 = time.perf_counter()

t0 = time.perf_counter()

# Use static full dump memory for more write performance,
# 50 bytes per item (which is an int)
ultra = UltraDict(full_dump_size=number_of_items * 50)

for pid in range(number_of_processes):

    # This will make sure the value is `0` for non-existing keys
    ultra.data = defaultdict(int)

    process = multiprocessing.Process(target=worker, args=(pid, ultra), daemon=True)
    process.start()
        # ultra.data is direct access to underlying local dict

last = 0
while True:

    # print stats roughly every second
    time.sleep(1)

    stats = ultra
    print(stats[0])
    t1 = time.perf_counter()
    delta = stats[0] - last
    t_delta = t1 - t0
    speed = int(delta / t_delta * number_of_items / 1000)
    print('kupds/sec:', speed, 'total:', stats[0], 'len:', len(stats))
    last = stats[0]

    t0 = time.perf_counter()

The appraoch is how you have described it. Each process has a local dict where it is mainly working on. Once in a while, each process sends a signal to an updater thread. This thread then updates a central UltraDict that is shared among all processes. As we already know there are many updates we use a full dump technique in UltraDict so all the updates are only written together in one go.
The result:
622
kupds/sec: 30833 total: 622 len: 50000
1432
kupds/sec: 40280 total: 1432 len: 50000
2170
kupds/sec: 36700 total: 2170 len: 50000
2844
kupds/sec: 33485 total: 2844 len: 50000
3636
kupds/sec: 39449 total: 3636 len: 50000

Note that these are kilo updates per second, so with 4 processes I achieve like 40 million updates per second. With one process it's around 10 million updates. I do run this on a Lenovo X1 with an Intel Core i7 CPU. This means for 50k updates, it takes only 1.25 ms.

@ronny-rentner thats really fast! i'll try it asap

I am closing this issue due to inactivity. Feel free to reopen if the problem persists.


fiona / update paris' office physical location

Changes:

...

Type of change

[ ] Bug fix
[ ] New feature
[x] Update feature
[ ] Refactor code
[ ] Translation to code
[ ] Translation to crowdin
[ ] Script configuration
[ ] Improve performance
[ ] Style only
[ ] Dependency update
[ ] Documentation update
[ ] Release

Hey @fiona-deriv, map is currently breaking for widths > 1980px.

Hey @fiona-deriv, map is currently breaking for widths > 1980px. Could you please help to address it even though it's outside the scope?

Hey @sean-binary , investigated this and it's stemming from wrapping the image within LocalizedLink, with the current implementation, may take a little longer to refactor, and considering this a bit immediate, can we handle it as an improvement?

Why aren't we using the google maps iframe for the Paris address since we have that already? @fiona-deriv


Update create-wallet.md

The guidance provided by the foundry book informs us to setup our tests via
contract MyTest is Test {
...
}

However, given the inheritance and import structures of forge-std/Test.sol, the documentation to create a wallet produces the following error:
Compiler run failed:
Error (7920): Identifier not found or not unique.
  --> test/MyTest.t.sol:86:5:
   |
86 |     Wallet memory _w = vm.createWallet(100);
   |     ^^^^^^

Error:

Instead, in order to store the result returned by vm.createWallet, I had to reference the Wallet type through the Vm interface, like so:
Vm.Wallet memory _w = vm.createWallet(100);

@all-contributors please add @juannyG for updating the createWallet docs


Streamline Library Dependencies

Could have selected play-services-maps library to be more streamlined so will consider for future efficiency. At the moment, coming across less red lines would be very helpful.
In reference to the resolution for Issue #2

Need to double-check for any unneccessary dependencies installed, minor


Incorrect smart indentation before open brace

TypeScript Version:  master
Code
[
    {

    },
    // indentation expected: 4, actual: 0
    {

    }
]

It's 4 now


Refactor transaction generation logic

The current way that transactions are generated leaves scope for a stale, or wrong, transaction to be broadcast. As the generation is asynchronous, we have no guarantee the current tx atom represents the latest form values (though, as it's fast, it almost always is).
We should refactor this so transactions are generated when the form is submitted.

@markmhx For Monday mtg discussion while I'm out, I put up a draft PR to start work on this. Will return to it on Wed unless there is a higher priority issue!

This was merged into the ledger poc branch.


Typescript + modern JS private

Remove Babel for esbuild and vitest
Use strict Typescript
modernize JS code:

use es6 fast equal
# private members
# private methods
nullish coalescing

@sakgoyal I got distracted with tweaking other things and forgot to get your credit into the readme
I have one more update and will add your username when I push it
You may not even care about it, but I wanted to ensure you knew it was coming :)

no problem!


Question about the missing checkpoints.

Thanks for your great work and codes!
How can I get the checkpoints:wres50_lego.pth ?

Hello, wres50_lego.pth is defined and trained by yourself. Please check variable "ckp_path" in main.py .


Any bullet

Adjusted so I could use ✔ emoji 😁

It looks fun but as that's not standard Markdown, and as I haven't really seen anybody else with this need, I'm closing this PR. Thanks anyway though.


Fixed room backend

This is the right way to fix #128 and thus replaces #132.
Changes:

Entities should not be saved in the instances, because they cannot be interacted with beyond unit of works (and the separate thread is not in a unit of work)
Introduced a song instance which knows its song id and the current time. It implements an observer pattern to unschedule its own timer and to load the new song on completion.
Loading rooms in lifecycle listener rather than in constructor prevents Guice initialization errors
Room init should be retriggered manually after using the bootstrapper

Work to do:

Javadoc, checkstyle
Refactoring, responsibilities can be split up more across files
API endpoints should be tested, the front end compatibility as well
I broke the tests.

Looks very good.