Critical security vulnerability CVE-2023-36665 needs to be corrected by upgrading protobufjs library to 7.2.5 or higher within gax library dependency in package.json file.

(feywind edited for brevity)
Environment details

OS: node18-Alpine
Node.js version: 18
npm version:  10.7.0
@google-cloud/pubsub version: 4.3.3

Steps to reproduce
See vulnerability details in: https://github.com/advisories/GHSA-h755-8qp9-cq85
protobuf.js (aka protobufjs) 6.10.0 until 6.11.4 and 7.0.0 until 7.2.4 allows Prototype Pollution, a different vulnerability than CVE-2022-25878. A user-controlled protobuf message can be used by an attacker to pollute the prototype of Object.prototype by adding and overwriting its data and functions. Exploitation can involve: (1) using the function parse to parse protobuf messages on the fly, (2) loading .proto files by using load/loadSync functions, or (3) providing untrusted input to the functions ReflectionObject.setParsedOption and util.setProperty. NOTE: this CVE Record is about Object.constructor.prototype. = ...; whereas CVE-2022-25878 was about Object.proto. = ...; instead.
Attack scenario
An external attacker for exposed endpoints.

Pub/Sub includes gax 4.0.3, which already pulls in protobufjs 7.2.4 (probably by semver). Renovate isn't issuing PRs to upgrade to newer gax versions, and there's a newer one that explicitly pulls in 7.2.4, so that'd be good to have.


BUG: UI Improvement in the feedback form frontend

Is there an existing issue for this?

[X] I have searched the existing issues

What went down?
We can improve the way the feedback form looks with simple tailwind and no extra library, so it's not a bug but an improvement.
It would reduce the space the feedback form takes and make it look more professional.

Please assign it to me. :)
If any other field is required, I can add that as well.

hacktoberfest

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a gssoc-24-extd contributor
[X] I want to work on this issue
[X] I check the issue doesn't exist

assigned @vishnuprasad2004
also you can create multiple issues at a time

solved #75


[Spire - The City Must Fall] Allow translations

Hi.
I don't know if @Medieve plans to update this sheet so that it can be translated. If not, is there any way to translate it? I'm not a programmer (I only know how to translate the sheets) and I don't have a Pro account, this is why I'm asking.
Thanks

I didn't have current plans to put in translation tags but since you asked nicely I can set out some in the next few days.

Submitted a translation file pull
#7028 https://github.com/Roll20/roll20-character-sheets/pull/7028

That's amazing! Thank you very much @Medieve

@WallaceMcGregor if this have been implemented, you could close this issue.


Remove accidentally added print statment

Removing a print statment that I think accidentally got included

This line has already been removed on main :)


Assets are not copied to the android project when install android build template.

Godot version:
Godot v3.2.4 beta6
OS/device including version:
Android / Galaxy S21+
Issue description:
I'm using the Android build template to use subview, but the build template has some problems.
First of all, it does not copy the configuration of signing, so it it required (minor).
And, after that, assets are not copied (or not linked), so when I build with android build template, it shows,
2021-03-25 19:41:55.452 27967-28005/com.godot.game E/godot: **ERROR**: Couldn't load file 'res://project.binary', error code 19.
2021-03-25 19:41:55.452 27967-28005/com.godot.game E/godot:    At: core/project_settings.cpp:621:_load_settings_text_or_binary() - Couldn't load file 'res://project.binary', error code 19.

So, I just check the apk file built with godot editor, and I found assets/ directory, so I copied that directory to the project.
After that, it works fine.
Steps to reproduce:

fork project https://github.com/uheartbeast/youtube-tutorials
load project
install android build template
load android project (generated by build template)
build / run

Minimal reproduction project:

The Android build template is only supposed to be used by the Godot editor.
A separate set of instructions will be provided when Godot 3.3 is stable for how to create a Android project that uses Godot subview.


Add OpenGraph and Meta tags

We should add OpenGraph and Meta Tags to the pages.
This package could be used
https://www.npmjs.com/package/react-meta-tags

I'm not sure if that's already implemented or not but we have seoTitle and seoDescription fields available for that purpose. They are available for the following models: Product, Category, Collection.

Related issue - #169 - could be done as one PR.


setup pingdom to run monitor status page nightly

The current production site has a pingdom script to load the page after 2:00am.  This load will fail, but starts the process of running all tests.  The script loads the page again after an hour (maybe hourly).  This load and all other loads on that day should pass.
The script can be the same used for the current production site.

Yes, Pingdom runs every hour. This isn't efficient and puts undue strain on the service because it's forced to run all tests every hour, so we should come up with a better system eventually, but that's another issue.

It does not run the tests every hour.  It loads the page every hour.  The application only runs the tests once a day unless given a token that forces the tests to run again.  The results are cached.  Every page load after the test run uses the cached values.

I've created a new Pingdom monitor pointed at qa-server-service.library.cornell.edu which will be the non-public service URL. That will work now, and should continue to work post-go-live. At that point, the old Pingdom monitor will also point to the new service (at the old address) and we can disable one of them.

I received a message from Pingdom that the monitor status page is down, and it is.  So it correctly reported the status.  I am going to leave this open until the status page is fixed and we can confirm that Pingdom correctly detects that the page is working.  Probably no more work to be done on this issue.

Close once this is confirmed to be working when the switch over is complete.


Players named "." delete all players when deleted

Not sure if this is a tModLoader bug or a vanilla bug, but it's been reported many times.
Make a player named "."
Delete it
All player files are gone from Players folder and even recycling bin.
Needs Confirmation.

Did we want to fix it in tML though?

No, as said fixed in vanilla due next update

Seems to be the filename, players are saved as PLAYER_NAME.plr, this player is ., so this is saved as ..plr, .. means parent directory, which means the parent directory with plr added on the end, which is: TERRARIA_TMODLOADER_DIR\plr
So it's deleting TERRARIA_TMODLOADER_DIR\plr! Problem solved!


EMQ V2.1 crash when public with QoS = 2

Environment

OS: Windows 7
Erlang/OTP:
EMQ: 2.1-beta

Description
I downloaded EMQ 2.1-beta binary package for Windows 7, then i use MQTTLens to test on my localhost
With public messages have QoS = 0 or 1 the broker works well, but when i public a message with QoS = 2 the connection is lost, i then install EMQ 2.0.7, it works well with all QoS level
In error log file:
2017-02-24 08:42:45.220 [error] <0.1062.0> gen_server <0.1062.0> terminated with reason: call to undefined function maps:take(10721, #{10721 => {mqtt_message,<<0,5,73,60,216,166,115,240,112,5,0,0,4,37,0,3>>,10721,{<<"lens_Kb0tX3S...">>,...},...}})
2017-02-24 08:42:45.220 [error] <0.1062.0> CRASH REPORT Process <0.1062.0> with 1 neighbours exited with reason: call to undefined function maps:take(10721, #{10721 => {mqtt_message,<<0,5,73,60,216,166,115,240,112,5,0,0,4,37,0,3>>,10721,{<<"lens_Kb0tX3S...">>,...},...}}) in gen_server2:terminate/3 line 1157
2017-02-24 08:42:45.221 [error] <0.242.0> Supervisor emqttd_session_sup had child session started with {emqttd_session,start_link,undefined} at <0.1062.0> exit with reason call to undefined function maps:take(10721, #{10721 => {mqtt_message,<<0,5,73,60,216,166,115,240,112,5,0,0,4,37,0,3>>,10721,{<<"lens_Kb0tX3S...">>,...},...}}) in context child_terminated
2017-02-24 08:42:45.221 [error] <0.295.0> Supervisor 'esockd_connection_sup - <0.295.0>' had child connection started with emqttd_client:start_link([{client_idle_timeout,30000},{client_enable_stats,false},{max_clientid_len,1024},{max_packet_size,...}]) at <0.1061.0> exit with reason call to undefined function maps:take(10721, #{10721 => {mqtt_message,<<0,5,73,60,216,166,115,240,112,5,0,0,4,37,0,3>>,10721,{<<"lens_Kb0tX3S...">>,...},...}}) in context connection_crashed
A description of the issue

I'm not on a windows machine to check, can you paste the header from emqttd console?  It should show erlang/otp version, among other things. It should be > 19.0, since maps:take was introduced then. Lower versions will cause the crash you reported.

Thank phanimahesh, my Erlang/OTP version is 18 and it make broker crash

You mentioned binaries, did you mean you downloaded precompiled version from github, and haven't built it yourself?
If the windows package is built with 18, it should get a rebuild. /cc @emqplus

@phanimahesh : yes, i downloaded binary package for windows, and it contains erts-7.3

@quanfehut The issues is caused by "maps:take/2" API, which is only available in Erlang/OTP R19+ release. We upgraded the OTP and fixed the issue in 2.1.0-beta.1.


Allow numba functions within zip files to be cached

Numba doesn't currently allow .zip archives to contain numba functions when caching is enabled. This is all but required when using Spark.
This change enables that, by recognizing a zip file and using a user-wide cache directory for the cache. It closes (most of?) https://github.com/numba/numba/issues/4908#issuecomment-1161476166
I'm the maintainer of numbagg, but haven't contributed directly to numba, so please let me know if the contribution needs any work, happy to make changes.
FWIW — hoping to be helpful — I found it not super easy to use the repo — I notice it doesn't use pytest / code formatters / etc. That does make contributions a bit harder for newcomers, though I'm sure it's for good reason...

Thanks for the PR! I'm just marking it as "2 - In progress" while we wait for CI.

FWIW — hoping to be helpful — I found it not super easy to use the repo — I notice it doesn't use pytest / code formatters / etc. That does make contributions a bit harder for newcomers, though I'm sure it's for good reason...

Thanks for the comments, I do think they're helpful. It doesn't use pytest because there's a lot of homegrown addons to unittest that have been added over the years, and porting them over to pytest wasn't trivial (there was an attempt some years ago but I believe it stalled).
WRT to code formatting, we only check for flake8 compliance (with a large number of files excluded as they were historically never checked and they never got fixed up yet). I think there's also more that could be done to make things a smoother experience here, but it's been difficult to move forward with relative to other items.

From a quick skim it looks like the tests aren't platform-agnostic WRT the way paths are formed (the root, and separators) - I haven't looked deeper, but perhaps pathlib.Path could be used to make the form of the tests platform-independent?

From a quick skim it looks like the tests aren't platform-agnostic WRT the way paths are formed (the root, and separators) - I haven't looked deeper, but perhaps pathlib.Path could be used to make the form of the tests platform-independent?

Thanks, fixed

It doesn't use pytest because there's a lot of homegrown addons to unittest that have been added over the years, and porting them over to pytest wasn't trivial (there was an attempt some years ago but I believe it stalled).

Yes totally re homegrown addons...
I would say that pytest is much much nicer, both from the perspective of maintainers being able to design better test suites, and the perspective of contributors who are running tests — for example I couldn't seem to enter pdb on a test failure with the current setup? Transitioning probably isn't trivial but also probably not herculean — pytest can collect & run standard unittests, and running two test commands for a while could be a way to get moving.

WRT to code formatting, we only check for flake8 compliance (with a large number of files excluded as they were historically never checked and they never got fixed up yet). I think there's also more that could be done to make things a smoother experience here, but it's been difficult to move forward with relative to other items.

OK, you might want to check out ruff for linting; it fixes most failures itself. For formatting: we moved to black in a big bang in xarray, was very successful. The one-off merge conflicts are a bit annoying, but tractable ("merge up to the formatting commit, format with black, merge in the formatting commit").

The pytest transition probably requires some expertise but if you wanted a hand with linting & formatting, happy to help.

This is ready for review!

This is ready for review!

Oops, sorry, I forgot to check back when I was waiting for CI the other day.

Hi team — the gentlest of pings here. I know things can get backlogged. I do think this is a fairly small unambiguous improvement, with some decent comments and a great test. If there's anything I can do to reduce the reviewing burden, please lmk.

Sorry! I'll review it today

Sorry, I wrongly pushed to your branch an empty commit

No prob re empty commit, push away.
Docs error seems unrelated though?
numba/_helperlib.c:22:44: error: ‘I’ undeclared (first use in this function)
   22 |     #define _complex_float_ctor(r, i) (r + I * i)
      |                                            ^
numba/_helperlib.c:147:12: note: in expansion of macro ‘_complex_float_ctor’
  147 |     *out = _complex_float_ctor((float) _out.real, (float) _out.imag);
      |            ^~~~~~~~~~~~~~~~~~~
numba/_helperlib.c:22:44: note: each undeclared identifier is reported only once for each function it appears in
   22 |     #define _complex_float_ctor(r, i) (r + I * i)
      |                                            ^
numba/_helperlib.c:147:12: note: in expansion of macro ‘_complex_float_ctor’
  147 |     *out = _complex_float_ctor((float) _out.real, (float) _out.imag);
      |            ^~~~~~~~~~~~~~~~~~~
error: command '/usr/bin/gcc' failed with exit code 1

Docs error seems unrelated though?

Yes! I guess something must have changed on conda-forge.

I think that fixing the docs issue requires merging numba:main into this branch so that #9675 is picked up.

Thanks a lot for the thorough review @guilhermeleobas. I've made those changes and it should be ready for merging / another review.

closed/reopened your PR to run CI again

I fixed some flake8 issues. On the topic of dev friendliness:

pre-commit uses a different version of flake8 which doesn't work (at least on my system)
flake8 generates lots of errors, because the command that runs in CI isn't just flake8

I managed to work this out in a few mins but less easy than other repos I contribute to! (hope this is helpful)

This is a bit off-topic, but anyway:
I understand that the lack of pytest use and a realistic approach to formatting and linting can be quite frustrating as I experience the same. These topics have been discussed repeatedly over the years but there is still no solution in place. I think that is evidence that the tasks are significantly harder than they may appear.
From memory, pytest can't be used with Numba as it messes with the bytecode to such an extent that tests start to become unreliable. Numba uses the bytecode as the input to the compiler pipeline, so messing with that was a no-go, not sure if that is still the case. Basically someone would need to take this one, investigate if pytest could be used and port the code these days. No small feat.
As for the formatting, any PRs that auto-format are usually quite large and contain many false positives, so the usually just go through multiple rounds of reviews and eventually just become stale and are closed -- as this is not the most exciting task and contributors tend to disengage. Also, auto-formatting (and spelling corrections) will often break stuff, so it needs to be done very, very carefully and each lines needs to be reviewed in earnest. Also, any large scale reformatting PRs tend to break in-flight PRs so that causes additional strain.
If someone were really inclined to help out on this, I think I would be realistically willing to accept about one PR on this topic per week with no more than a handful of changes (maybe 20-30 LOC?) on a single file. This would need to be reviewed very carefully and extensively tested.
One would have to take the files from the flake8 exclude list:
https://github.com/numba/numba/blob/main/.flake8
Currently that's ~ 230 files or so and so I would estimate the time taken to complete the task -- at a rate of one file per week -- of making the rest of the codebase flake8 complaint will be about 4 to 5 years (with holidays, release priorities, other random incidents...) -- so perhaps "herculean" is actually the correct adjective here 😇 ? I'm not saying it's impossible, just very tedious and laborious and boring...
Since this is quite off-topic, perhaps it makes sense to move this discussion to a new issue (or a discourse discussion) or find an existing issue where this has been discussed?

I fixed some flake8 issues. On the topic of dev friendliness:

pre-commit uses a different version of flake8 which doesn't work (at least on my system)
flake8 generates lots of errors, because the command that runs in CI isn't just flake8

I managed to work this out in a few mins but less easy than other repos I contribute to! (hope this is helpful)

Here, this could help:
https://github.com/numba/numba/pull/9701
But, this gives new flake8 errors:
 💣 zsh» pre-commit run --verbose --all-files
[INFO] Initializing environment for https://github.com/PyCQA/flake8.
[INFO] Installing environment for https://github.com/PyCQA/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
flake8...................................................................Failed
- hook id: flake8
- duration: 3.41s
- exit code: 1

docs/dagmap/render.py:12:14: E128 continuation line under-indented for visual indent
docs/dagmap/render.py:18:1: W293 blank line contains whitespace
docs/dagmap/render.py:35:9: E128 continuation line under-indented for visual indent
docs/dagmap/render.py:37:81: E501 line too long (84 > 80 characters)
docs/dagmap/render.py:39:81: E501 line too long (110 > 80 characters)
docs/dagmap/render.py:62:81: E501 line too long (84 > 80 characters)
docs/dagmap/render.py:63:81: E501 line too long (100 > 80 characters)
docs/dagmap/render.py:73:81: E501 line too long (82 > 80 characters)
docs/source/developer/compiler_pass_example.py:78:1: E305 expected 2 blank lines after class or function definition, found 1
docs/source/developer/inline_example.py:2:1: F401 'numba' imported but unused
docs/source/developer/inline_overload_example.py:1:1: F401 'numba' imported but unused
docs/source/extending/template.py:6:35: E999 SyntaxError: invalid syntax
docs/source/proposals/np-where-override.py:6:1: E302 expected 2 blank lines, found 1
maint/towncrier_checker.py:49:26: F541 f-string is missing placeholders
maint/towncrier_rst_validator.py:6:7: E225 missing whitespace around operator
maint/towncrier_rst_validator.py:11:5: E225 missing whitespace around operator
maint/towncrier_rst_validator.py:30:1: E302 expected 2 blank lines, found 1
maint/towncrier_rst_validator.py:33:36: E225 missing whitespace around operator
maint/towncrier_rst_validator.py:33:65: W291 trailing whitespace
maint/towncrier_rst_validator.py:42:58: W291 trailing whitespace
maint/towncrier_rst_validator.py:43:21: E128 continuation line under-indented for visual indent
maint/towncrier_rst_validator.py:49:57: W291 trailing whitespace
maint/towncrier_rst_validator.py:50:81: E501 line too long (85 > 80 characters)
maint/towncrier_rst_validator.py:51:81: E501 line too long (89 > 80 characters)
maint/towncrier_rst_validator.py:57:1: E305 expected 2 blank lines after class or function definition, found 1
maint/towncrier_rst_validator.py:59:7: F541 f-string is missing placeholders
maint/towncrier_rst_validator.py:68:54: W291 trailing whitespace
maint/towncrier_rst_validator.py:84:7: F541 f-string is missing placeholders
maint/towncrier_rst_validator.py:86:7: F541 f-string is missing placeholders
maint/towncrier_rst_validator.py:97:81: E501 line too long (85 > 80 characters)
maint/towncrier_rst_validator.py:98:81: E501 line too long (91 > 80 characters)
maint/towncrier_rst_validator.py:104:11: F541 f-string is missing placeholders
maint/towncrier_rst_validator.py:106:7: F541 f-string is missing placeholders
maint/towncrier_rst_validator.py:110:7: F541 f-string is missing placeholders

pre-commit run --verbose --all-files  15.72s user 1.52s system 187% cpu 9.172 total

But, this gives new flake8 errors:

I think CI runs flake8 numba so this wouldn't usually be an issue - all the errors are from files outside the numba directory.

But, this gives new flake8 errors:

I think CI runs flake8 numba so this wouldn't usually be an issue - all the errors are from files outside the numba directory.

Configs should generally be in config files rather than in CI scripts, so running flake8 alone works... I'm guessing there is a flake8 config that allows specifying which files to check...

As for the formatting, any PRs that auto-format are usually quite large and contain many false positives, so the usually just go through multiple rounds of reviews and eventually just become stale and are closed -- as this is not the most exciting task and contributors tend to disengage. Also, auto-formatting (and spelling corrections) will often break stuff, so it needs to be done very, very carefully and each lines needs to be reviewed in earnest. Also, any large scale reformatting PRs tend to break in-flight PRs so that causes additional strain.

To clarify, is this a claim that running black over the code would change its functionality?
That would be very surprising to me, and would be a bug in black.
Assuming it didn't change its functionality, would you support running black over all the code?

(I moved the dev tools discussions to other issues. This PR is ready for a second review — thanks!)

@sklam, are you happy with the current state or would like to review this one as well? If not, we can merge it.

@max-sixty, can you add a news fragment for this PR? It will then be part of the release note. See https://github.com/numba/numba/tree/main/docs/upcoming_changes for README and examples.

Great, added!

/azp run

Done!

No urgency but it states still Waiting on Author — I think it's now ready for you to merge — lmk if that's not correct

Anything else needed guys?

Anything else needed guys?

No, it's fine, just some delays here due to Anaconda-internal CI upgrades taking longer than expected. Hopefully we'll be able to merge this in the next few weeks. I know it's suboptimal, but please be patient, it will get merged eventually.

Thank you @esc !

Thank you @esc !

Apologies for the delay. @sklam made some progress on the recent 3.12 issue our test-suite was hitting:
https://github.com/numba/numba/pull/9727
so with a little luck this PR will be merged early next week.

thank you for the patch


create separate jest test suites

[x] synchronous
[x] async
[x] performance - not to be included in all (coverage) tests
[x] all - with coverage

CLOSED -> reduced runtime of sync test suite from ~30 secs to ~3 secs.


How to make entire TreeItemRenderer draggable?

Hi,
First, thanks for doing this. Vue tree dnd components are rare and usually years since their last update.
Quick question though - I'm looking at the demo and trying to drag via the name rather than the icon. Is that possible?

Ignore me, it is.


Adding diagnostic logging

Adding logging to all of the sensors and some of the other components to make
it easier to see what's going on while GreenPiThumb runs.

Coverage decreased (-3.3%) to 82.819% when pulling 6b787b4d626fc762f7b49e2222b84ee049c99daf on mtlynch:add-logging into 56b1dab8d1273ce9218b16bc14b6cc5dbf582da9 on JeetShetty:master.

Coverage decreased (-3.2%) to 82.95% when pulling af73721c907b50e7efd1bb9e87427ba2d468eaf9 on mtlynch:add-logging into 56b1dab8d1273ce9218b16bc14b6cc5dbf582da9 on JeetShetty:master.

Coverage decreased (-3.2%) to 83.333% when pulling 3d1234b801b47e6cf902be08a2848b41751eed59 on mtlynch:add-logging into 573694c3a6c571878fd9acbdde207bc377708971 on JeetShetty:master.


.NET 3.5 SP1 Missing

Information on the System Requirements/supported platforms for .NET 3.5 SP1 is missing.  The Lifecycle FAQ—.NET Framework https://support.microsoft.com/en-us/help/17455/lifecycle-faq-net-framework says "For a list of supported operating systems for .NET Framework 3.5 SP1 refer to the .NET Framework system requirements." and links to https://docs.microsoft.com/en-us/dotnet/framework/get-started/system-requirements

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8afd1f96-b008-8b2e-bf3e-89d91f8cca5e
Version Independent ID: 2010bd32-59f6-823c-f193-27439e12798d
Content: .NET Framework system requirements
Content Source: docs/framework/get-started/system-requirements.md
Service: dotnet-framework
GitHub Login: @rpetrusha
Microsoft Alias: ronpet

+1

We can only install dotnet framework 4.6.2 to win7 but we can install dotnet framework 4.7.2 in win7 sp1.

Thanks for your feedback, @JaredShaver. I apologize that it's taken so long to respond.
The System Requirements page is designed to provide information about .NET Framework 4.5 and later versions only. The .NET Framework versions and dependencies article provides information about all previous .NET Framework versions.  We've opened #7977 to add a cross-reference to the versions and dependencies article from the system requirement article.


Fix to not wrap API routes with superjson!

This is a bad bug 😬
Related: https://github.com/blitz-js/blitz/issues/1819

Published in https://github.com/blitz-js/babel-plugin-superjson-next/releases/tag/v0.2.1

Published in https://github.com/blitz-js/babel-plugin-superjson-next/releases/tag/v0.2.1


Fix issue #6, enable gracefully shutdown for PMS and ADS

Make two channels, one for Interrupt Signal, and the other one for error handling.

The change looks OK to me now.


requirements-extras.txt: Remove zcbor requirement

Add instructions to install it when needed to the lwm2m README.

 Compliance checks are not failing, hence this is somehow going through. However, requirements-extras.txt is not an area label.
I would have expected something like
Scripts: remove zcbor from req.txt.
LWM2M: update readme.
https://docs.zephyrproject.org/latest/contribute/guidelines.html#commit-message-guidelines


sysctl erring during boot, when set through config.yaml

Version - v0.2.1-rc1

Create a config.yaml with

k3os:
  sysctl:
    kernel.panic: 60
  password: asdf

During boot process, press e to get into GNU Grub
On linux cmdline add k3os.mode=install k3os.install.config_url="config yaml url"
Try to log in.

results:  Won't let you cause the sysctl field is erring on boot.  "ERRO[0000] json: cannot unmarshal nuber into Go struct field K3OS.sysctls of type string"

putting double quotes around kernel.kptr_restrict value solved this issue for me.
  sysctls:
    kernel.printk: 4 4 1 7
    kernel.kptr_restrict: "1"


How to use irregular image pyramids? IIIF

My underlying images are irregular image pyramids. E.g., a typical image has following IIIF structure:
{
"profile" : ["http://iiif.io/api/image/2/level2.json", {
    "supports": ["canonicalLinkHeader", "profileLinkHeader", "mirroring", "rotationArbitrary", "sizeAboveFull", "regionSquare"],
    "qualities" : ["default", "bitonal", "gray", "color"],
    "formats" : ["jpg", "png", "gif", "webp"]
}],
"tiles" : [{
    "width" : 512,
    "scaleFactors" : [1, 4.0001689138501408, 16.002226468764611]
    }],
"protocol" : "http://iiif.io/api/image",
"sizes" : [{
    "width" : 61917,
    "height" : 43923
    }, {
    "width" : 15479,
    "height" : 10980
    }, {
    "width" : 3869,
    "height" : 2745
    }],
"height" : 43923,
"width" : 61917,
"@context" : "http://iiif.io/api/image/2/context.json",
"@id" : "http://imageserver/image.jp2"
}

The scaling is not smooth at 1, 4, 16, but 1, 4.0001689138501408, 16.002226468764611.
Still, OSD will as me for tiles of a regular pyramid with sizes at the power of 2. Typical requests are:
/imageserver/image.jp2/49152,40960,8192,2963/512,/0/default.jpg
/imageserver/image.jp2/0,32768,8192,8192/512,/0/default.jpg
/imageserver/image.jp2/0,0,8192,8192/512,/0/default.jpg
/imageserver/image.jp2/0,40960,8192,2963/512,/0/default.jpg
/imageserver/image.jp2/57344,0,4573,8192/286,/0/default.jpg
/imageserver/image.jp2/57344,40960,4573,2963/286,/0/default.jpg
/imageserver/image.jp2/24576,16384,4096,4096/512,/0/default.jpg
/imageserver/image.jp2/20480,16384,4096,4096/512,/0/default.jpg
/imageserver/image.jp2/40960,26624,2048,2048/512,/0/default.jpg

resulting in a lot of rescaling work since e.g. when OSD asks for a 8192 wide region with a 512 size, the downsample is exactly 16.
The "best level for downsample" is in this case the orginal level 1 (with downsample 4.000169) which is then of course way too large and has to be downscaled.
Level 2 is not suitable, since it has the downscale factor 16.0022, and this would lead in a slightly to small patch which would have to be upscaled.
In any case, downscaling or upscaling is involved, slowing the image server down. Therefore, I have two related questions:

Why is OSD asking for a 8192 sized tile (downsample 16), and not for a 8193 sized tile (downsample 16.0020) which would better fit the provided scaling factor?

What is your recommendation for this kind of irregular image pyramids?

Thanks for your thoughts!

Any thoughts, @jpstroop @azaroth42?

It's not a valid IIIF Image API implementation, as scalefactors are required to be integers, not floats.  There is a feature request for 3.0 that the OP should submit their use cases to:  https://github.com/IIIF/api/issues/447  The issue is currently closed, as we lack use cases to support its inclusion.

Ok, even if this is not valid IIIF, some scanners scan at irregular pyramids (non-exact powers of 2). So what would be the best way to deal with that? I think it would be a large performance gain if OSD (or IIIF) could natively deal with that.

Is it just a pyramid or is it also tiled? If it's just an irregular pyramid, the LegacyTileSource is what you're looking for:
http://openseadragon.github.io/examples/tilesource-legacy/

It is also tiled. The image is unfortunaley too large to be in RAM in one piece.

Cool. Well, I don't think we have anything that supports that scenario yet, but the LegacyTileSource is probably a good starting point if you wanted to build a new tile source (or extend that one) to support irregular tiled pyramids.

Thanks, @iangilman, I was thinking the same, and wanted to have a close look at that.
But I have a question:
Internally, OSD always works with a full deepzoom pyramid with powers of two, regardless of the tilesource, right?
So, there is no chance for the tilesource to support irregular pyramids, since this would have to be implemented deeper in the OSD code. Otherwise, the tile would only serve as a translator between native pyramid from the imagefile and OSD query of a tile that it wants to put to a specific coordinate at a specific (power-of-two-)level. right?
I aim to skip this step of power-of-two-levels: If my imagefile has three levels, 1/1, 1/4 and 1/16 (but not 1/2 and not 1/8), then I want OSD two "fall back" to the next level only at those three zoom levels (and not inbetween already at e.g. 1/2, which triggers the artificial generation of an 1/2 tile out of the native 1/1 level).
Do you know which part of OSD would have to be altered or where should I look, to implement this behavior?

I actually don't know if that's the case. The LegacyTileSource certainly allows you to give it non-power-of-two tiles. I don't know if it's somehow translating them to powers of two internally. A quick look at the code for it hasn't really enlightened me; have you taken a look at it?
At any rate, the other side to look from TiledImage._updateViewport, and see how it handles levels.


[SPARK-16088][SPARKR] update setJobGroup, cancelJobGroup, clearJobGroup

What changes were proposed in this pull request?
Updated setJobGroup, cancelJobGroup, clearJobGroup to not require sc/SparkContext as parameter.
Also updated roxygen2 doc and R programming guide on deprecations.
How was this patch tested?
unit tests

@shivaram

Test build #61017 has started for PR 13838 at commit c3be3e8.

Test build #61017 has finished for PR 13838 at commit c3be3e8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61017/
Test PASSed.

Merged build finished. Test PASSed.

Test build #61060 has started for PR 13838 at commit 43df023.

Test build #61060 has finished for PR 13838 at commit 43df023.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61060/
Test PASSed.

Merged build finished. Test PASSed.

LGTM. Merging this to master, branch-2.0


userCanResolve prop

adding this prop for some more control over users being able to resolve a conversation

@kyleharper good point! changed to is-read-only


SUB to stats socket not working

The demo Python program (pasted below) that connects to the stats socket and prints the decoded messages is stuck at the sock.recv() and does not print anything, although I can successfully have pushpin play along with clients and origin server. The permissions on the socket look good (srwxr-xr-x 1 pushpin zurl ). Am I missing anything? Thanks!
import sys
import tnetstring
import zmq

ctx = zmq.Context()
sock = ctx.socket(zmq.SUB)
sock.connect('ipc://var/run/pushpin/pushpin-stats')
sock.setsockopt(zmq.SUBSCRIBE, '')

while True:
    m_raw = sock.recv()
    mtype, mdata = m_raw.split(' ', 1)
    if mdata[0] != 'T':
        print 'unsupported format'
        continue
    m = tnetstring.loads(mdata[1:])
    print '%s %s' % (mtype, m)

I believe zmq ipc requires writability regardless of which direction the data is logically flowing. Try 777 perms and see if that helps.

Done, the perms are now srwxrwxrwx, but it still doesn't work. I also restarted pushpin, just in case, to no avail.

The filename should probably be ipc:///var/run/pushpin/pushpin-stats (note the triple slash), otherwise it would be a relative path from where you run this code.

Good point: it might be wise to add a note or change the code at http://pushpin.org/docs/advanced/#stats-socket. Alas, it still doesn't work.

Thanks for pointing out that typo. Fixed.
The only other thing I can think of is a ZeroMQ versioning issue. What OS are you running? Did you install the python zmq package from the OS or pip?

I'm on Ubuntu 14.04 3.13.0-106-generic, Python 2.7.6 [GCC 4.8.2] in virtualenv-13.1.2. Uhm... could virtualenv be the issue?

When you install zmq via pip, it'll build its own copy of zmq if it can't find the dev files of your system zmq. This can lead to mismatching versions used between your programs in the virtualenv and programs on the system (such as Pushpin, assuming you're using the Ubuntu package). Try sudo apt-get install libzmq3-dev so the dev files exist on the system, and then pip uninstall/install zmq within the virtualenv again.

libzmq3-dev was already installed, I reinstalled pyzmq (version 16.0.2) but I still get nothing out of the socket.

Can you install the python-zmq Ubuntu package and run the code without the virtualenv to see if that works?

Also note if you restart Pushpin then the permissions on the pipe file may revert, so be sure to chmod again. Or set ipc_file_mode in pushpin.conf

The virtualenv-free install still did not work, but setting again the permissions did the trick! I re-tested it within the virtualenv and it's all working fine. As I installed libzmq3-dev long ago, the most likely issue was simply the triple-slash. Thank you!

Whew, glad we figured it out. :)


Program scrolling up automatically

Hi, here is my problem i'm adding text via websocket but the text appears at the bottom of the window and the program is scrolling up automatically so when i try to scroll down to see the new lines of text it will automatically scroll up again so i can't check the text

i'm adding text via websocket but the text appears at the bottom of the window

What do you mean by that? The new text copied from the WebSocket shouldn't appear at the bottom of earlier texts. Do you mean the new text is quite long and so it doesn't fit the main window, so you need to scroll down to look up a word that appears in the later section of the text? Or are you perhaps entering the backlog mode mistakenly?

the program is scrolling up automatically so when i try to scroll down to see the new lines of text it will automatically scroll up again

I can't reproduce this behavior. Are you using a device with a touchscreen, or are you scrolling it down with your mouse wheel? Does it automatically scroll up when a popup appears due to an unintended lookup? Can you reproduce the problem when you scroll up when using the scrollbar directly?
I'd appreciate a video demonstrating the problem.

Hi, thanks for the answer, here is the video demonstrating my problem : [
https://github.com/rampaa/JL/assets/107620399/43b554e7-9aa6-4d32-9c66-6e7947cdf3f8
](url)

Thanks for the video. If I'm interpreting the video right, you are using a Speech-To-Text program that generates Japanese text from Japanese audio. And it seems like it is doing the text generation in real-time instead of generating the sentences beforehand, so it constantly re-copies the whole text as it generates more of it like: "人の世の見つけぬ花や軒の栗... はるか山陰", "人の世の見つけぬ花や軒の栗... はるか山陰の"...  "人の世の見つけぬ花や軒の栗... はるか山陰の小里に石なかば土にうずもれてあり。". If my understanding is correct, then JL is behaving as expected.
You may still want to try the following though:

See if the Speech-to-Text tool you use allow you to generate Japanese texts for the audio file all at once beforehand. If it does, it may be able to copy the whole "はるか山陰の小里に石なかば土にうずもれてあり。" sentence at once when the voice starts to say "はる", in which case everything will work as expected.

If none of those is possible/realistic for your use case, I can look into whether I can add an option to make JL play better with your use case, though I am not sure if I can make the experience seamless, ideally this should be fixed by the Speech-To-Text program.

Added two options for your use case. Since I don't know what Speech-To-Text tool you use, I couldn't test it with it so I don't know if it will work as expected. The new options will be included in the next version of JL. When it comes out, you can enable it through "Preferences->Main Window->Merge sequential texts when they match", you can also fiddle with the value of "Preferences->Main Window->Only merge sequential texts when they match within milliseconds". When the new version of JL is out, please let me know whether or not the new options solve your problem.

Thanks you so much i love you
Le dim. 2 juin 2024 à 17:49, rampaa @.***> a écrit :

Added two options for your use case. Since I don't know what
Speech-To-Text tool you use, I couldn't test it with it so I don't know if
it will work as expected. The new options will be included in the next
version of JL. When it comes out, you can enable it through
"Preferences->Main Window->Merge sequential texts when they match", you can
also fiddle with the value of "Preferences->Main Window->Only merge
sequential texts when they match within milliseconds". When the new version
of JL is out, please let me know whether or not the new options solve your
problem.
—
Reply to this email directly, view it on GitHub
https://github.com/rampaa/JL/issues/89#issuecomment-2143908882, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AZVCQL5VDAIKZGYLLA54TFDZFM5J7AVCNFSM6AAAAABITXSKP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTHEYDQOBYGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

FYI: The new version of JL is out. Try enabling the "Preferences->Main Window->Merge sequential texts when they match" option and let me know how it goes.

https://github.com/rampaa/JL/assets/107620399/0a85e5c6-6319-44b6-9ef5-0188ea3806da
Hi, it's me again, I just tried the update and I wanted to thanks you !!! Everything works perfectly now i can watch live videos and use the pop up dictionnary on them, it's perfect
Thanks you so much


Support Python bodies for "short block"

It would be nice if in the following:
def aaa():
    bbb;
    ccc;

    ddd;

Saying "short block bat" referred to
    bbb;
    ccc;

We could probably do it by creating a version of SurroundingPairInteriorScopeHandler that looks not just for surrounding pair interiors but also interiors of any tree-sitter scopes that have an interior
Although there is a case to be made that what "short block" really wants to be looking at is indentation level 🤔. Would be a very different impl than what we just made tho I think 😅

That example sounds useful.
Using indentation is in many ways simpler than what we are currently doing :D


Use greenlight instead of dflow in the URLs of the apps

Please update the config of dflow to use "greenlight" in the various instances (e.g. greenlight.orgbook.gov.bc.ca) .  The urls are live.  Wade has added the routes to OpenShift already.

I chatted with @WadeBarnes and we decided to recap the steps that are required after a domain name change.
Preliminary steps:

new DNS entries are created for the new dev and test base URLs.
new routes matching the new DNS entries are created in OpenShift, using the appropriate CA certificates, for the web application service.

Agent updates:

each agent needs to be updated to reflect the new domain names. This includes:

updating the APPLICATION_URL field to reflect the new webapp URL.
re-creating the route exposing the agent endpoint, used in the ENDPOINT_URL environment variable.

if the ledger domain name was updated, the LEDGER_URL field will require updating as well. The ledger will at least need to be restarted in order to set the updated display name environment variable that drives the browser window name.

Please note: it is not possible to have two public URLs and configure the agents so that they use the one initially followed to reach them (e.g.: keep all the dFlow traffic on dFlow routes, and GreenLight on GreenLight routes), since the ledger only stores the one provided upon agent startup and registration.
Cleaning up:

all old routes can be deleted at this point, if desired.

dev and test environments have been updated to use the new URLs


Gallery of analysis examples for CMSAF@UNL facility

Collect and curate CMS, ATLAS and Opendata developed and tested on coffea-casa AF instance(s).

Started to collect repositories here: https://github.com/CoffeaTeam/coffea-casa#check-analysis-repositories-using-coffea-casa
Maybe we need to have dedicated page in coffea-casa readthedocs? @mat-adamec

For now, we have two places we are collecting notebooks with coffea-casa physics analysis:

https://github.com/CoffeaTeam/coffea-casa#analysis-repositories-using-coffea-casa
https://github.com/CoffeaTeam/coffea-casa-tutorials

I will open dedicated issues to add notebooks in coffea-casa readthe docs and make them available then inside coffea-casa during the start of the notebook pod.


build(docker): Release 0.5.0

Description

Development notes

Checklist

[x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Updated the documentation to reflect the code changes
[ ] Added a description of this change in the relevant RELEASE.md file
[ ] Added tests to cover my changes

https://github.com/yaml/pyyaml/issues/601#issuecomment-1813963845

Somehow the logs are wrapped in a more narrow max width now and that's why some assertions fail. For example one test expects INFO - Pipeline execution completed successfully but instead the logs show
                          INFO     Pipeline execution completed          runner.py:112
                                   successfully.

(notice how the line is broken)
https://github.com/kedro-org/kedro-plugins/actions/runs/7186100161/job/19570771296?pr=473#step:7:168

Somehow the logs are wrapped in a more narrow max width now and that's why some assertions fail. For example one test expects INFO - Pipeline execution completed successfully but instead the logs show
                          INFO     Pipeline execution completed          runner.py:112
                                   successfully.

(notice how the line is broken)
https://github.com/kedro-org/kedro-plugins/actions/runs/7186100161/job/19570771296?pr=473#step:7:168

Might be helpful: https://stackoverflow.com/questions/14446861/matching-discontinuous-interrupted-strings

I starting to suspect if this PR accidentally bring in a bug #92, I am not certain because before that our e2e-test never try to write on mounted storage. pandas-iris only read from it but now we changer to spaceflights-pandas which write to the mounted volume.
The problem only appear when using with the --uid and --gid options.

I starting to suspect if this PR accidentally bring in a bug #92, I am not certain because before that our e2e-test never try to write on mounted storage. pandas-iris only read from it but now we changer to spaceflights-pandas which write to the mounted volume.
The problem only appear when using with the --uid and --gid options.

I propose we do the following:

Create a simpler test started for kedro-docker e2e tests that doesn't need to write to mounted volume
In a separate task fix the issue with mounted volume in combination with the --uid and --gid options

It would be a shame to have the compatibility with Kedro 0.19.0 blocked by this issue which might have been broken for a much longer time already.

@merelcht After reading this carefully, I have changed my mind and I don't think it is a broken feature, it is just a bad test.

Create a simpler test started for kedro-docker e2e tests that doesn't need to write to mounted volume

I agree with this. The permission problem is not a bug because UID/GID need to match with the host. For someone who set this manually should be aware of the consequence.
I propose we update the test. Instead of testing with a pipeline run, we can use kedro docker ipython, and assert PID and UID to see if it match with the CLI arguments kedro docker ipython. The original test isn't really testing UID/GID, it could still work even if the UID and GID wasn't set properly.


Please don't lock the tsconfig.json file

I am attempting to use a separate git client and it keeps complaining about the tsconfig.json file being locked. If I close the folder from Visual Studio Code, the file is unlocked but also deleted.
It looks like Visual Studio Code is somehow locking the tsconfig.json file.
Expected:
Allow other applications to read and write the tsconfig.json file even if the folder containing that file is open in Visual Studio Code.
Environment:
Visual Studio Code:

OS: Windows 7

@claycephas  we don't do anything special with the tsconfig.json file and I wasn't able to reproduce this. Can you please use the handle tool from sysinternals (https://technet.microsoft.com/en-us/sysinternals/bb896655.aspx) to see who is locking the file in your case?

I have the same isue. I dont explicitly open the file but I have a task that modifies it for me. However even if that task is killed I still have to close Visual studio code to unlock the file.

+1

It happened again... code.exe is locking the file according to handle

@Clivar do you have any additional steps. I tried various scenarios (having tsconfig.json open, changing it, ....) but I couldn't make code holding a handle to it.
I suspect that the tsserver is holding the handle but before moving it to them I want to make sure that this is the case. When it happens the next time can you provide the command line of the code.exe process holding the lock. That would help ti understand which part of the code is the culprit here since code is multi process  and one VSCode instance start ~6 code.exe processes.

fwiw, try installing gitex and see if that has to do with anything. The last time I remember it happening was when I checkout/merge branches. Intellisense went crazy and I finally tracked it down to the tsconfig.json being missing. Happened a few more times with the tsconfig.json either being locked or gone all together.

@tonyeung Exactly, I have it when pulling or merging with git. File went missing or was empty sometimes aswell. Happens quite a bit. Closing and reopening code has always fixed the access denied issue so far

@Clivar  how do you interact with git (git bash or gitex or sourcetree)?

git bash. I do however commit from inside vs code

@Clivar  thanks. So to clarify, you commit within vscode, but do execute other git functions via git bash right?

I have the same issue of tsconfig.json file is locked and after exiting VS code the tsconfig.json file is deleted.
I use Git Extensions on Windows for all git commands (commit, push, pull).
C:\Apps>handle tsconfig.json

Handle v4.0
Copyright (C) 1997-2014 Mark Russinovich
Sysinternals - www.sysinternals.com

code.exe           pid: 16764  type: File           260: C:\Source\Modus\client\tsconfig.json
code.exe           pid: 12804  type: File           260: C:\Source\VSS.UI.Utilities\tsconfig.json

PID 12804:
"C:\Program Files (x86)\Microsoft VS Code\Code.exe" "c:\Program Files (x86)\Microsoft VS Code\resources\app\extensions\typescript\out\utils\electronForkStart" "c:\Program Files (x86)\Microsoft VS Code\resources\app\extensions\typescript\server\typescript\lib\tsserver.js"

PID 16764:
"C:\Program Files (x86)\Microsoft VS Code\Code.exe" "c:\Program Files (x86)\Microsoft VS Code\resources\app\extensions\typescript\out\utils\electronForkStart" "c:\Program Files (x86)\Microsoft VS Code\resources\app\extensions\typescript\server\typescript\lib\tsserver.js"

@tonyeung Exactly, Pull and Push sometimes from UI aswell but most of the time I use GIT Bash for repo actions

Was able to reproduce the tsconfig.json file lock again. Here's the output to handle.exe:
Handle v4.0
Copyright (C) 1997-2014 Mark Russinovich
Sysinternals - www.sysinternals.com

code.exe pid: 16136 type: File 24C: C:\VSS.UI.Utilities\VSS
.UI.Utilities\tsconfig.json

Process Explorer shows this is the command line for PID 16136:
"C:\Program Files (x86)\Microsoft VS Code\code.exe" "c:\Program Files (x86)\Microsoft VS Code\resources\app\extensions\typescript\out\utils\electronForkStart" "c:\Program Files (x86)\Microsoft VS Code\resources\app\extensions\typescript\server\typescript\lib\tsserver.js"

After exiting Visual Studio Code the tsconfig.json file was deleted.

I think the deletion of the tsconfig.json file makes this a pretty critical bug.
The first time it happened it took me a while to figure out what happened. The worse part is that the built in git client didn't pick up the change for a while, so I didn't notice anything missing. If for some reason someone was working without source control it would be a nightmare to figure out why intellisense doesn't work anymore.

I've seen the same thing with other files (it was a TypeScript file this time):
$ git rebase upstream/master
First, rewinding head to replay your work on top of it...
error: unable to create file gulp-typescript/gulp-typescript.d.ts (Permission denied)
Fast-forwarded master to upstream/master.

$ git rebase upstream/master
gulp-typescript/gulp-typescript.d.ts: Permission denied
Current branch master is up to date.

$ git rebase upstream/master
Cannot rebase: You have unstaged changes.
Please commit or stash them.

$ git status
On branch master
Your branch is ahead of 'origin/master' by 627 commits.
  (use "git push" to publish your local commits)
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        deleted:    gulp-typescript/gulp-typescript.d.ts

no changes added to commit (use "git add" and/or "git commit -a")

The file was open inside VS Code.
Windows 10.

I've seen the same thing with other files (it was a TypeScript file this time):
$ git rebase upstream/master
First, rewinding head to replay your work on top of it...
error: unable to create file gulp-typescript/gulp-typescript.d.ts (Permission denied)
Fast-forwarded master to upstream/master.

$ git rebase upstream/master
gulp-typescript/gulp-typescript.d.ts: Permission denied
Current branch master is up to date.

$ git rebase upstream/master
Cannot rebase: You have unstaged changes.
Please commit or stash them.

$ git status
On branch master
Your branch is ahead of 'origin/master' by 627 commits.
  (use "git push" to publish your local commits)
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        deleted:    gulp-typescript/gulp-typescript.d.ts

no changes added to commit (use "git add" and/or "git commit -a")

The file was open inside VS Code.
Windows 10.

@claycephas Thanks a lot for the information. As suspected it is the tsserver that locks the tsconfig.json file. I will open a bug against the TypeScript team. I have one additional question: are you seeing this with the latest VSCode as well (e.g. 0.10.6). This version ships with a new version of the tsserver.

Open https://github.com/Microsoft/TypeScript/issues/6337 in TS land

@dbaeumer  i just had an issue with the tsconfig.json file again last night.. or was it early this morning.. I'm pretty sure I'm on the latest vs code.

I'm having a weird problem with tsconfig.json when VSCode is open. If I do a git checkout someBranch tsconfig.json sometimes gets deleted or locked and I have to issue a git reset --hard HEAD despite having just switched branches. tsconfig.json exists in both branches.
All git commands are issued by commandline (ie: not through VSCode).
I think this is caused by VSCode locking the tsconfig file so git can't overwrite it or something, and it eventually ends up deleted. If VSCode is closed, this doesn't happen. However, no git errors ever occur and I don't notice until my tsc commands fail or I do a git status. Pretty fishy.

@Penryn  its an issue with tsc, @dbaeumer  already opened a ticket with the typescript team.

Ah, thank you for the update. I understand now.

vscode keeps deleting my tsconfig.json whenever I checkout a new git branch

@pocesar: could it be that git deletes it. We don't have any logic in regards to git operations. (e.g. what happens if you checkout the branch without having VSCode open)

@dbaeumer nope, tsconfig.json is on all of my branches

@pocesar: is your repository public so that I can try to reproduce.

Another repro with VS Code version 0.10.6.

Output from handle.exe:
C:\Users\canders2>c:\apps\Handle.exe tsconfig.json

Handle v4.0
Copyright (C) 1997-2014 Mark Russinovich
Sysinternals - www.sysinternals.com

code.exe           pid: 1520   type: File           260: C:\Source\Modus\client\tsconfig.json

Output from procexp.exe
Path:
C:\Program Files (x86)\Microsoft VS Code\code.exe

Command line:
"C:\Program Files (x86)\Microsoft VS Code\Code.exe" "c:\Program Files (x86)\Microsoft VS Code\resources\app\extensions\typescript\out\utils\electronForkStart" "c:\Program Files (x86)\Microsoft VS Code\resources\app\extensions\typescript\server\typescript\lib\tsserver.js"

Current Directory:
C:\Program Files (x86)\Microsoft VS Code\

PARENT:
PID 15536

PARENT Path:
C:\Program Files (x86)\Microsoft VS Code\code.exe

PARENT Command line:
"C:\Program Files (x86)\Microsoft VS Code\Code.exe" "c:\Program Files (x86)\Microsoft VS Code\resources\app\out\bootstrap" --type=pluginHost

PARENT Current directory:
C:\Program Files (x86)\Microsoft VS Code\

After killing PID 1520: client\tsconfig.json file is deleted.

@claycephas did you run a git operation when that happened (e.g. switching branches). I think the deletion of the tsconfig.json file happens as follows:

some one tries to delete tsconfig.json. However since it is locked it fails, however to my knowledge Windows queues the delete and executes it when the lock is gone.
when VSCode ends the file will be deleted.

@dbaeumer it's private, but like was demonstrated by other people in here, it's fairly easy to reproduce. I opened the Devtools inside vscode, and caught this error

intellisense for Typescript stops completly, even though my build task is running

@dbaeumer, the command I was running was git checkout to switch branches.

@dbaeumer I can reproduce on Windows 10. Here are some things I see that are weird:

it reproduces only once I open a TS file, so I speculate TS-Server locks the file
it does not reproduce in 0.10.7-dev
it does also not reproduce for me when I run 0.10.6 "out of dev"

I am puzzled what kind of difference it could make to run 0.10.6 built or "out of dev" with regards to file locking.

As puzzles as I am :-). As said today I was not able to reproduce this in the upcoming 0.10.7-dev build with the identical tsserver bits. When testing under 0.10.6 I noticed that the first file the tsserver reads get locked. But I have no explanation for it.
@bpasero you mentioned today that you disabled asar support. Could it be that asar support is still on in 0.10.6 when using code.exe as a node environment and in 0.10.7 it is off. That could be an explanation for the behavior.

@dbaeumer I reverted my change for noasar and it did not make a difference. Unfortunately I am not able to easily switch back to the Electron we shipped in 0.10.6 because of all the API changes in Electron that I adopted, otherwise we could nail this down more easily.
Btw I also once commented out all file watching code in TS-Server and could still reproduce the file locking issue, so it seems to be unrelated.

@bpasero yes, it is unrelated to file watching. I was able to reproduce on 0.10.6 with a simple fs.readFileSync but not on 0.10.7.

I will close this as fixed for 0.10.7 since all test cases we have that fail on 0.10.6 pass on 0.10.7 although we don't have the final answer why.
@bpasero can you please marked as verified.

This problem is in all Electron based editors. Atom has the same probs. So the real problem lays in the way Electron (Nodejs?) opens files on Windows. It should open them with no lock of course.


BEFORE LAUNCH CHECKLIST

This is to keep track of mathsteps related tasks before the launch mid-Jan:

[x] make a ROADMAP.md based on Socratic coverage goals see it here
[x] go through TODOs in code and move to github issues @evykassirer

code reorg:

[x] #57 @evykassirer
[x] do a lookover for structure consistency? (@aelnaiem)

after we finalize index.js:

[x] #61 publish mathsteps to npm
[x] #61 update the README to just have usage instructions (and how to download on npm) @evykassirer

doesn't necessarily have to be before launch:

[ ] make a wiki on GitHub (stuff like blog post, explanations, code structure)
[ ] Add remove unnecessary parens in NodeStatus creation, and get rid of addStep (in both expression and equations)

moved the rest to their own issues #66 #67


I suppose this is the end of packtpub-crawler?

https://www.packtpub.com/packt/offers/free-learning

They have done it before as part of some a/b tests, hopefully they revert it back after the stats drop (I don't think people manually check the site every day).
Maybe we can contact them, since this script turns a daily chore into a pleasant experience and all their free books are already downloadable from other sources anyways.
But otherwise, we can't do much about it…

oh no! just started implementing this script with the packtpub Alexa skill yesterday! How frustrating!

I have added the book title and the claim URL in the error messages, this way we can at least check if the book is interesting enough to claim it manually.
 Shame we can't automatically claim them anymore :(

going to close this, as https://github.com/niqdev/packtpub-crawler/pull/71 has now been merged

I have created a new branch with a proposal, I don't know if is worth it spend time.
I have fixed the claim, looking at the docs the recaptcha-token field should always be available in the page, but needs to be validated by the client and can be used only once. If you solve the captcha manually and plug the token here you are able to download the book.
If you run the script with an invalid captcha it will download the latest book claimed with the wrong title.
Would be interesting, just for fun, to try to de-couple the claim from the rest, solving only the captcha via mail :blush:
By the way, this document (although I think is already obsolete) is an alternative, but I don't think should be the way to go :disappointed:

Since we have duplicated issues #75 #76 related to this one I will re-open it.
The problem is related to the captha and the error looks like this
[-] <type 'exceptions.IndexError'> list index out of range | spider.py@97
Traceback (most recent call last):
  File "script/spider.py", line 97, in main
    packtpub.runDaily()
  File "/home/ubuntu/Projects/github/packtpub-crawler/script/packtpub.py", line 161, in runDaily
    self.__parseDailyBookInfo(soup)
  File "/home/ubuntu/Projects/github/packtpub-crawler/script/packtpub.py", line 93, in __parseDailyBookInfo
    self.info['url_claim'] = self.__url_base + div_target.select('a.twelve-days-claim')[0]['href']
IndexError: list index out of range

There is a a feature branch with a proposal, but it could be a black hole!

@niqdev really there is a problem with captcha, still, it doesn't work. Maybe implement it by using two steps with opened the page? as one more option

@develsites yep that was the idea/proposal in the feature branch, 2 step process solving the captcha manually via email for example, but yes at the moment the script is broken and we can't do much

Honestly, if you have to solve the captcha manually anyway, then you may as well just go to https://www.packtpub.com/packt/offers/free-learning and claim it manually.
Packtpub-crawler is still useful for notifying what the latest book is though :)

i had no captcha today... is it an error or did they remove it? Claiming still worked

umh, something changed for sure, the reCAPTCHA moved to the bottom-right of the page.
Were you able to download the book with the script?

Hi
I didn't use the script but I was able to get the book manually without
validating reCAPTCHA.
Thanks.
Le lun. 17 juil. 2017 à 21:09, niqdev<EMAIL_ADDRESS>a écrit :

umh, something changed for sure, the reCAPTCHA moved to the bottom-right
of the page.
Were you able to download the book with the script?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/niqdev/packtpub-crawler/issues/70#issuecomment-315851723,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC6cEF-HlWIVHN0GCb0EGbUZygY6wLA5ks5sO7EfgaJpZM4NnXgq
.

The CAPTCHA has not yet returned, but the script fails to claim the book with IndexError('list index out of range',).

yeah, you dont have to "do" anything for the captcha to work... maybe it detects the browser or something?
For me, using chrome, it just works. no box, nothing, but blocking google prints the error "no captcha" or whatever
its new kind of captcha from google?

Yes, they seem to analyze things like mouse movement patterns. It's called
"invisible recaptcha" and it's really interesting when you are into machine
learning.

Hello, we have managed to solve the captcha to make my script-grabber working, You can use the same solution or check my solution at: https://github.com/igbt6/Packt-Publishing-Free-Learning
Regards!

@igbt6 That's awesome, thanks a lot for sharing with us!

@niqdev I managed to get my Packt grabber working by using Selenium in headless mode AND setting useragent to Chrome (default for headless Chrome is, if I recall correctly, WebdriverChrome).

@katka-n great! is it easy to integrate with the current project?

[Hacktoberfest] Anyone interested in integrating Anti Captcha or other solutions? Thanks

@niqdev I am not that experienced but I will try to do so, if I succeed I will create a pull request ;)

here is a python solution for the recaptcha  https://github.com/ecthros/uncaptcha

Thanks @tjadanel , any interest in integrate it?


Updated jsdoc version in package.json

This fixes broken build, due to the fact that in jsdoc 3.3.2 there is reference to non existing repository https://github.com/ariya/esprima which is now https://github.com/jquery/esprima
Following jsdoc commit introduces fix (reference to npm not directly to github)
https://github.com/jsdoc3/jsdoc/commit/4ef7bb483282f1297544f7df1c9952846427fce9

@schmod 3.4.* sounds good to me

closed in favor of #101.
@allenhwkim - Can we push a release to NPM?


Point cloud filters in Python

Required Info

Camera Model
D435

Firmware Version
(Open RealSense Viewer --> Click info)

Operating System & Version
Ubuntu 16

Kernel Version (Linux Only)
(e.g. 4.14.13)

Platform
Raspberry Pi

SDK Version
2.16.2

Language
python

Segment
{Robot/Smartphone/VR/AR/others }

Issue Description
I've successfully been able to export a point cloud in Python! Since there is some noise, I'm looking to use filters and post-processing as shown in the post-processing c++ example. I haven't been able to find documentation for how to do this in Python. Is there any available? Thanks!

depth_filters.ipynb, you can actually launch it in browser for an interactive demo:
The python examples are somewhat in state of disarray, we need to add more and make them more discoverable.

Thank you!


Fast back-translation training with MMT

I have to build a FR-JP model, and I don't find large parallel resources.
First of all, do you know about a large FR-JP corpus somewhere ?
I just have an idea. I would like to know if someone tested it with MMT.

create 2 first models with existing low resources. With small data, this should be quite fast to get a convergence.
translate 2 large monolingual corpus, to get first bad translation pairs. Named them fr-JP and jp-FR, where the bad translations are put in the source side. Add these pairs to both model memories. This should be fast with MMT. It should bring you with much better models, since the bad sources should be used with the fuzzy near sentences to be translated to improve with the good original targets.
back to 2, and loop several times, replacing the MMT memories with better ones each time. This short loop should be fast, because there isn't any heavy neural training.
back to 1. This large loop should be slower, because there is now a neural training with a large parallel corpus

Hi @EtienneAb3d
For public parallel corpora I suggest the Opus Website, but probably you already knows it and there is no large FR-JA corpus (node: JP is not the right lang-code, JA is).
Using back translation for these tasks has proven to be very effective, and the methodology you suggested would definitely create the best quality synthetic data possible. So at the end of the day it's all about speed vs. quality: batch-translate a large monolingual corpora with static engines is still faster then adaptive MT, so if have no time, or quality is not that important, I suggest to go for low-level batch translation.
Just a quick note, not sure if I got it correctly but please note that ModernMT will still use a FR>JA pair to adapt on a JA>FR translation request: memory elements are bi-directional.
For this reason I suggest to keep the two systems (FR > JA and JA > FR) separated otherwise ModernMT will use also synthetic target to adapt, and this won't result in better quality of course.

I had a simpler setup in mind: (1) a NMT model trained on the available parallel data, (2) batch translate of a large monolingual corpus and finally (3) training on parallel + synthetic data.
But you're right: this way translation quality won't change and won't progressively improve.

Hi @EtienneAb3d,
This article could be useful for you.
http://iconictranslation.com/2018/08/issue-5-creating-training-data-for-nmt/

I read it quickly. As far as I understand, they do not really explain what can be obtained when looping the process. I think there is a lot to gain. Certainly I will test it in a near future.

Good luck @EtienneAb3d  and please keep us updated of your progress if possible. Good luck!

Suppose you want to train a DE-PT cooking dedicated model, you don't have any DE-PT well translated cooking data, but you have large monolingual cooking data.
Start from 2 generic DE-PT models, and link them in order to continuously train each other to repair bad translations produced by the other, being better and better at each step.
Would this bring us with high level cooking translation models ?
This video is a nice way of thinking of what appends in the back-translation training = a way to learn how to repair synthetically generated damaged data:
http://www.letribunaldunet.fr/insolite/retouche-photo-intelligence-artificielle-video.html

I think this paper sheds more light on how they did it. https://arxiv.org/pdf/1511.06709.pdf
But what about translating your monolingual data using Google or Bing since they have tons of bilingual data in most language pairs, then use these MT'd pairs in your baseline engine combined with the little parallel data you may have. I think this approach could get your good acceptable results, especially for in-domain engines like cooking engine.

I know that using back-translation is improving the model. My questions was:

what occurs when looping the training? Is the model continuously improving? I don't know a work with results on this question.
how to build a fast system for this, since a neural training can take 1 or 2 weeks, the loop is quite heavy? MMT instant adaptation can be an interesting feature for that.

For your Google proposition, I don't want to use Google, because:

I don't want to pay for it for a large translation of several million of sentence pairs. My goal is to use may own free system.
I can't loop the training on Google, since Google ins't trained by me, again, I don't want to pay for the new possibility of Google to be trained with my own data.
cooking was only an example, I could also be in a situation where the data are highly confidential, and can't go outside on the web
I don't want to be dependent on Google, to sell it to my customers, but to improve my own technology recognized by my customers for its advantages (confidentiality, translation quality, adaptability, ..)

Thanks for your reply @EtienneAb3d ! I haven't tested back-translation approach before. I will update this thread if I go this route.

Experimentation started here:
http://www.lingua-et-machina.eu/transinfo/
Updated every hour with new published news.
The models were not trained with news data, and they have quite special formulations and vocabulary.
For the moment, only the FR-EN couple is really fed with back-translations.
At this early point, the adaptive effect is near null.
Wait for data volume to bring a possible effect...
:)


Valve's servers didn't reply in time

The API always respond with this, even if I have a bunch of bots added (200) and the timeout limit increased to 5s

Proxies are now essential due to the implementation of new restrictions. Have you attempted to incorporate them into your setup?
#131 #132

Well, then why is it successfully setting up connections if it's a proxy issue?

From my understanding of the situation, the new limits impact fetching item data, not logging in. This is causing issues with floats and other data on many websites. Even the extension isn't working right.
Just a heads up, I don't use CSGOFloat/inspect at this time. Just here to help out!

Alright, thanks for the help! It is still unclear to me since it worked yesterday, and it's not working now.

Proxies are now essential due to the implementation of new restrictions. Have you attempted to incorporate them into your setup?
#131 #132

How a do that?

Proxies are now essential due to the implementation of new restrictions. Have you attempted to incorporate them into your setup?
#131 #132

How a do that?

Update CSGOFloat to the newest version and add the proxies parameter to your config, like in config.example.js.
Keep in mind, this is for those self-hosting CSGOFloat API. Since you created issue #140 about the extension, this suggestion won't apply if you're only using the extension.

Update CSGOFloat to the newest version and add the proxies parameter to your config, like in config.example.js.
Just to let you know, this is for those self-hosting CSGOFloat API. If you're only using the extension, as mentioned in issue #140, this suggestion won't apply.

It's my first time with this problem and I don't know JavaScript, There's s solution for the extension?

Update CSGOFloat to the newest version and add the proxies parameter to your config, like in config.example.js.
Just to let you know, this is for those self-hosting CSGOFloat API. If you're only using the extension, as mentioned in issue #140, this suggestion won't apply.

It's my first time with this problem and I don't know JavaScript, There's s solution for the extension?

At the moment, end-users can't fix this issue. It's a backend problem with CSGOFloat related to the new CSGO limits. The only solution is to wait for an updated version containing the necessary fixes.

At the moment, end-users can't fix this issue. It's a backend problem with CSGOFloat related to the new CSGO limits. The only solution is to wait for an updated version containing the necessary fixes.

thank bro

Closing since this is related to you spamming the GC too much.


put something massive in proposals.rst to warn randoms not to resolve comments

Rationale: Google docs doesn't have a setting to disable the 'resolve' button on comment threads. This means that anonymous users can resolve a thread.
Google's answer to the problem is to submit feedback via Help -> Report a problem on the doc (source)

I don't think we have this problem anymore now that randoms can't resolve threads (we switched to Github).


build: added key press tests

added in tests for each of the key press commands on the recording page to prove that the new key handlers work

Update PR title to follow the commit message convention

Is "build" an appropriate prefix? I wasn't sure since it's adding tests - "feat" and "fix" didn't feel right.


[MM-38153] Allow test-e2e to run all tests

Summary
This PR allows running all test using make test-e2e by removing the build tags for the non e2e tests. It also allows for easier developing as a developer can now set the local build tags to e2e and compile all files.
Ticket Link
https://mattermost.atlassian.net/browse/MM-38153

Codecov Report

Merging #246 (3dce29d) into master (3b5697e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #246   +/-   ##
=======================================
  Coverage   24.77%   24.77%
=======================================
  Files          52       52
  Lines        3257     3257
=======================================
  Hits          807      807
  Misses       2373     2373
  Partials       77       77

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b5697e...3dce29d. Read the comment docs.

@hanzei - wrt to "running all test using make test-e2e" - shouldn't we change the naming since not all test (including unit, etc. ?) are considered e2e now?

@catalintomai The target make test is currently an alias to make test-unit, which I think it a good things as most of the time a developers wants to run the non-e2e test as the e2e ones take a long time.
While renaming make test-e2e makes sense, I'm struggling to find a better name as make test is already taken. Maybe make test-all. Do you have suggestions?

@hanzei, wrt to "as most of the time a developers wants to run the non-e2e test as the e2e ones take a long time" - it looks like we expect a differentiation between e2e and non-e2e, so putting all tests under the e2e umbrella seems to go contrary to that.  On the naming, as a developer I'd be a bit confused on the difference between make test and make test-all - I'd rather that we don't have make test to begin with - then we can have only smth like make test-all. I'd say that we follow the taxonomy that we use in other repos/parts of the product. Do they also encompass all tests under the make test-e2e moniker?

As I mentioned before, imo we should be consistent to what we do in other places/repos -

In the mattermost server repo make test runs some of the tests, while leaving specific and long running tests to specific commands. Please note that similar to how it's done here the make test calls sub commands like make test-server. Are there other repos that you are referring to?
FWIW this PR doesn't change the semantic. Currently make test-e2 runs some of the unit tests, but without consistency which are run and which not. With this change it's runs all of them and is consistent. Maybe we can defer the make target discussion to another ticket?


AttributeError: 'SpooledTemporaryFile' object has no attribute 'readinto'

I get the following error while trying backup a Postgres db and upload it to the S3. The S3 setup works fine.
`Writing file to default-something-2020-12-18-002957.psql
AttributeError: 'SpooledTemporaryFile' object has no attribute 'readinto'
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/utils.py", line 118, in wrapper
func(*args, **kwargs)
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 61, in handle
self._save_new_backup(database)
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 88, in _save_new_backup
self.write_to_storage(outputfile, filename)
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/_base.py", line 88, in write_to_storage
self.storage.write_file(file, path)
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/storage.py", line 82, in write_file
self.storage.save(name=filename, content=filehandle)
File "/home/django/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save
return self._save(name, content)
File "/home/django/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 446, in _save
obj.upload_fileobj(reader, ExtraArgs=params)
File "/home/django/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 619, in object_upload_fileobj
return self.meta.client.upload_fileobj(
File "/home/django/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj
return future.result()
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
raise self._exception
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
self._submit(transfer_future=transfer_future, **kwargs)
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 558, in _submit
self._submit_multipart_request(
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 622, in _submit_multipart_request
for part_number, fileobj in part_iterator:
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 270, in yield_upload_part_bodies
fileobj, full_size = self._get_upload_part_fileobj_with_full_size(
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 337, in _get_upload_part_fileobj_with_full_size
data = fileobj.read(kwargs['part_size'])
File "/home/django/venv/lib/python3.8/site-packages/django/core/files/utils.py", line 17, in
readinto = property(lambda self: self.file.readinto)
Traceback (most recent call last):
File "manage.py", line 22, in
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/django/venv/lib/python3.8/site-packages/django/core/management/init.py", line 401, in execute_from_command_line
utility.execute()
File "/home/django/venv/lib/python3.8/site-packages/django/core/management/init.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/django/venv/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/django/venv/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
output = self.handle(*args, **options)
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/utils.py", line 118, in wrapper
func(*args, **kwargs)
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 61, in handle
self._save_new_backup(database)
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 88, in _save_new_backup
self.write_to_storage(outputfile, filename)
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/_base.py", line 88, in write_to_storage
self.storage.write_file(file, path)
File "/home/django/venv/lib/python3.8/site-packages/dbbackup/storage.py", line 82, in write_file
self.storage.save(name=filename, content=filehandle)
File "/home/django/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save
return self._save(name, content)
File "/home/django/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 446, in _save
obj.upload_fileobj(reader, ExtraArgs=params)
File "/home/django/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 619, in object_upload_fileobj
return self.meta.client.upload_fileobj(
File "/home/django/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj
return future.result()
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
raise self._exception
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
self._submit(transfer_future=transfer_future, **kwargs)
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 558, in _submit
self._submit_multipart_request(
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 622, in _submit_multipart_request
for part_number, fileobj in part_iterator:
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 270, in yield_upload_part_bodies
fileobj, full_size = self._get_upload_part_fileobj_with_full_size(
File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 337, in _get_upload_part_fileobj_with_full_size
data = fileobj.read(kwargs['part_size'])
File "/home/django/venv/lib/python3.8/site-packages/django/core/files/utils.py", line 17, in
readinto = property(lambda self: self.file.readinto)
AttributeError: 'SpooledTemporaryFile' object has no attribute 'readinto'
`
Settings.py
DBBACKUP_CONNECTORS = {
    'default': {
        'USER': os.environ['DB_USER'],
        'PASSWORD': os.environ['DB_PASSWORD'],
        'HOST': os.environ['DB_HOST']
    }
}

DBBACKUP_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DBBACKUP_STORAGE_OPTIONS = {
    "access_key": os.environ['AWS_ACCESS_KEY_ID'],
    "secret_key": os.environ['AWS_SECRET_ACCESS_KEY'],
    "bucket_name": os.environ['AWS_BUCKET_NAME'],
    'default_acl': 'private'
}

Versions
Django-dbbackup

pypi: 3.30

External tools

Python: 3.8
Django: 3.1
OS: Ubuntu 18.4

It looks like you're bumping into this issue described in the PR on comment here: https://github.com/python/cpython/pull/3249#issuecomment-719880320 As described in the comment a viable solution could be to monkeypatch SpooledTemporaryFile. If you've got time to test that I'd be keen to hear back.

A longer term fix would fix this in django itself..


Improve error title for error code 422 ("Unprocessable Entity") in partial search

If the rdap server has the "allow_search_wildcars_anywhere" property (in configuration.properties) turned off and a client executes a partial search like
http://localhost:8080/rdap-server/domains?nsLdhName=*test*
The following error message is displayed:

{"rdapConformance":["rdap_level_0"],"errorCode":"422","title":"Unprocessable Entity","description":"Partial search can only have wildcards at the end of each label"}

However, this message could be confusing because the standard title for error code 422 mentions an "unprocessable entity", so it could be confused with the rdap entity object when a client is searching for a domain.

The default title is now "Unprocessable HTTP Entity" to avoid confusion. The issue was at NICMx/rdap-data-access-api, still will be solved here.

Reopen (bug). When a 422 error code is returned from server, the title keep showing the text "Unprocessable entity".
{"rdapConformance":["rdap_level_0"],"errorCode":"422","title":"Unprocessable Entity","description":"Search pattern must be at least 5 characters, consecutive wildcards '*' are treated as only one character. Pattern received: *"}
This is probably caused because the DEFAULT_MSG in UnprocessableEntityException class (in the solution) is not used when the exception is created.

Yep, the solution was incomplete. The title was set at ExceptionResult, the real fix was to update the title of the error at that class; please, try again.

Verify. If an user perform a search that returns a 422 error code, the error message is displayed with the new title. For example
{"rdapConformance":["rdap_level_0"],"errorCode":"422","title":"Unprocessable HTTP Entity","description":"Partial search can only have one wildcard at each label"}


Show multiple model fields in one column

Is there a way to show multiple model fields in one column? I don't want to use grouped columns but would rather show multiple model fields stacked in one column for each table row.

You can create a template under templates/components/multiple-fields.hbs and put whatever content you need to display
{{row.field1}}
{{row.field2}}
{{row.field3}}

and in your column definition just add cellComponent: 'multiple-fields'

Thanks for the quick reply. I had the component defined in cellComponent but was failing to use row.content to refer to the field.


Data corruption

The latest changes in RNS have fixed the issues with channels and listeners unresponsive. I ran top with rnsh overnight over both ethernet and LoRa and the sessions are still running this morning--something that was not possible on the previous version.
However, I am noticing some subtle problems. In the session running over ethernet, I'm noticing text out of place after a minute or so. I simultaneously ran a session over SSH and it had no problems. (Screenshots at the bottom)
It seems like macOS top doesn't clear the terminal or the line, but rather only updates characters when they change by positioning the cursor and printing the updated text. This means if a data message is lost or duplicated, it could result in this garbling (particularly if relative cursor positioning is used). This could be a bug introduced by the recent changes in RNS, but I think I may have seen issues like this before that update.
SSH:

rnsh - notice the garbled first few lines compared to the SSH session:

An even more garbed top over rnsh session after about 20 minutes. The top over ssh session is still running and has no corruption.

I ran a test that printed a sequence of numbers, copied the terminal output into a file and compared the text to a locally generated file. There appear to be some out-of-order segments in the received text.
Text Generator i=0; while [ $i -lt 1000000 ]; do i=$((i+1)); printf "%032d\n" $i; done
$ diff -y --suppress-common-lines over-rnsh.txt locally-generated.txt
00000000000037510						|<PHONE_NUMBER>0000000000000000037498
								><PHONE_NUMBER>0000000000000000037499
								><PHONE_NUMBER>0000000000000000037500
								><PHONE_NUMBER>0000000000000000037501
								><PHONE_NUMBER>0000000000000000037502
								><PHONE_NUMBER>0000000000000000037503
								><PHONE_NUMBER>0000000000000000037504
								><PHONE_NUMBER>0000000000000000037505
								><PHONE_NUMBER>0000000000000000037506
								><PHONE_NUMBER>0000000000000000037507
								><PHONE_NUMBER>0000000000000000037508
								><PHONE_NUMBER>0000000000000000037509
								><PHONE_NUMBER>0000000000000000037510
0000000000000000000000037498					|<PHONE_NUMBER>0000000000000000037523
00000000000000000000000000037499				<
00000000000000000000000000037500				<
00000000000000000000000000037501				<
00000000000000000000000000037502				<
00000000000000000000000000037503				<
00000000000000000000000000037504				<
00000000000000000000000000037505				<
00000000000000000000000000037506				<
00000000000000000000000000037507				<
00000000000000000000000000037508				<
00000000000000000000000000037509				<
000000000000000000000000000000000000000000000037523		<
000000000000000000000000000000000000000000000037834		|<PHONE_NUMBER>0000000000000000037821
00000000000000000000000000037835				<
00000000000000000000000000037836				<
00000000000000000000000000037837				<
00000000000000000000000000037838				<
00000000000000000000000000037839				<
00000000000000000000000000037840				<
00000000000000000000000000037841				<
00000000000000000000000000037842				<
00000000000000000000000000037843				<
00000000000000000000000000037844				<
00000000000000000000000000037845				<
0000000000000000000000037821					<
00000000000037846						|<PHONE_NUMBER>0000000000000000037834
								><PHONE_NUMBER>0000000000000000037835
								><PHONE_NUMBER>0000000000000000037836
								><PHONE_NUMBER>0000000000000000037837
								><PHONE_NUMBER>0000000000000000037838
								><PHONE_NUMBER>0000000000000000037839
								><PHONE_NUMBER>0000000000000000037840
								><PHONE_NUMBER>0000000000000000037841
								><PHONE_NUMBER>0000000000000000037842
								><PHONE_NUMBER>0000000000000000037843
								><PHONE_NUMBER>0000000000000000037844
								><PHONE_NUMBER>0000000000000000037845
								><PHONE_NUMBER>0000000000000000037846

Just FYI, I am trying to debug/diagnose this now.

Ok, I'm a bafoon. When I added the windowing stuff, I "solved" a lock hang by adding a thread decoupler at the wrong place in the call chain, causing a potential race condition for the callback handling the extension of the buffer data.
Fix incoming :)

I prefer by a wide margin a one-line oopsie over an unforeseen architecture issue requiring an extensive rewrite. 😅

Haha, exactly! Well, I just pushed some commits that should fix the issue. Let me know when you get some time to test it out!
I've tested it out both with:

A small script reading a file from disk and sending it over a buffer. Comparing afterwards with diff is an exact match this time around :)
Running rnsh sessions and checking for any garbling over time. None so far. Even had four instances of top running inside tmux for a couple of hours, and all looked good.
Using rnsh to cat various files and comparing the output with diff. All checks out :)
Tested both over local instance, LAN and multi-hop over the internet.

Fingers crossed it's good this time around :)

The top test as well as the session dump test both pass. Good fix @markqvist! Released in v0.1.1


Compile regexs into application env

BugzID: 48704

I'm moving this to an immutable cache, this is too filthy.


Fix a number of typescript issues

Fixes 79 typescript errors. Discovered at least two bugs in notifications.ts, and I'm pretty sure this feature was at least partially broken and may still be, I don't really know how to test it.
After this, only like ~10 typescript errors remain in the codebase but those are harder to solve.

There should be no "tsc" error now

Those .vue import errors can be fixed by using vue-tsc instead of tsc, but it's currently broken for typescript 5.7 which we already use: https://github.com/vuejs/language-tools/issues/5018.

I guess I will just uninstall vue-tsc and we live with those few @ts-expect-error. vue-tsc is monkeypatching tsc and that will likely break again in the future, so I think we are better off using official typescript tooling only.

tsc is now enabled as part of make lint-js and is passing for me and should do too on CI.

I switched back to vue-tsc, to a fork of mine that is from their master branch which works with typescript 5.7, but for some reason was not released yet (see https://github.com/vuejs/language-tools/issues/5018).
.vue files are getting type-checked again and the errors on the imports on .vue files go away. Interestingly a new error shows in vitest.config.ts, so I added a @ts-expect-error there. I hope we can resolve this later.

FYI I opened https://github.com/vitejs/vite-plugin-vue/issues/487 regarding this not callable error.

So I think I found the magic incantation that requires the least amount of @ts-expect-error:
    "module": "esnext",
    "moduleResolution": "bundler",

This is suboptimal as we do have a few node.js scripts, but we can likely split the tsconfig into two later, one for node files and one for web_src.

I added type definitions for all named exports from *.vue files, so npx tsc is now also clean. It does not typecheck the .vue files because it can not parse them, so we need to use vue-tsc for that.
Long-term, I'd like to forbid named exports from Vue SFCs, they don't seem worth it.


Video+Audio downloads fail with error code 400

Steps to reproduce
Test video ID: H7i9Anh0_Dg
const stream = await this.videoInfo.download({
  type: "audio+video",
  quality: "best",
  format: "mp4"
});

Failure Logs

Expected behavior
The video downloads completely or rejects the promise with error code.
Current behavior
Trying to download videos with Youtube.js fails because a download request is rejected with a 400 error code. Most of the requests are successful, but one is not. Rate limiting requests client-side has no effect, it appears that the same exact request fails every time. Different youtube videos also fail to download with similar behavior.
When the download fails, the promise is not rejected. Instead, it hangs forever causing the application to be stuck in an waiting state.
Version
Default
Anything else?
No response
Checklist

[X] I am running the latest version.
[X] I checked the documentation and found no answer.
[X] I have searched the existing issues and made sure this is not a duplicate.
[X] I have provided sufficient information.

Hi!
This is likely due to the range being invalid, are you using the range option? As you can see here these audio+video formats are usually downloaded in one request unless you force the range (which is really only required for the adaptive formats).

Thank you for your help. I needed to use video+audio instead of audio+video since it checks for that string specifically. Note that the example download code also has the reversed order:
   const stream = await yt.download(song.id as string, {
      type: 'audio', // audio, video or audio+video
      quality: 'best', // best, bestefficiency, 144p, 240p, 480p, 720p and so on.
      format: 'mp4' // media container format
    });


SPICE reader: take into account additional distortion information

Describe the feature
The current SPICE FITS files include "WCSDVARR" ImageHDUs to represent additional distortion information (see the draft WCS IV paper). Distortion from the instrument optics is already corrected in SPICE L2 FITS files, but the additional distortion in the WCSDVARR HDUs is the one that comes from spacecraft pointing changes, as obtained by the pipeline from the Solar Orbiter SPICE kernels.
#208 was about fixing the SPICE reader so that it does not fail on SPICE FITS files including HDUs for additional distortion information. This fix has been implemented by ignoring the WCSDVARR HDUs.
The SPICE FITS file reader should have the option to take into account this information instead of ignoring it.
Proposed solution
How this should be done, and whether this should be part of sunraster or of a SPICE-specific package is still unclear. A possibility is to reproject on a non-distorted grid, but this has drawbacks, and it requires deciding what to do in problematic cases (e.g. when pointing jitter in the scanning direction is larger than the raster step).

Now described by Issue #9 in sospice and will be implemented there.


Python sklearn XGBClassifier cannot used in plot_partial_dependence

xgbModel = xgb.XGBClassifier(max_depth=5, learning_rate=0.3, silent=1, objective='binary:logistic',
min_child_weight=10, gamma=0.1, subsample=0.8, scale_pos_weight=111.5).fit(feat, labels)
fig, axs = plot_partial_dependence(xgbModel, feat, focus_features,
feature_names=featurenames,
n_jobs=3, grid_resolution=50)
I Got Error Message,
if not isinstance(gbrt, BaseGradientBoosting):
raise ValueError('gbrt has to be an instance of BaseGradientBoosting')
I Check the source code, Seems XGBClassifier is not based on BaseGradientBoosting
So the sklearn wrapper is not totally available for all feature in sklean?

I ran into the same issue. Is there a PDP function in python that works with XGBClassifier?

I use the following function (highly inspired from this notebook:
import numpy as np

def partial_dependency(model, X, features, selected_feature, floor):
    # The model could be an XGBoost sklearn fitted instance (or anything else with a
    # predict method)
    X_temp = X.copy()
    # Works only for numerical features. Find something else for categorical features.
    grid = np.linspace(np.percentile(X_temp.loc[:, selected_feature], 0.1),
                       np.percentile(X_temp.loc[:, selected_feature], 99.5),
                       50)
    y_pred = np.zeros(len(grid))

    for i, val in enumerate(grid):
        X_temp.loc[:, selected_feature] = val
        y_pred[i] = model.predict(X_temp.loc[:, features]).mean()

    return grid, y_pred

I think it is possible to use the sklearn plot_partial_dependence function if you remove the
if not isinstance(gbrt, BaseGradientBoosting) check.
To sum up, it would be nice to integrate a plot_partial_dependence into the XGBoost Python API.