MSBuild-15.9 not working with Godot 3.4.4-mono

Godot version
3.4.4
System information
Linux localhost 5.15.32-gentoo-r1-x86_64 #1 SMP Mon Apr 18 21:16:59 CEST 2022 x86_64 Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz GenuineIntel GNU/Linux
Issue description
I have MSBuild-15.9 installed along with Mono-6.12.0-stable and godot just fails to build any CS file.
The message its reporting is basically telling me that my version is either incompatible or I am missing some dependency.
Running: "/home/tino/bin/msbuild"  "/home/tino/Software/godot/test/New Game Projectxxx.sln" /restore /t:Build "/p:Configuration=Debug" /v:normal "/l:GodotTools.BuildLogger.GodotBuildLogger,/home/tino/Software/GodotVersions/Godot_v3.4.4-stable_mono_x11_64/GodotSharp/Tools/GodotTools.BuildLogger.dll;/home/tino/.local/share/godot/mono/build_logs/412526998dd3745cb2b486a94e16e341_Debug" /p:GodotTargetPlatform=x11
Microsoft (R) Build Engine version <IP_ADDRESS>856 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : Logger error MSB1029: The "GodotTools.BuildLogger.GodotBuildLogger" logger could not be instantiated from the assembly "/home/tino/Software/GodotVersions/Godot_v3.4.4-stable_mono_x11_64/GodotSharp/Tools/GodotTools.BuildLogger.dll". Please verify the logger assembly has been built using the same version of the Microsoft.Build.Framework assembly as the one installed on your computer and that your host application is not missing a binding redirect for Microsoft.Build.Framework. Specified cast is not valid.

/home/tino/bin/msbuild is a symlink to /usr/local/bin/msbuild-15.9
I have no clue how to proceed beyond this point. Any pointers would help.
Steps to reproduce

Install MSBuild-15.9
Create new project
Add a node
Add a C# script
Call GD.Print(""); somewhere
Make sure Godot can find the MSBuild binary
Build

Minimal reproduction project
test.zip

In addition, I have tried to rebuild godot from sources to make sure it gets built using the versions of the tools on my system. But somehow I am still getting the same error.
I followed the steps outlined here: https://docs.godotengine.org/en/stable/development/compiling/compiling_with_mono.html
Using the git branch for 3.4.4-stable, i generated the pre-build binary to generate the mono_glue, generated the mono_glue then recompiled with
scons p=x11 target=release_debug tools=yes module_mono_enabled=yes mono_glue=yes
After this i got this error:
> godot3 -e
Godot Engine v3.4.4.stable.mono.custom_build.419e713a2 - https://godotengine.org
OpenGL ES 3.0 Renderer: GeForce GT 240/PCIe/SSE2
OpenGL ES Batching: ON

Mono: Log file is: '/home/tino/.local/share/godot/mono/mono_logs/2022-05-02_17.34.15_5959.log'

Unhandled Exception:
System.TypeInitializationException: The type initializer for 'Sys' threw an exception. ---> System.DllNotFoundException: /home/tino/3rdpartysoft/godot/bin/GodotSharp/Mono/lib/../lib64/libmono-native.so assembly:<unknown assembly> type:<unknown type> member:(null)
  at (wrapper managed-to-native) Interop+Sys.LChflagsCanSetHiddenFlag()
  at Interop+Sys..cctor () [0x00000] in <a6a5ba8fc13a4797a32a4dc4ae25c772>:0
   --- End of inner exception stack trace ---
  at System.IO.FileSystem.FileExists (System.ReadOnlySpan`1[T] fullPath, System.Int32 fileType, Interop+ErrorInfo& errorInfo) [0x00007] in <a6a5ba8fc13a4797a32a4dc4ae25c772>:0
  at System.IO.FileSystem.FileExists (System.ReadOnlySpan`1[T] fullPath) [0x00006] in <a6a5ba8fc13a4797a32a4dc4ae25c772>:0
  at System.IO.File.Exists (System.String path) [0x00043] in <a6a5ba8fc13a4797a32a4dc4ae25c772>:0
  at GodotTools.Utils.File.Exists (System.String path) [0x00006] in <0daea5da6c9f4eeb9930d3675498733a>:0
  at GodotTools.GodotSharpEditor.EnablePlugin () [0x00130] in <0daea5da6c9f4eeb9930d3675498733a>:0
ERROR: System.TypeInitializationException: The type initializer for 'Sys' threw an exception.
   at: debug_send_unhandled_exception_error (modules/mono/mono_gd/gd_mono_utils.cpp:369)

During the build, only bin/GodotSharp/Mono/lib was created, no lib64 was created by the build.
So I tried symlinking it to /usr/lib64 where the library libmono_native.so resides. After this i got the exact same error as previously mentioned in the opening issue.

After examining the build, i should also mentioned I get this right after scons starts the build:
> scons p=x11 target=release_debug mono_glue=yes module_mono_enabled=yes tools=yes -j4
scons: Reading SConscript files ...
Checking for `thread_local` support... supported
Mono root directory not found. Using pkg-config instead
Checking for C header file mntent.h... (cached) yes
scons: done reading SConscript files.
scons: Building targets ...
[  0%] build_api_solution(["bin/GodotSharp/Api/Debug/GodotSharp.dll", "bin/GodotSharp/Api/Debug/GodotSharp.pdb", "bin/GodotSharp/Api/Debug/GodotSharp.xml", "bin/GodotSharp/Api/Debug/GodotSharpEditor.dll", "bin/GodotSharp/Api/Debug/GodotSharpEditor.pdb", "bin/GodotSharp/Api/Debug/GodotSharpEditor.xml"], [])
MSBuild path: /usr/bin/dotnet
[  1%] Microsoft (R) Build Engine version 17.1.0+ae57d105c for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

[  4%] ^CAttempting to cancel the build...
[  4%] scons: *** [bin/GodotSharp/Api/Debug/GodotSharp.dll] Build interrupted.
scons: Build interrupted.
scons: building terminated because of errors.
scons: writing .sconsign file.
[Time elapsed: 00:00:06.653]
^C
~/3rdpartysoft/godot

I tried with mono_static=yes and found a strange bug
> scons p=x11 mono_glue=yes module_mono_enabled=yes tools=yes mono_static=yes -j4
scons: Reading SConscript files ...
Checking for `thread_local` support... supported
Found Mono root directory: /usr/lib64/pkgconfig/../../lib64/..
RuntimeError: Could not find mono library in: /usr/lib64/pkgconfig/../../lib64/../lib:
  File "/home/tino/3rdpartysoft/godot/SConstruct", line 661:
    SConscript("modules/SCsub")
  File "/usr/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 660:
    return method(*args, **kw)
  File "/usr/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/usr/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/tino/3rdpartysoft/godot/modules/SCsub", line 21:
    SConscript(name + "/SCsub")  # Built-in.
  File "/usr/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 660:
    return method(*args, **kw)
  File "/usr/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/usr/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/tino/3rdpartysoft/godot/modules/mono/SCsub", line 34:
    mono_configure.configure(env, env_mono)
  File "/home/tino/3rdpartysoft/godot/modules/mono/build_scripts/mono_configure.py", line 251:
    raise RuntimeError("Could not find mono library in: " + mono_lib_path)
~/3rdpartysoft/godot

I changed a line in modules/mono/build_scripts/mono_configure.py at line 243 to
    mono_lib_path = os.path.join(mono_root, "lib64")

And it finally found my mono library.

Even after pointing the build to my native library, and attempting to build it statically, it still shows me the same error as the opening issue. Its clear to me that symlinking won't solve anything, but without it, godot just searches for non-existent files it didn't create during the build but expects them to be there:
Godot Engine v3.4.4.stable.mono.custom_build.419e713a2 - https://godotengine.org
OpenGL ES 2.0 Renderer: GeForce GT 240/PCIe/SSE2
OpenGL ES Batching: ON

Mono: Log file is: '/home/tino/.local/share/godot/mono/mono_logs/2022-05-02_18.26.48_14477.log'

Unhandled Exception:
System.TypeInitializationException: The type initializer for 'Sys' threw an exception. ---> System.DllNotFoundException: /home/tino/3rdpartysoft/godot/bin/GodotSharp/Mono/lib/../lib64/libmono-native.so assembly:<unknown assembly> type:<unknown type> member:(null)
  at (wrapper managed-to-native) Interop+Sys.LChflagsCanSetHiddenFlag()
  at Interop+Sys..cctor () [0x00000] in <a6a5ba8fc13a4797a32a4dc4ae25c772>:0
   --- End of inner exception stack trace ---
  at System.IO.FileSystem.FileExists (System.ReadOnlySpan`1[T] fullPath, System.Int32 fileType, Interop+ErrorInfo& errorInfo) [0x00007] in <a6a5ba8fc13a4797a32a4dc4ae25c772>:0
  at System.IO.FileSystem.FileExists (System.ReadOnlySpan`1[T] fullPath) [0x00006] in <a6a5ba8fc13a4797a32a4dc4ae25c772>:0
  at System.IO.File.Exists (System.String path) [0x00043] in <a6a5ba8fc13a4797a32a4dc4ae25c772>:0
  at GodotTools.Utils.File.Exists (System.String path) [0x00007] in <2fbd5b7d279040959105492c48f93d40>:0
  at GodotTools.GodotSharpEditor.EnablePlugin () [0x00149] in <2fbd5b7d279040959105492c48f93d40>:0
ERROR: System.TypeInitializationException: The type initializer for 'Sys' threw an exception.
   at: debug_send_unhandled_exception_error (modules/mono/mono_gd/gd_mono_utils.cpp:369)

This file: /home/tino/3rdpartysoft/godot/bin/GodotSharp/Mono/lib/../lib64/libmono-native.so doesn't exist and I have no idea why it doesn't get created during the build.

libmono-native.so is not built within Godot, it is obtained from your distribution, in your case it should be /usr/lib64/libmono-native.so.
I don't know yet the reason why it tries to dynamically load this path but there is a small workaround for it:
Go to bin/GodotSharp/Mono, create lib64 folder and do the symlink inside the folder:
ln -s /usr/lib64/libmono-native.so lib64/libmono-native.so

Should fix it.


direct link to contact us is not showing on mobile

direct link to contact us is not showing on mobile  in the articles compared to how it should show on desktop
https://monosnap.com/file/EDrvCLtR5gYCHOph6Qy74fMNMfUjE0

Probably solved by https://github.com/kiwicom/smart-faq/pull/404

Possibly duplicate of #402

so I guess it can be closed, right? 🙂

Will be fixed by https://github.com/kiwicom/smart-faq/pull/466


109.event reason word limit

Description: Fix bug where user may enter more than 3 words in reason editText field
Closes #109
Proof of Concept

Checklist

[x] Follows style guidelines and is properly commented
[x] Code changes generate no new warnings
[x] Testing (Unit and/or Intent)
[x] Documentation (UML, Wiki, etc.)

Changes have been completed :)


passing vm-specific configs

Hey,
I'm trying to add "fuel" to the wasmtime engine used here, but there's apparently no way to pass engine configs. It'd be nice to have them. I can work on it if you accept contributions :)

PR is welcome!

Could you show us your API design before coding in it?

Sure. I'll send the design before I start coding it.


add dummy implementations of the upcoming getTransform/Clip to _MulticastCanvas

These dummy implementations are needed so that the new methods can be added in the ui.Canvas that lives in the engine repo. They will be populated with a proper implementation when that class is rolled into the framework.

See https://github.com/flutter/engine/pull/33308

(landing this would be harmless though, so still LGTM)


Update information to be accurate to the semantics of string literals

@InKryption Thanks for bringing this to my attention! You're right, what I've written there is misleading. On the other hand, this is syntax the learner hasn't seen yet.
I also tease strings here:
https://github.com/ratfactor/ziglings/blob/5a7c4d07793e7193911372ee34195dfc64dd7e70/exercises/054_manypointers.zig
    // Take a good look at the array type to which we're coercing
    // the zen12 string (the REAL nature of strings will be
    // revealed when we've learned some additional features):

And the big reveal is here:
https://github.com/ratfactor/ziglings/blob/dedd787f2dd0badc03caa1299adaf57eaeaae156/exercises/077_sentinels2.zig
// Are you ready for the THE TRUTH about Zig string literals?
//
// Here it is:
//
//     @TypeOf("foo") == *const [3:0]u8
//
// Which means a string literal is a "constant pointer to a
// zero-terminated (null-terminated) fixed-size array of u8".

So,
what do you think about leaving the syntax in that comment as-is, but then changing the explanation from
// Is the same as:
//
//     const foo = [_]u8{ 'H', 'e', 'l', 'l', 'o' };

to something like
// Is almost* the same as:
//
//     const foo = [_]u8{ 'H', 'e', 'l', 'l', 'o' };
//
// (* We'll see what Zig strings REALLY are in Exercise 77.)

to make it accurate, but not overwhelming?

@ratfactor
That seems reasonable to me. I ultimately made this PR off-the-cuff in response to a person being irked after being explained the actual semantics of string literals, while providing them assistance in the zig discord, so I have no strong attachment to this edit exactly. :)

@InKryption Thank you! ❤


Allow gestures to be disabled

It's very easy for me to accidentally swipe my screen and reset my timer. Combined with the fact that I cannot jump my timer to a specific time, I lose my current work session and have to start from the beginning.
This could be solved with either:

Adding a setting to disable gestures, or
Adding a setting to confirm "destructive" actions such as resetting the session.

I agree. In addition, I feel that the gestures are unintuitive, and I forget what they do after not using the app for awhile. I think that a set of standard play/pause/fastForward/skip buttons would be more intuitive.

Same here, too easy to accidentally swipe, and sometimes you don't even notice that you lost your place and the timer reset, since it keeps running.

I also agree here, gestures are very inintuitive and destructive, for example: if swipe up adds a minute, wouldn't it make sense for swipe down to reduce a minute?  This behavior means that if you accidently make a gesture, even if you realize the mistake, it's very confusing how to undo it.
Since swipe left/right is also destructive, so the best alternative would probably be to disable them.

The next major update will feature a nicer UX for gestures and I will consider adding buttons as an alternative.


Align Filters Section to Category Cards

Increase the width of the filter element to match the width of category cards :)

Thanks! for submitting this issue!

Hi! I am looking for a project to do my first open-source contribution for a class. Do you mind if I try to fix this issue and submit a PR?

Absolutely! Tag me when you open up a PR.
Haven't had a chance to fully write out contribution guidelines, but you can use npx serve . at your repo root to get the test server running.


pointerWithin is not exported

I think pointerWithin is exactly what I need right now, having created a snapCornerToCursor modifier.
I see the type definitions, but it does not appear that it is exported properly from @dnd-kit/core
Version: @dnd-kit/core 5.0.1

Nevermind, had a corrupted build of node_modules


Fatal JavaScript out of memory on Deno

Really enjoying using this library, but lately I have trouble running the latest versions (0.0.9, 0.0.10) on Deno (versions 1.45.x, 1.46.x, 2.0.x)
Given:
// @deno-types="https://cdn.jsdelivr.net/npm/wasm-vips/lib/vips.d.ts"
import Vips from 'https://cdn.jsdelivr.net/npm/wasm-vips/lib/vips-es6.js';

const vips = await Vips({
    dynamicLibraries: [],
});

The output is:
error: Uncaught (in worker "em-pthread")(in promise) RangeError: WebAssembly.Memory(): could not allocate memory
    at https://cdn.jsdelivr.net/npm/wasm-vips/lib/vips-es6.js:15:84
    at https://cdn.jsdelivr.net/npm/wasm-vips/lib/vips-es6.js:242:14

#
# Fatal JavaScript out of memory:...

Things I've tried:

reducing the INITIAL_MEMORY:

const vips = await Vips({
    dynamicLibraries: [],
    INITIAL_MEMORY: 512 * 1024 * 128,
});

adding --v8-flags=--max-old-space-size=4096

Version 0.0.8 and before are working great.

I was able to reproduce the issue, and confirm that it only occurs with version 0.0.9 or later.
Commit https://github.com/kleisauke/emscripten/commit/67924ad25146e925cc8d43b46d81320a4b058389 fixes this, this will be in v0.0.11. Thanks for reporting this!

Upstream PR: https://github.com/emscripten-core/emscripten/pull/22778.

v0.0.10 is now available.


Application on non-temporal data

Thank you for your awesome work!
I would like to know whether it is possible to modify this code to be able to use with non-temporal data.
That is, just 1d data with a certain batch size and number of channels.

I think the code is not suitable for non-temporal data because BEST-RQ framework is a pre-training method for speech recognition. You should look for models or training methods that is suitable for the data you have.


[fix] cms: sest_children_state

Coverage decreased (-4.3%) to 80.251% when pulling 374d0241f95b8145484c05e8fa98cb5e0811f6b4 on fix-cms_folder_for_member into 43d1dbc367d1de1ecd8ab43b5caef558e75e7bf3 on master.


Use of uninitialized value in anonymous hash ({}) at C:/Strawberry/perl/site/lib/Map/Tube/Plugin/Graph.pm line 133.

map-tube --map=Berlin --generate_map
produces a proper map image (as far as I can tell), but it issues also the above warning. The same happens for NYC and Singapore
The same warning is issued for other maps, too, but with worse consequences: Nanjing, NizhnyNovgorod, Novosibirsk, and Tbilisi exhibit the same warning but also produce empty maps. (More exactly: they produce images that contain only the header line but no map.)

@gwselke Thank you for your efforts.
As far as Berlin map is concerned, I clearly see issue in the map data.
Unfortunately the current test doesn't catch the issue. I found nodes having invalid line id.
This should have been picked right at the beginning.

@gwselke Just to let you know, I have released patch to handle the invalid station line id.
https://metacpan.org/release/MANWAR/Map-Tube-3.81

@gwselke Thank you for your efforts.
As far as Berlin map is concerned, I clearly see issue in the map data.
Yes, Bielefeld data do have an issue!

That is a good thing! It's very easy, in fact, to get some detail wrong in a map, so it is a great help to have these caught by the Map::Tube infrastructure.


feat(mge/third_party): update cutlass version

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@luzzyzhang @xxr3376


Force using runc in F30

Suspect crun is being pulled in by installing podman RPM at F30 VM build time.  Attempting to fix that, and also force use of runc for Fedora < 31 at runtime.
Ref: https://github.com/containers/libpod/pull/5333#issuecomment-592048270

Wouldn't it be simpler to remove handle_crun

Ohohoh snap, I forgot about that damn thing, thanks for the reminder!  Yes, we need that also.
Waiting to push it's removal until VM build finishes, assuming it will pass.  It might fail due to dependency resolution on attempted removal of crun...

I don't really understand that logic anyway, since AIUI podman configures itself properly at install-time.

FWIW/IIRC: That was added b/c podman RPM brought one configuration and installing from source needed another.  So it was necessary while we had giuseppe's special 'F30 + crun' task going (since removed).

OMG, WTH happened, ugg.  I think maybe we still need to customize libpod.conf for F31.  Checking a VM image manually...

could I suggest switching depending on cgroups version?

Considering this now, since it applies across distributions and versions.

So...back to an earlier fear of mine, https://github.com/containers/libpod/pull/5342/checks?check_run_id=473384963 is showing that removal of 'crun' package (or something else) in F30 is also yanking out the podman RPM.  We need that for this particular task.  Investigating...

...confirmed.  The issue comes from podman in updates-testing, so we have a packaging problem.  Invoking the packaging $DIETY @lsm5, bowing to his greatness, and getting out my workaround-duct-tape.  Note: I tried this w/o updates-testing on F30, and podman pulls in runc as expected.

@cevich https://bodhi.fedoraproject.org/updates/FEDORA-2020-89326b1795 has Suggests: crun for f30.

Damnit! So now the bindings task in F31 is getting things wrong, despite forcing use of crun.  Did something else change between building VM images on Tuesday and Today?
https://cirrus-ci.com/task/5152549762236416?command=integration_test#L625

Errr wait, that's not the problem, it says level=warning.  Hrmmm...

Re-running flaked tests..

And again...

/approve
LGTM

/lgtm


Deflate Issue (Somewhere....)

OK, so please forgive this rather headscratching issue.
I've been digging at this for a couple of days now, and have come to the conclusion that we've got a bug in the Deflate decompressor somewhere.
This bug also appears to exist in the Microsoft Deflate stream decompressor, but not in the current Zlib.
Some background:
I maintain openBVE (game). One of the things this has, is an importer for Microsoft DirectX formatted objects.
I've discovered one object which is in compressed DirectX binary format, which simply fails to import.
In simplified terms, the DirectX binary format consists of 26 bytes of header, followed by deflate compressed data.
Broken File:
treeSet2.zip
Decompressing the stream data in this one returns 32768 bytes of data.
If I dump the header and decompress with Zlib, I get 35712 bytes of data.
(n.b. The two streams are identical until the first chokes)
Reconstructing the file in textual format tells me that the decompressor chokes when it encounters the following 4-byte single: 0.858470
Anyone with thoughts would be most welcome.

Without doing any digging: this sounds like a gzip file?
I guess that doesn't make any real difference.

If you plugged in another Deflate implementation does it work?
Also, I don’t use the Microsoft deflate implementation that’s in the framework.  This one came from DotNetZip.

Looking at the referenced issue from OpenBVE, it could be you’re right about the zlib implementations.  If there’s a newer implementation in C# I’m happy to use it

Hmm. From some further reading around and digging, I get the nasty suspicion that these may actually be MSZIP (cab) compression.
https://msdn.microsoft.com/library/bb417343.aspx#microsoftmszipdatacompressionformat
If I understand the implications from that documentation correctly, they're deflate, but flush all buffers other than the history buffer after each block.
Not sure that helps any, archive math is not an area I've ever had any major involvement in.

OK, so I've managed to get a working example using a slightly modified version DotNetZip
This is the main byte array extractor code, which plumbs into a very slightly modified DotNetZip Zlib implementation:
https://github.com/leezer3/OpenBVE/blob/b6e937b79c0ba1fc07d0b8d3ff0c47f74363f482/source/Plugins/Formats.DirectX/MsZip.cs
Essentially, for a MSZIP stream, we must read each compressed block (which must decompress to 32k or less data), and then unconditionally pass the decompressed data as the dictionary to our decoder for the next block.
This required a slight modification to the set dictionary method from DotNetZip, as in the default form it checks whether it is required & whether the adler of the dictionary is correct; Both of these must be ignored.
Porting these changes to SharpCompress woudn't be too difficult, but it would need a new stream class, e.g. MSZip stream which then performed the appropriate length checks and stuff internally, as SharpCompress doesn't expose the inflater by default; I presume this was a design choice?
Further, I don't see any easy way to do much error checking on this, other than block length checks & a  check on the output against a known value, as supplying a 'wrong' dictionary in this manner appears to work, but produces garbled data.

I’d probably rather have a custom Stream class like GZipStream instead of exposing the zlib inflater directly just for ease of use.
I think I have memories of using WinZip on cab files that are corrupted and didn’t know until too late.  Seems like no error checking is a “feature.”
I haven’t looked at the code yet.
If you feel like making a PR for a MsZipStream then it would appreciated :)


Integrate Eclipse Kuksa.VAL - GPS Feeder

The GPS feeder of Kuksa provides GPS coordinates as Vehicle Signal in the Data Broker according to the Vehicle Signal Specification.
The GPS feeder uses a locally running gpsd, which needs to be preinstalled into the Leda distribution.
Default configuration will not work out of the box, as the configuration depends on the specific hardware attached to the device. On raspi, that may be a GPS connected via USB or serial. On qemu, we may want to use a GPS faker application to simulate the location.

This is has been blocked due to compilation errors. Needs investigation.

For the default configuration, Leda should use a "sane default".
In case of gpsd, it seems we should be using /dev/ttyUSB0 as default, as it supports most of the GPS devices out of the box.

According to your comment here gpsd should run inside the container and not on the host, correct?

I would say yes, as it is an optional feature from an SDV point of view, and it's easier to configure at runtime via the container runtime.
Since our current supported HW platform is not static (eg raspi gps module can be attached and is not built in)
But i think it would be acceptable tradeoff to have gpsd preinstalled natively, once we know how to dynamically change the config at runtime through Desired State. Since that is conceptually not clear yet.
So with sane default for Raspi, we could just start by preinstalling it on the sdv-image packagegroup additions.
We could use this issue also for testing out the bitbake-built container images.
wdyt?


SQL History: Double click to copy it to Query Editor

SQL History is an awesome feature and is a must to have for everyone.
Now I'm missing resources to restore a past query in a more productive way.
First, there is no "Copy" option on the context menu, and Ctrl+C also doesn't work if a History Query Item is selected.
Second, I would love to have a "fast recover" way like this:

ME as an Azure Data Studio user
WHEN I'm at Query Editor
AND Double click at any Query History item
THEN the selected query should appear at Query Editor at the current cursor position;

My setup:

Thanks for the suggestion! We've had a few asks for double click instead to open the query in a new query window - would you prefer your suggestion over that assuming we implemented the ctrl + c copy to copy the text from the selecting cell? (which you could then paste into an existing document)

Hi @Charles-Gagnon
That would be a great start, for sure.
My expectation would be to double click and have it at cursor position at the editor. Maybe this behavior - double click to past at cursor position - could be toggled at settings.

I'm going to close this as a dupe of #7437 to consolidate the issues we're tracking for this feature.


Delegate Mission Request: Integration of Optimism Gov and RGPF into University Courses,

Delegate Mission Request: Making Impact Evaluation Accessible
Delegate Mission Request Summary:
This mission proposal centers around building partnerships with academics who are already at the vanguard of incorporating blockchain technology, specifically Ethereum, into their university courses. This mission will strengthen this ongoing commitment by introducing a specialized module focusing on Optimism and RPGF (Public Goods Funding) within the Ethereum ecosystem.
S5 Intent 6 : 4 Contributions that educate the broader community about Optimism governance and RetroPGF
Proposing Delegate: Jack Anorak 3
Proposal Tier 2: Fledgling
Baseline grant amount: 100,000 OP (20,000 OP per course, includes rewards to students). Mission management chooses to forego rewards in lieu of future RPGF rewards.
Should this Foundation Mission be fulfilled by one or multiple applicants: Multiple
Completion date: By Dec 31, 2024
Apply Here
Specification
How will this Delegate Mission Request help accomplish the above Intent?
This mission will broaden the reach of Optimism’s work in governance by introducing a specialized module focused on Optimism Governance and RetroPGF. This module not only aims to provide students with a nuanced understanding of OP Governance and RetroPGF but also seeks to create a framework that rewards and recognizes the dedicated efforts of these educators while doing outreach to onboard more educators around the globe.
The mission will result in an Open Source Syllabus and playbook allowing other educators to replicate anywhere in the world.
Onchain Impact report including a registry of wallets, onchain attestations from students and a case study. This reporting will allow anyone to evaluate the impact of the mission based on onchain metrics, quality of syllabus, and content shared on socials.
What is required to execute this Delegate Mission Request?

Minimum of three academics at three different institutions collaborating on open source syllabus
Reporting Progress through socials and gov forum
Outreach and onboarding of other educators focusing on similar topics
Onchain attestations from students rating experience
Onchain attestation from academics rating student participation and performance
Case Study co authored by academics
Database of onchain activities and txns

How should the Token House measure progress towards this Mission?

These measures should focus on progress towards completion. Including expected completion dates for each is recommended
Please be as specific as possible in defining measures of progress so that Token House delegates can accurately track execution
There should also be some measure of quality measurement to ensure there is a fully complete syllabus; administrators will be responsible for a rubric listing all of the focus areas to be covered

How should badgeholders measure impact upon completion of this Mission?

Progress towards mission should be rated on progress reports from academics.
This progress should commensurate with onchain activity demanded by the syllabus.

These projects were selected for this mission request
https://app.charmverse.io/op-grants/cenfotec-optimism-in-costa-rica-university-36967085286006984
https://app.charmverse.io/op-grants/optimism-at-rio-de-janeiro-federal-universities-a-collaboration-with-the-blockchain-innovation-hub-6395457446717472


Support pin-depends

Say I have two packages:

A
B, which depends on the master branch of A in opam's pin-depends field.

Now I have the following scenario:

Commit to B, CI runs normally.
Commit to A, adding a depext D.
Commit to B.
When the CI runs for that last commit, it fails during A install because of missing system dependency (it did not install D).

It seems that the pin is cached by the CI during the depext stage, and can't detect upstream changes, since the pin is against a branch name instead of a commit, and the pinned package is not pulled at this moment.
/cc @Julow

We tried to study a simple/naive fix yesterday:
We could use git ls-remote to query the last rev of each pin-depends (in a job placed outside of Analyse's cache). The rev could then be inserted in the command installing them, invalidating Docker's cache if that changes.
The problem is that opam files and opam show are very hard to parse. To solve that we should use Opam's opam-format library.

Note: another way to depend on dev versions is to use git submodule to add the exact commit you want to your project. ocaml-ci will automatically fetch these submodules (using git clone --recursive) and dune will automatically build against them.

Note: another way to depend on dev versions is to use git submodule to add the exact commit you want to your project. ocaml-ci will automatically fetch these submodules (using git clone --recursive) and dune will automatically build against them.

If we wanted to depend on a specific commit we could also pin-depend on that specific commit, and the CI works well in that case too.

Note that pin-depends is broken at the moment as the new solver (in #167) only has access to the Git clone and opam-repository. I guess we'll need to implement fetching of pin-depends manually somehow.

Title changed to reflect that this is now a more general issue.

pin-depends with a commit hash is working again now. I don't think we'll support pinning to branches any time soon.

It seems like there is an issue when pinning to a repo where the opam file is not at the root (maybe it's unrelated to pinning?), it can't find the opam file thus fails during analysis.

Do you have a link to an example? ocaml-ci currently assumes the opam file is at /$name.opam, although I think opam also allows some other forms.

Here's one: https://ci.ocamllabs.io/github/vocal-project/vocal/commit/363a91d49e1e89c202d5656c088f3a8ca538f68c/variant/(analysis)
The opam file is under a opam/ folder

I've pushed a fix - should be working now!


a version of the python

Will the author write a version of the python

No, this is a JavaScript project. The encoding is pretty straightforward so it would be very simple to port it to Python, but I'm not going to do it. Thanks for your question.

@zerorzeror there do seem to be some Python implementations of base2048, e.g. https://github.com/ionite34/base2048 -- thx for the reminder to give it a whirl!


Improvement to docs: ability to dedicate nodes to an app

I would like to have the ability to specify that I want X number of nodes dedicated to my app.  I believe this is possible - just not clear on how to do it in the docs.

Looks like the old version of the docs has it but not the new, http://docs.deis.io/en/latest/using_deis/manage-application/
deis tags is what you need, however you will have to add labels to each Kubernetes Node for it to match up

There is already documentation about deis tags in the "Security Considerations" section of the documentation. https://deis.com/docs/workflow/managing-workflow/security-considerations/#application-runtime-segregation
We should probably add a link to this page from within the "Managing Applications" to give more visibility to the docs that are already there.

Related: https://github.com/deis/workflow/issues/401


doc: Updated description grammar in mikro-bus.yaml file

Fixed some of the wording within the description section of the mikro-bus.yaml file.

@shaggygi thanks for the fix, could you reword the commit message to match the contribution guidelines? https://docs.zephyrproject.org/latest/contribute/guidelines.html#commit-guidelines

could you reword the commit message to match the contribution guidelines?

My apologies as it is my first Zephyr PR.  Have another look for me.  Thx.

could you reword the commit message to match the contribution guidelines?

My apologies as it is my first Zephyr PR.   Pushed just now.  Thx

@fabiobaltieri Updated, but forgot the sign-off. Pushed just now. Thx

Nearly there :-) break commit comment on 75 columns:
 3: UC4 Commit message body line exceeds max length (84>75): "Fixed some of the wording within the description section of the mikro-bus.yaml file."

@fabiobaltieri Updated, but forgot the sign-off. Pushed just now. Thx

Nearly there :-) break commit comment on 75 columns:
 3: UC4 Commit message body line exceeds max length (84>75): "Fixed some of the wording within the description section of the mikro-bus.yaml file."

Well blame.  🤕  Please hold 😄

@fabiobaltieri sorry for my troubles, thanks for your patience.


Support for the file with multibyte filename.

When I upload the file which jas Japanese filename, uploaded file has name which omit Japanese character.
{Japanese word}.txt yileds to .txt
foo{Japanese word}.jpg yield to foo.jpg

Thanks for reporting this, @w4yh!
This one is already open #321, which actually is going to have a fixed merged in the near future (see #520).

The fix has been merged to master and thus should be in the next release.


feat(module:resizable): add direction parameter in NzResizeEvent

PR Checklist
Please check if your PR fulfills the following requirements:

[✅] The commit message follows our guidelines: https://github.com/NG-ZORRO/ng-zorro-antd/blob/master/CONTRIBUTING.md#commit
[✅] Tests for the changes have been added (for bug fixes / features)
[✅] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[x] Documentation content changes
[ ] Application (the showcase website) / infrastructure changes
[ ] Other... Please describe:

What is the current behavior?

Issue Number: #7986
What is the new behavior?
NzResizeEvent will now contains a direction parameter with type NzResizeDirection
Does this PR introduce a breaking change?
[ ] Yes
[x] No

Other information

是否要对预览的ghostElement进行适配，目前示例里面不管是拖动左边还是右边的，预览效果都是右边变化
previewResize({ width, height, direction }: NzResizeEvent): void {
    this.createGhostElement();
    this.renderer.setStyle(this.ghostElement, 'width', `${width}px`);
    this.renderer.setStyle(this.ghostElement, 'height', `${height}px`);
    //todo adapt to the direction
  }


bots: Improve bots profile summary.

Clarify the "Bot" user in the bot profile summary.
Display "Manage bot" modal from bots profile summary.
Display the "Deactivate bot" button inside the bot edit modal.
Fixes: #22482

Screenshots and screen captures:

Screenshots

Gif

**Self-review checklist**

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

Merged, thanks for doing this @yogesh-sirsat!


add encryption library(s) for IDEA, DES, 3DES and AES

Confirmation:

Look for encryption library(s) for all the encryption methods we want to provide
Add encryption library(s) to encryptio.py
Add encryption library to requirements.txt


Update date in index.html

Now no one will ask if it's still active ;)

That's right! :-) :-)

:beers:


#1535 generating test artifacts with a specified size for Maven

Pull Request Description
Annotation used in tests improved to jar generation with given size.
This pull request closes #1535
Acceptance Test

[x] Building the code with mvn clean install -Dintegration.tests still works.
[x] Running mvn spring-boot:run in the strongbox-web-core still starts up the application correctly.
[x] Building the code and running the strongbox-distribution from a zip or tar.gz still works.
[x] The tests in the strongbox-web-integration-tests still run properly.

Questions

Does this pull request break backward compatibility?

[ ] Yes
[x] No

Does this pull request require other pull requests to be merged first?

[ ] Yes, please see #...
[x] No

Does this require an update of the documentation?

[ ] Yes, please see strongbox/strongbox-docs#{PR_NUMBER}
[x] No

Hi @wpater !
Thank you for your contribution!
Would you mind signing the ICLA, as described in the Contributing page?
Kind regards,
Martin

Thanks for signing the ICLA and for your contribution! Congratulations on your first pull request! :)


Update navbars example to include a centered nav with utilities

Addresses #36486.

In the above example, the entire navbar collapses. I think there may be a simpler approach to getting a truly centered Navbar with left and right align-items. See my reply, which includes a Codepen example of the basic concept: https://github.com/twbs/bootstrap/discussions/36486#discussioncomment-2903439).
Also, I would follow the example navbarTogglerDemo02 where only the menu and right-align item collapses, not the navbar-brand: https://getbootstrap.com/docs/5.2/components/navbar/#content

Also, see my codepen example for a potentially different approach to center the menu, using flex-1 on both left and right items only using the BS class "col" without specifying a number suffix. #36486 (comment)

I have provided a Codepen of this approach applied to a Bootstrap navbar. See comment, https://github.com/twbs/bootstrap/discussions/36486#discussioncomment-2917883


[Docker] Dockerize the app

Setting up a development environment can be cumbersome and almost all of the time redundant. Docker will make the developer experience seamless and we can focus on developing rather than fixing build errors.

Closing against #4


Make links to headers highlight the header after scrolling to it

Right now it is sometimes hard to find the header on screen after clicking on a link directing to that header. There should be an option to highlight headers after they are navigated to temporarily so they can be visually located easily.

This should be fixed in 1.8.0. Let me know if it still does not work as expected and reopen the issue. If you find any new issues please submit a new issue :)
Thanks!


Astro update

Bumping astro to keep up with the changes

Hey @jerzakm !
This is a cool update from Astro's community 🎉 I'm seeing a reduction in kb transferred 😄
Are there any other details to look into?
I noticed view transitions commented out. Hopefully that's coming back? Although I'm not noticing it within our current live so 🤷♂


feat: Support iOS 17 text content types

Summary:
While setting up a credit card form in RN, I discovered that iOS 17 supports a number of new content types (UITextContentType docs). In the docs these are marked with the Beta flag.
Setting up the new content types is relatively straightforward, but a change is required in https://github.com/facebook/react-native-deprecated-modules to update the TextInput prop types. I will open a PR in that repo shortly. I have opened a PR to update the prop types. Once that PR is merged, a version bump for that dependency will need to be added to this PR.
Changelog:

Test Plan:
The rn-tester app builds and runs successfully. I have added a few new examples of inputs using the new text content types.

Hmm, I think that something in the CI or in the rn-tester project config needs to be updated to handle iOS 17

Hmm, I think that something in the CI or in the rn-tester project config needs to be updated to handle iOS 17

Looks like the circleci config uses xcode 14.3 https://github.com/facebook/react-native/blob/44a507fdcf9068df40c0debbbf317b0c64988ad4/.circleci/config.yml#L49C14-L49C14
15.0.0 is supported on circleci https://circleci.com/docs/testing-ios/#supported-xcode-versions

Hmm, I think that something in the CI or in the rn-tester project config needs to be updated to handle iOS 17

Also, thinking about this, I'm not sure this change could be merged until XCode 15 is at a stable release, it would be required to build these changes, and even then it may be desirable to have a gap between XCode 15 release and the requirement to build RN apps using it 🤔

@philIip I've added compile time checks. 🤞 the circleci checks pass now.
The last remaining known issue is the version of react-native-deprecated-modules that is being used.
Once the recently merged PR there is published in a new version, I'll update the version in react-native.
Would you like me to prep a PR that bumps the minor version number and updates the changelog of react-native-deprecated-modules?

Would you like me to prep a PR that bumps the minor version number and updates the changelog of react-native-deprecated-modules?

In an effort to be helpful, I've opened a PR on react-native-deprecated-modules to bump the version and update the changelog. Once merged and released I'll make the final change on the PR to use the new react-native-deprecated-modules version.

PR for the docs website update - https://github.com/facebook/react-native-website/pull/3804


ULP stage_inc and jumps (IDFGH-2325)

Hi,
i am trying to write simple ULP app that would be running in deep sleep and i found that neither sleep can be use in deep sleep, which is clearly described in documentation and jumps in conjunction with stage_inc, which i didnt find any info about it.
https://docs.espressif.com/projects/esp-idf/en/v3.2/api-guides/ulp_instruction_set.html#stage-inc-increment-stage-count-register
Now is the question, is it bug or it is normal and jumps cant be used in deep sleep?
I have to add that ULP code works if is not used in deep sleep mode.
I am using arduino with ULP (esp-idf v3.2).
Thanks.

@chegewara can you please clarify how you are using the stage counter? Are you counting within one run of the ULP program, or you halt the program and then use the counter when the ULP wakes up again? Also, please report which ULP assembler version you are using (run esp32ulp-elf-as --version to check).

I am using ULP in arduino with this ulp toolchain:
https://github.com/duff2013/ulptool
This is version output:
./esp32ulp-elf-as --version
GNU assembler (GNU Binutils) 2.28.51-esp-20190801
Copyright (C) 2017 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `esp32ulp-elf'.

xtensa-esp32-elf-as --version
GNU assembler (crosstool-NG crosstool-ng-1.22.0-80-g6c4433a) 2.25.1
Copyright (C) 2014 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `xtensa-esp32-elf'.

During my learning process i found few issues.

is with stage_inc and jumps, it is working as long as i start ulp program without entering deep sleep mode,  if i remember test code was something based on this sketch:
https://github.com/duff2013/ulptool/blob/master/src/ulp_examples/ulp_adc/adc.s

In my other test code i am reading adc in deep sleep ULP code and i am using analogRead() in main app code, but few hours ago i found this is wrong. It is causing some very odd behavior. And by odd i mean that i have ULP variable which is increased by 1 in every timer reset with halt, it is working until variable value is multiply of 32. This is ULP code and i know it is not perfect:

#include "soc/rtc_cntl_reg.h"
#include "soc/rtc_io_reg.h"
#include "soc/soc_ulp.h"

  /* Define variables, which go into .bss section (zero-initialized data) */
  .bss

  /* ADC1 channel 7, GPIO35 */
  .set adc_channel, 7

  .global toggle_counter
toggle_counter:
  .long 0

  .global voltage
voltage:
  .long 0

  /* Code goes into .text section */
  .text
  .global entry
entry:

  and r0, r0, 0
  /* Read toggle counter */
  move r3, toggle_counter
  ld r0, r3, 0
  /* Increment */
  add r0, r0, 1
  /* Save counter in memory */
  st r0, r3, 0

  and r0, r0, 31
  jump toggle_set, eq

  /* Set the RTC_GPIO27 output LOW (clear output) */
  WRITE_RTC_REG(RTC_GPIO_OUT_W1TC_REG,RTC_GPIO_OUT_DATA_W1TC_S+10,1,1)
  WRITE_RTC_REG(RTC_GPIO_OUT_W1TC_REG,RTC_GPIO_OUT_DATA_W1TC_S+13,1,1)
  jump toggle_complete

  .global toggle_set
toggle_set:
  move r2, voltage
  ld r0, r2, 0
  and r0, r0, 0
  /* read adc1 channel 7 + 1 to register R1*/
  adc r1, 0, adc_channel + 1
  /* store ADC value to voltage */
  st r1, r2, 0
  /* load R1 to R0 */
  ld r0, r1, 0
  jumpr alert, 3825, lt
  /* Set the RTC_GPIO27 output HIGH (set output) */
  WRITE_RTC_REG(RTC_GPIO_OUT_W1TS_REG,RTC_GPIO_OUT_DATA_W1TS_S+13,1,1)
  jump toggle_complete

  .global alert
alert:
  WRITE_RTC_REG(RTC_GPIO_OUT_W1TS_REG,RTC_GPIO_OUT_DATA_W1TS_S+10,1,1)

  .global toggle_complete
toggle_complete:

  /* Get ULP back to sleep */
  .global exit
exit:
  halt

To see when it stuck i start ulp program without entering deep sleep and printing toggle_counter value.
Summarize, i am more suspecting my lack of knowledge in using ULP rather than bug, but i thought i will report it. In worst case i will learn something.
Thanks.

I will close this issue for now, because it seems that i dont fully understand ULP logic. I will open it when i confirm problems.

Thanks @igrr for suggestion. Like i said, i am learning ULP hard way, and i thought that stage value is preserved after resting timer/program with halt, but its not.
Also other problems i mentioned here were caused by my lack of knowledge. For example ive been trying to use ADC in ULP and analogRead() in main app, then i found this:
This function reconfigures ADC1 to be controlled by the ULP.
 * Effect of this function can be reverted using adc1_get_raw function.

and i understood i have to call adc1_ulp_enable() each time before entering deep sleep and running ULP.


All possible Unique permutations of a given String.

Is your feature request related to a problem? Please describe.
A string is given we have to generate all its possible permutations. A permutation of the string is defined as a word that contains all the characters of the original string but in any order. The output should contain only unique words in lexicographically sorted order.
Describe the solution you'd like
The solution is based on recursion and word is generated from left to right. A word is generated by placing a character from available at the current position and solving for subproblem.
For example,
(a) abc => abc, acb, bac, bca, cab, cba
(b) aba => aab,aba,baa
Do you want to work on it
Yes, and I would like to use C++ for this contribution under GSSOC21.

Can someone explain what is exactly needed to be done for following instruction ?
-Edit the corresponding README.md file to add the link to your code in the corresponding section in alphabetical order.

I would like to contribute to the above problem in java as I am a participant of GSSOC'21. Kindly assign this to me.

Hi,
I am GSSOC'21 participant,
My code will be in python with proper comments and test cases.
@tarun26091999 @rudrakshi99 Please assign me this. Thankyou!!

Hello, @shubhidua @unicode-Ind @Sathwika-02 @aaadddiii @anubhavitis, I think this is an important question and this should definitely be added in the Algo-Tree and I will start working on it as soon as I get assigned!! as I am a part of GSSoC'21, so I would like to implement this in c++.

@unicode-Ind @shubhidua @Sathwika-02 assigned.

Hi,
I would like to solve this issue in C++ with proper test cases and comments.
Please assign this issue to me if its not already solved.

@Srishti013 for C++ it's already done. Thanks for your interest!

I would like to contribute to the above code in java. Kindly assign this to me. Thanks.


Remote end closed connection without response after successful headless log in with v2.8

Describe the bug
After the fixes in https://github.com/mintapi/mintapi/pull/552, running headless locally on macOS led to urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) after signin.
What version of MintAPI are you using?
Local checkout of https://github.com/mintapi/mintapi/pull/552
What command[s] did you run / steps to reproduce?
mintapi --headless --beta --use-chromedriver-on-path username password

Stacktrace/error received:
Traceback (most recent call last):
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/Users/broz/.pyenv/versions/3.8.14/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/Users/broz/.pyenv/versions/3.8.14/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/Users/broz/.pyenv/versions/3.8.14/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/broz/.pyenv/versions/mintapi-3.8.14/bin/mintapi", line 33, in <module>
    sys.exit(load_entry_point('mintapi', 'console_scripts', 'mintapi')())
  File "/Users/broz/src/mintapi/mintapi/cli.py", line 518, in main
    data = mint.get_account_data(limit=options.limit)
  File "/Users/broz/src/mintapi/mintapi/api.py", line 265, in get_account_data
    return self.get_data(
  File "/Users/broz/src/mintapi/mintapi/api.py", line 245, in get_data
    data = self.__get_mint_endpoint(endpoint, **kwargs)
  File "/Users/broz/src/mintapi/mintapi/api.py", line 452, in __get_mint_endpoint
    response = self.get(
  File "/Users/broz/src/mintapi/mintapi/api.py", line 162, in get
    return self.driver.request("GET", url, **kwargs)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/seleniumrequests/request.py", line 159, in request
    self.requests_session.headers = get_webdriver_request_headers(self, proxy_host=self.__proxy_host)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/seleniumrequests/request.py", line 76, in get_webdriver_request_headers
    webdriver.switch_to.window(original_window_handle)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/selenium/webdriver/remote/switch_to.py", line 134, in window
    self._w3c_window(window_name)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/selenium/webdriver/remote/switch_to.py", line 142, in _w3c_window
    send_handle(window_name)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/selenium/webdriver/remote/switch_to.py", line 138, in send_handle
    self._driver.execute(Command.SWITCH_TO_WINDOW, {'handle': h})
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 427, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/selenium/webdriver/remote/remote_connection.py", line 344, in execute
    return self._request(command_info[0], url, body=data)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/selenium/webdriver/remote/remote_connection.py", line 366, in _request
    response = self._conn.request(method, url, body=body, headers=headers)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/Users/broz/.pyenv/versions/3.8.14/envs/mintapi-3.8.14/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/Users/broz/.pyenv/versions/3.8.14/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/Users/broz/.pyenv/versions/3.8.14/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/Users/broz/.pyenv/versions/3.8.14/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

What did you expect to happen?
Standard output
What actually happened?
Above exception
Additional context
As noted in https://github.com/mintapi/mintapi/pull/552, I appear to have some new HTML, at least on the signin side.
When not running headless, the error looks like https://github.com/mintapi/mintapi/issues/546 - so I would guess that this is related or a different way to reproduce the same error.

I am seeing this same error, also headless

Can you please try 2.9 from pypi and let me know if that resolves the issue? Thanks!

I will try 2.9 but I have been seeing this issue on main

@mrooney Same issue on 2.9
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/mintapi", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/mintapi/cli.py", line 518, in main
    data = mint.get_account_data(limit=options.limit)
  File "/usr/local/lib/python3.9/dist-packages/mintapi/api.py", line 262, in get_account_data
    return self.get_data(
  File "/usr/local/lib/python3.9/dist-packages/mintapi/api.py", line 242, in get_data
    data = self.__get_mint_endpoint(endpoint, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/mintapi/api.py", line 449, in __get_mint_endpoint
    response = self.get(
  File "/usr/local/lib/python3.9/dist-packages/mintapi/api.py", line 162, in get
    return self.driver.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/seleniumrequests/request.py", line 159, in request
    self.requests_session.headers = get_webdriver_request_headers(self, proxy_host=self.__proxy_host)
  File "/usr/local/lib/python3.9/dist-packages/seleniumrequests/request.py", line 76, in get_webdriver_request_headers
    webdriver.switch_to.window(original_window_handle)
  File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/switch_to.py", line 134, in window
    self._w3c_window(window_name)
  File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/switch_to.py", line 142, in _w3c_window
    send_handle(window_name)
  File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/switch_to.py", line 138, in send_handle
    self._driver.execute(Command.SWITCH_TO_WINDOW, {'handle': h})
  File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py", line 427, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/remote_connection.py", line 344, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/remote_connection.py", line 366, in _request
    response = self._conn.request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

@mrooney Confirmed that 2.9 does not resolve.  This occurs post-login, and not obviously tied to CSS selectors, so that is as expected.

Have been doing some experimentation. The problem seems to only exist on chromedriver 106. If I revert to chromedriver 105 the problem goes away.

I am also experiencing this issue on Ubuntu.

Confirming that 2.9 with a reversion to 105.0.5195.52 eliminates the issue when running in interactive mode. When running with --headless I'm oddly getting prompted for the "Mint e-mail:" despite passing it on the command line. Removing the --headless flag allows the command to proceed normally.
@ddollar and @pblesi were you testing in interactive or headless mode?

This just started feeling on me again, November 1. The complaint now appears to be that Chrome is a different version down the Chromedriver. Do those two need to be in lockstep from aversion, standpoint? And if so, is there a clear way to specify a separate Chrome app instance to use or will it always use the system version?

I'm having an issue similar.

package
version

mintapi
2.9

selenium-requests
2.0.0

configargparse
1.5.3

pandas
1.5.1

keyring
23.5.0

xmltodict
0.13.0

oathtool
2.3.1

requests
2.28.1

selenium
4.6.0

python-dateutil
2.8.1

pytz
2022.1

numpy
1.23.2

trio
0.21.0

certifi
2022.6.15

urllib3[socks]
1.26.5

trio-websocket
0.9.2

tldextract
3.3.1

charset-normalizer
2.1.1

idna
3.3

autocommand
2.2.1

path
16.4.0

filelock
3.8.0

requests-file
1.5.1

sortedcontainers
2.4.0

attrs
22.1.0

async-generator
1.10

sniffio
1.3.0

outcome
1.2.0

wsproto
1.2.0

PySocks
1.7.1

six
1.16.0

h11
0.13.0

mintapi --headless --no_wait_for_sync --accounts --transactions --include-investment --filename $mintTempDirectory my_username my_password

https://github.com/cryzed/Selenium-Requests/issues/53 has been fixed and released in selenium-requests 2.0.1.  I verified it resolves this issue for me.  Run pip install --upgrade mintapi 'selenium-requests>=2.0.1' to upgrade.

I just released mintapi 2.10 to pypi, which updates selenium-requests to 2.0.1. Please try that (or update your selenium-requests to the same) and let me know if that resolves the issue. Thanks!


integer tests are likely to miss zero

The integer tests use genrand_int32 to fill the input buffers used by the tests.  This is generally not a problem, except that for the larger scalar types, int and long, the probability that the relatively few samples that are tested contain zero is low.  This is particularly unfortunate for functions such as popcount(), clz(), and ctz() where getting the correct result for a zero input is especially important.  We should have a way to ensure that 0 and other interesting values are guaranteed to be tested.

See Khronos internal issue https://gitlab.khronos.org/opencl/conformance-tests/issues/22


API docs update

Update to use the latest antsibull-doc tooling and remove deprecated Github actions.

Committed via CLI


delete keep open label and any other label that is no longer used

clean up labels from previous bot(s) used.

@michelleN need any help with that?

closing as inactive.


Update main.py (added csv export)

added a function that exports the accuracy, speed, and size results as csv files.

Nice thanks! Adding a few comments

LGTM thanks!
Could you add a mention in the readme that the results are written to accuracy_results.csv, speed_results.csv and size_results.csv.

LGTM thanks!
Could you add a mention in the readme that the results are written to accuracy_results.csv, speed_results.csv and size_results.csv?

I also got rid of the TODO bullet point. I think there are a couple of other bullet points that we might be able to get rid of (add ohsumed dataset, has pytorch already been removed?) but you would know better than I would.


Toptracks doesn't begin from the top

Describe the bug
Currently, Your top tracks starts at the 47 instead of 1
To Reproduce
Steps to reproduce the behavior:

Go to /home
See the Top Tracks

Expected behavior
Show Top tracks from the 1 to the 50.
Screenshots

I tried to use this props but something strange it's occurring, even when I set a positive number nothing changes, when I set negative it changes.
Maybe the key props it's not assuming the expected value?

I used initialSlide: 0(this is default anyway) but the slider is starting at the -4 as you can see in the PrevArrow Component.

It's not a bug with the Slider component because I made some examples to test and it worked just fine.
Maybe it's something with the map and the index made here:
const slides = tracks.map((track, index) => slide(index, track));


Submit Table Choose Location on Map

Depends on #55
update the Submit Table page so that users can choose the latitude and longitude from a map.
Refactor the TableMap component to be able to handle this - probably want to rope in the auto-complete and the "Use My Location" button as well.

Handled by #59


WIP: Enhancements to spatial.cKDTree

This PR does the following:

Make the kd-tree data structure inspectable and walkable from Python
Add support for pickle
Removed memory leaks in cKDTree.query
Use PyMem_Malloc instead of libc malloc
Add an option for creating balanced trees
Add an option for compacting/shrinking tree nodes
Clean op the coding style
Change language to C++
Reimplement cKDTree.query in C++
Release GIL in cKDTree.query

Builds restarted, all failed trying to reach ftp.gnu.org.

Travis is happy.

I am adding compact kd-tree build similar to scikit-learn (potentially faster search) and cleaning up the old code. Mostly stylistic. I also want to add the sparse_distance_dict method that has been requested on several occations.
I also need to add tests for the pickle support.
Don't merge this PR yet. Consider it "WIP almost done".

Speaking as one who knows nothing about either implementation, could we
just swap theirs in instead of maintaining two similar but different
versions?
On Wednesday, January 7, 2015, Sturla Molden<EMAIL_ADDRESS>wrote:

I am adding compact kd-tree build similar to scikit-learn (potentially
faster search) and cleaning up the old code. Mostly stylistic. I also want
to add the sparse_distance_dict method that has been requested on several
occations.
I also need to add tests for the pickle support.
Don't merge this PR yet. Consider it "WIP almost done".
—
Reply to this email directly or view it on GitHub
https://github.com/scipy/scipy/pull/4374#issuecomment-69067022.

They are not similar except for the query function. I have wondered the same, currently there are three kd-tree implementations in the SciPy stack (two in SciPy). But the reality is we will break a lot of peoples' code if we do.

In a perfect world one could consider maintaining only one kd-tree implementation written in C, C++ or Fortran 90, and just expose it to SciPy and scikit-learn with Cython or f2py. That would also allow us to release the GIL, which we currently cannot do during lengthy queries. It is not just a problem for parallel processing, but we are also locking up the interpreter. So for all serious use of cKDTree I prefer to fork and do the query in a subprocess. The kd-tree in sciki-learn also has the same problem. Our Python kd-tree (scipy.spatial.KDTree) does not lock up the interpreter, but is slower by orders of magnitude.

Is there a way of e.g., taking then best implementation and adding a compatibility layer to the other one?

That is indeed a good question... Let me just link the other kd-tree codes first:
scikit-learn:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/binary_tree.pxi
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/kd_tree.pyx
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/ball_tree.pyx
scipy:
https://github.com/scipy/scipy/blob/master/scipy/spatial/kdtree.py
https://github.com/scipy/scipy/blob/master/scipy/spatial/ckdtree.pyx
There are two ways to implement a data structure. Either you lay it out in a flat array and use indexing scheme, or you do a lot of allocations and use pointer chasing. scikit-learn uses a flat memory model whereas scipy uses pointer chasing.
cKDTree used to be a tree of C malloc'ed C structs. With this PR it becomes a tree of Python objects. This does not affect the performance (it is identical), but it makes it visible and walkable from Python and serializable with pickle. Those have been the major reasons SciPy also has a Python based id-tree with identical functionality.
As for performance, the scipy cKDTree build trees much faster than scikit-learn, whereas scikit-learn (at least on my tests) gives up to 20 % faster queries. This probably due to the memory model. For practical purposes they are more or less equal.
SciPy cKDTree has been a notorious source of memory leaks. Basically because it uses malloc everywhere in a complex algorithm (involving two heaps and a binary tree) and then looses track of its allocations. I have spent considerable time tracking them down. @patvarilly and I squashed most of them in 2012, and I hope the last ones are gone with this PR.
Scikit-learn uses the median to balance the trees and also compacts hyperrectables to fit the data range. This PR adds both techniques as options. SciPy uses the "sliding midpoint rule" which is known to give quite efficient search trees. But in the worst case you can end up with a very unbalanced tree.
As for features, the nearest-neighbor query method is identical. The rest is different. scikit-learn has implemented the features they need for machine learning, basically point corrections and kernel-density estimation. Exactly how much they cover is difficult of say becuase their kd-tree is not documented but used internally. SciPy's kd-tree has a much richer set of search options, also methods e.g. used in data visualization, games and graphics. SciPy's version lacks kernel density estimation and only allows Minkowski distances.
If we want to implement everything in one kd-tree this will be a dauting task. It is clearly less work to implement the missing features of scikit-learn's kd-tree in cKDTree than vice versa. @patvarilly spent half a year implementing all the functions of spatial.KDTree in spatial.cKDTree. It will take a lot of work to join.
But it is not an optimal situation that the same community is maintaining three separate implementations of kd-trees.

I probably should also ping @jakevdp He might have some insight since he wrote the version in scikit-learn. But he recently make some comments here: https://github.com/scipy/scipy/issues/4236

At least Travis is happy.

Regarding the GIL: there is some work in sklearn to enable multithreading on the tree implementation: https://github.com/scikit-learn/scikit-learn/pull/4009
Seems to be pretty straightforward.

Also, my own work and exploration makes me doubt the necessity of the sliding midpoint rule except in very contrived situations.

Also, scikit-learn's KDTree is very well documented, but a sphinx error makes that documentation not appear on the website. Try IPython's help functionality, and you'll see the documentation.
Sorry for the piece-wise response :smile:

Regarding 3 implementations: we could probably get rid of the Python implementation in scipy.spatial, now that cKDTree is feature complete. The only real reason to keep it that I can see is educational (easier to read).

I am not so concerned about multi-threading as I am with not locking up the interpreter for long periods of time. Granted, it is not so bad on posix systems where we have a copy-on-write fork, but on Windows it is a PITA . There is no room for releasing the GIL in cKDTree as it uses Python objects everywhere. We might consider to make the object use multiprocessing or os.fork() internally, just to make sure this doesn't happen.
For parallel queries it is trivial to use multiprocessing instead of multithreading.

Personally I have never come across a situation where the sliding midpoint rule is of any help. But I know they theoretically exists.

So I did an experiment today and moved the query function to C++ using RAII and STL containers. C++ exceptions are mapped to Python exceptions. It wasn't particularly difficult becuase the Cython code is written almost like primitive C++. Most of it is just boring translation of . to ->and adding curly brackets. I tried to keep the amount of C++ to a minimum so the major portion of the code is still Cython.
Now the situation is that the GIL is released and query speed  is equal to scikit-learn. Not faster, not slower, almost exactly the same.
It seems the performance difference was not due to the memory model but rather suboptimal code generated by Cython. I strongly suspect the culprit is calls to virtual functions. Basically any cdef method in a Cython class is called through a vtab lookup.
Multithreading the queries internally with e.g. OpenMP or GCD is now trivial, but I haven't implemented that yet.
The search methods can be moved to C++ one by one in later PRs.
When I push this I expect there might be some Jenkins errors until I get the setup.py right. Currently I have everything ripped out of SciPy.

Now lets see what Jenkins says...

Meh, build failed on NumPy 1.6 because of NPY_LIKELY.

Added additional tests for the introduced features. If Jenkins is still happy it is time to squash this into one commit.

Bugger, something is not right with the balanced tree option. It generates a small frequency of search errors.

I strongly suspect the culprit is calls to virtual functions.

I think your suspicion is correct. I did a lot of experimentation to get around this type of thing in the sklearn code.

 Pickle error with Pyhton 2.7 is a bit weird, but lets see what Jenkins says about this.

No, pickle is still complaining on Python 2. :-(

The pickle issue on Python 2 might be related to this:
http://bugs.python.org/issue11969

After using Python 3 for two months, it seems I have forgotten how to use pickle in Python 2.

I think problem on Python 2 is related to this:
http://stackoverflow.com/questions/18503466/cython-pickling-in-package-not-found-as-error

Here is another example of the same problem:
http://stackoverflow.com/questions/12646436/pickle-cython-class

My Cython code for shared memory ndarrays, which depends on pickle, seems to be broken as well. Is it just me or is there something strange going on with pickling Cython classes on Python 2?

I give up trying to make pickle work in Python 2. I have no idea what is wrong there. For what I can tell the pickling code is now identical in this PR and sklearn, but it keeps failing on Python 2 -- and working without error on Python 3.
I have changed the memory layout to be a contiguous block (stored in a std::vector), which gives more compact storage and a faster pickle. Also the kd-tree is no longer stored as Python objects, which saves some memory per node. The full tree stucture is still exposed to Python.
I experimented with multithreading queries (with OpenMP and GCD), but the scalability had very bad characteristics on my MacBook. I got near perfect speedup on Windows before, so it is either the OS or something strang that I have not thought about. I have left the code in for educational purposes, or perhaps it can be improved, or perhaps it scales better on Windows or Linux, but only the code for single-threaded queries are used. It releases the GIL, so it should also be possible to experiment with multithtreading on the Python side.

Summary thus far:

Make sure there are no memleaks in cKDTree.query by using a memory pool.
Move all objects and code used by cKDTree.query to C++ and release the GIL during queries
Add support for pickle for Python 3 (pickling fails on Python 2)
Use a flat and contiguous memory layout
Expose the internal kd-tree structure to Python
Add options for splitting on the median and compacting kd-tree nodes
Experimental code for using GCD or OpenMP in cKDTree.query (not built by default)

Other query functions can be moved to C++ later on if this PR turns out ok.

Travis is happy, so I am rebasing on SciPy master.

I have now written a version that uses fork()and shared memory internally in C++. On my laptop it is one or two orders of magnitude times faster than cKDTree and the KDTree in sklearn, typically 20 to 80 times faster.  :smile:
Pushing the code soon!

E.g. searching for the 5 nearest neighbours for each point in np.random.randn(1000000,4) took 20.6 s with cKDTree, 24.8 s with sklearn and 990 ms with my new cKDTree.

Nice!

That seems pretty good. Does your fork version work on windows?
On Tuesday, January 13, 2015, Sturla Molden<EMAIL_ADDRESS>wrote:

E.g. searching for the 5 nearest neighbours for each point in
np.random.randn(1000000,4) took 20.6 s with cKDTree, 24.8 s with sklearn
and 990 ms with my new cKDTree.
—
Reply to this email directly or view it on GitHub
https://github.com/scipy/scipy/pull/4374#issuecomment-69733384.

Windows does not support POSIX fork, so no. But perhaps threads are more scalable on Windows?
Currently my code traises NotImplementedError if parallel queries are requested on Windows. Which reminds me I have to bypass the test for parallel queries on Windows.
The Travis error is not my fault. It seems the full coverage test took too long.

I am calling this a WIP PR until I have parallel queries working on Wintendo as well.

Here is experimental code for multithreading on Windows. I don't have access to a computer for debugging, but the logic is the same as for the posix fork() version so the only errors would be typos.

And speaking of typos...

I got fooled by clock() not giving wall time after fork(). Here are better timings:
http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/u/12464039/kd-tree copy.ipynb
The speed-up compared to the current cKDTree is 2.2 in the sequential case and 8.9 with parallel processing. This makes more sense when there are 4 CPU cores on my laptop... The speed-up of 2.2 is mostly due to tweaking allocation of nodeinfo structs. This seems to be a bottleneck in the query function.

Timing with time.clockfooled me more than once. It seems GCD and OpenMP scales up as well as forking.
http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/u/12464039/kdtree-with-gcd-and-openmp.ipynb

I am considering to remove the internal  parallel processing as a solution with Python threads scale equally well when the GIL is released. Also it does complicates the code and makes maintainence harder.
https://github.com/scikit-learn/scikit-learn/pull/4009#issuecomment-70315786
On the other hand it might be easier for users to have automatic parallel queries. But is seems this can just as well be done with Python code.

IIRC people have previously expressed that they wanted an explicit keyword to enable parallelization, so your parallel kw looks good. I've found the joblib n_jobs kw also easy to use.

despite the typos, your timings are looking pretty good :)

I like joblib too, but its multiprocessing backend is mostly useful on Linux. On Windows and Mac it memory maps a physical file on the hard drive instead of using shared memory for IPC.

Particularly because of an OS from Redmond which severely complicates parallel processing code in C++. It is easier to keep in Python (or Cython) anything that does not need to be in C++.

The main issue is in any case to release the GIL, so we don't lock up the interpreter for potential long periods of time like cKDTree currently do.

Removed all the C++ parallel computing crap add an n_jobs keyword.
http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/u/12464039/kdtree-jan17.ipynb

Squashed and rebased

Works fine for me also on 32-bit linux.

@sturlamolden there's no need to squash all your commits every time, it's actually easier to review if there's one commit per feature.

@sturlamolden maybe hold off pushing fixes to my comments one by one. TravisCI may start producing smoke:)

I was worried about that too.

I don't quite understand why I get this build warning (it is used once in a catch clause):
scipy/spatial/ckdtree/src/ckdtree_cpp_exc.h:48:1: warning: ‘void translate_cpp_exception_with_gil()’
defined but not used [-Wunused-function]
translate_cpp_exception_with_gil()

The header with translate_cpp_exception_with_gil() is also imported by ckdtree.pyx. Thus ckdtree.cxx ends up with an unused static void translate_cpp_exception_with_gil(). I will move the implementations to a .cxx file to silence the warning.

Removed #define NPY_NO_DEPRECATED_API NPY_1_9_API_VERSION similar to:
https://github.com/scipy/scipy/pull/4417

Because the data access during the brute-force search is so horrible, it helps to use explicit prefetching.
http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/u/12464039/kdtree-jan21.ipynb

SIMD acceleration of the Euclidian distances gives another speedup.
http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/u/12464039/kdtree-jan21-simd.ipynb

IIRC people have previously expressed that they wanted an explicit keyword to enable parallelization, so your parallel kw looks good. I've found the joblib n_jobs kw also easy to use.

The n_jobs keyword argument is easy to use but I find the name slightly misleading. It's not the total number of jobs scheduled but instead the maximum number of jobs running concurrently: you can have some workload that is naturally partitioned in 12 jobs and have 4 of them running concurrently at any point with n_jobs=4, typically on a 4 cores host. From this point of view, n_workers would be a better name in my opinion.
That being said there is already a quite extensive code base that is derived from the joblib convention (e.g. scikit-learn and statsmodel) so maybe it's better to reuse that convention even if it's a flaky name.

BTW +1 for using the Python threading + GIL released in C++ combo over the OpenMP option. OpenMP is not supported by compilers such as clang or requires different build flags (MSVC vs GCC) and the GCC implementation is problematic when used in a Python program that relies on Python multiprocessing forks for parallelization at some upper level:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60035

It is a flaky name. But here I am actually using static scheduling, so in some sence it is actually correct (the number of jobs is equal to the number of threads).
Actually it might spawn n_jobs+1 threads, but if that happens only n_jobs of the threads will do significant work. It simplifies the logic but doesn't do any harm.

I guess I could also have used joblib, but it is not bundled with SciPy.

Apple's GCD and Accelerate Framework also has the same issues with multiprocessing as GNU OpenMP. But the bug is actually in multiprocessing and fixed in Python 3.4. It is not correct to assume that APIs beyond a signal-safe subset of POSIX should be safe to call on both sides of a fork without an exec. Threads and forks don't mix very well.

I guess I could also have used joblib, but it is not bundled with SciPy.

No need to add a dependency on joblib just for a single use of the threading module.
BTW, the API in multiprocessing.pool.ThreadPool is slightly higher level. In Python 3 there is concurrent.futures.ThreadPoolExecutor that is even better but that would require a dependency on the external python futures package for Python 2 support.

Apple's GCD and Accelerate Framework also have the same issues with multiprocessing as GNU OpenMP. But the bug is actually in multiprocessing and is fixed in Python 3.4. It is not correct to assume that APIs beyond a signal-safe subset of POSIX should be safe to call on both sides of a fork without an exec. Threads and forks don't mix very well.

I agree but it's not fixed by default in Python 3.4 either: you have to select a distinct start method such as forkserver instead of fork.

Yes, and it cripples multiprocessing to use it, though it is never worse than multiprocessing on Windows.

Time to pick this up again, would be good to get it in before the release split. @sturlamolden here is a rebased branch with some build/doc/style fixes: https://github.com/rgommers/scipy/tree/pr-4374-ckdtree. Can you have a look at that and push it to this PR if it looks OK?
I also took the libery of reformatting your commit messages, mainly to use the standard prefixes as in http://docs.scipy.org/doc/numpy-dev/dev/gitwash/development_workflow.html#writing-the-commit-message. And to squash a couple of commit that were only a couple of lines. Note, my last two commits can also be squashed together, I kept them apart so that it's easier to look at the build warning fixes.

 Thanks @rgommers :)

attn @nkeim

@sturlamolden here's a recipe in case you need it:
$ git remote add rgommers https://github.com/rgommers/scipy.git
$ git fetch rgommers
$ git br -D ckdtree-query-fix-memleak
$ git co -b ckdtree-query-fix-memleak rgommers/pr-4374-ckdtree
$ git push origin ckdtree-query-fix-memleak -f   # assumes your own Github repo is called origin

Thank you :) @rgommers

Now lets see if this worked...

On entry to ZGEMM parameter number 10 had an illegal value
The code in this PR does not use BLAS apart from np.dot, but it should not call ZGEMM as complex numbers are not used. Is this an unrelated error?

@rgommers

Yep, unrelated error. It was in master for a while, but only detected when we fixed the relaxed strides TravisCI test. Due to gh-4736.
Will need to do something about that asap.

Rebase worked like a charm, and the other test runs pass for this PR.

Tested on 32-bit Linux and on OS X (w Accelerate). Still would like to test this on Windows with mingw32 before merging.
If anyone can test with MSVC, that would also be helpful. @sturlamolden reading back the discussion above, I'm not sure if you actually did test on Windows?

Windows, gosh, does anyone still use that OS? :smile: All my computers have OSX or Linux. The only thing I can say with respect to Windows is that I wrote standard C++98. So it should compile on MinGW and MSVC. But testing on Windows is a good idea before merging.
I have an old harddrive with Windows 7 and all the dev tools installed. But I need to plug it into a computer :)
I planned at some point to install Windows in a VirtualBox, but I never got around to do it.

No, I did not test on Windows, but I will do it now.

Thanks, that would be helpful. I'll test on mingw32.

@sturlamolden things work fine with mingw32, so I'd like to merge this today or tomorrow. Can you have a look at my last two comments?

OK, that's weird... :smile:
I just ran tests with MinGW and MSVC myself.
MinGW (gcc 4.6.1, 64-bit) compiled fine, but the .query method segfaulted. What version of MinGW did you try?
With MSVC I got a lot of crappy warnings and a few error messages (see below).
I don't think this should be merged just yet. :disappointed:
C:\DEVELOPMENT\cKDTree>python setup.py build_ext --inplace --compiler=msvc
non-existing path in '.': 'ckdtree.cxx'
running build_ext
running build_src
build_src
building extension "spatial.ckdtree" sources
build_src: building npy-pkg config files
No module named msvccompiler in numpy.distutils; trying from distutils
customize MSVCCompiler
customize MSVCCompiler using build_ext
No module named msvccompiler in numpy.distutils; trying from distutils
customize MSVCCompiler
Missing compiler_cxx fix for MSVCCompiler
customize MSVCCompiler using build_ext
building 'spatial.ckdtree' extension
compiling C sources
cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\
lib\site-packages\numpy\core\include -I.\ckdtree\src -IC:\Python27\lib\site-pack
ages\numpy\core\include -IC:\Python27\include -IC:\Python27\PC /Tp.\ckdtree\ckdt
ree.cxx /Fobuild\temp.win-amd64-2.7\Release\.\ckdtree\ckdtree.obj /Zm1000
Found executable c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\amd64
\cl.exe
ckdtree.cxx
.\ckdtree\src\ckdtree_cpp_methods.h(29) : error C3861: '__builtin_prefetch': ide
ntifier not found
c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Include\xlocale(342) : war
ning C4530: C++ exception handler used, but unwind semantics are not enabled. Sp
ecify /EHsc
.\ckdtree\ckdtree.cxx(8929) : warning C4244: '=' : conversion from '__pyx_t_5num
py_intp_t' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(9999) : warning C4244: '=' : conversion from '__pyx_t_5num
py_intp_t' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(10687) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(10691) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(10703) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11538) : warning C4244: 'argument' : conversion from '__py
x_t_5numpy_intp_t' to 'int', possible loss of data
.\ckdtree\ckdtree.cxx(11538) : warning C4244: 'argument' : conversion from '__py
x_t_5numpy_intp_t' to 'int', possible loss of data
.\ckdtree\ckdtree.cxx(11559) : warning C4244: 'argument' : conversion from '__py
x_t_5numpy_intp_t' to 'int', possible loss of data
.\ckdtree\ckdtree.cxx(11559) : warning C4244: 'argument' : conversion from '__py
x_t_5numpy_intp_t' to 'int', possible loss of data
.\ckdtree\ckdtree.cxx(11614) : warning C4244: 'argument' : conversion from '__py
x_t_5numpy_intp_t' to 'int', possible loss of data
.\ckdtree\ckdtree.cxx(11614) : warning C4244: 'argument' : conversion from '__py
x_t_5numpy_intp_t' to 'int', possible loss of data
.\ckdtree\ckdtree.cxx(11845) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11846) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11849) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11850) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11853) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11854) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11908) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11909) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11923) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11924) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11929) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11930) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11965) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11966) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11971) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(11972) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(12000) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(12001) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(12006) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(12007) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(13591) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(13792) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(15975) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(16713) : warning C4244: '=' : conversion from 'Py_ssize_t'
 to 'int', possible loss of data
.\ckdtree\ckdtree.cxx(17052) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(17054) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(17056) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(17085) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(17098) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
.\ckdtree\ckdtree.cxx(17124) : warning C4244: '+=' : conversion from 'Py_ssize_t
' to 'long', possible loss of data
error: Command "cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\includ
e -IC:\Python27\lib\site-packages\numpy\core\include -I.\ckdtree\src -IC:\Python
27\lib\site-packages\numpy\core\include -IC:\Python27\include -IC:\Python27\PC /
Tp.\ckdtree\ckdtree.cxx /Fobuild\temp.win-amd64-2.7\Release\.\ckdtree\ckdtree.ob
j /Zm1000" failed with exit status 2

Squashed the bug on MSVC. Building and running ok now.
Speedup on 4-core AMD Phenom 2  with MSVC is enormous:
Current cKDTree: 85.6 s
New cKDTree, one thread: 14.2 s
New cKDTree, multithreading: 4.8 s

MinGW still crashes on .query.

Speedup with _mm_prefetch on MSVC:
Current cKDTree: 85.6 s
New cKDTree, one thread: 9.2 s
New cKDTree, multithreading: 3.1 s

Can someone else test on MinGW?

It builds and runs correctly on my MinGW setup now. The cause of the crash with MinGW was bad compile flags set by an old NumPy (fixed with current NumPy).
https://github.com/numpy/numpy/blob/master/numpy/distutils/mingw32ccompiler.py

I think this is ready to merge now. Travis might complain on the unrelated QR issue because it is not rebased on a fixed master.

I've tested with mingw32 3.4.5 (which is what we still build binaries with). I'll re-test that, if that passes and MSVC + MinGW 4.6 work for you then it's time to hit the green button.

MinGW-w64 with gcc 3.x still has -DDEBUG in NumPy master. See line 126 and 127. Not much I can do about that. But the 32-bit MinGW does not have this flag, and we only provide pre-built 32-bit binaries, right?
https://github.com/numpy/numpy/blob/master/numpy/distutils/mingw32ccompiler.py
MSVC and MinGW-w64 [4.6] works fine for me now.

Indeed, only 32-bit binaries for now. But probably once we start creating them for 64-bit with MinGW-w64 (first numpy then scipy I expect) we'll want to go back and provide those also for recent releases - we did the same for OS X wheels.

There probably is not a MinGW-w64 past gcc version 4.x anyway. (But if there were it would crash regardless of this PR.)

Travis is happy too, it seems.

@aarchiba Do you have any comments on this PR?

still passes all tests with mingw32 3.4.5

Okay, this seems ready to go into 0.16.x - hitting the green button.

Thanks @sturlamolden!