Compilation errors with nrf24lu1

 > make BOARD=unirecv
In file included from ../src/key_handlers/key_handlers.h:12,
                 from ../src/key_handlers/key_normal.h:6,
                 from ../src/key_handlers/key_normal.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/key_handlers/key_handlers.h:12,
                 from ../src/key_handlers/key_mouse.h:6,
                 from ../src/key_handlers/key_mouse.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/key_handlers/key_mouse.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/key_handlers/key_handlers.h:12,
                 from ../src/key_handlers/key_media.h:6,
                 from ../src/key_handlers/key_media.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/key_handlers/key_handlers.h:12,
                 from ../src/key_handlers/key_macro.h:6,
                 from ../src/key_handlers/key_macro.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/key_handlers/key_handlers.h:12,
                 from ../src/key_handlers/key_hold.h:6,
                 from ../src/key_handlers/key_hold.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/key_handlers/key_hold.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/key_handlers/key_handlers.h:12,
                 from ../src/key_handlers/key_handlers.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/key_handlers/key_handlers.h:12,
                 from ../src/key_handlers/key_custom.h:6,
                 from ../src/key_handlers/key_custom.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/usb/util/requests.h:6,
                 from ../src/usb/device_descriptors.c:6:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/usb_reports/mouse_report.h:6,
                 from ../src/usb_reports/mouse_report.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/usb_reports/mouse_report.c:9:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/usb/descriptors.h:6,
                 from ../src/usb_reports/vendor_report.h:6,
                 from ../src/usb_reports/vendor_report.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/debug.h:14,
                 from ../src/core/ring_buf.h:8,
                 from ../src/usb_reports/vendor_report.h:9,
                 from ../src/usb_reports/vendor_report.c:4:
/usr/include/stdlib.h:131:36: error: missing binary operator before token "("
/usr/include/stdlib.h:174:17: error: missing binary operator before token "("
/usr/include/stdlib.h:188:36: error: missing binary operator before token "("
In file included from ../src/usb_reports/vendor_report.c:126:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/usb_reports/media_report.h:6,
                 from ../src/usb_reports/media_report.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/usb_reports/media_report.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/usb_reports/keyboard_report.h:6,
                 from ../src/usb_reports/keyboard_report.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/usb_reports/keyboard_report.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/keycode.h:6,
                 from ../src/core/keycode.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/keycode.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/flash.h:8,
                 from ../src/core/macro.h:6,
                 from ../src/core/macro.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/macro.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/usb_commands.h:26,
                 from ../src/core/usb_commands.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/usb_commands.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/debug.h:14,
                 from ../src/core/ring_buf.h:8,
                 from ../src/usb_reports/vendor_report.h:9,
                 from ../src/core/usb_commands.c:23:
/usr/include/stdlib.h:131:36: error: missing binary operator before token "("
/usr/include/stdlib.h:174:17: error: missing binary operator before token "("
/usr/include/stdlib.h:188:36: error: missing binary operator before token "("
In file included from ../src/core/mods.h:6,
                 from ../src/core/mods.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/mods.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/matrix_interpret.h:6,
                 from ../src/core/matrix_interpret.c:23:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/matrix_interpret.c:25:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/unifying.h:6,
                 from ../src/core/unifying.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/unifying.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/nonce.h:6,
                 from ../src/core/nonce.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/rf.h:6,
                 from ../src/core/rf.c:7:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/rf.c:9:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/debug.h:14,
                 from ../src/core/rf.c:12:
/usr/include/stdlib.h:131:36: error: missing binary operator before token "("
/usr/include/stdlib.h:174:17: error: missing binary operator before token "("
/usr/include/stdlib.h:188:36: error: missing binary operator before token "("
In file included from ../src/core/nrf24.h:6,
                 from ../src/core/nrf24.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/util.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/settings.h:7,
                 from ../src/core/settings.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/settings.c:8:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/ring_buf.h:6,
                 from ../src/core/ring_buf.c:1:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/debug.h:14,
                 from ../src/core/ring_buf.h:8,
                 from ../src/core/ring_buf.c:1:
/usr/include/stdlib.h:131:36: error: missing binary operator before token "("
/usr/include/stdlib.h:174:17: error: missing binary operator before token "("
/usr/include/stdlib.h:188:36: error: missing binary operator before token "("
In file included from ../src/core/packet.h:6,
                 from ../src/core/packet.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/flash.h:8,
                 from ../src/core/layout.h:7,
                 from ../src/core/layout.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/hardware.h:6,
                 from ../src/core/hardware.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/flash.h:8,
                 from ../src/core/flash.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/error.h:9,
                 from ../src/core/error.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/error.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/flash.h:8,
                 from ../src/core/crc.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/usb/descriptors.h:6,
                 from descriptors.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from usb_reports.c:4:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from nrf24lu1_usb_regs.h:6,
                 from usb_reports.c:7:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from usb.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/debug.h:14,
                 from ../src/core/ring_buf.h:8,
                 from ../src/usb_reports/vendor_report.h:9,
                 from usb.c:22:
/usr/include/stdlib.h:131:36: error: missing binary operator before token "("
/usr/include/stdlib.h:174:17: error: missing binary operator before token "("
/usr/include/stdlib.h:188:36: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/timer.h:6,
                 from timer.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from nrf24.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/nonce.h:6,
                 from nonce.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from main.c:7:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
In file included from ../src/core/usb_commands.h:26,
                 from main.c:13:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/debug.h:14,
                 from ../src/core/ring_buf.h:8,
                 from ../src/usb_reports/vendor_report.h:9,
                 from main.c:33:
/usr/include/stdlib.h:131:36: error: missing binary operator before token "("
/usr/include/stdlib.h:174:17: error: missing binary operator before token "("
/usr/include/stdlib.h:188:36: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/led.h:6,
                 from led.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/hardware.h:6,
                 from hardware.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/util.h:11,
                 from ../src/core/flash.h:8,
                 from flash.c:6:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from ../src/core/debug.h:14,
                 from ../src/core/ring_buf.h:8,
                 from ../src/usb_reports/vendor_report.h:9,
                 from flash.c:9:
/usr/include/stdlib.h:131:36: error: missing binary operator before token "("
/usr/include/stdlib.h:174:17: error: missing binary operator before token "("
/usr/include/stdlib.h:188:36: error: missing binary operator before token "("
In file included from ../src/core/aes.h:11,
                 from aes.c:4:
/usr/include/stdint.h:286:17: error: missing binary operator before token "("
In file included from aes.c:6:
/usr/include/string.h:36:42: error: missing binary operator before token "("
/usr/include/string.h:164:21: error: missing binary operator before token "("
/usr/include/string.h:173:43: error: missing binary operator before token "("
/usr/include/string.h:491:19: error: missing binary operator before token "("
make: *** No rule to make target 'build/unirecv/obj/aes.dep', needed by 'build/unirecv/keyplus_nrf24lu1-unirecv.hex'.  Stop.

When I run it again:
compiling: aes.c
compiling: flash.c
compiling: hardware.c
compiling: led.c
compiling: main.c
compiling: nonce.c
nonce.c:50: error 78: incompatible types
from type 'unsigned-int near* fixed'
  to type 'unsigned-char generic* fixed'
nonce.c:51: error 78: incompatible types
from type 'const-unsigned-long-int near* fixed'
  to type 'unsigned-char generic* fixed'
Makefile:206: recipe for target 'build/unirecv/obj/nonce.rel' failed
make: *** [build/unirecv/obj/nonce.rel] Error 1

The efm8 branch has scripts for building on nrf24lu1+ (it uses the SDCC compiler too). When it's ready I'll merge it to master and documentation on how to build it.
With the unifying receivers, need to be a little bit careful since if you load a bad build it is quite hard to recover it again.

I'm guessing this never got resolved? Can't find the efm8 branch now, and still getting the same errors on master :(

I'm guessing this never got resolved? Can't find the efm8 branch now, and still getting the same errors on master :(


[WIP] GH-192 Python release integration test

All builds with TRAVIS_TAG set to release will trigger a long running process to validate the release is correct. The script does the following

extract the tarball
start the node
get node's top every 5 seconds untill there are 5 blocks mined
stop the node and cleanup

TODO: check what's the real value of TRAVIS_TAG when building a release

TODO: check what's the real value of TRAVIS_TAG when building a release

@velzevur Wrt checking this for the case the build is a tag: I recommend you try this by forking this repo in your github account, enabling Travis CI there, pushing this branch there too, taking there (in your fork) a tag, observing result.


make the deploy modal side bar sticky

Unfortunately, I couldn't bootstrap scrollspy plugin to work in a modal, will get that working in a separate PR in the future.


Remove the tail spaces from all source files

Summary
Please ignore the false alarm: "error: Mixed case identifier found"
Impact
Minor
Testing
CI

Please remove the rst files unless you are going to check each of them manually. There are some places where they are intentional and impact formatting.

Please remove the rst files unless you are going to check each of them manually. There are some places where they are intentional and impact formatting.

Ok, I remove the change inside Documentation.


Sdk automation script

SDK automation task basically involves creating a PR in azure-sdk-for-net repo when something changes in the swagger files in azure-rest-api-specs repo. This will result in removing manual work for updating the SDKs in azure-sdk-for-net repo when something changes in the swagger files.
SDK Automation Doc: https://github.com/Azure/azure-rest-api-specs/blob/master/documentation/sdkautomation/sdk_customization.md
This PR is for Implementing powershell script which takes json file as an input, updates the input-file path in autorest.mds for the changed swagger files, generates sdk using track-2 code gen and produces output json as mentioned here.
For reference here are the examples of input and output json in the test env:

generateInput.json
generateOutput.json

nit: casing of the Sdk-init.ps1 name should be consistent with the Sdk-Generate. I would also added some prefix like Automation-* so people who see the script would know that this is not something they need to run.


Entering UI edit mode breaks view

Entering 'edit UI' mode with custom header breaks the view and make editing underlaying cards not possible as can be seen on screen:

There are cards and badges underneath.
version: 1.4.1

Related to this, if you enter 'edit UI' mode, switch to another panel, such as "Supervisor", and then switch back, the header is now gone and you cannot switch pages or exit 'edit ui' mode without reloading the page.

Thanks, on it. Both issues should be fixed in next release.

Yeah, since i installed the new Hass.io Core & OS / Supervisor updates this week Custom-Header won't even load anymore.. do you have any idea if I would need to remove it completely and reinstall? Or will this be fixed from the update?

Yes, this issue is fixed for me now as well - thank you very much maykar! Have a terrific weekend.

Sovled :) thx!


Error executing vim.schedule lua callback: Vim(call):E5555: API call: Invalid buffer id

Describe the bug
I see this error when I wiping all buffers.
To Reproduce

Open any file with LSP client.
Start typing to see a completion with preview. Trigger preview is important, you should see VS.Vim.Buffer.Default in ls!:

Execute %bwipeout
Go to the previous opened file (<C-o>)
Start typing to see a completion with preview.

Actual behavior
Instead of the preview window, you will see the following error:
Error executing vim.schedule lua callback: Vim(call):E5555: API call: Invalid buffer id

Expected behavior
The preview is displayed normally.
Additional context
I discovered a similar issue but with a different context: #266.
Can we have a check if this buffer is valid and recreate it if needed automatically?

@hrsh7th, I think if we add a check for VS.Vim.Buffer.Default that will fix #266 too.

I think it was fixed. Thank you for your pointing out.

@hrsh7th, it didn't help :(
The issue still exists.


 Project.toml:
name = "packageName"
uuid = "1c653b0a-0b5a-4cff-b25a-92f0db012773"
version = "0.1.0"

[deps]
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"

[compat]
JSON = "0.21"
julia = "1.8"

and Manifest.toml:
# This file is machine-generated - editing it directly is not advised

julia_version = "1.9.0"
manifest_format = "2.0"
project_hash = "fd1f7ad92e109f821f4124222ecf4f969f1af8a6"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "31e996f0a15c7b280ba9f76636b3ff9e2ae58c9a"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.4"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[deps.Parsers]]
deps = ["Dates", "PrecompileTools", "UUIDs"]
git-tree-sha1 = "a5aef8d4a6e8d81f171b2bd4be5265b01384c74c"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.5.10"

[[deps.PrecompileTools]]
deps = ["Preferences"]
git-tree-sha1 = "259e206946c293698122f63e2b513a7c99a244e8"
uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
version = "1.1.1"

[[deps.Preferences]]
deps = ["TOML"]
git-tree-sha1 = "7eb1686b4f04b82f96ed7a4ea5890a4f0c7a09f1"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.4.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.Random]]
deps = ["SHA", "Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[deps.SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
version = "0.7.0"

[[deps.Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
version = "1.0.3"

[[deps.UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

PkgToSoftwareBOM v0.1.1
Julia v1.9

It's erroring out trying to read a Project.toml from the registry.  It must be when trying to access files for packageName, but presumably packageName doesn't exist in the registry?
Does packageName exist as a set of source files that you added as a local path in Pkg?  Or does just the name exist in your Project.toml?

Indeed, this package isn't in any registry. It's actually just those two TOML files and nothing else.

This is an edge case I hadn't considered.  Where you're presented with a Project.toml that can't be fully resolved.  For example, if some packages are in a private registry that the SBOM creator doesn't have access to.  The package today does implicitly assume a completely resolved Manifest.toml.  Probably worth mentioning in the README.
Still, it shouldn't have gotten that far into the registry resolution process before erroring out.  It should have exited with missing right at the start.  Will need to investigate further.

The manifest can be fully resolved; all those packages are either stdlibs or are in General. We don't register all of our packages because we don't need to. Some of them are applications that are never installed by anyone but are instead containerized, so they aren't registered. I hope you will consider supporting this use case.

@Octogonapus
I want to solve your problem, but I can't reproduce the issue yet. I feel that I'm missing something in the setup. Here's what I'm doing

I create a new folder called dev_sbom
Inside that folder I create another folder called packageName.  I put the Project.toml and the Manifest.toml files in your original post in there
Launch Julia, cd to my dev_sbom folder
Launch interactive Pkg, create a new environment
Try to add packageName as a package. But at this point it all errors out for me. Here's a transcript

julia> cd("./JuliaWork/dev_sbom/")

julia> VERSION
v"1.9.0"

(@v1.8) pkg> activate .
  Activating new project at `~/JuliaWork/dev_sbom`

(dev_sbom) pkg> add ./packageName
ERROR: Did not find a git repository at `packageName`, perhaps you meant `Pkg.develop`?

(dev_sbom) pkg> dev ./packageName/
ERROR: expected the file `src/packageName.jl` to exist for package `packageName` at `packageName`

(dev_sbom) pkg> st
Status `~/JuliaWork/dev_sbom/Project.toml` (empty project)

If you could give me some pointers on how you are setting up your environment, I'll try to reproduce.

Here's a script that can reproduce it:
#!/bin/bash
mkdir testpkg

echo 'name = "packageName"
uuid = "1c653b0a-0b5a-4cff-b25a-92f0db012773"
version = "0.1.0"

[deps]
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"

[compat]
JSON = "0.21"
julia = "1.8"' > testpkg/Project.toml

echo '# This file is machine-generated - editing it directly is not advised

julia_version = "1.9.0"
manifest_format = "2.0"
project_hash = "fd1f7ad92e109f821f4124222ecf4f969f1af8a6"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "31e996f0a15c7b280ba9f76636b3ff9e2ae58c9a"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.4"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[deps.Parsers]]
deps = ["Dates", "PrecompileTools", "UUIDs"]
git-tree-sha1 = "a5aef8d4a6e8d81f171b2bd4be5265b01384c74c"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.5.10"

[[deps.PrecompileTools]]
deps = ["Preferences"]
git-tree-sha1 = "259e206946c293698122f63e2b513a7c99a244e8"
uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
version = "1.1.1"

[[deps.Preferences]]
deps = ["TOML"]
git-tree-sha1 = "7eb1686b4f04b82f96ed7a4ea5890a4f0c7a09f1"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.4.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.Random]]
deps = ["SHA", "Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[deps.SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
version = "0.7.0"

[[deps.Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
version = "1.0.3"

[[deps.UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"' > testpkg/Manifest.toml

julia --project=testpkg -e 'import Pkg; using PkgToSoftwareBOM; generateSPDX(spdxCreationData(rootpackages=filter(p -> !(p.first in ["PkgToSoftwareBOM", "SPDX"]), Pkg.project().dependencies)))'

I've run your script and it works just fine for me.
Looking at the stack trace in the original post, it's erroring out on line 60 of Registry.jl
Package= TOML.parse(registry.in_memory_registry[registryPath*"/Package.toml"])

The error message
ERROR: MethodError: no method matching getindex(::Nothing, ::String)

implies that registry.in_memory_registry is nothing.
Could you launch Julia in the configuration that fails for you and then run
Pkg.Registry.status()

and tell me what registries you have?  There would appear to be something different about the Registry memory structures than what I see.

I can actually reproduce this exception by running the same code on this project. I would guess that your General registry is a compressed file? Mine is not, so in_memory_registry is nothing. This is the relevant code https://github.com/JuliaLang/Pkg.jl/blob/master/src/Registry/registry_instance.jl#L305-L331
shell> ls ~/.julia/registries/General
A  B  C  CONTRIBUTING.md  D  E  F  G  H  I  J  jll  K  L  LICENSE.md  M  N  O  P  Q  R  README.md  Registry.toml  S  T  U  V  W  X  Y  Z

You are correct that my General registry is a compressed file, which is the out of the box behavior.  This looks like the cause of the error.
Out of curiosity, how do you get your General registry to be like this?.  It would be helpful if I could recreate this scenario.

I believe ENV["JULIA_PKG_SERVER"] = "" is what does it. If not, try adding General by URL.

I've forked the General registry, modded the UUID, and then added it to my environment. Now I have a compressed and uncompressed registry side-by-side.  With the uncompressed registry I get the same error as in the opening post.
Now that I can reproduce the issue, I'll start working on a fix.

Released as v0.1.2


Fixes to interactive figures in notebooks 18 and 19

See #2 and the commit log on this PR for more information.

thanks!


hanging event handlers

We are currently experiencing an issue with nameko consumers hanging forever not acking messages nor consuming more. We have three queues (prefetch_count 25 each) which are event handlers. One event handler calls tpool some times (but seems like inside tpool there are no context switching - only open/os.scandir calls and some lxml processing), other two handlers just use nameko-sqlalchemy.
All three queues have 25 unacked messages and the process is just hanging not consuming nor acking messages. After stracing these tpool threads seems like thery are just sleeping in a loop
select(0, NULL, NULL, NULL, {20, 647391}) = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {20152, 65235860}) = 0
clock_gettime(CLOCK_MONOTONIC, {20152, 65326908}) = 0
select(0, NULL, NULL, NULL, {60, 0})    = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {20212, 125662883}) = 0
clock_gettime(CLOCK_MONOTONIC, {20212, 125752436}) = 0
select(0, NULL, NULL, NULL, {60, 0})    = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {20272, 184288837}) = 0
clock_gettime(CLOCK_MONOTONIC, {20272, 184380661}) = 0
select(0, NULL, NULL, NULL, {60, 0})    = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {20332, 195832888}) = 0
clock_gettime(CLOCK_MONOTONIC, {20332, 195917909}) = 0
select(0, NULL, NULL, NULL, {60, 0}
stracing the main thread gives
recvfrom(6, 0x7f67bc13aac8, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_MONOTONIC, {20686, 120473825}) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=6, u64=2086511330193309702}}) = 0
clock_gettime(CLOCK_MONOTONIC, {20686, 120606518}) = 0
clock_gettime(CLOCK_MONOTONIC, {20686, 120649753}) = 0
epoll_wait(4, [], 1023, 1000)           = 0
clock_gettime(CLOCK_MONOTONIC, {20687, 121990770}) = 0
epoll_ctl(4, EPOLL_CTL_DEL, 6, 0x7ffd7fa5f9a0) = 0
clock_gettime(CLOCK_MONOTONIC, {20687, 122235665}) = 0
clock_gettime(CLOCK_MONOTONIC, {20687, 122304991}) = 0
recvfrom(6, 0x7f67bc13aac8, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_MONOTONIC, {20687, 122510493}) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=6, u64=2086511330193309702}}) = 0
clock_gettime(CLOCK_MONOTONIC, {20687, 122653462}) = 0
clock_gettime(CLOCK_MONOTONIC, {20687, 122698273}) = 0
epoll_wait(4, ^Cstrace: Process 1810 detached
6 descriptor is amqp connection
The worst problem is that it happens sporadically and we have no idea how to reproduce it.
The log file shows and error
Traceback (most recent call last):   File "/home/dev1-irm/.venv/lib/python3.5/site-packages/eventlet/hubs/hub.py", line 458, in fire_timers     timer()   File "/home/dev1-irm/.venv/lib/python3.5/site-packages/eventlet/hubs/timer.py", line 58, in __call__     cb(*args, **kw)   File "/home/dev1-irm/.venv/lib/python3.5/site-packages/eventlet/semaphore.py", line 147, in _do_acquire     waiter.switch() greenlet.error: cannot switch to a different thread
about at the same time the consumers started hanging. But there is no information about from where the trace came.
We are using
nameko==2.8.4
eventlet==0.21.0

You may be able to figure out what the greenthreads are doing by printing their stack traces. If you backdoor into the process, you can access the currently running managed and worker threads via the container.
Looking at their current stack may give you an idea what they're waiting for.
It looks like your queue consumer thread is waiting to receive data and not getting any, which makes sense if you're already holding max unacked messages. So I would start by trying to figure out what the worker threads are doing, and why they've not finished and acked their messages.
The "cannot switch to a different thread" message is suspicious. Whatever caused it may have broken the eventlet scheduler.

@mattbennett I managed to reproduce the issue, but the problem seems to have bad side-effects. What I did was dispatching a lot of events, and then dd from /dev/zero to a file until there is no space left on device (we are running nameko under supervisor). So when I am printing the tracebacks of all greenlets,
import traceback
from greenlet import greenlet
for ob in gc.get_objects():
    if not isinstance(ob, greenlet):
        continue
    if not ob:
        continue
    print(''.join(traceback.format_stack(ob.gr_frame)))

they are all stuck on self.greenlet.switch()
Here is an example:
    result = function(*args, **kwargs)
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/nameko/containers.py", line 393, in _run_worker
    result = method(*worker_ctx.args, **worker_ctx.kwargs)
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/ingest/base/retry.py", line 72, in wrapper
    return wrapped(*args, **kwargs)
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/ingest/base/retry.py", line 72, in wrapper
    return wrapped(*args, **kwargs)
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/ingest/release_manager/main/service.py", line 198, in handle_delivery_arrived
    delivery_handler.parse, release_uri)
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/eventlet/tpool.py", line 117, in execute
    rv = e.wait()
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()

Another interesting thing I noticed is that inside a tpool there is a context switch which is caused by datetime.strptime function
  File "/usr/lib/python3.5/_strptime.py", line 510, in _strptime_datetime
    tt, fraction = _strptime(data_string, format)
  File "/usr/lib/python3.5/_strptime.py", line 313, in _strptime
    with _cache_lock:
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/eventlet/semaphore.py", line 129, in __enter__
    self.acquire()
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/eventlet/semaphore.py", line 115, in acquire
    hubs.get_hub().switch()
  File "/home/dev1-irm/.venv/lib/python3.5/site-packages/eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()

But I can not understand what is actually causing a hub to stop processing greenlets. At this time there is no "cannot switch to a different thread" log message.
My question is if context switch inside a tpool must always trigger that error? If yes, why sometimes greenlets do not block?
And another question is how do I find out the actual cause why greenlets are stuck?

I'm confused. Why did you choose to consume all the disk space to reproduce the issue? Couldn't this be causing a completely different problem?
From what I understand of tpool, it's intended to wrap non-green functions that would otherwise block the event loop. If the thing you're wrapping yields, why are you using the tpool?

@mattbennett sorry for a little chaotic reply :) I chose to consume all disk space because about at the same time we had hanging consumers we had "no space left on device" errors and I thought that this might be causing the problem.
Gegarding tpool, I actually wrapped python code that seemed to have no yields but when I inspected greenlets with the help of backdoor, I noticed that strptime function uses some semaphore which is patched by eventlet and this is causing a yield.
I agree that full disk and hanging consumers are actually separate problems and the real cause was the implicit yield from strptime. So I am going to remove tpool (and use it more wisely next time) from that part of code and give it a try.

Ha, no worries. Yep, let's close this issue.
I guess the root cause is either the out of disk space error or the yield inside the tpool. You might want to check over at github.com/eventlet/eventlet whether a yield inside a tpool is expected to cause problems. Seems a bit fragile to me.


feat(opt-generation): defaults values

previous pdf support is removed. Defaults values for others generation options
BREAKING CHANGE: previous pdf support is removed.
fix #3

:tada: This PR is included in version 2.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


No file shows in the destination directory after downloading one non-current version

Storage Explorer Version: 1.15.0-dev
Build: 20200820.5
Branch: master
Platform/OS: Windows 10/ CentOS 7.6.1810 (Core) / MacOS Catalina
Architecture: ia32/x64
Regression From: Not a regression
Steps to reproduce:

Expand one Non-ADLS Gen2 storage account (make sure the blob versioning and soft delete are enabled) -> Blob Containers.
Create a new blob container -> Upload one blob to it -> Delete the blob.
Switch to 'Active blobs and blobs without current' view.
Download the non-current version to the specified destination directory.
Check whether the non-current version shows in the destination directory.

Expect Experience:
The non-current version shows in the destination directory.
Actual Experience:

Show a successful activity log.

No non-current version shows in the destination directory.

More Info:
This issue doesn't reproduce for the multiple non-current versions.

The bug is that we were incorrectly trying to download the non-existent current version. For you information, when downloading a  single previous version, in the activity log, the source file name will have its versionId included. e.g.
Transfer of 'version/Leasd.txt (2020-08-19T18:09:44.1472053Z)' to 'C:\Users\vlimu\Documents\version\New folder' complete (used SAS)
If the versionId is not there in the activity when downloading a single previous version, it is already a bug.

Verified on build 20200821.5. The version id shows in the activity log when downloading a single version and the file shows in the destination directory. So close it.


Crash report: "missing field jquery"

RUST_BACKTRACE=1 SHREDDIT_DRY_RUN=false ./shreddit-linux
<snipped output>
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("missing field `jquery`", line: 1, column: 50) }', src/things.rs:206:14
stack backtrace:
   0: rust_begin_unwind
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panicking.rs:64:14
   2: core::result::unwrap_failed
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/result.rs:1791:5
   3: shreddit::things::Thing::edit::{{closure}}::{{closure}}
   4: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
   5: shreddit::things::Thing::shred::{{closure}}
   6: tokio::runtime::park::CachedParkThread::block_on
   7: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
   8: tokio::runtime::runtime::Runtime::block_on
   9: shreddit::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

./shreddit-linux --version
shreddit 0.4.0

Could you try running the 0.4.1 release? That should surface the error you're seeing.

Actually, I see this too. It seems to only occur for posts, not comments. Looking into this now.

This is fixed and will be in the 0.4.2 release. Thanks for the report!

Thanks, confirmed it works now :)


Docs: Add top/bottom margin to highlighted code samples

Backport of #31036 to v4

before/after comparison (unhovered)


Quick buy - starting items

Hello, is there a possibility to get information about starting items which are quick buy. There is no information about these items. To sum up, i can retrieve information about purchased items, however starting items are inaccessible.

This library doesn't offer entity parsing which is one way to get that data (you can have it dump a snapshot of a player's owned items at any point.  If you need that I suggest using either clarity or manta replay parsing libraries.

Okay, thanks a lot.


Is it possible to pass valtio store/state top-down, if i have two same component instances which have independent store instance but share same store templates?

Hi all, i'm making some kind of realworld application, which has deep-nested components and need to pass the store/state top-down through the nested components.
3 typical cases as i describe below, and i'm not sure how could i solve these cases correctly with valtio.
Case 1: Single component instance has own single store instance
it's the most simple scenario, we have a singleton Component instance and singleton store isntance.
In this case, i think we could simply use import state + useProxy to pass store instance top-down  through components.

Case 2: Multiple component instances share  same store instance
sometimes，we probably instantiate the component multiple times, but they still share the same store instance.
it's just a simple advanced version of Case 1. we could also simply get it managed using import state + useProxy

Case 3: Multiple component instances use own store instance which shares same store template
In this situation, i have totally no idea
how could i instantiate the store instance? could i instantiate the store instance inside the react functional component lifecycle?  if yes, in which phase and how?
further more，
how could the store instance be passed top-down through components  like the way above or is there any other better practice?

I'm also looking forward for the best practice of Case 1/2/3, any discussions are welcomed!

I'm also looking forward for the best practice of Case 1/2/3, any discussions are welcomed!

Hi, thanks for opening the discussion up.

i'm make some kind of realworld application

That's nice!

import state + useProxy

In v1, useProxy from valtio is renamed to useSnapshot.

is it possible to provide an API like useProxyState() to simplify the useRef and proxy?

We used to have one, but removed afterward before v1. This is because a) it's not a very typical use case, meaning case 1&2 is a primary use case, and b) it's just possible if you understand how to use useRef correctly. I feel bad providing such an api that could hide how it's working. In other words, it's not the responsibility of the library.

if i instantiate the store inside the component, is it possible to pass it top-down using Context?

Yes, that's totally fine.
That said, I think it's a doc issue. For documenting best practices, we have wiki pages instead of readme. You are welcome to contribute a new page for case 3.
The following is my idea.
import { createContext, useContext, useRef } from 'react'
import { proxy } from 'valtio'

const ProxyContext = createContext()

const Provider = ({ children }) => {
  const proxyRef = useRef()
  if (!proxyRef.current) {
    // lazy initialization
    proxyRef.current = proxy(...)
  }
  return (
    <ProxyContext.Provider value={proxyRef.current}>
      {children}
    </ProxyContext.Provider>
  )
}

const useProxyContext = () => {
  return useContext(ProxyContext)
}

Closing as answered. Feel free to have further questions.

Thanks for your answer @dai-shi.  I hadn't used contexts until now, so I want to share a full code snippet
import React, { createContext, useContext, useRef } from "react";
import { proxy, useSnapshot } from "valtio";

interface IState {
  count: number;
}

const ProxyContext = createContext<IState>({ count: 0 });

const Provider = ({ children }: { children: React.ReactNode }) => {
  const proxyRef = useRef<IState>();
  if (!proxyRef.current) {
    // lazy initialization
    proxyRef.current = proxy({ count: 0 });
  }
  return (
    <ProxyContext.Provider value={proxyRef.current}>
      {children}
    </ProxyContext.Provider>
  );
};

const useProxyContext = () => {
  return useContext(ProxyContext);
};

const ComponentWithState = () => {
  const state = useProxyContext();
  const snap = useSnapshot(state);
  return (

      <button onClick={() => ++state.count}>{snap.count}</button>

  );
};

function App() {
  return (

      <Provider>
        <ComponentWithState />
      </Provider>
      <Provider>
        <ComponentWithState />
      </Provider>

  );
}

export default App;


Firestore: Non-existing Documents

Calls to DocumentSnapshot.to_dict() for non-existing documents should return None and not an empty dictionary.

Hi Sebastian,
The DocumentSnapshot should have already raised an exception on creation if the document is not found.  So I believe that returning None would be extraneous.  Please let me know if you don't think that's the case.

We should no longer raise exceptions for missing documents, but instead return None/Null for any getData() call.


RPG Exp glitch

Had a player tell me that they had something spam that they caught a mon several times over, giving them tons of increased rpg experience. The mon in question also received a lot of experience, but it stayed at lvl1 and didn't level. They used a rare candy on it, and it shot up to level 52.

Probably related to https://github.com/ObliqueNET/Stratos/issues/75, closing in favour of that one.

I imagine this is fixed and can be closed, was referenced over on Stratos and I've closed that.


in-canvas-for-x width:auto instead of width:inherit

Issue fix.
Discussion can be found https://github.com/zurb/foundation-sites/issues/11124

Hi @RvWensen,
Here are some recommentations your future PRs:

Use the same "format" as others PRs for your PR and commits titles: feat:, fix:, docs:, refactor:, clean:, chore:...
Provide a list of the changes mades with their reasons (even if you end in repeating yourself with what was said in the related issue).
If your PR closes an issue, said it explicitely (like "Closes #XXX" or "Fixes #XXX")

Thanks for the feedback!

lgtm 👍
The inherit was simply overused. Looks better now!


[HOLD for payment 2023-07-20] [HOLD for payment 2023-07-17] Login - Crash when navigating to the concierge chat via deeplink after login

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Log out of NewDot
Go to https://staging.new.expensify.com/concierge
Verify that you are taken to the login screen
Log in to NewDot with any account

Expected Result:
You're navigated to the concierge chat after login
Actual Result:
App crashes
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[x] Android / Chrome
[ ] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: <IP_ADDRESS>
Reproducible in staging?: Yes
Reproducible in production?: No
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/78819774/0fac22f8-7138-4e71-949d-fe94938b50d9

Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01cdf0596ececb29a0
Upwork Job ID:<PHONE_NUMBER>958860801
Last Price Increase: 2023-07-24

:wave: Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open StagingDeployCash deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

Identify the pull request that introduced this issue and revert it.
Find someone who can quickly fix the issue.
Fix the issue yourself.

@c3024 are you available to submit a PR for this please?

Since this is a deploy blocker, I'm going to submit a PR now.

Yes @c3024 we'll compensate since we used your proposal (tagging a BugZero team member). Also assigning to @thesahindia for the PR review and testing.

Not overdue, will pay tomorrow, which is the 20th!

Payments can be processed now. I am OOO for 10 days so I am going to let someone else on the BZ team take care of this.

I'm confirming the payment amounts and will prepare them tomorrow. Sorry for the delay!

Alright, the payment process has started in Upwork. @c3024 please accept and I can complete the payment process.
Heads up, I didn't include the bonus because we just used the proposal, and an Expensify engineer implemented the code. Let me know if there are other questions.

@thesahindia - please update the checklist and within ND, please request payment + urgency bonus since you were involved in the review process.

Here is the payment summary:

External issue reporter - N/A
Contributor that fixed the issue - @c3024 $1000
Contributor+ that helped on the issue and/or PR - @thesahindia $1500

Upwork Job:
https://www.upwork.com/jobs/~01cdf0596ececb29a0
*If applicable, the bonuses will be applied on the final payment
Extra Notes regarding payment: https://github.com/Expensify/App/issues/22502#issuecomment-1650567246

The payment has been sent to @c3024 via Upwork.

@thesahindia please complete the checklist, so we can close this GH. Heads up, please request payment from @JmillsExpensify. Anu is currently OOO. Thank you!

@thesahindia any update on the checklist here? I'd like to close this out soon. Thank you!

Thank you for that update! I'm going to close this out now.

$1,500 payment for @thesahindia approved based on BZ summary.


fixing typos

fixed typos and called the correct reranking index on last example

for blog https://www.elastic.co/search-labs/blog/reranking-elasticsearch-hugging-face


Financial Well-Being: Move js from apps to routes js

Financial Well-Being app has its initialization code inside the routes js directory, which calls js in the apps directory. The apps directory is really for applications that have dedicated templates that call the scripts from apps directly. This doesn't happen with Financial Well-Being.
Changes

Move js from apps to routes js

How to test this PR

Pull branch and gulp clean && gulp build and http://localhost:8000/consumer-tools/financial-well-being/ and http://localhost:8000/consumer-tools/financial-well-being/results/ should work like production.

After talking with @Scotchester we're going to move everything to apps and get rid of the URL-dependent route scripts.

@Scotchester moved the scripts to apps from routes.

This works as expected, but a question: Any reason not to just load fwb-questions.js and fwb-results.js in the templates? I.e., is it necessary to create home.js and results.js?

We could, but as a general best practice I think we should avoid modules initializing themselves. Since this goes through the bundler the two files shouldn't matter.


Model Load

Hi Team!
Framework version: dspy-ai-2.5.12
I'm encountering an issue when trying to load an optimized MiproV2 model and use it in my project. I have optimized my initial model using MiproV2 and saved it as a JSON file. However, when I load the model, it seems that the original model signature is being used, not the optimized one.
Despite following the example given in Issue #617,
I’m still unable to get it to work as expected.
Here's what I did:
Optimized my model with MiproV2 and saved it as mipro_optimized_20241017_100830.json.
Tried to load the model with the following approach:
python
Copy code
entity_extractor = EntityExtractor()
entity_extractor.load("mipro_optimized_20241017_100830.json")
I did some debuging and In dspy/primitives/module.py, I can see the load_state method:
def load_state(self, state, use_legacy_loading=False):
    for name, param in self.named_parameters():
        if isinstance(param, BaseModule):
            param.load_state(state[name], use_legacy_loading=use_legacy_loading)
        else:
            # `use_legacy_loading` is only applicable for BaseModule instances.
            param.load_state(state[name])

In my debuger, after this loop I saw the correct self.signature which came from the JSON manifest.
So It appears to correctly process the JSON file, but when returning to its invoker(the main load method):
def load(self, path, use_legacy_loading=False):
    with open(path) as f:
        self.load_state(ujson.loads(f.read()), use_legacy_loading=use_legacy_loading)

The original model is loaded (so the self.signature is the original one), not the optimized version (which was loaded in load_state).
So no suprise when I invoke the language model, it still uses the original signature rather than the optimized one from the JSON manifest.
Steps to reproduce:
entity_extractor = EntityExtractor() # My DSPy program
entity_extractor.load("mipro_optimized_20241017_100830.json")
Question:

Am I missing something in the model loading process? Is there a specific step needed to properly load and utilize the optimized model?
I use a TypedPredictor—is Mipro optimizing the signature for that? Should the optimized instructions be used as part of the system message in the LLM call?

Any help or guidance would be much appreciated. Thanks!
Darvi

Hey @fireking77 ! Thanks a lot for opening this.
It's possible that this is a bug in saving or loading. Let's figure out which one it is. Can you check the JSON file? Does it have your optimized instruction and/or examples? Check for extended_signature and signature.

Hi @okhat !

Can you check the JSON file? Does it have your optimized instruction and/or examples?
Yes it have the optimisation part is perfect. This is why I asked this question :)

Check for extended_signature and signature.
I can't see the extended_signature in the JSON, only the signature:

"signature": {
      "instructions": "Extract and analyze entities from input descriptions related to fleet management and security systems.......<continues>

I installed the following packages to monitor the LLM calls, and it’s sending my original prompt, not the one shown above:
arize-phoenix
openinference-instrumentation-dspy

If you can tell me what I should debug, I’m ready to work with you on this!
By the way, aside from this hiccup, the framework works great and is very easy to use.
Thanks for all the time and effort you’ve put into this!
Darvi

Hey @fireking77 ! Thank you, this is super helpful!
So the JSON file has the proper, optimized instructions? But they're just not getting loaded?
Or are they not getting saved?

adding @chenmoneygithub re: saving

Hey @fireking77 ! Thank you, this is super helpful!
So the JSON file has the proper, optimized instructions? But they're just not getting loaded?
Or are they not getting saved?

Not getting loaded, I think it is saved properly...I copied the part above from that JSON which was created after the mipro optimisation like this:
    # Optimize program
    print(f"Optimizing program with MIPRO...")
    optimized_program = teleprompter.compile(
        student=entity_extractor.deepcopy(),
        trainset=train_set,
        valset=validation_set,
        num_trials=NUM_OF_TRIALS,
        minibatch=False,
        # minibatch_size=15,
        # minibatch_full_eval_steps=10,
        max_labeled_demos=MAX_LABELED_DEMO,
        max_bootstrapped_demos=MAX_BOOTSTRAP_DEMO,
        requires_permission_to_run=False,
    )

    optimized_program.save("_data/entity_extractor_model.json")

I copied the whole thing from the doc

The original model is loaded (so the self.signature is the original one), not the optimized version (which was loaded in load_state).
@fireking77 could you elaborate a bit about this? load_state is invoked by this load, so once the state loading is successful it shouldn't be overwritten.

demo1: original instruction

demo2: optimized / expected instruction

demo3: original instruction

@fireking77  I just realized that it's weird that MiproV2 is changing your signature instead of extended_signature. @okhat  Omar I think we expect only the extended_signature to be changed by MiproV2?
For this specific loading, I am still looking into it.

I tried to manually change the signature in json file, and I am able to load it back successfully:

@fireking77 Could you share how you define the DSPy model? We cannot reproduce the error on our side.

I think I found the issue:
class TypedPredictor(dspy.Module):
    def __init__(self, signature, instructions=None, *, max_retries=3, wrap_json=False, explain_errors=False):
        """Like dspy.Predict, but enforces type annotations in the signature.

        Args:
            signature: The signature of the module. Can use type annotations.
            instructions: A description of what the model should do.
            max_retries: The number of times to retry the prediction if the output is invalid.
            wrap_json: If True, json objects in the input will be wrapped in ```json ... ```
            explain_errors: If True, the model will try to explain the errors it encounters.
        """
        super().__init__()
        self.signature = ensure_signature(signature, instructions)
        self.predictor = dspy.Predict(signature, _parse_values=False)
        self.max_retries = max_retries
        self.wrap_json = wrap_json
        self.explain_errors = explain_errors

@fireking77  In our TypedPredictor we store this self.signature, but saving/loading only handles this self.predictor. That means when you are loading the saved file, self.signature will diverge from self.predictor.signature. We have two options here:

Add saving/loading logic for TypedPredictor.
Remove the self.signature in TypePredictor.

@okhat  Omar please let me know which one you prefer.

@chenmoneygithub We can remove the self.signature but it may break other stuff. How much code is saving this signature?
FYI: TypedPredictors will be deleted in 2.6 since dspy.Predict will replace their functionality/

@chenmoneygithub , @okhat ,

I think I found the issue:
In our TypedPredictor we store this self.signature, but saving/loading only handles this self.predictor. That means when you are loading the saved file, self.signature will diverge from self.predictor.signature.

@chenmoneygithub Thanks so much for identifying this.
Omar mentioned that the entire TypedPredictor will be removed from the codebase with the upcoming Predict module.
In the meantime, how should I work around this issue? Should I use Typed CoT? Should I install the bleeding edge version of dspy from the repo? 😄
What do you suggest to help me move forward?
Actually, I manually added the optimized signature to my signature definition, and it worked! 😄 I noticed better performance. But how can I automate this until version 2.6?
Thanks,
Darvi

Hey @fireking77 ! I see a few paths.
The most temporary fix: If your predictor needs to be a TypedPredictor (i.e., dspy.Predict doesn't cut it), if your compile run is cached, you can recreate the same optimized program by compiling every time. It takes a few seconds and costs $0 if your cache is still there.
Beyond that, if you'd like to help us implement the fix "Add saving/loading logic for TypedPredictor" that would be awesome. The other path is to wait for @chenmoneygithub to do it, which may mean a Monday/Tuesday release.
Let me know what you think. Happy to help with any of these options.

Omar,
For now, I'm manually changing the signature, but I plan to deep dive into the framework once I'm able/would be able to contribute a fix I will contribute help. Until then, the best I can do for the sake of everybody :D is some debugging and testing. So there's no rush on my end.
May I ask another question? In the next release, you mentioned removing this module. I believe the base Predict module will become the typed predictor, so it should be able to pipeline an output like this, correct?
class ExtractEntityFromDescriptionOutput(BaseModel):
    entity_hu: str = Field(
        description="The extracted entity in Hungarian, refactored, lowercased, and cleaned from the input description, without any special Hungarian characters (only ASCII characters)."
    )
    entity_en: str = Field(
        description="The English translation of the extracted Hungarian entity."
    )
    is_inverted: bool = Field(
        description="Boolean flag indicating if the input is connected in an inverted way (for example, True if 'fordított', 'ford', 'for', or 'f' is present in the input description)."
    )
    categories: str = Field(
        description="It contains the categories, separated by '|', to which the extracted entity belongs."
    )
    review: bool = Field(
        description="Boolean flag indicating if the entity cannot be accurately identified or there's low confidence in the extraction."
    )

    def get_categories(self) -> Set[EntityCategory]:
        return set([EntityCategory(category.strip()) for category in self.categories.split("|")])

Thanks, guys! I really appreciate all the effort.
Darvi

@okhat  I think we can do a simple fix like this: https://github.com/stanfordnlp/dspy/pull/1651
This keeps TypePredictor.signature, but doesn't maintain it as a class attribute. @fireking77  I think the above PR should resolve your issue, please help patch and try, thanks!

Nice, thanks a ton @chenmoneygithub !!
Hey @fireking77 this is now on main but I can push to pypi if needed.

Published to 2.5.13 !

Guys! @chenmoneygithub @okhat
I upgraded from PyPi to 2.5.13
!! Works as a charm !! 🥇

This is the optimised signature :D
I am going to check the patch you have made...because i would like to gather deeper knowledge about this tech you have made.
Thanks so much..now I can play more :D
Darvi


I have a problem with my Flutter application.

Steps to Reproduce

Run flutter create bug.
Update the files as follows: ...
...

Expected results:
Actual results:

Logs

No detail is available here. could you please recheck

Looks like this was created by accident. @JuriBurakov if you meant to file an issue, please try again.


[RFC] Move editor to its own top-level directory

[RFC] Move editor to its own top-level directory
Not sure why it's under python/src/aiconfig

It's in there because the most obvious way to package the flask server with the react files was to build everything into the python-aiconfig package. This might break the packaging and we'll have to set up a new way to ship the editor.

If we still want to do this, let's do it after we're done MVP. We're touching everywhere in it now so not a good time

If we still want to do this, let's do it after we're done MVP. We're touching everywhere in it now so not a good time

Yeah, I have this as a follow-up item here: https://docs.google.com/document/d/1YwUlIQZX8CiuvtbVR4OvpFQ2nphFITRD_XZR3CDNtoE/edit#bookmark=id.8h0jqqaolx9z
Hopefully we will find another way to package this that allows sensible code structure.


Basket. Confirmation screen for single operation

Show confirm screen for operation when click on it or select only one and sign
List of confirm screens
Transfer/xcm transfer
Stacking operations
Manage proxy operations

https://github.com/novasamatech/nova-spektr/issues/1892

Status verification: ✅


[WIP] feat: set restrictive security contexts in KFM deployments

Description
Summary
It has been detected that starting from OCP 4.11 when KFM K8s Deployment is deployed the following warnings are shown:
serviceaccount/kas-fleet-manager configured
Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "migration", "service", "envoy-sidecar" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "migration", "service", "envoy-sidecar" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "migration", "service", "envoy-sidecar" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "migration", "service", "envoy-sidecar" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/kas-fleet-manager created

This warnings are related to the security context configuration of the deployments
This PR mitigates most of the warnings shown above with the exception of one which cannot be avoided at the moment (see below for details on why). After the changes the only remaining warning shown is the one related to seccompProfile:
Warning: would violate PodSecurity "restricted:v1.24": seccompProfile (pod or containers "migration", "service", "envoy-sidecar" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

This changes also ensure that KFM can be deployed in OCP 4.11 and earlier versions too even when a restricted security profile is enforced (see below for details)
Details and motivations of the change
Starting from K8s 1.25 a new K8s built-in admission controller called Pod Security Admission Control will allow cluster admins to enforce K8s Pod Security Standards with Namespace Labels. Additionally, OpenShift Security Context Constraints (SCC) are in the progress of changing to align with the new criteria to admit workloads according to the K8s Pod Security Standards definitions.
With the introduction of the new built-in admission controller that enforces K8s Pod Secuity Standards three policies belonging to those standards are defined. One of those policies is the Restricted policy.
As part of the changes done in OpenShift in the SCC to address the new needs new SCC policies have been defined starting in OpenShift 4.11. Specifically, the changes made align the deployments to comply with the new introduced SCC policy named restricted-v2 which is the most restrictive and it is the policy assigned by default to the namespaces created by non-privileged users.
The set of changes change the KFM deployments to align with the new SCC restricted-v2 policy. This:

Removes some of the warnings that started to appear when deploying KFM starting in OCP 4.11 in relation to the security context not being configured to comply with the restricted-v2 SCC policy
Ensure that KFM can be run with the restricted-v2 SCC policy. This allows non-privileged users to deploy KFM in OCP 4.11 and newer versions where admins decide to enforce restrictions according to the K8s Pod Security Standards
Ensure that KFM can run in K8s clusters where cluster admins decide to enforce restrictions according to the K8s Pod Security Standards (with some caveats in some K8s versions as explained below)
Follows best practices around giving the least amount of privileges needed for the workloads and follows K8s Pod Standard Security practices
Make explicit the security needs of the KFM workloads

Notice that although the changes align to the restricted-v2 SCC policy, it does not fully align with the Restricted K8s policy. That is intentional due to at the moment KFM needs to be compatible with OCP 4.10 and 4.11: In OCP 4.10 the SCC restricted policy does not allow to define the securityContext seccompProfile attribute required by the K8s restricted policy. To be compatible with 4.10 and 4.11 this attribute must be left unset. In that case SCC itself defaults it to runtime/default so it is ok to leave it unset.
References

K8s Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/
OpenShift Security Context Constraints (SCC):  https://docs.openshift.com/container-platform/4.11/authentication/managing-security-context-constraints.html
K8s Pod Security Admission controller: https://kubernetes.io/docs/concepts/security/pod-security-admission/
Pod Security Admission in OpenShift 4.11: https://cloud.redhat.com/blog/pod-security-admission-in-openshift-4.11
Operators Pod Security Standards best practices: https://sdk.operatorframework.io/docs/best-practices/pod-security-standards/

Verification Steps

Verify that KFM can be deployed and works correctly in OCP 4.10 and in OCP 4.11 without issues.
Verify that when deploying KFM in OCP 4.11 only the following warning related to seccompProfile is shown:

Warning: would violate PodSecurity "restricted:v1.24": seccompProfile (pod or containers "migration", "service", "envoy-sidecar" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/kas-fleet-manager created

Checklist (Definition of Done)

[ ] All acceptance criteria specified in JIRA have been completed
[ ] Unit and integration tests added that prove the fix is effective or the feature works (tested against emulated and non-emulated OCM environment)
[ ] Documentation added for the feature
[ ] CI and all relevant tests are passing
[ ] Code Review completed
[ ] Verified independently by reviewer
[ ] All PR comments are resolved either by addressing them or creating follow up tasks
[ ] Required metrics/dashboards/alerts have been added (or PR created).
[ ] Required Standard Operating Procedure (SOP) is added.
[ ] JIRA has been created for changes required on the client side

The reeason why we cannot completely remove all the warnings is due to we need to keep compatibility with OCP versions older than 4.11 at the moment: In stage and production we are currently using an older version.
cc @pb82 in theory it should be possible to completely remove all warnings when we upgrade the environments in the future (in a different PR) at the expense of not supporting OCP versions earlier than 4.11 in KFM. However, I am not sure if we will be able to do it due to Enterprise or other requirements. Are there some requirements around Enterprise and OCP versions support? and for non-enterprise?

@miguelsorianod for Enterprise, we're not running the control plane on the customer's cluster, and looking at the changes, the resources created on the data plane by the KFM haven't changed. So Enterprise should not be a concern here.

@miguelsorianod Out control plane clusters are already on 4.12, so I guess this PR can be unwipped?

@miguelsorianod Our control plane clusters are already on 4.12, so I guess this PR can be unwipped?

There are also testing pipelines clusters that we've to consider that uses kas-installer. Before this PR is merged, let's ping QE to make sure that their clusters are running on the version we expect

@miguelsorianod Our control plane clusters are already on 4.12, so I guess this PR can be unwipped?

Ok. In that case we should specify in the doc requirements to mention that the minimum OCP version supported for KFM is OCP 4.11. Are we ok with it?

@miguelsorianod let me confirm with QE first, Manyanda mentioned that there might still be pipelines running the KFM on < 4.11.

@miguelsorianod let me confirm with QE first, Manyanda mentioned that there might still be pipelines running the KFM on < 4.11.

Yes. I have been talking to them. At the moment they are using < 4.11. I am currently discussing what is the viability of upgrading with them

I talked to QE and changes have been made to upgrade the clusters to OCP 4.11 on that side.
I updated this PR adding the additional changes to fully eliminate all warnings.
This PR should be ready for final review and merge. cc @pb82 , @machi1990

Thanks @miguelsorianod
This looks good to me.


Language features

What do we think about having language features such as:
Function overloading
Default parameters
Operator overloading
Template classes
Multiple inheritance
Delegate constructors

This is a very importnat desicution once we have a general direction we can start making start making breaking it down into a list and work syntax in baced on the features list. let me start thinking a bou tthis we should probably pick Paradigms


Fix observe so new object assignments are also reported

This fixes a bug where changes to a property being assigned an object value would not be reported by observe, leading to missing information in the observer stream. This now calls observe on new assignments when applicable and updates tests.
Change-type: patch

I self-certify!


pull request for e2e test at 12/28/2016 19:56:12

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Use node 18 for circleci and AWS

:tada: This PR is included in version 1.16.2 :tada:
The release is available on:

v1.16.2
v1.16.2
GitHub release

Your semantic-release bot :package::rocket:


Remove DifferentialPrivacyParams from MeasurementSpec

Reasons for this proposal:

Raimundo, Pasin and Jiayu computed the optimal differential privacy parameters for the different types of queries.  There is not any reason to use values other than these.  See this doc https://docs.google.com/document/d/1JJA3ufmWgDNblDMKTB2KDLEQjBpgR12kVYVOozjdZ3M/edit
If we allow MeasurementConsumers to choose arbitrary values for the differential privacy parameters, then we will not be able to use adaptive composition to compute total privacy budget usage.  Instead, we will have to revert to a more conservative approach which would reduce the total number of queries that can be performed.
It is envisioned that there may be multiple implementations of Reporting and Planning Front-ends.  Can we trust that each of these implementations will use the correct values of epsilon and delta?

Instead of having the RPFE provide the values of epsilon and delta, they should be hard-coded into the Kingdom, and a table lookup for these values should be performed based on the type of query.

We should discuss this in one of the workstreams. maybe in "R&F Planning - Eval Framework & Conceptual Design".


Upgrade axios

Summary
Describe the goal of this PR. Mention any related Issue numbers.
Requirements (place an x in each [ ])

[x] I've read and understood the Contributing Guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.

Resolves this issue

#1985


Bug Fix: Profile not showing on website

Fixes #82

Fix: syntax error in profiles.json file.
Now all profiles are loading properly on website.

Screenshot

Updates

Along with the bug fix, I also added my Profile data into profiles.json with proper syntax.

Hi @pushkaraj2007, done changes. I will create a new PR for profile addition.
Preview: https://techcreators-git-fork-alkaison-fixprofilesloading-pushkaraj.vercel.app/


Implement Gitpod deployment.

@leenyburger had this working for her workshop at RailsConf. Would love to make the same option available for vanilla Bullet Train projects.

@andrewculver can you say a little more about what you're hoping to see here?


Fixed internal/xerrors/transportError.Error() method (replaced li…

…ne feed to carriage return)
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Pull request type

Please check the type of change your PR introduces:

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, renaming)
[x] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

What is the current behavior?

Issue Number: #293
What is the new behavior?

Other information

Codecov Report

Merging #295 (7d1417a) into master (b04d001) will decrease coverage by 44.63%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           master     #295       +/-   ##
===========================================
- Coverage   59.39%   14.76%   -44.64%
===========================================
  Files         168      131       -37
  Lines       16036    15228      -808
===========================================
- Hits         9525     2248     -7277
- Misses       5957    12773     +6816
+ Partials      554      207      -347

Flag
Coverage Δ

1.14.x
?

1.15.x
?

1.16.x
?

1.17.x
14.76% <0.00%> (-44.64%)
:arrow_down:

1.18.x
?

connection
?

discovery
?

e2e
14.76% <0.00%> (-31.05%)
:arrow_down:

integration
14.76% <0.00%> (-31.05%)
:arrow_down:

macOS-latest
?

ratelimiter
14.76% <0.00%> (ø)

scripting
?

table
?

ubuntu-latest
14.76% <0.00%> (-44.64%)
:arrow_down:

unit
?

windows-latest
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b04d001...7d1417a. Read the comment docs.


Call ExtendedKustoIngestClient.IngestFromStreamAsync with individual json document 20 times/second is fine or not?

Hi,
The page contains a lot of recommendations in direction of batching the data to be ingested. What I don't understand though is why the client application code should do the batching in case of Queued Ingestion? In that case Azure Data Explorer can itself decide how often (e.g. every 5 minutes) and how many queue messages to read at once, right?
Can I simply ingest individual json documents (for each one call ExtendedKustoIngestClient.IngestFromStreamAsync where the stream will contain only 1 json document), and do this e.g. 10-20/second?
Thanks in advance and best regards,
Deyan Petrov

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 47ff6172-c040-65e6-55aa-1e418e5c53b9
Version Independent ID: 74022ecc-45f6-99dd-873b-2cf6ebfd8676
Content: Kusto ingest client library best practices - Azure Data Explorer
Content Source: data-explorer/kusto/api/netfx/kusto-ingest-best-practices.md
Service: data-explorer
GitHub Login: @orspod
Microsoft Alias: orspodek

It should be actually the same as ingestion from Event Hubs where the client can write individual events to Event Hub at any time, and ADX ingests them every 5 minutes (batching them) ...

Thanks for the feedback and bringing this to our notice . At this time we are reviewing the feedback and will update the document as appropriate .

@deyanp - thanks for the feedback. @vladikbr - can you assist with an answer?

@deyanp,
First of all, ingesting your data record at a time will incur costs - every IngestFromStream call involves a storage write transaction.
It is true that the ADX ingestion service will batch the incoming data (or more correctly, the blobs containing that data), but the more source data is fragmented, the harder the ingestion service will work and the more storage transactions will be needed.
So, if you are seeking to ingest truly "trickling" data, are better off with Event Hub ingestion or with Streaming Ingestion (https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/streamingingestionpolicy).
In case you wish to stick to the classic queued ingestion, I would recommend to batch the data on the client for as long as your latency requirements allow.
And last, you should be switching away from the ExtendedIngestClient methods (those are there for back compat), and move to IKustoIngestClient.IngestFromStreamAsync (https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/netfx/kusto-ingest-client-reference#interface-ikustoingestclient)

@vladikbr, thanks for the answer and the hints about event hubs and IKustoIngestClient.IngestFromStreamAsync!
I am looking for the easiest (from client code point of view) way to ingest "trickling" data as you called it into ADX ...
Creating Event Hubs just for this sounds like an overkill (and additional costs) ... Do you mean that every call to IKustoIngestClient.IngestFromStreamAsync results in a separate blob written on a storage account? I was thinking you write a queue message only, which sounded equivalent to writing event hub messages, however it seems I am mistaken?

@deyanp,
Yes, every call to IKustoIngestClient.IngestFromXXX (other than IngestFromBlob) includes a write transaction to Blob storage. A queue message is also written to point to the written blob (or to the existing blob in case of IngestFromBlob).
If your scenario is trickling data, Streaming ingestion is for you. This mechanism was specifically designed for trickling scenarios.

@vladikbr thanks for the response.  @deyanp thanks for your feedback.
I will now close this issue.

Thanks @vladikbr , but are you sure Streaming Ingestion is applicable to my scenario where I have only a few events per second? I see this in the docs:
"This ingestion method is suitable for ingesting a high volume of data, such as thousands of records per second, spread over thousands of tables. Each table receives a relatively low volume of data, such as a few records per second."
Which is also a bit controversial, as the first sentence says thousands of records per second, and the second sentence ends with a few records per second ...


tests: Test with Django 5.1 RC

New RC is available.

@sentrivana Django tests are currently broken (not even running) in master, this PR also seems to be affected. I am investigating


kubeadm: update unversioned kubelet config map fg info

The feature gate goes GA in 1.25 and becomes locked by
default to "enabled". It can be removed from the list.
/kind feature
/sig cluster-lifecycle
(this is targeting 1.25)
xref https://github.com/kubernetes/kubeadm/issues/1582

/cc @pacoxu
PTAL

/milestone 1.25
/cc @kcmartin

Happy with changes since https://github.com/kubernetes/website/pull/34159#pullrequestreview-999024002
/lgtm

/approve


Head stabilization complex terrain & improved visual tracking algorithm

Description
This is a continuation of #136. Main tasks are:

Improve visual tracking algorithm after the track-cam bug fix (WIP)
Deploy head stabilization model in closed simulation loop (done)

Types of changes

[ ] Bugfix
[x] New feature
[ ] Refactor / Code style update (no logical changes)
[ ] Build / CI changes
[ ] Documentation Update
[ ] Other (explain)

Does this address any currently open issues?
#137

Should be all good now, but after #143 let me rerun everything again tomorrow.
https://github.com/NeLy-EPFL/flygym/assets/23410765/e971f31e-d262-49c9-aa22-4c27eef25cc7
https://github.com/NeLy-EPFL/flygym/assets/23410765/f3888964-706e-4d80-ad5a-cd90bd9ed500

@tkclam Would you mind taking a look at this? #143 has been merged into this (first merged into dev-v1.0.0, which was then merged into this).

Data collection and training are quite time-consuming. Should we store trained model weights in flygym/data?
Also, should we include the required packages in extras_require in setup.py?
"examples": [
    "lightning",
    "pandas",
    "scikit-learn",
    "seaborn",
    "torch",
    "tensorboardX",
    "flyvision @ https://github.com/TuragaLab/flyvis/archive/refs/heads/main.zip",
]

Data collection and training are quite time-consuming. Should we store trained model weights in flygym/data? Also, should we include the required packages in extras_require in setup.py?
"examples": [
    "lightning",
    "pandas",
    "scikit-learn",
    "seaborn",
    "torch",
    "tensorboardX",
    "flyvision @ https://github.com/TuragaLab/flyvis/archive/refs/heads/main.zip",
]

Yup sounds good. About tensorboard/tensorboardX though, I think it's shipped with lightning. Maybe it's better if we let lightning take care of installing the correct version/variant?

Data collection and training are quite time-consuming. Should we store trained model weights in flygym/data? Also, should we include the required packages in extras_require in setup.py?
"examples": [
    "lightning",
    "pandas",
    "scikit-learn",
    "seaborn",
    "torch",
    "tensorboardX",
    "flyvision @ https://github.com/TuragaLab/flyvis/archive/refs/heads/main.zip",
]

Yup sounds good. About tensorboard/tensorboardX though, I think it's shipped with lightning. Maybe it's better if we let lightning take care of installing the correct version/variant?
Edit: ~I will also add networkx since it's used in another example.~ networkx was previously included under an entry of extras_require called tutorials. I will move it to examples too.

Just tested the installation of lightning with a new environmen. Looks like tensorboardX is only included if we install it with pip install "lightning[extra]"

Data collection and training are quite time-consuming. Should we store trained model weights in flygym/data? Also, should we include the required packages in extras_require in setup.py?
"examples": [
    "lightning",
    "pandas",
    "scikit-learn",
    "seaborn",
    "torch",
    "tensorboardX",
    "flyvision @ https://github.com/TuragaLab/flyvis/archive/refs/heads/main.zip",
]

Yup sounds good. About tensorboard/tensorboardX though, I think it's shipped with lightning. Maybe it's better if we let lightning take care of installing the correct version/variant?
Edit: ~I will also add networkx since it's used in another example.~ networkx was previously included under an entry of extras_require called tutorials. I will move it to examples too.

Just tested the installation of lightning with a new environmen. Looks like tensorboardX is only included if we install it with pip install "lightning[extra]"

ok then i will include tensorboardX explicitly. Thanks!

All issues should be addressed now. I don't think the change broke anything but I will rerun all the data collection / training / closed look simulation again to make extra sure.


bootloader entry point

I'm sorry to bother you with this simple question.
I have a bootloader that I want to find a vulnerability, so I am testing it with the bootloaders you provided.
For some reason, my ida couldn't decompile if it was 32 bits, so I did huawei's bootloader
The problem is that I need to specify an entry point before running find_tint.py, but I don't know this
I couldn't find the satisfied answer because of my lack of search skills.
Could you tell me how you designated the entry point for the bootloader?

Usually it is a matter of static analysis - you need to find where most calls are started and where there is no other reference to your potential "main". Just try to search for this. From the other point, i don't understand why do you need "main" specifically. In a context of symbolic execution you need to find an "entry point" (nothing like actual main) of the interesting function, like memory management or verified boot

Usually it is a matter of static analysis - you need to find where most calls are started and where there is no other reference to your potential "main". Just try to search for this. From the other point, i don't understand why do you need "main" specifically. In a context of symbolic execution you need to find an "entry point" (nothing like actual main) of the interesting function, like memory management or verified boot

Thank you for your kind and quick reply.
The reason why I asked for bootloader's entry point was because of the error shown in the picture below.
The qualqumm where ELF Header exists did not show the error above, so I thought I need an entry point.
Anyway, thank you for your good comments.
I tried to change the direction and solve the above error itself, and there was an article with the same problem, so I solved it by referring to the article.
https://github.com/ucsb-seclab/BootStomp/issues/3
I think it's going well now.
Thank you for your research.

Thank you for your kind and quick reply.
The reason why I asked for bootloader's entry point was because of the error shown in the picture below.
The qualqumm where ELF Header exists did not show the error above, so I thought I need an entry point.
Anyway, thank you for your good comments.
I tried to change the direction and solve the above error itself, and there was an article with the same problem, so I solved it by referring to the article.
https://github.com/ucsb-seclab/BootStomp/issues/3
I think it's going well now.
Thank you for your research.


App window flashes at App launch on Android

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Kill the App
Launch the App
Watch the splash screen

Expected Result:
Splash screen should have the dark background
Actual Result:
Splash screen flashes, from white to dark background
Workaround:
None
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[ ] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number:
Reproducible in staging?:
Reproducible in production?:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
Expensify/Expensify Issue URL:
Issue reported by: @quinthar
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1671685986662779
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01b593296a186bb13a
Upwork Job ID:<PHONE_NUMBER>665060864
Last Price Increase: 2022-12-23

@Julesssss isn't this a duplicate report of what we encountered while solving https://github.com/Expensify/App/issues/8236? Not going to triage until we align on whether this is a bug or not.

I and Jason had a 1:1 call and we concluded that this is a different issue than what the team discussed before. As far as I understood, the previous issue was related to the status bar color, but this one is related to the App window background. Here is a screen recording with the current issue:
https://user-images.githubusercontent.com/18078393/209392752-83dc82a4-5542-48f2-90b8-7b2c52183af4.mp4
As you can see, after killing the App and opening it again, the App screen is white and then dark.
@Julesssss Related to the status bar and also navigation bar. Have we considered to make the App render behind them and add a padding  according to the bars height, to make sure that the App content does not overlap with the bars content? I guess the challenge is when the screen background changes, we have to delegate to the native code to update the activity background color as well, but maybe we could do some brainstorming and find a robust and scalable solution to support this. It would be a great achievement if we raise the Android App ui to the level of the iOS App.

Super productive call, thank you! Yeah it seems like we still have a couple of improvements left to make when it comes to Android – whether it's the status bar on app load, the background color, the app icon, as well as the bottom border where the swipe action happens. All of those seem to either flash white or are permanently white, so it's inconsistent with the overall branding and our official colors. I agree this is a good one to keep internal, at least until we have a better sense of scope and next steps.

@cristipaval is out for this week, but he has a good plan for the path forward and this bug is still young, so I'm going to keep him assigned.

Waiting for @Julesssss, he had some health emergencies, he'll be online tomorrow.
One thing I want to put down here, is that we're about to add the ability for the user to choose between light/dark themes. We need to find a solution for the App window to instantly set the background according to the user preference.

I guess the challenge is when the screen background changes, we have to delegate to the native code to update the activity background color as well, but maybe we could do some brainstorming and find a robust and scalable solution to support this. It would be a great achievement if we raise the Android App ui to the level of the iOS App.

Yeah, I agee and have come up with many solutions but unfortunately none of them are ideal. Will be looking into this again soon.

Interestingly, I'm not seeing this issue on Android 13. I'm on app version 1.2.48-2 and I see the dark window background no matter how the app is opened (from background, fresh open after kill, fresh open after back button press):
https://user-images.githubusercontent.com/10736861/210761018-4c16aed9-d957-469a-9683-5cbb034b484b.mp4
Did I miss something? Else I wonder if this is a device-specific issue.

Could you switch the theme in your phone settings? Set light theme if possible and try again. I wonder if this is related to the theme of the phone settings.

Closing this issue per this Slack convo. We're handling it elsewhere.


Gradio app returns error

Gradio app returns error for all images including the examples

My bad. It works.


Impersonate managed service account

Hi and thanks, @mj1856, for this great library! My sys admin has created a managed service account (MSA), which is tied to a machine and has no visible password. That account has read/write permissions to a folder on the network where I need to write files to from an MVC C# app.
I've created a question about this on StackOverflow: https://stackoverflow.com/questions/53783296/write-files-to-network-drive-from-iis-using-managed-service-account
Is it possible to impersonate an MSA for this purpose? I've tried this code:
var cred = new UserCredentials("dot", "msa-pims-dev$", "");
Impersonation.RunAsUser(cred, LogonType.Batch, () =>
	{
		System.IO.File.WriteAllText(@"\\SomeUncPath\Reports", "sample text");
	}
);

The above throws this:
System.ArgumentException: Password cannot be empty or consist solely of whitespace characters. Parameter name: password
Thanks.

Found the solution: needed to include the domain before the MSA when setting it on IIS.


Option to Auto-Download New Chapters for Series Only If No Unread Chapters

Describe your suggested feature
Basically an option that automatically downloads new chapters from a series ONLY if you don't have any unread chapters. It could be a suboption for "Download new chapters" that you can choose whether you turn on or off.
Other details
The only thing that I skip on the Smart Update is for entries with completed status. So if I do a global update, I'll get new chapters even if I have unread chapters. The reason for this is because I still want to know that the said series have new chapters, so I'll know that I have enough to binge-read haha. With that said, because of this I can't turn on the auto-download new chapters because it will download them all. I want an option to just download new chapters from entries that I have no unread chapters—series that I don't want to gather chapters first before I read because I really want to stay updated on it.
Before, I actually skip entries with unread chapters on Smart Update, so I have no problem with auto-download. But, that makes it so that I won't know whether the entries I'm waiting to gather more chapters to binge-read have new chapters. In the end, sometimes I even forgot the said series lol.
I hope you consider it, thanks!
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] If this is an issue with an official extension, I should be opening an issue in the extensions repository.
[X] I have updated the app to version 1.11.2.
[X] I will fill out all of the requested information in this form.

You can do this:

Create a category named "Skip-download"
Go to Settings/Downloads/Auto-download then add that category to Exclude.
If you are using preview version then you can hide that category from Library if you don't want to see it.

Yes, manual but better control rather than an option for entire library.

Yes, manual but better control rather than an option for entire library.

Hello again. So I tried it and it works, but the option you told me to hide categories doesn't work somehow. You need to click the "eye" button right? It's not hidden for some reason.

To actually hide them, go to your library, press the 3 horizontal lines, go to display and click on "Hide hidden categories"

Maybe I should enable the option by default.

@cuong-tran then by default the filter in the library will be yellow and so active?
How about switching the logic: the filter will be inactive by default but it's the opposite logic "show hidden categories". And so we also avoid the confusing double negation.

Cool

@cuong-tran then by default the filter in the library will be yellow and so active? How about switching the logic: the filter will be inactive by default but it's the opposite logic "show hidden categories". And so we also avoid the confusing double negation.

I did it just following your suggestion. Thank you.