Add device failed: {"ERROR":true,"MESSAGE":"Device exists elsewhere"}

Any idea what is causing this?
Apr 17 15:53:49 deckconnect08 node[1627]: Starting
Apr 17 15:53:49 deckconnect08 node[1627]: adding new device: 0001:0002:00
Apr 17 15:53:49 deckconnect08 node[1627]: existing = []
Apr 17 15:53:49 deckconnect08 node[1627]: Registering key events for AL49J2C11001
Apr 17 15:53:49 deckconnect08 node[1627]: Connecting to <IP_ADDRESS>:16622
Apr 17 15:53:49 deckconnect08 node[1627]: Connected
Apr 17 15:53:49 deckconnect08 node[1627]: connected
Apr 17 15:53:49 deckconnect08 node[1627]: registerAll [ 'AL49J2C11001' ]
Apr 17 15:53:49 deckconnect08 node[1627]: Connected to Companion: CompanionVersion=2.2.0+4003-493404d6 ApiVersion=1.1.0
Apr 17 15:53:49 deckconnect08 node[1627]: Add device failed: {"ERROR":true,"MESSAGE":"Device exists elsewhere"}

Nevermind.   I rebooted the companion machine and it works.

OK...I am opening this issue up again as I now have reproduction steps.
If a Satellite reboots, then it fail with this error.  If I then reboot the Companion server, it can reconnect again.  So everytime a Satellite needs to reconnect, the server needs a reboot.  This is very prone to failure as this level of intervention may not always be possible.

This seems to be happening with v2 StreamDecks only.

I can confirm this issue as well. I can also confirm it does not happen with the 0.4.0 release. Or at least loading and then unloading the 0.4.0 the stream decks work fine.

This sounds more like a problem with companion than satellite, but as it will be a bug in the satellite api tracking it here is fine.
I have seen this every now and then but havent been able to identify reliable steps to reproduce the issue.
I did have it just now when testing out satellite on a pi, after it rebooted following pulling the power, but after doing it again it was fine

I think that every time I see this is soon after the satellite device has been disconnected. So I am wondering if this is a case of sockets not being closed promptly enough.
A solution to this is to retry adding a device a short time after it has failed
Please try again with the latest 2.2 beta, and the latest commits to this repository

It happens for me when my VPN breaks the link to companion from the satellite. Regaining the satellite connection is touch and go.
Sent quickly from my phone

On Apr 20, 2022, at 4:43 PM, Julian Waller @.***> wrote:

This sounds more like a problem with companion than satellite, but as it will be a bug in the satellite api tracking it here is fine.
I have seen this every now and then but havent been able to identify reliable steps to reproduce the issue.
I did have it just now when testing out satellite on a pi, after it rebooted following pulling the power, but after doing it again it was fine
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.

I have a mix of local (on-net) Satellites and remote VPN connected Satellites, and see it in both circumstances.   I'll test the last commits in the next day or two.

Will do later tonight!
Sent quickly from my phone

On Apr 20, 2022, at 5:45 PM, Julian Waller @.***> wrote:

I think that every time I see this is soon after the satellite device has been disconnected. So I am wondering if this is a case of sockets not being closed promptly enough.
A solution to this is to retry adding a device a short time after it has failed
Please try again with the latest 2.2 beta, and the latest commits to this repository
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.

Git pulled both Companion and Satellite.
Now getting this failure:
Apr 20 15:32:44 deckconnect08 node[660]: Starting
Apr 20 15:32:44 deckconnect08 node[660]: adding new device: /dev/hidraw0
Apr 20 15:32:44 deckconnect08 node[660]: existing = []
Apr 20 15:32:44 deckconnect08 node[660]: Open "/dev/hidraw0" failed: TypeError: cannot open device with path /dev/hidraw0
Apr 20 15:32:44 deckconnect08 node[660]: adding new device: /dev/hidraw1
Apr 20 15:32:44 deckconnect08 node[660]: existing = []
Apr 20 15:32:44 deckconnect08 node[660]: Open "/dev/hidraw1" failed: TypeError: cannot open device with path /dev/hidraw1
Apr 20 15:32:44 deckconnect08 node[660]: adding new device: /dev/hidraw2
Apr 20 15:32:44 deckconnect08 node[660]: existing = []
Apr 20 15:32:44 deckconnect08 node[660]: Open "/dev/hidraw2" failed: TypeError: cannot open device with path /dev/hidraw2
Apr 20 15:32:44 deckconnect08 node[660]: Connecting to <IP_ADDRESS>:16622
Apr 20 15:32:44 deckconnect08 node[660]: Connected
Apr 20 15:32:44 deckconnect08 node[660]: connected
Apr 20 15:32:44 deckconnect08 node[660]: registerAll []
Apr 20 15:32:45 deckconnect08 node[660]: adding new device: /dev/hidraw0
Apr 20 15:32:45 deckconnect08 node[660]: existing = []
Apr 20 15:32:45 deckconnect08 node[660]: Open "/dev/hidraw0" failed: TypeError: cannot open device with path /dev/hidraw0
Apr 20 15:32:45 deckconnect08 node[660]: adding new device: /dev/hidraw1
Apr 20 15:32:45 deckconnect08 node[660]: existing = []
Apr 20 15:32:45 deckconnect08 node[660]: Open "/dev/hidraw1" failed: TypeError: cannot open device with path /dev/hidraw1
Apr 20 15:32:45 deckconnect08 node[660]: adding new device: /dev/hidraw2
Apr 20 15:32:45 deckconnect08 node[660]: existing = []
Apr 20 15:32:45 deckconnect08 node[660]: Open "/dev/hidraw2" failed: TypeError: cannot open device with path /dev/hidraw2
Apr 20 15:32:45 deckconnect08 node[660]: Connected to Companion: CompanionVersion=2.2.0+4003-493404d6 ApiVersion=1.1.0

I did force satellite to a different hid backend to get better performance, it looks like you might need to update your udev rules?
Make /etc/udev/rules.d/50-companion.rules or look for a similar name contain https://github.com/bitfocus/companion-satellite/blob/master/pi-image/50-satellite.rules

Good catch.... that was it.   Thanks!!!

Whoops.... Original issue still exists.
Apr 20 15:54:34 deckconnect11 systemd[1]: Started companionsatellite.service.
Apr 20 15:54:37 deckconnect11 node[1690]: Starting
Apr 20 15:54:37 deckconnect11 node[1690]: adding new device: /dev/hidraw0
Apr 20 15:54:37 deckconnect11 node[1690]: existing = []
Apr 20 15:54:38 deckconnect11 node[1690]: Registering key events for CL29K1A06374
Apr 20 15:54:38 deckconnect11 node[1690]: Connecting to <IP_ADDRESS>:16622
Apr 20 15:54:38 deckconnect11 node[1690]: Connected
Apr 20 15:54:38 deckconnect11 node[1690]: connected
Apr 20 15:54:38 deckconnect11 node[1690]: registerAll [ 'CL29K1A06374' ]
Apr 20 15:54:38 deckconnect11 node[1690]: Connected to Companion: CompanionVersion=2.2.0+4003-493404d6 ApiVersion=1.1.0
Apr 20 15:54:38 deckconnect11 node[1690]: Add device failed: {"ERROR":true,"MESSAGE":"Device exists elsewhere"}
Apr 20 15:55:52 deckconnect11 node[1690]: Exiting
Apr 20 15:55:52 deckconnect11 systemd[1]: Stopping companionsatellite.service...
Apr 20 15:55:52 deckconnect11 systemd[1]: companionsatellite.service: Main process exited, code=exited, status=130/n/a
Apr 20 15:55:52 deckconnect11 systemd[1]: companionsatellite.service: Failed with result 'exit-code'.
Apr 20 15:55:52 deckconnect11 systemd[1]: Stopped companionsatellite.service.
Apr 20 15:55:52 deckconnect11 systemd[1]: Started companionsatellite.service.
Apr 20 15:55:54 deckconnect11 node[1798]: Starting
Apr 20 15:55:54 deckconnect11 node[1798]: adding new device: /dev/hidraw0
Apr 20 15:55:54 deckconnect11 node[1798]: existing = []
Apr 20 15:55:54 deckconnect11 node[1798]: Registering key events for CL29K1A06374
Apr 20 15:55:55 deckconnect11 node[1798]: Connecting to <IP_ADDRESS>:16622
Apr 20 15:55:56 deckconnect11 node[1798]: Connected
Apr 20 15:55:56 deckconnect11 node[1798]: connected
Apr 20 15:55:56 deckconnect11 node[1798]: registerAll [ 'CL29K1A06374' ]
Apr 20 15:55:56 deckconnect11 node[1798]: Connected to Companion: CompanionVersion=2.2.0+4003-493404d6 ApiVersion=1.1.0
Apr 20 15:55:56 deckconnect11 node[1798]: Add device failed: {"ERROR":true,"MESSAGE":"Device exists elsewhere"}

@alatteri I can see from that log that your companion needs updating. Perhaps it is on the master branch and needs switching to the beta branch?

Just noticed I had git pulled  Companion master instead of beta branch. Now that I have the beta Companion running, I'll test again.

Ok... re-opening this.  Unfortunately, still happening.  Less, but still.
I don't know why there was a ping timeout, the Satellite is on the same hardwired network at Companion server.
Apr 21 18:07:23 deckconnect08 node[659]: ping timeout
Apr 21 18:07:23 deckconnect08 node[659]: Connection closed
Apr 21 18:07:23 deckconnect08 node[659]: disconnected
Apr 21 18:07:24 deckconnect08 node[659]: Trying reconnect
Apr 21 18:07:24 deckconnect08 node[659]: Connecting to <IP_ADDRESS>:16622
Apr 21 18:07:25 deckconnect08 node[659]: Connected
Apr 21 18:07:25 deckconnect08 node[659]: connected
Apr 21 18:07:25 deckconnect08 node[659]: registerAll [ 'AL50H1C01830', 'AL15I1C08292', 'AL50H1C03600' ]
Apr 21 18:07:25 deckconnect08 node[659]: Connected to Companion: CompanionVersion=2.2.0+4003-beta-493404d6 ApiVersion=1.1.0
Apr 21 18:07:25 deckconnect08 node[659]: Add device failed: {"ERROR":true,"MESSAGE":"Device exists elsewhere"}
Apr 21 18:07:25 deckconnect08 node[659]: message repeated 2 times: [ Add device failed: {"ERROR":true,"MESSAGE":"Device exists elsewhere"}]

The log line CompanionVersion=2.2.0+4003-beta-493404d6 ApiVersion=1.1.0 is suggesting that while you do have a beta build, it is an old beta build.
The ApiVersion needs to be 1.2.0 for half of the fix to work

Hi Julian.... I'm doing this... are there other git steps to get the proper beta?
git checkout -b beta
git pull

Try yarn update
Perhaps git is complaining about of conflict or something?

oh actually, I think you will have create a branch called beta based off your local master branch.
a git branch -u origin/beta might fix that, followed by yarn update

I'll have to test over the weekend, as that instance is currently in use.

Still not working properly.
Apr 25 12:00:00 deckconnect08 systemd[1]: Started companionsatellite.service.
Apr 25 12:00:01 deckconnect08 node[32275]: Starting
Apr 25 12:00:01 deckconnect08 node[32275]: adding new device: /dev/hidraw0
Apr 25 12:00:01 deckconnect08 node[32275]: existing = []
Apr 25 12:00:01 deckconnect08 node[32275]: Registering key events for AL50H1C01830
Apr 25 12:00:01 deckconnect08 node[32275]: adding new device: /dev/hidraw1
Apr 25 12:00:01 deckconnect08 node[32275]: existing = ["AL50H1C01830"]
Apr 25 12:00:01 deckconnect08 node[32275]: Registering key events for AL15I1C08292
Apr 25 12:00:01 deckconnect08 node[32275]: adding new device: /dev/hidraw2
Apr 25 12:00:01 deckconnect08 node[32275]: existing = ["AL50H1C01830","AL15I1C08292"]
Apr 25 12:00:01 deckconnect08 node[32275]: Registering key events for AL50H1C03600
Apr 25 12:00:01 deckconnect08 node[32275]: Connecting to <IP_ADDRESS>:16622
Apr 25 12:00:01 deckconnect08 node[32275]: Connected
Apr 25 12:00:01 deckconnect08 node[32275]: connected
Apr 25 12:00:01 deckconnect08 node[32275]: registerAll [ 'AL50H1C01830', 'AL15I1C08292', 'AL50H1C03600' ]
Apr 25 12:00:01 deckconnect08 node[32275]: Connected to Companion: CompanionVersion=2.2.0+4107-beta-5c59551f ApiVersion=1.2.0
Apr 25 12:00:01 deckconnect08 node[32275]: Add device failed: {"ERROR":true,"DEVICEID":"AL50H1C01830","MESSAGE":"Device exists elsewhere"}
Apr 25 12:00:01 deckconnect08 node[32275]: Add device failed: {"ERROR":true,"DEVICEID":"AL15I1C08292","MESSAGE":"Device exists elsewhere"}
Apr 25 12:00:01 deckconnect08 node[32275]: Add device failed: {"ERROR":true,"DEVICEID":"AL50H1C03600","MESSAGE":"Device exists elsewhere"}
Apr 25 12:00:02 deckconnect08 node[32275]: Add device failed: {"ERROR":true,"DEVICEID":"AL50H1C01830","MESSAGE":"Device exists elsewhere"}
Apr 25 12:00:02 deckconnect08 node[32275]: Add device failed: {"ERROR":true,"DEVICEID":"AL15I1C08292","MESSAGE":"Device exists elsewhere"}
Apr 25 12:00:02 deckconnect08 node[32275]: Add device failed: {"ERROR":true,"DEVICEID":"AL50H1C03600","MESSAGE":"Device exists elsewhere"}
Apr 25 12:00:03 deckconnect08 node[32275]: Add device failed: {"ERROR":true,"DEVICEID":"AL50H1C01830","MESSAGE":"Device exists elsewhere"}
Apr 25 12:00:03 deckconnect08 node[32275]: Add device failed: {"ERROR":true,"DEVICEID":"AL15I1C08292","MESSAGE":"Device exists elsewhere"}
Apr 25 12:00:03 deckconnect08 node[32275]: Add device failed: {"ERROR":true,"DEVICEID":"AL50H1C03600","MESSAGE":"Device exists elsewhere"}

hasn't happened again in awhile, so closing.


feat: connect state signals to C# events using Callable.From

This merge request introduces the ability to connect Godot state signals to C# events using Callable.From, providing a more idiomatic way to handle signals in C#.
Changes:

Updated the StateChartState constructor to connect signals to event handlers using Callable.From.
Added methods to handle signals with the correct argument types:

OnStateEntered
OnStateExited
OnEventReceived
OnStateProcessing
OnStatePhysicsProcessing
OnStateStepped
OnStateInput
OnStateUnhandledInput
OnTransitionPending

Ensured each method raises the corresponding C# event with the correct parameters.

Motivation:
Connecting signals using Callable.From aligns with the recommended practices for handling signals in Godot using C#. This change improves type safety and readability of the code.
~ Co Created With Chat GPT

Thank you very much for submitting this PR! While I agree on the intention of this, I think this approach has a few drawbacks.

The state charts library checks whether or not some signals are connected to improve performance. E.g. if you don't connect something to state_processing or state_physics_processing then the library will automatically disable the node's processing and thus avoid incurring runtime cost for something that isn't used. With the implementation given here, this optimization would no longer work.
C# events do not auto-disconnect themselves (except for events marked with [Signal] and even there it doesn't seem to work reliably). I find this to be a major hindrance in game development where actors can appear and disappear at any time. On the other hand, any connection made with Godot's Connect will be automatically removed if either the source or the destination are freed. This means when people use the new  events introduced by this change, they will have to manually disconnect their receivers when they are freed. This is an extra required step and will likely raise quite a bit of support tickets here.

I think a better approach would be to provide a type-safe Connect variant.  I have created a branch lining out the idea (check https://github.com/derkork/godot-statecharts/compare/main...csharp-type-safe-signals). I think the external API is quite nice, you can do a:

my_state.StateEntered.Connect(OnStateEntered);

// ...

void OnStateEntered() {
}

though internally it's a lot to write in the wrapper classes. Then again, it's a one and done.

I had a few chats with some people who know a lot more about the state of C# in Godot and me and they told me that my implementation approach didn't help with the signal disconnect issue at all. Since this is the case we can actually use events, so at least the usage mode is consistent with what the rest of Godot does. So i changed the implementation to something along these lines:
        public event Action<float> StateProcessing
        {
            add => Wrapped.Connect(SignalName.StateProcessing, Callable.From(value));
            remove => Wrapped.Disconnect(SignalName.StateProcessing, Callable.From(value));
        }

This way we get the += / -= syntax people are used to but still only subscribe to signals when actually needed.


HIGH: Verify sending UnreadMessageSummary to unvalidated accounts didn't break notification frequency

Problem:
I see this is merged, which is great!  However, testing is a little weird.  I have a test room where I mention a new email, and it does get the invite, which is great.  But for some reason that mention was sent out to everyone else in the room instantly — I would expect it to wait for an hourly update.  Additionally, subsequent messages seem to send instantly to everyone. Are you sure you didn’t break something causing messages to email out instantly to all members, or maybe all unvalidated users?  I've tested:
Testing steps:

Alice creates a workspace
Alice creates a workspace room named #test
Alice mentions<EMAIL_ADDRESS>in #test
Alice posts "Hello Bob!"
Alice mentions<EMAIL_ADDRESS>in #test
Alice posts "Hello Cathy!"

Expected result:

Bob immediately gets a summary with:

Alice invited<EMAIL_ADDRESS>

Cathy immediately gets a summary email with:

Alice invited<EMAIL_ADDRESS>

An hour later Bob should get a summary email with:

Hello Bob!
Alice invited<EMAIL_ADDRESS>Hello Cathy!

An hour later Cathy should get a summary email with:

Hello Cathy!

Actual result:

Bob immediately gets a summary with:

Alice invited<EMAIL_ADDRESS>

Bob immediately gets a summary with: (this is wrong; it should come an hour later)

Hi Bob!

Cathy immediately gets a summary email with:

Alice invited<EMAIL_ADDRESS>

Bob also immediately gets a summary email with: (this is wrong; it should come an hour later)

Alice invited<EMAIL_ADDRESS>

Bob immediately gets a summary email with: (this is wrong; it should come an hour later)

Hi Cathy!

Cathy immediately gets a summary email with: (this is wrong; it should come an hour later)

Hi Cathy!

Solution:
Can you do some testing to make sure that the summary notification frequency of newly invited/mentioned unvalidated users is being honored?

PR up


App home just shows loading icon

Upon entering the app or navigating home using the sidebar house icon, all I get is a throbber.
There's nothing in logs and I'm not actually sure what should be on there.
If you need me to investigate further, please let me know what to look into.
I'm hosting on docker through traefik 2.

Can you see in the api logs if it is loading tasks?

I have tested this further - so far as I understand it, the home page only shows tasks with due date.
I didn't have any prior, so the page was empty.
Upon adding some, they've shown on there without any issues.
So the 'issue' is only visual, i.e. it may be better to show that there are no tasks to display.
2023-02-03T17:48:58.647549203Z: WEB 	▶ <IP_ADDRESS>  GET 200 /api/v1/tasks/all?page=0 17.679032ms - Dart/2.18 (dart:io)
2023-02-03T17:48:58.498226149Z: WEB 	▶ <IP_ADDRESS>  GET 200 /api/v1/tasks/all 15.475018ms - Dart/2.18 (dart:io)
2023-02-03T17:48:58.483609027Z: WEB 	▶ <IP_ADDRESS>  GET 200 /api/v1/tasks/all?&sort_by%5B%5D=due_date&sort_by%5B%5D=id&order_by=asc&filter_by%5B%5D=done&filter_by%5B%5D=due_date&filter_value%5B%5D=false&filter_value%5B%5D=0001-01-02T00:00:00.000Z&filter_comparator%5B%5D=equals&filter_comparator%5B%5D=greater&filter_concat=and 11.58656ms - Dart/2.18 (dart:io)


Update request: Netbox 3.7.4 → 4.0

Package name: Netbox
Latest released version: 3.7.3
Current version on the unstable channel: 3.7.4
Current version on the stable/release channel: 3.6.9
[ X ] Checked the nixpkgs pull requests

there is an pull request for netbox 3.7.8 but not 4.0.0 https://github.com/NixOS/nixpkgs/pull/309483

Notify maintainers
@minijackson @n0emis @raitobezarius

Hey, I've seen this upgrade, but I think it' be best to wait until after NixOS' 24.05 branch-off, so that users don't have to upgrade through 2 breaking versions at once. The branch-off should happen on 2024-05-22

With with some (hacky) overlays I have managed to make netbox 4.0 run. Basically, here are the changes I made.
Django 5 and pytz

I'm overriding django to django 5.0 globally, as I'm trying to use GeneratedField. This further requires the following changes

django-filter: Add pytz to nativeCheckInputs.
django-timezone-field: Remove pytz from propagatedBuildInputs.
djangorestframework: Remove pytz from .propagatedBuildInputs. Add inflection to nativeCheckInputs.

Updated packages

django-mptt: Updated to 0.16, use hatchling build system and add model-bakery.
strawberry-graphql
asgiref

New packages

django-htmx
django-choices-field
strawberry-graphql-django

Netbox itrself

Add strawberry-graphql-django, django-htmx and nh3.

Not sure if I have enough time to contribute all of them, but if possible I will give it a try.

Maybe we can get this ball rolling again :)

Sooo, this is indeed going to be a big one. I've managed to get the NetBox test to be successful locally. Here's what needs to be done in nixpkgs:

[ ] Separate PR for upgrading asgiref from 3.7.2 -> 3.8.1, as it has a number of dependents
[ ] Separate PR for upgrading NetBox that basically does what @KiruyaMomochi explained (thanks!)

In my local branch, I have also moved NetBox to pkgs/by-name, and removed the part of the patch that allowed configuring UNIX sockets for Redis, since it was upstreamed, so if you don't mid I'll open those 2 PRs soon™.

Thanks @minijackson! I was just about to start working on this myself :D If you want you can ping me on the PRs (I can only review then)

Done in #331750!

I'm using the new package and experience some problems. I'm not sure if this is an issue in my own setup. I can't set relations on objects e.g. a site for a device, does this work for you?


Added log ls

Based on #73.
Implemented after seeing https://github.com/ipfs/go-ipfs/pull/2632.
I do not like the way that it is spit out. We should change this.

LGTM


Replication consistency issues

Steps to reproduce
See https://community.neon.tech/t/are-read-replicas-available-in-tech-preview/218/5
Expected result
No consistency issues
Actual result
consistency issues
Environment
Logs, links

hints
We might be seeing issues because of our manual XLOG_CHECKPOINT_SHUTDOWN record.
On hot standby nodes, we might instead need to use XLOG_CHECKPOINT_ONLINE records; because XLOG_CHECKPOINT_SHUTDOWN means there are no running transactions, even if the primary has running transactions. I'm not sure about that, though - it wouldn't explain the 2x larger count(*).

closed with #4579


ToView for entity types with owned types creates invalid migrations

See https://github.com/aspnet/EntityFrameworkCore/issues/18298#issuecomment-542804007

I'll take another look to see whether anything else is missing.

I couldn't find any other issues through manual testing.


Fix typo in README.md

Summary of the Pull Request
References and Relevant Issues
Detailed Description of the Pull Request / Additional comments
Validation Steps Performed
PR Checklist

[ ] Closes #xxx
[ ] Tests added/passed
[ ] Documentation updated

If checked, please file a pull request on our docs repo and link it here: #xxx

[ ] Schema updated (if necessary)
[x] Typo fixed

Please fix this upstream. Everything in oss/ is ingested from other repos, verbatim. See /dup #10659 #16162 #16176 et al.


LoRA on all linear transformer block layers

In general, it would be useful for the LoRa tuner and linear_to_lora_layers in mlx_lm.tuner.utils in particular to allow more control over how LoRa is applied (the ability to specify the rank and alpha, for example).  This PR is for the basic ability to indicate via CLI option (-l/--lora-all-linear) that LoRa is applied to all linear transformer block layers (adding the application of LoRa to k_proj and o_proj).  Currently, it only does this for the following models:

llama
mistral
mixtral
gemma
phi
plamo
qwen2
stablelm
starcoder2

This is mainly motivated by the finding of authors of the QLoRa paper

that the most critical LoRA hyperparameter is how many LoRA adapters are used in total and that LoRA on all linear transformer block layers are required to match full finetuning performance.

I think a more sustainable way to do this is the following:

Have a field in the Yaml which gives the layers keys to apply LoRA to
Make this a yaml config parameter only. So you can't do it with the CLI
Change the LoRA util to read from that

I think a more sustainable way to do this is the following:

Have a field in the Yaml which gives the layers keys to apply LoRA to
Make this a yaml config parameter only. So you can't do it with the CLI
Change the LoRA util to read from that

Wdyt?

I agree 100%.  I'll update the PR accordingly.  I'd also like to add the ability to configure LoRA alpha and rank via config parameters as well.  Should that be broken out into a separate PR?

I'd also like to add the ability to configure LoRA alpha and rank via config parameters as well. Should that be broken out into a separate PR?

Great! I think either is fine. Up to you

Ok.  I have incorporated that also into this PR


meson_init.sh not working

On a clean checkout running meson_init.sh does not complete correctly and returns with the following error (this is before trying to build and run any specific test)
sw/device/silicon_creator/rom_ext/meson.build:167:6: \
ERROR: Program '/projects/goo_opentitan/opentitan/build-out/sw/host/rom_ext_signer' \
not found or not executable.

Is there a soultion for this? I currently have to work around it by removing references to rom_ext_signer from the following:
sw/device/tests/meson.build
sw/device/silicon_owner/bare_metal/meson.build
sw/device/silicon_creator/rom_ext/meson.build
I appreciate that moving forward Bazel will be used instead but I believe that Meson is still used in the nightly regression so I assume we need it to be working at least for now and it is still the system I use to build and run tests during development. As an aside, is there an example of running a top level test using only bazel (any simple smoketest as an example)?
On an email with @timothytrippel it was suggested that there would be no need to use bazel with the meson build but I did notice in the file below it explicitly references a bazel generated file for the rom_ext_signer, I can generate this with bazel but meson does not pick it up.
sw/host/meson.build
line 22
command: ['cp', meson.project_source_root() / 'bazel-bin/sw/host/rom_ext_image_tools/signer/rom_ext_signer', '@OUTPUT@'],

On a clean checkout running meson_init.sh does not complete correctly and returns with the following error (this is before trying to build and run any specific test)

Hmmm, that's odd, I just pulled from master and it seems to be working on my machine.

sw/device/silicon_creator/rom_ext/meson.build:167:6: \
ERROR: Program '/projects/goo_opentitan/opentitan/build-out/sw/host/rom_ext_signer' \
not found or not executable.

Is there a soultion for this? I currently have to work around it by removing references to rom_ext_signer from the following: sw/device/tests/meson.build sw/device/silicon_owner/bare_metal/meson.build sw/device/silicon_creator/rom_ext/meson.build
I appreciate that moving forward Bazel will be used instead but I believe that Meson is still used in the nightly regression so I assume we need it to be working at least for now and it is still the system I use to build and run tests during development. As an aside, is there an example of running a top level test using only bazel (any simple smoketest as an example)?

Normally, we would triage and solve this issue, but I'm not sure it's worth the time in this case, since once #12761 merges (hopefully tomorrow), the nightly regressions will use Bazel only.
There are several examples I will share with you offline, but to also provide an example here, you can use the bazel build //<bazel package>:<bazel target> command to build any SW target. For example, to build the UART smoketest for the DV sim platform, you can use:
bazel build //sw/device/tests:sim_dv_uart_smoketest
Note: there was a slight cleanup in #12761, so after it merges you will use bazel build //sw/device/tests:uart_smoketest_sim_dv (the device/platform string will come after the test name)
Alternatively, you could also use: bazel build --define DISABLE_VERILATOR_BUILD=true --test_tag_filters=dv //sw/device/tests:uart_smoketest (but that is a little more verbose).

On an email with @timothytrippel it was suggested that there would be no need to use bazel with the meson build but I did notice in the file below it explicitly references a bazel generated file for the rom_ext_signer, I can generate this with bazel but meson does not pick it up. sw/host/meson.build line 22

Oh you are right! I forgot about this hack we added to bridge the gap until we delete meson (hopefully next week, we are finishing one more task).

command: ['cp', meson.project_source_root() / 'bazel-bin/sw/host/rom_ext_image_tools/signer/rom_ext_signer', '@OUTPUT@'],

Sorry for any inconvenience on this, and please let me know if you are having any Bazel issues. I'm going to close this issue, and will reopen it if #12761 does not get merged by tomorrow.


[IngestionClient] Delete realtime ingestion client

Purpose
Delete realtime ingestion client
Pull Request Type
What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[x] Other... Please describe: Removing unused project "Realtime Ingestion Client"

@HenryvanderVegte you might want to delete the references to realtime transcription from the setup guides:
https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/master/samples/ingestion/ingestion-client/README.md
https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/master/samples/ingestion/ingestion-client/Setup/guide.md


Crash during downloading large pictures

Sometimes, Andstatus crashes when it tries to download pictures like this one. I think it's necessary to implement an option (checked by default) which allows to download resized pictures to a smaller size.
The same uploading: if we use 3G, it should be better to upload a picture in a smaller size than if we do it using WiFi or just for making it faster.
Maybe sync options should be a good place to add these configurations.

It still happens and I think it should be implemented, maybe Twidere could be a good rely.

I will add some switch to choose smaller size images from a server (if a server pprovides such an option). E.g. downloading thumbnail...
Meanwhile please try these ways:

Turn off "Sync while using application" option.
Turn on "Log network level messages" and "Log everything to file". Then send me recent logs after a crash.

It's already disabled
I'ts already enabled

I'll send you the file to your email!

I could reproduce the bug by showing the below mentioned notice:
@andstatus 14.0. Now on 14.1 it crashes for instance when displaying my post with image: https://quitter.se/notice/3289953 Will logcat later
netmackan

I'm sorry to report that even with the upgrade to 14.2 (from f-droid) I still have the crash on my post with image (https://quitter.se/notice/3289953). I'll try to get the stacktrace.

Mm bad to know :(
Btw, wrong image

This fix was done after v.14.2. It may be tested only compiling the code from this repository. (See this instruction: https://github.com/andstatus/andstatus/wiki/Building-AndStatus-from-Source-code-on-your-Android-device-using-AIDE-application
Will be released in v.14.3

Oh sorry, that explains it :)
I don't use Google Play and in addition AIDE seems to be a non-free software proprietary app.  I'll try to build from source the conventional way instead.

I have now built with the latest on master and it works great! The image is displayed and everything. Big thanks!


APP crash bug:call body() method 2 times will  result in app crash

just like this，and app will crash
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.main_activity);
    new Thread() {
        @Override
        public void run(){
            HttpRequest response = HttpRequest.get("http://www.google.com");
            String t1=response.body();
            String t2=response.body();  //call body() 2 times
        }
    }.start();
}

Yes, this is happening to me also, I had to store the body() result in a var to be able to access it multiple times. Had me looking for the problem some hours.

@ueinohakono @kaoz70 can you provide the stack trace you are seeing? Thanks.

The body will open the stream to read all content each time you call it, the first call will close the input stream associated with the connection. So, the issue came out when you do the 2nd call. I think this can be avoid by handle by in the code like cache the response body.
The stack trace:
Exception in thread "main" com.github.kevinsawicki.http.HttpRequest$HttpRequestException: java.io.IOException: stream is closed
at com.github.kevinsawicki.http.HttpRequest$Operation.call(HttpRequest.java:689)
at com.github.kevinsawicki.http.HttpRequest.copy(HttpRequest.java:2573)
at com.github.kevinsawicki.http.HttpRequest.body(HttpRequest.java:1724)
at com.github.kevinsawicki.http.HttpRequest.body(HttpRequest.java:1739)
at com.foxty.http.HttpRequestTest.main(HttpRequestTest.java:11)
Caused by: java.io.IOException: stream is closed
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.ensureOpen(HttpURLConnection.java:3308)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3333)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.github.kevinsawicki.http.HttpRequest$8.run(HttpRequest.java:2579)
at com.github.kevinsawicki.http.HttpRequest$8.run(HttpRequest.java:2573)
at com.github.kevinsawicki.http.HttpRequest$Operation.call(HttpRequest.java:683)
... 4 more


Question: Custom template to create a multiweek plan

Awesome repo.  Thank you very much for all your efforts.
Can a-b-c be used to create a multiple week plan does not rely on a specific date.  For example, a university prof might have a 12 week plan semester (labeled weeks 1 thru 12) spelling out homework due.  Or this marathon training plan...
http://www.dailygarnish.com/wp-content/uploads/2011/01/Marathon-Training-Plan.jpg
Thank you

For a use case like this I'd recommend just building your own week view component rather than trying to hack something in with a custom template as the UX deviates too much.


InteractiveViewer sometimes treats Scale as Pan

Steps to reproduce

Run the code given below.
Attempt many different scale gestures.
Eventually, one may get stuck as a pan. You'll have two fingers on the screen, but no amount of pinching them together/apart will cause a zoom. Lifting one or more fingers will reset it.

import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Demo',
      home: Scaffold(
        appBar: AppBar(title: Text('Demo')),
        body: Center(
          child: ClipRect(
            child: InteractiveViewer(
              child: Container(
                color: Colors.brown,
                child: Image.network(
                  'https://1001freedownloads.s3.amazonaws.com/vector/thumb/116435/go_board_09x09.png',
                ),
              ),
            ),
          ),
        ),
      ),
    );
  }
}

Cause
_onScaleUpdate always receives a scale of 1.0 during this stuck gesture.  That would make me think that it's a bug with GestureDetector, but I was unable to reproduce this with a lone GestureDetector.  There must be something with InteractiveViewer's widget tree or configuration that causes the bug to show up.
Moved to new issue from some discussion in https://github.com/flutter/flutter/issues/63006.

Reproducible on the latest master 1.21.0-10.0.pre.114.

Minimal reproducible code sample
import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Demo',
      home: Scaffold(
        appBar: AppBar(title: Text('Demo')),
        body: Center(
          child: InteractiveViewer(
            child: Container(
              color: Colors.brown,
              child: Image.network(
                'https://1001freedownloads.s3.amazonaws.com/vector/thumb/116435/go_board_09x09.png',
              ),
            ),
          ),
        ),
      ),
    );
  }
}

flutter doctor -v
[√] Flutter (Channel master, 1.21.0-10.0.pre.135, on Microsoft Windows [Version 10.0.19041.450], locale en-US)
    • Flutter version 1.21.0-10.0.pre.135 at C:\Development\flutter_master
    • Framework revision 654022aecb (6 hours ago), 2020-08-18 03:01:02 +0300
    • Engine revision b9243713a0
    • Dart version 2.10.0 (build 2.10.0-36.0.dev)

[√] Android toolchain - develop for Android devices (Android SDK version 30.0.1)
    • Android SDK at C:\Users\marku\AppData\Local\Android\sdk
    • Platform android-30, build-tools 30.0.1
    • Java binary at: C:\Users\marku\AppData\Local\JetBrains\Toolbox\apps\AndroidStudio\ch-0\193.6626763\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

[√] Visual Studio - develop for Windows (Visual Studio Community 2019 16.6.5)
    • Visual Studio at C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
    • Visual Studio Community 2019 version 16.6.30320.27
    • Windows 10 SDK version 10.0.18362.0

[√] Android Studio (version 4.0)
    • Android Studio at C:\Users\marku\AppData\Local\JetBrains\Toolbox\apps\AndroidStudio\ch-0\193.6626763
    • Flutter plugin version 47.1.2
    • Dart plugin version 193.7361
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] Connected device (5 available)
    • SM G950F (mobile)           • ce12171c51cc001c03 • android-arm64  • Android 9 (API 28)
    • sdk gphone x86 arm (mobile) • emulator-5554      • android-x86    • Android 11 (API 30) (emulator)
    • Windows (desktop)           • windows            • windows-x64    • Microsoft Windows [Version 10.0.19041.450]
    • Web Server (web)            • web-server         • web-javascript • Flutter Tools
    • Chrome (web)                • chrome             • web-javascript • Google Chrome 84.0.4147.125

• No issues found!

I can reproduce this on both emulator and physical Android and iOS.
On both Android and iOS physical devices I can get into this stuck state where only gestures from one corner of the screen are detected. This obviously results in the same scale turning into pan issue. A strange detail is that lifting both fingers off the screen
doesn't reset it. the only thing that seems to work is panning back towards the middle from the working corner.

@markusaksli-nc I've realized that the specific problem with gestures happening near the corners is not related to this issue, it's a mistake in the sample code we're using.  If you wrap your InteractiveViewer in a ClipRect you'll see what the problem is: the gestures are happening outside of the InteractiveViewer.  I've updated the issue's description to use a ClipRect.
To avoid this problem without using a smaller area of the screen with a ClipRect, you'll have to make sure that the InteractiveViewer is taking up the full size of the screen. See my flutter-go code for a full example of positioning a widget within the screen while letting it zoom to the full size of the screen without hitting this problem.

Ah I see! Thanks for clarifying this. I suppose it should be mentioned in the documentation to some degree to make it easier for users to diagnose the resulting behavior (unless it's already mentioned somewhere and I can't find it) . The InteractiveViewer class page has plenty of room for this.
Sorry for causing a bit of a detour here with the faulty sample code. I'll keep this in mind if it pops up again during triage.

No worries!  I'm really trying to get a feel for how people are using InteractiveViewer and the problems they are having, so this is really useful.  I think you're right about the documentation, I'll add something.

This issue seems to have been fixed on the latest master channel as I cannot reproduce it there using the latest sample code at https://github.com/flutter/flutter/issues/63999#issue-680556122.
I don't get stuck when zooming in/out and panning as well. Verified on Samsung Galaxy Tab A7 Lite, Android 13.

Demo
https://user-images.githubusercontent.com/104349824/233035616-d9aac260-3dc7-427a-8cd7-7c760951f352.mov

flutter doctor -v (stable and master)
[✓] Flutter (Channel stable, 3.7.11, on macOS 13.0.1 22A400 darwin-x64, locale en-VN)
    • Flutter version 3.7.11 on channel stable at /Users/huynq/Documents/GitHub/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision f72efea43c (31 hours ago), 2023-04-11 11:57:21 -0700
    • Engine revision 1a65d409c7
    • Dart version 2.19.6
    • DevTools version 2.20.1

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 32.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E222b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)

[✓] IntelliJ IDEA Community Edition (version 2022.1.1)
    • IntelliJ at /Users/huynq/Library/Application Support/JetBrains/Toolbox/apps/IDEA-C/ch-0/221.5591.52/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.77.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.62.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-x64     • macOS 13.0.1 22A400 darwin-x64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 112.0.5615.49

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

[!] Flutter (Channel master, 3.10.0-7.0.pre.44, on macOS 13.0.1 22A400 darwin-x64, locale en-VN)
    • Flutter version 3.10.0-7.0.pre.44 on channel master at /Users/huynq/Documents/GitHub/flutter_master
    ! Warning: `flutter` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    ! Warning: `dart` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision d9392f2e2a (3 hours ago), 2023-04-18 20:31:26 -0400
    • Engine revision 099ed6c62d
    • Dart version 3.1.0 (build 3.1.0-12.0.dev)
    • DevTools version 2.23.1
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 32.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E222b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)

[✓] IntelliJ IDEA Community Edition (version 2022.1.1)
    • IntelliJ at /Users/huynq/Library/Application Support/JetBrains/Toolbox/apps/IDEA-C/ch-0/221.5591.52/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.77.3)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.62.0

[✓] Connected device (3 available)
    • SM T225 (mobile) • R9JT3004VRJ • android-arm64  • Android 13 (API 33)
    • macOS (desktop)  • macos       • darwin-x64     • macOS 13.0.1 22A400 darwin-x64
    • Chrome (web)     • chrome      • web-javascript • Google Chrome 112.0.5615.121

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

Could you please confirm if it's fixed on a PR?


Don't send full transactions with a merkleblock if already known

In merkleblock response, don't include transactions if they were already seen on the peer connection.
This long-standing optimization was removed in https://github.com/bitcoin/bitcoin/pull/7133.  The reason for the removal was that tracking of inventory already known to the peer was expanded and converted to a bloom filter with false-positive rate of 1e-6, whereas previously it was a simple set.
The use of a bloom filter in the already-seen check introduces a 1e-6 chance that a transaction will not be sent along with the merkleblock, even when it has not actually been seen before.  Nevertheless, the optimization should be reinstated for the following reasons:

Consistency with RelayTransaction, which retains the optimization
It is all that's needed for Classic to fully serve XT-style thin blocks (client support is separate)
SPV clients should be prepared to handle a merkleblock which is not accompanied by full transactions for every txid they may need

I have corresponded with the maintainers of the most popular P2P SPV clients, here is a summary:

breadwallet: @voisine reports that breadwallet already detects this condition and disconnects the peer.  This is effective because the next connection will have a freshly randomized bloom filter and a fresh 1e-6 fp rate.  He was going to check that the peer (us) is not banned in this case.
bitcoinj: @schildbach has opened an issue https://github.com/bitcoinj/bitcoinj/issues/1173, which is not considered urgent.

Supporting thin blocks is important, but even a one-in-a-million chance of an SPV client not hearing about a transaction is annoying.  How about doing this:
    if ((!pfrom->nServices & NODE_NETWORK) ||
                !pfrom->filterInventoryKnown.contains(pair.second))
       pfrom->PushMessage(NetMsgType::TX, block.vtx[pair.first]);

... so SPV clients end up getting extra data, but full nodes don't.

Done.  Why didn't I think of that?!

I like the idea but the solution you propose will unfortunately not work for pruned nodes. Is there another way to signal that we want filtered merkleblocks?

Right, you can be NODE_NETWORK and pruned and still wanna do the thin block thing...
This morning I checked, and (at least) bitcoinj doesn't do the BIP130 'sendheaders' thing, so SPV clients don't have the fPreferHeaders flag set. But checking that feels like a hack.
The right way is a new service bit for thin-block-preferring nodes-- and I think the thin blocks patch does have a new service bit, yes?

Yes, there is a new service bit, but still the same problem.
Node supports bloom, but does it support filtered bloom or unfiltered bloom (or both)?
vs.
Node supports thin blocks, but is it bloom thin blocks or xthin blocks (or both)?
There should be sub-service bits :-)

The service bit is for the "extreme" protocol changes.  I think we're back to the SPV node being prepared to notice a txid being sent without the full tx.  All it really needs to do is process merkleblocks from multiple peers.

If this is what you're thinking of @gavinandresen , then I'm happy with it. Not because it's a good solution but because it solves my problem :-)
 if ((!pfrom->nService & NODE_XTHIN) || !pfrom->filterInventoryKnown.contains(pair.second))
       pfrom->PushMessage(NetMsgType::TX, block.vtx[pair.first]);

@dagurval Ok, no remote clients currently available will benefit from this, but we are working on one.
Before this PR is updated, we have another question to answer -- how to best distinguish a classic peer (which has this fix) from peer which lacks it.

With the superior Xthin protocol supported, it will be used by the only client that supported XT-style thinblocks anyway.  Closing.

Thanks dgenr8!


Set a base category after ordering and renaming categories

There's is one enhancement and one bug with category data

The enhancement is that Daru should set the first category as base category after setting the category order.

The bug is that if base category is renamed then a new base category is not substituted in place of it.

Also need a function to remove unused categories.

Could you split this into two different issues - one for the enhancement and one for the bug?

Splited it to three issues:
#190 #189 #188


[BUG] log format: json and timestamp: true disables json format

Hello.
When I define log section in config like this:
log:
  config:    false
  level:     info
  format:    json
  requests:  false
  timestamp: false

it is working as expected and emitting logs in JSON format.
But when I change timestamp: true logs are emmited back in text format.
karma version: v0.94
Thanks

Fix in #3832, thanks!


added ui-ux label governance

What kind of PR is this?:

/kind chore
What this PR does / why we need it:
Commonly used on DeFiCh/wallet, added because it was missing in this repo.

Codecov Report

Merging #93 (291fb72) into main (fe49176) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #93   +/-   ##
=======================================
  Coverage   84.30%   84.30%
=======================================
  Files          14       14
  Lines         172      172
  Branches       27       27
=======================================
  Hits          145      145
  Misses         27       27

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe49176...291fb72. Read the comment docs.


Added a condition-based each function

New in Swift 2, iterates over an array but will only call the callback if a condition (the when param) is met.

Can we make the when params first before callback. The reason for this is to allow for a bigger callback closure body
$.each([1, 2, 3], when: { $0 % 2 == 0 }) {
  // Closure body doing a lot of things
  let x = $0
  let y = x * 10
  print(y)
}

Added your fix. Makes more sense. :+1:


Documentation: get the json result of the research for export.

Hi,
Documentation is a bite confuse on how to get the json response of the search queries  and how to log it in the console ( for test ).
I want to extract in another format (like csv ) the entire result of my research (with some of my facets  and my slider engaged), not just the current page result.
Exemple my querie return 500 results on 50 pages with 10 hitPerPage, i want the 500 results.
Maybe TransformData on the hit widget it's a clue, but honestly i'm little lost with this.
Thx :)

Hi @clementvp,
If you want to do an extract of your full data to a CSV file, you should use our JavaScript client directly: https://github.com/algolia/algoliasearch-client-js
Otherwise, you could also do you own hits widgets like so:
var myHitsWidget = {
  _container: $('#myHits'),
  getConfiguration: function() {
    return {
      hitsPerPage: 500
    }
  },
  init: function() {
    // if you need some jQuery code here, do it
  },
  render: function(args) {
    var out = '';
    args.results.hits.forEach(function(hit) {
      out += '...'; // do whatever you want here, any html structure
    });
    this._container.html(html);
  }
};

search.addWidget(myHitsWidget);

Thank you for your rapid reply, i will try that ;)


Log output no reload duplicated

The log messages for reload are duplicated many times for each change. This is unnecessary and likely a result of the number of feeds. Need to look into a fix.

Fixed when reworking the methods used to on('change')


Add CoroMapper to models friends

Hey! This pull request was tested by me and my actual private project tests, and everything is working perfectly as it should!

Hey! This pull request was tested by me and my actual private project tests, and everything is working perfectly as it should!

Hey! This pull request was tested by me and my actual private project tests, and everything is working perfectly as it should!

Thanks so much.


⚠ Samsung has degraded performance

In 65a055c, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2653 ms

Resolved: Samsung performance has improved in b6e59e2.


Support non-xml responses

Wanted to use invoice.getPDF but code is assuming every response will be XML/hash.

👍  is this going to get merged?


the setStrokeStyle method have a problem.

in setStrokeStyle() when i set the param to 0.3,the alpha of border be changed and the thickness not be set to 0.3
var shape=new createjs.Shape(); shape.x=100; shape.y=100; shape.graphics.setStrokeStyle(1).beginStroke("#ff0000"); shape.graphics.drawRect(0,0,50,50); shape.graphics.setStrokeStyle(0.3).beginStroke("#ff0000"); shape.graphics.drawRect(60,0,50,50); stage.addChild(shape); stage.update();

This is how the canvas deals with subpixel line widths. It cannot draw them any thinner than one pixel, so it alphas them out instead.
Note that you're seeing it as a fuzzy two pixel line because of a quirk of canvas where it considers a whole pixel coordinate to be between screen pixels, so it draws the lines on each side of the coordinate. Positioning lines on half pixels fixes this `graphics.moveTo(100.5, 0).lineTo(100.5, 100);

thanks,i set the value added 0.5 ,it's ok!


[tf] add ChainId

Motivation
Propagate ChainId change to terraform config.
Test Plan
Testing on own cluster.

@bors-libra r+

:pushpin: Commit 7b3c05d has been approved by sherry-x

Added b6d5f6c to correct a typo missed in #5098.

@bors-libra r=sherry-x

:pushpin: Commit b6d5f6c has been approved by sherry-x

:hourglass: Testing commit b6d5f6ce1aff25adbadb3d9ebc60a2292b3707bf with merge 7731a4eda7a158541a8c505eb4f221c805cdecb8...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: sherry-x
Pushing 7731a4eda7a158541a8c505eb4f221c805cdecb8 to master...


[incubator/chartmuseum] Introduce multi-tenancy option

A new version of ChartMuseum was released that introduces multi-tenancy, or multiple orgs/repos for your charts under one server. Update chart image to match latest release.

/ok-to-test

/lgtm
thanks stef!


Importing pixi tilemap does not work

I am trying to use a tilemap in my project, I am installing everything with npm.
Importing anything from pixi.js works just fine, but when I install pixi/loaders and or pixi/tilemap it turns from this: https://i.stack.imgur.com/RYUd2.png, to this: https://i.stack.imgur.com/Yltmb.png. When I hover over the items it says Cannot resolve symbol 'thing', and these are the declarations: https://i.stack.imgur.com/hAxT6.png.
I imported the loader and tilemap with npm i --save @pixi/loaders and npm i --save @pixi/tilemap, and installed pixi with npm i pixi.js.
Also when I try to use the loader like this: import {Loader} from "@pixi/loaders", I get this: https://i.stack.imgur.com/UIfXy.png error.
I don't now what I am doing wrong here.
Thanks in advance.

FYI: I am also unable to import @pixi/tilemap in my Pixi v7 application.
Unfortunately I don't have useful logs and I'm using Pixi via ClojureScript so my tooling configuration is unlikely to help in diagnosing the issue.

@noonian can you please check if issue gone with this branch: https://github.com/steel97/pixi-tilemap/tree/cjs-test
I removed "type": "module" from package.json
you can install this version by changing your package.json:
"@pixi/tilemap": "github:steel97/pixi-tilemap#cjs-test"

I had the same error. When the type is "module", *.js files are interpreted as modules and do not have "require" function available. Either CommonJS files should use cjs extension or the type should be "commonjs" to make it work with commonjs projects. I do not know if the second option breaks "module" projects though, did not test it.
Related:
https://stackoverflow.com/questions/69172129/webpack-uses-esm-part-of-library-although-it-requires-commonjs


local variable issue

Hello @vincentcox ,
I am facing this error after supplying the apk
UnboundLocalError: local variable 'sorted_tosort' referenced before assignment
[WARNING] <IP_ADDRESS> - - [10/Oct/2018 19:35:59] code 404, message File not found

Are you using a build from the release? Or do you use the python command line?
If it's possible, I can check with your APK to debug this issue.

Hi @vincentcox ,
Thanks for prompt reply. I am using from the command line.

Getting the same even with the test-apk.apk
`python3 stacoan.py -p test-apk.apk --disable-server --disable-browser
[INFO] Decompiling app...
[INFO] /mnt/Security/Tools/Android/StaCoAn/src/test-apk_apk/jadx_source_code
[INFO] "/mnt/Security/Tools/Android/StaCoAn/src/jadx/bin/jadx" -d "/mnt/Security/Tools/Android/StaCoAn/src/test-apk_apk/jadx_source_code" "test-apk.apk"
Invalid maximum heap size: -Xmx4g
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[INFO] --------- JADX OUTPUT BELOW ---------
[INFO] jadx return code: 1
[INFO] Decompiling done.
[INFO] Searching trough files
[INFO] Searching done.
[INFO] start generating report
[INFO] progress: 100%
[ERROR] ERROR: Unknown error: local variable 'sorted_tosort' referenced before assignment.
`

I know what happened. JADX could not allocate 4Gigs of memory so it didn't run. The problem tries to continue anyhow and there are 0 results, which causes a for loop to not run somewhere.
Please double check that you are using 64-bit java version. JADX sets max memory heap to 4GB by default. This only possible on 64-bit java.

Is this issue with system architecture, as I am having 32bit system.

Indeed, it's caused by the 32 bit architecture. However, you can try to run:
set DEFAULT_JVM_OPTS="-Xms128M" "-Xmx2g"
But I am not sure if it will work because StaCoAn spawns an own subshell.

Let me know if it works. I will try to make a fix for this if it's not working.

Still getting the same :(

I'll create a fix this weekend. Thanks for reporting this issue, I will let you know when the fix is available.

Hello @vincentcox ,
Thank you for your quick & positive response on this issue.
Waiting for the fix :1234:

Can you try the DEV branch?
git clone -b dev https://github.com/vincentcox/StaCoAn
cd StaCoAn
cd src
python3 stacoan.py -p test-apk.apk --disable-server --disable-browser

Let me know what the output is, If it works for you then I know the correct amount of memory I can allocate for 32 bit systems. Then I can build a fix to the master.

Hi @vincentcox ,
Sorry for the being late in response, still getting the same issue
python3 stacoan.py -p test-apk.apk --disable-server --disable-browser
[INFO] Decompiling app...
[INFO] /mnt/Security/Tools/Android/StaCoAn/src/test-apk_apk/jadx_source_code
[INFO] "/mnt/Security/Tools/Android/StaCoAn/src/jadx/bin/jadx" -d "/mnt/Security/Tools/Android/StaCoAn/src/test-apk_apk/jadx_source_code" "test-apk.apk"
[INFO] 32 bit detected, setting max memory for java to 2G instead of 4G. This might cause problems for JADX.
Invalid maximum heap size: -Xmx4g
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[INFO] --------- JADX OUTPUT BELOW ---------
[INFO] jadx return code: 1
[INFO] Decompiling done.
[INFO] Searching trough files
[INFO] Searching done.
[INFO] start generating report
[INFO] progress: 100%
[ERROR] ERROR: Unknown error: local variable 'sorted_tosort' referenced before assignment.

Can you try the dev branch again?
git clone -b dev https://github.com/vincentcox/StaCoAn
cd StaCoAn
cd src
python3 stacoan.py -p test-apk.apk --disable-server --disable-browser

Normally it should work now because I put the variable into the command. Let me know the output if it still doesn't work, we'll figure it out!

Hello @vincentcox ,
Don't know why its not working on my machine ??? Still getting the same issue check this

I'll download a 32 bit kali machine to figure this out. It will take me some time but I will dive into this! 💪

Great @vincentcox :+1:

Hi @vincentcox ,
Thank you, it Works :+1: :100:
This is the best support I ever got :100:

No problem, thank you for your patience because it took me some time to figure it out.
Soon I will push a global fix for this, but meanwhile you can use this fix to get it working for you.
Have a nice weekend!

Thank you :+1:


Language Server seemingly stuck at analysis stage

The status bar does not change from 695 items left....

Here's the log file:
log.log

I also have the same problem

Ubuntu 18.04 languageServer.0.2.41 have same problem

In at least my case it's not just seemingly stuck but it also takes up a lot of CPU.

@DonJayamanne , what module is opened in your case?

Sorry don't have that info anymore.

Same here on Ubuntu 16.04.
If I open an empty with system Python 2.7.12 it looks ok. Just typing some import lines (os or requests for example) it's also ok.
But as soon as I select a VirtualEnv with Python 3.6 and do the same, the Microsoft.Python.LanguageServer process seems to get stuck in an endless loop and uses 100% cpu.
I guess this started after upgrading to the latest version.

I think I probably have reproducible steps:

Create and activate a new Python 3.7.2 virtualenv (I used pyenv)
Install packages as detailed in this requirements.txt file (pip install -r requirements.txt)
Open vscode and select the interpreter created on step 1 above
Create and save a new file with this one line: import boto3

Sorry for a rather big requirements.txt file and for not being able to nail it down to a few packages at the moment. I'll update this issue if that changes.
If anyone could confirm those steps reproduce the issue, I'd be grateful.
Thanks!

We have a repro for this (but thanks either way!); it's just idna. https://github.com/Microsoft/python-language-server/issues/832#issuecomment-479609316

0.2.42 has been built and pushed to all of the channels (containing #873); it should fix the hangup introduced in 0.2.41.

How do we apply this fix?  Thanks very much!

Reload VS Code, and if that doesn't update it, you can delete the downloaded language server and have the extension download it again. See Microsoft/vscode-python#3977.

@zhen8838 , @BeanNan , @marconfus  - can you please check if the issue is fixed for you?

Yes, Now analysis successful. The analysis speed is now much faster. but I don't know why something can't autocomplete:

I have to import tensorflow.python as tf

Anyway, I am still very grateful for your work. thanks so much.

For me I'm still stuck at "Analyzing in the background, X items left..." where X is unchanging for a couple hours, but it appears that less CPU is being used.  Thanks for the fix so far!

@zhen8838 , this is probably #867
@andrew-weisman , can you enable full logging ("python.analysis.logLevel": "Trace"), copy full log before hanging and attach it to the bug as a file?

I don't think that'd be #867; the new code should still bail when it sees the complicated comprehension that defines __all__ in tensorflow. I'll check tomorrow to be sure.

Thanks for your efforts, now I working with python-language-server so good.

@AlexanderSher @jakebailey Thanks for your efforts.
For me, after upgrade languageServer to 0.2.43, the analysis ends in about 5~7 minutes, but I got some problems like  @zhen8838 https://github.com/Microsoft/python-language-server/issues/869#issuecomment-479729081 .
1 - Launch VS Code with shell command in iTerm2.
1.1 - languageServer:

Analysis stopped after 5 to 7 minutes, but CPU and RAM usage doesn't look good.

1.2 - Intelligent Code Completion
IntelliSense does not work properly,I try:
import tensorflow as tf
import pandas as pd

It seems that IntelliSense is not working for tensorflow, but working for pandas.

My python virtual environment is correct and the code works.

BUT parameter info for tf looks working :(

I don't know why, that's it.
2 - Launch VS Code.app directly.
It seems that code can't find the python virtual environment, and analysis seems not working.

I tried debugging and the code worked fine.

@zhen8838 , @BeanNan , @marconfus - can you please check if the issue is fixed for you?

Yes, the bug is gone now. Thanks!

@jakebailey:

Dang, okay. I'll file a separate issue for that and look into it. The tensorflow __all__ should have been marked as unusable.

Opened as #881 to keep this thread about the analysis getting stuck rather than unrelated issues.

I don't know why it can't Intellisense Class init parameter :

Just FYI it works for me now, i.e., the background analysis doesn't hang forever and finishes in about 30min or so, and the CPU usage is way down overall.  Thanks for all your work!

It still stays with hight CPU load even on small project like this Pyramid example project: https://github.com/Pylons/pyramid_blogr.
Just check it out and install it in a Python 3 virtualenv with pip install -e .
After that when ever i open one of the Python files, this process keeps working on 100% for ever: .vscode/extensions/ms-python.python-2019.3.6352/languageServer.0.2.43/Microsoft.Python.LanguageServer
I often work on bigger projects based on the CMS Plone, there VS Code is currently kind of useless.

@zhen8838 please file a separate issue, as that's unrelated to the analysis getting stuck.
@MrTango The analyzer still needs to inspect dependencies; even if the project is small the imports still need to be inspected. We're still working on more ways to improve performance, so things should continue to improve.

@jakebailey thx for the info. I'm aware of the fact that VS Code needs to analyse the libraries as Pyramid too, but that is the case for pretty much for every project. And Pyramid is not a big library not talking about frameworks like Django & co. Currently as much as i like VS Code in general, I must say for Python development it's not very much useful. Even the basic goto definition of Sumlime test works better and with the anaconda plugin the autocompletion just works. VS Code is heating the room for a long while and still eats up more than 10GB memory after a while. So i gues until the caching and performance issues are really solved, I'll switch back to sublime.

@MrTango

. So i gues until the caching and performance issues are really solved, I'll switch back to sublime.

You can still use VSCode with a work around, just change the following setting in your settings.json to turn off the language server and use Jedi:
"python.jediEnabled": true

Not really, if I do that, jedi is driving the CPU the same way and worse, the autocompletion is blocking even normal snippets. :(

@MrTango - yes, Jedi sometimes has CPU and RAM issues as well. Is your code uses just Pyramid or anything else as well? Is it Plain Python or Anaconda? What Python version is it?

Hi, it's plain Python 3 in this case, the example project i was talking about has just a view dependencies https://github.com/Pylons/pyramid_blogr/blob/master/setup.py
But I'm developing a lot with bigger stack like the CMS Plone or the ERP Odoo.
The Pyramid project I'm currently working on is bit bigger than the example, but even the smal project is slow in VS Code that's what i wanted to show. It might be also simpler to figgure out why on that code base, than on bigger ones. I gues i really need to wait for the caching feature, you talked about for the future.
but thx a lot for your effort, I'll keep an eye on vs code ;)

Since original issues appear to be resolved, closing this one. Opened https://github.com/Microsoft/python-language-server/issues/936 on @MrTango case. Please open separate issues with specific configurations if you are still experiencing problems. Thanks.

The original issue of hanging during "analyzing in background". Same symptoms as originally described, and I'm on server version <IP_ADDRESS>. I think I've pin-pointed which package import is causing it, but unfortunately I can't reveal which one it is. It's a very small package though, so I'm surprised it's causing this problem.

@raaaaaymond , can you try LS 0.2.74 from beta channel?

using LS 0.2.74 and today i got it.

@aaveidt, @raaaaaymond  - when you say 'stuck' do you mean it is actually running (i.e. CPU consumption is visible) or CPU is 0 and it is just the stale message?

@MikhailArkhipov I mean the CPU was running, but nothing happens for hours and hours.
Update @AlexanderSher I never tried languageServer 0.2.74 because I just switched to Jedi for a while (which has its own problems), but recently I upgraded to VSCode 1.34.0, with languageServer <IP_ADDRESS>, and it's working fine for me for now.

Is there a solution to this problem?
It is still happening to me causing VSCode to be laggy.

I'm having this problem with the last few items.
It looks like whoever implemented/modified this recently forgot to hide the message when the process finishes.

The issue is still here.
Any workarounds?

I don't understand why is there 2 processes running.

This issue is old, and the langauge server has had many changes and a version bump since then. Your issue may be different.
Please follow the instructions at this section and file a new issue with logs and some more info, thanks: https://github.com/microsoft/python-language-server/blob/master/TROUBLESHOOTING.md#filing-an-issue

This issue occured again. Language Server always stuck at analysis stage. Please check and provide possible solution as soon as possible.
Environment data

VS Code version: 1.36.0 (Stable Channel)
Extension version (ms-python.python): 2019.6.22090
OS and version: Ubuntu 18.04 LTS
Python version (& distribution if applicable, e.g. Anaconda): 2.7.15
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv

@jakebailey @DonJayamanne @MikhailArkhipov

See my previous comment; we need some code sample, list of requirements, etc, in order to reproduce this and figure out if something's wrong.

@jakebailey As per your instruction I've created a new issue with details, Please help.
Here is the url: https://github.com/microsoft/python-language-server/issues/1291

Hey @kaushall a temporary solution for you:
uninstall IntelliCode and enable jedi on settings.json
"python.jediEnabled": true,

@petrosschilling Thanks, Yes I'm using jedi right now.

I confirm I'm having the same problem again immediately after updating to VSCode 1.36.0. I've posted on @kaushall thread #1291 with more details.

1.Run the Extenesions: Open Extensions Folder command
2.Quit VS Code
3.Delete the folder containing the Python extension (name should have a prefix of ms-python.python)
4.Reinstall Python Extension
This step is work for me!

What  worked for me :

Change from virtualenv to  default python 3 interpreter .
Close and restart VSCODE
Change back to virtualenv.


Change LICENSE

I am stating to believe the GPL/LGPL GNU licenses are too restrictive. I would love to change to either:

MIT license
BSD license

Do I need to consider something?

BSD 3-clause.


ESM imports escape the sandbox & runfiles

The relevent code in node is in resolve.js where there is a realPathSync call
https://github.com/nodejs/node/blob/4c5b96b376aa778cbe651362c29a26e0d4c32ccd/lib/internal/modules/esm/resolve.js#L310
This realPathSycn call does not get the fs monkey paches while the cjs require loader's realPathSync call does. It is unclear why.
This is the underlying reason why we current need --preserve-symlinks-main on by default in js_library so the .mjs entry points don't escape their runfiles.
It is also the reason mocha was observed to escape the sandbox in the repro https://github.com/aspect-build/rules_js/pull/353#issuecomment-1206659854 and likely related to https://github.com/aspect-build/rules_js/issues/347.
This affects .mjs entry points and programs that use esm imports. For example, mocha uses the import() built-in that is affected.

We suspect the underlying reason why the fs patches apply to the commonjs path and not the esm path in node is the fs import style in https://github.com/nodejs/node/blob/4c5b96b376aa778cbe651362c29a26e0d4c32ccd/lib/internal/modules/esm/resolve.js
const {
  realpathSync,
  statSync,
  Stats,
} = require('fs');

while the commonjs import is
const fs = require('fs');

Presumably both modules are evaluated before our fs patches, which are done in a --require from the command line. Our patches would apply only to the commonjs path since it has a reference to the fs module which we monkey patch while the esm path has references to the fs functions it needs directly which are not monkey patched.

Product Planning: still relevant

I tested the changed needed in Node.js to resolve this and it works as expected. The change is just https://github.com/aspect-build/node/commit/6616ddd629f9417e0ac2826910ff9756b01cfe34.

Minimal repro: https://github.com/gregmagolan/repro_rules_js_362

https://nodejs.org/api/module.html#modulesyncbuiltinesmexports
Does this fix the issue?

https://nodejs.org/api/module.html#modulesyncbuiltinesmexports
Does this fix the issue?

It doesn't. This is being called already after the monkey patches are made, https://github.com/aspect-build/rules_js/blob/c95dda4058268f985ac7888e94730928d34ce690/js/private/node-patches/register.js#L50. I don't remember the context anymore however. @jbedard was working on that one.

I think that patching fs prob won't affect ESM loading as it happens in a seperate thread and you should use experimental-loader resolve in that case (or try the fs patch in a file loaded via --experimental-loader)

Thanks for the suggestion @mkg20001 . I'll spend a bit of time today looking into that.

@alexeagle For reference, there were a few attempts to make the ESM loader's fs calls monkey patchable but they were rejected by the node maintainers:
https://github.com/nodejs/node/pull/39711
https://github.com/nodejs/node/pull/39513
Other (possibly) related PRs & issues:
https://github.com/nodejs/node/pull/41076

Try to patch it in the loader thread aswell, by which I mean running the
code directly without globalPreload since the code would run in the right
thread unless each loader gets it's own thread (but you can keep the
globalPreload aswell for commonjs)
If that still won't work, your best bet is a custom resolve function (you
can use a custom load function to intercept and rewrite loading aswell)
https://nodejs.org/api/esm.html#examples
(you can always just call the "next" function with the real path you're
resolving and return thaz when you're just mapping paths)
Greg Magolan @.***> schrieb am Sa., 1. Apr. 2023, 20:48:

@mkg20001 https://github.com/mkg20001 I tested monkey patching the fs
module in a globalPreload hook via --experimental-loaders with no luck.
The require('fs') called in
https://github.com/nodejs/node/blob/4c5b96b376aa778cbe651362c29a26e0d4c32ccd/lib/internal/modules/esm/resolve.js#L30
appears to still happen before the globalPreload.
—
Reply to this email directly, view it on GitHub
https://github.com/aspect-build/rules_js/issues/362#issuecomment-1493073942,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AO2KD5D6K4NJ5DZ4ZFBTW7BZ67ANCNFSM55WWSRYA
.
You are receiving this because you were mentioned.Message ID:
@.***>

The custom resolve function on a loader sounds promising if it works and prevents nodejs forking which would be not ideal if it can be avoided.

@gregmagolan did you have any further luck with the loader approach?
I am currently debugging some sources of sandbox escape with Node.js v18, and this is with the patches to the ESM resolver.js. As you well know, it is tricky business. There are some places that use internalModuleStat rather than fs.statSync, which is not patchable. I noticed in Node v21 this API is now used instead of fs.statSync in the esm loader files, so it appears this problem is likely to get worse rather than better:
https://github.com/nodejs/node/blob/v21.0.0/lib/internal/modules/esm/resolve.js
Luckily, within the context of build and test actions, --experimental_use_hermetic_linux_sandbox seems to be extremely effective at ensuring hermetic JS execution thanks to using hardlinks (so no more symlinks to follow) and chroot (so no walking parent dirs up to the root looking for competing node_modules dirs). Reflecting on this, I do wonder whether aspects of this approach could be emulated to make bazel run of js_binary targets more robust. My main concern here is developers running tools and binaries locally - not deployed binaries, where we should be able to rely on the FS not being polluted, e.g. because they run in docker containers via js_image_layer. Obviously, it is annoying when a developer misses a dep and, rather than manifesting in a "module not found" error, there are subtle bugs due to node loading separate instances of the same module because one was resolved inside runfiles, and one under the bindir (etc.).
I wonder whether this "developer running a js_binary through bazel run" could be handled specifically by a wrapper script crawling the runfiles dir for all the node_module and application files pertinent to the binary (as rules_js would own this code, we can be sure to stay within the runfiles dir), and then creating a mirroring hardlink tree under /tmp with the node_modules and application files, and then launching node within this /tmp hardlink tree. This, like I say, would apply the benefits seen with the hardlink sandbox to bazel run for js_binary. There is clearly some overlap here with js_devserver.
What do you think?

Other things to look at:

Whether the fs guards could better handle Node.js "misbehaviour" here and redirect sandbox escapes back within the sandbox. I think I noticed realPathSync(path) returning path without modification if path is already outside of runfiles. In this case, returning the path inside runfiles that links onto path might be better, as then subsequent fs calls that base themselves on path will fall into the expected pathways.

The CJS loader uses this Module._nodeModulePaths function to get all possible node_modules directories from the current location up to the root...
e.g.

/my/bazel/path/execroot/bin/mybin.runfiles/app/node_modules
/my/bazel/path/execroot/bin/mybin.runfiles/node_modules
/my/bazel/path/execroot/bin/node_modules
/my/bazel/path/execroot/node_modules
/my/bazel/path/node_modules
/my/bazel/node_modules
/my/node_modules
/node_modules

These then feed into downstream operations, some of which use internalModuleStat. This is not good and I'm pretty sure accounts for some of the escapes I'm seeing.
Luckily, it looks like Module._nodeModulePaths is patchable:
https://github.com/ilearnio/module-alias/blob/dev/index.js#L17

I haven't looked further into the loader approach since the spike a while ago.
I came across this proposal however that looks interesting and could open a path for a solution: https://github.com/nodejs/node/issues/52219. Haven't had time yet to look in detail but at face value it sounds promising.

I had a passing thought since we tripped on this again. rules_js maybe could try and use a register from node to rewrite all fs imports it sees (or node:fs) to be it's monkey patched version before node loads the file. Then you could universally patch esm possibly? I don't know, but worth mentioning in case anyone wants to give it ago.
For now, we just disable the sandbox on macOS and leave it on during remote execution with linux.


Allow creating a CuratorTestingServerExtension that starts the testing server in the constructor?

Discussed in https://github.com/kiwiproject/kiwi-test/discussions/238

Originally posted by sleberknight July 14, 2021
Background
The original (i.e. pre-GitHub) implementation of CuratorTestingServerExtension did the following in its constructor:
testingServer = new TestingServer(port);

The above is equivalent to:
testingServer = new TestingServer(port, true);  // here, true means to start the server *right now*

When we re-implemented this in kiwi-test, we chose to do this in the constructor:
testingServer = new TestingServer(port, false);  // here, false means we will NOT start the server now

And then in the beforeAll we called testingServer.start(). Doing it this way is more correct in the way JUnit Jupiter extensions should behave.
However, this caused an issue when testing Dropwizard services using the DropwizardAppExtension, specifically because Dropwizard does not follow typical Jupiter extension conventions. It looks something like this when we're using the Dropwizard extension and the CuratorTestingServerExtension:
@ExtendWith(DropwizardExtensionsSupport.class)
class AppTest {

  private static final DropwizardAppExtension<AppConfig> APP =
          new DropwizardAppExtension<>(App.class, ResourceHelpers.resourceFilePath("config-app-test.yml"));

  private static final int ZOOKEEPER_TEST_SERVER_PORT = 37_456;

  @RegisterExtension
  static final CuratorTestingServerExtension ZOOKEEPER_TEST_SERVER =
          new CuratorTestingServerExtension(ZOOKEEPER_TEST_SERVER_PORT);

  // tests...

}

But when using the kiwiproject extension, the above does not work because there is no way to ensure the Curator TestingServer starts before the Dropwizard extension launches the test application. As a result, the App classes under test can never connect to a the Curator testing Zookeeper server, so we end up mocking things like leader latches inside App by doing something like:
public static class MockLeaderLatchApp extends App {
  @Override
  protected ManagedLeaderLatch buildManagedLeaderLatch(CuratorFramework curator) {
    return mock(ManagedLeaderLatch.class);
  }
}

//  and then...

  private static final DropwizardAppExtension<AppConfig> APP =
          new DropwizardAppExtension<>(MockLeaderLatchApp.class, ResourceHelpers.resourceFilePath("config-app-test.yml"));

This is not ideal, and doesn't test the actual leader latch in the App class.
To allow us to use the Dropwizard extension and the Curator testing extension, we could add a new constructor in the kiwiproject CuratorTestingServerExtension that accepts a second argument that forces the testing server to start immediately, i.e. it would call new TestingServer(port, true).
While this is not the "correct" way to do most Jupiter extensions, we don't have a choice in this case if we want to use it with the (nonstandard) Dropwizard extension mechanism. So we might do something like:
  @RegisterExtension
  static final CuratorTestingServerExtension ZK_TEST_SERVER =
          new CuratorTestingServerExtension(ZK_TEST_SERVER_PORT, TestingServerStart.IMMEDIATE);

where ServerStart is perhaps an enum inside CuratorTestingServerExtension:
public enum TestingServerStart {

  /** Start the Curator TestingServer immediately. */
  IMMEDIATE,

  /** Start the Curator TestingServer when beforeAll is called (this is the normal JUnit Jupiter lifecycle) */
  BEFORE_ALL
}

Again, while this isn't ideal and not the recommended way, we would no longer need to create the MockLeaderLatchApp and mock things out when we're trying to do integration testing.

Note that the solution in #242 doesn't add additional constructors and the enum mentioned in the original discussion in this issue. Instead, after initially trying that route (i.e. using the TestingServerStart enum), it didn't look good, and I changed to use two new static factory methods instead:

newStartingImmediately()
newStartingImmediately(int port)

Doing this avoided a few issues. First, using the enum forced adding multiple new constructors, one that accepted the enum and a second one that accepted the enum and a port in order to cover the various cases. Second, and more importantly, the existing constructors already cover the situation when you want to use the regular JUnit Jupiter extension lifecycle. Having the enum contain a BEFORE_ALL constant is therefore not necessary, and redundant. Therefore, the only useful enum constant is IMMEDIATE, and because having an enum with only one constant is a bit silly, it seemed to make more sense to create the static factory methods to cover the "immediate start" use case.
And the code reads better (to me anyway). For example:
@RegisterExtension
static final CuratorTestingServerExtension ZK_TEST_SERVER =
        CuratorTestingServerExtension.newStartingImmediately();

// or

@RegisterExtension
static final CuratorTestingServerExtension ZK_TEST_SERVER =
        CuratorTestingServerExtension.newStartingImmediately(9090);


Run windows builds on latest adoptium openjdk

@bot-gradle test qf

OK, I've already triggered the following builds for you:

QuickFeedback build


Fix a compile error in BN

Description
This is to fix the compile error reported in https://github.com/apache/incubator-mxnet/issues/10235

CI doesn't have a test for CUDA. It only has CuDNN


[TB] Font Weight issues

Problem/Concern
1.) Currently, the font weights are associated with the secondary Font rather than the primary font.
2.) We need a drop down field that allows the user to pick the available font weight for the header (the Secondary font)
(I am askign Keith to add a SDK variable --headerWeight
3.) Can we update the default line height for small text to 16px?
Proposed Solution

Looking at the code, it appears that the five fontweight selectables are changing the correct properties in the SDK. The issue appears to be how these properties are used. As Lise said, I don't see these changing the appearance for the Primary Font examples.

@lwnoble  - to clarify point 2 in the description: We want the five fontweight selectables to effect the primary font, and a new 6th dropdown that effects header weight? What would this mean for fontweight 2 which is described as For headers and small text that is important.?

waiting on @smithbk to add the secondary fontweight property in the SDK then the above will be a quick addition

The 6th drop down is present and the default line height mentioned is changed.
As for the Font Weight problem:
We can see that the SDK is working and that the css variables are changing correctly. The problem appears to be how they are being used. Looking at the code I don't see them being used.
For example, changing the fontWeight1 dropwdown changes the --fontWeight-1 variable which should effect all body1 text, but it doesn't change. Instead, the variable --body1FontWeight is used, which appears to not be calculated from fontWeight1.
@lwnoble - the question for you: Should changing these fontweight dropdowns directly change the typography styles for the classes? For example, changing fontWeight1, will automatically change the fontweight for body styles 1-3.
I went ahead and created a PR for this change so it is ready to go once you say so
https://github.com/discoverfinancial/a11y-theme-builder/pull/596/files
The logic is below, please confirm if this is the way it should work:
fontweight0 -> caption, cta, small
fontweight1 -> body
fontweight2 -> labels, subtitles, overlines, caption bold, small semibold
fontweight3 -> body bold
fontweight4 -> stat

this code has been merged

Changes are in and appear to have the desired fix. However, we should investigate why the fontweights 0-4 have been changed to unset somewhere along the dev process.

The font weight defaults have been changed to 300,400,600,700,800.
Note the sample design system has not changed.

@lwnoble is ok with closing w/o sample updates.  Closing.


HDDS-5700. Improve LOG message of decommission progress.

https://issues.apache.org/jira/browse/HDDS-5700

I wish we had a better way to list out the Unhealthy / Under replicated containers etc. Feel like we need a basic fsck command for Ozone to be able to debug things like this a bit more easily.
Patch looks mostly OK. I just had one comment about wrapping the lines in "if debug enable" statements.

I wish we had a better way to list out the Unhealthy / Under replicated containers etc. Feel like we need a basic fsck command for Ozone to be able to debug things like this a bit more easily.
Patch looks mostly OK. I just had one comment about wrapping the lines in "if debug enable" statements.

Thanks @sodonnel ， I will address the comments by a new commit.  I also agree we should have a better way to display these kind of information.  The patch is just an emergency fix, for we have decommission issue in our cluster and I need to resolve the issue ASAP.

@ChenSammi when a DN is decommissioned successfully, do you think we can delete all its' replicas by sending replica delete command from SCM, so that we can free up the space on the DN and cleanup the container map of SCM.

I don't think we should remove the containers from the DNs, for two main reasons:

Sometimes people decommission nodes for maintenance, and then bring them back to the cluster, so its better if they still have the data in that case.
If there are some bugs in decommission, then its good we can recommission the node and know its data is still intact.
If the node needs to be wiped its easy to just format the drives or delete the data folders manually later.

However, we should clear the replica references out of SCM. This should happen when the node goes dead, and the dead node handler runs. If that is not the case, then we might have a general problem with dead node handling.

@ChenSammi when a DN is decommissioned successfully, do you think we can delete all its' replicas by sending replica delete command from SCM, so that we can free up the space on the DN and cleanup the container map of SCM.

I don't think we should remove the containers from the DNs, for two main reasons:
1. Sometimes people decommission nodes for maintenance, and then bring them back to the cluster, so its better if they still have the data in that case.

2. If there are some bugs in decommission, then its good we can recommission the node and know its data is still intact.

3. If the node needs to be wiped its easy to just format the drives or delete the data folders manually later.

I see.  It makes sense to keep the replica data.

However, we should clear the replica references out of SCM. This should happen when the node goes dead, and the dead node handler runs. If that is not the case, then we might have a general problem with dead node handling.

The obervation is once the DN is dead, it's replicas are all cleared from SCM.

Thanks @sodonnel  for the code review.


Revert "Add closeness_centrality app and test case of duplicated mode…

Codecov Report

Merging #372 (e925b46) into main (6a98c7e) will decrease coverage by 10.88%.
The diff coverage is 81.81%.

@@             Coverage Diff             @@
##             main     #372       +/-   ##
===========================================
- Coverage   77.36%   66.48%   -10.89%
===========================================
  Files          57       55        -2
  Lines        5302     5051      -251
===========================================
- Hits         4102     3358      -744
- Misses       1200     1693      +493

Impacted Files
Coverage Δ

...ython/graphscope/analytical/app/katz_centrality.py
53.84% <ø> (ø)

python/graphscope/framework/app.py
86.75% <81.81%> (-1.13%)
:arrow_down:

python/graphscope/deploy/kubernetes/utils.py
12.38% <0.00%> (-56.67%)
:arrow_down:

python/graphscope/interactive/query.py
38.46% <0.00%> (-52.02%)
:arrow_down:

python/graphscope/deploy/kubernetes/cluster.py
25.38% <0.00%> (-50.39%)
:arrow_down:

...n/graphscope/deploy/kubernetes/resource_builder.py
30.00% <0.00%> (-21.90%)
:arrow_down:

python/graphscope/client/rpc.py
66.12% <0.00%> (-19.06%)
:arrow_down:

python/graphscope/analytical/app/triangles.py
85.71% <0.00%> (-14.29%)
:arrow_down:

python/graphscope/client/session.py
62.62% <0.00%> (-10.67%)
:arrow_down:

python/graphscope/framework/graph.py
84.11% <0.00%> (-8.12%)
:arrow_down:

... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 336ec94...e925b46. Read the comment docs.

close since the ci is fix.


Refactor ockam portal ... commands into ockam tcp-... commands

We currently have:
ockam portal create tcp-inlet ..
ockam portal create tcp-outlet ...

We want to change these to:
ockam tcp-inlet create ...
ockam tcp-outlet create ...

Without changing their behavior.
These commands are defined here:
https://github.com/build-trust/ockam/blob/develop/implementations/rust/ockam/ockam_command/src/portal/create.rs

Related: #3076 #3078

We love helping new contributors!
If you have questions or need help as you work on your first Ockam contribution, please leave a comment on this discussion.
If you're looking for other issues to contribute to, checkout issues labeled -  https://github.com/build-trust/ockam/labels/good first issue or https://github.com/build-trust/ockam/labels/help wanted

@spacekookie what are your thoughts on the above?

Hi I'd be interested in trying to do this. Would this require reworking the portal subcommand (by removing it?) would this mean creating tcp-inlet and tcp-outlet folders in the ockam_command/src folder? as the portal subcommand is being removed (as the only enum is used to point to the create command for these two right now).

@Hwatwasthat thank you for taking this on.

Would this require reworking the portal subcommand (by removing it?)

Yes

would this mean creating tcp-inlet and tcp-outlet folders in the ockam_command/src folder? as the portal subcommand is being removed (as the only enum is used to point to the create command for these two right now).

Let's create:
ockam_command/
  src/
    tcp/
      mod.rs
      inlet/
        mod.rs
        create.rs
      outlet/
        mod.rs
        create.rs

We want to keep the folders because we'll add other commands soon.
Keeping them under a tcp/ folder would be nice because we want to in the near future make TCP functionality a plugin so all of them living one folder would be helpful.
this also applied to tcp-connection and tcp-listener in https://github.com/build-trust/ockam/issues/3076

@Hwatwasthat note in https://github.com/build-trust/ockam/pull/3140 for https://github.com/build-trust/ockam/issues/3076 @anuvratsingh started the ockam_command/src/tcp/* directory structure.
The inlet & outlet directories for tcp-inlet & tcp-outlet command will live there as well. Let me know in case you have any questions.

Hey @mrinalwadhwa, there is a TODO: Add delete, list, show subcommands, which I might be able to do, so should I do that in tcp inlet/outlet or in portal which would be refactored later on?

@anuvratsingh I think if you create them directly under tcp/inlet and tcp/outlet then it would lead to less conflicts irrespective of the order in which the PRs are created

@Hwatwasthat 👋 checking back to see if you need help with this or have any questions about the issue? If you don't have time to work on it, that's okay too, we'll open it up for others to explore.

Hey, yeah I've gotten a bit busier than expected! I'll keep an eye on it
and do out anyway for the practice but open it up so it might get done
faster if I can't find the time to do it.
On Thu, 11 Aug 2022, 06:23 Mrinal Wadhwa, @.***> wrote:

@Hwatwasthat https://github.com/Hwatwasthat 👋 checking back to see if
you need help with this or have any questions about the issue? If you don't
have time to work on it, that's okay too, we'll open it up for others to
explore.
—
Reply to this email directly, view it on GitHub
https://github.com/build-trust/ockam/issues/3077#issuecomment-1211563418,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHDI7P347GDYRXXDULTD7RDVYSE45ANCNFSM55EHLGJA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Thank you, when you have time to do some for practice / get familiar with the code. Have a look at the new issue in the https://github.com/build-trust/ockam/labels/good first issue label.