Use multiprocess if available

Windows doesn't have a true fork implementation, so multiprocessing and pickling are limited compared to other platforms. There are also some situations in Linux / Mac OS X where we would like to pickle something, but it does not work without some code changes. This PR makes pickling for mulitprocessing work in more situations by using the multiprocess library, if it's available.multiprocess is a fork of multiprocessing that uses dill instead of pickle.
This PR is set up such that we do not require the user to have multiprocess installed, but if it is installed we'll use it -- except in Windows, where pickling is so limited that it will make things easier to require it. There are a few other routes we could go:

Don't require it in Windows: This makes things simpler for Windows users who don't want to use multiprocessing. Installing the multiprocess package should work on all platforms, but if there's a platform where there are likely to be issues, it's Windows.

Require it for all platforms: This will likely be a net benefit to everyone, but it introduces a new dependency. Also, dill will always be a little bit slower than pickle since it's able to pickle more things. However, if we're not pickling anything too large anyway, performance shouldn't be an issue, and it would be possible to simplify some parts of the codebase by using dill instead of cloudpickle and removing some of the hacks that we currently use to get Windows to pickle things that it normally can't.

I'm happy to modify this PR to implement either option if you want! I've been using doit for a relatively large data science project and it's been a huge help.

Thanks.

On runner.py MRunner there is a check if multiprocessing is available. That should be taken into account...

In that spirit I guess it would be better to NOT make multiprocess required on Windows.

Also it would be nice to consolidate try/catch of the imports. Maybe on "compat.py" file.

Thanks for taking care of CHANGES file. Please also mention this in the "install.rst" file.

Finally, need to think how to deal this on CI.

Require it for all platforms: This will likely be a net benefit to everyone

could you please elaborate

It will also be useful on macOS, where spawn became the default in 3.8, and fork doesn’t always work properly.

https://github.com/uqfoundation/multiprocess/issues/65
It seems multiprocess package forced a "regression" because one of the features added (Pool) does not work well with "spawn" method. This leads me to think that multiprocess could be accepted as a multiprocessing replacement but could not be default event for Windows/MAC.

Hi @schettino72, since I was touching the doit code for #377 I updated this one also to take your feedback into account.

On runner.py MRunner there is a check if multiprocessing is available. That should be taken into account...

Done, I implemented the check in compat.py and call it from MRunner now.

In that spirit I guess it would be better to NOT make multiprocess required on Windows.

Yes, that makes sense. I instead made it an optional dependency that you can install with pip install doit[multiprocess].

Also it would be nice to consolidate try/catch of the imports. Maybe on "compat.py" file.

Done, the imports are done in compat.py now.

Thanks for taking care of CHANGES file. Please also mention this in the "install.rst" file.

Done!

Finally, need to think how to deal this on CI.

I didn't add anything to CI, but you could either modify one of the existing builds in your build matrix to install with multiprocess, by changing
      - run: pip install . -r dev_requirements.txt

to
      - run: pip install .[multiprocess] -r dev_requirements.txt

for that build (might need quotes, like pip install '.[multiprocess]', not sure what kind of shell Github actions uses). Or you could add a new build to the matrix that does the above with the latest version of Python. Those are my ideas anyhow; running the tests locally with multiprocess works correctly.
If you like, I can try to add a test that would fail with multiprocessing but succeed with multiprocess.

If you like, I can try to add a test that would fail with multiprocessing but succeed with multiprocess.

That would be great.


chore(auto-instrumentations-node)!: update @opentelemetry/instrumentation* dependencies to latest

Refs: #749

Which problem is this PR solving?
Typings mismatch between the auto-instrumentations-node package and underlying instrumentations.
Short description of the changes
This updates the versions of @opentelemetry dependencies to the latest in the auto-instrumentations-node metapackage.
Relates to #749.

Closing in favor of #769 :man_technologist:


Customize FilePlayer

I am trying to add a video quality control button to the  FilePlayer
import FilePlayer from "react-player/lib/players/FilePlayer";.This is playing a .m3u8 file yet i am not getting a control to change the quality in the player customize the skin a little.any areas to focus?

ReactPlayer doesn't provide any standard controls (yet). The default player controls show you pass in controls={true}. In the case of FilePlayer it's just the standard browser controls for HTML5 video, which don't support quality switching.
You may be better off using video.js or something similar. They have an example of a quality switcher: https://kmoskwiak.github.io/videojs-resolution-switcher/


Speed up build by not building examples and tests

Building the default target which includes examples and tests:
$ time make
[...]
make  55.96s user 6.75s system 91% cpu 1:08.67 total

Only using the build target:
$ time make build
[...]
make build  37.11s user 2.96s system 98% cpu 40.699 total

merged! will be making a new npm version ASAP!

@darora thank you for your contribution! Please try this new version :)
+<EMAIL_ADDRESS>

@darora also if you are using the parser/deparser
 -<EMAIL_ADDRESS> -<EMAIL_ADDRESS>


Make a sweep through all tests and clean formatting / visibility of properties as necessary.

Looking for stuff like this https://github.com/opencfp/opencfp/pull/270/files#r46741586 that isn't breaking the build by any means but is otherwise mostly style-related.

@mdwheele Actually, can we close this or should I leave it assigned to you?

Leave it assigned. I really want to make a pass through this to make things more consistent.

I'm going to end up marking tests that interact with sqlite as I go through and clean property visibility as well so we have a clean way of beginning to corral.


Handle byte string format according to specification

Contributing to Twilio

All third-party contributors acknowledge that any contributions they provide will be made under the same open-source license that the open-source project is provided under.

[x] I acknowledge that all my contributions will be made under the project's license.

I can see you did some changes. How can I help you to move this forward?

@hanny24 The nudge was appreciated, just rebased and force-pushed the reformatted branch -- let's see how far it gets.

hey @blast-hardcheese, would it be possible to release guardrail with this code?

Can do, thanks for your patience!

no worries


Add predicate tracing

It would be nice to have a way to trace particular predicates for debugging.  Should be pretty simple -- we can add it in the call instruction and compile out support if debug is 0 for performance.

Added rudimentary tracing.  No support for return values yet.


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target_61971002-036c-4dd0-b625-33bebfe8350d/pull/1 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target_61971002-036c-4dd0-b625-33bebfe8350d.git

Docs Build status updates of commit d75ecc5:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: GitBranchDeletedOrForcePushed] Cannot sync git repo to specified commit because branch Release_Archive_main_2022-03-08-05-23-59 has been deleted or has been force pushed

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Update nscplugin, sbt-scala-native, ... to 0.4.16

About this PR
📦 Updates

org.scala-native:nscplugin
org.scala-native:sbt-scala-native
org.scala-native:scalalib
org.scala-native:test-interface

from 0.4.10 to 0.4.16
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-native" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.scala-native" }
}]

labels: library-update, early-semver-minor, semver-spec-patch, commit-count:1

Superseded by #354.


Do not include approimateTerrainHeightsPrecise in built Cesium

While reviewing https://github.com/AnalyticalGraphicsInc/cesium/pull/8224 I noticed we were including approimateTerrainHeightsPrecise.json into the Build folder. This is not a big deal because it doesn't actually get included, it just adds half a megabyte to the zip file.

Thanks for the pull request @OmarShehata!

:heavy_check_mark: Signed CLA found.
:grey_question: CHANGES.md was not updated.

If this change updates the public API in any way, please add a bullet point to CHANGES.md.

:grey_question: Unit tests were not updated.

Make sure you've updated tests to reflect your changes, added tests for any new code, and ran the code coverage tool.

Reviewers, don't forget to make sure that:

[ ] Cesium Viewer works.
[ ] Works in 2D/CV.
[ ] Works (or fails gracefully) in IE11.

Why was this file kept around at all?  Since we generate this file directly from terrain data, we can just generate the more compact version directly the next time we do it. CC @tfili @hpinkos

This was the original PR https://github.com/AnalyticalGraphicsInc/cesium/pull/7959 where we kept the original file because (at least I) thought we didn't have a documented process of regenerating the original numbers.
I can just remove the precise version and the gulp task that regenerates it to some given degree of precision if our process for generating this file is outside this repo.

@mramato yeah, this file wasn't deleted in case we needed to regenerate the values for approximateTerrainHeights at a different precision

@tfili has a node script that does this, but it relies on some non-public data coming from CWT.  So we should document the process (even if it requires some private data), and make sure the script outputs less precise numbers (which should be trivial).  For now, I would recommend just deleting the precise version and script and writing up an issue (it's in git history if we ever really need it).  Hopefully Fili can chime in with more details.

We have an issue open already: #6557

Not much to it. Run the bin/generateMinMax.js script in the generateMinMax branch of agi-node-terrain. It could probably use a merge of master and an update to new syntax. I think you just point it at the terraindb and let it run.


Training Script

It would be worth to provide a train script, in order to train larger models (for instance 7B, 13B).

You can use whichever training script / library you'd like, e.g. Megatron, DeepSpeed, lightning, hf accelerate etc. Just have to replace the model definition.
Examples:
Lightning has lit-gpt: https://github.com/Lightning-AI/lit-gpt
FlashAttention has training code, you can swap the model: https://github.com/Dao-AILab/flash-attention/tree/main/training

does not seem to be so straightforward with HF trainer, quite literally:
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1555, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2725, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2748, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py", line 680, in forward
    return model_forward(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py", line 668, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
TypeError: MambaLMHeadModel.forward() got an unexpected keyword argument 'labels'

no labels in forward() ?
would be very nice if you could provide a simple, minimal example of how to use the models with HF trainer. thank you!

We've managed to train mamba by modifying the Huggingface Trainer class. Here is our implementation, we were actually able to train a chat model that seems to perform quite well.

We've managed to train mamba by modifying the Huggingface Trainer class. Here is our implementation, we were actually able to train a chat model that seems to perform quite well.

Cool, nice work! Are you using fp32 for this finetuning work?

Hmm... Doesn't seem to work lit-gpt.
Minimal example:
from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
from mamba_ssm.utils.hf import load_config_hf
from mamba_ssm.models.config_mamba import MambaConfig

with fabric.init_module(
        empty_init=isinstance(fabric.strategy, DeepSpeedStrategy)
    ):
  config = load_config_hf('state-spaces/mamba-2.8b')
  model = MambaLMHeadModel(MambaConfig(**config))

This will give the following error
Traceback (most recent call last):
  File "/home/me/lit-gpt/pretrain/ahrefs_mamba.py", line 782, in <module>
    setup(run_config)
  File "/home/me/lit-gpt/pretrain/ahrefs_mamba.py", line 306, in setup
    main(fabric, run_config)
  File "/home/me/lit-gpt/pretrain/ahrefs_mamba.py", line 342, in main
    model = MambaLMHeadModel(MambaConfig(**model_config))
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 385, in wrapper
    f(module, *args, **kwargs)
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/mamba_ssm/models/mixer_seq_simple.py", line 199, in __init__
    self.backbone = MixerModel(
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 385, in wrapper
    f(module, *args, **kwargs)
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/mamba_ssm/models/mixer_seq_simple.py", line 118, in __init__
    [
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/mamba_ssm/models/mixer_seq_simple.py", line 119, in <listcomp>
    create_block(
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/mamba_ssm/models/mixer_seq_simple.py", line 42, in create_block
    block = Block(
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 385, in wrapper
    f(module, *args, **kwargs)
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/mamba_ssm/modules/mamba_simple.py", line 316, in __init__
    self.mixer = mixer_cls(dim)
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 385, in wrapper
    f(module, *args, **kwargs)
  File "/home/me/lit-gpt/mamba-venv/lib/python3.9/site-packages/mamba_ssm/modules/mamba_simple.py", line 99, in __init__
    self.dt_proj.bias.copy_(inv_dt)
RuntimeError: The size of tensor a (0) must match the size of tensor b (5120) at non-singleton dimension 0

if that's true, how the heck am I to pass attention for SFT?

Stage 2 works, but not stage 3. I don't have a fix at the moment.


Development Plan

The development tasks are broken down by topic, here we will brainstorm different components of Zambeze that need to be worked on. The hardest part is defining where one topic ends and another begins so we don't tried on each others toes while developing.
Detailed descriptions will be placed in separate issues once we flesh them out.
Topics
Zambeze command-line interface
The Zambeze command line interface is a python script that the user can use to interact with the orchestrator. The orchestrator should be running in the background on the same machine as the script. When a user makes a request using the command-line interface it connects to the orchestrator and passes the message along. The command-line interface should be able to report back if there are errors connecting and fail gracefully if the orchestrator does not respond in an appropriate amount of time.
Orchestrator file parser for reading Campaign file (Scientific Workflow File)
Scientific workflows that are understood by Zambeze should follow a declarative style and be written in .json (I think that is what we agreed on). We will need to be able to read these files and ensure that the file content and keywords are understood and validated.
Orchestrator logic for Task Submission to Queue (NATS)
Once a file is read in, the orchestrator will need to know how to split these into tasks with actions. It will also need a mechanism for submitting these tasks to the NATS queue. Once a submission is made the orchestrator will need to be able listen for a response... (The details of this I am uncertain about and need to be fleshed out)
Orchestrator Abstraction for interacting with supported third-party APIs i.e. (Globus/DataFed)
When an orchestrator is assigned a task it might be required to support data movement with Globus/DataFed/Some other application, or perform some other kind of computation. An abstraction layer is needed to be able to support communication with third-party applications outside of Zamebeze.
Orchestrator bid generation
When an orchestrator sees a task in NATS that is up for grabs, it will need to pull the message and evaluate if it has the resources to satisfy the request. It then needs to submit a bid to the orchestrator that originally submitted the request - it has to do this by placing a new task "Control task" back in the queue with a plan/bid for how it can satisfy the request.
Orchestrator bid evaluation
An orchestrator that has a "Data" or "Compute" task will receive bids back from the queue (NATS). It could receive anywhere between 0 and more bids back. It needs to have logic to evaluate which bid is the best and confirm the winning bid can proceed with the assignment.
.

Thanks @JoshuaSBrown for the initial description. Here are some thoughts:

Campaign file: this will be implemented in a later phase. Initially, we will use an API (i.e., a campaign.py file that will expose a Campaign class) in which the user will be able to declare their workflow (data and compute actions).
CLI: will also be implemented later. The API described above will be initial interface to use Zambeze.

For the orchestrator, I think it will be useful to create a graph so we can detail the functionalities described above. I will try to start a draft.

@tskluzac, I would go with option 2 for now, and once we have our own auth model, we will move all of it into the new system.
Hope you all are able to get some good hours of sleep :)

Another dev/design-related question --- now that the agent is a daemon that 'disconnects' from its parent process, I'm wondering what should happen when stopping an agent:

Campaign-created agents: should these be terminated as soon as the campaign terminates? At first I thought "definitely yes", but now I realize there's a case where that agent could have theoretically picked up work submitted by a different campaign (and stopping it would interrupt that work). This leads to my second point...

Stop behavior: in the current model, I assume that users stop agents when they are confident work is done -- therefore a hard cleanup can be had where the entire child subprocess is immediately killed. We could instead use a 'soft' stop model (i.e. where some "KILL" message is added to the queue that will prevent the agent from grabbing more tasks), but then we run the risk of users stopping an agent and it hangs indefinitely (and makes matters difficult when users try to restart their agent -- it would either have to create a second agent or force them to wait).

The best solution to both of these problems that I can think of is to continue to have hard shutdowns, but do:
A) the campaign doesn't start its own agent and just latches onto a user's locally-running agent, if one is running. Then the campaign DOES NOT automatically terminate at the end of the campaign. (user's responsibility)
B) we let the campaign continue to spin up/down an agent, but put some sort of limitation on campaign-adjacent agents that they cannot receive work. Then we can automatically spin it down without fear of deleting work.
Happy to discuss further (or have someone point out that I'm misunderstanding something!)

Can we close this issue as we are using Projects now?


Unable to connect using kerberos on macos

Issue Type: Bug
Followed the instructions outlined here: https://docs.microsoft.com/en-us/sql/azure-data-studio/enable-kerberos?view=sql-server-ver15
and still was not able to connect to SQL Server using Windows authentication.
Connection failed due to Kerberos error.
Help configuring Kerberos is available at https://aka.ms/sqlopskerberos
If you have previously connected you may need to re-run kinit.
running klist I can see my ticket.
Running on macOS Big Sur version 11.4
Azure Data Studio version: azuredatastudio 1.30.0 (59c4b8e90cf2d3a24bed72623197f10f8d090fdc, 2021-06-16T18:57:01.467Z)
OS version: Darwin x64 20.5.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz (16 x 2400)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onprotected_video_decode: unavailable_offrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
2, 2, 2

Memory (System)
64.00GB (1.98GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions: none

@shawn-gmr have you tried this? https://github.com/microsoft/azuredatastudio/issues/23#issuecomment-345361519

I haven't, but will give it a try and let you know

should we see  RestrictedKrbHost/ at all after running setspn -L, if the SPN has been registered for SQL Server? I see 7 other values as well, including HOST/, TERMSRV/ and WSMAN/

@shawn-gmr yes I see all of those. Also, I am not sure, its been a while but changing those files on macOS don't make a difference. I think setspn is what fixed it for me.

Our issue may be internal. Either SQL Server or Domain Controllers, since there was a change.


Question about figure 3 visualization

Hello, I am currently a final year student and I have a graduation project on this knowledge graph topic. I am very impressed with your approach to the problem. I have a question about "Figure 3" in the article, I don't quite understand how to draw this graph. Could you please be more specific about this?
Thank you very much and I look forward to hearing from you

Just save the learned relation embeddings and plot the phase part using matplotlib.pyplot.hist. You may need to tune the bins parameter to obtain a better visual effect.


Package Requested

[X] I would like someone else to build the manifest.
[ ] I would like help so I can submit the manifest.
[ ] I have performed a search and couldn't find this package.
[ ] I have checked that the installer for this package is not contained in a .zip file
[ ] I think there is a new version available and I have provided the URL.

Please provide the following information
Publisher: Virtualhere
Publisher Url: https://www.virtualhere.com/
Package Name: Virtualhere
Description: VirtualHere allows USB devices to be used remotely over a network just as if they were locally connected!
Package Version: 4.6.3
Installer URL: https://www.virtualhere.com/sites/default/files/usbclient/vhui64.exe
Current Package Identifier (for new versions): 4.6.3

#78644


Bug: const in createMachine type definition breaks the method's typing

XState version
XState version 5
Description
When using setup({ … }).createMachine({ … }), the machine definition passed to the latter method reports the error "Expected 0 arguments, but got 1" in the TypeScript linter (see screenshot).

This screenshot demonstrates the example from machines documentation. (Note: you also have an error in your types property. Lines should end with commas not semi-colons.)

Expected result
I expected the example from the documentation to be typed correctly. I did not expect to see an error when createMachine is provided a configuration object.
Actual result
The TypeScript linter displays an error. It expects zero arguments despite the actual requirement of a configuration object.
Reproduction
https://codesandbox.io/p/devbox/github/statelyai/xstate/tree/main/templates/vanilla-ts?file=%2Fsrc%2FfeedbackMachine.ts%3A30%2C13
Additional context
The error appears originate in the generated setup.d.ts.

When the const is removed, the code is correctly typed and the original error disappears.

The const was introduced in this commit of this PR.
Unsure what regressions are introduced by removing the offending code. I'll comment back if I find any while I continue to develop off of a patch branch.

The lint error does not appear to occur in isolation: https://codesandbox.io/p/devbox/admiring-star-7hjlsm?file=%2Fsrc%2FfeedbackMachine.ts%3A22%2C5
I'm using the same versions of TS and xstate:

Per documentation, the relevant tsconfig appears to match:

I may have solved my problem.
I had to add updated eslint dependencies ot the project.

The project lives in a monorepo and the root's eslint dependencies were out of sync with the project's typescript dependency.

From what I've seen so far the error exactly like this is usually an indication of using an older TS version (4.x).

The resolution here is to ensure you're using the latest version of TypeScript.
More info here: 📖 https://stately.ai/docs/typescript

I'd upgraded to the latest version of Typescript (5.3.3), but the error was still appearing. It wasn't until I upgraded our TS eslint plugins that the error disappeared. Might not be necessary to document, but good to remember if this comes up again.
Now for today's update: at the end of yesterday, I installed Pretty TypeScript Errors and the issue has resurfaced. Even after uninstalling, the problem remains. It must have modified a TS version setting somewhere, but I haven't found it yet.

Latest versions:

interesting… Restart TS Server is missing as well

Note that VS Code might use its own TS version until you manually "Select TypeScript Version..." in the command palette and choose to use the workspace version.

It looks like that was the issue. It was relying on the monorepo's TS version instead of the package. It HAD been using the package's version, but the plugin install must have modified it. 🤷

I was able to get things sorted.

Thanks for your advice!


Text-Code MRR

I have two questions on Text-Code readme:

Do you consider MRR of both validation and test dataset? The dataset (test.txt) includes both validation and test dataset of CodeSearchNet.
MRR of CodeBERT reported as "0.8685" here but it is indicated as {'MRR': 0.2719} in readme.

I have two questions on Text-Code readme:

Do you consider MRR of both validation and test dataset? The dataset (test.txt) includes both validation and test dataset of CodeSearchNet.
MRR of CodeBERT reported as "0.8685" here but it is indicated as {'MRR': 0.2719} in readme.

We have considered MRR of both validation and test dataset. This dataset is different from CodeSearchNet. We first filter the dataset as mentioned in the readme, and function names and variables in dev/test sets are replaced by special tokens. Then, we combine validation and test dataset of CodeSearchNet and re-split them so that test dataset have more examples. Urls in valid.txt and test.txt have no overlap.

valid=set()
test=set()
with open("valid.txt") as f:
...     for line in f:
...             valid.add(line.strip())
with open("test.txt") as f:
...     for line in f:
...             test.add(line.strip())
len(valid&test)
0

2 .The experiment setting is different from CodeBERT in the paper. The experiment setting in this dataset is to retrieve source code from whole validation/test dataset, while the setting reported in the CodeBERT paper is to retrieve source code from only 1000 candidates. Besides, to test the generalization ability of a model, function names and variables in test sets are replaced by special tokens. Therefore, this dataset and experiment setting is more difficult and MRR is lower.


It seems that brand and topline are not included in header?

Hi,
It seems that brand and topline are not included in header?
Why's that? Any consideration?
Wei

Yes, I also have this issue, any chance we could get a fix or a clarification on how to use these?
Also, my site's top bar is a grey instead of the black the screenshots show. Is this a parameter somewhere?

Change the theme in theme file:
hugo-bootstrap/layouts/partials/base/imports.html

or better way, create same directory in your site directory
/baseOfYourSite/layouts/partials/base/imports.html

And then change theme and highlight
  <link rel="stylesheet" href="{{ "/css/theme/" | absURL }}yeti.css">

  <link rel="stylesheet" href="{{ "/css/highlight/" | absURL }}yeti.css">

Also there is easier way to do it, though it is not documented. You can just specify bootswatch theme in config file. Example for config.toml:
[params]
  [params.theme]
    name = "yeti"

I have updated this theme to latest bootstrap 4. I am closing this issue for now. Please open a new one if the latest theme does not do what you need.

Closing


Firewall livelog: add dropdown selects for static fields

Entries like direction and action have a predefined set of valid options, we probably better load these in the form and flip the text input with a dropdown in these cases.

Yes 👍


[canary] Measure usage of external services along that provides file-related feature

need to add instrumentations to @vercel/blob methods on canary agent:

[x] put()
[x] del()
[x] list()

"canary" agent released by https://github.com/giselles-ai/giselle/pull/247


download fails with net::ERR_NAME_NOT_RESOLVED

I don't have the complete direct link for the file, but using my browser, when I click at the button it downloads it. So I try to do this using chromedp. Sometimes it works (e.g. 40% of success) and sometimes it doesn't.
Is there any way to detect when it doesn't I can re-trigger it until it works? It always completes with 0 exit code.
Some other related questions:

Doesn't always start to download
How to detect it started?
How to detect when it finishes?

What versions are you running?

github.com/chromedp/chromedp v0.5.3

$ google-chrome --version # this didn't work on my terminal. I opened the browser and I've got:
Version 85.0.4183.121 (Official Build) (64-bit)

go version go1.15.2 darwin/amd64

What did you do? Include clear steps.
package main

import (
  "fmt"
  "github.com/chromedp/cdproto/page"
  "github.com/chromedp/chromedp"
)

func main() {
  fetchDownloadLink("https://vimm.net/vault/1640")
}

func fetchDownloadLink(romLink string) {
  // create chrome instance
  ctx, cancel := chromedp.NewContext(context.Background(), chromedp.WithDebugf(log.Printf))
  defer cancel()

  // create a timeout
  ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
  defer cancel()

  err := chromedp.Run(ctx,
    chromedp.Navigate(romLink),
    page.SetDownloadBehavior(page.SetDownloadBehaviorBehaviorAllow).WithDownloadPath("."),
    chromedp.Click(`#download_form > button`),
    chromedp.Sleep(20*time.Second),
  )
  if err != nil {
    log.Fatal(err)
  }
  log.Println("success")
}

What did you expect to see?
e.g. when it works the logs look like this:
{"method":"Page.downloadWillBegin","params":{"frameId":"3E04B393578256B78EF50D7CC6E04D2A","guid":"99697456-1fa0-40a5-bd5b-8d81260fb045","url":"https://download.vimm.net/download/?mediaId=1745","suggestedFilename":"Wolfenstein 3-D (USA).zip"},"sessionId":"DCBECF64D8041046440B4683E5AF01C9"}

then it starts downloading ...
{"method":"Page.downloadProgress","params":{"guid":"99697456-1fa0-40a5-bd5b-8d81260fb045","totalBytes":535507,"receivedBytes":114680,"state":"inProgress"},"sessionId":"DCBECF64D8041046440B4683E5AF01C9"}

and finishes:
{"method":"Page.downloadProgress","params":{"guid":"99697456-1fa0-40a5-bd5b-8d81260fb045","totalBytes":535507,"receivedBytes":535507,"state":"completed"},"sessionId":"DCBECF64D8041046440B4683E5AF01C9"}

What did you see instead?
{"method":"Log.entryAdded","params":{"entry":{"source":"network","level":"error","text":"Failed to load resource: net::ERR_NAME_NOT_RESOLVED","timestamp":1.601630035812447e+12,"url":"https://cluster-na.cdnjquery.com/color/jquery.color-2.1.2.min.js?integrity=btjsonpcallback1601630035695&checksum=%7B%22cbc%22%3A0%2C%22st%22%3A2%2C%22au%22%3A%5B%5D%2C%22hau%22%3A%5B%5D%2C%22ref%22%3A%22https%3A%2F%2Fvimm.net%2Fvault%2F1640%22%2C%22aa%22%3A3%2C%22pgid%22%3A%2298907c77-048f-11eb-8f98-5e10fc12e5a5%22%2C%22v%22%3A1%2C%22format%22%3A%22jsonp%22%7D&o=https%3A%2F%2Fvimm.net%2Fvault%2F1640&csVersion=1.22.36&clearThroughOptions=undefined","networkRequestId":"58383.86"}},"sessionId":"8D3E26DC6CDE32DE49247056D9A3008B"}

It's a long time since the question was asked. Hope that it still helps.

You can subscribe to the page.EventDownloadWillBegin event and the page.EventDownloadProgress event to detect the download state.
When the function chromedp.Click() is called, it's possible that the button is not ready yet. I think that's the real reason why the download failed occasionally. And please note that the position of the button changes sometimes, which will make the function chromedp.Click() dispatch the mouse event to the stale position.

Here is the updated code (with changes highlighted) to make it more reliable:
diff --git a/download.go b/download.go
index fdace35..3f037cc 100644
--- a/download.go
+++ b/download.go
@@ -1,35 +1,57 @@
 package main

 import (
    "context"
-   "fmt"
-   "github.com/chromedp/cdproto/page"
-   "github.com/chromedp/chromedp"
    "log"
+   "sync"
    "time"
+
+   "github.com/chromedp/cdproto/page"
+   "github.com/chromedp/chromedp"
 )

 func main() {
    fetchDownloadLink("https://vimm.net/vault/1640")
 }

 func fetchDownloadLink(romLink string) {
    // create chrome instance
    ctx, cancel := chromedp.NewContext(context.Background(), chromedp.WithDebugf(log.Printf))
    defer cancel()

    // create a timeout
    ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
    defer cancel()

+   var wg sync.WaitGroup
+   wg.Add(1)
+
+   chromedp.ListenTarget(ctx, func(ev interface{}) {
+       switch ev := ev.(type) {
+       case *page.EventDownloadWillBegin:
+           log.Println("EventDownloadWillBegin", ev.URL)
+       case *page.EventDownloadProgress:
+           log.Println("EventDownloadProgress", ev.State)
+           if ev.State == page.DownloadProgressStateCompleted {
+               wg.Done()
+           }
+       default:
+           return
+       }
+   })
+
    err := chromedp.Run(ctx,
        chromedp.Navigate(romLink),
        page.SetDownloadBehavior(page.SetDownloadBehaviorBehaviorAllow).WithDownloadPath("."),
+       chromedp.WaitReady(`body`, chromedp.ByQuery),
        chromedp.Click(`#download_form > button`),
-       chromedp.Sleep(20*time.Second),
+       chromedp.ActionFunc(func(_ context.Context) error {
+           wg.Wait()
+           return nil
+       }),
    )
    if err != nil {
        log.Fatal(err)
    }
    log.Println("success")
 }

Many thanks


feat: prefilter

r?

Build failed.

Thanks for your contribution! Please add some descriptions to this PR about what is done in this PR (quantization? filter?). And also how to use the filter feature?


Update dependency.yml as label in not permitted in markdown block

Why are the changes needed?

bugfix for issue template
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

cc @pan3793

Thanks, merged to master


Allow redirecting sections to other manuals

To support some of the zendesk tickets in: https://trello.com/c/2pxsDfWf/103-unpublish-hmrc-manual-sections
We already support redirecting a section to another section, or to it's parent manual.  This change allows us to redirect a section to another manual root.

Looks good to me! 👍


Fix Facebook  Sing-In docs - activity element

According to FB documentation the meta-data and activity should be inside the application element:
See section 5 in the link:
Add the following meta-data element, an activity for Facebook, and an activity and intent filter for Chrome Custom Tabs inside your application element:

Thank you, good catch!


Provider class name in readme/docs differs from actual

Class Bramdevries\Oauth\Client\Provider\Slack differs from the docs, where League\OAuth2\Client\Provider\Slack is used. I couldn't get the latter to work at all, until it was pointed out to me that the former should be used (and works).

I had the same problem. I fixed it by using:
use Bramdevries\Oauth\Client\Provider\Slack;
// ...
$provider = new Bramdevries\Oauth\Client\Provider\Slack([
// ...


lookup('env', 'PWD') returns an empty string when deploying TDP via tdp-server running as a service.

Error
A deployment via TDP-server or TDP-ui, when TDP-server running as a Linux service, craches with the error like:
2023-07-06 19:02:45,683 p=28584 u=root n=ansible | fatal: [dcgfy6pq]: FAILED! => changed=false
  msg: |-
    Could not find or access '/files/jmx_prometheus_javaagent-0.16.1.jar' on the Ansible Controller.

Cause
The absolute path to components build files is defined in the variable binaries_local_dir: "{{ lookup('env', 'PWD') }}/files". This works fine when launching a deployment via TDP-lib (tdp deploy) or via TDP-server (or TDP-ui) when tdp-server is started as a process in terminal.
However, it doesn't work when TDP-server is started as a Linux service because {{ lookup('env', 'PWD') }} returns an empty string. So far, I couldn't find why Ansible executor doesn't bring PWD from Linux services.
To reproduce

Install TDP-server while installing TDP-getting-started

Configure TDP-server as a Linux service (configuring tdp-server.service file)

Example /etc/systemd/system/tdp-server.service file
[Unit]
Description=TDP Server

[Service]
User={{ service_user }}
WorkingDirectory={{ service_directory }}
ExecStart={{ service_directory }}/venv/bin/python {{ service_directory }}/venv/bin/uvicorn tdp_server.main:app --reload
EnvironmentFile=-/etc/sysconfig/tdp-server
Environment=PATH={{ service_directory }}/venv/bin/:/sbin:/bin:/usr/sbin:/usr/bin
Environment=GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git
Restart=always

[Install]
WantedBy=multi-user.target

Example /etc/sysconfig/tdp-server file:
# common
TDP_COLLECTION_PATH=ansible_collections/tosit/tdp:ansible_collections/tosit/tdp_extra
TDP_RUN_DIRECTORY=.
TDP_VARS=./tdp_vars
# tdp-lib
TDP_DATABASE_DSN=sqlite:///sqlite.db
# tdp-server
PROJECT_NAME=tdp-server
BACKEND_CORS_ORIGINS="[\"http://{{ tdp_ui.host }}:{{ tdp_ui.port }}\"]"
SERVER_NAME={{ tdp_server.host }}
SERVER_HOST=http://{{ tdp_server.host }}:{{ tdp_server.port }}
# OPENID_CONNECT_DISCOVERY_URL=http://localhost:8080/auth/realms/tdp_server/.well-known/openid-configuration
OPENID_CLIENT_ID=tdp_server
DATABASE_DSN=sqlite:///sqlite.db
DO_NOT_USE_IN_PRODUCTION_DISABLE_TOKEN_CHECK=True
UVICORN_HOST={{ tdp_server.host }}
UVICORN_PORT={{ tdp_server.port }}

Start TDP-server service with systemctl start tdp-service

Run deployment with the command like:

curl -X POST http://UVICORN_HOST:UVICORN_PORT/api/v1/deploy/dag

Workaround
There are multiple choices:

When running TDP-server as a Linux service, provision it with the environment variable PWD equals the root folder of your files folder with binaries.

Exemple /etc/sysconfig/tdp-server:
# ... previous vars

# pointing the root folder of `tdp-getting-started` (same as `{{ service_directory }}`)
PWD=.

Overwrite the binaries_local_dir variable with smth without lookup, for example:

binaries_local_dir: "{{ service_directory }}/files"

Solution
Either to learn why Ansible doesn't bring PWD from Linux services and fix it. Either to define binaries_local_dir differently without lookup.

@rpignolet what do you think, can we avoid using lookup('env', 'PWD')?


Can't deploy Python Azure TimerTrigger Function

When trying to deploy my Python Azure function towards Azure I stumble upon the fact that my TimerTrigger function is not deployed e.g. not visible in the Azure portal. The deployment goes without any errors. Other default template functions like the HTTP trigger and QueueTrigger work fine.
Repro steps:

Create an Azure Functions Python project
Add a Timer Function
Add code to write a message towards a DataTable to verify it ran
Run function locally --> works like a charm
Click "Deploy to Function App.."
Refresh DataTable and notice that there is no new message

VisualStudioCode: v1.43.0
AzureFunctionsExtension: v0.21.0
OS: win10x64
OS Release: 10.0.18362

@Smurfffje the DataTable stuff require a few app settings. Did you remember to set those when running in Azure? If you already have the settings on your local machine, you can right click on "Application Settings" under the function app and upload the settings
If this project is on GitHub somewhere I might be able to take a look. Seems weird that the timer trigger isn't even showing up in the portal

@EricJizbaMSFT thanks for the reply, The DataTable stuff is fine since the exact same code in an HTTP trigger function gives the desired output.
It is just the TimerTrigger function, If I create an empty project with just the timertrigger template and deploy it to Azure I get an empty function app. If I add a HTTP trigger function and then redeploy, the function app only has the HTTP trigger function

Okay so I can essentially ignore the DataTable stuff? I'm still gonna need more info here because it works fine for me. Maybe post a screenshot of your workspace and copy the deploy logs here. Also - how did you create the Function App in Azure - from VS Code or the portal? Anything special about it?

@EricJizbaMSFT yes that can be ignored, it is purely about the Timer function. I used VS code for everything, nothing special just the basic template like VS code creates them. I have uploaded a screenshot of the workspace, the buildlog as seen in VS code and the important files from the TimerTrigger function
gitbuildlog.txt

init.txt

This is how it looks from the Azure Portal

How about the function.json file for the timer trigger

{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "name": "mytimer",
      "type": "timerTrigger",
      "direction": "in",
      "schedule": "0 */5 * * * *"
    }
  ]
}

Yeah I honestly have no idea what's going on here. If this is still happening - you might have better luck with the Azure Functions support directly. Sometimes the Azure teams has access to more logs which might explain the problem. We also have https://github.com/microsoft/vscode-azurefunctions/issues/1848, but that is usually a momentary problem that goes away after a few minutes

Right now it is deploying as expected, however sometimes deployment says that it completed but it did not since it hang on a module installation from the requirements.txt file. But these errors do not end up in the deployment window.

however sometimes deployment says that it completed but it did not since it hang on a module installation from the requirements.txt file. But these errors do not end up in the deployment window.

This piece should be fixed as a part of https://github.com/microsoft/vscode-azuretools/pull/703. Going to close since that's already tracked in other issues


Add Dockerfile

As promised, I created a simple Dockerfile for GraphHopper.
Build an image:
docker build -t graphhopper:master .
Run tests:
docker run -it graphhopper:master ./maven/bin/mvn clean test verify
Run graphhopper:
docker run -it -p 8989:8989 -v /home/username/graphhopper-maps/:/data graphhopper:master ./graphhopper.sh web /data/europe-latest.osm.pbf
You could add this Project to docker hub and let them build the images for you, which everyone can use.
Please let me know if there is something i should change.

+1

Thanks a lot! As we have currently already many files in the project root, is it possible to move the 2 new docker files to e.g. core/files or would it make things ugly/complicated?

And I assume switching to a stable branch like 0.8 is possible via
docker build -t graphhopper:0.8 .
Or what does graphhopper:master mean?

If I move the Dockerfile to core/files, then it would be unpossible to COPY the repository data into the docker image.

The  path must be inside the context of the build; you cannot COPY ../something /something, because the first step of a docker build is to send the context directory (and subdirectories) to the docker daemon. -- https://docs.docker.com/engine/reference/builder/#/copy

graphhopper:master was just an example for a tag name of the docker image. One would build an image of the 0.8 branch by first checking it out and then use docker build -t graphhopper:0.8 .
https://docs.docker.com/engine/reference/commandline/build/#/tag-image--t

There could be a Dockerfile in core/files which downloads the released zip files, like i did in https://github.com/edy/graphhopper-docker

Hmmh, difficult. I thought I could just fetch the docker file and work with it (install it etc) without being attached to any repository or data?
For normal production use I would prefer fetching the stable zip. But sometimes a tweak or bug fix is necessary. Maybe like in your 2nd suggestion but with git checkout 0.8?

graphhopper:master was just an example for a tag name of the docker image

Sorry, have not yet any idea of the concepts for docker :) which makes it a bit hard to decide what to do

Maybe like in your 2nd suggestion but with git checkout 0.8

I'd go with the 0th (Dockerfile inside the project root).
Pros:

you can git checkout your branch/tag/commit and build an image for this version
Docker Hub has automated builds which are triggered on pushes (commits or tags). Every pushed tag will create a new docker image tagged as eg. graphhopper:0.8.2

Cons:

two more files inside the project root 😃

My other suggestions require more work because the version number must be changed manually inside the Dockerfile on every new release.

Ok, thanks for the explanation - convinced & merged :)


Question: sui::clock

I am writing a module using sui::clock, but when i write unit-test for it I get EEmptyInventory error when I try to use test_scenario::take_shared function to get globally shared clock object. Does anyone know how to solve it. Thanks !!

kamyardl

I have solved the problem. Tks


[Device Support Request] BlitzWolf BW-SHP13 smart plug (aka Tuya TS0121)

Is your feature request related to a problem? Please describe.
BlitzWolf BW-SHP13 smart plug 16A EU version, same device as TuYa TS0121 smart plug.

Device joins zigbee network and it's recognized by home-assistant.
Entities are created for:

On/Off switch: It works
Electrical power measurement (W): Values updated correctly.
Energy metering (kWh): Entity is created but values are never updated.

This entities are missing:

Current (A).
Voltage (V).

Describe the solution you'd like
Please consider adding support for the missing values and fixing the electrical power measurement value.
Device signature - this can be acquired by removing the device from ZHA and pairing it again from the add devices screen. Be sure to add the entire content of the log panel after pairing the device to a code block below this line.
{
  "node_descriptor": "NodeDescriptor(byte1=1, byte2=64, mac_capability_flags=142, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=0)",
  "endpoints": {
    "1": {
      "profile_id": 260,
      "device_type": "0x0051",
      "in_clusters": [
        "0x0000",
        "0x0004",
        "0x0005",
        "0x0006",
        "0x0702",
        "0x0b04"
      ],
      "out_clusters": [
        "0x000a",
        "0x0019"
      ]
    }
  },
  "manufacturer": "_TZ3000_3ooaz3ng",
  "model": "TS0121",
  "class": "zigpy.device.Device"
}

Additional context
This device doesn't support reporting of electrical measurement values, so it needs to be polled. It's supported on zigbee2mqtt, they fixed this by polling the values every 10 seconds.
Voltage value is correctly reported if I manually get the attribute value:

Hi, yes it would be great if it was fully supported as you describ it.
I also have the same device, but with a slightly different signature : (the manufacturer part differ)
 {
  "node_descriptor": "NodeDescriptor(byte1=1, byte2=64, mac_capability_flags=142, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=0)",
  "endpoints": {
    "1": {
      "profile_id": 260,
      "device_type": "0x0051",
      "in_clusters": [
        "0x0000",
        "0x0004",
        "0x0005",
        "0x0006",
        "0x0702",
        "0x0b04"
      ],
      "out_clusters": [
        "0x000a",
        "0x0019"
      ]
    }
  },
  "manufacturer": "_TZ3000_g5xawfcq",
  "model": "TS0121",
  "class": "zigpy.device.Device"
}

Z2M polls them every 60 seconds: https://github.com/Koenkk/zigbee-herdsman-converters/blob/c9ed6bb75a9682cf60b41e4e7647c653480f9a6c/devices.js#L1490-L1503 because they don't support attribute reports apparently (or at least don't do it properly).
It doesn't look like that zha-quirks has something similar yet. So I'm not sure how something like this would even be implemented correctly.

Z2M polls them every 60 seconds: https://github.com/Koenkk/zigbee-herdsman-converters/blob/c9ed6bb75a9682cf60b41e4e7647c653480f9a6c/devices.js#L1490-L1503 because they don't support attribute reports apparently (or at least don't do it properly).
It doesn't look like that zha-quirks has something similar yet. So I'm not sure how something like this would even be implemented correctly.

ZHA already polls the electrical measurement cluster. Is this not an EM cluster?

ZHA already polls the electrical measurement cluster. Is this not an EM cluster?

It should be. Perhaps it's missing these constant attributes (https://github.com/Koenkk/zigbee-herdsman-converters/blob/c9ed6bb75a9682cf60b41e4e7647c653480f9a6c/devices.js#L1479-L1483) (or the binding above that)? (Just guessing at this point though).
The Xiaomi quirk does something similar: https://github.com/zigpy/zha-device-handlers/blob/dev/zhaquirks/xiaomi/init.py#L410-L415
(Although the Xiaomi Smart Plug (EU) with power reporting also worked before doing that for me)

It should be. Perhaps it's missing these constant attributes (https://github.com/Koenkk/zigbee-herdsman-converters/blob/c9ed6bb75a9682cf60b41e4e7647c653480f9a6c/devices.js#L1479-L1483) (or the binding above that)? (Just guessing at this point though).
The Xiaomi quirk does something similar: https://github.com/zigpy/zha-device-handlers/blob/dev/zhaquirks/xiaomi/init.py#L410-L415
(Although the Xiaomi Smart Plug (EU) with power reporting also worked before doing that for me)

I just re-read the issue. It looks like the electrical power measurement (W) is working correctly, right? Only voltage and current do not have entities created for them. The total "smart" (?) energy meter isn't updated.
That looks like the same "issue" basically all power measuring plugs have (at least the Xiaomi Plugs) (?)

I just re-read the issue. It looks like the electrical power measurement (W) is working correctly, right? Only voltage and current do not have entities created for them. The total "smart" (?) energy meter isn't updated.
That looks like the same "issue" basically all power measuring plugs have (at least the Xiaomi Plugs) (?)

Yes, if that’s the case. We currently only support 1 sensor per cluster. This is currently a limitation in ZHA.

Yes, if that’s the case. We currently only support 1 sensor per cluster. This is currently a limitation in ZHA.

Got my Blitzwolf plugs today. They seem to work well so far. (On/off + current consumption).
With Z2M you can also set the initial power on state: https://www.zigbee2mqtt.io/devices/TS0121_plug.html#power_outage_memory-enum
I guess that's something ZHA could also implement. Since that shouldn't be hard, I'll see if I get around to it in the next couple of days.

Got my Blitzwolf plugs today. They seem to work well so far. (On/off + current consumption).
With Z2M you can also set the initial power on state: https://www.zigbee2mqtt.io/devices/TS0121_plug.html#power_outage_memory-enum
I guess that's something ZHA could also implement. Since that shouldn't be hard, I'll see if I get around to it in the next couple of days.

Yes, if that’s the case. We currently only support 1 sensor per cluster. This is currently a limitation in ZHA.

Any plans to extend ZHA to support more than one sensor per cluster?

BlitzWolf BW-SHP13 smart plug 16A EU version, same device as TuYa TS0121 smart plug.

Device joins zigbee network and it's recognized by home-assistant.
Entities are created for:

On/Off switch: It works
Electrical power measurement (W): Values updated correctly.
Energy metering (kWh): Entity is created but values are never updated.

This entities are missing:

Current (A).
Voltage (V).

Hope to see full support for these Tuya smart plugs as there are few inexpensive Zigbee plugs featuring power/energy-metering.
Lonsonho also has Tuya TS0121 branded variant which also features support. for power-metering and energy-metering
https://zigbee.blakadder.com/Lonsonho_TS0121.html
https://www.aliexpress.com/item/4001118546894.html
Really like that and BlitzWolf BW-SHP13 because power/energy-metering and fact that they are both inexpensive and very small.

Yes, if that’s the case. We currently only support 1 sensor per cluster. This is currently a limitation in ZHA.

Any plans to extend ZHA to support more than one sensor per cluster?

BlitzWolf BW-SHP13 smart plug 16A EU version, same device as TuYa TS0121 smart plug.

Device joins zigbee network and it's recognized by home-assistant.
Entities are created for:

On/Off switch: It works
Electrical power measurement (W): Values updated correctly.
Energy metering (kWh): Entity is created but values are never updated.

This entities are missing:

Current (A).
Voltage (V).

Hope to see full support for these Tuya smart plugs as there are few inexpensive Zigbee plugs featuring power/energy-metering.
Lonsonho also has Tuya TS0121 branded variant which also features support. for power-metering and energy-metering
https://zigbee.blakadder.com/Lonsonho_TS0121.html
https://www.aliexpress.com/item/4001118546894.html
Really like that and BlitzWolf BW-SHP13 because power/energy-metering and fact that they are both inexpensive and very small.

And this one from "Rehentele" looks to be yet another Tuya TS0121 variant with power-metering and energy-metering support:
https://www.aliexpress.com/item/4000449939329.html
Note that Rehentele seem to sell a TS0121 version with power-metering as well as a TS0111 version without that looks the same.

And this one from "Rehentele" looks to be yet another Tuya TS0121 variant with power-metering and energy-metering support:
https://www.aliexpress.com/item/4000449939329.html
Note that Rehentele seem to sell a TS0121 version with power-metering as well as a TS0111 version without that looks the same.

It shows consumption in kWh (energy metering) as "unknow"... any solution about?

Please, add the missing entities so i can move from z2m to ZHA.

same request from my side, this plug is perfect as size and power capability, so having the chance to see the consumption in ZHA will be the perfect match. I really hope this feature can be implemented, thanks.

That looks like the same "issue" basically all power measuring plugs have (at least the Xiaomi Plugs) (?)

For all other issues (voltage, current not displaying), ZHA would need to be "changed".

What exactly needs to be "changed" in the ZHA integration to overcome the limit of 1 sensor per cluster?
How many sensors per cluster does other Zigbee implementations like example Zigbee2MQTT support?

It would be nice to know if there is a roadmap for this "fix" (more than 1 sensor per cluster) or else we should make a decision to steer away from Zigbee power plugs.

+1. Bought a Tuya TS0121 from aliexpress to measure consumption of different devices.
On/Off switch works and Electrical power measurement (W) updated correctly.
I can retrieve the total usage from the device in the Metering cluster using the current_summ_delivered attribute, but unfortunately there is no sensor for this.

I did some research with my BW-SHP13 in Home Assistant via ZHA:
The current_summ_delivered attribute @rklomp mentioned gets written to zigbee.db. You could use a sql sensor to read the value from there.
You have to multiply that value by 10 to get Wh or by 0.01 to get kWh.
- platform: sql
  db_url: sqlite:////config/zigbee.db
  scan_interval: 30
  queries:
  - name: plug_metering
    query: "SELECT value * 0.01 as value FROM attributes WHERE ieee = 'YOUR_DEVICEID' AND cluster = 1794 AND attrid = 0"
    column: "value"
    unit_of_measurement: kWh

So, the data is there. What needs to be done to show that via the device sensor?
 The Blitzwolf shows 120W while my other plug shows 80W (power measurement). So in that case I would multiply by 0.0066 instead of 0.01 to get a more accurate value, but I'm unsure which device I should trust more. In the end both are very cheap and no high-precision metering devices 😉

I just re-read the issue. It looks like the electrical power measurement (W) is working correctly, right? Only voltage and current do not have entities created for them. The total "smart" (?) energy meter isn't updated.
That looks like the same "issue" basically all power measuring plugs have (at least the Xiaomi Plugs) (?)

Yes, if that’s the case. We currently only support 1 sensor per cluster. This is currently a limitation in ZHA.

FYI, there is a developers/development discussion about this "issue" in this related PR by @stattin42 -> https://github.com/home-assistant/core/pull/50471

While sql sensors or events for attribute changes could be a way to handle multiple attributes per cluster, I am still struggling a little with the rationale behind selecting an optional attribute as the base entity/sensor for a cluster or channel. Wouldn't it be more useful to use a mandatory attribute for the sensor?
Now that things are what they are, I have some sympathy for not breaking things by changing existing behaviour. I cannot help thinking, though, that there is a sense of urgency finding a solution. Hopefully a solution which does not require templating for commonly used mandatory attributes. Support is currently broken for many compliant devices unless they support a certain optional attribute.

You are correct, current_summ_delivered in in zigbee db, BUT it updates there very rarely ( once per day)
I have several different plugs and only one plug, BlitzWolf TS0121, updates correctly, every minute.

Same for me. Two plugs update regular, but four other plugs update only sporadically (sometimes after 5 minutes, sometimes after 30 minutes).

my understanding is that then current_summ_delivered is sent by the device when it reach a certain level of current delivered. So I would suggest to put something consuming more current to see if you see the update arriving more often.

my understanding is that then current_summ_delivered is sent by the device when it reach a certain level of current delivered. So I would suggest to put something consuming more current to see if you see the update arriving more often.

For current_summ I think this makes sense. But I also have the update problem with current_power. And there something like that would be absolutely pointless. For example, I want to be notified when the washing machine is ready and not 30 or even 60 minutes later.

For instant power, my understanding is that there is no reporting for  as the Blitzwolf TS0121 are concerned, and you must poll the information by doing a read attribute as per the time cycle you want

For instant power, my understanding is that there is no reporting for as the Blitzwolf TS0121 are concerned, and you must poll the information by doing a read attribute as per the time cycle you want

How do you read attribute with code? I can do this only from Manage Clusters dialog manually.

Maybe, but actually ZHA/Z2M polls independently. Otherwise no values would arrive, because I have not set this anywhere. Or am I misunderstanding this?
If I use the node "poll state" in nodered, then I get only the state that HA knows. That means: if the plug polls 100W and then 30 minutes no more (although the consumer is off), then the node polls always the 100W.
So I would have to set a poll interval directly in ZHA or what do you mean?

For instant power, my understanding is that there is no reporting for as the Blitzwolf TS0121 are concerned, and you must poll the information by doing a read attribute as per the time cycle you want

Maybe, but actually ZHA/Z2M polls independently. Otherwise no values would arrive, because I have not set this anywhere. Or am I misunderstanding this?
If I use the node "poll state" in nodered, then I get only the state that HA knows. That means: if the plug polls 100W and then 30 minutes no more (although the consumer is off), then the node polls always the 100W.
So I would have to set a poll interval directly in ZHA or what do you mean?

I'm sorry but I don't know how zigpy and zha works. But the way we are doing on other plugins not zigpy related) is to poll every 30s or every 60s , so you can catch if there is something consuming power or not.
Then as mentioned for the current_summation this is a based on some trigger on the device itself.
Blitzwolf is a Tuya based device, and you get what you pay for. If you want a full working plug with instant and summation you can go for Salus SP600 which works very well, with reporting and they are also other brands like Legrand, but the price is somehow more expensive :-(

Hey all, since the new energy dashboard, is this also the topic to see the opportunities to integrate this plug inside that? I'm now trying to do something with a template sensor to get things inside the energy dashboard.

Looks like this might be working soon, see:
https://github.com/home-assistant/core/pull/56909
https://github.com/home-assistant/core/pull/56666

For all users of TS0121 device is ZHA adding the device to summation_delivered channel but no TS0121 user have testing is it working with there devices.
One user have doing it with TS011F and the device is sending values with wrong multiplier and hi is fixing it on the device quirk that is ready for merging in the dev.
If like getting the TS0121 working OK its need testing and feedback from the user for getting it working OK and perhaps updating the quirk for the devices.

Hi, I have 4 of BW-SHP13 and after new HA version, they get new sensor and it shows data. But it show kWH unit, but it number is actually in Wh.

Are you sure the value is Wh and not "10Wh"? On my devices it's "10Wh" so we must divide by 100 and not by 1000. Can you confirm that?

Hi,
maybee it is in 10Wh. I am only shure, it is not in kWh. Because it is connectect to whashing mashine and showing 16 323,0 kWh. And that is too much after 1/2 year of using it...

I have plugs from Schwaiger, Innr SP 120, BlitzWolf, Lonsonho and Salus.
Now they all show smartenergy_metering_summation_delivered, but values are not updating in Lovelace.
electrical measurement utdates frequently but summation_delivery shows reading after restart.
Only Lonsonho plug updates also summation_delivery. I have made restarts, nothing helps.

.

Drilling a bit deeper into the WARNING above, it appears that in configure_reporting_multiple in zigpy/zcl/__init__.py it is expected that the return value from _configure_reporting_multiple may be a list.
However, in the case of the WARNINGs I see, _configure_reporting_multiple returns a one element nested list with [[ConfigureReportingResponseRecord(status=0)]] which does not go very well with the condition:
if (
            isinstance(res, list)
            and not (res[0].status == foundation.Status.SUCCESS and len(res) == 1)
            and len(res) >= 0
        ):

@Adminiuga, is this specific to my devices or more general? Should a separate issue be created for this?

Do you have same log, but as instead of zigpy_deconz.uart include logging for zigpy_deconz.zigbee.application and zigpy_deconz.api

I update HA to 2021.11. The BW-SPH13 has now another sensors - great!
But i think, that sensors electrical_measurement_rms_current and total comsumption has to be divided by 1000.
It show, tahat my Washingmachine consume 2.400 W (correct ) and 10.000 A (that is is not correct).
Correct is 10 A.

Is your energy sensor kWh updating correctly?
My BlitzWolf plug updates kWh sensor only once when restarted and is freezed after that.

It's updatete with no problem.
This is chart of my Washing Mashine cycle.

If there is small load, it change after every 1 or 10 W or so. The kWh shoud be divided by 1000 or 100 (I have no energymeter to test it)

@Celer21 please try if reconfiguring your plug fixes total consumption 😃

Works as expected since last HA update! Consumption (kWh) seems still not reliable

I can confirm now BlitzWolf plug kWh is updating, but Shwaiger and Innr SP120 are still freezed.

For Innr SP120 reconfigure fails

logInnrSP120.txt

#1131 adds the correct divisor for rms_current.
The devices likely need to be reconfigured to display the correct values.

#1131 adds the correct divisor for rms_current. The devices likely need to be reconfigured to display the correct values.
Yes, for a different device I had to re-set up the zigbee device in HA to get the new scaling.

In which standard version of HA will it be distributed? I only have one instance of HA and cannot use dev or beta channel.
Thanks!

In which standard version of HA will it be distributed? I only have one instance of HA and cannot use dev or beta channel.

No one can tell until a new release and it has been bumped in HA so keep an eye on https://github.com/zigpy/zha-device-handlers/releases in combination with "Bump ZHA quirks version" in https://github.com/home-assistant/core/pulls?q=zha

@Hedda Thank you for links.

Hi everyone,
i think this is not fixed yet? The current is showing wrong values.
Here yu have an example:

So in fact the current requires a divider of appr. 1.000 - in this case 224V multiplied with 662A would mean 148,29kw - I am not charing an electric car here ;-)
With the correction it would be 148Watts so way more reasonable and especially the consumption would be better.
cheers
zhaquirks.tuya.ts0121_plug.Plug
{   "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.Router: 1>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress|RxOnWhenIdle|MainsPowered|FullFunctionDevice: 142>, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False)",   "endpoints": {     "1": {       "profile_id": 260,       "device_type": "0x0051",       "in_clusters": [         "0x0000",         "0x0004",         "0x0005",         "0x0006",         "0x0702",         "0x0b04"       ],       "out_clusters": [         "0x000a",         "0x0019"       ]     }   },   "manufacturer": "_TZ3000_g5xawfcq",   "model": "TS0121",   "class": "zhaquirks.tuya.ts0121_plug.Plug" }

Hi everyone, i think this is not fixed yet? The current is showing wrong values.
Here yu have an example:
So in fact the current requires a divider of appr. 1.000 - in this case 224V multiplied with 662A would mean 148,29kw - I am not charing an electric car here ;-) With the correction it would be 148Watts so way more reasonable and especially the consumption would be better.
cheers
zhaquirks.tuya.ts0121_plug.Plug { "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.Router: 1>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress|RxOnWhenIdle|MainsPowered|FullFunctionDevice: 142>, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False)", "endpoints": { "1": { "profile_id": 260, "device_type": "0x0051", "in_clusters": [ "0x0000", "0x0004", "0x0005", "0x0006", "0x0702", "0x0b04" ], "out_clusters": [ "0x000a", "0x0019" ] } }, "manufacturer": "_TZ3000_g5xawfcq", "model": "TS0121", "class": "zhaquirks.tuya.ts0121_plug.Plug" }
Update: I fix it with a simple template right now but maybe someone wants to fix the zhaqurik:

Have you tried reconfiguring the switch in the device page? I did that with the switches that I had previosly paired and it fixed all the wrong values.

My plug shows wrong current value too.
I am running HA 2021.12.10 so maybe the patch will be included in next release?
Is it enough to reconfiguring device? Or is it necessary to remove from ZHA and repair it again?

Hi everyone, i think this is not fixed yet? The current is showing wrong values.
Here yu have an example:
So in fact the current requires a divider of appr. 1.000 - in this case 224V multiplied with 662A would mean 148,29kw - I am not charing an electric car here ;-) With the correction it would be 148Watts so way more reasonable and especially the consumption would be better.
cheers
zhaquirks.tuya.ts0121_plug.Plug { "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.Router: 1>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress|RxOnWhenIdle|MainsPowered|FullFunctionDevice: 142>, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False)", "endpoints": { "1": { "profile_id": 260, "device_type": "0x0051", "in_clusters": [ "0x0000", "0x0004", "0x0005", "0x0006", "0x0702", "0x0b04" ], "out_clusters": [ "0x000a", "0x0019" ] } }, "manufacturer": "_TZ3000_g5xawfcq", "model": "TS0121", "class": "zhaquirks.tuya.ts0121_plug.Plug" }
Update: I fix it with a simple template right now but maybe someone wants to fix the zhaqurik:

Have you tried reconfiguring the switch in the device page? I did that with the switches that I had previously paired and it fixed all the wrong values.

sure multiple times. its still showing crazy currents and wrong consumption - real consumption is about 0,33kwh ... ;-)

Have you tried reconfiguring the switch in the device page? I did that with the switches that I had previously paired and it fixed all the wrong values.

sure multiple times. its still showing crazy currents and wrong consumption - real consumption is about 0,33kwh ... ;-) There is just my work laptop running on this switch

What values do you get for attributes ac_current_divider and ac_current_multiplier in the TuyaZBElectricalMeasurement cluster?

I think repairing its needed or reading the "new" added (in the quirk) attribute then its null  if not being populated and then ZHA is not using them.

Hi all,
the cluster attributes where Null so i selected the right divisor/multiplier and then it changed to a reasonable value:

Thanks a lot!

In which standard version of HA will it be distributed? I only have one instance of HA and cannot use dev or beta channel.

No one can tell until a new release and it has been bumped in HA so keep an eye on https://github.com/zigpy/zha-device-handlers/releases in combination with "Bump ZHA quirks version" in https://github.com/home-assistant/core/pulls?q=zha then wait for it to appear under "All changes" in https://rc.home-assistant.io/latest-release-notes/

FYI: The new version of zha quirks (0.0.66) is included in the HA 2022.02 release :)

My device total usage readings (metering_summation_delivered) still appear to be wrong in HA (after reconfigure). I don't know what steps to take to fix this?
Amperage, voltage and wattage readings all seem fine, but summation shows a lot of kWh's which should be Wh i think. So probably a wrong multiplier somewhere?
metering_summation_delivered:

Energy dashboard - individual device:

My device total usage readings (metering_summation_delivered) still appear to be wrong in HA (after reconfigure). I don't know what steps to take to fix this?

Try setting the values of the attributes divisor (0x0603) and multiplier (0x0602) in the TuyaZBMeteringCluster cluster. Divisor should be 100 and multiplier should be 1.

My device total usage readings (metering_summation_delivered) still appear to be wrong in HA (after reconfigure). I don't know what steps to take to fix this?

Try setting the values of the attributes divisor (0x0302) and multiplier (0x0301) in the TuyaZBMeteringCluster cluster. Divisor should be set to 100 and multiplier should be set to 1.

Those are already set to 100 (divider) and 1 (multiplier), but I set them both anyway and now the values seem much more reasonable! Thanks! 😊

My device total usage readings (metering_summation_delivered) still appear to be wrong in HA (after reconfigure). I don't know what steps to take to fix this?

Try setting the values of the attributes divisor (0x0302) and multiplier (0x0301) in the TuyaZBMeteringCluster cluster. Divisor should be set to 100 and multiplier should be set to 1.

Those are already set to 100 (divider) and 1 (multiplier), but I set them both anyway and now the values seem much more reasonable! Thanks! 😊

I tried the same and it also fixed the issue for me completely!

Hi, can I ask you. Is it possible to delete sum consumption (power)? I have Zigbee2mqtt.

Hi, can I ask you. Is it possible to delete sum consumption (power)? I have Zigbee2mqtt.

@radek-foltyn  zha-device-handlers (zha-quirks) is for Home Assistant ZHA, not for Zigbee2MQTT. For Zigbee2MQTT see instead:
https://github.com/Koenkk/zigbee2mqtt/issues/
https://github.com/Koenkk/zigbee2mqtt/discussions


Fix deploy for last Elixir versions and update README

@sylane Why was it necessary to add Distillery? It's surpassed by the built-in release handling in Mix itself, so in my opinion it should not be used.

@sylane Ping!


DPL-541 Bioscan OKR: MVP Tag plate creation

User story
As R&D we need an MVP way to create the 24 x 384-well tag plate variants required for a month's set of work on the Bioscan project.
Who are the primary contacts for this story
Scott T
Emma B
Andrew S
Who is the nominated tester for UAT
Scott T
Acceptance criteria
To be considered successful the solution must allow:

[ ] New tag layout 'direction' option to get tag 2 indexes in the correct order to match Scotts requirements. Create a variation of the tag layout "InColumnsThenRows" with the tag 2 index calculation changed to (column % scale) * scale + (row % scale)
[ ] Script to generate the forward (1 x 96 tags) and reverse (24 x 4 tags) tag groups and the 24 tag layout templates using the tag sequences as supplied by Scott. Include a version number in the names so we can regenerate if required. Consider how the tag sequences could be imported into the script. Maybe as files? Consider that a new master batch of tags may include a different layout or different sequences in the tags.
[ ] Script to create a set of 24 x 384-well tag plates (with a list of the 24 barcodes as output) from the 24 tag layout templates. The output list of barcodes can then be used to manually print off the tag plate barcodes as required. If possible use Baracoda to prefix the barcodes with the tag layout template variation index (1-24) to aid the user (e.g.  BSTG-01-nnnnnnnn to BSTG-24-nnnnnnnn). Also consider having this creation script annotate the plate with the tag layout template variation index (1-24) and tag supplier batch reference to aid the user (perhaps as comments visible in Limber similar to the metadata being stored for the LEAN work).

Dependencies
None
References
This story has a non-blocking relationship with:
Other Bioscan OKR stories.
Additional context
See the tag plate creation tab in this document.
See the research story comments here https://github.com/sanger/General-Backlog-Items/issues/219 for info on the new 'direction' algorithm required.

Tag plates do not contain aliquots in their wells. The plate itself has a link to the tag layout template.
To determine which template was used when a tag plate was created:
See qcables table, which links asset (plate) to the lot id. When you create a tag plate, you create a qcables row.
Then see the lots table which links the lot to a tag layout template id.
Then see tag_layout_templates which links the template to the tag groups for tag 1 and 2, and to the direction and walking algorithms to use.
So when the tag plate is used later, the correct tags are applied to the aliquots in the wells which are created to hold the samples transferred into the tag plate from the parent (tag plate is now the child plate).


[INLONG-10360][Manager] Combine schedule state transition with group operations

Fixes #10360
Motivation
Combine schedule state transition with group operations. details see description in #10360
The finite state machine for schedule is

This is the transient status of the schedule info.
With specified operations, the status will change to corresponding value.

Status
Operation

NEW
inlong group created with schedule info

APPROVED
the new inlong group approved by admin

REGISTERED
schedule info registered to schedule engine

UPDATED
update schedule info for a group

DELETED
delete a group

Modifications

add schedule info in InlongGroupRequest
add configuration inlong.schedule.engine to indicate the schedule engine type
replace the implementation QuartzOfflineSyncJob with  the interface Job when building quartz job
add ScheduleOperator to process all the schedule related operations
register schedule info to schedule engine after group approved
save schedule info when group created
update schedule info to schedule engine when group updated
delete schedule info when group deleted

Verifying this change
(Please pick either of the following options)

[x] This change is a trivial rework/code cleanup without any test coverage.

Documentation

Does this pull request introduce a new feature? (no)

I resolved your comments, PTAL again @fuweng11


Found endless loop condition in phase 3

I have been trying to improve the code a bit for rotational drives, played a bit with the buffer sizes and I am not sure if my changes are causing it atm, or if I just hit the right settings to trigger a bug. Sadly I dont totally understand the whole algorithm yet and I am not a c++ developer, so its utterly painful for me, to read through the code.
But I noticed the same behaviour with the unchanged code as well, sometimes, that in phase 3 the process goes up to 100% on a single core, no further IO and keeps just spinning.
What happens is, that it reads an entry pos, that doesnt equals current pos,
the value gets cached and current pos gets increased till both match, but in this case, the entry_pos is already smaller than the current_pos.
Why it happens, I cannot say, but this case is not handled, not throwing any error and causing this endless loop.
first line 261 gets triggert, till current pos becomes bigger than greatest pos  which gets checked in line 235 and totally stops the whole reading process.
I have right now a version that consistently creates this case and I will try to see, how this value gets written there, but as I said, I don't fully understand yet, how everything is working, so if anyone would like to get in contact and help, that would be totally appreciated.

Ok, so I found why i constnaly caused this issue in my code. I added async reading to the buffered disk, so it would already preload the next buffer, while its processing the old data. But the special case in phase 2 table 7, where it reads and writes to the same file, caused seeks to execute, while it was reading or writing.  So data got written into the wrong position. At least this is my guess. I added a muted to the buffered disk and locked the internal read and writes and it went through without any issues.
So maybe there is nothing wrong with the original code and it the hanging processes might still be hardware related, but I think the condition described, is still what keeps the process stuck in an endless loop. wort to put at least a check and throw an exception, that data was inconsistent. It cannot hurt and if  I am right people would at least get some feedback, that something is wrong.
I had an issue with my memory before on unchanged code. It was set to vendor spec speed  3200, but got my processes stuck, then i underclocked it to 2666 and had no issues.
If I will find more infos, I will let you know.

Interesting. I have observed the same issue on my two plotters as well. Both AMD ryzen machines with 3600 MHz RAM Speed running Linux. One has cheap RAM and on this machine the plotting process got stuck in phase 3 with one core at 100% usage frequently (about 1 in 4 plots). I suspected a RAM issue and clocked it down to 3200 MHz which seamed to have worked, but it still happens from time to time (once every 2 days).
The other machine with the expensive RAM works fine at 3600 MHz most of the time, but it has happened twice to me here as well (done about 600 plots with this one so far).
I have switched to a custom version of chiapos (https://github.com/pechy/chiapos/tree/combined) about 2 weeks ago which I merged with 1.0.2. The observed behaviour of processes getting stuck in phase 3 happened with the original code and this modified version likewise, so I'd still say it is a RAM issue.

Closing issue - feel free to re-open if you have more actionable data. PRs are also welcome


Helping with how-tos

How-tos are goal oriented guides that solve a specific problem.
Throughout the years our community has solved many problems such as: firebase/keycloak integration, sending email with plpython, using Caddy with PostgREST, custom Nginx logic, etc. However these solutions remain undocumented for others to use and improve.
Recently I've added a how-to section in our docs, we can link these solutions there as how-to guides.
The guides can be part of our docs and you can credit yourself(url to github, website, etc) in the author part or they can also be in the form of a blog post of your personal website.
There are issues already labeled as how-tos so if you'd like you can pick one of those as a starter. I'll help reviewing it in a PR.

Seeing the low engagement on this issue(quite a while now), I'll close/unpin.


CA-63 Add LinkedIn Ads template support

TL;DR
Freshpaint GTM Template support for the LinkedIn Ads destination.
Summary

Accepts up to 3 Conversion IDs, as do the two community gallery templates (one published by LinkedIn).
The implementation here is to send up to 3 track calls, one for each Conversion ID - the same as the aforementioned gallery templates do (in their case, they're doing a LinkedIn tag call instead of track)
This is a different use case than multi-config, where the config param differs (e.g., Facebook Pixel ID, GA4 Measurement ID) - for those, we use multiple dest instances, and specify those in the integrations object.

Testing notes
Tested each of the following permutations, observed correct track call payloads:

[x] Single Conversion ID
[x] Multiple Conversion IDs (up to 3), with and without embedded spaces
[x] Too many (4 or more) Conversion IDs: OBSERVE console error message
[x] E2E test to Event Verification - OK (both required fields sent)

Screenshot

LGTM, is there a reason we only support three conversion IDs?

because that's what the gallery templates support, which we'd be migrating from. I think they chose 3 to to have some kind of limit to avoid possible throttling.


Update engine revision to pull in AOTC build fixes.

Bots are still working on that SHA. Will commit once those are green.
cc @abarth @jason-simmons

lgtm

This will break AOT builds because BuildApp isn't marked executable.

We should update to https://github.com/flutter/engine/commit/e0c48bc4e240e6c919c7b25b2c95aa93de905764

Rebased to that revision.

I've updated the engine past this rev.

Thanks Adam


[super_editor] While performing a hot reload and changing the position of the SuperTextField in the widget tree, an exception is thrown

During development if the SuperTextField is moved around the widget tree, and a hot reload is issued, a flutter assertion is thrown:
The following assertion was thrown during a scheduler callback:
This widget has been unmounted, so the State no longer has a context (and should be considered defunct).

Consider canceling any active work during "dispose" or using the "mounted" getter to determine if the State is still active.

I believe the issue is here, in reassemble:
https://github.com/superlistapp/super_editor/blob/5227a938611632f3241a6cca36c4d5f9833b26f8/super_editor/lib/src/infrastructure/super_textfield/android/android_textfield.dart#L237-L241
I'm not sure if text fields other than android also have this problem.

@jellynoone can you provide the minimal code to cause this error?

I'll try to provide the minimal code sample in a short time, but until then I want to clarify, that the movement of the widget I was referring to in the report is manual, i.e. if I go from:
Widget build(BuildContext context) {
  return Column(children: [
    SuperTextField(),
  ]);
}

to:
Widget build(BuildContext context) {
  return Row(children: [
    SuperTextField(),
  ]);
}

and then perform a hot reload.
I think the reason this issue hasn't come up yet might be because developers provided a GlobalKey on the SuperTextField and so such reparenting wasn't disposing the widget as it does in this case.

The minimal code sample:
import 'package:flutter/material.dart';
import 'package:super_editor/super_editor.dart';

void main() {
  runApp(const MainApp());
}

class MainApp extends StatelessWidget {
  const MainApp({super.key});

  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      home: Scaffold(
        body: SafeArea(
          child: Column(mainAxisAlignment: MainAxisAlignment.center, children: [
            SizedBox(
              height: 30,
              width: 300,
              child: ColoredBox(
                color: Colors.black,
                child: SuperTextField(),
              ),
            ),
          ]),
        ),
      ),
    );
  }
}

Steps to reproduce:

Run app in debug mode on Android
Change Column into `Row``
Save / trigger hot reload
Observe the assertion

Thanks. I was able to reproduce the bug, and I confirmed that checking the mounted state seems to resolve it. I'm going to put up a PR. The PR will include sweeping changes to our post frame callback behavior so that all of them check mounted automatically.


[BUG] enable storage-network, Networks statistics are incorrect

Describe the bug

To Reproduce
Steps to reproduce the behavior:

go to setting page
enable storage-network

Expected behavior

The network created by storage-network should not be counted
Support bundle

Environment

Harvester ISO version:
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630):

Additional context
Add any other context about the problem here.

Verified this bug has been fixed.

Test Information

Environment: baremetal DL360G9 4 nodes
Harvester Version: v1.1.0-rc3
ui-source Option: Auto

Verify Steps:

Install Harvester with any nodes
Follow Steps to reproduce in https://github.com/harvester/harvester/issues/2952#issue-1410829036


Online documentation out of date with npm module

It has been 11 months since the last npm publish according to the npm registry page.
The documentation page lists an isImmutable method for checking if an object is an Immutable object.
This method does not exist anywhere in the published npm module.
Is there a reason that the docs and released module are out of sync?
If this basic method is missing, what else is present in the documentation but absent from the published module?

The isImmutable() method listed in the docs threw me off as well. Docs should be updated accordingly.

Can I suggest someone put the Build Version in the documentation. It would be helpful to have some idea what version the documentation is in reference to. Thank you for your hard work, this product is  exceptional.

This issue is quite annoying. I thought isImmutable was in v.3.8.1 because docs do not specify which build is that included in.

+1 docs are incredibly confusing atm

+1 Build versions aren't absolutely needed, but the docs should be accurate for the latest release instead of partially updated for the release candidates.

Great library (thank you), but also just killed quite a bit of time reaching the same conclusion.

I've updated the website build script to include a version number and only build when we push a new version.
The website will always reflect the most recent version pushed, even if that is a release candidate.


Type bug on _purchases.purchase_items_create_params unit_amount

Issue:
When utilizing the Node Chargebee v2 Purchases API the types from chargebee-typescript suggest that unit_amount type should be  number. However, upon testing, we have encountered an error response suggesting that the unit_amount type should be number[] instead.
Error Response:
	purchase_items[unit_amount][0] : wrong format

This error response indicates a discrepancy between the expected and actual data type for unit_amount in the _purchase.create_params object. The API documentation states that unit_amount should be of type number, but the error message implies that it is being interpreted as an array of numbers (number[]). The API call returns no error when the parameters for unit_amount are of type number[].
Expected Behavior:
The API should accept unit_amount as a number type as per the documentation, and error messages should accurately reflect any issues related to the format or type mismatch.
Impact:
This issue prevents us from properly utilizing your chargebee-typescript types. It hinders our ability to rely on your typescript definitions for API calls.
Steps to Reproduce:
Versions:
"chargebee": "2.31.0",
"chargebee-typescript": "2.29.0",

Make a call to the Chargebee Purchases API to create a purchase with the unit_amount being set to a number.
Observe the error response indicating an incorrect format for unit_amount.

Additional Information:
We have verified that the data being sent to the API conforms to the documented specifications.
This issue occurs consistently across multiple API calls and environments.
No recent changes have been made to our integration setup that could have caused this behavior.
Request:
We kindly request your prompt attention to this matter. Please investigate and resolve this discrepancy in the Chargebee API as soon as possible. If there are any updates or workarounds available, we would appreciate receiving guidance on how to proceed in the interim.
Thank you,
Ash,
Launched

Hi @First-Derivative, sorry for the delay in looking into this. I'm not sure if I understand this correctly. Could you share a code snippet that results in the error
purchase_items[unit_amount][0] : wrong format

Also, I see you've mentioned both chargebee and chargebee-typescript versions. Are you seeing this error in both the SDKs?

@First-Derivative — can you please confirm if you're still having this issue and share more details about it?

Hi @cb-sriramthiagarajan,
I can confirm this is no longer an issue. Most likely your API updated on on the backend and this error was resolved.

Thank you @First-Derivative!


introduce signals

jangan di merge dulu bang

changes:

using signal to trigger initial dialog
fixed an issue where the dialog is not continued when the typing ends

monggo kalo mau merge


Prohibit image usage in local backend (v3)

Clear and concise description of the problem
Currently, image clause is used as a path to the executable.
Suggested solution
Use entrypoint or introduce new special clause for Local backend.
Alternative
Get rid of image and use only commands.
Additional context
https://github.com/woodpecker-ci/woodpecker/discussions/3263#discussioncomment-8233224 (point N3)
Validations

[X] Checked that the feature isn't part of the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

This needs a depreciation first, can't be removed in 3.0 directly

Sad, but OK.
The main point is to get rid of confusing image. Do something like environment was removed from plugins and schema was corrected.
So, there is entrypoint for shell and commands for whatever else, then.

https://docs.drone.io/pipeline/exec/syntax/steps/

Sadly, it has been told a long ago, even initial PR:
https://github.com/woodpecker-ci/woodpecker/pull/709#issuecomment-1046150138
https://github.com/woodpecker-ci/woodpecker/pull/1565#issuecomment-1427135440, bullet point 1


Gradlew and unit tests

100% pass and code coverage for JALSEBuilder.
Also did some gradle stuff to add a wrapper and added junit dependency

I think the default formatting for gradle entries is 4 spaces instead of a tab. May want to fix up that first commit.
Any idea why the second one isn't building?
#70 in the pipeline!

i think this is the ticket

oh, wait i notice the first merge, i did a copy of much of it, so we should discard this pull request and ill just commit the unit test and the refactoring


[Feature] Kernel iter head

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
Add kernel_iter_head which is a roi head in K-Net
Modification
support K-Net
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: FANG-MING:x: meng976537406You have signed the CLA already but the status is still pending? Let us recheck it.

Need to resolve conflicts and update the code, clean unnecessary code.


removing 3 and 4 years from validity range options

Coverage remained the same at 59.534% when pulling 972051a61eb8dd9e1c60c5de9ad5e3aeef860eb6 on hosseinsh:cert-year-validity into 9b2a1bebd13f6b028699afca366ed3d77695a0d4 on Netflix:master.