[RFC] Full Cluster Replication with Failover &  Failback support

What problems are you trying to solve?
Cross cluster replication, as it stands today, allows user to setup continuous replication on a set of indices from one cluster(leader) to another(follower). This happens right now at index level where customer can choose to enable replication on a particular index/pattern.
We rely on a logical replication model where each shard on follower index polls the leader shard for new changes(i.e. translog) which it then replays locally.
Problem statement: While the current implementation works fine, it is not feature complete. From a users point of view, we need cluster level replication with support for Failover/Failback. This proposal lists down the high level approach and the plan for achieving Full Cluster Replication with Failover and Failback support.

Why should it be built? Any reason not to?
These are the major use cases for Full Cluster Replication:

Disaster Recovery: Full Cluster Replication can be very useful for building tolerance against outages or complete failure of OpenSearch clusters. With Full cluster replication, user can setup backup clusters that continuously replicate data from the primary cluster and are ready to serve the production workload in case of Primary cluster outage.
Data Proximity: For search use cases, a single centralised cluster serving requests for end users across multiple regions increases the end to end latency . With Full Cluster Replication, cluster owners can configure multiple read-only clusters in closer proximity to the end users thus enhancing the overall customer experience.

What are you proposing?
We propose building support for Full Cluster Replication on OpenSearch. During Replication, Secondary clusters would periodically poll the primary for data (indices, index settings, index mapping, cluster settings, cluster state, etc). Duration for poll should be small enough to ensure the minimal lag(e.g. 1 min) in propagating these changes to the Secondaries. Incase of remote store, this flow can be further optimised such that all secondary clusters can fetch the data directly from remote store thus reducing the overhead of replication on leader.

Apart from customer data, we'd be replicating system indices as well. We'll work with other plugins(Dashboards, Security, ISM, etc) to ensure integration with Full Cluster Replication. For example, security plugin on secondary clusters should be able to work as expected with the replicated security plugin metadata from Primary.
Handling Failover/Failback:
Failover and Failback will be a manual customer invoked action. We can be further explore automating it via an external process but it's not planned as of now.
Individual clusters would have the capability to switch its role between Primary/Secondary but won’t be smart enough to do this switch automatically.
Why not automatic failover? To achieve auto-failover internally we'll need to establish a quorum between the Primary and Secondary clusters which will significantly increase the complexity of the replication process. On top of that, Replication will be more prone to failure in outages like network partition, as clusters might not be able to establish the quorum and thus fail to do the failover(similar to split-brain).

What is the user experience going to be?
User stories:

As a cluster admin, I want to continuously replicate all the data from my Primary cluster to multiple Secondary clusters so that my end users experience low search latency by requesting data from the nearest cluster.
As a cluster admin, I want to continuously replicate all the data from my Primary cluster to multiple Secondary clusters so that I  have backup cluster ready to serve traffic in case of any outage on Primary.
As a cluster admin, I want to switch the Replication Configuration so that I can promote one of the Secondary cluster as the new Primary incase of Failover/Failback.

Note: While we'll support multiple secondary clusters replicating from a primary, we won't support chained replication(secondary cluster replicating to another cluster)

Other dependencies:
Choice of Replication model: Although the choice of replication model shouldn't affect Full Cluster Replication design drastically, it'll still require us to build support for multiple scenarios and we might want to restrict the choice of replication model for Full Cluster Replication. The preference for replication model is mentioned below(highest on top).

Segment replication with remote store
Segment replication without remote store
Logical replication

Note: The preference above is not backed by any benchmark but rather based on our performance expectations from prior experiences. We'll cover this in more detail during design.

Thanks for the great writeup @ankitkala
Few open questions

Does the follower cluster need to have exact same topology configuration as leader cluster?
Similar to 1, what about version constraints and other configs? Are they expected to match? (I understand this will open up problem up version upgrades coordination)
Are there sample APIs if already thought about to demonstrate the user workflows?
Can the replication direction switch back and forth when the failover happens?

Does the follower cluster need to have exact same topology configuration as leader cluster?

It is recommended (at least for DR cases) but we might not be able to enforce it. At the least, secondary should have enough space to accommodate the Primary's data.

Similar to 1, what about version constraints and other configs? Are they expected to match? (I understand this will open up problem up version upgrades coordination)

Depends on choice of replication, For SegRep based approaches, we'll need to enforce that the version matches across Primary and secondary.

Are there sample APIs if already thought about to demonstrate the user workflows?

We need to discuss those as part of design and probably can then come up with sample APIs.

Can the replication direction switch back and forth when the failover happens?

Yes. Replication direction has to switch back and forth during failover and failback.

Lets add further details on -

For Data proximity case,

Additional details on support of secondary clusters replicating from multiple primary clusters.

How will the support of failover looks in this case?

Can we add expectation on achievable(/target) RPO and RTO at cluster-level with this feature?

What is the user experience going to be?
User stories:

This should be "As a cluster admin, I want to continuously replicate all the data from my Primary cluster to multiple Secondary clusters so that my end users experience low search latency by requesting data from the nearest cluster."

This is a very detailed proposal. Thanks @ankitkala! I have a few comments/quesitons:

Will this be enabled on the replica cluster? When a replica cluster is setup, is there a cluster wide setting we can use as an indicator in the UX that it is acting as a follower cluster?
With remote store enabled on the primary, will the recovery point objective (RPO) be 0?
From a UX perspective, we should plan a UI in the management tools section of Dashboards to manage CCR.
If a cluster is set to primary and the previous cluster that was a primary comes back online, will the new primary ignore it?
How do we reconcile multiple followers when the primary changes?
In the automatic failover, I think it is worth calling out one other aspect that makes this challenging is that traffic, load balances, and other systems will need to be rerouted which isn't managed by the OpenSearch process.

Can we add expectation on achievable(/target) RPO and RTO at cluster-level with this feature?

Yep. I deliberately not mentioned it here as RPO will heavily depend on how we want to design the complete solution. I'll update those here after we have more clarity on what we want to support.
RTO might be even more tricky given that we want to start with manual Failover/Failback

One other question is that there seems to be two features 1/ enabling full cluster replication and 2/ failover/failback. Do we want to split these into two releases since they are two features? Would that help accelerate releasing the full cluster replication capability?

This is very thorough. Looking forward to a proposal for APIs.

Need a section on conflict resolution, especially in failover scenarios. If the leader cluster is failing, you may fail over, then fail back, and end up with different data on the two clusters.
Consider whether replication may want to expose replication write concerns to the write APIs (e.g. return when "majority" or "at least 1" of the replicas has received/written/committed data).

Will this be enabled on the replica cluster? When a replica cluster is setup, is there a cluster wide setting we can use as an indicator in the UX that it is acting as a follower cluster?

It'll have to be introduced. How we're thinking of this is that cluster can behave in 2 mode(Primary and Secondary). We'll definitely need to provide that visibility to the customers.

With remote store enabled on the primary, will the recovery point objective (RPO) be 0?

We'd want to be as close to 0 as possible. We can revisit this we have basic high level design ready.

From a UX perspective, we should plan a UI in the management tools section of Dashboards to manage CCR.

Agreed.

If a cluster is set to primary and the previous cluster that was a primary comes back online, will the new primary ignore it?
How do we reconcile multiple followers when the primary changes?

We'll need to have a simplistic version of primary term to deterministically identify who is the new Primary.

In the automatic failover, I think it is worth calling out one other aspect that makes this challenging is that traffic, load balances, and other systems will need to be rerouted which isn't managed by the OpenSearch process.

Incase of DR, we're not thinking about distributing the load in any way. Instead, Secondary would be on complete standby .

Thanks for the thorough proposal.

Individual clusters would have the capability to switch its role between Primary/Secondary but won’t be smart enough to do this switch automatically.
Does the customer need to switch their indexer (indexing operations) to the secondary cluster manually? How to handle failed indexing operations during the primary's cluster outage.

Does the customer need to switch their indexer (indexing operations) to the secondary cluster manually? How to handle failed indexing operations during the primary's cluster outage.

Yes, Customer has to manually failover to one of the secondary cluster. There is expected to be some data loss in case of failover due to an outage(based on our RPO guarantees). Post failover, customer can continue indexing the data to the follower cluster.

Greetings,
Any update on this ?
Is there any progress related to RPO and RTO ?
Thank you.


Fix a typo in README.md

'Version' was written 'Verison'

Hi @brunosaboia, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. Real humans will now evaluate your PR.

TTYL, MSBOT;

Thanks for catching that @brunosaboia!

No problem @tomjebo, thank you for the excelent work with OpenXML SDL 👍


Support distance as user-facing similarity metric

fix for failing test ClientClassificationsTest is introduced in PR https://github.com/semi-technologies/weaviate-java-client/pull/111


@material-ui/pickers sandbox not working

https://codesandbox.io/s/8gpxl

Per the codesandbox description:

https://github.com/mui-org/material-ui/blob/master/docs/src/pages/components/pickers/MaterialUIPickers.js

I believe this issue should be moved to the main repository, and appears to be a duplicate of https://github.com/mui-org/material-ui/issues/19438.


Redshift Feature - Support unload/copy paradigm

Feature
Support inserting data to Redshift tables using unload and copy
Feature description
Despite the recommended process of using:
INSERT INTO table (SELECT * FROM tmp_table)

when transferring data from one table to another, we've found that the performance of Redshift inserts suffers heavily at scale. Rather it becomes much faster to unload the data to s3 first, followed by copying it back into the table using the Redshift unload/copy commands. Of course this shouldn't be the default behavior, but I think it would make sense to allow the user to choose to use this paradigm if it benefits them.
I feel like this could be done fairly easily by allowing the user to configure s3 access parameters in their profiles.yml, and maybe the upload path and any other parameters could be set within the model configs?
Who will this benefit?
Redshift users processing enough data to be able to take advantage of the parallelization provided by the unload/copy commands.

Flagging that Mamykin has documented a process for copy commands via DBT for s3 and snowflake. Not sure why we're not able to enable a similar workflow for Redshift?

Following up, confirmed that COPY also works as a post hook operation, and you can point to s3 as the source. For example:
on-run-end:
  - "truncate table dbt.sales"
  - "copy dbt.sales from 's3://analytics/sales/sales.csv' credentials 'aws_iam_role=arn:aws:iam::XXXX:role/XXXX' delimiter ',' removequotes IGNOREHEADER 1"
  - 'GRANT USAGE ON SCHEMA "dbt" TO GROUP ro'
  - 'GRANT SELECT ON ALL TABLES IN SCHEMA "dbt" TO ro_username'


OSError: libcudart.so.7.0: cannot open shared object file

I am using CentOS 7 with cuda 7.5 and cudnn v5.  Openblas is used as the blas library and opencv was installed using yum.  some of configure setting related to these is:
ADD_LDFLAGS =-L/usr/local/cudnn-v5/lib64 -L/usr/lib64
ADD_CFLAGS =-I/usr/local/cudnn-v5/include
USE_CUDA = 1
USE_CUDA_PATH = /usr/local/cuda
USE_CUDNN = 1
USE_BLAS = openblas
USE_OPENCV = 1

make -j32 gives no error message.  But when I try to run the code in the example/image-classification directory, it give the error:
 python example/image-classification/train_mnist.py
Traceback (most recent call last):
  File "example/image-classification/train_mnist.py", line 1, in <module>
    import find_mxnet
  File "/home/jdhao/local/mxnet/example/image-classification/find_mxnet.py", line 2, in <module>
    import mxnet as mx
  File "/home/jdhao/.local/lib/python2.7/site-packages/mxnet-0.5.0-py2.7.egg/mxnet/__init__.py", line 7, in <module>
    from .base import MXNetError
  File "/home/jdhao/.local/lib/python2.7/site-packages/mxnet-0.5.0-py2.7.egg/mxnet/base.py", line 43, in <module>
    _LIB = _load_lib()
  File "/home/jdhao/.local/lib/python2.7/site-packages/mxnet-0.5.0-py2.7.egg/mxnet/base.py", line 35, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/home/jdhao/anaconda2/lib/python2.7/ctypes/__init__.py", line 440, in LoadLibrary
    return self._dlltype(name)
  File "/home/jdhao/anaconda2/lib/python2.7/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcudart.so.7.0: cannot open shared object file: No such file or directory

And yes, I have added the cuda runtime library to my .bash_profile, it is
 export LD_LIBRARY_PATH="/usr/local/cuda/lib64:$LD_LIBRARY_PATH"

Is there anything wrong with my compilation or settings?

you can try the following script from bash
ldconfig /usr/local/cuda/lib64

@wshenx , I  tried, but still got the same error message. Where I can find a complete config.mk file? Maybe that will help me find what is wrong.

if you are using cuda 7.5, shouldn't it be libcudart.so.7.5?

@csucat Well, I don't think it has anything to do with your compilation since make reports no error.  As @Ldpe2G suggests, it is strange that python requires the 7.0 version of the lib instead of 7.5. You can check that first.

@csucat At a second thought, if you are using the latest Mxnet, the python version should not be version 0.5. Maybe 0.7 or higher, I'm not quite sure. I think you previously installed a older version of mxnet python wrapper and at that time cuda 7.0 was installed on your system. So I suggest you to reinstall your python wrapper again so your site-packages can be updated.

@wshenx yeah, I have installed mxnet when the server was using the cuda7.0 drivers, and then the cuda 7.5 driver was installed and the cuda 7.0 driver was removed.  I deleted the mxnet root directory and download the mxnet project again according to the install instructions(git clone --recursive https://github.com/dmlc/mxnet). By the way, originally, I use the system-provided python when I was compiling the mxnet package, but now I am using Anaconda.

yeah, a fresh new installation usually helps~

@wshenx, Problem Solved!!  I thought the system-provided python and anaconda have nothing to with each other.  Apparently I was wrong. The output of conda list | grep mxnet only shows that mxnet version 0.7 was installed. But when I use pip uninstall mxnet(this pip is from anaconda), surprisingly,  mxnet version 0.5 was uninstalled. Using the command again uninstalled the mxnet version 0.7.  Finally, reinstalling mxnet python wrapper solved the problem.

For anyone who has met this problem: make sure that any previous install of mxnet wrapper is removed and then reinstall the new mxnet wrapper.


"portIsFree" function's "timeout" parameter value increased to solve issues #48, #54 and #70

In windows, assigning a port may take a little longer than previously assumed value of 2000ms.
I am using Windows 10 1903 64bit, and it takes about 3000ms to initialize a port.
I increased the timeout to some higher value (9999ms) to give a little more space to Windows to initialize a port to use with the app.
This patch probably solves issues #48, #54 and #70

Hi - thanks for working on this! I'll try to review this eventually but unfortunately the bigger fire is that CI is currently broken for this repo (#72) and I don't have a lot of free time to work on this tool these days. :
(

Cool commit! A few minor suggestions:
Why 9.999 seconds? If it takes 3000 seconds to get a port, why not, say, 5000 or 6000ms?
I'd add a comment just above the line explaining the reason for the long timeout so people don't have to read this commit message
// Windows can be slow to get a port - Windows 10 1903 64bit takes about 3000ms in testing
Better yet, actually check if the platform is windows and only increase the timeout then.

I tried different values and the code gets the port earlier if it is available, so adding a check for OS is not necessary. I agree commenting inside the code could be useful, I couldn't decide if I should. 9999ms is seemed logical as all other numbers would be random values just like the originally assumed 2000ms.
A little downside of the code is that the Timer is still kept set by the system even if port check completed less than a second. But it is a small price to pay I think.
Thanks

Hi - thanks for working on this! I'll try to review this eventually but unfortunately the bigger fire is that CI is currently broken for this repo (#72) and I don't have a lot of free time to work on this tool these days. :
(

Hope you or someone with enough knowledge finds some time to work on it. For now, at least those that can access code can change the value just like I did.
I am new to this area of coding, maybe in few months I can help on that too.

@jkrems
I dig into the testing parts a bit more. First with Travis CI system, then cloning the project to my local disk then running tests with few different methods.

There are many fails occurs with tap test suit. Error on CI possibly all are these fails. problems are mostly related to cli.output. it is somehow scrambled during execution.
I tried all test scripts by hand with node test/cli/test_file.js. only launch.test.js fails on tests at different steps on different runs. Each fail occurs on forwards child output tests, again with scrambled cli.output. They sometimes pass, sometimes fail, but at least 1 of them fails to the end.

I wrote my findings here in this thread, because I don't know what else to do with them. At least you/we may use it as a reference later.


Feature request - BME280 Sensor Support

Wondering how hard it would be to add support for the Bosch BME280 Sensor.  Have a few on order, might give it a go myself once I get them.  Really just wanted to see if anyone else has already done it or not.
The sensor supports Temp, Hum, and Pressure.  All over and I2C connection.
https://www.adafruit.com/product/2652
Well, that and to see if there is any life left on this project.  Seems like a lot of people are using it with HA, but this has not been updated in some time.

it is in fact not so hard to do that. You need to connect it to the I2C pins. Standard these are D1 and D2. These are however taken up by the red and green pin. Though it is possible to use other I2C pins, it might be easiest to give the red and green pin other vales like D4 and D6.
Then it is just a matter of adding the library, chosing the right mode and doing a read that you add to the JSON


Clear clipboard of the mnemonic

Partially addresses #32
Changes
Use pyperclip, which works on macOS, Linux and Windows, to set the clipboard to a space character after user is done inputting data
Types of changes
What types of changes does your code introduce?

[x] New feature (a non-breaking change that adds functionality)

Testing
Requires testing

[x] Yes
[ ] No

Remarks
This introduces some host dependencies. Notably on Linux, clipboard mechanisms only exist if a window manager is present. So as to not bomb out on headless Linux, the code is in a try: except:
"On Windows, no additional modules are needed.
On Mac, this module makes use of the pbcopy and pbpaste commands, which should come with the os.
On Linux, this module makes use of the xclip or xsel commands, which should come with the os. Otherwise run “sudo apt-get install xclip” or “sudo apt-get install xsel” (Note: xsel does not always seem to work.)"

existing-mnemonic, the user already has the mnemonic outside the clipboard somewhere. We don't need to warn.
Added a warning (and a note in docs) for new-mnemonic.


Add Hardkernel HC4 support

Description of changes
I propose to include some specific board options for the Odroid HC4 in nixos-hardware.
I have been successfully using NixOS on my Odroid HC4 since December 2021.
Things done

[X] Tested the changes in your own NixOS Configuration
[ ] Tested the changes end-to-end by using your fork of nixos-hardware and
importing it via <nixos-hardware> or Flake input

Thanks @bors merge

@bors merge

bors merge


MudTable content not updated after deleting items from the source if inline editing is enabled

Bug type
Component
Component name
MudTable
What happened?
Take a MudTable with Items bound to a source of List<T>  or ObservableCollection<T>
Then you delete an item from the source, like this for example:
async Task DeleteEntry(string item)
{
    await Task.Delay(0);
    Test.Remove(item);
    StateHasChanged();
}

If no RowEditingTemplate is defined then the item will be deleted from the source and the UI will be updated accordingly.
If however, a RowEditingTemplate is defined to enable inline editing, like this for example:
<RowEditingTemplate>
      <MudTextField @bind-value="@context" T="string"/>
      <MudIcon Class="ma-4"<EMAIL_ADDRESS>@onclick="@( async() => await DeleteEntry2(context))" Size="Size.Small"></MudIcon>
</RowEditingTemplate>

The item is deleted from the source but the UI remains unchanged.
Extra note:
If you have a search filter applied, you can exit inline edit mode and change the SearchString a few times,
the UI will be updated and show the correct state of the source.
Expected behavior
When deleting the item from the source I expected the following to happen:

The item is deleted from the source
The MudTable row exits inline edit mode
The MudRow for that item is removed from the MudTable

In the reproduction link, which is based on a sample that was made for 3272 I've modified the MudTable bound to Items2 by adding a RowEditingTemplate in order to reproduce the issue.
Reproduction link
https://try.mudblazor.com/snippet/mamQubvccDAmEQha
Reproduction steps

Take a MudTable
Bind Items to some source
Define a RowEditingTemplate
Add logic to delete items from the source
Delete items from the source and watch how the UI is not updated

Relevant log output
No response
Version (bug)
6.0.4
Version (working)
No response
What browsers are you seeing the problem on?
Firefox, Microsoft Edge
On what operating system are you experiencing the issue?
Linux
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I have the same problem.   Is there a workaround to manual reset the selected row after the delete is done in the db?

I've faced the simular issue 😬.
The problem is that the MudTable saves previous items, when it in inline edit mode (in my case durring the switching between pages).
What I did: I reseted editing item
<MudTable @ref="_table"/>

@code
{
    private MudTable<T>?  _table;

    protected override async Task OnInitializedAsync()
    {
        _table?.SetEditingItem(null);
    }
}


Out Of Memory Error with Dask arrays

Describe the issue:
Hi, my team uses dask to combine features (reindex and forward fill) with respect to unique timestamps and then run a linear regression. We load feature data through a config which has the hashes and as of now we are running a fit for 30 features with 30 dates.
I am working on a server with 375 GB ram, but facing the Out of memory error which I wanted to discuss. I think my code breaks at the to_dask_array operation when all the data gets loaded into memory.
I will be highly obliged to you if you can help us out in tackling this issue, or any general comments on how I can prevent going out of memory.
Minimal Complete Verifiable Example:
def combine_feature_data(feature_hashes, objective_paths, dates, base_dir):
    combined_feature_dfs = []
    for feature_hash in feature_hashes:
        parquet_files = [os.path.join(base_dir, f"indicator_{feature_hash}_{date}/part.0.parquet") for date in dates]
        feature_df = read_parquet_files(parquet_files)
        combined_feature_dfs.append(feature_df)

    objective_dfs = [dd.read_parquet(path) for path in objective_paths]
    combined_feature_dfs.extend(objective_dfs)
    combined_features = dd.concat(combined_feature_dfs, axis=0, interleave_partitions=True)
    combined_features = forward_fill_combined_dataframe(combined_features)
    return combined_features

def forward_fill_combined_dataframe(combined_df):
    all_indices = sorted(set(combined_df.index))
    index_df = pd.DataFrame(all_indices, columns=['timestamp'])
    index_ddf = dd.from_pandas(index_df, npartitions=1).set_index('timestamp')
    reindexed_df = index_ddf.merge(combined_df, left_index=True, right_index=True, how='left').ffill().fillna(0)
    reindex_df = reindexed_df.drop_duplicates()
    return reindexed_df

X = combine_feature_data(feature_hashes, objective_paths, dates, base_dir)
X = X.to_dask_dataframe()
x = (X.iloc[:, :-1]).to_dask_array(lengths=True)
y = (X.iloc[:, -1]).to_dask_array(lengths=True)

model = LinearRegression()
model.fit(x, y)

@phofl Would like to hear your comments on this. Thanks in advance.

Hi,
I can't troubleshoot this from afar, the example is something I can't copy-paste because the data is coming from parquet. You'd either have to provide a more minimal example or run this on a platform where we have access to the metrics about the workload

Closing this for now, please ping to reopen if you can provide a reproducer


Wekan Snap does not work on Debian 9 for Windows Subsystem for Linux

I did get errors about snap socker. I did try to fix it with info from related issues at https://github.com/Microsoft/WSL/issues/534 https://github.com/Microsoft/WSL/issues/534 but with no success.

Wekan works with source install on Windows SubSystem for Linux, so it's enough for development.
https://github.com/wekan/wekan-maintainer/blob/master/virtualbox/rebuild-wekan.sh

Yeah at the moment snap packages don't work in Windows Linux sub system. Kernel Linux emulation is not there yet with feature set to allow full security continent and other needed features


fix(occlusion_spot): occlusion spot parameter disable as default

Signed-off-by: Muhammad Zulfaqar Azmi<EMAIL_ADDRESS>Description
This PR aims to disable occlusion spot module by default.
The occlusion spots feature was introduced to decelerate anticipating people jumping out from behind nearby vehicles, but it causes unnecessary slowing down in multi-lane driving. This design issue has yet to be fixed.
Due to the issue, currently, some functional test are facing complication. To reduce this occurrence, occlusion spot is disabled.
Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[X] I've confirmed the contribution guidelines.
[X] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

@zulfaqar-azmi-t4
can you fix pre commit?


Lost connenction to web api after last update

Hello,
I made an update from 0.3.9 to 0.3.10 today and after HA reboot I lost access to Web Api sensors.
After downgrade back to 0.3.9 everything is OK.
It is issue on my side or not?
Device: Nuki Smart Lock 3.0 Pro without bridge
thanks, Regards,
Petr

status is ok for me, but i have 404 error for lock/unlock actions

Same for me

Also a 404.

raise ConnectionError(f"Http response for {response.request.url}: {response.status_code} {response.reason}")
ConnectionError: Http response for https://api.nuki.io/smartlock/XXXXXXXXX/action: 404 Not Found

The discussion is in #76
There's a regression detected, but the fix isn't known yet

Hi, have the same here with thaat log message. I noticed that the integration is using wrong lock ID... even after removing and configuring a new, it is still the wrong lock ID.
I hope this helps...

There should be a new device which is called Nuki WebApi. The device contains a entity for every user. Including a user for webapi. Make sure that it is on.

Reverted to 0.3.8, still same issue :/
I deleted/added the integration.
Generated new API token with all authorizations and i still have the 404 error..
I dont understand..

I need more details to help you resolving the issue.
@chu11u when you say "wrong lock ID", what is "wrong" value and what should be the correct one?
@forzewin which API call gives your 404 and what is the ID of your device?

I need more details to help you resolving the issue.
@chu11u when you say "wrong lock ID", what is "wrong" value and what should be the correct one?
The part in the URL wuch hold the lockid is different than my own lockid.
I ran those api calls my self to get my lockid and got a different number.
Than I ran the same lock action call with the right lockid, and the device locked and unlock.
So now I use the integration only for the state of the lock and using direct calls for lock/unlock untill the integration is fixed.

@forzewin which API call gives your 404 and what is the ID of your device?
I got the 404 when I tried to lock or unlock. It is the action URL call. It showed briefly on the Lovelace bottom and also in the log file.

Sorry for the formatting, i am on my phone.
here the full error when trying to unlock the lock :
Échec d'appel du service "lock/unlock". Http response for https://api.nuki.io/smartlock/292XXXX48996/action: 404 Not Found
full log :
Logger: homeassistant.components.websocket_api.http.connection
Source: custom_components/nuki_ng/nuki.py:39
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 11:54:11 (17 occurrences)
Last logged: 13:57:27
[140127195676832] Http response for https://api.nuki.io/smartlock/292XXXXX996/action: 404 Not Found
[140127194185936] Http response for https://api.nuki.io/smartlock/292XXXXX48996/action: 404 Not Found
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 199, in handle_call_service
await hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 1713, in async_call
task.result()
File "/usr/src/homeassistant/homeassistant/core.py", line 1750, in _execute_service
await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 204, in handle_service
await service.entity_service_call(
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 676, in entity_service_call
future.result()  # pop exception if have
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 930, in async_request_call
await coro
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 713, in _handle_entity_call
await result
File "/config/custom_components/nuki_ng/lock.py", line 56, in async_unlock
await self.coordinator.action(self.device_id, "unlock")
File "/config/custom_components/nuki_ng/nuki.py", line 466, in action
await self.api.web_lock_action(self.web_id(dev_id), action)
File "/config/custom_components/nuki_ng/nuki.py", line 126, in web_lock_action
await self.web_async_json(
File "/config/custom_components/nuki_ng/nuki.py", line 184, in web_async_json
return await self.async_json(
File "/config/custom_components/nuki_ng/nuki.py", line 39, in async_json
raise ConnectionError(f"Http response for {response.request.url}: {response.status_code} {response.reason}")
ConnectionError: Http response for https://api.nuki.io/smartlock/2928XXXX48996/action: 404 Not Found
the id of my lock in nuki app looks like this
ID  Smart Lock: 30XXXF64

I need more details to help you resolving the issue.
@forzewin which API call gives your 404 and what is the ID of your device?
I got the 404 when I tried to lock or unlock. It is the action URL call. It showed briefly on the Lovelace bottom and also in the log file.

Mine is:
[140186346527696] Http response for https://api.nuki.io/smartlock/292805081XXX/action: 404 Not Found
I have a Smart Lock Pro 3.0 without Bridge.

Thanks for the input.
I've pushed a fix to the main branch, could you please test the it? (Install main version via HACS)

Just did it and i can confirm that the fix works like a charm.
Everything is back and working again
Thanks a lot @kvj

Thanks for the input. I've pushed a fix to the main branch, could you please test the it? (Install main version via HACS)

Yes, main version is working for me too!

Thanks for the input. I've pushed a fix to the main branch, could you please test the it? (Install main version via HACS)

Sorry for being novice... how do I do that? i looked at the integration and there is no update. how do I install it?

how do I install it?

Go to HACS, click on the integration, select "Re-download" from the popup menu and pick "main" from the list of all versions available

Thanks for the input. I've pushed a fix to the main branch, could you please test the it? (Install main version via HACS)

Sorry for being novice... how do I do that? i looked at the integration and there is no update. how do I install it?

Go to HACS > Integrations > Search for Nuki Lock and click on it > on the upper right, click on ... > Redownload > Select Main

Thanks for the input. I've pushed a fix to the main branch, could you please test the it? (Install main version via HACS)

Sorry for being novice... how do I do that? i looked at the integration and there is no update. how do I install it?

Go to HACS > Integrations > Search for Nuki Lock and click on it > on the upper right, click on ... > Redownload > Select Main
Done. Do i need to re-install the integration also?

You need to restart HA after the redownload

You just need to restart HA after the redownload

well... still get the 404 message:
Http response for https://api.nuki.io/smartlock/292882605412/action: 404 Not Found
I have no problem leaving the lockid because it is wrong one...
Ill try to remove and configure the lock again

Ok. I can confirm that main worked.

is 0.3.10 the corrected version?

is 0.3.10 the corrected version?

nope, I haven't pushed the updated release yet. That will be 0.3.11

After a complete removal and reinstall of 'main' it worked for me as well. confirm.

I can't confirm. Problem still exist.
Removal and reinstall did not work. Is equal which version i'll use: main or 0.3.10.
If a status is requested the bridge become a ping timeout for round about 2-3 Pings. This can be reproduced. I opened a ticket at Nuki, too.

If a status is requested the bridge become a ping timeout for round about 2-3 Pings. This can be reproduced. I opened a ticket at Nuki, too.

In case the bridge receives too many requests, it reboots, it's a known issue due to the hw limitations of the bridge. It can process only 1 request at a time. Probably you're sending it requests not only from this integration but from the mobile app, the web app, etc.
The bridge has a log, check it, you can dump it via curl and the /log endpoint.

I can't confirm. Problem still exist.
Removal and reinstall did not work. Is equal which version i'll use: main or 0.3.10.
If a status is requested the bridge become a ping timeout for round about 2-3 Pings. This can be reproduced. I opened a ticket at Nuki, too.

Remove and install "main" version

I had removed and install main, also tried to delete all files manually. Problem exists.
But a the nuki log, i can see the reboots. I have to wait for the answer from Nuki about this problem.

But a the nuki log, i can see the reboots. I have to wait for the answer from Nuki about this problem.

I knew that was the case, and in order to limit the reboots, you need to lower the queries to the bridge: nuki_ng does only one every 30s (polling), the rest are callbacks (bridge to nuki_ng) so there are no problems there.
Make sure you uninstalled the old (official) nuki integration, make sure there are no other apps communicating with the bridge. If you can check the bridge log, check the frequency of the reboots to have an idea.
I'm pretty confident you are running nuki_ng in parallel to the official nuki integration, double-check that.

You were right, I had the official Nuki integration running. I have now disabled and uninstalled it.
I also closed all apps and tried it only via HA with Nuki Lock. The errors have become less, but are still there. As soon as I try to buzz/unlatch the "Opener" or "Lock", it often takes up to a minute until something happens.
This is what I find in the logs:
Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht
Logger: custom_components.nuki_ng.nuki
Source: custom_components/nuki_ng/nuki.py:155
Integration: Nuki Lock (documentation, issues)
First occurred: 19:06:36 (1 occurrences)
Last logged: 19:06:36
Failed to update callback http://homeassistant.pandora:8123/api/webhook/nuki_ng_bridge_hook_xxxxxxxxxxxxxxxxxxx
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/local/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f66a2fe50>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='<IP_ADDRESS>', port=8080): Max retries exceeded with url: /callback/list?token=xxxxxx (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f66a2fe50>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/config/custom_components/nuki_ng/nuki.py", line 388, in _update
callbacks_list = await self.api.bridge_check_callback(
File "/config/custom_components/nuki_ng/nuki.py", line 154, in bridge_check_callback
callbacks = await self.async_json(
File "/config/custom_components/nuki_ng/nuki.py", line 37, in async_json
response = await self.hass.async_add_executor_job(lambda: cb(requests))
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/config/custom_components/nuki_ng/nuki.py", line 37, in
response = await self.hass.async_add_executor_job(lambda: cb(requests))
File "/config/custom_components/nuki_ng/nuki.py", line 155, in
lambda r: r.get(self.bridge_url("/callback/list"), timeout=BRIDGE_TIMEOUT)
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='<IP_ADDRESS>', port=8080): Max retries exceeded with url: /callback/list?token=xxxxxx (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f66a2fe50>: Failed to establish a new connection: [Errno 111] Connection refused'))
Logger: homeassistant.components.websocket_api.http.connection
Source: custom_components/nuki_ng/nuki.py:39
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 19:06:39 (2 occurrences)
Last logged: 19:06:45
[546950995376] Http response for http://<IP_ADDRESS>:8080/lockAction?token=xxxxxx&action=3&nukiId=xxxxxxxxx&deviceType=2: 503 Service Unavailable
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 199, in handle_call_service
await hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 1713, in async_call
task.result()
File "/usr/src/homeassistant/homeassistant/core.py", line 1750, in _execute_service
await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 204, in handle_service
await service.entity_service_call(
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 676, in entity_service_call
future.result()  # pop exception if have
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 930, in async_request_call
await coro
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 713, in _handle_entity_call
await result
File "/config/custom_components/nuki_ng/lock.py", line 59, in async_open
await self.coordinator.action(self.device_id, "open")
File "/config/custom_components/nuki_ng/nuki.py", line 461, in action
result = await self.api.bridge_lock_action(dev_id, action, device_type)
File "/config/custom_components/nuki_ng/nuki.py", line 99, in bridge_lock_action
return await self.async_json(
File "/config/custom_components/nuki_ng/nuki.py", line 39, in async_json
raise ConnectionError(f"Http response for {response.request.url}: {response.status_code} {response.reason}")
ConnectionError: Http response for http://<IP_ADDRESS>:8080/lockAction?token=xxxxxx&action=3&nukiId=xxxxxxxxx&deviceType=2: 503 Service Unavailable
Any idea?
Thanks in advance!

Yes I have an idea: your setup is a total disaster. :)
Jokes apart: the errors are "connection refused", it means that HA/Nuki_NG can't communicate with the bridge. So please exclude any network issue, and if you're confident it's not the network (firewalls, vlans, etc.) then I would suggest to uninstall nuki_ng, read the instructions carefully again and reinstall it configuring everything accordingly. Take time to read every step.
Make sure you disabled all apps that could communicate with the bridge.


[dv: tl ul] Host agent is not comforming to spec - back2back messages

in conjunction with #3208
While the TL-UL specification does not allow multi beat messages, I cannot find anything in the spec that disallows driving single beat messages back2back.
which would give a burst-like behavior.
Currently, our DV host does not support this functionality. the minimum distance between consecutive messages is 1 clock cycle gap. This basically cuts the bandwidth in half.
figure 4.1 in  the specification version 1.8.1

Message J has opcode 4, which on Channel A indicates a Get. Even though the Get operates on 16
bytes as indicated by a_size, message J itself carries no data and thus fits in a single beat which is
accepted immediately. Message K can then be issued and accepted the following cycle

in the wave below the request is accepted immediately, so the next beat should be presented in the next cycle. but this is not the case

good catch.  Back to back transactions is definitely something we need to check.
I assume at a minimum the icache will do this.

@rasmus-madsen
we do have test a lot of back2back cases in CSR tests.

If you would like to have back2back in the other IP seq, use csr_rd/wr(.blocking(0))
  task automatic csr_rd(input  uvm_object     ptr, // accept reg or field
                        output uvm_reg_data_t value,
                        input  uvm_check_e    check = UVM_CHECK,
                        input  uvm_path_e     path = UVM_DEFAULT_PATH,
                        input  bit            blocking = default_csr_blocking,
                        input  bit            backdoor = 0,
                        input  uint           timeout_ns = default_timeout_ns,
                        input  uvm_reg_map    map = null);
    if (backdoor) begin
        csr_peek(ptr, value, check);
    end else if (blocking) begin
      csr_rd_sub(ptr, value, check, path, timeout_ns, map);
    end else begin
      fork
        csr_rd_sub(ptr, value, check, path, timeout_ns, map);
      join_none
      // Add #0 to ensure that this thread starts executing before any subsequent call
      #0;
    end
  endtask

But if using non-blocking, csr_rw/rd will consume no time, you may need to add some blocking statement.
forever begin
  csr_rd(a_reg, .blocking(0));
  #dly;
end

I feel some TLUL behavior can be covered by CSR test suite, such as back2back access, TLUL error, csr reset value. In the IP seq, we can focus more on IP functionality rather than the register module. WDYT?

@weicaiyang I think I tried that and I concluded that the blocking was for some other feature as it didn't do what I expected.
I can look again tomorrow and try to come up with a solution.
but I disagree that back to back can be covered by CSR tests since AES will be bandwidth limited without this functionality
and I expect that the most common behavior for AES is to write the register groups back 2 back.

but I disagree that back to back can be covered by CSR tests since AES will be bandwidth limited without this functionality
and I expect that the most common behavior for AES is to write the register groups back 2 back.

agree, if it's to stress test the bandwidth limit. We certainly should enable back2back access and zero_delays.
blocking=0 can start next CSR access without waiting for completion of current access.
zero_delays will set all TL random delay to 0. (example).

@weicaiyang I will set the TL UL delays from my test,
I feel  both sides should be configured to one of 5 options.
0 delay (super fast)
fast (delay from 0-5)
slow (delay from 4-10)
very slow (delay 10)
I will randomize host and device to one of the 5 options and set the tl_ul agent config accordingly from my test

@weicaiyang I will set the TL UL delays from my test,
I feel both sides should be configured to one of 5 options.
0 delay (super fast)
fast (delay from 0-5)
slow (delay from 4-10)
very slow (delay 10)
I will randomize host and device to one of the 5 options and set the tl_ul agent config accordingly from my test

there are kinds of delay knobs in agent cfg. TLUL agent acts as a host for AES. You can just control those knobs for host.

that is the plan.

@weicaiyang after taking a closer look I do not believe that the blocking(0) option is appropriate for b2b transactions I don't fully understand this feature.
Each transaction should take 1 clock. but the driver should be ready to present the next transaction on the following clock.
which is why I believe this is a problem in the driver.
virtual task send_a_channel_request(tl_seq_item req);
  .
  .
  .
    // break delay loop if reset asserted to release blocking
    `DV_SPINWAIT_EXIT(repeat (a_valid_delay) @(cfg.vif.host_cb);,
                      wait(reset_asserted);)
    // wait until no outstanding transaction with same source id
    `DV_SPINWAIT_EXIT(while (is_source_in_pending_req(req.a_source)) @(cfg.vif.host_cb);,
                      wait(reset_asserted);)

    if (!reset_asserted) begin
      cfg.vif.host_cb.h2d_int.a_address <= req.a_addr;
      cfg.vif.host_cb.h2d_int.a_opcode  <= tl_a_op_e'(req.a_opcode);
      cfg.vif.host_cb.h2d_int.a_size    <= req.a_size;
      cfg.vif.host_cb.h2d_int.a_param   <= req.a_param;
      cfg.vif.host_cb.h2d_int.a_data    <= req.a_data;
      cfg.vif.host_cb.h2d_int.a_mask    <= req.a_mask;
      cfg.vif.host_cb.h2d_int.a_user    <= '0;
      cfg.vif.host_cb.h2d_int.a_source  <= req.a_source;
      cfg.vif.host_cb.h2d_int.a_valid   <= 1'b1;
      // bypass delay in case of reset
      `DV_SPINWAIT_EXIT(@(cfg.vif.host_cb);, // <<< **WHY IS THIS HERE?**
                        wait(reset_asserted);)
    end
    `DV_SPINWAIT_EXIT(while(!cfg.vif.host_cb.d2h.a_ready) @(cfg.vif.host_cb);,
                      wait(reset_asserted);)
  .
  .
  .

  endtask : send_a_channel_request

I don't see the need for the callback inside the if.
Removing it forces an Assertion to fire.
 `ASSERT(pendingReqPerSrc_A, h2d_pre_S |-> pendingReqPerSrc_S, !clk_i, !rst_ni)

it seems this assertion fires prematurely since there is a check in the driver that should avoid having multiple outstanding requests.
    `DV_SPINWAIT_EXIT(while (is_source_in_pending_req(req.a_source)) @(cfg.vif.host_cb);,
                      wait(reset_asserted);)

That being said I don't see anything in the spec prohibiting multiple outstanding requests.
I have discussed this with @rswarbrick
and we agree on that
The master can send a request on A. Once the slave has ack'ed the request, the master can send another without waiting for the response on D.  This allows clever slaves that have slow responses but support pipelining to still get high throughput.
So I think A rework of both host and device driver is needed to incorporate b2b support.

@rasmus-madsen
TL spec allows outstanding request, but each outstanding request has to use unique source ID.
We had a discussion last year and we decided to make driver wait if the ID of the request is the same as another pending request.
Total source ID is 32 and most of the IPs only supports up to 2 outstanding requests. If seq only sends a few requests, the chance of this case is low, but if sending a lot, it will eventually happens.
If you want to make sure the item is absolutely sent in an outstanding way, need to make sure it's sent with a unique ID.
I think you sends request through RAL, but RAL doesn't provide interface for source ID. You could override tl_seq_item to control the ID or we could consider to add a knob and let driver modify the ID.
    // wait until no outstanding transaction with same source id
    `DV_SPINWAIT_EXIT(while (is_source_in_pending_req(req.a_source)) @(cfg.vif.host_cb);,
                      wait(reset_asserted);)

The assertion checks if source ID is unique
  // a_source: there should be no more than one pending request per each source ID
  sequence pendingReqPerSrc_S;
    pend_req[h2d.a_source].pend == 0;
  endsequence

See my comment below for WHY IS THIS HERE?
    if (!reset_asserted) begin
      cfg.vif.host_cb.h2d_int.a_address <= req.a_addr;
      cfg.vif.host_cb.h2d_int.a_opcode  <= tl_a_op_e'(req.a_opcode);
      cfg.vif.host_cb.h2d_int.a_size    <= req.a_size;
      cfg.vif.host_cb.h2d_int.a_param   <= req.a_param;
      cfg.vif.host_cb.h2d_int.a_data    <= req.a_data;
      cfg.vif.host_cb.h2d_int.a_mask    <= req.a_mask;
      cfg.vif.host_cb.h2d_int.a_user    <= '0;
      cfg.vif.host_cb.h2d_int.a_source  <= req.a_source;
      cfg.vif.host_cb.h2d_int.a_valid   <= 1'b1;
      // bypass delay in case of reset
      `DV_SPINWAIT_EXIT(@(cfg.vif.host_cb);, // <<< **WHY IS THIS HERE?**. // this line is to make a_valid last at least 1 cycle, but looks like it can be merged into the following while loop.
                        wait(reset_asserted);)
    end
    `DV_SPINWAIT_EXIT(while(!cfg.vif.host_cb.d2h.a_ready) @(cfg.vif.host_cb);,
                      wait(reset_asserted);)

@weicaiyang
I was not a part of the referred discussion, but I don't see how "most Ps only support up to two outstanding requests" turns into an environment that only supports 1?
I think this is a place where we need to be compliant, and let the device do the throttling, not the host driver.
also not supporting back2back is effectively cutting AES performance as loading it takes 2x than it needs to.
why don't we give the Driver control of the source ID.
then it can simply increment when a request is accepted and decrement when the response is seen by the monitor.

another question, how did you create that back 2 back simulation you have copied waves from, it looks to me like the driver does not support that in its current state because of that extra @cb

are we testing the Icache with back2back?

@weicaiyang
I was not a part of the referred discussion, but I don't see how "most Ps only support up to two outstanding requests" turns into an environment that only supports 1?

the discussion was last July even before we created this repo. Driver usually doesn't modify data/control info, for example, in a xbar test, I want to use same source ID for all hosts. I set source ID in seq and don't expect driver to modify it.
Besides, there is only 1/256 chance (source is 8 bit wide) the source ID isn't unique. If we really want always back2back, still several ways to implement it in seq level.

I think this is a place where we need to be compliant, and let the device do the throttling, not the host driver.
also not supporting back2back is effectively cutting AES performance as loading it takes 2x than it needs to.
why don't we give the Driver control of the source ID.
then it can simply increment when a request is accepted and decrement when the response is seen by the monitor.

another question, how did you create that back 2 back simulation you have copied waves from, it looks to me like the driver does not support that in its current state because of that extra @cb

you can use this command to get the waveform I shared.
dvsim.py aes_csr_rw -ro "+zero_delays=1" -w
or modify these in your seq

set zero_delays=1 through plusarg or directly change env_cfg.zero_delays = 1
csr_rd/wr/update(..., .blocking(0))

there is only one CB here for a_valid if zero_delays is set and source ID is unique.

are we testing the Icache with back2back?

maybe @rswarbrick can answer this

are we testing the Icache with back2back?

The ICache testing doesn't use TileLink at all. In the actual design, the ICache connects to the rest of the system through an adapter. In the testbench, the DV agent plays the role of that adapter, so we don't get "as far" as the TileLink part.
Also note that the ICache acts as a master/host as far as TileLink is concerned: it responds to requests from the core (with a weird protocol) and then generates requests of the main memory.

are we testing the Icache with back2back?

The ICache testing doesn't use TileLink at all. In the actual design, the ICache connects to the rest of the system through an adapter. In the testbench, the DV agent plays the role of that adapter, so we don't get "as far" as the TileLink part.
Also note that the ICache acts as a master/host as far as TileLink is concerned: it responds to requests from the core (with a weird protocol) and then generates requests of the main memory.

ok, the reason I brought it up was that @tjaychen mentioned it earlier in an earlier comment.

@weicaiyang
I think we are talking past each other.
I understand I can disregard the RAL and go and do my own register accesses.
but I think instead we should fix our RAL layer so it supports outstanding requests.
I think this issue can be changed into RAL does not support back2back (unique IDs)

Since this issue has been rumbling on for a while, and touched on several different topics, maybe it would make the most sense to open a new one for that specific problem? It might make it easier to get stuff closed incrementally.

@weicaiyang
I think we are talking past each other.
I understand I can disregard the RAL and go and do my own register accesses.
but I think instead we should fix our RAL layer so it supports outstanding requests.
I think this issue can be changed into RAL does not support back2back (unique IDs)

I think I may miss some of your points. Sri has sent a meeting invite to discuss this issue.

As we discussed in the meeting, here are some potential solutions to guarantee unique ID for back2back transaction.

Sri: ID allocator in seq_item, when randomizing the item, look at the outstanding window to generate a unique ID. If max outstanding is 2, the ID of every 2 items are unique
Rasmus: have fifo in cfg, driver uses the ID in cfg fifo when fifo isn’t empty
Add a knob to let driver modify the ID to be unique rather than stalling the item

@sriyerg @weicaiyang
I think we can close this do you agree?

@sriyerg @weicaiyang
I think we can close this do you agree?

We can close it now. Sri has added the b2b support in the agent.

We can close it now. Sri has added the b2b support in the agent.


Add NonNull annotation and use requireActivity

See #483 and https://github.com/permissions-dispatcher/PermissionsDispatcher/pull/511#issuecomment-420142755

Thank you! Let me check 👀


Use amboss instead of 1ml

Amboss.space is a new explorer. Where 1ml has not had updates for years, amboss.space is a great explorer, actively developed, and shows much better info, such as readable fee reports.

Thanks for this!


Multi-engine support

Support file extensions and subfolder structure for additional intent engines in NSP.

Fixed in REL 2.0.0
Check: https://github.com/nokia/vscode-intent-manager/releases/tag/2.0.0


base16-schemes: unstable-2024-01-14 -> unstable-2024-06-21

Description of changes

Update base16-schemes package to latest commit here
Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[X] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

P.S.: the GitHub web interface is subpar.

Result of nixpkgs-review pr 321853 run on x86_64-linux 1

1 package built:

base16-schemes

Works for me.
Thank you for your work @IamPyu.


qual: make config hook types more precise

@brillout, validated with tsc --noEmit on the example. Is this going in the right direction? If yes I would like to:

do the same for other hooks
add types to all such hooks in all existing typescript examples.

Is it OK with you?

@brillout I think it is better now

@brillout does it now look more like what you had in mind? Thanks

examples/auth $ pnpm exec tsc --noEmit
pages/+config.h.ts:8:3 - error TS2322: Type '{ onRenderHtml: "import:vike-react/renderer/onRenderHtml"; onRenderClient: "import:vike-react/renderer/onRenderClient"; passToClient: string[]; clientRouting: true; hydrationCanBeAborted: true; meta: { ...; }; }' is not assignable to type 'Config | `import:${string}` | Config[] | `import:${string}`[] | undefined'.
  Type '{ onRenderHtml: "import:vike-react/renderer/onRenderHtml"; onRenderClient: "import:vike-react/renderer/onRenderClient"; passToClient: string[]; clientRouting: true; hydrationCanBeAborted: true; meta: { ...; }; }' is not assignable to type 'ConfigBuiltIn & Config & ConfigVikeSvelte'.
    Type '{ onRenderHtml: "import:vike-react/renderer/onRenderHtml"; onRenderClient: "import:vike-react/renderer/onRenderClient"; passToClient: string[]; clientRouting: true; hydrationCanBeAborted: true; meta: { ...; }; }' is not assignable to type 'ConfigBuiltIn'.
      Types of property 'meta' are incompatible.
        Type '{ Head: { env: "server-only"; }; Layout: { env: "server-and-client"; }; title: { env: "server-and-client"; }; description: { env: "server-only"; }; favicon: { env: "server-only"; }; lang: { env: "server-only"; }; ssr: { ...; }; }' is not assignable to type '`import:${string}` | ConfigMeta | undefined'.
          Type '{ Head: { env: "server-only"; }; Layout: { env: "server-and-client"; }; title: { env: "server-and-client"; }; description: { env: "server-only"; }; favicon: { env: "server-only"; }; lang: { env: "server-only"; }; ssr: { ...; }; }' is not assignable to type 'ConfigMeta'.
            Property 'ssr' is incompatible with index signature.
              Type '{ env: "config-only"; effect: ConfigEffect; }' is not assignable to type 'ConfigDefinition'.
                The types returned by 'effect(...)' are incompatible between these types.
                  Type 'import("/home/lourot/Documents/git/vike/vite-plugin-ssr/node_modules/.pnpm/vite-plugin-ssr@0.4.142_vite@4.3.9/node_modules/vite-plugin-ssr/dist/esm/shared/page-configs/Config").Config | undefined' is not assignable to type 'import("/home/lourot/Documents/git/vike/vite-plugin-ssr/vike/dist/esm/shared/page-configs/Config").Config | undefined'.
                    Type 'ConfigBuiltIn & Config & ConfigVikeReact' is not assignable to type 'Config | undefined'.
                      Type 'import("/home/lourot/Documents/git/vike/vite-plugin-ssr/node_modules/.pnpm/vite-plugin-ssr@0.4.142_vite@4.3.9/node_modules/vite-plugin-ssr/dist/esm/shared/page-configs/Config").ConfigBuiltIn & Vike.Config & VikePackages.ConfigVikeReact' is not assignable to type 'import("/home/lourot/Documents/git/vike/vite-plugin-ssr/vike/dist/esm/shared/page-configs/Config").ConfigBuiltIn & Vike.Config & VikePackages.ConfigVikeReact'.
                        Type 'ConfigBuiltIn & Config & ConfigVikeReact' is not assignable to type 'ConfigBuiltIn'.
                          Types of property 'extends' are incompatible.
                            Type 'Config | `import:${string}` | `import:${string}`[] | Config[] | undefined' is not assignable to type 'Config | `import:${string}` | Config[] | `import:${string}`[] | undefined'.
                              Type 'ConfigBuiltIn & Config & ConfigVikeReact' is not assignable to type 'Config | `import:${string}` | Config[] | `import:${string}`[] | undefined'.

I'm pretty sure this is because the vike auth example +config.h.ts includes the new vike Config that I have touched, but also wants to satisty vike-react's Config, which wants to satisfy the old vike Config. In other words there is a kind of circular type dep now accross vike and vike-react.
@brillout I believe we should tackle this circular dep first as this will hit us back whenever we touch the Config type. What do you think? Does that make sense to you if I start looking into this?

Yes, exactly.
If the user always extends PageContext{Client,Server} by using Vike.PageContext then I'm thinking we can even remove the generic type.
Btw. for onBeforeRender() specifically, since it's run on the server-side only, we should use PageContextServer instead of PageContext.
An interesting question is what happens when the user sets onBeforeRender's env to server-and-client? So far, I believe the only solution would be to code-gen types.

If the user always extends PageContext{Client,Server} by using Vike.PageContext then I'm thinking we can even remove the generic type.

You made me realize I wanted to use Vike.PageContext and not PageContext. I mixed up. I'll fix that and see what happens if I remove the generic.
Regarding "vike": "link:./vike/", we don't write that in the examples. Probably because we want the examples to work with npm?
Alternatively I was thinking of removing statisfies Config when using extends: vikeReact. Rationale:
// examples/auth/pages/+config.h.ts

import Layout from '../layouts/LayoutDefault'
import vikeReact from 'vike-react'

export default {
  Layout,
  passToClient: ['userFullName'],
  extends: vikeReact
} // `satisfies Config` not needed, as already implied by `extends vikeReact`, which is an object
  // that already states explicitly `satisfies Config`

What do you think?

Actually, since vike is a peer dependency, it should resolve to the same/local vike package 🤔.

In this example actually vike is also a normal dependency. Maybe it shouldn't be. Let me play with that first. Thanks for all the pointers :)

Does that make sense to you if I start looking into this?

Yes. (I had a quick look and it does seem like pnpm is linking everything correctly, so I'm not sure why we get that TypeScript error.)

Probably because we want the examples to work with npm?

Yes exactly. That's why we use pnpm's overrides feature instead.

I was thinking of removing statisfies Config when using extends: vikeReact

I believe this wouldn't type passToClient nor Layout; it would only type vikeReact.

In this example actually vike is also a normal dependency.

The example itself declares a direct dependency to vike which is what I would expect. What I meant is that vike-react shouldn't declare a direct dependency to vike and its dependency to vike should be a peer dependency instead, which is actually already the case (I just double checked). So I believe vike-react and the example are doing things correctly. And since pnpm seems to link everything correctly, I wonder why that error occurs.
Let me know if you want me to dig into that TS error.

@brillout I found the culprit: the example has a dependency to an old vike-react (0.2.0), which still has a dependency to vite-plugin-ssr instead of vike, so it doesn't benefit from "vike": "link:./vike/". Switching to<EMAIL_ADDRESS>solves the issue.
Thanks for the useful pointers. Updating the PR

Ah, indeed, I forgot to update that one. Thanks 🙏.

@brillout I'm happy with this now. Let me know if anything should be improved. Thanks!

👍 Thinking out loud:

The type of<EMAIL_ADDRESS>is actually (pageContext: PageContextServer | PageContextClient) => /* ... */ because of<EMAIL_ADDRESS>I think we can make it work by telling users that, if they modify meta['onBeforeRender'], then they should use Vike.Config to refine Config['onBeforeRender'].
It seems that const onBeforeRender: Config['onBeforeRender'] = async (pageContext: PageContextServer): Promise<{ pageContext: Partial<Vike.PageContext> }> is indeed the most correct/precise. I wonder if we can make it less verbose, but I guess we can't if we want to be strict about the return type. Which makes me wonder if the verbosity is worth the return type safety. Maybe I'm exaggerating but I imagine it to be tedious for the user to have to define 3 types for each hook (return type, argument type, and overall function type). I guess it's a trade-off situation. Thoughts?

An idea: const onBeforeRender: Config['onBeforeRender'] = async (pageContext: Parameters<Config['onBeforeRender']>[0]): ReturnType<Config['onBeforeRender']> (the types Paramaters and ReturnType are utilities provided by TypeScript like Partial). I've actually just tried (see my / the latest commit to this PR): turns out it isn't as simple as that. But maybe we can make it simpler by having Vike providing some utility type (I can further dig later this week if you want me to).

The return type Partial<Vike.PageContext> seems to be working well, as I can't think of a use case where a user would override a built-in pageContext property https://vike.dev/pageContext#built-in.
route() (and onBeforeRoute()) is also called on the client-side if the user uses Client Routing => pageContext: PageContextServer | PageContextClient. Since we'll increasingly promote the vike-* packages, I think it's fine if we assume the user to use Client Routing (Server Routing users would need to use their own types).
guard() can also be called on the client-side.

Very much looking forward to provide users with type safe Config['somHook'] 💫

The return type Partial<Vike.PageContext> seems to be working well, as I can't think of a use case where a user would override a built-in pageContext property https://vike.dev/pageContext#built-in.

Not sure I get this point. Is this about a particular line of code? Do you want me to change something?

@brillout I think I addressed everything. I'll give a try at documenting now all this, either in this PR or in another one.

I just woke up from a nap and got an idea, I hope it's ok if I share it with you. I don't know if it bothers you but I'm thinking it's worth it to take our time on this one. Let me know if it does bother you.
The idea:
- Hook<'onBeforeRender'>
- HookParam<'onBeforeRender'>
- HookReturn<'onBeforeRender'>
+ OnBeforeRender
+ OnBeforeRenderParam
+ OnBeforeRenderReturn

Why? Because, that way, the user can go-to-defintion:
// This is the Vike source code the user sees when he goes-to-the-definition of
// OnBeforeRender{Param,Return}
type OnBeforeRenderParam = PageContextServer
type OnBeforeRenderReturn = { pageContext?: Partial<Vike.PageContext> } | undefined
type OnBeforeRender = (pageContext: OnBeforeRenderParam) => OnBeforeRenderReturn

Whereas going-to-the-definition-of Hook{Param,Return} isn't helpful.
One drawback of OnBeforeRender{Param,Return} is that the source of truth isn't Config anymore, which means that the user cannot use Vike.Config to override the hook types. (Whereas with Hook{Param,Return} the source of truth is Config => if the user modifies Config by using Vike.Config then his changes will be reflected by Hook{Param,Return}.) I think a semi-satisfactory solution would be to tell users to not use OnBeforeRender{Param,Return} if they change the meta of onBeforeRender.
What do you think? Let me know if something is unclear.
Also, some random thoughts (clearly out-of-scope for this PR): I'm thinking that Vike.PageContextCustom could have been a better name than Vike.PageContext (which, I guess, is similar to what you had in mind back then with Vike.PageContextAdditionalData). Let's see how things turn out.
Thanks for the conversations btw. Your input has been very helpful. While it does delay the V1 design release, I think it's very well worth it: providing a neat TypeScript DX is super convenient for the user.

I'll give a try at documenting now all this, either in this PR or in another one.

I'm thinking, if we provide a clear path for users, then we may not need to teach him TypeScript knowledge at all. (Saving us time while lifting the user from the need for deep TypeScript knowledge: I can imagine that some/(many?) users don't want to dig into TypeScript.) That said, in the future, some extra docs explaining TypeScript could be really neat. So I agree: we can keep the docs minimal for now and see when/whether we'll need it.

No worries, you're right it's important to get this right. And it is interesting for me.
Regarding OnBeforeRender{Param,Return} it's a tiny bit more complicated but I have just pushed a minimum working commit: https://github.com/vikejs/vike/pull/1137/commits/2213c796 . I have added a bunch of inline NOTE(aurelien) as prompts for discussion, and I will eventually remove them. Let me know what you think.
Once we're happy with this part of the story, I'll reply on the other topics :) Namely:

Vike.PageContextCustom
documentation

NOTE(aurelien)

Good idea 👍 and I started doing the same.

Once we're happy with this part of the story, I'll reply on the other topics :) Namely:

👍 Sounds like a good plan.

// NOTE(aurelien): this is still my favorite style as a user. And as a Vike contributor I'll
// probably personally advertise for this.
const onBeforeRender: Config['onBeforeRender'] = async (
  pageContext: PageContextServer | PageContextClient
): Promise<{ pageContext: Partial<Vike.PageContext> }>

One thing I like about OnBeforeRender{Param,Return} is that it's simple for the user to set up: the user merely has to remember the ${HookName}{Param,Return} name convention. Whereas with the expanded style, the user has to go to https://vike.dev/onBeforeRender#typescript and copy-pase the hook type.
Also with OnBeforeRender{Param,Return} it's obvious that the hook type corresponds to the offical type and is therefore correct, whereas with the expanded style this isn't clear.
Another neat thing is that we can add JSDocs to OnBeforeRender{Param,Return}:

The screenshots also shows that the type is shown like in the expanded style.
Although, unfortunately, this doesn't work for OnBeforeRenderParam:

I've created a TypeScript Feature Request about it: https://github.com/microsoft/TypeScript/issues/56010.
That said, from a purely reading perspective (without TypeScript's quick info IntelliSense for example in a Pull Request review), the expanded style is indeed superior.

I'm thinking maybe we could/should forgo OnBeforeRenderParam while keeping OnBeforeRender{Return}:
const onBeforeRender: OnBeforeRender = async (pageContext: PageContextServer): OnBeforeRenderReturn

Vike.PageContextCustom

Yes I like this name more. Should we work on a renaming/alias?

I'm not sure about this yet, so I'd suggest we first finish this PR and then, at some point later, we revisit whether Vike.PageContextCustom turns out to be indeed better. (As always, feel free to disagree.)

FYI https://github.com/vikejs/vike/pull/1137/commits/b577406b1c23b895d4e4ae2813567bc22de6f315: I pushed a change to main so that hooks can be exported as export { hookName } instead of export default.

I pushed a change to main

The CI is failing because there is a bug in the changes I pushed. I'll fix it tomorrow.

FYI b577406: I pushed a change to main so that hooks can be exported as export { hookName } instead of export default.

Nice!

I'll fix it tomorrow.

Done. The CI is now green.

@brillout I think it looks good now. What do you think?
Should I work on adding all this to the documentation or do you think we can get away with not documenting this for now?

👍 I'm seeing you added the type utils for the hooks 💯.
Async route is actually deprecated, so the default should be sync. I also think the default for onPageTransition{Start,End} hooks should be sync onPageTransition{Start,End} hooks don't return any value so I think we may not need any (a)sync variant. Same for onHydrationEnd.
I'm thinking whethter other hooks should be sync by default. E.g. I expect most guard() hooks to be sync. Alternative: we always use async guard() hooks in the docs and examples. Any thoughts on this? (I'm thinking it could be nice to always use async hook when a hook can be async: it's then obivious that the hook can be async.)

I just tried and this works:
type OnHydrationEnd = (pageContext: PageContextClient) => Promise<void | undefined> | void | undefined;

So we indeed can remove the variants for hooks that return void | Promise<void>.
On the other hand, it's kinda nice to keep the convention HookName{Sync} consistent across all hooks.
Hm 🤔

Async route is actually deprecated, so the default should be sync. ~I also think the default for onPageTransition{Start,End} hooks should be sync~ onPageTransition{Start,End} hooks don't return any value so I think we may not need any (a)sync variant. Same for onHydrationEnd.

Make sense, yes

I'm thinking it could be nice to always use async hook when a hook can be async: it's then obivious that the hook can be async.

Yes I like this

type OnHydrationEnd = (pageContext: PageContextClient) => Promise<void | undefined> | void | undefined;

The problem with this is that we can't use ReturnType<OnHydrationEnd> anymore:
// error TS1064: The return type of an async function or method must be the global Promise<T> type. Did you mean to write 'Promise<void>'?
const onHydrationEnd: OnHydrationEnd = async (): ReturnType<OnHydrationEnd> => {
  console.log('Hydration finished; page is now interactive.')
}

This is why we stopped mixing Promise<something> | something in return-types. We could say it's OK not to support ReturnType<HookThatReturnsVoid> but I would prefer just providing the async variant instead.

Ah, you're right, The return type of an async function or method must be the global Promise<T> type was the issue and I just checked: it's also problematic even for a return type void.
Ok let's go with your with your suggestion. In other words, looking at the list:
  Guard,
  GuardSync,
  OnBeforePrerenderStart,
  OnBeforePrerenderStartSync,
  OnBeforeRender,
  OnBeforeRenderSync,
  OnBeforeRoute,
  OnBeforeRouteSync,
  OnHydrationEnd,
  OnPageTransitionEnd,
  OnPageTransitionStart,
  OnPrerenderStart,
  OnPrerenderStartSync,
  OnRenderClient,
  OnRenderClientSync,
  OnRenderHtml,
  OnRenderHtmlSync,
  Route

Only Route should be sync by default (i.e. Route | RouteAsync). All other hooks should be HookName | HookNameSync (async by default).

we try to make examples use async variants as much as we can. What do you think?

Agreed. I think it's easier for the user if we keep things consistent.

Idea: we provide OnBeforeRenderAsync | OnBeforeRenderSync instead of OnBeforeRender | OnBeforeRenderSync. More verbose but clearer.
I think I've a slight preference for that. Thoughts?
Let me think this a bit.

Only Route should be sync by default (i.e. Route | RouteAsync). All other hooks should be HookName | HookNameSync (async by default).

Actually, there is a caveat I think. When browsing Vike's source code, I came to the conclusion that the following hooks are sync-only: route, onPageTransitionStart, onPageTransitionEnd. The reason is that there is no await being done in Vike's source code for exactly these three hooks, so I think there shouldn't be an async variant for them. Or am I mistaken? I can dig that up again if needed.

I think I've a slight preference for that.

Yes, I prefer explicit Async/Sync as well

Yes, I prefer explicit Async/Sync as well

👍 Ok let's move with that then: we can add an alias type OnBeforeRender = OnBeforeRenderAsync later on anyways 👌.

Let me check about onPageTransition*.

Fixed: https://github.com/vikejs/vike/commit/58f822b8f8880f9168478a1956ff8ed6997d06da.

Yes, I prefer explicit Async/Sync as well

Since we're explicit, I think we don't need to use async hooks in all examples. Some hooks are trivial and obviously sync: I think it's fine to use the *Sync type for them.

@brillout I think it looks good now :)

A real beauty 😍
I've added a couple of commits (see the entire commit messages); let me know if you object. (I'm actually not 100% sure about my type Route = RouteSync commit but it feels weird to offer a RouteSync if RouteAsync is deprecated. Not sure. I guess we can keep my commit and experiment how we feel about it later on (we can always deprecated Route in favor of RouteSync).)
I'm 👍 for proceeding. Btw. we can already merge; we can add docs and update examples in an extra PR. Whatever works best for you.

(all the rest looks good to me :+1:)

(We can ignore Cloudflare's CI failing: we've reached Cloudflare's free tier daily rate liming.)

Merged 🎉
Thanks for the PR. Can't wait to release these really neat types.


Print test results on connected serial/terminal

Implement final print of results as plaintext and/or CSV output
sample, packaging as mere values, Group, Test, RepNo ecc.
A;1;1;-3;1;3.245;44,233;11,100;

done with commit 85277b5


Change the Router.serve method

Codecov Report

Merging #40 (b365190) into master (ea74102) will decrease coverage by 0.47%.
The diff coverage is 89.58%.

@@            Coverage Diff             @@
##           master      #40      +/-   ##
==========================================
- Coverage   98.66%   98.19%   -0.48%
==========================================
  Files          15       16       +1
  Lines        1048     1052       +4
==========================================
- Hits         1034     1033       -1
- Misses         14       19       +5

Flag
Coverage Δ

unittests
98.19% <89.58%> (-0.48%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

lib/response.lua
100.00% <ø> (ø)

lib/request.lua
73.33% <73.33%> (ø)

lib/router.lua
99.01% <96.96%> (-0.99%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea74102...b365190. Read the comment docs.


fix(build): Fix installation on Windows boxes with node@6.

Per https://github.com/randytarampi/lwip/commit/ee4352a.

:tada: This PR is included in version 3.0.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


fix(svg): remove xmlns

Summary

xml namespace is not required for svg elements (and has never been https://stackoverflow.com/a/18468348/3185307), only for svg documents.
Result

This saves a probably insignificant amount of bundlesize
xmlns nowhere left

bundlesize saved is .04 kb by the way :D


chron expression interpreter

aid time chron

Completed in https://github.com/Timmoth/aid-cli/commit/4bc3ae7e19fd6ece0cfe171c367b446d2883a1db


inmem: allow streaming of metrics as intervals complete

Related to one of the items in https://github.com/hashicorp/consul/issues/10320
We use the InMemSink to capture metrics for the debug archive, but the previous implementation resulted in us missing a lot of metrics because only 1/3 of the intervals were captured. Attempting to request the metrics at each interval is a challenge (maybe impossible), so instead this PR adds for an HTTP handler that sends metrics each time an interval completes.

I pushed another commit which reduces the interface that Stream uses. This should make it easier to integrate into an API handler. It would allow someone to customize the encoding and would also allow someone to use the method with a different type of API.


[HW] TikTok Clone

Name: NAMAN BHATEJA
Email<EMAIL_ADDRESS>Linkedin Profile: https://www.linkedin.com/in/naman-bhateja-018392171/
screenshots:-

Hey NAMAN BHATEJA, great job! Congrats! Here is your badge! https://api.badgr.io/public/assertions/PCc1nyuiRO6GO4n6piE8Jg


Add an --exit-non-zero-on-fix equivalent to the formatter.

Title says it all really. --exit-non-zero-on-change? I think this would be useful when using tools like pre-commit with CI in the same way as the checker's --exit-non-zero-on-fix.

How does your setup look like? Do you use --diff or --check? The formatter already exits with 0 when not using --check or --diff.
How do you detect changes when using ruff in CI when using --diff or --check when it exists with 0?

I already use --check and I know the formatter exits 0 without it. What I'm suggesting is a way to make it format and exit non-zero if anything changed in the same way as check --fix --exit-non-zero-on-fix.

I already use --check and I know the formatter exits 0 without it. What I'm suggesting is a way to make it format and exit non-zero if anything changed in the same way as check --fix --exit-non-zero-on-fix.

ruff format --check has different semantics than ruff check --fix. It only checks if files would be reformatted but it doesn't change the files on disk. You want to use ruff format (without check) if you want to format AND persist the files.

Updated my previous comment.

Oh sorry. I misunderstood you. If I understand you correctly, you're asking that ruff format --check exits with a non-zero exit code if it would reformat any files but it would also write the changes back to disk so that running ruff format isn't necessary locally.
I'm a bit torn where to integrate this functionality best. We could either to ruff format --exit-non-zero-on-fix (undecided on the name of the flag) without using check. However, the option would be incompatible when using with --check or --diff.
Did you use black before? If so, how did your setup look like?

We could either to ruff format --exit-non-zero-on-fix (undecided on the name of the flag) without using check. However, the option would be incompatible when using with --check or --diff.

This seems like a reasonable approach if we do implement it. It isn't incompatible with --check though, it just has no effect. With --diff I don't know our exit code behavior.

If I understand you correctly, you're asking that ruff format --check exits with a non-zero exit code if it would reformat any files but it would also write the changes back to disk so that running ruff format isn't necessary locally.

Exactly.

Did you use black before? If so, how did your setup look like?

I used black, just not with this behaviour. It was actually ruff's own --exit-non-zero-on-fix that gave me this idea for the formatter.

Hm so ruff format is all you should need in pre-commit? Having pre-commit tooling check for differences makes a lot more sense than adding more exit code flags to Ruff.

 How did I overlook the fact that pre-commit has its own exit code? Idiot. 😔

Good to hear it works :) I don't think this is justified then, I'm not sure what use-case would require this behavior.

@zanieb I do have a need for this, so perhaps this could be re-opened? My use case: I can't use pre-commit, because my project is a subdirectory in a larger git repo ("monorepo"). pre-commit decided to not support this use case, so I have my own "pre-commit" script, which indeed, I want to fail if it makes changes. Currently I wrote this:
if ! ruff format --check .; then ruff format .; false; fi

but having a --exit-non-zero-on-fix would be much nicer.
If this is approved, I will be happy to try and implement this.
Thanks!

@noamraph I'm trying to understand your use case and I'm sorry if this is obvious but my bash knowledge is rather terrible :sweat:
What I understand from the script is that it runs ruff format . only if ruff format --check failed. Can you explain the reason for it? I would also be interested to understand what functionality you miss from using ruff format --diff. It doesn't write the files as ruff format does.

@MichaReiser it's convenient for me to have one script which formats files and returns a nonzero code if it modified files. This is the behavior of the pre-commit tool. It's convenient because I use the same tool on my machine before committing, to format all the files, and also in CI, to block a pull request which isn't formatted according to the rules. For running on my machine, I need it to format the files. For running in CI, I need it to return nonzero if files aren't formatted. It's convenient to combine the two uses.

the use case for us would be to have a single format makefile target that can be used both for:

developers formatting their local repo
CI jobs checking the formatting

something like
.PHONY: format
format:
	poetry run ruff format . --exit-error-on-fix

without this we need separate targets for dev and CI
.PHONY: format-check
format-check:
	poetry run ruff format . --diff

.PHONY: format
format:
        poetry run ruff format .

@zanieb I have another use case: I'm trying leftcommit (a pre-commit alternative) and it doesn't check if the files were changed.
If you want, I hope I can submit a PR.

Also need this.  Like above its fine to make the changes "ruff format" - we just need to know if it ended up making any changes...
If "ruff format" returned a 1 if it made changes then we know.  We stop the pre-commit.  Developer does a git status and can see we already fixed some things.  They review and do a commit and all is good.
Otherwise, as mentioned before we have to run ruff twice.  (Though since it is so much faster than black that isn't as bad as it could be!  :)

I have a similar need.   I'm moving to moon for monorepo task management, and it has most of what I need for a pre-commit alternative, and by eliminating pre-commit I can have a single source of truth for task execution.
As a developer running a formatter on pre-commit, I want the formatter to format my code, but I also want pre-commit to abort so that the user can review the changes that were made (same behavior as with pre-commit tool).   As it stands, I have to run ruff format twice:  with and without --check.


Broken link: The Network Connection Status Icon

Broken link in section 14: The Network Connection Status Icon blog post on Technet was not migrated to the new Tech Community site
http://blogs.technet.com/b/networking/archive/2012/12/20/the-network-connection-status-icon.aspx

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9abd3113-b803-87b6-ef7f-6893bcc5ceb5
Version Independent ID: 6f7ca1ee-d19c-42bb-7db5-6158fc160e32
Content: Manage connections from Windows operating system components to Microsoft services (Windows 10)
Content Source: windows/privacy/manage-connections-from-windows-operating-system-components-to-microsoft-services.md
Product: w10
Technology: windows
GitHub Login: @Dansimp
Microsoft Alias: dansimp

@officedocsbot assign @jvsam

Hi @amandeepmand , thank you for letting us know about the broken link. I will get this issue over to the Windows writing team for correction.
We want users to have the best Windows 10 experience by ensuring that contents of the Microsoft Docs are useful, up-to-date and error-free. Suggestions, comments and feedback to improve the docs are always welcome!

Hello @amandeepmand ! The link has been updated on the site and it's now redirecting to the new Networking blog (main page). We're checking if the old content has been migrated to the blog site, if so, we will request for the new URL to be added. Thank you again for taking the time to share your feedback and for being part of the Microsoft Docs community!

Hi again @amandeepmand, a new update to this documentation via Pull Request (PR) #4298 has been approved and merged. It will be displayed at docs.microsoft.com in the next scheduled publishing run. You can verify the changes at this URL: https://docs.microsoft.com/windows/privacy/manage-connections-from-windows-operating-system-components-to-microsoft-services. We will now close this issue but we'll provide an update here to inform you when the changes are live.
Thank you for your feedback to help improve the contents of this document and for being part of the Microsoft Docs community!

@officedocsbot close

Hi @amandeepmand, we'd like to inform you that the changes are now displayed at docs.microsoft.com. You can verify by visiting this URL: https://docs.microsoft.com/windows/privacy/manage-connections-from-windows-operating-system-components-to-microsoft-services. Thank you for being part of the Microsoft Docs community!


[timm][inference][float16][regression] res2net50_14w_8s fail accuracy

res2net50_14w_8s fail accuracy at float16 in inference mode. It passes with v2.1.

There is no accuracy difference between A100 and PVC on this model with pytorch 2.1, and PyTorch upstream goal is on par with CUDA about accuracy and functionality pass rate. So won't fix for PyTorch upstream.


Use rtd for docs

Codecov Report

Merging #167 (30a8a29) into master (0b15808) will not change coverage.
The diff coverage is n/a.

@kntkb @mikemhenry the build doc is still failing, is it supposed to be fixed in this PR?

@mikemhenry @kntkb oh nvm I see that we just need to disable the previous build docs GA

Yeah, this one should be ready to go https://espaloma--167.org.readthedocs.build/en/167/
Anybody opposing merging this as it is now?

let's merge once we can disable "Build Doc / docs (push)" GA

Sorry! I was just about done with this PR and got distracted, I will clean it up now that it is building!

Okay auto merge should take care of this, I will keep an eye on it


E2 (update example)

Might fix the build

@RedstoneWizard08 you updated to a commit that does not exist

I know, I'll fix it at some point, there were changes to the example.


consistency: central_version => nightly version

Why, OOI?  This doesn't look particularly consistent to me...

@calixteman asked for this change in
https://github.com/mozilla/relman-auto-nag/pull/168
https://github.com/mozilla/relman-auto-nag/pull/168/commits/0476700a28e7e4c9be1c796a464fab117faccc86
So, just to be consistent with this change

to be clear, I don't care :)

@jcristau accept or reject it, it's up to you.
Here the consistency is to have the channel names.

Well this leaves central in most places with nightly in a couple, I'd rather keep saying central everywhere.


Archives is broken

http://artsy.github.io/blog/archives/
we should make it like Ash's too: https://ashfurrow.com/blog/ - no one cares about the previews when looking at archives.

I also don't really care much for the date. Probably care more for the tags: perhaps something like
| --------------------------------------------------- |
| [author] - [post title]                      [tags] |
| --------------------------------------------------- |
| [author] - [post title]                      [tags] |
| --------------------------------------------------- |

chunking by months seems fine though

We should do the same for tag pages too

and maybe author pages.