refactor aggregates

Unify aggregate operators: remove Single/MultiGroupByAggregate, make StreamingAggregate as a subclass of Aggregate, remove CountAllAggregator
Introduce an abstraction of tuple hash table, make Aggregate/SymmetricHashJoin use it
Make aggregators column-oriented: the internal state contains columns of "basic" aggregation ops (e.g. SUM, COUNT), and "complex" aggregation ops which emit results are mapped to them (e.g. AVG). Only internal aggregations are applied to input tuples. Results are generated by emit aggregations.

Works with https://github.com/uwescience/raco/pull/553.

Coverage decreased (-0.8%) to 26.58% when pulling fe7e3afca2fbe63fbad577256c8fa89d231e1e44 on refactor_agg into 48d79cacdc3a6678219568af228a08f4206572a0 on master.

Coverage decreased (-0.8%) to 26.571% when pulling 1dd35d5ab0b43982f543fe5fadc948d98c3cb207 on refactor_agg into 48d79cacdc3a6678219568af228a08f4206572a0 on master.

Coverage decreased (-0.8%) to 26.601% when pulling<PHONE_NUMBER>d58746ff09ea53a1573c064581ef65 on refactor_agg into 48d79cacdc3a6678219568af228a08f4206572a0 on master.

Could you find a way to avoid re-opening a PR that's under review? I have to copy/paste a few dozen comments by hand into the new PR...

For TupleHashTable, actually I'm expecting a lot of collisions from joins, since one join key may correspond to many tuples. Aggregates are fine. Since usually joins have much larger hash tables, I was not very motivated to do the optimization. I agree that it is a cool design if collisions are rare.

Right, that's why I suggested that maybe we should have different interfaces/impls for TupleHashTable based on whether keys are expected to reference unique indices. For aggregates this will be true, for joins no. I don't think we should necessarily assume that aggregate groups will be coarse-grained; someone will eventually run a query where that is not the case. Anyway, I opened https://github.com/uwescience/myria/issues/876 for this issue, so we can drop it from the PR :)

Coverage decreased (-0.9%) to 26.195% when pulling e34654b0a48a28be3e80c7b644ab8f8f3c79178e on refactor_agg into d282c443dc9ff4cefeb974201f73344ce6e10358 on master.

Coverage decreased (-0.8%) to 26.305% when pulling bf05ea2b19621ccabd50ca18b836997a82467502 on refactor_agg into d282c443dc9ff4cefeb974201f73344ce6e10358 on master.

I have addressed all comments and also adopted new changes in master. (Is there a way to make coverall pass? I didn't change much...) There are several things that I need to confirm with @parmitam, but the new commit is good to be reviewed first.

I will do #876 and #878 in other PRs.

Coverage decreased (-0.8%) to 26.273% when pulling 05618d237d3239ffe924c30ffc43143920aeee83 on refactor_agg into d282c443dc9ff4cefeb974201f73344ce6e10358 on master.

I also cherry-picked two commits from #867 here since one is relevant.

Coverage decreased (-0.8%) to 26.264% when pulling 716fbfd87ddd80a9fb90803f67fca433fd14a432 on refactor_agg into d282c443dc9ff4cefeb974201f73344ce6e10358 on master.

Coverage decreased (-0.8%) to 26.275% when pulling 1a1074c189cdee5152032b33bd119f9853812823 on refactor_agg into d282c443dc9ff4cefeb974201f73344ce6e10358 on master.

Coverage decreased (-0.8%) to 26.285% when pulling 72f6b9bbc4329688602d9381944fbb670ef021cc on refactor_agg into d282c443dc9ff4cefeb974201f73344ce6e10358 on master.

Coverage decreased (-0.8%) to 26.266% when pulling fea07e1218a86e398fbd2c0155f61081511f3b65 on refactor_agg into d282c443dc9ff4cefeb974201f73344ce6e10358 on master.

Coverage decreased (-0.8%) to 26.243% when pulling fea07e1218a86e398fbd2c0155f61081511f3b65 on refactor_agg into d282c443dc9ff4cefeb974201f73344ce6e10358 on master.

I think coverall was messed up since there is no way to change the coverage by 0.8% through modifying a few lines of Java docs...

Coverage decreased (-0.8%) to 26.263% when pulling 155f22946ada327566de59e16e806882ad6ac61b on refactor_agg into d282c443dc9ff4cefeb974201f73344ce6e10358 on master.


[RFC][DNM] device: device api extension proposal

A draft showing how to handle external API with devices.
Short summary:

device structure gets additional field - a pointer to extension api structure which contains mask of capabilities and pointer to data. if more than one extension is supported then data is an array of pointers associated with each api. https://github.com/zephyrproject-rtos/zephyr/blob/47ec30148aeedc61dcce6d2d1da3f3b0873473b6/include/device.h#L302
device.h gets internal api for checking if device supports given extension (z_device_ext_api_supported(dev, bitmask)) and getting data associated with that extension (z_device_ext_api_get_data(dev, bitmask)) https://github.com/zephyrproject-rtos/zephyr/blob/47ec30148aeedc61dcce6d2d1da3f3b0873473b6/include/device.h#L328
external api (onoff service in that example) implements simple inline like device_get_onoff_service(dev, &srv) https://github.com/zephyrproject-rtos/zephyr/blob/47ec30148aeedc61dcce6d2d1da3f3b0873473b6/include/sys/onoff.h#L434

Example:
err = device_subsys_get_onoff_service(clock, subsys, &srv);
if (err < 0) {
    LOG_ERR("Failed to get device onoff service (err: %d)", err);
}

err = onoff_request(srv, &client);

Signed-off-by: Krzysztof Chruscinski<EMAIL_ADDRESS>

As for spi and i2c, you are pushing a specialized feature (or if you prefer: a "not-generic enough one") to a generic node. Imo, your are taking the problem the wrong way. You are not responding to what we have been waiting for: a transaction manager RFC. If you do that you'll see what belongs to the transaction manager and what may belong to something else below.

@tbursztyka we already have similar case. There is an onoff service and clock_control driver. I would like to use onoff service on clock device and not extended clock control API. How to do that? Maybe you could propose something as you clearly have something in mind and it is different than this proposal.
Transaction manager is another case. We can have RFC with API for that but still there will be an issue how to use it with the device.

Could we please start developing a consensus on what we need before we start proposing technical solutions that have to be reviewed without anything to judge them against?
Also: https://docs.zephyrproject.org/latest/reference/drivers/index.html#device-specific-api-extensions
which might be extended to subsystem-specific-api-extensions if that seems like a good solution for whatever the problem is.

@pabigot issue created #22415

API 2020-07-07: Moved from triage to in-progress; relates to #23371.


WIP: Scots gaelic

First effort at Scots Gaelic translation.
Tha mi air leth taingeil airson gach ceartachadh agus moladh.
Sono estremamente grato per tutte le correzioni e i suggerimenti.
Je suis extrêmement reconnaissant pour toutes les corrections et suggestions.
Für alle Korrekturen und Anregungen bin ich überaus dankbar.
Any and all corrections and suggestions gratefully received.

Best effort of my limited learning.  If you feel like squashing and merging it might catch the eye of a native speaker one day to aid learning.  I'll keep trying to research and improve this while also learning the tooling.
btw my editor also snipped some trailing whitespace (blame Martin ;-P )


Add flag for image builder table tabs

also add a feature flag- to control the way we show tabs -related to that story https://issues.redhat.com/browse/THEEDGE-3367

Code looks good to me but when I run it the edge table is just empty. Is that expected?

pr is blocked until this pr https://github.com/RedHatInsights/image-builder-frontend/pull/1152 will merge


Choose a unit testing framework and write tests

There are so many unit test frameworks out there. Choose one and write unit tests.

Don't forget to use Travis CI and use a badge in README.md.


Change Debug/Display on BlindingKey / SpendingKey etc. to output a fingerprint or something

We don't want these leaking in the logs.

hey, I would like to work on this issue. just have to change all the keys displayed in various logs to output fingerprint by hashing it to a smaller value using function right? and also where is the good place to find which logs are storing keys?

Yes, though you don't need to look for specific logs. Only Display and Debug trait impls need to be changed.

Looks done.


⚠ Processors has degraded performance

In 339047b, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 87 ms

Resolved: Processors performance has improved in 5370c85.


Add custom GCP resources and filters

Add custom resources & filters created by @aq17 to implement the gcp_foundations_bench-1.2.0

these are upstream changes?

These are the custom changes made by @aq17, based on the <IP_ADDRESS> tag of the upstream

Ah these were additional changes that we did not open PR on the main repo?


Use string split instead of regex.

~30% speedup (python 3.7)

Cool! This is better than #8 for another reason too...

If sep is not specified or is None, a different splitting algorithm is applied: runs of consecutive whitespace are regarded as a single separator

https://docs.python.org/2/library/stdtypes.html#str.split
That's why the original used re: to consume consecutive whitespace.

Released v1.1.3 with this change: https://pypi.org/project/bgpdumpy/1.1.3/

Thank you!


'right' and 'bottom' issue in Phaser.Ellipse.prototype

https://github.com/photonstorm/phaser/blob/dev/src/geom/Ellipse.js#L196
this.width = this.x + value;
should be changed to:
this.width = value - this.x;
https://github.com/photonstorm/phaser/blob/dev/src/geom/Ellipse.js#L238
this.height = this.y + value;
should be changed to:
this.height = value - this.y;

Fixed locally in Phaser :)

Split to a separate issue please.


[Feature] Add option to associate an api key to a specific user

Use case
I want to grant other persons using my hosted server the ability to manage their own user and the connected devices but currently i can only give them full admin privileges, which i do not want for security reasons.
Description
Add an optional parameter to the cli for example headscale apikeys create --user testUser which only grants admin rights to the user testUser and the associated devices.
Contribution

[X] I can write the design doc for this feature
[X] I can contribute this feature

How can it be implemented?

Add the optional parameter --user to theheadscale apikeys create cli
update the api endpoints to only accept get and post request to users associated to the apikey (if no user is assigned, grant all privileges)

Hi, I found this idea to be interesting too. As a further improvement, can you also allow for view-only/read-write keys? This may be an optional --view-only flag that only allow GET requests to the Headscale server. I have a use case where I want to monitor online nodes but not control them, so all of this would be a great addition. Thanks!

Hi, I found this idea to be interesting too. As a further improvement, can you also allow for view-only/read-write keys? This may be an optional --view-only flag that only allow GET requests to the Headscale server. I have a use case where I want to monitor online nodes but not control them, so all of this would be a great addition. Thanks!

I definitely see how this would be useful. If this feature request is being approved, I can probably implement this without much extra effort

not stale

no


Code blocks in @typedef have no indentation

Search terms
code block typedef indentation
Expected Behavior
The rendered code block preserves indentations
Actual Behavior
The rendered code block does not preserve indentations
Steps to reproduce the bug

https://github.com/jackens/jc/blob/master/src/jcmp.js#L78
https://jackens.github.io/jc/modules/jcmp.html#JCMP_CONFIG

Environment

Typedoc version: 0.22.7
TypeScript version: 4.4.3
Node.js version: 14.17.4
OS: Windows 10 WSL

It turns out that this is a bug in TypeScript: https://github.com/microsoft/TypeScript/issues/46750


inlining not working with base tag

It looks like the base tag isn't honoured. inliner reports "no such file":
gist

I appreciate this was months ago, but do you want to take a shot at a PR? The tests are very easy to setup (follow the contributing doc - just need two HTML files). Thanks so much!


List of referencing_class values is not exhaustive

There are other possible values of referencing_class that is shown here. It's misleading that only some are mentioned without making it clear it's only an excerpt of the full list.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6ada44a9-e5a1-65b3-3bb2-9ebac215e5ab
Version Independent ID: 470c3c15-d19d-e29f-c5f9-0824881deaac
Content: sys.sql_expression_dependencies (Transact-SQL) - SQL Server
Content Source: docs/relational-databases/system-catalog-views/sys-sql-expression-dependencies-transact-sql.md
Product: sql
Technology: system-objects
GitHub Login: @stevestein
Microsoft Alias: sstein

@brikken - thanks for raising this issue. We've created an internal work item (1595947) to investigate and potentially update the article.
We're going to close to this item and track with our internal work item. Feel free to continue to add any additional info or comments here.
Thanks again for pointing this out.


Setting up routing endpoint for IoT Hub for service bus fails

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior.
Expected behavior
A clear and concise description of what you expected to happen.
Environment summary
Install Method (e.g. pip, interactive script, apt-get, Docker, MSI, edge build) / CLI version (az --version) / OS version / Shell Type (e.g. bash, cmd.exe, Bash on Windows)
Additional context
Add any other context about the problem here.

I'm a Sr Content Dev working on expanding the IoT Hub routing tutorial (https://docs.microsoft.com/en-us/azure/iot-hub/tutorial-routing) to add ARM templates, and to add routing setup scripts in PowerShell and CLI (and ARM). Everything works except CLI.
When setting up the routing endpoint using CLI for the storage account, it fails.
az iot hub routing-endpoint create
--connection-string $storageConnectionString
--endpoint-name $endpointName
--endpoint-resource-group $resourceGroup
--endpoint-subscription-id $subscriptionID
--endpoint-type $endpointType
--hub-name $iotHubName
--container $containerName
--resource-group $resourceGroup
Here's the error:
robin@Azure:~$ az iot hub routing-endpoint create --connection-string $storageConnectionString --endpoint-name $endpointName --endpoint-resource-group $resourceGroup --endpoint-subscription-id $subscriptionID --endpoint-type $endpointType --hub-name $iotHubName --container $containerName --resource-group $resourceGroup
Error occurred in request., RetryError: HTTPSConnectionPool(host='management.azure.com', port=443): Max retries exceeded with url: /subscriptions/a4295411-5eff-4f81-b77e-276ab1ccda12/resourceGroups/ContosoResources5/providers/Microsoft.Devices/IotHubs/ContosoTestHub4791?api-version=2018-12-01-preview (Caused by ResponseError('too many 500 error responses',))
Traceback (most recent call last):
File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 731, in urlopen
body_pos=body_pos, **response_kw)
File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 731, in urlopen
body_pos=body_pos, **response_kw)
File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 731, in urlopen
body_pos=body_pos, **response_kw)
File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 711, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
File "/opt/az/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='management.azure.com', port=443): Max retries exceeded with url: /subscriptions/a4295411-5eff-4f81-b77e-276ab1ccda12/resourceGroups/ContosoResources5/providers/Microsoft.Devices/IotHubs/ContosoTestHub4791?api-version=2018-12-01-preview (Caused by ResponseError('too many 500 error responses',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/az/lib/python3.6/site-packages/msrest/universal_http/requests.py", line 137, in send
**kwargs)
File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 507, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='management.azure.com', port=443): Max retries exceeded with url: /subscriptions/a4295411-5eff-4f81-b77e-276ab1ccda12/resourceGroups/ContosoResources5/providers/Microsoft.Devices/IotHubs/ContosoTestHub4791?api-version=2018-12-01-preview (Caused by ResponseError('too many 500 error responses',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 328, in execute
raise ex
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 386, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 379, in _run_job
six.reraise(*sys.exc_info())
File "/opt/az/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 356, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 171, in call
return self.handler(*args, **kwargs)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/init.py", line 448, in default_command_handler
return op(**command_args)
File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/iot/custom.py", line 567, in iot_hub_routing_endpoint_create
return client.iot_hub_resource.create_or_update(resource_group_name, hub_name, hub, {'IF-MATCH': hub.etag})
File "/opt/az/lib/python3.6/site-packages/azure/mgmt/iothub/operations/iot_hub_resource_operations.py", line 199, in create_or_update
**operation_config
File "/opt/az/lib/python3.6/site-packages/azure/mgmt/iothub/operations/iot_hub_resource_operations.py", line 140, in _create_or_update_initial
response = self._client.send(request, stream=False, **operation_config)
File "/opt/az/lib/python3.6/site-packages/msrest/service_client.py", line 336, in send
pipeline_response = self.config.pipeline.run(request, **kwargs)
File "/opt/az/lib/python3.6/site-packages/msrest/pipeline/init.py", line 197, in run
return first_node.send(pipeline_request, **kwargs)  # type: ignore
File "/opt/az/lib/python3.6/site-packages/msrest/pipeline/init.py", line 150, in send
response = self.next.send(request, **kwargs)
File "/opt/az/lib/python3.6/site-packages/msrest/pipeline/requests.py", line 72, in send
return self.next.send(request, **kwargs)
File "/opt/az/lib/python3.6/site-packages/msrest/pipeline/requests.py", line 137, in send
return self.next.send(request, **kwargs)
File "/opt/az/lib/python3.6/site-packages/msrest/pipeline/init.py", line 150, in send
response = self.next.send(request, **kwargs)
File "/opt/az/lib/python3.6/site-packages/msrest/pipeline/requests.py", line 193, in send
self.driver.send(request.http_request, **kwargs)
File "/opt/az/lib/python3.6/site-packages/msrest/universal_http/requests.py", line 328, in send
return super(RequestsHTTPSender, self).send(request, **requests_kwargs)
File "/opt/az/lib/python3.6/site-packages/msrest/universal_http/requests.py", line 140, in send
raise_with_traceback(ClientRequestError, msg, err)
File "/opt/az/lib/python3.6/site-packages/msrest/exceptions.py", line 51, in raise_with_traceback
raise error.with_traceback(exc_traceback)
File "/opt/az/lib/python3.6/site-packages/msrest/universal_http/requests.py", line 137, in send
**kwargs)
File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 507, in send
raise RetryError(e, request=request)
msrest.exceptions.ClientRequestError: Error occurred in request., RetryError: HTTPSConnectionPool(host='management.azure.com', port=443): Max retries exceeded with url: /subscriptions/a4295411-5eff-4f81-b77e-276ab1ccda12/resourceGroups/ContosoResources5/providers/Microsoft.Devices/IotHubs/ContosoTestHub4791?api-version=2018-12-01-preview (Caused by ResponseError('too many 500 error responses',))
robin@Azure:~$
Sapan Saxena verified on Friday 2/15 that this didn't work for him either. Today (Tuesday 2/19), he says it works for him. It did not work for me this morning, but when I tried it around 2 p.m., it also didn't work, but just never finished running. I left it for 15 minutes in Cloud Shell, and finally have to Ctrl-C to get past it. I checked the portal, it didn't add the endpoint.
If you want the CLI script, ping me internally.  I'm going to post the second problem setting up a routing endpoint for a service bus queue separately because it's a whole different problem.
Thx,
Robin

One more comment: This is related to GH issue 8579 about setting up an endpoint using CLI for a storage account. Different error, though.

The awesome @v-Ajnava has mucked around with this and found that the problem is that when you retrieve the connection string, it is returned with double-quotes around it. Then when you turn around and send it into this cmdlet, it fails. If you change the double quotes to single quotes, it works. It would be nice if that cmdlet would accept connection string in double quotes. Is that something that can be changed, or do I have to figure out how to do string manipulation in CLI to change them to single quotes?
/cc: @ash2017

Closing as the solution is the same describe din #8579.


Pro: Mask Input - Suffix and prefix is not reative

Reproduction
https://formkit.link/451004d401d0cb8ab3eaffe1370c9480
Describe the bug
Using a Mask Input, setting  :suffix prop to a ref and mutate the ref. The values is not updated.
Environment
• OS: Mac
• Browser Chrome
• Version 121

Any news on this? It's a Pro component and still nothing... :(

I should have specified there is a workaround. just reboot the input by dynamically setting the key.


Modified SingletonPropertyNameTransformer.cs to strip column suffixes…

…, convert property names to PascalCase, and handle special ID suffix property names.

Is this kind of how you wanted me to do the constructor?
class SingletonPropertyNameTransformer : IPropertyNameTransformer
    {
        private readonly char columnSeperator;

        public SingletonPropertyNameTransformer(char columnSeperator)
        {
            this.columnSeperator = columnSeperator;
        }

Question: When you run Upswing does it generate the CLR datatypes for you in the entity class? For example I get the following.
public partial class Players_T
{
		public  PlayerId { get; set; }
		public  TeamId { get; set; }
}


Can the robomaster ep model be exported to another common format like urdf or sdf?

Hi, Jeguzzi. I'd like to use the model in another simulator. Is there a way to use them, for example in gazebo or pybullet? Thank you very much:)

Yes, URDF/xacro models are available in the ROS2 package https://github.com/jeguzzi/robomaster_ros/tree/main/robomaster_description, and documented on https://jeguzzi.github.io/robomaster_ros/robomaster_description.html

Thank you so much for your prompt reply, this is really helpful!


About the training process of the model

Dear @cwmok
I sincerely appreciate your excellent work, but I have encountered several issues while reproducing this paper.
Firstly, regarding the number of iterations, I found that the iteration number in the code of this paper is 160001. Is this the actual number of iterations during the training process?
Secondly, I am currently reproducing the first part of Train-C2FVit_pairwise, but I have found that DiceLoss defined in C2FVit_model and Dice defined in Train-C2FVit_pairwise do not seem to be used during the training process. I would like to know how Dice should be used.
Thirdly, after downloading the Code, I saw two columns of data in the C2FViT.affineeCOM_pairwise _. txt file in the folder log. One column was Dice, and the other column was NCC. Is this correct my understanding? But why does the data obtained in the end not match the data provided by Tabe in the paper?

Hi @RexEmperor,
Thanks for your interest in our work.

Firstly, regarding the number of iterations, I found that the iteration number in the code of this paper is 160001. Is this the actual number of iterations during the training process?

Yes. After training, we select the best model based on the result of the validation set.

Secondly, I am currently reproducing the first part of Train-C2FVit_pairwise, but I have found that DiceLoss defined in C2FVit_model and Dice defined in Train-C2FVit_pairwise do not seem to be used during the training process. I would like to know how Dice should be used.

Train_C2FViT_pairwise_semi.py and Train_C2FViT_template_matching_semi.py are the semi-supervised models, which will use Dice loss in training. The other scripts are fully unsupervised. See Readme.md for more details.

Thirdly, after downloading the Code, I saw two columns of data in the C2FViT.affineeCOM_pairwise _. txt file in the folder log. One column was Dice, and the other column was NCC. Is this correct my understanding? But why does the data obtained in the end not match the data provided by Tabe in the paper?

Both columns are Dice. The first column is the dice of the "selected" anatomical structures, while the second column is the dice of the whole brain (binary) semengtation. See Code  here for more details. More importantly, the result reported in the log file is the result of the validation dataset. In our paper, we report the result of the test set, which is a disjoint dataset apart from the validation set.

Dear @cwmok
Thank you for patient answering.
I think I made a mistake. I did not untangle the validation part of the annotation. So, all I need to do is untangle the validation part of the unsupervised training code to obtain results similar to those in the Log file?
But I didn't find any part about calculating Dice scores in the test.py file . Do I need to manually add it? By calculating Dice in the Test section, I can obtain results similar to those in the Table in the article, right?
Thank you again for your patient answer.

I think I made a mistake. I did not untangle the validation part of the annotation. So, all I need to do is untangle the validation part of the unsupervised training code to obtain results similar to those in the Log file?

Yes.

But I didn't find any part about calculating Dice scores in the test.py file . Do I need to manually add it?

No, you don't have to. The code for calculating the Dice score is provided.
def dice(im1, atlas):
    unique_class = np.unique(atlas)
    dice = 0
    num_count = 0
    for i in unique_class:
        if (i == 0) or ((im1 == i).sum() == 0) or ((atlas == i).sum() == 0):
            continue

        sub_dice = np.sum(atlas[im1 == i] == i) * 2.0 / (np.sum(im1 == i) + np.sum(atlas == i))
        dice += sub_dice
        num_count += 1
    return dice / num_count

By calculating Dice in the Test section, I can obtain results similar to those in the Table in the article, right?

Yes, you're correct!

Dear @cwmok
Thank you very much for your patience and prompt response. You have been a great help to me.
I have another question. If I want to obtain the data in the article Table, which Python file do I need to modify it based on?
Is it Test-C2FVit_pairwise or Train-C2FVit_pairwise?
If the modification is based on Test-C2FVit_pairwise, then I need to apply def dice to Test-C2FVit_pairwise and adopt a structure similar to the validation part of Train-C2FVit_pairwise?
If it is a modification based on Train-C2FViT.pairwise, would it be like me to follow the structure of the validation section and add a section for calculating the Dice score for the test?
Thank you again for your reply.

I have another question. If I want to obtain the data in the article Table, which Python file do I need to modify it based on?
Is it Test-C2FVit_pairwise or Train-C2FVit_pairwise? If the modification is based on Test-C2FVit_pairwise, then I need to apply def dice to Test-C2FVit_pairwise and adopt a structure similar to the validation part of Train-C2FVit_pairwise?

You will have to create a new python script, similar to the structure of the validation section. Test-C2FVit_pairwise takes only one pair of the images as input and outputs the affine-transformed image. It will not compute the Dice score.

Okay, I roughly understand. Do I need to import the trained model, then use it to register the prediction set and calculate the Dice score?
Your timely reply has been of great help to me, thank you very much.

@RexEmperor
Yes, you're correct.

Dear @cwmok
I followed the validation section and created a new code file for testing.
I would like to ask, what I get will be DSC(23) is it?
This is the code I wrote:
`
def test():
print("Testing C2FViT...")
model = C2F_ViT_stage(img_size=128, patch_size=[3, 7, 15], stride=[2, 4, 8], num_classes=12,
embed_dims=[256, 256, 256],
num_heads=[2, 2, 2], mlp_ratios=[2, 2, 2], qkv_bias=False, qk_scale=None, drop_rate=0.,
attn_drop_rate=0., norm_layer=nn.Identity,
depths=[4, 4, 4], sr_ratios=[1, 1, 1], num_stages=3, linear=False).cuda()
affine_transform = AffineCOMTransform().cuda()
init_center = Center_of_mass_initial_pairwise()

# OASIS
imgs = sorted(glob.glob(datapath + "/OASIS_OAS1_*_MR1/norm.nii.gz"))
labels = sorted(glob.glob(datapath + "/OASIS_OAS1_*_MR1/seg35.nii.gz"))

testing_generator = Data.DataLoader(Dataset_epoch(imgs, labels, norm=True, use_label=False),
                                     batch_size=1,
                                     shuffle=True, num_workers=4)

model_path = "/home/guoke/Downloads/C2FViT-main/Model/C2FViT_affine_COM_pairwise/C2FViT_affine_COM_pairwise_stagelvl3_160000.pth"
model.load_state_dict(torch.load(model_path))
model.eval()

for X, Y in testing_generator:
    X = X.cuda().float()
    Y = Y.cuda().float()

    if com_initial:
        X, _ = init_center(X, Y)

    X = F.interpolate(X, scale_factor=0.5, mode="trilinear", align_corners=True)
    Y = F.interpolate(Y, scale_factor=0.5, mode="trilinear", align_corners=True)

    warpped_x_list, y_list, affine_para_list = model(X, Y)

    imgs = sorted(glob.glob(datapath + "/OASIS_OAS1_*_MR1/norm.nii.gz"))[265:]
    labels = sorted(glob.glob(datapath + "/OASIS_OAS1_*_MR1/seg35.nii.gz"))[265:]

    valid_generator = Data.DataLoader(
        Dataset_epoch(imgs, labels, norm=True, use_label=True),
        batch_size=1,
        shuffle=False, num_workers=2)

    use_cuda = True
    device = torch.device("cuda" if use_cuda else "cpu")
    dice_total = []
    brain_dice_total = []

    dice_total = np.array(dice_total)
    brain_dice_total = np.array(brain_dice_total)

    print("\nTesting...")

    for batch_idx, data in enumerate(valid_generator):
        X, Y, X_label, Y_label = data[0].to(device), data[1].to(device), data[2].to(
            device), data[3].to(device)

        with torch.no_grad():
            if com_initial:
                X, init_flow = init_center(X, Y)
                X_label = F.grid_sample(X_label, init_flow, mode="nearest", align_corners=True)

            X_down = F.interpolate(X, scale_factor=0.5, mode="trilinear", align_corners=True)
            Y_down = F.interpolate(Y, scale_factor=0.5, mode="trilinear", align_corners=True)

            warpped_x_list, y_list, affine_para_list = model(X_down, Y_down)
            X_Y, affine_matrix = affine_transform(X, affine_para_list[-1])
            F_X_Y = F.affine_grid(affine_matrix, X_label.shape, align_corners=True)

            X_Y_label = F.grid_sample(X_label, F_X_Y, mode="nearest", align_corners=True).cpu().numpy()[0,
                        0, :, :, :]
            X_brain_label = (X_Y > 0).float().cpu().numpy()[0, 0, :, :, :]

            # brain mask
            Y_brain_label = (Y > 0).float().cpu().numpy()[0, 0, :, :, :]
            Y_label = Y_label.data.cpu().numpy()[0, 0, :, :, :]

            dice_score = dice(np.floor(X_Y_label), np.floor(Y_label))
            dice_total = np.append(dice_total,dice_score)

            brain_dice = dice(np.floor(X_brain_label), np.floor(Y_brain_label))
            brain_dice_total = np.append(brain_dice_total,brain_dice)

            dice_total = np.array(dice_total)
            brain_dice_total = np.array(brain_dice_total)

            print("Dice mean: ", dice_total.mean())
            print("Brain Dice mean: ", brain_dice_total.mean())

            with open(log_dir, "a") as log:
                log.write(f"{dice_total.mean()}, {brain_dice_total.mean()} \n")

`
`
if name == 'main':
parser = ArgumentParser()
parser.add_argument("--modelname", type=str,
dest="modelname",
default='../Model/C2FViT_affine_COM_pairwise/C2FViT_affine_COM_pairwise_stagelvl3_160000.pth',
help="Pre-trained Model path")
parser.add_argument("--datapath", type=str,
dest="datapath",
default='/home/guoke/Downloads/C2FViT-main/Data/Train_pairwise/',
help="data path for testing images")
parser.add_argument("--com_initial", type=bool,
dest="com_initial", default=True,
help="True: Enable Center of Mass initialization, False: Disable")
opt = parser.parse_args()
datapath = opt.datapath
com_initial = opt.com_initial
model_name = opt.modelname

if not os.path.isdir("../Dice"):
    os.mkdir("../Dice")

log_dir = "../Dice/" + model_name + ".txt"

with open(log_dir, "a") as log:
    log.write("Test Dice log for " + model_name + ":\n")

print("Training %s ..." )
test()

`
Could you please give me some advice? I ended up with a result of around 0.59, while in the paper it is around 0.66, which confuses me.

@RexEmperor
What is your initial DSC23? We selected 23 critical anatomical structures for evaluation, as shown below.

More importantly, as mentioned in the paper, we randomly select 3 MRI scans as an atlas in the OASIS dataset. It seems that your code is reporting the pairwise registration results but not comparing them to the 3 MRI atlases.
Remember to set com_initial=True and load the correct model ('C2FViT_affine_COM_pairwise_stagelvl3_118000.pth).

Dear @cwmok
Sorry, I seem to be getting confused.
I want to realize this experiment——Atlas-Based Registration (OASIS).
Then, I trained the model based on the file——Train_C2FViT_pairwise.py.
I want to get the results of this experiment in Table1 and Table2.

Because, as I read in the paper, “In the atlas-based registration with the OASIS dataset, 23 subcortical structures re included”.
So I took it for granted that as long as I write a test file like the validation section of the file to use for testing it would be fine, and it seems I was wrong about that now.
I thought "Pairwise image registration" was the task——Atlas-Based Registration (OASIS).In hindsight, I was wrong.
If I want to do this task——Atlas-Based Registration (OASIS), which step should I do now? Should the model I trained be the one that will eventually be used to perform that task? Is it that I don't need to train another model.
Thanks again for your patience. I know I'm bothering you, but I'm really trying to figure out the experiment for this paper, thank you so much yet!

More importantly, as mentioned in the paper, we randomly select 3 MRI scans as an atlas in the OASIS dataset. It seems that your code is reporting the pairwise registration results but not comparing them to the 3 MRI atlases.
So according to what you are saying is that I need to compare one image (moving image) with three randomly selected images (fixed image) and then find their average Dice score?

@RexEmperor

If I want to do this task——Atlas-Based Registration (OASIS), which step should I do now? Should the model I trained be the one that will eventually be used to perform that task? Is it that I don't need to train another model.

You will have to train two models, one with the centre of mass initialization and one without a centre of mass initialization.

So according to what you are saying is that I need to compare one image (moving image) with three randomly selected images (fixed image) and then find their average Dice score?

You're correct. For each image in the test set, we compare it with three randomly selected images (fixed image) and then compute their average Dice score over 23 selected anatomical structures.

You're correct. For each image in the test set, we compare it with three randomly selected images (fixed image) and then compute their average Dice score over 23 selected anatomical structures.

Are（23 selected anatomical structures） they already included in the downloaded dataset？

Dear @cwmok
About the task of template match, I preprocessed the dataset like you did：.
target_dir = 'path/to/your/directory'

contents = os.listdir(target_dir)

folder_names = [name for name in contents if os.path.isdir(os.path.join(target_dir, name))]

for name in folder_names:
    dir_path = os.path.join(folder_names,name)
    file_path = os.path.join(file_path,"seg4.nii.gz")

    seg4_lable = nibabel.load(file_path)
    seg4_lable_npy = seg4_lable.get_fdata()

    seg4_lable_oasis = np.zeros(seg4_lable_npy.shape,dtype=seg4_lable_npy.dtype)
    seg4_lable_oasis[(seg4_lable_npy == 1)] = 1
    seg4_lable_oasis[(seg4_lable_npy == 2)] = 2
    seg4_lable_oasis[(seg4_lable_npy == 7)] = 3
    seg4_lable_oasis[(seg4_lable_npy == 9)] = 4

    new_nii = nibabel.Nifti1Image(seg4_lable_oasis,seg4_lable.affine,seg4_lable.header)
    nibabel.save(new_nii,os.path.join(dir_path,"seg4_mni.nii.gz"))

After preprocessing the data according to the above code, I trained my model according to Train_C2FViT_template_matching.py and uncommented the validation part of the code.
Then based on my observations, I noticed that the Dice scores for the validation part of the model I trained were way off from the ones in the log file you provided.
Here's my Dice score:

And here is the Dice score in your log folder:

This is very confusing to me, what do you suggest about this?
Once again, thank you for your tireless replies!

@RexEmperor
Your preprocessing is for the mni152 altas, but not for the MRI in OASIS dataset.
Correct one:
# for label_path in oasis_label_list:
#     print(label_path)
#     label_nib = nib.load(label_path)
#     label_npy = label_nib.get_fdata()
#
#     label_mni = np.zeros(label_npy.shape, dtype=label_npy.dtype)
#     label_mni[(label_npy==8)] = 1
#     label_mni[(label_npy==27)] = 1
#
#     label_mni[(label_npy==5)] = 2
#     label_mni[(label_npy==6)] = 2
#     label_mni[(label_npy==24)] = 2
#     label_mni[(label_npy==25)] = 2
#
#     label_mni[(label_npy==9)] = 3
#     label_mni[(label_npy==28)] = 3
#
#     label_mni[(label_npy==7)] = 4
#     label_mni[(label_npy==26)] = 4
#
#     new_nii = nib.Nifti1Image(label_mni.astype(np.int64), label_nib.affine, label_nib.header)
#     save_path = label_path.replace("seg35.nii.gz", "seg4_mni.nii.gz")
#     nib.save(new_nii, save_path)

You should always check your preprocessed data before you use it for training. I suggest using 3D Slicer/ITKSnap for visualization check to make sure you're selecting the correct ROI anatomical structures for both moving and fixed images.

Dear @cwmok
Thank you again for your help, you have taught me so much that was previously unknown to me.
I saw your Q&A with others in your Issue section, so I just called the preprocessing code directly.
Thanks again for your reply.

Dear @cwmok
I'm so sorry to bother you.
I have two questions:
One is that in the dataset division, the validation set is 10 volumes, but in the code section, it only includes 5 volumes.
imgs = sorted(glob.glob(datapath + "/OASIS_OAS1_*_MR1/norm.nii.gz"))[255:260]
The second question is about the testing part of template_matching.I'm wondering if the testing part is consistent with the validation part?Simply align the images in the test set one by one with the MNI template and calculate the Dice score.

@RexEmperor

@RexEmperor

One is that in the dataset division, the validation set is 10 volumes, but in the code section, it only includes 5 volumes.
imgs = sorted(glob.glob(datapath + "/OASIS_OAS1_*_MR1/norm.nii.gz"))[255:260]

Please strictly follow the result and setting reported in the paper. Do not rely on the commented-out code (it may not be up-to-date.) I acknowledge that the results and setting written in the paper are accurate and correct.

The second question is about the testing part of template_matching.I'm wondering if the testing part is consistent with the validation part?Simply align the images in the test set one by one with the MNI template and calculate the Dice score.

Yes. It is just aligning the images in the test set one by one with the MNI template and calculating the Dice score.

Thanks for the reply, I understand it.
Thank you very much for your patience.

Dear @cwmok
I've got a problem:
The Dice score I get after validating this phase of the model I trained is screwed up by 0.02 compared to what you got, is there something wrong in the middle?
Does the MNI template file need to be preprocessed? Or is it ok to use the ones in the Data folder you provided.

@RexEmperor
I think it is ok to use the one in the Data folder. But you have to make sure the index of each anatomical structure matches that of the moving image. I strongly recommend visualizing both the MNI template and its label using ITKSNAP before training.

The intensity values of the anatomical structures do not seem to match each other.
Could you confirm it?

I don't know if this is the right way to see if my intensity values of the anatomical structures are off.
import numpy as np
import nibabel as nib
# Assuming label_npy is the original label array and label_mni is the remapped label array
label_npy = nib.load('H:\data\seg35.nii.gz')
label_npy = label_npy.get_fdata()
label_mni = nib.load('H:\data\seg4_mni_cp.nii.gz')
label_mni = label_mni.get_fdata()
# Finds the index of all elements in label_mni with a value of 1
indices_mapped_to_1 = np.where(label_mni == 1)

# Check that these indexes correspond to values of 8 or 27 in the original label_npy array.
original_values_at_indices = label_npy[indices_mapped_to_1]
has_8_and_27 = (np.any(original_values_at_indices == 8)) & (np.any(original_values_at_indices == 27))

if has_8_and_27:
    print("Labels 8 and 27 are mapped to 1")
else:
    print("Labels 8 and 27 aren't mapped to 1")

I just tested a set that was mapped to 1 and the results were right.

Am I proving it the wrong way?Thank you for your patience in replying.

According to you, it is the MNI templates that are not a problem, since I downloaded these files from you as well.
In the code, the MNI152 image is MNI152_T1_1mm_brain_pad_RSP.nii.gz;MNI152 label is MNI-maxprob-thr50-1mm_pad_RSP_oasis.nii.gz.That should be right, right?

Above is their presentation in ITK-SNAP.
If there's no problem with this, then it's only possible that I'm having a problem pre-processing the data? Is my preprocessing code correct please? Just preprocess the oasis images and generate new images as their labels right?
import os
import nibabel as nib
import numpy as np

target_dir = '/home/guoke/Downloads/C2FViT-main/Data/Train_pairwise/'

contents = os.listdir(target_dir)

oasis_label_list = [os.path.join(target_dir,name) for name in contents if os.path.isdir(os.path.join(target_dir, name))]

for label_path in oasis_label_list:
    label_path = os.path.join(label_path,"seg35.nii.gz")
    print(label_path)
    label_nib = nib.load(label_path)
    label_npy = label_nib.get_fdata()

    label_mni = np.zeros(label_npy.shape, dtype=label_npy.dtype)
    label_mni[(label_npy==8)] = 1
    label_mni[(label_npy==27)] = 1

    label_mni[(label_npy==5)] = 2
    label_mni[(label_npy==6)] = 2
    label_mni[(label_npy==24)] = 2
    label_mni[(label_npy==25)] = 2

    label_mni[(label_npy==9)] = 3
    label_mni[(label_npy==28)] = 3

    label_mni[(label_npy==7)] = 4
    label_mni[(label_npy==26)] = 4

    new_nii = nib.Nifti1Image(label_mni.astype(np.int64), label_nib.affine, label_nib.header)
    save_path = label_path.replace("seg35.nii.gz", "seg4_mni.nii.gz")
    nib.save(new_nii, save_path)

This is the code I am using, is there a problem with this?
I'm really sorry to keep bothering you, and thank you for your reply, it's sincerely appreciated!

@RexEmperor Could you send me an example of preprocessed fixed and moving image pair along with the corresponding preprocessed segmentation? I can help you to have a look.

MNI152_T1_1mm_brain_pad_RSP.nii.gz
MNI-maxprob-thr50-1mm_pad_RSP_oasis.nii.gz
norm.nii.gz
seg4_mni.nii.gz
imgs = sorted(glob.glob("/OASIS_OAS1_*_MR1/norm.nii.gz"))
labels = sorted(glob.glob("/OASIS_OAS1_*_MR1/seg4_mni.nii.gz"))
MNI152_img = "MNI152_T1_1mm_brain_pad_RSP.nii.gz"
MNI152_label = "MNI-maxprob-thr50-1mm_pad_RSP_oasis.nii.gz"

Thank you very much indeed.

@RexEmperor
Your preprocessed data is correct. It is the same as my training data.

@RexEmperor
I believe I have spotted the issue.
In my code, I use one-hot encoded label.
imgs = sorted(glob.glob(datapath + "/OASIS_OAS1_*_MR1/norm.nii.gz"))
labels = sorted(glob.glob(datapath + "/OASIS_OAS1_*_MR1/seg4_mni_onehot.nii.gz"))
MNI152_img = "../Data/MNI152_T1_1mm_brain_pad_RSP.nii.gz"
MNI152_label = "../Data/MNI-maxprob-thr50-1mm_pad_RSP_oasis_onehot.nii.gz"

Your label is not one-hot encoded. See https://www.geeksforgeeks.org/ml-one-hot-encoding/ for more detail about one-hot encoding.

I read this page about one-shot coding and I learned a few things. However, I still don't understand how one-shotting should be done for .nii.gz files, because the entire .nii.gz file file should be tagged, right?
So, I would like to ask if you can send me your one-shot file or show me the preprocessor code.
Thank you very much.

Hi @RexEmperor,
It is in https://github.com/cwmok/C2FViT/blob/main/Data/image_A_seg4_mni_onehot.nii.gz
The segmentation for fixed image will be in similar format.

Sorry, I don't quite understand how to go about this. Can you send me the code that does the one-shot preprocessing of the data? Or the processed data (but it seems to be too big)?

Or how can this one-shot encoded moving image you provided help?What do I do with it?Thank you.

Hi @RexEmperor,
There are many ways to achieve one-hot encoding. For example, https://pytorch.org/docs/stable/generated/torch.nn.functional.one_hot.html.
You just need to inspect the provided the one-hot encoded image (np.shape, intensity etc) and preprocess the image in the same way as the provided one.

Dear @cwmok
Sorry,I'm sorry I'm having a problem.
It seems to me that the image_A_seg4_mni_onehot.nii.gz file is the image_A_seg4_mni.nii.gz file encoded by onehot right? But I found through the code that the num_classes of image_A_seg4_mni.nii.gz is 5, but the size of the resulting image_A_seg4_mni_onehot.nii.gz is 256x256x256x4. This is really confusing to me.
And, after converting the .nii.gz file to a tensor vector, it needs to be converted to long data before it can be input into the torch.nn.functional.one_hot function, do you do the same?

@RexEmperor
4 anatomical structures + 1 background = 5 classes.
We then take the 4 classes from the one-hot encoded data.
Yes, it accepts long data type only.

You mean you only coded the categories and not the context, right?
When I process the tensor, the system always reports an error:
img = nib.load('image_A.nii.gz')
data = img.get_fdata()
tensor = torch.from_numpy(data)
tensor = tensor.to(torch.long)
tensor_ontshot = torch.nn.functional.one_hot(tensor,4)

Am I handling this wrong?
Thank you.

I used this script below to process the one-hot encoded data.
import glob

import nibabel as nib
import numpy as np

'''
MNI152 label - OASIS 35 label
0 -
1 Caudate - 8, 27
2 Cerebellum - 5, 6, 24, 25
3 Frontal Lobe - None
4 Insula - None
5 Occipital Lobe - None
6 Parietal Lobe - None
7 Putamen - 9, 28
8 Temporal Lobe - None
9 Thalamus - 7, 26
'''

'''
OASIS seg35 list (selected 25 structures)

13    Brain-Stem                            119 159 176 0

7     Left-Thalamus                         0   118 14  0
26    Right-Thalamus                        0   118 14  0

6     Left-Cerebellum-Cortex                230 148 34  0
25    Right-Cerebellum-Cortex               230 148 34  0

3     Left-Lateral-Ventricle                120 18  134 0
22    Right-Lateral-Ventricle               120 18  134 0

5     Left-Cerebellum-White-Matter          220 248 164 0
24    Right-Cerebellum-White-Matter         220 248 164 0

9     Left-Putamen                          236 13  176 0
28    Right-Putamen                         236 13  176 0

8     Left-Caudate                          122 186 220 0
27    Right-Caudate                         122 186 220 0

10    Left-Pallidum                         12  48  255 0
29    Right-Pallidum                        12  48  255 0

14    Left-Hippocampus                      220 216 20  0
30    Right-Hippocampus                     220 216 20  0

11    3rd-Ventricle                         204 182 142 0

12    4th-Ventricle                         42  204 164 0

15    Left-Amygdala                         103 255 255 0
31    Right-Amygdala                        103 255 255 0

2     Left-Cerebral-Cortex                  205 62  78  0
21    Right-Cerebral-Cortex                 205 62  78  0

(too small) 19    Left-Choroid-Plexus                   0   200 200 0
(too small) 35    Right-Choroid-Plexus                  0   200 200 0

'''

print("Process MNI label...")

mni_label_path = "../Data/MNI-maxprob-thr50-1mm_pad_RSP_oasis.nii.gz"
mni_label = nib.load(mni_label_path)
mni_label_npy = mni_label.get_fdata()

print(f"Unique labels: {np.unique(mni_label_npy)}")

one_hot_img = np.zeros(mni_label_npy.shape + (4,), dtype=mni_label_npy.dtype)

for index, i in enumerate([1, 2, 3, 4]):
    one_hot_img[..., index] = (mni_label_npy == i)

new_img = nib.nifti1.Nifti1Image(one_hot_img.astype(np.int64), affine=mni_label.affine, header=mni_label.header)
save_path = mni_label_path.replace('MNI-maxprob-thr50-1mm_pad_RSP_oasis.nii.gz', 'MNI-maxprob-thr50-1mm_pad_RSP_oasis_onehot.nii.gz')
print("Saving... ", save_path)
nib.save(new_img, save_path)

OASIS_mni_path = sorted(glob.glob("../Data/OASIS/OASIS_OAS1_*_MR1/seg4_mni.nii.gz"))

for path in OASIS_mni_path:
    print(f"Processing {path}")
    label = nib.load(path)
    label_npy = label.get_fdata()

    print(f"Unique labels: {np.unique(label_npy)}")
    one_hot_img = np.zeros(label_npy.shape + (4,), dtype=label_npy.dtype)

    for index, i in enumerate([1, 2, 3, 4]):
        one_hot_img[..., index] = (label_npy == i)

    new_img = nib.nifti1.Nifti1Image(one_hot_img.astype(np.int64), affine=label.affine, header=label.header)
    save_path = path.replace('seg4_mni.nii.gz', 'seg4_mni_onehot.nii.gz')
    print("Saving... ", save_path)
    nib.save(new_img, save_path)

# print("Preprocessing OASIS label")
# left = [7, 6, 3, 5, 9, 8, 10, 14, 15, 2]
# right = [26, 25, 22, 24, 28, 27, 29, 30, 31, 21]
# left_right_pair = [list(a) for a in zip(left, right)]
# other = [11, 12, 13]
#
# OASIS_label_path = sorted(glob.glob("../Data/OASIS/OASIS_OAS1_*_MR1/seg35.nii.gz"))
#
# for path in OASIS_label_path:
#     print(f"Processing {path}")
#     label = nib.load(path)
#     label_npy = label.get_fdata()
#
#     print(f"Unique labels: {np.unique(label_npy)}")
#     one_hot_img = np.zeros(label_npy.shape + (13,), dtype=label_npy.dtype)
#
#     for index, pair in enumerate(left_right_pair):
#         left_image = (label_npy == pair[0])
#         right_image = (label_npy == pair[1])
#         one_hot_img[:, :, :, index] = left_image | right_image
#
#     for index, o in enumerate(other):
#         one_hot_img[:, :, :, index+10] = (label_npy == o)
#
#     new_img = nib.nifti1.Nifti1Image(one_hot_img.astype(np.int64), affine=label.affine, header=label.header)
#     save_path = path.replace('seg35.nii.gz', 'seg35_onehot.nii.gz')
#     print("Saving... ", save_path)
#     nib.save(new_img, save_path)

Sorry to bother you.
I would like to ask about this table, this 0.72±0.06 means that if I reproduce this experiment by myself, even if I come up with an effect of 0.78 it is still reasonable right?

Dear @cwmok
I have a question to consult: In the Template-Matching Normalization (MNI152) task, we have utilized the MNI152 coordinate space. However, I don't seem to find this practice in other research papers. May I ask if this will have any impact on our final registration results? Will it enhance or diminish the accuracy?

@RexEmperor
I am not clear about your question. Are you asking about the application of the Template-Matching Normalization?
It is common in medical image analysis. For instance, it is a common preprocessing step in Brain Tumor Sequence Registration (https://arxiv.org/abs/2112.06979).

No, I'm actually asking about MNI152. Since the Template-Matching Normalization task in the article utilizes MNI152, I would like to know what MNI152 is and what its functions are. I've searched for some information online, but I haven't found anything particularly helpful.
Thanks for your answer!

Thank you!
I would like to ask about this table, this 0.72±0.06 means that if I reproduce this experiment by myself, even if I come up with an effect of 0.78 it is still reasonable right?

@RexEmperor
It could be. To answer this question, you have to compute the Initial (CoM) DSC as well. If your initial DSC (before registration) is higher than mine, the chances are the selected atlas will probably be easier to register than the case I selected in my paper, which is also correct.

Thank you for your reply. May I ask where can I get the LPBA dataset?


Remove beta releases from changelog

Alaska Airlines Pull Request
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Resolves: # (issue, if applicable)
Summary:
Please summarize the scope of the changes you have submitted, what the intent of the work is and anything that describes the before/after state of the project.
Remove beta releases from changelog
Type of change:
Please delete options that are not relevant.

[ ] New capability
[ ] Revision of an existing capability
[ ] Infrastructure change (automation, etc.)
[x] Other (please elaborate)

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

Summary by Sourcery
Remove beta release entries from the changelog to maintain a cleaner and more concise document focused on stable releases.
Chores:

Remove beta release entries from the changelog to streamline the document and focus on stable releases.

:tada: This PR is included in version 3.0.0-beta.1 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fix small issues in the Todo tutorial

I added a separate step for rebuildRouter and removed the PostgreSQL configuration changes (not sure why they were needed on my machine).

thank you


stat: block size differences

Toybox 0.7.1, Busybox 1.24.2
On a<EMAIL_ADDRESS>Why is there a difference in blocksize?
root@hammerhead:/sdcard # busybox stat -c %B:%b:%o:%s twrp-3.0.0-0-hammerhead.img
512:28632:4096:14657536
root@hammerhead:/sdcard # toybox stat -c %B:%b:%o:%s twrp-3.0.0-0-hammerhead.img
4096:28632:4096:14657536

%B  Bytes per block

busybox says 512Byte while toybox says 4096Byte.
28632 Blocks * 512Byte = 14659584 Byte
Which is a lot closer to the actual file size of 14657536 Byte (reported by both toybox&busybox).
Other commands from both binaries also show a block size of 4096 Byte though.
root@hammerhead:/sdcard # busybox stat twrp-3.0.0-0-hammerhead.img                                       <
  File: twrp-3.0.0-0-hammerhead.img
  Size: 14657536        Blocks: 28632      IO Block: 4096   regular file
Device: 11h/17d Inode: 218474      Links: 1
Access: (0660/-rw-rw----)  Uid: (    0/ UNKNOWN)   Gid: ( 1015/ UNKNOWN)
Access: 2016-03-18 14:16:00.000000000
Modify: 2016-03-18 14:16:03.000000000
Change: 2016-03-18 14:16:03.000000000

root@hammerhead:/sdcard # busybox stat -f .
  File: "."
    ID: 0        Namelen: 255     Type: UNKNOWN
Block size: 4096
Blocks: Total: 7015287    Free: 189096     Available: 189096
Inodes: Total: 1785856    Free: 1747803

root@hammerhead:/sdcard # toybox stat twrp-3.0.0-0-hammerhead.img
  File: `twrp-3.0.0-0-hammerhead.img'
  Size: 14657536         Blocks: 28632   IO Blocks: 4096        regular file
Device: 11h/17d  Inode: 218474   Links: 1
Access: (660/-rw-rw----Segmentation fault

root@hammerhead:/sdcard # toybox stat -f .                                                 <
  File: "."
    ID:<PHONE_NUMBER>000000 Namelen: 255    Type: 65735546
Block Size: 4096    Fundamental block size: 4096
Blocks: Total: 7015287  Free: 189096    Available: 189096
Inodes: Total: 1785856  Free: 1747803

This stackoverflow post indicates that the stat %B parameter for block size should be ignored as it may not necessarily reflect the actual file system block size value but usually some unrelated OS value.
This would explain the values of 4096 which leads to huge discrepancies between allocated and actual file size (it's not a sparse file). What would be interesting though why busybox still gets the correct 512 blocksize value.
Is it planned to support something like --block-size for commands like stat,ls and du?

After more research, I think I got it now, see here.
%b comes from st_blocks which is always in units of 512 byte.
So the allocated file size on the filesystem is always calculated as block-count * 512 byte, with the result being in increments of the actual file system block size, which is the %o format i.e. I/O Block size.
So why does busybox print the "correct" value of 512 for "Bytes per Block" which should usually just be ignored? Well it's hardcoded.
Toybox returns the same value of st_blksize for both IO Block size and Bytes per block.
Busybox returns a hardcoded 512 for Bytes per block and st_blksize for IO Block size.
@landley
What would you say to also hardcoding 512 for Bytes per bock for toybox?
Returning the IO Block size for seems wrong.
Telling us the "Bytes per block" for the "Blocks allocated" count seems like stat's and the %B formats responsibility and would actually fit the parameters descriptions.
Or Bytes per block could be cut from code, due to serving no useful purpose... but that would break "spec" and could lead to compatibility issues. So hardcoding 512 seems like a good solution.

On 08/11/2016 03:49 AM, Matthias Urhahn wrote:

Toybox 0.7.1, Busybox 1.24.2
On a<EMAIL_ADDRESS>Why is there a difference in blocksize?
|root@hammerhead:/sdcard # busybox stat -c %B:%b:%o:%s
twrp-3.0.0-0-hammerhead.img 512:28632:4096:14657536
root@hammerhead:/sdcard # toybox stat -c %B:%b:%o:%s
twrp-3.0.0-0-hammerhead.img 4096:28632:4096:14657536 |
%B Bytes per block

In toybox, both %o and %B come from the stat() system call.
else if (type == 'B') out('u', stat->st_blksize);
} else if (type == 'o') out('u', stat->st_blksize);
I.E. we're telling you what the operating system told us (presumably
getting it from the filesystem driver).
A quick check shows busybox is printing a hardwired value for %B but the
stat value for %o:
    } else if (m == 'B') {
            strcat(pformat, "lu");
            printf(pformat, (unsigned long) 512); //ST_NBLOCKSIZE

    } else if (m == 'o') {
            strcat(pformat, "lu");
            printf(pformat, (unsigned long) statbuf->st_blksize);

I don't understand the point of printing a hardwired value for %B? FAT
block sizes vary from 512 bytes up to 65535k. ext2 can be 1k or 4k.
A few years ago hard drives went from 512 byte physical block to 4k,
which caused some problems because of longstanding assumptions:
https://lwn.net/Articles/322777/
https://lwn.net/Articles/377895/
And now the sectors are getting bigger:
https://lwn.net/Articles/582862/
Here's some articles about the damage conflicting block size assumptions
can do (data loss when an interrupted write changes stuff you didn't
think was being rewritten) and ways around it:
https://lwn.net/Articles/349970/
https://lwn.net/Articles/665299/
https://lwn.net/Articles/353411/
You shouldn't have to care about 90% of that (the OS handles it all for
you), but from my perspective having two ways to query block size would
be really nice if the OS gave me a way to determine the block size of
the physical media, as distinct from the block size of the filesystem.
Unfortunately, although it seems like %B would be filesystem block
size and %o would be physical media block size, Linux doesn't give me a
way to query physical media block size that I've noticed. (I might be
able to beat it out of the mtd layer for some types of flash?)

busybox says 512Byte while toybox says 4096Byte.

Busybox is returning a single hardwired answer.
That said, it looks like the Ubuntu version is also doing that. If %B
should always say "512" regardless of context, I can make it do that and
change the help text to say something like:
%B  prints "512"

28632 Blocks * 512Byte = 14659584 Byte
Which is a lot closer to the actual file size of 14657536 Byte (reported
by both toybox&busybox).

More stat fields: %b is st_blocks and %s is st_size.
The stat(2) man page says that st_blocks is "number of 512B blocks
allocated" so %B is units for %b (and yes, it's a hardwired value). So
the help text should be something like:
%B  units for %b (always 512)

Other commands from both binaries also show a block size of 4096 Byte
though.

The "man 1 stat" page says:
   %B     the size in bytes of each block reported by %b

The "man 2 stat" page says:
   blkcnt_t  st_blocks;  /* number of 512B blocks allocated */

So yes, you've found an inconsistency and I should fix it. %B should
output a hardwired "512", busybox is correct here.
Thanks. Good catch,
Rob

Fixed with https://github.com/landley/toybox/commit/4460e9f0ed16a3f4dea4dec13f90df9c956b4dc9 :+1: Thanks!


Remove unnecessary DataObject::write in fixture blueprint

This line doesn't seem necessary, and shaves ~1 min off overall build time without it.

seems legit - would it be worth doing this in the 3.5 line?

I suppose so. I assume it won't be cherry pickable but feel free to copy it through

How does this line work if you don't write it?
$fixtures[$class][$identifier] = $obj->ID;

@tractorcow I guess it doesn't. I based this off the test suite passing. I'm happy to close it as well since it doesn't add that much benefit.


Primary key detection is not used (in particular, by foreign key detection)

As detailed in the project description paper https://dspace.mit.edu/handle/1721.1/127410, foreign key detection should take primary key detection's outputs as arguments. But it is not so in the current version.

Could have FKD take in PKD's predictions and only look at column pairs in which at least one is a primary key


Added config for mix build

It creates a lot of junk errors and warnings while building inside mix projects. This config fixes this.

Thank you for the contribution.


How to put fonts?

i have fonts files in styles: http://take.ms/SSYZt
then i have file to loads fonts: http://take.ms/TkwJW
then i import this file to core.scss into :global and nothing.

Until better workarounds are found, I recommend reading through https://github.com/davezuko/react-redux-starter-kit/issues/403 and see if that helps.

Has there been any updates on this? I have looked through #403 and attempted to implement the solutions listed there with no avail. I saw you mentioned in that issue thread about adding some information to the FAQ on how to properly do this?

@jgo4th There have not yet, sorry. This has been a long running issue, so I apologize for that. I'm currently trying to limit my non-work-related coding/typing due to wrist pains, so it may still be a while until I can really dig into the heart of it.

Remove then from :global. If you look at the output code, you will see there are double brackets into every "@expression", therefore making it not work.
$red: #D95E40;
$green: #34bf49;
$linkColor: #298eea;

@import 'vendor/font-awesome';
@import 'vendor/roboto';
@import 'main';
@import 'base';

html {
  box-sizing: border-box;
}

html,
body {
  height: 100%;
}

body, input, textarea, button {
  font-family: Roboto, sans-serif;
}

It will work perfectly :D I struggled to make it work too, until I look at the ouput css.

@vini175pa how did you view the output.css? I am new to the whole react ecosystem.

Use 'Inspect' of your browser to view your page elements. There there's a link to a css file that is the output css. Its name is probably hashed ( e.g. app.912893u8na.css )

Closing per @vini175pa 's tips. If you are still having issues let me know and we can reopen.

I'm still having issues since I'm using a custom font.  If I remove them from :global I get a module not found error.  If I keep them inside the :global scope I can see the double {{ }} brackets in my @font-face preventing it from working.  Any help here?

@vinceh
Maybe the path to the font is wrong. Look, this is how I got it to work:
/src/styles/vendor/_roboto.scss, this is the file that loads Roboto font.
/* BEGIN Light */
@font-face {
  font-family: Roboto;
  src: url("../static/fonts/Roboto/Light/Roboto-Light.woff2?v=1.1.0") format("woff2"), url("../static/fonts/Roboto/Light/Roboto-Light.woff?v=1.1.0") format("woff"), url("../static/fonts/Roboto/Light/Roboto-Light.ttf?v=1.1.0") format("truetype");
  font-weight: 300;
  font-style: normal;
}
/* END Light */
...

My fonts are located in /src/static/fonts

Same issue as @vinceh here; trying to use Font Awesome installed via npm, my core.scss looks like:
$fa-font-path: '~font-awesome/fonts';

:global {
  @import 'base';
  @import '~normalize.css/normalize';
  @import '~font-awesome/scss/font-awesome.scss';
...

Webpack is happy but I get the double bracket issue (@font-face{{ ... }}) in the generated CSS. Am I missing something or is this not possible?

@bhj As I said: "Remove them from :global scope. If you look at the output css, you will see there are double brackets into every "@expression" ( e.g. @font-face{{ ... }} ), therefore making it not work."
@import '~font-awesome/scss/font-awesome.scss';
:global {
  @import 'base';
  @import '~normalize.css/normalize';

Will solve the problem

Hi @vini175pa I still get the same error as @vinceh if I do that ("Module not found" for each font)
If I work around that and copy /node_modules/font-awesome/fonts to /src/styles/fonts and then make core.scss
$fa-font-path: 'styles/fonts';
@import '~font-awesome/scss/font-awesome.scss';

:global {
  @import 'base';
  @import '~normalize.css/normalize';
...

... then the output CSS has all the FA classes namespaced to core__ and not usable from elsewhere.

Oh, I see. Here's a workaround: https://github.com/css-modules/css-modules/issues/95

Thanks for the hint @vini175pa! I ended up creating a folder like /src/styles/nomodule with a font-awesome.scss:
$fa-font-path: '~font-awesome/fonts';
@import '~font-awesome/scss/font-awesome.scss';

I import this file in CoreLayout.js rather than core.scss. Then over in /build/webpack.config.js, in the // Loaders for styles that need to be treated as CSS modules section, I added to the .scss loader configuration:
    exclude: /src\/styles\/nomodule/,

And below that in the // Loaders for files that should not be treated as CSS modules. section, I changed the .scss loader configuration from
  exclude: excludeCSSModules,

to
  include: /src\/styles\/nomodule/,

Hacky, but it seems to work and I can use Font Awesome straight from node_modules without copying files around. I totally see what @davezuko is getting at with #787 now and would love to know a better way of doing this.


Support a date-time primative

created_at_utc: "1981-11-30T10:44:38.567Z"
https://birchsolutions.slack.com/archives/C0222UL3XS9/p1651200517793859?thread_ts=1651199301.661399&cid=C0222UL3XS9

added to types.yml. has it been implemented in codegen?

moved issue: https://github.com/fern-api/fern-java/issues/128


Move a folder into itself puts file tree in a bad state

should also prevent folder from being moved into one of its children, or their children, etc


Fix: env-bootstrap-dir doesn't work

Description
The env-bootstrap-dir option doesn't work.
How has this been tested?
Manually. I didn't find appropriate tests for this.
Types of changes
I changed the templates/wp-config.php file
Checklist:

[x] My code is tested
[x] My code follows the project code style
[ ] My code has documentation (for new features or changed behavior)

Thanks @voronkovich


Reproduction issue

Fixed the issue where MCM masking and span selection augmentation were not being applied dynamically
Also, modified the preprocessing code to remove all punctuations and turn uppercase letters to lowercase letters

Reopend the pull request to merge it to the different branch.


Intelligent Go - Hypervolt Charger Integration

Describe the bug
Hypervolt Charger integration is now live for Intelligent Go. HA Octopus integration now reporting a unknown Intelligent provider ‘HYPERVOLT’.
Reproduction steps
Requires a Hypervolt EV Charger and signing up to Intelligent Go integration
Expected behaviour
No specific error it seems although the Intelligent Dispatcher sensor is present but no of the other bump charge switch or Intelligent Go sensors available. Presumably as it looks or functions the same as the OHME charger from a code perspective reading the documentation. Not sure if the other switches or sensors should be available for the Hypervolt EV charger integration. NB Zappi charger integration has also recently been added
Tariff Code
E-1R-INTELLI-VAR-22-10-14-H
Integration Version
12.1.0
Home Assistant Version
2024.8.2
Fresh Install?
After upgrading
Home Assistant Logs
This error originated from a custom integration.
Logger: custom_components.octopus_energy.intelligent
Source: custom_components/octopus_energy/intelligent/init.py:255
integration: Octopus Energy (documentation, issues)
First occurred: 08:00:21 (6 occurrences)
Last logged: 08:00:23
Unexpected intelligent provider 'HYPERVOLT'
Confirmation

[X] I confirm that I cannot find my solution within the documentation
[X] I confirm that I cannot find my solution within the FAQ

Thanks for raising this. Due to not all intelligent integrations being the same, new ones need to be enabled manually. For instance, not much is enabled for OHME because it's all handled their side. Could you please confirm what features are enabled by the app with a screenshot? Bump charge, smart charge, planned dispatches, charge target/limit, ready time?

Bump Charge, target charge%, ready time, and dispatches are available. Had a list of scheduled slots yesterday which were used. Looks like they might not fully have sorted the charge history bit.

Here's the completed and planned dispatches from the sensor attribute:
Planned dispatches
Completed dispatches

start: '2024-08-16T10:30:00+01:00' end: '2024-08-16T11:00:00+01:00' charge_in_kwh: -0.67 source: null location: UNABLE_TO_IDENTIFY
start: '2024-08-16T10:00:00+01:00' end: '2024-08-16T10:30:00+01:00' charge_in_kwh: -2.67 source: null location: UNABLE_TO_IDENTIFY
start: '2024-08-16T09:30:00+01:00' end: '2024-08-16T10:00:00+01:00' charge_in_kwh: -2.85 source: null location: UNABLE_TO_IDENTIFY
start: '2024-08-16T09:00:00+01:00' end: '2024-08-16T09:30:00+01:00' charge_in_kwh: -3.12 source: null location: UNABLE_TO_IDENTIFY
start: '2024-08-16T08:30:00+01:00' end: '2024-08-16T09:00:00+01:00' charge_in_kwh: -0.04 source: null location: UNABLE_TO_IDENTIFY
start: '2024-08-16T08:00:00+01:00' end: '2024-08-16T08:30:00+01:00' charge_in_kwh: -0.93 source: null location: UNABLE_TO_IDENTIFY
Data last retrieved
17 August 2024 at 14:33:58
Provider
HYPERVOLT
Vehicle battery size in kwh
64
Charge point power in kw
7
Current start
Unknown
Current end
Unknown
Next start
17 August 2024 at 23:30:00
Next end
18 August 2024 at 05:30:00

Have also unplugged and replugged car and it created a schedule for today in the Octopus App even though car is already at 100%:

These showed up somewhere in the Octopus API as the OctoAid App picked them up, but couldn't seem to find them as planned dispatches in the attributes of the dispatching sensor (yet).

The support should now be added as part of v12.2.0

Many thanks - updated and seems to picking up the Charge Limit and ready time values from Octopus. Will try another charge in a few days (as car currently at 100%!)

Glad to hear. Ill close this issue. If you have any other issues in the future, please raise a separate ticket.


Hiding sticker on hover

Same as doki-theme/doki-theme-vscode#143, but for JetBrains IDEs.

As you can see, the feed drop down (where it says nuget.org) and the uninstall buttons for the specific project and solutions are there but not really visible.

Yeah that's a good idea. Guess that would have been easier to implement that making it move 😄


Can't get response data's value

The response.data is ok,but response.data.access_token is undefined.
I don't known why, can anyone help me?

var access_token = data.data.access_token

var access_token = data.data.access_token

Thank you. I have found and modified it.


Report context cancel cause in DoBatchWithOptions instead of ctx.Err().

In https://github.com/grafana/tempo/issues/3957 it was found that analyzing the root cause of context cancelled logs can be quite hard to analyze. In the discussion on how to improve the situation, it was found that reporting the cancel cause, introduced by WithCancelCause in go 1.20, would be more helpful in these situations.
Note that this won't change the behavior of error reporting when WithCancel (without cause) is used.

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."


[question] Exempting on-nix derivations from NixOS GC cleanup

I occasionally do this to clean up disk space from older NixOS generations:
nix-store --gc
nix-collect-garbage -d
nix-env --delete-generations old

but it also cleans up locations in the nix store that belong to the various on-nix/python derivations. Now after cleaning up disk space, when I cd around into direnv enabled dirs with on-nix/python default.nix the derivations get rebuilt. Reproducibility is a good thing, but is there a way to save that time and somehow make NixOS cleanup aware of on-nix/python nix store locations?

The garbage collector do not remove paths in the store that are referenced by other paths or symlinks anywhere in the same volumne as /nix/store
So the trick there is to create a symlink into some stateful directory and not remove it
Something like this: https://github.com/kamadorueda/alejandra/blob/942dc8fd92dc7befa212c269b6c75163ad891ce2/.envrc#L15
Alternatives are keeping the ./result of nix-build, or to use nix-shell with a --profile, in short, keeping a symlink around

https://discourse.nixos.org/t/what-is-the-best-dev-workflow-around-nix-shell/418/4

Have you tried using the use nix hook for direnv? it seems to do all the magic
https://github.com/nix-community/nix-direnv/blob/60a6964eaeab12b3af8d43acc8e783992b917fc2/direnvrc#L207

Have you tried using the use nix hook for direnv?

I have a file .envrc with contents use_nix so I think I am using that already? I don't see any local files created in any case.

A python.on-nix.com shell is not different than a regular shell, so I think the behaviour you are encountering is due to a direnv missconfiguration
From what I see in the readme: https://github.com/nix-community/nix-direnv
It seems that you need to perform some extra steps while installing direnv:
Optional: To protect your nix-shell against garbage collection you also need to add these options to your Nix configuration.

If you are on NixOS also add the following lines to your /etc/nixos/configuration.nix:

{ pkgs, ... }: {
  nix.extraOptions = ''
    keep-outputs = true
    keep-derivations = true
  '';
}
On other systems with Nix add the following configuration to your /etc/nix/nix.conf:

keep-derivations = true
keep-outputs = true

It also seems that installing nix-direnv requires some extra steps too:

https://github.com/nix-community/nix-direnv#with-nix-env

Thanks for looking into this and the pointers!
Putting this into configuration.nix fixes it for me:
  nix.extraOptions = ''
    keep-outputs = true
    keep-derivations = true
  '';

Awesome!


Dashboard ONTAP: SVM - NFS throughput has wrong unit

Problem
I found two issues in the Dashboard ONTAP: SVM.

incorrect units for all NFS (v3,v4,v4.1) throughput panels. All panels show ops per second instead of bytes per second.
"NFSv4 Read IOPs" single stat panels is broken. The query is using wrong metric svm_read_total instead of  svm_nfs_read_total.
"NFSv4 Write IOPs" single stat panels is broken. The query is using wrong metric svm_write_total instead svm_nfs_write_total.

Version
harvest version 23.05.0-1 (commit 6f74c7a5) (build date 2023-05-03T08:08:46-0400) linux/amd64
OS and platform
RHEL8

Verified on bd0b0db7


Made it work with vis-2

The only important change is in widgets/materialdesign/js/widgets/materialdesign.viseditor.js

anything we can support you with @Scrounger?

Maybe it's possible to move to community adapters? Don't know if licensing is ok with this


Add nginx config to remove server versions

Fails are flappers

@cooperka  No I already did this in the wild. Just documenting.


Fix don't set up cookie on manual close session

Q
A

Is bugfix?
✔

New feature?
❌

Breaks BC?
❌

Fixed issues
#32

👍


Poor readme

Please keep things about you minimal in the readme.
People generally want to know what is there for them.
Basically, answer the questions they would ask you.
Again the questions always come from who, what, when, where, why and probably how:
Questions they may have:
What is this project about?
How does the project work?
What skills were used?
How does one setup the project locally to run it on their computer?
How can they contribute to your project
think about which questions a user may ask and answer them in the readme.

other questions may include :
What is the architecture of the app?
Architecture can be a big thing so to answer this question talk about how many pages it has and how to get to each of these pages.
What are the categories or organisations used in the app

you may also talk about folder structure.


Fix bug copying the pointer of the same for-loop variable

longhorn/longhorn#3980

Don't forget to backport to v1.2.5


WIN11: RealSense SDK ERROR rs2_create_context_ex..

Hey, I download the SDK 2.55/2.56 and the newer to my windows 11.

When I try to open to open the file after installing it I get this error.

The realsense-config.json doesn't exist (C/users/[user]/AppData/Roaming) and I couldn't find it.
How can I fix that problem?

Hi @RoyAmoyal  Can you confirm if you have downloaded and run the stabdalone RealSense Viewer Intel.RealSense.Viewer.exe from the 'Assets' file list of 2.55.1's Release page, please?
https://github.com/IntelRealSense/librealsense/releases/tag/v2.55.1

Once downloaded, the .exe file must be run to launch this version of the RealSense Viewer.  The shortcut launch icon that the full SDK installer places on the Windows desktop shold not be used to launch the standalone Viewer.

I got the same error, even after downloading the Intel.RealSense.Viewer.exe.
Do I need to download more files? Or just the SDK and the viewer exe files?

The errno=42 error has been reported most often for version 2.55.1.
You should not need to download anything other than the .exe when using the standalone Viewer.
Does the error still appear if you use the Viewer .exe for 2.54.2?
https://github.com/IntelRealSense/librealsense/releases/tag/v2.54.2

Hi @RoyAmoyal  Do you require further assistance with this case, please? Thanks!

Case closed due to no further comments received.


Error when using tap or tap-flat reporter

Describe the bug
When I try to use either of the tap-reporters I only get this error:
TypeError: this._clearScreen is not a function
 ❯ Vitest.log node_modules/vitest/dist/chunk-vite-node-externalize.80bed9b3.mjs:9290:10
 ❯ IndentedLogger.log node_modules/vitest/dist/chunk-vite-node-externalize.80bed9b3.mjs:7967:17
 ❯ TapReporter.onFinished node_modules/vitest/dist/chunk-vite-node-externalize.80bed9b3.mjs:8038:17
 ❯ node_modules/vitest/dist/chunk-vite-node-externalize.80bed9b3.mjs:9682:51
 ❯ Vitest.report node_modules/vitest/dist/chunk-vite-node-externalize.80bed9b3.mjs:9680:38
 ❯ node_modules/vitest/dist/chunk-vite-node-externalize.80bed9b3.mjs:9525:20
 ❯ process.processTicksAndRejections node:internal/process/task_queues:95:5
 ❯ Vitest.runFiles node_modules/vitest/dist/chunk-vite-node-externalize.80bed9b3.mjs:9532:12
 ❯ Vitest.start node_modules/vitest/dist/chunk-vite-node-externalize.80bed9b3.mjs:9449:5

Reproduction
Happens on every run with either --reporter tap or --report tap-flat. Here is a minimal repo. Run tests via vitest --run to see the error.
System Info
System:
    OS: macOS 12.4
    CPU: (10) arm64 Apple M1 Max
    Memory: 1.90 GB / 64.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.7.0 - ~/.nvm/versions/node/v18.7.0/bin/node
    Yarn: 3.1.0 - ~/.nvm/versions/node/v18.7.0/bin/yarn
    npm: 8.15.0 - ~/.nvm/versions/node/v18.7.0/bin/npm
  Browsers:
    Chrome: 103.0.5060.134
    Firefox: 100.0.2
    Safari: 15.5
  npmPackages:
    vitest: 0.21.0 => 0.21.0

Used Package Manager
yarn
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

I updated the original description with a link to a Stackblitz repo (https://stackblitz.com/edit/vitest-dev-vitest-uaoddv?file=vite.config.ts)


Global storage-backed user settings

Specific Demand
This is forked from the discussion in #1426. I think that Dioxus would benefit from a global, statically-defined user settings container. Different components would be able to declare a user setting using a hook like use_setting. There would then be another hook that can be used to retrieve all user settings to construct a common preferences page.
Implement Suggestion
Each setting should be tied to a struct that must derive Default to ensure a fallback behavior, as well as a friendly description and name for constructing the preferences page. On browser targets, the window.localStorage offers a good persistent store that will only reset if the browser cache gets purged. On desktop/mobile, this could be exposed as a platform-native collection such as UserDefaults for Swift or it could be written to some file. Ideally, if a user setting gets changed in one window/tab, all other windows/tabs should be re-rendered to reflect the new setting value. The new signals crate would be useful for this.

I think this could be split into two different crates: use_setting, use_cache, etc. with paths from directories and their web equivalents could be a good fit for dioxus-std. The full settings construct could be a different crate that handles the UI, defaults, and synchronizing between windows/tabs

If the backing is something like the ProjectDirs, wouldn't synchronization be required to ensure multiple instances don't conflict when writing back settings?

If the backing is something like the ProjectDirs, wouldn't synchronization be required to ensure multiple instances don't conflict when writing back settings?

You can create an unique id for the app based on something like the package name, author, git hash and version.

I agree that constructing the common settings UI and handling system defaults could be handled as a community crate, but synchronization seems like it should be packaged with the hook. I think either the synchronization code should be part of dioxus-std or it could all be done as a community crate.

Cross application synchronization could fit with dioxus-std. The interprocess crate may be useful to implement that feature

Cross application synchronization could fit with dioxus-std.

I've created a new issue to track this: #1456

I think for native targets, this can be accomplished without cross-application synchronization by using file locks and watchers. On web and fullstack, operations are synchronized by the browser. We can listen for StorageEvent to update the in-memory setting. The trickiest targets I see are liveview and SSR, I think for now I will leave these as unsupported.

Below is my more flushed-out proposal for this feature:
Summary
Use case
Developers should be able to specify a collection of user-facing settings, define a default value for these settings, and subscribe to them wherever relevant. These values should be consistent across all instances of an application for a given target.
Non-goals
I am not concerned with synchronizing values between targets as I am primarily focused on desktop and web targets, where there is no backing server. This may be a beneficial use case for dioxus-fullstack, though, and can be treated as a future consideration.
This is not a design for general state management across instances. I have a separate issue discussing persistent state management for a given instance #1426.
While I propose a concept for a common settings UI, I am not going to define this here. That said, it could follow closely the structure of the router layouts system, using attributes and nesting for each field.
Design
Developer experience
A developer would declare a struct like the following:
#[derive(SettingsCollection)]
struct UserSettings {
    // Optional attributes with the name and description of the settings, for use in a common settings UI.  If using i18n, these can be treated as keys into the i18n dictionary.
    #[name("Appearance")] // Defaults to the name of the field if this is not provided.
    #[description("Select what theme to use for the UI")]
    theme: ThemeSetting,
    #[description("Define how notifications are delivered.")]
    notifications: NotificationSetting,
}

#[derive(Setting, Default)] // A default value will be required for each setting.
enum ThemeSetting {
    Light,
    Dark,
    #[default]
    System,
}

#[derive(Setting)]
struct NotificationsSetting {
    #[name("Enable notifications")]
    enabled: bool,
    #[name("Show previews in notifications")]
    previews: bool,
}

impl Default for NotificationsSetting {
    fn default() -> Self {
        Self { enabled: false, previews: true }
    }
}

In your root node, define the settings as follows:
use_settings::<UserSettings>(cx);

When accessing a setting, a selector will be used. This should return some form of RefCell or Arc to the setting section, allowing the setting to be mutable. Ideally, this should only trigger renders on the setting that is being selected, though I am still thinking of how to enforce this.
let mut theme_settings = use_settings_selector::<UserSettings>(cx, |settings| settings.theme);

Under the hood
I am envisioning three implementations to start. One consideration for all three is whether to store the whole struct as a single entry or break it up and store each field as a separate key. Storing each field as a separate key could offer opportunities to only invalidate some of the entries if the structure of a setting gets changed.
WASM
This is probably the most straightforward. WASM targets have access to the localStorage API, which is a key value store that is synchronized for all sessions under a given origin domain. This is supported by all browsers. There is a window storage event which we can listen for to learn when the localStorage has been updated in another session to update the underlying settings struct.
Desktop
For Desktop targets, I see two approaches, related to the consideration above about storing the whole settings struct as one entry or breaking it up by field. The lightest-weight option is to serialize the settings struct as a JSON/CBOR and dump it into a file in the system's default project directory. Then, we can use file watchers and file locks to synchronize the settings between instances. Another option is to use a key/value DB like redb, which offers built-in consistency between instances, though it doesn't offer a channel for listening for changes.
Mobile
Mobile makes some things easier and others harder. Synchronization is not a concern because there is only ever one instance of an application. That said, we also cannot usually access the underlying filesystem to store data, meaning we need to use native APIs.
For iOS, the Swift UserDefaults API is a key/value store similar to localStorage. It technically also works for macOS, though since macOS can have multiple app instances and UserDefaults does not have a channel for subscribing to changes, it is easier to stick with the above Desktop approach.
I have yet to find an equivalent for Android that has Rust bindings so I may mark Android as unsupported for now.

When accessing a setting, a selector will be used. This should return some form of RefCell or Arc to the setting section, allowing the setting to be mutable. Ideally, this should only trigger renders on the setting that is being selected, though I am still thinking of how to enforce this.

You could use the signals crate with the use_selector hook. It will only rerender the component when the output of the selector changes.

Under the hood
I am envisioning three implementations to start. One consideration for all three is whether to store the whole struct as a single entry or break it up and store each field as a separate key. Storing each field as a separate key could offer opportunities to only invalidate some of the entries if the structure of a setting gets changed.

All of that sounds great! This could be part of the larger use storage crate in dioxus-std. If we make how the data is stored generic so that other platforms can inject there own storage, we can have a user storage handler as an additional implementation for each platform

Some prototyping for the storage abstraction: https://github.com/DioxusLabs/dioxus-std/pull/17

I've taken the prototype shared above and added to it the ability to differentiate between storage media that is shared between sessions and those that are ephemeral to the current session. With this, I've added a subscription mechanism for watching for changes to shared storage. https://github.com/ealmloff/dioxus-std/pull/1

Closing as this is now addressed in v0.5 :)


Select Division Stage

As a ministry coordinator
I want to be able to update where a division is in responding to a CFR
so that I can track and take action to complete the CFR

Assumptions & Scope
What are the assumptions for this story?
Many ministry requests will have more than one division responding to a CFR. Divisions will be in different stages in their response.
What is IN scope?
What is NOT in scope?
Selecting what divisions are to be canvassed; #566
The overall CFR state; #567
The overall state of the request. See Epic #451
Acceptance Criteria
Scenario 1: Click Division Stage dropdown

GIVEN I am a ministry coordinator
AND I am on a request where Divisions to be canvassed have been selected (see #566)
WHEN I click on a 'Select Division Stage' dropdown for a division
THEN the dropdown opens with the following choices: Clarification, Awaiting Fees, Fees Received, Gathering Records, Records Received, Awaiting Harms, Harms Received, Pending Sign Off, Sign Off Complete
https://jacklyn808742.invisionapp.com/console/share/3W2O64AP5C/775202206
AND the already selected stage will be highlighted in grey
AND if no stage is selected none will be highlighted

Scenario 2: Select Division Stage

GIVEN I am a ministry coordinator
AND I have opened the Select Division Stage' dropdown for a division
WHEN I click on a stage
THEN the dropdown menu will close
AND the stage selected will appear on the dropdown menu

...
Dependencies? What is the impact of this dependency? (If so, link dependency in the ticket, make it visible in a team´s backlog)
Validation Rules? (If yes, list here)
Design
@xxx - please link the Design here
Definition of Ready

[ ] Is there a well articulated User Story?
[ ] Is there Acceptance Criteria that covers all scenarios (happy/sad paths)?
[ ] If there is a user interface, is there a design?
[ ] Does the user story need user research/validation?
[ ] Does this User Story needs stakeholder approval?
[ ] Design / Solution accepted by Product Owner
[ ] Is this user story small enough to be completed in a Sprint? Should it be split?
[ ] Are the dependencies known/ understood? (technical, business, regulatory/policy)
[ ] Has the story been estimated?

Definition of Done

[ ] Passes developer unit tests
[ ] Passes peer code review
[ ] If there's a user interface, passes UX assurance
[ ] Passes QA of Acceptance Criteria with verification in Dev and Test
[ ] Confirm Test cases built and succeeding
[ ] No regression test failures
[ ] Test coverage acceptable by Product Owner
[ ] Ticket ready to be merged to master or story branch
[ ] Developer to list Config changes/ Update documents and designs
[ ] Can be demoed in Sprint Review
[ ] Tagged as part of a Release
[ ] Feature flagged if required
[ ] Change Management activities done?

@m-prodan, @abin-aot, here is an additional story for review in refinement tomorrow.

Close as duplicate of #566


Add support for sqlite3Storage.db.Exec

Hi, guess this is more or less a proposal at the moment as it's not directly adding any features to the command line tool itself:
I'm using your code programmatically and would like to be able to have access to sqlite3Storage.db.Exec, i.e.:
...
s := storage.NewSQLite3StorageWithDefaults()
s.LoadInput("/path/to.csv")
r, err := s.Exec("DELETE foo from BAR")
if err != nil {
  return nil, err
}
c, err := r.RowsAffected()
if err != nil {
  return nil, err
}
s.Close()
...

I like this proposal but can not merge it as is; mostly because the names of publicly accessible functions have changed, this will break projects that depend on this code.

I do want to say thank you for taking the time to write this proposal

Thanks for the implementation, great work!


fix multiverse sources for 1604

Wouldn't it be better to use the host fact ansible_distribution_release instead of a specific version at every point where a release name is specified?

maybe yes, but currently BBB itself is pinned to 16.04. for the next bbb release the role has to be adjusted anyway, but we don't know if the support for 18.04 is introduced and 16.04 dropped then.


Error code 218: GENERIC in method keybase.1.login.login

Hello.
Can't login into my Keybase app on Linux Mint 19.1 Cinnamon:
missing provisioning key in login context Error code 218: GENERIC in method keybase.1.login.login

Keybase GUI Version: 2.13.0-20181219142726+01e9fa19eb
my log id: 13a55142752879137d7dd91c

How can I fix that?
Thanks!

Someone?

Won't get to it until the holidays are over.


fix some linkrot

Using archive.org if available

This is really awesome. Sincerely appreciate you taking time out to do this.

 Thanks for your time too!


Linear gradiant in svgxml show a plain color

Bug
I've added an svg with linear gradient using svgxml tag but it doesn't show the gradient, only a plain color, both on ios and android, devices and simulators
Environment info
React native info output:
System:
    OS: macOS 10.14.4
    CPU: (4) x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
    Memory: 44.77 MB / 8.00 GB
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 10.3.0 - /usr/local/bin/node
    npm: 6.10.3 - /usr/local/bin/npm
    Watchman: 4.9.0 - /usr/local/bin/watchman
  SDKs:
    iOS SDK:
      Platforms: iOS 12.4, macOS 10.14, tvOS 12.4, watchOS 5.3
  IDEs:
    Android Studio: 3.5 AI-191.8<IP_ADDRESS>91312
    Xcode: 10.3/10G8 - /usr/bin/xcodebuild
  npmPackages:
    react: 16.8.6 => 16.8.6
    react-native: 0.60.4 => 0.60.4
  npmGlobalPackages:
    create-react-native-app: 1.0.0
    react-native-cli: 2.0.1
    react-native-git-upgrade: 0.2.7
    react-native-svg: 6.3.1

Library version: 2.5.5
Steps To Reproduce
Import an svg with gradial and render it:
import { SvgXml } from 'react-native-svg';
import FacebookIc from '../../assets/ic_facebook_small.svg';

...

render() {
    return (<SvgXml width="30" height="30" xml={FacebookIc} />);
}

Describe what you expected to happen:
The svg should be showed with the gradient
My svg file contains
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="30" height="30" viewBox="0 0 30 30">
  <defs>
    <linearGradient id="linear-gradient" x1="0.5" x2="0.5" y2="1" gradientUnits="objectBoundingBox">
      <stop offset="0" stop-color="#f8ec46"/>
      <stop offset="1" stop-color="#f2c52d"/>
    </linearGradient>
  </defs>
  <path id="ic_facebook" d="M15,30A15,15,0,0,1,9.161,1.179,15,15,0,0,1,20.839,28.821,14.907,14.907,0,0,1,15,30ZM11.484,12.5V14.76H12.95v7.558h2.9V14.76h2.2l.169-2.256H15.845V10.868a.769.769,0,0,1,.361-.706,1.532,1.532,0,0,1,.8-.177,4.716,4.716,0,0,1,1.2.188l.308-2.219a9.744,9.744,0,0,0-2.278-.271c-.132,0-.264,0-.393.009a2.821,2.821,0,0,0-2.572,1.366,2.716,2.716,0,0,0-.323,1.265V12.5Z" transform="translate(0)" fill="url(#linear-gradient)"/>
</svg>

Seems gradientUnits="objectBoundingBox" is not working, this seems to work for now:
  <linearGradient id="linear-gradient" x1="50%" x2="50%" y2="100%">
    <stop offset="0" stop-color="#f8ec46"/>
    <stop offset="1" stop-color="#f2c52d"/>
  </linearGradient>

Can you try with the latest commit from the master branch?

:tada: This issue has been resolved in version 9.9.7 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

The problem seams to be solved! Thank you!