LUCENE-9932: Performance improvement for BKD index building

https://issues.apache.org/jira/browse/LUCENE-9932
Description
In BKD index building, the input bytes must be sorted before calling BKD writer related API. The sorting method leverages MSB Radix Sort algorithm, and the comparing method takes both the bytes itself and the DocId, but in real cases, DocIds are usually monotonically increasing. This could yield one possible performance enhancer. I found this enhancement when I dig into one performance issue in our system. Then I research on the possible solution.
Solution
DocId is usually increased by one when building index in a thread-safe way, by assuming such condition, the comparing method can eliminate the unnecessary comparing input - DocId, only leave the bytes itself to compare. In order to do so, MSB radix sorting and its fallback sorting method must be stable, so that when elements are the same, the sorting method maintains its original order when added, which makes DocId still monotonically increasing. To make MSB Radix Sort stable, it needs a trivial update; to make fallback sort table, use merge sort instead of quick sort. Meanwhile, there should introduce a switch which is able to turn the stable option on or off.
Tests
Add testcases.
Add performance testcase (maybe removed when merging).
Test environment:
MacBook Pro (Retina, 15-inch, Mid 2015), 2.2 GHz Intel Core i7, 16 GB 1600 MHz DDR3
Java version:
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
Test scenarios:
bytesPerDim = [1, 2, 3, 4, 8, 16, 32]
dim = 1
doc num = 2,000,000
warm up 5 time, run 10 times to calculate average time used.

bytesPerDim\scenario
disable sort doc id (PR branch)
enable sort doc id (master branch)

1
30989.594 us
1151149.9 us

2
313469.47 us
1115595.1 us

3
844617.8 us
1465465.1 us

4
1350946.8 us
1465465.1 us

8
1344814.6 us
1458115.5 us

16
1344516.6 us
1459849.6 us

32
1386847.8 us
1583097.5 us

Result shows that, by disabling sort DocId, sorting runs 1.73x to 37x faster when there are many duplicate bytes (bytesPerDim = 1 or 2 or 3). When data cardinality is high (bytesPerDim >= 4, test cases will generate random bytes which are more scatter, not likely to be duplicate), the performance does not go backward, still a little better.
In conclusion, in the end to end process for building BKD index, which relies on BKDWriter for some data types, performance could be better by ignoring DocId if they are already monotonically increasing.

Wow, this is a neat idea! I wonder if we could find a less intrusive way to apply this change, e.g. could we avoid adding info about whether to sort by doc ID to BKDConfig? For instance could MSBRadixSorter detect whether data is sorted by itself instead of having to propagate this information in so many places?

@jpountz Good advice! Before that I am still struggling where to propagate this config up to the index builder layer.
I will give it a try, the first thing comes up my mind is to bring up a new prepare method, in which it will scan all docid from i to j to see if they are increasing. I will experiment on this, if the overhead is small enough, then it is worthwhile to sort without docid.
One more question, are there any places where doc id is not added increasingly? I mean the source code, not test cases.

I will give it a try, the first thing comes up my mind is to bring up a new prepare method,

One idea I had in mind was to create a new class, something like StableMSBRadixSorter that would extend MSBRadixSorter to:

add the two assign and finalizeAssign methods that you currently added to Sorter,
override the way data gets rearranged to guarantee stability,
change the fallback sorter,
modify radixSort(int,int,int,int) to check whether data is already sorted before computing the common prefix length and the histogram of the leading bytes.

One more question, are there any places where doc id is not added increasingly?

I don't remember how we deal with it, but we should check how this optimization plays with index sorting, since we would renumber doc IDs at flush time.

@jpountz Per your advice, I have updated the code.
In terms of performance, I refined TestBKDDisableSortDocId, to make it re-runnable as a benchmark. I have made the following benchmark.
doc num = 2,000,000, dim number = 1, bytePerDim = [1,2,3,4,8,16,32], run times = 10 (warm up 5 times)
Result is shown as below.
  -------------------------------------------------
| bytesPerDim | isDocIdIncremental | avg time(us) |
 -------------------------------------------------
|      1      |         N          |   1127688.5  |
|      1      |         Y          |     56464.7  |
|      2      |         N          |   1124137.8  |
|      2      |         Y          |    339150.2  |
|      3      |         N          |   1485020.4  |
|      3      |         Y          |    878251.1  |
|      4      |         N          |   1436003.9  |
|      4      |         Y          |   1376974.1  |
|      8      |         N          |   1444971.3  |
|      8      |         Y          |   1365877.5  |
|     16      |         N          |   1500235.5  |
|     16      |         Y          |   1385235.8  |
|     32      |         N          |   1516514.0  |
|     32      |         Y          |   1415364.9  |
 -------------------------------------------------

Meanwhile, I also reset to main branch to run the same test case. Result is shown as below.
  -------------------------------------------------
| bytesPerDim | isDocIdIncremental | avg time(us) |
 -------------------------------------------------
|      1      |         N          |   1144138.9  |
|      1      |         Y          |    390398.8  |
|      2      |         N          |   1121301.9  |
|      2      |         Y          |   1124444.9  |
|      3      |         N          |   1482451.3  |
|      3      |         Y          |   1471338.1  |
|      4      |         N          |   1424961.6  |
|      4      |         Y          |   1423907.1  |
|      8      |         N          |   1437768.5  |
|      8      |         Y          |   1474001.1  |
|     16      |         N          |   1464370.4  |
|     16      |         Y          |   1478291.3  |
|     32      |         N          |   1500323.9  |
|     32      |         Y          |   1508646.1  |
 -------------------------------------------------

I made a graph so that we can see more clearly.

If DocIds are increasing, PR branch out-performs in all scenarios.
If DocIds are not in order, we expect the performance to be the same with main branch. It does work almost the same, but here we introduce a small overhead to scan data beforehand, checking whether data is in order, so PR branch is a little bit (like 1% percent) backward.
I made a flame-graph, the right-most column is where checking order consumes, very small, like below 3% of total CPU consumption.

I left a few comments before your previous comment, I left some suggestions to make the code simpler but I think that it's getting close.
Thanks for the great performance analysis! Given that you are benchmarking with fully random bytes, I'm actually expecting the speedup to be greater with real data where the leading bytes are often zeroes. For instance I'd expect users who index integers (4 bytes) between 0 and 2^24 to notice speedups that are closer to the one that you computed for bytesPerDim=3 than for bytesPerDim=4.

What I read for this results is that the stable sorting is faster than the non-stable. I think it makes sense because re-ordering using a helper array is probably faster that in-place re-ordering.
If we are ok to take the memory penalty of using this extra array, then I think it make sense to always use the stable version of the algorithm?

For instance I'd expect users who index integers (4 bytes) between 0 and 2^24 to notice speedups that are closer to the one that you computed for bytesPerDim=3 than for bytesPerDim=4.
I can also construct a real case to verify :smile:

If we are ok to take the memory penalty of using this extra array, then I think it make sense to always use the stable version of the algorithm?

+1 to always use the stable version of the algorithm. This would only use transient memory and in reasonable amounts, so I'm not concerned with the memory usage.

+1 to always use the stable version of the algorithm. This would only use transient memory and in reasonable amounts, so I'm not concerned with the memory usage.

Per comment, I make stable reorder in MSB Radix Sort as default.
The benchmark of this PR branch goes like below. Starting from bytePerDim=4, non-stable MSB Radix Sort runs faster than stable version.
  -------------------------------------------------
| bytesPerDim | isDocIdIncremental | avg time(us) |
 -------------------------------------------------
|      1      |         N          |    981732.7  |
|      1      |         Y          |     56766.1  |
|      2      |         N          |    941666.1  |
|      2      |         Y          |    304395.9  |
|      3      |         N          |   1306374.9  |
|      3      |         Y          |    853744.8  |
|      4      |         N          |   1262810.3  |
|      4      |         Y          |   1345605.8  |
|      8      |         N          |   1282193.9  |
|      8      |         Y          |   1364078.3  |
|     16      |         N          |   1269967.4  |
|     16      |         Y          |   1433378.3  |
|     32      |         N          |   1293686.0  |
|     32      |         Y          |   1468472.1  |
 -------------------------------------------------

To make it more clear, I also add the new result into the graph I made above.

I try to analysis, according to the flame graph I generated (see above picture). Sorting includes the following stages.

build histogram
fallback intro sort (quick sort for non-stable, in place merge sort for stable)
reorder

Let's illustrate the total time partitioned by the cost per stage.
For main branch MSBRadixSort, it shows as below.
+--------------------------------------------------------------------------------------------------------+
|  build histogram |      fallback sort (quick sort)     |             non stable reorder                |
+--------------------------------------------------------------------------------------------------------+

For PR branch StableMSBRadixSort, it shows as below
+-------------------------------------------------------------------------------------------------+
|  build histogram |         fallback sort (in place merge sort)        |    stable reorder       |
+-------------------------------------------------------------------------------------------------+

For PR branch MSBRadixSort making stable reorder as default, it shows as below.
+----------------------------------------------------------------------------------+
|  build histogram |       fallback sort (quick sort)     |    stable reorder      |
+----------------------------------------------------------------------------------+

I think that stable reorder contributes most part of the speedup, but in place merge sort is slower than 3-pivot quick sort.
After all, regardless of the order of doc IDs, stable reorder is definitely a good choice. If data are spread evenly, not many duplicates, StableMSBRadixSort is a little slower (10%) than MSBRadixSorter. If not, StableMSBRadixSort will out-perform far better than MSBRadixSorter. This makes me reluctant on the way we should go, @jpountz could you give me some advice?

For instance I'd expect users who index integers (4 bytes) between 0 and 2^24 to notice speedups that are closer to the one that you computed for bytesPerDim=3 than for bytesPerDim=4.

@jpountz I add a new benchmark test case testBenchmarkWithLeadingZeroBytes in TestBKDDisableSortDocId. Your assumption is correct.
Result is shown as below (do not make stable reorder as default).
bytesPerDim=4, leadingZeroByteNum=[1,2,3]
 -------------------------------------------------
| bytesPerDim | isDocIdIncremental | avg time(us) |
 -------------------------------------------------
|      4      |         N          |   1476874.0  |  <- leadingZeroByteNum = 1
|      4      |         Y          |   1363280.3  | <- leadingZeroByteNum = 1
|      4      |         N          |   1601885.9  | <- leadingZeroByteNum = 2
|      4      |         Y          |    930728.1  | <- <- leadingZeroByteNum = 2
|      4      |         N          |   1210586.8  | <- leadingZeroByteNum = 3
|      4      |         Y          |    367141.6  | <- <- leadingZeroByteNum = 3
 -------------------------------------------------

Very cool, thanks for testing what happens with small integers!
Do the numbers get better if you use TimSorter as a fallback sort? InPlaceMergeSorter is convenient because it only requires implementing a few methods, but the logic it uses to merge sorted arrays in-place is very slow, so I wonder if that's the cause of the slow down we're seeing.

I use TimSort instead of InPlaceMergeSorter, expect it to be faster, but it turns out to be slower. @jpountz would you check my latest commit to see if I implement Tim Sort correctly?
Below is the latest benchmark of MSBRadixSort with stable reorder(isDocIdIncremental = N) and StableMSBRadixSort (isDocIdIncremental = Y)
 -------------------------------------------------
| bytesPerDim | isDocIdIncremental | avg time(us) |
 -------------------------------------------------
|      1      |         N          |    995541.5  |
|      1      |         Y          |     60399.2  |
|      2      |         N          |    951085.9  |
|      2      |         Y          |    322054.3  |
|      3      |         N          |   1333992.5  |
|      3      |         Y          |    756951.4  |
|      4      |         N          |   1340422.4  |
|      4      |         Y          |   1528955.5  |
|      8      |         N          |   1323878.8  |
|      8      |         Y          |   1494004.5  |
|     16      |         N          |   1305548.1  |
|     16      |         Y          |   1480329.4  |
|     32      |         N          |   1326447.5  |
|     32      |         Y          |   1589089.8  |
 -------------------------------------------------

Oh, this is disappointing, maybe the arrays are too small for TimSorter to actually perform better than InPlaceMergeSorter.
I'd be keen to proceed with the change that always performs a stable sort with InPlaceMergeSorter. Some cases do get slower but only by a few percents, and it's going to be unlikely noticed through the full indexing chain. On the other hand, some cases are getting several times faster, which I'm sure is going to be noticeable. We could still iterate later, but for now this sounds to me like a good performance-simplicity trade-off. What do you think?

I make StableMSBRadixSorter as default sorter and use InPlaceMergeSorter as fallback sorter. Please check the latest commit. (I did not squash commits, and save every commit as separate branch in my own forked repo for future reference).
Average sort time (unit in us) comparison.
(1) PR branch
(2) main branch
(3) PR branch, verify doc ID not in ascending order, original MSBRadixSorter with stable reorder.
 -------------------------------------------------------------------------------
| bytesPerDim | isDocIdIncremental |      (1)     |      (2)     |     (3)     |
 -------------------------------------------------------------------------------
|      1      |         N          |     31132.7  |   1144138.9  |    981732.7  |
|      1      |         Y          |     33049.8  |    390398.8  |              |
|      2      |         N          |    273975.1  |   1121301.9  |    941666.1  |
|      2      |         Y          |    281060.2  |   1124444.9  |              |
|      3      |         N          |    844291.7  |   1482451.3  |   1306374.9  |
|      3      |         Y          |    804839.8  |   1471338.1  |              |
|      4      |         N          |   1274670.8  |   1424961.6  |   1262810.3  |
|      4      |         Y          |   1289128.8  |   1423907.1  |              |
|      8      |         N          |   1357592.3  |   1437768.5  |   1282193.9  |
|      8      |         Y          |   1286732.0  |   1474001.1  |              |
|     16      |         N          |   1366177.6  |   1464370.4  |   1269967.4  |
|     16      |         Y          |   1353213.1  |   1478291.3  |              |
|     32      |         N          |   1403655.4  |   1500323.9  |   1293686.0  |
|     32      |         Y          |   1406872.4  |   1508646.1  |              |
 -------------------------------------------------------------------------------

In conclusion, PR branch runs faster than main branch regardless of bytePerDim and isDocIdIncremental, some cases 38x times faster. If we make original MSBRadixSorter with stable reorder, PR branch is little bit slower but acceptable. I think in real case, doc ID is usually in ascending order when index building, and that could be a noticeable performance improvement.

We could still iterate later, but for now this sounds to me like a good performance-simplicity trade-off. What do you think?
I have no problem 😄

@jpountz are there any testcases suitable to verify the end to end performance improvement, like through IndexWriter? maybe I could give it try.

Maybe try running this benchmark: https://github.com/mikemccand/luceneutil/blob/master/src/main/perf/IndexAndSearchOpenStreetMaps1D.java. Instructions on how to get the data are available at the top of https://github.com/mikemccand/luceneutil/blob/master/src/main/perf/IndexAndSearchOpenStreetMaps.java. It indexes lat,lon data from geonames as 1D points. It's not the most realistic benchmark (NYC taxis would be more realistic, but it's also much more complicated to set up) but should give us a good idea of how much better/worse things might get with a dataset that consists mostly of 1D points.

I spent some time trying to use the real case benchmark. The speedup of IndexWriter is what we expected, faster than main branch, total time elapsed (include adding doc, building index and merging) decreased by about 20%. If we only consider flush_time, the speedup is more obvious, time cost drops about 40% - 50%.

Run IndexAndSearchOpenStreetMaps1D.java against the two branches and take down the log.
note: comment query stage, modify some of the code to adapt to latest Lucene main branch.

main branch:
# egrep "flush time|sec to build index" open-street-maps.log
DWPT 0 [2021-04-27T11:33:04.518908Z; main]: flush time 17284.537739 msec
DWPT 0 [2021-04-27T11:33:37.888449Z; main]: flush time 12039.476885 msec
72.49147722 sec to build index

PR branch:
#egrep "flush time|sec to build index" open-street-maps-optimized.log
DWPT 0 [2021-04-27T11:35:00.619683Z; main]: flush time 9313.007647 msec
DWPT 0 [2021-04-27T11:35:29.575254Z; main]: flush time 8631.820226 msec
59.252797133 sec to build index

Further more, I come up with an idea to use TPC-H LINEITEM to verify. I have a 10GB TPC-H dataset and develop a new test case to import the first 5 INT fields, which is more typical in real case.

Run IndexAndSearchTpcHLineItem.java against the two branches and take down the log.
main branch:
egrep "flush time|sec to build index" tpch-lineitem.log
DWPT 0 [2021-04-27T11:17:25.329006Z; main]: flush time 13850.23328 msec
DWPT 0 [2021-04-27T11:17:50.289370Z; main]: flush time 12228.723665 msec
DWPT 0 [2021-04-27T11:18:15.546002Z; main]: flush time 12537.085005 msec
DWPT 0 [2021-04-27T11:18:40.140413Z; main]: flush time 11819.225223 msec
DWPT 0 [2021-04-27T11:19:04.850989Z; main]: flush time 12004.380921 msec
DWPT 0 [2021-04-27T11:19:29.435183Z; main]: flush time 11850.273453 msec
DWPT 0 [2021-04-27T11:19:54.016951Z; main]: flush time 11882.316067 msec
DWPT 0 [2021-04-27T11:20:18.932727Z; main]: flush time 12223.151464 msec
DWPT 0 [2021-04-27T11:20:43.522117Z; main]: flush time 11871.276323 msec
DWPT 0 [2021-04-27T11:20:52.060300Z; main]: flush time 3422.434221 msec
271.188917715 sec to build index

PR branch:
 egrep "flush time|sec to build index" tpch-lineitem-optimized.log
DWPT 0 [2021-04-27T11:24:00.362128Z; main]: flush time 7573.05091 msec
DWPT 0 [2021-04-27T11:24:19.498948Z; main]: flush time 7355.376016 msec
DWPT 0 [2021-04-27T11:24:38.602117Z; main]: flush time 7287.306154 msec
DWPT 0 [2021-04-27T11:24:57.541930Z; main]: flush time 7227.514396 msec
DWPT 0 [2021-04-27T11:25:16.474158Z; main]: flush time 7236.208865 msec
DWPT 0 [2021-04-27T11:25:35.339855Z; main]: flush time 7152.876269 msec
DWPT 0 [2021-04-27T11:25:54.111110Z; main]: flush time 7080.405571 msec
DWPT 0 [2021-04-27T11:26:12.985489Z; main]: flush time 7188.012278 msec
DWPT 0 [2021-04-27T11:26:31.857053Z; main]: flush time 7176.303704 msec
DWPT 0 [2021-04-27T11:26:38.838771Z; main]: flush time 2185.742347 msec
213.175509249 sec to build index

For benchmark command, please refer to my document.
Test environment:
CPU:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
Stepping:              4
CPU MHz:               2500.000
BogoMIPS:              5000.00
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              33792K
NUMA node0 CPU(s):     0-31

Memory:
$cat /proc/meminfo
MemTotal:       65703704 kB

Disk: SATA
$fdisk -l | grep Disk
Disk /dev/vdb: 35184.4 GB,<PHONE_NUMBER>8832 bytes,<PHONE_NUMBER>6 sectors

OS:
Linux 4.19.57-15.1.al7.x86_64

JDK:
openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.11+9-LTS, mixed mode, sharing)

Thanks for taking time working on my branch. I merged your change into this PR, the code looks much better 😄.
I was wondering which test case do I neglect besides TestIndexSorting? I agree that maybe we could work on improving index sorting scenario later.

Agreed, this change helped me notice that we are probably lacking coverage for index sorting. Since this is a pre-existing issue, let's work on it in a separate pull request. In my opinion, the two important bits that are important to test with your PR are MutablePointsReaderUtils in the case when doc IDs are incremental, which looks covered already, and StableMSBRadixSorter, let's add a new test for this one similar to TestMSBRadixSorter and with additional checks regarding stability?

Comments addressed.

Reuse offset array with size of HISTOGRAM_SIZE in reorder.
Update CHANGES document.
Remove benchmark test case.
Add logic to check doc IDs' ordering in TestMutablePointsReaderUtils.java.
Add new test case for StableMSBRadixSorter.java (by copying TestMSBRadixSorter and only change the sorter, is that ok?)

@neoremind Did you forget to push your latest commit?
No need to squash yourself, I'll do it upon merge via the GitHub UI. :)

@jpountz sorry, forget to push :sweat_smile:

This looks good to me. Can we better check that the sort is actually stable in the tests? E.g. maybe we could verify that the arrays are not only equal after sorting with Arrays#sort and StableMSBRadixSorter but also that they have the very same instance at every index?

Add a new assertion assertSame(points[i].packedValue, reader.points[i].packedValue);.

Can we test it in TestStableMSBRadixSorter too? It would be a pity if bugs in StableMSBRadixSorter were only caught by tests for points.

Comment addressed.

@jpountz It's great to work with you on this optimization :smile: Thanks for taking so much time to help me.

@neoremind I enjoyed it too. Thanks for identifying this opportunity for speedup and going through the many feedback iterations.


Won't compile on macOS 10.15

When I go to try and compile .scss files I get the following error:
Errno::EROFS: Read-only file system @ dir_s_mkdir - ./.sass-cache
There should be a way to disable the cache in the settings.

any update?


Adding a dual stack rule with IPv6-only L4 protocol prevent the firewall from starting

Important notices
Describe the bug
When adding a IPv4+IPv6 rule applying to a IPv6-ICMP protocol, I am not getting any error when applying the rule from the web UI, but at the next reboot of the device, the firewall service will fail to start and complain of an error at the following line (/tmp/rules.debug)
pass in quick on igb1_vlan101 inet proto ipv6-icmp from {any} to {any} keep state label "f319cdcfed31fd3af08b790a85cec5ef"

Disabling the rule in the web ui fix the issues, maybe there should be some kind of validation to avoid putting this kind of rule ?
To Reproduce
Steps to reproduce the behavior:

Go to your firewall rules
Add a new 'IPv4+IPv6' rule with a L4 protocol condition to IPv6-ICMP
Scroll down to 'Apply the firewall changes'
Reboot
You will see in the boot logs that the firewall rules have failed to load

Expected behavior
An error should show up when validating the form or when reloading the rules first, to prevent the whole firewall to break at reboot.
Environment
OPNsense 20.7.5-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1h  22 Sep 2020

What is the error message in this case?

/tmp/rules.debug:xxx: proto icmp6 doesn't match address family inet

Yes indeed this was the error message, I could not reproduce the issue without breaking my network as I don't have a lab setup for opnsense :)
Also do we know if the opposite happens ? (ICMP protocol when address family is IPv6 will break it too?)
Thanks for the fix :)


Regen with response error

Contributing to the Azure SDK
Depends on this PR https://github.com/Azure/azure-sdk-for-net/pull/27858
And this autorest update https://github.com/Azure/autorest.csharp/pull/2131
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check for Azure.ResourceManager.ServiceBus
API changes are not detected in this pull request for Azure.ResourceManager.ServiceBus

API change check for Azure.ResourceManager.Resources
API changes have been detected in Azure.ResourceManager.Resources. You can review API changes here
API changes
-         public ErrorDetail Error { get; }
+         public ResponseError Error { get; }
-         public ErrorDetail Error { get; }
+         public ResponseError Error { get; }
-         public ErrorDetail Error { get; }
+         public ResponseError Error { get; }
-         public ErrorDetail Error { get; }
+         public ResponseError Error { get; }
-         public ErrorDetail Error { get; }
+         public ResponseError Error { get; }

API change check for Azure.ResourceManager
API changes have been detected in Azure.ResourceManager. You can review API changes here

API change check for Azure.ResourceManager.EdgeOrder
API changes have been detected in Azure.ResourceManager.EdgeOrder. You can review API changes here
API changes
-         public ErrorDetail Error { get; }
+         public ResponseError Error { get; }

API change check for Azure.ResourceManager.ConnectedVMwarevSphere
API changes have been detected in Azure.ResourceManager.ConnectedVMwarevSphere. You can review API changes here
API changes
-     public class ErrorDetail {
-         public string Code { get; }
-         public IReadOnlyList<ErrorDetail> Details { get; }
-         public string Message { get; }
-         public string Target { get; }
-     }
-         public IReadOnlyList<ErrorDetail> ErrorDetails { get; }
+         public IReadOnlyList<ResponseError> ErrorDetails { get; }

API change check for Azure.ResourceManager.Dashboard
API changes are not detected in this pull request for Azure.ResourceManager.Dashboard

API change check for Azure.ResourceManager.Hci
API changes are not detected in this pull request for Azure.ResourceManager.Hci

API change check for Azure.ResourceManager.AppService
API changes have been detected in Azure.ResourceManager.AppService. You can review API changes here

API change check for Azure.ResourceManager.HybridConnectivity
API changes are not detected in this pull request for Azure.ResourceManager.HybridConnectivity

API change check for Azure.Core
API changes are not detected in this pull request for Azure.Core

API change check for Azure.ResourceManager.StoragePool
API changes are not detected in this pull request for Azure.ResourceManager.StoragePool

API change check for Azure.ResourceManager.Communication
API changes are not detected in this pull request for Azure.ResourceManager.Communication

API change check for Azure.ResourceManager.Network
API changes are not detected in this pull request for Azure.ResourceManager.Network

API change check for Azure.ResourceManager.WebPubSub
API changes have been detected in Azure.ResourceManager.WebPubSub. You can review API changes here
API changes
-         public virtual Response<PrivateEndpointConnectionResource> GetIfExists(string privateEndpointConnectionName, CancellationToken cancellationToken = default);
-         public virtual Task<Response<PrivateEndpointConnectionResource>> GetIfExistsAsync(string privateEndpointConnectionName, CancellationToken cancellationToken = default);
-         public virtual Response<SharedPrivateLinkResource> GetIfExists(string sharedPrivateLinkResourceName, CancellationToken cancellationToken = default);
-         public virtual Task<Response<SharedPrivateLinkResource>> GetIfExistsAsync(string sharedPrivateLinkResourceName, CancellationToken cancellationToken = default);
-         public virtual Response<WebPubSubResource> GetIfExists(string resourceName, CancellationToken cancellationToken = default);
-         public virtual Task<Response<WebPubSubResource>> GetIfExistsAsync(string resourceName, CancellationToken cancellationToken = default);
-         public virtual Response<WebPubSubHubResource> GetIfExists(string hubName, CancellationToken cancellationToken = default);
-         public virtual Task<Response<WebPubSubHubResource>> GetIfExistsAsync(string hubName, CancellationToken cancellationToken = default);

API change check for Azure.ResourceManager.Storage
API changes are not detected in this pull request for Azure.ResourceManager.Storage

API change check for Azure.ResourceManager.DnsResolver
API changes are not detected in this pull request for Azure.ResourceManager.DnsResolver

API change check for Azure.ResourceManager.Cdn
API changes are not detected in this pull request for Azure.ResourceManager.Cdn

API change check for Azure.ResourceManager.Monitor
API changes are not detected in this pull request for Azure.ResourceManager.Monitor

changes moved to https://github.com/Azure/azure-sdk-for-net/pull/27961


Proposal: move tooling that's used within nixos and nixpkgs in-tree

Issue description
Currently we have some important tooling developed out-of-tree:

nix-update, used as an update script and even recommended in the nixpkgs manual itself
nixpkgs-update, used by the update bot
calamares-nixos-extensions, used in the graphical ISOs
nix-init, used as a way to generate new package expressions, with explicitly designed support for nixpkgs
etc.
(There may be more projects that I've not recalled offhand.)

However, if we want to fix issues, this depends on an external project, which in almost all cases has a bus factor of 1. Therefore we are hindered by the pace of external development rather than being able to solve our own problems in-tree. Checking the list of open, unresponded-to issues and PRs in the above repos should give a good idea of the concern.
It also prevents community consensus from being able to improve said tooling, instead depending on a single maintainer to respond and agree.
If there's no defensible objection, I propose moving such tooling in-tree.
CC @Mic92 @ryantm @vlinkz @figsoda for the projects mentioned above

What I am worried about is that one needs to compromise on CI. For nix-update we need to impurely fetch stuff from the internet and I would like to keep doing this as mocking is very hard to do reliable. Also all the usual other CI checks such as treefmt/mypy and so on. Also issues related to these tools would get very easily drowned by all the other issues with have in nixpkgs already, so they are not visible to both the maintainers and the user.
Project wise I try to add at least one other person with git commit access to all my important tooling. For nix-update, it's currently @figsoda and @SuperSandro2000. If you have more maintainer suggestions, let me know. There is one open pull request and one pull request marked as draft. Releases happen often after important features are added.

Absolute not!
The nixpkgs tree is already filled with multiple projects which shouldn't be in here like yarn2nix.
The correct move would be to move them out but keep them in the NixOS org or at least in nix-community.
For nix-update I would suggest to move that to nix-community.

What I am worried about is that one needs to compromise on CI.

Thanks for raising the point about CI, it would definitely change how we have to test the relevant programs - if any of the tooling were to move in-tree, that must be a primary consideration. If it helps, I can draft how the testing would look like to make sure all the primary use cases are covered.

issues related to these tools would get very easily drowned by all the other issues with have in nixpkgs

I agree this risk is present. nix-update specifically is in a better shape, especially since as you mentioned there are others with commit access, so I am OK with considering it a special  case of "let's not break what seems to be working".
I'm still interested in addressing the other projects mentioned. If the filed issues aren't addressed in the respective upstream projects, then they get created here anyway, and we just have to slap a "waiting for upstream fix" label or such.

The correct move would be to move them out but keep them in the NixOS org or at least in nix-community.

nix-init is part of nix-community already, and the calamares scripts are part of the nixos org itself, but the bus factor in each of those cases is still 1. I myself have PRs open in both of these repos for months that basically got ignored, and it's not like we can request someone else to review because there is only one person to review. How does changing the repo owner solve the issues raised?
And for r-ryantm, the bot is somehow allowed to spam the repo, but we cannot make the code changes needed to stop it from doing so, because it's yet another a one-person external project; how is that reasonable either?

What I am worried about is that one needs to compromise on CI.

That's not even the biggest problem with CI. If it is out of tree, you can just add new CI checks without much consideration and if it breaks things, it's just an out of tree repo, not that painful. If it would be in tree, every CI change would need to be carefully reviewed to not impact any other files. Scalability is already a big concern and would then be a major factor in choosing CI actions.
Also some tools can only be configured by placing config files in the root of the repository. This would easily clutter the repository.
Also we will easily find projects which want conflicting settings which are then no longer possible.
Triggering ofborg for every push of every project is very expensive and just wasteful.
Configuring CI actions could also become more challenging if we need to carefully configure everything to not work tree wide.
Installing rust crates from git would also no longer be a viable option of cloning the repo can take multiple minutes.

I myself have PRs open in both of these repos for months that basically got ignored, and it's not like we can request someone else to review because there is only one person to review.

Then the root problem is that there are to little maintainers and we need to introduce more together with the original maintainers.

How does changing the repo owner solve the issues raised?

That we have more power in nix-community to add new maintainers in case the original are totally absent. It basically prevents a hard fork.
Also new maintainers can be added with just a few clicks without them having to accept invites to new orgs.

The main goal of moving this in-tree would be to raise the number of people that can merge from 1 to 200-something.

Will that solve the problem? I doubt it. It would at the same time diffuse responsibility between many people and it wouldn't increase the amount of people with good knowledge about the code and that feel responsible to keep the tool in good working condition.

And in-tree PRs get more opportunity for feedback from nixpkgs contributors (since these projects have an impact on nixpkgs).

For very deeply coupled projects this is correct but most tools you listed are just loosely coupled and work as well with our of tree projects.

Probably the best way is to open an issue, offer your help and see where that goes. If the repo is in nix-community and there is no reaction for a month or so, you can probably ask in the matrix, if you could take over the project.

I see, thanks for explaining.


feat: bump up cosmwasm from v1.1.9 to v1.4.0

Description

Adopted Cosmwasm/cosmwasm v1.4.0 to Finschia/cosmwasm.
This changes the base version of Finschia/cosmwasm from v1.1.9 to v1.4.0.
The test involving cosmwasm, wasmvm, and wasmd has not been done yet, so we will first merge into feat/bump_1.4.0. Once the test with all components working together is successful, will merge into main.
Version bump as follows

https://github.com/Finschia/cosmwasm/pull/331/commits/79abfdd897ffe3f024f3368b1e3e58c699a65d2b applied Cosmwasm/cosmwasm v1.4.0 to Finschia/cosmwasm.
Subsequent commits were made to adapt the changes made in v1.4.0 to Finschia/cosmwasm.

Main changes

https://github.com/Finschia/cosmwasm/pull/331/commits/e43cc7abd50269a0fc9ab0544ffa67b9225f7285 -new_native_with_env -> new_typed_with_env

The function names have been changed since wasmer3.

https://github.com/Finschia/cosmwasm/pull/331/commits/1812811e6c9a6cdc4c36c0803410ff05c79f1681, https://github.com/Finschia/cosmwasm/pull/331/commits/6bc3424ea99922d451dd3cef3e58be1e6c756074 - Changed the default rust toolchain for CI from 1.60.0 to 1.67.0

Cargo.lock is currently the latest among the package dependencies and a build error occurred, so only cosmwasm-vm is set to 1.70.0.

https://github.com/Finschia/cosmwasm/pull/331/commits/02a2d6983f6ddb804e34befe12d55376af6c8b69 - The current version of wat is v1.0.78, so I corrected the old wat expressions.
https://github.com/Finschia/cosmwasm/pull/331/commits/1dc3259484f04c861c9fbc7435e6f2560a79f19d - Since cosmwasm-storage has been deprecated, I made modifications to no longer use cosmwasm-storage.

See: https://github.com/CosmWasm/cosmwasm/commit/097593e413ad6ebd38815aab3a9ec16c7a26e9fa, https://github.com/CosmWasm/cosmwasm/commit/336ce91197738c0fb3bed12fa8345abc1667681c

https://github.com/Finschia/cosmwasm/pull/331/commits/811bc7c0f6ab8ce1e6761a9a53802bc99a56aa9a - Modified the tests to ensure that each engine and module pair properly corresponds one-to-one.

Types of changes

[ ] Bug fix (changes which fixes an issue)
[x] New feature (changes which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] ETC (build, ci, docs, perf, refactor, style, test)

Checklist

[x] I followed the contributing guidelines.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[x] The PR title and commits are followed conventional commit form.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 3 committers have signed the CLA.:white_check_mark: da1suk8:x: chipshort:x: webmaster128You have signed the CLA already but the status is still pending? Let us recheck it.

It seems that many links are linking to Cosmos, starting with: https://github.com/cosmos/cosmos-sdk/blob/.
Is it necessary to change them to FInschia/finschia-sdk


Handling for multiple dictionary files and selection of the latest version

Created a new utility function found at version-utils.R.
The function is responsible for parsing a single string or vector of string with file names and returning the latest version as well as file name with the latest version.
Modified _dictionary-version.qmd to utilize the new function.

Warning contains extra dictionaries to allow for testing. Should remove other versions prior to merge.

@yulric  Ready for review

@yulric Ready for review


I'm trying to take:

0 to a power of NaN;
NaN to a power of 0.

https://go.dev/play/p/OdD0TalqBrN
What did you see happen?
I see output:
NaN
1

What did you expect to see?
The comments to the Pow function give examples of return values:
https://github.com/golang/go/blob/db6097f8cbaceaed02051850d2411c88b763a0c3/src/math/pow.go#L31-L32
The result should be as in the comments or it should return 1 1 as it does in the IEEE Std. 754-2008 "Section 9.2.1 Special values". Either way it seems to me that these two edge cases should return the same value.
It's probably due to a case order problem:
https://github.com/golang/go/blob/db6097f8cbaceaed02051850d2411c88b763a0c3/src/math/pow.go#L55-L62
This example should handle the case on line 61. But due to a higher-priority case on line 57 an incorrect value is returned.
I think it would be more correct to move a case with NaN checks to the begin or move all edge cases at the beginning of the pow function from switch-case to if statement like this:
func pow(x, y float64) float64 {
	if IsNaN(x) || IsNaN(y) {
		return NaN()
	}

	if y == 1 {
		return x
	}

	// Other edge cases ...

	// Calculate pow ...
}

feel free to submit a CL/PR

This behavior is correct given the special cases listed in the docs for Pow. The zero exponent is the first, and highest priority, special case.

@randall77

This behavior is correct given the special cases listed in the docs for Pow. The zero exponent is the first, and highest priority, special case.

How behavior can be correct if there is one result in the function's comments and docs and another result in fact? Either the comments and documentation are lying or this case is being handled incorrectly. Please write why this is correct behavior, given that doc and the IEEE 754-2008 standard state the opposite.

Read the doc comment carefully.
Special cases are (in order):
//	Pow(x, ±0) = 1 for any x
...
//	Pow(NaN, y) = NaN

Note the "in order", and the fact that the first one matches the input described.
I don't know whether we diverge from IEEE 754 or not, but even if we did this is certainly a weird enough case that it doesn't warrant making a backwards-incompatible change to the spec and behavior.

From IEEE 754 2008:
For the pow function (integral exponents get special treatment):
pow(x, ±0) is 1 for any x (even a zero, quiet NaN, or infinity)


Disable AmmySidekick

I was thinking about a scenario where I wouldn't want to include the AmmySidekick.dll. For development it's great, but for shipping code I wouldn't want to include it. And I also wouldn't want someone being able to add it to a directory and suddenly have an open port that can be used to change the UI of the app. Security nightmare.
So, first I tried removing AmmySidekick.dll from the output folder. This causes the app to crash trying to look for the assembly.
So then I removed the reference from the project. This caused the Ammy compiler to crash.
1>------ Build started: Project: AmmyQuickstart, Configuration: Debug Any CPU ------
1><Default>(1,1,1,1): error : Ammy compilation error: System.NullReferenceException: Object reference not set to an instance of an object.
1><Default>(1,1,1,1): error :    at Ammy.Language.AstTopExtensions.BuildXaml(TopWithNode top, TypeSymbol nodeType, ImmutableArray`1 members, String rootSymbolId, DependentPropertyEvalContext context)
1><Default>(1,1,1,1): error :    at Ammy.Language.TopWithNode.EvalProperties(DependentPropertyEvalContext context)
1><Default>(1,1,1,1): error :    at Ammy.Language.Start.EvalProperties(DependentPropertyEvalContext context)
1><Default>(1,1,1,1): error :    at Nitra.Declarations.EvalPropertiesHost.EvalProperties(DependentPropertyEvalContext context, IDependentPropertyContainer obj, Single statistics)
1><Default>(1,1,1,1): error :    at Nitra.Declarations.ProjectEvalPropertiesHost.ExecutePass(DependentPropertyEvalContext context, String passName)
1><Default>(1,1,1,1): error :    at Nitra.Declarations.EvalPropertiesHost.EvalProperties(DependentPropertyEvalContext context, String passName, Int32 stage)
1><Default>(1,1,1,1): error :    at Ammy.Language.Start.RefreshProject(CancellationToken _cancellationToken, ImmutableArray`1 files, Object data)
1><Default>(1,1,1,1): error :    at Ammy.Build.AmmyProject.RefreshProject(String outputPath, String rootNamespace, String assemblyName)
1><Default>(1,1,1,1): error :    at Ammy.Build.AmmyCompiler.Compile(AmmyProject project, Boolean generateMetaFile, Boolean needBamlGeneration, Boolean needUpdate)
1>I:\src\other\AmmyQuickstart\packages\Ammy.1.1.35\build\Ammy.targets(16,5): error : Ammy compilation failed
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

It would be nice to have an Ammy compile only option that doesn't include the runtime UI updating stuff, and doesn't need the AmmySidekick.dll.

+1, we don't need this is in the release builds for WPF applications. So it would be great if Ammy doesn't include AmmySidekick if there are no DEBUG symbol defined.
It should be not hard as AmmyBackend already includes bool parameter determining whether it should include AmmySidekick into the produced XAML code. And AmmySidekick.dll perhaps should not be referenced by the project at all, but if it should be referenced, it could be excluded from release build by configuration checking in the csproj file, like this:
<ItemGroup Condition="'$(Configuration)' != 'Release'">
    <Reference Include="AmmySidekick" />
</ItemGroup>

I don't know if it possible to add a condition by symbol, but configuration condition like in this example should be also good for most cases.

Conditions for adding Register are following:

DEBUG compilation symbol defined
NO_AMMY_UPDATE compilation symbol is not define

Although, they both don't work in the most recent version of Ammy since I've added SupportsRuntimeUpdate option to platform configuration. This bug is already fixed and everything should work properly in the next version.
As for condition for including AmmySidekick.dll, I'm not sure it is possible to add condition with NuGet. If it's not possible, then the easiest way would be to just add documentation article on how to exclude AmmySidekick.dll conditionally.

@ionoy Awesome!
I ❤ this project.

Thanks, man!
Just remembered. AmmySidekick.dll contains both runtime update logic and ExpressionConverter that is used for binding expressions. So, to exclude this dependency would mean binding expression not working anymore. Is referencing AmmySidekick.dll in non-debug scenarios a problem?

@ionoy, I think it would be better to have separate assemblies for ExpressionConverter (AmmyHelper.dll?) and AmmySidekick.

Yeah that's a problem, given AmmySidekick.dll also has that open port that requires users to open their firewall. This blocks me from using Ammy in commercial projects.
Apart from @aienabled's suggestion of splitting the 2 libraries, could Ammy provide the code for the bits it needs to the compiler so it's compiled directly into the assembly, rather than being a reference? Given Ammy is performing compiler magic anyway.

Ammy wouldn't require an open port in non-debug or NO_AMMY_UPDATE configuration. Port is opened as soon as the first element is registered with Ammy.Register.

could Ammy provide the code for the bits it needs to the compiler so it's compiled directly into the assembly

What do you mean by that?

@distantcam, have you seen the generated XAML code when the inline binding converters are used? Ammy serializes C# code to the binding expression and uses ExpressionConverter as the binding converter, which is included into AmmySidekick.dll currently. So Ammy doesn't generate any C# code, in provides only MSBuild targets for converting Ammy to XAML files.
So I propose to move ExpressionConverter into the separate assembly.

@aienabled No I haven't played with binding in Ammy yet. So I'm not sure how it works under the hood.
@ionoy I've just noticed the Ammy task already outputs compiled files as AmmyGeneratedItems. That's what I meant. You could also add the ExpressionConverter that way.

Yeah, separating binding logic into its own assembly seems like a good idea. Runtime update code is very platform specific, so bundling it with code that is not platform specific seems wrong. There is also an SRP argument there.
@distantcam This is actually an interesting idea. Including code behind the scene instead of referencing platform dependent assembly. What would be the disadvantages?

Even if ExpressionConverter code is compact and doesn't carry any dependencies it will be generated per project. So each project will have it's own ExpressionConverter code. Might be not a really good idea.

There is more code than just ExpressionConverter, so it isn't exactly compact. I can see a problem if there are multiple XAML-based projects inside the same solution that reference each other. Nothing major would happen (probably), but if there would be different versions of Ammy for each project it could lead to interesting bugs.

What would be the disadvantages?

Code collisions because someone has an Ammy class in their project for some reason. This can happen so sometimes. It's best to use a name like ___COMPILED___AMMY, which hopefully would never be used. The harder solution would be to pre-compile the source using Roslyn, detect if there is a collision, and change the code output, but that's a lot of work and the simpler solution should work.
Also the project not having the right references for the code to compile. Again, just add to the references during the MSBuild task.

if there would be different versions of Ammy for each project it could lead to interesting bugs.

Another reason to compile it into the assembly. Make the class internal and it's not a problem. :)
The ExpressionConverter source would be revealed this way, if that's a concern.

Revealing ExpressionConverter code is not an issue. It's kinda open at the moment since you can decompile AmmySidekick.
I like the idea of ExpressionConverter being included as an internal code. This means we don't need a separate assembly, which means no problems with conflicting dlls. It makes Ammy compilation a little bit more complicated since there is no actual reference with ExpressionConverter inside. I would have to create this TypeSymbol manually before resolving namespaces.
So, my current understanding of a solution is:

Include ExpressionConverter as code inside ammy.targets
Include conditional reference to AmmyRuntimeUpdate.dll inside ammy.targets


Add support to install extensions directly from custom stores

Description: I would like to request that the Kiwi Browser development team add support for installing extensions from custom stores other than Google Chrome's Web Store. This would allow users to install and use extensions from various sources directly (preferably without having to switch to desktop mode, if possible), expanding the ecosystem of directly installable extensions for Kiwi Browser users without having to enable Developer Mode to load locally downloaded extension files.
Expected Behavior: When support for installing extensions from custom stores is added, I expect to be able to browse and install extensions from custom stores like Microsoft Edge Add-ons for example, and customize the extension stores list in the installation sources within the Extensions menu.
Additional Context: Microsoft Edge extensions are built using the Chromium extension framework, which is also used by Kiwi Browser. I was able to successfully install and use Edge extensions from locally downloaded files, which means that Edge extensions are already compatible with Kiwi Browser. However, I still can't install them directly from the Edge Add-ons store due to some incompatibility issue perceived by the website that results in disabling the "Get" installation button and viewing this message whenever I select any extension:

To install add-ons, you'll need the new Microsoft Edge.Download the new Microsoft Edge

Your suggestion is appreciated but in case you are looking for;
Microsoft Edge Canary (Android) now let's you use Extension from Microsoft Store and also allows side loading (.crx)

Your suggestion is appreciated but in case you are looking for; Microsoft Edge Canary (Android) now let's you use Extension from Microsoft Store and also allows side loading (.crx)

Thank you for this information, I didn't know this before. But as far as I'm aware, the canary version of Microsoft Edge is not stable and can have more bugs. Add to this the fact that Microsoft Edge is not open source.


Add cgkeenan's file

created a branch, created a file and made a commit

created a branch, created a file and made a commit


Add task for adding/moving forms in groups

What problem does this pull request solve?
Trello card: https://trello.com/c/7EkOqWib/1605-add-rake-task-to-move-forms-from-one-group-to-another

We expect that users will want to move forms from one group to another. As we don't yet have a design for users to do this themselves, this commit adds a Rake task that a developer can run as a support task.
Things to consider when reviewing

Ensure that you consider the wider context.
Does it work when run on your machine?
Is it clear what the code is doing?
Do the commit messages explain why the changes were made?
Are there all the unit tests needed?
Has all relevant documentation been updated?

Any form can be moved to any group regardless of form status/group status or form organisation/group organisation/creator organisation.

Would it be useful at this point to have it print more information about the new group/old group?

Any form can be moved to any group regardless of form status/group status or form organisation/group organisation/creator organisation.

Would it be useful at this point to have it print more information about the new group/old group?

Yes, i think a quick summary of group (name, org, creator) might be useful - not vital to merging this though 👍

Any form can be moved to any group regardless of form status/group status or form organisation/group organisation/creator organisation.

Would it be useful at this point to have it print more information about the new group/old group?

Yes, i think a quick summary of group (name, org, creator) might be useful - not vital to merging this though 👍

I've added more logging in 8d36a14, let me know what you think.


VSCode: Navigate to nested function

TypeScript Version:  master
Code
function fnc1() {
    function fnc2() {}
}

In Visual Studio, I can use ctrl-, to navigate to either fnc1 or fnc2.
In VSCode, I can use ctrl-p to navigate to fnc1, but not to fnc2.
The error also happens for functions nested in namespaces.
In both cases, I have the editors set up to point to my latest build of the master branch.

This doesn't repro for me in TypeScript or in JavaScript. What version of VSCode is this?

I am on version
Version 0.10.11
Commit f291f4ad600767626b24a4b15816b04bee9a3049
Date 2016-03-11T16:19:03.130Z
Shell 0.35.6
Renderer 45.0.2454.85
Node 4.1.1
which seems ancient, so maybe it's a VSCode regression.

I updated this morning to 1.3.0.

@andy-ms are you using an up-to-date language service?

Now that I've updated to 1.3.0, I can repro this bug.

I set the "typescript.tsdk" settings to point to my checkout of master, so it should be completely up-to-date.

Additionally, in 0.10.11, if I had a folder with files first.ts and second.js, then f1 – f4 were all found in the code below. In 1.3.0, only the outer symbol from the file with the caret is found. That is, f3 is not found when first.ts has the caret, and f1 is not found when test.js has the caret.
// first.ts
function f1() {
  function f2() {
  }
}

// second.js
function f3() {
  function f4() {
  }
}

I am also running a checkout of master. I don't think anything else changed in my testing besides upgrading VSCode from 0.10.11 → 1.3.0.

Closed along with Microsoft/vscode#9217


Multivariate Hawkes Process

I am interested in mutually exciting processes and I was wondering if you have a package for multivariate hawkes process

I am also interested in the process of mutual stimulation. Have you found the appropriate Hawkes process package?


Experiment should have qubits property

What is the expected behavior?
I find it terribly confusing that BaseExperiment takes a qubits argument in its constructor, but it gets renamed to physical_qubits as a property. I think the names should be kept consistent; at the very least, there should be a qubits property that returns the qubits that were passed to the constructor.

Closing due to lack of interest.


Search for Portfolio entries is bugged

Something I noticed on my website and I just noticed your base website is affected too. Not quite sure how to fix it, so I'm posting an issue instead.
Basically, Search functionality is semi-broken for Portfolio (and possibly any other containers which are not a _post container) containers. You cannot search for them by site content and searching by site title shows an incorrect excerpt.
Take this Portfolio entry for example: https://sylhare.github.io/Type-on-Strap/portfolio/ninja
Navigate to Search and try to find parts of the first sentence from this page:

Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium

Result:

Search for title ("Ninja") instead and you get a result, albeit glitched:

It doesn't seem to be affected by the layout Portfolio entry has - your entries use post layout and my entries use page layout - yet result is the same.

Indeed might be something with the search.json that does not get compiled correctly. I think it would require an update of the plugin, or maybe work around with the options.

It was just a typo in search.json, should be all good now.

Works perfectly, thanks for a timely fix!
Just update your gh-pages so results are visible straight away :)


[SE-0206] Incorporate acceptance notes; set status to Implemented

Remove Hasher.init(seed:)
Remove most Hasher.combine overloads
Change Hasher.finalize to be __consuming
Clarify that Hasher is order-sensitive
Clarify hashValue's deprecated status (only custom implementations are deprecated)
Clarify that SE-0206 is not going to be source-breaking; hashValue impls will compile with a warning in 4.2 mode.
Update links to implementation
Move to Implemented status

@jckarter This incorporates the changes in the core team's acceptance notes, and also clarifies that while hashValue is deprecated as a protocol requirement, this proposal is not source breaking.
The old Source Compatibility section implied that users will need to go through a forced migration step, which is not the case. I replaced it with text that describes the actual intended behavior.
apple/swift#16009 and apple/swift#16073 implemented the proposal. (Except for the deprecation warning when hash(into:) needs to be derived from a manual hashValue.)


MediaQuery.of(context).orientation returning incorrect value

I have an application where most pages are restricted to portrait view, but two pages support all orientations.  The main application is not restricted - I am restricting the orientations on each page.  The portrait pages are displaying correctly, but pages with landscape support are doing the following:

Begin in portrait
Rotate the phone to landscape (either direction)
The app flips to landscape (either direction), but immediately flips back to portrait
Rotate the phone back to portrait
Rotate the phone to landscape (either direction) and it doesn't even flip initially - it simply stays in portrait

I have the MediaQuery orientation printing on build for troubleshooting and here's what happens:
Step 1) Prints Orientation.portrait
Step 2) Prints the following:
Orientation.portrait
Orientation.landscape
Orientation.landscape
Orientation.portrait
All further rotations) Nothing is printed in the console
Note: This same code was working properly in past testing.  I am not certain at which point in time it ceased working properly.
Code Sample
import 'package:flutter/services.dart';
import 'package:purchys_client/models/property.dart';
import 'package:purchys_client/widgets/header.dart';
import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';
import 'package:photo_view/photo_view.dart';
import 'package:photo_view/photo_view_gallery.dart';
class PropertyGallery extends StatefulWidget {
final Property property;
final int index;
const PropertyGallery({Key key, this.property, this.index}) : super(key: key);
@override
_PropertyGalleryState createState() => _PropertyGalleryState(property: this.property, index: this.index);
}
class _PropertyGalleryState extends State {
final Property property;
int index;
_PropertyGalleryState({this.property, this.index});
@override
void initState() {
super.initState();
if (index == null) {
  index = 0;
}

SystemChrome.setPreferredOrientations([
  DeviceOrientation.landscapeRight,
  DeviceOrientation.landscapeLeft,
  DeviceOrientation.portraitDown,
  DeviceOrientation.portraitUp,
]);

}
@override
dispose(){
SystemChrome.setPreferredOrientations([
DeviceOrientation.landscapeRight,
DeviceOrientation.landscapeLeft,
DeviceOrientation.portraitUp,
DeviceOrientation.portraitDown,
]);
super.dispose();

}
@override
Widget build(BuildContext context) {
PageController _pageController = PageController(initialPage: index);
print(MediaQuery.of(context).orientation.toString());

return new Scaffold(
  appBar:
    header(context, page: property.address, removeBackButton: false),
  body: Container(
    color: Theme.of(context).canvasColor,
    child: PhotoViewGallery.builder(
      pageController: _pageController,
      enableRotation: true,
      transitionOnUserGestures: true,
      backgroundDecoration: BoxDecoration(
        color: Theme.of(context).canvasColor,
      ),
      scrollPhysics: const BouncingScrollPhysics(),
      builder: (BuildContext context, int index) {
        return PhotoViewGalleryPageOptions(
          imageProvider: NetworkImage(property.images[index].mediaUrl),
          initialScale: PhotoViewComputedScale.contained,
          minScale: PhotoViewComputedScale.contained,
          maxScale: PhotoViewComputedScale.covered * 1.1,
        );
      },
      itemCount: property.images.length,
    )
  )
);

}
}

@kmcdonald-purchys
Is this still a valid issue using latest Flutter stable version ?
If so, can you provide updated flutter doctor -v and complete minimal reproducible code sample ?

Without additional information, we are unfortunately not sure how to resolve this issue.
We are therefore reluctantly going to close this bug for now.
Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact description of what happens, logs, and the output of flutter doctor -v.
All system setups can be slightly different, so it's always better to open new issues and reference related issues.


Widget methods should care for sanitizing the input

Affected Symphony version(s) : 2.7.x (current)
PHP version(s) : all
OS(es) : all
In the Symphony core, the Widget class provides methods to build HTML output. Sanitizing of values is always done when Widget functions are called.
Wouldn't it be better to perform all sanitizing in the Widget methods?
This has initially been discussed in #2769.

This will break lots of thing. We would need to add a way to be able to add text in between child nodes.

If this occurs often (and can not be avoided), yes, it should be added. But at the end of the day having a strict API for the Widget functions would ease maintenance and would prevent issues.

Agreed. Can't wait to implement this.

Well, yes. Sounds like a lot of work.

Yes, we change from opt-in to opt-out. And would hurt perf.

Why would this hurt performance?

First because we would need to render things from the bottom up (instead of top-down like we do) and because we will need to call html entities a lot. Should not be a lot, but still.

I am afraid that my knowledge of the inner workings is a bit porous… Can you explain what you mean by bottom-up and top-down rendering? (I thought that most could stay as it is, and one could just move the sanitizing to a different method that can be called with a new parameter for this behaviour.)

Can you explain what you mean by bottom-up and top-down rendering?

When you call ->generate() it will call all of its children ->generate() functions, which will do the same on their children. The generation then becomes the element's own value (as a string). This removes the possibility to simply sanitize its value. This is top down (we start with the top-most element).
The inverse would be to sanitize all text values before rendering tags.


johndanz/ch9455/bad-position-shares-math

Clubhouse Story
Submitter checklist

[x] Test covering functionality added/fixed in
Check the linked story includes the following:

[x] Steps to reproduce
[ ] Screenshot (If applicable)

Reviewer Checklist

[ ] Test/lint pass

@JohnDanz I left a comment about the position adjustments in this PR and why I don't think they're necessary (or correct).  Another meta-comment is that the position adjustment stuff seems like it would belong in a different PR, as it appears to be out of the scope of ch9455.  Or, am I misunderstanding?

https://app.clubhouse.io/augur/story/9455/bad-position-shares-math
added some screenshots of the 9.99... display issue. John pointed it out, but there is a difference how the Order Book shows the 10 share order and how the Outcome List shows the Order.
I think we need to be consistent.

@tinybike I actually figured it went here since it wasn't updating correctly so it was "bad shares math" But i agree with your meta comment that it probably should be cleaned up elsewhere then in the selector so I'm going to pull out the changes I made to try and account for orders and create a new task to investigate why it appears to not be updating correctly.
So this next commit should have just the changes to use BigNumber Math in the accumulator.
@bthaile you're comment is OOS for the same reason as above, we should make a seperate issue to investigate that, I'll create a clubhouse issue for that.


Added List of Creditors report

This PR adds support for querying and retrieving the list of creditors' data.
The reports require only the following parameters: court, docket_number, and pacer_case_id.
It returns an HTML file where the creditor's raw data is embedded within a form. We parse that parameter and retrieve the creditor's raw data, which we can use from here.
In PACER, there is an additional step. If you click the "View Labels" button, it sends the data to a service that doesn't require authentication and formats the raw data.
So I added this additional step as method download_file(), in case we need to use the formatted data.
I'm attaching an example of the raw data file and the formatted one so you can see the difference:
cacb-1181603-raw.txt
cacb-1181603-formatted.txt
The data is the same, but in the formatted version, it adds the docket_number to each line, capitalizes some words, and in some lines, it adds an undeliverable or preferred field at the end.
If these differences are not significant, we could use the raw version and convert it to CSV, adding the docket_number to each line so that when we create the global CSV for a company, each line has the docket_number. Otherwise, we could use the formatted version, but it adds an additional point of failure since it needs an additional request. I noticed that sometimes the service might be unavailable.
Let me know what you think.

Thanks, this looks really good, but we do need to figure out what's up with the different costs.


tslint 0.0.4

Cleaned and fixed comments from #923
New Pull Request Checklist
Please mark the points which you did / accept.

[x] I read the Step Development Guideline
[x] I have a test for my Step, which can be run with bitrise run test (in the step's repository)
[x] I did run bitrise run audit-this-step (in the step's repository - note, if you don't have this workflow in your bitrise.yml, you can copy it from the step template.)
[x] I read and accept the Abandoned Step policy

Thank you @ibratoev ! :)


Fix the wrong style of icon in Mozilla Firefox, use margin instead of transform to set the position

#8 #9
In 6e5cafa, I used margin to replace translateY, but I think use flex would be better. However, some animate does not work well after that, so I reverted the change and wrote this to remind you to considerate about using flex to set the position of Toasts.
Tips: merge feat1 (#10) first.

Hey there! This is an interesting approach I love it! I have some suggested changes that I will include in the next comment

Also thanks for taking a look and making the suggested changes. This will make solid toast so much better 💯

Maybe I should make 1 pull request instead of 2, now we will have some merge conflicts. 😂

Thanks again for these changes. I have published v0.2.0 with all your changes! Let me know if something doesn't work for you 🕺

It works very nice now, nice work. Thanks for reviewing my code. 👍


Replacing state

Hi,
Is it possible to replace the state of a Store? When I look at the source code it looks like we can replace the reducers but not the sate? Is that correct?
It would be nice to replace the whole state for debugging purpose. Use case; when a user encounter an error he/she can submit the whole state which I then can load in my develop environment to reproduce the error.

You can pass the intial state as the second param of the provideStore function.

@fxck, sure but then you have to "reboot" the application, right? I want to replace the state on the fly. Like:
const myNewState = {
  ...
};
store.replaceState(myNewState)

If it's for debugging purposes, why does it matter?
If you want to do it on the fly, you can use redux dev tools, which I think support pasting in an object, or create your own meta reducer, similar to how this works.


ContentGPT

Hi, thanks for sharing the code!
I've been inspecting through your code and came up with few questions.
Do you expect your ContentGPT to reconstruct the input (i.e. biography, item content, or review) when the item_id token is given?
Is it normal that the loss for ContentGPT does not fall in the very first 10 epochs of training?
How are you sure that the natural language semantics and collaborative semantics all mingled well in item_id tokens?
Hope to hear from you soon! Thanks.

Hi Minseok,
Thanks for reaching out! I hope all is well with you.
Below are an itemized response to your questions :)

The purpose of the ContentGPT.

Basically, for the pretraining of the ContentGPT, you can refer to the example "(b) User/Item Textual Features" in the grey box of Section 3.2.2, Soft+Hard Prompting, for an intuitive understanding.
In a nutshell, for textual features associated with a user, an item, or a user/item pair, we design a prompt (composed of heterogeneous tokens of user/item/vocab) that describes the nature of the features (e.g., whether it is a user profile, an item description, or a review), and use it to predict the features themselves (composed of homogenous tokens), such that the features can be encoded into the user/item content token embeddings.

The loss of the ContentGPT.

Could you provide more details about the dataset used for experiments and the value of lambda_V? Thanks.

The mingle of collaborative and content token embeddings.

Actually, we use a mutual regularization strategy to constrain the user/item collaborative embeddings with user/item content embeddings, where collaborative embeddings can help the ContentGPT to find information relevant for recommendations in the user/item content features, whereas content embeddings can introduce side information to support CollaborativeGPT and RecGPT. This has its theoretical root in MaP estimation given the generative process defined in Sections 3.2 and 3.3. I guess this mutual-regularization process is the "mingling process" mentioned in your question.
In our optimization, there's actually a hyperparameter, lambda_V, that controls the strength of mutual regularization, where its value is selected by grid search. The sensitivity analysis can be found in Figure 4.

Hi, thanks for your prompt reply.
For the ContentGPT, due to the limited resource, I've partitioned one of your dataset, Amazon Beauty (~10%), and set the batch size as 4. For the initial training of ContentGPT, I reckon 'regularize=False'; the loss for reconstructing the hard prompt (e.g. review) does not decrease.
As you've mentioned, "such that the features can be encoded into the user/item content token embeddings.", I believe that after initially training the ContentGPT (w/o regularizing with CollaborativeGPT) should be able to reconstruct the hard prompt (e.g. review) it learned when provided with soft+hard prompt. However, I couldn't observe it (for my case, since the loss didn't fall at all, this phenomenon is evident). Were you able to reconstruct something semantically similar?
Thanks!

Hi, thanks for your prompt reply.
For the ContentGPT, due to the limited resource, I've partitioned one of your dataset, Amazon Beauty (~10%), and set the batch size as 4. For the initial training of ContentGPT, I reckon 'regularize=False'; the loss for reconstructing the hard prompt (e.g. review) does not decrease.
As you've mentioned, "such that the features can be encoded into the user/item content token embeddings.", I believe that after initially training the ContentGPT (w/o regularizing with CollaborativeGPT) should be able to reconstruct the hard prompt (e.g. review) it learned when provided with soft+hard prompt. However, I couldn't observe it (for my case, since the loss didn't fall at all, this phenomenon is evident). Were you able to reconstruct something semantically similar?
Thanks!

Hi Minseok,
Thanks for providing the detailed information. I didn't save the training details when I ran the experiments, as the codes automatically select the best model based on the validation set. I'm now rerunning the full-scale experiments to retrieve the training log, which may take some time :)
Here, for the ContentGPT, we only want to encode content information into the user/item token embeddings via language modeling. To generate a full review, i.e., "to reconstruct something semantically similar" mentioned in your question, based on only user/item content embeddings is a more difficult task. Nevertheless, for the loss issue, I'll get back to you as long as I have the results :)
For now, the effectiveness of the ContentGPT can be verified by Comparison with LLM-CF in Table 1&2, where we have removed the ContentGPT and mutual-regularization and use only CollaborativeGPT and RecGPT for recommendations.
Thank you for your patience.
Best,
Yaochen

Hi Minseok,
I've rerun the experiments on the Company server and I found that the training loss drop monotonically for the ContentGPT in the first ten epochs. Below is the screen shot of the training log.

I wonder whether your situation is because of the smaller batch size.
Best,
Yaochen

Hi, Yaochen.
Thanks for sharing the training log. The loss drop tendency is almost the same when I ran your code with my dataset. Guess the batch size or other settings may have led to the difference with your benchmark dataset. I do really appreciate your help and will get back to you if I encounter new questions! Wish you a good luck on your future research as well.
Best, Minseok

Hi Minseok,
I'm glad that the codes work on your own dataset.
Thanks for the kind words, and good luck with your research too!!
I'll mark this issue as closed. If you encounter new questions, feel free to reopen it or start a new issue :)
Best,
Yaochen


Add about us hero

Pull Request
Related issue
Fixes #477
What does this PR do?

Creation of the About us page
Addition of the AboutUsHero block with the first block of the page
Addition of the corresponding story for Storybook
Update of the Header and Footer accordingly

How to test
Go to the deploy of this branch
You can also yarn storybook and check the new AboutUsHero block
PR checklist
Please check if your PR fulfills the following requirements:

[x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
[x] Have you read the contributing guidelines?
[x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!

bors merge


Can we register as a Payor/Provider ourselves or do we need to contact the HCX team?

As of now, Integrators have to contact the HCX team to get onboarded. HCX team will create the participant record in the registry, upon which an email will be received by the participant to change the password of Keycloak for token generation and a participant code will be shared for the team to use.


CAMEL-10953 Add message attribute support to AWS SNS publisher

I copied the message attribute capability from the SQS Producer to SNS Producer. I didn't want to refactor the code too much to avoid the duplicate code.

Thanks for the PR it has been merged. Do you mind closing this?

Nice (y) This was my first contribution. What may I expect in terms of getting this code in the standard camel libraries (and thus removing my own customized library from my project)?


Dashboard / widgets / OpenVPN - typo

Hi!
https://github.com/opnsense/core/pull/6438 backport

I don’t mind prepping backports, but I would ask to use “git cherry-pick -x ” to get a reference to the original commit.
the core framework can do this too as “make mfc-” which is what I use for trivial backporting.

shoot. github desktop does not use -x )
replaced with https://github.com/opnsense/core/pull/6440


Stacktrace test

the previous regex created issues for custom adapters because the integration tests are inside the deps directory.

@josevalim I noticed one other thing while trying to help debug the sqlite adapter's integration test failures on the latest Ecto.
Their adapter has to dump booleans to 0/1 which is ok if they belong to a typed field but fails to load properly when for instance they are a value in a map. This made me curious if they would work in embeds, but I noticed that the map values aren't typed even there.
I wanted to investigate keeping the type for embed values but before I wasted a lot of time I wanted to see if you knew a reason not to do this. Thanks!

From a first glance it makes sense that we should try to keep the type as much as possible. :)


Problem with LazyColumn and clickable modifier

Hi
There is a problem with LazyColumn and clickable modifier, if I add clickable to item of LazyColumn then KLazyListItemNode can't find child.
For example I modified your sample and tried to run
@Composable
private fun ListItemHeader(item: LazyListItem.Header, modifier: Modifier = Modifier) {
    Box(
        modifier = Modifier
            .padding(horizontal = 16.dp, vertical = 8.dp)
            .clickable {  }
            .clip(RoundedCornerShape(8.dp))
            .background(Color.Green)
            .then(modifier)
    ) {
        Text(
            item.title,
            Modifier
                .padding(8.dp)
                .testTag("LazyListHeaderTitle")
        )
    }
}

I got error
java.lang.AssertionError: Failed to assert the following: (Text + EditableText = [Items from 1 to 10])
Can't retrieve node at index '0' of '(hasAnyAncestorThat(Semantics node id = 18)) && (TestTag = 'LazyListHeaderTitle')'
There are no existing nodes for that selector.

Hi!  Check that your nodes use the flag useUnmergedTree = true
This issue contains examples of how it can be configured

Yeah, it's working. Can you please explain why it's now working without this flag?

You can read more about it here


use syscall.Stdin for for read password

Use syscall.Stdin for terminal read password instead of 0 (which is not stdin on windows...)

close #57


rarity - distinguish between evo and base rarity with prefixes

[5:46 PM] River: ok
[5:46 PM] River: heres the deal
[5:46 PM] River: we need to do
[5:46 PM] River: 6*b
[5:46 PM] River: and 6*e
[5:46 PM] River: 6*b means that a card is 6* base
[5:46 PM] River: and 6*e means that the evo is 6*
[5:46 PM] River: 6* should do a disambig if both exist
[5:46 PM] River: until disambig exists, have it mean 6* base
[5:47 PM] River: also, to be mobile-friendly, add 6s as well as 6*
[5:47 PM] River: s for star

jk the prefixes should be 6* base and 6* evo but maybe let 6*b and 6*e also work (and same with s replacing *)

jk the prefixes should be 6* base and 6* evo but maybe let 6*b and 6*e also work (and same with s replacing *)


I have a question in the learning process.

First of all, thank you for the good research. I was struggling with ImageBind being commercially unavailable and found a good alternative.
I have a couple of questions.

Am I correct in understanding that the concept of the paper is to patch all the data, thus incorporating the idea of ViT?

Am I correct in my understanding of the learning pipeline?
For example, I'm assuming I'm using image classification and audio classification.

Method A

1) Section 3.3 Pretraining
    Train RAION-2B with ViT. (Training with clips?)
2) Section 3.3 Modality-Agnostic Learning
    ViT learns Freeze, Data2Seq Image, and Data Embedding.
3) Section 3.4 Task-specific Heads
    In the classic way, we freeze the Data2Seq part and train the Head and ViT sections respectively.
4) Done!
    You have one image classifier head and one audio  classification head!

Method B

1) Section 3.3 Pretraining
    Attach Data2Seq Image and Data Embedding, and train enough image and audio  data through Patch Masking self-learning of ViT.
2) Section 3.4 Task-specific Heads
    In the classic way, we freeze the Data2Seq part and train the Head and ViT sections respectively.
3) Done!
    We have one image classifier head and one audio  classification head!

When I think of something classical, method B looks familiar to me.
As for method A, you only showed an example for Image in the paper, right?
My question is, if I put audio patch data, etc. into a model trained with RAION, won't it act as an outlier because there is a chance that the model has never been trained on it?
Also, since the Transformer Encoder has limited parameters, I have a hypothesis that if I don't train Data2Seq in the pretraining interval in the form of Method B, the parameters of the Encoder will not converge properly on the data for image and audio classification. (In a very small probability, there may be cases where the patch data of image and audio data are almost similar? Or maybe we're going to train Head separately anyway, so it's okay?)

If I'm completely misunderstanding, could you please provide a pseudocode for the learning?

Looking at the issue, the similarity for each modality seems to be pretty good, but if the example in question 2 is a model that classifies dog species by image and dog species by sound, will the similarity of the two Encoder Outputs be similar?

This is a really interesting paper and seems to go beyond my existing imagination and common sense. Thanks again for a great contribution.

You can find the answer in our new project OneLLM.


blueman-applet: exception on startup

❯ systemctl --user start blueman-applet

I have not yet found any functionality that actually does not work after clicking OK though.

I'm not using Bluetooth but this seems to me like an issue that has to be resolved outside Home Manager. Basically, from what I can see the org.blueman.Mechanism D-Bus service is provided by the blueman package in Nixpkgs:
rycee@beta:~/devel/nixpkgs$ nix-build -A blueman
/nix/store/8abl7bvx83g7ql4ri8qzkyjs3fyp3nss-blueman-2.0.4
rycee@beta:~/devel/nixpkgs$ cat result/share/dbus-1/system-service/org.blueman.Mechanism.service
[D-BUS Service]
Name=org.blueman.Mechanism
Exec=/nix/store/8abl7bvx83g7ql4ri8qzkyjs3fyp3nss-blueman-2.0.4/libexec/blueman-mechanism
User=root
rycee@beta:~/devel/nixpkgs$

The issue seems to be that this is a system service so it can't set it up for a single user. Ideally, I guess there should be a NixOS module for blueman (e.g., services.blueman.enable = true) but you could try installing blueman using environment.systemPackages in your system configuration. If it is installed system wide I suspect that the system dbus daemon will be able to find it…

@rycee is right, the problem seems to be connected that dbus can't find the blueman package.
To make it work I've added blueman to services.dbus.packages in my system configuration file:
services.dbus.packages = with pkgs; [ blueman ];

That solved the issue and I learned something about D-Bus. Thanks!

Do you need both environment.systemPackages and services.dbus.packages at the same time specifying blueman?

@CMCDragonkai it should be enough with only services.dbus.packages = [ pkgs.blueman ]; set.


react: Return Canceler from AddCallback

This makes more sense than returning a CallbackHandle that has to then
get passed into RemoveCallback. It avoids (as in some solutions but not
all) having to cast the CallbackHandle back and forth.
The author of this commit believes this to be more idiomatic Go.

@petertseng I agree it's more idiomatic.

Notes:
One might say it is in violation of https://github.com/golang/go/wiki/CodeReviewComments#interfaces because we put the interface in the package producing one. Though the tests need to consume as well, and it's definitely true the interface is there for the tests.
I would be very interested if someone can find examples in the standard library that either support or refute this thought of returning Canceler, but otherwise since I have approval I'll merge it soon.

This is not part of the standard library, but see how https://godoc.org/gopkg.in/redis.v5#Client.Subscribe returns a *PubSub, on which you might call https://godoc.org/gopkg.in/redis.v5#PubSub.Unsubscribe , so this matches what we do here.

Standard library: https://golang.org/pkg/database/sql/#Open returns *DB, on which you might call https://golang.org/pkg/database/sql/#DB.Close

Another example is https://golang.org/pkg/net/#Dialer.Dial which returns net.Conn according to given network backed by different structs, on that Cancel might be called.


The combination of cage and reduce doesn't work correctly

[] > test
  cage (* (cage 1)) > arr
  reduce. > b
    list
      arr
    arr
    [a x]
      a.with (cage 2) > @
  seq > @
    (b.at 0).write 28
    (b.at 1).write 42
    stdout
      sprintf
        "%d %d\n"
        (b.at 0)
        (b.at 1)
    TRUE

[] > test-2
  cage (* (cage 1)) > arr
  arr.with (cage 2) > b
  seq > @
    (b.at 0).write 28
    (b.at 1).write 42
    stdout
      sprintf
        "%d %d\n"
        (b.at 0)
        (b.at 1)
    TRUE

Note that the code of these two tests differs only in the place where the second element is added. In the first test, we add it through the reduce method. And in the second test, we simply add it via the with method.
In both examples, the code should do the same thing, but the first test outputs "28 2" and the second "28 42".

@EugeneDar moved

@yegor256 should I close this issue, because we don't plan to use the 'cage' in the future?


User Management via SCIM

It would be a nice feature to add user management to the CLI. For example, add / edit / delete a user. Also, view / delete FIDO devices for a person.

@nynymike please create such issues on jans-config-api. cli just uses swagger.yaml so if it has no such feature, cli has nothing to do

@nynymike please create such issues on jans-config-api. cli just uses swagger.yaml so if it has no such feature, cli has nothing to do

I don't see it as a hard and fast rule that the config-api can't use the SCIM API also. BTW, SCIM has it's own swagger yaml file. Alternately, we could perhaps use a different cli command. But I think it should also go in this repo.

I don't see it as a hard and fast rule that the config-api can't use the SCIM API also. BTW, SCIM has it's own swagger yaml file. Alternately, we could perhaps use a different cli command. But I think it should also go in this repo.

jans-cli works with scim swagger ui with small changes (both cli and yam file)

jans-cli works with scim swagger ui with small changes (both cli and yam file)


name 'lstm_auto_encoder' is not defined

Has problem when run the example code
could not identify 'lstm_auto_encoder' parameter

Thanks @tabatabaei,
this is a little documentation misspecification. Currently pipline.add_feature() takes either DeepAutoEncoder() or TsFresh() object  (or both).

Hi, @tabatabaei. Thanks, actually this is a little documentation Misspecification!
Maybe, @amirabbasasadi can fix it.
pipeline.add_feature() gets DeepAutoEncoder object as input.
you can easily implement lstm auto-encoder with the following script, also I would try to add extra example that covers more feature extraction methods:
  ##########
  # dataset
  ##########

  LEN_TS = 39 # time series length
  FH = 7      # forecasting horizon

  # create synthetic data
  from metats.datasets import ETSDataset
  ets_generator = ETSDataset({'A,N,N': 512,
                              'M,M,M': 512}, length=LEN_TS, freq=4)
  data, labels = ets_generator.load(return_family=True)
  colors = list(map(lambda x: (x=='A,N,N')*1, labels))

  # scaling data
  from sklearn.preprocessing import StandardScaler
  scaled_data = StandardScaler().fit_transform(data.T)
  data = scaled_data.T[:, :, None]    # size: batch_dim x seires_length x series_dim
  print('data shape is:',data.shape)

  # creats MetaLearning pipeline
  from metats.pipeline import MetaLearning
  pipeline = MetaLearning(method='selection', loss='mse')

  ##########
  # LSTM
  ##########

  from metats.features.unsupervised import DeepAutoEncoder
  from metats.features.deep import AutoEncoder, LSTMDecoder, LSTMEncoder

  H = 5   # hidden size
  l = 3   # latent size
  NL= 2   # number of layers

  enc = LSTMEncoder(input_size=data.shape[2], latent_size=l,
                  hidden_size=H, num_layers=NL, directions=1)
  dec = LSTMDecoder(output_length=LEN_TS-FH, output_size=data.shape[2],
                  latent_size=l, hidden_size=H, num_layers=NL, directions=1)
  ae = AutoEncoder(encoder=enc, decoder=dec)
  ae_features = DeepAutoEncoder(auto_encoder=ae, epochs=100, verbose=False)
  pipeline.add_feature(ae_features)

  ##########
  # Forecasters
  ##########

  from sktime.forecasting.naive import NaiveForecaster
  from sktime.forecasting.compose import make_reduction
  from sklearn.neighbors import KNeighborsRegressor

  # creating two base-forecasters
  regressor = KNeighborsRegressor(n_neighbors=1)
  forecaster1 = make_reduction(regressor, window_length=15, strategy="recursive")
  forecaster2 = NaiveForecaster()

  # adding base-forecasters to pipline
  pipeline.add_forecaster(forecaster1)
  pipeline.add_forecaster(forecaster2)

  ##########
  # MetaLearner
  ##########

  from sklearn.ensemble import RandomForestClassifier
  # to adopt random foreset as meta-learner:
  # meta-learner makes connection between features and
  # each base-forecaster's weights
  pipeline.add_metalearner(RandomForestClassifier())
  pipeline.fit(data, fh=FH)
  predict = pipeline.predict(data, fh=FH)
  print(predict.shape)


Lonewolf: Pairing generation

The pairinggen functions need to be able to handle non-team seasons as well. The implementation can use the same JavafoInstance backend for the pairing algorithm.

Implemented.


Implement readDirs method

Implementing feature to read directories from provided directory path based on Idea here:
https://github.com/helmfile/helmfile/discussions/253
Within the README.md I have also documented the readDir function which was not documented but available for usage.

@vasicvuk Please add unittest and e2etest for this.

@vasicvuk Thanks for your idea. I think readDirs is ambiguous. Do you have a higher function name

@vasicvuk e2e test in test/e2e/template/helmfile/tmpl_test.go

Hi @yxxhero,
Maybe listDir or readDirsWithinDir?
Do you have some name you would like?

@vasicvuk  listDir will better.  Can exec meet your needs?

As I understood exec is plarform specific and we are using helmfile on both Windows and Linux so it would be hard do handle both cases

@mumoshu WDYT?

@yxxhero Maybe listDir can list both files and folders instead just folders, i think that would be better

Can you describe your application scenario in detail

Sure,
We have multiple Cloud native apps built with 30-50 microservices, each application uses some of the common services we call platform and some services specific to their product.
All of the microservices have their own charts and since we standardize the helm chart with internal generator tool we have a file named global which values from are applied for every microservice.
Other then that we have a folder for each team (owner of multiple microservice), and also an override which we use to optimize the helmfile for certain client were we implement product to.
.
├── ...
├── base # Default configuration for the product
│   ├── global #  Global configuration
│   │   ├── values.yaml # Global values
│   │   └──
│   ├── platform # Platform microservices
│   │   ├── service1.yaml # Platform microservice 1
│   │   ├── service2.yaml # Platform microservice 2
│   │   └──
│   ├── app  # App microservices
│   │   ├── app1.yaml # App microservice 1
│   │   ├── app2.yaml # App microservice 2
│   │   └──
│   └──
├── override # Override configuration for the specific client
│   ├── global #  Global override configuration
│   │   ├── values.yaml # Global values
│   │   └──
│   ├── platform # Platform override microservices
│   │   ├── service1.yaml # Platform microservice 1
│   │   └──
│   ├── app  # App override microservices
│   │   ├── app1.yaml # App microservice 1
│   │   └──
│   └──
└── ...

In override, the user can override only certain services, global values, or everything (it's up to the implementation team). Other than that we allow the option for an override file to begin with _ and then the microservice will not be deployed at all even if it is defined in base.
We implemented everything from the description above, but currently, we need to have a list of folders in YAML so we know what folders are there (app, platform, third-party) and we want to automate also this by having an option to list folders from code too.

@vasicvuk ok. Thanks very much. Perfect this PR according to your idea. I feel it is really useful.

@yxxhero, Thanks, I have changed the function to listDir and added unit tests for both readDir and listDir.
Since I am not really a GO developer, I need a little more time to understand how e2e part works, I am not sure if someone can jump in and help with e2e part?
Thanks

@yxxhero  I managed to add e2e tests also :)

@mumoshu ping

https://github.com/helmfile/helmfile/blob/4e9b99d10e07c072287f108569c5d27f018b28e7/pkg/argparser/args_test.go#L8

@yxxhero  Should I change whole file then, as everything is using deepEqual currently?

@vasicvuk whole file.

@yxxhero  Done

@vasicvuk please fix go test error

@yxxhero  Can you help with identifying what the problem is, since i didn't change snapshot_test.go and it is showing this test as failure:
snapshot_test.go:126: helm push /home/runner/work/helmfile/helmfile/test/e2e/template/helmfile/raw-0.1.0.tgz oci://localhost:5000/myrepo: Error: failed to do request: Head "http://localhost:5000/v2/myrepo/raw/blobs/sha256:82d11697f6b3c7c5bcf619fb237648b2a8daee142dd0972f7f6244e72ea921e7": EOF
    snapshot_test.go:126: Unable to run helm push /home/runner/work/helmfile/helmfile/test/e2e/template/helmfile/raw-0.1.0.tgz oci://localhost:5000/myrepo: exit status 1
--- FAIL: TestHelmfileTemplateWithBuildCommand (10.77s)
    --- PASS: TestHelmfileTemplateWithBuildCommand/chart_need (6.33s)
    --- PASS: TestHelmfileTemplateWithBuildCommand/issue_2098_release_template_needs (0.07s)
    --- FAIL: TestHelmfileTemplateWithBuildCommand/oci_need (4.37s)

Thanks for your efforts!
Probably that's a flaky test added via #239. Let me rerun it and see if the error disappears.

@mumoshu Yes, now they are passing :)


Issue488: Fix multiple authors text and avatar alignment

closes #488
What this PR does
This aligns text and avatars by their middle.
How/Why
It was looking a bit messy, so I just added a vertical-alignment to people-badge
Image of the new behavior 😉

@bebatut could you please review this PR? ❤

Thanks @vasconsaurus for the fixed there

@bebatut there were some bumps along the way, but I'm really happy it worked out ok in the end 🙂