Add Azure Blob Storage as backend

Environment variables:

ACCOUNT_NAME=storage-account-name
ACCOUNT_KEY=storage-account-key

Environment variables for test:

RESTIC_TEST_AZURE_ACCOUNT_NAME=storage-account-name
RESTIC_TEST_AZURE_ACCOUNT_KEY=storage-account-key
RESTIC_TEST_AZURE_REPOSITORY=azure:restic-test-container

Init repository:

$ restic -r azure:container-name/prefix/dir init
or
$ restic -r azure://container-name/prefix/dir init

Container location is fixed for a azure storage account. So we cant specify it in repository.

Codecov Report

Merging #1059 into master will decrease coverage by 4.7%.
The diff coverage is 9.37%.

@@            Coverage Diff            @@
##           master   #1059      +/-   ##
=========================================
- Coverage   53.81%   49.1%   -4.71%
=========================================
  Files         117     119       +2
  Lines        8895    9151     +256
=========================================
- Hits         4787    4494     -293
- Misses       3407    3984     +577
+ Partials      701     673      -28

Impacted Files
Coverage Δ

src/restic/backend/location/location.go
77.77% <ø> (ø)
:arrow_up:

src/restic/backend/azure/azure.go
0% <0%> (ø)

src/cmds/restic/global.go
20.75% <0%> (-1.16%)
:arrow_down:

src/restic/backend/azure/config.go
85.71% <85.71%> (ø)

src/restic/backend/b2/b2.go
0% <0%> (-80.59%)
:arrow_down:

src/restic/backend/swift/swift.go
0% <0%> (-73.69%)
:arrow_down:

src/restic/backend/swift/config.go
36.95% <0%> (-54.35%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9583dc8...00d72bb. Read the comment docs.

@fd0 , you will need a Azure storage account to run these tests. If you would like us to provide the account, please share an email address. I can give you access to our test Azure test storage account that you will be able to use to run tests. You can find my email on my Github profile.

FYI, I don't have much time at the moment, will check this later. Thanks for the contribution!

Format:
$ restic -r azure:container-name/prefix/directory init

@diptadas , we should change the URL to azure:container:/[prefix] .  There should be a : after container name.

Format: azure:container:/[prefix]

Our pr for reading partial blobs from Azure storage is now merged into the official sdk repo. https://github.com/Azure/azure-sdk-for-go/pull/664

I'm backing up to minio running in gateway mode. gateway mode supports azure storage. see documentation here.
it would be interesting which method is faster and more stable:
a) restic -> minio -> azure
b) restic -> azure

@flo82 , if you get a chance to run benchmarks, please share with us. It is kind of hard to see how an extra hop will perform any better.

i will try, if i find time for that :-)
will it be possible to switch between a) and b) on the fly (because minio impl and azure impl use the same object/file structure) or do i need to init a new repository?

will it be possible to switch between a) and b) on the fly (because minio impl and azure impl use the same object/file structure) or do i need to init a new repository?

It really depends on what Minio gateway does. It minio gateway maintains 1:1 mapping, it should be possible.

@diptadas @tamalsaha Hey, I'll test this in the next couple of days, is it okay for you if I squash the commits and add the necessary code to merge this? Or do you want to do this yourself, and I'll just add comments? I think the former will be much faster, but I'd like your approval before I do this. In order to build/verify the integration tests this needs to be reopened from a branch of the restic repo anyway.

@fd0, both the prs are squashed. We are happy with you adding necessary code to merge this into master.  As @diptadas employer, I also speak for himself on this matter.

Awesome, then I'll do that. Cherry-picking your commits was the plan, I should've made that clearer. I've done the same for the Swift backend (see https://github.com/restic/restic/pull/975/commits).

Superseded by #1149


Retrieve all bookshelves of a user

Since the frontend team needs a way to display all bookshelves related to one user, we need to implement this feature, starting from the lowest layer (repository) up to highest layers (route, controller).
The bookshelves data returned must follow a specific format aka DTO, with validations rules imposed on each property of such a format.
@rraneemm work your magic!

It is all done!


Mojo::Reactor::Poll: I/O watcher failed 请问？

Mojo::Reactor::Poll: I/O watcher failed: Wide character in subroutine entry at /usr/local/share/perl/5.26.0/Mojo/IOLoop/Stream.pm line 83.
[18/02/07 22:22:52] [error] reactor error: I/O watcher failed: Wide character in subroutine entry at /usr/local/share/perl/5.26.0/Mojo/IOLoop/Stream.pm line 83.
[18/02/07 22:22:52] [error]  at /usr/local/share/perl/5.26.0/Mojo/EventEmitter.pm line 15.
[18/02/07 22:22:52] [error] 	Mojo::EventEmitter::emit(Mojo::Reactor::Poll=HASH(0x5642040dc048), "error", "I/O watcher failed: Wide character in subroutine entry at /us"...) called at /usr/local/share/perl/5.26.0/Mojo/Reactor/Poll.pm line 143
[18/02/07 22:22:52] [error] 	Mojo::Reactor::Poll::_try(Mojo::Reactor::Poll=HASH(0x5642040dc048), "I/O watcher", CODE(0x5642059da970), 0) called at /usr/local/share/perl/5.26.0/Mojo/Reactor/Poll.pm line 58
[18/02/07 22:22:52] [error] 	Mojo::Reactor::Poll::one_tick(Mojo::Reactor::Poll=HASH(0x5642040dc048)) called at /usr/local/share/perl/5.26.0/Mojo/Reactor/Poll.pm line 99
[18/02/07 22:22:52] [error] 	Mojo::Reactor::Poll::start(Mojo::Reactor::Poll=HASH(0x5642040dc048)) called at /usr/local/share/perl/5.26.0/Mojo/IOLoop.pm line 136
[18/02/07 22:22:52] [error] 	Mojo::IOLoop::start(Mojo::IOLoop=HASH(0x564204391ab8)) called at /usr/local/share/perl/5.26.0/Mojo/Webqq/Client.pm line 29
[18/02/07 22:22:52] [error] 	Mojo::Webqq::Client::run(Mojo::Webqq=HASH(0x5642035f23c0)) called at /home/wizardmeow/Documents/Code/shell/mojo::webqq/qq line 7

Perl完全不懂
请问这是什么情况
不过，似乎也没有影响正常使用（躺

嗯，似乎是老生常谈的UTF-8问题？

是的，是按照说明文档配置的。问题不是在自己加的代码上，我对比过了。
不过似乎完全没有影响正常使用。
感谢回复


Adjust size of extreme bins in discretize_and_truncate (2)

Summary
discretize_and_truncate currently has an odd behaviour on the first and last bin, that result to be smaller than the others.
Details and comments
discretize_and_truncate splits data into (2 ** num_qubits[j]) bins of equal size and equally spaced, where the first bin is centered in (bounds[j, 0] and the last in bounds[j, 1]).
In the previous implementation, data were cut so that only values between bounds[j, 0] and bounds[j, 1] survived. So, half of the first bin fell out of the bounds, and therefore the number of samples in the first bin was misleadingly low. Similarly for the last bin.
Code was modified as discussed in https://github.com/Qiskit/qiskit-machine-learning/pull/7

@Zoufalc can I ask you to take a look over this please

Modified the paramater name to min_max_bin_centers as agreed offline

@Zoufalc This method is used from QGAN where it seems bounds is parameter  of its constructor and this bounds ends up getting passed to this newly named parameter (which was bounds before this adjustment). Does QGAN need to change in anyway given this now seems to have a different interpretation of what it means?
Also, is this method principally for QGAN as in arguably, given this code had now been released, existing names/behavior ought to be deprecated and phased out rather than simply being changed. I guess if this is only used by QGAN and the impact is low we might be able to avoid that; on QGAN itself, if that needs changing, then thats less obvious. Maybe even there the bounds param, given it was optional, is not often used? Ideally we deprecate though.

@woodsp-ibm, @Zoufalc what's the status of this PR?

@woodsp-ibm Sorry for the delayed reply.
The new name is reflective of what is actually happening so the qGAN code doesn't need to be changed.
I see your point. I would also be fine with keeping the parameter name 'bounds' as is. @gabrieleagl what is your opinion here? Do you think that the renaming of the parameter is vital or would it be sufficient to include the shifting changes?

Hello  @Zoufalc, @woodsp-ibm, I personally prefer the new name, as it is more representative.
As far as deprecation is concerned, I believe it's not really necessary, because the method is only used for the qGAN, and arguments are typically called positionally.
Anyway, if you really think it's better to have the deprecation for consistency, I can do that, which means:

making min_max_bin_centers and num_qubits optional
adding an additional optional parameter bounds at the end (so that the order is respected, in case of positional calls)
checking that num_qubits is always values, and raise an exception otherwise
checking min_max_bin_centers or bounds is valued
adding a deprecation message if bounds is valued.
If we take this path, I would need your help on which exceptions to raise, and how to create the deprecation message.

Let me know!

Either way is fine with me. For me, the most important point was to fix the behavior of the binning.

As far as the deprecation is concerned; which since this is a public API we should in principle do, the simplest way is to keep the old method as-is and just add another new method as the 'replacement' with this new interface.  In the old method you would add a deprecation warning indicating the new method which should be used instead; of course wherever we call the old one we would want to change over to the new one - just in QGAN I guess. Here is an example of a deprecation warning in code https://github.com/Qiskit/qiskit-terra/blob/68b5ede97b6bf5454250927fb522be217f3f3b7e/qiskit/circuit/library/probability_distributions/uniform.py#L57-L60


Missing support for smart plug with smart metering

Hi,
I purchased two different smart plugs from same site:

NEO ZigBee On Off Smart Metering Plug" from this site:
https://hubitat-shop.eu/product/neo-zigbee-on-off-power-metering-plug/

RTX ZigBee On Off Smart Metering Plug
https://hubitat-shop.eu/product/rtx-zigbee-on-off-smart-metering-plug-2/

Is it possible to add support for the smart metering for Homey? When I add them the Tuya app won't recognice them and they are added as general devices without support of smart metering.
Manufacturer ID: _TZ3000_gjnozsaz

Thanks Peter, I'll close this thread. I read in the comments it will be supported in next release.


dont put hypens in variable names

fixes: https://github.com/MarcoMuellner/openapi-python-generator/issues/64
dont put hypens in variable names

Thanks for the contribution 🙂


Slider Syncing bug

When you setup two sliders (one to control the other) and set the sliders to not infinite, the controlled slider will move one place more than the controller slider at the ends when using the arrows.  So if you have 5 slides the controller will have slide 5 selected and the controlled slider will be showing slide 4 or slide 1.  This is with using the arrows on the controller.  if you manually click on slide 5 in the controller, the controlled slider will show slide 5 properly.
My slick slider JS settings are as follows:
$('.desktop-content').slick({
slidesToShow: 1,
slidesToScroll: 1,
arrows: false,
fade: true,
asNavFor: '.desktop-solutions'
});
$('.desktop-solutions').slick({
infinite: false,
slidesToShow: 4,
slidesToScroll: 1,
asNavFor: '.desktop-content',
centerMode: false,
focusOnSelect: true
});

have you tried my approarch to solve syncing issue? https://github.com/kenwheeler/slick/issues/1971#issuecomment-165313300

I have  actually the same problem that die nav-for ist not syncing with the closest stage slider.

any ideas how to solve and not to modify the slick.js

Hi @mpy-pixum in my case, i have solved it by adding afterChange function. maybe that can help you too https://github.com/kenwheeler/slick/issues/1971#issuecomment-165313300

https://codepen.io/athemix/pen/RwNVYmw
It happens on this demo...if you press next on the last slide, the other slide overshoots what slide it should be on.

https://codepen.io/athemix/pen/RwNVYmw
It happens on this demo...if you press next on the last slide, the other slide overshoots what slide it should be on.


Don't make recursive symlink

In the cxx build process it makes a recursive symlink to the crate root:
~ ls -l /Users/timh/workspace/libxyz/crates/core_logic/target/debug/build/core_logic-2813b45ab0a66cc3/out/cxxbridge/crate/core_logic/target/debug/build/core_logic-2813b45ab0a66cc3/out/cxxbridge/crate/
total 0
lrwxr-xr-x  1 timh  staff  69 24 Feb 13:24 core_logic -> /Users/timh/workspace/libxyz/crates/core_logic

It would be good if that could be avoided - lots of code just blindly walks trees following symlinks to infinity.
Also I assume there is some alternative that doesn't involve symlinks otherwise how does this crate work on Windows?
It would be nice if this symlink didn't exist.

Ditto. Just ran into this issue with a C# project where Visual Studio walks the tree forever and throws an exception on solution build. This effectively makes the build.rs approach untenable for me. I can get around by killing the symlink as a post build step, but eh...


Performance of Windows PowerShell is better then PowerShell Core

I get very strange and disturbing performance results for Import-Csv cmdlet.
Import 111 Mb csv log file:

On Windows PowerShell always ~ 3.7 sec. Memory ~ 10 Mb.
-On PowerShell Core from 6 sec up to 27 sec - each subsequent launch increases the time by 1-2-3 seconds. Memory grow from 1 Mb to 105 Mb. After waiting some minutes memory decreases to ~1 MB and agaim the cmdlet take ~6 sec.

I suppose we get the same for any script.

@iSazonov we have seen similar memory growth with string related activities in JSON too. I suspect .NET Core is just greedier wiith ita GC than Framework.
But the slow down.. that is concerning.

@daxian-dbw If you are working on performance tests this may be of interest to you.

  Do you have a repro I can run?  It's probably worth measuring any difference between GCSettings.IsServerGC true and false.
FWIW, GCSettings.IsServerGC seems to be false by default for both powershell.exe and pwsh.exe.  As I understand it, that default means that there is one GC thread regardless of how many threads are simultaneously producing garbage.  If I'm understanding this right, any managed code invoked by PowerShell that is both CPU-bound and well-parallelized is likely to run out of memory if you have enough cores and it runs long enough.
It could be that whatever Import-Csv (and the JSON activities @markekraus reported) are doing became more parallelized from Windows PowerShell to PowerShell Core.

@alx9r The repo is simple
Measure-Command { Import-Csv file.csv }

The file I used has ~110 Mb size.

It could be that whatever Import-Csv (and the JSON activities @markekraus reported) are doing became more parallelized from Windows PowerShell to PowerShell Core.

Then my test had to be even faster than in Windows PowerShell but it is 3-10(!) times slower from the first run.

@iSazonov I'm not seeing the dramatic speed difference you reported.  Perhaps the specifics of the .csv file matter.  Here is my test code:
$PSVersionTable

"GCSettings.IsServerGC : $([System.Runtime.GCSettings]::IsServerGC)"
$filePath = "$([System.IO.Path]::GetTempPath())5abb9571859c48cea807d755cc48058b.csv"

# uncomment the following to generate a ~130MB .csv file first
<#
$numLines = (50 * [System.Math]::Pow(2,12))

1..$numLines |
    % { ($_..($_ + 100) | % {"$_"}) -join ',' } |
    Set-Content $filePath
#>

"Import-Csv $filePath"
Measure-Command { Import-Csv $filePath } | select TotalSeconds | Format-List

These are the results for Windows PowerShell and PowerShell core, respectively:
Name                           Value
----                           -----
PSVersion                      5.1.14409.1012
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14409.1012
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
GCSettings.IsServerGC : False
Import-Csv C:\Users\un1\AppData\Local\Temp\5abb9571859c48cea807d755cc48058b.csv
TotalSeconds : 37.3680663

Name                           Value
----                           -----
PSVersion                      6.1.0-preview.688
PSEdition                      Core
GitCommitId                    v6.1.0-preview.688
OS                             Microsoft Windows 6.3.9600
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0
GCSettings.IsServerGC : False
Import-Csv C:\Users\un1\AppData\Local\Temp\5abb9571859c48cea807d755cc48058b.csv
TotalSeconds : 57.3899851

Peak memory usage for the test runs was as follows:

Windows PowerShell: 23.8 MB
PowerShell Core: 35.4 MB

Perhaps the specifics of the .csv file matter.

Yes, but Windows PowerShell frees the memory so quickly that sometimes memory consumption can not be seen.
I also see that in your test PowerShell Core is much slower.

I did a bit of iterative testing with a smaller CSV file (104866 bytes). On my system, the results show that Import-Csv performance does vary between back-to-back runs. The minimum runtime was 71 ms and the maximum was 192 ms. It looks like the runtime would decrease each time the GC came along. I'm running PowerShell inside a Docker container (mcr.microsoft.com/powershell:centos-7).
I understand this issue was filed based on processing much larger input files (>100 MB), so I hope these findings are not a distraction; I thought the findings may still be relevant.
$PSVersionTable

Write-Host "Creating source data..."
"random_numbers" | Out-File source.csv
Get-Random -SetSeed 1 | Out-Null
for ($i=0; $i -le 10000; $i++) {
	(Get-Random).ToString() | Out-File source.csv -Append
}
Write-Host "Measuring Import-Csv performance over time..."
"index,duration_ms,bytes_consumed" | Out-File 7112_results.csv
for ($i=0; $i -le 5000; $i++) {
	$millis = (Measure-Command { Import-Csv stats.csv }).TotalMilliseconds
	$memory = [System.GC]::GetTotalMemory($false)
	$i.ToString() + "," + $millis.ToString() + "," + $memory | Out-File 7112_results.csv -Append
}
Write-Host "Done"

$PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.0.2
PSEdition                      Core
GitCommitId                    v6.0.2
OS                             Linux 4.9.87-linuxkit-aufs #1 SMP Wed Mar 14 15:12:16 UTC 2018
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

Quartile
Runtime
Memory

Minimum
71.37 ms
10.19 MB

Q1
125.87 ms
39.88 MB

Median
138.86 ms
50.56 MB

Q3
151.39 ms
59.35 MB

Maximum
192.12 ms
65.38 MB

7112_results.csv.txt

@rthardin Thanks for great tests! Could you please continue and test on Windows - for PowerShell Core and Windows PowerShell for comparison?

Updated scripts for testing
##############################
Write-Host "Creating source data..."
"random_numbers" | Out-File source.csv
Get-Random -SetSeed 1 | Out-Null
for ($i=0; $i -le 10000; $i++) {
	(Get-Random).ToString() | Out-File source.csv -Append
}Write-Host "Measuring Import-Csv performance over time..."
"index,duration_ms,bytes_consumed" | Out-File psobject_results.csv
for ($i=0; $i -le 5000; $i++) {
	$millis = (Measure-Command { Import-Csv source.csv }).TotalMilliseconds
	$memory = [System.GC]::GetTotalMemory($false)
	$i.ToString() + "," + $millis.ToString() + "," + $memory | Out-File psobject_results.csv -Append
}
Write-Host "Done"

######################
Write-Host "Measuring Import-Csv performance over time..."
"index,duration_ms,bytes_consumed" | Out-File psobject_results.csv
for ($i=0; $i -le 5000; $i++) {
 $millis = (Measure-Command { [psobject]::new() }).TotalMilliseconds
 $memory = [System.GC]::GetTotalMemory($false)
 $i.ToString() + "," + $millis.ToString() + "," + $memory | Out-File psobject_results.csv -Append
}
Write-Host "Done"

######################
Write-Host "Measuring Import-Csv performance over time..."
"index,duration_ms,bytes_consumed" | Out-File psobject_results.csv
for ($i=0; $i -le 5000; $i++) {
 $millis = (Measure-Command { [object]::new() }).TotalMilliseconds
 $memory = [System.GC]::GetTotalMemory($false)
 $i.ToString() + "," + $millis.ToString() + "," + $memory | Out-File psobject_results.csv -Append
}
Write-Host "Done"

I started with Import-Csv and temporary disable output in the C# code. Results was well. So I guess that root of problems in creating PSObject and next two tests confirm this. Windows PowerShell is 30-100 percent faster.
It's unclear where the problem is - in PowerShell Core or in .Net Core.
/cc @SteveL-MSFT @daxian-dbw

@iSazonov to see if we slowed down creating PSObjects (looking at the Import-CSV cmdlet, it's really just adding NoteProperty members to a PSObject), I ran this test:

measure-command { 1..1000 | % { $a = [PSObject]::new(); 1..10 | % { $a | Add-Member -NotePropertyName "a$" -NotePropertyValue $ }; $a | Add-Member -NotePropertyName id -NotePropertyValue $_; $a } }

Windows PowerShell 5.1 consistently took around 15.7s on my machine.  PSCore6.1-3 consistently took around 8.4s, so about twice as fast.
Will have to investigate more where the bottleneck is.

@SteveL-MSFT I can confirm that property adding is faster on PowerShell Core. I think [PSObject]::new() is slower.

When looking at performance traces, comparing 5.1 to 6.0.2 (and preview, almost indistinguishable from 6.0.2) I found the following using the sample csv from @alx9r :
It is calls to System.Collections.Hashtable.InitHash(Object, Int32, out UInt32, out UInt32) that is twice as slow on 6.0.2. (5263ms compared to 9250ms). This comes from the fact that the windows version uses System.OrdinalComparer.GetHashCode(String) where core is using System.Globalization.CompareInfo.GetHashCodeOfStringCore(String, CompareOptions)
This function is called from several code paths from System.Management.Automation.PSMemberInfoIntegratingCollection1.AddToTypesXmlCache(T, Boolean)` which I think is called when we add properties to our psobject.
@SteveL-MSFT - I think you have to compare creating a PSObject with many properties. That is what seems to be expensive - not the creation of a PSObject per se.

@SteveL-MSFT In general, core is faster the full framework, but I think the code sample you used is running so much pipeline stuff that the general speed improvements hides the fact that the addition of properties to psobjects is slower.

Basically, this seems to boil down to String.GetHashCode being twice as slow.
foreach($i in 1..10000000) {$Null = "PowerShell6HasSlowerHashCode".GetHashCode()}

4.165s on Windows PowerShell.
7.167s on PowerShell 6.0.2.

@powercode thanks for doing the analysis.  I can confirm that I have similar result on my machine where PSCore6 is slower than Windows PowerShell calling GetHashCode().  Opened https://github.com/dotnet/corefx/issues/30994

Hmm.. I bet the issue in #6199 is related too then. The sampled JSON there was creating a large number of properties on a large number of PSObjects and then the same JSON was converted with NewtonSoft, the performance improvement was significant in terms of both memory usage and execution time.

Per dotnetcore team, this is expected as they made an improvement to GetHashCode() to be randomized (per session) for security reasons.

"per session" - this should not affect because we see the slowdown inside the session each time an object is created.

@iSazonov It is about them having a more expensive algorithm now. Not just different per session.
However, it seems like we have an opportunity for optimization. As it looks now, we recalulate the hash code for the every property name for every line in the csv. If we were smarter about how we created custom objects (some sort of prototype object maybe) a lot of the work could be amortized.
I'm thinking in the lines of "Create a prototype object, doing all the work needed in the typesystem" and then "Instantiate 10 million of these objects" given this array of property values.

@powercode I tried cloning PSObject but don't get benefits. I believe we need more generic improvements. Also not only Import-Csv is slow - ConvertTo-Csv and web cmdlets too #6199.

@iSazonov We would need new API:s

@vancem Please look the PerfView traces (for Core and Windows)
https://ufile.io/s3t7v
You can use follow PowerShell scripts to record new traces
cd c:\tmp\

# For measuring creating PSObject-s
"random_numbers" | Out-File .\source1.csv
Get-Random -SetSeed 1 | Out-Null
for ($i=0; $i -le 1000; $i++) {
	(Get-Random).ToString() | Out-File .\source1.csv -Append
}

function perf_test($source, $result) {
    Write-Host "Measuring Import-Csv performance over time..."

    "index,duration_ms,bytes_consumed" | Out-File $result
    for ($i=0; $i -le 200; $i++) {

            $millis = (Measure-Command { Import-Csv $source }).TotalMilliseconds
            # Uncomment this if you want analize results in Excel
            $memory = [System.GC]::GetTotalMemory($false)
	    $i.ToString() + "," + $millis.ToString() + "," + $memory | Out-File $result -Append
    }
    Write-Host "Done"
}

perf_test .\source1.csv .\resultsWindows1.csv

###############
# For measuring adding new properties to PSObject-s
$fields = 0..19 | ForEach-Object { "random_numbers$_" }
($fields -join ",") | Out-File .\source2.csv
Get-Random -SetSeed 1 | Out-Null
for ($i=0; $i -le 1000; $i++) {
    $values = 0..19 | ForEach-Object { (Get-Random).ToString() }
    ($values -join ",") | Out-File .\source2.csv -Append
}

perf_test .\source2.csv .\resultsWindows2.csv

I forgot to say that my traces is based on PowerShell Core 6.0 Preview3 with .Net Core 2.1.301.

@vancem Have you any news?

Faster hash may help some, but the real perf boost is to enable a scenario where we know up front that we will create a lot of pscustomobjects with the same layout and create the metadata objects up front. Then we should be able to amortize the work of creating the properties. @lzybkr, didn't you have some ideas around this?

Of cause we can make some improvements in this code but main question remains open - why this code works fine on .Net Framework and is so bad on .Net Core?

I agree that while there are a bunch of interesting performance questions here, by for the most interesting is why running the same powershell benchmarks on .NET Core runs slower than .NET Desktop.
Thank you @iSazonov  for providing the traces.   Sorry it took so long for met to really take a look.  I have been multiplexing investigations.
First just to understand the data you put on https://ufile.io/s3t7v

You are measuring the time it takes for powershell to Import-Csv on a file.
In the first run, there is only one column in the comma separated file you are reading in.
  Data in PerfViewDataCore*.etl.zip,
Powershell for Desktop .NET (PowerShell.exe).  Data in PerfViewDataWindows*.etl.zip.

Everything is running as 64 bit processes.   Of the two runs, the second one (with 20 columns in the CSV file), has the most dramatic difference.  It takes about 17 seconds to run on .NET Core but only 9.5 seconds on .NET Desktop.   Thus that is the most interesting case to look at first.
In both cases, from the trace it is clear that these are both CPU bound cases.   There is no interesting blocked time involved in the scenario.
As usual with cases like this where you 'swap out' large things (like runtimes), the difference is caused by many things.   It actually is non-trivial to properly analyze it as you have to be sure you remain 'fair' to each case as you 'drill in' and find interesting differences.
One way of testing if the app is 'most doing the same thing' is to see what objects it allocates.   Looking at the 'GC Heap Alloc Ignore Free' shows that the .NET Core case allocates 4GB of objects and the Desktop does 3.7GB (about 8%more).   Most of this is because of a Func delegate that is allocated in the
Runspaces.TypeTable.GetMembers() method (called from AddToTypesXmlCache).
This method is definitely a method of interest.  It is not clear if it is being called less or if a JIT optimization is not happening. but it is definitely of interest.
However an important point allocation are 'about the same (within 10%), so you would expect GC overhead to be in line with that.   It is not.   .NET Core's GC time is 3.5 seconds, where Desktop is 1.3 (2.7x bigger).   Drilling into the stacks inside the CLR.dll CoreCLR we see that HndScanHandlesForGC is 2 seconds of CPU time in .NET Core and only .076 seconds for .NET Desktop).   This is very likely because .NET Core uses dependent handles more.   Clearly this is worth fixing (it is 11% of the cost RIGHT THERE)
Another thing I look at is to group CPU by DLL and see if there are unexpected anomalies.  In particular I expect the 'user' DLLs to basically use the same amount of CPU independent of the runtime (sure runtime or framework things might be different and thus more expensive, but the CPU in YOUR dll should be basically the same.   This is NOT the case.   In particular for the microsoft.powershell.commands.utility DLL,

.NET Core CPU - 1977 msec
.NET Desktop - 357 msec

Thus .NET Core used 5.5X more CPU.   WOW, that is a big increase.   Drilling into it most of it is in the ImportCsvHelper.ParseNextRecord and the methods it calls.    There is clearly differences in inlining but it really is hard to get 5X differences because of code quality differences on real world code.  Clearly we have to dig into this.
@iSazonov - are we sure we are running OPTIMIZED code?   Running under a debugger can definitely turn off optimization we should check this.
So the simple answer is, the answer is not simple.    More investigation is needed.
@iSazonov  - I have a Powershell that ships with windows, but I don't have one for .NET Core.   Can you zip up your .NET Core version (ideally with source), so I can collect more specialized data to do further investigation?
There does seem to be a bunch of stuff the .NET Core team could do.  We should log an issue at https://github.com/dotnet/coreclr
Thanks

@brianrob @valenis

Actually I see that download instructions for Powershell Core.  I will give those a whirl.

@vancem Thanks for your detailed analysis.  The PSCore6 from our releases page is both crossgen'd and built as Release

are we sure we are running OPTIMIZED code?

No, traces I pushed on the share is not crossgen-ed. Sorry that I did not mention this. Really I tried locally Preview3 version and did not find a difference in behavior (of cause crossgen-ed binaries is more fast).

I did a bit of hacking...
With the following:

using the overload Add(PSMember, bool prevalidated) for all objects except the first when adding the noteproperties.
with a new constructor to PSObject that preallocates the _instanceMembers collection with an initial capacity
Replace a Linq.Any() with a List.Count > 0

I was able to improve the performance of import-csv by a factor of 10x.
A csv with 200k lines and 100 columns took 00:04:34 before the optimizations and 00:00:26 after.
We still spend a sizeable time if that calculating hashes, but it is at least a lot better.

I believe we should continue this research. We still did not find all the root causes of why Windows PowerShell has better performance.

When I looked at the traces, the hash code calculation showed up over and over again, when comparing 5.1 to 6.0. So that at least one of the root causes. PowerShell is a very heavy user of GetHashCode.

We could implement fast GetHashCode for some (what?) internal classes.

I would rather push for corefx to have an opt-out option for randomized GetHashCode() (and use it in specific cases where it's not a security concern) rather than doing our own implementation if this is the perf blocker.

It's worth checking if we're computing the hash code too often. Can the algorithm be improved?

@lzybkr  Tracking issue https://github.com/dotnet/corefx/issues/30994 . There seems to be no chance of doing Marvin algorithm faster.

@SteveL-MSFT I guess that having an opt-out option for randomized GetHashCode() is a problem - CoreFX do de-virtualization optimization for default GetHashCode() so without the optimization other GetHashCode() can be slower then de-virtualized Marvin.

@lzybkr There are for sure things we do that could be improved, but I agree with @SteveL-MSFT  that at the end of the day, there still be quite a lot of calls to GetHashCode. So having a way to opt out seems like a good path.
In my view, one does not exclude the other.

I would rather push for corefx to have an opt-out option for randomized GetHashCode()

Already exists

We could implement fast GetHashCode for some (what?) internal classes.

For HashTable your can pass a IEqualityComparer to the constructor. Better would to be to use the strongly typed  HashSet<string> and pass IEqualityComparer<string> to its constructor; which implements a non-randomized HashCode. It can be a readonly static singleton object as it shouldn't require state.
Dictionary<string, TKey> already has an opitimization for this if you don't specify the comparer for the string; but this doesn't currently exist in HashSet<string> (I don't think could exist in HashTable at the framework level as it is not strongly typed, so couldn't apply a fast path for strings).

Sorry I completely forgot what Hashtable did 😄
Its strongly typed equivalent is Dictionary<TKey, TValue>

Current status.

We added some optimizations to reduce allocations and increase performance. (Thanks @powercode!)

I see @benaadams start work on 3 optimizations in CoreCLR. Thanks!

We get new Issue #7603 about memory garbage (without confirmation). The same we see in current issue too - PowerShell Core doesn't free memory so fast as Windows PowerShell.
I found related Issue https://github.com/aspnet/Home/issues/1976 It was splitted on some issues.

We should do test only on release/crossgened binaries because in debug mode GC works differently.

In 6.1.0 RC1 trace I see that we do many allocations in System.Management.Automation.ReservedNameMembers.

GeneratePSTypeNames(System.Object)
Most in MemberMatch..ctor(). I have no idea if we can improve this, but it's surprising that we create links to standard methods dynamically while they could be static.

@iSazonov I'd say that there is something strange with your profiling session. You have a call to MemberMatch.ctor, but that is never invoked. I tried changing the definition to a static class, and it still compiled fine.
There doesn't seem to be a path from GeneratePSTypeNames to PSNoteProperty.Copy either, so I'm not sure how much I would trust that data.
When debugging, it is the same old AddToTypesXmlCache that shows up as the source of the copy. Not sure why the data is off. @vancem, can you help us here?
A breakpoint in GeneratePSTypeNames is not hit at all during debugging, and yet it shows up in the profile.

@iSazonov, you must be running something else than me. Can you share your command?

@powercode I am using the test published above for Import-Csv (columns = 700, rows = 5000 - I start the test and stop trace recording after some seconds). PowerShell Core 6.1.0 RC1 with crossgen.exe coped into pshome directory.
Screenshort above is GC Heap Alloc Ignore Free.
Traces https://files.fm/u/b2pysja2
Perhaps I do something wrong :-(

If you have symbols to system.management.Automation, setting up you symbol path (File -> Set Symbol Path) and source path (File -> Set Source Path), and then you can right click on a node name and select Goto Source, and PerfView will take you to the line (attributed with metric).    This allows you to see exactly what line is involved.
Note that inlining can often make it hard to follow  because the name in the source is really the caller (or caller's caller …).   Having the line number (which by the way is approximate, it may be the line before as well), is helpful to disambiguate.
If you ever get suspicious about things, you can fall back to the debugger.   Set breakpoints (probably with hit counts, every 1000 or so) for some of the locations that PerfView pointed you at and you can see if they agree with the debugger.
I have looked at lots of traces, and I have only rarely seen something where a name could not be explained by inlining only very, very rarely (and it could have been me misinterpreting the code).   If by checking with the debugger, you believe you have found such a case, I would need all the source, PDBs and repro information to really investigate the issue.

@iSazonov now that we have moved to .NET Core 3.0, can you rerun your perf test?

I'm even afraid to think about it :-))))
Will do.

-
Windows PowerShell
PowerShell Core

One column
6-17s/9-16Mb
1.8-3.4s/7.6-9.3Mb

700 columns
55s/16Mb
16s/10Mb

@iSazonov So the perf issue with Import-Csv is now gone after moving to .NET 3.0 preview? What do you mean by one column and 700 columns?

@daxian-dbw After @powercode improvements for PSObject all cmdlets works faster with the objects and make less allocations.
After moving to .Net Core 3.0 we get better GC. Now GC does cleanups even better .Net Framework (I say about the tests I did here). Perhaps .Net Core 3.0 add a performance too but I don't compare with previous PowerShell Core released versions.

What do you mean by one column and 700 columns?

In my tests I generate csv files. In first test the csv file consists of one column. Second - 700 columns. The first test shows a general increase in performance (I would assume that most of it comes from .Net Core 3.0). Second one explicitly shows an increase in performance which comes from @powercode PR.

@iSazonov so can we close this?

We get great progress for Import-Csv cmdlet so I close the issue.
Make sense to open new issues if we will find another scenarios with bad performance.
(Although I would like to have something to see performance history - improvements and regressions after new commits)

(Although I would like to have something to see performance history - improvements and regressions after new commits)

Absolutely. We need to have some perf tests that serve as benchmarks to help us see the trend. Currently we have none of those tests. Opened https://github.com/PowerShell/PowerShell/issues/9996


deptry correctly ignores lines that start with #  in requirements.txt, but not ones that just start with #

deptry correctly ignores lines that start with #  requirements.txt, but not ones that just start with #
For instance, a line # dev-only requirements (tooling) is correctly ignored by deptry, producing no output, but a line #dev-only requirements (tooling) produces the erroneous error message requirements.txt: DEP002 '#dev-only' defined as a dependency but not used in the codebase. This is in contrast to https://pip.pypa.io/en/stable/reference/requirements-file-format/#comments, which specifies: "A line that begins with # is treated as a comment and ignored."
System info:
deptry 0.13.0
Windows 10 ( Microsoft Windows [Version 10.0.19045.3930] )
Python 3.11.0

Thanks for raising this! I will look into it later today.


Fix exception inside iframe without access

If there is no access then !_canAccessIFrame() calcs to true and passed immediately to add function causing exception inside jquery lib

The fix resolved my problem. I'm waiting merge request.
Thanks.

Can we please merge this 🙏🏻

1 This is a huge issue for us

@malihu can we please merge this bugfix? It works

Merge and close this bug, please. It has to have a very high priority.

This fix worked for me as well while I was having the same issue, please merge.


Climate change impacts on abiotic stresses in DSSAT

Hello DSSAT team,
I am writing here as I am new to DSSAT and can't find what I am looking for in the documentation. I want to do some climate change impact studies on cereal and maize cropping systems in SSA, so marginal conditions with little to no fertilisation and irrigation. Now I have some questions before I start delving into preparation of input data and setup of my simulation:

Are abiotic stresses, esp. heat and drought stress, implemented in DSSAT and if yes, to which degree and for which crop?
How are extremes regarding temperature and drought handled in DSSAT in general? Do they affect other variables beyond soil and crop, such as pest damage?
Is CO2 fertilisation implemented? Can I modify CO2 concentrations as part of an e.g. climate input file? Are there limits on possible CO2 concentrations, or could I use pre-industrial baseline concentrations (< 280 ppm)?

That is it for now, and I hope some can answer these questions, as I am quite curious about DSSAT and its applicability in this space!
Best,
Lennart

Hi @LenLon!
Thanks for your message and interest in DSSAT but please next time for questions use contact us channel at https://dssat.net/contact-us
Answering your question. We would advise you to download DSSAT at http://dssat.net and go through the documentation where you will find several papers talking about that. All your questions are answered there, especially the 2019 DSSAT Chapter.
Through here would be very difficult to explain in details and the papers are a better resource.
Thank you.
Fabio Oliveira


Support mass resume download

Closes #64.

tested on staging, worked smoothly. I don't really trust that I caught everything tho..let's try prod sometime?


NET Support (publish to NuGet)

👍 if you want to see this module published to NuGet

This module is being published on NuGet: https://www.nuget.org/packages/Cdklabs.CdkPipelinesGitHub/


SpEL support for @Query annotation [DATAJDBC-397]

sopeLi opened DATAJDBC-397 and commented

@Modifying
@Query(value = "update orders o set o.status = #{#order.status}")
Integer updateOrder(@Param("order") Order order);

I want to get the parameters of the order by means of order entity then update.

Reference URL: https://stackoverflow.com/questions/57040030/spring-data-jdbc-use-query-comments-how-to-match-the-parameters-registered-for
Issue Links:

DATAJDBC-468 Supporting SQL Function around inserting value

DATAJDBC-526 Ability to change default sql type mappings

DATAJDBC-605 Parameter concatenation in queries using named parameters

5 votes, 4 watchers

Guys, did this change? I am trying to use that but it doesn't work.
I am using the 3.0.1 version.


Make tests deterministic

Integration tests fail sometimes as data is shuffled without a seed.

Hey, I'd be willing to help you out on this :) Do you have any extra information before I can start?

I think the current version (0.13.2.) of breeze does not support shuffling with random generators with seed. It was added in version 1.0-RC2 (https://github.com/scalanlp/breeze/commit/92cc09c3bdf9952cfbbdbb72d7afae5c4bf91421) with the implicit RandBasis parameter.

You are right. Given that only indices are shuffled, it should be possible to simply use scala.util.Random.shuffle and make seed an implicit parameter to cv.score().

Maybe instead of passing just the seed, we can pass the whole Random object like: (implicit rnd: scala.util.Random = new scala.util.Random())? Or maybe specify the seed when creating the object, like randomState in scikit-learn?

Passing the Random object implicitly seems even better. I think this ensures that some particular seed (or instance of Random) is only used in functions that take implicit Random (in our case cross-validation).


<IP_ADDRESS>/traffic

Stratux version:v1-4r1 (f6bad36cdc)

Stratux config:
SDR

[ ] single
[ ] dual

GPS

[ ] yes
[ ] no
type:

AHRS

[ ] yes
[ ] no

power source:
usb cable:

EFB app and version: (e.g., WingX Pro7 8.6.2)
EFB platform: (e.g., iOS 9.2)
EFB hardware: (e.g., iPad Mini 2)

Description of your issue:
<IP_ADDRESS>/traffic as described in
https://github.com/cyoung/stratux/blob/master/notes/app-vendor-integration.md
returns  "not websocket protocol", /getTraffic & variants returns page not found
all the other <IP_ADDRESS>/get*  pages work
this would be helpful in testing audio alerts

If possible, enable "Replay Logs", reproduce the problem, and provide a copy of the logs in http://<IP_ADDRESS>/logs/stratux/ and http://<IP_ADDRESS>/logs/stratux.log.

I asked essentially the same question and it was answered here:
https://www.reddit.com/r/stratux/comments/7f0912/help_using_ws192168101traffic/


Ad service undefined

Hello, i'm trying to update an ad and i'm having some problems.
I can update an ad_group_ad perfectly but i try to update the "final_urls" field of the ad inside the ad_group_ad and it gets an error:
Field 'ad.final_urls' cannot be modified by 'UPDATE' operation.
Then, i try to use the client path of the ad but it's not defined in the "path_lookup_util.rb" file and it gets a NoMethod error:
NoMethodError: undefined method 'ad' for #<Google::Ads::GoogleAds::PathLookupUtil>
And finally, i try to define the method by myself in the gem but it gets another error:
ArgumentError: No service found with name Ad
So, it looks like the service of the Ad resource is not defined and i can't do any operations with them.
Any help? Thanks.

The AdService doesn't exist in the Google Ads API yet. See: https://developers.google.com/google-ads/api/reference/rpc/google.ads.googleads.v0.services
For now in the Google Ads API you'll have to do it the old way, by deleting the ad and creating a new one. I'll let the team know that this functionality is missing to make sure they're aware so we can be sure that it makes a future release.

This is the way i'm doing it right now but there are some problems deleting and creating a new one too, for example i can't update an ad.image_ad because i need the "media_file" field to create it but that field is not selectable, so i can't get it to use it in the new ad.
Thanks for the response.

For product questions like this (i.e. things that would apply the same regardless of what language you're using, as opposed to issues that are specific to the Ruby library), you should ask in our product forum.
We have a team of support specialists who monitor the forum to help.


📝[Docs]:  config.py

What Docs changes are you proposing?
To store configuration settings such as API keys, risk thresholds, etc.
Why do the Docs need this improvement? What is the motivation for this change? How will this change benefit the community?
Configuration parameters for easy modification and access.
Describe the solution you'd like
Provide a clear and concise description of the changes you'd like to see implemented in the documentation.
Describe alternatives you've considered
If you've thought about alternative approaches or solutions, briefly describe them here.
Additional context
Add any additional context, examples, or screenshots related to the proposed documentation changes.

add this file in risk assessment


Remove "/" at line 1 for Minecraft

Removed the / at the start of the first line due to the fact that MCF won't run any lines that start with /

Sorry no approve just comment.

Since this is an mcfunction file, it is supposed to contain commands that are executed without slashes
https://minecraft.fandom.com/wiki/Function_(Java_Edition)#Running_a_function


ISPN-6518 GlobalTransaction objects are not cleared properly

https://issues.jboss.org/browse/ISPN-6518

Pushed to master and 8.2.x. Create a PR for 8.1.x together with ISPN-6047


Update Secure Baseline parser (for Policy Id)

🗣 Description
Update secure baseline parser to properly identify Policy Id and not pick up Policy implementation sections.
Updated unit test to accomidate changes.
💭 Motivation and context
Required to properly parse secure baseline documents.
closes #405
🧪 Testing

Execute Unit test for CreateReport - Verify no errors
Execute Invoke-Scuba on all products - verify report policy id, description, etc.

📷 Screenshots (if appropriate)

✅ Pre-approval checklist

[X] This PR has an informative and human-readable title.
[X] Changes are limited to a single goal - eschew scope creep!
[X] All future TODOs are captured in issues, which are referenced
in code comments.
[X] All relevant type-of-change labels have been added.
[X] All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
[X] Tests have been added and/or modified to cover the changes in this PR.
[X] All new and existing tests pass.

✅ Pre-merge checklist

[ ] Revert dependencies to default branches.
[ ] Finalize version.

✅ Post-merge checklist

[ ] Create a release.

@schrolla Implemented yor proposed solution.

@nanda-katikaneni Pull request is ready for merge into Emerald


Remove dependency on wildfly com.google.guava

Camel 2.24.x comes with guava 19.0 and requires guava 20.0 for grpc. Currently, we don't install either but wire to the wildfly provided version, which is > 20.0
For portability wfc should come with those slot(s) for guava that Camel defines

In extraA we have hadoop -> guice-3.0 -> guava-12
In extraC we have jclouds -> guava-16


add HadGEM-GC31-MM parameter files

These passed the nice dearprudence check 👍

@dgergel nothing particular to note, except that only ssp126 and ssp585 are available, so that latter will be used to be concatenated to historical.
If this looks good, I will merge away and run this one !


adding / updating linux / android rules

@williballenthin @mr-tz we're failing lints because aarch64 isn't introduced until https://github.com/mandiant/capa/tree/feat/1755. That means we can't merge this until the referenced PR is merged?

hm let's pull the aarch64 and android features into master now so that we can support these rules. I'm afraid of too many things getting blocked behind #1755

It may just be this commit: https://github.com/mandiant/capa/commit/e1186c5b4b30ea477e51bef144413f90b7f72798 or we can take all the changes to capa/features/common.py.


python3Packages.neo4j-driver: 4.4.2 -> 4.4.3

Description of changes
https://github.com/neo4j/neo4j-python-driver/wiki/4.4-changelog#443

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 170308 run on x86_64-linux 1

2 packages built:

python310Packages.neo4j-driver
python39Packages.neo4j-driver


No attention

What happened?
The package is not maintained properly and causes outdated clusters.
Expected Behavior
Package follow the rke_cluster TF provier releases or the RKE release cycles
Steps to reproduce

Output of pulumi about

Additional context
ke:index/cluster:Cluster resource 'rke' has a problem: expected kubernetes_version to be one of [v1.22.4-rancher1-1 v1.21.7-rancher1-1 v1.21.6-rancher1-2 v1.21.6-rancher1-1 v1.21.5-rancher1-1 v1.21.4-rancher1-1 v1.20.13-rancher1-1 v1.20.12-rancher1-2 v1.20.12-rancher1-1 v1.20.11-rancher1-2 v1.20.11-rancher1-1 v1.20.10-rancher1-2 v1.20.10-rancher1-1 v1.20.9-rancher1-1 v1.20.8-rancher1-1 v1.20.6-rancher1-1 v1.20.5-rancher1-1 v1.20.4-rancher1-1 v1.19.16-rancher1-2 v1.19.16-rancher1-1 v1.19.15-rancher1-2 v1.19.15-rancher1-1 v1.19.14-rancher1-2 v1.19.14-rancher1-1 v1.19.13-rancher1-1 v1.19.12-rancher1-1 v1.19.10-rancher1-1 v1.19.9-rancher1-1 v1.19.8-rancher1-1 v1.19.7-rancher1-1 v1.19.6-rancher1-1 v1.19.4-rancher1-1 v1.19.3-rancher1-2 v1.19.3-rancher1-1 v1.19.2-rancher1-1 v1.18.20-rancher1-3 v1.18.20-rancher1-2 v1.18.20-rancher1-1 v1.18.18-rancher1-2 v1.18.18-rancher1-1 v1.18.17-rancher1-3 v1.18.17-rancher1-2 v1.18.17-rancher1-1 v1.18.16-rancher1-3 v1.18.16-rancher1-2 v1.18.16-rancher1-1 v1.18.15-rancher1-3 v1.18.15-rancher1-2 v1.18.15-rancher1-1 v1.18.14-rancher1-3 v1.18.14-rancher1-2 v1.18.14-rancher1-1 v1.18.12-rancher1-1 v1.18.10-rancher1-2 v1.18.10-rancher1-1 v1.18.9-rancher1-1 v1.18.8-rancher1-1 v1.18.6-rancher1-2 v1.18.6-rancher1-1 v1.18.3-rancher2-2 v1.18.3-rancher2-1 v1.17.17-rancher2-4 v1.17.17-rancher2-3 v1.17.17-rancher2-2 v1.17.17-rancher2-1 v1.17.17-rancher1-3 v1.17.17-rancher1-2 v1.17.17-rancher1-1 v1.17.16-rancher1-3 v1.17.16-rancher1-2 v1.17.16-rancher1-1 v1.17.14-rancher1-1 v1.17.13-rancher1-2 v1.17.13-rancher1-1 v1.17.12-rancher1-1 v1.17.11-rancher1-1 v1.17.9-rancher1-2 v1.17.9-rancher1-1 v1.17.6-rancher2-2 v1.17.6-rancher2-1 v1.17.5-rancher1-1 v1.17.4-rancher1-2 v1.17.4-rancher1-1 v1.17.3-rancher1-1 v1.17.2-rancher1-2 v1.17.2-rancher1-1 v1.17.0-rancher1-2 v1.17.0-rancher1-1 v1.16.15-rancher1-4 v1.16.15-rancher1-3 v1.16.15-rancher1-2 v1.16.15-rancher1-1 v1.16.14-rancher1-1 v1.16.13-rancher1-2 v1.16.13-rancher1-1 v1.16.10-rancher2-2 v1.16.10-rancher2-1 v1.16.9-rancher1-1 v1.16.8-rancher1-2 v1.16.8-rancher1-1 v1.16.7-rancher1-1 v1.16.6-rancher1-2 v1.16.6-rancher1-1 v1.16.4-rancher1-1 v1.16.3-rancher1-1 v1.16.2-rancher2-1 v1.16.2-rancher1-1 v1.16.1-rancher1-1 v1.15.12-rancher2-8 v1.15.12-rancher2-7 v1.15.12-rancher2-6 v1.15.12-rancher2-5 v1.15.12-rancher2-4 v1.15.12-rancher2-3 v1.15.12-rancher2-2 v1.15.12-rancher1-1 v1.15.11-rancher1-3 v1.15.11-rancher1-2 v1.15.11-rancher1-1 v1.15.11-rancher1-0 v1.15.10-rancher1-1 v1.15.9-rancher1-1 v1.15.7-rancher1-1 v1.15.6-rancher1-2 v1.15.5-rancher2-2 v1.15.5-rancher1-2 v1.15.5-rancher1-1 v1.15.4-rancher1-2 v1.15.4-rancher1-1 v1.15.3-rancher1-1 v1.15.2-rancher1-1 v1.15.0-rancher1-1 v1.14.10-rancher1-1 v1.14.10-rancher1-0 v1.14.9-rancher1-2 v1.14.9-rancher1-1 v1.14.8-rancher2-1 v1.14.8-rancher1-1 v1.14.7-rancher1-1 v1.14.6-rancher1-1 v1.14.5-rancher1-1 v1.14.3-rancher1-1 v1.14.1-rancher1-2 v1.14.1-rancher1-1 v1.13.12-rancher2-1 v1.13.12-rancher1-1 v1.13.11-rancher1-1 v1.13.10-rancher1-2 v1.13.10-rancher1-1 v1.13.9-rancher1-2 v1.13.9-rancher1-1 v1.13.7-rancher1-1 v1.13.5-rancher1-3 v1.13.5-rancher1-2 v1.13.5-rancher1-1 v1.13.4-rancher1-2 v1.13.4-rancher1-1 v1.13.1-rancher1-2 v1.13.1-rancher1-1 v1.12.10-rancher1-2 v1.12.10-rancher1-1 v1.12.9-rancher1-1 v1.12.7-rancher1-4 v1.12.7-rancher1-3 v1.12.7-rancher1-2 v1.12.7-rancher1-1 v1.12.6-rancher1-2 v1.12.6-rancher1-1 v1.12.5-rancher1-2 v1.12.5-rancher1-1 v1.12.4-rancher1-1 v1.12.3-rancher1-1 v1.12.1-rancher1-1 v1.12.0-rancher1-1 v1.11.9-rancher1-3 v1.11.9-rancher1-2 v1.11.9-rancher1-1 v1.11.8-rancher1-1 v1.11.6-rancher1-1 v1.11.5-rancher1-1 v1.11.3-rancher1-1 v1.11.2-rancher1-2 v1.11.2-rancher1-1 v1.11.1-rancher1-1 v1.10.12-rancher1-1 v1.10.11-rancher1-1 v1.10.5-rancher1-2 v1.10.5-rancher1-1 v1.10.3-rancher2-1 v1.10.1-rancher2-1 v1.10.0-rancher1-1 v1.9.7-rancher2-2 v1.9.7-rancher2-1 v1.9.5-rancher1-1 v1.8.11-rancher2-1], got v1.26.7-rancher1-1
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@zs-ko Thanks for bringing this to our attention! Unfortunately we paused work on this earlier as we hit a snag with getting tests to pass. We've now made another attempt to update this provider and hope to get this out fairly soon!
Note, the upstream provider we pull in has been experiencing issues regarding RKE and AWS nodes.  Eg:
https://github.com/rancher/terraform-provider-rke/issues/404
https://github.com/rancher/terraform-provider-rke/issues/37


separate checking weight-mask from checking doscaling

What changes does this PR introduce?
Bug fix: Separate the check for weight-mask from check for rescaling of a control vector.
What is the current behavior:
Currently, if a user make a change to the weight-mask to be different from the ocean mask, a typical thing one can do to mask out where in the ocean the control will not be active, the non-zero count of the control vector will change depending on whether doscaling=.true. or .false. . What doscaling is meant to do is to switch back and forth between non-dimensional and dimensional space.  Whether the user chooses to scale the control by its weight (doscaling=.true. to have unit-less control) or not (doscaling=.false. to have control with physical unit) at this time should not affect the nonzero count of the control vector.
What is the new behaviour
Now the non-zero count of the control vector will be the same when the user switches between doscaling=.true. and .false. , applied for the case when the control's weight-mask is different from mask of the ocean.
Does this PR introduce a breaking change?
There will be a change IF any existing experiment has (a) weight-mask different than the ocean mask and (b) has set doscaling=.false.
Other information:
Suggested addition to tag-index
(To avoid unnecessary merge conflicts, please don't update tag-index.  One of the admins will do that when merging your pull request.)

LGTM

Took a look a t code changes, seems OK. Also checked that global_oce_cs32 and global_oce_llc90
(from verification_other) are not affected. Should be ready to merge once we know what to add to tag-index.


Default value of github.repository property set to git origin remote url #196

I reverted commit accidentally pushed to the wrong branch :)

I moved error handling to InitConfigFileTask, because in this case since GitOriginRepoProvider is used for generating default values for config file, user should be informed what default value was used instead and how can they change it. Details about the error will be logged below anyway (ProcessRunner logs it).


Insufficient user privilege after updates.

Hi,
did some updates today - but now it seems to have a new issue (after working fine for some week)
DSM 7.2.2-72806
Surveilliance Station: 9.2.1-11380
Some more info: out.txt
Error:
i even switched the USER from MANAGER to ADMINISTRATOR (for testing) but same error (see picture)
kr

Hi,
i updated my DS420+ to the same version as you after i finished the last changes.
Even my SurveillanceStation has the same version as yours.
My setup works well. A quick https check worked, too.
The question is where your setup differs of mine.
Do you have the "Surveillance Video Extension" installed?
My user is in the "users" group only.
It has rights to read/write "surveillance".
It has rights to access the apps "DSM" and "Surveillance Station".
In the Surveillance Station the user is "Manager of all Objects".

It works again,
thy for sharing your Full-User-Settings!
My user is in the "users" group only.    (SAME)
It has rights to read/write "surveillance".    (CHANGED to R/W - before just READ)
It has rights to access the apps "DSM" and "Surveillance Station". (CHANGED to DSM, too. -before just SS)
In the Surveillance Station the user is "Manager of all Objects". (SAME)
thy

Great to hear it working again.
If I have time I write a whole Readme part about the user setup


Added customizable asset picker component docs

Added customizable asset picker component docs
Description

Related Issue
https://jira.corp.adobe.com/browse/SITES-19685
Motivation and Context

We should also have a section with current limitation.
We are limited to a single custom component: https://git.corp.adobe.com/dx-devex-acceleration/uix-examples-internal/blob/main/uex-asset-picker/src/aem-uex-1/web-src/src/components/ExtensionRegistration.js#L33
We cannot open open the custom asset picker from the remote app view. As in we cannot double click on the image in the wysiwyg and open the custom asset picker.

We should also have a section with current limitation.
We are limited to a single custom component: https://git.corp.adobe.com/dx-devex-acceleration/uix-examples-internal/blob/main/uex-asset-picker/src/aem-uex-1/web-src/src/components/ExtensionRegistration.js#L33
We cannot open open the custom asset picker from the remote app view. As in we cannot double click on the image in the wysiwyg and open the custom asset picker.

Added limitations section
Limited to single ctom component: this does not limit ability to crete many variations of asset picker by providing different configurations, so I think practically it won't matter. We can discuss and then add this.


Better theme compatibility

With the default theme selecting the open terminal windows with the little icons works well. With for exp. Atom Material theme not.
They flicker and can't be clicked. I have to click between the terminal icon and the line under the icon so switch between the different terminals. Seems the size gets messed up somehow.
I made a video
Newest Atom, Termination, macOS

Woah, that is an issue! I'll look into it. :smile:
Thanks for trying Termination!

This is also an #issue with Platform.io's version of the terminal for Atom when used in combination with Atom Material theme and someone already mentioned a fix that worked for me (and others apparently).
Link to comment with fix
Fix by @kbrgl:

I've been experiencing the same issue with atom-material-ui. How did you disable the transition from devtools?
EDIT: Nvm, I figured it out. Adding this to my stylesheet fixed the problem:

.tooltip {
  transition: opacity 500ms;
}

The problem is that atom-material-ui adds unnecessary transitions to the transform property of tooltips. This resets it to only having transitions on the tooltip opacity.

Agreed, that fixed it for me. Thanks!

Thank you @girlandhercode!
So it is a bit buggy (imho) but not on our end. This info and the workaround are definitely going into the wiki. :smile:
Cheers!
Fred


Helper for loading history from JSON

What was changed
Added client.HistoryFromJSON helper to load history
Why?
Other forms of obtaining history JSON need to be supported beyond what's in the replayer
Checklist

Closes #808

I only added that option because our replayer has overloads that accept that option


mcomix: 2.2.1 -> 3.0.0

Description of changes

Migrate to by-name
Bump: 2.2.1 -> 3.0.0

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 279598 run on x86_64-linux 1

2 packages built:

mcomix
mcomix.dist


⚠ Bofrost.be has degraded performance

In e9c172b, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1902 ms

Resolved: Bofrost.be performance has improved in 0c2fc03 after 18 minutes.


luci-mod-network Add description: Hidden SSID, WMM

luci-mod-network Describe issues: Hidden SSID, WMM
The performance and reliability implications of hiding SSIDs or disabling
WMM are not commonly known and these settings often end up being
misconfigured to harmful effect.
To seek to mitigate this, add descriptions explaining that:
Where the ESSID is hidden, clients may fail to roam and airtime efficiency
may be significantly reduced.
Where Wi-Fi Multimedia (WMM) Mode QoS is disabled, clients may be limited
to 802.11a/802.11g rates.
Signed-off-by: Nick Lowe<EMAIL_ADDRESS>

Thanks


setting allow_nudging_weights_to_use_fast_gemm_kernel in the python API does not work

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux, 18.04 Ubuntu
TensorFlow installed from (source or binary): pip3
TensorFlow version (or github SHA if from source): 1.x

Command used to run the converter or code if you’re using the Python API
...
from tensorflow.lite.toco import toco_flags_pb2 as toco_flags
toco = toco_flags.TocoFlags()
toco.allow_nudging_weights_to_use_fast_gemm_kernel = True
converter.convert()
...

The output from the converter invocation
Bad value for conv/weights at index 187, previous bad value at index 175, distance=12, kMinDistanceBetweenBadValues=16. Consider passing --allow_nudging_weights_to_use_fast_gemm_kernel if you don't care about accuracy.

In particular
Consider passing --allow_nudging_weights_to_use_fast_gemm_kernel if you don't care about accuracy.
Also, please include a link to the saved model or GraphDef
N/A - see below

Failure details

expected behaviour - the erroring code continues past this step and either fails elsewhere, or continues to pass. It should not suggest the same fix.

Any other info / logs
I am restricted in the details I can post of my application.
It looks like the source does not have a check in there, so probably still in 2.x

@ianfhunter Can you please provide simple standalone code for faster resolution? You could use any public data. Thanks!

I'll try to do so in the coming week

@ianfhunter Are you still working on providing simple standalone code? Thanks!

Closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!


Springfox #845: Configuration for global Operation-Parameters. (Documentation)

Now (better late than never) the docu on issue #845

@GitVhaos thanks! :metal:


fix README.md URLs

Could you replace URLs in readme? They are broken for sure

Thanks for reporting.


Update documentation on pages web

What changes were proposed in this pull request?
Update documentation on pages web
How was this patch tested?
Test

@zhixingheyi-tian Please take a review, help merge it.

@zhouyuan  Please take a review on Page demo, this is corresponding to web page. please help merge it, thanks!


Add internal metadata fields for tracking ACKs and NACKs

Description
During a conversation with a colleague we discussed the need for some internal metadata fields to track ACKs (acknowledgments) and NACKs (negative acknowledgments) in Deckard. This issue proposes adding these fields to the metadata map of our messages.
Background
We were exploring the best approach to limit message retries (using NACK + LockMs) to a maximum of N attempts. The two options we considered were by using metadata or breakpoint field, which we were uncertain about its usage.
After discussing the matter, it became clear that for now the breakpoint field would serve as a suitable workaround, because the metadata field shuold not be used since it is not updated in NACKs or ACKs.
The "breakpoint" field is commonly used when a message is used to process a large number of elements, enabling the worker applications to track the progress and resume from where it left off. For example, if we have a social media page with 10,000 posts and need to process them in batches of 10, the "breakpoint" field allows the application to store the last processed position, ensuring it can continue from there in subsequent iterations on different workers.
Proposition
It would be beneficial to include some standard metadata fields in Deckard's pull responses. These fields would provide information on the number of ACKs/NACKs a message has received and the consecutive number of ACKs/NACKs. This information could be useful in the case described in this issue. It also would be useful, for example, in dynamically adjusting the time lock for a message based on the number of consecutive NACKs. All this controlled by the client application, which would be good to increase Deckard's ability to help users in different use cases without relying on some additional database.
The proposed metadata fields to be added are as follows:
{
 "_deckard.total_nacks" : "0",
 "_deckard.total_acks" : "321",
 "_deckard.consecutive_acks" : "12",
 "_deckard.consecutive_nacks" : "0"
}

Where all our internal metadata would be prefixed by _deckard to allow users distinguish between their own metadata.
We could add helpers for each client to get these numbers, since the metadata field is a map[string]string and I don't like adding each field individually in our message's struct. Any suggestion?

Maybe will be better adding a new field on the message struct called diagnostic or something like that, where we can put diagnostic information using either a map[string]number or map[string]any. The way protobuf implemented the any attribute is not something I like being on Deckard.

I think this is an awsome idea,
Recently @lucasoares  has told me of an "eternal" message in our private enterprise, this is something that could also have helped me out.
Can further lead down the road to even some sort of deadletter architecture... who knows.
Since the best every event sourcing system of the galaxy (google pubsub) also delivers this so called diagnostic fields so I would say lets just mirror what works and this is pretty much it.
I back up this idea.

I will add a new message inside all messas named "Diagnostic" and concentrate this kind of field inside it.


456 basic exhibitiontool

Closes #456

Don't forget to scroll up, I have reopened some comments by @olschulz


Update dataset metadata in A single-cell transcriptional roadmap of the mouse and human lymph node lymphatic vasculature

The Homo sapiens dataset in A single-cell transcriptional roadmap of the mouse and human lymph node lymphatic vasculature should be corrected to:

[ ] Use 'human adult stage' rather than 'adult' for the development_stage label as indicated by HsapDv:0000087
[ ] Use 'HsapDv:0000087' rather than 'EFO:0001272' for the development_stage_ontology_term_id per the schema  HsapDv term if human, child of EFO:0000399 otherwise

Note that 'adult' is correct for the Mus musculus dataset.

updated via migration to 2.0.0


latest cli doesn't honor sass extension

Bug Report or Feature Request (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request

Versions.

@angular/cli: 1.4.2
node: 8.4.0
os: win32 x64
Repro steps.

In a project that uses scss as style.
Generate a new component.
The css for the component will be .css instead of .scss.
The log given by the failure.

Desired functionality.

correct extension used for css
Mention any other details that might be useful.

Duplicate of #7624, #7644, #7647,& #7522

Closing as a duplicate.


[Docs] Cilium integration test not showing the latest test

Current State
The test results for cilium service mesh is not updating to the latest test results.

The table shows test result from 2022-05-10 12:32:04 UTC Tue which is not the latest one which can be seen in the past results page:

Desired State
Change the logic for Cilium service mesh such that it shows the latest test results in the https://docs.meshery.io/project/compatibility-matrix page.

Contributor Guides and Resources

📚 Instructions for contributing to documentation

Meshery documentation site and source

🛠 Meshery Build & Release Strategy
🎨 Wireframes and designs for Meshery UI in Figma
🙋🏾🙋🏼 Questions: Layer5 Discussion Forum and Layer5 Community Slack

Hello @asubedy , Can I work on this?

Sure @karnatisrinivas

Hello @asubedy, I've tried to analyze the issue but I haven't found any leads.
-> Currently, we are sorting stable channels based on their " Meshery-component-version", (This approach is working except for Cilium)
-> I tried to change the Sort condition to "timestamp" from " Meshery-component-version". By this approach, the results are showing only for 3 Adapters.
-> Cilium only has stable versions.

@karnatisrinivas any updates here? were you able to solve the issue?

I reopen this issue because the PR which was related to this issue is still on draft status.

This issue has been solved, Closing it.


TFLite converter needs flatc command.

The conversion to TFLite format has been greatly improved in nnabla 1.20.0, but since the flatc command is used internally, the conversion will fail if the flatc command is not installed in the environment used beforehand.
We plan to make further improvements to make it easier to use, but as of now you can either build it yourself using https://google.github.io/flatbuffers/flatbuffers_guide_building.html as a guide or install the flatc command in snap install flatbuffers under Linux environment.

flatc is now included in nnabla-converter wheel from v1.30.0.
https://github.com/sony/nnabla/pull/1084
So, you don't need to install flatc by yourself.


JitPack: Build is broken (Downloader Subversion Unresolved reference: requestPasswordAuthenticationInstance)

The JitPack build appears to be broken since commit b1cbf50b49cd0d7230d295a6a21a585eb90e3538
JitPack error log:
> Task :downloader:compileKotlin FAILED
e: /home/jitpack/build/downloader/src/main/kotlin/vcs/Subversion.kt: (268, 46): Unresolved reference: requestPasswordAuthenticationInstance

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':downloader:compileKotlin'.
> A failure occurred while executing org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork
   > Compilation error. See log for more details
...
BUILD FAILED in 2m 21s
96 actionable tasks: 96 executed
Build tool exit code: 0
Looking for artifacts...
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
Looking for pom.xml in build directory and ~/.m2
[Fatal Error] pom.xml:1:1: Premature end of file.

Example: https://jitpack.io/com/github/oss-review-toolkit/ort/eb7d747de0/build.log

That's a side effect of @mnonnenmacher having to revert our migration to Java 11 for some internal reasons. @mnonnenmacher, are you still confident that you can restore my changes by today?

Unfortunately not, all my test builds failed with the error below which prevented me from making progress. I will have to continue on Monday. As a workaround we could upgrade to Java 10 instead if it is urgent and would help, until I could fix the issue on our side.
Could not determine the dependencies of task ':cli:installDist'.
> Could not resolve all dependencies for configuration ':cli:runtimeClasspath'.
> A problem occurred configuring project ':clients:clearly-defined'.
> Could not open cache directory be2vyyzc82zj0yzd1epubxm40 (/home/.../.gradle/caches/6.7/gradle-kotlin-dsl/be2vyyzc82zj0yzd1epubxm40).
> org.gradle.api.internal.initialization.DefaultClassLoaderScope@30dddb0c must be locked before it can be used to compute a classpath!

@mnonnenmacher isn't this just https://github.com/oss-review-toolkit/ort/issues/3306?

Also see https://github.com/oss-review-toolkit/ort/pull/3333 that I just filed.

@sschuberth Thanks for #3333, that helped. Yes, it was #3306, it just took a while to verify that everything works correctly now. I have restored your commits, so the JitPack build should work again.

The JitPack build for commit 0d7fa47e5c40b8ae50c3ea536a8eaa33d492dd05 succeeded, so I'm closing this.


Multiple bug &  "low speed"

Hello I have tried the last three version :

9.5.2 : usbip with or without parameters command output nothing
9.3.4 : EDIT: works but still low speed for a webcam
9.3.3 : usbip can list and attach but use "low speed" on USB cam where before it was using "fulll speed"

Let's try 9.3.2 !

usbip with or without parameters command output nothing

I don't get it. Use --debug option. Empty output is OK if nothing to output.

wait, I unistall/reinstall (i try multiple version)

I mean this :

this happen with the latest version 9.5.2 (9.3.4 works, but still use low speed where it should not)

Ok installing the debug version I got this :

this explain why the release version did not output anything to the console

I'll check it tomorrow, thank you.

For information, if it can help, with the previous version (9.3.3) I got :

Thanks to you !

Some more informations :
With the version 9.3.3 I managed to make it works at full speed, it seems there is a maybe a mess between the two drivers, dont know.
If I deactivate xhci driver I fot error on usbip pot, but my camera is working well, I can use it maximum resolution.

When 0.9.3.X and 0.9.5.X are installed, make sure that you are calling right usbip.exe. It is binary incompatibility. I checked the latest release, all is OK.

I always unsinstall the previous version, I use the one that come with the installer is that ok ?

Re-tested, same, uninstall/reinstall and I got :

Another question, why some device appear in "low speed" ?
They was working well before, but now they are set as low speed

You should completely uninstall all usbip drivers. You should do it manually as described in readme.

Well I ll try, in the meantime I have found a cheap software that works well for me and wich look stable (wich shows that you have great potential to sell this one once packager & reliable)

I am not aware about commercial usbip implementations for Windows.

I just tried Virtualhere, it works flawlessly for me :

I have virtualised three USB of a raspberrypi3 to control my laser from lightburn (+ a controle camera) using WIFI.
Everything works perfectly, it is free for one USB over IP on a device and 50$ for unlimited (I have bought their license)

It use it own server (server installation requiere only one line, and is finished in less than 30s on raspberry)
Client side everything is in a single program.
Full server/client installation took less than 5 minutes.

There are a lot of proprietary solutions for sharing USB devices. VirtualHere is one of them, it can't work with Linux usbip server.

Actually, I would have prefer to use your solution as it is open source and already pretty advanced, but for now Virtualhere do the job very well and very reliabily without requiring windows test mode. I will keep an eye here. Keep up the good work !


Can you add support for these two original versions?

Hi, Hello, developer. First of all, I want to make it clear that I did not register a successful audiogames forum here, so I came here.
I use a translator to write, because I don't know any language except Chinese, I can only do so.
At present, I have downloaded two versions, these two versions are also English, are official versions, but they do not belong to the annual version, no longer steam, but he is very popular, at least in our case, many people are playing with these two versions, these two versions of tutorials, and auxiliary plug-in tools are also the most complete and most, I do not know if it is very difficult for Yao to add two versions.
If possible, I hope developers can add support for these two versions, which are not supported at present. I will post the information about these two original versions below.
Version number: <IP_ADDRESS>1 and version number: <IP_ADDRESS>5 belong to the original version released in 2009.
Many people are using these two versions, especially the <IP_ADDRESS>1 version, technical articles, tutorials auxiliary plug-ins are also the most complete and most. Very popular.
I hope to be able to support these two versions. It is not clear whether it is very difficult to do so.

Hello!
Yes it is possible, but it will take a lot of time.
Supporting four versions will also mean I have to do twice as much testing for every update, which will slow development.
I won't make any promises, but if I still have motivation after the current versions are complete, I'll try to support the old games.

ok
---Original---
From: @.>
Date: Mon, Dec 18, 2023 19:30 PM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [CG8516/PvZA11y] Can you add support for these two originalversions? (Issue #9)
Hello!
Yes it is possible, but it will take a lot of time.
Supporting four versions will also mean I have to do twice as much testing for every update, which will slow development.
I won't make any promises, but if I still have motivation after the current versions are complete, I'll try to support the old games.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>


bugfix: Force pages to default width.

This change is to compensate for an API bug in Confluence where newly
create pages end up getting a random width of either "fixed" or "full".
Fixes #173

Thanks!


Item duplication with Magic Mirror and Portable Mirror.

If I place an item in my portable magic mirror, then immediately close the inventory, the item I put in the mirror will be thrown on the ground and a duplicated will come out of the attached magic mirror.

This is still a thing, its easier to pull off with high latency.

I couldn't actually duplicate this, but I made the code that handles this a bit more secure. Let me know if this still happens in B17 onwards.


DC Timer failing to send players back to login/end game on server disconnect

I had this happen during the start of the game in the Meet phase. I stopped in the middle of things to take a screencap on the iPod Touch for another issue, and the iPod got disconnected from the server while safari was minimized (while I was uploading the screencap). This was the game state when I returned to safari:

After it got disconnected, the DC timer appeared on the other devices (an iPad 4, the MotoG, and Chrome Desktop) and started to count down from 60. Desktop and the MotoG both counted down and then returned to the login screen, but the iPad hung at 0s indefinitely:

Going to try and reproduce and see if this is still happening specifically on ipad after updates from yesterday

Assigning to @mattwbenson to try to repro since this may be fixed along with other fixes.

Saw this intermittently yesterday, but couldn't lock down why or when consistently besides knowing it was happening during the second reconnect (when we got the safari web exception referenced in another issue). But agree we should keep moving.


/etc/init.d/nagira start doesn't start

When I run the nagira-setup config:all everything works except when the services have to start.
And I check on the log file and I got this error:
/usr/local/rvm/gems/ruby-1.9.2-p320/gems/nagira-0.3.3/lib/app.rb:70:in block in <class:Nagira>': undefined method status_file' for #Nagios::Config:0x00000015b5fcb0 (NoMethodError)
from /usr/local/rvm/gems/ruby-1.9.2-p320/gems/sinatra-1.3.6/lib/sinatra/base.rb:1356:in configure'         from /usr/local/rvm/gems/ruby-1.9.2-p320/gems/nagira-0.3.3/lib/app.rb:62:in class:Nagira'
from /usr/local/rvm/gems/ruby-1.9.2-p320/gems/nagira-0.3.3/lib/app.rb:53:in <top (required)>'         from /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in require'
from /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in require'         from /usr/local/rvm/gems/ruby-1.9.2-p320/gems/nagira-0.3.3/bin/nagira:4:in <top (required)>'
from /usr/local/rvm/gems/ruby-1.9.2-p320/bin/nagira:23:in load'         from /usr/local/rvm/gems/ruby-1.9.2-p320/bin/nagira:23:in '
from /usr/local/rvm/gems/ruby-1.9.2-p320/bin/ruby_executable_hooks:15:in eval'         from /usr/local/rvm/gems/ruby-1.9.2-p320/bin/ruby_executable_hooks:15:in '
It strange because I try with different version of nagira and different version of ruby.
Maybe I do something wrong but I don't know.
For the moment nagira works only in test when I run it in the command line "nagira"
Thanks in advance.

Same as #34


KAFKA-8326: Introduce List Serde

Introduce List serde for primitive types or custom serdes with a serializer and a deserializer according to https://cwiki.apache.org/confluence/display/KAFKA/KIP-466:+Add+support+for+List<T>+serialization+and+deserialization
Test cases for the new serde will be added once the PR is reviewed by repo maintainers.

Usage: new ListSerde<>(Serdes.String(), Comparator.comparing(String::length)) to create a serde for List<String> values

@mjsax

Thanks a lot for the PR @yeralin! I like the idea and think it's a good addition.
However, adding those classes is a public API change and not a MINOR PR. Thus, it should be backed by a JIRA ticket and it also requires a KIP: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
Let me know if you have any question about the KIP process (should actually be well document in the wiki), or if you need any other assistance.
Btw: You should also add unit tests :)

Great to see some progress on this PR, but we still need a KIP... I would recommend to work on the KIP first (or in parallel), @yeralin

@mjsax KIP, JIRA, and DISCUSS thread are started: https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List<T>+serialization+and+deserialization

Hmmm, strange I added a test case, but it still says that test results were not found

Any updates on this?

@mjsax Sorry, I thought to finish all the discussions first, that's why I did not push.
New commit is in place! 😄

Hey, I'm sorry I thought you wanted me to merge with trunk that's why so many files popped up.
I think I know how to resolve this. I'll force push before the merge, and then keep pushing my commits only related to introduce_list_serde work. And at the end, once all reviews are completed, I'll do the merge and we can close this PR.

@yeralin Seems there is a merge conflict again.
And yes, I usually rebase to trunk (not merge trunk) and then push force to update a PR.

Hmmm, I think I did it! I haven't worked with Git with a project on this scale, so bear with me :)
I think I performed a rebase with apache/kafka:trunk, resolved couple conflicts along the way, and now everything should be fine.

Build failed with spotbug issue. Can you address this?
To test locally before pushing, I recommend to run ./gradlew streams:clean streams:test -- this will run checkstyle and spotbug, too.

I fixed all checkstyle warnings, except for:
Disallowed import - org.apache.kafka.clients.CommonClientConfigs.

under ListSerializer.java and ListDeserializer.java
Also, lots of tests are failing because for some reason they are expecting
DEFAULT_LIST_KEY/VALUE_SERDE_INNER_CLASS, DEFAULT_LIST_KEY/VALUE_SERDE_TYPE_CLASS and other properties to have default values:
org.apache.kafka.common.config.ConfigException: Missing required configuration "..." which has no default value.

I understand that I can set those default values during .define(... call. I guess I should set them to null?

As per unchecked warnings, I suppressed them at the variable or method levels.

@mjsax Would you have time to take a look? :)

Retest this, please

Retest this, please
Those were actually related spotbug errors, my fault 😅

I think it is ready for the next round of review :)
Thank you in advance!

Bump :)

Ready for another round :)

Bump 🥇

Bump 🔢

Bump 🥇

@mjsax would you be so kind to take another look? :)

Rrrrready for another round :)
Thank you 🥇

Thanks for looking into that, @mjsax , I also tried it out just now, and I think you're right.
The only thing I think we could do to be able to spit out a "fully typed" result is to introduce a "dummy interface" like Jackson's TypeReference. This kind of interface can be used to capture the full generic argument list at run time and use it to construct the collection. Short of that, I believe we'd be doomed to be producing rawtypes warnings in user code. Which is probably not a good choice for the API. Note, this is all in retrospect. It's not obvious at all that this would crop up until you try it.
In light of that, I think we probably need to change up the interfaces a little. Something like:
class ListDeserializer<Inner> extends Deserializer<List<Inner>> with a constructor like public <L extends List> ListDeserializer(Class<L> listClass, Deserializer<Inner> innerDeserializer). When you call deserialize, you'd get a List<Inner> (e.g., List<Integer>) out, but this is probably fine, since best practice is to declare the variable type as the interface, not the implementation anyway. And the advantage is that the calling code (aka "user code") wouldn't have any "rawtypes" or "unsafe" warnings at all.
Again, thanks a million for catching this!

@vvcephei Thanks for your input! I agree that that it would be better to preserve the "inner type" instead of the "list type", ie, deserialize() should return List<Integer> instead of raw ArrayList.
@yeralin -- Do you agree this this assessment? If yes, please update the PR accordingly.

@vvcephei Just pushed a commit with changes according to your description. Not entirely sure if I got everything right.

Updated KIP and JIRA ticket as well

Thanks @yeralin -- I am currently fully loaded with 2.4 release issues... Will try to cycle back to your PR but might take some time.

@mjsax Happy New Year! :)
I rebased the branch according to upstream/trunk. I noticed that branches aren't getting built anymore.
Let me know if there is anything else I can do here!

@yeralin Any progress on this PR?

Sorry, have been busy at work and personal stuff.
Should start making progress soon!

I don't think this construct works:
listClass = (Class<List<Inner>>) configs.get(listTypePropertyName)

In TimeWindowedDeserializer, they are doing it differently:
https://github.com/apache/kafka/blob/f8994897d30ef7508756dceb1d19171d6204f2a2/streams/src/main/java/org/apache/kafka/streams/kstream/TimeWindowedDeserializer.java#L64-L69

@mjsax Introduced
org.apache.kafka.common.serialization.ListSerializerTest
org.apache.kafka.common.serialization.ListDeserializerTest

They test non-arg-constructors and use configure methods.
I had to also update configure logic in both classes.
Take a look :)
Thank you!

Hmmm I was thinking about this magic flag you were talking about, and there are some corner cases that I want to discuss.
My initial idea was to create an enum:
public enum SerializationStrategy {
    NULL_INDEX_LIST,
    NEGATIVE_SIZE
}

Then it becomes counterintuitive:
Case 1: encoding primitives arr={1, 2, null, 3}
User chooses SerializationStrategy.NEGATIVE_SIZE, I guess we'd have to throw an exception saying that this serialization strategy is forbidden.
Case 2: encoding non-primitives arr={'a', 'b', null, 'c'}
User chooses SerializationStrategy.NULL_INDEX_LIST, but then what do we store for size of arr[2]? Zero? Negative one? We are still wasting 4 bytes that way.
I think our best bet is to simply use "null index list" for all primitives, and -1 strategy for all non-primitives avoiding any magical, serialization strategy flags.
What do you think?
@mjsax @vvcephei

Sorry for the late reply... Adding an enum sounds like a good idea.

Case 1: encoding primitives with NEGATIVE_SIZE

I don't think we would need to throw, however, we would need to choose the non-optimized encoding and encode the length for each list item (even if it is the same length) over and over again.

Case 2: encoding non-primitives with NULL_INDEX_LIST

We would store zero bytes, because we can skip the entire entry. During deserialization we can just do a list.add(null) when we hit the corresponding index as encoded in the null-index-list header, ie, during deserialization we maintain a counter for the index of the element we deserialize.
Does this make sense?

I think our best solution is to simply use "null index list" for all primitives, and -1 strategy for all non-primitives avoiding any magical, serialization strategy flags.

That would also be possible as an initial non-configurable default implementation, but I would still encode a magic header byte in case we want to change it later (just to be future prove). My suggestion to only implement one strategy was aiming to limit the scope of the PR. But if you want to implement both strategies, I have no objection either. However, if we actually do implement both strategies, than we could make it configurable from the very beginning on?

However, if we actually do implement both strategies, then we could make it configurable from the very beginning on?
I will implement both strategies. My problem with making it configurable is
..., however, we would need to choose the non-optimized encoding and encode the length for each list item (even if it is the same length) over and over again.
User might choose inefficient strategy and shoot him/herself in the foot.

That's why I was thinking just

use "null index list" for all primitives, and "-1" strategy for all non-primitives avoiding any magical, serialization strategy flags.

User might choose inefficient strategy and shoot him/herself in the foot.

I guess it's called freedom of choice :) If we feel strong about it, we could of course disallow the "negative size" strategy for primitive types. However, it would have the disadvantage that we have a config that, depending on the data type you are using, would either be ignored or even throw an error if set incorrectly. From a usability point of view, this would be a disadvantage. It's always a mental burden to users if they have to think about if-then-else cases.
I guess, in the end it's subjective which approach is better. But we could discuss pros/cons on the mailing list. Feel free to propose whatever you prefer personally (and list all pros/cons for different approaches) and people can just chime in.
Personally, I have a slight preference to allow both strategies for all types as I think easy of use is more important, but I am also fine otherwise.

Aaaaaan I am back! 😄  Sorry for such a long break, was dealing with some personal stuff and work.
Pushed a few commits with null-index-list and negative-size serialization strategy functionalities.
Rrrready for another round!
The code syntactically kinda raw, made it so to facilitate the reviewing process.
To review latest changes, the following set of commits is recommended for reviewing:
https://github.com/apache/kafka/pull/6592/files/b9ab94486e8d58903e2271978521050c3e1ffab3..bc9cd4bc47f8524db255ff65480868315541ef84
@mjsax @vvcephei

Any updates?

@yeralin -- your PR is in my review queue. Not sure how quickly I will find time to have a look though atm -- maybe next week, but I can't promise.

Any updates :)?

This would be an awesome addition, as I wouldn't have to maintain my own impl 😄.

@mjsax Let me know when I can jump back to it :)

Hi, any updates on this?

@yeralin -- sorry for the delay. Happy to hear that you are still on top of your KIP!
I did not find time yet to review the PR after your latest updates. (Not sure if it makes you feel any better, I did also not review any other KIP related PR for a while, and yours is actually at the top of my list...) -- I hope to find time soon and hope we can get this merged before the end of the year... Cannot guarantee it though. At least, I would like to get it into 2.8 release.

@mjsax Hey, just rebased my branch with the trunk, updated my tests to use JUnit 5.
Let me know when you guys will have time to review it.

@mjsax Hey, just rebased my branch with the trunk, updated my tests to use JUnit 5.
Let me know when you guys will have time to review it.

Any updates on this? :)

@yeralin -- Sorry that I did not get to review your updates yet... I need to finish #9744 first that is very close to get merged. Your PR is next in the list... (Just trying to work through my KIP PR backlog one-by-one...)
Btw: seems there is a merge conflict. Could you rebase the PR in the mean time?

Ok, resolved all conflicts. Thank you for dedicating your time into this :)

Ok, resolved all conflicts. Thank you for dedicating your time into this :)

Bump

Hey @yeralin , are you still working on this/looking for reviews? I know everyone is often busy but if you can just respond to let us know you're still active, we can try to pitch in to get this across the finish line.
cc @mjsax @cadonna @lct45 @wcarlson5 for help reviewing

I am! :) Routinely checking this PR.
I totally understand, we all get busy sometimes 👍

Replied under each review comment, waiting for response before pushing the review changes.

@ableegoldman Did you have time to look at it? Let me know if you need more context on my comments.

Hey @yeralin one other thing, can you give the KIP document a quick pass and make sure everything in there is up to date with what we've discussed and anything else that's evolved during the PR review? For example we might want to point out the null handling, though it's technically an implementation detail it would be good to clarify what is and isn't possible with this new Serde

Ok, updated the KIP with serializing nulls for different strategies.

@ableegoldman Seems like all checks passed :)

Build has only some unrelated test failures in known flaky KTableKTableForeignKeyInnerJoinMultiIntegrationTest#shouldInnerJoinMultiPartitionQueryable and
RaftClusterTest

Merged to trunk! Thanks @yeralin for all the work and patience it took to get this PR in, and to all the reviewers who helped get it here along the way.
@yeralin, can you update the KIP to note that it's completed, and then move this KIP to the "Adopted" section on both the KIP main page and the Streams KIPs subpage? 🙏

Thanks @yeralin for pushing it through and thanks @ableegoldman for helping out reviewing! Really nice addition!

Hey @yeralin @mjsax @ableegoldman  Getting SerializationException in this serde
SerializationException: Invalid serialization strategy flag value
Flag value is derived from bytes size (which is coming >2) which is amount of enum variables hence its breaking
seems to be a bug
please check

This is while using List where E is class which is Inner
Serde used for Inner is JsonSerde of Type E

@venkatesh010 could you please provide a code snippet?


Helm enabling EBPF on GKE needs updating

/area documentation
What would you like to be added:
Under either the GKE or the Helm section there should be some detail on setting the ebpf.enabled = true flag for installing falco via helm on GKE.
Why is this needed:
This would help other users attempting to install falco on GKE nodes running COS as the documentation doesn't currently suggest this as an option.
The first two comments relate to this issue

/assign