Avoid guid at the end of outputDirectory

On Coverlet we have some users complain about the dynamic folder name generated for reports file.
I did some check on code and seem that guid is "mandatory" and not removable.
https://github.com/microsoft/vstest/blob/master/src/Microsoft.TestPlatform.Common/DataCollection/DataCollectionAttachmentManager.cs#L101-L125
Could be useful add a parameter, maybe in runsetting like <DeterministicOutputdirectory>true/false</DeterministicOutputdirectory> and in that case avoid to concat guid.ToString().
Dynamic folder makes CI more complex.
Issues refs https://github.com/tonerdo/coverlet/issues/500 https://github.com/tonerdo/coverlet/issues/767
cc: @DaleyKD

I don't know the reasons for the guid but I would rather avoid yet another option. I see the coverage file described in the TRX output, but even thought there is a href to it, it does not contain the guid. I also don't see the guid anywhere else in the repo. Imho the best course of action is to provide a way that will allow the test result to be programatically processed.
When you run with CC in azure pipeline, do you see the coverage file automatically uploaded? maybe they have this already figured out.

Admittedly the trx, and xml in general is not the easiest to process, but it's a start, and it is generic format that we don't have to re-invent.

When you run with CC in azure pipeline, do you see the coverage file automatically uploaded? maybe they have this already figured out.

Looking at publish result task https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml seem that a globbing filter is used
- task: PublishTestResults@2
  inputs:
    testRunner: VSTest
    testResultsFiles: '**/*.trx'
    failTaskOnFailedTests: true

Admittedly the trx, and xml in general is not the easiest to process, but it's a start, and it is generic format that we don't have to re-invent.

I'm ok with file format, but non deterministic path add some work to do in some scenario, tipical "find right file and move to".

I meant it more in the direction of returning a simple object that can be programatically processed to get to the info, instead of forcing the naming conventions to be deterministic. For example in Pester I simply return a result object that the caller can process. Here we could do something similar. Say you'd want to upload your CC file:
$result = dotnet test --collect="Code Coverage" ....params.... --json-result
($result | ConvertFrom-Json).Attachements | where Type -eq "Code Coverage" | Invoke-RestMethod -Method PUSH

This would be the end product where we would have to come up with the result etc... So a more achievable goal would be to hookup the trx file and read the info from there if it was correct.

$result = dotnet test --collect="Code Coverage" ....params.... --json-result

Are you forcing powershell usage?If I don't use powershell what does --json-result produce?Is it a json on standard output?

Yes, it was just an exam

Od: Marco Rossignoli<EMAIL_ADDRESS>Odesláno: Wednesday, April 1, 2020 8:05:48 PM
Komu: microsoft/vstest<EMAIL_ADDRESS>Kopie: Jakub Jareš<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Předmět: Re: [microsoft/vstest] Avoid guid at the end of outputDirectory (#2378)
$result = dotnet test --collect="Code Coverage" ....params.... --json-result
Are you forcing powershell usage?If I don't use powershell what does --json-result produce?Is it a json on standard output?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/vstest/issues/2378#issuecomment-607406005, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABLYLYIFL7VICTTBJS7R7ILRKN67ZANCNFSM4LVPYFMA.

Yes json, it was just an example to show a concept. I am not forcing powershell on anyone.

Od: Marco Rossignoli<EMAIL_ADDRESS>Odesláno: Wednesday, April 1, 2020 8:05:48 PM
Komu: microsoft/vstest<EMAIL_ADDRESS>Kopie: Jakub Jareš<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Předmět: Re: [microsoft/vstest] Avoid guid at the end of outputDirectory (#2378)
$result = dotnet test --collect="Code Coverage" ....params.... --json-result
Are you forcing powershell usage?If I don't use powershell what does --json-result produce?Is it a json on standard output?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/vstest/issues/2378#issuecomment-607406005, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABLYLYIFL7VICTTBJS7R7ILRKN67ZANCNFSM4LVPYFMA.

What if we have more than one collector https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md#enabledisable-a-datacollector?
A json array with different "Type"?

Just stumbled upon this.

I don't know the reasons for the guid but I would rather avoid yet another option.

I disagree. VSTest should respect the specified ResultsDirectory and CoverageFileName for data collectors. I can do that for loggers already so I would argue DataCollectors to behave the same:
<RunSettings>
  <RunConfiguration>
     <!-- Directory for test run reports. E.g. trx, coverage etc. -->
    <ResultsDirectory>.\</ResultsDirectory>
  </RunConfiguration>
  <LoggerRunSettings>
    <Loggers>
      <Logger uri="logger://Microsoft/TestPlatform/TrxLogger/v1">
        <Configuration>
          <LogFileName>testResults.trx</LogFileName>
        </Configuration>
      </Logger>
      <Logger uri="logger://Microsoft/TestPlatform/HtmlLogger/v1">
        <Configuration>
          <LogFileName>testResults.html</LogFileName>
        </Configuration>
      </Logger>
    </Loggers>
  </LoggerRunSettings>

I would expect the following behavior:

LogFileName inside a DataCollector configuration node is respected:

<DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <CoverageFileName>coverage.xml</CoverageFileName>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

The specified ResultsDirectory is respected without any intermediate folders in it.

  <RunConfiguration>
     <!-- Directory for test run reports. E.g. trx, coverage etc. -->
    <ResultsDirectory>.\</ResultsDirectory>
  </RunConfiguration>

In this example you can even see that coverage is explicitly listed in the comment (which I took from the docs). Specifying the log file name seems to have worked in the past with the pre-coverlet code coverage collector: https://github.com/microsoft/vstest/issues/1957.

I just observed another misbehavior:

The coverage report is created twice:

vihofer_DESKTOP-803H9R0_2020-04-22_06_59_42\In\DESKTOP-803H9R0\coverage.opencover.xml
4a408eed-656b-435d-a540-30c285d1c57a\coverage.opencover.xml

The coverage report is created twice:

@ViktorHofer  I think it's a dup of https://github.com/microsoft/vstest/issues/2334

You are right :)

FWIW I currently need to clean the TestResults directory manually to make sure that ReportGenerator picks the right coverage results file. Ideally we would not need to do this.

The directory where the coverage results will be published is required to exist. Otherwise it throws:
Data collector 'Code Coverage' message: Data collector caught an exception of type 'Microsoft.VisualStudio.Coverage.VanguardException': 'Running event not received from CodeCoverage.exe. Check eventlogs for failure reason.'. More details: ..

That's also bad.

I think I see the purpose for the dynamic folder that is appended-- if you run it at the root level against a solution with multiple test projects, each project is in its own dynamic GUID folder.
A better solution would be to at least use the project name in this scenario. These names are predictable and should be unique:
TestResults/ProjectA/<coveragefile>.xml
TestResults/Projectb/<coveragefile>.xml
...unless there's some weird nesting in folders that would allow duplicate project names. That sounds like an edge case that the user would have to fix, though, and in that case you could use GUID folders per duplicated project or something.

Spent the evening doing tests here and there trying to understand why I was getting random folders. Glad that I finally found this, but sad this is not implemented. This is definitively not CI/CL friendly.

This is also not IDE friendly, as there is no deterministic way to find the most recent result folder.  Coverage highlighting tools (VSCode Coverage Gutters, in my case) can't be effectively configured to work with this behavior.

This is also not IDE friendly, as there is no deterministic way to find the most recent result folder.  Coverage highlighting tools (VSCode Coverage Gutters, in my case) can't be effectively configured to work with this behavior.

dotnet test standard output provides the generated folder name
"Starting test execution, please wait...\r\nLogging Vstest Diagnostics in file: C:\Users\tonyh\Source\Repos\DataCollectorXUnit\XUnitTestProject1\bin\Debug\netcoreapp3.1\fine-code-coverage\coverage-tool-output\diagnostics.log\r\nA total of 1 test files matched the specified pattern.\r\n\r\n
Attachments:\r\n  C:\Users\tonyh\Source\Repos\DataCollectorXUnit\XUnitTestProject1\bin\Debug\netcoreapp3.1\fine-code-coverage\coverage-tool-output\**7ba6447d-a89f-4836-bffc-aeb4799e48ab\coverage.cobertura.xml**r\nPassed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 7 ms - XUnitTestProject1.dll (netcoreapp3.1)"
The diagnostic log also has the generated folder name
TpTrace Verbose: 0 : 4712, 10, 2021/02/21, 14:58:38.317,<PHONE_NUMBER>845, vstest.console.dll, DataCollectionRequestSender.SendAfterTestRunStartAndGetResult : Received message: (DataCollection.AfterTestRunEndResult) -> [
{
"Uri": "datacollector://Microsoft/CoverletCodeCoverage/1.0",
"DisplayName": "XPlat code coverage",
"Attachments": [
{
"Description": "",
"Uri": "file://C:/Users/tonyh/Source/Repos/DataCollectorXUnit/XUnitTestProject1/bin/Debug/netcoreapp3.1/fine-code-coverage/coverage-tool-output/7ba6447d-a89f-4836-bffc-aeb4799e48ab/coverage.cobertura.xml"
}
]
}
]
Agree that should avoid the guid though.

Is there any solution found for this issue?
I'm suffering from the same problem, while msbuild integration allows a specification of ouput dir. Why not vstest?

@psvaiter Here is my solution. I call this executable before using the coverage file. The tool search for the coverage file and copy at the specified file path.
`using System;
using System.IO;
namespace CopyCoverageFile
{
class Program
{
static void Main(string[] args)
{
if (args.Length == 2)
{
var files = Directory.GetFiles(args[0], "*.coverage", SearchOption.AllDirectories);
            var path = Path.Combine(args[1], "YOUR_TEST.coverage");

            Directory.CreateDirectory(args[1]);

            File.Copy(files[0], path);

            Console.WriteLine("Is coverage created? " + File.Exists(path));
        }
        else if (args.Length == 1)
        {
            Console.WriteLine($"Is new {args[0]} exist? " + File.Exists(args[0]));
        }
    }
}

}`

Hoping this may help a few people out there, I use the following snippet of PowerShell in my development scripts to extract the unit test results paths and feed them to report generator.
$TestOutput = dotnet test --collect "XPlat Code Coverage"
$TestReports = $TestOutput | Select-String coverage.cobertura.xml | ForEach-Object { $_.Line.Trim() } | Join-String -Separator ';'
dotnet reportgenerator "-reports:$TestReports" "-targetdir:./CoverageReport" "-reporttype:Html"

@ArchieCoder I tried to use MSBuild integration and that worked as expected. Besides that I could use the Fine Code Coverage extension for Visual Studio that seems to depend on coverlet.msbuild package.
Here's the command I run:
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=TestResults\

This way I stopped using the vstest.

This is also not IDE friendly, as there is no deterministic way to find the most recent result folder. Coverage highlighting tools (VSCode Coverage Gutters, in my case) can't be effectively configured to work with this behavior.

I am right there with you. I'm not sure what genius though an artifact directory's name should be non-deterministic, but it's completely ludicrous to me. Nobody has, from my reading, presented a single good argument s to why you'd want a GUID for this. I could see the argument for a timestamp, but honestly, if somebody wants their artifacts in a GUID-named folder, make them do it themselves as its a lot easier to go deterministic->random than vice-versa. I'm not sure why those who want consistency are the ones being punished.

FWIW, I'm using dotnet and Visual Studio for .NET Core applications.  I wanted to execute my unit tests and generate a coverage report, with log files going into .results/log and the coverage report in .results/report.  I ended up going the "shell script" route (hey, we're all *nix users with WSL now, right?).  Here's what works for me:
Step #1:  Install Report Generator as a local tool:
dotnet new tool-manifest
dotnet tool install dotnet-reportgenerator-globaltool

Step #2:  Make sure your project has coverlet collector:
dotnet add package coverlet.collector

Step #3:  Create a Bash script file with below contents, tweak to your liking.  Don't forget to chmod +x it.  The script creates the .results folder if it does not exist, cleans up any previous files in .results, and then executes unit tests and report generation.  It exits with the exit code from the unit tests, so if you are running this in a CI/CD pipeline, you can continue (or not) if the unit tests pass.
#!/bin/sh
mkdir -p .results
rm -r -f .results/*
dotnet test --collect:"XPlat Code Coverage" -r .results/log -l trx
result=$?
dotnet reportgenerator -reports:`find .results -name coverage.cobertura.xml` -targetdir:.results/reports
if [ $result = 0 ]; then
    echo "All tests passed!"
else
    echo "Some tests failed :("
fi
exit $result

You could get beef this up if you wanted to only exit successfully (exit code of not zero) based upon code coverage, successful report generation, etc. but I have messed with this enough this morning ;)

A better solution would be to at least use the project name in this scenario.

This is not enough since you can have a test project with multiple target frameworks.

I ran into same issue.
This may help in some cases:
You can deterministically generate a trx log file which contains a partial path and name of the .coverage file using:
dotnet test --collect:"Code Coverage" --logger "trx;LogFileName=TestResults.trx"
and you can parse the XML to get the file path using:
static string? GetAttachmentPath(string trxPath)
{
    var doc = new XmlDocument();
    doc.Load(trxPath);

    var namespaceManager = new XmlNamespaceManager(doc.NameTable);
    namespaceManager.AddNamespace("vs", "http://microsoft.com/schemas/VisualStudio/TeamTest/2010");

    XmlNode? firstCoverageAttachment = doc.DocumentElement?.SelectSingleNode(
        xpath: @"/vs:TestRun/vs:ResultSummary/vs:CollectorDataEntries/vs:Collector[@uri=""datacollector://microsoft/CodeCoverage/2.0""]/vs:UriAttachments/vs:UriAttachment[1]/vs:A",
        nsmgr: namespaceManager);

    return firstCoverageAttachment?.Attributes?["href"]?.Value;
}

Related issue https://github.com/microsoft/vstest/issues/2334#issuecomment-1156634841

@MarcoRossignoli @nohwnd is someone actively working on this? It's still causing unreasonable pain and the issue is upvoted quite a lot. Also, the milestone seems to be incorrect as 17.1 already shipped.

@ViktorHofer We are not working on this. I will bring it up at tomorrow's standup.

Any progress? Everything that needed to be said was said in this topic. User should have an option to set static output directory and given the feedback, pretty sure this should be the default behavior.
Remember that you can always work incrementally on the final solution. First integrate the proposal found here (or anything similar) and then follow up with something neater if an additional option is not something you guys are aiming for.
If this issue is gone, please ignore. Using VS 2022 17.3.1 and the problem is still reproducing with dotnet test.

Provided (another?) example of run script workarounding this issue. Tested on my Win10 with msys2 and Debian9.

I believe the pseudo code for the solution that Microsoft could do is the following code. I have an hard time thinking it would be more complex than this.
var fixedDirectory = String.Empty;
if (args.Contains(“/fixed_directory”))
fixedDirectory = GetValue(“/fixed_directory”);
// Regular code
if (!string.IsNullOrEmpty(fixedDirectory))
File.Copy(guidDirectoryFileName, Path.Combine(fixedDirectory, Path.GetFileName(guidDirectoryFileName));

Well, I've stumbled into this & the related issues as I'm encountering the same problems. I kept seeing duplicate files in the differing folder names (guid -vs- generated name).  A bit saddened to see this has been an ongoing issue for what looks to be 2+ years now.
So far, I've seen the suggestion of parsing the TRX file to get the associated coverage file may be an avenue (but definitely not preferred) and then it seems like the other option is that if we know the folder-naming convention for the UserName + HostName prefix variant we could possibly just ignore/purge those as known dupes.
What I wanted to report & share (and maybe get an answer for) is that I have observed some very odd behavior along these lines; In my case I have 6 test projects and I use the --results-directory argument to point to my desired output path at the repo root (TestResults) along with the --logger "trx;LogFileName=MyProject.trx" argument and that does yield the correct/expected behavior.
But what I've noticed in regard to the code coverage result files is that the GUID-based directory is always present, and the file is always there, -but- I have seen the other User/Host name prefixed folders NOT get emitted reliably.  And it's random it seems as I've not identified any pattern as to when it fails to emit that variant.
So, add me as +1 anxiously awaiting a good solution to this problem space. I'd very much prefer to have more control over both the location where the coverage file is output and the name of it without the need to detect & handle dupes. I realize this may require an effort by both the msft/vstest team and the coverlet team as they're just build atop the provided plumbing.

After investigating if parsing the TRX file was something I wanted to pursue using, it looks like that is going to prove to be inadequate as well. In my case the path being mentioned in the 'attachments' is using the host-name - not the GUID-based folders.  So, it's still ambiguous and as I was sharing in my case, I've seen those UserName+HostName folders not getting created reliably.  Seems like I'm basically stuck with the 1 option of just ignoring/purging the folders that follow the UserName+HostName convention.
One question I do have: One of my key concerns is that when you process/analyze the code coverage files there is the notion of a 'hit count' and these dupes would be incrementing that count when they are merged - yielding misleading metrics potentially. Is that a correct assumption?

Sharing the below workaround. Note the single /*/ glob that will resolve to a direct child folder where the test attachment exists. The other xml files (from other test projects) exist in deeper folders.
 - task: PublishCodeCoverageResults@1
   displayName: Publish code coverage from $(Agent.TempDirectory)/*/*.cobertura.xml
   inputs:
     codeCoverageTool: Cobertura
     summaryFileLocation: $(Agent.TempDirectory)/*/*.cobertura.xml

Hoping this may help a few people out there, I use the following snippet of PowerShell in my development scripts to extract the unit test results paths and feed them to report generator.
$TestOutput = dotnet test --collect "XPlat Code Coverage"
$TestReports = $TestOutput | Select-String coverage.cobertura.xml | ForEach-Object { $_.Line.Trim() } | Join-String -Separator ';'
dotnet reportgenerator "-reports:$TestReports" "-targetdir:./CoverageReport" "-reporttype:Html"

I was absolutely struggling with this and I'm so glad that you pasted this in here. Thanks for an easy fix!

Instead of using a PowerShell snippet to do the operation "find the file in a random directory and copy it to someplace known", one can simply use a tool that can copy with "wildcard source paths". However, neither copy, xcopy nor robocopy seem to support that. Funnily, scp does. And it should be part of any modern Windows installation.
So, make sure you give a known output file name and use scp:
vstest.console.exe --logger:trx;LogFileName=%WORKSPACE%\UnitTestResults.trx;verbosity=detailed ^
--collect:"Code Coverage;Format=Xml;CoverageFileName=vstest.coveragexml" /EnableCodeCoverage ^
<YOUR UNIT TEST FILES HERE>

REM Copy the code coverage file to where it is expected
scp TestResults\*\vstest.xml vstest.coveragexml

Note, how the Code Coverage collector doesn't even respect the CoverageFileName since it simply changes the file extension... Way to go!

Hi @jachstet-sea,
We discussed these issues and we cannot do much because any change will have consequences on a lot of people. We will keep this issue open in case we can plan some breaking changes in some version.

We discussed these issues and we cannot do much because any change will have consequences on a lot of people. We will keep this issue open in case we can plan some breaking changes in some version.

Thanks for the update :)

Sorry we cannot offer anything better at the moment

@Evangelink If adding a parameter would cause the command line tool to break (which I doubt really), did your team consider fork this project and remove the guid creation?

Too bad this has not been addressed. In an attempt to workaround this issue and get deterministic output paths I wrote a an MSBuild target that moves files and deletes the directory with the guid. And of course I shot myself in the feet, deleted the wrong directory and now I'm running recovery software. I'm crossing fingers that I can recover my git repository where I had a lot of local branches (and stashes).

After a long amount of searching around, what I had to do to solve this problem (alluded to in previous comments) is to add the following to my .Tests .csproj:
<PackageReference Include="coverlet.msbuild" Version="6.0.0"/>

And then I had to switch from the following command:
dotnet test --collect:"Code Coverage;Format=Cobertura;CoverageFileName=coverage.xml"

To:
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput="coverage.xml"

After a long amount of searching around, what I had to do to solve this problem (alluded to in previous comments) is to add the following to my .Tests .csproj:
<PackageReference Include="coverlet.msbuild" Version="6.0.0"/>

And then I had to switch from the following command:
dotnet test --collect:"Code Coverage;Format=Cobertura;CoverageFileName=coverage.xml"

To:
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput="coverage.xml"

As long as you don't mind this hanging over your head:
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md
If your use the coverlet.msbuild way of solving this problem, be aware that you may run into this and have to revert to using coverlet.collector and have to deal with the non-deterministic nonsense in the end anyway (like I did).

You can also use dotnet-coverage tool to resolve this issue:
dotnet-coverage collect -f cobertura -o "D:\coverage\report.cobertura.xml" "dotnet test"

Check how to use dotnet-coverage tool in this scenario:
https://github.com/microsoft/codecoverage/blob/main/samples/Calculator/scenarios/scenario25/README.md

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform


Problem with templates selector

Before the V2, the champ templates, like this :
'templates' => [       'default' =>  'module:'.$this->name.'/views/templates/block/default.tpl',       'test' => 'module:'.$this->name.'/views/templates/block/test.tpl', ],
this worked, but now the selected template is not updated in the block, making this field non-functional.

Fixed https://github.com/PrestaSafe/prettyblocks/pull/103

Thanks !


seal_delegate_call api function (support for library contracts)

This PR implements delegate call as a separate api function seal_delegate_call
It is a follow up to #7 (second reason)

Thanks. Currently ooO. Will review next week.

@athei, please take a look at the seal_caller approach i came up with.
It uses the property that if all gas is passed and the account id is the same then we assume it was delegate_call.
We look for a caller recursively.
Let me know if you don't like it and want to add caller field to the frame itself.

@athei, please take a look again and let me know what you think.

@athei, i feel like this is gonna be a problem, as i don't have this button and the issue is still there

@athei, can i ask you to run benchbot manually so I can just update weights.rs file? I don't see any other options right now.

/benchmark runtime pallet pallet_contracts

@athei, can i ask you to run benchbot manually so I can just update weights.rs file? I don't see any other options right now.

I think it will fail when committing but the file can be somehow recovered. Let's see what happens.

Let's see what happens.

Okay

ERROR: Unable to push ./frame/contracts/src/weights.rs

Unable to push :(

Yeah as I expected. I will need to ask around how to handle this. Please hang on.

FYI the weights were generated for the following tree:
commit e99c99d4a667b83032021ab08b6682c616132518 (HEAD -> delegatecall)
Author: Parity Bot<EMAIL_ADDRESS>Date:   Fri Jan 21 13:05:30 2022 +0000

    cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

commit 1b6dc66dc062bc95dfa1d60c58168021b1b204fc
Merge: c21d0f3aa4 48e0b415b9
Author: Parity Bot<EMAIL_ADDRESS>Date:   Fri Jan 21 11:36:44 2022 +0000

    Merge branch 'master' of https://github.com/paritytech/substrate into delegatecall

commit 48e0b415b94d8ce14e0ebbbdb813263afa27f02e (origin/master, origin/HEAD)
Author: Shawn Tabrizi<EMAIL_ADDRESS>Date:   Thu Jan 20 14:17:16 2022 -0400

    dont increase the allocated size of vec (#10701)

commit c21d0f3aa4161254942d96ce5543bd33ab99956f (pr/delegatecall)
Author: yarikbratashchuk<EMAIL_ADDRESS>Date:   Thu Jan 20 15:13:22 2022 +0200

Notably I cut the output off at c21d0f3aa4161254942d96ce5543bd33ab99956f which is this PR's branch current HEAD.
https://gist.github.com/joao-paulo-parity/cb2ec45c9423a426aa4925da382263e7
git format-patch -1 HEAD --stdout > weights.patch
git diff HEAD~1 > weights.diff
cat ./frame/contracts/src/weights.rs > weights.rs

Thanks alot. @yarikbratashchuk can you apply this patch?

@athei, updated weights.rs.

Need. Need a second reviewer now.

I missed this in my notification stream, will take a look soon

@HCastano, would really appreciate your review :)

@yarikbratashchuk can you also merge master again?

/benchmark runtime pallet pallet_contracts

@athei, same problem, sorry for the inconvenience.
Next time I make PR from my account.

No worries. We are trying to resolve this situation by improving the bot: https://github.com/paritytech/bench-bot/issues/54
But  for now it would be best to open from your account until that issue is resolved.

https://gist.github.com/joao-paulo-parity/881c211f652f3332f8553dbe1a2a96f1 for https://github.com/paritytech/substrate/pull/10617#issuecomment-1027797781

@yarikbratashchuk Can you push?

Can you address the two open comments?

Oh, sorry, missed that. Done.

We merged something in the mean time. You need to merge master again.

So I can't find an error there. I am willing to merge this in an imperfect since it is still unstable. If you merge master so that you keep the weight changes from master and only add the functions you added then we can merge.

bot merge


Boot into cold spare on next system boot

Is it possible to use rugpi-ctrl to indicate that the system should try booting into the cold set on next boot without calling rugpi-ctrl system reboot --spare? The use case is for scenarios in which a new image has been downloaded and installed to the cold spare, but the reboot should not occur immediately (e.g. because the user is in the middle of something). Ideally, the user could power the system off whenever they please, and the next time they power it on it'll boot into the cold spare.

Whether or not this is possible depends on the details of the boot flow. The recommended tryboot boot flow for Raspberry Pi is realized by setting certain flags with Raspberry Pi's firmware^1 using an argument to the reboot system call. I do not think that it will be possible to use this mechanism/boot flow across power cycling. There is only little documentation of what actually happens under the hood, but from what I know, the flags are stored in EEPROM (non-volatile storage) only on some models while they are presumably stored in volatile storage on others where they are erased on power cycling. As this is not officially documented/supported, I think, it would be a bad idea to rely on any specific behavior. For the u-boot boot flow, however, it would be possible (and also rather easy to implement).
What boot flow are you using?

Thank you for the detailed and prompt reply!
I'm currently using the tryboot boot flow on an RPi4, but for no special reason other than it being default/recommended. If there are no/few drawbacks to using u-boot instead of tryboot, I'd actually prefer that because of u-boot's maturity and what sounds like possibly a more consistent experience across different boards?

Well, tryboot has certain advantages. For instance, it will likely be supported on future Raspberry Pi models immediately, whereas it may take some time for U-Boot to catch up. This is due to it being built directly into Raspberry Pi's firmware. Furthermore, it supports updates of the config.txt file and firmware files.
Actually, after thinking about it a bit more, I have a different idea: We could set a flag for Rugpi itself. The system would then first boot into the default partition but Rugpi would immediately initiate a reboot very early before even starting Systemd. Technically, there is a reboot then, but from a user's perspective, this will be very quick and without the system coming fully up. What do you think about this idea?

That sounds like a great solution! I think a key requirement, to your point, is that the user doesn't perceive a difference in boot time. If the switch to the cold partition occurs early in the boot process (prior to systemd coming up as you mention), then that should cover it.
If the reboot into the cold partition fails, then rugpi would behave as usual, correct? In other words, if the cold partition doesn't get committed as "hot," then a reboot should fallback to the "working" partition rather than continuing to try to boot into the new one? Maybe in this case there's a slight difference in that the reboot into the cold partition should only occur a certain number of times if it fails to be committed? I think u-boot handles this with a "try times" feature to prevent an infinite boot loop to a bad new update.

I think a key requirement, to your point, is that the user doesn't perceive a difference in boot time.

Indeed. The difference should be minor. Technically, this is just part of a deferred update and updates may take some additional time anyway, for instance, in case migrations need to run. So, I guess, it would be fine.

If the reboot into the cold partition fails, then rugpi would behave as usual, correct?

Correct, that would be a one-shot flag stored on the data partition that makes Rugpi reboot into the spare partition set once. The flag would be cleared prior to rebooting such that the system falls back to the default partition set in case the system does not commit. We could also have a counter, if this is desired.

Correct, that would be a one-shot flag stored on the data partition that makes Rugpi reboot into the spare partition set once. The flag would be cleared prior to rebooting such that the system falls back to the default partition set in case the system does not commit. We could also have a counter, if this is desired.

I think the option of retrying more than once would be helpful when booting from network, and any other scenarios in which determining the set is "good" relies on non-deterministic tasks. That being said, once is fine for my use case.

I think the option of retrying more than once would be helpful when booting from network, and any other scenarios in which determining the set is "good" relies on non-deterministic tasks. That being said, once is fine for my use case.

For those cases, one probably wants something more elaborate than a simple counter (e.g., something like exponential backoff). I am currently a bit busy and will take care of the flag in the next weeks.

Installing an update with --reboot deferred will now defer rebooting to the spare partition to the next boot. As discussed, this is done by setting a flag which is checked early during boot and, if the flag is present, triggering an immediate reboot to the spare partition.


SSL error when using client

I'm getting this error on Linux: https://gist.github.com/dmix/a3d4a0b63f42a691c41a
The solution here is to default to SSLv3 instead of using TLSv1 (or TLSv1_2)
https://stackoverflow.com/questions/23223691/ssl-connect-syscall-returned-5-errno-0-state-sslv2-v3-read-server-hello-a-fara
This is good overall for security.

How to set this though? @dmix

see https://github.com/BitBalloon/bitballoon-ruby/pull/12

I just saw that, would this also solve for if you're using the access_token instead of oauth?

Works now thanks.
@HangingClowns not sure


Vnet 3D segmentation: Train loss remains 0.99 all the time

Hi
I am trying to train 3D segmentation network and using the Vnet config file in the github repo.
But the result seems that it is not training at all since the loss remains all the time.
Why would this happen and am I missing anything?
Here is the config.ini I use.
[T1]
path_to_search = ../data
filename_contains = T1
filename_not_contains =
spatial_window_size = (32, 32, 32)
pixdim = (1.0, 1.0, 1.0)
axcodes=(A, R, S)
interp_order = 3
[parcellation]
path_to_search = ../data
filename_contains = Mask
filename_not_contains =
spatial_window_size = (32, 32, 32)
pixdim = (1.0, 1.0, 1.0)
axcodes=(A, R, S)
interp_order = 0
############################## system configuration sections
[SYSTEM]
cuda_devices = ""
num_threads = 2
num_gpus = 1
model_dir = ./models/model_vnet
[NETWORK]
name = vnet
activation_function = prelu
batch_size = 1
decay = 0
reg_type = L2
volume level preprocessing
volume_padding_size = 21
histogram normalisation
histogram_ref_file = ./example_volumes/monomodal_parcellation/standardisation_models.txt
norm_type = percentile
cutoff = (0.01, 0.99)
normalisation = True
whitening = True
normalise_foreground_only=True
foreground_type = otsu_plus
multimod_foreground_type = and
queue_length = 1
window_sampling = uniform
[TRAINING]
sample_per_volume = 32
rotation_angle = (-10.0, 10.0)
scaling_percentage = (-10.0, 10.0)
lr = 0.0001
loss_type = Dice
starting_iter = 0
save_every_n = 5
max_iter = 10000
max_checkpoints = 20
[INFERENCE]
border = (5, 5, 5)
#inference_iter = 10
save_seg_dir = ./output/vnet
output_interp_order = 0
spatial_window_size = (0, 0, 3)
############################ custom configuration sections
[SEGMENTATION]
image = T1
label = parcellation
output_prob = False
num_classes = 160
label_normalisation = True
The training looks like:
INFO:niftynet: training iter 1319, loss=1.0 (0.137058s)
INFO:niftynet: training iter 1320, loss=0.9937499761581421 (0.134881s)
INFO:niftynet: iter 1320 saved: /ifshome/qjiang/Downloads/vnet/models/model_vnet/models/model.ckpt
INFO:niftynet: training iter 1321, loss=0.9937499761581421 (0.146447s)
INFO:niftynet: training iter 1322, loss=0.9941431879997253 (0.156982s)
INFO:niftynet: training iter 1323, loss=0.9937499761581421 (0.149890s)
INFO:niftynet: training iter 1324, loss=0.9937499761581421 (0.155439s)
INFO:niftynet: training iter 1325, loss=0.9937499761581421 (0.148608s)
INFO:niftynet: iter 1325 saved: /ifshome/qjiang/Downloads/vnet/models/model_vnet/models/model.ckpt
INFO:niftynet: training iter 1326, loss=0.993877649307251 (0.139796s)
INFO:niftynet: training iter 1327, loss=0.9937499761581421 (0.133824s)
INFO:niftynet: training iter 1328, loss=0.9941365122795105 (0.139889s)
INFO:niftynet: training iter 1329, loss=0.9937499761581421 (0.142670s)
INFO:niftynet: training iter 1330, loss=0.9939338564872742 (0.141735s)
INFO:niftynet: iter 1330 saved: /ifshome/qjiang/Downloads/vnet/models/model_vnet/models/model.ckpt
Thank you very much!

This may be related to #208. If your segmentations are really tiny then the background can dominate the dice calculation. Read through the discussion there to see if they have any relevant solutions.

Try changing the loss function from to GDSC


Update the repository description with the correct link

Try to do it directly in the remote repository (on GitHub)


Use a lib folder instead of a long directory

Simply copy all needed libraries/references to the lib folder to save having the long directory which may break at any time

Thanks again!


Release 1.7.0 - [merged]

In GitLab by @Asacolips on Jan 23, 2023, 17:58
Merges release/1.7.0 -> master
Release 1.7.0
Downloads
Manifest URL: https://asacolips-artifacts.s3.amazonaws.com/dungeonworld/1.7.0/system.json
Combatible Foundry versions: v10, v11
Features

Added compatibility for Foundry v11.
Updated DE localization.
Added Ukrainian localization.
Added Italian localization.
Added price to equipment sheet.
Added equipment toggle feature.
Added class viewer (next to class name on character sheet) to view class details without having to level up.
Updated damage calculations to account for +damage tags.
Updated icons in compendiums.
Updated dark mode to default its value based on browser's preference.

Bug Fixes

Fixed issue with drag and drop behaviors of moves on the character sheet.
Fixed issue with hotbar drag and drop.
Clarified localizations for bond and ask moves.
Fixed typos in GM screen journals.

Credits
Thanks to Lifthraser, Jensenator, GM online, jegasus, and Giuseppe Abe for their contributions!

In GitLab by @Asacolips on Mar 22, 2023, 20:48
added 7 commits
3c4df892...364905d5 - 6 commits from branch masterbcd24ec5 - Merge branch 'master' into release/1.7.0
Compare with previous version

In GitLab by @Asacolips on Jun 4, 2023, 14:29
added 3 commits
d9263393 - Merge branch 'release/1.7.0' into dungeon-world-system-lifthraser-masterc74c8832 - Fix concat4c4306ff - Merge branch 'dungeon-world-system-lifthraser-master' into release/1.7.0
Compare with previous version

In GitLab by @Asacolips on Jun 4, 2023, 15:29
added 1 commit
0b822d63 - Update changelog
Compare with previous version

In GitLab by @Asacolips on Jun 4, 2023, 14:49
added 1 commit
2cc73e75 - Add 1.7.0 release notes
Compare with previous version

In GitLab by @Asacolips on Jun 4, 2023, 15:31
added 1 commit
0d72a956 - Update node version in gitlab ci
Compare with previous version

In GitLab by @Asacolips on Jun 4, 2023, 14:28
added 3 commits
365cf5a8 - Foundry V11.300 compatibility for Dungeon World system8d88fdf8 - DE translation update by user Jensenator0415eb3c - Merge branch 'master' into 'release/1.7.0'
Compare with previous version

In GitLab by @Asacolips on Jun 4, 2023, 14:31
added 1 commit
601f7dd1 - Update manifest
Compare with previous version

In GitLab by @Asacolips on May 1, 2023, 20:12
added 27 commits
bcd24ec5...182c987d - 17 earlier commits0fee017d - code cleanup4a3c4bf8 - Merge branch 'class_viewer' into 'master'f5e64680 - Fix drop-to-hotbar behaviour for non-item objects.56665f78 - Merge branch 'dragmove-fix' into 'master'dab5ee4a - clarification about bond moves and ask moves21e0b16f - clarification about bond moves and ask moves v2d4376897 - clarification about bond moves and ask moves v3aa997bd7 - class viewer - add mouseover helper and change the name of the popup window6e979316 - Merge branch 'release/1.7.0' into dungeon-world-system-lifthraser-mastere11d535f - Code syntax cleanup
Compare with previous version

In GitLab by @Asacolips on Jun 4, 2023, 15:39
mentioned in commit 431679fd7328fb2d93e7af76a00f41a9a591480e


NOT READY TO MERGE: Use eternal_watch feature of our fork of python-etcd to:

get cluster UUID checking
do more efficient polling with fewer reconnects.

Can one of the admins verify this patch?


feat: Update L2GasOracle address to L1MessageQueueWithGasOracle

PR Summary
Our upcoming upgrade will merge L1MessageQueue with L2GasPriceOracle into a single contract L1MessageQueueWithGasPriceOracle. The old oracle will be deprecated.
This PR should only be merged after the upgrade has concluded.

Checklist

[X] There are breaking changes
[X] I've added/changed/removed ENV variable(s)
[ ] I checked whether I should update the docs and did so by updating /docs

Description
N/A

Fixed in #916.


Corrected spelling of environment

Can one of the admins verify this patch?

Thanks for the catch and fix @naunga

@bryevdv My pleasure.

Thanks @naunga !


activity bar invisible  no sense

VS Code Version: 1.72.0
OS Version: Darwin x64 21.6.0

Steps to Reproduce:

make activity bar invisible with toggle activity bar visibility
open remote project
while I open remote project, it pears file expolorer
see screenshot below

This is no an issue


auto load our old js on load event

first of all thank you for providing this package ,
i have a  asp.net core razor pages app and writing so many js code , now if i want to import them to blazor ssr with this way , need to go every js and wrapp them up inside of export function onLoad() {} , there is a way to tell auto loading the js file on some event like onLoad() ?
on mean somthing like

and no need to write the export function onload  anymore and this js file just run on load

Hi @AmBplus,
If all you want to do is load a script dynamically, you don't need to wrap it in another script that defines an onLoad() handler. You can just reference the script directly from the <PageScript> component. The onLoad, onUpdate, and onDispose callbacks are optional, and only exist to implement functionality that responds to navigation events.

TNX , in asp.net core mvc or razor pages , we have asp-apend-version = true , so i don't need to change the file by myself , in this package we have something similar ?


Memory overhead(?) for Convolution schedule

Not sure whether it is something with the memory pool.
For this schedule, https://github.com/yzhliu/topi-intel/blob/master/mem_issue/conv.py , if I simply separate data pack and convolution calculate (then add up the costs), I can always get 10% speedup against that is combined:
$ TVM_NUM_THREADS=4 python2 conv.py
MKLDNN - g1mb1_ic3ih224iw224_oc64oh112ow112_kh7kw7_sh2sw2_ph3pw3_n : 0.773982 ms
Run separately ...
Use memoize topi.tests.test_topi_conv2d.verify_con2d_nchw.get_ref_data.pkl(5, (1, 3, 224, 224), 'float32', 3, 2, (64, 3, 7, 7))
--- schedule data packing ---
data -> data_vec: 8.82842e-05 secs/op
--- schedule kernel packing ---
kernel -> kernel_vec: 2.59738e-05 secs/op
--- schedule conv & unpack ---
conv & unpack: 0.00110448 secs/op
total = 0.001219 sec/op
Run together ...
Use memoize topi.tests.test_topi_conv2d.verify_con2d_nchw.get_ref_data.pkl(5, (1, 3, 224, 224), 'float32', 3, 2, (64, 3, 7, 7))
conv all: 0.00146011 secs/op
MKLDNN - g1mb1_ic64ih56iw56_oc64oh56ow56_kh3kw3_sh1sw1_ph1pw1_n : 0.72514 ms
Run separately ...
Use memoize topi.tests.test_topi_conv2d.verify_con2d_nchw.get_ref_data.pkl(5, (1, 64, 56, 56), 'float32', 1, 1, (64, 64, 3, 3))
--- schedule data packing ---
data -> data_vec: 0.000432787 secs/op
--- schedule kernel packing ---
kernel -> kernel_vec: 2.77567e-05 secs/op
--- schedule conv & unpack ---
conv & unpack: 0.00089618 secs/op
total = 0.001357 sec/op
Run together ...
Use memoize topi.tests.test_topi_conv2d.verify_con2d_nchw.get_ref_data.pkl(5, (1, 64, 56, 56), 'float32', 1, 1, (64, 64, 3, 3))
conv all: 0.001539 secs/op
While I don't see similar issue with this simple GEMM schedule: https://github.com/yzhliu/topi-intel/blob/master/mem_issue/gemm.py
The test was made on an AWS C5 instance.

closing for now as @yidawang 's answer might be the case, can reopen and investigate later


Missing File for training

Dear Author,
I sincerely thank you for your great work.
Could you please add train.py to the utils folder?
It was missed in the current repo. Thank you!

Hi,
Thanks for your interests! Sorry for the missing file. We have uploaded the corresponding file. Please check it.

@Layne-Huang The models folder is missing in Zenodo, is this intentional or is zenodo outdated?

Hi, sorry for the late response. We have added another link for the model weights: https://drive.google.com/file/d/1Zn8KbumzQ5S_Rc6lb4ezVvydgTeHjzbI/view?usp=drive_link


Allow imagePullSecrets to be optional

Certain cluster schedulers, such as Run:ai, handle imagePullSecrets independently of what's listed in the manifest file. Additonally, it's possible that users could push a custom NeMo image to another container registry that doesn't require authentication. As such, the ability to omit the imagePullSecret from the config should be supported to allow users to run jobs on kubernetes without it.

Closing in favor of the combined #418


remove NO DATA when there ain't an editor

See this: al-Burhān fī wujūh al-bayān, NO DATA (n.plac., n.pub., 1930)
We don't want that NO DATA

This was a tricky one. The NO DATA was because there was no edition type, not because there wasn't an editor.

Well, in the process of finding solution you also found the problem....It should have been other way round...My apologies

haha, there is nothing to apologise for. You were right in making a ticket, because the NO DATA shouldn't have been in the text.


Merge sugar and glucose, touch up how some drinks metabolize.

About the PR
Changelog

:cl: Rane

tweak: Glucose and sugar are now the same thing.
add: Alcohols now metabolize into some ethanol.
tweak: Heavily caffeinated drinks metabolize into some theobromine instead of directly containing theobromine in their containers.

idk if i understand the ethanol change

idk if i understand the ethanol change

Just googled the alcohol contents of all the drinks and divided in half since it's per 0.5u

no i dont understand why its being done in the first place

like what does it add

All of the alcohol effects can be put on ethanol itself and avoid most of the jank of SS13 where to get drunk you need to just drink more types of drinks. Like, literally, if there was just 3% abv beer under 10 different names in ss13 then drinking 1/10th of the amount of each of them would be deadly but drinking all of one of them does nothing.

okay, so basically just metabolizing into amounts of ethanol rather than defining the alcohol power on the reagent itself (i am currently working on adding alcohol effects and thats what i was doing)? i guess i'm okay with this


Add Avril 14th rpc/grpc/api

@stefanopier can you resolve the conflict please? ill merge afterward.


Change default form action errors when progressively enhanced

Describe the problem
I hope I can convince the maintainers that the default behavior for errors in form actions should be changed.
Currently, if you have a progressively enhanced form (without a custom callback), and it errors, your entire page unloads, and displays the nearest error boundary (if present) or shows a default error page if not.
Errors happen, but there's no way this should be the default behavior. A user should not see their current page unloaded, and replaced with an error page because a form post happened to fail.
Describe the proposed solution
For progressively enhanced forms, if the action fails, leave the form in place, and send the error info into $page.form, and rely on the user to display whatever error ui they'd like.
Anticipating one possible response, this would cause use:enhance to break with default, non-enhanced behavior. If that's a big problem then I would submit (pun!) that a non-enhanced form should probably not redirect to an error page either, if there's an error, but rather re-render the same form, with info about the error, so the user can potentially try again. I led with enhanced forms only since that's what I care about most, and figured that would make this a smaller request; but changing the default behavior of form action errors might be the simpler (and better) solution here.
Alternatives considered
The current workaround is to return a custom callback, and simply not call update().
That works, but it basically makes use:enahnce (by itself) essentially useless, unless you can guarantee that the action will never error out (or you disagree with my premise, and actually want the entire page to unload, and show an error page).
Importance
would make my life easier
Additional Information
No response

I don't expect maintainers to design api's around Twitter polls, but I do feel strongly about this (and really want SvelteKit to succeed) so I opened a poll here to try to gauge opinions beyond just my own
https://twitter.com/AdamRackis/status/1599899980528951297

I'm against this change as well.
As you wrote, you can easily implement this on your own by passing a callback to the enhance action or create your own enhance action.
If you want this behavior for "un-enhanced"-forms as well, you can wrap your form action in a try-catch block and return invalid() in the catch block.

@matt-l-w it seems crazy to me that an application framework like SvelteKit should be beholden to how web forms happen to work by default, given a web platform that was designed decades ago, to serve static documents.
To say nothing of the fact that implementing user-hostile defaults will not fix those standards. Folks who maintain these standards historically believe in “don’t break the web” (a good thing) so it’s not like frameworks implementing different defaults will somehow get these standards changed.
And a number of people replied to tell me that popping the user to an error page goes against a11y.

Error pages aren't happy-path UI.  In this case, you're not handling unexpected errors inside your action. There are basically three possible reasons an action could fail:

user data was invalid, in which case return invalid(4xx, {...})
something went wrong on the server (an external API failed, or whatever) in which case return invalid(5xx, {...})
something went wrong on the server but you failed to anticipate it <-- YOU ARE HERE

(Side-note, but I wonder if invalid should be renamed fail or something, since 'invalid' feels more 4xx-y than 5xx-y.)
What you're saying, in effect, is that you can't possibly be expected to anticipate all the ways in which your app should go wrong, and so the framework should step in to save you from yourself — even though it has no special knowledge of the error — but only in the case of progressively-enhanced forms (not load, or references to missing data in your markup, or a manual fetch to a non-existent URL, etc etc etc). Handling errors is your job. The framework's +error.svelte boundaries are more of a backstop than anything.
If you don't handle the error and expect the framework to do (handwaves) something when it occurs, then maybe you've made the progressive enhancement case slightly nicer, but your non-progressively-enhanced forms (whether because you didn't add use:enhance or because JS failed for some reason) remain broken.
And that's only if you add code to handle the error case on the page. Let's talk about that. SvelteKit can't help you in this case...
<form method="POST" use:enhance action="?/newTodo">
  <input name="description" placeholder="What do you need to do?">
</form>

...because there's nowhere to show an error. We could add some UI there...
<form method="POST" use:enhance action="?/newTodo">
+  {#if $page.error}
+
{$page.error.message}

+  {/if}
  <input name="description" placeholder="What do you need to do?">
</form>

...but what happens when you have multiple forms on the page? Where does the {#if $page.error} UI go here?
{#each data.todos as todo}
  <form method="POST" action="?/updateTodo" class:done={todo.done}>
    <input hidden name="id" value={todo.id}>
    <input name="description" value={todo.description}>
    <button formaction="?/toggleTodo>{todo.done ? 'mark todo' : 'mark done'}</button>
  </form>
{/each}

If you're handling the error yourself, you can include the id of the todo when you return invalid(...) to create the appropriate UI. The framework can't possibly be expected to have that level of understanding, because that's not its job — it's yours.
In summary, the current behaviour is deliberate and isn't going to change. Please don't use context-free Twitter polls with leading questions to try and influence maintainers — we're smarter than that.

I did clarify that I'd be happy to see non PE forms change this default behavior, too.
There's a variation of option 2 that's not accounted for, above. If your network is spotty, and for whatever reason does not connect to your form's endpoint, your existing UI, iiuc, will unload and render an error page, rather than update some sort of $page.form.error, (or perhaps $page.form[id].error if there's more than one form)
More broadly, it sounds like your UI will break badly unless app developers catch every possible exception, and wrap with invalid().
Why isn't that the default, then? Shouldn't app frameworks try to put devs into a "pit of success"? Defaults matter.


Typescript integration

Is there any plans on typescript integration? Like adding an warframe.d.ts? If not i can start working on it sometime next week.

As I wrote in the Discord, I'm just gonna rebuild it in TS. I thought of doing it today, but family calls 🎁

i'm working on pulling in warframe-worldstate-parser types so i can auto-generate types for it

I kind of hold this opinion:

allow usage with typescript
don't lock any usage into either forcing those of use that will probably maintain it (.... me) into using typescript

I'm all for validations that something should work well with typescript, but forcing it isn't great to me?

I'm fine with a warframe.d.ts and just leaving it as JS for the app itself


Service Location US vs EU

ZOHO uses different API endpoints for US and EU customers. Currently, this information is stored in the ApiResource resx (ZsRootEndpoint).
It would be great if there was a possibility to set the URL or to have a switch property on the configuration object.
https://subscriptions.zoho.com/api/v1/
https://subscriptions.zoho.eu/api/v1/

@kruijs - thanks for pointing this out, would have never know about the location specific API base URL
I have commit some changes to accommodate this spec, will take a second look at it and push a NuGet soon.

That's great. Thanks a lot for your efforts.

Pushed option to set API Base URL (do not forget to end with '/'
Release Note
Nuget v0.6.15
Pardon on late response


Map View

I am not sure if Google maps offers this type of api call, I will do more research on this, I know Mapbox does so that will be our last resort.

This is probably going to be a ton of work. Maybe there is an example app out there that I could borrow from?


Extend GWT functionality

Hi!
This project looks interesting! But like other Material Design/Polymer GWT wrappers I've seen, some of the widgets lack GWT functionality. Take a look at GWT's built in TextBox for example. It implements a bunch of interfaces, for example:

HasChangeHandlers
HasText
HasValue<String>
IsEditor<ValueBoxEditor<String>
etc

PaperInput does not implement these interfaces. This makes it harder to work with the widget. For example, if you use a framework such as tessell, it's crucial to implement these interfaces, in order to do proper 2-way bindings.
Are there any plans regarding this?
Thanks!

Actually we have not thought on that.
We have the mechanism to implement  customised behaviours per element, but first we have to consider if there is interest on certain features. Could you enumerate which interfaces would be fundamental for you and in which web components?.
Anyway those customisations only can be perform in Widgets, not Elements.

Well, I would like to see most of the native interfaces being implemented. But there are of course some that are more important than others. If we take TextBox again, for example, the most important ones would be HasValue<String>, HasEnabled and HasChangeHandlers, I think. The value and the enabled status are things that you would most likely bind to a model.
If people would like to use the new GWT Editor framework, I assume the widget would have to implement IsEditor<ValueBoxEditor<String>> as well.
If we look at CheckBox, we would again need the most vital ones, HasValue<Boolean>, HasEnabled and IsEditor<LeafValueEditor<Boolean>> for editors.
But if you take a look at the GWT native class, FocusWidget, that many widgets extend, you will see a lot of interfaces that are very useful.
These are just ideas, but it's much easier to use the widgets if they work like regular GWT widgets.
About it only working on Widgets, yeah, that might be the case. I'm not very familiar of using Elements directly, as mentioned in the "Element vs Widget API" section of the readme.


Animal match

Title and Issue number

Title :
Issue No. :
Code Stack :
Close #193

Checklist:

[x] I have mentioned the issue number in my Pull Request.
[x] I have commented my code, particularly in hard-to-understand areas
[] I have created a helpful and easy to understand README.md
[] I have updated the Index.html file for my contribution

@shraddha761 please address the comments


Make 'use core::fmt' calls consistent

I started taking a look at #2869 and looked for everything that was implementing the Display trait:
impl fmt::Display for _

but found some places where the imports weren't consistent:
impl Display for _

There were only a few instances of the latter, so I went ahead and cleaned those up before starting #2869
I started pulling this thread when I saw the same thing was happening for fmt::Debug and fmt::Formatter so I updated the rest of the use core::fmt::* statements with a few exceptions:

No updates to use core::fmt::* if it was being called from within a function since I felt like the function scope was small enough to not cause confusion
No updates to use core::fmt::{self, Write as _};

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

1 of 4 (25.0%)  changed or added relevant lines in 3 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage remained the same at 83.094%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

bitcoin/src/bip158.rs
0
1
0.0%

bitcoin/src/psbt/mod.rs
1
2
50.0%

io/src/error.rs
0
1
0.0%

Files with Coverage Reduction
New Missed Lines
%

io/src/error.rs
1
23.21%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
19508

Relevant Lines:
23477

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

3 of 6 (50.0%)  changed or added relevant lines in 4 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage remained the same at 83.094%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

bitcoin/src/bip158.rs
0
1
0.0%

bitcoin/src/psbt/mod.rs
1
2
50.0%

io/src/error.rs
0
1
0.0%

Files with Coverage Reduction
New Missed Lines
%

io/src/error.rs
1
23.21%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
19508

Relevant Lines:
23477

💛  - Coveralls

Nice clean up, thanks man!


DSL Autosuggest problem

With fastAdapter being the initial dsl block, using any field named fastAdapter leads to the dsl being suggested first. It may become quite annoying for some users, so perhaps you'd rather make it operator invoke from the FastAdapter class. That way it needs to be capitalized

Sadly is that this does not function:

The user would be required to write:

So I think we should keep the current variant

Maybe others will provide inputs eventually, but I think the autosuggest problem is a pretty big hassle. It comes down to a naming preference. Those who don't name their field fastAdapter won't have a problem.

operator fun <Item : GenericItem> FastAdapter.Companion.invoke(block: (FastAdapter<Item>).() -> Unit): FastAdapter<Item> {
    return fastadapter(block)
}

fun test() {
    FastAdapter<GenericItem> {
        itemAdapter<> {  }
    }
}

It seems to work fine. You can decide whether you want it under companion or as an extension like above


convert and upload image to powervs fails

What happened:

What you expected to happen:
The command exited without complete execution and also did not give any error
docker run -it -e IBMCLOUD_API_KEY=$IBMCLOUD_API_KEY -e  RHEL_SUBSCRIPTION_USERNAME=$RHEL_SUBSCRIPTION_USERNAME -e RHEL_SUBSCRIPTION_PASSWORD=$RHEL_SUBSCRIPTION_PASSWORD -e RHEL_ROOT_PASSWORD=$RHEL_ROOT_PASSWORD quay.io/powercloud/image-upload:0.1 --service-name  powervs-wc  --region us-east --cos-bucket my-cos-bucket --cos-resource-group default --cos-instance-name cc-cos-service --rhel-url  "https://access.cdn.redhat.com/content/origin/files/sha256/0b/0b65cf119a90629ffe8a3d0c88bac12479e1c058c647fd00a8d838b9f65c21f2/rhel-8.4-ppc64le-kvm.qcow2?user=xxxxx&auth=xxx"
Running bellow script:
sh -x ./convert-upload-images-powervs --service-name powervs-wc --region us-east --cos-bucket my-cos-bucket --cos-resource-group default --cos-instance-name cc-cos-service --rhel-url https://access.cdn.redhat.com/content/origin/files/sha256/0b/0b65cf119a90629ffe8a3d0c88bac12479e1c058c647fd00a8d838b9f65c21f2/rhel-8.4-ppc64le-kvm.qcow2?user=xxxx&auth=xxxx
[setup] Running setup command...
[setup_artifacts] Downloading code artifacts master in ./automation
Attempt: 1/5
[setup] SUCCESS: setup command completed!
[setup_ibmcloudcli] Installing the latest version of IBM-Cloud CLI...
Attempt: 1/5
//ibmcloud version 2.6.0+df1953d-2022-03-24T15:00:50+00:00
[setup_poweriaas] Installing power-iaas plugin...
[setup_terraform] Installing the latest version of Terraform...
Attempt: 1/5
Terraform v0.13.6
[setup_occli] Installing the latest version of OpenShift CLI...
Attempt: 1/5
Client Version: 4.9.24
[variables] Trying to login with the provided IBMCLOUD_API_KEY...
Listing services under account CHANDAN CHOPRA's Account as user<EMAIL_ADDRESS>Targeting service crn:v1:bluemix:public:power-iaas:us-east:a/193e2e8a2ae01145918831b64557901f:d63809d3-bb89-48f9-998d-4ac8f0db0451::...
[variables] Gathering information from the selected Service Instance... Please wait
(base) chandanchopra@Chandans-MacBook-Pro OCP_PowerVS %
How to reproduce it (as minimally and precisely as possible):
I just ran above command and I got error
Anything else we need to know?:
Environment:

pvsadm version (use pvsadm version):
Environment - On-Prem or IBM PowerVS:
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Others:

This is running within container image

Can you please mention what exact error you are getting, I don't see any error in the above o/p..

There is no error, the command just ends there

/assign @sudeeshjohn

@sudeeshjohn @Shilpi-Das1 can you please update about this bug?

I made a few changes on local

The centos image which we were using is not working. So I ran with centos-8-stream catalog image. It's working fine but it's size is 120GB. We don't need 250GB now because we have reduced disk size

The default disk size of coreos is changed. Now we need to keep it 16GB instead of 11GB. But it's just for older version since 4.10 onwards ova is available so need of conversion

Image import is failing I had to add access and secret keys for that https://github.com/ppc64le-cloud/pvsadm/issues/275 Waiting for confirmation whether secret and access key is permanent change or not

I see change is merged, if any issues, please reopen it.. -Thanks.


update dagster version

Summary

updating dagster version to 1.1.15 (changelog). Realized I was running a different dagster version in my user code deployment that was causing issues with the dagster version powering the app/daemon running in k8s. Sorry if this PR is incomplete, I just tried to find where plural and helm look for dagster chart versions.
Test Plan

Checklist

[ ]  No images hosted from dockerhub
[ ]  Are dashboards present to understand the health of the application.  There must be at least 1 of these

[ ]  all databases should have dashboards
[ ]  ideally also have at least cpu/mem utilization dashboards for webserver tier of the app
[ ]  you can use plural from-grafana to convert a grafana dashboard found via google to our CRD

[ ]  Are scaling runbooks present

[ ]  all databases must have scaling runbooks
[ ]  you can use the charts in pluralsh/module-library to accelerate this

[ ]  do you need to add config overlays?

[ ]  inputing secrets
[ ]  configuring autoscaling

[ ]  If there’s a web-facing component to the app, we need to support OIDC authentication and setting up private networks if no authentication option is viable
[ ]  All major clouds must be supported

[ ]  Azure
[ ]  AWS
[ ]  GCP

hey @kwigley did you test this already w/ plural link?  It looks good otherwise

just tested with plural link! everything I tested checks out

awesome, we'll get it merged in


Use appropriate overloads (matching lane width) of CanonicalizeToBools

This should fix failures in SIMD.validation

@Cellule I'll keep an eye on the builds and let you know when the tests pass
fyi: @arunetm

Curious, do you know why this was okay in ch.exe, but not in jshost.exe ?

@Cellule there are no simd.js tests (only asm.js ones) for Bool8x16/Bool16x8.AllTrue in ChakraCore's test suite.

@Cellule alright, we're all green here! :-) Could you please help us to merge this?


Add FIPS related configuration data stubs to the default openssl app config file.

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

We have a misunderstanding, it seems, emanating from sudden double use if "default" in configuration terms.
There is such a thing as a "default section", which has nothing to do with the default provider, but is rather anything that comes before the first named section (i.e. the "default section" is usually unnamed in the config file).
Unfortunately, the way you have currently placed the provider related stuff, the following have been moved from the default section to the default provider section:
# Extra OBJECT IDENTIFIER info:
#oid_file		= $ENV::HOME/.oid
oid_section		= new_oids

# To use this configuration file with the "-extfile" option of the
# "openssl x509" utility, name here the section containing the
# X.509v3 extensions to use:
# extensions		=
# (Alternatively, use a configuration file that has only
# X.509v3 extensions in its main [= default] section.)

So, regarding location, the openssl_conf = openssl_init line should indeed be somewhere in the default section (i.e. anywhere before the [ new_oids ] section, but the rest must be placed after the whole default section (i.e. just before [ new_oids ] or anywhere further down the file).

Side note: the default configuration section does have a name.  [ default ].  It should be possible to actually name it, although that's something I've never experimented with...

config.pod seems to list the following..
[openssl_init]
oid_section = oids
providers = providers
alg_section = evp_properties
ssl_conf = ssl_configuration
engines = engines
Should it have the oid_section in here then? (That was where I got the info from)..

This pull request is ready to merge

Merged
e45d943665e806ff49d06cfbdd566a8e2d57d56d Add FIPS related configuration data to the default openssl application configuration file


ModuleNotFoundError: No module named 'app_logging'

I don't know if this error is just for lubuntu, but when running hiddeneye-reborn in the terminal I get the error:
ModuleNotFoundError: No module named 'app_logging'.

Because the Devs are still working on HiddenEye Reborn it actually says as you click on to the GitHub page of this repo in the description that this is not user end ready!


✨ Improve homepage sidebar - Adam's testimonial

Cc:  @jaydenalchin
Hi @JackDevAU
Pain
Since you're working on the sidebar widget, I thought we could make some more improvements to the sidebar
Suggested Solution / Tasks

I don't know who added the CSS to remove margins of H4. Please remove that line of CSS which sets H4 to 0

HELP AND IMPROVE OUR RULES - It currently seems disconnected to Adam's testimonial. Few changes:

We don't want people to contact us about changes anymore... Please update the text and link
From

If you spot a rule that is out of date, please contact us, or if you are cool...

To

If you spot a rule that is out of date, please suggest an edit, or if you are cool...

Increase the quote font size in CSS
From small

       #sidebar .testimonial blockquote {
          border-left: 0;
          font-size: small;
          font-style: italic;
      }

To smaller
   #sidebar .testimonial blockquote {
          border-left: 0;
          font-size: smaller;
          font-style: italic;
      }

Make the heading to be inside the testimonial div

Move Adam photo and name to be below the quotation

Add a closing quote mark at the end of quotation

Add a link to SSW.People from Adam's name - https://www.ssw.com.au/people/adam-cogan/

Expected Result

Thanks!
 Please update the text and link to be  'suggest an edit'

Done
> 3. Decrease the quote font size in CSS

You've stated to increase the font size but then give examples of changing FROM small TO smaller, which is decreasing the font size. To confirm, you would like this  decreased?
> 4. Make the heading to be inside the testimonial div

Done
> 5. Move Adam photo and name to be below the quotation

Done
> 6. Add a closing quote mark at the end of quotation

Done
> 7. Add a link to SSW.People from Adam's name

Done
Here is what it looks like

@tiagov8
CC @tkapa @JackDevAU @jaydenalchin
> 1. `<h4>` margin

We are using a plugin 'gatsby-plugin-postcss' which is bringing in CSS from the various plugins used throughout the solution. There is a plugin that seems to be overriding this value so I cant remove it, but I have enforced the original margin-top/bottom values of 1rem using
  margin-bottom: 1rem !important;
  margin-top: 1rem !important;

It now looks as follows

> 3. Decrease the quote font size in CSS

Just awaiting your confirmation before getting these changes reviewed and merged

Hi @drwharris
Thanks for all those changes!

Decrease the quote font size in CSS
Just awaiting your confirmation on if this should be an increase or decrease before getting these changes reviewed and merged

You may disregard this one - keep as "small" In my inspector it was increasing a little bit, but I think this depends on browser settings.

Hi @tiagov8
No problem, I have reverted that one. Here is what it looks like

I will send this for review.


Dimensioned.clone always transfers id to new object

This PR will make sure styles are transferred when applying various operations to an object including slices. This will probably affect some existing tests but it is better behavior.

Ok, the tests are passing and I do think this is a good change. Merging.


bug: unable to use million/jsx-runtime with next.js

Describe the bug
I'm trying to use million on Next.js using babel automatic runtime but i'm not able to get the next dev server to start.
Getting the following error:

To Reproduce
CodeSandbox: https://codesandbox.io/s/next-million-js-o4jlki?file=/.babelrc
Expected behavior
Next.js should be able to run...
Device (please complete the following information):

Node 16
React 18.2
Next latest

Hi@DaniAkash, Next.js is currently not compatible with Million.js. We are planning to work on a next plugin to allow for this in the future.

That would be awesome! If you can share more details... I'm also happy to contribute 👍

Sure! We're currently working towards some compatability with routers currently in order to support Next.js router in #203. Next steps are hydration, prerendering, etc. Feel free to join our Discord to discuss further


Double numbers in Y-axis labels in bar chart

When the highest value in my series is less than the number of horizontal grid lines I get double numbers. As shown here in the image. How can I avoid or fix this?

Can you show us your configuration?

 var dfaultChartOpts = {
            axisX: {
                showGrid: false
            },
            axisY: {
                labelOffset: {
                    x: 0,
                    y: 0
                },
                onlyInteger: false
            }
        };

I could never reproduce this issue. Please reopen if needed.


Unable to leave Group

User was unable to leave the Seeds group, logic for this function is probably not working properly.

Fixed in latest commit.


Standard layout type possibly supurfluous requirement.

Standard layout types
...

has no base classes of the same type as the first non-static data member. (1)

meets one of these conditions: (2)

no non-static data member in the most-derived class and no more than one base class with non-static data members, or

has no base classes with non-static data members

Doesn't condition labeled (2) being satisfied imply condition labeled (1) to be satisfied as well?
If not, it would help to add an example that meets condition labeled (2) but not condition label (1).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 249da7e2-4135-6468-221b-d62df3af0f22
Version Independent ID: 2e68cba5-e743-2d88-7b4c-7bb6f79c54cd
Content: Trivial, standard-layout, POD, and literal types
Content Source: docs/cpp/trivial-standard-layout-and-pod-types.md
Product: visual-cpp
Technology: cpp-language
GitHub Login: @corob-msft
Microsoft Alias: corob

Was able to create an example that shows the two requirements are independent.
# include <iostream>
# include <type_traits>

struct Base
{
   void Foo() {}
};

struct Derived : public Base
{
   Base b;
   int x;
};

int main(){
    std::cout << std::boolalpha << std::is_standard_layout_v<Derived>;
}

Closing the issue. It still might be worth it to add it into the document for clarity.


Add missing definition of cop to docs

Add a missing definition of cop to the getting started part of the
documentation.

[x] Closes #994
[x] Fully documented, including history.rst for all changes and api/*.rst for new API

Thanks @gerritholl 👍


Add kodi unique id based on discovery

Description:
Adds unique ID to avoid duplicate kodi players, which happens when netdisco decides to pick an advertisement from a different network interface.
Depends on https://github.com/xbmc/xbmc/pull/14089 .
Related issue (if applicable): fixes #
Pull request in home-assistant.github.io with documentation (if applicable): home-assistant/home-assistant.github.io#<home-assistant.github.io PR number goes here>
Example entry for configuration.yaml (if applicable):

Checklist:

[ ] The code change is tested and works locally.
[ ] Local tests pass with tox. Your PR cannot be merged unless tests pass

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.github.io

If the code communicates with devices, web services, or third-party tools:

[ ] New dependencies have been added to the REQUIREMENTS variable (example).
[ ] New dependencies are only imported inside functions that use them (example).
[ ] New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[ ] New files were added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

Can you keep track of unique ids of entities and check that before trying to add the same device again. If we need to add a new entity for the new ip address, the old corresponding entity should first be removed.

Fair enough, thanks for the feedback and merge! :+1:


ID - Send Errors from 5/4

DevOps notified that there have been failures communicating with receivers due to send errors. ID errors noted below. Need to resubmit failed files to ID.
[1;31m23:07:13.145 [pool-2-thread-7252][66033] FATAL-ALERT gov.cdc.prime.router.azure.SendFunction - FAILED Sftp upload of inputReportId e2e8132d-54b1-4af8-af5f-6a672011dd42 to SFTPTransportType(host=sftp.dhw.idaho.gov, port=22, filePath=./ELR/SimpleReport/Prod, credentialName=null) (orgService = id-phd.elr), Exception: Exhausted available authentication methods, All retries failed.  Manual Intervention Required.  Send Error report for: e2e8132d-54b1-4af8-af5f-6a672011dd42 to id-phd.elr[m
[1;31m11:08:04.439 [pool-2-thread-7931][88989] FATAL-ALERT gov.cdc.prime.router.azure.SendFunction - FAILED Sftp upload of inputReportId 85d50d76-ff37-4e1d-a9fe-1600cc405e67 to SFTPTransportType(host=sftp.dhw.idaho.gov, port=22, filePath=./ELR/SimpleReport/Prod, credentialName=null) (orgService = id-phd.elr), Exception: Exhausted available authentication methods, All retries failed.  Manual Intervention Required.  Send Error report for: 85d50d76-ff37-4e1d-a9fe-1600cc405e67 to id-phd.elr[m

The first failure for ID was noted on 5/2: https://prime.cdc.gov/metabase/question/91-last-mile-problems-in-the-last-3-days

The two files were uploaded to Idaho's server.

5/4: Notified ID about file upload to correct the send errors.


Overlapping text, odd wrapping, and odd breaking

https://ncatlab.org/nlab/show/!-modality
Trying to print this page using html2pdf. I can get it to print but the ! is printed about 30px to the right on top of modality. The wrapping is all off as if the margins are not being correctly used for wrapping. Page breaking is done in the middle of lines. Extra blank page.
my code is essentially
var printPage = function(name)
{

    /*
    var opt = {
        margin:       1,
        filename:     name,
        image:        { type: 'jpeg', quality: 0.98 },
        jsPDF:        { unit: 'in', format: 'letter', orientation: 'portrait', floatPrecision:'smart' },
        pagebreak:    { mode: 'avoid-all', after: '#Content' },
        html2canvas:  { allowTaint: true, scale: 1, logging: true, dpi: 300, letterRendering: true, scrollY: -window.scrollY, scrollX: -window.scrollX},
    };*/

    var opt = {
        margin:       1,
        filename:     name,
        html2canvas: { scale: 1, letterRendering: true },
        //image:        { type: 'jpeg', quality: 0.98 },
        //jsPDF:        { unit: 'in', format: 'letter', orientation: 'portrait', floatPrecision:'smart' },
        //pagebreak:    { mode: 'avoid-all', after: '#Content' },
        //html2canvas:  { allowTaint: true, scale: 1, logging: true, dpi: 300, letterRendering: true, scrollY: -window.scrollY, scrollX: -window.scrollX},
    };

    html2pdf().set(opt).from(document.getElementById('Content')).save().then(() => {
                    });
}

printPage(name);

I've been trying various options and things others have done but nothing fixed it. Seems to be a bug in html2pdf and/or html2canvas.

https://github.com/niklasvh/html2canvas/issues/2775#issuecomment-1204988157
Try this


Bitbucket Support

Can this tool also support Bitbucket?

Should be possible if Bitbucket has an API, I personally haven't worked with Bitbucket yet but I will take a look. It will take some time to get familiar with bitbucket and it would also require some refactoring of the code

@wernermorgenstern I have completed the initial draft, I need to a lot of refactoring to make this work, I could not re-use the existing bitbucket client and had to write the client on my own. I need to write more tests and update some more documentation, im gonna release within 10 days, but meanwhile you can already test when you build locally on branch. Feedback appreciated
export BITBUCKET_TOKEN="t****"
export BITBUCKET_USER="roflmoflcopter"
export REPO_NAME="cdk-notifier-test"
export REPO_OWNER="roflmoflcopter"
export BITBUCKET_PR_ID=1

git checkout support-bitbucket
go build && ./cdk-notifier -l data/cdk-small.log --vcs bitbucket --tag-id next

> INFO[0000] Created comment with id 308183615 and tag id next https://bitbucket.org/roflmoflcopter/cdk-notifier-test/pull-requests/1/_/diff#comment-308183615

:tada: This issue has been resolved in version 2.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Figure out what the 'e' format string actually means

Dataflash has a variety of format strings, some of which are deprecated. I do not know which ones. There is no documentation, except a cryptic forum response saying that dataflash is 'self documenting' good luck.

It is an int32. There were several format strings with similar errors.


Requirement for safety of cache_hash is stronger than documented

The documentation states:

cache_hash (bool) – Ensure that the object’s hash code is computed only once and stored on the object. If this is set to True, hashing must be either explicitly or implicitly enabled for this class. If the hash code is cached, then no attributes of this class which participate in hash code computation may be mutated after object creation.

A reader could read this as meaning that only the local fields of this object should not be assigned to and that using this on any frozen class is okay.  We should clarify that the object needs to be "deeply immutable" for this to be safe.

PR please? 😇

@hynek : Sure, I'll take care of this one sometime soon.


[WIP] Added support for arm service connection for KubernetesV0

Task name: KubernetesV0
Description:  Added support for arm service connection
Documentation changes required: Y
Added unit tests: N
Attached related issue: N
Checklist:

[X] Task version was bumped - please check instruction how to do it
[ ] Checked that applied changes work as expected

CODEOWNERS file should be updated accordingly if we decide to go with new version instead of modifying existing one.


Update Renovate config

Proposed changes
This updates the Renovate config to create better PR titles and provide better granularity when dependencies are updated.
Types of changes
What types of changes does your code introduce?
Put an x in the boxes that apply

[ ] CI/CD update (an update to the CI/CD workflows, scripts, and/or configurations)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[ ] I have followed the guidelines in the CONTRIBUTING document
[ ] I have signed the CLA
[ ] I have checked to ensure there aren't other open Pull Requests for the same update/change
[ ] I have created (or updated) an Issue to track the status of this update/change and updated the link in this PR description (see above in the Proposed changes section) using the wording Addresses #<issue_number>
[ ] I have performed a self-review of my code
[ ] My code follows the style guidelines of this project
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] Basic linting passes locally with my changes
[ ] I have added necessary documentation (if appropriate)
[ ] I have updated the Changelog with a brief description of my changes

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: nfelt14:x: renovate[bot]You have signed the CLA already but the status is still pending? Let us recheck it.

A lot of the version downgrades are from testing how renovate will update them


Feature Request: Google Drive Picker API for Angular5

I put in a feature request at Firebase support, but support told me to come here.
Could you direct me to the right person for this request?
I am very grateful for AngularFire2! I've been trying to get the Google Picker API for Drive working without luck.
Could Picker API be supported too?

Joe Leiman

I think that's out of scope for AF2, it could be a feature request for the JS SDK in general though. https://github.com/firebase/firebase-js-sdk


NIFI-5287 Made LookupRecord able to take in flowfile attributes and c…

…ombine them with lookup keys.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@ijokarumawak Review?

@MikeThomsen @ijokarumawak Hey guys, I'm sorry to chime in a bit late here. I don't think this is really the approach that we want to take here, though. What our interface provides currently is a mechanism to pass in the 'coordinates', i.e., the location for finding the data that we want. The approach here takes this concepts and muddies it by providing arbitrary key/value pairs that don't make sense for the lookup.
I would propose an alternative approach, which would be to add a new method to the interface that has a default implementation:
default Optional<T> lookup(Map<String, Object> coordinates, Map<String, String> context) throws LookupFailureException {
    return lookup(coordinates);
}

Where context is used for the FlowFile attributes (I'm referring to it as context instead of attributes because there may well be a case where we want to provide some other value that is not specifically a FlowFile attribute). Here is why I am suggesting this:

It provides a clean interface that properly separates the data's coordinates from FlowFile attributes.
It prevents any collisions between FlowFile attribute names and coordinates.
It maintains backward compatibility, and we know that it won't change the behavior of existing services or processors/components using those services - even those that may have been implemented by others outside of the Apache realm.
If attributes are passed in by a Processor, those attributes will be ignored anyway unless the Controller Service is specifically updated to make use of those attributes, such as via Expression Language. In such a case, the Controller Service can simply be updated at that time to make use of the new method instead of the existing method.

@markap14 @ijokarumawak updated based on the last comment.

Thanks @markap14 for pointing that. Separating the coordinate and context make sense and implementations will be cleaner by doing so. Although I agree with the idea, let me try to explain why I wanted to put everything into the same map.
I wanted to muddle all variables into the same map because some lookup target, such as URL for the RestLookupService have both lookup coordinate and environment depending part that can be passed as FlowFile attributes or variable registry.
For example, let's say we want to make following URL to be populated by a Expression Language to be used by RestLookupService. http://test.example.com:8080/service/john.smith/friend/12345
I think the URL has both lookup and contextual part in it.

john.smith and 12345` are lookup coordinates that vary from input, in LookupRecord context, it varies per recocrd
But hostname and port is more of contextual values, that varies per FlowFile or environment

If the URL property is configured as http://${apiHost}:${apiPort}/service/${userName}/friend/${friendId}, and if it can only refer lookup coordinate, then apiHost and apiPort need to be set for each record lookup. And to do so, user need to configure dynamic properties at LookupRecord processor using record paths, which can be awkward since RecordPathValidator doesn't allow literal String value. A work-around is to use concat('test.example.com') to return the constant value for every record lookup.
To make this scenario more flexible while meeting with the idea of separating lookup coordinates and context, I wrote MikeThomsen/nifi#1  so that target URL can be configured by 2 properties (if necessary). BASE_URL can use Variable Registry, and URL can use coordinate.
If we pursue the path to separate lookup and coordinate more implementations like that will be needed in different places. That has both pros / cons I think.
On the other hand, the variables those can be referred by an EL is already muddled a lot IMHO. It contains System properties, Variable registry, and FlowFile attribute values and EL can utilize those without knowing where it's configured.
Adding those values into lookup coordinate may not sound that wrong, if we keep the consistent overlaying order.

@ijokarumawak

Adding those values into lookup coordinate may not sound that wrong, if we keep the consistent overlaying order.

It is, however, dangerous to do with LookupService implementations that are based around query builders like the Mongo one. The Mongo one does a straight conversion of the coordinate map into a Mongo query, so if you add any value other than a key the user specifies you will break the query in most cases.

Maybe there should be a Context interface, and there can be support for implementations that support more than one map or type of backing.  I think the limitation here is using a literal Map instead of a logical construct.
This is similar to variable resolution in a custom language.  You may  need more than a map.
LookupContext context = new AttributeAndCoordinateLookup(attrs, coords);

@ottobackwards I think that's overkill because a Map is fine for the main variables and a Map also works for passing EL output. A major advantage of using two maps is that the "context" one is not set in stone as @markap14 pointed out and can change as our use changes without breaking much.

It may be overkill, that is true.  But if you have to keep adding new functions to account for different scenarios that isn't great either and may suggest something like that would be good to have. Having a context or resolver for this type of thing isn't that radical. Having a context sets the interface, that is true, but the implementation can be any kind of policy/strategy/composition you may require.
That being said, just a thought anyways.

In hindsight, a "Context" type of object would have been a good idea in this situation, had it been done that way in the first place. However, while it's a helpful design pattern, we have to be cognizant of the fact that we always have performed a release that contains the Service API the way that it is. We need to avoid changing that, as any custom Controller Services would break if we change the API in a non-backward-compatible way. It would also break any custom processors that try to use a Lookup Service.
While we do have some flexibility in changing Service API's after they have been released, it is kind of a nuclear option that should be avoided if we can. Especially for something that is as widely used as the lookup stuff.
This is largely why I want to avoid also including attributes in the same map - it would potentially break existing services if new things suddenly are fed into that map that are not part of the lookup's coordinates. By introducing the Map<String, String> context and providing a default implementation in the interface, all existing services will work as they previously did, and all existing processors will work as the previously did. However, new or updated implementations will still be able to take advantage of FlowFile Attributes.

@markap14 I agree with that. We shouldn't break existing stuffs. I've been thinking about this after I posted my last comment and now I believe having separate map is a better idea. It's possible to mix those to evaluate an EL with both values if new implementation really needs so, while keeping existing ones work as they are.
Sorry for making a confusion.
@MikeThomsen now we can remove Attributes Regular Expression from LookupRecord, and just pass all FlowFile attribute as context as you did with LookupAttribute.
In addition to that, there is one more things that I want to clarify on this improvement.
Do we also want to access Variable Registry values through the context map? If so, how can we do that?

@markap14 that certainly makes sense, thanks for taking the time to respond

As for Variable Registry, I think passing only FlowFile attribute is enough, same as the way AbstractRecordProcessor is doing right now. https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractRecordProcessor.java#L113
In that sense, I found a contradiction between current documentation and behavior. Schema Name, Schema Version, Schema Branch and Schema Text are documented that those supports Variable Registry and FlowFile Attributes, but it can only access to FlowFile attributes. I assume this pattern is not considered when ExpressionLanguageScope is added. Probably for another JIRA (not filed yet).

@ijokarumawak whenever Expression Language is evaluated from a PropertyDescriptor, it always has access to the Variable Registry. The framework provides the variable registry to the compiled expression when it is evaluated.

@markap14 @ijokarumawak cleaned it up by removing that attribute and it's rebased against master (as of my last pull this morning). Can one of you review?

Thanks @markap14 , that is true. My previous comment on the difference between doc and actual behavior was a false alarm.
I expected that but got different result, so I got confused. But now I understand what happened in my test. A LookupService was in a child group, using a RecordReader defined in a parent group. And I set a variable in the child group. But the RecordReader in the parent group was not able to use it. Configuring variable at the parent group worked as expected.

Parent PG

RecordReader: Schema Name=${schema.name}
Variable registry: schema.name=parent (this value can be used)
Child PG

LookupService using the RecordReader in the parent PG
Variable registry: schema.name=child (the RecordReader doesn't have access to variables defined in child PG)

Also, I found that how SchemaAccessStrategy classes caches PropertyValue to evaluate EL at runtime. @MikeThomsen RestLookupService PR #2723 needs to be updated to follow this pattern when it evaluates EL. I will comment on that PR for detail.

Thanks @MikeThomsen LGTM +1, merging.


REQUEST - Support for more UK only sites

I am very interested in using this tool however I'm concerned about its ability to work with some UK only sites as I saw there were issues with UK amazon and ebuyer.
Native support for:

currys.co.uk
box.co.uk
https://www.overclockers.co.uk/
etc.
This would possibly be useful

@jamesearls This is something I'm working on. I will let you know when this feature is ready.

Appreciate that man! Right now I'm currently trying to develop a fix for amazon UK on a forked version of the repo.

A lot of UK sites like Scan, Overclockers, Aria PC and Novatech are already working with the generic scraper. I've made a list of config files (#113) you might find useful. They helped me get a 6800 XT from scan this afternoon. Best of luck!