Benchmarks keep on failing - problem with python scripts

For the last few days I've been trying to get PerfkitBenchmarker working with Azure and AWS. I followed all the instructions that are provided here and on the official website. I have successfully installed everything needed, also got the accounts for Azure and AWS. The problem arises when I am trying to benchmark anything, i.e. while executing
./pkb.py --cloud=Azure --benchmark=iperf ...
Everytime I execute a benchmark my tests are failing while calling certain Python functions like (see attached pictures):

PrepareVm() - I get an error stating: Exception occurred while calling PrepareVM(...)
DeleteVm(), same thing
Another error that I am getting during the execution of iperf test is simply: Error during benchmark iperf.

Then, when the test finishes I get that the benchmark iperf failed, success rate is 0.0%. I also tried different test, same issue. And same issue for Azure and AWS.
I am using Python 2.7.3, Azure client 0.9.9, latest Perfkit (1.3.0).
If anyone knows what might be wrong I would very much appreciate it. I really need to get this working somehow.
I also attached images of the error. It follows from top to bottom, showing the execution of iperf test.

Hi @Eldainorn , your PKB command line looks good. As far as I can tell from your screenshots, the last PKB function in the stack trace for prepare-phase errors is always _GetDefaultImagesFromAzure(). Is that correct? (I would guess that the errors in DeleteVm() are because PrepareVM() didn't manage to create a VM, so they will go away once we fix the prepare step.)
Could you please run azure vm image list --json at your command prompt and paste the results here? That's the command that PKB is running that it can't interpret the output of. Thanks!

It looks like you are hitting and Azure quota issue.
Have you been able to create a VM with using the azure command like the following?
azure vm create --vm-size=Medium pkb-44ed3a3a-0 b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_4-LTS-amd64-server-20160314-en-us-30GB  --location='West US' --userName voellm
happy to help debug more...

Thank you for such a quick answer.
@noahl _GetDefaultImagesFromAzure() is indeed the last function, as we can see from the images I provided. I also think that you are right that DeleteVm() is failing because PrepareVm() failes beforehand.
I ran the azure vm image list --json command and what I find weird is that I do not get complete output. I checked if there are some other options for printing the output but it seems that there none that would help me (only verbose option). Here is the output I get (it's quite a lot):
[
{
"category": "Public",
"label": "CoreCLR x64 Beta5 (11624) with PartsUnlimited Demo App on Ubuntu Server 14.04 LTS",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Japan East;Japan West;Central US;East US;South Central US",
"logicalSizeInGB": 30,
"name": "03f55de797f546a1b29d1b8d66be687a__CoreCLR-x64-Beta5-Linux-PartsUnlimited-Demo-App-201504.29",
"operatingSystemType": "Linux",
"eula": "http://go.microsoft.com/fwlink/?LinkId=521895",
"description": "Linux VM image with coreclr-x64-beta5-11624 installed to /opt/dnx. This image is based on Ubuntu 14.04 LTS, with prerequisites of CoreCLR installed. It also contains PartsUnlimited demo app which runs on the installed coreclr. The demo app is installed to /opt/demo. To run the demo, please type the command '/opt/demo/Kestrel' in a terminal window. The website is listening on port 5004. Please enable or map a endpoint of HTTP port 5004 for your azure VM.",
"imageFamily": "CoreCLR x64 Beta5 (11624) with PartsUnlimited Demo App on Ubuntu Server 14.04 LTS",
"publishedDate": "2015-04-29T07:00:00.000Z",
"isPremium": false,
"privacyUri": "http://go.microsoft.com/fwlink/?LinkID=528096",
"recommendedVMSize": "Standard_D1",
"publisherName": "Microsoft Visual Studio Group",
"iconUri": "Ubuntu-cof-45.png",
"smallIconUri": "Ubuntu-cof-45.png",
"iOType": "Standard_LRS"
},
{
"category": "Public",
"label": "Team Foundation Server 2013 Update 4 on Windows Server 2012 R2",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Japan East;Japan West;Central US;East US;South Central US",
"logicalSizeInGB": 128,
"name": "03f55de797f546a1b29d1b8d66be687a__Team-Foundation-Server-2013-Update4-WS2012R2",
"operatingSystemType": "Windows",
"eula": "http://www.microsoft.com/en-us/download/details.aspx?id=13350",
"description": "Microsoft Team Foundation Server 2013 Trial on Windows Server 2012 R2 Update. Virtual Machines created with this trial image will require a product key for Team Foundation Server (such as from an MSDN Subscription). This image includes a complete installation of Team Foundation Server 2013 Update 4. Some components require additional setup and configuration. You can configure SQL Server using SQL Server Express included in this image, by downloading and installing SQL Server Standard edition (from an MSDN Subscription), or by connect to a pre-existing SQL Server. Minimum virtual machine size for this image is Medium. For more details on TFS server setup please see the [Team Foundation Server install guide|http://msdn.microsoft.com/en-us/library/dd631902.aspx].",
"imageFamily": "Team Foundation Server 2013 Update 4 on Windows Server 2012 R2",
"publishedDate": "2014-11-12T08:00:00.000Z",
"isPremium": false,
"privacyUri": "http://go.microsoft.com/fwlink/?LinkID=286720",
"recommendedVMSize": "Medium",
"publisherName": "Microsoft Visual Studio Group",
"iconUri": "VisualStudio2013_100.png",
"smallIconUri": "VisualStudio2013_45.png",
"iOType": "Standard_LRS"
},
{
"category": "Public",
"label": "Team Foundation Server 2013 Update 4 on Windows Server 2012 R2",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Japan East;Japan West;Central US;East US;South Central US",
"logicalSizeInGB": 128,
"name": "03f55de797f546a1b29d1b8d66be687a__Team-Foundation-Server-2013-Update4-WS2012R2-201503.20",
"operatingSystemType": "Windows",
"eula": "http://www.microsoft.com/en-us/download/details.aspx?id=13350",
"description": "Microsoft Team Foundation Server 2013 Trial on Windows Server 2012 R2 Update. Virtual Machines created with this trial image will require a product key for Team Foundation Server (such as from an MSDN Subscription). This image includes a complete installation of Team Foundation Server 2013 Update 4. Some components require additional setup and configuration. You can configure SQL Server using SQL Server Express included in this image, by downloading and installing SQL Server Standard edition (from an MSDN Subscription), or by connect to a pre-existing SQL Server. Minimum virtual machine size for this image is Medium. For more details on TFS server setup please see the [Team Foundation Server install guide|http://msdn.microsoft.com/en-us/library/dd631902.aspx].",
"imageFamily": "Team Foundation Server 2013 Update 4 on Windows Server 2012 R2",
"publishedDate": "2015-03-20T07:00:00.000Z",
"isPremium": false,
"privacyUri": "http://go.microsoft.com/fwlink/?LinkID=286720",
"recommendedVMSize": "Medium",
"publisherName": "Microsoft Visual Studio Group",
"iconUri": "VisualStudio2013_100.png",
"smallIconUri": "VisualStudio2013_45.png",
"iOType": "Standard_LRS"
},
{
"category": "Public",
"label": "Team Foundation Server 2013 Update 4 on Windows Server 2012 R2",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Japan East;Japan West;Central US;East US;South Central US",
"logicalSizeInGB": 128,
"name": "03f55de797f546a1b29d1b8d66be687a__Team-Foundation-Server-2013-Update4-WS2012R2-201503.27",
"operatingSystemType": "Windows",
"eula": "http://www.microsoft.com/en-us/legal/IntellectualProperty/UseTerms/Default.aspx",
"description": "Microsoft Team Foundation Server 2013 Trial on Windows Server 2012 R2 Update. Virtual Machines created with this trial image will require a product key for Team Foundation Server (such as from an MSDN Subscription). This image includes a complete installation of Team Foundation Server 2013 Update 4. Some components require additional setup and configuration. You can configure SQL Server using SQL Server Express included in this image, by downloading and installing SQL Server Standard edition (from an MSDN Subscription), or by connect to a pre-existing SQL Server. Minimum virtual machine size for this image is Medium. For more details on TFS server setup please see the [Team Foundation Server install guide|http://msdn.microsoft.com/en-us/library/dd631902.aspx].",
"imageFamily": "Team Foundation Server 2013 Update 4 on Windows Server 2012 R2",
"publishedDate": "2015-03-27T07:00:00.000Z",
"isPremium": false,
"privacyUri": "http://go.microsoft.com/fwlink/?LinkID=286720",
"recommendedVMSize": "Medium",
"publisherName": "Microsoft Visual Studio Group",
"iconUri": "VisualStudio2013_100.png",
"smallIconUri": "VisualStudio2013_45.png",
"iOType": "Standard_LRS"
},
{
"category": "Public",
"label": "Visual Studio Community 2013 Update 5 with Azure 2.8 on Windows Server 2012 R2",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Central India;South India;West India;Japan East;Japan West;Central US;East US;South Central US",
"logicalSizeInGB": 128,
"name": "03f55de797f546a1b29d1b8d66be687a__VS-2013-Comm-VSU5-AzureSDK-2.8-WS2012R2",
"operatingSystemType": "Windows",
"eula": "http://go.microsoft.com/fwlink/?LinkID=430755",
"description": "This image of Visual Studio Community 2013 Update 5 with Azure Tools 2.8 enables you to unleash the full power of Visual Studio to develop cross-platform solutions.  Create apps in one unified IDE, and incorporate new languages, features, and development tools into them with Visual Studio Extensions (available in the Visual Studio Gallery).",
"imageFamily": "Visual Studio Community 2013 Update 5 with Azure 2.8 on Windows Server 2012 R2",
"publishedDate": "2016-01-05T08:00:00.000Z",
"isPremium": false,
"privacyUri": "http://go.microsoft.com/fwlink/?LinkID=286720",
"recommendedVMSize": "Standard_D2",
"publisherName": "Microsoft Visual Studio Group",
"iconUri": "VisualStudio2013_100.png",
"smallIconUri": "VisualStudio2013_45.png",
"iOType": "Standard_LRS"
},
{
"category": "Public",
"label": "Visual Studio Community 2013 Update 5 with Azure 2.8 on Windows Server 2012 R2",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Central India;South India;West India;Japan East;Japan West;Central US;East US;South Central US",
"logicalSizeInGB": 128,
"name": "03f55de797f546a1b29d1b8d66be687a__VS-2013-Comm-VSU5-AzureSDK-28-WS2012R22016-03-03",
"operatingSystemType": "Windows",
"eula": "http://go.microsoft.com/fwlink/?LinkId=430755",
"description": "This image of Visual Studio Community 2013 Update 5 with Azure Tools 2.8 enables you to unleash the full power of Visual Studio to develop cross-platform solutions.  Create apps in one unified IDE, and incorporate new languages, features, and development tools into them with Visual Studio Extensions (available in the Visual Studio Gallery).",
"imageFamily": "Visual Studio Community 2013 Update 5 with Azure 2.8 on Windows Server 2012 R2",
"publishedDate": "2016-03-03T08:00:00.000Z",
"isPremium": false,
"privacyUri": "http://go.microsoft.com/fwlink/?LinkId=286720",
"recommendedVMSize": "Standard_D2",
"publisherName": "Microsoft Visual Studio Group",
"iconUri": "VisualStudio2013_100.png",
"smallIconUri": "VisualStudio2013_45.png",
"iOType": "Standard_LRS"
},
{
"category": "Public",
"label": "Visual Studio Community 2013 Update 5 with Tools for Apache Cordova CTP 3.2 on Windows Server 2012 R2",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Central India;South India;West India;Japan East;Japan West;Central US;East US;South Central US",
"logicalSizeInGB": 128,
"name": "03f55de797f546a1b29d1b8d66be687a__VS-2013-Comm-VSU5-Cordova-CTP3.2-AzureSDK-2.8-WS2012R2",
"operatingSystemType": "Windows",
"eula": "http://go.microsoft.com/fwlink/?LinkID=430755",
"description": "This image provides Visual Studio Community 2013 Update 5, Azure Tools 2.8, and CTP 3.2 of the Tools for Apache Cordova running on Windows Server 2012 R2.  It allows you to easily and quickly set up a development environment in Azure to build and test Android, iOS, and Windows apps using HTML, CSS, and JavaScript. Please see [http://go.microsoft.com/fwlink/?LinkID=397716|http://go.microsoft.com/fwlink/?LinkID=397716] for more information.",
"imageFamily": "Visual Studio Community 2013 Update 5 with Tools for Apache Cordova CTP 3.2 on Windows Server 2012 R2",
"publishedDate": "2016-01-15T08:00:00.000Z",
"isPremium": false,
"privacyUri": "http://go.microsoft.com/fwlink/?LinkId=286720",
"recommendedVMSize": "Medium",
"publisherName": "Microsoft Visual Studio Group",
"iconUri": "VisualStudio2013_100.png",
"smallIconUri": "VisualStudio2013_45.png",
"iOType": "Standard_LRS"
},
{
"category": "Public",
"label": "Visual Studio Premium 2013 Update 5 on Windows Server 2012",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Central India;South India;West India;Japan East;Japan West;Central US;East US;South Central US",
"logicalSizeInGB": 128,
"name": "03f55de797f546a1b29d1b8d66be687a__VS-2013-Prem-VSU5-AzureSDK-2.8-WS2012R2-2",
"operatingSystemType": "Windows",
"eula": "http://msdn.microsoft.com/subscriptions/eula",
"description": "The Visual Studio Premium 2013 Update 5 developer desktop is an offering exclusive to MSDN subscribers. The image includes Visual Studio Premium 2013 Update 5, SharePoint 2013 Trial, SQL Server 2012 Developer edition and configuration scripts to quickly create a development environment for Web, SQL and SharePoint 2013 development. To learn how to configure any development environment you can follow the links on the desktop. We recommend a Standard_D2 VM size for SQL and Web development and Standard_D3 VM size for SharePoint development. Please see [http://go.microsoft.com/fwlink/?LinkID=329862|http://go.microsoft.com/fwlink/?LinkID=329862] for a detailed description of the image. Privacy note: This image has been preconfigured for Microsoft Azure, including enabling the Visual Studio Experience Improvement Program for Visual Studio, which can be disabled.",
"imageFamily": "Visual Studio Premium 2013 Update 5 on Windows Server 2012",
"publishedDate": "2016-01-29T08:00:00.000Z",
"isPremium": false,
"privacyUri": "http://go.microsoft.com/fwlink/?LinkID=286720",
"recommendedVMSize": "Standard_D2",
"publisherName": "Microsoft Visual Studio Group",
"iOType": "Standard_LRS"
},
{
"category": "Public",
"label": "Visual Studio Ultimate 2013 Update 5 on Windows Server 2012",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Central India;South India;West India;Japan East;Japan West;Central US;East US;South Central US",
"logicalSizeInGB": 128,
"name": "03f55de797f546a1b29d1b8d66be687a__VS-2013-Ultimate-VSU5-AzureSDK-2.8-WS2012R2-2",
"operatingSystemType": "Windows",
"eula": "http://msdn.microsoft.com/subscriptions/eula",
"description": "The Visual Studio Ultimate 2013 Update 5 developer desktop is an offering exclusive to MSDN subscribers. The image includes Visual Studio Ultimate 2013 Update 5, SharePoint 2013 Trial, SQL Server 2012 Developer edition and configuration scripts to quickly create a development environment for Web, SQL and SharePoint 2013 development. To learn how to configure any development environment you can follow the links on the desktop. We recommend a Standard_D2 VM size for SQL and Web development and Standard A4 VM size for SharePoint development. Please see [http://go.microsoft.com/fwlink/?LinkID=329862|http://go.microsoft.com/fwlink/?LinkID=329862] for a detailed description of the image. Privacy note: This image has been preconfigured for Microsoft Azure, including enabling the Visual Studio Experience Improvement Program for Visual Studio, which can be disabled.",
"imageFamily": "Visual Studio Ultimate 2013 Update 5 on Windows Server 2012",
"publishedDate": "2016-01-29T08:00:00.000Z",
"isPremium": false,
"privacyUri": "http://go.microsoft.com/fwlink/?LinkID=286720",
"recommendedVMSize": "Standard_D2",
"publisherName": "Microsoft Visual Studio Group",
"iOType": "Standard_LRS"
},
{
"category": "Public",
"label": "Visual Studio Community 2015 with Azure SDK 2.8 on Windows Server 2012 R2",
"location": "Southeast Asia;Australia East;Australia Southeast;North Europe;West Europe;Central India;South India;West India;Japan
@voellm
I tried the command you provided but it doesn't work for me. Here is what I get:

I then changed the location to --location="East US"  and then I got different error that stated: error: The specified DNS name is already taken and at the end command failed.
I asked my friend to try to run the tests for AWS and he managed to get it to work without any problems. Maybe I'll try to reinstall everything (Python, pip, PerfKit, clients, ...). Could there be incompatibility between different versions of PerfKit/Python/... ?

@Eldainorn
I ran into this recently--with my Azure account, I'm not able to create VMs in West US, but I can in East US.
That is, when I do:
azure vm create --location='West US' testname 6a83c2d016534a7a917bcd21b6e1c0c9__Debian-9-DAILY-amd64-20160318.0  --userName=test
I get the same error you get above.   On the other hand, the following command runs successfully:
azure vm create --location='East US' hildrum-test 6a83c2d016534a7a917bcd21b6e1c0c9__Debian-9-DAILY-amd64-20160318.0  --userName=test
Try:
./pkb.py --benchmarks=iperf --cloud=Azure --zone='East US'

Truncated azure vm image list --json looks like a dup of #860, which should have been fixed by #878. Give me a little while to check if that's in the version of PKB you're using.

Looks like #878 is not in v1.3.0. This issue is already fixed in master and will presumably be in v1.4.0. You might be able to get around the specific failing command by explicitly picking an image via the --image flag, but your current environment may truncate the stdout and stderr of other commands. I recommend checking out a copy of the master branch.

@hildrum, @skschneider  I am still getting the same problem. Thx anyways. It seems that there is some problem with the Azure. My friend could run the tests for AWS and DigitalOcean but he also got the same problems as I did when he was running tests for the Azure. I will try it with the Benchmarker 1.4.0 when it releases. Or we will just change the platform if the problems persist.

v1.4.0 has been released. Any update on this issue?

Unfortunately I already deleted my trial account for Azure. We had to proceed with testing so we picked Google Cloude Platform over Azure.
Only thing I can comment further is that I tried with the v1.2.0 and I had the same problem as with the v1.3.0.

Sorry we weren't faster. Considering the similarity of this issue to #860, I'm quite confident that the fix in v1.4.0 has addressed this. Since there's nothing further planned, I'm closing this issue for now, but if you ever see this error again, we can resume investigating.


Title:  Add a KNN learning program

Is your feature request related to a problem? Please describe.
No.
Describe the solution you'd like.
Add a KNN learning program
Describe alternatives you've considered.
Add a KNN learning program
Add any other context or screenshots about the feature request here.
No response

Please assign me this issue and give this issue a gssoc'22 tag


fix: Properly restore kv.spec.uninstallStrategy

What this PR does
Currently serial tests rely on the global cleanup. Actually the global cleanup just ensure the kv.spec.config is restored, not the entire spec.
Test id [test_id:3683] changes the uninstallStrategy to BlockUninstallIfWorkloadsExist but does not restore the previous setting.
This can lead to failure to the subsequent tests, for example [test_id:3146]should be able to delete and re-create kubevirt install, in which uninstall strategy is evaluated.
Failure example: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/13139/pull-kubevirt-check-tests-for-flakes/1854890798374457344
/kind flake
Release note

NONE

Nice catch! should we maybe ensure kv is completely restored in global cleanup? I am afraid this is a similar symptom to what I bugged you about in #12766

I was tempted to do so, but I have preferred to do so in a separate wider PR and focus on the simple bug fix.

Nice catch! should we maybe ensure kv is completely restored in global cleanup? I am afraid this is a similar symptom to what I bugged you about in #12766

I was tempted to do so, but I have preferred to do so in a separate wider PR and focus on the simple bug fix.

Okay sure. Just my couple of cents - using patch here, I believe we'll be missing hotloop/test bugs like the one you're describing. But that's a whole different discussion and I am willing to accept that the operator changes the CR at it's discretion.
/lgtm


Question - Implementing Threads in Consumer code

Question
Can someone please share the best practices for implementing worker threads in Kafka Consumer program ? We would have one consumer subscribed to multiple topics and poll for messages. Once the message is received, need to hand this over to worker threads for processing the messages, so that main thread which polls are messages is not blocked.
Any sample code / reference link is appreciated.
Thanks,

we need to do an example or blog post outlining variations on this. you just want logic to maintain N worker threads at any given time. if a thread finishes, call Consumer.Consume and start a new worker with the result. Note that librdkafka will aggressively poll for new messages behind the scenes for you, so Consumer.Consume will not block if there are new messages to process. you may need to tune librdkafka parameters related to buffering (but probably not).

Is it good practice to have a queue in consumer side (May be a Blocking Collection ?) and fill it from On Message event.... We can stop polling if the queue size reaches a certain limit.
There would be certain set of Worker threads which would dequeue and process the messages in background...

librdkafka already gives you this queue internally, and you can tune the size - i don't think there would be any benefit to having another one. note also that OnMessage has been depreciated in the 1.0 API (use Consume instead).

Thanks @mhowlett . I will use Consume instead of Poll...
Do you have any sample to change the properties defined in librdkafka. I am using confluent-kafka-dotnet. How internal queue works in librdkafka.

@mhowlett  - I have put some sample code here with Consume Method / Tasks. I am adding a max of 10000 messages to a list & waiting for it to process before proceeding with next consume call.
Do you see any problems with this approach ?
` class Program
{
public static void Main(string[] args)
{
var conf = new ConsumerConfig
{
GroupId = "test-consumer-group",
BootstrapServers = "localhost:9092",
// Note: The AutoOffsetReset property determines the start offset in the event
// there are not yet any committed offsets for the consumer group for the
// topic/partitions of interest. By default, offsets are committed
// automatically, so in this example, consumption will only start from the
// eariest message in the topic 'my-topic' the first time you run the program.
AutoOffsetReset = AutoOffsetResetType.Earliest
};
        using (var c = new Consumer<string, string>(conf))
        {
            c.Subscribe(new List<string>() { "my-topic", "test-topic" });

            bool consuming = true;
            // The client will automatically recover from non-fatal errors. You typically
            // don't need to take any action unless an error is marked as fatal.
            c.OnError += (_, e) => consuming = !e.IsFatal;

            List<Task> eventTaskList = new List<Task>();

            while (consuming)
            {
                try
                {
                    var cr = c.Consume();

                    Task eventTask = Task.Run(() =>
                    {
                        Console.WriteLine($"{ DateTime.Now } - Consumed message '{cr.Key}' '{cr.Value}' at: '{cr.TopicPartitionOffset}'.");
                        //Do some work here
                        Thread.Sleep(500);
                        Console.WriteLine($"{ DateTime.Now } - Done Consuming message '{cr.Key}' '{cr.Value}' at: '{cr.TopicPartitionOffset}'.");

                    });

                    if (eventTaskList.Count >= 10000)
                    {
                        Task.WaitAll(eventTaskList.ToArray());

                        Console.WriteLine($"{ DateTime.Now } - Done Doing batch of " + eventTaskList.Count);

                        eventTaskList.Clear();
                    }
                    else
                    {
                        eventTaskList.Add(eventTask);
                    }

                }
                catch (ConsumeException e)
                {
                    Console.WriteLine($"Error occured: {e.Error.Reason}");
                }
            }

            // Ensure the consumer leaves the group cleanly and final offsets are committed.
            c.Close();
        }
    }
}`

Hi vjrag
I apreciate your example with tasks. I have a doubt the autoCommit in this case is the better way ?
Best Regards

vjrag,
I used this sample code in VB 2017 with confluent.kafka version 1.11. It complains that the type or namespace ConsumerConfig could not found. Do you know what I might miss?
Thanksm

ConsumerConfig is new - which version of Confluent.Kafka are you using? highly recommend 1.0-beta3 over 0.11.6.


Data serializer register

@mparent61 @MichaelButkovic
not ready yet.
Just looking for feedback.
Left it in bulbs, cause daddy wanted it there & it's currently light enough.

Looking good

Coverage decreased (-0.2%) to 86.035% when pulling d368de6925c6ed7426429e2d9437d786ea77f9ec on data-serializer-register into 0532a3b617512020751d7e63fa8257a943e10888 on master.

Replaced by https://github.com/theonion/django-bulbs/pull/285


Colab link returns 'File not found'

Hi there, I'm pretty fascinated by this work, and gonna to take a try on the Colab link, however I found the link returns 'File not found'. Could you please fix this bug, and release the Colab link again? Thanks a lot : )

I'm not sure why the colab link disappeared. Maybe @shangtse knows?
You can also run the notebook in the repository on your local machine or upload it to colab on your own. If I find time, I might upload the notebook again and change the link in the repo.

That would be great! @dxoigmn were you able to upload the file?


Change command format of creategroup command and improve…

…functionality of deletegroup command

Codecov Report

Merging #116 (8a504ad) into add-delete-group-modification (514b1f7) will decrease coverage by 0.57%.
The diff coverage is 0.00%.

@@                         Coverage Diff                         @@
##             add-delete-group-modification     #116      +/-   ##
===================================================================
- Coverage                            61.70%   61.12%   -0.58%
  Complexity                             407      407
===================================================================
  Files                                   80       81       +1
  Lines                                 1478     1492      +14
  Branches                               159      161       +2
===================================================================
  Hits                                   912      912
- Misses                                 527      541      +14
  Partials                                39       39

Impacted Files
Coverage Δ

...edu/address/logic/commands/CreateGroupCommand.java
0.00% <ø> (ø)

...edu/address/logic/commands/DeleteGroupCommand.java
0.00% <ø> (ø)

.../seedu/address/logic/parser/AddressBookParser.java
86.36% <ø> (ø)

.../seedu/address/logic/parser/ArgumentTokenizer.java
88.09% <0.00%> (-9.28%)
:arrow_down:

.../seedu/address/logic/parser/CreateGroupParser.java
0.00% <0.00%> (ø)

...s/logic/parser/exceptions/EmptyGroupException.java
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 514b1f7...8a504ad. Read the comment docs.


App Doesn't Log in Until I Refresh

Describe the bug
The "LOG IN" button doesn't sign me in. I have to refresh the page
To Reproduce

Go to /login
Enter login details
Click "LOG IN"

Expected behavior
I should be logged in and redirected to the main app page.
Desktop (please complete the following information):

OS: MacOS
Browser: Chrome

Messy but Quick Fix
Y'know... one solution to this is just to allow the page to refresh. Treat form submission like a regular form submission (do not :prevent the @submit).
Once refreshed, the app would load and redirect appropriately.

Actual Fix
I was looking at the actions in the store and we aren't actually returning a Promise.resolve.
I implemented one. It didn't fix the issue, but it is certainly a step in the right direction.
I suspect that... possibly the route hook that looks for an existing user-token is not being fired when the router push happens...
Might also be worth looking at the onAbort event of Vue's router.push: https://router.vuejs.org/guide/essentials/navigation.html#router-push-location-oncomplete-onabort
Near as I can tell, onAbort fires when navigation to the new page fails and the user stays on the current page. Worth scattering a few console.logs about, maybe including here, to see how far through the stack our process is actually getting.


Write messages

The asyncWriteMessages is like this:
def asyncWriteMessages(messages: Seq[PersistentRepr]): Future[Unit] = {
val sends = messages.groupBy(.persistenceId).map {
case (pid, msgs) => writerFor(pid).ask(messages)(writeTimeout)
}
Future.sequence(sends).map( => ())
}
I think it should be like this:
def asyncWriteMessages(messages: Seq[PersistentRepr]): Future[Unit] = {
val sends = messages.groupBy(.persistenceId).map {
case (pid, msgs) => writerFor(pid).ask(msgs)(writeTimeout)
}
Future.sequence(sends).map( => ())
}
Do i make a mistake?

Ouch, that's really a bug, good catch. It just didn't surface because a message batch sent by akka-persistence only contains messages with the same persistenceId. Do you want to create a pull request? Maybe along with a test that demonstrates that the bug is fixed?

I've created a pull request: https://github.com/krasserm/akka-persistence-kafka/pull/25
But no test.


How to get coord3D from plane3D ?

Hi,
i have a planed3D (using the a,b,c et d values) and i would like the get a ccord3D where the plane3D is the 0xy plane.
how can i do ?

Hi!
If you need to create arbitrary CS aligned with plane, you can do like that:
            Plane3d p = new Plane3d(1, 1, 1, 1);             Point3d point = Coord3d.GlobalCS.Origin;             Point3d projection = point.ProjectionTo(p);             Vector3d z = p.Normal;             Vector3d x = z.OrthogonalVector;             Vector3d y = x.Cross(z);             Coord3d new_coord = new Coord3d(projection, x, y);
If you already have specific point where origin of new CS should be located, use it instead of 'projection' point.


CORE-AAM: Add role-mapping results for macOS and Linux

N.B. The associated tests have not (at this time) landed. https://github.com/w3c/web-platform-tests/pull/7263


feedback component in all pages

Harness Developer Pull Request
Thanks for helping us make the Developer Hub better. The PR will be looked at
by the maintainers.
What Type of PR is This?

[ ] Issue
[ ] Feature
[x] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s)

House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[x] Tested Locally
[

] Optional Screen Shoot.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app

Just my two cents on this: There is a lot of text for just a feedback request, and with multiple sections we risk people closing the dialog before they've submitted their comment.
I think we could do something more simplistic, like this but without the rating:

"Let us know what you think about the Harness Developer Hub."
"Comment *" field (required).
"Email (optional)" field.
"Submit" button.

After they click submit, show a simple "Thank you for your feedback." message with a "Close" button.

@aimurphy that feedback form out of the box for what we use in product. @rohanmaharjan100 just for the PR as we discussed in Slack, would be better below the edit page link. Thanks!

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app

@rohanmaharjan100 per our convo this AM, looks like the feedback item is missing off the homepage. Thanks!

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app


Wso2 whitepaper

Doc - https://wso2.com/whitepapers/building-an-api-strategy-using-an-enterprise-api-marketplace/
Update the sentence as "look" is inappropriate here.
"Complex transformation might require an integration platform; most customers look to bring their own ID systems (BYOID) - in which case the API Manager needs to allow consumers to authenticate against the existing system by federating to them. "
"Complex transformation might require an integration platform; most customers seem to bring their own ID systems (BYOID) - in which case the API Manager needs to allow consumers to authenticate against the existing system by federating to them. "

I have edited the text as suggested.


fix: Change dbt functions args check for ast.Str that works on all versions

Deprecated since version 3.8: Old classes ast.Num, ast.Str, ast.Bytes, ast.NameConstant and ast.Ellipsis are still available, but they will be removed in future Python releases. In the meantime, instantiating them will return an instance of a different class.

So we need to check ast.Str for 3.7

fixed now: https://github.com/fal-ai/fal/runs/6530627497?check_suite_focus=true


Adds .dl-horizontal-breakpoint

Not sure how to build the project correctly; every time I try to build, it removes the license.
See #16562

Grunt is used to build the project, but PR's do not include the built files. You can use grunt To execute the default task which builds and runs tests. This would need Jekyll gem and rogue gem for building the docs.

I've updated the docs.


Hash-Pin Dependencies

Hi I'm Joyce from Google and I'm working on helping many open source projects on improving their supply chain security posture.
Description
I would like to suggest a security practice recommended by the OpenSSF Scorecard which is to hash pin dependencies to prevent dependency-confusion and typosquatting attacks.
The change would only be applied to GitHub workflows, dockerfiles and shell scripts dependencies.
This means:

Hash pinning GitHub Workflow actions.
Hash pinning the image of the OS/runner on dockerfiles.
Using --requirechecksums on choco installs.

I can submit one PR for each type of change above to be easier to review if you prefer. Just let me know if that's the case.
Along with hash-pinning dependencies, I also recommend adopting dependabot or renovatebot to help keep the dependencies up to date. Both tools can update  hashes and associated semantic version comments.
Let me know if you are open to evaluate those changes and I'll submit the PRs ASAP.
Any questions or concerns just let me know.
Thanks!
Additional Context
For more informations about dependency confusion / typosquatting attacks:

Detection of malicious packages
Counter dependency confusion (aka substitution) attacks

For more informations about the dependency-update tools:

Dependabot
Renovatebot

@joycebrum

This crate has 0 dependencies due to its particular role in the Rust ecosystem.
Its test crate has only a few minimal dependencies, all of which are either directly controlled by the Rust Project, or are directly controlled by a trusted member of the Rust Project.
Pinning, as you describe, is counter to recommended practice in the Rust ecosystem due to peculiarities of Cargo's resolver.

But thank you for the offer to help.

...and we control most of the GitHub Actions too.
For the ones we don't... we have much bigger things to worry about if actions/checkout@v3 is suborned, and the amount of maintainer time/energy to evaluate automated warnings from things like dependabot and renovatebot is extremely thin, so increasing the already-high maintenance burden of this crate, merely to obtain some points on a scorecard from the OSSF, would be... counterproductive. Because this crate changes incredibly slowly, including its CI, I am not seriously concerned.
If you have a more... holistic security recommendation that accounted for this being a crate being controlled by a gaggle of programmers who are often accused of being anxiety-ridden paranoiacs already, we would probably be interested.

Hi @workingjubilee thanks for the return!
No problems, there are really strong cons on hash pinning and there is no right answer on this matter. Besides, the workflows being with permissions minimally scoped as it is, it is safe enough! 🙏🏼


Re-Upload Scan error

Slack us first!
I spoke to Chris Fort on slack he told me to open an issue
Bug description
When performing the reimport-scan of a Generic Findings Import, the error by the GUI is " An exception error occurred during the report import:name 'ValidationError' is not defined ".
Chris Fort pointed this out to me as the error:

django.core.exceptions.ValidationError: ['Host must not be empty']

Performing the Generic Findings import-scan using a .csv file works normally, with reimpot-scan it is not working, I am using the same .csv file as the import-scan.
Using the API the same exception happens.
Steps to reproduce
Steps to reproduce the behavior:

Import the file with Import-scan using Generic Findings Import
Re-import the same file using ReuploadScan.

Expected behavior
Run the Re-Upload Scan without errors
Deployment method (select with an X)

[X] Docker Compose
[ ] Kubernetes
[ ] GoDojo

Environment information

Operating System: Ubuntu 20.04
DefectDojo version (see footer) or commit message: 2.2.1

Additional context (optional)
Add any other context about the problem here.
[2021-08-31 12:37:55 -0500] 00b4257: Merge pull request Release: Merge release into master from: release/2.2.0 #5014 from DefectDojo/release/2.2.0 [ (HEAD -> master, tag: 2.2.0, origin/master, origin/HEAD)]
LOG
log.txt
https://pastebin.com/FH0GLHSc

I can take this one

The problem with the missing declaration of ValidationError has already solved with #4962

Ah! That's why I can't found it on dev :D

Fixed in dev by #4962


PCR duplicates

Since pre-processing input BAM files prior to analysis isn't an option, we should be able to handle this on a partition-by-partition basis. We don't have access to both ends of a read, so duplicate sequences MIGHT not be PCR duplicates, but we can deduplicate anyway (make it an option to disable deduplication).

This was handled as part of #112.


How to combine groovy-language-server with monaco-editor

I look at the example using vscode-json-languageservice, and then I want to use groovy-language-server, but I failed to introduce it in the first step, please help me

I'm afraid that I have no experience working with Monaco editor directly. I know that VSCode uses Monaco, but it's abstracted away, so I don't know anything about its APIs. I wish I could provide more advice!


Single row query throws System.OutOfMemoryException

Hi Folks,
Im getting an exception when running a non async a query  on a single row with a column that contains a 20 mb xml object.
I was previously using async but went to non async due the size of the column and the way data is transfered for large cols.
Thanks for taking a look :)
Exception message:
Exception of type 'System.OutOfMemoryException' was thrown.
Stack trace:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at
 System.String.CtorCharArrayStartLength(Char[] value, Int32 startIndex, Int32 length) at
 System.Data.SqlClient.TdsParser.TryReadSqlStringValue(SqlBuffer value, Byte type, Int32 length, Encoding encoding, Boolean isPlp, TdsParserStateObject stateObj) at
 System.Data.SqlClient.TdsParser.TryReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj) at
 System.Data.SqlClient.SqlDataReader.TryReadColumnInternal(Int32 i, Boolean readHeaderOnly) at
 System.Data.SqlClient.SqlDataReader.GetValues(Object[] values) at
 Microsoft.EntityFrameworkCore.Storage.Internal.UntypedRelationalValueBufferFactory.Create(DbDataReader dataReader) at
 Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable.Enumerator.BufferlessMoveNext(Boolean buffer) at
 Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`2 operation, Func`2 verifySucceeded, TState state) at
 Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[TState,TResult](Func`2 operation, Func`2 verifySucceeded, TState state) at
 Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, Func`2 operation, TState state) at
 Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable.Enumerator.MoveNext() at
 Microsoft.EntityFrameworkCore.Query.QueryMethodProvider.<_ShapedQuery>d__3`1.MoveNext() at
 System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source) at
 lambda_method(Closure , QueryContext ) at
 Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass20_0`1.<CompileQueryCore>b__0(QueryContext qc) at
 Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query) at
 Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression) at
 System.Linq.Queryable.FirstOrDefault[TSource](IQueryable`1 source) at
 SomeNameSpace.DataModule.Repository`2.Get(Expression`1 where) in /opt/atlassian/pipelines/agent/build/src/Modules/SomeNameSpace.DataModule/Repository.cs:line 445 at
 SomeNameSpace.IntegrationServerModule.Services.IntegrationService.<>c__DisplayClass16_0.<SetResponseMessageAsync>b__0() in /opt/atlassian/pipelines/agent/build/src/Modules/SomeNameSpace.IntegrationServerModule/Services/IntegrationService.cs:line 287 at
 System.Threading.Tasks.Task`1.InnerInvoke() at
 System.Threading.Tasks.Task.Execute() --- End of stack trace from previous location where exception was thrown --- at
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at
 SomeNameSpace.IntegrationServerModule.Services.IntegrationService.<SetResponseMessageAsync>d__16.MoveNext() in /opt/atlassian/pipelines/agent/build/src/Modules/SomeNameSpace.IntegrationServerModule/Services/IntegrationService.cs:line 287 --- End of stack trace from previous location where exception was thrown --- at
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at
 SomeNameSpace.IntegrationServerModule.Services.IntegrationService.<MessageResponseAsync>d__15.MoveNext() in /opt/atlassian/pipelines/agent/build/src/Modules/SomeNameSpace.IntegrationServerModule/Services/IntegrationService.cs:line 261 --- End of stack trace from previous location where exception was thrown --- at
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at
 SomeNameSpace.IpApi.Integration.Controllers.IntegrationController.<MessageResponseAsync>d__4.MoveNext() in /opt/atlassian/pipelines/agent/build/src/SomeNameSpace.IpApi.Integration/Controllers/IntegrationController.cs:line 63 --- End of stack trace from previous location where exception was thrown --- at
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at
 Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__27.MoveNext() --- End of stack trace from previous location where exception was thrown --- at
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at
 Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__25.MoveNext() --- End of stack trace from previous location where exception was thrown --- at
 Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context) at
 Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at
 Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextExceptionFilterAsync>d__24.MoveNext()

Steps to reproduce

// How the query is run
// _entitySet.Where(row => row.id = 1).FirstOrDefault();

// context setup
public class IntegrationServerDataContext : DbContext, IDataContext
{
    private readonly IConfiguration _configuration;
    private readonly IEnumerable<IDataContextProvider> _contextProviders;
    private readonly IHostingEnvironment _env;

    public IntegrationServerDataContext(IHostingEnvironment env, IConfiguration configuration, IEnumerable<IDataContextProvider> contextProviders)
    {
        _env = env;
        _configuration = configuration;
        _contextProviders = contextProviders;

        Database.SetCommandTimeout(600);
    }

    public new DbSet<TEntity> Set<TEntity>() where TEntity : class
    {
        return base.Set<TEntity>();
    }

    protected override void OnConfiguring(DbContextOptionsBuilder options)
    {
        string connectionString;

        switch (_env.EnvironmentName)
        {
            case EnvironmentType.IntegrationTest:

                options.EnableSensitiveDataLogging();
                connectionString = _configuration.GetConnectionString("IntegrationTestConnection");
                break;

            default:

                connectionString = _configuration.GetConnectionString("DefaultConnection");
                break;
        }

        options.UseSqlServer(
            connectionString,
            optionsBuilder => optionsBuilder
                .MaxBatchSize(1000)
                .CommandTimeout(600)
                .EnableRetryOnFailure()
        ).ConfigureWarnings(
            warnings => warnings.Log(RelationalEventId.QueryClientEvaluationWarning)
        );
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        foreach (var provider in _contextProviders)
        {
            provider.OnModelCreating(modelBuilder);
        }
    }
}

Further technical details
EF Core version: 1.1.2
Database Provider: Azure P1
Operating system: Azure App service Standard: 1 Large

The error is coming from SqlClient. Can you test against pure ADO.Net code and see if that works?

I bashed up this console app. It runs fine.
using System;
using System.Data.SqlClient;

namespace ConsoleApp1
{
    public class Program
    {
        static void Main()
        {
            Console.WriteLine("start");

            const string connectionString =
                "Server=tcp:REDACTED;Initial Catalog=REDACTED;Persist Security Info=False;User ID=REDACTED;Password=REDACTED;MultipleActiveResultSets=True;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;";

            const string queryString =
                "SELECT id, [RequestXml], [ResponseXml] from [IntegrationQueues] WHERE id = 34517";

            using (var connection = new SqlConnection(connectionString))
            {
                var command = new SqlCommand(queryString, connection);
                try
                {
                    connection.Open();
                    var reader = command.ExecuteReader();
                    while (reader.Read())
                    {
                        Console.WriteLine("\t{0}\t{1}\t{2}",
                            reader[0], reader[1], reader[2]);
                    }
                    connection.Close();
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex.Message);
                }
                Console.ReadLine();
            }
        }
    }
}

@smitpatel sorry forgot to tag you see above :)

As a work around we have actually decided to go with the ado connector for now. Still very interested in fixing this for entity tho.

@Spaceman1861 - Can you share a repro code which demonstrate above issue?
You would need to provide a small app using similar model & querying. A sql script to insert some sample data which is large enough to show error. & details about your SqlServer instance in Azure. There are various things involved here so it would be hard to reproduce the error unless all parameters are matched up exact same.

@smitpatel ill put it my list and get it to you as soon as possible

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.
BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.


Linters setup fix and lint rules review for Flutter 2.5

Linters and Flutter 2.5
This is a review of linters to prepare for Flutter 2.5. I made a pull request, here (https://github.com/weta-vn/advance_image_picker/pull/23) that contains atomic updates, for your review to the linter setup and linter issues discussed below. This should help you with getting the linter working fully and make the code Flutter 2.5 ready as well.
Atomic commits included in the the pull request:

Photo manager version upgrade (minor) (https://github.com/rydmike/advance_image_picker/commit/8d20e47bf7466b67da8221dae6a26cd0cd152361)
Remove two lint rules that were put on manually that ARE already on (https://github.com/rydmike/advance_image_picker/commit/8d20e47bf7466b67da8221dae6a26cd0cd152361), a typo correction (https://github.com/rydmike/advance_image_picker/commit/c200ac7aaf7b4e985d25a19d6646e2137b07c223) together with Dart SDK 2.14 fixes the issues with the linter not being triggered correctly so those extra true rules are no longer needed, they are ON already. The idea with this linter setup is to only have to turn OFF rules in analysis_options.yaml never on, all available rules are put ON by default in all_lint_rules.yaml.
Added two new rules to all_lint_rules.yaml released by Dart team. (https://github.com/rydmike/advance_image_picker/commit/e5df4359bdd0ea44d865bb916f1ac233ca6d6353)
Set SDK to 2.14.0 (https://github.com/rydmike/advance_image_picker/commit/2308cfdc532fc025f3ef704caf7b17c4df39d091)

The above fixes led to that there were a massive 598 linter hints triggered. They should have been triggered already with past lint setup, but due to the issue mentioned above they were not.

To reduce the lint warnings dart fix --apply was run as well as removing the in Flutter 2.5.0 deprecated AppBar API usage that was still used temporarily via (https://github.com/weta-vn/advance_image_picker/pull/11/commits/17cd5c97d7466abd3d0ad43709fe52c2efaad6a2), but flagged as should be removed in later version, this also removes the deprecated AppBar backwardsCompatibility API usage.

These changes brought the lint errors to "only" 177 remaining.
After this I did a large amount of manual lint fixes as well, they are listed as atomic commits with comments in the pull request. Those fixes were very basic that the IDE could do pretty automatically but dart fix --apply could not.
What is left?
This still leaves 3 warning and 81 lint hints un-resolved. I left them un-fixed on purpose, some are just doc comments for you to simply add (it will improve your doc API percentage in pub.dev ) but others are good to review manually, most of them are straight forward. For the 3 cases of "warning: Don't reassign references to parameters of functions or methods." you may want to add and ignore of the lint for the line where you mutate the parameter value, if that is what you want to do. just put a:
// ignore: parameter_assignments

before the line where you need to do it and the warning goes away and at the same time the ignore is a good way to document that it is intentional. Alternatively rethink the re-assignment of the parameter, make the parameter final and create a local copy of it that you change if so needed.
The rest of the still open fixes are quite simple, but I can certainly advice if you need help.
I did a test build and tested the picker in an app, with all above changes applies. As far as I could see everything still worked as before. 😃
BTW yes the avoid lines longer than 80 chars is very annoying. However, at least not long ago, a package pub.dev score was penalized 10 points if it was not used, since the dartfmt that is used by pub.dev uses 80 chars as max allowed line as the format it check against when giving points for using dartfmt. I have not checked if they have abandoned that part of the dartfmt check yet. I don't think so, but there was some debate about using it raised in issues. Could be good to test if it is still used.
I think that is the format issue I see on pub.dev, might not be related to 80 char length anymore:

Definitely worth testing just turning it OFF and see if it affects the score ore not, there is already in analysis_options.yaml a placeholder, just uncomment it.

Remaining lint warnings and hints to be corrected:

@rydmike Thank you very much.
We'll confirm & merge asap.

Did some minor edits to the description above, mostly on the 80 char lint topic.

I realize this kind of merge might be tricky for your dev branches and therefore not feasible right now. In that case you can use my steps as a guide.
 I just left a few lint warnings in there for you to fix and did not add more API docs, better that you do that. The API doc lint is handy since it tells you were they are missing, where pub.dev want to see them for a better API doc coverage score. This is pretty good already, but you can easily get to 90% or higher:

As you know Flutter 2.5 comes with flutter_lints package used as default for new Flutter projects. The lint rules used here include all of the lint rules in flutter_lints, and many more, making this an even tougher lint rule set to comply by. But once it is done, using them and coding with them is imo no problem, the IDE tells you what to do 😃
Also you can use the dart fix --dry-run to test what lints need to be fixed and can be fixed automatically and then dart fix --apply to fix them, I already did that as part of this PR. It also takes care of deprecation API fixes, when possible, good tool to know: https://dart.dev/tools/dart-fix
If you use VS-Code you can set it to fix lints on save too.

I don't think it is available on IJ/AS yet, might come soon.
By the way my blog on linting has been updated to reflect Flutter 2.5 release and include updated rule comparison table between linters. https://rydmike.com/blog_flutter_linting


error: relation "videos" does not exist

got the error when deploy to vercel
error: relation "videos" does not exist

duplicated the issue when run locally

ok, fixed with this command
psql ""postgres://USER:PASSWORD@HOST/DB"" -f data/install.sql

suck

I encountered the same problem, can you tell me how to solve it? I didn't find where to enter the instructions when deploying on vercel

I encountered the same problem, can you tell me how to solve it? I didn't find where to enter the instructions when deploying on vercel

In the readme,

After create the postgresql database, we need Initialize the database. The sql file is in the source code at data/install.sql
if you familar the psql command, the simpliest way is
psql "postgres://USER:PASSWORD@HOST/DB" -f data/install.sql

If you don't have the psql command, you need install it first, please refer Postgresql official document for help.


Issue31 create and change scenario which are not calculating

Codecov Report

Merging #39 (21d43f5) into main (ec5b81d) will not change coverage.
The diff coverage is 100.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff            @@
##              main       #39   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           66        66
  Lines         3662      3683   +21
=========================================
+ Hits          3662      3683   +21

Impacted Files
Coverage Δ

ScenarioGUI/gui_classes/gui_calculation_thread.py
100.00% <100.00%> (ø)

ScenarioGUI/gui_classes/gui_combine_window.py
100.00% <100.00%> (ø)

.../test_gui_structure_elements/test_result_export.py
100.00% <100.00%> (ø)

...test_gui_structure_elements/test_results_figure.py
100.00% <100.00%> (ø)

...s/test_gui_structure_elements/test_results_text.py
100.00% <100.00%> (ø)

tests/test_main_window_functions/test_auto_save.py
100.00% <100.00%> (ø)

tests/test_main_window_functions/test_run.py
100.00% <100.00%> (ø)

... and 7 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


API DSL

Todos

[ ] Add Error type
[ ] Customize Response headers, status code, etc. in handler
[ ] Refine api
[ ] Flesh out Doc type
[ ] Unify Input & Params into single type
[ ] Improve Client
[ ] Extract model into shared project so we can have a Scala.js client
[ ] Improve OpenAPI support
[ ] Expose OpenAPI from server

Performance Explorations

[ ] Tuple Allocation Optimizations (during Zips)
[ ] OrElse Optimizations (create Trie-like request parsing structure)

Codecov Report

Merging #1209 (fefa20b) into main (bba6d8b) will decrease coverage by 8.32%.
The diff coverage is 0.00%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bba6d8b...fefa20b. Read the comment docs.

@tusharmath I added some (very basic) support for customizing headers/status codes when handling responses. There are a few ways of doing this, but I think an overload where the user gets to simply manually specify what to return from the handler is both the simplest and should work pretty well.
Next, I'm going to get zio-schema published for Scala 3 and fix the 2.12 compilation errors.


[feature card] theming is off

Detailed description
The feature card components both medium and large aren't changing themes. Looks like we're hard setting it g90 theme. The design specs appear to be using inverse tokens.

Steps to reproduce the issue

Go here (link is to web components, but same issue is in react)
Change theme

Additional information

https://github.com/carbon-design-system/carbon-for-ibm-dotcom/blob/master/packages/styles/scss/components/feature-card/_feature-card.scss#L151

Still an issue :scream:


[chore] :recycle: Refactor onConnect listener creation to a util function

What is the context for these changes?

Refactors and simplifies the onConnect listener logic. This allows for easier middleware creation for onConnect events by removing the possibility to missed actions (if the actions for onConnect logic are to ever change) and DRYs up the creation of listeners.
The inspiration behind this was not only to DRY up the code a bit, but also because in #4 I am going to re-use this logic (it could likely be reused in #13 as well) for looking up ENS names when a wallet connects.
Note: I was going to add tests for useMiddleware.ts as well, however I noticed that we need to create some form of app context in order to render hooks with the state provider. I'd prefer for that to be done the right way, and I believe that doing so will make this PR larger than it should be.

How can this be tophatted?

Primarily green CI (I've added tests for the functionality). However, I will also run /snapit and link a CodeSandbox with the snapshot version for testing.
🎩 Checklist

[ ] Tested on mobile
[x] Tested on multiple browsers
[x] Includes unit tests (see note in the context for the PR)
[ ] ~Tested for accessibility~ N/A
[ ] ~Updated relevant documentation for the changes (if necessary)~ N/A

You can test this PR using this CodeSandbox


Fatal error: 'cuda_runtime_api.h' file not found

The following error message occurs when I install the apex from source on my ROCm server(CentOS 7.6).
python setup.py install --cpp_ext --cuda_ext

In file included from /public/home/apex-master/apex/contrib/csrc/optimizers/multi_tensor_distopt_adam_kernel.hip:12:
In file included from /public/home/apex-master/csrc/multi_tensor_apply.cuh:3:
/public/home/.conda/envs/my_env/lib/python3.8/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: 'cuda_runtime_api.h' file not found
#include <cuda_runtime_api.h>
         ^~~~~~~~~~~~~~~~~~~~
1 error generated when compiling for gfx803.
error: command '/public/software/compiler/rocm/rocm-4.0.1/bin/hipcc' failed with exit code 1

It seems that it is building 'distributed_adam_cuda' extension.
My envirments:
Currently Loaded Modulefiles:
  1) compiler/devtoolset/7.3.1   2) compiler/rocm/4.0.1         3) mpi/hpcx/2.4.1/gcc-7.3.1

python                    3.8.13
torch                     1.10.1+rocm4.0.1

Is nvcc or any other dependencies necessary？

Hi @lvcc2018,
It seems that some files were not "hipified" properly. I recommend you to use the following options:

(Preferred) Use our published docker image: docker pull rocm/pytorch:latest-centos7 which has the latest prebuilt stable PyTorch and Apex. If you would like to make some changes to Apex, feel free to reinstall Apex from source.
Uninstall ROCm 4.0.1 and reinstall newer versions of ROCm and their dependencies. Then, build ROCm from source.

Hi @lvcc2018, It seems that some files were not "hipified" properly. I recommend you to use the following options:

(Preferred) Use our published docker image: docker pull rocm/pytorch:latest-centos7 which has the latest prebuilt stable PyTorch and Apex. If you would like to make some changes to Apex, feel free to reinstall Apex from source.
Uninstall ROCm 4.0.1 and reinstall newer versions of ROCm and their dependencies. Then, build ROCm from source.

Thanks for your time. Agree that it doesn't hipify properly. Unfortunately It's not allowed to use docker or use a newer version of ROCm. I find that all the .cu files are skipped, is it normal?
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/multihead_attn/dropout_hip.cuh -> None ignored
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/multihead_attn/layer_norm_hip.cuh -> None ignored
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/multihead_attn/strided_batched_gemm_hip.cuh -> None ignored
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/nccl_p2p/nccl_p2p_cuda.cuh -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/nccl_p2p/nccl_p2p_cuda.cuh ok
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/nccl_p2p/nccl_p2p.cpp -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/nccl_p2p/nccl_p2p.cpp ok
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/nccl_p2p/nccl_p2p_cuda.cu -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/nccl_p2p/nccl_p2p_hip.hip skipped
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/fused_adam_cuda.cpp -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/fused_adam_cuda.cpp ok
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/fused_adam_cuda_kernel.cu -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/fused_adam_hip_kernel.hip skipped
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/fused_lamb_cuda.cpp -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/fused_lamb_cuda.cpp ok
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/fused_lamb_cuda_kernel.cu -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/fused_lamb_hip_kernel.hip skipped
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/multi_tensor_distopt_adam.cpp -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/multi_tensor_distopt_adam.cpp ok
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/multi_tensor_distopt_adam_kernel.cu -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/multi_tensor_distopt_adam_kernel.hip skipped
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/multi_tensor_distopt_lamb.cpp -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/multi_tensor_distopt_lamb.cpp ok
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/multi_tensor_distopt_lamb_kernel.cu -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/optimizers/multi_tensor_distopt_lamb_kernel.hip skipped
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/transducer/transducer_joint_kernel.cu -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/transducer/transducer_joint_kernel.hip skipped
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/transducer/transducer_joint.cpp -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/transducer/transducer_joint.cpp ok
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/transducer/transducer_loss.cpp -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/transducer/transducer_loss.cpp ok
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/transducer/transducer_loss_kernel.cu -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/transducer/transducer_loss_kernel.hip skipped
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/xentropy/interface.cpp -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/xentropy/interface.cpp ok
/public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/xentropy/xentropy_kernel.cu -> /public/home/ach2ha8oau/megatron-deepspeed/apex-master/apex/contrib/csrc/xentropy/xentropy_kernel.hip skipped
Successfully preprocessed all matching files.
Total number of unsupported CUDA function calls: 0

Which version of pytorch do you have installed?

Which version of pytorch do you have installed?

Its 1.10.1+rocm4.0.1


Castle Wars Tables

Menu entry swapper addition for taking 5 bandages, barricades, explosives, boulders, instead of the default 1 for each click.

This can now be done with the changes in 92aac53f4273e7944c3da767637e083847c03808


Add new communication endpoint to node

Node should be able to serve raw data from storage.

We think we want a binary protocol to ferry stuff across with pretty low level access.

[ ] Juliet server in node (mvp)
[ ] Add some kind of connection limiter
[ ] Respect config
[ ] Add metrics
[ ] Take care of node_startup_instant
[ ] Cover with tests

Regarding the in-memory data that is used in the RPC queries, here's the summary from @jacek-casper: https://gist.github.com/jacek-casper/9dce32c2e4b0282efd497f6e7abffaba

Updated todo list:

[ ] support binary port config
[ ] add metrics
[ ] implement event system in binary port component
[ ] add some kind of connection limiter
[ ] add tests coverage
[ ] cleanup (especially, remove all remaining references to RPC server from node docs, comments, etc.)
[ ] update changelog


RuntimeError: GET was unable to find an engine to execute this computation

When running the patchcore model on the MvTEC dataset, no runtime errors during training, but I get this error when trying to train using a custom dataset.
To use wandb logger install it using `pip install wandb`
/mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/anomalib/config/config.py:238: UserWarning: The seed value is now fixed to 0. Up to v0.3.7, the seed was not fixed when the seed value was set to 0. If you want to use the random seed, please select `None` for the seed value (`null` in the YAML file) or remove the `seed` key from the YAML file.
  warn(
/mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/anomalib/config/config.py:275: UserWarning: config.project.unique_dir is set to False. This does not ensure that your results will be written in an empty directory and you may overwrite files.
  warn(
Global seed set to 0
2023-03-27 14:26:50,464 - anomalib.data - INFO - Loading the datamodule
2023-03-27 14:26:50,465 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-03-27 14:26:50,465 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-03-27 14:26:50,465 - anomalib.models - INFO - Loading the model.
2023-03-27 14:26:50,465 - anomalib.models.components.base.anomaly_module - INFO - Initializing PatchcoreLightning model.
/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric `PrecisionRecallCurve` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
  warnings.warn(*args, **kwargs)
2023-03-27 14:26:50,470 - anomalib.models.components.feature_extractors.timm - WARNING - FeatureExtractor is deprecated. Use TimmFeatureExtractor instead. Both FeatureExtractor and TimmFeatureExtractor will be removed in a future release.
2023-03-27 14:26:51,838 - timm.models.helpers - INFO - Loading pretrained weights from url (https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/wide_resnet50_racm-8234f177.pth)
2023-03-27 14:26:52,039 - anomalib.utils.loggers - INFO - Loading the experiment logger(s)
2023-03-27 14:26:52,039 - anomalib.utils.callbacks - INFO - Loading the callbacks
/mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/anomalib/utils/callbacks/__init__.py:142: UserWarning: Export option: None not found. Defaulting to no model export
  warnings.warn(f"Export option: {config.optimization.export_mode} not found. Defaulting to no model export")
2023-03-27 14:26:52,060 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True
2023-03-27 14:26:52,060 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores
2023-03-27 14:26:52,060 - pytorch_lightning.utilities.rank_zero - INFO - IPU available: False, using: 0 IPUs
2023-03-27 14:26:52,060 - pytorch_lightning.utilities.rank_zero - INFO - HPU available: False, using: 0 HPUs
2023-03-27 14:26:52,061 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
2023-03-27 14:26:52,061 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..
2023-03-27 14:26:52,061 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..
2023-03-27 14:26:52,061 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(limit_predict_batches=1.0)` was configured so 100% of the batches will be used..
2023-03-27 14:26:52,061 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
2023-03-27 14:26:52,061 - anomalib - INFO - Training the model.
2023-03-27 14:26:52,237 - anomalib.data.base.datamodule - INFO - No normal test images found. Sampling from training set using a split ratio of 0
/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric `ROC` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
  warnings.warn(*args, **kwargs)
/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:613: UserWarning: Checkpoint directory /mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/results/patchcore/mpi/run/weights exists and is not empty.
  rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")
2023-03-27 14:26:52,652 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py:183: UserWarning: `LightningModule.configure_optimizers` returned `None`, this fit will run with no optimizer
  rank_zero_warn(
2023-03-27 14:26:52,656 - pytorch_lightning.callbacks.model_summary - INFO -
  | Name                  | Type                     | Params
-------------------------------------------------------------------
0 | image_threshold       | AnomalyScoreThreshold    | 0
1 | pixel_threshold       | AnomalyScoreThreshold    | 0
2 | model                 | PatchcoreModel           | 24.9 M
3 | image_metrics         | AnomalibMetricCollection | 0
4 | pixel_metrics         | AnomalibMetricCollection | 0
5 | normalization_metrics | MinMax                   | 0
-------------------------------------------------------------------
24.9 M    Trainable params
0         Non-trainable params
24.9 M    Total params
99.450    Total estimated model params size (MB)
Epoch 0:   0%|                                                                                | 0/16 [00:00<?, ?it/s]/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py:138: UserWarning: `training_step` returned `None`. If this was on purpose, ignore this warning...
  self.warning_cache.warn("`training_step` returned `None`. If this was on purpose, ignore this warning...")
Epoch 0:  69%|████████████████2023-03-27 14:26:53,916 - anomalib.models.patchcore.lightning_model - INFO - Aggregating the embedding extracted from the training set.s]
2023-03-27 14:26:53,917 - anomalib.models.patchcore.lightning_model - INFO - Applying core-set subsampling to get the embedding.
   Traceback (most recent call last):                                                                                | 0/5 [00:00<?, ?it/s]
  File "tools/train.py", line 75, in <module>
    train()
  File "tools/train.py", line 61, in train
    trainer.fit(model=model, datamodule=datamodule)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
    results = self._run_stage()
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
    self._run_train()
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_train
    self.fit_loop.run()
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.on_advance_end()
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 250, in on_advance_end
    self._run_validation()
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 308, in _run_validation
    self.val_loop.run()
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance
    output = self._evaluation_step(**kwargs)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step
    output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1494, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 390, in validation_step
    return self.model.validation_step(*args, **kwargs)
  File "/mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/anomalib/models/patchcore/lightning_model.py", line 109, in validation_step
    anomaly_maps, anomaly_score = self.model(batch["image"])
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/anomalib/models/patchcore/torch_model.py", line 90, in forward
    anomaly_map = self.anomaly_map_generator(patch_scores)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/anomalib/models/patchcore/anomaly_map.py", line 55, in forward
    anomaly_map = self.compute_anomaly_map(patch_scores)
  File "/mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/anomalib/models/patchcore/anomaly_map.py", line 38, in compute_anomaly_map
    anomaly_map = self.blur(anomaly_map)
  File "/home/vk/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/anomalib/models/components/filters/blur.py", line 87, in forward
    output = F.conv2d(input_tensor, self.kernel, groups=self.channels, padding=0, stride=1)
RuntimeError: GET was unable to find an engine to execute this computation
Epoch 0:  69%|██████▉   | 11/16 [00:10<00:04,  1.03it/s, loss=nan]

config file used :
dataset:
  name: data
  format: folder
  path: /mnt/8014A82114A81C62/Vishal/Anomaly-MPI/anomalib-0.4.0/custom_dataset
  normal_dir: Good # name of the folder containing normal images.
  abnormal_dir: Anomaly # name of the folder containing abnormal images.
  normal_test_dir: null # name of the folder containing normal test images.
  task: segmentation # classification or segmentation
  mask: Mask
  extensions: null
  normalization: imagenet
  split_ratio: 0.2 # ratio of the normal images that will be used to create a test split
  image_size: 256
  train_batch_size: 8
  test_batch_size: 8
  num_workers: 8
  transform_config:
    train: null
    val: null
  create_validation_set: true
  test_split_mode: from_dir # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

model:
  name: patchcore
  backbone: wide_resnet50_2
  pre_trained: true
  layers:
    - layer2
    - layer3
  coreset_sampling_ratio: 0.1
  num_neighbors: 9
  normalization_method: min_max # options: [null, min_max, cdf]

metrics:
  image:
    - F1Score
    - AUROC
  pixel:
    - F1Score
    - AUROC
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 0
  path: ./results

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: null # options: onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 1
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 1
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0 # Don't validate before extracting features.
  log_every_n_steps: 50
  accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: false
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

@vishalk2999, can you check your pytorch and cuda versions? I haven't experienced this issue before, but based on a quick search, it sounds like it might be related to that.
https://stackoverflow.com/questions/75776895/runtimeerror-get-was-unable-to-find-an-engine-to-execute-this-computation-when
https://github.com/microsoft/visual-chatgpt/issues/283

@samet-akcay the cuda version in nvidia-smi and nvcc are matching and the cuda version is 11.3. Moreover, I am able to train using GPU when I am using the default config file for training patchcore using MvTEC dataset. I am getting this issue only when I am using a custom dataset

Please see this discussion
https://stackoverflow.com/questions/75776895/runtimeerror-get-was-unable-to-find-an-engine-to-execute-this-computation-when
This could potentially be due to your cuda installation or gpu memory issue when you try to fit a large custom dataset into your memory.
I'm closing this since this is not a bug. Feel free to re-open if you face any other problems

Try entering the code below.

$ conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia


Test Issue created at Thu, 03 Jun 2021 11:55:56 GMT

Test issue body  Thu, 03 Jun 2021 11:55:56 GMT

Comment on issue at Thu, 03 Jun 2021 11:55:57 GMT


Droolon F1 passed calibration of Vive pro eye but not working with the eye tracking

The Droolon F1 calibration using the Vive Pro Eye overlay in the steamVR dashboard. When loading the VRCfacetrack osc with "is vpe" removed, only blinking is working in the VRchat. Eye widening, eye squeeze, and eye tracking is not working. The Droolon drivers created some files inside the eye_camera_device folder inside SRanipal folder in Program Files when I run the Droolon driver setup. I have attached these files under this issue as google drive link.
https://drive.google.com/file/d/16a4xzKIDGViLQec4kzlEibaPQHfRTyFD/view?usp=sharing

I worked with someone who has the Droolon F1 very recently, we've managed to get the Droolon F1 fully functional with a modified version of VRCFaceTracking. Looks like the way the F1 is initialized seems really flakey, causing some parameters to not load properly sometimes (like gaze). Will be working with them further to hopefully get the F1 to be in a usable state.

Also they were using it with the Vive Cosmos Elite completely unmodified just to add, with no lip tracking at the moment

Removed the IsViveProEye check in latest update, hopefully this fixes your issue!


Using Python function as macro.

I can't seem to figure out how to use a Python function as a macro. I've tried using a function like:
def if_cond(arg, caller=None):
called as:
{% call if_cond("bar") %}
    APPLE
{% endcall %}

But, it seems that caller is <Macro anonymous>, which doesn't seem to have any way to retrieve the body. Looking at the source, it appears that the Macro object should have a body attribute, however it does not.

Macros are a special construct produces by a Jinja macro block. call is not intended to be used on Python functions passed in to the template.

@Dreamsorcerer What you want to do (at least in the example, can't see what is the full use case) can be achieved with a filter block:
import jinja2

def if_cond(body, arg):
    # do something
    return arg + body.strip() + arg

e = jinja2.Environment()
e.filters["if_cond"] = if_cond

template = e.from_string(
"""{% filter if_cond('bar') %}
    APPLE
{% endfilter %}""")

print(template.render())
# --> barAPPLEbar


[BUG]: The thread of the dbgpt_server.py service does not close

Describe the bug
dbgpt_server.py has been started for 3 days, and the number of threads has been increasing. As of today, it has increased to 15,000. It seems that the threads are only opened and not closed.
The start command for the service is:
nohup python dbgpt_server.py --host xxx --port xx > /home/debug.txt 2>&1 &
Expected behavior
The service can release the thread in time.
Additional context

Already fixed, just have to try the latest version.


Fixed unary not for pointers #316

For #316

This change is

Reviewed 2 of 2 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Add support for throwing errors with custom error code

Motivation:
In the rsocket-js implementation (v0 branch), all stream errors are serialized and transmitted to other end with fixed error code. Due to this bug, there is no way to send custom error codes over stream (Bug report https://github.com/rsocket/rsocket-js/issues/218)
Custom error codes are supported in the rsocket protocol and implemented in rsocket-java this using RSocketErrorException . When the application throws RSocketErrorExecption, the error code of the exception is sent over the stream. But this feature is not implemented in rsocket-js.
Modifications:

This PR adds custom Javascript Error class RSocketError in RSocketMachine.js.
When the application throws error of type RSocketError, the errorCode of the error object is sent over the stream.

Result:
This will allow application to send custom error codes to other party.

Hey @palamccc thanks for the contribution!
Would you be able to add tests for your changes? Additionally the DCO check will ideally need to be resolved before this can be merged.

Would you be able to add tests for your changes? Looks like RSocketServer-test.js might have good examples to build off of.

@viglucci  I've added tests now.

LGTM. Thanks!

I contributed types to DefinitelyTyped https://github.com/DefinitelyTyped/DefinitelyTyped/pull/68788
Waiting for 0.0.28 release :-).

@palamccc I've published a new version which should include this change. If you would be so kind, please validate this change with<EMAIL_ADDRESS>


Missing Git dependency for Kustomize. How to install Git on K3OS?

K3OS is supposed to be a minimal OS with only the packages needed to run Kubernetes. Apparently, this is not true because Kustomize as part of kubectl is also part of Kubernetes which relies on Git to fetch base config files through go-getter, but Git is not included in the K3OS image. This means that kubectl will crash when trying to deploy Kustomize manifests with certain URI formats (for example github.com/fluxcd/flux//deploy?ref=v1.19.0).
What is the best way to install git without building K3OS from scratch? I'm using Packer to create my own images from the K3OS base image, but I can't use it to install Git because K3OS doesn't come with a package manager.

You mean I should run kubectl from inside a pod? That doesn't sound like a good idea. I need git as a dependency to kubectl itself and not to some third party application that you would usually run in a pod.

yes, if you do not  have an additional machine, e.g a laptop,  to run kubectl outside of the cluster.  Please note that k3os is kind of immutable OS, it dose this by purpose to ensure system stability. If u wanna continue the traditional way of operations, install k3s into traditional os instead, e.g. alpine, ubuntu ... , but not taking k3os.

So you've got a few options here,
One is to include the kustomize and git binaries as part of your generation of your ISO and do some kustomize dance as part of runtime init and stick the resulting manifests in /var/lib/rancher/k3s/server/manifests
Or two is do the kustomization of your git-required remote base manifests before you write out the ISO and include it in a directory in which is copied to the k3os host from k3os/system and either copy them over at runtime, or run further kustomizations on them to stick the resulting output into /var/lib/rancher/k3s/server/manifests
Hope this helps


Highlight route point when hovering elevation/altitude chart

Highlight route point when hovering elevation/altitude chart

Done!


Merge google/yapf#1067, treat yapf.third_party as a third_party in isort

Merge google/yapf#1067, treat yapf.third_party as a third_party in isort

Lol, this actually merges in google/yapf#1093 😆


BanditScheduler: Add emitter_pool and active attr; remove emitters attr

Description

Currently, the BanditScheduler has an emitters attr that returns the _active_arr variable. This does not make much sense since BanditScheduler does not have emitters as input, and _active_arr gives emitters a different meaning from the emitters attribute of Scheduler.
Thus, this PR removes the emitters attr and instead adds emitter_pool and active to indicate the pool of emitters in the scheduler and which ones are currently active, respectively.
TODO

[x] Remove BanditScheduler.emitters
[x] Add BanditScheduler.emitter_pool
[x] Add BanditScheduler.active

Status

[x] I have read the guidelines in
CONTRIBUTING.md
[x] I have formatted my code using yapf
[x] I have tested my code by running pytest
[x] I have linted my code with pylint
[x] I have added a one-line description of my change to the changelog in
HISTORY.md
[x] This PR is ready to go

Will merge once #489 is resolved.