approach curating duplicate literature entries

In the process of merging the various literature corporate shared by DeeAnn, Nancy and Kendra, some duplicates appear in the batlit v0.1 .
For instance,

id
authors
date
title
journal
doi

https://www.zotero.org/groups/bat_literature_project/items/I8FSKML3
Goldstein | Anthony | Gbakima | Bird | Bangura | Tremeau-Bravard | Belaganahalli | Wells | Dhanota | Liang | Grodus | Jangra | DeJesus | Lasso | Smith | Jambai | Kamara | Kamara | Bangura | Monagin | Shapira | Johnson | Saylors | Rubin | Chandran | Lipkin | Mazet
2018-08-27
The discovery of Bombali virus adds further support for bats as hosts of ebolaviruses
Nature Microbiology
10.1038/s41564-018-0227-2

https://www.zotero.org/groups/bat_literature_project/items/7CL5RBDS
Goldstein | Anthony | Gbakima | Bird | Bangura | Tremeau-Bravard | Belaganahalli | Wells | Dhanota | Liang | Grodus | Jangra | DeJesus | Lasso | Smith | Jambai | Kamara | Kamara | Bangura | Monagin | Shapira | Johnson | Saylors | Rubin | Chandran | Lipkin | Mazet
2018-10
The discovery of Bombali virus adds further support for bats as hosts of ebolaviruses
Nature Microbiology
10.1038/s41564-018-0227-2

@ajacsherman as your role of curator of the batlit corpus, what is your approach to handling duplicates?
Some options I can think of :

merge records
link duplicate records
do nothing

Also, I assume that updates are expected to come in from various different sources . . . making further duplications likely. How do you imagine handling updates?

Hi Jorrit, I have been deduplicating within our contributor's individual libraries by merging the records. I am making copies of these named folders (i.e. "Reeder") in my own personal folders so I can keep a record of who contributed what. Now that I have these archived, I can eliminate the duplicates for the complete Bat Literature Collection. Will that help? Going forward, I plan to curate individual collections in a separate folder and then add the pdfs to the larger shared folder once my curation has been completed. I can work on that right now if it moves your process forward?

@ajacsherman thanks for sharing how you are handling duplicates.
Can you provide one or two examples of such merged duplicates?

Abdelgawad A, Damiani A, Ho SY, Strauss G, Szentiks CA, East ML, Osterrieder N, Greenwood AD. Zebra Alphaherpesviruses (EHV-1 and EHV-9): Genetic Diversity, Latency and Co-Infections. Viruses. 2016 Sep 20;8(9):262. doi: 10.3390/v8090262. PMID: 27657113; PMCID: PMC5035975.

With your example, I was able to trace down the original duplicate records related to doi:10.3390/v8090262
via
preston ls --algo md5\
 | grep items?\
 | preston grep "10.3390/v8090262"\
 | grep value

yielding the expected two search hits -
<line:hash://md5/1eeb212dfd65e7549522aef68497604d!/L1072> <http://www.w3.org/ns/prov#value> "            \"DOI\": \"10.3390/v8090262\"," <urn:uuid:34bd5285-9e7d-426b-b1fd-996516be6a7f> .
<line:hash://md5/0f362972b3eb321073cee168c180c307!/L9756> <http://www.w3.org/ns/prov#value> "            \"DOI\": \"10.3390/v8090262\"," <urn:uuid:8a08d8a3-9390-4d51-bc15-309183fccdab> .

curl 'https://linker.bio/line:hash://md5/0f362972b3eb321073cee168c180c307!/L9653-L9802'

    {
        "key": "2PWXAVQL",
        "version": 803,
        "library": {
            "type": "group",
            "id": 5435545,
            "name": "Bat Literature Project",
            "links": {
                "alternate": {
                    "href": "https://www.zotero.org/groups/bat_literature_project",
                    "type": "text/html"
                }
            }
        },
        "links": {
            "self": {
                "href": "https://api.zotero.org/groups/5435545/items/2PWXAVQL",
                "type": "application/json"
            },
            "alternate": {
                "href": "https://www.zotero.org/groups/bat_literature_project/items/2PWXAVQL",
                "type": "text/html"
            },
            "attachment": {
                "href": "https://api.zotero.org/groups/5435545/items/6AZNRAQN",
                "type": "application/json",
                "attachmentType": "text/html"
            }
        },
        "meta": {
            "createdByUser": {
                "id": 6296343,
                "username": "deeannreeder",
                "name": "",
                "links": {
                    "alternate": {
                        "href": "https://www.zotero.org/deeannreeder",
                        "type": "text/html"
                    }
                }
            },
            "creatorSummary": "Abdelgawad et al.",
            "parsedDate": "2016-09",
            "numChildren": 2
        },
        "data": {
            "key": "2PWXAVQL",
            "version": 803,
            "itemType": "journalArticle",
            "title": "Zebra Alphaherpesviruses (EHV-1 and EHV-9): Genetic Diversity, Latency and Co-Infections",
            "creators": [
                {
                    "creatorType": "author",
                    "firstName": "Azza",
                    "lastName": "Abdelgawad"
                },
                {
                    "creatorType": "author",
                    "firstName": "Armando",
                    "lastName": "Damiani"
                },
                {
                    "creatorType": "author",
                    "firstName": "Simon Y. W.",
                    "lastName": "Ho"
                },
                {
                    "creatorType": "author",
                    "firstName": "Günter",
                    "lastName": "Strauss"
                },
                {
                    "creatorType": "author",
                    "firstName": "Claudia A.",
                    "lastName": "Szentiks"
                },
                {
                    "creatorType": "author",
                    "firstName": "Marion L.",
                    "lastName": "East"
                },
                {
                    "creatorType": "author",
                    "firstName": "Nikolaus",
                    "lastName": "Osterrieder"
                },
                {
                    "creatorType": "author",
                    "firstName": "Alex D.",
                    "lastName": "Greenwood"
                }
            ],
            "abstractNote": "Alphaherpesviruses are highly prevalent in equine populations and co-infections with more than one of these viruses’ strains frequently diagnosed. Lytic replication and latency with subsequent reactivation, along with new episodes of disease, can be influenced by genetic diversity generated by spontaneous mutation and recombination. Latency enhances virus survival by providing an epidemiological strategy for long-term maintenance of divergent strains in animal populations. The alphaherpesviruses equine herpesvirus 1 (EHV-1) and 9 (EHV-9) have recently been shown to cross species barriers, including a recombinant EHV-1 observed in fatal infections of a polar bear and Asian rhinoceros. Little is known about the latency and genetic diversity of EHV-1 and EHV-9, especially among zoo and wild equids. Here, we report evidence of limited genetic diversity in EHV-9 in zebras, whereas there is substantial genetic variability in EHV-1. We demonstrate that zebras can be lytically and latently infected with both viruses concurrently. Such a co-occurrence of infection in zebras suggests that even relatively slow-evolving viruses such as equine herpesviruses have the potential to diversify rapidly by recombination. This has potential consequences for the diagnosis of these viruses and their management in wild and captive equid populations.",
            "publicationTitle": "Viruses",
            "volume": "8",
            "issue": "9",
            "pages": "262",
            "date": "2016/9",
            "series": "",
            "seriesTitle": "",
            "seriesText": "",
            "journalAbbreviation": "",
            "language": "en",
            "DOI": "10.3390/v8090262",
            "ISSN": "",
            "shortTitle": "Zebra Alphaherpesviruses (EHV-1 and EHV-9)",
            "url": "https://www.mdpi.com/1999-4915/8/9/262",
            "accessDate": "2021-01-22T15:11:19Z",
            "archive": "",
            "archiveLocation": "",
            "libraryCatalog": "www.mdpi.com",
            "callNumber": "",
            "rights": "http://creativecommons.org/licenses/by/3.0/",
            "extra": "Number: 9\nPublisher: Multidisciplinary Digital Publishing Institute",
            "tags": [
                {
                    "tag": "EHV-1",
                    "type": 1
                },
                {
                    "tag": "EHV-9",
                    "type": 1
                },
                {
                    "tag": "co-occurrence",
                    "type": 1
                },
                {
                    "tag": "diversity",
                    "type": 1
                },
                {
                    "tag": "latency",
                    "type": 1
                },
                {
                    "tag": "zebra",
                    "type": 1
                }
            ],
            "collections": [
                "DZKBQXJR"
            ],
            "relations": {
                "owl:sameAs": "http://zotero.org/groups/2719577/items/WNTGIJCX"
            },
            "dateAdded": "2024-03-07T00:47:03Z",
            "dateModified": "2024-03-07T00:47:03Z"
        }
    },

and
preston cat 'line:hash://md5/0f362972b3eb321073cee168c180c307!/L9652-L9802'

or https://linker.bio/line:hash://md5/0f362972b3eb321073cee168c180c307!/L9652-L9802
producing:
    },
    {
        "key": "2PWXAVQL",
        "version": 803,
        "library": {
            "type": "group",
            "id": 5435545,
            "name": "Bat Literature Project",
            "links": {
                "alternate": {
                    "href": "https://www.zotero.org/groups/bat_literature_project",
                    "type": "text/html"
                }
            }
        },
        "links": {
            "self": {
                "href": "https://api.zotero.org/groups/5435545/items/2PWXAVQL",
                "type": "application/json"
            },
            "alternate": {
                "href": "https://www.zotero.org/groups/bat_literature_project/items/2PWXAVQL",
                "type": "text/html"
            },
            "attachment": {
                "href": "https://api.zotero.org/groups/5435545/items/6AZNRAQN",
                "type": "application/json",
                "attachmentType": "text/html"
            }
        },
        "meta": {
            "createdByUser": {
                "id": 6296343,
                "username": "deeannreeder",
                "name": "",
                "links": {
                    "alternate": {
                        "href": "https://www.zotero.org/deeannreeder",
                        "type": "text/html"
                    }
                }
            },
            "creatorSummary": "Abdelgawad et al.",
            "parsedDate": "2016-09",
            "numChildren": 2
        },
        "data": {
            "key": "2PWXAVQL",
            "version": 803,
            "itemType": "journalArticle",
            "title": "Zebra Alphaherpesviruses (EHV-1 and EHV-9): Genetic Diversity, Latency and Co-Infections",
            "creators": [
                {
                    "creatorType": "author",
                    "firstName": "Azza",
                    "lastName": "Abdelgawad"
                },
                {
                    "creatorType": "author",
                    "firstName": "Armando",
                    "lastName": "Damiani"
                },
                {
                    "creatorType": "author",
                    "firstName": "Simon Y. W.",
                    "lastName": "Ho"
                },
                {
                    "creatorType": "author",
                    "firstName": "Günter",
                    "lastName": "Strauss"
                },
                {
                    "creatorType": "author",
                    "firstName": "Claudia A.",
                    "lastName": "Szentiks"
                },
                {
                    "creatorType": "author",
                    "firstName": "Marion L.",
                    "lastName": "East"
                },
                {
                    "creatorType": "author",
                    "firstName": "Nikolaus",
                    "lastName": "Osterrieder"
                },
                {
                    "creatorType": "author",
                    "firstName": "Alex D.",
                    "lastName": "Greenwood"
                }
            ],
            "abstractNote": "Alphaherpesviruses are highly prevalent in equine populations and co-infections with more than one of these viruses’ strains frequently diagnosed. Lytic replication and latency with subsequent reactivation, along with new episodes of disease, can be influenced by genetic diversity generated by spontaneous mutation and recombination. Latency enhances virus survival by providing an epidemiological strategy for long-term maintenance of divergent strains in animal populations. The alphaherpesviruses equine herpesvirus 1 (EHV-1) and 9 (EHV-9) have recently been shown to cross species barriers, including a recombinant EHV-1 observed in fatal infections of a polar bear and Asian rhinoceros. Little is known about the latency and genetic diversity of EHV-1 and EHV-9, especially among zoo and wild equids. Here, we report evidence of limited genetic diversity in EHV-9 in zebras, whereas there is substantial genetic variability in EHV-1. We demonstrate that zebras can be lytically and latently infected with both viruses concurrently. Such a co-occurrence of infection in zebras suggests that even relatively slow-evolving viruses such as equine herpesviruses have the potential to diversify rapidly by recombination. This has potential consequences for the diagnosis of these viruses and their management in wild and captive equid populations.",
            "publicationTitle": "Viruses",
            "volume": "8",
            "issue": "9",
            "pages": "262",
            "date": "2016/9",
            "series": "",
            "seriesTitle": "",
            "seriesText": "",
            "journalAbbreviation": "",
            "language": "en",
            "DOI": "10.3390/v8090262",
            "ISSN": "",
            "shortTitle": "Zebra Alphaherpesviruses (EHV-1 and EHV-9)",
            "url": "https://www.mdpi.com/1999-4915/8/9/262",
            "accessDate": "2021-01-22T15:11:19Z",
            "archive": "",
            "archiveLocation": "",
            "libraryCatalog": "www.mdpi.com",
            "callNumber": "",
            "rights": "http://creativecommons.org/licenses/by/3.0/",
            "extra": "Number: 9\nPublisher: Multidisciplinary Digital Publishing Institute",
            "tags": [
                {
                    "tag": "EHV-1",
                    "type": 1
                },
                {
                    "tag": "EHV-9",
                    "type": 1
                },
                {
                    "tag": "co-occurrence",
                    "type": 1
                },
                {
                    "tag": "diversity",
                    "type": 1
                },
                {
                    "tag": "latency",
                    "type": 1
                },
                {
                    "tag": "zebra",
                    "type": 1
                }
            ],
            "collections": [
                "DZKBQXJR"
            ],
            "relations": {
                "owl:sameAs": "http://zotero.org/groups/2719577/items/WNTGIJCX"
            },
            "dateAdded": "2024-03-07T00:47:03Z",
            "dateModified": "2024-03-07T00:47:03Z"
        }
    },

Note, however, that one of the original records available in the v0.2 version of batlit appears to be available in the current version as accessed on 2024-06-03 -
https://www.zotero.org/groups/5435545/bat_literature_project/items/YWNCWPYJ
and one is no longer available:
https://www.zotero.org/groups/5435545/bat_literature_project/items/2PWXAVQL
And, luckily, Zotero keeps a record of the replaced record 2PWXAVQL in statement "dc:replaces": "http://zotero.org/groups/5435545/items/2PWXAVQL", in the updated metadata retrieved via
ZOTERO_TOKEN=[SECRET] preston track https://api.zotero.org/groups/5435545/items/YWNCWPYJ

containing
{
    "key": "YWNCWPYJ",
    "version": 11620,
    "library": {
        "type": "group",
        "id": 5435545,
        "name": "Bat Literature Project",
        "links": {
            "alternate": {
                "href": "https://www.zotero.org/groups/bat_literature_project",
                "type": "text/html"
            }
        }
    },
    "links": {
        "self": {
            "href": "https://api.zotero.org/groups/5435545/items/YWNCWPYJ",
            "type": "application/json"
        },
        "alternate": {
            "href": "https://www.zotero.org/groups/bat_literature_project/items/YWNCWPYJ",
            "type": "text/html"
        },
        "attachment": {
            "href": "https://api.zotero.org/groups/5435545/items/JGTYJ4TR",
            "type": "application/json",
            "attachmentType": "application/pdf",
            "attachmentSize": 1609849
        }
    },
    "meta": {
        "createdByUser": {
            "id": 13229919,
            "username": "acsherman",
            "name": "",
            "links": {
                "alternate": {
                    "href": "https://www.zotero.org/acsherman",
                    "type": "text/html"
                }
            }
        },
        "creatorSummary": "Abdelgawad et al.",
        "parsedDate": "2016-09-20",
        "numChildren": 3
    },
    "data": {
        "key": "YWNCWPYJ",
        "version": 11620,
        "itemType": "journalArticle",
        "title": "Zebra Alphaherpesviruses (EHV-1 and EHV-9): Genetic Diversity, Latency and Co-Infections",
        "creators": [
            {
                "creatorType": "author",
                "firstName": "Azza",
                "lastName": "Abdelgawad"
            },
            {
                "creatorType": "author",
                "firstName": "Armando",
                "lastName": "Damiani"
            },
            {
                "creatorType": "author",
                "firstName": "Simon",
                "lastName": "Ho"
            },
            {
                "creatorType": "author",
                "firstName": "Günter",
                "lastName": "Strauss"
            },
            {
                "creatorType": "author",
                "firstName": "Claudia",
                "lastName": "Szentiks"
            },
            {
                "creatorType": "author",
                "firstName": "Marion",
                "lastName": "East"
            },
            {
                "creatorType": "author",
                "firstName": "Nikolaus",
                "lastName": "Osterrieder"
            },
            {
                "creatorType": "author",
                "firstName": "Alex",
                "lastName": "Greenwood"
            }
        ],
        "abstractNote": "Alphaherpesviruses are highly prevalent in equine populations and co-infections with more than one of these viruses’ strains frequently diagnosed. Lytic replication and latency with subsequent reactivation, along with new episodes of disease, can be inﬂuenced by genetic diversity generated by spontaneous mutation and recombination. Latency enhances virus survival by providing an epidemiological strategy for long-term maintenance of divergent strains in animal populations. The alphaherpesviruses equine herpesvirus 1 (EHV-1) and 9 (EHV-9) have recently been shown to cross species barriers, including a recombinant EHV-1 observed in fatal infections of a polar bear and Asian rhinoceros. Little is known about the latency and genetic diversity of EHV-1 and EHV-9, especially among zoo and wild equids. Here, we report evidence of limited genetic diversity in EHV-9 in zebras, whereas there is substantial genetic variability in EHV-1. We demonstrate that zebras can be lytically and latently infected with both viruses concurrently. Such a co-occurrence of infection in zebras suggests that even relatively slow-evolving viruses such as equine herpesviruses have the potential to diversify rapidly by recombination. This has potential consequences for the diagnosis of these viruses and their management in wild and captive equid populations.",
        "publicationTitle": "Viruses",
        "volume": "8",
        "issue": "9",
        "pages": "262",
        "date": "2016-09-20",
        "series": "",
        "seriesTitle": "",
        "seriesText": "",
        "journalAbbreviation": "Viruses",
        "language": "en",
        "DOI": "10.3390/v8090262",
        "ISSN": "1999-4915",
        "shortTitle": "Zebra Alphaherpesviruses (EHV-1 and EHV-9)",
        "url": "http://www.mdpi.com/1999-4915/8/9/262",
        "accessDate": "2024-04-18T21:02:03Z",
        "archive": "",
        "archiveLocation": "",
        "libraryCatalog": "DOI.org (Crossref)",
        "callNumber": "",
        "rights": "https://creativecommons.org/licenses/by/4.0/",
        "extra": "",
        "tags": [
            {
                "tag": "EHV-1",
                "type": 1
            },
            {
                "tag": "EHV-9",
                "type": 1
            },
            {
                "tag": "co-occurrence",
                "type": 1
            },
            {
                "tag": "diversity",
                "type": 1
            },
            {
                "tag": "latency",
                "type": 1
            },
            {
                "tag": "zebra",
                "type": 1
            }
        ],
        "collections": [
            "UAWY6DNP"
        ],
        "relations": {
            "dc:replaces": "http://zotero.org/groups/5435545/items/2PWXAVQL",
            "owl:sameAs": "http://zotero.org/groups/2719577/items/WNTGIJCX"
        },
        "dateAdded": "2024-03-07T00:47:03Z",
        "dateModified": "2024-05-30T19:18:31Z"
    }
}

Which gives us a possible strategy for dealing with duplicate literature entities:

Aja (or other curator) deduplicates entries using the Zotero merge tool
Jorrit (or some other data archivist) take a snapshot of the Zotero group
Preston (or some other robot) detects "dc:replaces": "http://zotero.org/groups/5435545/items/2PWXAVQL" and translates this into an action to annotate any existing Zenodo record associated with http://zotero.org/groups/5435545/items/2PWXAVQL as deprecated and being replaced by https://www.zotero.org/groups/bat_literature_project/items/YWNCWPYJ .

@ajacsherman @myrmoteras I've added the proposed deduplication workflow for your review.

@ajacsherman @jhpoelen can we add a deduplication step by comparing what is in BLR, coviho community?
We also need to consider how to do this routinely if we allow others to upload records to the batlit community.
May be we should deduplicate against the entire Zenodo?


How to select ROI and display in another window?

I have a las file, I need to crop select ROI and then to display it another window. How can we do it. Can anyone please guide me? Looking for positive response

I suggest learning how to do it in PCL first... The pclpy algorithm should be very similar.
Although the current limitation I see is related to issue #6. Because the selection of points will likely need some sort of callback function. I plan to look at it when I get the time.

Ok. But I couldn't understand the link which you mentioned. Can you please elaborate? :)

It's a reference for myself on how to implement the binding of c++ functions taking function pointer as arguments. So it's not intended for a pclpy user.

In the newest version 0.7.0, I implemented some functions that take a callback as an argument. So I think you can do what you want to achieve. Try the following (in the main viewer, press x to toggle rectangle selection, and select an area with the left mouse button)

def test_open_in_new_window():
    pc = pclpy.io.read(test_data("street.las"), "PointXYZRGBA")
    viewer = pcl.visualization.PCLVisualizer("viewer", False)
    viewer.addPointCloud(pc)
    viewer.resetCamera()

    viewers = [viewer]

    def handle_event_area(event):
        # use x to toggle rectangle selection
        assert isinstance(event, pcl.visualization.AreaPickingEvent)
        indices = pcl.vectors.Int()
        event.getPointsIndices(indices)

        other_viewer = pcl.visualization.PCLVisualizer("viewer", False)
        rgb = np.array([pc.r[indices], pc.g[indices], pc.b[indices]]).T
        other_pc = pcl.PointCloud.PointXYZRGBA.from_array(pc.xyz[indices], rgb)
        other_viewer.addPointCloud(other_pc)
        other_viewer.resetCamera()
        viewers.append(other_viewer)

    viewer.registerAreaPickingCallback(handle_event_area)

    while not all(v.wasStopped() for v in viewers):
        for v in viewers:
            if not v.wasStopped():
                v.spinOnce(50)

test_open_in_new_window()

It throws this error NameError: name 'test_data' is not defined
import pclpy
from pclpy import *

def test_open_in_new_window():
    pc = pclpy.io.read(test_data("E:/Lidar-practice/new-files/las14_type7_ALS_RIEGL_Q680i.las"), "PointXYZRGBA")
    viewer = pcl.visualization.PCLVisualizer("viewer", False)
    viewer.addPointCloud(pc)
    viewer.resetCamera()

    viewers = [viewer]

    def handle_event_area(event):
        # use x to toggle rectangle selection
        assert isinstance(event, pcl.visualization.AreaPickingEvent)
        indices = pcl.vectors.Int()
        event.getPointsIndices(indices)

        other_viewer = pcl.visualization.PCLVisualizer("viewer", False)
        rgb = np.array([pc.r[indices], pc.g[indices], pc.b[indices]]).T
        other_pc = pcl.PointCloud.PointXYZRGBA.from_array(pc.xyz[indices], rgb)
        other_viewer.addPointCloud(other_pc)
        other_viewer.resetCamera()
        viewers.append(other_viewer)

    viewer.registerAreaPickingCallback(handle_event_area)

    while not all(v.wasStopped() for v in viewers):
        for v in viewers:
            if not v.wasStopped():
                v.spinOnce(50)

test_open_in_new_window()

I added few lines in the code
import pclpy
from pclpy import *
from pclpy import pcl
import os
import pclpy.view.vtk

def test_data(*args):
    return os.path.join("test_data", *args)

def test_open_in_new_window():
    pc = pclpy.io.read(test_data("E:/Lidar-practice/new-files/las14_type7_ALS_RIEGL_Q680i.las"), "PointXYZRGBA")
    viewer = pcl.visualization.PCLVisualizer("viewer", False)
    viewer.addPointCloud(pc)
    viewer.resetCamera()

    viewers = [viewer]

    def handle_event_area(event):
        # use x to toggle rectangle selection
        assert isinstance(event, pcl.visualization.AreaPickingEvent)
        indices = pcl.vectors.Int()
        event.getPointsIndices(indices)

        other_viewer = pcl.visualization.PCLVisualizer("viewer", False)
        rgb = np.array([pc.r[indices], pc.g[indices], pc.b[indices]]).T
        other_pc = pcl.PointCloud.PointXYZRGBA.from_array(pc.xyz[indices], rgb)
        other_viewer.addPointCloud(other_pc)
        other_viewer.resetCamera()
        viewers.append(other_viewer)

    viewer.registerAreaPickingCallback(handle_event_area)

    while not all(v.wasStopped() for v in viewers):
        for v in viewers:
            if not v.wasStopped():
                v.spinOnce(50)

test_open_in_new_window()

I am not getting any error. But no output is being generated.

Yes, sorry about that, the second argument to the constructor of PCLVisualizer should be True (create_interactor=True). I set it to False in unit tests so that no window is displayed.
Also, remove the test_data function call. This is only a helper in unit tests.
def test_open_in_new_window():
    pc = pclpy.io.read("E:/Lidar-practice/new-files/las14_type7_ALS_RIEGL_Q680i.las", "PointXYZRGBA")
    viewer = pcl.visualization.PCLVisualizer("viewer", True)
    viewer.addPointCloud(pc)
    viewer.resetCamera()

    viewers = [viewer]

    def handle_event_area(event):
        # use x to toggle rectangle selection
        assert isinstance(event, pcl.visualization.AreaPickingEvent)
        indices = pcl.vectors.Int()
        event.getPointsIndices(indices)

        other_viewer = pcl.visualization.PCLVisualizer("viewer", True)
        rgb = np.array([pc.r[indices], pc.g[indices], pc.b[indices]]).T
        other_pc = pcl.PointCloud.PointXYZRGBA.from_array(pc.xyz[indices], rgb)
        other_viewer.addPointCloud(other_pc)
        other_viewer.resetCamera()
        viewers.append(other_viewer)

    viewer.registerAreaPickingCallback(handle_event_area)

    while not all(v.wasStopped() for v in viewers):
        for v in viewers:
            if not v.wasStopped():
                v.spinOnce(50)

test_open_in_new_window()

@davidcaron, how to specify Intensity in the code. Can I change RGBA to I in the code
 pc = pclpy.io.read("E:/Lidar-practice/new-files/las14_type7_ALS_RIEGL_Q680i.las", "PointXYZRGBA")
 pc = pclpy.io.read("E:/Lidar-practice/new-files/las14_type7_ALS_RIEGL_Q680i.las", "PointXYZI")

The colors shouldn't be read when you specify PointXYZI
when I try pc = pclpy.io.read(".../las14_type7_ALS_RIEGL_Q680i.las", "PointXYZI") with your file it seems to work fine...
can you paste the whole traceback message?

@davidcaron, the error will occur when I try to select ROI, I guess the problem is with this code,
other_viewer = pcl.visualization.PCLVisualizer("viewer", True)         rgb = np.array([pc.r[indices], pc.g[indices], pc.b[indices]]).T         other_pc = pcl.PointCloud.PointXYZRGBA.from_array(pc.xyz[indices], rgb)         other_viewer.addPointCloud(other_pc)         other_viewer.resetCamera()         viewers.append(other_viewer)


py23 module use within the project

I took a quick look into what it would take to part ways with the py23 module imports within the project and it is significant.  The following is a ballpark estimate of matched strings in modules under Lib/fontTools/ that does not account for a py23 module import statement in the match.  This may be an over count for the strings that are Python builtins/stdlib.

basestring: 43 matches across 19 files
bytechr: 65 matches across 16 files
byteord: 112 matches across 32 files
BytesIO: 35 matches across 19 files
bytesjoin: 89 matches across 32 files
open: 277 matches across 49 files
Py23Error: 3 matches in 1 file
range: 897 matches across 109 files
RecursionError: 4 matches across 2 files
round: 302 matches across 50 files
SimpleNamespace: 18 matches across 4 files
StringIO: 9 matches across 7 files
strjoin: 24 matches across 15 files
Tag: 744 matches across 47 files
tobytes: 70 matches across 37 files
tostr: 31 matches across 18 files
tounicode: 33 matches across 13 files
UnicodeIO: 5 matches across 3 files
xrange: 3 matches in 1 file
zip: 176 matches across 39 files

Any interest in beginning to chip away at this?
Related #2231

Some of these are easy to fix. basestring should be unconditionally replace with str, UnicodeIO with StringIO and both should be imported from io module together with BytesIO. SimpleNamespace should be imported from types module.
open, round, range, zip need no change, py23 is already using the Python 3 builtin, so removing the import is all needed.
bytechr, byteord, bytesjoin, strjoin, tobytes, tostr and Tag class seem useful and should probably be moved to a different module (misc.textTools?). tounicode is another name for tostr and should be simply replaced with it.

basestring to str https://github.com/fonttools/fonttools/pull/2241


broke build in  -- Performing Test HAVE_PCLMULQDQ_INTRIN - Failed

-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN - Success
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL - Success
-- Performing Test HAVE_BUILTIN_CTZL
-- Performing Test HAVE_BUILTIN_CTZL - Success
-- Performing Test HAVE_SSE2_INTRIN
-- Performing Test HAVE_SSE2_INTRIN - Success
-- Performing Test HAVE_SSE42_INTRIN
-- Performing Test HAVE_SSE42_INTRIN - Success
-- Performing Test HAVE_PCLMULQDQ_INTRIN
-- Performing Test HAVE_PCLMULQDQ_INTRIN - Failed
-- Architecture-specific source files: arch/x86/x86.c;arch/x86/insert_string_sse.c;arch/x86/fill_window_sse.c
-- Building: tcmalloc_minimal_internal
-- Link libtcmalloc_minimal : tcmalloc_minimal_internal
-- Using mysqlclient: /usr/include : ; static=/usr/lib64/libmysqlclient.a
-- Will build ClickHouse 1.1.54187
-- Using internal compiler: headers=/usr/share/clickhouse/headers : /usr/share/clickhouse/bin/clang   -D_GLIBCXX_USE_CXX11_ABI=1 -pipe -msse4.1 -msse4.2 -mpopcnt -std=gnu++1y  -fno-omit-frame-pointer -Wall -Wnon-virtual-dtor   -Werror -O3 -DNDEBUG -x c++ -march=native -shared -fPIC -fvisibility=hidden -fno-implement-inlines
-- Configuring incomplete, errors occurred!

what happen in this error?

Performing Test HAVE_PCLMULQDQ_INTRIN - Failed

this is not the reason of configuration failure.
It is just a check, and build will work without support for pclmulqdq intrinsic.

Please provide full log of running cmake.

how to fix this problem? I need check CPU info?
This OS is VMware

os 32 or 64 bit ?

64 bit

You need to provide complete cmake output (now we see just few tail lines).

The issue is incomplete: we did not see full cmake output.


Maven jars deployment rebuilds the jars and overrides code signing

Currently, jars deployment overrides code signing as it rebuilds the jars again. This needs to be fixed, so it only does the deployment of already existing jars.
Acceptance criteria:

Deployment doesn't; rebuild jars.

PR merged. Closing this.


Clarify the spec to allow optional or unspecified OAuth scopes

Referencing issue #513. Clarify the spec to accommodate OAuth schemes where scope is optional in the authorization request, or where scope is not used at all.
The OAuth Flow Object says that the scopes map is REQUIRED, and doesn't specify any meaning for an empty object value. We took this to mean that the scopes property is required to be present AND non-empty, otherwise an empty value would seem to be circumventing the intention of the spec.
This PR makes it explicit that the scopes map MAY be empty.
Likewise, Security Requirement Object says:

If the security scheme is of type "oauth2" or "openIdConnect", then the value is a list of scope names required for the execution. For other security scheme types, the array MUST be empty.

It doesn't explicitly disallow an empty scope list for an OAuth security requirement. But taken in context, it's easy to assume, mistakenly, that the scopes array must be non-empty, so this PR explicitly allows an empty scopes list on oauth2 security requirements.

Note: #513 suggests that there are OAuth implementations where scope is optional in the authorization request. In cases where the authorization request omits the scope parameter, the authorization server uses a pre-defined default value for the scope.
I tried to address this in the first commit by allowing a scope with an empty string as its name (i.e.  "" as its map key) to represent the default scope.  Other named scopes could also be defined alongside this one.
But I removed the empty-string provision in the second commit because it seemed to create ambiguities in the Security Requirement Object. For example:

If empty string is a defined scope, can Security Requirement include the empty string in its scope list?
In that case, what's the difference between a scopes array with empty string vs. an empty scopes array?

Maybe this:

"security": [ { "oauth_scheme": [""] } ] (scopes array with empty string) means "The caller must be authorized in the default scope."
"security": [ { "oauth_scheme": [] } ]  (empty scopes array) means "The caller must be authorized in some scope(s) that are not specified here, therefore the scope requirements are implementation-defined."

Or maybe they just mean the same thing.

Should we disallow scopes array with empty string and a non-empty string?

"security": [ { "oauth_scheme": ["", "read:pets"] } ] would seem to say, "the caller must be authorized in the default scope AND in the 'read:pets' scope."
This doesn't make much sense to me, but I guess this would be just one of many nonsensical things that designers can express in their API specs. So maybe we just leave it to API designers to specify a sane authorization scheme.

If anyone has good answers to these questions and/or feels strongly that this needs to be addressed in this PR, let's discuss.
Otherwise, I'm happy to defer these gnarly questions and just address the case of empty scopes in this PR.

@webron, done.


B 21374 ub fields int

[!IMPORTANT]
This ticket has a LOT of AC. Please do not rush this review.

B-21374
Summary
Introduces the optional and OCONUS specific fields of accompanied tour and dependent under/over 12 years of age. Feature flags the frontend portion of the application - the backend is not FF'd as it is the internal API and optional, nullable fields with no intent to ever make not nullable. This means the backend works as intended with or without the FF. The Prime cannot control these fields at this time, this is purely for customer and office user.
These OCONUS specific fields are for UB allowance enabling.
How to test
Please test with FF on and off
Setup

Think of two different duty locations
Once you have two different duty locations, open one up within the database, navigate to its address, and set the address is_oconus to true
Now you have a CONUS and OCONUS duty location
Update .envrc and adjust +export FEATURE_FLAG_UNACCOMPANIED_BAGGAGE=true

Customer

Navigate to milmovelocal and create a new account
Create a new order and fill out all forms until you reach "Tell us about your move orders"

Fill in current and new duty location, ensuring to set one of these locations to the OCONUS version you created in the "Setup" section above.
Test adding dependents
Remove your OCONUS duty location and check that the OCONUS fields dissappear
Try to break it
Fully submit your move and proceed to Office

Office

Once you have submitted the move as a customer, login to Office and preview the submitted move as a SC and TOO
The SC should be able to see the information under entitlements
The TOO should be able to see the information under entitlements and update them
These OCONUS entitlement fields shouldn't be present on CONUS moves, verify that
Try to break it

Recordings
https://github.com/user-attachments/assets/1d8f5474-5b7c-4caa-a20c-77e73c3df358
https://github.com/user-attachments/assets/480c886f-7ee9-4afa-9d0d-a1b9f84ea6f1
508 Screenshot
Tooltips do not violate ANDI

Test coverage on client failed.
Last run: Tue 22 Oct 2024 05:41:49 PM UTC
Refer to this report to see coverage details for the client app.
Documentation for test coverage can be found at this link.

I was able to change the Number of dependents under age of 12 as a SC. Not sure if that's correct or not because the testing instructions says SC should be able to see the information and TOO should be able to update it.

Converting back to draft to address changes and introduce new AC. New functionality will be as follows:

Customer being able to edit accompanied tour, dependents under twelve, dependents twelve and over on their orders after it has been submitted
Customer being able to preview these UB fields when selecting "Review request"
SC being able to edit UB fields
Checking with PO on tooltip preview

From the AC - "When I select yes,
I can enter values for the number of dependents "12 and Older" AND "Under 12" (clear out values entered when the user changes dependents included answer back to "No").
I wasn't seeing the number of dependents under the age of 12 and over 12 clearing out when toggling the "Are dependents included in your orders" radio buttons - values persisted

@traskowskycaci I did this intentionally so that if the user is just clicking around, they can keep their values without having to re-type information they've already entered - figured it'd be a smoother experience. I have conditionals on the submit that will default these to null if dependents are set to false/('no'). Let me know if this suffices and if it works as intended

I was able to change the Number of dependents under age of 12 as a SC. Not sure if that's correct or not because the testing instructions says SC should be able to see the information and TOO should be able to update it.

The AC required us to fill in the gaps here in there which is how we got some dev proposed AC. SC should be able to edit allowances and is fully able to now - ready for review again (Once I verify the pipeline is passing)

Failing test is flaky

Failing tests are flaky

From the AC - "When I select yes,
I can enter values for the number of dependents "12 and Older" AND "Under 12" (clear out values entered when the user changes dependents included answer back to "No").
I wasn't seeing the number of dependents under the age of 12 and over 12 clearing out when toggling the "Are dependents included in your orders" radio buttons - values persisted

@traskowskycaci I did this intentionally so that if the user is just clicking around, they can keep their values without having to re-type information they've already entered - figured it'd be a smoother experience. I have conditionals on the submit that will default these to null if dependents are set to false/('no'). Let me know if this suffices and if it works as intended

Morning! Honestly, I like your implementation. However, it would probably be good to run it by Dan that AC mentioning clearing out the values did come from the him & his team

Moving back to draft until new AC gets added in

I was able to change the Number of dependents under age of 12 as a SC. Not sure if that's correct or not because the testing instructions says SC should be able to see the information and TOO should be able to update it.

Customer confirms this is desired capabilities

From the AC - "When I select yes,
I can enter values for the number of dependents "12 and Older" AND "Under 12" (clear out values entered when the user changes dependents included answer back to "No").
I wasn't seeing the number of dependents under the age of 12 and over 12 clearing out when toggling the "Are dependents included in your orders" radio buttons - values persisted

@traskowskycaci I did this intentionally so that if the user is just clicking around, they can keep their values without having to re-type information they've already entered - figured it'd be a smoother experience. I have conditionals on the submit that will default these to null if dependents are set to false/('no'). Let me know if this suffices and if it works as intended

Morning! Honestly, I like your implementation. However, it would probably be good to run it by Dan that AC mentioning clearing out the values did come from the him & his team

Thank you for pointing that out, asked and concurred (See agility for POA)

Some UI requested changes... Can you shorten these a bit? Suggestions:

Accompanied tour with a Yes or No value instead of Is this an accompanied tour?
Dependents under age 12 instead of Number of dependents under the age of 12
Dependents over age 12 instead of Number of dependents of the age 12 or over

And you could keep the extended description in this part
Also could you please update/add the storybooks for the EditOrdersForm, AddOrdersForm and the AllowancesList form?
And... Don't want to open a can here, but remember the SC can add orders for a service member...

Proposed to customer and added as new AC, will action

Warnings

:warning:
This PR does not include changes to storybook, even though it affects component code.

  Generated by :no_entry_sign: dangerJS against 5d7d36b79fbe0d444c24d34e15167686b0f66fd0


Add malloc error checks, avoid double allocation of data block

This PR includes the following:

Adds a couple of missing malloc error checks.
Changes the infinite loop around fpga_download to exit in case of an error. The badge is then stuck on the last displayed error, preventing continuation after an error.
Changes fpga_req_add_file_data with an option to capture the buffer given in parameter, as well as the caller so the buffer is padded with the header required for the list record. This avoid a double allocation of the data block.

Err, no ... don't try to fit the header from fpga_req in there, it's an ugly mixing of abstraction layer for nothing .. the overhead of the malloc is negligible ...
Also this should be split in several commits.

Multiple commits: will split in a new PR.
Malloc and header: if the record is malloc-ed separately and then the buffer captured and placed in re->data, it won't be properly freed? My understanding is that it is always assumed that freeing the record also frees the memory pointed by re->data? This seemed the simplest to avoid additional complications in the records (but this can be solved by adding a boolean tracking whether to free data as well, and adding a call to free where needed?)

Yeah, I would add a field saying if data needs freeing or not.

Noted, closing, will send new PR.

you know you can just force push to the same branch to update the existing PR :)

I did not :) I tried, option to reopen is grayed out though?

Yeah, that works when the PR is open, probably should have re-opened it before the force push, now it seems locked ... ah well, next time :sweat_smile:

Created a different one, good to know in any case!


WIP [stable/prometheus-operator] Add variable to optionally set priorityClassName

Signed-off-by: Chris O'Brien<EMAIL_ADDRESS>What this PR does / why we need it:
Allow priorityClassName to be set for the prometheus-operator deployment's podSpec.
Additionally, update dependency charts containing the same change of adding priorityClassName options.  See #9478 #9427 #9483.
Which issue this PR fixes

fixes #9476

Checklist

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[ ] Wait until #9478 and #9427 are merged, then update Helm dependencies

/ok-to-test
/lgtm


Further improvements / maintenance

Hello,
Due to the library becoming stale, as @Kevinpgalligan suggested in #74, I'll keep working on it at my fork only: https://github.com/Gleefre/sketch. Feel free to open pull requests and raise new issues there :).
@vydd if you choose to resume your work on sketch, I would be glad to open several pull-requests to backport our changes.

Hey @Gleefre ! Sorry for not doing a good job at reviewing, super busy times.
You are of course free to maintain a fork, but I feel it'd be better for everyone if we found a way to get your changes into here.
If you would describe where you want to go with the fork/changes, it'd be much easier for me to review at a more regular cadence. As it stands, whenever I do it, it's a huge context switch for me, as figuring out why someone is proposing a certain change at a certain time can be difficult.

Hi!
Not sure what you mean by where I want to go; but my goal is to create a faster moving branch, that would be easier to contribute to (for me as well as for others).
Aside from small bug fixes I also plan to work on features like

easier building for applications. define-start-function is currently working, but I'm still thinking about its API.
fix the default width/height solution I've been using - which would probably include rewriting defsketch macro and removing the prepare method.
separate the public API from the SDL2/OpenGL backend. Maybe add other backends.
exposing some internals of the backend, for example to get a configurable mag filter in openGL to resize images without them getting blurred.
figure out for myself and maybe document sketch components like shaders, figures, controllers and channels.

I can go through PRs later today if you reopen them. Let's try to make this work.

Alright :).

I'll be closing this issue now.
Do you plan to actively work on sketch, or you are just reviewing PR for now?

Let's say it's reviewing PRs for now and discussion. I am actively thinking about channels again, though.

Here's something else you can do with channels (as a continuation from our irc discussion... maybe we should create an issue for it):
(defsketch add
    ((c (in :c 0)))
  (text (format nil "~a" c) 100 100))

(define-channel-observer
  (out :c (+ (in :a 0) (in :b 0))))

with that defined, you can see the number changing as you send numbers to :a and :b in repl (using (out :a 5))


SPManagedMetaDataServiceApp resource fails when setting Content Type Hub URL

The SPManagedMetaDataServiceApp resource fails when attempting to set a content type hub URL on an existing MMS instance that was created without a content type hub URL.  There is logic in the Set-TargetResource function that attempts to compare the CT hub URL that was provided as a parameter to the current CT hub URL.  As part of this operation, a "TrimEnd()" is attempted.  This fails when the existing instance's CT hub URL was not set because $result.ContentTypeHubUrl will be $null (resulting in an error of "You cannot call a method on a null-valued expression.").
Confirmed that SharePoint DSC 1.7, 1.9, 2.0, and 2.1 all have the incorrect code.
OS:  Windows Server 2012 / 2012 R2
SharePoint version:  SharePoint 2013
DSC version:  Using 1.7, but have confirmed issue still exists in newer releases.

Will be included in my next Bugfix PR

Closed via PR #775


feat: wallet_deploymentData

@janek26, before we proceed with the merge, could you please address the following issues:

The wallet_deploymentData call has the potential to return null.
The version field is returned from the request. Therefore, it should be included in the type definition.

Thank you for looking into these matters.

@dan-ziv pls check #194 which also covers for this request (and is already merged)

@janek26 perhaps we could close this one, now that https://github.com/starknet-io/get-starknet/pull/194 has been merged

https://github.com/starknet-io/get-starknet/pull/194 Already covered this PR.


Flag: input-path

Allows for an input file to be specified.
This replaces the data collected by data gatherers from clusters.

/approve
/lgtm


[backport] Fixes a grammar error

Backport of #7658

[automatic post] Jenkins, test this please.

Jenkins CI test (for commit 271168f9a3b3bd01238551783921d687b624609d, target branch v6) succeeded!


Use localStorage

It would be cool if this library would use localStorage to store the dismiss rather than a cookie. At least make it an option. Most developers don't care that a cookie is set and it just clutters every subsequent request.

Done. Not as good as cookie way, but should be OK.
Soon in 2.0.0


feat: add autoML notebook

@xczuba comments: https://app.reviewnb.com/kili-technology/kili-python-sdk/blob/feature/ml-1449-aanadine-on-colab-when-i-look-at-google-vertex-ai-automl%2Frecipes%2Fvertex_ai_automl_od.ipynb%2F/discussion

LGTM, just need a conclusion + cleanup sections at the end

I don't cleanup the project at the end because it will not be used on tests and we don't want the client to accidentely delete its project


Suggest adding three trivial overloads for a more uniform use of expr()

matchit.txt

Hi @hugoetchegoyen, thanks for bringing this up and that is a very good topic.
"expr" was introduced one year ago and now we have a chance to rethink about it.
Maybe there is a chance to remove all its usage and provide overloaded versions for
nullary callable, Id, and others for "when" and "operator=".
Then the sample would be like
#include "matchit.h"

template<typename T1, typename T2>
constexpr auto eval(std::tuple<char, T1, T2> const& exp)
{
    using namespace matchit;
    Id<T1> i;
    Id<T2> j;
    return match(exp)
    (
        pattern | ds('+', i, j) = i + j,
        pattern | ds('-', i, j) = i - j,
        pattern | ds('*', i, j) | when(i) = i, // previously not allowed
        pattern | ds('/', i, j) = 12345,        // previously not allowed
        pattern | _ = []
        {
            assert(false);
            return -1;
        }
    );
}

The only concern is about others. Would that be too risky if users provide a unary instead of nullary mistakenly and we silent the error and return the unary function as the result?
A compromise may be to keep expr for others only and deprecate its usage for Id.
What do you think?
Thanks,
Bowen

Hi Bowen,
Thanks for your reply.
I like the idea of eliminating the need to use expr() entirely by
overloading operator= and when() for nullary callables, Ids and others.
I realize that if the user mistakenly uses a unary instead of a nullary,
the unary will fall in the category of others and will be used as a plain
value, not called. I don't think this deserves a special protection.
I would not deprecate expr() but keep it around as an option, this way you
don´t break backward compatibility.
In some cases you might also like to use expr() for cosmetic reasons. For
example, consider a handler that returns a nullary lambda:
pattern | _ =  []{ return []{ return 123; }; }

In my opinion it looks nicer with expr():
pattern | _ =  expr([]{ return 123; })

Best regards,
Hugo
El mié, 9 nov 2022 a las 10:46, Bowen Fu @.***>)
escribió:

Hi @hugoetchegoyen https://github.com/hugoetchegoyen, thanks for
bringing this up and that is a very good topic.
"expr" was introduced one year ago and now we have a chance to rethink
about it.
Maybe there is a chance to remove all its usage and provide overloaded
versions for
nullary callable, Id, and others for "when" and "operator=".
Then the sample would be like
#include "matchit.h"
template<typename T1, typename T2>constexpr auto eval(std::tuple<char, T1, T2> const& exp)
{
using namespace matchit;
Id i;
Id j;
return match(exp)
(
pattern | ds('+', i, j) = i + j,
pattern | ds('-', i, j) = i - j,
pattern | ds('*', i, j) | when(i) = i, // previously not allowed
pattern | ds('/', i, j) = 12345,        // previously not allowed
pattern | _ = []
{
assert(false);
return -1;
}
);
}
The only concern is about others. Would that be too risky if users
provide a unary instead of nullary mistakenly and we silent the error and
return the unary function as the result?
A compromise may be to keep expr for others only and deprecate its usage
for Id.
What do you think?
Thanks,
Bowen
—
Reply to this email directly, view it on GitHub
https://github.com/BowenFu/matchit.cpp/issues/94#issuecomment-1308790293,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ2LRV6SFAMDHQUWNDL5XW3WHOTKZANCNFSM6AAAAAAR22EK5A
.
You are receiving this because you were mentioned.Message ID:
@.***>

PS:
With my three trivial overloads the situation is like your second alternative: expr() is compulsory for others only. But it is not deprecated, on the contrary, it is generalized to accept everything.

@hugoetchegoyen good pointer about "this doesn't deserve a special protection". And by deprecate I mean marked it with "[[deprecated]]", so to discourage its usage and remove in a later major version release.
If you are interested, will you create a pull request for it?
Thanks,
Bowen

Hi Bowen,
I think I suceeded. The changes are very small. You can use 'others' as when predicates or handlers now. I attach a file with the changes and the sample program. Function expr() is still available, I used it (unnecessarily) in expr(i+j). If you would like to remove it, I guess it can be moved to namespace impl.
Regards,
Hugo
matchit2.txt

@hugoetchegoyen looks good to me. And you can also define a helper toNullary function inside impl namespace and than you will only need one version of when / operator=.
Are you planning to create a pull request yourself or you want me to create one for you?
Please let me know if you meet any issues when creating a PR.
To create a PR, you need to fork the repo and commit the changes to your fork repo firstly and then create a PR from your fork repo to this repo.
You can find more info here. Hope this helps.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork
Thanks,
Bowen

Hi Bowen,
Great! The universal references solved it. If you think it's OK now I'll create a pull request.
Regards,
Hugo
matchit4.txt

PS sorry for the mistake, I did use std::decay_t in toNullary. Sent you an outdated file.

This is the updated file.
matchit4.txt

I already forked your repository, commited my changes, built the targets and ran unittest without  errors, so it seems the changes don't break existing code. I'll create a pull request as soon as you agree.

LGTM. Feel free to create a pull request. We can continue our discussions there (if any).
And can you also add some unit tests to cover the new code paths?

Thank you very much Bowen.
Best regards,
Hugo


Add creating combined reference dataset with a single code.

The HmtVar dataset is not working yet. It might not be able to combine all the datasets together. Hail reports running out of memory when combining all the datasets. It may be because of hail configuration.

An example of what the class for write_combined_reference_table would look like is as follows:
class WriteCombinedReferenceData(object):
  CONFIG = {...}
  ...
  @classmethod
  def get_ht(cls, dataset, reference_genome):
    config = cls.CONFIG[dataset][reference_genome]
    ...

  ...
  @classmethod
  def run(cls, args):
    ...
    cls.join_hts([...])

if __name__ == "__main__":
  ...
  WriteCombinedReferenceData.run(args)

and then for mito all you should need to do is
class WriteCombinedMitoReferenceData(WriteCombinedReferenceData):
  CONFIG = {...} # the mito config goes

if __name__ == "__main__":
  ...
  WriteCombinedMitoReferenceData.run(args)

Alternatively, you can leave write_combined_reference_table as it is and just add the mito data sources to the main config. Since the script specifically  sets which subset of configs it wants to use that won't break the exisiting script, and then for the mito you could import functionality and do something like
from hail_scripts.write_combined_reference_table import join_hts

if __name__ == "__main__":
   ...
   join_hts([...], args)

I updated download_and_create_reference_datasets/v02/hail_scripts/write_combined_reference_data_ht.py to support writing the combined mitochondrial reference data. Basically, its functionalities for nuclear SNPs don't change.

so this is still not what we discussed. As I have repeatedly mentioned, we want this to match the usage of/ share code with write_combined_reference_data_ht. That means there is one create_ht__<dataset> script per dataset type, and then we will have a new file write_combined_mito_reference_data_ht.py that basically just imports join_hts and calls it with the datasets we care about for mito. If you really want to add a helper that can allow you to call multiple create_ht__<dataset> at the same time you can do that, but that needs to be separate from the actual script for writing the combined file, and there still needs to be a way to run them independently. We discussed in standup that going forward we need to be able to easily update just one resource without needing to re-download the others. If you are still confused, please set up time to meet with me

You can not merge PRs to this repo without Mike's approval. My approval was very much conditional on him approving. Revert this immediately

actually I reverted it, new PR is here: https://github.com/broadinstitute/hail-elasticsearch-pipelines/pull/314


Question Regarding NQ320k Dataset Deduplication

Hi @varshakishore ,
Thank you for your excellent work and for sharing the code with the community. I have a few questions regarding the NQ320k dataset.
Following your instructions, I attempted to deduplicate the dataset using the title extracted from the "document_url" field. I identified the title by locating the text between "title=" and "&amp", for example, the title "2017_BBC_Sports_Personality_of_the_Year_Award" was extracted from the following URL:
url = https://en.wikipedia.org//w/index.php?title=2017_BBC_Sports_Personality_of_the_Year_Award&oldid=816169117"

I replicated this process for both the train and dev sets from Natural Questions dataset, resulting in 108,071 and 1,448 unique documents, respectively. Merging both train and dev sets yielded 108,399 unique documents. However, in your paper, there are at least 98,743 + 9,874 = 108,617 documents (initial + new).
My question is: Am I missing any preprocessing steps? It seems there is a discrepancy in the total number of unique documents.
Thank you for your time and assistance.

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."


NeoEden: add initial tests

This is really janky at the moment, but the idea is to get rid of the escrip and write tests using go testing infra.

Goal is to follow two principales, DRY and KISS.
Tests are grouped together (only smoke at the moment)
Tests broken into different categories based on what they test, like just EVE, or App or Network (for now just simple eve tests are ported)
Idea is setup once and use in multiple tests, for example setup and app once and share it between (no-destructive) tests.
This is WIP through and through.

TODO:

[ ] add documentation

Test
Simply build and prepare :
make clean
make build
make build-tests
./eden config add default
./eden setup
./eden start
./eden eve onboard

and then :
shah@zombie:$ cd neo-eden/workflows/ && ./smoke-tests.sh
Running smoke tests
INFO[0000] EVE Test Suite started
configName:  default
configFile:  /home/shah/.eden/contexts/default.yml
=== RUN   TestSmoke
=== RUN   TestSmoke/TestAppArmorEnabled
INFO[0000] TestAppArmorEnabled started
DEBU[0000] Will use config from /home/shah/.eden/contexts/default.yml
DEBU[0000] Try to add config from /home/shah/shah-dev/eden/neo-eden/workflows/eden-config.yml
INFO[0000] TestAppArmorEnabled finished
=== RUN   TestSmoke/TestVcomLinkTpmRequestEK
INFO[0000] TestVcomLinkTpmRequestEK started
    vcom_test.go:14: Checking if vcomlink is running on EVE
INFO[0000] TestVcomLinkTpmRequestEK finished
=== RUN   TestSmoke/TestVtpmIsRunningOnEVE
    vtpm_test.go:13: TestVtpmIsRunningOnEVE started
    vtpm_test.go:26: TestVtpmIsRunningOnEVE finished
--- PASS: TestSmoke (0.82s)
    --- PASS: TestSmoke/TestAppArmorEnabled (0.34s)
    --- PASS: TestSmoke/TestVcomLinkTpmRequestEK (0.28s)
    --- PASS: TestSmoke/TestVtpmIsRunningOnEVE (0.20s)
PASS

You sure you want to do this in one PR? I mean I'm going to help you all the way through, but my experience with that says we should do it in smaller steps

hmm make sense, lets me rework it so we can have two eden in parallel.

@shjala since #1037 is merged makes sense to rebase this, but let's have a discussion first about evetestkit design

@shjala since #1037 is merged makes sense to rebase this, but let's have a discussion first about evetestkit design

sure 👍

I'm going to add the setup and teardown to this and then lets have a final round of review, I don't want to add more tests to it to bloat it more that it is.


--zksync flag causes invariant tests to hang

Component
Forge
Have you ensured that all of these are up to date?

[X] Foundry
[X] Foundryup

What version of Foundry are you on?
forge 0.0.2
What command(s) is the bug in?
forge test --zksync
Operating System
macOS (Apple Silicon)
Describe the bug
GM, I have issues with running invariant tests while using the --zksync flag. Basically all tests works perfect both with and without the flag until I add invariant tests. Once I add them, it works only without the flag, with it the tests never finish / stall forever.
Steps to replicate:
Use simple contract like this:
`// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;
import<EMAIL_ADDRESS>error USDCTrasferFailed();
contract PaymentProcessor {
IERC20 public immutable usdc;
mapping (address => uint256) public deposits;
constructor(address _usdc) {
    usdc = IERC20(_usdc);
}

function depositUSDC(uint256 amount) public {
    bool success = usdc.transferFrom(msg.sender, address(this), amount);
    deposits[msg.sender] += amount;
    if (!success) revert USDCTrasferFailed();
}

}
`
now, a simple test:
`// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;
import {Test, console} from "forge-std/Test.sol";
import { PaymentProcessor } from "../src/PaymentProcessor.sol";
import { MockERC20 } from "./mocks/MockERC20.sol";
import { StdInvariant } from "forge-std/StdInvariant.sol";
contract PaymentProcessorTest is StdInvariant, Test {
PaymentProcessor public paymentProcessor;
MockERC20 public usdc;
function setUp() public {
    usdc = new MockERC20();
    paymentProcessor = new PaymentProcessor(address(usdc));

    // Invariant target
    targetContract(address(paymentProcessor));
}

// Statefull invariants
function invariant_testAlwaysCorrectUSDCAddress() public {
    assertEq(address(paymentProcessor.usdc()), address(usdc));
}

function testDepositUSDC(uint128 amount) public {
    usdc.mint(address(this), amount);
    usdc.approve(address(paymentProcessor), amount);
    paymentProcessor.depositUSDC(uint256(amount));
    assertEq(usdc.balanceOf(address(paymentProcessor)), amount);
    assertEq(paymentProcessor.deposits(address(this)), amount);
}

}`
and then try to run it with these two commands:
You will notice this goes through
forge test
And this stalls / never finishes
forge test --zksync
If I just remove the invariant testing from the tests file, both work.

Hey @davd-ops!
Thank you for bringing up the issue regarding the invariant tests hanging when running in --zksync mode. I’ve looked into this, and it appears that the tests aren’t actually hanging. Instead, the fuzzer is taking significantly longer to complete when running it in zksync mode.
If you reduce the number of runs or decrease the fuzzing depth, you’ll notice that the tests do eventually finish. I apologize for the inconvenience this slower performance is causing right now. You can adjust this values located in the foundry.toml. In particular I think the main issue is the depth and not the number of runs.  By default If I remember correctly the depth is 500 I believe.
[profile.default.invariant]
runs = 256
depth = 100

Let me know if this helps!

Hey @Jrigada thanks for the answer! However reducing the number the runs or depth kinda destroys the point of fuzzing (to try and run the code with as many as possible combinations of random inputs to see if there is nothing that would break it).
Even with the reduced settings that you posted, it still takes super long . In a situation where u have 3+ invariants, you're gonna have to go for a walk before it finishes up, that is not a realistic usage scenario at all.
Can we expect improvements to this in the near future? It's basically unusable right now in any real scenario

As we depend on zksync-era's multivm, we are bound by the performance of that library when executing a transaction in zkVM.
Having said that, we noticed a major speedup when running the release binary, compared to compiling locally without the --release flag.
We understand this is no final solution but, considering the stage of development of this project, we are not focused on performance where it is not critical or where there are workarounds available.
If using a release binary still proves to be too slow, I suggest limiting the number of runs for invariant tests like @Jrigada mentioned above, and create a separate foundry profile where the invariant runs are set to a number more appropriate for your use case, and use such profile before deployments or similar, where you need that extra security, but otherwise not use it during development to avoid unnecessary slowdown

I'll go ahead and close the issue as there's really no solution for this at this point in time


[BUG] Issue with cloning group datasets

Selecting fields and cloning seems to copy unwanted fields into the clone. This issue only applies to group datasets
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")
dataset = dataset.select_fields().clone("image-clone")
dataset.first() # Works

dataset = foz.load_zoo_dataset("quickstart-groups")
dataset = dataset.select_fields().clone("group-clone")
dataset.first() # Unexpected ground truth data

Traceback
Traceback (most recent call last):
  File "/Users/benjaminkane/code/fiftyone/fiftyone/core/dataset.py", line 6406, in _sample_dict_to_doc
    return self._sample_doc_cls.from_dict(d, extended=False)
  File "/Users/benjaminkane/code/fiftyone/fiftyone/core/odm/document.py", line 452, in from_dict
    return cls._from_son(d)
  File "/Users/benjaminkane/venvs/fo/lib/python3.10/site-packages/mongoengine/base/document.py", line 837, in _from_son
    obj = cls(__auto_convert=False, _created=created, **data)
  File "/Users/benjaminkane/venvs/fo/lib/python3.10/site-packages/mongoengine/base/document.py", line 99, in __init__
    raise FieldDoesNotExist(msg)
mongoengine.errors.FieldDoesNotExist: The fields "{'ground_truth'}" do not exist on the document "samples.64bea2e5c505526444f4d0c0"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/benjaminkane/code/scratch/mult-pcd.py", line 5, in <module>
    dataset.first()
  File "/Users/benjaminkane/code/fiftyone/fiftyone/core/dataset.py", line 1114, in first
    return super().first()
  File "/Users/benjaminkane/code/fiftyone/fiftyone/core/collections.py", line 712, in first
    return next(iter(self))
  File "/Users/benjaminkane/code/fiftyone/fiftyone/core/dataset.py", line 2183, in iter_samples
    for sample in samples:
  File "/Users/benjaminkane/code/fiftyone/fiftyone/core/dataset.py", line 2199, in _iter_samples
    sample = make_sample(d)
  File "/Users/benjaminkane/code/fiftyone/fiftyone/core/dataset.py", line 2212, in make_sample
    doc = self._sample_dict_to_doc(d)
  File "/Users/benjaminkane/code/fiftyone/fiftyone/core/dataset.py", line 6412, in _sample_dict_to_doc
    return self._sample_doc_cls.from_dict(d, extended=False)
  File "/Users/benjaminkane/code/fiftyone/fiftyone/core/odm/document.py", line 452, in from_dict
    return cls._from_son(d)
  File "/Users/benjaminkane/venvs/fo/lib/python3.10/site-packages/mongoengine/base/document.py", line 837, in _from_son
    obj = cls(__auto_convert=False, _created=created, **data)
  File "/Users/benjaminkane/venvs/fo/lib/python3.10/site-packages/mongoengine/base/document.py", line 99, in __init__
    raise FieldDoesNotExist(msg)
mongoengine.errors.FieldDoesNotExist: The fields "{'ground_truth'}" do not exist on the document "samples.64bea2e5c505526444f4d0c0"

Resolved by #3336


babel-preset-es2015-rollup versioning problem

The current version of babel-preset-es2015-rollup is 1.0.0 and it's based on babel-preset-es2015 6.1.2, so it's not updated as the latest version available is 6.3.13. Currently, in order to use the updated package, I'm loading babel-preset-es2015 and manually disabling the commonjs plugin.
What can be done to install an specific version of babel-preset-es2015 adapted to rollup?

That's very odd – 6.3.13 should satisfy the existing semver constraint. I've just tried installing the latest version of this, and these are the versions I got:
<EMAIL_ADDRESS>node_modules/babel-preset-es2015-rollup
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS>└──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
What happens if you rm -rf node_modules && npm i? (If you're on npm 3, trashing the entire node_modules folder appears to be the most efficient way to update as far as I can tell, unfortunately)

I will see what went wrong here then report you when I find it out.
2016/01/04 20:54 "Rich Harris"<EMAIL_ADDRESS>
That's very odd – 6.3.13 should satisfy the existing semver constraint.
I've just tried installing the latest version of this, and these are the
versions I got:
<EMAIL_ADDRESS>node_modules/babel-preset-es2015-rollup
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS><EMAIL_ADDRESS>└──<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>What happens if you rm -rf node_modules && npm i? (If you're on npm 3,
trashing the entire node_modules folder appears to be the most efficient
way to update as far as I can tell, unfortunately)
—
Reply to this email directly or view it on GitHub
https://github.com/rollup/rollup-plugin-babel/issues/27#issuecomment-168837523
.

@qgustavor did you figure it out? I'd like to close this out if there's a resolution.

I don't know why but it's working now.
2016/01/16 13:37 "Brian Donovan"<EMAIL_ADDRESS>
@qgustavor https://github.com/qgustavor did you figure it out? I'd like
to close this out if there's a resolution.
—
Reply to this email directly or view it on GitHub
https://github.com/rollup/rollup-plugin-babel/issues/27#issuecomment-172217142
.

If it comes up again let us know!


New Feature: Start LISAv2 tests using Parameters File

1.	Switch parameters can be enabled by setting it's value to "true"
2.	Update the parameters in .\XML\TestParameters.xml and run below command to start the tests.
3.	Syntax -
	Run-LisaV2.ps1 -TestParameters <Path to XML file>

	Example 1 :
	===========
	a.	Run-LisaV2.ps1 -TestParameters .\XML\NetworkTests.xml
		Here, we defined .\XML\NetworkTests.xml to include all required network tests.
		We also added necessary information like ARMImageName, TestLocation in the XML file.

	Example 2 :
	===========
	b.	Run-LisaV2.ps1 -TestParameters .\XML\NetworkTests.xml -TestLocation "westeurope"
		Here, we already have predefined XML file from Example 1, but now, you want to run
		the same tests in "westeurope" region. You can achieve this by editing the
		NetworkTests.xml file and adding westeurope region.
		OR
		You can "Override" TestLocation on top of XML parameters.
		This scenario is helpful in leveraging use of Predefined XML parameter files without
		editing them for small changes.

	Example 3 :
	===========
	c.	Run-LisaV2.ps1 -TestParameters .\XML\NetworkTests.xml -TestLocation "westeurope" -ARMImage "Canonical UbuntuServer 16.04-LTS latest".
	Again, we are leveraging same .\XML\NetworkTests.xml file to run tests agains different Image.

Conclusion:
===========
1.	When you use -TestParameters <File Path> LISAv2 will import all the paramters defined there.
2.	All other parameters are still available to use.
3.	Any parameter given to Run-LisaV2.ps1 in conjunction with -TestParameters will override it's value and value from XML parameters file will be ignored.

Execution is verified.
Scenario 1: Provide parameters using XML file only.
PS D:\GitHub\LISAv2> .\Run-LisaV2.ps1 -ParametersFile .\XML\TestParameters.xml
10/26/2018 15:23:48 : [INFO ] Setting parameter TestPlatform -> Azure
10/26/2018 15:23:48 : [INFO ] Setting parameter TestLocation -> westeurope
10/26/2018 15:23:48 : [INFO ] Setting parameter RGIdentifier -> SSDELETEME
10/26/2018 15:23:48 : [INFO ] Setting parameter ARMImageName -> Canonical UbuntuServer 16.04-LTS latest
10/26/2018 15:23:48 : [INFO ] Setting parameter TestNames -> VERIFY-DEPLOYMENT-PROVISION
10/26/2018 15:23:48 : [INFO ] Setting parameter XMLSecretFile -> D:\GitHub\LISAv2\XML\AzureSecrets.xml
10/26/2018 15:23:48 : [INFO ] Setting boolean parameter UpdateGlobalConfigurationFromSecretsFile -> True
10/26/2018 15:23:48 : [INFO ] Setting boolean parameter UpdateXMLStringsFromSecretsFile -> True
10/26/2018 15:23:48 : [INFO ] Setting boolean parameter ForceDeleteResources -> True
10/26/2018 15:23:48 : [INFO ] Setting boolean parameter UseManagedDisks -> True
10/26/2018 15:23:48 : [INFO ] Setting parameter VMGeneration -> 1
10/26/2018 15:23:48 : [INFO ] Setting boolean parameter ExitWithZero -> True
10/26/2018 15:23:48 : [INFO ] Created LogDir: .\TestResults\2018-26-10-20-53-48-8879
10/26/2018 15:23:48 : [INFO ] 7za.exe file found in Tools folder.
10/26/2018 15:23:48 : [INFO ] dos2unix.exe file found in Tools folder.
10/26/2018 15:23:48 : [INFO ] gawk file found in Tools folder.
10/26/2018 15:23:48 : [INFO ] jq file found in Tools folder.
10/26/2018 15:23:48 : [INFO ] plink.exe file found in Tools folder.
10/26/2018 15:23:48 : [INFO ] pscp.exe file found in Tools folder.
10/26/2018 15:23:48 : [INFO ] kvp_client32 file found in Tools folder.
10/26/2018 15:23:48 : [INFO ] kvp_client64 file found in Tools folder.
10/26/2018 15:23:49 : [INFO ] nc.exe file found in Tools folder.
10/26/2018 15:23:49 : [INFO ] Test parameters have been validated successfully. Continue running the test.
.
.
.
Test Results Summary
ICA test run on 10/26/2018 15:23:57
ARM Image under test Canonical : UbuntuServer : 16.04-LTS : latest
Total Executed TestCases        1 (       1 Pass,        0 Fail,        0 Abort)
Total Execution Time(dd:hh:mm) 0:0:6
XML file: TestConfiguration

#Sr. Test Name : Test Result : Test Duration (in minutes)
----------------------------------------------------
1. VERIFY-DEPLOYMENT-PROVISION : PASS : 6.9
        FirstBoot : PASS
        FirstBoot : Call Trace Verification : PASS
        Reboot : PASS
        Reboot : Call Trace Verification : PASS

Logs can be found at .\TestResults\2018-26-10-20-52-46-4150

10/26/2018 15:30:55 : [INFO ] Creating 'Azure-46238-buildlogs.zip' from '.\TestResults\2018-26-10-20-53-48-8879'
10/26/2018 15:30:55 : [INFO ] D:\GitHub\LISAv2\Azure-46238-buildlogs.zip created successfully.
10/26/2018 15:30:55 : [INFO ] Copied : .\report\report_TC-46238.xml --> .\report\report_TC-46238-junit.xml
10/26/2018 15:30:55 : [INFO ] Analysing results..
10/26/2018 15:30:55 : [INFO ] PASS  : 1
10/26/2018 15:30:55 : [INFO ] FAIL  : 0
10/26/2018 15:30:55 : [INFO ] ABORT : 0
10/26/2018 15:30:55 : [INFO ] LISAv2 exits with code: 0

Scenario 2: Provide parameters using XML file + Override Parameters value
PS D:\GitHub\LISAv2> .\Run-LisaV2.ps1 -ParametersFile .\XML\TestParameters.xml -TestLocation westus
10/26/2018 15:58:26 : [INFO ] Setting parameter TestPlatform -> Azure
10/26/2018 15:58:26 : [INFO ] Setting parameter TestLocation -> westeurope
10/26/2018 15:58:26 : [INFO ] Setting parameter RGIdentifier -> SSDELETEME
10/26/2018 15:58:26 : [INFO ] Setting parameter ARMImageName -> Canonical UbuntuServer 16.04-LTS latest
10/26/2018 15:58:26 : [INFO ] Setting parameter TestNames -> VERIFY-DEPLOYMENT-PROVISION
10/26/2018 15:58:26 : [INFO ] Setting parameter XMLSecretFile -> D:\GitHub\LISAv2\XML\AzureSecrets.xml
10/26/2018 15:58:26 : [INFO ] Setting boolean parameter UpdateGlobalConfigurationFromSecretsFile -> True
10/26/2018 15:58:26 : [INFO ] Setting boolean parameter UpdateXMLStringsFromSecretsFile -> True
10/26/2018 15:58:26 : [INFO ] Setting boolean parameter ForceDeleteResources -> True
10/26/2018 15:58:26 : [INFO ] Setting boolean parameter UseManagedDisks -> True
10/26/2018 15:58:26 : [INFO ] Setting parameter VMGeneration -> 1
10/26/2018 15:58:26 : [INFO ] Setting boolean parameter ExitWithZero -> True
10/26/2018 15:58:26 : [INFO ] Overriding specified parameter TestLocation = westus
10/26/2018 15:58:26 : [INFO ] Created LogDir: .\TestResults\2018-26-10-21-28-26-7759
10/26/2018 15:58:26 : [INFO ] 7za.exe file found in Tools folder.
10/26/2018 15:58:26 : [INFO ] dos2unix.exe file found in Tools folder.
10/26/2018 15:58:26 : [INFO ] gawk file found in Tools folder.
10/26/2018 15:58:26 : [INFO ] jq file found in Tools folder.
10/26/2018 15:58:26 : [INFO ] plink.exe file found in Tools folder.
10/26/2018 15:58:26 : [INFO ] pscp.exe file found in Tools folder.
10/26/2018 15:58:26 : [INFO ] kvp_client32 file found in Tools folder.
10/26/2018 15:58:26 : [INFO ] kvp_client64 file found in Tools folder.
10/26/2018 15:58:26 : [INFO ] nc.exe file found in Tools folder.
10/26/2018 15:58:26 : [INFO ] Test parameters have been validated successfully. Continue running the test.
.
.
.
Test Results Summary
ICA test run on 10/26/2018 15:58:41
ARM Image under test Canonical : UbuntuServer : 16.04-LTS : latest
Total Executed TestCases        1 (       1 Pass,        0 Fail,        0 Abort)
Total Execution Time(dd:hh:mm) 0:0:25
XML file: TestConfiguration

#Sr. Test Name : Test Result : Test Duration (in minutes)
----------------------------------------------------
1. VERIFY-DEPLOYMENT-PROVISION : PASS : 25.86
        FirstBoot : PASS
        FirstBoot : Call Trace Verification : PASS
        Reboot : PASS
        Reboot : Call Trace Verification : PASS

Logs can be found at .\TestResults\2018-26-10-20-53-48-8879

10/26/2018 16:24:35 : [INFO ] Creating 'Azure-27929-buildlogs.zip' from '.\TestResults\2018-26-10-21-28-26-7759'
10/26/2018 16:24:35 : [INFO ] D:\GitHub\LISAv2\Azure-27929-buildlogs.zip created successfully.
10/26/2018 16:24:35 : [INFO ] Copied : .\report\report_TC-27929.xml --> .\report\report_TC-27929-junit.xml
10/26/2018 16:24:35 : [INFO ] Analysing results..
10/26/2018 16:24:35 : [INFO ] PASS  : 1
10/26/2018 16:24:35 : [INFO ] FAIL  : 0
10/26/2018 16:24:36 : [INFO ] ABORT : 0
10/26/2018 16:24:36 : [INFO ] LISAv2 exits with code: 0

@juhlee-microsoft  Do you want me to update README.md file, right?

@juhlee-microsoft Do you want me to update README.md file, right?

Yes, also I want you to check out all other documents if needed.

@iamshital please squash the 7 commits into one

A general review note on this way of implementing parameterization: you are duplicating any work that involves parameter change/addition because you need to maintain separately the command line and the xml file.
Basically, you are doing this because the lisav2 interface has become so complicated, it cannot be easily changed between the runs. From the experience with previous projects, when the command line interface had become very hard to maintain because of adding more and more parameters, one way of dealing with it was to limit the command line parameters to the most used ones and move all the others to the XML (file) based configuration.
@iamshital  @juhlee-microsoft @chvalean what do you think of this suggestion?

@iamshital please squash the 7 commits into one

Ju is merging all the PRs with squash.

@iamshital There are two files with Run-LISAv2.ps1. Run-LisaV2.ps1 and ReplaceableTestParameters.xml

A general review note on this way of implementing parameterization: you are duplicating any work that involves parameter change/addition because you need to maintain separately the command line and the xml file.
Basically, you are doing this because the lisav2 interface has become so complicated, it cannot be easily changed between the runs. From the experience with previous projects, when the command line interface had become very hard to maintain because of adding more and more parameters, one way of dealing with it was to limit the command line parameters to the most used ones and move all the others to the XML (file) based configuration.
@iamshital @juhlee-microsoft @chvalean what do you think of this suggestion?

It is true that there are too many parameters in cmdline, but this is agile-style project pain. He put all dynamic values in the file. So that you can edit a xml file every run time, or keep the different xml files.


Display Google Sheets

A second HTML page that embeds a Google Sheet for displaying leaderboards would be a great feature to have.  This might have to be hard coded for a single spreadsheet rather than being able to pass in your own.

We can probably use a data table to show the user leaderboards and have a button to let them download it in an excel format. I know there is another issue to download as csv. We can build the logic to let the user decide what format they want to download their data.
I've used xlsx in the past to convert JSON data to excel for users to download. I believe they have a free one that we can use to start.


Shiny Updates Visual Bug

Expected behavior

The info should be in one line. I expect this is due to how 4.6 parses the info for shiny updates
Actual behavior

Steps to reproduce behavior

Upgrade to WP 4.6, then rollback the timber version number manually to force an update to show.
What version of WordPress, PHP and Timber are you using?

WordPress 4.6
How did you install Timber? (for example, from GitHub, Composer/Packagist, WP.org?)

Plugin

Closed by #1167


Using categorical slot values in different context

Rasa Core version: master
Python version: 3x
Operating system (windows, osx, ...): ubuntu
Issue:
Can categorical slots be used in different context. For example in the below domain file toppings slots will have different actions to be taken for each of items. The ask is can it be a generic story or it needs to be in serial.
say..
main menu

order{"item":"pizza"}

utter_toppingneeded

topping menu

inform{"toppings":"mushroom"}

slot{"item":"pizza"}
do_pizza

topping menu

inform{"toppings":"mushroom"}

slot{"item":"pasta"}
do_pasta

Content of domain file (if used & relevant):
slots:
  item:
    type: categorical
    values:
    - pasta
    - pizza
    - sandwich
  toppings:
    type: categorical
    values:
    - mushroom
    - onion
    - tomato

Please ask these kind of questions in our forum


Eloquent Relationship issue with hasOne

/**
 * @return \Illuminate\Database\Eloquent\Relations\HasOne
 */
public function phone()
{
    return $this->hasOne('App\Phone');
}

Gives the following error:
Class Illuminate\Database\Eloquent\Relations\HasOne referenced with incorrect case: Illuminate\Database\Eloquent\Relations\hasOne.

@sumityadav You don't have on your code any reference to "Illuminate\Database\Eloquent\Relations\hasOne" with hasOne instead of HasOne ?

@nunomaduro I did not reference "Illuminate\Database\Eloquent\Relations\hasOne" instead I am just defining the Eloquent relationship in User model (same as the code snippet)

Because this issue seems to be inactive for quite some time now, I will close it. This problem should be fixed by now.
Thanks! ☺

Just verified with fresh installation of Laravel 5.6 and it worked without issues.
Thanks


warnings.warn('resource_tracker: There appear to be %d '

Hello dear all. I had an error.
System: Ubuntu 24.04
GPU: 2x3090
Ram: 128Gb
Python version 3.9.8
Can you please help me, where I'm wrong?
Traceback (most recent call last):
File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/saif/saif/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 282, in run
train_and_evaluate(
File "/home/saif/saif/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 496, in train_and_evaluate
scaler.scale(loss_gen_all).backward()
File "/home/saif/saif/venv/env398rvc/lib/python3.9/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/saif/saif/venv/env398rvc/lib/python3.9/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:589] Read error [<IP_ADDRESS>]:20909: Connection reset by peer
/usr/local/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 20 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

Same here

I solved this problem by installing faiss and all its dependencies via conda. BTW, my machine is a Macbook, which can be used as a reference.


Internal Server Error with Local Login

When I enable local login, and then access manually to /login/whatever and that page is not handled by the LoginServlet, throws a NullPointerException

Please post the trace stack here.

Now it doesn't happen again, it's weird


Support for Headless Firefox

Based on @mpasternak's work in https://github.com/pytest-dev/pytest-splinter/pull/144

Coverage decreased (-0.3%) to 87.41% when pulling 980ccf085678ab9ab60d74d90ea8ee76cd8d5383 on zupo:add/headless_firefox into 36ce5a3ef6ddbf2fcd78c7b12941d2cbbb773b97 on pytest-dev:master.

3.3.0 is out


Fix migration ioerror

What this PR does / why we need it:
Live migration tests of a VMI with configMap fail occasionally with an I/O error.
This is most probably because of an inconsistency between the virtual/actual size of configMap/secret disks on the source and the destination. This may partially happen because the test doesn't' delete
the created configMaps and secrets.
This PR also changes the migration flag to override the VMI generated disks.
Fixes #2481
NONE

/retest

/retest

/retest

/retest

This is most probably because of an inconsistency between the virtual/actual size of configMap/secret disks on the source and the destination. This may partially happen because the test doesn't' delete
the created configMaps and secrets.

I would expect that we solely copy the disks over and don't create them again. We basically drag a disk-snapshot of the configMaps and secrets with us, from the timepoint where the VMI was created. So changing the entities on the cluster should  not have any effect.

/retest

I would expect that we solely copy the disks over and don't create them again. We basically drag a disk-snapshot of the configMaps and secrets with us, from the timepoint where the VMI was created. So changing the entities on the cluster should not have any effect.

Yes, that what I was trying to do at the beginning. The problem is that when we try to block migrate a disk that doesn't exist on the destination side with libvirt.MIGRATE_NON_SHARED_INCmigration flag, libvirt doesn't allow to create a new disk. With libvirt.MIGRATE_NON_SHARED_INC when the fine doesn't exist on dest. libvirt requires a store to be defined.
In this PR I let the dest. create the disk and completely override it during migration. I will look into how to define a store and what are the implications.

/retest

Let's ensure that we synchronize this work with the containerdisk PR. With the changes there libvirt can't override the image in the backing chain.

Let's ensure that we synchronize this work with the containerdisk PR. With the changes there libvirt can't override the image in the backing chain.

I reverted to libvirt.MIGRATE_NON_SHARED_INC flag. This change is only about fixing the test now.

/retest

/retest

/retest

/test pull-kubevirt-e2e-os-3.11.0-multus
/test pull-kubevirt-e2e-os-3.11.0-crio

/test pull-kubevirt-e2e-os-3.11.0-crio

/test pull-kubevirt-e2e-os-3.11.0-multus

/test pull-kubevirt-e2e-os-3.11.0-multus

/approve

/retest

/test pull-kubevirt-e2e-os-3.11.0-crio

/retest

/test pull-kubevirt-e2e-os-3.11.0-crio

/hold
I think the right approach is to add a tests.kubevirt.io/something label  to the configmaps on creation and delete them after every test in general.


Warn if sys.path is modified (.append, .insert)

We can more or less reliably detect sys.path modifications with the AST. Whenever this happens, it should be checked by a human because more often than not this isn't actually necessary (and relative imports can be used instead).

Closed by #15


doc update

@MaxJPRey could you help me on this error? What does this mean?
/github/home/vale --output=/lib/rdjsonl.tmpl --config=doc/.vale.ini doc   {"message": "[Vale.Spelling] Did you really mean 'Conda'?", "location": {"path": "doc/source/Getting_started/Installation.rst", "range": {"start": {"line": 141, "column": 19}}}, "severity": "ERROR"}   /bin/reviewdog -f=rdjsonl -name=vale -reporter=github-pr-check -fail-on-error=true -filter-mode=nofilter -level=error   2023/02/25 05:36:32 [vale] reported: https://github.com/pyansys/pyaedt/runs/1159315[30](https://github.com/pyansys/pyaedt/actions/runs/4268565939/jobs/7431141173#step:4:31)18 (conclusion=failure)

@MaxJPRey could you help me on this error? What does this mean? /github/home/vale --output=/lib/rdjsonl.tmpl --config=doc/.vale.ini doc {"message": "[Vale.Spelling] Did you really mean 'Conda'?", "location": {"path": "doc/source/Getting_started/Installation.rst", "range": {"start": {"line": 141, "column": 19}}}, "severity": "ERROR"} /bin/reviewdog -f=rdjsonl -name=vale -reporter=github-pr-check -fail-on-error=true -filter-mode=nofilter -level=error 2023/02/25 05:36:32 [vale] reported: https://github.com/pyansys/pyaedt/runs/1159315[30](https://github.com/pyansys/pyaedt/actions/runs/4268565939/jobs/7431141173#step:4:31)18 (conclusion=failure)

It is because Conda is a proper noun and thus not a common noun allowed by Vale -the tool used to check the typos, grammar... in our documentation-.
I just added Conda to the accept.txt file to notify Vale that it is intended and not a typo.


Rewrite and simplify Player

Hey,
This is a mostly complete rewrite of how logic is handled in the Player.

There are no dead code paths causing the player thread to exit sporadically. The only way that a player thread currently can shut down is either for the commands channel to disconnect or for it to panic.
The various player states have been simplified and documented. In particular the code has been rewritten to not make use of an Invalid state any longer. The EndOfTrack state has been removed.
The loaded track is no longer stored in the player state. Instead it is stored in an loaded_track: Option<LoadedTrack>. This makes it clearer what the lifetime of a loaded_track is and affords us to implement some things decoupled from the state.
The player has no interim panics. Instead all errors are propagated as errors. If an error reaches the thread it will panic.
The end_of_track oneshot is called in the Drop implementation of LoadedTrack to make sure we don't forget about it. Hopefully this means we can write clients without having to match against Err(Cancelled) in the receiving end.

This was also built on top of a previous patch which incorporated linear volume control:
This adds linear volume control at the same stage as normalization happens and adds player events to control the volume directly.
I'm pretty sure this can be implemented somewhere in the audio backend, as an audio filter, or as part of the mixer. But this avoids iterating over the samples twice if normalization happens anyways by multiplying it with the normalization factor.

Thanks for the work, this looks fairly substantial, so I'm probably not going to be able to review it in the near future and test it out, if one of the other colaborators wants to step in, that'd be great.

I'd like to clarify why I did this. I was testing a player that I'm working with for stability by letting it play for a couple of hours. After a while it randomly stopped playing without notifying that the loaded track had ended. I was trying to fix the issues I found with this rewrite and I'm now stability testing this patch again.

Are you sure it's not because of reconnection not being implemented? every issue opened about tracks dropping out seems to come from us not handling reconnection atm.

Can't say for sure. I was hoping this would improve available diagnostics so I could say for sure. All I know for sure is that the oneshot channel wasn't called even though the player had played the full track.

If it is the network, what may be happening is that librespot attempts to retrieve the track, but due to there being no active session, due to connection dropout, the client hangs, the sender isn't triggered, and a restart is required.

Ah. There's two things which make me think my particular problem wasn't reconnect issue. It always hangs at the end of a track. And I'm able to make it play again by loading a new track.
I have not seen this after the rewrite in this PR. Before it happened maybe once every 30 minutes. But I'll keep testing it.

If you can make it play again by loading a new track, it's definitely not a connection issue, as connection issues ultimately cause a crash/necessary restart.

Will try and test this tonight and let you know how it goes. Let me know when you're done adding commits and will review & merge.

Just tested it, I get some odd behaviour with this rewrite though. When starting a song on iOS/desktop and switching to librespot, the playback starts, but then switching back to the official client causes librespot to load the next track and play that, whilst the official client is playing the current song. Quitting the main client and relaunching also does not cause it to sync up with librespot's current state.

@sashahilton00 could this be because the PlayerEvent signals have been changed here:
https://github.com/librespot-org/librespot/pull/300/files#diff-f582d878256027ce3c970a26de626b55R50
Now these are only emitted when a track is loaded, unloaded, or changed. But not when the player for example pauses?

Thing is, changing device should trigger Stop event, as opposed to pause I believe.

Could you explain a bit more 1) what it is that you are doing and 2) what you expect to happen and why? I'm not familiar with the example client (I only use the Player directly), but maybe I can help troubleshoot.

Didn't test myself, but I will present what I think is the problem:
You have 2 Spotify instances running, one is librespot and the other a desktop client.
Steps to reproduce:

Play music on desktop client
Switch Playback device to librespot client
Switch back Playback device to desktop client

Expected behavior: Client should always play the same song
Actual behavior: After 2nd switch of playback device the next song is played by the desktop client
Hope I got the error right

Sure, the steps taken were the following:

Run cargo build
Run librespot
Start song on desktop client.
Switch playback to librespot.
Switch playback back to desktop.
At this point, the desktop continues playback as expected, but librespot skips to the next song in the queue and starts playing, so 2 songs are playing at the same time.

Given that this is still broken and there haven't been any new commits, shall I close it?

Ah, yeah sure. I'm not working on this any longer!