Composite Resource: Enable Adding Referenced File Content via User Interface

Add the ability to add references to content file(s) stored outside of HydroShare but available via an HTTP or FTP URL to a HydroShare resource. Detailed description of this functionality is in the discussion Google Doc: https://docs.google.com/document/d/1739C59tEO4SErKidCbYRsZDJYWkpJk4T1Ir07HICVMg/edit
NOTE: We will add references to map services later as part of another issue.

[x] Add button to toolbar of the file browser interface in edit mode on the resource landing page to "Add content from the web." This should display a modal window for adding content from the web. The user will input a URL for the reference in the modal window. The modal window should look like the following.  References to files can be located anywhere within the folder hierarchy of a resource (except nested within other aggregations).

NOTE: Initially there will be only one option for content type (add reference to file or website using an FTP or HTTP URL), but we will add additional content types (web services) via additional issues.

[x] Create validation for URLs. When a user specifies a URL as a reference to external content, HydroShare should validate the URL to ensure that a 400 error (content not found) is not returned and that a 401 error (user not authorized) is not returned.

[x] Create .url file to store reference. For each reference, a .URL file will be created that stores the URL of the referenced content. The .url files will be named using the name of the remote file with a .url appended to the end. When displaying .url reference files in the file browser interface, the .url file extension should be removed so the user doesn't see it.

[x] Display referenced files in file browser with unique icon(s). Referenced files (.url files) should show up in the file browser interface with a unique icon that visually indicates that they are referenced from another location.

[x] Verify ability to create metadata for a single referenced file (.url file) (e.g., create a single-file aggregation from the .url reference file). The user should be able to right click on a single .url file and select "Add metadata to file". This will have the same effect as the existing functionality for this action when initiated on any other file stored within HydroShare's file system - an OAI-ORE resource map and aggregation metadata document will be created. See notes for how the OAI-ORE resource map document should be encoded in the Google Doc.

[ ] Verify ability to create metadata for a folder containing referenced files (e.g., create a file-set aggregation from .url reference files in a folder). The user should be able to right click on a folder containing any .url files and select "Add metadata to folder". This will have the same effect as the existing functionality for this action when initiated on a folder with files stored within HydroShare's file system - an OAI-ORE resource map and aggregation metadata document will be created within the folder. See notes for how the OAI-ORE resource map document should be encoded in the Google Doc.

[x] Add messaging to user interface for referenced content. When a resource contains referenced content (of any type), add the following text directly below the Content browser:

Please note: This resource contains links to external content. Linked content is NOT stored in HydroShare, and we can't guarantee its availability, quality, or security.

[x] Implement packaging rules for download of resources or parts of resources containing referenced content.

Right clicking on an individual .url file and selecting "Download" should download the .url file.
Double clicking on an individual .url file should result in the user receiving the following warning, after which HydroShare will redirect a new browser tab to the URL contained in the file.

The content you have requested to access is not stored in HydroShare, and we can’t guarantee its availability, quality, security, or size. If the externally linked content is large, access may take time.

Right clicking on a folder that contains one or more .url files and selecting "Download" should result in the user receiving a zipped file containing the folder's contents as is current behavior.
Double clicking on or right clicking on a single referenced file aggregation should be the same as individual referenced files (above).
Requesting download of a fileset aggregation containing .url files should be the same as requesting download of a folder except that the zipped file delivered to the user should contain the OAI-ORE resource map and aggregation metadata documents (as is current functionality).
Requesting download of an entire resource containing referenced content should return a zipped BagIt bag with all resource map documents, metadata documents, any content files stored within HydroShare's file system, and any .url files (as is current functionality).

[ ] Disable publication of resources containing referenced content. As soon as referenced content has been added to a resource, disable the "Publish" button. If all referenced content has been removed from a resource, it can be publishable again. We may also want to remove the message about permanent publication with the URL-encoded DOI below the "How to Cite" for resources that can't be published.

[ ] Implement an option for modifying the URL for referenced content. Once a reference has been created and a .url file exists in a resource, an option should be created in the file browser interface to right click on the .url file and select "Edit reference URL."

Check that the following still work once complete:

[ ] Copying resource
[ ] Creating a new version of a resource

@horsburgh I don't see adding referenced content from create resource page mentioned in the task list on this issue. I assume in addition to allowing users to add referenced content from resource landing page after a resource has been created, we should also allow users to create resource using referenced contents initially. Want to confirm with you whether this is the case or not.

@hyi - this is a good question. I debated a bit about whether we should add this to the resource creation page. Once we have consolidated resources on a single resource type (composite), I think the create resource page will go away entirely. So, I'm not sure we want to add new functionality to that page. On the other hand, we've still got some work to do before we no longer require users to select a resource type, which is really the only reason to have that page at the moment.

@horsburgh Got it. If we got rid of resource creation page, it'd mean the resource would always be created with the default "Untitled resource" and empty contents initially. So there might still be merit to have a resource creation page initially even if we got rid of resource type selection, but that is a UX issue, and it'd most likely work better if we got rid of resource creation page and streamline resource landing page to make editing much easier for users. Agreed we should not consider adding any more feature into resource creation page at this point.

I modeled the messages after those that I had seen on other websites. If you want to change them go ahead. I added the text about clicking on .url files because I'm concerned that users will not know what to do with the .url files once they download them.

@horsburgh This is not urgent, but if you could ask @Maurier to create an icon for referenced .url files in file browser when he has time, that'd come in handy for me to use while working on this issue.

@horsburgh Regarding this message: The file you have requested to access has a size of XXXX. It is not stored in HydroShare, and we can’t guarantee its availability, quality, or security. I suggest to remove the size info as we most likely will not know the size info from the external URL. We could make the message as The file you have requested to access is not stored in HydroShare, and we can’t guarantee its availability, quality, or security.

@hyi - sure, given our conversation the other day I think it's OK to remove the size info. I suggested providing size info so that a user would know what they were asking for. I originally thought that HydroShare could do a header request to get the size info when the reference was created so we could present that information to users. But, I understand that there are reasons that we may not want to do that. Some referenced files may be quite large - as large files stored elsewhere is one of the use cases for referenced content.

@Maurier will work on an icon for you @hyi

@hyi
Here is how you can add the icon:
on https://github.com/hydroshare/hydroshare/blob/develop/theme/static/css/hs-file-icons.css add the following rules:
.fb-stack-url {
    color: #607D8B;
}

.fb-view-grid .fb-stack-url .fa-link, .dz-details .fb-stack-url .fa-link {
    top: 4px;
    font-size: 19px;
}

.fb-view-list .fb-stack-url .fa-link {
    top: -4px;
    font-size: 11px;
}

on https://github.com/hydroshare/hydroshare/blob/develop/theme/static/js/hs-file-icons.js add the following entry:
    fileIcons.URL =
        "" +
        "" +
        "" +
        "";

This should give you an icon that will be used for .url files:

@Maurier Got it. Many thanks for the quick and detailed response.

@horsburgh A quick question regarding your comment: References to files can be located anywhere within the folder hierarchy of a resource (except nested within other aggregations)
Does it mean if a folder can be created within the folder hierarchy of a resource, then references to files can be created as well? I am trying to see whether I can leverage existing flag for enabling or disabling folder creation action in a folder hierarchy of a resource to enable/disable adding reference content to resource as well, and it appears to me the folder creation enabling/disabling flag applies to adding reference content to a specific folder in a resource folder hierarchy as well. Let me know whether it is safe to make this assumption, i.e., use folder creation enabling/disabling flag to control enabling/disabling for adding reference contents as well. If the answer is yes, I can rename that flag to make it cover both cases; otherwise, if the answer is no, I'll create a new flag for adding reference contents, but I'll need to know what is different between actions of creating a new folder and adding reference content, so I can set the flag accordingly.

@hyi - I think the answer is yes. The point of this comment was that you can put references in any folder anywhere in the hierarchy - except within specific aggregation folders that do not permit it. For example, you should not allow creation of a reference file within a geographic feature aggregation folder or within a geographic raster aggregation folder. I believe that the same rules that have been implemented for creating new folders should apply to referenced files as well - if they have been implemented correctly.
@dtarb, @sblack-usu, and @pkdash - am I getting this right. Best to have more than one set of eyes before Hong uses her proposed method. I think it works, but let us know if not.

yes, this makes sense to me.  Perhaps name that flag to something more general or comment it well so the next person to come along isn't confused why a folder flag is used for reference content.

@sblack-usu Yes, that is what I intend to do, to rename that flag to cover both cases to eliminate confusion.

@hyi maybe we could use this one for .url files:
https://fontawesome.com/v4.7.0/icon/external-link
This one would indicate better that the file opens outside of HydroShare.

@Maurier Agreed that external-link icon looks like a better fit to add content from the web. Will update the code to use that icon. Thanks!

@horsburgh What is the rationale behind When displaying .url reference files in the file browser interface, the .url file extension should be removed so the user doesn't see it? Special logic will be needed to handle hiding .url file extension from the file browser interface. Before I add any such special logic, I am wondering whether this is really needed. For me personally, having .url extension in the file browser interface is preferable as it clearly indicates the file is actually a .url file. Let me know whether we can treat .url extension in the same way as any other uploaded files without hiding the .url extension.

@horsburgh Regarding the task The user should be able to right click on a folder containing any .url files and select "Add metadata to folder". This will have the same effect as the existing functionality for this action when initiated on a folder with files stored within HydroShare's file system - an OAI-ORE resource map and aggregation metadata document will be created within the folder., I confirmed with @sblack-usu that Add metadata to folder functionality is not in for 1.16 release yet but @pkdash may be planning on adding that after 1.16. This "Add metadata to folder" is a generic functionality for composite resource overall without any special handling needed for .url web reference file. So I suggest to remove this task from this issue but treat "Add metadata to folder" as a generic feature for composite resource. When that generic feature is implemented after 1.16 release, it will work for .url web reference file as well.

Add metadata to folder is generic Composite resource functionality and not specific to referenced content at all. But, I thought it had already been added to composite resource functionality? @pkdash can set us straight.

@horsburgh A update regarding the task: Verify ability to create metadata for a single referenced file (.url file) (e.g., create a single-file aggregation from the .url reference file). The user should be able to right click on a single .url file and select "Add metadata to file".  What I have done is to automatically create a generic logical file aggregation for the .url file at the time it was added to the resource. The reason for this is that I need to record url into a generic logical file field (not metadata per se, but only a generic extra_data field for internal use) for doing redirecting to the url when, e.g., double-clicking on the .url file. The result of this is that you can directly see and add metadata to .url file generic aggregation from content metadata tab after selecting the file, i.e., it skips the step to click on Add metadata to file and this menu item is actually disabled for .url file. I don't think this minor difference would matter much, and in fact, I like the current implementation since it saves one extra step from users, and thus more streamlined.

"Add metadata to folder" is not part of 1.16. I am currently working on this as part of the new FileSet aggregation/file type.

Thanks for the reminder @pkdash - I'll look forward to that functionality after 1.16.
@hyi - if you are automatically creating an aggregation for the .url files, I think that's fine unless there are any performance implications. I suspect that's not an issue.

@hyi - the purpose of my suggesting that we remove the ".url" part of the extension was that it is going to be confusing to users, most of whom will not know what a ".url" file is. If you want to leave it for now for the first implementation, I'm fine with that.

@horsburgh I don't notice performance implications resulting from automatically creating a generic aggregation for .url files since the generic aggregation creation is fast as it does not involve time-consuming metadata extraction for other specific aggregation types. Regarding keeping the ".url" extension, we may add a brief message about .url with a link such as https://fileinfo.com/extension/url after .url file is added to educate the user a bit on .url file we are using to encode web reference content.

@hyi - my concern with performance wasn't with auto-creating an aggregation for an individual .url file, but rather what would happen if I had a whole bunch (say hundreds) of .url files in a resource. If that's not a problem, then this is great.

@horsburgh That is a good question and valid concern. I don't anticipate performance issues on the server side even with hundreds of .url files in a resource, as long as the client can display all those files efficiently in file browser UI. We can do some stress testing perhaps after the REST API is also implemented so we can have an idea whether we need to address any potential performance impact.

@horsburgh Regarding Double clicking on an individual .url file should result in the user receiving the following warning, after which HydroShare will redirect a new browser tab to the URL contained in the file, since .url file is auto-aggregated into a generic logical file, double clicking on an .url file will download the single file aggregation as the current default behavior. I suggest we keep this default behavior, and add a menu entry redirect to referenced URL when right clicking on a .url file. Let me know what you think.

@horsburgh I took back my suggestion above after giving it a bit of more thought and after seeing your subsequent bullet point Double clicking on or right clicking on a single referenced file aggregation should be the same as individual referenced files (above). I will implement as you specified, i.e., doubling clicking on a .url file will result in the user receiving the warning and then redirecting a new browser tab to the referenced url.

@hyi - I'm not sure what your plan for implementation of the redirect will be, but I think it's worth building it in a way that a user can share the link to the referenced resource in hydroshare, where hydroshare will then redirect to the url in the .url file.  I actually believe that the download link would be good for this because it doesn't make much sense to ever download a .url file.
If you don't agree with the download link being repurposed for .url file redirection, I propose we setup a different url that is standalone from the UI, that gives the redirect warning with a continue button.
For example, I think if a user pastes a reference to the url file in the browser https://www.hydroshare.org/resource/4db3e81d3ccb4826b3c5451995ca2882/data/contents/example.url it should take you to the hydroshare redirect warning page and then redirect.  That same url could be opened into a new tab when double clicking the url file in the file browser.  This will give an easy way to access these referenced resources through hydroshare.  It would also encourage people to share the hydroshare link rather than the link inside the url, which could give us some useful metrics to track.

@sblack-usu That is a good point. Since .url file is auto-aggregated into a generic file aggregation, the download of the url file currently actually downloads a zipped aggregation with metadata included similarly as other aggregations. We allow users to add metadata into .url aggregation anyway, so this is consistent with aggregation design as detailed in this issue by @horsburgh. When users double click on the .url file, it will bring up a dialog and ask users for redirect. I think having a reference URL that allows to directly redirect to the referenced URL is a good idea, but having the download URL to download an aggregation with metadata included is also useful and consistent with HydroShare aggregation behavior across the board. Perhaps we can have a redirect parameter in the URL to inform doing redirect or not specific to .url files. For example, https://www.hydroshare.org/resource/4db3e81d3ccb4826b3c5451995ca2882/data/contents/example.url downloads a zipped aggregation, but https://www.hydroshare.org/resource/4db3e81d3ccb4826b3c5451995ca2882/data/contents/example.url/redirect redirects to the referenced URL? @horsburgh what do you think?

@hyi - I discussed this a bit with @sblack-usu yesterday. I think having one way to handle both of these situations would probably be best. In HydroShare's resource landing page, we have proposed changing the behavior of a double click on a ".url" file so that instead of downloading the .url file (as is done with all other files), the user is redirected to the URL contained in the .url file after receiving a warning. There should also be a right click menu item on a .url file that says something like "Open referenced content" or something like that.
As @sblack-usu suggests, instead of the warning being a modal window or pop up message on the resource landing page, could we precede the redirection with a separate page that opens in a new tab, warns the user about the redirection, and allows the user to choose whether they want to continue with the redirection? We could also give them the option to just download the .url file on this page.
The reason for having a separate page rather than a modal or pop-up of some sort on the resource landing page is that it would work for all cases: 1) when a user double clicks on a .url file on the resource landing page, 2) when a user right clicks on a .url file and selects "Open referenced content", or 3) when a user pastes a HydroShare link to a .url file into their browser (e.g., https://www.hydroshare.org/resource/4db3e81d3ccb4826b3c5451995ca2882/data/contents/example.url).

@horsburgh @sblack-usu A couple of comments:

The current spec does mention Double clicking on an individual .url file should result in the user receiving the following warning, after which HydroShare will redirect a new browser tab to the URL contained in the file. However, when users right click on the .url file and select download, the aggregation zip file will be downloaded similarly to other aggregations. I think this is reasonable. We can add a right click menu item like "Open referenced content" on a .url file that will do the redirect similarly to double clicking on the .url file. Will this be sufficient?
I like a modal popup warning better than a separate page since it is more tightly integrated with resource landing page. Opening up a separate page on a separate tab is a little distracting in my opinion. The modal pop up dialog can be shared between different actions as well, except when a user pastes a HydroShare link to a .url file into their browser. However, this is a different issue since this HydroShare link to a .url file is currently behaving as downloading the file or aggregation just like other files. Even if we want to replace modal pop up with a separate page, we have the same issue how to handle this URL link to a .url file. Our current URL mapping just maps this to download view function, we can easily add some logic to check for .url file extension and do redirect for .url file as needed in the download view function, if this makes more sense. If we want to support both download and redirect via the same URL, we can create such a page where both download and redirect button are added. However, I think having a modal popup warning to work with file browser on resource landing page is better in terms of consistency with rest of file browser UI. The only drawback as I can see is to replicate the warning language in both modal popup dialog (where only redirect button (i.e., no download button) is included) and the redirect page (where both redirect and download buttons will be included). Given more seamless UI in file browser, I feel this drawback is acceptable. Although replacing the modal popup dialog with redirect page is easy to do, I just like modal popup better when working within file browser and that is already implemented in my local branch.

@hyi - on your first point, yes, that would be great to have a right click option on a .url file to "Open referenced content"
On your second point - a single redirect page seems simpler and would be consistent across both use cases. If you feel strongly about the modal window for the HydroShare user interface, that's fine. But, we would still have to handle the case where a user pastes a HydroShare URL to a .url file into their browser. This would need to be a separate HydroShare page that handles those requests to see whether the user wants to download the .url file or be redirected to the referenced content. That page should probably have a message that says something like:

You are accessing referenced content that is not stored in HydroShare. Do you want to download an Internet Shortcut File that contains a link to the referenced content or have your browser redirected to the location of the referenced content?

@horsburgh We have a URL already that tracks external activity. @Castronova wrote it and uses it for tracking app opens. You could easily reuse that work @hyi
https://github.com/hydroshare/hydroshare/blob/develop/hs_tracking/views.py

@horsburgh @sblack-usu I believe I have implemented all tasks covered in this issue and I am deploying the first pass of the implementation to playground right now. Will let you know when the deployment is done and playground is up and running. One thing I want to bring up is that instead of implementing a separate page asking users to either download or redirect for a url file, I implemented redirecting to the referenced url directly if the URL file encodes a valid referenced URL. Since users can download the file or aggregation from within HydroShare file browser interface, I think it is acceptable to directly redirect if a file URL is put in URL field on web browser directly. Hopefully you have time to take a look on the first pass implementation on playground when it is up before our team call meeting today so we can discuss any comments/issues as needed.

Thanks @hyi - I'll have a look as soon as you are ready for testing.

@horsburgh - playground is up and running now. I have done a quick testing and everything appears to work as expected. You will have to sign up and create a user account first since it is not deployed with existing www resource data.

@horsburgh @sblack-usu @ChristinaB @danames  I have fixed the redirect bug when putting a file URL on a web browser on playground.hydroshare.org, discovered during the demo today. It now works regardless where the URL file is in the folder structure. Feel free to give a test drive on playground and chime in with any suggestions/comments/feedback.

@hyi - I just did some testing on playground. Here's some issues I encountered:

I tried to edit the URL for a reference (right click, edit referenced URL) to an invalid URL. I got a huge, long, red error message on the page. We need to handle this more gracefully. I suggest an error message that says:

"The URL that you entered is not valid. Please enter a valid URL."

Updating the URL for a referenced file removes all file/content level metadata. Was this by design? I don't think we should do this. I think the URL should be edited in the .URL file, but the file-level metadata should remain.
For some reason when I download the whole resource I created, the tagmanifest-md5.txt file is incorrect. It contains all of the Bagit tag files (which it should), but it also contains a listing of all of the content files (which it should not). This may be a separate issue that needs to be addressed.
For the window that pops up when you click on the "Add content from the web" button, can we add some additional text to the label above the "Name" field? I suggest:

Create a name for the referenced content. This name will be displayed in the file content area of the resource.

I create a new folder and dropped a shapefile into it. That automatically converted that folder to a geographic feature aggregation. I then dragged one of my .url file onto this folder, and it let me drop the .URL file in that folder. This should not be possible. I should not be able to nest a referenced file aggregation within a geographic feature aggregation. It also let me upload files to this same folder. There should be logic preventing this from happening right?

Implementation of the metadata file and the resource map document file at the single-file aggregation level looked good. It also looked good at the resource level. I was able to move .URL files into and out of folders no problem.
I also tried right clicking and selecting "Get file URL", which gave me a HydroShare URL for the .URL file. I pasted that into my browser window and it immediately redirected me to the remote URL, so that also looks good.
I did not do any testing via the API and there may need to be additional considerations there.

@horsburgh I believe I have addressed all your comments and updated playground for you to test again with one caveat regarding your comment item No. 3 about tagmanifest-md5.txt. First, the tagmanifest-md5.txt is generated by iRODS rule, which has not been changed for a long time. Second, I cannot reproduce it locally. Third, I tested on testservices.hydroshare.org which does not have my code, and it has exactly the same issue with tagmanifest-md5.txt content. I believe this is iRODS-related since all test VMs deployed automatically using Jenkins uses hydrotest41.renci.org iRODS host and thus exhibit same issue with content in tagmanifest-md5.txt. It may be related to iRODS server upgrade, but I have no plausible explanation how come the same iRODS rule (which does not write content file manifests into tagmanifest-md5.txt) somehow put content files into tagmanifest-md5.txt probably only when being executed on this hydrotest41.renci.org iRODS host. I tested on www and https://dev-hs-1.cuahsi.org/ which @sblack-usu deployed with latest develop branch and this tagmanifest-md5.txt problem does not occur there. I merged latest develop into my branch so what is deployed on playground now has latest develop content along with my referenced content work. Anyway, I don't think this tagmanifest-md5.txt issue is introduced by my referenced content work, it looks like a deployment-specific/irods-specific issue although I have no idea what could have caused that. Also, your comment item No. 5 is a generic bug in composite resource phase 2 implementation which I have fixed in my branch. Please test again when you have a chance and let me know any further issues you find.

@hyi - i went through all of the tests I described in my last comment above. Everything checked out fine except for number 3 - the BagIt tag manifest file still contains all of the resource content files. Since you have indicated that this is a configuration setting specific to the testing server, I won't worry about it.
One other minor note - I can no longer drop a ".URL" file into a folder that is a geographic feature aggregation. When I try to do that, I get an error message. But, if I open that folder and try to drag and drop a file from my machine onto the resource, it doesn't work (which is good), but I don't get an error message (or any message at all). I think this is unrelated to your functionality and so I think what you have done is great. Just noting that there's an inconsistency here.

Deployed with 1.16.1.  Thanks team.


⚠ LPDM Handbook (hosted on Divio) has degraded performance

In 5b2d1ee, LPDM Handbook (hosted on Divio) (https://repohubdev.tk) experienced degraded performance:

HTTP code: 500
Response time: 5358 ms

Resolved: LPDM Handbook (hosted on Divio) performance has improved in 1b68d2f.


[consensus] misc improvement

Motivation
See each commit message.
Have you read the Contributing Guidelines on pull requests?
(Write your answer here.)
Test Plan
(Share your test plan here. If you changed code, please provide us with clear instructions for verifying that your changes work.)
Related PRs
(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/diem/diem/tree/main/developers.diem.com, and link to your PR here.)
If targeting a release branch, please fill the below out as well

Justification and breaking nature (who does it affect? validators, full nodes, tooling, operators, AOS, etc.)
Comprehensive test results that demonstrate the fix working and not breaking existing workflows.
Why we must have it for V1 launch.
What workarounds and alternative we have if we do not push the PR.

@JoshLind good question this is another attempt to fix #9766, the callback is used to cleanup blocks. it should be good as long as the callback is only triggered after mempool acks the transactions (which previously not guaranteed).

/land

:broken_heart: Test Failed - ci-test

/land


Auto-globbing in cmd tasks makes it impossible to pass anything that looks like a glob to the called command

I'm having an issue with the following in my pyproject.toml
[tool.poe.tasks]
run-tests = "python -m unittest discover -v ./tests/"

This works fine:
PS D:\PS\services\projects\scaler> poetry run poe run-tests
Poe => python -m unittest discover -v ./tests/
.... <Tests run>

However, this fails:
PS D:\PS\services\projects\scaler> poetry run poe run-tests -p "test_unit*.py"
Poe => python -m unittest discover -v ./tests/ -p
usage: python.exe -m unittest discover [-h] [-v] [-q] [--locals] [-f] [-c] [-b] [-k TESTNAMEPATTERNS] [-s START] [-p PATTERN] [-t TOP]
python.exe -m unittest discover: error: argument -p/--pattern: expected one argument

Specifically, "py_unit*.py" has disappeared. This argument is a glob-pattern to be passed as-is to Pytest, so should not be expanded. The bash-like behaviour of "use glob as-is if not matched" would be less-surprising, producing
Poe => python -m unittest discover -v ./tests/ -p "py_unit*.py"

The behaviour of course is closer-to-expected if I happen to be in 'tests'
PS D:\PS\services\projects\scaler\tests> poetry run poe run-tests -p "test_unit*.py"
Poe => python -m unittest discover -v ./tests/ -p test_unit_all_the_tests.py
... <Runs tests from that file>

i.e. when I first set this up, it appeared to be working, but only because I happen to have just one test file matching the glob here. It also will then fail to match any subdirectory-held test files that aren't also named test_unit_all_the_tests.py.
In bash, 'py_unit*.py' would mean "Don't expand this", but there's no way to communicate that to poethepoet as the quotes are already gone before it gets the argument.
[tool.poe.tasks]
run-unit-tests = "python -m unittest discover -v ./tests/ -p 'test_unit*.py'"

has the same problem, except it's always globbed from the top-level, not from the directory I happen to be in, leading to:
PS D:\PS\services\projects\scaler\tests> poetry run poe run-unit-tests -p "test_unit*.py"
Poe => python -m unittest discover -v ./tests/ -p -p test_unit_all_the_tests.py

where the same glob has expanded two different ways.

I'm not sure the 'auto-glob' behaviour makes sense, as there's literally no way for me to express "This is a glob, that isn't" through a cmd task, as it unconditionally tries to expand globs.
It's also a somewhat surprising behaviour, since if I wanted that, I'd use the shell runner, or e.g., a pwsh runner per #15, and let it do the globbing.
I don't see a particular justification (or documentation) for the "auto-glob" behaviour, so personally I would suggest removing it from cmd.py, along with the vestigates in shell.py. That would mean the rm -rf example in the Command Tasks list moves to shell, where it belongs, and is rewritten to use a bash glob, instead of a Python glob, i.e. there's no ** in bash.
If it's preferred to be kept, then having a way for the task to specify "No, do not auto-glob my commands" would be handy. I'd just use that all the time, so that poetry run poe X Y Z is literally poetry run <definition of X> Y Z as I would expect.
As another argument towards dropping auto-globbing, I've just realised this would make grep or sls nigh-unusable here, as a regexp on the command line will always be globbed-and-deleted, unless you're very unlucky and it globs into an existing filename, which would not have matched the regexp.

Thanks for the feedback. The existing behaviour does seem problematic. The auto globbing was a very early feature from when I was still trying to avoid having something like shell tasks and I guess it wasn't that well thought through.
Looking at it now it seems a better way to achieve what I wanted would be just to set shell=True on the Popen call, rather than reimplementing part of the shell behaviour 🤨. One of the reasons I recall not doing this originally was that I'm not so sure whether it would behave consistently on windows (and I don't have ready access to a windows dev setup), but I guess that's what functional tests are for :P.
I should look into this some more. Or maybe make the globbing behaviour opt in as you suggest, though I think there's value in not having to use "shell" tasks for most things due to the complexity that introduces à la #15 .

Thanks for the feedback. The existing behaviour does seem problematic. The auto globbing was a very early feature from when I was still trying to avoid having something like shell tasks and I guess it wasn't that well thought through.
Looking at it now it seems a better way to achieve what I wanted would be just to set shell=True on the Popen call, rather than reimplementing part of the shell behaviour 🤨. One of the reasons I recall not doing this originally was that I'm not so sure whether it would behave consistently on windows (and I don't have ready access to a windows dev setup), but I guess that's what functional tests are for :P.
I should look into this some more. Or maybe make the globbing behaviour opt in as you suggest, though I think there's value in not having to use "shell" tasks for most things due to the complexity that introduces à la #15 .

I kind of expected, similar e.g., to Dockerfile CMD, that cmd vs shell would differ only in the shell parameter to Popen. These extra behaviours (cmd-type applies Python's glob, shell hunts really hard for bash of some kind, no matter the local platform) have been a surprise to me.
I'm not keen on cmd-type using Popen( ..., shell=True ) simply because that implies another cmd instance (per $COMSPEC) in the chain, and that's approximately never what I want. And unlike POSIX, there's no way to override that (according to the docs the executable parameter is only for POSIX). I really just want cmd-type to be Popen(args_from_task + args_from_sysargv, shell=False), so it does exactly what I wrote.
The current shell approach is better than Popen( ..., shell=True), as it's "bash or fail", but per #15 it probably needs to fail more easily than it does... And it probably needs to be clearer that it's not "system shell" but "bash". Which is why I suggested in #15 that shell isn't the right name for it: it implied to me that it was just Popen( ..., shell=True), and the examples for shell certainly matched, since many of the cmd examples appear to assume an underlying POSIX system too.
Supporting a shell binary override for shell types would probably help, as then you'd be able to specify exactly what to script against, and shell type could be used for, e.g. "Powershell scripts" as well. Or cmd scripts if you're desperate and only need to support Windows, I guess. It would also mean "default shell is bash" is surfaced in the docs, as part of documenting the override for it.
That would probably also be a good opportunity to get rid of the $SHELL check, since that's not really promised to be bash and kind-of steps away from the idea of a "fixed, known environment that works the same everywhere" that we get from Poetry.
It might be better to check if we are running under bash, and then try /usr/bin/bash, and then /usr/bin/sh, and then /bin/sh, and then give up. It's only very recently on recent Windows 10 where you could run Windows-native 'python.exe' under bash from recent gitforwindows or MSYS2, and get the right results; WSL would be right-out, and it's the most-likely bash.exe in the Windows path by default.
Particularly for me, the main use-case I have for poethepoet is to replace the various .bat and .sh scripts that have grown up to wrap the various python invocations my team uses, with platform-agnostic versions, ideally stored in pyproject.toml with the rest of the project definition.

I kind of expected, similar e.g., to Dockerfile CMD, that cmd vs shell would differ only in the shell parameter to Popen. These extra behaviours (cmd-type applies Python's glob, shell hunts really hard for bash of some kind, no matter the local platform) have been a surprise to me.
I'm not keen on cmd-type using Popen( ..., shell=True ) simply because that implies another cmd instance (per $COMSPEC) in the chain, and that's approximately never what I want. And unlike POSIX, there's no way to override that (according to the docs the executable parameter is only for POSIX). I really just want cmd-type to be Popen(args_from_task + args_from_sysargv, shell=False), so it does exactly what I wrote.
The current shell approach is better than Popen( ..., shell=True), as it's "bash or fail", but per #15 it probably needs to fail more easily than it does... And it probably needs to be clearer that it's not "system shell" but "bash". Which is why I suggested in #15 that shell isn't the right name for it: it implied to me that it was just Popen( ..., shell=True), and the examples for shell certainly matched, since many of the cmd examples appear to assume an underlying POSIX system too.
Supporting a shell binary override for shell types would probably help, as then you'd be able to specify exactly what to script against, and shell type could be used for, e.g. "Powershell scripts" as well. Or cmd scripts if you're desperate and only need to support Windows, I guess. It would also mean "default shell is bash" is surfaced in the docs, as part of documenting the override for it.
That would probably also be a good opportunity to get rid of the $SHELL check, since that's not really promised to be bash and kind-of steps away from the idea of a "fixed, known environment that works the same everywhere" that we get from Poetry.
It might be better to check if we are running under bash, and then try /usr/bin/bash, and then /usr/bin/sh, and then /bin/sh, and then give up. It's only very recently on recent Windows 10 where you could run Windows-native 'python.exe' under bash from recent gitforwindows or MSYS2, and get the right results; WSL would be right-out, and it's the most-likely bash.exe in the Windows path by default.
Particularly for me, the main use-case I have for poethepoet is to replace the various .bat and .sh scripts that have grown up to wrap the various python invocations my team uses, with platform-agnostic versions, ideally stored in pyproject.toml with the rest of the project definition.

After taking some time to analyse the problem I think you're right that it doesn't make sense to mess with args passed from the user's shell, and in general quoting an arg should disable glob expansion for that arg. I'll fix it accordingly, which I hope will address your pain point.

Thanks also for taking the time to share your perspective. The shell task needs some refinement, though I think there's value in the "simple" cmd task having a little shell-like magic for convenience.

After taking some time to analyse the problem I think you're right that it doesn't make sense to mess with args passed from the user's shell, and in general quoting an arg should disable glob expansion for that arg. I'll fix it accordingly, which I hope will address your pain point.

Thanks also for taking the time to share your perspective. The shell task needs some refinement, though I think there's value in the "simple" cmd task having a little shell-like magic for convenience.


ZFS boot support is broken for Virtio drives with unassigned serial numbers

Virtio drives w/o serial numbers aren't visible in /dev/disk/by-id but ZFS boot logic attempts to import zpools from /dev/disk/by-id only.
Possibly other paths (e.g.  /dev/disk/by-path) should be tried too.
An easy way to reproduce that would be to create a libvirt machine with one Virtio drive and unassigned serial number, install NixOS with ZFS support and try to boot.

This is a configurable behavior via boot.zfs.devNodes.  Just remember to set it correctly before nixos-install.  Or you can also use an Root on ZFS guide.

Ok, I get the point. Was there any specific motivation to make the defaults as restricted as they are at the moment?
Also, maybe, the boot error message should mention this option?..

Maybe it is because by-id aliases are human-readable and used by many
Root on ZFS guides.  For example, if, on a physical server, a disk
fails, you will be able to look up the serial number of the failed disk
in zpool status and replace it.
ZFS does not care about this, you can set it to /dev/ if necessary.  But
then the output will be /dev/sda /dev/nvme0 /dev/vda, etc.
"Paul S." @.***> writes:

Ok, I get the point. Was there any specific motivation to make the defaults as restricted as they are at the moment?
Also, maybe, the boot error message should mention this option?..
--
Reply to this email directly or view it on GitHub:
https://github.com/NixOS/nixpkgs/issues/156897#issuecomment-1479966240
You are receiving this because you commented.
Message ID: @.***>


Write a Swift program to divide two numbers

Description
Write a Swift program to divide two numbers
Input  : 2 4
Output : 0.5

How to contribute

Save the solution in DivideTwoNumbers.swift file
Add DivideTwoNumbers.swift file in divide-two-numbers folder

!assign


⚠ 366.ru has degraded performance

In d7ff1fa, 366.ru (https://366.ru) experienced degraded performance:

HTTP code: 200
Response time: 2167 ms

Resolved: 366.ru performance has improved in f9ea605.


MySQL error on stop

Hello,
I installed your plugin, but getting this error while stopping the server: https://paste.soleria.eu/iviqawukak.md
The database is empty (only tables are created). i didn't used any command of the plugin.
EDIT: Happening when someone is connecting / disconnecting too.

Does the plugin work with SQL-Lite?
What MySQL/MariaDB version are you using?

It works with SQLite.
I'm running on MariaDB 10.5.12

Investigating right now, expect a fix by the end of the day.

Fixed in release 1.1.7.
https://github.com/Cowings/PunishmentsX/releases/tag/1.1.7
Thank you for reporting this issue!


Feasibility of a future "one-liner" command for compiling a Rust lib to Wasm/JS?

💡 Feature description
Basically a wasm-pack command/feature that automatically adds wasm-bindgen attributes where they need to be throughout a Rust codebase (during the build process) to match the Rust API that is publicly exposed in lib.rs.
💻 Basic example
I'm a web developer, and have only dabbled with Rust and wasm-pack in the past, but recently I've been looking into how to compile huggingface/tokenizers to Wasm so it works on the web. The lib.rs file looks like this:
pub mod utils;
pub mod decoders;
pub mod models;
pub mod normalizers;
pub mod pre_tokenizers;
pub mod processors;
pub mod tokenizer;
pub use tokenizer::*;
...

Speaking very naively, it seems like this lib.rs could tell wasm-pack everything it needs to know about what I want to export in the resulting tokenizers.js module. Feature flags and cfg attributes would be added by the developer to remove certain things from the wasm build if needed, and maybe some wasm-bindgen attributes could be sprinkled around in cases where the default behavior is not desired.
So I'm just wondering whether this is remotely feasible, or whether there are some fundamental constraints that will mean we will always need to add wasm-bindgen attributes throughout the whole codebase?
Thanks! (And apologies if this sort of question has been answered somewhere before - it seems like it might be a common question, but I wasn't able to find anything on the topic)

Hey @drager, please completely ignore this if you don't have time or it's a silly question (i.e. indicates insufficient research on my part), but otherwise I was wondering if you could make a short comment here?
The TL;DR is basically the feature description in the above comment. Wondering what the barriers are to achieving that sort of thing, or if it's not plausible at all for some reason?


Describe ESV - when returning a URL, doesn't display correctly.

ESV value stored in FIDC:
https://login.salesforce.com/services/oauth2/token
Using:
frodo esv variable describe -i esv-endpoint-uri https://tenant.forgeblocks.com/am
Returns:
Describing variable esv-endpoint-uri...
Name         │esv-sf-token-endpoint-uri
Value        │gin.salesforce.com/services/oauth2/token

Looks like the word wrapping logic is causing this. I will look into this.


Dropdown field values should be sorted

In general, dropdown fields should be sorted alphabetically.  The exception is for Status when everyday use dictates the order.

Fixed in commit https://github.com/ytgov/eco-asset-tracker/commit/b749bb56b2566b565a5f04d98b5eabffa2b7a8f2


search function should be available for all kind of information (SC-TEI-6,8,10,12,23,..)

The current version (0.3) only allows filtering the entities. But, it should be a functionality for all kind of information: layers, models, risk configurations. I'm not sure for risk sessions, non-paginated lists (trees for now), this functionality seems not so necessary.
the TEI test cases SC-TEI-28 is about looking for risk indicators (inside the model)
For the paginated lists are more necessary

a search function has no sense for Layers, since there should be not more than a handful, for several practical and theoretical reasons
Risk configurations: not clear if searching has sense. Possibly filtering on entities or models has more sense.
Same for risk sessions.
For models it could be implemented. But we foresee to have only a small numeber of models for now


Develop a Jupyter notebook to demonstrate a native (not DAPA) call to SDAP time series

Jupyter notebook demonstrating calls to SDAP analytics endpoints via the native SDAP API is checked in here: https://github.com/unity-sds/unity-analytics-sdap/blob/main/notebooks/unity-sdap-native.ipynb


x86 Server functional tests fail locally

The ServerComparison.FunctionalTests x86 tests fail if run directly because they look for the x86 runtime in the wrong place. It was recently moved from %userprofile%\.dotnet\x86 to %reporoot%\.dotnet\x86
Failed   ServerComparison.FunctionalTests.HelloWorldTests.HelloWorld(variant: Server: Kestrel, TFM: netcoreapp3.0, Type: Portable, Arch: x86)
Error Message:
 System.Exception : Command exited unexpectedly with exit code: -2147450730
Stack Trace:
   at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber)
   at Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer.StartSelfHostAsync(Uri hintUrl) in d:\github\AspNetCore\src\Hosting\Server.IntegrationTesting\src\Deployers\SelfHostDeployer.cs:line 190
   at Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer.DeployAsync() in d:\github\AspNetCore\src\Hosting\Server.IntegrationTesting\src\Deployers\SelfHostDeployer.cs:line 65
   at ServerComparison.FunctionalTests.HelloWorldTests.HelloWorld(TestVariant variant) in d:\github\AspNetCore\src\Servers\test\FunctionalTests\HelloWorldTest.cs:line 52
--- End of stack trace from previous location where exception was thrown ---
Standard Output Messages:
 | [0.005s] TestLifetime Information: Starting test HelloWorld-Server: Kestrel, TFM: netcoreapp3.0, Type: Portable, Arch: x86 at 2019-02-21T19:37:07
 | [0.003s] TestLifetime Information: Starting test HelloWorld_Kestrel_netcoreapp3.0_x86_Portable at 2019-02-21T19:37:07
 | [0.004s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: Deploying [Variation] :: ServerType=Kestrel, Runtime=CoreClr, Arch=x86, BaseUrlHint=, Publish=False
 | [0.005s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: Executing C:\Users\chrross\.dotnet\x86\dotnet.exe d:\github\AspNetCore\src\Servers\testassets\ServerComparison.TestSites\bin\Debug\netcoreapp3.0\ServerComparison.TestSites.dll --urls http://<IP_ADDRESS>:0/ --server Microsoft.AspNetCore.Server.Kestrel
 | [0.006s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: Removing environment variable ASPNETCORE_ENVIRONMENT
 | [0.006s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: SET ASPNETCORE_DETAILEDERRORS=true
 | [0.007s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: SET ASPNETCORE_CONTENTROOT=d:\github\AspNetCore\src\Servers\testassets\ServerComparison.TestSites
 | [0.048s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: Started C:\Users\chrross\.dotnet\x86\dotnet.exe. Process Id : 18680
 | [0.169s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr: It was not possible to find any compatible framework version
 | [0.170s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr: The specified framework 'Microsoft.NETCore.App', version '3.0.0-preview3-27414-8' was not found.
 | [0.170s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:   - Check application dependencies and target a framework version installed at:
 | [0.171s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       C:\Users\chrross\.dotnet\x86\
 | [0.171s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:   - The .NET Core framework and SDK can be installed from:
 | [0.171s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       https://aka.ms/dotnet-download
 | [0.171s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:   - The following versions are installed:
 | [0.172s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       2.0.0 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.172s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       2.0.9 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.172s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       2.1.2 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.172s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       2.1.6 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.173s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       2.1.7 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.173s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       2.2.0 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.173s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       2.2.1 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.174s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       3.0.0-preview-27217-02 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.174s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       3.0.0-preview-27219-3 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.174s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       3.0.0-preview-27319-3 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.175s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       3.0.0-preview-27324-5 at [C:\Users\chrross\.dotnet\x86\shared\Microsoft.NETCore.App]
 | [0.175s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:   - Installing .NET Core prerequisites might help resolve this problem:
 | [0.175s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: C:\Users\chrross\.dotnet\x86\dotnet.exe stderr:       https://go.microsoft.com/fwlink/?linkid=798306
 | [0.180s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: host process ID 18680 shut down
 | [0.180s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: Host process shutting down.
 | [0.181s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: Host process already exited or never started successfully.
 | [0.181s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: [Time]: Total time taken for this test variation '0.1767904' seconds
 | [0.181s] TestLifetime Information: Finished test HelloWorld_Kestrel_netcoreapp3.0_x86_Portable in 0.1785128s
 | [0.190s] TestLifetime Information: Finished test HelloWorld-Server: Kestrel, TFM: netcoreapp3.0, Type: Portable, Arch: x86 in 0.18489519999999998s

The discovery logic is outdated:
https://github.com/aspnet/AspNetCore/blob/4451a804477f7f3d4d0ed89d56c51332d8c3ee25/src/Hosting/Server.IntegrationTesting/src/Deployers/ApplicationDeployer.cs#L108-L114
https://github.com/aspnet/AspNetCore/blob/4451a804477f7f3d4d0ed89d56c51332d8c3ee25/src/Hosting/Server.IntegrationTesting/src/Common/DotNetCommands.cs#L14-L21
@natemcmaster does this work from a full build because DOTNET_HOME gets set?

does this work from a full build because DOTNET_HOME gets set?

yup. https://github.com/aspnet/AspNetCore/blob/4451a804477f7f3d4d0ed89d56c51332d8c3ee25/build.ps1#L242

We should look at these.


[Feature] Make ethereum.getBalance ENS aware

Description
Developers using ethereum.getBalance() might want to reference ENS (or other) in addition to addresses
let balance = ethereum.getBalance(
    Address.fromString("0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045")
);

or
let balance = ethereum.getBalance(
    Address.fromString("vitalik.eth")
);

Are you aware of any blockers that must be resolved before implementing this feature? If so, which? Link to any relevant GitHub issues.
No response
Some information to help us out

[ ] Tick this box if you plan on implementing this feature yourself.
[X] I have searched the issue tracker to make sure this issue is not a duplicate.

I would like to work on this change. Just for clarity, should this change the Address.fromString() method, or only in the ethereum.getBalance() function? In my opinion, I think we should go with the Address.fromString() method. Other functions like ethereum.call() or ethereum.hasCode() might also want to use ens resolvers, so that will make it easier. Also, a direct type conversion from string to H160 of "vitalik.eth" will lead to problems.


warn if customize_ functions are defined more than once

If you define two cusomize_ function with the same name, only the latter gets executed.

This isn't possible without changes to the shell.


fix(User,Circle): fix Xs size dimension

PR includes

[ ] Bug Fix
[ ] Docs
[ ] Specs
[ ] Refactor
[ ] Feature
[ ] DX
[ ] DevOps

Related issues

Resolve #
Related issues #, #
QA Instructions, Screenshots, Recordings
Please replace this line with instructions on how to test your changes, a note on the devices and browsers this has been tested on, as well as any relevant images for UI changes.

/canary


Support multi-line values in INI files to enable table parsing

This issue is a followup issue after #120.
Example roughness.ini input:
# 2021-12-01: Adapted from D-Flow FM UM, C.15, example 1.

[General]
    fileVersion           = 3.01
    fileType              = roughness

[Global]
    frictionId            = Main
    frictionType          = Chezy
    frictionValue         = 45.000

[Branch]
    branchId              = Channel0
    frictionType          = Manning
    functionType          = constant
    frictionValues        = 0.10000

[Branch]
    branchId              = Channel1
    frictionType          = Manning
    functionType          = constant
    numLocations          = 2                     # at two locations
    chainage              = 0.000 100.000
    frictionValues        = 0.20000 0.30000

[Branch]
    branchId              = Channel7
    frictionType          = Chezy
    functionType          = absDischarge
    numLevels             = 3                     # for three values
    levels                = 100.000 200.000 300.000
    numLocations          = 2                     # at two locations
    chainage              = 0.000 300.000
    frictionValues        = 45.00000 55.00000     # values for |Q|=100
                            41.00000 52.00000     # values for |Q|=200
                            40.00000 50.00000     # values for |Q|=300

The frictionValues currently parse correctly for:

Channel0 -> [0.1]
Channel1 -> [0.2, 0.4]
But not for:
Channel7 -> [45, 55]

This prohibits turning the FrictBranch.frictionvalues from the current List[float] into a List[List[float]] (or some other table-like data structure)
Expected result:

Channel7 -> [[45, 55], [41, 52], [40, 50]]

Request: can we support in IniBasedModel the parsing of multiline values?
Note: a combination of multiline values (as in this roughness file) and datablocks (as known from .bc files) is currently not necessary.

@arthurvd Can this issue be closed?
The discussion that was held about this topic came to the conclusion that HYDROLIB-core will not support multiline values.

Closing issue (seems like a stale issue); if a user asks for it we can reconsider.


Cannot read . and .. entries whithin a dir object

A lot of small mistakes caused this - fixed now


JS Error

Version 0.2.0 (First production version)
Immediatly crashes when launch pack-o-bot.exe.
Was able to launch pre-release but it didn't seem to record anything.
Anything else I should offer to help investigate the issue?

Thank you so much!  We're currently on that. I hope we'll have it fixed in a little while.

@liweichen6 Now that we're officially released, can you please try again? Should have been fixed in<PHONE_NUMBER>2c3bf259c5794c9db26d552cd3642a.

Now I can launch the app, but how do I know if it's recording?
It says "Watching for packs..." but that's it. I opened a pack, then checked pitytracker and nothing's there.
I also tried different language settings in HS, but that isn't the issue.

@liweichen6 Did you enter your token from pitytracker?

@mlntn Yes, entered and saved.

You downloaded the app before we had it bug-free. Could you delete the app again and delete your pack-o-bot user folder (either in Application Support on Mac or in AppData on Windows) and then reinstall the app?

@geraldhiller Is 0.2.1 (released 17 mins ago) the latest one?
I delete the app folder and /AppData/Roaming/pack-o-bot then download this one, but when I launch it, it says needs to be updated.

@liweichen6 Yes it is. The update check is fixed now.

This is pretty weird.
Shows "Watching for packs..." and that's it.
Even tried to regenerate a new token and still the same.

{"region":"xx","unsentPacks":{}}
This is from packs.json under AppData. Does that mean the bot didn't get any data (not even the region)?

Yes. It looks like something went wrong with the setup. Did you close HS before opening Pack-o-Bot? If logging isn't enabled in HS already, POB will enable it in its setup routine, that's why HS needs to be closed, or at least restartet after starting POB. There is no way around it.

@geraldhiller Yes I tried several times, even restarted my PC.
And anyway to make sure logging is enabled? Like "Logs" folder under Hearthstone?

@liweichen6 Yes, can you please paste the contents of %USERPROFILE%\AppData\Local\Blizzard\Hearthstone\log.config

`[Achievements]
LogLevel=1
FilePrinting=true
ConsolePrinting=false
ScreenPrinting=false
Verbose=true
[Arena]
LogLevel=1
FilePrinting=true
ConsolePrinting=false
ScreenPrinting=false
Verbose=true
[FullScreenFX]
LogLevel=1
FilePrinting=true
ConsolePrinting=false
ScreenPrinting=false
Verbose=true
[LoadingScreen]
LogLevel=1
FilePrinting=true
ConsolePrinting=false
ScreenPrinting=false
Verbose=true
[Power]
LogLevel=1
FilePrinting=true
ConsolePrinting=false
ScreenPrinting=false
Verbose=true
[Rachelle]
LogLevel=1
FilePrinting=true
ConsolePrinting=false
ScreenPrinting=false
Verbose=true
[Zone]
LogLevel=1
FilePrinting=true
ConsolePrinting=false
ScreenPrinting=false
Verbose=true
[BattleNet]
LogLevel=1
FilePrinting=true
`

Hmm, I'm not sure what Verbose Achievement logging looks like. Can you post your Achievements.log? It's in C:\Program Files (x86)\Hearthstone\Logs\

Achievements.log
D 20:03:14.5852494 Validating achieve: 67
D 20:03:14.5877560 NetCache.OnProfileNotices(): sending notices to AdventureProgressMgr::OnNewNotices
D 20:03:14.5882572 NetCache.OnProfileNotices(): sending notices to AchieveManager::OnNewNotices
D 20:03:14.5882572 NetCache.OnProfileNotices(): sending notices to GenericRewardChestNoticeManager::OnNewNotices
D 20:03:14.5892600 NetCache.OnProfileNotices(): sending notices to AccountLicenseMgr::OnNewNotices
D 20:03:14.5892600 NetCache.OnProfileNotices(): sending notices to FixedRewardsMgr::OnNewNotices
D 20:03:14.5917666 NetCache.OnProfileNotices(): sending notices to PopupDisplayManager::OnNewNotices
D 20:03:14.5917666 NetCache.OnProfileNotices(): sending notices to DialogManager::OnNewNotices
D 20:03:14.5922679 NetCache.OnProfileNotices(): sending notices to StoreManager::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to GeneralStorePacksPane::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to AdventureProgressMgr::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to AchieveManager::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to GenericRewardChestNoticeManager::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to AccountLicenseMgr::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to FixedRewardsMgr::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to PopupDisplayManager::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to DialogManager::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to StoreManager::OnNewNotices
D 20:03:14.5927694 NetCache.OnProfileNotices(): sending notices to GeneralStorePacksPane::OnNewNotices
D 20:03:16.0706979 Validating achieve: 60
D 20:03:21.7748612 PopupDisplayManager: adding 0 rewards to load total=0
D 20:03:21.7748612 PopupDisplayManager: Calling AllAchievesShownListeners callbacks
D 20:03:42.7962409 PopupDisplayManager: adding 0 rewards to load total=0
D 20:03:46.7803316 NotifyOfCardGained: [name=Drain Soul cardId=ICC_055 type=SPELL] NORMAL 20
D 20:03:46.7808329 NotifyOfCardGained: [name=Breath of Sindragosa cardId=ICC_836 type=SPELL] NORMAL 18
D 20:03:46.7808329 NotifyOfCardGained: [name=Eternal Servitude cardId=ICC_213 type=SPELL] NORMAL 6
D 20:03:46.7808329 NotifyOfCardGained: [name=Acolyte of Agony cardId=ICC_212 type=MINION] NORMAL 9
D 20:03:46.7808329 NotifyOfCardGained: [name=Howlfiend cardId=ICC_218 type=MINION] NORMAL 12
D 20:03:53.5984555 NotifyOfCardGained: [name=Spirit Lash cardId=ICC_802 type=SPELL] NORMAL 24
D 20:03:53.5984555 NotifyOfCardGained: [name=Roll the Bones cardId=ICC_201 type=SPELL] NORMAL 10
D 20:03:53.5984555 NotifyOfCardGained: [name=Shadowblade cardId=ICC_850 type=WEAPON] GOLDEN 2
D 20:03:53.5984555 NotifyOfCardGained: [name=Drain Soul cardId=ICC_055 type=SPELL] NORMAL 21
D 20:03:53.5984555 NotifyOfCardGained: [name=Sanguine Reveler cardId=ICC_903 type=MINION] NORMAL 23
D 20:04:01.6498586 PopupDisplayManager: adding 0 rewards to load total=0

And my HS is not located in default location. It's under D://
Does it matter?

Yep, that's the problem. I'll see if we can come up with a solution for that.

@mlntn Move HS to the default location. Now everything works.
Can't believe it's that simple lol.

Awesome, thanks for all your help debugging this issue. We're working on making this tool more robust and your early feedback helps us tremendously.

Just found a new bug related to HS language setting.
zh-CN: only the 1st pack is recorded.
en-US: no issues.

Hmm, can you send us the email of your pitytracker account via the feedback button on pitytracker?

@Ekkstein Just sent.

It appears that you only have packs from the us server in our database.
Please login to the account (region) from where you have issues posting your packs and post the first line of C:\Program Files (x86)\Hearthstone\LogsBattleNet.log here.

@Ekkstein I only have an US account, but sometimes I play in zh-CN. It's the HS language setting, not the region.

@liweichen6 Can you paste a copy of Achievements.log with a pack opened while in zh-CN?

D 22:02:30.3199889 Validating achieve: 67
D 22:02:30.3224955 NetCache.OnProfileNotices(): sending notices to AdventureProgressMgr::OnNewNotices
D 22:02:30.3224955 NetCache.OnProfileNotices(): sending notices to AchieveManager::OnNewNotices
D 22:02:30.3224955 NetCache.OnProfileNotices(): sending notices to GenericRewardChestNoticeManager::OnNewNotices
D 22:02:30.3239995 NetCache.OnProfileNotices(): sending notices to AccountLicenseMgr::OnNewNotices
D 22:02:30.3239995 NetCache.OnProfileNotices(): sending notices to FixedRewardsMgr::OnNewNotices
D 22:02:30.3265061 NetCache.OnProfileNotices(): sending notices to PopupDisplayManager::OnNewNotices
D 22:02:30.3265061 NetCache.OnProfileNotices(): sending notices to DialogManager::OnNewNotices
D 22:02:30.3270077 NetCache.OnProfileNotices(): sending notices to StoreManager::OnNewNotices
D 22:02:30.3275099 NetCache.OnProfileNotices(): sending notices to GeneralStorePacksPane::OnNewNotices
D 22:02:30.3275099 NetCache.OnProfileNotices(): sending notices to AdventureProgressMgr::OnNewNotices
D 22:02:30.3275099 NetCache.OnProfileNotices(): sending notices to AchieveManager::OnNewNotices
D 22:02:30.3275099 NetCache.OnProfileNotices(): sending notices to GenericRewardChestNoticeManager::OnNewNotices
D 22:02:30.3280100 NetCache.OnProfileNotices(): sending notices to AccountLicenseMgr::OnNewNotices
D 22:02:30.3280100 NetCache.OnProfileNotices(): sending notices to FixedRewardsMgr::OnNewNotices
D 22:02:30.3280100 NetCache.OnProfileNotices(): sending notices to PopupDisplayManager::OnNewNotices
D 22:02:30.3280100 NetCache.OnProfileNotices(): sending notices to DialogManager::OnNewNotices
D 22:02:30.3280100 NetCache.OnProfileNotices(): sending notices to StoreManager::OnNewNotices
D 22:02:30.3280100 NetCache.OnProfileNotices(): sending notices to GeneralStorePacksPane::OnNewNotices
D 22:02:31.1486917 Validating achieve: 60
D 22:02:36.7280231 PopupDisplayManager: adding 0 rewards to load total=0
D 22:02:36.7285241 PopupDisplayManager: Calling AllAchievesShownListeners callbacks
D 22:02:57.5624089 PopupDisplayManager: adding 0 rewards to load total=0
D 22:03:00.0535308 NotifyOfCardGained: [name=黑暗裁决 cardId=ICC_039 type=SPELL] NORMAL 16
D 22:03:00.0540321 NotifyOfCardGained: [name=装死 cardId=ICC_052 type=SPELL] NORMAL 21
D 22:03:00.0540321 NotifyOfCardGained: [name=咆哮魔 cardId=ICC_218 type=MINION] NORMAL 15
D 22:03:00.0540321 NotifyOfCardGained: [name=正义保护者 cardId=ICC_038 type=MINION] NORMAL 24
D 22:03:00.0540321 NotifyOfCardGained: [name=命运骨骰 cardId=ICC_201 type=SPELL] NORMAL 11
D 22:03:10.6055806 PopupDisplayManager: adding 0 rewards to load total=0

Bot shows "found a pack...", "uploading...", then "watching for packs..."
Checked PityTracker but this pack isn't on there.
Does the bot read CardName or CardID?

That pack uploaded for me. Can you confirm that it's not showing out of order in the table of packs here? https://pitytracker.com/packs

I think I find out why.
I assumed the first shown on PT was the last I uploaded.
But it actually shows between No.20 and No.21.
Bot did upload that one, but I didn't see so I manually add that again.

@Ekkstein Do you want to take a look at that on PT?

@liweichen6 Thanks again. I'm going to close this ticket. Feel free to open another if you run into another bug.


Compute libssl path from LD_LIBRARY_PATH

Problem
We use the almalinux:macro docker image, which does not contain libssl.so.3 by default. So we use the environment variable LD_LIBRARY_PATH=/opt/lib to specify the dynamic library path. Other programs work well, but prisma seems to only look for libssl from a few fixed paths, such as /lib64, /usr/lib64, /lib.
https://github.com/prisma/prisma/issues/20905#issuecomment-2476370127
Suggested solution
If the LD_LIBRARY_PATH environment variable exists in the system, node_modules/prisma/libquery_engine-rhel-openssl-3.0.x.so.node will automatically search for libssl from the specified path, so we only need to modify the get-platform package to prioritize searching from LD_LIBRARY_PATH. A simple way to change this is to modify the getos method:
https://github.com/prisma/prisma/blob/5.22.0/packages/get-platform/src/getPlatform.ts#L73
  const libsslSpecificPaths = computeLibSSLSpecificPaths({ arch, archFromUname, familyDistro: distroInfo.familyDistro })
  if (typeof process.env.LD_LIBRARY_PATH === "string") {
    libsslSpecificPaths.unshift(...process.env.LD_LIBRARY_PATH.split(':'))
  }
  const { libssl } = await getSSLVersion(libsslSpecificPaths)

Alternatives

Provide detailed debug information like ldconfig and openssl version -v, and may need to refactor getSSLVersion
Provide an environment variable such as PRISMA_LIB_PATH to allow users to specify the path of the dynamic library

Additional context
I tried using linux-static-x64 but prisma seems to automatically add native binaryTargets for me and download rhel-openssl-3.0.x, I'm not sure if this is expected.
When I use linux-static-x64 and manually specify PRISMA_QUERY_ENGINE_BINARY=node_modules/.prisma/client/query-engine-linux-static-x64, it works. But prisma still outputs the warning message, which is confusing.
generator client {
  provider      = "prisma-client-js"
  binaryTargets = ["linux-static-x64"]
  engineType    = "binary"
}

bash-5.1$ ls node_modules/.prisma/client/
default.d.ts  edge.d.ts         index.d.ts                                  package.json                     schema.prisma
default.js    edge.js           index.js                                    query-engine-linux-static-x64    wasm.d.ts
deno          index-browser.js  libquery_engine-rhel-openssl-3.0.x.so.node  query-engine-rhel-openssl-3.0.x  wasm.js

The following is the debug information of get-platform
prisma:get-platform Found distro info:
{
  "targetDistro": "rhel",
  "familyDistro": "rhel",
  "originalDistro": "almalinux"
} +173ms
prisma:get-platform Trying platform-specific paths for "rhel" +0ms
prisma:get-platform Falling back to "ldconfig" and other generic paths +1ms
prisma:get-platform Couldn't find any version of libssl or OpenSSL in the system +9ms
prisma:warn Prisma failed to detect the libssl/openssl version to use, and may not work as expected. Defaulting to "openssl-1.1.x".

Actually I'm gonna mark it as a bug


Aggregate requests example should fetch bodies in parallel

Currently, the aggregate requests example requests each body in series:
const btc = await btcResp.json()
const eth = await ethResp.json()
const ltc = await ltcResp.json()

This should be done all at once
Proposal 1
This would be a pretty minimal change:
const [btc, eth, ltc] = await Promise.all([btcResp, ethResp, ltcResp].map(res => res.json()))

Proposal 2
I kinda just went all out
// Make multiple requests, aggregate the responses and
// send it back as a single response.

const currencies = ['BTC', 'ETH', 'LTC']

addEventListener('fetch', event => {
    event.respondWith(fetchAndLog(event.request))
})

async function fetchAndLog(request) {
    const [btc, eth, ltc] = await Promise.all(
      currencies
        .map(currency => fetch(`https://api.coinbase.com/v2/prices/${currency}-USD/spot`))
        .map(p => p.then(res => res.json()))
        .map(p => p.then(({ data: { amount } }) => amount))
    )

    const body = { btc, eth, ltc }

    return new Response(JSON.stringify(body, null, 2), {
      status: 200,
      headers: {'Content-Type': 'application/json'}
    })
}

Hello @jrf0110! This is very due for a response. Thanks for the recommendation to simplify the code. Closing this issue because this code has been updated.


blas1_rotmg_test and blas1_rotmg_test fail

Hello everyone,
I was trying your library but the tests in the title fail.
Steps to reproduce:
git clone --recursive https://github.com/codeplaysoftware/portBLAS.git
cd portBLAS
export CC=icx
export CXX=icpx
cmake -S . -B build -DSYCL_COMPILER=dpcpp
cd build
make all
make test

blas1_rotg_test output:
Device vendor: Intel(R) Corporation
Device name: Intel(R) UHD Graphics 620
Device type: gpu
...
[ RUN      ] Rotg/RotgFloat.test/alloc_usm__api_async__a_340282346638528859811704183484516925440__b_minus_340282346638528859811704183484516925440
/tmp/portBLAS/test/unittest/blas1/blas1_rotg_test.cpp:93: Failure
Value of: utils::almost_equal(a, a_ref)
  Actual: false
Expected: true
[  FAILED  ] Rotg/RotgFloat.test/alloc_usm__api_async__a_340282346638528859811704183484516925440__b_minus_340282346638528859811704183484516925440, where GetParam() = ("usm", 4-byte object <00-00 00-00>, 3.40282e+38, -3.40282e+38) (0 ms)
...
[ RUN      ] Rotg/RotgFloat.test/alloc_usm__api_sync__a_340282346638528859811704183484516925440__b_minus_340282346638528859811704183484516925440
/tmp/portBLAS/test/unittest/blas1/blas1_rotg_test.cpp:93: Failure
Value of: utils::almost_equal(a, a_ref)
  Actual: false
Expected: true
[  FAILED  ] Rotg/RotgFloat.test/alloc_usm__api_sync__a_340282346638528859811704183484516925440__b_minus_340282346638528859811704183484516925440, where GetParam() = ("usm", 4-byte object <01-00 00-00>, 3.40282e+38, -3.40282e+38) (0 ms)
...
[ RUN      ] Rotg/RotgFloat.test/alloc_buf__api_async__a_340282346638528859811704183484516925440__b_minus_340282346638528859811704183484516925440
/tmp/portBLAS/test/unittest/blas1/blas1_rotg_test.cpp:93: Failure
Value of: utils::almost_equal(a, a_ref)
  Actual: false
Expected: true
[  FAILED  ] Rotg/RotgFloat.test/alloc_buf__api_async__a_340282346638528859811704183484516925440__b_minus_340282346638528859811704183484516925440, where GetParam() = ("buf", 4-byte object <00-00 00-00>, 3.40282e+38, -3.40282e+38) (1 ms)
...
[ RUN      ] Rotg/RotgFloat.test/alloc_buf__api_sync__a_340282346638528859811704183484516925440__b_minus_340282346638528859811704183484516925440
/tmp/portBLAS/test/unittest/blas1/blas1_rotg_test.cpp:93: Failure
Value of: utils::almost_equal(a, a_ref)
  Actual: false
Expected: true
[  FAILED  ] Rotg/RotgFloat.test/alloc_buf__api_sync__a_340282346638528859811704183484516925440__b_minus_340282346638528859811704183484516925440, where GetParam() = ("buf", 4-byte object <01-00 00-00>, 3.40282e+38, -3.40282e+38) (1 ms)

blas1_rotmg_test output:
Device vendor: Intel(R) Corporation
Device name: Intel(R) UHD Graphics 620
Device type: gpu
...
[ RUN      ] Rotmg_Usm/Rotmg_UsmFloat.test/alloc_usm__d1_2p9__d2_27431224__x1_1p50__y1_0p0__will_overflow_0
/tmp/portBLAS/test/unittest/blas1/blas1_rotmg_test.cpp:134: Failure
Value of: isAlmostEqual
  Actual: false
Expected: true
[  FAILED  ] Rotmg_Usm/Rotmg_UsmFloat.test/alloc_usm__d1_2p9__d2_27431224__x1_1p50__y1_0p0__will_overflow_0, where GetParam() = ("usm", 2.1, 2.74312e+07, 1.5, 5.72622e-08, false) (0 ms)
...
[ RUN      ] Rotmg_Buffer/Rotmg_BufferFloat.test/alloc_buf__d1_2p9__d2_27095732__x1_1p50__y1_0p0__will_overflow_0
/tmp/portBLAS/test/unittest/blas1/blas1_rotmg_test.cpp:134: Failure
Value of: isAlmostEqual
  Actual: false
Expected: true
[  FAILED  ] Rotmg_Buffer/Rotmg_BufferFloat.test/alloc_buf__d1_2p9__d2_27095732__x1_1p50__y1_0p0__will_overflow_0, where GetParam() = ("buf", 2.1, 2.70957e+07, 1.5, 5.46859e-08, false) (1 ms)

Ubuntu version: 22.04.1
Cmake version: 3.27.4
icpx version: Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2<IP_ADDRESS>31213)
CPU: Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz

Hello @fbarbari
Thank you for your interest in portBLAS. Could you please share the name of the reference BLAS implementation you are using for these tests?
There is a known issue with rotg and rotmg where some of the tests fail because of the usage of fast-math compiler directive documented here: https://github.com/codeplaysoftware/portBLAS/blob/fec888faae176bb2f86f3eaa9a4cd7739606052a/test/unittest/CMakeLists.txt#L106
and we had to explicitly specify not to use fast-math for these tests to make sure that these tests would pass. We will try to investigate these further and get back to you soon. Thanks.

Could you please share the name of the reference BLAS implementation you are using for these tests?

Cmake tells me this:
...
-- Found SystemBLAS: BLAS_LIBRARIES
...

Which I don't think is that much informative. I have installed OpenBLAS 0.3.24 on this system.

Hello @fbarbari,
Looking at this issue I remember we had issues with these rotg and rotmg as we have found that they have many edge cases that are not well defined and libraries can give different output. In particular we have discussed this issue before with rotmg and OpenBLAS in https://github.com/codeplaysoftware/portBLAS/pull/376. In the end we decided to make our implementations match with cblas and cuBLAS. I have added a bit of documentation for this issue in https://github.com/codeplaysoftware/portBLAS/pull/506
I suggest using cblas if you want all tests to be green.

I'm sorry I was mistaken somewhere. Locally OpenBLAS is giving correct results for these tests although I am using OpenBLAS 0.3.20 and my integrated GPU is UHD Graphics 770. I see there has been some changes regarding rotg in the OpenBLAS release notes. We will revisit how we want to approach this issue.

Hi @fbarbari,
We looked into this issue and on our side everything works fine with openBLAS 0.3.26.
We were able to reproduce test failure when the configuration uses a different BLAS library provided by oneAPI toolkit. To fix it and give more clarity there is PR #509 open.
If you don't want to wait for it to be merged I suggest you to reconfigure and compile portBLAS adding two flags to specify openBLAS path:
-DOPENBLAS_LIBRARIES=/path/to/openblas/lib and  -DOPENBLAS_INCLUDE_DIRS=/path/to/openBLAS/include.

Hello @fbarbari,
I think this issue is solved so I am going to close it. If you feel your problem is not resolved, please reopen it or open another issue for us. Thank you!


Manually Adding a Node to an Existing Cluster

Documentation issue at https://gardener.cloud/blog/2019/06.11-manually-adding-a-node-to-an-existing-cluster/
The tutorial promised in this article is nowhere to be found.

Thank you for the heads-up. This blog post used to refer to an outdated guide which has since been removed.
I will be removing the guide as it provides no benefit.


Fix EC_POINT_new doc

Should be EC_POINT_set_affine_coordinates() sets x and y

Checklist

[x] documentation is added or updated

No, this is a misunderstanding. This sentence really documents the EC_POINT_get_affine_coordinates. The x and y are the output BIGNUM parameters of the function.


2.0 Compatibility?

Since 2.0 is out this mod isn't compatible any more.

Haven't bought 2.0 yet - It is more a time problem than  money/commitments.
I'll try to make some room this weekend.


[BACKPORT] [TC-491] [TC-492] - Fix SQL error in downgrade scripts

(cherry picked from commit 4a5962ed53720c6ff2286979a71161e782d725e7)

@hbeatty renamed my branch and recreated the PR

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR-trafficops-test/85/

@rawlinp Thank you for renaming your branch. This has been merged. Please close.


Detect SSO for When Org has SSO Set Up

We have SSO set up through Okta, but when a developer finds the hive login page, they are not prompted to log in with SSO. Two related features would be great:

Add a "login with Okta" (or other SSO provider) button on the home page when the org has it set up.
Detect if an email address should be associated with SSO or not. For example, it is not likely that<EMAIL_ADDRESS>will need a distinct Google login from their SSO login. If<EMAIL_ADDRESS>tries to log in with any other method, redirect them to the SSO page the org has configured.

Adding on / clarifying after setting up SSO for my company too. A few things security wanted to look for in this.
Would love to see:

Require SSO (I think essentially what Cassidy is describing in 2 above)
Better support for moving to SSO. I had signed in and created our org with Google SSO. Then after I added OKTA OIDC support, when I signed in, it made a totally new account for me despite the emails being the same. This was surprising as I'd expect email to be the primary key and it should have just logged me in. (see https://supertokens.com/docs/thirdparty/common-customizations/deduplication/implementing-deduplication)

Related https://github.com/kamilkisiela/graphql-hive/issues/4367


PowerShell and RegExp to convert code

PowerShell and RegExp to convert code

Name: Cláudio Silva

Your Timezone: UTC +1

Abstract: I had a project where I needed to convert more than 100K lines of code. Can you imagine the pain of doing it all by hand? I have created an PowerShell function that uses regular expression in order to do the find/replace and used it on +1000 individual files. I did in less than 2 weeks what I initial expected to be several months.
Note: I have used it to convert from Firebird SQL to T-SQL flavor but you can use it to other goals.

Optional links

Twitter: @claudioessilva
Slack ID: @claudioessilva on powershell.slack.com
Slack ID: @claudioessilva on sqlcommunity.slack.com
Blog URL: https://claudioessilva.eu

Thanks again!  Do feel free to submit more proposals : D


Load from ipynb and start sync

This enables loading directly from opened ipynb file from the browser to nvim and start syncing.
Adds 2 commands
:JupyniumLoadFromIpynbTab tab_index
:JupyniumLoadFromIpynbTabAndStartSync tab_index

Also this fixes system python startup issue with LegacyVersion from packaging.

Thank you for adding this idea so fast!

Thank you for adding this idea so fast!

No problem! I don't know how I missed this. Thanks for the feedback!


Fix ESLint

Fixes https://github.com/kungfooman/RuntimeTypeInspector.js/issues/15

CI is still failing, but that's because of a unit test that requires additional care for the Stringifier Comment node handling (we have another issue tracking that)


Add extra window functions

I have written some additional window functions that allow greater flexibility while choosing the FFT window to use:

Welch window
Cosine window
Nuttall Window
Blackman-Nuttall Window
Blackman-Harris Window

The coefficients used are listed at Wikipedia
Thanks for the great software!

I'll add tests for these windows shortly


Remove patch from Github Workflow

@yaronzz this removes the patch which is no longer necessary.

@yaronzz I also opened a PR where I removed the patch from AIGS:
https://github.com/AIGMix/AIGS/pull/5


A few improvements to the tools/modules/update.sh script: [phab:D1171]

Created by pcc at 2016-12-03 21:16:32:

Don't clone clang-tools-extra, kythe doesn't use it unless I'm mistaken
Use the github mirrors instead of llvm.org, as they are generally faster
Use ninja if available to build LLVM

zarko commented inline (context lost during migration) at 2016-12-05 20:37:45:
I'm dubious about the assertion that "anyone who cares about build speed should be using ninja anyway". I also don't have ninja installed on my various machines and am hesistant to add another (albeit optional) dependency. Is there a reason why make should be pessimized in this way?

pcc commented inline (context lost during migration) at 2016-12-05 20:55:58:
That seems like a fair position. Would you be happy with ninja as an optional dependency as long as it doesn't regress make?

zarko commented inline (context lost during migration) at 2016-12-05 21:02:07:
Yes, that's fine (and the rest of your changes LGTM and are appreciated).

pcc wrote at 2016-12-05 21:57:01:

Restore the logic for guessing number of cores
Document the optional ninja dependency

zarko commented inline (context lost during migration) at 2016-12-05 22:57:21:
build.*.*.* should probably still be an option; otherwise once we need to do a version bump, people will be left with old build directories.

zarko wrote at 2016-12-05 22:57:21:
Tested this in existing and no-prebuild configurations on Linux and OS X (and no-prebuild with Ninja on Linux).
This is causing a build failure on OS X:
cat .../kythe/cxx/extractor/testdata/metadata/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
sed: 1: "/-target/,+1d": expected context address

pcc commented inline (context lost during migration) at 2016-12-05 23:04:02:
But build.*.* matches anything that build.*.*.* would match, right?

pcc wrote at 2016-12-05 23:04:02:

! In (earlier comment), zarko wrote:
Tested this in existing and no-prebuild configurations on Linux and OS X (and no-prebuild with Ninja on Linux).

This is causing a build failure on OS X:
cat .../kythe/cxx/extractor/testdata/metadata/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
sed: 1: "/-target/,+1d": expected context address

I'll see if I can reproduce this on OS X. But are you sure that my change was the culprit here? I don't see what that test has to do with my change.

zarko wrote at 2016-12-05 23:06:18:
#1536 fixed the sed issue; your diff was cut ~2 minutes before it was landed.

zarko commented inline (context lost during migration) at 2016-12-05 23:08:12:
As it turns out, this is not some magic * that only matches a field, but the normal sort of find *. Never mind.

zarko closed via commit fe8961e78df4fe3b530069394cd97fee42bb8493 at 2016-12-06 00:02:11


FixedWindowRoller is unreliable under Windows

Windows allows troublesome filenames which are not tolerated in Rust Strings.  Some information is available here.  Rust has OsStr and OsString to help deal with the problem.
FixedWindowRollerBuilder::build takes a &str for the filename pattern.  That makes it impossible to place the log archives in a directory that has a troublesome name.

@Coding-Badly that sounds like it could be an issue, would you be interested in working on a fix?

I am.  The fix requires an interface change to FixedWindowRollerBuilder::build.
That change allows another as-yet-unreported bug to be fixed.
Still interested?

@gadunga Would love your input here

@Coding-Badly any other solution you can think of that wouldn't break the current API?

There really are two things at play.  The directory and the pattern.  While similar (they're both a sequence of characters) they do have different purposes, different constraints, and different behaviours.  I honestly believe they should be separated.  (And, there really is not a way around the Windows path trouble.)
However, nothing stops us from adding a new method and leaving the existing method intact.  The existing method could be marked deprecated with a warning for Windows users.  The change would not be breaking and the audience would be reasonably informed about the situation.
I suspect having the two separated would also make the roller just a tiny bit easier to use.

I like that idea, I would prefer to not deprecate the current method as I think that will be the 99% use case, but certainly, we should mention the limitation in the docs. I think a change such as this would be appropriate for log4rs. I think config file support is out of scope for now.

The R&D repository for this issue.


251 version information

https://github.com/uwlib-cams/MARC2RDA/blob/main/Working Documents/2XX.csv

https://docs.google.com/spreadsheets/d/1Sk_WqUWOTNr3mvadehS7QTOEOUjglH9S4jhXkz9pxmQ/edit#gid=687405068
Hello Everyone,
Please see first pass of the spreadsheet for field code 251 - Version Information (R). Would anyone be able to review this spreadsheet for me.
Thanks,
Alice


Fine-tuning for multiclass data

Hi, I am working with a dataset that has around 60 classes labeled from 0 - 59. I created a new DataProcessor for the task first which returns those labels on get_labels() function.
However, after following the  pre-training script in the READme.md i get a keyerror :
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/mr/miniconda3/envs/bert/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/mr/Workspace/Team8993/THESIS/DNABERT/examples/transformers/data/processors/glue.py", line 120, in glue_convert_examples_to_features
    label = label_map[example.label]
KeyError: '35'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "run_finetune.py", line 1283, in <module>
    main()
  File "run_finetune.py", line 1097, in main
    train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
  File "run_finetune.py", line 760, in load_and_cache_examples
    features.extend(result.get())
  File "/home/mr/miniconda3/envs/bert/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
KeyError: '35'

Can you please tell me where this might be originating from?

Hi,
I think you need to modify the src/transformers/data/processors/glue.py. First, define your data processor as the DnaPromProcessor, and update the three dictionaries at line 596-644 (you need to define a name for your own task). When running the code, specify the task name with --task_name YOURTASK.

Hey, thanks for your help. I have successfully ran fine-tuning, prediction and visualization script on a demo dataset. However, I have a little confusion about the dataset. During fine-tuning I don't see it using any validation set. Again, during visualization the script requires a dev.tsv.  could you kindly tell me

which scripts are actually utilizing the test.tsv and dev.tsv ?
Should I divide my dataset into non-overlapping 3 sections?
Which .tsv is being used for validation?
Thanks and regards.

@appledora

Fine-tune
The run_finetune.py script will fine-tune the DNABERT model using train.tsv .  Once fine tuning is complete, the script will test the performance of the fine-tuned model using dev.tsv (provide the argument --do_eval).  Additionally, run_finetune.py can evaluate performance on dev.tsv periodically during fine-tuning (provide the arguments --evaluate_during_training and
--logging_steps 1000).

Prediction
Once fine-tuning is complete, if you run the run_finetune.py script, it will generate predictions with dev.tsv and save them to pred.npy.

I divide my data into a train, a dev, and a test set.  However, this means I have to rename my dev set file as dev_true.tsv and my test set file as dev.tsv between steps 3 and 4!  I can see how one might become confused.
I hope this helps.

yeah...i eventually had to do the renaming step as i went along running the repo. Thanks for your detailed explanations, either way!

@appledora  or @Zhihan1996 can you please tell me how to finetune the unlabeled kmer data with pretrained models

Hi,
I think you need to modify the src/transformers/data/processors/glue.py. First, define your data processor as the DnaPromProcessor, and update the three dictionaries at line 596-644 (you need to define a name for your own task). When running the code, specify the task name with --task_name YOURTASK.

You'll also need to add your task name to your metrics  in /src/transformers/data/metrics/init.py or you're going to error during validation :)


Remove vcf_header option from write_vcf

vcztools view needs to produce the appropriate header in all situations, and having an input header as an option is not really very helpful. We need to subset the samples and fields in all sorts of ways, and parsing the source header here is not the right approach. Header parsing should be done in bio2zarr.
Note that do this fully, we will need to update the VCF Zarr specification and vcf2zarr in order to store sufficient information about the header, rather than the header itself. This will require some coordinated updates across repos.
I think the simplest approach here is to just delete the option for proving a complete header here, and see what we're missing.
See:

https://github.com/sgkit-dev/vcf-zarr-spec/issues/15

See the linked issues in https://github.com/sgkit-dev/vcf-zarr-spec/issues/15#issuecomment-2270942624 for notes on how we might avoid storing the full header, and retain lossless encoding of the information in it.


"Choose another page" button on group permissions page is cut off

Increasing the font size of buttons (#2985) has resulted in the text of the "Choose another page" button under Settings -> Groups -> Page permissions being cut off:

Oops, looks like this is specific to #3035 because of the extra column. Will fix it there...

BTW: if the design is dependent on an exact string length and font size, it will be almost impossible to translate well. When translating the length of a string could easily double (or halve) because of different word lengths and grammar in the other language. The layout should tolerate at least ± 50% string (ideally more) length change without breaking or overflowing for it to be easy to translate.
An added bonus is that if the design tolerates this, adding an extra column won't suddenly completely break the layout as well.