[test] Help testing the upcoming release!

Hi all.
I've refactored the code a bit and since the current release is rather stable I though I'd release a pre-production release via the "Issues" with the hope that some brave people are willing to test it. All feedback is welcome.
I've fixed some issues:

TumblThree now includes a working detection between hidden and offline blogs that doesn't require queuing the blog.
Fixes a bug that released the video connection semaphore to often. That means the slider in the settings for limiting the video downloads didn't work at all. Now it should properly limit the connections to the vt.tumblr.com host and prevent incomplete video downloads.

New features are:

There is now an option for regular and hidden blog downloads to dump all the data gathered by the crawler to the disk. Right now for each post a separate file is created. I wasn't sure if I should write it all in one single file per blog (could get large rather quickly) or split it. It completely obsoletes the current metadata download options since it contains more information. The SVC branch (v1.0.7.x) contains way more crawler data compared to the master (v1.0.8.x) branch.
You can now specify in the Details panel for each blog where its files should be downloaded. If the Textbox control is empty, the files are downloaded as in previous releases in the global download location specified in the settings plus the blog name.
I though of this feature as some kind of trade off for the feature suggestion to avoid downloading duplicates across different blogs (see #151).
Now you can download everything into a single folder, but the blog databases/indexes stay separated. I'm not sure if that's a good approach. There is still a back-mapping missing if you want to "restore" the files for each blog. Since the .tumblr databases are text files, one could script that. This feature will certainly break the metadata files if you download multiple blogs at once as each downloader will compete for accessing the files, but most likely not the dumping of the crawler data in the bullet point above.
The global database probably requires a lot of testing. I don't have any experience in how a disk based approach like sqlite will behave and perform if multiple concurrent downloads want to access it, nor how it will handle several million entries and/or how the large the memory usage will be. And a server-client based solution is certainly overkill.
I'm not sure about this at all, thus I might remove it again if it has no use. This is completely untested and how it should work in theory. I cannot easily test it due to lack of data/time.

Release A:

Fixed some bugs introduced during the refactoring.
Added an option to load all databases into memory and compare each to-downloaded binary file to all databases across TumblThree before downloading. If it's already in any database, the file is skipped and will not be counted as downloaded. I'm not sure if that's the right approach, since it seems that TumblThree will miss downloads in the user interface, but if we count them, then ppl will complain that files are missing. If we reduce the overall number, then TumblThree will not grab everything ..
You can enable it in the settings->global. You'll have to restart TumblThree afterwards.

Release B:

Fixed more bugs introduced during the refactoring. Should be working now even with the "load all databases into memory"-mode.

Release C:

Improved the blog detection during blog addition. Should reduce latency and offline blogs aren't added anymore at all.

Thanks!
TumblThree-v<IP_ADDRESS>C-Application.zip
TumblThree-v<IP_ADDRESS>C-Translations.zip
TumblThree-v<IP_ADDRESS>C-Application.zip
TumblThree-v<IP_ADDRESS>C-Translations.zip
TumblThree-v<IP_ADDRESS>C-Application.zip
TumblThree-v<IP_ADDRESS>C-Translations.zip

Hello,
Thank you very much for considering my request and implementing this.
I see several positives and several drawbacks with this implementation.
Pros:

Now it is possible to merge several thematic blogs into a single download folder, i.e. comics blogs, anime blogs, p*rn blogs, etc. This is very useful when less thematic directories is better than hundreds of directories per blog.

Cons:

Even blogs with different theme may still reblog same content.
Having folder with millions of images makes WinExplorer choke and it takes up to HOURS to index them. Photo management apps may take long time to index, thumbnail them. Makes these folders impossible to browse within acceptable timeframe,

IMHO the best solution would be to load all indexes in memory, and apply them in real time against scanned blog contents. Not sure how viable that solution is. However I also see the appeal of your implementation and would not suggest that you remove it - it has great potential.
I will be testing it over the next week, and let you know if I encounter issues.

IMHO the best solution would be to load all indexes in memory, and apply them in real time against scanned blog contents.

I can add this. It's not hard to do and is actually how it was before this commit. It certainly consumes more memory, but might not be that much of an issue with our systems nowadays and of course somewhat scales with the usage. Download speedwise it should certainly be the best option. Since the old releases also leaked some memory I've no real comparison whatsoever.
If you are going to test this, I'll add it later this week. I'd like someone to test it who has downloaded way more content than I have.

Thank you! I will definitely commit to testing this with a setup of 100+ blogs. I will test both the OP "merged directories" method and the latest proposal, when added.
I also started to like the "merged directories" functionality a LOT, so please really don't remove it.

Downloading images works fine. But the program does not download image metadata (in format images.txt), does this mean that this function has disappeared completely from the program? There are xml files here, do they replace this function? I looked at xml files inside, they contain more data, although for me useful information among them is actually not much more than it was in the previous metadata files (I found useful among the new data only the note count), there is more unnecessary information (like avatar urls and links to smaller files) and just a lot of redundant text.
Just as an example, here's downloaded xml file for only one file. In an unpacked size, the xml file occupies 15.6 kilobytes on the disk. In a metadata file, this would be only 6 to 8 lines approximately (and the useful information will be almost as much as there is in an xml file).
tumblr_omnfiaNZrm1r8wg3no1_1280.zip
For interest, I also compared the total size of xml files with the size of one metadata file from the same blog (using the old version of the program) and found that for a blog with 2500 images, xml files occupy 90 megabytes, and one images.txt file occupies 612 kilobytes. The difference is more than a hundred times. It would be nice to reduce the size of xml files somehow. Just personally, I download many metadata files, such a difference is very significant for me. Or if you would return the option to download metadata in the previous format, then it would be very good for me.
Thank you very much for hard work and for regular updates!

Or if you would return the option to download metadata in the previous format, then it would be very good for me.

It actually was a regression. I've messed up the refactoring that broke downloading of all text posts. I've brought it back in the -a release that I've uploaded in the entry post of this issue.
I think the dump crawler data makes only sense if you use the svc branch. I've uploaded a v<IP_ADDRESS>a release that you might want to test. The .json files the svc crawler generate look similar to this (100 posts).

Downloaded v<IP_ADDRESS>a release, image metadata are downloaded normally, thanks! I noticed that if enable the option to download files by specific size (1280), then the program for one file downloads json files in at least two versions (250/400/540 and 1280). And if there are files from the photosets, then for them the program creates files with different counters, but with the same content inside. And here there are also other files with names consisting only of digits, they also repeat the content from the files tumblr _ * .json. In general, if in the Total Comember run a search by same size and content, then the program finds many duplicate files.
For example, in my folder among the 5340 json files only 1127 are unique, the rest are copies.
Here is a screenshot of the search in Total Commander.
https://i.imgur.com/kXRDWQn.jpg
And here is a screenshot of the selection of all copies of files, except for the unique ones. https://i.imgur.com/cIg4Ldu.jpg
In general, if count file sizes, then json files together take up 55.6 megabytes and 47.6 megabytes from them are actually copies.
In v<IP_ADDRESS> about the same thing, but it seems the program does not create for one image two versions of xml file when I enable the option to download files by specific size.

Hello, Johannes I just downloaded the new version and found that I can not add more than 50-60 blogs from the clipboard at a time if I copy a lot more addresses. First blogs are added, and then this process stops completely and further blogs are not added. It seems to me that the reason is not in any particular blog, but simply the new version does not process than 50-60 blogs. Here is a list of 3500 blogs, you can see if there is a problem on other machines if you copy the entire list completely, and then from this program adds only the first 50-60 blogs and nothing more.
3500 blogs.txt

Ah, I see. I can already imagine what the problem is. I'll fix it right away.
Thanks for testing!

Using <IP_ADDRESS>, Windows 10
I have a suspicion that the new version crashes on attempt to add 18+ blogs. I add blogs directly via entry field in the app, and the application crashes. If I use the clipboard, then it doesn't catch the blog. Tried the same with kitten blogs, and not a single one caused such issue which makes me suspect that this may be related to NSFW tags.
Here is one fairly tame 18+ blog http://torikev.tumblr.com/ that crashes every time on attempt to add.
EDIT: Seems to be random thing, maybe not 18+ related. Many blogs work, others (such as the above) cause crash every time.

I downloaded the new version <IP_ADDRESS>d, the program already adds more blogs, but not all. Of 3500 blogs (most of them are exactly online) the program added only 747 blogs. When I cleared the list of 3500 blogs from the already added 747 blogs and copied the remaining part of the previously missed addresses, the program added only a part of them (288 blogs of 2751 blogs that the program previously missed). On the third time, when I cleared the list of addresses from successfully added blogs and copied it, the program again added part of them. Thus, the program still does not cope completely with the addition of all addresses in the program.
And probably the reason is not the big size of the list because when I tried to copy 50 blogs, the program of them added 28 blogs. And when I only tried to copy only 10 addresses, the program of them added only 5.
For an example, try adding these 20 blogs to the program. I checked all of them in the browser and they are exactly online. But of them the program adds only 8 blogs.
20 blogs.txt

Ah, looks like there were two issues with the new detection. The ones that aren't added in your 20 blogs text file example are safe mode blogs.

If you want to test, then here is a list of 2608 blogs, all the metadata from them were downloaded in the previous version a few days ago, so they are all exactly online. List of 3500 blogs do not use, it's my old list and some of the blogs are probably already offline.
blogs.txt

With the new version, adding blogs has really improved! I see that all blogs from the list have been added. Very good! Thanks!

Thanks for the quick response! Using 33e version, all blogs that crashed seem to resolve properly now and get catched by the clipboard service. No crashes so far.

Has anyone encountered a very long pause after a blog seems to be downloaded completely, without proceeding to the next one?

ya, after a blog has finished downloading it stalls on that last download and wont move to the next blog.

It's too general. I've had it briefly tested and added these 3 small blogs, queued them all, added all options, all finished downloading:
http://wallpaperfx.tumblr.com/
http://mywallpapercollections.tumblr.com/
http://nature-e.tumblr.com/
So, what options did you enable? Can you specify a specific blog?
I'll probably not code anything until next year. So if it always happens for you, I'll pull the latest releases and see if I can fix it next year.

i had global database enabled, images,linked and reblogged selected.
would it also be possible to have an option to load all blog database into one single file. i delete the folder after a blog goes offline because i use the folder names to generate a list to add blogs back the to programs blog list when they dissapear due to crashes ect.

Hello. I had global database enabled, images + videos, included reblogs, set a folder (shared with 1 other blog) and minimum date 20171101 with no maximum date.

Well, I cannot reproduce any error, stall after complete crawl or hang after shutdown.
I've added 400 and more blogs, downloaded 15000 and more posts, turned on/off global database, shutdown TumblThree during the crawl, added blogs during the crawl, it behaves exactly like version v<IP_ADDRESS> for me. There was someone on my website who has apparently the same issue, but I am unable to fix it if I cannot reproduce it.
If someone here can provide a more detailed description, or is capable of programming and can debug the issue himself that would be wonderful.
Do you get any error messages during normal operation about connection timeouts?

I have image,video,reblog,linked and force rescan enabled, I also have the global option selected. The blogs will begin to scan like normal with no apparent issues but will seemingly stall at the last file, no matter how long I leave it the crawl is never completed, no error message or crash or anything, it just wont complete the blog crawl, if I press stop nothing seems to happen, the blog just continues to be unsuccessfully downloaded, and the thing is when you re-released this new update it was downloading fine from what I could tell, but after shutting the program down for the night and restarting it the next day is when it began giving me issues again, deleting app data and redownloading the program doesn’t seem to change anything. I didn’t change or do anything different between that time, it just randomly decided it would no longer work.

Could you upload the database files (_files.tumblr + .tumblr or similar) for a blog that stalls and your settings from C:\Users\AppData\Local\TumblThree\Settings?

iv also noticed that on top of the previous symptoms if i whipe the blog list and add blogs and directly crawl them without closing the program everything downloads normal,  it will reference and block re downloading from the global database.  but if i close the program and reopen and then crawl the blog whether it was an old blog or a newly added one that i haven't done anything with the crawler will download all of the content even if its repeated files, so its ignoring the global database and then hangs on the final file in the blog and remain incomplete.
altho i can remove the blog from the index but keep the _files files and readd the blog to the crawl list and crawl without closing the program and it will work appropriately and remember the files that were downloaded from the failed crawl. so if i just keep deleting the blog list and leaving the files and reloading them into9t he blog list i am able to download things normally, but only if i constantly reload the blog list every time i open the program and sit though it loading the blogs back into the program.
New folder.zip

Thanks for the description! Based on it, I've an idea what the problem might be: Maybe the _files files aren't completely loaded and restored in memory when you start crawling. As most of the code is using async-code, and the database loading code too, it actually runs concurrently and might not be finished when the GUI is up. Thus, if you have several hundreds of blogs with a lot of data, and you start the crawl after TumblThree is up, the method for loading the blogs _files databases might still be loading, but the crawl already tries to access them and that fails something I'm catching somewhere else in the code.
Maybe if I find some time in the next days I can quickly add a notification event that fires when everything is loaded. Otherwise you might want to check the Task Manager and see for the Disk I/O of TumblThree after starting it, and then maybe start the crawl a bit delayed. If that fixes the problem, then that was the issue.
I've tested your settings and the single blog file with 2-3 additionally added blogs, closed TumblThree, re-opened it, and it worked.


[docs] Tweak the "Edit this page" button icon

[x] I have followed (at least) the PR section of the contributing guide.

This PR was originally tweaking both the icon of the "Edit this page" button and its positioning of it. But after Gerda's comment bellow, I figured we could benefit from more exploration (which is better done on Figma) and, therefore, am reducing this to just changing the icon ⎯ super simple.

Before: https://mui.com/joy-ui/getting-started/overview/
After: https://deploy-preview-37142--material-ui.netlify.app/joy-ui/getting-started/overview/

Thanks Danilo! Couple thoughts:

I love the addition of the GitHub icon to indicate where you will go. 👏
I'm curious if changing the alignment makes it less discoverable, though? Unless that is the intended outcome?

We do have a comment button aligned to the right of each section.

Additionally, these are all different styles of buttons, so I added in the chart below a potential third option:

making the button styles consistent with each other on the page

although noting that they're not exactly consistent with the entire system right now

keeping the right alignment since they are all categorically related to feedback

though I think there is room to explore a wholly different positioning in the overall layout if we wanted to increase the scope of this PR

keeping the icon change

Current
Solution 1
Solution 2

LMK what you think about this!

@gerdadesign love it ⎯ thanks for the comment and review! As we've talked about, I've reduced this PR to tweaking just the icon, which seems like a safer move now!


Change Form.File input to custom in QuestionDisplay

Change Form.File type in the QuestionDisplay form to be a custom type Form.File instead of the regular one as to match the file upload style present in the application.

Same applies to DataForm in ApplicantView.js


feat: pagination drop manager

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
This PR:

add pagination to token/[id]
add pagination and integrate nft/[id]
add pagination and integrate ticket/[id]
cleaning up
refactorisation

How to test
Show steps to replicate and test the feature/fixes in this PR

Go to https://4429d54c.keypom-airfoil.pages.dev?v=42
Log in into one of the dummy account wallet from @BenKurrek
Select the Wallet Select menu and click on Master Key. Set "MASTER_KEY" as the master key
Go to My Drops by selecting the Wallet Select menu and proceed to click My Drops
Try to change page by clicking at the black button with arrows

Screenshots/Screen Recording of Implementation
how to:
https://user-images.githubusercontent.com/40631483/220005495-ca241be7-c265-4d8c-a6c8-110f6e5f8ab1.mov
screenshot:

When a drop is cut-off, currently it cuts off very early. Can we make sure it's inline with the max width of the table similar to the other drops?

Typo in drops page:

I'm seeing some weird behaviour where I can't select drops that are at the bottom of the page. My cursor doesn't register them. See message in slack for recording

This drop says it has 3 keys claimed:

But on the all drops page, it shows as 0 claimed:

For slow internet connections, there is a static drop that shows up on the drop manager page. Can we get rid of that? There is a loading animation and so I would assume that once the loading animation finishes, your drops show up. It seems weird to have a loading animation followed by a dummy drop followed by more loading and THEN your drops show up:
https://user-images.githubusercontent.com/57506486/220141875-69f03bf0-1111-422f-b0cc-56810a33ee24.mov

This drop says it has 3 keys claimed:  But on the all drops page, it shows as 0 claimed:
(for reference, this is the drop: https://4429d54c.keypom-airfoil.pages.dev/drop/ticket/1676571027569 and the account ID is dummy-account-1676569420877.testnet)

I logged the drop's id, dropName, next_key_id, as well as the getKeySupplyForDrop value and below is the console log result

the logic for the claimed value was written by @mattlockyer :
              claimed: `${
                next_key_id - (await getKeySupplyForDrop({ dropId: id }))
              } / ${next_key_id}`,

and it has displayed the right value for the following dropId:<PHONE_NUMBER>876 as shown below

As I remembered from our discussion last time, this was because the rest of the drop has randomized key which caused the key value to not be retrievable using getKey. Do you think that this is the reason?


Display time spent and ETA to editor progress dialogs

This is useful for long tasks such as lightmap baking.
I worked on this on the 3.x branch since I needed a long task to test it on, but I can remake this PR for the master branch once we agree on its design.
I'll need some help improving the ETA calculations. While they work well for tasks that advance continuously, they're useless for tasks that have few long steps (such as the "main" tasks in the BakedLightmap progress dialog shown below, as opposed to the subtasks). Nonetheless, this is not critical for now as this PR already improves editor usability a bit.
Preview
https://user-images.githubusercontent.com/180032/123471214-bb3d6980-d5f5-11eb-9cfb-fc1ee85baa5b.mp4

For the main task the ever increasing ETA is more confusing than no ETA IMO, so I'd remove it for now until we can figure out a good estimate.
ETAs that change continuously are bad UX IMO (unless it's a fast change to converge on something precise).

To improve the ETA estimation, I was thinking about displaying it only if there's a large enough amount of steps to perform (more than 100). This should ensure accurate estimations such as lightmap baking are still present, but less accurate estimates wont be displayed anymore.
I think the most important ETA to have is for lightmap baking anyway. It's rarely useful in other situations.


Upgrade client-go

Closes #93

Rebased against new master because of conflicts, should be able to merge cleanly now.


Configure typescript architecture

Codecov Report

Merging #405 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master      #405   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           15        13    -2
  Lines          182       170   -12
=========================================
- Hits           182       170   -12

Flag
Coverage Δ

#cypressspecs
?

#jestspecs
100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

assets/index.js

assets/services/getUrlParams.js

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34b7a9a...3120151. Read the comment docs.


Updates odbc-fdw extension to version 0.4.0

We now have a new default version of the odbc_fdw extension, 0.4.0, released and available on all database servers for both PG10 and PG11. Context https://github.com/CartoDB/cartodb-platform/issues/5504#issuecomment-491825650

Tested in staging and everything seems OK, so merging


Update iOS docs and upload readme to demo app repo

[ ] Add a table of demo / code file / description of code
[ ] Linda generates readme in markdown for Pacu and hands it off with necessary pictures
[ ] Linda submits PR

We're giving Marshall until the end of the sprint to investigate the solution that we chose (#1). If we do not figure out how to get this working, then we will fall back to the alternative (#2).
--
#1: Host the scripts to generate the docs on the repo, CI runs the script whenever there is a new github tag.
+: automated documentation, fresh per release
+: abstracts away how documentation works from CI
-: more difficult to implement in CI; our mac infra is not mature and this may not be possible
#2: Host the docs on the repo (bloaty), and CI pulls .html files whenever there is a new github tag.
+: automated documentation, fresh per release
+: abstracts away how documentation works from CI
-: bloated repo (although our docs are small at the moment)
-: breaking conventions (people don't normally host docs with the source code)
Notes for #2:

All docs must be at repo/docs/rtd/index.html --having it in a consistent place across all repos means that he doesn’t have to sort of hard-code different paths for each repo. Name the file index.html, and do not change it--changing it would break the CI build.
Automate generating this documentation per build by adding a line to the makefile.


feat: add progress reporter for build

This PR adds the first step of adding progress reporting for the pixi build progress. This creates some reporter traits
that can be used to report progress, but only implements noop reporters. Will implement actuall reporting of progress in subsequent PR.

You are right it would probably need to be passed into the build_source_record functions and such.

Changed this.

I added an allow is that alright?

jeppers!


fixed bug whith empty datasets

if we have 4 datasets and one of them is empty, driver thinks that we have only 3 datasets and skip the empty one

This might have more value on the upstream project. This is just a fork for one PR I made


WIP: Overhaul

Aims:

Grand cleanup (removing dead code, following best practices, etc.)
Increasing test coverage (>80% on each 'module')

Why?
We want to have our test-suite more reliable. For that, we now use the bundled substance library from within tests, eating our own dogfood.
substance is now mature enough to flesh out the test-suite and reach a reasonably high test-coverage.
Status:

[x] /dom
[ ] /model
[ ] /packages
[ ] /ui
[ ] /util

Codecov Report

Merging #1020 into develop will decrease coverage by 5.71%.
The diff coverage is 84.98%.

@@             Coverage Diff             @@
##           develop    #1020      +/-   ##
===========================================
- Coverage    64.78%   59.06%   -5.72%
===========================================
  Files          130      267     +137
  Lines         8958    11213    +2255
===========================================
+ Hits          5803     6623     +820
- Misses        3155     4590    +1435

Impacted Files
Coverage Δ

packages/spell-check/CorrectionTool.js
0% <ø> (ø)

collab/DocumentClient.js
0% <ø> (ø)
:arrow_up:

model/documentHelpers.js
74.34% <ø> (ø)
:arrow_up:

ui/Clipboard.js
92.95% <ø> (ø)
:arrow_up:

model/DocumentSchema.js
50% <ø> (ø)
:arrow_up:

collab/computeSnapshot.js
90% <ø> (ø)
:arrow_up:

packages/list/ListItemHTMLConverter.js
66.66% <ø> (ø)

ui/EditInlineNodeCommand.js
0% <ø> (ø)

packages/gutter/Gutter.js
0% <ø> (ø)

model/ContainerAnnotation.js
52.17% <ø> (ø)
:arrow_up:

... and 329 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fe1c7...028adcf. Read the comment docs.

Codecov Report

Merging #1020 into develop will decrease coverage by 5.46%.
The diff coverage is 86.59%.

@@             Coverage Diff             @@
##           develop    #1020      +/-   ##
===========================================
- Coverage    64.78%   59.31%   -5.47%
===========================================
  Files          130      267     +137
  Lines         8958    11190    +2232
===========================================
+ Hits          5803     6637     +834
- Misses        3155     4553    +1398

Impacted Files
Coverage Δ

packages/codeblock/CodeblockHTMLConverter.js
40% <ø> (ø)

packages/input/Input.js
0% <ø> (ø)

ui/AnnotationTool.js
0% <ø> (ø)
:arrow_up:

ui/AnnotationCommand.js
72.91% <ø> (ø)
:arrow_up:

packages/heading/Heading.js
100% <ø> (ø)

collab/DocumentEngine.js
68.75% <ø> (ø)
:arrow_up:

packages/image/ImageNode.js
50% <ø> (ø)

packages/switch-text-type/SwitchTextTypeTool.js
1.61% <ø> (ø)

packages/image/ImageHTMLConverter.js
50% <ø> (ø)

model/documentHelpers.js
74.34% <ø> (ø)
:arrow_up:

... and 329 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fe1c7...1f43a78. Read the comment docs.

Codecov Report

Merging #1020 into develop will decrease coverage by 5.47%.
The diff coverage is 86.59%.

@@            Coverage Diff             @@
##           develop   #1020      +/-   ##
==========================================
- Coverage    64.78%   59.3%   -5.48%
==========================================
  Files          130     267     +137
  Lines         8958   11189    +2231
==========================================
+ Hits          5803    6636     +833
- Misses        3155    4553    +1398

Impacted Files
Coverage Δ

packages/heading/Heading.js
100% <ø> (ø)

model/Schema.js
55.55% <ø> (ø)

model/NodeFactory.js
0% <ø> (ø)

model/HTMLExporter.js
34.48% <ø> (ø)
:arrow_up:

packages/table/TableCellComponent.js
63.63% <ø> (ø)

model/DefaultChangeCompressor.js
3.84% <ø> (ø)

packages/paragraph/Paragraph.js
100% <ø> (ø)

model/PropertyIndex.js
56.25% <ø> (ø)

packages/prose-editor/ProseArticle.js
0% <ø> (ø)

collab/DocumentClient.js
0% <ø> (ø)
:arrow_up:

... and 329 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fe1c7...0cbd168. Read the comment docs.

Codecov Report

Merging #1020 into develop will decrease coverage by 5.41%.
The diff coverage is 86.43%.

@@             Coverage Diff             @@
##           develop    #1020      +/-   ##
===========================================
- Coverage    64.78%   59.36%   -5.42%
===========================================
  Files          130      267     +137
  Lines         8958    11197    +2239
===========================================
+ Hits          5803     6647     +844
- Misses        3155     4550    +1395

Impacted Files
Coverage Δ

model/Coordinate.js
63.63% <ø> (ø)
:arrow_up:

packages/heading/HeadingComponent.js
100% <ø> (ø)

packages/base/BasePackage.js
0% <ø> (ø)

packages/inline-wrapper/InlineWrapperComponent.js
84.61% <ø> (ø)

model/ContainerAnnotation.js
52.17% <ø> (ø)
:arrow_up:

model/Range.js
30.76% <ø> (ø)
:arrow_up:

ui/AnnotationTool.js
0% <ø> (ø)
:arrow_up:

packages/toolbar/ToolDropdown.js
0% <ø> (ø)

model/Fragmenter.js
96.69% <ø> (ø)
:arrow_up:

model/Transaction.js
91.11% <ø> (ø)
:arrow_up:

... and 329 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fe1c7...767f0f6. Read the comment docs.

Codecov Report

Merging #1020 into develop will decrease coverage by 4.98%.
The diff coverage is 88.75%.

@@             Coverage Diff             @@
##           develop    #1020      +/-   ##
===========================================
- Coverage    64.78%   59.79%   -4.99%
===========================================
  Files          130      267     +137
  Lines         8958    11157    +2199
===========================================
+ Hits          5803     6671     +868
- Misses        3155     4486    +1331

Impacted Files
Coverage Δ

packages/switch-text-type/SwitchTextTypeCommand.js
0% <ø> (ø)

packages/toolbar/Toolbar.js
0% <ø> (ø)

model/ContainerAnnotation.js
52.17% <ø> (ø)
:arrow_up:

packages/base/BasePackage.js
0% <ø> (ø)

ui/AbstractIsolatedNodeComponent.js
77.14% <ø> (ø)
:arrow_up:

packages/emphasis/EmphasisComponent.js
100% <ø> (ø)

packages/heading/HeadingHTMLConverter.js
100% <ø> (ø)

model/CoordinateAdapter.js
0% <ø> (ø)

packages/split-pane/SplitPane.js
0% <ø> (ø)

packages/persistence/SaveCommand.js
0% <ø> (ø)

... and 332 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fe1c7...743958d. Read the comment docs.

Codecov Report

Merging #1020 into develop will decrease coverage by 4.81%.
The diff coverage is 89.81%.

@@             Coverage Diff             @@
##           develop    #1020      +/-   ##
===========================================
- Coverage    64.78%   59.96%   -4.82%
===========================================
  Files          130      267     +137
  Lines         8958    11139    +2181
===========================================
+ Hits          5803     6679     +876
- Misses        3155     4460    +1305

Impacted Files
Coverage Δ

model/Node.js
89.21% <ø> (ø)

model/NodeFactory.js
0% <ø> (ø)

ui/AnnotationTool.js
0% <ø> (ø)
:arrow_up:

packages/quote-marks/InsertQuoteMarkCommand.js
10% <ø> (ø)

packages/link/LinkCommand.js
0% <ø> (ø)

model/Conflict.js
0% <ø> (ø)

packages/base/RedoCommand.js
0% <ø> (ø)

packages/list/ListItemHTMLConverter.js
66.66% <ø> (ø)

model/HTMLExporter.js
68.96% <ø> (+34.48%)
:arrow_up:

packages/emphasis/EmphasisHTMLConverter.js
100% <ø> (ø)

... and 332 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fe1c7...63ae8cb. Read the comment docs.

Codecov Report

Merging #1020 into develop will decrease coverage by 3.66%.
The diff coverage is 91.4%.

@@             Coverage Diff             @@
##           develop    #1020      +/-   ##
===========================================
- Coverage    64.78%   61.11%   -3.67%
===========================================
  Files          130      267     +137
  Lines         8958    11139    +2181
===========================================
+ Hits          5803     6808    +1005
- Misses        3155     4331    +1176

Impacted Files
Coverage Δ

packages/inline-wrapper/InlineWrapper.js
50% <ø> (ø)

packages/base/UndoCommand.js
0% <ø> (ø)

packages/button/Button.js
0% <ø> (ø)

packages/table/InsertTableCommand.js
0% <ø> (ø)

packages/tabbed-pane/TabbedPane.js
0% <ø> (ø)

packages/list/ListItemHTMLConverter.js
66.66% <ø> (ø)

model/DOMExporter.js
75.58% <ø> (+5.81%)
:arrow_up:

collab/DocumentClient.js
0% <ø> (ø)
:arrow_up:

packages/split-pane/SplitPane.js
0% <ø> (ø)

collab/CollabServer.js
0% <ø> (ø)
:arrow_up:

... and 333 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fe1c7...492ca0d. Read the comment docs.

Codecov Report

Merging #1020 into develop will decrease coverage by 1.59%.
The diff coverage is 88.66%.

@@            Coverage Diff             @@
##           develop    #1020     +/-   ##
==========================================
- Coverage    64.78%   63.18%   -1.6%
==========================================
  Files          130      318    +188
  Lines         8958    11717   +2759
==========================================
+ Hits          5803     7403   +1600
- Misses        3155     4314   +1159

Impacted Files
Coverage Δ

model/paste.js
86.66% <ø> (ø)
:arrow_up:

model/SelectionState.js
88.88% <ø> (ø)
:arrow_up:

packages/switch-text-type/SwitchTextTypeCommand.js
0% <ø> (ø)

packages/text-input/TextInputPackage.js
100% <ø> (ø)

model/Transaction.js
91.11% <ø> (ø)
:arrow_up:

model/DocumentNode.js
59.18% <ø> (ø)
:arrow_up:

packages/base/SelectAllCommand.js
0% <ø> (ø)

packages/grid/Grid.js
14.28% <ø> (ø)

packages/toolbar/ToolGroup.js
0% <ø> (ø)

packages/paragraph/ParagraphComponent.js
100% <ø> (ø)

... and 390 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fe1c7...b3e06ad. Read the comment docs.

Codecov Report

Merging #1020 into develop will decrease coverage by 1.16%.
The diff coverage is 90.04%.

@@             Coverage Diff             @@
##           develop    #1020      +/-   ##
===========================================
- Coverage    64.78%   63.61%   -1.17%
===========================================
  Files          130      318     +188
  Lines         8958    11711    +2753
===========================================
+ Hits          5803     7450    +1647
- Misses        3155     4261    +1106

Impacted Files
Coverage Δ

model/ContainerAnnotationIndex.js
100% <ø> (ø)
:arrow_up:

model/JSONConverter.js
88.88% <ø> (ø)
:arrow_up:

collab/Server.js
0% <ø> (ø)
:arrow_up:

model/ContainerAnnotation.js
52.17% <ø> (ø)
:arrow_up:

collab/CollabSession.js
0% <ø> (ø)
:arrow_up:

packages/heading/HeadingHTMLConverter.js
100% <ø> (ø)

ui/AnnotatedTextComponent.js
68.96% <ø> (ø)
:arrow_up:

packages/file/FilePackage.js
100% <ø> (ø)

packages/emphasis/EmphasisComponent.js
100% <ø> (ø)

model/NodeIndex.js
50% <ø> (ø)

... and 392 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fe1c7...e95ae22. Read the comment docs.

Let's call it a first round. dom and model looking good.
ui and util must be covered better in a second round.
packages and collab are utterly undertested.


Write a GO program to find maximum of n numbers

Description
Write a GO program to find maximum of n numbers
Input  : 10 20 30 40 50
Output : 50

How to contribute

Save the solution in find_maximum_of_n_numbers.go file
Add find_maximum_of_n_numbers.go file in find-maximum-of-n-numbers folder

!assign

!assign


[Question] Cannot find "mlc-chat-config.json" in path ""dist"/RedPajama-INCITE-Chat-3B-v1-q4f16_1

❓ General Questions

I followed the steps mentioned at https://mlc.ai/mlc-llm/docs/get_started/try_out.html
and i got this after running
(mlc-chat-venv) D:\textgen\dist\prebuilt>mlc_chat_cli --local-id RedPajama-INCITE-Chat-3B-v1-q4f16_1
Cannot find "mlc-chat-config.json" in path ""dist"/RedPajama-INCITE-Chat-3B-v1-q4f16_1
because the files are in prebuilt folder i think which is a problem

Thanks for the question @TanmayDoesAI! Could you try invoking mlc_chat_cli under D:\textgen?

@CharlieFRuan ohhhh, I completely forgot, that was a rookie error, thanks for help!

hey, I tried a similar thing
firstly cloned the llama 2 70b
(mlc-chat-venv) D:\text-gen\dist\prebuilt>git clone https://huggingface.co/mlc-ai/mlc-chat-Llama-2-70b-chat-hf-q3f16_1
then got this
(mlc-chat-venv) D:\text-gen>mlc_chat_cli --local-id Llama-2-70b-chat-hf-q3f16_1
Use MLC config: "D:\text-gen\dist\prebuilt\mlc-chat-Llama-2-70b-chat-hf-q3f16_1\mlc-chat-config.json"
Use model weights: "D:\text-gen\dist\prebuilt\mlc-chat-Llama-2-70b-chat-hf-q3f16_1\ndarray-cache.json"
Cannot find library "Llama-2-70b-chat-hf-q3f16_1-vulkan.dll" in "dist"/prebuilt/lib or other search paths
@CharlieFRuan could you help with this?

Hi @Myracodes20, unfortunately llama2 70b is only supported in -metal.so and -webgpu.wasm, as shown in this repo (the one you cloned in the tutorial), which contains all the model libraries.
This is because llama2 70b requires a great amount of memory (M2 Max can run it). We are currently working towards making 70b deployable in more platforms/devices.
In the mean time, you could perhaps try out Llama2 13B/7B, which are supported in many more platforms.

oh okay, thanks for the help!


recorder: cache downloaded tracks

... to allow a user to save as GPX, then instantly as CSV, rather than waiting to download the same data again.

Nice one!


Better debug formatting

Hello,
I would like to propose a config option that would improve the readability of bitmask variables printed using the debug formatting {:?}.
At the moment only the default derived implementation is available, which in the output looks like this: Bitmask { bits: 1 }.
Without looking at enum's declaration it is impossible to decipher, which value this corresponds to.
My proposition is a debug_as_vec config option. Adding it to enum's declaration would cause a custom Debug trait implementation to be created for the type. Said implementation would print the bitmask as a vec of all matching values that exist in the enum. Example:
#[bitmask]
#[bitmask_config(debug_as_vec)]
pub enum BitmaskDebugAsVec {
    Flag1,
    Flag2,
    Flag12 = Self::Flag1.or(Self::Flag2).bits,
    Flag3,
}

assert_eq!(format!("{:?}", BitmaskDebugAsVec::none()), "[]");
assert_eq!(format!("{:?}", BitmaskDebugAsVec::Flag1), "[Flag1]");
assert_eq!(format!("{:?}", BitmaskDebugAsVec::Flag2), "[Flag2]");
assert_eq!(format!("{:?}", BitmaskDebugAsVec::Flag12), "[Flag1, Flag2, Flag12]");
assert_eq!(format!("{:?}", BitmaskDebugAsVec::Flag3), "[Flag3]");
assert_eq!(format!("{:?}", BitmaskDebugAsVec::full()), "[Flag1, Flag2, Flag12, Flag3]");

Not adding the option would mean the default debug formatting, as it has always been.
I have already created a working implementation of this feature on my fork. If you like the idea I will make a PR.

Alternatively, we can add a simple no_debug config that would only prevent the Debug trait to be automatically derived. Then the user can implement it himself.

Hey,
I actually really like this idea, and it is a perfect use for a config option.
I already looked at your implementation on your fork, and it looks good, so I don't mind if you create a pr that I can merge.
It is missing the README, but I can do that when changing the version.
The only thing I am not 100% certain of is the name of the config option, but truth be told, I am not sure about the ones I can think of either.
Some of my options include:

vec_debug
list_debug
advanced_debug

You see, my main problem seems to be with the word as, because it is, in my opinion, unnecessary.
Any thoughts?

I'm totally fine with a different name. I wasn't even sure about mine in the first place. Out of your suggestions, I think I would go with vec_debug.
I will change the name, add an entry for it to the README and make a PR.

What do you think about instead of just the vec like [Flag, ...] we do it more in line with the Debug trait and include the name of the bitmask like BitmaskVecDebug [Flag, ...] or even BitmaskVecDebug { flags: [Flag, ...] }?

I think that I might prefer BitmaskVecDebug[Flag, ...], because it is more in line with the Debug trait but does not add to much bloat with braces and the identifier of the enum before every flag.

All right, I'll go with the BitmaskVecDebug[Flag, ...] form.
I didn't like the BitmaskVecDebug { flags: [Flag, ...] }, because it somewhat implies that this way there is a flags field in the type, which could be misleading.

Perfect, thank you for the contribution.
I will merge now but wait with publishing until tomorrow or Sunday, so if you have any changes or suggestions, they can be included.

Ok, I'll change the printing format to the one we've discussed just now then.

All done.

Just as I was going to move on I tried adding both config options to enum's declaration and realised the output is invalid, haha. I'll make a fix right away.

Interesting, good catch, can you add a test for that?

Ohh, i think I misunderstood, thought it was the config parsing, maybe no test needed!

No no, the test is needed, I am adding it. The funny thing about the mistake in parser is that it takes just one character to be changed to fix it, lol.

The bug was that when inverted_flags was also configured, regular values would be checked and printed instead of the inverted ones.
#[bitmask(u8)]
#[bitmask_config(vec_debug, inverted_flags)]
pub enum BitmaskVecDebug {
    Flag1,
    Flag2
}

println!("{:?}", BitmaskVecDebug::full()); // BitmaskVecDebug[Flag1, Flag1, Flag2, Flag2];


ARROW-13037: [R] Incorrect param when creating Expression crashes R

LGTM but @nealrichardson should take a look too.


Fix examples in docs and update for recent work.

Closes #105.

lgtm

Withdrawing lgtm based on looking more in detail :D

Updated for review feedback.

LGTM.

Added some commas. :)

lgtm - please squash

Squashed


Reproducing paper results

Unfortunately I wasn't able to get close to paper results. My results on MUSDB18HQ dataset are following:
SDR bass: 1.0778
SDR drums: 2.5914
SDR other: 0.6353
SDR vocals: 2.9829

My method - I used as query 100 train tracks from MUSDB18HQ dataset. Then I created averaged 2048-vector for ['bass', 'drums', 'other' and 'vocals']. Then I split test tracks using this vector and calculated the SDR value. May be I missing something?

For these results, I think there must be some errors. I have several suggestions/questions to see if you could find some problems:

Our checkpoint contains two parts: the first one is the separation model (zsp_full.ckpt), another is the htsat/st-sed detection system checkpoint (2048-d.ckpt). The sound event detection system's config file is htsat_config.py, did you also load this checkpoint in htsat_config.py?
if you download the musdb18 dataset from the official website, make sure it is converted to 32000 sample rate, because the original one is 44100.  You can also use our processed file, detail link in readme.md.
The query is from musdb18 training set, but it is the 30s highlight version and it does not contain the silence frame.  The testing set is the full testing set (i.e. not 30s). You can follow our readme.md link to find the highlight version, or you can extract it through the toolkit from sisec 2018.
Did you output some separation files to see how it separates? This number seems that it does not perform the separation.

Also, if you do not use the highlight version of training set, it will also produce the similar result but slightly lower than the paper result (because some silence frames drop the embedding from a better one). But it should not be 0-3 SDR.

Yes probably problem was with sample rate. I changed it in config from 32000 to 44100.
After our discussion I returned it back to 32000. My validation on MUSDB18HQ for bass increased from 1.0778 to 2.6361. But it's still far from 3.8. I'm trying to find what else I'm missing.
I split your code for inference in 2 parts (I createed the pull request with these files):

zero_shot_create_vector.py - create vector out of queries
zero_shot_process_file.py - process file with vector

May be you can just post your vectors which you obtained for MUSDB dataset?

Thank you for your commit, I will take a look and merge your commit.
I have further suggestions regarding your reproduction:

The config.py and htsat_config.py files have the sample rate and you change it to 32000. It is only responsible for how model loads the data. Our model is trained on 32000Hz (AudioSet is 32000Hz) so you need to set it 32000Hz.
You also need to convert the musdb18 dataset to 32000Hz using librosa.resample, the original one is 44100Hz, changing the sample rate variable in configuration files do not automatically change the data input sample rate. Check out code process_musdb in main.py to see more details. You can use our processed 32000Hz files musdb-32000.npy in readme.md
When calculating SDR values, we use the median SDR not average SDR. This follows the SiSec 2018 and other music source separation evaluation, which is different from speech source separation (use average SDR more).

I attach my query on vocal, drum, bass, other below. This model can perform the SDRs below (one of our attempts, we have three attempts to get the average values in the paper):
average_query_musdb.zip

Please let me know if you have further questions, I'm happy to help you achieve the results I got.


Bug: General validation in forms

What?

Forms need general validation of required fields before submission

@pKorsholm can u list all the places where the validations are needed.

The idea is to highlight which fields are missing to the user so they can easily identify which fields are missing data
Ideally this would apply to any form in the admin portal with required fields, I might have missed some, but here is a list:

When creating an order
When creating a discount
Product creation/update
Region details

Thanks a lot Rolwin 😄

ur welcome @pKorsholm :)


AbstractItem#isSelectable should probably be read only

In almost all situations, getting the value is through some other field. In that case, it doesn't make sense to add a setter.
I think the only case where the setter is not blank is
https://github.com/mikepenz/FastAdapter/blob/0cd1406eb3dcf361b963a8fd63ad9b8f72a15cb4/app/src/main/java/com/mikepenz/fastadapter/app/items/expandable/SimpleSubExpandableItem.kt#L56-L61
Note that subItems is never null, and calling isSelectable from the parent likely won't do anything

@AllanWang theoretically people can design the items to change the selectable state on per element base.
Not done in the sample, but I can think of usecases where this might be relevant.

Fortunately you can override val with var
open class A {
    open val a: Boolean = true
}

class B : A() {
    override var a: Boolean = true
}

Oh nvm. I see that you call the modifier yourself. In that case this is void


Customize styles

Have you considered making the styles like background color of the footer etc customizable?
At first, I thought that making those be variables in the templates config might be useful but I'm now leaning towards providing a stylesheet field (or array) for urls that are included after the built-in styles.
And maybe also a scripts array but I think that isn't quite as necessary.
If you think it's a good idea, I can definitely make a PR for that.

Yeah, that's a good idea.
I think we can conjugate with jsdoc's static files.
templates: {
  defaults: {
    staticFiles: {
      include: [...]
      exclude: [...]  <-- Maybe we can exclude the built-in styles
    }
  },
  css(?): [...]  <-- new feature with staticFiles property
}

I made a PR for an initial solution: #17.

v1.1.0 published to npm.js
Thank you :)


Use properties for service

Hrm. Still interested. I'll try to get this updated.

another one that i'll pick up

We really do want to do this, but at this point we're looking to take another angle.


Error get 'setpoint_raise_lower' from eCozy device

When I want to get the value {"setpoint_raise_lower":""} from an eCozy device, zigbee2mqtt crashes with the following:
                    zclData: [{attrId: zclId.attr(cid, attrId).value}],
                                                              ^

TypeError: Cannot read property 'value' of undefined
    at Object.convert (/app/node_modules/zigbee-shepherd-converters/converters/toZigbee.js:572:63)
    at keys.forEach (/app/lib/extension/devicePublish.js:147:41)
    at Array.forEach (<anonymous>)
    at DevicePublish.onMQTTMessage (/app/lib/extension/devicePublish.js:139:14)
    at results.extensions.filter.map (/app/lib/controller.js:134:27)
    at Array.map (<anonymous>)
    at Controller.onMQTTMessage (/app/lib/controller.js:134:14)
    at MQTT.onMessage (/app/lib/mqtt.js:81:18)
    at emitThree (events.js:136:13)
    at MqttClient.emit (events.js:217:7)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>start: `node index.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.

@Chrischi- do you know more about this?

@Koenkk @snoopyCode
As far as I know setpoint_raise_lower is just client-to-server and I think the get implementation is wrong.

Yes, get in toZigbee.js#L607 is wrong and should be removed. You should use thermostat_occupied_heating_setpoint or thermostat_unoccupied_heating_setpoint to read the value you changed with setpoint_raise_lower.

did a PR #308


MM-14322 Create E2E test for long post with multiple attachments

Summary
Create e2e to test long text posted with 4 attachemnets.
Ticket Link
https://github.com/mattermost/mattermost-server/issues/10370
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] Ran make check-style to check for style errors (required for all pull requests)
[x] Ran make test to ensure unit and component tests passed
[ ] Added or updated unit tests (required for all new features)
[ ] Needs to be implemented in mobile (link to PR or User Story)
[ ] Has server changes (please link)
[ ] Has redux changes (please link)
[ ] Has UI changes
[ ] Includes text changes and localization file (.../i18n/en.json) updates
[ ] Touches critical sections of the codebase (auth, posting, etc.)

I can't get a passing test of this spec, something related to uploading a file.  Could you please try to sort it out?
Also, please help rebase to master since I've added already the long_text_post.txt.  Another thing is you might want to take a look at collapsed_message_spec and apply some of helper functions there like verifyCollapsedPost/verifyExpandedPost and cy.getLastPostIdWithRetry(); actually might be better to move the test there instead of separate spec file.

This issue has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!
/cc @jasonblais @hanzei

Hey @nadaa,
Thanks for your PR. :+1: Is the provided feedback clear? Do you have questions?

Hey @nadaa,
Are you still working on this PR? Do you have any questions?
Please note that we will close this PR after another 10 days of inactivity. If you don't object, other contributors might complete this PR for you.

Hey @nadaa,
Thanks for this contribution. This PR has become orphaned and will be closed. If you still want to work on this, feel free to reach out.


dev: consider sonia-corporation/stale instead of actions/stale

I recently switched from actions/stale to sonia-corporation/stale.
actions/stale appears to have no further active development, and sonia-corporation/stale has a nice feature: exempting from marking as stale all issues that are in a project (along with issues that are in milestones, or assigned to someone, both of which actions/stale already had). So if people are proposing features, simply accepting them by adding them to a project, like say a version release plan, will keep them from being marked stale without having to add extra labels.
sonia-corporation/stale also has a much more comprehensive documentation site: https://sonia-corporation.github.io/stale/
The Yaml keys in the workflow have different names but there's basically a 1:1 correspondence, so it was very fast to update my existing workflow to a new one.
Anyway just passing this along in case it's helpful!

@rootwork, thanks a lot for sharing!
Seems to be that some commits has been done three days ago in the main branch. Is it really doesn't have any further development anymore?

@dec0dOS You're right! Maybe it is in fact still in active development.
I appreciated the sonia version for its project option, but if that's not generally applicable then maybe actions/stale is still fine.


DEADLINE_EXCEEDED: deadline exceeded after 29.999577300s

I used this code:
`
public class SimpleConsumerExample {
private static final Logger log = LoggerFactory.getLogger(SimpleConsumerExample.class);
private SimpleConsumerExample() {
}

@SuppressWarnings({"resource", "InfiniteLoopStatement"})
public static void main(String[] args) throws ClientException {
    final ClientServiceProvider provider = ClientServiceProvider.loadService();

    // Credential provider is optional for client configuration.
    String accessKey = "rocketmq2";
    String secretKey = "12345678";
    SessionCredentialsProvider sessionCredentialsProvider =
        new StaticSessionCredentialsProvider(accessKey, secretKey);

    String endpoints ="<IP_ADDRESS>:30402";
    ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder()
            .setRequestTimeout(Duration.ofSeconds(30))
        .setEndpoints(endpoints)
        .setCredentialProvider(sessionCredentialsProvider)
        .build();
    String consumerGroup = "yourConsumerGroup";
    Duration awaitDuration = Duration.ofSeconds(30);
    String tag = "yourMessageTagA";
    String topic = "testTopic";
    FilterExpression filterExpression = new FilterExpression(tag, FilterExpressionType.TAG);
    SimpleConsumer consumer = provider.newSimpleConsumerBuilder()
        .setClientConfiguration(clientConfiguration)
        // Set the consumer group name.
        .setConsumerGroup(consumerGroup)
        // set await duration for long-polling.
        .setAwaitDuration(awaitDuration)
        // Set the subscription for the consumer.
        .setSubscriptionExpressions(Collections.singletonMap(topic, filterExpression))
        .build();
    // Max message num for each long polling.
    int maxMessageNum = 16;
    // Set message invisible duration after it is received.
    Duration invisibleDuration = Duration.ofSeconds(15);
    // Receive message, multi-threading is more recommended.
    do {
        final List<MessageView> messages = consumer.receive(maxMessageNum, invisibleDuration);
        log.info("Received {} message(s)", messages.size());
        for (MessageView message : messages) {
            final MessageId messageId = message.getMessageId();
            try {
                consumer.ack(message);
                log.info("Message is acknowledged successfully, messageId={}", messageId);
            } catch (Throwable t) {
                log.error("Message is failed to be acknowledged, messageId={}", messageId, t);
            }
        }
    } while (true);
    // Close the simple consumer when you don't need it anymore.
    // consumer.close();
}

}
`
Then error is:
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 29.999577300s. [remote_addr=/<IP_ADDRESS>:30402]
at io.grpc.Status.asRuntimeException(Status.java:539)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

这个是5.0客户端问题， 建议issue放：https://github.com/apache/rocketmq-clients/issues

这个是5.0客户端问题， 建议issue放：https://github.com/apache/rocketmq-clients/issues
It's not a 5.0 client issue. This error was returned by the proxy, and it should be a bug in the proxy.

It's not a 5.0 client issue. This error was returned by the proxy, and it should be a bug in the proxy.

It's not a 5.0 client issue. This error was returned by the proxy, and it should be a bug in the proxy.

You can set useEndpointPortFromRequest to true and try again. See more information in #6268

I've already set it up，then I can send messages, but I can't consume

I've already set it up，then I can send messages, but I can't consume

I've already set useEndpointPortFromRequest =true，then I can send messages, but I can't consume

这个错误是GRPC client访问proxy超时了， 我看了下rocketmq client关于deadline的的超时时间设置代码，
应该是deadline= awaitDuration + request timeout, 从你的代码看应该是60s，但是实际报错是30s。

建议检查下

pod网络策略、安全组这种配置
确认下代码和报错是否匹配，再执行下，现在的报错知道是grpc遇到deadline了，看看能否把rocketmq client相关的堆栈打出来。
也可以用https://arthas.aliyun.com/doc/trace.html trace下客户端和proxy， 看看请求到proxy没有， 到了的话是真的grpc超时了， 还是长轮训超时了

这个问题是我用高版本JDK（java11）编译，低版本Jdk（java8）运行造成的，有些方法缺失造成的bug，换成JDK8编译后部署就可以了

I have the problem too, When i use rocketmq-client-java 5.0.6，5.0.7 同connect rockert proxt 5.3.1, when fetchTopicRoute it happend this problem:
org.apache.rocketmq.shaded.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2.999828700s. [remote_addr=/<IP_ADDRESS>:31831]
I am sure the <IP_ADDRESS>:31831 tcp network can connect, beacuse i can telnet it. but the rocketmqclient connect make the grpc exception.

the client runtime enriment is jdk17,rocket proxy server is open jdk8

这个问题是我用高版本JDK（java11）编译，低版本Jdk（java8）运行造成的，有些方法缺失造成的bug，换成JDK8编译后部署就可以了

你用jdk11编译。用jdk8运行，指的是proxy端吗？但是我用的官方5.3.1镜像也报这个错

I have the problem too, When i use rocketmq-client-java 5.0.6，5.0.7 同connect rockert proxt 5.3.1, when fetchTopicRoute it happend this problem:
org.apache.rocketmq.shaded.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2.999828700s. [remote_addr=/<IP_ADDRESS>:31831]
I am sure the <IP_ADDRESS>:31831 tcp network can connect, beacuse i can telnet it. but the rocketmqclient connect make the grpc exception.
 the client runtime enriment is jdk17,rocket proxy server is open jdk8

it's ok now, i find what ever wrong, for example sslEnable client server not equal, or secretKey wrong, all exception Message is "DEADLINE_EXCEEDED", It's misled me into investigating the wrong issue.


Fix dropped values from queries by using FlatParamsEncoder

The uri_template gem, which hyperclient uses to build urls, implements RFC6570, that expands a parameter in the {?foo*} format as ?foo=a&foo=b&foo=c.
Unfortunately, faraday's default argument parser (Faraday::Utils.default_params_encoder) is set to the NestedParamsEncoder which only considers parameters to be arrays if they have [] in the key.
The result is that values for a parameter are dropped in this process:
require 'faraday'
require 'uri_template'

Faraday::Utils.default_params_encoder.decode(
  URITemplate.new('http://example.com/{?foo*}').expand(foo: ['a', 'b']).split('?').last
)
=> {"foo"=>"b"}

And thus the resulting faraday request will be missing some of the values.
The solution is to use faraday's other parameter encoder (the FlatParamsEncoder), that correctly encodes such requests.
Fixes #114

Thanks. Feel free to nuke or ignore jruby from the build and get it to green.

The test doesn't seem to be testing the actual problem here, you should write a spec that demonstrates the issue at hand, please.

Thanks for the feedback. I tried to do so, but the stub_request also uses faraday to parse methods, so it also ignores multiple parameters. I will try to take a different approach and get back to you.

@dblock Updated! I ended up going for a spinach tests, since those are using webmock, but still had to configure webmock to behave properly, otherwise its default config also drops values... The ruby ecosystem really doesn't like RFC 6570.

Alright! Just need a passing build.

Merged, thank you.

Thanks for the quick feedback!


Add a tick box for support audit export users

Context
Support want the ability to export only school and rb users
Changes proposed in this pull request
Add a checkbox to the support export users form
Guidance to review

 Add scoping to support export user to only include school and rb users


DS-3209 AIP Import: Extend accepted handles for supports()

Adds an additional supported handle format to the supports() method of
VersionedHandleIdentifierProvider to make behavior more in-line with the
DS 5.4 code.
https://jira.duraspace.org/browse/DS-3209

Note: this PR was developed/tested at Georgetown and it resolved an AIP ingest issue locally.

+1 is just restoring code from 5.4 which seems to have been accidentally stripped out

Since DSpace 6 we have to VersionedHandleIdentifierProvider. We should enhance both. While doing so I thought it would be good to also enhance the method retrieveHandleOutOfUrl. I put my changes here: https://github.com/tuub/DSpace/tree/DS-3209. Can you please review and test my addition and tell me what you think about that? If you like my additions, you could just cherry-pick them to the branch you used to create this PR, that would update this PR automatically.

Thanks, @pnbecker. I looked through your suggested changes and thought they made sense to me.
Unfortunately, with the way we use our environment, one of the lines would break the item import process for us. In our environment, we have one production instance and three non-prod instances of DSpace. Each separate instance has it's own handle prefix (e.g. 12345, 12345.1, 12345.2, 12345.3) that we use to help determine which items were created where. So the getPrefix() comparison on line 353 would then break any item imports from our other environments since those items would have slightly different handle prefixes.
I think we've been able to get away with our setup in previous versions of DSpace since the retrieveHandleOutOfUrl() method was slightly less stringent. I'm not sure if we're the only folks with that sort of a setup or if other institutions might be affected by that problem as well.
Does anybody else watching this PR have a similar setup or know of a similar setup? We may be a special case.
Has there ever been talk about the possibility of having a sort of regular expression handle.prefix configuration to provide more flexibility on this front?

Hi @mjmarttila
I added the line in question to ensure, that we a) have a handle and b) that we have a handle we are responsible for. When I saw the message I became afraid that the VersionedHandleIdentifierProvider would accept URLs like http://dx.doi.org/10.14279/depositonce-5015 as handle. While DOI build up on handle, the VersionedHandleIdentifierProvider shouldn't deal with DOIs as if they were handles.
DSpace currently cannot deal with multiple different prefixes. I don't see how this could easily be achieved and to be honest, I don't see much sense in doing so. Of course I'm open for arguments... What I do understand is, that it would make sense to support the use case where someone is working with one prefix that is extended by a dot and something else. So instead of testing equality, we could use StringUtils.startsWithIgnoreCases(String, String). That would support the use-case of extending one prefix, but would still ensure we have a handle out of our range. Would that keep you compatible?
If my proposed solution fits your needs, can you extend your PR by cherry-picking my commit and adding another commit changing the equality checks or shall I create a new PR?

Hi @pnbecker,
The reasons you’ve introduced the additional handle validation in that line certainly make sense, but unfortunately even with your suggested startsWithIgnoreCases() modification, the solution would break a multi-instance environment like ours.
It sounds like you already understand the value of using both a primary prefix and derived prefixes that extend it with a dot, but for the sake of the PR conversation history I’ll provide some additional context around why we use derived handle prefixes: when we import a production item into our test instances, we want to retain the production handle (e.g. XXX/YYY). When one of our test systems assigns a handle, we want to use a derived handle prefix (e.g. XXX.1/YYY) to indicate that the object originated on a particular test server. This way regardless of what content gets transferred where, we know which instance each object originated from.
So if we set the handle.prefix property on all of our test instances to our production handle with the startsWithIgnoreCases() addition in place, it would technically allow our test systems to import any of our existing exported AIPs. But it would also cause our test systems to then generate new handles using only the production prefix (e.g. XXX/YYY for all instances), breaking our system of determining where the content originated from.
If other institutions have a similar setup with derived handle prefixes, the solution as it stands would prevent those test handles from being assigned. What would folks think about something along the lines of a property in the local.cfg that the supports() method checks and, when set/not default, has the method split the configured handle prefix on the user-specified regex and only uses the first value for the prefix comparison? So in our case we could specify “[.]” as our expression which would then let the supports() check only look at the primary prefix but would not affect handle assignment during new object creation.

Hi @mjmarttila,
thanks for the context, that helps to understand what your aiming to. Are items that originated on your production server ever moved to the testservers? If not, your setup should even work with the startsWithIgnoreCase() comparison. In case you copy items originated on your production server to the test instances, could you please locally override this class using DSpaces "additions" module?
While thinking about how to solve this issue, I wrote the following code. But this requirement is so specific that I really hesitate to include this into DSpace's official code base.
public static String retrieveHandleOutOfUrl(String url)
    {
        // We can do nothing with this, return null
        if (!url.contains("/")) return null;

        String[] splitUrl = url.split("/");

        if (splitUrl.length >= 2)
        {
            // Handle prefixes can be extended by a "." and arbitrary numbers.
            // We need to ensuere here, that we found a handle containing a prefix
            // we are responsible for. Check this a little bit fuzzy to support
            // setups running multiple instances using an extended prefix.
            String retrievedPrefix = splitUrl[splitUrl.length -2];
            if (StringUtils.substringBefore(getPrefix(), ".").equalsIgnoreCase(
                    StringUtils.substringBefore(retrievedPrefix, ".")))
            {
                return retrievedPrefix + "/" + splitUrl[splitUrl.length - 1];
            }
        }
        return null;
    }

What does others think of this discussion?

Hi @pnbecker,
Items originating from our production server do get transferred to our test servers so that we can test custom features that are in development on a slice of our production data before pushing those features to our prod instance.
If we're the only deployment using this sort of a multi-instance setup, then we could certainly do a local override (and thanks very much for providing a code example of how this might be done). We just thought we would use this opportunity to advocate for support of the multi-instance setup in case other institutions were working off of a similar model.
To that end, I'll echo your call for others to comment on this discussion.

To add to this discussion, there are known use cases for a production DSpace site actually having items with multiple handle prefixes (even though a single DSpace site only assigns a single handle prefix).
One of the best examples is a federated repository, where one DSpace is actually harvesting / importing its content from other DSpace sites.  One example of this in production is the Texas Digital Library's (TDL) DSpace site: https://tdl-ir.tdl.org/tdl-ir/browse?type=title   If you look closely at the Item handles in that site, they come from many different Handle prefixes, as the original item exists in a separate DSpace.  In the case of TDL, their main repository just allows you to search the DSpace sites of all their member institutions.
It's also worth noting that many production DSpace sites assign handles that include dots in the prefix. So the fact that a prefix has a dot also shouldn't exclude it in any way.  Again, one example: http://oaktrust.library.tamu.edu/handle/1969.1/2815
So, long story short, I'm not sure we should be validating prefixes based on the configured handle.prefix.  That setting is only meant to represent what prefix DSpace will use when it assigns new handles. But, DSpace should still be able to import Items having existing handles (whether it be from AIP or OAI-PMH or wherever).

I read the protocol from yesterday's developer meeting (http://irclogs.duraspace.org/index.php?date=2016-05-25) and the comments in this ticket above. I hope I can clarify things a little bit:

The PR by @mjmarttila is a good one and we should do the same change to the VerisonedHandleIdentifierProviderWithCanonicalHandles and to the HandleIdentifierProvider.
While reviewing it, I became afraid that any HandleIdentifierProvider would return true on supports("http://www.example.org/test123/arbitraryPath") and return test123/arbitraryPath as handle. To say it simple: we shouldn't rely on dc.identifier.uri to contain handles only. ;-) As this PR connects the methods support() and retriveOutOfURL() we need to ensure that we found a handle.
@mwoodiupui mentioned in the developer meeting that we need to ensure that the HandleProvider does not claim to support DOIs. That is true and we should keep that in mind. But I'm even more afraid of arbitrary URLs that could be stored in dc.identifier.uri and include two slashes or more.

Regarding a possible solution: I thought it would be a good idea to use the handle prefix to ensure that we have a handle there. But @mjmarttila and @tdonohue proofed that might have downsides. What can we use instead? Can we use handle.canonical.prefix or should we just check whether /handle/ is part of the URL?

There are several cases to consider.
http[s]://hdl.handle.net/* contains a Handle.
hdl:* contains a Handle (but we should not be producing representations with this unregistered scheme).
http[s]://dspace.example.net/handle/* contains a Handle.
http[s]://www.example.net/dspace3/handle/* contains a Handle.
123.4/56 is a Handle.
10.14279/33 is a DOI (which is also a Handle, but IMHO we should not recognize it as one).
As far as I can tell, anything else is not a Handle and does not contain one.
Here's what I would do:

If the identifier matches one of the "contains a Handle" rules, extract the part that should be a Handle.  Otherwise use the whole string.
If what we have now starts with the DOI prefix, return false.
If what we have now matches the rules for raw Handle representations, return true.
Return false.

Thanks to everybody for the clarification and thanks mwoodipui for the suggested solution outline. I've got the first draft of an implementation of that solution (still needs to be cleaned up):
// valid starting patterns for handle
    	String[] validPatterns = {
    			// http[s]://hdl.handle.net/*
    			"^http[s]?://hdl\\.handle\\.net/",
    			// hdl:*
    			"^hdl:",
    			// http[s]://dspace.example.net/handle/*
    			"^http[s]?://[a-zA-Z0-9/.\\-]+/handle/"
    		};

    	// trim out prefix, if found
        String handle = identifier;
        boolean match = false;
        for(String vp : validPatterns) {
        	if(match)
        		break;
        	Pattern pattern = Pattern.compile(vp);
        	Matcher matcher = pattern.matcher(identifier);
        	if(matcher.matches()) {
        		handle = handle.substring(matcher.end());
        	}
        }

        // fail if identifier starts with DOI prefix
        if(handle.startsWith("10."))
        	return false;

        // fail if identifier does not match handle format
        if(! handle.matches("\\d+(\\.\\d+)*/\\d+(\\.\\d+)*"))
        	return false;
        else
        	return true;

I do have a follow-up question before I clean up the code and add it to a commit, though.
@mwoodiupui, @pnbecker, @tdonohue, is there an established standard (from a DSpace perspective) of an acceptable handle value? The solution above assumes that all non-slash, non-dot characters will be numerical, but I also looked at the Handle.net Tech Manual (see Handle Syntax on page 6) which says, "Handles may consist of any printable characters from the Universal Character Set of ISO/IEC 10646, which is the exact character set defined by Unicode." Should DSpace be willing to accept any character from the UCS, or is there an agreed upon subset (such as the numerical assumption above).
Thanks for your help.

@mjmarttila Thanks for keep on working on this!
Regarding the patterns to detect handles, awful things like IDN (https://en.wikipedia.org/wiki/Internationalized_domain_name) came to my mind. => We cannot expect the URL to contain ASCII characters only. Perhaps it would be best to use something like http[s]?://.+/handle/ as third pattern?
We should support all characters from the UCS as the handle tech manual describes. DSpace currently produces numerical postfixes only, but the whole ticket just came up to support some kind of import workflow. Also it is possible that a handle contains more than one slash or perhaps once CNRI starts to use prefixes that contains characters... E.g. originally we didn't took care in DSpace about prefixes containing dots and had to resolve this later. Following this, we have to relax the check for the handle format to either "\d+(\.\d+)*/.+" or even ".+/.+" (ensuring it contains at least a prefix, suffix and a slash).
In general I have the feeling that we are on a good way here! :-)

Quick, and perhaps silly question... Is all of this Handle validation parsing really necessary?  I'm not sure I understand why we couldn't just parse out the DOIs (so they are treated separately), and then treat everything else as "probably a Handle".
I guess I'm not yet convinced all this fancy parsing is necessary. Is there something I'm missing that requires that we fully "validate" that a Handle is a Handle?  If so, why can't we simply check if it's a URL starting with "hdl:", "http://hdl.handle.net/" or whatever is configured in handle.canonical.prefix?

@mjmarttila: I discussed this with @tdonohue and @mwoodiupui in #dspace today. @tdonohue's last comment was related to the last five lines of your method: The form of a naked (=> without resolver or hdl: in front of it) handle is so arbitrary that it doesn't makes sense to check that. So we would suggest to relax the last one of the validPatterns (as described in my comment above), to add a pattern constructed by using handle.canonical.prefix from the configuration and to remove the last five lines. What do you think of those suggestions?

Thanks @tdonohue, @pnbecker, and @mwoodiupui. I'm onboard with removing the last validation as unnecessary due to the fairly broad handle specification.
There is one more caveat that I found in the 11th hour and (briefly brought up on the #dspace IRC) by looking through the stack trace of our original exception scenario: in the case of an API import, the identifier string being supplied to the supports() method originates in the getObjectHandle() method in the AbstractMETSIngester class, which essentially just pulls the OBJID from the METS file and strips out the "hdl:" prefix from the beginning. So the supports method should also be expecting the possibility of a valid handle without any of the other prefixes (e.g. "12345/6789").
I incorporated the suggestions outlined by @pnbecker and tried to reduce some of the regular expression complication to make the method more in-line with its original implementation's simplicity. The last four lines are a sort of high-level validation of handles coming in similarly to the API import stack I mentioned in the above paragraph. Basically, I just wanted to make sure it was meeting a very basic definition of a handle (contains a '/') and wasn't some other URL that did not match one of the known handle patterns (e.g. contained ".../handle/...").
        String prefix = handleService.getPrefix();
        String canonicalPrefix = DSpaceServicesFactory.getInstance().getConfigurationService().getProperty("handle.canonical.prefix");
        if (identifier == null)
        {
            return false;
        }
        // return true if handle has valid starting pattern
        if (identifier.startsWith(prefix)
                || identifier.startsWith(canonicalPrefix)
                || identifier.startsWith("hdl:")
                || identifier.startsWith("info:hdl")
                || identifier.matches("^https?://hdl.handle.net/")
                || identifier.matches("^https?://.+/handle/"))
        {
            return true;
        }
        // return false if identifier starts with DOI prefix
        if(handle.startsWith("10.") || handle.matches("^https?://dx.doi.org") || handle.startsWith("doi:"))
        	return false;
        // return false if identifier does not meet basic handle requirements (has a '/')
        if(! handle.contains("/") || handle.matches("^https?://.+"))
        	return false;
        // otherwise, assumed a valid handle
                return true;

Would this meet your suggestions? Are the last four lines an acceptable way to handle the prefix-stripped handle input? Thanks as always for any feedback.

I've added the code from my last comment into the latest commit. I'd be happy to incorporate any additional suggestions as they arise, though.

@mjmarttila, during a PR review there was a request to provide detailed instructions for someone wishing to test this PR.  Please review/revise these instructions.

Export an AIP from an instance with handle prefix X
Restore that AIP into a DSpace 6 instance with a handle prefix of Y
Result: AIP should be restored and the handle from the original repo should be retained

Thanks, @terrywbrady. I think your instructions look good to me. An example of the original exception that was thrown by the code in the scenario you described is available in the description of the accompanying JIRA ticket for this PR (see https://jira.duraspace.org/browse/DS-3209). With this PR in place, that exception should now only be thrown in the case of a non-recognizable identifier format (e.g. DOI format, non-handle URL, etc.).
If it helps, below is an explicit example of a test case:

An item AIP (with handle 12345.1/123456) is exported from SourceRepo with handle.prefix:12345.1.
That AIP is restored into DestRepo with handle.prefix:12345.2.
The item is successfully restored into DestRepo with its original handle value (12345.1/123456).

Note that since the change is being made to the import code of DS-6.0, I believe that the version of the DSpace instance that the AIP is being exported from shouldn't matter. I've been testing exporting from a v5.4 instance and restoring to a v6.0 instance.
If I can provide any additional clarification, please just let me know. Thanks in advance to any folks who test.

@pnbecker , I see that you are assigned to this.  Would you have time to test this change?

+1 via code review. The code all looks reasonable. This is just waiting on a tester to re-verify (and @pnbecker volunteered to test in today's Dev meeting)

+1, tested locally

Sorry, that I was so unresponsive in the last weeks.
I haven't tested it, but I'm pretty sure, that the code as it is now would return true for something like "http://dx.doi.org/10.14279/depositonce-5383" or even "doi:10.14279/depositonce-5383", which obviously aren't handles but DOIs. The problem is that we cannot assume that everything containing a slash and is not starting with "10." is a handle. And just think how this code would work together with other identifiers (not that we don't have enough, but that never prevented people from inventing new identifier systems).
I think we are putting the cart in front of the horse. Why do we try so hard to accept almost everything as handle? Instead we could change the AIP Import to not remove the information that a handle is a handle. If we keep the good code of this PR and change or even remove the method org.dspace.content.packager.AbstractMETSIngester.decodeHandleURN(String value) the handle recoginition and your use case should work fine.
The code would then look like following:
String prefix = handleService.getPrefix();
        String canonicalPrefix = DSpaceServicesFactory.getInstance().getConfigurationService().getProperty("handle.canonical.prefix");
        if (identifier != null &&
            (
                identifier.startsWith(prefix)
                || identifier.startsWith(canonicalPrefix)
                || identifier.startsWith("hdl:")
                || identifier.startsWith("info:hdl")
                || identifier.matches("^https?://hdl.handle.net/")
                || identifier.matches("^https?://.+/handle/")
            )
        )
        {
            return true;
        }
        return false;

Sorry for having objections and objections but this problem is bigger than it looks on the first wimp.

From talking with @pnbecker on IRC, it sounds like his primary objection (in the comment preceding this one) is that each of these methods return a default value of true.  Returning "true" by default essentially means that it's going to match a lot of URLs that are not handles (or DOIs)... e.g. "http://example.com/" would return true.
Simply setting these methods to a default value of false would both simplify the method (as @pnbecker shows above) and ensure we are not overly "greedy" with our matching of what is a valid handle.
The sidenote by @pnbecker about changing the behavior of AIP import may unravel a larger problem here. The reason the "hdl:" seems to be stripped from the handle is that the handle string ends up being passed to PackageUtils.createObject(), which in turn creates the necessary object (e.g. via CollectionService.create()).  Based on the type of object, it may or may not use the IdentifierService to assign the Handle. Currently, it seems only Items use IdentifierService in ItemInstallService.installItem(), whereas Communities and Collections simply call the HandleService.createHandle() which calls the HandleDAO to insert it directly into the database tables.
So, that previous paragraph is a long way of saying, I think the IdentifierService is not yet fully integrated into every layer of our API.  There are areas that simply insert Handles directly using HandleService (& therefore HandleDAO), especially Collections and Communities, instead of having the "matching" IdentifierService 'mint' a new Handle.  Since AIPs have to work for all types of objects, we end up having to "work around" this problem until IdentifierService fully replaces HandleService.
@mjmarttila and @terrywbrady : do you see any issue with simply changing these methods to "return false;" as the default (instead of true)?  Just curious if that's a reasonable "compromise" for now.

@pnbecker and @tdonohue, I believe that all of the examples you have cited ("http://dx.doi.org/10.14279/depositonce-5383", "doi:10.14279/depositonce-5383", and "http://example.com") appropriately return false with the logic as it stands (see Line 71 for examples one and two, and Line 74 for example three). (Although I see now that the pattern for DOI site matches on Line 71 was a bit off and should be corrected to "^https?://dx\.doi\.org.*" .) The method takes the approach of returning false for known non-DSpace namespaces (the various DOI prefixes, arbitrary URLs, etc.) and then returning true for anything that passes those tests and still lies within the rather large handle namespace. This was meant to accommodate handle.net's very loose definition of a handle in the technical documentation while still avoiding the known non-DSpace namespaces.
That being said, I'm fine going with the more restrictive approach recommended, but would suggest a small addition to allow for multi-instance DSpace deployments that utilize derived handles. The method plus addition would look like the following:
        String prefix = handleService.getPrefix();
        String canonicalPrefix = DSpaceServicesFactory.getInstance().getConfigurationService().getProperty("handle.canonical.prefix");
        if (identifier == null)
        {
            return false;
        }
        // return true if handle has valid starting pattern
        if (identifier.startsWith(prefix)
                || identifier.startsWith(canonicalPrefix)
                || identifier.startsWith("hdl:")
                || identifier.startsWith("info:hdl")
                || identifier.matches("^https?://hdl\\.handle\\.net/")
                || identifier.matches("^https?://.+/handle/"))
        {
            return true;
        }
        // return true if base prefix matches in case of multi-instance deployment with derived prefixes demarcated by a dot "."
        if(prefix.contains(".")) {
        	String[] splitPrefix = prefix.split("\\.");
        	if(splitPrefix.length > 1 && identifier.startsWith(splitPrefix[0])) {
        		return true;
        	}
        }
        return false;

Would this satisfy the objections? If so, I'll add a new commit with the changes.
I agree that the root issue is larger than just the supports() method, but created this PR and accompanying ticket just to try to bring the handle behavior more in-line with its behavior in v5.4. I'm afraid I'm not familiar enough with the inner mechanics of all the affected classes to go making the larger changes that will probably be required to get the object ingest/creation processes where they eventually should be.

I've incorporated the suggestions for a more restrictive supports() method into the latest commit.

@pnbecker : Could you take a look at these latest changes and give your feedback?  To me, they sound more in line with the solution you were looking for.  It'd obviously be good to get this ticket resolved (as it's been a long standing one)

We have continuing uncertainty about the set of recognized forms because we have no unit test suite to prove our expectations.  I would suggest putting the list of test cases in an external file so that it can be easily adjusted without rebuilding.

See https://github.com/mwoodiupui/DSpace/tree/DS-3209 for a sketch of a test suite.  Unfortunately I don't seem to be able to create a PR against the Georgetown repo, or I set mine up wrong.  The .properties resource can be edited as desired to set the single test's expectations.  The test can be run singly:  "mvn test -Dtest=HandleIdentifierProviderTest" for rapid turnaround.
That patch also makes small changes to the ServiceManager so that the test can bypass the Spring configuration and work with the specific flavor of provider that it expects, still autowired by Spring.  In this way we could test all flavors without any reconfiguration of Spring.  I didn't like making unrelated changes deep in dspace-services just for testing, but I couldn't work out a better approach.  The updated code should be equivalent to the original, except that the resulting ApplicationContext can receive new bean definitions later.  If this change is objectionable, we can just use it for testing identifier patterns during development and leave it out of the final merge.

@mwoodiupui , I merged your branch into @mjmarttila 's branch. Thank you for providing this.
Mike is out of the office for a few days.  I will plan to test this locally to confirm functionality.

Ha, shortly after posting my patch, I thought of a less intrusive way to do it.  The branch mentioned above has a new commit which reverses the changes to SpringServiceManager, replacing them with a new child ApplicationContext confined to the test suite.
Ugh, I just noticed that the .properties needs to have colons (such as "http:") escaped.  As I said, a sketch.

Discussed this in IRC with @tdonohue and he pointed out the following config setting: https://github.com/DSpace/DSpace/blob/master/dspace/config/dspace.cfg#L255
I will attempt to use this setting rather than coding for a derived prefix.

I made an update per @tdonohue 's suggestion to use the handle.additional.prefixes config setting.
I have tested this locally and the settings seem to validate properly.

@terrywbrady : looking good. I notice we still should update the Unit Test scenarios in handle-form.properties (noted in this comment: https://github.com/DSpace/DSpace/pull/1399#discussion_r74146331).  That'd just help us prove this is working properly

Latest commit contained only test driver changes

@tdonohue , @mjmarttila , @mwoodiupui , @pnbecker , test cases have been updated and ran successfully in travis.

+1 via code review.  Looks good to me, passes all the new unit test scenarios.

This looks really great now. Using the property handle.additional.prefixes seems to be a perfect solution for me. Thanks for all your hard work on this!
+1 => merging.


Support pyproject.toml for python

Why?
Turns out that https://python-poetry.org and https://github.com/takluyver/flit use this format. It is getting more popular and as it is a nicely structured format it should be pretty easy to support.
Also I got a request for support which is enough to prioritize.
Acceptance Criteria

Support pyproject.toml for loading dependencies as per https://www.python.org/dev/peps/pep-0518/

I have basic support for the base config ie. https://martin-thoma.com/pyproject-toml/ in <IP_ADDRESS>. Next up will be supporting poetry ie. tool.poetry.dependencies

I have 90% of the cases covered with poetry. I will create a new issues to track the remaining edge cases ways to include dependencies. I'd prefer to get this out and circle back for the remaining.


add kefir to cdnjs

it will be great if you add kefir to cdnjs

I've added it to jsdelivr, have no idea when it appears on the actual CDN though :smile:

Ah, it's up actually:
http://cdn.jsdelivr.net/kefir/2.7.1/kefir.js
http://cdn.jsdelivr.net/kefir/2.7.1/kefir.min.js
If anyone wants to add Kefir to cdnjs, please do. I guess it can be done by not an owner of the repo.


Fix Advisory.hash()

The hash is computed from a string, which is built by iterating over attributes of type Iterable/Sequence, such as impacted_package_urls.
This change sorts those attributes before concatenating to the string, to avoid different hashes when the order of impacted_package_urls, etc differs.
NB A better solution would be to avoid implementing __hash__() and use immutable collections like tuple and frozenset for the above mentioned attributes instead.
See https://eng.lyft.com/hashing-and-equality-in-python-2ea8c738fb9d

Works fine, tried running those tests 100 times, passed each time, feel free to merge.


Identify and implement CFF-wide standards to support supply-chain risk management

@pburkholder raised this as a discussion topic during the 2022-05-10 TOC meeting. Capturing some of the discussion context here:

Governmental bodies that use or audit software are increasingly asking OSS projects to prove integrity of software artifacts and to document artifact provenance.

https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-161r1.pdf
https://www.whitehouse.gov/briefing-room/presidential-actions/2021/05/12/executive-order-on-improving-the-nations-cybersecurity/

Software bills of materials (SBOMs) and initiatives such as SLSA.dev can help in these areas
While CFF processes genrate some of this SBOM information and can guarantee integrity of some artifacts, these processes are not consistent across CFF artifacts and not complete.
Paketo and kpack already generate SBOMs and annotate them on the images they build.

The TOC and the Working Groups should decide how far we would like the projects to go in supporting these emerging supply-chain standards, which standards to implement, and then assign responsibility to one or more bodies within the CFF to carry out the work.

Thanks for the succinct issue description, @emalm

This is still revenant and the TOC plans to look into this.