[REQUEST] Option to keep stalled torrents.

Hey, touching on my questions from earlier, thought I'd make some suggestions here.
I don't know if you like 1 post per topic. Sorry if it seems like I'm spamming!
So this suggestion is:

Option to NOT delete stalled torrents. Only add them.

Sort of like there's an option to never delete seeding torrents, have the same option for torrents when searching for new ones to replace.
It might look like this:
[Radarr-1080.Torrent]
AutoDelete = false
IgnoreTorrentsYoungerThan = 86400
MaximumETA = -1
MaximumDeletablePercentage = 0.75
DoNotRemoveSlow = true

KeepStalled = true

And the behavior would be:

qBitrr sees a stalled torrent.
Adds that to the blocklist as normal.
Searches for a new release as normal.
Keeps the stalled torrent as well.

This way, the old one has a chance to complete one day, but you still have qBitrr looking for new ones.
Some uses cases... sometimes there will be a torrent that has stalled for a week or two, but by the grace of the torrent gods, someone will connect with the rest.
I see there's the qbitrr-ignored tag, but that's not really the same thing I believe.
That just causes qBitrr to ignore that one being stalled if I'm not mistaken.
This would be super helpful for torrent from private trackers that are stalled.
If one is stalled, I don't want it delete because it'll probably give me a hit and run.
But still want qBitrr to look for a different release.
No delete. Just add! :P'
OK thank you!

This is doable, shouldn't be terrible to implement. At first I'll implement this using a tag, proceeding from there I'll make the changes for the no tagging version.
This change will also need to adjust the search behaviour as currently qBitrr only searches for items that aren't in the queue for Radarr/Sonarr

Dang dude you're awesome. Check your Patreon messages, by the way.

This specific issue should be resolved currently in docker tag pr-86. Give it a try and let me know how it goes

Great and thank you.
I tested it out with each of the following:

AllowedStalled = true

and

StalledDelay = 0
StalledDelay = -1
StalledDelay = 86400

Looks like it's just ignoring the new settings outright.
It's still going in and removing the stalled torrent and re-searching for a new one.
According to the logs, it's qBitrr v4.7.5-7c72ac4
Although I think that version number is the same as it was yesterday.
When I have it create the config.rename_me, it does in fact have the new AllowedStalled and StalledDelay fields.
Hope this helps!

The version number for now will remain the same, only the hash will change, until the change is completed. Can you set logs to trace and send over a log?

If not tonight, tomorrow, I'll be doing some testing myself too

Whoops shoulda had that for you already.
Here are the logs: qBitrr Logs
Also included my config.toml in there.

The reason for grabbing the same torrent is due to use of multiple trackers which would have the same torrent. For Radarr/Sonarr each torrent on each tracker is different, so this happens a lot

Also, let the trace logs run for 15-30 mins before sending them over. There's no info to work with in the logs you sent me

Also, let the trace logs run for 15-30 mins before sending them over. There's no info to work with in the logs you sent me

OK let it run for about an hour: qBitrr Logs
SHould show it replacing Lionheart every 30 minutes.

I went over everything and it just dawned on me. Can you test again tagging the torrent with the new tag qBitrr-allowed_stalled? Once it's working with the tag I can get to completing work on the tagless update

Made some additional changes, kindly update your container and test again please

If you can send me the magnet link to the Lionheart torrent so I can test please

If you can send me the magnet link to the Lionheart torrent so I can test please

OK all done. Same link: qBitrr Logs
Magnet is in there too.

I assigned the tag in qBittorrent.
I forced updated the container under the same pr-86 tag.
Now it isn't doing anything with Lionheart.
The log show that it seems like it's trying to grab a couple of random movies that already exist. Such as Bambi, The Thing, Metroplois, and a few others since zipping this log up. But I don't think it's actually performing those actions. (I see nothing being added to qbittorrent and the progress text doesn't show up in Radarr.)
I don't remember it sleeping for only 5 seconds each time but maybe I'm wrong on that.

Hope it helps!

I noticed an issue in the logs you sent, I'll fix it and hopefully the logic will be functional

Update to the latest version, I think it is now resolved and functional. Once confirmed I'll push the official update and continue work on the tagless update

Though remind me, you wanted a search to still occur if the torrent is stalled right?

Though remind me, you wanted a search to still occur if the torrent is stalled right?

OK. I did the same update. Forced updated the pr-86 tag.
Unfortunately it still is having issues.
This time I tried three scenarios so there are three sets of logs: qBitrr Logs
I also threw another movie in the mix - Anchorman. What I did was add a torrent I knew would be stalled for it. (no seeders).
Here's what happened:
logs_minus_one_no_tag: StalledDelay set to -1. Did not assign tag.

No change. Still continues to remove stalled torrent, adds it to blocklist, and re-search new one (which ends up being the same exact torrent even though it's on the blocklist).

logs_zero_no_tag: StalledDelay set to 0. Did not assign tag.

Same results.

logs_zero_using_tags: StalledDelay set to 0. Manually assigned qBitrr-allowed_stalled tag.

Removes stalled torrent, adds it to blocklist, re-searches, and the new one does NOT get auto-assigned any tags.

Also, with Anchorman, I noticed it will remove the stalled torrent, add it to blocklist, and re-add a new torrent that has a bunch of seeders and actively downloads (desired behavior). Then if I pause it, it'll get removed on the next loops (which I set to 5 minutes).
One more thing, each time I tested, I removed all the blocklist entries as well. Oh and I let them run for a while. At least 30 minutes each time I think.
OK that's it for now!

I just had another movie downloading, had plenty of seeds and was actively downloading.
Just on a hunch, I ran qBitrr and sure enough, it removed that torrent, blocklisted it, and re-searched it.
And I forgot to answer you...
Yeah what I was looking for originally is for it to go ahead and blocklist and re-search stalled torrents.
Just not remove them from qbittorrent. Allowing the stalled torrent and newly searched replacement torrent to run side by side.

Noted, have you tried with a price value for a delay? Say 60 (minutes)?

Otherwise I think I can have this ready by Sunday at the most (maybe Monday)

Noted, have you tried with a price value for a delay? Say 60 (minutes)?

I just tried with 3600 (1 hour) and it just did the same thing. Removed torrent despite (ignoring the `StalledDelay = 3600'), blocklist, re-search (adding the same exact torrent, ignoring Radarr's blocklist).
The odd thing about the blocklist situation is it seems like this is the only movie it's ignoring the blocklist for.
I can tell it's re-adding the same one because it's the same hash that it's blocking.
I'll do some more testing on that front, but seems like a Radarr issue.
If you ever want to remote into my server and see what it's doing in realtime, just let me know here or Discord (DevanteWeary).

Noted, I've made some changes and added more logs. I'm also testing this functionality with the Lionheart torrent you provided and things seem to be working correctly now. Please update and let me know, trace logs are now clearer on what the new functionality is doing

Also the delay is in minutes, so 3600 would be equal to 60 hours. I've updated the config description

Noted, I've made some changes and added more logs. I'm also testing this functionality with the Lionheart torrent you provided and things seem to be working correctly now. Please update and let me know, trace logs are now clearer on what the new functionality is doing

OK updated the pr_86 again and still didn't work.

Logs with no tagging: qBitrr Logs
Logs where I tagged it. Not all the loops but the last loop for sure: qBitrr Logs

Also note how the last time Lionheart we removed and re-searched, it was at 54% so was actively downloading. It stalls at about 98ish%.
Oh and I changed the StalledDelay to 60.

Actually man... I was thinking about it... it DID work in that it removed the Lionheart and re-searched it.
It just didn't work in that I think it ignored the delay (it's supposed to ignore anything below the delay, right?)
And also it removed Lionheart even though it was actively downloading at one point.
I also tested out some other movies: Bad Moms and Wayne's World.

It removed the stalled ones.
It also removed the PAUSED ones.

Then re-searched those as well.
And finally, I tagged them in this test.
More qBitrr Logs

So to sum up, the current issues are:

It's removing paused torrents.
It's removing active downloads.
It's ignoring StalledDelay (if I understand correctly what it's supposed to do)
In the case of Lionheart, it's re-searching the exact same torrent (and hash) even though it gets blacklisted.

That's it!

Ok so going through your logs I can deduce the following:
The stalled delay is working, I wasn't clear on how it works. It checks for the time delay from the time the torrent was added and when looking at your logs, the ones removed had been added more than 60 minutes prior to the run, so that logic is functional.
What's definitely wrong is the processing of paused torrents, that I'll need to adjust for sure

That's what I understood. In other words, if a torrent is only 57 minutes old, it will not be removed/replaced IF stalledDelay is set to 60.
I haven't looked at those logs in particular but I can tell you it's ignoring the 60 minutes rule for sure.
For example, it's just replacing the Lionheart every loop (5 minutes - ultimately I'll set it to 60 minutes).
And remember, it was ignoring the setting even when I had it set to 60 hours too.
Sorry man I know you've been working hard on it.
And like I said, I'm open to you remoting in and taking a look for yourself firsthand.
I did one quick test just now.
I had Lionheart in there since 9pm last night.
I just now turned on qBitrr and it immediately removed Lionheart then re-searched and added it back.
Then after the 5 minute loop, it removed and re-searched Lionheart again. And I noticed Lionheart wasn't even stalled yet. It was still actively downloading.
Here's a screenshot of a second or two before it removed Lionheart, and then a screenshot of the log right after.
Right before:

And right after:

And of course the actual logs.
This is a brand new set of logs cleared before the test: qBitrr Logs - Quick Test

Ok, think I know what's going on, as soon as I get the chance I'll make some adjustments

I just had a look, and I think you need to check your localisation (time and date). Ensure that both qbittorrent and qBitrr have the same time/timezone applied as per your logs the delay is functional:
Stalled check: Lionheart.1990.1080p.BluRay.x265.HEVC.10bit.2ch.(xxxpav69) [Current:2024-06-30 04:12:36.255702][Added:2024-06-29 21:44:57][Limit:2024-06-29 22:44:57]
I also pushed an update which should fix the processing of paused, uploading torrents too

OK got some results...
Completed uninstalled / reinstalled qBitrr. Reused my config.
Ensured the time was correct by 1) adding the TZ variable to the container settings and 2) running the date command in the CLI of Unraid, qbittorrent, and qBitrr to make sure they match.
1st run - 60 delay setting - qBitrr Logs

qBitrr removed the Lionheart that was already in there as expected.
I paused the download to see if it would remove a paused torrent.
It removed it.

2nd run - 480 delay setting just to test - qBitrr Logs - 480 delay

qBitrr removed the current Lionheart (it shouldn't have).
I tagged the replacement with stalled_allowed
I watched as the 5min loop came around and qBitrr removed Lionheart even though it wasn't stalled but rather mid-download.

By the way, are we sure there's nothing else in my config that could be causing it?

Took a look at your config file and logs and noticed a pattern with the File extension allow list. Try it this way and let me know if anything is different: config.zip

I've made adjustments which should allow an empty File extension allow list and upon further testing, on my end the logic is working flawlessly. I'll wait till I get feedback from your end before releasing though

So a quick update, just added logic for re-searching, untested yet and some config changes as seen below:

OK done testing!
Seems to be doing a lot better on my side.
After testing it out, it's almost perfect.
The only thing I see now is that it is NOT adding to blocklist unless it removes the stalled torrent.
It still searches, just doesn't block.
Check out the Delay settings part below to see what I mean.
Also, since you made some adjustments after posting your config, I didn't use it so let me know if you still want me to.

Logs (in order of testing)

qBitrr Logs
qBitrr Logs - 60min Delay
qBitrr Logs - 20min Delay

Notes

Active download and paused torrents are NOT getting removed.

Seems to be honoring the StalledDelay setting.

BEFORE the StalledDelay time is over: does NOT remove Lionheart ❌. Initiates a search ✔. Does NOT add to blocklist ❌.

AFTER the StalledDelay time: removes Lionheart ✔. Initiates a search ✔. Adds to blocklist ✔.

It's forcing a resume on paused torrents. (sometimes I keep torrents paused for a reason!)

At one point it said Lionheart was stalled and added the tag, though it was actively downloading.

Delay settings:

-1 Delay: Removed stalled torrent immediately, re-searched, blocked.
0 Delay: Never removes stalled torrent, re-searched, NOT blocked.
60 Delay: Acts like 0 delay before 60 minutes, acts like -1 after 60 minutes.

I also thought of something not related to the above that might complicate things... sorry!
Let's consider stalled torrents left in qBittorrent.
1st pass: It detects stalled torrent #1 and blocklists and re-searches. This search adds torrent #2. Torrent #1 is kept in qBittorrent like we want.
2nd pass: It now detects stalled torrent #1 and completed torrent #2.
Will it see torrent #1 as stalled again? And perform another blocklist and re-search?
And just forever block and re-search on every looped as long as the stalled torrent is in there?
I'm thinking some kind of list/db that says "I already processed this stalled one, ignore it from now on."
Or did I just type that for nothing and that's what the "Already been cleaned up" entry in the log means? ha

Excellent news overall, glad to hear. To address your concerns:

To be clear, the functionality you're looking for is for the lionheart to be removed from the Radarr queue and block listed but not removed from qbittorrent as soon as it stalls, then performs a search  as usual?

Could you specify what reason you have for pausing torrents? As overall all the way qBitrr works, it only pauses torrents if there isn't enough free space or if the torrent has reached it's seeding/ratio limit and it's not set to remove, otherwise it tries to keep torrents active. Maybe this could be another feature?

Regarding your looping concern, every time qBitrr starts it processes all torrents and those it's cleaned up shouldn't be retouched. This specific scenario might need a bit more testing to see what the current behaviour will be. At most I already have a potential fix in mind

And lastly to clarify, a torrent is detected as stalled when it either stalls or it detects that the availability is below that configured, though it does the check every 5 seconds and untags it if all is clear so no worries

To be clear, the functionality you're looking for is for the lionheart to be removed from the Radarr queue and block listed but not removed from qbittorrent as soon as it stalls, then performs a search  as usual?

Yep you got it!

Do NOT removed stalled torrent from qBittorrent. Keep it forever.
Add stalled torrent to blocklist in Radarr.
Re-search for new torrent.

Could you specify what reason you have for pausing torrents? As overall all the way qBitrr works, it only pauses torrents if there isn't enough free space or if the torrent has reached it's seeding/ratio limit and it's not set to remove, otherwise it tries to keep torrents active. Maybe this could be another feature?

Oh man, there are various reasons.
I'll give you an example of what I have currently.
I have 5 torrents for Looney Tunes that vary in sizes from 150GB to 500GB.
It was easy for me to add them as I was searching and then pause them and try one at a time so I didn't download all that space unnecessarily. Kind of like having them as a backup in case the others don't work.
I have another one where I added the torrent, then found another torrent of better quality but it's very slow (taking a week so far) with low seeders.
So I'm keeping the first torrent paused in case the second one loses all its seeders and never comes back.

Hope this clears things up a bit

Let me just say again that you're awesome man.
Thank you for working on everything so hard.

Yep you got it!\n\nDo NOT removed stalled torrent from qBittorrent. Keep it forever.\nAdd stalled torrent to blocklist in Radarr.\nRe-search for new torrent.

Ok I can fix this, I actually added logic to not cause a blocklist until the delay is finished thinking it would be preferred.

Oh man, there are various reasons.\nI'll give you an example of what I have currently.\nI have 5 torrents for Looney Tunes that vary in sizes from 150GB to 500GB.\nIt was easy for me to add them as I was searching and then pause them and try one at a time so I didn't download all that space unnecessarily. Kind of like having them as a backup in case the others don't work.\n\nI have another one where I added the torrent, then found another torrent of better quality but it's very slow (taking a week so far) with low seeders.\nSo I'm keeping the first torrent paused in case the second one loses all its seeders and never comes back.

This seems to me to be very manual and there's no real way to handle these outside of tagging them to be ignored using the tag qBitrr-ignored

Let me just say again that you're awesome man.\nThank you for working on everything so hard.

You're very welcome, thank you for the support and it's a pleasure seeing this tool and any changes I add to it being used

Oh man, there are various reasons.\nI'll give you an example of what I have currently.\nI have 5 torrents for Looney Tunes that vary in sizes from 150GB to 500GB.\nIt was easy for me to add them as I was searching and then pause them and try one at a time so I didn't download all that space unnecessarily. Kind of like having them as a backup in case the others don't work.\n\nI have another one where I added the torrent, then found another torrent of better quality but it's very slow (taking a week so far) with low seeders.\nSo I'm keeping the first torrent paused in case the second one loses all its seeders and never comes back.

This seems to me to be very manual and there's no real way to handle these outside of tagging them to be ignored using the tag qBitrr-ignored

Do you think something like a general PauseWhenLow = true and/or PauseResume = false would work?
If being honest, I don't think qBitrr needs to be starting, stopping, resuming, or even checking for free space.
There are so many other spots that check for free space.
For instance, Radarr/Sonarr checks for free space and most server OSes (like Unraid in my case) checks for free space.
Just throwing that out there!

PauseWhenLow how exactly? What parameters, that's the real issue. And the whole point of qBitrr is automating torrent management for Radarr/Sonarr.
Also although the Arrs do have logic in place for free space, they only account for importing not downloading and in my case for example, I have my downloads going to an SSD for faster downloads and general processing before the files are moved to HDDs for plex to read off of, which means that the Arr apps have no idea what storage capacity exists for the downloads folder.
I could add an option to disallow pausing and resuming, rather simple to do actually, but you will lose a large part (in my opinion) of the overall functionality that qBitrr provides.
If PauseResume = false is something you'd be interested in, let me know and I'll get it sorted, got no problem with that

Also although the Arrs do have logic in place for free space, they only account for importing not downloading and in my case for example, I have my downloads going to an SSD for faster downloads and general processing before the files are moved to HDDs for plex to read off of, which means that the Arr apps have no idea what storage capacity exists for the downloads folder.

Actually yeah this makes sense to me.

I could add an option to disallow pausing and resuming, rather simple to do actually, but you will lose a large part (in my opinion) of the overall functionality that qBitrr provides.

When you say large part, what are we talking here?

If PauseResume = false is something you'd be interested in, let me know and I'll get it sorted, got no problem with that

Yeah man that'd be great. I personally am not worried about any kind of space issues but can see how others would be for sure.

When you say large part, what are we talking here?

Mainly to do with ensuring seeding as configured, and overall automation, I guess it's larger for me and some others than it may be for you to be honest.
Yeah man that'd be great. I personally am not worried about any kind of space issues but can see how others would be for sure.

No worries, I'll get these changes sorted tomorrow I think

I have updated with the agreed changes. The new config is under the top most Settings section as seen below:

Let me know if things are working as intended

I have updated with the agreed changes. The new config is under the top most Settings section as seen below:
Let me know if things are working as intended

Hey sorry was busy all day yesterday and today.
I got to check it out just a little bit today and long story short, looks like it's back to removing stalled torrents.
But let me get home tonight and give it a full test and get back to you!

That's ok, might I suggest enabling and disabling the re-search functionality to see if the removing of the torrents is due to that? On my end I think that's what's causing it

I'm going to go ahead and release the changes as is, but I'm not closing this issue quite yet till I see the re-searching is working as intended

I'm going to go ahead and release the changes as is, but I'm not closing this issue quite yet till I see the re-searching is working as intended

Yeah sorry about that. Was super busy yesterday and then today my server was giving me all kinds of problems and been working on that most of the day. Tomorrow morning I'll 100% be able to test out those changes and report back!

No worries, update to the latest release (4.8.0) first

No worries, update to the latest release (4.8.0) first

OK, was able to update and test.
I ran six tests, three 5-minute loops each.
Delay 0, Delay -1, and Delay 1440.
Tagged and untagged each.
tldr;

it's removing stalled torrents every test.
It's ignoring the StalledDelay setting.
unless StalledDelay is set to 0, in which case it seems like it's crashing.
It always removes tags during the tagged tests.
CT (see below) seems to be only one that says it was checked before.

LH = Lionheart. In general, 0 seeders, always gets to 94.8% then stalls.
HD = Holidate. 0 seeders, gets to around 8% then stalls, though usually didn't get there during the tests.
CT = Conspiracy Theory. Seems to have a couple seeders. Never really stalls. Slow download.
I tried to get the status of the torrent right before it processed in (parenthesis).
Here are my findings.

--  [ Delay 0 - No Tag ]  --  [qBitrr Logs](https://devante.rocks/share/qbitrr_logs_delay0_untagged.zip)
Loop1: 	LH (stalled): tagged, no block, no search. error in log.
		HD (DLing): skipped
		CT (stalled): not processed due to LH error?
Loop2: 	LH (stalled, tagged): not processed due to HD error?
		HD (DLing): tagged as stalled even though DLing. Errors in log.
		CT (stalled): not processed due to HD error?
Loop3: 	LH (stalled, tagged): Seens as stalled, nothing done.
		HD (DLing): Seen as stalled. Skipping file check.
		CT (stalled): Tagged as stalled. Errors in log.

	Summary: 	Errors/crashes affecting further checks.
				No actual blocking and searching being done.

--  [ Delay 0 - No Tag ]  -- No Log
Didn't test due to crashes above.

--  [ Delay -1 - No Tag ]  --  [qBitrr Logs](https://devante.rocks/share/qbitrr_logs_delay-1_untagged.zip)
Loop1: 	LH (stalled): Says not stalled. Removing stale. Removed, blocked, searched.
		HD (DLing): skipped
		CT (stalled): Says not stalled
Loop2: 	LH (stalled): Skipped
		HD (DLing): Says not stalled. Removing stale. Removed, blocked, searched.
		CT (stalled): Skipped
Loop3: 	LH (stalled): Says not stalled. Removing stale. Removed, blocked, searched.
		HD (DLing): Says not stalled. Removing stale. Removed, blocked, searched..
		CT (DLing): Skipped.

	Summary: 	Now it's removing as "stale?" Even when stalled.
				No auto-tagging done.

--  [ Delay -1 - Tagged ]  --  [qBitrr Logs](https://devante.rocks/share/qbitrr_logs_delay-1_tagged.zip)
Loop1: 	LH (stalled): Says not stalled. Untagged. Removing stale. Removed, blocked, searched.
		HD (DLing): skipped, untagged
		CT (DLing): skipped, untagged
Loop2: 	LH (DLing): Untagged.
		HD (DLing): Says not stalled. Untagged. Removing stale. Removed, blocked, searched.
		CT (DLing): Skipping, already cleaned up. Untagged.
Loop3: 	LH (stalled): Says not stalled. Untagged. Removing stale. Removed, blocked, searched.
		HD (DLing): Says not stalled. Untagged.
		CT (DLing): Skipping, already cleaned up. Untagged.

	Summary: 	Now it's removing as "stale?" Even when stalled.
				Keeps removing tag.

--  [ Delay 1440 - No Tag ]  --  [qBitrr Logs](https://devante.rocks/share/qbitrr_logs_delay1440_untagged.zip)
Loop1: 	LH (stalled): Stale torrent: stalled? Removed, blocked, re-searched.
		HD (DLing): Not stalled.
		CT (DLing): Not stalled.
Loop2: 	LH (DLing): Not stalled.
		HD (DLing): Remove stale: unavailable?
		CT (DLing): Not stalled. Already checked.
Loop2: 	LH (stalled): Removed stale: stalled?
		HD (DLing): Not stalled.
		CT (DLing): Not stalled. Already checked.

	Summary: 	Says delayed even though it has only been a few minutes. Ignoring StalledDelay?

--  [ Delay 1440 - Tagged ]  --  [qBitrr Logs](https://devante.rocks/share/qbitrr_logs_delay1440_tagged.zip)
Loop1: 	LH (stalled): Stale torrent: stalled? Removed, blocked, re-searched.
		HD (DLing): Stale torrent: stalled? Removed, blocked, re-searched.
		CT (DLing): Not stalled. Untagged.
Loop2: 	LH (DLing): Not stalled. Untagged.
		HD (DLing): Not stalled. Untagged.
		CT (DLing): Not stalled. Untagged. Says already checked.
Loop3: 	LH (stalled): Stale torrent: stalled? Removed, blocked, re-searched.
		HD (?): Stale torrent: stalled? Removed, blocked, re-searched.
		CT (DLing): Not stalled. Untagged. Says already checked.

	Summary: 	Says already checked for CT only in this and other loops.
				Removes tags. Ignores StalledDelay.

Note: I don't see anywhere to confirm version. Also, no timestamps in logs.

Can you send me your config again please to check a few things? And logs too. Also sorry for the delayed reply, had a hectic weekend

Ignore the previous message, I found the logs. I made some changes (logging updates too) and the crashing error should be fixed, take another look please. Also delay -1 seems to be working correctly. I do plan on seeing how to add timestamps at some point but it hasn't been an issue given that actions which are time dependant are logged appropriately.

Ignore the previous message, I found the logs. I made some changes (logging updates too) and the crashing error should be fixed, take another look please. Also delay -1 seems to be working correctly. I do plan on seeing how to add timestamps at some point but it hasn't been an issue given that actions which are time dependant are logged appropriately. use feramance/qbitrr:nightly

I included logs and my config in each archive.
OK, updated to nightly and here's what I got.
tldr;

Paused untagged are removed. Mentioned because status is PAUSED_DOWNLOAD when its removed.
Tagged are NOT removed, but also not blocklisted/re-searched.
StalledDelay seems to be taken in account correctly.

-- [ Delay 0 - Untagged ] -- qBitrr Logs
Loop 1 - LH (stalled): Removed, blocked, re-searched.
Loop 2 - LH (stalled): Removed, blocked, re-searched.
Loop 3 - LH (stalled): Removed, blocked, re-searched.
Loop 4 - LH (paused): Not removed.
Loop 5 - LH (paused): Removed, blocked, re-searched.
Loop 6 - LH (paused): Removed, blocked, re-searched.
Loop 7 - LH (paused): Removed, blocked, re-searched.
-- [ Delay 0 - Tagged ] --  qBitrr Logs
Loop 1 - LH (stalled): Untouched and not blocked.re-searched.
Loop 2 - LH (stalled): Untouched and not blocked.re-searcheD
Loop 3 - LH (stalled): Untouched and not blocked.re-searcheD
-- [ Delay 1440 - Mixed ] --  qBitrr Logs
Loop 1 - LH (stalled, tagged): Removed, blocked, re-searched.
Loop 2 - LH (dling, untagged): Untouched and not blocked/re-searched.
Loop 3 - LH (paused, untagged): Untouched and not blocked/re-searched.
Loop 3 - LH (paused, tagged): Untouched and not blocked/re-searched.

Ok so I reopened a PR to handle things better for updates so use feramance/qbitrr:pr-84 again please goiung forward.
Overall the only incorrect behaviour I found was the paused download status handling. If you manually tag a torrent the logic won't work correctly as the torrent is blocked and researched only when the tag is added, not if it already there (to avoid blocklisting and researching on every loop run). Otherwise, besides the paused download, it seems to me that the expected behaviour was correct, please correct me if I am wrong.

Ok so I reopened a PR to handle things better for updates so use feramance/qbitrr:pr-84 again please goiung forward.
Overall the only incorrect behaviour I found was the paused download status handling. If you manually tag a torrent the logic won't work correctly as the torrent is blocked and researched only when the tag is added, not if it already there (to avoid blocklisting and researching on every loop run). Otherwise, besides the paused download, it seems to me that the expected behaviour was correct, please correct me if I am wrong.

Hey just to make sure, do you want me to use pr-84 or is it pr-86?
About the manual tagging... OK I actually didn't understand that before so thank you!
So in that case, the one I labeled [ Delay 0 - Untagged ] above is the most valid test, right?
If so, the issue it seems to be having goes back to the original intent of this feature request: it's removing stalled torrents from qBittorrent instead of keeping them.
Desired behavior:

✔ Block torrent in Radarr/Sonarr.
✔ Re-search for a new one.
✔ Do NOT remove stalled torrent from qBittorrent. Keep it in there.

Current behavior:

✔ Block torrent in Radarr/Sonarr.
✔ Re-search for a new one.
❌ qBitrr removes the torrent from qBittorrent.

Both are wrong actually, it's feramance/qBitrr:pr-95

Regarding the removing of the torrent from qbittorrent when blocking, I'll investigate further, need to consult the Arr APIs

Hey, came across this: line 43
Sorry if this is nothing, just saw it and wanted to throw it out there!

After looking into pyarr (the arr api used by qBitrr) further, it seems that the delete queue isn't working as intended. After some manual tested I found a solution and have implemented it. Kindly update to the latest pr-95 build and let me know how it goes.

Current behavior:

✔ Block torrent in Radarr/Sonarr.
✔ Re-search for a new one.
❌ qBitrr removes the torrent from qBittorrent.

Desired behavior:

✔ Block torrent in Radarr/Sonarr.
✔ Re-search for a new one.
✔ Do NOT remove stalled torrent from qBittorrent. Keep it in there.

I think this last update should meet the desired behaviour, as such I have started work on the TAGLESS update and made a good amount of progress already

I think this last update should meet the desired behaviour, as such I have started work on the TAGLESS update and made a good amount of progress already

Thank you so much for sticking with it!!! You're amazing man.
OK some good news and bad news.
Good news is it worked the first time, but bad news is after that, it stopped working and also did a couple things it didn't do before, like say it was blocking/researching but didn't actually do it.

-- [ Delay 0 ] <-- Worked perfectly!!! - qBitrr Logs
Loop 1 - LH (#1)(stalled): tagged as allowed, NOT removed, blocklisted, re-searched, new torrent added
Loop 2 - LH (#1)(stalled): already checked
Loop 2 - LH (#1)(downloading): not stalled
Loop ? - LH (#1)(stalled): already checked
- LH (#2)(uploading): tagged as imported
-- [ Delay 1440 ] - qBitrr Logs
Ignored delay, removed torrent, but did not block and did not re-search
Log says deleted stale torrent: stalled state
-- [ Delay 0 - IgnoreYoungerThan 1200 ] - qBitrr Logs
After 10min, just skipped
-- [ Delay 0 - Added torrent after launching qBitrr ] - qBitrr Logs
qBitrr tagged the torrent, then didn't do anything with it.
-- [ Delay 0 - Added torrent before launching qBitrr ] - qBitrr Logs
Loop 1 - LH (#1): Skipping file check
Loop 2 - LH (#1)(stalled, manually removed tag): qBitrr added tag, did not block, did not research
Loop 3 - LH (#1)(stalled, still using tag from loop 2): skipped again
-- [ Delay 0 - I Forced Update of qBitrr ] - qBitrr Logs
Loop 1 - LH (#1)(stalled): tagged, said it was blocking and researching but didnt actually do it in Radarr
Loop 2 - LH (#1): skipped
aaaaaand I just tried a test... nothing special just wanted to since it's now hours later from those previous test...
-- [ Delay 0  ] - qBitrr Logs
Loop 1 - LH (#1)(stalled): tagged, said it was blocking and re-searching but didn't actually do it in Radarr
Loop 2 - LH (#1): skipped

Ok so after checking all the logs and logic, I determined that I think the test with the 1440 delay needed a restart or needed the update to work as intended (I have added logs to log the stalled delay value for this reason). In addition when having a 0 delay the skipping is intended if its already been processed or has no changes required, so that's normal (skipping means 'let it be' essentially.
Lastly, the only real issue I saw was the not blocking and searching, I have made a small adjustment  with added logs to hopefully fix the issue and if not at least better determine what's going on. So please UPDATE first, then test... Fingers crossed

OK I wanna saaaay.... looking good so far!
Lionheart worked perfectly.
I went ahead and unleashed it on my series and that seems to be going great too.
I don't know man, I wanna say delay0 is a success!
As for an actual delay, I tested StalledDelay = 20 and...

Tagged torrent but did not do anything else.
After the 20 minutes was over, it removed the torrent and did NOT blocklist/re-search. (should not have removed)
Log said it was removing stale torrent.
qBitrr Logs

But for the intent of keeping stalled torrents while searching for new ones... looks like you got it working!!
I wanted to ask... so I just assumed by removing the qBitManager folder, I was "starting fresh" i.e. removing the flag that said a torrent had already been check. But I don't think that's the case at all, is it?

Very happy to hear, and regarding the researching, it'll only work if qBitrr itself tags the torrent, not if you do. You should never need to tag anything except for the ignore tag, but I'll take a look tomorrow before merging and getting down to having a function tagless version.
Regarding your question about deleting the folder, the way qBitrr works, it doesn't really care about the folder, it rebuilds all the data it needs with every database update to keep up to date with the Arr apps. In short, restarting it is enough to literally start fresh with whatever configs you have set in your config.toml

Also to clarify, stalled delay 20 means remove after 20 minutes from torrent added time of stalled. Some scenarios:

if torrent is 24 mind old when it stalled, then the 20 mins limit is already exceeded and is removed and re searched
if torrent is 10 mins old then torrent is researched and allowed to remain for an additional 10 minutes before being removed

Re search happens when the torrent is tagged as stalled only, or when removed if it exists in the Arr app's queue

Oh snap OK no I was totally mistaken.
I thought it was a "don't even re-search, block, even LOOK at this torrent until it's at least 20 minutes old."
Well that's a cool option then.
You've basically let us keep stalled ones in qbittorrent but also given us a way to remove them after a certain amount of time.
OK OK sorry one more thing. In qBittorrent, I don't think the status "Downloading metadata" is considered stalled even if it's been that way for a while. I see that those torrents show up in the Downloading filter and not the Stalled filter, so assuming that is relevant?

I forgot the exact tile bit after a while if a torrent is still stuck on downloading metadata, it should be detected as stalled yes, if not I'll amend it

So just to clarify regarding the Downloading Metadata, the way it's handled is if the download is younger than the config IgnoreTorrentsYoungerThan then it ignores it. If it exceeds the time configured and it's still Downloading Metadata, then its marked as stalled.

Just a follow up.
This function has been working like a charm!

Very happy to hear. Sorry on the delay for the tagless, been very busy lately and had to resolve some issues

Trying to use this feature to prevent H&R but it seems to be deleting the stalled torrents right away. Am I doing something wrong?
ReSearchStalled = true
StalledDelay = 11520 (required time to prevent H&R for tracker.

Did you restart qBitrr after changing these settings? And if so, please attach trace logs with this enabled. Thanks

Closing due to no response


"--device" option when deploy container in web UI

Hello everyone.
Thanks in advance!

@DeanChan We don't support this via the UI at the moment, should be reasonably easy for someone to add though, following the same code that that is used for volumes/links on the deploy screen.

Thank you @tombee . I'll close this issue now.


Temporarily disable golde-test

The golden test is still running as part of build_and_push_docker_image circleci task.
This presubmit test uses googleapis/artman:latest docker image. CircleCI doesn't pull the latest artman image every time it runs, causing misleading result.

It ended up to be a gapic config problem. The gapic yaml used in gapic is using the old style.


Internationalization (i18n) / multilingualism for some text fields ?

Hello.
All text fields included in the spec are provided on a single-language basis.
As often, English is the most widely used language, in the spec or examples found elsewhere.
It is usually not a problem when dealing with APIs, since these are seen only by developers or machines.
However, with MCP, I understand some texts (prompts, descriptions...) might be read / selected / typed directly by end users.
Are there any plans to provide an internationalization (i18n) / multilingualism mechanism in the spec?
Or shall we rely on the LLM to handle the (possibly not totally accurate) translations?
Thanks!

Would you be interested in drafting a proposal PR?

I'd love to.
However, this would be a somehow ambitious PR, with quite a few design choices to be made beforehand.
Do you really think this is a must-have feature? If so, I'll gladly give it some work.
I'll follow up with some more targeted questions.

What would be the most appropriate standard to encode language codes ? ISO 693-1 ? ISO 693-2 ? IETF BCP 47 ? Any other ?

Should we provide backward compatibility? If so, any advice on how to achieve this here?

In terms of workflow, where should language selection occur?

Server-side: The client adds an optional language code parameter to its requests and the server returns only the appropriate language, if available, or its default language.

This might not make sense if notifications sent by server must also be i18n-ized
More complex servers
Reduced load on network

Client-side: The server returns all available languages and the clients selects its preferred one.

Moxe complex clients
Heavier load on network

These options would have pretty different impacts on the spec.

The main things intended to be human-readable are names (e.g., of resources and prompts) and descriptions [...]

Names are used as identifiers. I fear we might end up having problems by making them multilingual, for prompts names or tools names in particular.
Especially if they end up being hardcoded in some clients when calling "well-known" servers, or for inter-servers communication someday, if this ever occurs.
On the other hand, descriptions can be translated harmlessly.

Regarding the specific questions, I'd be curious for your initial opinions. Always happy to discuss and suggest alternatives, if needed, but you've had a lot of good thoughts so far and I want to make sure to hear you first.

OK, my suggestions will follow. I'll address each topic in a specific "thread" inside this issue, to keep things separated.

What would be the most appropriate standard to encode language codes? ISO 693-1? ISO 693-2? IETF BCP 47? Any other?

Let's go for the wiiiiiiidely used IETF BCP 47. For instance:

Basic language codes: en for English, fr for French, es for Spanish, zh for "Generic Chinese" (usually defaulted to Mandarin), yue for Cantonese...
Language + Region Codes: en-US for American English, en-GB for British English, fr-CA for Canadian French...

The component that selects the language (either client-side or server-side, see other topic) SHOULD provide a fallback mechanism to a close variant if the preferred language is not available. For instance: en=>en-US or en-US=>en or en-GB=>en-US.
If no close variant is available, or if i18n is not handled by the component, the "default" language MUST be returned (text + corresponding language code).

Names are used as identifiers. I fear we might end up having problems by making them multilingual [...]

The more I think about it, the more I feel it would be a hassle to translate names for tools and prompts:

Servers would have to check for variants of names (much higher complexity)
The spec indeed states they are "unique identifiers"

On the other hand, names for resources could be i18n-compliant, since the identifier would be the uri in that case. The spec indeed states they are "human-readable names".
That being said, we have another problem, not spec-related: In Claude Desktop, the "Allow" popup says "Run tool_name from server", without description, which is some kind of flaw in itself. And even more if only the description is translated.

In terms of workflow, where should language selection occur?
[...]
These options would have pretty different impacts on the spec.

Let's avoid unnecessary network traffic.
For requests, clients should add an optional preferredLanguage parameter.
For responses, servers should add a (mandatory?) language field.
Language selection would be handled by the server.
The server would fall back to its default language if:

It does not support i18n
The preferred language is not available
No equivalence was found (see above)

Should we provide backward compatibility?

Not sure yet, but I feel me might figure out a smooth solution.
If not, this would probably join other major and impactful changes in a next version of the spec.

Thanks for all the thoughts! Broadly, this sounds great. I'd suggest the following:

preferredLanguage becomes part of the client's advertised capabilities, under a new locale subobject (i.e., capabilities.locale.preferredLanguage)
language is then part of the server's response in a similar object, iff supported (i.e., capabilities.locale.language)

Done this way, it should actually be 100% backwards compatible, since this doesn't involve changes to the shapes of any other responses—and we can add it to any protocol version.
What do you think?


types:Thread is one level too high in the subclass hierarchy

Background
Issue 470 posed a question about what the superclass of types:Thread should be within the UCO namespaces.  At the time that question was written, core:UcoThing was being introduced, and was considered the correct answer.
core:UcoThing was being introduced in Issue 430.  Between the beginning and ending of that issue, another intermediary class was introduced for a class hierarchy of those UcoThings that are not UcoObjects or UcoFacets: UcoInherentCharacterizationThing.
The types:Dictionary class became a subclass of core:UcoInherentCharacterizationThing, as part of resolving 430.  However, types:Thread, as a "Data structure" ontology class that we have chosen to not introduce into the UcoObject subclass hierarchy (particularly in discussion on Issue 393 that introduced types:Thread), is not a subclass of core:UcoInherentCharacterizationThing.  This appears to have been an implementation oversight during the resolution of 430.
Requirements
Requirement 1
OWL classes in UCO being used as "Data structure" classes should exist in the same subclass hierarchy below core:UcoInherentCharacterizationThing.
Requirement 2
types:Thread must be moved in the subclass hierarchy to alongside other "Data structure" classes.
Risk / Benefit analysis
Benefits

Design is aligned for current "Data structure" classes.

The one exception is core:Compilation and its subclass hierarchy.  core:Compilation is analogous to co:Set but has been around since UCO's beginning as a general, unordered object-aggregation.

Design is clarified for future users of types:Thread and other structures that will potentially be imported from the Collections Ontology.

Risks
Moving types:Thread to be a subclass of core:UcoInherentCharacterizationThing will mean it will no longer be permitted to define a node as, say:
kb:some-thing
  a types:Thread, observable:ObservableObject ;
  .

This would be a member of two disjoint classes (UcoObject and UcoInherentCharacterizationThing), and thus would be invalid.
Nor would it be permitted to define a new class:
kb:SomeThreadClass
  rdfs:subClassOf types:Thread, observable:ObservableObject ;
  .

(UCO test mechanisms don't currently review this in an automated manner.)
While this appears to capture the design intent of Issue 430, due to 1.0.0 defining types:Thread as a subclass of the most-general UcoThing, implementing this fix would be a backwards-incompatible change that would have to wait for UCO 2.0.0.
Competencies demonstrated
Competency 1
A user is interested in a yet-unexplored usage pattern for co:List based on some ObservableObject---let's say representing traceroute results---and wonders how to define a new class.
Competency Question 1.1
Before the change in this proposal, what looks like an appropriate subclassing, according to types:Thread?
Result 1.1
ex:IPAddressRoute
    a owl:Class , sh:NodeShape ;
    rdfs:subClassOf observable:ObservableObject, co:List ;
    sh:property [
        sh:class observable:IPv4Address ;
        sh:path ( co:item co:itemContent ) ;
    ] ;
    .

(This also breaks from the current UCO design pattern with Facets, which is another of the reasons we chose to move co:* subclasses over to UcoInherentCharacterizationThing.)
Competency Question 1.2
With this fix, what is the appropriate design style?
Result 1.2
ex:IPAddressRouteFacet
    a owl:Class , sh:NodeShape ;
    rdfs:subClassOf core:Facet ;
    sh:property
      [
        sh:class co:Thread ;
        sh:path ex:hasIPv4AddressRoute ;
      ] ,
      [
        rdfs:seeAlso observable:MessageThreadFacet ;
        sh:class observable:IPv4Address ;
        sh:path ( ex:hasIPv4AddressRoute co:item co:itemContent ) ;
      ]
      ;
    .

Solution suggestion

For UCO 2.0.0: Change types:Thread rdfs:subClassOf core:UcoThing to types:Thread rdfs:subClassOf core:UcoInherentCharacterizationThing.
For UCO 1.x.0: Because a superclass change may not be a tenable thing to "Deprecate" in the ontology with SHACL, add a test to uco-qc.ttl that fails CI on detecting a class definition that is a subclass of both uco-core:UcoObject and co:Collection.  (co:Collection is the CO topmost class, just under owl:Thing.)

Coordination

Tracking in Jira ticket OC-285
[x] Administrative review completed, proposal announced to Ontology Committees (OCs) on 2022-11-29
[x] Requirements to be discussed in OC meeting, 2022-12-08
[x] Requirements Review vote occurred, passing, on 2022-12-08
[x] Requirements development phase completed.
[ ] Solution announced to OCs on TODO-date
[ ] Solutions Approval to be discussed in OC meeting, date TBD
[ ] Solutions Approval vote has not occurred

[ ] Solutions development phase completed.
[ ] Implementation merged into develop
[x] Milestone linked
[ ] Documentation logged in pending release page

I'd suggest to defer resolution of this CP until clarification has been achieved about the meaning and use of the core:Facet by the FDP-WG, because this could change the meaning and position of UcoInherentCharacterizationThing

I'd suggest to defer resolution of this CP until clarification has been achieved about the meaning and use of the core:Facet by the FDP-WG, because this could change the meaning and position of UcoInherentCharacterizationThing

I don't think that will be necessary.
The current definition of disjoint classes in UCO's core namespace is here:
[]
	a owl:AllDisjointClasses ;
	owl:members (
		core:UcoInherentCharacterizationThing
		core:UcoObject
	) ;
	.

Facets are thus mostly out of scope of the discussion for this Issue.  The places where I used Facet in my illustrations in this ticket pertained to existing design patterns.  Further, the Facet Design Pattern Working-Group (FDP-WG) is highly likely to be proposing backwards-incompatible changes that would need to wait until UCO 2.0.0.
I think it would be helpful for us to approach whatever the working-group's 2.0.0 resolution is with only one design pattern, rather than the current pattern and patterns implemented without regard for Issue 509.

I would concur that this CP is likely not encumbered by any questions with facets.
And to clarify, Alex is correct above that core:UcoInherentCharacterizationThing and core:UcoObject are disjoint but it should also be noted (since the topic of facets weas raised in relation to this discussion) that core:Facet is a subclass of core:UcoInherentCharacterizationThing. All facets are "a grouping of characteristics unique to a particular inherent aspect of a UCO domain object". So, because of the assertion that core:UcoInherentCharacterizationThing and core:UcoObject are disjoint, it also follows that core:Facet and core:UcoObject are disjoint.

One interesting issue arose in testing.  I implemented the class of Competency Question 1.1 as an XFAIL test, and encountered that the Collections Ontology in some cases (including co:List) uses a subclassing pattern that would not show in RDFS inferencing that a class is (eventually) a subclass of co:Collection.  I excerpted the particular pattern for the test after converting CO's RDF-XML format to Turtle:
co:List
	a owl:Class ;
	rdfs:comment "This definition is excerpted from co:."@en ;
	rdfs:subClassOf [
		a owl:Class ;
		owl:intersectionOf (
			co:Bag
			[
				a owl:Restriction ;
				owl:onProperty co:item ;
				owl:allValuesFrom co:ListItem ;
			]
		) ;
	] ;
	.

RDFS inferencing would look for co:List rdfs:subClassOf co:Bag.  The above pattern would not work in RDFS inferencing, and instead OWL inferencing would be required to infer the class hierarchy.
This is the explanation for why the SHACL-SPARQL query looks more complex for the Collections Ontology side vs. UCO's side:
PREFIX co: <http://purl.org/co/>
PREFIX core: <https://ontology.unifiedcyberontology.org/uco/core/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT $this
WHERE {
	$this
		rdfs:subClassOf*/(owl:intersectionOf/rdf:rest*/rdf:first)?/rdfs:subClassOf* co:Collection ;
		rdfs:subClassOf* core:UcoObject ;
		.
}

This pattern is tested in PR 516 here.


[1.7.0] Clicking on a tab adds #react-mdl-tabs-hack into URL

It is this code that does it. Had to revert back to v1.6.1 which works as expected with HeaderTabs.
We should find a better solution, or fix it on the mdl side. Appending #react-mdl-tabs-hack to the URL is not an option for me.

Yay :/
I believe I'll make a monkey patch on the MDL side to remove the href check to show the panels. It doesn't make sense in a react world.
Trying to fix it since the beginning with the href but it definitely doesn't work.

Yeah mdl@v1 is not very react-friendly. I believe that monkey patch is the only solution for this problem ATM, because I think they wont accept any PRs on the mdl for this. Anyway, let me know how I can help. I can even write the patch if you just point me into the right direction.

The way I currently patch MDL is by fetching their 1.2.0 version, applying my current patch on it, and then modify the things I have to update. Generating the new MDL version with the changes (with gulp all on the MDL repo). Then creating the patch from the changed files/commits.
I'll do this tonight after work, but if you want to do it, go ahead ;)

Uhh I think I'll leave this one to you :) It would probably take me way too long to get everything right. But I will test as soon as you patch it ;)
I want to focus on missing form components which I require for my app: react-mdl-autocomplete, react-mdl-datetime and fix react-mdl-selectfield to use chips at least.
I really hope they release v2 alpha soon, it's kinda frustrating having to work with v1.

@tleunen when do you plan to release this? I would like to test it + HeaderTabs ripple effect.

Sorry @HriBB, I'll probably do it later today.

@tleunen no worries, did not mean to rush you or anything, I was just wondering when it's gonna be released. Take your time, I know that you've got more important stuff to do :)


Add eventmesh server

Added settings for etcd
Added constants for state
Introduced EventMeshServer interface
Added Etcd3EventMeshServer , implementation of EventMeshServer to use
etcd v3
Added new node package etcd3
Added unit tests.

Pull Request Test Coverage Report for Build 935

93 of 93 (100.0%) changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 87.584%

Totals

Change from base Build 928:
0.1%

Covered Lines:
6216

Relevant Lines:
6857

💛  - Coveralls

@muralisc / @subhankarc : Can this be merged now ? There is a travis build failing.

@subhankarc
I had a basic doubt. If a manager has a watch on path /deployments/<resourceType>/<resourceId> will the same watch be called even if there is a change in the annotation i.e. subpath? For ex., will the createManager be invoked even when a backup is triggered via annotation /deployments/<resourceType>/<resourceId>/<annotationName>/<annotationType>/<annotationId>?
Wouldn't this lead to performance issues later when multiple managers will all be getting fired and all managers except one would is supposed to process it?

@jagadish-kb @muralisc
Based on feedback from Jagadish, I feel we can restructure our event keys a bit to ensure two managers do not listen to same path and do not get unnecessary events.
For example,
SF BOSH provision manager listens to /deployments/bosh and look for a key /deployments/bosh/<service-instance-id>
SF's default Backup manager can look for a key /backups/default/<service-instance-id>/<backup-id> and can listen to /backups/default/
In a generic way, every Provision Manager watches on /deployments/ and every operation manager watches on /<operation-name>/<operation-type>
@muralisc @jagadish-kb do you see any issue with this?

@subhankarc @muralisc
I also had similar thoughts as you have posted in the last comment. No need to ensure every piece of data is under one root. Data can be segregated under different roots and if need be for retrieval / reporting they can be collated. So basically each manager has its own unique path, so that we dont have multiple watchers getting fired and each watcher having to figure out if he needs to do something or not. Rather it should be given that if a watcher is invoked that watcher must do necessary.
Also, another border/boundary condition during restarts. Does etcd ensure that all handlers are fired? for ex., for whatsoever reason the watcher had crashed by the time API server puts in the request and when the watcher job comes back up does etcd identify this and invoke the watcher?

@jagadish-kb
So during every restart the watch is registered by the manager. so while the manager is up, it will receive the event.
If it was down, it would not receive the events during which it was down. But that would not be needed as well, because we would not stagger any request and wait for managers to pick it later. It would reject the request if not immediately picked up by the manager.
Does it answer your question?

@subhankarc - when you say " It would reject the request if not immediately picked up by the manager." - you mean API server polls for inqueue status (for x duration) and within this xduration if its not picked up by manager it will send back failure response? -- If you meant so its fine.
So while rejecting this request then API server should also ensure that the state value is set to rejected. Just in case to avoid manager to pick it up sometime later and process it which could result in orphaned deployment.

@jagadish-kb
That is correct. API server will make the resource status as IN_QUEUE and wait/poll for it to change. Managers pick it up and make it IN Process and only then API server responds. It will be determined by the different status of resource.
We have so far decided on in_queue, in_process, completed and failed state. We plan to define the possible state values as a spec for any Controller so that writing new controller would be easy.

@subhankarc @jagadish-kb
As discussed ,
SF's default Backup manager can look for a key /backups/default/<resource-type>/<service-instance-id>/<backup-id> and can listen to /backups/default/
While provision manager listens to /deployments/<resource-type>
The above change will be made as the part of backupManager changes.


feat: CSP header

Proposed changes

NextSafe enables us to set default values for common security headers.

This PR focuses on updating the Content Security Policy, ensuring that only essential external resources are allowed to interact, such as loading content, images, and scripts exclusively from specified URLs.

Should pass all the CSP checks -

observatory.mozilla.org
csp-evaluator.withgoogle.com

Almost there but inline-css is required - (can be solved using nonce, but need to check how to implement it with nextJS)

https://github.com/valory-xyz/autonolas-registry-frontend/assets/22061815/729d5701-c87d-416e-93aa-9b7052acb5da

Here is how the headers would appear – please verify them in the network tab.

App should work as before with improved security

https://github.com/valory-xyz/autonolas-registry-frontend/assets/22061815/f357b8b1-1454-46a7-8019-263149af8d42

Minting of components/agents/services

https://github.com/valory-xyz/autonolas-registry-frontend/assets/22061815/286fccf7-e37c-4206-9c24-efce42014566

https://github.com/valory-xyz/autonolas-registry-frontend/assets/22061815/25ab036e-862a-4d89-bc3e-54a0b54f2f43
https://github.com/valory-xyz/autonolas-registry-frontend/assets/22061815/87f26c95-2b42-46bf-8e3f-795c7eca5a38
Types of changes
What types of changes does your code introduce to autonolas-registry-frontend?

[ ] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

[x] My pull request is targeted against the main branch (left side), and I've started my branch from our main.
[ ] I've ensured that linting and unit tests pass locally with my changes, and CI also passes successfully.
[ ] I've included tests to validate the effectiveness of my fix or to demonstrate that my feature functions correctly.
[ ] I've verified that my changes do not reduce code coverage.
[x] I've added any necessary documentation as needed.
[x] I've confirmed that any dependent changes have been merged and published in downstream modules.
[ ] If the onload page involves contract interaction, I've ensured it is client-side rendered using next/dynamic.
[x] I've updated next.config.js header to include the new URL for secure app interaction

I went through and tested further. I couldn't identify any broken functionality.
Note that there are still some errors related to CSP showing in console:

The Vercel feedback is probably specific to the preview url
Wallet Connect Verify is probably not working right now, but to be honest I don't know exactly what this is yet anyway
prefetch-src is deprecated – I assume NextSafe leaves this in there for support of older browsers


Add support for data from other formats

See available formats here:
https://pymatgen.org/pymatgen.io.html

I looked into this a bit, and I find that probably the best solution is something very close to the one that we already have.
The Data structure can be created with its arrays through the default constructor which can then be cached by the user if desired. This would generate the following code for any data format:
from pymatgem.io import correct_parser
from gemdat import Data

def generic_reader(file, cache):
  if Path(cache).exists():
          return Data.from_cache(cache)
  some_object = correct_parser(file);
  # Massage `some_object` into correct format for `Data`
  massaged_data = {
              'structure'=structure,
              'trajectory_coords': trajectory.coords,
              'species': structure.species,
              'lattice': structure.lattice,
              'time_step': run.parameters['POTIM'] * 1e-15,
              'temperature': run.parameters['TEBEG'],
              'parameters': run.parameters, }
  return Data(**massaged_data)

generic_reader("file","cache")
data.to_cache("cache")

I think we should offer this function as a way people can add new readers, for new data formats. If they have created one we can then implement it in GEMDAT.
It might be possible to modularize the from_xxx functionality, but that would come at the cost of increased complexity

@alexandrosvasi @AstyLavrinenko Please provide us with example data for Lamps/gromacs/cp2k

@alexandrosvasi @AstyLavrinenko Can you give us an update on this?

We now have some data to test with for LAMMPS


shell: fix history feature

When user was typing a new command and next pressed an up arrow shell has displayed previously executed command. Next it was not possible to display currently edited command using a down arrow.
Fixes #10766.
Signed-off-by: Jakub Rzeszutko<EMAIL_ADDRESS>

Codecov Report

Merging #10793 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #10793   +/-   ##
=======================================
  Coverage   53.24%   53.24%
=======================================
  Files         215      215
  Lines       25820    25820
  Branches     5692     5692
=======================================
  Hits        13748    13748
  Misses       9756     9756
  Partials     2316     2316

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53db000...3ff8a1d. Read the comment docs.

I need to print this post, no one will believe me I was praised by @pfalcon ;)
Regarding circular list it was intentional and we like it, please don't ask to change it :)


Hyundai & KIA & GENESIS Official steerRatio

Description : Apply the official specification information  to KIA_EV6 , IONIQ_5 , G70 ,GV70, G80
Verification Hyundai Motor's official website
KIA_EV6  : https://www.kiamedia.com/us/en/models/ev6/2022/specifications
IONIQ_5  : https://www.hyundainews.com/assets/documents/original/48175-2022Ioniq5ProductGuidespecs090821.pdf
G70 :  https://www.conceptcarz.com/s31009/genesis-g70.aspx
GV70 : https://www.genesis.com/content/dam/genesis-p2/au/assets/models/gv70/specs/documents/Genesis_GV70_MY23_Spec_Sheet.pdf
G80 : https://www.genesis.com/content/dam/genesis-p2/au/assets/models/g80/specs/documents/Genesis_New-G80_Specs-Table_2020.pdf

收到！

a lot of cars have variable ratio steering, this is probably the reason we don't match the spec numbers


Edit Dashboard title and Slice title in place

Currently the only way to edit the title of a dashboard or a slice is to use the CRUD view provided by FAB.
Instead of that, it'd be nice to have a way to edit the name in-place, the same way that it's done in Google docs for instance.

google docs example:

clicking outside the editable field or hitting enter should save the new name
use the notify module to give user feedback
https://github.com/airbnb/superset/blob/master/superset/assets/javascripts/SqlLab/components/VisualizeModal.jsx#L153

Does this supersede #2726 then? I had something started on that.

Oh I forgot I had opened it. Closing #2726


Chunks lost in translation

Hi,
Thank you for your work on indicTrans. I have been using this to translate some short paragraphs( 3-4 sentences) in various supported Indic languages.  I noticed that there is a certain amount of data that gets lost in translation. For example- I am trying to translate this English sentence to Tamil:
"In order to make the French capital safer, quieter and less dirty,a speed limit of 30 kmph for cars came into force in Paris on Monday"
This is translated as:
பிரான்ஸ் தலைநகர் பாரிஸில் கார்களுக்கு மணிக்கு 30 கிலோமீட்டர் வேகத்தில் செல்லலாம் என்ற கட்டுப்பாடு விதிக்கப்பட்டுள்ளது
The chunk- "In order to make the French capital safer, quieter and less dirty" is lost in the translation
I assumed that with the Transformer architecture, long sentences too could be translated more accurately.
I would like to know what could be done to fix this issue.

@gowtham1997  Thank you for your response. I am using the python interface. The paragraph is split into lines before attempting translation.
The Samanantar paper mentions - "sentence pairs with longer sentences are unlikely to have high alignment on LaBSE representations and thus be included in Samanantar"
Could this be an issue due to the data that IndicTrans was trained on( i.e no such long sentences in the training data) ?

I am not sure about the cause @PS-AI .
But we have now released newer version of the translation models (V0.3) and it seems to preserve the whole sentence during translation.
"In order to make the French capital safer, quieter, and less dirtier, a speed limit of 30 kmph for cars came into force in Paris on Monday"
translates to
"பிரான்ஸ் தலைநகரை பாதுகாப்பானதாகவும், அமைதியானதாகவும், தூய்மையற்றதாகவும் மாற்றுவதற்காக, கார்களுக்கான வேக வரம்பு மணிக்கு 30 கி. மீ. என பாரீஸ் நகரில் திங்கள்கிழமை முதல் அமலுக்கு வந்துள்ளது."
^ Note that I have changed "less dirty" to "less dirtier" (cleaner also works). For some reason, the model gives "அழுக்கானதாகவும்" for "less dirty", so it still needs some improvement.

Even google translate seems to make the same mistake for En to Tamil which is surprising.
Thanks for bringing this to our notice. If you find similar sentences where translations quality is bad, please feel free to let us know


Make mixin audit opt-in

See https://discord.com/channels/770691727568404521/1042399103822340146
The mixin audit messes up people trying to make add-ons for our mods. Making it opt-in would avoid this.
There should be some discussion, as this opt-in behavior would make it less likely that we end up running these audits often - maybe that's not bad though.

Oh the build failed. I'll fix this in a sec

Ayo, make sure to bump the version, didn't do it in the devel yeet pr.
Also, just a sec eh?

👌


Channel metadata not pulled correctly on short claim id collision

https://lbry.tv/@ivanontech:a
@ivanontech has a few entries, 2 of which where the claim id starts with an a, but only one that's controlling (other is spent). Double check our chainquery logic on how we look these up.

This is now fixed


MAINT: fix runtest.py overriding $PYTHONPATH: prepend instead

Reference issue
Closes #14501
What does this implement/fix?
runtest.py currently overrides the $PYTHONPATH environment variable when setting up the environment for child scripts to be invoked. This leads child processes, e.g. setup.py, to not run in the same environment as the parent, i.e. they cannot find python modules installed under a separate prefix.

Thanks @jkelling for your first contribution, and to reviewers


Initial Logical Optimiser + Integration

This is an initial take on a Logical Optimiser which operates on the alga-Graph-encoded streams and applies a set of built-in rules.
This is simply integrated into CompileIoT via a flag in the GenerateOpts type.
Presently this exposes a bug elsewhere that breaks the taxi example: the two adjacent Filter operations are fused by the Optimiser (good!) but the generated Haskell code has a syntax problem (bad!). I'm going to try and fix that and add that commit to this PR.
I've prepared it as separate commits that could be reviewed individually to make it easier.

The taxi example now works. Further work is needed to re-enable the "mapFilter" rewrite rule (and some changes to all the other rules) but perhaps not blocking this PR.

I found the bug (In CompileIoT, kinda) that was triggered by Taxi when mapFilter was enabled. It's now fixed, a test written to catch the situation, and mapFilter re-enabled.


WRITE_TRUNCATE appears to be ignored when passed as an option in a load job

The documentation suggests to set:
configuration.load.writeDisposition to WRITE_TRUNCATE.
However, when attempting to set the option for this, nothing works. The following array consists of all the options I've tried:
    $options = [
        'configuration.load.writeDisposition' => 'WRITE_TRUNCATE',
        'configuration.writeDisposition' => 'WRITE_TRUNCATE',
        'writeDisposition' => 'WRITE_TRUNCATE',
        'configuration' => [
            'writeDisposition' => 'WRITE_TRUNCATE',
            'load' => [
                'writeDisposition' => 'WRITE_TRUNCATE',
            ],
        ],
    ];

    /** @var $table \Google\Cloud\BigQuery\Table */
    $table->load($data, $options);

I added each option individually, observing each time that the table merely grew in size.

Problem appears to be in the JobConfigurationTrait where it constructs the array. The setting is present in $userDefinedOptions at the point of return, but it is being lost and is not present in the code it's returned to.
I've extracted it into a minimal test case:
<?php

$userDefinedOptions = ['configuration' => ['load' => ['writeDisposition' => 'WRITE_TRUNCATE']]];
$projectId = 'my_project';
$name = 'load';
$config = [
    'destinationTable' => [
        'tableId' => 'my_table',
        'datasetId' => 'data',
        'proejctId' => 'gbq_project',
    ]
];

$result = [
    'projectId' => $projectId,
    'configuration' => [
        $name => $config
    ]
] + $userDefinedOptions;

print_r($result);

https://3v4l.org/eBTgg
The behaviour of + with arrays is not to recursively array merge, I think that's what is missing.
$result = array_merge_recursive([
    'projectId' => $projectId,
    'configuration' => [
        $name => $config
    ]
], $userDefinedOptions);

I can confirm having tested the proposed change in the linked pull request and I now observe the table I'm replacing being truncated and having the appropriate number of records. The behaviour before was for it to always append, growing the table every time.

Wasn't sure how to get GitHub to link things properly: https://github.com/GoogleCloudPlatform/google-cloud-php/pull/623

Hey @bencromwell, thanks for the report!
The following is an example of the intended use, where the jobConfig key is to contain the properties found under configuration.load:
$table->load($data, [
    'jobConfig' => [
        'writeDisposition' => 'WRITE_TRUNCATE'
    ]
]);

I can certainly see where your confusion came up, and you aren't alone in having issues with the configuration. We'll see what we can do to help make things more clear. :)

Thanks for getting back to me, that does make it work. However, I have a few follow up queries and arguments for the case of changing it so you can pass things as defined in the main BigQuery service API documentation!
Firstly, why is it intended to be remapped from configuration.load to jobConfig? A good reason here may negate the rest of what I'm going to say (or it may not!).
My view is that the client should, where possible, match the service it's implementing, to reduce friction for implementers. That is one case for the properties being a direct match for those which are defined in the BigQuery documentation.
Additionally, people are most likely to go straight to the BigQuery documentation to find out what is possible and how to do it, not to the client documentation. As a result of that I feel that even if the documentation for this client included this nuance, it would still be missed up front. The chances are people will most likely arrive there having first experienced a failure.
Another question is does everything nest under jobConfig? In the case of writeDisposition, we're stripping the namespacing away. How much of the namespacing is meant to be removed for each of the options you can pass to a job as listed here? Or does only a subset apply to this particular (load) job, making this a moot point? If only a subset does apply at the moment, can you be sure there won't be a collision in the future as a result of a new option that isn't nested having the same name as something that is?
I don't know if I'm off base here, but if I'm not alone in missing this and with the other things I've raised, perhaps it is worth matching the client to the API?

Further to the above,
I've just retested with the original library code and with
    $options = [
        'jobConfig' => [
            'writeDisposition' => 'WRITE_TRUNCATE',
        ],
    ];

I am seeing the same bug - the table is not being truncated.

Firstly, why is it intended to be remapped from configuration.load to jobConfig?

My view is that the client should, where possible, match the service it's implementing, to reduce friction for implementers. That is one case for the properties being a direct match for those which are defined in the BigQuery documentation.

We agree! We try our best to make sure we expose every feature of the upstream API while helping to save our users as many keystrokes as possible a long the way. In this case, you can still supply all the options one might need to provide to configuration.load - we just don't expose it in the exact same way the REST API does. There are many times this comes up, and often we find it can help to create easier to interact with abstractions.

Additionally, people are most likely to go straight to the BigQuery documentation to find out what is possible and how to do it, not to the client documentation.

Interesting! That goes against my own experience, I have seen most people use a combination of client documentation and upstream documentation. That's a good point to consider, though, and is something we need to help bridge the gap on. Our upstream documentation often points to language specific examples of how to achieve certain tasks, have you found those snippets helpful in the past?

Another question is does everything nest under jobConfig? In the case of writeDisposition

Or does only a subset apply to this particular (load) job, making this a moot point?

As far as I understand it we should not expect configuration options for load to overlap with options from query, for example.

can you be sure there won't be a collision in the future as a result of a new option that isn't nested having the same name as something that is?

Could you provide an example of what you mean here, and how you would see it being problematic? It sounds to me changes like this would come with other considerations that may be breaking in nature - in which case, we would update our options in the library.
As far as how to solve the problem of providing configuration options, WDYT about some of these solutions?

Provide a job builder class, this could implement a fluent design to help make it clear which options are available. The downside here is, we need to manually add options to the class whenever the upstream API changes.

$job = new JobBuilder();
$job->writeDisposition($writeDisposition)
    ->anotherExampleOption($value);
$table->load($data, [
    'jobConfig' => $job
]);

Validate the request before delivering it, this may prove especially helpful when there are so many options to provide for a job. In this case, if one were to provide options of ['configuration' => ['load' => []]] you would immediately get an error with some direction of how to address it before we even hit the network.

If we had an example of how to provide writeDisposition in our source and library documentation, for example, do you think it would be valuable?

RE: The bug you're seeing, the following code sample successfully truncated the target table when I ran it. What does the output of the returned job look like for you after it has completed?
<?php

require 'vendor/autoload.php';

use Google\Cloud\BigQuery\BigQueryClient;

$bq = new BigQueryClient();
$table = $bq->dataset('dataset')
    ->table('table');

$job = $table->load(fopen('data.csv', 'r'), [
    'jobConfig' => [
        'writeDisposition' => 'WRITE_TRUNCATE'
    ]
]);

Hey @bencromwell, just wanted to follow up on this really quick. Are you still having trouble with the snippet provided above?

Hi @dwsupplee. Sorry I've not come back to you yet on it - I've not been able to work on it this week. Pretty sure it's still an issue but I'll need to extract the code down to a minimal repeatable example etc.

Hey @bencromwell, wondering if you'd had a chance to confirm whether @dwsupplee's snippet solved your issue. :)

Closed by #686.


[Tutorial: Start a private network] It is not clear which secret seed to use in the "Add key to keystore" step.

Content request
The tutorial says: "This example uses the secret seed generated by the key subcommand into the keystore. In this tutorial, the secret seed generated was 0x563d22ef5f00e589e07445a3ad88bb92efaa897d7f73a4543d9ac87476434e65" but I don't see this secret seed generated in the previous steps.
I think it's less confusing when it says: Use the secret seed that was generated in the Generate local keys using node template step. That's the 0x0087016ebbdcf03d1b7b2ad9a958e14a43f2351cd42f2f0a973771b90fb0112f instead of using a new secret seed and nobody knows where it comes from.
Are you willing to help with this request?
Yes!

Thanks for making this issue.
Adding some other points to clarify from a member of the Substrate Devs TG channel:

The tutorial says, to continue, I can re-use the same keys or I can use another computer to make more keys. What’s unclear to me, is why I would need another machine, and why I can’t use the same machine to generate more keys?

What I’m hearing in the chat is, I actually can use the same machine to generate more keys. But should I not do this, because the tutorial doesn’t say it’s an option?

@sanitu , @sacha-l
I'm not sure about this because I'm not deeply into the cryptography involved, but I think the issue is not that you can't generate more keys on a single machine, but that unless you change your root identity on a single machine, all of the keys map to a single identity. I guess that the tutorial could have you derive different keys from a single parent, then use those keys but I haven't tested that scenario or figured out if that would make the tutorial more difficult to follow.


Unbalanced  Data

May I ask what is the reason?

Version 6.3.0

This means, you have a plugin not working well. Hard to tell which one.

You need to check your typical trace in a test env first, make sure the trace is working as expected. And find out which span(plugin) is the first breaks.


Rate limit exceeded

I am trying to fetch tweets by tweet ids and getting the following output:
Rate limit exceeded, waiting for 30 seconds...
Rate limit exceeded, waiting for 30 seconds...
Rate limit exceeded, waiting for 30 seconds...
Rate limit exceeded, waiting for 30 seconds...
I am attaching my code, can you help me with it? What am I doing wrong or how can I make it work better?
const Rettiwt = require('rettiwt-api');
const { Auth } = require('rettiwt-auth');
const { TweetService } = require('rettiwt-api');
const auth = new Auth();
const fs = require('fs');
const readline = require('readline');
const outStream = fs.createWriteStream('tweets.txt');
// Helper function to write an index to a file
function writeIndexToFile(index) {
fs.writeFileSync("lastIndex.txt", index.toString());
}
// Helper function to read an index from a file
function readIndexFromFile() {
if (fs.existsSync("lastIndex.txt")) {
return parseInt(fs.readFileSync("lastIndex.txt", "utf8"));
}
return 0; // default value if file doesn't exist
}
async function fetchAndWriteTweets(credential, startIndex) {
const tweetService = new TweetService(credential);
const fileStream = fs.createReadStream('unique_arabicTrain-clean.csv');
const rl = readline.createInterface({
input: fileStream,
crlfDelay: Infinity
});
let index = 0;
for await (const line of rl) {
if (index < startIndex) {
index++;
continue;
}
let fetched = false;
while (!fetched) {
try {
const tweetId = line.split(',')[1];
const tweet = await tweetService.details(tweetId);
outStream.write(JSON.stringify(tweet) + '\n');
fetched = true;
writeIndexToFile(index);
} catch (error) {
if (error.response && error.response.status === 429) {
const waitTime = 30;  // Wait time is 30 seconds
console.log(Rate limit exceeded, waiting for ${waitTime} seconds...);
await new Promise(resolve => setTimeout(resolve, waitTime * 1000));
} else {
console.error(Failed to fetch Tweet at index ${index}:, error);
fetched = true;
}
}
}
index++;
}
}
auth.getGuestCredential()
.then(credential => {
let startIndex = readIndexFromFile();
fetchAndWriteTweets(credential, startIndex);
})
.catch(console.error);

896 seconds.

if (error.response && error.response.status === 429) {
// Get the reset timestamp from headers
const resetTimestamp = error.response.headers['x-rate-limit-reset'];
// Get the current timestamp
const currentTimestamp = Math.floor(new Date().getTime() / 1000);
// Calculate the difference in seconds
const difference = resetTimestamp - currentTimestamp;
// If difference is negative (i.e., the reset time is in the past),
// use a default delay of 30 seconds
const waitTime = difference > 0 ? difference : 30;
console.log(Rate limit exceeded, waiting for ${waitTime} seconds...);
await new Promise(resolve => setTimeout(resolve, waitTime * 1000));
}
This is how I am checking it


Loading stuck at 82%

When i try to compile to html5 the loading gets stuck at 82%

try refreshing again or closing out and trying again.

I think this is an actual bug, I saw it before.


Allowing to remove all listchars but keep indentation guides with 'set list/nolist'.

Addressing #195.
I renamed the option g:indent_blankline_respect_list to the more descriptive name g:indent_blankline_disable_with_list.

If this option is set to true, setting nolist will essentially disable the plugin as was implemented in #201.
If this option is set to false (default), setting nolist will replace all listchars with empty spaces (like nolist without this plugin) but keep indentation guides.

Beat me to it, I wanted to do the same :+1:
I think if we rename the option, it should be g:indent_blankline_disable_with_nolist
code looks good

Well, one should definitely commit before force-pushing 🙃
Addressed your suggestions, though.


ApyHub Document Readability (Independent Publisher)

When submitting a connector, please make sure that you follow the requirements below, otherwise your PR might be rejected. We want to make you have a well-built connector, a smooth certification experience, and your users are happy :)
If this is your first time submitting to GitHub and you need some help, please sign up for this session.

[X] I attest that the connector doesn't exist on the Power Platform today. I've verified by checking the pull requests in GitHub and by searching for the connector on the platform or in the documentation.
[X] I attest that the connector works and I verified by deploying and testing all the operations.
[X] I attest that I have added detailed descriptions for all operations and parameters in the swagger file.
[X] I attest that I have added response schemas to my actions, unless the response schema is dynamic.
[X] I validated the swagger file, apiDefinition.swagger.json, by running paconn validate command.
[X] If this is a certified connector, I confirm that apiProperties.json has a valid brand color and doesn't use an invalid brand color, #007ee5 or #ffffff. If this is an independent publisher connector, I confirm that I am not submitting a connector icon.

If you are an Independent Publisher, you must also attest to the following to ensure a smooth publishing process:

[X] I have named this PR after the pattern of "Connector Name (Independent Publisher)" ex: HubSpot Marketing (Independent Publisher)
[X] Within this PR markdown file, I have pasted screenshots that show: 3 unique operations (actions/triggers) working within a Flow. This can be in one flow or part of multiple flows. For each one of those flows, I have pasted in screenshots of the Flow succeeding.
[X] Within this PR markdown file, I have pasted in a screenshot from the Test operations section within the Custom Connector UI.
[ ] If the connector uses OAuth, I have provided detailed steps on how to create an app in the readme.md.

[[certify-connector]]

Hello @troystaylor,
There is a ApyHub title connector exist in the system.  This connector title named as ApyHub again.
Could you please change the title?

I have changed the other to ApyHub Archiving. This one should be ApyHub Document Readability.

Hello @troystaylor,
There are no files in this PR to review, can you please help me to find them?


Using ASP.NET Core Identity not working for asp.net core 2.1

Issue / Steps to reproduce the problem
Documented Using ASP.NET Core Identity
Follow the instructions exactly as stated but with an asp.net core 2.1 application.
The problem shows up with both the MVC and Page's versions of WebApp.
I stepped the code and logins are happening, the user is found, etc.
However something is going wrong with cookies where the app doesn't think its logged in.
Let me know if you want me to post a github project for this issue
 source code of issue

@SIkebe

There are some changes in the dev branch related to upgrading the library to .net core 2.1  to my knowledge they haven't been released yet.     I would suspect one of these changes are causing you your issues.
It would be nice in the future if you would post your logs in the issue as well.   While having a link to your repo may seam like a good idea if you delete it in two years the information on this repo will then be invalid.   Issues should contain all the information needed to recreate the issue within them.

I am experiencing the same problem following the ASP.NET Identity Quickstart. It does look like the quickstarts haven't been updated in a while.
When creating the boilerplate web application here, we now have the option to choose ASP .NET Core 2.1 in addition to 2.0. The quickstart example works great when choosing the 2.0 option, but doesn't work in 2.1.
What happens after entering valid creds is that the user is kicked back to the index page as if they hit the back button. The Register and Login links are in the navbar as opposed to "Hello<EMAIL_ADDRESS>as expected.
There's no indication of error. Registration works, but after doing so the user is not logged in nor can they log in. I copied the Serilog configuration from another quickstart example. Here's what happens:
2018-06-11 16:20:38.568 -05:00 [INF] You are using the in-memory version of the persisted grant store. This will store consent decisions, authorization codes, refresh and reference tokens in memory only. If you are using any of those features in production, you want to switch to a different store implementation. 2018-06-11 16:20:38.607 -05:00 [DBG] Using idsrv as default scheme for authentication 2018-06-11 16:20:38.610 -05:00 [DBG] Using Identity.External as default scheme for sign-in 2018-06-11 16:20:38.610 -05:00 [DBG] Using Identity.External as default scheme for sign-out 2018-06-11 16:20:38.611 -05:00 [DBG] Using oidc as default scheme for challenge 2018-06-11 16:20:38.611 -05:00 [DBG] Using oidc as default scheme for forbid 2018-06-11 16:20:39.891 -05:00 [WRN] Failed to determine the https port for redirect. 2018-06-11 16:21:17.248 -05:00 [INF] You are using the in-memory version of the persisted grant store. This will store consent decisions, authorization codes, refresh and reference tokens in memory only. If you are using any of those features in production, you want to switch to a different store implementation. 2018-06-11 16:21:17.304 -05:00 [DBG] Using idsrv as default scheme for authentication 2018-06-11 16:21:17.308 -05:00 [DBG] Using Identity.External as default scheme for sign-in 2018-06-11 16:21:17.308 -05:00 [DBG] Using Identity.External as default scheme for sign-out 2018-06-11 16:21:17.309 -05:00 [DBG] Using idsrv as default scheme for challenge 2018-06-11 16:21:17.309 -05:00 [DBG] Using idsrv as default scheme for forbid 2018-06-11 16:21:18.736 -05:00 [WRN] Failed to determine the https port for redirect. 2018-06-11 16:21:24.158 -05:00 [INF] AuthenticationScheme: Identity.External signed out. 2018-06-11 16:21:35.791 -05:00 [INF] AuthenticationScheme: Identity.Application signed in. 2018-06-11 16:21:35.796 -05:00 [INF] User logged in. 2018-06-11 16:23:11.341 -05:00 [INF] You are using the in-memory version of the persisted grant store. This will store consent decisions, authorization codes, refresh and reference tokens in memory only. If you are using any of those features in production, you want to switch to a different store implementation. 2018-06-11 16:23:11.380 -05:00 [DBG] Using idsrv as default scheme for authentication 2018-06-11 16:23:11.383 -05:00 [DBG] Using Identity.External as default scheme for sign-in 2018-06-11 16:23:11.383 -05:00 [DBG] Using Identity.External as default scheme for sign-out 2018-06-11 16:23:11.384 -05:00 [DBG] Using idsrv as default scheme for challenge 2018-06-11 16:23:11.384 -05:00 [DBG] Using idsrv as default scheme for forbid 2018-06-11 16:23:16.675 -05:00 [INF] AuthenticationScheme: Identity.External signed out. 2018-06-11 16:23:35.907 -05:00 [INF] AuthenticationScheme: Identity.Application signed in. 2018-06-11 16:23:35.911 -05:00 [INF] User logged in.
Please let me know if there's any other information I can provide.

I got this to work by making it look as close to the working Quickstarts/6_AspNetIdentity

Change

services.AddDefaultIdentity<IdentityUser>()
    .AddEntityFrameworkStores<ApplicationDbContext>();

to
services.AddIdentity<IdentityUser, IdentityRole>()
    .AddEntityFrameworkStores<ApplicationDbContext>()
    .AddDefaultTokenProviders();

source

Bring back the Identity Scaffold
This was a strange one, because technically it is supposed to work without it.  I did not change any code that was brought back.  This is all those cshtml files that make up the Identity UI.
The reason I brought it back was that the Login page didn't even come up so I wanted to debug it.  However simply bringing it back made it work.

Added IEmailSender implementation because now logout needs it.

I have a history of start to finish;
history

Thank you @ghstahl, that seems to work for me as well.
First I added
services.AddIdentity<IdentityUser, IdentityRole>()
    .AddEntityFrameworkStores<ApplicationDbContext>()
    .AddDefaultTokenProviders();

and was instantly logged in when I started the application. As you mentioned the login/logout/register buttons don't work until the ASP Identity bits are scaffolded into the application. After doing this things are working great.
I noticed when using the Razor Class Library (RCL) that the broken login button, for example, wanted to route to http://localhost:5000/?area=Identity&page=%2FAccount%2FLogin
However, once scaffolded, it would correctly route to: http://localhost:5000/Identity/Account/Login
I created a separate Core 2.1 MVC application for science and followed the Quickstart example. Everything works except the routing. For example when navigating to an authorized controller endpoint, the client MVC application kicks you back to the login site at the URL http://localhost:5000/account/login?returnUrl=..., which 404's.
However, if you add the identity prefix to it like this: http://localhost:5000/identity/account/login?returnUrl=... then everything works great. Similarly the RequireConsent routing won't work if enabled, for what I suspect is the same reason.
This might be a separate issue though. Happy to log it separately if desired. Cheers!

@jogleasonjr did you get Logout to work.  In my examples I can't get
await _signInManager.SignOutAsync();

to trigger the /connect/endsession/callback redirect.
The expectation is that when the HOST logs out that it fires a bunch of iFrame logouts to all the clients that it knows about.
It works in the IdentityServer4.AspNetIdentity host example
My startup is practically identical.

I think I am at feature parity with my POC asp.net core 2.1 webapp.
source
There was nothing that had to be done with any IdentityServer4 or IdentityServer4.AspNetIdentity stuff, as expected!
We have a documentation issue now with getting folks up to speed on an out of the box AspNet Core 2.1 webapp.  The main culprit is that Identity scaffolding, where bringing all the source back solved a lot of problems.
Frankly I find no value in what the AspNet team did to move out UI elements to separate dlls.  It makes it difficult to know what is going on, and I personally would never use it.  I have been down that road before where we moved all cshtml files into an assembly resource and all it got me was team discontent.

To use the shared UI library and not have to scaffold into the project, you can use .AddDefaultUI() as in:
services.AddIdentity<IdentityUser, IdentityRole>()
        .AddEntityFrameworkStores<ApplicationDbContext>()
        .AddDefaultTokenProviders()
        .AddDefaultUI();

In addition, because the shared UI places these Razor pages in an Identity area, I needed to override the IdSrv URL configuration.
services.AddIdentityServer(options =>
        {
            options.UserInteraction.LoginUrl = "/Identity/Account/Login";
            options.UserInteraction.LogoutUrl = "/Identity/Account/Logout";
        })

As @DarranShepherd  suggested, adding AddDefaultUi lets you avoid scaffolding all of the objects.
Regarding routing, IdentityServer4 is looking for old root or account routes, when now most of the routes are prepended with "/identity". You can't just point the client to "url/Identity" because that breaks idsrv's discovery functionality.  The solution seems to be to strip "/identity" from the routes in the server (Credit to UnknownQuestions from this thread):
services.AddMvc() .AddRazorPagesOptions(o => o.Conventions.AddAreaFolderRouteModelConvention("Identity", "/Account/", model => { foreach (var selector in model.Selectors) { var attributeRouteModel = selector.AttributeRouteModel; attributeRouteModel.Order = -1; attributeRouteModel.Template = attributeRouteModel.Template.Remove(0, "Identity".Length); } })) .SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
As an aside and off-topic re idsrv, I really don't like the changes to Asp.Net Core Identity. They've put so much of it into a black box - controllers and views - that even if you scaffold the UI, you're still left with a big black box in a high-visibility part of your project. I also take issue with the "/identity" routing, which basically announces to the world stuff about your stack that nobody needs to know.
/end rant/

I really don't like the changes to Asp.Net Core Identity

amen

All set on this issue -- can we close?

Follow-up on this thread: https://github.com/IdentityServer/IdentityServer4/issues/2510#issuecomment-411871543

Thanks all for the info. I wasted a day on this and decided just to go with the 2.0 asp.net identity template. Any chance of documenting the issues with 2.1 and ids4 on this link so others don't suffer :)??