Carny multi-threading and more efficient processing.

I apologize in advance for all of the unrelated changes. I added the interfaces for the new Scheduler and the new Seeker and made them only able to load the current implementations. It should be very easy to follow those changes, there's just a bunch of them making this look a lot worse than it really is. In some cases, I couldn't isolate those changes from the Carny changes without temporarily reversing things in annoying ways, so that's why some of those changes are here. Other related changes are just there so everyone will be working with the newer interfaces.
 From what I have been able to test, this code produces the exact same end result in a fraction of the time that the old implementation used. I also ended up creating a few custom objects because they performed a lot better than what Java provides out of the box.

I took a quick look through it...the Scheduler/Seeker changes you made seem like they have no impact since it's just prep for you inserting your new implementation. I'll need to take some time to go over the changes to Agent/Carny since those are significant and I want to be sure nothing got broken there.

Thanks. I wasn't sure if you were looking at things already or not and I spotted a small improvement that removed a synchronization requirement because it turned out the caching didn't make any noteworthy difference. Ergo, I remove it and the caching variables. I also started with synchronizing between all of the threads, but I quickly realized that even though the interference between all of the threads wasn't that great, the code didn't scale very well above 4 cores, so I gave each thread it's own callback and we just merge all of the results when they finish. The merging is very fast, so this actually turns out to be the fastest approach.

I did a little more analyzing and found a few more things that can be improved. One of them is we are mapping channels that have a zero length name which ends up being checked by all of the agents, yet none of them will ever match because I don't think we ever create an agent that would specifically look for a channel without a name. Fixing this gave me a full second speed up (mostly because of the keyword searches). I know these numbers sound small, but on larger databases, these should scale to substantial time savings.

I spent a bunch more time going through this, and due to the complexity of the change...I don't think I'm going to feel confident enough just merging it without further verification. What would make me feel much better was if I was able to run something with the prior version which then dumped out all of the results from one pass of Carny to a file....and then ran the new version, and it did the same thing and then we verified that the results were identical. We would also need to force a specific time of execution since that effects the results from Carny.  I know you said you tested something like that already yourself...but I'd like to run that on my DB as it's been around a lot longer and has far more entries in it. As of now, I have 613k Airings, 101k Agents, 34k Watched and 345k Wasted in my DB....and it takes a couple days for Carny to finish....so I'm very much looking forward to this optimization, but I want to verify it with a stress test like this before I merge it. :)  I still want to review the code more, but it's going to take more time until I can get around to finishing that.  Let me know what you think.

That's fine. I found a few small improvements that I would like to add to this merge if that's ok. Also if it's not a privacy issue, can I have a copy of your wiz.bin just to see what that looks like myself. I am most interested in optimizing for worse case and you're probably as bad as it gets. :)

I'll write a simple information dump into the version of Carny I have submitted that will be copy and paste compatible with the old version of Carny.

Please hold off on adding anymore to this commit. :) I'll upload my Wiz.bin to Drive and send you a link to it....I'm not that concerned about people seeing what I've watched.
And if you do the lengthy init in Carny and then have SageTV exit right after that...then I think it'll do the whole run without worrying about MediaFile existence and you'll basically get the same result as if you ran it on my system. At startup, my DB should probably take half a day or so to run...maybe less because then it's not throttling down CPU usage for the processing.  I'm going to be out of town for a few days, so if you just want to test mine yourself...I'll take your word for it that it gets matching results. :)

No problem on not committing. I don't know what kind of hardware you have, but I have an i7 4790k I can run your wiz.bin through. I'm fairly certain it will get through it in less than a day using the old approach.

I think my server is running an Athlon XP 1800+...close to 10 years
old...but it gets the job done. :D
On Wed, May 10, 2017 at 10:52 AM, Joseph Shuttlesworth <
<EMAIL_ADDRESS>wrote:

No problem on not committing. I don't know what kind of hardware you have,
but I have an i7 4790k I can run your wiz.bin through. I'm fairly certain
it will get through it in less than a day using the old approach.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/sagetv/pull/281#issuecomment-300562353, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANEIDAsNQULH7JjKmRhbayeZZdYRuU94ks5r4flvgaJpZM4NO1-m
.

--
Jeffrey Kardatzke
<EMAIL_ADDRESS>Google, Inc.

I  sold my Athlon XP 1800+ on eBay over 7 years ago due to a high power consumption for lower performance than other options at the time. Just for fun, I looked up a benchmark comparison of my best CPU vs your CPU. http://www.cpubenchmark.net/compare.php?cmp[]=221&cmp[]=2275
I forgot that CPU only has one core. I guess you won't benefit as much as others.

The optimized carny without throttling processed all of your agents in just under 39 seconds. :) Now I have to move things around and run it through the old carny for comparison.

Looks like it's going to be 3 1/2 hours. I marked the date and time I started so I can reproduce this a few times if I need to.

That sounds awesome. :)
On Wed, May 10, 2017 at 1:09 PM, Joseph Shuttlesworth <
<EMAIL_ADDRESS>wrote:

Looks like it's going to be 3 1/2 hours. I marked the date and time I
started so I can reproduce this a few times if I need to.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/sagetv/pull/281#issuecomment-300598032, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANEIDDDVa_6GR7IkwIh0B6mHCpFpu5WCks5r4hlqgaJpZM4NO1-m
.

--
Jeffrey Kardatzke
<EMAIL_ADDRESS>Google, Inc.

So this is really annoying. Everything is very consistent between runs. The same airings are selected every single time between both the old and new carny, but the WP and Cause maps are incorrect/inconsistent in the new carny compared to the old carny. I paid very close attention to how the WP value is calculated when I modified that step and I cannot see how the outcome should be different from what it was before. There aren't any discrepancies in numWatchedAirs, numWastedAirs or numManualWaste between the old and the new implementation, so the final calculations are very suspect at the moment. I'll figure it out, but I just thought I would share my progress.

I've fixed 99% of the problem and the funny thing is that fixing the problem actually resulted in another speed boost.

Sounds great. You can of course add the commits with fixes to the pull
request, and it's fine to add the other performance enhancements since you
were probably using those during your testing. I'd like the pull request to
match what you were testing with (minus the test export code....unless you
want to comment/if it out so it can be used to test any later changes)
Jeff Kardatzke
Sent from my Android
On May 11, 2017 12:15 PM, "Joseph Shuttlesworth"<EMAIL_ADDRESS>wrote:
I've fixed 99% of the problem and the funny thing is that fixing the
problem actually resulted in another speed boost.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/sagetv/pull/281#issuecomment-300889453, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANEIDMmdI7M44mRjKllwWP0AcYXZHpTWks5r414zgaJpZM4NO1-m
.

It would probably be a good idea to leave the exporting code in there, but commented out for anyone that wants to make changes so they can prove that the end result doesn't change.
So that I didn't need to keep changing the time, I have it performing a difference between now and yesterday at 4pm calculation at startup and then it subtracts the difference from the returned time for both time() and eventTime() so that it's always 4pm yesterday at startup, but also so that the time continues to progress so we don't break anything relying on the time moving forward. I don't know if this might be something of value for future testing applications and it won't be in this commit, but I'm just putting it out there.
I'm performing hopefully one last comparison to the old version which is chugging away right now. Hopefully this time I've got an exact match. Right now the WP being returned for some agents is 0.000000001-0.000000003 off from the old implementation and I think it's because something changed in Sage.properties. I should have made a backup of it before starting, so that's my fault and I'm paying for it with a 3 hour wait.

That last 1% is that the favorite priorities are not working as expected. Any time I call doBattle(), it always returns null which results in the higher priority favorite not being selected. This specifically happens on Extreme Homes. There's a first run favorite and a first run and rerun favorite. The first run and re-run favorite is the higher priority, but it consistently gets overridden.
I noticed that both agents don't have anything in the weakAgents array which is why the results are always null, but I don't understand the logic here. The order that the agents are processed in a single threaded scenario still ends up with the wrong favorite being selected for the airing. The first runs agent has a lower id and as such ends up being processed before the first runs and reruns agent. Is my logic backwards? Should I be processing in the opposite direction?

It looks like the creation time is a factor too. Is that correct? I guess I missed that kind of ordering in the old implementation.

So if I correct the ordering to account for the creation time 99.99% of the cause map between the old and the new agrees. Basically I added a small check after all of the workers are done to make sure things are in a specific order if they are out of order. This will also ensure that for non-favorites the cause agents don't shift around based on what thread processed what agent.  This really large spread of data made that more pronounced. Every run produces the exact same map regardless of thread count, so I can say that while the cause agent doesn't match for about 10 out of the 4,204 airings on the cause map, it's always the same difference in agents between the old and new. Also, they all contain the exact same airings (no more, no less).
This is what the old carny returns:
Agent[Trend=PersonTrend, Person=Bruce Mahler id=386317 watchProb=0.33333334 numWatchedAirs=4 numWastedAirs=4 numManualWaste=0 createTime=Mon 5/19 2:26:26.616 del=auto enabled=true]	A[104946335,98978,"Police Academy",33668@0511.21:00,100,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=PersonTrend, Person=Bruce Mahler id=386317 watchProb=0.33333334 numWatchedAirs=4 numWastedAirs=4 numManualWaste=0 createTime=Mon 5/19 2:26:26.616 del=auto enabled=true]	A[104950030,98978,"Police Academy",33668@0512.01:05,100,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=PersonTrend, Person=Bruce Mahler id=386317 watchProb=0.33333334 numWatchedAirs=4 numWastedAirs=4 numManualWaste=0 createTime=Mon 5/19 2:26:26.616 del=auto enabled=true]	A[105256539,98978,"Police Academy",33668@0517.21:00,105,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=PersonTrend, Person=Bruce Mahler id=386317 watchProb=0.33333334 numWatchedAirs=4 numWastedAirs=4 numManualWaste=0 createTime=Mon 5/19 2:26:26.616 del=auto enabled=true]	A[105263052,98978,"Police Academy",33668@0518.04:35,105,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=PersonTrend, Person=Bruce Mahler id=386317 watchProb=0.33333334 numWatchedAirs=4 numWastedAirs=4 numManualWaste=0 createTime=Mon 5/19 2:26:26.616 del=auto enabled=true]	A[105366994,98978,"Police Academy",33668@0520.08:00,105,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=PersonTrend, Person=Bruce Mahler id=386317 watchProb=0.33333334 numWatchedAirs=4 numWastedAirs=4 numManualWaste=0 createTime=Mon 5/19 2:26:26.616 del=auto enabled=true]	A[105368667,102357,"Police Academy 2: Their First Assignment",33668@0520.09:45,95,T]	Show[Title=Police Academy 2: Their First Assignment, Desc=After graduating in the first "Police Academy," Carey Mahoney (Steve Guttenberg) and the rest of his hapless fellow officers are assigned to a precinct commanded by Capt. Pete Lassard (Howard Hesseman), whose district is being harassed by a gang led by Zed (Bob Goldthwait). Given 30 days to shape up or lose his post, Lassard struggles to discipline his incompetent new charges, while subordinate Lt. Mauser (Art Metrano) plots to take his job away., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Michael Winslow, Colleen Camp, Bubba Smith, David Graf, Bruce Mahler, Marion Ramsey, Howard Hesseman, Art Metrano, George Gaynes, Ed Herlihy, Bob Goldthwait, Jerry Paris, ], Rated=PG-13, ERs=[Brief Nudity, Language, Mild Violence, ], Year=1985, Bonus=[93725, 93952, 9193271, ], id=102357, lastWatched=Wed 7/2 23:00:00.000, extID=MV0187610000]
Agent[Trend=PersonTrend, Person=Bruce Mahler id=386317 watchProb=0.33333334 numWatchedAirs=4 numWastedAirs=4 numManualWaste=0 createTime=Mon 5/19 2:26:26.616 del=auto enabled=true]	A[105376587,98978,"Police Academy",33668@0520.19:00,105,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=PersonTrend, Person=Bruce Mahler id=386317 watchProb=0.33333334 numWatchedAirs=4 numWastedAirs=4 numManualWaste=0 createTime=Mon 5/19 2:26:26.616 del=auto enabled=true]	A[105398904,102357,"Police Academy 2: Their First Assignment",33668@0520.20:45,90,T]	Show[Title=Police Academy 2: Their First Assignment, Desc=After graduating in the first "Police Academy," Carey Mahoney (Steve Guttenberg) and the rest of his hapless fellow officers are assigned to a precinct commanded by Capt. Pete Lassard (Howard Hesseman), whose district is being harassed by a gang led by Zed (Bob Goldthwait). Given 30 days to shape up or lose his post, Lassard struggles to discipline his incompetent new charges, while subordinate Lt. Mauser (Art Metrano) plots to take his job away., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Michael Winslow, Colleen Camp, Bubba Smith, David Graf, Bruce Mahler, Marion Ramsey, Howard Hesseman, Art Metrano, George Gaynes, Ed Herlihy, Bob Goldthwait, Jerry Paris, ], Rated=PG-13, ERs=[Brief Nudity, Language, Mild Violence, ], Year=1985, Bonus=[93725, 93952, 9193271, ], id=102357, lastWatched=Wed 7/2 23:00:00.000, extID=MV0187610000]

This is what the new carny returns:
Agent[Trend=TitleReRunsTrend, Title=Police Academy id=385391 watchProb=0.33333334 numWatchedAirs=1 numWastedAirs=0 numManualWaste=0 createTime=Tue 3/25 1:54:07.061 del=auto enabled=true]	A[104946335,98978,"Police Academy",33668@0511.21:00,100,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=TitleReRunsTrend, Title=Police Academy id=385391 watchProb=0.33333334 numWatchedAirs=1 numWastedAirs=0 numManualWaste=0 createTime=Tue 3/25 1:54:07.061 del=auto enabled=true]	A[104950030,98978,"Police Academy",33668@0512.01:05,100,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=TitleReRunsTrend, Title=Police Academy id=385391 watchProb=0.33333334 numWatchedAirs=1 numWastedAirs=0 numManualWaste=0 createTime=Tue 3/25 1:54:07.061 del=auto enabled=true]	A[105256539,98978,"Police Academy",33668@0517.21:00,105,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=TitleReRunsTrend, Title=Police Academy id=385391 watchProb=0.33333334 numWatchedAirs=1 numWastedAirs=0 numManualWaste=0 createTime=Tue 3/25 1:54:07.061 del=auto enabled=true]	A[105263052,98978,"Police Academy",33668@0518.04:35,105,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=TitleReRunsTrend, Title=Police Academy id=385391 watchProb=0.33333334 numWatchedAirs=1 numWastedAirs=0 numManualWaste=0 createTime=Tue 3/25 1:54:07.061 del=auto enabled=true]	A[105366994,98978,"Police Academy",33668@0520.08:00,105,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=TitleReRunsTrend, Title=Police Academy id=385391 watchProb=0.33333334 numWatchedAirs=1 numWastedAirs=0 numManualWaste=0 createTime=Tue 3/25 1:54:07.061 del=auto enabled=true]	A[105376587,98978,"Police Academy",33668@0520.19:00,105,T]	Show[Title=Police Academy, Desc=When the mayor of a crime-ridden city loosens the restrictions on entering the police academy in order to get more cops on the street, all manner of oddball characters enlist to join the force. Among the cadets are suave Carey Mahoney (Steve Guttenberg), hulking Moses Hightower (Bubba Smith), beautiful Karen Thompson (Kim Cattrall) and sound effects-generating Larvell Jones (Michael Winslow), who all have to show initiative and courage when they are faced with tough situations out on patrol., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Kim Cattrall, Michael Winslow, G.W. Bailey, George Gaynes, Bubba Smith, Andrew Rubin, David Graf, Bruce Mahler, Leslie Easterbrook, Hugh Wilson, ], Rated=R, ERs=[Nudity, Mild Violence, Adult Situations, Language, ], Year=1984, Bonus=[93725, 93952, 9193271, ], id=98978, lastWatched=Sun 3/23 3:00:00.000, extID=MV0171520000]
Agent[Trend=PersonTrend, Person=Ed Herlihy id=389680 watchProb=0.33333334 numWatchedAirs=2 numWastedAirs=2 numManualWaste=0 createTime=Sun 11/2 16:29:05.370 del=auto enabled=true]	A[105368667,102357,"Police Academy 2: Their First Assignment",33668@0520.09:45,95,T]	Show[Title=Police Academy 2: Their First Assignment, Desc=After graduating in the first "Police Academy," Carey Mahoney (Steve Guttenberg) and the rest of his hapless fellow officers are assigned to a precinct commanded by Capt. Pete Lassard (Howard Hesseman), whose district is being harassed by a gang led by Zed (Bob Goldthwait). Given 30 days to shape up or lose his post, Lassard struggles to discipline his incompetent new charges, while subordinate Lt. Mauser (Art Metrano) plots to take his job away., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Michael Winslow, Colleen Camp, Bubba Smith, David Graf, Bruce Mahler, Marion Ramsey, Howard Hesseman, Art Metrano, George Gaynes, Ed Herlihy, Bob Goldthwait, Jerry Paris, ], Rated=PG-13, ERs=[Brief Nudity, Language, Mild Violence, ], Year=1985, Bonus=[93725, 93952, 9193271, ], id=102357, lastWatched=Wed 7/2 23:00:00.000, extID=MV0187610000]
Agent[Trend=PersonTrend, Person=Ed Herlihy id=389680 watchProb=0.33333334 numWatchedAirs=2 numWastedAirs=2 numManualWaste=0 createTime=Sun 11/2 16:29:05.370 del=auto enabled=true]	A[105398904,102357,"Police Academy 2: Their First Assignment",33668@0520.20:45,90,T]	Show[Title=Police Academy 2: Their First Assignment, Desc=After graduating in the first "Police Academy," Carey Mahoney (Steve Guttenberg) and the rest of his hapless fellow officers are assigned to a precinct commanded by Capt. Pete Lassard (Howard Hesseman), whose district is being harassed by a gang led by Zed (Bob Goldthwait). Given 30 days to shape up or lose his post, Lassard struggles to discipline his incompetent new charges, while subordinate Lt. Mauser (Art Metrano) plots to take his job away., Cat=Movie, SubCat=Comedy, People=[Steve Guttenberg, Michael Winslow, Colleen Camp, Bubba Smith, David Graf, Bruce Mahler, Marion Ramsey, Howard Hesseman, Art Metrano, George Gaynes, Ed Herlihy, Bob Goldthwait, Jerry Paris, ], Rated=PG-13, ERs=[Brief Nudity, Language, Mild Violence, ], Year=1985, Bonus=[93725, 93952, 9193271, ], id=102357, lastWatched=Wed 7/2 23:00:00.000, extID=MV0187610000]

If this is an acceptable deviance, then I'm ready to commit what I have.

Sorry for the delay, was out of town. :)  Yeah, if that's the only difference then that's fine. The cause maps being identical doesn't really matter, as long as the watch probabilities are the same.

I knew you weren't available. Yes. The favorites have the correct agent and the watch probability is exactly the same between implementations. I was extra particular about the favorite agents because you had a note in the code about when that's wrong we can end up not correctly prioritizing scheduling.

I'm happy with what is currently committed. I don't think I'll have much desire to revisit this, barring any issues that need to be fixed, since I've explored tons of avenues based on profiling to get the most out of this new implementation. The only downside is if you have a ton of shows, memory consumption can become a concern and that's why I added the option to turn the maps off which comes at the sacrifice of a lot of speed. As a reference, no maps uses 352MB total heap and with maps uses 420MB total heap when using the massive wiz.bin.

Initial review done....that took some time. :D

And by initial....I mean I read over all the code, so please act on the comments and then once they are all resolved we can merge.

Have you thought about just how much energy savings this is going to be across the whole user base? It's A LOT. :D My own electric bill will likely go down by at least a few $$ a month.

Strangely enough I didn't really think about energy savings since my motivation was getting Carny up and running faster for my testing of Scheduler2 and Seeker2. I suspect this will enable many computers to actually idle, but I did note that on my testing computer as soon as all of the recording space filled up, old recordings were being deleted frequently enough that it didn't idle as much as you might hope. However, because we're processing less recordings per agent it should turn into some cost savings. :)

Oh, and before you think I'm trying to pull a fast one on you, getHashes() was casting to short from some  testing I was doing before I accidentally committed the casting which isn't supposed to happen. If that casting is there, we you will end up not getting all related recordings. I forgot to note that I removed the casting in the commit. :) The reason I  didn't notice was because I was shifting by 16 bits. Also as a fun fact, I tried Short objects before I realized due to alignment and overhead, Short, Integer and Long all use the same amount of memory. There is literally no memory consumption advantage if it's boxed that I could see.

All looks good, just wanted you to confirm that you verified with this final version that we get the same results for processing both of our databases (with the allowed exceptions we already discussed). Once you confirm that, then I shall merge. :)

I shall test those results once more today and hopefully will be done by this evening.

I can positively conclude that switching if (doBattle(compareAgent, agent) == compareAgent) now ensures that we get the wrong agent for favorites every single time.

Based on what you said I'm not sure why that's happening. I have some ideas, but I still find it a little odd.

I keep feeling like somehow very subtly I'm using doBattle() backwards because when I'm working with an agent that should be a higher priority, it always returns null which then results in no agent change.

If I add checking for null and replacing on null everything works correctly again.

Can you be more specific about what you mean by checking for null?
Jeff Kardatzke
Sent from my Android
On May 21, 2017 7:21 AM, "Joseph Shuttlesworth"<EMAIL_ADDRESS>wrote:
If I add checking for null and replacing on null everything works correctly
again.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/sagetv/pull/281#issuecomment-302939718, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANEIDPLacjjcDheu9lQI8od7o8tn5tSFks5r8EhsgaJpZM4NO1-m
.

doBattle() seems to always return null when the other agent is stronger. It doesn't make a lot of sense.

This is what the code looks like after I changed it if that help paint the picture better:
      if (compareAgentWP == wp && compareAgent.isFavorite())
      {
        // Check for agent priority. This returns the higher priority agent.
        Agent winner = doBattle(agent, compareAgent);
        if (winner == null || winner == compareAgent)
        {
          agent = compareAgent;
          wp = compareAgentWP;
          return true;
        }
      }

That makes sense because that's the case when their is no explicit priority
set yet between those agents. This is what leads to an unconfirmed
recording conflict vs. a confirmed. It also then matches the old code
exactly if you do it this way. ☺
Jeff Kardatzke
Sent from my Android
On May 21, 2017 10:12 AM, "Joseph Shuttlesworth"<EMAIL_ADDRESS>wrote:

This is what the code looks like after I changed it if that help paint the
picture better:
  if (compareAgentWP == wp && compareAgent.isFavorite())
  {
    // Check for agent priority. This returns the higher priority agent.
    Agent winner = doBattle(agent, compareAgent);
    if (winner == null || winner == compareAgent)
    {
      agent = compareAgent;
      wp = compareAgentWP;
      return true;
    }
  }

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/sagetv/pull/281#issuecomment-302949740, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANEIDObkewyX1K9a4zEu-ir3e-5hk50cks5r8HB9gaJpZM4NO1-m
.

Ah. Cool. So it's not wrong, it's just not obvious that this is acceptable. :)

The old code looks like this:
        if (wp != null && currAgent.watchProb == wp.floatValue() && currAgent.isFavorite())
        {
          // Check for agent priority
          Agent oldAgent = newCauseMap.get(agePots[j]);
          if (oldAgent == null || doBattle(oldAgent, currAgent) == currAgent)
            replaceInMap = true;
        }

I'm not sure where you're doing a null check here other than checking if it's on the map. Are you just saying it matches the old code because the old code does this in order?

I was wrong about it matching the old code...and then actually what you're doing with the null check doesn't actually make sense, because then you're replacing the value even though there is no explicit prioritization by one over the other.
I'm still wondering about where you compare creation times for the Agents, and then swap them if the create time is newer...because I never did anything like that in my code. Maybe that behavior, combined with the inverse battle behavior is what is ending up with the matching result.  I wonder if you removed that creation time comparison, used Wizard.getAgents() instead of getRawAccess, and then did the battle the 'correct' way....if that would then overall give the same matching result.

If I use AGENT_SORTER when I get null, I seem to get the correct results now. That might have been what I was missing when you suggested it earlier to get the ordering correct.
      if (compareAgentWP == wp && compareAgent.isFavorite())
      {
        // Check for agent priority. This returns the higher priority agent or null if there isn't
        // any explicit priority set. When this is null there is an implicit priority that based on
        // creation time. The older of the two takes priority.
        Agent winner = doBattle(agent, compareAgent);
        if (winner == null && ordered && AGENT_SORTER.compare(agent, compareAgent) > 0 ||
          winner == compareAgent)
        {
          agent = compareAgent;
          wp = compareAgentWP;
          return true;
        }
      }

The new parameter ordered is a boolean on if they need to be sorted or not. We don't need to sort until the last step, so it's a waste to do it when the agents are being pulled from the queue because at that stage, they are ordered according to each thread. The final merge is when everything needs to be re-ordered on matching WP.

I got the create time sorting idea from agentCreateTimeSorter, which is used by sortAgentsByPriority(), BTW.

In the old implementation we use DBObject[] allAgents = wiz.getRawAccess(Wizard.AGENT_CODE, (byte) 0); to get all of the agents and then process them in that order. I believe 0 is sorted by ID in all cases from the Wizard and I tried enforcing the sort by ID, but that didn't get me the results I was expecting either. I'm reasonably sure that's why AGENT_SORTER doesn't always get me results fully comparable to the old implementation.

It does then make another call and sets allAgents from Wizard.getAgents,
which copies from the Index that is sorted by AGENT_SORTER. Another example
of sloppy coding by me from 15 years ago. ☺ I had a lot of late nights
writing code.
Jeff Kardatzke
Sent from my Android
On May 21, 2017 3:33 PM, "Joseph Shuttlesworth"<EMAIL_ADDRESS>wrote:
In the old implementation we use DBObject[] allAgents =
wiz.getRawAccess(Wizard.AGENT_CODE, (byte) 0); to get all of the agents and
then process them in that order. I believe 0 is sorted by ID in all cases
from the Wizard and I tried enforcing the sort by ID, but that didn't get
me the results I was expecting either. I'm reasonably sure that's why
AGENT_SORTER doesn't always get me results fully comparable to the old
implementation.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/sagetv/pull/281#issuecomment-302967942, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANEIDGCD_pbWz4IADoA4YG5oIQtPpQhsks5r8LucgaJpZM4NO1-m
.

And that last code example you showed looks like it should be logically
correct. So if that gets matching output, go ahead and commit that and we
can be done with this. 😀
Jeff Kardatzke
Sent from my Android
On May 21, 2017 3:44 PM, "Jeffrey Kardatzke"<EMAIL_ADDRESS>wrote:

It does then make another call and sets allAgents from Wizard.getAgents,
which copies from the Index that is sorted by AGENT_SORTER. Another example
of sloppy coding by me from 15 years ago. ☺ I had a lot of late nights
writing code.
Jeff Kardatzke
Sent from my Android
On May 21, 2017 3:33 PM, "Joseph Shuttlesworth"<EMAIL_ADDRESS>wrote:
In the old implementation we use DBObject[] allAgents =
wiz.getRawAccess(Wizard.AGENT_CODE, (byte) 0); to get all of the agents
and then process them in that order. I believe 0 is sorted by ID in all
cases from the Wizard and I tried enforcing the sort by ID, but that didn't
get me the results I was expecting either. I'm reasonably sure that's why
AGENT_SORTER doesn't always get me results fully comparable to the old
implementation.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/sagetv/pull/281#issuecomment-302967942, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANEIDGCD_pbWz4IADoA4YG5oIQtPpQhsks5r8LucgaJpZM4NO1-m
.

Dang. I missed that. Let me see what where I can go with that approach. This must be agonizing to watch. :)

Awesome. This is even better at getting the agents correct. It also fixed the Police Academy example.

Great!
Jeff Kardatzke
Sent from my Android
On May 21, 2017 4:11 PM, "Joseph Shuttlesworth"<EMAIL_ADDRESS>wrote:

Awesome. This is even better at getting the agents correct. It also fixed
the Police Academy example.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/sagetv/pull/281#issuecomment-302969797, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ANEIDOceQPV5xEuNIpEplLirQbsUTuzyks5r8MSdgaJpZM4NO1-m
.

Ok. I compared my own database and your database against the old Carny and everything is matching.

Just waiting for Travis to finish it's test and then I will merge. Congrats on the largest contribution to the SageTV core outside of myself. 😀

You're in the SageTV core??? ;) Thanks. I really like SageTV and I want to see it only get better.


Expose (Q)SPI pins to access flash memory

This patch exposes pins connected to the flash memory on board (IS25LP064A) through the Board abstraction.
Although this does not introduce a high level library to access the memory, it does allow for experimentation with standard SPI and QuadSPI interfaces available in stm32h7xx-hal.
Meanwhile, I'm preparing a simple driver for the memory,

Thank you for your contribution @phoracek.
The work you're doing on a driver look really promising!
I've released your PR in v0.4.2.

This BSP and your posts on flowdsp.io kickstarted my new hobby and helped me a ton :) So thank you!


maxWidth/maxHeight loops when interacting with preferred sizes

There seem to have been two buggy cases:

Buttons with certain settings and appearance strategies were larger than their preferred sizes. Thus if the preferred and maximum sizes were the same, this would create an infinite loop. Fixing the sizing of the node resolves this issue.
Setting a maximum size that is SMALLER than the preferred size creates an infinite loop.

I'll be committing shortly resolutions to (1), however (2) is an outstanding issue of how to handle. Presumably we should either:

Detect it and error out with a helpful message, instead of an "unclear stack error"
Silently handle this by using some sort of "effective preferred size" that is internally used, that is defined as "the minimum between the preferred size and max size if both are non-null".
Adjust the preferred size to be the max size.

Unclear yet what the best approach here is to me. @marlitas thoughts?

Hmmm... from a developer standpoint I would appreciate receiving a helpful error message as this seems like behavior I would not intentionally create. Although handling it silently may be more graceful and save me from pitfalls I don't know about, it feels sneaky and like it could potentially be hiding a bigger bug in code I would be writing further down the line.
Option 3 feels similar to option 2 in that it would handle this buggy situation silently, and that seems less preferable to me right now.

I'm definitely feeling the "assertion" route right now, we can always reevaluate later if we're hitting these in "valid" cases somehow.
@marlitas does the above commit look good to you?

Does auditMaxDimensions not need to return a boolean so that the asserts that trigger that function can move through?
I thought a void function would return undefined which evaluates to falsy and then trigger the original assertion no matter what?

Talked through my questions with @jonathanolson and feels good to close!


Updated angular ui-router to avoid

Using older angular router causes "transition silent error"

@torabian thank you for your contribution!


New release coming ?

Last release was on Jun 3, 2015

@7aleb I've been maintaining a isort linter for Atom: https://github.com/lexicalunit/atom-isort The last update I made was just a couple months ago.


escape character on search

Description
escape character to avoid error on package search
Fixes # (THEEDGE-2871)
Type of change
What is it?

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Tests update
[ ] Refactor

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I run make pre-commit to check fmt/vet/lint/test-no-fdo

Codecov Report
Base: 42.58% // Head: 42.58% // No change to project coverage :thumbsup:

Coverage data is based on head (d5eb1cc) compared to base (7f71770).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1706   +/-   ##
=======================================
  Coverage   42.58%   42.58%
=======================================
  Files          67       67
  Lines        9571     9571
=======================================
  Hits         4076     4076
  Misses       5058     5058
  Partials      437      437

Impacted Files
Coverage Δ

pkg/clients/imagebuilder/client.go
40.23% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Adressing

I'm planning to add a new adress register, allowing to perform some adress math (finally?).
For example:
bpl r1      ; Base Pointer Load
stm r1, 2 ; Store and Multiply
sta r1, 5  ; Store and Add
lev r2      ; Load Effective Value (deref)
lea r1      ; Load Effective Adress

; r1 = 2*r1+5
; r2 = rcl (2*r1+5)

It may be possible to add an preprocessor to parse the algebra expression and output the valid adress math code.

closed in favor of d325292


Generate pdf with graph subcomponents

@petrs Any idea what you meant by this issue? I'll probably close this unless it's really related to our current state of art.

@adamjanovsky This was meant to generate multiple pdf files instead of single one with all cert_ids (which is large and messy). But is only of potential use, no priority

Graphs are not interactive on web or can be rendered on-demand in notebooks. Closing this as won't fix.


Fix a race condition in the bundle command

These two changes fix a particular egregious race condition in the rnws bundle command on really performance-challenged machines, e.g. whatever steam-powered 19th century hardware Travis uses. The symptom would be Error: socket hang up in the part that attempts to fetch the bundle from the webpack server, which isn't ready yet. The only explanation that I have for this is that fetch is simply called too early sometimes. So the solution is to ensure it's only called when the server(s) is(are) definitely ready.
In this PR, we ensure that

the RN package server doesn't get started until the webpack compiler not only has written its entry file but the webpack server is actually listening
fetch won't be called until the RN package server is definitely up, which given the previous bullet point implies that the webpack server is listening

(Disclaimer: I'm not entirely certain the first change is 100% necessary, but it helps us a lot in reasoning about when the webpack server has actually started up. It makes the promise chain simpler and I doubt that having it in place will have any impact on startup perf -- I haven't observed any, but I also haven't done any scientific measurements tbh.
If there are any concerns, we could alternatively have the RN package server continue to be launched after the entry file was written, and then make Server.start() return a Promise.all based on the RN package server and the webpack server. So long as we don't fetch before both servers are definitely listening on their ports... I chose not to go down that route because IMHO having _startWebpackDevServer resolve the promise when the server is actually listening made more sense to me.)
cc @lightsofapollo

@elliottsj do you think you could look at these changes and perhaps release a 0.7.2 if you're ok with them?

\o/ thanks!

:+1: released as 0.7.2


Dashboard-Resource: Same Description is shown for Bucket, Volume, VolumeGroup... in REsource Menu

1 Click REsource Menu
2. Click on each Tabs .. Bucket, Volume, Volume Grp...
3. Same description is shown..
4. Should show description about the clicked tab-feature
or.. if the description is about Resource Menu.. it should be above the Tabs

@satya-gorli , please comment on the expected behavior as well...

expected Behavior:
Should show description about the clicked tab-feature
and.. if the description to be given about Resource Menu.. it should be above the Tabs

@satya-gorli  We have made the following change:
Buckets: View and manage Buckets that have been manually created or applied for through service templates.
Volumes: View and manage Volumes that have been manually created or applied for through service templates.
Volume Groups: View and manage Volume Groups that have been manually created or applied for through service templates.
File Share: View and manage File Shares that have been manually created or applied for through service templates.


Invalid Syntax at Line 51

fatih@fatih-Workstation:~/huion-linux-drivers$ python ./huion-tablet-driver.py
  File "./huion-tablet-driver.py", line 51
    file=sys.stderr)
        ^
SyntaxError: invalid syntax

You ran it with python2, it's supposed to be run with python3


[BUG] After the automatic update, all download progress reset

Describe the bug
After the automatic update of the launcher, all the download progress of Five Nights at Freddy Security Breach has reset. The version before the update was 1.2.4
Steps to Reproduce

Downloaded the launcher version 1.2.4 (a long time ago)
Started downloading Five Nights at Freddy Security Breach, download option - KaOS
Closed the program completely
Started the program again
Automatic update started
After installing and running the update, all download progress disappeared

Expected behavior
That downloads that due to slow internet (100-200 kilobytes per second) and lack of distribution will not be lost from the library
Screenshots
No response
Operating System
Windows 10 Pro
Hydra Version
2.0.1
Additional Information
No response

Could you please let me know if this bug persists?


An error from counter-based RNG example: .parent is of type Array{Tuple{Tuple{Int64}},1} which is not isbits.

https://julialang.zulipchat.com/#narrow/stream/225542-helpdesk/topic/Aside.3A.20Reproducible.20multithread.20random/near/215662714

The demo (examples/monte_carlo_pi.jl) fails with FoldsCUDA.jl v0.1.0.
It still works with the latest upstream packages in dev version https://github.com/JuliaFolds/FoldsCUDA.jl/pull/31/commits/ee66b3dcdb7a612203e066b98cb10cbf898d1b51
Apparently, this was already fixed by 400341bcc750a86ca5ba7e0ddf341909900445e0 but I forgot to release the new version.

0.1.1 released now 9737a42fe3a900e2d3fc8e4191cbecfc8c73b7f0


Mac - The tray icon is duplicated when connected and disconnected

The tray icon in mac replicates multiple times each time a disconnect happens.
Version : OSX Yosmite 10.10.2

@yohangz s perspective was, with the Mac behavior we don't need to tray icon in Mac, just need to remove it for Mac build.

Fixed with Mac Latest Build


Add RestartPolicy to QueueJob

referering #271

@etiennecoutaud , I think we may need a design doc for that; for example, what the action we will take if task failed? e.g MPI job.

@etiennecoutaud , I'm going to close this PR for QueueJob, that'll be great if you can contribute PodGroup controller, similar to this PR.

Sure, I will check first good issue label, do you have priority issues to work on ?

for first good issues, you can try the issues that I opened recently :)


Incorrect pp calculation if it isn't a fc in other gamemodes (catch/taiko/mania)

Example code:
import fetch from 'node-fetch';
import { ctb_ppv2 } from 'booba';

const API_KEY = 'put api key here'; // osu! api v1 key
const USER = '4158549';

(async () => {
  const response = await fetch(`https://osu.ppy.sh/api/get_user_recent?k=${API_KEY}&u=${USER}&limit=1`);
  const json = await response.json();
  const [score] = json;

  const pp = new ctb_ppv2().setPerformance(score);

  console.log(await pp.compute())
  console.log(await pp.compute(true))
  // Both have the same output
})();

To make it simpler in the example code I use api v1 but in my project I'm using api v2
Attachments:

I have now added fc simulation to catch and taiko in 0.0.15, but it doesn't make much sense for mania since we can't know how much score the player would've gotten without messing up and mania pp is mostly based on score.


podman: add mac-helper patch

Description of changes
closes #204874

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

I would have preferred that the other package maintainers had a chance to review/approve this.


Filter projects_and_groups configuration using project topics

Problem description
The current configuration syntax allows the usage of * wildcards to apply a configuration to all projects or groups (or group/subgroup). However, sometimes multiple projects that share a common characteristic are scattered across multiple (sub-)groups. This makes it hard to apply the same configuration to multiple projects with said common characteristic without moving them into the same (sub-)group.
Imagine the following structure:
my_group
└─ my-product-a (group)
   └─ cool-python-app (python project)
   └─ really-cool-java-app (java project)
└─ my-product-b (group)
   └─ awesome-python-app (python project)
   └─ cool-application (python project)

In this scenario, I would like to apply a configuration to all python projects... (and only to the python projects!)
Implementation proposal
GitLab allows adding "topics" to a project. Topics are single strings that annotate a project and allow searching / filtering for a topic. As described in the example above, all python projects could be annotated with the a topic identified as python.
This proposal would introduce a new configuration syntax that filters based on topics. The topic filter is indicated by the ? character that follows appended to the * wildcard. The topic filter then allows specifying one or multiple topics (separated by ,) that are required for the specified configuration:
projects_and_groups:
  "*?<topic>":
    # common-level config filtered by <topic>
  group_1/*?<topic>:
    # group-level config filtered by <topic>
  group_2/*?<topicA>,<topicB>:
    # group-level config filtered by <topicA> and <topicB>

Example configuration
projects_and_groups:
  "my_group/*?python":
    # common-level config for python projects

Related issues / features
https://github.com/gitlabform/gitlabform/issues/325

One work-around could be yaml anchors. You can move the shared configs into one or more yaml anchor and then apply them wherever needed.

One work-around could be yaml anchors. You can move the shared configs into one or more yaml anchor and then apply them wherever needed.

That is true in terms of re-usability of configuration. I was thinking more in the direction of not having to explicitly specify my projects in the config.yml but rather apply the configuration based on a predicate (e.g. "a python project").

Hi @armingerten!
Thank you for the detailed feature proposal.
We will take it into consideration when designing v4 of the app, but I cannot commit to any deadlines for it, sorry (see #343 for more info).

After thinking a bit more about it I think this feature could be very useful and fill an important feature gap - allow to provide shared configs for entities regardless of their location in the group hierarchy.
I even had a case where I would use it at Egnyte very recently - a Python & Golang team's projects are scattered within a few GitLab groups split by product/service, but they should all have some shared config related to that team...
But I am not sure if I completely understand how it would be used so can you please provide a semi-complete usage example, @armingerten? I mean a config where you define f.e. 2 topics in the config and then use it for a few projects/groups.
(We could also then compare how this would in practice differ from just using YAML anchors.)

One of the unclear to me things is that in the GitLab API projects can have topics assigned (there is "topics" attribute listed here), but groups cannot (no "topics" here).

@jimisola / @lfvjimisola: can you take a look at this proposal too, please? As we have discussed the syntax of the next major version of GitLabForm.

But I am not sure if I completely understand how it would be used so can you please provide a semi-complete usage example, @armingerten? I mean a config where you define f.e. 2 topics in the config and then use it for a few projects/groups.

Yes, of course.
Example 1
The problem
Let's imagine, we have a team called "team red" that is responsible for multiple projects. The team decided that they have strict rules about merge requests:

Only "fast forward" merges should be allowed
At least one team member other than the author must review and approve every merge request

Unfortunately, the projects of team red are scattered across multiple groups within the GitLab instance.
Usage Example
Step 1: Define the "annotated" configuration
The first step is to define a configuration you would like to apply. Unless we intend to limit this to a specific group (and its descendent groups), we can use the * wildcard. Next, we define a topic team-red that will later be added to all of team red's projects.
The following entries in the config.yml describe team red's merge request policies:
projects_and_groups:
  "*?team-red":
    project_settings:
      merge_method: ff

    merge_requests:
      approvals:
        approvals_before_merge: 1
        merge_requests_author_approval: false

Step 2: Add the topic to all projects that belong to team red
There is now essentially two options to do that.
The first option (that inspired me for this feature proposal) is obviously through the user interface. While GitLab allows managing topics on an instance level (e.g. through the Topics API), it is not a hard requirement that topics need to be declared on an instance level before a project can reference it. Once you add the string team-red to the project's list of topics, it is instantly available. Do that with all of team red's projects and you are good to go!
The second option uses GitLabForm. Let's say team red is working on the products foo and bar. The have a project called awesome as part of product foo and a project cool as part of product bar. A GitLabForm configuration for projects would then look like this
projects_and_groups:
  product-foo/awesome:
    project_settings:
      topics: "team-red"

  product-bar/cool:
    project_settings:
      topics: "team-red"

Especially if you are intending to manage every single project individually with GitLabForm, this implementation is very similar to simply using YAML anchors.
Example 2
The problem
Another example would be to use this proposed feature to introduce "rules" that can generically be applied to your projects without needing to know the configuration of GitLabForm. This examples evolves around two roles:

The GitLab instance operator that also operates GitLabForm
The developer that has no deeper knowledge about GitLabForm but wants to apply the rule to their project

Let's assume, we want to add two rules on an instance level:

Rule 1: Strict merge request policies (the example from above)
Rule 2: Use conventional commits

Usage example
As with the first example, we start by adding the annotated configuration to GitLabForm's config.yml. This is done by the GitLab instance / GitLabForm operator:
projects_and_groups:
  "*?strict-merge-requests":
    project_settings:
      merge_method: ff

    merge_requests:
      approvals:
        approvals_before_merge: 1
        merge_requests_author_approval: false

  "*?conventional-commits":
    project_push_rules:
      commit_message_regex: '^((build|chore|ci|docs|feat|fix|perf|refactor|revert|style|test)(\(\w+\))?(!)?(: (.*\s*)*))'

Now, whenever a developer (unaware of the implementation details of GitLabForm) wants to add such policies to their projects, they simply add the topics strict-merge-requests and/or conventional-commits to their project. Not only does this de-couple the implementation detail from the usage - it also well visible within the GitLab UI, to which rules a developer/contributor needs to adhere when contributing to the project.
Conclusion
Of course these are just very simple examples. I am convinced that this feature would give the users a lot of flexibility to design their project configuration (and probably required repo files, etc.), without minding the group / project hierarchy.

One of the unclear to me things is that in the GitLab API projects can have topics assigned (there is "topics" attribute listed here), but groups cannot (no "topics" here).

Yes, to my knowledge that is true.
The way I am intending to use this feature, the topic "marks" projects that should receive a specific configuration. If I want to mark a whole group to receive a specific configuration, I am already quite happy with the wildcard (i.e. <my-group>/*) GitLabForm is offering today. I can see how this also has its limits as described in https://github.com/gitlabform/gitlabform/issues/325 .

From a config syntax perspective, the following can be confusing.
projects_and_groups:
  "*?team-red":
    project_settings:
      merge_method: ff

To me, it is unclear what project I'm applying the config. Maybe add topic as a query parameter. Something like ?topic=team-red.
There could also be use case to restrict the above to certain groups only instead of all projects in the entire instance. Then maybe we can do following.
projects_and_groups:
  "group1/?team-red":
    project_settings:
      merge_method: ff

Also does syntax potentially make it harder to capture list of projects using regex pattern instead of wildcard, in the future. Using regex pattern could be another useful pattern. For example all projects that has docs- prefix should have same config. Although this could be addressed by utilizing topics too.

I think the inheritance feature might need to be enhanced or clarified where it can be used.

One of the unclear to me things is that in the GitLab API projects can have topics assigned (there is "topics" attribute listed here), but groups cannot (no "topics" here).

Another one of those classic GitLab situations where there are differences between projects and groups. We already know that they are working on merging groups and projects into namespaces (here) but it I'm hoping that we'll be releasing the next major version of GitLabForm before :)
That said, we can implement this but specifying a topic for a group will only affect projects under that group.

@armingerten Thank you for your feature request will detailed examples and use-cases. I like it a lot. Can see how we can use this at my workplace as well. I don't think that I have paid attention to topics before, so thank you.

One of the unclear to me things is that in the GitLab API projects can have topics assigned (there is "topics" attribute listed here), but groups cannot (no "topics" here).

Yes, to my knowledge that is true.
The way I am intending to use this feature, the topic "marks" projects that should receive a specific configuration. If I want to mark a whole group to receive a specific configuration, I am already quite happy with the wildcard (i.e. <my-group>/*) GitLabForm is offering today. I can see how this also has its limits as described in #325 .

I honestly see the same type of use-case that you presented for projects but for groups.
Let's assume that we have topics for groups as well. If I specify a topic for a group what should it be applied to?

only group and sub-groups with that topic
only sub-projects with that topic
1 and 2

I think we should find a solution that is 1) logical and 2) will work the same way when GitLab merges groups and projects into a common entitiy: namespaces.

From a config syntax perspective, the following can be confusing.
projects_and_groups:
  "*?team-red":
    project_settings:
      merge_method: ff

To me, it is unclear what project I'm applying the config. Maybe add topic as a query parameter. Something like ?topic=team-red.
There could also be use case to restrict the above to certain groups only instead of all projects in the entire instance. Then maybe we can do following.
projects_and_groups:
  "group1/?team-red":
    project_settings:
      merge_method: ff

Also does syntax potentially make it harder to capture list of projects using regex pattern instead of wildcard, in the future. Using regex pattern could be another useful pattern. For example all projects that has docs- prefix should have same config. Although this could be addressed by utilizing topics too.

Thank you for your comment. Our current draft of configuration syntax v4 allows for use of regexp on at least group/project names.
We have couple of suggestion so far (pls correct me if I missed any):

"*?"
"?

We might also need to differentiate if it should apply to only groups, only projects or both.
Ideally, I would like for our new syntax to backwards/forward compatible ones we release v4.
With that I mean that we should be able to add something else like topics in the future and still be backwards compatible.
Also, should I be able to only specify one topic or list several?
With your syntax proposal it have reworked it into alternative 1. We have to lookup what characters are valid in a group/project path and choose something else. Using colon ":" here as I want to avoid * and ? which are used in wildcards, in regexps in URLs etc:
Alternative 1:
projects_and_groups:
  "*group1/:topics=team-red,team-blue;xxx=strict_project,more_strict_project":
    project_settings:
      merge_method: ff

However, for some of the configuration syntax in v4 we're planning to use Node tags (https://yaml.org/spec/1.2.2/#tags).
v4  could have a comma-separated list for topics and later on we kan add other.
Alternative 2:
Also, regarding v4 configuration syntax the current draft is using keys include and exclude on group/project level (where value can be id, name or regexp (by using a custom resolver with node tags).
namespaces:
  groups:
    “description/name for some group rules”:
      include: <name, id and PCRE2 regexp>
      exclude: <name, id and PCRE2 regexp>

We have some variations in the draft for the include/exclude value: should it singular/plural, a list etc. But, to simplify the value can be e.g.:

"123": group/project id (implicit)
"id: 123" group/project id (explicit)
"name: someProjectOrGroup": group or project name
"regexp: ": regular expression (match id, name or both)

it would be convenient to add topics as well.
namespaces:
  groups:
    “description/name for some group rules”:
      include:
         - some/path
         - topic: team-read
      exclude: <name, id and PCRE2 regexp>

Would match all groups and projects under path "some/path" with topic = team-read.

I wonder whether we should consider converting this feature request to be more generic.
Ability to retrieve projects using various filter option instead of being restricted to just topic.
According to GitLab's project api doc on list projects, we can retrieve projects using various filters. For example:

archived
id_after
id_before
imported
last_activity_after
last_activity_before
membership
min_access_level
owned
starred
topic
topic_id
visibility
with_issues_enabled
with_merge_requests_enabled
with_programming_language

Aside from topic, I could see use case for applying different configuration using visibility field. For example: different set of config based on whether the project is private, public or internal. Otherwise, you'll have to duplicate the same info as a "topic" value to achieve the same thing.
Although I can't think of any right now but may be there could be interesting uses cases through other fields listed above. Perhaps some special config for starred projects.
The syntax implementation probably shouldn't hard code those fields. Instead we could try think of "raw parameter passing" type setup that we already have for configuring project level settings.

@animas I like the idea and agree. There is a big difference between include and exclude in that the API does not allow us to exclude directly using the API. We have to do that in a 2nd step. But, that's a implementation detail.

@jimisola @amimas Thanks for all the input and suggestion. I really like how this idea is evolving!

There could also be use case to restrict the above to certain groups only instead of all projects in the entire instance. Then maybe we can do following.
projects_and_groups:
  "group1/?team-red":
    project_settings:
      merge_method: ff

Indeed, I think the "filter for topics" should filter the list of projects (or groups) that precedes the "filter separator". So assuming the filter separator is ?, the following principles would apply:

*?team-red: From all groups and projects, only those with topic team-red
group1/*?team-red: Group1, descendent groups and all projects within group1 (and its decedents) filtered with topic team-red

Also, should I be able to only specify one topic or list several?

Several 😉

From a config syntax perspective, the following can be confusing.
projects_and_groups:
  "*?team-red":
    project_settings:
      merge_method: ff

To me, it is unclear what project I'm applying the config. Maybe add topic as a query parameter. Something like ?topic=team-red.

Hmm, I see your point ... I was already thinking that the ? character might be confused with a "query parameter". Maybe # would be a better fit? 🤔
projects_and_groups:
  "groups/*#team-red,team-blue":
    project_settings:
      merge_method: ff

Also, regarding v4 configuration syntax the current draft is using keys include and exclude on group/project level (where value can be id, name or regexp (https://github.com/yaml/pyyaml/issues/457#issuecomment-1030691684).
it would be convenient to add topics as well.
namespaces:
  groups:
    “description/name for some group rules”:
      include:
         - some/path
         - topic: team-read
      exclude: <name, id and PCRE2 regexp>

Hmm, I suppose using include/exclude rather than modifying the path string definitely increases readability.

I wonder whether we should consider converting this feature request to be more generic.
Ability to retrieve projects using various filter option instead of being restricted to just topic.

You are right - that is actually a brilliant idea! Using topics seemed obvious to me because a topic acts as a "label" and "labeling" / "tagging" / etc. and then filtering by label / tag is a popular approach in many systems ... But why should this be limited to filtering by topic if projects expose other properties as well?
One could even consider filtering projects / groups by processing the returned API objects using yamlpath 🤔 .

I've updated #331 and created a wiki page for the syntax here


[Feature Request] the split editor can be drag onto another screen

Now VsCode can split  editor. But It's not comfortable when the  screen is small, like Mac pro 13 inches. So I hope the split editor can be drag onto other screen. This is very useful for our coding.
Thanks.

https://github.com/Microsoft/vscode/issues/10121


proc macro 2.0

Documents the changes in https://github.com/vulkano-rs/vulkano/pull/1062
TODO: Remove test paths in Cargo.toml before merging.

Reopening against dev branch


Maintained

Is this plugin still maintained?

yep. but need to recheck on current version of pritunl

Cool :) good to hear!


RuntimeError on SoundScrape OSX

Hi all, I get this error when trying to use SoundScrape, even when trying to update it or uninstall it. I have already updates my OpenSSL so I'm not sure what's causing the problem, and this occurs when I try to uninstall SoundScrape through "pip uninstall soundscrape", so I can't reinstall it. Sorry for the formatting, my first time opening an issue.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/pip", line 7, in
from pip import main
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/init.py", line 15, in
from pip.vcs import git, mercurial, subversion, bazaar  # noqa
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/vcs/mercurial.py", line 10, in
from pip.download import path_to_url
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 38, in
from pip._vendor import requests, six
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/init.py", line 53, in
from .packages.urllib3.contrib import pyopenssl
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/contrib/pyopenssl.py", line 53, in
import OpenSSL.SSL
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/OpenSSL/init.py", line 8, in
from OpenSSL import rand, crypto, SSL
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in
from OpenSSL._util import (
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in
from cryptography.hazmat.bindings.openssl.binding import Binding
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 250, in
_verify_openssl_version(Binding.lib.SSLeay())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 230, in _verify_openssl_version
"You are linking against OpenSSL 0.9.8, which is no longer "
RuntimeError: You are linking against OpenSSL 0.9.8, which is no longer support by the OpenSSL project. You need to upgrade to a newer version of OpenSSL.

pip install requests[security]
pip install soundscrape
Does that work?


EventRepository codehale metrics

Today metrics relative to feature can go to JDBC or Inmemory databases with custom schema.
We wloud like to leverage on metrics and its numerous exporters. As a consequence will will publish the monitoring information into a metrics registry.

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/23492614-eventrepository-codehale-metrics?utm_campaign=plugin&utm_content=tracker%2F1038413&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F1038413&utm_medium=issues&utm_source=github).

not an issue


Dont snapshot files and filelike objects when saving a session

Some of the instability of copying session data into a new interview comes from file permissions. This removes file objects from saved sessions to avoid those complications.
A review of this PR can just be an "it still runs" review as this is still a developer-oriented feature. I'm going to continue to make additional small PRs to refine different aspects of the feature and eventually make it usable by members of the public.

When I ran manage_sessions.yml, the "To a new session of" field listed a few options that I don't recognize except one. I picked that one but got an error, which is true, that package is not in my playground. But why did it make to the list? Is it because it's in another playground on the same server?
DANotFoundError: Interview docassemble.LRFGuideMe:data/questions/HousingGuide.yml not found

It's using the list of "advertised" / dispatch / shortcut interviews on the server to build that list (see https://docassemble.org/docs/config.html#dispatch). It's in the config.yml file for the server. There's not any other good way to see what interviews are installed on the server.

Just a thought: would it be helpful if we allow the user to choose if they want to use this functionality for current interview only, or for all all interviews on the server? I suspect the first case may be no less popular than the second; in that case it would save the trouble of having to scan thru unrelated sessions by the user.

Just a thought: would it be helpful if we allow the user to choose if they want to use this functionality for current interview only, or for all all interviews on the server? I suspect the first case may be no less popular than the second; in that case it would save the trouble of having to scan thru unrelated sessions by the user.

That's a good idea. Also, see my notes here: https://github.com/SuffolkLITLab/docassemble-AssemblyLine/issues/421


Right Click Element alias for Open Context Menu

TODO: Add tests

Not ready yet, just trying to get the new pipeline to execute ..


[Locale] [Basque] Add new strings

Hi ikerib,
We are updating Stirling-PDF's license to MPL 2.0 (Mozilla Public License Version 2.0) to continue fostering our open-source commitment while introducing more flexible usage of the project.
What We Need From You:
In order to transition to the MPL 2.0 license, we need the explicit consent of all our contributors.

If you're comfortable relicensing your contributions for this specific PR under MPL 2.0, please respond with:
I, ikerib, grant permission to relicense my contributions from this PR to Stirling-PDF under the Mozilla Public License 2.0.
If you'd prefer to give blanket permission for all of your contributions to Stirling-PDF (be it through pull requests, commits, or any other form of contribution) rather than on a PR-by-PR basis, please respond with:
I, ikerib, grant permission to relicense all my contributions to Stirling-PDF under the Mozilla Public License 2.0.

What does this do and why?
As your old code was contributed under a different license it legally cannot move to a new license without your permission, and since we dont want to lose your code we wanted to reach out. Remember this project is remaining open-source!.
Deadline:
If we dont receive a response, or if you decline, unfortunately, we will have to remove your contributions from Stirling-PDF before the license change, which is scheduled  in coming months.
Thank You!
Your contributions are invaluable, and we hope to continue having them in Stirling-PDF under the new license. Feel free to ask if you have any questions or concerns about this change.
Best,
Frooodle
Stirling-PDF Maintainer


[Feature Request] Add a global worker registry

Is your feature request related to a problem? Please describe.
In order to enable Eager Workflow Start we need to find compatible workers that are associated with the current core instance. This feature will provide an internal interface that given a (namespace, task_queue_name) pair provides a list of suitable workers.
Describe the solution you'd like
WorkerRegistry is a singleton object that is lazily initialized and protected by a RwLock. Workers register at creation time (Worker.new_with_pollers()) and unregister during shutdown (Worker.initiate_shutdown()).
The main consumer of the query interface is the client, and since we don't want the client crate to depend on other internal crates,
we use dependency injection with a trait defined by the client.
Additional context
This is just a stepping stone towards enabling Eager Workflow Start for core-based languages

global [...] WorkerRegistry is a singleton object that is lazily initialized and protected by a RwLock.

Disagree. The client can just keep a collection of weak references to known workers that are associated with it. Eager workflow start is only a per client thing, not a process-wide one.

Disagree. The client can just keep a collection of weak references to known workers that are associated with it.

What this is trying to do is to help the client find workers that are "associated with it", in the sense that they target the same queue+namespace.
There is no reason why the local worker should use the same client connection, this is a needed restriction for Go and Java because otherwise there is no shared state between them. And this flexibility makes it easier to make it transparent for the Core SDKs, otherwise it is confusing that we have to configure them to use the same client to enable it...

Can be a client-crate-defined trait like EagerWorker that exposes calls the worker would implement. Eager workflow start is only a per client thing, not a process-wide one.

Internally this is how Eager Workflow Start will work, it just that this client trait is stateless, and just finds all the relevant workers using the registry injected by Core with the first "start workflow". All this happens within core abstractions, the SDKs clients are not modified...

this is a needed restriction for Go and Java because otherwise there is no shared state between them. And this flexibility makes it easier to make it transparent for the Core SDKs, otherwise it is confusing that we have to configure them to use the same client to enable it

Technically Go or Java could have had globals for this info too. They choose to make this client level state. It's not confusing that eager workflow start is per client, and it would make hardly any sense to create multiple clients for different workers of the same task queue and need to start workflows on each others' workers. Keeping the client-worker relationship works well in all SDKs, not just Go/Java.
My only objection is globals. I don't think we want a process-wide global in Go, Java, or core for a client knowing which workers it can use.

This is similar to metrics in Core, it also uses lazy static to maintain some state accessible.
I think keeping track of which workers are available, and what tasks they can process, should be a Core-level abstraction and not a client abstraction. Sooner or later we will need this in other contexts, for example, when flexing workers. And I agree that directly exposing a global in Java or Go SDKs is a hack, but here it is all internal to Core, not directly visible to application code...  @Sushisource comments?

And just to make clear what I mean by "global", it is not going to be a Rust "naked global" visible everywhere, I will wrap it in a struct, and then the new() will define a static map  within the local scope,  only indirectly visible to methods of this struct.
Similar to default_resource_kvs() in https://github.com/temporalio/sdk-core/blob/master/core/src/telemetry/mod.rs#L246-L249

This is similar to metrics in Core, it also uses lazy static to maintain some state accessible.

Not mutable state. Global mutable state is bad and unnecessary.

I think keeping track of which workers are available, and what tasks they can process, should be a Core-level abstraction and not a client abstraction.

It should be "core-client level"

directly exposing a global in Java or Go SDKs is a hack

Not about exposing, even having mutable global state is a hack/bad

Another reason is Typescript, where the workers never share a connection with the client because they use a different protobuf implementation

Then there needs to be an in-TS way to associate workers with clients.

Let me clarify what is "global mutable state" in this context.
I'm creating a singleton service that internally has mutable state protected by locks and exposes a narrow, and safe, interface to modify that state.
If we had a dependency injection mechanism in core we  wouldn't have this discussion, exactly the same singleton service would be created at start time, and "magically" appear where it is needed.
AFAIK, Rust is no friend of fancy pants dependency injection, so the alternative is to create a placeholder for the service at compile time, i.e., static, and initialize it with first use. And to make it available where it is needed we have a global factory method that returns that instance.
So this is not the wild west of global mutable state...

I think global is fine. As long as the map holds Weak refs the concerns about lifetimes aren't a problem. Rust makes concerns about improper access patterns a non-issue. Clients may not always be explicitly associated with the worker in situations where you might still like this to work. That's admittedly rare, but a small plus to using a static map.
I'd also be totally fine with storing it in the client, but I think either path is reasonable and whatever is simpler to implement makes sense to me.

We may have to explain that eager on some SDKs works when you reuse the client and on others is process wide. I'm not really concerned about access. It would have been plenty easy to have private static final ConcurrentMap<TargetNamespaceTaskQueueTuple, TemporalWorker> WORKERS = new MapMaker().weakValues().makeMap(); in Java too, but I think it's clearer to users to reuse at a client level instead of process level, but meh.

I realized another reason why we should discourage globals if we can. Core is currently multi-tenant and isolated today. There has been talk of a "core remote" where you can have a general core that supports multiple different remote workers. Similarly with WASM, there is an overhead of starting a WASM instance, so one may want to reuse the same WASM instance but with completely different runtime. It is very normal for core to be used with different runtimes and different thread pools (often to isolate telemetry).
In general, isolated core is a good thing. If you must share this map across clients, can we at least put it on the runtime?

That's reasonable - putting it on the runtime would be nice but clients do not know that the runtime exists right now or have access to it (since it's defined in core and the client crate doesn't depend on that, and won't).
I don't know that we'll ever actually do a remote core, but the WASM concern is real. Given that users are indeed kinda doing a dumb thing if they have multiple clients for the same endpoint, then I'm fine giving that up and saying the map exists on the client instance and is populated when the client is passed into a worker.

Yes, if we want to reuse WASM instances with different runtimes then we cannot have a worker registry per process, and we need to attach state to the client.
I saw that init_worker() takes a client of type AnyClient "to allow passing different kinds of clients into things that want to be flexible". Are there client types that I need to ignore? What is the cleanest way to unwrap the original client?

It was decided that we will not have a global registry, and instead associate state directly to clients, similar to the Go and Java SDKs


Provide chatbot instance to adaptors move up session creation.

Fixes the hipchat example by providing a usable backlink of the Chatbot instance to the Adapter instance.

@Achimh3011 Thanks for your interest.
The tests are failing due flake8 issue
2.94s$ flake8

./chatterbot/chatterbot.py:26:9: E303 too many blank lines (2)

The command "flake8" exited with 1.

Blinking test  tests/storage_adapter_tests/test_sqlalchemy_adapter.pyis also unstable on master.


ng build --prod removes 'var' variables from css file

Hello,
I have an Angular project.
My css looks like :
 :root{ 	--table-row:#f3f3f3; 	--table-shadow:#9EA1A2; 	--execution-log-background:#000; 	--tooltip-arrow:#24272A; 	--invalid-ele-border:#ff6170; } .invalidElement input.ng-invalid, .invalidElement textarea.ng-invalid{     border: 1px solid var(--invalid-ele-border) !important;     border-radius: 3px !important; }
When I run command for production version:ng build --prod=true --deleteOutputPath=true --optimization=true --progress=true --showCircularDependencies=true
it results as (from dist folder):
:root{--white:#fff;--theme-red:#bc0134;--bright-green:#73d94b;--bg-light-gray:#eeeeee;--bg-med-gray:#dddddd;--bg-dark-gray:#cecece;--bg-green:#d5f3c9;--bg-light-red:#ffe2ea;--bg-switch:#cccccc;--menu-divider:#a8a8a8;--text-dark-gray:#4c4c4c;--text-med-gray:#777777;--text-light-gray:#999999;--text-icon-green:#47ad33;--btn-icon-gray:#bbbbbb;--border-gray:#ddd;--border-dark-gray:#999;--border-light-red:#d5aab6;--shadow-light-red:#d5aab640;--disabled-bg:#eee;--disabled-text:#666666;--disabled-btn-text:#c3c3c3;--transparent:rgba(0,0,0,0);--trans-black:rgba(0,0,0,0.1);--focus-border-red:#d8bdc4;--focus-shadow:red;--focus-input-border:#999999;--trans-white:#ffffffad;--bg-tooltip:#444;--bg-used:#c0dedd;--table-row:#f3f3f3;--table-shadow:#9EA1A2;--execution-log-background:#000;--tooltip-arrow:#24272A;--invalid-ele-border:#ff6170}.invalidElement input.ng-invalid,.invalidElement textarea.ng-invalid{border:1px solid!important;border-radius:3px!important}
Here u can see in result var(--invalid-ele-border) is missing property form css.
So why this is happening, In a development environment, it works perfectly fine.
Note: extractCss parameter can solve my problem, but it increases files in the dist folder. So I don't want to go with it.
Please help..
Thank You
My Package.json looks like:
{
"name": "css-test",
"version": "0.0.0",
"scripts": {
"ng": "ng",
"start": "ng serve",
"build": "ng build",
"test": "ng test",
"lint": "ng lint",
"e2e": "ng e2e"
},
"private": true,
"dependencies": {
"@angular/animations": "~8.2.3",
"@angular/common": "~8.2.3",
"@angular/compiler": "~8.2.3",
"@angular/core": "~8.2.3",
"@angular/forms": "~8.2.3",
"@angular/platform-browser": "~8.2.3",
"@angular/platform-browser-dynamic": "~8.2.3",
"@angular/router": "~8.2.3",
"rxjs": "~6.4.0",
"tslib": "^1.10.0",
"zone.js": "~0.9.1"
},
"devDependencies": {
"@angular-devkit/build-angular": "~0.803.0",
"@angular/cli": "~8.3.0",
"@angular/compiler-cli": "~8.2.3",
"@angular/language-service": "~8.2.3",
"@types/node": "~8.9.4",
"@types/jasmine": "~3.3.8",
"@types/jasminewd2": "~2.0.3",
"codelyzer": "^5.0.0",
"jasmine-core": "~3.4.0",
"jasmine-spec-reporter": "~4.2.1",
"karma": "~4.1.0",
"karma-chrome-launcher": "~2.2.0",
"karma-coverage-istanbul-reporter": "~2.0.1",
"karma-jasmine": "~2.0.1",
"karma-jasmine-html-reporter": "^1.4.0",
"protractor": "~5.4.0",
"ts-node": "~7.0.0",
"tslint": "~5.15.0",
"typescript": "~3.5.3"
}
}

Hi @harshkoralwala, this should be fixed in version 9.
To update please following the instructions here: https://update.angular.io/#8.0:9.0
As a workaround until updating you can also disable optimization for the problematic code.
Example:
/* clean-css ignore:start */
.invalidElement input.ng-invalid, .invalidElement textarea.ng-invalid{ border: 1px solid var(--invalid-ele-border) !important; border-radius: 3px !important; }
/* clean-css ignore:end */

It is important to point out that "extractCss": true is not recommended to be used for non development builds.
If the problem persists after upgrading, please open a new issue, provide a simple repository reproducing the problem, and describe the difference between the expected and current
behavior.

Hello @alan-agius4 ,
Thank you for a quick reply.
As per your response, I can say its an issue for Angular 8.2.3.
As of now, I can not upgrade my project's angular version.
So temporary I m going with your solutions of add comments before and after problematic code.
Thanks.

It is important to point out that "extractCss": true is not recommended to be used for non development builds.

@alan-agius4 Newly created projects have this param in angular.json for production builds. Where can I read why this is not recommended way?

@destus90, apologies I updated the above post. I meant false and not true.
Ie: extractCss is recommended for non development built.

@destus90, apologies I updated the above post. I meant false and not true.
Ie: extractCss is recommended for non development built.

Got it. At first I thought I had missed something.

Hello @alan-agius4,
Is there any way to get a list of all problematic css code.
I have added a comment before and after problematic css which I have identified. But I want to take care of all the problematic css lines which are unknown to me.
As my css file is big, so it's not possible to identify problematic css lines manually.
I can identify it by comparing individual original css with css produced after the build, but its a very time-consuming task.
Kindly suggest a solution.
Thank You.

@harshkoralwala, problematic css code should result in errors or warnings.
That said, in this case the original code was not problematic but rather a bug in the optimizer which ending up dropping valid code.
Unfortunately, I am not aware of any tool to compare original vs output css. Apart from something like file diff utils.
My recommendation would be to update to version 9 as a lot of bugs are not longer reproducible because we replaced the css optimizer.


The latest web build won't run

Failed to load resource: the server responded with a status of 404 () /rip/master/app.4f0f24292f7c921909dd70d31d451d2c1a759632.wasm:1
TypeError: Failed to execute 'compile' on 'WebAssembly': HTTP status code is not ok

Good catch! Fixed. I've improved the CI script so this problem shouldn't happen again.


gif was in slow motion in TransitionToImage widget

I tried using gifs, but the gif was in slowmotion.

Try https://github.com/mchome/flutter_advanced_networkimage/commit/54c0c940b94af735cc865e6ad3f17d2e5ff015c5 with this setup:
https://www.dartlang.org/tools/pub/dependencies#git-packages


NameID namespace for LogoutRequest

When run against OneLogin::RubySaml::SloLogoutrequest is_valid? method, the existing XML LogoutRequest structure fails with:
Element '{urn:oasis:names:tc:SAML:2.0:protocol}NameID': This element is not expected. Expected is one of ( {http://www.w3.org/2000/09/xmldsig#}Signature, {urn:oasis:names:tc:SAML:2.0:protocol}Extensions, {urn:oasis:names:tc:SAML:2.0:assertion}BaseID, {urn:oasis:names:tc:SAML:2.0:assertion}NameID, {urn:oasis:names:tc:SAML:2.0:assertion}EncryptedID ). (OneLogin::RubySaml::ValidationError)

This is because the NameID element requires an explicit xmlns attribute to put it in the assertion space, per saml-schema-protocol-2.0.xsd.
I was not (easily) able to add a test for this against OneLogin::RubySaml::SloLogoutrequest because that class is not part of the omb-awong/ruby-saml fork.

Excellent. There were a few hacks that were made for our previous partner. Reviewing 👀

this is upstream now.

🎉 Bravo PK, bravo for wrangling the SAML beast 🐲


Unknown Licence

Hi, thank you for sharing this project. I would like to build upon it a little, but I can not find the licence it is under.
Is it MIT, Creative Commons or BSD?
Thanks
Ryan

updated, adding Unlicense license.

Wonderful, thankyou :)


renamed files, some files are left alone as they need refactoring and…

… cleaning later

Coverage decreased (-2.8%) to 20.354% when pulling db65cee8e27b6d52d017c29e00b64309bb3a514b on u9520107:refactor/file-naming into 0981f566f463b2d806c0ab8aab95b6e03ddb79b6 on ringcentral:master.

Coverage decreased (-2.7%) to 20.464% when pulling f2be2c47b2c6ced51e9d203ce8ce1a1e392f99b4 on u9520107:refactor/file-naming into 0981f566f463b2d806c0ab8aab95b6e03ddb79b6 on ringcentral:master.

Coverage increased (+1.0%) to 24.138% when pulling 5248e178393f10959c2a068437e3436c5d8614ce on u9520107:refactor/file-naming into 0981f566f463b2d806c0ab8aab95b6e03ddb79b6 on ringcentral:master.


Please invite me to the community

Name
Ishimoto Yamada
Discord Username (if applicable)
No response
Additional Context
Hello
This is Ishimoto Yamada.
I would love to join this community.
Please invite me.
Kind regards.

Name
Ishimoto Yamada
Discord Username (if applicable)
No response
Additional Context
Hello
This is Ishimoto Yamada.
I would love to join this community.
Please invite me.
Kind regards.


Add fuzzing

Simulating a user load and verifying that the behavior of Weaviate is correct.

Closing because it is solved by @etiennedi's script.


Upvoting or Downvoting a comment crashes Carpeddit

I'm using the latest commit

Including deleting a comment crashes


Questions: rustfmt provides incorrect format

Hi all!
I'm trying to work on #7833 . But when I try to use rustfmt, the resulted format is not as expected.
The input format and other functions' format is:
pub fn rpad_utf8(arg: &Option<Bytes>, len: &Option<Int>, pad: &Option<Bytes>) -> Result<Option<Bytes>> {

But the formatted version would become multiline indentation:
pub fn rpad_utf8(
    arg: &Option<Bytes>,
    len: &Option<Int>,
    pad: &Option<Bytes>,
) -> Result<Option<Bytes>> {

Since I'm not sure which config file rustfmt.toml I should look for, I uninstalled and re-installed rustfmt to make sure it stays the default settings, but the ouput is the same. Is there some point I'm missing?

This is because the line is too long (105 chars), so the rustfmt will format it to the vertical mode, I think this is ok. See here for more details.

Thank you @glorv for your kind reply! I should've read configurations of rustfmt more carefully.