Linuxbrew team

My new son, Jake, is two months old, and I'm on parental leave from work. I will be traveling with my family until Christmas. I'd love for some help with maintaining Linuxbrew in my absence.
Dana @danabrand, a system administrator at UBC, has kindly completed the last two merges of Homebrew into Linuxbrew. She's newish to Linuxbrew and doesn't have commit rights  to Homebrew/linuxbrew. When she's prepared a merge commit, I'd appreciate someone's help to take a look at it and push danabrand/master to linuxbrew/master. Less pressing, but it would be great to have a couple volunteers to review pull requests and comment on issues.
Xu @xu-cheng, Dominyk @DomT4, Tim @tdsmith, Homebrew-science friends @Homebrew/science, would you be interested in joining the Linuxbrew team? Please cc anyone that I've missed.

I'll gladly help as time allows.

Happy to help where I can.  We look forward to your return in 21 years time ;-)

Homebrew already chews up a more than reasonable amount of time that I have left after various chunks of life heh, but I might be able to stick my head in Linuxbrew and help with various simple issues. Dana is also welcome to ping my @ if there's something that needs merging quickly. Enjoy the time away Shaun :)

I will help now and then whenever I'm available. But I cannot keep too many promise as Homebrew has already occupied most of my spare time, similar as DomT4.
BTW, congratulations.

@DomT4 @xu-cheng Any effort that you can pitch in is appreciated.
Would you please add @dpo and @tseemann to the Homebrew/linux team?

Would you please add @dpo and @tseemann to the Homebrew/linux team?

Done. Also I think you have the access to change Linux team whatever you want.

I tried, but I don't believe I do have the rights to change the linux team.

@danabrand Would you be able to merge Homebrew into Linuxbrew again this week?
@dpo @tseemann @DomT4 @xu-cheng Could one of you take a look at Dana's merge and push the branch danabrand/master to linuxbrew/master and danabrand/homebrew to linuxbrew/homebrew?
After a few weeks of that arrangement, could you please add @danabrand to the Homebrew/linux team so that she can push herself?

@xu-cheng Can you make the Homebrew/linux team public, and if you think it's a good idea, make all the Homebrew/* teams public?

@sjackman You now have rights to add/remove people from the @Homebrew/linux team.

I should have some more free time starting next month, would be glad to help out then.

Thanks, Mike.

Great. I've added you to the team, Ian.

Hi all, great to e-meet. And Shaun, yes, I can merge again into Linuxbrew this week.

Dana has merged homebrew/master into linuxbrew/master. That commit is at danabrand/master. Could someone in the @homebrew/linux team please take a look at that commit and push it to linuxbrew/master? I've tested it, and it looks good, but I won't be doing any testing while I'm in Thailand. =)

Nobody else has responded here so I took the nervous plunge and merged Dana's master branch into Linuxbrew/master, which brings Linuxbrew up to ~8 days ago in up-to-date-ness.

Please also push the branch danabrand/homebrew to linuxbrew/homebrew. It's a branch at the point that homebrew/master was merged.

This does not want to work for me locally, but at least the master is done.

Thank you! I was just wondering what's happening with that. Ok, I will get onto merging the latest and let you know when it's ready to be pushed again.

Thanks Dana. Feel free to ping if you want to. I was trying to give it a bit to see if anyone who uses Linux on a more frequent basis wanted to handle this, but ¯\_(ツ)_/¯.
I don't really have time to keep on top of Linuxbrew and I don't want to routinely become the person who gets pinged when something breaks, because most of the time my reaction is going to be a slightly confused look when it comes to Linux specific changes :smile:.
I can handle occasional merges of branches here though, so if things are ready and tested and such let me know and I'll bite the bullet. Thank you again for the willingness to help out with this sort of thing.

How are things going @danabrand? No pressure intended - Know life gets busy and it's apparently just me and you doing Linuxbrew for a while.

Shaun here checking in from Thailand. Thanks so much for feeding and watering Linuxbrew in my absence, Dana and Dominyk. I appreciate your help!

Hi Dom, this has been on my mind for the last weeks, but the UBC job was indeed very busy.
From: Dominyk Tiller<EMAIL_ADDRESS>Sent: Thursday, November 05, 2015 9:19 PM
To: Homebrew/linuxbrew
Cc: Brand, Dana
Subject: Re: [linuxbrew] Linuxbrew team (#600)
How are things going @danabrandhttps://github.com/danabrand? No pressure intended - Know life gets busy and it's apparently just me and you doing Linuxbrew for a while.
—
Reply to this email directly or view it on GitHubhttps://github.com/Homebrew/linuxbrew/issues/600#issuecomment-154288322.

@sjackman Hope the time away so far has been pleasant! :smiley:. Know getting away from FOSS for a bit can be a weight off one's shoulders.
@danabrand No worries Dana. Whenever you have time. Things that pay the bills trump things which very much don't.

@DomT4 Finally got around pushing the latest to my branch (danabrand) - would you care to push it to live linuxbrew? I know it's extra work for you, but I don't have permissions to do that. Hopefully the fixes were all correct or at least not breaking current ones:-)
@sjackman Shaun, I am trying to carry your brew while you away, just a bit swamped here, but hopefully you'll find it up to date enough. when are you back?:-)

@danabrand Thanks Dana, you're a star. Merged into Linuxbrew and pushed. If anything broke we'll probably see issues and someone can look at it. Thank you again :)

Looks like I broke it – see issue 648. I am in transit right now, will get to this only in a few hrs, if anyone can fix it, please go ahead. Thank you!
From: Dominyk Tiller<EMAIL_ADDRESS>Sent: Monday, November 16, 2015 12:55 PM
To: Homebrew/linuxbrew
Cc: Brand, Dana
Subject: Re: [linuxbrew] Linuxbrew team (#600)
@danabrandhttps://github.com/danabrand Thanks Dana, you're a star. Merged into Linuxbrew and pushed. If anything broke we'll probably see issues and someone can look at it. Thank you again :)
—
Reply to this email directly or view it on GitHubhttps://github.com/Homebrew/linuxbrew/issues/600#issuecomment-157167074.

Hello, can someone either apply fix suggested in issue 648 or possibly repush my branch, whatever you think it's better. i don't have access to the main repo. thanks!

Will merge your fork in again now. Apologies for the few hours delay.

Hello all, worked on a massive merge today. I did what I could, but I need your help.
I went through all conflicts (100+) reported by the merge and for better or worse, resolved them. I pushed all these in my branch.
However, when testing, I noticed that all formulas using "unneeded" for a bottle, failed. At a closer look, the auto-merged formula.rb doesn't contain a   "def bottle_unneeded?" line.
When merging, these were reported auto-merged by git:
Auto-merging Library/Homebrew/keg.rb
Auto-merging Library/Homebrew/formula_installer.rb
Auto-merging Library/Homebrew/formula.rb
Auto-merging Library/Homebrew/extend/pathname.rb
Auto-merging Library/Homebrew/extend/ENV/std.rb
Auto-merging Library/Homebrew/exceptions.rb
Auto-merging Library/Homebrew/dependency_collector.rb
Auto-merging Library/Homebrew/cmd/test-bot.rb
Auto-merging Library/Homebrew/cmd/pull.rb
Auto-merging Library/Homebrew/cmd/bottle.rb
Auto-merging Library/Homebrew/cmd/audit.rb
But whatever the auto-merging did, brew fails when trying to make sense of an unneeded bottle. This is the error reported when testing on zzz, tomcat, jenkins (all in that boat):
Error: block not supplied
Testing at random on other modified formulas worked fine.
Any ideas appreciated. thank you.

Hmm, just noticed issue 660 reports this as well, so the error is not necessarily related to this merge.

However, when testing, I noticed that all formulas using "unneeded" for a bottle, failed. At a closer look, the auto-merged formula.rb doesn't contain a "def bottle_unneeded?" line.

It sounds like the auto-merge dropped that line for some reason, maybe there's a conflict there or something. Can you diff between the merged formula.rb and the homebrew/homebrew formula.rb from the merge point?

@DomT4 It's a bit more complex than that, the unneeded reference appears in a few more files in Homebrew, while not present in Linuxbrew.
Alternatively, we could push all new formulas other than the few involving unneeded. It at least brings a bunch of them up to date, while we figure out (or Shaun does when he's back) the underlining mechanism that breaks. What do you guys think?

@danabrand Depending how comfortable you are with it another potential rough hack is to use something like awk or sed to run through the formulae directory and remove any bottle :unneeded or bottle :disable lines temporarily.

@DomT4 hmm, that's not a bad idea. I see 21 formulas using that, not that many. let me try a few. of course it's a hack and not pretty, but it does the job done for the moment.

@danabrand What's the breakage process out of curiosity? Does the entirety of Linuxbrew break if there are bottle :xyz lines detected or only when those specific formulae are invoked in some manner?

@DomT4 It seems that only brew installing/updating or otherwise parsing those formulas is affected.  I discovered the behavior in my testing when testing at random brew install on some of the modified formulas in this latest merge. But then I noticed report from issue 660 as well, so I believe this happened a few merges ago. When did you guys introduced bottle unneeded/disable?

@danabrand Alright, that's manageable at least - I had in mind that homebrew/versions can be tapped by Linuxbrew users and there's a bunch of bottle :xyz in there these days, but as long as it's only going to die on certain formulae it's not too horrible a UX.
The new bottle :xyz DSL was added on October 19th in https://github.com/Homebrew/homebrew/pull/43935.

CC @xu-cheng in case he has ideas on why it wouldn't function for Linuxbrew specifically.

Thank you very much for tackling this merge even though you're busy, Dana, and thanks Dominyk for supporting her. If bottle :unneeded was broken before this merge, I suggest leaving it broken in this merge and fixing it with a separate commit after the merge. If this merge otherwise works, push! Thank you everyone.

@DomT4 Ok. good news. Worked on an older branch of Linuxbrew pre Oct 19 and merged with Homebrew just to test the formula.rb and related formulae merging; this time the automerging worked just fine and voila, we have good Homebrew formulae. Tested installs for reported formula in issue 660, as well as a few other that have the bottle:unneeded line, they worked fine. Checked in these formulae into my latest danabrand repo.
So theoretically this last commit in danabrand solves the bottle:unneeded line issue, as well as it has the 100+ updated formulae in Library/Formula. I did not test them all, I tested some at random and they worked for me. I really can't test that many, you guys did a lot of upgrades and I haven't kept up, so this is some all-in-one commit and I am sure I slipped somewhere.
Do you think you have time to test a bit this last commit? Last experience traumatized me:-) and I would want to be more cautious this time, especially with such a large merge.

Yeah, I'll merge you into my VM later today and play with it a bit. May not be till mid-afternoon as have quite a busy day but will get around to it by the evening. Thank you Dana.
Don't worry about not testing everything; It isn't humanly possible to test everything Homebrew tweaks on a weekly basis, let alone longer, without mainlining caffeine and not moving from your computer for a few days :smile:.

Seems to be okay. Merges cleanly, can run a brew update and brew upgrade on an outdated formula. Can run brew cleanup and brew prune to remove old versions without issue. Can execute commands on bottle :unneeded formulae. Can run brew audit and brew audit --strict --online fine. Can execute the brew command itself with no arguments with no errors. brew doctor runs without error. Same with brew config. brew tap works fine.
There's a slight merge conflict in test-bot on line 628 of your most recent commit, but I could fix that on pull.
Looks pretty good to me Dana :).

@DomT4 Thank you Dominic. Please go ahead with the conflict resolve and if you feel we can bite the bullet, push the new commit. Fingers crossed.

Merged. Did a bit more testing locally, everything seems to work as expected, which probably means there's some edge case breakage we'll suddenly get flooded with :smile_cat:, but fingers crossed all is alright. Thanks again Dana. We owe you a :beer: some day for all the help here.

@DomT4  BTW, managed to injure myself this weekend on a first ski trip this winter, so one hand typing now and less computer glued, but will try to watch the issues.  @sjackman  Shaun, come back!!!!!!! :)

@danabrand We just get pouring rain in the UK :wink:. Hope the injury isn't too painful and you recover soon!

I hope that your ski injury has recovered, Dana. Thanks again both of you for your efforts! Liz and I have left Thailand and are now in Bali, Indonesia.

Hi guys, apologies for dropping off the grid. Still recovering, boy I forget my age... I will try to poke at an up-to-date all-in-one merge this coming week, but most likely I will truly get onto it only on the 22nd. Hope to have it ready for all as Xmas present;)

Hi, friends. I'm back from vacation and sleeping in my own bed again after three months away. Thanks for all your help in my absence. As much as you have time available, I encourage you all to continue contributing to Linuxbrew! I really need the help.

Now on to some trickier technical stuff. I started a git merge homebrew/master today and got 200 merge conflicts! Whew! I believe that these merge conflicts are a results of the merges on 2015-11-10 and 2015-11-16 and the subsequent reverts on 2015-11-17. git now thinks that a bunch of files are modified on the linuxbrew branch, and these files create merge conflicts. Here's a good blog post on the issue: Git: merging after a revert. His suggestion is to revert the revert commits. In our case however there are three revert commits and there have been 8 commits since the revert commits, so I don't think that strategy is straight forward.
My plan is to checkout the merge of 2015-10-15 602d9b461deec71f1886ddfc2d8df932e702fc37 and merge homebrew/master into that commit. With that merge done, I'll then merge that commit into the current linuxbrew/master bb453ad33892df0d5560a177890a94810ecc71af, taking the tree exactly as it is from the previous merge commit. i.e. git merge --strategy=ours. This should, if it works as intended, produce a nice clean tree that reverts the most recent 14 commits:
bb453ad  2015-12-10<EMAIL_ADDRESS>travis: remove conflict marker
b82c4cf  2015-11-30<EMAIL_ADDRESS>mysql: de-dupe dependency block
0d0823e  2015-11-30<EMAIL_ADDRESS>profanity: needs libcurl on linux
bdcfe9b  2015-11-30<EMAIL_ADDRESS>test-bot: resolve merge conflict
c5ead6b  2015-11-26<EMAIL_ADDRESS>Update formula to use unneeded
d88a3b3  2015-11-25<EMAIL_ADDRESS>Fixed merge error
e2b369e  2015-11-25<EMAIL_ADDRESS>Merge branch homebrew/master into linuxbrew/master
da44282  2015-11-24<EMAIL_ADDRESS>Update libpng.rb
092bbb6  2015-11-17<EMAIL_ADDRESS>Revert "Merge branch homebrew/master into linuxbrew/master"
787d7f9  2015-11-17<EMAIL_ADDRESS>Revert "Merge branch homebrew/master into linuxbrew/master"
c257139  2015-11-17<EMAIL_ADDRESS>Revert "OS_VERSION fixed"
3bd7bda  2015-11-16<EMAIL_ADDRESS>OS_VERSION fixed
85d75ac  2015-11-16<EMAIL_ADDRESS>Merge branch homebrew/master into linuxbrew/master
e9b2f2b  2015-11-10<EMAIL_ADDRESS>Merge branch homebrew/master into linuxbrew/master

I'll cherry-pick the following commit
0d0823e  2015-11-30<EMAIL_ADDRESS>profanity: needs libcurl on linux

I want to emphasize that I do really appreciate all the work that was done by everyone between 2015-11-10 and 2015-12-10, and I apologize for my plan to wipe these commits away. My git-fu is just not strong enough to continue with the repo in its current state, and the most efficient and likely-to-succeed strategy for me is to backtrack to the last known-good state. Thanks again for all your help!

Here's what I plan to do:
git checkout master
git branch linuxbrew-old
git checkout 602d9b4
git branch --force master
git merge homebrew/master
git merge --strategy=mine linuxbrew-old
git push

This strategy reduces the number of merge conflicts from 200 to 42, so I think that I'm on the right track. The real test will be in future merges.

The merge is going well. I'm testing it now at https://github.com/sjackman/linuxbrew/
If all goes well, I'll push it to linuxbrew/master

The merge went well and is now pushed to linuxbrew/master!

I'm going to close this issue now. If anyone would like to contribute more work to Linuxbrew, I would very much appreciate your help.

Welcome back @sjackman. No offence taken to doing it your way. Things got kind of complicated with merging and then having to revert the merge commits to avoid manually reverting 200+ commits. Linuxbrew has been a bit of a handful. I'm not really sure how you manage to keep on top of it to be honest, given you also contribute to at least the Science tap.
Dana was a complete godsend. Very little would have happened here without her doing the vast majority of the heavy lifting and I'm extraordinarily grateful to her for all the help. Dana, if you ever run into me somewhere, feel free to request a large drink of some description, I more than owe you one.


Where is ares-novacom

You can't connect to a TV without ares-novacom...
cf. https://webostv.developer.lge.com/develop/app-test/using-devmode-app/

https://www.webosbrew.org/pages/configuring-sdk.html

https://webostv.developer.lge.com/sdk/installation/download-installer/

You can't connect to a TV without ares-novacom...
cf. https://webostv.developer.lge.com/develop/app-test/using-devmode-app/

webosose/ares-cli supports webOS OSE. If you want to download CLI that supports webOS TV, download webOS TV SDK from the link below.
https://webostv.developer.lge.com/sdk/installation/download-installer/


[Update] [SkyfallRPG] Update Sheet json

[x] The pull request title clearly contains the name of the sheet I am editing.
[x] The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[x] The pull request makes changes to files in only one sub-folder.
[x] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

Changes / Description
Updated Description

Character Sheet Info Roll20 Internal Use only.


Hi,should I create a directory cam-baseline-voc12?

Should I create a directory cam-baseline-voc12 and put res50_cam.pth in it?
Is the res50_cam.pth pre-trained on ImageNet by your team?
What is the difference between this res50_cam.pth and net/resnet50_cam.py trained by myself?
If I run the run_sample.py command line, am I just train your .pth more time(epoch) or I train the model from scratch?
Sorry to ask such many easy questions, I am the beginner of WSSS. Thank you so much!

Hi,

I provided res50_cam.pth which was the pre-trained model on VOC2012. Actually, you can try to train such a model or directly use ours.
Then, run_sample.py will load the pretrained model to train clims.

Hi,

I provided res50_cam.pth which was the pre-trained model on VOC2012. Actually, you can try to train such a model or directly use ours (create a directory called cam-baseline-voc12 then put the pretrained res50_cam.pth in it).
resnet50_cam.py is the basic network (baseline cam). res50_cam.pth is the saved model of resnet50_cam.py
run_sample.py will load the pretrained model to train clims.
btw, welcome to star our repository

Hi,

I provided res50_cam.pth which was the pre-trained model on VOC2012. Actually, you can try to train such a model or directly use ours (create a directory called cam-baseline-voc12 then put the pretrained res50_cam.pth in it).
resnet50_cam.py is the basic network (baseline cam). res50_cam.pth is the saved model of resnet50_cam.py
run_sample.py will load the pretrained model to train clims.
btw, welcome to star our repository

1.感谢，我目前的理解是先用res50_cam.pth的模型去初始化resnet50_clims里的CLIMS对吗？
2.还有一个问题就是如果我想CLIP的image encoder之前加一些模块（比如对特征表示做增强这种的）的话，可以在resnet50_clims里的CLIMS的stage后面直接加吗？这样的话还能用res50_cam.pth去初始化吗？

可以的，如果你添加了一些模块的话，也可以导入预训练的模型，只要把这里的strict=True换成strict=False https://github.com/CVI-SZU/CLIMS/blob/f37162c1fba3a0011600a6eadfe97fc16edd7edd/step/train_clims.py#L65

可以的，如果你添加了一些模块的话，也可以导入预训练的模型，只要把这里的strict=True换成strict=False
https://github.com/CVI-SZU/CLIMS/blob/f37162c1fba3a0011600a6eadfe97fc16edd7edd/step/train_clims.py#L65

我大概了解了，还有一个小小的问题，就是如果不导入res50_cam.pth的话，训练resnet50_clims大概需要多花多少时间呢？您有记录过吗？

最好导入一下预训练模型，效果会好些，总体训练时间是一样的应该是10epoch

最好导入一下预训练模型，效果会好些，总体训练时间是一样的应该是10epoch

OK! 感谢同学耐心的指导！！！祝申到理想的PHD!

感谢！


Adding consumergroup and consumer to defaulting webhook plus validation logic updates

Signed-off-by: aavarghese<EMAIL_ADDRESS>As per title
/cc @pierDipi

/test unit-tests

/test unit-tests

/test unit-tests


Associating WI from Kebab , distorted dropdown for iteration list with long strings

Associating WI from Kebab , distorted dropdown for iteration list with long strings

@mindreeper2420 Do we need type ahead drop in the associate iteration modal?

I think here we need to set a max width and overflow hidden, at least for the time being.
There's no real solution to this problem.

fixed in https://github.com/fabric8io/fabric8-planner/pull/1744

Verified !


Typescript is useless when using Proxy

Suggestion
Es6 Proxy is one of the most powerful and sweet features of JavaScript, it can particularly give JavaScript programmers ability to have different setter/getter behaviors on index properties. But TypeScript just ruins the experience, despite similar practices found largely in DOM Api's.
If Typescript can't infer Proxy types, at least provide more powerful index signatures typing, allowing getter setter to have different types.
📃 Motivating Example
/*
interface TargetObj<S extends object> {
  [K in keyof S]: Wrapper<S[K]> // wrapper object just stores that value, with type from generic
}
targetObj: TargetObj<skipping S signature> = {
  key1:  new Wrapper(1)
  key2: new Wrapper('string')
}
obj = new Proxy(targetObj , {
  get(target, prop) return  target[prop].value
  set(target, prop, value) {
    obj[prop] = new Wrapper(value)
  }
})
obj.key1 // -> Wrapper { value: 1 }
obj.key1 = 5
obj.key2 = 'another string'
obj.key1 // -> Wrapper { value: 5 }
obj.key2 // -> Wrapper { value: 'another string' }

Now TypeScript wont allow this and I can't even set index signatures with varying signatures of getter and setter.
⭐ Suggestion
Infer Proxy types or at least support different getter/setter for index signatures like this.
interface TargetObj<S extends object> {
  get [K in keyof S]: Wrapper<S[K]> // wrapper object just stores that value
  set [K in keyof S]: S[K]
}

🔍 Search Terms
List of keywords you searched for before creating this issue. Write them down here so that others can find this suggestion more easily and help provide feedback.
Proxy type inferring, different getter/setter signatures on index properties, etc
✅ Viability Checklist
My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
[x] This feature would agree with the rest of TypeScript's Design Goals.

💻 Use Cases
Proxy support.

Either this is a duplicate of #43662, or #43662 is a prerequisite for this.

Duplicate of #43826

Thank you @KaelWD 👍


Error when using post- hooks

After a successful test of this script in our testing environment, deploying our production workload caused the deployment job to exit with a failure. While the deployment still ran successfully, the script exited with a non-zero exit code, thus causing the job to be marked as failed.
Release "limes-api-production" has been upgraded. Happy Helming!
NAME: limes-api-production
LAST DEPLOYED: Tue Nov 15 14:44:04 2022
NAMESPACE: default
STATUS: deployed
REVISION: 474
TEST SUITE: None
NOTES:
Successfully deployed the application at https://limes-api.mll/
0s          Normal    Completed                job/limes-api-production-send-additional-files-to-order-27808725      Job completed
0s          Normal    SawCompletedJob          cronjob/limes-api-production-send-additional-files-to-order           (combined from similar events): Saw completed job: limes-api-production-send-additional-files-to-order-27808725, status: Complete
/usr/lib/ruby/gems/3.1.0/gems/helm_upgrade_logs-0.4.0/exe/helm_upgrade_logs:31:in `rescue in <top (required)>': Failed to find logs before helm finished (HelmUpgradeLogs::Error)
	from /usr/lib/ruby/gems/3.1.0/gems/helm_upgrade_logs-0.4.0/exe/helm_upgrade_logs:26:in `<top (required)>'
	from /usr/bin/helm_upgrade_logs:25:in `load'
	from /usr/bin/helm_upgrade_logs:25:in `<main>'
/usr/lib/ruby/gems/3.1.0/gems/helm_upgrade_logs-0.4.0/exe/helm_upgrade_logs:27:in `waitpid': No child process (Errno::ECHILD)
	from /usr/lib/ruby/gems/3.1.0/gems/helm_upgrade_logs-0.4.0/exe/helm_upgrade_logs:27:in `<top (required)>'
	from /usr/bin/helm_upgrade_logs:25:in `load'
	from /usr/bin/helm_upgrade_logs:25:in `<main>'
[INFO] Waiting for pod logs: Defaulted container "migrate-limes-api-dwh" out of: migrate-limes-api-dwh, migrate-limes-api-archive
Error from server (NotFound): pods "limes-api-production-hook-migrate-secondary-rp6lj" not found

I think the issue is that limes-api-production-hook-migrate-secondary is defined with the annotation "helm.sh/hook": post-install,post-upgrade. This causes it to be executed only after helm finishes running, thus causing this script to fail to find the logs. How can this be solved?

Hmm sorry didn't see this much earlier.
What parameters do you use when you run this executable?
Did you use something like
helm_upgrade_logs --install RELEASE_NAME CHART_NAME --wait
The --wait is required for the sub process to wait for things otherwise the release may finish without waiting for pods to start and I think this script will break
I have used this in many places that use the post install, post-upgrade hooks. Do you see logs before the release finishes?
I have just installed a new version 0.4.1 so if you add the environment variable helm_upgrade_logs_ignore_child_error it will not exit with the same error though not sure exactly if it will properly clean things up though not sure if it will really make it work

@spawnia  have just released version 0.5.0. I found there was an issue with getting of logs and specific pods in a certain state causing issues which I have just fixed. Keen to know if this fixes your issue

Thank you for your answer and sorry for the lack of feedback on your question. I will try v0.5.0 and see if that fixes the issue I was seeing.

I know this is probably unrelated to your package, but I am having trouble installing the latest version in a Docker image I use to run it. https://rubygems.org/gems/helm_upgrade_logs/versions seems to have 0.5.0, and I can install it in Ubuntu just fine - but somehow the following Dockerfile fails for me with Could not find a valid gem 'helm_upgrade_logs' (= 0.5.0) (required by 'helm_upgrade_logs' (0.5.0)) in any repository.
FROM alpine
RUN apk add --no-cache ruby \
 && gem install helm_upgrade_logs:0.5.0

Do you have an idea what I could do to fix this? Feel free to ignore this, as I said it seems unrelated to your package.

Hmm perhaps the alpine ruby package does not have ruby gems as a source? Maybe adding
gem sources --add https://rubygems.org/  might help.

To have ruby inside an alpine image, what I've done in the past is simply the following using ruby's own docker alpine image
FROM ruby:3-alpine

RUN mkdir /app
WORKDIR /app
COPY . /app
RUN gem install bundler

Maybe one of the images on https://hub.docker.com/_/ruby/tags like https://hub.docker.com/layers/library/ruby/3.3-rc-alpine/images/sha256-f8d2fb7879f2999c045cd3d75c29ff030ebac43443fece49823d433153b1c181?context=explore

Ok I found the version of alpine seems to make a difference. I don't why.
This docker file did work
https://gitlab.com/samuel-garratt/helm_upgrade_logs/-/blob/master/Dockerfile
See the pipeline run that builds it at https://gitlab.com/samuel-garratt/helm_upgrade_logs/-/jobs/5548800145

The latest version of the gem seems to be working for me @spawnia . I think it just took a while to be fully indexed for some images


No Dockerfile was found in the project directory: .

Following the instruction from here:
No Dockerfile was found in the project directory: .
but around local push the following error pop up on the command line:
`No Dockerfile was found in the project directory: .
If you need help, don't hesitate in contacting us at:
GitHub: https://github.com/resin-io/resin-cli/issues/new
Forums: https://forums.resin.io`
My tree files are attached.
resin-base-python.zip
My goal is to use the local environment for testing a prototype. This project (a bit modified) was successfully running already in production. The Dockerfile in the same directory as the other files.

After removing .template extension from Dockerfile the building is started. However the following error msg came up:
`* Building..

Stopping and Removing any previous 'base-resin-python-local' container
Building new 'base-resin-python-local' image
Step 1 : FROM resin/%%RESIN_MACHINE_NAME%%-python
rdt push failed. Error: Error parsing reference: "resin/%%RESIN_MACHINE_NAME%%-python" is not a valid repository/tag Error: Error parsing reference: "resin/%%RESIN_MACHINE_NAME%%-python" is not a valid repository/tag
`

Since my project was in production previously the %%RESIN_MACHINE_NAME%% was used in the Dockerfile, so the base Docker image was not loaded properly I guess.
Please correct me if I am using improper terms, but it solved my problem when I replaced the %%RESIN_MACHINE_NAME%% with raspberrypi.

Hey @Biaggio74 ,
It looks like you figured it out :)
Currently resin local push does not support Dockerfile.template resolving (i.e. generating proper Dockerfile from a Dockerfile.template).
However, we are already working on this and we will soon integrate our OSS module that performs template resolving into the cli. When this is done, Dockerfile.template will be supported without requiring the manual steps you had to go through.

Hi @lekkas,
One more thing which is still related for deploying resin local.
In some post I've seen the following how to read env variables from the .yml file which is created in the project folder.
I tried this:
In .yml file
env-vars:
-varUUID: RESIN_DEVICE_UUID
However, in this way it was not working when I called the varUUID var from somewhere in my code, in the python file. How should I reference it  properly?

Hey @Biaggio74 ,
Sorry for the late response. There are two ways you can specify environment variables:

The first one is to pass --env options in resin local push, e.g.

resin local push --env 'varUUID=RESIN_DEVICE_UUID'. resin-cli will automatically save your environment variables in .resin-sync.yml

The second one is to edit .resin-sync.yml directly and in that case the proper format is:

.resin-sync.yml
---------------
[..]
local_resinos:
  environment:
    - varUUID=RESIN_DEVICE_UUID
[..]

Hi @lekkas,
Thank you

Hi @lekkas
My .resin-sync.yml is looks like this:
local_resinos:
  app-name: myawsthing
  build-triggers:
    - Dockerfile: ba1f8906445414018ca1af696ad0f34b178adb839a28e28e36c7bfbbf5204c7e
  environment:
    - varUUID=RESIN_DEVICE_UUID
destination: /usr/src/app
ignore:
  - .git
  - node_modules/

But I am still getting the following error:
...
rdt push completed successfully!
* Streaming application logs..
Traceback (most recent call last):
  File "awsiotpub.py", line 23, in <module>
    clientId = varUUID
NameError: name 'varUUID' is not defined

Closing due to inactivity, but please ping if you're still facing this.

Sorry for the late reply, I just saw the mail now.
Its solved my issue ;)
Br,
B.
On Tue, Nov 14, 2017 at 6:54 PM, Tim Perry<EMAIL_ADDRESS>wrote:

Closing due to inactivity, but please ping if you're still facing this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/resin-cli/issues/504#issuecomment-344342412,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK3pY_TnKCXXa_gQ1gWe7MZ9BZXj1s9wks5s2dPugaJpZM4NMRJT
.


Fix NOT IN -> NIN

The Azure Digital Twins Query Language uses NIN rather than NOT IN.
Also removed reference STARTSWITH and ENDSWITH in query comparison operators, which were never a thing.
Removed confusing reference to nextAsTwin which is not a thing.

@johngallardo : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@johngallardo  Thank you for these changes! Approved from the docs side.  #sign-off


Yaw,Pitch,Roll

Hello, I noticed that in line 160 of main.py, the values of the yaw,pitch roll can be obtained
P, pose = parse_pose(param)  # Camera matrix (without scale), and pose (yaw, pitch, roll, to verify)
However, when I attempted to print the results of pose using the sample image of test1 ( as in the readme), the values obtained were (-0.2982479758550047, 0.16620569715835162, 0.28968614317706604), which looked too small to be correct seeing that the face is tilted quite a bit. (From the resulting pose image as well) Can I check if this is correct?

Radian to degree converting? For example: pitch * 180 / pi

i tried this but this takes more time. i want real time pose estimation without gpu.Does anybody know better method?


Languagetool: API doesn't accept query.

I just installed your Package.
This Error showed up.
Error: langCode cannot be empty or whitespace only.
Can't find an logfile.
I tryed "atom.workspace.getActiveTextEditor().getText()" in the Console.
This works fine.
Update:
I found the Problem. If you leave Mother Tongue empty. The Error occurs.

@fhchl Any intention in accepting the PR?

I have the same problem

I haven't used Atom in ages and will not be able to check any PRs. I am archiving this repo.


Sidebar Redux

Set Sidebar Redux for async loading and Actions dispatch on select/deselect
Signed-off-by: Peter Fitzgibbons<EMAIL_ADDRESS>

Codecov Report

Merging #1200 (6ed358e) into feature/metrics (0e6fef2) will not change coverage.
The diff coverage is n/a.

@@                Coverage Diff                 @@
##             feature/metrics    #1200   +/-   ##
==================================================
  Coverage              71.87%   71.87%
  Complexity               291      291
==================================================
  Files                     42       42
  Lines                   2311     2311
  Branches                 240      240
==================================================
  Hits                    1661     1661
  Misses                   509      509
  Partials                 141      141

Flag
Coverage Δ

opensearch-observability
71.87% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Manually merged by @sejli into #1211


Added guidance for scaling applications to multiple regions

Fixes https://github.com/cockroachdb/docs/issues/10455.
This PR introduces a new page with guidance on scaling from single-region to multi-region.
The page includes some general guidance on representing crdb_region columns in an ORM, for DML statements, with an example for SQLAlchemy. We can add other examples (notably, in Hibernate) after publishing this page.

This change is

@awoods187 @rmloveland
Thank you both for the detailed responses! I'll work in your feedback as soon as I can.

@awoods187 @rmloveland
Everything looking good from your points of view?
@ajstorm
When you get a chance, it would be nice to have an engineering review on this PR as well. :)

@ajstorm Friendly ping on review :)


Prefer [] over [""] for blank values that are split

irb(main):002:0> [].blank?
=> true
irb(main):003:0> [""].blank?
=> false

This ^ is the only potential issue with this change that I can think of off the top of my head; if a project is doing presence checks on "empty" values and that logic depends on the original behavior.

This may offer a partial solution to:

https://github.com/scientist-softserv/palni-palci/issues/624


Updates cudf 21.10

@rjzamora merged your PR #1197 here
@benfred added cudf backwards compatibility


Support setup from pyproject.toml

Thanks for your work on this project. I have a suggestion: it would be great if the configuration could be read from pyproject.toml as an alternative to having a separate file mkdocs.yml since pyproject.toml has become the standard place for configuration.

Hi @mcarans, unfortunately we don't have control over this: MkDocs expects configuration to exist in mkdocs.yml.

Ok thanks, i have raised this upstream: https://github.com/mkdocs/mkdocs/issues/3178


build(example-utils): Add explicit return type to getter

api-extractor/typescript do not guarantee that the emitted order of an inferred string-literal union type is stable. This can lead to "flip-flopping" where api-extractor sometimes emits "foo" | "bar" and sometimes "bar" | "foo".
Adding explicit types instead of relying on inference is the best workaround. In this case there was already an exported type defined so I used that instead of an explicit literal union.

/azp run Build - client packages


Make Material Theme Work Again!

Changes the Material Theme Names to the ones in the unpgk Repo.
Material-Darker > Material-Theme-Darker, etc.

Ty for this


Clarification on how to specify what HR DB logs to clear

Making it clear on how to specify what criteria to use when clearing the HR database log

:white_check_mark: Organization membership check passed
Requester is not a member of the MicrosoftDocs organization.


Support float32?

16:52:07 kris ~/faster-whisper $ python test_transcription.py  Traceback (most recent call last):   File "/home/kris/faster-whisper/test_transcription.py", line 8, in <module>     model = WhisperModel(model_path, device="cpu", compute_type="float16") # compute_type="float16"   File "/home/kris/faster-whisper/faster_whisper/transcribe.py", line 71, in __init__     self.model = ctranslate2.models.Whisper( ValueError: Requested float16 compute type, but the target device or backend do not support efficient float16 computation.
I get somewhat of the same error with whisper however it automatically changes to float32. Is there a possible fix for this?

Simply remove the compute_type argument and it will automatically convert the model to float32.

that was easier than expected :D


Picker crops the text

[✓] Review the documentation: https://facebook.github.io/react-native
[✓] Search for existing issues: https://github.com/facebook/react-native/issues
[✓] Use the latest React Native release: https://github.com/facebook/react-native/releases

Environment:
OS: macOS High Sierra 10.13.3
Node: 8.9.4
Yarn: 1.3.2
npm: 6.1.0
Watchman: 4.9.0
Xcode: Xcode 9.2 Build version 9C40b
Android Studio: 3.1 AI-173.4819257
Packages: (wanted => installed)
react: ^16.3.0-alpha.1 => 16.3.0-alpha.1
react-native: 0.54.0 => 0.54.0
I am totally aware of the similar issue and do not understand why it was closed, because issue still prevails.

Select is basically Picker with styled-components wrapping.

<Select
        selectedValue={selected ? selected : value}
        onValueChange={onValueChange}
        itemStyle={!isAndroid && {height: 66, fontSize: 16}}
      >
        {childrens}
      </Select>

#14575

#14575

Why 0.54 is considered to be an old version??


DSS Metrics: lowercase namespace

change namespace for dss to lowercase, allows more flexibility with deployment in data-store repo.
Connected To: https://github.com/HumanCellAtlas/data-store/issues/2177

lgtm
On a related topic, the number of bundled reported deleted on #data-store-eng is always 0 in dev, even though deletions occur every day. Do we know why?

Ive had subscriptions turned on at the endpoint for several weeks now and dont think I've ever seen a event pass through with event_type: DELETE. Although Karon runs everynight, it might not be triggering the notification.... Are bucket events used to trigger off DELETE notifications?

@amarjandu yes:
https://github.com/HumanCellAtlas/data-store/blob/69d412cfc1ecd2c83dfdb597b276f024dcbee3a7/daemons/dss-notify-v2/app.py#L51
https://github.com/HumanCellAtlas/data-store/blob/69d412cfc1ecd2c83dfdb597b276f024dcbee3a7/daemons/dss-notify-v2/app.py#L68
There may be a bug in either the v2 notifier or the metrics code. We should track it down.

not required currently, might come back and merge it in at a later time, if we want to move the DSS to another namespace.


Pipeline components tokenizer_whitespace and tokenizer_spacy errors

rasa NLU version (e.g. 0.8.0a11):
Used backend / pipeline ("nlp_spacy", "nlp_mitie", "tokenizer_whitespace", "ner_mitie", "ner_synonyms"):
Operating system (Linux):
Issue:
When using  "tokenizer_whitespace" or  "tokenizer_spacy":
  File "trainer.py", line 106, in training
    trainer = Trainer(RasaNLUConfig("config_mitie.json"))
  File "build/bdist.linux-x86_64/egg/rasa_nlu/model.py", line 119, in __init__
    components.validate_requirements(config.pipeline)
  File "build/bdist.linux-x86_64/egg/rasa_nlu/components.py", line 89, in validate_requirements
    for package in component_class.required_packages():
AttributeError: 'NoneType' object has no attribute 'required_packages'

Content of configuration file (if used & relevant):
{
  "pipeline": ["nlp_spacy", "nlp_mitie","tokenizer_spacy", "ner_mitie", "ner_synonyms"],
  "language": "en",
  "num_threads": 1,
  "mitie_file": "../../MITIE-models/english/total_word_feature_extractor.dat",
  "fine_tune_spacy_ner": false,
  "max_number_of_ngrams": 7,
  "duckling_processing_mode": "append",
  "entity_crf_BILOU_flag": false,
  "entity_crf_features": [
    ["low", "title", "upper", "pos", "pos2"],
    ["bias", "low", "word3", "word2", "upper", "title", "digit", "pos", "pos2"],
    ["low", "title", "upper", "pos", "pos2"]]
}

Thanks, fixed it. These components were not registered so they couldn't be created.

Hi ,
I facing the issue while sending curl post. Below are the details.
Python version:3.6.5
Rasa_Core Version: 0.11.3
Rasa_NLU Version: 0.13.2
/GET are working fine
--> >curl -H "Accept: application/json" -H "Content-Type: application/json" -XGET localhost:5005/domain
Response:{"actions":["utter_greet","utter_get_pizza_size","utter_get_pizza_toppings","action_order_pizza"],"config":{"store_entities_as_slots":true},"entities":["size","toppi
ngs"],"intents":[{"order_pizza":{"use_entities":true}},{"greet":{"use_entities":true}}],"slots":{"size":{"initial_value":null,"type":"rasa_core.slots.TextSlot"},"top
pings":{"initial_value":null,"type":"rasa_core.slots.TextSlot"}},"templates":{"utter_get_pizza_size":[{"text":"What pizza size?"}],"utter_get_pizza_toppings":[{"text
":"What toppings would you like on your pizza?"}],"utter_greet":[{"text":"Hello, how can I help you?"},{"text":"Hi, I am here to help."}]}}
Where as while for /post method I'm getting error

curl -H "Accept: application/json" -H "Content-Type: application/json" -XGET localhost:5005/webhooks/rest/webhook -d '{"text":"Hello"}'

Response:

500 Internal Server Error
Internal Server Error
The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.
Rasa_core provide the method
custom_webhook.receive    OPTIONS, POST   /webhooks/rest/webhook

Please create your own issue. this does not seem related to this one.


Release v0.41.1

https://github.com/vmware-tanzu/carvel-ytt/releases/tag/v0.41.1
https://community.chocolatey.org/packages/ytt/0.41.1


[Suggestion] A NumberBox that binds directly to decimal values and not double

Hi.  I posted this issue in the 1.7 SDK Plans thread:
https://github.com/microsoft/WindowsAppSDK/discussions/4710#discussioncomment-11365351
You seem like the type of person smart enough to figure out a way to address this problem, but we have a major issue with NumberBoxes not binding directly to decimal values (only double).  When working with currency values stored in databases, you use decimal because double can cause rounding issues.  I don't understand why this isn't a bigger issue for more devs up to this point yet, but the only way to bind the NumberBoxes to decimal values is to use a converter, which seems like could cause the same rounding issues eventually.
I asked SyncFusion to address this and they said that their number box controls are based on the SDK ones so they can only support decimal when SDK does.  Do you think you can address this somehow in DevWinUI?
Thanks for listening/reading!  :)

@MPITech Yes, I am aware of this issue and unfortunately I don't have a solution in mind, the only option is to create a new control.
This problem needs to be solved by Microsoft and as you mentioned, the converter is a solution.
I voted for your comment.
I hope Microsoft fixes this issue soon.


Detected blocking call to open inside the event loop by custom integration 'mila' at custom_components/mila/update_coordinator.py,

Hello,
I am currently on 2024.6.0b7 release and there like they added blocking call detection in 2024.6.0. not sure how to reolve this but reporting so it not a surprise.
Detected blocking call to open inside the event loop by custom integration 'mila' at custom_components/mila/update_coordinator.py, line 37: self._api = MilaApi(MilaConfigEntryAuth(hass, config_entry, MilaOauthImplementation(hass, config_entry))) (offender: /usr/local/lib/python3.12/site-packages/milasdk/api.py, line 38: with open(Path(__file__).parent / './gql/mila_schema.gql') as f:), please create a bug report at https://github.com/sanghviharshit/ha-mila/issues Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module> sys.exit(main()) File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main exit_code = runner.run(runtime_conf) File "/usr/src/homeassistant/homeassistant/runner.py", line 188, in run return loop.run_until_complete(setup_and_run_hass(runtime_config)) File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete self.run_forever() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever self._run_once() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1988, in _run_once handle._run() File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/src/homeassistant/homeassistant/setup.py", line 165, in async_setup_component result = await _async_setup_component(hass, domain, config) File "/usr/src/homeassistant/homeassistant/setup.py", line 447, in _async_setup_component await asyncio.gather( File "/usr/src/homeassistant/homeassistant/setup.py", line 449, in <genexpr> create_eager_task( File "/usr/src/homeassistant/homeassistant/util/async_.py", line 37, in create_eager_task return Task(coro, loop=loop, name=name, eager_start=True) File "/usr/src/homeassistant/homeassistant/config_entries.py", line 742, in async_setup_locked await self.async_setup(hass, integration=integration) File "/usr/src/homeassistant/homeassistant/config_entries.py", line 594, in async_setup result = await component.async_setup_entry(hass, self) File "/config/custom_components/mila/__init__.py", line 19, in async_setup_entry coordinator = MilaUpdateCoordinator(hass, entry) File "/config/custom_components/mila/update_coordinator.py", line 37, in __init__ self._api = MilaApi(MilaConfigEntryAuth(hass, config_entry, MilaOauthImplementation(hass, config_entry)))

this is not in this project, you need to open this in the lib under this project look up milasdk

I haven't had much time to look into this, but it would involve refactoring the sdk a bit - this call is to get the GraphQL schema from a file.  It shouldn't pose much of an issue (small file) for the time being, but is something that should be fixed.


OSD-12071 Add annotation to disable global namespace resolution

This adds an annotation to the OperatorGroup to allow OLM to be more resilient if a catalog source becomes in a bad state.
References
https://bugzilla.redhat.com/show_bug.cgi?id=2076323
https://issues.redhat.com/browse/OSD-12071

@dustman9000 Thanks for creating this PR!

/approve


Aurora master MBR disk creation broken

Tests are failing and it seems located on the MBR creation
It times out because it wont create the disk

Times out because it wont start the install. Kairos service is failing to start
× kairos.service - kairos installer
     Loaded: loaded (/etc/systemd/system/kairos.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2024-10-23 07:41:38 UTC; 2s ago
   Duration: 914ms
    Process: 1713 ExecStartPre=/bin/sh -c dmesg -D (code=exited, status=0/SUCCESS)
    Process: 1714 ExecStartPre=/usr/bin/kill -SIGRTMIN+21 1 (code=exited, status=0/SUCCESS)
    Process: 1715 ExecStart=/usr/bin/kairos-agent install (code=exited, status=1/FAILURE)
    Process: 1716 ExecStartPost=/usr/bin/kill -SIGRTMIN+20 1 (code=exited, status=0/SUCCESS)
   Main PID: 1715 (code=exited, status=1/FAILURE)
        CPU: 422ms

No users found in any stage that are part of the 'admin' group.
In Kairos 3.3.x we no longer ship a default hardcoded user with the system configs and require users to provide their own user.Please provide at least 1 user that is part of the 'admin' group(for sudo) with your cloud configs.If you still want to continue without creating any users in the system, set 'install.nousers: true' to be in the config in order to allow a system with no users.

now after estoring the user+admin, it seems like the kairos service wont start automatically?
sudo systemctl status kairos
○ kairos.service - kairos installer
     Loaded: loaded (/etc/systemd/system/kairos.service; disabled; vendor preset: disabled)
     Active: inactive (dead)

seems like its missing the nodepair.enable in the cmdline to auto start the service?
BOOT_IMAGE=(cd)/boot/kernel cdroot root=live:CDLABEL=COS_LIVE rd.live.dir=/ rd.live.squashimg=rootfs.squashfs net.ifnames=1 console=ttyS0 console=tty1 rd.cos.disable vga=795 nomodeset install-mode selinux=0 rd.live.overlay.overlayfs

But the config files in the image seems to be a bit old, so they dont support the new install-mode thing

done, it was old versions + missing admin group for kairos user


Discord bot gif backdrop color

We currently use a white background for gif previews (https://github.com/webaverse/app/blob/master/screenshot.html#L294), we could use the Discord background to make it look like the items are moving / living "inside" of discord
Discord uses the hex color #36393f for their background.

actually, in this case, the bg is different when your @ named

What about a grad?


New utilities classes should be global

Record type, field information, etc. utilities classes should be global to allow external code to use these.

Included in beta release version 3.32 (Beta 8) and higher

Included in production release version 3.32


Add Polygon support

Overview
This PR adds support for rasterizing Polygons
Closes https://github.com/pyviz/datashader/issues/181.
For example usage, see notebook at https://anaconda.org/jonmmease/datashader_polygons_pr/notebook
GeomArray ExtensionArrays
In order to rasterize polygons efficiently, we need a data structure that can store an array of polygon definitions in a form that is directly accessible to numba.
To accomplish this, I added a new RaggedArray (see https://github.com/pyviz/datashader/pull/687) subclass called PolygonsArray.  Each element of this array can store one or more polygons with holes.  So elements of a PolygonArray are roughly equivalent to a shapely Polygon or MultiPolygon.  The entire PolygonsArray is roughly equivalent to a geopandas GeometryArray of Polygon/MultiPolygon elements.
The new PolygonsArray pandas extension array could eventually grow to support many of the operations supported by the geopandas GeoSeries.  The advantage would be that these operations could be implemented in vectorized form using numba for potentially improved performance, and Dask DataFrame support would largely come for free.
To demonstrate the potential, I added length and area properties to the PolygonArray class. These operations are ~8x faster than the equivalent GeoSeries operations, and they could also be naturally parallelized using Dask for large datasets.
Canvas methods
New Canvas.polygons() method has been added to rasterize polygons, and the Canvas.line() method has been updated to support these new geometry arrays, making it easy to draw polygon outlines.
Examples
For code and timing, see https://anaconda.org/jonmmease/datashader_polygons_pr/notebook

cc @jbednar @philippjfr

This looks great! Particularly glad to see the conversion functions from geopandas. My main question is about all the extension array work and how that overlaps with the work that is going on within geopandas itself. Do we want to continue maintaining these extension arrays? Do we push to merge the implementations with geopandas?  Do we work with a project like awkward-array to get a shared implementation across all the different projects?

Do we push to merge the implementations with geopandas?

This work is currently living in Datashader, but one of the motivations was to have (somewhere, not sure yet!) an implementation of Shapely-like processing in Dask and Pandas that is fully independent of the geospatial stack that GeoPandas uses, i.e. Shapely, GEOS, GDAL, etc.  Being independent of that stack makes it much more flexible and easier to deploy on a variety of systems, and so I don't think we should be trying to move this code into GeoPandas itself.  That said, it would be possible to abstract some of it into something that this code and GeoPandas can both depend on, but that doesn't rely on the legacy stack.  I don't know how feasible that is, but it does sound desirable!
In the meantime, we'll have the limitation that we don't have a standard way to persist these objects, which means that for I/O we are still tied to the legacy stack. :-(

As for awkward-array, it definitely looks interesting and having it be externally maintained is very attractive, but I haven't yet seen anything significant it would add for these use cases.  E.g. I don't see any persistence format in https://github.com/scikit-hep/awkward-1.0 , and the operations to be supported here don't seem to have much overlap with those proposed for awkward...

That said, it would be possible to abstract some of it into something that this code and GeoPandas can both depend on, but that doesn't rely on the legacy stack.

Yes, that was my hope, duplicating these things all over the place does not seem ideal (even if it's fine for the time being).

In the meantime, we'll have the limitation that we don't have a standard way to persist these objects, which means that for I/O we are still tied to the legacy stack. :-(

Right there are quite a few issues that are unsolved which is why I would like to push towards some shared implementation that will solve all these issues, and awkward-array seems like the solution most likely to get there particularly if we encourage it in some form. It has a community of hundreds of folks which will be relying on it for their daily work.

I don't see any persistence format in https://github.com/scikit-hep/awkward-1.0

Quoting the README: "Interoperability with ROOT, Arrow, Parquet and Pandas, as well as planned interoperability with Numba and Dask.", with parquet and arrow both being persistence formats. In the longer term goals it also states: "Translation to and from Apache Arrow and Parquet in C++."

the operations to be supported here don't seem to have much overlap with those proposed for awkward...

This is absolutely true but the idea as I understood it is that it would allow implementing numba CPU and GPU kernels on top of the arrays, which could then be exposed as functions or on custom array subclasses.

Interoperability with...

At this point those are all just plans, though. E.g. they mention Parquet export, but Parquet doesn't have any support for ragged arrays that I know of, so presumably they only mean export of non-ragged arrays to Parquet.  Extending Parquet (via Arrow/Fletcher?) to support ragged data is a separate task and necessary with or without Awkward, as far as I can see.

would allow implementing numba CPU and GPU kernels

How would that differ from what is already implemented here?  Doesn't this ExtensionArray already support those?

Yes, I think this would definitely be a longer term plan.

How would that differ from what is already implemented here? Doesn't this ExtensionArray already support those?

Good question I don't have an answer to.

so presumably they only mean export of non-ragged arrays to Parquet.

This seems very unlikely to me since almost all their data has ragged arrays in some form, so parquet support without that would be entirely useless for them.

parquet support without that would be entirely useless for them.

Maybe that's a question you could raise with that community, because Parquet is very explicitly a columnar storage format (it's the first line on their homepage!), and a Google search for parquet+ragged only comes up with awkward, not anything in Parquet.

Worked fine in the old awkward already so I don't see why it wouldn't work in the new one. In the end ragged arrays are just stored as columns with offsets to compute the start and stop of each ragged chunks:

In the end ragged arrays are just stored as columns with offsets to compute the start and stop of each ragged chunk:

Hmm. I can't tell if that would work here or not.  What we need is columnar, but with one chunk per row, and each row lining up with other Pandas columns containing e.g. one scalar per row.  I can see how this column works, but not how it maps to other columns in the same Parquet file.

Looks like it does come back organized that way when your file above is read into Pandas, resulting in an object column:
>>> pd.read_parquet("test.parq")

0  [0.7556205061410939, 0.6017337297767158, 0.162...
1  [0.021352689589037355, 0.8700607878641867, 0.2...
2  [0.42894486202061544, 0.6228601294191625, 0.81...

So maybe it will work, just not getting an object column in our case.

Cross reference https://github.com/pyviz/datashader/issues/720#issuecomment-537548914 regarding extension array serialization.
For the geometry constructs, I think we only really need one level of nesting, which is what the current RaggedArray supports.  This maps 1-to-1 with the arrow ListArray type, which already has parquet serialization support through pyarrow.  So serialization to parquet should already be possible. Once the pandas and pyarrow teams settle on metadata conventions it should also be possible to read a parquet file into a pandas DataFrame and automatically restore the extension arrow columns.

Great to hear!

Btw, I'm totally not objecting to the idea that this implementation is the one we continue to use internally. I'm just saying that it'd be a shame if it was used exclusively by datashader and lots of other people end up duplicating the work. So I'm just suggesting that we should either move it out of datashader eventually OR switch to using other implementation (but only if it makes sense to do so).

Oh, I definitely don't want the underlying ragged array support to stay in Datashader, so we're on the same page there.  It's here until we figure out what to do with it, to avoid committing to a particular implementation until we are really ready for that. The ideal is if it can migrate to a suitable project already maintained by others, and barring that I think it would need to be in its own project that can have a life outside of Datashader.

For the geometry constructs, I think we only really need one level of nesting, which is what the current RaggedArray supports. This maps 1-to-1 with the arrow ListArray type, which already has parquet serialization support through pyarrow.

That's a really good reason to stick with the +Inf/-Inf convention that lets us use flat arrays rather than lists of arrays. Good thing we went with that; whew!

Yeah, I agree that it would make sense to eventually move these geometry focused extension arrays into a separate package.  But, I'm not familiar enough with the communities around GeoPandas to know whether there are any existing packages that would be a good fit.

@jonmmease , can you add an example above of plotting the same data as filled polygons, polygon outlines, and isolated points? The first two are illustrated, but not the third.

I'm not familiar enough with the communities around GeoPandas to know whether there are any existing packages that would be a good fit.

I'm pretty sure @jorisvandenbossche will be able to help guide us in that...

can you add an example above of plotting the same data as filled polygons, polygon outlines, and isolated points?

Sure, I updated the notebook at https://anaconda.org/jonmmease/datashader_polygons_pr/notebook with examples of lines and points.

Perfect, thanks!  I figured there might need to be some small fix to support that. :-)

Thanks for pinging me!
(first answering on some other aspects of the discussion above)

In the meantime, we'll have the limitation that we don't have a standard way to persist these objects, which means that for I/O we are still tied to the legacy stack. :-(

It already has become clear in the above discussion I think, but to be explicit: parquet notably does support ragged arrays (or at least the kind of ragged arrays we are dealing with here with a regular nesting). In "parquet parlance" it is called "nested" data types, and specifically a "list" type.
Arrow also natively supports this kind of data (ListArray / ListType), and thus arrow + parquet can be a natural choice for a good serialization format of this kind of data.
As I also mentioned on one of the other issues, I think this is a good reason to choose a memory format that maps to one of arrow's types (another reason is that this then will directly work on the GPU as well, as RAPIDS is using arrow memory).
But @jonmmease already mentioned this is the case right now with the RaggedArray, so that is great.
Note that it would actually be possible to not use the inf/inf approach, but to use a second level of nesting like a "list of list of float array" (the data itself stays a flat array, but you get an extra set of offsets), which still perfectly maps to arrow/parquet types. Now, I don't know the code that uses this format for generating the glyphs, and it might be that the inf approach gives a better performance?

So serialization to parquet should already be possible. Once the pandas and pyarrow teams settle on metadata conventions it should also be possible to read a parquet file into a pandas DataFrame and automatically restore the extension arrow columns.

Yes, that is something I am actively working on right now, and normally, with the next arrow version, it should be possible to have a perfect roundtrip to arrow/parquet for a custom ExtensionArray.
The pandas -> arrow/parquet direction already works if you implemented ExtensionArray.__arrow_array__ on the custom extension arrays here. The reverse will (probably, this is still an open PR) work by implementing a ExtensionDtype.__from_arrow__. And with those two, a full roundtrip preserving the types should be possible.
See https://github.com/pandas-dev/pandas/issues/20612#issuecomment-530749254 for the pandas issue, and https://issues.apache.org/jira/browse/ARROW-2428 / https://github.com/apache/arrow/pull/5512 for the Arrow work-in-progress. Feedback on that is very welcome! (as it is exactly this kind of applications that this new work should enable)

(and then some comments regarding the geospatial aspects)

I'm not familiar enough with the communities around GeoPandas to know whether there are any existing packages that would be a good fit.

I am also not aware of any existing packages in this space in Python. But, I think it would certainly be useful to have this functionality somewhere (and not semi-hidden in datashader). Where the best place would be, not directly sure, will think a bit more about this (short term it is of course perfectly fine to have it here).
At the moment, I think that GeoPandas would rather keep focus on geometries in sense of what is supported by GEOS, and not necessarily representations based on ragged arrays.
On the other hand, if exposing such functionality, it would be nice to have good integration with and a similar interface as geopandas (would it eg be possible to enable having a GeoDataFrame but with geopandas' GeometryArray swapped with a PolygonArray? That could be an interesting option to explore, where the new array types could still live in a side project)
Regarding performance:

The new PolygonsArray pandas extension array could eventually grow to support many of the operations supported by the geopandas GeoSeries. The advantage would be that these operations could be implemented in vectorized form using numba for potentially improved performance, and Dask DataFrame support would largely come for free.
To demonstrate the potential, I added length and area properties to the PolygonArray class. These operations are ~8x faster than the equivalent GeoSeries operations, and they could also be naturally parallelized using Dask for large datasets.

We are working on better performance in GeoPandas in the PyGEOS project (https://github.com/pygeos/pygeos/), so for comparisons, that is also an interesting target. I took some comparisons from Jon's notebook and redid the comparison of the geopandas method but now with pygeos:
In [19]: import pygeos

In [19]: arr = pygeos.from_wkb(geopandas.array.to_wkb(world.geometry.array))

In [20]: length_gp = %timeit -o world.geometry.array.length
696 µs ± 37.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [21]: length_pg = %timeit -o pygeos.length(arr)
156 µs ± 6.72 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [22]: print("pygeos.length speedup: %.2f" % (length_gp.average / length_pg.average))
pygeos.length speedup: 4.47

In [23]: area_gp = %timeit -o world.geometry.array.area
709 µs ± 55.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [24]: area_pg = %timeit -o pygeos.area(arr)
118 µs ± 1.6 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [25]: print("pygeos.area speedup: %.2f" % (area_gp.average / area_pg.average))
pygeos.area speedup: 6.01

I didn't compare with this PR on my laptop, but compared to geopandas, there is also a considerable speed up with pygeos. The numba-solution based on the ragged array representation in this PR still gives a slightly bigger speed-up, but with pygeos the difference gets much smaller.
Now that said, the implementation in this PR could maybe be further optimized (further optimizing pygeos means optimizing GEOS C++, which is a harder target). And there will be cases where a custom implementation can certainly be a lot faster than GEOS (distances between points is a notable one ..). And of course there are also other reasons to want to have an array-based, non-GEOS representation of geometries (such as the specific usecase here in datashader, avoiding dependencies, ..). Just don't underestimate the complexity of much of the functionality that now GEOS provides for us.
For those conversions between GEOS-based geopandas and ragged-array based ExtensionArrays here: to have the best possible performance for those conversions, it might be useful to see if certain aspects can be implemented in PyGEOS.
In a quick experiment, I check getting the flat numpy array of coordinates from the "countries" dataset (the example used in Jon's notebook):  when using pygeos instead of the PolygonsArray.from_geopandas method of this PR, the time goes down from 45ms to 150µs (factor 300). See https://nbviewer.jupyter.org/gist/jorisvandenbossche/de8bfd1c1a7f62e25fd6f29bd9c3c0e3. Now to be clear, the pygeos method I used only gives you the coordinates, not the array of indices. But at least it shows a faster method is possible (I think also returning an array of indices in pygeos should only give a limited slowdown).

Thanks for taking the time to weigh in @jorisvandenbossche!

Note that it would actually be possible to not use the inf/inf approach, but to use a second level of nesting like a "list of list of float array"

Oh cool, I didn't realize that you could have multiple nested levels like that.  The way things work right now is that the +inf/-inf values are used to indicate whether the polygon that follows is in counter clockwise (filled) or clockwise (hole) order.  I'll think a bit on whether there are any particular pros/cons to these representations.

Yes, that is something I am actively working on right now...

Awesome, thanks for taking that on!

We are working on better performance in GeoPandas in the PyGEOS project...

Yes, I saw your announcement for PyGEOS and that looks like a great step forward in terms of performance.  And based on your benchmarking, it looks like working at the level of PyGEOS for the geopandas <-> ragged conversions could be very beneficial.

Now that said, the implementation in this PR could maybe be further optimized

By parallelizing the numba implementation of area/length I was able to get a significant speedup over the single threaded version timed above.  Do you know if it would be possible to run the PyGEOS ufuncs in a multi-threaded context?

I am also not aware of any existing packages in this space in Python. But, I think it would certainly be useful to have this functionality somewhere (and not semi-hidden in datashader).

@jbednar @jorisvandenbossche Here are some notes I started on what a new project might look like that extracts and generalizes the geometric/spatial concepts that Datashader needs.  There would certainly be a long tail to implement the various GEOS features on the ragged representation in numba, but I expect it could be useful before everything is fully fleshed out.  Let me know if you have any other general thoughts on this approach and how it fits in the ecosystem.
https://github.com/jonmmease/spatialpandas/issues/1
Thanks!

The way things work right now is that the +inf/-inf values are used to indicate whether the polygon that follows is in counter clockwise (filled) or clockwise (hole) order. I'll think a bit on whether there are any particular pros/cons to these representations.

I thought the np.inf was used to mark the beginning of a new part of a MultiLineString/MultiPolygon and the -np.inf to mark beginning of the interiors (holes) of a single Polygon (in shapely terminology)? But writing this, I suppose that means the same in the end (if the winding of your shapely polygons / input is correct).
It could be similar to how eg GeoJSON does it with nested lists: inside the list for a single geometry, the first list is the exterior, and any following lists are holes, if they are present. And for a MultiPolygon, you then get a list of such nested lists. You get a lot of (virtual) nesting then (but the actual coordinates are still just a flat array), where you have offsets to the beginning of each MultiPolgyon, and then offsets to beginning of each polygon inside that, and then offsets to the parts (exterior / holes).

Do you know if it would be possible to run the PyGEOS ufuncs in a multi-threaded context?

Yes, that is possible. There is still some work to get there (releasing the GIL, I have have POC branch that does this for some operations), but I did a quick experiment on it: https://jorisvandenbossche.github.io/talks/2019_FOSS4GBE_pygeos/#32

Here are some notes I started on what a new project might look like that extracts and generalizes the geometric/spatial concepts that Datashader needs.

Thanks! I will try to comment on that issue shortly.

I thought the np.inf was used to mark the beginning of a new part of a MultiLineString/MultiPolygon and the -np.inf to mark beginning of the interiors (holes) of a single Polygon (in shapely terminology)? But writing this, I suppose that means the same in the end (if the winding of your shapely polygons / input is correct).

Yeah, that's correct. In this PR I'm using the winding number fill algorithm and assuming that the main polygon is CCW and holes are CW.  I run the shapely polygons through shapely.polgon.orient during import to make sure this is true.  It occurs to me that this might be part of why the import is a bit slow.

It could be similar to how eg GeoJSON does it with nested lists: inside the list for a single geometry, the first list is the exterior, and any following lists are holes, if they are present. And for a MultiPolygon, you then get a list of such nested lists. You get a lot of (virtual) nesting then (but the actual coordinates are still just a flat array), where you have offsets to the beginning of each MultiPolgyon, and then offsets to beginning of each polygon inside that, and then offsets to the parts (exterior / holes).

For this approach, are you picturing that the extension array itself would store the polygons and holes as nested Arrow ListArrays, rather than numpy arrays?  Or just that this format could be used for parquet serialization?  In either case, it's not clear to me how to get the offsets back out of a pyarrow ListArray. For example
import pyarrow as pa
la = pa.ListArray.from_arrays(offsets=[0, 3, 5], values=[1, 2, 3, 4, 5])
la

<pyarrow.lib.ListArray object at 0x7f3535fbcbe8>
[
  [
    1,
    2,
    3
  ],
  [
    4,
    5
  ]
]

There's a values property to get the flat array out.
la.values

<pyarrow.lib.Int64Array object at 0x7f3535fbcb28>
[
  1,
  2,
  3,
  4,
  5
]

But I don't see how to get the [0, 3, 5] offsets array back out.  Looking quickly at the pyarrow source, it looks like maybe this isn't implemented in Python yet.
https://github.com/apache/arrow/blob/89b5a2484161c9cab7bb33d899d696729f119126/python/pyarrow/array.pxi#L1200
Apart from this implementation question, it would be pretty compelling if we could come up with a canonical ragged representation of homogenous geometry arrays and if PyGEOS could efficiently create these arrays directly from geopandas geometry arrays.

But I don't see how to get the [0, 3, 5] offsets array back out

Ahh, never mind. I see that I can get at it through the chunk's buffers.

I run the shapely polygons through shapely.polgon.orient during import to make sure this is true. It occurs to me that this might be part of why the import is a bit slow.

Ah, yes, indeed. From a quick profile, it seems that 2/3rd of the time is coming from orient. And so that part was not included in my pygeos timing. From a quick look at the source of orient, a pygeos version can probably be faster as well. Opened an issue for that here: https://github.com/pygeos/pygeos/issues/71

For this approach, are you picturing that the extension array itself would store the polygons and holes as nested Arrow ListArrays, rather than numpy arrays? Or just that this format could be used for parquet serialization?

Yes, just that it can be used for serialization, if you use numpy arrays with the same layout (so without depending directly on pyarrow). Eg if we store the arrays here in a PolygonArray, we could easily construct the arrow array with pa.ListArray.from_arrays when doing serialization.

But I don't see how to get the [0, 3, 5] offsets array back out

Ahh, never mind. I see that I can get at it through the chunk's buffers.

Yes, but that is not very user friendly I think. We should probably improve the interface in pyarrow to have easier access to the offsets. I opened https://issues.apache.org/jira/browse/ARROW-7031 for this.

@jonmmease , I'll leave that to your discretion; I don't have any inside knowledge of it other than that I believe @mrocklin proposed using fletcher originally for this purpose. I don't see any problematic dependencies...

I don't know to what extent you want pyarrow as a hard dependency for this functionality, it is still a quite big dependency.
Although I am actually somewhat biased here to use pyarrow, I am not sure there is already that much added value for directly using pyarrow, since there is not much functionality of pyarrow that you would use (except for storing the arrays).
In the end, for much of the functionality you implement in datashader, you need access to the actual data (flat array + offsets): eg looping through the inner lists in numba to create glyphs.
So whether you store the array + offsets yourself in an ExtensionArray, or directly in a pyarrow.ListArray (wrapped in an ExtensionArray), for the code using this extension array in datashader, it doesn't matter much I think.
Of course, directly using pyarrow might make the implementation of the PolygonArray a bit simpler.
And also, most people will probably want to use this with good serialization trough eg parquet, and in that case they need to have pyarrow installed anyway.

In addition the text functionality, fletcher also has a base FletcherArray (ExtensionArray that wraps a pyarrow Array) that could be used as a base class to inherit from for a LineArray/PolygonArray/..
Now, I am not familiar enough with fletcher's development to know it is already stable enough to use as dependency.

I'm very excited about this work. I was playing around with applying this to a project that fell by the wayside a couple of years ago. Here are some initial results:

This dataset has just over 800K polygons. As you can see converting the dataset from GeoPandas takes a significant amount of time. @jonmmease you mentioned that this may be because of the running the geometries through shapely to ensure orientation is correct. I wonder if that check could be optional, and turned off for data that has been preprocessed. I'm less familiar with the other formats being discussed, but if a faster IO approach could be supported that would be great!

| I don't know to what extent you want pyarrow as a hard dependency for this functionality, it is still a quite big dependency.
To make that concrete, here are the additional conda packages that get installed when you make an datashader environment with pyvarrow, beyond what you get with datashader itself:
arrow-cpp
boost-cpp
brotli
bzip2
double-conversion
gflags
glog
icu
libboost
libevent
libiconv
libprotobuf
lz4-c
pyarrow
re2
snappy
thrift-cpp

A few of those are no big deal, and already likely to be in a real environment anyway (snappy is used in many Datashader examples, bzip2 and lz4-c are useful for anyone reading input files, and some others look straightforward), but others do look substantial and potentially problematic.  That said, only a fraction of our users need the ragged array support, and so we can certainly consider requiring pyarrow just for those users, if it increases interoperability and reduces our maintenance burden and codebase size or complexity. Jon, I think I have to leave that decision to you; please go with whichever choice you feel most comfortable with, keeping all the above in mind!

Apart from data import time, what are you're impressions of the rendered result and the performance? Do you have anything to compare it to? Do the speckled locations toward the lower left make sense?

The rendered result look great, and is much faster than any other tool I've used (ArcGIS) takes several minutes to visualize this dataset and it has to re-render anytime you do anything to the map. Rending this in a notebook with GeoViews takes hours/days (if it doesn't crash).
The speckled appearance is accurate for the data. In fact the whole thing is much more speckled than it  appears with the blue shade color map. This render helps to show how speckled the whole area is:

It would now be useful to be able to zoom in to explore the data better. Is there a way to plot this with Bokeh so the zoom/pan controls would be available. I'd also like to be able to overlay this on some map tile. I've done that with other datasets using GeoViews/HoloViews, but can't work out how to do that with this data type.

I've done that with other datasets using GeoViews/HoloViews, but can't work out how to do that with this data type.

Once there is agreement that the data format for this is finalized I'll be opening a HoloViews PR to add support for this glyph type, so you'll be able to be able to use the usual approaches.

@jorisvandenbossche, quick question on __arrow_array__ serialization.  It looks like pyarrow doesn't currently support returning a ChunkedArray in  __arrow_array__, which is what fletcher uses as the storage data structure.  Would this be difficult to support at the pyarrow level?

Thanks for taking a look at fletcher. First and foremost: I'm actively looking to develop it further and also happy to spent some cycles to make it more useful for others. Important things to note about it:

fletcher started out as a library for faster strings but nowadays supports any pyarrow-backed array type. My main personal use case for it will be efficient string and boolean arrays, I'm not seeing other types where I have the need at the moment.
I would see it as a place where we could develop general algorithms on top of pyarrow-backed ExtensionArrays. Probably we should keep it limited to general purpose algorithms but I'm happy to temporarily house geo-related algorithms if that makes development simpler.
The long list of dependencies of the conda-forge pyarrow package can be reduced by splitting the recipe into multiple outputs. I plan to do this at some time but no work has been done there yet. This will only happy for the conda package, not the wheel as pip/setuptools isn't sophisticated enough to handle this. This is meant as an FYI if the list of dependencies becomes a problem in future.
Long-term I hope that most of it gets absorbed into pandas and pyarrow. While this is not the case, it is there to faster iterate on getting the missing pieces together and find out what is best to go into the both mentioned packages.
At the moment only the latest "major" version (i.e the latest 0.x release series)  of pyarrow and pandas is supported and I plan to keep it that way as things are rapidly improving in both pyarrow's and the pandas.ExtensionArray API. Supporting older version will be quite of a lot of a burden. Happy to accept PRs from the outside but I doubt that anyone will go the extra effort.

Also I see that @jonmmease has started prototyping https://github.com/jonmmease/spatialpandas, I guess this on top of fletcher seems like a good idea.

Hi @xhochy, thanks for taking the time.  That was very helpful.  As I mentioned in https://github.com/pyviz/datashader/pull/826 and https://github.com/jonmmease/spatialpandas/pull/2, after a bunch of prototyping I decided not to subclass FletcherArray for the time being.  The class works great, and I got pretty far down the road using the subclass approach.  I just ran into some issues that I think stem from the fact that I want __getitem__(int_key)  to return instances of a custom class wrapping the arrow result.  I ended up using FletcherArray as a helpful reference in writing a custom ExtensionArray that wraps a numeric ListArray.  This "numeric ListArray" restriction also helped simplify the required logic a bit (e.g. no string or ChunkedArray handling).  Thanks again!

@sdc50

The rendered result look great, and is much faster than any other tool I've used (ArcGIS) takes several minutes to visualize this dataset and it has to re-render anytime you do anything to the map. Rending this in a notebook with GeoViews takes hours/days (if it doesn't crash).

That's great news! Thanks for taking the time to try this out and report back.

This dataset has just over 800K polygons. As you can see converting the dataset from GeoPandas takes a significant amount of time. @jonmmease you mentioned that this may be because of the running the geometries through shapely to ensure orientation is correct. I wonder if that check could be optional, and turned off for data that has been preprocessed

I think the best way forward here is probably to build import export capabilities on top of (or in) the PyGEOS library.  For the time being, we can at least add the option to disable the orient logic. I did this in the new extension arrays that will replace those in this PR at https://github.com/jonmmease/spatialpandas/pull/2.

This PR is now superseded by https://github.com/pyviz/datashader/pull/826 and https://github.com/jonmmease/spatialpandas/pull/2.
With this approach the extension arrays are built on nested pyarrow ListArrays as part of the new (not yet published) spatialpandas library, and there is a separate extension array corresponding to each shapely/geopandas shape class.  With the multi-level nesting support, nan/+inf/-inf values are no longer needed as separators which cleans up the Datashader rendering logic a bit.
Efficient serialization to parquet already works, and reading from parquet will be possible when such support is added to pandas.
Thanks for the discussion all. I'm going to close this PR and we can continue on https://github.com/pyviz/datashader/pull/826 regarding polygon rendering and on https://github.com/jonmmease/spatialpandas/pull/2 regarding the geometric extension arrays.


Wrong whitelist tags management

"Something we noticed in the Mario Party discord is that the bot posts if a channel is playing a Mario Party game and has Drunk in its title beause we have run as a keyword
It isnt a huge deal, but I just though I'd let you know that it was happening due to how the bot currently handles strings"
Found by renn on Discord

Changed the way tags are checked in the titles.
Before : If the title contains the tag somewhere, no matter how, it's validated
Now : If the tag is the beginning of a word (helps for plural forms : "run" => "runs" accepted, "Drunk" is not) it's validated


Setup auto unit testing with different number of distributed machines

It's especially important to unit test with many different numbers of machines, since some code has different logic based on whether contiguous amplitude sets fit entirely into a chunk.

Unit testing buffed to 8 processes


Fix localServePath for servePath’s without protocol.

@finik replaces and extends #275.
I think the test in 7e758db covers the case you were explaining — take a look and double check, if you will.

I think this is right (and @finik can correct me if it’s wrong) so I’ve published it in 4.4.0.


[vm][data cache] Small cleanup of effects return type

Changes will becoming to make the type layout eager
This change will make those changes (and any other future changes) easier to grok

Motivation

Normally I think this change wouldn't be needed now, but for changing the type layout code and making it "eager", this will make that whole process easier

Test Plan

cargo test

/land


Runtime check - Cycle should end even if lookupProvider fails

For this usage, the cycle runtime check would be spoiled.
try {
  scope.getInstance(A.class);
} catch (XXX e) {
  ...
}

A would stay inside cycleDetectionStack, even though the injection for A finished.

Coverage decreased (-0.01%) to 94.965% when pulling 15108be7827e5698db3d760fcff48a560ea09f9b on dlemures/runtimecheck-closecycle into 2e6fb96653c3942727a7862f58a015d8d981cdba on master.

Coverage increased (+0.1%) to 95.092% when pulling e29060496e73caf778668b288fa7cbae332bd27f on dlemures/runtimecheck-closecycle into 2e6fb96653c3942727a7862f58a015d8d981cdba on master.


Fix main content area on developers/show from butting up against site footer

Purpose
The main content area on the developers/show page was butting right up against the footer. This PR fixes that.
Screenshots
Before:

After(desktop):

After(mobile):

Approach
Add mb-16 to the outer most div on developers/show page. mb-16 seemed to be the standard elsewhere across the site, so I rolled with that.

Thanks for the PR! Can you take a screenshot on mobile for documentation sake?

@joemasilotti done!

Awesome, thanks for the contribution!


Can jinsta run on a Mac?

I've installed nodejs but everytime I run jinsta I'm asked to enter sms / email code. After entering the code, I get an error. I would appreciate it if someone can share documentation or post a video on how to install and run jinsta on a Mac. Thanks in advance.
Here's the error:
sharman@iMac sps_jinsta % jinsta -u ******* -p ***** -w ./jinsta_data -t vegan -likelimit 30
info:    logging in with user: ******** ...
? Enter sms / email code: 062874
(node:3711) UnhandledPromiseRejectionWarning: IgLoginRequiredError: GET /api/v1/loom/fetch_config/ - 403 Forbidden; login_required
at Request.handleResponseError (/usr/local/lib/node_modules/jinsta/node_modules/instagram-private-api/dist/core/request.js:93:24)
at Request.send (/usr/local/lib/node_modules/jinsta/node_modules/instagram-private-api/dist/core/request.js:49:28)
at async LoomRepository.fetchConfig (/usr/local/lib/node_modules/jinsta/node_modules/instagram-private-api/dist/repositories/loom.repository.js:6:26)
(node:3711) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3711) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

i just fixed this issue in the latest version some minutes ago :)
can you update and try again please ? and show me the output of jinsta -v so i know that you run the latest version :)

I had the same issue - with the update & !! disabling 2factor authentication it now works fine :-)

@breuerfelix I updated to 0.0.37 but the errors continue.
sharman@iMac sps_jinsta % jinsta -v
0.0.37
sharman@iMac sps_jinsta % node index.js
info:    logging in with user: sharman.shah ...
? Enter sms / email code: 934081
(node:9600) UnhandledPromiseRejectionWarning: IgLoginRequiredError: GET /api/v1/direct_v2/inbox/?visual_message_return_type=unseen&thread_message_limit=10&persistentBadging=true&limit=20 - 403 Forbidden; login_required
at Request.handleResponseError (/Users/sharman/sps_jinsta/node_modules/instagram-private-api/dist/core/request.js:93:24)
at Request.send (/Users/sharman/sps_jinsta/node_modules/instagram-private-api/dist/core/request.js:49:28)
at async DirectInboxFeed.request (/Users/sharman/sps_jinsta/node_modules/instagram-private-api/dist/feeds/direct-inbox.feed.js:21:26)
(node:9600) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:9600) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

@die-tueftelei on my phone, I checked Settings>Security>Two-factor authentication but it already seems to be off and it's asking me to set up Two-factor authentication. Can you guide me how you managed to disable it?

Exactly there in the app I had activated it previously.. As I found no other possibility I remembered I activated that feature 2 month before.. Now my login issues with jinsta are solved..

It's been over a month now and I'm still unable to login. My Two-factor authentication is switched off.

Should 2FA be on or off?
If it's supposed to be on, should 2FA be set to 'Text message' or 'Authentication app'?


Image Card: Repetitive Links

The problem
There are repeated links within the image card that to the same destination, one for the image and one for the title. This creates additional tab stops (# of items in keyboard focus) that over time will lengthen the amount of time required by a keyboard-only or screen reader user to interact with the site.
The solution
Ideally, there should be a single link for an "item" that takes users to the detail page. If possible, have the items title become the keyboard focusable link, and set up the click event for the image as a separate mouse-only action that does not receive focus.
Alternatives considered
There are also ways to expand the 'clickable' area of a link through CSS.
Additional information
Additionally, depending on the implementation, the Pharos documentation should include information about not have the item's container gain keyboard focus unless there is some action that can be taken while receiving focus, ie. pressing Enter to go to detail page, using arrow keys to move to the next item, etc. If this approach is taken, then neither the image or title would gain focus after the outer container, and would move directly to any other inner links (author/collection).

@sirrah-tam is this a duplicate of #353? If there's nuance between them, might we reconcile them together?

Aha, yes that is exactly it! Should this one be closed and include my comments on the other one?

@sirrah-tam That sounds good!

Closed as duplicate of #353


Multi version lsm docs (#7973)

Pull request opened by the merge tool on behalf of #7973

Merged into branches iso7 in cab67dde053c01a6cdcb06080df4816685585e25


Compatible api extension version v1

What type of PR is this?
/enhancement
Which issue(s) this PR fixes?
#3
What this PR does?
Summary:
Compatible api extension version v1
Special notes for your reviewer?
N/A

LGTM


Remove Chat

Remove the Chat system from all files.

@faradaym
This is what I was referring to during the TH. Where chat calls into the Duck::sendData() method to actually send the CDP Packet containing the chat message, across the mesh.
This is really a terrible design and in order to make do this, some bad design decisions were made like now all methods in DuckRadio were made private and we have 5 CDP classes added a friends (including DuckNet)
  webServer.on("/chatSubmit.json", HTTP_POST, [&](AsyncWebServerRequest* request) {
    int err = DUCK_ERR_NONE;

    std::vector<byte> message;
    std::string clientId = "";

    AsyncWebParameter* p = request->getParam(0);
    std::string msg = p->value().c_str();
    message.insert(message.end(), msg.begin(), msg.end());
    logdbg_ln("chatSubmit: sending message. length: %d",message.size());
    std::vector<byte> muid;

    err = duck->sendData(topics::gchat, message, BROADCAST_DUID, &muid);
    addToChatBuffer(duck->buildCdpPacket(topics::gchat, message, BROADCAST_DUID, muid));

    switch (err) {
      case DUCK_ERR_NONE:
      {
        request->send(200, "text/html", "OK.");
      }
      break;
      case DUCKLORA_ERR_MSG_TOO_LARGE:
      request->send(413, "text/html", "Message payload too big!");
      break;
      case DUCKLORA_ERR_HANDLE_PACKET:
      request->send(400, "text/html", "BadRequest");
      break;
      default:
      request->send(500, "text/html", "Oops! Unknown error.");
      break;
    }
  });

class DuckRadio {
  friend class Duck;
  friend class DuckDetect;
  friend class DuckLink;
  friend class MamaDuck;
  friend class PapaDuck;

private:
  // Everything is private to force Duck (and Duck descendants) to be the only
  // way to interact with the radio. There should only be one Duck per sketch
  // so that arbitrary pieces of code cannot interfere with the radio. Also,
  // Duck does things like recording the outgoing MUIDs so that it can wait for
  // acknowledgments to those MUIDs.

  DuckRadio();

So once chat is removed we need to take care in reverting all of these other changes in how they were in CDP 2.x
It is also critical that we add DuckRadio unit tests (which is only possible once we redefine the class to contain public methods as it was before).

Changes to DuckRadio were unrelated to chat, chat was removed in #390


adding  delay in subjectpermission chcks to allow  applying roles

What type of PR is this?
(bug/feature/cleanup/documentation/test/refactor)
test e2e
What this PR does / why we need it?
allows for 120 seconds to check if subjectpermissions successfully applies rolebindings  in a newly created namespace
Which Jira/Github issue(s) this PR fixes?
Fixes #
Special notes for your reviewer:
Pre-checks (if applicable):

[x] Tested latest changes against a cluster
[ ] Included documentation changes with PR

/lgtm
/approve


Rename formState to state and separate it from the options

I'd love to hear more about why you think this would be an improvement. Can you elaborate?

@aitboudad can this be closed?