[fix][broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector

Fixes #23315
Fixes #23321
Motivation
See #23315 and #23321. This PR fixes a bug in ConsistentHashingStickyKeyConsumerSelector where the selected consumer for a hash could change across 2 existing consumers when a consumer was removed.
In addition it fixes the problem that ConsistentHashingStickyKeyConsumerSelector.getConsumerKeyHashRanges() returned invalid information. These are fixed together since they are in the same code area and fixing getConsumerKeyHashRanges is required for testing the consistent selection of a consumer when there are hash collisions.
Modifications
In PR #8396 a solution was added to fix a problem where all hash ring points were assigned to the last arrived consumer when there were hash collisions, for example when the consumers had the same name:
https://github.com/apache/pulsar/blob/4f96146f13b136644a4eb0cf4ec36699e0431929/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ConsistentHashingStickyKeyConsumerSelector.java#L127
That solution causes problems since it causes the bug #23315. A different solution is needed to distribute the consumers evenly when there are hash collisions.
The solution in this PR is to keep track how many times a specific consumer is "selected" as a consumer in the hash ring. When a consumer is added, it will replace the previously selected consumer if the sorting order is less than the existing consumer. The sorting order takes into account the consumer priority, customer name and the selection count.
When a consumer is removed and the consumer was selected, a new consumer is selected after sorting the consumers. If the consumer wasn't previously selected, it will be only removed from the consumers for that hash ring entry.
This solution ensures that consumers aren't switching across two existing consumers and that the consumers are evenly balanced when there are hash collision, for example when the consumers have the same name.
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

I renamed this to "reduce" out-of-order issues since I'm not sure if the current model in this PR prevents the case that a consumer gets swapped when an unrelated consumer leaves.
I was able to get the behavior fairly consistent, but it's not going to prevent all cases.
It seems that another hash ring layer is needed for the colliding consumers. That would prevent unnecessary changes. A hash ring inside a hash ring...

Codecov Report
Attention: Patch coverage is 93.10345% with 4 lines in your changes missing coverage. Please review.

Project coverage is 74.57%. Comparing base (bbc6224) to head (f6dac16).
Report is 596 commits behind head on master.

Files with missing lines
Patch %
Lines

...ce/ConsistentHashingStickyKeyConsumerSelector.java
93.10%
1 Missing and 3 partials :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23327      +/-   ##
============================================
+ Coverage     73.57%   74.57%   +1.00%
- Complexity    32624    33940    +1316
============================================
  Files          1877     1934      +57
  Lines        139502   145015    +5513
  Branches      15299    15846     +547
============================================
+ Hits         102638   108149    +5511
+ Misses        28908    28598     -310
- Partials       7956     8268     +312

Flag
Coverage Δ

inttests
27.78% <84.48%> (+3.20%)
:arrow_up:

systests
24.56% <82.75%> (+0.23%)
:arrow_up:

unittests
73.93% <93.10%> (+1.08%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines
Coverage Δ

...ce/ConsistentHashingStickyKeyConsumerSelector.java
95.40% <93.10%> (+1.17%)
:arrow_up:

... and 607 files with indirect coverage changes

@lhotari @poorbarcode Thanks for bring up the customer's priority discussion. I think the Key_Shared subscription missed the consumer's priority support before. The expected behavior for consumer's priority is if you have a higher priority consumer under the subscription, the consumer with lower priority should not receive message until the consumer with higher priority get disconnected. I would like to suggest to take this one out of this PR. This issue is not easy to resolve in the Consumer Selector since different key will select different consumers unless we remove all the low priority consumers out of the Consumer Selector.

@codelipenghui  Valid point. I have removed the priority handling. We can come back to that after PIP-379 has been implemented since it's easy to solve there. The current PIP-379 implementation will also handle cases where hash ranges switch from one existing consumer to another existing consumer. That will be essential in addressing the priority handling. It's also possible to later add the logic that you are describing to the model that is in this PR.

I'm thinking another simple solution to replace the NavigableMap<Integer, List<Consumer>> hashRing; to NavigableMap<Integer, HashRangeAutoSplitStickyKeyConsumerSelector> hashRing;. Since the HashRangeAutoSplitStickyKeyConsumerSelector used a fixed hash range and it can also distribute loads to multiple consumers. ConsistentHashingStickyKeyConsumerSelector + HashRangeAutoSplitStickyKeyConsumerSelector together will resolve the both load balance and hash collision issue.

@codelipenghui I was also thinking about that type of solution; however, it won't be effective. The reason for this is that each hash range in the actual hash ring is not evenly sized. Even if the embedded hash ring would perfectly distribute the duplicate consumers, it wouldn't solve the overall distibution.
That's also why the previous solution in this PR didn't work properly, even though the consumers were evenly assigned across all hash ranges when there were collisions. That's why any embedded hash ring or container solution work and I have reverted those changes from this PR and taken a different approach.
This would be a simple problem to solve with an additional key. I guess @poorbarcoded suggested using a client IP address or port. The client address would be a useful additional key, but in many k8s environments, the IP address is masked when the ingress solution doesn't sustain the original IP address. The IP address of a proxy isn't very useful.
I have refactored the approach in this PR to use a hash-based solution where the idea is that the additional key for the hash would be the arrival index of the consumer for a particular name. If a consumer leaves and reconnects immediately, it would retain the same ranges. If consumers leave and reconnect in different order, the result wouldn't be consistent from a single consumer's viewpoint. The consumer would have to have an unique name in that case.
In the current version there's no consistency for this type of case at all. consumerList.get(hash % consumerList.size()) depends on the number of connected consumers and that will vary. It's not a "breaking change" to have a way to mitigate the duplicate consumer name behavior, even when there isn't full consistency between different phases.
The distribution is properly even now with the changes (I added a test case for proving that), but there are some remaining issues to fix since a test case derived from @poorbarcode's test case isn't yet passing locally.

One detail that I noticed in ee7a83e7945ca966e7b47fde589179469f4e71b9 is that by increasing the hash ring points from 100 to 200, it will improve the even distribution across the consumers a lot.

@lhotari How about just adding a rehash method for the consumer will put to the same position in the Hash Ring? The benefits for this solution:

Less memory overhead (we don't create to create list, maps for each point)
No major changes
Easy to read and understand

As you mentioned increasing the hash ring points from 100 to 200". It should be more common way for users to get a better distribution to increase the ring points, but for the existing solution or new solution in this PR will use much more heap memory that user expected because we will create list or map for each point. If they have 100 consumers (not a very large case), we will have 20k lists or maps.

@lhotari How about just adding a rehash method for the consumer will put to the same position in the Hash Ring? The benefits for this solution:

@codelipenghui There wouldn't be any consistency in that type of solution. In the solution in this PR, the consumer will get assigned the same hash ranges if it disconnects and comes immediately back.

Less memory overhead (we don't create to create list, maps for each point)

There's not much memory overhead in the solution in this PR. I don't see how this could be a problem in this PR.

No major changes

This PR doesn't contain a "major change"

Easy to read and understand

Please be more specific of this feedback. What isn't easy to read and understand in this PR and I could try to improve it.

As you mentioned increasing the hash ring points from 100 to 200". It should be more common way for users to get a better distribution to increase the ring points, but for the existing solution or new solution in this PR will use much more heap memory that user expected because we will create list or map for each point. If they have 100 consumers (not a very large case), we will have 20k lists or maps.

This PR doesn't increase inaccuracy in any way. The inaccuracy is an existing property of the hash ring solution and not related to the hash collisions.

I have refactored the code in this PR. I hope it now appears how simple the code is and there isn't a need for alternative solutions.  Please take another look. /cc @codelipenghui  @merlimat @equanz @poorbarcode

Hash ranges are very rare with the current hash range size of 2^31-1 (Integer.MAX_VALUE).
In PIP-379, the hash range size is reduced to 2^16-1 (65535). In that case, hash range conflicts are common.
However, due to the large amount of hash ring points, there's no significant difference in distributions due to the conflicts.


Add script to run matched rekt tests (#3344)

EigenAI demo

Signed-off-by: Cali0707<EMAIL_ADDRESS>
Update test/e2e-matched-test.sh

Co-authored-by: Pierangelo Di Pilato<EMAIL_ADDRESS>
Addressed PR comments

Signed-off-by: Calum Murray<EMAIL_ADDRESS>
Signed-off-by: Cali0707<EMAIL_ADDRESS>Signed-off-by: Calum Murray<EMAIL_ADDRESS>Co-authored-by: Pierangelo Di Pilato<EMAIL_ADDRESS>

/retest

/retest

/retest

/retest

/retest

OCF Webhook is merging this PR


Improve Grafana dashboard

Hello,
I'm in a platform team and we are running a multi-cluster setup and I don't want the provide too many different UIs for the developers. Therefore, I want them to only use the Grafana dashboard.
However, currently the Grafana dashboards does not have the same quality as the purpose built UI. I'm currently missing:

More metadata about the specific SLO. Currently it is difficult to understand the SLO, especially for the latency SLO.

Would also be nice to have a duration panel for the latency SLO's.

Have you considered feature parity between the Pyrra UI and the Grafana dashboard?
I had one idea to create an API which aggregates the SLO CRDs across clusters which Grafana can make query, but not sure if it is the best approach.
UPDATE:
One step forward would be to add a prometheus label to the pyrra_objective named e.g. description. Then add the value for latency SLOs with 90.000% in 4w must be faster than 500ms or use the .spec.description value.

Hey, thanks for opening this issue.
There is definitely more improve for sure.
The biggest problem I faced was limitations with Grafana itself. I can take a look at some of these things.
Feel free to copy the current dashboard in the meantime and make the adjustments that you want.


Extend the info in the placement log to include the node type

Small suggestion:
In the function SimplePlacer::AssignAndLog, change the print and log to be:
printf("%s (%s): %s\n", node->name().c_str(),
       node->type_string().c_str(),
       node->assigned_device_name().c_str());

thanks! you could submit a PR and go through the review process.

oops - didn't see your response. sorry. thanks for doing the work @burness.  nice job.


Build failed for Execute Shell command

Hi @adrianth
I have successfully ran Jenkins job using 'Execute Windows batch command' by following your video on Opentest - Jenkins integration.
I'm trying to run the same on a Linux server(using 'Execute Shell') and fetching code from repository by Source Code Management - Git.
I'm getting below error in Console output

I have used below command.

Please let me know if I have to change my command.

My best guess for why you're seeing that error is that the template file really doesn't exist in the test repo. Make sure you have a "templates/CAPSmokeTestsSprint1.yaml" file in there. This is probably just have a capitalization error. Because Linux is case-sensitive and Windows is not, if the name of your file on disk is spelled with a slightly different capitalization, the command will still work on Windows, but will fail on Linux.


Farvardin 2,  is not recognized!

The object for the second day of Farvardin is always as the same object for the first day of Farvardin for all years. For example:
moment("1398/01/02", "jYYYY/jMM/jDD").format("jYYYY/jMM/jDD") // retruns 1389/01/01
I should mention that I am using this package on an android phone using React-native. I have not this propblem in the node server. The propblem only exists on the phone
What can I do for this propblem?

It must be a timezone problem.

Yes it is a timezone problem. When I change it to US timezone, it works fine, but when I change the timezone to Tehran, It does not work properly.

Add 12 hour to your input string, and parse that too. This is a way to have enough offset, so timezones cannot interfere.


Vulnerability caused by EIP-1884

Describe the bug
https://eips.ethereum.org/EIPS/eip-1884  increases the gas cost of certain opcodes, possibly making contracts go over the 2300 gas limit imposed by 'transfer', making them unable to receive funds via 'transfer'.
Configuration

Severity: MEDIUM

Impact
The Ether transfer function (sendETH) does not work.
Recommendation
LINE 40:
payable(_recipients[i]).transfer(_amounts[i]);

Modify the function to use call instead of transfer.
When transferring Ether using the call function, control is transferred to the recipient, so care must be taken regarding reentrancy vulnerabilities.

Exploit Scenario

Reference site:
https://gist.github.com/ritzdorf/1c6bd72955391e831f8a397d3152b4e0/
https://github.com/holiman/eip-1884-security

Demo

I checked.


Remove api/edge/v1 path from API docs since these are already the v1 docs

This was causing duplication on the path on c.rh.c.

In the future, can we change the openapi.json to be jq output type format? It's hard to review when it's all on one line

We can, but it's automatically generated and we shouldn't have to read it 🤔 Maybe a validator to see if its format is okay? Or do you think we really need to read the output and check if it's correct? If so we can format the generated output

There are a number of Check annotations in the unchanged files. Are any of these significant?

They are mostly to enforce comments on exported functions and structs and are highly recommended by Go's best practices


Xcode 8 says the files are not Swift 3.0 in version 5.2.1

The newest version is giving errors that it is not Swift 3.0 although the CHANGELOG.md claims it is Xcode 8 Swift 3.0 compatible.

Hi @cbauer10,
Thank you for your feedback !
Could you give us some details on the errors you have? And how the library is integrated in your project?
Thanks !

Same here. As noted below:

My setup is:

Xcode Version 8.0 (8A218a)
CocoaPods 1.0.1

Podfile:

platform :ios, '9.0'
use_frameworks!
target "Gigsterous" do

pod 'OHHTTPStubs'
pod 'OHHTTPStubs/Swift'

end

Console output while pod install:

Installing OHHTTPStubs (5.2.1)
Generating Pods project

Hope it helps, looking forward to using it again! 👍

You should update to CocoaPods 1.1.0 (currently rc2) to be used with Xcode 8: gem update cocoapods --pre.

@ikesyo  It did help indeed! Thanks a lot!

@ikesyo Glad to know that the fix is coming.  I just can't work with pre-release code at my company.  You don't have any insight as to when it will be released do you?

CocoaPods 1.1.0 is coming very soon. Just a couple of PRs to finish to be sure not handle every possible tricky Xcode8/Swift3 complex configurations out there.
CP 1.1.0.rc.2 is super stable enough though, no reason to not use it. CP is not in beta anymore, and we use the RC at work without any problem.
Gonna close this issue now as it's silver by latest CP, and is not an OHHTTPStubs issue itself anyway.


ci: fix changed resolved ref for dataverse test, fix recent github ratelimit PR

There was a new version of a DOI that was tested against, so "resolved_ref" transitioned from v3.0 to v4.0 it seems. The last update to this DOI was several years ago. I've made it into a regexp now though so if new releases happen, it should not break this again.

Closes #1614

Complements #1612 with a typo fix, and with that, our test suite should become reliable I think

@manics I squeezed in a refactor of a function that could lead to a silent error and a complementary fix for #1612.
This is done now I think!


InstantParam: Improved parsing capabilities

So that it's identical to the InstantRFC3339Module in the sense that
it can also:

parse strings using the Zulu indicator
parse strings with none or parts of fractional seconds
use 'uuuu' (year) instead of 'yyyy' (year-of-era) for serializing the
year

Sorry for modifying your code from yesterday @izrobin but I didn't really get your underlying reason for not making the parsing more forgiving, as in the InstantRFC3339Module. Making the two of them behave identical should be a goal, no?
Anyway, please have a look @izrobin

Yes please :)

Thanks @izrobin : I'll perform a maven release


⚠ Bofrost.lu has degraded performance

In 6d67343, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1402 ms

Resolved: Bofrost.lu performance has improved in b694120 after 6 minutes.


Added VideoFiltersMode on Set/New-CsTeamsMeetingPolicy

Docs Build status updates of commit cac8dab:
:clock10: Pending: waiting for processors (9 builds ahead of you)
:warning: Docs Build is busy, currently there are 9 builds ahead of this one, for more information you can view the Build queue graph on the Docs Portal.

Docs Build status updates of commit cac8dab:
:clock10: Full build: the average full build time is 16 min(s) 32 sec(s), this is based on the last 1000 build(s) for this repository.

Docs Build status updates of commit cac8dab:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/New-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit dc7f562:
:clock10: Full build: the average full build time is 16 min(s) 33 sec(s), this is based on the last 1000 build(s) for this repository.

Docs Build status updates of commit dc7f562:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/New-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit ce15290:
:clock10: Pending: waiting for processors (41 builds ahead of you)
:warning: Docs Build is busy, currently there are 41 builds ahead of this one, for more information you can view the Build queue graph on the Docs Portal.

Docs Build status updates of commit ce15290:
:clock10: Full build: the average full build time is 16 min(s) 36 sec(s), this is based on the last 1000 build(s) for this repository.

Docs Build status updates of commit ce15290:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/New-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 340ec2a:
:clock10: Full build: the average full build time is 16 min(s) 48 sec(s), this is based on the last 1000 build(s) for this repository.

Docs Build status updates of commit 340ec2a:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/New-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 09a17d5:
:clock10: Full build: the average full build time is 16 min(s) 49 sec(s), this is based on the last 1000 build(s) for this repository.

Docs Build status updates of commit 09a17d5:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/New-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Fetch all transactions on member transactions page

Enhancements included in this PR:

Added "getAllTransactions" function to Soonaverse Member API that will implement pagination to retrieve all transactions for a memberId from Build.5 API.
Member profile transactions page now will populate with full history of member transactions.
Member profile transactions page will also populate the "Export CSV" with all member transactions.

I've kept the legacy UX in place where desktop device mode will populate a paginated table with 50 transactions per page and mobile will implement infinite scroll to display transactions, they should now just have all transactions for the member listed.
Thank you.

@amenconi I confirm it is working, all the data is there now!
Although I think this should be improved, either in this sprint or in a future release.
My thoughts:
When you first visit the Transactions page, it says "No data" and it takes a lot of time to load (and I'm not a whale, I just have 43 pages of records).
Options to improve this could be: add a spinner gif or something indicating page is loading. Or, better yet, add a filter to this page where the user can filter by periods of time.
It can be a date range filter (date from - date to) or a drop down list with fixed time frames, like: Last 6 months, Last year, All time transactions. I think a date from - to would be best as it is more versatile to the user needs.
Also, having an improvement like that would reduce API/database hits, as the default filter range will bring fewer results, only increased manually by those who really need it.

Totally makes sense, shouldn't be too hard to implement.
I generated a couple hundred transactions on my testnet account in debug and it all loaded almost instantly but didn't think about how many transactions some power users may have racked up over the years and no reason to hit the API that many times every time they pull that page up.
Thanks.

Hm, I think it would make sense if the exported CSV file contains the exact same data that the user filtered to see in the screen.

Ok latest push contains:

Ensure DataService has had time to populate member$ that on initial page load it doesn't try to fetch transactions with an undefined memberId input.
Initial page load fetches 100 transactions by default.
Desktop view has radio select for options: 100, 200, 500, 1000, ALL.  Selecting any of these will fetch that number of transactions order by date starting with the most recent.  This selection will also apply to export to CSV function.

I explored using a date range to filter transactions but on initial glance it didn't seem that Build.5 API supported date inputs as filters to fetch a subset of transactions.  If this is a highly sought after change versus the quantity of transactions select filter currently implemented, we can fetch all transactions and filter from there to make it work but didn't want to go that route since it would be less efficient and a bit more complex to build out but can definitely swap or add that if needed.
Thank you.

Great work guys,
In my case I've over 6000+ transactions so the full list can take some time. I personally think we should remove the pagination and just do the infinite scroll on the desktop with this approach. It'll make more sense. I think it's little confusing to have the radio buttons at the top as well as paging. People might paginate to the end and they not necessary realize there might be more transactions. This can actually might happen even with infinite scroll.
I would maybe rephrase this and say something along the lines of "Select number of recent transactions to view/export:"
Just curious, there wasn't an option to expand the current pagination so when they get to the end you'll try to retrieve more. For export, it'll always do ALL. If not with pagination, I'm sure the infinite scroll supports server call to get more content right? I feel this approach would be more intuitive.
Lastly, could you try to use same loading that we use on other pages? You should be able to use ng-zorro skeleton: https://ng.ant.design/components/skeleton/en#ng-content
see example:
https://github.com/soonaverse/app/assets/130610/0d174f98-c3fb-483c-b6ce-516d54dca9a3

Thanks Adam.  Makes sense to keep to infinite scroll for both device views to me.  As you say, infinite scroll is efficient as it always loads one "page" of transactions (backend "page" of 100) which is the minimal request we can do from the API (I think) and then incrementally grabs additional pages as needed when scrolled one fetch at a time.
Current infinite scroll logic should allow scrolling until there are no more transactions left to fetch from the API.  Plus the infinite scroll logic is already there for mobile so should be easy to adapt the same for the desktop view.
With infinite scroll in place for mobile and desktop, would it make sense to keep the transaction count select in place but only apply it to the export?  I would then change the text to something more appropriate and move it next to the Export CSV button if so.
Also I did mean to call out the loading icon as a placeholder, thanks for the ng-zorro skeleton link and will make sure that is in place on my next push.

Thanks Adam. Makes sense to keep to infinite scroll for both device views to me. As you say, infinite scroll is efficient as it always loads one "page" of transactions (backend "page" of 100) which is the minimal request we can do from the API (I think) and then incrementally grabs additional pages as needed when scrolled one fetch at a time.
Current infinite scroll logic should allow scrolling until there are no more transactions left to fetch from the API. Plus the infinite scroll logic is already there for mobile so should be easy to adapt the same for the desktop view.
With infinite scroll in place for mobile and desktop, would it make sense to keep the transaction count select in place but only apply it to the export? I would then change the text to something more appropriate and move it next to the Export CSV button if so.
Also I did mean to call out the loading icon as a placeholder, thanks for the ng-zorro skeleton link and will make sure that is in place on my next push.

❤
I would simplify it for the user and just export everything. Eventually there should be an option to select date range instead. This is a filter they will care for. This should work differently though, user should hit export and options should pop-up. This way it's clear what those options are for. At the moment, it's not very clear what those options are for.


No modules named mcubes

Description
When we run the scripts python main.py --0, we get the error No module named 'mcubes'. And then we try to use pip install mcubes to install this package, but there is no matching distribution found for mcubes. We also search the package in PyPI, but we cannot find this package. So how can be install this package ?
Steps to Reproduce
Run the scripts python main.py --0, and get the error No module named 'mcubes'
Expected Behavior
Expect to install this package mcubes.
Environment
Ubuntu18.04, Cuda10.2

I find that we should use pip install pymcubes to install this packages.


catalog: deprecate EntitySystemDiagramCard

Feature Suggestion
Deprecate EntitySystemDiagramCard from @backstage/plugin-catalog.
The @backstage/plugin-catalog-graph provides a nicer graph card that works for all kinds of entities, EntityCatalogGraphCard, so we should use that instead. It's already used in the example-app, but the create-app template still uses the older card.
Possible Implementation
We think this is a good first issue and would be happy if anyone from the community wants to pick this up! ❤
A couple of things need to be done:

[ ] Mark the EntitySystemDiagramCard as @deprecated and remove our own usage of it. Provide instructions in a changeset for how to switch to the EntityCatalogGraphCard instead.
[ ] Update the create-app template and provide a changeset with instruction for how to update existing apps.
[ ] After at least two releases have gone by since the deprecation, completely remove the EntitySystemDiagramCard component. This can be done immediately with a PR that we don't merge until the desired date.

👋  Hey! Eoghan here from Roadie. Mind if I pick this up?

@eoghanmcilwaine sorry, it was already picked up in #9066 😅
There's actually the last point left though. You could prep the removal PR that we merge in two weeks.

Ah, no worries. Sure, I'll prep that removal PR now 👍

Alright, now completely removed, thank you both!


Playing an AnimationGroup changes the order of self.mobjects

Description of bug / unexpected behavior
Overlapping objects on a screen are drawn in front of one another. This is a fact of life. As an animator, this must be taken into account. In manim, this layering is, as far as I can tell, decided by the order of Scene.mobjects. Initializing an AnimationGroup doesn't change this order, but play()ing it does. This results in difficulties animating things that are supposed to be layered behind other objects.
Note the two print statements in the below code example, and their output in the terminal.
Expected behavior
I would expect that
a = AnimationGroup(animation)
self.play(a)

did not change scene.mobjects, and specifically that it was visually indiscernible from scene.play(animation).
I am not particularly against implementation details such as mobject.copy()ing and replacing the involved Mobjects as part of scene.play()ing an AnimationGroup, if that's what is going on under the hood. But it should then reorder scene.mobjects to reflect the old ordering of things.
How to reproduce the issue

Code for reproducing the problem
import manim as m

class Test(m.Scene):
  def construct(self):
    s1 = m.Square()
    s1.shift(2*m.LEFT + 1.1*m.UP)
    s1.set_fill(m.WHITE, opacity=1)
    s2 = s1.copy()
    s2.shift(2.2*m.DOWN)
    self.add(s1, s2)

    t1 = m.Square()
    t1.shift(1.1*m.UP)
    t1.set_fill(m.RED, opacity=1)
    t2 = t1.copy()
    t2.shift(2.2*m.DOWN)
    self.add(t1, t2)

    # The white squares are behind the red squares,
    # because they were added to the scene first
    self.play(
            s1.animate.shift(4*m.RIGHT),
            s2.animate.shift(4*m.RIGHT),
            run_time=3)

    # The upper white square is still behind the upper red square,
    # but now the lower white square is in front of the lower red.
    a1 = s1.animate.shift(4*m.LEFT)
    a2 = m.AnimationGroup(s2.animate.shift(4*m.LEFT))
    print(*[mob.get_color() for mob in self.mobjects])
    self.play(a1, a2, run_time=3)
    print(*[mob.get_color() for mob in self.mobjects])

Additional media files

GIF

Logs
Terminal output
Manim Community v0.15.2

[05/07/22 08:44:55] DEBUG    Hashing ...                          hashing.py:343
                    DEBUG    Hashing done in 0.012204 s.          hashing.py:355
                    DEBUG    Hash generated :                     hashing.py:358
<PHONE_NUMBER>_2346739728_856931009
                    DEBUG    List of the first few          cairo_renderer.py:84
                             animation hashes of the scene:
                             ['3163782288_2346739728_856931
                             009']
                    INFO     Animation 0 : Partial      scene_file_writer.py:504
                             movie file written in '/Us
                             ers/uname/python/test
                             /media/videos/test
                             /480p15/partial_movie_fi
                             les/Test/3163782288_234673
                             9728_856931009.mp4'
white white #fc6255 #fc6255
                    DEBUG    Hashing ...                          hashing.py:343
                    DEBUG    Hashing done in 0.011929 s.          hashing.py:355
                    DEBUG    Hash generated :                     hashing.py:358
<PHONE_NUMBER>_3981890268_883942231
                    DEBUG    List of the first few          cairo_renderer.py:84
                             animation hashes of the scene:
                             ['3163782288_2346739728_856931
                             009', '2201830969_3981890268_8
                             83942231']
[05/07/22 08:44:56] INFO     Animation 1 : Partial      scene_file_writer.py:504
                             movie file written in '/Us
                             ers/uname/python/test
                             /media/videos/test
                             /480p15/partial_movie_fi
                             les/Test/2201830969_398189
                             0268_883942231.mp4'
white #fc6255 #fc6255 white
                    INFO     Combining to Movie file.   scene_file_writer.py:597
                    DEBUG    Partial movie files to     scene_file_writer.py:538
                             combine (2 files): ['/User
                             s/uname/python/test
                             /media/videos/test
                             /480p15/partial_movie_file
                             s/Test/3163782288_23467397
                             28_856931009.mp4', '/Users
                             /uname/python/test
                             /media/videos/test/
                             480p15/partial_movie_files
                             /Test/2201830969_398189026
                             8_883942231.mp4']
                    INFO                                scene_file_writer.py:718
                             File ready at '/Users/uname
                             /python/test/me
                             dia/videos/test/480p1
                             5/Test_ManimCE_v0.15.2.gif
                             '

                    INFO     Rendered Test                          scene.py:240
                             Played 2 animations
                    INFO     Previewed File at: '/Users/test     file_ops.py:213
                             /python/test/media/videos/test
                             /480p15/Test_ManimCE_v0.15.2
                             .gif'

System specifications
System Details

OS (with version, e.g., Windows 10 v2004 or macOS 10.15 (Catalina)): macOS 12.3.1
RAM: 8GB
Python version (python/py/python3 --version): 3.10.0
Installed modules (provide output from pip list):

Package                   Version
------------------------- -----------
backports.cached-property 1.0.1
certifi                   2021.10.8
charset-normalizer        2.0.12
click                     8.0.4
click-default-group       1.2.2
cloup                     0.13.0
colour                    0.1.5
commonmark                0.9.1
Cython                    0.29.28
decorator                 5.1.1
glcontext                 2.3.5
idna                      3.3
isosurfaces               0.1.0
manim                     0.15.2
ManimPango                0.4.0.post2
mapbox-earcut             0.12.11
moderngl                  5.6.4
moderngl-window           2.4.1
multipledispatch          0.6.0
networkx                  2.7.1
numpy                     1.22.2
Pillow                    9.0.1
pip                       22.0.4
pycairo                   1.21.0
pydub                     0.25.1
pyglet                    1.5.23
Pygments                  2.11.2
pyobjc-core               8.4.1
pyobjc-framework-Cocoa    8.4.1
pyrr                      0.10.3
requests                  2.27.1
rich                      12.3.0
scipy                     1.8.0
screeninfo                0.8
setuptools                57.4.0
six                       1.16.0
skia-pathops              0.7.2
srt                       3.5.2
tqdm                      4.63.0
urllib3                   1.26.9
watchdog                  2.1.6

FFMPEG
Output of ffmpeg -version:
ffmpeg version 5.0.1 Copyright (c) 2000-2022 the FFmpeg developers
built with Apple clang version 13.0.0 (clang-13<IP_ADDRESS>)
configuration: --prefix=/usr/local/Cellar/ffmpeg/5.0.1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox
libavutil      57. 17.100 / 57. 17.100
libavcodec     59. 18.100 / 59. 18.100
libavformat    59. 16.100 / 59. 16.100
libavdevice    59.  4.100 / 59.  4.100
libavfilter     8. 24.100 /  8. 24.100
libswscale      6.  4.100 /  6.  4.100
libswresample   4.  3.100 /  4.  3.100
libpostproc    56.  3.100 / 56.  3.100

AnimationGroup should be seen rather as an animation on a group of mobjects rather than a group of animations.
Can you try passing group=s2 to your AnimationGroup a2? I haven't tried it locally, but would assume it will fix the issue as specifying an existing group (or mobject) the animation is bound to explicitly circumvents the mechanism that collects mobjects from self.mobjects and re-adds them in a group.
While it would be easy to fix the behavior when only one animation is passed, I'm not so sure about how a clean fix for the situation where multiple animations are passed could look like.

You're right, adding group=s2 is a usable workaround
GIF
![Test_ManimCE_v0 15 2](https://user-images.githubusercontent.com/92792178/167251184-93532ba4-f4c0-4259-80ab-647c5221c6d4.gif)

After working a little with the above mentioned workaround, the current state of this issue is starting to feel more natural. Maybe it is the way it's supposed to be after all. Perhaps the real issue here is a lack of documentation of this phenomenon, rather than the fact that it happens at all.


Implement Serialize for Reflect?

I'd thought of building this kind of thing in the past so thanks a lot for making it!
The use case that I had in mind was to be able to have serializable types for dumping program state without having to have the compile time and code size costs of #[derive(Serialize)]
I haven't looked closely, but do you think that this would be possible to implement?

This is a great idea! I've whipped up an initial implementation in #2, but there's still some work to do before it's ready to merge. If you have time to try it out and provide feedback, that'd be much appreciated.


UI call-graph: sample count inconsistency.

https://drive.google.com/a/flipkart.com/file/d/0B3Pn1JEAsmALMVI3TFA5SXJEZlU/view?usp=sharing
In snapshot, you can see that count 4526 split into 4 child frames with count [1327, 79, 4, 3] which is inconsistent. The first child node should have had the count of 3116.
corresponding json response:
https://gist.github.com/gauravAshok/b19cf84f0e78e4819a02bbf9ce755adf

This is fixed in 98b4a3f367ea37184840ca3c2fdde170ab486f15, Screenshot :


Update trac/wiki URLs.

this has to be done, in my view, as the old trac urls at pcmdi do not resolve.
however, there are now conflicts, so rebase please

Rebased... and with a couple more whitespace fixes.

excellent


OutOfMemoryError occurs when executed for > 8000 genome

I want to get CSBs for over 8000 genomes set.
The genomes include only E. coli and have very low phylogenetic diversity.
I excuted following command,
% java -jar /path/to/CSBFinder/CSBFinder-0.5.1-jar-with-dependencies.jar \
>       -in ${INPUT} \
>       -out TXT \
>       -q 1 \
>       -out-dir ${OUTPUT}

and got following errors.
21/10/2019 17:40:00: Parsing input genomes file
21/10/2019 17:41:05: Executing workflow
21/10/2019 17:41:05: Extracting CSBs from 8288 input sequences.
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.Integer.valueOf(Integer.java:832)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.multimapAddAll(InstanceNode.java:127)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:115)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCount(InstanceNode.java:98)
        at Model.SuffixTreePatternFinder.SuffixTrees.GeneralizedSuffixTree.computeCount(GeneralizedSuffixTree.java:428)
        at Model.SuffixTreePatternFinder.SuffixTreeAlgorithm.findPatterns(SuffixTreeAlgorithm.java:140)
        at Model.CSBFinderWorkflow.run(CSBFinderWorkflow.java:73)
        at Model.Controller.pipeline(Controller.java:168)
        at Model.Controller.run(Controller.java:65)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

What do you think is the cause of this?

How much RAM do you have on your computer?
If you have 8GB for example, you can run the program using the option
"java -Xmx8g"
to increase the RAM memory used by the program.
In addition, you could try and use the option "-alg match_points"
This is a memory efficient algorithm that should decrease the memory
requirement.
Because you are running the program on a very large datase, I would also
advice to change the quorum parameter to "-q 10" or to a larger number, as
the program could take a very long time to run. You can also improve the
runtime by using the option "-procs 0" to use all available threads on the
computer. This option should be combined with the option "-alg match_points"
On Mon, Oct 21, 2019, 19:22 y-naka<EMAIL_ADDRESS>wrote:

I want to get CSBs for over 8000 genomes set.
The genomes include only E. coli and have very low phylogenetic diversity.
I excuted following command,
% java -jar /path/to/CSBFinder/CSBFinder-0.5.1-jar-with-dependencies.jar >       -in ${INPUT} >       -out TXT >       -q 1 >       -out-dir ${OUTPUT}
and got following errors.
21/10/2019 17:40:00: Parsing input genomes file
21/10/2019 17:41:05: Executing workflow
21/10/2019 17:41:05: Extracting CSBs from 8288 input sequences.
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.Integer.valueOf(Integer.java:832)
at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.multimapAddAll(InstanceNode.java:127)
at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:115)
at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113)
at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113)
at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113)
at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCount(InstanceNode.java:98)
at Model.SuffixTreePatternFinder.SuffixTrees.GeneralizedSuffixTree.computeCount(GeneralizedSuffixTree.java:428)
at Model.SuffixTreePatternFinder.SuffixTreeAlgorithm.findPatterns(SuffixTreeAlgorithm.java:140)
at Model.CSBFinderWorkflow.run(CSBFinderWorkflow.java:73)
at Model.Controller.pipeline(Controller.java:168)
at Model.Controller.run(Controller.java:65)
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
What do you think is the cause of this?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dinasv/CSBFinder/issues/2?email_source=notifications&email_token=ACWSNFFLO4YZOGUDX6OMR4LQPV7E5A5CNFSM4JC3H272YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HTEAFAA,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACWSNFCFMZIAG7LJBM57I6LQPV7E5ANCNFSM4JC3H27Q
.

I tried the option java -Xmx10g and everything went well with us.
Thank you for nice advise.


added an initial calendar date to allow for bi-directional pagination

Added the functionality to use an initial date to which the calendar jumps at start. I have orientated on this principle:
https://github.com/EdsonBueno/infinite_scroll_pagination/issues/56#issuecomment-890392710
If no startDate is given, it takes DateTime.now(). same for initDate. If initDate is larger than endDate, it is taken endDate.
In addition I added a localization parameter and weekdays as option.
Example:
PagedVerticalCalendar(
      invisibleMonthsThreshold: 1,
      startDate: DateTime(2021, 9, 1),
      endDate: DateTime(2022, 3, 1),
      initDate: DateTime(2022, 2, 1),
      weekDays: true,
      languageCode: 'de',
      onMonthLoaded: (year, month) {
        // on month widget load
      },
      onDayPressed: (value) {
        // on day widget pressed
      },
      onPaginationCompleted: () {
        // on pagination completion
      },
    )

Alright, interesting work. I like the initDate feature to allow pagination in both directions. this would be a nice addition to the package. However tests are currently failing and i'd recommend adding a few more to make sure it works.
you're trying to push several features in one commit, I wouldn't recommend that.
weekdays is feature that won't be added. this package is a minimal framework for creating your own looks. the included example already shows how to add weekdays to the top of every month.
In the same reasoning, included localisation is not needed since you can customize any date visualizations using the builder.
tldr, remove the weekdays and localization and fix the minor issues and I'll happily get this merged. Thanks for your work!

thanks for still working on this! I haven't forgotten this pull request, I have been very busy. I'll find some time to integrate this one of these days.

thanks for still working on this! I haven't forgotten this pull request, I have been very busy. I'll find some time to integrate this one of these days.

now available on pub..dev thanks for your contribution! 🎉


Use scoped keys for Theming

Description
With this PR, theming won't be done using the theme context key anymore, but rather defaults to _FELA_THEME_ now. withTheme now also accepts an optional propName argument to rename the prop that is passed to the wrapped component. Within createComponent and connect we're using _felaTheme to avoid conflicts with other theme props.
Versioning

Major
Minor

fela-bindings

Patch

react-fela
inferno-fela
preact-fela

Checklist
Quality Assurance

You can also run yarn run check to run all 4 commands at once.

[x] The code was formatted using Prettier (yarn run format)
[x] The code has no linting errors (yarn run lint)
[x] All tests are passing (yarn run test)
[x] There are no flow-type errors (yarn run flow)

Changes
If one of the following checks doesn't make sense due to the type of PR, just check them.

[x] Tests have been added/updated
[x] Documentation has been added/updated
[x] My changes have proper flow-types

@wcastand you're welcome :P
cc @HugoGiraudel

Haha, nice. Thank you. We renamed our prop into themeName anyway. :)

That's what i call reactivity x)
Thanks !!!


[QUESTION] Protocol of adding a new model (Stella_en_<*>_v5  family) implementation with Candle

Hi,
I have a working implementation of Stella_en_<*>_v5 family of models which is one of the top ranking model in the MTEB leaderboard for reranking and retrieval.
It's basically built on top of the candle-transformers::qwen2 implementation with the language modeling head swapped for their pre-trained dense layer.
I was hoping to open a pull request with candle along with an example.
Questions:

Would it be OK?
If it is, is there a protocol to follow for candle_transformers? Something akin to How to add a model to transformers
For embedding models are there any candle standard API implementations required? My implementations just spits out the logits from the forward pass.

Looking for some guidance. Thanks in advance.
CC:
@EricLBuehler @LaurentMazare

Adding popular models such as these ones is certainly fine (as long as the license permit it, e.g. yolo-v8 directly in candle-transformers or llama 3.2 vision would be problematic).
There isn't a very formal process for this, the best is probably to start by looking at similar models and how they are implemented and do it in the same style. If you go along this way and want to formalize it in a "How to add a model" document, the contribution would be welcome.
There is no standard api provided in candle-transformers just spitting the logits seems fine as long as there is an example showing how to use them.
On best practices:

Ideally one should follow the python reference implementation if it exists and have links in the code to the different part of this reference implementation.
The code should be validated on some examples against the reference implementation and the logits should not be far apart (usually better to use f32 than bf16 for this).
It's good to have an example with a readme that links to the model card and gives some input/output samples.

Thanks a ton for the elaborate response @LaurentMazare, I'll take a stab at it after verifying the license.

Am I missing something, the smaller stella_400M docs say its based off the same model but the config lists different activations functions and swapping of the model address produces:
Error:` Tokenizer doesn't contain expected `<|endoftext|>` token

I don't think you are missing anything .. I was stumped there too. The 1.5B variant and 400M variant are not just config changes. I've not had the chance to work on the 400M variant yet for Candle transformers. Candle transformers support the 1.5B variant for now.

Ah thanks for the heads up!
If you dont mind what was your process for the 1.5B?
Did you start by converting the modeling_qwen.py then just implement the task specific queries and the tokenizer.
The 400 has a modeling.py that Iv got converted and outputting embeddings but not matching the sample and they're way off.

Yes, thats what I did .. picked up the base Qwen model Candle implementation, removed everything and kept adding one module at a time while validating the output with HF Transformers implementation.
Just noting down a few misses I had during the phase you are in that resulted in output mismatch:

The tokenizer padding - it's padding left not right the way most embedding models do, I had missed this and ended up having similar but not same outputs
Check if you can reproduce the results reported by the authors - during AutoModel I missed trust_remote_code=True - gave me a lot of headache before I figured it out.
The file you should look at is modeling.py - with trust_remote_code=True this is the file which is getting executed
float32 seemed to be important for stability for the final embedding projection layer - with bf16 I had NaNs IIRC

Some of the above may be obvious .. these are just mistakes I made.
By the way .. a PR with 400M would be welcome. Would very much love to hear about the progress you make.

The tokenizer padding_side comes from tokenization_qwen.py right but the 400M doesn't use that as far as I can tell.
AutoTokenizer has trust_remote_code=True but theres no specific .py file for the 400M.
Seems like more based on gte-large-en-v1.5

The tokenizer config actually comes from tokenizer_config and yes they are using padding: right here unlike 1.5B variant and looks like they are using BertTokenizer.
Most importantly, they define custom AutoModel and AutoConfig calls here which is referencing the modeling.py and config.py.
To implement this in candle we'll need to walk through and re-write starting with modeling.py -> class NewModel and config.py -> class NewConfig.
Checkout the linked tokenizer_config, that should give you relevant pointers.
By the way, for the stella_en_400M_v5 I don't see tokenization_qwen.py. Use this as your reference implementation instead of qwen.

Yea got which repo just wasn't clear on if there was an existing architecture reference in candle.
Ended up just starting from scratch, seems like it needs this custom rope implementation also.
Can get this output but not clear on how to hook in with the MLR layer.
For the moment just copied your mean pooling and stuck it on the end.
Similarity scores: [[0.8398, 0.2990],
[0.3282, 0.8095]]

Here's what I got, can reproduce the example and followed your example.
Still a little unclear on what's needed to submit for the model. Doesn't include any of the training related dropouts and the config is hardcoded in the model. More of inference only for these specific weights.
Here's the PR


Whisparr Support?

Is it possible to support Whisparr? It should be straightfoward as it has a similar API to sonarr/radarr/readarr

Just saw your answer to Mylar support. Looks like this might be a no go since there is no Go api for it (yet).

It supports Whisparr if you add it as a Radarr instance. One day I'll add Whisparr to the official starr API, just not in hurry since the app isn't completely ready yet.

@hongkongkiwi Can you test a build with whisparr support?


Bug: OpenAPI unexpected ImproperlyConfiguredException

Description
We're getting an unexpected ImproperlyConfiguredException in litestar/_openapi/schema_generation/schema.py", line 594, in process_schema_result.
In our app, we defined a basePlayer model using Pydantic. This model is returned on its own and also as a subdocument in other endpoints.
Schema generation fails recognizes two different schemas with the same title. However, the schemas only differ in their description. The description itself is set as a field-level annotation when Player is used as a subdocument.
This makes setting the description at the field level much less useful.
I may be missing something. Looking at the code it hashes the schemas to ensure a 1-1 relationship between titles and schemas. I don't believe this was previously a constraint and schemas were produced based on the module path. I may not understand why this is now a requirement.
I've provided an example showing this behavior for both msgspec and Pydantic.
This behavior is new to Litestar 2.0 and did not exist in Litestar 1.51
URL to code causing the issue
No response
MCVE
### Example showing the issue with Pydantic

import traceback
from litestar import Litestar, get
from litestar.params import Parameter
from pydantic import BaseModel, Field

from litestar.types.asgi_types import Scope

class Player(BaseModel):
    id: str
    name: str
    position: str
    jersey: str

class Play(BaseModel):
    id: str
    ball_carrier: Player = Field(description="Sub document representing player who carried the ball")

@get("/player", sync_to_thread=True)
def get_player_by_id(id: str = Parameter(str)) -> Player:
    return Player(id=id, name="Chris", position="RB", jersey="00")

@get("/play", sync_to_thread=True, include_in_schema=True)
def get_play_by_id(id: str = Parameter(str)) -> Play:
    return Play(id=id, ball_carrier=Player(id="asdasd", name="chris", position="rb", jersey="00"))

async def after_exception_hook(exc: Exception, scope: "Scope") -> None:
    traceback.print_exc()

app = Litestar(route_handlers=[get_player_by_id, get_play_by_id], after_exception=[after_exception_hook])

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app)

Example using msgspec
import traceback
from litestar import Litestar, get
from litestar.params import Parameter

from litestar.types.asgi_types import Scope

from msgspec import Meta, Struct
from typing import Annotated

class Player(Struct):
    id: str
    name: str
    position: str
    jersey: str

class Play(Struct):
    id: str
    ball_carrier: Annotated[Player, Meta(description="Sub document representing player who carried the ball")]

@get("/player", sync_to_thread=True)
def get_player_by_id(id: str = Parameter(str)) -> Player:
    return Player(id=id, name="Chris", position="RB", jersey="00")

@get("/play", sync_to_thread=True, include_in_schema=True)
def get_play_by_id(id: str = Parameter(str)) -> Play:
    return Play(id=id, ball_carrier=Player(id="asdasd", name="chris", position="rb", jersey="00"))

async def after_exception_hook(exc: Exception, scope: "Scope") -> None:
    traceback.print_exc()

app = Litestar(route_handlers=[get_player_by_id, get_play_by_id], after_exception=[after_exception_hook])

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app)

### Steps to reproduce

```bash
1. Create a base app
2. Define a model and set it as the return type of an endpoint
3. Define a second model, containing a field annotated with the model defined in step 2. Set a `description` on the field (i.e. Pydantic `Field(description="{desc}"` or msgspec  Annotated[Player, Meta(description="{desc}")]). Create a second endpoint returning this model
4. See error

Screenshots
No response
Logs
litestar.exceptions.http_exceptions.ImproperlyConfiguredException: 500: Two different schemas with the title Player have been defined.

first: {"properties":{"id":{"type":"string"},"name":{"type":"string"},"position":{"type":"string"},"jersey":{"type":"string"}},"type":"object","required":["id","jersey","name","position"],"title":"Player"}
second: {"properties":{"id":{"type":"string"},"name":{"type":"string"},"position":{"type":"string"},"jersey":{"type":"string"}},"type":"object","required":["id","jersey","name","position"],"title":"Player","description":"Sub document representing player who carried the ball"}

Litestar Version
Python 3.11
litestar[full] @ 2.2.1
pydantic @ 2.4.2
Platform

[ ] Linux
[X] Mac
[ ] Windows
[ ] Other (Please specify in the description above)

[!NOTE]
While we are open for sponsoring on GitHub Sponsors and
OpenCollective, we also utilize Polar.sh to engage in pledge-based sponsorship.
Check out all issues funded or available for funding on our Polar.sh Litestar dashboard

If you would like to see an issue prioritized, make a pledge towards it!
We receive the pledge once the issue is completed & verified
This, along with engagement in the community, helps us know which features are a priority to our users.

Naively hacking around.
Using the field's annotation + name instead of just title as the schema key/ref extension does seem to work. process_schema_result of litestar._openapi.schema_generation.schema seems to work.
if schema.title and schema.type in (OpenAPIType.OBJECT, OpenAPIType.ARRAY):
            # base_title = Schema.parse_obj(schema.to_schema())
            name_title = get_class_full_path(field.annotation) + field.name
            if name_title in self.schemas and hash(self.schemas[name_title]) != hash(schema):
                raise ImproperlyConfiguredException(
                    f"Two different schemas with the title {schema.title} have been defined.\n\n"
                    f"first: {encode_json(self.schemas[name_title].to_schema()).decode()}\n"
                    f"second: {encode_json(schema.to_schema()).decode()}\n\n"
                    f"To fix this issue, either rename the base classes from which these titles are derived or manually"
                    f"set a 'title' kwarg in the route handler."
                )

            self.schemas[name_title] = schema
            return Reference(ref=f"#/components/schemas/{name_title}")
        return schema

Thanks for the excellent report! Hopefully someone with more knowledge on this than me can chime in. Git blame is no help.
Do all tests pass with your change?

No, they would need to be updated to reflect the new naming scheme as they're currently hardcoded to the old one.
I'd make a PR for this but I'm afraid I'm not particularly familiar with the OpenAPI spec and I'm not quite sure if this was changed for compliance purposes?

@peterschutt My research leads me to believe my implementation would not be a violation of OpenAPI's spec. I've made an early PR pending feed back.
https://github.com/litestar-org/litestar/pull/2475

Thanks @chris-telemetry

Is there a workaround for this issue in the meantime?

Is there a workaround for this issue in the meantime?

@LonelyVikingMichael
The best workaround I've found is fudging the names of the schema using either the 'title' keyword on the Field or by creating a dummy class:
class Player(BaseModel):
    id: str
    name: str
    position: str
    jersey: str

class PlayerWithDescription(Player):
    pass

class Play(BaseModel):
    id: str
    ball_carrier: PlayerWithDescription = Field(description="Sub document representing player who carried the ball")


[GSK-2022] Fix Giskard nightly test

Description
Fix removed append method of pandas2 DataFrame
Related Issue

Type of Change

[ ] 📚 Examples / docs / tutorials / dependencies update
[x] 🔧 Bug fix (non-breaking change which fixes an issue)
[ ] 🥂 Improvement (non-breaking change which improves an existing feature)
[ ] 🚀 New feature (non-breaking change which adds functionality)
[ ] 💥 Breaking change (fix or feature that would cause existing functionality to change)
[ ] 🔐 Security fix

Checklist

[ ] I've read the CODE_OF_CONDUCT.md document.
[ ] I've read the CONTRIBUTING.md guide.
[ ] I've updated the code style using make codestyle.
[ ] I've written tests for all new methods and classes that I created.
[ ] I've written the docstring in Google format for all the methods and classes that I used.

I started an ad-hoc nighly build https://github.com/Giskard-AI/giskard/actions/runs/6706047430


yubikey-manager-qt: 1.1.4 -> 1.1.5

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/yubikey-manager-qt/versions.
meta.description for yubikey-manager-qt is: "Cross-platform application for configuring any YubiKey over all USB interfaces."
meta.homepage for yubikey-manager-qt is: "https://developers.yubico.com/yubikey-manager-qt/"
meta.changelog for yubikey-manager-qt is: ""
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 1.1.5 with grep in /nix/store/h4w1qi7lh09s0clwnrgnh7agkvh4fkxn-yubikey-manager-qt-1.1.5

directory tree listing: https://gist.github.com/f35a9c9fd845a17b6823edb2c01f3426

du listing: https://gist.github.com/f50f71e57765bec27f6411c3b9613d7e

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
0 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
yubikey-manager-qt

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/h4w1qi7lh09s0clwnrgnh7agkvh4fkxn-yubikey-manager-qt-1.1.5 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A yubikey-manager-qt https://github.com/r-ryantm/nixpkgs/archive/22dbd2cbcaf29828fd5b5758b831b5591eb449bf.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/h4w1qi7lh09s0clwnrgnh7agkvh4fkxn-yubikey-manager-qt-1.1.5
ls -la /nix/store/h4w1qi7lh09s0clwnrgnh7agkvh4fkxn-yubikey-manager-qt-1.1.5/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review 1

1 package built:

yubikey-manager-qt

Maintainer pings
cc @avdv for testing.

Tested, works on my machine (sees my yubikey 4, allows me to swap OTP slots)


About the dataset cruise.bag!

Hey, I really appreciate your excellent project.From youtube and ROScon2019, I have both viewed your demo during the presentation.And I can use webviz on my device and see the same result by play demo.bag. For further, I would like to realize the 3d rendering just like you showed in the demo :the car is a 3d model and the road is more like the real environment and so on. I notice that you used a bag named cruise.bag, so how could I get that bag to help my research? Thank you.

Hi, I saw you also posted a comment here about the bag file which I answered: https://github.com/cruise-automation/webviz/issues/421#issuecomment-626001381
For the map you can use nav_msgs/OccupancyGrid; see e.g. how that works in RViz here: http://wiki.ros.org/rviz/DisplayTypes/Map (it works similarly in Webviz).
 We currently don't have a command for building a static page, but we're tracking that here #406 and I have some suggestions there for implementing it if you want to give it a shot; shouldn't be too difficult. Alternatively you can also directly host the code from the gh-pages branch: https://github.com/cruise-automation/webviz/tree/gh-pages

Hi, I saw you also posted a comment here about the bag file which I answered: #421 (comment)
For the map you can use nav_msgs/OccupancyGrid; see e.g. how that works in RViz here: http://wiki.ros.org/rviz/DisplayTypes/Map (it works similarly in Webviz).
 We currently don't have a command for building a static page, but we're tracking that here #406 and I have some suggestions there for implementing it if you want to give it a shot; shouldn't be too difficult. Alternatively you can also directly host the code from the gh-pages branch: https://github.com/cruise-automation/webviz/tree/gh-pages

Thanks for your quick reply.I have used Rviz expertly so I know how to display the map on it and also have successfully run a local server with my own data. Above all, I think that the map is too plain to show it to others. From your video at ROScon2019, I find a fancy map  which is quite similar with the realworld, we can see the road and zebra stripes aside with the 3d model of the car.And from the previous question, you said the car was hardcoded. So may you give me some adive about how to create the map like the realworld and how to add a car model(maybe it is not supported?)
Thanks again!

Our map was constructed from lidar data by special mapping vehicles, and is currently proprietary, unfortunately. It would be cool to one day support loading map tiles from Mapbox or so though!
Adding your own car model is currently not supported, though if you want to fork Webviz yourself you could replace our car model in this directory: https://github.com/cruise-automation/webviz/tree/1c5cfa783eab39f237c59054d0dc899aee58228a/packages/webviz-core/src/panels/ThreeDimensionalViz/commands/CarModel and then publish a Pose, and then in the settings for that Pose topic enable the car model. But yeah, that's less than ideal.
It's a good feature request though to formally support different 3d models at some point!

Our map was constructed from lidar data by special mapping vehicles, and is currently proprietary, unfortunately. It would be cool to one day support loading map tiles from Mapbox or so though!
Adding your own car model is currently not supported, though if you want to fork Webviz yourself you could replace our car model in this directory: https://github.com/cruise-automation/webviz/tree/1c5cfa783eab39f237c59054d0dc899aee58228a/packages/webviz-core/src/panels/ThreeDimensionalViz/commands/CarModel and then publish a Pose, and then in the settings for that Pose topic enable the car model. But yeah, that's less than ideal.
It's a good feature request though to formally support different 3d models at some point!

Okay, get it! Thanks again.

Our map was constructed from lidar data by special mapping vehicles, and is currently proprietary, unfortunately. It would be cool to one day support loading map tiles from Mapbox or so though!
Adding your own car model is currently not supported, though if you want to fork Webviz yourself you could replace our car model in this directory: https://github.com/cruise-automation/webviz/tree/1c5cfa783eab39f237c59054d0dc899aee58228a/packages/webviz-core/src/panels/ThreeDimensionalViz/commands/CarModel and then publish a Pose, and then in the settings for that Pose topic enable the car model. But yeah, that's less than ideal.
It's a good feature request though to formally support different 3d models at some point!

Hey! I just watch the video on youtube https://www.youtube.com/watch?v=_3_Bb-dlq0Qagian. And when you show the demo at 17:50 in the video, I see several options on the  top left corner of the page  just like the topic to be chosen in Rviz ,including Map、Car、Localization ……
So, I wonder
1)Is the Map rendered based on the  occupancied grid map with WORLDVIEW? I haven't figured out the relationship between webviz and worldview yet.I just want to create a simliar map of the area of a park, not too large, and I know how to create the map with lidar data but the map is not so fancy as yours.
2)the option Caris binding with the pose you mentioned in the previous answer? I try to make clear your reply ' then in the settings for that Pose topic enable the car model'How to enable the model?
3)I am really interested in those options on the  top left corner of the page(shown in picture uploaded),for example when you choose Lidar,the other objects are displayed separately, so the Lidar would be the processed information based on raw data from lidar?

@janpaul123

Ah in the open source version we don't have those categorizations currently, but we render topics in the same way there. So just enable the topics that you want to see. In the future we'd like to make it so you can make your own categorizations like "Lidar" and such.
And in our internal version we render the map in a different way (using a tile server), but you should be able to get a similar effect using the occupancy grid.
When you have a geometry_msgs/PoseStamped message you can turn on our car model like this:

Btw I set up a Slack workspace to make it easier for us to chat; join us using the link here: #461


Models/servers are not persisted between launches

I haven't been able to get server configs to persist between launches of PicoMLXServer, and have had to recreate each server each time. I did briefly try to look through the code, and saw UserDefaults was used for preferences, but not (at least not obviously) for servers, and confirmed the contents of ~/Library/Preferences/com.starlingprotocol.PicoMLXServer.plist did not contain model/server info. Nothing stored in ~/Library/Application Support, either.

I'm reworking server management in the next version. Will add persistency.


[Suggetion] Add Per Block Replace Option

Please Add Per BLock Replace Option like i watn diamond ore to be replaced with bedrock and wood with air like this

Actually you can
https://github.com/AIPTU/BlockReplacer/blob/ed83ea6b87451b437223ddf84c1c9cc3ba086ea0/resources/config.yml#L18

https://github.com/AIPTU/BlockReplacer/blob/ed83ea6b87451b437223ddf84c1c9cc3ba086ea0/resources/config.yml#L21

ok


Generate styles for the sessions#new view template

The new sessions generator has an html template that needs to be styled

@yshmarov Great idea, are you interested in submitting a PR?

The generator could looks like this:
# app/views/sessions/new.html.erb

Sign in

  <% if alert.present? %>

<%= alert %>

  <% end %>

  <%= form_with url: session_url, class: "contents" do |form| %>

      <%= form.email_field :email_address, required: true, autofocus: true, autocomplete: "username", placeholder: "Enter your email address", value: params[:email_address], class: "block shadow rounded-md border border-gray-400 outline-none px-3 py-2 mt-2 w-full" %>

      <%= form.password_field :password, required: true, autocomplete: "current-password", placeholder: "Enter your password", maxlength: 72, class: "block shadow rounded-md border border-gray-400 outline-none px-3 py-2 mt-2 w-full" %>

      <%= form.submit "Sign in", class: "rounded-lg py-3 px-5 bg-blue-600 text-white inline-block font-medium cursor-pointer" %>

  <% end %>

Result:

For reference, posts#index with the scaffold generator looks like this:

OK, I dug into this today. A few things need to happen in order for this gem to be able to override the new session view template ...
The naive change to the upstream generator to parameterize the template engine might look like:
diff --git a/railties/lib/rails/generators/rails/sessions/sessions_generator.rb b/railties/lib/rails/generators/rails/sessions/sessions_generator.rb
index 21ed4204..29520a06 100644
--- a/railties/lib/rails/generators/rails/sessions/sessions_generator.rb
+++ b/railties/lib/rails/generators/rails/sessions/sessions_generator.rb
@@ -3,6 +3,10 @@
 module Rails
   module Generators
     class SessionsGenerator < Base # :nodoc:
+      hook_for :template_engine, as: :sessions do |template_engine|
+        invoke template_engine
+      end
+
       def create_session_files
         template "models/session.rb", File.join("app/models/session.rb")
         template "models/user.rb", File.join("app/models/user.rb")

This would allow this gem to implement Tailwindcss::Generators::SessionsGenerator in lib/generators/tailwindcss/sessions/sessions_generator.rb.
However, when we do this, the generator will look locally for all of the model and controller templates as well, which we should not be copying into this gem just to override the view:
https://github.com/rails/rails/blob/c01ee683aa700b109f704a5b6611107fe2dc5476/railties/lib/rails/generators/rails/sessions/sessions_generator.rb#L7-L12
I think this requires breaking up the sessions generator into multiple generators, similar to how the scaffold generation is broken up into Rails::Generators::ScaffoldGenerator and Erb::Generators::ScaffoldGenerator, which is a conversation we should have upstream.
Given the sessions generator is intended to be bare-bones, it's not clear to me that it's intended to be overridden by gems like this; and if it is, we may need to wait for it to firm up a bit before making a change like I'm describing. It's possible that @dhh has some opinions that might guide next steps.

Yeah, I'm a bit of two minds about this. The generator as it exists in the rails repo is intended to be unstyled. But I am actually starting to think that some degree of styling for these built-in generators could be nice. So maybe adding the hooks like proposed above would be fine.
Could also just let it sit and cook for a little longer. And see if we can't come up with an approach to default styling that feels proportionate.

@dhh The necessary changes are pretty small, PR is at https://github.com/rails/rails/pull/52442

Draft pr is at #384

Just a note that the upstream Rails PR was merged, so I'm moving forward with #384.


Dataset not found or corrupted.

Hello, i installed GOT-10k dataset and the toolkit and save it to the my drive. I run : pip install -r requirements.txt and move got10k folder to  my current direcory. But i tried to loop over GOT-10k dataset, i get an error like : Dataset not found or corrupted. I checked my root_dir (data/GOT-10k) and it is correct. What can cause this error ?
The script i try to run :
from PIL import Image
from got10k.datasets import GOT10k
from got10k.utils.viz import show_frame
dataset = GOT10k(root_dir='data/GOT-10k', subset='train')
img_file, anno = dataset[10]
for s, (img_files, anno) in enumerate(dataset):
seq_name = dataset.seq_names[s]
print('Sequence:', seq_name)
for f, img_file in enumerate(img_files):
    image = Image.open(img_file)
    show_frame(image, anno[f, :])

Python version at the Colab i use is 3.6.5
I would be grateful if you help me about that.

Hi, I run into the same problem. Have you sove it yet?


DefaultJiraClient - No result was available from Jira unexpectedly - JSONObject["type"] not found

Hi, I am getting an error when trying to run the Jira collector. Below is the error and below that are my properties used. I looked at issue #1369 however, that one seems to have been closed due to inactivity from the reporter and not because of a resolution.
Jira v7.2.7
2017-11-08 10:34:59,998 INFO  c.c.d.collector.CollectorTask - Running Collector: Jira
2017-11-08 10:35:00,007 INFO  c.c.d.collector.CollectorTask - -----------------------------------
2017-11-08 10:35:00,007 INFO  c.c.d.collector.CollectorTask - https://my.jira.com/
2017-11-08 10:35:00,007 INFO  c.c.d.collector.CollectorTask - -----------------------------------
2017-11-08 10:35:00,009 DEBUG c.c.d.client.team.TeamDataClientImpl - Constructing data collection for the feature widget, team-level data...
2017-11-08 10:35:00,024 INFO  c.c.d.collector.CollectorTask - Team Data               0        0s
2017-11-08 10:35:00,025 DEBUG c.c.d.c.p.ProjectDataClientImpl - Constructing data collection for the feature widget, project-level data...
2017-11-08 10:35:02,673 DEBUG c.c.d.c.p.ProjectDataClientImpl - Size of paged Jira response: 947
2017-11-08 10:35:04,434 INFO  c.c.d.collector.CollectorTask - Project Data          947        4s
2017-11-08 10:35:04,437 DEBUG c.c.d.c.story.StoryDataClientImpl - Constructing data collection for the feature widget, story-level data...
2017-11-08 10:35:05,430 DEBUG c.c.d.c.story.StoryDataClientImpl - Obtaining story information starting at index 0...
2017-11-08 10:35:13,707 ERROR c.c.d.client.DefaultJiraClient - No result was available from Jira unexpectedly - defaulting to blank response. The reason for this fault is the following:RestClientException{statusCode=Optional.absent(), errorCollections=[]}
2017-11-08 10:35:13,723 DEBUG c.c.d.client.DefaultJiraClient - Exception
com.atlassian.jira.rest.client.api.RestClientException: org.codehaus.jettison.json.JSONException: JSONObject["type"] not found.
	at com.atlassian.jira.rest.client.internal.async.DelegatingPromise.claim(DelegatingPromise.java:47) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.capitalone.dashboard.client.DefaultJiraClient.getIssues(DefaultJiraClient.java:101) ~[jira-feature-collector.jar!/:2.0.5-SNAPSHOT]
	at com.capitalone.dashboard.client.story.StoryDataClientImpl.updateStoryInformation(StoryDataClientImpl.java:162) [jira-feature-collector.jar!/:2.0.5-SNAPSHOT]
	at com.capitalone.dashboard.collector.FeatureCollectorTask.collect(FeatureCollectorTask.java:119) [jira-feature-collector.jar!/:2.0.5-SNAPSHOT]
	at com.capitalone.dashboard.collector.FeatureCollectorTask.collect(FeatureCollectorTask.java:28) [jira-feature-collector.jar!/:2.0.5-SNAPSHOT]
	at com.capitalone.dashboard.collector.CollectorTask.run(CollectorTask.java:63) [core-2.0.5-SNAPSHOT.jar!/:2.0.5-SNAPSHOT]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: com.atlassian.jira.rest.client.api.RestClientException: org.codehaus.jettison.json.JSONException: JSONObject["type"] not found.
	at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient$3.apply(AbstractAsynchronousRestClient.java:188) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient$3.apply(AbstractAsynchronousRestClient.java:182) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.httpclient.api.ResponsePromiseMapFunction.apply(ResponsePromiseMapFunction.java:62) ~[atlassian-httpclient-api-0.13.2.jar!/:na]
	at com.atlassian.httpclient.api.ResponsePromiseMapFunction.apply(ResponsePromiseMapFunction.java:12) ~[atlassian-httpclient-api-0.13.2.jar!/:na]
	at com.atlassian.util.concurrent.Promises$Of$3.apply(Promises.java:285) ~[atlassian-util-concurrent-2.4.0-M9.jar!/:na]
	at com.atlassian.util.concurrent.Promises$2.onSuccess(Promises.java:162) ~[atlassian-util-concurrent-2.4.0-M9.jar!/:na]
	at com.google.common.util.concurrent.Futures$4.run(Futures.java:1154) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:150) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:135) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53) ~[guava-14.0.jar!/:na]
	at com.atlassian.util.concurrent.Promises$Of$3.apply(Promises.java:285) ~[atlassian-util-concurrent-2.4.0-M9.jar!/:na]
	at com.atlassian.util.concurrent.Promises$2.onSuccess(Promises.java:162) ~[atlassian-util-concurrent-2.4.0-M9.jar!/:na]
	at com.google.common.util.concurrent.Futures$4.run(Futures.java:1154) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:150) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:135) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) ~[guava-14.0.jar!/:na]
	at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53) ~[guava-14.0.jar!/:na]
	at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient$1$1.run(SettableFuturePromiseHttpPromiseAsyncClient.java:46) ~[atlassian-httpclient-apache-httpcomponents-0.13.2.jar!/:na]
	at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient$ThreadLocalDelegateRunnable$1.run(SettableFuturePromiseHttpPromiseAsyncClient.java:197) ~[atlassian-httpclient-apache-httpcomponents-0.13.2.jar!/:na]
	at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.runInContext(SettableFuturePromiseHttpPromiseAsyncClient.java:90) ~[atlassian-httpclient-apache-httpcomponents-0.13.2.jar!/:na]
	at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient$ThreadLocalDelegateRunnable.run(SettableFuturePromiseHttpPromiseAsyncClient.java:192) ~[atlassian-httpclient-apache-httpcomponents-0.13.2.jar!/:na]
	... 3 common frames omitted
Caused by: org.codehaus.jettison.json.JSONException: JSONObject["type"] not found.
	at org.codehaus.jettison.json.JSONObject.get(JSONObject.java:360) ~[jettison-1.1.jar!/:1.1]
	at org.codehaus.jettison.json.JSONObject.getString(JSONObject.java:487) ~[jettison-1.1.jar!/:1.1]
	at com.atlassian.jira.rest.client.internal.json.IssueJsonParser.parseSchema(IssueJsonParser.java:344) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.jira.rest.client.internal.json.IssueJsonParser.parseFields(IssueJsonParser.java:309) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.jira.rest.client.internal.json.IssueJsonParser.parse(IssueJsonParser.java:223) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.jira.rest.client.internal.json.IssueJsonParser.parse(IssueJsonParser.java:88) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.jira.rest.client.internal.json.GenericJsonArrayParser.parse(GenericJsonArrayParser.java:39) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.jira.rest.client.internal.json.SearchResultJsonParser.parse(SearchResultJsonParser.java:40) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.jira.rest.client.internal.json.SearchResultJsonParser.parse(SearchResultJsonParser.java:27) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient$1.handle(AbstractAsynchronousRestClient.java:145) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient$3.apply(AbstractAsynchronousRestClient.java:186) ~[jira-rest-java-client-core-3.0.0.jar!/:na]
	... 26 common frames omitted
2017-11-08 10:35:13,724 DEBUG c.c.d.c.story.StoryDataClientImpl - Story information query took 8293 ms
2017-11-08 10:35:13,724 INFO  c.c.d.c.story.StoryDataClientImpl - Loop i 0 pageSize 0
2017-11-08 10:35:13,724 INFO  c.c.d.collector.CollectorTask - Story Data              0        9s
2017-11-08 10:35:13,724 INFO  c.c.d.collector.CollectorTask - Finished                        13s

I do have "Task" and "Story" as my feature.jiraIssueTypeNames. I tried just "Story" however that did not work as well (same stack trace). Moreover, I want to be able to find issues categorized as "Task", would they still appear even if I use "Story"?
I have already confirmed that the customerfield_xxxxx values are correct. Any help would be much appreciated, thank you.
# Database Name
dbname=dashboarddb

# Database HostName - default is localhost
dbhost=localhost

# Database Port - default is 27017
dbport=27017

# MongoDB replicaset
dbreplicaset=[false if you are not using MongoDB replicaset]
dbhostport=[host1:port1,host2:port2,host3:port3]

# Database Username - default is blank
dbusername=dashboarduser

# Database Password - default is blank
dbpassword=dbpassword

# Logging File location
logging.file=./logs/jira.log

# PageSize - Expand contract this value depending on Jira implementation's
# default server timeout setting (You will likely receive a SocketTimeoutException)
feature.pageSize=100

# Delta change date that modulates the collector item task
# Occasionally, these values should be modified if database size is a concern
feature.deltaStartDate=2017-06-01T00:00:00.000000
feature.masterStartDate=2017-06-01T00:00:00.000000
feature.deltaCollectorItemStartDate=2017-06-01T00:00:00.000000

# Chron schedule: S M D M Y [Day of the Week]
feature.cron=0 * * * * *

# ST Query File Details - Required, but DO NOT MODIFY
feature.queryFolder=jiraapi-queries
feature.storyQuery=story
feature.epicQuery=epic

# JIRA CONNECTION DETAILS:
# Enterprise Proxy - ONLY INCLUDE IF YOU HAVE A PROXY
feature.jiraProxyUrl=
feature.jiraProxyPort=
feature.jiraBaseUrl=https://my.jira.com/
feature.jiraQueryEndpoint=rest/api/2/
# For basic authentication, requires username:password as string in base64
# This command will make this for you:  echo -n username:password | base64
feature.jiraCredentials=[my encoded credentials]
# OAuth is not fully implemented; please blank-out the OAuth values:
feature.jiraOauthAuthtoken=
feature.jiraOauthRefreshtoken=
feature.jiraOauthRedirecturi=
feature.jiraOauthExpiretime=

#############################################################################
# In Jira, general IssueType IDs are associated to various 'issue'
# attributes. However, there is one attribute which this collector's
# queries rely on that change between different instantiations of Jira.
# Please provide a string name reference to your instance's IssueType for
# the lowest level of Issues (for example, 'user story') specific to your Jira
# instance.  Note:  You can retrieve your instance's IssueType Name
# listings via the following URI:  https://[your-jira-domain-name]/rest/api/2/issuetype/
# Multiple comma-separated values can be specified.
#############################################################################
feature.jiraIssueTypeNames=Story,Task

#############################################################################
# In Jira, your instance will have its own custom field created for 'sprint' or 'timebox' details,
# which includes a list of information.  This field allows you to specify that data field for your
# instance of Jira. Note: You can retrieve your instance's sprint data field name
# via the following URI, and look for a package name com.atlassian.greenhopper.service.sprint.Sprint;
# your custom field name describes the values in this field:
# https://[your-jira-domain-name]/rest/api/2/issue/[some-issue-name]
#############################################################################
feature.jiraSprintDataFieldName=customfield_XXXXX

#############################################################################
# In Jira, your instance will have its own custom field created for 'super story' or 'epic' back-end ID,
# which includes a list of information.  This field allows you to specify that data field for your instance
# of Jira.  Note:  You can retrieve your instance's epic ID field name via the following URI where your
# queried user story issue has a super issue (for example, epic) tied to it; your custom field name describes the
# epic value you expect to see, and is the only field that does this for a given issue:
# https://[your-jira-domain-name]/rest/api/2/issue/[some-issue-name]
#############################################################################
feature.jiraEpicIdFieldName=customfield_XXXXX

#############################################################################
# In Jira, your instance will have its own custom field created for 'story points'
# This field allows you to specify that data field for your instance
# of Jira.  Note:  You can retrieve your instance's storypoints ID field name via the following URI where your
# queried user story issue has story points set on it; your custom field name describes the
# story points value you expect to see:
# https://[your-jira-domain-name]/rest/api/2/issue/[some-issue-name]
#############################################################################
feature.jiraStoryPointsFieldName=customfield_XXXXX

#############################################################################
# In Jira, your instance will have its own custom field created for 'team'
# This field allows you to specify that data field for your instance
# of Jira.  Note:  You can retrieve your instance's team ID field name via the following URI where your
# queried user story issue has team set on it; your custom field name describes the
# team value you expect to see:
# https://[your-jira-domain-name]/rest/api/2/issue/[some-issue-name]
#############################################################################
feature.jiraTeamFieldName=

# Set this to true if you use boards as team
feature.jiraBoardAsTeam=false

logging.level.com.capitalone.dashboard.collector=DEBUG
logging.level.com.atlassian.jira=INFO
#logging.level.org.apache.http.impl.nio.client.DefaultHttpAsyncClient=DEBUG
logging.level.com.capitalone.dashboard=DEBUG
#logging.level.org.springframework=DEBUG

The exception appears to be thrown from
com.atlassian.jira.rest.client.internal.json.IssueJsonParser.parseSchema(IssueJsonParser.java:344)
and that line looks like this:
(line 343) JSONObject fieldDefinition = json.getJSONObject(fieldId);
(line 344) res.put(fieldId, fieldDefinition.getString("type"));
The root json object contains in it this json {"custom":"com.onresolve.jira.groovy.groovyrunner:scripted-field","customId":12813}. The root json object itself is pretty large and there are many inner objects that contain the "type" key, however this one does not and not sure what this one represents.
Any help would be much appreciated, thanks.

I am not able to recreate this issue but if you are not behind a proxy then could you comment out
feature.jiraProxyUrl= feature.jiraProxyPort=

Hi @Sbrenthughes,
Regarding customfields, I do have actual numbers, I only replaced with "xxxxx" to post here.
I commented out both proxy fields, however the issue still remains. While trying to recreate, are you working with a json that does not have the "type" field, as mentioned in my second comment?

Hi @e-macias:
I see that @Sbrenthughes has already responded.
If I may add: From what I understand, this issue that you are seeing is happening when the collector is trying to fetch Story Data.
The main properties needed to fetch Story data are:
feature.jiraIssueTypeNames=Story,
feature.storyQuery=story,
feature.pageSize=100
In your case, the story information comes the api call such as this below:
https://my.jira.com/rest/api/latest/search?jql=updatedDate>%3D'2016-03-01+00:00'+AND+issueType+IN+('Story')+ORDER+BY+updated+ASC&expand=schema,names&fields=*all,-comment,-watches,-worklog,-votes,-reporter,-creator,-attachment&maxResults=100&startAt=0
On the response JSON that you get back, there is sub object called "schema".
And it essentially defines the "type" for each custom field found on the issues.
And it looks like in your case that "type" information may be missing and hence causing the error. Thanks !

@subodhbattina pretty much yes. I am not entirely sure about the "sub object called 'schema'" part of your comment. So I cannot say for sure that you are correct there, but you might be.

So inside the JSON response, there is another section for "schema" which has type information, basically a type field in each of those entries inside the "schema" section. So basically that type field might be missing on the response JSON. Hope this helps. Let me know if I can go ahead close this issue. Thanks !

@subodhbattina I think I may have found the "schema" object you were referring to:
"schema":{"parent":{"type":"array","items":"issuelinks","system":"parent"}, and it goes on for a while..
I reached out to Atlassian regarding the issue being thrown by their jira-rest-java-client, and they suggested I reach out to Adaptavist (the creators of the jira-groovy plugin). I am still waiting to hear back from them, and I was going to close this ticket afterwards, however, as this does not seem to be an issue with Hygieia, we may close this issue. I will continue to post if a solution is found for reference. Thank you for your help thus far.

@e-macias Sure, you are very welcome. Please do keep us updated on your findings. I'll go ahead and close this ticket. Thanks !

Just FYI I think this may be related to this bug: https://bitbucket.org/atlassian/jira-rest-java-client/pull-requests/78


Bump version to 0.0.7

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I have canceled pending stalled appveyor builds and triggered a new one. I hope it will help.

All build on merge commit have failed, since pypi is currently down. I will manually restart the jobs when the server becomes available again.


Fix all project warnings

Remove all compiler warnings from all targets.

Probably add something to detect warnings on CI


retrieve shasum

Cut v2.0.0
Made it a major release because I don't think it'll work with older versions of the lambda.