Transfer code from CourtListener

This PR copies the relevant code from CourtListener's citations module with minimal substantive changes. Cf. https://github.com/freelawproject/courtlistener/issues/1521.
Key points:

References to the Django DB have been replaced with calls to the courts_db package (cf. https://github.com/freelawproject/courts-db/pull/12)
The as_html() methods have been removed from the Citation objects as they serve no purpose outside of CL
To make find_citations.py easier to understand, I moved some of the helper functions to a new file (helpers.py), but no logic was changed
Typing has been added to all functions
Rather than importing get_visible_text() from juriscraper (https://github.com/freelawproject/juriscraper/blob/master/juriscraper/lib/html_utils.py#L163), I just copied it into utils.py. I thought this made sense because it's the only function that we need from juriscraper, and so it seemed silly to add that as a dependency just for that.
The relevant tests were also copied from CL, and they all pass
What is the purpose of this function: https://github.com/freelawproject/courtlistener/blob/master/cl/citations/reporter_tokenizer.py#L19? It is never actually invoked in CL, but I copied it over here anyway. Should we be calling that somewhere?

More changes are probably necessary to prepare this for publishing, but I think I've transferred the bulk of the code from CL in a working fashion.

This is still a WIP, as apparently there are several typing errors I need to fix. (Has Python become Java???) I will push some additional, squashed commits when I fix that.

This is still a WIP, as apparently there are several typing errors I need to fix. (Has Python become Java???) I will push some additional, squashed commits when I fix that.

All your points sound good. Your one q:

What is the purpose of this function: https://github.com/freelawproject/courtlistener/blob/master/cl/citations/reporter_tokenizer.py#L19? It is never actually invoked in CL, but I copied it over here anyway. Should we be calling that somewhere?

I don't know, which means it can be deleted. I checked the git blame for it, and it's all me, and it looks like my style, but....I guess whatever it was supposed to be never happened. We can only hope my git skills have improved.
Sorry about the type checking. We started doing it during the move to py3, and I've found it to be extremely helpful. The checker catches bugs regularly (especially around bytes vs strs), and it helps me to actually understand other people's code, since I'm often reviewing something I've never seen before. I've also noticed that making myself and others do the typing forces people to do simpler functions. If the typing is so horrible that you can't stand it, well, you might simplify your function. I once swore off java. I still think I'd never touch it. But...the typing I enjoy so far.

All your points sound good. Your one q:

What is the purpose of this function: https://github.com/freelawproject/courtlistener/blob/master/cl/citations/reporter_tokenizer.py#L19? It is never actually invoked in CL, but I copied it over here anyway. Should we be calling that somewhere?

I don't know, which means it can be deleted. I checked the git blame for it, and it's all me, and it looks like my style, but....I guess whatever it was supposed to be never happened. We can only hope my git skills have improved.
Sorry about the type checking. We started doing it during the move to py3, and I've found it to be extremely helpful. The checker catches bugs regularly (especially around bytes vs strs), and it helps me to actually understand other people's code, since I'm often reviewing something I've never seen before. I've also noticed that making myself and others do the typing forces people to do simpler functions. If the typing is so horrible that you can't stand it, well, you might simplify your function. I once swore off java. I still think I'd never touch it. But...the typing I enjoy so far.

I actually first learned to code in Java, worshiping at the altar of static typing, polymorphism, etc. etc., and I thought I had left all that behind...
Do we even enforce the type checking in CL yet? When I run mypy there I get tons of errors. I have fixed them all in this repo now though.
Currently, the problem is that the pylint reporting task is not passing. Can you advise as to what am I supposed to do to fix that? When I run it locally, it generates a file with some suggested refactors, but surely I don't have to implement all of its suggestions?

I actually first learned to code in Java, worshiping at the altar of static typing, polymorphism, etc. etc., and I thought I had left all that behind...
Do we even enforce the type checking in CL yet? When I run mypy there I get tons of errors. I have fixed them all in this repo now though.
Currently, the problem is that the pylint reporting task is not passing. Can you advise as to what am I supposed to do to fix that? When I run it locally, it generates a file with some suggested refactors, but surely I don't have to implement all of its suggestions?

it's really annoying lol.

Well, we really don't want to annoy our volunteers unless it gives us a solid improvement in code quality. I think typing does that, but if there are better ways to handle this, I'm happy to think about it. You're the first to use our new project template since we added typing. It doesn't surprise me that there are some issues. There must be a better way to do this. Maybe @drewsilcock has thoughts.

Do we even enforce the type checking in CL yet? When I run mypy there I get tons of errors. I have fixed them all in this repo now though.

No, not yet. I think we should turn it on though in some form b/c the current state of things is that we're adding typing to various places, but mypy isn't doing anything. It'd be better to check them as we add them, I suspect.

Currently, the problem is that the pylint reporting task is not passing. Can you advise as to what am I supposed to do to fix that? When I run it locally, it generates a file with some suggested refactors, but surely I don't have to implement all of its suggestions?

I don't know what's normal here. I poked through and fixed a bunch of these, so things are better, but there are some that look really difficult to fix without introducing bugs. Some of the ones I fixed were boring, others made the code better. Maybe the easiest way to deal with the remaining ones is to ignore them with #pylint comments in the code. That way we can keep the linting and express our aspiration to someday fix the old code (and it doesn't have to be fixed now).
What do you think?

it's really annoying lol.

Well, we really don't want to annoy our volunteers unless it gives us a solid improvement in code quality. I think typing does that, but if there are better ways to handle this, I'm happy to think about it. You're the first to use our new project template since we added typing. It doesn't surprise me that there are some issues. There must be a better way to do this. Maybe @drewsilcock has thoughts.

Do we even enforce the type checking in CL yet? When I run mypy there I get tons of errors. I have fixed them all in this repo now though.

No, not yet. I think we should turn it on though in some form b/c the current state of things is that we're adding typing to various places, but mypy isn't doing anything. It'd be better to check them as we add them, I suspect.

Currently, the problem is that the pylint reporting task is not passing. Can you advise as to what am I supposed to do to fix that? When I run it locally, it generates a file with some suggested refactors, but surely I don't have to implement all of its suggestions?

I don't know what's normal here. I poked through and fixed a bunch of these, so things are better, but there are some that look really difficult to fix without introducing bugs. Some of the ones I fixed were boring, others made the code better. Maybe the easiest way to deal with the remaining ones is to ignore them with #pylint comments in the code. That way we can keep the linting and express our aspiration to someday fix the old code (and it doesn't have to be fixed now).
What do you think?

Oh, one other thought: It'd be nice to have some check that says, "Did pylint get worse? Did mypy get worse?" And if so reject it, but if not allow it. I don't know how to set that up though....

Oh, one other thought: It'd be nice to have some check that says, "Did pylint get worse? Did mypy get worse?" And if so reject it, but if not allow it. I don't know how to set that up though....

There are bots that tell you whether code coverage got better or worse, maybe there's one for telling you whether pylint / mypy got better or worse? The other option is just to use mypy to raise warnings but not fail builds, and then a human reviewer can choose whether typings should be added or not. You could also raise the typing failures as annotations on the review using GitHub Action magic (I'm not sure how easy this would be).
The pylint report is annoying - I didn't know that when pylint writes a report it returns non-zero exit code when warnings are issued, with the exit code representing a bitmask of what types of errors / warnings were issued (???). I'll put a fix in for this.

There are bots that tell you whether code coverage got better or worse, maybe there's one for telling you whether pylint / mypy got better or worse? The other option is just to use mypy to raise warnings but not fail builds, and then a human reviewer can choose whether typings should be added or not. You could also raise the typing failures as annotations on the review using GitHub Action magic (I'm not sure how easy this would be).
The pylint report is annoying - I didn't know that when pylint writes a report it returns non-zero exit code when warnings are issued, with the exit code representing a bitmask of what types of errors / warnings were issued (???). I'll put a fix in for this.

I found that pylint has a parameter you can use to set the threshold and I just added it. Seems like an ok fix. I didn't find anything that'd enforce improvement.
Now the tests are failing...

I found that pylint has a parameter you can use to set the threshold and I just added it. Seems like an ok fix. I didn't find anything that'd enforce improvement.
Now the tests are failing...

In case it's helpful, I enjoyed this article Why Pylint is both useful and unusable, and how you can actually use it. The author recommends turning on pylint checks in small batches and then turning off ones that are mostly noise, to find the ones that are going to have high enough signal-to-noise ratio in your project to be useful. I've been tempted but haven't actually turned on pylint for any projects, so can't say how this pans out.

In case it's helpful, I enjoyed this article Why Pylint is both useful and unusable, and how you can actually use it. The author recommends turning on pylint checks in small batches and then turning off ones that are mostly noise, to find the ones that are going to have high enough signal-to-noise ratio in your project to be useful. I've been tempted but haven't actually turned on pylint for any projects, so can't say how this pans out.

This just generates the HTML report so it seems fine to me to just add --exit-zero. There's a separate job that does pylint -E to fail on errors but not warnings.

OK, I was wondering what was going on there since I didn't recognize the short flag offhand. I think I'll crank that one up so it dies if the threshold is below 9 or so, and so it includes warnings too. I think @jcushman's blog points to the right way. We're actually in pretty good shape here. I do shiver at ever getting this turned on in CL though. I guess I've got my work cut out for me (I wouldn't want anybody else to have to do all of CL).

This just generates the HTML report so it seems fine to me to just add --exit-zero. There's a separate job that does pylint -E to fail on errors but not warnings.

OK, I was wondering what was going on there since I didn't recognize the short flag offhand. I think I'll crank that one up so it dies if the threshold is below 9 or so, and so it includes warnings too. I think @jcushman's blog points to the right way. We're actually in pretty good shape here. I do shiver at ever getting this turned on in CL though. I guess I've got my work cut out for me (I wouldn't want anybody else to have to do all of CL).

OK, all tests are passing. To get this working, I tweaked the lint CI file and I removed support for python 3.6. Hopefully that's OK. It's scheduled for deprecation at the end of this year, so this is perhaps a bit aggressive, but 3.6 doesn't have support for re.Match, which we used for typing. We could hack around it, but it seemed easier to just remove support until somebody needs it.
@mattdahl is there anything else outstanding before I do a full review?

OK, all tests are passing. To get this working, I tweaked the lint CI file and I removed support for python 3.6. Hopefully that's OK. It's scheduled for deprecation at the end of this year, so this is perhaps a bit aggressive, but 3.6 doesn't have support for re.Match, which we used for typing. We could hack around it, but it seemed easier to just remove support until somebody needs it.
@mattdahl is there anything else outstanding before I do a full review?

3.7 is fine with me. It's the version we have in production, being the
python version on Debian stable (Buster) which was released 18 months ago.
On Tue, Jan 12, 2021 at 1:16 PM Mike Lissner<EMAIL_ADDRESS>wrote:

OK, all tests are passing. To get this working, I tweaked the lint CI file
and I removed support for python 3.6. Hopefully that's OK. It's scheduled
for deprecation at the end of this year, so this is perhaps a bit
aggressive, but 3.6 doesn't have support for re.Match, which we used for
typing. We could hack around it, but it seemed easier to just remove
support until somebody needs it.
@mattdahl https://github.com/mattdahl is there anything else
outstanding before I do a full review?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/freelawproject/microscope/pull/1#issuecomment-758844448,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC33UAR7CWCJK3AD5ANIR3SZSGXFANCNFSM4V34FQWA
.

3.7 is fine with me. It's the version we have in production, being the
python version on Debian stable (Buster) which was released 18 months ago.
On Tue, Jan 12, 2021 at 1:16 PM Mike Lissner<EMAIL_ADDRESS>wrote:

OK, all tests are passing. To get this working, I tweaked the lint CI file
and I removed support for python 3.6. Hopefully that's OK. It's scheduled
for deprecation at the end of this year, so this is perhaps a bit
aggressive, but 3.6 doesn't have support for re.Match, which we used for
typing. We could hack around it, but it seemed easier to just remove
support until somebody needs it.
@mattdahl https://github.com/mattdahl is there anything else
outstanding before I do a full review?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/freelawproject/microscope/pull/1#issuecomment-758844448,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC33UAR7CWCJK3AD5ANIR3SZSGXFANCNFSM4V34FQWA
.

I have this installed locally alongside our (CAP's) cite extractor, and it's great to see it working! 🎉
I'm going to drop some feedback here as I mess around -- please don't hold pulling this on my account, though. I'm happy to refile stuff as bugs if it doesn't end up making sense to fold into this initial PR. Just want to share now in case it's handy to make changes along the way.
First thing I've noticed is the default behavior of get_citations() with disambiguate=True is surprising. Right now by default it returns only unambiguous citations, so for example get_citations("23 Wall. 307") == []. If you include disambiguate=False then all citations are returned but no disambiguation is done. At a minimum I'd switch the default.
What might be more useful is if the default behavior is for each cite to come back with a list of all the reporters in reporters_db that they potentially match (including reporter metadata like start and end date), so callers can do their own disambiguation with more context, and the disambiguate=True flag causes get_citations to attempt to narrow that list down to one but not filter anything out.
I think it'll turn out there's an elegant way to implement this internally by having tokenize() return class ReporterStr(str) subclasses instead of strs, that can also have a list of candidate reporters attached, and then use those when scanning the tokens instead of re-checking the master list of reporter strings. This also lets CAP do OCR-friendly tokenization just by plugging the tokenize() function. I can send this part as a separate PR later if helpful.

I have this installed locally alongside our (CAP's) cite extractor, and it's great to see it working! 🎉
I'm going to drop some feedback here as I mess around -- please don't hold pulling this on my account, though. I'm happy to refile stuff as bugs if it doesn't end up making sense to fold into this initial PR. Just want to share now in case it's handy to make changes along the way.
First thing I've noticed is the default behavior of get_citations() with disambiguate=True is surprising. Right now by default it returns only unambiguous citations, so for example get_citations("23 Wall. 307") == []. If you include disambiguate=False then all citations are returned but no disambiguation is done. At a minimum I'd switch the default.
What might be more useful is if the default behavior is for each cite to come back with a list of all the reporters in reporters_db that they potentially match (including reporter metadata like start and end date), so callers can do their own disambiguation with more context, and the disambiguate=True flag causes get_citations to attempt to narrow that list down to one but not filter anything out.
I think it'll turn out there's an elegant way to implement this internally by having tokenize() return class ReporterStr(str) subclasses instead of strs, that can also have a list of candidate reporters attached, and then use those when scanning the tokens instead of re-checking the master list of reporter strings. This also lets CAP do OCR-friendly tokenization just by plugging the tokenize() function. I can send this part as a separate PR later if helpful.

(Again just recording notes for later use, don't mind me.)
I did a bit of speed profiling, which is easier with this code separated out, running py-spy record -o profile.svg -- python profile.py on a script that just extracts cites from a large case ten times in a loop.
Looks like 61% of runtime goes to the line if citation_token in list(EDITIONS.keys()) + list(VARIATIONS_ONLY.keys()):, which can be eliminated by caching REPORTER_STRINGS = set(list(EDITIONS.keys()) + list(VARIATIONS_ONLY.keys())) earlier in the file.
20% goes to elif strip_punct(citation_token.lower()) == "supra":, which can be substantially reduced by replacing strip_punct with a less fancy regex like re.sub(r'[^a-z]', '', citation_token.lower()).
10% (or 50% once those are fixed) goes to strings = REPORTER_RE.split(text) in tokenize(), because of the giant "|".join() regex. I think we can eliminate the tokenize() runtime either with an optional hyperscan dependency or by doing more post-processing on a simpler regex, but that one will take a little more thought.
So, good news -- there should be an easy 5x speedup here from tweaking a couple of lines, and then another 2x or so by messing with tokenize after that.

(Again just recording notes for later use, don't mind me.)
I did a bit of speed profiling, which is easier with this code separated out, running py-spy record -o profile.svg -- python profile.py on a script that just extracts cites from a large case ten times in a loop.
Looks like 61% of runtime goes to the line if citation_token in list(EDITIONS.keys()) + list(VARIATIONS_ONLY.keys()):, which can be eliminated by caching REPORTER_STRINGS = set(list(EDITIONS.keys()) + list(VARIATIONS_ONLY.keys())) earlier in the file.
20% goes to elif strip_punct(citation_token.lower()) == "supra":, which can be substantially reduced by replacing strip_punct with a less fancy regex like re.sub(r'[^a-z]', '', citation_token.lower()).
10% (or 50% once those are fixed) goes to strings = REPORTER_RE.split(text) in tokenize(), because of the giant "|".join() regex. I think we can eliminate the tokenize() runtime either with an optional hyperscan dependency or by doing more post-processing on a simpler regex, but that one will take a little more thought.
So, good news -- there should be an easy 5x speedup here from tweaking a couple of lines, and then another 2x or so by messing with tokenize after that.

(Again just recording notes for later use, don't mind me.)
I did a bit of speed profiling, which is easier with this code separated out, running py-spy record -o profile.svg -- python profile.py on a script that just extracts cites from a large case ten times in a loop.
Looks like 61% of runtime goes to the line if citation_token in list(EDITIONS.keys()) + list(VARIATIONS_ONLY.keys()):, which can be eliminated by caching REPORTER_STRINGS = set(list(EDITIONS.keys()) + list(VARIATIONS_ONLY.keys())) earlier in the file.
20% goes to elif strip_punct(citation_token.lower()) == "supra":, which can be substantially reduced by replacing strip_punct with a less fancy regex like re.sub(r'[^a-z]', '', citation_token.lower()).
10% (or 50% once those are fixed) goes to strings = REPORTER_RE.split(text) in tokenize(), because of the giant "|".join() regex. I think we can eliminate the tokenize() runtime either with an optional hyperscan dependency or by doing more post-processing on a simpler regex, but that one will take a little more thought.
So, good news -- there should be an easy 5x speedup here from tweaking a couple of lines, and then another 2x or so by messing with tokenize after that.

(Again just recording notes for later use, don't mind me.)
I did a bit of speed profiling, which is easier with this code separated out, running py-spy record -o profile.svg -- python profile.py on a script that just extracts cites from a large case ten times in a loop.
Looks like 61% of runtime goes to the line if citation_token in list(EDITIONS.keys()) + list(VARIATIONS_ONLY.keys()):, which can be eliminated by caching REPORTER_STRINGS = set(list(EDITIONS.keys()) + list(VARIATIONS_ONLY.keys())) earlier in the file.
20% goes to elif strip_punct(citation_token.lower()) == "supra":, which can be substantially reduced by replacing strip_punct with a less fancy regex like re.sub(r'[^a-z]', '', citation_token.lower()).
10% (or 50% once those are fixed) goes to strings = REPORTER_RE.split(text) in tokenize(), because of the giant "|".join() regex. I think we can eliminate the tokenize() runtime either with an optional hyperscan dependency or by doing more post-processing on a simpler regex, but that one will take a little more thought.
So, good news -- there should be an easy 5x speedup here from tweaking a couple of lines, and then another 2x or so by messing with tokenize after that.

I literally slapped my forehead reading those performance notes, @jcushman. Those feel like a nice opportunity for a lean PR once this one is merged. Hyperscan kind of freaks me out, but making it optional seems fine. We're not using Intel processors ourselves...I wonder if it'd still be faster (we are x86).
I'm down generally with the API changes you're describing in the disambiguation code. The current code is very much "Mike working in a vacuum," so I expect some low-hanging fruit in ergonomics.
Thanks for the comments.

I literally slapped my forehead reading those performance notes, @jcushman. Those feel like a nice opportunity for a lean PR once this one is merged. Hyperscan kind of freaks me out, but making it optional seems fine. We're not using Intel processors ourselves...I wonder if it'd still be faster (we are x86).
I'm down generally with the API changes you're describing in the disambiguation code. The current code is very much "Mike working in a vacuum," so I expect some low-hanging fruit in ergonomics.
Thanks for the comments.

@mlissner I don't think anything else remains on my end. Substantively, I think this should be all the code that merit copying from CL.
Thanks for fixing all the linting issues. And great work on those performance calculations, @jcushman!
Once this is merged I also see some areas where I'd like to clean up the code a bit (if only for my own sanity)...

@mlissner I don't think anything else remains on my end. Substantively, I think this should be all the code that merit copying from CL.
Thanks for fixing all the linting issues. And great work on those performance calculations, @jcushman!
Once this is merged I also see some areas where I'd like to clean up the code a bit (if only for my own sanity)...

LGTM. Some next steps from the thread:

merge into CL, yes, duh
Fix biggest performance issues
Clean up code to keep Matt sane
Get a deploy going (see #3)

LGTM. Some next steps from the thread:

merge into CL, yes, duh
Fix biggest performance issues
Clean up code to keep Matt sane
Get a deploy going (see #3)

Thank you everybody!

Thank you everybody!


Qt programs segfault on startup in master

Describe the bug
All (?) Qt programs segfault on startup for me in nixpkgs master.
To Reproduce
Steps to reproduce the behavior:

Run konsole or kate
Observe segfault

I'm using NVIDIA, but not sure if relevant. @adisbladis observes this too.
Additional context
This started happening in dcea5637d847d640e0f2fae8839ebe8f97456527 or later.
# /nix/store/31jl621yn5bcb69kd4spkji4p7d4c9yi-kate-19.04.1/bin/kate
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/8x8/legacy/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/16x16/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/16x16/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/22x22/legacy/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/24x24/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/24x24/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/32x32/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/32x32/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/48x48/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/48x48/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/256x256/legacy/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/512x512/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/scalable/ui/"
sonnet.core: No language dictionaries for the language: "en_US"
sonnet.core: No language dictionaries for the language: "en_US"
sonnet.core: No language dictionaries for the language: "en_US"
sonnet.core: No language dictionaries for the language: "en_US"
zsh: segmentation fault  /nix/store/31jl621yn5bcb69kd4spkji4p7d4c9yi-kate-19.04.1/bin/kate

# valgrind --trace-children=yes /nix/store/31jl621yn5bcb69kd4spkji4p7d4c9yi-kate-19.04.1/bin/kate
==25308== Memcheck, a memory error detector
==25308== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==25308== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==25308== Command: /nix/store/31jl621yn5bcb69kd4spkji4p7d4c9yi-kate-19.04.1/bin/kate
==25308==
==25308== Memcheck, a memory error detector
==25308== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==25308== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==25308== Command: /nix/store/31jl621yn5bcb69kd4spkji4p7d4c9yi-kate-19.04.1/bin/.kate-wrapped
==25308==
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/8x8/legacy/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/16x16/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/16x16/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/22x22/legacy/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/24x24/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/24x24/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/32x32/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/32x32/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/48x48/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/48x48/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/256x256/legacy/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/512x512/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/scalable/ui/"
sonnet.core: No language dictionaries for the language: "en_US"
sonnet.core: No language dictionaries for the language: "en_US"
sonnet.core: No language dictionaries for the language: "en_US"
sonnet.core: No language dictionaries for the language: "en_US"
==25308== Stack overflow in thread #1: can't grow stack to 0x1ffe801000
==25308==
==25308== Process terminating with default action of signal 11 (SIGSEGV)
==25308==  Access not within mapped region at address 0x1FFE801FF8
==25308== Stack overflow in thread #1: can't grow stack to 0x1ffe801000
==25308==    at 0x1A507BF4: ??? (in /nix/store/j7cfivh5qb9yyxlg511dzwgpg0zqf33f-qtbase-5.12.0-bin/lib/qt-5.12/plugins/xcbglintegrations/libqxcb-glx-integration.so)
==25308==  If you believe this happened as a result of a stack
==25308==  overflow in your program's main thread (unlikely but
==25308==  possible), you can try to increase the size of the
==25308==  main thread stack using the --main-stacksize= flag.
==25308==  The main thread stack size used in this run was 8388608.
==25308== Stack overflow in thread #1: can't grow stack to 0x1ffe801000
==25308==
==25308== Process terminating with default action of signal 11 (SIGSEGV)
==25308==  Access not within mapped region at address 0x1FFE801FF0
==25308== Stack overflow in thread #1: can't grow stack to 0x1ffe801000
==25308==    at 0x482C12A: _vgnU_freeres (in /nix/store/aip6mvglkqlrw47g5zyczdx2yjjay5nq-valgrind-3.15.0/lib/valgrind/vgpreload_core-amd64-linux.so)
==25308==  If you believe this happened as a result of a stack
==25308==  overflow in your program's main thread (unlikely but
==25308==  possible), you can try to increase the size of the
==25308==  main thread stack using the --main-stacksize= flag.
==25308==  The main thread stack size used in this run was 8388608.
==25308==
==25308== HEAP SUMMARY:
==25308==     in use at exit: 9,996,224 bytes in 124,802 blocks
==25308==   total heap usage: 1,310,542 allocs, 1,185,740 frees, 7,446,616,049 bytes allocated
==25308==
==25308== LEAK SUMMARY:
==25308==    definitely lost: 352 bytes in 2 blocks
==25308==    indirectly lost: 136 bytes in 3 blocks
==25308==      possibly lost: 311,663 bytes in 31 blocks
==25308==    still reachable: 9,618,497 bytes in 124,229 blocks
==25308==                       of which reachable via heuristic:
==25308==                         length64           : 3,656 bytes in 65 blocks
==25308==                         newarray           : 2,872 bytes in 71 blocks
==25308==         suppressed: 0 bytes in 0 blocks
==25308== Rerun with --leak-check=full to see details of leaked memory
==25308==
==25308== For lists of detected and suppressed errors, rerun with: -s
==25308== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

# ulimit -s unlimited
# valgrind --main-stacksize=500000000 --trace-children=yes /nix/store/31jl621yn5bcb69kd4spkji4p7d4c9yi-kate-19.04.1/bin/kate
==28189== Memcheck, a memory error detector
==28189== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==28189== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==28189== Command: /nix/store/31jl621yn5bcb69kd4spkji4p7d4c9yi-kate-19.04.1/bin/kate
==28189==
==28189== Memcheck, a memory error detector
==28189== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==28189== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==28189== Command: /nix/store/31jl621yn5bcb69kd4spkji4p7d4c9yi-kate-19.04.1/bin/.kate-wrapped
==28189==
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/8x8/legacy/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/16x16/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/16x16/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/22x22/legacy/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/24x24/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/24x24/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/32x32/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/32x32/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/48x48/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/48x48/ui/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/256x256/legacy/"
Invalid Context= "Legacy" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/512x512/legacy/"
Invalid Context= "UI" line for icon theme:  "/run/current-system/sw/share/icons/Adwaita/scalable/ui/"
sonnet.core: No language dictionaries for the language: "en_US"
sonnet.core: No language dictionaries for the language: "en_US"
sonnet.core: No language dictionaries for the language: "en_US"
sonnet.core: No language dictionaries for the language: "en_US"
==28189== Stack overflow in thread #1: can't grow stack to 0x1fe132a000
==28189==
==28189== Process terminating with default action of signal 11 (SIGSEGV)
==28189==  Access not within mapped region at address 0x1FE132AFF8
==28189== Stack overflow in thread #1: can't grow stack to 0x1fe132a000
==28189==    at 0x6018680: QWindow::flags() const (in /nix/store/nas96m3mkssidm5mzl9c153y1w2xacg5-qtbase-5.12.3/lib/libQt5Gui.so.5.12.3)
==28189==  If you believe this happened as a result of a stack
==28189==  overflow in your program's main thread (unlikely but
==28189==  possible), you can try to increase the size of the
==28189==  main thread stack using the --main-stacksize= flag.
==28189==  The main thread stack size used in this run was 500002816.
==28189== Stack overflow in thread #1: can't grow stack to 0x1fe132a000
==28189==
==28189== Process terminating with default action of signal 11 (SIGSEGV)
==28189==  Access not within mapped region at address 0x1FE132AFF0
==28189== Stack overflow in thread #1: can't grow stack to 0x1fe132a000
==28189==    at 0x482C12A: _vgnU_freeres (in /nix/store/aip6mvglkqlrw47g5zyczdx2yjjay5nq-valgrind-3.15.0/lib/valgrind/vgpreload_core-amd64-linux.so)
==28189==  If you believe this happened as a result of a stack
==28189==  overflow in your program's main thread (unlikely but
==28189==  possible), you can try to increase the size of the
==28189==  main thread stack using the --main-stacksize= flag.
==28189==  The main thread stack size used in this run was 500002816.
==28189==
==28189== HEAP SUMMARY:
==28189==     in use at exit: 9,998,414 bytes in 124,862 blocks
==28189==   total heap usage: 1,311,370 allocs, 1,186,508 frees, 7,437,822,590 bytes allocated
==28189==
==28189== LEAK SUMMARY:
==28189==    definitely lost: 352 bytes in 2 blocks
==28189==    indirectly lost: 136 bytes in 3 blocks
==28189==      possibly lost: 311,663 bytes in 31 blocks
==28189==    still reachable: 9,620,687 bytes in 124,289 blocks
==28189==                       of which reachable via heuristic:
==28189==                         length64           : 3,656 bytes in 65 blocks
==28189==                         newarray           : 2,952 bytes in 73 blocks
==28189==         suppressed: 0 bytes in 0 blocks
==28189== Rerun with --leak-check=full to see details of leaked memory
==28189==
==28189== For lists of detected and suppressed errors, rerun with: -s
==28189== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
zsh: segmentation fault  valgrind --main-stacksize=500000000 --trace-children=yes

Metadata
Please run nix run nixpkgs.nix-info -c nix-info -m and paste the result.

system: "x86_64-linux"
host os: Linux 4.19.61-hardened, NixOS, 19.09.git.60dcc0f (Loris)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.2.2
channels(root): "nixos-18.09.1922.97e0d53d669"
channels(at): ""
channels(play): ""
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

can confirm the issue when running kate at 618ecd930d0475b377885e34a31086882b5d6bcc on nixos with plasma at b5f5c97

cc @vandenoever @ttuegel
@samueldr bisected it to the 5.12.3 bump https://github.com/NixOS/nixpkgs/issues/65399#issuecomment-515727404

I've reverted the changes in c220e00d7a4524f59e19d7337aa3d47e9a678e64 and bf39fc17d46b48e3883dd646bc0ecc0ccfb67df4 on master.

I'm not sure if the revert is enough - VirtualBox still fails for me with the following:
Qt WARNING: Could not find the Qt platform plugin "xcb" in ""
Qt FATAL: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

[1]    29115 abort      VirtualBox

That's a different issue with the plugins (see e.g. https://github.com/NixOS/nixpkgs/pull/65372 for a potential solution).

Thanks for the reverts, I confirmed they no longer segfault in master.


[RFC] Plugin Developer Journey

Purpose
This RFC describes a possible journey of a future Backstage plugin developer as they build a plugin that touches many different aspects of a Backstage. The story invents many new things that are not part of Backstage today, but are things that I'm suggesting we should add as long term or north star goals. The idea is to discuss what parts of the story makes sense to aim for, and what we'd want to do differently or not at all. The "chapters" are numbered to make it a bit easier to comment on parts of the story.
The Protagonist
Sam is an experienced developer that has worked with Backstage for a while, and knows the best practices and tools available to build plugins. Sam also likes music and wants to have a theme tune for every service in Backstage.
The End
Sam built a Spotify plugin for Backstage that allows service owners to define a theme tune for their service. The theme tune plays whenever a user visits the service page in Backstage. The plugin is published to NPM and available for any organization to easily install and add to their Backstage installation.
1. A New Plugin
Sam chooses to develop this plugin in a standalone project and creates a new plugin using npx @backstage/cli create-plugin, which detects that it's not being run in an existing project and therefore creates a separate plugin repo.
Spinning up the frontend with yarn start, Sam goes to work with getting the base functionality of a Spotify web player going. By installing a couple of dependencies and whipping together a nice UI, the player is pretty much done.
2. The Auth Menace
Sam realizes users need to be authenticated towards the Spotify API to be able to play music, and Backstage doesn't support Spotify login yet. Sam adds the @backstage/plugin-auth-backend as local development middleware in the project, and provides the necessary wrapping logic and configuration for the passport-spotify strategy. The Spotify auth provider is now available in the local development backend, and by adding a frontend SpotifyAuth Utility API that implements the OAuthApi type, it's now working in the frontend too.
const spotifyAuthApiRef = createApiRef<OAuthApi>({
  id: 'core.auth.spotify',
  description: 'Provides authentication towards Spotify APIs',
});

Sam realizes that Spotify auth might be useful to others, and that it would be more convenient if it was a part of the Backstage Core. After submitting and merging a Pull Request with the additions to the @backstage/plugin-auth-backend and @backstage/core packages, Spotify auth is now available for everyone to use. Since the Backstage Core team also adds it to the public demo server, Sam can now get rid of it in the local setup and rely on the shared development auth providers instead.
The only thing left now is making sure that users of the plugin provide Spotify auth in the app. Sam ensures this by adding spotifyAuthApiRef to the plugin's list of required APIs, as well as listing it in the requirements section in the README.
## Requirements

This plugin requires the following APIs to function:

- `spotifyAuthApiRef` from `<EMAIL_ADDRESS>
3. The Catalog Awakens
Sam now has a working player and a method for users to log in to listen to music, but the goal is to provide theme songs for services. Sam adds this functionality by defining a new metadata annotation called sam.wise/spotify-track-id. The annotation's value is a Spotify track ID and can be defined in a component like this:
apiVersion: backstage.io/v1
kind: Component
metadata:
  name: my-component
  annotations:
    sam.wise/spotify-track-id: '4uLU6hMCjMI75M1A2tKUQC'
spec:
  type: service

Sam creates a JSON schema that documents the annotation and allows it to be used in validation and documentation for organizations that choose to adopt the plugin.
{
  "sam.wise/spotify-track-id": {
    "$id": "https://raw.githubusercontent.com/sam/backstage-spotify-theme/master/annotation.json#/sam.wise/spotify-track-id",
    "type": "string",
    "title": "Spotify Track Annotation",
    "description": "Spotify track ID to associated with the entity",
    "examples": ["4uLU6hMCjMI75M1A2tKUQC"]
  }
}

4. The Rise of Widgets
Sam also wraps the music player in an entity page widget. This allows anyone that wants to use the plugin to add the player to any of their entity layout templates, which will make it show up for every entity of that kind.
export const PlayerWidget = plugin.createEntityWidget({
  component: WebPlayer,
  locations: ['header', 'card', 'footer'],
  cardSize: [2, 4],
});

The widget receives information about the entity in whose page it's being embedded, which makes it simple to grab the track id from the annotations and hook up the player.
Sam also modifies the standalone plugin development setup to include this new widget inside a basic entity page, adding it to a couple of different places where users of the plugin may want to put the player, just to make sure they all work.
5. The First User
At this point the only things that anyone that wants to use Sam's plugin needs to do is to add https://raw.githubusercontent.com/sam/backstage-spotify-theme/master/annotation.json#/sam.wise/spotify-track-id to their catalog schema, import and add the PlayerWidget on the desired entity template pages, and make sure they're providing Spotify auth.
Sam soon sees the first "Used by" show up on GitHub, and feedback starts rolling in. Users really like the plugin, and some a requesting the possibility to select a theme tune when creating a new component. Sam jumps on the idea and adds a new creation hook that is exported by the plugin. The hook can be installed either in a single, all, or component templates that match a label. It adds a field as a part of the component creation process with a nice search box that allow users to search for a track that they want to use as the theme tune.
6. Return to the Repo
Sam is pretty content at this point, but would like to make it easier for users to change the track after creation, preferable using the same search box that was made for the creation form. By adding an edit button to the PlayerWidget, and a nice empty state, Sam is able to provide the appropriate hooks in the GUI to open up a search dialog.
To save the selected track, Sam uses the RepoApi to suggest a change to the entity definition file. This will create a Pull Request for organizations that use GitHub, a Merge Request for users of GitLab, and so on.
const repoApi = useApi(repoApiRef);
const alertApi = useApi(alertApiRef);

const onSave = async () => {
  const { url } = await repoApi.createChangeRequest({
    title: `Change theme tune to ${track.title} by ${track.artist}`,
    changes: [
      {
        path: entityYamlPath,
        content: newEntityYamlContent,
      },
    ],
  });

  alertApi.post({ message: `Requested change, ${url}` });
};

Now it's much simpler for users to change the theme tune, as they no longer need to go look up a track ID and edit a yaml file. Instead, they can now stay inside Backstage and search for the track and request the change from there. In addition, the requested change can be reviewed by the regular process of each organization.
7. The User Strikes Back
Sam's plugin is pretty popular at this point, and has been picked up and used by many organizations. But some users start voicing concerns that they have too many different hand-crafted annotations in their entity descriptions, and would like to be able to avoid some of them. They really like the theme tunes though, and wish they could keep them without having to put them in the entity description, even if that means it won't go through the regular source control review process.
One day Sam receives a Pull Request for the plugin. It adds an option to use the Database provided by @backstage/backend-common to store the track ID. It's all packaged into a new backend plugin that will also extend the catalog backend with schema and functionality to automatically load the value of the sam.wise/spotify-track-id annotation from the backend plugin and database. The backend plugin also extends the common GraphQL schema with a mutation that updates the track ID in the database.
On the frontend the Pull Request doesn't change much. It defines the save action that was previously using the RepoApi in its own API.
type ThemeTuneStorageApi = {
  save(entity: Entity, trackId: string): Promise<void>;
};

The plugin also provides two different implementations of the API, one that uses the old behavior of the RepoApi, and a new one that calls the GraphQL API. The new API relies on the IdentityApi as a mechanism for authorizing changes, instead of source control reviews. The IdentityApi provides a token that is included in the request to the backend, which then must match the owner of the component for which the user is trying to change the theme tune.

Author breaking the 4th wall here. I actually think every GraphQL request should include the ID token of the user, but invented a reason to include it here anyway.

The API is selected based on a configuration parameter for the plugin, but defaults to the original RepoApi behavior.
if (config.getBoolean('storeTrackInDatabase')) {
  return new GraphQLThemeTuneStore(graphqlClient, identityApi, alertApi);
} else {
  return new RepoThemeTuneStore(repoApi, alertApi);
}

Sam is amazed by the pure awesomeness of this change, replies with a "👍" and hits merge.
8. Attack of the Clones
Sam just released v1.8.4 of the plugin, and at this point it's so popular that a couple of other plugins has started depending on the sam.wise/spotify-track-id annotation. One such plugin being the spotify-album-art plugin that can display the album art of the theme tune as the background of the entity header. Sam thinks it's all pretty cool, but doesn't like that the annotation that was once an internal concern of the plugin is now becoming a standard in the community.
In order to standardize the annotation in Backstage, Sam submits a Pull Request to the Backstage Core repo. The request suggest a new well-known metadata annotation called spotify.com/track-id, with the same schema definition as Sam's label, and refers to Sam's own plugin and the spotify-album-art plugin as existing usages. The Backstage maintainers merge the Pull Request, after checking with the folks over at Spotify that they're cool with the annotation, and faffing about over some minor grammar mistake in the annotation description.
With the annotation now available inside Backstage Core, Sam releases v2 of the plugin, which uses the new annotation. It can still consume the old annotation for backwards compatibility, but new users of the plugin no longer need to add the https://raw.githubusercontent.com/sam/backstage-spotify-theme/master/annotation.json#/sam.wise/spotify-track-id extension to their catalog schema, as it's now part of the core schema. The new release of Sam's plugin specifies a dependency on Backstage with a minimum version set to the same release as the one were the annotation was added to the core schema.

9. Revenge of the Sam
Sam, now in full control of all theme tunes in Backstage, releases v2.0.1, which switches all tracks to 4uLU6hMCjMI75M1A2tKUQC. Sam wanted to do something more nefarious, but since Backstage sandboxes sensitive actions and is mostly read-only with strict CSP, Sam's hands were tied.

@nikek Yeah a core schema would be a tricky thing to manage. I'm hinting at some possible rules we could use, for example usage by multiple plugins, and that org-specific fields would preferably be approved by the org.
Regarding the widgets, I was trying to imply that the configuration given by the plugin developer just limits the options for how a widget can be used. Kinda like the Android widget API lets you specify resize modes and minimum dimensions, https://developer.android.com/guide/topics/appwidgets. Then it'd be up to the app developer to choose how to use the widget, but within the constraints set up by the widget author.

Moved to docs.


Invoices

in the endpoints file there is some documentation about invoices.
I would like to automatically download my supercharging invoices, but can not get it to work.
"DOWNLOAD_CHARGING_INVOICE": {
    "TYPE": "GET",
    "URI": "bff/v2/mobile-app/charging/invoice/{uuid}",
    "AUTH": true
  },
  "DOWNLOAD_CHARGING_SUBSCRIPTION_INVOICE": {
    "TYPE": "GET",
    "URI": "bff/v2/mobile-app/charging/subscription/invoice/{invoiceId}",
    "AUTH": true
  },

using these endpoints needs a uuid or invoiceId, which I am trying to obtain with this endpoint:
  "MANAGE_GET_SUBSCRIPTION_INVOICES": {
    "TYPE": "GET",
    "URI": "bff/v2/mobile-app/subscriptions/invoices",
    "AUTH": true
  },

I am using this url:
https://owner-api.teslamotors.com/bff/v2/mobile-app/subscriptions/invoices
I am using GET, with requestheader application/json
params is: {"vehicle_id":"xxxxxxxxxxxx"} (id is correct, and works for all other api requests)
response is:
{"response":null,"error":"param is missing or the value is empty: vehicle_id","error_description":""}
I tried attaching the vehicle_id in the url, as needs to be done for other calls, but then I get a 404 error.
Any help?

Curious about this too.  I've tried for days to get supercharging costs from the API but have not gotten anywhere.

What headers are you sending?

The BFF (backend-for-frontend) endpoints don't appear to authenticate the same way as the other vehicle endpoints do. I'd love to figure out what's going on here. Since they started obfuscating the JavaScript bundle in the app, it's gotten a little more difficult to figure out what the app is up to with these endpoints. I either need to packet sniff or just de-obfuscate the JS bundle, but we should probably focus a bit on figuring out how to call these endpoints.

The BFF (backend-for-frontend) endpoints don't appear to authenticate the same way as the other vehicle endpoints do. I'd love to figure out what's going on here. Since they started obfuscating the JavaScript bundle in the app, it's gotten a little more difficult to figure out what the app is up to with these endpoints. I either need to packet sniff or just de-obfuscate the JS bundle, but we should probably focus a bit on figuring out how to call these endpoints.

I've been able to packet sniff the app and I'll post that information here when I get a chance. Currently trying to packet sniff the actual car as well, but that's proving to be a little tricky so far.

Through checking the HTTP requests on the app, I've been able to get Invoice data through:
https://ownership.tesla.com/graphql?deviceLanguage=en&deviceCountry=US&vin={vin}&operationName=getChargingHistoryV2

The request is as follows:
POST /graphql?deviceLanguage=en&deviceCountry=US&vin={VIN}&operationName=getChargingHistoryV2 HTTP/1.1
Host: ownership.tesla.com
Accept: */*
Authorization: Bearer {token}
Accept-Encoding: gzip, deflate, br
Accept-Language: en
Cache-Control: no-cache
charset: utf-8
Content-Length: 1376
User-Agent: Tesla/1032 CFNetwork/1331.0.7 Darwin/21.4.0
Connection: keep-alive
x-tesla-user-agent: TeslaApp/4.8.1/5e1bfb8d0d/ios/15.4.1
Content-Type: application/json
x-txid: {x-txid}
Cookie: bm_sv={value}; ak_bmsc={value}

Which in turn will return roughly the following response
"charging": {
        "historyV2": {
          "data": [{
            "countryCode": "US",
            "programType": "PTSCH",
            "billingType": "IMMEDIATE",
            "vin": "{VIN}",
            "credit": null,
            "invoices": null,
            "chargeSessionId": "1234565-0-12345566",
            "siteLocationName": "LOCATION",
            "chargeStartDateTime": "2022-05-21T21:06:18-04:00",
            "chargeStopDateTime": "2022-05-21T21:46:20-04:00",
            "unlatchDateTime": "2022-05-21T21:46:21-04:00",
            "fees": [{
              "sessionFeeId": 1234566,
              "feeType": "CHARGING",
              "payorUid": null,
              "amountDue": null,
              "currencyCode": "USD",
              "pricingType": "PAYMENT",
              "usageBase": 61,
              "usageTier1": 10,
              "usageTier2": 12,
              "usageTier3": 17,
              "usageTier4": 0,
              "rateBase": 0.17,
              "rateTier1": 0,
              "rateTier2": 0,
              "rateTier3": null,
              "rateTier4": null,
              "totalTier1": 0,
              "totalTier2": 0,
              "totalTier3": 0,
              "totalTier4": 0,
              "uom": "kwh",
              "isPaid": true,
              "uid": 1234567,
              "totalBase": 10.37,
              "totalDue": 10.37,
              "netDue": 10.37,
              "status": "PAID"
            }],
            "vehicleMakeType": "TESLA",
            "sessionId": 1234567,
            "surveyCompleted": null,
            "surveyType": null,
            "postId": "0",
            "cabinetId": "vehicleMakeType",
            "din": ""
          }

I've only been able to view the request on the Tesla app. I'm still trying to figure out the header situation since I believe the x-txid or the cookies are single use.

Any update on this? As there is no way to get my invoices (premium connectivity and supercharging) via mail, I’d like to access them via api…

I'm also curious if you found anything here. If i were to guess, the secret is in the cookies.

PDF invoices can be accessed with
GET https://ownership.tesla.com/mobile-app/charging/invoice/{INVOICE_ID}?deviceCountry=US&deviceLanguage=en&vin={VIN}

The inconvenience is that you'd need to know the INVOICE_ID.

thanks @idriskhenchil, we're still a bit stuck on getting the list of invoices. Presently getting back "Bad Request". Perhaps missing some headers.

@davidhodge Also the same error I get back when trying to access the request outside of the app. I used the same headers present in the mobile request as the headers I used in the client.
My wild guess is that I think the app may generate a client-side token that is used to validate where the request is coming from, in order to prevent the request from being made elsewhere. In a sense each request would only be single-use which could be why it works once when analyzing the request in the app, and doesn't work elsewhere.

@idriskhenchil @wannesdemaeght
I was able to get invoice history response off the app without cookies with this endpoint:
POST https://akamai-apigateway-charging-ownership.tesla.com/graphql?deviceLanguage=en&deviceCountry=US&ttpLocale=en_US&vin={vin}&operationName=getChargingHistoryV2
I've seen other "akamai-apigateway-" host prefixes but I searched a while and couldn't find "akamai-apigateway-charging-ownership.tesla.com".
I did not need to add the "x-txid" key to the header (or even the bm_sv cookies).
I hope I'm on the right track and this is helping.

hi @chadhurin
unfortunately it's not working for me. responsestatus is 400 - Bad Request.
accompanying text:  {"errors":[{"message":"Oops! Something went wrong. 387be28a-4046-4479-9d86-61bebf6a10cf","txId":"387be28a-4046-4479-9d86-61bebf6a10cf"}]}
I used both POST and GET, with the url you provided, and a working bearer-token...

Hey, @wannesdemaeght
Try this in your body
{   "query": "\n    query getChargingHistoryV2($pageNumber: Int!, $sortBy: String, $sortOrder: SortByEnum) {\n  me {\n    charging {\n      historyV2(pageNumber: $pageNumber, sortBy: $sortBy, sortOrder: $sortOrder) {\n        data {\n          ...SparkHistoryItemFragment\n        }\n        totalResults\n        hasMoreData\n        pageNumber\n      }\n    }\n  }\n}\n    \n    fragment SparkHistoryItemFragment on SparkHistoryItem {\n  countryCode\n  programType\n  billingType\n  vin\n  credit {\n    distance\n    distanceUnit\n  }\n  chargingPackage {\n    distance\n    distanceUnit\n    energyApplied\n  }\n  invoices {\n    fileName\n    contentId\n    invoiceType\n  }\n  chargeSessionId\n  siteLocationName\n  chargeStartDateTime\n  chargeStopDateTime\n  unlatchDateTime\n  fees {\n    ...SparkHistoryFeeFragment\n  }\n  vehicleMakeType\n  sessionId\n  surveyCompleted\n  surveyType\n  postId\n  cabinetId\n  din\n}\n    \n    fragment SparkHistoryFeeFragment on SparkHistoryFee {\n  sessionFeeId\n  feeType\n  payorUid\n  amountDue\n  currencyCode\n  pricingType\n  usageBase\n  usageTier1\n  usageTier2\n  usageTier3\n  usageTier4\n  rateBase\n  rateTier1\n  rateTier2\n  rateTier3\n  rateTier4\n  totalTier1\n  totalTier2\n  totalTier3\n  totalTier4\n  uom\n  isPaid\n  uid\n  totalBase\n  totalDue\n  netDue\n  status\n}\n    ",   "variables": {     "sortBy": "start_datetime",     "sortOrder": "DESC",     "pageNumber": 1   },   "operationName": "getChargingHistoryV2" }
with these headers:
--header 'User-Agent: Tesla/1195 CFNetwork/1388 Darwin/22.0.0'
--header 'x-tesla-user-agent: TeslaApp/4.11.1/12ad93c62a/ios/16.0'
--header 'Content-Type: application/json'
--header 'Authorization: Bearer {token}

@wannesdemaeght Did that work?

Hi @chadhurin, thanks for your help!
I got everything working exactly as I wanted.
could you maybe help me along with the correct body to get a list of the premium connectivity invoices?

@chadhurin do you know what endpoint I could use for the premium connectivity invoices, as well as what to put in the body?
Would be a great help!

@chadhurin Hey, do you know if it is possible to get the supercharger location's latitude and longitude from the graphql query from your comment:
https://github.com/timdorr/tesla-api/issues/539#issuecomment-1206834760
If so, can you let me know what needs to be added to that query? Thank you.

@idriskhenchil @wannesdemaeght I was able to get invoice history response off the app without cookies with this endpoint: POST https://akamai-apigateway-charging-ownership.tesla.com/graphql?deviceLanguage=en&deviceCountry=US&ttpLocale=en_US&vin={vin}&operationName=getChargingHistoryV2
I've seen other "akamai-apigateway-" host prefixes but I searched a while and couldn't find "akamai-apigateway-charging-ownership.tesla.com".
I did not need to add the "x-txid" key to the header (or even the bm_sv cookies).
I hope I'm on the right track and this is helping.

So this is still working in 2023, but I noticed it no longer returns the invoices for each VIN. It'll instead send a list of the most recent invoices regardless of VIN, but you still need to provide a VIN query parameter or it'll return an error 400.
Is that your experience or am I sending my request incorrectly?
I also noticed invoice.invoices.contentId is null for some invoices. Does this just mean a PDF isn't available to download yet?

I wonder, what is the query that feeds the graph located in the Charging Stats section in the app?

I want to create a periodic charging report, to include data such as “charge start time”, “charge end time”, “energy consumed”, “location” (home, supercharge, work, other) and the cost, for each vehicle.
Sadly, Tesla do not provide charging data export, so I wonder if it’s possible to retrieve it using the GraphQL query or the REST API? Because, all the data I need does exists in the app.
Thanks!
P.S:
I am JS developer. I would appreciate if someone can send me the obfuscated JS files from the app. I want to try to read them in order to extract as much information as possible.

getChargingHistoryV2 and GetNearbyChargingSites return http 403 since approx 24h
does it still work for you?

I was getting the same error for getChargingHistoryV2, so I found the invoices from this uri.
https://ownership.tesla.com/mobile-app/charging/history?

Query Params:
vin=%
deviceLanguage=en
deviceCountry=US

Headers:
Authorization: Bearer %

It only returns the 5 most recent invoices and I am not sure how to query for more pages. Would you or anyone else know how?

getChargingHistoryV2 and GetNearbyChargingSites return http 403 since approx 24h
does it still work for you?

Just checked and I too am getting error 403 now ://

@superfloh247 I was getting the same error for getChargingHistoryV2, so I found the invoices from this uri.
https://ownership.tesla.com/mobile-app/charging/history?

Query Params:
vin=%
deviceLanguage=en
deviceCountry=US

Headers:
Authorization: Bearer %

It only returns the 5 most recent invoices and I am not sure how to query for more pages. Would you or anyone else know how?

Thank you for the solution! I implemented it and I got ALL of my invoices since late November of 2022

@superfloh247 I was getting the same error for getChargingHistoryV2, so I found the invoices from this uri.
https://ownership.tesla.com/mobile-app/charging/history?

Query Params:
vin=%
deviceLanguage=en
deviceCountry=US

Headers:
Authorization: Bearer %

It only returns the 5 most recent invoices and I am not sure how to query for more pages. Would you or anyone else know how?

Thank you for the solution! I implemented it and I got ALL of my invoices since late November of 2022

No problem. Did you add any extra query params?

@superfloh247 I was getting the same error for getChargingHistoryV2, so I found the invoices from this uri.
https://ownership.tesla.com/mobile-app/charging/history?

Query Params:
vin=%
deviceLanguage=en
deviceCountry=US

Headers:
Authorization: Bearer %

It only returns the 5 most recent invoices and I am not sure how to query for more pages. Would you or anyone else know how?

Thank you for the solution! I implemented it and I got ALL of my invoices since late November of 2022

No problem. Did you add any extra query params?

No, this is how I make the request:

What'd you find? @superfloh247

It was a combination of HTTP headers to make it work again
https://github.com/superfloh247/TeslaLogger/commit/f403495c95f76761d3a1c18aeae930dc29498f3a

and by "it works" I mean the GraphQL queries, not the GET request

this endpoint works, and is a lot cleaner to implement.
Too bad that there are only 5 invoices generated...
Anyone have an idea on how to get the premium connectivity invoices?
I tried url  https://ownership.tesla.com/mobile-app/premium-connectivity/history?vin=5YJ3E7EB9KF48520&deviceLanguage=en&deviceCountry=US but no dice... (VIN spoofed).

I wonder, what is the query that feeds the graph located in the Charge Stats section in the app?
I want to create a periodic charging report (monthly), to include data such as “charge start time”, “charge end time”, “energy consumed”, “location” (home, supercharge, work, other) and the cost, for each vehicle.
Sadly, Tesla do not provide charging data export, so I wonder if it’s possible to retrieve it using the GraphQL query or the REST API? Because, all the data I need does exist in the app.
Thanks!
P.S: I am JS developer. I would appreciate if someone can send me the obfuscated JS files from the app. I want to try to read them in order to extract as much information as possible.

Found the request! :)
Request
POST https://owner-api.teslamotors.com/api/1/vehicles/{{vehicle-id}}/charge_history?interval={{interval}}&currency_code=ILS&client_country=IL&time_zone=Indian/Mayotte&start_date={{start-date}}&end_date={{end-date}}`

Headers
Authorization: Bearer {{token}}
Content-Type: application/json
x-tesla-user-agent: TeslaApp/4.24.0-1911

[!IMPORTANT]
x-tesla-user-agent header is important!

Body
You can leave it empty.
Variables
{{vehicle-id}}
Your vehicle ID, that can be retrieved using the https://owner-api.teslamotors.com/api/1/vehicles request.
{{interval}}
Optional. The interval of the records. As far as I know, it supports only the values day and month. If no interval param is provided, day is the default.
{{start-date}}
Optional. The start date from which to show the records.
It should look like this: 2023-07-01T00%3A00%3A00%2B03%3A00.
If start_time is omitted, will return the recent records.

[!IMPORTANT]
Make sure value is URL encoded.

[!NOTE]
The resolution of each record, depends on the {{interval}} value.

{{end_date}}
Required if start_date is provided. The start date from which to show the records.
It should look like this: 2023-07-31T00%3A00%3A00%2B03%3A00.

[!IMPORTANT]
Make sure value is URL encoded.

{{token}}
You know. Your token in Tesla, generated by OAuth 2.0 protocol.

this endpoint works, and is a lot cleaner to implement. Too bad that there are only 5 invoices generated...
Anyone have an idea on how to get the premium connectivity invoices? I tried url https://ownership.tesla.com/mobile-app/premium-connectivity/history?vin=5YJ3E7EB9KF48520&deviceLanguage=en&deviceCountry=US but no dice... (VIN spoofed).

hi @wannesdemaeght
i know it has been a long time, but this week I figured out the correct endpoint for Premium Connectivity. It seems like that we also have the same use case "invoices per mail" https://github.com/aSauerwein/tesla-invoices/blob/main/download_v2.py
this is what you need for Premium Connectivity Invoices

url to get subscription invoice ids
https://ownership.tesla.com/mobile-app/subscriptions/invoices?deviceLanguage=en&deviceCountry=AT&httpLocale=en_US&vin=XXXXYYYZZZZ&optionCode=$CPF1
url to download subscription invoice:
https://ownership.tesla.com/mobile-app/documents/invoices/{{invoice-uuid}}?deviceLanguage=en&deviceCountry=AT&httpLocale=en_US&vin=XXXXYYYZZZZ

hope this helps anyone


Support Spock tests with Groovy 4.0.0 and Spock 2.2-M1-groovy-4.0

Expected Behavior
I want to compile a Micronaut application using the Micronaut Application and Spock Test plugins using Groovy 4.0.0 and  Spock 2.2-M1-groovy-4.0.  (The application is in Java, only the tests are using Groovy/Spock)
Actual Behaviour
I get the following error from the Gradle build:
Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:4.0.0' also provided by [org.codehaus.groovy:groovy:3.0.9(compile)]

Full error:
> Could not resolve all files for configuration ':cj-btc-daemon:testCompileClasspath'.
[1800](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1800)
   > Could not resolve org.apache.groovy:groovy:4.0.0.
[1801](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1801)
     Required by:
[1802](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1802)
         project :cj-btc-daemon
[1803](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1803)
         project :cj-btc-daemon > org.spockframework:spock-core:2.2-M1-groovy-4.0
[1804](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1804)
         project :cj-btc-daemon > org.apache.groovy:groovy:4.0.0 > org.apache.groovy:groovy-bom:4.0.0
[1805](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1805)
      > Module 'org.apache.groovy:groovy' has been rejected:
[1806](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1806)
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:4.0.0' also provided by [org.codehaus.groovy:groovy:3.0.9(compile)]
[1807](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1807)
   > Could not resolve org.codehaus.groovy:groovy.
[1808](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1808)
     Required by:
[1809](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1809)
         project :cj-btc-daemon
[1810](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1810)
      > Module 'org.codehaus.groovy:groovy' has been rejected:
[1811](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1811)
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:3.0.9' also provided by [org.apache.groovy:groovy:4.0.0(groovyApiElements)]
[1812](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1812)
   > Could not resolve org.codehaus.groovy:groovy:3.0.9.
[1813](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1813)
     Required by:
[1814](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1814)
         project :cj-btc-daemon > io.micronaut:micronaut-bom:3.3.3
[1815](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1815)
         project :cj-btc-daemon > io.micronaut:micronaut-bom:3.3.3 > org.codehaus.groovy:groovy-bom:3.0.9
[1816](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1816)
         project :cj-btc-daemon > io.micronaut:micronaut-inject-groovy:3.3.3
[1817](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1817)
      > Module 'org.codehaus.groovy:groovy' has been rejected:
[1818](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1818)
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:3.0.9' also provided by [org.apache.groovy:groovy:4.0.0(groovyApiElements)]
[1819](https://github.com/ConsensusJ/consensusj/runs/5281594437?check_suite_focus=true#step:6:1819)

Steps To Reproduce

Use OpenJDK 11 or greater.
Checkout https://github.com/ConsensusJ/consensusj/ commit c954987ef095f811848f5f7ecda2cfc02e96c26c
./gradlew build

(The msgilligan-groovy-4.0 branch currently has 2 commits. The first c954987ef095f811848f5f7ecda2cfc02e96c26c shows the failure, the second commit 19a9632f523348a345fdbc10e92707c6dac2beb8 has the workaround of commenting out the two Micronaut "daemon" modules)
Environment Information
Problem occurs on Github Actions using JDK 11 or 17, and on macOS, Windows, and Linux.
Example Application
https://github.com/ConsensusJ/consensusj/commit/c954987ef095f811848f5f7ecda2cfc02e96c26c
Version
3.3.3 and Gradle Plugin 3.2.2

This is an expected error. Gradle is basically telling you that now you have 2 different versions of Groovy on classpath, both living at different GAV coordinates. The fix is to tell it which one to use:
configurations.all {
    resolutionStrategy.capabilitiesResolution.all {
        if (capability.group.equals("org.codehaus.groovy")) {
            selectHighestVersion()
        }
    }
}

(I haven't actually tested the fix on your project but I've used something similar a few weeks ago, see https://twitter.com/CedricChampeau/status/1486257246350852097)
The Gradle plugin can't do much by itself, because Micronaut itself depends on an older Groovy version.

Thanks @melix !  I'm so happy to see you working on Micronaut! Sorry for not getting back to you sooner.
I'm integrating the fix right now and will let you know what happens. The Micronaut apps in this multi-project are both very simple and the only usage of Groovy is in the (sadly minimal) Spock tests, so it seems I should be fine.

@melix's suggestion fixed my project and I believe handles the case where the Micronaut application does not use Groovy, but the tests use Spock -- which is what this Issue was asking for -- so I'm closing the issue.
Thanks again, Cedric!


Add RESTful API Using FastAPI

Issue Description:
Let's enhance Hezar by adding a RESTful API. This will allow users to easily access and utilize Hezar's AI models via HTTP endpoints.
Objective:
The goal of this enhancement is to provide a user-friendly and standardized way for the community to interact with Hezar's AI models. By exposing these models through a RESTful API, users can leverage AI capabilities with ease, making them accessible to a broader audience.
Proposed Implementation:
I suggest implementing a RESTful API (OpenAPI Specification) using FastAPI. This API will expose endpoints for various AI models and functions offered by Hezar.
If the community finds this feature useful, I'm willing to work on its implementation.

Hello @AmirLavasani jan, thanks for this awesome feature request/ideation.
We're currently working on adding Hezar to the Hugging Face Hub Inference API so that users can easily interact with our models through HTTP requests which comes free for us. It also enables our models to be directly inferenced on the Hub. You can find out more in #56.
The solution you proposed is also desirable. The only issue here might be that we'll need access to servers to host our services.
One idea that we had a while ago, was to provide some plug-and-play Docker images for different tasks that serves the same purpose that you proposed. The reason to do so is that some models are not suitable for use in real life out of the box. For example our OCR models only work on already detected text box images and you cannot use them for an image that has multiple text segments inside. So the solution is to make use of two models in a pipeline to extract all texts correctly (A CTPN + CRNN pipeline for instance), hence needing a separate end-to-end service to do so. Let me know what you think <3 .

Hi @arxyzan jan, Thanks. Let's break down the points.

I'd like to contribute to adding Hezar to the Hugging Face Hub Inference API. It's a very valuable feature.
To clarify, I wasn't suggesting hosting Hezar services online. I meant creating a local API interface with FastAPI for easier model interactions. Think of it as a web UI interface, similar to the CLI discussed in issue #100. Here is an example of the FastAPI Docs interactive UI. Using this web UI interface, users can make inferences using HTTP requests. The models will run locally.
I think the concept of an end-to-end solution is very powerful. It could serve as an abstract layer for utilizing low-level models in pipelines, such as OCR as you mentioned, or a voice chat with Whisper and LLMs, or YouTube translation using ASR and translation models. Nevertheless, it requires initial design decisions on how to structure these pipelines.
I'll reach out to set up an online meeting to explore this further.

Thanks once more, and great job on this project!

Thanks for the follow up @AmirLavasani jan.
Now I see. I think we would need more discussion in this regard. My ideation for now is that we must add an api  module that can handle serving any model and make it ready for inference. We've all had such experience for ML models and I don't think it would be a hard thing to do. The important part here is that, as you mentioned, we need some design decisions for this section.
Some additional notes on your points:

Adding Hezar to the HF Hub Inference API has a minor blocker which we haven't solved yet. It's not that complicated but we don't want to just fix it. we need a nice and easy schema for converting Hezar model outputs to HF Inference API compatible outputs. You can learn more about this challenge in this issue #56
For end-to-end solutions I think having another repo like hezar-apps is more convenient. We can discuss this later.

Unfortunately, I won't be available until next weekend. For now you can message me on Telegram (@arxyzan) or email and we'll set a meeting for later.
Thanks for your contribution Amir jan.

You're welcome @arxyzan jan! 😊
I completely agree; adding an api module, following FastAPI structure guidelines, should be a great starting point.
Regarding the HF API, I'll take a look at HF Inference API-compatible outputs.
The concept of "hezar-apps" is intriguing. We can certainly delve into this idea further.
Next weekend works for me, we'll schedule a meeting at your convenience.

I don't the status of this implementation but I have created a repo that contains the overall structure for wrapping Hezar around FastAPI.
https://github.com/rezashabrang/hezar-api
Currently it only support NLP domain and the models that only require text.
I wanted to make the Docker image on github container registry but there are limitations due to insane size of Hezar packages. I'll look into it in future as how we can reduce the overall image size (Maybe using specifc domains like hezar[nlp].
This is just the starting point and we need to add other routers depending on the domain: e.g for computer vision we need to be able to accept images via router for generating captions or accepting sound file for speech recognition domain.

Hi @rezashabrang, thanks for the effort man. I'm sorry to see this so late, my GH notifs were bugged for a while and I missed some of the issues and mentions. I just saw the repo and I think it's pretty solid. If you're still willing to help us out on this task just let me know.

Hey @arxyzan! No problem and I'm still on board for this implementation. You can also state your ideas on how to integrate this into hezar (e.g: separate repo) or any opinion for the API itself and I'd be happy to contribute.

@rezashabrang So glad to have you with us.
As you might have noticed, Hezar models prediction follows the same pipeline for any task meaning that all models (independant of the task) take an input or a batch of inputs and output a list of results (which is dependant on the task). So one main challenge is to implement the service in a way that the same flexibility is kept there too. I know that this might have some overheads for now, so my best solution is to implement one POST route for each of the tasks (6-7 tasks for models and also word embeddings, we can add preprocessors later too) so that each one has its own input/output request schema.
Regarding the second question, I think we can easily add an api or serve module in the root module (hezar.serve for example) and put everything there.
Unfortunately, I'm not familiar with FastAPI design patterns and best practices so I'd be glad to leave this to you.
Let me know what you think.


Fix bug and code quality

Fix typo in #224
Add more assertions and exceptions

Codecov Report
Base: 50.40% // Head: 50.32% // Decreases project coverage by -0.08% :warning:

Coverage data is based on head (c6ee238) compared to base (5aede6b).
Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #237      +/-   ##
============================================
- Coverage     50.40%   50.32%   -0.09%
+ Complexity      652      651       -1
============================================
  Files           127      127
  Lines          2964     2967       +3
  Branches        349      351       +2
============================================
- Hits           1494     1493       -1
  Misses         1345     1345
- Partials        125      129       +4

Impacted Files
Coverage Δ

...ect/logic/commands/customise/CustomiseCommand.java
81.96% <0.00%> (-5.97%)
:arrow_down:

src/main/java/soconnect/ui/PersonCard.java
0.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Create new array if parsed babel file does not have the plugin section.

Fixing the "babylonParser" expects that "plugins" is set in error #2819

Thanks for sending a PR!


weird result when evaluate with lora

I finetune the llama-7b model on alpaca by runing run_finetune_with_lora.sh for 0.01 epochs, and evaluate by run_evaluate_with_lora.sh. I only change the path without any modification to the setup, but the result is weird, the accuracy on the test set was 0.0, do anyone know why this would happen?
Here is the result of the last test sample:

Hi,
alpaca is an  instruction-following dataset and the outputs are not supposed to be measure by accuracy. You can manually check if the prediction answers are satisfied. Or you can try finetune your model on other datasets like MedMCQA which can be measured by accuracy. Also, when fine-tuning your model, try increase your epoch number, otherwise the model can learn little about the training data.

@SHUMKASHUN Thank you! now the result is reasonable.


can you create its executable version

can you create its executable version

like an executable application? no...
but we will release the source code (in Python, of course) in a few days.


⚠ ClamAV has degraded performance

In 9f9d10b, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 177 ms

Resolved: ClamAV performance has improved in 8826816.


HSEARCH-2813 Add Elasticsearch performance tests

https://hibernate.atlassian.net//browse/HSEARCH-2813

Jenkins, retest this please

An example run (it takes quite a bit of time): http://ci.hibernate.org/view/Search/job/hibernate-search-performance-elasticsearch/20/execution/node/20/log/

merged! great job


Ubuntu build from wrong (19.04) version. Should be built against 16.04 for maximum compatibility. (As suggested by godot docs)

x11 builds against Ubuntu 18.04; I think it's safe to mark that as oldest supported as 20.04 just released


suggestions for search query

Add suggestions for search query as the user enters text into the search bar.

assign me this.

Go ahead @Krunal-Karena

assign me as well!!

go ahead @DeepRahangdale


Regarding course access

Hi,
Just wanted to know whether these demos are recorded or available some where?
Thanks,
Rahul

Hi @rahulsahay19 ,
There are recorded in Bulgarian language for now. Maybe you want an English version?
Kind regards,
Ivaylo

Yes.


Gradle plugin is no longer compatible with Kotlin 2.0.0 and by Dependency Management Plugin

Expected Behavior

The current approach for mismatching versions of your project and the embedded one in detekt (in the context of Gradle runtime dependencies) is described here:
https://detekt.dev/docs/gettingstarted/gradle/#dependencies.
This was working in all versions except 2.0.0
configurations.all {
    resolutionStrategy.eachDependency {
        if (requested.group == "org.jetbrains.kotlin") {
            useVersion(io.gitlab.arturbosch.detekt.getSupportedKotlinVersion())
        }
    }
}

With this prior 2.0.0 you fix the message like when you build your project:
detekt was compiled with Kotlin 1.8.0 but is currently running with 1.7.0
Observed Behavior

With the same approach in kotlin 2.0.0 with the configuration, the :compileKotlin task now fails with this message:
Execution failed for task ':compileKotlin'.
> Could not resolve all files for configuration ':detachedConfiguration1'.
   > Failed to transform kotlin-reflect-1.9.23.jar to match attributes {artifactType=classpath-entry-snapshot, org.gradle.libraryelements=jar, org.gradle.usage=java-runtime}.
      > Execution failed for BuildToolsApiClasspathEntrySnapshotTransform: /home/anonym/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.9.23/ab01e1cc8428c2140891318838ad7039f6fa389a/kotlin-reflect-1.9.23.jar.
         > Artifact org.jetbrains.kotlin:kotlin-build-tools-impl must have version aligned with the version of KGP when compilation via the Build Tools API is disabled.

Steps to Reproduce

I create this repo with the main branch having the bug in kotlin 2.0.0 and the olderexample branch with this working in kotlin 1.9.4, both with detekt 1.23.6
Context
I'm trying to run detekt with Gradle in Kotlin 2.0.0 wit spring boot, but breaks the compilation of the project.
Can be related or duplicated from: https://github.com/detekt/detekt/issues/7304
Your Environment

Version of detekt used: 1.23.6
Version of Gradle used (if applicable): 8.8
Gradle scan link (add --scan option when running the gradle task):
Operating System and version: Ubuntu 22.04
Link to your project (if it's a public repository): https://github.com/vcruzmj/demo-detekt

We don't want or need the Kotlin version to be overridden on all configurations. We just need the detekt configuration to be left alone.
If you apply the config you've provided then you're overriding all configurations. That's having an impact on Kotlin itself which is what the warning is about.
There isn't much detekt can do about this directly. Your Gradle config needs to be fixed, or you need to find out which plugin is overriding the detekt configuration and stop it from doing that.

If you apply the config you've provided then you're overriding all configurations. That's having an impact on Kotlin itself which is what the warning is about.

But that is detekt official documentation https://detekt.dev/docs/gettingstarted/gradle/#dependencies, it's even in the logs of the warning.

There isn't much detekt can do about this directly. Your Gradle config needs to be fixed, or you need to stop the dependency plugin overriding the configs.

But below Kotlin 2.0.0 this was working, so, was the official way to do it now? The only real change in my gradle configuration is kotlin from 1.9.4 to 2.0.0.

reading #6198 more in detail, I tried this:
configurations.matching { it.name == "detekt" }.all {
    resolutionStrategy.eachDependency {
        if (requested.group == "org.jetbrains.kotlin") {
            useVersion(io.gitlab.arturbosch.detekt.getSupportedKotlinVersion())
        }
    }
}

With that the project is fixed.
So instead of using configurations.all { I should use configurations.matching { it.name == "detekt" }.all {
I think (maybe it's just me) the documentation on https://detekt.dev/docs/gettingstarted/gradle/#dependencies should be updated to precise that configuration as an example, or I need to learn how to read better.
Thanks anyway for your help, and for this project.

I think (maybe it's just me) the documentation on https://detekt.dev/docs/gettingstarted/gradle/#dependencies should be updated to precise that configuration as an example, or I need to learn how to read better.

Can you provide a PR fixing the documentation?

for me with spring boot project and build script plugins this
configurations.matching { it.name.contains("detekt") }.all {     resolutionStrategy.eachDependency {         if (requested.group == "org.jetbrains.kotlin") {             useVersion(io.gitlab.arturbosch.detekt.getSupportedKotlinVersion())         }     } }
does not work.
Still detekt was compiled with Kotlin 1.9.23 but is currently running with 2.0.0.


Failed to start gremlin container

Gremlin container failed to start when I run command like below.
docker-compose -f ./entc/integration/compose/docker-compose.yaml up --build gremlin

I think the settings are not correct to the latest image of tinkerpop/gremlin-server
If ent is intended to work with Amazon Neptune, fixing the gremlin version 3.4.11 would be better.
Because Amazon Neptune latest engine version still supports 3.4.11.
https://docs.aws.amazon.com/neptune/latest/userguide/engine-releases-<IP_ADDRESS>.html
And in version 3.4.11, both docker and test code work well.

[x] The issue is present in the latest release.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior 😯

I got errors.
gremlin_1     | [ERROR] GremlinServer - Configuration file at conf/gremlin-server.yaml could not be found or parsed properly. [Cannot create property=scriptEvaluationTimeout for JavaBean=org.apache.tinkerpop.gremlin.server.Settings@42e99e4a
gremlin_1     |  in 'reader', line 18, column 1:
gremlin_1     |     host: <IP_ADDRESS>
gremlin_1     |     ^
gremlin_1     | Unable to find property 'scriptEvaluationTimeout' on class: org.apache.tinkerpop.gremlin.server.Settings
gremlin_1     |  in 'reader', line 20, column 26:
gremlin_1     |     scriptEvaluationTimeout: 30000
gremlin_1     |                              ^
gremlin_1     | ]
compose_gremlin_1 exited with code 0

Expected Behavior 🤔

Gremlin container can run.
Your Environment 🌎

Tech
Version

Go
go version

Ent
master(6e48129b2974c1bfa14d4efcd182ed5bcc5c7256)

Thanks for sending a patch for this @seiichi1101
@alexsn, would you mind adding me to the entgo org at Dockerhub? Thanks 🙏

@alexsn, would you mind adding me to the entgo org at Dockerhub? Thanks 🙏
sure, what's your docker id?

Any updates?
@a8m, are you still going to use Docker Hub?

@alexsn, would you mind adding me to the entgo org at Dockerhub? Thanks 🙏

sure, what's your docker id?

mashraki. Thanks @alexsn

@alexsn, would you mind adding me to the entgo org at Dockerhub? Thanks 🙏

sure, what's your docker id?

mashraki. Thanks @alexsn

Added.

I updated the image in docker-hub to be 3.4.11

dockerhub entgo/gremlin-server
All that is left now is just to update the integration test (update the dockerfile in the integration test).
@seiichi1101 can you please update your PR just to update that?
Thanks.

Thanks @yonidavidson, @alexsn, and @seiichi1101 🙏


Add cache arguments of renderer.new

Codecov Report

Merging #9 (33f7cb2) into master (77c87ae) will decrease coverage by 0.25%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master       #9      +/-   ##
==========================================
- Coverage   98.79%   98.54%   -0.26%
==========================================
  Files           8        8
  Lines         416      413       -3
==========================================
- Hits          411      407       -4
- Misses          5        6       +1

Flag
Coverage Δ

unittests
98.54% <100.00%> (-0.26%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

lib/renderer.lua
100.00% <100.00%> (ø)

lib/global.lua
95.65% <0.00%> (-4.35%)
:arrow_down:

lib/install.lua
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77c87ae...33f7cb2. Read the comment docs.


adding noise to the uncond images

have you experimented adding a little noise to the zeroed tensors?
I made a few tests with the Plus model and the results are... interesting. Basically instead of a zeroed embed I'm passing a random +/- 0.5 noise (or even higher).
This is a quick example, sometimes the result is quite better, you just need to keep it low otherwise it starts "burning" the image.

wondering if this makes any sense

@cubiq hi, I think this is reasonable, you can think of it as negative image prompt.
For the text-to-image models, we trained them with randomly dropping text (set text=""). In the inference stage, we can set uncond text = "", but in fact, we can use a non-empty string, which is negative prompt.
For the IP-Adapter, we trained it with randomly dropping image (set image to a zero). Hence, we also can use a non-zero image as uncond image in the text stage.

I did what you suggested @xiaohu2015 basically sending a negative image instead of a "zero" one and it works as expected. It's pretty impressive.
In the picture below sending a black and white negative image you get a very saturated result, by sending a very saturated RGB image you get a very desaturated result.
Need to play a little more with it. The base model doesn't seem influenced by what image is actually inside the negative image but just by the colors/saturation/brightness.


data: Hacktinium

Changes proposed
Create LinkFree JSON file. Add basic info & links.
Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[ ] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[x] All new and existing tests passed.
[x] This PR does not contain plagiarized content.
[x] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

Thanks everyone! 😄


Remove duplicate paras from GORM insecure example

Fixes #4140.
Summary of changes:

Delete an extra header and include that had snuck into the "insecure"
example section.  Looks like good old-fashioned human error.

This change is

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/7f68ef45874f4a4764680614c5eb4d454e73e687/


ISSUE#317 Refactor CreationTagsScreen Tags selection

created tests + commented some parts of the code

resolved in PR #301


Added requirement for more recent versions of electron

This change is required for newer versions of electron. See issue 94 of the electron/remote repo.

LGTM


Limit DropDownButton's length

Connects-To: #410
Change-Type: patch
---- Autogenerated Waffleboard Connection: Connects to #410

Just noticed that the updated snapshot belongs to NavBar

@dimitrisnl  Oh thats weird, yeah, discard it please

@LucianBuzzo The thing is that every time I run jest --updateSnapshot I get the same changes.
Could you confirm it if you have some spare time?

@dimitrisnl I can't replicate the issue you're seeing, perhaps you need to run rm -rf node_modules && npm install? In the meantime I'm happy to merge this PR.


bypass 403 IIS

Is there any way to bypass 403? I have used many tools but it did not work. Any tip?

The shortscan technique does work on 403 pages but it wont "bypass" them whatever that's supposed to mean. In the case of the screenshot the 403 happens because directory listing is not enabled.

As @meme-lord says this error is a standard message when directory listing is disabled. The best you can do is to run the tool against the directory and see what it turns up.


Setup VM

[x] Move installation from Tomer's user on the VM to \opt\mehdie\gui
[x] Make sure the website and the google cloud proxy load when the server is restarted
[x] Make sure it is accessible from anywhere on port 80
[ ] After works well, replace the domain mapping on google cloud from our current cloud-run service to this VM.

@kalycoding I can't access the website at https://<IP_ADDRESS>/ are you sure its accessible from anywhere?

Why are all the images not showing?


fix panic when guess isn't 5 letters

when entering anything that isn't 5 letters long, the program calls panic!(). this could be fixed by using Vec::get(), which returns an Option.
instead, it should let the player try again without counting it as an attempt.

Fixed in commit 3c83eaf


GUVNOR-2503: Asset screen Validate-button should use incremental builder

See https://issues.jboss.org/browse/GUVNOR-2503 and https://github.com/droolsjbpm/kie-wb-common/pull/421

There are five PRs all together:-

https://github.com/droolsjbpm/drools/pull/873
https://github.com/droolsjbpm/guvnor/pull/346
https://github.com/droolsjbpm/kie-wb-common/pull/421
https://github.com/droolsjbpm/drools-wb/pull/265
https://github.com/droolsjbpm/optaplanner-wb/pull/89

@mcimbora Could you please look at this for @karreiro?

@manstis @karreiro +1 This looks good to me.


[Bug] Not copying directories containing spaces

As title says.
Also, I tried to use exclude to avoid copying .github/workflows/ but that also doesn't seem to work.
Otherwise this is an awesome action!

Hi! Glad you like the action ☺
Unfortunately i am very bussy at the moment, so I will come back to you in a couple of weeks or so.
If you want, please open a pull request fixing the bug. Contributions are much appreciated!

interested in this issue

Sorry for the delay, I have been extremely busy 😅 However, I hope to be able to fix this bug shortly!
It seems that the problem is that some of the variables regarding path names are not double quoted. This results in directories (and maybe files) with spaces to not be copied over. Adding double quotations around these variables should resolve the issue.

#42 I did a pull request to solve the problem

Great 👌
The pull request is now merged, and a new release v3.2.2 is published.


Fix build warnings

In release mode, 2 warnings could lead to errors if compile flags
-Werror=unused-but-set-variable and -Werror=unused-function are enabled
Signed-off-by: Phong LE<EMAIL_ADDRESS>Issue #, if available:
if compile flags -Werror=unused-but-set-variable and -Werror=unused-function are enabled
Description of changes:
Remove static in unused function and void use for set but not used variable
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hi @gilsken Thanks for reaching out. Our policy is to not accept pull requests, instead make changes internally so that they can be tested by our QA. I will open an internal ticket to get this fixed.