Fix CI on i686-unknown-linux-musl

I suspect focal-20210119 breaks the build, uses an older image to fix the Could not configure 'libc6:i386' failure.

r? @Amanieu
(rust-highfive has picked a reviewer for you, use r? to override)

r? @Amanieu
(rust-highfive has picked a reviewer for you, use r? to override)

@bors r+ p=11

@bors r+ p=11

:pushpin: Commit 464ccfb357f7a4faccf504a215811e51bcfc19fa has been approved by JohnTitor

:pushpin: Commit 464ccfb357f7a4faccf504a215811e51bcfc19fa has been approved by JohnTitor

:evergreen_tree: The tree is currently closed for pull requests below priority 10. This pull request will be tested once the tree is reopened.

:evergreen_tree: The tree is currently closed for pull requests below priority 10. This pull request will be tested once the tree is reopened.

:hourglass: Testing commit 464ccfb357f7a4faccf504a215811e51bcfc19fa with merge 397a33d79ff106b0867472e77787c2e2af0c7a84...

:hourglass: Testing commit 464ccfb357f7a4faccf504a215811e51bcfc19fa with merge 397a33d79ff106b0867472e77787c2e2af0c7a84...

I believe ubuntu has dropped 32-bit x86 support.

I believe ubuntu has dropped 32-bit x86 support.

Hm then some other i686 targets may fail as well, let's check...

Hm then some other i686 targets may fail as well, let's check...

:broken_heart: Test failed - checks-actions

:broken_heart: Test failed - checks-actions

@bors r+

@bors r+

:pushpin: Commit faa0184ad2cf3ac0351dac23a639669d15e6ba1b has been approved by JohnTitor

:pushpin: Commit faa0184ad2cf3ac0351dac23a639669d15e6ba1b has been approved by JohnTitor

:hourglass: Testing commit faa0184ad2cf3ac0351dac23a639669d15e6ba1b with merge f7a9029c2c59a2c0bee820c1ec9ae9319f4e7797...

:hourglass: Testing commit faa0184ad2cf3ac0351dac23a639669d15e6ba1b with merge f7a9029c2c59a2c0bee820c1ec9ae9319f4e7797...

:sunny: Test successful - checks-actions, checks-cirrus-freebsd-11, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Approved by: JohnTitor
Pushing f7a9029c2c59a2c0bee820c1ec9ae9319f4e7797 to master...

:sunny: Test successful - checks-actions, checks-cirrus-freebsd-11, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Approved by: JohnTitor
Pushing f7a9029c2c59a2c0bee820c1ec9ae9319f4e7797 to master...


add database versioning and migration to docs

creating this issue as a placeholder / temporary reference
from gitter channel:
version is a number representing the current database version.
onVersionUpdate is a function that is passed the current version and should return the migrated version.
 nSQL().createDatabase({
    id: “my-db”,
    tables: […],
    version: 3,
    onVersionUpdate: (prevVersion) => {
         return new Promise((res, rej) => {
             switch(prevVersion) {
                 case 1:
                     // migrate v1 to v2
                    res(2);
                    break;
                 case 2:
                     // migrate v2 to v3
                     res(3);
                     break;
             }

         });

     }

also thinking that it might be more useful to have versioning for tables (rather than the top-level databases) since table data is what gets migrated
that's at least what I need for my project since I'm autoloading most of the logic based on table entities, and so I was planning to do the same for versioning/migration - but I imagine this could be built on top of the existing setup so going to play around with it and see

Migration docs are live on this page: https://nanosql.io/databases.html#creating-a-database


Prevent failed pings from filling logs

Currently every single failed ping posts a full exception with stack trace which clogs logs up. My fault x.x

Not entirely as there is also an excessive number of pings for... reasons #266 does help reduce this a bit


FakeStreamingListLLM.astream() yields strings while ChatOpenAI yields AIMessageChunk

System Info
langchain==0.0.352
langchain-community==0.0.6
langchain-core==0.1.3

Python 3.12.1 running inside Docker from python:3.12-bookworm on Linux.
Who can help?
I have a FastAPI app that streams the output of an LLM. The app uses langchain.chat_models.ChatOpenAI at runtime, but during test I mock the LLM with langchain.llms.fake.FakeStreamingListLLM. However, when my app calls .astream() on each of them I'm getting different results:

While ChatOpenAI yields instances of AIMessageChunk...
FakeStreamingListLLM yields instances of strings.

This mismatch between the fake and the real class makes the fake unsuitable for some mocking purposes as the interface of AIMessageChunk is different from a str.
@baskaryan @agola11
Information

[ ] The official example notebooks/scripts
[X] My own modified scripts

Related Components

[X] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[X] Async

Reproduction
See an snippet below on how to reproduce the output of strings within ipython:
from langchain.llms.fake import FakeStreamingListLLM
import asyncio

async def pp(llm):
    async for chunk in llm.astream('some input'):
        yield(f"{type(chunk)}: {chunk}")

async def consume():
    results = []
    async for item in pp(FakeStreamingListLLM(responses=['a'])):
        results.append(item)
    return results

asyncio.run(consume())

# OUTPUT:
["<class 'str'>: a"]

Expected behavior
I'd expect FakeStreamingListLLM to return instances of AIMessageChunk as the real chat model.

try:
import asyncio

- from langchain.llms.fake import FakeStreamingListLLM
+ from langchain_community.chat_models import FakeListChatModel

async def pp(llm):
    async for chunk in llm.astream('some input'):
        yield (f"{type(chunk)}: {chunk}")

async def consume():
    results = []
-    async for item in pp(FakeStreamingListLLM(responses=['a'])):
+    async for item in pp(FakeListChatModel(responses=['a'])):
        print(item, type(item))
        results.append(item)
    return results

asyncio.run(consume())

That worked perfectly @chyroc . Than you very much!
I don't know if you intend to close this issue as there is a working solution, or do you intend to leave it open to update the behavior ofFakeStreamingListLLM. Feel free to close it whenever you like.


Fix links in README

Somehow, we screwed up the org name in all these URLs! :facepalm:
Fixes #79.

Thanks for the review! Merging now.


Kong ingress controller user defined volume mounts

Hello,
The kong.controller-container helper function is missing the capability to mount the user defined volume mounts, where we are able to inject the user defined volumes.
We need that capability to secure the provisioning of the  CONTROLLER_KONG_ADMIN_TOKEN using the Vault injector agent.
this would imply updating the helper function from this
  {{- if .Values.ingressController.admissionWebhook.enabled }}
  volumeMounts:
  - name: webhook-cert
    mountPath: /admission-webhook
    readOnly: true
  {{- end }}

to this
  volumeMounts:
  {{- include "kong.userDefinedVolumeMounts" . | nindent 2 }}
  {{- if .Values.ingressController.admissionWebhook.enabled }}
  - name: webhook-cert
    mountPath: /admission-webhook
    readOnly: true
  {{- end }}

Any concerns / thoughts about us making a PR in that direction ? We have validated the use case ?
Thanks.

Handled in https://github.com/Kong/charts/pull/560


Entangleporter Improvements

Implements #4503, #3846, and half of what #3953 wants.
As for #3953 I've not changed the amount by which tiles emit to pipes, but have increased the buffer capacity to the same as the ultimate tiers of cable/pipe/tube can handle, so in theory transmitters can pull at their full rate, on one side of the porter at least.
Frequency storage in the itemstack stores only the name & public/private flag, so private frequencies should attach themselves to the placing player (e.g. should it be different to the frequency's).
Anchor upgrade is copied from the teleporter - loads just the tile's chunk.

Great work, thanks!


Add an option to skip files with no changes

https://github.com/linksplatform/RegularExpressions.Transformer/blob/6598f6a314ff98eb742c685eee952ee6d6f58476/csharp/Platform.RegularExpressions.Transformer/TransformersListExtensions.cs#L24

Closed by https://github.com/linksplatform/RegularExpressions.Transformer/commit/0384ddcaa7f4617e1aae67984202926df9358b57


Make dashboard links more visible

The dashboard links are too subtle or lacking text.

I really think this should be a button, or something. And consistent everywhere. The button can be used both on individual list elements as well as on the pages themselves.


{Storage} Bump azure-mgmt-storage to api version 2023-01-01

This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

Storage


Image not working?

HI, thank you for this great project! Currently I'm using your api for my project. Now I wanto to get the image of a product, but this isnt working:
app.get('/image/:imageid.:extension', cors(corsOptions), async (req, res) => {
  const imageID = req.params.imageid;
  const extension = req.params.extension;
  const image = await picnicClient.getImage(imageID, ImageSizes.LARGE);
  // const base64Image = new Buffer.from(image.toString('binary'), 'base64')
  // res.writeHead(200, {
  //    'Content-Type': `image/${extension}`,
  //    'Content-Length': image.length
  //  });

  res.set('Content-Type', `image/${extension}`);
  res.set('Content-Length', image.length);
  res.set('Image-ID', `image/${extension}`);

  return res.end(image);
})

In postman I only get this:

Am I doing it wrong?

Correct, the api sends a request with some incorrect headers. This needs to be fixed, which I will do soon, thanks for reporting. 👍
If you want a temporary fix you can send a request like this: 'https://storefront-prod.nl.picnicinternational.com/static/images/IMAGE_ID/SIZE.png'.
Example: 'https://storefront-prod.nl.picnicinternational.com/static/images/8560e1f1c2d2811dfefbbb2342ef0d95250533f2131416aca459bde35d73e901/medium.png'
This is basically what the api does as well.

Thank you for investigation! Actually I load them by the url manually like you proposed :)

Got distracted, but fixed it just now in v1.4.2. 👍


Errors when a file has been deleted

When a file is staged for deletion (git rm $file), then rubocop -r rubocop_lineup issues the following error instead of running:
undefined method `+' for nil:NilClass
/Users/jamie/code/vpay/eft3/vendor/bundle/ruby/2.3.0/gems/rubocop_lineup-0.4.0/lib/rubocop_lineup/line_number_calculator.rb:11:in `git_line_summary_to_numbers'`

This also appears to be the case after committing the deleted file, so it's of ongoing relevance.

Thx @jamie - sorry just now seeing this.
I thought we were appropriately ignoring deletions ... but ... we'll need to take another look under the hood.


Is it possible to combine minihttp with anything else in the same loop?

I can't just call tokio_minihttp::serve for that because I don't have the access to Core this way. And I can't copy the code from tokio_minihttp::Server::serve because there are lots of private stuff.
Am I missing something obvious?

No... minihttp is only example code :) I didn't feel like adding all the details for configuring the reactor that it runs on, etc...


Install fails while parsing go.mod

Observed behavior
Following the install instructions results in an error while parsing go.mod
Steps to replicate

run go install github.com/google/gnostic@latest
run go install github.com/google/gnostic-grpc@latest

result
go install github.com/google/gnostic-grpc@latest
go install: github.com/google/gnostic-grpc@latest: <EMAIL_ADDRESS>parsing go.mod:
	module declares its path as: github.com/googleapis/gnostic-grpc
	        but was required as: github.com/google/gnostic-grpc

Possible resolution
I was able to install gnostic-grpc using go install github.com/googleapis/gnostic-grpc@latest

Is this still an issue?

It's still an issue for me also.

Still an issue.
Solution suggested above works though

Possible resolution
I was able to install gnostic-grpc using go install github.com/googleapis/gnostic-grpc@latest


Fix and improve parallel mode

Motivation
Parallel mode was crashing with bad memory access due to data races accessing Frontend.configurationLoader, more specifically its cache.
After serializing access (initially with an NSLock), I observed that despite not crashing anymore, the performance was surprisingly worst than in single threaded mode.
That led me down a road of investigating why this was happening, and after some profiling I discovered that Rule's nameCache was the main source of contention - causing around 30% of total spent time waiting for locks (ulock_wait) on my tests.
After replacing the synchronization mechanism on nameCache with a more efficient os_unfair_lock_t (pthread_mutex_t in Linux), the contention dropped significantly, and parallel mode now outperformed single threaded mode as expected. As a bonus, these changes also improved single threaded mode performance as well, due to the reduced overhead of using a lock vs a queue.
I then used the same Lock approach to serialize access to Frontend.configurationLoader which increased the performance gap even further.
After these improvements, I was able to obtain quite significant performance gains:

serial (non optimized) vs parallel (optimized): ~5.4x (13.5s vs 74s)
serial (optimized) vs serial (non optimized): ~1.6x (44s vs 74s)
serial (optimized) vs parallel (optimized): ~3.2x (13.5s vs 44s)

Tests were made on my MacBookPro16,1 (8-core i9 @2.4GHz), on a project with 2135 Swift files, compiling swift-format in Release mode.
Update 1
Sadly, a custom Lock implementation is not ideal for swift-format to maintain and Windows support was not provided. As such, The alternative would be to use NSLock which being a part of Foundation is supported on all major platforms.
Using NSLock the improvements were not so good, unfortunately:

serial (non optimized) vs parallel (NSLock): ~1,9x (38s vs 74s)
serial (NSLock) vs serial (non optimized): ~1,4x (52s vs 74s)
serial (NSLock) vs parallel (NSLock): ~1,3x (38s vs 52s)

Update 2
The best solution was then to try and remove all concurrency contention from swift-format. By flattening the FileToProcess array and making the Rule's nameCache static, the best overall results were achieved:

serial (non optimized) vs parallel (optimized): ~6.7x (11s vs 74s)
serial (optimized) vs serial (non optimized): ~1.8x (40.6s vs 74s)
serial (optimized) vs parallel (optimized): ~3.6x (11s vs 40.6s)

In order to have both Rule.ruleName caching and avoid synchronization, a static cache of rule names is now generated via the generate-pipeline tool.
To make the rule types available to the Rule.ruleName protocol extension, it was moved to SwiftFormatRules target, together with SyntaxFormatRule and SyntaxLintRule.
Update 3
Base rules SyntaxFormatRule and SyntaxLintRule should be kept in the SwiftFormatCore target to eventually support dynamic discovery of rules (ditching generate-pipeline) and only require rule implementers to import SwiftFormatCore.
On the other hand, any statically generated ruleName cache must be in SwiftFormatRules target, because  that's where rule implementations reside to capture their ObjectIdentifier at compile time.
So, in order to have a static rule name cache that's accessible from SwiftFormatCore (especially from SyntaxFormatRule), a solution was to inject a ruleNameCache dictionary in Context, which is injected into every Rule.
This allows using generate-pipelines to a produce a ruleNameCache in SwiftFormatRules that's injected on all Contexts by each of the pipelines (which is done in SwiftFormat target).
While not as ideal as having the cached accessed in a single place (this approach incurs in the cache being copied for each Context), it achieves a good tradeoff given current constraints). Lets hope the compiler is smart enough to optimize this somehow.
All things considered, results were quite good:

serial (non optimized) vs parallel (optimized): ~7x (10.5s vs 74s)
serial (optimized) vs serial (non optimized): ~1.7x (42.5s vs 74s)
serial (optimized) vs parallel (optimized): ~4x (10.5s vs 42.5s)

Changes

Make file preparation serial by calculating all FileToProcess before processing them in parallel.

Move SyntaxFormatRule and SyntaxLintRule to SwiftFormatRules target.

Add new RuleNameCacheGenerator to generate-pipeline which outputs a static Rule name cache to RuleNameCache+Generated.swift file in SwiftFormatRule target containing:

The Rule.ruleName protocol extension.
A static nameCache dictionary containing all possible rules.

Benchmarks
Below are benchmarks of some configurations attempted during development, using hyperfine.
(Some of these configurations' implementations are available in branches on my fork).
The flattened FileToProcess array was an attempt to remove the need to synchronize access to Frontend.configurationLoader completely, by making initial file preparation sequential. It results in a slight performance improvement in single threaded mode (~3.6%), but a performance loss in parallel mode (~4.6%). Opted for maximizing parallel mode's performance, as that should be the more optimized mode.

In order to have both Rule.ruleName caching and avoid synchronization, a static cache of rule names is now generated via the generate-pipeline tool. To make the rule types available to the Rule.ruleName protocol extension, it was moved to SwiftFormatRules target, together with SyntaxFormatRule and SyntaxLintRule.
I think this achieves quite a good tradeoff, and doesn't mess too much with project organization. SyntaxFormatRule and SyntaxLintRule are rules after all 😅

Almost. They're not concrete rules; they're the base types that other rules should extend. It's important that we keep those two types in the SwiftFormatCore module and separate from the SwiftFormatRules module that implements concrete rules, because eventually we'd like to be able to support dynamic discovery of rules and get rid of generate-pipeline completely. That would let folks write their own rules as plug-ins where they would import SwiftFormatCore, and we'd also factor out the built-in rules in the same fashion.
It may be some time before we have the runtime metadata APIs we need to make that a reality, but I'd like to keep the design with that in mind, otherwise we'd just have to back the changes out later anyway.
I don't mind the generation of the cache itself, but can you make it work without moving those types around?

Almost. They're not concrete rules; they're the base types that other rules should extend.

Yes, I am aware that they are base classes to the other rules (hence placing them in a directory called Base), but didn't consider the future plans of having dynamic discovery of rules without generate-pipeline.

It may be some time before we have the runtime metadata APIs we need to make that a reality, but I'd like to keep the design with that in mind, otherwise we'd just have to back the changes out later anyway.

It would quite a trivial change to do, but it's a valid point nonetheless.

I don't mind the generation of the cache itself, but can you make it work without moving those types around?

I can definitely try. But there's both a circular dependency and a dynamic dispatch problem as things are, which makes this complicated:

SyntaxFormatRule, SyntaxLintRule and Rule live in SwiftFormatCore
All current rule implementations live in SwiftFormatRules, which depends on SwiftFormatCore
generating the cache as it is requires that the types are visible to the compiler at build time, so need to be in SwiftFormatRules (because that's where rule implementations are)
We can't import SwiftFormatRules (to access the codegen'ed cache) in Rule.swift because it would create a circular dependency
As they are, SyntaxFormatRule, SyntaxLintRule's ruleName invocation will always use dynamic dispatch and use the protocol extension implementation in Rule.swift.

These were the reasons why I ended up moving those two base classes and the Rule.ruleName protocol extension to SwiftFormatRules, and why I consider it an elegant solution given the constraints.
To work around issue in point 5 a solution would be to implement ruleName in base classes and override in subclasses. I've attempted to do so using codegen but it would require classes to use ObjC runtime to allow static vars to be declared in extensions, which then creates other issues related with SyntaxRewriter and SyntaxVisitor already being base classes.
The absolute best approach for this performance-wise would be for rules to simply declare a static let ruleName: String = "Foo" abandoning the protocol extension completely, which although well intended and nicely abstracted, given the project and code structure is causing a significant performance penalty.
I'll give it yet another go to see what I can come up with. Do you have any idea of what could work?

Well, I have yet another possible solution, which hopefully will tick all the necessary boxes 😅
So, in order to have a static rule name cache that's accessible from SwiftFormatCore (especially from SyntaxFormatRule), a solution was to inject a ruleNameCache dictionary in Context, which is injected into every Rule.
This allows using generate-pipelines to a produce a ruleNameCache in SwiftFormatRules that's injected on all Contexts by each of the pipelines (which is done in SwiftFormat target).
While not as ideal as having the cached accessed in a single place (this approach incurs in the cache being copied for each Context), it achieves a good tradeoff given current constraints). Lets hope the compiler is smart enough to optimize this somehow.
All things considered, results were quite good nonetheless:

serial (non optimized) vs parallel (optimized): ~7x (10.5s vs 74s)
serial (optimized) vs serial (non optimized): ~1.7x (42.5s vs 74s)
serial (optimized) vs parallel (optimized): ~4x (10.5s vs 42.5s)

flattened FileToProcess array, static ruleNameCache injected in Context
branch: swift-5.4-branch-rule-name-cache-in-context
serial
Benchmark #1: ./swift-format format --configuration .swift-format -r -i Sources
  Time (mean ± σ):     43.524 s ±  1.003 s    [User: 41.400 s, System: 1.676 s]
  Range (min … max):   42.648 s … 45.514 s    10 runs

Benchmark #2: ./swift-format format --configuration .swift-format -r -i Sources
  Time (mean ± σ):     41.646 s ±  0.714 s    [User: 39.650 s, System: 1.610 s]
  Range (min … max):   40.381 s … 42.525 s    10 runs

parallel
Benchmark #1: ./swift-format format --configuration .swift-format -r -i -p Sources
  Time (mean ± σ):     10.088 s ±  1.160 s    [User: 108.481 s, System: 3.700 s]
  Range (min … max):    8.441 s … 11.655 s    10 runs

Benchmark #2: ./swift-format format --configuration .swift-format -r -i -p Sources
  Time (mean ± σ):     11.207 s ±  0.735 s    [User: 118.658 s, System: 4.291 s]
  Range (min … max):    9.315 s … 11.871 s    10 runs

So, in order to have a static rule name cache that's accessible from SwiftFormatCore (especially from SyntaxFormatRule), a solution was to inject a ruleNameCache dictionary in Context, which is injected into every Rule.
This allows using generate-pipelines to a produce a ruleNameCache in SwiftFormatRules that's injected on all Contexts by each of the pipelines (which is done in SwiftFormat target).

This feels like a good way to do it! I can also see it scaling well to support dynamic discovery whenever we're able to do that, because we'd inject a different cache into the context instead of mutating some global singleton.
It looks like there are still some outdated changes from the previous push in this PR (the old generated cache file is still there along with the new one, and the lint/format rule base types are still in the SwiftFormatRules module instead of back in SwiftFormatCore). Can you fix those up please?

While not as ideal as having the cached accessed in a single place (this approach incurs in the cache being copied for each Context), it achieves a good tradeoff given current constraints). Lets hope the compiler is smart enough to optimize this somehow.

I don't think the cache will be copied here; the dictionary is never mutated after it's created, so creating a new Context will just store a Dictionary value that shares the same underlying storage with the original. That should be relatively fast.

This feels like a good way to do it! I can also see it scaling well to support dynamic discovery whenever we're able to do that, because we'd inject a different cache into the context instead of mutating some global singleton.

That's great news!

It looks like there are still some outdated changes from the previous push in this PR (the old generated cache file is still there along with the new one, and the lint/format rule base types are still in the SwiftFormatRules module instead of back in SwiftFormatCore). Can you fix those up please?

Whoops 🙈  Cherry-pick fail. Will fix right away

I don't think the cache will be copied here; the dictionary is never mutated after it's created, so creating a new Context will just store a Dictionary value that shares the same underlying storage with the original. That should be relatively fast.

Yes, that makes sense. COW's got our backs here 😁

We're almost there! I ran the tests locally and realized that the old hardcoded list of rules suppressed from RuleCollector in generate-pipeline means they won't be in the name cache either, and the tests for those rules crash 🙃
I've got #263 out to fix this, which should unblock this change.

Nice catch, and thanks for the quick fix! 🙏🏼
If I'm not mistaken, that was caused by the new assert in Context.isRuleEnabled, right?. I added it as a reminder to favor using the cache and not rely on the Rule.ruleName fallback which will use the (slow) protocol extension 😅. I can remove it if you believe it doesn't add value and is prone to creating more "friction" rather than help.

I think it's fine to have that assertion; if a rule's name isn't in the cache, it's probably a sign of other problems (like generate-pipeline not being run after adding something new), and falling back to the slower implementation would lead to slower performance for those rules. I think it's better not to risk that kind of inconsistency.
I merged #263, so if you rebase and rerun generate-pipeline again, the cache should pick those up.

I think it's fine to have that assertion; if a rule's name isn't in the cache, it's probably a sign of other problems (like generate-pipeline not being run after adding something new), and falling back to the slower implementation would lead to slower performance for those rules. I think it's better not to risk that kind of inconsistency.

Yeah, that was precisely my thought process! 😄

I merged #263, so if you rebase and rerun generate-pipeline again, the cache should pick those up.

Indeed it did! Pushing rebased and updated changes.


Language server keeps waiting

Description
The python language server is recognized and seems to be running, but it keeps saying "Waiting..." even if the kernel is not busy,

This is what appears in the developer tool console

Reproduce
I have no idea how to reproduce this bug.
I just installed the packages from the arch linux official repository (arch does not allow installing packages globally using pip).
P.S. I would like not to create a virtual environment and install a text editor + lsp server + formatter + extensions + libraries for every folder with a python script. So I don't know how to deal with this

Expected behavior
The LSP should run
Context

Operating System and version: Arch linux (kernel version 6.5.9)
Browser and version: Firefox 119.0
JupyterLab version:

$ jupyter --version
Selected Jupyter core packages...
IPython          : 8.17.2
ipykernel        : 6.26.0
ipywidgets       : 8.1.1
jupyter_client   : 8.6.0
jupyter_core     : 5.5.0
jupyter_server   : 2.10.0
jupyterlab       : 4.0.8
nbclient         : 0.8.0
nbconvert        : 7.7.4
nbformat         : 5.9.2
notebook         : not installed
qtconsole        : not installed
traitlets        : 5.9.0

Language server and version: pylsp v1.8.0

Required: installed server extensions

Paste the output from running `jupyter server extension list` (JupyterLab >= 3)
or `jupyter serverextension list` (JupyterLab < 3) from the command line here.
You may want to sanitize the paths in the output.

Config dir: /home/**************/.jupyter

Config dir: /usr/etc/jupyter

Config dir: /usr/local/etc/jupyter

Required: installed lab extensions

Paste the output from running `jupyter labextension list` from the command line here.
You may want to sanitize the paths in the output.

JupyterLab v4.0.8
/usr/share/jupyter/labextensions
        jupyterlab_pygments v0.2.2 enabled  X (python, jupyterlab_pygments)
        jupyter-matplotlib v0.11.3 enabled OK
        @jupyter-lsp/jupyterlab-lsp v4.2.0 enabled OK (python, jupyterlab-lsp)
        @jupyter-widgets/jupyterlab-manager v5.0.9 enabled OK (python, jupyterlab_widgets)

   The following extensions are outdated:
        jupyterlab_pygments

   Consider checking if an update is available for these packages.

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output (recommended for all interface issues)

Paste the output from your browser JavaScript console replacing the text in here.
To learn how to open the developer tools in your browser:
https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_are_browser_developer_tools#How_to_open_the_devtools_in_your_browser
If too many messages accumulated after many hours of working in JupyterLab,
consider refreshing the window and then reproducing the bug to reduce the noise in the logs.

I have the same issue on archlinux 6.7.8. Have you solved it yet?

Anyone managed to solve it ?


ide glitches when using names under "inverted apices":

if you have code under   the ide glitches and sometimes you have to restart
Unit tests sometimes are more readable allowing spaces in their names,
so you may want to use   to allow this, having tests like
let two plus two must be equal to four() =
but something may go wrong when you try to do it
To Reproduce
in a project using a unit test library like NUnit
create a test like:
[<Test>]
let ``two plus two must be equal to four``() =
   Assert.AreEqual(4,2+3)

Expected behaviour
everything should be fine, and you should be able to continue
working. But Vs code sometimes stop working
and you have to restart Vs code.
Environment (please complete the following information):

OS: Windows 10 pro
Ionide version 4.17
VSCode version: 1.49.1
dotnet SDK version: 3.1.100
mono / .Net Framework version:  4.8.03752

Additional context
That was true also for previous versions of Ionide under Mac os X using Mono.

Probably related to https://github.com/ionide/ionide-vscode-fsharp/issues/1073

Duplicate of #1073


Manual cherry-pick (#41)

Manual sync:

open-cluster-management-io/governance-policy-framework-addon#41

/override "Pipelines as Code CI / governance-policy-framework-addon-dfsa-on-pull-request"

/override "SonarCloud Code Analysis"


Can you bump version to include read-preference fix?

I'm currently using https://github.com/nicolaiarocci/eve and it requires flask-pymongo 0.3.0.  However, I can't use READ_PREFERENCE because the latest merge with the fix is after 0.3.0.

0.3.1 is about to be released to the CheeseShop.


will litedb v5 support query Thenby in the future?

hello david
Just i used collection.Query().OrderBy(),but i found there's no Thenby supported now,My application target machine has poor memory,so i really like love now litedb can use linq OrderBy on disk,I hope litedb will support Thenby,thanks for your time.

Hi, for my first release there is no plans yet for ThenBy. If you have all fields in same order, you can sort a new array:
.OrderBy(“[Name,Age]”)

This .OrderBy(x => new object[] { x.Name, x.Age }) seems to work as well, for those who prefer expression syntax. But I am still scared to face a scenario where I would need to order two columns in different order. 😨


mpd: fix 0.23.8 on darwin

0.23.8 includes https://github.com/MusicPlayerDaemon/MPD/commit/c975d8b94316c86bf5950ed3abeba394e1263677
said change fixes deprecation warnings introduced in the 12.0 sdk, using definitions from the 12.0 sdk
nixpkgs uses 10.12 and 11.0, neither of which are the 12.0 sdk
this will gracefully degrade into a no-op when that changes

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[X] aarch64-darwin

[X] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

✓

Result of nixpkgs-review pr 182299 run on x86_64-darwin 1

2 packages built:

mpd
mpd-small

It builds for me on Darwin, but I cannot approve since I am a bit lost about stdenv.hostPlatform.darwinSdkVersion (if it should be targetPlatform or hostPlatform), since no other package seems to use it.

targetPlatform seems to be appropriate going by https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/xcbuild/wrapper.nix

Would it be possible to backport this patch to nixpkgs-22.05-darwin?
Right now the mpd package is broken in that branch


Vulkan mod doesn't work in Prismlauncher on AMD due to not finding vulkan-loader

Describe the bug
If you try to use Vulkan mod in Minecraft it will fail to open, as the package seems to expect the game to only need OpenGL I assume at least. I think to fix this the package vulkan-loader needs to be included in the build. Since this is a mod it doesn't effect the base game, so adding support for this might be debatable but I would like to see it work to compare the renderers and try RT mods that uses hardware RT.
Steps To Reproduce
Steps to reproduce the behavior:

Download Vulkan mod
Try to run the game
See that it crashes and cannot find vulkan-loader

Expected behavior
The game loads with Vulkan mod installed
Additional context
The error:
[22:16:17] [Render thread/WARN]: Failed to create window:
net.minecraft.class_1041$class_4716: GLFW error 65542: Vulkan: Loader not found
	at net.minecraft.class_1041.method_4501(class_1041.java:196) ~[client-intermediary.jar:?]
	at org.lwjgl.glfw.GLFWErrorCallbackI.callback(GLFWErrorCallbackI.java:43) ~[lwjgl-glfw-3.3.1.jar:?]
	at org.lwjgl.system.JNI.invokePP(Native Method) ~[lwjgl-3.3.1.jar:?]
	at org.lwjgl.glfw.GLFWVulkan.nglfwGetRequiredInstanceExtensions(GLFWVulkan.java:117) ~[lwjgl-glfw-3.3.1.jar:?]
	at org.lwjgl.glfw.GLFWVulkan.glfwGetRequiredInstanceExtensions(GLFWVulkan.java:150) ~[lwjgl-glfw-3.3.1.jar:?]
	at net.vulkanmod.vulkan.Vulkan.getRequiredExtensions(Vulkan.java:425) ~[VulkanMod_1.20.1-0.3.9.jar:?]
	at net.vulkanmod.vulkan.Vulkan.createInstance(Vulkan.java:239) ~[VulkanMod_1.20.1-0.3.9.jar:?]
	at net.vulkanmod.vulkan.Vulkan.initVulkan(Vulkan.java:138) ~[VulkanMod_1.20.1-0.3.9.jar:?]
	at net.vulkanmod.vulkan.VRenderSystem.initRenderer(VRenderSystem.java:58) ~[VulkanMod_1.20.1-0.3.9.jar:?]
	at com.mojang.blaze3d.systems.RenderSystem.initRenderer(RenderSystem.java:1536) ~[client-intermediary.jar:?]
	at net.minecraft.class_310.<init>(class_310.java:505) ~[client-intermediary.jar:?]
	at net.minecraft.client.main.Main.main(Main.java:211) ~[minecraft-1.20.1-client.jar:?]
	at net.fabricmc.loader.impl.game.minecraft.MinecraftGameProvider.launch(MinecraftGameProvider.java:470) ~[fabric-loader-0.15.7.jar:?]
	at net.fabricmc.loader.impl.launch.knot.Knot.launch(Knot.java:74) ~[fabric-loader-0.15.7.jar:?]
	at net.fabricmc.loader.impl.launch.knot.KnotClient.main(KnotClient.java:23) ~[fabric-loader-0.15.7.jar:?]
	at org.prismlauncher.launcher.impl.StandardLauncher.launch(StandardLauncher.java:87) ~[NewLaunch.jar:?]
	at org.prismlauncher.EntryPoint.listen(EntryPoint.java:129) ~[NewLaunch.jar:?]
	at org.prismlauncher.EntryPoint.main(EntryPoint.java:70) ~[NewLaunch.jar:?]
Process exited with code 0.

Tried this as well but didn't work due to a build error, likely user error on my part
{pkgs, ...}: {
  nixpkgs.overlays = [
    (_final: prev: {
      prismlauncher = prev.prismlauncher.override ({extraLibraries ? _pkgs': [], ...}: {
        extraLibraries = pkgs':
          (extraLibraries pkgs')
          ++ [
            pkgs'.vulkan-loader
          ];
      });
    })
  ];

  environment.systemPackages = with pkgs; [
    prismlauncher
  ];
}

Also, I imagine that this effects Multimc so it's package should be updated as well. I have also tested on the unwrapped package, same issue there as well :/
Notify maintainers

@minion3665 @Scrumplex @getchoo
Metadata
~
❯ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.8.5, NixOS, 24.05 (Uakari), 24.05.20240410.1042fd8`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"home-manager, nixos"`
 - nixpkgs: `/nix/store/r1g09wfd968168f6drdl7agfaf3msik4-source`

Add a :+1: reaction to issues you find important.

First of all, to add additional libraries to the game's search path, you need to override additionalLibs.
prev.prismlauncher.override {
  additionalLibs = with final; [vulkan-loader];
}

After I did that, I was getting a SIGSEGV from libwayland-client though.
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007b7c82abd820, pid=213589, tid=213599
#
# JRE version: OpenJDK Runtime Environment (17.0.7+7) (build 17.0.7+7-nixos)
# Java VM: OpenJDK 64-Bit Server VM (17.0.7+7-nixos, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libwayland-client.so.0+0x8820]  wl_proxy_get_version+0x0
#
# Core dump will be written. Default location: Core dumps may be processed with "/nix/store/2skwm7ilmd4xd0sb9r5x6bmvl05vbhd2-systemd/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/scrumplex/.local/share/PrismLauncher/instances/vkmod/.minecraft/core.213589)
#
# An error report file with more information is saved as:
# /home/scrumplex/.local/share/PrismLauncher/instances/vkmod/.minecraft/hs_err_pid213589.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

Mod loads for me with the following
{ pkgs, ... }:

{
    environment.systemPackages = with pkgs; [
        (prismlauncher.override { additionalLibs = [ vulkan-loader ]; })
    ];
}

This works for me too, I guess this can be closed in favor of the MR


Generate signatures for functions

Now there is no signatures. It harms usability in IPython shell much.
Here is some PoC code:
from jpype import java

import jpype
import jpype.imports

from pathlib import Path

scalaLibsPath=Path(".").absolute() / "scala-2.12.6" / "lib"
kotlinLibsPath=Path(".").absolute() / "kotlin-compiler-1.2.60" / "kotlinc" / "lib"

classPaths=[]
classPaths.extend(( str((scalaLibsPath/("scala-"+n+".jar")).absolute())  for n in ("library", "reflect") ))
classPaths.extend(( str(p.absolute())  for p in kotlinLibsPath.glob("kotlin-stdlib*.jar") ))
classPaths.extend(( str(p.absolute())  for p in kotlinLibsPath.glob("kotlin-*runtime*.jar") ))
classPaths.extend(( str(p.absolute())  for p in kotlinLibsPath.glob("kotlin-*reflect*.jar") ))
classPathz=";".join(classPaths)

if not jpype.isJVMStarted():
	jpype.startJVM(jpype.getDefaultJVMPath(), "-ea", "-Djava.class.path="+classPathz)

jpype.imports.registerDomain("jpio", alias="io")
jpype.imports.registerDomain("scala")
jpype.imports.registerDomain("kotlin")

import re
import ast
from enum import IntFlag
from enum import _decompose as decomposeEnum
from collections import defaultdict
from jpype import reflect
import scala.reflect
import scala.reflect.runtime.currentMirror as cm
import scala.reflect.runtime.universe._
import scala.reflect.runtime.universe.runtimeMirror
#import astor

import java.lang.reflect.Modifier

class JavaParamModifier(IntFlag):
	final=java.lang.reflect.Modifier.FINAL
	synthetic=java.lang.reflect.Modifier.SYNTHETIC
	implicit=java.lang.reflect.Modifier.MANDATED
	mandated=java.lang.reflect.Modifier.MANDATED

class JVMLang(IntFlag):
	java = 0
	scala = 1
	kotlin = 2

from typing import T, Generic

class JNIModifiers:
	class final(Generic[T]):
		pass
	class synthetic(Generic[T]):
		pass
	class implicit(Generic[T]):
		pass

incorrectCharRx=re.compile("\W")

def prepareName(name):
	return incorrectCharRx.sub("_", name)

scalaReflectionPresent=None
try:
	scala=jpype.JPackage("scala")
	jun=scala.reflect.runtime.package.universe()
	ttCtor=jun.TypeTag()
	def ja(o):
		return list(scala.collection.JavaConverters.asJavaCollection(o))
	scalaReflectionPresent=True
except:
	scalaReflectionPresent=False

class Param:
	__slots__=("name", "typeName", "default", "modifiers")
	def __init__(self, name, typeName, modifiers, default=None, annotations=None):
		self.name=name
		self.typeName=typeName
		self.default=default
		self.modifiers=modifiers
	def __repr__(self):
		return " ".join((str(self.modifiers), self.typeName, self.name))

builtinRemap={
	java.lang.String: str,
	scala.Int: int
}

def findMethodInJavaClass(cls, methName):
	#global meth
	meth=None
	try:
		meth=cls.class_.getMethod(meth)
	except:
		for m in cls.class_.methods:
			print(m.name, methName, str(m.name) == methName)
			if str(m.name) == methName:
				meth=m
				break
	return meth

class LanguageReflector:
	markerAnnotationType = None

	@classmethod
	def check(cls, origClass):
		signs=origClass.getAnnotationsByType(cls.markerAnnotationType)
		return signs
	pass

class ScalaReflector(LanguageReflector):
	markerAnnotationType=scala.reflect.ScalaSignature
	def unpickle(bytes:JNIModifiers.final[bytes], offset:JNIModifiers.final[int], classRoot:JNIModifiers.final, moduleRoot:JNIModifiers.final, filename:JNIModifiers.final[str]):
		return scala.reflect.internal.pickling.UnPickler.unpickle(bytes, offset, classRoot, moduleRoot, filename)

	def reflect(self, origClass, methName, signs):
		print("it's a scala class")
		#scala.reflect.internal.pickling.UnPickler.unpickle(signs[0].bytes, 0, None, None, None)
		#unpickle(bytes: Array[Byte], offset: Int, classRoot: Symbol, moduleRoot: Symbol, filename: String)
		#unpickle(bytes:__main__.JNIModifiers.final, offset:__main__.JNIModifiers.final, classRoot:__main__.JNIModifiers.final, moduleRoot:__main__.JNIModifiers.final, filename:__main__.JNIModifiers.final, **kwargs)
		#unpickle(bytes, offset, classRoot, moduleRoot, filename)
		#__class__.unpickle(signs[0].bytes, 0, None, None, "")

languageReflectors={
	JVMLang.scala: ScalaReflector
}

class ReflectedMethod():
	__slots__ = ("params", "returnType", "language", "wrapper", "name", "prepName", "javaName", "meth", "cls")

	def __init__(self, cls, methName):
		meth=None
		self.cls=cls
		self.name=methName
		self.meth=getattr(cls, self.name)

		self.prepName=prepareName(methName)
		self.javaName="_"+self.prepName # original java method will use that name to avoid name collision and infinite recursion
		self.params=None
		self.wrapper=None
		self.language=JVMLang.java

		print(cls.class_)
		print(dir(cls.class_))
		annotations=list(cls.class_.declaredAnnotations)

		meth=findMethodInJavaClass(cls, methName)

		if meth is None:
			raise Exception("Method `"+methName+"` not found")

		origClass=meth.declaringClass

		print("origClass.name", origClass.name, origClass, origClass == cls.class_)

		print("return type:", meth.getReturnType(), meth.getGenericReturnType())

		self.returnType=meth.getReturnType().name

		import java.lang.reflect

		paramz=[]
		for p in java.lang.reflect.Method.getParameters(meth):
			paramz.append(Param(p.name, p.type.name, p.getModifiers()))
		annotations=[]
		self.params=paramz

		for langId, langRefl in languageReflectors.items():
			signs=langRefl.check(origClass)
			if signs:
				self.language=langId
				langRefl.reflect(self, origClass, meth.name, signs)
				break

		#self.__call__=self.toWrapped()

	def typeNameToAST(typeName, invertedJDomains, imports):
		from jpype.imports import _JDOMAINS
		typeName=typeName.split(".")
		if typeName[0] in invertedJDomains:
			typeName[0]=invertedJDomains[typeName[0]]
		if typeName[0] in _JDOMAINS:
			imports.append(ast.Import(names=[ast.alias(name=".".join(typeName), asname=None)]))
		else:
			imports.append(ast.Assign(targets=[ast.Name(id=typeName[0], ctx=ast.Store())], value=ast.Call(func=ast.Name(id='JPackage', ctx=ast.Load()), args=[ast.Str(s=typeName[0])], keywords=[])))

		hint=ast.Name(id = typeName[0], ctx = ast.Load())
		for part in typeName[1:]:
			hint=ast.Attribute(
				value = hint,
				attr = part,
				ctx = ast.Load()
			)
		return hint

	def makeWrapper(self):
		from jpype.imports import _JDOMAINS
		invertedJDomains={v:k for k, v in _JDOMAINS.items() if k!=v}

		repeatedNames=defaultdict(int)
		argNamez=[ ]

		for p in self.params:
			argName=prepareName(p.name)
			repeatedNames[argName]+=1
			if repeatedNames[argName]>1:
				argName+="_"+str(repeatedNames[argName])
			argNamez.append(argName)

		defPosArgz = []
		imports = []
		for i, p in enumerate(self.params):
			hintAttr=self.__class__.typeNameToAST(p.typeName, invertedJDomains, imports)

			for mod in decomposeEnum(JavaParamModifier, p.modifiers)[0]:
				if mod:
					hintAttr=ast.Subscript(
						value=ast.Attribute(
							value=ast.Name(id="JNIModifiers", ctx=ast.Load()),
							attr=mod.name, ctx=ast.Load()
						),
						slice=ast.Index(value=hintAttr), ctx=ast.Load()
					)

			defPosArgz.append(ast.arg(arg = argNamez[i], annotation = hintAttr))
			#, list(p.annotations), JavaParamModifier(p.getModifiers()),  p.isNamePresent()))

		#defValuez=[ast.NameConstant(value = None)]*len(defPosArgz)
		defValuez=[]

		callPosArgz=[ast.Name(id = nm, ctx = ast.Load()) for nm in argNamez]
		#callPosArgz.append(Starred(value = Name(id = 'args', ctx = ast.Load()), ctx = ast.Load()))

		code=imports
		code.append(ast.FunctionDef(name = self.prepName,
			args = ast.arguments(
					args =defPosArgz,
					#vararg = arg(arg = 'args', annotation = None),
					vararg = None,
					kwonlyargs = [],
					kw_defaults = [],
					kwarg = ast.arg(arg = 'kwargs', annotation = None),
					defaults = defValuez),
			body = [
				ast.Return(
					value = ast.Call(
						func = ast.Name(id = self.javaName, ctx = ast.Load()),
						args = callPosArgz,
						keywords = [
							ast.keyword(arg = None, value = ast.Name(id = 'kwargs', ctx = ast.Load()))
						]
					)
				)
			],
			decorator_list = [],
			returns = self.__class__.typeNameToAST(self.returnType, invertedJDomains, imports)
		))
		return ast.Module(body=code)

	def toWrapped(self):
		mast=self.makeWrapper()
		ast.fix_missing_locations(mast)
		#print(ast.dump(mast))
		import astor
		print(astor.to_source(mast))
		print(repr(self.language))
		compiled=compile(mast, filename=self.language.name+" method <"+self.name+">", mode="exec")

		from jpype import java, JPackage

		resEnv={"__builtins__":{"__import__":__import__}, "java":java, "JPackage":JPackage, self.javaName:self.meth, "JNIModifiers":JNIModifiers}
		exec(compiled, resEnv, resEnv)
		return resEnv[self.prepName]

	def __call__(self):
		pass

def toWrapped(cls, methName):
	return ReflectedMethod(cls, methName).toWrapped()

toWrapped(scala.collection.mutable.Map, "copyToArray")
toWrapped(scala.Int, "toString")
#toWrapped(java.lang.String, "length")

ToDo:

Scala reflection, arguments default values and names
test on kotlin-generated jars
now method reflection is not optimal: we reflect whole class and search for methods.
ctors are not reflected

I am guessing this is to make it so that java methods will have a valid signature for autocomplete.  Though this is a worthy goal I am a bit worried about how this will interact when we have multiple method overloads.  It may be possible to put this on the bound method of the _jpype but I don't know enough about how python and function definitions interact such that ipython works.  I have seen some cases that outright crash ipython so we should fix this eventually.

@Thrameos, I have updated the source, I guess it's ready to being integrated into jpype. Unfortunately I haven't managed make it work, it seems that C++ part needs some modifications to use it.

I am doing a major refactorization of the C++ pieces.  The C++ is currently overbuilt mess with two of just about everything.  I am not sure what your tolerance level for seeking endlessly for car keys, but I will warn you that that code is about as convoluted as it can be and still function. It is partly the inevitable collision between concepts when two languages are being bridged using a third meaning naming of concepts are randomly selected depending on the mindset of the author, partly the hopelessness of supporting python 2/3 at the same time, partly the need to wrap all interfaces that so that there is a single point of maintenance should a need arise, and last trying to be a larger framework that was ever fully realized.  It sort of gives the feel of walking into a large half finished Cathedral that was turned into a makeshift hospital,  But I am rambling...
Hopefully it will be a bit easier to add support of this when the layer gets flattened and simplified.

Now the state is unusable, added some unfinished reflection for scala.

Is this one resolved?

It's not, I will improve it better somewhen when I have more time. Java part works (except output type, I have not fixed getting it yet), scala part doesn't, and the design is not optimal. As you see I have tried scala reflection, unfortunately it won't give me parameters default args (it will, but they are generated by methods, and I guess these methods are generated by scala runtime, so we won't get the code generating default arguments) and I have not managed to get all the methods via scala reflection. Also it will require us to bundle scala reflection lib. I guess I should try to deserialize ScalaSignature (presence of this marks the class as scala one) and look into it. So I guess you can start integrating this, but keep in mind what I have said.

I meant is there any action required in terms of jpype changes to get you the calls you need to access the reflection pieces?  I thought that you would using the class_ but I wanted to be sure.
As far as the scala bundling it is likely best if you do this the same way that jpype.imports works.   Create the support hooks for scala you need in a jpype.scala module that is not included in the core module inputs.   You can place any package independent hook points in the main code, but keep those that required the other module separate.  You can add support methods to classes dynamically using a customizer hook.  Then just like jpype imports if the user wants those addition hooks installed you just need to import that module at the top of the code.  Add a "try: import scala; except: error("this module requires scala" to the top of the module so that it will only import properly if the required resources are available.

I meant is there any action required in terms of jpype changes to get you the calls you need to access the reflection pieces? I thought that you would using the class_ but I wanted to be sure.

Yes, I'll try to use it, for now it seems to be enough for getting the info I need.

Then just like jpype imports if the user wants those addition hooks installed you just need to import that module at the top of the code. Add a "try: import scala; except: error("this module requires scala" to the top of the module so that it will only import properly if the required resources are available.

I guess the thing should try to initialize all the hooks. I mean let we have a jar importing some other jars. Let some of them be written in scala, some of them in kotlin, some of them in another jvm lang. And we wanna auto-completion and helt for everything without explicitly enabling help for scala and kotlin. So the approach may be the following: have plugins for each language and try to initialize them all, if lang's runtime is not present, its plugin won't be initialized and it is not fatal (there will be a little bit less convenience), but if the runtime is present in classpath, I guess we can assumme that some module needs it, so I guess we can expect seeng some classes in that language.
Wnd I guess that I may need some hooks in the things creating python counterpart for jvm methods.
I see 2 alternative ways:
1 the class calls my hooks after it have created the python method, and it passes me the method it has created. And my hooks wraps that method into own compiled from generated AST.
2 the class calls my hook before it has created the python method, my hook reflects the method signature, creates the data structure with method signature like (friendlyName, return type, ((argument, type, defaultValue), ....), some other metadata, docstring)), types are according PEP 484, and jpype native code creates a builtin with the needed signature.
How do you think it's better to implement that?

Well the question is what is the purpose of the mod.  If it is to be included in the jpype distribution as a requirement then I would leave it to the maintainer to decide if it is worth adding the requirement.  If it is plugin that adds capabilities that makes it easier because then it just becomes an option.  If this is something entirely separate from jpype distribution then do it however you want.
For example, I have a full set of routines that plugs my math library into array classes using customizers.  It adds a whole bunch of new methods to double[], float[] and int[] so that I can do stuff like numpy without the cost of pulling the arrays from java to python.  This is a huge savings for my uses because I have massive arrays stored in java that would take forever if I were to convert them back and forth as I manipulate.  But to make use of it in jpype I would need to add a big jar of math routines.  As there is no provision to store or add jars to jpype so I haven't contributed it.    There is a provision to add thunks (individual support java classes).  Thus I could add some level of support by creating a thunk building it into the python module and having it install.  But as I am not sure anyone would use that math library it hasn't been worth it.
I don't have any experience with the tools you are describing so I cant really tell what the scope of this.
I am guessing that you want to add a method to classes to give back a method signature.  That is clearly something that can be done with a class customizer.  However, I am guessing that making that signature is required a bunch of reflection libraries (most likely that you are only using one of two methods from).  In that case, we would either have to require those libraries, or make it an optional module.  If you are using only one or two functions from a library and you can easily write those routines yourself, then if gets much simpler.  We just add those functions as the thunk and place the appropriate call hooks into jp_jni and expose the functions as part of JPMethod, JPClass, JPArrayClass.   It is also possible like the forName that we don't need to change the module at all and we can just use the thunk from python.  But I would need to have a more clear of what is happening before I could give input.  I suppose I should try to read the PEP.

I should point out my general goal with this project is to strip the C++ layer to the absolute bare minimum to support the python and push all stuff from the C++ layer to python or if it must be present into a thunk where it is safely contained in a virtual machine.  Everything that goes wrong here is in the C++ where there is no recovery but segfaulting.  I am not opposed to new functionality, but hopefully that functionality is in the python side where it is much easier to debug and for others to contribute.

Hmm.  I read the PEP which seems to be about how to declare type info.  But what is not clear to me is what is the mechanics of the system.  What method do we need to provide such that a JPMethod will be callable?
class Foo(object):
     method = _jpype.JavaMethod(bar)

f=Foo()
f.method()  <-- calls into the module
__XXX__(f.method) --> give the signature?
f.method.__XXX__()  --> or is it this?

what is XXX in the PEP? Sorry if am asking a silly question.  I am a java programmer so python is sometimes a mystery to me.

Well the question is what is the purpose of the mod. If it is to be included in the jpype distribution as a requirement then I would leave it to the maintainer to decide if it is worth adding the requirement.

I guess this is not hard requirement - without reflection libs the methods will be reflected as java methods. So the code doing reflection for these langs using langs reflection libs still should be a part of jpype.

What method do we need to provide such that a JPMethod will be callable?

I don't understand what you mean, but I try to guess that you mean the following
class Foo:
    def __call__(self):
        print("bar")
a=Foo()
a() # prints "bar"

I mean something like
#inside jpype
def signatureHook(cls, methName):
  ...
_jpype.registerSignatureHook(signatureHook)

from java import Something
# the hook is called by c++ code in jpype internally for every method in Something, then control is resulted to jpype, then jpype does everything needed, the resulting object has all the methods with signatures

a=Something.something() # the same, I guess

If I monkey-patched classes replacing JavaMethods builtins with my own wrappers it crashes the interpreter. Also my wrappers are not JavaMethods, so cannot be used the way they are used. I can try to write a proxy class, but I'm sure it will crash the interpreter too. Could you fix that?

I mean I am not sure how signature hooks work on regular classes enough to know how to make this work.  If I have the method __call__  is the signature hook on the class or the method, or someplace else entirely.  If I have a simple example where you can show me how to monkey patch an existing method to have a signature I should be able to tell you where to put it.
class Foo(obj):
   def myMethod(x):
       return 1

applyMonkeyPatch(Foo)  <-- make myMethod into (str) -> int

I just don't know the mechanics of what you are doing in the general sense.   So if you can show me what should be in applyMonkeyPatch to make the signature work, then the rest will be easy.
The JavaMethod object is pretty minimal and does not implement a __dict__ so you can't monkey patch it currently.  It is possible to add a dictionary with a few commands in the C++.  The JavaClass is all completely reworkable using customizers as we never expose the _jpype version.  You can even intercept all methods to your own and proxy to the java ones pretty easily by customizing the __getattribute__  The only complexity there is the mess of having __class__, __metaclass__ and instance.   I am thinking long term it would be much better to have the java class and object instances both use __metaclass__ so that there is no way to mess with the raw dictionary which tends to break the whole works.
Internally, I have a general ability to add python resources with _jpype.setResource.  So I just need to know what is the resource name and where it needs to appear in terms of fields, methods, and bound methods.  We also need to make it safe such if the resource is not set it gives a safe landing.    Thus
we can certainly have a _jpype.setResource('methodSignature', usermodule.getMethodSignature())

@Thrameos I have updated the code. It's a draft. Works for any jvm-based language. Haven't managed to extract default functions values though.

Do you have a good description of what the python function signature should look like on the python side?  We have full signatures available from java.  I just don't understand how to have a python function with more than one argument specification?
Is there are PEP that give an example of the resulting python function?  If there is, then we can populate it pretty easily in jpype 0.7 given existing meta data.

Do you have a good description of what the python function signature should look like on the python side?
https://www.python.org/dev/peps/pep-0484/
https://www.python.org/dev/peps/pep-0483/
https://docs.python.org/3/library/typing.html

If there is, then we can populate it pretty easily in jpype 0.7 given existing meta data.

The problem is when overloads have different arguments or returns.
void func1( int i )
void func1( int i, String s)
int    func1( String s)

would be close to impossible to specify.  Thus it is best if we only add this for non-overloaded in the first cut.
I do agree that if a method has only one overload that it would be fairly trivial to generate a signature.  I will just need to see how the specification is applied to the actual python function.
In jpype version 0.7, the methods both bound and unbound are stored directly in the class structure.  We would need to add a signature hook which would call a piece of python code to create the signature meta data and cache it.

The problem is when overloads have different arguments or returns.

yes, it is, that's what I meant when I have said that this is limited.

Thus it is best if we only add this for non-overloaded in the first cut.

I agree.
Another way to solve it is to disalow positional args for these functions and use only keyword args, but this is not suitable for functions with variable argumet count.

Okay so if that is the case we would need to modify PyJPMethod to have a getattr() looking for __annotation__ request.  It would then need to get to check the method overload list and see if it has only one overload.  If there is one overload then it would return a dictionary with contents arg# and return.  If it has multiple it would return the type as being variable arguments with just generic object.  If all the return types of the overloads match it can return that type otherwise it would specify a generic object.
It is likely best if we use a python hook rather than trying to code this in C++.  It would be a pain to debug and very inflexible otherwise.  We use setResource to define hooks for things like setting up classes and creating objects.  Thus the first step would be to add a resource hook and the getattr that intercepts the request for __annotation__.   We will need to convert the method id into a list of java.reflect.Method.
I am not sure how often this code will get hit.  If it gets hit often we may want to set up a cache like the toString in PyJPField.   The hook can then scan the overloads and produce the required dict.
I am on vacation this week and wont be able to look into it deeply.  We should work from the devel branch as that has a cleaned up code base.  The only problem I see if we need to make sure it doesn't trigger the __get__ request otherwise the logic detecting attempts to call a method without an object will trigger an exception.

I'm not sure that __annotation__ is everything we need add arguments to a function.

It is likely best if we use a python hook rather than trying to code this in C++. It would be a pain to debug and very inflexible otherwise.

:+1: We also need to make sure that the decision on what whether the method should be reflected is done on python side. This way we can add overloaded methods later. And we also need a better API to reflect a specific method and all its overloads than the one used in findMethodInJavaClass.

I am not sure how often this code will get hit. If it gets hit often we may want to set up a cache like the toString in PyJPField. The hook can then scan the overloads and produce the required dict.

 Does it do everything lazily and reflect a method only on being called or dir() rather than class load?

The reason i suspect that __annotation__ is needed is that when I specify a function with argument specifications then dir the function that is the only information I can find that holds the info.  But I may be mistaken.  I am not a particularly skilled python programmer most of my experience is C++ and Java.
As to the load hook, it isn't all that complex, but I haven't seen much docs on the process.
There appear to be two paths for loading classes in the jpype code.

Classes created explicitly from python.
Classes created implicitly as the result of a return.

The process is as follows.

There is a call to C++ to create a JPClass instance as a result of cache miss in the TypeManager.
The class preloads as much as stuff as it can including all the methods.  See native/common
But it does not actually process those methods yet because processing the methods arguments may trigger more class loads so those are deferred.
Super classes and interfaces are all created.
After all of the C++ elements are created there is a call to create the Python wrapper for the C++ object.
_jpclass.py constructs the class by pulling each element with reflection including the methods and fields.  Each of these class construct a python module object PyJP* under native/python
Customizer are called to add user code to the constructed class.
Python class is returned to user

Load pattern is aggressive rather than lazy because it needs to know all types to expose the proper python methods and base class.  Memory is held in global tables for classes and methods and are connected to the jvm instance.  Objects are owned by the PyJP _jpype module.
Adding the C++ hooks to call a python routine when a specific attribute is request is likely a 1 hour task if one knows the structure of jpype.  The hook needs to be added to the setResource table and added to the globals.  The PyJPModule type definition then has to add something catch the request for attributes not already in a slot.  Then one mixed cpython module call has to pack the required arguments and pass the return back using the jp_pythonenv calls.  Then the python hook must be initialized in the initialized method in _jclass.py.  Then if is just a matter of processing through the list of reflected methods to create the return.  If we need more than just __annotation__, we just add another hook using the same procedure.  It was a bit harder in the released version as you had communicate with HostRef objects.
The only tricky piece that can go wrong is handling the reference counting when talking to cpython.

https://www.python.org/dev/peps/pep-0563/

This one still isn't addressed.  Guess it will move to 0.8 target.

I think the first place to start on this would be to see if we can place an annotation on arrays.  Arrays are much simpler that other classes because they only have one type to return.  Thus we just have to get to the component type and find a way to get it to the annotation.  The tricky part here is that all arrays share the same get/set methods, thus we may have a hard time adding the annotation unless it can be done as a property.
For methods I think that we can use the same system as was used to populate __doc__.  I have already demonstrated I can get that to dynamically allocate.  Again on this one, I think we should focus mainly on returns so that we can at least try to get autocomplete without evaluation to work.

Hmm.  Arrays seem like a very poor choice to start on.  I immediately run into the issue that __annotation__ is a static dictionary rather than a function that operates using the class.  After 3.7 there will be a function to access the type hints rather than getting them directly, but this is still not very flexible.  Also I have serious concerns as to whether this can actually work without interfering with Python2.  The level of type hacking that may be required to overcome the limitations of the static system may make this completely incompatible.

IMHO: since python 3.4 runs even on ancient systems python 2 compatibility should be just dropped.

Though I agree that Python 2.7 should be dropped, I will hold out until the end of service life.
So far my exploration has told me that I really need to contact the author of the type hinting system in Python.  The system is designed entirely for static typing (which seems strange for a dynamic typed language).  They do not let the __self__ instance be passed into get_type_hints nor did they consider that the type list of arguments on a method call could be used to produce a more specific return type.  A few trivial changes could have made this job much easier.
That said I did come up with solutions that we could implement to get the return type determined (not dealing with the argument list for now.)  For methods like those on JArray the difficulty comes from the fact that these are raw Python function descriptors.  As such they don't have any hooks to generate the required attribute dynamically.  It has to be dynamic as the same method is shared with all instances of JArray.  Thus we have two methods we could deal with this, both rather hackish.   We can use a decorator to convert the JArray methods into a wrapper type that proxies everything through to the real function.  When it gets a call to __annotations__ the wrapper intercepts the call and uses __self__ to get the real type of the call where the component type is housed.  Option 2, rather than implementing the methods directly in JArray, we instead use the customizer  system to copy the methods onto the new array class.  It would need to clone the methods so that we have a fresh copy that we can place the annotations on.  Option 1 adds overhead to every method call (slowing the whole thing down), Option 2 adds memory overhead as we now have to clone the functions to add the required annotations.   Option 2 seems like a better choice, but method cloning in Python seems really hackish.
Methods are actually the simpler case.  The dynamic _doc__ method I used can be applied to methods pretty trivially.  We just need to check the PyJPMethod cache to see if there is already an annotations present.  It so then pass it, otherwise call a module defined Python method defined using _jpype.setResource.  That method can then call get methods on the list of java.lang.reflect.Method and see if there is a consensus  on the return types.  If everything is one type (or perhaps one is a void in which case we ignore it), then we can add that to the dictionary we pass back to be cached.   More sophisticated would be to try to find a common type, through I suspect that will not be possible.  Overhead on this is trivial.   The mod requires the following steps...

[ ] Add __annotations__ as a getset property on PyJPMethod that calls PyJPMethod::getAnnotations
[ ] Write PyJPMethod::getAnnotations to consult the cache or call JPPyEnv::getMethodAnnotations
[ ] Add a annotations_cache to the PyJPMethod
[ ] Add the cache to the GC list for PyJPMethod
[ ] Add a setResource to the module in jp_pythonenv.cc
[ ] Add getMethodAnnotations to the module in jp_pythonenv.cc that makes the overload list and calls the python hook.
[ ] Write the Python hook and install it with setResource in jpype/_jclass.py

I implemented the __annotations__ protocol and as far as I can tell ipython never called it.  Do you know how to trigger ipython to use the return annotation?

In ipython shell (or jupyter notebook) you should type
<function you wanna get help on>?

This should show that function signature with annotations, if they are present.

I believe that <function>? uses the __doc__ api and not the __annotations__ api.  Can you test using the __doc__ support pull request and see if it covers what you are looking for?

In [1]: def a(b:int)->int:
   ...:     return b

In [2]: ann=a.__annotations__

In [3]: def a(b):
   ...:     return b

In [4]: a.__annotations__=ann

In [5]: a?
Signature: a(b:int) -> int

Okay.  My testing was with Spyder running iPython which must have a very different concept of how to get that info.   I tried exactly what you had above and it never called the __annotation__ API.  Currently I get the following with the doc_string pull request.
$ ipython3
Python 3.6.3 |Anaconda, Inc.| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import jpype
In [2]: jpype.startJVM()
F:\nelson85pl\nelson85\devel\open\jpype\jpype\_core.py:210: UserWarning:
-------------------------------------------------------------------------------
Deprecated: convertStrings was not specified when starting the JVM. The default
behavior in JPype will be False starting in JPype 0.8. The recommended setting
for new code is convertStrings=False.  The legacy value of True was assumed for
this session. If you are a user of an application that reported this warning,
please file a ticket with the developer.
-------------------------------------------------------------------------------

  """)
In [3]: s = jpype.java.lang.String('foo')
In [4]: s.substring?
Call signature: s.substring(*args, **kwargs)
Type:           PyJPMethod
String form:    java.lang.String.substring
File:           f:\nelson85pl\nelson85\devel\open\jpype\_jpype.cp36-win_amd64.pyd
Docstring:
Java method dispatch 'substring' for 'java.lang.String'

Virtual Methods:
  * java.lang.String substring(int)
  * java.lang.String substring(int, int)

Returns:
  java.lang.String

In [5]:

We can try to get the annotations PR working as well.   But I am guessing that even without the annotations this covers the original need of this issue.

No wait I was right.   No matter what I do it does not appear to use the __annotations__ API.  Instead of printing "Signature: "  it prints "Call Signature:"  meaning it is hitting a different code path entirely.
$ ipython
Python 3.6.3 |Anaconda, Inc.| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import jpype

In [2]: jpype.startJVM()
F:\nelson85pl\nelson85\devel\open\jpype\jpype\_core.py:210: UserWarning:
-------------------------------------------------------------------------------
Deprecated: convertStrings was not specified when starting the JVM. The default
behavior in JPype will be False starting in JPype 0.8. The recommended setting
for new code is convertStrings=False.  The legacy value of True was assumed for
this session. If you are a user of an application that reported this warning,
please file a ticket with the developer.
-------------------------------------------------------------------------------

  """)

In [3]: s=jpype.java.lang.String('foo')

In [4]: s.replaceAll.__annotations__
Out[4]:
{'return': jpype._jclass.java.lang.String,
 'arg0': jpype._jclass.java.lang.String,
 'arg1': jpype._jclass.java.lang.String}

In [5]: s.replaceAll?
Call signature: s.replaceAll(*args, **kwargs)  <===  IT IGNORED THE __annotations__!
Type:           PyJPMethod
String form:    java.lang.String.replaceAll
Docstring:
Java method dispatch 'replaceAll' for 'java.lang.String'

Virtual Methods:
  * java.lang.String replaceAll(java.lang.String, java.lang.String)

Returns:
  java.lang.String

In [6]:

Okay I figured it out.  We are properly adding __doc__ and __annotation__ but we did not complete the contract required to be consider a function
def isfunction(object):
    """Return true if the object is a user-defined function.

    Function objects provide these attributes:
        __doc__         documentation string
        __name__        name with which this function was defined
        __code__        code object containing compiled function bytecode
        __defaults__    tuple of any default values for arguments
        __globals__     global namespace in which this function was defined
        __annotations__ dict of parameter annotations
        __kwdefaults__  dict of keyword only parameters with defaults"""
    return isinstance(object, types.FunctionType)

Not sure how I would ever manage to satisfy __code__ unless I can somehow make a stub code that just turns around and calls the method itself.  I recommend that we close this request as it is fall astray and make a ticket to upgrade PyJPMethod to satisfy the function contract.  It is unfortunately way outside the usual how to guide of Python as googling gives me nothing on how to accomplish this.   It does not appear the this layer is duck typed, but instead requires implementing the full function API if I want it to work.
The __annotations__ API is really a red herring.  It isn't really the issue as nothing will consult it until inspect tells other tools that our type is a function.
So bottom line is we have doc strings and they are respected by ipython "?" .  We can't get "help" or any other command to work until we complete a much larger contract.

But I am guessing that even without the annotations this covers the original need of this issue.

It doesn't. IPython also has autocompleteon. It is also essential.
It should look something like
In [3]: s = jpype.java.lang.String('foo')
In [4]: s.substring? Call signature: s.substring(beginIndex:int, beginIndex:typing.Optional[int]=None)

but better - with all the info from Java and scala world.

replaceAll.__annotations__

It is an object. You probably need replaceAll.__call__.__annotations__

Unfortunately, making __call__ have annotations is a non-starter as it is a Python native concept with no real presence in the current system.   The PyJPMethod can give anything that is required to pretend to be the type using getset.  But as far as I can tell in tracing the code only a "real function" will do because there is no duck typing.  We could add another class which just serves as a holder for PyMethod, but that is a big performance hit for a low value.  Every method takes twice the memory and twice the call overhead just to insert a dictionary.   And we have many hundreds of calls per attempt to access annotations thus the value prospect isn't there.
The other thing is I have been using JPype for several years using autocomplete just fine under Spyder.  It always completes just fine when I hit tab when greedy matching is turned on.  Thus before I can help on this I would need some way to replicate the actual problem.
But here are a few ideas....

First we need to find a way to get annotations out of the system and a way to test its function without interaction.  Trying to start an interactive session and calling "?" or "" is just a horrible way to do development.  And I would never be able to make a test bench, thus I would never know if it was broken.

Assuming we can get the annotation to work, it sounds like you want to install a custom hook in place of getMethodAnnotation that I am putting in place.  I can put that hook on the class wrapper so that you just install your method on the base object type of Scala and everything will go to you to decide how to annotate it.

Example:
@JCustomizer("my.base.scala.Object")
class MyBaseScalaObjectCustomizer(object):
    # Define a custom function which will create the annotations when requested.
    def __jmethod_annotator__(self, method):
         # Implement your own custom method to create annotations.

I can give an api that places a decorator on a class customizer that installs the __annotations__.  It will never be possible for reflection to give the signature s.substring(beginIndex:int, beginIndex:typing.Optional[int]=None) -> str.  There is no way to know the function of substring to the level you are requesting from within reflection.  Thus the best we can do is come up with a way to annotate a method in the customizer that will pass this information onto the real method.

(I am just going to wing it here.)
@JCustomizer("java.lang.String")
class JStringCustomizer(object):
      @JStub  # Mark this routine as a stub which will be used to annotate the real method
      def substring(beginIndex: int, endIndex:typing.Optional[int]=None) -> str:
          """Gets a portion of a string."""
          pass

If you can help me on number one by calling trace or some other function to try to find the actual code that is being used by <tab> it would be much easier for me to be productive in getting you a solution.   I traced down pydoc.document as the source of the documentation issue.  But I have no clue on how <tab> actually does its work.  Just have a PR with some tests that can only be passed by completing what you are looking for would be a big help.
Just one last test
import typing

class B(object):
    def a(begin:int, end:typing.Optional[int]=None) -> str:
        pass

print("doc:",B.a.__doc__)
print("name:",B.a.__name__)
print("code:",B.a.__code__)
print("globals:",B.a.__globals__)
print("defaults:",B.a.__defaults__)
print("annotations:",B.a.__annotations__)
print("kwdefaults:",B.a.__kwdefaults__)

Gives...
doc: None
name: a
code: <code object a at 0x0000000001F28B70, file "testAnn.py", line 4>
globals: {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x0000000001F2E3C8>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': 'testAnn.py', '__cached__': None, 'typing': <module 'typing' from 'C:\\ProgramData\\Miniconda3\\lib\\typing.py'>, 'B': <class '__main__.B'>}
defaults: (None,)
annotations: {'begin': <class 'int'>, 'end': typing.Union[int, NoneType], 'return': <class 'str'>}
kwdefaults: None

Oh another weird impediment detail that I noticed in the doc strings dive.  We are naming our classes with names like "java.lang.String" in module "jpype._jclass" (by default), but many portions of Python are taking our name and making it "String" in module "jpype._jclass.java.lang" which when they look for the module comes up with nothing which causes Sphinx and other tools to bust.  I tried making our module "java.lang" with name "String" which actually works just fine when jpype.imports is active, but then it tended to print only "String" on the error messages which I find very confusing.    We may run into this problem any place in which they don't consult the __module__ directly.

Another search for relevant PEP:

PEP 3107
PEP 0484 previously noted.
PEP 0526 May apply to fields/properties

I am sorry you are incorrect.  The __annotations__ API is not used by IPython for completions.  Here is a simple example.  We create a method with a signature call IPython completion and we get the same
garbage output that we when we try to complete from a JPype method.
$ ipython
Python 3.7.3 (default, Mar 26 2019, 01:59:45)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import IPython

In [2]: ip=IPython.get_ipython()

In [3]: def a(s:str)->str:
   ...:     pass
   ...:

In [4]: print(ip.Completer.all_completions('a("foo").'))
['a("foo")..bumpversion.cfg', 'a("foo")..convert.py.swp', 'a("foo")..eggs/', 'a("foo")..git/', 'a("foo")..gitattributes', 'a("foo")..gitignore', 'a("foo")..log.swp', 'a("foo")..pytest_cache/', 'a("foo")..trace.txt.swp', 'a("foo")..travis.yml']

In [5]: a?
Signature: a(s: str) -> str
Docstring: <no docstring>
File:      /mnt/devel/open/jpype/<ipython-input-1-be6273b5667b>
Type:      function

The problem is with the default configuration of IPython.
Using the spider configuration I get...
In {1]: import jpype
In [2]: jpype.startJVM(convertStrings=False)
In [3]: s=jpype.java.lang.String("foo")
In [4]: import IPython
In [5]: ip=IPython.get_ipython()
In [6]: print(ip.Completer.all_completions('s.substring(1).'))
['s.substring(1).CASE_INSENSITIVE_ORDER', 's.substring(1).charAt', 's.substring(1).chars', 's.substring(1).class_', 's.substring(1).codePointAt', 's.substring(1).codePointBefore', 's.substring(1).codePointCount', 's.substring(1).codePoints', 's.substring(1).compare', 's.substring(1).compareTo', 's.substring(1).compareToIgnoreCase', 's.substring(1).concat', 's.substring(1).contains', 's.substring(1).contentEquals', 's.substring(1).copyValueOf', 's.substring(1).endsWith', 's.substring(1).equals', 's.substring(1).equalsIgnoreCase', 's.substring(1).format', 's.substring(1).getBytes', 's.substring(1).getChars', 's.substring(1).getClass', 's.substring(1).hashCode', 's.substring(1).indexOf', 's.substring(1).intern', 's.substring(1).isBlank', 's.substring(1).isEmpty', 's.substring(1).join', 's.substring(1).lastIndexOf', 's.substring(1).length', 's.substring(1).lines', 's.substring(1).matches', 's.substring(1).notify', 's.substring(1).notifyAll', 's.substring(1).offsetByCodePoints', 's.substring(1).regionMatches', 's.substring(1).repeat', 's.substring(1).replace', 's.substring(1).replaceAll', 's.substring(1).replaceFirst', 's.substring(1).split', 's.substring(1).startsWith', 's.substring(1).strip', 's.substring(1).stripLeading', 's.substring(1).stripTrailing', 's.substring(1).subSequence', 's.substring(1).substring', 's.substring(1).toCharArray', 's.substring(1).toLowerCase', 's.substring(1).toString', 's.substring(1).toUpperCase', 's.substring(1).trim', 's.substring(1).valueOf', 's.substring(1).wait']

The problem is not with JPype.  It is with the configuration of IPython.

The problem is a bug in jedi.   If you can find the API that is required for jedi to follow through return types I will give this another shot.  Until then this will have to be wont fix as it is not our bug.
Relevant link

Trying to start an interactive session and calling "?" or "" is just a horrible way to do development. And I would never be able to make a test bench, thus I would never know if it was broken.

inspect.signature

Assuming we can get the annotation to work, it sounds like you want to install a custom hook in place of getMethodAnnotation that I am putting in place. I can put that hook on the class wrapper so that you just install your method on the base object type of Scala and everything will go to you to decide how to annotate it.

Not quite that. I wanna a mechanism allowing me to wrap methods (including magic ones like init) and/or classes in python code.

It will never be possible for reflection to give the signature s.substring(beginIndex:int, beginIndex:typing.Optional[int]=None) -> str.

I doubt that.

These kind of overloads often folow the pattern

methodName(Arg1T arg1) -> Rt
methodName(Arg1T arg1, Arg2T arg2) -> Rt
methodName(Arg1T arg1, Arg2T arg2, Arg3T arg3) -> Rt
methodName(Arg1T arg1, Arg2T arg2, Arg3T arg3, Arg4T arg4) -> Rt

We can detect that pattern
and generate a wrapper automatically:
def methodName(Arg1T arg1, typing.Optional[Arg2T] arg2=None, typing.Optional[Arg3T] arg3=None, typing.Optional[Arg4T] arg4=None) -> Rt:
	if arg4 is not None:
		return wrappedMethod(arg1, arg2, arg3, arg4)
	elif arg3 is not None:
		return wrappedMethod(arg1, arg2, arg3)
	elif arg2 is not None:
		return wrappedMethod(arg1, arg2)
	else:
		return wrappedMethod(arg1)

For the cases like
methodName(Arg1T arg1) -> Rt
methodName(Arg2T arg2, Arg1T arg1) -> Rt

the wrapper can be
def methodName(*, Arg1T arg1, typing.Optional[Arg2T] arg2=None) -> Rt:
	if arg2 is not None:
		return wrappedMethod(arg2, arg1)
	else:
		return wrappedMethod(arg1)

* requires all the args be specified via keywords
For the cases like
methodName(Arg1T arg1) -> Rt
methodName(Arg2T arg2) -> Rt

the wrapper can be
def methodName(*, typing.Optional[Arg1T] arg1, typing.Optional[Arg2T] arg2=None) -> Rt:
	assert arg1 is None or arg2 is None, "Incorrect signature, valid signatures are ..." # optimization will cut it, if we don't need it be cut, we can use if
	if arg2 is not None:
		return wrappedMethod(arg2)
	else:
		return wrappedMethod(arg1)

All of these is automatable.

I am sorry you are incorrect. The annotations API is not used by IPython for completions.

Yes, it is not used for completions. But function signature is. I mean if you put cursor into braces and hit tab, you will get arguments names autocompleteon.

From my prospective the function signature is the __annotations__ API.  There is no other programmatic hook to supply that information.
I implemented every possible attribute and even managed to fool python into calling our methods python functions... nothing.  I added a tracer to our methods that caught every single attribute call that was user accessible and got... not one single hit.  I ran a trace file following the entire process from start to end grepping for calls into jpype... only hit once on a __doc__ request.  I then grepped the trace for every use of inspect and __annotations__ as well as every keyword that you have pointed me towards....  Not a single hit that was relevant (just found uses of ismodule).  So I have now spent 12 hours on this and am frustrated and tired, and I am no closer to being able to generate this very reasonable feature request.  I have tried dutifully to explore each option you have suggested.  But the process does not appear to match what you think it is.  Thus my only conclusion is that you are mistaken about the process.
After all that exploring, I do have a clear picture of what is going on.   The IPython tab completion engine is jedi.  The point of jedi is to read minds (or so they say).  Thus rather than using the Python __annotatations__ API or any other user definable API, they instead do everything indirectly.  That is jedi will open the source file of the method is declared in and try and find the def foo(s:str)->str: line and the doc string.  Which is why in their documentation they specify that the type declaration must appear on one line.   Or they can read it from one of four python documentation formats, but they would only do so under certain situations none of which I can currently trigger.  That is because it doesn't ask for the doc string, instead it scans the file trying to find the docstring.  Feel free to read the section.   It would not be the first time I have been mistake about an API, but I gave it the very best college try.
The problem with this approach is no matter how clever we are in generating signatures, it will never be called because jedi is unwilling to actually ask us what we will be returning.   Instead it wants to read minds for opaque objects.
Please don't get me wrong, I am not in anyway upset with you for pointing me at red herrings.  I appreciate the feedback greatly  (though if it may have been easier if you had simply request "please make tab completion work" rather than "add signatures" as I would have been working on the correct problem from start).  I can see perfectly how the Python documentation would lead you to believe that it is actually using the signature information for tab completion.  IPython docs are even worse in that regard at it all points to dir as being the relevant entry point which was only relevant to the greedy and previous gnu readline implementation.  Thus I sadly must declare as this entire thread is one giant red herring.  It is best to close it and redirect efforts to something that is achievable.
IPython does use signatures in the loosest sense.  It is but so indirectly that we could never satisfy it.   Because of the way jedi is constructed it always calls object.__getattribute() or type.__getattribute() or lighter probing methods so it is very careful to NEVER execute a piece of user defined code.  That is by design.  Thus there is never going to be an opportunity for us to execute our hooks to provide the information.
The only solution I can give you is use greedy rather than jedi.  It has the side effect that it calls the methods sometime triggering Java code, but it works perfectly for me when I am developing.
Further, I posted a support request to jedi.  Assuming they can point me at a user defined hook then I can provide this functionality.  I am still trying some investigations checking to see if the python trace module is somehow missing the call (they have a restriction of not reporting stackframes that don't have an associated file.)   If jedi does give me some way to place a user hook into the process, then I can add the decorators we discussed earlier and you can hack to your hearts content adding as detailed signature generation as you would like.  Until then I am sorry but this has to be "won't fix".

The author of jedi was kind enough to correct the defects the were leading to __annotations__ to be ignored from compiled objects.  This allows annotations to work for all JPype methods and fields.  With regards to your request...

Not quite that. I wanna a mechanism allowing me to wrap methods (including magic ones like init) and/or classes in python code.

I am not sure that this PR addresses your specific concerns regarding Scala.  The process by which jedi determines where to get the signature for a method or class is very complicated and it is not possible in general to wrap a function as a descriptor and expect it to work when you have mixed or dynamic classes which do not have a presence in python source code.   To make this work for JPype required hacking the access tables in jedi to force a specific access path.  You would need to formulate what you are trying to do, test it using the following jedi pattern, and (assuming it wasn't a JPype failure) contact the jedi author to find a solution.
import jedi
src='s.substring(1).CAS'  # or whatever code you want to be able to complete
script = jedi.Interpreter(src,[locals()])
compl = [i.name for i in script.completions()]
print(compl)

You can already customize __init__ using class customizers.  I also required so fairly beefy tracing scripts to flush out errors to submit bug reports for jedi.  It is a bit of a monster producing 10 Mb trace files.
I can add the ability to define a different __annotations__ routine using class customizers if required.  Just open a fresh ticket describing what you would require.


[Cannot cast value of type NULL to type string error in Inline mode]

Describe the bug
Hello, The library does not support Inline mode and I tested it with a simple bot and got
Cannot cast value of type NULL to type string error in log channel
To Reproduce
Steps to reproduce the behavior:
1.Turn on Inline mode in BotFather
2.use Inline mode : @bot_username some_thing
Then you will receive the following error in log channel if you add it:
Laravel ERROR (local): Cannot cast value of type NULL to type string {"exception":"[object] (WeStacks\\TeleBot\\Exceptions\\TeleBotException(code: 0): Cannot cast value of type NULL to type string at /var/www/vendor/westacks/telebot/src/Helpers/Type.php:96)"} []
Additional context
The first time I encountered this problem was September 3. on September 2nd you released version 2.2.2 but I installed earlier versions and it still didn't work.

Dublicate of #59. I tested inline mode and it works well, attaching script below. Again - your description is unclear. You are only telling that something is not working, but not providing code examples and full error trace. I'm not responsible for mistakes you make in your project.
<?php

use WeStacks\TeleBot\Objects\Update;
use WeStacks\TeleBot\TeleBot;

require_once 'vendor/autoload.php';

$bot = new TeleBot([
    'token' => '<token>'
]);

$bot->addHandler(function (TeleBot $bot, Update $update, callable $next) {
    if (!isset($update->inline_query)) {
        return $next();
    }

    return $bot->answerInlineQuery([
        'inline_query_id' => $update->inline_query->id,
        'results' => [[
            'type' => 'article',
            'id' => 'google',
            'title' => 'Google',
            'url' => 'http://google.com/',
            'input_message_content' => [
                'message_text' => $update->inline_query->query
            ]
        ]]
    ]);
});

$last_offset = 0;
while (true) {
    $updates = $bot->getUpdates([
        'offset' => $last_offset + 1
    ]);
    foreach ($updates as $update) {
        $bot->handleUpdate($update);
        $last_offset = $update->update_id;
    }
}

This is a test and in the file SearchMoviesInlineQuery.php
You can see is a very simple handler
But the update doesn't reach the handler and it exits in Type.php:96 because $value_type is equal to NULL

I've just cloned your project, and everything works as intended.

I really don't understand that it works properly in long polling mode

But it gives an error in the webhook

I figured it out. It is actually a bug. There is a weird behavior in Telegram's API: when the incoming update is inline query, there is a field - inline_query.offset. Documentation  says that this field is actually a string, however - when you receive updates using long polling, it's actually an empty string. But when they send a webhook request - they send NULL, that is not passing validation in our library. It's kind of weird, but I guess we can write some "crutches".

Actually, by second though, it's not Telegram's API, but Laravel's middleware, that converts empty strings to NULL on every request. It's even easier to fix then

For quick fix: enter app/Http/Kernel.php and remove ConvertEmptyStringsToNull middleware from your global middleware list. In the next update, I'll write the webhook route in a way to bypass Laravel's global middleware
    protected $middleware = [
        // \Illuminate\Foundation\Http\Middleware\ConvertEmptyStringsToNull::class,
    ];

ok thank you


The app crashed!

I was following tutorial. After I changed parking lanes to bike lanes, this happened:

panicked at 'PolyLine::make_polygons() failed: Ring has repeat non-adjacent points near Pt2D(709.1134, 719.801)', geom/src/polyline.rs:526:20

The url was http://play.abstreet.org/0.3.39/abstreet.html?--cam=19.63%2F47.64048%2F-122.31122

Thanks for reporting! Will look into this when there's time

Duplicate of #1078