Renamed courseware_object to credentialed_object

This was just a search/replace + generate migrations on the existing code. Tests passing should be adequate.

Codecov Report

Merging #22 (bfda7d6) into main (ed979bf) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              main       #22    +/-   ##
==========================================
  Coverage   100.00%   100.00%
==========================================
  Files           30        23     -7
  Lines          986       834   -152
  Branches        95        82    -13
==========================================
- Hits           986       834   -152

Flag
Coverage Δ

mitol-django-common
100.00% <ø> (ø)

mitol-django-digital-credentials
100.00% <100.00%> (ø)

mitol-django-mail
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...al-credentials/mitol/digitalcredentials/backend.py
100.00% <100.00%> (ø)

...-credentials/mitol/digitalcredentials/factories.py
100.00% <100.00%> (ø)

...grations/0002_rename_courseware_to_credentialed.py
100.00% <100.00%> (ø)

...tal-credentials/mitol/digitalcredentials/models.py
100.00% <100.00%> (ø)

...redentials/mitol/digitalcredentials/serializers.py
100.00% <100.00%> (ø)

mitol-django-mail/mitol/mail/__init__.py

mitol-django-mail/mitol/mail/forms.py

mitol-django-mail/mitol/mail/apps.py

mitol-django-mail/mitol/mail/messages.py

mitol-django-mail/mitol/mail/views.py

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed979bf...bfda7d6. Read the comment docs.


Perf compatibility for python3 print log with sys.stdout.write instead of print

Thanks for the PR. I don’t use this much personally anymore so feel free to raise any other issues you run into with python 3 etc.


Warning[:deprecated] = true by default in RSpec

Subject of the issue
See https://bugs.ruby-lang.org/issues/17591.
The plan of ruby-core hiding deprecation warnings in Ruby 2.7.2 is that test frameworks, like rspec, should enable them by default (with Warning[:deprecated] = true if Warning.respond_to?(:[]=), which only enables deprecation warnings).
This is important so that developers see warnings in development, and that they see the warnings before updating to the next Ruby version.
Of course, RSpec users can still choose to disable deprecation warnings explicitly if they really want with Warning[:deprecated] = false (but then they should not be surprised when it breaks hard when updating to a newer Ruby, that's the cost of ignoring deprecation warnings).
Somewhat related to https://github.com/rspec/rspec-core/issues/2809.
Your environment

Ruby version: 2.7.2
rspec-core version: 3.10.1

Steps to reproduce
# some_spec.rb
def foo(**kwargs)
  kwargs
end

RSpec.describe 'Test' do
  it "should warn about keyword arguments" do
    expect(foo({a: 1})).to eq({a: 1})
  end
end

rspec some_spec.rb
Expected behavior
$ rspec some_spec.rb
/home/eregon/code/rspec-ws/some_spec.rb:7: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/eregon/code/rspec-ws/some_spec.rb:1: warning: The called method `foo' is defined here
.

Finished in 0.00154 seconds (files took 0.05064 seconds to load)
1 example, 0 failures

Actual behavior
No warnings in 2.7:
$ rspec some_spec.rb
.

Finished in 0.00157 seconds (files took 0.05947 seconds to load)
1 example, 0 failures

Breaks hard in 3.0:
$ rspec some_spec.rb
F

Failures:

  1) Test should warn about keyword arguments
     Failure/Error:
       def foo(**kwargs)
         kwargs
       end

     ArgumentError:
       wrong number of arguments (given 1, expected 0)
     # ./some_spec.rb:1:in `foo'
     # ./some_spec.rb:7:in `block (2 levels) in <top (required)>'

Finished in 0.00121 seconds (files took 0.05637 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./some_spec.rb:6 # Test should warn about keyword arguments

For RSpec 3.x I won't change the default, I'd consider that a breaking change. If there was overwhelming support for it in 4.x I might consider making it the default as long as it was suppressible, but I'd prefer it remained an opt in behaviour (because there are very verbose libraryes out there) which we already allow (with --warnings, which is supported via .rspec etc).
An alternative I'd happily consider, is to place it into our generated spec helper, with a comment explaining what it is. This has the effect of effectively being on by default (as most new projects end up using our default spec helper from what I've seen), but also explaining how to not have it for verbose projects without having to look it up.

I think you are confusing with ruby -w/$VERBOSE = true.
This is only about deprecation warnings.
Such warnings if ignored pretty much guarantee the library will not work and break on the next Ruby version.
I'd bet most gem maintainers wouldn't want that, they'd rather be warned in advance if something is going to change in an incompatible way, that's the whole purpose of these warnings.
For instance, it doesn't make much sense to test a gem on 2.7 but not address or even see the keyword arguments deprecation warnings.
Also, this would have no effect on Ruby < 2.7, since Warning#[]= does not exist before 2.7.
In Ruby <= 2.6, deprecation warnings would typically be shown even with $VERBOSE = false (the default), so in one way the breaking change would be to not shown them on 2.7+ with RSpec.

because there are very verbose libraries out there

Do you have examples that actually emit many deprecation warnings and support Ruby 2.7?

to place it into our generated spec helper

It would only apply to new projects though, which seems a big drawback.
Finally I'd like to highlight this is not a personal request but part of the plan of ruby-core for deprecation warnings in Ruby 2.7+, as mentioned on the ruby-core issue.
The fact that test-unit also did this change is IMHO a good motivation for the other popular test frameworks to consider it.

For more background on the change in 2.7.2, I think this comment is helpful: https://bugs.ruby-lang.org/issues/16345#note-31
The idea is: deprecation warnings are useless for end users, but important for developers. The chosen way to address that is to hide them by default, and enable them by default in development (test frameworks, REPLs, etc).

I think you are confusing with ruby -w/$VERBOSE = true.
This is only about deprecation warnings.

I am not confused, it is not up to us to decide to emit deprecation warnings, if Ruby makes that choice, that is there decision, but they got push back and added this option to turn them on at a users discretion.

For instance, it doesn't make much sense to test a gem on 2.7 but not address or even see the keyword arguments deprecation warnings.

But that is not our decision to make for other libraries. We already check that RSpec itself produces no warnings, it's even part of our spec suite.

Do you have examples that actually emit many deprecation warnings and support Ruby 2.7?

Rails projects where the majority of code lives upstream in gems of various ages.

It would only apply to new projects though, which seems a big drawback.

It prevents us making a breaking change and allowing to introduce it earlier.

Finally I'd like to highlight this is not a personal request but part of the plan of ruby-core for deprecation warnings in Ruby 2.7+, as mentioned on the ruby-core issue.

They made this opt in after push back by the community about the level of deprecation warnings, that same push back applies here, we are following the Ruby community here.

I am not confused,

I guessed that because you mentioned --warnings, -w and warnings in general, which are basically unrelated to this issue.

They made this opt in after push back by the community about the level of deprecation warnings, that same push back applies here, we are following the Ruby community here.

That's a simplistic view of it. It was made opt-in because for end users (!= developers) the deprecation warnings cannot be acted on.
For developers, the warnings should be on by default, I believe ruby-core agrees to that without a doubt.
RSpec is a clear case of being run in development, by developers, as such it's important to enable deprecation warnings, which BTW are already shown by default in Ruby <= 2.6.
If you still disagree, how about asking RSpec users or gem maintainers using RSpec about it?
@mame @jeremyevans Could you share your opinion here?

BTW, part of my motivation to fix this is I've seen many gems that try to be compatible with Ruby 3, often do it incorrectly, and they probably completely missed the deprecation warnings of 2.7, which could have been useful to migrate earlier or more easily. Due in part to test frameworks not showing these important warnings by default.

My opinion is that Ruby should always emit deprecation warnings, and software designed for end users should use $VERBOSE = nil.  I think it was a mistake to change Ruby to not emit deprecation warnings by default.
@eregon is right that ruby-core expected that test library developers would enable deprecation warnings by default.  If RSpec doesn't want to do that, considering RSpec's popularity, we (ruby-core) should consider reverting the change in Ruby and enable deprecation warnings by default.

It's not up to us to ease the migration of the code that is outside of our control.
We'll do everything that is possible to properly delegate in our code.
Even if it was a mistake to turn warnings off I'm Ruby 2.7.2, it would be a strange move to turn them back on.

I guessed that because you mentioned --warnings, -w and warnings in general, which are basically unrelated to this issue.

Deprecations are warnings. I mentioned RSpec's own --warning mode which includes setting Warning[:deprecated] = true a change stemming from an earlier issue you yourself opened.

That's a simplistic view of it. It was made opt-in because for end users (!= developers) the deprecation warnings cannot be acted on.

I saw developer push back about it on the Ruby issue tracker, as I am not privy to the internal debates of the Ruby team I took that to be the cause of the subsequent change, I apologise for not investigating further at the time, but I've yet to come across a user of Ruby who is not a developer.

@eregon is right that ruby-core expected that test library developers would enable deprecation warnings by default. If RSpec doesn't want to do that, considering RSpec's popularity, we (ruby-core) should consider reverting the change in Ruby and enable deprecation warnings by default.

It would have been an option for us to do that at the release of Ruby 2.7 had someone reached out and made that expectation, we could have flagged this mode for Ruby 2.7 only and turned it on. However it is now a breaking change, in that previous Ruby code bases that passed without warnings would suddenly gain warnings from a point release of RSpec.
I'm happy to discuss turning this on in RSpec 4.

Actually up to and included Ruby 2.7.1, deprecation were shown by default.
So if RSpec did this, I see it as no breaking change but rather a continuation of existing behavior (it was like that up to Ruby 2.7.1) and I would consider it a help in development and for testing (e.g., might save a lot of time to understand why some specs don't work).
I do see it might actually break CI for someone when updating RSpec though, if they test on Ruby 2.7.2 or 3.0, they check stderr output and still have deprecation warnings in the tested code.

It would have been an option for us to do that at the release of Ruby 2.7 had someone reached out and made that expectation

Yes, I think such issues should have been opened right after the decision was made to disable deprecation warnings by default in Ruby 2.7.2. Unfortunately it seem committers at the time forgot or did not think about it (I guess they might have been busy with the 2.7.2 release).
I think RSpec 4 would be a fine target for this change if it's deemed too incompatible in RSpec 3.

@JonRowe I can work on this for RSpec 4, ok to tag this ticket as such?
Should I include an opt-out configuration option? Any suggestion for the name? config.suppress_ruby_2_7_2_deprecation_warnings!?

We have a couple of options either it could just be a true/false config that is used to decide wether we call Warning[:deprecated] = true or not before a spec run or not but then theres an argument for where we turn it on, wether it should be before spec file loading (which would be my vote) or before specs run. We could defer that decision to the user however, and default to before spec files, e.g. something like:
def ruby_deprecation_warnings=(value)
  @ruby_deprecation_warnings =
    case value
      when -> x { [:both, :on_spec_load, :on_spec_run].include?(x) } then value
      when true then :on_spec_load
    else
      :none
    end
end

FWIW, RSpec is the only popular test framework still not showing deprecation warnings to its users (https://bugs.ruby-lang.org/issues/17591). That means it makes it much harder for RSpec users to notice deprecated methods, and most likely they won't notice until something breaking, effectively destroying a significant part of ruby-core's efforts to have a nice transition path with deprecation, etc.
Yes, it's unfortunate this wasn't asked immediately after 2.7.2 came out, but the current situation is problematic.

Maybe we should think wider how to address this and be compatible enough. It has been two years.
I'm thinking the next release of RSpec 3 could maybe print a single line asking if one wants to enable deprecation warnings or not, and then when set that line would not be printed.
Like:
[rspec] this spec suite does not specify if deprecation warnings should be printed. Set it via `config.ruby_deprecation_warnings = true/false`.
[rspec] It is recommended to enable them to address such warnings in time, before it breaks hard.

Or maybe a message as gem/bundle install time suggesting to enable deprecation warnings and showing how.
Any thoughts on that?

I'll happily review a PR with a config option that sets this, and prints a warning if its not set for Rubies that support it; the message should specify how to remove the warning, and the config option should support just silencing the warning without turning it on.


We should support searching on canonical urls with version

Given
MeasureReport
{
   "resourceType": "MeasureReport",
   "measure": "http://StructureDefinition.com|1.2.3"
}

The following queryString should work:
MeasureReport?url=http://StructureDefinition.com&version=1.2.3

invalid issue - ignore


[ENG-61] Fix sorting on serializer fields with source attributes

Purpose
Currently, all sorting must be done on the model name. This is not intuitive to a user as some fields such as date_created have a source of created.

Changes
api/base/filters.py - Modifying OSFOrderingFilter to support sorting on serializer fields.
api_tests/base/test_filters.py - Adding tests to ensure ordering is working correctly

QA Notes
Test ordering on models with ?sort=date_created. The order should match ?sort=created

Documentation
There is currently no documentation on parameterized sorting in the documentation. Perhaps there should be?

Side Effects
Sorting by source name was not deprecated, so side effects should be minimal

Ticket
https://openscience.atlassian.net/browse/ENG-61

For posterity - I talked to Corbin about some small changes to tests that would help streamline them. Outside of those, everything looks good to me!

LGTM 🚢

merging into feature/staging-1 for testing


Delay slice()ing the listeneres

Redux introduced a few optimizations in 3.x, and we now delay slice()ing listeners unless absolutely necessary, and don’t slice() them more than once per dispatch(). This project should probably adopt similar optimizations.

Thanks for the heads up.
Was just looking over the redux release log an these look like the major changes:
reactjs/redux@b7031ce3acb23b6ecadbd977b1cfa32486447904
reactjs/redux#1325
reactjs/redux@c031c0a8d900e0e95a4915ecc0f96c6fe2d6e92b
reactjs/redux@5b586080b43ca233f78d56cbadf706c933fefd19
I wonder if I can avoid duplicating this logic by wrapping the listener callback instead of reimplementing the subscribe logic.

Give it a try, let me know if there’s something I can do on our side

So I ended up implementing both. Not sure if wrapping the listener ends up simplifying things, still end up needing all the tests as a regression.
Direct port of redux core: 4093d5c783f04192f9337e15ad1b8eba487d7308
Experimental version: ad8935d72d180e1e867091fc2e1afff9f3dea898
One idea on the redux side that could help is moving the subscription logic into a util function this library could consume: tappleby/redux@6ca7c8d265afa2d8521fcf6f4a62d8086f4a9aff

Mind publishing this emitter on npm?

I can, though does it make sense as a standalone package?
Also there's a bunch of emitters already on npm, wonder if it should be scoped to redux. eg redux-emitter or redux-core-emitter.

So I merged the direct port (4093d5c) and released with v0.1.5. Wasnt sure the timeline for reactjs/redux#1702, plus these optimizations will still be useful for non redux v4 users.

:+1:


Kit documentation needs updated?

Brief Issue Summary
The below excerpt from src/kit.ts seems to indicate there was a change in naming for the kit file.
/**
 * The path to the user-local kits file.
 */
export const USER_KITS_FILEPATH = path.join(paths.dataDir, 'cmake-tools-kits.json');

/**
 * The old path where kits were stored. Upgraded in 1.1.3
 */
export const OLD_USER_KITS_FILEPATH = path.join(process.platform === 'win32' ? paths.roamingDataDir : paths.dataDir, 'cmake-tools.json');

The current docs/kits.md seems to reference the old naming. If this is indeed an error, I will do a pull request to fix it, but I wanted to make sure I was understanding correctly before doing so.
CMake Tools Diagnostics
No response
Debug Log
No response
Additional Information
No response

Thank you for pointing out this error in the documentation. The user-local kits' file name should be modified to 'cmake-tools-kits.json'.
Please feel free to send a PR.

Fixed in #2772.


BATMAN Theme added and also resolved conflicts

Hey @JLambertazzo I resolved those conflicts as you mentioned and now the PR is ready to merge .... btw thanks for your guidance 💯

Please don't forget to add hactoberfest label @JLambertazzo .

Unfortunately the changes are still here. If you check your files changed it's about 600 lines added, but it should be more like 3 or so for adding a theme. Can you look into this?

Yeah I saw those changes but those were just alignment changes made by vs code as some lines were not aligned properly hence while saving these lines they were properly aligned you may have a look at the files changed only spaces and tabs are added

Unfortunately I still can't merge the alignment changes since if anyone else merges using a different style formatter we will have the same issue all over 😟
You can either change your formatter settings temporarily by pressing ctrl+shift+p and typing Formatter in vscode or make the updates using any text editor that doesn't have autoFormatOnSave

ok @JLambertazzo  I will try to make those changes ....


Private proxy

The official library documentation for the addProxy() function does not say anything about the username and password arguments. Does the library support private proxies, and if so, how?

proxyTypeHttp.
proxyTypeSocks5.
proxyTypeMtproto.

proxyTypeHttp.
proxyTypeSocks5.
proxyTypeMtproto.

td_send({'@type': 'testProxy',          'server': ip,          'port': port,          'type': ?})
How is it possible to implement type?

For example, {'@type': 'proxyTypeSocks5.', ...}.

For example, {'@type': 'proxyTypeSocks5.', ...}.
@type denotes the method to dispatch. Before "?" everything is fine, instead of this symbol the program asks for an object. I don't understand in what format to transfer the last element of the dictionary.


Fail to transfer Trial App to subscription which has a web app with "Free Linux App Service Plan" existing

OS: Win10
Build Version: 20200701.1
Repro Steps:

Make sure there is a web app with "Free Linux App Service Plan" in your subscription.
Right click one Trial App and select “Transfer to Subscription…” --> Sign in and select the subscription --> Enter an unique web app name.

Actual:
Fail to transfer Trial App to subscription.

@nturinski tested and this does seem to be a region-specific limit. Debatable how often users would run into this error, but we could just prompt for location as a workaround

Closing because of https://github.com/microsoft/vscode-azureappservice/issues/1880


can not add routes(version 0.13.rc2 )

Summary
Kong shows these errors during add routes:
HTTP/1.1 400 Bad Request
Date: Thu, 15 Mar 2018 10:17:54 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *
Server: kong/0.13.0rc2

{"fields":{"hosts":"expected an array","service":{"id":"missing primary key"}},"name":"schema violation","code":2,"message":"2 schema viola
tions (hosts: expected an array; service.id: missing primary key)"}

Steps To Reproduce

create services

curl -i -X POST
--url http://localhost:8001/services/
--data 'name=DevOps-docs-1'
--data 'protocol=http'
--data 'port=18181'
--data 'host=service.node.a.com'

successful, service id could get from table:

kong=> select * from services;
id                  |       created_at       |       updated_at       |     name      | retries | protocol |        host        | port  | path | connect_timeout | write_timeout | read_timeout
--------------------------------------+------------------------+------------------------+---------------+---------+----------+--------------------+-------+------+-----------------+---------------+--------------
757afb91-ccfe-43a4-b02e-d4d2e6892394 | 2018-03-15 09:59:00+08 | 2018-03-15 09:59:00+08 | DevOps-docs-1 |       5 | http     | service.node.a.com | 18181 |      |           60000 |         60000 |        60000
(1 row)

add routes

--url http://localhost:8001/routes/
--data 'hosts=doc.yercc.com'
--data 'service=757afb91-ccfe-43a4-b02e-d4d2e6892394'

error comes
{"fields":{"hosts":"expected an array","service":{"id":"missing primary key"}},"name":"schema violation","code":2,"message":"2 schema viola
tions (hosts: expected an array; service.id: missing primary key)"}

Additional Details & Logs

Kong version (0.13.0rc2)

We have a new syntax for arrays. In future we will make your example to work when we add schema aware parsing. Use hosts[] for now as an argument.

@bungle thanks, what about second issue, service id missing

@bungle got it, thanks.
BTW, where can I find these new style syntax reference?

@kswen we usually make docs available when we have a release (betas, alphas, rcs don't count). There are some docs already added in these branches:

https://github.com/Kong/getkong.org/tree/0.13.x
https://github.com/Kong/getkong.org/tree/0.13.x-plugins

Even i am getting error for hots or hots[] for both
when hots[]
curl -XPOST  http://localhost:8001/routes -d '{"name":"kongdemo","hosts[]": ["localhost"], "paths[]": ["/stock/btc"],"protocols[]": ["http", "https"],"methods[]": ["GET"]}'
{"message":"2 schema violations (must set one of 'methods', 'hosts', 'headers', 'paths', 'snis' when 'protocols' is 'https'; '{name:kongdemo,hosts[]:: unknown field)","name":"schema violation","fields":{"'{name:kongdemo,hosts[]:":"unknown field","@entity":["must set one of 'methods', 'hosts', 'headers', 'paths', 'snis' when 'protocols' is 'https'"]},"code":2}curl: (3) [globbing] bad range in column 2
when hots
=======curl -XPOST  http://localhost:8001/routes -d '{"name":"kongdemo","hosts": ["localhost"], "paths": ["/stock/btc"],"protocols": ["http", "https"],"methods": ["GET"]}'
{"message":"2 schema violations (must set one of 'methods', 'hosts', 'headers', 'paths', 'snis' when 'protocols' is 'https'; '{name:kongdemo,hosts:: unknown field)","name":"schema violation","fields":{"'{name:kongdemo,hosts:":"unknown field","@entity":["must set one of 'methods', 'hosts', 'headers', 'paths', 'snis' when 'protocols' is 'https'"]},"code":2}curl: (3) [globbing] bad range in column 2


find a way to clean up ffmpeg output and provide progress bar

Ok, I think I have it!  After lots of hours of testing and tweaking and an idea from Rupert Plumridge I have come up with a pretty slick solution.  I run ffmpeg in the background outputing to a tmp file and parse the tmp file pulling out just what i want and massaging the data to give a rough sense of progress.  A TODO could be to calculate percentage and display that as well.
This is what it looks like when running:
Converting DTS to AC3:
size=   11961kB/119254kB time=218.72 bitrate= 448.0kbits/s
Here is the whole conversion section for review.  It should be a direct cut from the first line and replace up to the last line:

# ------ CONVERSION ------
# Convert DTS to AC3
doprint $"Converting DTS to AC3."
doprint "> ffmpeg -i \"$DTSFILE\" -acodec ac3 -ac 6 -ab 448k \"$AC3FILE\""

dopause
if [ $EXECUTE = 1 ]; then
        color YELLOW; echo $"Converting DTS to AC3:"; color OFF
        nice -n $PRIORITY ffmpeg -i "$DTSFILE" -acodec ac3 -ac 6 -ab 448k "$AC3FILE" 2>/tmp/ffmpeg_output &
        PID=$!
        DTSFILESIZE=$($DUCMD "$DTSFILE" | cut -f1) # Capture DTS filesize for end summary
        let AC3FILESIZE=$DTSFILESIZE/5  #Estimate on AC3 ending filesize based on typical use
        while [ -e /proc/$PID ];do
                LINE=$(tail -1 /tmp/ffmpeg_output|sed "s/kB/kB\/${AC3FILESIZE}kB/g") #Inject size in output
                echo -ne "\r$LINE"   # Provide updates from ffmpeg_output every second
                sleep 1
        done
        checkerror $? $"Converting the DTS file to AC3 failed" 1
        cleanup /tmp/ffmpeg_output

        # If we are keeping the DTS track external copy it back to original folder before deleting
        if [ ! -z $KEEPDTS ]; then
                color YELLOW; echo $"Moving DTS track to MKV directory."; color OFF
                rsync --progress -a "$DTSFILE" "$DEST"
                checkerror $? $"There was an error copying the DTS track to the MKV directory. You can perform
 this manually from \"$DTSFILE\"." 1
        fi
        cleanup $DTSFILE

        timestamp $"DTS track conversion took: "
fi

I might be able to use this same concept to provide a progress bar for Extracting DTS.  Need to revisit this again.

Submitted a version with this to dev branch.  Please test and ensure no issues

Output is cleaned up and progress percentage works as expected.  Closing issue.


Add Release workflow to version deployments

This adds a release workflow that creates a new GitHub Release (and git
tag) with an incremental version number. This version tag is then used
to tag images and referenced in the rest of the deployment system. This
makes it easier to compare references (vs commit SHAs). The version tag
is calculated as the last version incremented by the number of merge
commits until the reference commit.

Test bulk - recreating


docs: add potential flow tutorial

Description
Adds a tutorial based around an angle sweep of a flow over an airfoil using a simple mfem-based potential flow solver.
Type of change
Please select one below (Please click check boxes AFTER submitting ticket)

[ ] Bug fix
[ ] New feature
[x] New Documentation
[ ] Other (please describe below)

Supers Powers merge for Docs.


Use local connection with proxy

Hello,
I tried to use local connection with hub but it does not work. I view IDENTIFY request, my response is :

This is my first question, why I need verificationId, doc say it's necessary but reference say it's optionnal (https://developers.google.com/actions/smarthome/reference/local/interfaces/smarthome.intentflow.identifyresponsepayload). For me in my case it's not need because I send isLocalOnly true (True if this device does not appear in the `SYNC response. This is common for hub devices.). So i device is not in SYNC response it's does not need a verificationId, no ?
After that I receveid REACHABLE_DEVICES, I said all device it's reachable (home is proxy for all my device in google home).
After that I get "action.devices.PROXY_SELECTED", I don't know what I have to do with that... I can't find anything about that in documentation... I try to resend same response as REACHABLE_DEVICES and it's seem ok.
So according to the console all it's ok but I don't received any local execute exept if I test on verifcationId device.
Did anyone success with local api and hub systeme ?
Thank in advance for your reply

Let me try to reproduce your issue, ideally we should also have a sample for the hub device (filed #2).

Thank I make some other try but I think I miss something...

@zoic21 Did you manage to make the non-hub case work?

Yes for one device I make it work (I view execute command from google home hub). But I develop it for an home hub so I need proxy mode.

@zoic21 can you confirm if your hub device is assigned to a Room?

Yes not all device but 90% of them are affected to room. For the hub I take first presented device from IDENTIFY and I say it's a hub and proxy. Maybe I need to create a specify device for this ?

I also get the proxy selected event, after which time it errors out. I took a look at the sdk code, and it’s possible to respond to it, but I’m unsure what to send back.

I have also PROXY_SELECTED but I can't find any documentation on it... I tried to resend same response as REACHABLE_DEVICES. No error but I it doesn't work...

I test also with :

Did not work...

I also get the same error

smarthome.DeviceManager handle PROXY_SELECTED not implemented

Managed to reproduce the issue:
Ready, App version: 0.0.1
bundle.js:288 Ready
app.ts:37 IDENTIFY request {requestId: "E425620A102646A3C9FD3F8EF9CF430D", inputs: Array(1), devices: Array(1)}
app.ts:37 IDENTIFY request {requestId: "3FA3DEF5A470BB9DA52E667F1396C17F", inputs: Array(1), devices: Array(1)}
app.ts:74 IDENTIFY response {intent: "action.devices.IDENTIFY", requestId: "E425620A102646A3C9FD3F8EF9CF430D", payload: {…}}
app.ts:74 IDENTIFY response {intent: "action.devices.IDENTIFY", requestId: "3FA3DEF5A470BB9DA52E667F1396C17F", payload: {…}}
smarthome_sdk.js:197  [ 56.980s] [smarthome.DeviceManager] Handler for REACHABLE_DEVICES not implemented
smarthome_sdk.js:197  [ 57.007s] [smarthome.DeviceManager] Handler for PROXY_SELECTED not implemented

Note that PROXY_SELECTED is triggered even if REACHABLE_DEVICES is not implemented (or fails)

Adding:
smarthomeApp
    .onProxySelected((request: any) => {
        console.log("PROXY_SELECTED request", request);
        return {};
    });

Seems to workaround the failure.

It seems that onProxySelected doesn't exist.
Where and how should it be implemented ?

@GatienB it does exists in the JS SDK, but it's not exposed thru the typescript typing.
Tracking the issue internal at b/https://github.com/actions-on-google/smart-home-local/issues/1#issuecomment-515706997
See:
https://github.com/actions-on-google/smart-home-local/issues/1#issuecomment-515706997

Thanks ! The error disappeared.
Are you able to control your device behind the hub ?

I test, no error but nothink work...

@proppy do you have any clue why it's still not working and how to solve it ?

Any update on this?
Do we have an ETA for an updated to the typescript types and doc for the SDK so we can get hub devices working?

We have @home-assistant working as a hub device with the local SDK 💃
I documented all the different interactions between Home Assistant and Google Assistant, including the quirks in the current implementation and incorrect types in the docs, to help the people in this issue 👍 Find it here
If you want to try it out with Home Assistant, you will need to use this PR https://github.com/home-assistant/home-assistant/pull/27428

@balloob thanks for working on that!
Let's work together one surfacing the issue you found in the smarthome public issue tracker

@proppy already reported it, b/142429438

Hello,
Googd news, but I try and it does not work... All it's ok in console, but I received only once REACHABLE_DEVICES (not every minute).
Weird stuff too on I do :
var reachablesDeviceHandler = function (request) {   console.log('reachablesDeviceHandler : '+(new Date().toLocaleString())+' request : ',request);   var reachables_devices = [];   for(var i in request.devices){     if(request.devices[i].id == 'fake-local'){       continue;     }     reachables_devices.push({verificationId: request.devices[i].id});   }   console.log('reachables :',reachables_devices)   var response = {     intent: 'action.devices.REACHABLE_DEVICES',     requestId: request.requestId,     payload: {       devices: reachables_devices     }   };   console.log('response :',response)   return response; };
But on my console.log I get :

Why id instead of verificationId ?
Last thing, in documentation of home assistant I see "To trigger this intent, tell Google "force local" before saying a command.", but I am in french langage, I try "Force local #command#", "Force l'éxecution locale #commande#", nothcink work....

Reachable devices need to return verificationId not Id.
For force local, try putting your Google Assistant in English

Thanks for your réponse, i can't return verificationid, my code name Key correctly but something change key by id...

Am I misunderstanding how this should work ? I'm using UDP discovery, and sending response as this:
{
  deviceId: 'openhab-local',
  port: 8089,
  itemPath: '/rest/items/'
}

Identify is called and it has this data in request.inputs[0].payload.device.udpScanData so kind of ok.
From identify handler I return this:
{
  intent: "action.devices.IDENTIFY",
  requestId: "1168E8DC3A64CD2BB3240DC270F0342D",
  payload: {
    device: {
       id: "openhab-local",
       isLocalOnly: true,
       isProxy: true
    }
  }

Now reachable devices handler is invoked with this request:

but proxyData is empty, though docs about ProxyDevice state that it has a member udpScanData which Contains data if this result came from a UDP scan
What am I missing?

Thanks for your réponse, i can't return verificationid, my code name Key correctly but something change key by id...

I think this is expected, if you look at minified smarthome_sdk.js source for some reason they are assigning  value to idand deleting verificationId
 var k = Sc(d, f);
      k
        ? ((f.id = k), delete f.verificationId, a.push(f))
        : console.error(
            "Failed to match device " + f + " with cloud synced devices "
          );

Though I must say this got me really confused while debugging...

Hey Folks, wanted to provide a quick update here. The current preview release has a few known issues in both the SDK and the platform firmware blocking proper testing and implementation of hub devices. We are working to get these addressed and they will be fixed in the next preview release. At that point we will be able to publish updates docs and samples regarding hubs.
We'll of course post an update when the updated preview is available.

Hi @devunwired , thanks for the comment which will save a lot of time.
I noticed one issue for hub, in the identify stage, google home will trying to match the hub id to an unverified device (which is one of the otherDeviceIds in sync resp), however sync will normally not include any hub devices to the resp, so this should be fixed in the next preview.
Please correct me if I misunderstood.

Another one:
If the device is a hub, in the identify resp:
device.isLocalOnly: true
device.isProxy: true
these two field is documented well and are clear enough.
how about device.type? just ignore this field or use action.devices.types.HUB or something else?

Related link #13 .

Hi @ufosaga

I noticed one issue for hub, in the identify stage, google home will trying to match the hub id to an unverified device (which is one of the otherDeviceIds in sync resp), however sync will normally not include any hub devices to the resp

You are correct, this is why we have to separate flags for isProxy and isLocalOnly:

isProxy: This device is a proxy/hub that can reach other local devices
IsLocalOnly: This device is not present in the SYNC response

A device that returns isLocalOnly during IDENTIFY will not be matched to one of the otherDevicesIds in SYNC (if you observe different behavior, please file an issue).

These two fields are documented well and are clear enough. But how about device.type?

This field is optional and you can ignore it for your hubs. If appropriate for your device, you can set this field to one of the device type strings.

Is there still a bug in Execute for Hub type devices ?
In identify I hardcoded my HubID (isProxy = true, isLocal = true)
In execute when I try to send (via DeviceManager) using the hardcoded HubID it says the device can not be found.

Found my problem ... if I used the Proxied DeviceId, i.e. the DeviceId the command is sent to,  as opposed to the harcoded HubId, the Device Manager sent the info on to the hub.
I was happy to see that traits with 2FA authentication was sent via the Local path witch because of common code,  returned an error that requested a PIN ... and the follow up execution was sent via the Cloud path.  So it is not quite as "BROKEN" as i though it might be.

Yesterday cast firmware version 1.44.184686 got published to Google Home's on the preview channel (you can opt-in to this via Google Home app -> settings of a Google Home). It includes a bunch of bug fixes for the local SDK. We've been getting better results. The Google Home no longer restarts whenever we quit Home Assistant 👍
Did notice one breaking change, and that's for mDNS data. Instead of additionals the data is now nicely parsed and ready to use 👍
I've updated the request-responses that we see when interacting with the local SDK: REQUEST-RESPONSE.md. It also includes an example of an IDENTIFY intent with the new mDNS format.

We've updated this sample to include a hub use case, and included some additional detail in the implementation guide for hubs.
Please note one item in the updated documentation, as @balloob mentioned, you have to opt your test devices into the Cast Preview Program to test this functionality with the latest Cast firmware, which has not rolled out yet to production devices.


Check if found folder is not null when loading fragments

When loading fragments, check if the public folder is not null first
Closes #9473

YUP. OOPS.


Flow visualization

TauFactor version: 2
Python version: Anaconda Python 3.7x

Description
is there a way to visualize "flow"? (e.g. like PoreSpy's "concentration" at the bottom of this page? https://porespy.org/examples/simulations/tutorials/finding_tortuosity_from_image.html)

Hi @pcmkible, soon.
If you want you can pull the development branch and call s.calc_vertical_flux() after solve.

Solved with #97


bug: MasonUI is opened and is refreshing EVERY startup of neovim

Did you check docs and existing issues?

[X] I have read all the plugin docs
[X] I have searched the existing issues
[X] I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)
nvim v0.10.2
Operating system/version
macOS 14.7
Describe the bug
everytime i open neovim when nvim-java is installed, the mason ui is opening and refreshing.
Steps To Reproduce

install nvim-java
install the plugin like the documentation (not using lazy, kickstart or astro)
enjoy mason UI being shown every nvim startup

Expected Behavior
this only happens for the first time, when all dependencies are installed, so it seems that it doesnt handle dependencies being installed and not showing the mason UI.
for people having the same problem in
~/.local/share/nvim/site/pack/packer/start/nvim-java/lua/java/startup/mason-dep.lua
outcomment line 39 & 40 and you are fine

I think it is related to something else... (or maybe I am wrong). I mean when I successfully added nvim-java mason repo this stopped to happen (without the need to comment out 39&40).

@PiotrRaszkowski i have added all repos in my packer config and it still shows up, trying to update and install v1.37.0 as mentioned in my other issue.
If there is a another mason registry that is working that isn't mentioned in the documentation feel free to make a PR that fixes the missing registry in the documentation

https://github.com/nvim-java/nvim-java/wiki/Q-&-A#no_entry-cannot-find-package-xxxxx


esp32_allocateheap.c: Don't allocate ROM regions when running in QEMU.

Summary
This PR fixes instabilities found when running ostest in QEMU with SMP.
Impact
ESP32/QEMU/SMP
Testing
esp32-devkitc:smp + ostest

@masayuki2009 I basically had to revert what I did last time but only for QEMU.  This looks too hacky, I'll try to push to fix the issue from the QEMU side.

a curious question: besides qemu, can there be different ROM versions with different ROM area usage?

@Ouss4
Thanks!
It works with QEMU now.

a curious question: besides qemu, can there be different ROM versions with different ROM area usage?

@yamt Generally speaking, yes. Different ECO versions can have different ROM area usage.  However, this is not the case for ESP32.


[DNM] peer gc raft_log itself, not by leader CompactLog command

Does it relate to region checksum? @busyjay

Seems default raft_log_gc_count_limit and raft_log_gc_size_limit are too large for peer gc.  As follower don't know replicated_index, it can only use apply index for compact.

Maybe we can remove raftlog_gc.rs and ignore the compact log handling when applying the CompactLog admin command.

If we need backward compatibility, the CompactLog admin command still need handling in apply worker.
Because there may have some raft log not applied before restart,  we still have to handle it after restart, no matter we restart a single server or the whole cluster.

We can ignore backward compatibility before 1.0 GA

we can ignore the CompactLog and mark it as applied directly, later the GC scheduler will compact the log automatically.


Enhancement of membership page

The membership page is not looking good I want to improve the UI of that page. Please assign me for this issue under JWOC

Please assign me this issue under JWOC.

The membership page is not looking good I want to improve the UI of that page. Please assign me for this issue under JWOC

Go on @ArpitaSaga

Deadline is over for @ArpitaSaga please assign this project. I want to work on this issue under JWOC


fix(app-config): invalid app-config.yaml

The app-config.yaml file was found to be invalid.
To address this issue and ensure consistency with backstage-showcase, the scaffolder property has been set to an empty object.

/lgtm


Please replace jschardet with a different library

Please remove jschart. Many people have reported the problem of file coding detection in the community. These problems are caused by the design concept of jschart. In addition, these problems are labeled as "upstream", which leads to ineffective processing, and some of them are even closed directly.
https://github.com/aadsm/jschardet/issues
https://github.com/Microsoft/vscode/issues/23570
https://github.com/microsoft/vscode/issues/24195
https://github.com/Microsoft/vscode/issues/27419
https://github.com/microsoft/vscode/issues/33720
https://github.com/microsoft/vscode/issues/36230
https://github.com/microsoft/vscode/issues/36951
https://github.com/Microsoft/vscode/issues/41393
https://github.com/Microsoft/vscode/issues/47872
https://github.com/microsoft/vscode/issues/51125
https://github.com/microsoft/vscode/issues/52132
https://github.com/Microsoft/vscode/issues/61663
https://github.com/Microsoft/vscode/issues/64419
https://github.com/microsoft/vscode/issues/64931
As the https://github.com/microsoft/vscode/wiki/Issue-Grooming#out-of-scope-feature-requests point out, we need more up-votes, if you have trouble in 中国简体,please give a up-vote.

Has the community at large expressed interest in this functionality? I.e. has it gathered more than 10 up-votes or more than 10 comments? This criterion alone covers more than 650 of the 2850 open feature requests as of right now, October 9th, 2019.

Chinese Version:
大家好，本issue希望vscode 移除一个第三方依赖 jschar.
jschart 通过检测遍历文档内容的方式输出预期编码类型。这种机制在纯西方文字环境下能够很好的工作。但是，如果代码中混合部分中文特有字符，比如“中国人”，则很可能返回为“Windows 1252”编码类型，导致文件内容无法正常展示。这种缺陷本质上是jschar无法很好平衡“精确度”与“鲁棒性”相抗衡的结果。
为此，我希望大家通过投票的方式支持改进文件编码检测逻辑。
真诚希望所有遇到问题，或者将来可能遇到问题的朋友，为本issue点赞。
您的点赞将直接影响vscode对这项改动的支持力度。
根据 vscode 团队的 feature-requests 处理流程，https://github.com/microsoft/vscode/wiki/Issue-> Grooming#out-of-scope-feature-requests  ，如果点赞数量不够，本issue很可能不会被处理，而是被关闭。

Has the community at large expressed interest in this functionality? I.e. has it gathered more than 10 up-votes or more than 10 comments? This criterion alone covers more than 650 of the 2850 open feature requests as of right now, October 9th, 2019.
#84503 didn't have enough up-votes,

It's hard to believe that this issues has existed for so long.

The same file works fine in these editors: Visual Studio, Notepad ++, Beyond Compare, etc., no one cares.☹

Is there a better alternative than jschardet?

I see the issues with jschardet and we can certainly remove the support for auto guessing encodings but we can also investigate to either fix the issues or replace jschardet with something else if that is possible.
Encoding detection is currently only used in 2 places:

from the encoding picker of the status bar (we offer the guessed encoding as one of the top entries if guessed)
for any editor that is opened if files.autoGuessEncoding is set to true

Note: it is unlikely that VSCode would make an investment in improving the libraries, so this would need community help. But I am certainly willing to adopt a different library if it has good test coverage and shows improvements over the current one.

Thank you for your interest in this issue.
In order to provide higher quality coding detection, we want to collect some test cases.
The test case consists of three parts: the original file, the feature, and the expected encoding type (which can be multiple encoding types)
Demo:
File1, containing only simplified Chinese characters (酷酷的哀殿), GBK
File2, including simplified Chinese and English letters (My name is  酷酷的哀殿), GBK
File2, including simplified Chinese and English letters (My name is  酷酷的哀殿), UTF-8
谢谢大家对本issue的关心。
为了提供更高质量的编码检测，我们希望收集一些测试case。
测试case应该包含三部分：原始的文件，特征、期望的编码类型（可以是多个编码类型）
demo：
file1，只包含简体汉字（酷酷的哀殿），GBK
file1，同时包含简体中文+英文字母（My name is  酷酷的哀殿），GBK
file1，同时包含简体中文+英文字母（My name is  酷酷的哀殿），UTF-8

I think it would also be worthwhile seeing for existing test cases for jschardet and their underlying library.

When we search thousands of files, there are still many problems. A lot of results will not be shown.
let the user make the choice of changing it. will not solve this.

@sunbohong can you please elaborate a bit more what's your suggestion here. Sorry I didn't catch up with it.

For example,if i have a folder,which contain a GBK file,a UTF8 file.
when i search with a key.
Case 1. You can locate UTF8 file in the index by entering 酷酷的哀殿.

Case 2.If we open GBK file,You can locate both files in the index.

If there were thousands of files, we have to open thousands of files.

@sunbohong the issue you ran into is related to the search area, please file a separate issue as I think we don't have encoding detection in search, cc @roblourens is this a duplicate?

How to remove jschart? I can't find it

Search can only work in one encoding at a time, based on files.encoding. If you open the file, then it will be searched with encoding detection if that's enabled. I think that explains the above.


Make struct repr transparent so transmute is safe.

A lot of the code assumes that BpfInstruction and BpfProgram have the same size and layout as their raw counterparts, but they have to have a repr(transparent) for that to be safe.
The nomicon says:

When transmuting between different compound types, you have to make sure they are laid out the same way! If layouts differ, the wrong fields are going to get filled with the wrong data, which will make you unhappy and can also be UB (see above).
So how do you know if the layouts are the same? For repr(C) types and repr(transparent) types, layout is precisely defined. ...

Here's one example:
https://github.com/rust-pcap/pcap/blob/bc8a3c0cbb8bcf4365fd959b9c79b0729fd79482/src/lib.rs#L1389
It's not a std::mem::transmute, rather a pointer cast, but the structs still need to be transparent for it to be safe :)


Any plans to migrate to npm?

I know a lot of meteor packages are planning to migrate to npm.
Any idea if Astronomy has any plans on doing so...?

Probably not until version 3.0. I'm waiting until Meteor switch all packages to npm that Astronomy is using. Before that it's not possible for Astronomy to switch.

I'm curious, what are the Meteor dependencies? I might be able to help with migrating, I would find this useful in my react-native projects, for which I currently have to use simple schema.

Astronomy is heavily dependent on meteor collections and any save/remove operation goes through the meteor method. I was thinking about creating separate package that can be used in any JS project, which would be stripped of the storage functionality. It would allow defining schemas, validation, values casting etc. But I have several projects going on and I lack of the time to develop it :)


Eagerly create the backing configurations of aggregation reports

Eagerly create the backing configurations of AggregateTestReport and JacocoCoverageReport

Allows customization/filtering by specifying additional Attributes

--
This change is necessary for Android compatibility, where two variants containing test results from the debug and release buildtypes are indistinguishable. In order to allow adding the BuildType attribute to the consumer configurations, we need to create the configuration slightly earlier. Previously the configuration was created in a task configuration action and not accessible.

@bot-gradle help

Currently, the following commands are supported:
@bot-gradle test <BuildStage>

triggers the specified build for this PR. This command takes a single argument, i.e. @bot-gradle test QF.  Common stages are:

SanityCheck/CompileAll/QuickFeedbackLinux/QuickFeedback/ReadyForMerge/ReadyForNightly/ReadyForRelease
Shortcuts: SC/CA/QFL/QF/RFM/RFN/RFR
Specific builds:

AST: AllSmokeTestsReadyForMerge
AFT: AllFunctionalTestsReadyForMerge
ASB: AllSpecificBuildsReadyForMerge
ACC: AllConfigCacheTestsReadyForMerge
ACT: AllCrossVersionTestsReadyForNightly
APTN: AllPerformanceTestsReadyForNightly
AFTN: AllFunctionalTestsReadyForNightly
ACTR: AllCrossVersionTestsReadyForRelease
AFTR: AllFunctionalTestsReadyForRelease

@bot-gradle test and merge

queues this PR for testing and merges if all tests pass by:

Creating a merge commit from your PR branch HEAD and the target branch
Running a ReadyForNightly build against the merge commit
When it passes, fast-forward the target branch to this merge commit (i.e. merge the PR)

The merge commit is called a pre-tested commit, which means that it fully tests the integration
of your branch HEAD and latest master, instead of only testing your branch HEAD.
@bot-gradle cancel

cancel a running pre-tested commit build or remove it from queue

@bot-gradle clean

clear the conversation history

@bot-gradle help

display this message

To run a command, simply submit a comment.  For detailed instructions see here.

@bot-gradle test ACC

OK, I've already triggered AllConfigCacheTestsReadyForMerge build for you.

@bot-gradle test and merge

Your PR is queued. See the queue page for details.

OK, I've already triggered a build for you.

Pre-tested commit build failed.

@bot-gradle test and merge

OK, I've already triggered a build for you.

Pre-tested commit build failed.

@bot-gradle test and merge

OK, I've already triggered a build for you.


Using an HTTP proxy that is already configured

Is it possible to use a proxy setting located mainly within the editor (Settings > Appearance & Behavior > System Settings > HTTP Proxy) to make requests to OpenAI?

I have never used IntelliJ's built in HTTP Proxy, but as far as I understand, it should work. Based on the docs, it redirects all http traffic through that proxy, so I would assume it does the same for Open AI requests.


Filter Descriptors are only default when they have a default value set

Filter Descriptors can only be set as default when they have a default value set, even if a value is not required.
Cheers


Added support for the BTCe exchange

Warning - Some of the functions need testing by someone in posession of a BTCe account. In particular this is the case for  the buy(), sell(), cancelOrder() and getOrder() functions

@MrSiNiSt3R might be interested in testing

https://www.justice.gov/usao-ndca/pr/russian-national-and-bitcoin-exchange-charged-21-count-indictment-operating-alleged <- Hmm... ;)


onClick area not correctelly defined

onclick gesturedetector is not put at the right place . It should be moved before padding for better user experiance when clicking a selected tab. currently we struggle to click on a selected tab.
Line 153 to 159
`
child: Container(
child: AnimatedAlign(
          duration: Duration(milliseconds: ANIM_DURATION),

          curve: Curves.easeOut,

          alignment: Alignment(_circleAlignX, 1),

          child: GestureDetector(

            onTap: widget.tabs[currentSelected].onclick,

            child: Padding(

            padding: const EdgeInsets.only(bottom: 15),

            child: FractionallySizedBox(

              widthFactor: 1 / widget.tabs.length,

              child: Stack(

...
`

My suggestion didn't really solve the problem. The click on selected tab don't always work as expected. We need to click multiple time to make it work.

I found this issue also. Debugging in the widget inspector, the _RawMouseRegion ends at the top of the icon rather than the top of the button.  So the _RawMouseRegion starts at the top of the icon and carries down to the top of the widget title text.
This means the tap works correctly if you tap towards the bottom of the button. I was looking for a way to wrap this in a gesturedector of some sort to be able to control the tap region, but no luck yet.
Overall love the widget, just wish the tap area was configurable or covered the entire button.


SNV / SV Filter bug

Detailed information for SNVs are not displayed

SVs filtering is not working

Steps to reproduce the behavior:

Go to 'Filter Variants for Case'
Click on 'Filter&Display'
Scroll down to 'Results'
Click on little arrow for each variant
See error "Error during AJAX call:"

Expected behavior

I would like to see the detailed description of the variant

Steps to reproduce the behavior:

Go to 'Filter SVs'
Click on 'Filter&Display'
See error: nothing happens

Expected behavior
2) I would like to see the filtered SVs
Desktop (please complete the following information):

OS: [e.g. Windows]
Browser [e.g. Mozilla Firefox]

@boschann should be working again. Thanks for the report!


Files marked as Dirty force parsing of all files

Saving a .py file (occasionally closing an already saved file) triggers pylance to mark a file as dirty, triggering parsing of not only all files within the workspace but, all stubs located in c:\Users<User>.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\ and files in d :\repos\XXX.venv\Lib\site-packages.
The time spent parsing is approximately 1min 30sec, with CPU usage between approximately at 60% until parsing completes

After parsing is complete vscode will idle at <1% but RAM usage can be 2.0GB - 4.0GB (RAM usage will spike during parsing)

During parsing VSCode can become unresponsive.
Environment data

Language Server version: Pylance language server 2022.1.1 (pyright 4f61d443)
OS and version: Windows 10 PC, VS Code 1.63.2
Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.10

requirements are set in setup.py for each package. Nothing out of the ordinary, packages can depend on another including dependencies on third party libraries (i.e. numpy, pandas, scipy, matplotlib, etc)
our folder structure is:

Expected behaviour
Expect parsing of types for the saved file. On opening a new file with a dependency on changes to the type would be recognised.
Potentially, parsing all libraries in the workspace which have a dependency on the changed file is required,
Actual behaviour
All files from all libraries irrespective if they have a dependency on the saved file are parsed.
All stub files in c:\Users<User>.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\ are parsed.
All files in  d :\repos\XXX.venv\Lib\site-packages\ are parsed.
Logs
These are logs produced simply from opening a file.
Background analysis message: setFileOpened
Background analysis message: markFilesDirty
[FG] parsing: d:\repos\XXX\XXX.py (5ms)
[FG] binding: d:\repos\XXX\XXX.py (1ms)
Background analysis message: analyze
[BG(4)] analyzing: d:\repos\XXX\XXX.py ...

[BG(11)] parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi [fs read 1ms] (67ms)
[BG(15)] parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi [fs read 1ms] (83ms)
[BG(2)] parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi [fs read 1ms] (79ms)
[BG(6)] binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi (45ms)
...
...
[IDX(251)]     parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\openpyxl\__init__.pyi [fs read 0ms] (81ms)
[IDX(251)]     binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\openpyxl\__init__.pyi (1ms)
[IDX(251)]     indexing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\openpyxl\__init__.pyi ...
...
[IDX(251)]       parsing: d:\repos\XXX\.venv\Lib\site-packages\openpyxl\compat\numbers.py [fs read 1ms] (19ms)
[IDX(251)]       binding: d:\repos\XXX\.venv\Lib\site-packages\openpyxl\compat\numbers.py (1ms)

Plus many many more logs including all files in d:\repos

What setting are you using for python.analysis.diagnosticMode? Are you using the default value of "openFilesOnly", or are you using "workspace"? If you're using "openFilesOnly", the most you should see is a reanalysis of the files that are open. If you are using "workspace", then modification of a file should trigger reanalysis of any files that depend on the modified file. If it's a "leaf file", one which no other files depend upon, then it should be very fast. If it's a core file (like the __init__.py in a module that is directly or indirectly referenced by thousands of other source files), then all reanalysis can be quite expensive.
Parsing or binding should not be done again for any file other than the one whose contents were modified. The output of the parser and binder are cached in memory for all files unless/until the contents of the file is modified. The only exception is if heap space grows past a high-water mark (which is around 1.5GB), in which case the in-memory cache of already-parsed file information is purged to avoid heap overflow. If this occurs, you should see a line in the log file indicating such.
Pylance is an on-demand (lazy) analyzer. If you open a source file that imports other files, pylance will parse and bind the imported file but not type check it. It will evaluate type information selectively — only for symbols that are referenced by the opened file. It needs to do this recursively, of course. So if you're importing from a large, complex library with deep module dependencies, it may need to do significant work to determine the type of an imported symbol. This can be especially expensive if the library is missing type annotations because it then needs to fall back on type inference to determine types as best it can. If you would prefer that pylance avoid any attempt to infer types from unannotated libraries, you can disable the setting python.analysis.useLibraryCodeForTypes.
How big is your project in terms of source files?
By any chance, is it an open-source code base that's publicly available? If so, it would be useful to have repro instructions. If it's private, then we won't be able to repro it directly, and we'll need to rely on your assistance to further diagnose the issue.

python.analysis.diagnosticModel is set to openFilesOnly
ptyhon.analysis.useLibraryCodeFortypes is set to true
It is a flat structure such that most modules/classes are referenced in the main init.py and we're using absolute imports which refer back to this main init.py.  However, I still don't expect changes to unrelated packages to trigger a parse of all packages within the workspace.
After many parsers over the files I have caught the in-memory cache dump, however, this does not occur that regularly.
The library does include unannotated code, as we started with unannotated code, but i guess this is now <10% of the code base. The workspace as a whole excluding projects in other languages would include <1000 python the majority being <100 lines with larger files existing of 5000. I'll get some more concrete numbers

Ok 1677 python files totaling 25mb.

OK, thanks for the additional details. With openFilesOnly, the file count is less important because type checking will be performed only for the open files.
You said "the library does included unannotated code". Are you referring to a library that your project is importing (and is installed in your python environment), or does "the library" refer to your project itself?
I think it would help if you could post a log that contains all of the lines for a single file modification — or at least a few hundred lines with none of them replaced with intervening "..."'s. It's fine to redact privacy-sensitive information, of course.
I'm interested in a couple of things. First, how many of the lines are labeled with "IDX", which refers to the background indexing mechanism, and how many are labeled with "BG" which is the normal analysis logic (done on a background thread to allow for cancellation from the foreground thread). If many (most?) of the lines refer to "IDX", I'll ask you to temporarily set "python.analysis.indexing" to false to see if that noticeably changes the behavior.
Second, I'm interested in seeing if there are unexpected "parsing" and "binding" operations logged. After a file's contents are modified, I'd expect to see it parsed and bound, but other files should not need to be parsed or bound because they should be in memory still.
Great github username, BTW. :) I'm a big fan of "O Brother, Where Art Though" (the movie and the soundtrack).

Thanks for the quick responses. Great movie, great soundtrack, for those who don't get the reference, they're missing out.
I'll be back at my PC hopefully later and provide some more logs. I have thought the issues are linked to the way we have handled imports, however, i have seen open source projects use similar... I've found python projects aren't terribly consistent with imports.

I've jumped back on and I deleted my .venv which in turn meant no python interpreter is selected. Now things are responding as expected CPU usage typically sits at 7% during typing/file save with spikes not exceeding 10%. RAM usage is not building at sitting at 1GB.
I made changes to the options.py file and everything looks great!
Background analysis message: setFileOpened
Background analysis message: markFilesDirty
[FG] parsing: d:\repos\package\model\src\core\lib\models\design\options.py (4ms)
[FG] binding: d:\repos\package\model\src\core\lib\models\design\options.py (1ms)
Background analysis message: getSemanticTokens delta
[BG(4)] getSemanticTokens delta previousResultId:1642750642127 at d:\repos\package\model\src\core\lib\models\design\options.py ...
[BG(4)]   parsing: d:\repos\package\model\src\core\lib\models\design\options.py (1ms)
[BG(4)]   binding: d:\repos\package\model\src\core\lib\models\design\options.py (0ms)
[BG(4)] getSemanticTokens delta previousResultId:1642750642127 at d:\repos\package\model\src\core\lib\models\design\options.py (7ms)
Background analysis message: analyze
[BG(4)] analyzing: d:\repos\package\model\src\core\lib\models\model.py ...
[BG(4)]   checking: d:\repos\package\model\src\core\lib\models\model.py (67ms)
[BG(4)] analyzing: d:\repos\package\model\src\core\lib\models\model.py (191ms)
Background analysis message: resumeAnalysis
[BG(4)] analyzing: d:\repos\package\model\src\core\lib\models\design\options.py ...
[BG(4)]   checking: d:\repos\package\model\src\core\lib\models\design\options.py (1ms)
[BG(4)] analyzing: d:\repos\package\model\src\core\lib\models\design\options.py (130ms)
Background analysis message: resumeAnalysis
[BG(4)] indexing: d:\repos\package\model\src\core\lib\models\design\options.py [found 1] (0ms)
Indexing Done: d:\repos\package\model\src\core\lib\models\design\options.py
Background analysis message: setFileOpened
Background analysis message: markFilesDirty
[FG] parsing: d:\repos\package\model\src\core\lib\models\design\options.py (1ms)
[FG] binding: d:\repos\package\model\src\core\lib\models\design\options.py (1ms)
Background analysis message: analyze
[BG(4)] analyzing: d:\repos\package\model\src\core\lib\models\model.py ...
[BG(4)]   checking: d:\repos\package\model\src\core\lib\models\model.py ...
[BG(4)]     parsing: d:\repos\package\model\src\core\lib\models\design\options.py (1ms)
[BG(4)]     binding: d:\repos\package\model\src\core\lib\models\design\options.py (0ms)
[BG(4)]   checking: d:\repos\package\model\src\core\lib\models\model.py (61ms)
Background analysis message: markFilesDirty
[BG(4)] analyzing: d:\repos\package\model\src\core\lib\models\model.py (216ms)
Background analysis message: getDiagnosticsForRange
Background analysis message: getSemanticTokens delta
[BG(4)] getSemanticTokens delta previousResultId:1642750733679 at d:\repos\package\model\src\core\lib\models\design\options.py (2ms)
Background analysis message: markFilesDirty
Background analysis message: resumeAnalysis
[BG(4)] analyzing: d:\repos\package\model\src\core\lib\models\design\options.py ...
[BG(4)]   checking: d:\repos\package\model\src\core\lib\models\design\options.py (1ms)
[BG(4)] analyzing: d:\repos\package\model\src\core\lib\models\design\options.py (129ms)
Background analysis message: getDiagnosticsForRange
Background analysis message: resumeAnalysis
[BG(4)] indexing: d:\repos\package\model\src\core\lib\models\design\options.py [found 1] (0ms)
Indexing Done: d:\repos\package\model\src\core\lib\models\design\options.py
Background analysis message: analyze
Background analysis message: analyze

Now after i set up my .venv (which is a folder in the workspace) and set my python interpreter to .venv\Scripts\python.exe the behaviour change to this (FYI not the full logs)
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: setFileOpened
Background analysis message: markFilesDirty
[FG] parsing: d:\repos\package\core\src\core\lib\models\design\options.py (8ms)
[FG] binding: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
Background analysis message: getSemanticTokens delta
[BG(4)] getSemanticTokens delta previousResultId:1642751375311 at d:\repos\package\core\src\core\lib\models\design\options.py ...
[BG(4)]   parsing: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
[BG(4)]   binding: d:\repos\package\core\src\core\lib\models\design\options.py (0ms)
[BG(4)] getSemanticTokens delta previousResultId:1642751375311 at d:\repos\package\core\src\core\lib\models\design\options.py (11ms)
Background analysis message: analyze
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\model.py ...
[BG(4)]   checking: d:\repos\package\core\src\core\lib\models\model.py (117ms)
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\model.py (246ms)
Background analysis message: resumeAnalysis
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py ...
[BG(4)]   checking: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py (127ms)
Background analysis message: resumeAnalysis
[BG(4)] indexing: d:\repos\package\core\src\core\lib\models\design\options.py [found 1] (0ms)
Indexing Done: d:\repos\package\core\src\core\lib\models\design\options.py
Background analysis message: setFileOpened
Background analysis message: markFilesDirty
[FG] parsing: d:\repos\package\core\src\core\lib\models\design\options.py (3ms)
[FG] binding: d:\repos\package\core\src\core\lib\models\design\options.py (0ms)
Background analysis message: analyze
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\model.py ...
[BG(4)]   checking: d:\repos\package\core\src\core\lib\models\model.py ...
[BG(4)]     parsing: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
[BG(4)]     binding: d:\repos\package\core\src\core\lib\models\design\options.py (0ms)
[BG(4)]   checking: d:\repos\package\core\src\core\lib\models\model.py (132ms)
Background analysis message: markFilesDirty
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\model.py (369ms)
Background analysis message: getSemanticTokens delta
[BG(4)] getSemanticTokens delta previousResultId:1642751431261 at d:\repos\package\core\src\model\lib\models\design\options.py (4ms)
Background analysis message: markFilesDirty
Background analysis message: getDiagnosticsForRange
Background analysis message: resumeAnalysis
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py ...
[BG(4)]   checking: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
[BG(4)] indexing: d:\repos\package\core\src\core\lib\models\design\options.py (0ms)
Background analysis message: getDiagnosticsForRange
Background analysis message: analyze
Background analysis message: analyze
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py ...
[BG(4)]   checking: d:\repos\package\core\src\core\lib\models\design\options.py (8ms)
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py (230ms)
Background analysis message: resumeAnalysis
Background analysis message: markFilesDirty
Background analysis message: markFilesDirty
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: invalidateAndForceReanalysis
[Info  - 8:50:34 pm] Searching for source files
[Info  - 8:50:34 pm] Auto-excluding d:\repos\package\.venv
[Info  - 8:50:34 pm] Found 1677 source files
Background analysis message: setTrackedFiles
Background analysis message: markAllFilesDirty
Background analysis message: analyze
[BG(17)] parsing: d:\repos\package\AddIn\package.py [fs read 1ms] (18ms)
[BG(17)] parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi [fs read 0ms] (91ms)
[BG(17)] binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi (44ms)
[BG(17)] binding: d:\repos\package\AddIn\package.py (3ms)
[BG(17)] indexing: d:\repos\package\AddIn\package.py [found 6] (0ms)
Indexing Done: d:\repos\package\AddIn\package.py
[BG(17)] parsing: d:\repos\package\AddIn\bin\Debug\Python\package.py [fs read 1ms] (10ms)
[BG(17)] binding: d:\repos\package\AddIn\bin\Debug\Python\package.py (1ms)
[BG(17)] indexing: d:\repos\package\AddIn\bin\Debug\Python\package.py [found 6] (0ms)
Indexing Done: d:\repos\package\AddIn\bin\Debug\Python\package.py
...
...
...

the ... is repeating this behaviour with every single .py in the workspace producing about 1000s of logs.
 This is still not ideal compared to the behaviour/responsiveness prior to setting up the venv but still very managable.
Now i have closed VSCode and re-opened it, changed nothing. I add a space in a file (i.e. the options.py file is the one I'm using, i save the file now i am experiencing the same behaviour my issue noted. Here are some more logs (there are 33000+ in total).
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: setFileOpened
Background analysis message: markFilesDirty
[FG] parsing: d:\repos\package\core\src\core\lib\models\design\options.py (8ms)
[FG] binding: d:\repos\package\core\src\core\lib\models\design\options.py (3ms)
Background analysis message: analyze
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py ...
[BG(4)]   parsing: d:\repos\package\core\src\core\lib\models\design\options.py (3ms)
[BG(4)]   binding: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
[BG(4)]   checking: d:\repos\package\core\src\core\lib\models\design\options.py (38ms)
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py (162ms)
Background analysis message: getSemanticTokens delta
[BG(4)] getSemanticTokens delta previousResultId:1642751545222 at d:\repos\package\core\src\core\lib\models\design\options.py (4ms)
Background analysis message: resumeAnalysis
[BG(4)] indexing: d:\repos\package\core\src\core\lib\models\design\options.py [found 1] (0ms)
Indexing Done: d:\repos\package\core\src\core\lib\models\design\options.py
Background analysis message: setFileOpened
Background analysis message: markFilesDirty
[FG] parsing: d:\repos\package\core\src\core\lib\models\design\options.py (6ms)
[FG] binding: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
Background analysis message: analyze
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py ...
[BG(4)]   parsing: d:\repos\package\core\src\core\lib\models\design\options.py (4ms)
[BG(4)]   binding: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
[BG(4)]   checking: d:\repos\package\core\src\core\lib\models\design\options.py (68ms)
Background analysis message: markFilesDirty
[BG(4)] analyzing: d:\repos\package\core\src\core\lib\models\design\options.py (467ms)
Background analysis message: getSemanticTokens delta
[BG(4)] getSemanticTokens delta previousResultId:1642751637839 at d:\repos\package\core\src\core\lib\models\design\options.py (10ms)
Background analysis message: getDiagnosticsForRange
Background analysis message: markFilesDirty
Background analysis message: resumeAnalysis
Background analysis message: analyze
[BG(4)] indexing: d:\repos\package\core\src\core\lib\models\design\options.py (1ms)
Background analysis message: analyze
Background analysis message: getDiagnosticsForRange
Background analysis message: markFilesDirty
Background analysis message: markFilesDirty
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: invalidateAndForceReanalysis
[Info  - 8:54:01 pm] Searching for source files
[Info  - 8:54:01 pm] Auto-excluding d:\repos\package\.venv
[Info  - 8:54:02 pm] Found 1677 source files
Background analysis message: setTrackedFiles
Background analysis message: markAllFilesDirty
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: getDiagnosticsForRange
Background analysis message: invalidateAndForceReanalysis
Background analysis message: invalidateAndForceReanalysis
Background analysis message: getDiagnosticsForRange
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
Background analysis message: analyze
[BG(8)] parsing: d:\repos\package\processor\src\processor\__init__.py [fs read 1ms] (481ms)
[BG(15)] parsing: d:\repos\package\functional_tests\tests\__init__.py [fs read 1ms] (482ms)
[BG(14)] parsing: d:\repos\package\hub\src\hub\__init__.py [fs read 1ms] (565ms)
[BG(11)] parsing: d:\repos\package\tools\src\tools\__init__.py [fs read 1ms] (626ms)
[BG(12)] parsing: d:\repos\package\viewer\src\__viewer_pyinstaller\__init__.py [fs read 1ms] (643ms)
[BG(16)] parsing: d:\repos\package\developer\example.py [fs read 1ms] (664ms)
[BG(6)] parsing: d:\repos\package\model_interactor\src\model_interactor\__init__.py [fs read 0ms] (666ms)
[BG(2)] parsing: d:\repos\package\controllers\src\controllers\__init__.py [fs read 1ms] (734ms)
[BG(1)] parsing: d:\repos\package\library\src\library\__init__.py [fs read 0ms] (761ms)
[BG(5)] parsing: d:\repos\package\export\src\export\__init__.py [fs read 1ms] (757ms)
[BG(9)] parsing: d:\repos\package\jobs\src\jobs\__init__.py [fs read 1ms] (799ms)
[BG(7)] parsing: d:\repos\package\modeller\src\modeller\__init__.py [fs read 1ms] (915ms)
[BG(8)] parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi [fs read 50ms] (494ms)
[BG(16)] parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi [fs read 2ms] (360ms)
[BG(8)] binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi (232ms)
[BG(8)] binding: d:\repos\package\processor\src\processor\__init__.py (0ms)
[BG(8)] indexing: d:\repos\package\processor\src\processor\__init__.py [found 0] (0ms)
Indexing Done: d:\repos\package\processor\src\processor\__init__.py
[BG(7)] parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi [fs read 2ms] (344ms)
[BG(16)] binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi (225ms)
[BG(8)] parsing: d:\repos\package\processor\src\processor\_version.py [fs read 1ms] (143ms)
[BG(16)] binding: d:\repos\package\developer\example.py (111ms)
[BG(16)] indexing: d:\repos\package\developer\example.py [found 21] (1ms)
Indexing Done: d:\repos\package\developer\example.py
[BG(16)] parsing: d:\repos\package\developer\examples.py [fs read 1ms] (11ms)
[BG(16)] binding: d:\repos\package\developer\examples.py (4ms)
[BG(16)] indexing: d:\repos\package\developer\examples.py [found 46] (2ms)
Indexing Done: d:\repos\package\developer\examples.py
[BG(16)] parsing: d:\repos\package\developer\pythonnet.py [fs read 0ms] (1ms)
[BG(16)] binding: d:\repos\package\developer\pythonnet.py (0ms)
[BG(16)] indexing: d:\repos\package\developer\pythonnet.py [found 0] (0ms)
Indexing Done: d:\repos\package\developer\pythonnet.py
[BG(11)] parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\typeshed-fallback\stdlib\builtins.pyi [fs read 1ms] (806ms)
[BG(8)] binding: d:\repos\package\processor\src\processor\_version.py (102ms)
[BG(8)] indexing: d:\repos\package\processor\src\processor\_version.py [found 21] (2ms)
Indexing Done: d:\repos\package\processor\src\processor\_version.py
...
...
...
[Info  - 8:54:06 pm] Indexer background runner(42) root directory: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist
[Info  - 8:54:06 pm] Indexing(42) started
[Info  - 8:54:06 pm] Indexer background runner(36) root directory: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist
[Info  - 8:54:06 pm] Indexing(36) started
[BG(9)] parsing: d:\repos\package\jobs\src\jobs\main.py [fs read 1ms] (145ms)
[BG(9)] binding: d:\repos\package\jobs\src\jobs\main.py (0ms)
[BG(9)] indexing: d:\repos\package\jobs\src\jobs\main.py [found 1] (1ms)
Indexing Done: d:\repos\package\jobs\src\jobs\main.py
...
...
...

[IDX(40)]     parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\animation.pyi [fs read 2ms] (99ms)
[IDX(40)]     binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\animation.pyi (1ms)
[IDX(40)]     indexing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\animation.pyi [found 22] (1ms)
[IDX(40)]     parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\artist.pyi [fs read 0ms] (1ms)
[IDX(40)]     binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\artist.pyi (0ms)
[IDX(40)]     indexing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\artist.pyi [found 2] (0ms)
[IDX(40)]     parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\__init__.pyi [fs read 0ms] (0ms)
[IDX(40)]     binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\__init__.pyi ...
[IDX(40)]       parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\_subplots.pyi [fs read 0ms] (2ms)
[IDX(40)]       binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\_subplots.pyi (1ms)
[IDX(37)]     indexing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\animation.pyi [found 22] (61ms)
[IDX(37)]     parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\artist.pyi [fs read 0ms] (1ms)
[IDX(37)]     binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\artist.pyi (1ms)
[IDX(37)]     indexing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\artist.pyi [found 2] (0ms)
[IDX(37)]     parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\__init__.pyi [fs read 0ms] (2ms)
[IDX(37)]     binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\__init__.pyi ...
[IDX(37)]       parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\_subplots.pyi [fs read 1ms] (3ms)
[IDX(37)]       binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\_subplots.pyi (0ms)
...
...
...
[IDX(42)]       parsing: d:\repos\package\.venv\Lib\site-packages\execnet\gateway_base.py [fs read 1ms] (182ms)
[IDX(46)]       parsing: d:\repos\package\.venv\Lib\site-packages\flask_restx\marshalling.py [fs read 1ms] (69ms)
[IDX(46)]       binding: d:\repos\package\.venv\Lib\site-packages\flask_restx\marshalling.py (1ms)
[IDX(38)]     indexing: d:\repos\package\.venv\Lib\site-packages\cryptography\__init__.py ...
[IDX(38)]       parsing: d:\repos\package\.venv\Lib\site-packages\cryptography\__about__.py [fs read 1ms] (50ms)
[IDX(38)]       binding: d:\repos\package\.venv\Lib\site-packages\cryptography\__about__.py (1ms)
[IDX(38)]     indexing: d:\repos\package\.venv\Lib\site-packages\cryptography\__init__.py [found 9] (52ms)
[IDX(38)]     parsing: d:\repos\package\.venv\Lib\site-packages\dateutil\__init__.py [fs read 0ms] (2ms)
[IDX(38)]     binding: d:\repos\package\.venv\Lib\site-packages\dateutil\__init__.py (0ms)
[IDX(38)]     indexing: d:\repos\package\.venv\Lib\site-packages\dateutil\__init__.py [found 0] (1ms)
[IDX(38)]     parsing: d:\repos\package\.venv\Lib\site-packages\execnet\__init__.py [fs read 0ms] (4ms)
[IDX(38)]     binding: d:\repos\package\.venv\Lib\site-packages\execnet\__init__.py (1ms)
[IDX(38)]     indexing: d:\repos\package\.venv\Lib\site-packages\execnet\__init__.py ...
[IDX(38)]       parsing: d:\repos\package\.venv\Lib\site-packages\execnet\_version.py [fs read 0ms] (1ms)
[IDX(38)]       binding: d:\repos\package\.venv\Lib\site-packages\execnet\_version.py (1ms)
[IDX(38)]       parsing: d:\repos\package\.venv\Lib\site-packages\execnet\deprecated.py [fs read 0ms] (8ms)
[IDX(38)]       binding: d:\repos\package\.venv\Lib\site-packages\execnet\deprecated.py (0ms)

This following behaviour is very prevalent. Some of the time taken is quite large compared to the files in our project.
Both this directory c:\Users\user.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs
and d:\repos\package.venv\Lib\site-packages\
[IDX(44)]       parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\_axes.pyi [fs read 0ms] (787ms)
[IDX(44)]       binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\_axes.pyi (1ms)
[IDX(44)]     binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\__init__.pyi (794ms)
[IDX(44)]     indexing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\__init__.pyi [found 5] (1ms)
[IDX(43)]       parsing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\_axes.pyi [fs read 1ms] (485ms)
[IDX(43)]       binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\_axes.pyi (1ms)
[IDX(43)]     binding: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\__init__.pyi (489ms)
[IDX(43)]     indexing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\matplotlib\axes\__init__.pyi [found 5] (0ms)

I found this particular log in the trace. I believe these show up when logs are set to Information and libraries like numpy, scipy etc all show up.
[Info  - 8:54:20 pm] [IDX(35)] Long operation: indexing: c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\pandas\__init__.pyi (4079ms)
another example
[Info  - 8:54:21 pm] [IDX(47)] Long operation: indexing: c:\Users\michaelo\.vscode\extensions\ms-python.vscode-pylance-2022.1.1\dist\bundled\stubs\pandas\__init__.pyi (4728ms)
Hope this is enough let me know if you want more.

@heejaechang, it looks like significant (unexpected) activity is being done by the indexer. I wouldn't expect that we'd need to reindex anything in the venv based on a modification to a project source file. Could you take a closer look here?

I feels like files in .venv folder is recognized as user files rather than excluded as virtual env folder. Can u provide either all logs or the begining of the logs?

alright, root cause seems invalidateAndForceReanalysis and we need to figure out why that is requested.
can you create a file called "pyrightconfig.json" at the root of your workspace. Copy the following into that file:
{
"verboseOutput": true
}
and provide us full log, not the part of logs like you shared above, but full log which include starting of pylance server.

Here are the logs the on_open is what is produced upon opening VSCode the second set are what occur when a file is changed.
on_open.txt
file_changed.txt
in our .venv we install our packages as links via pip install -e <package_name>

@SoggyBottomBoy Thank you very much for the logs!, it looks like you are using multi root workspace with extraPaths?
so, the initial cost is due to us going through all workspace roots for indexing. In our current design, each workspace root starts its own python server so nothing is shared between them even if they share same python interpreter (virtual environment), so it ends up index libraries per workspace root. that seems the root cause of the cost.
the cost on changing file seems due to multiple issues.
first one is [Info  - 3:00:30 pm] SourceFile: Received fs event 'change' for path 'd:\repos\solution\model.pytest_cache\v' and similar fs events.
Background analysis message: invalidateAndForceReanalysis
it causes us to abandon current data and restart from scratch for all user files.
second one is [Info  - 3:00:29 pm] SourceFile: Received fs event 'change' for path 'd:\repos\solution\model.vscode\settings.json' and similar fs events.
this one is even more serious since it will cause all third party library info to be dropped and re-parsed.
this is what I have so far. for now, you can put this "python.analysis.indexing": false in your settings section in workspace file to turn off indexing that is causing high CPU.
I will take a look a bit more to see what we can do to make this situation better.
by the way, can you share your workspace file for multi root workspace for me to see what options you have set?
thank you

Thanks for looking into it further.
workspace.txt

@SoggyBottomBoy you don't have extraPaths? or did you put in pyrightconfig.json? if you are using pyrightconfig.json for something other than verbose, can you share one of pyrightconfig as well?

Nothing is set in the workspace root (with exception of the verbose setting). In each project is a pyrightconfig.json
Each project includes a settings.json, but this is only to manage whether pyright is used for test. i.e contents of settings.json is
{
    "python.testing.pytestEnabled": true,
}

The root level settings is only used to hide the folders which show up under the workspace folder alias "root" so that they don't appear twice and hide other project in C# for which we opt to use visual studio.
pyrightconfig.txt

@heejaechang, thanks for the analysis. I have a few questions though.
We install two fs event handlers. The first one covers the project roots and therefore includes the project source files, and the second covers any library sources that are outside of the project roots.
It looks to me (correct me if I'm wrong) that all of the fs events in these logs are being handled by the first of these two handlers. I think that all of the paths in the logs are under the project root directories. The logic for the first handler ignores all fs events for files unless the path end in ".py" or ".pyi". If I'm correct, then it is simply ignoring all of the "*.json" modifications.
The directory creation events for **\.pytest_cache\** are problematic though. I think this is the real source of the problem. How do we work around this? We can't ignore directory creation/deletion in general, but we might want to add a heuristic for some common known-to-be-safe directory names like ".pytest_cache". Adding special cases like this is not a great solution because there will always be exceptions that we don't know about, but I can't think of a better answer in this case. Any other ideas?
The other thing that sticks out to me is that we seem to be discarding more index data than necessary. If we receive an fs event for a path in the project sources, we may need to re-index the project sources, but we shouldn't need to re-index the imported libraries or the bundled stubs. Could we be more surgical in what we discard and re-index?

@SoggyBottomBoy, do you know why pytest is updating its cache on ever file save? Do you have your project set up to run tests on every file modification? That might explain why you're seeing this issue when most other pylance users are not.

The first one covers the project roots and therefore includes the project source files, and the second covers any library sources that are outside of the project roots.

Pylance only uses the first one, we don't use the second one. we have PR to re-enable outside of workspace file watcher but still not finished.

If I'm correct, then it is simply ignoring all of the "*.json" modifications.

if settings.json is changed, vscode (LSP) will call configuration change message which, in turn, will invalidate everything (InvalidateAndForceReanalysis)

The directory creation events for *.pytest_cache* are problematic though. I think this is the real source of the problem.

that cause all user files to be re-indexed. how to fix it, I am not sure yet.

If we receive an fs event for a path in the project sources, we may need to re-index the project sources, but we shouldn't need to re-index the imported libraries or the bundled stubs. Could we be more surgical in what we discard and re-index?

it already does. third party library re-index require either setting change (vscode calling us to tell configuration is changed) or file change in library path.
one of reasons why I want to see whether user is using extra path.
from logs, I see this
Search paths for d:\repos\solution\library
  c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.3\dist\typeshed-fallback\stdlib
  d:\repos\solution\library
  d:\repos\solution\library\typings
  c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.3\dist\typeshed-fallback\stubs\...
  c:\Users\user\.vscode\extensions\ms-python.vscode-pylance-2022.1.3\dist\bundled\stubs
  c:\Program Files\Python38\DLLs
  c:\Program Files\Python38\Lib
  c:\Program Files\Python38
  d:\repos\solution\.venv
  d:\repos\solution\.venv\Lib\site-packages
  d:\repos\solution\helpers\src
  d:\repos\solution\tools\src
  d:\repos\solution\model\src
  d:\repos\solution\library\src
  d:\repos\solution\processor\src
  d:\repos\solution\modeller\src
  d:\repos\solution\jobs\src
  d:\repos\solution\viewer\src
  d:\repos\solution\interactor\src
  d:\repos\solution\exporter\src
  d:\repos\solution\designer\src
  d:\repos\solution\.venv\Lib\site-packages\win32
  d:\repos\solution\.venv\Lib\site-packages\win32\lib
  d:\repos\solution\.venv\Lib\site-packages\pythonwin

so some of them seems indicate the user are using extra path, but it might got there using different mechanism.

We only run tests manually (except in our pipelines). We use a test explorer extension, i was unaware by this does refresh tests each time a file is saved.

Apologies i thought the test explorer was an extension. Does this come packed with the Python extension from microsoft?

@SoggyBottomBoy I couldn't figure out how you put these in your search path without extraPath. did you do something to put these in python search path?
  d:\repos\solution\helpers\src
  d:\repos\solution\tools\src
  d:\repos\solution\model\src
  d:\repos\solution\library\src
  d:\repos\solution\processor\src
  d:\repos\solution\modeller\src
  d:\repos\solution\jobs\src
  d:\repos\solution\viewer\src
  d:\repos\solution\interactor\src
  d:\repos\solution\exporter\src
  d:\repos\solution\designer\src

by the way, your workspace setting below cause us to create python server for solution folder on top of one per each projects.
{ "name": "root", "path": "." } part.

can I ask why you need this? when you already have one per each projects?

We have some assets, CI/CD configuration and global config (i.e. a dev-requirements.txt) in the root that we manage along with our documentation.
I believe what you're asking re the python search path is that our task to setup out venv will run the command pip install -e <project> which adds .egg-link into site-packages for all of our projects

Thank you, I think I have all info I need. I will get back to you once I have something.

Perfect! Thanks for your time.

@SoggyBottomBoy I have been playing with your setup and have one question. is there a reason why you have this multiple root workspace setup? when you have also root (solution) as one of workspace root?
some of this complication comes from the fact that you are using multi root workspace + having root (solution in your case) as one of workspace root as well.
I tried your setup and wasn't sure why you need all those other roots (projects) when opening solution as root, I could do all things we can do with multi roots setup.

Hi @heejaechang, thanks for looking into this. I work on the project with @SoggyBottomBoy and set it up like that.
At the time, we had lots of our tool configurations for each project in setup.cfg, e.g., flake8 and the Python plugin wouldn't pick those up unless the file was in the root of the workspace folder. We have since then moved the most of the configurations out into separate config files, e.g., .flake8, so perhaps they could all be merged into one file at the repo root, although, in some ways it's nice to have each project with it's own configuration.
I'll check on Monday when I'm back at work how this works.

@heejaechang Getting rid of the multiroot mostly seems to work. These are fine

I can merge the individual .flake8 files using per-file-ignores where appropriate.
.isoft.cfg is the same for every project so that should be fine, but perhaps would be an issue if it wasn't.
.pylintrc is the same for every project and we can override by adding a file in the python package where necessary.

These aren't quite:

pyrightconfig.json is different for each project as some are slower to catch up with the stricter typing we are adding. I can't see how to apply different settings to different files though. Is it possible?
pytest.ini is different for some projects but if we had to, we could make them the same.

@erictraut is there a way to set different type checking mode per sub dir?

Yes, to some extent. You can specify specific subdirectories or files that should use "strict" mode. See https://github.com/microsoft/pyright/blob/main/docs/configuration.md#main-pyright-config-options for details. The intent is that you can use less-strict settings for most of the files in your code base but gradually move to strict typing over time (e.g. as you rewrite or add new subcomponents).

@parched by the way, I think you guys should add "**/.pytest_cache" to the exclude list so that we don't pick up changes in the cache as user file change.

We should consider auto-excluding ".pytest_cache" in pylance & pyright.

part of this issue is related to https://github.com/microsoft/pylance-release/issues/2318

@parched can you check the latest pylance release (2022.2.0) to see whether perf experience is better than before?

we are putting more mitigation for this situation. basically, we now will only index workspace (root) that has file opened. so in your case, only workspace that currently actively used will be indexed otherwise, it will be postponed until a file is opened for the workspace.

This issue has been fixed in version 2022.2.1, which we've just released. You can find the changelog here: CHANGELOG.md