Enable syntax highlighting in JupyterLab by default

I think Haskell syntax highlighting should be enabled by default. One way to do that would be to insert a switch in kernel.json, maybe like
++ ["--codemirror", "Haskell"]

after the --ghclib argument.
https://github.com/gibiansky/IHaskell/blob/ef698157f44960566687a308e3455b5ba031eb43/src/IHaskell/IPython.hs#L140-L144
But if we did this, then how would syntax highlighting be disabled?
Maybe there's a more natural way to accomplish this.
Reference: https://github.com/gibiansky/IHaskell/pull/1191

I have some observations to make about the current behavior of ihaskell Codemirror on the master branch:

--codemirror haskell will enable syntax highlighting in Lab.
--codemirror Haskell will enable syntax highlighting in Lab.
--codemirror ihaskell will not enable syntax highlighting in Lab.
In the classic Notebook no switch is needed, Codemirror is already enabled.

Soo... what's going on here is that JupyterLab now, by default, already sets notebook cells to
"language_info": {
   "codemirror_mode": "ihaskell",

But the kernel.js script for the special "ihaskell" multiplexing mode isn't working anymore.

I just realized that the special "ihaskell" Codemirror mode in kernel.js was never working in JupyterLab.
kernel.js is for classic notebook only.
The commandline switch which @MMesch added in #1191 allows us to specify "haskell" mode for Codemirror, which works because Codemirror comes with "haskell" mode built-in. But that mode will highlight command code cells prefixed with : as if they were Haskell. Which is the special case that the special "ihaskell" mode addresses.

Apparently adding Codemirror modes to kernels in JupyterLab is a secret art, but here are some clues:
https://github.com/jupyterlab/jupyterlab/pull/9067
https://github.com/jupyterlab/jupyterlab/pull/9123
https://github.com/jupyterlab/jupyterlab/issues/10095
https://discourse.jupyter.org/t/how-to-find-the-codemirror-instance/9275

I think that the extension which we already have is the best way to add Codemirror modes to kernels in JupyterLab: https://github.com/gibiansky/IHaskell/blob/master/jupyterlab-ihaskell/src/codemirror-ihaskell.ts

What is this about?

A kernel may specify a codemirror-mode.js

https://github.com/jupyterlab/jupyterlab/issues/3010

Also note the extension directory name was changed in https://github.com/gibiansky/IHaskell/commit/e108c9cd166777c5aeb15e6485a1bc49d6f82818
so IHaskell README needs update.

I don't follow, the README specifies jupyterlab-ihaskell. Am I misunderstanding?

The top README: https://github.com/gibiansky/IHaskell/blob/master/README.md#stack-and-docker

Updated.

Thx.
Do you have any thoughts on this issue, Vaibhav? I would like to make Haskell syntax highlighting in JupyterLab work automatically without making users install the labextension. But that doesn't currently work, and I don't see any way to improve on the current situation.

Have you considered using prebuilt extensions to eliminate the need for Node.js and npm?
Your installer would just need to drop the pre-compiled jupyterlab-ihaskell in an appropriated directory. jupyterlab_robotmode uses this approach.

I think this is the problem though, we still need an extra installation step for something that "just works" with the older Classic Notebook interface. I can't see a way around that given the JupyterLab architecture.

Maybe we could get ihaskell install to copy a pre-compiled jupyterlab-ihaskell to the right place?

I tried using a prebuilt version and it does seem easy to install but I ran into this issue: https://discourse.jupyter.org/t/whats-wrong-with-the-way-im-using-codemirror-in-this-prebuilt-version-of-jupyterlab-ihaskell/10539, does anyone have any ideas?

Okay, I've fixed the issue and the prebuilt extension is available here, you should be able to copy it to the labextensions/jupyterlab-ihaskell directory in your jupyter --data-dir (e.g. for me that's ~/.local/share/jupyter/labextensions/jupyterlab-ihaskell). Unfortunately the files contain more than one dot in their filename so because of https://github.com/haskell/cabal/issues/5057 I can't include them in IHaskell without dropping support for Cabal < 3.0. I'll keep thinking about this but for now I can't see a straightforward way past this issue.

Whoops, didn't mean to close this.

I think a potential solution is to use a custom webpack config to remove one of the dots in the filename so we can then use the currently support file globbing functionality. I've asked about this here but any other suggestions are welcome: https://discourse.jupyter.org/t/custom-webpack-configuration-for-jupyterlab-prebuilt-extension-with-special-filename-formatting/10570.

I've implemented this and intend to merge soon.

Fixed by #1270.


[Windows] Add shell symlinks

Description
Add shell symlinks for Windows:

Name
Target

gitbash.exe
C:\Program Files\Git\bin\bash.exe

msys2bash.cmd
C:\msys64\usr\bin\bash.exe

wslbash.exe
C:\windows\System32\bash.exe

Related issue:
https://github.com/actions/virtual-environments/issues/1525
Check list

[x] Related issue / work item is attached
[x] Tests are written (if applicable)
[x] Documentation is updated (if applicable)
[ ] Changes are tested and related VM images are successfully generated

@eine, Could you please review this PR?

@eine, Could you please review this PR?

@al-cheb, I believe that C:\msys64\usr\bin\bash.exe and C:\msys64\usr\bin\sh.exe are, in fact, the same shell. I don't think it's necessary to add both of them. @lazka, wdyt?
At the same time, I would suggest adding a wrapper, instead of symlinking directly. Based on https://github.com/msys2/setup-msys2/blob/master/main.js#L165-L169, the wrapper might be msys2bash.cmd:
@echo off
setloca
IF NOT DEFINED MSYS2_PATH_TYPE set MSYS2_PATH_TYPE=strict
set CHERE_INVOKING=1
C:\msys64\usr\bin\bash.exe -leo pipefail %*

@al-cheb, I believe that C:\msys64\usr\bin\bash.exe and C:\msys64\usr\bin\sh.exe are, in fact, the same shell. I don't think it's necessary to add both of them. @lazka, wdyt?
At the same time, I would suggest adding a wrapper, instead of symlinking directly. Based on https://github.com/msys2/setup-msys2/blob/master/main.js#L165-L169, the wrapper might be msys2bash.cmd:
@echo off
setloca
IF NOT DEFINED MSYS2_PATH_TYPE set MSYS2_PATH_TYPE=strict
set CHERE_INVOKING=1
C:\msys64\usr\bin\bash.exe -leo pipefail %*

@al-cheb, I believe that C:\msys64\usr\bin\bash.exe and C:\msys64\usr\bin\sh.exe are, in fact, the same shell. I don't think it's necessary to add both of them. @lazka, wdyt?
At the same time, I would suggest adding a wrapper, instead of symlinking directly. Based on https://github.com/msys2/setup-msys2/blob/master/main.js#L165-L169, the wrapper might be msys2bash.cmd:
@echo off
setlocal
IF NOT DEFINED MSYS2_PATH_TYPE set MSYS2_PATH_TYPE=strict
IF NOT DEFINED MSYSTEM set MSYSTEM=mingw64
set CHERE_INVOKING=1
C:\msys64\usr\bin\bash.exe -leo pipefail %*

Thank you. Remove the link for sh shell and add a wrapper for bash.

@al-cheb, I believe that C:\msys64\usr\bin\bash.exe and C:\msys64\usr\bin\sh.exe are, in fact, the same shell. I don't think it's necessary to add both of them. @lazka, wdyt?
At the same time, I would suggest adding a wrapper, instead of symlinking directly. Based on https://github.com/msys2/setup-msys2/blob/master/main.js#L165-L169, the wrapper might be msys2bash.cmd:
@echo off
setlocal
IF NOT DEFINED MSYS2_PATH_TYPE set MSYS2_PATH_TYPE=strict
IF NOT DEFINED MSYSTEM set MSYSTEM=mingw64
set CHERE_INVOKING=1
C:\msys64\usr\bin\bash.exe -leo pipefail %*

Thank you. Remove the link for sh shell and add a wrapper for bash.


Refactored Select internaly for Arrays to prepare for other changes

I improved the internal structure of the component, because I saw that there where no Array used for the Options.
So there are no visible changes for the users, but the code is working now internally better and faster.
This is the preperation of dynamically loading Options without reinitializing the Component. But I also thougt
about making the Select similiar to the Autocomplete and give a "data" Object which can then be rendered.
Future Plan:
This should also be done in the Autocomplete. In the Autocomplete it will be a breaking change.
I want to make a combination of the Autocomplete and the Multi-Select with dynamic Loading Data
because I need it for my own Project, but I saw that it was already multiple times requested by other users.
For Reference see issues here:
#219
#67
#115
#171
#124
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue).

[ ] New feature (non-breaking change which adds functionality).

[ ] Breaking change (fix or feature that would cause existing functionality to change).

[x] I have read the CONTRIBUTING document.

[ ] My change requires a change to the documentation.

[ ] I have updated the documentation accordingly.

[ ] I have added tests to cover my changes.

[x] All new and existing tests passed.

I think if nobody objects about this, then lets merge this one
thanks btw @wuda-io 👍


Unable to read data using .value property from a BluetoothCharacteristic object.

Hi,
After going through a ton of effort, I'm still unable to read data. Please help!
I'm using this simple code to read data from a characteristic but the value stream seems to be empty.

readCharacteristic = await services
                  .firstWhere((service) => service.uuid == SERVICE_UUID)
                  .getCharacteristic(READ_UUID);

print("read characteristic: ${readCharacteristic.uuid}");
await readCharacteristic.startNotifications();

try {
  await readCharacteristic.value
      .timeout(Duration(seconds: 5))
      .firstWhere((element) {
    print("buffer: ${element.buffer}");
    print("elementSizeInBytes: ${element.elementSizeInBytes}");
    print("lengthInBytes: ${element.lengthInBytes}");
    print("offsetInBytes: ${element.offsetInBytes}");
    return false;
  });
} catch (e) {
  print("err: $e");
}

I printed out the readCharacteristic.uuid to make sure that the characteristic is recognized properly. But when I'm expecting to receive data immediately, after 10 seconds the timeout exceeds and I catch a timeout error which I suppose indicates that the stream was empty until that time. If I'm being correct, at least my terminal should have prompted the properties of whatever ByteData is being received.
I have also tried .lastValue.
To make sure that my transmitter device is indeed transmitting, I successfully read data using two other platforms using a similar method.
Regards.

You might be running in to the bug where the event that listens to value change event has a typo in it (I still need to release a new version for that)
Try readValue() to see if it can read a value at all.

@jeroen1602
Hi, thank you for your response.
I tried readValue() in my code and it threw the error:
Operation not supported for uuid 0000fff1-0000-1000-8000-00805f9b34fb
which is a valid uuid that I've tested before.
I also removed the package from pubspec.yaml and tried to get the latest version from the repo directly:
flutter_web_bluetooth:
    git:
      url: git://github.com/jeroen1602/flutter_web_bluetooth.git

and I got the same result.
Am I missing anything?

Some operations are not supported based on how the characteristic is set up. You should be able to check what is supported by reading the properties of the characteristic.
But did you also try startNotifications() again with the version directly from GitHub? Because that one should work with the GitHub version.

@jeroen1602
Hi, Sorry for the delay!
I downloaded the package from the repo again and monitored the characteristic's properties (the ones I thought might be relevant) with the following code:
if (readCharacteristic != null) {
        print(
            "is notifying: ${readCharacteristic.isNotifying}");
        print(
            "notify: ${readCharacteristic.properties.notify}");
        print(
            "reliableWrite: ${readCharacteristic.properties.reliableWrite}");
        print("read: ${readCharacteristic.properties.read}");
      }

and here are the results:
is notifying: true
notify: true
reliableWrite: false
read: false

Couple of notes:

I called startNotifications() right after declaring the characteristic. The print("is notifying: ${readCharacteristic.isNotifying}"); returns true which I thinkg verifies that.
I previously used this exact characteristic via NRF connect application and flutter blue package before reading from my Bluetooth device. So I'm confident about the UUIDs that I chose. (getCharacteristic() method also did not throw NotFoundError and I confirmed the characteristic by printing its UUID print(readCharacteristic.uuid);)


Typst v12

Hey :)
I haven't read through the Typst changelog (I'm not sure they even have one), but the changes seem straightforward and the readme example works.

Great! Thanks for your contribution again :)


Let media users signup for event

I think it would be nice if those who consider themselves media could sign up as covering an event. This would be a different type of status compared to participating in the event.
We could let media people vote on games. These votes would not be taken into account when ranking games like we do at the end, because this should only be for participants. But we could have a separate media-category (if we have enough grades). The number of grades needs to be lower than normal though. Maybe we don't even show the grade, just the rank to allow for super few grades. My suggestion would be to use the "one angry voter strategy" and let all games, even if 0 votes, be included. It is sensitive to cheaters but it wouldn't hurt trying. A media voter maybe needs to vote on at least 10 different games to be included in the stats. As an example.
One reason to allow this would be that media-people would have access to features such as filtering away games they've already voted on (when that is implemented).
Another would be that the list of what you have graded might be a nice tool for them. Both in remembering what has been played, but also how good a game was.

I got serious backlash when I suggested this before.
Instead what's going to happen is there's going to be a separate category for non-participants.

Alright, that pretty much would have the same effect though. Good.

So i have an idea very similar to pov's, but what i will say is that we should keep this open for at least a couple of LD's and get lots of feedback on it before we implement it. Because being rated by your peers is one of key points of Ludum Dare, we should be careful when making changes around it.
I would propose adding two new categories.
Public and Alumni. The first being open to anyone with an account who hasn't participated in any event and the second being open to anyone who has participated in a previous event but not the current one.
Adding these as separate categories means that the value and meaning of the existing categories don't change. Whilst still encouraging more people to engage with Ludum Dare when there not participating.

The only concern with separating them is the number of "alumni" that rate games in events they didn't participate in are few and far between. Nowhere near enough for a result.
That said, the plan isn't necessarily for a result anyway. Once an LD event has passed, you're going to want to be able to leave a rating on games as a general viewer of the website. That's what this is.

Fair enough. We could let people rate though not do anything with it though in case they want to keep track of their views on the games. We need to be clear though whichever way.


Question about the training labels

I find that the training data is literally from your another repo UWCR. But the annotation in the release dataset doesn't match the template_files you provided in this repo. How can I transfer the raw annotation into the annotation in this repo?

Hi. For your question about annotation conversion, I have made a function called "convert_annotations.py" and added that to my RAMP-CNN github repository.
I hope that helps.


Question: Can this be configured to create a new file at a different interval?

👋Hi, I was wondering if there is an easy way to configure org-journal to create a new journal file every week or month or year etc. rather than every day.
Thank you!

Ah, it's right there in the synopsis! Thank you 😊


GH506: Corrected issue with package id

In chocolatey when it gets webstorm package:
IPackage availablePackage = packageManager.SourceRepository.FindPackage(packageName, version, config.Prerelease, allowUnlisted: false);
It returns package with id WebStorm. But when it gets local package:
IPackage installedPackage = packageManager.LocalRepository.FindPackage(packageName);
it returns id webstorm.
So there is probably a bug.
And in the fix chocolateyGUI takes packageName instead of id which seems to work.

@DamianMaslanka5 thank you for submitting this fix.  Could you please provide some steps to reproduce the problem that you are seeing?
I just want to make sure that we are talking about the same thing.  Thanks

@gep13
Just install an old version of webstorm, for example 2017.3.3 and check your installed packages. It doesn't show latest version and you can't update the package.

Relates to #506

@RichiCoder1 can I get you to review this one?  Thanks!

That is weird. Really weird. Id should be the same universally. Or I thought so. @ferventcoder ?

Id should not be case sensitive.

@ferventcoder So after getting available package it should replace its id with id.to_lower() ?

@DamianMaslanka5 yes, I think that makes sense.  I "think" what is going on here is that Chocolatey treats everything as lowercase, however, in this case, we are using the NuGetService to interrogate the packages and get the information about them.  As a result, we are seeing some package ids coming through without the lower case being applied.  Can you try with simply doing a switch to lower case and let me know if the problem is resolved?  If it is, if you can fix up this PR to reflect that, we can get it pulled in.

Following this up, case sensitivity shouldn't be an issue, but there is something somewhere that is likely applying it.

@gep13 Updated.

@DamianMaslanka5 thank you for taking the time to correct this issue.  I have merged this into the develop branch here: https://github.com/chocolatey/ChocolateyGUI/commit/bf0d7c129f7b661dd909bd08b52266a0f081449e


⚠ Samsung has degraded performance

In 7d35efa, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2516 ms

Resolved: Samsung performance has improved in 57737cd.


Changed return type of Guild.defaultChannel

The default channel for a Guild is always the first TextChannel in the Guild, it can't be a VoiceChannel.
I know this might come off as petty, and I'm sorry if it does, just came across this today and wanted to make it a bit more specific.

For clarification, it's not "the first TextChannel" in the sense of the topmost TextChannel in the list but the first TextChannel to exist within the guild upon creation. But you are correct, it cannot be a VoiceChannel. Good catch.


Link-Checker-API not working inside Collections Publisher

Inside collections-publisher:
govuk-docker startup e2e

Then navigate to http://collections-publisher.dev.gov.uk/step-by-step-pages and create or view a step by step and then click "Check for broken links"; we get a 503 error.
~This happens even if you run govuk-docker startup inside link-checker-api in another tab.~ As of https://github.com/alphagov/link-checker-api/pull/284, you can now have link-checker-api running in another tab and this is available to collections-publisher, fixing the issue. But we shouldn't have to do this - the e2e stack should account for the link-checker-api dependency and resolve it automatically.

@ChrisBAshton I've tried this myself and it seems to work OK now, without any extra steps. I did have to pull the latest versions of the two repos in question.

Thanks Ben. Yes I think this issue crops up now and again and required an image rebuild of the repos in question. Think I updated the govuk-docker readme to that effect, but forgot to close this issue. Closing now :)


[AIFlow]Refactor model version.

What is the purpose of the change
Change model version's version type from string to int.

Codecov Report

Merging #295 (f89f17c) into master (e932701) will increase coverage by 0.10%.
The diff coverage is 67.64%.

@@            Coverage Diff             @@
##           master     #295      +/-   ##
==========================================
+ Coverage   66.80%   66.90%   +0.10%
==========================================
  Files         206      207       +1
  Lines       15573    15606      +33
  Branches     2337     2342       +5
==========================================
+ Hits        10403    10441      +38
+ Misses       4602     4598       -4
+ Partials      568      567       -1

Impacted Files
Coverage Δ

ai_flow/endpoint/server/util.py
79.91% <ø> (ø)

ai_flow/metadata_store/utils/ProtoToMeta.py
85.43% <ø> (ø)

ai_flow/model_center/entity/model_version.py
100.00% <ø> (ø)

ai_flow/store/mongo_store.py
14.14% <0.00%> (+0.02%)
:arrow_up:

ai_flow/endpoint/client/metadata_client.py
94.36% <100.00%> (ø)

ai_flow/endpoint/client/metric_client.py
78.07% <100.00%> (ø)

ai_flow/endpoint/client/model_center_client.py
96.00% <100.00%> (ø)

ai_flow/endpoint/server/__init__.py
100.00% <100.00%> (+23.07%)
:arrow_up:

ai_flow/meta/model_meta.py
91.66% <100.00%> (+0.75%)
:arrow_up:

ai_flow/metadata_store/utils/MetaToProto.py
100.00% <100.00%> (ø)

... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e932701...f89f17c. Read the comment docs.


Tagbox widget displaying in Builder but not in Survey

Are you requesting a feature, reporting a bug or asking a question?
Reporting a bug
What is the current behavior?
The Tagbox widget displays in Builder but not in Survey mode. In Survey mode, I see: "There is no visible page or question in the survey." When viewing inside the builder, it displays correctly.
What is the expected behavior?
It should ideally display in Survey mode too.
How would you reproduce the current behaviour (if this is a bug)?
Copy and paste this Survey JSON and try to load as a survey:
{
  "pages": [
    {
     "name": "page1",
      "elements": [
        {
          "type": "dropdown",
          "renderAs": "select2",
          "choicesByUrl": {
            "url": "https://restcountries.eu/rest/v2/all"
          },
          "name": "countries",
          "title": "Please select the country you have arrived from:"
        }
      ]
    }
  ]
}

surveyjs platform Vue
surveyjs version: 1.0.59

It looks like all neseccary scripts were not loaded on the page.
Could you check this sample - https://surveyjs.io/Examples/Library/?id=custom-widget-select2-tagbox&platform=Vue&theme=default
or this repo - https://github.com/surveyjs/surveyjs_vue_quickstart
(https://github.com/surveyjs/surveyjs_vue_quickstart/blob/master/src/App.vue)

I'm testing now but it seems that even when importing all the necessary package it doesn't render - I'll keep you updated. Thanks!

@willgdjones you can also try this codesandbox example https://codesandbox.io/s/zrkjozrnn4
You can modify it to reproduce the issue and provide the link.

Hi @dmitrykurmanov - thank you for providing this example, it allowed me to very quickly see what the problem was.
I had included the following line of code:
widgets.select2tagbox(SurveyKo);

when it needed to be:
widgets.select2tagbox(SurveyVue);

Double checking the quickstart documentation, it seems that I did not read the example closely enough. Thanks for both of your quick help! @tsv2013 @dmitrykurmanov

@willgdjones  what do you think about codesandbox? Is it comfortable for you? We think about creating examples there.

It was very useful for me because it allowed me to compare my Vue setup, where I am not importing the js files from CDNs, with the setup that you provide. The Plunker examples import the JS files from a CDN, therefore I could confirm that something was wrong with my setup, I had to play around for a while to find out exactly what my specific problem was.


Release for Javascript 1.0

Are there plans to make a release with ScalaJS  version 1.0?

We have to change the build configuration for dependencies.
Feel free to create PRs. See https://github.com/Atry/Curried.scala/pull/35 and https://github.com/Atry/Curried.scala/pull/42 as an example.


Update renovatebot/github-action action to 34.12.1

This PR contains the following updates:

Package
Type
Update
Change

renovatebot/github-action
action
patch
v34.12.0 -> v34.12.1

Release Notes

renovatebot/github-action
v34.12.1
Compare Source
See the the changelog for changes in all releases.
34.12.1 (2022-11-02)
Bug Fixes

deps: update renovate/renovate docker tag to v34.12.1 (7ad0c6c)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

@balena-ci I self-certify!


paho-mqtt-c: 1.3.8-5 in 'noetic/distribution.yaml' [bloom]

Increasing version of package(s) in repository paho-mqtt-c to 1.3.8-5:

upstream repository: https://github.com/eclipse/paho.mqtt.c.git
release repository: https://github.com/nobleo/paho.mqtt.c-release.git
distro file: noetic/distribution.yaml
bloom version: 0.10.7
previous version for package: 1.3.8-4

Because of a typo in the rules.em file..


OpenSSL Engine Implementation for post-quantum key exchange algorithms

Good morning.
I am trying to provide OpenSSL post-quantum security.
We base post-quantum security on the exchange of keys following guidelines.
I was wondering if it is possible to, through an openSSL engine, implement the key exchange functions to make the key exchange over another interface that is not ECDH or ECDSA.
Anyway, is there any official documentation of the use and implementation of openSSL engines?
I have found an example but it is not clear to me at what level are implemented the openSSL engines, that is, what layer of the library are.
I think I am not explaining myself correctly, if you have any questions I am at your disposal.
Thanks and best regards,
Jaime.

For completely new key types that OpenSSL doesn't know, it's an impossible task.
For RSA, DH and DSA, i.e key types that OpenSSL knows about, it's like that example, but different names... more or less.
If you're developing for the future, though, I would recommend building a provider instead.
I can provide links for you to read about later.

Thank you.
The point is we need to encapsulate calls to an Quantum Key Distributed API that provides keys as a stream of bytes into openSSL, and I was wondering if it is possible make it through an openSSL engine.
Anyway, I would appreciate to have these links in case at some point we decide to implement the provider.

So Quantum Key is a separate type of key?  Not RSA, not DH, not DSA, not EC, not ECDH, not ECDSA?
Re links, there is the Strategic Architecture, which is our view of the long term future, and OpenSSL 3.0 Design, which is for the next upcoming version.    Some details from the design document aren't exactly the same any more, but on a broader level, those documents are accurate.
As for manuals, I hope that provider(7) and provider-base(7) provide enough of a starting point to at least tickle you.

I was wondering if it is possible to, through an openSSL engine, implement the key exchange functions to make the key exchange over another interface that is not ECDH or ECDSA.

I assume you mean ECDH or DH here, since ECDSA is no a key exchange algorithm!
It depends what you mean. OpenSSL 3.0 (i.e. the master branch) has implemented provider support for key exchange meaning that you could write a provider to implement any arbitrary key exchange algorithm and it should (in theory) just work. Application code can call the key exchange APIs, requesting your new algorithm to be used, and away you go.
However there are some important caveats on that:

Implementing a new key exchange algorithm at the libcrypto layer does not mean that it will be used by libssl - so if that is your goal then this is not possible. libssl has no support for pluggable key exchange.

Providers are "in development" at the moment. Most significant key management is a work in progress - meaning that you can't (yet) do key generation inside a provider. Without that piece of the puzzle having the key exchange algorithm implemented is of limited usefulness! Key generation will be implemented before too long (certainly before 3.0 is released), but you'll have to wait a bit for that.

As for manuals, I hope that provider(7) and provider-base(7) provide enough of a starting point to at least tickle you.

Also of specific relevant to key exchange is this man page:
provider-keyexch(7)

Most significantly key management is a work in progress - meaning that you can't (yet) do key generation inside a provider.

This is upcoming functionality, I currently have that in a branch, top front and center.

I was wondering if it is possible to, through an openSSL engine, implement the key exchange functions to make the key exchange over another interface that is not ECDH or ECDSA.

I assume you mean ECDH or DH here, since ECDSA is no a key exchange algorithm!
It depends what you mean. OpenSSL 3.0 (i.e. the master branch) has implemented provider support for key exchange meaning that you could write a provider to implement any arbitrary key exchange algorithm and it should (in theory) just work. Application code can call the key exchange APIs, requesting your new algorithm to be used, and away you go.
However there are some important caveats on that:

Implementing a new key exchange algorithm at the libcrypto layer does not mean that it will be used by libssl - so if that is your goal then this is not possible. libssl has no support for pluggable key exchange.
Providers are "in development" at the moment. Most significantly key management is a work in progress - meaning that you can't (yet) do key generation inside a provider. Without that piece of the puzzle having the key exchange algorithm implemented is of limited usefulness! Key generation will be implemented before too long (certainly before 3.0 is released), but you'll have to wait a bit for that.

So Quantum Key is a separate type of key? Not RSA, not DH, not DSA, not EC, not ECDH, not ECDSA?

The point is I just need to make some calls to an API that calls a QKD Key Server, which will provide the same key to the openSSL client and the openSSL server.
The openSSL server will be running in the same enviroment that the key server, so the sequence diagram is the next one:

openSSL is running in the same enviroment than keyServer.
openSSL client request a key to the keyServer.
the keyServer provides a key to both the client and the server.
the openSSL client performs the connection to the openSSL server.

I just need to pass some parameters with both calls to make sure the key server provides them the same key.
Do you think it is possible to implement this "exchange" thorugh an openSSL engine? Or through a provider?

What protocol are you intending to use for client <-> server communication? Something custom you are writing yourself? Or SSL/TLS?

What protocol are you intending to use for client <-> server communication? Something custom you are writing yourself? Or SSL/TLS?

SSL/TLS.
I am just wondering what approach is better to integrate the QKD-Key exchange into openSSL to use QKD Keys with SSL communication.

SSL/TLS.
I am just wondering what approach is better to integrate the QKD-Key exchange into openSSL to use QKD Keys with SSL communication.

This is not possible without code changes to OpenSSL itself. As I said above:

Implementing a new key exchange algorithm at the libcrypto layer does not mean that it will be used by libssl - so if that is your goal then this is not possible. libssl has no support for pluggable key exchange.

If you do decide to hack the OpenSSL source to integrate this then you'll also need to decide whether you plan to support TLSv1.2, TLSv1.3 or both. Key exchange between the two protocol versions works quite differently. If its just for a proof-of-concept then I imagine you would have an easier time with the TLSv1.3 route.

Do you think it is possible to implement this "exchange" thorugh an openSSL engine? Or through a provider?

It is possible to implement "standalone" (i.e. not integrated into SSL/TLS) key exchange through an engine. You would need to implement a custom EVP_PKEY_METHOD to implement the algorithm itself, (and possibly  a custom EVP_PKEY_ASN1_METHOD to implement the necessary functions for key serialisation/deserialisation).
The other route is to do this via a provider as per the documentation links @levitte already supplied.
Either way should work, although my guess is that the provider route is probably easier (but has the disadvantage that support for it is "cutting edge", not yet released in a stable branch and not all there yet).

SSL/TLS.
I am just wondering what approach is better to integrate the QKD-Key exchange into openSSL to use QKD Keys with SSL communication.

This is not possible without code changes to OpenSSL itself. As I said above:

Implementing a new key exchange algorithm at the libcrypto layer does not mean that it will be used by libssl - so if that is your goal then this is not possible. libssl has no support for pluggable key exchange.

If you do decide to hack the OpenSSL source to integrate this then you'll also need to decide whether you plan to support TLSv1.2, TLSv1.3 or both. Key exchange between the two protocol versions works quite differently. If its just for a proof-of-concept then I imagine you would have an easier time with the TLSv1.3 route.

Do you think it is possible to implement this "exchange" thorugh an openSSL engine? Or through a provider?

It is possible to implement "standalone" (i.e. not integrated into SSL/TLS) key exchange through an engine. You would need to implement a custom EVP_PKEY_METHOD to implement the algorithm itself, (and possibly a custom EVP_PKEY_ASN1_METHOD to implement the necessary functions for key serialisation/deserialisation).
The other route is to do this via a provider as per the documentation links @levitte already supplied.
Either way should work, although my guess is that the provider route is probably easier (but has the disadvantage that support for it is "cutting edge", not yet released in a stable branch and not all there yet).

Ok, thank you very much for the info.
I will reconsiderate the implementation.
Thanks!

Good afternoon.
After reading some documentation about providers, I have some questions.
I am going to explain my design for your understanding:
There exists a Key Server next to each connection end (both client and server), which performs communication between both to transport the same key both the customer and the server.
I just need to indicate that this will be the key used to perform all the operations.
The key distribution can happen at any time and layer (SSL / CRYPTO / PROVIDER / ENGINE), it is just needed to perform a call to an API from both sides at the same layer.
I have considered several possibilities and I want to know if it is possible to perform the QKD-API function call (returns a string byte with the key) in the Provider initialization (or the Protocol Initialization, I mean a high level layer), so that way the key is passed by parameter in the digests, cipher, MAC... functions and both sides have the same key.
Is there any function in the provider initialization where the key is set? Or is it made in a lower level?
I also were wondering if you have some documentation about the flow through openSSL (openSSL application call, TLS/SSL, Core, Providers, Engines) to know where will be better to perform the API calls to set the key to use.
I do not know if I am being clear enough, for any doubt please do not hesitate to ask me.
Thank you,
Jaime.

I have considered several possibilities and I want to know if it is possible to perform the QKD-API function call (returns a string byte with the key) in the Provider initialization (or the Protocol Initialization, I mean a high level layer), so that way the key is passed by parameter in the digests, cipher, MAC... functions and both sides have the same key.
Is there any function in the provider initialization where the key is set? Or is it made in a lower level?

Keys are managed on the provider side via the keymgmt API. See:
https://www.openssl.org/docs/manmaster/man7/provider-keymgmt.html
Through this API keys can either be generated directly in the provider, imported from libcrypto (if it is a type of key that libcrypto understands such as RSA, DSA, ECDSA etc), imported from the application via libcrypto using "parameters", or loaded directly by the provider using some opaque "key id" passed from the application via libcrypto.
At least that is the theory. In practice though all that is currently present in master is the importing of well known types (and only for those algorithms that have been migrated to providers so far). This is an area of very active development. @levitte is currently working on this and might be able to give you a better insight as to when this will become available.

At least that is the theory. In practice though all that is currently present in master is the importing of well known types (and only for those algorithms that have been migrated to providers so far). This is an area of very active development. @levitte is currently working on this and might be able to give you a better insight as to when this will become available.

Ok, thank you very much.
I am considerating the Provider approach.
Would you know any site where I can find documentation about interaction between TLS Protocol, Core and 3rd party providers?
I would appreciate to have a detailed picture of the SSL flow between the TLS Protocol and the openSSL Core.

For the benefit of anyone else stumbling across this issue, here's a pointer to a provider integration of PQC.


Upgrade Handlebars to v4.0.x

Closes issue #110

Tests are failing; they must pass before merge. Also please follow existing convention for dependencies.
What new features does handlebars 4 bring or breaking changes? Since this library is closely coupled to handlebars syntax for the templates we need to potentially document the breaking changes or an upgrade path for users.

Tests are failing for Node v0.8. I think we should stop supporting it going forward. As for breaking changes, I've noticed nothing so far.

@riyadhalnur so what new features were added then to warrant an upgrade?

a lot has changed as outlined in their release notes of which I personally love the decorators and inline partials feature. the upgrade from 3 is actually non trivial -

Depthed paths are now conditionally pushed on to the stack. If the helper uses the same context, then a new stack is not created. This leads to behavior that better matches expectations for helpers like if that do not seem to alter the context. Any instances of ../ in templates will need to be checked for the correct behavior under 4.0.0. In general templates will either reduce the number of ../ instances or leave them as is. See #1028.
The = character is now HTML escaped. This closes a potential exploit case when using unquoted attributes, i.e. . In general it's recommended that attributes always be quoted when their values are generated from a mustache to avoid any potential exploit surfaces.
AST constructors have been dropped in favor of plain old javascript objects
The runtime version has been increased. Precompiled templates will need to use runtime of at least 4.0.0.

Sounds good. I went ahead and bumped the handlebars version in master. We will need to bump the major version on hbs given the changes.

closing this then since you already upgraded the version


ElasticSearch: Add builder method to create an index

Hi,
I am currently looking into using the ElasticSearch Testcontainer. I can see you did have methods like username and password.
Another really handy method would be to be able to add an index while creating the docker image as well. Not sure how/if that is possible though

Hi @huehnerlady , Did you get the Above query. how to create Index and push some data in to elasticsearch Test Containers ?
If yes, please paste the respective code here which will be helpfull

Hi, this is not Testcontainers responsibility. Testcontainers provides the pre-configure container in a state that can be used. Username and password are set via env vars.


Fix bug

Solves #89
The bug happens when add a new person with visit field not empty then try to mark the person's visit as done.
It is because a visit added upon adding the person does not have frequency or occurrence so they will throw exceptions when trying to update in done.
A quick is to check their values and treat the visit as an ad-hoc one if frequency or occurrence are not present / set.

Codecov Report

Merging #90 (f1d9219) into master (1ca37e6) will increase coverage by 0.01%.
The diff coverage is 75.00%.

@@             Coverage Diff              @@
##             master      #90      +/-   ##
============================================
+ Coverage     75.31%   75.33%   +0.01%
- Complexity      525      526       +1
============================================
  Files            80       80
  Lines          1584     1585       +1
  Branches        173      173
============================================
+ Hits           1193     1194       +1
  Misses          338      338
  Partials         53       53

Impacted Files
Coverage Δ

...java/seedu/address/logic/commands/DoneCommand.java
89.28% <66.66%> (ø)

...in/java/seedu/address/model/person/Occurrence.java
93.75% <100.00%> (+0.41%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ca37e6...f1d9219. Read the comment docs.


Put legacy note in small print

I'd leave it as is - the note was moved below as it applies to both options (standalone / OSGeo4W installer) - it's small it might be overlooked esp. for the more prominent placed standalones at the top.

Support for Windows 7 and Windows 8 has reached EOL, does this apply to a platform that is still actively supported?

Support for Windows 7 and Windows 8 has reached EOL, does this apply to a platform that is still actively supported?

Windows 7 is apparently still used. It didn't take long from taking down the earlier note to the first question why it didn't work on Windows 7 or whether it's 64 or 32bit.

I'd still assume that almost everyone visiting this page is not interested in Windows 7 and that this note should be big enough to be readable by the few people looking for it but small enough not to be auto-discovered by the majority.


some test scripts for Automated Soft Credits

Critical Changes
Changes
Issues Closed
New Metadata
Deleted Metadata

@davisagli  How do you checkout this branch locally to test?

@force2b the name of this branch is weird because usually in git origin/* is reserved for git's local copies of the remote called origin -- but Sravani somehow managed to create a branch with that name on github. I've been able to check it out locally using origin/origin/feature/robot
I'm hoping that at some point in the near future @skristem can switch to working on branches with names starting with feature/robot/ that we can merge directly to master instead of to feature/robot first (the unstable robot tests run on MetaCI for all branches starting like that). But first we'll need to get the current changes in feature/robot merged to master...

@force2b and @davisagli can I merge this PR

I'm going to go ahead and merge this into feature/robot because then we can see the status of the tests on MetaCI, which will help with any further fixes/refinements.

@davisagli thank u


URL Rewriting.

Would grab a video from a directory and writes the video name to url (domain.com/videoname) and removes the file extension too.

It's very easy to do, depends on if you've written the grabbing part already and mainly want to remove the file extension.
The Video's name should be picked up automatically, so you should only work on the file extension part.
Best Regards,
Rith

Removing the file extension isn't hard but i will work on the rewriting the URL Part.


"destroy" event not triggered if dialog.destroy()

My implementation sets up a call using b2bua and has "destroy" event handling per the documentation that receives the "destroy" event when one side terminates the call and then handles end of call logic and calls dialog.destroy() on the other side of the call.
This works and has been working as expected for some time. A new feature requires retrieving the dialog via srf.findDialogById(stackDialogId), then calling dialog.destroy() on that dialog.
When I do that the dialog does get destroyed, but I don't see the "destroy" event being invoked and the rest of the call handling that usually is then kicked off in response to one dialog being destroyed doesn't occur.
Is this expected behavior, and is there a way to get the event to trigger?

Was able to resolve my issue by manually emitting the destroy event, like so:
dialog.emit('destroy', stateObj);

Closing the issue

Was able to resolve my issue by manually emitting the destroy event, like so:
dialog.emit('destroy', stateObj);

Closing the issue


Removing googleapis-common-protos from deps in non-core packages.

Also

removing grpcio from non-core packages.
manually specifying the grpcio dep in core (rather than getting it from googleapis-common-protos[grpc])

(NOTE: This is a followup to #4096.)

@tseaver It already was, this is the line being removed from core/setup.py:
'googleapis-common-protos[grpc] >= 1.5.3, < 2.0dev',

@dhermes Yeah, but literally just now in #4096, right?

Ahh good call. That was some extra fail copy-pasta by me in #4096.

ISTM that adding unconditional dependencies to an already-GA API should be worth some debate.

@tseaver Yes I did it by mistake in #4096, will undo here.

@tseaver PTAL, I made grpc an extra in core.
@jonparrott Does this look like the right approach for packaging?

@lukesneeringer Can you follow-up on my question (https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4098#discussion_r142552072) when you get a chance?


Minify Razor @helper functions by implementing a build provider, resolves #5

See #5.
Because ASP.NET will not use the configured Razor host factory for App_Code files, a minifying build provider has been implemented. For App_Code files, ASP.NET uses WebCodeRazorHost instead of MvcWebPageRazorHost, so I have also implemented a MinifyingWebCodeRazorHost similar to the existing MinifyingMvcWebPageRazorHost.
I updated the sample and documentation. Additional configuration is required to make minification of Razor @helper functions work:
<compilation debug="true" targetFramework="4.7.2">
    <buildProviders>
        <add extension=".cshtml" type="RazorHtmlMinifier.Mvc5.MinifyingRazorBuildProvider, RazorHtmlMinifier.Mvc5" />
    </buildProviders>
</compilation>

I tested a new build of RazorHtmlMinifier.Mvc5 in my own codebase to verify all minification works as expected.

Thanks for contributing, I had a brief look and it looks great!
I'll look at this closer soon and try to merge it and release it.


Question: Is there a way to achieve pythonw behavior?

There is no way for making a Nuitka made executable run without a console pop-up, like when a python file is saved with ".pyw" behavior.
I scanned all switches once and searched all over google
I don't flag it rn, cause it may be my mistake, as I'm newbie in this project, if this feature is not out there, this shall be flagged as "enhancement"

Yes, there is an option, checkout  --windows-disable-console


Concretize Initialized Vars

I only changed statemate and nsichneu, but all previous commits are added to. I think the branch wasn't up to date. Let me close this PR and add a new PR.


Fix Ruby 2.6 warning

Quick fix to address a syntax ambiguity for splatted hashes that now issues a warning on Ruby 2.6. The warning was:
dry-transaction-0.13.0/lib/dry/transaction/step_adapter.rb:7: warning: in `[]': the last argument was passed as a single Hash

Thanks :)


hello，What causes redis to have too many idle connections

I used celery to processing tasks, the task of redis connection, when running after a period of time, will quote us was forced to shut down a remote connection error, I view the Windows port usage, found that there are a lot of TCP connections, I doubt is free connection too much.What do you think

Celery uses multiple connections to Redis so that tasks can be consumed in parallel. If you have more questions about how Celery uses redis-py, I would suggest posting on the Celery or Kombu issues list.


cls: scan rados objects having parquet data

Currently, we store read parquet data from objects. For query performance optimization, we want to read parquet tables from objects using the parquet fragment API .

closed by #61

closed by #61


Sparse Graph submodule

This is an initial pull request aimed at adding some graph routines based on the scipy sparse matrices (see mailing list discussion here: http://mail.scipy.org/pipermail/scipy-dev/2011-December/016773.html )
The initial commit includes some routines modified from the scikit-learn utility functions:

Dijkstra's algorithm w/ Fibonacci heaps
used to compute shortest path, implemented in cython

Floyd-Warshall algorithm
also used for shortest path, implemented in cython

graph Laplacian
implemented in pure python, for both sparse and dense inputs

This is very much a work in progress: I'd love to have a discussion about what routines should be included here.
I'd like this package to be a compendium of very fast, very general graph algorithms based on the scipy.sparse data model (i.e., no custom graph/node classes).  Though this won't allow for every graph algorithm to be included, I think it is the best fit to the scipy philosophy.  I think any routines included here should be very general, and address use cases in derived libraries, e.g. scikit-learn, scikit-image, networkx, etc.  I'd appreciate input from the developers of those libraries.
I have not benchmarked the below algorithms against those in NetworkX, but I think they will compare favorably.

Looks good at first glance. Just commenting to CC myself.

I have not reviewed in detail, but this looks like a very nice addition.   Thank you!  Register my +1

I'd be interested in comments from some NetworkX developers regarding other functionality that would be useful to include.

Just a quick benchmark against NetworkX: I think this is a fair comparison, but someone with more familiarity with networkX should double-check this:
In [1]: import networkx as nx

In [2]: import numpy as np

In [3]: from scipy.sparse.graph import graph_shortest_path

In [4]: G = nx.gnm_random_graph(100, 200)

In [5]: M = nx.to_numpy_matrix(G)

In [6]: timeit nx.shortest_path(G)
10 loops, best of 3: 23.1 ms per loop

In [7]: timeit graph_shortest_path(M)
100 loops, best of 3: 4.89 ms per loop

I moved the algorithms from the graph subdirectory to the csgraph subdirectory, and renamed them to unify with the convention set by cs_graph_components.
I haven't gotten any feedback on this from networkx developers.  I'm curious if people have ideas for other algorithms that could fit in the compressed sparse graph framework.
What else needs to be done before merging?

I took a look through the code and it seems like a good module.
I have two suggestions/questions based on a first read-thru.
They are both about graph_shortest_path which is what I spent
time looking at.

shouldn't the names be switched for dist_matrix and graph?
The input matrix describes the graph and the output describes the
shortest path length (distance) between pairs.  So it seems like
the input should be named graph and the output named dist_matrix.
Minor I know--but maybe easier to read?

Some users will want the path length and others want the path itself.
We found in NetworkX that it worked well to have the same routine optionally
return both a distance_matrix and a predecessor_matrix.  Row i of the
predecessor_matrix gives the predecessor of each node j in a shortest
path from node i to node j.  You can build paths quickly from the
predecessor_matrix.   Such a routine could be included in this module for speed,
or left up to the user to post-process.

As for other algorithms, how about unweighted path lengths, depth first order
and breadth first order?  How much do you want to put here vs in a specialty
package like NetworkX?

Thanks for the comments.
Good point on the naming.  I'll switch that to be more clear
I'll think about how to modify the code to return the predecessor matrix as well if the user passes a return_paths flag.
Unweighted path lengths is a good idea, and should be easy to implement.

How much functionality to include here is an interesting question.  I started with these particular routines because they're used in scikit-learn, and folks on the scipy-dev mailing list thought that these algorithms were fundamental enough to warrant inclusion in scipy.  I have to admit I have no idea how to draw a good line between what belongs here and what doesn't.

It's probably not possible to draw a clear line. As rules of thumb perhaps:

should have potential uses on more than one project that depends on scipy
should appear in an introductory algorithm book such as "Introduction to Algorithms", Cormen et al.

Thanks for the input, Ralf.  In light of that, I think I'll work to add the functionality suggested above by @dschult and call it good for now.  Those algorithms are fairly basic and broadly applicable, and most are used downstream in scikit-learn.  If more use-cases come up in the future, I'd be happy to work on adding them in.

Sounds good.

@dschult - can you take a look at the depth-first and breadth-first search functions I added?  Is this what you had in mind?
I still want to create a utility routine which will take the output of these and construct a sparse representation of the graph they represent.

Indeed.... those functions are what I had in mind.  The expansion of the original routines also look great.
The next thing I wonder is perhaps not in the spirit of the original intent, but I'm curious.
How dangerous would it be to use these routines on objects that aren't csr sparse matrices
but look similar--have .shape, .indptr, .indexes, .data?
I'm thinking of a graph data structure with csr storage that replaces the slicing features of spmatrix with more graph-like operations.  Right now the cdef code would work on these objects so long as those 4 attributes are defined.  Is it unlikely that the code would be rewritten so that it requires other csr_matrix features?  If so, it might be nice to enable access to the cdef functions directly.
I'm not sure this is possible, or even wise, and I'm sure it is hackish. :)  But my question is really about whether these routines are abstractable to csr representations that are not quite spmatrix.
Thanks---this is great stuff!
Dan

Dan,
I think you're right that the only attributes of the csr matrix that will be used by the routines are data, indices, indptr, and shape.  We could certainly allow arbitrary objects to be passed which have these attributes, but we'd have to be careful.  For example, a csc_matrix has all these attributes, but the code would have to decide whether the user means the csc matrix to represent an abstract graph object, or represent a regular matrix which should be converted to csr.  It sounds kind of messy.
As far as providing user access to the cdef routines, I'd be hesitant at this point.  Currently these routines don't perform any validation, and many have index-checking turned off for speed.  This is fine because the input is pre-validated by the calling function, but allowing user-access to these could easily lead to segfaults, unless we turn on index checking, which will lead to a reduction in speed.
Interesting ideas, though!

That all makes good sense!
Thanks again for putting this together.
Dan
On Feb 9, 2012, at 9:28 AM, Jacob Vanderplas wrote:

Dan,
I think you're right that the only attributes of the csr matrix that will be used by the routines are data, indices, indptr, and shape.  We could certainly allow arbitrary objects to be passed which have these attributes, but we'd have to be careful.  For example, a csc_matrix has all these attributes, but the code would have to decide whether the user means the csc matrix to represent an abstract graph object, or represent a regular matrix which should be converted to csr.  It sounds kind of messy.
As far as providing user access to the cdef routines, I'd be hesitant at this point.  Currently these routines don't perform any validation, and many have index-checking turned off for speed.  This is fine because the input is pre-validated by the calling function, but allowing user-access to these could easily lead to segfaults, unless we turn on index checking, which will lead to a reduction in speed.
Interesting ideas, though!

Reply to this email directly or view it on GitHub:
https://github.com/scipy/scipy/pull/119#issuecomment-3888033

The tests should use full import paths, either from scipy.sparse import cs_graph or from scipy.sparse.cs_graph import func_a, func_b. The package (csgraph/__init__.py) can also use from graph_components import func_a, as you did. This is actually more common than the full path. Don't use relative imports with dots.
The way you structured it is very good I think. Just a few minor things to add to the docs:

You didn't add an import of csgraph in sparse/__init__.py, which I think is fine. But that means csgraph is a public module, you can document it as such in doc/API.rst.txt.
As you said, the package needs a slightly longer description in the docstring in __init__.py and a listing of functions.
The package should be added to the reference guide by adding a sparse.csgraph.rst file with an automodule statement in doc/source/, and an entry in index.rst.
A few docstrings can be completed, for example cs_graph_laplacian needs params/returns, and some examples would be nice. The breath/depth first tests are easy to understand by looking at input/output arrays, they can be directly used as examples I think.

Good one Pauli, forgot about the underscores. +2 for those.
scipy.sparse.csgraph looks good to me.

Yep, dropping the prefixes and leaving the alias sounds OK. The alias could also be deprecated in due course.

I did a large number of updates, including the renaming discussed above, additional tests, examples in doc strings, documentation, and adding information to the release notes.
Let me know if you see anything I've missed!

Can you add: https://github.com/rgommers/scipy/commit/8228f76da6774541d198e1d67d77a757380f3cb1
It has fixes for numscons and bento builds. The numscons one I tested, I'm not 100% sure about the bento build. I couldn't find an example of usage of a .pxi file, perhaps that should be included in a builder (in which case a bscript file is needed).

Why not also underscore validation.py and tools.pyx?

@rgommers - regarding the pxi file: I think that because I've cythonized the scripts and committed the resulting c code, users won't have to worry about the details of the cython implementation.

In [6]: csgraph.<TAB>
csgraph.breadth_first_order    csgraph.depth_first_tree       csgraph.reconstruct_path
csgraph.breadth_first_tree     csgraph.dijkstra               csgraph.shortest_path
csgraph.connected_components   csgraph.floyd_warshall         csgraph.tools
csgraph.construct_dist_matrix  csgraph.laplacian              csgraph.validation
csgraph.cs_graph_components    csgraph.minimum_spanning_tree
csgraph.depth_first_order      csgraph.numpy

Shouldn't contain numpy, tools and validation. construct_dist_matrix and reconstruct_path are also not in __all__, should they be? If not, why are they imported in __init__.py?

I think construct_dist_matrix and reconstruct_path are tools which could be useful to users.  Perhaps I should add some examples to show how they're used.

Some nitpicks on the docs (which look excellent overall):

All sentences should start with a capital letter and end with a dot.
array-like should be array_like.
Parameters/Returns type specifiers:

keywords get , optional appended
default value and shape are noted in the description, not the specifier.
integer should be int, boolean should be bool (only in the type
specifier, not in a sentence).
if >1 type is possible, use "or". For example
node_array: np.ndarray, int, shape=(N_nodes,) should be
node_array: ndarray or int, optional.

"1d" should be "1-D"
the list in shortest_path (method parameter) won't render correctly I
think.  Probably best to indent it and add blank lines between the
auto/FW/D items.
the examples in depth/breadth_first_tree look incorrect in a terminal
(see below).  In minimum_spanning_tree it looks fine.

::
     (0)                         (0)
    /   \                       /              3     8                     3     8
  /       \                   /               (3)---5---(1)               (3)       (1)
  \       /                           /
   6     2                           2
    \   /                           /
     (2)                         (2)

You're right about the .pxi file, so the Bento fix should also be correct.

I think I've addressed all your concerns listed above.  Let me know if you see any other issues.

The info.py files have all been removed a while ago. Can you undo commit 3a5b909?
Is it still documented somewhere that info.py files should be added?

One last algorithm which would be nice to add would be the Bellman-Ford algorithm (and perhaps Johnson's algorithm) to perform shortest path searches on graphs with negative edges.  But that brings up the slightly awkward question of how to deal with edges of zero weight.  I think that zero-weight edges are probably a fundamental limitation of compressed sparse graph representations.

I've got three more commit at https://github.com/rgommers/scipy/tree/sparse-graph to fix a compile error, add C files to .gitattributes and add a test() command for the package. Could you pull those?

I moved things to info.py because that's how it's done in scipy.sparse and scipy.sparse.linalg.  I didn't know that we had changed that convention.  I'll undo that commit.

There's no info.py in scipy.sparse, but I see I forgot to remove the one in sparse.linalg (and its subfolders). I'll go fix that now.

Zero-weight edges will be difficult, but in principle there's nothing to stop you from explicitly storing 0's in csr format, right? Just getting them in there from a dense matrix doesn't work, so you'd have to create a matrix with csr_matrix((data, indices, indptr), [shape=(M, N)]). I don't know how much of a pain that would be.
Bellman-Ford and Johnson's algorithm meet the rules of thumb I suggested above, so I certainly won't stop you from adding them.

Good point - I think I will work on Bellman-Ford and Johnson's algorithm.  They can be included as alternate methods in csgraph.shortest_path.
One concern I have: some of these methods will give inconsistent results if passed a csr matrix in non-canonical form.  e.g. if there are repeated indices in a row, then dijkstra's algorithm will choose the smallest value among them (it essentially treats them as multiple edges between the nodes) while the floyd-warshall algorithm would sum them in the conversion to dense format.  A similar problem would occur with zero-weight edges: dijkstra's algorithm treats them as true zero-weight edges, while the floyd-warshall algorithm treats them as unconnected nodes.
Any thoughts on what to do with this?

My impression is that it would be best to treat these issues in the docstring with warnings.
Something like:
It is not well defined how to handle graphs
with zero weight edges.  We have chosen one
way to treat them but you are warned that
this may not be what you expect.  Be careful!!
Then explain how you treat them in that particular routine.
Negative weights could/should be discouraged for djikstra, but I don't think a check of the input for them is necessary if the docstring is clear.  It sounds like a warning about non-canonical csr matrices might be good too.  Not sure if its best to have the warnings in each function's docstring or if the module docstring is sufficient.
Dan
On Feb 13, 2012, at 4:26 PM, Jacob Vanderplas wrote:

Good point - I think I will work on Bellman-Ford and Johnson's algorithm.  They can be included as alternate methods in csgraph.shortest_path.
One concern I have: some of these methods will give inconsistent results if passed a csr matrix in non-canonical form.  e.g. if there are repeated indices in a row, then dijkstra's algorithm will choose the smallest value among them (it essentially treats them as multiple edges between the nodes) while the floyd-warshall algorithm would sum them in the conversion to dense format.  A similar problem would occur with zero-weight edges: dijkstra's algorithm treats them as true zero-weight edges, while the floyd-warshall algorithm treats them as unconnected nodes.
Any thoughts on what to do with this?

Reply to this email directly or view it on GitHub:
https://github.com/scipy/scipy/pull/119#issuecomment-3949727

For people following this - I haven't had time to do much work in the last week or so, but I hope to get to it in the near future.
These are my thoughts - CSC and CSR matrices can be set up to allow for zero-edges, but these get lost on conversion to dense arrays with the toarray() function.  My most recent commit includes a couple routines for sparse-dense conversion which are designed for compressed sparse graphs, and are more flexible in allowing the presence of meaningful zero-entries.
I need to update the documentation, examples, and some of the code to reflect these .  With these tools in place, I should be able to implement Johnson's algorithm and the Bellman-Ford algorithm, link them into the shortest_paths wrapper function, add some test cases, and then I think the submodule will be ready for merge.

The idea of the csgraph_from_dense and csgraph_to_dense should work well, but I'd choose the keywords such that they match and that a matrix roundtrips correctly. Something like
def csgraph_from_dense(graph, null_value=np.inf)
    ....

def csgraph_to_dense(csgraph, null_value=np.inf)
    ....

I added the beginnings of the Bellman-Ford algorithm.  The basic implementation works, but I need to add more to make it full-featured.  My list of remaining tasks is under TODO at the top of _shortest_path.pyx.
@rgommers - the reason I did slightly different call signatures for csgraph_from_dense and csgraph_to_dense is that there may be situations where one would want multiple values (say 0 and +/-infinity) to be non-edges when converting dense to sparse.  If we follow your suggestion, the code would look cleaner, but that flexibility would be lost.  Perhaps a compromise is to allow only one value to be passed in an argument null_value, but keep the infinity_null flag in csgraph_from_dense.

That's a good reason - it would be annoying to have to zero infty's by hand if they are somehow present. I just liked being able to roundtrip from an aesthetic/principled POV, but if there's a good reason not to do that it's not a big deal.

I just saw you added some more commits (Github doesn't send notifications for those). Looks good, and your TODO list is getting shorter.

Thanks!  Once I'm finished with pydata and pycon, I hope to finish this off.

Pydata sprint tonight!  I finished implementing Johnson's algorithm, and made all shortest path methods deal with negative cycles appropriately.
I'd still like to add some tests for negative cycles, and double-check that all the specialized conversion functions are consistent.  Getting very close - if anyone feels like doing a detailed review of the submodule, now would be a great time to start!

A few notes on this commit: For consistency across the submodule, I decided to define a canonical dense graph representation, where null-edges are represented by infinite weights.  This alleviates some problems I was having for graphs with zero-edges.  There are several utility routines to efficiently convert graphs to this canonical form from other representations (e.g. where null edges are denoted by 0).
I also cleaned up the conversion functions, so they're closer to the form that @rgommers suggested.
I need to take a look at the laplacian and connected_components functions, and make sure they behave with this new canonical form.  I also want to add a piece to the narrative documentation with some examples.  The to-do list is getting much shorter!

Another thing - there are several locations where nearly pure python functions are in .pyx files.  It may be nicer for people using the code (I'm thinking of source inspection with ipython, for example) if I work to factor pure python code to .py files.  Thoughts on that?

Moving pure-Python code out of Cython files has also the advantage that it decreases the size of the generated code.

I've been looking closer at the function scipy.sparse.csgraph.cs_graph_components, which I've moved to scipy.sparse.csgraph.connected_components in this PR.  I think it will need a complete rewrite, as there are several issues:
- if a directed graph is passed, it does not return the correct component label on nodes with no children.
- the doc string says that the function only uses the top triangle of the matrix; this is not true
- the source is a c++ module which is not well documented, and difficult to debug

For these reasons, I'd like to re-implement this function in cython, and fix these bugs along the way.  Thoughts?


[firebase_messaging] Use UNUserNotificationCenter for ios 10+

Description
Switch to use UNUserNotificationCenter once https://github.com/flutter/engine/pull/9864 lands in flutter master.
Follows instructions from: https://firebase.google.com/docs/cloud-messaging/ios/client#register_for_remote_notifications
Related Issues
Should fix https://github.com/flutter/flutter/issues/22099
Possibly Related Issues
https://github.com/flutter/flutter/issues/35375
https://github.com/flutter/flutter/issues/36201
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] If the pull request affects only one plugin, the PR title starts with the name of the plugin in brackets (e.g. [cloud_firestore])
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[ ] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[ ] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
[x] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
[x] I updated CHANGELOG.md to add a description of the change.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[x] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[ ] No, this is not a breaking change.

The messaging v1 API might be a good option for integration testing: https://firebase.google.com/docs/reference/fcm/rest/v1/projects.messages

I think the README should also contain code snippet for those who use Swift when it comes to delegating UNUserNotificationCenter.
Therefore, Swift users must add following line:
if #available(iOS 10.0, *) {
    UNUserNotificationCenter.current().delegate = self
}

It looks like https://github.com/flutter/engine/pull/9864 landed a while back - is this PR ready to resume?

@tvolkert I'm still working on adding tests for this.

any update ? :D

Also needs to provide a swift alternative for the AppDelegate code.

As mentioned before, also needs support for swift. I believe it's something like this:
     if #available(iOS 10.0, *) {         UNUserNotificationCenter.current().delegate = self as? UNUserNotificationCenterDelegate     }
but not 100% on that

How long would it take till it’s merged to master?

@kroikie @bparrishMines There are many same or similar issues after updating to the latest. Please take a look and fix it.

I am still experiencing issues where my onResume and onMessage callbacks are not firing properly. I made no code changes that I can think to be relevant.

@rscsam The latest library doesn't work on iOS. This isn't about wrong configuration. The library itself just doesn't work at the moment. I guess this repo isn't maintained by any Google developers because I have never had this kind of issue with Google products before.  #1644 This is the one I opened up 3 weeks ago and it hasn't been fixed.

"Projects that are not officially staffed by Googlers but may be of use to Firebase developers. Volunteers are welcome to contribute and maintain." I guess not but this is a dependency of their official firebase plugins so..

@njovy This isn't a "solution" per se, but removing
if #available(iOS 10.0, *) {   UNUserNotificationCenter.current().delegate = self as? UNUserNotificationCenterDelegate }
from my AppDelegate.swift fixed this problem for me.

@rscsam Getting rid of that line is a solution!!

@njovy This isn't a "solution" per se, but removing
if #available(iOS 10.0, *) { UNUserNotificationCenter.current().delegate = self as? UNUserNotificationCenterDelegate }
from my AppDelegate.swift fixed this problem for me.

work fine for me.
Flutter (Channel stable, v1.12.13+hotfix.5, on Mac OS X 10.15.1 19B88


Packaging for Inclusion in Fedora Linux

Hi!
This package is a candidate for inclusion as an official package in Fedora Linux! Per Fedora packaging guidelines, there are some issues that should be addressed before the package can be officially reviewed, built, and made available in the Fedora RPM repositories.
Since this package is part of a group of packages, I have been prototyping the packages in a Fedora Copr repository [1]. The preliminary sources for this package can be accessed in a Git repository [2].
Here are the initial issues which should be addressed:

[ ] Upstream package(s)
Since this package needs libpotassco, we should prioritize addressing issues with that package first [3].
[ ] .SO Name Versioning
Fedora guidelines strongly suggest that the version be included in any shared object libraries distributed in the package [4]. Some packages in this collection appear to be setting the SONAME value, though the value doesn't match the version of the package.
[ ] Installation Directory Override
Any files installed by the packages must conform to Fedora's filesystem hierarchy. In order for this package to install the files as such, it must not override CMake's default install locations without any mechanism to change them. I have sketched out a preliminary patch to address this [5]. Please feel free to use those changes as you see fit.
[ ] Header Installation Bug
It looks like there is a bug the header installation procedure which causes the directory structure to be lost. This will block downstream packages from consuming the headers properly. I've made a preliminary patch which addresses this [6].
[ ] Implicit Dependencies
In order to consume this package through CMake, the implicit dependencies of this package should be resolved. This package always depends on libpotassco, and if built with threads, also depends on Threads::Threads. It would be much easier for consuming packages to build against this package if they didn't need to explicitly resolve this package's dependencies on their own. I've made a preliminary patch which addresses this [7].

Once we have at least acknowledged each of the above issues, I'll initiate the review. If there is a valid reason that any of the issues can't be addressed, please let me know and we'll talk about an FPC exemption. If you have any further questions, or if there is anything I can do to help, please reach out.
Thanks very much!
--scott
[1] https://copr.fedorainfracloud.org/coprs/cottsay/potassco
[2] http://copr-dist-git.fedorainfracloud.org/cgit/cottsay/potassco/clasp.git/tree/
[3] https://github.com/potassco/libpotassco/issues/1
[4] https://fedoraproject.org/wiki/Packaging:Guidelines#Downstream_.so_name_versioning
[5] http://copr-dist-git.fedorainfracloud.org/cgit/cottsay/potassco/clasp.git/tree/clasp-3.3.2-install_dir.patch
[6] http://copr-dist-git.fedorainfracloud.org/cgit/cottsay/potassco/clasp.git/tree/clasp-3.3.2-header_install.patch
[7] http://copr-dist-git.fedorainfracloud.org/cgit/cottsay/potassco/clasp.git/tree/clasp-3.3.2-cmake-deps.patch

Hi Scott.
Thank you for your report. I tried to address your points in commit f139715 to dev.
However, at the moment I think the easiest and best thing to do would be to focus on clingo and to
forgo separate packages for libpotassco/clasp.
The clingo source releases (e.g. https://github.com/potassco/clingo/archive/v5.2.1.zip) already contain (and build) clasp and lpconvert. They also (by default) limit library installation to libclingo, which currently is the only library that is intended to be build and used as a shared library.

Awesome, thanks @BenKaufmann. I'll take a look as soon as I get back from vacation next week.


Multiple commands produce GoogleService-Info.plist

Hi,
I have 2 different flavors with different firebase configs. It works flawlessly in Android.
But I can't able to build the app in Xcode as it throws Multiple commands produce GoogleService-Info.plist error. Heres the error
Multiple commands produce '/Users/rameshvel/Library/Developer/Xcode/DerivedData/Runner-gqmldwjuahwuwkgijmfokisjscam/Build/Products/Debug-AppleApp-iphonesimulator/Runner.app/GoogleService-Info.plist':
1) Target 'Runner' (project 'Runner') has copy command from '/Users/rameshvel/work/storeJam/modernkirana/apps/customer/ios/Runner/OrangeApp/GoogleService-Info.plist' to '/Users/rameshvel/Library/Developer/Xcode/DerivedData/Runner-gqmldwjuahwuwkgijmfokisjscam/Build/Products/Debug-AppleApp-iphonesimulator/Runner.app/GoogleService-Info.plist'
2) Target 'Runner' (project 'Runner') has copy command from '/Users/rameshvel/work/storeJam/modernkirana/apps/customer/ios/Runner/AppleApp/GoogleService-Info.plist' to '/Users/rameshvel/Library/Developer/Xcode/DerivedData/Runner-gqmldwjuahwuwkgijmfokisjscam/Build/Products/Debug-AppleApp-iphonesimulator/Runner.app/GoogleService-Info.plist'

A quick look at the copy bundle resource in the build phase revealed that flutter_flavorizr added a copy phase for   GoogleService-Info.plist for both the flavors.
I know I can remove just a unused flavor and add it back when I am building that one. Is there a simple way?

Hi @Rameshv , this looks weird. flutter_flavorizr checks if a previously defined build phase was there, and if so, it removes it and readds with the updated references. Maybe you previously customized the name of the "Firebase Setup" script?

Hi @Rameshv, I had the same problem, try to remove all references to GoogleService-Info.plist in Build Phases -> Copy Bundle Resources.

Thank you @Cyril212 , this solved my problem also.

Hi @Rameshv, I had the same problem, try to remove all references to GoogleService-Info.plist in Build Phases -> Copy Bundle Resources.

great bro, tks

Hi @Rameshv , this looks weird. flutter_flavorizr checks if a previously defined build phase was there, and if so, it removes it and readds with the updated references. Maybe you previously customized the name of the "Firebase Setup" script?

Thank you this solve my problem

Thank you @Cyril212 , this solved my problem also.

Hi @Rameshv, I had the same problem, try to remove all references to GoogleService-Info.plist in Build Phases -> Copy Bundle Resources.

u are my hero 😄

Hi @Rameshv, I had the same problem, try to remove all references to GoogleService-Info.plist in Build Phases -> Copy Bundle Resources.

What do you mean I do not understand.

Hi @Rameshv, I had the same problem, try to remove all references to GoogleService-Info.plist in Build Phases -> Copy Bundle Resources.

What do you mean I do not understand.

Here's how you can do that:

Open your Xcode project.
In the left-hand navigation pane, select your project to open the project editor.
In the project editor, select your target.
Select the "Build Phases" tab.
Expand the "Copy Bundle Resources" section.
Look for any references to GoogleService-Info.plist in the list of resources. You can identify it by its icon, which should look like a blue file with a white "G" on it.
If you find any references to GoogleService-Info.plist, select them and press the "-" button to remove them from the list.
Repeat step 7 for any additional references to GoogleService-Info.plist.
Once you have removed all references to GoogleService-Info.plist from the "Copy Bundle Resources" build phase, save your changes and try building your Xcode project again.

By removing all references to GoogleService-Info.plist from the "Copy Bundle Resources" build phase, you are ensuring that Xcode does not try to copy the file to the app bundle more than once, which can cause conflicts and errors like the one described in the GitHub issue.

Hi @Rameshv, I had the same problem, try to remove all references to GoogleService-Info.plist in Build Phases -> Copy Bundle Resources.

What do you mean I do not understand.

Here's how you can do that:

Open your Xcode project.
In the left-hand navigation pane, select your project to open the project editor.
In the project editor, select your target.
Select the "Build Phases" tab.
Expand the "Copy Bundle Resources" section.
Look for any references to GoogleService-Info.plist in the list of resources. You can identify it by its icon, which should look like a blue file with a white "G" on it.
If you find any references to GoogleService-Info.plist, select them and press the "-" button to remove them from the list.
Repeat step 7 for any additional references to GoogleService-Info.plist.
Once you have removed all references to GoogleService-Info.plist from the "Copy Bundle Resources" build phase, save your changes and try building your Xcode project again.

By removing all references to GoogleService-Info.plist from the "Copy Bundle Resources" build phase, you are ensuring that Xcode does not try to copy the file to the app bundle more than once, which can cause conflicts and errors like the one described in the GitHub issue.

thanks it works

@Rameshv, I had the same problem, try to remove all references to GoogleService-Info.plist in Build Phases -> Copy Bundle Resources.

Hi, I had the same issue and I did this solution but if I remove these references, I get this error while building using vscode with a launch.json configuration:
Error (Xcode): lib/main.dart: Error: Error when reading 'lib/main.dart': No such file or directory
Here is my launch.json:
"configurations": [
{
"name": "my-app dev",
"request": "launch",
"type": "dart",
"program": "lib/main_development.dart",
"args": [
"--target",
"lib/main_development.dart",
"--flavor",
"development",]
},
{
"name": "my-app staging",
"request": "launch",
"type": "dart",
"program": "lib/main_staging.dart",
"args": [
"--target",
"lib/main_staging.dart",
"--flavor",
"staging",]
},
{
"name": "my-app production",
"request": "launch",
"type": "dart",
"program": "lib/main_production.dart",
"args": [
"--target",
"lib/main_production.dart",
"--flavor",
"production",]
}
]
}

@Cyril212
Thanks for the reply
It worked for me


[Memory Leak] Underlying native Android views aren't destroyed after JS component is unmounted

In react-native android 0.17 , after releasing component instance, android native Views doesn't seems to be destroyed. this has caused memory consumption by my app to hundreds of MB for just few navigation . to test this a wrote a simple component to test this:
'use strict';

var React = require('react-native');
var {
  AppRegistry,
  StyleSheet,
  Text,
  View,
  ScrollView,
  TouchableHighlight ,
} = React;

var app8 = React.createClass({
  getInitialState: function(){
    return {
      load: false
    }
  },
  load: function(){
    this.setState({load:true})
  },
  unload: function(){
    this.setState({load:false})
  },
  render: function() {
    var txts=[]
    if(this.state.load ){
      for (var i = 0; i < 4000; i++) {
        txts.push(<Text key={i} style={styles.item}>item-{i}</Text>)
      };
    }
    return (
      <View style={styles.container}>
        <Text>load? :{""+this.state.load}</Text>

        <View style={styles.buttonsHolder}>
          <TouchableHighlight  style={styles.button} onPress={this.load} >
            <Text>Load</Text>
          </TouchableHighlight >
          <TouchableHighlight  style={styles.button} onPress={this.unload} >
            <Text>clean</Text>
          </TouchableHighlight >
        </View>

        <ScrollView>
          {txts}
        </ScrollView>
      </View>
    );
  }
});

var styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: '#F5FCFF',
  },
  buttonsHolder: {
    justifyContent: 'center',
    alignItems: 'flex-start',
    flexDirection: 'row',
  },
  item: {
    fontSize: 14,
    textAlign: 'center',
    margin: 5,
  },
  button: {
    backgroundColor: '#eee',
    margin: 5,
    padding: 20,
  },
});

AppRegistry.registerComponent('app8', () => app8);

then after runing in unloaded state:
adb shell dumpsys meminfo com.app8
E:\books3\_pdfs\js>adb shell dumpsys meminfo com.app8
Applications Memory Usage (kB):
Uptime: 29327774 Realtime: 63386034

** MEMINFO in pid 25713 [com.app8] **
                   Pss  Private  Private  Swapped     Heap     Heap     Heap
                 Total    Dirty    Clean    Dirty     Size    Alloc     Free
                ------   ------   ------   ------   ------   ------   ------
  Native Heap        0        0        0        0     7384     5205      202
  Dalvik Heap     7982     7736        0     1304    18924    14562     4362
 Dalvik Other     2036     1908       12      736
        Stack      164      164        0        0
    Other dev      932      928        4        0
     .so mmap     5108     1348     2712     3264
    .apk mmap      227        0       24        0
    .ttf mmap       81        0       68        0
    .dex mmap     3274       64     2584      196
   Other mmap       19        4        4        0
      Unknown    13674    13668        0      168
        TOTAL    33497    25820     5408     5668    26308    19767     4564

 Objects
               Views:       17         ViewRootImpl:        2
         AppContexts:        4           Activities:        1
              Assets:        3        AssetManagers:        3
       Local Binders:        9        Proxy Binders:       21
    Death Recipients:        0
     OpenSSL Sockets:        0

 SQL
         MEMORY_USED:        0
  PAGECACHE_OVERFLOW:        0          MALLOC_SIZE:        0

what here is important is 'Views' that is number of android native Views that is 17.
after pressing 'load':
adb shell dumpsys meminfo com.app8
Applications Memory Usage (kB):
Uptime: 29405723 Realtime: 63463982

** MEMINFO in pid 25713 [com.app8] **
                   Pss  Private  Private  Swapped     Heap     Heap     Heap
                 Total    Dirty    Clean    Dirty     Size    Alloc     Free
                ------   ------   ------   ------   ------   ------   ------
  Native Heap        0        0        0        0    29428    29162       77
  Dalvik Heap    29659    29380        0     1296    39412    35901     3511
 Dalvik Other     3185     3084       12      704
        Stack      160      160        0        0
    Other dev     1336     1332        4        0
     .so mmap     5141     1364     2720     3196
    .apk mmap      281        0       24        0
    .ttf mmap       81        0       68        0
    .dex mmap     3452       64     2636      196
   Other mmap       20        4        4        0
      Unknown    85577    85568        0       72
        TOTAL   128892   120956     5468     5464    68840    65063     3588

 Objects
               Views:     4015         ViewRootImpl:        2
         AppContexts:        4           Activities:        1
              Assets:        3        AssetManagers:        3
       Local Binders:        9        Proxy Binders:       19
    Death Recipients:        0
     OpenSSL Sockets:        0

 SQL
         MEMORY_USED:        0
  PAGECACHE_OVERFLOW:        0          MALLOC_SIZE:        0

number of Views is 4015,that is expected, but after pressing 'clean' button native android Views instanse should go back to around 17, but it dosn't:
after few minutes;
E:\books3\_pdfs\js>adb shell dumpsys meminfo com.app8
Applications Memory Usage (kB):
Uptime: 30152367 Realtime: 64210627

** MEMINFO in pid 25713 [com.app8] **
                   Pss  Private  Private  Swapped     Heap     Heap     Heap
                 Total    Dirty    Clean    Dirty     Size    Alloc     Free
                ------   ------   ------   ------   ------   ------   ------
  Native Heap        0        0        0        0    29996    12684    16931
  Dalvik Heap    31468    31244        0     1296    41068    37999     3069
 Dalvik Other     3330     3236        8      696
        Stack      144      144        0        0
    Other dev     1080     1076        4        0
     .so mmap     5080     1364     2632     3192
    .apk mmap      226        0       24        0
    .ttf mmap       76        0       64        0
    .dex mmap     3212       64     2388      196
   Other mmap       19        4        4        0
      Unknown    87412    87404        0       72
        TOTAL   132047   124536     5124     5452    71064    50683    20000

 Objects
               Views:     4017         ViewRootImpl:        3
         AppContexts:        4           Activities:        1
              Assets:        3        AssetManagers:        3
       Local Binders:        7        Proxy Binders:       19
    Death Recipients:        0
     OpenSSL Sockets:        0

 SQL
         MEMORY_USED:        0
  PAGECACHE_OVERFLOW:        0          MALLOC_SIZE:        0

somehow those native views is still referenced and GC doesn't clean them up(could the source of this leak be in js if those created views is still referenced. ). this is serous bug, in any none trivial app after few navigation memory consumption sky rocks.
Memory usage is biggest hold back of react-native for me.
After pressing 'load' and 'clean' few times, number of views and memory usage just goes up, this seem to be js side problem because memory usage of 'Unknown' (javascriptcore is included in Unknown) is increasing linear.

"skevy added Android Needs Response"
what i should respond? nothing asked!

He means that someone in RN team needs to respond, not you.

Thank you, I'm not much active in github and not much familiar with it's
conventions, it's a shame ! :)
On Thu, Dec 24, 2015 at 11:41 PM, Richard Lai<EMAIL_ADDRESS>wrote:

He means that someone in RN team needs to respond, not you.
—
Reply to this email directly or view it on GitHub
https://github.com/facebook/react-native/issues/4963#issuecomment-167155310
.

No any update on this? this is a huge problem.

cc @kmagiera @mkonicek @astreet

Pinging people internally to take a look.

Hey!
I tried to repro this on master but was unable to. I see views starting at 45 and going up to 4045 when I click load. Clicking clear makes the count go down to 45 again.
Before:
blom$ adb shell dumpsys meminfo com.facebook.catalyst.launcher
Applications Memory Usage (kB):
Uptime: 6584076 Realtime: 6584076

** MEMINFO in pid 2381 [com.facebook.catalyst.launcher] **
                   Pss  Private  Private  Swapped     Heap     Heap     Heap
                 Total    Dirty    Clean    Dirty     Size    Alloc     Free
                ------   ------   ------   ------   ------   ------   ------
  Native Heap     4590     4492        0        0    24576    14836     9739
  Dalvik Heap     4044     3964        0        0     4895     3761     1134
 Dalvik Other      216      204        0        0
        Stack      200      200        0        0
    Other dev        5        0        4        0
     .so mmap     4792      280     3216        0
    .apk mmap      346        0       60        0
    .ttf mmap      106        0       64        0
    .dex mmap     8388        0     8376        0
    .oat mmap     2398        0      444        0
    .art mmap     1255      716        4        0
   Other mmap       56        4       12        0
      Unknown    28869    28868        0        0
        TOTAL    55265    38728    12180        0    29471    18597    10873

 Objects
               Views:       45         ViewRootImpl:        1
         AppContexts:        3           Activities:        1
              Assets:        2        AssetManagers:        2
       Local Binders:       14        Proxy Binders:       17
       Parcel memory:        4         Parcel count:       18
    Death Recipients:        0      OpenSSL Sockets:        0

 SQL
         MEMORY_USED:        0
  PAGECACHE_OVERFLOW:        0          MALLOC_SIZE:        0

After clicking load:
blom$ adb shell dumpsys meminfo com.facebook.catalyst.launcher
Applications Memory Usage (kB):
Uptime: 6612108 Realtime: 6612108

** MEMINFO in pid 2381 [com.facebook.catalyst.launcher] **
                   Pss  Private  Private  Swapped     Heap     Heap     Heap
                 Total    Dirty    Clean    Dirty     Size    Alloc     Free
                ------   ------   ------   ------   ------   ------   ------
  Native Heap    35649    35556        0        0    69632    60122     9509
  Dalvik Heap    30139    30064        0        0    39718    23907    15811
 Dalvik Other      852      852        0        0
        Stack      196      196        0        0
    Other dev        5        0        4        0
     .so mmap     4820      280     3240        0
    .apk mmap      346        0       60        0
    .ttf mmap      106        0       64        0
    .dex mmap     8508        0     8496        0
    .oat mmap     2593        0      572        0
    .art mmap     1407      728      140        0
   Other mmap       56        4       12        0
      Unknown    76377    76376        0        0
        TOTAL   161054   144056    12588        0   109350    84029    25320

 Objects
               Views:     4045         ViewRootImpl:        1
         AppContexts:        3           Activities:        1
              Assets:        2        AssetManagers:        2
       Local Binders:       13        Proxy Binders:       17
       Parcel memory:        4         Parcel count:       18
    Death Recipients:        0      OpenSSL Sockets:        0

 SQL
         MEMORY_USED:        0
  PAGECACHE_OVERFLOW:        0          MALLOC_SIZE:        0

After clicking clean:
blom$ adb shell dumpsys meminfo com.facebook.catalyst.launcher
Applications Memory Usage (kB):
Uptime: 6643315 Realtime: 6643315

** MEMINFO in pid 2381 [com.facebook.catalyst.launcher] **
                   Pss  Private  Private  Swapped     Heap     Heap     Heap
                 Total    Dirty    Clean    Dirty     Size    Alloc     Free
                ------   ------   ------   ------   ------   ------   ------
  Native Heap    15365    15272        0        0    65536    23516    42019
  Dalvik Heap    12307    12232        0        0     5449     3345     2104
 Dalvik Other      500      500        0        0
        Stack      204      204        0        0
    Other dev        5        0        4        0
     .so mmap     4820      280     3240        0
    .apk mmap      346        0       60        0
    .ttf mmap      106        0       64        0
    .dex mmap     8520        0     8508        0
    .oat mmap     2615        0      592        0
    .art mmap     1411      732      140        0
   Other mmap       56        4       12        0
      Unknown    77717    77716        0        0
        TOTAL   123972   106940    12620        0    70985    26861    44123

 Objects
               Views:       45         ViewRootImpl:        1
         AppContexts:        3           Activities:        1
              Assets:        2        AssetManagers:        2
       Local Binders:       13        Proxy Binders:       17
       Parcel memory:        4         Parcel count:       18
    Death Recipients:        0      OpenSSL Sockets:        0

 SQL
         MEMORY_USED:        0
  PAGECACHE_OVERFLOW:        0          MALLOC_SIZE:        0

Could you try to repo this on 0.18 (or better yet, master)?

Hi,
I tested in RN  version 0.18 and can confirm problem has been solved in this version. :ok_hand:
And tank you for your response and your awesome product.

+1 I tried with 0.18.1 Memory is not released after scene popped (Android),

@Nareshbathina I remember when i was checking this problem for version 0.18 memory Views did get released from react Native but it's memory in heap will not released  until Android garbage collection triggered. android 4.4 and below GC is a bit funny and doesn't  trigger until its be necessary and its really is funny how it works. in android +5.0 GC works very good.
Which version of android did you tested?
And remember this issue  was about React Native didn't release reference to Views.
if in the example that i wrote you press 'clean' and 'load' few times and 'Views' count remain 4000 (in android 4.4 and below will) , it's nothing to do with RN, it's just how Android works.

@jozn Can you please provide steps for how can I manage memory (GC) in android, I'm using react native 0.30 version and android above 5.0,


Fix #1363

Fix copied from IThundxr, https://github.com/Fabricators-of-Create/Create/commit/16408e75ff79aedc05043fa1752d3c4ad7de852f
Don't merge, just opening PR to say the fix works on 1.20.1 too

This was already cherry picked this morning? https://github.com/Fabricators-of-Create/Create/commit/16408e75ff79aedc05043fa1752d3c4ad7de852f

I opened the 1.20.1 dev brach and the commit wasn't made there, only in 1.18.2 dev branch I believe

Seems it was missed somehow during merge, fixed on all dev branches now


Refactor and Prepare to add Neukolln Example

No noteworthy changes in the code moved around, I'm assuming

the only semantic change was in some of the error paths regarding access by lane, in tests I trust.