Makes jsPsych an exportable module

Picked up where #482  left off, and cleaned up the tests to work, took advantage of the fact that jest has built in jsDOM functionality.
My thought going forward, is to have all the plugins in folders, with package.json and main.js and make them exportable as well. That way users can follow semver and use only specific versions of a plugin.
I'm using this fork successfully in a React project like so:
import jspsych from 'jspsych';
window.jsPsych = jspsych;
import 'jspsych/plugins/jspsych-audio-button-response.js';
import 'jspsych/plugins/jspsych-audio-keyboard-response.js';
import 'jspsych/plugins/jspsych-call-function.js';
import 'jspsych/plugins/jspsych-survey-likert.js';
import 'jspsych/plugins/jspsych-survey-multi-choice.js';
import 'jspsych/plugins/jspsych-instructions.js';
import 'jspsych/plugins/jspsych-html-keyboard-response.js';

Then handling all the jspsych calls in my componentDidMount
This also doesn't break any tests, and all the examples still work fine.

@RobAWilkinson thanks for getting this process started. I need to learn more about the build tools that (I think) would be required to implement to generate an easy-to-use, node-free experience that closely matches how the current process works. Basically, I don't want to scare off the many people who use jsPsych as their first adventure into JavaScript programming by further complicating how the plugin loading process works from the developer point of view.

Hi @jodeleeuw,
Recently updated my fork, I set up  a post install task that will take all the regular normal files we have, copy them, and wrap them in a UMD module definition so they can be required by people who might be using browserify, webpack etc.
I also took the liberty of seperating the code out and moving code that can be tweaked and developed into a lib folder, and leaving the build folder as the code that is required.
Long term, this will also allow us to possibly add in ES6 features and just build and shim for older browsers that do not support those features yet.
Would love to get some feedback from you or the other contributors. As it stands now, all the examples in examples/ work fine, and the entire test suite passes.
I believe we may have to update some of the documentation in order to deal with the code now being in lib. Can we create an issue for updating docs and connect it to this?

created an issue #529 to help track the work related to this.

Showing my GH ignorance here: Can I modify this PR to create a new branch? I think this is a big enough change that it should happen on a separate branch for now.

@jodeleeuw if you add the new branch to the master fork that you want this merged to, you then click the edit button near the title of the PR and redirect the merge to that branch instead

@teonbrooks thats correct, as long as it is in lib/plugins/ directory than it should work fine, just remember to import from build/plugins.

@teonbrooks thats correct, as long as it is in lib/plugins/ directory than it should work fine, just remember to import from build/plugins.

@RobAWilkinson, just for clarification, say I have my experiment repo, p300,
p300
|_ package.json
    files
         |_ experiment.js
            custom_plugin.js

and I have my experiment script, experiment.js, would I be able to do the following to import a custom plugin:
experiment.js
import jspsych from 'jspsych';
import './custom_plugin.js';

@jdpigeon, this thread might be useful for you

Any updates? This would be extremely helpful for the app I'm working on where I'd like to incorporate jsPsych with as little fuss as possible (ideally as a React component that I can just hand experimental details to)

@RobAWilkinson, I have a question about the gulp within react components. so, with jspsych standalone, npm install works and the gulp does it thing postinstall.
I was attempting to update my react component that I made for jspsych but I noticed that if I didn't include gulp at my jspsych-react package level, it wouldn't properly install the jspsych module.
you mentioned that you got this to work for your react component; do you have to include gulp with your react components or does it work properly for you?


Updated localizer documentation for mobile

Updated links talking about mailing list
Updated Firefox for Android information, since android-l10n mentions are now irrelevant
Updated Firefox iOS info as well

@flodolo Hopefully fixed it all!

Merging since just a nit was left and this was approved. thanks flod!


add auto-rotating particles

This was totally not a 4...

Could we remove the black borders of the particles, or are they intended?

well, simple design ^^ they are there by default


Add German Accents

This is just a minor contribution, but one needs to get started somehow.
Btw. this plugin works great for me. Thanks for your work (and the very useful documentation hidden in a resolved bug report).

Ok so people are actually using this, I'm gonna clean the codebase a tiny bit and try to write a start of a proper readme.
Thank you for your contribution !!!!

@tobiasgrosser Just published version 0.0.3 on npm that contains your changes and an actual readme :)
Thanks again

@tobiasgrosser Just published version 0.0.3 on npm that contains your changes and an actual readme :)
Thanks again

Amazing. The new README provides exactly the information I needed! (I figured it out myself, but it will be super useful for future users)

@giltho, thanks a lot for the super-fast response. I opened a couple of feature requests that list the features I am personally using on grosser.es as well as spcl.inf.ethz.ch. These are the last missing pieces to get my bibliography fully moved to gatsby.

My pleasure :) I've added all the features you asked for now, if anything else, don't hesitate to ask I'll see what I can do, or write a PR yourself if I'm in a day where it's harder for me to be responsive :)

Wow, you are very quick!
I tested all of them and updated the corresponding issues. I also found a couple minor extensions that I would potentially use in the future.
Finally, I am not 100% certain, but I feel https://www.gatsbyjs.com/dashboard is not handling dependencies to unpacked packages well. I am not certain, but it seems I cannot use these new features before your next release. Hence, I would appreciate if you could spin another release (hoping this is not too hard).
Also, I realized you do not 'git tag' your releases. This might be a good idea.


databricks: switching parameters to be a proper struct

This is a continuation of #6917
From #6917
As a part of the provisioning process, Databricks Clusters allow users to override certain values via a parameters block. Since there appears to be a fixed list of these:

customPrivateSubnetName, customPublicSubnetName, customVirtualNetworkId, enableNoPublicIp, relayNamespaceName, storageAccountName, storageAccountSkuName, vnetAddressPrefix, resourceTags, amlWorkspaceId.

from a usability perspective this would match other SDK's if these values were exposed as a native struct/class depending on the language, rather than being a generic dictionary; as such this PR changes parameters to a struct rather than a dictionary.

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

Automation for azure-sdk-for-python

Automation for azure-sdk-for-go

@mbfrahry
Please add code owner to reviewer. Also please try to pass CI as well. If you think any validation is not your fault please raise it.

You don't have permission to trigger SDK Automation.
Please add yourself to Azure group from opensource portal if you are MSFT employee,
or please ask reviewer to add comment *** /openapibot sdkautomation ***.
Please ask<EMAIL_ADDRESS>(or NullMDR in github) for additional help.

@weidongxu-microsoft @KrisBash thanks for reviewing - the tests pass now 👍

/azp run


Dataset file missing

I found that the Annotations file is missing compared to the prepared dataset when downloading the YouTube VIS 2021 dataset. Can you help solve this problem? Download link:https://drive.google.com/drive/folders/12DxR2HWTVjULNwKVMdYAvhOmZ9gBxsX2

You can download it from here.

Yes, I  checked the files and it seems like the annotation files are missing after downloading from https://codalab.lisn.upsaclay.fr/competitions/7680#participate-get_data

Yes, I checked the files and it seems like the annotation files are missing after downloading from https://codalab.lisn.upsaclay.fr/competitions/7680#participate-get_data

Can you download the train.zip file, extract it, and you will get JPEGImages and instances.json, which are the video and annotation files respectively?

Yes, I can, but I can't find the annotation files,and is instance.json just train.json?

Yes, instances.json is indeed the annotation file.

Thank you, I would like to know how to convert instances.json to annotation files in the step of Preparing Datasets for DVIS


base: webadmin: Don't take substring of nil variable

When converting interface names to UCI network names
webadmin fails if there is no UCI network name because
webadmin failed to ensure uciname has a value before
attempting to take a substring.
Signed-off-by: Daniel Dickinson<EMAIL_ADDRESS>

I've tested this and am hoping you agree it's simple and and non-controversial enough to just merge.


Search engine indexing issue for routes that require data to be fetched

I have an angular app and I've submitted sitemaps to Google. The app uses a firebase back-end and content is located in Firestore so fetched for each route/url.
The issue: The vast majority of results indexed in Google appear to be processing the page before the remote content has been retrieved so Google indexes a blank page. This results in the page becoming an 'omitted' from google results.
If you want to see an example of this then you look through some of the pages here: https://goo.gl/tDqKcu
Is there a way I can announce a page as loaded or declare a page as loading until the content is ready? (I guess the http 200 code is only sent once with it being a single page app).

I do not think this has anything to do with AngularDart, sorry. You'd have to see how other SPAs handle it.

I do think that this is to do with the Angular Dart framework. Other SPA frameworks have created solutions such as server side rendering to solve this problem such as 'Angular Universal'. I don't know what the solutions are for AngularDart but this is/will become an issue for anyone who chooses people using this framework. Indexing and search prominence is after all a fundamental cornerstone of any web app.
I do believe as a framework, Angular Dart should at very least have a documentation around this subject, even if it's just recommending approaches if there is nothing baked in to the actual framework.
I had hoped that introducing sitemaps would solve the issue but it hasn't it's just highlighted the issue I described above where the page isn't fetched quick enough for GoogleBot to snapshot it.
My personal concern is due to the fact that I'm readying the launch a large Angular Dart project which will replace a successful web service that has a very strong search presence. I would feel more comfortable about this if I had a solution to the search & indexing issue of Angular Dart apps.


Typo Typing Tester Game Design Enhancement

This pull request contains the updated code for the "Typo Typing Tester Game Design Enhancement". It is the result of following issue:
Typo Typing Tester Game Design Enhancement #916
Output Screenshot

Hi @panwar8279
Kindly review this pull request as soon as possible so that I can publish other issues and pull requests.


chore(deps): update module github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper to v1.8.5

This PR contains the following updates:

Package
Type
Update
Change

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.8.4 -> v1.8.5

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper
v1.8.5
Compare Source
Bug Fixes

deps: update module github.com/ibm/platform-services-go-sdk to v0.35.0 (#494) (020338a)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 5.1.0 :tada:
The release is available on:

GitHub release
v5.1.0

Your semantic-release bot :package::rocket:


Capability to re-write actual request/response

Pardon my ignorance if this has already been addressed - I am looking for a way to rewrite actual requests and responses as they are proxied via wiremock.  Is this possible w/ wiremock given the new extension point capability?
Thanks in advance for your answer.
/veli
BTW - this tool is AMAZING!!

Unfortunately you can only replace the responses to specific requests, rather than using the original request or response as an input. The response transformer extension isn't any help either because it allows you to transform the response definition (which simply says "go and fetch the actual response from here"), rather than the actual response.

Agree this might be a useful additional extension though.

Where do I vote for such elegant feature? :-)

I've also hit this problem.  I need a way to adjust headers coming back from the proxy to (for example) reduce cache expiry times.  We found the hard way that this doesn't work, so would also like to see this feature.

I've had similar feedback to this a few times now so clearly we should add this.
I'm really struggling to time-wise to do anything more than merge PRs at the moment. Would you consider creating one?

I'll look into it.
On Friday, 19 June 2015, Tom Akehurst<EMAIL_ADDRESS>wrote:

I've had similar feedback to this a few times now so clearly we should add
this.
I'm really struggling to time-wise to do anything more than merge PRs at
the moment. Would you consider creating one?
—
Reply to this email directly or view it on GitHub
https://github.com/tomakehurst/wiremock/issues/243#issuecomment-113578247
.

I've just pushed 2.0.3-beta to Maven central, which includes the ability to transform Responses in addition to ResponseDefinitions. You can also now supply per-stub parameters in both cases.


Prevent sending messages which render as empty

#19559 addressed the rendering of messages which are empty; however, it is still possible to send such messages.
Specifically, the empty fence:
```
```

...renders as an empty message.  There are likely other ways as well.
We should prevent sending such messages, with a pop-up similar to when you try to send a truly empty message.

@zulipbot claim

@zulipbot claim


Unable to download symbols when using traefik

Describe the issue
I have created two containers on an azure vm (not using the arm template). The containers are up and running. WebClient works fine and no errors were thrown at container creation. When I try to download symbols via vscode from outside the vm (Published Ports: 80, 443, 8080, 8180, 7046-7049) it fails because of "Bad Gateway". Trying to download them on the vm results in the same error.
I am not sure if I am missing something, haven't found any issues on this. Is this not supported?
Traefik Setup
I'm using a wildcard certificate, following the guide of @tfenster . The traefik container is up and running.
TOML seems fine and the container logs don't show any errors.
Scripts used to create container and cause the issue
Setup-TraefikContainerForBCContainers -PublicDnsName"<Dns name within wildcard cert>" -ContactEMailForLetsEncrypt noneed

New-NavContainer -accept_eula `
                -alwaysPull `
                -accept_outdated `
                -containerName "<Placeholder name>" `
                -auth NavUserPassword `
                -shortcuts Desktop `
                -includeTestToolkit `
                -includeTestLibrariesOnly `
                -doNotUseRuntimePackages `
                -doNotExportObjectsToText `
                -imageName "mcr.microsoft.com/businesscentral/sandbox:de-ltsc2019" `
                -licenseFile "<URL to our license for 16.x>" `
                -Credential $Credential `
                -useSSL `
                -useBestContainerOS `
                -updateHosts `
                -restart always `
                -assignPremiumPlan `
                -useTraefik `
                -PublicDnsName "<Dns name within wildcard cert>"

Full output of scripts
Both containers were created successfully, getting the expected output + new url's for traefik.
I don't have the output at hand, could recreate the containers and add to this issue if you want.

Screenshots

I tried every combination of the url inside the launch.json I could think of (for example https://DNS/ContainerName and https://DNS/ContainerNamedev - the dev url from the output when using traefik). Also tried other ports like 443.
Do you need more info? Had to hide stuff because it's company info.

Figured it out. Working launch.json now looks like this:
"server": "https://{DNS only}/",
"serverInstance": "{ContainerName joined by dev}",
"port": 443


Not Issue : Java form definition generator for Angular Schema form

Hi Team,
I wasn't able to join you by email, so i'm creating an issue in order to announce you my SF Java UI Library. It's related to your project angular-schema-form.
The SF-JAVA-UI is a server side extension for the Json Schema Form library,  It make the integration of your library into Java-Angular Web Project easier. Developers will not care about how to define your schema and your screen form definition. He will just annotate his fields, and just get the screen.
At this point, we start supporting the basic fields types and widgets that you support. You can check the release note
The library is extendable also. Developers can support their custom angular schema field (addons), so they add their own annotation and properties, and implement their custom generator, and they start using that into their project also. A separate repository was added to centralize the custom addons. (Documentation is missing for this feature)
Please, can you take a look at the main project and give us your feedbacks, and if you like it, you can share this announcement with the large community.

@SaifJerbi I like it :trophy:, feel free to join the Gitter channel and let others know, you can always contact me quicker via Gitter as well
I like what you did with my logo too, perhaps java-schema-form should be a part of this org. :wink:

Hi @Anthropic we really appreciate your collaboration. I joined the Gitter channel.
I hope that there will be contributers.
I like to name it : "java-schema-form", it would a pleasure for us to be part of your org

I asked now in Gitter for some insight into how it works internally. I was thinking to be called java-schema-form it should be able to operate as a server for a folder of schema and forms as well, is that how it works, does it generate then serve or does it compile direct to server output?


Missing Python packages due to cleared $PYTHONPATH

Problem Description
We install Spyder alongside other globally installed Python packages that are in non-standard locations. To make Python find them we either add the locations to $PYTHONPATH or use a custom sitecustomize.py whose location must be in $PYTHONPATH
When starting Spyder it fails with errors such as No module named 'zmq'.
I traced that to https://github.com/spyder-ide/spyder/blob/a857397bb4fc80c597ae3ba1ac478765137a1117/spyder/app/start.py#L13-L18
That removes the entries that are required such that Python finds the modules and our sitecustomize
Removing that results in a similar problem as the IDE console shows

Error while finding module specification for 'spyder_kernels.console' (ModuleNotFoundError: No module named 'spyder_kernels')

That arises due to removing $PYTHONPATH from the environment the kernel starts in:
https://github.com/spyder-ide/spyder/blob/a857397bb4fc80c597ae3ba1ac478765137a1117/spyder/plugins/ipythonconsole/utils/kernelspec.py#L172
I'm not fully sure what the intention was, but maybe that can be solved differently. Is that possible?
Versions

Spyder version: 5.5.1
Python 3.11.3 64-bit | Qt 5.15.10 | PyQt5 5.15.10 | Linux 4.18.0-513.24.1.el8_9.x86_64 (x86_64)

Hey @Flamefire, thanks for reporting. You said:

We install Spyder alongside other globally installed Python packages that are in non-standard locations. To make Python find them we either add the locations to $PYTHONPATH or use a custom sitecustomize.py whose location must be in $PYTHONPATH

We haven't contemplated the possibility of people building a Python installation by adding different directories to PYTHONPATH.

I'm not fully sure what the intention was

We remove PYTHONPATH before starting Spyder and from kernels to avoid users adding directories to it with modules named random.py, string.py, math.py, etc. Those modules would mask the respective standard library ones and cause major breakages. Most people are unaware that that is even possible and were very frustrated when it happened because it's not an easy problem to understand and find help about.

but maybe that can be solved differently. Is that possible?

I don't think there is a simple way to prevent the problem I mentioned without dropping PYTHONPATH from the env. But I'm open to suggestions.

We remove PYTHONPATH before starting Spyder and from kernels to avoid users adding directories to it

How exactly are they adding directories to it? IIRC I read in one of the related issues that it was about the current directory being used implicitly as a python search path. Maybe that could be detected and only that one removed from PYTHONPATH, potentially together with any empty values as IIRC PYTHONPATH=/foo: would mean the same as /foo:$PWD

PYTHONPATH is for user-provided and non-standard-library paths and should not be relied upon for Python or installed packages to work properly. Unfortunately, any user-provided path via PYTHONPATH presents a potential hazard to Spyder, not just the current working directory, so we cannot selectively "clean" this environment variable.
Rather than PYTHONPATH, I would recommend using a .pth file located in ../lib/python3.11/site-packages, relative to the Python executable you are using.
See https://docs.python.org/3.11/library/site.html#module-site

Unfortunately that doesn't work for our use case: We have environment modules managed via LMod to allow users to choose their desired version of software in a (heavily) shared environment/filesystem.
Those "modules" can be though of as (bash) scripts that are sourced and set/update and export environment variables. Hence the only thing a module providing a set of Python packages can do is append or prepend to $PYTHONPATH.
In particular we cannot modify the global Python installation and put additional files in there, so the .pth approach does not work for us.
We are switching to a mechanism using sitecustomize.py that reads a custom env variable to modify sys.path (as also described in your link). However we do put that into a separate folder and pass its location via $PYTHONPATH.
IIRC the $PWD is added to sys.path not $PYTHONPATH, I confused that. So any $PYTHONPATH would be something explicitly set by a user, so most "should" be aware of it.
As the intention is that people "were very frustrated when it happened because it's not an easy problem to understand and find help about." maybe another solution is to detect a set $PYTHONPATH and show a warning on startup to educate users that don't know about that but still allow this mechanism for users that do know and intended that (at least some like us)
If there is a message like "You have set $PYTHONPATH. If you encounter errors related to imports of Python packages you can try unsetting that environment variable before starting Spyder" then user that are not aware of that have some help right there without taking away control from all users.
I can also imaging another variable like $SPYDER_CLEAN_PYTHONPATH=0/false to disable that which we can set in the module. That would be in line with the standard $PYTHONNOUSERSITE or $PIP_REQUIRE_VIRTUALENV variables.

@Flamefire, thank you for the deeper explanation. I think I understand better now what your situation is.
How much control do you have over these module scripts?
Could you do something like the following?
Assuming there is a module located at /apps/modulefiles/mymodule/<ver>.lua, create a .pth file in /apps/modulefiles/mymodule/lib/python3.11/site-packages, containing the necessary paths. Rather than setting the PYTHONPATH variable in the module, set the PYTHONUSERBASE environment variable to /apps/modulefiles/mymodule. After loading this module, running Python will result in sys.path containing all the paths listed in the .pth file, without needing to modify PYTHONPATH.

@mrclary Thanks for the reply and suggestions

How much control do you have over these module scripts?

 BUT:
I was reporting this in the context of EasyBuild that installs the software and creates the modules. So we can control the contents of the modules. However there are many existing modules already installed on multiple HPC clusters so changing those existing modules is virtually impossible.

Rather than setting the PYTHONPATH variable in the module, set the PYTHONUSERBASE environment variable

I didn't know about that variable and it might come in handy.
However for our situation it doesn't help: There are multiple modules with Python packages. At the extreme it could be one module per package but in practice we have something like: 1 for Python (binaries etc.) and some base packages, 1 for commonly used packages from PyPI (~80 packages), 1 for numpy&scipy. And then some on top, e.g. h5py which requires HDF5 to be installed first, so it's sensible to put that in a extra modules in case some sites don't need it (and hence HDF5)
PYTHONPATH has the advantage of being a list. Hence we can prepend paths to this list in the same way we do with  $PATH and $LD_LIBRARY_PATH.
So one solution I see is putting our sitecustomize.py into the site-packages folder and using that mechanism to read possible sys.path entries from our custom env variable, so we no longer require $PYTHONPATH for that.
However that doesn't help with existing modules and we only recently introduced an option to use that as the default mechanism instead of $PYTHONPATH. But sites can still opt to use $PYTHONPATH for whatever reasons which would then cause that problem again for them.
And as confusing as it might be if users set up $PYTHONPATH and put modules/files inside those shadowing standard library ones it was very confusing for us to find the (by now 2) places where the PYTHONPATH we purposely set up was removed.
Hence the idea of only showing a warning as a starting point for users that are not aware of that.

 BUT: I was reporting this in the context of EasyBuild that installs the software and creates the modules. So we can control the contents of the modules. However there are many existing modules already installed on multiple HPC clusters so changing those existing modules is virtually impossible.

Yes, I see. That would require updating the EasyBuild and redeploy/reinstall across your clusters.

Rather than setting the PYTHONPATH variable in the module, set the PYTHONUSERBASE environment variable

I didn't know about that variable and it might come in handy. However for our situation it doesn't help: There are multiple modules with Python packages. At the extreme it could be one module per package but in practice we have something like: 1 for Python (binaries etc.) and some base packages, 1 for commonly used packages from PyPI (~80 packages), 1 for numpy&scipy. And then some on top, e.g. h5py which requires HDF5 to be installed first, so it's sensible to put that in a extra modules in case some sites don't need it (and hence HDF5)

Yes, I see. Python also provides PYTHONHOME for the purpose of using a non-standard location for the python libraries, but only for up to two locations and doesn't support scaling to 3 or more locations. In or order for PYTHONUSERBASE to work, each module would have to add/remove a .pth file to the $PYTHONUSERBASE/lib/python3.11/site-packages upon load/unload, which I don't think is possible.

So one solution I see is putting our sitecustomize.py into the site-packages folder and using that mechanism to read possible sys.path entries from our custom env variable, so we no longer require $PYTHONPATH for that.

I suppose that sitecustomize.py would require logic to determine which modules are loaded and modify sys.path accordingly.

However that doesn't help with existing modules and we only recently introduced an option to use that as the default mechanism instead of $PYTHONPATH. But sites can still opt to use $PYTHONPATH for whatever reasons which would then cause that problem again for them.

Yep. Either the change is made with EasyBuild and redeploy/reinstall across your systems, or changes are made to Spyder and you update Spyder. How is Spyder installed in your situation? Does it have its own module or is it installed in the base Python location?
Python 3.11 introduced the PYTHONSAFEPATH which prevents prepending potentially unsafe paths to sys.path. I don't know to what extent this applies to paths in PYTHONPATH but nevertheless cannot be used since Spyder still supports Python <3.11; but it might be something to look at in the future.
So, in conclusion, I agree that @Flamefire's suggestion to use an environment variable to switch PYTHONPATH treatment in Spyder may be the best solution for now. I propose SPY_ADD_PYTHONPATH. Spyder will not add paths from PYTHONPATH to sys.path upon startup if SPY_ADD_PYTHONPATH is unset or evaluates to False; Spyder will add paths from PYTHONPATH to sys.path upon startup if SPY_ADD_PYTHONPATH evaluates to True. What do @spyder-ide/core-developers think?

In or order for PYTHONUSERBASE to work, each module would have to add/remove a .pth file to the $PYTHONUSERBASE/lib/python3.11/site-packages upon load/unload, which I don't think is possible.

Actually, I think this is possible, but still not convenient.

Yes, I see. Python also provides PYTHONHOME for the purpose of using a non-standard location for the python libraries, but only for up to two locations and doesn't support scaling to 3 or more locations. In or order for PYTHONUSERBASE to work, each module would have to add/remove a .pth file to the $PYTHONUSERBASE/lib/python3.11/site-packages upon load/unload, which I don't think is possible.

Yes that is not possible because it is a shared filesystem, so that location needs to be at least specific to each user as they must not interfere with each other. But users can also have multiple independent tasks (from different shell sessions) running concurrently so that location must also be per-user-per-shell-session and still on the global filesystem which is impractical.

I suppose that sitecustomize.py would require logic to determine which modules are loaded and modify sys.path accordingly.

Correct. We have a shell variable that is prepended to by modules and that file reads it and adds the appropriate paths to sys.path

Yep. Either the change is made with EasyBuild and redeploy/reinstall across your systems, or changes are made to Spyder and you update Spyder. How is Spyder installed in your situation? Does it have its own module or is it installed in the base Python location?

Yes it has it's own module. So updating that is much easier, especially as currently it cannot be used as a module, although we have a workaround: We remove the unset-code with a patch. But a general solution avoiding the need for patches going forward would be appreciated.

Python 3.11 introduced the PYTHONSAFEPATH which prevents prepending potentially unsafe paths to sys.path. I don't know to what extent this applies to paths in PYTHONPATH

That seems to avoid the common issue with "wrong" files next to the file you want to execute. Interesting option indeed.

So, in conclusion, I agree that @Flamefire's suggestion to use an environment variable to switch PYTHONPATH treatment in Spyder may be the best solution for now. I propose SPY_ADD_PYTHONPATH. Spyder will not add paths from PYTHONPATH to sys.path upon startup if SPY_ADD_PYTHONPATH is unset or evaluates to False; Spyder will add paths from PYTHONPATH to sys.path upon startup if SPY_ADD_PYTHONPATH evaluates to True. What do @spyder-ide/core-developers think?

Just to be exact: Currently spider unsets $PYTHONPATH and removes those from sys.path after the fact. So maybe an inverse name is better SPY_CLEAN_PYTHONPATH which defaults to True to keep the current behavior but can be set to False/0/Off e.g. in our module. Or default to False and show a warning with a hint that this can be used given that the cleaning is basically a workaround for user mistakes, not something strictly required.

Just to be exact: Currently spider unsets  $PYTHONPATH and removes those from sys.path after the fact. So maybe an inverse name is better SPY_CLEAN_PYTHONPATH which defaults to True to keep the current behavior but can be set to False/0/Off e.g. in our module. Or default to False and show a warning with a hint that this can be used given that the cleaning is basically a workaround for user mistakes, not something strictly required. So I'd use the more common case by default to have more testing/experience for that.

CLEAN does describe what's going on better, however the default value of the variable will be unset and it will be anti-idiomatic to have the unset variable and the variable set to truish both perform the same action. The unset variable and setting the variable to falsey should produce the same result. ADD indicates that the paths should be (re)added to Spyder's runtime sys.path.

CLEAN does describe what's going on better, however the default value of the variable will be unset and it will be anti-idiomatic to have the unset variable and the variable set to truish both perform the same action. The unset variable and setting the variable to falsey should produce the same result. ADD indicates that the paths should be (re)added to Spyder's runtime sys.path.

True, I was just confused of what "ADD" would mean from the perspective of someone using $PYTHONPATH for that already so it sounds like it would add them twice.
Maybe SPY_USE_PYTHONPATH or SPY_ALLOW_PYTHONPATH? Or as in PYTHONNOUSERSITE/PYTHONDONTWRITEBYTECODE it could be something like SPY_NOCLEAN_PYTHONPATH

So, in conclusion, I agree that @Flamefire's suggestion to use an environment variable to switch PYTHONPATH treatment in Spyder may be the best solution for now. I propose SPY_ADD_PYTHONPATH. Spyder will not add paths from PYTHONPATH to sys.path upon startup if SPY_ADD_PYTHONPATH is unset or evaluates to False; Spyder will add paths from PYTHONPATH to sys.path upon startup if SPY_ADD_PYTHONPATH evaluates to True. What do @spyder-ide/core-developers think?

I agree that that's the best we could do to allow this use case.

`` it could be something like SPY_NOCLEAN_PYTHONPATH

I like this. 👍🏼

Yes I can test that as time allows. So there might be a bit of delay but it isn't so urgent that it should be fine.

Note: Could this be related/should supersede https://github.com/spyder-ide/spyder/issues/21633 ?

Note: Could this be related/should supersede #21633 ?

No; I think it is related to Windows subsystem for Linux. I left a comment.
However, the proposed solution for #22681 (current issue), may provide a workaround for #21633. But I think #21633 requires a dedicated fix.

@Flamefire @smax1,
I'll summarize what I think are the primary issues:

Spyder removes PYTHONPATH paths from its runtime environment sys.path, causing Spyder to fail to launch due to its inability to find necessary packages in non-standard locations (specified in PYTHONPATH). Spyder-kernels may be affected in the same way.
To pass PYTHONPATH on to user processes (IPython Console and others), Spyder only obtains PYTHONPATH from shell startup files (interactive + login) and not from the environment variables in its current launch environment, precluding possible changes from e.g. environment modules.

@mrclary Correct. And just to be clarify on the 2nd point: When started from a console on Linux PYTHONPATH from the environment is honored except paths matching a certain pattern
And for both we want an option to disable any filtering

@Flamefire, is it possible to have a dedicated python environment for Spyder to run from?
One issue with having library files located in non-standard locations, and presumably separately maintained, is that there could arise package conflicts that affect Spyder's operation.

I propose the following solution.

Spyder will move PYTHONPATH paths to the end of sys.path in its runtime environment at startup. This will allow Spyder to launch if non-standard library locations are listed in PYTHONPATH. Currently Spyder removes these paths, however moving them to the end should provide the same protection.
Spyder will not remove PYTHONPATH from the subprocess environment used to start Spyder-kenels.
Spyder-kernels will move PYTHONPATH paths to the end of sys.path at the beginning of startup. Currently, Spyder-kernels is protected when Spyder removes PYTHONPATH from the subprocess environment used to launch Spyder-kernels; Spyder-kernels will now be responsible for its own protection.
Spyder-kernels will remove PYTHONPATH paths from sys.path immediately after startup. This allows proper integration with the PYTHONPATH Manager plugin. Unlike Spyder, Spyder-kernels' runtime environment is the same as the user's Python environment (where the user runs his own code) so paths from the PYTHONPATH Manager will be injected into the sys.path in this environment. In order to work as expected, after Spyder-kernels starts, these paths will be removed (from the end of sys.path) and reapplied via the sig_pythonpath_changed signal from the PYTHONPATH Manager plugin.
get_user_environment_variables will continue to use os.environ from Spyder's runtime environment if Spyder was launched from the commandline, otherwise it will attempt to get them from an interactive login shell subprocess. Note this is important for Spyder launched from a shortcut: macOS will not see any shell startup variables, while Linux will only see login startup variables (i.e. ~/.bash_profile) but not interactive variables (i.e. ~/.bashrc).
Add a preference "Allow unsafe PYTHONPATH paths" to Preferences -> Application -> Advanced settings (default unchecked). This will allow bypassing the path filtering performed by PYTHONPATH Manager. The PYTHONPATH path filtering resides entirely within the PYTHONPATH Manager, is not used anywhere else, and is unrelated to Spyder startup; therefore I think it is more appropriate to use a preference setting rather than an environment variable.

Since #21769 includes a lot of refactoring of the PYTHONPATH Manager, I propose making these changes either after that PR or as part of that PR.

@ccordoba12, I think we need to reconsider how Spyder obtains the environment variables. After our discussion and updating the proposed solution above, I realized that we have an issue with remote kernels.
Environment variables, and by extension PYTHONPATH paths, are currently obtained from Spyder's runtime os.environ or an interactive login subprocess, both of which reflect local system variables. This works fine if the IPython console kernel environment is also local, but in the case of remote kernels, this is a problem. The reported environment variables from Tools -> Current user environment variables... will report environment variables from the local system, and will be different from "Show environment variables" in the IPython console hamburger menu (obtained from Spyder-kernel's os.environ).
Ideally, we should obtain "Current user environment variables..." and "Show environment variables" from the same source: Spyder-kernels' runtime environment. This makes sense because the only relevant environment variables (including PYTHONPATH) are those in the Spyder-kernels runtime environment. If the environment is local, then it will be effectively the same as what we currently do, but for remote kernels it will reflect the correct variables in the code execution environment.
I would propose doing the following:

Move spyder/utils/environ.py to spyder-kernels and have Spyder-kernels handle environment retrieval operations. Thus os.environ or interactive login subprocesses will be derived from the kernel environment, whether local or remote.
Remove Tools -> Current user environment variables... This does not make much sense in the context of multiple kernels that are possibly mixed local and remote. Environment variables are only relevant to the code execution environment and therefore should only be accessible from IPython Console related menus.
PYTHONPATH Manager also obtains PYTHONPATH from Spyder-kernels. This may allow users to use the PYTHONPATH Manager even for remote kernels. While it may not be possible to add remote paths via navigation, the user may still be able to type in remote paths to use. Path validation could also occur by Spyder-kernels rather than Spyder prior to updating sys.path.

What do you think?

@Flamefire, is it possible to have a dedicated python environment for Spyder to run from? One issue with having library files located in non-standard locations, and presumably separately maintained, is that there could arise package conflicts that affect Spyder's operation.

We more or less do have that which is why we set PYTHONPATH the way we do: We create an python environment with packages suitable for a given Spyder version by setting up LD_LIBRARY_PATH, PYTHONPATH and others to locations of suitable dependencies.
As all of them comes from recipes (think something like Docker or Singularity files) all software is (mostly) commonly maintained.

Add a preference "Allow unsafe PYTHONPATH paths" to Preferences -> Application -> Advanced settings (default unchecked). This will allow bypassing the path filtering performed by PYTHONPATH Manager.

What exactly is that filtering? And what are "unsafe PYTHONPATH paths"?

What exactly is that filtering?

I'm referring to the same thing you commented on here.

And what are "unsafe PYTHONPATH paths"?

Anything that is caught by the filtering is considered (potentially) unsafe 😄.


Update golang to 1.9.2 in install_golang.sh

This fix updates install_golang.sh to 1.9.2, see release:
https://golang.org/doc/devel/release.html#go1.9.minor
Signed-off-by: Yong Tang<EMAIL_ADDRESS>

Can one of the admins verify this patch?

Jenkins, test this please.


[error] impossible to get artifacts when data has not been loaded.

Hi,
After adding Esper 4.5 to my project dependencies, your plugin fails. Here the logging details:
[info] Trying to create an Idea module Hydrodesk Analytics
[error] impossible to get artifacts when data has not been loaded. IvyNode = org.antlr#stringtemplate;3.2.1
[error] {file:/home/mathias/Development/hydrodesk/analytics/}Hydrodesk Analytics/*:update-classifiers: java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded. IvyNode = org.antlr#stringtemplate;3.2.1
[info] Excluding folder target
....
My import:
"com.espertech" % "esper" % "4.5.0" excludeAll(
ExclusionRule(organization = "log4j"),
ExclusionRule(organization = "commons-logging")
)
Same happens with or without the excludeAll stuff.

Seems to be more Ivy related than the plugin or sbt related problem. Also the problem happens in sbt-level, try running for example the 'update-classifiers' sbt task.
Declaring the stringtemplate explicitly as a dependency seems to be a workaround.
But anyway, seems like nothing can be done in plugin code to fix this.


NETOBSERV-1017 Metrics: pod or service talkers

Requires:

https://github.com/netobserv/network-observability-operator/pull/388
https://github.com/netobserv/flowlogs-pipeline/pull/450

/hold

Rebased on https://github.com/netobserv/network-observability-operator/pull/388

@jpinsonneau I asked @memodi , we merged the split infra/apps PR so this one can be rebased and I'll go ahead with the review after that

@jpinsonneau I asked @memodi , we merged the split infra/apps PR so this one can be rebased and I'll go ahead with the review after that

:+1: thanks ! I cleaned the commit list. It's ready to go

Reviewed, something I think to change wrt ServiceMonitor API, apart from that lgtm!

Heads up on the impact, I have done more tests: I'm comparing with an alternative solution where I directly expose the pods metric without the pre-agg topK stage; and also, trying to see the impact on prometheus itself.
Here's FLP CPU / memory:

=> As seen before, the Aggregation stage has a huge impact on FLP memory. New metrics without pre-aggregation shows almost no difference compared with old metrics.
Prometheus memory:

=> The difference isn't easily identifiable bc prometheus already takes a lot of memory that always slightly fluctuates. I didn't see a significative difference between pre-aggregated and non-aggregated.
I also looked at other prometheus performance indicators, such as the "Head chunks" increase, there's indeed an increase ~+5K but that's about the same for pre-aggregated and non-aggregated (tbh, here I would have expected an advantage for the pre-aggregated ones)
So I have the feeling it doesn't worth it to pre-aggregate with topK, as it impacts us negatively while it doesn't show clear benefit for prometheus.

So I have the feeling it doesn't worth it to pre-aggregate with topK, as it impacts us negatively while it doesn't show clear benefit for prometheus.

Interesting, thanks for sharing that @jotak
Do you feel we should run some perfscale testing here to ensure we get the same results on bigger cluster ?
Else it would be straightforward to remove that stage

/hold

yeah perf tests would always be nice
btw, my methodology for checking prom's memory is probably not very good, I'll do again a few tests

@jotak following your feedback I removed the topk stage and took the opportunity to add packets / flows metrics disabled by default.
https://github.com/netobserv/network-observability-operator/pull/391/commits/6fe0612fbbc59447c90a72732f0480dc0c7db183
Let's see the impact on perfscale testing to decide if it's the right solution or if we should rollback the last commit

Note for QE: Do a large-scale pre-merge performance test, perhaps the 120 node scenario

Note for QE: Do a large-scale pre-merge performance test, perhaps the 120 node scenario

@memodi @nathan-weinberg the code is ready test as is. I'm waiting for the perf test results before merging as we have concerns about the impact on Prometheus performances.
:crossed_fingers: thanks !

we want cluster with large number of services and pods and measure prometheus before and after we have default ignoreTags and empty ignoreTags?

We can focus on with / without resources tag here since it's the purpose of that PR but having all of them is also good to see the extreme config.

Do you have metrics on prometheus side we should be looking at to measure impact there?

Storage / Memory would be good enough I guess. CPU would be a plus !

In terms of scale, I am thinking a 25 worker node cluster with ~2500 pods and 100 services, wdyt?

That's perfect :ok_hand:

Metrics measured:

prometheus_tsdb_head_series
prometheus rss
prometheus CPU

Measured in 3 iterations at above scale with catalog image on this PR:

with default config on ignoreTags - active TSDB: 2.05 Mil, RSS: ~10.9GB
with just egress, packets under ignoreTags - active TSDB: 2.44 Mil, RSS: ~13.5GB
with everything published (empty ignoreTags)- active TSDB: 3.77 Mil, RSS: ~18GB  - in this case you'll find it was stable at 3:09 Mil active TSDB, I noticed some FLP pods were crashing due to memory limits, so around 17:13, I increased FLP memory to 2GB and the active TSDB increased further to 3.77 Mil.

So where is the prom CPU ? Do you see an impact on it ?
What happened to FLP & prom after the memory limit increase  ? did it stabilized or just raise infinitely ?

@jpinsonneau @jotak - here's re-run of the test in Loki-less mode, with 3 iterations as shown in graph at same scale as previous run, in the end ignoreTags had config:
        ignoreTags:
        - egress
        - packets
        - nodes-flows
        - namespaces-flows
        - workloads-flows

towards the end Prometheus RSS and active TSDB were ~28.5 GB and 2.54 Mil respectively. CPU usage didn't impact as much so leaving it out.

IMHO As part of this feature, we restrict users to not letting them take off all tags from above list since it could make cluster prometheus unstable.

IMHO As part of this feature, we restrict users to not letting them take off all tags from above list since it could make cluster prometheus unstable.

Thanks for your testings @memodi ! The impact is quite huge as expected. I wonder if the topK stage would solve this cc @jotak
I agree we should improve and avoid letting the user select everything.
Here are some suggestions that could help in that way:

change ignoreTags to tags so the user actually needs to explicitly specify expected items
use enums:

resources / namespaces / workload to avoid picking these 3 at the same time
ingress / egress
packets / bytes  / flows

We could also think about a less flexible approach where we only have preconfigured cases the user can pick in a single enum such as:

nodes & pods ingress bytes
nodes & workload ingress bytes
nodes & namespaces ingress bytes
...

would it make sense to choose one of the tags of packets and bytes to reduce cardinality? can always add if customer asks for it.

@memodi , sorry for chiming in late here, but I'd say what we want to compare here is the impact on prometheus between the previous version that @jpinsonneau had, and the current version; the diff being that in the previous version we had a pre-aggregate stage in FLP that basically puts more load on FLP but that could potentially be better for prometheus.
I rebuilt an image based on Julien's previous commit (c1de0784) : quay.io/jotak/network-observability-operator:with-aggregate-stage : so the goal would be to run test perfs with this image and compare with the current PR image.

@memodi @jpinsonneau on the other discussion about preventing users to remove all ignored tags: what you suggest would be a breaking change, right? What about simply writing a big WARNING in the crd doc ?
fwiw, this is how the monitoring team does, in several places they write "if you modify this setting, closely monitor the prometheus-k8s deployment for excessive memory usage."  e.g: https://github.com/openshift/cluster-monitoring-operator/blob/master/pkg/manifests/types.go#L496

What about simply writing a big WARNING in the crd doc ?
fwiw, this is how the monitoring team does, in several places they write "if you modify this setting, closely monitor the prometheus-k8s deployment for excessive memory usage."

+1 that seems like best approach, we could also add similar Warning notes in documentation.

What about simply writing a big WARNING in the crd doc ?
fwiw, this is how the monitoring team does, in several places they write "if you modify this setting, closely monitor the prometheus-k8s deployment for excessive memory usage."

+1 that seems like best approach, we could also add similar Warning notes in documentation. cc @skrthomas

@memodi @jotak @jpinsonneau we can add the warning to the documentation as well. We currently don't have a docs tracker for https://issues.redhat.com/browse/NETOBSERV-139, due to no-doc impact on all the associated epics. So if we move forward with this, I'll create a docs epic for this small piece.

due to no-doc impact on all the associated epics. So if we move forward with this, I'll create a docs epic for this small piece

Epic NETOBSERV-139 containing this story would definitely need doc attention, not sure where it's flagged as no-doc?

@skrthomas Epic NETOBSERV-139 containing this story would definitely need doc attention, not sure where it's flagged as no-doc?

I guess it isn't actually labeled as no-doc, sorry for misrepresenting that. What I meant is that in documentation planning, this was discussed as not having any components with doc impact, and Julien's comment to me that the updates don't have any documentation impact. There are updates to the Health dashboard, but we don't go into specifics about the metrics shown there. Are there other stories within 139 that you think have doc impact?

@jotak would the warning be with the FlowCollector api doc re-generation?

@jotak would the warning be with the FlowCollector api doc re-generation?

yes

is aggregation performed only on netobserv_top_pods_ingress/egress_bytes_total ? I have workloads enabled but I don't see any aggregation happening there.

is aggregation performed only on netobserv_top_pods_ingress/egress_bytes_total ? I have workloads enabled but I don't see any aggregation happening there.

Yes, the top_bytes stage only affect new bytes metrics for pods / services. The already existing metrics are not altered by this PR.

Closing this as it will need to be reworked with NETOBSERV-1284 approach


Bump version to v1.24.0

Bump version to v1.24.x

Basic Tests C/C++ MacOS - #19614
Basic Tests Node MacOS - #19669
Basic Tests Python Windows - #19170


WIP: workshops/models.py: Add LanguageTag and LanguageQualification models

Based on a very restricted subset of RFC 5646.  This will
allow us to match instructors (and potentially students) with events
in an appropriate language.
As of the current tip (63fa782), this stores language ↔ person
associations, but doesn't currently allow you to edit weights (the
database has weights, but the UI doesn't provide a way to edit them,
and will clobber with 1s on person-edit).
There's currently no support for @pbanaszkiewicz's requested:

store event language in events
ask for the language in profile update form

I'm not familiar enough with the latter to know how to handle it.  Do
I have to do anything besides updating ProfileUpdateRequest?  I looked
around briefly for a template that we email instructors, since that
probably needs to be updated to list the languages we know about, but
couldn't find anything.  Can anyone give me some pointers or a refence
to docs or previous similar work?
I haven't stored event languages, because I expect there's not much
use for this information after the event is setup.  The EventRequest
form now uses a many-to-many language table, so we don't have to parse
user-entered languages.  And once the EventRequest is translated into
an Event, the language bits have already taken care of themselves
(unless we want to list them on out upcoming events feed?)
So this is a bit of a work-in-progress, but I think it's far enough
along to get some feedback ;).

ref #582

@wking thank you for the contribution. I went through your work and below are my comments.
The weights issue
The problem with 0÷1 grading is that no-one says what "1" and what "0" means. Does "1" mean I can speak English fluently, or does it mean I finished English studies?
Remember that people tend to under-evaluate their capabilities when self-assessing (what @gvwilson trains instructors to avoid).
So if we keep the weight system, I'd force LanguageQualification.weight to use predefined values (choices). This way we can help users self-assess.
For example we could use these values:

1 - I'm comfortable being a lead instructor for an event
0.75 - I'm comfortable being a helper during the event
0.5 - I'd rather not speak to the audience

They aren't perfect and assume that stress and discomfort when public speaking are related to lower language capabilities (which may or may not be true).
Relationship
It's not explicitly stated, but I'd like to have a relation Person ↔ Language, with this design probably with an intermediate model LanguageQualification.
If we, however, dropped weights (which I am in favor of personally), then:

we lose intermediate model (huge win for programming efforts)
and we lose UI for setting weights,
and we lose N more fields users would have to fill out.

Default languages

I'm against getting languages from the internet
I'm trying to get rid of data-import migrations (see #626)

Instead of using IANA registry (dd3c641f01f52ba76e5aa681e8c7a7d7ef7f5458 " workshops/migrations/0068: Add Language-import data migration"), I'd just copy Django's list:
https://github.com/django/django/blob/59f861fcb49a43acfbf2f75e683a7cb971600f52/django/conf/global_settings.py (it's shorter!)
Event's languages

I haven't stored event languages, because I expect there's not much
use for this information after the event is setup. The EventRequest
form now uses a many-to-many language table, so we don't have to parse
user-entered languages. And once the EventRequest is translated into
an Event, the language bits have already taken care of themselves
(unless we want to list them on out upcoming events feed?)

This is not true. When a new EventRequest comes in, admins turn it into an unfinished Event. It stays that way until all details are sorted out (location, dates, instructors, money, etc.).
So I want to store event's languages with the event, and also populate EventForm's languages when EventRequest is being accepted.

On Sun, Jan 03, 2016 at 01:25:46PM -0800, Piotr Banaszkiewicz wrote:

For example we could use these values:

1 - I'm comfortable being a lead instructor for an event
0.75 - I'm comfortable being a helper during the event
0.5 - I'd rather not speak to the audience

Sounds good to me, in which case we may want to change the …__gte=0.5
to …_gt=0.5.  But I otherwise I think this is a UI/docs decision that
doesn't need backend changes.

They aren't perfect…

I think this is going to be true of any system ;).  I wouldn't worry
about finessing it too much.

If we, however, dropped weights (which I am in favor of personally),
then:

we lose intermediate model (huge win for programming efforts)

But programming efforts are completed now, no?  Excepting the
weight-setting UI, which we obviously can't have without a
weight-storage field.

and we lose UI for setting weights,

Agreed.

and we lose N more fields users would have to fill out.

?  Setting weights (assuming we had a UI) sounds like one more
optional radio/select per language you speak.  I'm not sure how many
languages our instructors speak, but I'd be surprised if anyone speaks
so many that setting weights would laborious.  And if the instructor
is comfortable enough with an effective default weight of one (what
you get if there's nowhere to store weights), then they don't need to
bother configuring their weights.  So I see weights causing a marginal
increase in user effort, but only if the user opts into sharing that
level of detail.

Default languages

I'm against getting languages from the internet

I don't see a way to avoid that ;), and in this case the IANA URL is
the canonical source for RFC 5646 language tags.

I'm trying to get rid of data-import migrations (see #626)

I can change my data migration to a fixture-creating script.  Then we
can re-run the script if/when we want to update the fixture to pull in
IANA changes.

Instead of using IANA registry …, I'd just copy Django's list …
(it's shorter!)

Shorter because many languages don't have out-of-the-box Django
translations 1.  The IANA approach scales well: you need a langauge
we don't have listed?  Just bump up the subtag size limit to three
characters.  You need a language that the IANA doesn't list?  Register
it with IANA 2, and we'll pick it up once it's accepted.  I'm not
sure how the Django approach scales.  If you need a new language,
write a Django translation?  Or we just add it by hand?

So I want to store event's languages with the event, and also
populate EventForm's languages when EventRequest is being accepted.

Ok.  I feel like having Event and a separate EventRequest (and Person
and a separate ProfileUpdateRequest is causing more duplication than I
want to code/test ;).  But I guess that gets back to #70, where I'd
just let event-requestors create an Event or edit their Person
directly, with permissions controlling sensitive fields (e.g. need an
admin to finish the Event off for publication, or to badge a person).
I'll look at writing an Event CRUD API to mock something like that up.
So my proposed plan for this PR:

Convert the IANA-pulling data migration to a fixture-generator.
Commit a fixture generated by that script.
Add Event.language (so multi-language events would get multiple
Event entries?) which is initialized from EventRequest.langauge for
EventRequest-based Events.

Do I need to do more around ProfileUpdateRequest?  I'm still lost on
that front.

@wking:

Shorter because many languages don't have out-of-the-box Django
translations [1].  The IANA approach scales well: you need a langauge
we don't have listed?  Just bump up the subtag size limit to three
characters.  You need a language that the IANA doesn't list?  Register
it with IANA [2], and we'll pick it up once it's accepted.  I'm not
sure how the Django approach scales.  If you need a new language,
write a Django translation?  Or we just add it by hand?

But do we need to have 200 languages in the database when our users only claim to speak less than 30 of them? Or will we ever have a workshop in Interlingua, Esperanto or Ido?
Also related issue I've just found: IANA registry's logic is more complex than your script's logic. For example: some entries have multiple descriptions (like cu) and some are deprecated (see ik, id).

So my proposed plan for this PR:

Convert the IANA-pulling data migration to a fixture-generator.
Commit a fixture generated by that script.
Add Event.language (so multi-language events would get multiple Event entries?) which is initialized from EventRequest.langauge for EventRequest-based Events.

We're not switching to fixtures yet, so for now can you please:

Write a management command (not script!) for getting data from IANA
Hardcode that data in a data migration
Add Event.languages

Do I need to do more around ProfileUpdateRequest? I'm still lost on that front.

Add languages M2M field - that's the easy part
Figure out what should we do about a widget for the form - that's the hard part.

But before that, I want to hear second opinion on including weights into the system. If we end up using them, then we'd have to somehow come up with a proper, easy to use widget (and from what I remember this is not a trivial task). Probably some JS would be needed as well.
However, if we dropped the weights, our forms would be significantly easier to do.

@gvwilson should we include weights in relations between person and their languages to indicate how comfortable a person is speaking in these languages?
For example I'd give 1 to Polish but something around 0.8 to English and 0.1 to German.
We can use hardcoded values to help users decide what level of fluency they have:

1 - I'm comfortable being a lead instructor for an event
0.75 - I'm comfortable being a helper during the event
0.5 - I'd rather not speak to the audience

On Sun, Jan 03, 2016 at 03:21:59PM -0800, Piotr Banaszkiewicz wrote:

But do we need to have 200 languages in the database when our users
only claim to speak less than 30 of them? Or will we ever have a
workshop in Interlingua, Esperanto or Ido?

Probably not ;).  But I'd rather not be in the business of deciding
which subset of IANA languages we need.  Autocomplete should cover us
here.

Also related issue I've just found: IANA registry's logic is more
complex than your script's logic. For example: some entries have
multiple descriptions (like cu)…

Good point.  In this case, email.parser.Parser collects all the
entries, but getitem returns a single entry 1.  That should be
fine for us.  Supporting multiple descriptions doesn't seem to be
worth the trouble, but if we want to concat (with semicolon
separators?) and push into a single field for auto-completion, we
could do that with get_all 2.

… and some are deprecated (see ik, id).

I think this is also more trouble than it's worth for the level of
data cleanliness I expect us to have.  Ideally, somebody else would
have written a conformant RFC 5646 Python package… and maybe they have
3?  We could use that instead of my current Language table, with a
character field containing the full tag instead of a primary-key
pointing into Language.

Write a management command (not script!) for getting data from IANA
Hardcode that data in a data migration

These could both be replaced by a language-tag 3 dependency, if we
want to go that way.

Add Event.languages

Is the plural intended?  If so, we probably want to update
EventRequest as well, since it's currently using a singular field
name.

Do I need to do more around ProfileUpdateRequest? I'm still lost on that front.

Add languages M2M field - that's the easy part
Figure out what should we do about a widget for the form - that's the hard part.

The M2M autocomplete widgets I'm using already (e.g. in the
person-detail form) seem straigtforward enough.  If we can relax the
login requirement for lookups 4 or use a UI based on 5 (with
client-side autocompletion), it shouldn't be too bad to do something
similar with language-tags.

But before that, I want to hear second opinion on including weights
into the system. If we end up using them, then we'd have to somehow
come up with a proper, easy to use widget (and from what I remember
this is not a trivial task). Probably some JS would be needed as
well.

Agreed.  I'm personally still a fan of an API (in Django, or whatever)
and separate JavaScript UI (in a JavaScript framework like Angular),
because it makes this sort of problem trivial 6.

@wking:

Ideally, somebody else would have written a conformant RFC 5646 Python package… and maybe they have [3]?

+10 to using a 3rd party Python provider, however you may have troubles listing all included language tags.

Supporting multiple descriptions doesn't seem to be worth the trouble, but if we want to concat (with semicolon separators?)

language-tags returns a list of descriptions:
>>> print(tags.language('nl').description);
['Dutch', 'Flemish']

I'd concat them.

Add Event.languages

Is the plural intended?  If so, we probably want to update
EventRequest as well, since it's currently using a singular field
name.

No, sorry, that was my mistake. Event can have only one language.

The M2M autocomplete widgets I'm using already (e.g. in the
person-detail form) seem straigtforward enough.  If we can relax the
login requirement for lookups [4] or use a UI based on [5] (with
client-side autocompletion), it shouldn't be too bad to do something
similar with language-tags.

Okay, I checked and indeed M2M autocomplete widget works excellent (it just needs some better styling).
And we can have relaxed permission requirements for language tag lookup.
Anyway, let's see if language-tags works okay for us and what does @gvwilson say about weights.

@wking since Greg is focusing on other projects (and others agreed share my opinion) I'll make a design decision here: let's wipe out the LanguageQualification model (and therefore weights). It will save us from using an intermediary model for M2M relation with Person, and we won't have to think about UI.

On Mon, Jan 04, 2016 at 02:55:53PM -0800, Piotr Banaszkiewicz wrote:

… let's wipe out the LanguageQualification model (and therefore weights).

Ok, I'll post a not here once I've rerolled to drop them and converted
Language over to language-tags.

@wking any progress here?

On Mon, Feb 01, 2016 at 09:20:59AM -0800, Piotr Banaszkiewicz wrote:

@wking any progress here?

No, I haven't had time to implement the planned reroll.  Maybe this
week?

I agree to most of the suggestions and the code that exist here.

let's wipe out the LanguageQualification model (and therefore weights). It will save us from using an intermediary model for M2M relation with Person, and we won't have to think about UI.

+1 for that. Lets make it easier for the instructors to rate their ability to deliver a workshop in a particular language.

Ideally, somebody else would have written a conformant RFC 5646 Python package… and maybe they have [3]?

How often is official registry updated? I don't think the the list of languages with two-lettered tags (the popular languages we are interested in) receive additions/modification regularly that we should use a separate Python package to track changes.
The migration that is included seems the right solution to me.
Opinions?
I will be pushing a PR by tonight which will probably borrow a lot of code from this PR.

I feel very strongly that we should get rid of weights and just have
"person X can teach in language Y".  After all, we don't ask people to
rate their proficiency in lessons - we trust them when they say "I can
do this".

We're ditching this one in favor of #798.


Get/Add-Content throws improved error when targeting a container

PR Summary
Fix #7770
Fix #2686
Enhancing the error message as per #7770 and #2686
PR Checklist

[x] PR has a meaningful title

Use the present tense and imperative mood when describing your changes

[x] Summarized changes
[x] Change is not breaking
[x] Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
[x] This PR is ready to merge and is not Work in Progress.

If the PR is work in progress, please add the prefix WIP: to the beginning of the title and remove the prefix when the PR is ready.

User-facing changes

[x] Not Applicable
OR
[ ] User-facing Documentation needed

[x] Issue filed - Issue link: #7770

Testing - New and feature

[ ] Not Applicable or can only be tested interactively
OR
[x] Make sure you've added a new test if existing tests do not effectively test the code changed

[ ] Add [feature] if the change is significant or affects feature tests

I think more rules are added to CodeFactor, issues are in untouched lines.
Even CI failure is something different.

The package validation on NJsonSchema is a known issue not related to this PR

Reopen the PR to restart all CIs.

@kvprasoon can you rebase off master to pick up test fixes?

@kvprasoon I rebased for you.
Further please create new branch in your fork for every new change (don't put commits in forked master).

@iSazonov Sure, will do it, learning OSS collaboration via each PR... thanks


Add appropriate enum validators

There are currently no validators for enum types. This is because they of the discrepancy of comparability between int and string. However, some operators are still valid and therefore should be implmeneted:

BindValidatedEnumContains/NotContains
BindValidatedEnumIsValid - CUSTOM, validates against teh enuminfo (might not be possible, if we can't define a generic type on the method)

Need to clear up the code generation. It was a bit hap-hazard leading to unexpected results


How the max theory peak can be achieve of MIOpenGEMM?

MIOpenGEMM cannot achieve peak theoretical throughput. Consider Tensile: https://github.com/ROCmSoftwarePlatform/Tensile (which is partly written in assembly) to get within a few percent of the theoretical peak.


[Feature] Send a message to the website when user switches account

Feature description
Websites that use window.webln or window.nostr currently do not know when a user switches to a different account inside the extension, and show information from the previous account. It would be useful if a website could listen to a message so it can stay in sync.
Describe the solution
Post a message to the window when the Alby account changes
Describe alternatives
No response
Additional context
No response
Are you working on this?
None

closed in favour of #2504


RethinkDB 2.1.0 Install Failure Darwin

I am trying to install the newest RethinkDB released yesterday, ver 2.1.0. Only thing I have changed from the default.nix file is the src:
      src = pkgs.fetchurl {
        url = "http://download.rethinkdb.com/dist/rethinkdb-2.1.0-1.tgz";
        sha256 = "0i1bb6k2bsl15f6a9kq9708hh5n4b6rcg8dbdxq4lh07a4f4c9sx";
      };

Here is the tail end of the build that errors:
68 warnings generated.
13 warnings generated.
    [398/398] LD build/release_clang_jemalloc/rethinkdb
/nix/store/70k3xljbry6952xbwxhcflfkrh2ch0f9-bash-4.3-p39/bin/bash: objdump: command not found
Failed to link in jemalloc.
src/build.mk:355: recipe for target 'build/release_clang_jemalloc/rethinkdb' failed
make[1]: *** [build/release_clang_jemalloc/rethinkdb] Error 1
make[1]: *** Deleting file 'build/release_clang_jemalloc/rethinkdb'
Makefile:52: recipe for target 'make' failed
make: *** [make] Error 2
builder for ‘/nix/store/x7s39mdah1ak8byp2xagz5d9rzqmaz6w-rethinkdb-2.1.0-1.drv’ failed with exit code 2
error: build of ‘/nix/store/x7s39mdah1ak8byp2xagz5d9rzqmaz6w-rethinkdb-2.1.0-1.drv’ failed

So I tried adding binutils-raw to the buildInputs, but then I get this error:
68 warnings generated.
    [398/398] LD build/release_clang_jemalloc/rethinkdb
dyld: lazy symbol binding failed: Symbol not found: _libintl_dgettext
  Referenced from: /nix/store/3mx4zr2q68v7jhj1x15vfjngfxhk0rl7-binutils-2.23.1/lib/libbfd-2.23.1.dylib
  Expected in: flat namespace

dyld: Symbol not found: _libintl_dgettext
  Referenced from: /nix/store/3mx4zr2q68v7jhj1x15vfjngfxhk0rl7-binutils-2.23.1/lib/libbfd-2.23.1.dylib
  Expected in: flat namespace

Failed to link in jemalloc.
src/build.mk:355: recipe for target 'build/release_clang_jemalloc/rethinkdb' failed
make[1]: *** [build/release_clang_jemalloc/rethinkdb] Error 1
make[1]: *** Deleting file 'build/release_clang_jemalloc/rethinkdb'
Makefile:52: recipe for target 'make' failed
make: *** [make] Error 2
builder for ‘/nix/store/g7hrsdn0gz9f48s6jyj6lcjwnng2g88b-rethinkdb-2.1.0-1.drv’ failed with exit code 2
error: build of ‘/nix/store/g7hrsdn0gz9f48s6jyj6lcjwnng2g88b-rethinkdb-2.1.0-1.drv’ failed

Not sure how to proceed. cc @wkennington @thoughtpolice @domenkozar

@wkennington Turns out this is related to this commit https://github.com/NixOS/nixpkgs/commit/5b0a676d5b29ed1586a18cb35ed508ec48600a8f#diff-47969f195e59d7d15bf8e0b587a7f8e3. Jemalloc does not work with Darwin so --with-jemalloc is the main problem but really on Darwin the configureFlags can be blank. I will try and submit a PR for that change if no one else gets to it first.

@codedmart: any news here?

This was fixed (without updating this issue) but there is now another issue on Darwin:
lders/g4/76q469c16sg48h2xcl17lm900000gn/T/nix-build-rethinkdb-2.2.4.drv-0/rethinkdb-2.2.4/build/external/v8_<IP_ADDRESS>/build/out/x64.release/obj.target/v8_base/src/ic/x64/ic-compiler-x64.o /private/var/folders/g4/76q469c16sg48h2xcl17lm900000gn/T/nix-build-rethinkdb-2.2.4.drv-0/rethinkdb-2.2.4/build/external/v8_<IP_ADDRESS>/build/out/x64.release/obj.target/v8_base/src/ic/x64/stub-cache-x64.o
Usage: /nix/store/n9s1gn8hrxgkz26kz5r5813pyw3c21ri-libtool-2.4.6/bin/libtool [OPTION]... [MODE-ARG]...
Try 'libtool --help' for more information.
libtool:   error: unrecognised option: '-static'
tools/gyp/v8_base.target.x64.release.mk:479: recipe for target '/private/var/folders/g4/76q469c16sg48h2xcl17lm900000gn/T/nix-build-rethinkdb-2.2.4.drv-0/rethinkdb-2.2.4/build/external/v8_<IP_ADDRESS>/build/out/x64.release/libv8_base.a' failed
make[3]: *** [/private/var/folders/g4/76q469c16sg48h2xcl17lm900000gn/T/nix-build-rethinkdb-2.2.4.drv-0/rethinkdb-2.2.4/build/external/v8_<IP_ADDRESS>/build/out/x64.release/libv8_base.a] Error 1
make[3]: Leaving directory '/private/var/folders/g4/76q469c16sg48h2xcl17lm900000gn/T/nix-build-rethinkdb-2.2.4.drv-0/rethinkdb-2.2.4/build/external/v8_<IP_ADDRESS>/build/out'
Makefile:286: recipe for target 'x64.release' failed
make[2]: *** [x64.release] Error 2
make[2]: Leaving directory '/private/var/folders/g4/76q469c16sg48h2xcl17lm900000gn/T/nix-build-rethinkdb-2.2.4.drv-0/rethinkdb-2.2.4/build/external/v8_<IP_ADDRESS>/build'

It cannot build v8, which is an extra dependency it builds itself (that is, if you change the meta.platforms to include Darwin, because it was excluded).
@thoughtpolice, @bluescreen303, @pikajude, any ideas? Do you know why it is building v8 in the first place, is there a way to use the one from nixpkgs?

@wmertens What’s the status? I suggest you should open a new issue if it’s still broken.


Update docs with missing operations

The following operations:
Logical_ANDI
Logical_ORI
Logical_XORI
Logical_NOTI
aren't documented in: https://tenstorrent.github.io/tt-metal/latest/ttnn/index.html

Closing since these ops have been renamed as follows and the documentation is present in documentation:
Logical_ANDI --> logical_and_
Logical_ORI --> logical_or_
Logical_XORI --> logical_xor_
Logical_NOTI --> logical_not_


"Cannot read property 'Pointer' of undefined"

Hi,
we are using the cli tool with qx compile --watch for a new project. When reloading the application we often get this error:
TypeError: Cannot read property 'Pointer' of undefined
    at wrapper._fireEvent (http://localhost:3000/source/transpiled/qx/event/handler/Pointer.js:176:29)
    at wrapper._onPointerEvent (http://localhost:3000/source/transpiled/qx/event/handler/Pointer.js:202:14)
    at HTMLDocument.<anonymous> (http://localhost:3000/source/transpiled/qx/lang/Function.js:294:25)

The line in question is:
qx.event.type.dom.Pointer.normalize(domEvent);

I estimate it happens every other reload, but sometimes it happens multiple reloads in a row and then we can reload it for like five times and don't get an error. So it is really seems to appear random.
Is someone having this issue too?

Please can you try editing qx/event/handler/Pointer.js and changing the class comment to include @require(qx.event.type.dom.Pointer), eg the class should start like this:
/**
 * Unified pointer event handler.
 * @require(qx.event.dispatch.DomBubbling)
 * @require(qx.event.type.Pointer) // load-time dependency for early native events
 * @require(qx.event.type.dom.Pointer)
 */
qx.Class.define("qx.event.handler.Pointer",
{
  extend : qx.event.handler.PointerCore,

Hi, thanks for the fast reply! I added that require line and will observe if the error appears again.

That actually solved it, thank you very much!
How do we deal with this? Shall I prepare a PR for the main qooxdoo repo?

no leave it with me, I'll put it in with a PR Im preparing for a couple of similar issues.  Glad it's fixed it for you!


Mark TestStartDoesNotBlock flaky

What does this PR do?
Mark TestStartDoesNotBlock flaky
Motivation
#incident-32001
Describe how to test/QA your changes
Possible Drawbacks / Trade-offs
Additional Notes

Force merging


Add time_units to the y-axis labels

Most tree sequences now have a time_units attribute. It should be really easy to add this in brackets to any axis labels that say "Time xxx", e.g. "Time of parent node" would become "Time of parent node (generations)".

Dupe of #83


Uploaded logo SVGs

Uploaded new logo svgs under public/logo
Changed logo in navbar, feel free to revert or use anywhere else

Pull request approved. Thank you Marina :)


Rescue old/expired hit_ids

callback_models at this point in time is nil, It's defined when it's in the assignments loop: https://github.com/throttleup/turkee/blob/use-aws-sdk-mturk/lib/models/turkee_task.rb#L36 but it wouldn't reach there at all.
Does it make sense to update the turk directly?
            begin
              assignments = turk_api.assignments_for_hit(turk.hit_id)
            rescue Aws::MTurk::Errors::RequestError => e
              if e.message =~ /Hit .* does not exist.*/
                turk.update(complete: true)
              else
                raise e
              end
            end


World.setGameRule missing.

World has defined methods for getting a single gamerule, getting a map of gamerules, but has no setting to set gamerules.
I was hoping to create a plugin that would auto-setup worlds for UltraHardcore which involves setting gamerule settings.
At the moment the only alternative is to run commands with a given input.

You can use the WorldProperties world.getProperties().setGameRule(rule, value)


Major change in a route not listed when impact filter is on

https://knooppuntnet.nl/en/analysis/route/13139760/changes
The topology of the route changed, one node was replaced, and by side effect, the name change.
Impact off: last version v6 is listed.

Impact on: only the v1 is listed, but v6 is filtered 😢

The analysis logic can set two flags for a change:

happy
investigate

The happy flag is intended to indicate that the mapping quality has improved (for example: issues have been resolved, new nodes were added, etc.).
The investigate flag is intended to indicate that mapping quality may have become worse (for example: there are new facts, routes or nodes have been removed, etc.).
The impact filter looks for changes where at least one of these flags is set.
A route name change does not result in either the happy flag or the investigate flag to be set. The route quality is ok before the change, and is still ok after the change.  So although a route name change may be experienced as a significant change, it is probably not really necessary to draw other mappers attention to it by setting the happy or investigate flag.

In this case the geometry and topology got significant change: the beginning node was replaced by a completely different node (not an OSM node with the same name/ref near the previous node).


Collection of issues with byref analysis

There are a number of holes in checking the use of byref values.  I'm collecting these into one issue since if we fix any one of them we may as well go through and address them all.
Addressed:

[x] Compiler allows invalid uses of byref in interface methods https://github.com/Microsoft/visualfsharp/issues/127 (Related to #5459)
[x] Tupling causes problems in argument poitions https://github.com/Microsoft/visualfsharp/issues/3362 (Doesn't fail in Release, but dies in Debug)
[x] Base calls and byrefs: https://github.com/Microsoft/visualfsharp/issues/130
[x] Inlined tupled arguments and byrefs: https://github.com/Microsoft/visualfsharp/issues/820 and https://github.com/Microsoft/visualfsharp/issues/558
[x] Struct field returns https://github.com/Microsoft/visualfsharp/issues/1401
[x] Some uses of currying and byrefs: https://github.com/Microsoft/visualfsharp/issues/819

I'll close the individual bug entries and reactivate if we don't fix them all in one go.

Would it make sense to tackle (parts of) this planned User Voice item at the same time, since it will also affect the safety rules by loosening them in some cases?

Regarding my last comment, byref locals and returns have now been merged into the Roslyn tree - see here.

This is worth going through.

I've updated the checklist.
These still have issues:

Compiler allows invalid uses of byref in interface methods #127 (Related to https://github.com/Microsoft/visualfsharp/issues/5459)
Tupling causes problems in argument poitions #3362

The tupling one is a bit more concerning. We need to always error if we can't guarantee the compiler will not make a tuple, which I don't think we are doing.

We have resolved these in 15.9 now.


set numeric as true but sometimes get random string no any number

function getRandomString(length, options) {
const params = {
numeric: get(options, 'numeric', true),
letters: get(options, 'letters', true),
special: get(options, 'special', true),
exclude: get(options, 'exclude', []),
};
return randomString({
length,
...params,
});
}
......
Can see the function detail as above, I have only set the options as non-mandatory params, if no options passed here, all options will be true, but what I got the string have no at least one number...not always like this, but sometimes can not works.

Can you please provide more details (full formatted code example) on your question? ty


AppImage doesn't create folder in home (no permissions to access directory)

System: Fedora 37
binaries: Firo-Client-3.0.1.AppImage
If i leave the default path for "Data Directory" (.firo) in the "Select network and blockchain" screen, i get this error:
You don't have permissions to access the directory you selected. Please try again

Changing to /tmp or other paths seem to work

Actually, this seems to be unrelated to firo-client. Looks like that the .firo folder was created as root and the appimage cannot rightfully access it. Not sure how that happened, because i didn't create the folder manually, so it was probably generated when trying to run firo binaries from the main repo. Closing the issue here in the meantime, as it's not an issue of this repo.


Automate Releases

This issue is open to track our github actions. When deploying a new release, we want to be able to upload our package to npm.
cc: @costrouc

This works, closing.


Added more extensive test for adjacency lists constructed via vectors of vertices

… of vertices

[Current coverage][cc-pull] is 85.64%

Merging [#223][cc-pull] into [master][cc-base-branch] will not change coverage

@@             master       #223   diff @@
==========================================
  Files            24         24
  Lines          1435       1435
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1229       1229
  Misses          206        206
  Partials          0          0

Powered by Codecov. Last updated by [63be068...bf40526][cc-compare]
[cc-base-branch]: https://codecov.io/gh/JuliaLang/Graphs.jl/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/JuliaLang/Graphs.jl/compare/63be068a45cd74d9a311cbb32b9e9ae670557130...bf40526815c0c65c340d1596a1b90805665397d5
[cc-pull]: https://codecov.io/gh/JuliaLang/Graphs.jl/pull/223?src=pr


Support of codegen for exponentiation in the GCE

Currently GCE supports codegen of exponentiation in transition constraints. The task is to implement codegen of exponentiation for boundary constraints.

Superseded by #333.


Masternodes should be allowed to annonce ipv4 addresses only from now

Controversial, and my heart is bleeding. But IHMO a quick win for our "heterogenous network connectivity issue" until we have a better solution.
utACK

So about 140 Masternodes need to change providers?
Anyway...I guess this is in preparation to directly connect to nodes for IP verification?
If so, I'd rather have 2 individual Masternodes lists (IPv4 and IPv6. We ignore TOR for now because of lack of nodes), and IPv4 nodes verify IPv4 nodes and IPv6 nodes verify IPv6 nodes.
Each node knows in which network it runs, so if your implementation works for IPv4 it'll work for IPv6 as well.
There's no reason to exclude IPv6.

The reason to exclude IPv6 as masternode is because IPv4 only nodes don't have access to these IPv6 masternodes so IPv4 should be mandatory requirement of masternode service, IMO.

IPv4 only nodes don't have access to these IPv6 masternodes

I know, that's why I wrote

IPv4 nodes verify IPv4 nodes and IPv6 nodes verify IPv6 nodes

Though both is nice, I would think IPv6 only should not be excluded...

There is no reason to have ipv6 just to have ipv6, there must be some usefulness of it. The reason to exclude all non-ipv4 is exactly because they do not provide service they are paid for - connectivity and mixing. It's still very rare this days for end users to have ipv6 connectivity, most of them still rely on ipv4 and that's why it's should be a requirement for masternodes, not an option. We can revert this once ipv6 became the norm but that's way too far from being the reality right now imo.

I can accept this, but I do not agree.
It's the same chicken-egg argument ISPs tell you why they don't offer more IPv6 services, because there are not much IPv6 users who would be able to use it.
Removing IPv6 support would also remove the already existing IPv6 users. Welcome back in 2010.
And somewhere suddenly everyone cries because we're running out of IPv4 addresses.
Build the (Dash-) infrastructure and the users will follow.

I'm not happy to implement this too but sadly that

Build the (Dash-) infrastructure and the users will follow.

is not the case. I have full ipv6 connectivity inside my local home network but my ISP doesn't provide it and I can't force them to do so. I bet most of users suffer from the same issue. I think ISPs will only upgrade when they themselves run out of IPs or they have no other choice but to upgrade all their hardware one day because the old one simply doesn't work anymore. I doubt they will invest in new hardware just because ipv6 is nice or because some user think it's nice - they already made huge capital investments building their current infrastructure so they are not going to throw that away that fast imo. We, on the other hand, need network connectivity and masternode layer consistency right now, we can't wait another 5-10 years for them to upgrade and just let our users/infrastructure suffer from this issue all these years.
Also keep in mind that this does NOT remove ipv6 support from node networking, (master)nodes still can advertise itself on network layer as ipv6 nodes, same as before, it's only our custom masternode announce message that requires ipv4 now.

We, on the other hand, need network connectivity and masternode layer consistency right now

I 100% agree.
But your argument against IPv6 was NOT consistency (as I wrote above, when your code works with IPv4 it'll also work with IPv6. You only have to split the check into a IPV4 check executed by IPv4 nodes and a IPV6 check executed by IPv6 nodes. The result is, independently of the IP network, broadcasted to the Dash network ).
Your argument above was that right now IPv6 nodes get paid even they don't provide any services.
And that's simply wrong. They provide exactly the same services as IPv4 nodes, only on a smaller scale (proportional to the percentage of IPv6 nodes/peers in our network).
I would like to know why you think your code for IPv4 won't work for IPv6.

I would like to know why you think your code for IPv4 won't work for IPv6.

I did not see anyone arguing that the code is not working for IPv6 - it is working.
The problem is that we do not enforce dualstack at the moment, which leads to two distinct networks at the moment: IPv4 only nodes/wallets can't connect to IPv6 only masternodes and vice versa.
For me there are three solutions:

enforce dualstack installations, so IF you have a IPv6 masternode you ALSO need a IPv4 entry (common in real life installations, you do not get a IPv6 connection without at least a 6to4 gateway at home)
Prohibit IPv6 masternodes: This is the solution we are discussing here
Select TWO masternodes in each round for mixing: One for IPv6, one for IPv4. Rewards are not affected.

From technical point of view i favor 1) or 3), while 2) is the quick win - still not happy about degrading to IPv4, it is so 2000s

@schinzelh , I think current solution is not totally 2).
This solution just require IPv4 announcement , If you have a IPv6 masternode you need a IPv4 entry for announcement (as mentioned in your 1) ).

@thelazier You are right, actually this PR is enforcing any masternode (IPv4 AND IPv6) to have a IPv4 entry in masternode list, so this is in fact option 1) here :)

I like 3) :-)
As for 1), when you're running 2 Masternodes with 2 IPv6 IPs on one host, do you mean one IPv4 address for all Masternodes (which I think would be difficult to implement), or one IPv4 address for each IPv6 address (which would make the dual-stack unnecessary because we would have a IPv4 IP per node anyway)?

@crowning- Obviously it must be one IPv4 for each IPv6, otherwise: What masternode will bind to the IPv4 address? Node1 or Node2? Given Node1: How do we reach Node2 via IPv4?

Maybe we should drop the port 9999 requirement, too?

Maybe we should drop the port 9999 requirement, too?

This would make the most from the existing IPv4s.
Most services (e.g. HTTP) can be configured to use (almost) any available port, you only have to make sure that the clients KNOW (via the port in the URL) that they have to use a different port to connect. And the clients usually have all ports open for outgoing connections, only the Masternodes itself would have to open a different port for incoming connections.
The port is already part of the Masternode broadcast, so we would have covered this already.

Ok, thanks everyone for participating 👍 (I see we drift to port discussion which is kind of offtopic here ;) )
As I said we can always revert this later if ipv6 become more popular/available option for the majority of end users or maybe we could implement smth more sophisticated but for now I don't really see another solution to quickly fix our infrastructure so I'm going to merge this. Let's see how this plays out on testnet.

quickly fix our infrastructure

That's actually my main concern. Our network's working fine with this for 2 years now, so it certainly would for another month or so.
I vote for fixing it right instead of quick, even when my vote is ignored.

Our network's working fine with this for 2 years now

It has wrong incentives right now imo: ipv6 nodes do not provide service for majority of end users and yet they are paid just like ipv4. I see it more as a "fork" in the infrastructure rather than a valuable feature. ipv6 adoption is simply laughable unless you live in USA, Germany or Greece https://www.google.com/intl/en/ipv6/statistics.html#tab=per-country-ipv6-adoption&tab=per-country-ipv6-adoption

even when my vote is ignored

It's not. I hear you concerns but I see no other solutions so far. If you have one - you know how/where to submit your code :)

http://ipv6-test.com/stats/country/US

http://ipv6-test.com/speedtest/


BuddyBuildSDK Integration

@dennispi, here's the BuddyBuildSDK. You'll get an email that lets you install and try out the build

Make this 2!

Too green


NO-TICKET: Enable the "sync over upgrades" IT test.

master branch is now ready for this test. I have also disabled the regular sync_test as the upgrade version tests the superset of the functionality.

bors merge

bors merge

bors r-

bors r+