Docs: generated ToC links is not working propertly on website

A small problem with doc generation: the links from table of contents section works fine in github pages, but didn't work on project website.
Example:

github doc page (ToC links is OK): https://github.com/keystonejs/keystone-5/blob/master/docs/discussions/mutation-lifecycle.md
same page on keystone's 5 website (ToC links is broken): https://v5.keystonejs.com/discussions/mutation-lifecycle

(Looks like a problem with a docs generation script)

Thank you 🙏
Looks like titles with some non-alpha-num characters are broken:
/discussions/mutation-lifecycle#1-check-list-access-create-update-delete - CORRECT
/discussions/mutation-lifecycle#1-check-list-access-createupdatedelete - WRONG (TOC link)

The titles themselves have the correct links, but the TOC doesn't.

I think this is fixed now? Can close this issue 👍

Checked out doc links, they all are fixed now. Closing issue.

This may be still be an issue. Links work on github but not website docs. Is there a specific way we should link to other docs pages?
Eg. https://v5.keystonejs.com/guides/add-lists

I just noticed the same issue today with links in https://v5.keystonejs.com/discussions/schema.

link to createLists() API:

- https://v5.keystonejs.com/discussions/api/create-list
+ https://v5.keystonejs.com/api/create-list

Link to quickstart

- https://v5.keystonejs.com/discussions/quick-start
+ https://v5.keystonejs.com/quick-start


integrate with CCA and pyscenic

Dear,
Is it possible to integrate scanpy with CCA and pyscenic?
CCA (canonical correlation analysis to alignment different datasets and batch effect correction):
https://satijalab.org/seurat/immune_alignment.html
pyscenic (single-cell regulatory network inference and clustering):
https://github.com/aertslab/pySCENIC

CCA does not have code in python, which will make it difficult to integrate, pySCENIC is probably easier but I would rather ask the developers.
@falexwolf We should consider a way to facilitate scanpy 'plugins'. A quick search shows me that this could be possible: https://packaging.python.org/guides/creating-and-discovering-plugins/ but honestly I don't know how it works. Nevertheless, given the number of tools that continue to appear we should consider a scheme that facilitate how developers can take advantage of scanpy preprocessing, storing, analysis and visualization tools.

I like Seurat's CCA. A pull request using rpy2 similar to the R wrapper of Haghverdi et al.'s version of MNN would be welcome.
Regarding "plugins": I guess a lot of Scanpy's functionality already consists in "plugins":

https://scanpy.readthedocs.io/en/latest/api/scanpy.api.pp.mnn_correct.html
https://scanpy.readthedocs.io/en/latest/api/scanpy.api.pp.dca.html
https://scanpy.readthedocs.io/en/latest/api/scanpy.api.pp.magic.html
https://scanpy.readthedocs.io/en/latest/api/scanpy.api.tl.phate.html
https://scanpy.readthedocs.io/en/latest/api/scanpy.api.tl.louvain.html
and a lot more are on the way, as far as I know. I guess the strategy of having an optional dependency of the respective and a small wrapper in Scanpy is a scalable strategy. Do you think we need to do more?

Sklearn has its implementation of CCA but that would allow the alignment of two samples only. Recently a multi sample approach was implemented in pyrcca library for which there is a biorXiv paper.

@falexwolf I think it is worth to have a scanpy.plugin or 'scanpy.extensionor something shorter likescanpy.pg orscanpy.ext` that aggregates all plugins. First, this clarifies for the user that the tool he/she is using is not directly developed by scanpy. Second, this allows plugins to be installed separately without having to update scanpy's code. The idea is that scanpy will be able to discover any plugins installed. On the side of developers, this could facilitate integration with scanpy. We can get inspired by flask extensions: http://flask.pocoo.org/docs/1.0/extensiondev/

Sklearn has its implementation of CCA but that would allow the alignment of two samples only. Recently a multi sample approach was implemented in pyrcca library for which there is a biorXiv paper.

Is it suitable for single cell data ?

@fidelram Yes, makes sense. Let's see whether we manage to organize it this way. There will be a few plugins coming soon and I'll talk with the one doing it about this.
@wangjiawen2013 The Seurat developers did a bit more than simply fitting a standard CCA. So I'd assume that it'd be some work to wrap sklearn's CCA or pyrcca so that it performs similar to Seurat's CCA on single cell data...

Given that UMAP can be used for manifold learning, shouldn’t be possibile to align experiments using UMAP? Who wants to join me in this evaluation?

UMAP won't do any correction of batch effects for you, like CCA (it looks at the basis that leads to the greatest overlap between the batches, assuming that this captures the common biological variation and projects out everything else, assuming it's nuisance/technical batch effects). Similar for all other "alignment tools": you throw away some information in order to align. When you map a new dataset into an existing dataset using UMAP, this will do an exact mapping. If you have pronounced batch effects, the second dataset will cluster as a whole far away from the first.
So, I don't think that there will be much to gain.
Why not give BBKNN (https://github.com/Teichlab/bbknn) a try? It integrates nicely with Scanpy.

@falexwolf you are right, although I have to say that in my (surely limited) experience, batch effect doesn't seem to play a major role when using UMAP. I don't know BBKNN, I will give it a look for sure. I guess this evaluation may be a side project (for volunteers) which may eventually merge into scanpy.

Let’s continue the discussion about a general plugin mechanism in #271, and this thread for CCA specifically.

Just curious about the case of 'batch effect', it looks like to me library construction protocol/chemicals is main source of batch effects.
However, if I use same protocols, sequencing platform and slight difference of sequencing depth for some sample, but in different time course, would you call it batch effect?

In different time course, the batch effect and true biological variation will be entangled.
Batch effects, which occur because measurements are affected by laboratory conditions,reagent lots and personnel differences. This becomes a major problem when batch effects are correlated with an outcome of interest and lead to incorrect conclusions. However, in single cell RNAseq, different datasets should be integrated with suitable algorithm (such as mnn, CCA, bbknn, harmony, scvi et al.), even no batch effect exists.

CCA does not have code in python, which will make it difficult to integrate, pySCENIC is probably easier but I would rather ask the developers.
@falexwolf We should consider a way to facilitate scanpy 'plugins'. A quick search shows me that this could be possible: https://packaging.python.org/guides/creating-and-discovering-plugins/ but honestly I don't know how it works. Nevertheless, given the number of tools that continue to appear we should consider a scheme that facilitate how developers can take advantage of scanpy preprocessing, storing, analysis and visualization tools.

Pyscenic has been integrated into scanpy now! Here is the hyper link:
https://github.com/aertslab/pySCENIC/blob/master/notebooks/pySCENIC - Integration with scanpy.ipynb

Great! Should we announce this somehow?

Maybe we should have a section in the docs for linking to packages that work with scanpy?


[10.x] Add 'Roundrobin' Symfony mailer transport driver

Symfony mailer provides a 'roundrobin' transport driver for distributing the mailing workload across multiple transports.
Here is the doc for reference.
In this PR, I've added support for roundrobin transport driver to Laravel. I've also added tests which is similar to the tests of the failover transport driver.
Basically, failover transport driver is extended from the roundrobin transport driver class by only overriding the logic for selecting the next transport. But the use case is different for these two drivers.
failover helps to achieve High availability and roundrobin helps to achieve load-balancing.
P.S.: I'll send PR for updating the doc and config file when this PR is merged.

This implementation is concerning because it always tries the broken connection instead of remembering across requests, that is, to actually do a failover.


Cleanup temp files and fix iOS build

Ran into an issue where file system was filling up on Android. Also fixed broken iOS build.

@amerchDev Thank you for the pull request. Will review and merge.


Fixed issue with keymap case sensitivity

Keymaps used to be case sensitive, leading to a small
handful of bugs (minimization in windows), and making custom
key mapping more difficult.
see https://github.com/zeit/hyper/pull/1509

Thanks for working on this!


Refactor variant_annotation

Is your feature request related to a problem? Please describe.
The current variant_annotation step is broken.
Describe the solution you'd like
Refactor the step allowing for multiple tools and implement VEP only for now. We currently only need annotation with gnomAD frequency and ENSEMBL-based transcript annotation.
Describe alternatives you've considered
N/A
Additional context
N/A

@holtgrewe Two things:

For cBioPortal export, we need additional control of the VEP parameters. In particular, we need to be able to pick the "best" transcript (--pick option) & other output options. For the latter, perhaps we could have a output_options parameter in the config file, with different defaults in variant_annotation/__init__.py & somatic_variant_annotation/__init__.py. For example:

  output_options:
  # Germline pipeline
  - "terms SO"
  - symbol
  - hgvs
  - af_gnomade
  - af_gnomadg
  # Somatic pipeline
  - everything

The wrapper code would simply be (ugly, but just as an illustration):
# In the python part:
current_step = snakemake.config["pipeline_step"]["name"]
vep_config = snakemake.config["step_config"][current_step]["vep"]
script_output_options = " ".join(["--" + x for x in vep["output_options"]])

...

# In the bash part:
vep --verbose --cache --offline --force_overwrite \
    --fork {vep_config[num_threads]} --buffer_size {vep_config[buffer_size]} \
    --cache_version {vep_config[cache_version]} --assembly {vep_config[assembly]} \
    $(if [[ ! -z "{vep_config[cache_dir]}" ]]; then \
        echo --dir_cache {vep_config[cache_dir]}; \
    fi) \
    {script_output_options} \
    $(if [[ "{vep_config[pick]}" = "yes" ]]; then \
        echo --pick; \
    fi) \
    --{vep_config[tx_flag]} \
    --fasta {snakemake.config[static_data_config][reference][path]} \
    --input_file {snakemake.input.vcf} --format vcf \
    --output_file {snakemake.output.vcf} --vcf --compress_output bgzip

As an additional benefit, it would make the wrapper more independent of the VEP version (for example, the options af_gnomade & af_gnomadg are not present in VEP version 102. gnomAD annotation is required by the option af_gnomad).

We would like to use VEP & the associated cache for version 102. Because that version is used for GDC/TCGA, and I'd like to keep the annotation fairly consistent, and comparable to public studies if possible.

Sure, I'll fix #384 now and worry about #387 later....


elastic vectorstore

I work at Elastic, so keep this in mind. I'm not a real end user, rather just someone presenting this library.
I'm doing a meetup soon in Sydney which is simultaneously both Elastic and Golang. I decided to use parakeet for my talk.
https://www.meetup.com/en-AU/sydney-elastic-fantastics/events/302290920/
My talk is pure go, and I'm sure many would love to know how to use Elastic as a vectordb. I personally am too new and slammed to attempt this prior (kubecon china, then kubeday japan before then). I wasn't planning to attempt to do both topics at the same time.
However, if you have time to add Elastic vector db support, I would be happy to show it to folks!
Here is something similar, but not sure if it helps much as it is in python.
https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/vectorstores/elasticsearch.py

@codefromthecrypt I found how to persist the embeddings, but the similarity search seems to be a little bit more tricky (not sure to be ready for the 29th)

if you have a draft, I can try to get someone to help. thanks for taking a swing meanwhile!

Baby steps (my first time with ES):

@codefromthecrypt
I admit I asked ChatGPT for help. I'm even surprised that it works the first time. 🤣

@codefromthecrypt You should have it before the end of the week

😍


improve performance of glTFLoader._applyExtensions

for a specific function name, cache extensions which are enabled and have the specific function.
cache extension function id
use Map to replace {} for ILoaderProperty._activeLoaderExtensionFunctions, to imporve has/set/delete operation

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Snapshot stored with reference name:
refs/pull/15913/merge
Test environment:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15913/merge/index.html
To test a playground add it to the URL, for example:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15913/merge/index.html#WGZLGJ#4600
Links to test babylon tools with this snapshot:
https://playground.babylonjs.com/?snapshot=refs/pull/15913/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/15913/merge
https://gui.babylonjs.com/?snapshot=refs/pull/15913/merge
https://nme.babylonjs.com/?snapshot=refs/pull/15913/merge
To test the snapshot in the playground with a playground ID add it after the snapshot query string:
https://playground.babylonjs.com/?snapshot=refs/pull/15913/merge#BCU1XR#0

Visualization tests for WebGPU (Experimental)
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15913/merge/testResults/webgpuplaywright/index.html

WebGL2 visualization test reporter:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15913/merge/testResults/webgl2playwright/index.html

Visualization tests for WebGL 1 have failed. If some tests failed because the snapshots do not match, the report can be found at
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15913/merge/testResults/webgl1/index.html
If tests were successful afterwards, this report might not be available anymore.

Visualization tests for WebGL 1 have failed. If some tests failed because the snapshots do not match, the report can be found at
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15913/merge/testResults/webgl1/index.html
If tests were successful afterwards, this report might not be available anymore.

Thanks for the PR. Just curious, have you seen measurable performance issues with this code?
@alexchuber also recently identified an issue with the loader extension ordering that needs to be address. It might be good to think about both of these issues together before making a complete fix for both as they will likely touch the same code.

Any progress on this PR?

@Starryi do you want me to close and you ll reopen when ready ?

closing as stale for now


How to install it?

Can you please give  short instruction of how to install it into Intellij. Thanks a lot.

Never mind. I realized that I can install it easily through the settings dialog directly. Nevertheless, it might be good to know for developers how to install a local version?


Uncaught SyntaxError: Invalid regular expression: /(^|.)rails(.|$)/: Stack overflow

I have a select element which has a remote: true to my contacts controller. However, if one of the options is selected, then I want to cancel that remote call and invoke a different controller instead. In the ajax:beforeSend event, I check if the specific option is selected and then return false:
if( $select.val() == "1" ){
    $('.loading').hide();
    $select.find('option:selected') .trigger('change.rails');
   return false;

The return false works and the original ajax call is stopped. However, the selected option looks like this:
<option data-url="/contacts/edit_multiple" data-remote="true" data-type="html" data-method="post" data-params="filterable=Lead" data-toggle="modal" data-target="filterEdit" value="1">Edit/Delete Filter</option>
And when I trigger the change.rails event on it, I get the following error:
Uncaught SyntaxError: Invalid regular expression: /(^|.)rails(.|$)/: Stack overflow
Anyone know why this error occurs?

I have the same problem it looks like a jQuery issue


Failures executing stdlib tests with debug builds of Ponyc on Windows

When building Ponyc on current master from sources on windows (Windows 10 Home, 64 bit) with Visual Studio Community 2017 15.9.7 and Visual Studio Build Tools 2017 15.9.7 with these commands:
python -x waf clean --config debug --llvm 7.0.1
python -x waf build test --config debug --llvm 7.0.1

And then executing:
.\build\debug-llvm-7.0.1\stdlib-debug.exe --only=file

most of the time the execution just stops with an exit code of -1073740940. When running the same command in the VSCode debugger I sometimes get Critical error detected c0000374.
According to @kulibali this seems to be a heap corruption. I was not able to pinpoint the actual issue, but it seems to be about the function pony_os_cwd() and in particular about this call to free: https://github.com/ponylang/ponyc/blob/master/src/libponyrt/lang/directory.c#L77 but this needs to be verified.
Interestingly builds with --config release run the tests just fine.

It seems to be independent of LLVM version. The same behavior appears with LLVM 3.9.1 and LLVM 6.0.1.

@kulibali was this fixed when we moved over to building our own Cmake?

Seems to have been. There's a debug build in CI that doesn't seem flaky nowadays.

This doesn't happen anymore.


FreshRSS account: "The data couldn’t be read because it isn’t in the correct format."

Version: Version 6.0d5 (6005) on macOS 11.1
When I try to add a self-hosted FreshRSS Account, I get the following error: The data couldn’t be read because it isn’t in the correct format.
I can see that folders from my FreshRSS instance, but not the feeds.
I used as

URL: https://[url_of_my_freshrss_instance]/api/greader.php (I tried https://[url_of_my_freshrss_instance]/api/ and https://[url_of_my_freshrss_instance]/api/fever.php as well but both gave me a Network error)
username: admin
Password: API Password I set up in FreshRSS

What version of FressRSS are you using?

What version of FressRSS are you using?

1.17.0 (the latest release listed on https://github.com/FreshRSS/FreshRSS/releases)

1.17.0 (the latest release listed on https://github.com/FreshRSS/FreshRSS/releases)

Do you have "Enable API Access" checked in Authentication?
If you set up a test account for me I will help you debug the problem.  Just send the credentials to<EMAIL_ADDRESS>

Do you have "Enable API Access" checked in Authentication?
If you set up a test account for me I will help you debug the problem.  Just send the credentials to<EMAIL_ADDRESS>

Yes, API is activated (and in use by the Reeder app on iOS). Just sent you credentials - many thanks!

Yes, API is activated (and in use by the Reeder app on iOS). Just sent you credentials - many thanks!

Version: Version 6.0d5 (6005) on macOS 11.1
When I try to add a self-hosted FreshRSS Account, I get the following error: The data couldn’t be read because it isn’t in the correct format.
I can see that folders from my FreshRSS instance, but not the feeds.
I used as

URL: https://[url_of_my_freshrss_instance]/api/greader.php (I tried https://[url_of_my_freshrss_instance]/api/ and https://[url_of_my_freshrss_instance]/api/fever.php as well but both gave me a Network error)
username: admin
Password: API Password I set up in FreshRSS

I had the same problem. For me the URL that worked was: https://[url_of_my_freshrss_instance]/p/api/greader.php. It's none of the ones you've tried.
I determined what URL would work by searching Keychain Access for FreshRSS. I use FreshRSS with Reeder, so there was an application password saved by it. It had the working URL as part the name.

Version: Version 6.0d5 (6005) on macOS 11.1
When I try to add a self-hosted FreshRSS Account, I get the following error: The data couldn’t be read because it isn’t in the correct format.
I can see that folders from my FreshRSS instance, but not the feeds.
I used as

URL: https://[url_of_my_freshrss_instance]/api/greader.php (I tried https://[url_of_my_freshrss_instance]/api/ and https://[url_of_my_freshrss_instance]/api/fever.php as well but both gave me a Network error)
username: admin
Password: API Password I set up in FreshRSS

I had the same problem. For me the URL that worked was: https://[url_of_my_freshrss_instance]/p/api/greader.php. It's none of the ones you've tried.
I determined what URL would work by searching Keychain Access for FreshRSS. I use FreshRSS with Reeder, so there was an application password saved by it. It had the working URL as part the name.

I had the same problem. For me the URL that worked was: https://[url_of_my_freshrss_instance]/p/api/greader.php. It's none of the ones you've tried.
I determined what URL would work by searching Keychain Access for FreshRSS. I use FreshRSS with Reeder, so there was an application password saved by it. It had the working URL as part the name.

I tried your suggestion but it doesn't work either. Also, when I enter https://[url_of_my_freshrss_instance]/**p/**api/greader.php in my browser, I get a 404, but using https://[url_of_my_freshrss_instance]/api/greader.php I receive an "OK"

I had the same problem. For me the URL that worked was: https://[url_of_my_freshrss_instance]/p/api/greader.php. It's none of the ones you've tried.
I determined what URL would work by searching Keychain Access for FreshRSS. I use FreshRSS with Reeder, so there was an application password saved by it. It had the working URL as part the name.

I tried your suggestion but it doesn't work either. Also, when I enter https://[url_of_my_freshrss_instance]/**p/**api/greader.php in my browser, I get a 404, but using https://[url_of_my_freshrss_instance]/api/greader.php I receive an "OK"

The same problem when using FreshRSS. I can see all my feed and folders but can't refresh. (it just get stuck half way) I am using RSSHub for many websites and services that don't provide rss feed. These feeds may have less standard format. (I'm not sure)
I attach my opml file for your debug if necessary.
FreshRSS_opml.zip

@Boycottangent Which version of FreshRSS are you on? I just imported your OPML file and didn't have any problems syncing with it on FreshRSS 1.17.1.

@Boycottangent Which version of FreshRSS are you on? I just imported your OPML file and didn't have any problems syncing with it on FreshRSS 1.17.1.

1.17.1-dev, I'll upgrade to stable release and see if problem still exists.

That version should be fine. Are your RSSHub feeds included in your OPML file?

But do upgrade to a stable version to see if it helps.

I upgrade to version 1.18.0 and now everything works well. That OPML file has all my feeds including RSSHub feeds so the problem is probably caused by freshRSS itself.
Thank you!

@necrevistonnezr did you ever get your problem resolved?

I just tried again with my FreshRSS instance updated to 1.18. It's still weird: Now it syncs to one of my five categories (= folder) of feeds, i.e. I see one folder and one feed, synced. The rest of the folders and feeds are not visible.

Have you tried deleting and adding the account again?

It's a brand new installation of Netnewswire.

After completely removing, reinstalling NNW and adding the account again, FreshRSS works! Thanks!


glinternet example

Would be nice to support glinternet.

For a full support, we need the following matrix: ad.matrix.one_hot. It should one-hot encode the specified columns of mat with user-provided levels.


Doesn't recognize symbol "

Describe the bug
Doesn't recognize symbols copied from elsewhere
Expected behavior
A poll must be created nomatter where it is copied from.
Additional context
When I retyped in the % " ' the poll came back

https://www.reddit.com/r/Unicode/comments/h9xwm5/the_tallest_unicode_character_in_discord/

The single quotation  ’ (sent from iPhone in this case)  is different from the single quotation ' that we normally use. ASCII code of the former is 8217 and the latter is 39. The discord.py library itself seems to have an exception handler for special characters when passed as an argument to a coroutine, therefore the bot crashes right when the argument is read - this happens before executing any instructions written in said coroutine so couldn't use try/exceptions.


Finalized v2 Refactor and Semantic Release

This PR moves the last v2 refactor updates into the master branch. It also activates semantic release for future versioning of this repository.

Pull Request Test Coverage Report for Build 788

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+20.6%) to 82.124%

Totals

Change from base Build 769:
20.6%

Covered Lines:
2060

Relevant Lines:
2348

💛  - Coveralls

:tada: This PR is included in version 1.0.0 :tada:
Your semantic-release bot :package::rocket:


Elixir w/Phoenix has Live Reloading built in and it functions great

Hey @mgnshawn
Thanks for pointing out.
Could you please make a pull request? I have not used Elixir so I think it'll be better if you add it.


Error with rightclicks

I'm gonna fix the issue myself, just posting for documentation

Ok it's an issue with how it handles numbers in dropdown, i was going to fix some issues with that today anyway so i might as well do it now

https://github.com/alesan99/mari0_ae/pull/466


Icon request: Canapii

URL of the site or app (required): https://www.canapii.com/
Logo resources for the icon (optional, but recommended): https://twitter.com/CanapiiOfficial/photo

Here you go!
https://github.com/aegis-icons/aegis-icons/blob/master/SVG/Canapii.svg
❓ How to add icon(s) to Aegis?
P.S. Please do not add logo resources unless it fills the requirements writted in the issue template


Block list checker for the allowed address resolver

@FabricMC/yarn-triage anyone mind reviewing this? Should be relatively small changes.


Change: Use separate labels for compose-service and sub

Use separate labels for the compose-service name and the "sub" (derived from the ref).
The reason for this is that with a single label it is impossible to correctly find out which part is the compose-service name and which is the sub, e.g. in "portal-foo-bar" the compose-service name could be "portal" (with sub "foo-bar") or "portal-foo" (with sub "bar") or even "portal-foo-bar". Being able to correctly distinguish between compose-service name and sub is important for displaying information in the Web UI.
This change fixes this by using two separate labels. The same labels are used in the K8s service spec as selectors, which should work fine.

Won't this break for singletons?
What does "sub" stand for? Should we start a "terminology" section in the Readme?

Singletons don't have the "k8ify.sub" label. I've tested this and it works fine.
Considerations:

sub CAN be the same as ref (as in git ref), but not necessarily. So I do not want to call this ref because that would lead to some expectations that it wouldn't meet.
sub is used for naming sub-variants of compose services, so you can read it as "sub-variant", "sub-service", "sub-deployment". Because the value in sub is used in different places to name various different things I could not give it a more specific name as that would not be correct everywhere. Hence I left it at "sub".
"sub" refers to how the value is used, not where it comes from. If we wanted to call it according to where it comes from, its name would be something like "something-like a-git-ref-that-is-safe-to-use-in-k8s-resource-names". Gitlab has the same idea, they call it "[...]-slug".
The sub is used internally in k8ify and the Web UI, and not something the user directly needs to interact with, hence at this point I don't think this needs to be documented for the end-user.
I would very much like to have a more self-explanatory name but I just can't come up with anything (well and "something-like a-git-ref-that-is-safe-to-use-in-k8s-resource-names" is a bit too long for my taste). The only alternative I can see at the moment is to use the Gitlab terminology and call it "ref-slug" (which honestly doesn't seem like the worst idea).

As discussed I have changed "k8ify.sub" to "k8ify.ref-slug" including the corresponding go variable names.


Dev Readme Update

Readme Update

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

update REAME


realTouch() Absolute Coordinates does not respect Viewport size

Hi,
I have the viewport size set in a Cypress test to cy.viewport(1280, 720);, however the realTouch() does not always have the correct absolute coordinates when the browser is resized - I believe this is because the coordinates are calculated based on the iframe rather than the Html or Body element of the page.
For example, if I run my test on a full page width, the calculated absolute coordinates are x: 1254 and y: 177. If I resize my window, the coordinates change to x:1221 and y:177, even though my viewport is locked to 1280x720px.

It calculates a real coordinate. On the main tab (not your iframe) so it can be any number bigger than viewport. The main question – can you make a reproduction so we can understand exactly why it can not detect particular element

Sure, it should be pretty easy to replicate. Let's say you have a Cypress test with a fixed viewport set at 1280x720px, you add a test to do a touchstart realTouch with the coordinates { x: 650, y: 130}. The test passes until you resize your browser window, then it starts to fail. As an example, if my browser window is 2000px wide the test passes, if I resize to 1500px the test fails. My target is at the same location with the fixed viewport.

I forgot to mention a very important detail, I'm running the test in the Cypress browser (via cypress open command). This is where I can see the behaviour I explained above.

This is how it works by the way. Chrome CDP doesn't know anything about your viewport. We need to calculate element coordinate on the tab with a correct shift to the iframe scale() and so on.
https://github.com/dmtrKovalenko/cypress-real-events/blob/4fd016db8f855dac3a0651d55257573fa8bd5a5f/src/getCypressElementCoordinates.ts#L25
So your custom coorniate should relate also the whole tab root


[SCB-762] resolve eclipse compile warnings

Coverage remained the same at 87.081% when pulling d805dbae87f6208886ee697e109b0f88f548ad7c on wujimin:resolve-eclipse-compile-warnings into 8227f0ccbe7d69db51df1fc22b11a364662c1d9d on apache:master.

CI passed, but show error here, what happened?

The CI build just started,  code coverage is updated.

that's because i triggered it again......


Trouble with dropdown CSS

Hi,
I installed and added drop-down with the guidelines told in the Readme, but while rendering dropdown CSS is not getting applied correctly. I checked Webpack logs and found out that only react-dropdown.css is getting compiled. Not able to figure out why CSS is not getting applied.
Can someone here help me out?
Thanks in advance
SS of react-dropdown

SS of webpack

@sudeeptarlekar
Did you directly import react-dropdown.css ?
I used this dropdown as below:
import<EMAIL_ADDRESS>import Dropdown from '@trendmicro/react-dropdown';

export default Dropdown;
export {
    DropdownToggle,
    DropdownMenu,
    DropdownSubMenu,
    MenuItem,
    DropdownButton
} from '@trendmicro/react-dropdown';

and import this everywhere if you want to use it.

I closed this issue.
If need to reopen, do it


Deprecates the old string expansion tags in header_rewrite

[approve ci docs]

[approve ci autest]
[approve ci docs]

Cherry picked into 8.1.x


Content window is blocking text on deepspeed.ai

Is it somehow possible to remove or move the Content window on deepspeed.ai?

Good catch @sorenwacker - I'll take a stab at getting this moved to the side.  We thought this was fixed here, but seems not to be?

This should be fixed now, let us know if you're still seeing this, but I see it fixed on my side now.


Move calendar integrations into plugins

shouldnt be a separate menu item, it's a perfect demo of a plug-in

Done by salman

#1192


Fix shares_increment bug & Add New Feature in Cash Penalty Env

Found bugs in shares_increment logic:

Negative actions should be added first with self.shares_increment
Actions clipping should be performed AFTER not BEFORE the rounding actions by shares_increment

Add patient parameter to control whether we have to end the cycle when we're running out of cash or just don't buy anything until we got additional cash

Added tests module for shares_increment


⚠ POST to /fact_check has degraded performance

In 37e6fad, POST to /fact_check (https://dev.factiverse.no/v1/fact_check) experienced degraded performance:

HTTP code: 200
Response time: 28987 ms

Resolved: POST to /fact_check performance has improved in e0daa93.


Point "changes" place at zooming out

I attached the scene and MainActivity to reproduce the issue.
Correct/At zoom level 16:

At Zoom level 15:

You can see the upper-left point moves to another position somewhy and one point is missing as well.
PointGlich.zip

PointGlich.zip

Hi @vilinet - thanks for the report. I peeked at the example file you sent and noticed that these points are encoded in GeoJSON as a linestring. Have you tried encoding them as a MultiPoint geometry or just as separate Point features? The reason I ask is that GeoJSON layers like this have their LineString and Polygon geometries split into separate tiles and, in some cases, simplified for rendering.

Could work, but initally we show the Line and the points the line is made of. So we show and the joint points. Is it possible to achive without linestring?

Aha, that's good to know! I think for now you should encode both a LineString and a MultiPoint, even though it is somewhat redundant.
However, I would expect point rendering with LineString data to work in this scenario. I'll look into this behavior. It will be easier to figure out what's going on now that I know it works with MultiPoint data.


improve lane departure alert detection

Improve left lane departure detection

I like that it's still symmetric!
Can we also add a timer to not show warnings a few seconds after using the blinker. I got a lot of false positives when using the short tap on the blinker. The blinker is then already off when finishing the lane change.

merged internally first. Could not merge this directly due to conflicts


Set the file prefix in image_cache_manager

Summary
Set the file:// prefix for android in one central place instead in every component that displays an image and should solve the unlink issue that is being reported on Sentry for Android
Ticket Link

Do we think this is the cause of that Sentry spam about not being able to delete cached files? Also, does this have a Jira ticket?

@hmhealey Yes I do think this is the cause of the sentry spam and no, there is no ticket for this, that is why I wanted to discuss as I mentioned in the channel as well as decide if this goes in 1.14 or 1.15


Add something like use_compile parameter for Trainer

Description & Motivation
For below example, model is being compiled, DDPStrategy is passed to Trainer, then during fit method DDPStrategy is being applied, so forward is compiled but _pre_forward/_post_forward in DDP class is not.
Due to this in DDP _pre_forward/_post_forward cpp_reducer is not being disabled later on causing problem with queueing callback.
When DDP is also compiled cpp_reducer is disabled as expected.
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset
import pytorch_lightning as pl
from pytorch_lightning.strategies import DDPStrategy
import functools
from torch._dynamo import compiled_autograd

torch._dynamo.config.optimize_ddp = "python_reducer"

def compiler_fn(gm):
    return torch.compile(gm)

torch._C._dynamo.compiled_autograd.set_autograd_compiler(
    functools.partial(compiled_autograd.AutogradCompilerInstance, compiler_fn)
)

class SomeModel(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = nn.Linear(10, 1)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        return torch.mean((self(x) - y) ** 2)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)

def create_dataset(num_samples=1000):
    x = torch.randn(num_samples, 10)
    y = torch.sum(x, dim=1, keepdim=True)
    return TensorDataset(x, y)

def run_training():
    dataset = create_dataset()
    train_loader = DataLoader(dataset, batch_size=32)

    model = SomeModel()

    # First compile whole model
    model = torch.compile(model)

    # static_graph has to be true causing _DDPSink.backward method queueing callback
    ddp_strategy = DDPStrategy(static_graph=True)

    trainer = pl.Trainer(
            max_epochs=2,
            accelerator='cpu',
            devices=1,
            strategy=ddp_strategy
    )

    # DDP will be applied inside fit method, so DDP pre/post forward won't be compiled while forward is
    trainer.fit(model, train_loader)

if __name__ == "__main__":
    run_training()

Expected repro:
[rank0]:     Variable._execution_engine.queue_callback(  # type: ignore[call-arg,misc]
[rank0]: RuntimeError: Final callbacks can only be installed during backward pass.

Pitch
It seems useful to compile after applying strategy, so my suggestion is to add something like bool use_compile parameter for Trainer which would help for example in this situation, and also be cleaner to use.
Looks like it should be more advanced than just bool to setup specific backend and other optional compile parameters.
Alternatives
src/lightning/pytorch/trainer/trainer.py
class Trainer:
    def _run(
        self, model: "pl.LightningModule", ckpt_path: Optional[_PATH] = None
    ) -> Optional[Union[_EVALUATE_OUTPUT, _PREDICT_OUTPUT]]:
        (...)
        # ----------------------------
        # SET UP THE TRAINER
        # ----------------------------
        (...)
        self.strategy.setup(self)
        (...)

------> ### Pseudo proposition
------> if self.use_compile
------>     self.model = torch.compile(self.model)

        # ----------------------------
        # RUN THE TRAINER
        # ----------------------------
        results = self._run_stage()

        (...)

Additional context
File: torch/nn/parallel/distributed.py
class DistributedDataParallel(Module, Joinable):
    def _should_disable_cpp_reducer(self) -> bool:
        return self._use_python_reducer and (
            torch._utils.is_compiling() or self._force_to_disable_cpp_reducer
        )

    def _pre_forward(self, *inputs, **kwargs):
        if self._should_disable_cpp_reducer():
            return inputs, kwargs
        (...)

    def _post_forward(self, output):
        if self._should_disable_cpp_reducer():
            return output
        (...)

Dynamo replaces output of torch._utils.is_compiling() to True when code is compiled, False otherwise.
cc @borda

cc @jerome-habana


scrollview: Clean include statements

cstring was included twice (reported by Martin Strunz).
Use C++ header files and sort them alphabetically.
Signed-off-by: Stefan Weil<EMAIL_ADDRESS>

Rebased now to fix merge conflict. limits.h / climits was also redundant and is fixed here.


Strange Problem when I use ds4drv. I have a new"SyntaxError: invalid syntax"

Traceback (most recent call last):
File "/usr/local/bin/ds4drv", line 11, in
load_entry_point('ds4drv==0.5.1', 'console_scripts', 'ds4drv')()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 480, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2693, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2324, in load
return self.resolve()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2330, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "build/bdist.linux-x86_64/egg/ds4drv/main.py", line 8, in
File "build/bdist.linux-x86_64/egg/ds4drv/servers/init.py", line 1, in
File "/usr/local/lib/python2.7/dist-packages/ds4drv-0.5.1-py2.7.egg/ds4drv/servers/udp.py", line 126
*mac,  # MAC,
^
SyntaxError: invalid syntax

How could I fix it ?


[BUG] Prompt regarding Data Volumes shown when destroying a VM is misleading

Describe the bug
When destroying an instance the user should be shown any data disks attached and given the option to delete them.
To Reproduce
Steps to reproduce the behavior:

Create a VM (without a data disk)
Click on 'Destroy' the VM
See the prompt (compare to Legacy UI)

Expected behavior
The prompt should read 'Data Volumes to be deleted:'  (incorrect mapping used - should be "label.delete.volumes")
When there are no Data volumes attached, the label "label.volume.empty" should be used ("No data volumes attached to this VM")
Screenshots
Primate:

Legacy:

Desktop (please complete the following information):

OS: [e.g. Windows, Mac, iOS, Android with device/screen details if applicable]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

@davidjumani please create a new custom component for this action; the autogenerated widget will limit the UX which users may not like, like @PaulAngus 's expected behaviour.

With @davidjumani


Check type of dot syntax

In 3.x, we added several convenient aliases for standard optic composition, e.g.
case class Person(name: String, emails: List[String])
val emails: Lens[Person, List[Email]] = ...

emails.at(2)       // equivalent to emails.composeLens(at(2))
emails.index(2) // equivalent to emails.composeLens(index(2))

Since optics extend one another in 3.x, we have used overloaded methods to define at, index, etc ... Now, the problem is that it is easy to use the wrong type in those aliases. For example, Lens doesn't have an index method because the inherited method from Optional is correct. However, Lens overloads at method because the return type should be more precise (Lens composed with Lens gives another Lens).
Shall we add some tests for each alias to check? e.g.
for all optics o, for all alias a,
o.$a.type == o.compose(a).type

fixed in #789 for the 2 dot-syntax methods


refactor!: prove iIndepFun.prod in terms of iIndepFun.pi'

This trades a Nonempty assumption for a Fintype ι assumption. Is this OK? i.e. if we need independence of an infinite family of finite sets of independent random variables, we might not be able to use iIndepFun.pi' (as is currently).
I'm not familiar enough with math to know if the Fintype ι assumption is necessary to prove iIndepFun.pi in "real life"; I can only intuitively justify the Fintype (κ i) since the relevant preimages are intersections of only Finsets of ι but all of κ i.
in our current iIndepFun.pi formalization, Fintype ι is used to apply iIndepFun_iff_pi_map_eq_map.
Maybe the optimal end state involves adapting (instead of deleting) Terence's iIndepFun.prod argument to generalize iIndepFun.pi to infinite ι.

iIndep_fun.pi can be generalised to infinite ι, but it's not as trivial as it ought to be.


Gre reading comprehension

Added a reading comprehension task with GRE passages

Thank you for your submission!
Raising a potential concern now, before we enter the task review process. It seems like this task may be taken verbatim from the GRE/ETS website. If this is the case, and this is a copy of a copyrighted GRE test or sample GRE test, then you would need to get permission from ETS in order to add it to BIG-bench. If this is a novel task, that is only inspired by the GRE, then that is great, and you should make this clear in the task description.
Thank you again!

Hi Sohl!
Thank you for your message.
I didn't take this from the ets website, questions were compiled from various websites. However, to your point, I believe I have to confirm those passages weren't taken from the ets website originally.
Thank you!

Yeah, if you're building your task out of potentially copyrighted material, you need to make sure you have permission/rights to do so. I know it poses a practical barrier to a lot of material that would be fascinating to include.

Thank you for submitting your task! We are in the process of pairing submissions with task reviewers and will be in touch soon.

Update for one of the two websites I contacted to request permission to use:
Got full permission to use

Thank you very much for contributing this! I am one of your assigned reviewers.
Overall, I like the idea behind this submission. It is indeed fascinating how much implicit context LMs pick up on. I think this task is close to being in a state where it should accepted, but I think the authors should slightly beef up the justification in the readme and should address the issue of whether this task can be solved by memorizing the internet.
What I understood
This task puts asks language models to respond to long form queries similar to those from the reading comprehension portion of the GRE. In each case, the questions consist of a paragraph followed by a multiple choice question with one or more correct answers.
Review Criteria
Correctness
It appears that the JSON file is valid and passed the automatic GitHub tests
Specificity
I think the question of whether LMs can reason about reading comprehension problems that go beyond recall is certainly quite interesting and is reasonably specific.
Thoroughness
I think the task as submitted is fairly robust. My only comment in this direction is that there are arguably two different things being measured: (a) whether LMs can infer the required information and (b) whether LMs can produce the required information in exactly the required form. For instance, in the example paragraph with the sharks, with if the LM output ‘B’ or ‘B)’ or ‘the second answer appears more appropriate’ or ‘B) impartial observer’ or something like that. In all these cases I would say the LM has understood exactly what is required but I think would not get the right answer. I am really not sure if there is a simple way to mitigate this.
Difficulty
The readme file did not include any mention of how difficult this task is for older LMs, so it is hard to say.
Memorizing The Internet
It seems from the discussion between the authors and Jascha Sohl-Dickstein above that the reading comprehension examples being tested are taken directly from some websites. If so, I am worried that perhaps this task is in fact solvable by memorizing the internet.
Novelty
I am certainly not an expert on this stuff, but the work appears novel to me!
Justification
I think it would be useful if more justification were included in the readme file. For example, have other LMs been tested on such tasks? If so, how did they do?
Size and Compute Resources
No issues I can see.

@stanislavfort You are the second reviewer on this PR. Please provide feedback before Friday, March 26th.

Hi @medinabaitemirova - As a reminder you have until Friday, March 26th to revise your task to be accepted in the workshop. Please tag your reviewers when you have addressed their suggestions.
Reviewers:
@boris-hanin and @stanislavfort if the submitter has made revisions please be sure to notify me with a formal "accept" or "does not meet criteria" by Friday, March 26th (if you haven't already done so).

Hi @boris-hanin !
Thank you so much for your feedback. I added a section on related work. Please let me know if you think it would be useful to add more information.

Reassigning @vedant as the second reviewer on this task. Please provide feedback and then tag me with comment to "accept" or "reject" by Friday, March 26th.

@medinabaitemirova - the updated related work section is great
@chiafullo accept

@medinabaitemirova Thank you for this submission! I'm one of your assigned reviewers.

Correctness: It appears the task is specified correctly and passes automated tests.
Specificity: Evaluating the capability of LMs to perform graduate-level reading comprehension tasks is interesting and specfic.
Thoroughness: The task appears to be clearly probe a specific capability with trusted targets. The correct answers to GRE questions are presumably robust and specific unless the exam itself is poorly constructed.
Difficulty: This task is not likely to be fully solvable by many existing language models.
Not solvable by memorizing the internet: Because much of the training data appears on the internet, it is possible that models might be able to memorize answers to such questions. However
Novelty: To my knowledge this is not a type of task that is already included in the benchmark.
Justification: The task is motivated by the sound goal of evaluating models' capability in deriving implicit meaning.
Size: No issues I can foresee.
Compute resources: No issues I can foresee.

@chiafullo: Accept

@timothyn617 for meta review and merge (if acceptable)


Addressing #245: fixes for SQL Server

Big improvements in SQL Server syntax as proposed by @FilipDeVos in #245.
In particular, datetime2 and other new data types from SQL Server 2008, as well as better chosen types for big numbers.
Legacy software users can still load the dialect through gorp.SqlServerDialect{"2005"} to get SQLServer2005-Access2007 compatible types.

Hey there.
This PR doesn't seem to break anything. Any advice from projects collaborators?

Hmm ... Honestly, I'm not sure if any of us use SQL Server.  I certainly don't - I only know the bare minimum required to convert SQL Server DBs to something else.
It's possible that all of us are waiting for someone else to handle the SQL Server tickets, because we all think one of the others knows SQL Server better than we do.

I use SQL Server at work.
I wrote the dialect when the project was @coopernurse's (see the README... hey, I'm particularly proud of that! :).
The edits suggested by @FilipDeVos are a certain improvement over what I had written at first.
I am actually using this PR in production. I hope to see the automated testing (#249) working soon, but until then I see no worries on merging this this PR as it is nothing but an improvement on the SQL Server syntax.

Yeah, I'm not saying anything against merging it, just trying to explain why it may be that none of us have gotten to it yet.  I'm not yet comfortable merging things that I have no knowledge of.  :)
@GeertJohan I've got no issues with merging this.  What do you think?

@nelsam @GeertJohan Guys, are you really taking into account PRs from non-collaborators?

@qrawl, apologies for not looking at this, must've totally missed the notification.
I have no knowledge of SQL Server at all, but since you wrote the dialect I trust the changes are good and will merge them now.
Would you like to come on board as a collaborator to maintain the SQL Server part and, if you have time for it, more?

@GeertJohan Would be glad to!

In issue #245 you wrote:
It seems like you can't use a nvchar(max) field as primary key. Do you have such experience?

Is this relevant to this PR? Does this mean that it shouldn't return nvchar(max) when the field is used as primary key?

This incompatibility was already present in the first version of the dialect. GORP could check for PKs and automatically set the greater accepted length when max is specified. But wouldn't this be a little intrusive?

Ah okay, if behavior hasn't changed with this PR, then lets leave it this way.
Could it be interesting to add in documentation?


Terrible example; please revise

This example has 2 fundamental flaws:

the lock is unnecessary; since PreDrawEvent is declared as a private event (which may or may not be wise, but that's another matter), it already has thread-safety implemented by the compiler via the "field-like events" specification
the lock is invalid - it locks on a changing value (bad) that might be null (very bad)

Very bad code. Please fix.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5a96a00d-16c2-78da-1394-58e9db154d2f
Version Independent ID: ce67b2fa-8c51-06f5-fe8c-ba98b697bc2c
Content: add (C# Reference)
Content Source: docs/csharp/language-reference/keywords/add.md
Product: dotnet-csharp
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

Thanks for pointing this out @mgravell
Your analysis is correct. This example needs to be updated.

This would also throw an exception when you try to register to OnDraw if PreDrawEvent is still empty. An awful fix would be to initialize the PreDrawEvent with an empty delegate.


Adding custom Python libraries: No module named 'numpy.core._multiarray_umath'

I'm trying to build the flet_example app for Android on a Ubuntu 23.10 system with a simple numpy import.
The p4a create --requirements numpy ... command for compiling the files works fine and the dart run serious_python:main package app/src --mobile also.
When building the apk with flutter build apk this results in an error:
/home/janik/Android/Sdk/ndk/25.2.9519653/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip: error: '/home/janik/Downloads/serious-python/src/serious_python/example/flet_example/build/serious_python_android/intermediates/merged_native_libs/release/out/lib/arm64-v8a/libpythonbundle.so': The file was not recognized as a valid object file
Nevertheless I can install the apk to my Android device but I get a black screen.
Have you any ideas how to fix that?

The reason why I got a black screen was that I used Python 3.10 with p4a but Serious-Python expects Python 3.11 to run properly.
With the correct Python version the app starts but I get a new ImportError from NumPy which is shown in the app:
Traceback (most recent call last):
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/core/__init__.py", line 23, in <module>
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/core/multiarray.py", line 10, in <module>
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/core/overrides.py", line 6, in <module>
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 40, in <module>
  File "<frozen runpy>", line 229, in run_module
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/serious_python_tempGFODNN/main.py", line 4, in <module>
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/__init__.py", line 144, in <module>
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/core/__init__.py", line 49, in <module>
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.11 from ""
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

My requirements file in the app/src directory says numpy==1.22.3 and I used p4a create --requirements numpy==1.22.3 ... to create the Python executables. I guess this is somehow related to #75 .

The reason why I got a black screen was that I used Python 3.10 with p4a but Serious-Python expects Python 3.11 to run properly.
With the correct Python version the app starts but I get a new ImportError from NumPy which is shown in the app:
Traceback (most recent call last):
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/core/__init__.py", line 23, in <module>
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/core/multiarray.py", line 10, in <module>
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/core/overrides.py", line 6, in <module>
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 40, in <module>
  File "<frozen runpy>", line 229, in run_module
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/serious_python_tempGFODNN/main.py", line 4, in <module>
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/__init__.py", line 144, in <module>
  File "/tmp/serious_python_tempGFODNN/__pypackages__/numpy/core/__init__.py", line 49, in <module>
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.11 from ""
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

My requirements file in the app/src directory says numpy==1.22.3 and I used p4a create --requirements numpy==1.22.3 ... to create the Python executables. I guess this is somehow related to #75 .

My problem was that I thought I need the numpy==1.22.3 line in the app/src/requirements.txt file. This line should be deleted as NumPy is already part of the app when using p4a create --requirements numpy ... and export SERIOUS_PYTHON_P4A_DIST=.... Now everything works fine.


use client-go's default client config loading rules

ref https://github.com/istio/istio/pull/5330#issuecomment-385736342
cc @linsun

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: nmittler
Assign the PR to them by writing /assign @nmittler in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

istioctl/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/LGTM
cc @esnible

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: linsun
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

istioctl/OWNERS [linsun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-e2e.sh
07fe75a07978a2cad9f229677a9596924c862aad
link
/test istio-pilot-e2e

prow/e2e-bookInfoTests-v1alpha3.sh
07fe75a07978a2cad9f229677a9596924c862aad
link
/test e2e-bookInfo-envoyv2-v1alpha3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Help needed compiling. / Please provide bianry. [wemos_d1_mini32]

Hi, thanks for you work on this. I installed vs code and platform io but unfortunately I'm unable to compile your latest realease.
Here is my console ouptput. I would apprechiate if you could have a look and let me know why it suddenlty want's the ESPAsyncDNSServer?
*  Executing task: C:\Users\Max\.platformio\penv\Scripts\platformio.exe run --environment esp32dev

Warning! `test_build_project_src` configuration option in section [env:googletest_esp32] is deprecated and will be removed in the next release! Please use `test_build_src` insteadWarning! `src_filter` configuration option in section [env:googletest_esp32] is deprecated and will be removed in the next release! Please use `build_src_filter` instead
Warning! `src_filter` configuration option in section [env:native] is deprecated and will be removed in the next release! Please use `build_src_filter` instead
Processing esp32dev (platform: https://github.com/platformio/platform-espressif32.git; board: esp32dev; framework: arduino)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (5.2.0+sha.03254d4) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
- framework-arduino-esp32 @ 2.0.5+sha.a5f03a8
- framework-arduinoespressif32 @ 3.20005.220925 (2.0.5)
- tool-esptoolpy @ 1.40201.0 (4.2.1)
- toolchain-xtensa-esp32 @ 8.4.0+2021r2-patch3
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 41 compatible libraries
Scanning dependencies...
Dependency Graph
|-- ArduinoJson @ 6.19.4
|-- ESP Async WebServer @ 1.2.3+sha.f71e3d4
|   |-- AsyncTCP @ 1.1.1
|   |-- FS @ 2.0.0
|   |-- WiFi @ 2.0.0
|-- ESPAsync_WiFiManager @ 1.15.0
|   |-- AsyncTCP @ 1.1.1
|   |-- ESP Async WebServer @ 1.2.3+sha.f71e3d4
|   |   |-- AsyncTCP @ 1.1.1
|   |   |-- FS @ 2.0.0
|   |   |-- WiFi @ 2.0.0
|   |-- ESP_DoubleResetDetector @ 1.3.2
|   |   |-- SPIFFS @ 2.0.0
|   |   |   |-- FS @ 2.0.0
|   |   |-- EEPROM @ 2.0.0
|   |   |-- FS @ 2.0.0
|   |   |-- LittleFS @ 2.0.0
|   |   |   |-- FS @ 2.0.0
|   |-- WiFi @ 2.0.0
|-- NimBLE-Arduino @ 1.4.0
|-- FauxmoESP @ 3.4.0
|   |-- AsyncTCP @ 1.1.1
|   |-- WiFi @ 2.0.0
|-- LinakDeskEmbedded
|   |-- LinakDeskCore
|   |-- NimBLE-Arduino @ 1.4.0
|-- ESPmDNS @ 2.0.0
|   |-- WiFi @ 2.0.0
|-- ESP_DoubleResetDetector @ 1.3.2
|   |-- SPIFFS @ 2.0.0
|   |   |-- FS @ 2.0.0
|   |-- EEPROM @ 2.0.0
|   |-- FS @ 2.0.0
|   |-- LittleFS @ 2.0.0
|   |   |-- FS @ 2.0.0
|-- FS @ 2.0.0
|-- LittleFS @ 2.0.0
|   |-- FS @ 2.0.0
|-- LinakDeskCore
Building in release mode
Compiling .pio\build\esp32dev\src\main.cpp.o
Archiving .pio\build\esp32dev\libf5d\libAsyncTCP.a
Archiving .pio\build\esp32dev\lib4ed\libFS.a
In file included from .pio/libdeps/esp32dev/ESPAsync_WiFiManager/src/ESPAsync_WiFiManager.h:40,
                from src/main.cpp:13:
.pio/libdeps/esp32dev/ESPAsync_WiFiManager/src/ESPAsync_WiFiManager.hpp:221:10: fatal error: ESPAsyncDNSServer.h: No such file or directory

***************************************************************************
* Looking for ESPAsyncDNSServer.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:ESPAsyncDNSServer.h"
* Web  > https://registry.platformio.org/search?q=header:ESPAsyncDNSServer.h
*
***************************************************************************

#include <ESPAsyncDNSServer.h>
         ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Compiling .pio\build\esp32dev\lib586\WiFi\WiFiScan.cpp.o
Compiling .pio\build\esp32dev\lib586\WiFi\WiFiServer.cpp.o
Compiling .pio\build\esp32dev\lib586\WiFi\WiFiUdp.cpp.o
Compiling .pio\build\esp32dev\libdc2\ESP Async<EMAIL_ADDRESS>Compiling .pio\build\esp32dev\libdc2\ESP Async<EMAIL_ADDRESS>Compiling .pio\build\esp32dev\libdc2\ESP Async<EMAIL_ADDRESS>Compiling .pio\build\esp32dev\libdc2\ESP Async<EMAIL_ADDRESS>Compiling .pio\build\esp32dev\libdc2\ESP Async<EMAIL_ADDRESS>Compiling .pio\build\esp32dev\libdc2\ESP Async<EMAIL_ADDRESS>*** [.pio\build\esp32dev\src\main.cpp.o] Error 1
=========================================================================== [FAILED] Took 18.82 seconds ===========================================================================

Environment    Status    Duration
-------------  --------  ------------
esp32dev       FAILED    00:00:18.825
====================================================================== 1 failed, 0 succeeded in 00:00:18.825 ======================================================================

*  The terminal process "C:\Users\Max\.platformio\penv\Scripts\platformio.exe 'run', '--environment', 'esp32dev'" terminated with exit code: 1.
*  Terminal will be reused by tasks, press any key to close it.

Hi, thanks for your interest in the project!
This is unexpected, I have set up the CI explicitly to catch any such issues, the latest run finished today successfully on all platforms: https://github.com/krzmaz/LinakDeskEsp32Controller/actions/runs/3264545031
What is even more unexpected is that form the logs it seems that your build is using different versions of the dependencies than explicitly specified in platformio.ini.
For example

ESPAsync_WiFiManager @ 1.15.0

is different than https://github.com/krzmaz/LinakDeskEsp32Controller/blob/main/platformio.ini#L78:

khoih-prog/ESPAsync_WiFiManager @ 1.9.8

Did you change your platformio.ini file, or ran any automatic dependencies upgrades?

Hi, thank's for you fast response. I'm sorry, I was already fiddeling to long with this yesterday and forgot that I had changed something, because it wouldn't compile straight from the beginning.
Here are my steps with a fresh setup:

Downloading your latest release:

Opening it with VS Code --> search for PlatfomIO --> Enable (Workspace):

Click Build:

Console Output:

 *  Executing task: C:\Users\Max\.platformio\penv\Scripts\platformio.exe run --environment esp32dev

Warning! `test_build_project_src` configuration option in section [env:googletest_esp32] is deprecated and will be removed in the next release! Please use `test_build_src` instead
Warning! `src_filter` configuration option in section [env:googletest_esp32] is deprecated and will be removed in the next release! Please use `build_src_filter` instead
Warning! `src_filter` configuration option in section [env:native] is deprecated and will be removed in the next release! Please use `build_src_filter` instead
Processing esp32dev (platform: https://github.com/platformio/platform-espressif32.git#feature/idf-v4.0; board: esp32dev; framework: arduino)
-----------------------------------------------------------------------------------------------------------------------------------------------------
Platform Manager: Installing git+https://github.com/platformio/platform-espressif32.git#feature/idf-v4.0
git version 2.37.2.windows.2
Cloning into 'C:\Users\Max\.platformio\.cache\tmp\pkg-installing-6vixnn_o'...
warning: Could not find remote branch feature/idf-v4.0 to clone.
fatal: Remote branch feature/idf-v4.0 not found in upstream origin
Error: VCS: Could not process command ['git', 'clone', '--recursive', '--depth', '1', '--branch', 'feature/idf-v4.0', 'https://github.com/platformio/platform-espressif32.git', 'C:\\Users\\Max\\.platformio\\.cache\\tmp\\pkg-installing-6vixnn_o']

 *  The terminal process "C:\Users\Max\.platformio\penv\Scripts\platformio.exe 'run', '--environment', 'esp32dev'" terminated with exit code: 1.
 *  Terminal will be reused by tasks, press any key to close it.

Thanks for checking on this! :)

Sorry for that, this is a problem with upstream deleting needed branches.

fatal: Remote branch feature/idf-v4.0 not found in upstream origin

Please try using the mainline code:

I will try to keep in mind to create another tag

I cloned the main branch but it won't compile either.
 *  Executing task: C:\Users\Max\.platformio\penv\Scripts\platformio.exe run

Warning! `test_build_project_src` configuration option in section [env:googletest_esp32] is deprecated and will be removed in the next release! Please use `test_build_src` insteadWarning! `src_filter` configuration option in section [env:googletest_esp32] is deprecated and will be removed in the next release! Please use `build_src_filter` instead
Warning! `src_filter` configuration option in section [env:native] is deprecated and will be removed in the next release! Please use `build_src_filter` instead
Processing esp32dev (platform: https://github.com/platformio/platform-espressif32.git#feature/arduino-idf-master; board: esp32dev; framework: arduino)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
patch<EMAIL_ADDRESS>patches\gcc-ar-ranlib.patch
'patch' is not recognized as an internal or external command,
operable program or batch file.
*** Error 1
Exception: Problem while applying platform patches!

See scripts/apply_patches.py for more details!
:
  File "C:\Users\Max\.platformio\penv\lib\site-packages\platformio\builder\main.py", line 186:
    env.SConscript(item, exports="env")
  File "C:\Users\Max\.platformio\packages\tool-scons\scons-local-4.4.0\SCons\Script\SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "C:\Users\Max\.platformio\packages\tool-scons\scons-local-4.4.0\SCons\Script\SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "C:\Users\Max\dev\github\LinakDeskEsp32Controller\scripts\apply_patches.py", line 21:
    raise Exception("Problem while applying platform patches!\n\n"
=========================================================================== [FAILED] Took 14.11 seconds ===========================================================================

Environment    Status    Duration
-------------  --------  ------------
esp32dev       FAILED    00:00:14.108
====================================================================== 1 failed, 0 succeeded in 00:00:14.108 ======================================================================

 *  The terminal process "C:\Users\Max\.platformio\penv\Scripts\platformio.exe 'run'" terminated with exit code: 1.
 *  Terminal will be reused by tasks, press any key to close it.

I got it working by installing patch and adding it to my env path. Then I had to log in and out from my windows session to make it work. My first try was pip install patch which wasn't working.
This was the patch executable I downloaded: https://sourceforge.net/projects/gnuwin32/
Thankyou for your help. If their is any more help from my side needed to get everything more streamlined please let me know.

Great to see that it worked eventually. And this is a good datapoint that the log

See scripts/apply_patches.py for more details!

doesn't work 😄
There is an instruction about manually patching the needed files there:
https://github.com/krzmaz/LinakDeskEsp32Controller/blob/main/scripts/apply_patches.py#L17-L19
I will think about fixing this in the python script maybe.

Yeah, that was good help. :)
Thanks a lot. Unfortuanteley the esp reboot's a lot and I opened a seperate issue for that.
https://github.com/krzmaz/LinakDeskEsp32Controller/issues/17


MiddlewareTCP ipwhitelist weird working for custom entrypoint

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?
EKS v1.25, Traefik v2.9.10
Created custom entrypoint for some TCP service which does not have TLS
Created IngressrouteTCP and MiddlewareTCP with ipWhiteList, but looks like the application is accessible from any IP
What did you see instead?

the application is accessible from any IP if I include in-cluster IPs

    sourceRange:
      # In-cluster IPs
      - <IP_ADDRESS>/22
      - <IP_ADDRESS>/22
      - ip1
      - ip2

I tried with nc and some binary output was there, meaning the answer from service. Works from IP from the list and not from the list
nc -v tcp-service.domain.com 26656
Connection to tcp-service.domain.com port 26656 [tcp/*] succeeded!
"
 ���
    �Kq�PD7?d�|8����=>�5!E8)

the application is not accessible from any IP if I comment in-cluster IPs

    sourceRange:
      # In-cluster IPs
      # - <IP_ADDRESS>/22
      # - <IP_ADDRESS>/22
      - ip1
      - ip2

nc shows the following without binary output, which means it is connected to the node port, but no answer from the application. The answer is the same if use IP from the list and not from it
nc -v tcp-service.domain.com 26656
Connection to tcp-service.domain.com port 26656 [tcp/*] succeeded!

What version of Traefik are you using?
Traefik v2.9.10
What is your environment & configuration?
My Ingressroute and middleware
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: tcp-service
spec:
  entryPoints:
  - tcp-test
  routes:
  - match: HostSNI(`*`) # For TCP without TLS you must use HostSNI(`*`)
    services:
    - name: tcp-p2p
      port: 26656
    middlewares:
      - name: ip-whitelist
  # don't need TLS here
  # tls:
  #   passthrough: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: MiddlewareTCP
metadata:
  name: ip-whitelist
spec:
  ipWhiteList:
    sourceRange:
      # In-cluster IPs
      - <IP_ADDRESS>/22
      - <IP_ADDRESS>/22
      - ip1
      - ip2

My configuration for custom entrypoint (in helm chart values)
additionalArguments:
- --entryPoints.tcp-test.address=:26656/tcp

ports:
  cometbft-test:
    expose: true
    port: 26656
    exposedPort: 26656
    protocol: TCP
     ]

If applicable, please paste the log output in DEBUG level
If in-ckuster IPs in the list traefik just allows connections from any internal IP, but no external ones are logged
{"entryPointName":"tcp-test","level":"debug","msg":"Creating TCP server 0 at <IP_ADDRESS>:26656","routerName":"service-tcp-673acf455cb2dab0b43a@kubernetescrd","serverName":0,"serviceName":"service-tcp-673acf455cb2dab0b43a","time":"2023-10-25T14:52:39Z"}
{"entryPointName":"tcp-test","level":"debug","middlewareName":"service-ip-whitelist@kubernetescrd","middlewareType":"IPWhiteListerTCP","msg":"Creating middleware","routerName":"service-tcp-673acf455cb2dab0b43a@kubernetescrd","time":"2023-10-25T14:52:39Z"}
{"entryPointName":"tcp-test","level":"debug","middlewareName":"service-ip-whitelist@kubernetescrd","middlewareType":"IPWhiteListerTCP","msg":"Setting up IPWhiteLister with sourceRange: [<IP_ADDRESS>/22 <IP_ADDRESS>/22 ip1 ip2]","routerName":"service-tcp-673acf455cb2dab0b43a@kubernetescrd","time":"2023-10-25T14:52:39Z"}
{"entryPointName":"tcp-test","level":"debug","msg":"Adding route for \"HostSNI(`*`)\"","routerName":"service-tcp-673acf455cb2dab0b43a@kubernetescrd","time":"2023-10-25T14:52:39Z"}
{"level":"debug","middlewareName":"service-ip-whitelist@kubernetescrd","middlewareType":"IPWhiteListerTCP","msg":"Connection from <IP_ADDRESS>:61835 accepted","time":"2023-10-25T14:52:39Z"}
{"level":"debug","middlewareName":"service-ip-whitelist@kubernetescrd","middlewareType":"IPWhiteListerTCP","msg":"Connection from <IP_ADDRESS>:30462 accepted","time":"2023-10-25T14:52:39Z"}

If in-cluster IPs commented, traefik rejects all connections regrdless from which IP
{"entryPointName":"tcp-test","level":"debug","msg":"Creating TCP server 0 at <IP_ADDRESS>:26656","routerName":"service-tcp-673acf455cb2dab0b43a@kubernetescrd","serverName":0,"serviceName":"service-tcp-673acf455cb2dab0b43a","time":"2023-10-25T15:07:14Z"}
{"entryPointName":"tcp-test","level":"debug","middlewareName":"service-ip-whitelist@kubernetescrd","middlewareType":"IPWhiteListerTCP","msg":"Creating middleware","routerName":"service-tcp-673acf455cb2dab0b43a@kubernetescrd","time":"2023-10-25T15:07:14Z"}
{"entryPointName":"tcp-test","level":"debug","middlewareName":"service-ip-whitelist@kubernetescrd","middlewareType":"IPWhiteListerTCP","msg":"Setting up IPWhiteLister with sourceRange: [ip1 ip2]","routerName":"service-tcp-673acf455cb2dab0b43a@kubernetescrd","time":"2023-10-25T15:07:14Z"}
{"entryPointName":"tcp-test","level":"debug","msg":"Adding route for \"HostSNI(`*`)\"","routerName":"service-tcp-673acf455cb2dab0b43a@kubernetescrd","time":"2023-10-25T15:07:14Z"}
{"level":"error","middlewareName":"service-ip-whitelist@kubernetescrd","middlewareType":"IPWhiteListerTCP","msg":"Connection from <IP_ADDRESS>:51480 rejected: \"<IP_ADDRESS>:51480\" matched none of the trusted IPs","time":"2023-10-25T15:07:45Z"}
{"level":"error","middlewareName":"service-ip-whitelist@kubernetescrd","middlewareType":"IPWhiteListerTCP","msg":"Connection from <IP_ADDRESS>:27968 rejected: \"<IP_ADDRESS>:27968\" matched none of the trusted IPs","time":"2023-10-25T15:07:46Z"}

Hello @tkatrichenko,
Thanks for reporting this and your interest in Traefik!
If you are using a load-balancer in front of the Traefik service, which seems to be the case, then Traefik is always reached by the load-balancer IP.
To be able to filter connections on the client IP, one should use [proxy-protocol] (https://doc.traefik.io/traefik/routing/entrypoints/#proxyprotocol) configuration.
For the AWS side, here is some documentation.

Hello @rtribotte , thanks for the answer.
Yes, AWS Classic LB is in front of Traefik having cidr_blocks = ["<IP_ADDRESS>/0"] in its SG for this port.
I also annotating traefik service with
service:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'

When I add the following to traefik helm values
    forwardedHeaders:
     trustedIPs: [
       ip1,
       ip2
      # In-cluster IPs
      <IP_ADDRESS>/22
      <IP_ADDRESS>/22
     ]
    # Enable the Proxy Protocol header parsing for the entry point
    proxyProtocol:
     trustedIPs: [
       ip1,
       ip2
      # In-cluster IPs
      <IP_ADDRESS>/22
      <IP_ADDRESS>/22
    ]

And not touching middlewareTCP created earlier. Then scenario 2 happens, no traffik goes to the service from any IP.
But no logs with rejected status...

Hello @tkatrichenko,
After discussing this issue with other maintainers, we still think that's a question.
To keep the repository focused, we ask that all questions be discussed in the community forum.
So please, open a thread to allow moving forward on this topic.


Squaring for MM

Montgomery multiplication consists of 2 parts: big multiplication followed by reduction. In several cases the multiplication could be replaced by more efficient squaring algorithm, that preserves some already computed values.

Will do.

As more code gets added to P256, it would really be good to see more #set-options "--fuel 0 --ifuel 0" make their way into your codebase, along with adequate usage of #push-options and #pop-options (as opposed to #reset-options). This is something that was requested by Santiago and I on the previous pull request.
For instance, the first file that appears in the diff uses rlimit 1000: https://github.com/project-everest/hacl-star/pull/254/files#diff-d9fbb714d22989cbd15bd967a6a84396R597
Here are the reasons why I'm nervous about this:

rlimit 1000 means that z3 is having a genuinely hard time discharging the proof obligation; this proof is likely fragile or brittle and might break in the future, and will be very hard to fix for someone not familiar with the proof
without fuel and ifuel zero, proofs are slower because z3 explores many inversions / unrollings -- as more code arrives in HACL*, this means slower builds
there is no documentation on what is the expected level of fuel or ifuel for any given proof, and this means that if a proof breaks and someone needs to fix it, they can't tell if the proof was meant to work with fuel/ifuel 0, or with some other value -- this means more struggle waiting to happen in the future.

(Side-note: my preferred style is to never use fuel or ifuel, relying on local calls to allow_inversion for ifuel and normalization-based unrollings for fuel, a trick I learned from Santiago... I wrote a bunch about this topic in the upcoming refresh of the Low* tutorial, see for instance https://github.com/FStarLang/kremlin/blob/protz_tutorial/book/tutorial/specs/Spec.Bignum.fst#L42 or https://github.com/FStarLang/kremlin/blob/protz_tutorial/book/tutorial/code/Impl.Bignum.fst#L99 or https://github.com/FStarLang/kremlin/blob/protz_tutorial/book/tutorial/code/Impl.Bignum.fst#L149 for a preview)
Thanks,
Jonathan

I strongly support @protz suggestion.
This is a larger issue with the P-256 codebase, not specific to this PR.
I've been setting "--fuel 0 --ifuel 0" and reasonable --z3rlimit for files I had to update, but there are many others that still use the default or even greater fuel settings, mostly for the sake of retrying fragile proofs. This can now be better achieved using --quake n, although this is still suboptimal,

I agree in principle, and I am trying to be more disciplined about this in my own code.
@protz: the new Low* tutorial is certainly going to help, since may of the current HACL* developers
evolved their programming style without this guidance.
More generally, though, it sounds like we’re saying that the F* defaults are just plain wrong for Low* code.

Maybe we would like all files to have fuel/ifuel set to 0 and change this for specific functions?
Could we offer generic lemmas that should be called for unrolling lemmas/inverting types, instead
of relying on obscure-sounding F* options?

——
The rest of this comment is a more philosophical point that is unrelated to the current issue:
At an even higher level, I am worried that the programming style we are promoting for Low*,
even with the new tutorial, is focused too much on pragmatism and less on user experience.
We are telling peoplt that if you have no idea why F*/Z3 is doing something, do a step by step
proof using calc, often for trivial equalities. I really dislike this direction and think that asking
people to use super-detailed proofs is a little bit of a failure in our methodology. At that stage,
many people would rather switch to Coq.

On 16 Mar 2020, at 18:05, Santiago Zanella-Beguelin<EMAIL_ADDRESS>wrote:
I strongly support @protz https://github.com/protz suggestion.
This is a larger issue with the P-256 codebase, not specific to this PR.
I've been setting "--fuel 0 --ifuel 0" and reasonable --z3rlimit for files I had to update, but there are many others that still use the default or even greater fuel settings, mostly for the sake of retrying fragile proofs. This can now be better achieved using --quake n, although this is still suboptimal,
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub https://github.com/project-everest/hacl-star/pull/254#issuecomment-599653205, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDABHQGMPKLLCEWXVYZLZLRHZL4RANCNFSM4LMFLL3A.

FWIW, I tend to agree with @prosecco's philosophical point.  One of the things that attracted me to software verification (in the form of Dafny at the time) and made it seem usable was that function definitions "just worked", unlike in, say TLA+ or Coq, where everything has to be unfolded explicitly.  Having lived in the automatic world for quite a while (in both Dafny and F*), when I had to move to fuel 0, ifuel 0 settings to interact with the HACL* specs, it led to some rather unintuitive proof failures, since my mental model didn't account for the lack of fuel.
That said, using rlimits of 1000+ is also a bad habit, as @msprotz notes, since having fragile proofs like that are just mines waiting to be hit later.  It would be nice to be able to produce stable proofs without giving up on automation.

Thanks for the comments!
I will close the pull request for now and open it when the suggested changes will be done :)

I don't believe @msprotz is suggesting giving up on automation. That isn't my suggestion either.
A basic knowledge of the concepts of fuel and ifuel and how F* interacts with Z3 goes a long way, and probably can be summarized in a couple of paragraphs.
Insisting on always using --fuel 0 --ifuel 0 would be a bad idea, but asking F* programmers to develop an intuition of why they need to use a higher setting, and asking them to use it sparingly and knowingly in my opinion improves the programmer experience with F* besides improving proof stability and maintainability.

Reopened the PR to keep the discussion alive.
Also this is not an issue with this PR, so I think that it could be merged now.

There's several points being brought up in this discussion, let me try to summarize.

Yes, we should adopt better defaults and discipline ourselves to always use fuel/ifuel 0 by default, if only for pragmatic reasons. Let's hope we can convey that to current and future users of Low*. I agree that it's not ideal but that's the reality of working with a large-scale proof development in the presence of a large SMT context. (I was entirely pragmatic in my comment, and I don't argue for giving up on automation; rather, I'm arguing that a precise understanding is likely to give you more automation rather than a hard-to-diagnose failure followed by a very manual proof.)

I agree that having nicer lemmas to selectively allow inversion and unrolling of functions would go a long way. Documentation is also important -- I don't know where allow_inversion is documented, for instance. Having an equivalent of allow_inversion (e.g. allow_unrolling) would also be excellent, and has been proposed on Slack by @Chris-Hawblitzel (https://everestexpedition.slack.com/archives/C0N8CV6NA/p1583372040038400)

There are larger philosophical questions, i.e. why do we have to be so explicit with unrolling/inversion, or with calc proofs, when one could reasonably hope to rely on automation, since it's the promise of F* somewhat. This is a long-running concern, and has multiple explanations. Some, stemming from underlying limitations of the SMT solver (z3 just doesn't work for non-linear arithmetic, period) -- maybe we should forbid non-linear arithmetic (like Vale does) in HACL* code and demand that anything be proven with the (now much-improved) ring tactic? Some other limitations are open problems, i.e. is there any way to automatically figure out which definitions should be unrolled for a given proof (I doubt it?), or can something be done to improve the SMT context, or the performance of z3...?


Added reset method

This is required when the same form is shared across different pages.

@rohitdatta thanks for merging it! When are you planning to release a new version? I'd like to use it in my project. Thanks!

@gazpachu we realized there’s an issue preventing it from building. Also I don’t think it’s returning the reset method, so a component couldn’t call it.
We’ve added a task to our engineering calendar to take a look, although you can also submit a new patch if you’re familiar with how to fix it.


Throw friendly errors

Fixes #331
It would be nice if we were able to adapt the message when you attempt to run for distance with a negative sign, but commands are read-only and I didn't see an easy way to pipe that through. Any suggestions for other properties?
robot@ev3dev:~$ python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ev3dev.ev3 as ev3
>>> m = ev3.Motor()
>>> m.connected
True
>>> m.speed_sp = -100000
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 210, in _set_attribute
    attribute.write(value.encode())
OSError: [Errno 22] Invalid argument

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 604, in speed_sp
    self._speed_sp = self.set_attr_int(self._speed_sp, 'speed_sp', value)
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 238, in set_attr_int
    return self._set_attribute(attribute, name, str(int(value)))
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 213, in _set_attribute
    self._raise_friendly_access_error(ex, name)
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 229, in _raise_friendly_access_error
    raise ValueError("The given speed value was out of range. Max speed: +/-" + str(max_speed)) from driver_error
ValueError: The given speed value was out of range. Max speed: +/-1050
>>> m.speed_sp = 1000000
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 210, in _set_attribute
    attribute.write(value.encode())
OSError: [Errno 22] Invalid argument

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 604, in speed_sp
    self._speed_sp = self.set_attr_int(self._speed_sp, 'speed_sp', value)
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 238, in set_attr_int
    return self._set_attribute(attribute, name, str(int(value)))
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 213, in _set_attribute
    self._raise_friendly_access_error(ex, name)
  File "/usr/lib/python3/dist-packages/ev3dev/core.py", line 229, in _raise_friendly_access_error
    raise ValueError("The given speed value was out of range. Max speed: +/-" + str(max_speed)) from driver_error
ValueError: The given speed value was out of range. Max speed: +/-1050

Another potential candidate for friendlyifying:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "core.py", line 487, in position
  File "core.py", line 210, in get_attr_int
  File "core.py", line 192, in _get_attribute
OSError: [Errno 19] ENODEV

(instantiate motor, disconnect motor, access attribute... BAM! error.)


feat: Unstructured splitter

Added:
UnstructuredSemanticSplitter
Bug fixes for finding split index
Statistics of the splits

Need more testing


no ./src directory when npm install baguettebox.js

Why is there no ./src directory when installing through npm?

Because it's the src and not the dist files.
@feimosi: that is why I insisted we need to have a common dist folder with unminified files too.

@milose as it was said, src/ is excluded from the npm package. Why do you need source code of your dependencies?
@XhmikosR will do that soon

@feimosi I need it to include the scss files in to my main sass file:
@import 'node_modules/baguettebox.js/src/baguetteBox'

Then I can override classes and settings, without touching the original package :)

@milose I understand, v1.6.3 should satisfy your needs  :)


proposal: x/time/rate: add tokens apart of time

What did you do?
I'm trying to add tokens to the bucket apart of the time so this can help to have distributed rate-limiting so that another process can update tokens in the other one if the token is used there.
What did you expect to see?
There can be a function called AddToken that let you add tokens to the bucket apart of the time, but because the scenarios that anyone can have are different (like using AllowN and not waiting to fill the tokens again) there should be also a function called Tokens to return the current tokens apart of time so based on the current tokens there can be a possibility to decrease the tokens apart of the time.
// AddTokens will decrease the remaining tokens by n
func (lim *Limiter) AddTokens(n float64) {
	lim.tokens -= n
}

// Tokens will return the current Limiter tokens apart of time
func (lim *Limiter) Tokens() float64 {
	return lim.tokens
}

func (lim *Limiter) AddToken() {
	lim.AddTokens(1)
}

If it makes sense I can propose my change for it. I've tested it locally and I could achieve my goal with it!

cc @Sajmani

I'm sorry but I do not understand this proposal. There is an accepted proposal to expose Tokens, but not one to change the number of tokens directly:
https://github.com/golang/go/issues/50035#issuecomment-1013416864

@Sajmani well actually as I described above I wanted to add tokens to the bucket (limiter) apart of time which means for example If I have two threads that are using different limiters they can update each other by adding a token to their bucket (limiter).
Do you have any better idea to add tokens to the limiter apart from the time? because if you use AllowN or other functions they are based on the time so if another thread wants to add tokens it won't make sense.
A good example is to assume you have two HTTP servers and you want to rate-limit requests. Both of them are active and exposed to the user so they should have the same value of current req/s for rate-limiting otherwise user can heat the API double than what it should be.
With this proposal, we can add tokens to the buckets from other threads/processes so it can help to solve this problem.

Is the goal to enable balancing of rates across multiple instances of rate.Limiter (possibly in multiple processes)?

@Sajmani Actually the goal is to replicate the rates to the multiple instances of rate.Limiter by making fake tokens, so it can assume that there are more tokens gone without needing to call AllowN/ReserveN that updates the tokens by time durations.

Are you replicating the rates or the rate limits? SetLimit and SetBurst allow you to sync the rate limits across multiple rate limiters.

@Sajmani I'm not going to sync the limits and bursts with the rate.Limiter instances. I'm going to replicate/aggregate the current tokens (rate) between the rate.Limiter so if one token is gone from one rate.Limiter, it would be gone from the others as well.
So in this case no matter the user is hitting which rate.Limiter, all have the same consumed tokens (rate).

Ah, got it! So you have per-user rate limiters on multiple servers, and you want to deduct tokens on all servers whenever the user consumes tokens on any of the servers.
In theory if the user's requests were distributed evenly across the servers, then you could just set the per-user rate at each server to limit/(number of servers) and not do any synchronization.
But if you're using a load-balancing policy that's not evenly distributed (or intentionally keeps users on a single server for improved locality, cache use, DB client reuse, etc), then they will only consume tokens at the servers they're using.
If I'm understanding this correctly, the problem mainly occurs when the user's requests are distributed among a subset of K servers, is that right?
If so, wouldn't it be more efficient to update the rate at each of the servers that the users' requests are hitting to limit/K? This should involve a lot less synchronization than updating the tokens at each server, and you can use the existing SetLimit functionality.

the problem mainly occurs when the user's requests are distributed among a subset of K servers, is that right?

That is correct.

If so, wouldn't it be more efficient to update the rate at each of the servers that the users' requests are hitting to limit/K?

@Sajmani I didn't get that clearly what you mean. If you mean to stick a user to a server, actually, for scaling and being HA I'm going to expose 3 (n) load balancers to the user so a single client (source IP) can for example do 10k req/s and is being distributed (with some DNS tricks that let's skip this part =) ) to all of my load balancers (each 3.3k req/s). Now all of my load-balancers should be aware that the client (source IP) is doing 10k req/s not (3.3k req/s - or in some scenarios 1k on one server 3k on the other and 6k on the third one).
How can it be possible with SetLimit? In my implementation I'm setting the limit to 10k so how can setting a new limit help here? Can you provide me with an example, please?

ping @Sajmani =)

Any updates here?  We have some use cases for getting current number of tokens for observability reasons

@blevz I think that's covered in #50035

@clwluvw Unfortunately, I really don't understand your use case. The approved proposal #50035 will add the Tokens accessor, but AddTokens does not seem like an appropriate addition. If you need it, please consider forking this package and modifying it yourself.

Related: #68677 asks for being able to sync available tokens with external state.


Is v1 > v2 a breaking change?

I can not see anything in the changes stating it's a breaking change

Some time ago, we made changes to our lua.scripts and were concerned it might break something. Fortunately, no one has been affected.