feat: allow highlighting of ion-item

Allow functionality of ui-sref-active="current" on ion-item

Greetings @awebdeveloper!
My sensors indicate that you need to update your issue through our custom issue form. We are now requiring all issues to be submitted this way, to ensure that we have all of the information necessary to fix them as quickly as possible.
Click Here To Update Your Issue
I will have no choice but to close this issue if it is not resubmitted through the form. Please fill out the rest of the form, so that I may use my friendly robot powers to assist you.
Thank you!

If can write your own logic to "activated" the item ..
Just add the class: activated to item.
Example:
<ion-item ng-repeat="item in itens" ng-class="{'activated': (item.current)}"> </ion-tem>

This is already possible.
http://codepen.io/mhartington/pen/YXyZKR

Thanks. This is missing in docs. Plz update it.
On 05-May-2015 8:57 pm, "Mike Hartington"<EMAIL_ADDRESS>wrote:

This is already possible.
http://codepen.io/mhartington/pen/YXyZKR
--
Reply to this email directly or view it on GitHub
https://github.com/driftyco/ionic/issues/3644#issuecomment-99114515.


Modal not showing, because bootstrap renamed fade.in to fade.active

As the title says, i have downloaded the latest version of bootstrap 4 from the v4-dev branch along with the latest version of ng2-bootstrap and none of my modal windows are displaying.  After checking the elements, the modal window is there, but the fade.in css class has been removed from bootstrap and has been renamed to fade.active
.fade.in {
opacity: 1;
}
CHANGED TO
.fade.active {
opacity: 1;
}

Nice reason for small PR changing bs4 template

Happy to make the PR, but question.  Readme says that this should all work bs3 and bs4.  Is there anywhere that the user sets this, as to whether they are using bs3 or bs4, or do you just add the classes for both when applying css to elements?


Route updates are delayed with large number of services

We have a large number of services > 2500.  The watchBackend func is taking over 15-20 seconds due to the sequential service queries.   I am proposing an update to the below code to be a go func with worker pools.
https://github.com/fabiolb/fabio/blob/master/registry/consul/service.go#L57
I will PR this hopefully early next week.

Closing with #564

I'll keep this open as the issue which is the motivation for the PR.


Model fitting produces NaNs for certain inputs

First of all I want to thank you for the library and a great work you did in order to make it work with very efficient implementation.
While using the library I noticed that some of the data that I'm using can converge to NaN values. I managed to find an example that converges to NaN in two iterations
Input json:
{
  "search_keys": {
    "search_term": "xyz"
  },
  "judgment_keys": [
    {
      "session": [
        {"click": 0, "purchase": 0, "doc": "doc0"},
        {"click": 0, "purchase": 0, "doc": "doc1"},
        {"click": 0, "purchase": 0, "doc": "doc2"},
        {"click": 1, "purchase": 1, "doc": "doc3"},
        {"click": 0, "purchase": 0, "doc": "doc4"},
        {"click": 0, "purchase": 0, "doc": "doc5"},
        {"click": 0, "purchase": 0, "doc": "doc6"},
        {"click": 0, "purchase": 0, "doc": "doc7"},
        {"click": 0, "purchase": 0, "doc": "doc8"},
        {"click": 0, "purchase": 0, "doc": "doc9"},
        {"click": 0, "purchase": 0, "doc": "doc10"},
        {"click": 0, "purchase": 0, "doc": "doc11"},
        {"click": 1, "purchase": 1, "doc": "doc12"},
        {"click": 0, "purchase": 0, "doc": "doc13"},
        {"click": 1, "purchase": 0, "doc": "doc14"},
        {"click": 0, "purchase": 0, "doc": "doc15"},
        {"click": 0, "purchase": 0, "doc": "doc16"},
        {"click": 0, "purchase": 0, "doc": "doc17"},
        {"click": 0, "purchase": 0, "doc": "doc18"},
        {"click": 0, "purchase": 0, "doc": "doc19"}
      ]
    }
  ]
}

Result after one iterations
{
  "search_term:xyz": {
    "doc18": 0.32974863052368164,
    "doc17": 0.21041131019592285,
    "doc13": 0.1666666716337204,
    "doc12": 0.3333333432674408,
    "doc11": 0.1666666716337204,
    "doc8": 0.1666666716337204,
    "doc16": 0.23937760293483734,
    "doc7": 0.1666666716337204,
    "doc14": 0.26920539140701294,
    "doc3": 0.3333333432674408,
    "doc5": 0.1666666716337204,
    "doc4": 0.1666666716337204,
    "doc9": 0.1666666716337204,
    "doc19": 0.3070479929447174,
    "doc15": 0.2477390617132187,
    "doc0": 0.1666666716337204,
    "doc2": 0.1666666716337204,
    "doc10": 0.1666666716337204,
    "doc6": 0.1666666716337204,
    "doc1": 0.1666666716337204
  }
}

Result after two iterations:
{
  "search_term:xyz": {
    "doc18": null,
    "doc17": null,
    "doc13": 0.1666666716337204,
    "doc12": 0.3333333432674408,
    "doc11": 0.1666666716337204,
    "doc8": 0.1666666716337204,
    "doc16": null,
    "doc7": 0.1666666716337204,
    "doc14": null,
    "doc3": 0.3333333432674408,
    "doc5": 0.1666666716337204,
    "doc4": 0.1666666716337204,
    "doc9": 0.1666666716337204,
    "doc19": null,
    "doc15": null,
    "doc0": 0.1666666716337204,
    "doc2": 0.1666666716337204,
    "doc10": 0.1666666716337204,
    "doc6": 0.1666666716337204,
    "doc1": 0.1666666716337204
  }
}

It might be just a coincidence, but it looks like nulls only appear in the later documents (from 15th to 20th) and the first null appears at position 15 which is the place where last final happened.
I would appreciate if you can help me to identify the problem.
Let me know if I can provide you with more information.

Hi @itdxer ,
Thanks for opening this issue. This is indeed a weird behavior, it does seem to be a bug. I'll further debug it and reply back as soon as I figure out what's happening.

Hi @itdxer ,
I've finally debugged the issue and made a few changes.
So what's happening here is that your input data contains two distinct documents that had a purchase (doc3 and doc12). This unfortunately cannot happen due the way the model was implemented. The problem specifically happens in this line of code: once the first purchase is observed then the probability that the customer will continue examining new documents goes to 0 (zero) and therefore when the new purchase happens the result is that it cancels the whole probability making it effectively zero as well.
Later on this value is used as the denominator to find the continuation probability and therefore it gives the error that you observed.
What I did then was to install some exception raising on this step. Unfortunately that's all that can be done in this model as it only allows for 1 purchase.
You can verify if by removing one of the purchases flags if it works on the new code. The new code can be installed with:
pip install -U pyClickModels==0.0.2rc1

And with the new input data it should work.
Please let me know if this fixes this issue for you :)
Best Regards,

Hi,
Thank you for taking a look and adding exception for this use case. Now I can see that that's the case from the model since P(S=1|C=1,P=1)=1 and P(E_r=1|S_{r-1}=1)=0. I'll add some changes, test a new version and I'll let you know if nans completely disappeared or not.

It looks like when I get rid of the repeated purchases problem gets resolved (at least it converges to some values). Although I've encountered another problematic case. It raises exception which you added, but all of the purchases for this example are equal to zero. It looks like it might be failing because there are too many documents in the list (80 to be precise)
Do you think it might happen because list is to large and calculations might get quite unstable?
Exception:
RuntimeError: An error occurred during the optimization process. Please check if your data input files have the correct format, such as each session not having more than 1 sale in total.

Input:
{
  "search_keys": {
    "search_term": "xyz"
  },
  "judgment_keys": [
    {
      "session": [
        {"click": 0, "purchase": 0, "doc": "doc0"},
        {"click": 0, "purchase": 0, "doc": "doc1"},
        {"click": 0, "purchase": 0, "doc": "doc2"},
        {"click": 0, "purchase": 0, "doc": "doc3"},
        {"click": 0, "purchase": 0, "doc": "doc4"},
        {"click": 0, "purchase": 0, "doc": "doc5"},
        {"click": 0, "purchase": 0, "doc": "doc6"},
        {"click": 0, "purchase": 0, "doc": "doc7"},
        {"click": 0, "purchase": 0, "doc": "doc8"},
        {"click": 0, "purchase": 0, "doc": "doc9"},
        {"click": 0, "purchase": 0, "doc": "doc10"},
        {"click": 0, "purchase": 0, "doc": "doc11"},
        {"click": 0, "purchase": 0, "doc": "doc12"},
        {"click": 0, "purchase": 0, "doc": "doc13"},
        {"click": 0, "purchase": 0, "doc": "doc14"},
        {"click": 0, "purchase": 0, "doc": "doc15"},
        {"click": 0, "purchase": 0, "doc": "doc16"},
        {"click": 0, "purchase": 0, "doc": "doc17"},
        {"click": 0, "purchase": 0, "doc": "doc18"},
        {"click": 0, "purchase": 0, "doc": "doc19"},
        {"click": 0, "purchase": 0, "doc": "doc20"},
        {"click": 0, "purchase": 0, "doc": "doc21"},
        {"click": 0, "purchase": 0, "doc": "doc22"},
        {"click": 0, "purchase": 0, "doc": "doc23"},
        {"click": 0, "purchase": 0, "doc": "doc24"},
        {"click": 0, "purchase": 0, "doc": "doc25"},
        {"click": 0, "purchase": 0, "doc": "doc26"},
        {"click": 0, "purchase": 0, "doc": "doc27"},
        {"click": 0, "purchase": 0, "doc": "doc28"},
        {"click": 0, "purchase": 0, "doc": "doc29"},
        {"click": 0, "purchase": 0, "doc": "doc30"},
        {"click": 0, "purchase": 0, "doc": "doc31"},
        {"click": 0, "purchase": 0, "doc": "doc32"},
        {"click": 0, "purchase": 0, "doc": "doc33"},
        {"click": 0, "purchase": 0, "doc": "doc34"},
        {"click": 0, "purchase": 0, "doc": "doc35"},
        {"click": 0, "purchase": 0, "doc": "doc36"},
        {"click": 0, "purchase": 0, "doc": "doc37"},
        {"click": 0, "purchase": 0, "doc": "doc38"},
        {"click": 0, "purchase": 0, "doc": "doc39"},
        {"click": 0, "purchase": 0, "doc": "doc40"},
        {"click": 0, "purchase": 0, "doc": "doc41"},
        {"click": 0, "purchase": 0, "doc": "doc42"},
        {"click": 0, "purchase": 0, "doc": "doc43"},
        {"click": 0, "purchase": 0, "doc": "doc44"},
        {"click": 0, "purchase": 0, "doc": "doc45"},
        {"click": 0, "purchase": 0, "doc": "doc46"},
        {"click": 0, "purchase": 0, "doc": "doc47"},
        {"click": 0, "purchase": 0, "doc": "doc48"},
        {"click": 0, "purchase": 0, "doc": "doc49"},
        {"click": 0, "purchase": 0, "doc": "doc50"},
        {"click": 0, "purchase": 0, "doc": "doc51"},
        {"click": 0, "purchase": 0, "doc": "doc52"},
        {"click": 0, "purchase": 0, "doc": "doc53"},
        {"click": 0, "purchase": 0, "doc": "doc54"},
        {"click": 0, "purchase": 0, "doc": "doc55"},
        {"click": 0, "purchase": 0, "doc": "doc56"},
        {"click": 0, "purchase": 0, "doc": "doc57"},
        {"click": 0, "purchase": 0, "doc": "doc58"},
        {"click": 0, "purchase": 0, "doc": "doc59"},
        {"click": 0, "purchase": 0, "doc": "doc60"},
        {"click": 0, "purchase": 0, "doc": "doc61"},
        {"click": 0, "purchase": 0, "doc": "doc62"},
        {"click": 0, "purchase": 0, "doc": "doc63"},
        {"click": 0, "purchase": 0, "doc": "doc64"},
        {"click": 0, "purchase": 0, "doc": "doc65"},
        {"click": 0, "purchase": 0, "doc": "doc66"},
        {"click": 1, "purchase": 0, "doc": "doc67"},
        {"click": 0, "purchase": 0, "doc": "doc68"},
        {"click": 0, "purchase": 0, "doc": "doc69"},
        {"click": 0, "purchase": 0, "doc": "doc70"},
        {"click": 0, "purchase": 0, "doc": "doc71"},
        {"click": 0, "purchase": 0, "doc": "doc72"},
        {"click": 0, "purchase": 0, "doc": "doc73"},
        {"click": 0, "purchase": 0, "doc": "doc74"},
        {"click": 0, "purchase": 0, "doc": "doc75"},
        {"click": 0, "purchase": 0, "doc": "doc76"},
        {"click": 0, "purchase": 0, "doc": "doc77"},
        {"click": 0, "purchase": 0, "doc": "doc78"},
        {"click": 0, "purchase": 0, "doc": "doc79"}
      ]
    }
  ]
}

Hi @itdxer ,
What happened is that with lots of documents the chain probabilities of clicks at initial points ended up being zero (this is due maximum precision limit) and then some instances of the gamma updating denominator will be zero, which hits the Exception installed before.
The unit tests that I set up for the package didn't include this type of data so I didn't realize it.
I decided then to solve these problems by simply not updating gamma on points where the probability is either close to zero (like 10e-45) or zero itself. This means that your previous data containing two purchases now will work but for the second purchased doc the gamma factor simply won't be updated. Also you can send any amount of given documents for each session and it should work just fine.
Probably the proper approach after all would be to install data validation right at the beginning of code execution (something I haven't implemented yet).
I published an rc with the new code:
pip install -U pyClickModels==0.0.2rc2

Please let me know if it's working for you now.
Best Regards,

Thank you again for the quick reply and update. I'll try to test it today and I'll let you know about the results.

It looks like everything works, thank you so much for your help!


[Persistence][Utility] Separate all CreateAndApply functions into more functional components - Issue #508

Description
This PR tends to the deliverables in the parent issue
Issue
Fixes #508
Type of change
Please mark the relevant option(s):

[ ] New feature, functionality or library
[ ] Bug fix
[x] Code health or cleanup
[ ] Major breaking change
[ ] Documentation
[ ] Other

List of changes

WIP

Testing

[x] make develop_test; if any code changes were made
[X] Docker Compose LocalNet; if any major functionality was changed or introduced
[x] k8s LocalNet; if any infrastructure or configuration changes were made

Required Checklist

[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added, or updated, godoc format comments on touched members (see: tip.golang.org/doc/comment)
[ ] I have tested my changes using the available tooling
[ ] I have updated the corresponding CHANGELOG

If Applicable Checklist

[ ] I have updated the corresponding README(s); local and/or global
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added, or updated, mermaid.js diagrams in the corresponding README(s)
[ ] I have added, or updated, documentation and mermaid.js diagrams in shared/docs/* if I updated shared/*README(s)

Sweet

This AI has an Indian accent!! Brilliant! 😍

@Olshansk I believe I tackled everything

Can you just rerun tests & localnet on your machine to make sure that the failure in consensus/e2e_tests/state_sync_test.go was a one-off flaky issue?
Otherwise, just see my suggestion and :shipit:

https://user-images.githubusercontent.com/29378614/230515272-df710d26-8ffe-448f-99fd-87b89098b15c.mp4
👍


Math issue in metric definitions

🐞 Problem

All terms of the metrics in the metric module are divided by 1 / c**2. This is incorrect. I am linking the lines of code here, along with a Wiki page to compare them to:

Schwarzschild - metric.Schwarzschild - Wikipedia
Kerr - metric.Kerr - Wikipedia
Kerr-Newman - metric.KerrNewman - Wikipedia

Metric, g, is defined as ds^2 = c^2 * d\tau^2 = g_{ab}dx^adx^b. But in the code, ds^2 seems to have been equated with d\tau^2, but that cannot be the case, until c is set to 1.
🎯 Goal

To fix mathematical inaccuracies in the package.
💡 Possible solutions

Check the metric definitions and make corresponding changes.
📋  Steps to solve the problem

Comment below about what you've started working on.
Add, commit, push your changes
Submit a pull request and add this in comments - Addresses #<put issue number here>
Ask for a review in comments section of pull request
Celebrate your contribution to this project 🎉

This issue has been confirmed after a short review. Updating the issue description. First part to be fixed in #521. Second in short PR, right after.


Status of QUIC transport

What is the status of QUIC and HTTP over QUIC?
Is now an appropriate time point to visit implementing HTTP over QUIC? (I assume that QUIC is the transport layer)
Thanks

HTTP/3 which is HTTP over QUIC is still being worked on by the IETF and is currently still in draft. We will revisit when once this has been ratified.
I'm closing this as a dupe of #4695


avoid create-or-replace of deployment when scaling

While I understand the rationale for this change, thinking about it some more I think we should try to fix the "race condition" rather than change from a "describe what state we want" (which we have currently) to a "describe the changes we we think we need to reach the wanted state".
One of the reasons I'm saying this, is the following:
If the controller crashes and restarts, it will read the current CRD state of topics and functions and fire fake "create" events. With your changes this will fire createNew API calls, but the deployments are already there!

In other words I think this is kubectl apply vs. kubectl create, and my understanding is that we want to strive for the former

I understand the point for the restart/replay situation, but I don't want to revert to createOrReplaceWithNew for every scale operation. IIUC simply updating the replica count via scale is far lighter (and we need to be as light/responsive as possible for scaling). In fact, I think the apply vs create analogy is the opposite of what you're suggesting, since we are currently creating (replacing) every time, whereas an actual kubectl apply on a Deployment with a different replica count would simply "scale".

for the replay of fake "added" events when a Deployment already exists (e.g. after deleting the function-controller pod and letting it restart automatically), this currently happens on the PR branch:
[function-controller-597283684-49lf9] io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default.svc/apis/extensions/v1beta1/namespaces/default/deployments. Message: deployments.extensions "square" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=extensions, kind=deployments, name=square, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=deployments.extensions "square" already exists, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
So... we definitely do need to address that, but I think ignoring it and logging that it already exists might be sufficient. WDYT @ericbottard?

I think it's fine, this was just an immediate example of what I think could go wrong.
I'd be interested in learning more about the actual implementation of kubectl apply vs kubectl create though. Is it your understanding that apply does a diff clientside?

https://kubernetes.io/docs/tutorials/object-management-kubectl/declarative-object-management-configuration/#how-apply-calculates-differences-and-merges-changes
also in that doc:

Directly update the replicas field in the live configuration by using kubectl scale. This does not use kubectl apply:

kubectl scale deployment/nginx-deployment --replicas=2

I pushed a commit to handle (ignore) the already-exists case on restart/replay.

Merged as 3c4def8

Not the subject of this PR, but I played a bit with kubectl delete while this was running:
Shouldn't a deployment be recreated when a function for it exists? That is, in the same way that k delete po/foo-xxx ends up triggering the recreation of a new pod as long as there is a deployment, shouldn't k delete deploy/square end up being forcibly recreated as long as the function for it exists? @markfisher

Interesting question. I suppose if we consider the Deployment to be an implementation detail of how we deploy Functions, then yes. If someone intervenes manually, we should be able to restore the correct state (as long as the corresponding Function does exist). If so, I think this should be handled by catching the exception for a scale() call when the underlying Deployment no longer exists. In other words, we should still rely on just scale() for the "normal" case.

I like that better. We are already watching Deployment events, so if we see a delete but a Function does still exist for that Deployment, we can call our FunctionDeployer.deploy(). Scaling will kick in from there... so at least the manual deletion does reset the replica count to 0 thus "resetting" the Function overall, as deleting a Pod triggers a reset (new instance).


Redirection after admin login

Is there a way to change the redirection after admin/login.
I would like to redirect to something else than the users/dashboard page.

You need to extend the authcontroller@portadminlogin to use a configuration option. A pr will be appreciated

Closed because of the following pr: https://github.com/intrip/laravel-authentication-acl/pull/129


bug(mat-tabs): When clicking to another tab prev tab content is destroyed without calling a components onDestroy

Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
No response
Description
If you have a component in a mat-tab as content when that tab is no longer in focused the body is removed but the components onDestroy method is not called.  You can observe the content is removed in the dom and https://material.angular.io/components/tabs/overview#keeping-the-tab-content-inside-the-dom-while-its-off-screen supports this notion but I was expecting a components onDestroy method to be called when removed for cleanup purposes (resizeObservers et el)
Reproduction
StackBlitz link: https://stackblitz.com/edit/stackblitz-starters-vntmid?file=src%2Fmain.ts
Steps to reproduce:

Click on any tab.
Observe console log not called from the tab one component

Expected Behavior
Console log called indicating the onDestroy method fired.
Actual Behavior
NA
Environment
Angular CLI: 18.0.1
Node: 18.20.3
Package Manager: npm 10.2.3
OS: linux x64
Angular: 18.0.0
... animations, cdk, common, compiler, compiler-cli, core, forms
... material, platform-browser, router
Package                         Version
@angular-devkit/architect       0.1800.1
@angular-devkit/build-angular   18.0.1
@angular-devkit/core            18.0.1
@angular-devkit/schematics      18.0.1
@angular/cli                    18.0.1
@schematics/angular             18.0.1
rxjs                            7.8.1
typescript                      5.4.5
zone.js                         0.14.6

Wrap your tab-one component with <ng-template matTabContent> and it will work as expected:
<mat-tab label="Second">
  <ng-template matTabContent>
      <app-tab-one/>
  </ng-template>
</mat-tab>

This is not a bug.

Indeed it does. I kinda expected the component to be destroyed when the tab content is removed from the DOM without the ng template and directive. Maybe a documentation note? Thank you! @hakimio


[feature] Support GPU scheduling

Originally posted by @gaocegege in https://github.com/kleveross/klever-model-registry/pull/75#r478819061

/assign @simon-cj


Feature/base data mssql imperative

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.thailu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Remove version: Orwell.Dev-C++ version 5.11

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

URL: https://sourceforge.net/projects/orwelldevcpp/files/Setup Releases/Dev-Cpp 5.11 TDM-GCC 4.9.2 Setup.exe/download
OK
(Automated message - build 923)

Close with reason: Package still available.;


Fix failing macvim installation on rake install

Fix macvim installation failing since homebrew updates to not support options anymore.

Need help to test this on mac since I can't test this using docker-compose because its on linux and not macOS.

Hey @hazz1925, thanks for your contribution! Since I'm not actively using this fork anymore nor macvim (i'm using nvim), I'll wait until someone confirm this works before merging. Also, I have left a comment. Please check it out :-)

@hazz1925
https://github.com/macvim-dev/homebrew-macvim is marked as deprecated in favour of 'native' HomeBrew formula. To keep it simple, my proposal would be to simplify failing command to just
run %{brew install macvim}

This works for me locally for some time already.
If no reply for some time here, I'll create PR by myself.
@lfilho Just out my curioucity, what fork do you use? :)

Merged #827 that takes care fo this. Thanks for your contributions.
@vovinacci Yup, you found it! :) Thanks again for the PR


Signatures still not loaded

Apologies for opening a new issue, but the previous one https://github.com/jaeles-project/jaeles/issues/12 was closed before I had a chance to test the changes.
After pulling the new Jaeles and running a config -a update, still says "No signatures loaded".
Is there a specific directory or environment in which Jaeles should be run? For some reason it's not locating signatures when running a simple scan command.

Is there any signature file in your ~/.jaeles/base-signatures folder?

No, that directory does not exist.
I've even tried manually cloning https://github.com/jaeles-project/jaeles-signatures to ~/.jaeles/base-signatures but still get "No signature loaded".

It's weird. What is your OS and Jaeles version?
Can you delete ~/.jaeles/ folder then run jaeles config -a init --debug too see if it correct clone the signature repo?
For manual install, you need to copy all signature from https://github.com/jaeles-project/jaeles-signatures to ~/.jaeles/base-signatures  then run jaeles config -a reload.
Otherwise, you need to select the signature manually by running jaeles scan -s "/tmp/jaeles-signatures/cves/.*" -u your_url

Ubuntu; even with a reload and all the signatures in ~/.jaeles/base-signatures directory, it says No signature loaded.
However, if I do -s ~/.jaeles/base-signatures/*, it only runs the first yaml signature in alphabetical order. It was expected that it would run all of them in the directory, so maybe this is a feature to implement.
Strange indeed..

You need to do -s ~/.jaeles/base-signatures/.* with the . before  *.
what your output after delete ~/.jaeles/ folder then run jaeles config -a init --debug too see if it correct clone the signature repo?

Yes they appear after removing ~/.jaeles and running init again

I had the same problem.System is windows10

hit this bug too. What is the solution? (Pentoo Linux)
dev-lang/go-1.13.9

You need to delete old config or signatures with this command rm -rf ~/.jaeles/
Then run this command
jaeles config -a update --repo http://github.com/jaeles-project/jaeles-signatures

to reload the default signatures.

done that, did not help. (I'm using the latest git):
blshkv@pt ~ $ rm -rf ~/.jaeles/
blshkv@pt ~ $ jaeles config -a update --repo http://github.com/jaeles-project/jaeles-signatures
bash: jaeles: command not found
blshkv@pt ~ $ ./go/bin/jaeles config -a update --repo http://github.com/jaeles-project/jaeles-signatures
Jaeles beta v0.7 by @j3ssiejjj
[+] Cloning Plugins from: https://github.com/jaeles-project/jaeles-plugins
[+] Cloning Signature from: https://github.com/jaeles-project/jaeles-signatures
Enumerating objects: 55, done.
Counting objects: 100% (55/55), done.
Compressing objects: 100% (46/46), done.
Total 55 (delta 7), reused 32 (delta 3), pack-reused 0
[+] Cloning Plugins from: https://github.com/jaeles-project/jaeles-plugins
[+] Cloning Signature from: http://github.com/jaeles-project/jaeles-signatures
blshkv@pt ~ $ ./go/bin/jaeles scan -s 'jira' -u target.com
Jaeles beta v0.7 by @j3ssiejjj
[Error] No signature loaded
blshkv@pt ~ $

this is weird, but "~/.jaeles/base-signatures/ folder is getting deleted after the update command. I can see it while command is still running. What's going on?

The workaround is to specify a full path:
./go/bin/jaeles scan -s '/home/blshkv/.jaeles/base-signatures/.*' -u target

Try to clone signatures folder to somewhere like this
git clone https://github.com/jaeles-project/jaeles-signatures /tmp/jaeles-signatures/

then reload them in the DB with this command.
jaeles config -a reload --signDir /tmp/jaeles-signatures

can you fix that so it works by default without all these woodoo please? There is some logic error in the code too, since it does not create/update required files if the home directory exists.

I'm stuck with the same error:
Jaeles beta v0.9 by @j3ssiejjj
[Error] No signature loaded
Even after following the

Try to clone signatures folder to somewhere like this
git clone https://github.com/jaeles-project/jaeles-signatures /tmp/jaeles-signatures/

then reload them in the DB with this command.
jaeles config -a reload --signDir /tmp/jaeles-signatures

Please suggest...

had the same problem. tried all the suggestion but still.


Python Bindings: Numpy Support

@salasoom and @lindstro: just wanted to post this PR to update you on the progress of the python bindings.  Support for compression/decompression from/to numpy arrays has been added, as well as a few tests to verify that a "round-trip" of compression/decompression works.  @salasoom, if you have some time before the holidays, it would be great to discuss what other tests would be beneficial.
TODO (updated 03/24/2019):

[x] wrap main zfp functions and testing utility functions
[x] add tests and add to Travis CI
[x] floating-point (de-)compression for 1->4D matches known-good checksums
[x] write_header argument for compress_numpy
[x] add support for arbitrary strides rather than rearranging the array into fortran-contiguous
[x] integer (de-)compression matches the known-good checksums
[x] wrap the random strided array functions in test/utils and add to python tests
[x] support decompression into a pre-allocated buffer
[x] roll back changes to travis.sh
[x] release GIL before calls to compress/decompress
[x] update Cython flags to embed function signatures and docstrings
[ ] documentation

[ ] dependencies, configuration, compilation, testing, and installation
[ ] API docs and example usage

[ ] support for lossless compression

Oh. It's probably worth mentioning that the bindings aren't built by default.  You have to provide -DBUILD_PTYHON on the cmake line to build them.
TODO: add a python test to travis

@SteVwonder This is fantastic!  I'm up against a paper deadline this week but either I and/or @salasoom will take a closer look over the next few days.

Can you base the PR from the develop branch? We exclude documentation and 99% of tests when making releases on master.
Now that we are finishing bindings in a couple languages, we may mess around with the directory structure in the next release. For now, please separate the tests into dir tests/python.
When on develop, you'll see all the tests (and docs). For testing array compression/decompression, we generate ~million element arrays and pass them into the compressor, and then checksum the resulting bitstream and decompressed array (once decompressed). You can find C libs in tests/utils for generating the smooth random arrays, and for checksum-ing resulting memory (hash32/64). Calling those from Python would seem to be the way to go. We compare checksums against what's stored in tests/constants (macro-defined constants).
I've been meaning to refactor some of those utility libraries, so your input can help re-shape those APIs to be the least painful when calling from Python (ex. removing double pointers).
As far as benchmarking goes, that's something we've been meaning to get around to, an executable we can run on local machines, because we see a large range of test runtimes on Travis and Appveyor. For now, we time how long compression and decompression take for those million element arrays, and print them out as part of the test. At least then we have a history of CI log output with some timings.

@SteVwonder I've had a chance to skim your PR.  I like a lot of what I'm seeing.  Great job!
I also have a few comments and concerns:
compress_numpy (which is perhaps the most important function) supports fixed-accuracy mode only, while decompress_numpy supports all modes.  Although we have to start somewhere, this is a significant limitation.  While we support only fixed-rate compression in the CUDA version, there are good technical reasons for this having to do with lack of knowledge where in the compressed stream each thread should start.  I know we'll be making changes to the Python API before the release, but I wanted to point out that we'll have to change this function to support the other modes.  For now, it might be best to rename the function to make it explicit that it only handles the fixed-accuracy mode.  On the other hand, only a single line (zfp_stream_set_accuracy) has to change to support fixed-precision and fixed-rate also.  Alternatively, we might consider passing in an object that encodes the compression mode, which is essentially what zfp_stream does, and then add three more short functions for setting the compression parameters.  I think Markus or I can make this change.
I don't think we should have a default tolerance.  The user should be aware that this is something they need to specify.  If they don't know how to set this, then I think it would be safer to use a tolerance of zero.
Another difference wrt. zfp_compress is that only one field can be compressed at a time.  That might be fine for the Python interface since one can simply make multiple calls and concatenate the compressed byte strings.  However, this does cause an incompatibility with files compressed via the C interface, as zfp_compress does not insert a header between each field.  This is something worth documenting at least.
Does Cython expose internal implementation functions like _init_field in the global namespace?  That would violate the ANSI standard, which reserves all public symbols starting with _.
class Memory(): Maybe check the return value of malloc?
The return value to zfp_decompress is not checked.  It will be zero in case of failure.  Maybe throw an exception?
One potential concern: By default, zfp's compressed data is in multiples of 64-bit "words."  Are there any concerns that Python might pass an unaligned array of bytes to decompress_numpy?  That could cause issues on some platforms.
Regarding the changes to the LICENSE, LLNL has gotten picky about what can go in this file: see https://software.llnl.gov/about/licenses/.  We may have to update the license and place this additional information in the python directory.

Codecov Report

Merging #29 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop      #29   +/-   ##
========================================
  Coverage    92.43%   92.43%
========================================
  Files           46       46
  Lines         2248     2248
========================================
  Hits          2078     2078
  Misses         170      170

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 24ee287...9587f29. Read the comment docs.

@SteVwonder Currently, there's an assumption that the numpy array uses C ordering, which may or may not be true.  I think it would be safer to call zfp_set_stride_* in _init_field by translating the ndarray.strides byte strides to zfp's scalar strides.  See this discussion regarding this issue: https://github.com/zarr-developers/numcodecs/pull/160.

@lindstro, I looked into the type and exec problems reported in zarr-developers/numcodecs#160.  I had no issues with referencing the type field in the zfp_field struct (it is already included in this PRs commits), but I did have an issue referencing the exec field in the zfp_stream struct (Cython would error out with "invalid syntax").
I haven't found a general way to reference fields/variables in C that overlap with reserved keywords in Python/Cython. Ultimately though, I don't think this is much of an issue for zfp.  Cython does not require that every field in a struct be listed.  It only requires those fields that you need to directly access from Cython code.  So we can just omit exec from the zfp_stream definition in Cython and then set exec via the zfp_stream_set_execution function.  If we need to reference the exec field directly from Cython/Python for some reason, we might need to change the name in C.

@SteVwonder I'm fine with that for the current Python implementation.  I still think we should take this into consideration and change C struct member names for the next release to avoid issues like this.
With regards to the type field, is it possible that the behavior differs between Python 2 and 3, or did you test both?

Just pushed a commit adding support for fixed-rate and fixed-precision modes.  The compress_numpy function now has two additional "optional" arguments.  Exactly one of the "optional" arguments (i.e., rate, precision, tolerance) must be provided by the user, otherwise an exception is thrown.  This eliminates the need for having three separate functions, but it is a bit odd to require an "optional" argument.  As a quick sanity check, is -1 a valid value for rate or tolerance? (I'm currently using -1 as the default value for all three optional arguments under the assumption its an invalid value).
Hopefully this works for the EOY milestone, and we can come up with a better interface for the March release.  Also, same caveat as before that the current python test cases just verify that nothing catastrophic happens when doing a round-trip compression/decompression using the bindings.

I've been meaning to refactor some of those utility libraries, so your input can help re-shape those APIs to be the least painful when calling from Python (ex. removing double pointers).

After peeking at the utility libraries, I think I have a few ideas on how to make them more usable from Python. @salasoom, can we meet in person after the holidays to go over these tests a bit more in-depth and brainstorm?

Can you base the PR from the develop branch?
For now, please separate the tests into dir tests/python

👍 Done.

I don't think we should have a default tolerance. The user should be aware that this is something they need to specify. If they don't know how to set this, then I think it would be safer to use a tolerance of zero.

Makes sense.  Removed the default tolerance as part of the fixed-rate/precision commit.

Another difference wrt. zfp_compress is that only one field can be compressed at a time.

That's a good point.  I can document that as a limitation of the current numpy-specific interface.  Would adding a boolean output_header argument that toggles the writing of the header make it compatible with files compressed via the C interface?

Does Cython expose internal implementation functions like _init_field in the global namespace? That would violate the ANSI standard, which reserves all public symbols starting with _.

__init__ doesn't get exported as a symbol in the shared library, but other functions that start with _ are exported, like _PyBytes_Type and ___pyx_module_is_main_zfp.  I don't think this should be an issue since only Python will be dlopening the shared library, but maybe I'm missing something.

class Memory(): Maybe check the return value of malloc?
The return value to zfp_decompress is not checked. It will be zero in case of failure. Maybe throw an exception?

Good ideas.  Both are done.

I still think we should take this into consideration and change C struct member names for the next release to avoid issues like this.

👍 sounds good to me.  Also, I spoke too soon. It does appear that Cython has support for struct members whose names conflict with python keywords (cython reference). I believe the follow should work (will try out tomorrow).
ctypedef struct zfp_stream:
    zfp_exec _exec "exec"

With regards to the type field, is it possible that the behavior differs between Python 2 and 3, or did you test both?

That is possible.  I have only been able to test against python 3 while on my mac. I can test against both py2 & 3 on Linux tomorrow.

@SteVwonder Well done!  I have a few comments below.

Just pushed a commit adding support for fixed-rate and fixed-precision modes. The compress_numpy function now has two additional "optional" arguments. Exactly one of the "optional" arguments (i.e., rate, precision, tolerance) must be provided by the user, otherwise an exception is thrown. This eliminates the need for having three separate functions, but it is a bit odd to require an "optional" argument. As a quick sanity check, is -1 a valid value for rate or tolerance? (I'm currently using -1 as the default value for all three optional arguments under the assumption its an invalid value).

Thanks--I think this is fine for now.  -1 is not valid for any of the three parameters, so that'll work.  But I agree that we need a cleaner interface for the release.

Hopefully this works for the EOY milestone, and we can come up with a better interface for the March release. Also, same caveat as before that the current python test cases just verify that nothing catastrophic happens when doing a round-trip compression/decompression using the bindings.

OK.  I'm still concerned about the memory layout of the numpy array.  Did you get a chance to look into using strides?  I believe asfortranarray actually permutes the array, which could be expensive and unnecessary.  Instead, I would suggest mapping shape[3] to nx, shape[2] to ny, and so on.  This corresponds to the default ndarray layout and presumably the most common case.  Then set the zfp strides to guard against other layouts.  This ensures that in the most common case, we will traverse the numpy array in sequential order, and no deep copies are needed.

I've been meaning to refactor some of those utility libraries, so your input can help re-shape those APIs to be the least painful when calling from Python (ex. removing double pointers).

After peeking at the utility libraries, I think I have a few ideas on how to make them more usable from Python. @salasoom, can we meet in person after the holidays to go over these tests a bit more in-depth and brainstorm?

Can you base the PR from the develop branch?
For now, please separate the tests into dir tests/python

+1 Done.

I don't think we should have a default tolerance. The user should be aware that this is something they need to specify. If they don't know how to set this, then I think it would be safer to use a tolerance of zero.

Makes sense. Removed the default tolerance as part of the fixed-rate/precision commit.

Another difference wrt. zfp_compress is that only one field can be compressed at a time.

That's a good point. I can document that as a limitation of the current numpy-specific interface. Would adding a boolean output_header argument that toggles the writing of the header make it compatible with files compressed via the C interface?

Yes, I think that makes sense.

Does Cython expose internal implementation functions like _init_field in the global namespace? That would violate the ANSI standard, which reserves all public symbols starting with _.

__init__ doesn't get exported as a symbol in the shared library, but other functions that start with _ are exported, like _PyBytes_Type and ___pyx_module_is_main_zfp. I don't think this should be an issue since only Python will be dlopening the shared library, but maybe I'm missing something.

That's fine.  I was just concerned about polluting the global namespace with internal functions and doing so in an ANSI non-compliant manner.

class Memory(): Maybe check the return value of malloc?
The return value to zfp_decompress is not checked. It will be zero in case of failure. Maybe throw an exception?

Good ideas. Both are done.

I still think we should take this into consideration and change C struct member names for the next release to avoid issues like this.

+1 sounds good to me. Also, I spoke too soon. It does appear that Cython has support for struct members whose names conflict with python keywords (cython reference). I believe the follow should work (will try out tomorrow).

Nice.

ctypedef struct zfp_stream:
    zfp_exec _exec "exec"

With regards to the type field, is it possible that the behavior differs between Python 2 and 3, or did you test both?

That is possible. I have only been able to test against python 3 while on my mac. I can test against both py2 & 3 on Linux tomorrow.

Sounds good.  I noticed that this PR does not pass on Travis.  Do you know why?

Did you get a chance to look into using strides?

I have not, but I agree that strides is definitely the cleaner/better way to go in the long term. Unfortunately, I will not be able to implement that before the EOY.

I noticed that this PR does not pass on Travis. Do you know why?

I'm not 100% sure of the root cause, but I think it has to do with either the version of python (v3.4) or Cython (v?) in Ubuntu Trusty.  After switching to Ubuntu Xenial, everything passes.  I will investigate that further.

Regarding the changes to the LICENSE, LLNL has gotten picky about what can go in this file: see software.llnl.gov/about/licenses. We may have to update the license and place this additional information in the python directory.

Forgot to mention earlier that I made this change.  The top-level LICENSE is unchanged, and there are two subdirectories under the python directory which contain copied code along with a copy of the original licenses for that code.

That's a good point. I can document that as a limitation of the current numpy-specific interface. Would adding a boolean output_header argument that toggles the writing of the header make it compatible with files compressed via the C interface?

Yes, I think that makes sense.

Is there a file currently generated as part of the testsuite to test this against?
PS - I added a commit that removes references to reserved keywords like bytes and type.

Did you get a chance to look into using strides?

I have not, but I agree that strides is definitely the cleaner/better way to go in the long term. Unfortunately, I will not be able to implement that before the EOY.

No worries.  Let's tackle that next year.  Would be nice to verify that the Python version can decompress a file written by the zfp command-line tool (see more below).

I noticed that this PR does not pass on Travis. Do you know why?

I'm not 100% sure of the root cause, but I think it has to do with either the version of python (v3.4) or Cython (v?) in Ubuntu Trusty. After switching to Ubuntu Xenial, everything passes. I will investigate that further.

Nice!

Regarding the changes to the LICENSE, LLNL has gotten picky about what can go in this file: see software.llnl.gov/about/licenses. We may have to update the license and place this additional information in the python directory.

Forgot to mention earlier that I made this change. The top-level LICENSE is unchanged, and there are two subdirectories under the python directory which contain copied code along with a copy of the original licenses for that code.

Sounds good.

That's a good point. I can document that as a limitation of the current numpy-specific interface. Would adding a boolean output_header argument that toggles the writing of the header make it compatible with files compressed via the C interface?

Yes, I think that makes sense.

Is there a file currently generated as part of the testsuite to test this against?

No, we don't include binaries with zfp.  But, what I would suggest is that you generate a small rectangular 2D or 3D floating-point array and compress it with the zfp command-line tool (use -h to write a header).  A linear sequence of integers will do: f(x, y, z) = x + nx * (y + ny * z).  You want to make sure nx != ny != nz, e.g., nx = 15, ny = 16, nz = 17.  Then feed the compressed byte stream to your decompressor and verify that it works correctly.  You could use -a 0 to achieve near lossless compression.
Then repeat with zfPy compressing the file and the command-line tool decompressing.

PS - I added a commit that removes references to reserved keywords like bytes and type.

Can you please elaborate?  Surely zfp_field.type needs to be referenced somewhere?
We should be careful to avoid Python keywords in the zfp C implementation.  I tried to locate a list of reserved keywords but did not see either type or bytes referenced.  Are these provided by Python modules?

No, we don't include binaries with zfp. But, what I would suggest is that you generate a small rectangular 2D or 3D floating-point array and compress it with the zfp command-line tool (use -h to write a header). A linear sequence of integers will do: f(x, y, z) = x + nx * (y + ny * z). You want to make sure nx != ny != nz, e.g., nx = 15, ny = 16, nz = 17. Then feed the compressed byte stream to your decompressor and verify that it works correctly. You could use -a 0 to achieve near lossless compression.
Then repeat with zfPy compressing the file and the command-line tool decompressing.

zfPy: I like that stylization 😄
👍 Sounds like a solid plan. We should be able to write the "compress with zfPy and decompress with CLI" test after adding the write_header argument to compress_numpy.  I think we'll need to expose zfp_field as a "first-class citizen" before being able to write the reverse test, since that requires passing metadata about the array (i.e., type & dimensonality) to the decompression function.

We should be careful to avoid Python keywords in the zfp C implementation. I tried to locate a list of reserved keywords but did not see either type or bytes referenced. Are these provided by Python modules?

Sorry, I was being sloppy with my terms.  type and bytes are technically built-in functions.  You can find the full list of built-ins here. As far as reserved keywords, the best reference I could find was in the Cython source code.

PS - I added a commit that removes references to reserved keywords like bytes and type.

Can you please elaborate? Surely zfp_field.type needs to be referenced somewhere?

You are absolutely correct; it does need to be referenced.  In the case of type, I used a feature of Cython to "rename" the struct member to _type.  The struct definition now looks like:
ctypedef struct zfp_field:
    zfp_type _type "type"
    ...

And the references to the member look like: cdef zfp_type ztype = field[0]._type.  I didn't actually run into any issues personally with type, other than syntax highlighting oddities, but out of an abundance of caution (paranoia?), I made the switch.  Since it wasn't actually an issue, I can revert the change if you want.  Since exec is actually a reserved keyword and not just a built-in function, we will need to do something similar to handle that case (at least until the definition in C changes).
In the case of bytes, it was only used in function definitions at the top of the file (i.e., bitstream* stream_open(void* data, size_t bytes);).  Cython actually only requires type information for the parameters, so I changed it to bitstream* stream_open(void* data, size_t);.

@SteVwonder We now have support for reversible (lossless) compression (see feature/lossless branch). I think a quite natural default would be to use reversible compression when the user does not specify a rate, precision, or tolerance. We'll have to work on integrating these branches, but I wanted to mention this as a perhaps better alternative than throwing an exception when no optional argument is passed.

Just wondering about this choice and how numpy+compression really winds up looking to an end user. I mean, why does a numpy user even decide they want to use compression? If they're looking at memory usage, its probably in top, right?  I mean, thats the first place I'd look.  Ok, make a quick switch to ZFP arrays because those compress. Look at it again in top? Will lossless ZFP mode result in noticeable reductions? Note, this isn't a critical question about ZFP or lossless mode. Its just a question...is their likely to be a noticeable memory reduction? If not, user may, without looking further, decide to try something else? OTOH, if default for ZFP is a decent lossy mode, then in this simple example, user would be encouraged by what top says in the memory savings. Then, maybe they start looking at more of the ZFP details, maybe their computations wind up looking not as good, whatever. But their first sanity check (e.g. what top is telling them) suggests looking further.

I have a question that I couldn't quite answer from reading the code: can the decompress_numpy decompress directly into a pre-allocated buffer? This would be useful for zarr-developers/numcodecs#117.

@rabernat: just a heads up that the latest code in this PR now supports decompression into a user-provided buffer

Compared to this PR, imagecodecs's Cython bindings use zfp_field_set_stride_, release the GIL, can write to user provided output buffers, does more error checking, and allows to specify the execution model.

@cgohlke: Thanks for the feedback. Just a heads up that the latest code in this PR set the strides, releases the GIL (on the calls to zfp_(de)compress), and improves error checking and input validation.   Specifying the execution model is a longer term milestone, so it won't be added for this release, but hopefully soon.

There are a few things that, when improved, would enhance the experience inside notebooks a lot.
I understand that, at this point, documentation is not a priority, but having at least some documentation would definitely make it easier. My suggestions:

Use -DCYTHON_FLAGS=-Xembedsignature=True,binding=True by default for building (or specify it in the pyx source) for the functions which should be part of the public API (see Cython compiler directives)
disable the --no-docstrings option for release builds, which is set in python/scikit-build-cmake/UseCython.cmake so that docstrings are also available in release mode (default)

@romankarlstetter: Thanks for giving the bindings a spin and for the feedback.  I modified the cmake to add -Xembedsignature=True and not include --no-docstrings (except in the case of CMAKE_BUILD_TYPE=MinSizeRel).  It appears that -Xbinding=True has performance implications, so I opted to leave it out for now.   For computationally-intensive functions (like compress & decompress), it doesn't look like the overheads are bad, but I wanted to err on the performance-side here.   Do you know if embedsignature is enough to be useful in Jupyter Notebooks?  Or do you need the introspection enabled by binding=True?

Regarding the API: consider using a mode parameter which specifies which mode to use (one of precision, rate, accuracy, expert, as string or enum, maybe with a good default (accuracy?)). And then use one of the kw-args, (depending on the mode) to further configure compression settings.

Per the default mode: That is a good suggestion.  In previous conversations with @lindstro, my impression is that it is hard to define a "good default" since it is so data/domain dependent.  Luckily, lossless compression is being added in this release, so I think we are moving towards @markcmiller86's suggestion of lossless being the default.
Per the kwargs suggestion: I'm open to changing the compression settings part of the API to be two arguments (mode + mode parameter) rather than three (precision, rate, tolerance).  One nice thing about the current API is that the the various compression parameters types (tolerance and rate are doubles while precision is an int) are statically defined and then checked at runtime automatically by Cython.  Not a big deal to do that checking manually though.

@romankarlstetter: Thanks for giving the bindings a spin and for the feedback. I modified the cmake to add -Xembedsignature=True and not include --no-docstrings (except in the case of CMAKE_BUILD_TYPE=MinSizeRel). It appears that -Xbinding=True has performance implications, so I opted to leave it out for now. For computationally-intensive functions (like compress & decompress), it doesn't look like the overheads are bad, but I wanted to err on the performance-side here. Do you know if embedsignature is enough to be useful in Jupyter Notebooks? Or do you need the introspection enabled by binding=True?

I just tried that, binding=True is not necessary, the other two options are sufficient.

Per the default mode: That is a good suggestion. In previous conversations with @lindstro, my impression is that it is hard to define a "good default" since it is so data/domain dependent. Luckily, lossless compression is being added in this release, so I think we are moving towards @markcmiller86's suggestion of lossless being the default.

That sounds great 🙂

@SteVwonder I've set up Travis CI to always build & test Python. It's using Python 3.5 by default, with some build jobs testing 2.7. (I saw a compiler error on 2.7 with Cython)

@salasoom: thanks!  I saw the Cython error last night.  It looks like the Cython in Travis is not happy when argument names are left out of function signatures (in this case bytes in stream_open). I'm wondering if it is related to the Cython version.  I'm currently using Spack to install several older versions of Cython in an attempt to reproduce the error.

@SteVwonder At one point I did see 2.7 pass on trusty, when I was trying a bunch of different configs to get Python CI working.
https://travis-ci.org/LLNL/zfp/jobs/511857066
That passing job used cython 0.23.4, while the failing 2.7 jobs use 0.20.1

I believe there are still some issues with the handling of the strides argument to _decompress, but since that interface is still "experimental", I don't think that should block this PR.

@SteVwonder Thanks for quick fixes and for pointing out the strides issue.  We can live with this for now with proper documentation.

I think the AppVeyor failure is a false negative.  Only one builder failed, and within that builder, no tests failed.  The submission to cdash timed out:
Submit files (using https)
   Send to track: AppVeyor
   Using HTTP submit method
   Drop site:https://open.cdash.org/submit.php?project=zfp
   Error when uploading file: C:/projects/zfp/build/Testing/20190409-0345/Test.xml
   Error message was: Failed to connect to open.cdash.org port 443: Timed out
   Problems when submitting via HTTP
Command exited with code 127


graphics: Add 'color scale' values to vertices

CC @seebs
Not only for triangles, but also this should be useful for performance improvement for rendering quad images with ColorM.

Seems useful.
I tried a few variants of an API for this, and eventually concluded that the least-bad solution obviously available to me was to have a slice of color values in DrawImageOptions. A slice can be nil ("don't specify colors"), or can contain one color ("use this color") or can contain multiple colors (three for a triangle, or four for a quad).

I'd not want to add such slice options for quads, and ideally scales should be detected from the color matrix. On the other hand, I think specifying color scales for each vertices makes sense when triangles are rendered.
https://github.com/hajimehoshi/ebiten/tree/issue-662-color-wip Experimental branch


Add on block to cask DSL

This is my proposal for handling casks where some information differs between different macOS versions and CPU types. I've added an on block to the cask DSL to easily articulate which conditions must be met for the given block to be evaluated. My goal for the syntax here is that it is easily readable and simple for people who aren't as comfortable with Ruby and Homebrew to understand and use. Here are several examples of how the method can be used:
cask "foo" do
  on :monterey do
    sha256 "..."
  end

  on :arm do
    version "..."
    sha256 "..."
  end

  on :intel, and: :ventura do
    url "..."
  end

  on :mojave_or_older do
    app ".."
  end

  on :arm, and: :big_sur_or_newer do
    desc "..."
  end
end

One thing that I know is missing is a default case. An option for this is to have the default case be defined like normal, and just have the on blocks used for exceptions. For example:
cask "foo" do
  version "2.0.0"

  on :sierra_or_older do
    version "1.0.0"
  end
end

Another option is to require a default on block. For example:
cask "foo" do
  on :arm do
    url "..."
  end

  on :default do
    url "..."
  end

  # or...
  default do
    url "..."
  end
end

I'm also open to other ideas, of course.

This PR isn't quite ready yet, but I wanted to open it now to gather feedback. @Homebrew/cask, especially, please let me know if you have any thoughts or suggestions on this DSL.

I've also added a Homebrew::Override module which can be used to override the OS and CPU type in these on blocks so that you can load a cask as a different machine.

Review period will end on 2022-06-22 at 17:41:16 UTC.

In general, I like it! The and: is confusing, though. I feel it doesn’t fulfil the explicit goal:

simple for people who aren't as comfortable with Ruby and Homebrew to understand and use

It’s fine to understand; not sure about use. I think contributors will be confused by the colon shifting sides. Also, I don’t have better suggestions for _or_newer and _or_older but would be interested in seeing how else we can represent that.

An option for this is to have the default case be defined like normal, and just have the on blocks used for exceptions.

I’m all behind that one.

No, I'd say the simplest option is the best.

(Only raising this as something that might want to be thought about before bowing right back out:)
 Would it make any sense to have these new, generalized '`:on`' blocks become available for use in formulas at some point as a potential improvement over the existing ones limited to expressing OS conditionals, too, or would it be better to keep specialized conditionals more tightly scoped and/or targeted in formulas? (Not a blocking suggestion for this PR, naturally.)

We do potentially need something for architectures (#13103) so there's some growth potential there.

@Rylan12 Concept looks good to me. I'm not opinionated on the format.
One question is whether the approach to variable definition that we currently use regularly would change?
arch = Hardware::CPU.intel? ? "x64" : "arm64"

A few thoughts, but nothing amounting to a strong objection:

I worry that on is too generic a name.
Is it possible for :on to return a boolean when no block is passed? That way we can do things like if on :arm64 or similar.

I realise that my question about booleans above could be asked about our on_os blocks too, so I imagine there is something about needing to pass blocks here that I'm missing.

I've added an on block to the cask DSL to easily articulate which conditions must be met for the given block to be evaluated.

How do you think this compares to on_linux and on_macos in formulae? Should we consider adopting that format in formulae for consistency or vice-versa?

One thing that I know is missing is a default case. An option for this is to have the default case be defined like normal, and just have the on blocks used for exceptions.

This makes sense to me.

The and: is confusing, though.

Yeh, I feel so too. I wonder if these blocks could be nested to solve this instead?

As for the syntax, how about making the extensive version strictly define what the arguments mean? on arch: :intel, os: :big_sur do

I like this.

Would it make any sense to have these new, generalized ':on' blocks become available for use in formulas at some point as a potential improvement over the existing ones limited to expressing OS conditionals, too, or would it be better to keep specialized conditionals more tightly scoped and/or targeted in formulas? (Not a blocking suggestion for this PR, naturally.)

I agree. I'd like to see a similar approach to both, eventually.

It's just that for build instructions those don't make a whole lot of sense so we switched back to if statements.

Basically: we need this DSL for cases where we are generating brew info --json output that differs per platform without relying on if or unless that ends up "hiding" information unless repeatedly rerun.

We do potentially need something for architectures (#13103) so there's some growth potential there.

Yes. We need to handle that case in formulae sooner rather than later.

I worry that on is too generic a name.

Same here. only_on or something could work.

Thanks for the feedback, everyone! A few thoughts before I start moving forward:

I'm happy with only_on as a name.
I played a little more with some of the more complicated casks, and there are actually only 10 that have both a Hardware::CPU and MacOS.version call. So, since there are so few times when an and: is needed, I don't think we need to worry so much about optimizing the DSL for those cases. In looking at some of the examples, I think the nested approach is actually the easiest to understand. Here's an example of the old and new way for a more complex cask.
If we go with the nesting approach, then we probably don't need to have the arch: and os: argument names. Would we still prefer to include them or is it clear enough without them?
# With the argument names
only_on arch: :intel do
  only_on os: :monterey do
    version "..."
end

# Without the argument names
only_on :intel do
  only_on :monterey do
    version "..."
  end
end

Is it possible for :on to return a boolean when no block is passed? That way we can do things like if on :arm64 or similar.

I really like this idea!

I haven't really heard any suggestions about the _or_newer and _or_older, so I think I'm going to stick with that for now.
However, if we go with the nesting strategy, we could probably this information as a second argument like, for example:
only_on :monterey, :or_newer do
  version "..."
end

Not sure if that's just adding even more unnecessary complexity

Basically: we need this DSL for cases where we are generating brew info --json output that differs per platform without relying on if or unless that ends up "hiding" information unless repeatedly rerun.

I'd like to clarify one thing about this:
Currently, the way I envision this working is we change the values in Homebrew::Override but then still need to re-evaluate the cask. We should be able to just run @dsl.instance_eval(&@block) instead of the entire Cask::Caskloader.load method, but this still requires some level of "rerunning." Is this what you're picturing too?

Review period ended.

only_on is better, but still a bit generic. How about on_system? All of the following make sense to me:
on_system :macos do
  foo
end

on_system :arm64 do
  bar
end

on_system :intel, :linux do
  baz
end

If we go with the nesting approach, then we probably don't need to have the arch: and os: argument names. Would we still prefer to include them or is it clear enough without them?

I think it's clear enough without.

However, if we go with the nesting strategy, we could probably this information as a second argument like, for example:

This works. I'm happy with either this or the single parameter.

Is this what you're picturing too?

I was hoping we could do this in a way that would require repeated re-evalution for generating JSON but this isn't a blocker.

If you're asking about syntax, then yeah I think it would be great to have the syntax be shared between casks and formulae.

Yeh, syntax mainly 👍🏻

In terms of implementation, it's pretty different at the moment since formulae use the "override the no-op method on each individual os" instead of the "evaluate whether the condition passes and if not just return early" approach that is used here. I'm not sure if there's a reason for this difference, but I'm sure we can adapt formulae to match what's going on here.

👍🏻 cool.

Edit: we could also do the opposite and make on_arm, on_intel, on_ventura, on_mojave_or_older, etc. for use in formulae and casks

I don't feel strongly either way. I kinda like the idea of not having to change the formula definition and unifying on that rather than another deprecation/rewrite cycle there. It also makes a bit clearer that the API doesn't accept arbitrary arguments.

I guess it just depends on how willing we are to go through another change for formulae to use a similar syntax. This would be pretty simple since it would just be changing on_linux to only_on :linux, plus would allow us to remove the if Hardware::CPU conditions that are still present.

Yeh, I'm willing to do this but I think it's worth avoiding if so.

I'd still like to know people's thoughts on #13451 (comment) since sticking with if statements will make migrating much easier, but I think we can manage with what we have now.

I would prefer to avoid the if statements outside of methods when we're e.g. generating brew info --json so we could, eventually, do in a single pass.

only_on is better, but still a bit generic. How about on_system? All of the following make sense to me:

I like on_system. I think I still prefer on_linux, on_arm, etc. moreso.

I would prefer to avoid the if statements outside of methods when we're e.g. generating brew info --json so we could, eventually, do in a single pass.

So, if we don't use if statements to do things like arch = on_intel ? "intel" : "arm64", how would you suggest we handle these variables. Two options are:
cask "foo" do
  arch = "intel"
  version = "..."

  on_intel do
    arch = "arm"
    version = "..."
  end
end

cask "foo" do
  arch = on_intel "intel", otherwise: "arm"
  version "..."

  on_intel do
    version = "..."
  end
end

Has any thought been put into replacing a series of if/elsif statements (as shown here in the yasu.rb cask) with this dsl? To be fair that use case doesn't seem to be that common (51/4016 casks in Homebrew/Casks by my count) so it could just be left as is but I at least wanted to bring it up.

Currently, you could do:
cask "yasu" do
  version '5.0.1,701'
  sha256 '1fee2a849d9473b3d4776cf275ba48403a9b2c641077d4f4b61cd331d7f6459f'
  url "https://yasuformac.com/appcasts/10.13/yasuformac_#{version.after_comma}.zip",
      user_agent: :fake
  appcast 'https://yasuformac.com/appcasts/10.13/yasu.xml',
          checkpoint: '466c7e40951036fe7137b65c09d77fd42020face479abadfdc4b9d5e1c9a4cfc'

  on_el_capitan :or_older do
    version '3.0.4,504'
    sha256 'f1bd1e48a2ff0e9839f4b21c651e89a4e18d8aa85a9b9fa8642f333ef5b0053b'
    url "https://yasuformac.com/appcasts/10.11/yasuformac_#{version.after_comma}.zip",
        user_agent: :fake
    appcast 'https://yasuformac.com/appcasts/10.11/yasu.xml',
            checkpoint: 'd13bbaeca2a2c3610fa770ed3b2080fa040f525bbcc15c144d1802ecaf3ee487'
  end

  on_sierra do
    version '4.0.5,605'
    sha256 'dc18c2ee91144bd37b94ce6bab5cf2861b698432a301d1dd03be2bba23dbd609'
    url "https://yasuformac.com/appcasts/10.12/yasuformac_#{version.after_comma}.zip",
        user_agent: :fake
    appcast 'https://yasuformac.com/appcasts/10.12/yasu.xml',
            checkpoint: '53e665930618eaff3dabec8946d5884d8d25f949001a83516faf700d60a27e26'
  end
end

Another option that is not currently allowed but could easily be made okay is:
cask "yasu" do
  version '5.0.1,701'
  sha256 '1fee2a849d9473b3d4776cf275ba48403a9b2c641077d4f4b61cd331d7f6459f'
  url "https://yasuformac.com/appcasts/10.13/yasuformac_#{version.after_comma}.zip",
      user_agent: :fake
  appcast 'https://yasuformac.com/appcasts/10.13/yasu.xml',
          checkpoint: '466c7e40951036fe7137b65c09d77fd42020face479abadfdc4b9d5e1c9a4cfc'

  on_sierra :or_older do
    version '4.0.5,605'
    sha256 'dc18c2ee91144bd37b94ce6bab5cf2861b698432a301d1dd03be2bba23dbd609'
    url "https://yasuformac.com/appcasts/10.12/yasuformac_#{version.after_comma}.zip",
        user_agent: :fake
    appcast 'https://yasuformac.com/appcasts/10.12/yasu.xml',
            checkpoint: '53e665930618eaff3dabec8946d5884d8d25f949001a83516faf700d60a27e26'

    on_el_capitan :or_older do
      version '3.0.4,504'
      sha256 'f1bd1e48a2ff0e9839f4b21c651e89a4e18d8aa85a9b9fa8642f333ef5b0053b'
      url "https://yasuformac.com/appcasts/10.11/yasuformac_#{version.after_comma}.zip",
          user_agent: :fake
      appcast 'https://yasuformac.com/appcasts/10.11/yasu.xml',
              checkpoint: 'd13bbaeca2a2c3610fa770ed3b2080fa040f525bbcc15c144d1802ecaf3ee487'
    end
  end
end

Which do you prefer?

So, if we don't use if statements to do things like arch = on_intel ? "intel" : "arm64", how would you suggest we handle these variables.

Can you provide some examples of how we use these sorts of variables?

If we go with the on_* methods instead of passing the conditions, thoughts on handling _or_newer and _or_older as on_monterey :or_older do

This looks nice to me 👍🏻

Can you provide some examples of how we use these sorts of variables?

The main uses that I've seen are string interpolation in the url or in livecheck. For example, github. A quick search for arch = Hardware::CPU.(intel|arm)\? returns 276 casks.

     ARM/Apple Silicon variants/'variations' won't exist for macOS versions older than Big Sur, as you'll recall; maybe they shouldn't be allowed in the DSL?

The main uses that I've seen are string interpolation in the url or in livecheck. For example, github. A quick search for arch = Hardware::CPU.(intel|arm)\? returns 276 casks, so this is pretty extensively used.

Personally, I feel like we should lean more into duplication than interpolation here.
If we rely on if/else we cannot (ever) generate these URLs in a single pass.

I'll plan on merging this tomorrow unless anyone has any more thoughts/suggestions/reviews.
At that point, we'll need a new tag before using the new blocks in formulae and casks. I'll probably start working on those changes locally without a tag to give developers a little bit of time to notice any issues that arise with the on_macos and on_linux blocks (which use the new system after this PR) before tagging for everyone.

Thanks for the reviews @MikeMcQuaid and @apainintheneck! And thanks for everyone's feedback on the DSL!


Add badge for notificaitons -iOS

We are now showing a red dot to indicate when new notifications are available. Can we close it?

Unable to see the badge for dev app.
-Tested on dev app v1.8.0(304)

Verified this issue on dev app v1.9.5(330)


[Coverity CID: 179978] Error handling issues in /subsys/bluetooth/host/mesh/health_srv.c

Static code scan issues seen in File: /subsys/bluetooth/host/mesh/health_srv.c
Category: Error handling issues
Function: send_health_period_status
Component: Bluetooth
CID: 179978
Please fix or provide comments to square it off in coverity in the link: https://scan9.coverity.com/reports.htm#v32951/p12996

Already fixed in the master branch


Using 3.12.0, software simply crashes

Bug description
After installing @prisma/client and prisma CLI at version 3.12.0, node simply crashes without any output.
the code works when removing:
const { PrismaClient } = require("@prisma/client");
const prisma = new PrismaClient();

So it's clearly a Prisma issue, but there is not output to trace it.
prisma generate works fine without errors.
How to reproduce

install version 3.12.0
add prisma client after generating
crash.

Expected behavior
no crash. code should run.
Prisma information
generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "mysql"
  url      = env("DATABASE_URL")
}

model Account {
  id             Int       @id @default(autoincrement()) @db.UnsignedInt
  profile_id     Int       @db.UnsignedInt
  name           String    @default("") @db.VarChar(255)
  color          String?   @default("") @db.VarChar(100)
  initialBalance Decimal   @default(0.00) @db.Decimal(10, 10)
  currency       String    @default("USD") @db.Char(3)
  created_at     DateTime? @db.DateTime(0)
  updated_at     DateTime? @db.DateTime(0)

  Transaction Transaction[] @relation("account")
  @@map("accounts")
}

model Balance {
  id         Int       @id @default(autoincrement()) @db.UnsignedInt
  account_id BigInt    @db.UnsignedBigInt
  year       Int       @db.UnsignedInt
  balance    Decimal   @db.Decimal(10, 2)
  updated_at DateTime? @db.DateTime(0)

  @@map("balances")
}

model Category {
  id                   Int           @id @default(autoincrement()) @db.UnsignedInt
  profile_id           BigInt        @db.UnsignedBigInt
  name                 String        @default("") @db.VarChar(255)
  isBusinessExpense    Int           @default(0) @db.UnsignedTinyInt
  percentageDeductable Decimal       @default(0.00) @db.Decimal(10, 2)
  created_at           DateTime?     @db.DateTime(0)
  updated_at           DateTime?     @db.DateTime(0)
  Transaction          Transaction[] @relation("category")
}

model Project {
  id         Int       @id @default(autoincrement()) @db.UnsignedInt
  profile_id Int       @db.UnsignedInt
  name       String    @default("") @db.VarChar(255)
  created_at DateTime? @db.DateTime(0)
  updated_at DateTime? @db.DateTime(0)

  @@map("projects")
}

model Subject {
  id         Int      @id @default(autoincrement()) @db.UnsignedInt
  profile_id Int      @db.UnsignedInt
  name       String   @default("") @db.VarChar(255)
  created_at DateTime @db.DateTime(0)
  updated_at DateTime @db.DateTime(0)

  Transaction Transaction[] @relation("subject")
  @@map("subjects")
}

model Transaction {
  id             Int          @id @default(autoincrement()) @db.UnsignedInt
  profile_id     Int          @db.UnsignedInt
  account_id     Int          @db.UnsignedInt
  account        Account      @relation("account", fields: [account_id], references: [id])
  category_id    Int?         @db.UnsignedInt
  category       Category?    @relation("category", fields: [category_id], references: [id])
  subject_id     Int?         @db.UnsignedInt
  subject        Subject?     @relation("subject", fields: [subject_id], references: [id])
  date           DateTime?    @db.Date
  amount         Int          @default(0)
  vat            Int          @default(0)
  note           String?      @db.VarChar(255)
  transaction_id Int?         @unique @db.UnsignedInt
  transfer       Transaction? @relation("transfer", fields: [transaction_id], references: [id])
  transaction    Transaction? @relation("transfer")
  is_transfer    Int          @default(0)

  created_at DateTime? @db.DateTime(0)
  updated_at DateTime? @db.DateTime(0)

  @@map("transactions")
}

model UserProfile {
  id         Int     @id @default(autoincrement()) @db.UnsignedInt
  user_id    Int     @db.UnsignedInt
  profile_id Int     @db.UnsignedInt
  profiles   Profile @relation("profilesTouser_profiles", fields: [profile_id], references: [id])
  users      User    @relation("user_profilesTousers", fields: [user_id], references: [id])

  @@index([profile_id], name: "fk_profiles")
  @@index([user_id], name: "fk_users")
  @@map("user_profiles")
}

model User {
  id            Int           @id @default(autoincrement()) @db.UnsignedInt
  email         String        @unique @default("") @db.VarChar(255)
  name          String?       @db.VarChar(255)
  password      String?       @db.VarChar(255)
  created_at    DateTime      @db.DateTime(0)
  updated_at    DateTime      @db.DateTime(0)
  user_profiles UserProfile[] @relation("user_profilesTousers")

  @@map("users")
}

model Profile {
  id            Int           @id @default(autoincrement()) @db.UnsignedInt
  name          String        @default("") @db.VarChar(255)
  created_at    DateTime?     @db.DateTime(0)
  updated_at    DateTime?     @db.DateTime(0)
  user_profiles UserProfile[] @relation("profilesTouser_profiles")

  @@map("profiles")
}

Environment & setup

OS: mac os 12.3
Database: mysql 8.0.24
Node.js version: v16.13.2

Prisma Version
as mentioned, 3.12.0

@SirCameron thanks for the error report. But I cannot reproduce the issue. Just confirming you running npx prisma generate ?
Could you try that schema on a new project and let me know the exact steps you are doing to set it up.

Same here, using node v16.13.1.
App crashes at startup.

@Simskii could you provide a simple reproduction for us to take a look at?

I believe I just encountered the same issue @garrensmith, after starting my backend dev server on 3.12.0 I noticed after around 7-10 minutes, I received this error:
`
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF6C6A74C6F napi_wrap+111007
2: 00007FF6C6A181F6 v8::base::CPU::has_sse+59910
3: 00007FF6C6A190F6 node::OnFatalError+294
4: 00007FF6C72F220E v8::Isolate::ReportExternalAllocationLimitReached+94
5: 00007FF6C72D6FDD v8::SharedArrayBuffer::Externalize+781
6: 00007FF6C718011C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516
7: 00007FF6C718B53A v8::internal::Heap::ProtectUnprotectedMemoryChunks+1258
8: 00007FF6C7188679 v8::internal::Heap::PageFlagsAreConsistent+2457
9: 00007FF6C717D211 v8::internal::Heap::CollectGarbage+2049
10: 00007FF6C717B415 v8::internal::Heap::AllocateExternalBackingStore+1349
11: 00007FF6C719B88B v8::internal::Factory::NewFillerObject+203
12: 00007FF6C6ECA361 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+1409
13: 00007FF6C737B0CD v8::internal::SetupIsolateDelegate::SetupHeap+465325
14: 00007FF6C73CC34F v8::internal::SetupIsolateDelegate::SetupHeap+797743
15: 00007FF6C736BEC2 v8::internal::SetupIsolateDelegate::SetupHeap+403362
16: 0000007B80E95571`
So I then started it again and this time had no jobs/tasks/processes running, just the api server on it's own and I watched node's memory usage slowly crawl to full usage until there was no memory left and then received the same error again:
`<--- Last few GCs --->
[5992:00000262EF8627F0]   336261 ms: Mark-sweep (reduce) 2043.8 (2050.5) -> 2042.9 (2051.5) MB, 1356.1 / 0.0 ms  (average mu = 0.167, current mu = 0.031) allocation failure scavenge might not succeed
[5992:00000262EF8627F0]   337956 ms: Mark-sweep (reduce) 2043.9 (2053.5) -> 2043.0 (2053.7) MB, 1674.5 / 0.0 ms  (average mu = 0.086, current mu = 0.013) allocation failure scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF6C6A74C6F napi_wrap+111007
2: 00007FF6C6A181F6 v8::base::CPU::has_sse+59910
3: 00007FF6C6A190F6 node::OnFatalError+294
4: 00007FF6C72F220E v8::Isolate::ReportExternalAllocationLimitReached+94
5: 00007FF6C72D6FDD v8::SharedArrayBuffer::Externalize+781
6: 00007FF6C718011C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516
7: 00007FF6C718B53A v8::internal::Heap::ProtectUnprotectedMemoryChunks+1258
8: 00007FF6C7188679 v8::internal::Heap::PageFlagsAreConsistent+2457
9: 00007FF6C717D211 v8::internal::Heap::CollectGarbage+2049
10: 00007FF6C717B415 v8::internal::Heap::AllocateExternalBackingStore+1349
11: 00007FF6C719B88B v8::internal::Factory::NewFillerObject+203
12: 00007FF6C6ECA361 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+1409
13: 00007FF6C737B0CD v8::internal::SetupIsolateDelegate::SetupHeap+465325
14: 000003B0222CE4FA`
Hope this is enough to go off of - Seems to be repeatable for me - start Node, wait 7-10 minutes, crash 🙏

Thanks for the repo but unfortunately that still doesn’t give enough information. Are you able to create a simplified reproduction that I could run locally?

From: Niko @.>
Sent: Wednesday, April 6, 2022 5:35:31 PM
To: prisma/prisma @.>
Cc: garren smith @.>; Mention @.>
Subject: Re: [prisma/prisma] Using 3.12.0, software simply crashes (Issue #12684)
I believe I just encountered the same issue @garrensmithhttps://github.com/garrensmith, after starting my backend dev server on 3.12.0 I noticed after around 7-10 minutes, I received this error:
`
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF6C6A74C6F napi_wrap+111007
2: 00007FF6C6A181F6 v8::base::CPU::has_sse+59910
3: 00007FF6C6A190F6 node::OnFatalError+294
4: 00007FF6C72F220E v8::Isolate::ReportExternalAllocationLimitReached+94
5: 00007FF6C72D6FDD v8::SharedArrayBuffer::Externalize+781
6: 00007FF6C718011C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516
7: 00007FF6C718B53A v8::internal::Heap::ProtectUnprotectedMemoryChunks+1258
8: 00007FF6C7188679 v8::internal::Heap::PageFlagsAreConsistent+2457
9: 00007FF6C717D211 v8::internal::Heap::CollectGarbage+2049
10: 00007FF6C717B415 v8::internal::Heap::AllocateExternalBackingStore+1349
11: 00007FF6C719B88B v8::internal::Factory::NewFillerObject+203
12: 00007FF6C6ECA361 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+1409
13: 00007FF6C737B0CD v8::internal::SetupIsolateDelegate::SetupHeap+465325
14: 00007FF6C73CC34F v8::internal::SetupIsolateDelegate::SetupHeap+797743
15: 00007FF6C736BEC2 v8::internal::SetupIsolateDelegate::SetupHeap+403362
16: 0000007B80E95571`
So I then started it again and this time had no jobs/tasks/processes running, just the api server on it's own and I watched node's memory usage slowly crawl to full usage until there was no memory left and then received the same error again:
`<--- Last few GCs --->
[5992:00000262EF8627F0] 336261 ms: Mark-sweep (reduce) 2043.8 (2050.5) -> 2042.9 (2051.5) MB, 1356.1 / 0.0 ms (average mu = 0.167, current mu = 0.031) allocation failure scavenge might not succeed
[5992:00000262EF8627F0] 337956 ms: Mark-sweep (reduce) 2043.9 (2053.5) -> 2043.0 (2053.7) MB, 1674.5 / 0.0 ms (average mu = 0.086, current mu = 0.013) allocation failure scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF6C6A74C6F napi_wrap+111007
2: 00007FF6C6A181F6 v8::base::CPU::has_sse+59910
3: 00007FF6C6A190F6 node::OnFatalError+294
4: 00007FF6C72F220E v8::Isolate::ReportExternalAllocationLimitReached+94
5: 00007FF6C72D6FDD v8::SharedArrayBuffer::Externalize+781
6: 00007FF6C718011C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516
7: 00007FF6C718B53A v8::internal::Heap::ProtectUnprotectedMemoryChunks+1258
8: 00007FF6C7188679 v8::internal::Heap::PageFlagsAreConsistent+2457
9: 00007FF6C717D211 v8::internal::Heap::CollectGarbage+2049
10: 00007FF6C717B415 v8::internal::Heap::AllocateExternalBackingStore+1349
11: 00007FF6C719B88B v8::internal::Factory::NewFillerObject+203
12: 00007FF6C6ECA361 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+1409
13: 00007FF6C737B0CD v8::internal::SetupIsolateDelegate::SetupHeap+465325
14: 000003B0222CE4FA`
Hope this is enough to go off of - Seems to be repeatable for me - start Node, wait 7-10 minutes, crash 🙏
—
Reply to this email directly, view it on GitHubhttps://github.com/prisma/prisma/issues/12684#issuecomment-1090412329, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AABL2AWH26NBWWE3OG46G4TVDWVMHANCNFSM5SUCK7HA.
 Does it crash instantly or over time?
@Simskii How are you running your application?
@PatricioNG How do you start your application? Does it do anything in idle if no requests come in?
And for everyone:
Are these all applications that were upgraded from a previous Prisma version? Which one? I assume you did generate a new Prisma Client after the upgrade - manually or behind the scenes?  Is this also reproducible if you create a new project with npx prisma init in an empty folder?
This sounds bad, but right now we are indeed struggling to reproduce. For us everything works as it should, but we totally believe you that this is not true for you (at no fault of yours) and want to understand and fix.

@janpio thanks. I'm just running node index.js..
I got this working by deleting node_modules and reinstalling... obviously some old trash was still in there causing issues.
all working now.
Hopefully this can help the others having the same issue.
Thanks @janpio @garrensmith for your help!

Ok, that would have been my next suggestion as a potential workaround or solution.
For anyone else who has the same problem: Please zip your node_modules folder into an archive before trying this! We would really love to be able to compare a before and after of a node_modules folder where this happens, and deleting node_modules turns out to be the solution. This is what could help us identify the problem and hopefully fix it. Thanks.
@PatricioNG Based on this information, I think your problem is different from what the other 2 reported and you could optimally create a new bug report with all the information about this that could be useful. Thanks.

Hi @janpio - Thanks for responding, here's the answers to your question off the side of my desk, I'll try and see if I can get a small repro up and running later this afternoon

How do you start your application?
For me, this is just in my dev environment so I have a start script that runs npx tsc then cd's into the build folder and runs node server.js

Does it do anything in idel if no requests come in?
No, in my dev environment all batch jobs are turned off, so if the server is running it's simply waiting for requests

Was it upgraded from a previous Prisma version?
Yes "@prisma/client": "^3.8.1"

I assume you did generate a new prisma client after the upgrade
Yes, correct I have a script that runs post install things for me that includes npx prisma generate --schema ./prisma/EcommSchema.prisma

Am I generating my client to a custom location with output
Yes, looks like:  output   = "./generated/EcommSchema"

If I get time will create a new project this evening and see if I can reproduce

Could not reproduce in a new project.
After removing node_modules & yarn.lock all issues were solved.
Tried downgrading to 3.10 and upgrading to 3.12 again but the error does not occur anymore.

Did you by chance still have a backup of the broken node_modules folder @Simskii?

Unfortunately not... @janpio

@Simskii did find another case and send me both node_modules folder. They are mostly identical, besides tiny, irrelvant differences in the newer versions of some random apcakges of his project, and a difference in the .yarn-integrity file. Here is a unified diff:
--- C:/Users/Jan/Downloads/broken_node_modules/node_modules/.yarn-integrity	Fri Apr  8 15:17:06 2022
+++ C:/Users/Jan/Downloads/working_node_modules/node_modules/.yarn-integrity	Fri Apr  8 17:34:51 2022
@@ -428,7 +428,7 @@
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/autolinker/-/autolinker-0.28.1.tgz#0652b491881879f0775dace0cdca3233942a4e47",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/autoprefixer/-/autoprefixer-9.8.8.tgz#fd4bd4595385fa6f06599de749a4d5f7a474957a",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/awesome-phonenumber/-/awesome-phonenumber-2.70.0.tgz#81ed3abd1cecd08f974563b0973275f4c0f51efc",
-<EMAIL_ADDRESS>"https://registry.yarnpkg.com/aws-sdk/-/aws-sdk-2.1109.0.tgz#1be11188303f6545891edb0f7facf295f3ff9a40",
+<EMAIL_ADDRESS>"https://registry.yarnpkg.com/aws-sdk/-/aws-sdk-2.1110.0.tgz#bb4fe12a7768957b8df275bcd5e4188f5b5a9393",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/aws-sign2/-/aws-sign2-0.7.0.tgz#b46e890934a9591f2d2f6f86d7e6a9f1b3fe76a8",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/aws4/-/aws4-1.11.0.tgz#d61f46d83b2519250e2784daf5b09479a8b41c59",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/axios-logger/-/axios-logger-2.6.0.tgz#8baa8164846481530b1f822bf760eaf004d33d98",
@@ -520,8 +520,8 @@
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/camelcase/-/camelcase-5.3.1.tgz#e3c9b31569e106811df242f715725a1f4c494320",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/camelcase/-/camelcase-5.3.1.tgz#e3c9b31569e106811df242f715725a1f4c494320",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/camelcase/-/camelcase-6.3.0.tgz#5685b95eb209ac9c0c177467778c9c84df58ba9a",
-<EMAIL_ADDRESS>"https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001325.tgz#2b4ad19b77aa36f61f2eaf72e636d7481d55e606",
-<EMAIL_ADDRESS>"https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001325.tgz#2b4ad19b77aa36f61f2eaf72e636d7481d55e606",
+<EMAIL_ADDRESS>"https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001327.tgz#c1546d7d7bb66506f0ccdad6a7d07fc6d668c858",
+<EMAIL_ADDRESS>"https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001327.tgz#c1546d7d7bb66506f0ccdad6a7d07fc6d668c858",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/capture-exit/-/capture-exit-2.0.0.tgz#fb953bfaebeb781f62898239dabb426d08a509a4",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/capture-stack-trace/-/capture-stack-trace-1.0.1.tgz#a6c0bbe1f38f3aa0b92238ecb6ff42c344d4135d",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/caseless/-/caseless-0.12.0.tgz#1b681c21ff84033c826543090689420d187151dc",
@@ -730,7 +730,7 @@
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/ecc-jsbn/-/ecc-jsbn-0.1.2.tgz#3a83a904e54353287874c564b7549386849a98c9",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz#ae0f0fa2d85045ef14a817daa3ce9acd0489e5bf",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/ee-first/-/ee-first-1.1.1.tgz#590c61156b0ae2f4f0255732a158b266bc56b21d",
-<EMAIL_ADDRESS>"https://registry.yarnpkg.com/electron-to-chromium/-/electron-to-chromium-1.4.105.tgz#30de5e4ba020140b698539b2d366cd9c3a337ec7",
+<EMAIL_ADDRESS>"https://registry.yarnpkg.com/electron-to-chromium/-/electron-to-chromium-1.4.106.tgz#e7a3bfa9d745dd9b9e597616cb17283cc349781a",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/email-validator/-/email-validator-2.0.4.tgz#b8dfaa5d0dae28f1b03c95881d904d4e40bfe7ed",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/emittery/-/emittery-0.7.2.tgz#25595908e13af0f5674ab419396e2fb394cdfa82",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/emoji-regex/-/emoji-regex-8.0.0.tgz#e818fd69ce5ccfcb404594f842963bf53164cc37",
@@ -745,8 +745,8 @@
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/error-symbol/-/error-symbol-0.1.0.tgz#0a4dae37d600d15a29ba453d8ef920f1844333f6",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/es-abstract/-/es-abstract-1.19.2.tgz#8f7b696d8f15b167ae3640b4060670f3d054143f",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/es-to-primitive/-/es-to-primitive-1.2.1.tgz#e55cd4c9cdc188bcefb03b366c736323fc5c898a",
-<EMAIL_ADDRESS>"https://registry.yarnpkg.com/es5-ext/-/es5-ext-0.10.59.tgz#71038939730eb6f4f165f1421308fb60be363bc6",
-<EMAIL_ADDRESS>"https://registry.yarnpkg.com/es5-ext/-/es5-ext-0.10.59.tgz#71038939730eb6f4f165f1421308fb60be363bc6",
+<EMAIL_ADDRESS>"https://registry.yarnpkg.com/es5-ext/-/es5-ext-0.10.60.tgz#e8060a86472842b93019c31c34865012449883f4",
+<EMAIL_ADDRESS>"https://registry.yarnpkg.com/es5-ext/-/es5-ext-0.10.60.tgz#e8060a86472842b93019c31c34865012449883f4",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/es6-denodeify/-/es6-denodeify-0.1.5.tgz#31d4d5fe9c5503e125460439310e16a2a3f39c1f",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/es6-iterator/-/es6-iterator-2.0.3.tgz#a7de889141a05a94b0854403b2d0a0fbfa98f3b7",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/es6-promise/-/es6-promise-4.2.8.tgz#4eb21594c972bc40553d276e510539143db53e0a",
@@ -1340,7 +1340,7 @@
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/lru-cache/-/lru-cache-4.1.5.tgz#8bbe50ea85bed59bc9e33dcab8235ee9bcf443cd",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/lru-cache/-/lru-cache-4.1.5.tgz#8bbe50ea85bed59bc9e33dcab8235ee9bcf443cd",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/lru-cache/-/lru-cache-6.0.0.tgz#6d6fe6570ebd96aaf90fcad1dafa3b2566db3a94",
-<EMAIL_ADDRESS>"https://registry.yarnpkg.com/lru-cache/-/lru-cache-7.7.3.tgz#98cd19eef89ce6a4a3c4502c17c833888677c252",
+<EMAIL_ADDRESS>"https://registry.yarnpkg.com/lru-cache/-/lru-cache-7.8.0.tgz#649aaeb294a56297b5cbc5d70f198dcc5ebe5747",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/make-dir/-/make-dir-1.3.0.tgz#79c1033b80515bd6d24ec9933e860ca75ee27f0c",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/make-dir/-/make-dir-3.1.0.tgz#415e967046b3a7f1d185277d84aa58203726a13f",
<EMAIL_ADDRESS>"https://registry.yarnpkg.com/make-error/-/make-error-1.3.6.tgz#2eb2e37ea9b67c4891f684a1394799af484cf7a2",
@@ -2095,13 +2095,13 @@
   },
   "files": [],
   "artifacts": {
-    "<EMAIL_ADDRESS>[
+    "<EMAIL_ADDRESS>[
       "introspection-engine-darwin-arm64",
       "libquery_engine-darwin-arm64.dylib.node",
       "migration-engine-darwin-arm64",
       "prisma-fmt-darwin-arm64"
     ],
-<EMAIL_ADDRESS>[
+<EMAIL_ADDRESS>[
       "libquery_engine-darwin-arm64.dylib.node"
     ],
     "<EMAIL_ADDRESS>[
@@ -2443,15 +2443,6 @@
       "lib/protocol/crypto/build/config.gypi",
       "lib/protocol/crypto/build/gyp-mac-tool",
       "lib/protocol/crypto/build/sshcrypto.target.mk"
-    ],
-    "<EMAIL_ADDRESS>[
-      "introspection-engine-darwin-arm64",
-      "libquery_engine-darwin-arm64.dylib.node",
-      "migration-engine-darwin-arm64",
-      "prisma-fmt-darwin-arm64"
-    ],
-<EMAIL_ADDRESS>[
-      "libquery_engine-darwin-arm64.dylib.node"
     ]
   }
 }
\ No newline at end of file

Here only the interesting part:
--- C:/Users/Jan/Downloads/broken_node_modules/node_modules/.yarn-integrity	Fri Apr  8 15:17:06 2022
+++ C:/Users/Jan/Downloads/working_node_modules/node_modules/.yarn-integrity	Fri Apr  8 17:34:51 2022
@@ -2095,13 +2095,13 @@
   },
   "files": [],
   "artifacts": {
-    "<EMAIL_ADDRESS>[
+    "<EMAIL_ADDRESS>[
       "introspection-engine-darwin-arm64",
       "libquery_engine-darwin-arm64.dylib.node",
       "migration-engine-darwin-arm64",
       "prisma-fmt-darwin-arm64"
     ],
-<EMAIL_ADDRESS>[
+<EMAIL_ADDRESS>[
       "libquery_engine-darwin-arm64.dylib.node"
     ],
     "<EMAIL_ADDRESS>[
@@ -2443,15 +2443,6 @@
       "lib/protocol/crypto/build/config.gypi",
       "lib/protocol/crypto/build/gyp-mac-tool",
       "lib/protocol/crypto/build/sshcrypto.target.mk"
-    ],
-    "<EMAIL_ADDRESS>[
-      "introspection-engine-darwin-arm64",
-      "libquery_engine-darwin-arm64.dylib.node",
-      "migration-engine-darwin-arm64",
-      "prisma-fmt-darwin-arm64"
-    ],
-<EMAIL_ADDRESS>[
-      "libquery_engine-darwin-arm64.dylib.node"
     ]
   }
 }
\ No newline at end of file

Anyone an idea if this could be connected? What does this mean exactly? An artifact of 3.10.0 being present although they  updated Prisma to 3.12.0 already seems weird. But is it?

@pChausseC @SirCameron @Simskii are you on M1 macs by any chance?
If so, this could be #14058.
We implemented a workaround in 4.1.0. Could you check most recent Prisma version and see if it is still happens?
@PatricioNG your issue looks like something different than reported originally. Could you open a separate one and include a reproduction?

@SevInf yes I'm on M1 mac. Updated to @latest (4.3.1) without running into any issues. (haven't dealt with this issue since 3.12)

I am going to close this since it likely fixed in 4.1.0. Feel free to reopen if you manage to reproduce it on newer Prisma version