Update API token link

Addresses #3

Thanks!

sounds active.  Would you kindly advise how to use the app and/or get it running?


Increase action flexibility

Currently an action can specify multiple parents, but bindings have no way of choosing what parent they are for.
Therefor if an action decides to have multiple parents, all bindings will become bindings for all those parents too.
We can improve this by allowing bindings to specify 'targets'. This will specify what actions the binding should directly or indirectly be used for.
We can then also increase flexibility even more by allowing actions to specify children, instead of only parents. Currently actions always specify their parents, such that new handlers can always be created without the parents having to know of their existence. But it would also be useful to allow for the opposite; Have a new action be able to use an already existing action as a handler.
This way you can reuse all possible action handlers of an already existing action.
An action binding couldn't usually be traced back to the source ancestor action this way however (since some action doesn't know about one of its parents, its parent only know about it), but this is where the targets come in. When using this child behavior, the binding would have to specify the target in addition in order to allow for the action to be used in the actrion tree creation.
All of this has to be thought trough a little better, but will allow for an even greater amount of flexibility in the action system.

I gave it some more thought, and think no changes to the core action system are required. The mentioned changes would be rather involved and would increase the complexity of the whole system. I Think instead a simple pattern can be used to solve both issues:

I gave it some more thought, and think no changes to the core action system are required. The mentioned changes would be rather involved and would increase the complexity of the whole system. I Think instead a simple pattern can be used to solve both issues:


Invalid API endpoint causes the API endpoint to be reset

By default, ibmcloud API endpoint is set to https://cloud.ibm.com.
But when someone passed an invalid/wrong endpoint and it fails to set the endpoint, still it resets the endpoint.
Current Behaviour:
After an invalid usage of 'ibmcloud api INVALID_ENDPOINT', endpoint is reset, and you again have to set the endpoint to https://cloud.ibm.com, using ibmcloud api https://cloud.ibm.com

This might be intentional, but I felt it should not reset the API if usage failed, and just retained the last context/endpoint

➜  ~ ibmcloud api
API endpoint: https://cloud.ibm.com
➜  ~ ibmcloud api watsonx
Setting api endpoint...
FAILED
Could not determine the API endpoint

# Endpoint has been reset
➜  ~ ibmcloud api
No api endpoint set. Use 'ibmcloud api' to set an endpoint
Default api endpoint: https://cloud.ibm.com

# Need to set the endpoint manually to the default
➜  ~ ibmcloud api https://cloud.ibm.com
Setting api endpoint...
OK

API endpoint: https://cloud.ibm.com
Not logged in. Use 'ibmcloud login' to log in.
➜  ~ ibmcloud api
API endpoint: https://cloud.ibm.com

Expected Behaviour
➜  ~ ibmcloud api
API endpoint: https://cloud.ibm.com
➜  ~ ibmcloud api watsonx
Setting api endpoint...
FAILED
Could not determine the API endpoint

# Maybe better to not change anything, since the last command failed
➜  ~ ibmcloud api
API endpoint: https://cloud.ibm.com

Thank you for this suggestion. We will look into this.


docker pull thebloke/cuda12.1.1-ubuntu22.04-pytorch denied: requested access to the resource is denied

when doing
docker pull thebloke/cuda12.1.1-ubuntu22.04-pytorch
I get
denied: requested access to the resource is denied
Error response from daemon: manifest for thebloke/cuda12.1.1-ubuntu22.04-pytorch:latest not found: manifest unknown: manifest unknown

trying a different
docker pull long24ieccorp/cuda12.1.1-ubuntu22.04-pytorch
it works

The PyTorch repo is called :1 not :latest - I never pushed a :latest as I only made the pytorch repo as a sub-step of making the textgen and oneclick repos.
You know that the PyTorch one only contains PyTorch, no inference or even transformers? Is that what you want?

I just pushed :latest as well, so that will work too now
(pytorch2)  ubuntu@a10:/workspace ᐅ docker tag thebloke/cuda12.1.1-ubuntu22.04-pytorch:1 thebloke/cuda12.1.1-ubuntu22.04-pytorch:latest
(pytorch2)  ubuntu@a10:/workspace ᐅ docker push thebloke/cuda12.1.1-ubuntu22.04-pytorch:latest
The push refers to repository [docker.io/thebloke/cuda12.1.1-ubuntu22.04-pytorch]
db0bff7a716c: Layer already exists
89e404e93f0e: Layer already exists
638c8d9de20f: Layer already exists
70d32d75fc9f: Layer already exists
0f7c883f1a4f: Layer already exists
7aee75a70a2f: Layer already exists
24e1e08aaa60: Layer already exists
3416903c2cc4: Layer already exists
83687aeafbbf: Layer already exists
83ecbf43a888: Layer already exists
b15b1df4adac: Layer already exists
bea30ebbe843: Layer already exists
8afeff4e91d7: Layer already exists
6ec2b659c9ab: Layer already exists
566cd9dd29d6: Layer already exists
256d88da4185: Layer already exists
latest: digest: sha256:1808e29d42b38e637dfd52e1e03af86495961bd6c236e3bbd0fa038d38b1627f size: 3683
(pytorch2)  ubuntu@a10:/workspace ᐅ

Sorry I didn't explain the context, it came when execution the build_oneclick.py
Here is the full quote
#10 naming to docker.io/thebloke/cuda12.1.1-ubuntu22.04-pytorch:1
#10 naming to docker.io/thebloke/cuda12.1.1-ubuntu22.04-pytorch:1 done
#10 DONE 0.0s
2023-12-06 12:29:25 INFO [root] Pushing cuda12.1.1-ubuntu22.04-pytorch using command: docker push thebloke/cuda12.1.1-ubuntu22.04-pytorch:1
The push refers to repository [docker.io/thebloke/cuda12.1.1-ubuntu22.04-pytorch]
1a28f7d7cad4: Preparing
a51aa039af3d: Preparing
8bf2364d839c: Preparing
6a7e743bf135: Preparing
0f7c883f1a4f: Layer already exists
7aee75a70a2f: Waiting
24e1e08aaa60: Waiting
3416903c2cc4: Waiting
83687aeafbbf: Waiting
83ecbf43a888: Waiting
b15b1df4adac: Waiting
bea30ebbe843: Waiting
8afeff4e91d7: Waiting
6ec2b659c9ab: Waiting
566cd9dd29d6: Waiting
256d88da4185: Waiting
denied: requested access to the resource is denied
2023-12-06 12:29:26 ERROR [root] Got error while executing docker command: Command 'docker push thebloke/cuda12.1.1-ubuntu22.04-pytorch:1' returned non-zero exit status 1.
2023-12-06 12:29:26 ERROR [root] Process aborted due to error running Docker commands

Ah I see, yeah that script isn't really designed for other people to run it.  It's trying to push to my Docker Hub, which won't work of course
If you want to build your own version of the containers, edit build_docker.py and change thebloke to your own Docker Hub account.
Is there a need to re-build the containers, like new features or something broken? I can rebuild the main one if so

Ah I see, yeah that script isn't really designed for other people to run it. It's trying to push to my Docker Hub, which won't work of course
If you want to build your own version of the containers, edit build_docker.py and change thebloke to your own Docker Hub account.
Is there a need to re-build the containers, like new features or something broken? I can rebuild the main one if so

Ok it makes sense now. Thx.

No worries. I'll look to add a --username option to that script in future, to make this easier. I have another version of that script which has that, but I didn't update this one yet.


[FEATURE] adding k8sobjects receiver as preset

Adding k8sobjects receiver as a preset to collect Kubernetes Events as agreed on #669

@nicolastakashi you need to bump the chart version and then run make generate-examples in order for the CI to pass

@nicolastakashi you need to bump the chart version and then run make generate-examples in order for the CI to pass. GUIDE

Done, I'm not sure if it's right all those files being deleted.

Also needs a rebase (probably bc of the chart version/examples).

@povilasv indeed, it was by mistake


Let emulator run .casm files

The emulator can be able to run .copper bytecode and .casm files by compiling them to bytecode and run it

Close this because it's good the way it's done at the moment... even java does that


windows 11 cant find any shortcut and files after installed

Hi, I cannot find any file named including "Virtual Display" in my windows 11 after installed the software. I run install-cert.bat and then run virtual-display-driver-0.2.5-x86_64.msi with admin privileges.
It seems that it did not installed successfully.
OS name: Microsoft Windows 11 64bit
OS version:  10.0.22621  Build 22621

Did you see any errors during installation?
What do you see in device manager?

Is there anything at C:\Program Files\VirtualDisplayDriver?

No error during installation.

There are 3 virtual display adapters because i installed the driver for 3 times.

The path C:\Program Files\VirtualDisplayDriver does not exist.

No shortcut or folder named or related to Virtual Display Driver.

There is no any shortcut or file named "Virtual Display Driver Control" in the whole PC.

I will try the command msiexec /l*v "install.log" /i virtual-display-driver-0.2.5-x86_64.msi, and paste the log soon.

No error during installation.
There are 3 virtual display adapters because i installed the driver for 3 times.
The path C:\Program Files\VirtualDisplayDriver does not exist.
No shortcut or folder named or related to Virtual Display Driver.
There is no any shortcut or file named "Virtual Display Driver Control" in the whole PC.
I will try the command msiexec /l*v "install.log" /i virtual-display-driver-0.2.5-x86_64.msi, and paste the log soon.

Please follow the instructions here to manually uninstall all "virtual display drivers" (though the fact that it says it's working looks right)

Sorry for that there was a error during the installation i did not noticed.

This is the detail log.
install.log

The cause of your error is this:

DIFXAPP: ERROR: InstallDriverPackages failed with error 0xE0000247
DIFXAPP: RETURN: InstallDriverPackages()<PHONE_NUMBER> (0xE0000247)

It's failing to install the driver. My best guess why this might be happening (assuming you deleted and uninstalled all those drivers first before giving me this log) is because of the certificate might not be installed correctly.
Could you also follow this suggestion about nefconc in this comment? The uninstall part specifically.
./nefconc --remove-device-node --hardware-id Root\VirtualDisplayDriver --class-guid 4d36e968-e325-11ce-bfc1-08002be10318
I may have to start thinking of including a portable package with just the exe and driver by itself.

I added a portable package with just the files on their own. Before doing this, the obvious thing is, make sure everything is uninstalled, and no drivers are installed. With my original comment here, you can use pnputil to verify no more exist and delete any if they do.
You can install using the install.reg, add certificate (important of course), then go to device manager, click an any item in the list, go to action -> "add legacy hardware"

Press Have Disk

And finish the process

The last log was generated before uninstall the driver.
This log is generated after i uninstalled the driver, and i checked there is no virtual display adapter in Device Manager.
Here is my step:

run install-cert.bat with admin privileges in right click menu.
run msiexec /l*v "install.log" /i virtual-display-driver-0.2.5-x86_64.msi in windows terminal with admin privileges.
new_install.log

The last log was generated before uninstall the driver.
This log is generated after i uninstalled the driver, and i checked there is no virtual display adapter in Device Manager.
Here is my step:

run install-cert.bat with admin privileges in right click menu.
run msiexec /l*v "install.log" /i virtual-display-driver-0.2.5-x86_64.msi in windows terminal with admin privileges.
new_install.log

Looks like the same error. We got a build in the queue now. You can try the steps to manually install it above (should make things simpler)

I can't find the install.reg file. Missing the download link ?

I can't find the install.reg file. Missing the download link ?

The build is in the queue, hold on a moment. It'll be ready shortly

It seems that the cert was not imported successfully.
I run the command and get the output:
PS C:\Users\zfr\Desktop\virtual-display-driver-x64> certutil -addstore -f root "DriverCertificate.cer"
root "Trusted Root Certification Authorities"
Signature matches Public Key
Certificate "DriverCertificate" added to store.
CertUtil: -addstore command completed successfully.
PS C:\Users\zfr\Desktop\virtual-display-driver-x64> certutil -addstore -f TrustedPublisher "DriverCertificate.cer"
TrustedPublisher "Trusted Publishers"
Signature matches Public Key
Related Certificates:

Exact match:
Element 6:
Serial Number: 0dee6d126821b7456b9f248f3d9b2a
Issuer: CN=DriverCertificate
 NotBefore: 9/14/2023 12:24 AM
 NotAfter: 1/1/2040 7:59 AM
Subject: CN=DriverCertificate
Signature matches Public Key
Root Certificate: Subject matches Issuer
Cert Hash(sha1): 5dfa7e68997ed52d4b10420407860d1c29b6f427

Certificate "DriverCertificate" already in store.
CertUtil: -addstore command completed successfully.

The second command says 'Certificate "DriverCertificate" already in store' because i double click to import the cert file to TrustedPublisher before i run the command.

I added a portable package with just the files on their own. Before doing this, the obvious thing is, make sure everything is uninstalled, and no drivers are installed. With my original comment here, you can use pnputil to verify no more exist and delete any if they do.
(All that remains is for the package to actually be built) Edit: Build is in the queue now
You can install using the install.reg, add certificate (important of course), then go to device manager, click an any item in the list, go to action -> "add legacy hardware"

Install hardware that I manually select from a list

Next on show all devices

Press Have Disk and select the folder containing the driver
And finish the process

It works. The devices is normal in device manager. Thanks a lot.

Glad to hear it! Enjoy!


Add StableLM support

Related issue:

#327

Features:

Add stabilityai/stable-code-3b support
Support partial ROPE in QuantAttentionFused (partial rotary embedding used in stablelm and phi-2)

Just for clarification, this code only works with use_parallel_residual false, right? If use_parallel_residiual was true stablelm has basically the same architecture as phi-2 right? Have you experimented with implementing this?
Thanks


3DSRNGTool keeps on freezing when I type in the seed

I'm on the latest version of 3DSRNGTool and it just keeps on freezing everytime I enter the seed I get in PCalc.

I'm on the latest version of 3DSRNGTool and it just keeps on freezing everytime I enter the seed I get in PCalc.

Can you explain exactly what happens?

Closing this cause can't replicate. I'll reopen in case.


Connection Pool for SFTP and FTP

This is proof of concept that pooling connections will speed up snakemake for workflows with hundreds of remote inputs.
It allows snakemake to re-use connections when checking the metadata of multiple files without having too many connections open at once.
major changes
DomainObject
I added a connection_pool property to DomainObject and an abstract connect() method that subclasses need to implement. There are also a couple methods for getting connection parameters from args and kwargs that can be over-ridden as needed.
SFTP and FTP
The STFP.sftpc() and FTP.ftpc() methods have been replaced with the connect() method. And calls to self.ftpc() and self.sftpc() have been replaced with self.connection_pool.item().
I dropped the immediate_close parameter from FTP because I think it's no longer necessary with connection pooling.
ConnectionPool
This is imported from https://github.com/jmeppley/ConnectionPool (also available as conda install -c jmeppley connectionpool). There may be better libraries out there.
performance
2000+ RefSeq files over FTP
I was able to use FTP.remote to build a DAG to download the ~2400 protein files that make up the RefSeq 200 release in about 30 minutes. Without connection pooling, it was still working after 4 1/2 days.
250 to 1000 files over SFTP
I tested with this Snakefile that downloads files over SSH:
from snakemake.remote.SFTP import RemoteProvider
SFTP = RemoteProvider(private_key='/home/jmeppley/.ssh/id_rsa')

N = config.get('count', 250)

file_template = 'tyr.soest.hawaii.edu/mnt/delong/dataproducts/mesoscope/mapping/qc/stats/{file}.stats'
files, = SFTP.glob_wildcards(file_template)
files = files[:N]

rule total:
    input: expand("read_counts/{file}.txt", file=files)
    output: 'report.txt'
    shell:
        "cat {input} >> {output}"

rule read_count:
    input: SFTP.remote(file_template)
    output: 'read_counts/{file}.txt'
    shell: """echo -ne "{wildcards.file}\t" > {output}
              grep reads {input} | cut -f 3 >> {output}"""

I tested it using dry-runs with a few different values for count:

100 files: 1m 44s with the current snakemake and 3.0s with pooled connections
250 files: 4m 14s with the current snakemake and 3.5s with pooled connections
1000 files: 16m 50s with the current snakemake and 4.5s with pooled connections

Can you please format your PR with black?

I'll get the reformatted code up soon.
It's been a minute since I've looked at this, but I think only a few of the connection types lent them selves to pooling. It looks like I did FTP and SSH/SFTP. HTTP is supposed to disconnect between actions. I think other remotes, eg S3, already have some sort of background object managing the connection.
The question that should be answered before we make this live is what to do about the underlying connection pool code. I had to add a .close() method to the version that's available in pip.
I've been using my personal conda channel for testing, but that's not going to cut it for a real release. I can try to get it into conda-forge. I'm not sure about the process there. I've put things into bioconda, but this is more general.
It's a single file under the MIT License, so we could easily fold it into the snakemake/remote code with credit, but I'm not sure of the legality of that.
I think conda-forge is probably the way to go, but I'm not wedded to anything.

Yes, conda-forge sounds good.

conda-forge will take a little bit because I want to give the original developer a chance to weigh in. The project is only a couple years old.

I updated the PR code to do a few things:

fall back to creating new connections for every action if connection_pool is not installed
adds connection_pool as a python dependency in setup.py and test.environment.yml

There is no conda package yet for connection_pool, but the pip version works. I've used the pip version in the environment yaml file.

Once the package is on conda-forge, please ping me for merging! Thanks!

It's in pip. Can we pull it from there?
On Wed, Nov 11, 2020 at 2:47 AM Johannes Köster<EMAIL_ADDRESS>wrote:

Once the package is on conda-forge, please ping me for merging! Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/snakemake/snakemake/pull/483#issuecomment-725351064,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANDP3FOPH6DXPFWSQWXQJLSPJTVRANCNFSM4OUZDFMA
.

Technically yes, but for consistency, I'd like to have it in conda-forge.

Otherwise, the conda snakemake package cannot depend on it.

apart from performance, multiplexing would be nice in a server setup where only a certain amount of new connections (although successful) are allowed in a certain time-window (20 in 5mins).  Snakemake seems to check access to all files on DAG creations which let's me run into a paramiko.ssh_exception.SSHException: Unable to connect to server.com: [Errno 111] Connection refused.
@jmeppley Do you want help packaging ConnectionPool into a conda package? Pur Python projects should be relatively easy to setup I heard (and I would be interested to do it).

@enryH  I'll gladly accept help. I was able to set up a conda package, but getting it into conda-forge was harder. I'm more familiar with bioconda, but -- since it's more general -- this belongs in conda-forge.
I got as far as creating a conda-forge PR and going through one round of code-review, but it stalled.
What exactly are you picturing for a flag in this code? Just a simple boolean when the import fails that we can check later (EG at init.py:L347)? Or something more sophisticated?

Finally I read the conda-forge docs:) So it seems all 5 checks passed this time?!
I would simply add a boolean variable specifying it. CONNECTION_POOL_INSTALLED or connection_pool_installed (I would prefer the upper case as it is a environment variable, although it is set dynamically?!).
Most docstrings in the original package are Non-English. Do you auto-translate them somehow? (The code is quite clear, but I think English comments are often preferred?) Ah I just see you also use google translate already on some of the non so clear ones:)

It looks like I finally got all my ducks in a row with conda-forge and they
merged my PR. I think it will take some time to get from the staging
repository to actually being available in conda-forge, but I think all I
can do at this point is wait.
On Mon, Dec 21, 2020 at 12:55 PM Henry<EMAIL_ADDRESS>wrote:

Finally I read the conda-forge docs:) So it seems all 5 checks passed
https://github.com/conda-forge/staged-recipes/pull/12683#event-4137432669
this time?!
I would simply add a boolean variable specifying it.
CONNECTION_POOL_INSTALLED or connection_pool_installed (I would prefer
the upper case as it is a environment variable, although it is set
dynamically?!).
Most docstrings
https://github.com/jmeppley/ConnectionPool/blob/master/connection_pool/__init__.py
in the original package are Non-English. Do you auto-translate them
somehow? (The code is quite clear, but I think English comments are often
preferred?) Ah I just see you also use google translate already on some of
the non so clear ones:)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/snakemake/snakemake/pull/483#issuecomment-749191633,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANDP3F5HT3OIKEPPDN7O63SV6Y3PANCNFSM4OUZDFMA
.

upvote:)

Sorry for being so slow. The last months were crazy. I try to go through all PRs soon.


Make AWS S3 bucket name lowercase.

S3 requires bucket names to be lowercase in some regions. (Ireland for example)

Looks good to me.

Hey @maikelvl, could I have you rebase this branch with master? I merged in a few PRs just now so this is a few commits out of date.

I've rebased it.


PDF generation code

Great project and site. I'm trying to find the code that you use to generate the pdf version of the site that you have in your repo at https://github.com/ablwr/halt-and-catch-fire-syllabus/blob/main/HaltAndCatchFireSyllabus.pdf
I'm interested in jekyll plugins that do a good job generating PDFs and I like how your site offers both html and pdf versions.

Hi @leebrian!
I didn't do anything fancy or automated, and the script doesn't even have a cute bash loop, but I generated it like this: pandoc -f markdown -t pdf pandoc-header.md _classes/01.md _classes/02.md _classes/03.md _classes/04.md _classes/05.md _classes/06.md _classes/07.md _classes/08.md _classes/09.md _classes/10.md _classes/11.md _classes/12.md _classes/13.md _classes/14.md _classes/15.md pandoc-header.md -o syllabus.pdf --pdf-engine=xelatex
It does not automatically update if I update the site. I'm trying to think of if I have done automatic PDF generation through Jekyll before... I know there's a plug-in for it, but I think I've always just done it manually using bash when necessary!

Hi @leebrian!
I didn't do anything fancy or automated, and the script doesn't even have a cute bash loop, but I generated it like this: pandoc -f markdown -t pdf pandoc-header.md _classes/01.md _classes/02.md _classes/03.md _classes/04.md _classes/05.md _classes/06.md _classes/07.md _classes/08.md _classes/09.md _classes/10.md _classes/11.md _classes/12.md _classes/13.md _classes/14.md _classes/15.md pandoc-header.md -o syllabus.pdf --pdf-engine=xelatex
It does not automatically update if I update the site. I'm trying to think of if I have done automatic PDF generation through Jekyll before... I know there's a plug-in for it, but I think I've always just done it manually using bash when necessary!

Thanks so much @ablwr. This script is helpful and similar to something I do too. Surprisingly, I haven't found an easy jekyll plugin yet, but think it's do to me just not googling well. I appreciate you taking the time to answer.

Thanks so much @ablwr. This script is helpful and similar to something I do too. Surprisingly, I haven't found an easy jekyll plugin yet, but think it's do to me just not googling well. I appreciate you taking the time to answer.


🐛 Fix: monitoring networks tracing

What do these changes do?
Fix https://github.com/ITISFoundation/osparc-ops-environments/issues/825
Related issue/s

https://github.com/ITISFoundation/osparc-ops-environments/issues/825

Related PR/s
Checklist

[x] I tested and it works

It seems that if we want to "jaeger" trace all our services, we need to put all of them to the same docker network. Is this assumption correct? @mrnicegyu11

Almost correct @YuryHrytsuk , it is about the otel-collector specifically, but this is part of the jaeger stack


Migrate to Python 3 (Dual Python 3 and Python 2)

[x] Update filename analysis
[x] Update sqlitefid
[x] Remove todos
[x] Pylint --py3k work
[x] Unit tests - analysis code.
~[ ] Unit tests (integration) - report.~
[x] Sample-data, inc. utf8, and other analysis markers. (Provided as fixtures).
[x] Up to version 1 and remove beta.
~[ ] Github actions.~

Connected to #42
Connected to https://github.com/exponential-decay/demystify/issues/49
Python 3 related issues are tagged: here.

Closing in favor of rebased https://github.com/exponential-decay/demystify/pull/66


Plot Additional Data and Make Scrollable

Would be excellent to be able to plot 12 - 24 hours and make it scrollable. The monitor may not be the best place for this, but making the live view also the historical / analysis view would be very cool.

Yeah there's a few pieces here. First, we need to change the code that takes system time and ensures the horizontal axis includes it. We should replace current time with a vertical line or something, and update it on a regular interval.
Second, we should decide on the maximum time to show in the viewport, something pixel-based so smaller screens trigger scrolling. The amount of data is currently based on the pump history, glucose history, and prediction glucose provided.

Would we need to change the iter_glucose_hours and iter_pump_hours values in the loop, or simply append the new data each time?

If we're moving to Highcharts ...
http://www.highcharts.com/docs/chart-concepts/understanding-highstock

Openaps only supports overwriting files, so you'd need to just increase those values.


hovers callback doesn't work unless mouse is down

Steps to reproduce:
Go to current button demo:
https://kaboomjs.com/play?demo=button
Hover over the buttons
Nothing happens.
Hold the left button down and then hover over buttons
They respond like you would expect without having the left mouse button down.
hovers() should not depend on having the mouse button down.

Note, it doesn't matter which button ( right, left middle) you hold down - hover "works" if any of them are down, but NOT when you aren't holding a mouse button down.

What browser & machine you're on? Can't repro on my Macbook with Chrome / FF / Safari

Curious if you're picking up mouse move events at all. Does mousePos() give you the latest mouse position?

Version 95.0.4638.69 (Official Build) (64-bit) Windows
Device name	DESKTOP-ACN1IPP
Processor	Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz   1.80 GHz
Installed RAM	8.00 GB (7.83 GB usable)
Device ID	D2C430AF-398B-4796-8FF7-BDAFBD508D6C
Product ID	00325-81370-62044-AAOEM
System type	64-bit operating system, x64-based processor
Pen and touch	Pen and touch support with 10 touch points

Edition	Windows 10 Home
Version	21H1
Installed on	7/5/2020
OS build	19043.1288
Experience	Windows Feature Experience Pack 120.2212.3920.0

I tested on my own MacBook Pro and hover works there as expected. Weird.

Tested on Microsoft Edge on my PC and it also doesn't work.
Microsoft Edge
Version 95.0.1020.38 (Official build) (64-bit)

So it doesn't work on Windows on any browser? Does mousePos work there?

That example DOES work - the sprites follow the mouse and animate.

I think it's because your windows device might support touch, and it checks for mouseDown for hover event if it's on touch device here.
The isTouch is checked here As a workaround before the fix, can you try if
window. ontouchstart = null;
navigator.maxTouchPoints = 0;

before kaboom() fixes this case?

I'm testing on another windows desktop and hover does work.
I tested on my laptop with Firefox and it doesn't work.
Sorry this is frustrating for me as well.

I will try that in the demo

I ran this:
window.ontouchstart = null;
navigator.maxTouchPoints = 0;
kaboom();

function addButton(txt, p, f) {

	const btn = add([
		text(txt, 8),
		pos(p),
		area({ cursor: "pointer", }),
		scale(1),
		origin("center"),
	]);

	btn.clicks(f);

	btn.hovers(() => {
		const t = time() * 10;
		btn.color = rgb(
			wave(0, 255, t),
			wave(0, 255, t + 2),
			wave(0, 255, t + 4),
		);
		btn.scale = vec2(1.2);
	}, () => {
		btn.scale = vec2(1);
		btn.color = rgb();
	});

}

addButton("Start", vec2(200, 100), () => debug.log("oh hi"));
addButton("Quit", vec2(200, 200), () => debug.log("bye"));

// reset cursor to default at frame start for easier cursor management
action(() => cursor("default"));

in the demo viewer. No difference. ( No hover )
It seems this is a pretty obscure issue -  wouldn't blame you if you set this as low priority.

Yes, I'm using an all in one laptop with a touch screen.

There's a chance the browser won't let you change navigator.maxTouchPoints so kaboom still thinks it's a touch device, but this should be fixable on next release maybe we just don't special case touch devices here.

Ok if you get a build you'd like me to test I'd be happy to test the fix on this laptop.

Yeah MDN says it's read only :
The maxTouchPoints read-only property of the Navigator interface returns the maximum number of simultaneous touch contact points are supported by the current device.
https://developer.mozilla.org/en-US/docs/Web/API/Navigator/maxTouchPoints

In my console:
10```

That's a lot of touch points


Override package settings / categories in home-manager or darwin module

If i import the flake as a homeModule or darwinModule I would like to override just a few settings / categories but not redefine the whole definition. So instead of redefining everything:
home-manager.users.${config.user} = {
  imports = [ inputs.chris-nvim.homeModule ];

  chrisNvim = {
    enable = true;

    packages = {
      chrisNvim = { pkgs, ... }: {
        settings = {
          # will check for config in the store rather than .config
          wrapRc = true;
          configDirName = "chrishrb-nvim";
          aliases = [ "vi" "vim" ];
          # caution: this option must be the same for all packages.
          # nvimSRC = inputs.neovim;
        };
        categories = {
          lazy = true;
          generalBuildInputs = true;
          general = true;

          # languages
          go = true;
          python = true;
          web = true;
          java = true;
          devops = true;
          latex = false;

          # you could also pass something else:
          colorscheme = "catppuccin";
        };
      };
    };

};

, i would like to override only a few values e.g.
home-manager.users.${config.user} = {
  imports = [ inputs.chris-nvim.homeModule ];

  chrisNvim = {
    enable = true;

    packages = {
      chrisNvim = { pkgs, ... }: {
        categories = {
          python = true;
          web = true;
        };
      };
    };

};

Is this possible right now?

Yes, kinda.
The flake outputs its packageDefinitions function separately as well, so when using the flake or nixExpressionsFlakeOutputs templates you can just take the value and override specific ones?
Theres a utils.mergeCatDefs that can be used to merge function outputs together? Im sorry its not named better but its literally just is a function that takes 2 functions and an argument, and calls both of them with the argument and updates the output of the second from the output of the first.
However in classic nix fashion, its poorly documented rn
from :help nixCats.flake.outputs.exports
<mergeCatDefs> oldCats: newCats:
for merging category definitions,
will recursively update up to the first thing not an attrset.
For our purposes, we do not consider derivations to be attrsets.

You could do something like this
home-manager.users.${config.user} = {
  imports = [ inputs.chris-nvim.homeModule ];
  inherit (inputs.chris-nvim) utils packageDefinitions;

  chrisNvim = {
    enable = true;
    packageNames = [ "chrisNvim" ];

    packages = {
      chrisNvim = (utils.mergeCatDefs packageDefinitions.chrisNvim ({ pkgs, ... }: {
        categories = {
          python = true;
          web = true;
        };
      }));
    };

};

This is great for things like, if you wanted to make a dev shell with only 1 or 2 things from your normal config for example.
However, if Im not using it in a dev shell specific setting, I usually find it easier to just make a new package in my flake.
I have mine kinda factored into a base that I override like this:
https://github.com/BirdeeHub/birdeeSystems/blob/314a7e1622ef165435f950bb987d24e9af81717b/common/birdeevim/default.nix#L313-L414
And then I install it via my module
https://github.com/BirdeeHub/birdeeSystems/blob/314a7e1622ef165435f950bb987d24e9af81717b/homes/birdee.nix#L22C1-L25C5

big thanks for the fast and detailed reply! thats what I was looking for - it works 👍


read-write lookup implementation via permutation fingerprint in (Super)Nova R1CS

Implementation idea derived from Lookup Argument in Nova tricks: permutation checking without re-ordering
This idea as a demo, aiming to bridge the last mile of Nova R1CS to handle below scenarios (well known of high cost of constraints in R1CS)

read/write lookup table cross folding steps
memory execution trace proof in VM
bitwise operations
...

in pretty low cost (comparing with Merkle Tree approach, maybe SOTA?) per incremental computation (folding step): few field arithmetics + one random oracle hash for challenge, agnostic to lookup table size |T|! At the end just invoke an product sumcheck which can be integrated in nature to CompressionSnark with other claims together.
The test case show heapify creation on an array to build minheap. Array are representative as table support read/write. There will be O(N) round of foldings, where N is the size of array. In folding step i, there will be 3 read (parent node, 2 child nodes) and 4 write (update nodes). This case simply show that a table write in folding step i can be read later in folding step j for j > i.
R1CS constraints cost regarding to rw lookup

poseidon: ~992 constraints => for compute challenge in step circuit
others constrains for r/w lookup consistency: ~ log_2(number of read write lookup in 1 step). For example,  2^10 (1024) read write cost ~ 10 constraints
extra variables in Nova state: 5 (rolling challenge, challenge, rolling_READ, rolling_WRITE, read write counter)

read write counter can be omit in read-only table

Table read/write are wrapped into a gadget so it can be easily integrated in other applications. Gadget are supposed to be used in non-deterministic primary circuit, so there are other changes involved, basically set primary circuit as mutable reference so the lookup gadget can be updated and feeding non-deterministic aux witness easily.
PR still under drafted with following pending tasks (excuse for bad coding style in early stage :P )

[x] pre-process global challenge
[x] product sumcheck in CompressSnark
[x] rollback to immutable step circuit design
[x] debug: check why compress commitment in prove/decompress in verify got different result
[ ] thorough soundness check and documentation

Will keep ongoing work, and any review and feedback is welcome and appreciated :)

Hi @huitseeker Thanks for the review :)
I have addressed all the comments in latest commits

If you rebase on top of #76 you should get better results with CI!

First off, great work on getting this type of extension working on top of Nova! I've looked through your hackmd and the code, here are some thoughts. I've got a few code-related comments coming later too.
Soundness
Both the lookup gadget and the logic behind the lookup SNARK look good to me. I found the latter a bit confusing though, do you have any spec you're basing this off of?
What does the process look like for finalizing the proof? The LookupSNARK seems to only prove the correctness of the permutation argument, but I don't see any mention of verifying the R1CS instance. Would a user also create a CompressedSNARK alongside?
It seems like a good idea to have some separation between both proofs to ensure users can opt out of lookups, but there are also some missed opportunities for batching and minimizing the number of verifier hashes.
Cost estimation
It would be great to get an estimate of the number of constraints/variables required per read and write operation in each folding step.
This seems to be dominated by the cost of absorbing 3 field elements.
Would it be possible to get such an estimate?
Depending on the context (read vs. r/w, table size, number of accesses), it may be cheaper to consider the "vanilla" approach of performing these accesses solely with R1CS constraints.
Compressing SNARK
The Quarks approach for proving a grand product of $2^m$ terms requires a commitment to a vector of length $2^{m+1}$. For this lookup protocol, there are two such products to compute for both sides of the permutation check, leading to a total of $2^{m+2}$, or $4\cdot |T|$ committed elements, where $T$ is the size of the RAM.
The polynomial also needs to be opened at 2 different points, which can double the cost of the PCS opening. In the case of ZeroMorph, the reduction to univariate polynomials needs to be performed twice, leading to $\approx m$ additional group elements in the proof, and doubling the verifier costs.
In contrast, when the proof system has a mechanism for accessing "shifted" values or "rotations" of a vector, the grand product argument from PlonK can be used, which only requires the commitment to a single column of length $|T|$.
With the introduction of ZeroMorph, it would be possible to leverage this new PCS to implement the more efficient variant of product-check.
Witness generation
From a user-perspective, using lookups in this way means that the entire witness trace needs to be precomputed ahead of time, due to challenges alpha and gamma. Fundamentally, the amount of storage required to prove the program becomes linear in the number of folding steps, whereas it is constant for general folding.
This is fine for some applications, but can be impractical for others. It would be good to note what type of computations this SNARK is well-suited for. I think your example is a good one: An algorithm defined in terms of a loop, where the body has access to an array. In contrast to a non-folding-based circuit, the complexity of the compressing SNARK only depends on the size of the body.
Different techniques
Efficient lookup arguments will most likely all require modifications to some parts of the proof system. There are also some proposals (such as Origami and the LogUp variant in Protostar) that integrate lookups directly as part of the folding.
The main issue is that they require the recursive folding verifier to perform more scalar multiplications. With an implementation of CycleFold, this may have negligible effect on the recursive circuit, and may be competitive compared to the number of Poseidon hashes performed with the approach in this PR.
What your work shows though, is that the RAM/ROM abstraction is far more useful than fixed lookups. A recent paper surveys the different approaches to memory-consistency, and highlights how different proof systems implement this abstraction. It would be very neat to see these adapted to Nova, to get some intuition about the different performance profiles of the techniques.
In any case, being able to benchmark against an existing implementation will be very useful to figure out what approach works best!

Hi @adr1anh thanks for the insightful feedback! I will try to address thoughts in comments gradually :)
Soundness
The specs/flows in LookupSNARK::prove/verify is basically referred from RelaxedR1CSSNARK::prove() and RelaxedR1CSSNARK::verify() whereas in a simpler version. In LookupSNARK we only need offline-memory-check product-sumcheck + univariate PCS.
And yes, the R1CS verification should create CompressSNARK alongside. I choose to separation is due to I don't want to affect the exist CompressSNARK logic and I think it's easy for preliminary review, and ofc downside sacrificing efficiency on batching. I think the further optimisation on batching for LookupSNARK together with CompressSNARK definitely works because they share almost same flow of prove and verify. We can have a new CompressLookupSNARK, which tightly couple and allowing proving R1CS + Lookup together. The optimisation is open question an It will be nice to have more opinions from application side.
Computations of this SNARK well-suited
Actually, this SNARK is especially useful for applications which involves ALL scenarios in step circuit

random-memory-address
bitwise-operation
range check
One part not fully demonstrate in this PR (and docs) is we can combine all the tables into a single big table, with each table occupied range of address space in setup phase! We can define different constant offset for them so utilize more on large field, and address space does not overlap. Since there is just single table to cover all usage, in Nova state the state variable remain constant size. and challenge are shared so the cost are amortized significantly. In the final LookupSNARK, values in final table are composed of dynamic table commitment from prorver + fixed table commitment pre-defined in verifier side in setup phase.

Because current approach are based on indexed-lookup, means to prove v exist in table T we need to provide index (or address) a such that T[a] = v. Then a question might raise: if we combine all into single table, how to assure the non-deterministic address (e.g. range check) a from prover are within a valid address range ? Intuitively we shouldn't have range check on a via lookup otherwise it's a chicken-egg problem.
A strategy is, we can placement different tables into single table in a well-defined order, such that the table which address are well-constrainted (such as RAM where the address to rw is derived from computation) are in low memory address, while the tables (address are supposed to be non-deterministic from prover, such as range check) are in higher address. Then, we can replace the non-deterministic address from prover by T[offset + v] == v where the v is the value to range check. We put fixed lookup table at higher address consecutively so the soundness: T[offset + v'] == v' where v' is outside of range is not possible. (This property can be proved easily and intuitively)

Cost estimation

This seems to be dominated by the cost of absorbing 3 field elements.
In more detail, the cost can be spilited into 2 parts

absorbing 3 field elements -> poseidon hash circuit got padding, therefore the marginal cost for adding first read/write vs non first is different. The marginal cost when adding non-first read/write remain constant
extra R1CS var/constraints per read/write cost as below

name
extra var
extra constrains

Read on RWTable
+15 new var
+16 constraints

Write on RWTable
+15 new var
+16 constraints

Read on ReadOnlyTable
+ 8 new var
+ 10 R1CS constraints

Write on ReadOnlyTable
+ 8 new var
+ 10 R1CS constraints

table for RW cost more due to there are extra circuit to constraints global counter to avoid out-of-order r/w
read/write operation cost is no different

Giving we concat all table into a single table with separating address space.
On one scenarios: fixed lookup, e.g. 8 bits range check comparison

8 bit range check
extra var
extra constrains

vanilla  approach
+8 new var (bit)
+9 constraints (8 constrains as bit + 1 bit decompositions)

lookup approach
+15 new var
+16 constraints

Which indeed "vanilla approach" outperform.
While in another fixed lookup, e.g. 64 bits XOR

64 bit XOR
extra var
extra constrains

vanilla approach
64 * 3 = 192 new var (bit)
64 * 2 (constrains as bit) + 64 (bit XOR)+ 3 (bit decomposition) = 195

lookup approach (16 bit limbs XOR)
4*3(limb decomposition) + 1 (result) + 4 * (15 for lookup) = 73
3 (limb decomposition) + 4 * (16 for lookup) =  67 constraints

Where this lookup approach shows a significant improvement over vanilla approach.
Above estimated show current lookup approach is fruitful on R1CS for application which got complex step circuit, e.g. zkVM.

there is an simple optimisze to erach less var/constraints via reuse constant/constraints for gamma/gamma^2 in same step circuit => now it's duplicated per read/write, will optimize and update this PR

Witness && memory overhead
For this LookupSNARK, the witness for lookup need to be know in advance outside of circuit, means there need to be an external witness generator. The external witness generator is very common in zkVM application to firstly execute the program first on fast x86 machine then talk to zk proving system to generate the zk proof. It's a good idea to document this in some place, or maybe let unit test demonstrate it ?

Fundamentally, the amount of storage required to prove the program becomes linear in the number of folding steps, whereas it is constant for general folding.
I think the linear amount of storage is not due to this LookupSNARK design, whereas it's due to Nova non-deterministic step circuit design so we need to instantiate immutable step circuits in advanced. Each step circuit only hold the non-deterministic input for its round, so the size remain constant. @adr1anh would you mind to explain more for the linear in the number of folding steps?

New approach & techniques
In this PR, product sum augment are leveraged on old ppsnark->ProductSumcheck, while now it was refactor renamed to  ppsnark-> LookupSumcheck => ppsnark->MemorySumcheck recently and ported to Arecibo dev. I will try to adapt the I noticed there is another open PR https://github.com/lurk-lab/arecibo/pull/136 which use MemorySumcheck logUp protocol. I think logUp approach introduce extra cost in folding step for read/write lookup, like one extra constraint to have the field inverse. Ofc it will benefit on verifier cost.
The verifier cost of final SNARK will be amortized for application with massive folding steps. Comparing with existing STOA lookup attempt on R1CS Nova, I think first attempt of offline-memory-check tricks in Nova folding which lack of rotation still be a good milestone to save quite of cost on circuit size for complex application. We can left benchmark with other approach when other implementation also ready, whats your opinion :) ?

updated PR with below change summary

[ ] Refactor MemorySumcheckInstance to support different request of logUp proving -> I named with new name MemorySumcheckInstanceV2 so I can test it in ppsnark without touch batched_ppsnark quite a lot in POC phase
[ ] Lookup switch to MemorySumcheckInstance and design CompressedSNARKV2 to support lookup sumcheck batching with all other sumcheck stuff, so user dont need to have separate SNARK to improve efficiency
[ ] rebase to latest arecibo dev and fixed all compiling error
[ ] [Doing] run unittest cargo test --package arecibo --lib -- tests::test_ivc_rwlookup --exact got InvalidSumcheckProof failed, I am debugging and guessing just some small bug exist. Except for this, all other test in ppsnark are passed. So the mem_row, mem_col refactor I have verified success
[ ] Remove challenge poseidon circuit from folding circuit. Based on previous discussion, I think we can remove all challenge from circuit, and instead in final snark we just have singleread trace field value. Actually we already accumulate this rlc value in folding circuit (this is read_row doing). And in final snark prove phase, we let prover to aux a plonkish circuit to flatten this rlc field value (field value = read_row + final_table_read_row) to a column, then using this column to as input to compute poseidon or ECMH value in CompressedSNARK, then assert it equal with first challenge giving in Nova state. I am thinking from plonkish circuit perspective, and encode plonkish verify circuit in R1CS, although I think R1CS should work as well.

So with above optimized refactor, my lookup approach in folding will be extreme low cost, since in folding everything are working on field value without touch chanllenge circuit. challenge is defered to final snark. I guess this should be ultimate solution :smile: ?
Because so far I do not have clear idea for how to implement my idea about challenge defer in final CompressedSNARK R1CS, so it will be good if can gain some input from you

[ ] after that, the final thing is multiple table support. I think your idea works by having a const table value, e.g. + gamma^3 * TABLE_CONST so we can support multiple table easily.

Thanks for this great write-up and PoC @hero78119 ! This approach seems very promising for zkVM where the execution trace is done in advanced followed by ZKP work.
This all boils down to, how do we know that the prover did not cheat and change alpha and gamma, while keeping them deterministic. The portion  'Here share a Fiat-Shamir variant for non-interactive incremental computation' https://hackmd.io/C6XlfTRFQ_KoRy1Eo-4Pfw?view
Removing challenge Poseidon Circuit (costs, benefits)

I do not follow all of the details specifically in your last checkbox above, but having access to alpha and gamma with the execution trace upfront, allows for a lot of flexibility. For example a small separate SNARK could be created where the the Prover commits to the init values and checks that they match the final returned value; I think this is the approach you are planning on doing. circa: 'first challenge giving in Nova state'.
This seems like a very good first version.
How do we make this more modular?

If  bi, ai, ci are  assumed as public and known in advanced the final check can be computed on the fly by either the prover or verifier. In this case we would simply be doing a check that the NOVA IVC returned the correct alpha == gamma value for the table iterations against publicly known values.

Would removing the hashes in folding circuit make it less modular? i.e. preventing IVC (stop at anytime and verify). In your comment above it seems like IVC would still be possible with just the read_row.

But keeping the hash allows devs to add more values into it in a compressed form: hash(alpha and gamma, bi, ai, ci, more if we want..) at increased cost of absorbing field elements. They could also toggle out 'ci' or other values and make them public inputs where needed, this would reduce the cost of the hash; In practice there may not be much need to keep the multiplicities private, or in circuit. Or if most of them are 0, tricks can be used from other sparse polynomial papers.
For psuedo-IVC the prover could pre-commit to values at possible 'stop points' representing what the hash should be at those points (or public values), if the IVC circuit is stopped at a break point it could run the SNARK needed for proof. This would work something like Risc0 continuations.
While completely getting rid of the costs of the hash by removing the challenge Poseidon circuit, it might be helpful to think in terms of a execution trace, what values are public, and/or prover pre-commitments to possible stop points.
In the case where the full execution trace is not known, or is not needed, a user could decide only to do these steps where lookups are used. MLE-structured tables F.3.1 would be used instead.
It would really be cool if we get this and also have great modularity :)

Thanks for this great write-up and PoC @hero78119 ! This approach seems very promising for zkVM where the execution trace is done in advanced followed by ZKP work.
This all boils down to, how do we know that the prover did not cheat and change alpha and gamma, while keeping them deterministic. The portion 'Here share a Fiat-Shamir variant for non-interactive incremental computation' https://hackmd.io/C6XlfTRFQ_KoRy1Eo-4Pfw?view
Removing challenge Poseidon Circuit (costs, benefits)
I do not follow all of the details specifically in your last checkbox above, but having access to alpha and gamma with the execution trace upfront, allows for a lot of flexibility. For example a small separate SNARK could be created where the the Prover commits to the init values and checks that they match the final returned value; I think this is the approach you are planning on doing. circa: 'first challenge giving in Nova state'.
This seems like a very good first version.
How do we make this more modular?

If  bi, ai, ci are  assumed as public and known in advanced the final check can be computed on the fly by either the prover or verifier. In this case we would simply be doing a check that the NOVA IVC returned the correct alpha == gamma value for the table iterations against publicly known values.
Would removing the hashes in folding circuit make it less modular? i.e. preventing IVC (stop at anytime and verify). In your comment above it seems like IVC would still be possible with just the read_row.

But keeping the hash allows devs to add more values into it in a compressed form: hash(alpha and gamma, bi, ai, ci, more if we want..) at increased cost of absorbing field elements. They could also toggle out 'ci' or other values and make them public inputs where needed, this would reduce the cost of the hash; In practice there may not be much need to keep the multiplicities private, or in circuit. Or if most of them are 0, tricks can be used from other sparse polynomial papers.
For pseudo-IVC the prover could pre-commit to values at possible 'stop points' representing what the hash should be at those points (or public values), if the IVC circuit is stopped at a break point it could run the SNARK needed for proof. This would work something like Risc0 continuations.
While completely getting rid of the costs of the hash by removing the challenge Poseidon circuit, it might be helpful to think in terms of a execution trace, what values are public, and/or prover pre-commitments to possible stop points.
In the case where the full execution trace is not known, or is not needed, a user could decide only to do these steps where lookups are used. MLE-structured tables F.3.1 would be used instead.
It would really be cool if we get this and also have great modularity :)
EDIT Other observations:
'Address bi constrained by step circuit opcode. If its for memory load/store, its constrained by instruction itself.' from here https://hackmd.io/C6XlfTRFQ_KoRy1Eo-4Pfw?view - in the case of SuperNova bi would be zi; any need to absorb it again?

Hi @wyattbenno777 thanks for the feedbacks. Let me try to expand more context on challenge part refactor plans. You proposed many good points which is not covered by this PR, as this PR still in preliminary version with goals

based on indexed lookup to support random memory access.
lookup cost in folding step circuit just incur field operation
make a lookup gadget for cross folding step lookup (including random memory access)

For this lookup implementation, the read_set/write_set accumulation which implemented in folding step circuit, are already in pretty low cost by just few fields operations. Here are a raw summary https://github.com/lurk-lab/arecibo/pull/48#issuecomment-1827462868 as can see, the cost for an RW access just +15 R1CS var + 16 R1CS constraints. But when we introduce this lookup approach, there is an cost for challenges alpha, gamma computation in folding circuit. Although this cost is agnostic to number of read/write and only introduce once, however it's still costly, as around ~992 constraints for Poseidon RO.
So one optimisation suggested by @adr1anh is can we seek another low cost way to accumulate and verify challenge alpha, gamma integrity in folding circuit. After some quick thoughts, I proposed an approach that we can defer RO in folding circuit to final SNARK.

Prover still compute challenges alpha/gamma in witness generation phase and put alpha/gamma in z0 state
Based on offline memory check formula (in LogUp variant), alpha, gamma should be derived from sequence of (bi, ai, ci). which can be viewed as sequence of read_set trace
In folding step circuit, we already accumulated read_set trace as a field value, which in core is linear combinations of (bi, ai, ci) with alpha/gamma
in final compression snark, verifier can get read_set field value from final state zn. To verify integrity of challenge, prover can witness (auxiliary), generating a proof, to attest the challenge-integrity to prove below
- public input: alpha, gamma, read_set, audit_value_commitment, audit_ts_commitment
- assert read_set == logUp_sum([(bi, ai, ci), (bi, ai, ci),.... (bi, ai, ci)], alpha, gamma) // (bi, ai, ci) sequence are witnessed from prover
- computed_alpha := poseidon([(bi, ai, ci), (bi, ai, ci),.... (bi, ai, ci), audit_value_commitment, audit_ts_commitment]) // use (bi, ai, ci) to compute alpha
- computed_gamma := poseidon(computed_alpha)
- assert alpha == computed_alpha
- assert gamma == computed_gamma

With this optimization, we do not need to encode poseidon circuit in folding step circuit, which is somehow achieve the match the real meaning of low-cost.

Would removing the hashes in folding circuit make it less modular? i.e. preventing IVC (stop at anytime and verify). In your comment above it seems like IVC would still be possible with just the read_row.

Actually I am not quite sure about this question related to modularity if removing challenge from folding circuit. (The vision seems bigger), because challenges in lookup gadget are assumed to be blackbox and do not expect other application-level manipulation, so it will be easily to use and bug-pruning.
I think IVC still can stop at any point and just depends on what you want to verify if stop earlier. For example, we can still verify recursive_snark correctness, but we can't take intermediate field value from z_i state and verify it business logic are satisfied. For example in zkVM case, we need to verify the pc are terminated at correct step. In this case, pc value will be in memory commitment.
Hopefully my elaboration for deferring challenge-integrity is clear :)

Thank you for that very clear write-up on the LogUp proposed changes @hero78119 . That all seems very clear to be a SOTA implementation :)
I am messing around with your code a bit in a fork specifically in the context of memory consistency checks.
In the original Blum work, which the offline memory check from JOLT comes from, and also the way it was explained to me by one of the JOLT authors, the procedure goes as follows:

W is init to (addr, val, 0 (multiplicities)) for all entries.
for each 'read' there is a following 'write'.
In the last step there is a read with no following write (added to R). This ensures that W.len == R.len and is ready for a permutation check.
The tuples are made checkable with a Reed-solomon fingerprint (alpha and gamma). (LogUp work here.)

I assume it was done this way in research and in Lasso for soundness.
Please excuse me if I missed it in the fork or gadget here — but are we still following this 1,2,3,4 paradigm?

Hi all, and thank you for all the insightful discussion and work! I'd like to explain some thoughts around the construction using logUp, and to try and figure out how to make it more modular. Apologies in advance for the length of this response 😅.
The main point I would like to make though, is that I think we may want to consider doing the audit as a circuit, rather than a specialized SNARK. Even though this may be less efficient, it will allow us to explore these primitives and figure out how they can be applied more generally to zkVM designs. It should also allow us to explore different approaches such as ECMH (described in the Spice paper ) which do not depend on challenges r, gamma to be generated in advance.
Background
In the framework of Blum referenced by @wyattbenno777, we are maintaining two multi-sets R and W that contain entries (v, a, t), and after all the read/write operations, we check that R = W.
The original paper instantiated the multi-sets R, W using a hash-chain of all entries. Given the full pre-image of both sets, the verifier can check that these two hashes to indeed contain the same entries, though this takes time linear in the number of reads/writes.
The scheme can be more efficient for the verifier by implementing the multi-set abstraction in a more "ZK way" using some newer techniques.
As I understand in Lasso, we are constructing a set S by defining it as a pair of field elements acc_S, h_S. Given n elements in S, we would can define them as:
$$
acc_{S}(r, \gamma) = \prod_{i = 1}^n (r + v_i + \gamma \cdot a_i + \gamma^2 \cdot t_i)
$$
$$
h_S = H((v_1, a_1, t_1), \ldots, (v_n, a_n, t_n))
$$
Alternatively using logUp, we can write acc_S as
$$
acc_{S}(r, \gamma) = \sum_{i = 1}^n \frac{1}{r + v_i + \gamma \cdot a_i + \gamma^2 \cdot t_i}
$$
Both of these formulations are equivalent, assuming that r, gamma are both derived from h_S.
When applied to memory checking, both parties need to derive acc_R, h_R, acc_W, h_W. The way this is done in circuit is that the prover will supply r, gamma as public input, and the verifier is able to iteratively compute acc_S, h_S by adding entries to the running sum/product acc_S and hashing the entry to h_S. At the end of the program, we can add any remaining elements to W and then check that r, gamma was derived from (h_R, h_W) and that acc_R = acc_S.
The observation that @hero78119 implemented using logUp, is that we can combine both R,W into a single set S, by using the fact that we can add and remove elements from a set. I have written down the following construction, so that we can understand together how it works, and then figure out how to prove it secure in our specific context of (N)IVC.
Description
Init:

Get r, gamma as public input, which are derived by the prover beforehand using the same access patterns as defined below.
Set acc = 0, h = 0, t = 1

Read a:

Get (v', t') non-deterministically as the current value and timestamp of the RAM
Check that t' < t
Check that (t' = 0) => (v' = 0) for ensuring that the initial value was set to 0
Remove (v', a, t') from S

Update h = H(h, (v', a, t'))
Update acc = acc - 1/(r + v' + a * gamma + t' * gamma^2)

Add (v', a, t) to S

Update h = H(h, (v', a, t))
Update acc = acc + 1/(r + v' + a * gamma + t * gamma^2)

Optimization: merge remove and add

Update h = H(h, (a, v', t, t'))
Update acc = acc - 1/(r + v' + a * gamma + t' * gamma^2) + 1/(r + v' + a * gamma + t * gamma^2)

Update current timestamp t = t + 1
Return v'

Write (a, v):

Get (v', t') non-deterministically as the current value and timestamp of the RAM
Check that t' < t
Check that (t' = 0) => (v' = 0) for ensuring that the initial value was set to 0
Remove (v', a, t') from S

Update h = H(h, (v', a, t'))
Update acc = acc - 1/(r + v' + a * gamma + t' * gamma^2)

Add (v, a, t) to S

Update h = H(h, (v, a, t))
Update acc = acc + 1/(r + v + a * gamma + t * gamma^2)

Optimization: merge remove and add

Update h = H(h, (a, v', v, t, t'))
Update acc = acc - 1/(r + v' + a * gamma + t' * gamma^2) + 1/(r + v + a * gamma + t * gamma^2)

Update current timestamp t = t + 1

Audit:

Let A = {(v_1, a_1, t_1), ..., (v_n, a_n, t_n)} be the current state of the RAM, sorted by addresses a_i.

Define a_0 = 0 as sentinel

For i = 1, ..., n

Check that a_{i-1} < a_i to ensure that all addresses are unique.
Check that (t_i = 0) => (v_i = 0)
Insert the initial entry (0, a_i, 0) into S

Update h = H(h, (0, a_i, 0))
Update acc = acc + 1/(r + 0 + a_i * gamma + 0 * gamma^2)

Remove the final entry (v_i, a_i, t_i) from S

Update h = H(h, (v_i, a_i, t_i))
Update acc = acc - 1/(r + v_i + a_i * gamma + t_i * gamma^2)

Optimization: merge remove and add

Update h = H(h, (a_i, v_i, t_i))
Update acc = acc - 1/(r + v_i + a_i * gamma + t_i * gamma^2) + 1/(r + a_i * gamma)

Check that r, gamma are derived from h
Check that acc = 0

Application to NIVC
The above construction can easily be implemented purely in an NIVC setting. In the first phase, we modify each circuit's IO, so that it takes as in

Program IO z_in -> z_out
Timestamp t_in -> t_out
Hash chain h_in -> h_out
logUp accumulator acc_in -> acc_out
Constant challenges r, gamma that are passed as-is from input to output.
Two auxiliary variables  (is_audit, a_prev) which are initialized to (0, -1) which are also passed as-is by the circuits which are not the audit one. It is only used by the audit circuit to keep track of the uniqueness check, and needs to be passed to the other circuits to ensure all SuperNova circuits have the same IO.

The initial input of the program in the first iteration is (z_0, t = 0, h = 0, acc = 0, r, gamma, is_audit = 0, a_prev = -1)
Each step circuit updates t, h, acc according to the reads and writes it performs. At the very end, we run an audit circuit as another NIVC circuit that will be initialized with the last output of the actual computation. The goal is to run audit iteratively until the we get to acc = 0
audit(z, t, h, acc, r, gamma, is_audit, a_prev):
	if (acc = 0)
		// we can safely reset t, a_prev to 0
		// since they are no longer required
		// and we can hide the number of accesses
		// and the largest address accessed
		return (z, t=0, h, acc=0, r, gamma, is_audit a_prev=-1)

	let (v', a', t') <- ... // non-deterministic alloc

	// in the first iteration, skip the comparison to allow for the zero address
	if is_audit = 0
		is_audit = 1
	else
		// check uniqueness
		assert(a_prev < a')

	// base case checking
	assert((t' = 0) => (v' = 0))

	// update hash
	h' = H(h, (a', v', t'))

	// add init
	acc' = acc + 1/(r + gamma * a')

	// remove current entry
	acc' = acc' - 1/(r + v' + gamma * a' + gamma^2 * t')

	return (z, t, h', acc', r, gamma, is_audit, a')

The final SuperNova verifier will need to check

r, gamma is derived from h
acc = 0 and that the audit circuit was successful
the output program_counter corresponds to the audit circuit index
t, is_audit, a' can safely be ignored
The audit circuit can be run iteratively until we reach acc = 0. Subsequent calls will ensure that the total number of accesses defined by t is hidden from the verifier. For folding efficiency, we can write the audit circuit as one which repeats itself many times.

We can probably find some better set of conditions to ensure uniqueness in the NIVC setting (I'm not yet 100% sure it is sound), but for now the definition of the audit circuit should satisfy our requirements.
By implementing the audit circuit entirely in-circuit, we can ensure that the construction is entirely compatible with an existing SuperNova verifier, and would be a lot cheaper since we do not need any additional Sumcheck or PCS openings to verify the state.
It is slightly more expensive in-circuit, but it does bring us closer to a more modular framework as @wyattbenno777 was talking about.
Using a SNARK for the audit
We need to impose several extra restrictions if we want to use a SNARK-like approach for performing the audit.
Firstly, we will need to define a fixed sized RAM where all addresses are integers in a range [0,N[, as a way of ensuring uniqueness of the addresses.
The read and write operations can be defined as before, and we can use the known bound 0 ≤ t < N to enforce a cheaper comparison check (as I think is already implemented in this PR).
During the audit SNARK, we need to supply 2 committed vectors V, T of size N such that (V_i, i, T_i) is the entry at the i-th address. Address i was not used, then we set V_i = T_i = 0.
The Sumcheck relation is then given by
$$
acc_S = \sum_{i=0}^{N-1} \frac{1}{r + V_i + \gamma \cdot i + \gamma^2 \cdot T_i} - \frac{1}{r + \gamma \cdot i }
$$
Here, exploit the fact that the verifier can easily evaluate the polynomial whose evaluations are the first N integers (if N is a power of 2).
We then need to check that r, gamma were properly derived from h, V_i, T_i.
Removing hashes
Unfortunately, we cannot avoid the hashes that are performed during the read and write operation by the circuit. The main reason has to do with the semantics of the underlying interactive protocol that implements the multi-set that we are emulating in-circuit. Essentially, every tuple we add or remove from the set can be thought of as a value provided by the prover. It therefore appears in the corresponding Fiat-Shamir transcript. In our circuit implementation, we can think of the interaction as follows:

The prover sends a list of the tuples (v, a, t) which are added and removed from the multi-set, with any particular order.
The verifier needs to add all these tuples to the transcript to derive the next challenges r, gamma.
The verifier can now compute the logUp accumulator acc by adding or subtracting all tuples sent by the prover.

In the circuit, we are exploiting the non-determinism and can supply r, gamma in advance, which allows the verifier implemented in the circuit to compute acc and h iteratively. However, it is crucial that the tuple of every single insertion and removal is added to the transcript, even when it is seemingly added twice.
Soundness
It seems to me that we can easily adapt the existing security proof from the existing literature to the above construction. Though for the sake of discussion, we can try to understand intuitively why it should hold.
Before the audit happens, we can enforce the following invariant. By performing the reads and writes inside the circuit, we are ensuring that every update to S is valid. We are ensuring the property that if an address a was accessed at some timestamp t' and has value t', then the acc will contain the term
$$
\frac{1}{r + v' + a\cdot \gamma + t' \cdot \gamma^2 } - \frac{1}{r+ \gamma \cdot a}
$$
At time t > t' a read or a write of v at address a transforms the term respectively into.
$$
\frac{1}{r + v' + a\cdot \gamma + t \cdot \gamma^2 } - \frac{1}{r+ \gamma \cdot a}
$$
and
$$
\frac{1}{r + v + a\cdot \gamma + t \cdot \gamma^2 } - \frac{1}{r+ \gamma \cdot a}.
$$
Two terms for the same address a can never occur twice, since reads and writes can never update acc by adding an entry with t=0. This is only possible during the audit phase. Moreover, by removing the addresses in order and checking that they are unique will ensure that the negative term is removed re-inserted only once.

@adr1anh your summary and terminologies to formailize overall idea, is SUPER SUPER x 100 helpful for rest of us to speak on same math language 💯 💯 Thanks again for such a thorough and detail post. Based on your formalized terms, I can expand more on what current PR doing so it will helpful for other audience
Background & Implementation in (N)IVC
Giving multi-set W = R

W/R both encoded in LogUp sum

in this PR, TS polynomial are set as constant 1, so it equivalently checking 2 set equality via any element in one set is lookup ONCE in another set.

we can further expand W == R into W_initial_acc + W_acc = R_acc + R_audit_acc
obviously
W_initial_acc - R_audit_acc = R_acc - W_acc

To elaborate each terms

W_initial_acc: this is initial state of RAM
R_audit_acc: this is final state of RAM
R_acc, W_acc: both field value manipulated in IVC circuit by removing/adding rules

R_acc - W_acc != 0 in overwhelming probability.

Currently PR implement LogUp sumcheck with R_acc - W_acc as  initial claim on target formula W_initial_acc - R_audit_acc, while W_initial_acc, R_audit_acc can be both derived in audit phase via SNARK solution (no audit circuit yet)
So

cited from @adr1anh Check that acc = 0  in audit phase

Currently in audit phase, I am checking acc = R_acc - W_acc instead via logUp sumcheck
r, gamma derived from H_s
To derived challenges , theoretically in sound we should derived from h_R, h_W.
However in current PR i made an optimisation by just derived challenges from h_R only, according to 2 observed fact, informally sound analysis

in the end R = W need to be satisfied, so imply h_R = h_W.
entries adding to h_R, h_W are constraints by IVC circuit implicitly

Furthermore, h_R field elements can be splitted into two subset:

h_R  := h_R_ivc + h_R_audit
h_R_ivc := H({(v_i, a_i, t_i)…}), while tuple lists(v_i, a_i, t_i) are non-deterministic input from prover when perform read/write in step circuit
h_R_audit := final RAM state This set can be represented by non-deterministic polynomial commitment from prover via comitment({vi...}) and comitment({t_i…})

commitment({ai...}) are pre-known address vector [0, 1, 2, ….] so naturally in can be encode in verifier key or SNARK verifier flow.

For current PR implementation, h_R_ivc computation is implemented as Poseidon circuit in IVC (which lead to higher one time cost in step circuit) h_R_audit are from final SNARK
In the audit phase, challenges r gamma are derived from h_R, and comparing with prover supplied r gamma from public input.
Removing hashes
My proposed plan was to derived h_R_ivc tuples by a smaller scope audit circuit. Observed h_R_ivc can be derived from R_acc because both can be derived from raw set {(v_i, a_i, t_i)…}.  So in the audit circuit, we can

assert R_acc == logUp_sum([(bi, ai, ci), (bi, ai, ci),.... (bi, ai, ci)], r, gamma) // (bi, ai, ci) are non-deterministically
computed_r := RO([(bi, ai, ci), (bi, ai, ci),.... (bi, ai, ci), comitment({vi}), comitment({t_i…}]) // use (bi, ai, ci) to compute r, this part is similar to audit psuedo flow by @adr1anh https://github.com/lurk-lab/arecibo/pull/48#issuecomment-1906206682
computed_gamma := RO(computed_r)
assert r == computed_r
assert gamma == computed_gamma

Proposed audit circuit scope is smaller, just covering challenges integrity verification.
Limitation of proposed Removing hashes from IVC
One of the drawback for the solution is, if we implemented audit circuit in SRS bounded fashion, then we add a limitation cap that this audit circuit not handle massive read/write operations, for example zkEVM in riscv which can lead to billion number of read/writes. If we implemented in separate folding circuit, then the cost will still higher. I am still refer other literature you shared above and see other better alternative. If there are low cost alternative by still having challenge in folding circuit, or we can have challenge-free solution it will be super nice!

from @adr1anh: Unfortunately, we cannot avoid the hashes that are performed during the read and write operation by the circuit. ….(omitted partially)…. However, it is crucial that the tuple of every single insertion and removal is added to the transcript, even when it is seemingly added twice….

I am not pretty this description for why it’s infeasible. Because in my solution, every single insertion/removal is still adding to transcript, just it’s implemented in audit circuit. Audit circuit force the entries sum equal to R_acc, then it’s still being used to computed final challenge. Besides there is no chicken-egg soundness like computing challenge using challenge as input.
I believe what you point out is some critical soundness, and appreciate for further elaboration 🙏

Challenge derivation

in the end R = W need to be satisfied, so imply h_R = h_W.

Even if the two sets are equal, they are constructed in a different order and therefore the insertions into each set will lead to h_R != h_W. Unless you are talking here about the logUp accumulator acc_R and acc_W, in which case you're right that they are equal, but the accumulator has to be computed using r, gamma, and these need to be derived from every single element that is used to compute the acc. More precisely, we need r, gamma <- H(h_R, h_W), where the pre-image of h_R and h_W contains all the individual terms that make up acc_R, acc_W.
In the description of the protocol I wrote about, there is a possible optimization that reduces the number of hashes, where we only use a single hash-chain h_RW, where we only need to hash (a, v', v, t, t') for a write, and ( v', a, t, t') for a read to h_RW.  This is because we are doing the following operations in each operation:

Read: Add (a, v', t') to R and add (a, v', t) to W
Write: Add (a, v', t') to R and add (a, v, t) to W

During any read or write, the prover has to supply (v', t') non-deterministically, essentially an untrusted input from the prover. Therefore it must be added to the transcript h_RW. It seems that we may be able to get away from adding (v, t), since these values are derived by the IVC circuit. However, this requires some very specific conditions on the IVC circuit. I think we can write it as

The IVC circuit input z_0 needs to be added initially to the h_RW
Given z_0 and all the previously inserted entries (v', t'), it should be possible to deterministically derive (a, v, t).
The second condition seems hard to check thoroughly, and I'm not even sure what the exact requirement would be. Unless we can derive a proper definition and accompanying security proof, I think we should still add the "deterministic" values (a, t) to h_RW.

Hashing using PCS
Adding polynomial commitments to the transcript/hash accumulator h should provide the same guarantees as using a field hash function like Poseidon due to the binding property of Pedersen commitments.
At the moment though, we can only use polynomial commitments alongside a SNARK-like protocol where we can use Sumcheck. Long-term, I think we should be able to incorporate these directly into the arithmetization layer, so that the Poseidon hashes can be removed entirely from the circuit. The main idea is that we allow the R1CS circuit to have additional witness columns A, V, T which are committed to before we start synthesizing W, X. We could then write the circuit by considering Z' = [A, V, T, W, X]. For an IVC sequence of witness columns [Z'_0, ..., Z'_n], the X component would have to contain r, gamma, which would be derived the sequence of polynomial commitments [(comm(A_0), comm(V_0), comm(T_0)), ..., (comm(A_n), comm(V_n), comm(T_n))]. For those familiar with halo2, this would be akin to allowing committed instance columns that are used for lookups.
Both techniques are morally equivalent, yet the PCS approach requires significantly more engineering to be usable. However, it seems like this will be the ultimate way of making the construction as efficient as possible in the long term. Moreover, I think it will lead to a cleaner overall design, where the entirety of the lookup logic is contained in-circuit, which gives an easier way to "opt-out".

One of the drawback for the solution is, if we implemented audit circuit in SRS bounded fashion, then we add a limitation cap that this audit circuit not handle massive read/write operations

I don't think this is any more true than with a SNARK, and in fact it is more flexible. With the SNARK approach, we are forced to use a fixed-size memory and the run-time of the prover will always be the size of the array. With the circuit approach, we only need to remove entries that were actually accessed. This allows us to consider a much larger address space, allowing for a mapping with arbitrary keys. Given that the number of unique addresses is variable in each IVC execution, we can write the audit circuit as I did above to be recursive, which allows us to run it an arbitrary number of times, and each iteration can audit a fixed number of entries.
From a practical approach though, and given how much we still need to figure out about the construction in terms of efficiency and security, I think we should design the construction in such a way that it's efficiency can be better analyzed from a more theoretical perspective, which will help guide us towards an "optimal" solution that is also practical to implement.

I am not pretty sure about this description for why it’s infeasible. Because in my solution, every single insertion/removal is still adding to transcript, just it’s implemented in audit circuit. Audit circuit force the entries sum equal to R_acc, then it’s still being used to computed final challenge. Besides there is no chicken-egg soundness like computing challenge using challenge as input.

The problem is that each entry is loaded non-deterministically twice and could be different (once when it is written, and once when it is read). In the honest case, both of these values will be equal, but we must assume it is not the case and protect against this case. We need to prevent against the case where the value and address being written can be chosen non-deterministically as they could then be chosen in a way that depends on r, gamma and would allow us to make the acc_W equal anything, including acc_R.
It may technically be possible to argue that the entry added to the write set W does not need to be hashed, but we would have to argue more thoroughly that it is ok.
As I'm writing this though, I think we might be able to argue that adding t to h_RW is not necessary if we initialize h_RW = H(h_RW_prev, t_init), where t_init is the current time-stamp at the start of the IVC circuit.
The address a seems a lot harder to argue since it computed by the circuit in ways that may not be linked to the IVC input z.

I am rebasing this here and will make a PR in the next few days. I also moved/renamed some things into their own files so we do not have a bunch of V2 in one long file.

rebased and PRed here
https://github.com/lurk-lab/arecibo/pull/386


add sha256 hash check for powershell

Signed-off-by: Batuhan Apaydın<EMAIL_ADDRESS>Fixes #2

https://stackoverflow.com/questions/63396620/how-can-i-compare-a-files-sha256-hash-in-powershell-to-a-known-value

thanks for a quick contribution, we need to refactor a bit the SHA256SUMS contains all the files and not just one
see https://github.com/cpanato/vault-installer/actions/runs/3740062309/jobs/6348045175 it returns false but it passes the check :/

is there any update on this @cpanato 👋

@developer-guy can you please rebase?

of course my friend!

thank you!


Add test case for save method.

Fixes #{issue number here}
Changes proposed in this pull request:

Add test case for save method.

This PR looks abandoned to me so I'm closing it. Happy to reopen if you want to go back to it.


JSON Schematic Format

The format is just a proof of concept at the moment and needs feedback from people that might be interested.
An up to date example of what the schematic will eventually look like can be found and discussed here. Please post any concerns/ideas about the format in the gist.

Maybe at a later time if anyone shows any interest.


⚠ Bofrost.be has degraded performance

In 87da755, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1805 ms

Resolved: Bofrost.be performance has improved in bf0e604.


add a rotation input to vr camera (additive, like position), resolves #758

also remove FOV from presets, as the default fov is already 90 and shouldn't really be messed with, especially in VR where this is overridden anyway and

lgtm


Change default action of TextField to resemble form submission

Add a listener to prevent user from inputting more than 5 digits to textfield

Codecov Report

Merging #165 (8280279) into master (2f781a5) will decrease coverage by 0.84%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master     #165      +/-   ##
============================================
- Coverage     69.80%   68.96%   -0.85%
  Complexity      708      708
============================================
  Files           112      112
  Lines          2451     2481      +30
  Branches        293      298       +5
============================================
  Hits           1711     1711
- Misses          630      660      +30
  Partials        110      110

Impacted Files
Coverage Δ

src/main/java/seedu/address/ui/SupplyItemCard.java
0.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Wizard creates double underscore in Rollup API Name which causes error thinking it is namespace

Describe the bug
The wizard creates a double underscore in the rollup API name when typing in the label " - " (space dash space) and that causes an error to think it's referencing a namespace when trying to save the rollup

We are rolling out a new LWC based wizard in v2.21 and will deprecate the VisualForce editor & wizard in the future. I'm going to marked this as "will not do" as this feature will not be further developed.
If you get a chance please take a look at the new v2.21 and brand new wizard available there.


added avatar image and quote from Kent Huang

added avatar image and quote from Kent Huang

This is broken please fix and try again.


multiple pacman installed libraries have invalid cmake links

when using msys to install packages with pacman, some of the cmake files in those packages (vtk and one other, ...glib? ...zlib?) including a non-existent directory called "building" before everything else when trying to link to other files. This means cmake can't compile these packages.
pacman: Pacman v4.2.1-503-g62b54-dirty - libalpm v10.0.1
vtk: mingw-w64-x86_64-vtk 6.3.0-2
gcc: mingw-w64-x86_64-gcc 5.3.0-2
cmake: mingw-w64-x86_64-cmake 3.4.1-1 ; mingw-w64-x86_64-extra-cmake-modules 5.18.0-1

Move this to mingw-packages.

moved.


Add Import Extensions Documentation

Add Documentation For Importing Extensions
SwifterSwift is a large framework and sometimes our extensions conflict with extensions provided in other libraries.
Example: SwifterSwift provides many Date extensions that may conflict with a library that provides a calendar UI element or other date tools.
How can we resolve conflicts?
By importing SwifterSwift components at the type level instead of the entire framework:
import SwifterSwift // imports all SwifterSwift extensions and prone to conflict
import struct SwifterSwift.Date // only imports SwifterSwift Date extensions

@SD10 The above solution didn't work for me, I'm still getting Ambiguous use of 'hour' error although I imported just UIColor from SwifterSwift

import UIKit
import DateToolsSwift
import class SwifterSwift.UIColor

class ViewController: UIViewController {

	override func viewDidLoad() {
		super.viewDidLoad()

		let date: DateToolsSwift.Date = Date()
		let hour = date.hour
	}

}

@omaralbeik Thanks for testing this!
I have some ideas as to what's happening and will look into it further.

Looks like adding module_name = "SwifterSwift" in SwifterSwift.podspecs will help with this


V2 certificate format

This is a near complete implementation of an ipv6 enabled overlay network. There are a handful of pull requests targeting this branch in addition to a number of incomplete issues within this PR.
I have tried to highlight the main trouble spots with comments on the PR and in addition here are my internal notes:

nebula-cert is defaulting to mint both v1 and v2 certificates and nebula config is defaulting to transmit v1 certificates if both are present. This will lead to a situation where net new adopters will be stuck using v1 certificates. Should this be the default behavior?

Currently nebula wont allow you to remove a v1 certificate on reload but this means a restart is required to complete a migration to v2. A comment on this review highlights this, we should allow it.

Currently there are a few spots (control, ssh) where we return the certificate in use but they only return 1 cert, should we leave it returning the default or force folks to consume both? How do we indicate which one is default?

Every time we encounter a certificate we need to sort the networks as well as only record the union of applicable addresses. See #1261

We need to take extra care to ensure we never allow a ipv4 mapped ipv6 address. This is not currently addressed in this PR.

There may be an opportunity to improve the lighthouse protocol by directly using the MarshalBinary and UnmarshalBinary functions for netip objects. This is more of a nit than anything but we should agree on this before merging.

nebula-cert sign help text needs to talk about primary vpn address (or the union of effective addresses) being the first one after being sorted and what that means for tunnels.

The firewall config still uses naming like ca-sha should we alias this to ca-fingerprint to normalize the verbiage?

The firewall config still uses naming like ip should we alias this to network to normalize the verbiage

Normalize all names for Marshal* and Unmarshal*, we have some From*/To* and others without From/To.

Should we use this moment to swap p256 keys to use their compressed form? Saves ~32 bytes.

Once this is merged the upgrade path should likely be:

Give every host dual certs and leave the default pki.default_version to 1. Everything continues to use v1 protocols.
Switch all am_relay: true hosts pki.default_version to 2. Relays can now initiate v2 tunnels, but this is an uncommon flow.
Switch all non lighthouse hosts pki.default_version to 2. Lighthouses will reply to v2 hosts using v2 protocols.
Switch all lighthouses pki.default_version to 2.
Remove v1 certs from all hosts

This PR no longer exports the certificate structs; neither v1 or v2:
type certificateV2 struct {
	details detailsV2

	// RawDetails contains the entire asn.1 DER encoded Details struct
	// This is to benefit forwards compatibility in signature checking.
	// signature(RawDetails + Curve + PublicKey) == Signature
	rawDetails []byte
	curve      Curve
	publicKey  []byte
	signature  []byte
}

type detailsV2 struct {
	name           string
	networks       []netip.Prefix
	unsafeNetworks []netip.Prefix
	groups         []string
	isCA           bool
	notBefore      time.Time
	notAfter       time.Time
	issuer         string
}

type certificateV1 struct {
	details   detailsV1
	signature []byte
}

type detailsV1 struct {
	name           string
	networks       []netip.Prefix
	unsafeNetworks []netip.Prefix
	groups         []string
	notBefore      time.Time
	notAfter       time.Time
	publicKey      []byte
	isCA           bool
	issuer         string

	curve Curve
}

Before:

type NebulaCertificate struct {
	Details   NebulaCertificateDetails
	Signature []byte

	// the cached hex string of the calculated sha256sum
	// for VerifyWithCache
	sha256sum atomic.Pointer[string]

	// the cached public key bytes if they were verified as the signer
	// for VerifyWithCache
	signatureVerified atomic.Pointer[[]byte]
}

type NebulaCertificateDetails struct {
	Name      string
	Ips       []*net.IPNet
	Subnets   []*net.IPNet
	Groups    []string
	NotBefore time.Time
	NotAfter  time.Time
	PublicKey []byte
	IsCA      bool
	Issuer    string

	// Map of groups for faster lookup
	InvertedGroups map[string]struct{}

	Curve Curve
}

It is going to be a big breaking change and inconvenience for anyone who has used and uses the libraries. Would prefer to keep these exported.

I see now it’s switched to a factory approach, with TBSCertificate that is exported. 👍
A few thoughts to consider:
A helper function for converting a Certificate back to a TBSCertificate would be useful to complete the factory loop and allow manipulation of existing signed certificates (as unsigned ones).
There are also some Marshal functions like Marshal(), but without a corresponding Unmarshal(). I imagine most people are using pem encoded certs, but if Marshal() is exposed there should probably be a corresponding Unmarshal().


fix the outdate  link in  using-openstack-cloud-controller-manager.md

Signed-off-by: maoyangLiu<EMAIL_ADDRESS>[occm] Fix the outdate link in using-openstack-cloud-controller-manager.md

/lgtm

/release-note-none

/approve


[Improvement]: Give compile-time error for negative startIndex in the regexp langlib functions

Description
$title
Describe your problem(s)
regexp:Span? _ = regExpr1.find(str1, -5); Here better to give CE instead of RE.
Describe your solution(s)
No response
Related area
-> Compilation
Related issue(s) (optional)
No response
Suggested label(s) (optional)
No response
Suggested assignee(s) (optional)
No response

This shall also be addressed in findAll, findGroups(), findAllGroups methods as well.

A similar approach to what's used in the string:substring() could be used here.
Therefore, a Runtime Error shall be expected when a negative value is passed here.


QSIPREP preprocessing pipeline "AttributeError: Can only use .str accessor with string values!"

Hello everyone,
I'm putting this message here, but I'm not sure it's really a problem with QSIPREP as such...
I am trying to use qsiprep to create my own preprocessing pipeline, but the execution crashes pretty fast...
      * Participant list: ['CTR01'].
(base) bash-4.2$ ./qsiprep.sh
Switching to home... Launching QSIPREP...
230809-15:25:19,451 nipype.workflow INFO:
         Running with omp_nthreads=8, nthreads=32
230809-15:25:19,459 nipype.workflow IMPORTANT:

    Running qsiprep version 0.18.1:
      * BIDS dataset path: /data.
      * Participant list: ['CTR01'].
      * Run identifier: 20230809-151721_144e3b1b-1514-4beb-b929-28db4f42a5ff.

230809-15:30:09,986 nipype.workflow INFO:
         Running nonlinear normalization to template
230809-15:30:15,33 nipype.workflow INFO:
         Combining all dwi files within each available session:
230809-15:30:15,33 nipype.workflow INFO:
                - 2 scans in session 01
230809-15:30:15,33 nipype.workflow INFO:
                - 0 scans in session pre
230809-15:30:15,33 nipype.workflow INFO:
                - 0 scans in session pst
230809-15:30:15,81 nipype.workflow INFO:
         [{'dwi_series': ['/data/sub-CTR01/ses-01/dwi/sub-CTR01_ses-01_dir-PA_dwi.nii.gz'], 'dwi_series_pedir': 'j', 'fieldmap_info': {'suffix': 'rpe_series', 'rpe_series': ['/data/sub-CTR01/ses-01/dwi/sub-CTR01_ses-01_dir-AP_dwi.nii.gz']}, 'concatenated_bids_name': 'sub-CTR01_ses-01'}]
230809-15:30:16,224 nipype.workflow IMPORTANT:
         Creating dwi processing workflow "dwi_preproc_ses_01_wf" to produce output sub-CTR01_ses-01 (1.17 GB / 119 DWIs). Memory resampled/largemem=1.66/1.84 GB.
Process Process-2:

Error log:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/cli/run.py", line 950, in build_qsiprep_workflow
    retval['workflow'] = init_qsiprep_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/workflows/base.py", line 228, in init_qsiprep_wf
    single_subject_wf = init_single_subject_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/workflows/base.py", line 709, in init_single_subject_wf
    dwi_preproc_wf = init_dwi_preproc_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/workflows/dwi/base.py", line 315, in init_dwi_preproc_wf
    pre_hmc_wf = init_dwi_pre_hmc_wf(scan_groups=scan_groups,
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/workflows/dwi/pre_hmc.py", line 143, in init_dwi_pre_hmc_wf
    merge_plus = init_merge_and_denoise_wf(raw_dwi_files=plus_files,
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/workflows/dwi/merge.py", line 153, in init_merge_and_denoise_wf
    dwi_df = get_acq_parameters_df(raw_dwi_files)
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/workflows/dwi/merge.py", line 506, in get_acq_parameters_df
    merged_acq_params['PhaseEncodingDirection'].str.replace("-","")
  File "/usr/local/miniconda/lib/python3.8/site-packages/pandas/core/generic.py", line 5902, in __getattr__
    return object.__getattribute__(self, name)
  File "/usr/local/miniconda/lib/python3.8/site-packages/pandas/core/accessor.py", line 182, in __get__
    accessor_obj = self._accessor(obj)
  File "/usr/local/miniconda/lib/python3.8/site-packages/pandas/core/strings/accessor.py", line 181, in __init__
    self._inferred_dtype = self._validate(data)
  File "/usr/local/miniconda/lib/python3.8/site-packages/pandas/core/strings/accessor.py", line 235, in _validate
    raise AttributeError("Can only use .str accessor with string values!")
AttributeError: Can only use .str accessor with string values!

Thank you for your help!

could you try running with --containall instead of --cleanenv?

Same error :/
I've also tried to delete this two options:
--hmc_transform Affine \
--hmc_model eddy \

But still the same error

The problem seems to be in your json sidecars. The PhaseEncodingDirection needs to be a string. More details are here I strongly suggest not using --skip-bids-validation when you're trying out qsiprep on new data


⚠ Cioos National CKAN Sitemap has degraded performance

In 263c632, Cioos National CKAN Sitemap (https://catalogue.cioos.ca/sitemap/sitemap-1.xml) experienced degraded performance:

HTTP code: 200
Response time: 12277 ms

Resolved: Cioos National CKAN Sitemap performance has improved in 622e2c9 after 7 minutes.


Langchain & Langsmith integration for UpTrain

The idea is to provide a single-line integration for anyone using Langchain & Langsmith to be able to evaluate their LLM applications using UpTrain

@sourabhagr I can take this up, does this follow similar pattern as the llamaindex integration here or if you could elaborate a little?

yes @sky-2002 . It should follow a similar pattern like how we are doing it in llamaindex integration.

@sky-2002 Did you make any progress with this? Else I can take up this Issue and work on it.

@rumbleFTW Sure, Please go ahead

@ashish-1600 the llamaindex file allow you to only use the EvalLlamaIndex class to evaluate using Llamaindex.
The correct way to do this is using the debughandler for llamaindex and Callback handler for Langchain.
@sourabhagr I can pick this up as discussed before on our call!


updates dex poll changes and add gateway list

Before submitting this PR, please make sure:

[x] Your code builds clean without any errors or warnings
[x] You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:

Dex Poll changes
Add gateway connector to the list of tiers
Add gateway list command

Tests performed by the developer:
Tips for QA testing:

Merged the PR in behalf of Foundation team to development branch and the next Hummingbot version 1.12.0