Loading external javascript on hosted Nuxt app (AWS Amplify)

Environment

Operating System: Linux
Node Version:     v16.19.1
Nuxt Version:     3.4.3
Nitro Version:    2.3.3
Package Manager<EMAIL_ADDRESS>Builder:          vite
User Config:      app
Runtime Modules:  -
Build Modules:    -

Reproduction
You can find the code here: https://github.com/simoroma/test_external_javascript
Also hosted on AWS Amplify here: https://main.d29qqvykn4nywd.amplifyapp.com/
Describe the bug
This happens on built Nuxt code that is hosted on AWS Amplify. I honestly don't know if it is an AWS Amplify issue. Though, everything else works correctly.
I have an external javascript file added to nuxt.config.ts:
// https://nuxt.com/docs/api/configuration/nuxt-config
export default defineNuxtConfig({
  app: {
    head: {
      script: [
        {
          hid: "my_external_javascript",
          src: "my_external_javascript.js",
        },
      ],
    },
  },
});

And two pages in the folder pages: index.vue and page.vue.
If I load / (the root) the javascript is loaded correctly. If I load /page the javascript is not loaded.
Additional context
No response
Logs
No response

Path to files in public folder should starts with / character like src: /my_external_javascript.js

Thanks @ucw ! It was really matter of adding only /


feat(wallets): reuse session_id across multiple TrezorSigner

Component
Forge
Describe the feature you would like
https://github.com/foundry-rs/foundry/blob/398ef4a3d55d8dd769ce86cada5ec845e805188b/crates/wallets/src/multi_wallet.rs#L369-L375
Currently, every new instance of TrezorSigner is created with their own session_id. If a passphrase is enabled on the device, it means requesting it for every mnemonic index.
Ideally, each TrezorSigner reuses the first created session_id.
Additional context
No response

May I be assigned to this?

Could I take on this issue?

Hello @joshieDo
can i take this issue
I can modify the function to cache and reuse a single session ID.

I'd be happy to do this.


ARM Template Options for Adding Key to KeyVault is not documented

Official documentation for ARM Template Reference Guide doesn't have Keys resource details. However for achieving the same I found there is a 101 template.
ARM Template Documentation: https://docs.microsoft.com/en-us/azure/templates/microsoft.keyvault/2019-09-01/vaults/secrets
Adding KV Key 101 Template : https://github.com/Azure/azure-quickstart-templates/blob/master/201-machine-learning-encrypted-workspace/prereqs/prereq.azuredeploy.json

I think you should report issue to https://github.com/MicrosoftDocs/feedback/issues

No ARM manifest reference pages are generated based on swagger, so this is the correct place. KV team should update their swagger spec

Are you looking for this doc ?
https://docs.microsoft.com/en-us/azure/key-vault/keys/quick-create-template?tabs=CLI

I am looking for official ARM template resource documentation containing this "Microsoft.KeyVault/vaults/keys" resource documented

This link https://docs.microsoft.com/en-us/azure/templates/microsoft.keyvault/2019-09-01/vaults/secrets should have keys resource mentioned as well.

Hey @msmbaldwin, did you get a chance to add the template to that repo? Is there a PR to follow?

@msmbaldwin can you look into this request

@jlichwa can you help review this?

@mvvsubbu documentation is generated based on swagger (stable version) since it is still in preview there will not be documentation till GA.

Ok thanks for updating.

#please-close


[Cypress] Make cypress UI tests green again

Several items to tackle and investigate:

About page tests failing again in the reload part.
Some tests in Prague runners seem to work ok when running individually but fail when running nightly.

PR for disabling reload on About tests here: https://github.com/epinio/epinio-end-to-end-tests/pull/380

Also spaced the different test workflows' starting times here: https://github.com/epinio/epinio-end-to-end-tests/pull/382

It seems the above did not helped after today's results on NUE runners:
STD-UI-Latest-Chrome #670
Applications: error verifying credentials: error while connecting to the Epinio server: empty
Almost all the rest: AssertionError: Timed out retrying after 240000ms: Expected to find content: '===> EXPORTING' but never did.
 STD-UI-Latest-Firefox #553
AssertionError: Timed out retrying after 240000ms: Expected to find content: '===> EXPORTING' but never did.
AssertionError: Timed out retrying after 10000ms: Expected to find content: 'Namespaces 3' within the element: [ <h1>, 2 more... ] but never did.
NS_ERROR_UNEXPECTED
Rancher-UI-1-Chrome #760
s3gw: AssertionError: Timed out retrying after 240000ms: Expected to find content: '===> EXPORTING' but never did.
minio: stucked. (had to pull out)
Rancher-UI-2-Firefox #751
1 test failing with AssertionError: Timed out retrying after 240000ms: Expected to find content: '===> EXPORTING' but never did.

This issue above due to: https://github.com/epinio/epinio/issues/2583; already solved.
Apart from that Rancher Chrome tests, presented an issue with runner gha08-runner-0so I moved it and open this issue: https://github.com/epinio/epinio/issues/2598
After all this, status of tests are green again, so closing issue:


F401 triggers based on renaming imports

Ruff's F401 triggers if you import something and rename it, but not if you rename it but keep the name the same.
For example, this file doesn't use any of it's imports:
from . import clients as clients
from .config import config as config
from .config_training import config as config_training # E: F401 [*] `.config_training.config` imported but unused
from .thing import other as foobar # E: F401 [*] `.thing.other` imported but unused
from . import x as y # E: F401 [*] `.x` imported but unused
from a import b as c # E: F401 [*] `a.b` imported but unused

Which I'm running in VI via ALE using the LSP to get the comments, but I can also run it manually:
$ ruff check --isolated --select=F401 example.py
example.py:3:40: F401 [*] `.config_training.config` imported but unused
example.py:4:29: F401 [*] `.thing.other` imported but unused
example.py:5:20: F401 [*] `.x` imported but unused
example.py:6:20: F401 [*] `a.b` imported but unused
Found 4 errors.
[*] 4 potentially fixable with the --fix option.

Notice that the error doesn't trigger if you from a import b as b, but only if you use from a import b as c. This was from an __init__.py file that didn't set __all__, which isn't a good pattern, and it goes away as it should if you set __all__). But I think it should be consistent. I'm pretty sure the problem is that from a import b as b is being treated as from a import b; b = b which "uses" b, making it not unused. That's a guess.

We allow from a import b as b because it's respected by type checkers as a way to "export" the name similar to __all__. See the Pyright documentation section that discusses redundant aliases.

This looks correct and intended. Whilst both import examples lead to unused symbol in the current module, both also lead to symbols that exist.
from example import clients and from example import y are both valid at runtime.
To keep public interfaces sane, type checkers (and some import linters like pycln, isort, etc.) have a bit of heuristic to define what is private and public, and the form import X as X is how you explicitly flag an import as an intended re-export.

That being said, mixing explicit re-exports and __all__ isn't a great pattern, and maybe a rule dedicated to that might satisfy your needs and expectations?

Okay, I see why it happens, then, but the code that someone originally was confused about:
from .config import config as config
from .config_training import config as config_training

still seems surprising - you can't "reexport" the second config, since it can't keep the original name. The current form obviously was trying to reexport, but if it has to be renamed, that doesn't work then.
I'll just suggest that they use __all__, explicit is better than implicit. And makes the imports look less odd. :)

Thanks, by the way!

maybe a rule dedicated to that might satisfy your needs and expectations?

I was thinking the best solution would be to not do the explicit reexports, but move entirely to __all__.
I wouldn't mind having a rule that enforced all files (that don't start with an underscore, anyway) have an __all__, and all non-__init__.py files have a def __dir__() too (actually, I already enforce exactly this in pytest for scikit-build-core). But that's pretty specific to following this pattern.


vkCmdExecuteCommands marks the buffer "in flight" right away

I'm recording a primary command buffer that executes a secondary command buffer (in pseudocode):
vkBeginCommandBuffer(secondaryBuf, &secondaryBeginInfo)
vkEndCommandBuffer(secondaryBuf)

vkBeginCommandBuffer(primaryBuf, &primaryBeginInfo)
vkCmdExecuteCommands(primaryBuf, 1u, &secondaryBuf)
vkEndCommandBuffer(primaryBuf)

// primary buffer is NOT submitted for execution

vkDestroyCommandPool(device, cmdPool, NULL)

However, the command buffer is never submitted to a queue. Therefore, neither primary nor secondary buffer is actually executed. Regardless, I'm getting this message when I free the command buffers:

DS(ERROR): object: 0x601f560 type: 6 location: 5904 msgCode: 54: Attempt to free command buffer (0x601f560) which is in use.

I think the problem is in vkCmdExecuteCommands() in core_validation.cpp, where the secondary buffer is being added to globalInFlightCmdBuffers right away. Instead, I would expect this to happen only after the primary buffer was submitted to a queue.
EDIT
I'm also getting this:

DS(ERROR): object: 0x5f99f40 type: 6 location: 10119 msgCode: 53: Attempt to simultaneously execute CB 0x5f99f40 w/o VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT set!

even though the secondary command buffer is used only once. Note though, that this is intermittent and happens only when running without debugger attached.

Submitted #345 to fix this issue

Fix was cherry-picked and pushed to master


Atom can't be used with hungaian keyboard!

I cant type the '{' and '[' keys.

Ohh I see we already have this:  #35


Missing breaking changes in v2.0.0 changelog.

Hi there,
I think you've forgotten to inform us of an important change in v2.0.0: the name of the algorithm option has changed from SHA1 / SHA256 / etc. to SHA-1, SHA-256, etc.
PS: I also see that your README and some of your comments in the source code refer to the old format.
✌

Thanks! Release notes updated. Would you like to make a pull request for the readme?

Thanks @kentcdodds, here is the PR: https://github.com/epicweb-dev/totp/pull/14


adjust limits in deployment causes OOMKilled

The default limits in the Deployment causes a OOMKilled.
As a workaround i removed the limits. Is it not an idea to have a custom crd to adjust the patch-operator deployment?

Assuming you are using OLM Subscriptions to provision the operator, it is possible to configure custom requests and limits on the subscription resource spec.
spec:
  config:
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

That said, if the defaults are too low under normal conditions, it would be best for us to tune them, so appreciate that feedback. Are there any special circumstances that are causing your operator instance perform any more processing than usual? Or are you seeing OOM out of the box? Also, I assume the OOM is the controller container, but please note if the OOM container is kube-rbac-proxy.

Thanks @davgordo for claryfing that.
I will look into that. I see the controller pod is using almost 1GiB of RAM with just a few patches

Oh, that doesn't sound expected under most conditions. If you have some time to provide the patch resource configurations that cause these memory conditions, that would be interesting for us to study, and possibly identify a problem with memory management.

I get OOMKilled if i install the operator and not even have a patch object defined.

I am seeing a low memory footprint at rest on a fairly standard cluster.

Perhaps there is information you can provide about your environment that could help us recreate the memory problem?
Some places to start:

What namespace did you install the operator?
In the Patch Operator Subscription, what is showing as the Installed version? (latest should be patch-operator:v1.0.4)
Do you see logs after initial startup in the operator manager container?

The operator get's installed in namespace patch-operator. The subscription uses the latest channel wich is v1.0.4.
I reinstalled the operator it didn't OOMKilled right after the installation. I will adjust the subscription to allow it to use more resources.
I think for now this issue can be closed.


Ng serve fails with Angular CLI version 9.1.4 and Angular version 9.1.5

🐞 Bug report
Command (mark with an x)

[ ] new
[ ] build
[x] serve
[ ] test
[ ] e2e
[ ] generate
[ ] add
[ ] update
[ ] lint
[ ] xi18n
[ ] run
[ ] config
[ ] help
[ ] version
[ ] doc

Is this a regression?

 Yes, the previous version in which this bug was not present was: Angular CLI version 9.1.4 and Angular version 9.1.4
Description
I recently installed node version 12.16.3 LTS, then Angular CLI using the command npm install -g @angular/cli. Then I created a new project with the command ng new myProject.
These steps installed CLI version 9.1.4 and Angular version 9.1.5. After the installation, I tried to launche the project wih the command ng serve and get the error bellow.
🔬 Minimal Reproduction

🔥 Exception or Error

><EMAIL_ADDRESS>start C:\Users\Public\workspace\test
> ng serve

Compiling @angular/core : es2015 as esm2015
Compiling @angular/animations : es2015 as esm2015
Compiling @angular/compiler/testing : es2015 as esm2015
Compiling @angular/core/testing : es2015 as esm2015
Compiling @angular/common : es2015 as esm2015
Compiling @angular/animations/browser : es2015 as esm2015
Compiling @angular/platform-browser : es2015 as esm2015
Compiling @angular/common/http : es2015 as esm2015
Compiling @angular/common/testing : es2015 as esm2015
Compiling @angular/platform-browser-dynamic : es2015 as esm2015
Compiling @angular/platform-browser/testing : es2015 as esm2015
Compiling @angular/router : es2015 as esm2015
Compiling @angular/animations/browser/testing : es2015 as esm2015
Compiling @angular/common/http/testing : es2015 as esm2015
Compiling @angular/forms : es2015 as esm2015
Compiling @angular/platform-browser/animations : es2015 as esm2015
Compiling @angular/platform-browser-dynamic/testing : es2015 as esm2015
Compiling @angular/router/testing : es2015 as esm2015

chunk {main} main.js, main.js.map (main) 2 kB [initial] [rendered]
chunk {polyfills} polyfills.js, polyfills.js.map (polyfills) 676 bytes [initial] [rendered]
chunk {runtime} runtime.js, runtime.js.map (runtime) 6.15 kB [entry] [rendered]
chunk {styles} styles.js, styles.js.map (styles) 12.7 kB [initial] [rendered]
chunk {vendor} vendor.js, vendor.js.map (vendor) 339 kB [initial] [rendered]
Date: 2020-05-08T15:28:59.528Z - Hash: ce7b123bfb39e55c84dd - Time: 6563ms

ERROR in TypeError: Cannot read property 'flags' of undefined
    at resolveAlias (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:36513:37)
    at checkAliasSymbol (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:64502:26)
    at checkImportBinding (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:64534:13)
    at checkImportDeclaration (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:64552:29)
    at checkSourceElementWorker (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:64961:28)
    at checkSourceElement (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:64800:17)
    at Object.forEach (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:317:30)
    at checkSourceFileWorker (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:65123:20)
    at checkSourceFile (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:65091:13)
    at getDiagnosticsWorker (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:65179:17)
    at Object.getDiagnostics (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:65165:24)
    at C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:98703:85
    at runWithCancellationToken (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:98665:24)
    at getBindAndCheckDiagnosticsForFileNoCache (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:98691:20)
    at getAndCacheDiagnostics (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:98956:26)
    at getBindAndCheckDiagnosticsForFile (C:\Users\Public\workspace\test\node_modules\typescript\lib\typescript.js:98688:20)
** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **

🌍 Your Environment

Angular CLI: 9.1.4
Node: 12.16.3
OS: win32 x64

Angular: 9.1.5
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router
Ivy Workspace: Yes

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.901.4
@angular-devkit/build-angular     0.901.4
@angular-devkit/build-optimizer   0.901.4
@angular-devkit/build-webpack     0.901.4
@angular-devkit/core              9.1.4
@angular-devkit/schematics        9.1.4
@angular/cli                      9.1.4
@ngtools/webpack                  9.1.4
@schematics/angular               9.1.4
@schematics/update                0.901.4
rxjs                              6.5.5
typescript                        3.8.3
webpack                           4.42.0

Anything else relevant?

+1 getting this too.
I assume because the CLI is stuck in build-limbo again.

I assume because the CLI is stuck in build-limbo again.

Not correct. There is a regression in the Framework that we are working on resolving. See https://github.com/angular/angular/issues/36992#issuecomment-625856812

Please follow along at that issue.


koyama's homework of 20150624

add screen shot of GAZEBO and Rviz

my student number is 48-156517.


ReferenceError: Strict mode forbids implicit creation of global property 'service'

Strict mode causes bug in postserver.js example

I think I've encountered a similar error. It looks like PhantomJS behaving as per strict mode, but neither Firefox nor Chrome report any errors.


feat(chat): implement provider auto-redirect to skip provider selection page

Description:
Auto-redirect to default provider, bypassing the provider selection page for improved user flow.
Issues:
UI changes
Checklist:

[x] the pull request name complies with Conventional Commits
[x] the pull request name starts with fix(<scope>):, feat(<scope>):, feature(<scope>):, chore(<scope>):, hotfix(<scope>): or e2e(<scope>):. If contains breaking changes then the pull request name must start with fix(<scope>)!:, feat(<scope>)!:, feature(<scope>)!:, chore(<scope>)!:, hotfix(<scope>)!: or e2e(<scope>)!: where <scope> is name of affected project: chat, chat-e2e, overlay, shared, sandbox-overlay, etc.
[ ] the pull request name ends with (Issue #<TICKET_ID>) (comma-separated list of issues)
[x] I confirm that do not share any confidential information like API keys or any other secrets and private URLs

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review

/deploy-review


Bumped glob-parent dependency to 5.1.2

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 87.292%

Totals

Change from base Build<PHONE_NUMBER>:
0.02%

Covered Lines:
9382

Relevant Lines:
10380

💛  - Coveralls


Initial API and UI

Really crappy on both front- and back-end, but at least it provides a UI to input a string (complete with a default string to make it easy to show it does something) and displays the complexified result. More details about potential approaches to complexification are provided in the code comments.
No tests whatsoever. Maybe one day.

I'll branch off of this to make client code something that's buildable rather than static.  May shove server stuff in to a subdirectory just to make it somewhat clean.

Totally agree.

On Feb 25, 2017, at 11:34 PM, Zack Birkenbuel<EMAIL_ADDRESS>wrote:
@zbirkenbuel commented on this pull request.
In index.js:

const port = process.env.PORT || 8080

-const requestHandler = (request, response) => {

response.end('OK')
+const restify = require('restify');

+const transform = (request, response, next) => {

var s = request.params.s;
// Step 1: Measure existing complexity.
// For a first pass, let's just use string length.
const initialComplexity = s.length;

// Step 2: Do stuff.
// Some ideas:
// 1. Use a thesaurus
// 2. Translate to and back from another language
I like the thesaurus idea more than this. The idea is to make it pompous, not gibberish.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Also totally agree.

On Feb 25, 2017, at 11:42 PM, Zack Birkenbuel<EMAIL_ADDRESS>wrote:
I'll branch off of this to make client code something that's buildable rather than static. May shove server stuff in to a subdirectory just to make it somewhat clean.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.


Allow building X11 without Vulkan

This limit was likely introduced when Vulkan was the only option.
After some small testing this seems to work fine, although the experience could be improved, as one can still create a forward plus project and run it, although they will be greeted by an error.

Thanks!


Parser seems to swallow all preprocessor directives

When trying to parse some shader code, I noticed that any preprocessor directives were missing from the AST.
Minimal reproducible example
fn main() {
    use glsl_lang::{
        ast::TranslationUnit,
        parse::DefaultParse,
        transpiler::glsl::{show_translation_unit, FormattingSettings},
    };

    // Define and parse shader code
    let source = "#version 460\n#define TEST 3\nvoid main() {\n}\n";
    let ast = TranslationUnit::parse(&source).unwrap();

    // Directly print it out again
    let mut transpiled = String::new();
    show_translation_unit(
        &mut transpiled,
        &ast,
        (&FormattingSettings::default()).into(),
    )
    .unwrap();
    println!("{transpiled}");
}

Expected behavior
This program should print out the shader code as it was before, it should look like this:
#version 460
#define TEST 3
void main() {
}

Actual behavior
The actual output looks like this:
void main() {
}

The preprocessor directives just get swallowed. If I add them manually to the ast like so:
fn main() {
    use glsl_lang::{
        ast::{
            ExternalDeclarationData, IdentifierData, Node, PreprocessorData,
            PreprocessorDefineData, PreprocessorVersionData, TranslationUnit,
        },
        parse::DefaultParse,
        transpiler::glsl::{show_translation_unit, FormattingSettings},
    };

    let source = "#version 460\n#define TEST 3\nvoid main() {\n}\n";
    let mut ast = TranslationUnit::parse(&source).unwrap();

    let version = Node::new(
        ExternalDeclarationData::Preprocessor(Node::new(
            PreprocessorData::Version(Node::new(
                PreprocessorVersionData {
                    version: 460,
                    profile: None,
                },
                None,
            )),
            None,
        )),
        None,
    );
    let define = Node::new(
        ExternalDeclarationData::Preprocessor(Node::new(
            PreprocessorData::Define(Node::new(
                PreprocessorDefineData::ObjectLike {
                    ident: Node::new(IdentifierData("TEST".into()), None),
                    value: String::from("3"),
                },
                None,
            )),
            None,
        )),
        None,
    );
    ast.0.insert(0, version);
    ast.0.insert(1, define);

    let mut transpiled = String::new();
    show_translation_unit(
        &mut transpiled,
        &ast,
        (&FormattingSettings::default()).into(),
    )
    .unwrap();
    println!("{transpiled}");
}

Then the preprocessor directives are printed like expected, so this should really be a problem with parsing and not transpiling/printing.
Non bug explainations
Maybe there is a feature flag or a parsing setting that I have overlooked, that just en-/disables the parsing of preprocessor directives, but I couldn't find anything.
I hope this helps in any way :)

Thanks for the detailed bug report! Unfortunately there isn't an easy answer to this, but I'll still try to explain.
The main reason this is an issue is that the main crate, glsl-lang, only represents an AST (Abstract Syntax Tree) for historical reasons. I added position information to the nodes in the AST, so technically with node positions and the original source you could rebuild the source, including comments (they are stripped by the preprocessing/lexing stage), etc.
The transpiling part is based solely on the AST (again, historical reasons) so as you noticed with your second example, if the preprocessor directives are present in the AST, they'll get printed out as expected.
The issue is, the AST enforces a rigid structure that follows GLSL's specification, which means having those Preprocessor external declarations (i.e. top-level declaration) is more a hack than a faithful representation of the spec. If we want accurate parsing (per the spec) we need to follow the whole lexer -> preprocessor -> parser pipeline. However, this staging is both critical to the preprocessor functioning as expected, but also why we can't propagate preprocessor directives into the AST (at least in the current form).
Your example is an easy answer: we can parse it into the following AST:
TranslationUnit
+ VersionDirective
+ DefineDirective
+ FunctionDecl
  + Block

However, preprocessor directives can be placed literally anywhere:
#version 460 core
void main()
#define TEST 3
{
}

This would result in the same AST after preprocessing, but needs an entirely different data structure to allow for this (something more along the lines of a concrete syntax tree which supports preprocessing, so it does need some thought to get it correct).
Aside from preprocessor directives in random places, they could also be essential to parsing. As a contrived but technically legal by the spec example, this should parse to the same AST as well:
#version 460 core
#define BEGIN {
#define END }
void main()
BEGIN
END

To parse this correctly we need a preprocessing step to happen before parsing, but we'd also need to carry a lot of extra information to keep the transformation lossless when transpiling.
Another issue with dealing with concrete syntax trees is that they're harder to mutate than the current AST in the glsl-lang crate, and the results of mutations might be undefined depending on what preprocessor directives were used to generate this input.
As a workaround, what's supported is to inject pre-processor directives into the AST after parsing (see https://github.com/alixinne/glsl-lang/blob/master/lang-cli/src/main.rs for a full example). This only supports:

Preprocessor directives with runtime meaning: versions, pragmas, extensions, etc.
Directives located at the translation unit level, so no #pragma inside a function body, unfortunately.

Changing your example a bit to use this results in the following:
fn main() {
    use std::path::Path;

    use glsl_lang::{
        lexer::v2_full::fs::PreprocessorExt,
        parse::IntoParseBuilderExt,
        transpiler::glsl::{show_translation_unit, FormattingSettings},
    };

    // Define and parse shader code
    let source = "#version 460\n#define TEST 3\nvoid main() {\n}\n";

    let mut processor = glsl_lang_pp::processor::fs::StdProcessor::new();
    let ast = processor
        .open_source(
            source,
            Path::new("inline.glsl")
                .parent()
                .unwrap_or_else(|| Path::new(".")),
        )
        .builder()
        .parse()
        .map(|(mut tu, _, iter)| {
            iter.into_directives().inject(&mut tu);
            tu
        }).unwrap();

    // Directly print it out again
    let mut transpiled = String::new();
    show_translation_unit(
        &mut transpiled,
        &ast,
        (&FormattingSettings::default()).into(),
    )
    .unwrap();
    println!("{transpiled}");
}

With the transpiled result being:
#version 460
void main() {
}

Maybe this is enough for your use case?

Hey, thanks for the timely answer. I understand the problem with the preprocessor directives. The main thing I need is, that whatever is parsed and then transpiled back to glsl should compile iff the original compiled. So if defines are already resolved, that's no problem, if the version just vanished, then anything relying on a specific version might just not compiling anymore. I will look into injecting the directives back into the AST, like you showed there. Since I am only building a prototype, maybe I won't bother at all and just slap a #version 460 on everything for good measure :)
On a sidenote, thank you very much for your work. This library is very pleasant to work with. I am writing something where I need to find the expression where a cursor in the text file is. The spans are invaluable for that.

Thanks for the kind comment! I'm glad to know you find it a pleasure to work with (even if it does have its quirks after years of organic development).
I'll close the issue since you are okay with the solution I proposed (with the caveat that defines are always resolved) and I am not planning on changing this soon (it'd likely require a massive refactor to have a CST instead of an AST, which would also be a pretty big breaking change). Feel free to reopen the issue if something's wrong.


measure cost of instancing bullets into the root, is an object pool needed?

I don't think this is an issue. Object pool in the future will be easy.


[RFE] Spotify Connect support

I am using spotify connect a lot to play music on different devices around the home (running mostly raspotify) and controlling them from other devices (phones, laptop) so it came as a disappointment that this client does not support the connect protocol, does not show up in the devices selection and can not allow me to control music playing on other device. I think this support is already in librespot so it should not be that complicated to add (but I did not investigate in detail).

Hi! Thanks for your interest in the project.
librespot does support connect for authentication, not so sure about playback control -- looks like it would involve some work on my end! I'll dig eventually into it, but it is not a top priority for me atm to be honest :)

Thank you for your work! Spot works great for me so far. Looks like this rfe caught some attention so it looks like a common use case.
The librespot links spotcontrol, which should do exactly this (but in go ...) so it should be technically possible. If my list of things that I would like to work on would be a bit shorter, I would offer to have a look into that but this still hangs under the "learn rust". But maybe somebody else will be able to jump on that :)

Hi! Thank you for taking a look at this :)
Here a few pointers/an overview of the implementation which I had in mind:

the UI should be updated to allow selecting between devices, don't hesitate to sketch it first so we can agree on a design for this :) ;
the list of devices can be fetched once the AppEvent::LoginEvent(LoginEvent::LoginCompleted(_)) event fires;
the app state should be updated to know about which device is selected; either a Device::Local or a Device::Remote(some_id);
depending on that value, the PlayerNotifier either controls librespot as it currently does, or it controls another device over HTTP.

As for the HTTP calls, I would recommend not using rspotify since we already have a HTTP client, which just needs to implement the extra endpoints.
I have limited time to contribute, but I always have the time for quick messages like this one, don't hesitate if you have questions! As you can see, there are multiple tasks that we could even share if needed!

Hey is there any update on this?

hi! I was away a long while, but I had a branch for spotify connect that was WIP : #427
I could have a look again, but I remember it being fairly painful to implement

Would also love to see this feature!

Would love to see this as well. I use car thing to have separate controls for what I'm playing. But it relies on spotify connect to work. If Spot had it, I could ditch the official app again!
But I also understand if it is too much of a nightmare to implement.

It would be nice to have this feature! The absence of a connect-to-device feature - only stops me from using Spot, instead of the official client.


platform: Allows sysfs to implement utimesns natively

This moves away from syscall.UtimesNano as it has intentionally avoided common features in POSIX, such as handling UTIME_NOW and UTIME_OMIT. When we eventually expose this API, users will be free to override UTIME_NOW with a fake clock, possibly the same that was supplied to wazero's ModuleConfig.

ok only windows left

Thanks @evacchi for the help overnight my time as that's the only reason we are able to land this your morning!


Support LLVM 16 informally

This PR adds preliminary support for LLVM 16
Related to #730

Things that need to be changed:

Uses of getInstList on a basic block
getABITypeAlign[ment]


Ambiguous situation for files loaded into Bunch Tab

It is unclear how the user should interact with the Bunch tab after importing a file to it that was generated externally. Below is a screenshot after importing the bunch file mentioned in #130.

All well and good. However, if the user clicks Generate Beam or exports the tab to a file, another beam will be regenerated with the calculated Twiss parameters, resulting in a different beam with more generic features (see below picture).

At this point, the user's imported beam cannot be recovered.
So, here's the question: should the Bunch tab only be for creating new beams and not for importing beam files? I think so. There are not fine enough controls on that tab for working with non-generic beams that result from external simulations. I would suggest that the DCP tab is much more suited for viewing imported beam files--SDDS or otherwise.
I suggest removing the import capability from the Bunch tab aside from files generated by the tab.

New bunch tab behavior as of commit 9cb7eb3ccf87d962092c84f7c9590976f6b754d1:
If the user is creating a beam, everything works as before. The Generate Bunch button was removed since the plots will be updated after every change to the Twiss tables.
If the user imports a beam file, editing the Twiss tables are disabled after calculating the Twiss parameters. This calculation happens automatically on import, so the Calculate Twiss button was removed. The user can only specify:

The number of particles. If the number is lower than in the original file, a random sample will be extracted.
The momentum of the bunch.
The charge of the bunch.
Plotting parameters

On running a simulation or exporting, these new parameters will be written to the resulting file along with the original particle data from the file (or a sample of the original).

It's no longer possible to create a new bunch from within the tab.


Unable to autolaunch a dbus-daemon without a $DISPLAY for X11

Hi
After installing on ubuntu server 18.04 and calling cmusd I get  this errror :
/usr/local/lib/node_modules/cmusd/node_modules/dbus/lib/bus.js:24
self.connection = _dbus.getBus(1);
^
Error: Unable to autolaunch a dbus-daemon without a $DISPLAY for X11
at new module.exports (/usr/local/lib/node_modules/cmusd/node_modules/dbus/lib/bus.js:24:27)
at Function.DBus.getBus (/usr/local/lib/node_modules/cmusd/node_modules/dbus/lib/dbus.js:39:9)
at Function.DBus.registerService (/usr/local/lib/node_modules/cmusd/node_modules/dbus/lib/dbus.js:78:17)
at Player.init (/usr/local/lib/node_modules/cmusd/node_modules/mpris-service/index.js:35:22)
at new Player (/usr/local/lib/node_modules/cmusd/node_modules/mpris-service/index.js:28:7)
at Player (/usr/local/lib/node_modules/cmusd/node_modules/mpris-service/index.js:13:40)
at Object. (/usr/local/lib/node_modules/cmusd/bin/index.js:88:43)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)

Hi @djbouzz
This is not a cmusd error
Rather, you don't have an X11 Session running for dbus to be auto-launched on
If you want to use this without the X11 session, you have to configure dbus to launch on startup
For more info, look here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=690530
If you intend to connect to a remote X11 session, please ensure your SSH X11 Forwarding configuration isn't misconfigured. See: https://unix.stackexchange.com/questions/12755/how-to-forward-x-over-ssh-to-run-graphics-applications-remotely
Or, just ensure that the $DISPLAY variable points to the proper display. :1, or something


Need to allow temporal uncertainty

For example see date field types allowed here: https://dwc.tdwg.org/terms/#occurrence


Why did the new version of the 1080p webm format disappear?

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I'm running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood
Is this a youtube limitation issue?
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
C:\Tools\yt-dlp>yt-dlp --list-formats --proxy socks5://<IP_ADDRESS>:1080/ https://www.youtube.com/watch?v=UAXS7X6VjWo
[youtube] Extracting URL: https://www.youtube.com/watch?v=UAXS7X6VjWo
[youtube] UAXS7X6VjWo: Downloading webpage
[youtube] UAXS7X6VjWo: Downloading android player API JSON
[info] Available formats for UAXS7X6VjWo:
ID  EXT   RESOLUTION FPS CH │  FILESIZE   TBR PROTO │ VCODEC        VBR ACODEC      ABR ASR MORE INFO
─────────────────────────────────────────────────────────────────────────────────────────────────────────────
sb2 mhtml 48x27        1    │                 mhtml │ images                                storyboard
sb1 mhtml 80x45        1    │                 mhtml │ images                                storyboard
sb0 mhtml 160x90       1    │                 mhtml │ images                                storyboard
139 m4a   audio only      2 │   1.15MiB   49k dash  │ audio only        mp4a.40.5   49k 22k low, m4a_dash
140 m4a   audio only      2 │   3.05MiB  129k dash  │ audio only        mp4a.40.2  129k 44k medium, m4a_dash
251 webm  audio only      2 │   3.07MiB  130k dash  │ audio only        opus       130k 48k medium, webm_dash
17  3gp   176x144      7  1 │   1.73MiB   73k https │ mp4v.20.3     73k mp4a.40.2    0k 22k 144p
160 mp4   256x144     30    │   2.77MiB  118k dash  │ avc1.4d400c  118k video only          144p, mp4_dash
134 mp4   640x360     30    │  13.61MiB  579k dash  │ avc1.4d401e  579k video only          360p, mp4_dash
18  mp4   640x360     30  2 │  15.88MiB  675k https │ avc1.42001E  675k mp4a.40.2    0k 44k 360p
243 webm  640x360     30    │   8.28MiB  352k dash  │ vp9          352k video only          360p, webm_dash
22  mp4   1280x720    30  2 │ ~50.03MiB 2081k https │ avc1.64001F 2081k mp4a.40.2    0k 44k 720p
136 mp4   1280x720    30    │  45.89MiB 1952k dash  │ avc1.64001f 1952k video only          720p, mp4_dash
137 mp4   1920x1080   30    │  80.24MiB 3413k dash  │ avc1.640028 3413k video only          1080p, mp4_dash

C:\Tools\yt-dlp>

Unless you have reason to believe the format is available in browser, the only answer I can give you is "ask youtube". If they don't make it available, we can't get it...


💥Dropped link to add node and automatically connect its link via components panel

Description
This will enable to add node via context menu. It's also possible to automatically connect its link by dropping a link to canvas.
The components panel will only display what component it can linked but not smart enough to know 'Can it connect?' as there is no type check when the link is connecting.
There are 3 condition to display 3 different components list which are:

Right-clicking canvas will show all component that can be added.
Dropping link from ▶ either inPort or Outport will show all but General Components. (Can be added and linked)
Dropping link from any parameter inPort(beside ▶) will only display General Components. (Can be added and linked)

Take note:
Dropping link from any parameter OutPort(beside ▶) will not display anything as it's hard to capture what it can link to.
As this component panel can be activated by right-clicking, it make the previous Jupyterlab's context menu obsolete. With that, new custom context menu was created with fewer command mostly for nodes. It can be activated by ctrl+left-click.
Also, refactored and removed all the general components except literal and hyperparameter.
Pull Request Type

[x] Xircuits Core (Jupyterlab Related changes)
[x] Xircuits Canvas (Custom RD Related changes)
[ ] Xircuits Component Library
[ ] Testing Automation
[ ] Documentation
[ ] Others (Please Specify)

Type of Change

[x] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Tests

Try right-clicking canvas.

A panel showing components will opened. (The previous context menu was removed)
Choose any node to add it.

Try dropping link based on the condition mentioned above and see it's the same as mentioned

Link from ▶ either from inPort or OutPort.
Link from any parameter inPort(beside ▶).

Also, when choosing the node, does the link automatically connect?
When component panel is displayed, try using the search bar and click the display node.

Try dropping different link and search again.
It should only show the allowable components.

Try opening the new context menu by ctrl+left-click.

Try each option and does it work as expected.

Try ctrl+left-click any node excpet general components at the components panel

It'll open its python script

Tested on?

[ ] Windows
[x] Linux Ubuntu
[ ] Centos
[ ] Mac
[ ] Others  (State here -> xxx )

Notes
Currently, there is no type check when connecting the link. Also, feels like some of the files in /src/component folder need to move to different folder.

Thanks Fahreza for the review. It seems the problem occur when we built the wheel. It doesn't built the frontend when packaging the xircuits.

So to deploy simultaneously the frontend and the backend, the frontend NPM package needs to be built and inserted in the Python package. Ref

The workaround is I just add the labextension so that it can build the frontend when packaging.

Thanks for the feedback. Fixed both of the issues. Another thing I notice while fixing this is if you open the panel near to right side of screen, it'll go off screen. What do you think about this?


Feature: Delay loading Properties button until details are finished loading

Description
I have noticed that when the preview pane is open and I click on a folder or file, the Property button appears before the file information. It would be preferable if the Property button appeared after the file information, similar to the "Edit Tags" button.
Steps To Reproduce

Open preview pane
Click on a file or folder

Requirements
Modify the code to ensure that the Property button appears after the file information in the preview pane.
Files Version
<IP_ADDRESS>
Windows Version
22H2 22621.1992
Log File

Thank you for the feedback


Add a script to run the acceptance test

Co-authored-by: Sebastian Heid<EMAIL_ADDRESS>

Nice and simple, thanks yo.
Aakash


Sound Browser - Show from what mounted sources a sound is from

Fixes https://github.com/wiremod/wire/issues/1220

What was wrong with my PR for the same issue? (#1279)

Well I haven't seen it as posted the request. I did more or less the same, but I did a few things more. The other stuff you did seems to be some clean ups I am going apply to this PR as well.

That should be everything. The remaining changes are just in the .luacheckrc config file, which should be in a separate PR.

well, your PR has 149 luacheck warnings
mine only has 26 :)
Also, I separated out the cleanups into a bunch of commits so a person reading the logs can see what's going on.
Can you maybe base your changes on my PR instead, while making sure you don't introduce new warnings?
Or did I modify different parts than you did?

I changed more and different parts (in all 4 files the sound browser uses). I didn't include your .luacheckrc config changes (Should I?). All warnings left should be either caused by the GMod globals or aren't relay fixable in a sensible way.

Currently, luacheck only checks code that surrounds changed code, not everything. The more you modify, the more likely it will pick up an error unrelated to the code currently being modified.

@oldmud0 All changed Lua files are related to the topic of this PR or have also Bug fixes. I did not change unrelated files.


Base method module

addresses #50 but does not resolve. Adds t_bw_sites.
possibly solves issue with #25, but still includes Time series in state.
closes #40

Mo is doing final review of scheduling. Do not merge without his approval

closes #52

I think this is looking awesome - but it would benefit from documentation of new parameters, linting, and a spell check.
I think the architecture looks good - its hard to get this perfect for every situation. But feeling still some tidying and cleanup needed to get closer to PEP8, or at least consistent. It's a little thing, but consistent code 'smells good' to people browsing and they are more likely to trust it or put in the effort to learn it.
This is a big chunk of edits - it needs functionality testing as well before merging, but I can put some effort in on this when the parameters are documented so I know what to change and what each parameter change does.

@tbarchyn  @AkaMrmuffin  Thanks for reviewing! I agree and I can help with PEP8izing the document. I will wait until Mo resolves or responds to the last two review requests.

I can do a review session focused on functionality following your work @KeeganShaw-GIS .

Hi, @tbarchyn  and @KeeganShaw-GIS  I've updated the Haversine metric and added descriptions to these distance functions in the generic_fuctions.py.

@tarcadius

Yes the plan will be to remove the old programs that do not use base programs (in main folder). Tested folders should eventually go in the module folder as well.

This is a very good question. Basically we would have to add a method name, and a method company name to allow that. This does work currently if they are in different programs.

Definitely needs more comments / better code organization. See https://stackoverflow.com/questions/448271/what-is-init-py-for , second and third answer for its use. Basically allows us to have a folder structure.


Refactor interactive pose viewer backend to 3Dmol

We're switching to a 3DMol backend instead of NGL. Beside having a more 'PyMol-like' atom/bonds representation and controls, this package is much more flexible and will allow us to:

show interactions in a modular way
show bond orders from SDF

Overall the code should become somewhat cleaner/more interpretable.
Main points to implement:

[x] use airium to write HTML rather than just slapping strings together
[ ] add view orientation for all targets (possibly remove 272)
[x] feed in color_dict for both subpockets and fitness coloring methods (so remove bfactor workaround)
[ ] add interactions by default. may need to use PLIP for this because ProLIF seems to sensitive to protonation variations
[ ] finalize bond orders and protonation representation for both ligand and protein
[ ] make sure all components work in the workflow
[ ] remove all redundant code

Preview of new view:

@JenkeScheen is this ready for review, or is there more to hit on your checklist first?

Thanks, this is still WIP. I'd like to get the bypass in place, then get a review in before I attempt to remove the current flow.

Sounds good!

Have you considered using templates for the html generation part?   I am a big fan of jinja, since, in addition to just straight up find/replace, it allows you to define formatting macros and loops within the templates, minimizing the amount of processing/preparation you need to do inside the python code.

noting down here so I don't forget to update the conda env.yml: we're adding a dependency to PLIP, of which the openbabel dependency is a bit fiddly. This ended up doing the trick:
conda install -c conda-forge openbabel lxml
conda install -c conda-forge --no-deps plip

PLIP conda install has abi issues on running my_mol.analyze():
libc++abi: terminating with uncaught exception of type swig::stop_iteration
Abort trap: 6

for now, going with singularity -> load PLIP docker image approach.

This is almost ready for review, I just need to resolve some issues with protonation. I'll check in with @kaminow for this, looks like it's mostly just OE being difficult.
@dotsdl, I'm mostly unsure about https://github.com/choderalab/asapdiscovery/pull/443/commits/73cfad4187c1b5c9368ca88b64d1230098369a1d. It works on my end, but I'm not sure that conda-installing plip will result in a plip bin for all users, if not then https://github.com/choderalab/asapdiscovery/commit/ecdf23db1443406e48a22a6be44bf45363a0ec4f?diff=unified#diff-8ad06bf1f92be6cc4e927ac68d370da58af3abfef7c2fa0e9c4ceac4a12c8756R278 would fail. Can you check that after install, that CLI call works correctly?

Running the tests also leaves a bunch of log files in the working directory; can we disable these?

CI currently failing. Looks like you'll need to remove some of the test items since this PR removes things like view_coords_MERS_CoV_Mpro_7ene

ah, I'll have a look. I haven't been paying attention to CI as it was breaking because of an issue with fragalysis, thanks for spotting.

@JenkeScheen looks like CI is broken in a different spot now in get_interactions_plip. Can you address this?

ugh

@dotsdl looks like CI still fails after my bugfixes, but on something to do with fragalysis.

looks like CI is still failing stochastically on fragalysis stuff. python 3.10 succeeded this time, so this PR is good to go. @dotsdl did you want to have one last look?

looks like CI is still failing stochastically on fragalysis stuff. python 3.10 succeeded this time, so this PR is good to go. @dotsdl did you want to have one last look?

Yeah sorry about this known issue #463

Codecov Report

Merging #443 (6470cb2) into main (f90b2f3) will increase coverage by 0.04%.
Report is 1 commits behind head on main.
The diff coverage is 86.06%.

Additional details and impacted files


fix(Vulnerable dependency): dependency version update for jquery and bootstrap

SUMMARY
1, The application is running a vulnerable version of jQuery, v3.4.1, which was released on May 1, 2019.
https://snyk.io/test/npm/jquery/3.4.1
Impact
The noted version of the product is vulnerable to cross-site scripting.
2, The application is running a vulnerable version of Bootstrap v3.4.1, which was released on February 15, 2019.
Impact
The noted version of the product is vulnerable to cross-site scripting attacks. However this vulnerability was not identified during this test and the finding is based on the vulnerable version returned, hence warrants a low severity rating.
CVE ID:
• CVE-2019-8331

@eschutho @michael-s-molina @dpgaspar  Just pointing out this fairly old PR is still open... I'm not sure if it can be easily made mergeable, or if we should open a new/rebased one, but it does underscore that the move to a React Single Page app is not just a feature and maintenance issue, but a security one as well.

I merged a couple of PRs, one removing bootstrap, and one bumping jquery. I think we're all good here now. Thanks for the contribution, and nag me on slack if you ever open more that need attention :D


[Enhancement] Lake pk index load optimization

Why I'm doing:
In shared-data mode, the data is stored in remote. When pk index needs to be loaded, the cost will be expensive. The segments can be loaded paralleled to speed up the pk index loading.
What I'm doing:
Optimize the lake pk index load. For a tablet which has 155 segments, pk index loading will be 5x faster.
Fixes #issue
What type of PR is this:

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.2
[x] 3.1
[ ] 3.0
[ ] 2.5

Optimize the lake pk index load. For a tablet which has 155 segments, pk index loading will be 5x faster.
It doesn't make sense that only improve 5x.

Optimize the lake pk index load. For a tablet which has 155 segments, pk index loading will be 5x faster.

It doesn't make sense that only improve 5x.

For a segment, the loading costs 600ms, the insert costs 200ms. But lock is needed in insert. Insert can't be parallel.

3.29

Optimize the lake pk index load. For a tablet which has 155 segments, pk index loading will be 5x faster.

It doesn't make sense that only improve 5x.

need check the iops from the monitoring, whether this will increase the get/put api rate significantly that may hit the server side rate limit.


Issues updating to latest WASI SDK

I'm in the process of updating this to use the latest WASI SDK for .NET — https://www.nuget.org/packages/Wasi.Sdk/0.1.2-preview.10061
Bumped the package version in both the sample and this SDK:
<PackageReference Include="Wasi.Sdk" Version="0.1.2-preview.10061" />

However, I am seeing two distinct problems:

when UseWizer is enabled, pre-initialising the module fails with the following error related to the bulk memory proposal not enabled

  Running wizer to preinitialize bin/Debug/net7.0/HelloWorld.Spin.wasm...
EXEC : error : failed to compile the Wasm module [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/samples/hello-world/HelloWorld.Spin.csproj]

  Caused by:
      0: WebAssembly translation error
      1: Invalid input WebAssembly code at offset 4252388: bulk memory support is not enabled
/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/build/Fermyon.Spin.Sdk.targets(18,3): error MSB3073: The command "wizer bin/Debug/net7.0/HelloWorld.Spin.wasm -o bin/Debug/net7.0/HelloWorld.Spin.wasm.pre.wasm --allow-wasi" exited with code 1. [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/samples/hello-world/HelloWorld.Spin.csproj]

Using a branch of Wizer that enables the bulk memory proposal fixes this issue — however, that is not currently enabled in Wizer, nor is it configurable. Depending on the cause of this issue, we should consider adding an optional configuration to Wizer to enable this proposal.

Note that enabling Wizer on a simple WASI console application from C# does not encounter this issue.

running a release build results in a linking error:

wasm-ld : error : /var/folders/rb/3k3_t3zd0j32fh_b7kx54rq80000gn/T/dotnet_wasi_getbundledfile-6d21e9.o: undefined symbol: Fermyon_Spin_Sdk_dll_B1FE6C56_o_len [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/Fermyon.Spin.Sdk.csproj]
wasm-ld : error : /var/folders/rb/3k3_t3zd0j32fh_b7kx54rq80000gn/T/dotnet_wasi_getbundledfile-6d21e9.o: undefined symbol: Fermyon_Spin_Sdk_dll_B1FE6C56_o [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/Fermyon.Spin.Sdk.csproj]
clang-14 : error : linker command failed with exit code 1 (use -v to see invocation) [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/Fermyon.Spin.Sdk.csproj]
/Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/Wasi.Sdk.targets(224,3): error MSB3073: The command ""/Users/radu/.wasi-sdk/wasi-sdk-16.0/bin/clang" /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../native/main.c /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-ee-interp.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-ilgen.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-wasi-driver.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmonosgen-2.0.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-component-hot_reload-stub-static.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-component-marshal-ilgen-stub-static.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-component-diagnostics_tracing-stub-static.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-component-debugger-stub-static.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libSystem.Native.a obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/dotnet_wasi_getbundledfile.c obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/Fermyon.Spin.Sdk.dll.B1FE6C56.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Net.Http.dll.BFD8CFD7.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Net.Security.dll.4C5B4FF4.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Collections.dll.CB61E222.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Private.CoreLib.dll.CA1E5D08.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Memory.dll.4CE36FB8.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Security.Cryptography.dll.F320C359.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Collections.NonGeneric.dll.DE2976E7.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Threading.dll.7D26D3DA.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Runtime.dll.F876D186.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Private.Uri.dll.EA8A7CD0.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Net.Primitives.dll.1E0BF110.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/Microsoft.Win32.Primitives.dll.7502414D.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Diagnostics.Tracing.dll.2755647A.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Runtime.InteropServices.dll.D3ADDD17.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Text.Encoding.Extensions.dll.B1243078.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Collections.Concurrent.dll.D9EB7C68.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Runtime.Numerics.dll.404B5944.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Formats.Asn1.dll.923319B4.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Diagnostics.DiagnosticSource.dll.D9275F9C.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Threading.Thread.dll.59374F80.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Runtime.InteropServices.JavaScript.dll.6EAC5B7A.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Threading.Channels.dll.2396D734.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Threading.ThreadPool.dll.1C2B1053.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Collections.Immutable.dll.FFD8A487.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Linq.dll.78C0A655.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Numerics.Vectors.dll.D9C4D162.o obj/Release/net7.0/dotnet-wasi-sdk/entrypoint_Fermyon.Spin.Sdk.c --sysroot="/Users/radu/.wasi-sdk/wasi-sdk-16.0/share/wasi-sysroot" -I"/Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/include" -Wl,--export=malloc,--export=free,--export=__heap_base,--export=__data_end -Wl,-z,stack-size=1048576,--initial-memory=52428800,--max-memory=524288000,-lwasi-emulated-mman -Wl,-s -o "obj/Release/net7.0/dotnet-wasi-sdk/Fermyon.Spin.Sdk.wasm"" exited with code 1. [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/Fermyon.Spin.Sdk.csproj]

Note that running a release build on a simple WASI console application from C# does not encounter this issue.

cc @itowlson @SteveSandersonMS

@itowlson - is this for 0.6.0?

No, at last pass we didn't have a solution for this, and we won't have one in the 0.6 timeframe.

Following up on this because it's something that I've been testing out locally with new versions of the Wasi.Sdk. I have a branch in my fork that does seem to work for me link to diff but yielded some interesting results, so I thought I'd share. For context, I have no idea how wizer works or should work.
Then I ran spin build for the hello-world sample four times, twice on the current main branch and twice on my fork's branch. The interesting result was that the wizer processed wasm module was actually larger than without Wizer.

Branch
UseWizer
Size

main
false
17M / 17662772

main
true
18M / 19286298

wasi-sdk-0.1.3-preview.10012
false
10M / 10589360

wasi-sdk-0.1.3-preview.10012
true
12M / 12130194

Not sure what to do with this information but it does seem like the latest preview of Wasi.Sdk at least works and has some improvements for file size.

This is a bit of an aide but

The interesting result was that the wizer processed wasm module was actually larger than without Wizer.

Wizer isn't a minimiser.  A wizerised module coming out larger than an unwizerised one seems pretty likely, as the wizered one likely contains additional state.

Aha, thank you for clarifying for me!

Thanks for the investigation and testing - do you want to send a PR?  Weird that iterating the headers worked before but not now - the fake request during wizerisation does include headers (https://github.com/fermyon/spin-dotnet-sdk/blob/0b320bad3c661bb2d3af5950f1b76af8f7fb655d/src/native/http-trigger.c#L135) so something has gone amiss somewhere - we probably need to pass the literal strings through the string ABI or something.

Actually, let me clarify this a little. It does include the "key: val" header but it doesn't include the "expected" headers like "spin-full-url", which the handler was calling without any null checking.
I'll get the PR submitted and we can discuss how to handle the wizer headers there.

Ah, that makes sense.  Weird that it worked before, so this might still be a regression in the marshalling - will take a look.  Thanks!

It doesn't seem to work on the current main branch either. Might have to go back in time to figure out when it was introduced which would probably mean testing with multiple versions of spin.


Updated examples and added links to Azure Firewall workbook

Updated PowerShell example to send logs to Log Analytics

Added Azure CLI example

Added link to Azure Firewall Workbook under both "View and analyze the network and application rule logs" and "Next steps" heading

@StefanIvemo : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

Hi @vhorne - It looks like master was updated to add a section about enabling diagnostic logging by using Azure CLI (also). So, there isn't a clean slot for the version in this PR to be added to the article. In resolving the merge conflict, I wouldn't know which parts to keep. Could you give me some guidance, for example, "replace the new section in master with the section as it's written in this PR"? Thanks.

@ktoliver and @vhorne I noticed that the article had been updated the other day. I do find the CLI example
in the current version overly complicated.

@ktoliver , OK let's go with the recommendation from @StefanIvemo and use his CLI example.

@ktoliver : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Thanks @vhorne and @StefanIvemo


Estimator variance does not decrease with number of shots taken

Describe the bug
The variance of an observable should decrease with the number of shots taken.  However, this seems to not be the case, eg.

N = 4
qc1 = QuantumCircuit(N)
qc1.x(N-1)
qc1.h(range(N))
qc1.cx(range(N-1),N-1)
qc1.h(range(N-1))

op = Pauli('IZZZ')

estimator = Estimator(circuits=[qc1], observables=[op], service=service, options={ "backend": backend })

out = []
shots = [100, 1000, 2500, 5000, 10000, 25000, 50000, 100000]
for samples in shots:
    results = estimator(circuit_indices=[0], observable_indices=[0], shots=samples)
    out.append(results.metadata[0]['variance'])

gives

suggesting that performing more shots has no additional value in determining the precision of expectation values
Steps to reproduce
Expected behavior
The variance should improve with the number of shots taken.
Suggested solutions
Additional Information

qiskit-ibm-runtime version: 0.4.0
Python version:
Operating system:

@t-imamichi @ikkoham Thoughts?

Well it is independent of backend, but those particular results were on Ithaca

thanks! (wanted to see if simulator or real backend)

Variance is not standard error (SEM). Variance is square of the standard deviation (SD).

When we developed Estimator, we have not concluded which to include as the additional information to the expectation value, variance, standard error or standard deviation. So, we added variance as part of metadata so that we can modify it without changing the interface in the future.
There is a discussion to update the result class https://github.com/Qiskit/qiskit-terra/pull/8105. I think it's related to this issue.

I personally would expect the variance, or standard deviation, to highlight the  fact that increasing shots bounds the precision which is observable via repeated execution of a circuit.  For example, the following data (for a different circuit than the one given here) shows this:

Has there been further understanding on this in the year since the bug was marked high priority?
As a new runtime user encountering this issue, I feel returning an indicator of the uncertainty of the estimate (standard error or variance) is unambiguously the expected behavior here. To the point that new users will almost always wrongly assume variance is a measure of the uncertainty of the result, not the single-shot variance (or variance of the observable) that is currently returned. This confusion can be mostly avoided by instead returning the SEM (maybe {'std_err': sqrt(variance/num_shots)} instead of {'variance': variance}?) which is hopefully less ambiguous.

@nonhermitian I have investigated this issue and in my opinion, it has been resolved.
Here is the code that I run (same as your original code, but with all the modifications necessary to avoid errors):
import matplotlib.pyplot as plt
from qiskit.circuit import QuantumCircuit
from qiskit.transpiler.preset_passmanagers import generate_preset_pass_manager
from qiskit_ibm_runtime import EstimatorV2 as Estimator, QiskitRuntimeService, Session, Batch
from qiskit.quantum_info import Pauli

service = QiskitRuntimeService()
backend = service.backend("ibm_bangkok")

N = 4
circuit = QuantumCircuit(N)
circuit.x(N-1)
circuit.h(range(N))
circuit.cx(range(N-1),N-1)
circuit.h(range(N-1))

pm = generate_preset_pass_manager(backend=backend, optimization_level=3)
isa_circuit = pm.run(circuit)
isa_circuit.draw(idle_wires=False)

op = Pauli('IZZZ')
isa_op = op.apply_layout(isa_circuit.layout)

estimator = Estimator(backend)

jobs = []
with Batch(backend=backend) as batch:
    shots = [100, 1000, 2500, 5000, 10000, 25000, 50000, 100000]
    for samples in shots:
        estimator.options.default_shots = samples
        job = estimator.run([(isa_circuit, isa_op)])
        jobs.append(job)

results = [job.result() for job in jobs]

plt.scatter(shots, [r[0].data.stds for r in results])
plt.ylabel("std")
plt.xlabel("shots")

I have attached the figure that is produced.
In the figure, one can see how the std drops as the number of shots start to increase.
Past a certain point, the increase is not as clean as one would expect (e.g., shots=50000 yields an std that is larger than at shots=25000). However, this may be due to a number of factors that are outside of the runtime's control, such as statistical fluctuations (the estimated std is itself a random variable), noise drifts in the HW, etc.
We have a number of open issues regarding target_precision and default_shots that will allow us to improve our documentation. Our intent is to make sure that when these issues are closed, it will be very clear to the user that the estimator tries to do all it can to ensure the desired accuracy, but factors outside of its control may sometimes cause it to underperform.
Let me know if you have additional questions or comments, or if I can close this issue.


feat: track lines rejected in prometheus metrics

This adds the metric influxdb3_write_lines_rejected_total metric which tracks the total number of lines rejected from incoming writes.
Note, that this only tacks the number of rejected lines when the default accept_partial of true is provided to incoming write requests.
This also changes the names of the influxdb3_write_lines_total and influxdb3_write_bytes_total internally to influxdb3_write_lines and influxdb3_write_bytes, respectively. Since the Prometheus exporter adds the _total for counter metrics for us see here, these were previously being exported as influxdb3_write_lines_total_total and influxdb3_write_bytes_total_total - FYI @saraghds.
Closes #25696

We could also probably track rejected lines in requests that are made with accept_partial=false, by counting the lines in the request body.


Novel Downloading

Describe your suggested feature
It'd be nice to have the feature to download 4/5 chapters of a novel at once instead of 1 after 1. It takes more time that ways
Other details
Let's say i put 100 chaps on download together. It takes too much time with "1 after another" download setting
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] If this is an issue with a source, I should be opening an issue in the sources repository.
[X] I have updated the app to version 1.1.18.
[X] I will fill out all of the requested information in this form.

Duplicate of #471


Remove ability to add a blank issue

Per these instructions: https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository#configuring-the-template-chooser

@jehchow #sign-off


VADCOP design

Overview:
VADCOP is a mechanism where you add challenges (in the sense of "a challenge from the verifier to the prover") and turn lookups into regular polynomial equality constraints. This allows lookups for systems that do not support lookups. It also allows some bigger flexibility with regards to number of rows (you can have columns with different number of rows in the system), and it supports some optimizations (lookups can be combined, etc).
A challenge is always just a random field element, nothing more. One difficulty is that the challenges and the witness columns need to be ordered: In the interactive proof, the prover commits to a set of witness columns and sends those commitments to the verifier. The verifier answers with a random number (the challenge). Then the prover commits to another set of witness columns (whose values then can depend on the challenge).
Things we need to figure out:
Syntax:
Create syntax for creating challenges:
let x = challenge(n), where n is the "stage".
or
col <n> name; / col <n> name[k];
Create syntax for how witness columns depend on a certain challenge or are within a certain stage.
At best, the syntax should not require too big changes to the type system. If we have structs, we could even "just" use some structs that we mark "known" by the compiler.
Language for transforming lookups / links
We need language for specifying how lookups (or more abstractly "links") are transformed into stuff that makes use of challenges. This could just be a PIL function that takes a lookup as argument:
let transform_lookup = |left_sel, left_cols, right_sel, rigt_cols| ...;

This is not too far out - we already support it in the type system.
It would be even nicer if we could have structs for lookups and then use syntactic sugar for destructuring a lookup. This way, we could pass an array of all the lookups in the source:
let transform_lookup = |lookups| std::array::map(|lookup| match lookup {
  lsel { lcols } in rsel { rcols } => ...
  _ => panic("Not a lookup")
});

Another question is how we "call" these transforms once they are defined. They will probably be defined in some standard library, but I guess the idea is that we want the user to be able to choose between different ways to transform lookups.
Maybe it is better to do this at the machine level? Then we could have something like:
machine Main { ... }

transform_machine_to_vadcop(Main);

Witgen:
The witness generator needs to work in stages: It first needs to compute the first-stage witnesses, somehow receive the challenge from the verifier and then compute the next stage of witnesses.
Backend
Integrate this whole thing with polygon's vadcop data structure and halo2's challenges.
Our estark intergation already uses challenges internally and transforms the lookups, we can just reuse that.

Here is some example code that shows how to transform lookups, how to handle the challenges in constraints and maybe even how to fix the problem with the degree:

enum constr {
    Lookup(Lookup),
    Identity(expr, expr),
}

struct Lookup {
    left: SelectedExpressions,
    right: SelectedExpressions,
}

struct SelectedExpressions {
    sel: expr,
    exprs: expr[],
}

// The arithmetic machine, a 'constr' function.
let arith = constr || {
    let latch = ...;
    let A;
    let B;
    let C;
    ...

    ArithMachine {
        // Query the machine. Returns a lookup.
        // TODO If we call this function twice, we get two lookups. Do we want to prevent that?
        query: |sel, (x, y, z)| Lookup {
            left: SelectedExpressions {
                sel: sel,
                exprs: [x, y, z],
            },
            right: SelectedExpressions {
                sel: latch,
                exprs: [A, B, C]
            },
        },
    }
};

// The main machine.
let main = constr || {
    // This instantiates an arith machine.
    let arith_machine = arith();
    let X;
    let Y;
    let Z;
    // This is still the higher-level code.
    instr add X, Y -> Z = |flag, X, Y, Z| arith_machine.query(flag, (X, Y, Z));
    // It gets desugared to
    // constr (|flag, X, Y, Z| arith_machine.query(flag, (X, Y, Z)))(instr_add, X, Y, Z);
    // which essentially adds the lookup to the list of constraints.
}

// "capture_constraints" is a new builtin function. TODO we still have to find a better name.
//
// It creates a new "constraint context", i.e. the list of constraints is empty.
// We could also say that it creates a new pil namespace (the polygon feature of namespaces),
// taking the degree as an argument.
//
// Then it calls its first argument, which should be a `->()` function, i.e. takes no arguments and returns nothing.
//
// It returns all the constraints generated during the evaluation of the function.
// transform_lookups then takes this list and transforms all lookups into regular polynomial constraints
// (see below).
//
// TODO: Maybe a call to `capture_constraints` also just increments the stage after its return?
// At least it knows which witness cols were defined during its evaluation.
//
// If the evaluation of a pil program does not result in the right type or in a call to `capture_constraints`,
// some default call is added that does not transform the lookups (and does not use challenges).
constr transform_lookups(capture_constraints(main, 2**20));

/// This transforms an array of constrainst such that all lookups
/// are combined into a single identity constraint.
/// This is a 'constr' function, so it returns the constraints only implicitly.
/// NB: This is actually code for permutations. For lookups we somehow need to add multiplicities.
let transform_lookups: constr[] -> = constr |c| {
    let alpha = challenge();
    let beta = challenge();
    let (constrs, left_acc, right_acc, _) = array::fold(c, ([], 1, 1), |(constr_acc, left_acc, right_acc, id), c| {
        match c {
            Lookup(l) => {
                let (left, right) = transform_lookup(alpha, beta, l, id);
                (constr_acc, left_acc * left, right_acc * right, id + 1),
            c => (constr_acc + [c], left_acc, right_acc, id),
        }
    });
    // Now we add the identity constraints again.
    constr constrs;
    // This is a new witness column, and it automatically is in the next stage, since we
    // called `capture_constraints`
    let gprod;
    let L1: col = |i| if i == 0 { 1 } else { 0 };
    // TODO: the "'" here is a "previous row" reference.
    // not sure if we can write this with "next row" only.
    constr gprod * right_acc = ('gprod * (1 - L1) + L1) * left_acc;
    constr L1' * (gprod - 1) = 0;
}

let transform_lookup = |alpha, beta, l, id| {
    std::check::assert_eq(l.left.exprs.len(), l.right.exprs.len());
    let left = compress_selected_exprs(id, (alpha, beta), l.left.sel, l.left.exprs);
    let right = compress_selected_exprs(id, (alpha, beta), l.right.sel, l.right.exprs);
    (left, right)
};

/// Combines one part of a lookup into a single expression
/// using two challenges.
/// The id is the unique id of the lookup.
/// NOTE: We need to ensure that the stage of the expressions is
/// less than the stage of the challenge.
let compress_selected_exprs: int, (challenge, challenge), expr, expr[] = |id, (alpha, beta), sel, exprs|
    sel * (compress_columns(alpha, exprs) + id + beta - 1) + 1;

/// Combines an array of expressions into a single
/// expression using a challenge.
/// NOTE: We need to ensure that the stage of the input
/// is less than the stage of the challenge.
let compress_columns: challenge, expr[] -> expr = |alpha, cols|
    // TODO polygon uses reversed order.
    std::array::fold(cols, 0, |acc, c| (acc + c) * alpha);

As a summary of the above comment, we need the following features:
Destructuring of constr
constr needs to be enum types that we can destructure.
Access to constraints of stages
We want the code to be stage-independent, this means there is no way to create a challenge at a specific stage or with a specific ID, there is only the built-in functtion std::prover::new_challenge() that returns a new challenge at the current stage.
There is a built-in function std::prover::next_stage: (->) -> constr[], that takes a constr-function. It evaluates the function and returns the set of constraints generated while evaluating the function. It also increases the current stage.
Add builtins:
std::prover::new_challenge // could have name suggestion, but nothing else
std::prover::next_stage
std::prover::new_witness_column // no stage, but maybe name suggestion

Remove the current builtins:
std::prover::challenge // it has access to the stage
std::prover::new_witness_in_stage

Of course since at the end we want to transform to actual pil again, there needs to be a way to have explicit stages both for challenges and for witness columns, but we could check that in that case, none of the other mechanisms are used.

Some more ideas about how we could implement VADCOPs using a proof object / struct at the pil level:
https://hackmd.io/@4tYcUGjTRT6U93PllxQGww/rkL3UQyIA/edit

VADCOP itself is done so closing. We can make a new issue for proof objects if needed


Fixes #1052 - Review DistributedCache usage - Changed how token files and property files get cached

I tried to demonstrate that DistributedCaching is working and demonstrated how we can create a pseudonym for a cached file by putting at the end of a URI the # symbol followed by a pseudonym like in this example: job.addCacheFile(new URI("hdfs://localhost:8020/tokenfile.json#tokenfile"));
I tested this new code with a running instance of Hadoop on port 8020 and it works.  I had to remove much of the code that relies on this so the TokenFileIT will run and clean up properly during the build process.
The system call: 'hdfs dfs -put tokenfilepath /'  in the setTokenFileInHDFS function is vestigial now.  So is the 'hdfs -rm call' .  Since I had to test a real HDFS file system I have left this code in for further investigation in case someone wants to test it the way I did originally.  The system calls in the annotated After cleanUpHDFS function are vestigial too but left in to demonstrate the steps taken to evaluate this issue of concern that distributed caching was not working as expected.

I had to comment out a lot of the functionality in the test that I used to verify the distributed caching because it would not pass the security and style checks.  I left the commented code in as a reference or to be experimented with if individuals choose to do so at a later date.

@ctubbsii.  Do you think I need to work on this a little more?

@jzgithub1 This isn't quite right. This change confirms that the StackOverflow description of how to use the DistributedCache correctly actually works as described. However, what we really need, is to modify our (non-test) code to use the DistributedCache correctly, now that we have confirmed that the description on StackOverflow is correct.

Thanks for the feedback.  It seemed to me that our implementation is OK.  We are not using that "URI with the pound sign with an alias string"  technique they use on Stack Overflow.  Otherwise I see the file our code pulls from the cache just fine in the test directory whether the URI alias or expected file name is used.
I will revisit the issue and modify the non-test code to use the StackOverflow technique since this is what you want.

Thanks for the feedback. It seemed to me that our implementation is OK. We are not using that "URI with the pound sign with an alias string" technique they use on Stack Overflow. Otherwise I see the file our code pulls from the cache just fine in the test directory whether the URI alias or expected file name is used.

The problem is that our code appears to be using the DistributedCache, but it's not actually using it. That's a bug in our code. We are using it incorrectly. Instead of reading the file stored in the cache, we're merely confirming that the file at the URI was requested to be put in the cache... and then we're promptly ignoring the cached copy and instead reading the file directly from HDFS anyway.
The correct usage is the technique you verified works from StackOverflow. Step 1 was to verify that the StackOverflow technique worked... which you did, and that was great. Step 2 is to actually fix our broken usage. :smiley_cat:

OK.  I understand what is going on now.

@ctubbsii  I think this is ready for review now.

Yeah.   "Mnemonic" is really  a term @ctubbsii and I started to use to refer to the link that HDFS created in the working directory of the process that would point to the real file.
I will go ahead and change it to what you suggest to make is more understandable.

Made requested variable name changes.

I made the requested changes and will improve the RangePartitioner next week.

I am still working on testing the RangePartitioning use of distributed cache with the cut key files.  I just pushed this change to let the community know where I am on this issue right now.  I need to use the bulk import testing in Accumulo-testing to do this and need to figure out how to get maven to use the fork on my box as the accumulo repository to use and not the one github.  The present master branch has issues reading the key cut files from distributed cache without my modification.  Here is a small stack trace:
INFO mapreduce.Job: Task Id : attempt_1559227935495_0010_m_000004_2, Status : FAILED
Error: java.lang.RuntimeException: java.io.FileNotFoundException: /tmp/bt/10/splits.txt not found in distributed cache
at org.apache.accumulo.hadoop.mapreduce.partition.RangePartitioner.getPartition(RangePartitioner.java:59)
at org.apache.accumulo.hadoop.mapreduce.partition.KeyRangePartitioner.getPartition(KeyRangePartitioner.java:36)
at org.apache.accumulo.hadoop.mapreduce.partition.KeyRangePartitioner.getPartition(KeyRangePartitioner.java:31)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:728)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:125)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)

I am comfortable with the changes to RangePartitioner @ctubbsii.  I tested the hdfs caching mechanism using the "fragment" after a hash tag in the URI when adding to hdfs cache.  I verified proper caching in the debugger of the Intellij while stepping through the BulkIngestExample.java class during execution.  The main function of the BulkIngestExample class calls RangePartitioner.setSplitFile which calls job.addCacheFile the URI with the fragment. The during the actual MR job execution ,  in RangePartitioner.getCutPoints ,  the splits.txt file is access by the fragement URI called: org.apache.accumulo.hadoop.mapreduce.partition.RangePartitioner.cutFile  in the present working directory of the script or project.  This was my last concern about the code changes and I am confident it is ready for merging.
Let me know if there is anything that you feel I have missed here.

I believe that I have made all of the requested changes that you asked for @ctubbsii.  Please check on that.  My most recent effort have been in verifying the range partitioning caching of the splits file.  I am satisfied it is working as desired.

One more push is on the way.  I did miss one of the RangePartitioners. Sorry about that.

I have verified that the changes to RangePartitioner work by using the Bulk Ingest Example in the Accumulo-Examples project which uses Hadoop mapreduce.   The RangePartitioner caches and retrieves the splits file properly now.  It just did not do that before.  I have traced through the code in the debugger in both versions to see what is happening and observed the fragment creation in the working directory with the new code.   I will create an new test inside the Accumulo project that is like TokenFileIT to demonstrate the caching of the "split file" in the RangePartitioner.

This has been superseded by #1188; @jzgithub1 's changes here were incorporated into that PR, before I made further alterations.


Prepare release of SDK 1.6.5

The only real change is the removal of some schematron rules.

Errors in the checks by SDK Analyzer are expected for this version.


wait_for_everyone with timeout

System Info
I want to make the main process download a dataset. Other processes need to wait until it is finished.

# Pseudo code
if accelerator.is_local_main_process:
    download_dataset()

accelerator.wait_for_everyone()

However, it will only wait for 30 minutes and crash after that. It can take more than 30 minutes to download the dataset.
Is there any way to add timeout to this function?

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[ ] My own task or dataset (give details below)

Reproduction
# Pseudo code
if accelerator.is_local_main_process:
    download_dataset()

accelerator.wait_for_everyone()

Expected behavior
All processes have to wait until the main process finishes downloading the dataset, even if it's longer than 30 minutes

Yes, you can specify the maximum timeout in InitProcessGroupKwargs and then pass it along when creating your Accelerator:
from accelerate.utils import InitProcessGroupKwargs

accelerator = Accelerator(kwargs_handler=InitProcessGroupKwargs(timeout=3600))

Thank you. I think it will work although I haven't tested.
By the way, there are lots of type errors when I use Pyright LSP. It would be nice to explicitly set type hints for this function.

As long as those type hints are not too complex, we welcome any PR.

My recommendation is to use | instead of Optional[], Union[] etc. However this will only work from Python 3.7. It will make the type hint much more simplified. Because of the compatibility, I wonder if you would approve this change

I'm happy to look at a PR with small changes. We do not support Python versions older than 3.7, so happy to see what you have in mind!


[WIP] Add testing notes to markdown doc.

@timabbott DId I miss anything?

Awesome, this looks great, merged!  I ended up adding a few new sections to the docs as well about how to change the markdown processor while I was thinking about this.  Your review on the new changes would be great.
Thanks @hackerkid!