Failed to load staticwebapp.config.json schema on start

I'm using the latest version 1.1.4
Are you accessing the CLI from the default port :4280 ?

[ ] No, I am using a different port number (--port) and accessing the CLI from that port
[X] Yes, I am accessing the CLI from port :4280

Describe the bug
When starting an app with swa start I get WARNING: Failed to load staticwebapp.config.json schema. Continuing without validation!.
Even if it's just a warning and is referring to the schema file (not the config itself) staticwebapp.config.json doesn't work as expected. e.g. none of the routes declared is working.
To Reproduce
Steps to reproduce the behavior:

Go to any swa app
Type in command 'swa start'
See warning in console
Check in browser any routes declared -> won't work.

Expected behavior
staticwebapp.config.json to work. Routes and any other settings declared to work.
Desktop (please complete the following information):

OS: Windows
Version: 11

Additional context
After investigating this issue i have discovered that the URL used to load the config schema is hardcoded: https://github.com/Azure/static-web-apps-cli/blob/affefb24872286a542fe7bbd54340648ec3294d0/src/core/utils/user-config.ts#L168C13-L168C13 .
At this moment, the website https://www.schemastore.org/ which is hosting the schema file, is down (error 503).
Allowing loading the schema file only from a single website has created a single point of failure which we are experiencing right now.
staticwebapp.config.json has a property $schema, but as you can see the value is not used for actually loading the schema.
My suggestion is to allow the user to provide a value for $schema (in order to no longer depend on https://www.schemastore.org/ ).
Thank you.
Edit: there's currently an issue on Schemastore repository: https://github.com/SchemaStore/schemastore/issues/3303

I'm also seeing this warning, but routes are loading okay. I am using a customised config file swa-cli-config.json
My redirect from /login causes an error and the swa auth emulator does not seem to be protecting/allowing routes correctly. Trying to determine if related to above issue or not...

I have this issue also. It fails silently and won't load my settings.
I use the config to set CORS, so my deployment is not accessible at all from my app.
I cannot deploy anymore until I have a solution for this. I'm completely blocked.
Please, do you have a solution ?

I'm also running into this issue. Just had a new deployment, and users are redirected to 404 page if they try to access links directly or refresh the page.

Seeing the same problem. This does also affect the swa deploy command leading to a not working SWA

The same problem

Temporary solution (what I did):

Go to "C:\Users\%userprofile%\AppData\Roaming\npm\node_modules\@azure\static-web-apps-cli\dist\core\utils (for Windows).
Open user-config.js.
Search for https://json.schemastore.org/staticwebapp.config.json.
Upload staticwebapp.config.json to blob storage (for example).
Replace https://json.schemastore.org/staticwebapp.config.json by your remote path.

Temporary solution (what I did):

Go to %userprofile%\AppData\Roaming\npm\node_modules\@azure\static-web-apps-cli\dist\core\utils (for Windows).
Open user-config.js.
Search for https://json.schemastore.org/staticwebapp.config.json.
Upload staticwebapp.config.json to blob storage (for example) or use my file: https://stcuberdonintegration.blob.core.windows.net/assets/staticwebapp.config.json (Will be removed in 24H)
Replace https://json.schemastore.org/staticwebapp.config.json with your remote path.

Confirm this fix for the issue I observed...

I'm also seeing this warning, but routes are loading okay. I am using a customised config file swa-cli-config.json
My redirect from /login causes an error and the swa auth emulator does not seem to be protecting/allowing routes correctly. Trying to determine if related to above issue or not...

I can confirm that issue as well on 1.1.4. Also, the fix posted by @aresofficial is working for me.
That's really strange, how Azure position Static Web Apps as production-ready service, where your app can be totally inaccessible because of downtime on 3rd party service.
 That's "amazing"

As we are on linux when deploying the workaround does not work. One approach could be to setup a webserver which serves the schema. One could then point the building/deploying machine for the SWA to this server via a hosts entry :thinking:

It seems to be fixed.

Yes and no... someone threw more hardware on the problem so it works for now but the problem persists. One still depends on the availability of https://www.schemastore.org/.

There was some problem in https://json.schemastore.org so that swa-cli got Error 500 Timeout. I have validate that it works well now, so I will close the issue. Feel free to repoen it if the error occurs again.

Really? Is it fixed so that when the schemastore isn't available it at least throws some kind of error and does not deploy a not working swa?..
I think it is not fixed.

I agree with @avalor1 . We should not depend on a third-party service which can become unavailable without notice.

The issue is not fixed, Static Web Apps CLI is still unstable.
It's only a matter of time before problem recurs (e.g. because of outage on schemastore)
Deployment should at least not silently destroy routes, otherwise Static Web Apps can't be production-ready (as now).
@cjk7989 can you please escalate this issue?

As others mentioned, this is still not fixed.

Hi all, we have published<EMAIL_ADDRESS>which will throw an error if staticwebapp.config.json schema is failed to be download. And we plan to design a fallback method to completely fix the issue in the next version. Thanks very much for your feedback and suggestions.

The current error is still "Continuing without validation". That will confuse users since it's aborting execution.

I see this issue in 1.1.6. Emulator does not work without internet. Fails to start.
[swa] ✖ Failed to load staticwebapp.config.json schema. Continuing without validation!
[swa] node ".pnpm/@azure+static-web-apps-cli<EMAIL_ADDRESS>exited with code 0

Good morning.  Can't locally start my Azure static web app because ....
network timeout at: https://json.schemastore.org/staticwebapp.config.json

The schema store is just sending 429 and 503s

I agree that this is a weird remote dependency on what should should be "Airplane mode" type local dev work.

I followed @LiamOSullivan steps above but rather than putting in a new blob address I increated the time out

Go to %userprofile%\AppData\Roaming\npm\node_modules@azure\static-web-apps-cli\dist\core\utils (for Windows).
Open user-config.js.
Search for https://json.schemastore.org/staticwebapp.config.json.
Two line below increase the time out to 10000 milliseconds
Save and retry

It's happening, right now.... AGAIN!!!!!
Please, reopen and FIX this.

Temporary solution (what I did):
1. Go to `%userprofile%\AppData\Roaming\npm\node_modules\@azure\static-web-apps-cli\dist\core\utils` (for Windows).

2. Open `user-config.js`.

3. Search for `https://json.schemastore.org/staticwebapp.config.json`.

4. Upload `staticwebapp.config.json` to blob storage (for example) or use my file: `https://stcuberdonintegration.blob.core.windows.net/assets/staticwebapp.config.json` **(Will be removed in 24H)**

5. Replace `https://json.schemastore.org/staticwebapp.config.json` with your remote path.

Thank you for solution!
I was able to patch @azure/static-web-apps-cli for my deployment pipelines to use custom url instead of https://json.schemastore.org/staticwebapp.config.json.
<EMAIL_ADDRESS>diff --git<EMAIL_ADDRESS><EMAIL_ADDRESS>index 00861dd..0f2f1fe 100644
---<EMAIL_ADDRESS>+++<EMAIL_ADDRESS>@@ -149,7 +149,7 @@ function findLineAndColumnByPosition(content, position) {
     return notFound;
 }
 async function loadSWAConfigSchema() {
-    const schemaUrl = "https://json.schemastore.org/staticwebapp.config.json";
+    const schemaUrl = "https://simkuns.dev/staticwebapp.config.schema.json";
     try {
         const res = await (0, node_fetch_1.default)(schemaUrl, { timeout: 10 * 1000 });
         if (res.status === 200) {

package.json
{
  "scripts": {
    "postinstall": "patch-package"
  },
  "dependencies": {
    "@azure/static-web-apps-cli": "1.1.6",
    "patch-package": "^8.0.0",
    "postinstall-postinstall": "^2.1.0"
  }
}

@cjk7989 this should really be reopened, https://json.schemastore.org being down caused a monorepo with hundreds of devs to block all pull requests. We should be able to provide a local fallback.

@cjk7989  Friday, I was able to bring my SWA up just fine locally for testing during development. Tonight, I ran everything with the exact same configuration and I get this error. Are there really no parameters I could supply to have it not reach out to a spotty third party?

Currently swa fails to start locally because of this error:
 network timeout at: https://json.schemastore.org/staticwebapp.config.json
[swa] ✖ Failed to load staticwebapp.config.json schema. Continuing without validation!
...
✖ SWA emulator stopped because API server exited with code 1.

Having a way of doing local validation is not the solution. We would still see default pipelines to fail each time someone decides to attack schemas tore.org
I think validation shloud be non-fatal at build time, it will be redone by swa api during publish. An optional parameter to force fatal errors on validation would be nice to have for some scenarios.

Here's a simple fix that skips the validation (as the message says) instead of killing process. Some reordering of rows and missing schema returns config instead of null.
File<EMAIL_ADDRESS>async function validateRuntimeConfigAndGetData(filepath) {
    const ajv4 = new ajv_draft_04_1.default({
        strict: false,
        allErrors: true,
    });
    logger_1.logger.silly(`Reading content from staticwebapp.config.json...`);
    const content = (await readFile(filepath)).toString("utf-8");
    let config;
    try {
        logger_1.logger.silly(`Parsing staticwebapp.config.json...`);
        config = JSON.parse(content);
    }
    catch (err) {
        printJSONValidationWarnings(filepath, content, err.message);
        return null;
    }
    logger_1.logger.silly(`Loading staticwebapp.config.json schema...`);
    const schema = await loadSWAConfigSchema();
    if (!schema) {
        logger_1.logger.error(`Failed to load staticwebapp.config.json schema. Continuing without validation!`);
        return config;
    }
    logger_1.logger.silly(`Compiling schema...`);
    const validate = ajv4.compile(schema);
    logger_1.logger.silly(`Validating staticwebapp.config.json...`);
    const isValidSWAConfigFile = validate(config);
    if (!isValidSWAConfigFile) {
        printSchemaValidationWarnings(filepath, config, validate);
        return null;
    }
    logger_1.logger.silly(`File validated successfully. Continuing with configuration!`);
    return config;
}

Thanks for the feedback. It’s an intermittent bug but occurs more common recently. We are going to have a local backup schema as a fallback option if the download fails. The fix will come soon.

Here's a simple fix that skips the validation (as the message says) instead of killing process. Some reordering of rows and missing schema returns config instead of null. File<EMAIL_ADDRESS>async function validateRuntimeConfigAndGetData(filepath) {
    const ajv4 = new ajv_draft_04_1.default({
        strict: false,
        allErrors: true,
    });
    logger_1.logger.silly(`Reading content from staticwebapp.config.json...`);
    const content = (await readFile(filepath)).toString("utf-8");
    let config;
    try {
        logger_1.logger.silly(`Parsing staticwebapp.config.json...`);
        config = JSON.parse(content);
    }
    catch (err) {
        printJSONValidationWarnings(filepath, content, err.message);
        return null;
    }
    logger_1.logger.silly(`Loading staticwebapp.config.json schema...`);
    const schema = await loadSWAConfigSchema();
    if (!schema) {
        logger_1.logger.error(`Failed to load staticwebapp.config.json schema. Continuing without validation!`);
        return config;
    }
    logger_1.logger.silly(`Compiling schema...`);
    const validate = ajv4.compile(schema);
    logger_1.logger.silly(`Validating staticwebapp.config.json...`);
    const isValidSWAConfigFile = validate(config);
    if (!isValidSWAConfigFile) {
        printSchemaValidationWarnings(filepath, config, validate);
        return null;
    }
    logger_1.logger.silly(`File validated successfully. Continuing with configuration!`);
    return config;
}

Saved my teams morning productivity, thank you!

Temporary solution (what I did):

Go to %userprofile%\AppData\Roaming\npm\node_modules\@azure\static-web-apps-cli\dist\core\utils (for Windows).
Open user-config.js.
Search for https://json.schemastore.org/staticwebapp.config.json.
Upload staticwebapp.config.json to blob storage (for example) or use my file: https://stcuberdonintegration.blob.core.windows.net/assets/staticwebapp.config.json (Will be removed in 24H)
Replace https://json.schemastore.org/staticwebapp.config.json with your remote path.

Thank you. I have found another way to solve it. And it works without hosting the staticwebapp.config.json file
So instead of simply replacing the URL, I increased the timeout to 5 minutes instead of just 10 seconds, as the only issue was I was able to load the file in browser but it would take more than 10 seconds.
Here's how you can do it:

Go to user-config.js
Find const schemaUrl = "https://json.schemastore.org/staticwebapp.config.json"; and below there's a line with timeout set to 10 * 1000
Change the 10 to whatever number you think works
Reload your terminal (it wasn't working without it)

PS. Finding user-config was tricky because it was installed globally and since I am using nvm mine was located at<EMAIL_ADDRESS>

I disabled validation until this gets resolved:
    /*
    const schema = await loadSWAConfigSchema();
    if (!schema) {
        logger_1.logger.error(`Failed to load staticwebapp.config.json schema. Continuing without validation!`, true);
        return null;
    }
    logger_1.logger.silly(`Compiling schema...`);
    const validate = ajv4.compile(schema);
    */
    const validate = () => true;

A quick way to use the CLI until this is resolved is to remove the "true" parameter to the logger.error method, as this causes the script to exit:
This:
logger_1.logger.error(`Failed to load staticwebapp.config.json schema. Continuing without validation!`);
instead of this:
logger_1.logger.error(`Failed to load staticwebapp.config.json schema. Continuing without validation!`, true);
Credit to @cristianbretti

I probably understood this problem.
In loadSWAConfigSchema(), there is a process to load data from the following SchemaURL, right?
const schemaUrl = "https://json.schemastore.org/staticwebapp.config.json";

Currently, it seems that the data delivery for this URL is very slow.
This is causing a timeout.

In conclusion, it seems that this issue occurs at least during a period when the distribution of the above SchemaURL is unstable.

A quick way to use the CLI until this is resolved is to remove the "true" parameter to the logger.error method, as this causes the script to exit:
static-web-apps-cli/src/core/utils/user-config.ts
This: logger_1.logger.error(`Failed to load staticwebapp.config.json schema. Continuing without validation!`); instead of this: logger_1.logger.error(`Failed to load staticwebapp.config.json schema. Continuing without validation!`, true);
Credit to @cristianbretti

Removing the "true" from logger_1.logger.error will cause the routing to not work. I recommend the two temporary solution mentioned by @zainzafar90, to save the schema into blob, or set the timeout to a larger value.

I think the following measures are necessary.
Add processing that allows individuals to specify Schema for development by dividing the URL that distributes Schema into multiple parts or as an option.
Please have an image like the following.
swa start --schema "https://[alterbaseurl]/staticwebapp.config.json"

Can we skip accessing the network for the commands that does not require it? I am often find myself working on the plane with unstable internet connection. I don’t understand why do I need to download anything from the Internet if I just want to test the app. For deployment we may have additional arguments.

Solved with workaround!!
After few hrs of struggle and understand I finally got it work by storing schema to different host and pointing updating the same in user-config.js.
Thanks @cjk7989
Note: For mac os, path for file is<EMAIL_ADDRESS>depends on node installation directory.
change the schemaUrl in the file to your hosted schema path.
Those who need the content of the schema to host, I have a copy of it at: https://anuraggandhi.com/schema-swa.config.json.
(please don't use it directly from this host. It will be removed shortly.)

I think the following measures are necessary. Add processing that allows individuals to specify Schema for development by dividing the URL that distributes Schema into multiple parts or as an option.
Please have an image like the following.
swa start --schema "https://[alterbaseurl]/staticwebapp.config.json"

While I acknowledge that this issue arises due to current circumstances, it undermines the essence of validation.
Allowing users to easily upload their own flawed schemas poses a risk, particularly if they become outdated and desynchronized with the latest schema over time.
If validation doesn't happen against latest schema, this could subsequently lead to additional complications in production environments.

All builds fall down because of this issue

All builds fall down because of this issue

I understand, I also have the same set of frustration as my deployment delayed by many hrs due to this.
Hope MS will fix it intelligently and will not rely on 3rd party sites for their production ready tools.
Better solution, i think: MS should host the schemas themselves to a relied host and use them for their tools. They should take care of updating the schema when needed.

@softgandhi

以下のような対策が必要かと思います。Schemaを配布するURLを複数に分割またはオプションとして、個人が開発用のSchemaを指定できる処理を追加します。
以下のようなイメージを持ってください。
swa start --schema "https://[alterbaseurl]/staticwebapp.config.json"

この問題が現在の状況によって生じていることは認めますが、検証の本質が損なわれます。 ユーザーが欠陥のある独自のスキーマを簡単にアップロードできるようにすると、特にスキーマが古くなり、時間の経過とともに最新のスキーマと非同期になった場合にリスクが生じます。 最新のスキーマに対して検証が行われない場合、実稼働環境でさらに複雑な問題が発生する可能性があります。

以下のような対策が必要かと思います。Schemaを配布するURLを複数に分割またはオプションとして、個人が開発用のSchemaを指定できる処理を追加します。
以下のようなイメージを持ってください。
swa start --schema "https://[alterbaseurl]/staticwebapp.config.json"

この問題が現在の状況によって生じていることは認めますが、検証の本質が損なわれます。 ユーザーが欠陥のある独自のスキーマを簡単にアップロードできるようにすると、特にスキーマが古くなり、時間の経過とともに最新のスキーマと非同期になった場合にリスクが生じます。 最新のスキーマに対して検証が行われない場合、実稼働環境でさらに複雑な問題が発生する可能性があります。

I think the following measures are necessary. Add processing that allows individuals to specify Schema for development by dividing the URL that distributes Schema into multiple parts or as an option.
Please have an image like the following.
swa start --schema "https://[alterbaseurl]/staticwebapp.config.json"

While I acknowledge that this issue arises due to current circumstances, it undermines the essence of validation. Allowing users to easily upload their own flawed schemas poses a risk, particularly if they become outdated and desynchronized with the latest schema over time. If validation doesn't happen against latest schema, this could subsequently lead to additional complications in production environments.

Yes, I do not intend to use this in a production environment, but rather to use it for development to temporarily deal with issues that are currently occurring.

The solution shouldn't be reliant upon any specific host, MS controlled or otherwise. Its a dependency that can cause the current pain points many of us are receiving.
Why not ship a fallback schema with the tool, kept up to date with each update.

このソリューションは、MS が制御するかどうかにかかわらず、特定のホストに依存すべきではありません。この依存性は、私たちの多くが現在直面している問題の原因となる可能性があります。
更新ごとに最新の状態に保たれるフォールバック スキーマをツールに同梱してみてはいかがでしょうか。

I think this is the proposal that should be given the most preferential treatment.

Better yet, don't perform any validation at all at deploy time. If the schema is messed up in the config file at the time of deployment that is a fault of something that should have been fixed and adjusted at build time. And even then, there shouldn't be any hard-coded dependency on a remote server being available.

Better yet, don't perform any validation at all at deploy time. If the schema is messed up in the config file at the time of deployment that is a fault of something that should have been fixed and adjusted at build time. And even then, there shouldn't be any hard-coded dependency on a remote server being available.

I wouldn't use the CLI for production deployments anyway, it's mainly made for local development and validation. You should use the GitHub action directly, as the CLI deploy command is missing a lot of configuration features for deployment like customizing data-api path's etc. and the GitHub action doesn't rely on the CLI.
You could even directly use the deployment container image or the StaticSitesClient directly for deployment (probably unsupported), to not be dependent on the CLI for production.
Besides that, I 100% agree that there should never be a dependency on a remote server for any local development tooling.

All of the other options rely on this under the hood. So doing anything that doesn't rely on this isn't an option. The real tool, the StaticClientSites binary, is kept closed source. Otherwise I'd just modify and build that directly myself and use it. This CLI tool and any other cli tool that publishes to Azure Static Sites use that closed-source binary because it's the only way to get resources to Azure Static Sites.

That is not correct @nathanblair, the GitHub action doesn't rely on the CLI at all, which is why that is the preferred way for production deployments.
There is no production tooling around SWA relying on the CLI, as the SWA CLI is mainly for local development.

I hope that we can settle on the fact that "local development" does not require fetching anything from the internet. :)
One other interesting observation that I would like to point out: I didn't know that there is a validation until I went offline. I am not clear on why this is required and can this be optional? (activated using --strict or --validate arg)
In any case making start command not rely on the internet - that is my main request.
Thank you!

+1 on not relying on the internet by default
A warning about impossible validation is enough

Hi all, the fix has been merged in PR #808. In sumarry, we added a logic to load static-web-apps-cli/schema/staticwebapp.config.json as a backup, in case https://json.schemastore.org/staticwebapp.config.json get timeout again.
Before a new version of swa-cli is released, you can feel free to download and test it from GitHub. Here are the steps:

git clone https://github.com/Azure/static-web-apps-cli.git
cd into the folder "static-web-apps-cli"
run "npm install", then "npm run build"
run "npm link ./", which allows you to use this version of swa-cli globally.
* if you want to update the version of swa-cli, just delete this local folder "static-web-apps-cli" and run "npm install -g @azure/static-web-apps-cli" to reinstall the official version.

Thanks a lot for checking if this update works for you.

@cjk7989 Thank you so much for this - I think I speak for many that this is a great fix and will help loads. Was able to check PR and LGTM, not able to test it out right this second though sorry. Is there an estimated release window for the next version?

Better yet, don't perform any validation at all at deploy time. If the schema is messed up in the config file at the time of deployment that is a fault of something that should have been fixed and adjusted at build time. And even then, there shouldn't be any hard-coded dependency on a remote server being available.

I wouldn't use the CLI for production deployments anyway, it's mainly made for local development and validation. You should use the GitHub action directly, as the CLI deploy command is missing a lot of configuration features for deployment like customizing data-api path's etc. and the GitHub action doesn't rely on the CLI.

You could even directly use the deployment container image or the StaticSitesClient directly for deployment (probably unsupported), to not be dependent on the CLI for production.

Besides that, I 100% agree that there should never be a dependency on a remote server for any local development tooling.

All of the other options rely on this under the hood. So doing anything that doesn't rely on this isn't an option. The real tool, the StaticClientSites binary, is kept closed source. Otherwise I'd just modify and build that directly myself and use it. This CLI tool and any other cli tool that publishes to Azure Static Sites use that closed-source binary because it's the only way to get resources to Azure Static Sites.

That is not correct @nathanblair, the GitHub action doesn't rely on the CLI at all, which is why that is the preferred way for production deployments. The GitHub action is just a wrapper around the SWA container (mcr.microsoft.com/appsvc/staticappsclient:stable) and you can currently use the SWA container as well as the GitHub action without problems.
There is no production tooling around SWA relying on the CLI, as the SWA CLI is mainly for local development.

Not everyone has ability to use the GitHub Action for their deployment and should not be required to do so. Again, the underlying tool should be open sourced to begin with

...

That is not correct @nathanblair, the GitHub action doesn't rely on the CLI at all, which is why that is the preferred way for production deployments. The GitHub action is just a wrapper around the SWA container (mcr.microsoft.com/appsvc/staticappsclient:stable) and you can currently use the SWA container as well as the GitHub action without problems.
There is no production tooling around SWA relying on the CLI, as the SWA CLI is mainly for local development.

Not everyone has ability to use the GitHub Action for their deployment and should not be required to do so. Again, the underlying tool should be open sourced to begin with

That is what the container mcr.microsoft.com/appsvc/staticappsclient:stable is for. This is not the right place to request a certain tool to be open sourced. If you want other tools or something else to be open sourced, I would open a separate issue for that.
The important part is, that the SWA CLI should not be part of production release tooling and that the tools for that are the container image and the GitHub action. There is currently no dependency on the CLI for production deployments.

Hi all, the fix has been merged in PR #808. In sumarry, we added a logic to load static-web-apps-cli/schema/staticwebapp.config.json as a backup, in case https://json.schemastore.org/staticwebapp.config.json get timeout again.
Before a new version of swa-cli is released, you can feel free to download and test it from GitHub. Here are the steps:

git clone https://github.com/Azure/static-web-apps-cli.git
cd into the folder "static-web-apps-cli"
run "npm install", then "npm run build"
run "npm link ./", which allows you to use this version of swa-cli globally.

if you want to update the version of swa-cli, just delete this local folder "static-web-apps-cli" and run "npm install -g @azure/static-web-apps-cli" to reinstall the official version.

Thanks a lot for checking if this update works for you.

Thanks a lot for the quick fix @cjk7989 🚀  I tested the PR branch locally and the original problem is fixed.
I will close this issue as soon as the new release is available via npmjs.

Hello,

Hello,
I still have problem to deploy my application through Azure DevOps.
Here is the output :
Deploying to environment: production

network timeout at: https://json.schemastore.org/staticwebapp.config.json
✖ Failed to load staticwebapp.config.json schema. Continuing without validation!
##[debug]Exit code 1 received from tool 'C:\Program Files\Git\bin\bash.exe'
##[debug]STDIO streams have closed for tool 'C:\Program Files\Git\bin\bash.exe'
##[debug]task result: Failed
##[error]Script failed with exit code: 1
##[debug]Processed: ##vso[task.issue type=error;]Script failed with exit code: 1
##[debug]Processed: ##vso[task.complete result=Failed;]Script failed with exit code: 1

Even if it says that it would "Continuing without validation!", the deployment still fail.
It seems that https://json.schemastore.org/staticwebapp.config.json still have a timeout.
How can I manage to avoid that validation ?
Do I need to deploy throught Azure CLI, would it help ?
Maybe I could use a script in my deploy.yml template, but not sure how to do it...
Thanks for your help !

@PothieuG I don't think the npm package has been published yet: https://www.npmjs.com/package/@azure/static-web-apps-cli?activeTab=versions.
I imagine it'll get published later once @cjk7989 see's that it's been tested by some individuals here, see the request here. I'm also guessing there are timezone limitations too

1.1.7-alpha works good, problem seems to be fixed.

If someone is using docker in CI, i did the following: copied a copy of  https://json.schemastore.org/staticwebapp.config.json into my github repository and in my CI file, just after "npm install -g @azure/static-web-apps-cli", i added:
 "- sed -i 's,https://json.schemastore.org/staticwebapp.config.json,{{MY_FILE_ON_MY_GITHUB_REPO}},g'<EMAIL_ADDRESS>

Hello,
I still have problem to deploy my application through Azure DevOps. Here is the output :
Deploying to environment: production

network timeout at: https://json.schemastore.org/staticwebapp.config.json
✖ Failed to load staticwebapp.config.json schema. Continuing without validation!
##[debug]Exit code 1 received from tool 'C:\Program Files\Git\bin\bash.exe'
##[debug]STDIO streams have closed for tool 'C:\Program Files\Git\bin\bash.exe'
##[debug]task result: Failed
##[error]Script failed with exit code: 1
##[debug]Processed: ##vso[task.issue type=error;]Script failed with exit code: 1
##[debug]Processed: ##vso[task.complete result=Failed;]Script failed with exit code: 1

Even if it says that it would "Continuing without validation!", the deployment still fail. It seems that https://json.schemastore.org/staticwebapp.config.json still have a timeout.
How can I manage to avoid that validation ? Do I need to deploy throught Azure CLI, would it help ? Maybe I could use a script in my deploy.yml template, but not sure how to do it...
Thanks for your help !

For SWA deployment in a Azure DevOps CI/CD pipeline, the recommended way is to use the “AzureStaticWebApp” task.
The CLI is mainly for local development not for production deployments. The same action exists for Github, Gitlab and Bitbucket and as container for other deployment systems.

I followed these instructions from cjk7989. This installs 1.17-alpha ,which solved the problem for me.

Hi all, the fix has been merged in PR #808. In sumarry, we added a logic to load static-web-apps-cli/schema/staticwebapp.config.json as a backup, in case https://json.schemastore.org/staticwebapp.config.json get timeout again.
Before a new version of swa-cli is released, you can feel free to download and test it from GitHub. Here are the steps:

git clone https://github.com/Azure/static-web-apps-cli.git
cd into the folder "static-web-apps-cli"
run "npm install", then "npm run build"
run "npm link ./", which allows you to use this version of swa-cli globally.

if you want to update the version of swa-cli, just delete this local folder "static-web-apps-cli" and run "npm install -g @azure/static-web-apps-cli" to reinstall the official version.

Thanks a lot for checking if this update works for you.

Downgrade temporary to 1.1.4. A warning is visible but not error

Hello,
I still have problem to deploy my application through Azure DevOps. Here is the output :
Deploying to environment: production

network timeout at: https://json.schemastore.org/staticwebapp.config.json
✖ Failed to load staticwebapp.config.json schema. Continuing without validation!
##[debug]Exit code 1 received from tool 'C:\Program Files\Git\bin\bash.exe'
##[debug]STDIO streams have closed for tool 'C:\Program Files\Git\bin\bash.exe'
##[debug]task result: Failed
##[error]Script failed with exit code: 1
##[debug]Processed: ##vso[task.issue type=error;]Script failed with exit code: 1
##[debug]Processed: ##vso[task.complete result=Failed;]Script failed with exit code: 1

Even if it says that it would "Continuing without validation!", the deployment still fail. It seems that json.schemastore.org/staticwebapp.config.json still have a timeout.
How can I manage to avoid that validation ? Do I need to deploy throught Azure CLI, would it help ? Maybe I could use a script in my deploy.yml template, but not sure how to do it...
Thanks for your help !

For SWA deployment in a Azure DevOps CI/CD pipeline, the recommended way is to use the “AzureStaticWebApp” task (learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/azure-static-web-app-v0?view=azure-pipelines). The CLI is mainly for local development not for production deployments. The same action exists for Github, Gitlab and Bitbucket and as container for other deployment systems.

Requiring the use of actions/tasks for deploying static assets to a web server is not a good enough solution. That's unrelated to the problem with the CLI but then so is recommending alternatives that "don't use it" (PS, looked at the container image manifest, it does indeed use the same tooling and scripting that the CLI uses).

Hi all, the fix has been merged in PR #808. In summary, we added a logic to load static-web-apps-cli/schema/staticwebapp.config.json as a backup, in case https://json.schemastore.org/staticwebapp.config.json get timeout again.
Before a new version of swa-cli is released, you can feel free to download and test it from GitHub. Here are the steps:

git clone https://github.com/Azure/static-web-apps-cli.git
cd into the folder "static-web-apps-cli"
run "npm install", then "npm run build"
run "npm link ./", which allows you to use this version of swa-cli globally.

if you want to update the version of swa-cli, just delete this local folder "static-web-apps-cli" and run "npm install -g @azure/static-web-apps-cli" to reinstall the official version.

Thanks a lot for checking if this update works for you.

Worked for me publishing from within VS2022.

Here's a solution that worked for us until a release is published. We are using the SWA CLI in GitHub Actions, installed locally as a dev dependency of the project. So we installed the CLI using the latest commit hash to get the unreleased fix:
"@azure/static-web-apps-cli": "github:Azure/static-web-apps-cli#34864d0a78e971ebdf2b8721fe0a7bfccc0000c3",

In our case, the deployment was going through instead of failing and it resulted in the SWA configuration not being applied, which meant our 404 page and redirects were not working anymore in production.

Hello,
I still have problem to deploy my application through Azure DevOps. Here is the output :
Deploying to environment: production

network timeout at: https://json.schemastore.org/staticwebapp.config.json
✖ Failed to load staticwebapp.config.json schema. Continuing without validation!
##[debug]Exit code 1 received from tool 'C:\Program Files\Git\bin\bash.exe'
##[debug]STDIO streams have closed for tool 'C:\Program Files\Git\bin\bash.exe'
##[debug]task result: Failed
##[error]Script failed with exit code: 1
##[debug]Processed: ##vso[task.issue type=error;]Script failed with exit code: 1
##[debug]Processed: ##vso[task.complete result=Failed;]Script failed with exit code: 1

Even if it says that it would "Continuing without validation!", the deployment still fail. It seems that https://json.schemastore.org/staticwebapp.config.json still have a timeout.
How can I manage to avoid that validation ? Do I need to deploy throught Azure CLI, would it help ? Maybe I could use a script in my deploy.yml template, but not sure how to do it...
Thanks for your help !

For SWA deployment in a Azure DevOps CI/CD pipeline, the recommended way is to use the “AzureStaticWebApp” task (https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/azure-static-web-app-v0?view=azure-pipelines). The CLI is mainly for local development not for production deployments. The same action exists for Github, Gitlab and Bitbucket and as container for other deployment systems.

But the “AzureStaticWebApp” task does not work, when bash is not installed on the agent (ex. on self-hosted windows agents).
https://github.com/microsoft/azure-pipelines-tasks/issues/19447
Even when using Azure DevOps CI/CD pipeline it is not always possible to use the task.

Close it as<EMAIL_ADDRESS>has been released and fixed this issue, thanks very much for all your feedback and suggestions.


Light group does not honor members "supported_features"

The problem

Light group does not have the same "supported_features" as the light member of it.
I have 2 identical lights (Hue White Ambiance) through Hue Integration that are only able to dim, no color support.
Here the attributes of the 2 entities:
friendly_name: Salon Gauche
supported_features: 41
icon: 'mdi:outdoor-lamp'

and
friendly_name: Salon Droite
supported_features: 41
icon: 'mdi:outdoor-lamp'

I created a Light Group with both lights
  - platform: group
    name: TV Lights
    entities:
      - light.salon_droite
      - light.salon_gauche

Resulting in theses attributes for the Light Group Entity
min_mireds: 153
max_mireds: 500
effect_list:
  - random
  - colorloop
friendly_name: Salon
supported_features: 63
icon: 'mdi:sofa'

As you can see the "Supported_features" of the created group does not match the ones from the members lights
Hence the create Light Group is showing color picker and much more information in Lovelace UI.
Expected behavior
Created Light Group should have the same "Supported_features" as their members
Steps to reproduce

Create a Light Group with lights that does not support color
Click on the Light Group entity in Lovelace
You are prompted with a color picker that the lights does not support

Environment

Home Assistant release with the issue: Home Assistant 0.105.3
Last working Home Assistant release (if known):
Operating environment (Hass.io/Docker/Windows/etc.):  Hass.io
Integration causing this issue: Light Group
Link to integration documentation on our website: Light Group

Problem-relevant configuration.yaml

  - platform: group
    name: TV Lights
    entities:
      - light.salon_droite
      - light.salon_gauche

Traceback/Error logs

Additional information

I am sorry, unfortunately I cannot reproduce this issue.
Tested the supported_features for a light group with two mqtt lights:
light:
  - platform: mqtt
    schema: json
    name: mqtt_json_light_1
    state_topic: "home/rgb1"
    command_topic: "home/rgb1/set"
    brightness: true

  - platform: mqtt
    schema: json
    name: mqtt_json_light_2
    state_topic: "home/rgb2"
    command_topic: "home/rgb2/set"
    brightness: true

Which results in the following entity, which both have the same supported_features as your lights:
light.mqtt_json_light_1

friendly_name: mqtt_json_light_1
supported_features: 41

light.mqtt_json_light_2

friendly_name: mqtt_json_light_2
supported_features: 41

Grouped them in a light group:
light:
  - platform: group
    name: My Lights
    entities:
      - light.mqtt_json_light_1
      - light.mqtt_json_light_2

Which resulted in the following entity:
light.my_lights

friendly_name: My Lights
supported_features: 41

I have created a group with two hue ambiance (which only supports color temperature) and it worked as expected.
However, I have a similar problem with groups of lights created on the hue app. They show in home assistant as if all the light in it had all the features. I have submitted a pull request to solve this problem.
In yours example, you created a group called TV Lights, but, then, you showed the supported features of a group called Salon. Is the group Salon created in the HA or in the hue app?

Glanced at the code, could also only see |= bitwise operations, which could never result in a higher number.
As we cannot reproduce this, will close this issue.


Implement Integration Registry

[x] Declared integration is persisted across sessions
[x] Different entry points declare themselves to the registry
[x] Provide a method to retrieve profile ID
[x] Mediation adapters are detected
[x] If mediation are detected, mediation is kept even if standalone declare itself
[x] If no info is available (for prefetch, remote config) use fallback profile ID = 235

JIRA: EE-1195

Thanks for splitting / commenting the commits, please next time try to do this before review ;)


bug(mat-chip-listbox): Harness getChips() does not return mat-chip

Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
No response
Description
Hello,
When using MatChipListboxHarness.getChips() it only returns mat-chip-option elements, but not mat-chip. Because of this a mat-chip-listbox like this example cannot be tested using the listbox harness. Not sure what is the correct solution for this because it can be solved using mat-chip-set instead of listbox.
Possible solutions imho:

Change example to use mat-chip-set instead of mat-chip-listbox.
Change getChips() signature to return MatChipHarness[] which would be a breaking change.
Add an additional method like getStaticChips().

Reproduction
StackBlitz link: https://stackblitz.com/edit/qykqid
Expected Behavior
MatChipListboxHarness.getChips() should include mat-chip as well.
Actual Behavior
MatChipListboxHarness.getChips() only returns mat-chip-option.
Environment

Angular: 17.0.1
CDK/Material: 17.0.1
Browser(s): -
Operating System (e.g. Windows, macOS, Ubuntu): All

Hello,
Thank you for reaching out. Using mat-chip inside a mat-chip-listbox is unsupported behavior :). The only supported elements that may be used inside a mat-chip-listbox is mat-chip-option.
Best Regards,
Zach

Hey, thanks for the quick reply. But like I said, it's literally used that way in one of the examples which is kind of confusing. Shouldn't that be changed? I'm happy to provide a PR if you like.

Hello, sure, go ahead and send us a PR. We'd love to take a look.
-Zach

Took a while, but created a PR. Let me know what you think.


feat: reset_fork block number argument

What I did

fixes: #
How I did it
How to verify it
Checklist

[ ] Passes all linting checks (pre-commit and CI jobs)
[ ] New test cases have been added and are passing
[ ] Documentation has been updated
[ ] PR title follows Conventional Commit standard (will be automatically included in the changelog)

PR already approved elsewhere, just making sure build passes before merge


Support --on-error=log

This PR revamps Bottled Water's logging and error handling, with a primary goal of supporting an more forgiving error handling policy.
If Bottled Water encounters an error - such as failure to communicate with Kafka or the Schema Registry - its behaviour prior to this PR is for the client to exit, halting the flow of data into Kafka.  Since Postgres will retain and replay the logical replication stream until Bottled Water acknowledges it, this policy ensures that:

BW will never miss an update (every update made in Postgres will eventually be written to Kafka) - provided that whatever caused the error is resolved externally (e.g.  restoring connectivity to Kafka) and the Bottled Water client is then restarted;

BW will never write corrupted data to Kafka: e.g. if unable to obtain a schema id from the Schema Registry for the current update, rather than writing to Kafka without a schema id (which would leave consumers unable to parse the update), it will wait until the problem is resolved.

However, in some scenarios, exiting on the first error may not be desirable:

if there is an error publishing for one table (e.g. if Kafka is configured not to autocreate topics and the corresponding topic has not been explicitly created), you may not want to halt updates for all other tables.

if the reason for the error cannot be resolved quickly (e.g. Kafka misconfiguration or prolonged outage), Bottled Water may threaten the stability of the Postgres server.  This is because Postgres will store WAL on disk for all updates made since Bottled Water last acknowledged (i.e. successfully published) an update.  If Postgres has a high write throughput, Bottled Water being unavailable may cause the disk on the Postgres server to fill up, likely causing Postgres to crash.

To support these scenarios, this PR adds an alternative error handling policy where BW will simply log that the error occurred and drop the update it was attempting to process, acknowledging the update so that Postgres can stop retaining WAL.  This policy can be enabled via the --on-error command-line switch.  N.B. that in this mode Bottled Water can no longer guarantee to never miss an update.

As of PR #115, the --on-error switch now also applies to (most) errors that arise in the Bottled Water extension, for example if a row is too large to be encoded (#36).


What import to use in package shared between node app and frontend app

I was not able to find any resources in the documentation that lays out how to use the lib vs es6 imports
I would expect to import without lib or es6 in a shared lib (e.g. import { pipe } from 'fp-ts/function').  When this code is transpiled to js and run in node I get the error
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '<proj path>/node_modules/fp-ts/function' imported from <proj path>/dist/validation/item.js
Did you mean to import fp-ts/lib/function.js?

( is actual path to directory)
However, if this code is consumed by a frontend application I want the es6 version to be used.  I would think this would be done by referencing 'fp-ts/function' in shared lib, and letting webpack or rollup use module instead of main, this does not seem to work with pure TS transpile + running in node.
Is there something I am missing or any documentation that lays out which to use?

I would expect to import without lib or es6 in a shared lib (e.g. import { pipe } from 'fp-ts/function')

That's correct

using import { pipe } from 'fp-ts/function' transpiles fine but im getting an error when I run the code in node.  are there any additional steps?  I can create a project with reproducible err to link here if needed

I can create a minimal project with reproducible err to link here if needed

it would definitely help 👍


Testing This Library

I've been wanting to add some tests for a while, but I'm not sure there's any means of automating the testing of if audio plays / sounds right, in various browsers.
Any ideas?

Firstly don't test if audio API works! Test only your own code. Testing Audio API depends not only on browser, but also codecs supported by that browser, OS and other software. It should be done by API developers not you.
You can set up event listeners eg.: https://developer.mozilla.org/en-US/docs/Web/Guide/Events/Media_events to listen if it was triggered. Some of the events may have access to useful data to test too.

@tlenex Thanks for the great input! This is a good starting point for sure

What I can tell more... consider using travis or other CI tool to run tests on each PR / push

@tlenex yes, I think Travis makes a lot of sense for this as it is an open source friendly site. If you wanted to take the lead here, I'd be happy to set you up as a repo maintainer.

I'll think about it and let you know. Currently working on other projects


Add OpenAI Stream Client implementation

Add OpenAI Stream Client implementation.
Add openAiStreamClient test in ClientIT.

Resolves #116

This PR is very specific to the openai-java implementation and does not resolve #116 .

@JM-Lab as @making pointed out we can not expose proprietary libraries throughout the Spring AI public API.
Subclassing is not an option either. We need to define a new generic chunk interface or interfaces.
Also the Spring projects is somewhat biased toward Flux instead of rx.
Please share your design thoughts, apologies for not providing any guidelines so far.

Regarding the feedback from @tzolov, the development was revisited.
OpenAiStreamClient has been re-implemented to directly adhere to the latest Request/Response specifications of OpenAI's Chat Completion API.
Using Flux, the existing 'generate' method was re-implemented, and the 'generateStream' method was added.
The commit details are as follows:

Add the latest Request and Response Classes of OpenAI's Chat Completion API
Add the spring-boot-starter-webflux dependency
Implement the AiStreamClient interface extending AiClient
Implement OpenAiStreamClient using Reactor Flux

If you have any additional inquiries or recommendations, please don't hesitate to reach out.

Rebased, squashed and merged at 8139af468e9e3e8ba5f1d7f81f7e6e56c1c17fe0
@JM-Lab  thank you for the contribution and expect a follow-up changes as I'm heavily refactoring the chat generation and embedding clients.


adding conditional performance tracking via TIMED_FUNC_IF and TIMED_SCOPE_IF

As discussed in issue #413, I have added conditional performance logging and also extended the documentation accordingly.
This enables e.g. verbosity level based logging using
 TIMED_FUNC_IF( timer, VLOG_IS_ON( 4 ) );

Best,
Martin

Thanks for the pull request, i'm doing this as part of v9.85 and will use most of these changes but i cannot merge this as it will cause conflicts with the existing changes. I will use this code though

welcome.. :)


Improve documentation for obr

The documentation of needs some improvement. The usage section should contain examples on:

yaml files, including features like reading environment variables, simple calculations, if blocks, includes ...
basic queries
submit and using submit templates
apply

Like #54 , it is an ongoing effort and only cluttering up the issue board.


Slim 748 firmware file to protocol adapter

SonarQube analysis reported 2 issues

 1 major
 1 info

Watch the comments in this conversation to review them.
1 extra issue
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 DomainResponseMessageListener.java#L54: Complete the task associated to this TODO comment.

All is well.

SonarQube analysis reported 1 issue
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 DomainResponseMessageListener.java#L54: Complete the task associated to this TODO comment.

All is well.

SonarQube analysis reported 1 issue
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 DomainResponseMessageListener.java#L54: Complete the task associated to this TODO comment.

All is well.


Ignore null or undefined data when retrieving pinned row data (Fixes #1303)

Hoist P/R Checklist
Review and check off the below. Items that do not apply should be checked to indicate they have been considered. If unclear if a step is relevant, please leave unchecked and note in comments.

[x] Up to date with develop branch as of last change.
[x] Ready for review (or add wip label).
[N/A] Added CHANGELOG entry (or N/A)
[N/A] Checked for breaking changes (add breaking-change label + CHANGELOG if so, or N/A)
[N/A] Updated doc comments / prop-types (or N/A)
[N/A] Reviewed and tested on Mobile (or N/A)
[N/A] Created Toolbox branch / PR (link provided here, or N/A)

Pull request reviewers: when merging this P/R, please consider using a squash commit to collapse multiple intermediate commits into a single commit representing the overall feature change. This helps keep the commit log clean and easy to scan across releases.

👍 thank you.


Missing 'opensource' and 'commercial' images

https://github.com/ellisonleao/magictools#modeling
opensource link to missing https://github.com/ellisonleao/magictools/blob/master/img/opensource.png
commercial linked to missing https://github.com/ellisonleao/magictools/blob/master/img/commercial.png

hey @wizzup thanks for the heads up! that should be replaced with the emojis listed on #123


gRPC make fails

Hi,
I am facing the following error while building gRPC on Mac. I am following the instructions here.
[MAKE]    Generating cache.mk
[HOSTCXX] Compiling src/compiler/cpp_generator.cc
In file included from /usr/include/libkern/OSAtomic.h:43:0,
                 from /usr/local/Cellar/google-protobuf/3.2.0/include/google/protobuf/stubs/atomicops_internals_macosx.h:36,
                 from /usr/local/Cellar/google-protobuf/3.2.0/include/google/protobuf/stubs/atomicops.h:202,
                 from /usr/local/Cellar/google-protobuf/3.2.0/include/google/protobuf/stubs/atomic_sequence_num.h:33,
                 from /usr/local/Cellar/google-protobuf/3.2.0/include/google/protobuf/arena.h:54,
                 from /usr/local/Cellar/google-protobuf/3.2.0/include/google/protobuf/message.h:118,
                 from include/grpc++/impl/codegen/config_protobuf.h:47,
                 from ./src/compiler/config.h:37,
                 from ./src/compiler/cpp_generator.h:44,
                 from src/compiler/cpp_generator.cc:36:
/usr/include/libkern/OSAtomicDeprecated.h:770:2: error: #error Cannot use inlined OSAtomic without <atomic> and C++11 atomics
 #error Cannot use inlined OSAtomic without <atomic> and C++11 atomics
  ^~~~~
make: *** [/Users/devanshujain/Documents/Tools/grpc/objs/opt/src/compiler/cpp_generator.o] Error 1

Can someone help me in resolving the issue ?
Thanks

I was able to resolve it by omitting -DOSATOMIC_USE_INLINED=1 in Makefile

A lot of things changed since then, and this may very well be obsolete. Closing this one.


Improve onboarding experience for contributors

Summary
We create an onboarding process for code and non-code contributions. Since we have ~45 repositories, I believe we should centralize contributions documents on a repository.
This will be a living document; we don't need to get right on the first shot. Instead, we can create the documents and improve them over time.
Expected Outcome
We are prepared to receive code and non-code contributions on our priority repositories.
Requirements

[ ] We create an onboarding process for both code and non-code contributions
[ ] The document should state which repositories are ready to accept contributions
[ ] Whenever possible, we use GitPod to make it easy for newcomers to try a given repository
[ ] The onboarding should allow a one-time contribution
[ ] Repositories should have well-documented good-first-issue
[ ] We should find ways to recognize contributions via all possible channels
[ ] Repositories might have their additional contribution notes when necessary

Ideas to borrow from

https://github.com/EddieHubCommunity
https://docs.naas.ai/contributing-to-naas/contributing
https://glossary.cncf.io/
https://glossary.cncf.io/contribute/
https://github.com/Virtual-Coffee/virtualcoffee.io/blob/main/CONTRIBUTING.md
https://ngrx.io/contributing
https://rasa.com/community/contribute/

Adding the Apache Sling contribute guide as an additional reference
https://github.com/apache/sling-org-apache-sling-engine/blob/master/CONTRIBUTING.md
https://sling.apache.org/contributing.html

Any progress on this ?

I'm open to adding on my skills as well

@luzpaz @elidakirigo
Sorry for the very late response. And no, no much progress on this ticket. Maybe we should collab on it.


How to use  TimeDistributedDense layer before a RNN layer

@braingineer @fchollet
I found we always use TimeDistributedDense layer after a RNN layer. If we can use it before  a RNN layer to reduce the dimension of the Input to RNN layer?

If you have a 4-dimensional tensor and want to reduce it to 3-dimensions (or in general, d+1 to d), then you can do something akin to this, yes.  I have a layer called Summarize which does this.


add dev info

MAINTENANCE.md or DEVELOPMENT.md or CONTRIBUTING.md would be more appropriate?

Merged into CONTRIBUTING.md


Add tedge_logfile_request_plugin to purge step

Proposed changes
purge step fails because of lacking tedge_logfile_request_plugin.
https://github.com/thin-edge/thin-edge.io/runs/5511562408?check_suite_focus=true
Stop using continue-on-error, which took us so long time to find this issue, actually purging didn't happen!
At least, dpkg returns 0 if there was no package. Also confirmed on Ubuntu 20.04 on amd64.
pi@raspberrypi:~ $ sudo dpkg -P tedge_agent tedge_logfile_request_plugin tedge_mapper tedge_apt_plugin tedge_apama_plugin tedge mosquitto libmosquitto1 collectd-core
dpkg: warning: ignoring request to remove tedge_agent which isn't installed
dpkg: warning: ignoring request to remove tedge_logfile_request_plugin which isn't installed
dpkg: warning: ignoring request to remove tedge_mapper which isn't installed
dpkg: warning: ignoring request to remove tedge_apt_plugin which isn't installed
dpkg: warning: ignoring request to remove tedge_apama_plugin which isn't installed
dpkg: warning: ignoring request to remove tedge which isn't installed
dpkg: warning: ignoring request to remove mosquitto which isn't installed
dpkg: warning: ignoring request to remove libmosquitto1 which isn't installed
dpkg: warning: ignoring request to remove collectd-core which isn't installed
pi@raspberrypi:~ $ echo $?
0

Types of changes

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
[ ] Documentation Update (if none of the other choices apply)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

[x] I have read the CONTRIBUTING doc
[x] I have signed the CLA (in all commits with git commit -s)
[ ] I ran cargo fmt as mentioned in CODING_GUIDELINES
[ ] I used cargo clippy as mentioned in CODING_GUIDELINES
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)

Further comments

I'm fine with trying without continue-on-error. I believe there could still be cases that we would need to manually repair then (we had that, but I do not remember the case anymore).
Since the return code of dpkg -P is 0 even when packages were not installed it should work.

BTW it wouldn't have solved the issue in https://github.com/thin-edge/thin-edge.io/runs/5511562408?check_suite_focus=true . There was a wrong file name in the code.


Fixed Interval accepting negative values on server

Backend Fix
fixes https://github.com/bench-routes/bench-routes/issues/455

This requires a fetch/pull from the master, otherwise, there will be compilation errors.


remove aws scout2 parser

As we don't have a clear timeline for 3.X.X, we could remove awsscout2 parser earlier as the notification to merge to Scout2 was first published in January 2023.

@Maffooch, this is a followup

I will not fix Ruff Linter here as this is already covered here: https://github.com/DefectDojo/django-DefectDojo/pull/9903

@manuel-sommer That just got merged so hopefully it's no longer an issue for PRs :crossed_fingers:

Thank you for the hint @mtesauro . Rebasing is done.

We can probably remove it here as well if we're removing the parser entirely. It looks like PARSER_EXCLUDE should work as expected even if its value is empty (ref).

Done @cneill, thank you for the hint.

@manuel-sommer This is ready to merge once the merge conflicts are fixed.

Ready to get merged @mtesauro


Get process status and PID via CLI

Thank you for process-compose. I recently transitioned a project to it and it worked great.
The following is a request for something that would have made my life easier. I suspect others will find it useful too.
Feature Request
I'd like to query process status via CLI.
Specifically I'd like to get PID and status (other fields would be nice to have too, like restarts, exit code, healthchecks).
At the moment, the only commands giving me some information are process list (which only prints names of configured processes) and project state (which tells me process count and process running count).
I would find it useful to have:
$ process-compose process status <name> --json
{name: <name>, pid: 1234, status: "running", ...}

And/or:
$ process-compose project state --json
{
  <name_1>: {name: <name_1>, pid: 12345, status: "running", ...},
  <name_2>: {name: <name_2>, pid: 23456, status: "running", ...},
 ...
}

(* invalid JSON for the sake of brevity)
Use Case:
I'd like to know the PID in order to send signals to the running processes.
I'd like to know the state to orchestrate (confirm something is running, or not running, etc).
This mostly revolves around better troubleshooting: if I learn what is not running I can present useful error messages.
Proposed Change:
Add CLI commands to return process information that is currently available via TUI.
Maybe default to human-readable, but have the option for structured data (e.g., --json)
Who Benefits From The Change(s)?
I certainly do, and since I don't think I am special, this will probably help other users using process-compose to do similar things in a similar way.
Alternative Approaches
Here's the current workarounds I've had to resort to. Documenting various options for anyone else lurking on this issue in my same situation.
I'd love to get rid of all this junk and just query process-compose directly for this information.
1. Learning PID via process tagging and ps | grep
Configure processes by adding a dummy environment variable tag so they're easy to grep, e.g.
processes:

  FooService_1:
    command: PC_ID=foo_service_1 foo-server --config foo1.conf

Then learn the PID via pid=$(ps aux | grep 'PC_ID=foo_service_1' | grep -v grep | awk '{print $2}')
2. Learning PID via pidfile
Some processes support writing a pidfile, e.g.:
processes:
  FooService_1:
    command: foo-server --config foo1.conf --pidfile /temp/services/foo1.pid

Then learn the pid via something like: pid=$(cat ${pid_file_path})
3. Write pidfile via wrapper
For processes that don't support writing a pidfile, I need to wrap via script, e.g.:
processes:
  FooService_1:
    command: ./scripts/write-pidfile.sh /temp/services/foo1.pid -- foo-server --config foo1.conf

Where write-pidfile.sh writes its pid to the given file then execs the arguments after the -- separator.
4. Check running state via zero signal
Once I learned the PID, my favorite way to see if something is running is via kill -0

Hey @mprimi,
Thank you for using Process Compose.
This feature for sure was on my radar, I just didn't know if there was a need for it.
BTW Have you considered something like:
# status for a single process:
curl http://localhost:8080/process/{PROC_NAME}

# statuses for all the processes:
curl http://localhost:8080/processes

For a single process the output will be:
{
  "name": "_process2",
  "namespace": "default",
  "status": "Running",
  "system_time": "13m",
  "age":<PHONE_NUMBER>50,
  "is_ready": "Ready",
  "restarts": 0,
  "exit_code": 0,
  "pid": 4994,
  "IsRunning": true
}

Oh my! I had completely missed the REST API!
I have been looking at your docs up and down for the last week, and I guess I glossed over it the entire time. 🤦♂
This is great.


Change parser to new language proposal

@wiz has made a proposal for new .plan language:
https://gist.github.com/wiz/fd7249de5485f52e3102c0773fc1daff
which is much better because it enforces (what I think is) good practice of defining properties together, and also I think is more readable.

To confess, I really had no idea were to put the actual algebra bits.
I thought about making a keyword or something, but it doesn't quite fit the bag-of-attributes style of the rest.
Then I tried to interject between the node name and its attributes (step_one_of_many something + more { ... }), it didn't felt like it, so I just dumped it at exrpression's end.
And someone noted it turned to be something akin to XML notation, if you squint a little: <step title="title" ...>x + y</step> and XML's are related to s-expressions. Here's his take at it, in some imaginary LISP flavour. I'm not a huge fan of such, but maybe it'd inspire you to something.
Another option is, of course, to retain the = symbol for binding blocks together. Maybe it is reasonable to have both inlined and free-standing expressions.

I think as a core syntax your proposal is great - I like the minimalism. The "=" bindings can came later, if necessary... I think I need to use this for a while for my real plannings to know if it's desirable.
I slightly prefer curly than square brackets between attributes - just because is more common in general; but for sure parenthesis in the expressions - otherwise is too odd.


Add support for generating update actions for product state transitions

Changes

check if byId or byKey were used before calling delete. This previously would lead to a program error in the SDK, which is also fixed now with this PR 4d2f4b3
handle 405 errors and show descriptive error message, since the CTP API does not provide one
fix handling of graphql 400 errors. Previously these would not occur with status code 400. Since the now do this update was needed. abae23c
add support for generating update actions for product transitions
add unit and integration tests for product state transition actions

DOD

[x] commit messages are commitizen-friendly
[ ] code is unit tested
[x] code is integration tested
[x] public code is documented
[ ] code is reviewed by at least one person
[x] code satisfies linter rules

cc/ @emmenko @hajoeichler @butenkor

Coverage decreased (-0.6%) to 96.752% when pulling e840989ad526c2559f32a2880c60880180e89884 on feature/product-state-sync into 7b3162fe7d83e56645cae81e1592b303c1b94ae5 on master.

LGTM 👍

Coverage decreased (-0.1%) to 97.224% when pulling 397e2c19474652e12fe5286da11b01714559a8c0 on feature/product-state-sync into 7b3162fe7d83e56645cae81e1592b303c1b94ae5 on master.

So, if the desired state is not reachable from the current state, we will get a 400 from our platform.

LGTM

Coverage decreased (-0.1%) to 97.224% when pulling 397e2c19474652e12fe5286da11b01714559a8c0 on feature/product-state-sync into 7b3162fe7d83e56645cae81e1592b303c1b94ae5 on master.

@hajoeichler should we ad a test for unreachable states?

That would be great.

Coverage decreased (-0.1%) to 97.224% when pulling 397e2c19474652e12fe5286da11b01714559a8c0 on feature/product-state-sync into 7b3162fe7d83e56645cae81e1592b303c1b94ae5 on master.

can you explain why we need this? We are just forwarding whatever error the API throws for the given request, there is no logic on the SDK side for this. The test would just approve that the API is working as expected, not that the SDK is working as expected.

👍

@PhilippSpo Your are right, it's nothing specific here in the client.


Cannot use either wget, conda, or mamba to install chopper

Excuse me. I'm currently using Apple M3 Pro, and I'm trying to download "chopper".
I tried conda install and it resulted in "PackagesNotFoundError". Using bioconda and mamba also gave the same result.
I tried wget and it stopped downloading for no reason. Only around 300 KB was downloaded.
I tried looking on every solved issue and none of the solution helped.
I have absolutely no clue what to do next. What should I do?

Did you try the wget once, or does the download stop every time? Can you download it with your internet browser?

I have tried wget several times already, and everytime it failed to download more than 300 KB. I tried changing Wifi and the results were the same.

Can you please show me your wget command?

Looks like it stopped downloading for some reasons.

Because you cannot wget the whole repository. You need to download the right binary from the releases.

e.g. wget https://github.com/wdecoster/chopper/releases/download/v0.8.0/chopper-macos.zip


Does not use system dylib when cross-compiling on macOS

macOS has fat binary dylibs with executable code for multiple architectures. When cross-compiling between macOS on x86 and ARM, all system libraries are usable on both architectures, including libz.1.dylib.
The current logic gives up on system zlib when target != host, which causes unnecessary static linking when the target and host are x86_64-apple-darwin and aarch64-apple-darwin.

Detection whether /usr/lib/libz.dylib is actually available as a fat binary is unfortunately complicated, because since macOS 11 all system libraries are part of a monolithic shared cache, and don't exist as individual files in the file system any more (library paths are merely magic keys for lookup in system cache). Apple recommends checking their existence by running dlopen, but that's highly problematic for cross-compilation. For libz it may be enough to just assume it's present.

Similar issue when cross-compiling to iOS. iOS has libz shared library.

I don't think this is fixed. As a matter of fact I have a cross-compile job that ends up compiling zlib (and failing, because it's using clang trunk, which broke building zlib < 1.3.1).

There is now v1.1.15 of libz-sys with libz 1.3.1 in case this changes anything.


Proposal: DeFi Tokens Distribution platform

Description

A platform where users can distribute tokens using Permit2 for approvals, minimizing gas costs.

Features

Batch airdrop distribution with one approval.
Gas-efficient token distribution.
Easy management of large-scale airdrop campaigns.

Bounty track

DeFi

Approved


[Bug] Alerting failed on publishzip task on macOS

https://build.ci.opensearch.org/job/distribution-build-opensearch/5902/execution/node/276/log/
+ ./gradlew publishPluginZipPublicationToZipStagingRepository -Dopensearch.version=2.2.0-SNAPSHOT -Dbuild.snapshot=true -Dbuild.version_qualifier=
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Mac OS X 12.4 (x86_64)
  JDK Version           : 17 (Eclipse Temurin JDK)
  JAVA_HOME             : /private/var/jenkins/tools/hudson.model.JDK/openjdk-17
  Random Testing Seed   : 993604F0DFB01868
  In FIPS 140 mode      : false
=======================================

> Task :ktlint
/private/var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/tmpwp0wt1qz/alerting/alerting/src/test/kotlin/org/opensearch/alerting/TriggerServiceTests.kt:40:1: Exceeded max line length (140) (cannot be auto-corrected) (max-line-length)
/private/var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/tmpwp0wt1qz/alerting/alerting/src/test/kotlin/org/opensearch/alerting/TriggerServiceTests.kt:42:1: Exceeded max line length (140) (cannot be auto-corrected) (max-line-length)
/private/var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/tmpwp0wt1qz/alerting/alerting/src/test/kotlin/org/opensearch/alerting/TriggerServiceTests.kt:63:1: Exceeded max line length (140) (cannot be auto-corrected) (max-line-length)
/private/var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/tmpwp0wt1qz/alerting/alerting/src/test/kotlin/org/opensearch/alerting/TriggerServiceTests.kt:65:1: Exceeded max line length (140) (cannot be auto-corrected) (max-line-length)

> Task :ktlint FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':ktlint'.
> Process 'command '/private/var/jenkins/tools/hudson.model.JDK/openjdk-17/bin/java'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org/

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 3s
1 actionable task: 1 executed
2022-08-09 00:57:02 ERROR    Error building alerting, retry with: ./build.sh manifests/2.2.0/opensearch-2.2.0.yml --component alerting --snapshot
Traceback (most recent call last):
  File "./src/run_build.py", line 81, in <module>
    sys.exit(main())
  File "./src/run_build.py", line 68, in main
    builder.build(build_recorder)
  File "/private/var/jenkins/workspace/distribution-build-opensearch/src/build_workflow/builder_from_source.py", line 55, in build
    self.git_repo.execute(build_command)
  File "/private/var/jenkins/workspace/distribution-build-opensearch/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /private/var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/tmpwp0wt1qz/alerting/scripts/build.sh -v 2.2.0 -p darwin -a x64 -s true -o builds' returned non-zero exit status 1.

Hi @prudhvigodithi please work with @opensearch-project/alerting-plugin owners on this.
Thanks.


Additional Resources links broken

Links in Additional Resources return 404s.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 245ea389-75e9-1321-3be4-dc8d8e47a925
Version Independent ID: 1db547e8-a0b0-cf44-dba4-b65786c576fe
Content: Testing ASP.NET Core services and web apps
Content Source: docs/architecture/microservices/multi-container-microservice-net-applications/test-aspnet-core-services-web-apps.md
Product: dotnet-architecture
Technology: microservices
GitHub Login: @nishanil
Microsoft Alias: nanil

Thanks @oneor11
We've fixed this as part of PR https://github.com/dotnet/docs/pull/19901 Closing this issue.


MAINT:sparse.linalg: Use _NoValue for deprecated kwargs

Reference issue
Continuation of #18488, as per #18624
What does this implement/fix?
Adds deprecation _NoValue check mechanism to iterative solvers in order to deprecate the tol  keyword.

Ah restrt turned a few warnings on, is everyone OK if I include those test filters here?


docs: Explain systemd needed for debug console

Add a note to the developer guide explaining that the debug console requires systemd support (hence nominally you cannot use alpine linux for example as that doesn't use systemd).
Fixes #412.
Signed-off-by: James O. D. Hunt<EMAIL_ADDRESS>

Related: https://github.com/kata-containers/osbuilder/issues/96.

Hahaaa, from the Travis CI log:

ERROR: Invalid URL 'https://github.com/kata-containers/runtime' found

Let's hope that's a transitory error. Restarting...


Additions to Service, Domain, and Backend

Service now supports deleted_at; added deleted, created, updated at
Updates documentation on domain; it's for the Service, not account
Added Hostname to backend; if a hostname is used in address, it will show up
here. If an IP is used, hostname will be empty

I had difficulty running tests... it seems they are hardcoded to an account?
I took a spike at refactoring that, but I don't know a way to clean up a service after each test/suite

Hey @catsby,
They are hardcoded to an account. I'm going to try to setup something like VCR to record and commit the responses soonish

Thanks Seth!


ctr reuse Client's GetGRPCConnection

fix #1560
Signed-off-by: Tao Qingyun<EMAIL_ADDRESS>

Codecov Report

Merging #1670 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1670   +/-   ##
=======================================
  Coverage   48.75%   48.75%
=======================================
  Files          27       27
  Lines        4080     4080
=======================================
  Hits         1989     1989
  Misses       1672     1672
  Partials      419      419

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3679a55...faae42e. Read the comment docs.

I'm not sure what you're trying to fix. If the user is specifying its own dial options, then he/she has to specify all of the one it wants, not juste one.

@marcosnils I want to fix this
$ ./ctr images ls
ctr: failed to dial "/run/containerd/containerd.sock": dial unix /run/containerd/containerd.sock: connect: permission denied

$ ./ctr namespaces ls
ctr: grpc: the connection is unavailable: unavailable

this casued by the two client command use different dial options. the images cmd use containerd.New, the namespces cmd use getGRPCConnection in cmd/cyt/utils_unix.go  And I think they should be same.

I see, thanks for clarifying.
I think it would be better for the namespace commands to actually create a new client:
		client, err := newClient(clicontext)
		if err != nil {
			return err
		}
		namespace := client.NamespaceService()

should all cmd create a new client? And the functions  getXXXservice in cmd/utils.go could be removed.

I'd say yes. I think all the underlying services can be accessed from the client.
It just happen that ctr is a dev tool and that those methods probably were introduced later on.
Feel free to update the ones that you found if you so please.

Thank you for explaining. I did the wrong way, feel free to close it.

You can simply force push into your branch with the update 😄
On Sun, Oct 22, 2017, 5:55 PM qingyunha<EMAIL_ADDRESS>wrote:

Thank you for explaining. I did the wrong way, feel free to close it.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/containerd/containerd/pull/1670#issuecomment-338522861,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APUifmj_qPY0ON_3wJ-DGjU5ugUB3cXSks5su-P4gaJpZM4QBw_3
.

--
Sent from my phone.

I really don't like this because it introduces a non-thread safe global to the main client package. This should largely be handled by a client.
Closing in favor of the suggested alternative.

I make a PR as @mlaventure  suggested. PTAL #1673 @mlaventure @stevvooe


Ignore data models and UI related code in coverage reports

I think we should just leave the coverage as it is.

There is no logic stored data models. This  means what we would need to test getters and setters. That is at least in my opinion a complete waste of time.


Added GoDoc badge to README. Fixes #396

Close #396

This change is

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Make initial delay selection more sane

See VelocityPowered/Velocity#476 for justification.

Thank you for the PR!
The current design with two random delays was fully intentional and hitting ratelimits more often is an accepted compromise.
Let me start by explaining why the random delay even exists. The initial logic was very simple: Wait 5 minutes after server startup and then send the data every 30 minutes. What I didn't consider back then, was that many servers tend to restart at a fixed time at a full hour mark (xx:00). This caused a spike in requests every 30 minutes at xx:05 and xx:35 which resulted in a high CPU usage at these times:

The first idea to solve this issue was very similar to your PR: Just make the initial delay a bit random (e.g., MiniDigger suggested 5 min + random between 0 and 15 min). This however comes with a new set of issues:

If the random delay is too large (like the suggested 15 min), it might take a very long time for the server to send data initially and we could miss some
The load would still be unevenly distributed, just more spread out. This becomes especially relevant for smaller random delays like the 5 minutes in this PR. Instead of having about 100% more requests for ~1 minute at xx:05 and xx:35, these extra requests would now be distributed over 5 minutes. This would definitely better but still cause about ~20% more requests in that period.

This means that it is necessary to find a good balance and choose a not too short but also not too long random delay. Or alternatively, find another approach.
The current solution is a hybrid that tries to get the best of both worlds: It still uses a (short) initial random delay (between 3-6 minutes) but then instead of just sending the data every 30 minutes, it introduces a second delay between 0-30 minutes and only then starts the regular 30 minutes interval. This makes it very likely to hit a ratelimit once initially (in fact, in nearly 50% of all cases). Since ratelimits are super (CPU-)inexpensive for the backend to handle, this is no issue at all. However, it also makes sure that requests are evenly distributed and thus solves the issue quite nicely.
That being said, I agree that there is still space for improvements:

It happens quite often that people report bugs because they received a 429 response from the backend. While you have to enable the logFailedRequests config option to see it (which is only intended for debugging), it indeed looks scary with its huge stacktrace and WARN log level. It might be a good idea to handle it differently and log a less verbose message without a stack trace. Something along these lines: "Received 429 ratelimit from bStats. This is expected and not causing issues".
The fixed initial delay of 3 minutes (3 min + random between 0 and 3 min) is a remnant of the old bStats Metrics classes that sent the data for all plugins in a single request and thus needed it to give the other plugins some time to startup (otherwise the first loaded plugin would only send its own data initially). This is not really necessary anymore and can probably be removed.

Let me start by explaining why the random delay even exists.

I know why it exists - having made the Buycraft plugin, we ran into this issue too (and we had the advantage of doing near-infinite scaling on AWS!) and I implemented randomization to spread out the load from servers sending requests to pull purchases from Buycraft.
I'll probably implement your ideas here though, just to make the issue much less scary.

Decided to not pursue this anymore - more pressing things to do.

Thanks anyway 🙂
I've created a new issue for it to not forget it: https://github.com/Bastian/bStats-Metrics/issues/86


Fix Image/Table/Section links in asciidoc

Currently all references in generated Asciidoc for images, tables and sections are just a static text.
We need to make these references interactive, such that on click the forwarding to the position with an according Image/Table/Section will be executed.
We need to define for each image, table referenced in text link names, which will be used for referencing in the text as links.
Former issue: https://github.com/rwth-iat/aas-specs/issues/2

I have locally tackled this issue. But before creating a pull, I want to get some things straight. Here is what my approach has been.
Create a python function to do the following in order to an ascii doc:

Loop through the document and identify all the references to figures by identifiy all the pieces of texts in form Figure XX using regex and replace it with <<figure-xx, Figure XX>>. This makes the text form stay the same but creates a link to item with the id figure-xx. Unfortunately this id is not present in the document, but we create them in the next step so no problem.
Loop through the file and identify all the figures, that is every line that has the form .something
Before these lines, add [#figure-{counter:figure}, reftext="Figure {figure}"]. This sets the figure id in the document from its title (e. g. metamodel-of-referables) to its figure number (i. e. figure-xx).

Now, every mention of figures refer to the respective figure. I have the script and it works perfectly. I did not test it for tables just yet but it should work just fine perfectly too.
The reason I didn't create a pull just yet is that this solution is severely limited to countable references. So we can't use the same solution for sections subsections chapters and clauses. Because these are not countable in a normal manner. For example, a reference to Section <IP_ADDRESS> let's say. We can enumarate sections and subsections with a counter to get this hierarchical structure.
I also am not very satisfied with the solution I have for tables and figures as well. Not that it does not work or anything. However, I would rather have something that is more flexible and does not rely on figure numbers and instead refers to figure titles. I think I can do this as well with a new script.
What would your advice be? Should I create a pull for figures and tables and leave sections/subsections as is for the time, or should I try a new implementation to address all problems at once for the better. I have some ideas for the latter, but I would like to hear from you first.

Let's discuss it tomorrow in the meeting

Try to open word file with this (https://python-docx.readthedocs.io/en/latest/) and add some identifiers to all internal references, so that you can find these when you've converted the word to asciidoc

Fixed with this commit. Please test and report anything needs changing.

Try to open word file with this (https://python-docx.readthedocs.io/en/latest/) and add some identifiers to all internal references, so that you can find these when you've converted the word to asciidoc

This wasn't necessary. At first I tried to open and modify the docx but this causes infinetely many side effects some of them are practically impossible to fix. Any other library or method would come with their own problems. Luckily, this turned out to be completely superfluous anyway. Everything is handled in the ascii doc.
The references for everything are in the table of contents in the ascii doc. What I do is, read this toc, create a dictionary with the necessary replacements, and run them on the text. Then we delete the toc like before.
Unfortunately this isn't enough, because what we are referring is also needed to have the same if as in the toc for reference. This is not the case, but the id is there right next to these elements. These can be formatted in the desired order to get the functionality. That is what we do in the script. This comes with some changes in the old implementation because the old implementation just destroys every reference header completely instead of fixing the formatting and the order of them.
With the default pandoc behavior, only the stuff that refer to the said elements are converted into plain text, what is being referenced and its id stays in the ascii doc. The previous implementation completely destroys these id's that are meant for referencing, which actually are not deleted by pandoc. So I had to completely get rid of the code block responsible for this behavior. The id's for referencing are formatted and the sequence of reference-id, block-name, image itself is made sure to be the same accross the document, in their desired respective order. So the script handles the correct formatting and ordering and functionality of these reference headers now.

Initially solved then reverted the merge. It still works but it adds extra manual fixing work. I am going to make it optional and test it more to make sure I am not missing anything.


updt dl

updt to mosaicml-streaming==0.4.0
To enable training with streams replace this:
# Dataloaders
train_loader:
  name: text
  dataset:
    local: ${data_local}
    remote: ${data_remote}
    split: train_small

with something like this:
# Dataloaders
train_loader:
  name: text
  dataset:
    streams:
      c4_pruned:
        remote: remote_c4_pruned
        split: train
        local: /tmp/c4_pruned/
        proportion: .9
      s2:
        local: ${data_local}
        remote: ${data_remote}
        split: train_small
        proportion: .1

fix some failing tests (@dakinggg sup wit dis? it fine?)

um, maybe the default to low precision layer norm changed the tolerances needed? for the tests that you had to change tolerances, can you try specifying lpln=false?


Building things cost money over time

This is a decision to be made.

In C&C you can buy things even when you don't have money. You pay for the progress made so far (and the price is the total amount it would cost when construction is completed).
In Starcaft / Warcraft you pay the whole price at once and the unit is constructed.

The latter one is easy to get right. The first is easy to get wrong (with rounding errors, etc).
The 2nd option is built in by default, but in case we want the 1st option, we need to do:

[ ] withdraw money over time
[ ] when not enough money for constructing a thing, it should stop constructing. (ie, it is always possible to build a thing regardless if there is enough money)

For now you buy it at once, the build time takes time, credits are substracted immediately. (Starcraft like)