Error interpreting SCSS file: @import '../../../scss/styles';

Link to reproduction
test:int _community
To Reproduce
I installed Payload CMS via the recommended method (npx create-payload-app ) with Node 18.15.0, set up MongoDB, used the blog option for the base installation with full Typescript and ScSS support. I successfully added a number of collections and field hooks. All good.
Then add a custom component as detailed in your documentation:
 admin: {
    defaultColumns: [
      "displayName",
      "user",
      "edition",
      "themes",
      "published",
      "updatedAt",
    ],
    useAsTitle: "displayName",
    components: {
      views: {
        List: SubmissionList,
      },
    },
  },

So far I only have a bare bones component (again based on your documentation):
import React from "react";
import { List, type Props } from "payload/components/views/List";

export const SubmissionList: React.FC<Props> = (props) => (

      Some text before the default list view component. If you just want to do
      that, you can also use the admin.components.list.BeforeList hook

    <List {...props} />

);

Describe the Bug
On enabling any custom collection component, I get this error message when running yarn dev. The project builds correctly, but fails on yarn serve:
`/Users/neil/Sites/platform3/rca/payload/plrca/node_modules/payload/dist/admin/components/icons/Chevron/index.scss:1
@import '../../../scss/styles';
^

SyntaxError: Invalid or unexpected token
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1176:20)
    at Module._compile (node:internal/modules/cjs/loader:1218:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Object.require.extensions.<computed> [as .js] (/Users/xxxx/Sites/platform3/rca/payload/plrca/node_modules/ts-node/src/index.ts:1045:43)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Function.Module._load (node:internal/modules/cjs/loader:958:12)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.<anonymous> (/Users/xxxx/Sites/platform3/rca/payload/plrca/node_modules/payload/src/admin/components/icons/Chevron/index.tsx:3:1)`

Am I missing something? Did I skip a configuration step? It seems to be interpreting the file as typescript. I checked sass and sass-loader have been correctly installed.
Thanks in advance
Payload Version
1.11.8

Is your custom component located within your rootDir defined in your tsconfig.json file?

No, this is what mine looks like after running npx create-payload-app and installing the blog template:
{
  "compilerOptions": {
    "target": "es5",
    "lib": [
      "dom",
      "dom.iterable",
      "esnext"
    ],
    "allowJs": true,
    "strict": false,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "./dist",
    "rootDir": "./src",
    "jsx": "react",
    "paths": {
      "payload/generated-types": [
        "./src/payload-types.ts",
      ],
    }
  },
  "include": [
    "src",
  ],
  "exclude": [
    "node_modules",
    "dist",
    "build",
  ],
  "ts-node": {
    "transpileOnly": true,
    "swc": true,
  }
}

After upgrading to Payload 1.12.0 we got this error:
[nodemon] starting `ts-node --project tsconfig.server.json src/payload/server.ts`
uncaughtException /.../node_modules/payload/dist/admin/components/forms/Label/index.scss:1
@import '../../../scss/styles.scss';
^

SyntaxError: Invalid or unexpected token

Looks like a similar issue. Tried to rebuild Payload but still remains.

I got this error with @payloadcms/plugin-form-builder like @cbratschi

I found that the pure payload project is going well, but face this issue again in the demo project(https://github.com/payloadcms/next-payload-demo) with @payloadcms/plugin-form-builder

I got this fixed by removing the custom import of the Payload config in my Express server. For some reasons I had to use this code at the beginning but this is no longer needed. The Node code should not directly use the Payload config, only the generated schema files and Payload itself.

@neilg63 we haven't heard back in a couple weeks, are you still experiencing this error?
@cbratschi it sounds like you've sorted out your issue - do you need any more help here?
@xyxc0673 are you still stuck with this error? did it happen after you installed @payloadcms/plugin-form-builder and does it go away if you disable the plugin? I'll try to reproduce with this setup.

@JessChowdhury yes, my case is solved.

@JessChowdhury Sure，only happen if I add this plugin to payload config

@xyxc0673 can I take a look at your next.config.js file?
Try adding this line to your next.config.js  and let me know if the error persists:
transpilePackages: ["@payloadcms/plugin-form-builder"],

Same issue here. Bare payload project without NextJS.
Everything runs fine until I add any custom component to a field as described in the docs. As soon as I do this, the error appears.

I manage to track down the issue. The problem is that when the config is Loaded in NodeJS on server side, it will require the compiled Component jsx files. Some of these files contain a require call to SCSS resources to bundle them in webpack, but this will fail on server side.
I managed to add a workaround with intercept-require and return an empty object for these resources, but this is a bit dirty.

Same thing here, was working for a while and when I continued to work on my project, I had the same exact error.
In another project we used transpilePackages: ["@payloadcms/plugin-form-builder"]. But this time payload cms is in standalone / api mod.
I tried this nodemon dev command:
cross-env PAYLOAD_CONFIG_PATH=src/payload.config.ts nodemon -e ts,js --exec ts-node -r tsconfig-paths/register ./src/server.ts
With this custom webpack configuration:
  admin: {
    webpack: config => ({
        ...config,
        externals: [nodeExternals()],
        module: {
          ...config.module,
          rules: [
            ...config.module.rules,
            {
              test: /\.scss$/,
              exclude: /node_modules/,
              use: ['sass-loader'],
            },
          ],
        },
      }),
  },

Using webpack-node-externals and a default sass-loader to just try to dodge this error.
As tux-rampage said previously, it's because of .tsx compilation on server side.
Why a core admin package is crashing issues ? The .scss file is located inside /node_modules/payload/dist/admin/components/forms/Label/index.scss:1. So it's located in dist folder. Shouldn't this file has  been already transpiled to just CSS ?
Is there an easy way to fix it ? thx

Is there an easy way to fix it ? thx

I can only give you a (dirty) work around.
I imported the following code at the beginning of the server entry script (server.ts):
/**
 * This will fix an issue with react component loading on server side.
 * See: https://github.com/payloadcms/payload/issues/3120
 */

import intercept from 'intercept-require';

const cssPattern = /\.(s?css|svg|png|jpe?g|gif|ttf)$/i;

intercept(
    (module, info) => {
        if (info.didShortCircuit) {
            return {};
        }

        if (info.error) {
            throw info.error;
        }

        return module;
    },
    {
        shortCircuit: true,
        shortCircuitMatch: (info) => cssPattern.test(info.extname ?? ''),
    }
);

I am using intercept-require for this. This has no Typings, so you may have to add it manually when using typescript:
declare module "intercept-require" {
    export type InterceptFn = (module: unknown, info: RequireInfo) => unknown;
    export type RestoreFn = () => void;

    export interface RequireInfo {
        readonly moduleId: string;
        readonly callingFile: string;
        readonly native: boolean;
        readonly extname: string|null;
        readonly thirdParty: boolean;
        readonly absPath: string;
        readonly absPathResolvedCorrectly: boolean;
        readonly local: boolean;
        readonly error: Error|null;
        readonly attemptedShortCircuit?: boolean;
        readonly didShortCircuit?: boolean;
    }

    export interface InterceptShortCircuitOptions {
        shortCircuit: true;
        shortCircuitMatch?: (info: RequireInfo) => boolean;
    }

    export type InterceptOptions = InterceptShortCircuitOptions | {shortCircuit: false};

    export default function intercept(interceptor: InterceptFn, options?: InterceptOptions): RestoreFn;
}

Thx for the snippet, I hope I will not have to do this :disappointed:
I think that 50% of my issues are because of things exported inside the payload package. For example, when I want to type things using types inside payload/dist/payload, I have a bad mix between things that should stay in the node ecosystem but are trying to be bundled to my front.
Additional errors are coming form the fact that tsconfig.json is not handle at 100% in standalone mod. Aliases break and we have to fix things using webpack inside payload.config.ts. And then after fixing things with webpack plugins, we have weird behaviors at the build time. We fix things one more time using other webpack configuration like resolve:
{
resolve: {
  modules: [...],
  fallback: {
    "fs": "browserify-fs",

Then, we can have new errors at the run time (using serve after build).
Our solution:

We just removed all  @/ aliases from our app.
Limit what we import from payload or payload/dist and duplicate types in our repository when possible.
Empty webpack config but we still need browserfied packages like : "browserify-fs": "^1.0.0",.

I don't know why I can have issues relative to SCSS setup or browserified-fs when I import things from payload/dist/payload.
Do we have a way to be sure that when importing things like types, 20% of all payloadCMS bundle is not also included ?

Up

Yeah,  seems like we're experiencing the same issue

I'm having similar issues. I'm trying to upgrade to Payload 2.0.13 as well as convert my project to use payload-next.
> next build

WARNING(withPayload): Custom CSS file not found at undefined. Please update your next.config.js file.
 ✓ Creating an optimized production build
 ✓ Compiled successfully
   Linting and checking validity of types  .. ⨯ ESLint: Failed to load config "next/core-web-vitals" to extend from. Referenced from: /Users/someuser/Projects/jd-farm-next/.eslintrc.json
 ✓ Linting and checking validity of types
   Collecting page data  .../Users/someuser/Projects/jd-farm-next/next-app/node_modules/payload/dist/admin/components/elements/Tooltip/index.scss:1
@import '../../../scss/styles.scss';
^

SyntaxError: Invalid or unexpected token
    at Object.compileFunction (node:vm:360:18)
    at wrapSafe (node:internal/modules/cjs/loader:1126:15)
    at Module._compile (node:internal/modules/cjs/loader:1162:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)
    at Module.load (node:internal/modules/cjs/loader:1076:32)
    at Function.Module._load (node:internal/modules/cjs/loader:911:12)
    at Module.require (node:internal/modules/cjs/loader:1100:19)
    at Module.mod.require (/Users/someuser/Projects/jd-farm-next/next-app/node_modules/next/dist/server/require-hook.js:64:28)
    at require (node:internal/modules/cjs/helpers:119:18)
    at Object.<anonymous> (/Users/someuser/Projects/jd-farm-next/next-app/node_modules/payload/dist/admin/components/elements/Tooltip/index.js:13:1)
/Users/someuser/Projects/jd-farm-next/next-app/node_modules/payload/dist/admin/components/elements/Tooltip/index.scss:1
@import '../../../scss/styles.scss';
^

I do have a few custom components, but none of them import Tooltip, and I can't seem to get a stack trace that goes any higher than this.

Following up on this. I had my custom components declared in the same file I was declaring my collection, and therefore a .tsx extension on my Collection.tsx file.
All of my other collections were in .ts files except for this one. I moved my custom component into its own .tsx file and imported it into my Collection.ts and this error went away.

Having the same issue when trying to use form builder on 2.0 via next-payload. Works fine when I build locally but when I deploy to Vercel I get these same errors.

Running into the same issue.

Upon further testing I found something very peculiar: This error breaks all valid SCSS, and as the above mentioned, it seems to be caused by importing payload/dist. I was referencing this component in payload.config.ts like this:
    admin: {
        user: Users.slug,
        bundler: webpackBundler(),
        components: {
            views: {
                GateListDocument: {
                    Component: GateListDocument,
                    path: "/collections/volunteers/gate-list/output",
                },
            },
        },
    },

This is the problematic component:
import { useSearchParams } from "payload/dist/admin/components/utilities/SearchParams";
import React from "react";

import "./index.scss";

const GateListDocument = () => {
    const params = useSearchParams();
    return (

            things

    );
};

export default GateListDocument;

and I don't think it matters (as the compilation will fail for all valid SCSS documents), but here's my index.scss:
@import "~rfs/scss";

@at-root {
  @page {
    size: auto;
    /* 0.84cm */
    margin: 0.84cm 0;
  }

  @page :first {
    margin: 0 0 0.84cm 0;
  }
}

@media print {
  .pagebreak { break-after: page; } /* page-break-after works, as well */
}

.gate-list-document {
    margin: 0;
}

In order for the error to go away, I had to:

Start the backend with yarn dev and wait for it to error.
Comment out these two lines in the component:

import { useSearchParams } from "payload/dist/admin/components/utilities/SearchParams";
...
    const params = useSearchParams();

Restart the server. The error will persist.
Comment out this line:

import "./index.scss";

Restart the server. Now uncomment all 3 lines and wait for nodemon to restart; this time no errors. Edit one of the files, webpack should restart and there shouldn't be any errors.
Restart the server, webpack once again errors.

It seems like there is an issue with compiling SCSS files when a component is imported from payload.config.ts.

It looks like removing PAYLOAD_CONFIG_PATH (which is commonplace for payload v1 package.json content) from the run configs fixed it for me.

Same issue when combining Payload + Nest.js
https://github.com/CloudeyIT/payload-nestjs/issues/1

Same issue when combining Payload + Nest.js CloudeyIT/payload-nestjs#1

Have you tried removing PAYLOAD_CONFIG_PATH=src/cms/payload.config.ts?

Gave it up and then it started working!
Resolved it like this: (note that most of these things relate to Nest.js)

Downgrade to Payload 1 to resolve the scss issue
Add process.env.PAYLOAD_CONFIG_PATH = __dirname.replace('dist', 'src') + '/payload.config.ts' before init
Disable the nest swc builder
Remove the dist folder (or run nest start --watch twice or something)

I started seeing this today, after many hours I narrowed it down to an import statement within an unrelated management function. The import is importing a simple javascript object. I spent long long time trying to figure out why this could possible happen, seemed potentially related to webpack alias' but I really couldn't say definitively. For the moment I'm refining a "global" constant object in multiple places to avoid this. Very weird, but I can say fairly definitively it has nothing specifically to do with the @import '../../../scss/styles'; line

Can anyone link a reproduction?
I just tried to recreate this:

npx create-payload-app
choose website
run yarn dev

This template has a custom component being imported and used on the dashboard. I also tried the custom route/component and that worked fine for me as well.
A reproduction would be very helpful at this point.

Can anyone link a reproduction?
I just tried to recreate this:

npx create-payload-app
chose website
ran yarn dev

This template has a custom component being imported and used on the dashboard. I also tried the custom route/component mentioned above and that worked fine for me as well.
A reproduction would be very helpful at this point.

I don't have any custom components, I think this is a big red herring. I do think this has something to do with imports in server.ts or the payload config, but I think the error itself that payload is displaying is just coincidentally occurring in Tooltip.
To clarify what I was saying before, for me this is happening during import of some command line management scripts that aren't even run during yarn dev and have no react/UI whatsoever. Within my management scripts I am importing some configuration information from my block definitions, but when I comment out my management scripts everything loads and works just fine (admin/API/etc).

Sure, but could you get me something I can pull down on my machine and have it appear? That would be much easier to debug.

Is anyone on this thread still able to reproduce this issue? Please drop us a reply if you are, we're actively trying to resolve this.

Hey everyone! While we would love to help you fix this issue, it is very hard to do so without an actual reproduction. We are going to close this for now, but if anyone wants to submit a link to a branch or fork that throws this error we will gladly re-open it.
If any of you are using next-payload packages or the next-custom-server template - ensure that you are adding use client to the top of any files that import .scss files inside the app folder.
Again, if anyone can link a reproduction we will take a look.
Thanks!

Hi everyone,
Took some time yesterday to create a repro repo, but my usecase was too complex to have a minimal working repository.

More information about the stack here :arrow_down:
Context
Moving from a big Monolith using [Next.js Payload](https://nextjs-vercel.payloadcms.com/) to a Turborepo monorepo.
Our Monorepo

Next.js Website
Standalone PayloadCMS API
Libs mainly imported inside the PayloadCMS instance:

Prisma (done this way: https://turbo.build/repo/docs/handbook/tools/prisma)
Apollo GQL
Types+Config package (done this way: https://turbo.build/repo/docs/handbook/linting/eslint).

For a little more context, I encountered this issue in 4 different way:

Import my monorepo prisma library inside Payload. . Then load this library inside a Payload custom endpoint, to just do a basic push data in both Payload Collection + Prisma collection.
Using a custom webpack resolve.alias like @. Might be relative to the fact that we were resolving .tsx files and not .jsx files since we want to use typescript in our monorepo libraries. At the end, we removed all imports and used relative paths.
Not using Nextjs transpilePackage anymore. Since we are not using Nextjs, we encounter many issues relative to webpack resolve.fallback. Sometimes, after adding packages like os-browserify, we can have this error again.
Importing types from payload/dist/payload. Seems like things other than types are shipped to the browser.

Over time doing this migration, I formed an opinion (maybe bad sorry), but I think those time wasting issues are coming for 2 big concepts:

The bundling ecosystem using webpack + express is awful. As James said during the meeting about "Move to Next.js?", there is a big issue with bundler. Vitejs could resolve the problem, any other solution also might, but at the moment, using PayloadCMS+Typescript with ts-node and express is incredibly daunting. Also, there are differences between development and production builds, costing time for Devops / Infra relative tasks.
All PayloadCMS types not being exposed in its own namespace like payload/types/....

I hope this comment will be somewhat helpful, and I look forward to reading further discussions about this issue.

@thgh Thank you for the reproduction. I'm getting the same error message as you when trying to upgrade Payload CMS to version 2.0.0.

If I add useAuth() from payload/dist/utilities I also get .svg and .png errors. (fixable by overriding require())

why is this closed?
i have dozens of places in libs where i have things like
import {   FieldType,   Options, } from 'payload/dist/admin/components/forms/useField/types';
which is causing this error

Experiencing same problem, but im using astrojs framework, and decided to convert lexical json to html on "frontend", but actually its on server at the time of html generating, copied the code from payload docs,
import type { SerializedEditorState } from 'lexical'
import {
  type SanitizedEditorConfig,
  convertLexicalToHTML,
  consolidateHTMLConverters,
} from '@payloadcms/richtext-lexical'

async function lexicalToHTML(editorData: SerializedEditorState, editorConfig: SanitizedEditorConfig) {
  return await convertLexicalToHTML({
    converters: consolidateHTMLConverters({ editorConfig }),
    data: editorData,
  })
}

and getting such error:
/home/office/projects/20231120_domainRU_bani/node_modules/payload/dist/admin/components/elements/Banner/index.scss:1
@import '../../../scss/styles.scss';

For some reason @payload/richtext-lexical, imports payload with its admin part which has a lot of scss, but it does not make a lot of sense for me, why would we need any scss to convert json to html.

Hey all,
Jarrod and I have just caught up on this thread and we have gotten to the bottom of this.
It appears that potentially -all- of the above issues, including the one reproduction that we received from @thgh (THANK YOU) are importing the Payload config directly, and / or using the Payload config within another service outside of Payload itself.
As you all know, the Payload config is isomorphic, in that it loads both server-side and client-side code into the same config.  This means it loads SCSS files.
But node itself (not even ts-node) understand what to do with a .scss file (or a .css, .svg, etc). If you try to run a little test app that imports one of these files, it will crash.
That's where Webpack / etc. come in. Webpack knows how to handle CSS / SCSS / etc. And that's how Payload currently makes -use- of those files for the client side.
But on the server side, we just need to completely disregard any client-side only files. So to do this, we do the following before we load the Payload config:
  clientFiles.forEach((ext) => {
    require.extensions[ext] = () => null
  })

What the above code does is just says to Node that whenever it finds an .scss file, instead of loading the file, just return null instead.
This effectively dodges the issue.
Here's the important part - we are doing this for you before we load the Payload config. But if you're loading the config directly elsewhere, you need to do this same thing in your own code.
Here's where we perform this code inside of Payload, before we load your config:
https://github.com/payloadcms/payload/blob/main/packages/payload/src/config/load.ts
This is just a side-effect of the beautiful JavaScript world we live in, and there's little Payload can do to make this easier I'm afraid. If you import your Payload config into your own project, you need to be able to handle the client-side files that it comes with

@Extazykeep on your point:

it does not make a lot of sense for me, why would we need any scss to convert json to html.

This is 100% the truth. @AlessioGr has plans to resolve this so that Lexical converters don't load anything unnecessary.

@Extazykeep on your point:

it does not make a lot of sense for me, why would we need any scss to convert json to html.

This is 100% the truth. @AlessioGr has plans to resolve this so that Lexical converters don't load anything unnecessary.

seems like problem already solved, they swap react imports for lazy imports, now it does not require unnecessary admin ui, i guess

@Extazykeep on your point:

it does not make a lot of sense for me, why would we need any scss to convert json to html.

This is 100% the truth. @AlessioGr has plans to resolve this so that Lexical converters don't load anything unnecessary.

seems like problem already solved, they swap react imports for lazy imports, now it does not require unnecessary admin ui, i guess

It's currently only partly solved. SCSS might still make it to the frontend SCSS (even if no actual import happens, because of webpack's static analysis) because it's bundled by webpack. See https://github.com/payloadcms/payload/issues/4085#issuecomment-1847389064

As @thgh mentioned I still having the error
I'm trying to have a NestJs app with payload.
node_modules/payload/dist/admin/components/elements/Banner/index.scss:1
@import '../../../scss/styles.scss';

I'm trying to use payloadCMS on a NestJS app but no luck. I keep getting the error SyntaxError: Invalid or unexpected token even though when I applied the solution of telling node that it shouldn't process certain files
[
  '.scss',
  '.css',
  '.svg',
  '.png',
  '.jpg',
  '.eot',
  '.ttf',
  '.woff',
  '.woff2',
].forEach((ext) => {
  require.extensions[ext] = () => null;
})

I'm trying to use payloadCMS on a NestJS app but no luck. I keep getting the error SyntaxError: Invalid or unexpected token even though when I applied the solution of telling node that it shouldn't process certain files
[
  '.scss',
  '.css',
  '.svg',
  '.png',
  '.jpg',
  '.eot',
  '.ttf',
  '.woff',
  '.woff2',
].forEach((ext) => {
  require.extensions[ext] = () => null;
})

Can you share the full error?

The error is not different of the ones shown before.
/Users/dianjuar_cripto/Documents/rpg-marketplace-project/nest-payloadcms/node_modules/payload/dist/admin/components/elements/Banner/index.scss:1
@import '../../../scss/styles.scss';
^

SyntaxError: Invalid or unexpected token
    at internalCompileFunction (node:internal/vm:128:18)
    at wrapSafe (node:internal/modules/cjs/loader:1280:20)
    at Module._compile (node:internal/modules/cjs/loader:1332:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1427:10)
    at Module.load (node:internal/modules/cjs/loader:1206:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1022:12)
    at Function.Module._load (/Users/dianjuar_cripto/Documents/rpg-marketplace-project/nest-payloadcms/node_modules/@nx/js/src/executors/node/node-with-require-overrides.js:18:31)
    at Module.require (node:internal/modules/cjs/loader:1231:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/Users/dianjuar_cripto/Documents/rpg-marketplace-project/nest-payloadcms/node_modules/payload/dist/admin/components/elements/Banner/index.js:21:1)

I put the suggested code in several places (on the payload.config.ts file, on the main.ts file) but same results

For anyone who may still be having this issue, it is fixed in the latest version of @payload/richtext-lexical. Make sure to update to 0.9.3 and update any lexical packages to 0.13.1. Hope this helps

I tried it again but getting now the node-fetch ESM issue:
 ⨯ ../../node_modules/payload/dist/uploads/getExternalFile.js:61:34
Module not found: ESM packages (node-fetch) need to be imported. Use 'import' to reference the package instead. https://nextjs.org/docs/messages/import-esm-externals

Import trace for requested module:
../../node_modules/payload/dist/uploads/generateFileData.js
../../node_modules/payload/dist/collections/operations/create.js
../../node_modules/payload/dist/collections/requestHandlers/create.js
../../node_modules/payload/dist/collections/buildEndpoints.js
../../node_modules/payload/dist/collections/initHTTP.js
../../node_modules/payload/dist/initHTTP.js
../../node_modules/payload/dist/index.js
<EMAIL_ADDRESS><EMAIL_ADDRESS>./src/app/_components/richtext/richtext-lexical.tsx
./src/app/[lang]/[[...slug]]/_page-types/generic.tsx
./src/app/[lang]/[[...slug]]/_page-types/index.tsx
./src/app/[lang]/[[...slug]]/page.tsx

Somehow the Lexical code initializes a full payload instance and generateFileData() gets called.
I guess we have to wait for Payload 3 to fully use Lexical from Next.js.
More details about the ESM issue are here:
https://github.com/payloadcms/payload/issues/4421#issuecomment-2064129906

The upload feature is causing this:
https://github.com/payloadcms/payload/blob/main/packages/richtext-lexical/src/field/features/Upload/index.ts#L3


Use Angular 1 Promises

Using angular 1 promises would aid in compatibility of the angular 1 integration. Wrapping the $ionicSomething factories in angular 1 promises would provide features such as the finally function and integration with angular change detection. This issue is to investigate how to do that.
Possibly use when?
Thanks @astik for the original issue: https://github.com/driftyco/ionic-platform-web-client/issues/110

One option for angular 1 projects would be to use the bluebird promise library. It exposes its scheduler which can then be tied to angular's digest cycle effciently.
The angular-bluebird-promises library takes care of the integration and replaces the $q service implementation with the bluebird implementation for you.

@meticoeus what is the advantage of using bluebird over the builtin $q?
Currently I wrap my promises with $q.all([promise]), but I'm not sure why... Will test with $q.when(promise) instead, since that seems to be the preferred way to do it.

I guess bluebird adds more features? We don't need the additional features and I'd rather not have another dependency included in our bundle. We should be able to make use of $q to wrap our promises.

An option for a user of the library is to replace the global Promise from es6-promises (or native implementation) with bluebird promises. All uses of Promise in this and other libraries and your app code will be using a single implementation that can now inter-op with the digest cycle. You can then use the API directly instead of the wrapped versions, which may make sense in project that is being migrated. Just import the module in the normal way

Oh, I see! That native implementation replacement and digest cycle hook-in is something I've been wondering about. An option would be to document how to set up that library in Ionic 1 projects. We can't depend on it because our client must work for angular 1, Angular 2, and non-angular projects.
I personally think wrapping the promises with angular's $q would be better because it's an api that angular devs are familiar with. It just seems more consistent.

The $q wrapper approach is probably the best way to go to keep things simple and familiar for ng1 devs.
Documenting that you can and how setup bluebird so that you can avoid the wrapper might be useful for anyone doing a 1->2 migration so they can use the library directly without the ng1 modules.

For reference to anyone considering this strategy, there was already a discussion about making bluebird usable in angular 2 before they dropped $q for native promises.
The main take away is that $q is silent about uncaught promise rejections while bluebird is not. Some ng1 libraries like ui-router are guilty of this so your console will blow up until you change bluebird's configuration. Of course, that feature is useful when reviewing your own code.

@meticoeus @Maistho Got this working! If window.Promise is undefined, set it to $q, otherwise patch our use of promises with $q.when. 🎉


checkpoint lag is increasing while sink rows decreasing a lot

What did you do?

3 cdc node, 1 changefeed, kafka sink
workload: sysbench oltp_write_only 16 thread, 16 tables, prepare and run

What did you expect to see?
No response
What did you see instead?

Versions of the cluster
Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):
(paste TiKV version here)

TiCDC version (execute cdc version):
(paste TiCDC version here)

Duplicate of https://github.com/pingcap/tiflow/issues/5459


AKAudioFile init Fail

i got error like that
Error Domain=com.apple.coreaudio.avfaudio Code=1954115647 "(null)" UserInfo={failed call=ExtAudioFileOpenURL((CFURLRef)fileURL, &_extAudioFile)}
when i was init an audioFile "self.audioFile = try AKAudioFile(forReading: url)"
url = file:///var/mobile/Media/DCIM/106APPLE/IMG_6837.MOV
is this is the format problem or others, thanks

I got this error, too. How to fix this? Thanks

This is an old query but I'd like to help out, I had the same problem when I tried the same working code (on the simulator) on the device. The problem is happening somehow when the URL is not correct. Url variable is not nil but not correct. Check if you append a folder by adding isDirectory = true
like:  url.appendingPathComponent(songsFolderName, isDirectory: true)
So it is not about akfile.
I hope this helps to others.

This is an old query but I'd like to help out, I had the same problem when I tried the same working code (on the simulator) on the device. The problem is happening somehow when the URL is not correct. Url variable is not nil but not correct. Check if you append a folder by adding isDirectory = true
like:  url.appendingPathComponent(songsFolderName, isDirectory: true)
So it is not about akfile.
I hope this helps to others.


MacOS Input Fixes is Incompatible

Mod Name
MacOS Input Fixes
Mod Homepage
https://modrinth.com/mod/macos-input-fixes
Describe the bug
Crashes on startup
Steps to reproduce

Download necessary mods
Start the game
Crashes

Logs
https://mclo.gs/LcQdb2W
Additional context
No response

https://mclo.gs/JqJjacx crashes when I go to the settings in the mod
Options > Controls > Mouse settings

However the mod itself works now. thank you

@Aces-and-Jacks My PR to MacOS Input Fixes should fix that issue: https://github.com/hamarb123/MCMacOSInputFixes/pull/31


switchLocalePath() returns empty string on error 404

Environment

Operating System: Linux
Node Version:     v20.11.0
Nuxt Version:     3.13.2
CLI Version:      3.13.2
Nitro Version:    2.9.7
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      compatibilityDate, devtools, ssr, build, nitro, modules, vite, i18n, eslint, security
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules:    -

Reproduction
<script setup lang="ts">
import { computed } from 'vue'
import { useI18n } from 'vue-i18n'

const { availableLocales } = useI18n()

const localeSelectorItems = computed(() => {
  return availableLocales.map((loc: string) => {
    return {
      locale: loc
    }
  })
})
</script>

<template>
  <NuxtLink
    v-for="item in localeSelectorItems"
    :key="item.locale"
    :to="switchLocalePath(item.locale)"
  >
    {{ item.locale }}
  </NuxtLink>
</template>

Describe the bug
According to nuxt docs, the error.vue page to generate a custom error page has to be located outside of the pages directory in the src directory side by side with App.vue.
If I use the switchLocalePath() function on this page, it will always return a empty string instead of the actual locale path.
I'm using the same piece of code in src/error.vue and in src/pages/index.vue, so I can say for sure that it should work.
Additional context
No response
Logs
No response

So what's missing from the code I already posted?

You have provided a code snippet but your issue could be caused by a variety of factors such as config and project structure. A minimal reproduction (stackblitz or repository) with steps on how to reproduce your issue would allow me to actually debug what's happening.

https://stackblitz.com/edit/bobbiegoede-nuxt-i18n-starter-za36nq?file=components%2FLangSwitcher.vue
I used your starter template. I copy & pasted the code from app.vue into error.vue and it already shows --> locale switcher links are broken

Hmm I see what you mean, what do you expect switchLocalePath to return in this case?

The same as everywhere else. I use the same top bar for all pages, so I expect the language switcher to work everywhere the same. The error is just a different kind of content, but I don't see that page any different from the other pages. And I wonder why Nuxt does.

TBH I'm not a fan of all the crude magic going on inside Nuxt. In Vue I would just have defined a catchall route in the router. But as Nuxt is stubborn about being different, I think that the nuxt-i18n module should conform to that stubbornness.
At least the Nuxt error page takes care of sending the right http status code to the client. I suppose that I would have to create all that my self with the catchall approach?

The catch all route would only be used for non existent pages, check the docs on how to make one here https://nuxt.com/docs/guide/directory-structure/pages#catch-all-route.
The language switcher will keep working on any error page as long as the page actually exists for there to be localized variants of those, errors other than 404 should render the error.vue page.

catch all route would only be used for non existent pages

But a catchall is different from an error page. Catchall will send 200 for pages that should send 404.
I'll look into that approach, should be possible to mimick a real error page using setResponseStatus().

But a catchall is different from an error page. Catchall will send 200 for pages that should send 404.

I agree they're different, but it would resolve the issue of the language switcher not working.
It seems like the underlying behavior is clear for this issue, language switching does not work on non existent pages combined with Nuxt considering the error page not to be a 'page', does that sound right? For Nuxt specific behaviors you'll have better luck opening an issue on the Nuxt repository or asking in the Discord.

I agree they're different, but it would resolve the issue of the language switcher not working.

The package ist called nuxt-i18n. So I expect it to work with Nuxt. Not doing things the Nuxt way to make the package work is an anti-pattern. There should at least be some error or note in the docs that explains the behavior. Just silently returning an empty string is a bug that is hard to catch.

So what do you expect switchLocalePath to return on a non existent path (404)? The error 'page' is not a real route or page, so what would you consider the localized equivalent of no path?

I want to be able to switch languages, even on the error page. If the user is on the non-existing route /en/foo and they switch to de, the route should be/de/foo. It doesn't matter if the route exists within Nuxt or not. It's the route that is visible in the url bar, that's what the user sees.

and maybe /en/foo even was a broken link that should have been /de/foo, so the route might exist in one locale but not in the other. If there are no route localization rules specified in i18n.pages, I would expect it to stupidly swap out the locale part of the path, no matter if the route exists or not.

If a route exists in one language then it does work the way you describe, which ironically is actually an issue for other users, see https://github.com/nuxt-modules/i18n/issues/2782.
You can switch language fine on the error page (using setLocale for example), you're just not getting a localized path from switchLocalePath since there's no path to localize.

I would expect it to stupidly swap out the locale part of the path, no matter if the route exists or not.

If you're on a non existent page, like /foo, would you expect router.resolve({ query: { bar: '123' } }) (resolves current route with query) to return /foo?bar=123?

If a route exists in one language then it does work the way you describe, which ironically is actually an issue for other users, see https://github.com/nuxt-modules/i18n/issues/2782.

I see the confusion here. But however you want to solve this, it should be consistent from the users' point of view.

You can switch language fine on the error page (using setLocale for example), you're just not getting a localized path from switchLocalePath since there's no path to localize.

But how would this work with prefixed routes?
Imagine the user to be on the non-existing route /en/foo but they want to read that page in German, so they switch the language. Would the url still be /en/foo but in German? This doesn't make sense, because for all other pages it would switch to /de/foo.
From the users' POV, the url of an existing and non-existing route should follow the same rules. Everything else is just confusing. The only difference between existing and non-existing routes is on the technical side.
I think the high-level question that causes the confusion is: do we expect that a page exists in every locale? I would think so. If there's no translation, the page will show it's content in the fallback-locale.
So we can also assume that every page that does not exist in one locale doesn't exist in all others. So a 404 in English will also be a 404 in German.


[VCF] Question on usage

Follow up question after resolving this(https://github.com/sbslee/fuc/issues/62) on the usage.

I want to find common variants, and unique to each of the vcf. (2 vcf files from mutect both)
Secondly, I want to concatenate vcf calls from strelka and mutect, and remove duplicates.

Could you suggest which methods to use for these?
thanks,
Rohan

Because I'm currently out of town and don't have my computer with me, please excuse me for not being able to provide the fullest answers. But I will try my best :) I should be back by Sunday (in Korean time), so if below doesn't get you where you need, please let me know and I will try to provide additional support as soon as I get back.
For the first question, there are a number of ways to achieve the goal, but I suggest to use the pyvcf.VcfFrame.to_variants() method:
from fuc import pyvcf

vf1 = pyvcf.VcfFrame.from_file('in1.vcf')
vf2 = pyvcf.VcfFrame.from_file('in2.vcf')

variants1 = vf1.to_variants()
variants2 = vf2.to_variants()

common_variants = list(set(variants1).intersection(variants2))
unique_variants1 = [x for x in variants1 if x not in variants2]
unique_variants2 = [x for x in variants2 if x not in variants1]

For the second question, again, there are a number of ways but I recommend using the pyvcf.merge() method with the collapse=True option:
merged_vf = pyvcf.merge([vf1, vf2], how='outer')
merged_vf.to_file('out.vcf')

This will merge two VcfFrame objects while removing duplicate records.

The answer to the first question worked , thank you!
Regarding second question on merging variants from 2 callers,  when doing outer join, it does not work.
File "/home/ubuntu/miniconda3/envs/fuc/lib/python3.9/site-packages/fuc/api/pyvcf.py", line 5277, in
key=lambda col: [CONTIGS.index(x) if isinstance(x, str)
ValueError: 'MT' is not in list

@ironb25,
Thanks for the patience, I'm back! Glad to hear the first question is resolved.
For the second question, the problem occurred because currently the pyvcf.VcfFrame.sort method expects you to use either M or chrM for indicating the mitochondria contig (you are using MT). Clearly, this needs to be fixed because people need to be able to use pyvcf.VcfFrame.sort regardless of what contigs their input VCF file has. I will update this for the next release 0.34.0 -- it should be an easy fix. In the meantime, can you send me the two VCF files you are trying to merge? This way, I can make sure there are no other surprises for you. But it's totally fine if you can't send them to me.

I shared the file in the email.
On Mon, Jun 6, 2022 at 6:44 AM sbslee @.***> wrote:

@ironb25 https://github.com/ironb25,
Thanks for the patience, I'm back! Glad to hear the first question is
resolved.
For the second question, the problem occurred because currently the
pyvcf.VcfFrame.sort method expects you to use either M or chrM for
indicating the mitochondria contig (you are using MT). Clearly, this
needs to be fixed because people need to be able to use
pyvcf.VcfFrame.sort regardless of what contigs their input VCF file has.
I will update this for the next release 0.34.0 -- it should be an easy
fix. In the meantime, can you send me the two VCF files you are trying to
merge? This way, I can make sure there are no other surprises for you. But
it's totally fine if you can't send them to me.
—
Reply to this email directly, view it on GitHub
https://github.com/sbslee/fuc/issues/63#issuecomment-1147312248, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/APQ4JY2VW5NNJKPLYT2EELLVNXJCBANCNFSM5XWVFWDA
.
You are receiving this because you were mentioned.Message ID:
@.***>

@ironb25,
I updated the pyvcf.VcfFrame.sort method to handle custom contigs such as MT. This resolves your second question. I sent you the output VCF after merging.
FYI, I noticed that the output VCF has no genotype data (e.g. 0/1) for the samples NORMAL and TUMOR. When I looked into this more closely, I found that the strelka2.vcf file does not have the GT field at all. Was this intended?

FYI, you can try out the updated fuc yourself with the following:
$ git clone https://github.com/sbslee/fuc
$ cd fuc
$ git checkout 0.34.0-dev
$ pip install -e .

Since the two original issues have been resolved, I will close this issue. Please feel free to re-open it if necessary.


setup my webhook facebook issue

I have a simple issue when I am going to setup my webhook Facebook gives me following error:
The URL couldn't be validated. Response does not match challenge, expected value="1909028616", received="Error, wrong token"
Note that, I have correctly used git and Heroku.
Thank you

Hey, this means that the token verification did not work. Have you set the WEBHOOK_VERIFY_TOKEN in your .env file? and it is the same as in your FB app?


Roll engine 849fa42a06a5..208011758ffe (3 commits)

https://github.com/flutter/engine/compare/849fa42a06a5...208011758ffe
git log 849fa42a06a5c09ade2d88e48d982e069f68d969..208011758ffe7705b2b8a88fef3b6c38a3d3c3dd --no-merges --oneline
208011758 Roll src/third_party/skia 3f79f07a14eb..348880c3f67f (3 commits) (flutter/engine#8971)
45599515d Roll src/third_party/skia c8a84d2b7ff9..3f79f07a14eb (1 commits) (flutter/engine#8968)
f5fa03b5e Use Android text selection shifting API to handle keyboard backspace (flutter/engine#8956)
The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
If the roll is causing failures, please contact the current sheriff (jsimmons@google.com), and stop
the roller if necessary.

Auto-roller completed checks. About to merge.


Preprocessing does not progress and ends with OOM error

Hi,
When I want to use autrainer preprocess in order to obtain features previous to training I have the following problem.
For a long time it looks like this:
Preprocessing datasets...
 - BIRDNETV2-32k

BIRDNETV2-32k:   0%|          | 0/92741 [00:00<?, ?it/s]

And then after some time it ends up in an OOM error:
slurmstepd: error: Detected 1 oom_kill event in StepId=68059.batch. Some of the step tasks have been OOM Killed.
I already increased the gpu memory and descreased the batch size, but it's still not working which makes me wonder what I am doing wrong?
I use this dataset conf and my data splits are train, dev, test as required:
id: BIRDNETV2-32k
_target_: birdnetv2.BIRDNETV2

path: /nas/staff/data_work/AG/HearTheSpecies/birdnet_v2/splits
features_subdir: log_mel_32k_adapted
index_column: file
target_column: species
file_type: npy
file_handler: autrainer.datasets.utils.NumpyFileHandler

criterion: autrainer.criterions.BalancedCrossEntropyLoss
metrics:
  - autrainer.metrics.Accuracy
  - autrainer.metrics.UAR
  - autrainer.metrics.F1
tracking_metric: autrainer.metrics.Accuracy

transform:
  type: grayscale

My dataset class simply extends BaseMLClassificationDataset and does not change any of the implemented functions, as it should work with the default settings.
Do you have an idea how I could solve this?

I already increased the gpu memory and descreased the batch size, but it's still not working which makes me wonder what I am doing wrong?

During preprocessing, we keep all data on the CPU at the moment (this also holds true for any type of transform; unless explicitly changed by the user in the AbstractDataset implementation), so no GPU memory should be allocated.
The batch_size argument in the main configuration is also not used during preprocessing.
My guess would be that us setting -n / --num-workers to -1 (e.g. trying to use os.cpu_count() as a sane default) should be reconsidered and be 1 by default.
It could be possible that os.cpu_count() reports all resources available on a node and not just the allocated CPU cores through Slurm.
I will investigate this soon, and in general default to 1 as it is generally more save, thanks for opening the issue!
In the mean time, please let me know if running with a single thread fixes the OOM kill:
autrainer preprocess -n 1

Can you add a pointer to the code? I've only found this here, which seems like it's storing each feature to disk? Why do we keep them all to memory anyway? That won't work for big datasets..

In the mean time, please let me know if running with a single thread fixes the OOM kill:
autrainer preprocess -n 1

I tried it like this and let it run for a while now (45 minutes), but now it does not even show the progress bar anymore which was the case before at this time. Maybe as it takes longer now, but still, after 45minutes I thought it should be there right?

Edit: You probably also have to manually delete the log_mel_32k_adapted directory, as the preprocessing pipeline would be skipped otherwise and assume that the features are already extracted (pretty sure it exists since the OOM would kill the process after the creation).

It seems it does not even create this folder. At least I couldn't find it.

This is where we select the number of workers (threads):
https://github.com/autrainer/autrainer/blob/260ccf9a3ed1cfa90c524f5c043a89394cfcd8b3/autrainer/core/scripts/preprocess_script.py#L139-L142

I've only found this here, which seems like it's storing each feature to disk? Why do we keep them all to memory anyway?

We load each file separately here, i assumed this would be automatically collected as data is overridden in each iteration, however we can explicitly remove it after saving with del data. But i dont think it would hold the entire dataset.

Okay I checked an older job where I specified a larger memory (32GB) and smaller batch size (16) and after some hours it actually started to process now:
Preprocessing datasets...
 - BIRDNETV2-32k

BIRDNETV2-32k:   0%|          | 0/92741 [00:00<?, ?it/s]
BIRDNETV2-32k:   0%|          | 100/92741 [2:35:59<2408:31:05, 93.59s/it]
BIRDNETV2-32k:   0%|          | 200/92741 [2:36:44<997:31:50, 38.81s/it]
BIRDNETV2-32k:   0%|          | 300/92741 [2:36:58<543:07:04, 21.15s/it]
BIRDNETV2-32k:   0%|          | 400/92741 [2:36:59<328:24:03, 12.80s/it]
BIRDNETV2-32k:   0%|          | 400/92741 [2:37:11<328:24:03, 12.80s/it]
BIRDNETV2-32k:   1%|          | 500/92741 [2:37:28<212:25:46,  8.29s/it]
BIRDNETV2-32k:   1%|          | 600/92741 [2:37:54<142:16:24,  5.56s/it]
BIRDNETV2-32k:   1%|          | 700/92741 [2:37:59<96:08:20,  3.76s/it]
BIRDNETV2-32k:   1%|          | 700/92741 [2:38:10<96:08:20,  3.76s/it]

So it seems now it's working without OOM error.  I'll keep you updated.

I tried it like this and let it run for a while now (45 minutes), but now it does not even show the progress bar anymore which was the case before at this time. Maybe as it takes longer now, but still, after 45minutes I thought it should be there right?

Yes it will take longer, but it is definitely strange that no progress bar is showing at all. As a last resort, maybe try to increase the threads slightly, and update the progress bar after each iteration:
autrainer preprocess -n 4 -p 1

It seems it does not even create this folder. At least I couldn't find it.

This is also strange, as all (sub) directories should be created before extracting the features and after all csv files present in the root directory of the dataset are read in to identify all unique files.

Yes it will take longer, but it is definitely strange that no progress bar is showing at all. As a last resort, maybe try to increase the threads slightly, and update the progress bar after each iteration:
autrainer preprocess -n 4 -p 1

Okay I'll try this for the newest run.

It seems it does not even create this folder. At least I couldn't find it.

This is also strange, as all (sub) directories should be created before extracting the features and after all csv files present in the root directory of the dataset are read in to identify all unique files.

I thought so, too. But even now that the one extraction seems to run, there is still no log_mel_32k_adapted folder. The folder should be created in the specified path of the dataset yaml right?

Edit:

Okay I checked an older job where I specified a larger memory (32GB) and smaller batch size (16) and after some hours it actually started to process now:

is this using autrainer preprocess -n 4 -p 1?

No this is actually still using the old settings where I just ran autrainer preprocess.

The folder should be created in the specified path of the dataset yaml right?

Very interesting, yes the final path should be {path}/{features_subdir}/some_file.npy for some extracted file. Both variables come from the dataset configuration YAML file.

Summary:

set -n and -p to 1 by default, and add additional information on how exactly samples are process during autrainer preprocess (sequentially, one by one, optionally with threads (num_workers > 1) to avoid I/O bottlenecks)
explicitly delete the processed data after it is saved (del data) to decrease memory usage

Okay I checked an older job where I specified a larger memory (32GB) and smaller batch size (16) and after some hours it actually started to process now:

Also weird that it takes "some hours" to arrive there or not? For other datasets I've tried this was all rather fast (done in minutes, at most an hour)?

Also weird that it takes "some hours" to arrive there or not? For other datasets I've tried this was all rather fast (done in minutes, at most an hour)?

This was most likely the reason of too many threads and inefficient code. I will open a PR fixing both.
It may still take some time to process data if e.g. data is stored on a NAS.


Implement a lazy set for the migration to the new node key format

Context
The migration to the new node key format is tricky. To avoid stopping the network, it introduces a lazy set. We will accept the existence of both new and old format nodes in the tree.
Proposal

Refactor the Node structure to accept both old and new nodes, like this

// Node represents a node in a Tree.
type Node struct {
	key           []byte
	value         []byte
	hash          []byte
	nodeKey       *NodeKey
	leftNodeKey   []byte  // it can be a node hash for the old node and be a node key serialization for the new node.
	rightNodeKey  []byte
	size          int64
	leftNode      *Node
	rightNode     *Node
	subtreeHeight int8
}

Setting new nodes will store the node as a new node key format. It will refactor them to the new format for old nodes that face while updating the tree.

Sounds great, so there are just different versions of node key, new version could reference old version, but not the other way.

Sounds great, so there are just multiple versions of node key, new version could reference old version, but not the other way.

exactly


Intermittent test failure in folder reprocessing

See https://travis-ci.org/oaeproject/Hilary/builds/40199911 and S3 logs at oae-testing/travisci/logs/hilary/build-2138-33cba0f68c755c73f434dea7659325197b48bace-Thu Nov 06 2014 17:09:33 GMT+0000 (UTC).log.gz

1202 passing (13m)
2 failing

Folders Indexing verify folders can be reindexed:
Uncaught AssertionError: 0 === 1
at /home/travis/build/oaeproject/Hilary/node_modules/oae-folders/lib/test/util.js:1009:16
at /home/travis/build/oaeproject/Hilary/node_modules/oae-search/lib/test/util.js:81:28
at Request._callback (/home/travis/build/oaeproject/Hilary/node_modules/oae-rest/lib/util.js:9:8822)
at Request.self.callback (/home/travis/build/oaeproject/Hilary/node_modules/oae-rest/node_modules/request/request.js:121:22)
at Request.emit (events.js:98:17)
at Request. (/home/travis/build/oaeproject/Hilary/node_modules/oae-rest/node_modules/request/request.js:978:14)
at Request.emit (events.js:117:20)
at IncomingMessage. (/home/travis/build/oaeproject/Hilary/node_modules/oae-rest/node_modules/request/request.js:929:12)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:938:16
Preview processor Preview generation verify folder processing:
Uncaught AssertionError: "undefined" == true
at /home/travis/build/oaeproject/Hilary/node_modules/oae-preview-processor/tests/test-previews.js:1576:64
at /home/travis/build/oaeproject/Hilary/node_modules/oae-folders/lib/test/util.js:440:16
at Request._callback (/home/travis/build/oaeproject/Hilary/node_modules/oae-rest/lib/util.js:9:8822)
at Request.self.callback (/home/travis/build/oaeproject/Hilary/node_modules/oae-rest/node_modules/request/request.js:121:22)
at Request.emit (events.js:98:17)
at Request. (/home/travis/build/oaeproject/Hilary/node_modules/oae-rest/node_modules/request/request.js:978:14)
at Request.emit (events.js:117:20)
at IncomingMessage. (/home/travis/build/oaeproject/Hilary/node_modules/oae-rest/node_modules/request/request.js:929:12)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:938:16

I think this has been fixed now


Setting Shards for an index

[ ] elastic.v6 (for Elasticsearch 6.x)
How to set shards to 0 from default 5

You can either set the number of shards at creation time (example code) or later via Update Indices Settings API (example code).


None

We've been here before. See #203.

This was my first attempt for contribution.
Should I verify with you before my next pull request ?

This was my first attempt for contribution.

Welcome!

Should I verify with you before my next pull request ?

This is always a good idea irrespective of the project you're planning to contribute to so that you don't waste your time and effort if the contribution isn't going to be accepted. Have a look at the “Opening an issue” and “Opening a pull request” sections of the “How to Contribute to Open Source” guide for some direction if you're new to contributing generally.
I'm going to close this for now for reasons already discussed in #203. If you want to debate those, feel free to add to the conversation there.


Adding AllWorkSheets option to Import-Excel to return the complete document in one PSObject

I Needed to be able to import a full Workbook instead of a single sheet, if possible without iterating through worksheets until it failed.
To avoid breaking backward compatibility I implemented the option as another parameter set, while setting the default parameter set as the original function syntax.
I think the form
if($AllWorkSheets) { ... }
else { ... }

is clearer/less verbose than
switch ($PsCmdlet.ParameterSetName) {
         'AllSheets' { ... }
         'SingleSheet' { ... }
}

Let me know if you're happy with this quick hack, or if you have any suggestion.

There's a weird behavior that I just realized... Closing pull request.
Looks like it reads only the properties set in the first sheet for the other sheets. Sounds like some variable needs to be re-initialized in the foreach loop...
Sorry for the inconvenience

No problem. I really like the idea. Let me know if I can help


UART - design task - timeout counters safety twards the RC if a write/read request is not served

Resolved - done


Timed out running command when passing forecast data

Hassos 2023.2.5
Emhass addon 0.2.29
Home Assistant Yellow

I am trying to pass cost and sell electricity prices from Nordpool addon to emhass.
I am using the shell_command from the emhass documentation:
shell_command:
  trigger_nordpool_forecast: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim"

which outputs this in the Template editor:

Emhass gives no error, but the load_cost_forecast and prod_price_forecast are not updated

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
[2023-03-03 16:59:38,141] INFO in web_server: Launching the emhass webserver at: http://<IP_ADDRESS>:5000
[2023-03-03 16:59:38,143] INFO in web_server: Home Assistant data fetch will be performed using url: http://supervisor/core/api
[2023-03-03 16:59:38,143] INFO in web_server: The data path is: /share
[2023-03-03 16:59:38,148] INFO in web_server: Using core emhass version: 0.3.36
[2023-03-03 17:00:00,583] INFO in command_line: Setting up needed data
[2023-03-03 17:00:00,700] INFO in web_server:  >> Publishing data...
[2023-03-03 17:00:00,700] INFO in command_line: Publishing data to HASS instance
[2023-03-03 17:00:00,770] INFO in retrieve_hass: Successfully posted to sensor.p_pv_forecast = 60.62
[2023-03-03 17:00:00,807] INFO in retrieve_hass: Successfully posted to sensor.p_load_forecast = 1281.76
[2023-03-03 17:00:00,846] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable0 = 0.0
[2023-03-03 17:00:00,886] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable1 = 0.0
[2023-03-03 17:00:00,922] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable2 = 1450.0
[2023-03-03 17:00:00,961] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable3 = 0.0
[2023-03-03 17:00:00,994] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable4 = 0.0
[2023-03-03 17:00:01,028] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable5 = 0.0
[2023-03-03 17:00:01,061] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable6 = 947.0
[2023-03-03 17:00:01,097] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable7 = 0.0
[2023-03-03 17:00:01,129] INFO in retrieve_hass: Successfully posted to sensor.p_grid_forecast = 3618.14
[2023-03-03 17:00:01,162] INFO in retrieve_hass: Successfully posted to sensor.total_cost_fun_value = -18.05
[2023-03-03 17:00:18,726] INFO in command_line: Setting up needed data
[2023-03-03 17:00:18,731] INFO in forecast: Retrieving weather forecast data using method = scrapper
[2023-03-03 17:00:25,001] INFO in forecast: Retrieving data from hass for load forecast using method = naive
[2023-03-03 17:00:25,009] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2023-03-03 17:00:28,665] INFO in web_server:  >> Performing dayahead optimization...
[2023-03-03 17:00:28,665] INFO in command_line: Performing day-ahead forecast optimization
[2023-03-03 17:00:28,700] INFO in optimization: Perform optimization for the day-ahead
[2023-03-03 17:01:13,395] INFO in web_server: EMHASS server online, serving index.html...
[2023-03-03 17:05:00,661] INFO in command_line: Setting up needed data
[2023-03-03 17:05:00,678] INFO in web_server:  >> Publishing data...
[2023-03-03 17:05:00,679] INFO in command_line: Publishing data to HASS instance
[2023-03-03 17:05:00,791] INFO in retrieve_hass: Successfully posted to sensor.p_pv_forecast = 60.62
[2023-03-03 17:05:00,855] INFO in retrieve_hass: Successfully posted to sensor.p_load_forecast = 1281.76
[2023-03-03 17:05:00,935] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable0 = 0.0
[2023-03-03 17:05:01,002] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable1 = 0.0
[2023-03-03 17:05:01,062] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable2 = 1450.0
[2023-03-03 17:05:01,118] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable3 = 0.0
[2023-03-03 17:05:01,178] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable4 = 0.0
[2023-03-03 17:05:01,230] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable5 = 0.0
[2023-03-03 17:05:01,290] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable6 = 947.0
[2023-03-03 17:05:01,344] INFO in retrieve_hass: Successfully posted to sensor.p_deferrable7 = 0.0
[2023-03-03 17:05:01,399] INFO in retrieve_hass: Successfully posted to sensor.p_grid_forecast = 3618.14
[2023-03-03 17:05:01,450] INFO in retrieve_hass: Successfully posted to sensor.total_cost_fun_value = -18.05
[2023-03-03 17:10:00,622] INFO in command_line: Setting up needed data
[2023-03-03 17:10:00,630] INFO in web_server:  >> Publishing data...

Hassos logs Timed out running command :
Logger: homeassistant.components.shell_command
Source: components/shell_command/__init__.py:83
Integration: Shell Command (documentation, issues)
First occurred: 17:01:18 (1 occurrences)
Last logged: 17:01:18

Timed out running command: `curl -i -H "Content-Type: application/json" -X POST -d '{"load_cost_forecast":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},"prod_price_forecast":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim`, after: 60s
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/streams.py", line 502, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/subprocess.py", line 195, in communicate
    stdin, stdout, stderr = await tasks.gather(stdin, stdout, stderr)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/shell_command/__init__.py", line 83, in async_service_handler
    stdout_data, stderr_data = await asyncio.wait_for(
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

Hassos log:
home-assistant_2023-03-03T16-07-07.463Z.log

Ok could you please restart the add-on then don't do any data publish task, then show the complete logs of just running the day-ahead optimization task?

I have only Nordpool today prices when doing this test.
Running this day-ahead automation:
alias: EMHASS day-ahead optimization
description: ""
trigger:
  - platform: time
    at: "23:10:00"
condition: []
action:
  - service: shell_command.trigger_nordpool_forecast
    data: {}
mode: single

I use this Shell_command:
shell_command:
  publish_data: "curl -i -H 'Content-Type:application/json' -X POST -d '{}' http://localhost:5000/action/publish-data"

  trigger_nordpool_forecast: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim"

Emhass is logging:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
[2023-03-04 11:33:32,938] INFO in web_server: Launching the emhass webserver at: http://<IP_ADDRESS>:5000
[2023-03-04 11:33:32,939] INFO in web_server: Home Assistant data fetch will be performed using url: http://supervisor/core/api
[2023-03-04 11:33:32,939] INFO in web_server: The data path is: /share
[2023-03-04 11:33:32,944] INFO in web_server: Using core emhass version: 0.3.36
[2023-03-04 11:33:51,192] INFO in command_line: Setting up needed data
[2023-03-04 11:33:51,276] ERROR in utils: ERROR: The passed data is either not a list or the length is not correct, length should be 24
[2023-03-04 11:33:51,276] ERROR in utils: Passed type is <class 'list'> and length is 24
[2023-03-04 11:33:51,277] ERROR in utils: ERROR: The passed data is either not a list or the length is not correct, length should be 24
[2023-03-04 11:33:51,278] ERROR in utils: Passed type is <class 'list'> and length is 24
[2023-03-04 11:33:51,283] INFO in forecast: Retrieving weather forecast data using method = scrapper
[2023-03-04 11:33:56,062] INFO in forecast: Retrieving data from hass for load forecast using method = naive
[2023-03-04 11:33:56,065] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2023-03-04 11:34:00,636] INFO in web_server:  >> Performing dayahead optimization...
[2023-03-04 11:34:00,636] INFO in command_line: Performing day-ahead forecast optimization
[2023-03-04 11:34:00,645] INFO in optimization: Perform optimization for the day-ahead

Hassos is logging:
Logger: homeassistant.components.shell_command
Source: components/shell_command/__init__.py:86
Integration: Shell Command (documentation, issues)
First occurred: 11:34:51 (1 occurrences)
Last logged: 11:34:51

Timed out running command: `curl -i -H "Content-Type: application/json" -X POST -d '{"load_cost_forecast":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},"prod_price_forecast":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim`, after: 60s
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/streams.py", line 501, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/shell_command/__init__.py", line 87, in async_service_handler
    stdout_data, stderr_data = await process.communicate()
  File "/usr/local/lib/python3.10/asyncio/subprocess.py", line 195, in communicate
    stdin, stdout, stderr = await tasks.gather(stdin, stdout, stderr)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/shell_command/__init__.py", line 86, in async_service_handler
    async with async_timeout.timeout(COMMAND_TIMEOUT):
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

For testing I loggged into the main Home Assistant Docker container and tried the curl command in the bash shell following this guide:
Install [community "SSH & Web Terminal" addon](https://github.com/hassio-addons/addon-ssh/blob/main/ssh/DOCS.md) (not the official "Terminal & SSH" addon!) and configure your credentials. Note: the addon will fail to start if you use an insecure password so it's better to configure it with public key authentication.

Disable "Protection mode" in the addon's configuration.

At the shell, log into the main Home Assistant Docker container:

docker exec -it homeassistant bash

I used verbose mode in curl:
bash-5.1# curl -i -v -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim
Note: Unnecessary use of -X or --request, POST is already inferred.
* Could not resolve host: application
* Closing connection 0
curl: (6) Could not resolve host: application
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying <IP_ADDRESS>:5000...
* Connected to localhost (<IP_ADDRESS>) port 5000 (#1)
> POST /action/dayahead-optim HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.83.1
> Accept: */*
> Content-Length: 389
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 BAD REQUEST
HTTP/1.1 400 BAD REQUEST
< Content-Length: 167
Content-Length: 167
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Date: Sat, 04 Mar 2023 10:47:57 GMT
Date: Sat, 04 Mar 2023 10:47:57 GMT
< Server: waitress
Server: waitress

<
<!doctype html>

<title>400 Bad Request</title>

Bad Request

The browser (or proxy) sent a request that this server could not understand.

* Connection #1 to host localhost left intact
bash-5.1#

Emhass server did not reply to the curl command:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
[2023-03-04 11:46:23,713] INFO in web_server: Launching the emhass webserver at: http://<IP_ADDRESS>:5000
[2023-03-04 11:46:23,714] INFO in web_server: Home Assistant data fetch will be performed using url: http://supervisor/core/api
[2023-03-04 11:46:23,715] INFO in web_server: The data path is: /share
[2023-03-04 11:46:23,720] INFO in web_server: Using core emhass version: 0.3.36

So that error means that your list is not long enough.
Check this post from an iser that effectively implemented this: https://community.home-assistant.io/t/emhass-an-energy-management-for-home-assistant/338126/113?u=davidusb

Nordpool has updated the today and tomorrow prices now. I tried to send the two list values:
bash-5.1# curl -v -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":[2.21, 2.26, 2.28, 2.34, 2.44, 2.48, 2.47, 2.45, 2.43, 2.4, 2.37, 2.35, 2.33, 2.32, 2.34, 2.32, 2.33, 2.35, 2.36, 2.39, 2.39, 2.35, 2.38, 2.38],\"prod_price_forecast\":[2.21, 2.26, 2.28, 2.34, 2.44, 2.48, 2.47, 2.45, 2.43, 2.4, 2.37, 2.35, 2.33, 2.32, 2.34, 2.32, 2.33, 2.35, 2.36, 2.39, 2.39, 2.35, 2.38, 2.38]}' http://localhost:5000/action/dayahead-optim
Note: Unnecessary use of -X or --request, POST is already inferred.
* Could not resolve host: application
* Closing connection 0
curl: (6) Could not resolve host: application
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying <IP_ADDRESS>:5000...
* Connected to localhost (<IP_ADDRESS>) port 5000 (#1)
> POST /action/dayahead-optim HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.83.1
> Accept: */*
> Content-Length: 336
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 BAD REQUEST
HTTP/1.1 400 BAD REQUEST
< Content-Length: 167
Content-Length: 167
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Date: Sat, 04 Mar 2023 12:26:14 GMT
Date: Sat, 04 Mar 2023 12:26:14 GMT
< Server: waitress
Server: waitress

<
<!doctype html>

<title>400 Bad Request</title>

Bad Request

The browser (or proxy) sent a request that this server could not understand.

* Connection #1 to host localhost left intact
bash-5.1#

Emhass server logs:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
[2023-03-04 13:25:51,058] INFO in web_server: Launching the emhass webserver at: http://<IP_ADDRESS>:5000
[2023-03-04 13:25:51,058] INFO in web_server: Home Assistant data fetch will be performed using url: http://supervisor/core/api
[2023-03-04 13:25:51,059] INFO in web_server: The data path is: /share
[2023-03-04 13:25:51,063] INFO in web_server: Using core emhass version: 0.3.36

And refresh the add-on logs

Launching the shell_command from Services:

shell_command:
  publish_data: "curl -i -H 'Content-Type:application/json' -X POST -d '{}' http://localhost:5000/action/publish-data"

  trigger_nordpool_forecast: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim"

EMhass log:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
[2023-03-04 14:20:33,780] INFO in web_server: Launching the emhass webserver at: http://<IP_ADDRESS>:5000
[2023-03-04 14:20:33,780] INFO in web_server: Home Assistant data fetch will be performed using url: http://supervisor/core/api
[2023-03-04 14:20:33,781] INFO in web_server: The data path is: /share
[2023-03-04 14:20:33,786] INFO in web_server: Using core emhass version: 0.3.36
[2023-03-04 14:20:49,070] INFO in command_line: Setting up needed data
[2023-03-04 14:20:49,169] INFO in forecast: Retrieving weather forecast data using method = scrapper
[2023-03-04 14:20:53,726] INFO in forecast: Retrieving data from hass for load forecast using method = naive
[2023-03-04 14:20:53,728] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2023-03-04 14:20:56,769] INFO in web_server:  >> Performing dayahead optimization...
[2023-03-04 14:20:56,769] INFO in command_line: Performing day-ahead forecast optimization
[2023-03-04 14:20:56,813] INFO in optimization: Perform optimization for the day-ahead

Hassos log:
Logger: homeassistant.components.shell_command
Source: components/shell_command/__init__.py:86
Integration: Shell Command (documentation, issues)
First occurred: 14:21:49 (1 occurrences)
Last logged: 14:21:49

Timed out running command: `curl -i -H "Content-Type: application/json" -X POST -d '{"load_cost_forecast":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},"prod_price_forecast":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim`, after: 60s
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/streams.py", line 501, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/shell_command/__init__.py", line 87, in async_service_handler
    stdout_data, stderr_data = await process.communicate()
  File "/usr/local/lib/python3.10/asyncio/subprocess.py", line 195, in communicate
    stdin, stdout, stderr = await tasks.gather(stdin, stdout, stderr)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/shell_command/__init__.py", line 86, in async_service_handler
    async with async_timeout.timeout(COMMAND_TIMEOUT):
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

Ok thanks, so your're timing out!
From the logs we cannot see any errors from inside EMHASS. It is just that it doesn't seem to be able to finish the optimization.
From your config I see that you are using the GPK solver, why is that?
May be that's the problem. Try to use the default CBC solver.
May be that your optimization problem is just to big.
How many deferrable loads are you trying to optimize?
Do you have also a battery?

I do not have battery, but I have solarpanels. Today I have 8 deferrable loads (6 heating cabel, 1 heat pump, 1 water heater).
My hardware is a Home Assistant Yellow with Raspberry Pi CM4 module. In the emhass documentation you have written this:

Troubleshooting
Some problems may arise from solver related issues in the Pulp package. It was found that for arm64 architectures (ie. Raspberry Pi4, 64 bits) the default solver is not avaliable. A workaround is to use another solver. The glpk solver is an option.

So I changed to GLPK_CMD and path None to try to solve the problem.
I have now changed to COIN_CMD and path: /usr/bin/cbc

and the forecast data are passed to emhass, but I get "timing out" i the hassos log. Tt doesn't seem to be able to finish the optimization.
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
[2023-03-04 14:59:19,438] INFO in web_server: Launching the emhass webserver at: http://<IP_ADDRESS>:5000
[2023-03-04 14:59:19,439] INFO in web_server: Home Assistant data fetch will be performed using url: http://supervisor/core/api
[2023-03-04 14:59:19,441] INFO in web_server: The data path is: /share
[2023-03-04 14:59:19,445] INFO in web_server: Using core emhass version: 0.3.36
[2023-03-04 14:59:41,275] INFO in command_line: Setting up needed data
[2023-03-04 14:59:41,371] INFO in forecast: Retrieving weather forecast data using method = scrapper
[2023-03-04 14:59:44,864] INFO in forecast: Retrieving data from hass for load forecast using method = naive
[2023-03-04 14:59:44,866] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2023-03-04 14:59:48,161] INFO in web_server:  >> Performing dayahead optimization...
[2023-03-04 14:59:48,162] INFO in command_line: Performing day-ahead forecast optimization
[2023-03-04 14:59:48,198] INFO in optimization: Perform optimization for the day-ahead
[2023-03-04 15:00:35,074] INFO in web_server: EMHASS server online, serving index.html...
[2023-03-04 15:00:39,015] INFO in web_server: EMHASS server online, serving index.html...
[2023-03-04 15:00:49,804] INFO in optimization: Status: Optimal
[2023-03-04 15:00:49,804] INFO in optimization: Total value of the Cost function = -254.56
[2023-03-04 15:01:00,428] INFO in web_server: EMHASS server online, serving index.html...
[2023-03-04 15:01:19,953] INFO in web_server: EMHASS server online, serving index.html...

Logger: homeassistant.components.shell_command
Source: components/shell_command/__init__.py:86
Integration: Shell Command (documentation, issues)
First occurred: 15:00:41 (1 occurrences)
Last logged: 15:00:41

Timed out running command: `curl -i -H "Content-Type: application/json" -X POST -d '{"load_cost_forecast":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},"prod_price_forecast":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim`, after: 60s
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/streams.py", line 501, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/shell_command/__init__.py", line 87, in async_service_handler
    stdout_data, stderr_data = await process.communicate()
  File "/usr/local/lib/python3.10/asyncio/subprocess.py", line 195, in communicate
    stdin, stdout, stderr = await tasks.gather(stdin, stdout, stderr)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/shell_command/__init__.py", line 86, in async_service_handler
    async with async_timeout.timeout(COMMAND_TIMEOUT):
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

I forgot to mention it, but my HA Yellow have only 2GB with RAM. Maybe Pulp need more RAM when doing the optimization?

Ok so look no further. Your optimization problem is just too big. The Linear Programming has just too many equations to solve for 8 deferrable loads and that type of solver.
You have two options now:
Option #1: you can try to get the CBC solver to work with PULP_CBC_CMD or...
Option #2: reduce your problem size by either reducing the number of deferrable loads or by reducing the time period of your optimization. To reduce the optimization period you can switch to MPC instead of dayahead optimization. Look to the docs examples in order to help you set MPC. You will just need to pass some additional params, most notably the prediction horizon, fix to something like 6h and launch the MPC every 6 hours. This way you will reduce the optimization problem size.

Thanks for your help David. I am going to try the options.
If I upgrade my HA Yellow to 8 GB RAM or change to a Intel NUC will it help ?

The RAM may help but not guaranteed. The solver could really help, the CBC is way more efficient.
If you have already set the dayahead optimization, the MPC is really not that difficult to set. You just need to change a little bit your shell command.

Thank you for your help. I going to try the MPC controller.

Using the MPC controller with the prediction horizon fixed to 6 hour and a automation which do the optimization every 6 hour fixed the problem when the optimization is timing out:
trigger_nordpool_mpc: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}, \"prediction_horizon\":6, \"def_total_hours\":[5,8,8,8,8,8,8,8]}' http://localhost:5000/action/naive-mpc-optim"

Great, just be careful to maybe re-evaluate the def_total_hours in this case for MPC those will be the total number of hours inside the prediction horizon window, in your case 6h.

Here are the updated shell commands with total number of hours inside the prediction horizon window, in my case 6h:
Waterheater: 2 hours
Floorheating: 3 hours
"def_total_hours":[2,3,3,3,3,3,3,3]
shell_command:
  publish_data: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/publish-data"

  trigger_nordpool_forecast: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim"

  trigger_nordpool_mpc: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}, \"prediction_horizon\":6, \"def_total_hours\":[2,3,3,3,3,3,3,3]}' http://localhost:5000/action/naive-mpc-optim"

  trigger_entsoe_mpc: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.entsoe_average_electricity_price_today', 'prices_today') | map(attribute='price') | list + state_attr('sensor.entsoe_average_electricity_price_today', 'prices_tomorrow') | map(attribute='price') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.entsoe_average_electricity_price_today', 'prices_today') | map(attribute='price') | list  + state_attr('sensor.entsoe_average_electricity_price_today', 'prices_tomorrow') | map(attribute='price') | list))[now().hour:][:24]}}, \"prediction_horizon\":6, \"def_total_hours\":[2,3,3,3,3,3,3,3]}' http://localhost:5000/action/naive-mpc-optim"

👍


No network after uninstall of Tailscale Addon

Problem/Motivation
I installed the tailscale addon and HA node would come online for a few seconds and disappear. I was not able to access home assistant via my tailscale network. I uninstalled the addon and now My HA VM has no local networking. I cannot access my home assistant web ui.
Home Assistant OS v 8.2024
Proxmox v8
Expected behavior
Uninstalling the addon should revert network settings to what they were prior to the addon being installed.
Actual behavior
Home Assistant VM has no network.

I reviewed all network settings on proxmox
Other VMs and LXCs are working
Tried changing dns server from ha cli
Tried restarting NetworkManager
Tried deleting nmcli connections NetworkManager
Tried adding connection
nmcli devices show enps018 as unmanaged and will not add to managed
tbh I've been working on this for 2 days now and I am at the end of my rope.

Steps to reproduce
Install the tailscale addon in an HAOS Proxmox VM and uninstall it

Turn on an off the DNS option

Fix
After hours of troubleshooting heres the solution for Home Assistant Device unmanaged network device that cannot be switched to managed
Boot home assistant into rescue mode
Start NetworkManager
systemclt start NetworkManager
Set device to Managed
nmcli dev set <deviceName> managed yes
Exit the rescue terminal
exit
Wait for home assistant to start there will be an error
Repair supervisor
ha supervisor repair
start core
ha core start
You should now have home assistant operating system running again

Without logs it's hard to say anything about what happened. You wrote "Turn on an off the DNS option", the DNS option is default "on", you can disable it with adding accept_dns: false to the add-on config. But this seems to be a Tailscale issue, because the add-on doesn't modify the host's networking too much, nearly everything is done by Tailscale.


Custom named entities

Hi @richardpaulhudson
Congratulations with your work on Holmes. It's a very cool approach.
I'm wondering if it's possible to provide custom named entities - e.g. using the SpanRuler (or EntityRuler) component from spaCy.  I'm thinking the answer is no at this point but what would it require - I imagine it requires some training?

I would also like to be able to do this. I am wondering if rather than passing a string representing a model name, one could pass an nlp instance (or factory) directly.

I would also like to be able to do this. I am wondering if rather than passing a string representing a model name, one could pass an nlp instance (or factory) directly.

You can make your own custom models accessible by string representation using entry points. However I'm pretty sure that Holmes require specifically trained model to work.

@adelevie, that will work, although the penultimate line should read
doc.set_ents([Span(doc, 0, 1, "FELINE"), Span(doc, 4, 5, "CANINE")])

Thanks, and good catching my set_ents typo.


RFC: <template> in CallExpression argument position

Vote by reacting to this issue. Comment with more details below. Do you prefer:
Option A (🚀)
This is what v2.0.0-0 is currently returning.
await render(
  <template>

I am a multi-line template

With some text

  </template>,
);

Option B (👎)
This is what v1 currently returns.
await render(<template>

I am a multi-line template

With some text

</template>);

I always format my own stuff as the option a anyway. I think it looks better


Make proxy manager required prop of core/App component

We get the following errors otherwise:
[Vue warn]: Error in nextTick: "TypeError: Cannot read property 'onProxyRegistrationChange' of undefined"

found in

---> <Datasets> at node_modules/paraview-glance/src/components/core/Datasets/index.vue
       <VTabItem>
         <VTabsItems>
           <VTabs>
             <VNavigationDrawer>
               <VApp>
                 <DragAndDrop> at node_modules/paraview-glance/src/components/widgets/DragAndDrop/index.vue
                   <App> at node_modules/paraview-glance/src/components/core/App/index.vue
                     <Glance> at src/views/Glance.vue
                       <App> at src/App.vue
                         <Root>,TypeError: Cannot read property 'onProxyRegistrationChange' of undefined,[Vue warn]: Error in nextTick: "TypeError: Cannot read property 'onModified' of undefined"

found in

---> <GpuInformation> at node_modules/paraview-glance/src/components/widgets/GPUInformation/index.vue
       <GlobalSettings> at node_modules/paraview-glance/src/components/core/GlobalSettings/index.vue
         <VTabItem>
           <VTabsItems>
             <VTabs>
               <VNavigationDrawer>
                 <VApp>
                   <DragAndDrop> at node_modules/paraview-glance/src/components/widgets/DragAndDrop/index.vue
                     <App> at node_modules/paraview-glance/src/components/core/App/index.vue
                       <Glance> at src/views/Glance.vue
                         <App> at src/App.vue
                           <Root>,TypeError: Cannot read property 'onModified' of undefined,[Vue warn]: Error in nextTick: "TypeError: Cannot read property '$on' of undefined"

found in

---> <Notification> at node_modules/paraview-glance/src/components/core/Notification/index.vue
       <VApp>
         <DragAndDrop> at node_modules/paraview-glance/src/components/widgets/DragAndDrop/index.vue
           <App> at node_modules/paraview-glance/src/components/core/App/index.vue
             <Glance> at src/views/Glance.vue
               <App> at src/App.vue
                 <Root>,TypeError: Cannot read property '$on' of undefined,[Vue warn]: Error in nextTick: "TypeError: Cannot read property 'getViews' of undefined"

found in

---> <BrowserIssues> at node_modules/paraview-glance/src/components/core/BrowserIssues/index.vue
       <VApp>
         <DragAndDrop> at node_modules/paraview-glance/src/components/widgets/DragAndDrop/index.vue
           <App> at node_modules/paraview-glance/src/components/core/App/index.vue
             <Glance> at src/views/Glance.vue
               <App> at src/App.vue
                 <Root>,TypeError: Cannot read property 'getViews' of undefined,Script error.

For more information, right now proxyManager and widgetManager are expected to be provided using the provide directive. Unless there's a compelling reason to do it this way, I think we should just pass them as required props, since that is more idiomatic to vue.js.
It might also be nice to have some default values so that the App component could be used without needing to know a lot of internals of vtk.js, e.g. without having to write the following sort of code:
  const proxyConfiguration = proxyConfig || activeProxyConfig || Config.Proxy;

  const proxyManager = vtkProxyManager.newInstance({ proxyConfiguration });
  const renderListener = vtkListenerHelper.newInstance(
    proxyManager.autoAnimateViews,
    () =>
      [].concat(
        proxyManager.getSources(),
        proxyManager.getRepresentations(),
        proxyManager.getViews()
      )
  );

  proxyManager.onProxyRegistrationChange(renderListener.resetListeners);

  const widgetManager = vtkWidgetManager.newInstance({ proxyManager });
  widgetManager.registerWidgetGroup(Widgets.CROP, CropWidget);

Fixed with #129. proxyManager is part of a vuex store now.