next.js ENOENT trying to open schema.prisma

Bug description
In a monorepo, when exporting PrismaClient from one package and consuming it in a next.js api route, I get
error - Error: ENOENT: no such file or directory, open 'C:\Users\sbere\Desktop\Projects\prisma-pnpm-workspace-test\apps\frontend\.next\server\pages\api\schema.prisma'

How to reproduce

Git clone https://github.com/SimonBerens/prisma-pnpm-workspace-test
run

pnpm install
cd packages/prisma
pnpx prisma generate
cd ../../apps/frontend
pnpm run dev

send a GET request to localhost:3000/api/hello

Expected behavior
The api runs as expected
Prisma information
prisma:tryLoadEnv Environment variables not found at null +0ms
  prisma:tryLoadEnv Environment variables not found at undefined +0ms
  prisma:tryLoadEnv No Environment variables loaded +1ms
  prisma:tryLoadEnv Environment variables not found at null +1ms
  prisma:tryLoadEnv Environment variables not found at undefined +0ms
  prisma:tryLoadEnv No Environment variables loaded +0ms
  prisma:client dirname C:\Users\sbere\Desktop\Projects\prisma-pnpm-workspace-test\apps\frontend\.next\server\pages\api +0ms
  prisma:client relativePath ..\..\..\..\..\..\packages\prisma +0ms
  prisma:client cwd C:\Users\sbere\Desktop\Projects\prisma-pnpm-workspace-test\packages\prisma +0ms
  prisma:client clientVersion: 3.12.0 +2ms
  prisma:client clientEngineType: library +0ms

Environment & setup

OS: Windows 11
Database: PostgreSQL
Node.js version: v16.14.2

Prisma Version
prisma                  : 3.12.0
@prisma/client          : 3.12.0
Current platform        : windows
Query Engine (Node-API) : libquery-engine 22b822189f46ef0dc5c5b503368d1bee01213980 (at ..\..\node_modules\.pnpm\@prisma+engines<EMAIL_ADDRESS>Migration Engine        : migration-engine-cli 22b822189f46ef0dc5c5b503368d1bee01213980 (at ..\..\node_modules\.pnpm\@prisma+engines<EMAIL_ADDRESS>Introspection Engine    : introspection-core 22b822189f46ef0dc5c5b503368d1bee01213980 (at ..\..\node_modules\.pnpm\@prisma+engines<EMAIL_ADDRESS>Format Binary           : prisma-fmt 22b822189f46ef0dc5c5b503368d1bee01213980 (at ..\..\node_modules\.pnpm\@prisma+engines<EMAIL_ADDRESS>Default Engines Hash    : 22b822189f46ef0dc5c5b503368d1bee01213980
Studio                  : 0.459.0

I'm actually experiencing the same but it's not limited to nextjs it's everywhere I use prisma.
Yarn workspaces
Custom output path
Prisma output is a private package

I think https://github.com/prisma/prisma/issues/13052 would address this.

This does seem to be a PNPM thing. I can get @SimonBerens's repo running using Yarn3 and nodeLinker: node-modules: https://github.com/orbiteleven/prisma-yarn-workspace-test.

Since this seems to be related to PNPM+Next.js, I've cross-posted this here

Neither is a Next.js thing. I'm experiencing the same issue with Remix 1.4.3

I fixed this adding to .npmrc the following, this forces the prisma package to remain on top of the monorepo i think
public-hoist-pattern[]=*prisma*

public-hoist-pattern[]=prisma

That solves the issue thank you 👍

I fixed this adding to .npmrc the following, this forces the prisma package to remain on top of the monorepo i think
public-hoist-pattern[]=*prisma*

This worked!
Thanks

Currently having the same issue with pnpm workspaces (turborepo) + Next.js. public-hoist-pattern[]=*prisma* did not help :(

Currently getting the same error with pnpm workspaces, turborepo and Next.js. public-hoist-pattern[]=*prisma* did not do the trick :/

Currently getting the same error with pnpm workspaces, turborepo and Next.js. public-hoist-pattern[]=*prisma* did not do the trick :/

Try adding this to your .npmrc, this did the trick for me:
node-linker=hoisted

Can confirm, this is still an issue.

Currently getting the same error with pnpm workspaces, turborepo and Next.js. public-hoist-pattern[]=*prisma* did not do the trick :/

Try adding this to your .npmrc, this did the trick for me:
node-linker=hoisted

Also tried, still not helping 😢

I fixed this adding to .npmrc the following, this forces the prisma package to remain on top of the monorepo i think
public-hoist-pattern[]=*prisma*

This worked for us since July and inexplicably stopped working for us this week.

This worked for us since July and inexplicably stopped working for us this week.

Ah... just fixed it that way, hope it will keep working for us. 😅
If you're using Next.js or some other framework bundling via Webpack you can try adding @prisma/client to config.externals. I tried that once and it worked, because it keeps Prisma out of the bundle (it's used server-side only anyways). Prisma is probably just required via Node resolution out of node_modules, so the schema file stays next to it.

prisma 4.2.0
@prisma/client 4.2.0
<EMAIL_ADDRESS>M1 mac
I think I'm running into the same thing. I have 2 "packages" in turborepo that use prisma (mongodb, postgres).
I've tried using a custom "output" path for both packages so that the schemas wouldn't overwrite each other in the shared @prisma monorepo node_modules. It seemed to work ok until I saw this error.
I have pivoted and now I'm trying to use the default behavior of importing from @prisma/client but not hoisting prisma deps to the monorepo node_modules a la
    "nohoist": [
      "**/prisma/**",
      "**/@prisma/client/**",
      "**/@prisma/**",
      "**/@prisma/engines/**"
    ]

but still seeing this same error
ditto:dev: error - Error: ENOENT: no such file or directory, open '/Users/trenton/Desktop/dev/polaris/apps/ditto/.next/server/pages/schema.prisma'
ditto:dev:     at Object.openSync (fs.js:498:3)
ditto:dev:     at Object.readFileSync (fs.js:394:35)
ditto:dev:     at new LibraryEngine (webpack-internal:///../../packages/blueprint/node_modules/@prisma/client/runtime/index.js:25308:41)
ditto:dev:     at PrismaClient.getEngine (webpack-internal:///../../packages/blueprint/node_modules/@prisma/client/runtime/index.js:29321:16)
ditto:dev:     at new PrismaClient (webpack-internal:///../../packages/blueprint/node_modules/@prisma/client/runtime/index.js:29293:29)

Is it a problem with LibraryEngine.datamodel? Just looking at the sourcecode it seems like dirname on this line might be going to the wrong place.
datamodelPath: path.join(config.dirname, config.filename ?? 'schema.prisma')

Our stack is pretty specific, but here is what worked for us on pnpm v7 / turborepo v1.4.7 / vercel / prisma 4.4.0
Several of our builds were failing with this Prisma-related error: error - Error: ENOENT: no such file or directory, open '/Users/jbranchaud/code/clients/egghead/products/apps/epic-web/.next/server/pages/api/skill/schema.prisma'
This was originally fixed for us in July using public-hoist-pattern[]=*prisma* in .npmrc as @joelhooks pointed out:

This worked for us since July and inexplicably stopped working for us this week.

Because it suddenly stopped working, I started look for what could have changed most recently. pnpm v7.13.5 was released just a couple days ago. That seemed worth exploring, especially since our codebase isn't controlling the pnpm version. The theory being that our builds would pass on an earlier version of pnpm, e.g. v7.13.2.
What we learned is that, as of right now, the only way to control the specific version of pnpm that Vercel uses is to enable an experimental corepack feature.
With ENABLE_EXPERIMENTAL_COREPACK set to 1 as a Vercel environment variable, we noticed that our builds started respecting the packageManager config in the monorepo's root package.json.
So we set it to:
{
  ...,
  "packageManager"<EMAIL_ADDRESS>}

With those two things in place, we started seeing Vercel pin pnpm to that specific version and the above error went away again.

PR with more details: https://github.com/skillrecordings/products/pull/535

Someone was able to make it work ?
Same issue with:
pnpm: 7.14.0
NextJS
Prisma 4.5.0

This worked for us since July and inexplicably stopped working for us this week.

Ah... just fixed it that way, hope it will keep working for us. 😅
If you're using Next.js or some other framework bundling via Webpack you can try adding @prisma/client to config.externals. I tried that once and it worked, because it keeps Prisma out of the bundle (it's used server-side only anyways). Prisma is probably just required via Node resolution out of node_modules, so the schema file stays next to it.

This works for me.

output into project root node_modules

generator client {
  provider = "prisma-client-js"
  output   = "../../../node_modules/.prisma/client"
}

Add @prisma/client to externals in next.config.js

 defineNextConfig({
    webpack: config => {
      config.externals = [...(config.externals || []), '@prisma/client'];
      // Important: return the modified config
      return config;
    }
  })

@Pong420 this worked for me. Thank you so, so much!

If you're using nextjs, try adding prisma to the serverComponentsExternalPackages in your config:
// next.config.js
const nextConfig = {
  experimental: {
    appDir: true,
    serverComponentsExternalPackages: ['@prisma/client'],
  },
};

https://beta.nextjs.org/docs/api-reference/next.config.js#servercomponentsexternalpackages

Still having this issue with
yarn version 1.22,
prisma 4.8.1
setting the webpack config.externals works on dev, but on builds nextjs is complaining
Unexpected character '@' (1:35)
--
 \| var __WEBPACK_NAMESPACE_OBJECT__ = @prisma/client;
while analyzing module external assign "@prisma/client" for concatenation

this all started for me after upgrading prisma from version 3 to 4, any help appreciated!

Thanks for the excellent reproduction @SimonBerens - it was super easy to reproduce the problem with it. We are testing a workaround, so we will hopefully have good news soon 🔜 .

Meanwhile:
@nvsd Can you maybe share a reproduction of a project not using Next.js where this is happening? Maybe even as a new issue as this is different enough for us to look at it separately. Thanks.
@wladiston Can you maybe share a reproduction of this problem with Remix? Maybe even as a new issue as this is different enough for us to look at it separately. Thanks.
Thanks everyone else who provided all their information about their projects. If you want, any additional reproduction can help us refine and test our potential solutions. Thanks.
@orbiteleven @remorses @baptisteArno @joelhooks Can you please describe your projects a bit? Monorepo? Next.js? Custom output or not? Package Manager? Thanks.

I have a workaround working for @SimonBerens's repro
https://github.com/SimonBerens/prisma-pnpm-workspace-test/pull/1

@orbiteleven @remorses @baptisteArno @joelhooks Can you please describe your projects a bit? Monorepo? Next.js? Custom output or not? Package Manager? Thanks.

https://github.com/baptisteArno/typebot.io. It's a monorepo powered by Turborepo. It uses pnpm and it's a Next.js app.

The best solution would be to embed the schema.prisma data in the generated code instead of reading the file from file system

@baptisteArno Would you be able to give us a reproduction? (Btw, amazing work on Typebot!)

@remorses Then the schema would load successfully, and it would fail for the Query Engine which is needed to run queries and can not be in the generated code. Unfortunately not a solution.

@baptisteArno Would you be able to give us a reproduction? (Btw, amazing work on Typebot!)

Thank you 🙏
Here is the most minimal reproduction I could get: https://github.com/baptisteArno/prisma-turbo-pnpm

@Weakky you have to fix that buddy.

@baptisteArno Thanksfully the same workaround as for @SimonBerens also works for your minimal reproduction: https://github.com/baptisteArno/prisma-turbo-pnpm/pull/1/files Thanks for creating that, so we could confirm our workaround and will have another project later to confirm the proper fix!

I made the change on Typebot, works perfectly 👌

Whoops guys, my bad, it worked because I had a build cache. It didn't work when I triggered a rebuild of my app in production.
It crashed with the following error: Cannot find module '@prisma/client'

Can you shared the branch of that? Happy to look into it to see how to adapt our workarounds and understanding what is going on.

So this fix won't work if deploying the application to Vercel because they will ignore
output: "standalone",
experimental: {
    outputFileTracingRoot: path.join(__dirname, "../../"),
  },

Because Vercel is deploying serverless functions.
I reproduced this on: https://github.com/baptisteArno/prisma-turbo-pnpm
The deployment is here: https://prisma-turbo-pnpm-web.vercel.app (check the console to see the api response failing)
There is maybe a way to detect in the webpack function whether we're building for serverless or standalone? 🤔

Hey guys, thanks for the feedback. We have been working on a simpler way to make things work for you, as the workaround we proposed had clear limitations for TS projects. We worked on an experimental webpack plugin that ensures your Prisma files are copied correctly. I'd love if you could give it a try (https://www.npmjs.com/package/experimental-prisma-webpack-plugin). We have tested it in many different scenarios, so we hope this works for you.
const { PrismaPlugin } = require('experimental-prisma-webpack-plugin')

module.exports = {
  output: 'standalone',
  webpack: (config, { isServer }) => {
    if (isServer) {
      config.plugins = [...config.plugins, new PrismaPlugin()]
    }

    return config
  },
}

@millsp It works for a ton of scenarios I tried. Thank you for this plugin 👏
However, it crashes in dev mode when you import multiple prisma clients. Only when it has compiled at least 1 folder deep in the api folder:
web:dev: error: Error validating: This line is invalid. It does not start with any known Prisma schema keyword.
web:dev:   -->  schema.prisma:15
web:dev:    |
web:dev: 14 | }
web:dev: 15 |  String
web:dev: 16 | }
web:dev:    |
web:dev: error: Error validating: This line is invalid. It does not start with any known Prisma schema keyword.
web:dev:   -->  schema.prisma:16
web:dev:    |
web:dev: 15 |  String
web:dev: 16 | }
web:dev: 17 |
web:dev:    |
web:dev:
web:dev: Validation Error Count: 2
web:dev:     at RequestHandler.handleRequestError (webpack-internal:///(api)/../../packages/db/prisma-clients/postgresql/runtime/index.js:32859:19)
web:dev:     at RequestHandler.handleAndLogRequestError (webpack-internal:///(api)/../../packages/db/prisma-clients/postgresql/runtime/index.js:32808:18)
web:dev:     at eval (webpack-internal:///(api)/../../packages/db/prisma-clients/postgresql/runtime/index.js:33351:35)
web:dev:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
web:dev:     at async PrismaClient._executeRequest (webpack-internal:///(api)/../../packages/db/prisma-clients/postgresql/runtime/index.js:33931:30)
web:dev:     at async PrismaClient._request (webpack-internal:///(api)/../../packages/db/prisma-clients/postgresql/runtime/index.js:33917:24)
web:dev:     at async handler (webpack-internal:///(api)/./pages/api/posts.ts:8:5)
web:dev:     at async Object.apiResolver (/Users/baptistearno/Dev/prisma-turbo-pnpm/node_modules/.pnpm/next@13.1.6_pjwopsidmaokadturxaafygjp4/node_modules/next/dist/server/api-utils/node.js:372:9)
web:dev:     at async DevServer.runApi (/Users/baptistearno/Dev/prisma-turbo-pnpm/node_modules/.pnpm/next@13.1.6_pjwopsidmaokadturxaafygjp4/node_modules/next/dist/server/next-server.js:488:9)
web:dev:     at async Object.fn (/Users/baptistearno/Dev/prisma-turbo-pnpm/node_modules/.pnpm/next@13.1.6_pjwopsidmaokadturxaafygjp4/node_modules/next/dist/server/next-server.js:751:37)
web:dev:     at async Router.execute (/Users/baptistearno/Dev/prisma-turbo-pnpm/node_modules/.pnpm/next@13.1.6_pjwopsidmaokadturxaafygjp4/node_modules/next/dist/server/router.js:253:36)
web:dev:     at async DevServer.run (/Users/baptistearno/Dev/prisma-turbo-pnpm/node_modules/.pnpm/next@13.1.6_pjwopsidmaokadturxaafygjp4/node_modules/next/dist/server/base-server.js:384:29)
web:dev:     at async DevServer.run (/Users/baptistearno/Dev/prisma-turbo-pnpm/node_modules/.pnpm/next@13.1.6_pjwopsidmaokadturxaafygjp4/node_modules/next/dist/server/dev/next-dev-server.js:743:20)
web:dev:     at async DevServer.handleRequest (/Users/baptistearno/Dev/prisma-turbo-pnpm/node_modules/.pnpm/next@13.1.6_pjwopsidmaokadturxaafygjp4/node_modules/next/dist/server/base-server.js:322:20) {
web:dev:   clientVersion: '4.9.0',
web:dev:   errorCode: undefined,
web:dev:   page: '/api/another/hi'
web:dev: }

I managed to reproduce a minimalist issue here: https://github.com/baptisteArno/prisma-turbo-pnpm/pull/2
Can you think of a solution for this?

I'd love if you could give it a try (https://www.npmjs.com/package/experimental-prisma-webpack-plugin). We have tested it in many different scenarios, so we hope this works for you.

Yes! Thanks for this, it solved our problem, we are using yarn, and turbo, with a database package for Prisma. 🍻

I was testing out the plugin for my use case and ran into an issue. I tried to make an accurate reproduction of it here
It's a NextJS app. I think the issue is when I use prisma db functions in both an /api file and in a getStaticProps (maybe getServerSideProps too). I also tried to add a little more context in the Readme.

Thanks @trentonsnyder. I haven't looked in detail yet, so quick question for you: Have you tried removing if (isServer) { so that it is triggered at static build time too? I'm not sure if that is your problem 100% but maybe worth trying.

@millsp good suggestion. Seems to behave the same way though even with isServer removed.

Thanks, I'll take a look in the coming days

For those using multiple Prisma Clients, we are going to schedule follow-up work via https://github.com/prisma/prisma/issues/18069. Please upvote it if you need multi-client support, and don't hesitate to share more information there.

It's unfortunately super flaky and happens even with the patch – I suspect when another package is importing our database package and both are being transpiled into Next.

Prisma 4.14.0
pnpm
Next 13.4.2`
Turborepo

@janpio - I'm having a very similar usecase but I cannot provide a reproduction right now. But I would like to share in case it sparks some ideas.
I'm using Next 13 with the app dir, the monorepo workaround works for almost every usecase but it seems it hits some race conditions at times.
When going to a route the "compiling" messages shows and as soon as it's ready the page loads and throws the error. I can see that during compilation the .next/server/app/<example> get's created and before the prisma files get copied the app tries to load and fails.
Most times a refresh solves the issue and I can see the prisma files being copied into the newly created .next folder, where some other times it wont.
This happens a lot more consistently when doing client side navigation to a route which is not compiled yet, and needs to be compiled on the fly.
I hope this information is useful, if there is a better format/place to add this I can attempt to do so.

Thanks @maxtechera, you can just open a new issue even with this limited information. The bug issues template asks for a few interesting bits that could help us understand this - and if you ever figure out more you will have a place to add that information. Thanks!

Thanks @maxtechera, you can just open a new issue even with this limited information. The bug issues template asks for a few interesting bits that could help us understand this - and if you ever figure out more you will have a place to add that information. Thanks!

Here's the issue: https://github.com/prisma/prisma/issues/19784


Fix rest endpoint documentation to indicate that userId in PKI mode should be DN

PR: https://github.com/NationalSecurityAgency/skills-docs/pull/279


N/A

N/A

@Michaelvirga ah, good call. My guess is that it wouldn't take too much code change to get this to work (I would hope!). Animation support should be coming out in the next day or two, and I'll see if I can also make it work in the background.

Here is a beta release with animation support. It should work well. Please let me know if you find any bugs!
The last thing I'm working on (that isn't in this build) is support for animated prompt text!
AI-Render-v0-5-1.zip

@benrugg I am getting an error post intallation when I try to enable the addon - AI-Render.properties has no attribute ensure_sampler.  I can then enable it after if i click it again, but when I go to render it does not call the SD API.  In the compositing tab it looks like the AI-Render step is greyed out too


Changing search text value dynamically shows correct text  in search option but when selecting the option its taking the old option text.This is of type select2 remote data source

its in Select2 4.0.0 version.

Search “123”
Backend returns “{id:’123’,value:”this is 123”}”;
Anything else in the app (not in the search page) updates the db
Now search “123”
It shows “123: ‘this is new 123’”
But when we click on this option
The text that appears in select2 remains “this is 123”

This is because select2 adds s to the select and if something which matches the an old selected value it’s simply shown with old optionText instead of creating a new  everytime a search-result is selected.

i came across this issue today in my project, a very confusing issue. i can reproduce it with these 7 steps.


Undid Polish translation for tl;dr

In accordance with Untranslated words and expressions section.

Terrific, thanks. :)


Can't build unstable.kicad

Describe the bug
Was looking to upgrade to KiCAD 6, but get the below error related to wxWidgets when trying to build KiCAD from unstable
Steps To Reproduce
Run nix develop with this flake.
Expected behavior
Working KiCAD build.
Screenshots
-- Found Phoenix 4.1.1/gtk3 (wxWidgets 3.1.5)
CMake Error at /nix/store/sgsvy949kd2sr2vsxcffz5jkv855rmib-cmake-3.22.2/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find wxWidgets: Found unsuitable version "3.1.4", but required is
  at least "3.1.5" (found
  -L/nix/store/pcc4pldpa2xwqh65gazrlvv2z5y9xmbs-wxwidgets-3.1.4/lib;-pthread;;;-L/nix/store/5lzlfgrc37wqj400b4qgpaa6kc9d2wr1-libglvnd-1.4.0/lib;-L/nix/store/04yj0nllw5bqzzlay5m6zzbmyw7csgnk-g>
Call Stack (most recent call first):
  /nix/store/sgsvy949kd2sr2vsxcffz5jkv855rmib-cmake-3.22.2/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:592 (_FPHSA_FAILURE_MESSAGE)
  CMakeModules/FindwxWidgets.cmake:1019 (find_package_handle_standard_args)
  CMakeLists.txt:876 (find_package)

Notify maintainers
@evils @Kiwi

wxWidgets was updated in https://github.com/NixOS/nixpkgs/pull/157848
its currently in staging-next so you may either use that branch or overlay the package
https://nixos.wiki/wiki/Nixpkgs/Reviewing_changes#Pre-flakes

thanks, building out of staging-next seems to be working


I'm curious

I'm curious to know what's going to happen here 😜

@leandrodamascena this is basically it - just a simple program that prints the list of resource types in a CFN template :-)


HipChat cmd-c not working

I've found that cmd-c is not working to copy content in HipChat content. Right click and then selecting Copy works, but cmd-c is not.

Hello, I'm MacOS High Sierra (10.13).
This bug appears on all my services (HipChat, FB Messenger, WhatsApp).
I think, it's not service relevant, it will be something global for webview and event handlers.
I'll try to find solution and send PR today.

Duplicate of #33


Amaro Filter

Hey, thanks for the PR. This feels a little saturated to me, try adding a layer and applying color through it maybe. Right now it seems like there's a layer being screened without providing anything to screen.

Closing due to inactivity and failing checks. I'd love to see another PR open!


NCBI Datasets: switch to "host-name" field

Description of proposed changes
Previously used the "host-common-name" field to pull in the common names of hosts (e.g. "human"), however this field is empty in recent downloads. This commit switches to the other host field, "host-name", which has the scientific name of the host.
Related issue(s)

Resolves #383
Testing

[x] Checks pass
[x] Test run (GH Action - AWS Batch job 89cd19a7-4c18-4a2e-9b29-d9590adcec62)

Compared first 1k lines of old and new and indeed, the only change is homo sapiens is added where there was nothing `` before


Update index.md

Addition of table of content

Description
Addition of table of content for easy access.
Contribution Guidelines
Please Read through the Contribution Guidelines.
Fixed Issues
Please mention any issues fixed in the PR by referencing it properly in the commit message.
As per the convention, use appropriate keywords such as fixes, closes, resolves to automatically refer the issue.
Please consult official github documentation for details.
Fixes #(issue)
"BEFORE" and "AFTER" output
To verify that the change works as desired, please include an output of terminal before and after the changes under headings "BEFORE" and "AFTER".
BEFORE
Any Terminal Output Before Changes.
AFTER
Any Terminal Output Before Changes.
Contribution checklist

[x] I have read the Contributing Guidelines.
[x] The commits refer to an active issue in the repository.
[x] I have added automated testing to cover this case.

Notes
Please mention if you have not checked any of the above boxes.

Hi @rishabh710
We already have a table of contents on our main README.md


Add swarupmohalik to the team

Can't assign him to do a PR review and his login does not show in the autocomplete.

He still hasn't accepted my invite.

@swarupmohalik we are waiting for your action.

Please reopen @swarupmohalik if there any problems.


Unable to see OpenCR board as /dev/ttyACM0

ISSUE TEMPLATE ver. 0.2.0

Which TurtleBot3 you have?

[ ] Burger
[ x] Waffle
[ ] Waffle Pi

Which SBC(Single Board Computer) is working on TurtleBot3?

[ ] Raspberry Pi 3
[ x] Intel Joule 570x
[ ] etc (PLEASE, WRITE DOWN YOUR SBC HERE)

Which OS you installed in SBC?

[ x] Ubuntu MATE 16.04 or later
[ ] Raspbian
[ ] etc (PLEASE, WRITE DOWN YOUR OS)

Which OS you installed in Remote PC?

[ x] Ubuntu 16.04 LTS (Xenial Xerus)
[ ] Ubuntu 18.04 LTS (Bionic Beaver)
[ ] Linux Mint 18.x
[ ] etc (PLEASE, WRITE DOWN YOUR OS)

Write down software version and firmware version

Software version: [x.x.x]
Firmware version: [x.x.x]

Write down the commands you used in order

HERE

Copy and Paste your error message on terminal

HERE

Please, describe detailedly what difficulty you are in

I am unable to update the OpenCR firmware following the instructions here:
http://emanual.robotis.com/docs/en/platform/turtlebot3/applications/#ros2
The /dev/ttyACM0 device is no longer showing up. I have tried to go into FIrmware Recover Mode using these instructions, and I see the LED blinking but that doesn't make the device show up.
http://emanual.robotis.com/docs/en/parts/controller/opencr10/#update-bootloaderlinux
I also tried booting into the bootloader mode, which makes it so I can see the device using lsusb, but still can't connect to it.  I'm not sure if I need to burn a new bootloader?
Please advise.

Hi, @mkhansen-intel
Please make sure that OpenCR is connected to another port. (eg. /dev/ttyACM1 ...)
The description of the manual assumes that the port of OpenCR is /dev/ttyACM0.

I'm not sure if I need to burn a new bootloader?

Since we have not updated the boot loader, you do not need to update the boot loader separately.


[FLINK-6195] [build] Move gelly-examples jar from opt to examples

The opt directory should be reserved for Flink JARs which users may optionally move to lib to be loaded by the runtime. flink-gelly-examples is a user program so is being moved to the examples folder.

Changes to the build and docs look good. I would propose moving the changes to the EdgeList/AsmTestBase into a separate commit though.


Initialize observability feature missing in the go sdk.

We will require trace propagation in the providers built using the go sdk.
We will need to replicate this code in the go sdk.

OTEL support has been added & example has been updated in #26


Typo in ftd2xx.py

Line 10
shouln't the line say
if sys.platform == 'linux' instead of 'linux2' ?
Because on my System (CentOS 6 and Python 3.5) sys.platform returns 'linux' and not 'linux2'

Never mind. It's updated on the most recent version https://github.com/ctrl-shift-esc/ftd2xx/blob/master/ftd2xx/ftd2xx.py


[lexical] Fix flow error: change this to any

Seems that flow isnt able to compile this type as an input param type: see sample
error
This results in downstream errors such as this scenerio:
export class QuoteLineNode extends TextNode {

  updateDOM(
    prevNode: TextNode,
    dom: HTMLElement,
    config: EditorConfig,
  ): boolean {
    // errors here
    const shouldUpdate = super.updateDOM(prevNode, dom, config);
    return shouldUpdate;
  }
}

In this example, super is a TextNode, and prevNode, being this typed, should be a TextNode, but flow isnt able to determine that. The error:
Cannot call super.updateDOM with prevNode bound to prevNode because TextNode [1]
is incompatible with this [2]. [incompatible-call]

In this PR, changing this to any as a quick fix for the flow apis. Will look into adding stricter typing for each node's updateDom in future PRs, refer to https://github.com/facebook/lexical/pull/6894/files for possible places
I also tried changing the flow api to updateDOM(prevNode: LexicalNode, ... but that resulted in errors everywhere as flow isnt able to eg. tell that a ParagraphNode is a LexicalNode, strangely.

In this case the downstream code's type annotation is wrong, up to you whether you put FlowFixMe here to work around the wrong annotation in downstream code or not. It doesn't affect any OSS users because almost nobody uses flow outside of Meta.
export class QuoteLineNode extends TextNode {

  updateDOM(
    prevNode: TextNode,
    dom: HTMLElement,
    config: EditorConfig,
  ): boolean {
    // errors here
    const shouldUpdate = super.updateDOM(prevNode, dom, config);
    return shouldUpdate;
  }
}

Should be:
export class QuoteLineNode extends TextNode {

  updateDOM(
    prevNode: this,
    dom: HTMLElement,
    config: EditorConfig,
  ): boolean {
    // errors here
    const shouldUpdate = super.updateDOM(prevNode, dom, config);
    return shouldUpdate;
  }
}

Specifically updateDOM is always called with another instance of the exact same class (prevNode.constructor === this.constructor). The TextNode annotation in the downstream code only worked because there wasn't any code that needed to inspect anything that isn't in the base class other than calling the super implementation, so it passed the type checker but it was never really correct.

thanks for looking into this, changing to updateDOM(prevNode: this,... in QuoteLineNode threw the

prevNode: this,
^ Cannot use this [1] in an input position because this [1] is expected to occur only in output positions. [incompatible-variance]

same as sample error linked in the description

That might be a problem with Flow then, it might not be able to represent this constraint. Possibly always specifying the exact class that is overriding the method will work? Or maybe there is another Flow specific trick that can be used here? Last time I used Flow it was very different.


0.3.3 : Cannot find module 'core-js/modules/es6.array.is-array'

jimp 0.3.3
ubuntu 18
package.json:
{
  "name": "foo",
  "version": "0.0.1",
  "devDependencies": {
    "jimp": "0.3.3"
  }
}

app.js:
"use strict";
const jimp = require("jimp");

run it:
node app.js
result is an error with:
Error: Cannot find module 'core-js/modules/es6.array.is-array'

same here

@hipstersmoothie I should have mentioned I'm on LTS of everything, so
ubuntu 18.04
npm -v = 3.5.2
node -v = 8.10.0
Maybe you're on bleeding edge?

Did my above fix help until a release can get out?

@hipstersmoothie I don't use yarn, so don't know if that works, sorry. But I downgraded to 0.3.2 which works for me.

Can you verify the babel polyfill? you can instead just
npm i @babel/polyfill

npm i and yarn add are the same thing

@hipstersmoothie I tried that and it works. Might be useful for those using those tools, thanks.
I'll wait for the next version though.

I'm gonna keep this open til a release happens

Thought my node installation was broken, was going insane. Thanks for the quick release.

I'm trying to get it release ASAP. But I don't have the key yet. In the meantime please install @babel/polyfill and jimp should work as expected

I've published the release but the problem seems to be still there in 0.3.4.

This is from my smoke test as follows:
var Jimp = require("jimp");

// open a file called "lenna.png"
Jimp.read("lenna.png", function (err, lenna) {
    if (err) throw err;
    lenna.resize(256, 256)            // resize
         .quality(60)                 // set JPEG quality
         .greyscale()                 // set greyscale
         .write("lena-small-bw.jpg"); // save
});

I've published the release but the problem seems to be still there in 0.3.4.

This is from my smoke test as follows:
var Jimp = require("jimp");

// open a file called "lenna.png"
Jimp.read("lenna.png", function (err, lenna) {
    if (err) throw err;
    lenna.resize(256, 256)            // resize
         .quality(60)                 // set JPEG quality
         .greyscale()                 // set greyscale
         .write("lena-small-bw.jpg"); // save
});

It got added as a dev dep so it wasn't installed 😢

still Error: Cannot find module 'core-js/modules/es6.array.is-array'...

I just pushed a commit that should resolve the issue. https://github.com/oliver-moran/jimp/commit/97ed2a524ea4263af2581dd20578f166d714fd34 @oliver-moran

@hipstersmoothie sorry but still Error: Cannot find module 'core-js/modules/es6.array.is-array'

@hipstersmoothie unfortunately, last commit didn't...

My branch here seems to work. I ran
yarn add hipstersmoothie/jimp#test

and running the above code yields no errors.

npm uninstall --save jimp
npm install --save hipstersmoothie/jimp#test

And still Error: Cannot find module 'core-js/modules/es6.array.is-array'

did you rm -rf node_modules/

@johnnykramer what version of node are you using?

@hipstersmoothie
> node -v
> v8.11.3

yes, i've done rm -rf node_modules/

I've published 0.3.5. Smoke test works again from my end.
I'm also using v8.11.3.

I just tested with v8.11.3 also and it seems to work

@johnnykramer can you test with v0.3.5 ?

@oliver-moran @hipstersmoothie
node v10.8.0 also Error: Cannot find module 'core-js/modules/es6.array.is-array'

I can no longer reproduce the issue on any version of node. Could you throw a sample project together.
this is my setup right now:
{
  "name": "jimp-testing-project",
  "version": "1.0.0",
  "main": "src/index.js",
  "dependencies": {
    "jimp": "^0.3.5"
  }
}

var Jimp = require("jimp");

// open a file called "lenna.png"
Jimp.read("./exampleImages/lenna.png", function (err, lenna) {
    if (err) throw err;
    lenna.resize(256, 256)            // resize
         .quality(60)                 // set JPEG quality
         .greyscale()                 // set greyscale
         .write("lena-small-bw.jpg"); // save
});

maybe its a lock file problem? try deleting any of those if you have them

@hipstersmoothie yes, @babel/polyfill present in the jimp's dependencies

can i see the full stack trace?

core-js is in node modules too?

><EMAIL_ADDRESS>start /home/ec2-user/environment/test-app
> node index.js

internal/modules/cjs/loader.js:583
    throw err;
    ^

Error: Cannot find module 'core-js/modules/es6.array.is-array'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
    at Function.Module._load (internal/modules/cjs/loader.js:507:25)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:20:18)
    at Object.<anonymous> (/home/ec2-user/environment/test-app/node_modules/jimp/dist/index.js:3:1)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>start: `node index.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!  I can't reproduce this at all. Is this within a AWS instance?

The dependancies are installed so I don't know why it can't resolve them...

does it work if you add @babel/polyfill or core-js directly?

How about what npm version are you using?

If you are on npm@2 it could be a problem https://github.com/babel/babel-preset-env/issues/162

it is cloud9 on the aws ec2 instance.
npm v6.2.0

And i've installed @babel/polyfill and core-js manually — still not working! :(

I have no experience with aws. You have any ideas? I'll search around for similar issues

you definitely have core-js/modules/es6.array.is-array installed and babel is for sure trying to include it. maybe you could globally install it on the instance? (I'm just taking guesses at this point)

Awesome!

now to figure out why core-js 3 was being installed

I'm looking at the lock file and see no reference to core-js v3

So on a clean install it defaults to core-js@3-ish?

These are my only entries for core-js in my yarn.lock
"<EMAIL_ADDRESS>  version "7.0.0-rc.1"
  resolved<EMAIL_ADDRESS>  dependencies:
    core-js "^2.5.7"
    regenerator-runtime "^0.11.1"

and in the package-lock.json
    "@babel/polyfill": {
      "version": "7.0.0-rc.1",
      "resolved"<EMAIL_ADDRESS>      "integrity": "sha512-x6s1nu32T0Ayzh/d0SXB4UPIZNcK7Oc01tmZQEhBVwFeOF+ynUlyS4b+zKX1kfbjD7o4qF6PXvDXLRX6i28hsA==",
      "requires": {
        "core-js": "2.5.7",
        "regenerator-runtime": "0.11.1"
      }
    },

what does your lock file look like?

@hipstersmoothie Thanks for fixing this.
Off topic, what terminal are you using in that image?

@lonix1 Integrated vscode terminal, oh-my-zsh, with my custom theme https://github.com/hipstersmoothie/PowerlevelHipstersmoothie

🤘 Weed leafs and flames 🤘

@johnnykramer npm i -s<EMAIL_ADDRESS>works for me :)
Thanks man


enh: add dt.timestamp

What type of PR is this? (check all applicable)

[ ] 💾 Refactor
[x] ✨ Feature
[ ] 🐛 Bug Fix
[ ] 🔧 Optimization
[ ] 📝 Documentation
[ ] ✅ Test
[ ] 🐳 Other

Related issues

Related issue #
Closes #1182

Checklist

[x] Code follows style guide (ruff)
[x] Tests added
[x] Documented the changes

If you have comments or can explain your changes, please do so below.

I am catching up with Marco's today's livestream - as he explains how the integer representation works, wouldn't make sense to have something like the following pseudocode (details such as time units, time zones, date vs datetime, etc need to be taken care)
diff = series - datetime(1970, 1, 1, 0, 0, 0, tzinfo=...)
if time_unit == "ns":
    return diff.dt.total_nanoseconds()
elif ...
...

Maybe it ends up being similar effort, but feels less redundant than all the nested if cases.
Edit: pyarrow also has nanoseconds_between, microseconds_between and milliseconds_between two date/datetimes

thanks @FBruzzesi for taking a look! total_nanoseconds is for Duration, not Datetime, so we can't use it here unfortunately 😅

thanks @FBruzzesi for taking a look! total_nanoseconds is for Duration, not Datetime, so we can't use it here unfortunately 😅

I am most likely missing something - what I am suggesting is, instead of using the date(time) representation, compute the difference between the series/expr of interest and the scalar 1970-01-01 00:00:00 (unix_epoch). This would result in a Duration object (wouldn't it?) and then use the Duration methods. Sorry if I misunderstood the internal representation explanation

ah sorry you're right 🤦 maybe that's indeed simpler, will try!

i think a cast would always be more efficient, from a little timing test:
In [50]: %timeit (s - datetime(1970,1,1)).dt.total_seconds()
465 μs ± 64.1 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [51]: %timeit s.astype('int64')
52.9 μs ± 10.8 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)


Integrate docs changes to enable advanced search with autocomplete

Thanks Cliff! I've unpinned the version.

Thanks @bryanressler-idmod ! Re: the issue of the parallel build failing locally -- are you able to reproduce that?

@bryanressler-idmod I looked at the doc build from your branch and it doesn't seem to be searching across other docsets or displaying the text under the search box. Does this require additional changes to be made to the parent docset?


[QUESTION] Import only imports the last row of a CSV

Prerequisites

[X] Have read and understood: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/.github/SUPPORT.md
[X] Checked if your Laravel Excel version is still supported: https://docs.laravel-excel.com/3.1/getting-started/support.html#supported-versions
[X] Checked that your question isn't already asked before.
[X] Filled in the entire issue template

Versions

PHP version: 7.3.7
Laravel version: 5.8.30
Package version: 3.1.15

Description
Let's say I have the following CSV:
Table Name,LastUTC Column,Version Table Name,API Key,Pipeline ID
PassengerFee,ModifiedUTC,PassengerFeeVersion,7fa395c18448de9a650d4a449e07ebe280f2ed9ed852a587fced2ca949abba28,1
Booking,ModifiedUTC,BookingVersion,d81ce11cf559f94636913f847909b655ec14a1ed47f5f924dfbb79feb3016e20,2
BookingPassenger,ModifiedUTC,BookingPassengerVersion,739369eb42db13b6d786f15ecda9327b473d58fcdb0cd3d5bdf32897273e6686,3

and I have this controller method:
    public function store(IngestRequest $request)
    {
            Excel::import(new VersionCSVImport($request->schema_id), $request->csv);
            return Excel::download(new VersionCSVExport($request->schema_id), $request->file()['csv']->getClientOriginalName());
    }

and I have this import:

namespace App\Imports;

use App\Pipeline;
use Illuminate\Support\Collection;
use Maatwebsite\Excel\Concerns\ToCollection;
use Maatwebsite\Excel\Concerns\WithHeadingRow;

class VersionCSVImport implements ToCollection, WithHeadingRow
{
    protected $schema_id;

    public function __construct($schema_id)
    {
        $this->schema_id = $schema_id;
    }

    /**
    * @param Collection $collection
    */
    public function collection(Collection $rows)
    {
        foreach ($rows as $row) {
            $table_name = $row['table_name'];
            $modified_column_name = $row['lastutc_column'];
            $version_table_name = $row['version_table_name'];

            $pipeline = Pipeline::create([
                'name' => $table_name,
                'table_name' => $table_name,
                'modified_column_name' => $modified_column_name,
                'version_table_name' => $version_table_name,
                'schema_id' => $this->schema_id
            ]);
        }
    }
}

it only ever imports the last line of the CSV.   If I add a dump($pipeline); after the create, it shows the model has been created, but it doesn't actually persist to the database.
Additional Information
n/a

For the record, I've also tried using OnEachRow as well as ToCollection, and they both result in the same issue.

Obviously this won't be the solution for everyone, but I switched the transactions.handler setting in config/excel.php to null and it worked fine.


update: autocompletion icons

They look very nice! You made them?

They look very nice! You made them?

oh, I wish I could make them myself, I always use svg from flaticon since they are the vast majority of open source

So, they can be used in open source projects directly? Do they need some kind of attribution?
I just wanna be sure

So, they can be used in open source projects directly? Do they need some kind of attribution?
I just wanna be sure

yes, you just have to put this in the readme.
Iconos diseñados por Freepik from www.flaticon.es

Well, I think it would be better to have our own icons since I wanna make their's color customizable, I might need to modify them to do so
🙏 Sorry

No problem, I think it is good that they can be personalized, I close this pr?


ValueError: math domain error

Expected Behavior
No Crash
Actual Behavior
Traceback (most recent call last):
File "pokecli.py", line 291, in
main()
File "pokecli.py", line 257, in main
bot.tick()
File "/home/joe/PokemonGo-Bot/pokemongo_bot/init.py", line 65, in tick
self.navigator.take_step()
File "/home/joe/PokemonGo-Bot/pokemongo_bot/spiral_navigator.py", line 85, in take_step
point['lng']
File "/home/joe/PokemonGo-Bot/pokemongo_bot/cell_workers/utils.py", line 40, in distance
return 12742 * asin(sqrt(a)) * 1000
ValueError: math domain error
Steps to Reproduce
latest git pull
run with standard example configuration
logs in fine
then produces this error, most likely taking a log of a negative number...
Other Information
OS:
Git Commit: (run 'git log -n 1 --pretty=format:"%H"' and paste it here)

Did you do a "pip install -r requirements.txt"?
Probably not...

Can you give us a full log?

I'm going to close this as a new pip install -r requirements has made me unable to reproduce. good catch friederschueler.


Potential efficiency improvement of density-fitting (cderi, get_jk)

Hi devs!
While performing RI-JK (SCF) computations, I found two possible efficiency bottlenecks. These problems may be alleviated by minor code modifications.
For some remarks below, I think I think I'm not ready to propose a pull request for this issue.
Coulomb integral not paralleled using numpy.einsum without optimize=True
For evaluation of coulomb integrals, codes like https://github.com/pyscf/pyscf/blob/231da24f746d3a7988a4c48c9237c4788bda6a36/pyscf/df/df_jk.py#L260-L261 and https://github.com/pyscf/pyscf/blob/231da24f746d3a7988a4c48c9237c4788bda6a36/pyscf/df/df_jk.py#L290-L291 envokes numpy.einsum, which have optional parameter optimize=False.
For optimize=False, numpy.einsum just contract tensors or matrices without parallel. That could be great efficiency loss when using more CPU threads.
Simply add optimize=True to numpy.einsum will resolve this problem.
Since many codes in PySCF uses numpy.einsum without optimize=True, I think that this is not only a problem of RI-JK SCF, but also other tasks.
Cholesky decomposed ERI with scipy.linalg.solve_triangular
I found that, probably, scipy.linalg.solve_triangular just copies or transposes the large 3c-2e integrals ints $J_{P, \mu \nu}$ to $J_{\mu \nu, P}$. Since NumPy/SciPy often performs copy or transposition with only one thread, this can be extremely slow.
For evaluation of cholesky decomposed ERI $Y_{Q, \mu \nu}$ (mf.with_df._cderi, discuss incore case for simplicity)
$$\sum_{Q} L_{PQ} Y_{Q, \mu \nu} = J_{P, \mu \nu} \text{ or } \mathbf{L} \mathbf{Y} = \mathbf{J}$$
where $L_{PQ}$ is the lower triangular matrix of cholesky-decomposed 2c-2e ERI $J_{PQ}$, as well as $J_{P, \mu \nu}$ the 3c-2e ERI
$$\sum_{R} L_{PR} L_{QR} = J_{PQ} \text{ or } \mathbf{L} \mathbf{L}^\dagger = \mathbf{J}$$
https://github.com/pyscf/pyscf/blob/231da24f746d3a7988a4c48c9237c4788bda6a36/pyscf/df/incore.py#L180-L188
PySCF's gto.moleintor.getints3c will probably generates ints $J_{P, \mu \nu}$ as C-contiguous matrix, or $J_{\mu \nu, P}$ as F-contiguous matrix.
SciPy seems only accepts fortran-type array as arguments of its BLAS interface. If matrix transposition is to be avoided, then $J_{\mu \nu, P}$ as F-contiguous matrix is better to be passed as argument. Then our problem is to solve $\mathbf{Y}^\dagger$ from $\mathbf{Y}^\dagger \mathbf{L}^\dagger = \mathbf{J}^\dagger$ F-contiguously; then transpose $\mathbf{Y}^\dagger$ to $\mathbf{Y}$ as a C-contiguous matrix.
What's tricky is that, for function scipy.linalg.solve_triangular, it calls dtrtrs. This BLAS function could not handle $\mathbf{L}^\dagger$ lying on right-side. So the underlying BLAS function dtrsm is required to solve $\mathbf{Y}^\dagger \mathbf{L}^\dagger = \mathbf{J}^\dagger$. However, seems there's no function in scipy wraping ?trsm as high-level API. So to avoid too much data copy and transposition, we may need to explicitly call scipy.linalg.blas.dtrsm:
if decompose_j2c == 'cd':
    if not ints.flags.c_contiguous:
        log.warn("`ints` is not c_contiguous here. Additional transpose is involved.")
        ints = np.array(ints, order='C')
    cderi[:, p0:p1] = scipy.linalg.blas.dtrsm(
        1, low.T, ints.T, side=1, lower=False, overwrite_b=True).T

This code also has some potential problems:

This code almost asserts ints as result from transposed gto.moleintor.getints3c is C-contiguous; but I don't know if there could be any exceptions.
This code assumes integrals to be double-type.
This code avoids transpose, but still requires some data copy. Assignment to cderi[:, p0:p1] also requires data copy, which seems to be unavoidable if ints and cderi uses different buffers in memory. I think this data copy process is possible to be avoided, but that may involve some code logic changes to buffer assignment.
Above mentioned code is only for incore.py; I haven't validated case of outcore.py.

Simple benchmark efficiency improvement
C25H52, RI-JK/def2-TZVP (202 electrons, 1087 basis, 2811 auxbasis), Intel 6150 with 32 physical cores, fully incore
PySCF compiled with MKL, haswell

use np.einsum(optimize=True)*
use dtrsm
RI-JK time

x
x
80.66 sec

√
x
48.25 sec

√
√
35.64 sec

* This option is applied globally, meaning that I manually modified signature in numpy/core/einsumfunc.py directly.
benchmark_code.py.txt

simple one:
rho = lib.dot(dmtril, eri1.T)
vj += lib.dot(rho, eri1)

openmp with dtrsm may be faster, a ctypes should be use.

@baopengbp

simple one:
rho = lib.dot(dmtril, eri1.T)
vj += lib.dot(rho, eri1)

Yes, exactly 😄

openmp with dtrsm may be faster, a ctypes should be use.

I guess that's equally fast. Given F-contiguous matrices, scipy's wrapper of Lapack's dtrsm (linked with MKL) may be equally fast compared to MKL itself. It is also multi-threaded. So I think that ctypes could be utilized, but is certainly not the only option.
I haven't tested the case of OpenBLAS.
Though it seems (I used to believe) that scipy runs dtrtrs or dtrsm single-threaded, that's actually because scipy applies transposition of input matrices to F-contiguous. The transposition costs lots of time (could be accelarated but only single threaded by scipy), and then scipy performs dtrsm fast and multi-threaded.

Additional note for get_k
This seems not a great improvement, but it's simple and probably worth a try. This following code can actually utilize numpy.dot, instead of lib.dot, though it is possible the latter one is more stable in other types of matrix multiplication computations.
https://github.com/pyscf/pyscf/blob/14d88828cd1f18f1e5358da1445355bde55322a1/pyscf/df/df_jk.py#L304
Possible reason that numpy.dot could be faster than lib.dot here, is that numpy will handle something like $\mathbf{A}^\dagger \mathbf{A}$ by dsyrk instead of dgemm.
https://github.com/numpy/numpy/blob/c35050022fe9e613b316c4d586d59686b9dbc34d/numpy/_core/src/umath/matmul.c.src#L161-L196

That is useful, I saw a 30% faster for this dot.

openmp with dtrsm may be faster, a ctypes should be use.

I guess that's equally fast. Given F-contiguous matrices, scipy's wrapper of Lapack's dtrsm (linked with MKL) may be equally fast compared to MKL itself. It is also multi-threaded. So I think that ctypes could be utilized, but is certainly not the only option. I haven't tested the case of OpenBLAS.
Though it seems (I used to believe) that scipy runs dtrtrs or dtrsm single-threaded, that's actually because scipy applies transposition of input matrices to F-contiguous. The transposition costs lots of time (could be accelarated but only single threaded by scipy), and then scipy performs dtrsm fast and multi-threaded.

Yes，It took long times to transpose to the array for dtrsm and back.

openmp with dtrsm may be faster, a ctypes should be use.

I guess that's equally fast. Given F-contiguous matrices, scipy's wrapper of Lapack's dtrsm (linked with MKL) may be equally fast compared to MKL itself. It is also multi-threaded. So I think that ctypes could be utilized, but is certainly not the only option. I haven't tested the case of OpenBLAS.
Though it seems (I used to believe) that scipy runs dtrtrs or dtrsm single-threaded, that's actually because scipy applies transposition of input matrices to F-contiguous. The transposition costs lots of time (could be accelarated but only single threaded by scipy), and then scipy performs dtrsm fast and multi-threaded.

Yes，It took long times to transpose to the array for dtrsm and back.

I like the improvements discovered in this thread so much that I made a pull request (#2205). Some differences:

the trsm variation of appropriate type is found using scipy.linalg.get_blas_funcs
fall back to scipy.linalg.solve_triangular if F contiguous
outcore
I've always found np.matmul to be slightly faster than np.dot for unclear reasons, perhaps some weird openmp behavior, so I used it.


feat: div that encapsulate PageList component

Following my #749, I needed to tweaks the pageList component, without using cursed css like .no-thing + hr + p that can be broke.
Using this, it allow to customize the pageList for specific class/page.
In my case, for example, I could make this:
.no-folder-list .folder-content-hint {
	display: none
}

So I can disable the component only on a page that have the no-folder-list classes.

@Lisandra-dev can you address the review comment and also apply it to TagContent before I merge?

I dunno why the commit was not applied, wait a minute!

Done ✅


[WFLY-16372] Wrong mechanism realm name in host.xml test configurations.

Issue: https://issues.redhat.com/browse/WFLY-16372

Thanks, @ivassile


daos: daos_mknod implementation

daos_mknod is implemented in two steps:

dfs_open(..., O_CREATE,...)
dfs_release(...)

This works for regular files, but links are not
supported in dfs.
closes #377
Signed-off-by: Dalton Bohning<EMAIL_ADDRESS>

closes #377

Thanks @daltonbohning .  You're really knocking out a lot of items over this past week!  Nice!
It's fine to just support regular files for now, especially since the tools don't generally support types other than directories, regular files, and symlinks just yet.  However, let's add a check and print a warning message to alert the user if this mknod function happens to get called with something other than a regular file.  In that case, perhaps we could also set errno and return -1 if there is good errno value that matches up.
Also, it'd be helpful for future developers to add a comment in the function just to mention that the goal is to emulate a mknod operation using an open/release.

If printing/returning errors would cause problems, like lots of harmless warnings under typical usage, we can reconsider.  I think the main reason we call mknod right now is to set lustre striping on files, so it's kind of a special case.  We could double check the rest of mpiFileUtils code to be sure.

I revised my comment to be more clear, after some further investigation into dfs_open.
dfs_open allows regular files, symbolic links, and directories. If anything else is supplied, dfs_open returns an appropriate error, which daos_mknod captures and propogates:
https://github.com/daos-stack/daos/blob/73b42ec7870d74091a074e2b189f8d2d48f4fd53/src/client/dfs/dfs.c#L2315-L2340

As for mpifileutils calling mknod, I found this was necessary for DAOS (and possible regular POSIX as well, though I'm not sure).
I ran into an issue where if multiple processes were writing different chunks to the same file, and that file did not yet exist, then the destination file would sometimes not contain all of the written chunks. But if the destination file did exist before copying any chunks, then there were no issues.
mfu_flist_copy calls mfu_file_create, which calls mfu_file_mknod BEFORE copying any data. And since mknod creates the file, it solves this issue.

As for mpifileutils calling mknod, I found this was necessary for DAOS (and possible regular POSIX as well, though I'm not sure).
I ran into an issue where if multiple processes were writing different chunks to the same file, and that file did not yet exist, then the destination file would sometimes not contain all of the written chunks. But if the destination file did exist before copying any chunks, then there were no issues.
mfu_flist_copy calls mfu_file_create, which calls mfu_file_mknod BEFORE copying any data. And since mknod creates the file, it solves this issue.

Interesting.  Sounds like a race condition on creating the "inode" or something.  Based on your debugging, it sounds like this avoids the problem.  Do you know whether that is expected behavior in DAOS?

Interesting. Sounds like a race condition on creating the "inode" or something. Based on your debugging, it sounds like this avoids the problem. Do you know whether that is expected behavior in DAOS?

I checked with regular POSIX files (removing the mknod call) and was not able to reproduce the issue.
I'm not sure if it is expected behavior in DAOS, but since POSIX works without issues, it seems daos_mknod "indirectly" corrects the issue. I would like to find a more proper fix for this. I'll need to do some further digging. It may be that the dfs library just isn't explicitly handling race conditions, since it doesn't use MPI, etc.

Interesting. Sounds like a race condition on creating the "inode" or something. Based on your debugging, it sounds like this avoids the problem. Do you know whether that is expected behavior in DAOS?

I checked with regular POSIX files (removing the mknod call) and was not able to reproduce the issue.
I'm not sure if it is expected behavior in DAOS, but since POSIX works without issues, it seems daos_mknod "indirectly" corrects the issue. I would like to find a more proper fix for this. I'll need to do some further digging. It may be that the dfs library just isn't explicitly handling race conditions, since it doesn't use MPI, etc.

@adammoody @daltonbohning I can't say for sure either. It is a bit puzzling to me since I would think that on the open call in mfu_copy_open_file it would open and create the file if it doesn't exist, then write to its particular offset with the length of bytes it was given.

I can't say for sure either. It is a bit puzzling to me since I would think that on the open call in mfu_copy_open_file it would open and create the file if it doesn't exist, then write to its particular offset with the length of bytes it was given.

I'll look into this as well, to rule out mfu as being the culprit.

las p1cont1 p1cont2; rm p1cont2/a

@adammoody Turns out this was in fact a bug in dfs.c that was fixed on Thursday. But we needed the daos_mknod implementation anyway, so that's a win!

If printing/returning errors would cause problems, like lots of harmless warnings under typical usage, we can reconsider. I think the main reason we call mknod right now is to set lustre striping on files, so it's kind of a special case. We could double check the rest of mpiFileUtils code to be sure.

I think it will be okay to do as you suggest, since mfu_file->type == DAOS is really only being used from dcp.c at this time, and the DAOS team (Danielle and I) will have to run our tests again if ever mpifileutils is updated, so we should catch any future issues.

@adammoody I think I've finished with this one now. I added a little more cleanup, and switched an error check in the spirit of "keeping an rc check close to the function call", i.e. if parent == NULL then error instead of if parent != NULL... else error


Update README.md

Add instructions on where to run the inception download commands

Can one of the admins verify this patch?

I signed it!

Looks like there is still a problem with the CLA.
Could you verify your email you used to sign the CLA is the same one as the email that is in your commit?

@mortenjust the CLA still seems to be an issue.
Could you make sure your commit email and the email you used to sign the CLA are the same?

hey @gunan - thanks, should be good now. Thought it was enough to add my other CTA-signed emails to github. Anyway, signed the @genstart.dk email just now.


Replace lodash/debounce with custom impl

NextJS middleware can't deal with node modules, so we need to be careful
with adding them.


Add Twilio subscription via websub

Purpose
Add Twilio subscription via websub
Goals
Resolves https://github.com/wso2-enterprise/choreo/issues/14190
Security checks

Followed secure coding standards in http://wso2.com/technical-reports/wso2-secure-engineering-guidelines? yes
Ran FindSecurityBugs plugin and verified report? no
Confirmed that this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets? yes

Test environment
2201.2.1

Since this involves an API change we have to update related samples with this PR. And need to do a major version bump with it for the trigger.

Closing as per https://github.com/wso2-enterprise/choreo/issues/14190#issuecomment-2329032315


[mesos] collect mesos.registrar.log.recovered metric

This collects the mesos.registrar.log.recovered metric, as defined at https://mesos.apache.org/documentation/latest/monitoring/.  The metric was added to Mesos 1.0.0 as per MESOS-5618.
I have been successfully monitoring Mesos masters (1.1.0) with this change deployed on them.
Sidenote: Runningrake ci:run[mesos_master] locally displays "Ran 0 tests in 0.000s" (even before making changes), despite the test defined intest_mesos_master.py.  I encountered this on both Mac and Debian Linux.

Hi @darnould, thanks for the PR! This looks great, I'll merge. It will be packaged with the upcoming 5.12.0 release.
Regarding your sidenote, rake ci:run[mesos_master] doesn't run the tests because @attr(requires='mesos_master'] is not before TestMesosMaster class. The checks will run under rake ci:run[default].


Advisory Severity

Currently the only ones who have severity are Vulnerabilities. Technically, an Advisory can contain multiple Vulnerabilities.

The Trustification (v1) UI has aggregated severity to define the severity of the advisory itself

The question is: are we going to also have aggregated severity? If so, let's use this issue to track the work done in that regard.  If we won't have aggregated severity then let's document it here so the UI can also remove that column from the table (image below)

CSAF Spec: https://docs.oasis-open.org/csaf/csaf/v2.0/cs01/csaf-v2.0-cs01.html#3212-document-property---aggregate-severity

Aggregate severity (aggregate_severity) of value type object with the mandatory property text and the optional property namespace is a vehicle that is provided by the document producer to convey the urgency and criticality with which the one or more vulnerabilities reported should be addressed. It is a document-level metric and applied to the document as a whole — not any specific vulnerability. The range of values in this field is defined according to the document producer's policies and procedures.

So I think there's value in this when viewing advisories. It's an indication, expressed by the advisory author, on how important the document is.

On a more philosophical side. I always thought "Advisories" are a source of data/information but not worth showing it in the UI. An Advisory's data should serve to enrich metadata of other entities/models like "products", "sboms", "vulnerabilities" but no more.
We do have a compromise in terms of "compatibility" between v1 and Trustify. So I support keeping Advisories as a main model/entity exposed through the APIs (and also in the UI). So if possible, we should be as close as we can to the v1 of Trustification (there might be deviations as we all know tought)

Just general thoughts :)

On a more philosophical side. I always thought "Advisories" are a source of data/information but not worth showing it in the UI. An Advisory's data should serve to enrich metadata of other entities/models like "products", "sboms", "vulnerabilities" but no more.

That's true to some degree. Ideally, you don't interact it with it.
Unless you reach a point where you want to understand "why" things are as they are. At this point it might make sense to direct the user to source of the information. And that's the ingested advisory, which contains the information as authored by the supplier.

The PR https://github.com/trustification/trustify/pull/300 has added "aggregated_severity" to advisories. That is a good first step. However, the aggregated_severity of each advisory obtained through /api/v1/advisory is null. So there is still work to do

Not sure in which PR it was been implemented by I can see a field average_severity in the Advisory entity


Some components like the OpenSearch core project, common-utils, job-scheduler and security integration test are skipped

Is your feature request related to a problem? Please describe
Not sure if this correct but components like the OpenSearch core project, common-utils, job-scheduler and security integration test are skipped. Are they just missing in the manifest (which would be a bug) or are there no integration tests for those components or plugins?:
2024-01-22 18:06:51 INFO     Entering integ test for OpenSearch
2024-01-22 18:06:51 INFO     Skipping integ-tests for OpenSearch, as it is currently not declared in the test manifest
2024-01-22 18:06:51 INFO     Skipping integ-tests for common-utils, as it is currently not declared in the test manifest
2024-01-22 18:06:51 INFO     Skipping integ-tests for job-scheduler, as it is currently not declared in the test manifest
2024-01-22 18:06:51 INFO     Skipping integ-tests for security, as it is currently not declared in the test manifest

Describe the solution you'd like
Either include the components in the manifest or create integration tests for them or make it explicite in the documentation why they are skipped.
Describe alternatives you've considered
Unsure if at least the unit test are execute during the release process, see https://github.com/opensearch-project/opensearch-build/issues/4364
Additional context
No response

Hello @salyh, it appears that the output snippet in your question likely originated from an early stage of a build release cycle. This is when the test manifest isn't fully set for running tests on certain/any/many components.
The INFO message, indicating that it's "currently not declared in the test manifest," supports this.
In the community, manifests like https://github.com/opensearch-project/opensearch-build/blob/main/manifests/2.12.0/opensearch-2.12.0-test.yml are used to specify the components to be tested with. Furthermore, these manifests can be updated by the community as needed, depending on the progress and requirements of the release.
The Wiki link here hopefully can explain much better how community use the test manifest to run the test during the release cycle.  https://github.com/opensearch-project/opensearch-build/wiki,  thanks !!

Not everyone has on-boarded to integ-test framework. We are tracking the components here: https://github.com/opensearch-project/opensearch-build/issues/58
However, we are also gatekeeping the new components being added to the distribution to include integ-test.
Rest assured if the component is a part of the test manifest, the tests will be run.
Thanks!
Adding @bbarani if we should close this issue as duplicate of https://github.com/opensearch-project/opensearch-build/issues/58

Hi, @salyh, @gaiksaya, @bbarani, just want to follow up if we can close this issue, and using https://github.com/opensearch-project/opensearch-build/issues/58 to track further, thanks.


easy-search with --cov-mode 1 -c 1 --greedy-best-hits drops exact matches

I've bumped into what appears to be a bug when using easy-search with the --greedy-best-hits flag and --cov-mode 1 -c 1, where hits containing targets that fully cover the query are being filtered out. I've outlined the behaviour in the minimal repro below:
Using file seq.fasta with content (the sequence is arbitrary):
>random_seq
NTLGEQNARIWSEREVGICLARKHPFLDFSVVKDMIATGLGTDYKALSPQ

and running easy-search with the above fasta as both the query and target:
mmseqs easy-search seq.fasta seq.fasta result.tsv tmp --cov-mode 1 -c 1 --greedy-best-hits produces an empty result.tsv
Running the above command without the --greedy-best-hits flag results in the expected result.tsv:
random_seq	random_seq	1.000	50	0	0	1	50	1	50	1.586E-38	108

My guess is that there is a bug in the summarizeresult command being run when --greedy-best-hits is passed in:
summarizeresult tmp/15248047069833069253/result tmp/15248047069833069253/result_best -a 1 --overlap 0 -c 1 --threads 8 --compressed 0 -v 3
But it's quite possible I'm missing something... thanks for your time!
Expected Behavior
A result.tsv with content
random_seq	random_seq	1.000	50	0	0	1	50	1	50	1.586E-38	108

Current Behavior
An empty result.tsv
Steps to Reproduce (for bugs)
echo ">random_seq\nNTLGEQNARIWSEREVGICLARKHPFLDFSVVKDMIATGLGTDYKALSPQ" > seq.fasta
mmseqs easy-search seq.fasta seq.fasta result.tsv tmp --cov-mode 1 -c 1 --greedy-best-hits

MMseqs Output (for bugs)

Create directory tmp
easy-search seq.fasta seq.fasta result.tsv tmp --cov-mode 1 -c 1 --greedy-best-hits

MMseqs Version:                        	13.45111
Substitution matrix                    	nucl:nucleotide.out,aa:blosum62.out
Add backtrace                          	false
Alignment mode                         	3
Alignment mode                         	0
Allow wrapped scoring                  	false
E-value threshold                      	0.001
Seq. id. threshold                     	0
Min alignment length                   	0
Seq. id. mode                          	0
Alternative alignments                 	0
Coverage threshold                     	1
Coverage mode                          	1
Max sequence length                    	65535
Compositional bias                     	1
Max reject<PHONE_NUMBER>
Max accept<PHONE_NUMBER>
Include identical seq. id.             	false
Preload mode                           	0
Pseudo count a                         	1
Pseudo count b                         	1.5
Score bias                             	0
Realign hits                           	false
Realign score bias                     	-0.2
Realign max seqs<PHONE_NUMBER>
Gap open cost                          	nucl:5,aa:11
Gap extension cost                     	nucl:2,aa:1
Zdrop                                  	40
Threads                                	8
Compressed                             	0
Verbosity                              	3
Seed substitution matrix               	nucl:nucleotide.out,aa:VTML80.out
Sensitivity                            	5.7
k-mer length                           	0
k-score<PHONE_NUMBER>
Alphabet size                          	nucl:5,aa:21
Max results per query                  	300
Split database                         	0
Split mode                             	2
Split memory limit                     	0
Diagonal scoring                       	true
Exact k-mer matching                   	0
Mask residues                          	1
Mask lower case residues               	0
Minimum diagonal score                 	15
Spaced k-mers                          	1
Spaced k-mer pattern
Local temporary path
Rescore mode                           	0
Remove hits by seq. id. and coverage   	false
Sort results                           	0
Mask profile                           	1
Profile E-value threshold              	0.001
Global sequence weighting              	false
Allow deletions                        	false
Filter MSA                             	1
Maximum seq. id. threshold             	0.9
Minimum seq. id.                       	0
Minimum score per column               	-20
Minimum coverage                       	0
Select N most diverse seqs             	1000
Min codons in orf                      	30
Max codons in length                   	32734
Max orf gaps<PHONE_NUMBER>
Contig start mode                      	2
Contig end mode                        	2
Orf start mode                         	1
Forward frames                         	1,2,3
Reverse frames                         	1,2,3
Translation table                      	1
Translate orf                          	0
Use all table starts                   	false
Offset of numeric ids                  	0
Create lookup                          	0
Add orf stop                           	false
Overlap between sequences              	0
Sequence split mode                    	1
Header split mode                      	0
Chain overlapping alignments           	0
Merge query                            	1
Search type                            	0
Search iterations                      	1
Start sensitivity                      	4
Search steps                           	1
Exhaustive search mode                 	false
Filter results during exhaustive search	0
Strand selection                       	1
LCA search mode                        	false
Disk space limit                       	0
MPI runner
Force restart with latest tmp          	false
Remove temporary files                 	true
Alignment format                       	0
Format alignment output                	query,target,fident,alnlen,mismatch,gapopen,qstart,qend,tstart,tend,evalue,bits
Database output                        	false
Overlap threshold                      	0
Database type                          	0
Shuffle input database                 	true
Createdb mode                          	0
Write lookup file                      	0
Greedy best hits                       	true

Alignment backtraces will be computed, since they were requested by output format.
createdb seq.fasta tmp/15248047069833069253/query --dbtype 0 --shuffle 1 --createdb-mode 0 --write-lookup 0 --id-offset 0 --compressed 0 -v 3

Converting sequences

Time for merging to query_h: 0h 0m 0s 9ms
Time for merging to query: 0h 0m 0s 8ms
Database type: Aminoacid
Time for processing: 0h 0m 0s 38ms
createdb seq.fasta tmp/15248047069833069253/target --dbtype 0 --shuffle 1 --createdb-mode 0 --write-lookup 0 --id-offset 0 --compressed 0 -v 3

Converting sequences

Time for merging to target_h: 0h 0m 0s 9ms
Time for merging to target: 0h 0m 0s 8ms
Database type: Aminoacid
Time for processing: 0h 0m 0s 40ms
Create directory tmp/15248047069833069253/search_tmp
search tmp/15248047069833069253/query tmp/15248047069833069253/target tmp/15248047069833069253/result tmp/15248047069833069253/search_tmp -a 1 --alignment-mode 3 -c 1 --cov-mode 1 -s 5.7 --remove-tmp-files 1

prefilter tmp/15248047069833069253/query tmp/15248047069833069253/target tmp/15248047069833069253/search_tmp/6923777973734096903/pref_0 --sub-mat nucl:nucleotide.out,aa:blosum62.out --seed-sub-mat nucl:nucleotide.out,aa:VTML80.out -k 0 --k-score<PHONE_NUMBER> --alph-size nucl:5,aa:21 --max-seq-len 65535 --max-seqs 300 --split 0 --split-mode 2 --split-memory-limit 0 -c 1 --cov-mode 1 --comp-bias-corr 1 --diag-score 1 --exact-kmer-matching 0 --mask 1 --mask-lower-case 0 --min-ungapped-score 15 --add-self-matches 0 --spaced-kmer-mode 1 --db-load-mode 0 --pca 1 --pcb 1.5 --threads 8 --compressed 0 -v 3 -s 5.7

Query database size: 1 type: Aminoacid
Estimated memory consumption: 977M
Target database size: 1 type: Aminoacid
Index table k-mer threshold: 112 at k-mer size 6
Index table: counting k-mers
[=================================================================] 100.00% 1 eta -
Index table: Masked residues: 0
Index table: fill
[=================================================================] 100.00% 1 eta -
Index statistics
Entries:          41
DB size:          488 MB
Avg k-mer size:   0.000001
Top 10 k-mers
    DFVKIA	1
    ATLTKA	1
    SEEGLA	1
    IWEEIC	1
    PFDSKD	1
    CLRHLD	1
    DMAGTD	1
    QNRWRE	1
    GENRSE	1
    LAKPDF	1
Time for index table init: 0h 0m 0s 778ms
Process prefiltering step 1 of 1

k-mer similarity threshold: 112
Starting prefiltering scores calculation (step 1 of 1)
Query db start 1 to 1
Target db start 1 to 1
[=================================================================] 100.00% 1 eta -

602.620000 k-mers per position
41 DB matches per sequence
0 overflows
0 queries produce too many hits (truncated result)
1 sequences passed prefiltering per query sequence
1 median result list length
0 sequences with 0 size result lists
Time for merging to pref_0: 0h 0m 0s 0ms
Time for processing: 0h 0m 1s 740ms
align tmp/15248047069833069253/query tmp/15248047069833069253/target tmp/15248047069833069253/search_tmp/6923777973734096903/pref_0 tmp/15248047069833069253/result --sub-mat nucl:nucleotide.out,aa:blosum62.out -a 1 --alignment-mode 3 --alignment-output-mode 0 --wrapped-scoring 0 -e 0.001 --min-seq-id 0 --min-aln-len 0 --seq-id-mode 0 --alt-ali 0 -c 1 --cov-mode 1 --max-seq-len 65535 --comp-bias-corr 1 --max-rejected<PHONE_NUMBER> --max-accept<PHONE_NUMBER> --add-self-matches 0 --db-load-mode 0 --pca 1 --pcb 1.5 --score-bias 0 --realign 0 --realign-score-bias -0.2 --realign-max-seqs<PHONE_NUMBER> --gap-open nucl:5,aa:11 --gap-extend nucl:2,aa:1 --zdrop 40 --threads 8 --compressed 0 -v 3

Compute score, coverage and sequence identity
Query database size: 1 type: Aminoacid
Target database size: 1 type: Aminoacid
Calculation of alignments
[=================================================================] 100.00% 1 eta -
Time for merging to result: 0h 0m 0s 0ms
1 alignments calculated
1 sequence pairs passed the thresholds (1.000000 of overall calculated)
1.000000 hits per query sequence
Time for processing: 0h 0m 0s 28ms
rmdb tmp/15248047069833069253/search_tmp/6923777973734096903/pref_0 -v 3

Time for processing: 0h 0m 0s 1ms
rmdb tmp/15248047069833069253/search_tmp/6923777973734096903/aln_0 -v 3

Time for processing: 0h 0m 0s 1ms
rmdb tmp/15248047069833069253/search_tmp/6923777973734096903/input_0 -v 3

Time for processing: 0h 0m 0s 1ms
rmdb tmp/15248047069833069253/search_tmp/6923777973734096903/aln_merge -v 3

Time for processing: 0h 0m 0s 1ms
summarizeresult tmp/15248047069833069253/result tmp/15248047069833069253/result_best -a 1 --overlap 0 -c 1 --threads 8 --compressed 0 -v 3

[=================================================================] 100.00% 1 eta -
Time for merging to result_best: 0h 0m 0s 1ms
Time for processing: 0h 0m 0s 17ms
convertalis tmp/15248047069833069253/query tmp/15248047069833069253/target tmp/15248047069833069253/result_best result.tsv --sub-mat nucl:nucleotide.out,aa:blosum62.out --format-mode 0 --format-output query,target,fident,alnlen,mismatch,gapopen,qstart,qend,tstart,tend,evalue,bits --translation-table 1 --gap-open nucl:5,aa:11 --gap-extend nucl:2,aa:1 --db-output 0 --db-load-mode 0 --search-type 0 --threads 8 --compressed 0 -v 3

[=================================================================] 100.00% 1 eta -
Time for merging to result.tsv: 0h 0m 0s 1ms
Time for processing: 0h 0m 0s 11ms
rmdb tmp/15248047069833069253/result_best -v 3

Time for processing: 0h 0m 0s 2ms
rmdb tmp/15248047069833069253/result -v 3

Time for processing: 0h 0m 0s 1ms
rmdb tmp/15248047069833069253/target -v 3

Time for processing: 0h 0m 0s 1ms
rmdb tmp/15248047069833069253/target_h -v 3

Time for processing: 0h 0m 0s 1ms
rmdb tmp/15248047069833069253/query -v 3

Time for processing: 0h 0m 0s 2ms
rmdb tmp/15248047069833069253/query_h -v 3

Context
See above.
Your Environment
Include as many relevant details about the environment you experienced the bug in.

Git commit used (The string after "MMseqs Version:" when you execute MMseqs without any parameters):
Which MMseqs version was used (Statically-compiled, self-compiled, Homebrew, etc.): 13.45111
For self-compiled and Homebrew: Compiler and Cmake versions used and their invocation:
Server specifications (especially CPU support for AVX2/SSE and amount of system memory):
Operating system and version:

Thanks a lot! Makes sense, hits that match the coverage threshold should not be rejected.

You can download the precompiled static binary that contains the fix (and latest git changes) at https://mmseqs.com/latest in about an hour when all tests pass through successfully.

Thank you so much for the incredibly fast fix!

@milot-mirdita out of curiosity, any plans for a release in the next while?


Segmentation fault when running tensorflow.python.ops.gen_sparse_ops.sparse_fill_empty_rows

Click to expand!
Issue Type
Bug
Have you reproduced the bug with TF nightly?
No
Source
binary
Tensorflow Version
2.11.0
Custom Code
Yes
OS Platform and Distribution
Ubuntu 22.04
Mobile device
No response
Python version
3.9
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
Segfault due to Large or empty tensor.

Standalone code to reproduce the issue
import tensorflow as tf
import numpy as np
from tensorflow.python.ops import gen_sparse_ops
try:
  try:
    with tf.device('/CPU'):
      indices_tensor = tf.constant(-8647818616220, shape=[6, 0], dtype=tf.int64,)
      indices = tf.identity(indices_tensor)
      values_tensor = tf.random.uniform([6], minval=-256, maxval=257, dtype=tf.int32)
      values = tf.identity(values_tensor)
      dense_shape_tensor = tf.random.uniform([2], minval=-256, maxval=257, dtype=tf.int64)
      dense_shape = tf.identity(dense_shape_tensor)
      default_value_tensor = tf.random.uniform([], minval=-256, maxval=257, dtype=tf.int32)
      default_value = tf.identity(default_value_tensor)
      out = gen_sparse_ops.sparse_fill_empty_rows(indices=indices,values=values,dense_shape=dense_shape,default_value=default_value,)
  except Exception as e:
    print("Error:"+str(e))
  try:
    with tf.device('/GPU:0'):
      indices = tf.identity(indices_tensor)
      indices = tf.cast(indices, tf.int64)
      values = tf.identity(values_tensor)
      values = tf.cast(values, tf.int32)
      dense_shape = tf.identity(dense_shape_tensor)
      dense_shape = tf.cast(dense_shape, tf.int64)
      default_value = tf.identity(default_value_tensor)
      default_value = tf.cast(default_value, tf.int32)
      gen_sparse_ops.sparse_fill_empty_rows(indices=indices,values=values,dense_shape=dense_shape,default_value=default_value,)
  except Exception as e:
    print("Error:"+str(e))
except Exception as e:
  print("Error:"+str(e))

### Relevant log output

```shell
2023-01-22 04:57:38.410770: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-22 04:57:39.683814: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/lib/:/home/nimashiri/anaconda3/envs/cuda11.2/lib/:/home/nimashiri/anaconda3/envs/cuda11.2/lib/
2023-01-22 04:57:39.684083: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/lib/:/home/nimashiri/anaconda3/envs/cuda11.2/lib/:/home/nimashiri/anaconda3/envs/cuda11.2/lib/
2023-01-22 04:57:39.684094: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-01-22 04:57:40.720528: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 04:57:40.744520: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 04:57:40.744847: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 04:57:40.745340: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-22 04:57:40.745919: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 04:57:40.746106: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 04:57:40.746271: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 04:57:41.596881: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 04:57:41.597155: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 04:57:41.597418: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 04:57:41.597585: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4261 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
Segmentation fault

</details>

Hi, @SuryanarayanaY
Could you please look into this issue ? Thank you!

I have tried the code on a VM with Ubuntu 22.04 OS, executed it multiple times and got Segmentation fault (core dumped) error some times or Invalid argument error other times with random -ve number.
F tensorflow/core/framework/tensor_shape.cc:201] Non-OK-status: InitDims(dim_sizes) status: INVALID_ARGUMENT: Expected shape dimensions to be non-negative, got -60
Aborted (core dumped)

Please refer to below log.
(tf2.11) suryanarayanay@surya-ubuntu-22-04:~$ python 59393.py
2023-01-27 11:09:01.400497: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-27 11:09:01.522686: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-01-27 11:09:02.267038: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-01-27 11:09:02.267139: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-01-27 11:09:02.267154: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2.11.0
2023-01-27 11:09:07.133089: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-27 11:09:08.466948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38235 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:04.0, compute capability: 8.0
2023-01-27 11:09:08.468519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38235 MB memory:  -> device: 1, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
Segmentation fault (core dumped)
(tf2.11) suryanarayanay@surya-ubuntu-22-04:~$ python 59393.py
2023-01-27 11:09:21.109153: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-27 11:09:21.230401: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-01-27 11:09:21.973994: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-01-27 11:09:21.974104: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-01-27 11:09:21.974119: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2.11.0
2023-01-27 11:09:26.837099: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-27 11:09:28.167949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38235 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:04.0, compute capability: 8.0
2023-01-27 11:09:28.169675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38235 MB memory:  -> device: 1, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
2023-01-27 11:09:28.203125: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at sparse_fill_empty_rows_op.cc:253 : INVALID_ARGUMENT: Expected shape dimensions to be non-negative, got -60
Error:{{function_node __wrapped__SparseFillEmptyRows_device_/job:localhost/replica:0/task:0/device:CPU:0}} Expected shape dimensions to be non-negative, got -60 [Op:SparseFillEmptyRows]
2023-01-27 11:09:28.895343: F tensorflow/core/framework/tensor_shape.cc:201] Non-OK-status: InitDims(dim_sizes) status: INVALID_ARGUMENT: Expected shape dimensions to be non-negative, got -60
Aborted (core dumped)
(tf2.11) suryanarayanay@surya-ubuntu-22-04:~$ python 59393.py
2023-01-27 11:09:44.122100: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-27 11:09:44.245698: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-01-27 11:09:45.003299: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-01-27 11:09:45.003407: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-01-27 11:09:45.003422: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2.11.0
2023-01-27 11:09:49.880163: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-27 11:09:51.217684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38235 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:04.0, compute capability: 8.0
2023-01-27 11:09:51.219280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38235 MB memory:  -> device: 1, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
2023-01-27 11:09:51.252687: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at sparse_fill_empty_rows_op.cc:253 : INVALID_ARGUMENT: Expected shape dimensions to be non-negative, got -238
Error:{{function_node __wrapped__SparseFillEmptyRows_device_/job:localhost/replica:0/task:0/device:CPU:0}} Expected shape dimensions to be non-negative, got -238 [Op:SparseFillEmptyRows]
2023-01-27 11:09:51.950059: F tensorflow/core/framework/tensor_shape.cc:201] Non-OK-status: InitDims(dim_sizes) status: INVALID_ARGUMENT: Expected shape dimensions to be non-negative, got -238
Aborted (core dumped)
(tf2.11) suryanarayanay@surya-ubuntu-22-04:~$ python 59393.py
2023-01-27 11:10:00.082756: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-27 11:10:00.205665: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-01-27 11:10:00.952821: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-01-27 11:10:00.952924: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-01-27 11:10:00.952940: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2.11.0
2023-01-27 11:10:05.819320: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-27 11:10:07.152613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38235 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:04.0, compute capability: 8.0
2023-01-27 11:10:07.154214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38235 MB memory:  -> device: 1, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
Segmentation fault (core dumped)
(tf2.11) suryanarayanay@surya-ubuntu-22-04:~$

@nimashiri, Thanks for reporting this. It may needs investigation.

@learning-to-play

Hi, This issue has been addressed to check the shape of indices matches the shape of dense_shape in sparse_fill_empty_rows op.
The changes made in the file below and the test file added here.
https://github.com/tensorflow/tensorflow/blob/a6866df48e1aba0996bd99565fa76e79e1875915/tensorflow/core/kernels/sparse_fill_empty_rows_op.cc#L241-L247
Attached the gist here with the expected error message. Thanks!


fix: allow autosharding nodes to get peers

Problem

Solution

Fixes some failing tests in #1756 and stops throwing errors if node is setup to use autosharding
Notes

Resolves
Related to

@danisharora099 added test cases for getPeers with js-waku configured for autosharding