My first impressions

Just tried out the demo on the site and wanted to give a bit of feedback with maybe bugs / features included.
First like the idea - it's a novel way of collecting information. However, it's realistically unusable in it's present form:

features:

need to be able to drag page / chapter order around around, maybe the Chapter / Page could have a property int Order that's mutable. There are tons of jquery plugins that allow things to be sortable. on each "drop" you'd need to update the order of the chapter/page thta's been dragged.
having the order on the right in bright colours is distracting, please the right-side menu it foldable / minimisable at least when creating content
why such big lettering on book creation - it looks unprofessional.
what about setting a cover image when declaring a book as well,
meta data like author/s, keywords/tags would be appreciated for better search and categorising, and an icon maybe for when listing the books,
we need "subchapters" - online / pdf books are not real books, and it's possible for chapters to need grouping of sub-related data - this is urgent! - so Chapter can have a property SubChapters.
i see no theme in the settings,
please don't choose those slidey checkboxes over a normal checkbox - it scares users.
please present the chapters and pages in a smaller font, it can get unruly for big books beyond a few pages you make for testing,
new pages / chapters are more common than export, so move the export link to the far right, not the far left as the hand has to travel further to click it.
also when entering a book,  should then move focus to the save button, not the cancel button (you can then 'press it' with the space bar so it allows you to create content without touching the mouse.)
search by author / category would be nice once the metadata is implemented. all relatively simple features on top of what you have! opening attachments on page breaks the toolbox for wysiwyg

bugs:

clicking on cancel in the edit book redirectss to translation.js,
exporting to pdf is nasty - code of long lines  aren't wrapped and vanish - i assume an issue of whatever pdf converter you're using.

looking forward to see what this grows into. I'm not too confident with laravel, but it'd make most of the above changes fairly simple from what I remember.

Note: not affiliated with Bookstack development - just a user of Bookstack.

need to be able to drag page / chapter order around around, maybe the Chapter / Page could have a property int Order that's mutable. There are tons of jquery plugins that allow things to be sortable. on each "drop" you'd need to update the order of the chapter/page thta's been dragged.

Page and chapter order can be reordered from the "Book" page. The sort option is hidden in the 3-dot menu on the right hand side.

i see no theme in the settings,

Currently there are no additional themes. There is the option to implement your own style via authoring a custom stylesheet and injecting it into the head of the page via the "custom html head" section in the settings. This would also allow you to fix your text size complaints. If you wanted per-user themes or toggle-able themes this could be a valid feature request.
That being said, you should probably break out your feature requests into individual issues so they are manageable and trackable.

clicking on cancel in the edit book redirectss to translation.js,

Clicking on cancel in the edit book section does not redirect me to translation.js using the hosted demo. What browser are you using and what version is it?

hi, thanks for the feedback. using firefox 52 windows.

Hi @ChangePlaces, Thanks for your feedback. Sorry to hear it's unusable for you at the moment.
In regards to the 'cancel' button, It was set up to direct you back to the last page you were on. I've updated this behaviour for the next release to go to the correct page.
Thanks @domainzero for your answers above, All the information there is good so I won't go over the same points.

having the order on the right in bright colours is distracting, please the right-side menu it foldable / minimisable at least when creating content

I have some plans to renovate the sidebar a little as its feature set is growing and will grow in the future. I agree that making it collapsible would be nice for those who want a clean reading experience.

why such big lettering on book creation - it looks unprofessional.

A matter of opinion but I understand what you mean. Really it was to best utilise the space and to emulate the importance of writing the title of a new book. I'll consider this when I come to the next design update.

what about setting a cover image when declaring a book as well

This could lead to views becoming messy. The use of cover images is great in an ideal world where everyone would upload consistent and correct art but the world is not ideal. This can be discussed further under #181.

meta data like author/s, keywords/tags would be appreciated for better search and categorising, and an icon maybe for when listing the books,

search by author / category would be nice once the metadata is implemented. all relatively simple features on top of what you have!

I agree. I plan to open up the ability to search on database columns as part of #64 and #271.

we need "subchapters" - online / pdf books are not real books, and it's possible for chapters to need grouping of sub-related data - this is urgent! - so Chapter can have a property SubChapters.

I'm aware BookStack books are not real books. The system and analogy used is to make the structure easily and instantly relate-able to users. This can be discussed further in #95.

please don't choose those slidey checkboxes over a normal checkbox - it scares users.

[citation needed]
To me they are good at representing binary options. When there's multiple related options I use check boxes instead.

please present the chapters and pages in a smaller font, it can get unruly for big books beyond a few pages you make for testing,

I don't only use BookStack for testing. You can override font sizes via custom CSS as @domainzero has described.

new pages / chapters are more common than export, so move the export link to the far right, not the far left as the hand has to travel further to click it.

That's going to depend on how people use BookStack but that area does need a little cleanup after the exports options were added recently so I'll have a look when I'm next editing around that area.

also when entering a book, should then move focus to the save button, not the cancel button (you can then 'press it' with the space bar so it allows you to create content without touching the mouse.)

Yeah, That's a good idea. I think all the tab-focus paths need a check over and also the focus css states.

opening attachments on page breaks the toolbox for wysiwyg.

Could you confirm the steps to replicate this? On Firefox/Ubuntu I could not get this to happen.

exporting to pdf is nasty - code of long lines aren't wrapped and vanish - i assume an issue of whatever pdf converter you're using.

Yeah, clean PDF generation from HTML appears to be a difficult task. There are two different PDF rendering libraries supported by BookStack so switching to the WKHTML engine may prove better results:
https://www.bookstackapp.com/docs/admin/pdf-rendering/

Since this is a long list I'm going to close this issue for now since it's a bit unmanageable from a issue tracking point of view. Please refer to the linked issues for discussion on individual points.


Installation instructions consolidation

Description
Different installation pages were out of date or out of sync. The install page was over-complicated. The build from source instructions weren't really working.
This PR consolidates install info, directs users to where the more recent updates happen to be, and simplifies the basic user install. I started off just trying to fix the C++ setup info, but found so many other issues, I worked on those too.
Features

Fixes a bug that made it so you couldn't preview the dev version's install page without it redirecting you. (options.js)
Updates the build_from_source page

Point to the OS-focused pages where this information is most up-to-date.
Added a table that maps links to each binding.
Leaves the centos/other linux sections there since this info isn't covered elsewhere
Leaves the NCCL section alone (is this even current info?)
Links this page in the Download Source section found at the bottom of install

Creates a new C++ setup page that brings together C++ info found around the project.
Refactors the install page.

Moves the validation section to a new page
Removes virtualenv (this was repeated multiple times in the file w/o much real value
Super simplifies the pip install sections
Links back to each OS's detailed setup guides
Links to specific install guide sections for Perl and Julia
and more... the file went from 2531 lines to 1062. So much easier to deal with... probably could take more pruning...

Fixes some issues on the OS guide pages - the Python binding steps were missing

Preview
http://<IP_ADDRESS>/versions/cplusplus_install/install/index.html

@lebeg thanks for the review! I've address most of you comments. Please take a look at the build from source page again since I made a lot of changes after incorporating your cmake documentation on BLAS. That was all really good stuff to add here!
A big update would be to switch using cmake as the default and change all of the examples, but I think we should hold off until your cmake PR is merged, right?

I could also add this table to the pip footer...

A big update would be to switch using cmake as the default and change all of the examples, but I think we should hold off until your cmake PR is merged, right?

Yes, absolutely.

This PR also fixes #10020

This PR broke the installation guide nightly test: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTests/detail/master/79/pipeline

I'll take a look. I removed the virtualenv section so it should be an easy fix.


Fix coordinate space translation bug

Translation between room coordinate space and GUI/device coordinates resulted in incorrect values if the current view was rotated and not at the origin of the room.

Thank you for this fix!


Fix Exception If package.json Has No Scripts

Yep. :sparkle:  Thank you :cherries: :smile:

You're welcome!


Sylwia

👷 Deploy Preview for adoring-leavitt-e31a11 processing.

Name
Link

🔨 Latest commit
a2fc74273e5f3a3b5884c86b17e9de23b07423e9

🔍 Latest deploy log
https://app.netlify.com/sites/adoring-leavitt-e31a11/deploys/628778758c48d20008c92f71


Converting Types containing Sets or Maps

I'm trying to use this wonderful library to convert a data structure containing a set of my custom data type as follows:
data Group =   Group     { _liberties :: S.Set CustomType     , _members :: S.Set CustomType     }   deriving (Show, Eq, Generic, ToJSON, SOP.Generic, SOP.HasDatatypeInfo)
When writing the instance declarations for HasElmType, I get an error stating that there is no instance for HasElmType (S.Set CustomType).
I tried to define this myself, but found that these instances rely on SOP.Generic and Generic instances which I could not derive for a Set or a Map (using StandaloneDerivations) as all the required constructors are not exported.
This isn't a bug report, just a humble request for guidance. Is there a way to convert the above custom data type to elm code? I love the idea of this library and have it working for my other types, awesome work!

And I just saw the pull request. Closing this, sorry!

No problem! If you were able to find a solution that you liked, would you like to contribute it to the readme?


@serialport/bindings are not properly rebuild

Summary of Problem
I was looking for options on using serialport in a more "sync" approach, and found #1758  (which has been closed) and wanted to try out the bindings examples posted by @HipsterBrown.
Installed latest @serialport/bindings and had electron-rebuild rebuild it for my Electron Node version, but once I open Electron I get these errors:
App threw an error during load
Error: The module<EMAIL_ADDRESS>was compiled against a different Node.js version using
NODE_MODULE_VERSION 72. This version of Node.js requires
NODE_MODULE_VERSION 80. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
    at process.func [as dlopen] (electron/js2c/asar.js:140:31)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1034:18)
    at Object.func [as .node] (electron/js2c/asar.js:140:31)
    at Module.load (internal/modules/cjs/loader.js:815:32)
    at Module._load (internal/modules/cjs/loader.js:727:14)
    at Function.Module._load (electron/js2c/asar.js:738:28)
    at Module.require (internal/modules/cjs/loader.js:852:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at bindings (D:\electron\node_modules\bindings\bindings.js:112:48)
    at Object.<anonymous> (D:\electron\node_modules\@serialport\bindings\lib\win32.js:1:36)

Btw, rebuilding serialport itself via electron-rebuild works just fine.
Code to Reproduce the Issue
package.json
{
    "name": "electron",
    "version": "1.0.0",
    "description": "",
    "main": "main.js",
    "scripts": {
        "start": "electron .",
        "install": "electron-rebuild",
        "test": "echo \"Error: no test specified\" && exit 1"
    },
    "author": "",
    "license": "ISC",
    "devDependencies": {
        "serialport": "^9.0.0",
        "@serialport/bindings": "^9.0.0",
        "electron": "9.0.4",
        "electron-devtools-installer": "^3.0.0",
        "electron-rebuild": "^1.11.0"
    }
}

main.js
const {app, BrowserWindow} = require('electron');

const Binding = require('@serialport/bindings');

function createWindow()
{
    // Create the browser window.
    let win = new BrowserWindow({
        width: 800,
        height: 600,
        webPreferences: {
            nodeIntegration: true
        }
    });

    win.webContents.openDevTools({});

// and load the index.html of the app.
    win.loadFile('index.html');
}

app.whenReady().then(createWindow);

run Electron
npm start
Versions, Operating System and Hardware
"serialport": "^9.0.0",
"@serialport/bindings": "^9.0.0",
"electron": "9.0.4",
"electron-rebuild": "^1.11.0"
Node 12.18.0 x64
Win 10 x64

@petef19 I also have a similar issue https://github.com/serialport/node-serialport/issues/2105.
It works with "electron rebuild" and "electron ." command, but does not work when I use electron-builder for packaging the app.
Let me know, if you manage to find a solution.

@petef19 I also have a similar issue #2105.
It works with "electron rebuild" and "electron ." command, but does not work when I use electron-builder for packaging the app.
Let me know, if you manage to find a solution.

@SaikatCool: my problem is w/ rebuilding @serialport/bindings package, not the general serialport package as that rebuilds just fine here using electron-rebuild.
So the commands you posted are for @serialport/bindings package ? Both of them don't work but I did not install the packages globally, just as dev dependencies...
Lmk how to execute these and I'll give it a try...

@petef19  got the same problem. any news on that?
for me it seams that the package is also not existing within the release folder.

@petef19  got the same problem - any update on that.
For mit it seams that in the ticket #2105 the whole v80 is missing.

I downgraded Electron 9.0.0 to 8.2.5 to solve it.
The ABI for Electron 9.0.0 is not yet released by serialport.


webrtcd: lazy import of aiortc

Did we figure out specifically what's taking so long?

aiortc took 60% of the import time, and its submodules import some bigger dependencies like google_crc32, aioice, PyAV and cryptography. Another 20% was taken by aiohttp.web

Import time of webrtcd and webjoystick reduced by 50%

I noticed recently while running fw_versions.py locally, that the startup time was around 5s, mostly spent in imports. Wonder if this is a regression, will check it out!


Help Menu exists but does not do anything

The help menu in  version 2.0 does just shows this message:

Have it go to the github site or remove the menu. I like the option to go to the github site.

Fixed in #8, the Help menu now redirects to the GitHub project.


Potted plant occasionally prevents malware usage

The carnivorous potted plant is often equipped when adventuring in the daily dungeon and will occasionally kill the monster there before malware can be used, preventing 3 avatar keys from being acquired.

Thanks for the report, the script should avoid equipping the carn plant when using the malware now.


DeploymentAwaiter fails to update in a project with serverless containers enabled

How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
v20.13.1
Amplify CLI Version
12.12.4
What operating system are you using?
macOS Sonoma 14.5
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
No manual changes made
Describe the bug
In a project with Serverless containers enabled, updating a backend deployed with Amplify CLI v12.12.3 or earlier to Amplify CLI v12.12.4 and then running amplify push results in a failure to update DeploymentAwaiter.
Context:

DeploymentAwaiter is deployed using a CloudFormation Custom Resource.
Previously ran on Node 16, changed to Node 18 in #2529.
Node 18 runtime image does not include aws-sdk v2, requiring updated source code aws-amplify/amplify-cli#13757.

Issue:

The runtime is updated to Node 18, but the Lambda source code is not updated during amplify push, causing the following error:

INIT_START Runtime Version: nodejs:18.v31	Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
2024-07-30T12:16:16.780Z	undefined	ERROR	Uncaught Exception 	{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module 'aws-sdk'\nRequire stack:\n- /var/task/outbound.js\n- /var/task/cfn-response.js\n- /var/task/framework.js\n- /var/runtime/index.mjs",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module 'aws-sdk'",
        "Require stack:",
        "- /var/task/outbound.js",
        "- /var/task/cfn-response.js",
        "- /var/task/framework.js",
        "- /var/runtime/index.mjs",
        "    at _loadUserApp (file:///var/runtime/index.mjs:1087:17)",
        "    at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1119:21)",
        "    at async start (file:///var/runtime/index.mjs:1282:23)",
        "    at async file:///var/runtime/index.mjs:1288:1"
    ]
}

Reviewing the CloudFormation Template reveals the following entry:

File: amplify/backend/api/containera76cba34/containera76cba34-cloudformation-template.json

"AwaiterMyProviderframeworkonEventB703D418": {
  "Type": "AWS::Lambda::Function",
  "Properties": {
    "Code": {
      "S3Bucket": {
        "Ref": "deploymentBucketName"
      },
      "S3Key": {
        "Ref": "awaiterS3Key"
      }
    },
    "Description": "AWS CDK resource provider framework - onEvent (ContainersStack/AwaiterMyProvider)",

Code: https://github.com/aws-amplify/amplify-category-api/blob/2e75a79816da2f5c928bdaa900f9385b96fd5581/packages/amplify-category-api/src/provider-utils/awscloudformation/base-api-stack.ts#L542-L545

The S3Key is fixed, which likely causes the source code not to be updated.

Changes to a deployment package in Amazon S3 or a container image in ECR are not detected automatically during stack updates. To update the function code, change the object key or version in the template.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-code.html
The source code custom-resource-pipeline-awaiter.zip uploaded to the deployment bucket has been confirmed to be updated. However, the code on the lambda was confirmed to be NOT updated.
Expected behavior
The DeploymentAwaiter source code should be updated to the latest version, and CloudFormation should complete successfully.
Reproduction steps

Create an Amplify app with Amplify CLI v12.12.3 and enable Serverless containers.
Run amplify push to update the backend.
Update Amplify CLI to v12.12.4.
Run amplify push to update the backend.
An error occurs.

Project Identifier
Project Identifier: 52a166481f025be404045fcd549de961
Log output
No response
Additional information
No response
Before submitting, please confirm:

[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
[X] I have removed any sensitive information from my code snippets and submission.

Hey @ashphy, Thank you for raising this issue. I am working on reproducing it and will keep you updated on the outcome.

Hey @ashphy, We can reproduce the issue by following the steps mentioned in the description. Hence, marking this as bug for the team to evaluate further. To unblock, could you please downgrade to the previous CLI version until we resolve this issue?

Thank you for your prompt response. As suggested, I will downgrade to the previous version. Please feel free to close the issue once the fix is merged.

We are also affected by the error and would appreciate a fast fix.


haskell.compiler.ghcHEAD: support JS backend

Description of changes
Was able to compile a test jsexe (impurely as the package set doesn't work yet, see the last commit for details)!
To try it out, build nix-build -A pkgsCross.ghcjs.buildPackages.haskell.compiler.ghcHEAD on this branch.
Closes #208879.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

The gtk3 issue will be fixed by #208945.


add bower.json to 'gulp bump' & fix #1415

Offering this if it's helpful.  RE #1465.
Simply put it...

adds './bower.json' to the gulp bump task in gulpfile.js (jump to it)
manually bumps the bower.json version number to match package.json (jump to it)
& also inadvertantly fixes #1415 by checking that ctrl.$animate.on exists before using it (jump to it)

bower issues closed by: #1465 & https://github.com/angular-ui/ui-select/commit/c79f0c8928259b6ad1ec4485bfe25d3945220304
ngAnimate issues closed by: #1456  &  https://github.com/angular-ui/ui-select/commit/96632ff84df8204b5ba443216a758f02e23ef123


Update to 1.19.4, I guess

I tested a bit, did not find a crash. So I guess it works /shrug

Thanks, you're the best!


Mysterious Modal Animation Behavior

I'm having a problem with modals that I've been unable to debug over the past week. I assume it's some idiosyncrasy of my code or my understanding of liquid-fire modals, but I'm wondering if this is something anyone has seen and might have an idea about what might cause it.
I have two separate sections of a site that are both hooking into the same modal, code is almost identical from section to section, and yet mysteriously one works as expected, while the other is produces a strange animation effect – it appears that perhaps the modal starts full size, then shrinks a bit, and then expends again in a sort of accordion motion.
Clicking on a schedule box or show box opens the modal...
http://staging.ckdu.ca/schedule
http://staging.ckdu.ca/shows

I've realized that I only see this behavior when I use query parameters.

I am seeing this on liquid-fire master with ember 1.11 beta 4

I've realized that I only see this behaviour with query parameters present. For some reason which I am as yet unable to detect liquid fire is looping back to modal => observerForParam and then modal => update. This is using liquid-fire master with ember 1.11 beta 4

As it turned out this had nothing to do with liquid fire. The query param that I was using in one section of app was a default model id, while the other was based on a DS.attr('number') in model. When I changed that attribute to a string this problem died. This solved my problem:
http://stackoverflow.com/questions/23823343/ember-observe-returns-callback-twice-when-used-with-query-params

I wonder if this is something that would be worth noting as a possible gotcha in the liquid-fire documentation, or if liquid fire should/could care about this conversion? I was not aware of this queryParam conversion and it strikes me as something that others may also stumble on?


Instance assigned to constant inside class definition not initialized until/unless accessed

Code
class Foo
  FOOS = {} of Int32 => self

  def initialize(@id : Int32)
    FOOS[@id] = self
  end

  FOO_1 = Foo.new(1)
  FOO_2 = Foo.new(2)
end

Example 1
Foo::FOOS # => {}

Example 2
Foo::FOO_1 #=> #<Foo:0xfc6fd0 @id=1>
Foo::FOOS # => {1 => #<Foo:0xfc6fd0 @id=1>}

Example 3
Foo::FOOS # => {1 => #<Foo:0xfc6fd0 @id=1>}
Foo::FOO_1 #=> #<Foo:0xfc6fd0 @id=1>

This is, at least now, expected behaviour. Constants are only initialized if they are used.
This for example lets us avoid initializing ARFG, STDIN, STDERR, and basically any constant if it's not used.
Is this something that we want to continue having? I don't know, maybe it's confusing. At least it's not similar to Ruby... On the other hand, Swift works like that too, so it means that at least another language thinks this is good.

My use case is something like this:
class Foo
  FOOS = {} of Int32 => self

  private def initialize(@id : Int32)
    FOOS[@id] = self
  end

  FOO_1 = Foo.new(1)
  FOO_2 = Foo.new(2)

  def self.[](id : Int32)
    FOOS[id]
  end
end

Foo[1] # missing hash key

If it's set in stone I'll leave a dirty workaround here in case somebody else encounters this issue.
class Foo
  FOOS = {} of Int32 => self

  def initialize(@id : Int32)
    FOOS[@id] = self
  end

  FOO_1 = Foo.new(1)
  FOO_2 = Foo.new(2)

  def self.[](id : Int32)
    FOOS[id]
  end

  {% for constant in @type.constants %}
    {{constant}}
  {% end %}
end

Foo[1] # works

@Exilor If I'm not mistaken you could also use @@class variable for that.

@Sija I also need access to Foo_1, Foo_2... directly by name (Foo::Foo_1), that's why they are constants :stuck_out_tongue_winking_eye:

@Exilor I don't understand exactly your use case - that is, why do you have to use constants for that? why not use for instance Foo[:FOO1]?

@Sija I'm porting Java code that uses Java's enums (which can be of any type unlike Crystal's) and also has a class method to retrieve a specific enum instance by one of its fields. I have to use constants to access the instances by their name and that's surely faster than doing a hash lookup.
I have already circumvented this issue and my use case is solved, but if you're that interested... :smile:  here's the Java original and my Crystal port.

@Exilor since you've found the solution I go find some grandma next to pedestrian crossing :P thanks for explanation tho', your Crystal port looks so much cleaner (Java is good as a coffee...)


Ubuntu 20.04.6 LTS runs scrcpy，The mobile adb connected to Pixel4 automatically disconnects！

Environment

OS: [Debian]
scrcpy version: [2.1.1]
installation method: [git install.sh]
device model:Pixel 4
Android version: [13]

Describe the bug
scrcpy 2.1.1 https://github.com/Genymobile/scrcpy
INFO: ADB device found:
INFO:     -->   (usb)  99161FFAZ007M6                  device  AOSP_on_flame
/usr/local/share/scrcpy/scrcpy-server: 1 file pushed. 14.7 MB/s (56995 bytes in 0.004s)
[server] INFO: Device: [Google] Android AOSP on flame (Android 13)
INFO: Renderer: opengl
INFO: OpenGL version: 3.1 Mesa 21.2.6
INFO: Trilinear filtering enabled
INFO: Texture: 1080x2280
ERROR: Demuxer 'audio': stream disabled due to connection error
ERROR: Demuxer error
On errors, please provide the output of the console (and adb logcat if relevant).
2023-07-18 17:19:13.985 0-0/? E/: c2   4535 power_supply main: driver failed to report `flash_trigger' property: -1
2023-07-18 17:19:22.984 1082-4738/? E/QC2V4l2Driver: [avcE_20] failed to set buffer size limit to 4
2023-07-18 17:19:22.991 1082-4738/? E/QC2V4l2Codec: [avcE_20]  OPAQUE is not a supported pixel format!
2023-07-18 17:19:22.991 1082-4738/? E/QC2Comp: [avcE_20] configureCodec: Failed to set updated param Input::PixelFormat
2023-07-18 17:19:22.992 1082-1482/? E/QC2Interface: Few parameters failed to configure
2023-07-18 17:19:23.008 1082-4738/? E/QC2C2DEngine: calcYSize: unsupported color format, RGBA8888(1)
2023-07-18 17:19:23.019 1082-4738/? E/ECOSession: Failed to create ECOSession with w: 1080, h: 2280, isCameraRecording: 1
2023-07-18 17:19:23.019 1082-4738/? E/ECOService: ECOService failed to create ECOSession w: 1080, h: 2280, isCameraRecording: 1
2023-07-18 17:19:25.983 1731-2926/? E/AppOps: Attribution not found: uid=2000 pkg=com.android.shell(null) op=RECORD_AUDIO
2023-07-18 17:19:25.984 1310-2556/? E/mediaserver: unlinkToDeath: removed reference to death recipient but unlink failed.
2023-07-18 17:19:25.984 1310-2556/? E/IPCThreadState: attemptIncStrongHandle(37): Not supported
2023-07-18 17:19:25.991 1731-4771/? E/AppOps: Operation not started: uid=2000 pkg=com.android.shell(null) op=RECORD_AUDIO_OUTPUT
2023-07-18 17:19:26.878 0-0/? E/: c1   1065 iaxxx-spi spi5.0: Timeout for wakeup event rc :0 wake flag :0
2023-07-18 17:19:27.410 0-0/? E/: c3   4533 power_supply main: driver failed to report `flash_trigger' property: -1
2023-07-18 17:19:27.618 1038-3256/? E/audio_hw_utils: send_app_type_cfg_for_device: Could not get ctl for mixer cmd - Audio Stream Capture 42 App Type Cfg
2023-07-18 17:19:27.631 0-0/? E/: c0   3256 send_afe_cal_type cal_block not found!!

But everything is good at the same environment on Mac！I don't know why?

But when I used XiaoMi device,it won't happen!  XiaoMi can show screen on Ubuntu System~

It's specific to your device (and "quickstep", whatever it is).


BE | Blog Listing

Link to Ticket
Description of Changes Made
This is the backend counterpart to #40 and primarily consists of the following changes:

removal of the intro field from BlogIndexPage
removeal of the related_links reverse relation for BlogIndexPage
removal of the get_popular_tags method on BlogIndexPage
updating the blog_posts context variable by adding extra keys to each item
refactoring of the pagination mechanism (we're no longer using AJAX) and also account for query params in pagination
cleanup of the BlogPage model by removing unused reverse relation: blog_page_tag_select (this ideally should have been done as part of #35)

In addition, the pagination template has been copied from #40 and updated with the logic required to replicate the design:

Note
To avoid duplication of work and potential conflict with #40, no further changes have been made to the templates. However, I did test the above changes locally, by modifying the following templates (which are either removed/updated in #40)

tbx/project_styleguide/templates/patterns/molecules/blog-item/blog-item.html
{% load wagtailcore_tags wagtailimages_tags %}

{{ item.title }}

    {% if item.intro %}

{{ item.intro }}

    {% endif %}

    {% if item.description %}

{{ item.description|richtext }}

    {% endif %}

    {% if item.author %}
        {% image item.author.image fill-100x100 format-webp %}
        {{ item.author.name }}
        {{ item.author.role }}
        {% if item.event_type %}
            {{ item.event_type }}
        {% endif %}
        {% if item.date %}
            {{ item.date }}
        {% endif %}
    {% endif %}

        {% if item.read_time %}
            {{ item.read_time }} min read
        {% endif %}

        {% for tag in item.related_services %}
-            {% include "patterns/atoms/tag/tag.html" with item=tag small=True tag_link_base=blog_index_url %}
+            {% include "patterns/atoms/tag/tag.html" with item=tag tag_slug=tag.slug tag_name=tag.name small=True tag_link_base=blog_index_url %}
        {% endfor %}

tbx/project_styleguide/templates/patterns/molecules/title-block/title-block.html
{% load wagtailcore_tags %}
{% if item.client %}

{{ item.client }}

{% endif %}

{% firstof title|richtext item.title %}

{% if meta %}
    {{ meta }}
{% endif %}
{% if tags %}

Filter blog posts by category

    {% pageurl item as index %}
    {% if request.GET.filter == '' or not request.GET.filter %}
        {% include "patterns/atoms/tag/tag.html" with tag_link_base=index has_all_filter=True item=None is_active=True %}
    {% else %}
        {% include "patterns/atoms/tag/tag.html" with tag_link_base=index has_all_filter=True item=None %}
    {% endif %}

    {% for tag in tags %}
        {% if request.GET.filter == tag.slug %}
-            {% include "patterns/atoms/tag/tag.html" with item=tag is_active=True %}
+            {% include "patterns/atoms/tag/tag.html" with tag_slug=tag.slug tag_name=tag.name is_active=True %}
        {% else %}
-            {% include "patterns/atoms/tag/tag.html" with item=tag %}
+            {% include "patterns/atoms/tag/tag.html" with tag_slug=tag.slug tag_name=tag.name %}
        {% endif %}
    {% endfor %}
{% endif %}

tbx/project_styleguide/templates/patterns/pages/blog/blog_listing.html
{% extends "patterns/base_page.html" %}
{% load wagtailcore_tags wagtailimages_tags static %}

{% block content %}

    {% include "patterns/molecules/title-block/title-block.html" with item=page tags=related_services %}

-

        {% include "patterns/organisms/blog-listing/blog-listing.html" with blog_posts=blog_posts %}

+

        {% include "patterns/molecules/pagination/pagination.html" with pages=blog_posts %}

-    {% if blog_posts.has_next %}
-        {% include "patterns/atoms/see-more/see-more.html" with text="See more posts" fetch_data="blog-pages" %}
-    {% endif %}
-
-    {% include_block page.call_to_action %}

{% endblock %}

How to Test
Got to http://localhost:8000/blog/.
Please refer also to the above note regarding small modifications to some templates
Screenshots

Expand to see more
https://github.com/torchbox/torchbox.com/assets/7713776/b8d5f529-f571-4931-a1bc-6deb0c5bceb5

MR Checklist

[x] Add a description of your pull request and instructions for the reviewer to verify your work.
[x] If your pull request is for a specific ticket, link to it in the description.
[x] Stay on point and keep it small so the merge request can be easily reviewed.
[x] Tests and linting passes.

Unit tests

[ ] Added
[x] Not required

Documentation

[ ] Updated build docs
[ ] Updated editor guidelines (https://docs.google.com/document/d/1PAWccdQ4tfaZsrEWmpDhvP3GH5RRmBOARFVp4b-kje8/edit?usp=sharing)
[x] Not required

Browser testing

[x] I have tested in the following browsers and environments (edit the list as required)

Latest version of Chrome on Linux (Ubuntu)
Latest version of Firefox on Linux (Ubuntu)

Data protection

[x] Not relevant
[ ] This adds new sources of PII and documents it and modifies Birdbath processors accordingly

Accessibility

[ ] Automated WCAG 2.1 tests pass
[ ] Manual WCAG 2.1 tests completed
[ ] I have tested in a screen reader
[ ] I have tested in high-contrast mode
[ ] Any animations removed for prefers-reduced-motion
[x] Not required

Sustainability

[ ] Images are optimised and lazy-loading used where appropriate
[ ] SVGs have been optimised
[ ] Perfomance and transfer of data considered
[ ] If JavaScript is needed alternatives have been considered
[ ] New font variants have not been added
[x] Not required

Pattern library

[ ] The pattern library component for this template displays correctly, and does not break parent templates
[ ] The styleguide is updated if relevant
[x] Changes are not relevant the pattern library

TODO:

rebase onto integration branch
make relevant updates to templates as needed

Closing this PR and creating a new one, as it's easier to just create a new branch & copy in the changes than rebase.

Superseded by #43


Telecommand: LFS delete file (#227)

Resolves #227:
Summary

Included telecommand that uses LFS_delete_file helper function to delete a file from LFS
Accepts 1 argument: The name of the file to be deleted

This all looks good. This just needs to be tested.

I was able to quickly test this using my NAND memory module. Looks good!
✅ Able to successfully delete a file if it exists.
✅ Reports appropriate error when trying to delete a file that doesn't exist.
Here are some things to consider that I found during testing:
⚠ I'm unable to reach the "Error parsing file name arg" scenario when I send an empty string "" as an input. Also to note as a result "" is accepted as a parameter when creating a file. Able to reach case: 3 (filename too long for parsing) successfully.
⚠ Able to send input both with and without "", are both ways acceptable? May be worth including in the arg doc so it is more clear.

This all looks good. This just needs to be tested.

I was able to quickly test this using my NAND memory module. Looks good! ✅ Able to successfully delete a file if it exists. ✅ Reports appropriate error when trying to delete a file that doesn't exist.
Here are some things to consider that I found during testing (while not directly linked to the issue itself but worth getting investigated if possible): ⚠ I'm unable to reach the "Error parsing file name arg" scenario when I send an empty string "" as an input. Also to note as a result "" is accepted as a parameter when creating a file. Able to reach case: 3 (filename too long for parsing) successfully. ⚠ Able to send input both with and without "", are both ways acceptable? May be worth including in the arg doc so it is more clear.

Thank you for testing! I'll take a look into the issues and get them fixed right away! Passing a filename argument of "" should not be acceptable, there should be an error case for an empty argument.

While looking through the code, I found that the "Error parsing file name arg" condition was trivial since passing no arguments to this telecommand prompts a telecommand parsing error. However, I will keep the statement in the code just to be safe.
I added a note in the doc string indicating to not add quotations around the argument when parsing fs_delete_file to reduce any confusion.

another good thing to mention is that it might be nice to wait until Saksham's LFS telecommand list files [issue #7]  is merged into main to merge this branch.


5.3.5. DomainParticipantListener Discovery Callbacks lacks info on including StatusMask::none() for not sabotaging subscribers

The issue described https://github.com/eProsima/Fast-DDS/issues/2106 seems to originate in improperly configuring DomainParticipantFactory::create_participant(), like described on page https://fast-dds.docs.eprosima.com/en/latest/fastdds/discovery/disc_callbacks.html . I think the example code should add the StatusMask::none() to the create_participant() code.

Hi @ma30002000,
I have issued #280 trying to clarify this issue.
I do not think that the example you mention should be changed. That example only refers to the discovery callbacks that can be used with DomainParticipantListener. That specific example is not related with how the data is notified to the user or by which callback. I have tried to clarify that point in the Entity Listener section, as well as adding links to this section both in this example you mention as in the DomainParticipantListener section so the user is aware of what is being done and can configure the callbacks correctly for its use case. I hope it helps for further reference!


OVER_QUERY_LIMIT error

I have set a Dark Sky API in options, but I receive the error OVER_QUERY_LIMIT when I type anything into the "Default Location" entry. I haven't made any calls yet of the 1000/day allowed by Dark Sky's free API.

I have the same problem. Any workarounds?

Default location gets generated from Google Maps, not through Dark Sky. Look at file geo.go
The problem was that Google Maps now requires an API key to perform calls -- this is what was tripping the OVER_QUERY_LIMIT error.
See: https://maps.googleapis.com/maps/api/geocode/json?address=1600
{
 "error_message" : "You have exceeded your rate-limit for this API.
We recommend registering for a key at the Google Developers Console:
https://console.developers.google.com/apis/credentials
 project=_",
 "results" : [],
 "status" : "OVER_QUERY_LIMIT"
}

I've added that option to plug in the key through the wtr options menu, I hope it works.
Not sure if this project is being maintained though.

Is it working for you now? would you mind sending your new build?

As the Dark Sky API is no more, this issue is no longer relevant & can be closed.


Engine Web Server Wiki

Finish writing the first part of the wiki
Wiki Pages to be written:

[x] Setting up the Raspberry Pi
[x] Installing Raspberry Pi OS
[ ] Setting up SSH and optionally Wi-Fi
[ ] SSHing into the Raspberry Pi
[ ] Installing Updates
[ ] Installing Environments

Setting up the Raspberry Pi done

Installing Raspberry Pi OS done

this is an article on how to connect via ssh using rasberry pi.   https://www.raspberrypi.org/documentation/remote-access/ssh/windows10.md

img directory created in this is the commit. Screen shots and wiki pitures are put there for reference in page creation.

Video made for wiki

Steps for the Setting up SSH and optionally Wi-Fi page:

make ssh file somewhere (not boot on sd card)
make wpa_supplicant.json on boot on sd card
populate wpa_supplicant.json
power your board
locate your boards ip
ssh into your board
credentials for your pi

screen cap converted into gif

new screen cap

[SSHing into the Raspberry](SSHing into the Raspberry Pi) Pi page done.

[SSHing into the Raspberry](SSHing into the Raspberry Pi) Pi page done.

Setting up SSH and Optionally Wi-Fi Done.

Setting up SSH and Optionally Wi-Fi Done.

Installing Environments Done.

Installing Environments Done.

Final page Enabling SSH Server Done.

Final page Enabling SSH Server Done.


Add testing framework

Probably Karma/mocha. See esri-karma-tutorial.

I know the mantra says "test only your code", but does bootstrap-map-js already have tests that aren't in the repository? Is writing tests for that going to be necessary, too?

No, i'm not hiding secret tests. I wish I was.
I created this repo before I had any experience w/ fron-end testing tools, and I've never had time to add them along the way. I'm about to add IE config to esri-karma-tutorial, after which it would be a good starting point.


"Exchange" link at top is a 404

The link is dead.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3b640892-b0a1-4f20-1e25-fe5f0e97f5b6
Version Independent ID: 51774b36-90fa-1d9a-5800-1a54d9f70590
Content: Get-Mailbox (exchange)
Content Source: exchange/exchange-ps/exchange/mailboxes/Get-Mailbox.md
Service: exchange-powershell
GitHub Login: @chrisda
Microsoft Alias: chrisda

@chrisweis, currently, all Exchange cmdlet reference topics are grouped together under a single 'Exchange' moniker. That may or may not change, but we're definitely working on a mechanism that allows you to select an Exchange version (2010, 2013, 2016, Exchange Online, Office 365 Security & Compliance Center, or Exchange Online Protection) that filters the list of cmdlets based on your selection. Once we get that worked out, I'm pretty sure the link would start working.
But, even if the link was working, it won't take you anywhere interesting; you are effectively at the final destination that would be provided by the link.
In the Skype PowerShell cmdlet reference topics, if you click on the corresponding 'Skype' link at the top of a cmdlet ref topic (for example, in https://docs.microsoft.com/powershell/module/skype/add-cssladelegates), it simply takes you to the top of the Skype cmdlet reference node (https://docs.microsoft.com/powershell/module/skype/).
That link is likely a requirement of the repo, but doesn't provide anything interesting in our implementation. Exchange is further complicated by the fact that 500+ cmdlet reference topics are shared between the various flavors of on-premises Exchange Server and the cloud-based services, yet we need to only have one version of the topic for sanity's sake.
Thanks for the feedback.

The Exchange module link issue was fixed this week. Thanks again, for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


Split extension into clients and streams

fixes: https://github.com/creek-service/creek-kafka/issues/31
Reviewer checklist

[ ] Ensure relevant issues are linked (description should include text like "Fixes #")
[ ] Ensure any appropriate documentation has been added or amended

Coverage decreased (-0.2%) to 97.995% when pulling 432fdb910e315acee98509cd29e8288be1c4a52e on client-ext into 9dcbebe8ffd7b94ede63fc745d03cc2437071c7b on main.


Uncaught Error: Cannot read property 'object' of undefinedTypeError: Cannot read property 'objec...

[Enter steps to reproduce:]

...
...

Atom: 1.17.0 x64
Electron: 1.3.15
OS: Unknown Windows version
Thrown From: Atom Core
Stack Trace
Uncaught Error: Cannot read property 'object' of undefined
TypeError: Cannot read property 'object' of undefined
at ObjectsRegistry.get (C:\Users\hschang\AppData\Local\atom\app-1.17.0\resources\electron.asar\browser\objects-registry.js:41:28)
at EventEmitter. (C:\Users\hschang\AppData\Local\atom\app-1.17.0\resources\electron.asar\browser\rpc-server.js:306:31)
at emitMany (events.js:127:13)
at EventEmitter.emit (events.js:201:7)
at WebContents. (C:\Users\hschang\AppData\Local\atom\app-1.17.0\resources\electron.asar\browser\api\web-contents.js:222:13)
at emitTwo (events.js:106:13)
at WebContents.emit (events.js:191:7)
At C:\Users\hschang\AppData\Local\atom\app-1.17.0\resources\electron.asar\renderer\api\remote.js:217

Error: Cannot read property 'object' of undefined
TypeError: Cannot read property 'object' of undefined
    at ObjectsRegistry.get (~/AppData/Local/atom/app-1.17.0/resources/electron.asar/browser/objects-registry.js:41:28)
    at EventEmitter.<anonymous> (~/AppData/Local/atom/app-1.17.0/resources/electron.asar/browser/rpc-server.js:306:31)
    at emitMany (events.js:127:13)
    at EventEmitter.emit (events.js:201:7)
    at WebContents.<anonymous> (~/AppData/Local/atom/app-1.17.0/resources/electron.asar/browser/api/web-contents.js:222:13)
    at emitTwo (events.js:106:13)
    at WebContents.emit (events.js:191:7)
    at metaToValue (~/AppData/Local/atom/app-1.17.0/resources/electron.asar/renderer/api/remote.js:217:13)
    at undefined.remoteMemberFunction (~/AppData/Local/atom/app-1.17.0/resources/electron.asar/renderer/api/remote.js:113:18)
    at AtomEnvironment.module.exports.AtomEnvironment.getWindowDimensions (~/AppData/Local/atom/app-1.17.0/resources/app/src/atom-environment.js:663:1)
    at AtomEnvironment.module.exports.AtomEnvironment.storeWindowDimensions (~/AppData/Local/atom/app-1.17.0/resources/app/src/atom-environment.js:697:1)
    at WindowEventHandler.module.exports.WindowEventHandler.handleWindowBlur (~/AppData/Local/atom/app-1.17.0/resources/app/src/window-event-handler.js:184:1)
    at ~/AppData/Local/atom/app-1.17.0/resources/app/src/window-event-handler.js:3:1

Commands
     -4:24.5.0 core:move-left (input.hidden-input)
     -4:23.5.0 core:copy (input.hidden-input)
     -4:23.1.0 core:move-right (input.hidden-input)
     -4:22.5.0 core:paste (input.hidden-input)
     -4:21.7.0 core:move-left (input.hidden-input)
     -4:21.2.0 editor:delete-to-beginning-of-word (input.hidden-input)
     -4:19.4.0 core:select-up (input.hidden-input)
     -4:18.9.0 editor:toggle-line-comments (input.hidden-input)
     -4:18.2.0 core:undo (input.hidden-input)
     -4:16.8.0 core:move-down (input.hidden-input)
  2x -4:16.4.0 core:move-up (input.hidden-input)
     -4:16 editor:toggle-line-comments (input.hidden-input)
     -4:15.7.0 core:save (input.hidden-input)
     -2:09.6.0 core:move-up (input.hidden-input)
     -2:09.2.0 editor:toggle-line-comments (input.hidden-input)
     -2:08.9.0 core:save (input.hidden-input)

Non-Core Packages
atom-beautify 0.29.24
atom-html-preview 0.1.22
autoclose-html 0.23.0
file-icons 2.1.6
fold-comments 0.6.0
fold-functions 0.5.0
goto-definition 1.3.3
language-awk 0.1.12
merge-conflicts 1.4.5
minimap 4.28.2
minimap-autohide 0.10.1
symbols-tree-view 0.14.0

Thanks for taking the time to contribute!
We noticed that this is a duplicate of https://github.com/atom/atom/issues/11692. You may want to subscribe there for updates.
Because we treat our issues list as the Atom team's backlog, we close duplicates to focus our work and not have to touch the same chunk of code for the same reason multiple times. This is also why we may mark something as duplicate that isn't an exact duplicate but is closely related.
For information on how to use GitHub's search feature to find out if something is a duplicate before filing, see the How Can I Contribute? section of the Atom CONTRIBUTING guide.


License

This project is missing a license :(. Can you add it, so that it can be safely used in open source projects (e.g MIT).

Done.  Thanks for the suggestion.


[Feature][Flink] Flink on kubernetes

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
I hope the next official version can support the scheduling of Flink on kubernetes.
Use case
I hope the next official version can support the scheduling of Flink on kubernetes.
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi @ChenpiDog can you add some detail description of it?

@caishunfeng For the task type of Flink, the dolphinscheduler supports scheduling Flink deployed in local mode or yarn mode, but it does not seem to support scheduling Flink deployed on kubernetes. I can't find the relevant instructions on the official website.

My Flink is deployed with kubernetes, and my ETL is Flink SQL. I expect to use the dolphinscheduler to schedule or manage my Flink SQL.

@caishunfeng For the task type of Flink, the dolphinscheduler supports scheduling Flink deployed in local mode or yarn mode, but it does not seem to support scheduling Flink deployed on kubernetes. I can't find the relevant instructions on the official website.

Now ds has k8s task type in dev, I'm not sure it's what you need, but you can have a try.

I build the dev , flink still uses yarn by default in application mode ,  I find the k8s task , but  cannot be applied to flink  , you can  look at this  application mode   Pod creation, operation, and destruction are all operated by flink itself ,  like this:
./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=my-first-application-cluster \
    -Dkubernetes.container.image=custom-image-name \
    local:///opt/flink/usrlib/my-flink-job.jar

flink  will create  jobmanager, taskmanager, and copy config .........does many things, So ,simply specifying a image for a k8s task cannot solve the problem.  we don't have any path to run a flink task through the k8s tab

@caishunfeng For the task type of Flink, the dolphinscheduler supports scheduling Flink deployed in local mode or yarn mode, but it does not seem to support scheduling Flink deployed on kubernetes. I can't find the relevant instructions on the official website.

Now ds has k8s task type in dev, I'm not sure it's what you need, but you can have a try.

I build the dev , flink still uses yarn by default in application mode , I find the k8s task , but cannot be applied to flink , you can look at this application mode Pod creation, operation, and destruction are all operated by flink itself , like this:
./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=my-first-application-cluster \
    -Dkubernetes.container.image=custom-image-name \
    local:///opt/flink/usrlib/my-flink-job.jar

flink will create jobmanager, taskmanager, and copy config .........does many things, So ,simply specifying a image for a k8s task cannot solve the problem. we don't have any path to run a flink task through the k8s tab

I build the dev , flink still uses yarn by default in application mode , I find the k8s task , but cannot be applied to flink , you can look at this application mode Pod creation, operation, and destruction are all operated by flink itself , like this:
./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=my-first-application-cluster \
    -Dkubernetes.container.image=custom-image-name \
    local:///opt/flink/usrlib/my-flink-job.jar

flink will create jobmanager, taskmanager, and copy config .........does many things, So ,simply specifying a image for a k8s task cannot solve the problem. we don't have any path to run a flink task through the k8s tab

I get your point, they are different way. So it's better to expand flink task to support the k8s deployment modes.
@ikiler Are you interested in expanding flink task? like #9577

I build the dev , flink still uses yarn by default in application mode , I find the k8s task , but cannot be applied to flink , you can look at this application mode Pod creation, operation, and destruction are all operated by flink itself , like this:
./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=my-first-application-cluster \
    -Dkubernetes.container.image=custom-image-name \
    local:///opt/flink/usrlib/my-flink-job.jar

flink will create jobmanager, taskmanager, and copy config .........does many things, So ,simply specifying a image for a k8s task cannot solve the problem. we don't have any path to run a flink task through the k8s tab

I get your point, they are different way. So it's better to expand flink task to support the k8s deployment modes. @ikiler Are you interested in expanding flink task? like #9577

I'd love to if I could, but I have no good idea to solve it now.  we are still using  placing the api in the jar and using the history-server api to monitor the app  in ds, which is obviously a bad practice

Apache StreamPark (StreamX) and Dinky (DLink) seem to realize the management of Flink on Kubernetes, and their methods can be referred to.

I'd like to join this discussion.
init

write Dockerfile to workspace and add extra data
pull flink base image
build job image with current workspace
push job image to the docker registry

handle

check kubernetes has the same app name
start flink application mode through ShellCommandExecutor
waiting for kubernetes deployment ready
listening kubernetes event and process(logHandler errorHandler...)

cancel

cancel flink job through the flink client(use kubernetesClient exec, so we don't have to care about kubernetes.rest-service.exposed.type.)

FlinkParameters for kubernetes need add the parameter(extJarLibs) , becausewe must import libs into the docker image when building it.

Hi, @RuiFG Flink and Spark can submit application by Flink/Spark Operator, and this way is so native k8s. In my opinion,I think we maybe clould use Operator mode,not ShellCommandExecutor.  @caishunfeng

Hi, @RuiFG Flink and Spark can submit application by Flink/Spark Operator, and this way is so native k8s. In my opinion,I think we maybe clould use Operator mode,not ShellCommandExecutor. In this way,we can submit an application by a yaml file @caishunfeng

+1, the operator way is good, can you add some design detail for this way? @dpengpeng
cc @liqingwang

I'd like to join this discussion.
init

write Dockerfile to workspace and add extra data
pull flink base image
build job image with current workspace
push job image to the docker registry

handle

check kubernetes has the same app name
start flink application mode through ShellCommandExecutor
waiting for kubernetes deployment ready
listening kubernetes event and process(logHandler errorHandler...)

cancel

cancel flink job through the flink client(use KubernetesClient exec, so we don't have to care about kubernetes.rest-service.exposed.type.)

FlinkParameters for kubernetes need add the parameter(extJarLibs) , becausewe must import libs into the docker image when building it.

I think the k8s task can support it, right? see https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/guide/task/kubernetes.html

cc @litiliu

hi, I'm the colleague of @liqingwang, We have implemented using flink-operator to submit flink-batch job in our production. I will write the detail about it in the following days.

hi @caishunfeng, I don't think k8s task can support it. K8sTask uses KubernetesClient to execute tasks, we have to do more to be compatible with flink task(eg: flink-config Config Map Flink kubernetes HA), and K8sTaskExecutor currently only supports batch job, for Flink k8s task, it is better to monitor pod(Flink TaskManager).

Hi, @RuiFG Flink and Spark can submit application by Flink/Spark Operator, and this way is so native k8s. In my opinion,I think we maybe clould use Operator mode,not ShellCommandExecutor. In this way,we can submit an application by a yaml file @caishunfeng

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
https://github.com/apache/flink-kubernetes-operator
cool, DS build image, uses operator through KubernetesClient

Hi, @RuiFG Flink and Spark can submit application by Flink/Spark Operator, and this way is so native k8s. In my opinion,I think we maybe clould use Operator mode,not ShellCommandExecutor. In this way,we can submit an application by a yaml file @caishunfeng

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator https://github.com/apache/flink-kubernetes-operator cool, DS build image, uses operator through KubernetesClient

Hi @RuiFG I checked the official documentation of operator, but I feel that the key problem is still unsolved, that is how to get his status after the batch task ends, the operator only provides how to manage a running task, but for the batch task to get his information after the end is important.
Or did I miss some points in doc?

@ikiler Yeah, I didn't see it in doc, maybe have to implement KubernetesWatcher on DS.
For DS, open source operators maybe have more management troubles. gcp spark k8s operators and flink kubernetes operators API are still beta, and they both have to deploy in k8s to make the task run, what did you think? @dpengpeng

@ikiler Yeah, I didn't see it in doc, maybe have to implement KubernetesWatcher on DS. For DS, open source operators maybe have more management troubles. gcp spark k8s operators and flink kubernetes operators API are still beta, and they both have to deploy in k8s to make the task run, what did you think? @dpengpeng

Well, by the way our implementation, we use flink native api to submit task ，and no need  deploy anything else in k8s ,maybe using the operator would be better to submit,

@ikiler Yeah, I didn't see it in doc, maybe have to implement KubernetesWatcher on DS. For DS, open source operators maybe have more management troubles. gcp spark k8s operators and flink kubernetes operators API are still beta, and they both have to deploy in k8s to make the task run, what did you think? @dpengpeng

Well, by the way our implementation, we use flink native api to submit task ，and no need  deploy anything else in k8s ,maybe using the operator would be better to submit,

hi @caishunfeng @ikiler , this is the design of our implementation based on flink-kubernetes-operator.
Background
We are using Flink Kubernetes Operator to run Flink batch jar job on k8s cluster.
And all files in resource are stored on S3.
Summary design

In flink-task plugin's frontend, namespace & cluster & image & compute resources need to be specified by user to run this job.
In the ds-worker, the CRD represents this batch job was built and submitted to k8s cluster by KubernetesClient.
After that CRD has been submitted, a watcher is registered in ds-worker to monitor the status of that batch job(error or finish or timeout).
After getting the final status of that job, that watcher will be closed.
The resources(jar or config files) are downloaded from S3 by the initContainer. Before the mainContainer run.

Detail design

In flink-task plugin's frontend, user need to specify the namespace & cluster & image in which to run that job.
In the ds-master, the resources(jar or config files)'s absolute path on S3 will be calculated and set to context in the BaseTaskProcessor.getTaskExecutionContext()
In the ds-worker, the CRD of FlinkDeployment
including the initContainer was built and submit to k8s cluster by KubernetesClient.
After that CRD has been submitted, a watcher is registered in ds-worker to monitor the status of that batch job(error or finish). A CountDownLatch is used to monitor job timeout.

limitation

Because we are using the "error" filed to monitor job status，only flink version >= 1.15 are supported, because of a flink bug, see status no clear when deploying batch job with flink-k8s-operator

Tks for discussion.

Hi, any update for submitting spark tasks to kubernetes?


R packages not persistent across restarts

Packages installed in R via install.packages are not persistent across restarts. Additionally, programs installed via apt to support R packages (ie libgdal-dev) suffer the same fate.
Is there a way to persist installed packages & applications?

R packages should persist across restarts, but that is failing due to a permissions issue. I implemented the fix and will push soon.
Installed applications/libraries/drivers will not persist across restarts; the RStudio/Shiny Docker image needs to be built with the necessary drivers and libraries. To do this, modify the file "rshiny_rstudio/Dockerfile". Add a RUN command anywhere between the first and last line to apt-get any apps/drivers/libs you want built into the image. After you edit the Dockerfile, rebuild the site with setup.sh.

This is fixed (for me) after the latest commits. Please clone a fresh copy, setup again, and let me know if the issue persists. Note that a few other changes were included. Namely, to setup the site, use control.sh setup instead of setup.sh.


Stop using Guava's immutable collection types in public getters?

This library can cause issues for projects that use its shaded jar and a different version of guava. Should we make a breaking change and stop using Guava's immutable collection types in public getters?
See https://github.com/spotify/helios/issues/1142

I know that when the abstract getters return Immutable* collection types, AutoValue will generate implementations that also use the Immutable* types. Do you know what collection types AutoValue will generate if your abstract getters do not return the Immutable* types?
I don't know if this matters.


The buffer should only work for INJ transfers

Looks like the buffer keeps a small amount of any token the user is trying to transfer? For instance here I wasn't able to transfer the full amount of USDC, the buffer should only be there for INJ since otherwise it will be confusing.

Nope, we only apply buffer for INJ


FSDP with PPO trainer won't work because FSDP doesn't support model.generate

Hi! This issue seems to be a feature request or bug report. How to fine tuning with PPO models that doesn't fit one GPU? I'm using FSDP from torch.dist with Accelerate. Unfortunately, ppo_trainer.py and ppo2_trainer.py uses model.generate(..) (GenerationMixin from transformers). Now, when I try to generate tokens it fails with "runtimeerror: 'weight' must be 2-d" (see this ). It looks that TRL now doesn't support FSDP because of this dependency to unsupported model.generate.
The error:
> Traceback (most recent call last):
>   File "/data/artyom_karpov/rl4steg/train.py", line 385, in <module>
>     train(context)
>   File "/data/artyom_karpov/rl4steg/train.py", line 308, in train
>     res = ppo_trainer.model.generate(
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/trl/models/modeling_value_head.py", line 204, in generate
>     return self.pretrained_model.generate(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/peft/peft_model.py", line 1491, in generate
>     outputs = self.base_model.generate(*args, **kwargs)
>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
>     return func(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 1758, in generate
>     result = self._sample(
>              ^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 2397, in _sample
>     outputs = self(
>               ^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
>     return self._call_impl(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
>     return forward_call(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 1166, in forward
>     outputs = self.model(
>               ^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
>     return self._call_impl(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
>     return forward_call(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 926, in forward
>     inputs_embeds = self.embed_tokens(input_ids)
925 modelling_llama.py LlamaModel(
  (embed_tokens): Embedding(128256, 8192)
>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
>     return self._call_impl(*args, **kwargs)
module.py 1532 self=Embedding(128256, 8192)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
>     return forward_call(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
module.py 1542 self=Embedding(128256, 8192) 128257*8192 = 1 050 673 152
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/nn/modules/sparse.py", line 164, in forward
>     return F.embedding(

sparse.py 163: torch.Size([525340673]) torch.Size([1, 9])  525 340 673

>            ^^^^^^^^^^^^
>   File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/nn/functional.py", line 2267, in embedding
>     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

functional.py: weight.shape=torch.Size([525340673]), input.shape=torch.Size([1, 9]), padding_idx=-1, scale_grad_by_freq=False, sparse=False

>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> runtimeerror: 'weight' must be 2-d

My setup:
Python 3.11.5
accelerate 0.30.1
torch 2.4.0.dev20240515+cu121
torchaudio 2.2.0.dev20240515+cu121
torchvision 0.19.0.dev20240515+cu121
transformers 4.41.2
Thanks for your attention

Thanks for reporting. This is a known issue. Unfortunately currently FSDP for PPO is not supported, but we are considering it for future iterations.

Thanks for reporting. This is a known issue. Unfortunately currently FSDP for PPO is not supported, but we are considering it for future iterations. For now, please use deepspeed stage 2 or 3.

Thanks for the reply. Yeah, I see it fails here and there. Now there is this error:
2024-06-17 08:37:15,707::93731__main__:DEBUG Before PPO step
[rank1]:[E617 08:47:15.010046090 ProcessGroupNCCL.cpp:572] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=7930, OpType=_ALLGATHER_BASE, NumelIn=214016000, NumelOut=856064000, Timeout(ms)=600000) ran for 600029 milliseconds before timing out.
[rank1]:[E617 08:47:15.011359495 ProcessGroupNCCL.cpp:1587] [PG 0 (default_pg) Rank 1] Exception (either an error or timeout) detected by watchdog at work: 7930, last enqueued NCCL work: 7931, last completed NCCL work: 7929.
[rank2]:[E617 08:47:15.047646334 ProcessGroupNCCL.cpp:572] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=7930, OpType=_ALLGATHER_BASE, NumelIn=214016000, NumelOut=856064000, Timeout(ms)=600000) ran for 600073 milliseconds before timing out.
[rank2]:[E617 08:47:15.048060595 ProcessGroupNCCL.cpp:1587] [PG 0 (default_pg) Rank 2] Exception (either an error or timeout) detected by watchdog at work: 7930, last enqueued NCCL work: 7931, last completed NCCL work: 7929.
[rank0]:[E617 08:47:15.098972015 ProcessGroupNCCL.cpp:572] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=7930, OpType=_ALLGATHER_BASE, NumelIn=214016000, NumelOut=856064000, Timeout(ms)=600000) ran for 600070 milliseconds before timing out.
[rank0]:[E617 08:47:15.099381586 ProcessGroupNCCL.cpp:1587] [PG 0 (default_pg) Rank 0] Exception (either an error or timeout) detected by watchdog at work: 7930, last enqueued NCCL work: 7931, last completed NCCL work: 7929.
[rank3]:[E617 08:47:15.617045435 ProcessGroupNCCL.cpp:572] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=7930, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=600000) ran for 600043 milliseconds before timing out.
[rank3]:[E617 08:47:15.617461256 ProcessGroupNCCL.cpp:1587] [PG 0 (default_pg) Rank 3] Exception (either an error or timeout) detected by watchdog at work: 7930, last enqueued NCCL work: 7930, last completed NCCL work: 7929.
[rank1]:[E617 08:47:15.836224160 ProcessGroupNCCL.cpp:1632] [PG 0 (default_pg) Rank 1] Timeout at NCCL work: 7930, last enqueued NCCL work: 7931, last completed NCCL work: 7929.
[rank2]:[E617 08:47:15.836226690 ProcessGroupNCCL.cpp:1632] [PG 0 (default_pg) Rank 2] Timeout at NCCL work: 7930, last enqueued NCCL work: 7931, last completed NCCL work: 7929.
[rank1]:[E617 08:47:15.836557209 ProcessGroupNCCL.cpp:586] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank2]:[E617 08:47:15.836835376 ProcessGroupNCCL.cpp:586] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E617 08:47:15.837097713 ProcessGroupNCCL.cpp:592] [Rank 1] To avoid data inconsistency, we are taking the entire process down.
[rank2]:[E617 08:47:15.837359220 ProcessGroupNCCL.cpp:592] [Rank 2] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E617 08:47:15.838567322 ProcessGroupNCCL.cpp:1632] [PG 0 (default_pg) Rank 0] Timeout at NCCL work: 7930, last enqueued NCCL work: 7931, last completed NCCL work: 7929.
[rank1]:[E617 08:47:15.838647484 ProcessGroupNCCL.cpp:1448] [PG 0 (default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=7930, OpType=_ALLGATHER_BASE, NumelIn=214016000, NumelOut=856064000, Timeout(ms)=600000) ran for 600029 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:574 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f710bfecde6 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f710d2938f2 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x247 (0x7f710d299f67 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f710d29bd6c in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7f7159f43bf4 in /data/artyom_karpov/miniconda3/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x7ea5 (0x7f716211fea5 in /lib64/libpthread.so.0)
frame #6: clone + 0x6d (0x7f716173fb2d in /lib64/libc.so.6)

[rank0]:[E617 08:47:15.838852370 ProcessGroupNCCL.cpp:586] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank2]:[E617 08:47:15.838951683 ProcessGroupNCCL.cpp:1448] [PG 0 (default_pg) Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=7930, OpType=_ALLGATHER_BASE, NumelIn=214016000, NumelOut=856064000, Timeout(ms)=600000) ran for 600073 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:574 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7efc19573de6 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7efc1a81a8f2 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x247 (0x7efc1a820f67 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7efc1a822d6c in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7efc674cabf4 in /data/artyom_karpov/miniconda3/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x7ea5 (0x7efc6f6a6ea5 in /lib64/libpthread.so.0)
frame #6: clone + 0x6d (0x7efc6ecc6b2d in /lib64/libc.so.6)

[rank0]:[E617 08:47:15.839506898 ProcessGroupNCCL.cpp:592] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E617 08:47:15.841090820 ProcessGroupNCCL.cpp:1448] [PG 0 (default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=7930, OpType=_ALLGATHER_BASE, NumelIn=214016000, NumelOut=856064000, Timeout(ms)=600000) ran for 600070 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:574 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f4573079de6 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f45743208f2 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x247 (0x7f4574326f67 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f4574328d6c in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7f45c0fd0bf4 in /data/artyom_karpov/miniconda3/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x7ea5 (0x7f45c91acea5 in /lib64/libpthread.so.0)
frame #6: clone + 0x6d (0x7f45c87ccb2d in /lib64/libc.so.6)

[rank3]:[E617 08:47:16.241666661 ProcessGroupNCCL.cpp:1632] [PG 0 (default_pg) Rank 3] Timeout at NCCL work: 7930, last enqueued NCCL work: 7930, last completed NCCL work: 7929.
[rank3]:[E617 08:47:16.242020870 ProcessGroupNCCL.cpp:586] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank3]:[E617 08:47:16.242297407 ProcessGroupNCCL.cpp:592] [Rank 3] To avoid data inconsistency, we are taking the entire process down.
[rank3]:[E617 08:47:16.243859509 ProcessGroupNCCL.cpp:1448] [PG 0 (default_pg) Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=7930, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=600000) ran for 600043 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:574 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f22a993dde6 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f22aabe48f2 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x247 (0x7f22aabeaf67 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f22aabecd6c in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7f22f7894bf4 in /data/artyom_karpov/miniconda3/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x7ea5 (0x7f22ffa70ea5 in /lib64/libpthread.so.0)
frame #6: clone + 0x6d (0x7f22ff090b2d in /lib64/libc.so.6)

After I run ppo_trainer.step

Is this with DS3? Also, would you like to try out the PPOv2Trainer?

Is this with DS3? Also, would you like to try out the PPOv2Trainer?

@vwxyzjn Sorry for the half-baked bug report. See it at https://github.com/huggingface/accelerate/issues/2868 . I thought to use PPOv2Trainer but it doesn't suit me because I do inference twice for my policy and use both results to assess, i.e. res1 = policy(x), res2 = policy(res1, x), and then I compute reward using another model and those two results. But PPOv2Trainer allows only one inference as I see it.

@vwxyzjn , and to anyone else who can help,
So an update re this issue. I've tried DeepSpeed 3 stage. And I see timeouts from NCCL operations. I didn't find exactly what operation times out. Tracing / debugging so far tells that it gets to the 713 line with model_inputs["input_ids"] = self.accelerator.pad_across_processes(... in ppo_trainer.py. At that point, I saw time outs when it tried to allocate 1EB (see above) in CUDA and got OOM. Now, I guess it passes through that somehow but hangs somewhere else. All this is painful and takes long time. As I wrote I can't use PPO trainer v2 . I have many options and ideas to investigate. Thank you for your thoughts on how can i fix it.
Here is the log from the last run. It fails after 20 min of waiting:
2024-06-20 11:41:34,839:260555:__main__:DEBUG Tensors, 3: q.shape=torch.Size([278]) q.device=device(type='cuda', index=2) response.shape=torch.Size([130]) response.device=device(type='cuda', index=2)reward=tensor(0.1001, device='cuda:2', dtype=torch.bfloat16)
[rank3]:[E ProcessGroupNCCL.cpp:563] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=775693, OpType=_ALLGATHER_BASE, NumelIn=262668288, NumelOut=1050673152, Timeout(ms)=1800000) ran for 1800042 milliseconds before timing out.
[rank1]:[E ProcessGroupNCCL.cpp:563] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=775693, OpType=_ALLGATHER_BASE, NumelIn=262668288, NumelOut=1050673152, Timeout(ms)=1800000) ran for 1800044 milliseconds before timing out.
[rank0]:[E ProcessGroupNCCL.cpp:563] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=775693, OpType=_ALLGATHER_BASE, NumelIn=262668288, NumelOut=1050673152, Timeout(ms)=1800000) ran for 1800058 milliseconds before timing out.
[rank3]:[E ProcessGroupNCCL.cpp:1537] [PG 0 Rank 3] Timeout at NCCL work: 775693, last enqueued NCCL work: 775693, last completed NCCL work: 775692.
[rank3]:[E ProcessGroupNCCL.cpp:577] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank3]:[E ProcessGroupNCCL.cpp:583] [Rank 3] To avoid data inconsistency, we are taking the entire process down.
[rank3]:[E ProcessGroupNCCL.cpp:1414] [PG 0 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=775693, OpType=_ALLGATHER_BASE, NumelIn=262668288, NumelOut=1050673152, Timeout(ms)=1800000) ran for 1800042 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ffadf608897 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7ffae08e1c62 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7ffae08e6a80 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7ffae08e7dcc in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7ffb2c39fbf4 in /data/artyom_karpov/miniconda3/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x7ea5 (0x7ffb3457bea5 in /lib64/libpthread.so.0)
frame #6: clone + 0x6d (0x7ffb33b9bb2d in /lib64/libc.so.6)

[rank2]:[E ProcessGroupNCCL.cpp:563] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=775693, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800022 milliseconds before timing out.
[rank0]:[E ProcessGroupNCCL.cpp:1537] [PG 0 Rank 0] Timeout at NCCL work: 775693, last enqueued NCCL work: 775693, last completed NCCL work: 775692.
[rank0]:[E ProcessGroupNCCL.cpp:577] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E ProcessGroupNCCL.cpp:583] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E ProcessGroupNCCL.cpp:1414] [PG 0 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=775693, OpType=_ALLGATHER_BASE, NumelIn=262668288, NumelOut=1050673152, Timeout(ms)=1800000) ran for 1800058 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f34f7a64897 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f34f8d3dc62 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f34f8d42a80 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f34f8d43dcc in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7f35447fbbf4 in /data/artyom_karpov/miniconda3/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x7ea5 (0x7f354c9d7ea5 in /lib64/libpthread.so.0)
frame #6: clone + 0x6d (0x7f354bff7b2d in /lib64/libc.so.6)

[rank1]:[E ProcessGroupNCCL.cpp:1537] [PG 0 Rank 1] Timeout at NCCL work: 775693, last enqueued NCCL work: 775693, last completed NCCL work: 775692.
[rank1]:[E ProcessGroupNCCL.cpp:577] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E ProcessGroupNCCL.cpp:583] [Rank 1] To avoid data inconsistency, we are taking the entire process down.
[rank1]:[E ProcessGroupNCCL.cpp:1414] [PG 0 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=775693, OpType=_ALLGATHER_BASE, NumelIn=262668288, NumelOut=1050673152, Timeout(ms)=1800000) ran for 1800044 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6bc5247897 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f6bc6520c62 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f6bc6525a80 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f6bc6526dcc in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7f6c11fdebf4 in /data/artyom_karpov/miniconda3/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x7ea5 (0x7f6c1a1baea5 in /lib64/libpthread.so.0)
frame #6: clone + 0x6d (0x7f6c197dab2d in /lib64/libc.so.6)

[rank2]:[E ProcessGroupNCCL.cpp:1537] [PG 0 Rank 2] Timeout at NCCL work: 775693, last enqueued NCCL work: 775693, last completed NCCL work: 775692.
[rank2]:[E ProcessGroupNCCL.cpp:577] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank2]:[E ProcessGroupNCCL.cpp:583] [Rank 2] To avoid data inconsistency, we are taking the entire process down.
[rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 0 Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=775693, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800022 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f2f91617897 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f2f928f0c62 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f2f928f5a80 in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f2f928f6dcc in /data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7f2fde3aebf4 in /data/artyom_karpov/miniconda3/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x7ea5 (0x7f2fe658aea5 in /lib64/libpthread.so.0)
frame #6: clone + 0x6d (0x7f2fe5baab2d in /lib64/libc.so.6)

  0%|          | 0/200 [42:10<?, ?it/s]
W0620 12:11:55.877000<PHONE_NUMBER>51360 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 260553 closing signal SIGTERM
W0620 12:11:55.877000<PHONE_NUMBER>51360 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 260554 closing signal SIGTERM
W0620 12:11:55.878000<PHONE_NUMBER>51360 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 260555 closing signal SIGTERM
E0620 12:11:56.592000<PHONE_NUMBER>51360 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: -6) local_rank: 3 (pid: 260556) of binary: /data/artyom_karpov/rl4steg/.venv/bin/python
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
Traceback (most recent call last):
  File "/data/artyom_karpov/rl4steg/.venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1082, in launch_command
    deepspeed_launcher(args)
  File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 786, in deepspeed_launcher
    distrib_run.run(args)
  File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/artyom_karpov/rl4steg/.venv/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=======================================================
train.py FAILED
-------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-06-20_12:11:55
  host      : compute-permanent-node-990.local.vcn
  rank      : 3 (local_rank: 3)
  exitcode  : -6 (pid: 260556)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 260556
=======================================================

@vwxyzjn Sorry for the half-baked bug report. See it at https://github.com/huggingface/accelerate/issues/2868 . I thought to use PPOv2Trainer but it doesn't suit me because I do inference twice for my policy and use both results to assess, i.e. res1 = policy(x), res2 = policy(res1, x), and then I compute reward using another model and those two results. But PPOv2Trainer allows only one inference as I see it.

To support your use case, how about making a copy of the PPOv2Trainer to do your desired changes?

@vwxyzjn , I'm thinking about this. But limitations are as follows as I see it, please feel free to correct me
PPO trainer v2:

It needs to load ref model to do KL divergence, which takes additional memory. I saw v1 doesn't do it if it is PEFT.
Coupling of the way the policy generates continuations inside train method which makes it harder to do something like res1 = policy(X), res2 = policy(res1, X).
I don't have reward model locally, I mean I use API of other models (GPT and other) to get me a reward.
Padding. I see V2 uses no padding but concatenates query and response as they did in the original paper. But I don't want my reward model to see query of the policy.
(And something else)

Anyway, I'm thinking how to work this out. Thanks

@artkpv Were you able to find a way around this problem? I am running into something similar.

@artkpv Were you able to find a way around this problem? I am running into something similar.

@5-Jeremy  I use now RLOOTrainer which uses this https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L1261 to generate completions. Should work for PPOTrainer second version (https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L57C5-L57C21).

@artkpv Were you able to find a way around this problem? I am running into something similar.

@5-Jeremy I use now RLOOTrainer which uses this https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L1261 to generate completions. Should work for PPOTrainer second version (https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L57C5-L57C21).

I see. Thanks!


Add some snippets

I did some experiments locally. However, for some reason, I couldn't yet produce a PDF with the Coq answers, so I will use this PR to check if that's produced correctly.

Which kind of Coq answer did cause problems ? Answers to simple requests as Print, About, etc. Or goal displaying ?
Envoyé de mon iPad

Le 20 juil. 2021 à 12:31, Théo Zimmermann @.***> a écrit :

I did some experiments locally. However, for some reason, I couldn't yet produce a PDF with the Coq answers, so I will use this PR to check if that's produced correctly.
You can view, comment on, or merge this pull request online at:
https://github.com/coq-community/hydra-battles/pull/47
Commit Summary
Expand .gitignore after #44.
Insert some new snippets extracted from the code.
File Changes
M .gitignore (3)
M doc/chapter-primrec.tex (28)
M doc/movies/Makefile (1)
M theories/ordinals/Ackermann/extEqualNat.v (4)
M theories/ordinals/MoreAck/PrimRecExamples.v (19)
Patch Links:
https://github.com/coq-community/hydra-battles/pull/47.patch
https://github.com/coq-community/hydra-battles/pull/47.diff
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Everything, but that's because my coq-serapi is not correctly installed. The problem doesn't show up in the version built by the CI.

Ok, so it's safe to merge  this PR to master now ?

The Check were printing output contrary to the previous version so I've pushed a new commit to try to mask the output. I'm waiting for the result of CI to see if that works.

OK, it worked! So this is ready to merge.

How about adding in the pdf doc and README.md a paragraph like:
" An increasing part of the hydras.pdf document is written with the help of the Alectryon software [...], replacing old
copy-pasted quotations  with snippets  generated from the source and answers by the system.
<descrition of the snippet tools, and credits>. "

Good idea


VCS lint issue in br_counter.sv

VCS caught a lint issue of continuous assignment width mismatch in br_counter. This is caught in VCS when all lint warnings are elevated to full-blown errors.
Next steps:

Fix the lib
Understand why verilog_lint_test doesn't catch it
Consider elevating all lint warnings in verilog_sim_test to be errors.

Found by @awijaya-openai

Lib was fixed in #197.
Filed internal issues on tooling so this doesn't escape these unit tests in the future.


Add docs & small fix to makefile

Add docs to bit allocator so its responsibilities is clarified.

Original makefile is missing a mkdir build, causing build failure if the user is starting from a clean repository.

In fact I think it'll be clearer to name it BitSet, as C++ already does do. But let it be anyway.