CoreUi - why nightwatch cannot run e2e tests?

I want to start working with CoreUi Admin template and create project containing unit and e2e testing.
In the begining, I install CoreUi like shown in documentation:
git clone https://github.com/coreui/coreui-free-vue-admin-template.git CoreUI-Vue
cd CoreUI-Vue
npm install

Everything is ok. Now I want to run whole unit and e2e tests:
npm run test:unit
npm run test:e2e

All unit tests run properly. But when I run e2e test I got the following errors:
><EMAIL_ADDRESS>test:e2e
> vue-cli-service test:e2e

 INFO  Starting development server...
[BABEL] Note: The code generator has deoptimised the styling of<EMAIL_ADDRESS>as it exceeds the max of 500KB.

 WARNING  Compiled with 2 warnings                                                                                2:47:38 PM
 warning

asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets:
  css/app.10fee0d6.css (383 KiB)                                                                                js/chunk-28d6db65.e7636129.js (488 KiB)
  js/chunk-2d0a400c.650c6f3a.js (3.37 MiB)                                                                                js/chunk-2d0ae5e6.22010046.js (390 KiB)
  js/chunk-2d237b34.468fc7b7.js (1.36 MiB)
  js/chunk-vendors.03b8fe62.js (687 KiB)

 warning

entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  app (1.07 MiB)
      js/chunk-vendors.03b8fe62.js
      css/app.10fee0d6.css
      js/app.94002fbe.js

                                                                                App running at:
  - Local:   http://localhost:8080/
  - Network: http://<IP_ADDRESS>:8080/

  App is served in production mode.
  Note this is for preview or E2E testing only.

 INFO  Running end-to-end tests ...

[Test] Test Suite
=================
⚠ Error connecting to localhost on port 9515.
_________________________________________________
TEST FAILURE: 1 error during execution; 0 tests failed, 0 passed (348ms)

 ✖ test
   An error occurred while retrieving a new session: "unknown error: Chrome failed to start: exited abnormally."
Error: An error occurred while retrieving a new session: "unknown error: Chrome failed to start: exited abnormally."
    at endReadableNT (_stream_readable.js:1201:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

  Error: An error occurred while retrieving a new session: "unknown error: Chrome failed to start: exited abnormally."                                                                                at endReadableNT (_stream_readable.js:1201:12)
       at processTicksAndRejections (internal/process/task_queues.js:84:21)

   SKIPPED:
   - CoreUI Vue e2e tests

 ERROR  Error: Command failed: /home/bagsiur/Projects/B2B/test-coreui/CoreUI-Vue/node_modules/nightwatch/bin/nightwatch --config<EMAIL_ADDRESS>--env chrome
Error: Command failed: /home/bagsiur/Projects/B2B/test-coreui/CoreUI-Vue/node_modules/nightwatch/bin/nightwatch --config<EMAIL_ADDRESS>--env chrome
    at makeError (/home/bagsiur/Projects/B2B/test-coreui/CoreUI-Vue/node_modules/execa/index.js:174:9)
    at /home/bagsiur/Projects/B2B/test-coreui/CoreUI-Vue/node_modules/execa/index.js:278:16
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
bagsiur@DESKTOP-718MBR1:~/Projects/B2B/test-coreui/CoreUI-Vue$
bagsiur@DESKTOP-718MBR1:~/Projects/B2B/test-coreui/CoreUI-Vue$ npm run test:e2e

><EMAIL_ADDRESS>test:e2e
> vue-cli-service test:e2e

 INFO  Starting development server...

 WARNING  Compiled with 2 warnings                                                                                3:06:39 PM
 warning

asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets:
  css/app.10fee0d6.css (383 KiB)
  js/chunk-28d6db65.e7636129.js (488 KiB)
  js/chunk-2d0a400c.650c6f3a.js (3.37 MiB)
  js/chunk-2d0ae5e6.22010046.js (390 KiB)
  js/chunk-2d237b34.468fc7b7.js (1.36 MiB)
  js/chunk-vendors.03b8fe62.js (687 KiB)

 warning
entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  app (1.07 MiB)
      js/chunk-vendors.03b8fe62.js
      css/app.10fee0d6.css
      js/app.94002fbe.js

  App running at:
  - Local:   http://localhost:8080/
  - Network: http://<IP_ADDRESS>:8080/

  App is served in production mode.
  Note this is for preview or E2E testing only.
 INFO  Running end-to-end tests ...
[Test] Test Suite
=================
⚠ Error connecting to localhost on port 9515.
_________________________________________________

TEST FAILURE: 1 error during execution; 0 tests failed, 0 passed (348ms)

 ✖ test
   An error occurred while retrieving a new session: "unknown error: Chrome failed to start: exited abnormally."
Error: An error occurred while retrieving a new session: "unknown error: Chrome failed to start: exited abnormally."
    at endReadableNT (_stream_readable.js:1201:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

  Error: An error occurred while retrieving a new session: "unknown error: Chrome failed to start: exited abnormally."                                                                                at endReadableNT (_stream_readable.js:1201:12)
       at processTicksAndRejections (internal/process/task_queues.js:84:21)

   SKIPPED:
   - CoreUI Vue e2e tests

 ERROR  Error: Command failed: /home/bagsiur/Projects/B2B/test-coreui/CoreUI-Vue/node_modules/nightwatch/bin/nightwatch --config<EMAIL_ADDRESS>--env chrome
Error: Command failed: /home/bagsiur/Projects/B2B/test-coreui/CoreUI-Vue/node_modules/nightwatch/bin/nightwatch --config<EMAIL_ADDRESS>--env chrome
    at makeError (/home/bagsiur/Projects/B2B/test-coreui/CoreUI-Vue/node_modules/execa/index.js:174:9)
    at /home/bagsiur/Projects/B2B/test-coreui/CoreUI-Vue/node_modules/execa/index.js:278:16
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

I work on wsl2 and Debian. My node.js version is v13.14.0 and npm is 7.12.1. I installed chromedriver in version: ChromeDriver 2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706) and last stable google chrome: google-chrome-stable 90.0.4430.212-1
What do I miss? Where could be the cause of the error in running e2e tests?
Stackoverflow: https://stackoverflow.com/questions/67519595/coreui-why-nightwatch-cannot-run-e2e-tests

@bagsiur
version mismatch of chromedriver and chrome is the most common reason of e2e failing like this.

ChromeDriver 2.41 supports Chrome v67-69
your google-chrome-stable 90.0.4430 needs ChromeDriver 90.0.4430


🍒 Cherry pick PR #4641 to staging 🍒

🍒 Cherry pick https://github.com/Expensify/App/pull/4641 to staging 🍒

🚀 Deployed to production by @isagoico in version: 1.0.85-9 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Add sample reporting data

Overview
Add sample data for dfGroups, dfMetrics, dfSummary and dfBounds for use in reporting workflow examples and tests. Follow the format- dfGroups = sampleGroups
closes #1626
Test Notes/Sample Code

Notes:

@jonthegeek Good call- I think we can make a new dfSummaryLong object with a few different snapshot_date values based on what is in dfSummary so that it can be used to test longitudinal charts, but we also want to be able to test that those charts get skipped for dfSummary objects with only one snapshot date. While we could just filter the long df, to avoid confusion i think i'll just make another small df for longitunidal data.
Since I'll just be changing dates of the existing data, i don't think i'll need new dfBounds, but in the future we plan to add variable thresholds across time (#1595), so depending on that workflow, a longitudinal dfBounds might be necessary.

@jwildfire pointed out https://github.com/Gilead-BioStats/gsm/blob/dev/inst/examples/LongitudinalReport.R over in #1570 as an option for generating longitudinal data.

change sampleMeta back to sampleGroups

@jonthegeek @jwildfire made the edits to the sample data files we discussed yesterday and added docs. also merged in dev, so checks should pass.

Thanks! Will review and merge now.

@jwildfire I should be able to dig into this, and hopefully fix whatever is broken in the GHAs while I'm at it!

Datasets that are still used. I didn't delete these/the scripts to generate them. If we want to get rid of them, we should probably explicitly remove their usage outside of this PR (or, alternatively, change the things that use them to use stuff in this PR).
In function defaults:

meta_workflow (Widget_SiteOverview())

In test_util-clindata.R:

config_param
config_workflow
meta_param
meta_param

In qualification tests:

config_param

I also created an example_groups() function while making R CMD check happy. It generates groups for Widget_SiteOverview() using clindata::ctms_site and clindata::rawplus_dm. I strongly suspect that SHOULD be sampleGroups or something, but it doesn't match much at all at the moment.
I left unit_test_data_script.R for the moment, but IF we want to keep it, it should probably move into the testthat folder (potentially even turned into a helper with functions to load datasets). I strongly suspect we don't need it once we universally use this sampleData, though.

@jwildfire The column nMinDenominator in sampleMetrics isn't mentioned for dfMetrics in the vignette. Should it be? dfMetrics is also missing information in the bulleted list describing it (in both instances). I'm not 100% certain what should be there.
Also see my notes above about datasets that are still used. I suspect they SHOULD be eliminated, but I'm not sure that this is the PR to do that in.

Thanks John. Agree that we can continue this in another PR. Filed #1642 to deal with removal of legacy data sets and cleaning up dependencies.

@jwildfire The column nMinDenominator in sampleMetrics isn't mentioned for dfMetrics in the vignette. Should it be? dfMetrics is also missing information in the bulleted list describing it (in both instances). I'm not 100% certain what should be there.
Also see my notes above about datasets that are still used. I suspect they SHOULD be eliminated, but I'm not sure that this is the PR to do that in.

nMinDenominator is a parameter for Summarize() so documentation there is probably sufficient. However, the concept of using workflow$meta to document custom parameters for a workflow probably does need to be explained in the vignette (we could possibly use `nMinDenominator as a representative example). I'm still planning to do a full read-through of the vignettes soon, so I can keep this on my radar.


The same video is repeatedly downloaded

This worked on August 4th and now suddenly stopped working. Tested from multiple computers.
PowerShell_transcript.qfpEhXoA.20190806132901.txt
$ ll -lah
total 61M
drwxr-xr-x 1 user 197121   0 Aug  6 13:32  ./
drwxr-xr-x 1 user 197121   0 Aug  6 13:30  ../
-rw-r--r-- 1 user 197121 21M Aug  6 13:31 '1. About the Course Author.mp4'
-rw-r--r-- 1 user 197121 21M Aug  6 13:31 '2. How to Use the Linux Academy Cloud Playground for this Course.mp4'
-rw-r--r-- 1 user 197121 21M Aug  6 13:32 '3. Course `Syllabus.mp4'

will it be fixed soon?. I am also facing the same issue.

Currently, I do not have an active LinuxAcademy subscription. If someone can voluntarily share their subscription with me, I shall work on the issues/requests. I'm sorry :(

Currently, I do not have an active LinuxAcademy subscription. If someone can voluntarily share their subscription with me, I shall work on the issues/requests. I'm sorry :(

Is there a way to send  a PM

You may

Currently, I do not have an active LinuxAcademy subscription. If someone can voluntarily share their subscription with me, I shall work on the issues/requests. I'm sorry :(

Is there a way to send a PM

You may reach me here https://www.facebook.com/TheRahulShaw

@RahulShaw I've sent you a message on facebook, please check.

Please update the youtube-dl package and that will fix the problem.
pip install youtube-dl --upgrade


General volume mesh overlay fix

This works for images that are uniform over all dimensions. There is an upstream bug that leads to an off-by-one error in 3D texture allocation (see JOGL GLBuffers image size allocations). The plan is to account for this in ClearGL.

The upstream issue has been fixed in https://github.com/scenerygraphics/scenery/commit/d7033e174e9da4c00f91f24c11f24b4be655eea7 👍


[Feature Request] Support for video formats

Currently, you can make animated banners with gif and animated webp and png files. But gif is generally a low quality format, and webp and apng aren't very popular, so most of the time you need to convert them beforehand. The ability to use mp4 and webm files with autoplay and loop enabled would be much simpler.

mp4 can be converted to webp using ffmpeg:
$ ffmpeg -i video.mp4 -vcodec libwebp -lossless 1 -loop 0 -preset default -an -vsync 0 -strict -1 output.webp

There are few online services that can do similar conversions too.

Oh that's an interesting one. Currently it supports image formats only, but I'm intrigued how well that can work. Autoplay should work fine if muted and without controls, so I'll get back to you on that when I get to it.
Though for file size purposes (and more so if you're using a cloud service like Obsidian Sync), the conversion of mp4 -> webp might work better for now


[NCHWc tensor layout support] Integration of MISA NCHWc asm kernel.

This PR is based on #1532
[New API]
extern "C" miopenStatus_t miopenSet4dTensorDescriptorWithLayout(miopenTensorDescriptor_t tensorDesc,
                                                                miopenDataType_t dataType,
                                                                miopenTensorLayout_t tensorLayout,
                                                                int n,
                                                                int c,
                                                                int h,
                                                                int w)

NCHWc layout will be identified as a 4-D tensor with last dimension stride equal to vector length (Currentlly only vector_c).
This API create the 4-D tensor descriptor with
dim lengths of (n, c/vector_c, h, w)
strides of (chw, hw*vector_c, w*vector_c, vector_c).
[Host side]

New enum type miopenTensorLayout_t to support different user input tensor layout.
New data type miopenHalfx4 and miopenHalfx8.
New base argument convfp16x4 and convfp16x8 for miopen main driver
New cmode convfp16x4 and convfp16x8 for miopen test driver
New member of tensorDescriptor, vectorLength and tensorLayout
New tensor and tensorDescriptor constructor include the tensorLayout parameter.

[CPU convolution]

A new way to get index of a pixel in "VECT_C" type tensor.
CPU convolution of NCHWc * NCHWc.

[Kernel side]

New conv_asm_implicit_gemm_gtc_fwd_nchwc solver support NCHWc(input) x NCHWc(weight) fp16x4 and fp16x8 fwd kernels based on Dlops for gfx1030.
NCHWc(input) x CHWNc(weight) fp16x4 and fp16x8 fwd kernels based on Dlops for gfx1030 in this

usage:
For building the ctest:

cmake --build . --config Release --target test_conv_igemm_dynamic_dlops_nchwc_nchwc_fwd_fp16x4
cmake --build . --config Release --target test_conv_igemm_dynamic_dlops_nchwc_nchwc_fwd_fp16x8
cmake --build . --config Release --target test_conv_igemm_dynamic_dlops_nchwc_chwnc_fwd_fp16x4
cmake --build . --config Release --target test_conv_igemm_dynamic_dlops_nchwc_chwnc_fwd_fp16x8

For executing MIOpenDriver:

./bin/MIOpenDriver convfp16x4 -I NCHW_VECT_C -O NCHW_VECT_C -f NCHW_VECT_C -n 2 -W 32 -H 32 -c 32 -k 32 -x 5 -y 5 -p 2 -q 2 -F 1 -V 1
./bin/MIOpenDriver convfp16x8 -I NCHW_VECT_C -O NCHW_VECT_C -f NCHW_VECT_C -n 2 -W 32 -H 32 -c 32 -k 32 -x 5 -y 5 -p 2 -q 2 -F 1 -V 1
./bin/MIOpenDriver convfp16x4 -I NCHW_VECT_C -O NCHW_VECT_C -f CHWN_VECT_C -n 2 -W 32 -H 32 -c 32 -k 32 -x 5 -y 5 -p 2 -q 2 -F 1 -V 1
./bin/MIOpenDriver convfp16x8 -I NCHW_VECT_C -O NCHW_VECT_C -f CHWN_VECT_C -n 2 -W 32 -H 32 -c 32 -k 32 -x 5 -y 5 -p 2 -q 2 -F 1 -V 1

cc: @atamazov

Does something go wrong? 240 files changed?

Does something go wrong? 240 files changed?

Please ignore this one... I see, they are kernel files.

@JehandadKhan @atamazov @pfultz2 ping for review request

@aska-0096

dim lengths of (n, c/vector_c, h, w)

Does this match cudnn conventions?

Please split this into two (or more, if necessary) PRs. The first one should contain API changes + minimal necessary changes of MIOpen's core (no applicable solvers).
[Hint] How I would do that:

Duplicate this branch and revert all but API changes there. Create a new "API" PR from it.
Convert this PR into draft and explicitly indicate its dependence on "API" PR.

I see, I will create a API PR and edit this PR.

I think so.
The CUDNN_TENSOR_NCHW_VECT_C can also be interpreted in the following way: The NCHW INT8x32 format is really N x (C/32) x H x W x 32 (32 Cs for every W), just as the NCHW INT8x4 format is N x (C/4) x H x W x 4 (4 Cs for every W). Hence, the VECT_C name - each W is a vector (4 or 32) of Cs.
Refer from https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnTensorFormat_t

Convert this PR into draft and explicitly indicate its dependence on "API" PR.

I still recommend doing the above and editing the PR description (the topmost comment). Thank you!

Converted.

Hi @atamazov. NCHWc API part has been merged, I think it's time to move the kernel side forward.

CI passed.
ping for review @atamazov

@DrizztDoUrden @shurale-nkn @atamazov
Ping for review. We can have a code review meeting if necessary.

@DrizztDoUrden @shurale-nkn @atamazov Ping for review. We can have a code review meeting if necessary.

Yes a very brief meeting might be helpful.

Yes a very brief meeting might be helpful.

+1

Yes a very brief meeting might be helpful.

+1

@atamazov What time is work for you?  😃


[storage] rename should_exist to should_exist_vec in schemadb test

Motivation
rename should_exist to should_exists to avoid name collision.
Have you read the Contributing Guidelines on pull requests?
Yes.
Test Plan
cargo xtest
Related PRs
(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/libra/website, and link to your PR here.)

@bors-libra r+

@huitseeker: :key: Insufficient privileges: Collaborator required

@bors-libra r+

:pushpin: Commit 898c11e has been approved by davidiw

:hourglass: Testing commit 898c11ec7c0867742c88279aa1915b5dedb4656e with merge 5c2ae647ba2e0eefded7cae35dbd69f66b10541e...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: davidiw
Pushing 5c2ae647ba2e0eefded7cae35dbd69f66b10541e to master...


update unit test for cmd: apply, check, compile

What type of PR is this?
unit test

What this PR does / why we need it:
Upgrade the unit test framework
Which issue(s) this PR fixes:

Fixes #431
Special notes for your reviewer:
The file plan.out had been accidentally added into this commit, which should be the same as the previous version, I don't know why it was recognized as a new version.
Does this PR introduce a user-facing change?

Additional documentation e.g., design docs, usage docs, etc.:

I've updated the unit testing framework for some of the files according to [mockey's official usage](https://github.com/bytedance/mockey/blob/main/README_cn.md).

I have read the CLA Document and I hereby sign the CLA


Tidying leads to significant losses when applied on a dense track or params are chosen too large

The reason behind it is that the algorithm only compares the current point to the next one, not to the last tidied one.
For simplification, I therefore suggest using simplify-geojson

I've looked at the code and this was my initial reaction too.
If you have data points collected at 4 seconds, then only the first and last data points are added.
The loop should not use deleted points at the 'current' (i'th) point, but instead use the last kept data point.
Something like:
const keptPoints = dataPoints.reduce((acc, dataPoint, i) => {
  // Always add first data point
  if (i === 0) return acc.concat(dataPoint)
  // Always add last data point
  if (i === dataPoints.length-1) return acc.concat(dataPoint)

  const lastKeptPoint = acc.slice(-1)
  return shouldKeep(lastKeptPoint, dataPoint)
    ? acc.concat(dataPoint)
    : acc
}, [])


Audit and Reporting - tagui_report.csv column LOG FILE location

Hello
In Audit and reporting option create Log html file for respective automation script in same location.
Do we have any config setting to change location of log file ?
Regards
Saket Lopes

Hi Saket, I'm afraid there isn't such an option now to configure this. However you can add a line in src\tagui (macOS / Linux) or src\tagui.cmd (Windows), to always move report files to a certain location. To do that, search for the line in above files with tagui_report.php and add a command  after that to move all the .html files to another folder. Copying @siowyisheng for info.

Thanks Kensoh

@siowyisheng @saketlopes  : Hello, if possible please share the code to move the file to desired location which we need to add under src\tagui.cmd

Hi @Anujpalu24
If you are on windows and you want to move the main report (without the running number) you can go to src\tagui.cmd and add the following line of code behind this line move /Y "%flow_file%.html.tmp" "%flow_file%.html" > nul
move /Y "%flow_file%.html" "your_folder_location" > nul

If you want to move the persistent running number reports you can replace your tagui_report.php file with the one in this attached zip file. Then specify your folder in the tagui_report.php file at the top.

This will both shift the file to the specified folder, as well as update the csv file with the new file path

tagui_report.zip


No module named 'utils.qual_util'"

Hi, in shape completion code I get a "No module named 'utils.qual_util'" error or the line "SDFusion = create_model(opt)". What could be the cause of this?

fixed! please run again and see if it works now or not


Reusable Enums

Hello everybody,
Is your feature request related to a problem? Please describe.
In our API we are making use of enums quite heavily and also reuse them over on several structs. Right now there is no possibility to reuse these enums. This leads to quite a lot of code duplication.
example:
//Struct for Request Body
type FooRequestBody struct {
   Category string `enums:"CategoryA, CategoryB"`
   // ....
}

//Struct for Request Body
type BarRequestBody struct {
   Category string `enums:"CategoryA, CategoryB"`
   // ...
}

Describe the solution you'd like
I know that reusable enums are only available from OpenAPI 3.0 on but it would be nice to have a similar behaviour using YAML anchors.
One solution how this could be approached is to include a new tag (for example // @Def ...) that allows to create independent definition schemas that then can be referenced on fields.
// @Def Category string Enums(CategoryA,CategoryB)

//Struct for Request Body
type FooRequestBody struct {
   Category string `swaggertype:"Category"`
   // ....
}

//Struct for Request Body
type BarRequestBody struct {
   Category string `swaggertype:"Category"`
   // ...
}

Describe alternatives you've considered
Custom types in go don't allow to use field tags in the sense of:
type Category string `enums:"CategoryA,CategoryB"`

One possibility would be to to wrap the primitive type in a struct:
type Category struct {
   string `enums:"CategoryA,CategoryB"`
}

This would result in in the API spec json in an extra nested objected which would needed to be flattened out again by adding 'swaggertype:"string"' to the field of type Category
This would also require a lot of custom funcs and wrappers and makes it therefore very inconvenient to work with.

@jo-fr In most common use case scenarios, these enums may be strings or integers, and I feel to be an overhead to add all possible values to documentation.
If you need this feature, please write a PR, and I will be more than happy to assist you with a code review.

Normally Golang application also defines enums like:
type Pill int

const (
	Placebo Pill = iota
	Aspirin
	Ibuprofen
	Paracetamol
	Acetaminophen = Paracetamol
)

It would be better if there is no need to manually write out enum values:
// @Def Enum
type Pill int

const (
	Placebo Pill = iota
	Aspirin
	Ibuprofen
	Paracetamol
	Acetaminophen = Paracetamol
)

In this way, the enum variants and values are only maintained in one place.

I don't like the idea for now because the swag generator is quite slow on large projects. Adding this enum evaluation will make it slower.
In most cases, you are using enums as discriminatory (describing object types or statuses) and their actual number of values is really small.
If anyone here intends to contribute with a PR, I'm open to reviewing and approving this feature.


Improve stdlib plugin to remove offline mode for nightly builds

Description:
$subject
This is needed because DB connectors use drivers that are not packaged into the distribution. With offline mode the drivers cannot be resolved.

As part of the task, I followed up and fixed distribution daily build.

Plugin is released and relevant modules updated.


Fix example of appending to githubWorkflowPublishTargetBranches

The example of appending to githubWorkflowPublishTargetBranches fails with:
build.sbt:23: error: No Append.Value[Seq[sbtghactions.RefPredicate], Seq[sbtghactions.GenerativePlugin.autoImport.RefPredicate.StartsWith]] found, so Seq[sbtghactions.GenerativePlugin.autoImport.RefPredicate.StartsWith] cannot be appended to Seq[sbtghactions.RefPredicate]
ThisBuild / githubWorkflowPublishTargetBranches += Seq(RefPredicate.StartsWith(Ref.Tag("v")))
                                                ^
[error] Type error in expression

This is because a Seq[RefPredicate] instead of a sole RefPredicate is appended.

LOL silly me…


[Test] Add update issue test for GitHubIssuesProvider

When using something like the following
  var issueUpdate = new IssueUpdate{Body = description};

 await client.Issue.Update(owner, name, issueNumber, issueUpdate);

Github deletes some elements of the issue, this is a known scenario for us, let's create an issue so this doesn't happen again, I think there are two instance in which we update the issue so just test that when we are updating an issue we always share all the elements of an issue not only the part we want to update.

Release Note Category

[ ] Feature changes/additions
[ ] Bug fixes
[ ] Internal Infrastructure Improvements

Release Note Description
No release note needed

I submitted a PR for this: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-service/pullrequest/36940
It goes through the whole process of creating the issue and deleting it, even though we are just testing the update. I think this is okay, but I also wonder if just updating a previously opened issue would be enough.
The problem with the first option is that we are performing more operations that could fail and are not tested, but it should be fine. Additionally, we are creating an issue every time, so we would end up with a bunch of issues (all of them closed, of course).
The problem with the second option is that we are not controlling the whole scenario and we are not starting with a clean state every time. Some people can manually remove the milestone and such.
I went for the first option of having a clean state every time, but if anyone has concerns about this, please let me know.
FYI @missymessa


Want to contribute? Here's how to add buffs/debuffs to be tracked!

Step 1) Get the icon from RS Wiki. Ideally it doesn't have text or a background - if it does you'll want to use image editing software to clean it up as best as you can. Don't use tools like remove.bg because the process needs to be as exact as possible. I recommend photopea.com if you don't have good image editing software installed. Save the icon as is to /asset/resource/ as buff.png. If this is a debuff that is meant to track cooldown (eg. weapon specs) change the border color from "debuff red" (#cc0000) to "buff green" (#5a9619).
Step 2) Create a desaturated (set saturation to 0 and darken the image slightly) version of the buff. On Photopea I use 0 saturation and -25 lightness. Check for any black (#000000) pixels and change them to #010101 as the overlay will render #000000 as transparent otherwise. Save the image to /asset/resource/ as buff-desat.png
Step 3) Crop the buff/debuff border out of the image and save the image to /asset/data/ as buff.data.png
Step 4) Add the buff to /src/index.html above the spacers but below the last existing buff/debuff. The ID should be the name of the buff/debuff, data-id should be incremented by 1, and the <span> for the buff name should be the name of the buff. if the buff is meant to have 100% uptime add the maintainable class. Look at existing buffs/debuffs as examples of what this should look like.
Step 5) Update /src/css/betterbuffsbar.css in a similar fashion to other buffs. You should be adding two selectors - one for your #BuffID and another for #BuffId.inactive. Set the background-image of #BuffID to the buff.png image and #BuffID.inactive to the buff-desat.png image. Look at existing buffs/debuffs as examples of what this should look like.
Step 5.5) You can send a PR at this point - as the most tedious part is now complete. If you would like to complete the remaining steps you can but it is not necessary for me to accept your PR.
Step 6) Update /src/index.ts adding your buff or debuff to buffsList and debuffsList in a similar fashion to existing buffs or debuffs.
Step 7) Update buffImages or debuffImages in a similar fashion to existing buffs or debuffs.
Step 8) Inside of watchBuffs() add a findStatus() for your buff in a similar fashion to existing buffs or debuffs. For transparent buffs/debuffs the threshold is usually around 60-80 but can be as low as 20 for opaque buffs/debuffs the threshold is usually 300+.
Note: RS Wiki icons are not always usable and may include backgrounds and text. While some may be possible to use for detection they cannot be displayed nicely in the buffs tracker and so will not be used unless a nice enough buff image is made. It's OK if it isn't absolutely perfect but it should be close.

Half of these steps are no longer required (Steps 2, 3, 4) - only the default buff image is required and the desaturated image is faked using CSS.


Bug: bdgpeakcall output zeros

Describe the bug
This is very similar to #362 in that using bdgpeakcall generates output that contain zeros in columns 7-9.
Below, example lines from bdgpeakcall output file
track type=narrowPeak name="peaks.bed" description="peaks.bed" nextItemButton=on
chr1	12237	15856	peaks.bed_narrowPeak1	367	.	0.00000	0.00000	0.00000	905
chr1	16017	16585	peaks.bed_narrowPeak2	90	.	0.00000	0.00000	0.00000	425
chr1	16786	18924	pval_peaks.bed_narrowPeak3	218	.	0.00000	0.00000	0.00000	704

To Reproduce
I am following the steps in https://github.com/taoliu/MACS/wiki/Advanced%3A-Call-peaks-using-MACS2-subcommands .
At the last step to call peaks on score track using a cutoff, this what I did:
macs2 bdgpeakcall -i peak_pvalue.bdg -c 2 -l 175 -g 150 -o peaks.bed
Expected behavior
I can see peak_pvalue.bdg have pvalues that are not zero
Here are the first few  peaks in peak_pvalue.bdg sort by 5th column (i.e. -log10(pvalue))
 sort -nrk 4 peaks_pvalue.bdg | head
chr3    122870739       122870740       2952.45776
chr3    122870740       122870742       2940.46948
chr3    122870742       122870743       2935.67627

Here are the first few sorted peaks based on 8th column which should be -log10(pvalue)
sort -nrk 8 peaks.bed | head -n 10
track type=narrowPeak name="peaks.bed" description="peaks.bed" nextItemButton=on
chrY    959237  959448  peaks.bed_narrowPeak212237     68      .       0.00000 0.00000 0.00000 166
chrY    7717573 7717766 peaks.bed_narrowPeak212349     83      .       0.00000 0.00000 0.00000 45
chrY    7311337 7311645 peaks.bed_narrowPeak212348     631     .       0.00000 0.00000 0.00000 230

Here are the  same narrowpeak file but sorted based on 5th column
sort -nrk 5 peaks.bed | head -n 10
chr3    122867452       122877691       peaks.bed_narrowPeak151954     29524   .       0.00000 0.00000 0.00000 3287
chr12   72773195        72777012        peaks.bed_narrowPeak48999      25339   .       0.00000 0.00000 0.00000 1398
chr15   90017533        90024659        peaks.bed_narrowPeak68747      24055   .       0.00000 0.00000 0.00000 506

Screenshots
If applicable, add screenshots to help explain your problem.
System (please complete the following information):

OS: Linux
MACS Version  2.2.6 and 2.1.4

Additional context
I first discovered the problem running macs version 2.1.4. Since #362 mentioned narrow peak calling with bdgpeakcall works fine, I upgraded to macs version 2.2.6 and had the same problem.
Thank you!

@olechnwin It's impossible to let bdgpeakcall to output the last three columns. This tool is a straightforward simple peak calling tool which will check the 4th column in bedGraph file and call the regions above a given cutoff. Nothing else. It has NO idea of what type of value in the 4th column of bedGraph file. In the output of bdgpeakcall, the 5th column is related to the score in peak summit from bedGraph.  The 5th column score = 10 * score in the summit from bedGraph.

Ahhh....Thank you very much for your quick response. In the subcommand tutorial, it mentioned that the output is a narrowpeak file so I assumed it should have all the values as in narrowpeak file.
Sorry about that.

@olechnwin You are welcome! I will revise my tutorials.

@taoliu, Nah. Your tutorial is correct. It stated that the output is in narrowpeak format. I just need to learn how to read properly :-)
I just have to mention how grateful I am with the support you provided. It is also amazing how you do this by yourself!


[NEMO-351] Empowering Nemo with fast I/O using Apache Crail

JIRA: NEMO-351: Empowering Nemo with fast I/O using Apache Crail
Major changes:

Users can specify -remote_option as crail if they intend to use CrailFileStore during their job execution. (Crail configuration and deployment has to take place.)

Any update on this PR?

@wonook Let's close stale PRs. Do you plan to merge this to master?


Dynamic inquire about type or node type?

Once I serialized into dynamic, internally this is a JsonObject instance. Is there a way inquire about node type or find out whether it's an array, object or primitive?
I know JsonObject supports casts I'd like to avoid exception catching if I need to test for several different possibilities.

Dynamic inspection is supported through TypeConverters.
You can find examples in this test.

Thanks a bit clunky but works.

Yeah, I wish I could make the is and as keywords do that lifting - but there's no way to hook into them w.r.t. dynamic dispatch.  So since I'm stuck with clunky, I went with the clunk that's built in.

Any chance you'd consider making JsonObject public instead of internal?

Nah, it's too unstable in practice.  Some funky reflection in there too, that I suspect is gonna get real fun with a core port in the future.

Then maybe something like serialization into internal structure a la JToken from you know which library.


Enhance appBasePath() to handle multiple end directories

This PR updates the appBasePath() function to:

Check for multiple possible end directory names ('public', 'pub', 'wp-admin')

This change improves compatibility with various project structures (e.g., Laravel, Magento 2, WordPress).
Also fixes https://github.com/laradumps/laradumps-core/issues/7

That’s what we need

Thank you!


Merge scheduler into deployer

Merge scheduler impl into k8s deployer to benefit from existing functionality re-use, less duplication, etc

closing in favor of https://github.com/spring-cloud/spring-cloud-deployer-kubernetes/issues/316


Re-enable MSVC target

In a comment you said that ...

... if I am not mistaken a windows-gnu host can build windows-msvc targets and it should be easy to add windows-msvc target. I'll add windows-msvc target again when I am free.

Since there seemed to be no issue tracking that, I thought I'd create one. I think this is important because some Windows crates can't build for the MinGW target and MSVC is the preferred target anyway.

Thanks for reporting. I'll add MSVC windows targets again with the next rustc update. I've got some issues building rust compiler for the all tier one platforms, hopefully I'll fix it in the next few days.

Alright, I re-enabled MSVC targets, sorry it took me awhile to figure out right build tools for compiling rustc msvc targets (unfortunately I tried many). For the future reference: Visual C++ 2015 Build Tools is the right toolset for building rustc msvc targets.


ITestDataSource: access to RunSettings in method GetData

Summary
For data-driven tests, where the test-data is specified in the .runsettings (e.g. a folder with files to test) it is very difficult to generate specific test-names for each test case, that are also shown in the azure ui.
Background and Motivation
During test-discovery the method GetData does not allow accessing the TestContext of a surrounding class (since it is not a method of it), neither got a [ClassInitialize] method called, which would allow to copy the current TestContext to a static variable.
To be clear: it is the "test-data discovery" that fails (which leads to all tests getting the same name in the azure ui); the tests itself work correctly, as then the [ClassInitialize] method already got called, and I could store the TestContext to a static variable.
The only way I could solve it, was working down this list in GetData:

get current-process
get its parent-process
get parent-processes command line
search in the specified command-file for the .runsettings-file line
read in that file and search for the data

Here is a code snipped:
                    // GetCommandLine and ParseCommandLineArguments defined elsewhere

                    var parentProcess = GetParentProcess(System.Diagnostics.Process.GetCurrentProcess().Id);
                    if (parentProcess != null)
                    {
                        string commandLine = GetCommandLine(parentProcess);
                        string[] arguments = ParseCommandLineArguments(commandLine);
                        foreach (string arg in arguments)
                        {
                            if (arg.StartsWith("@"))
                            {
                                var args_file = arg.Substring(1);
                                string[] lines = File.ReadAllLines(args_file, Encoding.ASCII);
                                foreach (var line in lines)
                                {
                                    if (line.StartsWith("/Settings:"))
                                    {
                                        var name = line.Substring("/Settings:".Length);
                                        name = name.Trim('\"');
                                        StreamReader Reader = new StreamReader(new FileStream(name, FileMode.Open, FileAccess.Read, FileShare.ReadWrite), Encoding.UTF8);
                                        XmlDocument runsettings = new XmlDocument();
                                        runsettings.Load(Reader);
                                        Reader.Close();
                                        // read runsettings ...

Proposed Feature
Either an altenative GetData method which also has the TestContext as parameter, or a public static method that returns current TestContext.
Alternative Designs
See my workaround, but IMHO that very fragile and may break unexpectedly.

This is a good suggestion but it feels unlikely to be fixed soon so I'll move the ticket as won't fix.


np.take and np.einsum aren't fused properly

I'm trying to translate Product Key Memory  in PyTorch into JAX, and this requires the translation of nn.EmbeddingBag with per_sample_weights, as I haven't found any counterpart in JAX (but if you know, please let me know). For this, I wrote scatter_weighted_sum, the weighted sum version of scatter_add, in hope that it'll be efficient with fusing. However, jit didn't fuse np.take, reshape and np.einsum properly, which resulted in a huge intermediate object. Since #1979 concluded this sort of ops will be fused on GPU, I was wondering what is causing this problem. If by any chance this isn't supported on GPU, should this work on TPU? I'm using JAX ver. 0.1.67 with various Colab GPUs.
hidden_dim = 512
n_keys = 512
batch = 2 ** 15
knn = 32
heads = 4
key = random.PRNGKey(0)
values = random.normal(key, (n_keys ** 2, hidden_dim))
indices = random.randint(key, (batch*heads, knn), 0, n_keys ** 2)
weights = random.normal(key, (batch*heads, knn))
@jit
def scatter_weighted_sum(inputs, indices, weights):
    num_bags = weights.shape[-1]
    dim = inputs.shape[-1]
    indices = indices.reshape(-1)
    tmp = inputs.take(indices, axis=0).reshape(-1, num_bags, dim)
    return np.einsum('ind, in -> id', tmp, weights)

I agree, XLA is not fusing the gather with the reduction, which would avoid materializing the intermediate value, and I don't see a good reason for that. I filed a couple of bugs for the XLA teams.
A possible workaround would be to use lax.map to compute the elements in chunks.

We are having the same problem in TF-addons (and so a green light for a custom-op PR). https://github.com/tensorflow/addons/issues/2201
And in XLA/Pytorch:
https://github.com/pytorch/xla/issues/2403

We are having the same problem in TF-addons (and so a green light for a custom-op PR). https://github.com/tensorflow/addons/issues/2201
And in XLA/Pytorch:
https://github.com/pytorch/xla/issues/2403

@hawkinsp Are these XLA bugs/tickets public on github or internal?

@hawkinsp Are these XLA bugs/tickets public on github or internal?

It's currently an internal bug (b/157497992). It's worth noting that I raised this issue in particular with respect to GPUs; the issues might be different on CPU, GPU, or TPU. I'll ping it, although there are no guarantees they will have time...

It's currently an internal bug (b/157497992). It's worth noting that I raised this issue in particular with respect to GPUs; the issues might be different on CPU, GPU, or TPU. I'll ping it, although there are no guarantees they will have time...

@hawkinsp Do you think that you can open one at https://github.com/tensorflow/mlir-hlo so that It would be easier to track?

@hawkinsp Do you think that you can open one at https://github.com/tensorflow/mlir-hlo so that It would be easier to track?

Given that in this context we are talking about XLA classic not any of the MLIR work, the best place for a public XLA issue is the TF Github tracker. Feel free to file one if you like, although I think this issue covers it just as well.

Given that in this context we are talking about XLA classic not any of the MLIR work, the best place for a public XLA issue is the TF Github tracker. Feel free to file one if you like, although I think this issue covers it just as well.

In core TF we have already two users tickets that depend on this XLA missing fusion. I don't know if the team plans to cover that on MLIR-HLO or it is still plausible to expect something for classic XLA:
https://github.com/tensorflow/tensorflow/issues/32675
https://github.com/tensorflow/tensorflow/issues/43784

In core TF we have already two users tickets that depend on this XLA missing fusion. I don't know if the team plans to cover that on MLIR-HLO or it is still plausible to expect something for classic XLA:
https://github.com/tensorflow/tensorflow/issues/32675
https://github.com/tensorflow/tensorflow/issues/43784

Using embeddingbag instead of mebedding+allreduce will have a "huge" impact not only on performance but also on memory footprint for large number of indices. By computing the "sum" operation on the fly will save great amount of memory. We do not need to store all the table lookup results. A higher priority should be given to this issue.

Using embeddingbag instead of mebedding+allreduce will have a "huge" impact not only on performance but also on memory footprint for large number of indices. By computing the "sum" operation on the fly will save great amount of memory. We do not need to store all the table lookup results. A higher priority should be given to this issue.


Slow wslg performance with Intel integrated graphics

Windows build number:
10.0.22621.0
Your Distribution version:
22.04
Your WSL versions:
WSL version: <IP_ADDRESS>
Kernel version: <IP_ADDRESS>
WSLg version: 1.0.47
MSRDC version: 1.2.3575
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.1105
Steps to reproduce:

Update Windows on a laptop with Intel integrated graphics
Intel drivers are rolled back (this happens consistently after Windows updates)
Observe slow WSL performance (Wayland): lags in display updating after making an input (windows apps do not experience such a decrease in speed)
Update Intel drivers
Observe normal speeds

WSL logs:
No response
WSL dumps:
No response
Expected behavior:
Apps are laggy with the standard drivers provided after Windows update. Response times are so large that even text editors become almost unusable.
Actual behavior:
This post is primarily to guide users to one root of the problem when WSL is laggy. I doubt the wslg crew can convince Microsoft that it is a terrible idea to automatically roll back drivers every time you update Windows.

You can use this troubleshooter to hide the Intel graphics update coming from Windows Update:
https://social.technet.microsoft.com/wiki/contents/articles/53184.show-or-hide-updates-utility-fixing-automatic-installation-of-a-problematic-update-in-windows-10-version-1903.aspx

Unfortunately that does not work. It simply does not find updates to hide.
The fault is not with wslg though. It is just Microsoft who keeps "updating" the Intel drivers to a two year old version on its Surface Pro models, ignoring drivers directly installed from Intel. The reason why I posted this here is so that Surface Pro users know how to fix these performance issues: reinstall the Intel drivers after every. single. Windows. update.

Did you follow the instructions on the page? It wants you to restart the computer. It worked for me.

It can be simpler to:

first uninstall the device from Device Manager selecting the checkbox "Delete the driver software for this device"
then use DriverStore Explorer to delete the driver files from the Windows Driver Store

Maybe Windows doesn't properly remove them when uninstalling the device

I ran into this same problem with Windows downgrading the Intel graphics driver on my Dell Latitude 7490 :(
But I think I found a solution to this problem, which I'll describe below:

I installed the Intel graphics driver from their website.
Then after a reboot, I started PC Settings app and selected Windows Update where I saw it already getting the previous driver
At this point I let it continue installing, but before rebooting I went to Control Panel/Device Manager where I selected "Display Adapters/Intel(R) UHD Graphics 620" and then right click/Properties.
In the Driver tab, I clicked on "Roll Back Driver" and then wrote there the reason why I wanted to roll back the driver to a previous driver.
Then after a reboot, Windows Update didn't any more offer to install its own selected driver, but let me keep the driver installed from Intel's website.

I hope this helps you also.


Display a list of syntax errors

This shall be a FE substitute for #47
While it's possible to go through all the markers by pressing F8 - good tip to share with community 😉 - I think it's much better to display it as a list, right next to a button, which shall send template to the "chain".

Solution looks something like this (when one of the items in that list is hovered):

Solution looks something like this (when one of the items in that list is hovered):

@MaxStalker Very helpful!

@MaxStalker Very helpful!

@10thfloor Personally, I don't like to go lower than 14px (which is what I set it to now) on a text that you need to actually read 😅

@10thfloor Personally, I don't like to go lower than 14px (which is what I set it to now) on a text that you need to actually read 😅


RabbitMq IO Connector Refactor + Bugfixes

Background
My first attempts at using RabbitMq in Beam were quite problematic. Watermarking did not advance as expected, CheckpointMarks would fail as amqp channels could be severed during processing, deduplication of messages was optional but hard-coded to an awkward implementation that only looked at the 'correlation id' amqp property, which isn't necessarily unique. In short, this did not behave like a top-tier IO connector.
I had a great Q&A with the Beam Dev DL (see<EMAIL_ADDRESS>) and I have a pretty substantial refactor ready for review.
Open Questions
I don't know the best process to follow here. This is a lot of change, and there isn't a backing ticket for much (or any) of it right now. Seems like there at least should be.
Also, I'm wondering if I could/should be part of the OWNERS for this module if this gets merged. It seems there aren't many eyes on rabbitmq IO and I'm happy to help out if I can.
Changes
*Note This will not be binary compatible with previous versions of rabbitmq io. End users will have to update code. I think given the level of 'not working' I experienced as a user of rabbitmq with beam, this is ok. But we can discuss.
I'll break this apart into manageable sections:
Structural Changes

I've followed the KafkaIO connector's convention of separating many of these classes into separate files. It's been very difficult to trace the code with everything in RabbitMqIo.java.
Standardized on file/class prefix RabbitMq

RabbitMqIO Changes

I've explicitly dropped support for declaring an exchange. I can't find a valid use case for this with Beam and it's unnecessary code complexity.

Refactored the API and documentation to drop terminology indicating one 'writes to / reads from an exchange vs writes to / reads from a queue'. In AMQP 0.9, one always writes to an exchange and always reads from a queue. Instead, the caller now just specifies whether the queue already exists (simple use case) or if it may need creating (more complex), with a ReadParadigm interface defining this. Similarly, writing is always done to an exchange. RabbitMqMessage should already contain an appropriate routing key.
Supports pluggable connection handlers (more on this in Resource Management below)
Supports pluggable message deduplication strategies, but a strategy is now required). More on this in Message Deduplication below.
Supports total backlog bytes estimation, with some caveats. Mostly stolen from KafkaIO's implementation.
Reworked RabbitMqMessage to use AutoValue.

Message Deduplication

Previously, this was user-toggled. Given CheckpointMark finalizing can fail, this should not be considered an optional operation. requiresDeduping is now hard-coded to true
Strategy for detecting duplicate messages is pluggable, with several implementations already built-in to choose from. The previous default was the amqp property correlationId but this is not a safe choice. Many messages could be correlated to the same upstream event. Amqp property messageId would be a better choice, if implemented, but there are others the user can choose from, including always unique which effectively disables deduping.

Resource Management

I borrowed connection management from JdbcIO in terms of maintaining a static map of Connections tied to a remote URI. In AMQP, one multiplexes over Connections via Channels.
For better Channel management, each reader and writer now has a unique id (UUID) and I use the borrower (or lender) pattern to define what work should be done along the channel, but keep channel/connection management outside of the caller's concern to ensure consistent error handling, recovery, etc.

Checkpoint Marks
*Assumption This refactor is oriented around the following assumptions: 1) a CheckpointMark is associated with zero or one current Readers, but no more; 2) a CheckpointMark may outlive a Reader, but the converse is not true.

Dropped support for amqp transactions. They serve no purpose in this streaming model and in general should just be adding overhead. Messages are acknowledged or they aren't; that's transactional enough.
RabbitMqCheckpointMark may outlive a reader, so it's now responsible for closing an active channel if the checkpoint mark is finalized and there are no more reads happening
RabbitMqCheckpointMark is no longer re-used across readers. A new one will be instantiated any time a Reader is instantiated as there's no way safe way to 'recover' a deserialized CheckpointMark and still acknowledge messages.

Watermarks and Timestamps

I've basically re-implemented a dynamic watermarking strategy based off Kafka's. Like Kafka, I've made the implementation pluggable, with the dynamic CustomTimestampPolicyWithLimitedDelay and the default ProcessingTimePolicy.

I've done some testing in Dataflow and locally and I believe things are behaving correctly and this is in a pretty good state, so I wanted to start soliciting feedback now.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Thank you @je-ik !

a) what is the role of lessee in Channel? Are the multiple simultaneus "leases" on a single Channel at specific moment?

It is possible. A Reader may advance (read a message) on the same Channel the CheckpointMark is acknowledge previousl-read messages on. A CheckpointMark/Reader pair should effectively share a Channel and will 'lease' (or 'borrow') using the same id (in this case the checkpointMarkId) but no two readers and no two CheckpointMarks should ever use the same Channel.

b) is there any timeout for messages to be acknowledged before RabbitMQ redelivers them?

Per https://stackoverflow.com/a/30645242/796064 it does not appear to be the case. However if the channel or connection is ever severed, they will be redelivered.
c) what if there are no data flowing in, how would you update watermark?
I followed the Kafka model here, but there are two ways:

Using the ProcessingTimePolicy it will always advance
Using the more adaptive, Kafka-like approach, we update the context to show 'no backlog' when we advance and get no messages (see https://github.com/apache/beam/pull/10509/files#diff-447be8b9c794ae46f8d44c6cc1670201R152 ) which is used in the timestamp policy to update the watermark to the current time if at least one message has been seen before and there are no messages now (see https://github.com/apache/beam/pull/10509/files#diff-4fdb4b453f9bf31e0fecda73e2c97663R94 )

Thanks Daniel! I'll try to give this at least a quick look in the next couple of days but next 2 weeks I'll be on vacation, so you'll need to rely on other reviewers.

I gonna do a pass as well (as one of the original RabbitMQIO author).

a) what is the role of lessee in Channel? Are the multiple simultaneus "leases" on a single Channel at specific moment?

It is possible. A Reader may advance (read a message) on the same Channel the CheckpointMark is acknowledge previousl-read messages on. A CheckpointMark/Reader pair should effectively share a Channel and will 'lease' (or 'borrow') using the same id (in this case the checkpointMarkId) but no two readers and no two CheckpointMarks should ever use the same Channel.

OK, but then I don't quite understand why not to enforce single Channel on each Connection? Each reader will then have single connection and single channel, there would be no need for any "leases". Would this be possible? If yes, then it could pretty simplify the logic.

b) is there any timeout for messages to be acknowledged before RabbitMQ redelivers them?

Per https://stackoverflow.com/a/30645242/796064 it does not appear to be the case. However if the channel or connection is ever severed, they will be redelivered.

If messages don't timeout, then it might be actually the case that no duplicate messages are possible. It would require failing reader once channel/connection is broken (throwing exception), which would cause the source to be reset to last checkpoint, which would cause all not-yet-confirmed messages to be redelivered. It would require message commit in checkpoint to be atomic (don't know if that is possible). If these conditions could be satisfied the logic could be simplified even more.

c) what if there are no data flowing in, how would you update watermark?

I followed the Kafka model here, but there are two ways:
1. Using the [ProcessingTimePolicy](https://github.com/apache/beam/pull/10509/files#diff-8df6894f45062f43b381506a3663e5b4R108) it will always advance

2. Using the more adaptive, Kafka-like approach, we update the context to show 'no backlog' when we advance and get no messages (see https://github.com/apache/beam/pull/10509/files#diff-447be8b9c794ae46f8d44c6cc1670201R152 ) which is used in the timestamp policy to update the watermark to the current time if at least one message has been seen before and there are no messages now (see https://github.com/apache/beam/pull/10509/files#diff-4fdb4b453f9bf31e0fecda73e2c97663R94 )

:+1:

a) what is the role of lessee in Channel? Are the multiple simultaneus "leases" on a single Channel at specific moment?

It is possible. A Reader may advance (read a message) on the same Channel the CheckpointMark is acknowledge previousl-read messages on. A CheckpointMark/Reader pair should effectively share a Channel and will 'lease' (or 'borrow') using the same id (in this case the checkpointMarkId) but no two readers and no two CheckpointMarks should ever use the same Channel.

OK, but then I don't quite understand why not to enforce single Channel on each Connection? Each reader will then have single connection and single channel, there would be no need for any "leases". Would this be possible? If yes, then it could pretty simplify the logic.

Channels are not shared, but there can be multiple Channels per connection. A canonical use case would be a pipeline that reads messages from RabbitMq and publishes responses to the same RabbitMq host. In this case, the Reader will use a Channel and each Writer will use a Channel, but the single Connection will be shared between all of them.

b) is there any timeout for messages to be acknowledged before RabbitMQ redelivers them?

Per https://stackoverflow.com/a/30645242/796064 it does not appear to be the case. However if the channel or connection is ever severed, they will be redelivered.

If messages don't timeout, then it might be actually the case that no duplicate messages are possible. It would require failing reader once channel/connection is broken (throwing exception), which would cause the source to be reset to last checkpoint, which would cause all not-yet-confirmed messages to be redelivered. It would require message commit in checkpoint to be atomic (don't know if that is possible). If these conditions could be satisfied the logic could be simplified even more.

I don't follow. In the approach in this PR, the previous CheckpointMark is not re-shared as it can't be re-used safely. Messages will be redelivered (they were not acked) and at that point they should be caught be the 'dedpue' strategy. I'm not following what logic could be simplified or how this isn't 'duplicate message' delivery.

c) what if there are no data flowing in, how would you update watermark?

I followed the Kafka model here, but there are two ways:
1. Using the [ProcessingTimePolicy](https://github.com/apache/beam/pull/10509/files#diff-8df6894f45062f43b381506a3663e5b4R108) it will always advance

2. Using the more adaptive, Kafka-like approach, we update the context to show 'no backlog' when we advance and get no messages (see https://github.com/apache/beam/pull/10509/files#diff-447be8b9c794ae46f8d44c6cc1670201R152 ) which is used in the timestamp policy to update the watermark to the current time if at least one message has been seen before and there are no messages now (see https://github.com/apache/beam/pull/10509/files#diff-4fdb4b453f9bf31e0fecda73e2c97663R94 )

+1

Channels are not shared, but there can be multiple Channels per connection. A canonical use case would be a pipeline that reads messages from RabbitMq and publishes responses to the same RabbitMq host. In this case, the Reader will use a Channel and each Writer will use a Channel, but the single Connection will be shared between all of them.
Why to try to share connections to the broker between Reader and Writer? Why not create two connections, enforce single channel per connection and greatly simplify the overall logic? The benefits of the optimization seem to be low (multiple connections from single host instead of single connection) given the complexity it brings.

I don't follow. In the approach in this PR, the previous CheckpointMark is not re-shared as it can't be re-used safely. Messages will be redelivered (they were not acked) and at that point they should be caught be the 'dedpue' strategy. I'm not following what logic could be simplified or how this isn't 'duplicate message' delivery.
Generally, source requires deduplication if messages can be redelivered even in case application doesn't crash. Someone in this PR might correct me, but as I see it, provided that:
a) message acknowledgements don't timeout in the broker (that is message cannot be redelivered unless connection breaks) and
b) messages can be acknowledged atomically (i.e. either all messages are acked or no message is acked)
then the source might not need deduplication (requiresDeduping() could return false) and all the logic extracting deduplication ID could be dropped. It seems to me that condition (a) is satisfied, not sure if condition (b) is satisfied as well.

Thanks for the feedback so far, everybody. I'm much more dedicated to getting a first class rabbitmq io into beam than I am about any particular implementation details, so I'm going to drop most of the connection management, channel management, caching code and go back to something a bit more "java-y" which should both simplify the PR and scope the conversations in this PR down to the important Beam/Rabbit concerns and away from connection management.


Calculating Gravity in JS Animation

May I make a section showing how to calculate gravity in JS through canvas animation?

This would be cool, but it is very application specific. The main goal of this repo is to show mathematical notation as code, so that the reader will have the skills to implement something like this on their own.
For now, I would like to keep the scope small and focused. Eventually I might be open to having more application-specific examples in a separate folder within this repo. However, it would still be focusing on translating literature and notation into code, and it would try to cover areas that are not as well dissected. Whereas there are a lot of tutorials on the web for animating things with gravity/physics in code.
:smile:


Additional content for ACES presentation

@kristjank moar

@roks0n tnx awesome...
can you do some kind of step by step.. in the sense of How to do aces bla bla,
step 1 (slajd 1) ... and guide for them to see/read

@kristjank would an example for ARK to ETH service alone be ok?

yes yes 👍


Ext panel crash after wrong path

Just one more issue I have found :d
When I write wrong path in "wp path" The ext panel crashed with error:
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/acp/phpbbwpunicorn_module.php on line 38: require_once(..//wp-load.php): failed to open stream: No such file or directory
Fatal error: require_once(): Failed opening required '..//wp-load.php' (include_path='.:/usr/local/php/pear5') in /forum/ext/wardormeur/phpbbwpunicorn/acp/phpbbwpunicorn_module.php on line 38
Solution: disable ext, delete data, enable ext

Now I think I have correct path but half of the settings are missing after I save path in ext panel. They are saved(just non changeable)? or it is a bug?

After creating a new user I have this error:(in phpbb)(It creates user in wp but I can't log into this test user.
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
Warning: Cannot modify header information - headers already sent by (output started at /forum/includes/functions.php:3899) in /forum/includes/functions.php on line 5235
Warning: Cannot modify header information - headers already sent by (output started at /forum/includes/functions.php:3899) in /forum/includes/functions.php on line 5235
Warning: Cannot modify header information - headers already sent by (output started at /forum/includes/functions.php:3899) in /forum/includes/functions.php on line 5235

I'll look into this asap, thakns for the feedback

I'm checking the latest version of this 2(pl and ext) but I can't get it working. Do I have to set some special options in wordpress or phpbb to get it working?

I'm still on it

Updated :)

not working(session and user sync) it creates user in wp but password not working, email not set in wp(also something strange happens, it changed my admin password and I had to recover it by email)
after new register:
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/ext/wardormeur/phpbbwpunicorn/user.php on line 103: Invalid argument supplied for foreach()
Warning: Cannot modify header information - headers already sent by (output started at /forum/includes/functions.php:3899) in /forum/includes/functions.php on line 5235
Warning: Cannot modify header information - headers already sent by (output started at /forum/includes/functions.php:3899) in /forum/includes/functions.php on line 5235
Warning: Cannot modify header information - headers already sent by (output started at /forum/includes/functions.php:3899) in /forum/includes/functions.php on line 5235

Ok, i just had it, Ididn't know you had specified some association between roles :)
BRB with the solution

Thanks to @Elsensee for his help, this should be fixed. Sorry for misunderstanding this error :/


[BUG] failed to solve with frontend dockerfile.v0: failed to create LLB definition: no match for platform in manifest

Describe the bug
I'm using this project for a couple of months now on my Windows computer without any problems. I now want to make it work on my MacBook too, but I'm running into the following error when I do docker-compose up -d:
Building database
[+] Building 0.8s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                   0.0s
 => => transferring dockerfile: 139B                                                                                0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                0.0s
 => ERROR [internal] load metadata for docker.io/library/mysql:8                                                       0.7s
------
 > [internal] load metadata for docker.io/library/mysql:8:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: no match for platform in manifest

which Branch / PHP Version are you using ?
php8
Steps to reproduce
On a brand new, untouched MacBook Pro (M1 Pro chip):

Clone the repo
Use default env file (rename sample.env to .env)
docker-compose up -d

Expected behavior
A running container
Desktop (please complete the following information):

OS: MacOS
Docker Version: Docker version 20.10.10, build b485636

Noticed that this is a duplicate.


fix: code hidden issue for JS Enabled fields

Description
https://codemirror.net/doc/manual.html#addon_autorefresh
auto refresh was imported but not enabled, this will fix it.
Fixes #13227
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Tested manually by repeating steps provided in issue video

Hello @ganganimaulik 👋 Thank you for contributing to Appsmith.
As this is a bug fix we would like you to add a cypress test to validate the fix.

Hello @ganganimaulik 👋 Thank you for contributing to Appsmith.
As this is a bug fix we would like you to add a cypress test to validate the fix.

added test

/ok-to-test sha=0134db8

@ashit-rath Tested this and it is working as expected now


Fix compatibility issue for Django 3.0

staticfiles template tag library is deprecated in Django 2.1. Change it to {% load static %}.
Quick Reference: https://stackoverflow.com/questions/55929472/

Codecov Report

Merging #671 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #671   +/-   ##
=======================================
  Coverage   73.76%   73.76%
=======================================
  Files          72       72
  Lines        2725     2725
=======================================
  Hits         2010     2010
  Misses        715      715

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d6d26d...5838a3d. Read the comment docs.


UI: Showing Block Size Stats

This PR adds details about block sizes to the blocks UI page.
Each source block row shows an aggregated sum of all chunks and indices of all blocks in this row, respecting the currently selected compaction level.
When selecting a block, the details view shows the following:

total size
index size & percentage of total
chunk size & percentage of total
size per day, based on block total size and duration

All sizes are humanized up to Pebibytes, and display the raw bytes value on mouse over via their title text.
As I wasn't sure about all compatibility, block stats are only calculated and taken into account if

block has .thanos.files array set
individual files have their size_bytes set

At the moment, only chunks and index are counted - meta.json might be present as well, but often hasn't a size assigned to it.
Other rel_path than index and chunks/* are ignored as well.

Fixes #7205

[x] I added CHANGELOG entry for this change.

Changes

displaying size stats (if applicable) for selected block in details view
displaying aggregated block size per source row

Verification

added new tests
ran existing React tests
manual testing proxying a production Thanos instance of 1-2 TiB (~ 2-4 weeks) from about 10 sources

@outofrange this is awesome stuff, as others said... thank you!
If you want another idea for a contribution, here's one: each block stream (indexed from 1 to n) has a set of external labels. Currently we have to click on any block inside a stream to see them. It would be super helpful to be able to see the external labels there together with the block stream's index and size (that you just added). 💡

@outofrange this is awesome stuff, as others said... thank you!
If you want another idea for a contribution, here's one: each block stream (indexed from 1 to n) has a set of external labels. Currently we have to click on any block inside a stream to see them and this is annoying when you are looking for an specific stream. It would be super helpful to be able to see the external labels there together with the block stream's index and size (that you just added), so that you could just use your browser "find text" feature to find the block stream you want. 💡

I created this ticket for further discussions about this: #7237


KubeArmor daemonset pods failing to deploy successfully on x86 Ubuntu EKS-A Baremetal Env.

Bug Report
General Information

Environment description: EKS Anywhere Baremetal
Kernel version (run uname -a) Linux 5.4.0-162-generic #179-Ubuntu SMP Mon Aug 14 08:51:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Orchestration system version in use: 1.27
Link to relevant artifacts: https://github.com/aws-samples/eks-anywhere-addons/tree/main/eks-anywhere-common/Addons/Partner/KubeArmor
Target containers/pods: Kubearmor daemonset pods
Link to Paralell issue in EKS-Anywhere-Addons repo: https://github.com/aws-samples/eks-anywhere-addons/issues/232

To Reproduce

Deploy Kubearmor on EKS Anywhere baremetal environment matching the above via flux

The Kubearmor controller and relay pods start up and enter a 'ready' state, but the daemonset pods for all nodes enter a CrashLoopBackOff state shortly after deployment . The logs for the daemonset Kuberarmor container in the daemonset pods show that shortly after start the container begins to drop events due to replay timeouts. The failing deployment also causes the openebs PVC provisioner to fail to create new claims, preventing other workloads on the cluster from obtaining storage necessary to their successful function and deployment
Expected behavior
Kubearmor agent is expected to fully deploy and report ready
Screenshots

kubearmor.logs.txt

Opening for tracking per the request of a member of the EKS-Anywhere Partner Addons team
@daemon1024  @rksharma95

@daemon1024 Is this fixed?

Hey @elamaran11 , Yes this is fixed in our recent releases. Hope we can get https://github.com/aws-samples/eks-anywhere-addons/pull/235 merged to upgrade our version on EKS-A

@daemon1024 Thanks for the confirmation. we will be validating this soon. Thanks!

@daemon1024 Thanks for the attention on this issue! We've validated the updated PR and have just merged the PR that fixed the issue.


bw file output choice

It's always problematic to create too many alternative output files by default (see TDF thread), but in the case of bw files it may make sense to create at least two:

including multimappers, excluding blacklisted regions, scaled
excluding multimappers (MQ>0), excluding blacklisted regions, scaled

then I previously used a "lowres" ("supersmooth") version for IGV visualisation since IGV has a problem with visualising regions with scarce reads properly. There is no easy way to create a smoothened bw file out of an existing bw file unfortunately (all bwtools, WiggleTools etc can read bw but not write a bw). If there was I would say we can always create it ad hoc.

including multimappers, excluding blacklisted regions, smoothened and binned, scaled

Regarding blacklisting, there are in principle two ways, 1) remove reads from BAM file, 2) remove regions during bw file generation. I would prefer option 2) since then the BAM file is untouched and can be used for any purpose including to read out repetitive region enrichments. However, bamCoverage crashed on me when I tried to specify a blacklist file before. Please see if it works for you.

I am closing this one also, the multimapping was now fully adressed in #173 , blacklisting was already included and the smoothing does not always have the values we expect. Smoothened bigWigs we rarely generate and they can always still be manually generated from the BAM.


Feature Request: Jamf Pro 10.46.0 provides API access to management account name and password

Change in Jamf Pro 10.46.0 … Local Administrator Password Solution (LAPS) provides API access to look up an existing account and password.
Obviously we would still need to make sure the account has a Secure Token, but that is relatively easy to do, this would enable super to collect a known changing password and use that to trigger softwareupdate.

Not saying no... but this would significantly increase the potential attack surface for whatever Jamf API account is stored by super in the System Keychain. There is potential to gain an admin password for not just the local computer... but for every computer in an organization.

Thankfully admin access is required to look at that keychain entry, but I agree, that said, the security issues are something Jamf will need to consider regarding their API.
I had written something more, but you have given me an idea for a decent feature request for Jamf, for the jamf command, be able to do something like jamf adminLogin softwareupdate --install --all --restart, and then the jamf command could on Apple Silicon Macs, automatically add or pass the admin login credentials to softwareupdate, and perhaps the same for sysadminctl as well as any other command we could have it handle. Perhaps adding something from the policy to allow a script to use the command (for sysadminctl).

@wakco
I have been trying on Apple Silicon Macs to get the MDM workflow for upgrades to work, so far unsuccessfully and I am willing to believe this is mainly down to Apple. ☹
As such I am now considering the possibility of using a local admin account like you appear to be. I have recently joined an organisation and currently they have the same credentials on all Macs which is on my list of things to fix, i.e. to implement LAPS. I have had a brief look at Jamf's new LAPS feature but I get the impression it is not adequate yet to the purpose. I have previously used a script which stores the LAPS password in an extension attribute.
It looks like it might be possible to use the value of this extension attribute as a parameter to a policy script parameter. If so I could then run the Super policy with parameters which pass the LAPS user name and password to Super during deployment and allow super to then create a local account.
According to the Jamf docs, this is done by using a Jamf variable in the script parameter so it might looks something like the following -
--admin-password=$EXTENSIONATTRIBUTE_5
Note: The 5 refers to the number of the extension attribute so will be dependent on your setup.
I have a string suspicion however that the eventually (hopefully) created local credential will lack the required volume owner etc. permissions as Apple for example make it difficult to script giving a secure token to a script created user account. (Another Apple issue.)

You can not set the value of a script parameter via that mechanism. It's only designed for Configuration Profiles.

@Macjutsu
The $EXTENSIONATTRIBUTE was to read the LAPS password for the Mac that Jamf was running the policy and script on, not to set it. It would be set by the LAPS script using the Jamf API.
The idea was to read the LAPS password from the extension attribute and pass it to Super. As mentioned I have previously used a LAPS script which stored the password in plain text in the extension attribute and hence should work as is. However these days it is considered better to store the password in an encrypted form even though you have to be an approved Jamf user to access it.
I have had a look at some other LAPS solutions for Jamf and the following one looks better to me.
macOSLAPS
The author has provided an example script to retrieve and decode the LAPS password. In theory Super could be enhanced to use the $JSSID to look up the relevant extension attribute and decode its contents and then have the correct LAPS password for the Mac in question and hence be able to use it to do the required authentication locally.
LAPS Decode Password Script
The script will need some modifying for this purpose to make it fully automated and hence remove some of the interactive aspects. (Basically strip out everything except reading and decoding the extension attribute.) I could myself build this in to Super but I would have to do this each time you updated it. Ideally Super would be modified to have an additional script parameter e.g.
--use-macOSLAPS
and
--use-macOSLAPS-off
And accordingly use this or not. It should be possible to put this modified script in the Super folder and call it based on this additional parameter. I could create the modified script for you if you are willing to add it to Super.

I am aware of LAPS solutions... but there are two fundamental security problems with leveraging them in any local process or script like super.

Just because you have a local admin account, doesn't mean it can authenticate a macOS update/upgrade. The account requires Volume Ownership permissions which can not be programmatically given to a new account without first providing the credentials of an existing Volume Owner user. So even the act of creating a LAPS account capable of performing a macOS update/upgrade can not be automated. It's my understanding that the new Jamf LAPS solution will suffer from this same macOS security design choice.
In order to lookup any EA value super must authenticate to the Jamf Pro API... this means that super would be storing that credential in the System Keychain. Because of Jamf Pro's security model, that credential, if discovered, would allow anyone to have full read access to your entire Jamf Pro inventory database. Obviously, this would be a significant security risk.

Yes... but due to the limitations of Jamf Pro's security model... the permissions required for the Jamf Pro API account used by super to read EAs would also allow it read EVERY SINGLE EA for ALL of your computer records.
And it doesn't matter if you have added the extra (technically not actually more secure "encrypted" strings) method, it still doesn't matter because super would have to store the decryption secrets in the Keychain as well.

@Macjutsu
I see your point about Super directly reading EA fields.
However I think if we used as a script parameter something like -
--admin-password=$EXTENSIONATTRIBUTE_5
This would not require giving the SuperApi credentials full EA access and the content of the designated EA would be passed to your script. A Jamf admin would like all the other script parameters specifically set this up so it cannot be auto scraped and it would also be for a specific script i.e. Super. Furthermore the content of the EA in this case would be in encrypted form and would need decrypting in your script to utilise. This likely will require using two script parameters one for the encryption key and one for the encrypted admin password as above. Basically the same way that the macOSLAPS script works except your script would not have full EA access.

This is NOT something Jamf Pro can do... As I have already stated earlier....
"You can not set the value of a script parameter via that mechanism. It's only designed for Configuration Profiles."
Again... sending the value of Jamf Pro attributes is only possible with CONFIGURATION PROFILES.

@Macjutsu
Ok thanks for the correction. Whilst technically it would be possible to send the encrypted password to the Mac as part of a profile this is not going to be desirable even though it is encrypted.
I will consider writing my own script to retrieve the encrypted password - via the Jamf API, decode it and call super with command line parameters to allow it to create the service account.

Another issue with the configuration profile is that Jamf Pro does not send new configuration profiles when the value of a $VARIABLE changes. So when you cycle the LAPS password you would have to push a new configuration profile manually or with some other external automation.
This last point would be the best approach, as long as you don't store any data on the local drive while the pre-script is running.

What I ended up doing is use the same component that populates the LAPS Extension Attribute in a script to pass local admin creds to super to create the super service account. If for some reason LAPS hasn't been setup on a particular Mac, it pivots to deploying super using the MDM workflow. It's been working out well for me so far.

This is a good method if your plan is to only give super the admin password long enough to create it's own service account. Further, since it only needs to be run once, keeping it as a separate "installation" script makes sense as you wouldn't need any of this code in super itself.
However, how does using $JSSID limit the permissions for the API account? Perhaps for the execution of this single script it will... but not for the API account itself. If the API account has permissions to read EAs... it can read ALL EAs... even if this particular script only reads a single EA.
In other words, if this script is using the same Jamf Pro API account that is also passed to super, then a local admin could easily access the Jamf Pro API credentials that super stores in the System Keychain and then gain access to all of the data stored in every computer's EA value.

I had misunderstood your use of the $JSSID so no, it slightly helps reduce access but not materially.
As it happens, in order to read extension attributes you have to have read access to the computer record(s). The only way I can see to further secure this with respect to your concern would be either -

Use different dedicated API credentials for this purpose which Super itself does not know and hence is not stored anywhere on the Mac, if this is done using additional command line parameters it would increase the likelihood of running out of fields
Use different dedicated API credentials and hard code them in to the script.

It should be noted I went to great effort with my initial script design to provide masses of flexibility for LAPS support but only need a single extra parameter field.
Regarding the keychain storing Super API credentials.
I seem to recall seeing some software products create their own dedicated keychain with their own password to that keychain so only their software can access it.

super could create another keychain, but then where would it store the password for that keychain?
Any "secret" method super could use would be visible to the public due to its open source nature. As such, we literally couldn't hide access to that second keychain.
This is why most processes use the System Keychain... it's the only keychain that doesn't need a password... just admin/root access.

@Macjutsu
For what it's worth, I found an error in the last few lines of my proposed script.
Where I pass the regenerated command line parameters to the real super script this needs changing to the following to ensure the parameters are handled properly.
curl --silent -o /tmp/super -L -O https://github.com/Macjutsu/super/raw/main/super
chmod +x /tmp/super
array=($commandPARAMS)
mycmd=(/tmp/super "${array[@]}")
"${mycmd[@]}"

I can see that my script now successfully launches the real super script. I am not convinced however that Super is doing anything.
It seems to suggest the next scheduled event is right now except nothing ever happens. Super.log attached below.

If you post this to your own repository I can link to it in the Wiki.

@Macjutsu I can do that, but I would like to resolve the two queries first.

What would error 137 as per signify?

Error running script: return code was 137.

Any idea why the dialogs failed to appear but it did do an upgrade. It was a different Mac that did the upgrade so will not be in the above two logs.

This is a Jamf Pro issue: https://community.jamf.com/t5/jamf-pro/issue-running-script-from-policy/m-p/260226

@Macjutsu
Based on that Jamf article are you advising I add code to my script to call the real Super script via the logged in user? Or to create a new fresh policy with my unaltered script?

From the solution, "I was able to fix this by just creating a new policy, copying everything over and deleting the old policy. Something must have become corrupted or sticky in that old one that was causing all the issues. Go figure, had nothing to do with the script."
Sometimes things in Jamf Pro become corrupted and this results in error 137. It's resolved by simply re-uploading the scripts and creating fresh Policies with the same settings.

@Macjutsu
Thanks for the help.
I did make a brand new policy, I made a new script entry in Jamf Pro - although I copied and pasted the script itself and tried again, it still gave an error 137. I then made a new policy again, a new script but this time just the portion of my script that calls yours with hard coded $commandPARAMS (see below) - most of the script is building the commandPARAMS.
This stripped down script worked without an error 137 being triggered. :(
Anyway, whilst I myself continue looking in to this, here are two other questions. I am also getting an error as follows.
Mon Jun 19 09:55:39 G4-0001257 super[76778]: Exit: LaunchDaemon com.macjutsu.super.plist is scheduled to start right now.
/Library/LaunchDaemons/com.macjutsu.super.plist: Service is disabled
Bootstrap failed: 119: Service is disabled
Mon Jun 19 09:55:39 G4-0001257 super[76778]: **** S.U.P.E.R.M.A.N. 3.0 EXIT ****

My stripped down script is as follows and whether run locally or via a Jamf Policy event gives the 119 error.
#!/bin/sh

commandPARAMS="--jamf-account=superapi --jamf-password=password --admin-account=ladmin --admin-password=password"

curl --silent -o /tmp/super -L -O https://github.com/Macjutsu/super/raw/main/super
chmod +x /tmp/super

array=($commandPARAMS)

mycmd=(/tmp/super "${array[@]}")

"${mycmd[@]}"

I am also attaching below a copy of the Super.log from a Mac that did successfully run the Super upgrade cycle but did
not show any dialogs or prompts. So it 'suddenly' rebooted.
super.log

@Macjutsu
FYI - I have been working with the author of the macOSLAPS solution I am using and he has added the ability to work with existing LAPS accounts all using the same credentials and then take over randomising them. I have been working on a script to then read those details and pass them to your Super script.
I think this is pretty much working, I am now modifying my script to use a different Jamf API account so Jamf calls my script, my script uses the passed API credentials which are not stored and my script passes the normal SuperAPI credentials and the device specific local admin credentials to your Super script. I think this as much as possible prevents global access to credentials since the SuperAPI credentials cannot read the extension attributes and the API credentials which can are not stored on the client Mac.
I have created my own repo for this although I have not yet finished with testing.
https://github.com/jelockwood/Super-Glue
I have noticed you have recently updated the recommend API settings and added more access to Managed Software Update Plans. It could be that this might help my own experience and why I have felt I needed to enable failover to local admin credentials.
As a separate issue, I do get the impression that even after I have successfully installed your Super script, it only seems to run once and not after a successful macOS update/upgrade run again even if a new version is available. As it happens the way I will update the credentials for Super after the macOSLAPS has changed them should get round this by effectively re-installing and starting Super again. Looking at your launchdaemon I see it is set as launch only once so this might explain matters.
How is Super supposed to automatically re-check for new updates? Does it rely on the user regularly rebooting? What happens if a user does an update which does a reboot, the check sees no other updates and then Super 'quits' and the user does not do another reboot for a number of days? It would appear no further checks happen.

Adding the tag of "Help Wanted" to indicate that unless a safer method is found, I won't be working on this.


Added: Error Page

Pull Request
Related Issues:
Added 404 page
Fixes #335
Description:
[Provide a brief description of your changes.]
Checklist:

[x] I have tested my changes.
[x] My code follows the project's coding standards.
[ ] I have updated the documentation (if applicable).

Screenshots:

Please add KWOC tag!

@AaadityaG the 'KWoC' tag has already been added in the issue you are working on. Dont worry

@AaadityaG your PR is facing problem while deployment

@AaadityaG seems like the deployment related issue from my end. Please wait while I check the reason behind the issue.

@AaadityaG seems like your code has not followed the ESLint rules and hence the PR is not deploying successfully. Please commit your changes after solving the ESLint errors properly.

@ani1609 bro any idea/suggetion how can I fix this?

@AaadityaG pretty easy, you just need to follow up the ESLint rules and those are very straight forward

@AaadityaG Feel free to reach out to me on our Slack channel if you need assistance or guidance to resolve the issue. I'm more than happy to help you out!

raise new pr


LavinMQ crashing during startup because of "Lease lost "

Describe the bug
LavinMQ failed to start on a single node cluster with a "Lost lease" error from lease_keepalive.
Nov 07 10:35:48 systemd Starting LavinMQ...
...
<launching, loading data...>
...
Nov 07 10:35:55 lavinmq lmq.etcd Lost leadership of lavinmq/leader: Lost lease<PHONE_NUMBER>974389534
....
Nov 07 10:35:58 lavinmq lmq.launcher Lost leadership lease
Nov 07 10:35:58 lavinmq lmq.launcher Stopping

This unexpected shutdown during the startup then caused other errors in the startup, and left the LavinMQ service in an inactive (dead) state.
Logs from after "Stopping":
Nov 07 10:35:58 lavinmq Unhandled exception in spawn(name: HTTP listener): Can't re-start closed server (Exception)
Nov 07 10:35:58 lavinmq   from usr/share/crystal/src/http/server.cr:457:5 in 'listen'
Nov 07 10:35:58 lavinmq   from usr/share/crystal/src/fiber.cr:143:11 in 'run'
Nov 07 10:35:58 lavinmq   from ???
Nov 07 10:35:58 systemd Started LavinMQ.

Service status
lavinmq.service - LavinMQ
     Loaded: loaded (/lib/systemd/system/lavinmq.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Thu 2024-11-07 09:03:28 UTC; 4min 58s ago
       Docs: https://github.com/cloudamqp/lavinmq
    Process: 988772 ExecStart=/usr/bin/lavinmq -c /etc/lavinmq/lavinmq.ini (code=exited, status=0/SUCCESS)
   Main PID: 988772 (code=exited, status=0/SUCCESS)

Describe your setup
One single node running ubuntu 22.04.
LavinMQ v2.0.0 and etcd 3.5.16 running on the same machine.
How to reproduce
Have not been able to reproduce it again.
Expected behavior
LavinMQ starts, gets lease from etcd and keeps running.

The problem is probably that the lease TTL expires during startup if the startup takes more than 5s, leading to there being no lease when LavinMQ has actually started.

Fixed by #834


[PoC] Work on task containers from the frontend

I want to be able to spin up task containers and solve tasks from the frontend. Ideally there would be some kind of list of available tasks, with a "solve" button next to them (or something like that), which when clicked would spin up a task container (or use a preexisting one if available) and also display a terminal which I could use to solve the tasks. There would also have to be a "submit" and potentially "grade" button there.
This PR is a tentative first stab at it, mainly as a way of getting feedback whether this even makes sense to be doing. It currently looks like this:

but would of course require a lot more fleshing out and prettifying.
The way this works is that the frontend sets up an xterm object that comunicates with the backend via websockets. The backend directly runs docker exec /bin/bash and acts as an intermediary via the websocket. This seems to work quite well, though there are a couple of things that would have to be ironed out.
Feel free to just close this if you think it's a waste of time.

Could you tell me a bit about the benefits of this compared to using the viv CLI? That's current way to start task environments, get a terminal inside them, and grade submissions for them.

That's fine, but requires having it all set up. This would allow for qa etc. by just pointing someone at an url and letting them have at it.
So it's mainly a matter of convenience, but also would limit access, as here all that is needed is access to the viv UI (+ a bunch of other stuff in the background, of course). It would be nice to be able to look at tasks just from the browser.
I was also thinking this might be nice to have attached to runs, as it would make it easier to mess about on the agents container for quick checks and what not

OK makes sense. I see how this is more convenient than installing the viv CLI.
On the other hand, I get the sense that human baseliners want to use tools like VS Code when accessing task environments. They want to use something more powerful than a shell.
Re: limiting access, right now I consider Vivaria UI access to be more sensitive than SSH access to a VM host. To give more detail: If we wanted human baseliners to use this UI, we'd probably want to add a way to prevent them from seeing any data about any other task environments or any runs.
So personally, right now, I feel like the benefits of this feature aren't worth the costs of implementing and maintaining it.