[FR] RAG: Add support for Int8 embeddings

It would be great to have support for embeddings compressed to Int8 as per HuggingFace: Embedding Quantization.
Potential implementation would be to:

define an embedder (<:AbstractEmbedder for get_embeddings) and the corresponding finder (<:AbstractSimilarityFinder for find_similar)
Both would have the vectors with necessary min_values and max_values fields to hold the effective range for each embedding dimension (eg, length(min_values)=length(max_values)=D)
define methods for these types
The conversion to Int8 could be done post hoc (after build_index) via a utility function and then the resulting finder with the range to allow converting to Int8 (to be provided to the airag)
It should implement the two-stage pass with rescore_multiplier=4 (first on Int8 embeddings, then with Float x Int8)

I am going to take care of this one.
I would happily help to move the RAG functionality into a separate package too. Let me know if you want to move forward with that.

I was hoping to do the RAGTools migration after we merge in the Pinecone support, I don't suppose you would be interested in finishing that?
On the Int8, cool! You can probably re-use a lot from the "bitpacked" embedder. I don't mind if it's signed or not.
On the dep addition, where do you see the benefits to outweigh the costs. It's just a minor performance tweak (no big gains in any direction compared to what we have already), so I'm not sure we would need to support more than one simple implementation of this. Do you have a different view?

I would like to implement this first, since I have already invested some time on it. I can take care of the Pinecode support afterwards, it that's stopping the RAG package from being born.
LinLogQuantization.jl implements exactly what we need and nothing else. It's a very small package (less than 300 lines of code) and I cannot see a simpler implementation of linear quantization. In my opinion, anything else than using the package would be wasting effort in reinventing the wheel.
If you really want to avoid the dependency, we could take only the part of the package that implements linear quantization to avoid adding the code for logarithmic quantization.

By the way, here's a more detailed explanation of scalar quantization. It's a reference in the article you provided

Sorry for the slow response! I was at a hackathon the whole weekend.
I don't think it would be appropriate to add LLQ (with StatsBase as a dep) as a direct dependency of PromptingTools for everyone.
RAG is used by only a subset of PT users, within that subset only a few users will ever look at quantization, within that picking Int8 is quite a niche (the trade-offs are quite nuanced and it's probably not worth it for most).
In addition, if we'll only ever have Int8 (I don't see any benefit from having more Int versions - there are more low-hanging fruits to get performance), it's just 2-3 functions we need, so it's a very simple problem to solve directly.
If you still insist on using the LLQ package, I'd ask you to add it as an extension (weak dep). Then I'm happy to review the PR.

Understood.  I have extended the package for support linear quantization of signed integers (see PR). I can copy the two necessary functions to implement the Int8 index without adding the package as dep.


MudDataGrid : Fix Resize column when neighboring is hidden

Description
When a column is resized and the neighboring column is hidden, then the application crash.
Fixes #9585
How Has This Been Tested?
I added a test case that reproduce the bug.
Type of Changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation (fix or improvement to the website or code docs)

Checklist

[x] The PR is submitted to the correct branch (dev).
[x] My code follows the code style of this project.
[x] I've added relevant tests.

The fix work in a demo project, but not in the test.
In test, ElementReferenceExtensions.MudGetBoundingClientRectAsync return always null for HeaderCell.
I don't know why.

I just see a difference in the tag of the header :
In the demo (where it works) :

In the demo (where it don't work) :

Sound like blazor:elementReference isn't translated in the test.

Does anyone have any hints?

The test display a data grid with 3 columns. When a column is hidden, the element reference on  disappears :
...
// Open column menu
var columnMenu = comp.FindAll("th .mud-menu button").ElementAt(1);
columnMenu.Click();

// Click on 'Hide' menu item
comp.WaitForAssertion(() => comp.FindAll(".mud-list-item").ElementAt(1));
var hideMenuItem = comp.FindAll(".mud-list-item").ElementAt(1);
hideMenuItem.InnerHtml.Contains("Hidde");

var thBeforeHidde = comp.Find("th").OuterHtml;
//
hideMenuItem.Click();
var thAfterHidde = comp.Find("th").OuterHtml;
//

In test, ElementReferenceExtensions.MudGetBoundingClientRectAsync return always null for HeaderCell.
I don't know why.

MudGetBoundingClientRectAsync calls a JavaScript method mudElementRef.getBoundingClientRect, since bUnit doesn't support Js invocation, you need to mock it to return whatever you desire:
Context.JSInterop.Setup<int>("mudElementRef.getBoundingClientRect").SetResult(new BoundingClientRect(...?));

More info here: https://bunit.dev/docs/test-doubles/emulating-ijsruntime.html

@ScarletKuro, thank. It's worked and it progress.

Now, I need to simulate the mouse move.
The datagrid listen the global event document.pointermove by EventListener.
Have you a hint/example to mock it?

I don't think you can really simulate the pointer* events in bUnit: https://github.com/bUnit-dev/bUnit/discussions/1497
You need to do it on a different level, either on the DataGridColumnResizeService or the IEventListener.
As I understand the datagrid subscribes to event:
https://github.com/MudBlazor/MudBlazor/blob/e279503b602a031e7c433354b162f837dc0972a6/src/MudBlazor/Components/DataGrid/DataGridColumnResizeService.cs#L67-L68
And then the EventListener fires the OnEventOccur when the even acquires:
https://github.com/MudBlazor/MudBlazor/blob/e279503b602a031e7c433354b162f837dc0972a6/src/MudBlazor/Services/EventManager/EventManager.cs#L81-L82
And it invokes this lambda OnApplicationPointerMove(eventArgs, rightToLeft).
Technically you could just simulate the OnEventOccur similarly on how I'm doing it with the IBrowserViewportObserver:
https://github.com/MudBlazor/MudBlazor/blob/e279503b602a031e7c433354b162f837dc0972a6/src/MudBlazor.UnitTests/Components/HiddenTests.cs#L89
I'm just invoking the method directly in the test and the RaiseOnResized is also an [JSInvokable].

I think last commit contains things that it shouldn't.
I see you have a separate project for testing, I think in future you can use the MudBlazor.UnitTests.Viewer directly, if you add a test component with XYZTest naming, you can start this project and find this test and play with it.

I think last commit contains things that it shouldn't.

My mistake... I forget to select files before commit... so demo projects were pushed. It's cleaned.

Finally, the test reproduce the bug.
@ScarletKuro, thank for the help.

@ScarletKuro, thank for the help.

Thanks for the fix. That fix makes sense, all read and writes should be done via the hidden state not the parameter, and apparently there is one more place where parameter is used instead of the state.

@ScarletKuro, nice catch.
Column.Hidden is also used in SelectColumn.razor and HierarchyColumn.razor.
The razor template need to use the parameter property or the parameter state field?

Column.Hidden is also used in SelectColumn.razor and HierarchyColumn.razor.

This ones are fine.

Thanks.
It's interesting that EventListener is transient. I would make it a singleton so that if two components subscribe on same event, you could have one subscription on the JavaScript side and two on the C# side, rather than a 1:1 ratio. Something similar does the IBrowserViewportObserver.
It would allow to use:
var eventListener = (EventListener)Context.Services.GetRequiredService<IEventListener>();

in tests.
But that's a totally different topic.


Support Databricks struct literal

Databricks supports struct literal (struct function). Struct literal is already added for BigQuery. Databricks' syntax is very similar to the BigQuery one, except the fields definition part.
I added supports_struct_literal to Dialect trait to specify if the dialect support this literal or not
Databricks docs: https://docs.databricks.com/en/sql/language-manual/functions/struct.html

Thanks @ayman-sigma and @iffyio


Correct usage of getSideBar in config to an array of themes

Themes is now an array of objects, and not just one object

No this is wrong, sorry. There must be something else wrong as I can not get the sidebar to work. It produces an array of objects but it is not displayed in frontend using "vitepress": "^1.0.0-alpha.64"


Post-build step for asset canister fails because directory is considered to be outside the workspace root

My dfx.json looks like this:
{
  "version": 1,
  "dfx": "0.9.2",
  "canisters": {
    "myproject": {
      "type": "custom",
      "build": "nix build",
      "candid": "backend/src/myproject/src/myproject.did",
      "wasm": "result/target/wasm32-unknown-unknown/release/myproject.wasm"
    },
    "myproject_assets": {
      "type": "assets",
      "dependencies": [
        "myproject"
      ],
      "frontend": {
        "entrypoint": "result/dist/index.html"
      },
      "source": [
        "result/dist/"
      ]
    }
  },
  "networks": {
    "local": {
      "bind": "<IP_ADDRESS>:8000",
      "type": "ephemeral"
    }
  }
}

There's apparently some issue with using "result/dist/" as the source directory because when I run dfx deploy myproject_assets I see Error: The post-build step failed for canister 'ryjl3-tyaaa-aaaaa-aaaba-cai' with an embedded error: Directory at '/Users/py/projects/paulyoung/myproject/main/result/dist/' is outside the workspace root.
I suspect this is because result is actually a symlink to /nix/store/<hash>-myproject. I'm working around this for now but it might be nice if people could opt out of this check.
Also, is there a way to specify a build command for asset canisters? The above relies on the dependency of myproject to put the assets in the right place. It would help with my workaround if I could decouple the two.

My workaround is as follows.
"build": "bash -c \"nix build && rm -rf result-assets && mkdir result-assets && cp -R result/dist result-assets && chmod 775 -R result-assets\"",
I'm using bash -c because otherwise is treated && as a command and I get an error complaining that it doesn't exist.
It would be great if things like foo && bar just worked.

Hi Paul, I feel your pain! We have reworking the custom build command on our backlog and know the current state is mediocre at best. Thanks for giving good examples that we can test against!

I think I saw some code that looked like it might try to write to the provided directory as well. If that's true then that would have been another problem I would have run into.

@sesi200 I saw some PRs mentioning this so I wondered if I had missed anything that might allow me to work around this now.

I think it would help if "build" was allowed when "type": "asset".
I'm surprised that doesn't work; the assets need to get built somehow, and having something like "build": "npm run build" seems very reasonable.

It currently runs npm run build for asset canisters, but there's no reason we can't make that customizable


Automatic rush update --recheck

This is an automatic PR generated daily with changes from running the command rush update --recheck

We did. Latest run of aggregate report doesn't generate this PR. But this was created few days back. https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1080408&view=logs&j=0874ffeb-2919-5b4b-20de-16a97970b94c


Cannot build

Version
d3f0571c19ada5113a1add8313190ecb6ab1eab
Details
After clean clone of the repo I cannot build it :(
Steps to reproduce
Simple clone repo, then either build with VS 2022 or with dotnet build

Remove these lines from the csproj for now:
https://github.com/Tyrrrz/YoutubeDownloader/blob/2d3f0571c19ada5113a1add8313190ecb6ab1eab/YoutubeDownloader/YoutubeDownloader.csproj#L31-L34
I will fix it later


Backport Honor context termination

This backports https://github.com/containerd/containerd/pull/336
Commit: 034682e4db9326b9a2d1659f53aac0dddf77fdbb
Signed-off-by: Brian Goff<EMAIL_ADDRESS>

ping @crosbymichael

LGTM
Looks the same as the source commit.


Move to C++17

As noted in https://github.com/tier4/scenario_runner.iv.universe/pull/7#discussion_r517464547, requiring C++17 may help us get rid of some dependencies (namely Boost). Let's evaluate the feasibility of using C++17 after the ROS 2 port is done.

I know C++17 can remove a lot of boost dependencies and I'd love to use it, but I have one concern.
Can Autoware.Auto accept C++17? If not, I'm worried about we'll need to re-replace C++17 with boost when we contribute to Autoware.Auto.

@dejanpan Do you have any comments?

@esteve @kenji-miyake @mitsudome-r I provided a slightly longer answer here: https://github.com/eclipse/iceoryx/issues/220#issuecomment-724952814.
Since Adaptive Autosar Guidelines are not used in Autoware.Auto, I think that we could switch to C++17.
There are however 2 things that I am still uncertain about:

https://stackoverflow.com/a/49119902:

Where you have problems is if you link together objects compiled with different versions of GCC and you have used unstable features from a new C++ standard before GCC's support for that standard is complete.

and the other question is - do all the static code analysers and linter tools that we use from ROS 2 understand C++17?

I'd appreciate your comments.

@esteve Could you comment about the two concerns that @dejanpan pointed out?

@dejanpan @mitsudome-r if we agree to allow C++17 in our code, then we should work on https://github.com/tier4/AutowareArchitectureProposal/issues/106 to make sure that we don't use any unstable feature to avoid any ABI issues. It shouldn't be a problem, because we ourselves don't use anything from std::experimental anyway. However, any dependency that in fact uses an experimental feature may be affected, but just need to ensure that we're using the same compiler as the one used for building the ROS 2 binary packages.
I looked into the tools we use:

uncrustify supports C++17 (e.g. https://github.com/uncrustify/uncrustify/blob/master/src/options.h#L480-L482)
clang-format supports C++17 (it uses Clang as the backend)
cpplint should support C++17, but there's no clarification. I say should because it follows the Google style guide for C++, which targets C++17
cppcheck supports C++17 since at least version 1.88 (https://sourceforge.net/p/cppcheck/news/2019/06/cppcheck-188/)

@esteve thanks and works for me.
Now I do want to make also 1 thing clear: we agreed with T4 to port https://github.com/tier4/AutowareArchitectureProposal to ROS 2 as fast as possible and to leave the code as identical as possible as it was in ROS 1 (so no tests, design docs, documentation, refactoring). However as soon as we will finish the simulation packages, we will stop the work here and move to a proper porting in Autoware.Auto. That is also where I'd switch to C++17 and not in https://github.com/tier4/AutowareArchitectureProposal.
@kfunaoka @mitsudome-r


[No QA] Use pretext in chrerry pick workflow to tag mobile-deployers

cc @roryabraham
Details
In https://github.com/Expensify/App/pull/4829 we added a message to tag mobile deployers when a CP fails, but slack doesn't tag properly when adding @mobile-deployers to the text value:

This PR adds the mobile-deployers tag via pretext like we did in https://github.com/Expensify/App/pull/4767
Fixed Issues
Fixes https://github.com/Expensify/Expensify/issues/175742
Tests

Once deployed, attempt to CP an issue instead of a PR 🤯
When the CP fails, verify that it was announced in the #announce Slack channel🤞

QA Steps
None

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by @roryabraham in version: 1.0.90-3 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by @roryabraham in version: 1.0.91-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
failure ❌

🕸 web 🕸
success ✅


Run benchmarks on every PR

The motivation is that performance changes should be visible by default (as opposed to manually triggering the benchmarks).

When benchmarks are run on the master, the results will be saved to the cache.
On PRs with benchmark label, benchmarks are run and compared to the base branch as the baseline.
Metrics are now exposed to Github job summary, so one doesn't have to search through logs.

I am not adding benchmark label, as the job will fail. Most likely we need to merge this first, so permissions are propagated correctly. Here is the preview: https://github.com/xThaid/coreblocks/pull/2#issuecomment-2054193764
Fixes #657.

I removed benchmark label as it will trigger the job that issues a comment in this PR, which will fail due to lack of permissions. I believe that putting this modified workflow on the master branch should solve permission issues.


modified ui changes on features and scaled bosque on footer to make i…

Pull request for issue #51
Additional changes (mobile):

Footer's .bosque is now scaled to 1.5 to make it a little bit bigger
Footer's margin top is now decreased from 400 to 200 for shorter scrolling on the page, just a little adjustment
.homeBackground component's margin bottom is now 100px

@eyedeceiver03 Thank you very much for the contribution

You're welcome @luisDanielRoviraContreras !! 😄 🎉


Make ConventionBasedStartup internal

ConventionBasedStartup was made pubternal because it depends upon a pubternal type. We think this should actually just be made internal if possible.
https://github.com/aspnet/AspNetCore/blob/master/src/Hosting/Hosting/src/Startup/ConventionBasedStartup.cs
cc @davidfowl @Tratcher

Only if we're going to do the rest of Hosting.

Sure why not.

This was done in 3.0. We just missed the issue.

https://github.com/aspnet/AspNetCore/pull/11505


proposal: Go2 error handling for error-tree

Please add into wiki Go2ErrorHandling, my proposal for error handling in error-tree. Preview:
func printSum(a, b string) error {
  // create error tree
  et := errors.New("check input data")
  errorhandling(et interface{Add(error)}){ // add type just for clearification
    x, et := strconv.Atoi(a)
    y, et := strconv.Atoi(b)
  }
  // is error-tree have errors
  if et.IsError() {
    return et
  }
  fmt.Println("result:", x + y)
  return nil
}

In detail see: Go2ErrorTree

CC @jba @neild

It looks like there are two ideas here. One is an "error tree" type that can collect multiple errors. It seems that this type is itself an error, but it can also represent the state of there being no errors. So it is already incompatible with the basic property of Go error handling, that there is an error if and only if err != nil.
The second idea is some new syntax for automatically collecting errors into an error tree. That results in simpler control flow (no if err checks) for code where several actions are taken independently. I believe that is rare, however; it's much more common for an action to depend on the result of a previous action. In those cases, the special syntax will obscure bugs. For example, in this code:
errorhandling(et interface{Add(error)}) {
   a, et := f()
   b, et := g(a)
}

we erroneously pass a garbage a to g when f returns an error.

Yes, @jba, you are right. Syntax errorhandling is work fine, only if expessions inside are independancy.
And you example looks like(it is valid code see https://play.golang.org/p/xl5dRJgvKqU):
  a, err := f()
  // no error checking
  b, err := g(a)
  if err != nil{
    return err
  }

but in that case we will not see error from function f(). If we use errorhandling, then error will garantee added in error-tree.

@Konstantin8105 what are the problems and solution requirements that motivated your proposal?
FYI, I've collected a menu of Requirements to Consider for Go 2 Error Handling.
Also, please feel free to add a link to your proposal to the wiki:
https://github.com/golang/go/wiki/Go2ErrorHandlingFeedback

Now, I add to wiki https://github.com/golang/go/issues/32099.

Thanks for this proposal.  This combines error handling and error values into a single proposal.  But the idea is somewhat complicated: it requires a builtin type more complicated than error, and it requires additional control within the errorhandling construct.  It also seems to implicitly call an Add method, which is not something we ever do in the language today.
We aren't going to adopt this idea.


The full spell of quad in SigProfilerExtractor/SigProfilerExtractor/data/CN_classes_dictionary.txt

Hi. It wil be termed quad when the copy number is between 5-8. I wonder if you can provide the full spell of "quad".

The names for the channels have been set according to our preprint. If you want to modify these you will have to fork the repository and change the relevant lines of code.
Thanks,
Azhar


Help files improvements

The help files (html) are derived from our Open Roberta Wiki, Here we only have two sites (one in German, one in English) which describes all common blocks. A common block is each block that does not appear in either the Action, Sensor or Message categories.
Some supported Systems, like e.g. WeDo or Bob3, have not implemented all of these common blocks. For this we will add a special CSS class, starting with "not" and followed by the groups name of the systems, e.g. "notWedo", and set them to be invisible.

i am unable to locate these help files. Can you help me with their exact location?

i am unable to locate these help files. Can you help me with their exact location?
@phoenix-kd

Changes are made in this commit. link
Files are

OpenRobertaServer/staticResources/help/progHelp_bob3_de.html
OpenRobertaServer/staticResources/help/progHelp_wedo_de.html
OpenRobertaServer/staticResources/help/progHelp_wedo_en.html
OpenRobertaServer/staticResources/js/app/roberta/controller/progHelp.controller.js

Made some changes to the files here
Open to suggestions
PS: Had to submit an early incomplete PR as I am applying in #GSoC2020 and took this as a first issue to work on!

The help files themselves are autogenerated from the wiki, which is maintained by one of our colleagues.
As for the aspects stated in the issue, @bjost2s is working on those.
For general proposed changes to the help files (and thus the wiki) like typos or grammatical errors you can send us any issues you find and we will pass it on to our colleague to incorporate into the wiki.

Help files for BOB3 and Wedo show up now.


Create a prior art and alternatives section in readme

Make
https://github.com/jakedeichert/mask
MS Just
https://github.com/egoist/maid
https://github.com/sagiegurari/cargo-make
https://github.com/VladimirMarkelov/haku
https://github.com/go-task/task

Hey @casey! You might want also to mention my creature https://github.com/xonixx/makesure
Some exclusive features of it:

The syntax of Makesurefile is a valid shell, so user gets highlighting for free.
The tool doesn't need traditional installation, since it's single-file POSIX shell script (in fact 99% AWK) that you put in project repo.
The recipe may include very simple but very powerful @reached_if modifier. It allows to skip execution (togeter with sub-dependencies) if already satisfied (idempotence).  I wonder if just needs this feature and/or could incorporate.

Other than that, conceptually and feature-wise it's close to just except for syntax and this.

Nice, that looks really cool. I especially like that it's just shell and zero install. Adding it here.

Thank you!


Update topology-considerations.md

Fixed a small formatting typo in the first bulleted list

@fmigacz : Thanks for your contribution! The author(s) have been notified to review your proposed change.


[RFC] Add session date field in CSV

It would be useful to have the date of when the cooking session has taken place.

@jjohnstz - Can I please get some eyes on this?

@jjohnstz @jmaha - Can I please get some eyes on this?

Hi Michael, thanks for the PR. This is a useful change. We'll need to rev the CSV schema version and update a number of internal CSV parsing tools before merging this. Sorry for the delay, we've been buried with feature development for the holiday season.

@mschinis -- you get a chance to see the comment I put in the PR?  I'm one of the maintainers of Combustion's Android framework.  I haven't seen a response.

@jmaha Hi - great, thank you. Happy to help if you need help with this.
@miwright2 🤔 I don't see a comment in the PR. Is it in draft maybe?

@mschinis -- yup, just submitted.  🤦♂ take a look when you get the chance.

Would you also increment the CSV schema version to 5 as @miwright2 requested? These CSV files get used by a lot of different tools within Combustion, and that's how we differentiate parsing methods.

@mschinis could you retarget this PR for the develop branch? Thanks!

done and done 👍

Hi everyone, I've just received my combustion inc thermometer and plan on building some more functionality with this repository.
Is there a plan to merge this in, so I can contribute a few more things I have in mind?

@jmaha any luck with this?


Enhancement request: 'Open Project' list

The 'Open Project' list could do with more metadata (e.g. 'Date Created', number of lines/records, number of columns, ...), and if possible sortable.
As a workaround I created a 'Google Refine Projects' that pieces this together from /command/core/get-all-project-metadata, /command/core/get-models and /command/core/get-columns-info.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

This should potentially include any additional metadata from #1045 as well, especially description

Closing as a duplicate of #1221.

Usually we try to keep the oldest of a pair of duplicates, so we probably should have closed #1221 instead of this, but for the record, this was included with OpenRefine 2.8 in November 2017.


fix(kernel): "any" serialization breaks private type instances

When a "private" type (aka a class that is not visible in the .jsii
assembly) is passed from JS to Host through an any-typed entity, the
value was serialized by-value, passing only the instance's properties,
and omitting any methods.
This changes the behavior of the SerializationClass.Any serializer,
so that when an anonymous object is encountered, if it has a dynamic
getter, setter or any method (including a constructor), it is passed
by-reference instead of by-value.
Fixes aws/aws-cdk#6746

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 7b171d8a1d29c881f125265facaffeffa464ba53
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 8221e5ede10a4114def334aeec339cce9036ec15
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 5768ccca87f0ea6e98e622cfae74c97a631885d7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 940ec239f15484f87efd2d17bde7f89a67ef7103
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 7cb08ae8d7fd6f081231e65ec482f00f9ceff7bd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 63a887d59fde058538bb015510a39650194e5157
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 24341f2f1e6f3b56e8b51a62a428ee122cbb13ac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 5e414a083ecfa7c49a09495c1517f7af2c1e92bb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


Use size based capacity for mutable state and event cache

Currently our cache capacity is count based which does not work well with workflows that has large payloads

events part https://github.com/uber/cadence/pull/3294


Develop

Deploy preview for oscartegiffel processing.
Built with commit 47085f7d06de7e704b9a363fe4eaf56c090e5cc5
https://app.netlify.com/sites/oscartegiffel/deploys/5b453af0fdd72a74624062d8


Fix typo in android example

Fixes a small typo in the node_modules path inside the android example

👍 Nice catch! Thanks


Enable STS tests

Enable all the STS integration tests.
https://github.com/istio/istio/issues/20133

@JimmyCYJ: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio
38b5c90930479517a85217db975ac85c40b148ad
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Need to update proxy SHA when this PR is merged.
https://github.com/istio/proxy/pull/2700

--- FAIL: TestTokenFetchFailureTwo (1.05s)
    mockserver.go:203: Registered handler for endpoints:
        /v1/projects/-/serviceAccounts/service-1234567@gcp-sa-meshdataplane.iam.gserviceaccount.com:generateAccessToken
        /v1/identitybindingtoken
    setup.go:200: 2020-02-21 10:56:39.459970039 -0800 PST m=+12.337127383 check if all servers in the STS flow are up and ready
    setup.go:206: set up server address to dial <IP_ADDRESS>:20711
    setup.go:218: 2020-02-21 10:56:39.462870844 -0800 PST m=+12.340028110 all servers in the STS flow are up and ready
    token_failure_test.go:76:
        Expected
            <int>: 1
        to be >
            <int>: 1
FAIL
[2020-02-21 10:56:39.489][168113][warning][main] [external/envoy/source/server/server.cc:489] caught SIGTERM
[2020-02-21 10:56:39.489][168113][info][main] [external/envoy/source/server/server.cc:602] shutting down server instance
[2020-02-21 10:56:39.489][168113][info][main] [external/envoy/source/server/server.cc:556] main dispatch loop exited
[2020-02-21 10:56:39.489][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:83] Caught Segmentation fault, suspect faulting address 0x0
[2020-02-21 10:56:39.489][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-02-21 10:56:39.489][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: 1d997c17e2321d43565c80eb828f7bdcb78de7e2/1.13.0/Clean/RELEASE/BoringSSL
[2020-02-21 10:56:39.489][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: __restore_rt [0x7fbcca095520]
[2020-02-21 10:56:39.498][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: Envoy::Server::ConnectionHandlerImpl::ActiveTcpListener::~ActiveTcpListener() [0x55b39c77f76e]
[2020-02-21 10:56:39.506][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: Envoy::Server::ConnectionHandlerImpl::ActiveTcpListener::~ActiveTcpListener() [0x55b39c77f915]
[2020-02-21 10:56:39.514][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: Envoy::Server::ConnectionHandlerImpl::~ConnectionHandlerImpl() [0x55b39c78162b]
[2020-02-21 10:56:39.522][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: Envoy::Server::InstanceImpl::terminate() [0x55b39c7225b8]
[2020-02-21 10:56:39.530][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: Envoy::Server::InstanceImpl::run() [0x55b39c7260eb]
[2020-02-21 10:56:39.538][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #6: Envoy::MainCommonBase::run() [0x55b39b484a38]
[2020-02-21 10:56:39.546][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #7: main [0x55b39b483952]
[2020-02-21 10:56:39.546][168113][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #8: __libc_start_main [0x7fbcc9ee8bbb]
FAIL    istio.io/istio/security/pkg/stsservice/test/failure_sts_token_fetch     12.604s

I think its an issue locally

These tests do not seem to run locally, I get errors like E0221 10:56:28.942559587 168113 oauth2_credentials.cc:220] oauth_fetch: {"created":"@1582311388.942539213","description":"Failed to load file","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/load_file.cc","file_line":71,"filename":"","referenced_errors":[{"created":"@1582311388.942538001","description":"No such file or directory","errno":2,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/load_file.cc","file_line":45,"os_error":"No such file or directory","syscall":"fopen"}]}

It seems that gRPC library fail to open files, the test sets up two files, one for initial JWT token and one for CA cert. Perhaps your Istio ROOT PATH is not equal to istioEnv.IstioSrc used in the test.
The tests pass in prow:
https://storage.googleapis.com/istio-prow/pr-logs/pull/istio_istio/21334/unit-tests_istio/8974/build-log.txt

@JimmyCYJ: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio
179f9a6f2e3cd91dd12d62c6d2b16ccfd06b483f
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

Looks like the failing test is an STS one, are we enabling a flaky test?
On Sun, Feb 23, 2020, 11:01 AM Jimmy Chen<EMAIL_ADDRESS>wrote:

/retest
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/pull/21334?email_source=notifications&email_token=AAEYGXMMSCJIFK62ACHEX6TRELBZLA5CNFSM4KY2A6M2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMWELNA#issuecomment-590103988,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXI6STGJ2DHOIOF364TRELBZLANCNFSM4KY2A6MQ
.

Looks like the failing test is an STS one, are we enabling a flaky test?

It is a known flaky test, I am going to fix it. https://github.com/istio/istio/issues/19918

In response to a cherrypick label: #21334 failed to apply on top of branch "release-1.5":
Applying: update proxy SHA
error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	istio.deps
Falling back to patching base and 3-way merge...
Auto-merging istio.deps
CONFLICT (content): Merge conflict in istio.deps
Patch failed at 0002 update proxy SHA


Windows Certificate Store Issue

Windows Dart 2.14.0-93-dev and later.
The majority of this issue was discussed in the pub repo at https://github.com/dart-lang/pub/issues/2990 and it appears to be an issue with dart.
A recent change to the runtime made all certificate stores on Windows accessible to dart. I am in an environment with an internal CA that issues certificates to the proxy servers. When the root certificates for a site are not known to the proxy, it uses the internal certificates for the communication between the workstation and the proxy. The internal certificates are regularly pushed to the workstations to prevent invalid certificate errors.
The issue is that the expired certificates remain in the store and dart is attempting to use them. The runtime appears to be using the first certificate it encounters. If it is invalid for any reason, it throws an exception and moves on. This happens even when it does find a valid certificate.
import 'dart:io';

void main(List<String> arguments) {
  var client = HttpClient();

  client.badCertificateCallback = (cert, host, port) {
    print('Bad certificate connecting to $host:$port:');
    _printCertificate(cert);
    print('');
    return true;
  };
  client
      .getUrl(Uri.parse('https://pub.dartlang.org/api/packages/pedantic'))
      .then((request) => request.close())
      .then((response) {
    print('Response certificate:');
    _printCertificate(response.certificate);
    response.drain();
    client.close();
  });
}

void _printCertificate(X509Certificate? cert) {
  if (cert != null) {
    print('${cert.issuer}');
    print('${cert.subject}');
    print('${cert.startValidity}');
    print('${cert.endValidity}');
  } else {
    print('Certificate is null');
  }
}

Running this code produces this result. (I had to mask some of the names with XXX due to policy.)
Bad certificate connecting to pub.dartlang.org:443:
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Intermediate CA 1
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Signing CA 4
2020-01-06 17:59:49.000Z
2020-07-06 17:59:49.000Z

Bad certificate connecting to pub.dartlang.org:443:
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Signing CA 4
/CN=pub.dartlang.org
2021-05-04 01:59:01.000Z
2021-07-10 14:07:57.000Z

Response certificate:
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Signing CA 4
/CN=pub.dartlang.org
2021-05-04 01:59:01.000Z
2021-07-10 14:07:57.000Z

The first certificate displayed is expired but the second one is not. I have tried this URL in the browser and the valid certificate here is the one being used.
This issue can be replicated using the --root-certs-file option for dart. I used a hand crafted certificate bundle that removed all expired certificates and this code worked. I tried using it with pub and was getting local issuer certificate errors because I did not have the time to hand craft a certificate bundle that included 100+ items from my environment.

The first certificate displayed is expired but the second one is not. I

There got to be a reason why these certificates deem to be bad by boringssl/openssl.
Is it a root certificate, if not, are all certificates in its certificate chain are valid?
Which store are they in? (personal, local machine) (roots, intermediate cas...)
It is surprising you got two Bad certificate callbacks too - first one seems to have been sufficient since it returns true(meaning it can be trusted).

There got to be a reason why second certificate is deemed to be bad by boringssl/openssl.

I hand crafted a certificate bundle with that cert and all certs in the chain. I used it with the --root-certs-file option to dart. The bad certificate callback was not called.

Is it a root certificate, if not, are all certificates in its certificate chain are valid?

When I navigate to the URL in the browser, all certificates in the chain are valid.

Which store are they in? (personal, local machine) (roots, intermediate cas...)

The certificates are in the local machine store. The certs listed in the output are intermediate. The root certs for them are stored in root and are trusted and valid.

It is surprising you got two Bad certificate callbacks too - first one seems to have been sufficient since it returns true(meaning it can be trusted).

I thought it was odd as well.

I hand crafted a certificate bundle with that cert and all certs in the chain

Have you been able to confirm that any(or all) of these certificates that you bundled are present in your Windows localmachine/currentuser certificate store(via certmgr.msc)?

It is surprising you got two Bad certificate callbacks too - first one seems to have been sufficient since it returns true(meaning it can be trusted).
I thought it was odd as well.

This is actually working as intended as per https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_verify.html the callback is called for every certificate in the chain that failed openssl builtin verification.

I am in an environment with an internal CA that issues certificates to the proxy servers. When the root certificates for a site are not known to the proxy, it uses the internal certificates for the communication between the workstation and the proxy. The internal certificates are regularly pushed to the workstations to prevent invalid certificate errors.

I am not familiar with this process. pub.dev certificate chain has certain certificates in it's chain. In order for dart to connect to pub.dev, certificates that pub.dev shows need to be present in Windows certificate store.

The first certificate displayed is expired but the second one is not.

So if the first certificate printed(which seems to be an intermediate certificate in the chain) has expired/is invalid, then whole certificate won't be trusted neither.
Question is how does this intermediate certificate become part of the chain that dart app sees? Is it part of the chain that you can see in the browser when you check certificates for pub.dev website too?

Have you been able to confirm that any(or all) of these certificates that you bundled are present in your Windows localmachine/currentuser certificate store(via certmgr.msc)?

I have verified that all certificates are in the local machine store.

I am not familiar with this process. pub.dev certificate chain has certain certificates in it's chain. In order for dart to connect to pub.dev, certificates that pub.dev shows need to be present in Windows certificate store.

The proxy servers in my environment do not always use the certificates of the host site for secure communication with the workstations. For example, if I navigate to microsoft.com, the browser shows that the certs I am using are the ones from microsoft.com. When I navigate to pub.dev, the browser shows that the certs are from the internal CA. All of the certificates are valid and present in the local machine cert store.

Question is how does this intermediate certificate become part of the chain that dart app sees? Is it part of the chain that you can see in the browser when you check certificates for pub.dev website too?

When I open the URL in the browser, an intermediate certificate with the same name is in the chain but it is valid. Several intermediate certificates with the same name are in the store and all but one of them are expired. I exported the valid ones in the chain to create the bundle I was testing with.

I have finally been able to come back to this issue. I downloaded the most recent DEV build (2.15.0-42.0) and this issue is still present.
I decided it was time to reduce the issue down to its core. I left the wonky proxy configuration out of the equation and focused specifically on the certificate store.
I created a small .NET app that exported all certificates out of the local machine stores. The file produced by this code worked with the --root-certs-file switch. I did not get the bad certificate callback result.
class Program
{
	static void Main(string[] args)
	{
		// The locations here are CurrentUser and LocalMachine
		IEnumerable<StoreLocation> locations = (StoreLocation[])Enum.GetValues(typeof(StoreLocation));

		// The AddressBook, Disallowed, and TrustedPeople stores were explicily omitted.
		IEnumerable<StoreName> names = new StoreName[] { StoreName.AuthRoot, StoreName.CertificateAuthority, StoreName.My, StoreName.Root, StoreName.TrustedPublisher };

		List<X509Store> stores = new List<X509Store>();

		System.Console.WriteLine("Getting certificate stores.");
		foreach (StoreLocation location in locations)
		{
			foreach (StoreName name in names)
			{
				stores.Add(new X509Store(name, location));
			}
		}

		IEnumerable<X509Certificate2> storeCerts = null;

		System.Console.WriteLine("Writing certificates to valid_certs.pem");
		using (StreamWriter writer = new StreamWriter("valid_certs.pem"))
		{
			foreach (X509Store store in stores)
			{
				storeCerts = Program.GetCertificates(store);

				foreach (X509Certificate2 cert in storeCerts)
				{
					writer.WriteLine("-----BEGIN CERTIFICATE-----");
					writer.WriteLine(Convert.ToBase64String(cert.RawData, Base64FormattingOptions.InsertLineBreaks));
					writer.WriteLine("-----END CERTIFICATE-----");
					writer.WriteLine(String.Empty);
					writer.Flush();
				}

				if (store.IsOpen)
				{
					store.Close();
				}
				store.Dispose();
			}
		}

		System.Console.WriteLine("Done");
	}

	private static IEnumerable<X509Certificate2> GetCertificates(X509Store store)
	{
		IEnumerable<X509Certificate2> certs = new List<X509Certificate2>();

		store.Open(OpenFlags.ReadOnly);

		if (store.Certificates.Count > 0)
		{
			certs = store.Certificates.Cast<X509Certificate2>().Where(c => c.NotAfter > DateTime.Now);
		}

		store.Close();

		return certs;
	}
}

I then changed the GetCertificates method to omit the expiration date filter. I used the file created after the change with the --root-certs-file switch and the bad certificate callback result returned.
private static IEnumerable<X509Certificate2> GetCertificates(X509Store store)
{
	IEnumerable<X509Certificate2> certs = new List<X509Certificate2>();

	store.Open(OpenFlags.ReadOnly);

	if (store.Certificates.Count > 0)
	{
		// The call to Where was removed. This ensures all certificates are returned.
		certs = store.Certificates.Cast<X509Certificate2>();
	}

	store.Close();

	return certs;
}

The two files were also tested with pub: dart --root-certs-file=C:\certs\valid_certs.pem pub get --verbose. The first file resulted in pub running to completion with no errors. The second file resulted in a series of expired certificate errors in the pub output.
It looks like the certificate lookup is stopping on the first one found and ignoring all others.

Okay, thank you @JSanford42 . Will you be able to give a try to dart sdk with a proposed fix https://dart-review.googlesource.com/c/sdk/+/211160 I can build for you? I don't have a clear plan on how to test this yet in ci setting.

@aam I will gladly test the change. I usually download the SDK as a zip so I can switch between builds for testing.

Okay @JSanford42 thanks, can you ping me your email to<EMAIL_ADDRESS>so I share manually-built dart sdk google drive link with you?

@aam I received the link, downloaded the build, and tested it. I did not enounter any of the certificate issues. The bad certificate callback was not executed in the code from the original post. I also tested the build with pub by installing several packages with sizable dependency trees. I even installed some global packages (grinder and webdev) just to be sure.
I look foward to seeing this fix in a future dart build. In the mean time, I can use my manually built certificate bundle for development and package installations.


⚠ ClamAV has degraded performance

In b38fc28, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 196 ms

Resolved: ClamAV performance has improved in 21d3c41.


Support literal haskell!

Ahoy there!
Literal haskell is a fantastic writing medium, it also helps that it's markdown compatible. This issue description is in literal haskell, and can be compiled by any haskell compiler (as a file with a .lhs extension).
Code to compile is in markdown blockquotes like so:

factorial' :: [Int] -> Int
factorial' [] = 0
factorial' (x:xs) = x * factorial xs

In theory all you should have to do is add .lhs to your list of supported file types somewhere and it should just work. I totally get that there probably won't be much demand for this though.

🤔 Is still an issue?
Unless I'm mistaken, there is no longer any Haskell in this repo?


Failied to clone the project

After trying to clone the project I have some issue:
Cloning into 'The-Modern-Vulkan-Cookbook'...
remote: Enumerating objects: 1604, done.
remote: Counting objects: 100% (99/99), done.
remote: Compressing objects: 100% (78/78), done.
remote: Total 1604 (delta 35), reused 53 (delta 15), pack-reused 1505
Receiving objects: 100% (1604/1604), 55.73 MiB | 5.71 MiB/s, done.
Resolving deltas: 100% (265/265), done.
Updating files: 100% (1290/1290), done.
Downloading source/common/resources/assets/Bistro.glb (259 MB)
Error downloading object: source/common/resources/assets/Bistro.glb (5cc23d5): Smudge error: Error downloading source/common/resources/assets/Bistro.glb (5cc23d52b95fa1ba8b585d998185904becf8bcaa8f424fd74cf6950addb63deb): batch response: Post "https://github.com/PacktPublishing/The-Modern-Vulkan-Cookbook.git/info/lfs/objects/batch": x509: certificate signed by unknown authority

Errors logged to 'C:\Users\EDUKAJ\projects\dev\OReilly\The-Modern-Vulkan-Cookbook\book\The-Modern-Vulkan-Cookbook\.git\lfs\logs\20240429T094427.23756.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: source/common/resources/assets/Bistro.glb: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

You probably need to install git-lfs (https://git-lfs.com), let us know if it works.


Install fails on immutable OS like fedora silverblue

Hi folks,
not sure if this is the right place, but I can't install mdatp on fedora silverblue. Silverblue is based on ostree and rpm-ostree.
$ rpm-ostree install mdatp
...
error: Running %post for mdatp: bwrap(/bin/sh): Child process killed by signal 1; run `journalctl -t 'rpm-ostree(mdatp.post)'` for more information

the issue seems to be the post install script that tries to move files after installation and it is not allowed because it is run in a sandbox (see https://coreos.github.io/rpm-ostree/architecture-core/#sandboxing-scripts)
$ journalctl -t 'rpm-ostree(mdatp.post)'
...
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3359]: mkdir: cannot create directory ‘/var/opt’: Read-only file system
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3360]: chmod: cannot access '/var/opt/microsoft': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3361]: mv: cannot move '/opt/microsoft/mdatp/definitions/libmpengine.so' to '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3363]: chmod: cannot access '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000/libmpengine.so': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3364]: mv: cannot move '/opt/microsoft/mdatp/definitions/mpavbase.vdm' to '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3365]: mv: cannot move '/opt/microsoft/mdatp/definitions/mpasbase.vdm' to '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3366]: mv: cannot move '/opt/microsoft/mdatp/definitions/mpavdlta.vdm' to '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3367]: mv: cannot move '/opt/microsoft/mdatp/definitions/mpasdlta.vdm' to '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3368]: chmod: cannot access '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000/mpavbase.vdm': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3368]: chmod: cannot access '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000/mpasbase.vdm': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3368]: chmod: cannot access '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000/mpavdlta.vdm': No such file or directory
Mar 13 13:04:48 myhost rpm-ostree(mdatp.post)[3368]: chmod: cannot access '/var/opt/microsoft/mdatp/definitions.noindex/00000000-0000-0000-0000-000000000000/mpasdlta.vdm': No such file or directory

related issue (on the rpm-ostree site):
https://github.com/coreos/rpm-ostree/issues/233
Is it possible to have the files in the right directory already in the RPM package, so we don't have to move them in the post install script?

Please open a support request. This repository is for Defender open source tools that are not the parts of the product itself.


ESLint: Remove ts-ignore comments

SUMMARY
I removed some @ts-ignore comments and refactored the code accordingly. Some components required a simple dependency bump, installing @types package or properly casting the type. In some cases (particularly react-select components) workarounds were necessary - e.g. wrapping an Input component in a div to pass onPaste prop.
There are a few ts-ignores left. I decided to leave them for another PR as they might require some bigger refactors.
TEST PLAN
Run npm run lint, verify there are no new linter errors.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

@rusackas Can you please take a look?

Codecov Report

Merging #10933 into master will decrease coverage by 6.03%.
The diff coverage is 42.85%.

@@            Coverage Diff             @@
##           master   #10933      +/-   ##
==========================================
- Coverage   65.84%   59.81%   -6.04%
==========================================
  Files         815      781      -34
  Lines       38335    37222    -1113
  Branches     3601     3353     -248
==========================================
- Hits        25243    22265    -2978
- Misses      12990    14776    +1786
- Partials      102      181      +79

Flag
Coverage Δ

#cypress
56.78% <42.85%> (+0.06%)
:arrow_up:

#javascript
?

#python
61.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...et-frontend/src/SqlLab/components/LimitControl.tsx
65.78% <0.00%> (-22.85%)
:arrow_down:

...src/components/FilterableTable/FilterableTable.tsx
2.61% <0.00%> (-83.03%)
:arrow_down:

superset-frontend/src/components/Link.tsx
7.69% <ø> (-79.81%)
:arrow_down:

...erset-frontend/src/components/ListView/Filters.tsx
84.74% <ø> (-4.81%)
:arrow_down:

...ponents/Select/WindowedSelect/WindowedMenuList.tsx
96.29% <ø> (-0.26%)
:arrow_down:

...perset-frontend/src/datasource/DatasourceModal.tsx
81.08% <ø> (-14.67%)
:arrow_down:

...rontend/src/explore/components/PropertiesModal.tsx
45.76% <0.00%> (-16.36%)
:arrow_down:

superset-frontend/src/featureFlags.ts
100.00% <ø> (ø)

superset-frontend/src/setup/setupApp.ts
28.57% <ø> (+3.57%)
:arrow_up:

...perset-frontend/src/views/CRUD/welcome/Welcome.tsx
0.00% <0.00%> (-85.00%)
:arrow_down:

... and 265 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5623cd6...dc0c007. Read the comment docs.

There were issues on master earlier, please rebase!

@kgabryje @rusackas This change is causing failures on master https://github.com/apache/incubator-superset/runs/1146648924

2020-09-21T22:49:21.6155663Z > eslint --ignore-path=.eslintignore --ext .js,.jsx,.ts,.tsx . && npm run type
2020-09-21T22:49:21.6156105Z
2020-09-21T22:50:19.3926195Z
2020-09-21T22:50:19.3929001Z /home/runner/work/incubator-superset/incubator-superset/superset-frontend/spec/javascripts/components/TableSelector_spec.jsx
2020-09-21T22:50:19.3930832Z   179:5  warning  Skipped test  jest/no-disabled-tests
2020-09-21T22:50:19.3932051Z   221:5  warning  Skipped test  jest/no-disabled-tests
2020-09-21T22:50:19.3932744Z
2020-09-21T22:50:19.3934154Z /home/runner/work/incubator-superset/incubator-superset/superset-frontend/spec/javascripts/explore/utils_spec.jsx
2020-09-21T22:50:19.3935922Z    59:5  warning  Test has no assertions  jest/expect-expect
2020-09-21T22:50:19.3939217Z    68:5  warning  Test has no assertions  jest/expect-expect
2020-09-21T22:50:19.3940312Z    80:5  warning  Test has no assertions  jest/expect-expect
2020-09-21T22:50:19.3941191Z    92:5  warning  Test has no assertions  jest/expect-expect
2020-09-21T22:50:19.3942061Z   104:5  warning  Test has no assertions  jest/expect-expect
2020-09-21T22:50:19.3942930Z   116:5  warning  Test has no assertions  jest/expect-expect
2020-09-21T22:50:19.3943799Z   195:5  warning  Test has no assertions  jest/expect-expect
2020-09-21T22:50:19.3944281Z
2020-09-21T22:50:19.3945490Z /home/runner/work/incubator-superset/incubator-superset/superset-frontend/spec/javascripts/sqllab/TableElement_spec.jsx
2020-09-21T22:50:19.3946832Z   52:3  warning  Test has no assertions  jest/expect-expect
2020-09-21T22:50:19.3947260Z
2020-09-21T22:50:19.3948419Z /home/runner/work/incubator-superset/incubator-superset/superset-frontend/spec/javascripts/sqllab/actions/sqlLab_spec.js
2020-09-21T22:50:19.3949738Z   110:5  warning  Skipped test  jest/no-disabled-tests
2020-09-21T22:50:19.3950579Z   178:5  warning  Skipped test  jest/no-disabled-tests
2020-09-21T22:50:19.3950980Z
2020-09-21T22:50:19.3952147Z /home/runner/work/incubator-superset/incubator-superset/superset-frontend/spec/javascripts/sqllab/reducers/sqlLab_spec.js
2020-09-21T22:50:19.3953464Z   247:5  warning  Test has no assertions  jest/expect-expect
2020-09-21T22:50:19.3953857Z
2020-09-21T22:50:19.3955078Z /home/runner/work/incubator-superset/incubator-superset/superset-frontend/src/views/CRUD/data/savedquery/SavedQueryList.tsx
2020-09-21T22:50:19.3956720Z   31:3  warning  'addDangerToast' is defined but never used   @typescript-eslint/no-unused-vars
2020-09-21T22:50:19.3958073Z   32:3  warning  'addSuccessToast' is defined but never used  @typescript-eslint/no-unused-vars
2020-09-21T22:50:19.3958698Z
2020-09-21T22:50:19.3959246Z ✖ 15 problems (0 errors, 15 warnings)
2020-09-21T22:50:19.3959572Z
2020-09-21T22:50:20.2057788Z
2020-09-21T22:50:20.2059601Z ><EMAIL_ADDRESS>type /home/runner/work/incubator-superset/incubator-superset/superset-frontend
2020-09-21T22:50:20.2060451Z > tsc --noEmit
2020-09-21T22:50:20.2060663Z
2020-09-21T22:50:44.1527383Z src/components/ListView/Filters.tsx:140:11 - error TS2322: Type '{ label: string; value: string; }' is not assignable to type 'ValueType<GroupType<GroupType<never>>>'.
2020-09-21T22:50:44.1529417Z   Property 'options' is missing in type '{ label: string; value: string; }' but required in type 'GroupType<GroupType<never>>'.
2020-09-21T22:50:44.1530168Z
2020-09-21T22:50:44.1530855Z 140           value={selectedOption}
2020-09-21T22:50:44.1531543Z               ~~~~~
2020-09-21T22:50:44.1531916Z
2020-09-21T22:50:44.1533246Z   node_modules/@types/react-select/src/types.d.ts:11:3
2020-09-21T22:50:44.1534417Z     11   options: OptionsType<OptionType>;
2020-09-21T22:50:44.1537461Z          ~~~~~~~
2020-09-21T22:50:44.1538732Z     'options' is declared here.
2020-09-21T22:50:44.1539523Z   node_modules/@types/react-select/src/Select.d.ts:205:3
2020-09-21T22:50:44.1540192Z     205   value?: ValueType<OptionType>;
2020-09-21T22:50:44.1540648Z           ~~~~~
2020-09-21T22:50:44.1541964Z     The expected type comes from property 'value' which is declared here on type 'IntrinsicAttributes & Props<GroupType<GroupType<never>>> & UseAsyncPaginateParams<GroupType<GroupType<never>>, any> & ComponentProps & { ...; } & { ...; }'
2020-09-21T22:50:44.1542896Z
2020-09-21T22:50:44.1544256Z src/components/ListView/Filters.tsx:141:11 - error TS2322: Type '(selected: SelectOption | null) => void' is not assignable to type '(value: ValueType<GroupType<GroupType<never>>>, action: ActionMeta<GroupType<GroupType<never>>>) => void'.
2020-09-21T22:50:44.1545588Z   Types of parameters 'selected' and 'value' are incompatible.
2020-09-21T22:50:44.1546437Z     Type 'ValueType<GroupType<GroupType<never>>>' is not assignable to type 'SelectOption | null'.
2020-09-21T22:50:44.1547291Z       Type 'undefined' is not assignable to type 'SelectOption | null'.
2020-09-21T22:50:44.1547649Z
2020-09-21T22:50:44.1548063Z 141           onChange={onChange}
2020-09-21T22:50:44.1548491Z               ~~~~~~~~
2020-09-21T22:50:44.1548650Z
2020-09-21T22:50:44.1549206Z   node_modules/@types/react-select/src/Select.d.ts:169:3
2020-09-21T22:50:44.1550056Z     169   onChange?: (value: ValueType<OptionType>, action: ActionMeta<OptionType>) => void;
2020-09-21T22:50:44.1550753Z           ~~~~~~~~
2020-09-21T22:50:44.1551913Z     The expected type comes from property 'onChange' which is declared here on type 'IntrinsicAttributes & Props<GroupType<GroupType<never>>> & UseAsyncPaginateParams<GroupType<GroupType<never>>, any> & ComponentProps & { ...; } & { ...; }'
2020-09-21T22:50:44.1552741Z
2020-09-21T22:50:44.1554123Z src/components/ListView/Filters.tsx:142:11 - error TS2322: Type '(inputValue: string, loadedOptions: SelectOption[], { page }: { page: number; }) => Promise<{ options: { label: string; value: string; }[]; hasMore: boolean; additional: { page: number; }; }>' is not assignable to type 'LoadOptions<GroupType<GroupType<never>>, any>'.
2020-09-21T22:50:44.1555467Z   Types of parameters 'loadedOptions' and 'options' are incompatible.
2020-09-21T22:50:44.1556260Z     Type 'OptionsList<GroupType<GroupType<never>>>' is not assignable to type 'SelectOption[]'.
2020-09-21T22:50:44.1557211Z       The type 'OptionsType<GroupType<GroupType<never>>>' is 'readonly' and cannot be assigned to the mutable type 'SelectOption[]'.
2020-09-21T22:50:44.1557691Z
2020-09-21T22:50:44.1558183Z 142           loadOptions={fetchAndFormatSelects}
2020-09-21T22:50:44.1558686Z               ~~~~~~~~~~~
2020-09-21T22:50:44.1558828Z
2020-09-21T22:50:44.1559425Z   node_modules/react-select-async-paginate/ts/types.d.ts:44:5
2020-09-21T22:50:44.1560143Z     44     loadOptions: LoadOptions<OptionType>;
2020-09-21T22:50:44.1560590Z            ~~~~~~~~~~~
2020-09-21T22:50:44.1561767Z     The expected type comes from property 'loadOptions' which is declared here on type 'IntrinsicAttributes & Props<GroupType<GroupType<never>>> & UseAsyncPaginateParams<GroupType<GroupType<never>>, any> & ComponentProps & { ...; } & { ...; }'
2020-09-21T22:50:44.1562604Z
2020-09-21T22:50:44.1562739Z
2020-09-21T22:50:44.1562943Z Found 3 errors.
2020-09-21T22:50:44.1563099Z
2020-09-21T22:50:44.2013547Z npm ERR! code ELIFECYCLE
2020-09-21T22:50:44.2014256Z npm ERR! errno 1
2020-09-21T22:50:44.2048042Z npm ERR<EMAIL_ADDRESS>type: `tsc --noEmit`
2020-09-21T22:50:44.2048746Z npm ERR! Exit status 1
2020-09-21T22:50:44.2049520Z npm ERR!
2020-09-21T22:50:44.2050518Z npm ERR! Failed at the<EMAIL_ADDRESS>type script.
2020-09-21T22:50:44.2051630Z npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2020-09-21T22:50:44.2134011Z
2020-09-21T22:50:44.2134573Z npm ERR!  Exit status 1
2020-09-21T22:50:44.2235443Z npm ERR!
2020-09-21T22:50:44.2236440Z npm ERR! Failed at the<EMAIL_ADDRESS>lint script.
2020-09-21T22:50:44.2237161Z npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2020-09-21T22:50:44.2381461Z
2020-09-21T22:50:44.2382258Z npm ERR! A complete log of this run can be found in:
2020-09-21T22:50:44.2383564Z npm ERR!     /home/runner/.npm/_logs/2020-09-21T22_50_44_223Z-debug.log
2020-09-21T22:50:44.2420245Z ##[error]Process completed with exit code 1.

Opened a PR to (at least temporarily) revert that commit.


Consolidation process is not expected

Version

Karpenter Version: v0.27.0

Kubernetes Version: v1.22
I have just migrated from Cluster Autoscaler + Auto scaling group to Karpenter.
Well, when I compare the request/allocatable ratio, I find Karpenter provisioned nodes have only 30% cpu&memory utilization. I have opened consolidation for the Karpenter controller, not sure what cause this problem.
FYI, I didn't uninstall Cluster Autoscaler but instead I adjust the corresponding min/max replicas of ASG to 0. So the certain nodes can only be provsioned by Karpenter.
The pods only have spreadConstraints for availibility zones
(There is only one instance_type: c6g.8xlarge and node type is ondemand)
Expected Behavior
The nodes with low cpu&memory utilization will be deleted and reschedule the pods to other nodes.
Actual Behavior
The nodes have low resource utilization and the logs keep saying "Not all pods will be schedule".
Steps to Reproduce the Problem

Before then, I have a cluster autoscaler and autoscaling group, the asg has min/max 1/100
I launched provisioner for the same node type
Manually scale down the min/max replcas of asg, 1/100 -> 25/25 -> 20/20 -> ... -> 0/0
Then all the workload is managed by Karpenter
But however, Karpenter did not deprovision though the pods only request few resources.

Resource Specs and Logs
Here is the log for the nodes which should be consolidated:
`Non-terminated Pods:          (8 in total)
  Namespace                   Name                        CPU Requests  CPU Limits   Memory Requests  Memory Limits  Age
  ---------                   ----                        ------------  ----------   ---------------  -------------  ---
  default                     vcamsegs-59b8dc555-2kxmm    1 (3%)        2 (6%)       300Mi (0%)       0 (0%)         3d11h
  default                     vcamsegs-59b8dc555-d2zfv    1 (3%)        2 (6%)       300Mi (0%)       0 (0%)         3d11h
  default                     vsubmit-78c49fbbcf-tpczs    5400m (16%)   5500m (17%)  6400Mi (10%)     8448Mi (13%)   2d3h
  kube-system                 aws-node-b24gl              10m (0%)      0 (0%)       0 (0%)           0 (0%)         3d11h
  kube-system                 ebs-csi-node-vhqq4          0 (0%)        0 (0%)       0 (0%)           0 (0%)         3d11h
  kube-system                 kube-proxy-vftq4            100m (0%)     0 (0%)       0 (0%)           0 (0%)         3d11h
  monitoring                  datadog-86gxf               200m (0%)     0 (0%)       256Mi (0%)       0 (0%)         3d11h
  monitoring                  node-exporter-r2clz         10m (0%)      100m (0%)    100Mi (0%)       100Mi (0%)     3d11h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         7720m (24%)   9600m (30%)
  memory                      7356Mi (11%)  8548Mi (13%)
  ephemeral-storage           0 (0%)        0 (0%)
  hugepages-1Gi               0 (0%)        0 (0%)
  hugepages-2Mi               0 (0%)        0 (0%)
  hugepages-32Mi              0 (0%)        0 (0%)
  hugepages-64Ki              0 (0%)        0 (0%)
  attachable-volumes-aws-ebs  0             0
  vpc.amazonaws.com/pod-eni   3             3
Events:
  Type    Reason            Age                    From       Message
  ----    ------            ----                   ----       -------
  Normal  Unconsolidatable  52s (x318 over 3d11h)  karpenter  not all pods would schedule

For Karpenter logs, I didn't see any useful information related to this provisioner.
Why not all pods would schedule, was this caused by Karpenter can not reschedule daemonset?
Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Name:         cpu-c6g-streaming Namespace:     Labels:       app.kubernetes.io/instance=vkarpenter Annotations:  <none> API Version:  karpenter.sh/v1alpha5 Kind:         Provisioner Metadata:   Creation Timestamp:  2023-03-22T00:23:30Z   Generation:          4   Managed Fields:     API Version:  karpenter.sh/v1alpha5     Fields Type:  FieldsV1     fieldsV1:       f:status:         .:         f:resources:           .:           f:attachable-volumes-aws-ebs:           f:cpu:           f:ephemeral-storage:           f:memory:           f:pods:           f:vpc.amazonaws.com/pod-eni:     Manager:      karpenter     Operation:    Update     Subresource:  status     Time:         2023-03-22T10:49:53Z     API Version:  karpenter.sh/v1alpha5     Fields Type:  FieldsV1     fieldsV1:       f:metadata:         f:annotations:           .:           f:kubectl.kubernetes.io/last-applied-configuration:         f:labels:           .:           f:app.kubernetes.io/instance:       f:spec:         .:         f:consolidation:           .:           f:enabled:         f:kubeletConfiguration:           .:           f:evictionHard:             .:             f:memory.available:             f:nodefs.available:           f:evictionMaxPodGracePeriod:           f:evictionSoft:             .:             f:memory.available:             f:nodefs.available:           f:evictionSoftGracePeriod:             .:             f:memory.available:             f:nodefs.available:           f:kubeReserved:             .:             f:ephemeral-storage:             f:memory:           f:systemReserved:             .:             f:ephemeral-storage:             f:memory:         f:labels:           .:           f:asg-name:           f:graviton-c6g:           f:lifecycle:           f:streaming:           f:type:         f:providerRef:           .:           f:name:         f:requirements:         f:taints:         f:ttlSecondsUntilExpired:         f:weight:     Manager:         argocd-application-controller     Operation:       Update     Time:            2023-03-22T22:33:03Z   Resource Version:<PHONE_NUMBER>   UID:               8a1f95aa-578d-4dc2-926c-7d5ef53a3839 Spec:   Consolidation:     Enabled:  true   Kubelet Configuration:     Eviction Hard:       memory.available:             200Mi       nodefs.available:             10%     Eviction Max Pod Grace Period:  60     Eviction Soft:       memory.available:  500Mi       nodefs.available:  15%     Eviction Soft Grace Period:       memory.available:  1m30s       nodefs.available:  1m30s     Kube Reserved:       Ephemeral - Storage:  1Gi       Memory:               300Mi     System Reserved:       Ephemeral - Storage:  1Gi       Memory:               200Mi   Labels:     Asg - Name:    cpu-c6g-streaming     graviton-c6g:  true     Lifecycle:     ondemand     Streaming:     true     Type:          cpu   Provider Ref:     Name:  default   Requirements:     Key:       node.kubernetes.io/instance-type     Operator:  In     Values:       c6g.8xlarge     Key:       topology.kubernetes.io/zone     Operator:  In     Values:       us-west-2a       us-west-2b       us-west-2c     Key:       kubernetes.io/arch     Operator:  In     Values:       arm64     Key:       kubernetes.io/os     Operator:  In     Values:       linux     Key:       karpenter.sh/capacity-type     Operator:  In     Values:       on-demand   Taints:     Effect:                   NoSchedule     Key:                      streaming     Value:                    true     Effect:                   NoSchedule     Key:                      graviton-c6g     Value:                    true   Ttl Seconds Until Expired:  1209600   Weight:                     50 Status:   Resources:     Attachable - Volumes - Aws - Ebs:  3861     Cpu:                               3168     Ephemeral - Storage:<PHONE_NUMBER>Ki     Memory:<PHONE_NUMBER>Ki     Pods:                              23166     vpc.amazonaws.com/pod-eni:         5346 Events:                                <none>
Provisioner info

From the provisioner you pasted above, it looks like c6g.8xlarge is the only instance type allowed. This will prevent consolidation from happening since there are no cheaper instance types that will fit your pods.
I'd recommend removing the following requirement from your provisioner:
    Key:       node.kubernetes.io/instance-type
    Operator:  In
    Values:
      c6g.8xlarge

and see if consolidation works after that.  Karpenter will still select a graviton node since you have a kubernetes.io/arch=arm64 requirement.

Thanks for the quick response.
I think if Karpenter can not replace that with a cheaper node, the log should be "Can not replace with a cheaper node "? And let's suppose I have two c6g.8xlarge A and B. Let's assume both of them use 20% cpu and 20% memory. I think the Karpenter can schedule the workload from node A to node B correctly? Not sure the internal process of deprovisioning. Is this expected or a corner case if I only allow one instance_type and enabled consolidation also?

If you have multiple c6g.8xlarge nodes running and they're not utilized to where the pods can be consolidated to a smaller number of nodes, then Karpenter should take that action. I'm not sure why it is not in this case.
I don't think there should be any issues with the daemonsets scheduling.
Are you able to post the deployment specs for the pods running on the node?

Sure, I can do that. Just want to confirm, the consolidation process only takes care of non-daemonset pods right? Like when we run the command kubectl drain nodes --ignore-daemonset? Also, during my experimentation, if hpa scales down, like all the service pods were removed from the node, Karpenter can trigger deprovision behavior(not consolidation). For consolidation, it keeps logging
"  Normal   Unconsolidatable         2m57s (x2 over 18m)  karpenter        not all pods would schedule"
The deployment has a spreadConstraints, but I don't think that's the issue since there are no consolidation behavior at all. Here is an attached configuration for the deployment and it's controlled by a HPA
`
apiVersion: apps/v1
kind: ReplicaSet
metadata:
annotations:
argocd.argoproj.io/sync-wave: '4'
deployment.kubernetes.io/desired-replicas: '0'
deployment.kubernetes.io/max-replicas: '0'
deployment.kubernetes.io/revision: '63'
creationTimestamp: '2023-03-27T17:56:52Z'
generation: 1
labels:
admission.datadoghq.com/enabled: 'true'
app: vsubmit
appInst: vsubmit
pod-template-hash: 7d984bdfff
tags.datadoghq.com/service: vsubmit
tags.datadoghq.com/version: a734957
vertical: streaming
name: vsubmit-7d984bdfff
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: Deployment
name: vsubmit
uid: 94fe7e03-cf5d-474d-9452-463efec4b1d3
resourceVersion: '5522817073'
uid: b40c5dd5-5ec3-4d5e-afbd-91b9576356d5
spec:
replicas: 0
selector:
matchLabels:
appInst: vsubmit
pod-template-hash: 7d984bdfff
template:
metadata:
creationTimestamp: null
labels:
admission.datadoghq.com/enabled: 'true'
app: vsubmit
appInst: vsubmit
pod-template-hash: 7d984bdfff
tags.datadoghq.com/service: vsubmit
tags.datadoghq.com/version: a734957
vertical: streaming
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- arm64
- key: streaming
operator: In
values:
- 'true'
containers:
- env:
- name: CLUSTER
value: prod1
- name: OPENBLAS_NUM_THREADS
value: '4'
- name: DISABLE_LOGZIO
value: 'true'
image: :)
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sleep
- '25'
livenessProbe:
exec:
command:
- test
- '-S'
- /app/sockets/vsubmit.sock
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 1
name: vsubmit
readinessProbe:
exec:
command:
- test
- '-S'
- /app/sockets/vsubmit.sock
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 20
resources:
limits:
cpu: 5500m
memory: 8Gi
requests:
cpu: 5144m
memory: 6Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /app/config
name: config
- mountPath: /app/sockets
name: socket-data
- env:
- name: CLUSTER
value: prod1
- name: DISABLE_LOGZIO
value: 'true'
image: >-
:)
imagePullPolicy: Always
lifecycle:
preStop:
exec:
command:
- /bin/sleep
- '25'
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: 4003
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 20
name: vsubmit-nginx
ports:
- containerPort: 4003
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: 4003
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 20
resources:
limits:
memory: 256Mi
requests:
cpu: 256m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /app/config
name: config
- mountPath: /var/run/sockets
name: socket-data
dnsPolicy: ClusterFirst
priorityClassName: high-priority
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000
serviceAccount: vsubmit
serviceAccountName: vsubmit
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: graviton-c6g
operator: Equal
value: 'true'
- effect: NoSchedule
key: streaming
operator: Equal
value: 'true'
topologySpreadConstraints:
- labelSelector:
matchLabels:
appInst: vsubmit
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
volumes:
- configMap:
defaultMode: 420
name: :)
name: config
- emptyDir: {}
name: socket-data
status:
observedGeneration: 1
replicas: 0
`

Also, I didn't see there are any logs in Karpenter that shows the consolidation process but only some node events show unable to consolidate. Is there any places or log level I should set to see some detailed logs?

Just found out the pods have priorityClass, high-priority with a value of 100. Is that blocked the eviction process??

a high-priority class shouldn't block consolidation, it should only be used to determine the order of pod preemption if kube-scheduler needs to evict pods to make room on a node.

Using preferred anti-affinity and topology spreads can reduce the effectiveness of consolidation. At node launch, Karpenter attempts to satisfy affinity and topology spread preferences. In order to reduce node churn, consolidation must also attempt to satisfy these constraints to avoid immediately consolidating nodes after they launch. This means that consolidation may not deprovision nodes in order to avoid violating preferences, even if kube-scheduler can fit the host pods elsewhere.
referenced from official doc.
Do we have some plan to enable force evict even though it will temporarily violate the current constraints like spreadConstraint?

Another issue is we have a service running in only us-west-2a. It's used for setting up connections with camera. In the hpa, it does not allow scale down behavior. This behavior will block Karpenter from evicting pods(Like put 3 or 4 pods in 2 machines into 1 machine). I think we should we should not respect this behavior during scheduling but respect this "do not scale down" in the final state? Like though consolidation will make some pods unavailable, they should still evict those instead of keep them on the nodes to respect the rule?
Cluster AutoScaler can handle this well. I think this is a bug which we probably need to pay attention on. Maybe we can come up with another parameters to enable force eviction?

Closing in preference to #3694


"True" value doesn't work, needs to be "true"

Hi, in the page the Booleen value is invalid. See screenshot below. When its set to "True" the portal returns an error. When its set to "true" it works. The documentation on this page says to set it to "True" with an uppercase T, which is incorrect.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a007c4e9-6ec1-4be0-9cfb-a31e62013039
Version Independent ID: 3aab65db-895f-2f1f-3ffe-2e4f718bca56
Content: Shared device mode for iOS devices - Microsoft identity platform | Azure
Content Source: articles/active-directory/develop/msal-ios-shared-devices.md
Service: active-directory
Sub-service: develop
GitHub Login: @brandwe
Microsoft Alias: brandwe

@bubbletroubles
Thanks for your feedback! We will investigate and update as appropriate.

@bubbletroubles
I have created a PR to update the documentation and will now close out this issue. Please allow some time for the changes to reflect and if you don't see the document update, please let me know.
I will be more than happy to re-open this issue and look into what happened.
Thank you for your time and patience!


programatic usage with koa fails since migration from 1.0.0-alpha.4 to v1.0.0-rc3

This is my koa app that serves nuxt app.

if NODE_ENV=production it always serves "OK" and nothing happens on server
if NODE_ENV=development it always serves "OK" and in server logs I see that app fetches async data and then there is error UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Can't set headers after they are sent.
it works with express

import Koa from 'koa'
import { Nuxt, Builder } from 'nuxt'
import config from './../nuxt.config.js'

const app = new Koa()

config.dev = !(process.env.NODE_ENV === 'production')

const nuxt = new Nuxt(config)

if (config.dev) {
  const builder = new Builder(nuxt)
  builder.build()
}

app.use(async (ctx, next) => {
  ctx.status = 200
  await nuxt.render(ctx.req, ctx.res)
})

app.listen(3000)
console.log('Server listening')

Note that following piece of code is exactly the same I used with nuxt 1.0.0-alpha.4
app.use(async (ctx, next) => {
  ctx.status = 200
  await nuxt.render(ctx.req, ctx.res)
})

In<EMAIL_ADDRESS>they reworked the renderer, nuxt.render is now a connect middleware and not an async function like it used to be before. Moreover, it's a "terminating" middleware (it doesn't call next()), so it can't be directly promisified. You need to use a mocked response object, like this:
app.use(async ctx => {
  ctx.status = 200 // koa defaults to 404 when it sees that status is unset

  return new Promise((resolve, reject) => {
    const mockedResponse = {
      __proto__: ctx.res,
      end() {
        ctx.res.end.apply(ctx.res, arguments)
        // nuxt.render doesn't call next() on successful render,
        // resolve promise manually
        resolve()
      }
    }
    nuxt.render(ctx.req, mockedResponse, promise => {
      // nuxt.render passes a rejected promise into callback on error.
      promise.then(resolve).catch(reject)
    })
  })
})

@IlyaSemenov Thanks! I found lib that allows to use connect middleware with koa: https://github.com/vkurchatkin/koa-connect. I use it like this and it works:
import koaConnect from 'koa-connect'

app.use(async (ctx, next) => {
  ctx.status = 200
  await koaConnect(nuxt.render)(ctx, next)
})

@alekbarszczewski Yes I'm aware of koa-connect but it doesn't work correctly.
Add debug output to your code:
app.use(async (ctx, next) => {
  ctx.status = 200
  console.log("koaConnect called")
  await koaConnect(nuxt.render)(ctx, next)
  console.log("koaConnect returned")
})

You will see that await never returns. This is because koa-connect relies on the middleware to call next(), which nuxt.render doesn't.
That most probably leads to a memory leak in Node's async stack in the long run.

@IlyaSemenov Thanks man! I am using your code snippet now and it works (though it looks ugly haha).

 I should probably make a PR for koa-connect instead.

Hmm, there is another bug that might be related when in dev mode:
const builder = new Builder(nuxt)
builder.build()

When I have any file in static folder (for example static/images/sprites.svg and I visit http://localhost:3000/images/sprites.svg then it loads forever with no error and no response. However it works when I first nuxt build and then start my custom server in production mode.

Why is this ticket Closed ?
I am having issues with this ...

[ ] I know have to include some kind of Promise lib in my dependencies
[ ] CSS nano Nuxt package is briging up errors about promises
[ ] The solution proposed is ugly

I just "migrated" to express because I had too many problems with it :(

Even after "migrating" to express still had the same issues as running rc-3 in production with the ugly Koa hacks, page just hangs, staying on alpha4

See https://github.com/nuxt-community/koa-template/pull/21 for the page hanging fix.

@IlyaSemenov  when i use koa template the problem is sitll   Can't set headers after they are sent.
use middleware or redirect url like:
res.writeHead(302, {
'Location':'http://baidu.com'
});
res.end();

@shawon1220 Koa documentation clearly says:

Bypassing Koa's response handling is not supported. Avoid using the following node properties:

res.statusCode
res.writeHead()
res.write()
res.end()

So it's not a problem with Koa template, it's a problem with you not using Koa properly.


URL to long on candle socket

Hi there.
When I try to open a candle socket with no markets specified in the parameters on Binance, the call will crash i URL longer then 2048 characters exception. Can I work around this?
System.InvalidOperationException: URL length over the limit of 2048 characters. Consider splitting instruments in multiple connections.
at ExchangeSharp.BinanceGroup.BinanceGroupCommon.OnGetCandlesWebSocketAsync(Func`2 callbackAsync, Int32 periodSeconds, String[] marketSymbols)
Thanks,
Jacob

What is the url?

You need to split the instruments in multiple ws connections.
It's because we subscribe for Binance data by adding instrument's names to the url.


Linux support

I noticed that this requires Visual Studio to compile and only seems to work on Windows. Linus support when?

I currently don't have a linux machine set up so I probably won't be working on this.
This being a windows screensaver also complicates things.
That said, feel free to port it to linux if you want to, I will happily merge that.


Vector XYZW fast path not supported for vectors within immutable globals

The XYZW properties for Luau vectors are supported through the GETTABLEKS opcode, but not through the GETIMPORT opcode. Thus, access to the XYZW properties directly through immutable globals does not work.
Consider the global table foo with a single vector field vec:
// Create global "foo" with "vec" vector:
lua_newtable(L);
lua_pushvector(L, 0, 0, 0);
lua_setfield(L, -2, "vec");
lua_setglobal(L, "foo");

-- Error accessing "x" (or "X", or other properties)
print(foo.vec.x) -- error (.x goes through GETIMPORT opcode, which does not support the XYZW property fast path)

-- Works if the access to the vector is wrapped in parentheses:
print((foo.vec).x) -- ok

-- Works if the vector is first stored in a variable:
local v = foo.vec
print(v.x) -- ok

Workarounds:

Add the global to the mutableGlobals compilation option (not ideal).
Add XYZW fields as needed to a custom vector index metamethod.

I was running this in v0.609.

Second workaround is correct, you are supposed to setup the vector type metatable.
For an example, see this discussion https://github.com/luau-lang/luau/discussions/1298#discussioncomment-9818536


Cannot embed images in Linux, macOS and WSL.

Unintended behavior
The full path of the OS is displayed in the URL section and is not DataUrl.

@yoshi389111
I have made a few improvements and added support for macOS and Linux.
f928dd54878
Thank you so much! :+1:


Removing publish on master

Before creating my pull-request, I ensure:

[x] Javascript documentation is written
[x] Unit tests are written and green
[x] I've cleaned my WIP or meaningless commits
[x] Examples and documentation site is up-to-date

Codecov Report

Merging #68 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #68   +/-   ##
=======================================
  Coverage   91.56%   91.56%
=======================================
  Files           7        7
  Lines         320      320
  Branches       45       45
=======================================
  Hits          293      293
  Misses         14       14
  Partials       13       13

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c7d25d...291ac5d. Read the comment docs.


No license in pom file

I'm using Google OSS License plugin to show libraries' licenses to end users.
And after add commonmark-java dependency, I found license activity don't show license of commommark-java
I found there is license declaration on root (commonmark-parent) project.
https://github.com/commonmark/commonmark-java/blob/main/pom.xml#L264-L270
But, main java project(commonmark-parent) don't have license declaration.
https://github.com/commonmark/commonmark-java/blob/main/commonmark/pom.xml
And there is no license declaration on Maven POM File.
https://central.sonatype.com/artifact/org.commonmark/commonmark
Could you add license declaration on POM file of commonmark-java?

I found https://maven.apache.org/pom.html#inheritance and it looks Google OSS License plugin does not implement inheritance case from parent.
I'll submit issue on https://github.com/google/play-services-plugins

Agreed that this is a bug in play-services-plugins, but I'm happy to add it to at least the core pom to work around it, see:

https://github.com/commonmark/commonmark-java/pull/351

Confirmed that license appeared on OSS License activity with commonmark-java 0.24.0
Thank you very much.


CORS Issue?

The solution is probably really simple, I'm just not certain what it is.

Hello @faddat Thanks for pointing this out. Yes the solution is simple, we need to install Allow-Control-Allow-Origin: * chrome plugin. I will update that in readme.
link: https://chrome.google.com/webstore/detail/allow-control-allow-origi/nlfbmbojpeacfghkpbjhddihlkkiljbi?hl=en

@faddat  updated!

Awesome, thank you!


Research - Comparison of multiple Large Language Base Models

User story

As a software architect
I want / need to make a comparison of different Large Language Models and identify a collection that fits best to the goal of creating a LLM for Kubernetes
So that our created LLM will perform best.

Acceptance criteria

Different Large Language Base Models are compared to each other
The comparison contains at least a statement about a LLMs

size
adaptability
how complex the prompts to train with the LLM are (how complex will it be for the devs to work with the base model?)
how a new dataset must look like, so that the base model can be retrained with the new dataset (which requirements given our use case has the base model)
(other key decision criterias that are identified by the assignee)

Googles Gemma Model is part of the comparison
A candidate selection is created
The selected LLMs should fulfill the following requirements

The LLM is available under a permissive license
English must be supported

The selection should include at least 3 base models
Recommendations are made with which base model we should start training (like f.e. a ranking of the base models with an explanation for the ranking)

Definition of done (DoD)

Added only after week 5
The same for all features
Here goes the project specific part

DoD general criteria

Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release

My PO collegue sliced this issue into 3 other parts, added some other requirements and put them into the backlog, so this issue is no longer required.


Unable to use JSON Schema with "definitions"

When using a JSON Schema with "definitions" blocks, speccy returns the error Schema object cannot have additionalProperty: definitions.
For example, consider an OpenAPI v3.0.2 definition in yaml format with the following reference to a JSON schema:
...
components:
  schemas:
    Journal:
      $ref: 'mes-journal-schema/simple-schema.json'
...

Then a really simple JSON schema as follows:
{
  "id": "https://example.com/arrays.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Journal",
  "type": "object",
  "additionalProperties": false,
  "description": "Simple JSON Schema",
  "properties": {
    "message": {
      "$ref": "#/definitions/message"
    }
  },
  "definitions": {
    "message": {
      "type": "string"
    }
  }
}

Then invoking speccy via an npm script within my package.json:
  ...
  "devDependencies": {
    "speccy": "^0.8.7"
  },
  "scripts": {
    "check": "speccy lint -v -s info-contact -j openapi.yaml"
  },
  ...

When running the linter:
$ npm run check

><EMAIL_ADDRESS>check /Users/chris.nappin/Development/mes/mes-api-definitions
> speccy lint -v -s info-contact -j openapi.yaml

GET openapi.yaml
GET /Users/chris.nappin/Development/mes/mes-api-definitions/mes-journal-schema/simple-schema.json
Specification schema is invalid.

#/components/schemas/Journal
Schema object cannot have additionalProperty: definitions

Context
I'm attempting to add OpenAPI definitions to en existing project with JSON Schema already used for other purposes (e.g. generating Typescript interface types)
Your environment

Node Version: v8.10.0
Operating system and version: MacOS Mojave (v10.14.3)
Speccy Version: 0.9.0, 0.8.7, 0.8.6, 0.8.5

This is as designed, from the specification:

Additional properties defined by the JSON Schema specification that are not mentioned here are strictly unsupported.

Theoretically, https://github.com/wework/json-schema-to-openapi-schema might be able to rewrite your JSON Schema file (as I see you're using the -j option) in some way, and depending on how you're $refing into it, but I suspect this is out of scope for speccy itself.

Thanks for getting back to me.
Yes, we're trying to use speccy to convert a JSON schema to an Open API schema. I couldn't see any details on limitations in speccy in this area?
We'd like to have our schema files separate to our main openapi.yml (since the schema are used for other purposes) with a single $ref to each schema.
An example of the complex JSON schemas we're using is https://github.com/dvsa/mes-api-definitions/blob/develop/mes-journal-schema/schema-examiner-work-schedule.json. I'm not that familiar with JSON schema but I can't see how we can structure repeating nested objects without putting them under a "definitions" section?
Do you have any advice on how we can use speccy schema conversion with complex schemas?

Hey @chrisnappin what approach did you end up taking? I am doing something similar. Need to validate an OpenAPI spec that is in the request body.

We had to abandon any use of open api! This was a complete show stopper for us I'm afraid.

Ah darn.  Ok, thanks for responding.

Speccy should be migrating JSON schema "definitions" into the spec's "components/schemas" object, right?


Add Authentification

In this milestone I and @kennedyowusu  ,  we implement the following tasks
#PR Description

Remove the current_user method in ApplicationController
Install and setup devise gem
Register a new user.
logs users with a combination of email and password.
Store hashed password in the database.
Ask for confirmation by email.
Reset password view.
Style devise views
Refactor code and fix linters error

Hi @tarikbouari, this is a  a pair programming project your partner needs to add commits to this pr

Status: Invalid 🔴
Hi @tarikbouari, @kennedyowusu 🙋♀,
While you made a great effort in this project, I can't review your code.
Invalid Code Review Request
You have submitted a project that has a README that is not professional. Here is a key point to consider when documenting your README.

Kindly delete the live demo section if the application is not deployed. If project requirements say to deploy the project, the section should be restored and the link updated.

Your Code Review Request will be marked as invalid in your Dashboard, so please submit a new one once you are ready 🙏
Cheers 🥂  and Happy coding!!! 👯
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Remember to tag me @KDlamini  in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

Invalid Code Review Request does not count into the code reviews limit.

@KDlamini Thank you for your suggestion. We have gone ahead to implement the feedback you gave.


Psyringe cycle check

Includes #127
Updates psyringe.

Current coverage is 52.06% (diff: 46.31%)

Merging #130 into master will increase coverage by 0.41%

@@             master       #130   diff @@
==========================================
  Files            97         98     +1
  Lines          4885       4934    +49
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           2523       2569    +46
+ Misses         2124       2121     -3
- Partials        238        244     +6

Powered by Codecov. Last update c4ef48f...6b04c6d