Theme Versioning Proposal

Theme Versioning
Issue
Applications that transitively consume multiple versions of terra components are unable to reconcile theme variables.
Proposal
Version each theme property with the major version of the component.
Example:
terra-button - v3.0.0
terra-button-v3-background-color
Strategy
To enforce theme versioning a custom lint rule should be added into stylelint-config-terra. An example of the new rule can be found here.
This rule requires each custom property to include the package name and component version.
For passivity the new rule can have severity set to warning during the uplift and be changed to throw an error when all the initial warnings are satisfied.
Advantages

Applications can simultaneously theme multiple versions of a component
Aggregated theme libraries can passively add new theme versions on a component per component basis without requiring a major version bump of the entire package

Disadvantages

Each major version bump will require a new set of theme variables
Because of how lerna handles releases the package.json version is bumped after lint has run, this requires awareness of major version bumps before hand to not release the wrong set of theme variables

Additional Changes
As part of this change there is an opportunity to standardize the custom property pattern and enforce it with the lint rule. Here are a couple possible patterns.
(package-name)-(version)-(identifier+)$
(package-name)-(version)-(identifier+)-(pseudo selector?)-(property-name)$

+1

I've added an Additional Changes section for discussion regarding whether or not to enforce a more strict naming convention of custom property names.

I lean towards (package-name)-(version)-(identifier+)$

+1

+1

+1 looks good

+1:  (namespaced-package-name)-(version)-(identifier+)-(pseudo selectors)-(property-name)$
And if using classes in place of pseudo selectors/state modifiers, best practice recommendation will be to include them in the (identifier+) node.

What's an example of the current approach not working?   While this is simple to reason about with a single component, this gets more complex when defining a theme for a suite of applications.

Which version of variables is the theme configured to?
How do we migrate existing themes defined in configuration?
How does a tool allowing configuration of themes know which version to use?

What's an example of the current approach not working?

Select version 3 used --terra-select-option-selected-color to determine the background-color of a selected option. https://github.com/cerner/terra-core/blob/26d1abc138c571f78e1def38280a433a3cbf317c/packages/terra-form-select/src/SelectOption.scss#L22
Select version 4 used --terra-select-option-selected-color to set the text color of a selected option. https://github.com/cerner/terra-core/blob/9a61ec466c4d68504f69f9e674d922ecac2769a2/packages/terra-form-select/src/_Option.module.scss#L24
Application A was consuming version 3 and version 4 with the same theme resulting in both version of the component displaying incorrectly.

Which version of variables is the theme configured to?

There should be a theme for each version of a component.

How do we migrate existing themes defined in configuration?

Existing themes should be updated with versioning. 1 theme for each version after this change. Most components will only have 1 version.

How does a tool allowing configuration of themes know which version to use?

Eventually we plan to optimize theme repositories to analyze the node modules and only consume the necessary version of each theme for each component. In general I don't think there will be many versions of each theme.
Without versioning it is not possible for a single monolithic theme repository to be reliably consumed by an application that may consume multiple versions of the same component.
With versioning a monolithic theme repository can provide all versions of a theme for a component and allow an application to consume multiple versions of the same component.

This is a result of poor variable naming.  Specificity in naming would resolve this better than versioning in this specific case. For example: --terra-select-option-selected-color should be --terra-select-option-selected-backgroundcolor because that is what it is setting.. not the color.
IMO, more thought needs to go into the variable names to ensure they are specific to what they are doing.  Versioning will introduce a lot of complexity for client tooling to customize themes as well as terra development to analyze necessary themes.

Without versioning it is not possible for a single monolithic theme repository to be reliably consumed by an application that may consume multiple versions of the same component.

Yes it is possible. The property name IS the contact and we avoid breaking it at all costs.  --terra-select-option-selected-color  did not do what the name implied which is what cause that issue. If it were done correctly, --terra-select-option-selected-color would always set the option selected text color no matter what vesion of the component is being consumed. The text color is always the text color.

Versioning does introduce additional complexity, but it is the only way to resolve conflicts when they arise. Conflicts are rare, but they are present and usually are not found early enough to resolve the issue with variable naming without breaking other teams or major version bumping a component which creates the same theme versioning conflict.

So we're admitting that this is a rare occurrence that we're solving with the nuclear option that will make the use case of higher-order custom themes impossible to manage, map and maintain.  Any themes not applied automatically using a prebuilt theme will break when a component updates.  The theming strategy was conceived with client theming in mind and this is an incredibly non-passive and not taking that use case into consideration at all. This solution to tooling is not practical:

The tool should analyze the available theme variables on the page or within the bundle. This would allow theming a single version or multiple versions of a component. The tool could obscure the version of each variable if necessary.

If a tool can safely map a single variable to different versioned variables then why can't we just do this in code and abstract it from the consumers.  I would like to see terra move in the direction of using better property naming and stronger tests to find theme regressions before making this massive change.

this is an incredibly non-passive

Versioning the themes behind major versions of components is passive. If you major version bump the component a new version of the theme is created. They are protected behind major version bumps of the component.

This solution to tooling is not practical:

How does the tool work today? If the tool is not analyzing the variables available on the page / bundle then they are being statically set. Statically set variables do not account for changes or versions. Variables are added and removed regularly within components. This tool would end up having variables that don't work because they were removed, would be missing variables that were added, and after the lint changes it would have variables that don't exist.
At the end of the day we need a solution that allows a single monolithic theme repository to work for multiple versions of a component.  If we enforce the lint rules above and develop stronger naming conventions then theme versioning will not be as necessary, but it does mean older versions of components will cause conflicts. The theme repositories will need to specifically indicate starting at what version of each component they theme.

I'd like to collect concerns / problems / issues we are trying to solve. Let me know if there are others to add to the list.
Here are the ones I'm aware of:
a. Components use same variable name with 2 or more different values.
a {
  /*    Custom Property ↓             ↓ Value    */
  color: var(--terra-example-color, #000);
}
 .anchor {
  /*    Custom Property ↓             ↓ Value    */
  color: var(--terra-example-color, #111);
}

b. Component use a variable name that doesn't match the CSS property
a {
  /*    Custom Property ↓             ↓ Value    */
  background-color: var(--terra-example-color, #000);
}

c. Component changes the way a variable is used
c1. UX decideds border should be different between horizontal and vertical where before it was uniform
// Old version
.foo {
  /*    Custom Property ↓             ↓ Value    */
  border: var(--terra-example-border, 5px);
}
// Changes to
// New version
.foo {
  /*    Custom Property ↓             ↓ Value    */
  border-top: var(--terra-example-border-top, 3px);
  border-bottom: var(--terra-example-border-bottom, 3px);
  border-left: var(--terra-example-border-left, 7px);
  border-right: var(--terra-example-border-right, 7px);
}

c2. UX decideds border should be uniform where before it was different between horizontal and vertical
// Old version
.foo {
  /*    Custom Property ↓             ↓ Value    */
  border-top: var(--terra-example-border-top, 3px);
  border-bottom: var(--terra-example-border-bottom, 3px);
  border-left: var(--terra-example-border-left, 7px);
  border-right: var(--terra-example-border-right, 7px);
}
// Changes to
// New version
.foo {
  /*    Custom Property ↓             ↓ Value    */
  border: var(--terra-example-border, 5px);
}

d. Component removes a variable.
// Old version
.icon {
  display: inline-block;
  margin-top: var(--terra-example-margin-top: 3px);
}
// Changes to
// New version
.icon {
  display: inline-block;
}

e. Component adds a variable
// Old version
.icon {
  display: inline-block;
}
// Changes to
// New version
.icon {
  display: inline-block;
   margin-top: var(--terra-example-margin-top: 3px;
}

f. Client interfaces with versioned variables
// Client CSS
:root {
 --terra-example-color: #f00;
}

// Cerner Application CSS
.example {
  color: var(--terra-example-v1-color, #000);
}

I think a. and b. can be mitigated with stylelint rules, @StephenEsser has some of these in place.
In regards to c1. and c2., my thoughts are maybe always writing long-hand CSS properties is best. This way it doesn't matter if they all need to be the same value or different for horizontal and vertical.
In regards to d., I see this as passive so long as the visuals stay the same. I'm thinking of an instance where margin-top is used to vertically center and switch to an implementation to use flexbox for vertical center where margin-top is no longer neccesary. One issue I see with this is people including variables that are no longer used in components.
In regards to e., my thoughts are this is could be a non-passive change, for example if we add a color / background color, that could cause issues in themes like the low light theme.
In regards to f., my thoughts are this is a passive change so long as the variable fallback is the same as it was prior to making it a theme variable.
In regards to g. I see two options. Remove the version number from the theme variable name or add an abstraction layer between client theme and terra theme.
// Client CSS
:root {
 --cerner-example-color: #f00;
}

// Abstraction layer
:root {
 --terra-example-v1-color: --cerner-example-color;
}

// Cerner Application CSS
.example {
  color: var(--terra-example-v1-color, #000);
}

Abstraction layer would have to expand for all major versions of a variable:
// Abstraction layer
:root {
 --terra-example-v1-color: --cerner-example-color;
 --terra-example-v2-color: --cerner-example-color;
 --terra-example-v3-color: --cerner-example-color;
 --terra-example-v4-color: --cerner-example-color;
   etc....
}

Proposal
Version each theme variable independently as needed.
Example:
For the terra select example where the --terra-select-option-selected-color variable was reused for a different intent. With this proposal the variable would be renamed:--terra-select-option-selected-color-v1
So the lifecycle of a normal variable would be:
The variable starts off un-versioned
--terra-button-color
Then if color is co-opted to mean something else the variable would be changed to:
--terra-button-color-v1
If its intent changes again it would be named:
--terra-button-color-v1
Pros:
Variables don't change versions unless there is a conflict.
Gives us a standard way to address version conflicts without resorting to non standard naming.
Cons:
Versioning is reactionary and hard to test for. An issue may be found and a .1 version may need to be released with the updated versioned variable name.
The theme variables are difficult to relate to a version of a component.

I much prefer the versioning variables as needed. I will say that we should also strive to use versioning as a last resort. For example with the code above, we could do this
// select v4
&.is-selected {
  background-color: var(--terra-select-option-selected-background-color, #0079be);
  color: var(--terra-select-option-selected-color, #fff);
  color: var(--terra-select-option-selected-font-color);
}

Supporting the old property as a passivity layer until its safe to remove it.  When versioning is required, I'd like that we only support a versioned variable for a 1 or 2 major releases before it becomes un-versioned again. This will prevent a situation where we have all  different variable versions.   Also, an alternative to versioning the variables is to scope them. For example see https://codepen.io/anon/pen/LXgjoa
Having scoped properties I think works using  https://www.npmjs.com/package/postcss-css-variables#caveats. This would allow us to not change variable names and delete versioned scopes once they are no longer of concern. Not sure this is 100% bullet proof but it does address the example above.

I looked through the alernative proposal and I tested postcss-css-variables plugin and it seems to be a passive change to swap out. It can compile the variables the same as postcss-custom-properties with the addition of scoping variables under selectors other than :root.
A concern I see with the proposed alterntive @mhemesath is that I belive it will require theme authors to use the selector with the version class name on it in their theme definition file. For example, in the codepen demo, it seems a theme author would have to use .v4 .is-selected-xyz to set the value for --terra-select-option-selected-color in their theme file.
If v4 .is-selected-xyz only exists in the component's CSS, I'm not sure how a theme author would customize the value of --terra-select-option-selected-color.  I also think this would be more challenging to document how to use theme variables if we need to note using specific selectors to target versions.
Using only CSS custom properties, and versioning them if there is a conflict, it will be easier to aggregate the theming variables for documentation purposes. I like the idea that we should also strive to use versioning as a last resort. I think if/when there are conflicts with css custom properties, we should look at ways to avoid introdcing a version number if we can make it work like the example @mhemesath posted.
// select v4
&.is-selected {
  background-color: var(--terra-select-option-selected-background-color, #0079be);
  color: var(--terra-select-option-selected-color, #fff);
  color: var(--terra-select-option-selected-font-color);
}

Based on that, I think we should move forward with the proposal noted here.

A concern I see with the proposed alterntive @mhemesath is that I belive it will require theme authors to use the selector with the version class name on it in their theme definition file.

Good point, probalby have the scope be something like
.terra-button-v4 {
  --terra-button-color: red;
}

Theme Propsal Resolution

Version theme variables independently as needed when naming collison occurs.
Match the variable version with the version the of component it was introduced with.

If a conflict in variable naming occurs in terra-button v3, variable version number would use v4 and a new terra-button v4 package would be released.
The variable starts off un-versioned in package terra-button v3
--terra-button-color
Then if color is co-opted to mean something else the variable would be changed to:
--terra-button-color-v4
If its intent changes again it would be named:
--terra-button-color-v5

Create tool to find css custom properties that are not used in an app.

Update all current vairables to match stylelint-config-terra rules.

Create migration guide when theme variables change

Future consideration

Drop off version number after a set amount of time


Update sample2.cs

Add the Main()

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

@foreverlearner  thanks!


OCM-3366 | fix: CLI options not validated client-side

Applying the same validations that are done interactively when the fields are provided via CLI options.
This also refactors the code a bit.

/test all

@osherdp do we want to include this in the release?

@osherdp do we want to include this in the release?

@ciaranRoche this one is not critical

/hold
requires some more testing

/unhold

/lgtm


docs: add warning to generated files

Summary of Changes
Add a warning at the top of generated documentation files that they must not be edited directly.

:tada: This PR is included in version 0.18.0 :tada:
The release is available on:

npm package (@latest dist-tag)
npm package (@latest dist-tag)
v0.18.0
GitHub release

Your semantic-release bot :package::rocket:


feat: (structure redesign) platform docs restructuring for 4.0

Summary
This PR reorganizes the remaining sections of the platform docs improving navigation and structure.

Administer
Understand
Learn How-To
Use Platform

Key Changes

moving, renaming and restructuring
no content updates, just small typos and naming
adjust vale style rules
fix vale style violations

Dependencies

PR #269 must be merged first to avoid conflicts.

TODO

[x] re-flowing the menu order
[ ] convert @site calls to relative paths

Closes DOC-253

@deniseschannon thank you for the commit, I managed to fix all broken links and references in the last commit. Re-tagging you for review.

Closing in favor of #279


docs-bug(COMPONENT): v17.1 (the last version to omit m3) is not present in version switcher

Documentation Feedback
The version switcher on the docs site allows me to switch between latest and v16, but not between latest and v17.1, which is the version of Angular Material before the components began switching to m3 specs. Because v16 is not 1:1 compatible with v17, this means there's no way to view the documentation for a version that I (and probably many others) am stuck using for an indefinite period of time.
Affected documentation page
https://material.angular.io/components/categories

See also: https://github.com/angular/components/issues/28788

Valid point, it's a bit confusing that the v17.3 docs app is already on M3 while all v17.3 projects have M2 by default.

I misunderstood that both were available; rather than making available version 17.1, it would also be fine if there was another way to switch to m2 on latest 17.x in the docs. Ideally that way would also be in the URL so one can send coworkers links to either one specifically.

Yeah, it would be good to have the last Angular 17 version of Angular Material with m2.
It would be awesome to have something like that:

...
17.3.1 (latest) M3
17.3.1 (latest)
v16
...

Starting from 17.3.1 (latest) M3, all new version of the doc would be m3. So for example the next version will be 18.0.0 (latest) and it will use m3.

I agree.
Currently, Angular 17 is the latest LTS version, and M.2 is the default appearance when used in a project, M.3 was still experimental in Angular Material 17:
https://v17.material.angular.io/guide/material-3
This behaviour causes a lot of confusion, since our customers/UX team see the M.3 styling in the documentation for verson 17, even though M.2 is the "stable" and default styling.


fix: TabsControl Buttons in MyTheme Demo working again

Hello!
Found something interesting. If yout try to use your custom theme (aka MyTheme) in Tabalonia.Demo, you'll find out that all buttons stop working
Before (App.axaml)
        <custom:CustomTheme />

         <!--<StyleInclude Source="Styles/MyTheme.axaml" />-->

After (App.axaml)
        <!-- <custom:CustomTheme /> -->

         <StyleInclude Source="Styles/MyTheme.axaml" />

I've noticed that MyTheme in Demo is your implementation of Fluent Theme. So, I compared them and found out that in MyTheme there are some little typos in command bindings.

Sorry, there was some mistakes in my git author config, so I had to rebase and force-push. But there should not be any conflicts


New-SelfSignedCertificate into "Trusted Root CAs"

I have created CA certificate using  New-SelfSignedCertificate and created dev certificate, which is signed by the CA certificate.
The problem is, I am disallowed by the cmdlet to store the CA certificate under "Trusted Root CAs", I have to do it manually.
Without having the CA certificate in the "Trusted Root CAs" there will be no trust.
Why Is this use case not supported?
EDIT:
When creating an unsigned cert with -TestRoot switch, there is again no trust.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: a2cc6356-761f-c845-6ccb-8fd12b1dc7ba
Version Independent ID: 08045f21-38f3-4c0a-bbeb-c51cdb6615e8
Content: New-SelfSignedCertificate (pki)
Content Source: docset/winserver2022-ps/pki/New-SelfSignedCertificate.md
Product: w10
GitHub Login: @JasonGerend
Microsoft Alias: jgerend

To make it easier for you to submit feedback on articles on learn.microsoft.com, we're transitioning our feedback system from GitHub Issues to a new experience.
As part of the transition, this GitHub Issue will be moved to a private repository. We're moving Issues to another repository so we can continue working on Issues that were open at the time of the transition. When this Issue is moved, you'll no longer be able to access it.
If you want to provide additional information before this Issue is moved, please update this Issue before December 15th, 2023.
With the new experience, you no longer need to sign in to GitHub to enter and submit your feedback. Instead, you can choose directly on each article's page whether the article was helpful. Then you can then choose one or more reasons for your feedback and optionally provide additional context before you select Submit.
Here's what the new experience looks like.
Note: The new experience is being rolled out across learn.microsoft.com in phases. If you don't see the new experience on an article, please check back later.
First, select whether the article was helpful:

Then, choose at least one reason for your feedback and optionally provide additional details about your feedback:

Article was helpful
Article was unhelpful

Finally, select Submit and you're done!


executable file for macOS

Hi,
Is it possible to get an executable version of the app.
I cant install it via App Store since my Mac is managed.
Thanks

Hi @nivibilla, thanks for reaching out. Would you able to install using brew or executables are necessary? Using "Completions" feature requires Accessibility permission, would you be able to grant it? Other features work without any permissions.

Not sure. If brew is easier then pls lmk how to do it I'll be happy to test it out.

Hi @AugustDev wanted to circle back on this. Is it possible to install via homebrew?

Same here. I love the mac and ios apps on my personal devices, but would like a way to install it on my work mac.

🆙


Best way to create a cell array of character vectors instead of strings

Excellent, and useful tool. Thanks.
I'm using it to store information on variables with names.  For example, my Yaml file might contain:
cvs:
  names: [SAG_WT, SAG_POW, SAG_CC, SAG_OF_P80]

After loading the yaml file to a struct called model_data these names become a cell array as expected:
>> model_data.cvs.names

ans =

  1×4 cell array

    {["SAG_WT"]}    {["SAG_POW"]}    {["SAG_CC"]}    {["SAG_OF_P80"]}

However, tables don't seem to accept cell arrays of strings and seem to require char
>> array2table(rmse, 'VariableNames', model_data.cvs.names)
Error using table (line 326)
The VariableNames property is a cell array of character vectors. To assign multiple variable names, specify names in a string
array or a cell array of character vectors.

Is there a way to force a cell array of character vectors instead of strings?  Or, any other ideas on how to easily populate the RowNames or VariableNames properties of tables?

The only way I've found to convert them is this, but its not very pretty...
cv_names = convertStringsToChars([model_data.cvs.names{:}]);

Hey, you can use cellstr to convert a cell of strings to a cell of char arrays.
Also, they implemented string array support for tables somewhere between R2018a and R2021b if upgrading is an option for you. In that case you could set the ConvertToArray option of yaml.loadFile to true to get an array of strings, which you can pass to table.

Thanks very much.  Didn't know about cellstr. Also, yes I need to upgrade. I'm still using R2019b.


File Menu Shortcut Key Clash

VSCode Version: Code 1.18.0 (dcee2202709a4f223185514b9275aa4229841aa7, 2017-11-08T21:22:49.932Z)
OS Version: Windows_NT x64 10.0.16299
Extensions:

Extension
Author (truncated)
Version

vscode-great-icons
emm
2.1.20

cpptools
ms-
0.14.2

File menu have 2 sub menu used "S" key for shortcut key.
One is "将工作区另存为(S)",  another is "保存(S)  Ctrl + S".
In English is "Save To Workspace" and "Save".

Steps to Reproduce:

Alt + F -> S, can not save current file now
Check the File menu, will find 2 sub menu used same key("S"), Save to workspace and Save
Save to workspace is first match menu.

Reproduces without extensions: Yes/No

I can reprodcue this. There is also a clash with the "O" key.
@bpasero do you mind if I change these 2 mnemonics?

Ok actually a duplicate of https://github.com/Microsoft/vscode/commit/f8a0b7dcc5e42d841d5f31f9d4d19faae6d27083
@bpasero keep in mind that there seems to be one more clash. The one with O

Closing this one as @bpasero you already closed the other.


Implement working ranges

We need something like working ranges in ComponentKit. Something to allow components to listen for whether they're visible, nearing the viewport, or are being focused on by the user.
I wrote a threadsafe ringbuffer like we have in working ranges: https://gist.github.com/ocrickard/d222f9d744fee2ad0fdee53625d42ded
Not sure if it's useful, but it could serve as a strong base for recording layouts on the UI thread inside the scroll component, and then reading them on a BG thread to traverse the hierarchy and do the frame computations.

This task is perfect for Kfir, but I can't find you on Github using your name, so hope you're checking this.


Autocomplete fails with paths listed in PYTHONPATH via .env file

Environment data
VS Code version:
Code 1.18.1 (929bacba01ef658b873545e26034d1a8067445e9, 2017-11-16T18:23:26.125Z)
Python Extension version:
0.8.0
Python Version:
Python 2.7.14 :: Anaconda, Inc.
OS and version:
Darwin x64 16.7.0
Actual behavior
Auto complete does not work properly with additional packages listed in PYTHONPATH added via the .env file.
Expected  behavior
Auto complete will list documentation, methods, functions, etc. from packages listed in PYTHONPATH.
Steps to reproduce:

Add paths to PYTHONPATH via the workspace .env file
Notice that pylint seems to pick up the extra paths just fine. But auto complete fails to work.

Please could you provide a sample (sample repo).
Also, have you tried reloading VS Code (currently you'll need to reload vscode after editingthe .env file)

Here is an example repo: https://github.com/xgalaxy/python_test
I created a .env file that exposes the libs folder modules to PYTHONPATH. While this exposes the libs path to the linters, autocompletion and cmd+click navigation and documentation for the modules in libs doesn't function unless you also add the libs path to python.autocomplete.extraPaths.
In my opinion I think the extra paths and .env system needs an overhaul. I shouldn't have to list the same paths in multiple places to get a complete functioning development environment.
I would remove the .env file completely and rename python.autocomplete.extraPaths to python.extraPaths and expose the paths in that setting to every python tool; the linters, autocompletion system, documentation system, debugger, etc.

Thanks for the sample repo.

In my opinion I think the extra paths and .env system needs an overhaul. I shouldn't have to list the same paths in multiple places to get a complete functioning development environment.

Agreed, this will be resolved with the issue #228.
Just confirming, this issue would get resolved along with #228. All you'll need to do is add the settings in .env. file (and you wouldn't need to reload vscode).


Updating conda package for PyQt in Python 3.4

Working to solve issue #60 to resolve building in Appveyor for Python 3.4.

For Windows pyqt-4.11.4-py34_7 works fine, but there is no 34_7 version in Linux. Instead its 34_4 in Linux, which support sip 4.18.

Problem solved, despite the double CI webhook. Working on a fork in the future will prevent this.


{{imgix-image}} should respect and maintain any parameters passed in as part of the path

Fixed as of 0.1.0


Additional clarity for pylintrc_tweaks how-to.

@nedbat @clintonb @cpennington

I'm still not sure it makes sense to include the Django plugin(s) by default, since this isn't only intended for use with Django. However, we can hash that out later.
:+1:

I agree that this is an unfortunate way to solve the problem.  I'll figure out a better way, but in the meantime, :+1:


Adjust path of back button on FAQ page

Overview
When a user clicks on a hyperlink in the FAQ responses on page 2 and clicks back to return to the FAQ page, the user is returned to page 1. To improve the user experience, the back button should return to the page that the hyperlink was found on.
Action Items

[ ] Adjust back button path to return back to original page that hyperlink was found on

Resources/Instructions
If I click on the donate hyperlink which is on page 2 of the FAQ and click back from the hyperlink, I get returned to the first page of the FAQ.

Progress - Storing the page state in the URL is mostly done
Blockers - Need to migrate FAQ data from prod backend to test backend.  Also, do we need to remember which question dropdowns were opened?
Availability - Maybe 6 hours this week
ETA - by next meeting (12/9)

Ideally, we should remember all dropdowns in case a user accidentally clicks a link in a different dropdown than they were expecting. However, I believe this is not a very common occurrence so it would be okay if the only dropdown remembered is the dropdown that contained the link that the user clicked on.

@fyliu Do you know a quick way to migrate the FAQ data to the test backend?

@kevindphan I did an export table from prod and then import into the staging. Just the faq table.


fix: index out of bounds in GLTFAnimationFormat

Current Situation
public void updateFrame(float time, MeshAnimationFrame frame) {
    int upperFrame = 0;
    while (upperFrame < times.size() && times.get(upperFrame) < time) {
        upperFrame++;
    }

(https://github.com/MovingBlocks/Terasology/blob/develop/engine/src/main/java/org/terasology/rendering/gltf/GLTFAnimationFormat.java#L212-L216)
In updateFrame, upperFrame is initialized to 0 and then increased if was smaller than the size of the TFloatList times and some other condition. This can lead to upperFrame == times.size().
In the case where I faced the index out of bounds error, TFloatList times was the same size as List<T> data. This means, upperFrame could became equal to data.size() as well.
Now lowerFrame is set to the maximum of 0 and upperFrame - 1 in https://github.com/MovingBlocks/Terasology/blob/develop/engine/src/main/java/org/terasology/rendering/gltf/GLTFAnimationFormat.java#L217 .
Further below, we see
if (upperFrame == lowerFrame) {
    interpolator.interpolate(data.get(lowerFrame), data.get(lowerFrame), 0, target);
} else {
    float t = (time - times.get(lowerFrame)) / (times.get(upperFrame) - times.get(lowerFrame));
    interpolator.interpolate(data.get(lowerFrame), data.get(upperFrame), t, target);
}

(https://github.com/MovingBlocks/Terasology/blob/develop/engine/src/main/java/org/terasology/rendering/gltf/GLTFAnimationFormat.java#L219-L224)
Due to how lowerFrame is set, the if condition can only be true if upperFrame == lowerFrame == 0.
In all other cases, the else branch will be executed.
As we can see, in that branch both TFloatList times and List<T> data will be accessed with index upperFrame. If indeed upperFrame equals the size of one or both of them, we'll run into an index out of bounds error.
Proposed Solution
As we try to access times and data with index upperFrame in all cases where upperFrame != 0, IMO we should not be able to increase upperFrame up to the size of times. That's why, I propose to change the while condition to
public void updateFrame(float time, MeshAnimationFrame frame) {
    int upperFrame = 0;
    while (upperFrame < times.size() - 1 && times.get(upperFrame) < time) {
        upperFrame++;
    }

How to Reproduce

Checkout https://github.com/Terasology/LightAndShadowResources/pull/62
Start Terasology and try to start LightAndShadow
Observe the index-out-of-bounds error when the game tries to load the [red|black]Queen or [red|black]King prefabs.

If you'd like to use the debugger, you can set a conditional breakpoint at if (upperFrame == lowerFrame) {, conditioned to upperFrame == 41 and when it breaks observe that data.size() == times.size() == 41.

Pinging @immortius as the author of GLTFAnimationFormat.java to take a look and assess whether the proposed solution makes sense in this context.

I'm fine with merging it now to allow us to test LaS in the playtest with https://github.com/Terasology/LightAndShadowResources/pull/62 merged.
@immortius It would be highly appreciated though, if you could double-check this anyway and ping me if you see anything that seems off to you, so we can clean that up in a follow-up.


Convoy Gluster FS issues with rancher 1.1  and docker 1.11

Currently gluster FS is broken in rancher 1.1 and docker 1.11.
There have been numerous issues open regarding this but no effective solution yet.
https://github.com/rancher/convoy/pull/134
https://forums.rancher.com/t/glusterfs-glusterfsvolume-doesnt-have-required-field-name/3417
Any idea when this will be resolved ?

Duplicate of #4411.


Some elements have been added and rectified as discussed in the review.

Portal changes have been made and other errors have been rectified.
Worked on making the website more responsive.
Linking has been started.

Portal changes have been made and other errors have been rectified.
Worked on making the website more responsive.
Linking has been started.

@muskan278  why does a commit made by you have license copyright added by @rmaggon6

@DiptoChakrabarty , @rmaggon6 was facing some issues with Github account password. This is why I made that commit.

Portal changes have been made and other errors have been rectified.
Worked on making the website more responsive.
Linking has been started.

@muskan278 why does a commit made by you have license copyright added by @rmaggon6

@DiptoChakrabarty , @rmaggon6 was facing some issues with Github account password. This is why I made that commit.

Portal changes have been made and other errors have been rectified.
Worked on making the website more responsive.
Linking has been started.

@muskan278 why does a commit made by you have license copyright added by @rmaggon6

@DiptoChakrabarty , @rmaggon6 was facing some issues with Github account password. This is why I made that commit.

Okay @muskan278  is   @rmaggon6   still facing the issues  with GitHub  account , his last commit was a long back , has he been able to log back in

@DiptoChakrabarty bhaiya I have sorted the issue .No problem now .

@DiptoChakrabarty bhaiya I have sorted the issue .No problem now .

Cool then


Add generic functions to call clGet*Info functions

Issue #6 shows that not all values for calls to functions such as clGetDeviceInfo are documented.
Also OpenCL extensions often add new values to such function calls.
The current implementation requires all param_names to functions such as clGetDeviceInfo to be known beforehand.
It would be more useful (and future proof) to provide functions to get data from functions such as clGetDeviceInfo for any param_name values, so that they can be called for new or undocumented values.

Delivered last month.


Write a C program to convert minutes to seconds

Description
Write a C program to convert minutes to seconds
1 minute = 60 seconds
Input  : 5
Output : 300

How to contribute

Save the solution in ConvertMinutesToSeconds.c file
Add ConvertMinutesToSeconds.c file in convert-minutes-to-seconds folder

!assign


Add to docs how to setup lineage callbacks by default on airflow

Describe your changes :

Added a note explaining how airflow users can set the OpenMetadata lineage callbacks by default for DAGs, Tasks or specifics operators.

Type of change :

[] Bug fix
[] Improvement
[] New feature
[] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] Documentation

Checklist:

[x] I have read the CONTRIBUTING document.
[x] I have commented on my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] All new and existing tests passed.

Reviewers

Thank you @mariotaddeucci


Polymer does not work in Chrome Extensions

Polymer does not work in Chrome Extensions without modifications. The reason is restrictions by the Chrome Extensions infrastructure preventing inline scripts from running.
To allow Polymer to function in Chrome Extensions without using tools that will slow development (tools like Vulcanize), and to help with its adoption, please consider having the Polymer components use external scripts only. I believe that some components already do this.

Hey guys,
Is PolymerLabs/Crisper the (best) solution for this problem?

@baazoo : I don't think it is.
Crisper seems to do something weird...it actually builds a standard html file structure, ie it makes a, html element, with a head and body within it.
On the following html :
<link rel="import" href="../bower_components/some-element.html">

<polymer-element name="my-element">
<template>

  <some-element id="someElement"></some-element>

</template>
</polymer-element>

it produces this :
<link rel="import" href="../bower_components/some-element.html">

<polymer-element name="my-element">
<template>

  <some-element id="someElement"></some-element>
  </template>
</polymer-element>

It's not the right thing at all. It almost seems like it is simply the -csp part of vulcanize separated into a different command.
If it we to simply do what it says and extract script from html, then it would be possible to wrap it in some script to run on a whole directory tree, but it really doesn't do the right thing, imo.

@davidmaxwaterman
I am new to Polymer and haven't dug deep into Vulcanize. As far as I understood, it doesn't have CSPize feature by itself, however, Crisper by definition does split the inline script (not styles though) to make them CSP proof.
Could you please elaborate a bit on what exactly should be done to use Polymer for Chrome Extension development? Is it the problem with Custom Elements?
I am building a Chrome App / Extension duo and I don't want to create them in a totally separate line.

@baazoo vulcanize used to have a --csp option, and looking up a reference, I see that, indeed, it was simply moved to 'crisper':
https://github.com/polymer/vulcanize#what-happened-to-feature-from-0x
Actually, I've found following this thread a bit confusing, but let me try to answer your question....er, in answering your question, I began to wonder if the additional , , and  tags make any difference - it makes the file look very odd, but it might not make any difference. Indeed, a quick test on one of my 0.5 projects suggest that the extra stuff does not make any difference. I didn't realise that was the case. A bit annoying, but anyway.
Perhaps this means it'll work just fine, as a build step anyway.
I was thinking that cripser (and vulcanize --csp) where designed to only work on the main index.html file; but that seems wrong. It'd be nice if it were able to process a whole tree of files, but a script can do that, I think.

Crisper solved the issue
I tested with crisper on Polymer 1.0 and it worked like  a charm.
After installing crisper, I modified the Vulcanize task as below:
var vulcanize = require('gulp-vulcanize'); // added for use in chrome extensions
var crisper = require('gulp-crisper'); // added for use in chrome extensions

// Vulcanize imports
gulp.task('vulcanize', function () {
  var DEST_DIR = 'dist/elements';

  return gulp.src('dist/elements/elements.vulcanized.html')
    .pipe($.vulcanize({
      stripComments: true,
      inlineCss: false, // changed from 'true' for use in chrome extensions
      inlineScripts: false // changed from 'true' for use in chrome extensions
    }))
    .pipe(crisper()) // added for use in chrome extensions
    .pipe(gulp.dest(DEST_DIR))
    .pipe($.size({title: 'vulcanize'}));
});

PS:

This should work for Chrome Extensions popup (and future sidebar) pages but not for Content Scripts without WebComponents polyfil since Chrome does not allow registerElement in content scripts
Crisper only separates scripts from HTML not the styles. I am not sure if Inline styles are allowed in Content Scripts.
I am sure there is a workaround for using Polymer in Content Scripts but I have no interest in that.

@baazoo yes, it will work because it doesn't much matter about the extra , , and  tags...iinm. they're kind of added anyway as part of the import code (if you look at what is 'inside' an import using dev tools, it's basically another html document).
So, as part of a build step, or if you're 'converting' 3rd party code, like in bower_components, then it would work fine. You can probably make it work in a hierarchy too, with an appropriate wrapper.
It isn't so good at converting our own code, but perhaps that is not such a big deal since we can do that ourselves fairly easily (or just always do it using a build step).
Yeah, I think I agree that it's pretty much good enough and so this issue could be close....just my personal opinion.
That is...if only it had a grunt plugin (I guess 'brocolli' users might also have an issue).

So, as a supplement, however as a main point try this amazing free app. Createer.com is an online software for easily designing online ads, responsive landing pages and print materials such as flyers, posters, covers, invitations and more. Createer lets you easily share your designs on social media, send them via email, download as HTML, image or PDF. You can even design UI mockups for mobile as for standard display (html, js, css always generating without any coding required from you). Createer also includes 3,000 predesigned templates (browse here: https://beta2.createer.com/Templates?ghi) and over 250,000 images (illustrations, photos and icons) to work with in its user-friendly drag’n’drop enviroment. And finally this amazing stuff is here: https://beta2.createer.com/?ghi

baazoo: I don't understand how that can be used. Who runs that script at what point?
The best solution for me has been to:

install the Chrome Dev Editor from [1],
Open the directory containing my bower_components,
Right-click and say "Refactor for CSP" in the context menu.

I have to do this each time I install a new component, but it is a no-op for previously CSPized components, and it leaves my own components alone if they already use external scripts. Fairly low footprint.
[1] https://chrome.google.com/webstore/detail/chrome-dev-editor-develop/pnoffddplpippgcfjdhbmhkofpnaalpg

@viridium
You should change the gulpfile.js as I explained above and then run gulp. Don't remember the specific task, maybe the default will do the job.

I can't help but think there's a more direct equivalent to the CDE 'refactor for CSP' function. Perhaps something like this (remove 'echo' to make it do something):
$ for file in $(find bower_components -name \*.html); do js=${file/\.html/.js}; echo crisper --source "$file" --html "$file" --js "$js"; done

The best tool is polybuild, it combines vulcanize, crisper and also handles bower_components
$ polybuild popup.html
> popup.build.html and popup.build.js

Yeah, that looks quite nice - they've come up with quite a few new tools since this issue was opened (probably because this issue was opened). I haven't tried polybuild yet, but it looks hopeful :)
I'm curious if there is any hope regarding fixing the browser so this isn't an issue to start with, as @sjmiles and @ebidel mentioned.

Are there still issues? and is polybuild the easiest path?

As of a couple months ago, there were still issues. To me, the easiest path right now is to extract the inline resources using the Chrome Dev Editor. It's easiest, but not easy. I have to run the CSP-ize function available in that app each time a new non-CSP-ized component enters my source tree.
It would be best if components worked out of the box, without the dev having to run any builds.

The right tool for this is polybuild, which will make your elements follow CSP guidelines.
@justinfagnani, good candidate for CLI tooling.

Thanks @azakus, got it.

Managed to figure out how to split files to adhere to default extension CSP settings using polymer-build:
const gulp = require('gulp');
const PolymerBuild = require('polymer-build');

const HtmlSplitter = PolymerBuild.HtmlSplitter;
const PolymerProject = PolymerBuild.PolymerProject;

gulp.task('build', function() {
  const project = new PolymerProject(require('./polymer.json'));
  const sourcesHtmlSplitter = new HtmlSplitter();

  return project.sources()
    .pipe(sourcesHtmlSplitter.split())
    .pipe(gulp.dest('./build/'));
});

Be sure to configure sources in your polymer.json file.
See https://www.polymer-project.org/2.0/docs/tools/polymer-json#sources


Multiple node toggle editable children

Godot version: 3.1 custom
Issue description:
Would be really nice to be able to toggle editable children for multiple nodes at once.

This is already implemented on master, closing.


incidents POC

Codecov Report
Attention: 16 lines in your changes are missing coverage. Please review.

Comparison is base (5aadaf5) 100.00% compared to head (98de5e6) 90.12%.

Files
Patch %
Lines

src/graphql/incidents.ts
5.88%
16 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##            master     #108      +/-   ##
===========================================
- Coverage   100.00%   90.12%   -9.88%
===========================================
  Files           16       18       +2
  Lines          137      162      +25
  Branches        18       19       +1
===========================================
+ Hits           137      146       +9
- Misses           0       16      +16

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Option to add owned shares in config and possibly calculate market value of holdings?

It would be nice to have the option to add number of shares owned in the config like this:
symbols:
  - symbol: AAPL
    shares: 100
  - symbol: MSFT
    shares: 10

This would in turn make it possible to add the attributes shares and market_value (shares * state/regularMarketPrice) to each stock entity :-)

Sorry this is not intended for portfolio management; that extends into buy/sell/multiple purchases details. But you can use template to create custom sensor.

Thanks for you answer. Template sensors would work fine technically but not practically when your dealing with many symbols together with the flex-table-card. I'm trying to bend it a bit by adding extra attributes via customize.yaml and shares works fine somehow calculations (shares*regularMarketPrice) don't seem to be supported in customize. I might end up forking your component...

Agreed.
I myself was capturing and doing some custom calculations. Plus the fact that my count was dynamically changing due to auto-investment of dividends made it very custom. I opted to do this outside through AppDaemon and some of it available here (StockAggregator).
I also use a combination of auto-entities to decide which stock entities to include in a card and then display additional info via custom-secondary-info which supports templates.


Page titles containing ' (single quote) render incorrectly

Page titles containing ' (single quote) render with their html code
for example : ' (single quote) is replaced by ’
I'm using confluence-publisher 0.14.0 and Confluence 7.4.11

Hi @azocankara, thank you for reporting the issue. In fact, all HTML special characters are rendered escaped. This is due to the use of the HTML backend of AsciidoctorJ. I will prepare a bug fix.

Support for HTML special characters in page titles is not available in the current 0.0.0-SNAPSHOT version. @azocankara, would you be able to give it a try and let me know whether that works for your case? Thank you!


Extract copywriting text to a single file

Making it easier for non-developers to help with copywriting

Codecov Report

Merging #34 into master will increase coverage by 0.19%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #34      +/-   ##
==========================================
+ Coverage   28.56%   28.76%   +0.19%
==========================================
  Files          73       73
  Lines        7824     7906      +82
==========================================
+ Hits         2235     2274      +39
- Misses       5279     5319      +40
- Partials      310      313       +3

Impacted Files
Coverage Δ

...controllers/component_plugin_binding_controller.go
67.56% <0.00%> (-2.71%)
:arrow_down:

controller/lib/files/files.go
49.28% <0.00%> (+1.00%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4f2633b...bf7f544. Read the comment docs.


Request new release

Can we get a new release?
Specifically need Bump xml2js and aws-sdk due to vulnerability identified in pre xml2js 0.5.0 versions.
Thannks!

Sure. Will work on this.

Is there a timeframe expected?


Support injection of variable determined at runtime (Node Instance ID) into userdata template (passed as --node-labels via bootstrap.sh --kubelet-extra-args)

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?
I want my managed nodes to be named by their EC2 instance id. To achieve this, I'm taking this approach, and trying to set the node labels using pre_userdata and/or post_userdata in my nodegroups config to fetch the node's instance id:
# Early, using pre_userdata, fetch my instance id from instance metadata:
INSTANCE_ID=$(curl http://<IP_ADDRESS>/latest/meta-data/instance-id)

Then add --node-labels=instance=$${INSTANCE_ID} to the list of --kubelet-extra-args passed to the bootstrap.sh script in userdata.
This part is the problem.
It appears this list is currently generated from the kubelet_extra_args terraform input variable to the aws_eks_node_groups contained in the managed_ng input to the aws-eks-managed-node-groups submodule, the --kubelet-extra-args in my actual userdata ends up looking like:
--kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup-image=ami-xyz,eks.amazonaws.com/sourceLaunchTemplateVersion=3,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=managed-ondemand-m5large-lt-123,eks.amazonaws.com/sourceLaunchTemplateId=lt-123 --max-pods=29'

so I see that the terraform variable kubelet_extra_args is being filled by the submodule, but there's no way I can see to inject something determined at runtime, like my --node-labels=instance=<my-dynamically-determined-instance-id to the list of --kubelet-extra-args passed to the bootstrap.sh script in userdata.
Describe the solution you would like
Since the current template for my ami_type clearly substitutes the kubelet_extra_args terraform variable value verbatim into what gets passed  to --kubelet-extra-args, I thought I could just do this in my node group config:
      # This value is appended to the --kubelet-extra-args arg passed to bootstrap.sh.
      # Note the use of double $$ to delay variable interpolation of INSTANCE_ID until runtime, per
      #     https://developer.hashicorp.com/terraform/language/expressions/strings#escape-sequences-1
      kubelet_extra_args = "--node-labels=instance=$${INSTANCE_ID}"

..thinking it would insert --node-labels=instance=${INSTANCE_ID} literal into the bootstrap.sh call, resulting in something like this:
--kubelet-extra-args '--node-labels=instance=i-0c4c273fd4402d894,eks.amazonaws.com/nodegroup-image=ami-xyz,eks.amazonaws.com/sourceLaunchTemplateVersion=3,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=managed-ondemand-m5large-lt-123,eks.amazonaws.com/sourceLaunchTemplateId=lt-123 --max-pods=29'

Alas, what I'm getting is that my instance=i-0c4c273fd4402d894 is omitted from --node-labels, and I'm guessing it's because something in the submodule is overwriting my --node-labels with their --node-labels.
Describe alternatives you have considered
I next tried to specify k8s_labels in my node group config to refer to the env var INSTANCE_ID (which as you'll recall I set in pre_userdata), like this:
      # INSTANCE_ID var is set in pre_userdata before bootstrap.sh is called
      k8s_labels = { instance = "$${INSTANCE_ID}" }

..thinking that might get appended to the --node-labels list by the submodule, but it does not. It errors out of course because k8s_labels value "${INSTANCE_ID}" is evaluated by terraform and validated as a label (sigh):
main  | Error: error updating EKS Node Group (mycluster:managed-ondemand-m5large-lt-20221021180859713400000002) config: InvalidParameterException: Invalid value: ${INSTANCE_ID} : field must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')
main  | {
main  |   RespMetadata: {
main  |     StatusCode: 400,
main  |     RequestID: "d42ea2a6-5c50-4112-9706-3bb91f367678"
main  |   },
main  |   ClusterName: "mycluster",
main  |   Message_: "Invalid value: ${INSTANCE_ID} : field must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')",
main  |   NodegroupName: "managed-ondemand-m5large-lt-20221021180859713400000002"
main  | }
main  |
main  |   with module.eks_blueprints.module.aws_eks_managed_node_groups["mng_lt"].aws_eks_node_group.managed_ng,
main  |   on .terraform/modules/eks_blueprints/modules/aws-eks-managed-node-groups/main.tf line 1, in resource "aws_eks_node_group" "managed_ng":                                                                                main  |    1: resource "aws_eks_node_group" "managed_ng" {

Another Approach
I suppose another obvious approach would be to insert the node ec2 instance id from terraform using a module output or data source, but I had assumed this wouldn't/couldn't work bc the instance isn't created at the time the launch template is created. Am I wrong here?
Yet Another (Better) Approach
Instead of reverse-engineering the way all the modules are passing and evaluating variables, how about if we just simply added this to the template and exposed it as a controllable feature of modules/aws-eks-managed-node-groups just like pre_userdata, format_mount_nvme_disk, and post_userdata?
Additional context
I'm using version 4.7.0 of the module and:
main  | Initializing provider plugins...
main  | - Reusing previous version of hashicorp/null from the dependency lock file
main  | - Reusing previous version of hashicorp/helm from the dependency lock file
main  | - Reusing previous version of hashicorp/random from the dependency lock file
main  | - Reusing previous version of hashicorp/tls from the dependency lock file
main  | - Reusing previous version of hashicorp/cloudinit from the dependency lock file
main  | - Reusing previous version of hashicorp/aws from the dependency lock file
main  | - Reusing previous version of hashicorp/time from the dependency lock file
main  | - Reusing previous version of gavinbunney/kubectl from the dependency lock file
main  | - Reusing previous version of hashicorp/kubernetes from the dependency lock file
main  | - Reusing previous version of terraform-aws-modules/http from the dependency lock file
main  | - Reusing previous version of hashicorp/local from the dependency lock file
main  | - Using previously-installed hashicorp/random v3.4.3
main  | - Using previously-installed hashicorp/tls v3.4.0
main  | - Using previously-installed hashicorp/cloudinit v2.2.0
main  | - Using previously-installed hashicorp/aws v4.34.0
main  | - Using previously-installed hashicorp/time v0.9.0
main  | - Using previously-installed hashicorp/kubernetes v2.14.0
main  | - Using previously-installed hashicorp/null v3.1.1
main  | - Using previously-installed hashicorp/helm v2.7.1
main  | - Using previously-installed gavinbunney/kubectl v1.14.0
main  | - Using previously-installed terraform-aws-modules/http v2.4.1
main  | - Using previously-installed hashicorp/local v2.2.3

Not stale. Have assigned this to a teammate and will provide more info ASAP.

this is due to the way that EKS managed node groups control and manage the bootstrap script invocation. There are workarounds available though https://github.com/awslabs/amazon-eks-ami/issues/844

Thanks @bryantbiggs for putting me on the right trail to solve that mystery. Seeing your doc updates in the aws_eks module, I see that:

It is the EKS Managed Node Group service in the default eks ami which prevents customization of the execution of the bootstrap.sh script via userdata, by overwriting at runtime (at least) the userdata section that calls bootstrap.sh.
Until the merging of this PR, which adds support for sourcing an optional bootstrap-env.sh script and fixing bootstrap.sh to not overwrite KUBELET_EXTRA_ARGS, one must hack away at the actual bootstrap.sh script with sed.

I interpret this to mean that I should be able to achieve what I want (customizing node labels to include instance ids) by simply specifying pre_userdata in my managed_node_groups input to eks_blueprints and that modifying eks_blueprints itself should be unnecessary.
Since I'm trying to prepend to one of the args contained by KUBELET_EXTRA_ARGS (--node-labels) instead of just appending to KUBELET_EXTRA_ARGS itself, I'm using a slightly different sed incantion, using Bash parameter expansion with regex pattern replacement. I'm not otherwise changing anything in KUBELET_EXTRA_ARGS, so I must also ensure that this env var is set, and I follow the pattern of sourcing a custom script:
      pre_userdata = <<-EOT
        #!/bin/bash
        set -ex
        # Insert "instance=$${INSTANCE_ID}," into the --node-labels arg passed to kubelet config bootsrap process in /etc/eks/bootstrap.sh.
        # See here for details on why it's necessary to overwrite bootstrap script like this:
        #     https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/eks_managed_node_group/README.md#container-runtime--user-data
        #     https://github.com/aws-ia/terraform-aws-eks-blueprints/issues/1073#issuecomment-1339537583
        #     https://github.com/awslabs/amazon-eks-ami/issues/844
        EC2_METADATA_API_TOKEN=$(curl --silent -X PUT "http://<IP_ADDRESS>/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
        INSTANCE_ID=$(curl -H "X-aws-ec2-metadata-token: $${EC2_METADATA_API_TOKEN}" --silent http://<IP_ADDRESS>/latest/meta-data/instance-id)
        cat <<-EOF > /etc/profile.d/bootstrap.sh
        # environment variables to be sourced in /etc/eks/bootstrap.sh
        export KUBELET_EXTRA_ARGS="$${KUBELET_EXTRA_ARGS\/--node-labels=\/--node-labels=instance=$${INSTANCE_ID},}"
        EOF
        # Patch bootstrap.sh to source extra environment variables
        sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
        # Patch bootstrap.sh to respect/use existing KUBELET_EXTRA_ARGS env var
        sed -i 's/KUBELET_EXTRA_ARGS=$2/KUBELET_EXTRA_ARGS="$2 $KUBELET_EXTRA_ARGS"/' /etc/eks/bootstrap.sh
      EOT

Upon creation of the nodes' launch template, I confirmed in the console that the final, generated userdata looks as expected (I think):
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex

# User-supplied pre userdata
#!/bin/bash
set -ex
# Insert "instance=${INSTANCE_ID}," into the --node-labels arg passed to kubelet config bootsrap process in /etc/eks/bootstrap.sh.
# See here for details on why it's necessary to overwrite bootstrap script like this:
#     https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/eks_managed_node_group/README.md#container-runtime--user-data
#     https://github.com/aws-ia/terraform-aws-eks-blueprints/issues/1073#issuecomment-1339537583
#     https://github.com/awslabs/amazon-eks-ami/issues/844
EC2_METADATA_API_TOKEN=$(curl --silent -X PUT "http://<IP_ADDRESS>/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
INSTANCE_ID=$(curl -H "X-aws-ec2-metadata-token: ${EC2_METADATA_API_TOKEN}" --silent http://<IP_ADDRESS>/latest/meta-data/instance-id)
cat <<-EOF > /etc/profile.d/bootstrap.sh
# environment variables to be sourced in /etc/eks/bootstrap.sh
export KUBELET_EXTRA_ARGS="${KUBELET_EXTRA_ARGS\/--node-labels=\/--node-labels=instance=${INSTANCE_ID},}"
EOF
# Patch bootstrap.sh to source extra environment variables
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
# Patch bootstrap.sh to respect/use existing KUBELET_EXTRA_ARGS env var
sed -i 's/KUBELET_EXTRA_ARGS=$2/KUBELET_EXTRA_ARGS="$2 $KUBELET_EXTRA_ARGS"/' /etc/eks/bootstrap.sh

--//--

However I'm not seeing my "patches" in the /etc/eks/bootstrap.sh in newly-created node instances. Any ideas what I did wrong here?

For future onlookers frustrated by 3 levels of nested character escape sequence rules (terraform, bash, sed), this is what I ended up on that works, properly hacking the setting of KUBELET_EXTRA_ARGS inside bootstrap.sh:
      pre_userdata = <<-EOT
        #!/bin/bash
        #
        # Patch bootstrap.sh handling of --kubelet-extra-args to prepend our desired label to the --node-labels embedded in it.
        # This script hacking is required because the EKS managed node group service in the default AMI is overwriting parts of the userdata
        # most notably the line that calls `/etc/eks/bootstrap.sh`, in order to inject --kubelet-extra-args generated from external configuration:
        # --node-labels, --max-pods, --b64-cluster-ca, etc. Also, because `bootstrap.sh` doesn't currently respect the KUBELET_EXTRA_ARGS env var,
        # the only option is to hack the line _inside_ `bootstrap.sh` which handles the `--kubelet-extra-args` arg, so that it prepends
        # 'instance=$INSTANCE_ID,' into the comma-separated `--node-labels` list passed into `bootstrap.sh` inside the `--kubelet-extra-args` arg.
        # See here for details on why it's necessary to overwrite bootstrap script like this:
        #     https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/eks_managed_node_group/README.md#container-runtime--user-data
        #     https://github.com/aws-ia/terraform-aws-eks-blueprints/issues/1073#issuecomment-1339537583
        #     https://github.com/awslabs/amazon-eks-ami/issues/844
        set -ex
        EC2_METADATA_API_TOKEN=$(curl --silent -X PUT "http://<IP_ADDRESS>/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
        INSTANCE_ID=$(curl -H "X-aws-ec2-metadata-token: $${EC2_METADATA_API_TOKEN}" --silent http://<IP_ADDRESS>/latest/meta-data/instance-id)
        sed -i "s/KUBELET_EXTRA_ARGS=\$2/KUBELET_EXTRA_ARGS=\"\$${2\/--node-labels=\/--node-labels=instance=$INSTANCE_ID,}\"/" /etc/eks/bootstrap.sh
      EOT

Even more time consuming for me here was the fact that I wrongly thought I needed to set create_launch_template = true for the pre_userdata to work, and was (and still am) getting errors with nodes not connecting to the control plane. Once i set create_launch_template = false the issues went away. I would however like to get the launch template working, but of course that's a separate issue...

Reopening bc although I've got pre_userdata script functionally doing what I need, now I can't get my managed nodes to join the cluster as long as I have create_launch_template = true. @bryantbiggs @Zvikan could you take a quick look at my blueprints config at the bottom and let me know if you see any configuration issue?
I can aws ssm start-session into the nodes before the group creation times out, and see that their userdata and my bootstrap.sh hack look good, but in the kubelet logs I see this:
Dec 15 22:35:21 ip-10-0-11-202.us-west-1.compute.internal kubelet[3165]: E1215 22:35:21.023225    3165 kubelet.go:2469] "Error getting node" err="node \"ip-10-0-11-202.us-west-1.compute.internal\" not found"
Dec 15 22:35:21 ip-10-0-11-202.us-west-1.compute.internal kubelet[3165]: E1215 22:35:21.083298    3165 kubelet.go:2394] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"

It appears this is caused by the CNI not being configured in the cluster. I believe this to be something blueprints took care of when I was using create_launch_template = false, and I must be missing some essential bit of config now that I've set it true.
It looks like the blueprints vpc-cni submodule is only created when the enable_amazon_eks_vpc_cni input variable to modules/kubernetes-addons is set. Since I'm not setting this, and it defaults to false, I’m assuming we haven’t needed to explicitly use this module since something else in blueprints is doing that. I first assumed it was probably the aws_eks module underneath, which has input variable cluster_addons that can be used to specify the vpc-cni addon to be installed. However, this also defaults to off/empty, and here you can see that eks_blueprints does not even specify cluster_addons when invoking the eks module, so I’m perplexed ATM about how/if the vpc-cni is being created.
Here's my config. It's mostly copied from the mng lt example. Note it's using my fork, but this is basically blueprints 4.15 with minor changes made to crossplane module to add unrelated support for helm/terraform providers:
module "eks_blueprints" {
  # source = "github.com/aws-ia/terraform-aws-eks-blueprints?ref=v4.15.0"
  source = "github.com/timblaktu/terraform-aws-eks-blueprints?ref=crossplane-helm-provider"
  cluster_name    = local.name
  cluster_version = local.cluster_version
  vpc_id             = module.vpc.vpc_id
  private_subnet_ids = module.vpc.private_subnets
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access = true
  create_cluster_security_group = false
  cluster_security_group_id     = aws_security_group.eks.id
  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node all ports/protocols"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
    egress_all = {
      description      = "Node all egress"
      protocol         = "-1"
      from_port        = 0
      to_port          = 0
      type             = "egress"
      cidr_blocks      = ["<IP_ADDRESS>/0"]
      ipv6_cidr_blocks = ["::/0"]
    }
    ingress_cluster_to_node_all_traffic = {
      description                   = "Cluster API to Nodegroup all traffic"
      protocol                      = "-1"
      from_port                     = 0
      to_port                       = 0
      type                          = "ingress"
      source_cluster_security_group = true
    }
  }
  cluster_kms_key_arn = data.aws_kms_key.eks_secrets.arn

  managed_node_groups = {
    mng_lt = {
      node_group_name = "managed-ondemand-m5large-lt"
      ami_type               = "AL2_x86_64"
      release_version        = ""
      create_launch_template = true
      launch_template_os     = "amazonlinux2eks"
      capacity_type          = "ON_DEMAND"
      instance_types         = ["m5.large"]
      format_mount_nvme_disk = true
      desired_size           = 2
      max_size               = 2
      min_size               = 2

      create_iam_role         = false
      iam_role_arn            = aws_iam_role.managed_ng.arn
      additional_iam_policies = []

      subnet_type = "private"
      subnet_ids  = module.vpc.private_subnets
      public_ip   = false

      enable_monitoring = true
      eni_delete        = true

      http_endpoint               = "enabled"
      http_tokens                 = "optional"
      http_put_response_hop_limit = 3

      update_config = [{
        max_unavailable_percentage = 30
      }]
      timeouts = [{
        create = "10m"
        update = "20m"
        delete = "20m"
      }]

      k8s_taints = []

      pre_userdata = <<-EOT
        EC2_METADATA_API_TOKEN=$(curl --silent -X PUT "http://<IP_ADDRESS>/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
        INSTANCE_ID=$(curl -H "X-aws-ec2-metadata-token: $${EC2_METADATA_API_TOKEN}" --silent http://<IP_ADDRESS>/latest/meta-data/instance-id)
        sed -i "s/KUBELET_EXTRA_ARGS=\$2/KUBELET_EXTRA_ARGS=\"\$${2\/--node-labels=\/--node-labels=instance=$INSTANCE_ID,}\"/" /etc/eks/bootstrap.sh
      EOT

      block_device_mappings = [
        {
          device_name = "/dev/xvda"
          volume_type = "gp3"
          volume_size = 100
        }
      ]

      additional_tags = {
        subnet_type = "private"
      }

      remote_access = false
    }
  }
  tags = local.tags
}

module "eks_blueprints_kubernetes_addons" {
  # source = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=v4.15.0"
  source = "github.com/timblaktu/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=crossplane-helm-provider"

  eks_cluster_id       = module.eks_blueprints.eks_cluster_id
  eks_cluster_endpoint = module.eks_blueprints.eks_cluster_endpoint
  eks_oidc_provider    = module.eks_blueprints.oidc_provider
  eks_cluster_version  = module.eks_blueprints.eks_cluster_version

  tags = local.tags

  # enable_metrics_server     = true
  enable_cluster_autoscaler = true
  enable_argocd = true

  argocd_helm_config = {
    namespace        = "argocd"
    timeout          = "1200"
    create_namespace = true
    values           = [templatefile("${path.module}/templates/argocd-values.yaml", {})]
    set_sensitive = [
      {
        name  = "configs.secret.argocdServerAdminPassword"
        value = bcrypt_hash.argo.id
      }
    ]
  }
  argocd_manage_add_ons = true
  crossplane_aws_provider = {
    enable               = true
    provider_aws_version = "v0.33.0"
    additional_irsa_policies = [
      "arn:aws:iam::aws:policy/AdministratorAccess",
    ]
  }

  crossplane_kubernetes_provider = {
    enable                      = true
    provider_kubernetes_version = "v0.5.0"
  }
  crossplane_helm_provider = {
    enable                = true
    provider_helm_version = "v0.12.0"
  }

  crossplane_terraform_provider = {
    enable                     = true
    provider_terraform_version = "v0.5.0"
    additional_irsa_policies = [
      "arn:aws:iam::aws:policy/AdministratorAccess",
    ]
  }

  argocd_applications = var.cleanup_argocd_applications ? {} : {
    addons = {
      path               = "chart"
      repo_url           = "https://github.com/aws-samples/eks-blueprints-add-ons.git"
      add_on_application = true
    }
  }
}

Next area to double-check was public/private endpoint access and associated DNS settings in my VPC. Since I had recently added these two lines in my eks_blueprints config:
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access = true

and the previous defaults being used were, effectively:
  cluster_endpoint_private_access = false
  cluster_endpoint_public_access = true

..I had effectively turned on private endpoint access to restrict all node<-->controlplane traffic to intra-vpc.
Reverting back to the default settings (only public endpoint enabled), I saw some improvement in node behavior. On node group creation, instead of not ever seeing the nodes populated in the nodegroup console (and having to look their instance ids up using the cluster name in ec2 console), I now am seeing the nodes immediately, albeit they are still not joining due to same node condition error reported above:
Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"

I'm still flummoxed as to why the vpc-cni addon isn't being installed by default, or in any of the examples (aside from the "custom networking" or "custom ami" types which I'm not using), and what are the circumstances under which it needs to be explicitly added to or enabled in eks_blueprints config.
So, I decided to just add this to my eks_blueprints_kubernetes_addons config:
  enable_amazon_eks_vpc_cni = true
  amazon_eks_vpc_cni_config = {
    most_recent       = true
    resolve_conflicts = "OVERWRITE"
  }

and for the first time since setting create_launch_template = true, my nodes successfully joined the cluster (as well as setting my node labels via custom userdata per this original issue)! So, my questions are:

Why/when is it necessary to explicitly enable_amazon_eks_vpc_cni?
Is it necessary to explicitly enable_amazon_eks_vpc_cni when using a node group launch template with an ami_type but no custom_ami_id?
When not using a node group launch template, and letting enable_amazon_eks_vpc_cni default to false, why did I not get this same "cni config uninitialized" error?


SUBMARINE-758. Resolve duplicate 'storage' key in values.yaml

What is this PR for?
Fix bug: Duplicate key 'storage' in values.yaml.
What type of PR is it?
[Bug Fix]
Todos

[ ] - Task

What is the Jira issue?
https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-758
How should this be tested?
https://travis-ci.org/github/MortalHappiness/submarine/builds/761097136
Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

Thank you for the contribution. LGTM.


Pasting gb18030 string in integrated terminal modifies the string.

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version:
OS Version: Win 11
Version: 1.81.0 (user setup)
Commit: 6445d93c81ebe42c4cbd7a60712e0b17d9463e97
Date: 2023-08-02T12:37:13.485Z
Electron: 22.3.18
ElectronBuildId: 22689846
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621

Steps to Reproduce:

Open a powershell instance in an empty directory
Run dotnet new sln --name MySolution
Run dotnet new console --output "ᠮᠣᠩᠭᠣᠯ  ᠦᠰᠦᠭ  བོད་ཡིག  ۋېڭكقق ھس قك كدسدق د  ꆈꌠꁱꂷ"
Run dotnet new console --output "ᥘᥣᥭᥰ ᥖᥭᥰ ᥖᥬᥲᥑᥨᥒᥰ  ᦎᦷᦑᦺᦖᦺᧈᦉᦲᧇᦉᦸᧂᦗᧃᦓᦱ"
Open the folder in VS Code
Run dotnet add '.\ᠮᠣᠩᠭᠣᠯ  ᠦᠰᠦᠭ  བོད་ཡིག  ۋېڭكقق ھس قك كدسدق د  ꆈꌠꁱꂷ\ᠮᠣᠩᠭᠣᠯ  ᠦᠰᠦᠭ  བོད་ཡིག  ۋېڭكقق ھس قك كدسدق د  ꆈꌠꁱꂷ.csproj' reference '.\ᥘᥣᥭᥰ ᥖᥭᥰ ᥖᥬᥲᥑᥨᥒᥰ  ᦎᦷᦑᦺᦖᦺᧈᦉᦲᧇᦉᦸᧂᦗᧃᦓᦱ\ᥘᥣᥭᥰ ᥖᥭᥰ ᥖᥬᥲᥑᥨᥒᥰ  ᦎᦷᦑᦺᦖᦺᧈᦉᦲᧇᦉᦸᧂᦗᧃᦓᦱ.csproj' in the integrated terminal. It should complete correctly but the message may be incorrect (e.g. Reference '..\???? ??? ???????  ?????????????????\???? ??? ???????  ?????????????????.csproj' added to the project.). Note: add the project paths using tab completion. Pasting the command can fail (see below).
Remove the project reference to ᥘᥣᥭᥰ ᥖᥭᥰ ᥖᥬᥲᥑᥨᥒᥰ  ᦎᦷᦑᦺᦖᦺᧈᦉᦲᧇᦉᦸᧂᦗᧃᦓᦱ.csproj inside of project file ᠮᠣᠩᠭᠣᠯ  ᠦᠰᠦᠭ  བོད་ཡིག  ۋېڭكقق ھس قك كدسدق د  ꆈꌠꁱꂷ.csproj
Copy the previous dotnet add [proj1] reference [proj2] command from the integrated terminal
Paste the command using ctrl + v
The text is identical to the previous command and runs correctly
Delete the project reference again
Paste the command via right-click. Notice that the text is different.
Running the command fails because the project could not be found

/duplicate https://github.com/xtermjs/xterm.js/issues/701


Apache failure when using Microsoft as an OpenID Connect Provider

Hello,
I've configured my app to work with Google as an OP - and it does so flawlessly - and now I'm trying to get Microsoft to work as well.
I followed the configuration setup as detailed from one of the links in the wiki here.
When I go to a protected page, I get the Discovery page - good. I click on the link for Microsoft and as expected it asks me to go through the Microsoft account login flow. Once I've signed in to my Microsoft account successfully, I get sent back to my app, but then after a little bit of processing, the page fails to load stating "The connection was reset."
The Apache logs states failure here:
[core:notice] [pid 429] AH00051: child pid 434 exit signal Segmentation fault (11); possible coredump in /etc/httpd

Where openidc seems to be failing is during the process of parsing and validating the id_token. It will always fail after oidc_proto_get_key_from_jwks. The lines in the error log right before the failure are:
[authz_core:debug] [pid 434] mod_authz_core.c(809): [client <IP_ADDRESS>:50592] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[authz_core:debug] [pid 434] mod_authz_core.c(809): [client <IP_ADDRESS>:50592] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[auth_openidc:debug] [pid 434] src/mod_auth_openidc.c(2938): [client <IP_ADDRESS>:50592] oidc_check_user_id: incoming request: "/oauth2callback?(null)", ap_is_initial_req(r)=1
[auth_openidc:debug] [pid 434] src/util.c(853): [client <IP_ADDRESS>:50592] oidc_util_get_cookie: returning "mod_auth_openidc_session" = <null>
[auth_openidc:debug] [pid 434] src/util.c(956): [client <IP_ADDRESS>:50592] oidc_util_request_matches_url: comparing "/oauth2callback"=="/oauth2callback"
[auth_openidc:debug] [pid 434] src/mod_auth_openidc.c(1805): [client <IP_ADDRESS>:50592] oidc_handle_post_authorization_response: enter
[auth_openidc:debug] [pid 434] src/util.c(1233): [client <IP_ADDRESS>:50592] oidc_util_read_form_encoded_params: parsed: 1034 bytes in to 3 elements
[auth_openidc:debug] [pid 434] src/mod_auth_openidc.c(1676): [client <IP_ADDRESS>:50592] oidc_handle_authorization_response: enter, response_mode=form_post
[auth_openidc:debug] [pid 434] src/mod_auth_openidc.c(1359): [client <IP_ADDRESS>:50592] oidc_authorization_response_match_state: enter (state=pgS8pcHJSPH8uBv6VHoTxQdykDE)
[auth_openidc:debug] [pid 434] src/mod_auth_openidc.c(686): [client <IP_ADDRESS>:50592] oidc_restore_proto_state: enter
[auth_openidc:debug] [pid 434] src/util.c(204): [client <IP_ADDRESS>:50592] oidc_util_jwt_verify: enter: JWT header={"alg": "dir", "enc": "A256GCM"}
[auth_openidc:debug] [pid 434] src/util.c(1712): [client <IP_ADDRESS>:50592] oidc_util_create_symmetric_key: key_len=32
[auth_openidc:debug] [pid 434] src/util.c(853): [client <IP_ADDRESS>:50592] oidc_util_get_cookie: returning "mod_auth_openidc_state_pgS8pcHJSPH8uBv6VHoTxQdykDE" = "eyJhbGciOiAiZGlyIiwgImVuYyI6ICJBMjU2R0NNIn0..Oe-5CFHzKVKhDh5T.VVszg\
fhxtTvD_XVpEnkfMxAOPB0XFIGTabu3ekb2Ir5S0g_QCMyDURIt3Ppsc20pa4cVHphZPzioODfu2wteQSi9C3c7LTB-az-5z2iwXlo3c1sBNZyFNAjseSWv3IiEs23Jy5rgN5ETIzzBNSAUbK-N9uWYqUCh9kWuyXxiIFcjoNuxZxMitc6nLAgLH71p3WyXoLuP0GYGBmOqCmAwJTVk0iMpIPyEwBKbwFxp4f\
BAFVINfGPGJH_0jfMcjDJdUIH7_MpcUUZTwZbvbeTO49vQlb1xArHBZ7s6cc2BXgCSC7sBdVVwldaM1J2LSIEKMX_4-H-QQMfPHHEjntc8R96PjtHu0AXd_-JHsc3UpbqaIhF3N6QUraSiZP0LAL9qOqcx7Weh2QPQcxkwxR8-m0TA9XXYxhiKN-B_w1qX5QiRyAlZTfliJ4rd1FvXhSyBo0WSTeCX9sZdj_P\
NL6Z0X3QTMQLAV5UcL9TsIxfMWSz9mVHXIXcDMI7URJi1Bm9f_AXj0gegAOJ5ebbRj6ya3A9Pj4g-doflMKLoHxRifWyhqF9TsrZ0cDmSxw86v_Cvl8ZDtMZmAwnTxT-oUAOyr-Rp7GXbZYR6dsUSvRGfsd15A4Y0ZeA.Eab_TDcRa9aFzdThkvc8uA"
[auth_openidc:debug] [pid 434] src/util.c(811): [client <IP_ADDRESS>:50592] oidc_util_set_cookie: adding outgoing header: Set-Cookie: mod_auth_openidc_state_pgS8pcHJSPH8uBv6VHoTxQdykDE=;Path=/; expires=Thu, 01 Jan 1970 00:00:00 GMT\
;HttpOnly
[auth_openidc:debug] [pid 434] src/util.c(204): [client <IP_ADDRESS>:50592] oidc_util_jwt_verify: enter: JWT header={"alg": "dir", "enc": "A256GCM"}
[auth_openidc:debug] [pid 434] src/util.c(1712): [client <IP_ADDRESS>:50592] oidc_util_create_symmetric_key: key_len=32
[auth_openidc:debug] [pid 434] src/mod_auth_openidc.c(197): [client <IP_ADDRESS>:50592] oidc_get_browser_state_hash: enter
[auth_openidc:debug] [pid 434] src/mod_auth_openidc.c(734): [client <IP_ADDRESS>:50592] oidc_restore_proto_state: restored state: {"original_url": "http://localhost/NoAuth/Logout.html", "timestamp":<PHONE_NUMBER>, "state": "pgS8pcHJSP\
H8uBv6VHoTxQdykDE", "original_method": "get", "issuer": "https://login.microsoftonline.com/9188040d-6c67-4c5b-b112-36a304b66dad/v2.0", "response_mode": "form_post", "response_type": "code id_token", "nonce": "6iwIMXMyHwYI2wRpw45S\
8EoEaP_8yUMe6NeU4Aj8sJ4"}
[auth_openidc:debug] [pid 434] src/util.c(1312): [client <IP_ADDRESS>:50592] oidc_util_file_read: file read successfully "/opt/software/rt/etc/OIDCMetadata/login.microsoftonline.com%2F9188040d-6c67-4c5b-b112-36a304b66dad%2Fv2.0.pro\
vider"
[auth_openidc:debug] [pid 434] src/util.c(1312): [client <IP_ADDRESS>:50592] oidc_util_file_read: file read successfully "/opt/software/rt/etc/OIDCMetadata/login.microsoftonline.com%2F9188040d-6c67-4c5b-b112-36a304b66dad%2Fv2.0.con\
f"
[auth_openidc:debug] [pid 434] src/util.c(1312): [client <IP_ADDRESS>:50592] oidc_util_file_read: file read successfully "/opt/software/rt/etc/OIDCMetadata/login.microsoftonline.com%2F9188040d-6c67-4c5b-b112-36a304b66dad%2Fv2.0.cli\
ent"
[auth_openidc:debug] [pid 434] src/metadata.c(318): [client <IP_ADDRESS>:50592] oidc_metadata_client_is_valid: client (https://login.microsoftonline.com/9188040d-6c67-4c5b-b112-36a304b66dad/v2.0) metadata did not contain a "client_\
secret_expires_at" setting
[auth_openidc:debug] [pid 434] src/proto.c(2233): [client <IP_ADDRESS>:50592] oidc_proto_authorization_response_code_idtoken: enter
[auth_openidc:debug] [pid 434] src/proto.c(2101): [client <IP_ADDRESS>:50592] oidc_proto_validate_issuer_client_id: iss and/or client_id matched OK: (null), https://login.microsoftonline.com/9188040d-6c67-4c5b-b112-36a304b66dad/v2.\
0, (null), af744108-c699-4f44-ac47-3316b4de71fe
[auth_openidc:debug] [pid 434] src/proto.c(1087): [client <IP_ADDRESS>:50592] oidc_proto_parse_idtoken: enter: id_token header={"typ":"JWT","alg":"RS256","kid":"1LTMzakihiRla_8z2BEJVXeWMqo"}
[auth_openidc:debug] [pid 434] src/util.c(1712): [client <IP_ADDRESS>:50592] oidc_util_create_symmetric_key: key_len=32
[auth_openidc:debug] [pid 434] src/proto.c(1109): [client <IP_ADDRESS>:50592] oidc_proto_parse_idtoken: successfully parsed (and possibly decrypted) JWT with header={"typ":"JWT","alg":"RS256","kid":"1LTMzakihiRla_8z2BEJVXeWMqo"}, a\
nd payload={"ver":"2.0","iss":"https://login.microsoftonline.com/9188040d-6c67-4c5b-b112-36a304b66dad/v2.0","aud":"af744108-c699-4f44-ac47-3316b4de71fe","exp":1479586412,"iat":1479500012,"c_hash":"qBgGnPf4EZ3MlrzZ47Dj-w","nonce":\
"6iwIMXMyHwYI2wRpw45S8EoEaP_8yUMe6NeU4Aj8sJ4","email":"myemail@hotmail.com","sub":"AAAAAAAAAAAAAAAAAAAAAJuWkB7FlTqUTylxwPaTKMw","tid":"9188040d-6c67-4c5b-b112-36a304b66dad"}
[auth_openidc:debug] [pid 434] src/util.c(1712): [client <IP_ADDRESS>:50592] oidc_util_create_symmetric_key: key_len=23
[auth_openidc:debug] [pid 434] src/metadata.c(617): [client <IP_ADDRESS>:50592] oidc_metadata_jwks_get: enter, jwks_uri=https://login.microsoftonline.com/common/discovery/v2.0/keys, refresh=0
[auth_openidc:debug] [pid 434] src/cache/shm.c(158): [client <IP_ADDRESS>:50592] oidc_cache_shm_get: enter, section="jwks", key="https://login.microsoftonline.com/common/discovery/v2.0/keys"
[auth_openidc:debug] [pid 434] src/proto.c(876): [client <IP_ADDRESS>:50592] oidc_proto_get_key_from_jwks: search for kid "1LTMzakihiRla_8z2BEJVXeWMqo" or thumbprint x5t "(null)"

Any help would be appreciated.

Which version of the module are you on?

2.1.0
I'll go update it now and see if the problem gets fixed

I've updated to 2.1.1, and the problem persists. This is running on CentOS 7.2.1511.

thanks; I need to release 2.1.2; I hope you can build from source in the mean time


[SUPPORT] Clustering throwing exception

Hello,
I am trying to run clustering and the job is erroring out without much indication as to why.
Here's the command I am using to run clustering:
spark-submit \
--class org.apache.hudi.utilities.HoodieClusteringJob \
/usr/lib/hudi/hudi-utilities-bundle.jar \
--props s3://path-to-test/clustering.properties \
--mode scheduleAndExecute \
--base-path s3://path-to-test/data/hudi/test/country \
--table-name country --spark-memory 1g

Here's the properties file:
hoodie.clustering.async.enabled=true
hoodie.clustering.async.max.commits=1
hoodie.clustering.plan.strategy.target.file.max.bytes=1073741824
hoodie.clustering.plan.strategy.small.file.limit=629145600
hoodie.clustering.execution.strategy.class=org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy
hoodie.clustering.plan.strategy.sort.columns=enrich_selector_id

And here's the console output of clustering job.
22/02/13 20:00:48 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 1541, ip-172-31-74-236.ec2.internal, executor 3, partition 0, PROCESS_LOCAL, 7927 bytes)
22/02/13 20:00:48 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 1542, ip-172-31-69-5.ec2.internal, executor 2, partition 1, PROCESS_LOCAL, 7922 bytes)
22/02/13 20:00:48 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-172-31-74-236.ec2.internal:46827 (size: 102.2 KB, free: 4.8 GB)
22/02/13 20:00:48 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-172-31-69-5.ec2.internal:43517 (size: 102.2 KB, free: 366.1 MB)
22/02/13 20:00:49 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/hoodie.properties' for reading
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213084348.replacecommit' for reading
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213103928.replacecommit' for reading
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213122909.replacecommit' for reading
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213142348.replacecommit' for reading
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213162102.replacecommit' for reading
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213181256.replacecommit.requested' for reading
22/02/13 20:00:50 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 1541) in 1859 ms on ip-172-31-74-236.ec2.internal (executor 3) (1/2)
22/02/13 20:00:50 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 1542) in 1865 ms on ip-172-31-69-5.ec2.internal (executor 2) (2/2)
22/02/13 20:00:50 INFO YarnScheduler: Removed TaskSet 3.0, whose tasks have all completed, from pool
22/02/13 20:00:50 INFO DAGScheduler: ResultStage 3 (collect at HoodieSparkEngineContext.java:78) finished in 1.891 s
22/02/13 20:00:50 INFO DAGScheduler: Job 3 finished: collect at HoodieSparkEngineContext.java:78, took 1.893633 s
22/02/13 20:00:50 INFO Javalin: Stopping Javalin ...
22/02/13 20:00:50 INFO Javalin: Javalin has stopped
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213084348.replacecommit' for reading
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213103928.replacecommit' for reading
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213122909.replacecommit' for reading
22/02/13 20:00:50 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213142348.replacecommit' for reading
22/02/13 20:00:51 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213162102.replacecommit' for reading
22/02/13 20:00:51 INFO S3NativeFileSystem: Opening 's3://path-to-test/data/hudi/test/country/.hoodie/20220213181256.replacecommit.requested' for reading
22/02/13 20:00:51 ERROR HoodieClusteringJob: Clustering with basePath: s3://path-to-test/data/hudi/test/country/, tableName: country, runningMode: scheduleAndExecute failed
22/02/13 20:00:51 INFO SparkUI: Stopped Spark web UI at http://ip-172-31-66-151.ec2.internal:4041
22/02/13 20:00:51 INFO YarnClientSchedulerBackend: Interrupting monitor thread
22/02/13 20:00:51 INFO YarnClientSchedulerBackend: Shutting down all executors
22/02/13 20:00:51 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
22/02/13 20:00:51 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
22/02/13 20:00:51 INFO YarnClientSchedulerBackend: Stopped
22/02/13 20:00:51 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/02/13 20:00:51 INFO MemoryStore: MemoryStore cleared
22/02/13 20:00:51 INFO BlockManager: BlockManager stopped
22/02/13 20:00:51 INFO BlockManagerMaster: BlockManagerMaster stopped
22/02/13 20:00:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/02/13 20:00:51 INFO SparkContext: Successfully stopped SparkContext
22/02/13 20:00:51 INFO ShutdownHookManager: Shutdown hook called
22/02/13 20:00:51 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-3dd6c522-17c3-48a5-a809-8a0ad56c6da7
22/02/13 20:00:51 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-f841de1d-4101-49bc-8178-7bc79ede16b3

Do any of you guys have any insight as to why this error is happening?

Hi @aznwarmonkey Would you mind to have a try using
Step 1--mode schedule
Step 2 --mode execute --instant-time instant step1 generated

Also could you please provide full log of this test?

@aznwarmonkey : gentle ping.

Hello,
Apologies for the delay. Attached is the full console output of the schedule job.
output.txt

@zhangyue19921010 : Can you followup on this please.

Ack. Sorry, almost forget this issue. Will do more research and response ASAP

Hi @aznwarmonkey Sorry for the delay. I just took a closer look at the log you provided. And found something weird.
Based on your configs, you used --mode scheduleAndExecute. And Hoodie will log some info such as LOG.info("Running Mode: ["xxx"],   LOG.info("Step 1: Do schedule");  no matter it's successful or not. But I couldn't find any hudi-related logs in your provided files.
So that could you please check and tell us what version of hudi you used
Check your log related properties or find some logging by HoodieClusteringJob something like
22/03/14 05:30:51,634 INFO HoodieClusteringJob: Running Mode: [execute]; Do cluster

I am running version 0.9 that is deployed on AWS EMR clusters

do you think you can enable debug logs and share it here. above logs does not have much info to investigate. unfortunately.

will go ahead and close it for now. but feel free to re-open when you have some updates.


xs:annotations not support

The code seems to not support xs:annotations.
Here is an example of failing xsd:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" >
  <xs:simpleType name="XXXX">
    <xs:annotation>
      <xs:documentation>
        <usedBy>Used somewhere over a rainbow</usedBy>
      </xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:token">
      <xs:pattern value=".{1,24}" />
    </xs:restriction>
  </xs:simpleType>
</xs:schema>

Remove the xs:annotations and it works fine :)

Hi @ghilainm  - Thanks for the feedback.  I can see what's wrong.  It's actually the xs:documentation processing.  I have a fix I will publish tomorrow after I complete unit tests.

@andrewbober  Thx a lot! By the way how do you activate debug of the tool?

Xsd2JsonSchema uses the debug library for logging.  Enable debugging output by setting the DEBUG environment variable.
I like to use this setting:
export DEBUG=xsd2jsonschema:BaseConverter,xsd2jsonschema:ParsingState
For all possible logs use this setting (warning - this results in a ton of logging)
export DEBUG=xsd2jsonschema:*

This is resolved in version 0.3.2


revert fix for duplicate log update.

@amaltaro, I reverted the fix, but weird thing is that I don't see duplicate record for all the log.
We can push this now and investigate more for next release. What do you think?

The problematic logs used to be those "Create request ..." and "Assign request...", like this one
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/ReqMgr/Service/Request.py#L560
Logs from both testbed backends (vocms0132 and vocms0731) seem fine to me. The last day I injected/assigned workflows was on July 25 and I see all those log records over there (scanning by eye).
Did you also see a weird behaviour in your VM? Anyhow, if there are no dup records with this patch, I think we can just merge it (even though there is no explanation at all about the dup logs then).

Weird, Now it works without this patch and create a duplicate with this one. I am closing it. But I am not sure why it has cherrypy.error message event though there is no errors.

I found an openstack link giving some explanation about this cherrypy.error thing. However, you can also see it in my first comment/documentation in this PR:
https://github.com/dmwm/WMCore/pull/8691#issue-202559993


Loading .ply mesh created in Colmap and processed in Meshlab renders dark

Habitat-Sim version
main, verified in v0.2.5
🐛 Bug
I am trying to recreate a part of a room in Habitat. For that, I created a mesh using Colmap and Meshlab. Rendering in Meshlab seems to work fine:

But when loading the same mash in habitat-viewer, it is very dark:

It seems like changing the shader option and config does not make any difference. I tried adding lights to it, but it doesn't make a difference either. I furthermore read in the documentation that this is not necessary for scanned environments.
Steps to Reproduce
Steps to reproduce the behavior:

Dowload the mesh file from here.
Run habitat-viewer bug_scene.ply
To verify, download and run Meshlab and import the mesh.

Expected behavior
I expect the environment to be rendered clearly and brightly.
Additional context
I am not sure if I properly configured the scene as specified here, but I tried without success. Habitat test scenes are rendered correctly.
System Info
ENVIRONMENT INFO:
Platform: Linux-5.11.0-41-generic-x86_64-with-glibc2.31
Machine: x86_64
Processor: x86_64
Libc version: glibc 2.31
Mac version:
Python version: 3.9.17
Architecture: 64bit ELF
Win version:
System OS: Linux
Release: 5.11.0-41-generic
Version: #45~20.04.1-Ubuntu SMP Wed Nov 10 10:20:10 UTC 2021
Operational System: linux
GCC version: b'gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0\n'
CMAKE version: b'cmake version 3.14.0\n'
NVIDIA-SMI: b'Wed Aug  2 16:43:14 2023       \n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 470.86       Driver Version: 470.86       CUDA Version: 11.4     |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|                               |                      |               MIG M. |\n|===============================+======================+======================|\n|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |\n| 30%   43C    P8    24W / 250W |   3112MiB / 11019MiB |     26%      Default |\n|                               |                      |                  N/A |\n+-------------------------------+----------------------+----------------------+\n'
Pip packages versions:
b''
Conda packages versions:
b''

I have the same issue, where is your color information stored? I have vertices colors and I suspect habitat looks for textures instead


chore: convert to TypeScript

@alexander-fenster the lint step won't resolve itself until this PR merges sadly.  After this gets a review, it'll need to be hammer merged.

@JustinBeckwith I've just removed the lint step from the list of required tasks to merge.

@alexander-fenster @stephenplusplus  @googleapis/node-team friendly ping 😄

Oh. Totally missed this PR. Let me have a deeper look, this library is used from the autogenerated code and we must be careful here.

I tried running system and samples tests for pubsub here and it looks the generated code still works.