[shopify-app-remix] Managed installation and parallel route loading

Issue summary
Hi,
I'm not sure if this is strictly a bug or if I simply need some advice how to handle the new managed installation flow correctly.
I'm using the default remix template created by the Shopify CLI.
After the client has installed the app through managed installation, Shopify redirects the client to the app on route /app. Since it's the first request after installation, the shopifyApp() starts the Authenticating admin request flow and creates a new session and stores it on the session storage. This process is executed multiple times by each loader() function matching the current route. In this case, there are two loaders being executed for the request /app:

/app/routes/app.tsx
/app/routes/app._index.tsx.

Both loaders run in parallel due to Remix's parallel route loading. Then, after Authenticating admin request has finished the afterAuth() hook is called. This hook would actually be called twice because of each loader. However, the managed installation via the TokenExchangeStrategy dedupes this call so it only runs once:
https://github.com/Shopify/shopify-app-js/blob/7045fe4e3a9c39b858ca8e8337978ee42d496b82/packages/apps/shopify-app-remix/src/server/authenticate/admin/strategies/token-exchange.ts#L159-L172
I use the afterAuth hook to create a database entities (user, shop, etc) in my own system. All other authenticated routes need this information and query this API in their respective loaders. Now the problem is that the managed installation through TokenExchangeStrategy doesn't provide a sequential authentication flow like with the previous AuthCodeFlowStrategy. The authentication in a managed installation seem to be started by multiple loaders in parallel, and the "fastest" loader calls the afterAuth hook. That means there are multiple routes rely on the information being created in the afterAuth hook but it may not be created yet. So I end up with lots of not found errors in loaders because they run before the afterAuth has finished. The errors disappear after a reload.
My first idea was to throw a redirect at the end of afterAuth to force the entire route to reload when the entity was created. But redirect is not exposed on the params of afterAuth and any error thrown inside afterAuth it caught by an outer try-catch anyway.
What is the right way to handle this issue? I think doing API calls inside afterAuth after installation sounds like a normal thing, right?
Before opening this issue, I have:

[X] Upgraded to the latest version of the relevant packages

@shopify/* package and version:
Node version:
Operating system:

[X] Set { logger: { level: LogSeverity.Debug } } in my configuration, when applicable
[X Found a reliable way to reproduce the problem that indicates it's a problem with the package
[X] Looked for similar issues in this repository
[X] Checked that this isn't an issue with a Shopify API

If it is, please create a post in the Shopify community forums or report it to Shopify Partner Support

Expected behavior
After the managed installation, all loaders should wait for the afterAuth hook to be finished before returning.
Actual behavior
The -> app.tsx loader and -> app._index.tsx loader are started in parallel. Both loaders start the Authenticating admin request flow. The first loaders calls the -> afterAuth hook while the second loader finishes with <- app._index.tsx loader before <- afterAuth and <- app.tsx loader have finished.
14:45:48 │ remix         │ [shopify-api/INFO] version 11.0.0, environment Remix
14:45:48 │ remix         │ -> app.tsx loader
14:45:48 │ remix         │ -> app._index.tsx loader
14:45:48 │ remix         │ [shopify-app/INFO] Authenticating admin request
14:45:48 │ remix         │ [shopify-app/INFO] Authenticating admin request
14:45:48 │ remix         │ [shopify-app/INFO] No valid session found
14:45:48 │ remix         │ [shopify-app/INFO] Requesting offline access token
14:45:48 │ remix         │ [shopify-app/INFO] No valid session found
14:45:48 │ remix         │ [shopify-app/INFO] Requesting offline access token
14:45:49 │ remix         │ [shopify-api/INFO] Creating new session | {shop: nebula-etl-test.myshopify.com, isOnline: false}
14:45:49 │ remix         │ [shopify-app/INFO] Running afterAuth hook
14:45:49 │ remix         │ -> afterAuth
14:45:49 │ remix         │ Creating user
14:45:49 │ remix         │ [shopify-api/INFO] Creating new session | {shop: nebula-etl-test.myshopify.com, isOnline: false}
14:45:49 │ remix         │ <- app._index.tsx loader
14:45:49 │ remix         │ No user found
14:45:50 │ remix         │ User created
14:45:50 │ remix         │ [shopify-api/INFO] Registering webhooks | {shop: nebula-etl-test.myshopify.com}
14:45:50 │ remix         │ <- afterAuth
14:45:50 │ remix         │ <- app.tsx loader

Steps to reproduce the problem
Repo: https://github.com/zirkelc/shopify-managed-installation
pnpm shopify app dev

After an app was approved to be installed via the managed installation flow, Shopify redirects the merchant to the app. What is the target URL after in this case? I assume it's the application_url in the shopify.app.toml (same asappUrl on shopifyApp()), right?

Right!

The route (and loader) which is active on this application_url route must authenticate the request and trigger the afterAuth hook to create the user in the database. Then, its should redirect to any other routes that assumes the user already exists. So the key is that only one route (loader) after installation is active.

Right again. What happens here is that the template runs 2 loaders in this case - one in routes/app._index.tsx, and one in routes/app.tsx - the second one is a Remix layout that is run for every route under routes/app.*.tsx.
What I meant was that you don't need to have a layout, so you could for instance delete routes/app.tsx, and the routing would still work, and only the loader in routes/app._index.tsx would be run.
That would probably solve the problem you're having with 2 loaders running for that action. The downside is that you would need to replicate the code from routes/app.tsx in every route in routes/app.*.tsx for authenticate.admin to work properly.
Hope that made it a little clearer!

On a sidenote, what is the purpose of the _index/route.tsx and who calls this route?

This is a landing / marketing page that you can customize however you want, if you want to show it to users to encourage them to try the app / other features that aren't app related. You can reach it by visiting https://<your-app-url> directly from the browser :)

Thanks again for the quick reply, I got it! :) I will try that out and see how it works.
I have two alternative ideas and I'd like to hear your opinion on them:

each loader receives the current session from authenticate.admin.  let's say there is a session (e.g. created in the last 60 seconds) but no existing user yet, that could mean the user is just being created by another loader. to solve this issue, I could just throw a redirect to trigger a reload of the current URL. I would just need to know the current URL / route somehow. Do you know if there is a way to get this information?

when afterAuth() runs, it could throw a Response in order to redirect the user to some kind of post-installation route (e.g. to show a welcome message, intro, etc) or to simply trigger a refresh of the current URL and routes. This is currently not possible because the thrown object from afterAuth is swallowed by this try-catch:
https://github.com/Shopify/shopify-app-js/blob/ed8c30e016d061558dd4321accbcd492524bf569/packages/apps/shopify-app-remix/src/server/authenticate/admin/strategies/token-exchange.ts#L81-L93
Wouldn't it make sense to allow the Response to bubble up so we have more flexibility of what happens post-auth? I know this try-catch block is embedded in the authenticate that is supposed to return a session, so this feature would require some kind of re-write to make it work. On the other side, it looks like authenticate is being called by the authStrategyFactory here:
https://github.com/Shopify/shopify-app-js/blob/ed8c30e016d061558dd4321accbcd492524bf569/packages/apps/shopify-app-remix/src/server/authenticate/admin/authenticate.ts#L137-L155
This function authStrategyFactory.authenticateAdmin() lets a possible Error or Response  bubble up, so if authenticate would return a session and/or a Response it could just be re-thrown here.

That would be great! Sure, I'll be happy to work on this and submit a PR in the next few days.
Thanks for your support!

Hi @minhduc17801 your code looks fine, did you test it locally? Are you using managed installation oder OAuth?

Hi @zirkelc , thank you for your response, and I also apologize for my delay. I am using the default managed installation from the Remix template. I have also added logs and made some adjustments to the code in my previous comment; you can review it. I have tested it locally, and here are some results from my tests.

If I throw a redirect response in afterAuth, it seems that the response lacks headers or queries for authentication, and I keep getting redirected to /auth/login

2.If I don't return a response in afterAuth, as you mentioned above, both loaders will be called, and both authenticate.admin will run, but only one will wait for afterAuth. The other will skip authentication and continue executing the code below, encountering my conditional logic and throwing a 500 server error. However, it seems that the app.tsx loader is called again every time afterAuth finishes. So, if afterAuth is triggered by app._index.tsx, my application works fine, but if it’s triggered by app.tsx, it will encounter a 500 server error.
-afterAuth run in app.tsx loader, app crash with 500 server error

-afterAuth run in app._index.tsx loader, app works fine.

I think the simplest solution would be to make redirect response from afterAuth pass through authenticate and potentially trigger a reload of app.tsx and app._index.tsx. However, due to my lack of experience with both coding and Shopify, I haven't been able to implement it successfully. I would really appreciate your help with this. Thank you!


constructors to create wasm contract call json payloads

Description
Moves verifyMembership and verifyNonMembership types into their respective constructor functions for cleaner (1q line) payload generation.
closes: #4044
Commit Message / Changelog Entry
type: commit message

see the guidelines for commit messages. (view raw markdown for examples)

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

[x] Targeted PR against correct branch (see CONTRIBUTING.md).
[x] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[ ] Code follows the module structure standards and Go style guide.
[ ] Wrote unit and integration tests.
[ ] Updated relevant documentation (docs/) or specification (x/<module>/spec/).
[ ] Added relevant godoc comments.
[ ] Provide a commit message to be used for the changelog entry in the PR description for review.
[x] Re-reviewed Files changed in the Github PR explorer.
[ ] Review Codecov Report in the comment section below once CI passes.

Thank you for picking up a bunch of the 08-wasm issues, @Reecepbcups. :) This one in particular will require some changes, since we received a recommendation by Ethan Frey to organise the payload types a bit differently. I will open an issue and post it here when it's ready.

np, sounds good

These structs will be replaced with two higher level structures for Sudo and Query contract calls with each containing a pointer to a struct to represent each action, note the omit_empty is required for json serialization. These structures play well with having the msg enums defined in the rust code for match expressions etc:
// encapsulates read-only contract calls
type QueryPayload struct {
  MsgStatus *StatusPayload `json:"msg_status,omitempty"`
  MsgVerifyMembership *VerifyMembershipPayload`json:"msg_verify_membership,omitempty"`
  // ..etc
}

// anything which writes to store
type SudoPayload struct {
  MsgInitialize *InitializePayload `json:"msg_initialize,omitempty"`
  MsgUpdateState *UpdateStatePayload `json:"msg_update_state,omitempty"`
  // ..etc
}

Hi @Reecepbcups, this PR implements the changes mentioned by @damiannolan above. Take a look and let us know if you would like to continue the work of your PR. If you're busy, no problem, we can take over :) Also, it might be better to use the branch of #4133 to implement the changes on top of it.

@crodriguezvega Hey sorry for the late reply here. I will be out of town the next 7 - 10ish days so I will not be able to take this on.
I will get to more cleanup PRs once I get back if this has not been handled by then. (+ older issues I find)


Enable storage account infrastructure encryption

Resolves #4001
What is being addressed
Enable storage account infrastructure encryption.
Since storage account infrastructure encryption can only be turned on when a storage account is first created (unless you allow terraform to destroy the storage account and recreate it), this change is only applied on first deployment of a TRE (or a template component).  Storage accounts that exist within an existing TRE deployment will not be affected.
The terraform pattern used to acheive this behaviour is as follows:
resource "azurerm_storage_account" "stg" {

  ... existing attributes ...

  # changing this value is destructive, hence attribute is in lifecycle.ignore_changes block below
  infrastructure_encryption_enabled = true

  lifecycle { ignore_changes = [infrastructure_encryption_enabled ] }
}

@tamirkamara yes, they are compatible

Hey @tamirkamara do you have any more outstanding thoughts or feedback with your review?  Thanks

/test 019a1b4bde593168e26824397d8cd4978cffbe04

@tim-allen-ck this ok to merge now? thanks

/test 2200ac98751cef790f79ae31f8d2a6519d72e860


Updated to show existing driver

Resolved: now it will display that existing drivers is exists.

Resolved: now it will display that existing drivers is exists.

in header section, manage status and logout not working

@mansiruhil13 ,   I went through and checked in file changed section ,  I realised there is no error from my side.
you can also go through and let me know
pls review this file:
url:   https://github.com/mansiruhil13/Bobble-AI/pull/100/files

@mansiruhil13 ,   hey Mansi
i only just raised a issue for, duplication of driver names .... so for solving that i just added that features only..
i am not able to understand what you are trying to say

after being made change by me, that is why it is not working or clickable??
or something else??


Functionality storing backup in AWS Cloud Storage

Simply this idea is that, instead of storing backup locally we can store it for example in AWS S3, EBS, EFS or Glacier. Implementation is not difficult but might have very big value. Later it can be expanded to another cloud providers storage service and even to google drive :)

:) That can be useful in fact. We can do it for next release.

Hey. Are there any plans to implement storing backups to S3? Or is there any way to handle it by hand?


Lowered iOS deployment target to 8.0

The iOS framework still compiles, so I imagine there was no reason for this to be 8.3 :)

Does it go back any further than this?

Since it's a dynamic framework, iOS 8 is required to be able to link it on iOS, so this is the lowest it can go :)

Cool :sparkles:.  Last thing:
Can you alter the README to mention the new target bounds (down near the bottom there).

Good point, done!

Once Travis gives us the green :boat:
Do you need a release for this ASAP or are you willing to wait?

Nah, I'm building with Carthage from a fork anyway, so I'm in no rush!
I just figured I'd send this one commit :)

Well, thank you so much for it then.
:boat:


Multiline equation no longer rendering after upgrade (pandoc 2.1 ->2.2)

Brief Summary
I began building my site from a fresh machine. In moving to a newer version of nbsphinx and its dependencies, my mulit-line latex equations broke.
I can confirm that downgrading pandoc 2.2.3 down to pandoc 2.1.3 restores the proper behavior.
Example of What Broke
The following markdown cell:
\begin{equation}
\left(
\begin{array}{*{6}{X}}
  y[2] & y[4] & y[0] & y[4] & y[4] & y[4]
\end{array}
\right)
%
\rightarrow
\left(
\begin{array}{*{6}{X}}
  -2 & -4 & 0 & -4 & -4 & -4
\end{array}
\right)
\end{equation}

used to render, via nbsphinx, as:

It now renders as:

System Information

OS: Windows
Python 3.7
nbsphinx 0.4.2
sphinx 1.7.9
pandoc 2.2.3
nbconvert 5.3.1

As I mentioned above. Holding all of these things constant and downgrading pandoc to 2.1.3 resolves this issue. I tried 2.2.0 and it failed as well - it seems like the breaking change was introduced here.

Thanks for the report!
I guess this is the same problem as #279, so the same work-around should work.
If you have an idea how to fix this properly, please let me know!

I will look into this. Should I look into #257 to understand what code the solution should be touching?

Yes, #257 is the culprit.
The problem is that pandoc changed its behavior regarding using RawBlock and RawInline at some point.
I was trying to find a way to handle both old and new pandoc versions, but obviously it didn't work that well.
I hope you have more success!

I'm closing this, given that you raise a runtime warning for being on pandoc >= 2.0.
Hope all is well!


Updating to v0.17.0

v0.17.0 was released 10 days ago.

Hi, please get the tests passing and we can review.


Miscellaneous layout changes

Description
Monster PR that implement a number of proposed and inter-dependent layout changes (and some other bits and bobs):

Full-bleed layout is now opt-in. By default, <body> will have some padding and margin around it so that its contents don't touch the edges of the viewport and also have a sensible max-width on large viewports. This is, of course, a breaking change.

To restore the "old", full-bleed layout behaviour you can add the .grav-o-full-bleed class to <body>.
The old .grav-o-container class has been renamed to grav-o-full-bleed__content to better illustrate its relationship with grav-o-full-bleed.
Since we'd like page header and footer components to be flush with the top / bottom of the viewport, they have negative margins to counteract the <body> default padding. These get switched off when the grav-o-full-bleed class is used, so that header and footer will be correctly positioned in both scenarios. Similarly, their inner <div> autmatically adopt container behaviour when they appear within a full-bleed page.
Closes #333
This change also removes the body > * exception to the lobotomised owl rule and thus solves the problem of a missing margin or padding above the page footer. Closes #182

Sticky footer is configurable and more robust. By default, the sticky footer effect still gets applied to <body>, however you now have the ability to disable that at build time via a SASS variable and also via a new .grav-o-undo-sticky-footer class for ad-hoc overrides.

To support SPA cases, where the page footer is not a direct descendent of <body> there is now also a .grav-o-sticky-footer class that can switch on this behaviour for arbitrary container elements.
The implementation of the sticky footer has been changed and should have less side-effects. It also no longer relies on the presence of a <main> element.
The implementation has been wrapped up a SASS mixin, so it can be easily re-used.
Closes #133

Smart borders. I made a few mixins for adding borders to containers. They have a "smart" option that makes them disappear if you apply a color scheme class to the same element. This can be useful for boxing in sections (e.g. a sidebar or similar). When that section doesn't have an explicit color scheme (and so, shares the same background color as its surroundings) it gets a visible border. But, if you apply a color scheme to it - which will change its background color - then the border disappears. The assumption is that the difference in background color is sufficient to visibly differentiate that container from its surroundings.

I was motivated to add this by the page footer. So far, unless you apply a color scheme to it (like we do on the Buildit website, so make it dark gray), there's no visible separation between it and the page content above. I've therefore given it a "smart" top border.
This is obviously a subjective design choice, so if you disagree, I'm happy to ditch it. However, even if we don't use it on the footer, I believe the smart border facility will be useful. I therefore suggest we keep the mixin in any case.

Removed .grav-o-section class. All it did was apply a larger than usual margin-top. Probably should have killed it a long time ago. It's barely used and in light of all the other code & design changes redundant. If you really need a bespoke, bigger gap somewhere you can probably achieve it just as well with the recently added padding utility classes.
Adds margin utility classes. This is useful for the homepage, where - as an exception - we want to remove the standard margin top that gets applied to <main> so that the hero is flush with the page header. Since we already have a full suite of padding util classes, I thought I may as well add all the equivalent margin ones too.
Standardises gap after page header. The page intro component, used on story pages, previously had an extra large margin-top (which, strictly speaking, didn't follow Gravity's own layout conventions). On any pages where that component was not used, the gap above the page header's first sibling was the standard vertical gap, which itself varied, depending on that element's font size. When navigating between different pages this can cause the start of the page content to seamingly "jump" up and down. This change therefore removes the outer margin on the page-intro component (so that component now adheres to our usual conventions) and now sets an equivalent (and font-size indenpendent) top margin to any element that follows the page header.

To support testing the above changes, I have added a few page templates into the pattern library: "Simple HTML" (simulates the raw output you'd get from rendering a markdown file), "Header and footer" (simple HTML that has been topped and tailed by Gravity's page header and footer components), "Nested Root" (same as "Header and footer", but with everything wrapped in a top-level <div> so that they are not direct descendents of <body>.
I had previously been prototyping these in various CodePens, but it was getting a bit cumbersome to keep them all in sync, so doing it within our pattern library was easier. I've tried to keep each one as a seprate commit, so if we decide to ditch any of these ideas, we should be able to rebase and cherry-pick just the ones we want to keep.

@james-nash When i'm looking at this branch locally on pattern lab, i'm not getting any padding on the body of Templates and Pages. Do I need to run something? Or is this a bug?

@dw-buildit Hmm... it should _just work_TM
Maybe the rebase broke something. Let me try and, once it's all clean, I'll push it.

Ok, just pushed the renamed "undo sticky footer" class: ..grav-o-sticky-footer--unstick.

:tada: This PR is included in version 3.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[TFLite Export] Add Unit Test

In this PR
Add Unit Test case For Test Fused Op in Tensorflow lite export
Test Model consist like below
Input -> Conv2D -> Batch Norm -> ReLU -> Flatten

make this model with nntrainer and export to tensorflow lite
set input {1,3,4,4} Dim -> in Tensorflow lite native make same input with 3,4,4 and transpose this input

latency
(nntrainer exported has transpose layer)
10000 forward tflite native (sec)              : 0.0069193975830078125
10000 forward nntrainter exported (sec) : 0.0070416927337646484

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2204. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

:octocat: cibot: @DonghakPark, The last line of a text file must have a newline character. Please append a new line at the end of the line in test/unittest/compiler/unittest_interpreter.cpp.


EBS volume creation logging is missing data

   Recipe: aws_test::ebs_volume
     * aws_ebs_volume[db_ebs_volume] action create
       - create a volume with id= size=50 availability_zone= and update the node data with created volume's id

It appears that we're not populating the new_resource. availability_zone properly.  We need to investigate that and correct the provider

Fixed


Megamenu subtitle

Forgot the export

While you're in there, want to add the missing Table exports? 🤗

@mercedesb added it in https://github.com/lob/ui-components/pull/74 ! Will merge that one now


Feature/#185 Update font family Chalk_3 to title BID

Issue Addressed

Updated the font family of the bid-text class
Updated ternary if syntax

Proposed Changes and Benefits

Resolve issue #185

Screenshots

Notes to Reviewers
I change syntax of HTML element of BID-TEXT to dont repeat code.
Please, send to hacktoberfets this pull request

Checklist

[x] I have thoroughly read and understand The Contribution Guidelines.
[x] The title of this PR follows the keyword: brief description of change format, using one of the following keywords:

Feature - adds new or amends existing user-facing behavior.
Chore - changes that have no user-facing value, refactors, dependency bumps, etc.
Fix - bug fixes.
Docs - fixing typos and adding new content.

[x] I have listed my proposed changes and their benefits.
[x] I have linked the corresponding issue in the Issues section.

@XanderRubio Can you review my pull request? I need add my profile in contributors?

@XanderRubio Can you review my pull request? I need add my profile in contributors?

Yes, @DiguyaDeveloper, please go ahead and add what you want to do before you die with text and image by following issue #5, and in the process, you will add yourself to the contributors list. Great work as it is much appreciated! Have a great day, Diego!

Xander


search_json view displays incorrect URLs

The view should only be showing the URLs related to the given project. There is validation on the passed project slug, raising a 404 if the slug does not match a project, and yet if it does match, all the URLs for every project are returned. Just need to add a simple filter on the query to limit the returned URLs to those related to the project.

Or perhaps the URL itself should be changed to not include the project slug if a universal list of all submitted URLs is actually the desired behavior.

I think it is meant to return the URLs for that specific project only.

Alright, I'll add a filter to the query found here.


Build pdf component for win-print API support.

@hokein i am wroking on supporting ppapi, glad that you shipped ppapi headers in this patch. But there are a few ppapi components that needs to be built (ppai_*), shall i base it on this PR or you will be adding it ?

@deepak1556  Which ppapi components will you need? The pdf component already has dependency on ppapi modules (ppapi_cpp_object and ppapi_internal_module).
It may require a long time to get win-print patch merged. If you need it eariler, you can directly ship ppapi without basing on this PR, and I will update this PR later.

Ah cool just needed ppapi_internal , didnt check pdf.gyp . But we are yet to confirm whether to ship pdfium as an optional shared library right ? I will just work on ppapi for now without basing on this PR. Thanks!

:sparkles:


R18 problem

Expected Behavior
Move file to folder
Create .nfo, covers
Current Behavior
It creates the folder but it doesn't move the file or create .nfo or cover. Comes up with Error: Could not find part of path
Steps to Reproduce (for bugs)
Your Environment

Module version used:
Operating System and PowerShell version:
Win 10 Pro
2.55 GUI
Powershell: 7.1.3

Are you using the Windows GUI javinizer -opengui or docker?

Can you post your settings file? I'm unable to reproduce on my end.

Yep Javinizer -OpenGUI
Do I paste the jvsetting.json here?

Can you attach the file instead? I think some of the fields in your paste got deleted due to github's markdown.

jvSettings.txt
Hope this works.

I think I found your issue.
Remove the extra space at the end of your setting and try sorting again.
### Before
"sort.format.folder": "<ID> <TITLE><ACTORS> "  <-- Extra space here after <ACTORS>

### After
"sort.format.folder": "<ID> <TITLE><ACTORS>"

So that's what it was. All good here now thank you.


Subscription count reduction

Would like to see further clarification on what happens when the count is reduced but not expiring.  Example.  300 licenses is updated to 250 through renewal.   What happens if there are 300 licenses assigned.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e5e656ac-5f86-6f46-55fc-5ea2cef0b51c
Version Independent ID: 74553c8e-e34c-8f5b-c2c2-95100c357d74
Content: What happens to my data and access when my subscription ends?
Content Source: microsoft-365/commerce/subscriptions/what-if-my-subscription-expires.md
Service: o365-administration
GitHub Login: @cmcatee-MSFT
Microsoft Alias: cmcatee

@USAGonzo00 If you have 300 licenses and try to reduce to 250, that action is blocked and you will be told to unassign licenses from users until you're at or below 250.


Add livepeer.org UI tracking

Info gets recorded when user visits a page on livepeer.org and what page(s)


added sinon and chai as dev dependencies; added some unit tests

This PR not exactly solves #128, but it's a start.
Added chai dependency because with it tests look a bit cleaner.
Also added sinon which is also very useful library for spying and mocking everything. I haven't exactly used it in the PR, it's an investment for the future.

Cool stuff, thanks!
Any reason why you added sinon and chai instead of using the built-in Jasmine utils?

I don't like Jasmine and I'm looking for alternatives, using chai and sinon is perfectly fine to me because they make it easier the transition to a different framework. I just wanted to know if there was any particular reason. Thanks!


[Candidate 1v1 Natural Threats ZH v4] Wind audio crackles a bit when scrolling

Issue Description
Wind audio crackles a bit when scrolling over map.
Expected behavior
Wind audio does not crackle when scrolling over map.
Videos
https://github.com/TheSuperHackers/GeneralsRankedMaps/assets/4720891/b06790a6-fcb3-45de-a3d5-20f2ffaf0bbf

Perhaps can be improved by giving wind audio emitters a short delay before they kick in.


TypeError: configure_model() missing 1 required positional argument: 'cpu_offload_'

TypeError: configure_model() missing 1 required positional argument: 'cpu_offload_'
start gradio
I will encounter this problem regardless of whether I pass this parameter or not
Here are some startup parameters I have tried
python3 ./demos/gradio_ui.py --model_dir weights/ --cpu_offload true
python3 ./demos/gradio_ui.py --model_dir weights/ --cpu_offload false
python3 ./demos/gradio_ui.py --model_dir weights/

Hi @ysxk, this is a bug introduced by our LoRA trainer. I'm working on a fix which should be out soon. In the meantime, ./demos/cli.py works. Thanks for reporting!

@ysxk gradio_ui.py is now fixed. You can use it with:
python3 ./demos/gradio_ui.py --model_dir weights/ --cpu_offload

or
python3 ./demos/gradio_ui.py --model_dir weights/

thanks


Try to find FQDN of host

If resolving the local FQDN fails, fall back to previous method of using MachineName.
Fixes #15.

@pall-valmundsson - thanks for the PR!


Affinity Management

Is it possible to manage the affinity by groups of nodes? and manage by tenant? ( taints, region, zone,...)

Hi! Yes, but please provide a more complete use-case. Example:
"Like a DevOps engineer I want to put a tenants on dedicated nodes of management cluster for the better isolation"
...

:-)
The idea:
3 datacenters, multiples hosts per datacenter, stretch cluster between datacenters.
When a user create a VM, DB, etc.. He should be able to select his datacenters, and from storage point of view, where are replication for High Availability beteween datacenter? in the DC or splited between DC?.. etc...
Let me know if I'm not clear enough.

So you may run into different options.
Like:

I don't care at all (so system will place the DB on the first appropriate node in any DC)
choose from one of DC: DC1, DC2, DC3
use local volume vs replicated volume (this option should be configurable even we are using particular DC).

Which ones do you need?
The idea to have cluster of DB stretched between DC is reasonable as it is the only option to have HA. But it is not the only use-case

if we have a VM in DC1 with replication storage. It should be interesting if we can select " Replication between DCs" or "replication between nodes in my DC".
In the first case, the VM can restart on a second DC (in case of DC1 failure)
In the second case, the VM can restart only on a node of the DC1, and no HA in case of DC1 failure.
and the mixed version: I have 2 replicas of my VM on the same DC, and a third replica on a external DC (2 or 3).
The final idea is to purpose to our users to select the level of HA.
What do you think?

we don't use VM for databases, queues, storage... tenant k8s service is working on top of VMs and you can still order VMs on your own and install something into it.


Make File Transfer compatible with electrons with iterable outputs

Environment

Covalent version: 0.177.0
Python version: 3.8.13
Operating system: mac OS

What is happening?
When transferring files and returning the results as a list / tuple / dict, we get an error saying:
[2022-09-06 19:25:18,248] [ERROR] execution.py: Line 372 in _run_task: Run task exception
Traceback (most recent call last):
  File "/Users/faiyaz/opt/anaconda3/envs/dnn-tutorial/lib/python3.8/site-packages/covalent_dispatcher/_core/execution.py", line 345, in _run_task
    output, stdout, stderr = await execute_callable()
  File "/Users/faiyaz/opt/anaconda3/envs/dnn-tutorial/lib/python3.8/site-packages/covalent/executor/base.py", line 572, in execute
    result = await self.run(function, args, kwargs, task_metadata)
  File "/Users/faiyaz/opt/anaconda3/envs/dnn-tutorial/lib/python3.8/site-packages/covalent/executor/executor_plugins/dask.py", line 112, in run
    result, worker_stdout, worker_stderr = await dask_client.gather(future)
  File "/Users/faiyaz/opt/anaconda3/envs/dnn-tutorial/lib/python3.8/site-packages/distributed/client.py", line 2037, in _gather
    raise exception.with_traceback(traceback)
  File "/Users/faiyaz/opt/anaconda3/envs/dnn-tutorial/lib/python3.8/site-packages/covalent/executor/executor_plugins/dask.py", line 60, in dask_wrapper
    output = fn(*args, **kwargs)
  File "/Users/faiyaz/opt/anaconda3/envs/dnn-tutorial/lib/python3.8/site-packages/covalent/executor/base.py", line 95, in wrapper_fn
    output = fn(*new_args, **new_kwargs)
TypeError: get_item() got an unexpected keyword argument 'files'

The UI shows the following:

How can we reproduce the issue?
Add a train.csv and test.csv in the login node in beehive under your account. Accordingly replace the filepaths in the example workflow shown below.
remote_train_file = ct.fs.File('/federation/faiyaz/source_data/train.csv', is_remote=True)
remote_test_file = ct.fs.File('/federation/faiyaz/source_data/test.csv', is_remote=True)
local_train_file = ct.fs.File('/Users/faiyaz/Code/datasets/train.csv')
local_test_file = ct.fs.File('/Users/faiyaz/Code/datasets/test.csv')

strategy = ct.fs.strategies.Rsync(
    user='faiyaz',
    host='beehive.agnostiq.ai',
    private_key_path='/Users/faiyaz/.ssh/id_ed25519'
)

@ct.electron(
    files=[
        ct.fs.FileTransfer(remote_train_file, local_train_file, strategy=strategy, order=ct.fs.Order.BEFORE),
        ct.fs.FileTransfer(remote_test_file, local_test_file, strategy=strategy, order=ct.fs.Order.BEFORE),
    ]
)
def fetch_and_validate_data(files=[]) -> List[str]:
    _, local_train = files[0]
    _, local_test = files[1]
    train = pd.read_csv(local_train, parse_dates=['date'])
    test = pd.read_csv(local_test, parse_dates=['date'])
    return train, test

@ct.lattice
def workflow():
    train, test = fetch_and_validate_data()

dispatch_id = ct.dispatch(workflow)()
res = ct.get_result(dispatch_id, wait=True)
print(res)

What should happen?
The iterables should be unpacked without any errors. Also, note that for a non-iterable return, there is no issue.
Any suggestions?
It's possible that during L218 of electron.py, particularly in def __iter__() when consructing the electron we copy the metadata self.metadata.copy() which would also include the callbefore & callafter deps with return value keys defined, which would in turn try to inject the files kwarg. We may want to disable this for internally generated electrons or figure out a way to make the injected keywords optional.

It's possible that during L218 of electron.py, particularly in def iter() when consructing the electron we copy the metadata self.metadata.copy() which would also include the callbefore & callafter deps with return value keys defined, which would in turn try to inject the files kwarg. We may want to disable this for internally generated electrons or figure out a way to make the injected keywords optional.

PR #965 might be relevant here.


bugfix: VERILOG_INCLUDE_DIRS not a list of Paths

Yosys.JsonHeader, Yosys.Synthesis

Fixed VERILOG_INCLUDE_DIRS being a list of strings instead of a list of
Paths.

Resolves #615

@kareefardi It is a duplicate. It's declared twice. I just noticed.


go to definition not working

Issue Type: Bug
When I'm trying to get the definition of any method from called place. no definition found is coming.
kindly give me any solution.
VS Code version: Code 1.39.2 (6ab598523be7a800d7f3eb4d92d7ab9a66069390, 2019-10-15T15:33:40.634Z)
OS version: Linux x64 4.4.0-142-generic

System Info

Item
Value

CPUs
Intel(R) Core(TM) i3-7100 CPU @ 3.90GHz (4 x 897)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwareoop_rasterization: disabled_offprotected_video_decode: unavailable_offrasterization: disabled_softwareskia_deferred_display_list: disabled_offskia_renderer: disabled_offsurface_synchronization: enabled_onvideo_decode: unavailable_offviz_display_compositor: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
1, 1, 1

Memory (System)
7.71GB (0.71GB free)

Process Argv
--unity-launch

Screen Reader
no

VM
0%

Extensions (6)

Extension
Author (truncated)
Version

vscode-eslint
dba
1.9.1

php-debug
fel
1.13.0

php-intellisense
fel
2.3.13

php-pack
fel
1.0.2

sublime-keybindings
ms-
4.0.3

vetur
oct
0.22.6

This is not very informative and doesn't allow me to reproduce anything. Go to definition works for me, so please add more information, like in what language, in what file, at what position you are trying this etc

I confirm @shamsulaman786's problem. I have it in Visual Studio Code 1.38.1, working with C#. It is actually hard to reproduce, because it sometimes works (on simple cases like 2 new C# classes and one of them references the other) and sometimes not.

I can reproduce it with the following code:
using System;
using Microsoft.AspNetCore.Hosting;

namespace MyCompany.Tests
{
    public class TestServerBuilder
    {
        private IWebHostBuilder hostBuilder;

        public TestServerBuilder()
        {
            hostBuilder = new WebHostBuilder();
        }
    }
}

When I right-click on IWebHostBuilder and then choose "Go to definition", then there is the caption: "No definition found for 'IWebHostBuilder'".


[feature request] link zsh statically to make it portable

To reproduce:

Install zsh-portable.

mkdir /tmp/zsh-portable
cd /tmp/zsh-portable
curl -fsSL https://github.com/xxh/zsh-portable/raw/master/result/zsh-portable-Linux-x86_64.tar.gz | tar -pxz

Type ldd zsh.

Expected: statically linked binary.
Actual: dynamically linked binary with a dependency on glibc.

This makes the binary usable only on systems with a compatible version of glibc. On systems without glibc (e.g., Alpine or Android), zsh-portable doesn't work.
Would it be possible to create a statically-linked binary, perhaps linked against musl?

🔴 CURRENT SOLUTION: https://github.com/xxh/zsh-portable/issues/2#issuecomment-599216563

Good questions)

About glibc. Because it has backward compatibility there is a common solution: just building the binary on most earlier version of linux. For example AppImage for python uses glibc 2.17 from manylinux 2010 and it will work on any linux with glibc older than 2010. For Zsh building I used here Ubuntu 14 has glibc 2.19 and looks enough.

About musl. I've tried to build zsh on Alpine. It could be achieved by replacing Ubuntu to Alpine and replace ubuntu apt to alpine apk commands. As result I got zsh with musl just for interest. No commits about it.

I mentioned musl not because I want the dependency on musl to be replaced by a dependency on glibc but because with musl it's easier to create a statically linked binary. With glibc it's generally impossible (although in some cases you can do it).

I haven't experience with musl and if you could create musl version PR is very welcome. I tried once and it was looking as working way.

I'm not interested in a binary with a dependency on musl. I'm asking if it's possible to create a statically linked binary to make it portable.

Could you please give an example of OS where it's not working? I could be use it for testing.

Alpine Linux

If you can build a binary that works on Alpine and Ubuntu, it'll work pretty much on any Linux.

Added alpine version:

Dockerfile-alpine
build-alpine.sh
zsh-portable-alpine-Linux-x86_64.tar.gz

And in Ubuntu it woks if add musl to lib:
# Ubuntu
mkdir /tmp/zsh-portable
cd /tmp/zsh-portable
curl -fsSL https://github.com/xxh/zsh-portable/raw/master/result/zsh-portable-alpine-Linux-x86_64.tar.gz | tar -pxz
cp libc.musl-x86_64.so.1 /lib/ld-musl-x86_64.so.1
./zsh.sh

The question is: how to avoid cp libc.musl-x86_64.so.1 /lib/ld-musl-x86_64.so.1?

The question is: how to avoid cp libc.musl-x86_64.so.1 /lib/ld-musl-x86_64.so.1?

You need to find a way to link zsh binary statically.
You use powerlevel10k, right? In the powerlevel10k directory there is gitstatus/bin/gitstatusd-linux-x86_64-static. You can copy this binary to Ubuntu or Alpine and run it. It'll just work. This is because it's statically linked. If you run ldd on it, it'll tell you as much. Compare it with the result of ldd on gitstatusd-linux-x86_64.

I've did the trick. A bit hacky but it works. Try:
mkdir /tmp/zsh-portable
cd /tmp/zsh-portable
curl -fsSL https://github.com/xxh/zsh-portable/raw/master/result/zsh-portable-musl-Linux-x86_64.tar.gz | tar -pxz
./zsh.sh

It works for me on ubuntu and alpine like a charm )

If it works for you as expected I'm going to use this for xxh.

Have you tried building a statically linked zsh binary?

Sad but I'm not so experienced in building static binaries. I've tried to rebuild with --enable-libc-musl (help says compile with musl as the C library) and the zsh binary has grown like it includes musl. But no changes while trying execution.
As result I just replace /lib/musl to .////musl and it works.

As result I just replace /lib/musl to .////musl and it works.

Yeah, I saw that. This isn't what I would like to have. Shipping dynamic libraries with executables can work OK if you use chroot or similar. Otherwise it's prone to clashes.

I've tried to rebuild with --enable-libc-musl

This doesn't seem like the right flag.
When I get some free time, I'll see if it's not too hard to build a static binary.

FYI: I've used zsh-portable without xxh: https://www.reddit.com/r/zsh/comments/fiq9w2/bring_zsh_with_ohmyzsh_wherever_you_go_through/fkka097/. Works very well.

Took a shot at it. The build process is very hacky but the result is robust and works very well. See https://github.com/romkatv/zsh-bin/blob/master/build-zsh-5.8-static. If you try this, please let me know whether it works for you.
There are a couple things that could be improved.

There is no way to create global and zshenv and zshrc. This isn't a big deal because the binary is supposed to be installed in a home directory, so global rcs are unnecessary. Nevertheless, it would be nice to support them. It's fairly easy to do but I just ran out of time.
I've built man pages for Zsh and included them in the archive, but they don't get picked up. I haven't looked at what it would take to make man zsh work.

xxh deleted a comment from romkatv yesterday

I really don't like when things I post publicly get deleted.

Feel free to close this issue as I no longer need this feature.
If you want, you are welcome to use any part of https://github.com/romkatv/zsh-bin. No attribution necessary.

Thank you! I'm reviewing it now!

No attribution necessary.

Hah, because:

magic marker, trailing p10k totally accidental

We all want a bit of attention :)

In March of 2020 an announcement was posted on /r/zsh. It mentioned that "xxh uses the portable version of Zsh". I thought it would be cool to migrate my ssh.bash script to Zsh and install the portable version of Zsh on the remote machine if there isn't one already installed (this is basically what xxh does).

Thank you! It's appreciated.

My pleasure :wink:

Took a shot at it. The build process is very hacky but the result is robust and works very well. See https://github.com/romkatv/zsh-bin. If you try this, please let me know whether it works for you.

It completely works. Thanks! I'm going to create xxh-shell-zsh-<romkasuper/p10k/zsh-bin please choose> and testing it with xxh.

There is no way to create global and zshenv and zshrc.

I read this as feature not bug )

It completely works.

Thanks for the confirmation!

I'm going to create xxh-shell-zsh-<romkasuper/p10k/zsh-bin please choose> and testing it with xxh.

🤣

Done:
pip install --upgrade xxh-xxh # to xxh 0.6.1
xxhp install xxh-shell-zsh-romkatv xxh-plugin-zsh-powerlevel10k
xxh myhost +s xxh-shell-zsh-romkatv +if +q


fix(RELEASE-1178): fetching of Dockerfile oci artifact

The pull spec is composed from the digest,
but we need to replace : with -.

/test release-pipelines-e2e-suite


Support multiple providers per chainid

We will soon have multiple enterprise partners per chain id (Infura/Coinbase cloud). Naively let's just round-robin between those providers.

If this feature is added, can you please add a "provider" column in Athena?


MudNavGroup Expanded attribute value not respected

Things to check

[X] I have searched the existing issues for this bug
[X] To rule out a caching problem I made sure the bug also happens in an incognito tab

Bug type
Component
Component name
MudNavGroup
What happened?
The expanded state changes regardless of the value provided by the 'Expanded' attribute.  When the provided bool is true, the menu is still collapsed.
Attempt was to prevent the menu collapsing on click (I'd like it to persist in an open state).  The following was attempted:
<MudNavGroup Title="Builder" HideExpandIcon="true" Expanded="isExpanded"  ExpandedChanged="KeepExpanded">
...
</MudNavGroup>

bool isExpanded = true;

private void KeepExpanded(bool _val)
{
    isExpanded = true;
    StateHasChanged();
}

Expected behavior
I was expecting the isExpanded value to be respected after the StateHasChanged call and the menu to remain in an expanded state.
Reproduction link
https://try.mudblazor.com/snippet/QOmePwPxxvXMPBHl
Reproduction steps

Create a MudNavGroup component.
Create private bool isExpanded.
Create function to catch ExpandChange - set isExpanded to true.
Provide attribute Expanded="isExpanded".
Provide attribute ExpandedChanged to function in step 3.
Set Breakpoint on function created in step 3.
Run - Clicking the NavGroup will hit the breakpoint, isExpanded is set to true but the menu state still changes.

Relevant log output
No response
Version (bug)
7.15.0
Version (working)
No response
What browsers are you seeing the problem on?
Firefox, Chrome, Edge, Other
On which operating systems are you experiencing the issue?
Windows
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Currently we do not support intercepting two way binding values.
I would just recommend to disable the click on the expansion:
https://try.mudblazor.com/snippet/caQSlcbdHiOHDPTp


Running insert_worm.py fails

This is my full stack trace when I run the script on both the alpha0.5 and alpha0.5-slarson branches.
The error occurs in the step update_neurons_and_muscles_with_lineage_and_descriptions().
└─▪ python insert_worm.py
uploaded muscles
uploaded lineage and descriptions
uploaded synapses
uploaded receptors and innexins
Traceback (most recent call last):
  File "insert_worm.py", line 277, in do_insert
    update_neurons_and_muscles_with_lineage_and_descriptions()
  File "insert_worm.py", line 100, in update_neurons_and_muscles_with_lineage_and_descriptions
    cells = {next(x.name()) : (norn(x.lineageName()), norn(x.description())) for x in P.Cell().load() }
  File "insert_worm.py", line 100, in <dictcomp>
    cells = {next(x.name()) : (norn(x.lineageName()), norn(x.description())) for x in P.Cell().load() }
  File "insert_worm.py", line 92, in norn
    return next(x)
TypeError: unicode object is not an iterator

Hi @travs. Could you have a look at #76? In that pull request, update_neurons_and_muscles_with_lineage_and_descriptions is no longer called since the relevant activity takes place in upload_lineage_and_descriptions

Looks like this is fixed by #76 !


Accessibility issue in testing peek view

The original issue comes from https://github.com/microsoft/vscode-java-test/issues/1387
Comment from @sevapopov2:

Hello, here are more observations I have just made. It is possible to invoke the peek output view by shortcut, it's CTRL+; and m. Each time I invoke it, NVDA starts reading the test output out loud, but the focus is always placed at the line in the test file and as far as I understand it's selected. You can check that it's placed each time by closing the opened test file and invoking the output command again. Screen reader focus stays in the editor with a possibly selected code. I haven't found a way to browse the results by reading them slowly, line by line yet and be able to access links that are mentioned in the report if there are any. Thank you!

@connor4312 Looks like there is some problem with the editor focus when the test message view is peeked out?

I found a way to view test results and work with tests.
When the tests are executed, I need to press CTRL+m for tab key to move between elements, then tab until the close button and after that there will be a tree with test results and different buttons to access such as go to the previous, next failures and others. That's accessible with NVDA, I am sure it's also accessible with Orca. Thank you for help all!

Thanks for the issue!
@isidorn at the moment we reveal the failed position and then focus the editor. Instead is it preferable to focus within the peek view like the references peeks do?
https://github.com/microsoft/vscode/blob/c1d916181410459a82e9b66f8e7fd3b67e2138e2/src/vs/workbench/contrib/testing/browser/testingOutputPeek.ts#L751-L753

@connor4312 yeah, it would be better if you focus inside the peek automatically for accessibility reasons.
If you want, you can also differentiate if the command is triggered via keybinding, if yes then move focus, otherwise do not. This is what the editor hover does.
Though I think it might be a bit confusing in this case, so I suggest to just pass focus inside the peek.

Thank you for such an active feedback! I also think that it would be more convenient if the focus moves directly to the peek instead of moving it to the editor with the selected line that can be easily deleted by accedent. Thank you!

This bug has been fixed in the latest release of VS Code Insiders!
@jdneo, you can help us out by commenting /verified if things are now working as expected.
If things still don't seem right, please ensure you're on version 15253f1528573fc8f84e956b21af4bad57f38c07 of Insiders (today's or later - you can use Help: About in the command palette to check), and leave a comment letting us know what isn't working as expected.
Happy Coding!

@connor4312 how about some verification steps unless @sevapopov2 resurfaces?

Steps in the original issue should still be accurate. Opening test peek output (for a failed test) should focus and screen-read the peek view, rather than staying in the editor.


Update script to check Homebrew path for Apple Silicon

This is related to issue #62

Thanks for this change. I'll run the tests. If they pass, I'll merge this.


How can I change the formatting?

How can I make it render like "1. " or "1 - "?

Hi @bibek-maharjan It is not currently possible, but I'm going to look into it.

Hi @bibek-maharjan: I've added the support like you requested. Try it out!


VS Code 1.85.1 jumps into wrong file line from Search menu

Type: Bug
Working with Cisco .cfg configuration export files.
The linux "file" command reports the file as: ASCII text, with CRLF, CR line terminators.
VSCode reports the file as UTF-8 CRLF.
Notepad++ shows every line in a file ends with CRCRLF.
The issue goes away if I change the line endings with a command.
Steps to reproduce:

Get a file with CRCRLF line endings.
Search for string in Search menu
Click on Search results in any file
Code will jump into wrong line most of the time
After closing and opening that file in editor and cliking again will solve the problem for that file only

Looks the same as https://github.com/Microsoft/vscode/issues/32547
VS Code version: Code 1.85.1 (0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2, 2023-12-13T09:49:37.021Z)
OS version: Windows_NT x64 10.0.19042

relevant? -> https://github.com/microsoft/vscode/issues/127#issuecomment-159306903

relevant? -> https://github.com/microsoft/vscode/issues/127#issuecomment-159306903

Yes, it seems that, according to that comment, we don't support mixed line endings, at least for find/search. I would prefer that the find functionality support this before search does.


support awareness of in-flight nodes & startup taints

1. Issue, if available:
Fixes #628 and #972
2. Description of changes:

don't launch nodes if the pod can schedule on a recently launch
node that hasn't become ready
support startup taints

Co-authored-by: Bryan Stenson<EMAIL_ADDRESS>3. How was this change tested?
Unit testing & scaling cluster on EKS. I tested with two EKS clusters, one using Cilium and one standard cluster.
4. Does this change impact docs?

[X] Yes, PR includes docs updates
[ ] Yes, issue opened: link to issue
[ ] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Tested and I am able to launch the cilium connectivity test on a single node cluster. This leads to unschedulable pods for the multi-node tests, so Karpenter launches a second node and doesn't bind the pods to the node due to the startup taint.  Eventually cilium becomes ready, removes the startup taint and kube-scheduler binds the pods.
NAMESPACE     NAME                                                     READY   STATUS    RESTARTS   AGE     IP                NODE                                           NOMINATED NODE   READINESS GATES
cilium-test   echo-a-c4cdff77c-jr6bh                                   1/1     Running   0          4m7s    <IP_ADDRESS>   ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
cilium-test   echo-b-598c78b9fc-h458m                                  1/1     Running   0          4m6s    <IP_ADDRESS>     ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
cilium-test   echo-b-host-5556f9488-rxdkk                              1/1     Running   0          4m5s    <IP_ADDRESS>    ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
cilium-test   host-to-b-multi-node-clusterip-7f8c9699d6-ppc62          1/1     Running   0          4m1s    <IP_ADDRESS>     ip-192-168-11-65.us-west-2.compute.internal    <none>           <none>
cilium-test   host-to-b-multi-node-headless-7df4d6fdb-zkfc4            1/1     Running   0          4m1s    <IP_ADDRESS>     ip-192-168-11-65.us-west-2.compute.internal    <none>           <none>
cilium-test   pod-to-a-679f686cb-frndq                                 1/1     Running   0          4m5s    <IP_ADDRESS>     ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
cilium-test   pod-to-a-allowed-cnp-54755cc9c6-7tr2v                    1/1     Running   0          4m3s    <IP_ADDRESS>   ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
cilium-test   pod-to-a-denied-cnp-7bfb7d69b8-kv4wf                     1/1     Running   0          4m4s    <IP_ADDRESS>   ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
cilium-test   pod-to-b-intra-node-nodeport-6bd8d9468d-4cncq            1/1     Running   0          3m59s   <IP_ADDRESS>     ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
cilium-test   pod-to-b-multi-node-clusterip-7984ccf8c6-v9vm9           1/1     Running   0          4m2s    <IP_ADDRESS>   ip-192-168-11-65.us-west-2.compute.internal    <none>           <none>
cilium-test   pod-to-b-multi-node-headless-7fb5f5c84b-g4654            1/1     Running   0          4m2s    <IP_ADDRESS>    ip-192-168-11-65.us-west-2.compute.internal    <none>           <none>
cilium-test   pod-to-b-multi-node-nodeport-54c89d778-hhggm             1/1     Running   0          4m      <IP_ADDRESS>    ip-192-168-11-65.us-west-2.compute.internal    <none>           <none>
cilium-test   pod-to-external-1111-c98db84d4-6qdph                     1/1     Running   0          4m4s    <IP_ADDRESS>    ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
cilium-test   pod-to-external-fqdn-allow-google-cnp-5f55d8886b-846hw   1/1     Running   0          4m3s    <IP_ADDRESS>    ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
karpenter     karpenter-c4954dbf-wwmtl                                 2/2     Running   0          5m20s   <IP_ADDRESS>   ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
kube-system   cilium-52q78                                             1/1     Running   0          12h     <IP_ADDRESS>    ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
kube-system   cilium-node-init-9x2xg                                   1/1     Running   0          3m55s   <IP_ADDRESS>     ip-192-168-11-65.us-west-2.compute.internal    <none>           <none>
kube-system   cilium-node-init-z9vb7                                   1/1     Running   0          12h     <IP_ADDRESS>    ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
kube-system   cilium-operator-6689cf95b6-thpwb                         1/1     Running   0          12h     <IP_ADDRESS>    ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
kube-system   cilium-vb6l7                                             1/1     Running   0          3m13s   <IP_ADDRESS>     ip-192-168-11-65.us-west-2.compute.internal    <none>           <none>
kube-system   coredns-85d5b4454c-4p9s9                                 1/1     Running   0          13h     <IP_ADDRESS>    ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
kube-system   coredns-85d5b4454c-rq4vr                                 1/1     Running   0          34h     <IP_ADDRESS>     ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
kube-system   kube-proxy-5lrqf                                         1/1     Running   0          34h     <IP_ADDRESS>    ip-192-168-55-185.us-west-2.compute.internal   <none>           <none>
kube-system   kube-proxy-x92cm                                         1/1     Running   0          3m13s   <IP_ADDRESS>     ip-192-168-11-65.us-west-2.compute.internal    <none>           <none>

I'm wary of the long term impact of including startup taints in the api. It's discussed in a number of places, but I'll restate here. Upstream has a concept to address this use case (including the node restart case) called the NoAdmit taint effect. This allows scheduling but prevents the kubelet from running the pod, which is exactly what we need.
I'd hate to include this in karpenters API as a point in time workaround for something that will eventually be fixed upstream. One option is to support this taint effect in karpenter before it's merged upstream and do the upstream work in parallel.
There may be reasons to not do this, and we can always deprecate a field using an API bump, but I want to make sure we're rigorously evaluating our options for changes that expand the API.

I'm wary of the long term impact of including startup taints in the api. It's discussed in a number of places, but I'll restate here. Upstream has an unimplemented concept to address this use case (including the node restart case) called the NoAdmit taint effect. This allows scheduling but prevents the kubelet from running the pod, which is exactly what we need. One benefit of doing this upstream is that the change applies to all integrators like CAS, Karpenter, and the Descheduler.

I don't think that NoAdmit solves this problem.  From the docs, NoAdmit is "NoSchedule" + don't start pods that are scheduled.  If it were "don't start pods only", maybe it could work but as it, it doesn't fully convey the concept of a temporary taint applied at startup that will be eventually removed.   Even if it were "don't start pods", I don't think it still fully implies that the taint is temporary.

I'd hate to include this in karpenters API as a point in time workaround for something that will eventually be fixed upstream. One option is to support this taint effect in karpenter before it's merged upstream and do the upstream work in parallel.

The NoAdmit taint was introduced, commented out, in 2016 . That appears to be the initial implementation of taints and it hasn't been needed yet.  Agree that it doesn't seem to be a priority and we would need to do the work, but I also don't think it solves this particular problem.
To solve the problem, we would also need to treat 'NoAdmit' as a temporary taint by provisioning capacity for workloads that don't tolerate it and then assume something would remove the taint.  We would have implemented the concept of startup taints only hidden behind a magic effect name. If NoAdmit is ever implemented upstream, Karpenter would then be supporting it incorrectly as nothing in the NoAdmit docs indicate that it will always be removed.

I'm wary of the long term impact of including startup taints in the api. It's discussed in a number of places, but I'll restate here. Upstream has an unimplemented concept to address this use case (including the node restart case) called the NoAdmit taint effect. This allows scheduling but prevents the kubelet from running the pod, which is exactly what we need. One benefit of doing this upstream is that the change applies to all integrators like CAS, Karpenter, and the Descheduler.

I don't think that NoAdmit solves this problem. From the docs, NoAdmit is "NoSchedule" + don't start pods that are scheduled. If it were "don't start pods only", maybe it could work but as it, it doesn't fully convey the concept of a temporary taint applied at startup that will be eventually removed. Even if it were "don't start pods", I don't think it still fully implies that the taint is temporary.

Think of these independently:

NoSchedule: prevent scheduling
NoExecute: evict any running pods
NoAdmit: prevent the pod from running

In the initial implementation, there was some concept of combining the taint effects into a single effect, e.g. NoScheduleNoAdmitNoExecute. Wisely, upstream never went this direction and instead recommended that users specify both NoSchedule and NoExecute separately if both behaviors are desired. Similarly, I'd expect NoAdmit to behave this way, and don't expect that NoScheduleNoAdmitNoExecute would be accepted upstream.
The idea that a taint exists at startup and will eventually be removed is superceded by the more general "don't run if this exists.". NoAdmit captures cases of startup and then removed, as well as flip flop (restart) scenarios. If we separate NoScheduleNoAdmit from NoAdmit, then we can just follow natural scheduling rules and the node will sort everything out at runtime. It's possible that the taint may take hours to be removed (think maintenance windows, or similar mechanisms), and neither karpenter nor the scheduler need to care.

I'd hate to include this in karpenters API as a point in time workaround for something that will eventually be fixed upstream. One option is to support this taint effect in karpenter before it's merged upstream and do the upstream work in parallel.

The NoAdmit taint was introduced, commented out, in 2016 . That appears to be the initial implementation of taints and it hasn't been needed yet. Agree that it doesn't seem to be a priority and we would need to do the work, but I also don't think it solves this particular problem.

See above.

To solve the problem, we would also need to treat 'NoAdmit' as a temporary taint by provisioning capacity for workloads that don't tolerate it and then assume something would remove the taint. We would have implemented the concept of startup taints only hidden behind a magic effect name. If NoAdmit is ever implemented upstream, Karpenter would then be supporting it incorrectly as nothing in the NoAdmit docs indicate that it will always be removed.

Agree that a magical taint that is never implemented upstream could result in an awkward customer experience. I'm not sure whether or not it's worse than a special field to capture this edge case. There are are a couple paths here:

We implement NoAdmit, and successfully upstream the concept 🎉
We implement NoAdmit, and upstream does something else -- may require deprecation in Karpenter
We implement startupTaints and upstream implements NoAdmit -- may require deprecation in Karpenter
We implement startupTaints and nothing happens upstream 🥈 .

How hard do you think it would be to get a directional sense from SIG Scheduling on this? My guess is that they would be supportive, since this concept is simple to implement, has precedence, and applies generally to CAS/Karpenter/etc.

Think of these independently:
* NoSchedule: prevent scheduling

* NoExecute: evict any running pods

* NoAdmit: prevent the pod from running

NoExecute does not just evict running pods though, it also implies NoSchedule.  I would expect NoAdmit to work similarly if it were to be implemented, though it could conceivably behave differently.

In the initial implementation, there was some concept of combining the taint effects into a single effect, e.g. NoScheduleNoAdmitNoExecute. Wisely, upstream never went this direction and instead recommended that users specify both NoSchedule and NoExecute separately if both behaviors are desired. Similarly, I'd expect NoAdmit to behave this way, and don't expect that NoScheduleNoAdmitNoExecute would be accepted upstream.

This isn't correct, if you put a NoExecute taint on a node it also prevents scheduling any pods unless they tolerate the taint.  If it didn't, the pod would just rapidly get scheduled and evicted immediately.

The idea that a taint exists at startup and will eventually be removed is superceded by the more general "don't run if this exists.". NoAdmit captures cases of startup and then removed, as well as flip flop (restart) scenarios. If we separate NoScheduleNoAdmit from NoAdmit, then we can just follow natural scheduling rules and the node will sort everything out at runtime. It's possible that the taint may take hours to be removed (think maintenance windows, or similar mechanisms), and neither karpenter nor the scheduler need to care.

We need awareness of which taints are temporary and even if NoAdmit were to be accepted and implemented upstream, we can't assume that it would always be a temporary taint.  For in-flight nodes we care about the difference between a taint that we place on a node at startup which will exist for a short period of time vs a taint that can exist for hours.  In the latter case, we need to provision a new node.  In the first case, we would not.
The other issue is that for startup taints, we provision capacity even though the pod doesn't tolerate the taint because of the temporary nature of the taint.  This is different from other taints in which we only provision capacity if the pod tolerates the taint.

Agree that a magical taint that is never implemented upstream could result in an awkward customer experience. I'm not sure whether or not it's worse than a special field to capture this edge case. There are are a couple paths here:
1. We implement NoAdmit, and successfully upstream the concept 🎉

Assuming that:

we can implement NoAdmit and get it merged upstream
it doesn't work as described in its initial docs and doesn't work similarly to the existing NoExecute and instead only prevents pods from running

Internally, we would have to add to 'NoAdmit' the concept of provisioning capacity for pods even if they don't tolerate the taint or it doesn't solve the current problem.   For in-flight nodes we have to treat NoAdmit as temporary to prevent us from provisioning excess capacity. If NoAdmit is then supported upstream, customers now can't taint their nodes with "NoAdmit" using Karpenter unless they want us to treat that with our special temporary taint handling. We then need to explain how/why we treat "NoAdmit" differently from every NoSchedule and NoExecute taints and are left in an awkward state.

2. We implement NoAdmit, and upstream does something else -- may require deprecation in Karpenter

We still have a special Karpenter only taint effect whose name doesn't clearly describe what it does.

3. We implement startupTaints and upstream implements `NoAdmit` -- may require deprecation in Karpenter

See 1, NoAdmit doesn't imply temporary so we still need the concept of a temporary taint.  We need something like startupTaints that lets customers express to Karpenter that a taint is temporary so pods don't need to tolerate it for the purposes of provisioning and so we can recognize it on in-flight nodes and know that the capacity will soon be available.  For the CCM and Cilium case, that capacity will be available faster than we can launch a node since every node goes through the same initialization process.

4. We implement startupTaints and nothing happens upstream 🥈 .

How hard do you think it would be to get a directional sense from SIG Scheduling on this? My guess is that they would be supportive, since this concept is simple to implement, has precedence, and applies generally to CAS/Karpenter/etc.

I could see NoAdmit potentially solving some issues with pre-binding, but not all (e.g. extended resource zero'ing).  I don't think it can solve the 'temporary startup taints' problem at all.

I'll be offline due to a medical emergency. Don't block on this.
You make some good points and I've learned a bunch from this discussion. However, I think you're missing the core of what I'm saying. Startup taints is a perfectly reasonable short term (and maybe long term) approach, even if I think we might be able to coordinate a few changes that would collapse the user facing complexity with a more general solution.


Retire Packages, Metapackages, and Frameworks

Fixes #1311

Thanks @pkulikov!

This PR introduced some warnings

@mairaw Thanks for letting me know.


Add onchange event when selecting on click in empty ListBox

The onchange event was not being triggered when selecting from ListBox without having typed anything in the .hw-combobox__input.
I tried to see if the navigate/ClickOnTheOutside stimulus actions were doing something SelectOnClick wasn't, but soon I realised that the onchange event was being triggered only due to its nature.

Hey @RickHPotter thanks for the PR,
I think this is what you're looking for: https://github.com/josefarias/hotwire_combobox/issues/137#issuecomment-2035555941
The combobox isn't actually changing in this scenario. So I don't love the idea of having it emit a change event. However, we are emitting our own custom events from the canonical component element to support the kind of interaction you're describing.
Closing for now. But let me know if the above doesn't work for you and we can discuss alternatives, including the one proposed here.

Thanks for your thorough answer. That works really well with what I plan to do in my app.
I guess I should have read more thouroughly the issues before trying to "fix" something, lol.


expression: has

has expression
This PR provides a has expression to ol/expr which returns a boolean based on the context properties having a certain key (or nested key).
Reasoning
It is not trivial to get a has or defined operation with the current expressions. Things like ["any", ["get", "bar"], ["==", ["get", "bar"], ""], ["==", ["get", "bar"], false], ["==", ["get", "bar"], 0]] would work, but aren't very convenient.
Solution
Similar to mapbox has this expression will return true if the key was found in the properites, making the above expression reduce to ["has", "bar"]. The syntax is identical to the get expression, where you can provide more then one key for nested properties.

@jahow this is ready for review. im not a webgl expert, but from my understanding of shader programming, there is no undefined in glsl. if you have a proposal to get has to work for webgl as well, id happily give it a shot

thanks! indeed in WebGL we need to introduce a special "undefined" value somehow, not sure yet how that would work out. I'll give this a review asap

I'm reworking our accessor expression handling (get, var) in https://github.com/openlayers/openlayers/compare/main...tschaub:openlayers:accessor-expressions.  This is going to involve changing the WebGL style parsing and evaluation, ideally sharing the evaluation of accessor expressions on the CPU.  I'll explain more when opening a pull request for that branch.  Just mentioning it here because I don't think it is worth trying to include has handling in WebGL as part of this pull request.


Do you know where is the source of vgg2_fp?

Do you know where is the source of VGG2_FP?  what I can find is the aligned vgg2_fp, not original vgg2_fp.

@twmht Do you know a publication link of vgg2_fp?


Provide back-up beacon node URLs for validator processes and fail-over automatically

For discussion.
Description
As a staker, I want my validators to automatically failover to a backup beacon node if the first beacon node is not available or out of sync or otherwise in distress, so that I can continue the work of validation without downtime.

Closing as duplicate of #5237.


Unlock Resource Loader before calling copy_from

Unlock the resource loader before calling copy_from as copy_from can trigger resource change emissions leading to further resource loads, for example when calling copy_from on a shader resource, it will trigger a change emission leading to the shader preprocessor resource loading any shader includes in the .gdshader.

Could you open an issue first, ideally with an MRP? I'd like to assess the problem so I can actually tell if this is the right fix.


filterTemplate js console error

I created a jsFiddle to show the problem https://jsfiddle.net/ecropolis/whcmzr5q/2/
on my app I'm loading the data via ajax and that breaks so my grid does not get populated. I'm not sure if it is a bug or if I'm doing something wrong but it seems like the UI is rendering how I expect. I haven't finished the functionality here but my plan is to execute a new ajax/query and return a new result so I really want the filter done on the server side for the status column. (This fiddle has the error but when the data is static rather than loaded via ajax, the data still loads. https://jsfiddle.net/ecropolis/whcmzr5q/)

The reason is that when you redefine filterTemplate you either should redefine filterValue, or if you want to keep existing implementation, you need to save this.filterControl variable (grid just cannot predict what you have you the filterTemplate to get value from it).
So if you prefect second option the fix will be:
filterTemplate:function(){
	console.log(this);
	var $result = this.filterControl = $('<select><option value="1">Confirmed</option><option value="0">Non-Conf</option><option value="8">Not-In</option></select>').on("change", function(o,v) {
		console.log(v);
		console.log(o);
	});
	return $result;
}


Bugfix: Fix delivery action in tray loader

Changelog Description
Fix delivery action after recent refactoring of ayon core.
Additional info
Fixes #351
Since directory and file is now separate in the delivery templates I've also made them two labels in the delivery UI for better clarity as to what is what.

Testing notes:

Set up a delivery template in Anatomy settings ayon+anatomy://PROJECTNAME/templates/delivery, for example like these examples:

[
  {
    "name": "client",
    "directory": "{root[work]}/{folder[name]}/{product[name]}",
    "file": "{folder[name]}_{product[name]}_{<EMAIL_ADDRESS>  },
  {
    "name": "previews",
    "directory": "{root[work]}",
    "file": "{folder[name]}_{product[name]}_{<EMAIL_ADDRESS>  },
  {
    "name": "per_asset",
    "directory": "{root[work]}/{folder[name]}",
    "file": "{folder[name]}_{product[name]}_{<EMAIL_ADDRESS>  },
  {
    "name": "main",
    "directory": "{root[work]}/{project[name]}/delivery/{yyyy}-{mm}-{dd}/{hierarchy}/{folder[name]}/",
    "file": "{project[code]}_{folder[name]}_{task[short]}_{<EMAIL_ADDRESS>  }
]

In Tray Launcher open the Loader
Using the Delivery action in the tray loader initiate a delivery to an output root.

Meanwhile been successfully tested :)
Delivery templates created:

and in the Loader

Been tested with those 3 templates, all working well!


Include field names when using query.

Something we noticed after upgrading from mysql2 to trilogy was the lack of field names when running a query manually.
Here is a simple example of the differences:
mysql2
Mysql2::Client.new( config ).query( "SELECT * FROM instruments" ).first
#=> {"SerialNumber"=>"ABCD1234", "Manufacturer"=>"Thermo", "Model"=>"TVA2020"}

trilogy
temporary_database_connection = Trilogy.new( config )
temporary_database_connection.select_db( config[ "database" ] )
temporary_database_connection.query( "SELECT * FROM instruments" ).first
#=> ["ABCD1234", "Thermo", "TVA2020"]

Differences
A few differences to point out:

In order to establish a database connection and run a query, you need to create a new Trilogy instance and then use select_db or change_db and provide the database name again, otherwise, you get a "No database selected" error.

mysql2 returns a Hash while trilogy returns an Array from the query method.

In an otherwise incredible drop-in replacement, this prevents the usage of fetching the row values based on the column names, like this:
temporary_database_connection.query( "SELECT * FROM instruments" ).first.fetch( "SerialNumber" )

Which worked very well on mysql2.

We're reverting back to mysql2 in the meantime but thought I would post it here to get your input because this is (so far) the only thing preventing us from moving forward with trilogy.
Many thanks!
Joshua

Related: https://github.com/trilogy-libraries/trilogy/issues/99

Note that temporary_database_connection.query( "SELECT * FROM instruments" ) returns a Trilogy::Result, which does have #fields available. So as an immediate workaround you could do something like:
result = temporary_database_connection.query( "SELECT * FROM instruments" )
result.rows.map { |row| result.fields.zip(row).to_h }

to get hashes.

Very true, and that's a great use of zip!
We use this in quite a few places so it would mean changing a lot of lines, minor as it is.
Is there a desire to have rows return a Hash with the fields as keys or is that not going to work? If so, then we'll help with that before moving to trilogy. If not, then we'll go this workaround route at some point.
Let me know. And thanks again!

Trilogy isn't a intended as a 1:1 drop in replacement for mysql2, though it is similar due to shared authorship and consumers.
I don't think it's a good idea for us to change the behaviour of rows as that would break backwards compatibility. I think you can achieve what you want using result.each_hash.first

Oh right, I totally forgot about each_hash. Thanks John. Yeah, I agree with closing this.

Totally fair. I'll give that a shot and go from there. Thanks for all your work on this.

Just wanted to say, .each_hash.with_index( 1 ) do |row, row_number| works a treat!
I didn't know that method existed. Is there a good place to add some documentation for that? Cheers!

Our documentation is limited at the moment, but we do have https://github.com/trilogy-libraries/trilogy/blob/3c3a3865a3896f3056b1d7c69c5244964b60937c/contrib/ruby/README.md?plain=1#L33. The method is implemented in https://github.com/trilogy-libraries/trilogy/blob/3c3a3865a3896f3056b1d7c69c5244964b60937c/contrib/ruby/lib/trilogy/result.rb#L9-L25


update the Android migration docs

The Android SDK 2.x has a breaking change for the network override use case, which we missed mentioning in the migration docs.
The iOS SDK also has a similar breaking change in the latest versions. I will update the iOS migration docs in another PR since it's sitting in the acp doc repo. https://developer.adobe.com/client-sdks/previous-versions/documentation/mobile-core/migration/

@praveek @calebk1  your review comments have been addressed.


Do not allow writing records which content_stream() has been modified as it results in partial or empty content

Related to #64:
When writing a record into an archive one can read out its content via the streaming API and write the record (now with partial or empty content) into the archive resulting the loss of content. We clearly do not or at least should not want that. An Exception or a Warning should be raised instead of loosing content silently.
from warcio.warcwriter import WARCWriter

filename = 'test.warc.gz'

# Write WARC INFO record with custom headers
output_file = open(filename, 'wb')
writer = WARCWriter(output_file, gzip=True, warc_version='WARC/1.1')

# Custom information and stuff
info_headers = {'custom': 'stuff'}
info_record = writer.create_warcinfo_record(filename, info_headers)

custom_headers_raw = info_record.content_stream().read(6)  # TODO: After this writing should not be allowed

writer.write_record(info_record)

output_file.close()

Result (notice the partial payload):
WARC/1.1
WARC-Type: warcinfo
WARC-Record-ID: <urn:uuid:67a981ea-fece-49b9-834a-e3b660042cf5>
WARC-Filename: test.warc.gz
WARC-Date: 2019-08-15T10:44:53.778034Z
Content-Type: application/warc-fields
Content-Length: 15

: stuff

Yes, this is an easy mistake to make when doing anything complicated.

I was bit by this recently as well. In my tool, I'm writing a log file (using logging). Before the process exits, I copy this log file into the WARC as a resource record by open()ing it in read mode and passing it to the WARCWriter. I also pass the record length (= file size) to avoid the temporary copy created in ensure_digest otherwise.
The above works perfectly fine – unless the log file is appended to between creating the record and writing it. I recently started adding very detailed debug-level logging messages, and some of those are produced in that window. What happens is that the length and digest are taken at create_warc_record time, but then the file is copied in its entirety into the record body. Due to the size mismatch, the file becomes unreadable.
Here's a small example to illustrate this (logging also flushes the file automatically on every message as of Python 3.8.4):
import os
import warcio

with open('test.txt', 'w') as logfile:
	logfile.write('foo\n')
	logfile.flush()
	with open('test.warc.gz', 'wb') as fp:
		writer = warcio.warcwriter.WARCWriter(fp)
		with open('test.txt', 'rb') as recordinput:
			recordinput.seek(0, os.SEEK_END)
			length = recordinput.tell()
			recordinput.seek(0)
			record = writer.create_warc_record('file://test.txt', 'resource', payload = recordinput, length = length)
			logfile.write('bar\n')
			logfile.flush()
			writer.write_record(record)

The result is a WARC like this:
WARC/1.0
WARC-Type: resource
WARC-Record-ID: <urn:uuid:7d16bc43-5eeb-4b63-b3a0-64119ec8890f>
WARC-Target-URI: file://test.txt
WARC-Date: 2020-07-15T01:51:23Z
WARC-Payload-Digest: sha1:6HJNF6JE5GDKZBX566ZWZFF434ZL53AV
WARC-Block-Digest: sha1:JZEOFSND2LFIU4EMWDGFIVYAKRHPWUBB
Content-Type: application/warc-record
Content-Length: 4

foo
bar

The content length and payload digest are from b'foo\n' but the block contents and block digest are from b'foo\nbar\n'. The digest difference happens because the block digest is only calculated on the write, not on record creation (which also means the file needs to be read thrice in total instead of only twice; is there a good reason for that?).
This is of course a special case. First, the file object passed into warcio isn't actually manipulated directly; the modification happens indirectly through the file system, which makes it much less obvious to spot. And second, it's only a file append, not a movement of the file pointer (through read or seek) or an overwrite of existing data. This specific case can actually be fixed fairly easily. I'm now using a wrapper around the file object which prevents access beyond the initial file length. It's also fixable in warcio: _iter_stream just needs to only read length bytes instead of reading until EOF. In fact, by requiring _iter_stream to read exactly that many bytes, the issue in the original report can also be kind of solved by raising an error if _iter_stream is unable to read that much. This would at least clearly indicate that something's wrong.

I think that this should definitely be addressed, writing after reading should be disallowed:
record.content_stream()
record.write_record() <- this should throw an error

Adding a non-streaming source as per #64 could provide an alternative.
@JustAnotherArchivist
Yes, perhaps the ensure_digest() in create record is unnecessary, it should compute both digests only when writing.
If someone needs the digest sooner, they can call ensure_digest() explicitly..
Adding a length check is probably a good idea, but in general there's no way to prevent user from modifying record between create_warc_record() and write_record() except maybe combining that into a single call.
Sometimes that can be useful for adding additional data to record or have a reference to the record, but its still possible to change it in a way that makes it invalid since you can always manipulate the stream still until its actually written..

@ikreymer
Indeed, short of making a full copy (as ensure_digest does when the length is unknown) with its significant performance and disk space impact for large records, it's impossible to prevent that. The simple cases which modify the stream position (original issue) or the stream length (mine) can be solved though: keep the value of the stream position at create_warc_record time, then seek to that position before writing, and only write length bytes. This only works for seekable streams, of course.


Update Get Started with Linux prerequisites

Title
On the title describe
what you've fixed (or created) with this Pull Request (PR).
Summary
Update Get Started to remove Linux installation links, and add a link to the .NET install page and to the Linux Prerequisites page.
Suggested Reviewers
@mairaw

@JRAlexander,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot


Expose migration engine with some utility functions

Would be useful for Spacedrive and other desktop apps. Probably needs to be behind a feature flag.

W

Closed by #158


fix:  suspend job status can not turn into Suspended and report error…

What this PR does / why we need it:
Fixed
Suspend job status can not turn into Suspended condation and report error "job completion time is nil, cannot cleanup"
Checklist:

[ ] Docs included if any changes are user facing

/assign @kubeflow/wg-training-leads

/assign @andreyvelich

/assign @tenzen-y

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 4 (0.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 35.42%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/controller.v1/common/job.go
0
4
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.01%

Covered Lines:
4380

Relevant Lines:
12366

💛  - Coveralls

+1 on this issue, been experiencing the same behavior on 1.7 as well, when the feature was introduces. Can confirm @xuxianzhang's fix is working and suspend\resume is working properly with it.
@andreyvelich, can we move forward with merging it?
Thanks!


feat: installer component support for Houdini 20.0 and 20.5

Fixes: #123
What was the problem/requirement? (What/Why)
The submitter installer component was written in a way that it would only work for Houdini 19.5 installations.
What was the solution? (How)

Generalized portions of the installer component so that it will install for 19.5, 20.0 and 20.5 while also hopefully making it easier to update in the future for new versions

Each version of Houdini is now a subcomponent and the overall Houdini component is a component group
There did not appear to be a way to have the folder section copy to multiple destinations so instead the package file is copied to a temp location before then being copied over to the Houdini packages folders for the selected versions. The swapping of the INSTALL_DIR_PLACEHOLDER has now been moved to the folder actionList which runs immediately after the file is copied to the temp location before it's then copied to the Houdini package folders.

Changed the substitution of INSTALL_DIR_PLACEHOLDER to use unix paths on Windows to fix #123 so that there are no weird escape issues with certain characters and unescaped backslashes on Windows.

Selection of the version of Houdini will look like below:

New CLI flags to select the components:

What is the impact of this change?

A future release of the installer will use the new component and be able to install for Houdini 20.0 and 20.5

How was this change tested?

Built dev builds of the installer for Windows and Linux
Ran through the installer on both OSes in GUI mode and verified that I could select individual Houdini versions to install for and validated that nothing was being changed for the other versions.
Ran through the installer on both OSes in CLImode and verified that I could select individual Houdini versions via flags during installation and validated that nothing was being changed for the other versions.
I verified that if the Houdini package directory hadn't been made yet the installer would create it(the packages directory doesn't exist by default and the user pref directory won't until Houdini has been launched at least once).
Verified that on unisntallation the package files were removed
Verified that on installation over an existing installation the package files were still updated
Verified on each OS that I could use the submitter in Houdini 19.5, 20.0 and 20.5

Was this change documented?
No
Is this a breaking change?
No

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Needs rebase then should be good 👍


feat: add endpoint for listing deployments

If you are happy with the implementation, I will add tests.

I need to add tests before merging the PR


Confluent Cloud Integration

Purpose
These changes allow the PPM to communicate with Confluent Cloud.
Notes

I've added "using namespace std" to a few files to increase readability.
This PR also contains the changes in the feature/cdot-edges branch.
There is now a file called docker-compose-confluent-cloud.yml which uses the Docker.cdot file to spin up the PPM by itself.

These changes have been verified to work with both Confluent Cloud and a locally hosted kafka instance.

I think the last thing to do for this PR is to get the "do_kafka_test.sh" script to work.

I think the last thing to do for this PR is to get the "do_kafka_test.sh" script to work.

I've created a new work item for updating the "do_kafka_test.sh" script.


fix(sgtrivy): update misconfig-scanners

Trivy has added support for scanning terraform plan snapshots and
thus so updated the config names.
See PR
Fixes error:
[terraform:security-check] 2024-04-09T09:37:47.378Z	INFO	Misconfiguration scanning is enabled
[terraform:security-check] 2024-04-09T09:37:47.378Z	INFO	Need to update the built-in policies
[terraform:security-check] 2024-04-09T09:37:47.378Z	INFO	Downloading the built-in policies...
[terraform:security-check] 46.13 KiB / 46.13 KiB [-----------------------------------------------------------] 100.00% ? p/s 0s
[terraform:security-check] 2024-04-09T09:37:47.570Z	ERROR	Invalid misconfig scanners specified: [terraform terraformplan] defaulting to use all misconfig scanners
[terraform:security-check] 2024-04-09T09:37:50.125Z	INFO	Detected config files: 4

$ ../.sage/bin/trivy config --help
...
Misconfiguration Flags
      --misconfig-scanners strings        comma-separated list of misconfig scanners to use for misconfiguration scanning (default [azure-arm,cloudformation,dockerfile,helm,kubernetes,terraform,terraformplan-json,terraformplan-snapshot])
...

@thall Don't we need to bump the trivy version?

@odsod did so, https://github.com/einride/sage/commit/154b169b434e3dae1b5a5db1dccff12a6b5807fb

Failing here when bumping sage