ASP.Net Core Reverse Proxy / API Gateway sample

ASP.Net Core Reverse Proxy / API Gateway sample
Connects to https://github.com/Particular/Workshop.Microservices.Private/issues/16
Fix https://github.com/Particular/Workshop.Microservices.Private/issues/16
PoA

[x] define custom routes
[x] allow users to define their routes
[x] inject custom routs into the default routing engine
[x] gateway pipeline terminator
[x] support GET requests
~support POST requests~
~support PUT requests~
~support PATCH requests~
~support DELETE requests~
[x] add sales and shipping samples (with hardcoded data)
[x] WebAPI hosts to expose real sales and shipping data
[x] Mimic the same pure MVC sample as in the first Demo
~super simple SPA to interact with back-end~
[x] is this an API Gateway or a reverse proxy? Once defined rename accordingly.

Description
This PR introduces a demo of a composite UI using as composition infrastructure a gateway/reverse proxy built on top of ASP.Net Core 1.1.
This composition infrastructure allows to run a .Net Core web server as simple as:
public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddRouting();
        services.AddViewModelComposition();
    }

    public void Configure(IApplicationBuilder app, ILoggerFactory loggerFactory)
    {
        app.RunViewModelCompositionWithDefaultRoutes();
    }
}

The above configures a default route, whose template is {controller}/{id:int?}, that mimics the WebAPI default behavior. All HTTP requests are intercepted and if matching ViewModelAppenders are found they are invoked, allowing composition to happen.
Similarly to the full MVC Classic sample lists are handled via subscribing to events raised by appenders.

This is ready for review, I know it's huge but I have no other idea on how to commit such a large sample other than this type of PRs (suggestions are more than welcome).
BTW take a look at the PoA, there is a pending question:

is this an API Gateway or a reverse proxy? Once defined rename accordingly.

I like the approach! There are some high level questions that I had during the review which I would also like to ask:

Did we consider the composition to be made on the basis of explicit template? Currently it's the DynamicViewModel which is a structure to which appenders and subscribers glue their fragments?
What is the failure handling approach? Do we support partial reponses? If yes should we notify clients somehow?
What is the assumed security model? Authorization by the Gateway and no verification later on?
Is the url the only source of data for request handling? Should we enable/show how to get data from cookies as well?
Does the caching leave only in the CompositionHandler?

Did we consider the composition to be made on the basis of explicit template? Currently it's the DynamicViewModel which is a structure to which appenders and subscribers glue their fragments?

no, even if it's a thing that would be interesting to explorer. When I started this the goal was to demo a gateway-only composition to return json structures. But on top of that now I don't see why such an evolution should not be tested. @tmasternak can you expand a little bit on the idea? (maybe in its own issue)

What is the failure handling approach? Do we support partial reponses? If yes should we notify clients somehow?

There is no failure handling approach at all ATM.

What is the assumed security model? Authorization by the Gateway and no verification later on?

Appenders can retrieve security related headers from the incoming request and pass along to their own backend. I haven't thought about it all

Is the url the only source of data for request handling? Should we enable/show how to get data from cookies as well?

uhm. Interesting, tell me more

Does the caching leave only in the CompositionHandler?

in a large system one may end up with thousands of appenders, that's the first thing that needs to be cached since they can be easily cached by URL

@adamralph @seanfarmar since you saw the presentation at DDD SouthWest, thoughts on this:

is this an API Gateway or a reverse proxy?

I'm on the fence, from the implementation perspective is a reverse proxy, conceptually is an API Gateway. The decisions affects the components names.
/cc @tmasternak @DavidBoike

no, even if it's a thing that would be interesting to explorer. When I started this the goal was to demo a gateway-only composition to return json structures. But on top of that now I don't see why such an evolution should not be tested. @tmasternak can you expand a little bit on the idea? (maybe in its own issue)

Will do.

uhm. Interesting, tell me more

I think you already answered my question when talking about security. Appenders already have access to the request so they can grab pretty much anything that client sends.

is this an API Gateway or a reverse proxy

Isn't an API gateway an implementation of a reverse proxy? They're not mutually exclusive.
Since API Gateway is better explaining the implementation and better known, I'd go for API Gateway.

I have another question @dvdstelt @adamralph @tmasternak:

Actually the demo supports GET requests only, in light of this request handlers were named IViewModelAppender. The semantic of POST, PUT, PATCH and DELETE doesn't align at all with the concept of "appending" data.
It is much more request handling. I'm actually in favor of changing IViewModelAppender to IHandleRequests because from the HTTP transport perspective a GET and a POST are both requests.

Thoughts?

@mauroservienti 👍 for using IHandleRequets.
Which leads us to other data source which are ISubscribeToCompositionEvents.
First of all I was thinking if something along the lines of IEnrichResponse would be more appropriate Secondly I would really like to hear your opinions about coupling that we currently have between: http request, appenders and subscribers. Especially the subscribers are things that I would like to question. Here are the problems that I see:

They need to match request to work properly - I'm not sure if that is by design or working around framework behavior
It introduces strong coupling between artifacts that are owned by different teams (Sales owns OrdersListViewModelAppender and Shipping owns OrdersLoadedSubscriber).
Are those indicators that the template approach with dedicated team creating template of responses and defining response composition is something we might want to PoC?

@mauroservienti Don't you think IHandleRequests sounds too generic and IViewModelAppender describes better what the purpose of a class does, implementing this interface? Or does it do other things?
I'll let @mauroservienti respond to @tmasternak his questions, because he knows best. But afaik most of it is because of the framework and the fact that it's really hard to come up with a good solution that doesn't involve writing TONS of interfaces and classes, without providing anything additional.

@tmasternak

Which leads us to other data source which are ISubscribeToCompositionEvents.

I haven't thought to ISubscribeToCompositionEvents yet, but your comment sounds reasonable 😉

and defining response composition is something we might want to PoC?

What about, starting from this, add a new "demo" or create a PoC?

Don't you think IHandleRequests sounds too generic and IViewModelAppender describes better what the purpose of a class does, implementing this interface? Or does it do other things?

@dvdstelt I agree but it's not aligned with the HTTP behavior, so it might create a lot of confusion when it comes to strage HTTP usages such as POST requests to overcome GET URL length limitations. That means that you might want to have a IViewModelAppender invoked by a POST. As it stands now you cannot, appenders are bound to GET requests.
And HTTP has always an incoming URL + payload and always expects a response payload. Verbs are on top of the transport.

I'm playing with the new syntax here with the goal of building a slightly more real life sample that handles reads and writes.

Found a bug https://github.com/Particular/Workshop.Microservices/issues/64, I'll fix before merging.

@mauroservienti any chance you can do a screencast to give an overview of the solution? (similar to your presentation in DDD)?

Good idea @seanfarmar, I can do that this Friday

(rebased on master)

fixed history, reverted a couple of wrong commits. Thanks to @adamralph for his support and git knowledge.

@adamralph, as per our call:

demo is now split into 2 different solutions

ASP.Net Core API Gateway
ASP.Net MVC Core

renamed the demo root folder from ASP.Net Core API Gateway to ASP.Net Core

In the coming days I'll have a look at the last item:

In the solution that demos the Gateway alone add a super-simple js client

Or is it better to merge this as is and raise a new issue for that, this is already big enough on its own

👍 thanks @mauroservienti
Regarding the JS client for the gateway, I think it can be done separately, since the gateway is a completely new aspect to the demo. However, I think you can start work on it now. If it gets done in time to go in with this PR then that's good, but if not, it can be sent in another PR.Perhaps you should push any WIP to a new branch based on this one. If this branch is not merged by the time the work is done. You can fast forward this branch to the tip of that new branch and include the JS client in this PR.

re the "Solution and project naming" being the one that worked on it for several weeks to me as it is now makes a lot of sense, just because I know what's going on. All your suggestions make sense. Pick the one you prefer and makes most sense from the point of view of someone that never saw this before and I'll adopt that.

What is the rationale behind the split of types between ITOps.ViewModelComposition.csproj and ITOps.ViewModelComposition.Gateway? I see that ITOps.ViewModelComposition.csproj even has some types in the ITOps.ViewModelComposition.Gateway. Is that by design?

ITOps.ViewModelComposition is the engine, it has no knowledge at all of the presentation technology. ITOps.ViewModelComposition.Gateway knows how to setup a reverse proxy with ASP.Net Core, and ITOps.ViewModelComposition.Mvc know how to deal with ASP.Net MVC Core.

ITOps.ViewModelComposition.Json.csproj seems to contain only some very generic serialization methods which take an HttpContent, assume that it contains JSON, and then asynchronously deserialize that JSON into an ExpandoObject or ExpandoObject[]. Is this specific to view model composition in any way, or should ViewModelComposition be removed from the project name?

it can be called JsonHelper but I prefer to have a finger cut than calling an assembly *Helper 😄
Theoretically it could be merged into ITOps.ViewModelComposition, I split it to avoid making ITOps.ViewModelComposition dependant on Json.Net, this is it.

Regarding the solution and project naming, the one that makes the most sense to me is

Divergent.WebApi.sln

Divergent.WebApi.csproj

the thing is that it's not WebAPI, WebAPI is already an established term that refers, when we talk in the context of .NET, to ASP.Net WebAPI.
It is a CompositionGateway or APIGateway or CompositionAPIGateway. Because it not only an API Gateway, it's an API Gateway with composition support.

Divergent.Website.sln

Divergent.Website.csproj

OK.

Thanks for the explanation regarding the ITOps projects. One thing I still don't understand is why ITOps.ViewModelComposition.csproj has types in a ITOps.ViewModelComposition.Gateway namespace, whereas ITOps.ViewModelComposition.Gateway.csproj also exists. Is it that the ITOps.ViewModelComposition.Gateway types in ITOps.ViewModelComposition.csproj are generic gateway components, whereas the types in ITOps.ViewModelComposition.Gateway.csproj are the ASP.NET Core specific implementations?

I don't understand what you mean by

ITOps.ViewModelComposition.csproj has types in a ITOps.ViewModelComposition.Gateway

In ITOps.ViewModelComposition.Gateway there are just 3 classes that strictly related to how a reverse proxy works in .NET Core, they act as a bridge allowing the composition engine to be hosted in a ASP.Net Core reverse proxy. The composition engine is hosting aware, it just composes.

Regarding ITOps.ViewModelComposition.Json.csproj, how about renaming it to ITOps.Json.csproj?

OK

I should have addressed all the comments @adamralph, please take a look.

rebased on master

Somewhat related to the last point, in ITOps.ViewModelComposition.csproj, there is only one type in each of the namespaces ITOps.ViewModelComposition.Engine and ITOps.ViewModelComposition.Gateway. Should we collapse those two types into the ITOps.ViewModelComposition namespace?

Yeah, we should. I believe I noticed another couple of things that are in the wrong namespace, I'll do another pass.

Did you get anywhere with the JSON casing spike?

No, not yet. It affects a lot of projects in this demo, to keep changes to this PR in scope I propose we revisit as we have this merged. It's something that we can drop but it's also transparent to the logic of the sample, the problem is that changes will be pervasive to all projects, only complicating this PR.

In ITOps.ViewModelComposition.Gateway there are just 3 classes that strictly related to how a reverse proxy works in .NET Core, they act as a bridge allowing the composition engine to be hosted in a ASP.Net Core reverse proxy. The composition engine is hosting aware, it just composes.

That tells me that ITOps.ViewModelComposition.Gateway.csproj contains the ASP.NET Core-specific implementation of abstractions defined in ITOps.ViewModelComposition.csproj. Is that correct? In that case, I would consider renaming ITOps.ViewModelComposition.Gateway.csproj to ITOps.ViewModelComposition.AspNetCore.

Originally ITOps.ViewModelComposition.Gateway was collapsed into ITOps.ViewModelComposition, but then whenever you're using MVC you have available at configuration time all the Gateway configuration extensions that for MVC make no sense, that's the main reason I split the Gateway into its own project, so that its purpose was clear. We can easily merge ITOps.ViewModelComposition.Gateway back into ITOps.ViewModelComposition.

Although, I'm unsure about that, since I see that ITOps.ViewModelComposition.csproj also has a dependency on Microsoft.AspNetCore.Routing.nupkg.

ITOps.ViewModelComposition (the engine) depends on HTTP only, the routing dependency is a convenient way to have all the facilities that the routing provide to analyze HTTP requests, otherwise we would have needed to reinvent the wheel. Luckily routing is not anymore double linked to MVC as it was before, and now is part of the core part of AspNetCore, and the engine is designed for AspNetCore only.

@adamralph addressed most of the changes, and replied to a couple of comments.

All comments addressed.

should this be upgraded to netcoreapp2.0?

squashed into 1 single commit @adamralph. And yes, netcoreapp2.0 can be handled as a separate PR

🎆 🎉


shareViaTwitter : link automatically shortened not work

$scope.shareTwitter = function () {
      $cordovaSocialSharing.shareViaTwitter(content.title, null, content.url).then(function (result) {
        // Success
      }, function (err) {
        // An error occurred. Show a message to the user
      });
    };

On Android, tweet contain full url not shortened.

The plugin has no built-in shortener or anything.. that's up to the Twitter app. But did your total message fit the 140 chars?

In README plugin

Twitter: message, image (other filetypes are not supported), link (which is automatically shortened).

I'm confuse with that. Lenght of message is lower that 140 character

Ah, that makes sense. I've updated the readme - it now indicates this is optionally done by the Twitter client, not something the plugin does.


most of our runtime is spent in to_string()

We should speed this up, not that it really matters. But it will effect our timing performance if we do the time until you actually print things or put in json or anything. I guess most timing metrics wouldn't care for it but it'd be nice at some point.
You should be able to just make a write_child style method that take a formatter and writes to it, and is smart about uncurrying without actually doing any allocations by doing an application_parse() style thing

This is largely fixed by various changes


Update README.md

Instruction must be given as new users can have problems.

Can you explain what happened as the composer create-project will install all the dependencies, as you can see in https://gist.github.com/akrabat/2c9a03d01a1fbf154b102d4ebcb1933d

I am closing this as this project should be used via composer create-project


Add Laravel 8 support

@antonkomarev can you tag a new release? thanks 😃

Thanks for pointing this out! I forgot to push it :+1:

Laravel Ban v4.3.0 has been released.


fix double evaluations...

fix UIBean and its subclasses double evaluations
fix Param tag used with Bean tag double evaluations
(to be continued)

Reference:
https://securitylab.github.com/research/apache-struts-double-evaluation/

Coverage increased (+0.008%) to 49.918% when pulling a7884f9df923fa1e07222d131ab0a413bdd13823 on fix/double_evaluations into 09f969a9bebe31370df64702a61420f14ead6271 on master.

@yasserzamani Will this work with the following code?
<s:iterator value="..." status="stat">
  <s:textfield id="itemId%{#stat.index}" key="items[%{#stat.index}].name" />
</s:iterator>

Can we have some tests for that too?

Thanks for your review @aleksandr-m ! Yes I think it will work. Assume <s:sometag someAttr="someComplexExp".... I don't disturb the evaluation of someComplexExp at all. It will be evaluated as before. But once evaluated for first time, if it's going to go for another evaluation for second time, then I check and validate it again against accepted/excluded patterns. I know it might limit developer in some special not-usual usages but I have to do so if I want to reduce the risk of end-user raw data evaluation by mistake. Furthermore developer always can change her/his design if this breaks it.
For example, I think in your case, in first evaluation we will have e.g. id->itemId2 and key->items[2].name where both will pass accepted/excluded patterns if they are needed to be re-evaluated.
I'll add corresponding test:+1:thanks!

@aleksandr-m as promised, here you are. It addresses your case right?

I have a large concern with using *PatternsCheckers here, they supposed to be used as a border pass control - validating if incoming parameters are allowed or not. Now they have been used internally to validate expression evaluation which can be hard to understand by users. Also there is no possibility to have more restrictive patterns on enter (in interceptors) and looser patterns internally. Another thing is that those *PatternsCheckers are created per interceptor, the same will happen here, but having a large number of tags will consume a lot of memory instating new *PatternsCheckers per each tag.
Having that said I would introduce a dedicated Pattern Checker instance specifically used in tags, with simple one pattern used to verify expression evaluation. It also has to be a Singleton to reduce memory consumption.

Thanks a lot! Clever catches :) I'll reform as suggested  :+1:

@lukaszlenart the thing is they're not completely decoupled. They're dependent to each other. For example currently by default getFoo().name is not accepted (should be changed to foo.name to get accepted) but if developer wants to add it to the accepted patterns then it must implicitly be also accepted for re-evaluations. That being said, even they must be equal. This has no meaning to restrict a pattern to be evaluated again if developer wants to allow, vice versa, no meaning to allow a pattern to be re-evaluated if developer hasn't explicitly allowed it. Evaluation patterns and re-evaluation patterns are naturally same. wdyt?
I think the only remaining issue is memory. wdyt? May I simply define singletons from default accept/exclude patterns classes and use them instead in tags?

Hi @yasserzamani .  This looks like an interesting PR and discussion for sure.  😃
One other consideration may be allowing a choice (by configuration) as to whether to allow re-evaluation or not (at all).
If a developer wants to permit re-evaluation then they could enable a configuration option to do so, otherwise the framework could continue to apply the more restrictive original logic (https://github.com/apache/struts/pull/483/files#diff-cfe644a2b24b492d6835fa1f38e7a770dad354b286cbe6b056a5fe7e80e669caL808-L809).
Also, having an opt-in choice might mitigate some of the potential concerns with additional overhead, since the developer would be making a choice to use the feature.
In terms of having the ability to configure different enter/interceptor accept/reject patterns vs. tag/component accept/reject patterns, there might be hard-to-predict scenarios where one might need some flexibility to change them independently.  Having the tag/component accept/reject patterns default to the same as the enter/interceptor ones, but with a user-configurable override, might be a safer path to take ?

Hi @JCgH4164838Gh792C124B5 thanks! :)
Actually it's not a feature requested, it's a security mitigation. Its main purpose is to mitigate when developer is using %{} or ${} against end user not validated input by mistake. It prevents it only in suspicious cases because reevaluation is naturally needed in Struts and cannot be opt-out.
Moving on your second paragraph, I personally think it's not a new thing. It's a "complement" for current already exist accept/exclude policy. I think it's same and current policy had merely missed this piece. My reasoning: Consider <s:text name="foo[%{index}].name"/>. It's just same as <s:text name="foo[0].name"/><s:text name="foo[1].name"/><s:text name="foo[2].name"/><s:text name="foo[3].name"/>.... My inference: To validate that if it's ok to double evaluate foo[%{index}].name we just need to validate if the result of the first evaluation is already considered as a valid input! Do you know what I mean? I feel they aren't different, the second evaluation is same as the first evaluation regarding validation. e.g. if in some places we had third evaluation, then we could say the result of the second evaluation also must be an already considered as valid before going to third validation and so on.
So I think it's not a new concept, it merely was a missing piece in previous impl. The only issue here is the memory. Frankly what I've never understood yet in Struts is if e.g. OGNL exclusions or Interceptor exclusions are for security so why developer is allowed to override them?! because otherwise I could change current impl to singleton and solve the mem issue:)

@yasserzamani you can use the same default patterns in both cases but I would give user a chance to change them independently. You cannot turn a bean into Singleton programmatically, the scope is defined in struts-default.xml as prototype so each time a new instance will be created. This is fine for interceptors as they are created per stack, each interceptor is a Singleton for a given interceptors stack. This won't work for tags, as tags are re-created per each request and each tag is a separated instance.

you can use the same default patterns in both cases but I would give user a chance to change them independently.

:+1: thanks! @lukaszlenart I think I did so in last commit now. Could you please review?

Let's leave it as is, no better idea so your is good enough :)

NOTE
BTW please use Squash and merge.

I think this ready to be Squash & merged

@lukaszlenart I think so,  yes please


[Usage] Not able to save model weights after Lora Finetuning

Describe the issue
Issue: Lora finetuning with Zero2.json and also Zero3.json. During finetuning, train and validation loss are reducing but when I see the weights in saved model checkpoint, it has only initialised weights.
Command:
Model finetuning is done using: sh finetune_task_lora.sh

Log:
With **Zero3.json**, all weights are same as initialisation:
Vision Lora A mean values: tensor(5.9843e-05, device='cuda:0', dtype=torch.bfloat16)
Non-vision Lora A mean values: tensor(5.1260e-05, device='cuda:0', dtype=torch.bfloat16)
Vision Lora B sum values: tensor(0., device='cuda:0', dtype=torch.bfloat16)
Non-vision Lora B sum values: tensor(0., device='cuda:0', dtype=torch.bfloat16)

With **Zero2.json**, only weights of Lora-B Non-vision values have changed from initialisation:
Vision Lora A mean values: tensor(5.9843e-05, device='cuda:0', dtype=torch.bfloat16)
Non-vision Lora A mean values: tensor(5.1260e-05, device='cuda:0', dtype=torch.bfloat16)
Vision Lora B sum values: tensor(0., device='cuda:0', dtype=torch.bfloat16)
Non-vision Lora B sum values: tensor(**2.1562**, device='cuda:0', dtype=torch.bfloat16)

Screenshots:
Train and Validation loss during Model finetuning in below plot:

I had the same issue. Your checkpoint is, by default, saved in "./checkpoints". Then I ended up just checking the modification times to find the right one, lol. Then, you can merge.
Hope this helps.

Thanks for the reply. I have been passing correct checkpoints folder only. I haven't been using merge script from here as it is throwing error like this:
OSError: LLava_fine_tune/checkpoints/llava-v1.6-7b-lora/checkpoint-159 does not appear to have a file named config.json. Checkout 'https://huggingface.co/LLava_fine_tune/checkpoints/llava-v1.6-7b-lora/checkpoint-159/tree/main' for available files.
I think it is referring to Huggingface for the checkpoint but I have it on my local. Do you know how to fix this?

Thanks for the reply again. Now, I am able to merge the checkpoints. But next problem is: Model is not giving any output after "Assistant: ". Whereas base model is generating some response.


[2.7] Restart submariner-addon pod on managed cluster after restore

This is required for https://github.com/stolostron/backlog/issues/22005
When hub is backed up and restored on backup cluster, the submariner-adodn pods on managed clusters continue to use old hub. To make them use new hub, need restart submariner-addon pods.

/assign @vthapar


fix(item): handle multiple inputs inside ion-item correctly

Pull request checklist
Please check if your PR fulfills the following requirements:

[x] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been reviewed and added / updated if needed (for bug fixes / features)
[x] Build (npm run build) was run locally and any changes were pushed
[x] Lint (npm run lint) has passed locally and any fixes were made for failures

Pull request type

Please check the type of change your PR introduces:

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

What is the current behavior?
Issue number: fixes #15850
What is the new behavior?

Items with multiple clickable inputs position slotted (slot=end) inputs relatively
Items with inputs (checkbox, toggle, ...) and clickables (ion-button, a) add a z-index of 3 for the clickable elements

Does this introduce a breaking change?

[ ] Yes
[x] No

Other information

@brandyscarney Could you please either merge this or close it, otherwise I'm not able to submit another pull-request :)

@DavidStrausz Sure! There are conflicts with master, did you want me to close it to re-open it with conflicts resolved?

@brandyscarney I could rebase on master in my repo and resolve the conflicts if you're still planning on merging this, but as its been sitting around since may I would have submitted a PR for something else now as the user-land workaround for this works well for us :)

@DavidStrausz thanks for the effort, even if we can't appreciate it in production yet.

@DavidStrausz I haven't had time to review the UX portion of this. There were some changes I made awhile back to fix some of the problems with item's clickable elements related to disabled inputs: https://github.com/ionic-team/ionic/blob/1a8b7a4559860b3efa4778a78c905e30f18587bf/core/src/components/item/item.tsx#L130-L142
Here's the commit: https://github.com/ionic-team/ionic/commit/dfa2b13c3a408c6288acc38ce5de480ff04c88fc
I created a Codepen of some of the scenarios listed to see what was still not working and it seems like buttons are the only remaining issue where it isn't working entirely: https://codepen.io/brandyscarney/pen/wvvxjXE?editors=1100
I think it may just need CSS updates for the slots in order to fix the covers, but I need to review the native UX for it.

@brandyscarney I resolved the conflicts and added a fix which also accounts for the remaining issue with ion-button.

This is great! Thank you so much for making the updates, adding tests, and being patient with us. I pushed a few small changes, mostly adding the ion-anchor and button elements to look for as clickables.
I added a small change to account for the problem @lppedd is experiencing with setting the item itself to clickable when there is a input cover, but I think that issue itself is outside of the scope of this problem (will comment on the original issue with more steps).
Once the build passes this is ready to go. Thanks again!

No problem, always glad go help out and improve the framework!
Nice thinking with the additional clickables, I wasn‘t sure how many to add, most probably it will be necessary to add some more over time when people want to place some other elements there.

I published a dev build for this if anyone wants to try it out: 5.0.0-dev.201911132228.c139c9a
It's based on master which includes some Ionic 5 changes since we aren't doing new development on Ionic 4 at the moment. Changes for Ionic 5 can be seen here: https://github.com/ionic-team/ionic/releases/tag/v5.0.0-beta.0


5.1 Version still hasn't resolved the issue either.

Script version
5.1
Describe the bug
YouTube is still blocking you from playing YouTube videos because AdBlock has been detected.

OS : Windows 11, MacOS Monterey
Browser : Chrome

Its because they have added algorithmic detection, they're really going ham at us now.
Any interference from any source without manual user input will trigger it.
That would easily explain how people are randomly getting the popup out of nowhere without any adblockers.

https://github.com/TheRealJoelmatic/RemoveAdblockThing/issues/439


Pelican 4 prevent TypeError

Follow up on issue #174. See https://pelican.readthedocs.io/en/stable/changelog.html#id1

Thank you @iranzo for reviewing it.
Thank you @ashwinvis for your contribution.


"istioctl inspect"

This work item tracks the progress of
Istioctl inspect
https://docs.google.com/document/d/1OnJpw1Hi5vtfnIajOFwvuJPlnPlB2rJeCkRvSVJx9vI/edit#
Initially we will implement this as "istioctl experimental inspect" so that we can get feedback on the output and implementation before finalizing.

+1 would love to see an experimental cmd on this.
@esnible what is the outlook for this in in 1.3?

should this be service level inspect given many istio resources are applied to service not pod?

+1 for using experimental and getting user feedback before finalizing. In particular, it would be useful to understand whether it makes sense to anchor on workload (pod), host (service), or a higher-level service concept (IstioService?).

Much of the needed data is now in the Envoy configuration, in the filter_medata/istio JSON.
If you do kubectl exec $POD -c istio-proxy -- curl localhost:15000/config_dump | jq '.. | . as $parent | .metadata?.filter_metadata? | select(.!=null) | { route: $parent.name, config: $parent.metadata.filter_metadata.istio.config} you'll see output like
...
{
  "route": "inbound|8079|grpc-ping|fortio-echo.default.svc.cluster.local",
  "config": "/apis/networking/v1alpha3/namespaces/default/destination-rule/fortio-echo"
}
{
  "route": "inbound|8080|http-echo|fortio-echo.default.svc.cluster.local",
  "config": "/apis/networking/v1alpha3/namespaces/default/destination-rule/fortio-echo"
}
...

We could clean up that information and produce something human-readable that is very similar to what was discussed in the proposal.  I think we can get a minimal version of this in 1.3.

Totally re-architected to get all information from sidecar:15000/config_dump and pilot:/debug/authenticationz.
The code already validates many Kubernetes Service things, such as port names.  I will add support for "a port 80 service being called tcp" as requested in https://github.com/istio/istio/issues/6892

Still looking for PR reviewers.
I am using this to validate all meshed pods on a cluster using
for ns in $(kubectl get ns -o custom-columns=NAME:.metadata.name --no-headers); do
	for pod in $(kubectl get pods -n $ns -o custom-columns=NAME:.metadata.name --no-headers); do
	~/go/out/darwin_amd64/release/istioctl experimental inspect pod $pod.$ns --ignoreUnmeshed | grep -i warning
	done
done

I designed inspect as an inspection tool but it is proving handy as a validation tool.


starting to integrate ruby 3.1 into the builds

Signed-off-by: John McCrae<EMAIL_ADDRESS>
Description
Testing out a fix for the borked gemspec files
Related Issue

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Chore (non-breaking change that does not add functionality or fix an issue)

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] I have run the pre-merge tests locally and they pass.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.
[ ] All commits have been signed-off for the Developer Certificate of Origin.

Changes were for testing and now not needed


Expansion of supported version range of macOS

Descrpition
If possible, the minimum supported version should be 10.14.

PHAssetChangeRequest in photo-kit is only available in macOS 10.15 or newer.
The minimum supported version could be 10.15, but the next version after 10.15 is 11.0. 11.0 has changes that make some api deprecated, and we decide that the maintenance cost is not worth it to go to 10.15. Fortunately, macOS 11.0 has a very wide range of supported devices, so this will not be an issue in many cases.
See:

macOS versions
macOS 11.0 (Big Sur) supported devices


SpreadsheetTest fails on non-English OS settings

Required Information

[ ] AEM Version, including Service Packs, Cumulative Fix Packs, etc: (irrelevant)
[ ] ACS AEM Commons Version: 4.0.1-SNAPSHOT (latest)
[ ] Reproducible on Latest? yes

Expected Behavior
When running unit tests using e.g. "mvn clean package", all unit tests complete.
Actual Behavior
If your OS locale is Hungarian, the SpreadsheetTest unit test fails with the following error:
[ERROR]   SpreadsheetTest.testSheetTypesAsStrings:176 expected:<123[.]456> but was:<123[,]456>

I'm sure the issue will also happen on other non-English locales that use something other than the dot for decimal separator.
Steps to Reproduce

Set your OS locale to something that uses decimal comma (German or French can work). How to do this will depend on your OS.
Execute mvn test in the root source folder.

The issue is coming from the fact that the Variant class implicitly depends on the default Java locale in multiple places. This behavior makes CompositeVariant and Spreadsheet also depend on the default Java locale, so the assertions in the unit test can either succeed or fail based on an external factor.
How to fix this will depend on whether the locale dependency of Variant is considered a feature or a bug. If it's a feature, the unit test should either make its string-based assertions with dynamically-formatted expected values, or not make those assertions at all. If it's a bug, then Variant should get a way to specify its locale, and this change should be also applied to CompositeVariant and Spreadsheet; then the unit test could specify that it wants an en_US spreadsheet and freely make its US-based assertions.

I'd like to help with fixing this issue, but I'm really not sure which of the two options is better (i.e. embracing dependency on the default locale vs. removing it).

Actually it might make sense to enforce the locale detected in the spreadsheet if possible.  Another point to consider is that we're relying on a library and need to at least isolate if it's a bug in the library or in our code.

Actually it might make sense to enforce the locale detected in the spreadsheet if possible.

I don't think Excel files have any locale saved with them. If I open the xlsx file from the test folder in Excel, I'm just seeing the numbers with my locale's proper decimal separator without any indication that the spreadsheet was created with another separator. I can't see any locale either on the Excel UI, or the public API of POI (the library used to parse the xlsx file).

Another point to consider is that we're relying on a library and need to at least isolate if it's a bug in the library or in our code.

As far as I can tell, there is only one interaction between Variant and POI that relies on the default locale. Variant has a constructor that takes a POI Cell instance as an argument. The constructor eventually ends up calling DataFormatter.formatRawCellContents() to fill the string value of the variant in line 112. The no-args constructor is used to create a DataFormatter, which will implicitly use the default locale for formatting. If DataFormatter was instantiated with an explicit locale, we could control the number, date and time formats that end up used in the string form of the variant. In other words, POI is working fine, but Variant code asks it to rely on the default locale.
One way to fix this would be to add a Variant constructor that takes both a Cell and a Locale (the latter for formatting purposes), but this change would need to "ripple through" CompositeVariant and Spreadsheet to allow client code to specify the locale they expect.

Are we talking here just about a testcase, which only works in a limited number of locales, or is functionality affected as well?
The travis build runs in a combination of multiple languages and countries (-Duser.language=X -Duser.country=Y) (see https://travis-ci.org/Adobe-Consulting-Services/acs-aem-commons), but this does not seem to trigger this test to fail either. Should we run more builds also with different locale settings?

Are we talking here just about a testcase, which only works in a limited number of locales, or is functionality affected as well?

The answer really depends on what you consider the expected functionality. The behavior of the Spreadsheet class depends on the JVM default locale all the time, not just for the test case. This may be expected, but isn't documented as far as I can tell.

The travis build runs in a combination of multiple languages and countries (...), but this does not seem to trigger this test to fail either.

Hmm, I'm pretty sure the test should fail with both German and French. Maybe it has to do with how the tests are run. As far as I remember Surefire forks a new JVM to run the tests, and that new process may not inherit the JVM args passed to Maven. I could successfully force the default locale to en_US and make the test pass locally by setting the _JAVA_OPTIONS environment variable, so I think you should be able to force a failure by setting that variable in Travis. Whether it's a good idea is another question - apparently this variable affects all newly started JVMs, so it may have some unexpected side effects.

Created #1864 as a proposed minimal fix.
Turns out CompositeVariant doesn't need to be touched because it doesn't directly deal with Cell instances. I've opted to add a setter to set Spreadsheet locale since it already has a bunch of constructors and it didn't feel right to add a whole new set of them. It may be worthwhile to introduce a builder for this class instead and make client code more readable.
Please note that I haven't changed the fact that Variant still uses the default locale to convert between dates and strings. I think it would be better design to be explicit about this locale and allow specifying it, but this isn't necessary to fix the current issue.

This was fixed by applying https://github.com/Adobe-Consulting-Services/acs-aem-commons/pull/1882.


Mark the *printf functions with attribute((format)).

See issue #79.

Codecov Report

Merging #80 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master       #80   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            1         1
  Lines          359       359
=========================================
  Hits           359       359

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d3b9846...d8aeaa9. Read the comment docs.

When the format attribute is added, we get a massive amounts of warnings from test_suite.cpp, including:
warning: flag ' ' results in undefined behavior with 's' conversion specifier [-Wformat]
warning: format specifies type 'char' but the argument has type 'int' [-Wformat]
warning: invalid conversion specifier 'b' [-Wformat-invalid-specifier]
 warning: flag '#' results in undefined behavior with 'u' conversion specifier [-Wformat]

Some can just be fixed by explicit casting (and TBH it's a bit surprising that they work... are we saved by the ABI perhaps?) ; others cannot.
What are your thoughts?...
Naturally, this is not my repository, but I've been thinking of merging this into my fork, which already has, merged, the work of some others around here (e.g. warning fixes, glibc %p behavior etc.). I even created a PR for that, but while merging I noticed all those warnings.

When the format attribute is added, we get a massive amounts of warnings from test_suite.cpp

I believe I fixed this in my fork. See joshklod:format-attr for my implementation of this feature.


Drawing is only allowed inside NOTIFICATION _DRAW, _draw() function or 'draw' signal

This error makes absolutely no sense. I have brought this code into my project and it works perfectly fine, except for this error message I get every frame.
Do you have any idea why? The code is calling draw's inside _draw and queue_redraw() is getting called.
This maybe could be related to https://github.com/godotengine/godot/issues/75376 ?
Let me know if you have any idea, thanks!

What is your Godot version?

Im in Godot 4.1, However the demo you provided works just fine in 4.1.
However in my game I am getting this error every-frame, despite it actually drawing succesfully.
My Tree Setup:

Video Example that it is working:
https://github.com/daveTheOldCoder/godot-radial-progress-indicator/assets/94399429/c525773d-d498-4243-a439-0ff056fb7020
I have been trying to re-create a minimal replicating project to no success yet. I may have to just start removing things from my main project  one-by-one.

Thanks for the additional information.
I don't know a lot about custom drawing in Godot. I created this Godot 4 plugin by making the minimal changes to a Godot 3 plugin.
I just tested your suggested change in Godot 4.3-beta1, and it worked correctly. I'll repeat the test in some other 4.x versions, and if it works correctly, I'll submit an update for the plugin.

Do you think it would be better to remove the _process() method, and leave the calls to _draw() and queue_redraw() in the setters? In my test, that works too. That has the advantage that the drawing calls occur only when needed, rather than every frame.
Could you test that and see if it removes the error?

I would leave it in the _process() function since that guarentees it is only called once per frame.
In my implementation I am updated both the progress and the max progress each frame which which cause it to be called more than once.
I do agree there is some performance difference involved but I am not sure how much.
Anyways, here it is in action as a stamina/health meter. Thanks again !
https://github.com/daveTheOldCoder/godot-radial-progress-indicator/assets/94399429/900f8a91-9c81-4460-8a7b-88989b212c26

That makes sense.
I removed the setters, since their only purpose was to make the drawing calls.
I've committed the change here, and will shortly be submitting the new version to the Asset Library.
Thanks again for your help.

New version is 2.1.1:
https://github.com/daveTheOldCoder/godot-radial-progress-indicator/releases/tag/2.1.1
https://godotengine.org/asset-library/asset/2193


Get tests

Can we leverage scanpy's tests?

Download data instead of committing it, like sklearn does.

Ok, we have at least very basic tests now in 7fcf3fd903093c322094dc679a456dd8319dc0f6.

Need aggregation test.


The command 'nuxt build' finished but did not exit after 5s

I cannot find an open issue for this problem but it still exists, even in a newly created nuxt project, so there is little chance that it's caused by timers or something else in my components:
⚠ Nuxt Warning                                                                     │
   │                                                                                │
   │   The command 'nuxt build' finished but did not exit after 5s                        │
   │   This is most likely not caused by a bug in Nuxt.js                                 │
   │   Make sure to cleanup all timers and listeners you or your plugins/modules start.   │
   │   Nuxt.js will now force exit                                                        │
   │                                                                                │
   │   DeprecationWarning: Starting with Nuxt version 3 this will be a fatal error

Versions

nuxt: ^2.14.6 latest
node: 14.9.0

Reproduction

Follow the guide to create a minimal project.
I've created one here: https://github.com/gera2ld/nuxt-bug
Steps to reproduce

Create a minimal nuxt project according to the guide or clone my minimal project: https://github.com/gera2ld/nuxt-bug
yarn
yarn build

What is Expected?
The build process should exit after finishing and should not show the warning.
What is actually happening?
The build process hangs and shows the warning above.

Something really weird happens, everything works now without any change to the code.
I guess there might be something wrong with the Internet for some time, sockets hung and prevented the process from exiting.

R


add timezone info

Hi, appreciate the attempt. Always nice to see people jumping in to contribute!
but you're trying to add something that's already there 😅

On every "card" there's a last updated UTC timestamp

ack, thanks.


Host GHSL GeoJSON for developers / users

[ ] Host GHSL GeoJSON on GitHub or S3?

I created the bucket encompass-public-data for this purpose.

make discoverable in Github

The updated README.md doesn't contain any references to data sources -- does it make more sense to put this in the app's Methodology dialog?

This is now part of the Methodology dialog.


add docker env for go installation

could add description for docker environment in Chapter 1.1 https://github.com/astaxie/build-web-application-with-golang/blob/master/zh/01.1.md

install the docker Go environments? I will introduce the docker in my new book: https://github.com/astaxie/go-system-programming

No, prepare the Go environment using docker.
$ docker run -it -v $PWD:/go/src golang bash
go ...
On Sun, Jan 4, 2015 at 4:06 PM, astaxie<EMAIL_ADDRESS>wrote:

install the docker Go environments? I will introduce the docker in my new
book: https://github.com/astaxie/go-system-programming
—
Reply to this email directly or view it on GitHub
https://github.com/astaxie/build-web-application-with-golang/issues/424#issuecomment-68624780
.

--
True software development embraces consistent inconsistency.
blog: http://larrycaiyu.com/blog (en), http://larrycaiyu.com (chinese)


Polygon rendering clipped when opacity is set

Polygon rendering is clipped when opacity is set
When opacity is set it seems that polygon rendering cuts off the left side of the polygon when it is a rhombus, such as in the example below.
When "opacity" is 1 it renders correctly, whereas any other value chops of the left side.
Example code
This example was created manually in the XmlOutputTester application in "Samples".
        private void Form1_Load(object sender, EventArgs e)
        {
            float opacity = 0.9F;
            float w = 800;
            float h = 600;

            svgDoc = new SvgDocument
            {
                Width = w,
                Height = h,
                ViewBox = new SvgViewBox(0, 0, 15000, 3000),
            };

            var group = new SvgGroup();
            svgDoc.Children.Add(group);

            var p1x = new SvgUnit(6112);var p1y = new SvgUnit(0);
            var p2x = new SvgUnit(6936);var p2y = new SvgUnit(1099);
            var p3x = new SvgUnit(7124);var p3y = new SvgUnit(1099);
            var p4x = new SvgUnit(6299);var p4y = new SvgUnit(0);

            var points = new SvgPointCollection
            {
                p1x,p1y,
                p2x,p2y,
                p3x,p3y,
                p4x,p4y
            };

            group.Children.Add(new SvgPolygon()
            {
                Points = points,

                Fill = new SvgColourServer(Color.LightGray),
                Stroke = new SvgColourServer(Color.Black),
                StrokeWidth = 1,
                Opacity = opacity
            });

            var stream = new MemoryStream();
            svgDoc.Write(stream);

            // xml is correct
            textBox1.Text = Encoding.UTF8.GetString(stream.GetBuffer());

            // BUG: ...but image is drawn incorrectly
            pictureBox1.Image = svgDoc.Draw();
        }

Example Screen Shots
Opacity=1

Opacity=0.9

Just wanted to add that setting FillOpacity on the other hand does not produce the same issue.


Replace import status elements with displaying status on buttons

Existing behavior:

Click on async button
Import status element near button changes state to "Loading"
Element changes state to "Success" on operation success or "Fail" on operation failure

New behavior:

Click on async button
Button is disabled, text changes to describe action (eg. "Updating...")
Button changes color and re-enables on operation success or failure, text changes to reflect result

Completed in 3.2.0


Ensure consistent margins

See, for example, the Checklist page. This looks pretty bad:

The code blocks also have the wider margins:

See, for example, the Checklist page. This looks pretty bad:

@holly-cummins What do you want in this page?


Vibration notification settings do not get saved

Summary
Vibration notification settings do not get saved, still getting vibrations when setting turned off.
Environment Information

Device Name: LG Nexus 5X, Huawei Mate 9
OS Version: 8.1, 7.0
Mattermost App Version: 1.8.0
Mattermost Server Version: 4.8.1

Steps to reproduce
Turn off vibration in notification settings and receive any notification.
Expected behavior
See above. I don't think this really needs more thorough explanation
Observed behavior
Another user observed a change in the vibration pattern when turning the setting on and off.

Hi @svelle, thank you for reporting this!
I will try the repro steps and report back here my findings.

Quick update on this - I realized that this is on Android, so I will ask someone on our team with Android to test this.

Hi @svelle, a team member tested this and they were unable to reproduce the issue. My recommendation is to upgrade to 4.10 or another option is to test on our nightly build server to see if you can reproduce this there. Would one of these options sound good?

Okay. I'll try that later today and report back to you. I'll also see if I
can take a short video of the bug.
On Wed, May 23, 2018, 4:18 PM amyblais<EMAIL_ADDRESS>wrote:

Hi @svelle https://github.com/svelle, a team member tested this and
they were unable to reproduce the issue. My recommendation is to upgrade to
4.10 or another option is to test on our nightly build server
https://pre-release.mattermost.com/signup_user_complete/?id=f1924a8db44ff3bb41c96424cdc20676
to see if you can reproduce this there. Would one of these options sound
good?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/1688#issuecomment-391364088,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHLYliyN13hNhBtzYQxWkqryhBWJtQ9Rks5t1W-dgaJpZM4UHh3Q
.

Sorry for the late reply. I got unexpectedly busy last week. I did some more checking and could confirm the same behavior with the mobile-app on a Xiamoi running Lineage 15.1 (Android 8.1).
I also experienced this behavior using different Clients, with the same latest app version, with different Servers (4.8-4.10). I think this could also be an issue with the react-native implementation of the vibration interface not just with the vibration setting in mattermost-mobile.
@amyblais did the colleague that tested this have a Samsung phone, by any chance? Because we tested this on a Galaxy Note 8 and that was the only phone so far that did not experience this behavior.

Hi @svelle! The colleague tested on a Galaxy S6 as well as on Moto G4 S, but he is not able to repro this.
As next steps, can you help:

Confirm if you are seeing the issue only on notifications while the app is in the background or does it also happen when the test notification is pushed from the settings area?

Double check that the affected devices do not have the ringer set to vibrate? If there is an alert sound selected (as is the default), the device would still vibrate if that's what the ringer is set to.

The first point I can definitely confirm. It happens when testing and also with real notifications out of the app.
But that second point might be it, it stops vibrating as soon as I go into regular ringer mode (not vibrate). I could only check this on my Nexus right now. But will report back when I can test on other devices tomorrow.  Even though I find that behavior a bit irritating seeing as I actively turned off the vibrations. But I guess that's something out of our hands cause it's probably an Android Notification thing.
I did run into another bug, though. When selecting the ringtone and selecting 'Default (None)' It plays a ringtone and continues to play that ringtone until I close the app. Closing the ringtone selector does not stop the ringing. I'll see tomorrow if this also happens on other phones and then open up another issue.
If I can confirm this with the other phones tomorrow I guess I can close this. Unless there is a way to really make the notifications silent even in Vibrate mode because in my opinion this should be expected behavior when (de-)activating such a setting.

okay, after testing this out on some other devices this sure seems like an android "bug" and not a mattermost bug so I'll close this. Thanks for all the help in figuring this out @amyblais !


Callback for custom Field when rendered

So basically what I want to do is to add a bootstrap tooltip for a text field and it requires to call methods on elements that are attached to DOM.
I know that I can listen to onRefreshed on the grid level, but it would be much better to do it on the custom field level (so I don't have to remember to re-initialize tooltips for all tables where I use this custom field).
Thanks.

Please, checkout this issue https://github.com/tabalinas/jsgrid/issues/72
I guess setTimeout should do the trick.

Thank you for the suggestion, I'll use it for now.
IMO it would be good to have some kind of callback available for these cases.

Yes, it sounds reasonable. The problem is: currently data appended by rows (not cells) so it's easy to add a callback for rendered row, but then I don't see big difference between using onRowRendered and onRefreshed. Introducing a callback for cells requires some artificial firing of callbacks, when a row is rendered, and this may cause performance issues for large grids. Let's keep it like this for now and see maybe at some point we'll introduce it in the future.


How to make Traefik issue SSL certs

Hey, I want to connect a domain to a nomad service. How can I make traefik issue an SSL cert?
Would be great if you have some code lying around that can help.

hey @nilansaha, sorry I missed this issue.
Let me double-check if I'm getting this straight:
you want ANY certificate created for your service (aka a self-signed or locally signed cert) or you need a VALID (say letsencrypt signed) cert created for your service?
Let me know, I'll do my best to help you ;)

All good. I figured it out.
FYI, I meant a letsencrypt signed certificate for my service.

cool! if you think it may be useful for others, may I ask you a patch to this code?
I'm sure others had your same issue!
thanks!


For a small percentage of HTTP2/3 Envoy doesn't read backend's responses and returns 504 timeout

Description:
We use emissary-ingress 3.5.2 with varnish that caches static content. It looks like Envoy can't read responses from backends for a small percentage of traffic (around 0.002%) when backends properly generate responses.
What's common in such scenario is that the client browser issues many parallel requests for objects.
I'm also attaching Wireshark screenshot where you can see that the backend generates response but envoy refuses to read it.

Can you provide some details about the configuration for that Envoy? (e.g., is this using both h2 and h3? is health-checking being used?)
cc @alyssawilk

Can you provide some details about the configuration for that Envoy? (e.g., is this using both h2 and h3? is health-checking being used?)
cc @alyssawilk

Can you provide some details about the configuration for that Envoy? (e.g., is this using both h2 and h3? is health-checking being used?)
cc @alyssawilk

@alyssawilk - I've checked it and it response_code_details shows: response_timeout.
As a workaround for this problem I've tried to add 3 retries to upstream, but it doesn't seem to work.
The configuration for envoy is:
                            "route": {
                              "cluster": "cluster_varnish_least_request",
                              "prefix_rewrite": "xxx",
                              "priority": null,
                              "retry_policy": {
                                "num_retries": 3,
                                "per_try_timeout": "1s",
                                "retry_on": "5xx"
                              },
                              "timeout": "20.000s",

In this case durations shows 1 or 4 second, response_code_details shows upstream_per_try_timeout - it doesn't look like it's retring

I guess without a trace I'm not sure how you can tell it's an Envoy problem and not an actual upstream timeout?


world.removeCollider fix

Calls to World.removeCollider currently fail because the wrong method on physics pipeline is invoked. This PR fixes the call.

Thanks!


CIF-1907 - Build navigation component based on WCM Core Component

added navigation component v2
added unit tests
updated examples library
updated IT

Related Issue
CIF-1907
How Has This Been Tested?
Manually, unit tests, ITs.
Screenshots (if appropriate):

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have signed the Adobe Open Source CLA.
[X] My change requires a change to the documentation.
[X] I have updated the documentation accordingly.
[X] I have read the CONTRIBUTING document.
[X] I have added tests to cover my changes and the overall coverage did not decrease.
[X] All unit tests pass on CircleCi.
[X] I ran all tests locally and they pass.

Should we have this PR on the 2.0.0-ee branch?

In principle it should break anything and must be backward compatible so I don't really it looks fine to me as is too.
(Who started using navigation v1 can keep using it just fine.)


[full-ci] Bump reva

Bump reva to test https://github.com/cs3org/reva/pull/4117

@aduffeck I guess this can be closed?


Grant permission to generated binaries under openj9.test.sharedClasses.jvmti/bin/native

Grant chmod 755 to generated binaries under openj9.test.sharedClasses.jvmti/bin/native
This is required for z/OS where the copy task doesn't retain permission bits of the generated binaries.
Related : openj9-openjdk-jdk11-zos/issues/738

Signed-off-by: Mesbah-Alam<EMAIL_ADDRESS>

Test : Grinder/17352


Use the default fg for STYLE_NORMAL

Using "white" explicitly causes problems on terminals with a light
background (e.g. Solarized White): just use the default fg.
I've not changed STYLE_HIGHLIGHT, but there Fore.RESET + Style.BRIGHT
might be better, too.

:+1:
Thank you, @blueyed!


device_reference not stripped before passing values to placeholder actors in some cases

I've faced a bug that I believe is caused by thrust::device_reference<T> in some cases not being stripped to T before being passed to function objects that are composed using placeholders (actors?). The minimal example I could come up with:
#include <thrust/device_vector.h>
#include <thrust/inner_product.h>
#include <thrust/functional.h>

void foo()
{
    using namespace thrust::placeholders;

    thrust::device_vector<float> v1, v2;

    thrust::inner_product(v1.begin(), v1.end(), v2.begin()
        , 0.0f, thrust::plus<float>{}, _1 + 1.0f);
}

Here, the problem is related to the last argument (BinaryOp2). The same can be demonstrated with this function object:
struct erroneous_thing
{
    template <typename T>
    __host__ __device__
        T operator()(const T& lhs, const T& rhs) const {
        return lhs + rhs;
    }
};
//...........
    thrust::inner_product(v1.begin(), v1.end(), v2.begin()
        , 0.0f, thrust::plus<float>{}, erroneous_thing{});

Somehow erroneous_thing is instantiated with T being thrust::device_reference<float>, while it should be float, resulting in a compile error. I haven't faced the problem with any other algorithm I've used; thrust::transform properly strips reference, as well as reduction part of inner_product (e.g. passing erroneous_thing{} as a first parameter is fine).
CUDA toolkit 10.2.

Compiler Explorer link: https://www.godbolt.org/z/-8gpEp

Fixed in #1237.


Modify DNS considerations

Hi,
This part is not clear I think : "Use the Azure Active Directory (Azure AD) domain controllers deployed on the Hub (described in Identity considerations) as name servers."
There is no such thing as Azure AD domain controllers, what is meant here is domain controllers deployed in Azure but it doesn't "belong" to Azure AD.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b0737988-cef2-4848-4584-cef58e823f94
Version Independent ID: 2540a5be-f09f-49c6-da17-04b2ae2284c2
Content: Concept - Integrate an Azure VMware Solution deployment in a hub and spoke architecture - Azure VMware Solution
Content Source: articles/azure-vmware/concepts-hub-and-spoke.md
Service: azure-vmware
GitHub Login: @shortpatti
Microsoft Alias: v-patsho

@DufferFromParis
Thanks for the feedback! I have assigned the issue to content author to check and update the document as appropriate.

@DufferFromParis
Thanks for the feedback! I have assigned the issue to content author to check and update the document as appropriate.

@shortpatti
Can you please check and add your comments on this doc update request as applicable.

I'll work with the engineering team to clarify this more :)

@shortpatti
Can you please check and add your comments on this doc update request as applicable.

I'll work with the engineering team to clarify this more :)

Updated content.
#please-close

Updated content.
#please-close


x/text: tag new release

What version of Go are you using?

$ go version 1.18

Does this issue reproduce with the latest release?
yes
Does this issue reproduce with go 1.17.8?
no
What operating system and processor architecture are you using (go env)?
go env Output
GO111MODULE=""
GOARCH="amd64"
GOBIN="/home/joris/go/bin"
GOCACHE="/home/joris/.cache/go-build"
GOENV="/home/joris/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/joris/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/joris/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.8"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4031452016=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I was trying to create translations for my Go (fiber) project, at first I thought this was an issue related to Fiber.
So I created a new empty project, started all over (without Fiber), and got the same error.
The tutorial I was following: https://www.alexedwards.net/blog/i18n-managing-translations
Then I realised that I've updated Go last week, so I downgraded my Go version to 1.17, and everything worked again.
What I saw when I my Go version was at 1.18:

panic: in crypto/sha256.Sum224: cannot convert slice t0[:] ([]byte) to *[28]byte

How to reproduce?
This is the repo for my project: https://github.com/Joris1Jansen/SimpleNotesApi/tree/GOLANG-ISSUE_WITH_GOTEXT
Please, don't look at the complete mess of my code, this is a project I'm trying out all new things I want to learn.
The current version in go.mod is 1.17, so everything should work now.

go generate ./locale/locale.go

Should return something like:

de-DE: Missing entry for "Welcome!".
fr-CH: Missing entry for "Welcome!".

BUT, if we upgrade to go version 1.18

go generate ./locale/locale.go

We get a big output, but the main isseu:

panic: in crypto/sha256.Sum224: cannot convert slice t0[:] ([]byte) to *[28]byte

I hope that this is enough information, this is my first time opening an issue/bug. So feel free to provide me with any feedback to learn from for the next time.
Cheers

please include the go env output

Looks like gotext was installed using @latest, which is<EMAIL_ADDRESS>and requires<EMAIL_ADDRESS>CL 353350 updates it to<EMAIL_ADDRESS>to pick up the required updates (latest is x/tools@v0.1.10), but there hasn't been a release since
cc @mpvl @golang/release

Tagging a new x/text release is possibly blocked by #47566 /  #45093 (CL 304029 ) (?)

The above were fixed by https://go-review.googlesource.com/c/text/+/304029, I think.
I think we're all good to do a release soon? cc @rolandshoemaker

@mvdan @rolandshoemaker   x/text has had 2 releases (v0.3.8 and v0.4.0) since the original post, I think this issue is good to close.

Agree. Closing then.


Allow running builds sequentially

In some settings, memory/disk space constraints can make running the builds in parallel not feasible (especially when there are many tags/branches). This PR adds an option to Driver to run the builds sequentially. One could also think about surfacing this option to the CLI.
Note that I am not very familiar with asyncio, so if you want to incorporate this maybe you want to use a more idiomatic way instead of the loop over gather.

Thank you very much for this PR! I did some minor refactoring (see my commit). Would you mind adding a note to docs about this option?

One could also think about surfacing this option to the CLI.

Sounds like a good idea.

Nice, thanks for the refactoring! I will add the docs and take a closer look at the CLI.

I have added the docs and slightly modified the existing text. I also added the command line option --sequential to set SEQUENTIAL from the CLI. Please modify as you see fit.

I just noticed that I overlooked updating the README, I will add those changes soon.


Added a very good question from leetcode

Remove Nth Node From End of List

Remove Nth Node From End of List

Added good questions from leetcode


Shooter game

Description
Here is the game , i’m still a junior developer so the code can be a bit messy and poorly optimized , and the game is kinda heavy in weight sorry for that 🙏🏽
Fixes:  #425
ATTACH SCREEN-SHOTS / DEPLOYMENT LINK
The game is already deployed in this link ⬇
https://jackson93150.github.io/JsGame/

After pressing E button, no function is working?

Rest is fine @Jackson93150, Game is Amazing!

you need to be on the new weapon box and press e and game will continue

@VidhiBhatt01 yes it was a project that i made this year for my video gaming programmation course .
You can see my progress in this project on the original repository here : https://github.com/Jackson93150/JsGame
And i took a base from a similar game that i programmed past year but in Rust here : https://github.com/Jackson93150/RetroRunner-

@Jackson93150 the game is quite impressive can you do some changes
w->up
a->left
d->right

@arpit456jain you mean to change to QWERTY so up -> w , left -> a , down -> s and right -> d ?

well it is changed now 👍🏽

@Jackson93150 ur project is very heavy can u decrease the size of images take help from mentors

https://deploy-preview-427--amazing-js-projects.netlify.app/2dshootergame/

@arpit456jain i resized and compressed from 60+mb to 23mb is it enough ?


data['kind'] missing & unexpected red agent behaviour

Hi,
I have modified the environment to include additional nodes / vulns / attack vectors, keeping things mostly the same with respect to environment generation; graph = graph.add_edge[(s, t), (port)] which models fine and AIGym seems happy with the change. The only difference in network generation is that I am using an erdos-renyi generator instead of the nx.stochastic_block_model included in environment_generation.py
The issue is that when running the simulation, i'm met with two unexpected behaviors;

KeyError: 'kind' from commandcontrol.py.

This appears to fail randomly on certain nodes, but with no obvious pattern. For example, one play-through fails when data['kind'] from node 6 to node 3 is missing (default nodes, not ones I have added) which in-turn ends the simulation. E: Just to add, I guess i'm asking if there are any instances where a node wouldn't be assigned an EdgeAnnotation or is it likely something I've done that has broken something / not accounted for a change. This only seems to be an issue with the rendering of the network in network_as_plotly_traces(_) in cyberbattle_env.py and doesn't appear to affect the game.

Red agent can try to exploit RDP / SMB in targets that it shouldn't be able to

It seems that the attacker can try to exploit nodes it shouldn't be able to see. In the network, one node is selected to be a pivot point between other networks, this node has a single vulnerability allowing the pivot, with firewall settings set to block all. The red agent still appears to be able to scan this node for SMB / RDP from any node, and assigns them each EdgeAnnotation.REMOTE, which shouldn't be considered valid moves.
Any advice on where to look would be greatly appreciated,
Thanks

@Dawn-RL Are you hitting these bugs after making your changes or are you also able to repro on the main branch?
Could you provide a small repro? That would help us investigate. Since the bug is intermittent you may have to wrap your repro in a loop to run until the bug surfaces.
Also, if the two bugs are unrelated, I suggest you file a separate bug for the second one.

Hi Bluma,
Thanks for the quick reply! I have added a repo: https://github.com/Dawn-RL/CBS_Bug that demonstrates the second issue. I believe both issues are connected but cannot confirm. Jupyter notebook was too big to render properly so have included 3 screen captures showing what the output is. Happy to provide further info if needed.
Thanks

@dawn-RL If you cannot repro the bug on our main branch then it might be caused by your own changes. I suspect it's because you are adding nodes to the network graph using h1.add_node without specifying the 'data' field. See here to see how it's done in the code:
https://github.com/microsoft/CyberBattleSim/blob/33fd6643a76ec4deca771b01f3a757710649042c/cyberbattle/simulation/model.py#L378
You may want to consider forking the repo so you can point us to the commit with your changes in the future. Since your current repo is not a fork we cannot easily locate where the changes were made in our code base (the line numbers are not always helpful due to code drift). (Or alternatively you can share a diff file with the hash of the base commit)

Hi Blumu,
Fork can be found here with 2 branches, main and second_bug_branch.
Main is a fresh fork of your repo, showcasing that in 'notebooks/Random_Network_Kind_Error.ipynb' we get the data['kind'] error on new installation. This displayed the error on the first run, then ran 3 times without error, and finally the error came back on the fourth trial.
Second_bug_branch includes 'notebooks/Random_Network_Invalid_Move.ipynb' which showcases the invalid moves being made by the agent.
Hope this helps!
Cheers

@Dawn-RL Thanks for doing this, it's helpful! I just took a quick look at it. I think bug 1 may be located in function:
https://github.com/microsoft/CyberBattleSim/blob/9691fa83e2b2012fe81e2e7a3e9db19efe1a6e86/cyberbattle/simulation/generate_network.py#L87
When the environment graph gets constructed, it is first initialized by copying the traffic graph using nx.relabel_nodes. After that the nodes data gets initialized using edge information from the traffic graph. The problem is that the edges that get copied from the traffic graph should then be removed the environment graph before being returned. This can be done with a call to clear_edges at the end of the function.
As for bug number 2, I recommend opening a separate issue for it.

@Dawn-RL Could you try branch wiblum/48 and let me know if it fixes issue 2.?

Hi Blumu,
Looks good so far, much appreciated!
Cheers


fix incorrect code style

Please provide a description of this PR:
Fixed incorrect code style.
To help us figure out who should review this PR, please put an X in all the areas that this PR affects.

[ ] Ambient
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Please check any characteristics that apply to this pull request.

[X] Does not have any user-facing changes. This may include CLI changes, API changes, behavior changes, performance improvements, etc.

Hi @UramnOIL. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@UramnOIL: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
886bac5407580d2c5578d12f861db449493d9a93
link
true
/test release-notes

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest
@UramnOIL You still need to satisfy the CLA requirements.

/retest

@ericvn I've agreed the CLA, is there anything I still have to do?

@UramnOIL No. The test is now showing as passed.


Add many brands to brands.txt

I have personal experience with all of these brands, have bought their products on Amazon, and have confirmed they all have a website and have been around for years. Additionally, I had a good experience with each product. I have confirmed that products are actually sold on Amazon with these spellings.
For about 25% of these brands I've had some contact with either the Amazon seller or the manufacturer and had a positive experience. Here's what I bought from each of them:

Acqua Panna: Bottled water
Airpura: Air purifier
Amazing Abby: Hard plastic dishwasher-safe cups
Antigravity Batteries: Car battery
Berkey: Water filtration system
Butter Buds: Substitute butter food product
CalDigit: Laptop dock
Califia Farms: Oat milk
CyberPower: Uninterruptible Power Supplies
DALI: Headphones
Drano: Sodium hypochlorite drain cleaner
Dyson: Air purifier and fan
Fenix: Flashlights
FIJI: Bottled water
First Alert: Smoke detector
GearIT: Quality Ethernet cables
HyperX: Headphones (subsidiary of Hewlett Packard)
iFixit: Repair tools and guides
King Size: clothing
Level: Smart devices
LiCB: Disposable batteries
Listerine: Mouthwash
MG Chemicals: Pharmacy-grade alcohol
Meraki: WiFi Access Points (subsidiary of Cisco)
MxVol: Remanufactured laser printer toner (mvtoner.com is their website)
Nespresso: Coffee maker
Nordic Ware: Cast-iron waffle maker
Nu Calgon: HVAC evaporator coil cleaner
Paper Mate: Office supplies
Parodontax: Toothpaste
Phanteks: PC case
Plugable: USB hub
Polaroid: Fixed Typo - Pretty sure there isn't a legitimate company with the name "Poloroid" but "Polaroid" makes analog cameras, film, batteries, etc.
Purell: Hand sanitizer
Red Duck: Taco sauce
Ryobi: Power tools
Skechers: Fixed Typo - Pretty sure there isn't a legitimate company with the name "Sketchers" but "Skechers" is a major shoe manufacturer.
Smith & Nephew: Medical supplies
Speck: Smartphone cases
Triple Paste: Medical supplies
Vashe: Medical supplies
Vertiv: Uninterruptible power supplies
VTOMAN: Battery-powered car jumpstarter and tire inflator device. This brand has the format (all caps) and randomness of the fake brands we're often used to, but I looked them up and they have their own storefront, have apparently been around since 2014, and their products get very good reviews from legitimate testers like Project Farm on Youtube. I have been happy with one of their car jumpstarter products, and it doesn't appear to be a simple "re-label" of an existing product.
WypAll: Paper towels
ZEISS: Eyeglasses lens wipes (they make a lot of other stuff too; big multinational corporation)
Zojirushi: Small kitchen appliances like hot water kettles (sold mostly in Japan but people import them and put them on Amazon)

The whole concept of remanufactured toner cartridges is a bit eyebrow-raising in general, as it's hard to verify they didn't give you just a tiny bit of toner to get through the 30 day return period, kinda like those $15 fake SD cards that claim 1 TB of space but just throw your data into the ether after 16 GB :).
But, that's neither here nor there, as (1) I've had a good experience with MxVol's cartridges so far, at a fraction of the price that HP charges (theirs are true highway robbery, even if the "quality" is guaranteed by a big corporation); and (2) they do have their own website, even though it isn't called "mxvol.com".
There don't appear to be any remanufactured toner cartridge brands willing to "fly by the light of day" too openly, at the risk of being sued by someone like HP, probably. But if you get what they advertise, you can save hundreds of dollars per toner refill compared to HP, which sells one of the most expensive substances on Earth in the form of their color toner.


fix: env vars from secrets and configmap + values.schema update

This is needed because environment variables can be defined from configMaps and/or Secrets.
Example:
extraEnv:
- name: KAFKA_SSL_TRUSTSTORE_PASSWORD
  valueFrom:
    secretKeyRef:
      name: apicurio
      key: user.password

This currently fails with:
Error: values don't meet the specifications of the schema(s) in the following chart(s):
apicurio-registry:
- registry.extraEnv.0: value is required
- registry.extraEnv.0: Additional property valueFrom is not allowed

Thank you, will be abailable in 3.2.1
https://github.com/eshepelyuk/apicurio-registry-helm/releases/tag/3.2.1


Adding snippets for ergo language

Fixes  #43
This Pr focus on adding snippets for Ergo Language the same as Concerto language. This is an initial pr. I will keep adding the snippets into this branch :)

Due to prettier configuration, there is some change in package.json and concerto.json but if anyone have any  problem I can safely remove these all changes. But from my point of view since we are living this era we must have prettier configuration.

@dselman Thanks for the clarification. I will revert all the reformat files. Thank you :)

Merged to master


Open one substream for each notifications protocol

In #4284 we introduced a new API for the networking crate, where one can register a "notifications protocol". The implementation, however, was simply translating notifications into old messages and sending them the same way as before.
This PR is the next step.
In addition to the current unique substream with the remote, we now also try to open a separate substream for each notifications protocol that has been registered.
If the remote doesn't support the new protocol, we don't care. If the remote does support the new protocol, then we use it to send our notifications.
(Note: connections that are already open when a new protocol is registered will not use the new protocol. This really should be fixed eventually by forbidding registering new protocols after the initialization process (cc #4587 ).)
Right now the only protocol concerned is GrandPa. I went with /paritytech/grandpa/1 for the protocol name.
cc'ing @rphmeier @andresilva
New protocol description
The new substreams have the following properties:

They are unidirectional w.r.t notifications. If A wants to send notifications to B, it needs to open a substream. If then B wants to send notifications to A, then it needs to open its own substream for that. It is encouraged, but not mandatory, for B to open a substream with A if A opens one with B. The reason is to avoid collisions if both sides want to open a substream at the same time.
After A opens a substream with B, it starts by sending an initial message. B then needs to send back a handshake message. A can then start sending notifications. B can refuse the substream (if it is full) by closing the substream without replying back. This handshake isn't accessible from the public API yet.
It is a protocol error to open another substream for a given protocol while one already exists.

Backwards compatibility
This PR is fully backwards compatible with the current network.
Also note that if we receive a notification the "old way", we still accept it even if the new protocol is in use.
Right now the legacy protocol is still the only way to perform syncing and light client requests, and therefore we make the presence of the legacy substream act as "the authority".
In other words, we are considered "connected" to a node if we have a legacy substream open with a node, then we are considered connected for all purposes. The new substreams exist in parallel but are optional. Also note that it is not possible to open a new substream without opening the legacy substream as well, but this will change in the future.
However nodes that have this PR will immediately talk to each other using the new protocol. It should therefore be strongly tested before it is deployed "for real" on Kusama.

I just remembered that at the moment GRANDPA uses the legacy substream handshake to determine whether the remote is a full node or an authority.
This "role" should instead be passed as part of the handshake. Before we merge, I will expose this handshake API and make it pass the role for GRANDPA.


How to restrict the use of keywords go ?

Hi, I have a doubt. When I develop it with my co-worker, I don't want anyone to use the go func() directly. how do i limit them ？

Sorry I missed this issue being opened. This must be handled by code analysis. A linter could provide a warning. If you look at the codemod directory, I have some source code analyzers that auto-convert to use go-recovery. So on a CI server these could be ran to produce a warning (I forget if there are options for that or if you would have to modify and run a git dirty check).
Catching some errors is better than none though, so I don't see this as a reason to not use this library where you are able to.


feat(apps/prod/jenkins/release): increase agent cap from 150 to 200

current max is over 100, but cluster usage is 30%

/hold

/lgtm


Images built from Dockerfiles are 2x larger than they need/should to be

Problem
After building images using Dockerfiles in this repository, according to instructions here: https://github.com/opea-project/GenAIExamples/blob/main/ChatQnA/docker-composer/xeon/README.md
And investigating their huge download & image sizes, I found out that most of that is due to unwanted Nvidia libraries, pulled in by sentence-transformers => torch Python module dependencies.
To make matters worse, those deps are specified in Dockerfile so that container manager cannot de-duplicate layers on which that content is.
Solution
Better would be to build such images in multiple steps.  First building intermediate image(s) that contains common components like sentence-transformers and its huge deps, and then using that as base for all the images that need those components.
That way Docker and image registries would need to transfer & store that huge (in my case unused) content on disk only once.
Longer term solution
In longer term, I think upstream Torch project should be encouraged to decouple drivers from its downloads to separate modules and document how their users can install what they actually need (in my case, I would prefer OneAPI Level-Zero Intel GPU drivers).

Huge images:
$ docker images | grep opea
opea/llm-tgi            latest     139c71c9b631   About an hour ago   2.14GB
opea/reranking-tei      latest     25343403b23b   2 hours ago         6.36GB
opea/retriever-redis    latest<PHONE_NUMBER>26   3 hours ago         7.55GB
opea/embedding-tei      latest     2b93d18d6f56   3 hours ago         7.27GB

Due to (useless to me) Nvidia support :
$ docker run -it --rm --entrypoint /bin/sh --user 1000 opea/retriever-redis -c "du -ks /home/user/.local/lib/python3.11/site-packages/* | sort -nr"
2892656  /home/user/.local/lib/python3.11/site-packages/nvidia
1618908  /home/user/.local/lib/python3.11/site-packages/torch
429964   /home/user/.local/lib/python3.11/site-packages/triton
107692   /home/user/.local/lib/python3.11/site-packages/scipy
92780    /home/user/.local/lib/python3.11/site-packages/transformers
...

In multiple Python modules:
$ find /home/ -name '*nvidia*' -o -name '*cuda*' | xargs du -ks | sort -nr
2892660  /home/user/.local/lib/python3.11/site-packages/nvidia
861020   /home/user/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
83440    /home/user/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda_linalg.so
80160    /home/user/.local/lib/python3.11/site-packages/triton/third_party/cuda
...

(triton is torch dependency, so one gets single instance of also that with what I proposed.)

Following change gets rid of the nvidia site-package (which still keeping torch) and reduces image size by 3GB:
 $ git diff
diff --git a/comps/retrievers/langchain/docker/Dockerfile b/comps/retrievers/langchain/docker/Dockerfile
index 2db4bfc..425d3e3 100644
--- a/comps/retrievers/langchain/docker/Dockerfile
+++ b/comps/retrievers/langchain/docker/Dockerfile
@@ -30,7 +30,12 @@ RUN chmod +x /home/user/comps/retrievers/langchain/run.sh
 USER user

 RUN pip install --no-cache-dir --upgrade pip && \
-    pip install --no-cache-dir -r /home/user/comps/retrievers/requirements.txt
+    pip install --no-cache-dir -r /home/user/comps/retrievers/requirements.txt && \
+    pip uninstall -y nvidia-cublas-cu12 nvidia-cuda-cupti-cu12 \
+      nvidia-cuda-nvrtc-cu12 nvidia-cuda-runtime-cu12 \
+      nvidia-cudnn-cu12 nvidia-cufft-cu12 nvidia-curand-cu12 \
+      nvidia-cusolver-cu12 nvidia-cusparse-cu12 \
+      nvidia-nccl-cu12 nvidia-nvjitlink-cu12 nvidia-nvtx-cu12

 ENV PYTHONPATH=$PYTHONPATH:/home/user

Special torch + triton versions that are built without CUDA support could reduce the image sizes by extra 1GB.

thank you for raising the question. we will feedback to the dev team and update if any progress.

As an example, here's documentation on how Kubeflow images are layered: https://www.kubeflow.org/docs/components/notebooks/container-images/
(Once this ticket has been fixed, similar graph of OPEA image layers would nice.)

I think this PR could solve your issue, https://github.com/opea-project/GenAIComps/pull/221

Feel free to reopen it if you have other concerns.

I think this PR could solve your issue, #221
Feel free to reopen it if you have other concerns.

Only project members can re-open tickets.  I cannot.
#221 solves only half of the container size problem I reported (while getting rid components with Nvidia proprietary license, which IMHO was at least as important).
=> @chensuyue Please reopen.
Because several containers are using PyTorch, container size can be reduced further (significantly), by putting PyTorch (and other shared dependencies) to an intermediate image that is used as a (shared) base image for the final images needing those deps.
That way:

Such large dependencies are downloaded / built only once during image builds
Rest of the images are small as PyTorch is on shared layer, instead bloating sizes of all the images
Smaller images mean first run of the given containers (on given host) being much faster

I.e. it will take significantly less bandwidth, disk space and time, which all are cost reductions on rented clusters.
Dockerfiles will also be more maintainable, with (almost) duplicated content being removed.

Filed https://github.com/opea-project/GenAIComps/issues/265 for rest of the issue.


make interval between synchronizations configurable

useful and also needed to support https://github.com/wearemolecule/route53-kubernetes/blob/master/service_listener.go#L38

LGTM

is there any doc about how to configure this?
with --interval flag?


GRCh38_nestedRepeats_20200703.tsv in update version  or   nested_repeat_file=/path/to/nested/repeat.tsv in initial version

Hi andr-kun,
I am trying to replicate your script manuscript/bash_files/TE_isoform_filter/transposon_isoforms_filter_hipsci.sh but I am having trouble finding the file GRCh38_nestedRepeats_20200703.tsv. I have searched for it on Google and UCSC, but I couldn't find a suitable download link. Can you please share the download link for this file? I would really appreciate it.
Thank you.

Hi Kratos12138,
You can download the nested repeats file from UCSC genome browser, using their table export tool. The specific options you need to download the nested repeats file can be found in the image below:

Kind regards,
Andrian


Chromium docs 3971750

Docs Build status updates of commit a4aa593:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error] Cannot sync git repo to specified state. Please make sure the pull request is mergeable. Details: Git merge results in conflict, repository remote url: https://github.com/MicrosoftDocs/edge-developer, working directory: W:\xq0m-s, target branch: HEAD, target branch commit: c6acf2d3bb4861e8fafd7432b4a28f3b26c27527, source commit: a4aa5930c958cb821b1cc65b82024d0d937185a6.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 9591543:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-edge/devtools-guide-chromium/beginners/html.md
:white_check_mark:Succeeded

microsoft-edge/devtools-guide-chromium/customize/dark-theme.md
:white_check_mark:Succeeded

microsoft-edge/devtools-guide-chromium/customize/images/command-menu-dark.msft.png
:white_check_mark:Succeeded

microsoft-edge/devtools-guide-chromium/customize/images/elements-styles-console-dark-theme.msft.png
:white_check_mark:Succeeded

microsoft-edge/devtools-guide-chromium/customize/images/options-settings.msft.png
:white_check_mark:Succeeded

microsoft-edge/devtools-guide-chromium/customize/images/settings-preferences-appearance-theme-dark.msft.png
:white_check_mark:Succeeded

microsoft-edge/devtools-guide-chromium/javascript/background-services.md
:white_check_mark:Succeeded

microsoft-edge/devtools-guide-chromium/progressive-web-apps.md
:white_check_mark:Succeeded

microsoft-edge/devtools-guide-chromium/shortcuts.md
:white_check_mark:Succeeded

microsoft-edge/devtools-guide-chromium/workspaces/index.md
:white_check_mark:Succeeded

microsoft-edge/toc.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


just looping one driver for release canvas 1.0

Description
just looping one driver for release canvas 1.0

Looks good


Merging all the latest code from Gitcorp aem-markdown-importer

This PR is for issue #…
Following things have changed:

Updates the version of flex mark used in standalone importer to 0.28.34.
Improved scrolling experience in Imported API/Developer Documentation
Better formatting for the lists and nested lists in Imported API docs
Improved connection communication between standalone importer and AEM
Better Image Alignment, formatting and rendering from markdown sources by standalone importer
Email, white spaces and HTML links handled better in imported documentation

Looks good. But it seems there are unresolved conflicts and in some areas massive white space changes.


feature: support concurrent multipart object upload with s3 backend?

rgw logs:
2023-01-30T09:06:29.411+0000 7f4a8d2be700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx [30/Jan/2023:09:06:28.269 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=1&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>00 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.142009735s

2023-01-30T09:06:37.914+0000 7f49e1166700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:06:36.585 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=2&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>00 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.329011321s

2023-01-30T09:06:46.531+0000 7f4a5c25c700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:06:45.225 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=3&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>00 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.306011081s

2023-01-30T09:06:54.822+0000 7f4a8a2b8700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx [30/Jan/2023:09:06:53.265 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=4&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>00 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.557013273s

2023-01-30T09:07:02.879+0000 7f49cd13e700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:07:01.416 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=5&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>00 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.463012457s

2023-01-30T09:07:10.677+0000 7f4a181d4700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:07:09.415 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=6&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>00 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.262010694s

2023-01-30T09:07:18.327+0000 7f49bf122700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx [30/Jan/2023:09:07:17.081 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=7&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>00 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.245010734s

2023-01-30T09:07:26.367+0000 7f4a2c1fc700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx [30/Jan/2023:09:07:24.893 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=8&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>00 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.473012567s

2023-01-30T09:07:33.966+0000 7f49e596f700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:07:32.752 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=9&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>00 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.214010358s

2023-01-30T09:07:37.338+0000 7f4a35a0f700  1 beast: 0x7f499a681630: [x.x.x.x](http:/x.x.x.x/) - xx[30/Jan/2023:09:07:35.505 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=10&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1"<PHONE_NUMBER>86 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.832015634s

from ceph rgw logs: we know the distribution push the object with multiparts one by one  and not in concurrency.

+1, seems to be a great improvement for accelerating artifact pushes, e.g. a very large model stored as an OCI artifact.

Feel free to open a PR, but I have a feeling this might require some serious refactoring 🤔

is multipart upload still serial now?

is multipart upload still serial now?

yes