Breakpoints crash flutter application

Observed on master building from a mac for either an android device or iOS simulator. Insert a debugger() call somewhere (I used the update methods in editable_text.dart) or use the observatory break x command.  When the breakpoint is hit the application crashes immediately.
Android:
F/libc    ( 6857): Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x2504260c258 in tid 6890 (1.ui), pid 6857 (er.demo.gallery)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/walleye/walleye:9/PPP3.180510.008/4811556:user/release-keys'
Revision: 'MP1'
ABI: 'arm64'
pid: 6857, tid: 6890, name: 1.ui  >>> io.flutter.demo.gallery <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x2504260c258
    x0<PHONE_NUMBER>731e10  x1<PHONE_NUMBER>7f2800  x2<PHONE_NUMBER>000001  x3<PHONE_NUMBER>cd5698
    x4<PHONE_NUMBER>000000  x5<PHONE_NUMBER>000000  x6<PHONE_NUMBER>a27d80  x7<PHONE_NUMBER>a27250
    x8<PHONE_NUMBER>60c259  x9<PHONE_NUMBER>e59919  x10<PHONE_NUMBER>000001  x11 ffffffffffffffff
    x12<PHONE_NUMBER>000000  x13<PHONE_NUMBER>000000  x14<PHONE_NUMBER>200000  x15<PHONE_NUMBER>00008e
    x16 000000767c3af4d0  x17<PHONE_NUMBER>703630  x18 000000767c3b86a0  x19<PHONE_NUMBER>731dc0
    x20<PHONE_NUMBER>731db0  x21 000000767ab00041  x22<PHONE_NUMBER>000000  x23<PHONE_NUMBER>da4cc0
    x24<PHONE_NUMBER>7bffff  x25 000000767c3b1538  x26 000000767c3b1660  x27 000000767c36d888
    x28 000000767c372898  x29<PHONE_NUMBER>a21c68
    sp  000000767b386ac0  lr  000000767b9a6f08  pc  000000767b9a6f78
backtrace:
    #00 pc<PHONE_NUMBER>61df78  /data/app/io.flutter.demo.gallery-SMkSqt7HPu3poMwax5SB3g==/lib/arm64/libflutter.so
    #01 pc<PHONE_NUMBER>61b61c  /data/app/io.flutter.demo.gallery-SMkSqt7HPu3poMwax5SB3g==/lib/arm64/libflutter.so
    #02 pc<PHONE_NUMBER>626cfc  /data/app/io.flutter.demo.gallery-SMkSqt7HPu3poMwax5SB3g==/lib/arm64/libflutter.so
    #03 pc<PHONE_NUMBER>596fd0  /data/app/io.flutter.demo.gallery-SMkSqt7HPu3poMwax5SB3g==/lib/arm64/libflutter.so
    #04 pc<PHONE_NUMBER>000ad4  <an

iOS:
io.flutter.1.ui (9): EXC_BAD_ACCESS (code=1, address=0x5f9b86318)

The stack is something like:
dart::ActivationFrame::GetAsyncCompleterAwaiter
dart::Debugger::CollectAwaiterReturnStackTrace
dart::Debugger::PauseDeveloper
...

Works as expected in 87a067704

Broken in 7ebf2728dc9173ad63a41aaa4c13e1fcafa51d4a

Original culprit is f4a038da6328d437c9f0ca54d77ceb22779224a3, which was rolled back a few times, but it looks like it was the dart update that broke it

cc @Hixie

also cc'ing @aam, @a-siva, and @jacob314

@jonahwilliams, thanks for bisecting!

we should revert that dart roll, between this and the performance regressions, we can't ship it.

Not able to reproduce the crash somehow(on e7832a389755a1894494be1a26c2debf2cac745e).
With flutter gallery I am setting breakpoints(in observatory and in IJ) in editable_text.dart:831, get the breakpoint triggered as I navigate to Material/Text fields, able to step and continue.
What line are you setting breakpoint on, what app are you debugging?

I actually started experiencing this yesterday as well, it happens on debug-mode and when an exception is thrown.

@aam
Investigated more - breaking on something like Foo.build works fine, breaking on EditableTextState.updateEditingValue or other methods crashes.  Not sure what the distinction is.

GetAsyncCompleterAwaiter

Perhaps related to the async debugging functionality?

@aam I did two more cycles of rm bin/cache/fluter_tool.s* and flutter clean.  Not sure what kicked it over but it did start working again - I think your change fixes it.  Thank you for taking another look and sorry for sounding the alarm!

This was fixed for me on Friday,
but an update I just did broke it again.
I'm not sure it's the same issue. It seems to work reproducible when run from console using flutter run, but when I debug-run it from IntelliJ I get
Launching lib/main_zoechi.dart on Android SDK built for x86 in debug mode...
Initializing gradle...
Resolving dependencies...
Running 'gradlew assembleDebug'...
registerResGeneratingTask is deprecated, use registerGeneratedResFolders(FileCollection)
registerResGeneratingTask is deprecated, use registerGeneratedResFolders(FileCollection)
registerResGeneratingTask is deprecated, use registerGeneratedResFolders(FileCollection)
Built build/app/outputs/apk/debug/app-debug.apk.
Installing /Users/zoechi/dart/customers/lookatyou/lookatmybaby/lookatmybaby_flutter/build/app/outputs/apk/app.apk...
Syncing files to device Android SDK built for x86...
F/libc    (29177): Fatal signal 11 (SIGSEGV), code 2, fault addr 0x97b9ee44 in tid 29207 (1.ui), pid 29177 (ou.lookatmybaby)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/sdk_gphone_x86/generic_x86:8.1.0/OSM1.180201.007/4586646:user/release-keys'
Revision: '0'
ABI: 'x86'
pid: 29177, tid: 29207, name: 1.ui  >>> com.lookatyou.lookatmybaby <<<
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x97b9ee44
    eax 0c20f410  ebx 8ef9eaa0  ecx 7f780625  edx 8dd7f970
    esi 8ef9fd1c  edi 8c0a91b0
    xcs 00000073  xds 0000007b  xes 0000007b  xfs 0000003b  xss 0000007b
    eip 8e566db2  ebp 8dd7d9e8  esp 8dd7d9a0  flags 00010202
backtrace:
    #00 pc 00624db2  /data/app/com.lookatyou.lookatmybaby-NkwTGzygWQJLeHi-3XYZYg==/lib/x86/libflutter.so
    #01 pc 00625284  /data/app/com.lookatyou.lookatmybaby-NkwTGzygWQJLeHi-3XYZYg==/lib/x86/libflutter.so
    #02 pc 00622524  /data/app/com.lookatyou.lookatmybaby-NkwTGzygWQJLeHi-3XYZYg==/lib/x86/libflutter.so
    #03 pc 0062a040  /data/app/com.lookatyou.lookatmybaby-NkwTGzygWQJLeHi-3XYZYg==/lib/x86/libflutter.so
    #04 pc 00637cc5  /data/app/com.lookatyou.lookatmybaby-NkwTGzygWQJLeHi-3XYZYg==/lib/x86/libflutter.so
    #05 pc 00637c74  /data/app/com.lookatyou.lookatmybaby-NkwTGzygWQJLeHi-3XYZYg==/lib/x86/libflutter.so
    #06 pc 00639661  /data/app/com.lookatyou.lookatmybaby-NkwTGzygWQJLeHi-3XYZYg==/lib/x86/libflutter.so
    #07 pc 0091ae5b  /data/app/com.lookatyou.lookatmybaby-NkwTGzygWQJLeHi-3XYZYg==/lib/x86/libflutter.so
    #08 pc 0000058b  <anonymous:8c440000>
    #09 pc 000005dd  <anonymous:8c440000>
    #10 pc 000062e4  <anonymous:7f200000>
    #11 pc 00006058  <anonymous:7f200000>
    #12 pc 000348d7  <anonymous:89600000>
    #13 pc 00005d00  <anonymous:7f200000>
    #14 pc 00005c55  <anonymous:7f200000>
    #15 pc 00005af9  <anonymous:7f200000>
    #16 pc 00002f65  <anonymous:7f200000>
    #17 pc 000189df  <anonymous:7f480000>
    #18 pc 00017e72  <anonymous:7f480000>
    #19 pc 000178f1  <anonymous:7f480000>
    #20 pc 0001bc2c  <anonymous:7f480000>
    #21 pc 00016bd4  <anonymous:7f480000>
    #22 pc 00015c52  <anonymous:7f480000>
    #23 pc 000151c7  <anonymous:7f480000>
    #24 pc 00018c81  <anonymous:7f480000>
    #25 pc 00017e72  <anonymous:7f480000>
    #26 pc 000178f1  <anonymous:7f480000>
    #27 pc 0001bc2c  <anonymous:7f480000>
    #28 pc 00016bd4  <anonymous:7f480000>
    #29 pc 00015c52  <anonymous:7f480000>
    #30 pc 000151c7  <anonymous:7f480000>
    #31 pc 00018c81  <anonymous:7f480000>
    #32 pc 00017e72  <anonymous:7f480000>
    #33 pc 000178f1  <anonymous:7f480000>
    #34 pc 00016bd4  <anonymous:7f480000>
    #35 pc 00015c52  <anonymous:7f480000>
    #36 pc 000151c7  <anonymous:7f480000>
    #37 pc 000145dc  <anonymous:7f480000>
    #38 pc 0000dee1  <anonymous:7f480000>
    #39 pc 0000dd78  <anonymous:7f480000>
    #40 pc 000003f1  <anonymous:89600000>
    #41 pc 0000b663  <anonymous:7f480000>
    #42 pc 000085d5  <anonymous:7f480000>
    #43 pc 000073b9  <anonymous:7f480000>
    #44 pc 00022f0f  <anonymous:7fa00000>
    #45 pc 0000f900  <anonymous:80180000>
    #46 pc 000003f1  <anonymous:89600000>
    #47 pc 0000e931  <anonymous:80180000>
    #48 pc 0000c12f  <anonymous:80180000>
    #49 pc 0000b823  <anonymous:80180000>
    #50 pc 000003f1  <anonymous:89600000>
    #51 pc 0000b1c2  <anonymous:80180000>
    #52 pc 0000afcd  <anonymous:80180000>
    #53 pc 0000aafa  <anonymous:80180000>
    #54 pc 0000a64b  <anonymous:80180000>
    #55 pc 000046b4  <anonymous:80180000>
    #56 pc 000030bd  <anonymous:80180000>
    #57 pc 0001d9be  <anonymous:89600000>
    #58 pc 0001d55e  <anonymous:89600000>
    #59 pc 000003f1  <anonymous:89600000>
    #60 pc 0001caf6  <anonymous:89600000>
    #61 pc 0001c854  <anonymous:89600000>
    #62 pc 0001c794  <anonymous:89600000>
    #63 pc 0000096f  <anonymous:8c440000>
Lost connection to device.

✓] Flutter (Channel master, v0.5.7-pre.124, on Mac OS X 10.13.5 17F77, locale en-AT)
    • Flutter version 0.5.7-pre.124 at /Users/zoechi/flutter/flutter
    • Framework revision f6b234a0ea (33 minutes ago), 2018-07-16 14:23:25 +0200
    • Engine revision c5a63d28bf
    • Dart version 2.0.0-dev.67.0.flutter-84ca27a09e

[✓] Android toolchain - develop for Android devices (Android SDK 27.0.3)
    • Android SDK at /usr/local/opt/android-sdk
    • Android NDK at /usr/local/opt/android-sdk/ndk-bundle
    • Platform android-27, build-tools 27.0.3
    • ANDROID_HOME = /usr/local/opt/android-sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)
    • All Android licenses accepted.

[✓] iOS toolchain - develop for iOS devices (Xcode 9.4.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 9.4.1, Build version 9F2000
    • ios-deploy 1.9.2
    • CocoaPods version 1.5.3

[✓] Android Studio (version 3.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 23.1.2
    • Dart plugin version 173.4700
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)

[✓] IntelliJ IDEA Ultimate Edition (version 2018.1.5)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin version 26.0.2
    • Dart plugin version 181.4892.1

[✓] Connected devices (1 available)
    • Android SDK built for x86 • emulator-5554 • android-x86 • Android 8.1.0 (API 27) (emulator)

I've got an integration test failing for what I believe to be this issue:
https://github.com/flutter/flutter/pull/19473
It passes if patched into e22f997 but fails on current master. I can't land it because it requires changes to vm_server_client (I've opened a PR, but it'll need to be reviewed, merged and rolled into Flutter) but you can run it locally by cloning my vm_server_client PR branch and overriding the path in pubspec.yaml (see the PR for more info) if you want to use it to test fixes.

@zoechi wrote:

This was fixed for me on Friday, but an update I just did broke it again.

How was it fixed on Friday? https://github.com/flutter/flutter/pull/19378 never went in because we should do it differently: https://github.com/flutter/flutter/pull/19378#issuecomment-405285521.

@aam I'm not sure. It seems it just sometimes works. Probably better to just ignore my comment.

This no longer seems to be an issue on ToT.

Ignore my last comment. I am attempting to build with a local engine and trying to reproduce.

@aam and I have verified that https://github.com/flutter/engine/commit/1d126dab31595748fc7abc38b6be29bfe8ffeadd fixes the issue and once it is rolled in, we can close this.

Update: We believe this is fixed, we will do an engine roll and find out where we stand.

The failing test in my PR above now passes 👍
✓ flutter run does not terminate
✓ flutter run does not terminate when a debugger is attached
✓ flutter run does not terminate when a debugger is attached and pause-on-exceptions

I still need to get the vm_server_client changes landed and released before I can merge the tests in though.

Confirmation of similar issue fixed also at https://github.com/flutter/flutter/issues/19464#issuecomment-405844970. I suspect this can be closed, but will leave for @jonahwilliams to confirm since the exact symptoms were slightly different.

This appears to be fixed at head, closing!

Breakpoints are now working for me as well. Thanks for fixing this!

I have the same issue since I moved to flutter 2 stable

@papmodern see https://github.com/flutter/flutter/issues/80733#issuecomment-823088387.


Ci enhancements

CI-enhancements

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 37.35%. Comparing base (cab53c4) to head (3908177).
Report is 4 commits behind head on develop.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #204      +/-   ##
===========================================
- Coverage    37.91%   37.35%   -0.57%
===========================================
  Files           36        1      -35
  Lines         3373      779    -2594
===========================================
- Hits          1279      291     -988
+ Misses        2012      479    -1533
+ Partials        82        9      -73

Flag
Coverage Δ

integrationtests
37.35% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


⚠ Bofrost.es has degraded performance

In 34a8d12, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1768 ms

Resolved: Bofrost.es performance has improved in e48ff84 after 6 minutes.


Navigate grid using cursor keys. Is it posible?

Can I use the cursor keys to navigate the grid by highlighting the current cell or row?. Thank you

Hi @valientegaston!
No, because the highlight is for mouse hover only. I believe that if the highlight continued even without any mouse hover, it would be the selection concept (which I still need to create). The two could coexist at the same time. The cursor moves the selection and applies the scroll when necessary.
Once I implement the selection, you can use it as a highlight of course.
I just don't know if I'll get it before my vacation ok? :stuck_out_tongue_winking_eye:

Hi! I really appreciate the speed of your response. Thank you so much!
When you say "selection" do you mean marking a set of cells that belong or not to the same row, to later copy the content, or do you mean a highlight to know which cell I'm currently standing on in the grid?
Can a cell take a "focus"? Basically I need to navigate the grid by each cell, using the arrow keys. I have also not found a way to select a specific cell with the mouse.
I love this grid, and I want to use it to replace another one in a project, that's why I want to know with the limitations that I can find.
Thank you!

Selection could be by row or by cell. I thought about being per row only first.
So when clicking on a row, it would have the status selected and could have a different color. The mouse hover would highlight the row (selected or not).
The selection could be single or multiple (holding CTRL or SHIFT).
And the model would have a method to get the selected rows.
Currently, the grid can already have focus because it captures page down/up events. I would have to add the up/down arrows as well. And in the future, sideways arrows because of per-cell selection.
The idea is that this grid makes everything possible in the universe :stuck_out_tongue_winking_eye:
But since I keep other packages too, it takes a certain amount of time :wink:

thank you!

when can we expect selection row or cell by keyboard keys (arrow up/down/left/right)
This is very useful for desktop or web

Im waiting for it too :)


Error: sendInteraction API call failed: Error: 3 INVALID_ARGUMENT: Request contains an invalid argument.

Good afternoon
I'm trying to automate some of the tests in my action, but I always give the following error
sendQuery called with 'Talk to my test app'
Error: sendInteraction API call failed: Error: 3 INVALID_ARGUMENT: Request contains an invalid argument.
at ActionsApiHelper.sendInteraction (node_modules \ @assistant \ conversation-testing \ dist \ actions-api-helper.js: 47: 19)
at async ActionsOnGoogleTestManager.sendInteraction (node_modules \ @assistant \ conversation-testing \ dist \ action-on-google-test-manager.js: 79: 31)
at async Context.  (test \ balance.test.js: 37: 5)
What may be happening.
I've tried to configure the sentence in several ways including leaving it blank but it doesn't work.
Thanks

Hello @LuizPiresS how did you fix it?

I have the same problem. Would really like to know how you fixed it.

We were asking for authorization to get the username from the Google account.
At first, we chose to disable the use of the user name.
We did this so that we were able to deliver the POC on time.


would be useful if :as => Range could always return a Range (if possible)

opt.on :i, :in, "Gist entries from Pry's input expression history. Takes an index or range.", :optional => true, :as => Range, :default => -5..-1 do |range|

If i invoke it with gist -i 1 it would be useful if range could be yielded as 1..1

I don't see the logic behind -i 1 returning 1..1, then again I don't really see the logic behind the current return of 1 (as an Integer), but that was also a request from yourself. The only logical thing I see -i 1 returning (as a Range) is probably 0..1. But I need much more input on this.

The logic behind returning 1..1 instead of 1 is that this way, we don't have to check if the user supplied a range or a single value, we can always use Range methods (e.g. #each).


manifest documentation is out of date

Page Affected: https://developers.google.com/web/fundamentals/web-app-manifest/
What needs to be done?
The article does not reflect the current state of how Chrome (and other browsers) use the manifest - for example, note at the end says you can't update the manifest (you can now with web apk and we should detail the process clearly). We now also need a 512 icon for splash screen iirc. We should check and confirm how safari use's this data. I see a lot of people using description now, we should at least document it even though it's not used in Chrome.
There's probably a lot more I missing, that we can add as best practice.

As we're at it, consider adding documentation on:

"screenshots" (spec deeplink)
"iarc_rating_id" (spec deeplink)

Both properties could (but as far as my analysis goes currently aren't) used by Microsoft for the Windows Store.

I think it's worth ignoring these until there is usage of it.
On Wed, Apr 11, 2018, 14:59 Thomas Steiner<EMAIL_ADDRESS>wrote:

As we're at it, consider adding documentation on:

"screenshots" (spec deeplink
https://www.w3.org/TR/appmanifest/#screenshots-member)
"iarc_rating_id" (spec deeplink
https://www.w3.org/TR/appmanifest/#iarc_rating_id-member)

Both properties could (but as far as my analysis goes currently aren't)
used by Microsoft for the Windows Store.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/WebFundamentals/issues/6029#issuecomment-380461980,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACxxgM7cmIeF0IXUHPcqRbzbmG6P0Qeks5tngw5gaJpZM4TPilx
.


Handle validation error

Motivation and context

Error
CVATValidationError
ValidationError

some detail
'some detail'
"[ErrorDetail(string='some detail', code='invalid')]"

["some detail 1", "some detail 2"]
'some detail 1; some detail 2'
"[ErrorDetail(string='some detail 1', code='invalid'), ErrorDetail(string='some detail 2', code='invalid')]""

{"key_1": "some detail 1", "key_2": "some detail 2"}
'key_1: some detail 1; key_2: some detail 2;'
"{'key_1': ErrorDetail(string='some detail 1', code='invalid'), 'key_2': ErrorDetail(string='some detail 2', code='invalid')}"

{"key_1": ["some detail 1.1", "some detail 1.2"], "key_2": ["some detail 2.1", "some detail 2.2"]}
"key_1: ['some detail 1.1', 'some detail 1.2']; key_2: ['some detail 2.1', 'some detail 2.2'];"
"{'key_1': [ErrorDetail(string='some detail 1.1', code='invalid'), ErrorDetail(string='some detail 1.2', code='invalid')], 'key_2': [ErrorDetail(string='some detail 2.1', code='invalid'), ErrorDetail(string='some detail 2.2', code='invalid')]}"

How has this been tested?

Checklist

[x] I submit my changes into the develop branch
- [ ] I have added a description of my changes into CHANGELOG file
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (read github docs)
- [ ] I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

[x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.

@zhiltsov-max, Could you please take a look at this small PR?

From the PR description I see that the errors are still strings, while they represent more complex structures. Is there a reason for this? Wouldn't it be better to pass json structures instead?

@Marishka17 , could you please resolve the conflict?


setTrustedCertificatesBytes not working on IOS

I use self signed certification(and with client side verification) for ssl https purpose, works perfectly on Android and Postman, but not IOS!!!
setTrustedCertificatesBytes with rootCA.crt (pem, p12):
IOS  : ' HandshakeException: Handshake error in client (OS Error: CERTIFICATE_VERIFY_FAILED: application verification failure(handshake.cc:393))'
setTrustedCertificatesBytes with rootCA.der:
IOS Android : ' BAD_PKCS12_DATA(pkcs8_x509.c:657), errno = 0'
If I use setTrustedCertificates, can not load the assets file:  (OS Error: No such file or directory, errno = 2)

ByteData bytes = await rootBundle.load('assets/ca/client.p12');
ByteData rootPem = await rootBundle.load('assets/ca/rootCA.p12');
(_dio.httpClientAdapter as DefaultHttpClientAdapter).onHttpClientCreate =
(client) {
SecurityContext sc = SecurityContext(withTrustedRoots: false);
sc.setTrustedCertificatesBytes(rootPem.buffer.asUint8List(),
password: 'A#1kQ98!I2hwi12wB!ISC$5');
sc.useCertificateChainBytes(bytes.buffer.asUint8List(),
password: 'A31q4fVWe$@bTCYT%^5!64HBJ');
sc.usePrivateKeyBytes(bytes.buffer.asUint8List(),
password: 'A31q4fVWe$@bTCYT%^5!64HBJ');
HttpClient httpClient = HttpClient(context: sc);
// httpClient.badCertificateCallback =
//     (X509Certificate cert, String host, int port) => true;
return httpClient;
};

stuck here for couple days, please tell me this is a bug

Hi @what2003, please provide a completed and minimal reproducible code sample so that we may verify this. Also, please provide the output of flutter doctor -v as well. Thank you!
May be related/similar: #87717

https://github.com/what2003/flutter_application_1
ssl files all in assets/ca, my server api is working fine, please check it out.

╰─$ flutter doctor -v
Building flutter tool...
[✓] Flutter (Channel stable, 3.3.4-0.0.pre.1, on macOS 11.6 20G165 darwin-x64, locale zh-Hans-CN)
• Flutter version 3.3.4-0.0.pre.1 on channel stable at /Users/jack/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 27d7ec9563 (11 minutes ago), 2022-10-04 10:59:06 -0500
• Engine revision 5c984c26eb
• Dart version 2.18.2
• DevTools version 2.15.0

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
• Android SDK at /Users/jack/Library/Android/sdk
• Platform android-33, build-tools 30.0.3
• Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 13.0)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Build 13A233
• CocoaPods version 1.11.3
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 2021.2)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
[✓] VS Code (version 1.71.2)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.48.0
[✓] Connected device (3 available)
• SM G975U (mobile)    • R58M304QW1Z                              • android-arm64  • Android 11 (API 30)
• 吴杰的 iPhone7 (mobile) • eed6e3b2d815420621d441672707634130e89e16 • ios            • iOS 15.0.1 19A348
• Chrome (web)         • chrome                                   • web-javascript • Google Chrome 104.0.5112.101
[✓] HTTP Host Availability
• All required HTTP hosts are available
• No issues found!

Hi @what2003, thanks for providing the sample code.
But from what I can see, you're using dio package for network requests. It's similar to someone using http package here that should be opened in http repository. If you still want to use dio, please open the issue in the dedicated repository.
For the issue to be workable, it needs to be reproducible with a completed and minimal reproducible code sample that doesn’t include 3rd party plugins or complex production code. Please provide such a sample.
Thank you!

https://github.com/what2003/flutter_application_1
Sorry for 3rd party plugins, now it is pure flutter, and only one page, please check it out!
Thank you! @huycozy

@what2003 thanks for updating the sample code. I also see there's an open issue addressing the case you described #50699. There is also someone who fixed the issue by various workaround. Please check that issue if it can help.

Thanks @huycozy , #50699 is a different problem, after many times test, clearly "setTrustedCertificatesBytes" is not performed on IOS, easy to reproduce.

@what2003 I label this issue for further investigation/insight from the team due to this case using a self-signed certificate whilst #50699 uses GoDaddy.
This issue is reproducible on the latest stable and master channels with sample code.

Completed exception logs
[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: HandshakeException: Handshake error in client (OS Error:
	CERTIFICATE_VERIFY_FAILED: application verification failure(handshake.cc:393))
#0      _SecureFilterImpl._handshake (dart:io-patch/secure_socket_patch.dart:99:46)
#1      _SecureFilterImpl.handshake (dart:io-patch/secure_socket_patch.dart:142:25)
#2      _RawSecureSocket._secureHandshake (dart:io/secure_socket.dart:911:54)
#3      _RawSecureSocket._secureHandshake (dart:io/secure_socket.dart:914:15)
<asynchronous suspension>
#4      _RawSecureSocket._tryFilter (dart:io/secure_socket.dart:1040:13)
<asynchronous suspension>

flutter doctor -v (stable and master)
[✓] Flutter (Channel stable, 3.3.4, on macOS 12.6 21G115 darwin-x64, locale en-VN)
    • Flutter version 3.3.4 on channel stable at /Users/huynq/Documents/GitHub/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision eb6d86ee27 (23 hours ago), 2022-10-04 22:31:45 -0700
    • Engine revision c08d7d5efc
    • Dart version 2.18.2
    • DevTools version 2.15.0

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 31.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
    • Xcode at /Applications/Xcode14.0.1.app/Contents/Developer
    • Build 14A400
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] IntelliJ IDEA Community Edition (version 2022.2.2)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 70.0.5
    • Dart plugin version 222.4167.21

[✓] IntelliJ IDEA Community Edition (version 2022.1.1)
    • IntelliJ at /Users/huynq/Library/Application Support/JetBrains/Toolbox/apps/IDEA-C/ch-0/221.5591.52/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.71.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.48.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-x64     • macOS 12.6 21G115 darwin-x64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 105.0.5195.125

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

[✓] Flutter (Channel master, 3.5.0-3.0.pre.5, on macOS 12.6 21G115 darwin-x64, locale en-VN)
    • Flutter version 3.5.0-3.0.pre.5 on channel master at /Users/huynq/Documents/GitHub/flutter_master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 9ca314afd0 (12 minutes ago), 2022-10-06 23:06:11 -0400
    • Engine revision a4174f29a7
    • Dart version 2.19.0 (build 2.19.0-286.0.dev)
    • DevTools version 2.18.0

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 31.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
    • Xcode at /Applications/Xcode14.0.1.app/Contents/Developer
    • Build 14A400
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] IntelliJ IDEA Community Edition (version 2022.2.2)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 70.0.5
    • Dart plugin version 222.4167.21

[✓] IntelliJ IDEA Community Edition (version 2022.1.1)
    • IntelliJ at /Users/huynq/Library/Application Support/JetBrains/Toolbox/apps/IDEA-C/ch-0/221.5591.52/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.71.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.48.0

[✓] Connected device (5 available)
    • SM T225 (mobile)           • R9JT3004VRJ                              • android-arm64  • Android 12 (API 31)
    • iPhone (mobile)            • d9a94afe2b649fef56ba0bfeb052f0f2a7dae95e • ios            • iOS 15.7 19H12
    • iPhone 14 Pro Max (mobile) • 332079AF-E895-4685-910E-7B1E18B0C6B8     • ios            • com.apple.CoreSimulator.SimRuntime.iOS-16-0 (simulator)
    • macOS (desktop)            • macos                                    • darwin-x64     • macOS 12.6 21G115 darwin-x64
    • Chrome (web)               • chrome                                   • web-javascript • Google Chrome 106.0.5249.103
    ! Error: iPhone is busy: Fetching debug symbols for iPhone. Xcode will continue when iPhone is finished. (code -10)

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

Wasted time to reporting bug,  only result is my server exposed here now getting someone's attack.
This flutter security issue makes every flutter build app face MITM attack risk.
Appreciate  Google, but BIG disappointed by this team get paid not doing their jobs!

@what2003 please refer to our code of conduct in regards to respectful, welcoming communication.
https://github.com/what2003/flutter_application_1 is a 404.

I'm having the exact same issue. setTrustedCertificatesBytes doesn't work on iOS.

@kandrelczyk are you able to provide a minimal reproducible code sample app then? When I investigated this issue the one that was previously provided is no longer available (my 404 comment).

@jmagman here you go: https://github.com/kandrelczyk/flutter-selfsigned-example
All this app does is make a http call to https://<IP_ADDRESS>:8443/test which uses a self-signed cert. The root CA is added to security context using setTrustedCertificatesBytes. When running is iOS simulator it results in:
[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: HandshakeException: Handshake error in client (OS Error:
CERTIFICATE_VERIFY_FAILED: application verification failure(handshake.cc:393))
I've manually verified the cert with the CA and it's correct. I've also added the rootCA to OS X keychain and tested with curl. It works. I've tried both p12 and PEM formats for the CA, both give me the same error. This could be some silly error on my side but I've went through all the docs and can't find any issue. It really looks like the new SecurityContext is just not used by the HttpClient.

Until now, I still have this issue. I got an error CERTIFICATE_VERIFY_FAILED: application verification failure(handshake.cc:393))
Has this issue been resolved yet? If not, what is the workround alternative?

I'm having the exact same issue. setTrustedCertificatesBytes doesn't work on iOS.

@kandrelczyk Thanks very much for the example. As far as I can tell, only DER formatted certificates are acceptable on iOS.
You already have the DER file in your assets directory (rootCA.der) so could you change the path in main.dart and try again?

Hi @brianquinlan ,
When I try to use the rootCA.der file I'm getting following error:
[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: TlsException: Failure trusting builtin roots (OS Error:
	BAD_PKCS12_DATA(pkcs8_x509.c:657), errno = 318767204)
#0      _SecurityContext.setTrustedCertificatesBytes (dart:io-patch/secure_socket_patch.dart:249:26)
#1      main (package:selfsigned/main.dart:12:11)

Looks like setTrustedCertificatesBytes requires p12 and as far as I know you can't convert a DER to a p12, not usuing openssl.

@kandrelczyk Thanks for trying this. Let me take a more detailed look.

Running @kandrelczyk 's example on macOS (I assume that the connection is supposed to fail?):
Building macOS application...
[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: SocketException: Connection failed (OS Error: Operation not permitted, errno = 1), address = <IP_ADDRESS>, port = 8443
#0      _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:721:35)
#1      _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1914:26)
#2      RawSocket.startConnect (dart:io-patch/socket_patch.dart:27:23)
#3      RawSecureSocket.startConnect (dart:io/secure_socket.dart:297:22)
#4      SecureSocket.startConnect (dart:io/secure_socket.dart:77:28)
#5      _ConnectionTarget.connect (dart:_http/http_impl.dart:2482:26)
#6      _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2924:12)
#7      _HttpClient._getConnection (dart:_http/http_impl.dart:2929:12)
#8      _HttpClient._openUrl (dart:_http/http_impl.dart:2784:12)
#9      _HttpClient.getUrl (dart:_http/http_impl.dart:2627:48)
#10     main (package:selfsigned/main.dart:15:32)
<asynchronous suspension>

running on iOS:
[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: HandshakeException: Handshake error in client (OS Error:
	CERTIFICATE_VERIFY_FAILED: application verification failure(handshake.cc:393))
#0      _SecureFilterImpl._handshake (dart:io-patch/secure_socket_patch.dart:99:46)
#1      _SecureFilterImpl.handshake (dart:io-patch/secure_socket_patch.dart:142:25)
#2      _RawSecureSocket._secureHandshake (dart:io/secure_socket.dart:918:54)
#3      _RawSecureSocket._secureHandshake (dart:io/secure_socket.dart:921:15)
<asynchronous suspension>
#4      _RawSecureSocket._tryFilter (dart:io/secure_socket.dart:1047:13)
<asynchronous suspension>

The failure when I switch the version to use the .der file (same on iOS/macOS):
[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: TlsException: Failure trusting builtin roots (OS Error:
	BAD_PKCS12_DATA(pkcs8_x509.c:609), errno = 318767204)
#0      _SecurityContext.setTrustedCertificatesBytes (dart:io-patch/secure_socket_patch.dart:249:26)
#1      main (package:selfsigned/main.dart:12:11)
<asynchronous suspension>

Hi @brianquinlan ,
I didn't try running it on macOS so not sure what's causing the SocketException. The HandshakeException seen on iOS is the reported issue.

Hi @brianquinlan ,
I didn't try running it on macOS so not sure what's causing the SocketException. The HandshakeException seen on iOS is the reported issue.

I was able to create a working example using a PEM file:
https://github.com/kandrelczyk/flutter-selfsigned-example/compare/main...brianquinlan:flutter-selfsigned-example:modified-test?expand=1
After converting to a DEX file (openssl x509 -outform der -in assets/cert/trusted_certs.pem -out assets/cert/trusted_certs.der), I get the exception on macOS, iOS and Android:
[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: TlsException: Failure trusting builtin roots (OS Error:
	BAD_PKCS12_DATA(pkcs8_x509.c:609), errno = 318767204)
#0      _SecurityContext.setTrustedCertificatesBytes (dart:io-patch/secure_socket_patch.dart:249:26)
#1      main (package:selfsigned/main.dart:14:11)

Does someone have a self-contained example that works on Android but does not work on iOS?


Y-axis defect at RTL mode

When I tried to add dir="rtl" in body tag at this example Line with Missing Data
hint: I replace "https://d3js.org/d3.v4.js" with "//d3js.org/d3.v4.0.0-alpha.4.min.js"-
The following error appears:

This is expected. If you add dir="rtl", you’ll need to restyle your axis as appropriate since the default tick text-anchor for a left or right axis will no longer be correct. For example, you may need to add the following style:
.axis--y {
  text-anchor: start;
}

@mbostock I have this issue too.
my solution was to set the everything to ltr which I don't think is a great workaround.
Not supporting this out-of-the-box requires every developer that needs RTL support to know this and take it into account when using d3.
I think that it should be build-in in this library.
Please consider re-opening this issue.


import export simulation test for x/wasm is missing

Currently, x/wasm is not added to store prefix list should be checked inside TestAppImportExport. Is any reasons not to add the check for x/wasm?
https://github.com/CosmWasm/wasmd/blob/main/app/sim_test.go#L158
By the way, the simulation test happens error as follows since the random genesis state does not generate proper LastInstanceID.
https://github.com/CosmWasm/wasmd/blob/main/x/wasm/simulation/genesis.go#L17
The error would be like:
compared 2 different key/value pairs between KVStoreKey{0xc001218320, wasm} and KVStoreKey{0xc000302450, wasm}
    sim_test.go:275:
                Error Trace:    /home/kilem/forbole-dev/desmos/app/sim_test.go:275
                Error:          Not equal:
                                expected: 2
                                actual  : 0
                Test:           TestAppImportExport
                Messages:       store A  =>

                                store B lastContractId =>
                                store A  =>
                                store B  =>

May I take care of this issue if the thing I proposed  is worth to you?

Thanks for reporting! Good 👁 .
There is no reason to exclude the wasm module from the test. The CircleCI sim run should have caught this but did not.


live555: 2018.02.28 -> 2018.10.17

Motivation for this change
#48825 (fix for CVE-2018-4013) reworked to :

explicitly mark live555 broken on aarch64
remove live555 in vlc only for aarch64 platform but not on other x64 unixes

I rebuilt vlc successfully on nixos with these changes.
/cc @peterhoeg @timokau @oxij
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

No attempt on aarch64-linux (full log)
The following builds were skipped because they don't evaluate on aarch64-linux: live555
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: live555
Partial log (click to expand)
/nix/store/9q3054sam63vmv65jk2y0q2f67sffhzk-live555-2018.10.17

@GrahamcOfBorg build vlc

Success on aarch64-linux (full log)
Attempted: vlc
Partial log (click to expand)
/nix/store/rydhnc4x5p7gmzsnxnnyvzawsy9prlz5-vlc-3.0.4/bin/nvlc: interpreter directive changed from " /bin/sh" to "/nix/store/dsyc1z7ck08ga7l0b1jcxx35wj69qcii-bash-4.4-p23/bin/sh"
/nix/store/rydhnc4x5p7gmzsnxnnyvzawsy9prlz5-vlc-3.0.4/bin/qvlc: interpreter directive changed from " /bin/sh" to "/nix/store/dsyc1z7ck08ga7l0b1jcxx35wj69qcii-bash-4.4-p23/bin/sh"
/nix/store/rydhnc4x5p7gmzsnxnnyvzawsy9prlz5-vlc-3.0.4/bin/rvlc: interpreter directive changed from " /bin/sh" to "/nix/store/dsyc1z7ck08ga7l0b1jcxx35wj69qcii-bash-4.4-p23/bin/sh"
/nix/store/rydhnc4x5p7gmzsnxnnyvzawsy9prlz5-vlc-3.0.4/bin/cvlc: interpreter directive changed from " /bin/sh" to "/nix/store/dsyc1z7ck08ga7l0b1jcxx35wj69qcii-bash-4.4-p23/bin/sh"
/nix/store/rydhnc4x5p7gmzsnxnnyvzawsy9prlz5-vlc-3.0.4/share/vlc/utils/video-vlc-default.sh: interpreter directive changed from "/bin/sh" to "/nix/store/dsyc1z7ck08ga7l0b1jcxx35wj69qcii-bash-4.4-p23/bin/sh"
/nix/store/rydhnc4x5p7gmzsnxnnyvzawsy9prlz5-vlc-3.0.4/share/vlc/utils/audio-vlc-default.sh: interpreter directive changed from "/bin/sh" to "/nix/store/dsyc1z7ck08ga7l0b1jcxx35wj69qcii-bash-4.4-p23/bin/sh"
/nix/store/rydhnc4x5p7gmzsnxnnyvzawsy9prlz5-vlc-3.0.4/share/vlc/utils/gnome-vlc-default.sh: interpreter directive changed from "/bin/sh" to "/nix/store/dsyc1z7ck08ga7l0b1jcxx35wj69qcii-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/rydhnc4x5p7gmzsnxnnyvzawsy9prlz5-vlc-3.0.4...
postPatchMkspecs
/nix/store/rydhnc4x5p7gmzsnxnnyvzawsy9prlz5-vlc-3.0.4

Success on x86_64-linux (full log)
Attempted: vlc
Partial log (click to expand)
/nix/store/l4h7sqblghs5nnb4aa7jh6mip20p1rp4-vlc-3.0.4/share/vlc/utils/audio-vlc-default.sh: interpreter directive changed from "/bin/sh" to "/nix/store/r47p5pzx52m3n34vdgqpk5rvqgm0m24m-bash-4.4-p23/bin/sh"
/nix/store/l4h7sqblghs5nnb4aa7jh6mip20p1rp4-vlc-3.0.4/share/vlc/utils/gnome-vlc-default.sh: interpreter directive changed from "/bin/sh" to "/nix/store/r47p5pzx52m3n34vdgqpk5rvqgm0m24m-bash-4.4-p23/bin/sh"
/nix/store/l4h7sqblghs5nnb4aa7jh6mip20p1rp4-vlc-3.0.4/share/vlc/utils/video-vlc-default.sh: interpreter directive changed from "/bin/sh" to "/nix/store/r47p5pzx52m3n34vdgqpk5rvqgm0m24m-bash-4.4-p23/bin/sh"
/nix/store/l4h7sqblghs5nnb4aa7jh6mip20p1rp4-vlc-3.0.4/bin/nvlc: interpreter directive changed from " /bin/sh" to "/nix/store/r47p5pzx52m3n34vdgqpk5rvqgm0m24m-bash-4.4-p23/bin/sh"
/nix/store/l4h7sqblghs5nnb4aa7jh6mip20p1rp4-vlc-3.0.4/bin/cvlc: interpreter directive changed from " /bin/sh" to "/nix/store/r47p5pzx52m3n34vdgqpk5rvqgm0m24m-bash-4.4-p23/bin/sh"
/nix/store/l4h7sqblghs5nnb4aa7jh6mip20p1rp4-vlc-3.0.4/bin/rvlc: interpreter directive changed from " /bin/sh" to "/nix/store/r47p5pzx52m3n34vdgqpk5rvqgm0m24m-bash-4.4-p23/bin/sh"
/nix/store/l4h7sqblghs5nnb4aa7jh6mip20p1rp4-vlc-3.0.4/bin/qvlc: interpreter directive changed from " /bin/sh" to "/nix/store/r47p5pzx52m3n34vdgqpk5rvqgm0m24m-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/l4h7sqblghs5nnb4aa7jh6mip20p1rp4-vlc-3.0.4...
postPatchMkspecs
/nix/store/l4h7sqblghs5nnb4aa7jh6mip20p1rp4-vlc-3.0.4

On aarch64:
checking for live555... no
configure: WARNING: No package 'live555' found.
checking for live555 version<PHONE_NUMBER> or later... no
configure: WARNING: live555 is missing or its installed version is too old:
Version 2011.12.23 or later is required to proceed.
You can get an updated one from http://www.live555.com/liveMedia .
checking for libdc1394-2 >= 2.1.0... yes

On x64-linux (same behaviour as previous Hydra builds):
checking for live555... no
configure: WARNING: No package 'live555' found.
checking for live555 version<PHONE_NUMBER> or later... yes
checking for main in -lliveMedia_pic... no
checking for main in -lliveMedia... yes
checking for libdc1394-2 >= 2.1.0... yes

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/how-to-make-rpi4-aarch64-run-vlc-with-rtsp/14837/1


make.js: Set bundleBuild to 'unknown' if 'git log' fails

This attempts to fix #5920.

I'm not sure if this needs to be fixed on the PDF.js side. The error indicates to me that the Git clone is somehow not properly set up...

I'm not sure if this needs to be fixed on the PDF.js side.

I agree!
Given the amount of references to git in make.js, we would really have to "guard" all of them to prevent any possible issues (and I don't think we want to do that). Without git properly setup, there's a lot of things that just won't work as intended (or only work partially) in make.js.
(I'm actually wondering if we perhaps should throw an error at the top of the file, and prompt the user to setup/configure git when it's not available.)

The error indicates to me that the Git clone is somehow not properly set up...

There's also issue #5403, which is a (somewhat) similar issue.

seems reasonable! thanks for your feedback :)


Ignore certain Tekton annotations when creating resources

Expected Behavior
When creating a TaskRun or PipelineRun resource, any annotation that starts with *.tekton.dev/ should be removed.
Actual Behavior
Chains completely ignores the resource because
Steps to Reproduce the Problem

Create a TaskRun or PipelineRun with the following annotation: chains.tekton.dev/signed: 'true'
Notice that Chains completely ignores the resource.

Furthermore, other annotations, like chains.tekton.dev/transparency, can cause confusion to users.
This is, of course, not isolated to Chains. Tekton Results is also affected by this issue.
Additional Info

Kubernetes version:
Output of kubectl version:

🐚 kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.24.2
Kustomize Version: v4.5.4
Server Version: v1.25.4+77bec7a

Tekton Pipeline version:
Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

🐚 tkn version
Client version: 0.29.0
Chains version: v0.13.1
Pipeline version: v0.41.1
Triggers version: v0.22.2
Operator version: v0.63.0

/assign

@lcarva I'm a bit confused why this is something that should be addressed in Pipelines rather than in chains/results? I'm not familiar with how chains deals with annotations

@lcarva I'm a bit confused why this is something that should be addressed in Pipelines rather than in chains/results? I'm not familiar with how chains deals with annotations

@lbernick , Chains doesn't have an AdmissionController so it doesn't have a chance to reject/modify such resources. By the time Chains sees the resource, it's too late. I think something in the Tekton ecosystem should probably do this, and Tekton Pipeline seems like the right place. Wdyt?

That is my reading too. Note that #6441 tries to handle this for the "known cases" (aka anything with *tekton.dev/*) but that could as well be the same for any other tool (outside of tekton) that uses annotations to drive some behavior. A follow-up to #6441 could be to have a way to configure those "ignores" 🙃.

I think that sounds reasonable!

I think this makes sense, but I would not categorise it as bug, it looks more like a new feature to me.

Pipelines WG - @vdemeester please move this to the next milestone if we cannot merge the fix in time


CSP file read failing fix

Edited CSP read function to use pystring to split, strip and format the lines being checked.

Previously the (line != "CSPLUTV100") would error when reading a CSP.
Writing was fine.
I thought it would be better to match the pattern matching in other
fileFormat source files.

Signed-off-by: Andrew Bunday<EMAIL_ADDRESS>

I had to change the code a bit as the existing unit test didn't work. 0dae004fc006284620424e0db7472b969a6e6f1b
Merged into master


Translate docs/installation page into Korean

What type of PR is this?
/kind content
/kind translation
Any specific area of the project related to this PR?
/area documentation
What this PR does / why we need it:
Translate Installation page(/docs/installation) to Korean.
Which issue(s) this PR fixes:
Fixes #275
Special notes for your reviewer:
@jmyung

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign lucperkins
You can assign the PR to them by writing /assign @lucperkins in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Preview page
👉 https://deploy-preview-280--falcosecurity.netlify.app/ko/docs/installation/

/assign

LGTM label has been added.  Git tree hash: 94e8bff700f5d5f2047f52769faab4758170cde5

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: leodido
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [leodido]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Thanks for approving! 🎉


edgectl install --ci does not work in windows

Describe the bug
edgectl install --ci does not work in Windows (I cannot tell if it does work in other OS)
To Reproduce
Steps to reproduce the behavior:

Execute edgectl install --ci in command line

Expected behavior
According to this, it should install ambassador in kubernetes without the host crd.
Versions (please complete the following information):

Ambassador: 1.2.0
Kubernetes environment : Azure
Version 1.5

NOTE:
I'm able to complete the installation with edgectl install yet it would be nice if --ci flag does work

Hi @katlimruiz, thank you for spotting this issue! We've tested this and it does not work on any OS at the moment. We are removing it to test and fix, and will return it back to the documentation once it is fixed. thank you!


return rows that are empty

the function $reader->get()
return rows even empty

Same problem. I dont want to run a loop to disregard the null data. Library should be able to do that

$reader->ignoreEmpty()

Still leaves a few null keys.
[
      {
        "id": "11K81A0558",
        "email"<EMAIL_ADDRESS>        "role": "Administrator"
      },
      {
        "id": "11K81A0559",
        "email"<EMAIL_ADDRESS>        "role": "Staff"
      },
      {
        "id": null
      },
      {
        "id": null,
        "email": null
      }
    ]

This packages just wraps PHPExcel, nothing we can do about that.


chore(main): release 1.1.3

:robot: I have created a release beep boop
1.1.3 (2024-09-26)
Bug Fixes

compatibility with polars > 0.20.30 (738fe8d)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/BLSQ/openhexa-toolbox/releases/tag/v1.1.3 :sunflower:


Pre-release and model run checks

We don't want to release models without doing run testing first. In order to test a model we need to deploy it in some way first.
This is a chicken and egg or perhaps a cart before the horse problem.
We can solve this by having a pre-release staging area where we automatically deploy models without run testing, and then trigger a pull-request to https://github.com/ACCESS-NRI/access-om2-configs/ with updated model versions (exe paths) to the deployed model in pre-release.
(TBC)

Dupe of https://github.com/ACCESS-NRI/ACCESS-OM2/issues/5 so made this a comment there.


fix: set diagd gunicorn keepalive to 300s

Description
The Problem
Intermittent 503's from the /ambassador/v0/check_alive endpoint even though gunicorn/diagd is healthy.
This is to help avoid 503's on the /ambassador/v0/check_alive and similar endpoints. Currently, use-cases that probe this health check are subject to race-conditions where gunicorn closes idle TCP connections after 2s and does not send a FIN or RST to envoy. Envoy, believing the tcp connection is still active, attempts to route the check_alive request to gunicorn, but receives a RST as the TCP connection is no longer valid according to gunicorn. This results in a failing health check even though gunicorn is 'up'. This false-positive failure mode is problematic especially in situations where one is leveraging this health check to detect ambassador health from a CDN or other downstream load balancer.
See Wireshark image of the default 2s tcp idle timeout from gunicorn, and the implications of this low default. Our CDN is probing the backend between once every 500ms and 2.5s. When the HTTP probes do this pod are more than 2s apart, the above race condition is triggered and Envoy reports receiving a 503 from gunicorn, indicating a 'failed' health check.

Setting the keepalive timeout to 300s should be a safe default as typical implementations will probably be polling this health endpoint more than once every 5 minutes.
Related Issues
N/A
Testing
I've manually tested & captured this in Production some time ago, but since have upgraded back to mainline ambassador and am no longer running this fork. Would love to get this change merged into Ambassador to fix my 503 problem 😬 .
Checklist

[x] I made sure to update CHANGELOG.md.
Remember, the CHANGELOG needs to mention:

Any new features
Any changes to our included version of Envoy
Any non-backward-compatible changes
Any deprecations

[x] This is unlikely to impact how Ambassador performs at scale.
Remember, things that might have an impact at scale include:

Any significant changes in memory use that might require adjusting the memory limits
Any significant changes in CPU use that might require adjusting the CPU limits
Anything that might change how many replicas users should use
Changes that impact data-plane latency/scalability

[x] My change is adequately tested.
Remember when considering testing:

LEGACY MODE TESTS DO NOT RUN FOR EVERY PR. If your change is affected by legacy mode, you need
to run legacy-mode tests manually (set AMBASSADOR_LEGACY_MODE=true and run the tests).
(This will be fixed soon.)
Your change needs to be specifically covered by tests.

Tests need to cover all the states where your change is relevant: for example, if you add a behavior that can be enabled or disabled, you'll need tests that cover the enabled case and tests that cover the disabled case. It's not sufficient just to test with the behavior enabled.

You also need to make sure that the entire area being changed has adequate test coverage.

If existing tests don't actually cover the entire area being changed, add tests.
This applies even for aspects of the area that you're not changing – check the test coverage, and improve it if needed!

We should lean on the bulk of code being covered by unit tests, but...
... an end-to-end test should cover the integration points

[x] I updated DEVELOPING.md with any any special dev tricks I had to use to work on this code efficiently.

@esmet @kflynn @acookin @rhs I'd appreciate your review and feedback on this. If further clarification is needed on how this solves my issue let me know what further detail you'd like. Let me know if you have any questions.
Thanks

This keepalive would be super helpful, any chance of getting this feature merged in?

Superseded by #4008 -- many thanks for the original, @neufeldtech!


Run minikube in Travis CI

We have a stub created for testing all Pull Requests in minikube. With the change in apache/incubator-openwhisk-deploy-kube#5, we are able to run OpenWhisk in minikube. We need to implement this in our Traivs CI testing.

It looks like this may not be possible. See failures in this Travis run:
E0605 21:52:41.509372    4641 start.go:127] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: virError(Code=1, Domain=20, Message='internal error: no supported architecture for os type 'hvm'').
 Retrying.
E0605 21:52:41.509779    4641 start.go:133] Error starting host:  Error creating host: Error creating machine: Error in driver during machine creation: virError(Code=1, Domain=20, Message='internal error: no supported architecture for os type 'hvm'')

Oh, is your OpenWhisk working on Minikube? Right now after I changed the API version to 1.23 and completed the openwhisk config job, my Invoker failed to ping the controller when I try to make a wsk action.

@Tomcli: I wasn't able to get the config job to complete in my testing, so I figured I would start working on getting Travis testing to work in my add_minikube_tests branch. Based on my testing (see log output in my last comment), Minikube is not going to work in Travis CI with KVM as the docker-machine-driver. I can say from past experience that VirtualBox doesn't work in Travis CI.
Today, I plan on looking into alternative options.

@Tomcli You need to minikube ssh and sudo ip link set docker0 promisc on I believe. Several members of my team have successfully run OpenWhisk on both minikube and minishift if you need additional help debugging things.

@bbrowning: The output from Travis CI shows: internal error: no supported architecture for os type 'hvm' when running minikube start --vm-driver kvm, and I know from past attempts that VirtualBox is not an option in Travis CI.
In the apache/incubator-openwhisk-deploy-kube, I found that this is a problem that has already been solved. I am about to start testing with a configuration based of the scripts in apache/incubator-openwhisk-deploy-kube's .travis.yml and the scripts in apache/incubator-openwhisk-deploy-kube's .travis/.

@bbrowning Thank you, that fixes my problem. I think we can start adding instructions for Minikube then.

@mlangbehn Yes - I'm not aware of any way to get virtual machines running on top of Travis CI. That may be a blocker for running minikube itself in a Travis job.

Branch where I am working to get Travis CI testing on all branches and pull requests:
https://github.com/mlangbehn/OpenWhisk-on-Kubernetes/tree/split_out_travis_install_script

May I suggest that you try kubeadm-dind-cluster? It works on Travis and CircleCI rather well (it has it's own test suite running on Travis plus we're using it to run Virtlet tests). Unlike Minikube it doesn't need VMs and unlike hack/local-up-cluster.sh you don't have to build Kubernetes from source.

Hi, I found this issue by coincidence (I am now looking how to run MiniShift in Travis).
Anyway, I found this script, that I adapted and it lets you run Minikube in Travis.
Give it a try :)


Camel - Add properties view for compoents

like we have for route and endpoints. We should have the same for components.

And we got this now also - A little nav bar cleanup is needed as a change in workspace is needed to know if its component vs endpoint etc.


Fix/i18 deployment

Fixes for some issues discovered in I18 deployment testing:

PositionLabellerPart not uploading additional points after initial 5,000, and now waits until unique ID reaches last ID expected using same logic as DetectorDriverPart. This was working using the AreaDetector simulator with pausing, breakpoints and pausing with breakpoints
Savu kinematics can now scan without any axes in AxesToMove for no motion and added NeXus templates to the package data
Cast data type in OdinWriterPart to lower case for NumPy V1.2 update

Then for convenience I added a basic block for the UnrollingPart so that it could be enabled/disabled in a design based on visibility. This allows the same scan block YAML to be used by Odin and non-Odin detectors.

I found additional issues when using a software-triggered detector and outer axes. If I paused part way through an inner scan, then the number of images to take would be updated for the remainder of that inner scan in Seek and then not be reset in PostRunArmed. This should now be fixed.
I want to do more testing with hardware triggering with scans with both breakpoints and outer axes with pausing to confirm these are all working together.


[BUG]: Error when loading pre-trained Pegasus model through transformers

🐛 Describe the bug
When I tried to load a pre-trained Pegasus model through transformers, it seemed that an error was encountered when initializing one of the Embedding layers.
Here is the Traceback:
Building prefix dict from /home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/jieba/dict.txt ...
[08/03/22 10:38:47] DEBUG    colossalai - jieba - DEBUG: Building prefix dict
                             from /home/liuzhaofeng/anaconda3/lib/python3.9/site
                             -packages/jieba/dict.txt ...
Dumping model to file cache /tmp/jieba.cache
[08/03/22 10:38:49] DEBUG    colossalai - jieba - DEBUG: Dumping model to file
                             cache /tmp/jieba.cache
Dump cache file failed.
Traceback (most recent call last):
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/jieba/__init__.py", line 100, in initialize
    replace_file(fpath, cache_file)
PermissionError: [Errno 1] Operation not permitted: '/tmp/tmpp3dplm6z' -> '/tmp/jieba.cache'
[08/03/22 10:38:50] ERROR    colossalai - jieba - ERROR: Dump cache file failed.
                             ╭─────── Traceback (most recent call last) ───────╮
                             │ /home/liuzhaofeng/anaconda3/lib/python3.9/site- │
                             │ packages/jieba/__init__.py:100 in initialize    │
                             │                                                 │
                             │    97 │   │   │   │   │   from shutil import mo │
                             │    98 │   │   │   │   else:                     │
                             │    99 │   │   │   │   │   replace_file = os.ren │
                             │ ❱ 100 │   │   │   │   replace_file(fpath, cache │
                             │   101 │   │   │   except:                       │
                             │   102 │   │   │   │   logger.exception("Dump ca │
                             │   103                                           │
                             ╰─────────────────────────────────────────────────╯
                             PermissionError: [Errno 1] Operation not permitted:
                             '/tmp/tmpp3dplm6z' -> '/tmp/jieba.cache'
Loading model cost 3.289815902709961 seconds.
                    DEBUG    colossalai - jieba - DEBUG: Loading model cost
                             3.289815902709961 seconds.
Prefix dict has been built succesfully.
                    DEBUG    colossalai - jieba - DEBUG: Prefix dict has been
                             built succesfully.
[08/03/22 10:38:51] INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Added key: store_based_barrier_key:1 to store
                             for rank: 0
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Rank 0: Completed store-based barrier for
                             key:store_based_barrier_key:1 with 1 nodes.
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Added key: store_based_barrier_key:2 to store
                             for rank: 0
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Rank 0: Completed store-based barrier for
                             key:store_based_barrier_key:2 with 1 nodes.
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Added key: store_based_barrier_key:3 to store
                             for rank: 0
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Rank 0: Completed store-based barrier for
                             key:store_based_barrier_key:3 with 1 nodes.
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Added key: store_based_barrier_key:4 to store
                             for rank: 0
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Rank 0: Completed store-based barrier for
                             key:store_based_barrier_key:4 with 1 nodes.
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Added key: store_based_barrier_key:5 to store
                             for rank: 0
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Rank 0: Completed store-based barrier for
                             key:store_based_barrier_key:5 with 1 nodes.
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Added key: store_based_barrier_key:6 to store
                             for rank: 0
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Rank 0: Completed store-based barrier for
                             key:store_based_barrier_key:6 with 1 nodes.
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Added key: store_based_barrier_key:7 to store
                             for rank: 0
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Rank 0: Completed store-based barrier for
                             key:store_based_barrier_key:7 with 1 nodes.
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Added key: store_based_barrier_key:8 to store
                             for rank: 0
                    INFO     colossalai - torch.distributed.distributed_c10d -
                             INFO: Rank 0: Completed store-based barrier for
                             key:store_based_barrier_key:8 with 1 nodes.
                    INFO     colossalai - colossalai - INFO: /home/liuzhaofeng/a
                             naconda3/lib/python3.9/site-packages/colossalai/con
                             text/parallel_context.py:521 set_device
                    INFO     colossalai - colossalai - INFO: process rank 0 is
                             bound to device 0
[08/03/22 10:39:00] INFO     colossalai - colossalai - INFO: /home/liuzhaofeng/a
                             naconda3/lib/python3.9/site-packages/colossalai/con
                             text/parallel_context.py:557 set_seed
                    INFO     colossalai - colossalai - INFO: initialized seed on
                             rank 0, numpy: 1024, python random: 1024,
                             ParallelMode.DATA: 1024, ParallelMode.TENSOR:
                             1024,the default parallel seed is
                             ParallelMode.DATA.
                    INFO     colossalai - colossalai - INFO: /home/liuzhaofeng/a
                             naconda3/lib/python3.9/site-packages/colossalai/ini
                             tialize.py:117 launch
                    INFO     colossalai - colossalai - INFO: Distributed
                             environment is initialized, data parallel size: 1,
                             pipeline parallel size: 1, tensor parallel size: 1
Traceback (most recent call last):
  File "/tmp/pycharm_project_471/pegasus_train.py", line 104, in <module>
    model = PegasusForConditionalGeneration.from_pretrained(hf_model_name)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/modeling_utils.py", line 1843, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/colossalai/utils/model/utils.py", line 52, in wrapper
    f(module, *args, **kwargs)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1294, in __init__
    self.model = PegasusModel(config)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/colossalai/utils/model/utils.py", line 52, in wrapper
    f(module, *args, **kwargs)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1140, in __init__
    self.encoder = PegasusEncoder(config, self.shared)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/colossalai/utils/model/utils.py", line 52, in wrapper
    f(module, *args, **kwargs)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 653, in __init__
    self.embed_positions = PegasusSinusoidalPositionalEmbedding(
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/colossalai/utils/model/utils.py", line 52, in wrapper
    f(module, *args, **kwargs)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 114, in __init__
    self.weight = self._init_weight(self.weight)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 122, in _init_weight
    n_pos, dim = out.shape
ValueError: not enough values to unpack (expected 2, got 1)

Although there is an error when loading the jieba package at the top, I don't think this is the reason for the error below, because this problem will also occur when ColossalAI is not used, and the file permission problem does not affect the fine-tuning of my model.
The model I want to fine-tune is Pegasus open sourced by IDEA on HuggingFace: https://huggingface.co/IDEA-CCNL/Randeng-Pegasus-238M-Summary-Chinese
It should be noted that they customized a tokenizer: https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/main/fengshen/examples/pegasus
Below is my script:
import colossalai
import torch as th

from torch.utils.data import Dataset, DataLoader
from colossalai.utils import get_current_device
from colossalai.core import global_context as gpc
from colossalai.zero.init_ctx import ZeroInitContext
from transformers import AdamW, get_scheduler, PegasusForConditionalGeneration

from tokenizers_pegasus import PegasusTokenizer

max_input_length = 512
max_target_length = 256
train_batch_size = 4
test_batch_size = 4
learning_rate = 2e-5
epoch_num = 8
beam_size = 4
no_repeat_ngram_size = 2

colossalai.launch_from_torch(config="./configs/colossalai_zero.py")
hf_model_name = "IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese"
tokenizer = PegasusTokenizer.from_pretrained(hf_model_name)

class SummaryDataset(Dataset):
	def __init__(self, data_file):
		self.data = None
		self.data_file = data_file
		self.load_data()

	def load_data(self):
		data_list = []
		with open(self.data_file, 'rt', encoding='utf-8') as f:
			for idx, line in enumerate(f):
				items = line.strip().split(',')
				if len(items) == 2:
					data_list.append({
						'title': items[0],
						'content': items[1]
					})
		self.data = data_list

	def __len__(self):
		return len(self.data)

	def __getitem__(self, idx):
		return self.data[idx]

def collote_fn(batch_samples):
	batch_inputs, batch_targets = [], []
	for sample in batch_samples:
		batch_inputs.append(sample['content'])
		batch_targets.append(sample['title'])
	batch_data = tokenizer(
		batch_inputs,
		padding=True,
		max_length=max_input_length,
		truncation=True,
		return_tensors="pt"
	)
	with tokenizer.as_target_tokenizer():
		labels = tokenizer(
			batch_targets,
			padding=True,
			max_length=max_target_length,
			truncation=True,
			return_tensors="pt"
		)["input_ids"]
		batch_data['decoder_input_ids'] = model.prepare_decoder_input_ids_from_labels(labels)
		end_token_index = th.where(labels == tokenizer.eos_token_id)[1]
		for idx, end_idx in enumerate(end_token_index):
			labels[idx][end_idx + 1:] = -100
		batch_data['labels'] = labels
	return batch_data

if __name__ == '__main__':
	train_data = SummaryDataset("dataset/summary/train.csv")
	valid_data = SummaryDataset("dataset/summary/test.csv")

	train_dataloader = DataLoader(train_data, batch_size=train_batch_size, shuffle=True, collate_fn=collote_fn)
	valid_dataloader = DataLoader(valid_data, batch_size=test_batch_size, shuffle=False, collate_fn=collote_fn)

	with ZeroInitContext(target_device=get_current_device(), shard_strategy=gpc.config.zero.model_config.shard_strategy,
	                     shard_param=True):
		model = PegasusForConditionalGeneration.from_pretrained(hf_model_name)
		optimizer = AdamW(model.parameters(), lr=learning_rate)
		lr_scheduler = get_scheduler(
			"linear",
			optimizer=optimizer,
			num_warmup_steps=0,
			num_training_steps=gpc.config.NUM_EPOCHS * len(train_dataloader),
		)

	engine, train_dataloader, eval_dataloader, lr_scheduler = colossalai.initialize(model=model,
	                                                                               optimizer=optimizer,
	                                                                               train_dataloader=train_dataloader,
	                                                                               test_dataloader=valid_dataloader,
	                                                                               lr_scheduler=lr_scheduler)

	for epoch in range(gpc.config.NUM_EPOCHS):
		engine.train()
		for batch, batch_data in enumerate(train_dataloader, start=1):
			engine.zero_grad()

			batch_data = batch_data.to(get_current_device())
			outputs = model(**batch_data)
			loss = outputs.loss

			engine.backward(loss)
			engine.step()

		lr_scheduler.step()

		total_loss = 0
		engine.eval()
		for batch, batch_data in enumerate(eval_dataloader, start=1):
			engine.zero_grad()

			batch_data = batch_data.to(get_current_device())
			with th.no_grad():
				outputs = model(**batch_data)
				loss = outputs.loss
				total_loss += loss.item()

		print(f"epoch {epoch} loss {total_loss / len(eval_dataloader)}")

Environment
$ python --version
Python 3.9.12

$ pip list
colossalai                      0.1.8+torch1.10cu11.3
tokenizers                      0.12.1
torch                           1.10.0+cu113
transformers                    4.18.0

$ nvidia-smi
Wed Aug  3 13:09:42 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.129.06   Driver Version: 470.129.06   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:04:00.0 Off |                  N/A |
| 23%   28C    P8     8W / 250W |   2695MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:05:00.0 Off |                  N/A |
| 23%   26C    P8     9W / 250W |   3589MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:08:00.0 Off |                  N/A |
| 23%   26C    P8     8W / 250W |   3371MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:09:00.0 Off |                  N/A |
| 23%   26C    P8     9W / 250W |   1227MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA GeForce ...  Off  | 00000000:84:00.0 Off |                  N/A |
| 23%   27C    P8     8W / 250W |      8MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA GeForce ...  Off  | 00000000:85:00.0 Off |                  N/A |
| 23%   25C    P8     8W / 250W |      8MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA GeForce ...  Off  | 00000000:88:00.0 Off |                  N/A |
| 23%   27C    P8     9W / 250W |      8MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA GeForce ...  Off  | 00000000:89:00.0 Off |                  N/A |
| 23%   24C    P8     8W / 250W |      8MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   1326351      C   /home/venv/bin/python            2687MiB |
|    0   N/A  N/A   2146382      G   /usr/lib/xorg/Xorg                  4MiB |
|    1   N/A  N/A   2146382      G   /usr/lib/xorg/Xorg                  4MiB |
|    1   N/A  N/A   3715225      C   /home/venv/bin/python            3581MiB |
|    2   N/A  N/A   2146382      G   /usr/lib/xorg/Xorg                  4MiB |
|    2   N/A  N/A   3715763      C   /home/venv/bin/python            3363MiB |
|    3   N/A  N/A   1174978      C   /home/venv/bin/python            1219MiB |
|    3   N/A  N/A   2146382      G   /usr/lib/xorg/Xorg                  4MiB |
|    4   N/A  N/A   2146382      G   /usr/lib/xorg/Xorg                  4MiB |
|    5   N/A  N/A   2146382      G   /usr/lib/xorg/Xorg                  4MiB |
|    6   N/A  N/A   2146382      G   /usr/lib/xorg/Xorg                  4MiB |
|    7   N/A  N/A   2146382      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

Hi @koking0,
I'm very sorry to tell you that the old version of ZeRO is not capable to handle the Pegasus model.
Because it slices each parameter during the construction of the model, every parameter is changed to a unidimensional partial tensor after initializatioin. This means that it is not legal to use some functions, such as shape and size, to get the shape of any parameter in the initialization of the model.
If you want to run the Pegasus model properly, I suggest to use our new version of ZeRO.
Though the new ZeRO has some performance issues now, it can tackle the problem mentioned above.
You can try it out. Just write initilaization like this:
    with ColoInitContext(device=torch.device('cpu')):
        model = ...

    chunk_size = ChunkManager.search_chunk_size(model, 64 * 1024 ** 2, 32)
    pg = ProcessGroup()
    placement_policy = 'auto'
    chunk_manager = ChunkManager(chunk_size, process_group=pg,
                                 enable_distributed_storage=True,
                                 init_device=GeminiManager.get_default_device(placement_policy))
    gemini_manager = GeminiManager(placement_policy, chunk_manager)
    model = ZeroDDP(model, gemini_manager)

    optimizer = HybridAdam(optimizer_grouped_parameters, lr=args.learning_rate)
    optimizer = ZeroOptimizer(optimizer, model, initial_scale=2 ** 14)

Here is an example about the OPT model.
Feel free to ask any question about your errors.

Hi @koking0,
I'm very sorry to tell you that the old version of ZeRO is not capable to handle the Pegasus model. Because it slices each parameter during the construction of the model, every parameter is changed to a unidimensional partial tensor after initializatioin. This means that it is not legal to use some functions, such as shape and size, to get the shape of any parameter in the initialization of the model.
If you want to run the Pegasus model properly, I suggest to use our new version of ZeRO. Though the new ZeRO has some performance issues now, it can tackle the problem mentioned above. You can try it out. Just write initilaization like this:
    with ColoInitContext(device=torch.device('cpu')):
        model = ...

    chunk_size = ChunkManager.search_chunk_size(model, 64 * 1024 ** 2, 32)
    pg = ProcessGroup()
    placement_policy = 'auto'
    chunk_manager = ChunkManager(chunk_size, process_group=pg,
                                 enable_distributed_storage=True,
                                 init_device=GeminiManager.get_default_device(placement_policy))
    gemini_manager = GeminiManager(placement_policy, chunk_manager)
    model = ZeroDDP(model, gemini_manager)

    optimizer = HybridAdam(optimizer_grouped_parameters, lr=args.learning_rate)
    optimizer = ZeroOptimizer(optimizer, model, initial_scale=2 ** 14)

Here is an example about the OPT model. Feel free to ask any question about your errors.

Process_group argument in the ChunkManager seems to not exists anymore. Is it true ?
Version of colossalai: 0.1.8+torch1.10cu10.2

Hi @AntoineBlanot,
Currently, we will keep the argument process_group in the ChunkManager for a while.
I don't know how you conclude that, could you tell me the reason why you have an opinion like that.

Hi @AntoineBlanot,
Currently, we will keep the argument, process_group, in the ChunkManager for a while. I don't know how you conclude the remove of the argument, could you tell me the reason why you have an opinion like that.

It seems like in my version of colossalai, processgroup is an argument of ZeroDDP and not ChunkManager.

Ok. We'll publish a new release soon. If you want to use the new version of ZeRO, you have to be in the newest release.


Small tweaks for clarity and easier maintainability.

Hey hey! I am learning how to get more comfortable with github so I can actually do more stuff and be more useful. This was the perfect repo to try out my hand at a pr! Below are the adjustments I made.

Added venv setup instructions for Linux
made a new dir under templates for javascript files
took out the inline js in index
chunked the inline code into two distinct files: chat.js for handling the actual api calling and messages, and event-listener.js to handle the event listener.
placed the references to those back into index

I figured this structure would be easier for people to read and understand exactly what this code is doing.

Thank you!


Translation

Hello. I apologize if the message is not in the correct format.
I offer my own version of Russian language support. Translated as closely as possible to the meaning in the context of the site.
The result of the work can be viewed on my website

I apologize if the message is not in the correct format.

That's no problem at all, I can squash the commit message. Thanks for your contributions!


Added script to update chatops integration setting

Added the script chatops-update.sh to help customer update integration settings for Chatops.

@stacyped ^^

@stacyped ^^

@stacyped Thanks for your review, I have updated the scripts with your comments, could you pls help to review again? Thank you.

@stacyped Thanks for your review, I have updated the scripts with your comments, could you pls help to review again? Thank you.

@stacyped could you pls help to review the updated script? Thank you!

@stacyped could you pls help to review the updated script? Thank you!


Updated external-provisioner to 3.2.1 for 1.20 - 1.23

Issue #, if available:
close #1071
Part of 1.20-19, 1.21-16, 1.22-9, and 1.22-3 releases in #1067
Description of changes:

Updates external-provisioner  for all compatible Kubernetes versions
Updated Go version, as they did too

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve


🐛 [4.0.0-beta.15] - Recording video with useSkiaFrameProcessor freezes my app

What's happening?
When I try to record video during frame processing using the new useSkiaFrameProcessor everything works fine, the problem occurs when I want to stop the video recording. At that point my application freezes and in some cases crashes.

All of this did work until I updated to beta.15 so i asume that is caused by this update.
I tried to reproduce it on example app but getting error: Error: Unable to resolve module @babel/runtime/helpers/interopRequireDefault from /Users/XYZ/Git/react-native-vision-camera/package/src/Camera.tsx: @babel/runtime/helpers/interopRequireDefault could not be found within the project.
Reproduceable Code
const frameProcessor = useSkiaFrameProcessor(frame => {
  'worklet';
  const detectedPose = poseDetection(
    frame,
    frameNumber.current,
  ) as SkeletonPose;
  frame.render();
  if (isRecording.current) {
    if (Object.values(detectedPose).length > 0) {
      const poseArray = getFramePoseData(
        detectedPose,
        frame.width,
        frame.height,
      );

      const pointsPaint = Skia.Paint();
      pointsPaint.setColor(Skia.Color('green'));
      pointsPaint.setAntiAlias(true);
      pointsPaint.setStyle(PaintStyle.Fill);
      for (let i = 0; i < poseArray.length; i++) {
        frame.drawCircle(
          poseArray[i].x * frame.width,
          poseArray[i].y * frame.height,
          10,
          pointsPaint,
        );
      }

      frame.restore();
      frameNumber.current++;
    } else {
      frame.render();
    }
  }
}, []);

<ReanimatedCamera
  ref={cameraRef}
  frameProcessor={frameProcessor}
  style={StyleSheet.absoluteFill}
  device={device}
  video={true}
  pixelFormat="rgb"
  isActive
  fps={30}
  format={format}
  preview={false}
/>

Relevant log output
15:35:41.555: [info] 📸 VisionCamera.startRecording(options:onVideoRecorded:onError:): Starting Video recording...
15:35:41.561: [info] 📸 VisionCamera.startRecording(options:onVideoRecorded:onError:): Will record to temporary file: /private/var/mobile/Containers/Data/Application/A47FCB4D-79DE-4162-98E5-A12594FFE8FF/tmp/ReactNative/61032842-6B19-4449-84E1-4DF712515D5D.mov
15:35:41.568: [info] 📸 VisionCamera.initializeVideoWriter(withSettings:): Initializing Video AssetWriter with settings: ["AVVideoCompressionPropertiesKey": {
    AllowFrameReordering = 1;
    AllowOpenGOP = 1;
    AverageBitRate = 4847616;
    ExpectedFrameRate = 30;
    MaxAllowedFrameQP = 41;
    MaxKeyFrameIntervalDuration = 1;
    MinAllowedFrameQP = 15;
    MinimizeMemoryUsage = 1;
    Priority = 80;
    ProfileLevel = "HEVC_Main_AutoLevel";
    RealTime = 1;
    RelaxAverageBitRateTarget = 1;
}, "AVVideoCodecKey": hvc1, "AVVideoHeightKey": 1280, "AVVideoWidthKey": 720]
15:35:41.578: [info] 📸 VisionCamera.initializeVideoWriter(withSettings:): Initialized Video AssetWriter.
15:35:41.578: [info] 📸 VisionCamera.start(clock:): Starting Asset Writer(s)...
15:35:45.966: [info] 📸 VisionCamera.start(clock:): Asset Writer(s) started!
15:35:45.967: [info] 📸 VisionCamera.start(clock:): Started RecordingSession at time:<PHONE_NUMBER>28166
15:35:45.967: [info] 📸 VisionCamera.startRecording(options:onVideoRecorded:onError:): RecordingSesssion started in 4411.938375ms!

nw_socket_handle_socket_event [C20.1.1:1] Socket SO_ERROR 54
nw_protocol_socket_reset_linger [C20.1.1:1] setsockopt SO_LINGER failed 22

15:35:49.452: [info] 📸 VisionCamera.stop(clock:): Requesting stop at<PHONE_NUMBER>44666 seconds for AssetWriter with status "writing"...
nw_socket_handle_socket_event [C21.1.1:1] Socket SO_ERROR 54
nw_protocol_socket_reset_linger [C21.1.1:1] setsockopt SO_LINGER failed 22
SocketStream read error [0x301425d60]: 1 54
15:35:53.563: [error] 📸 VisionCamera.stop(clock:): Waited 4.0 seconds but no late Frames came in, aborting capture...
15:35:53.563: [info] 📸 VisionCamera.finish(): Stopping AssetWriter with status "writing"...
15:35:53.587: [info] 📸 VisionCamera.startRecording(options:onVideoRecorded:onError:): RecordingSession finished with status completed.

Error occurred: The operation couldn’t be completed. (OSStatus error -50.)

(lldb)

Camera Device
{
  "hasTorch": true,
  "hardwareLevel": "full",
  "maxZoom": 123.75,
  "isMultiCam": false,
  "position": "back",
  "maxExposure": 8,
  "supportsLowLightBoost": false,
  "supportsRawCapture": false,
  "supportsFocus": true,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "minZoom": 1,
  "name": "Back Camera",
  "hasFlash": true,
  "minExposure": -8,
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "sensorOrientation": "landscape-right",
  "minFocusDistance": 12,
  "formats": []
}

Device
iPhone 12 Mini (iOS 17.4.1)
VisionCamera Version
4.0.0-beta.15
Can you reproduce this issue in the VisionCamera Example app?
I didn't try (⚠ your issue might get ignored & closed if you don't try this)
Additional information

[ ] I am using Expo
[X] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

good feature request, will look into this soon

I think this should fix your issue: https://github.com/mrousavy/react-native-vision-camera/pull/2754 although I haven't tested it yet.
Wanna pull in the changes (e.g. via patch-package) and try if it works? You can ignore android/ changes if you only test iOS obviously.

Thank you, I will try to patch that :)

fixed in #2754.


Indent levels of headings?

Hi, maybe I'm just missing the option, I'd like to format my different levels of headings not via font-size but via indention:

Is that possible and if yes, what option do I need to use?

That is not something the theming system supports.
What you can do is set indentation on the section body. https://github.com/asciidoctor/asciidoctor-pdf/blob/v1.6.x/docs/theming-guide.adoc#keys-section

In the future, please ask questions about usage and use cases in the community chat at https://asciidoctor.zulipchat.com. Thanks!

Okay, thanks! I've managed to get it done (for all headings) using a negative section.indent and a positive page.margin.

Nice trick! I suppose we could mention that in the theming guide.


[coin.move] CoinMetadata getter functions

Currently, CoinMetadata can be accessed for off-chain consumption, however there is a class of use-cases where we would like to consume the metadata fields (decimals, name, symbol, etc.) on-chain. For example, a protocol might want to perform checks on the metadata corresponding to a CoinType on-chain, e.g. check that decimals==8. Currently this is impossible because the metadata fields are opaque. The following getters allow inspection of CoinMetadata fields.

Thanks for the contribution @optke3!
The test failures come from our snapshot tests for genesis and gas costs which have changed because your change affects the framework library (which is published as part of genesis, and loaded by the transactions that calibrate gas costs).  To fix these please:

Rebase your commit (there were some recent changes for gas costing).
Go to the sui-cost crate (sui/crates/sui-cost), run cargo insta test --review, and accept the changes.
Go to the sui-config crate (sui/crates/sui-config), run cargo insta test --review there, and accept those changes as well.
Commit the snapshot changes.

You may need to install insta.rs for this as well (cargo install cargo-insta).


Closes #90. Update dev files

This PR updates the docker-based and vagrant-based development scripts, configuration, and READMEs to reflect the current state of cultivar on develop. This PR falls just short of making the Vagrant set up full-service since it doesn't set up the messaging backend, but at least now it has a site.
How to test
You might want to start TOTALLY from scratch (re: Vagrant install, Docker install) but if not that far back, at least get rid of your old vagrant box and docker images just to be safe.
Follow the README sections from this PR on the dev environment with

Docker
Vagrant
and confirm for each in turn that the site really goes up where the README says it will

You can also smoke test the messaging backend for the docker configuration by triggering a background tasks via uploading a data set and follow logs for celery (and rabbitmq if you like) to make sure that background tasks are being executed by workers to prove out the messaging backend
NOTE: I considered setting up an emailing backend for dev not worth fixing from a docker perspective (and it's already non existent from the Vagrant perspective) since for dev purposes we have no email configuration necessary at this point (no email logging and we don't appear to use it elsewhere in the project).
Implementation notes/notable things
To fix the Vagrant setup:

This repo is not python 2.7 safe anymore as dicussed in #90, but the default python executable for the base vagrant box we're using (ubuntu/trusty32) points to python 2.7. So I switched the default by having the virtualenvironment the Vagrantfile provisions use the python3 executable (which is really python 3.4.1 in this ubuntu version, but we can deal with upgrades later if we want).

To fix the Docker setup:

There was legacy stuff related to redis instead of rabbitmq as the backend, so switched it all over to rabbitmq which is what the django settings imply.
Config management was off so I created a separate sample env file that handles the docker case, following the same env file antipattern as bare metal or vagrant dev installs
Took out some irrelevant/configuration-complicating pieces from entrypoints (or entire entrypoints altogether) from the django and celery service containers, and added a wait loop so the django service will wait for postgres to allow connections since runserver won't exit or otherwise kick runserver again on a connection refused error while the postgres container is still starting up.
Dropped configuring an smpt server since I decided the email backend was irrelevant for dev purposes right now

Overall fixes:

There was a duplicate requirement in one of the inherited requirement files that pip was freaking out on for celery installs
The README directions for adding null values to .env files were thinking pythony (ENV_VAR=None) instead of bashy (ENV_VAR=1) which is not ze same

Out of scope

I didn't touch the 'prod' docker setup (docker-compose.yml)
By my eye there is some complexity in the docker configuration that could be removed/streamlined, but only really touched the configuration-complexity introducing entrypoints for the celery and django service this round
I didn't add any more db seeding/migration seeding then was already present

@bahadasx if you want to take a look at this and see if the dev instructions work for you, I think this would be good to merge before we get anyone else into the codebase. I put testing instructions and an overview, but can go over it with you/anyone tomorrow(/today....) also cc @ojedatony1616 @bbengfort


Duplicate new message loaders

Duplicates each time you send a message

I believe I fixed this issue. My branch