SecureSocket handshake can stall the main thread for long time

In my application, the SDK used before was 1.9 and everything worked fine. After upgrading to 1.12, android still works well, but ios will get stuck on the splash screen.
Detailed questions:

Sometimes the first time you start it will get stuck on the splash screen.
Manually kill the process and then enter the app, there is a high possibility that it will get stuck on the splash screen page.

Code situation

pubspec.yaml
http: 0.12.0+4
lib/SplashPage.dart

class SplashPage extends StatefulWidget {
  @override
  State<StatefulWidget> createState() {
    return _StdPageState();
  }
}

class _StdPageState extends State<SplashPage> {
  @override
  void initState() {
    super.initState();

    // send Http here
  }
}

lib/util/HttpsUtils.dart

import 'dart:convert' show utf8;
import 'package:flutter/material.dart';
import 'package:http/http.dart' as http;
import 'package:inlan_flutter/Config.dart';
import 'package:inlan_flutter/util/DataUtils.dart';
import 'package:inlan_flutter/util/I18nUtils.dart';
import 'package:inlan_flutter/util/JsonUtils.dart';
import 'package:inlan_flutter/util/LogUtils.dart';
import 'package:inlan_flutter/util/ToastUtils.dart';
import 'package:inlan_flutter/util/UpgradeUtils.dart';
import 'package:inlan_flutter/util/UuidUtils.dart';

class HttpsUtils {
  static void apiPostWithUrl(
      String url,
      Map<String, dynamic> body,
      BuildContext context,
      StackTrace callPositionStackTrace,
      State state,
      callback(json)) async {
    body["version"] = Config.apiVersion;
    body["httpId"] = UuidUtils.v4();
    body["lang"] = I18nUtils.languageCode();
    body["path"] = url;

    String fullUrl = Config.apiUrl + url;

    final apiClient = http.Client();

    try {
      await apiClient
          .post(fullUrl,
              headers: {"Content-Type": "application/json"},
              body: JsonUtils.encode(body))
          .then((response) {
        LogUtils.log(response.statusCode, StackTrace.current,
            tag: "statusCode");

        LogUtils.devLog(Config.apiUrl + url, callPositionStackTrace,
            tag: "apiUrl " + url);

        LogUtils.devLog(JsonUtils.prettyJson(body), callPositionStackTrace,
            tag: "req");

        LogUtils.devLog(
            JsonUtils.prettyStr(response.body), callPositionStackTrace,
            tag: "resp ${response.statusCode}");

        switch (response.statusCode) {
          case 200:
            callback(JsonUtils.parse(response.body));
            break;
          case 400:
            Map<String, dynamic> jsonObj = JsonUtils.parse(response.body);
            state.setState(() {
              ToastUtils.short(jsonObj["message"]);
            });
            break;
          case 401:
            Map<String, dynamic> jsonObj = JsonUtils.parse(response.body);
            state.setState(() {
              ToastUtils.withCallback(jsonObj["message"], () {
                DataUtils.setByService('login_data', '-1');
                DataUtils.setByService('isLogined', '-1');
                Navigator.pushNamedAndRemoveUntil(
                    context, '/account', (Route<dynamic> route) => false);
              });
            });
            break;
          case 403:
            break;
          case 406:
            UpgradeUtils().doForceUpgrade(context);
            break;
          case 422:
            Map<String, dynamic> jsonObj = JsonUtils.parse(response.body);
            state.setState(() {
              ToastUtils.short(jsonObj["message"]);
            });
            break;
        }
      });
    } finally {
      apiClient.close();
    }
  }
}

My temporary solution
  void initState() {
    super.initState();
    Timer timer = new Timer(new Duration(milliseconds: 1200), () {
      // send Http here
    });

  }

Expected solution

What causes the freeze?
A better solution?

I'm using HttpClient from dart:io. I tried  to set the badCertificateCallback, the program did not enter callback.
I've only had this kind of stuck situation since last month, and there are some people who have met earlier than me. It used to run well with the same code. This is only on IOS, Android is good.

I don't think there is enough information here for us to diagnose problem.
HTTP processing is done asynchronously so it can't just block your UI thread.
I would start by looking for exceptions in the logs.

Many people have encountered this situation.
Maybe this is the problem of iOS, I don't know.

this

I also have this problem, I tried the same request in Safari, It's also slow,
Server ENV:

Nginx + HTTPS(Let's Encrypt)
Client ENV:
Device: iPhone XR , iOS 13.4
Flutter 1.12.13+hotfix.9 • Tools • Dart 2.7.2

TEST:
My Flutter App:

http: response in 200ms,
https: about 6 - 10 seconds and the UI frozen, it freeze about 30 seconds when I send 5 request at the same time
Safari:
http: response in 200ms,
https: about 5 - 8 seconds, safari is not freeze

This is the stack trace,  I pause debug when the UI frozen

@CyrilHu could you try running the following code on your iOS device (replace url to point to your https server)
import 'dart:async';
import 'dart:developer' as developer;
import 'dart:io';

void log(String s) {
  print(s);
  developer.log(s);
}

Timer stallDetector() {
  final sw = Stopwatch()..start();
  return Timer.periodic(Duration(milliseconds: 5), (_) {
    if (sw.elapsedMilliseconds > 10) {
      log('EVENT LOOP WAS STALLED FOR ${sw.elapsedMilliseconds} ms');
    }
    sw.reset();
  });
}

void main() async {
  final url = 'https://gank.io';
  final timer = stallDetector();
  var sw = Stopwatch()..start();
  var httpClient = HttpClient();
  try {
    var request = await httpClient.getUrl(Uri.parse(url));
    var response = await request.close();
    log('REQUEST COMPLETE IN ${sw.elapsedMilliseconds} ms');
  } finally {
    httpClient.close(force: true);
    timer.cancel();
  }
}

what does it print back?

@CyrilHu could you try running the following code on your iOS device (replace url to point to your https server)
what does it print back?

https test 1:
flutter: EVENT LOOP WAS STALLED FOR 48 ms
[log] EVENT LOOP WAS STALLED FOR 48 ms
flutter: EVENT LOOP WAS STALLED FOR 24 ms
[log] EVENT LOOP WAS STALLED FOR 24 ms
flutter: EVENT LOOP WAS STALLED FOR 30 ms
[log] EVENT LOOP WAS STALLED FOR 30 ms
[log] EVENT LOOP WAS STALLED FOR 3274 ms
flutter: EVENT LOOP WAS STALLED FOR 3274 ms
[log] EVENT LOOP WAS STALLED FOR 62 ms
flutter: EVENT LOOP WAS STALLED FOR 62 ms
flutter: REQUEST COMPLETE IN 3967 ms
[log] REQUEST COMPLETE IN 3967 ms
https test 2:
flutter: EVENT LOOP WAS STALLED FOR 124 ms
[log] EVENT LOOP WAS STALLED FOR 124 ms
flutter: EVENT LOOP WAS STALLED FOR 29 ms
[log] EVENT LOOP WAS STALLED FOR 29 ms
[log] EVENT LOOP WAS STALLED FOR 34 ms
flutter: EVENT LOOP WAS STALLED FOR 34 ms
flutter: EVENT LOOP WAS STALLED FOR 2744 ms
[log] EVENT LOOP WAS STALLED FOR 2744 ms
[log] EVENT LOOP WAS STALLED FOR 39 ms
flutter: EVENT LOOP WAS STALLED FOR 39 ms
flutter: REQUEST COMPLETE IN 3079 ms
[log] REQUEST COMPLETE IN 3079 ms
http test 1:
flutter: EVENT LOOP WAS STALLED FOR 70 ms
[log] EVENT LOOP WAS STALLED FOR 70 ms
[log] EVENT LOOP WAS STALLED FOR 30 ms
flutter: EVENT LOOP WAS STALLED FOR 30 ms
[log] EVENT LOOP WAS STALLED FOR 90 ms
flutter: EVENT LOOP WAS STALLED FOR 90 ms
flutter: REQUEST COMPLETE IN 245 ms
[log] REQUEST COMPLETE IN 245 ms
http test 2:
flutter: EVENT LOOP WAS STALLED FOR 67 ms
[log] EVENT LOOP WAS STALLED FOR 67 ms
[log] EVENT LOOP WAS STALLED FOR 32 ms
flutter: EVENT LOOP WAS STALLED FOR 32 ms
[log] EVENT LOOP WAS STALLED FOR 68 ms
flutter: EVENT LOOP WAS STALLED FOR 68 ms
[log] REQUEST COMPLETE IN 227 ms
flutter: REQUEST COMPLETE IN 227 ms

@mraleph
Is there a temporary solution? This is a very urgent problem!

Yeah, this seems pretty bad. Based on my cursory analysis of the code most likely cause of the the problem is the fact that we install a certificate verification callback (via SSL_CTX_set_cert_verify_callback) which uses SecTrustEvaluateWithError.
However SecTrustEvaluateWithError comes with the following warning:

Don’t call SecTrustEvaluateWithError from your app’s main run loop because it might require network access to fetch intermediate certificates, or to perform revocation checking. To perform evaluation asynchronously, use SecTrustEvaluateAsyncWithError instead.

Which seems to match the situation we are experiencing here - I think the stall occurs when SecTrustEvaluateWithError is attempting to check revocation status of the certificate. I suspect that you should be able to speed things up by configuring OCSP stapling.
I am extremely unfamiliar with this code (e.g. it is unclear to me why we use SSL_CTX_set_cert_verify_callback to verify the whole certificate chain, rather than using SSL_CTX_set_verify to verify individual certificates in the chain - the latter API supports asynchronous verification, while SSL_CTX_set_cert_verify_callback API is completely synchronous. If we must use SSL_CTX_set_cert_verify_callback - then we have to offload handshaking into a thread pool to make it asynchronous, otherwise we will stall UI thread too much.
@bkonyi @zanderso @dnfield you are the ones changing this code most. Can you take a look at this?
/cc @zichangg @sortie

@ghostgzt I think the only client side workaround I can offer is to use a separate isolate to handle networking and then send results back into the main isolate. This would prevent UI from stalling - but hand shake would still take very long time.

It looks like there's a SecTrustEvaluateAsyncWithError that we might be able to use here instead.
We were using SSL_CTX_set_cert_verify_callback as we needed to be able to access the SSLCertContext associated with the connection from the verification callback but SSL_CTX_set_verify doesn't allow for passing an argument to the callback. Can you point me to where in the documentation it says SSL_CTX_set_cert_verify_callback is synchronous whereas SSL_CTX_set_verify is not? Open/BoringSSL documentation is abysmal, so maybe I just missed that somewhere.

Can you point me to where in the documentation it says SSL_CTX_set_cert_verify_callback is synchronous whereas SSL_CTX_set_verify is not?

It turns out that I actually meant SSL_CTX_set_custom_verify rather than SSL_CTX_set_verify (this library is amazing 🙄).
// SSL_CTX_set_cert_verify_callback sets a custom callback to be called on
// certificate verification rather than |X509_verify_cert|. |store_ctx| contains
// the verification parameters. The callback should return one on success and
// zero on fatal error. It may use |X509_STORE_CTX_set_error| to set a
// verification result.
//
// The callback may use |SSL_get_ex_data_X509_STORE_CTX_idx| to recover the
// |SSL| object from |store_ctx|.
OPENSSL_EXPORT void SSL_CTX_set_cert_verify_callback(
    SSL_CTX *ctx, int (*callback)(X509_STORE_CTX *store_ctx, void *arg),
    void *arg);

enum ssl_verify_result_t BORINGSSL_ENUM_INT {
  ssl_verify_ok,
  ssl_verify_invalid,
  ssl_verify_retry,
};

// SSL_CTX_set_custom_verify configures certificate verification. |mode| is one
// of the |SSL_VERIFY_*| values defined above. |callback| performs the
// certificate verification.
//
// The callback may call |SSL_get0_peer_certificates| for the certificate chain
// to validate. The callback should return |ssl_verify_ok| if the certificate is
// valid. If the certificate is invalid, the callback should return
// |ssl_verify_invalid| and optionally set |*out_alert| to an alert to send to
// the peer. Some useful alerts include |SSL_AD_CERTIFICATE_EXPIRED|,
// |SSL_AD_CERTIFICATE_REVOKED|, |SSL_AD_UNKNOWN_CA|, |SSL_AD_BAD_CERTIFICATE|,
// |SSL_AD_CERTIFICATE_UNKNOWN|, and |SSL_AD_INTERNAL_ERROR|. See RFC 5246
// section 7.2.2 for their precise meanings. If unspecified,
// |SSL_AD_CERTIFICATE_UNKNOWN| will be sent by default.
//
// To verify a certificate asynchronously, the callback may return
// |ssl_verify_retry|. The handshake will then pause with |SSL_get_error|
// returning |SSL_ERROR_WANT_CERTIFICATE_VERIFY|.
OPENSSL_EXPORT void SSL_CTX_set_custom_verify(
    SSL_CTX *ctx, int mode,
    enum ssl_verify_result_t (*callback)(SSL *ssl, uint8_t *out_alert));

In SSL_CTX_set_cert_verify_callback case callback can return either success or failure, but in SSL_CTX_set_custom_verify it can return ssl_verify_retry which means verification is in progress asynchronously.

We were using SSL_CTX_set_cert_verify_callback as we needed to be able to access the SSLCertContext associated with the connection from the verification callback but SSL_CTX_set_verify doesn't allow for passing an argument to the callback.

Apparently there is an ex_data mechanism that allows you to associate arbitrary data with SSL object, see e.g. SSL_set_ex_data. This should allow us to use SSL_CTX_set_custom_verify.

We now have confirmation that my speculation about the source of the delay is correct, see screenshots in flutter/flutter#55586.
@bkonyi Ben, I am tentatively assigning to you because you have written the current version of the code, so you might be able to fix it faster than others.

I've found kinda temporary work around to increase the response speed with platform channel methods
https://github.com/dart-lang/sdk/issues/41451#issuecomment-619243422

I noticed similar behavior too with lets encrypt cert too. Haven't investigated deep into the issue yet.
I suspect if it's related to some kind of network blocking in China. The reasoning comes from that we didn't receive report from users out of China.

The OCSP server for lets encrypt (ocsp.int-x3.letsencrypt.org)  is blocked from China.
And in our business, our SLB service provider (ALIYUN) not support OCSP stapling.
We have change the https cert to solve this problem.

The OCSP server for lets encrypt (ocsp.int-x3.letsencrypt.org) is blocked from China.
And in our business, our SLB service provider (ALIYUN) not support OCSP stapling.
We have change the https cert to solve this problem.

But I also suspect the network is part of the problem. Looks like only some users using iOS flutter app have the problem. Other users on android or H5 platforms in similar network environment didn't report issue too.

But I also suspect the network is part of the problem. Looks like only some users using iOS flutter app have the problem. Other users on android or H5 platforms in similar network environment didn't report issue too.
If it's the network blocking to be blamed, why it doesn't happen to all users.

@rxwen  We also notice that Android users not affected by this issue. I guess there may some different default setting between iOS and Android. I am not sure.

@rxwen the code that causes stall is iOS specific, we don't have similar code on Android. That would explain why Android uses are unaffected.
(The code was added in 644862bf966e6c079460258807ab3364b7e3759a to make sure that BoringSSL uses system level root certificates)

@mraleph thanks for linking the docs for those methods. I'll take a look this week and see what I can do.

We first reported the same issue in https://github.com/flutter/flutter/issues/55586.
We found that there have been reports earlier this month that letsencrypt OCSP is experiencing troubles in China.
The problem severely affects app usability, so getting rid of the stall would definitely help a lot. Nonetheless, the delay of the overall request would still exist. So is there any way we can help devs notice such problems that certain step of communication is unreasonably slow? (so that they wouldn't mistakenly think flutter/dart networking is slow or unreliable :))

The OCSP server of letsencrypt (ocsp.int-x3.letsencrypt.org) has been DNS poisoned in China. So your server OCSP Stapling will not work, and the browser will send a request to OCSP server and lead to a lag.
You can run a OCSP proxy on your server to solve this problem.

I'm true it's a problem of certificate of https, most of the cases occured whiling using Let's Encrypt certificates. We can find the resolution in a issue: (https://github.com/flutterchina/dio/issues/703)
In letsencrypt's compatibility list, there is no flutter-platform: letsencrypt certificate compatibility.

#https://github.com/flutterchina/dio/issues/786

any news?

@ghostgzt Checkout previous posts. I don't think this is a problem for Dart.

The OCSP server of letsencrypt (ocsp.int-x3.letsencrypt.org) has been DNS poisoned in China. So your server OCSP Stapling will not work, and the browser will send a request to OCSP server and lead to a lag.
You can run a OCSP proxy on your server to solve this problem.

@zichangg I believe @mraleph and @bkonyi were looking at using some async methods to do the certificate resolution rather than the current sync ones. It wouldn't fix the underlying DNS poising issue, but it would avoid blocking UI for a Flutter application.

@zichangg while it is correct that certificate validation delay is indeed not Dart problem by itself (Safari has the same issue), dart:io should not be performing this verification on the main thread and stall UI by doing so. So this needs to be fixed.
I don't think either me or @bkonyi are actively working on this issue - so it falls on @zichangg's plate.

any new?
is there any plan to fix this on dart:io?

@a-siva @franklinyow - is this a candidate for the Sept milestone?

Team need more time for the fix, moved to Oct

@a-siva Who's working on this now?

@aam has picked this up.

Per discussion with @rmacnak-google the plan is to leave existing synchronous openssl/boringssl API in place, but have SSLFilter::Handshake() run on dart worker thread using IOService infrastructure that is used for similar synchronous os api calls(file, socket, etc)

One problem with running SSLFilter::Handshake on separate worker thread is its inability to invoke dart's onBadCertificate  synchronously during handshake.

One problem with running SSLFilter::Handshake on separate worker thread is its inability to invoke dart's onBadCertificate  synchronously during handshake.

SSLCertContext::CertificateCallback() running on an IOService thread can send a message to the isolate telling it to run the onBadCertificate, then block to wait for the result.

Thanks @zanderso . Yeah, that is a possibility, concern would be clean up of blocked IOService thread in case onBadCertificate isolate exits.
As a temporary solution I thought of having async handshake only in case when there is no onBadCertificate provided.
Yet another possibility is to use ssl_verify_retry return code Slava mentioned above with only SecTrustEvaluate part of CertificateVerificationCallback running on a separate thread(spawned via Dart_NewNativePort). This callback will check if it got a response from that thread for that certificate and if so it will synchronously call onBadCertificate (if provided), then return the response to continue ssl's handshake, otherwise it will send a request to that thread and return ssl_verify_retry to suspend ssl's handshake.

https://dart-review.googlesource.com/c/sdk/+/165520 with proposed fix.

Can we close this issue not that the CL has landed ?

dart-review.googlesource.com/c/sdk/+/165520 with proposed fix.

The fix landed without tests. Is it possible to add some to prevent regressions in the future?

https://github.com/dart-lang/sdk/issues/43674 to track adding a test.

which version fixed this problem?

So far this landed as 63852d2073c776d6d849e306b2e36274f2143eb3 in flutter
master channel(
https://github.com/flutter/flutter/commit/63852d2073c776d6d849e306b2e36274f2143eb3
).
On Sat, Oct 10, 2020 at 4:41 AM Gentle Kwan<EMAIL_ADDRESS>wrote:

which version fixed this problem?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/dart-lang/sdk/issues/41519#issuecomment-706535718,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC5BUOAIFGQ3MSB4DANNBDSKBB55ANCNFSM4MIZKDZA
.

I've got the same issue, with the above update,  I will have to test with what dart version, flutter version and what channel, thank you.

For anyone who is using 1.22.6 but not full prepared migrate into Flutter 2.0 still facing about this problem. My workaround below might help you.
Get  the current Flutter version path by using this command.
which flutter

open the path printed.
delete path ./cache
cd ./internal
then you need modify three file to change the flutter verison.
engine: 1d12d82d9cb54876f58044aa52198d53ec841c3d
fuchsia-linux: h-DeV4tgEk6jHzUlsoZri9lklKQJDEAk7UVXOEpNKFMC
fuchsia-mac: FFpTJfmj1EGxY5dJDoGNmZ4dwub3GsxHTJHDtKGmQcMC
now you can still using Flutter 1.22.6 but migrate flutter engine to flutter 1.23.0-18.1pre
If you can not compile the IOS app after mofications, try modify ./ios/PodFile
# Uncomment this line to define a global platform for your project
platform :ios, '10.0'

and add
 target.build_configurations.each do |config|
    config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '10.0'
end

then run
flutter clean \
&& rm ios/Podfile.lock pubspec.lock \
&& rm -rf ios/Pods ios/Runner.xcworkspace

now you can flutter run --release

I would strongly advise against just manually rolling engine like that because you might end up some bizarre issues due to ABI skew between Flutter Framework and Flutter engine. Instead you should always just switch to a newer version of Flutter (e.g. if you want to use engine from 1.23.0-18.1pre you should better use Flutter at 1.23.0-18.1pre as well).

Using 1.23.0-18.1pre might cause Android GestureDetector not working.  we do not have any choice because migrate into Flutter 2.0 is not possible for us at present.  But Flutter team might not know this issue is A BIG TROUBLE for Flutter users in China!


Make scrollbar thinner

12px is too much for a scrollbar. I think 8 or 6 is a good value.

I agree with you on that. It would be great if someone using a high density screen could also comment on that. (which was the initial idea of having 12).

I have a 27" 1440p screen and a 1080p 14" laptop. Is that high DPI enough? It seems okay to be currently. Maybe 8/10 instead of 12, but I think 6 is gonna be too small.

this is from my laptop. But with high DPI everything should double (if you use 2x scaling like on a 4K screen) right? So it should not affect high DPI.

6 looks small on any screen size huh, 8 is good I think

@Defman21 I think 8 is good too! Let's change it to that!
Also, looking at @vooze's screenshot, I think, for the menubar in the terminal, we should also add some padding-bottom ?

So there should be a padding at the bottom of terminal? I don't get it

In the top header of the applications. Where the Red, Yellow and Green buttons are!

Looks just fine to me :)

It is okay, just noticed that the top and the bottom are not equal.


zero copy example does not compile on devcloud

Summary
I am trying to run the zero-copy example on devcloud. It seems that the default platform specified in the CMakelist.txt cannot be found on devcloud, so I changed it to pac_s10_dc. And I got the compilation error from aoc compiler as detailed below.
Is this example supported on devcloud S10 board? If not, where can I find a board that supports USM and zero-copy?
Version
I am on the latest commit of master branch. Here is the dpcpp version
u68165@s001-n143:~$ dpcpp -v
Intel(R) oneAPI DPC++ Compiler 2021.2.0 (2<IP_ADDRESS>10317)

For Quartus and BSP version
u68165@s001-n143:~$ tools_setup -t S10DS
sourcing /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/init_env.sh
export QUARTUS_HOME=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/quartus
export OPAE_PLATFORM_ROOT=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237
export AOCL_BOARD_PACKAGE_ROOT=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237/opencl/opencl_bsp
Adding $OPAE_PLATFORM_ROOT/bin to PATH
export INTELFPGAOCLSDKROOT=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld
export ALTERAOCLSDKROOT=/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld
Adding $QUARTUS_HOME/bin to PATH
source /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/init_opencl.sh

Environment
Intel devcloud cluster. S10 node with oneAPI
--------------------------------------------------------------------------------------
Nodes with Stratix 10 OneAPI: (1 available/3 total)
s001-n143
--------------------------------------------------------------------------------------

Steps to reproduce
Just clone the repo, generate Makefile and run make fpga as mentioned in the instructions.
Observed behavior
Here is what I got after running make fpga
u68165@s001-n144:~/oneAPI-samples/DirectProgramming/DPC++FPGA/Tutorials/DesignPatterns/zero_copy_data_transfer/build$ make fpgaScanning dependencies of target zero_copy_data_transfer.fpga
[ 50%] Building CXX object src/CMakeFiles/zero_copy_data_transfer.fpga.dir/zero_copy_data_transfer.cpp.o[100%] Linking CXX executable ../zero_copy_data_transfer.fpga
aoc: Running OpenCL parser....
Error: SPIRV to LLVM IR FAILED
dpcpp: error: fpga compiler command failed with exit code 1 (use -v to see invocation)
src/CMakeFiles/zero_copy_data_transfer.fpga.dir/build.make:94: recipe for target 'zero_copy_data_transfer.fpga' failedmake[3]: *** [zero_copy_data_transfer.fpga] Error 1CMakeFiles/Makefile2:229: recipe for target 'src/CMakeFiles/zero_copy_data_transfer.fpga.dir/all' failed
make[2]: *** [src/CMakeFiles/zero_copy_data_transfer.fpga.dir/all] Error 2
CMakeFiles/Makefile2:268: recipe for target 'src/CMakeFiles/fpga.dir/rule' failed
make[1]: *** [src/CMakeFiles/fpga.dir/rule] Error 2
Makefile:183: recipe for target 'fpga' failed
make: *** [fpga] Error 2

Expected behavior
to generate bitstream successfully.

@tyoungsc - Can you or someone on your team take a look at this please

I believe this error has been mis-characterized.  I just tried running cmake then make fpga on the devcloud, and this is the error I got:
u73448@login-2:~/samples.1/DirectProgramming/DPC++FPGA/Tutorials/DesignPatterns/zero_copy_data_transfer/build$ make fpga
Scanning dependencies of target zero_copy_data_transfer.fpga
[ 50%] Building CXX object src/CMakeFiles/zero_copy_data_transfer.fpga.dir/zero_copy_data_transfer.cpp.o
[100%] Linking CXX executable ../zero_copy_data_transfer.fpga
warning: -reuse-exe file '/home/u73448/samples.1/DirectProgramming/DPC++FPGA/Tutorials/DesignPatterns/zero_copy_data_transfer/build/zero_copy_data_transfer.fpga' not found; ignored
Out of memory in module quartus_sh (1023 megabytes used).
Error: The patches required to compile for the target board (0.05dcp) is not installed for the following Quartus:
/glob/development-tools/versions/oneapi/2021.2/inteloneapi/intelfpgadpcpp/2021.2.0/QuartusPrimePro/19.2/quartus/bin/quartus_sh
dpcpp: error: fpga compiler command failed with exit code 1 (use -v to see invocation)
src/CMakeFiles/zero_copy_data_transfer.fpga.dir/build.make:94: recipe for target 'zero_copy_data_transfer.fpga' failed
make[3]: *** [zero_copy_data_transfer.fpga] Error 1
CMakeFiles/Makefile2:229: recipe for target 'src/CMakeFiles/zero_copy_data_transfer.fpga.dir/all' failed
make[2]: *** [src/CMakeFiles/zero_copy_data_transfer.fpga.dir/all] Error 2
CMakeFiles/Makefile2:268: recipe for target 'src/CMakeFiles/fpga.dir/rule' failed
make[1]: *** [src/CMakeFiles/fpga.dir/rule] Error 2
Makefile:183: recipe for target 'fpga' failed
make: *** [fpga] Error 2
So the platform used in this sampe (intel_s10sx_pac:pac_s10_usm) IS found, but the Quartus version installed on the DevCloud does not include the required patch.  This is not a problem with the sample, it is a problem with the binaries installed on the DevCloud.
@JoeOster or @Hecmay can you help me figure out who this issue should be re-directed to?

Alby, I'm assigning this to you as it seems to be a problem with the FPGA Add-on installed on the Dev Cloud.  It's not clear to me if the wrong add on is installed, or if there's a problem with the add-on that was shipped and it doesn't support the USM board.

Hi there, I think the root of the error is a misunderstanding about what DevCloud nodes to use. It looks like you're using n144, which is part of the fpga_runtime:stratix10 group, meant for running the designs after they've been compiled. However, the runtime nodes don't necessarily have Quartus installed. If you want to compile your design for hardware, you should be using the fpga_compile nodes to submit your jobs, ex:
qsub -l nodes=1:fpga_compile:ppn=2 -d . build_fpga_hw.sh
These don't have a board, but do have Quartus. Running it on one of these nodes (ex: n056) goes through the aoc stage fine for me, so please let me know if you still have problems. You can see more about the flow/job submission here:
https://devcloud.intel.com/oneapi/documentation/base-toolkit/
https://devcloud.intel.com/oneapi/documentation/job-submission/

Thanks Alby, I made a similar mistake (I used the login node to try to run my FPGA compile).  I've launched a compile on an fpga_compile node, and while it's still in progress it did not error out immediately as I saw previously, so I assume this will be fine.  Sorry for not catching this myself, I don't have much experience using DevCloud (obviously :-) ).

Thanks @AlbyB @mdbtucker. At the beginning I was trying to use the fpga_compile node, but the qsub request hanged forever. So I switched to FPGA node instead. I will give it a try later.

If they're all still busy and you know exactly what you want to execute, you can also try writing it up in a .sh file and submitting that to the fpga_compile group, and it'll run whenever there's a free node.

It is being compiled now. Thanks!
One thing to notice is that: do not run tools_setup -t S10DS as suggested by the devcloud document before the compilation. This will mess things up somehow.
u68165@s001-n053:~/oneAPI-samples/DirectProgramming/DPC++FPGA/Tutorials/DesignPatterns/zero_copy_data_transfer/
build$ make fpga
Scanning dependencies of target zero_copy_data_transfer.fpga
[ 50%] Building CXX object src/CMakeFiles/zero_copy_data_transfer.fpga.dir/zero_copy_data_transfer.cpp.o
[100%] Linking CXX executable ../zero_copy_data_transfer.fpga
Board 'pac_s10' not found.
Searched in the board package at:
  /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237/opencl/open$
l_bsp
Board list:
  pac_s10_dc
     Board Package: /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_$
_b237/opencl/opencl_bsp

If you are using a 3rd party board, please ensure:
  1) The board package is installed (contact your 3rd party vendor)
  2) You have used -board-package=<bsp-path> to specify the path to
     your board package installation
Error: No board_spec.xml found for board 'pac_s10' (Searched for: /glob/development-tools/versions/fpgasupport$
tack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237/opencl/opencl_bsp/hardware/pac_s10/board_spec.xml).
dpcpp: error: fpga compiler command failed with exit code 1 (use -v to see invocation)
src/CMakeFiles/zero_copy_data_transfer.fpga.dir/build.make:94: recipe for target 'zero_copy_data_transfer.fpga$
 failed
make[3]: *** [zero_copy_data_transfer.fpga] Error 1
CMakeFiles/Makefile2:229: recipe for target 'src/CMakeFiles/zero_copy_data_transfer.fpga.dir/all' failed
make[2]: *** [src/CMakeFiles/zero_copy_data_transfer.fpga.dir/all] Error 2

No problem, glad to hear that it's working!
If you wouldn't mind, could you link what documentation you're following and let us know how you came across it/these samples?

@AlbyB Here is the document I referred to: https://github.com/intel/FPGA-Devcloud/tree/master/main/Devcloud_Access_Instructions#53-development-tool-access-and-setup. I asked Larry before (he is the maintainer of devcloud cluster I believe?), and he forwarded me this link.
Unfortunately, my interactive job on fpga_compile node was terminated because of ssh broken pipe. And my batch job to fpag_compile node is hanging forever again (even though I can see many available compilation-only nodes in the cluster). I guess I need to try my luck on fpga_runtime node again. lol

@AlbyB
Sorry to bother you again.. The compilation on compile_fpga can run successfully and it can generate .fpga file as expected. However, the bitstream cannot be executed on real FPGA on devcloud.
Here is the detailed error message
u68165@s001-n143:~/oneAPI-samples/DirectProgramming/DPC++FPGA/Tutorials/DesignPatterns/zero_copy_data_transfer/build$ ./zero_copy_data_transfer.fpga
Error enumerating AFCs: not found
Error enumerating AFCs: not found
Error enumerating AFCs: not found
Error enumerating AFCs: not found
Segmentation fault


bug: Swipe to go back shows blank screen while using abstract nested states on iOS

Type: bug
Platform: ios 8 webview
I found that if you have a parent abstract state, you will see an extra blank screen(parent abstract state) when you swipe to go back on iOS. Since this issue only occurs on iOS device, I will not create a codepen but only put some related code with a picture demonstrates this issue here:
states in app.js:
$stateProvider
            .state('signup', {
                abstract: true,
                templateUrl: 'templates/signup.html',
                controller: 'SignupCtrl'
            })
            .state('signup.index', {
                url: '/signup',
                templateUrl: 'templates/signup/index.html',
                controller: 'SignupIndexCtrl'
            })

// www/index.html
<body>

<ion-nav-bar></ion-nav-bar>

<ion-nav-view></ion-nav-view>

</body>

// templates/index.html
<ion-view>
    <ion-nav-title>
        welcome
    </ion-nav-title>
    <ion-content>

    </ion-content>
</ion-view>

// templates/signup.html
<ion-view hide-nav-bar="true">
    <ion-nav-view>
    </ion-nav-view>
</ion-view>

// templates/signup/index.html
<ion-view>

</ion-view>

related discussions on ionic forum: click here

Greetings @Orthocenter!
My sensors indicate that you need to update your issue through our custom issue form. We are now requiring all issues to be submitted this way, to ensure that we have all of the information necessary to fix them as quickly as possible.
Click Here To Update Your Issue
I will have no choice but to close this issue if it is not resubmitted through the form. Please fill out the rest of the form, so that I may use my friendly robot powers to assist you.
Thank you!

You're missing abstract: 'true' in your main state
            .state('signup', {
                templateUrl: 'templates/signup.html',
                abstract: 'true',
                controller: 'SignupCtrl'
            })

@mhartington thanks for your reply. Sorry, I forgot it when I typed this code on Github, but I have set 'abstract: true' in my real code actually. In addition, I have tested the issue I mentioned above with default view cache settings again, but nothing changed.

Can you please provide a codepen then to test.
Mike Hartington

On Jul 23, 2015, at 11:10 PM, Yuxin Chen<EMAIL_ADDRESS>wrote:
@mhartington thanks for your reply. Sorry, I forgot it when I typed this code on Github, but I have set 'abstract: true' in my real code actually. In addition, I have tested the issue I mentioned above with default view cache settings again, but nothing changed.
—
Reply to this email directly or view it on GitHub.

Hi, @mhartington  . Here is a codepen showing this issue:  http://codepen.io/Orthocenter/pen/oXaPoq .
It was forked from ionic official codepen Tabs Example, and removed some unnecessary parts. You can see a black screen when you click ‘go to tabs.home’ link first time. And If you swipe to go back at state 'tabs.home', you will see an extra white screen (the abstract view). The second case only occurs in native app, since swiping gesture has no effect in browsers. So please test with built native app. Thx
this picture illustrates the second case:

So you have your states setup wrong. Take a look at this codepen.
http://codepen.io/ionic/pen/CbBsA
You should have them setup something like this.
    .state('tabs', {
      url: "/tab",
      abstract: true,
      templateUrl: "templates/tabs.html"
    })
    .state('tabs.home', {
      url: "/home",
      view: {
        "tab-view": {
          templateUrl: "templates/home.html"
        }
      }
    })
    .state('tabs.facts', {
      url: "/facts",
      view: {
        "tab-view": {
          templateUrl: "templates/facts.html"
        }
      }
    });

Note that this will not allow you to swipe back from tabs.home to index.

Sorry, in my codepen, I just use some behaviors of UI-router but not ion-tabs.
I wrote my states config according to this example: https://github.com/angular-ui/ui-router/wiki/Nested-States-and-Nested-Views#abstract-states, and I think I have set up correctly.
The root of this problem may be applying redundant animation on a parent view. Say that we are on state  B.C, previous state is A. When we swipe left, B should not be shown separately, but should be moved with C simultaneously instead. Thx

Not necessarily, since we're dealing with abstract states. Since state B is abstract, the navigation and history management aren't sure how to handle. Do we swipe back from B to A or do we do nothing since there is no state to swipe back to from C.
This is a bug, but it's a bug that would require massive changes to ionic's history and navigations stacks as well as ui-router. This is being working on in V2, but as it stands it's not something that we could fix ATM.

Thanks, that answers the question. I think that we should swipe back to A in that case. We should swipe back to the previous view, not parent state right? Maybe I have missed some cases.

Right, we think it should go back to A, but the way our codebase stands ATM doesn't allow for this :disappointed:
This is something that we are working on for V2, so we are aware of this shortcoming. Closing for now as there's no easy fix.

@mhartington Any possibilities of bringing Ionic's push and pop navigation to Ionic 1? As of now the navigation is just a nightmare for a moderately complex app in Ionic 1

I am facing the same problem for this, anyone knows a good solution to fix that? Thanks


Adds WebSocket connection cross-site protection using SameSite cookie strategy

Fixes https://github.com/kubetail-org/kubetail/issues/3

Closing in favor of CSRF token strategy https://github.com/kubetail-org/kubetail/pull/6


Authorization

In this Pull Request, I have done the following:

[x] Installed CanCanCan.

[x] Added admin role column to the users table.

[x] A user can delete a post if it is theirs or if they have an admin role.

[x] A user can delete a comment if it is theirs or if they have an admin role.

Thank you @AmaduKamara, I truly appreciate your time and efforts in reviewing my work


[bug] collab settings change doesn't prompt wallet

Describe the bug
When trying to change the username and description for a collab contract, when I click Save Profile nothing happens. This happens regardless of whether or not an image is being uploaded.
Reproduction

make a new collab contract (eg. https://teia.art/tz/KT1MARbF9KvMjGJWNX3j3bVdkatUjWbD4zU1)
"log in" to it, via the collaborate menu
go to settings
fill out new profile information
press "Save Profile"
nothing happens

Expected behavior
When pressing Save Profile, a prompt to sign in your wallet should come up, and then the changes will be uploaded and saved.
Platform and versions
- OS: Windows 11
- Browser: tried Firefox and Chrome
- tested teia.art and teia.rocks

Console output
XHRPOSThttps://ipfs.infura.io:5001/api/v0/add?stream-channels=true&progress=false
[HTTP/1.1 401 Unauthorized 39ms]

POST
	https://ipfs.infura.io:5001/api/v0/add?stream-channels=true&progress=false
Status
401
Unauthorized
VersionHTTP/1.1
Transferred360 B (20 B size)
Referrer Policystrict-origin-when-cross-origin
Request PriorityHighest

    Access-Control-Allow-Credentials
    	true
    Access-Control-Allow-Origin
    	https://teia.art
    Content-Length
    	20
    Content-Type
    	text/plain; charset=utf-8
    Date
    	Tue, 16 Aug 2022 15:17:18 GMT
    Vary
    	Origin
    Www-Authenticate
    	Basic realm="Project ID is required"
    X-Content-Type-Options
    	nosniff
    X-Robots-Tag
    	noindex

    Accept
    	*/*
    Accept-Encoding
    	gzip, deflate, br
    Accept-Language
    	en-US,en;q=0.5
    Connection
    	keep-alive
    Content-Length
    	259
    Content-Type
    	multipart/form-data; boundary=---------------------------100955003518257656423327817725
    DNT
    	1
    Host
    	ipfs.infura.io:5001
    Origin
    	https://teia.art
    Referer
    	https://teia.art/
    Sec-Fetch-Dest
    	empty
    Sec-Fetch-Mode
    	cors
    Sec-Fetch-Site
    	cross-site
    User-Agent
    	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0

Additional context
No response

Also fixed via https://github.com/teia-community/teia-ui/pull/159


[feature request]Reactions to undefined args

When using an undefined argument (e.g. using -? after -h is defined to display help message), the program will directly report an error and exit. I wonder if it is possible to show the correct usage and help of the args before the program exits in that case.

It is very hard to track all undefined arguments. Always use --help as the long name to pring help message. Then you can always get the right help message by program_name --help.

Always use --help as the long name to pring help message. Then you can always get the right help message by program_name --help.

What I mean by this example is that the help message is hoped to be displayed before exiting when the program encounters an undefined argument.

It is very hard to track all undefined arguments.

When encountering a mismatched argument, the program will jump to the 'position argument number missmatching' branch, and perhaps adding a   call this%print_help()  statement before line 265 error stop  can deal with the mismatched args.

I make a small update, print the unparsed arguments when position argument number missmatching. Now it will show the unparsed -? if you define -h for help.
Because there are several kind of parse error in the library, I think it is not suitable to only print help message just in the specific condition. And you can always get the help message by run program_name --help, so print the help message may be unnecessary.
May be a better error display is needed for this library. I will compare some cmdline tools and try to improve the entire error display later.

I make a small update, print the unparsed arguments when position argument number missmatching.

Thanks a lot !!!

Because there are several kind of parse error in the library, I think it is not suitable to only print help message just in the specific condition. And you can always get the help message by run program_name --help, so print the help message may be unnecessary.
May be a better error display is needed for this library. I will compare some cmdline tools and try to improve the entire error display later.

My idea is to assume that the user doesn't know the correct usage (even --help) when the wrong argument is entered. As a result, a help message can be displayed before the error stops to remind the user of the correct usage.

I compared some cmdling tools, and find that there is no common way to display errors, and no one can deal with every unexpected input clearly. So finally, I think your idea is a not bad solution. Now I add this%print_help() for every parse errors.
Now I use stop instead of  error stop for parse errors, because these errors are shown for users. The build error and get error are the bug of the program, and should be fixed by developers, so they remain using error stop.
Tanks for your comments.

Now I use stop instead of error stop for parse errors, because these errors are shown for users. The build error and get error are the bug of the program, and should be fixed by developers, so they remain using error stop.

Thank you very much, this seems reasonable.
But maybe the error message could be shown at first, then the help message, and then stop, like this.

Some times the help message may be very long, so I prefer print_help first.

Some times the help message may be very long, so I prefer print_help first.

From a logical point of view, it is because there is an error that the help message is shown. So it should be the error message first, the help message last.
In addition, there is one more error that will occur,that is the conversion error. This error occurs when an integer is required but a real number or string is entered.
This type of error maybe could be handled in the get method, for example

However, if we do so, the incorrectly entered value will replace the default value in the help message displayed, so other methods may also need to be adjusted.

The error message if shown for users. So people should see it directly. If show error message first, when the help message is too long, then people must scroll up to find the error message. It is inconvenience.
I will add the type check later.

Thank you very much!!!


ci: E2E Framework [Background steps]

Reason for Change:

Run steps in background, require corresponding stop later on to avoid orphaned steps
Issue Fixed:

Requirements:

[ ] uses conventional commit messages

[ ] includes documentation
[ ] adds unit tests
[ ] relevant PR labels added

Notes:

source branch is a fork, dup: https://github.com/Azure/azure-container-networking/pull/2541


📖 Add a book for the Cluster API Operator

User Story
As a developer/user I would like to learn about CAPI operator features, designs and other CAPI operator guides, like installation, quickstart and day 2 operations, release procedures and more. Currently all information is stored inside a single markdown document, making it hard to read.
Detailed Description
CAPI uses a concept of a book - https://cluster-api.sigs.k8s.io/, CAPI operator should follow the same pattern.
Anything else you would like to add:
/kind feature

Thanks for issue (was postponing to open one 😄), I already started investigating it, so
/assign

I’ve separated this into sub-tasks, so anyone can help getting it done. Initial book implementation follows CAPI path, and is in https://github.com/kubernetes-sigs/cluster-api-operator/issues/330. This will deploy book to the github pages.
Currently looking like this, should be a sufficient template to start:

cc @Fedosin @furkatgofurov7 @alexander-demicev

/remove-lifecycle stale

@Danil-Grigorev I just found out that most of the CAPI providers if not all only have one version of their provider book and don't have versioned books per release branch as for example CAPI does. I can imagine that is because of maintenance work and as just simple as they did not have a need for it.
So my question would be: do we need really versioned books of CAPI operator hosted in Netlify as CAPI or not?
/cc @alexander-demicev @Fedosin @damdo @JoelSpeed if you folks have opinions, welcome!

@furkatgofurov7 I think for now we don't need multiple versions in our documentation, but we can return to this question in the future, if needed.


BrokenPipeError: [Errno 32] Broken pipe

try to use the tool after a 20 minutes struggle because of using automatically installation which did nothing then return to manually and finally made it,,
so i run the tool and start to search but later there some error happen
see the ss

Does the Browser open? Does the Browser navigate to google.com/imghp ?
Seems like the Browser is closed when it tries to find the DOM Element.

I have the exact same issue.
The browser is well opened and access to google.com/imghp.
I can see it starting the searches but after a few seconds (when it found the results for the picture I put in ./known), I have this error, EagleEye exits but not Firefox.

i Tried using the latest Geckodriver (Ver 0.21.0) and FF 60.0.2, everything works. No crashes.
Note: If you are using Firefox ESR(like Kali does) please use the Geckodriver Version 17

I'm also having this issue.
Linux Mint 19
Firefox 60.0.2
Geckodriver 0.21.0
Also i had a bit of trouble to get this script to even start.
`:: Google Reverse Image Search
==> Opening Webdriver
:: Uploading Local Known Images
-> Inserting Path
Traceback (most recent call last):
File "eagle-eye.py", line 230, in
main(skipFB=args.skipfb, skipIR=args.skipir, skipY=None, FBUrls=[])
File "eagle-eye.py", line 116, in main
g.collectLinksLocal()
File "/home/mint/EagleEye/grabber/google.py", line 94, in collectLinksLocal
pred = driver.find_element_by_xpath("/html/body/div[6]/div[3]/div[3]/div[1]/div[2]/div/div[2]/div[1]/div/div[2]/a")
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 393, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 966, in find_element
'value': value})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 318, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/remote_connection.py", line 472, in execute
return self._request(command_info[0], url, body=data)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/remote_connection.py", line 495, in _request
self._conn.request(method, parsed_url.path, body, headers)
File "/usr/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1065, in _send_output
self.send(chunk)
File "/usr/lib/python3.6/http/client.py", line 986, in send
self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe
`

It worked when i changed geckodriver  to 0.19.1 :)


Comma split problem

If identity of the mailbox or displayname cointains "comma", the split function do not work correctly causing exception (Index out of range)

Fixed in the next version, tests in progress

Fixed in Release Candidate 1 version


Make react a peer dependency for Connect

Unresolved issue:
https://github.com/blockstack/connect/issues/74

@hstove mind providing your latest thinking about this dependency here?

Yes, we definitely need to update this. I have the fix in the PR for transaction signing - I'll get it pulled into the master branch so it can be published. Thanks @njordhov , sorry this has taken so long.


use official Ansible container for ansible-lint

Basically, run ansible-lint locally the same way that the ansible-lint
github action runs.  This helps when you get errors in the github CI,
you can reproduce and fix locally.
https://github.com/ansible/ansible-lint-action/blob/main/action.yml#L24
Get rid of the lsr_ansible-lint configuration.  If you want to configure
ansible-lint, you specify .ansible-lint in the role repo root directory.
This also helps simplify and make the local action work just like the
github action.
By default, it will use podman to run the container.  You can set
LSR_CONTAINER_RUNTIME to use a different runtime.
You can pass extra command line arguments to ansible-lint like this:
tox -e ansible-lint -- -vvv --exclude=my/dir/ ...

@nhosoi @spetrosi this is ready for review


Eliminate race conditions

When using threads for timer or libserialcpp SPI (that is internally using threads also), there may be additional race conditions that are not usually hit if running a mainloop (eg: on Raritan framework).
Serial read/write coud can be run simultaneously and race conditions must be handled.
Timer elapsing are implemented using threads when not running on the Raritan mainloop and the actions for the code executed in timers should be protected using mutexes.

First race condition should be fixed by https://github.com/Legrandgroup/libezsp/commit/f7d20b7d5368f484b3ef19d38556c00c162843c0#diff-edb1b438ffb18d61ccd84f60a3077c03
Second race condition should be fixed by https://github.com/Legrandgroup/libezsp/commit/31a2e1623dc246299f8ba4ce6bb882475aba5fa4

Fixes done on branch https://github.com/Legrandgroup/libezsp/tree/thread-safety are part of solving this.

Implemented in PR #76


v4 is not working on IE8

options.daysOfWeekDisabled = daysOfWeekDisabled.reduce(function (previousValue, currentValue)
the function reduce is not supported until IE9
can you please fix it?

You can add the Polyfill to your project


feat: implement aborted transactions retry mechanism

This is the first part of the transaction retry mechanism implementation. Here is what it consists of:
Class ResultsChecksum is written according to the design and covered with unit tests.
Every transaction inits a checksum property on creation.
Transaction.execute_update() and Transaction.batch_update() methods are now adding their results into transaction checksum.
StreamedResultSet now have an optional constructor arg - checksum. If checksum is passed result set will be pushing every streamed row into the checksum before returning it to user.
At the next step I'm gonna implement propagation of transaction checksum into the result sets of operations runned in this transaction.

@mf2199, @AVaksman

@AVaksman, @mf2199, seems like only adding a big unit test for Session.run_in_transaction() is left. It's a tough thing, requires a lot of mocks, I'm still in progress with it.

Two tests failed:
FAILED tests/system/test_system.py::TestSessionAPI::test_transaction_query_w_concurrent_updates
FAILED tests/system/test_system.py::TestSessionAPI::test_transaction_read_w_concurrent_updates

Seems to me it's a right behavior: data concurrently changed, so results of the retried transaction are not equal to the results of the original transaction.

@mf2199, @AVaksman, okay, I've pushed the last part

cc @skuruppu and @larkee, who will understand better than I do how this interacts with the existing python-spanner retry logic.

@IlyaFaer I get the impression that there is a misunderstanding about when retry logic based on checksums are needed for aborted transactions. The Java client contains two different APIs for interacting with Cloud Spanner:

The 'normal' Spanner client. This client requires client applications to define a transaction as a function, and to pass that function in to a TransactionRunner. The Spanner client will start a transaction and then call the function that the client application supplied. The function can then execute queries and updates on the transaction. If the transaction is aborted by Cloud Spanner, queries and/or updates on the transaction will return an Aborted error. This will trigger the Spanner client to retry the transaction by starting a new transaction and calling the transaction function once again. This retry strategy does not require any checksums, as the entire transaction is defined as a function that can be retried in its entirety.
The 'Connection API': This is a generic API based on a more traditional database API that has a permanent 'connection' with the database. The client application can execute transactions and statements on a connection, and it is not known in advance what statements will be executed in a transaction, as each transaction can be a random list of statements and is not defined as a function. If a transaction is aborted by Cloud Spanner, it cannot be retried by simply 'calling the transaction function again using a new transaction ID', as that function is simply not there. Instead, the Connection API keeps track of all statements that are executed during a transaction and the results that were returned by those statements. If the transaction is aborted, the Connection API will start a new transaction and execute the exact same statements once more on the new transaction, and compare each the result of each statement with the result of the original statement. If they match, the Aborted error from Cloud Spanner is 'ignored' and the transaction is allowed to proceed. If the results do not match, the Aborted error is propagated to the client application and the transaction is aborted.

The way that the Connection API keeps track of the results per statement depends on the type of statement:

All statements remember whether the statement returned an error or success. During the retry, the error or success should be the same.
Update statements and batch update statements remember the update count that was returned by the statement. During a retry, the update counts are compared with each other.
Queries can potentially return a large number of rows, and it is not feasible to keep all rows that have been seen during a transaction in memory. So instead queries keep track of a running checksum based on each row that has actually been returned to the client application. During a transaction retry, the same query is executed again, and the retry will consume the exact same number of rows as the original transaction. If the checksum that is calculated during that retry matches the checksum during the original transaction, the results are deemed equal and the transaction may proceed.

From what I see in this PR, the checksum retry logic is being added to the 'normal' client. The run_in_transaction method takes a function that contains all transaction logic. If the transaction is aborted by Cloud Spanner, the client will retry the transaction by starting a new one and calling the same transaction function once more with the new transaction. There is no need to add any checksum checking to this API, as this retry logic already works as intended.
Retries based on checksums are only needed if client applications can interact with Cloud Spanner in a more traditional way through some kind of 'connection' object (or similar) that does not require you to define a transaction as a function.
Consider the following (pseudo-)code example based on a simple example on how to work with MySQL in Python:
try:
    connection = mysql.connector.connect(host='spanner.googleapis.com',
                                         database='Electronics',
                                         user='pynative',
                                         password='pynative@#29')
    if connection.is_connected():
        connection.autocommit = false
        cursor = connection.cursor()
        cursor.execute("SELECT * FROM Foo;")
        for row in cursor:
            print(row)
        cursor.execute("""UPDATE Foo SET Bar = 'baz' WHERE Id = 2""")
        connection.commit()

except Error as e:
    connection.rollback()
    print("Error ", e)
finally:
    if (connection.is_connected()):
        cursor.close()
        connection.close()
        print("Spanner connection is closed")

Cloud Spanner could for example abort the transaction before/during the statement cursor.execute("""UPDATE Foo SET Bar = 'baz' WHERE Id = 2"""). This will cause this statement to return an Aborted error. The underlying connection API should in that case catch the Aborted error, start a new transaction, retry the SELECT * FROM Foo query and check whether the results of that query is still equal to the results of the original attempt. If they are, then the transaction can proceed and the UPDATE Foo SET Bar = 'baz' WHERE Id = 2 can be executed again.
All the above should be transparent to the client application (except for a delay during the execution of UPDATE Foo SET Bar = 'baz' WHERE Id = 2, as the entire retry has be executed and checked before the result of that statement can be returned to the client application).
Please don't hesitate to ask if you have additional questions on this (or let me know if I completely misunderstood how the current PR works).

@olavloite,

From what I see in this PR, the checksum retry logic is being added to the 'normal' client.

Yes, that's correct. The thing is that the connection API we have for now is actually delegating to the original Spanner client, and exactly run_in_transaction() (which takes a function and runs it like a transaction, like you mentioned), for example:
https://github.com/googleapis/python-spanner-django/blob/e6f6179b4100c7c995849dc521a02c63aea1d791/google/cloud/spanner_dbapi/cursor.py#L116
where in_transaction() is:
https://github.com/googleapis/python-spanner-django/blob/e6f6179b4100c7c995849dc521a02c63aea1d791/google/cloud/spanner_dbapi/connection.py#L59
where db_handle is Database object.
In fact our Python connection API always works in autocommit mode:
https://github.com/googleapis/python-spanner-django#transaction-management-isnt-supported
Thus, our way of use is little bit different (at least in some parts) from Java. We probably can write several wrappers in the connection API, to use checksums comparisons only in connection API, not in traditional Spanner client. But it looks like an overkilling way, as we will have to inherit and then override several original Spanner methods from several classes, and these new methods will be very-very similar to the original ones, like 15-20% defference. It's probably easier to make checksums comparison enablable (and always enabled when using connection API), if it's critical not to compare checksums in the original Spanner client.
For now I don't see simpler way to add checksums comparison into connection API without adding it into original Spanner, as to make comparison work it takes to make changes in several classes, that are embedding each other on different levels.

Yes, that's correct. The thing is that the connection API we have for now is actually delegating to the original Spanner client, and exactly run_in_transaction() (which takes a function and runs it like a transaction, like you mentioned), for example:

Thanks for the clarification, that makes sense.

In fact our Python connection API always works in autocommit mode:

Uhm... If it is not the intention to change that, then you don't need retry logic for transactions, as there are no transactions.

From what I'm getting from the code, any query that is executed on a connection will be handled here: https://github.com/googleapis/python-spanner-django/blob/e6f6179b4100c7c995849dc521a02c63aea1d791/google/cloud/spanner_dbapi/cursor.py#L103
That will use a single-use read-only transaction (a 'read snapshot'): https://github.com/googleapis/python-spanner-django/blob/e6f6179b4100c7c995849dc521a02c63aea1d791/google/cloud/spanner_dbapi/cursor.py#L180
Such a query will never abort (read-only transactions never abort), so any retry logic in the result set is not necessary.

Update statements will also not abort, even though they do use transactions, as any aborted transaction is handled by the normal retry logic:

Any update on a connection will be handled here: https://github.com/googleapis/python-spanner-django/blob/e6f6179b4100c7c995849dc521a02c63aea1d791/google/cloud/spanner_dbapi/cursor.py#L116
That calls https://github.com/googleapis/python-spanner-django/blob/e6f6179b4100c7c995849dc521a02c63aea1d791/google/cloud/spanner_dbapi/connection.py#L59 which again calls the normal run_in_transaction with a standard function that only executes a single update statement. If that statement is aborted, the single statement will be retried until it succeeds.

@olavloite thanks for the detailed explanation. That is also my understanding. This fix relates to https://github.com/googleapis/python-spanner/issues/34 and based on the discussion and example, it seems that the intention is to change the retry mechanism in run_in_transaction. I think I'm missing something here. If the cased discussed in the example was an issue, then we would've changed the implementation in all the client library a long time ago. @olavloite do you have any thoughts on Ben's example in that issue?

Yeah, this getting a little bit confusing as there are a number of different topics getting mixed up a little bit.
Issue googleapis#34 describes a feature request for AUTOCOMMIT=off. A copy-paste from the issue:

In AUTOCOMMIT=off mode, we need to hold a Transaction for perhaps an indefinitely long time.
Cloud Spanner will abort:
a) Transactions when not used for 10seconds or more -- we can periodically send a SELECT 1=1 to keep it active
b) Transactions even when refreshed, can and will abort. This is because Cloud Spanner has a high abort rate

Thus we need to retry Transactions!

So:

The current retry logic in the 'normal' client library for Python works as intended, and is similar to the retry logic in most other client libraries as well: It has a run_in_transaction function that takes another function as a parameter. The run_in_transaction function starts a transaction, calls the function that was given as an input parameter, and commits the transaction. If the transaction is aborted, the run_in_transaction function will start a new transaction, call the input function once again, and try to commit again. This strategy will always succeed in the end.
The current Connection API implementation for Python always uses AUTOCOMMIT=on. All queries in the Python Connection API are currently executed using single-use read-only transactions. These transactions will never be aborted by Cloud Spanner and do not need any retry mechanism in the client library. All DML statements in the Python Connection API are executed as a single statement transaction. That means that a very simple function is defined that only executes the single statement, and that statement is passed in to the run_in_transaction method mentioned in 1. These statements therefore also do not need any additional retry logic. This AUTOCOMMIT=on behavior is equal to the behavior of the Java Connection API (and JDBC driver) when that is used in AUTOCOMMIT=on.
The feature request that seems to be the base of this PR is only relevant if the Python Connection API is to support AUTOCOMMIT=off, and the additional retry strategy should also only be used for the Connection API when using AUTOCOMMIT=off.

As explained above, the normal retry strategy that uses a run_in_transaction cannot be used for the Connection API in AUTOCOMMIT=off, as a transaction is not defined as a function. It would instead require the API to keep track of all statements that have been executed on a read/write transaction and the results these statements returned, and if the transaction is aborted, re-execute the statements and verify that these returned the same results as during the initial attempt. If the results are equal, the transaction should be allowed to continue. If they are not, the Aborted error should be propagated to the client application.

Ok that makes more sense to me now @olavloite, thanks so much for making sense of it. So if the intention is to support AUTOCOMMIT=off, it wouldn't help to change the logic in run_in_transaction. It has to be done in the Connection API.


feat: Make limit field mutable in the Timer class

Description

As stated in #2357, the field limit of the class Timer had no reason to be final. By removing this keyword, we make it possible to add more time to a given Timer object (so that they can be used as a countdown for example).
Checklist

[x] I have followed the Contributor Guide when preparing my PR.
[x] I have updated/added tests for ALL new/updated/fixed functionality.
[x] I have updated/added relevant documentation in docs and added dartdoc comments with ///.
[x] I have updated/added relevant examples in examples or docs.

Breaking Change?

[ ] Yes, this PR is a breaking change.
[x] No, this PR is not a breaking change.

Related Issues

Closes #2357.

I would like to add a little note here : in the description of the current getter, it is written :

The current amount of ms that has passed on this iteration

Wouldn't it rather be ``the number of seconds passed on this iteration'' ?

Wouldn't it rather be ``the number of seconds passed on this iteration'' ?

Agree!

@spydon Should I open a separate PR ?

@spydon Should I open a separate PR ?

Nah, you can do it in this one.

Just mutating that value won't work if _running is set to false.
Wouldn't that be confusing?

Is it really confusing? That is what I would expect would happen if I used this at least, would you have expected the timer to start again?

Might be better to expose a method to "reconfigure" the timer and make all state variables private

I would just update the _running value in the setter, if the expected behavior would be that it started again if it was lower than the limit.

not sure what I would expect haha. just crossed my mind. if you think it's clear as is, no problem for me :)

not sure what I would expect haha. just crossed my mind. if you think it's clear as is, no problem for me :)

Maybe you can add a dartdoc line about it @Skyost? :)
Just describing that the timer won't start automatically if the limit is raised and the timer currently is stopped.

I've left a little note in the header of the Timer class. I've reused your words 🙂


Remove dependency on "gulplog"

I notice you have a dependency on "gulplog" in your package.json, but you use gutil.log with the source code.
Do you need this dependency any more?

Good catch.  It isn't used.  Solved with dcd5e60.


hw: Add optional hardwired cluster alias region

This adds an optional hardwired alias region to the snitch_cluster in addition to the pseudostatic cluster_base_addr_i. This is useful in multi-cluster systems (e.g. Occamy) to enable static linking and optimization of TCDM contents.
Along the way, I removed some unconnected address inputs from snitch_cluster_peripheral.
For Occamy, I recommend enabling the prefix and mapping it to 0x1800_0000.

To keep track of the status, only Banshee tests are failing as this architectural feature is yet to be modeled in Banshee.

As requested, this should be made configurable instead. Will look into that next.

As requested, this should be made configurable instead. Will look into that next.

After clarifying this, it shouldn't be necessary anymore.


Add deploymentclient types

@treydock @bastelfreak
This is a rebase of #94
Lets get this merged before it falls behind again.

@TraGicCode can you rebase please?

I'm on it

Done! @bastelfreak


SCTransform normalization error

Hi there,
I had a question regarding using SCTransform with CHOIR. If I choose the SCT assay for use_assay= I get a warning:
For best performance of CHOIR with SCTransform normalization, please provide the unnormalized count matrix and set the 'normalization_method' parameter to 'SCTransform' .
I can see that when selecting the normalization method to SCTransform  I get vst.flavor = "v2"  by default.  However I can't seem to be able to regress out unwanted variation (mito %, cell cycle, etc...).  I was wondering what I should do in the case I wanted to regress out these variables and still use CHOIR.
Also I tried running it as is and got the following error
 choir_seurat <- CHOIR(seurat, use_assay= "RNA", n_cores = 1, normalization_method = "SCTransform")
----------------------------------------
- CHOIR - Part 1: Build clustering tree
----------------------------------------
Warning: Key 'CHOIR' already exists in provided object. Existing data may be overwritten.
Input data:
 - Object type: Seurat (v4)
 - # of cells: 848
 - # of modalities: 1

Proceeding with the following parameters:
 - Intermediate data stored under key: CHOIR
 - Normalization method: SCTransform
 - Subtree dimensionality reductions: TRUE
 - Dimensionality reduction method: Default
 - # of variable features: Default
 - Batch correction method: none
 - Maximum clusters: auto
 - Minimum cluster depth: 2000
 - Distance approximation: TRUE
 - Alpha: 0.05
 - Multiple comparison adjustment: bonferroni
 - Features to train RF: var
 - # of excluded features: 0
 - # of permutations: 100
 - # of RF trees: 50
 - Use variance: TRUE
 - Minimum accuracy: 0.5
 - Minimum connections: 1
 - Maximum repeated errors: 20
 - Maximum cells sampled: Inf
 - Downsampling rate: 1
 - # of cores: 1
 - Random seed: 1

2024-02-14 21:18:10 : (Step 1/6) Running initial dimensionality reduction..
                      Preparing matrix using 'RNA' assay and 'data' slot..
                      Running PCA with 2000 variable features..
Warning: Key 'PC_' taken, using 'choirp0reduction_' instead2024-02-14 21:20:26 : (Step 2/6) Generating initial nearest neighbors graph..
2024-02-14 21:20:27 : (Step 3/6) Identify starting clustering resolution..
                      [[ Current tree: 4 iterations in  1s ]]
                      Starting resolution: 0.1
2024-02-14 21:20:28 : (Step 4/6) Building parent clustering tree..
                      [[ Current tree: 13 iterations in  2s ]]

                      Identified 3 clusters in parent tree.
2024-02-14 21:20:30 : (Step 5/6) Subclustering parent tree..
Error in h(simpleError(msg, call)) :
  error in evaluating the argument 'x' in selecting a method for function 'as.matrix': subscript out of bounds

Here is the Traceback

Hi! All of the benchmarking we've done for CHOIR for RNA data has been using log normalization, rather than SCTransform, so I can't vouch for how the performance may change using SCTransform. We don't currently have plans to add customizable parameters for SCTransform normalization, until we've verified how the normalization method impacts performance.
That said, if you're worried about mitochondrial read percentage and cell cycle impacting your clusters, I'd recommend excluding those features using CHOIR parameter exclude_features. Doing so will prevent those features from being used as input to the random forest classifiers, so they won't impact the pruning of the clustering tree and the final clusters.
For the error you encountered—first, can you try setting use_slot to "counts"? When using SCTransform normalization, we want to use the "counts" slot/layer as input, rather than the (default) "data" slot/layer.
And try setting the Seurat object default assay to "RNA" before running CHOIR, if you haven't already. I fixed a bug related to this recently in the 'dev' branch, and it may be something similar that you've run into.
DefaultAssay(seurat) <- "RNA"

If neither of those things help, could you try running this quick reproducible example? This runs without errors for me.
library(Seurat)
library(CHOIR)
library(scRNAseq)

data_matrix <- LaMannoBrainData('mouse-adult')
colnames(data_matrix@assays@data$counts) <- colnames(data_matrix)
seurat_object <- CreateSeuratObject(data_matrix@assays@data$counts)
seurat_object <- CHOIR(seurat_object, use_slot = "counts", normalization_method = "SCTransform")


⚠ Bofrost.be has degraded performance

In 863c4ad, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2061 ms

Resolved: Bofrost.be performance has improved in 73f8914.


Autorest Regen Preview to 3.0.0-alpha.20231122.5 by Pan Shao from refs/pull/3957/head

Triggered from https://github.com/Azure/autorest.csharp/pull/3957

API change check
API changes are not detected in this pull request.


ADCS Firmware

ADCS functions

Discussed with Parker L:

ADCS code that's currently floating around in the main folder should be in its own adcs_drivers subfolder under Src and Inc
MEMORY_MODULE_FUNCTION should instead be unique for all the relevant structs (pick 2-3 for this PR); see EPS_result_system_status_TO_json function in electrical-eps-i12 branch (eps_types_to_json.c) for how do do this.

Please provide an update, including when this will be ready to merge to main; feel free to propose a plan to split this up.
This is reaching a state where we will have to write code completely devoid of this branch in order to have testing capabilities for upcoming vibration tests if a Minimum Viable Product (MVP) is not ready soon.

I think I've successfully addressed everything that needs to be addressed, so my next step is to do some testing with the ADCS (which won't be able to be done until next Saturday, September 7) before it's ready to be merged to main.

I don't anticipate too many difficulties in the testing, but it's important to be thorough and at least test some of the new JSON functions.

Considering possible ways to split this up, now that I've developed the generic ADCS command and telemetry telecommands:

MVP: functions required for generic ADCS command and telemetry telecommands (plus all of adcs_types)
Remainder of telecommands and associated functions (62 telecommands) -- possibly split arbitrarily into smaller groups for ease of review ex. commands, then telemetry

Does this PR have a "send arbitrary bytes to the ADCS and get the response" command? If not, we can put it in a separate issue, but we'll need that in case there's a bug in any of this work which we need to bypass

Does this PR have a "send arbitrary bytes to the ADCS and get the response" telecommand? If not, we can put it in a separate issue, but we'll need that in case there's a bug in any of this work which we need to bypass

It has two, actually! Those would be the generic_telemetry and generic_command telecommands, which will be the first things I test on Saturday -- my thinking was exactly the same.

Good stuff! I propose that, if we do split this up (which I don't think we're on track to do), then the appropriate place to split it would be to get that MVP implemented (with the adcs_generic_command and adcs_generic_telemetry_request telecommands) in one PR, and then all the individual commands in another PR. Looking forward to hearing how testing goes.

Update on testing for now (though I wasn't able to test today): the generic telemetry and command functions are working pretty well (still need to test them with the larger max buffer size, but it really shouldn't be a problem at all)!
We've been going through the list of other commands at a rate of about 22/week that we're able to test (so, 2 weeks of testing remaining until everything is fully tested). Since we've done a bunch of testing of all these commands already, the only thing we are really testing with these are the TO_json functions -- which now we've tested a few of for both commands and telemetry, I'm pretty confident about.
Otherwise, the requested code changes are made and I think it should be just about ready to merge!


Add 2 new args to include web vital metrics and audits

Added 2 new arguments to generate the output

-n to include the metrics from LH6 beta (bumped the LH version to V6 beta.
-t to include the audits

Thanks Yohan! 👍👍


ENH/DOC: special: document or extend domain of Legendre incomplete elliptic integrals

Describe your issue.
ellipeinc and ellipkinc are broken. Please either a) extend the domain of the Legendre elliptic integrals to include $1 < m < \frac{1}{\sin^2\phi}$ or b) document that arguments in the interval will result in NaN.
Reproducing Code Example
# Pick some k between 0 and 1.
ellipkinc(np.arcsin(k), 1 / k**2)
ellipeinc(np.arcsin(k), 1 / k**2)

Error message
Output is incorrect. not a number even though the function can be defined in this domain.
SciPy/NumPy/Python version and system information
1.12.0 1.26.4 sys.version_info(major=3, minor=12, micro=2, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /usr/local/include
    lib directory: /usr/local/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= HASWELL MAX_THREADS=2
    pc file directory: /usr/local/lib/pkgconfig
    version: 0.3.21.dev
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /usr/local/include
    lib directory: /usr/local/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= HASWELL MAX_THREADS=2
    pc file directory: /usr/local/lib/pkgconfig
    version: 0.3.21.dev
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 2.11.1
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.8
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  pythran:
    include directory: ../../tmp/pip-build-env-tfqmvy7q/overlay/lib/python3.12/site-packages/pythran
    version: 0.15.0
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /opt/python/cp312-cp312/bin/python
  version: '3.12

Thanks for reporting this. Actually, like https://github.com/scipy/scipy/issues/20506, we haven't touched the elliptic integrals at all (yet). But they are going to be updated via https://github.com/scipy/scipy/pull/20390. Once that is merged, I think we can look into this easily.

This bug is the same as I have recently encountered and comes from an explicit check in the incomplete elliptic integrals that is overly strict.
The functions are called, for example, as ellipkinc(phi, m). The code explicitly returns nan if  m>1 (e.g., line 166 in ellik.h). However, the integrals are well defined for all $m < 1 / \sin^2(\phi)$.
For example ellipkinc(0.015, 1.1) returns nan, however quad(lambda t: 1 / np.sqrt(1 - 1.1 * np.sin(t)**2), 0, 0.015) correctly returns 0.015000618791072898. This value is consistent with the functions elliptic_f from sympy and  gsl_sf_ellint_F() from libgsl, both of which handle this case without error.

cc @steppi now that gh-20390 is in

Here, the domain is overly restrictive, which could just be considered a documentation oversight and enhancement request, so I will update the label. Note that Boost is even more restrictive with the domain of this function, according to its documentation.
That said, l'll i look into this shortly as a follow-up to https://github.com/scipy/scipy/issues/4452#issuecomment-2323062848. ellipe probably has the same feature. In the meantime, see how you might implement this in terms of elliprf yourself following Boost's implementation notes at the end of their documentation.

Someone more familiar with the existing implementations of the special functions may have better luck, but I wasn't able to simply remove that condition to extend the usable domain. In case that doesn't work:
Assuming the output of this function is real and the domain of $m$ is extended, the domain of $\phi$ is $-\phi_c < \phi < \phi_c$ if $m > 1$, where $\phi_c = \sin^{-1} \sqrt{ \frac{1}{m} }$. (Outside of that, the output would be complex, and implementing that is a different sort of enhancement.) In this region, special.ellipkinc ($F$) can be implemented in terms of special.elliprf ($R_F$) as:
$$
F(\phi, m) = \sin \phi \ R_F \left( \cos^2 \phi , 1 - m \ \sin^2 \phi, 1 \right).
$$
The Boost documentation doesn't mention it explicitly, but this assumes $\phi > 0$ and $F(-\phi, m) =  -F(\phi, m)$. Including that logic:
def k(phi, m):
    sign = np.sign(phi)
    phi = np.abs(phi)
    val = np.asarray(np.sin(phi) * Rf(np.cos(phi)**2, 1 - m*np.sin(phi)**2, 1))
    return val * sign

Focusing on $m > 1$, we can compare the results of this implementation against mpmath.ellipf:

import numpy as np
import matplotlib.pyplot as plt
from scipy.special import elliprf as Rf
from mpmath import mp
mp.dps = 50

@np.vectorize
def k_ref(phi, m):
    # in case mpmath has tiny imaginary component
    return complex(mp.ellipf(phi, m)).real

@np.vectorize
def k(phi, m):
    sign = np.sign(phi)
    phi = np.abs(phi)
    val = np.asarray(np.sin(phi) * Rf(np.cos(phi)**2, 1 - m*np.sin(phi)**2, 1))
    return val * sign

def log10_relative_error(res, ref):
    with np.errstate(all='ignore'):
        out = np.where(res != ref, abs(res - ref) / abs(ref), 0)
        out = np.log10(out)
    out[out < -16] = -16
    return out

m = np.logspace(0, 3, 30)
phi_c = np.asin(np.sqrt(1/m))
phi = phi_c * np.linspace(-1, 1, 25)[:, np.newaxis]
m, phi = np.broadcast_arrays(m, phi)

res = k(phi,  m)
ref = k_ref( phi,  m)

logerr = log10_relative_error(res, ref)
plt.contourf(m, phi, logerr, cmap='viridis')
plt.xlabel(r"$m$")
plt.ylabel(r"$\phi$")
plt.colorbar()
plt.title(r"$\log_{10}$ error")
plt.xscale('log')
plt.show()

Note the the plot disappears for $| \phi | > \phi_c$ and the error is greater at $\phi \approx \phi_c$, but the error is very small   throughout most of the domain. So if simply removing the explicit condition for $m > 1$ doesn't work, the function could branch to follow this implementation instead.
Alternatively, the Cephes implementation could be replaced. For $-1 < m < 1$ and $-\frac{\phi}{2} < \phi < \frac{\phi}{2}$, the error looks like:

The logic for extending the domain to arbitrary $\phi$ for $m < 1$ could look like the following, although there might be a simpler way to do it:
def k_unit(phi, m):
    sign = np.sign(phi)
    phi = np.abs(phi)
    val = np.asarray(np.sin(phi) * Rf(np.cos(phi)**2, 1 - m*np.sin(phi)**2, 1))
    return val * sign

@np.vectorize
def k(phi, m):
    # standardize to positive phi
    sign = np.sign(phi)
    phi = np.abs(phi)

    # shift to [-np.pi, np.pi], but count the number
    # of complete periods
    n = (phi + np.pi/2) // np.pi
    phi = (phi + np.pi/2) % np.pi - np.pi/2

    # Add the appropriate number of complete integrals
    # to the result
    K = k_unit(np.pi/2, m)
    val = k_unit(phi, m)
    return sign*(val + 2*n*K)

It looks like the error would be comparable to what we have now for any values of $m$, $\phi$.

Here, the function does not produce incorrect results; rather, the domain is restricted compared to some versions of the function, and this restriction is not mentioned in the documentation. I've updated the label to reflect that this is really a documentation oversight and enhancement request rather than a "bug".

I disagree with this. If it's helpful I can give a simple proof to show that both of these integrals, using their common definition, are finite and real-valued for any $k \in (0, 1)$.
ellipkinc(np.arcsin(k), 1 / k**2)
ellipeinc(np.arcsin(k), 1 / k**2)

Maybe I have just cherry-picked your comment and we're actually both in agreement, but pointing out in case not.

I agree that $F(\sin^{-1}k, 1/k^2)$ is real-valued for $k < 1$ . But the implementation ellipkinc does not currently include k < 1 in its domain. Likewise, $F$ is defined in regions where the output would be complex. ellipkinc doesn't include those arguments in its domain either. Authors may choose to restrict the domain of their implementation of a function to a region in which they are prepared to accurately perform the calculation and test the results. There are many reasons why the domain of an implementation might not include all regions where a function might be defined. For instance, it may be challenging to perform the calculation numerically in some regions. I'm not sure why the authors the underlying Cephes function chose to restrict the domain here, but whatever it was, it looks intentional. There are many other instances of this in scipy, and even the Boost implementation of this function has the same restriction.
That said, It should definitely be documented, and requesting that the domain be extended is an excellent feature request.


Add "edit this page" link to every page

This will require a global prop containing the git base path.

Done


nixos/tests/qemu-vm-volatile-root: init

Description of changes
Add a test for mounting the root filesystem as a volatile tmpfs via the qemu-vm module.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@lilyinstarlight do you think it's a good idea to merge this first and then once we get it working with the systemd initrd, we parameterize the test?

Created the issue: https://github.com/NixOS/nixpkgs/issues/240175

@ofborg test qemu-vm-volatile-root

That's a good idea! I added it to the QEMU VM test refactor board. I can't see any systemd stage 1 board. Maybe only the team can see it?

That's a good idea! I added it to the QEMU VM test refactor board. I can't see any systemd stage 1 board. Maybe only the team can see it?

It's probably because it's a classic project. You should be able to see https://github.com/NixOS/nixpkgs/projects/51, but the qemu-vm test board is probably fine


[Addition]: Black Panther X From Italy Over 70 Self Witness Clusters

Hotspot b58 Addresses
11qRraCq1Z4kqReCqNXs1RJyQTAT1h4C11yLuvYRJxZkQVeTyQ
11n1Tr8BKDE7BPebh61VXa4zbddKGmB2roKjEL2g2aSi9yfiM19
112WckGK2YwFMVdh6EgLaLH2VQzTkASfNvapNYbKNwBppCGiQ14o
11rx3W3p4zMcYGoHqNuh9JrvmpEw3E52bpcWQVBHq1EUmPcL4Ua
11UFNQsBwUT1gwmfnAqP4QE6uKwCjFDN2Zeu3MsMxUtLocNoZtE
14rsXm7XB1uisbsswCLUNaEG9oAimR46qZk5LnNnshM7pNbiztY
112KHXMNhuJH8jQuLxoCqBWN41JGaEJPmJgLtrB7MTfsFd5B5KGD
112D1qMmegj1ThbiTpzVfMy9xvGXxXGj6BBEArhxz2Berxh9uYR9
112AHFyd3wvfQa32JyRqcjaeDu7fpgs173X58bPHwVejCyCmWMMH
112SXE3Nucpa2z9G87nEoVHFWouJYkHhxYGDyJNwM7i7JYt3YREF
11q4tjrGqLRV7tPwsbrhcBJs7Lqr4AQacYYsEZUtMHEFiJ8PNcu
11YQFiEUJvs79YQGdyqV1gTbmVD2KPT8xHKubGi6Lm9bMvbP6dB
112mLx2uqrCDuwuD38YM1hSaYV7jcnumHAjUwL7oZKNP2GgcH43Q
112fZuTTkorVd6zE19B5nKSph5PoDgdfrhVBTVsppNqu4tUdsrUJ
112aa2cJHjTAbZ9b3u4K6DgV85vM6AJtXo3R7TDKaWCA6sB8wAZu
Discord Handle
No response
Reason(s)

14rsXm7XB1 uisbsswCLUNaEG9oAimR46qZk5LnNnshM7pNbiztY This is just one of those wallets, Only a few of the hot spots are listed, and I believe there are more, from 5 purses, the same manufacturer Panther, only exist to witness each other, hoping to shut down the deceiver as soon as possible

Please look into #5277
The ip of these hotspots is Chinese. It is super big gaming.

These were in PR #6020 added via #5277

Please investigate # 6018 and # 5859, Panther Cluster

这些是在公关部 #6020 通过添加 #5277

Please investigate #6018 and #5859, Panther Cluster


Feature request: query-string strip (pick, exclude)

To eliminate unneeded params, preserve only needed
queryString.strip('http://example.com/?a=1&b=2', [ 'a' ] )
// result : http://example.com/?a=1

It would be helpful when page has many redudant params (like statistic params) and want to get a cleaner one

Better function names from @Uzlopak:
queryString.pick('http://example.com/?a=1&b=2', [ 'a' ] )
// result : http://example.com/?a=1

queryString.exclude('http://example.com/?a=1&b=2', [ 'a' ] )
// result : http://example.com/?b=2

strip is not a very obvious naming. Do you keep the ones you name or do you drop the ones you name?
pick or exclude should better names for this type of operation.

@Uzlopak strip is inspired by python string strip (remove spaces at two ends)
this word means 'remove all coverings from.'
And yes, it's not suit for exact function name for its inaccuracy，pick or exclude are better choices. (Maybe both can be implement)

The python strip equivalent in javacscript would be trim

I would name it .removeElements()

For a better implementation, this could preserve the order of url queries, cause input and output are both strings, which has order.

I would still prefer pick and exclude as removeElements could imply that you make not an idempotent operation.

Just opened a PR here that provides what you want : https://github.com/sindresorhus/query-string/pull/261 ;)

@BenoitSerrano Good job! How about consider the order? https://github.com/sindresorhus/query-string/issues/259#issuecomment-629629534?

@zhangciwu
@BenoitSerrano
I read the code, and I think it is reversing the order, by doing
return Object.assign({[currentKey]: query[currentKey]}, accumulator);
To conserve the order I suppose you would actually have to do it the other way round.
like
return Object.assign(accumulator, {[currentKey]: query[currentKey]});
or
return Object.assign({}, accumulator, {[currentKey]: query[currentKey]});
But i didnt test it...

@Uzlopak In ES2020 Object do preserve create order, but early versions there is no limitation, so it may have different result for different browser/node/other runtime

With this proposal, under most circumstances, for..in, Object.keys / values / entries, and JSON.stringify are guaranteed to iterate in order over:
(1) Numeric array keys
(2) non-Symbol keys, in insertion order
(3) Symbol keys, in insertion order

Relate: https://stackoverflow.com/questions/30076219/does-es6-introduce-a-well-defined-order-of-enumeration-for-object-properties

Updated it, it should preserve the parameters order ;)

looks good for me

If anyone wants to work on this, see the initial attempt and feedback in https://github.com/sindresorhus/query-string/pull/261.


Skip obfuscation for *.winmd library

ConfuserEx Version: 1.5.0
Target Framework: .NET Framework 4.8
Operating System: Windows 10 Pro

*.csproj v1
Here is my initial *.csproj file:
<project baseDir="obj\Release" outputDir="..\Release" xmlns="http://confuser.codeplex.com">
    <rule pattern="true" preset="aggressive" />
    <module path="MyProduct.exe">
        <rule pattern="true" preset="aggressive" inherit="false"></rule>
    </module>
</project>

Build fails with:

Confuser.MSBuild.targets(69, 5): Could not resolve method: System.Void Windows.Storage.Streams.InMemoryRandomAccessStream::Dispose() (Windows, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null, ContentType=WindowsRuntime)

My application uses Windows.winmd library.

*.csproj v2
Now I am trying to avoid Windows.winmd obfuscation with the following updates:
<project baseDir="obj\Release" outputDir="..\Release" xmlns="http://confuser.codeplex.com">
    <rule pattern="true" preset="aggressive" />
    <module path="MyProduct.exe">
        <rule pattern="true" preset="aggressive" inherit="false"></rule>
    </module>
    <module path="..\..\..\libs\Windows.winmd">
        <rule pattern="true" preset="none" inherit="false"></rule>
    </module>
</project>

This also failed, but this time on the other type:

Confuser.MSBuild.targets(69, 5): Could not resolve method: System.Object Windows.Foundation.Collections.IMapView`2<System.String,System.Object>::Lookup(System.String) (Windows, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null, ContentType=WindowsRuntime)

Since Windows.winmd does not relate to my business code, I would like to avoid obfuscation for that. Is it possible? Or how I can resolve the latest issue?

You can always declare a library as external.
You can do that by adding external="true" in the module tag. Those attributes are considered dependencies and will not get obfuscated.
The second error still points to confuserex being unable to find a specific type. Where is Windows.Foundation.Collections.IMapView2` located? You may need to add that library as well to the project.

Windows.Foundation.Collections.IMapView also in the same Windows.winmd library.

Regard external=true attribute, if I use it:
<project baseDir="obj\Release" outputDir="..\Release" xmlns="http://confuser.codeplex.com">
    <rule pattern="true" preset="aggressive" />
    <module path="Define.Pdf.exe">
        <rule pattern="true" preset="aggressive" inherit="false"></rule>
    </module>
    <module path="..\..\..\libs\Windows.winmd" external="true"></module>
</project>

it does not affect — it still throws initial exception about Windows.Storage.Streams.InMemoryRandomAccessStream::Dispose():

Confuser.MSBuild.targets(69, 5): Could not resolve method: System.Void Windows.Storage.Streams.InMemoryRandomAccessStream::Dispose() (Windows, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null, ContentType=WindowsRuntime)

There is a bug in 1.5 that may cause this. Could you give the 1.6 RC version a try. This should sort it out.

I was able to resolve this issue with "ref proxy".mode=strong setting. Here is my work .crproj:
<project baseDir="obj\Release" outputDir="..\Release" xmlns="http://confuser.codeplex.com">
    <rule pattern="true" preset="aggressive" inherit="false" />
    <module path="MyProduct.exe">
        <rule pattern="true" preset="aggressive" inherit="false">
            <protection id="ref proxy">
                <argument name="mode" value="strong" />
            </protection>
        </rule>
    </module>
</project>


Implement suggestions from the registry

See https://bholley.net/cargo-vet/performing-audits.html#suggestions-from-the-registry

The registry is getting big enough that it's worth doing this now.


Pasting in input without updating data in Vuejs

I am having problems when the user paste the value of the field(ctrl + v). The field fullfil the mask correctly, but the element in the data property in vuejs stays without the mask. Have you been through this?
Code:
import AwesomeMask from 'awesome-mask'

Vue.directive('mask', AwesomeMask);

new Vue ({
    el: '#inicialEmail',
    data: {
        action: '',
        requestType: '',
        inscricao: {
            nomcan: '',
            emailcan: '',
            celcan: ''
        }
    })

Example:
Copy paste:

Write:

Are you using which event on the template? Have you tried to get the value after Vue.nextTick?

@JoohnCruz

@iCaio i will try to use the  Vue.nextTick,
I just tried to use, important and as directive.
Example:
`import AwesomeMask from 'awesome-mask'
Vue.directive('mask', AwesomeMask);`
<input name="cpfcan" id="cpfcan" v-model="inscricao.cpfcan" v-mask="'999.999.999-99'">
No event is intended for this.

After some hours looking for it, I found this:
https://github.com/vuejs/vue/issues/2804
This is related to this Vue issue.
I think I have fixed it on version 0.4.2, can you try it @JoohnCruz ?

Issue closed due one week or ore of inactivity