Add directive to change plain HTTP port

Would be nice if we could have a directive to let caddy listen on a >1024 port for the plain HTTP requests.
Currently the vhost:port notation is only for HTTPS traffic, but I want the HTTP listener to listen on >1024 too to circumvent the bind permission denied problem.
Something like http_port 8080 would do fine.
Ideas?

Maybe I don't understand your question but Caddy can listen on > 1024 port for both http and https. And no, the vhost:port notation is for both http and https.

I would like to redirect HTTP to my HTTPS port, but due to a limitation in docker, I cannot let caddy listen on <1024 ports. So I would need two ports; one that I can map to 80 on the host, and one I can map to 443 on the host.

If you are not willing to use Let's Encrypt's automatic https, that should be pretty straightforward.
Share your Caddyfile and let us try to see what went wrong.

Example:
website.com:8080 {
    ...
}

If I now map the host 443 port to 8080 of the container, HTTPS will work. But what about HTTP traffic on port 80?

Looks like you're doing it wrong. That is gonna serve HTTP, not HTTPS.
Easy way out is to be explicit.
HTTP -> http://website.com:8080
HTTPS -> https://website.com:8080
The specified scheme will determine what happens.

Is this what you mean?
website.com:8080 { }
https://website.com:8081 {
    ....
}

Caddy logs;
Activating privacy features... done.
website.com:8080
website.com:8081
website.com:80

And a connection refused on port 80.

Based on that Caddyfile, I do not expect to see website.com:80. Do you have any other server block where the port is not specified ?
Ordinarily, this should be enough.
http://website.com:8080 { ... }
https://website.com:8081 { ... }

My caddyfile;
http://datapeak.hazcod.com:8080 { }
https://datapeak.hazcod.com:8081 {
    header / {
        "Strict-Transport-Security" "max-age=63072000; includeSubdomains; preload"
        "X-Frame-Options" "DENY"
        "X-Content-Type-Options" "nosniff"
        "X-XSS-Protection" "1; mode=block"
        -Server
    }

   gzip

    proxy / mgmt:8080 {
        #header_downstream -Server
        proxy_header X-Forwarded-For {host}
    }
}

http://dav.datapeak.hazcod.com:8080 { }
https://dav.datapeak.hazcod.com:8081 {
    header / {
        "Strict-Transport-Security" "max-age=63072000; includeSubdomains; preload"
        "X-Frame-Options" "DENY"
        "X-Content-Type-Options" "nosniff"
        "X-XSS-Protection" "1; mode=block"
        -Server
    }

    gzip

    proxy / dav:8080 {
        #header_downstream -Server
        proxy_header X-Forwarded-For {host}
    }
}

This is expected. Automatic HTTPS uses port 80 to redirect plaintext traffic to encrypted traffic.

@mholt I do want HTTPS, but I cannot use ports <1024 because else I need to run caddy as root.
In docker, the storage backend AUFS does not support xattr, so no setcap.

@HazCod With Caddy, there is difference between HTTPS and automatic HTTPS. For now, you need ports 80 and 443 to use automatic HTTPS.
Looks like the option you have now is to specify your certs.

Yes, in that case then you need to use old-fashioned manual TLS.
Now that Caddy fully supports all 3 challenge types (in the 0.9 beta), maybe we can think about a flag to change the HTTP port from 80 to something else.

@mholt make sense :)

@mholt do you think this will be easy to implement?

@HazCod Yes, probably will. I guess a command line flag registered from the httpserver package should do the trick; then use its value instead of the hard-coded one.

Although, on second thought, we should consider the ramifications of such an option. Changing HTTP from port 80 to something else will break the ACME HTTP challenge.

Well it will, unless that port is on port 80 externally. Maybe add a disclaimer in the README?

Sure, we can document it on the CLI page.

Side note: I would really love this feature, as being able to run a docker container as a non-root user means that we can't bind to any protected ports (433 or 80). I would love to be able to run automatic https on ports 8080 and 4433 inside of Caddy, so I can run as a non-root user, even though I'm mapping the ports outside 80->8080 443->4433
:)

The more I think about this the more I don't like it, it seems like a hack to work around a deficiency in some software that frankly you probably don't even need in order to run Caddy, a fully self-contained, static binary.

The way I see it it's an effective way to not needing to run caddy as root. Personally, I don't feel comfortable doing this until it has at least privilege downgrading.

I run one main caddy on the host and proxing inside my containers running also caddys... where's the problem using this approach?

In my opinion it would be nice to let users tell their default http and https port via cli. It might be, that all sites need to run on 8080 and one would dislike having to type out every :8080. Or the aforementioned non-root issues.
Why not just have something like: ./caddy -default-http:8080 -default-https:8443. This would let caddy listen on these ports by default even for the https challenge.
example.com {
}

^ this would listen on :8080 and :8443 easily without setcap and root privileges and also enable https through these ports.

While this is possible, are the implications of this clear? You will need ports 80 and 443 to solve the ACME challenges, unless you configure the DNS challenge. Right now that's basically a given. But then you change the default HTTP and HTTPS ports because you can't bind to 80 or 443 and then suddenly your sites break when your certs expire because they require ports 80 and 443 (unless you configure DNS too).

I think the implications are quite clear. One reason to expose this as cli instead of a directive in my opinion. This is a workaround in my opinion for setups with standard ports not usable. Be that due to running as non-root in containers or in enterprise setups, where multiple ports are already bound.
Not saying it makes sense to jump into this immediately, but should be considered in my opinion. Right now in my specific use cases either the startup process runs as root or caddy itself is, which is not perfect. Still trying to get a workaround working for kubernetes without using root to set the bind capability.
Thanks for taking the time to let us reason around this even so it might not be considered standard use case.

Considering the effort I went through tracking the possibilities to deescalate the permissions in different environments (I have to admit mainly chroot and containers) the sane way would be to use setcap, which in itself needs root access. Making it possible to bind to higher ports and doing the translation to the lower ports on the scheduler/NAT etc. side might make sense. This would enable starting up caddy without any need for root.
So there are various ways to address this:

setcap + root for starting up
root (not every environment supports setcap)
suggest DNS challenge as a workaround
make ports configurable

As said above, I might now consider switchable ports the easiest solution. For any feedback how to deescalate better, I am all ears.

@stp-ip : mind you that setcap will not work on all docker storage backends.
(Aufs (the default) does not support xattr)

Yeah that's what I meant with not supported everywhere. Currently looking into using capsh and deescalation on that route, but that's not the discussion here.

For informations sake this is were I am stopping for today:
Still needs root to run capsh, but circumvents the xattr support issue or at least I think it should.
capsh --keep=1 --user='www-data' --caps='cap_net_bind_service=+ep' --inh='cap_net_bind_service' -- -c '/usr/bin/caddy --quic --conf=/etc/config/caddyfile'
This will spawn caddy as a unprivileged user with the added cap (not yet working on my side so).
Then you also need to take care of the /root/.caddy path and also make /root/.ssh writable to the "www-data" user, when using git.
Perhaps that helps someone until we figure out, what the best solution is.

This seems to work with a from source build of libcap:
capsh --caps='cap_net_bind_service+ep cap_setpcap,cap_setuid,cap_setgid+ep' --keep=1 --user='www-data' --addamb='cap_net_bind_service' -- -c '/usr/bin/caddy --conf=/Caddyfile'"

Hi @mholt, what is the current stopper for the PR?

@HazCod This is an issue, not a PR -- but if you're asking about the previous PR linked to this issue: it was implementing a change that was actually orthogonal to this issue. This issue isn't about making ACME ports configurable, it's about changing the port that Caddy considers the plaintext HTTP port (currently hardcoded as 80). So we simply need a new PR that sets a variable to 80 and has Caddy use that variable in all relevant places. :)

https://github.com/HazCod/caddy/tree/patch-1 but i'm still trying to compile my first go project. This might take a while ;-)

@HazCod Nice -- not too far off! I've left a comment inline for you to review.

Chiming in: I'm on OpenBSD where I don't have cap_net_bind_service, so I have to redirect ports with pf instead.
However, http_port sounds confusing to me. The result would be:
https://example.com:4443 {
    http_port 4080
    listen <IP_ADDRESS>
}

And the result will be that Caddy binds on <IP_ADDRESS>:4080 and <IP_ADDRESS>:4443, thinking the host is example.com:4443 when it's actually example.com.
I like (and run) @tdsmith's patch. The HTTPS port should definitely be configurable, too.
Secondarily, if I use the DNS challenge and don't want the redirect, the HTTP port should be possible to disable.

Looking a bit more at @tdsmith's patch, I think the command line option should not just change the ACME port, but also the default listening and redirect one.
I'll give a patch a try.

@mholt I tried writing my first bits, only thing left is getting this config from httpserver.go to the tlsConfig variable in the server.go/NewServer function. Ideas?

@HazCod The name acmePort is misleading; ACME uses multiple ports for different reasons. Try calling it HTTPPort instead (and the flag being httpport). It really has nothing to do with ACME, it's just the port that Caddy considers to be the "default" port for HTTP.

Yeah sorry, I currently don't have enough time to learn myself go for this PR. Just wanted to let you know.. :(

Thanks @HazCod, I appreciate the effort anyway!

Thanks again @mholt, can finally run it as non-root! :D


RefreshIndicator with Flexible widget crashes

Steps to Reproduce

code sample
  Widget buildBody(BuildContext context) {
    final listView = RefreshIndicator(
      key: _refreshIndicatorKey,
      onRefresh: reloadData,
      child: Flexible(
        child: Scrollbar(
          child: ListView.builder(
            scrollDirection: Axis.vertical,
            shrinkWrap: true,
            itemCount: list.length,
            itemBuilder: (context, index) {
              if (index == list.length - 1 && !isLoading && !endReached) {
                SchedulerBinding.instance.addPostFrameCallback((_) {
                  print('reloading - last element reached: $index');
                  loadData();
                });
              }
              var w = listBuilder(context, index);
              if (widget.canEditItems)
                w = GestureDetector(
                  child: w,
                  onTap: () => editItem(index),
                );
              return w;
            },
          ),
        ),
      ),
    );

    final header = buildHeader();
    var widgetTree = <Widget>[listView];
    if (header != null) widgetTree.insert(0, header);
    widgetTree = <Widget>[
      Column(
        children: widgetTree,
      )
    ];

    if (isLoading)
      widgetTree.add(Stack(
        children: [
          new Opacity(
            opacity: 0.3,
            child: const ModalBarrier(dismissible: false, color: Colors.grey),
          ),
          new Center(
            child: new CircularProgressIndicator(),
          ),
        ],
      ));
    return new Stack(children: widgetTree);
  }

Logs

logs
Performing hot reload...
Syncing files to device LGM G600L...
Reloaded 17 of 669 libraries in 1,204ms.
I/ViewRootImpl(21658): ViewRoot's Touch Event : ACTION_DOWN
I/ViewRootImpl(21658): ViewRoot's Touch Event : ACTION_UP
I/flutter (21658): ══╡ EXCEPTION CAUGHT BY WIDGETS LIBRARY ╞═══════════════════════════════════════════════════════════
I/flutter (21658): The following assertion was thrown building
I/flutter (21658): RefreshIndicator-[LabeledGlobalKey<RefreshIndicatorState>#499ec](dependencies:
I/flutter (21658): [_LocalizationsScope-[GlobalKey#5edc2], _InheritedTheme], state:
I/flutter (21658): RefreshIndicatorState#57572(tickers: tracking 2 tickers)):
I/flutter (21658): 'package:flutter/src/widgets/basic.dart': Failed assertion: line 4272 pos 12:
I/flutter (21658): 'renderObject.parentData is FlexParentData': is not true.
I/flutter (21658):
I/flutter (21658): Either the assertion indicates an error in the framework itself, or we should provide substantially
I/flutter (21658): more information in this error message to help you determine and fix the underlying cause.

I/flutter (21658): reloading - last element reached: 11
I/flutter (21658): Another exception was thrown: 'package:flutter/src/widgets/framework.dart': Failed assertion: line 4971 pos 12: '_children.contains(child)': is not true.
I/flutter (21658): Another exception was thrown: RenderStack object was given an infinite size during layout.
F/flutter (21658): [FATAL:flutter/flow/layers/transform_layer.cc(28)] Check failed: transform_.isFinite().

D:\dev\pavement>flutter analyze
Analyzing pavement...

   info - Name non-constant identifiers using lowerCamelCase - lib\map\measure.dart:144:11 - non_constant_identifier_names
   info - Name non-constant identifiers using lowerCamelCase - lib\map\measure.dart:148:11 - non_constant_identifier_names
   info - The member 'fromMap' can only be used within 'package:google_maps_flutter/src/camera.dart' or a test - lib\map\measure.dart:200:42 - invalid_use_of_visible_for_testing_member

3 issues found. (ran in 1.9s)

flutter doctor -v
D:\dev\pavement>flutter doctor -v
[√] Flutter (Channel stable, v1.5.4-hotfix.2, on Microsoft Windows [Version 10.0.17134.765], locale en-GB)
    • Flutter version 1.5.4-hotfix.2 at C:\tools\flutter
    • Framework revision 7a4c33425d (8 weeks ago), 2019-04-29 11:05:24 -0700
    • Engine revision 52c7a1e849
    • Dart version 2.3.0 (build 2.3.0-dev.0.5 a1668566e5)

[√] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at c:\tools\Android-SDK
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = c:\tools\Android-SDK
    • Java binary at: C:\tools\android-studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1343-b01)
    • All Android licenses accepted.

[√] Android Studio (version 3.4)
    • Android Studio at C:\tools\android-studio
    • Flutter plugin version 36.1.1
    • Dart plugin version 183.6270
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1343-b01)

[!] VS Code, 32-bit edition (version 1.34.0)
    • VS Code at C:\Program Files (x86)\Microsoft VS Code
    X Flutter extension not installed; install from
      https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[√] Connected device (1 available)
    • LGM G600L • LGMG600Lf8469764 • android-arm64 • Android 8.0.0 (API 26)

! Doctor found issues in 1 category.

We are going to need more information in order to investigate this. Looking at the code snippet and stack trace, I don't see anything obvious that would be causing the issue. Can you provide a complete, runnable example with instructions on how to reproduce the problem?

We are going to need more information in order to investigate this. Looking at the code snippet and stack trace, I don't see anything obvious that would be causing the issue. Can you provide a complete, runnable example with instructions on how to reproduce the problem?

Run
Pull to refresh
Crash with I/flutter (13511): 'package:flutter/src/widgets/basic.dart': Failed assertion: line 4272 pos 12: I/flutter (13511): 'renderObject.parentData is FlexParentData': is not true.

import 'package:flutter/material.dart';
import 'package:flutter/scheduler.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        // This is the theme of your application.
        //
        // Try running your application with "flutter run". You'll see the
        // application has a blue toolbar. Then, without quitting the app, try
        // changing the primarySwatch below to Colors.green and then invoke
        // "hot reload" (press "r" in the console where you ran "flutter run",
        // or simply save your changes to "hot reload" in a Flutter IDE).
        // Notice that the counter didn't reset back to zero; the application
        // is not restarted.
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  final canEditItems = false;

  MyHomePage({Key key, this.title}) : super(key: key);

  // This widget is the home page of your application. It is stateful, meaning
  // that it has a State object (defined below) that contains fields that affect
  // how it looks.

  // This class is the configuration for the state. It holds the values (in this
  // case the title) provided by the parent (in this case the App widget) and
  // used by the build method of the State. Fields in a Widget subclass are
  // always marked "final".

  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  var _refreshIndicatorKey = GlobalKey();

  bool endReached = false;

  bool isLoading = false;

  @override
  Widget build(BuildContext context) {
    // This method is rerun every time setState is called, for instance as done
    // by the _incrementCounter method above.
    //
    // The Flutter framework has been optimized to make rerunning build methods
    // fast, so that you can just rebuild anything that needs updating rather
    // than having to individually change instances of widgets.
    return Scaffold(
      appBar: AppBar(
        // Here we take the value from the MyHomePage object that was created by
        // the App.build method, and use it to set our appbar title.
        title: Text(widget.title),
      ),
      body: buildBody(context),
    );
  }

  final list = List.generate(22, (index) => Map());

  buildBody(BuildContext context) {
    final listView = RefreshIndicator(
      key: _refreshIndicatorKey,
      onRefresh: reloadData,
      child: Flexible(
        child: Scrollbar(
          child: ListView.builder(
            scrollDirection: Axis.vertical,
            shrinkWrap: true,
            itemCount: list.length,
            itemBuilder: (context, index) {
              if (index == list.length - 1 && !isLoading && !endReached) {
                SchedulerBinding.instance.addPostFrameCallback((_) {
                  print('reloading - last element reached: $index');
                  loadData();
                });
              }
              var w = listBuilder(context, index);
              if (widget.canEditItems)
                w = GestureDetector(
                  child: w,
                  onTap: () => editItem(index),
                );
              return w;
            },
          ),
        ),
      ),
    );

    final header = buildHeader();
    var widgetTree = <Widget>[listView];
    if (header != null) widgetTree.insert(0, header);
    widgetTree = <Widget>[
      Column(
        children: widgetTree,
      )
    ];

    if (isLoading)
      widgetTree.add(Stack(
        children: [
          new Opacity(
            opacity: 0.3,
            child: const ModalBarrier(dismissible: false, color: Colors.grey),
          ),
          new Center(
            child: new CircularProgressIndicator(),
          ),
        ],
      ));
    return new Stack(children: widgetTree);
  }

  Future<void> reloadData() {
    return Future(null);
  }

  listBuilder(BuildContext context, int index) {
    return ListTile(
      title: Text(index.toString()),
    );
  }

  void loadData() {}

  editItem(int index) {}

  buildHeader() {
    return Container(
      height: 33,
      color: Colors.green,
    );
  }
}

Thanks for taking the time to put together the sample app. It does indeed illustrate the problem. The core issue here is that the Flexible widget needs to be either a direct child of the Column widget or at least have no StatefulWidgets between it and the Column. In this case you have a RefreshIndicator (which is a StatefulWidget) between the Flexible and the Column. This restriction is described in more detail in the docs for Flexible.
So to fix the code above you can just make Flexible the parent of RefreshIndicator instead of the other way around:
...

  buildBody(BuildContext context) {
    final listView = Flexible(
      child: RefreshIndicator(
        key: _refreshIndicatorKey,
        onRefresh: reloadData,
        child: Scrollbar(
            child: ListView.builder(

...

Normally this would be reported with a better error message that would have helped point out the problem. However there appears to be something specific about the way the RefreshIndicator is interacting with the other widgets that prevented reporting the problem with the better error message. We will need to look into why that is and hopefully come up with a better way of handling this situation.

@darrenaustin
The docs by your link say different

the path from the Flexible widget to its enclosing Row, Column, or Flex must contain only StatelessWidgets or StatefulWidgets (not other kinds of widgets, like RenderObjectWidgets).

@duzenko, you are right, I misread the docs there. Mea culpa.
The above fix should still work for the sample code. I think the RefreshIndicator widget's implementation causes it to add other kinds of widgets between the Column and the Flexible widget when it becomes active. We need to look into this and see if there is a way to handle this situation better.

Issue still exist

flutter doctor -v
[✓] Flutter (Channel dev, v1.18.0-8.0.pre, on Mac OS X 10.15.4 19E287, locale
    en-GB)
    • Flutter version 1.18.0-8.0.pre at /Users/taha/Code/flutter_dev
    • Framework revision e0c63cd35e (5 days ago), 2020-04-23 22:29:01 -0400
    • Engine revision d2ec21221e
    • Dart version 2.9.0 (build 2.9.0-3.0.dev b0d35855d8)

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/taha/Code/sdk
    • Platform android-29, build-tools 29.0.3
    • ANDROID_HOME = /Users/taha/Code/sdk
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_212-release-1586-b4-5784211)
    • All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 11.4.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.4.1, Build version 11E503a
    • CocoaPods version 1.9.1
[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 3.6)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 45.1.1
    • Dart plugin version 192.7761
    • Java version OpenJDK Runtime Environment (build
      1.8.0_212-release-1586-b4-5784211)
[✓] VS Code (version 1.44.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.9.1
[✓] Connected device (5 available)
    • SM M305F      • 32003c30dc19668f          • android-arm64  • Android 10
      (API 29)
    • Taha’s iPhone • 00008020-001059882212002E • ios            • iOS 13.4.1
    • macOS         • macOS                     • darwin-x64     • Mac OS X
      10.15.4 19E287
    • Web Server    • web-server                • web-javascript • Flutter Tools
    • Chrome        • chrome                    • web-javascript • Google Chrome
      81.0.4044.129
• No issues found!

flutter doctor -v
[✓] Flutter (Channel stable, 1.22.5, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.22.5 at /Users/tahatesser/Code/flutter_stable
    • Framework revision<PHONE_NUMBER> (4 weeks ago), 2020-12-10 11:54:40 -0800
    • Engine revision ae90085a84
    • Dart version 2.10.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.0

[!] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.1

[✓] Connected device (2 available)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E            • ios • iOS 14.3
    • iPhone 12 (mobile)   • 181EDE6A-AA9B-4619-8352-9CD6F7A9CD61 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)

! Doctor found issues in 1 category.

[✓] Flutter (Channel master, 1.26.0-2.0.pre.222, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.26.0-2.0.pre.222 at /Users/tahatesser/Code/flutter_master
    • Framework revision 655cd1e2a6 (6 hours ago), 2021-01-07 03:19:03 -0500
    • Engine revision 7f00e2f6c5
    • Dart version 2.12.0 (build 2.12.0-179.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.1

[✓] Connected device (4 available)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E            • ios            • iOS 14.3
    • iPhone 12 (mobile)   • 181EDE6A-AA9B-4619-8352-9CD6F7A9CD61 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)      • macos                                • darwin-x64     • macOS 11.1 20C69 darwin-x64
    • Chrome (web)         • chrome                               • web-javascript • Google Chrome 87.0.4280.141

• No issues found!

flutter doctor -v
[✓] Flutter (Channel stable, 1.22.5, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.22.5 at /Users/tahatesser/Code/flutter_stable
    • Framework revision<PHONE_NUMBER> (4 weeks ago), 2020-12-10 11:54:40 -0800
    • Engine revision ae90085a84
    • Dart version 2.10.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.0

[!] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.1

[✓] Connected device (2 available)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E            • ios • iOS 14.3
    • iPhone 12 (mobile)   • 181EDE6A-AA9B-4619-8352-9CD6F7A9CD61 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)

! Doctor found issues in 1 category.

[✓] Flutter (Channel master, 1.26.0-2.0.pre.222, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.26.0-2.0.pre.222 at /Users/tahatesser/Code/flutter_master
    • Framework revision 655cd1e2a6 (6 hours ago), 2021-01-07 03:19:03 -0500
    • Engine revision 7f00e2f6c5
    • Dart version 2.12.0 (build 2.12.0-179.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.1

[✓] Connected device (4 available)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E            • ios            • iOS 14.3
    • iPhone 12 (mobile)   • 181EDE6A-AA9B-4619-8352-9CD6F7A9CD61 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)      • macos                                • darwin-x64     • macOS 11.1 20C69 darwin-x64
    • Chrome (web)         • chrome                               • web-javascript • Google Chrome 87.0.4280.141

• No issues found!

Hi @duzenko, I ran your code sample I am not able to reproduce the issue. I believe this is fixed

code sample
import 'package:flutter/material.dart';
import 'package:flutter/scheduler.dart';

void main() => runApp(const MyApp());

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: const MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  final canEditItems = false;

  const MyHomePage({Key? key, required this.title}) : super(key: key);

  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  final _refreshIndicatorKey = GlobalKey();

  bool endReached = false;

  bool isLoading = false;

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: buildBody(context),
    );
  }

  final list = List.generate(22, (index) => {});

  buildBody(BuildContext context) {
    final listView = RefreshIndicator(
      key: _refreshIndicatorKey,
      onRefresh: reloadData,
      child: Flexible(
        child: Scrollbar(
          child: ListView.builder(
            scrollDirection: Axis.vertical,
            shrinkWrap: true,
            itemCount: list.length,
            itemBuilder: (context, index) {
              if (index == list.length - 1 && !isLoading && !endReached) {
                SchedulerBinding.instance.addPostFrameCallback((_) {
                  print('reloading - last element reached: $index');
                  loadData();
                });
              }
              var w = listBuilder(context, index);
              if (widget.canEditItems) {
                w = GestureDetector(
                  child: w,
                  onTap: () => editItem(index),
                );
              }
              return w;
            },
          ),
        ),
      ),
    );

    final header = buildHeader();
    var widgetTree = <Widget>[listView];
    if (header != null) widgetTree.insert(0, header);
    widgetTree = <Widget>[
      Column(
        children: widgetTree,
      )
    ];

    if (isLoading) {
      widgetTree.add(Stack(
        children: const [
          Opacity(
            opacity: 0.3,
            child: ModalBarrier(dismissible: false, color: Colors.grey),
          ),
          Center(
            child: CircularProgressIndicator(),
          ),
        ],
      ));
    }
    return Stack(children: widgetTree);
  }

  Future<void> reloadData() {
    return Future(() => null);
  }

  listBuilder(BuildContext context, int index) {
    return ListTile(
      title: Text(index.toString()),
    );
  }

  void loadData() {}

  editItem(int index) {}

  buildHeader() {
    return Container(
      height: 33,
      color: Colors.green,
    );
  }
}

output 1
Restarted application in 319ms.

════════ Exception caught by widgets library ═══════════════════════════════════
The following assertion was thrown while applying parent data.:
Incorrect use of ParentDataWidget.

The ParentDataWidget Flexible(flex: 1) wants to apply ParentData of type FlexParentData to a RenderObject, which has been set up to accept ParentData of incompatible type StackParentData.

Usually, this means that the Flexible widget has the wrong ancestor RenderObjectWidget. Typically, Flexible widgets are placed directly inside Flex widgets.
The offending Flexible is currently placed inside a Stack widget.

The ownership chain for the RenderObject that received the incompatible parent data was:
  RepaintBoundary ← NotificationListener<ScrollNotification> ← NotificationListener<ScrollMetricsNotification> ← CupertinoScrollbar ← Scrollbar ← Flexible ← NotificationListener<OverscrollIndicatorNotification> ← NotificationListener<ScrollNotification> ← Stack ← RefreshIndicator-[GlobalKey#a15c5] ← ⋯
When the exception was thrown, this was the stack
#0      RenderObjectElement._updateParentData.<anonymous closure>
#1      RenderObjectElement._updateParentData
#2      RenderObjectElement.attachRenderObject
#3      RenderObjectElement.mount
#4      SingleChildRenderObjectElement.mount
...     Normal element mounting (45 frames)
#49     Element.inflateWidget
#50     MultiChildRenderObjectElement.inflateWidget
#51     MultiChildRenderObjectElement.mount
...     Normal element mounting (9 frames)
#60     Element.inflateWidget
#61     MultiChildRenderObjectElement.inflateWidget
#62     MultiChildRenderObjectElement.mount
#63     Element.inflateWidget
#64     MultiChildRenderObjectElement.inflateWidget
#65     MultiChildRenderObjectElement.mount
...     Normal element mounting (25 frames)
#90     Element.inflateWidget
#91     MultiChildRenderObjectElement.inflateWidget
#92     MultiChildRenderObjectElement.mount
...     Normal element mounting (136 frames)
#228    Element.inflateWidget
#229    MultiChildRenderObjectElement.inflateWidget
#230    MultiChildRenderObjectElement.mount
...     Normal element mounting (172 frames)
#402    Element.inflateWidget
#403    MultiChildRenderObjectElement.inflateWidget
#404    MultiChildRenderObjectElement.mount
...     Normal element mounting (498 frames)
#902    Element.inflateWidget
#903    Element.updateChild
#904    RenderObjectToWidgetElement._rebuild
#905    RenderObjectToWidgetElement.mount
#906    RenderObjectToWidgetAdapter.attachToRenderTree.<anonymous closure>
#907    BuildOwner.buildScope
#908    RenderObjectToWidgetAdapter.attachToRenderTree
#909    WidgetsBinding.attachRootWidget
#910    WidgetsBinding.scheduleAttachRootWidget.<anonymous closure>
#914    _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:192:26)
(elided 3 frames from class _Timer and dart:async-patch)
════════════════════════════════════════════════════════════════════════════════

════════ Exception caught by rendering library ═════════════════════════════════
A RenderFlex overflowed by 558 pixels on the bottom.
The relevant error-causing widget was
Column
════════════════════════════════════════════════════════════════════════════════
flutter: reloading - last element reached: 21

Refreshindicator has been Swapped with Flexible to fix the exception in code sample1

modified code sample
import 'package:flutter/material.dart';
import 'package:flutter/scheduler.dart';

void main() => runApp(const MyApp());

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: const MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  final canEditItems = false;

  const MyHomePage({Key? key, required this.title}) : super(key: key);

  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  final _refreshIndicatorKey = GlobalKey();

  bool endReached = false;

  bool isLoading = false;

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: buildBody(context),
    );
  }

  final list = List.generate(22, (index) => {});

  buildBody(BuildContext context) {
    final listView = Expanded(
      child: RefreshIndicator(
        key: _refreshIndicatorKey,
        onRefresh: reloadData,
        child: Scrollbar(
          child: ListView.builder(
            scrollDirection: Axis.vertical,
            shrinkWrap: true,
            itemCount: list.length,
            itemBuilder: (context, index) {
              if (index == list.length - 1 && !isLoading && !endReached) {
                SchedulerBinding.instance.addPostFrameCallback((_) {
                  print('reloading - last element reached: $index');
                  loadData();
                });
              }
              var w = listBuilder(context, index);
              if (widget.canEditItems) {
                w = GestureDetector(
                  child: w,
                  onTap: () => editItem(index),
                );
              }
              return w;
            },
          ),
        ),
      ),
    );

    final header = buildHeader();
    var widgetTree = <Widget>[listView];
    if (header != null) widgetTree.insert(0, header);
    widgetTree = <Widget>[
      Column(
        children: widgetTree,
      )
    ];

    if (isLoading) {
      widgetTree.add(Stack(
        children: const [
          Opacity(
            opacity: 0.3,
            child: ModalBarrier(dismissible: false, color: Colors.grey),
          ),
          Center(
            child: CircularProgressIndicator(),
          ),
        ],
      ));
    }
    return Stack(children: widgetTree);
  }

  Future<void> reloadData() {
    return Future(() => null);
  }

  listBuilder(BuildContext context, int index) {
    return ListTile(
      title: Text(index.toString()),
    );
  }

  void loadData() {}

  editItem(int index) {}

  buildHeader() {
    return Container(
      height: 33,
      color: Colors.green,
    );
  }
}

flutter doctor -v (mac)
[✓] Flutter (Channel master, 3.7.0-3.0.pre.33, on macOS 12.6 21G115 darwin-arm64, locale en-IN)
    • Flutter version 3.7.0-3.0.pre.33 on channel master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision<PHONE_NUMBER> (38 minutes ago), 2022-12-05 18:27:21 -0800
    • Engine revision a309d239c4
    • Dart version 2.19.0 (build 2.19.0-463.0.dev)
    • DevTools version 2.20.0
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly
      to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0-rc4)
    • Android SDK at /Users/mahesh/Library/Android/sdk
    • Platform android-33, build-tools 33.0.0-rc4
    • ANDROID_HOME = /Users/mahesh/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14A400
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] IntelliJ IDEA Community Edition (version 2021.2.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 61.2.4
    • Dart plugin version 212.5080.8

[✓] VS Code (version 1.70.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.55.20221129

[✓] Connected device (3 available)
    • iPhone 12 Pro (mobile) • 026D5789-9E78-4AD5-B1B2-3F8D4E7F65E4 • ios            •
      com.apple.CoreSimulator.SimRuntime.iOS-14-5 (simulator)
    • macOS (desktop)        • macos                                • darwin-arm64   • macOS 12.6 21G115 darwin-arm64
    • Chrome (web)           • chrome                               • web-javascript • Google Chrome 108.0.5359.94

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

[✓] Flutter (Channel stable, 3.3.9, on macOS 12.6 21G115 darwin-arm, locale en-IN)
    • Flutter version 3.3.9 on channel stable at /Users/mahesh/Development/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision b8f7f1f986 (24 hours ago), 2022-11-23 06:43:51 +0900
    • Engine revision 8f2221fbef
    • Dart version 2.18.5
    • DevTools version 2.15.0

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0-rc4)
    • Android SDK at /Users/mahesh/Library/Android/sdk
    • Platform android-33, build-tools 33.0.0-rc4
    • ANDROID_HOME = /Users/mahesh/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14A400
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] IntelliJ IDEA Community Edition (version 2021.2.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 61.2.4
    • Dart plugin version 212.5080.8

[✓] VS Code (version 1.70.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.53.20221101

[✓] Connected device (3 available)
    • iPhone 12 Pro (mobile) • 026D5789-9E78-4AD5-B1B2-3F8D4E7F65E4 • ios            •
      com.apple.CoreSimulator.SimRuntime.iOS-14-5 (simulator)
    • macOS (desktop)        • macos                                • darwin-arm64   • macOS 12.6 21G115 darwin-arm
    • Chrome (web)           • chrome                               • web-javascript • Google Chrome 107.0.5304.110

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

Closing as fixed. Incase you still encounter the issue please feel free to write in the comments and we will reopen it.
Thank you.


Help to Marcin Purtak

at first: Could you have a look which flows can we take ( from each of the connectors type) and use it as a development test ?

2nd: prepare them for dev prefect container and register with project = dev

RUN_CONFIG = DockerRun(
    image="ghcr.io/dyvenia/viadot/viadot:dev_test",
    labels=["dev"],
)

3rd: prepare develpment container that will have office365 installed

4rd: run those flows on the prefect dev agent with the link to this dev container

Finally, I sent all my progress to Marcin because I must keep working on SERI project.


Allow jsonBlob updates to include null values

See Hoist Core PR: https://github.com/xh/hoist-core/pull/216
Hoist P/R Checklist
Pull request authors: Review and check off the below. Items that do not apply can also be
checked off to indicate they have been considered. If unclear if a step is relevant, please leave
unchecked and note in comments.

[x] Caught up with develop branch as of last change.
[x] Added CHANGELOG entry, or determined not required.
[N/R] Reviewed for breaking changes, added breaking-change label + CHANGELOG if so.
[N/R] Updated doc comments / prop-types, or determined not required.
[N/R] Reviewed and tested on Mobile, or determined not required.
[N/R] Created Toolbox branch / PR, or determined not required.

If your change is still a WIP, please use the "Create draft pull request" option in the split
button below to indicate it is not ready yet for a final review.

Pull request reviewers: when merging this P/R, please consider using a squash commit to
collapse multiple intermediate commits into a single commit representing the overall feature
change. This helps keep the commit log clean and easy to scan across releases. PRs containing a
single commit should be rebased when possible.

Not eager to add a change that requires a Hoist Core upgrade for this minor
release.
In any case, the hoist-react PR should probably specify a bumped minimum
version.
On Thu, Oct 21, 2021 at 11:42 AM Petra Laohakul @.***>
wrote:

@.**** commented on this pull request.
In svc/JsonBlobService.js
https://github.com/xh/hoist-react/pull/2744#discussion_r733807196:

@@ -69,12 +69,13 @@ export class JsonBlobService extends HoistService {
* @param {string} [data.description]
*/
async updateAsync(token, {name, value, meta, description}) {

   const params = {token};

   if (name) params.name = name;

   if (value) params.value = JSON.stringify(value);

   if (meta) params.meta = JSON.stringify(meta);

   if (description) params.description = description;

   return XH.fetchJson({url: 'xh/updateJsonBlob', params});

   return XH.fetchJson({

       url: 'xh/updateJsonBlob',

       params: {

           token,

           payload: JSON.stringify({name, value, meta, description})

yup! :) i love that about Json.stringify and undefined properties!
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/xh/hoist-react/pull/2744#discussion_r733807196, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AARTL2OQWONSX5M3BKZJY73UIAYIHANCNFSM5GOJSEAA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


desktop client 2.7 cannot connect to ocis-eos setup

This may not be a valid test against ocis 1.0.0-beta4 due to https://github.com/owncloud/ocis/issues/260
Setup: https://gitea.owncloud.services/jw/hetzner/src/branch/master/make_ocis_eos_test.sh

Reference: https://owncloud.github.io/ocis/eos/
on an ubuntu 20.04 host
using default master branch of owncloud/ocis.

To make openidconnect work, we map port 9200 to localhost via ssh -f -L 9200:localhost:9200 root@$ipaddr sleep 300
test with ownCloud version 2.7.0daily20200510 (build 13907)

https://localhost:9200 shows a certificate warning

after [x] trust anyway,  firefox also shows a certificat warning

After 'accept the risk', the user is redirected to the IDP, einstein can be logged in, the client is authenticated. -> OK
The client can select what to sync, existing folders at einstein's account are shown. -> OK
The installation wizard offers ( ) choose what to sync. Existing folders are shown -> OK

When the user chooses what to sync (or selects sync everything), the sync fails with a PROPFIND error:

2020-05-12T01:33:01+02:00 DBG userinfo request unauthorized error=invalid_token desc="token is expired by 4s" service=konnectd
2020-05-12T01:33:01+02:00 DBG  bytes=0 duration=0.542016 method=GET path=/konnect/v1/userinfo proto=HTTP/1.1 request=08a2b49b-1a84-46fb-b835-3de1bdb51ee3 service=konnectd status=401
2020-05-12T01:33:01+02:00 WRN root/go/pkg/mod/github.com/cs3org/reva@v0.1.1-0.20200427161359-c1549a8110eb/internal/grpc/services/authprovider/authprovider.go:107 > error authenticating user error="authsvc: error in Authenticate: oidc: error getting userinfo: +401 Unauthorized: " pkg=rgrpc service=reva traceid=b31f3067fb70e2b28cebc71d132e4ce9
2020-05-12T01:33:01+02:00 INF unary code=OK end="12/May/2020:01:33:01 +0200" from=tcp://<IP_ADDRESS>:51682 pkg=rgrpc service=reva start="12/May/2020:01:33:01 +0200" time_ns=8152063 traceid=b31f3067fb70e2b28cebc71d132e4ce9 uri=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate user-agent=grpc-go/1.26.0
2020-05-12T01:33:01+02:00 ERR error authenticating credentials to auth provider for type: bearer error="gateway: grpc failed with code CODE_UNAUTHENTICATED" pkg=rgrpc service=reva traceid=b31f3067fb70e2b28cebc71d132e4ce9
2020-05-12T01:33:01+02:00 WRN root/go/pkg/mod/github.com/cs3org/reva@v0.1.1-0.20200427161359-c1549a8110eb/internal/grpc/services/gateway/authprovider.go:66 >  error="gateway: grpc failed with code CODE_UNAUTHENTICATED" pkg=rgrpc service=reva traceid=b31f3067fb70e2b28cebc71d132e4ce9
2020-05-12T01:33:01+02:00 INF unary code=OK end="12/May/2020:01:33:01 +0200" from=tcp://<IP_ADDRESS>:60194 pkg=rgrpc service=reva start="12/May/2020:01:33:01 +0200" time_ns=10067909 traceid=b31f3067fb70e2b28cebc71d132e4ce9 uri=/cs3.gateway.v1beta1.GatewayAPI/Authenticate user-agent=grpc-go/1.26.0
2020-05-12T01:33:01+02:00 ERR error generating access token from credentials error="auth: grpc failed with code CODE_UNAUTHENTICATED" pkg=rhttp service=reva traceid=b31f3067fb70e2b28cebc71d132e4ce9

User Einstein can log in on the web UI and create folders. These folders are visible at the 'choose what to sync' dialog.

Same error, when using an ubuntu 18.04 host.
Same error, when using the remote ipaddr with port 9200, instead of the mapped port to localhost.

re-tested with ocis-1.0.0-rc1 via https://gitea.owncloud.services/jw/hetzner/src/branch/master/make_ocis_eos_test.sh and owncloud-client version 2.7.0 (build 1992)
Connection works just fine. OK.


Paybill and Buygoods validation & confirmation:

What is the ETA for this. I need to understand how this is supposed to work for example who is the broker? Am I supposed to develop a server that is complant with the wsdl provided or how is it supposed to work the documentation is crap.

No timeline :-1: Safaricom is not providing proper documentation of their API; If you want to work directly against their API, you should contact them :wink:


"Ghercești" is listed in the drop-down menu

Description:
"Ghercești" is listed in the drop-down menu alongside its corresponding county and commune, according to the law.
Precondition:
The website is up an running.

Step 1
Type "Ghercești" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Ghercești, DOLJ".


Numerical accuracy test in 03-matrix-multiplication.py is fauing; atol and rtol values

Describe the bug
Hi all,
I am investigating the failure of the numerical accuracy test failure of 03-matrix-multiplication.py on the AMD MI300 GPUs. This example uses float16 and compares numerical results obtained with Torch and Triton.
https://github.com/triton-lang/triton/blob/f062089a11bc87deef2160e461e9a3820b2ad3a4/python/tutorials/03-matrix-multiplication.py#L367-L371
As one can see, we increase rtol for MI200 GPUs because we know that the MFMA units flush denorms to zero in the case of float16. As far as I know, this issue has been resolved on MI300.
The specific atol and rtol values (1e-2 and 0.0, respectively) were introduced in this PR by @kernhanda. However, I couldn't find any reasoning why those values were chosen. It seems to me that those values are too high for float16. Below is a script which compares numerical results obtained by Torch and Numpy. It fails on both MI300 and H100 machines with default command line options (which replicates the ones in the example) - i.e., while comparing Torch results with the Numpy ones obtained with float16 and float64.
import torch
import numpy as np
import argparse
import sys

parser = argparse.ArgumentParser()
parser.add_argument("-t", "--type", choices=['fp16', 'fp32'],
                    default="fp16",
                    help="data type under the test")
parser.add_argument("--atol", type=float, default=1e-2, help="absolute tolerance")
parser.add_argument("--rtol", type=float, default=0.0, help="relative tolerance")
parser.add_argument("-v", "--verbose", action='store_true', help='verbose output')
args = parser.parse_args()

if args.type == 'fp64':
    print('using float64...')
    torch_fp = torch.float64
if args.type == 'fp32':
    print('using float32...')
    torch_fp = torch.float32
elif args.type == 'fp16':
    print('using float16...')
    torch_fp = torch.float16

print(f'using {args.atol=}; {args.rtol=}')

torch.manual_seed(0)
M = N = 512
K = 512
print(f'{M=}; {N=}; {K=}')
a = torch.randn((M, K), device='cuda', dtype=torch_fp)
b = torch.randn((K, N), device='cuda', dtype=torch_fp)

torch_output = torch.matmul(a, b)

torch_output_host = torch_output.cpu()
a_host = a.cpu()
b_host = b.cpu()

numpy_output = np.matmul(a_host.numpy(), b_host.numpy())

# See: https://numpy.org/doc/stable/reference/generated/numpy.allclose.html
def difference(a_vec, b_vec, atol, rtol):
    assert a_vec.shape == b_vec.shape
    m, n = a_vec.shape
    err_counter = 0
    max_diff = -sys.float_info.min
    min_diff = sys.float_info.max
    for i in range(m):
        for j in range(n):
            a = a_vec[i,j].item()
            b = b_vec[i,j].item()
            limit = atol + rtol * abs(b)
            delta = abs(a - b)
            max_diff = delta if delta > max_diff else max_diff
            min_diff = delta if min_diff > delta else min_diff
            if delta > limit:
                if args.verbose:
                  print(f'[{i}, {j}]: {a} | {b} | {delta}')
                err_counter += 1
    print(f'total errors found: {err_counter}')
    return max_diff, min_diff

print('-' * 80)
print(f'Comparing with: {torch_fp}')
if torch.allclose(torch_output_host, torch.from_numpy(numpy_output), atol=args.atol, rtol=args.rtol):
    print("\U00002705: Torch and Numpy match")
else:
    print("\U0000274C: Torch and Numpy differ")

max_diff, min_diff = difference(torch_output_host, numpy_output, atol=args.atol, rtol=args.rtol)
print(f'{max_diff=:10.8f}; {min_diff=:10.8f}')

print('-' * 80)
print(f'Comparing with: {torch.float64}')
a_host = a_host.double()
b_host = b_host.double()

numpy_output = np.matmul(a_host.numpy(), b_host.numpy())
max_diff, min_diff = difference(torch_output_host, torch.from_numpy(numpy_output), atol=args.atol, rtol=args.rtol)
print(f'{max_diff=:10.8f}; {min_diff=:10.8f}')

I suggest we need to increase tolerance values (i.e., the current values are to low for float16 for a given GEMM configration (M=512; N=512; K=512)).
I need to know whether OpenAI developers are going to be ok with this change.
Environment details
Triton: doesn't matter b/c the reproducible example uses Torch and Numpy
Docker:

ROCm: rocm/pytorch:rocm6.0.2_ubuntu22.04_py3.10_pytorch_2.1.2
CUDA: pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel

GPU:

MI300X
H100-PCIe

Hi all, I observed 2 things while working with Nvidia H100 GPUs around the following lines:
https://github.com/triton-lang/triton/blob/cc89dac07b7acf3af9962d83250a8bc015fc5a91/python/tutorials/03-matrix-multiplication.py#L358-L371

I can set any small value of atol (e.g., 1e-16, 1e-32 or 1e-64) and the accuracy test passes
The accuracy test fails as soon as I am using the CPU version of torch.matmul. This can be reproduced by replacing the following line

https://github.com/triton-lang/triton/blob/cc89dac07b7acf3af9962d83250a8bc015fc5a91/python/tutorials/03-matrix-multiplication.py#L361
with
torch_output = torch.matmul(a.cpu(), b.cpu())
torch_output = torch_output.to(device='cuda')

Hi all, I observed 2 things while working with Nvidia H100 GPUs around the following lines:
https://github.com/triton-lang/triton/blob/cc89dac07b7acf3af9962d83250a8bc015fc5a91/python/tutorials/03-matrix-multiplication.py#L358-L371

I can set any small value of atol (e.g., 1e-16, 1e-32 or 1e-64) and the accuracy test passes
The accuracy test fails when I am switching to the CPU version of torch.matmul. This can be reproduced by replacing the following line

https://github.com/triton-lang/triton/blob/cc89dac07b7acf3af9962d83250a8bc015fc5a91/python/tutorials/03-matrix-multiplication.py#L361
with
torch_output = torch.matmul(a.cpu(), b.cpu())
torch_output = torch_output.to(device='cuda')

@antiagainst @zhanglx13


Case edit form for logged in editor users

Create form where logged in editor can add new case pages and edit their own case pages.

Implemented in PR #165.


QUIC support

It would be generally interesting to see QUIC support, which would allow to compare TCP and QUIC implementations performance.

Sure, I'd love to add QUIC support. Do you have any context/suggestions that might be helpful?

@xnuter On my side, I'm interested in this tool firstly as a tool to make experiments and compare TCP vs QUIC for educational purposes and to optimize/benchmark our client/server application. I'm currently using quinn crate as QUIC implementation. Performance is not always obvious to me.
PS funny fact, we were both graduated from NSU

I wouldn't expect QUIC to be faster than TCP for a single data stream or relatively small payloads. If you run tests on closely located boxes, then unlikely there would be a considerable difference.


when train at last 15 epoch，meet a problem

hithere， very appreciate  for your code, but when i try to ytain on my coco's format dataset,i meet a problem, it is when the training go to the 285 epoch, the aug is closed, and a error occurs, as fllows:
2021-08-28 19:32:16 | INFO     | yolox.core.trainer:191 - --->No mosaic aug now!
2021-08-28 19:32:16 | INFO     | yolox.core.trainer:193 - --->Add additional L1 loss now!
2021-08-28 19:32:16 | INFO     | yolox.core.trainer:318 - Save weights to ./YOLOX_outputs/CoCo
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
2021-08-28 19:32:19 | INFO     | yolox.core.trainer:184 - Training of experiment is done and the best AP is 0.00
Process Process-2:2:
2021-08-28 19:32:19 | ERROR    | yolox.core.launch:147 - An error has been caught in function '_distributed_worker', process 'SpawnProcess-1' (2676), thread 'MainThread'<PHONE_NUMBER>03424):
Traceback (most recent call last):
File "", line 1, in
File "/home/ljh/.conda/envs/ylx/lib/python3.6/multiprocessing/spawn.py", line 105, in spawn_main
exitcode = _main(fd)
│     └ 9
└ <function _main at 0x7f7c9f7f02f0>
File "/home/ljh/.conda/envs/ylx/lib/python3.6/multiprocessing/spawn.py", line 118, in _main
return self._bootstrap()
│    └ <function BaseProcess._bootstrap at 0x7f7c9f9177b8>
└ <SpawnProcess(SpawnProcess-1, started)>
File "/home/ljh/.conda/envs/ylx/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
│    └ <function BaseProcess.run at 0x7f7c9f907f28>
└ <SpawnProcess(SpawnProcess-1, started)>
File "/home/ljh/.conda/envs/ylx/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
│    │        │    │        │    └ {}
│    │        │    │        └ <SpawnProcess(SpawnProcess-1, started)>
│    │        │    └ (<function _distributed_worker at 0x7f7c390a2d08>, 0, (<function main at 0x7f7c2d6ace18>, 2, 2, 0, 'nccl', 'tcp://<IP_ADDRESS>:3...
│    │        └ <SpawnProcess(SpawnProcess-1, started)>
│    └ <function _wrap at 0x7f7c3cb44840>
└ <SpawnProcess(SpawnProcess-1, started)>
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
│  │   └ (<function main at 0x7f7c2d6ace18>, 2, 2, 0, 'nccl', 'tcp://<IP_ADDRESS>:36773', (╒══════════════════╤══════════════════════════...
│  └ 0
└ <function _distributed_worker at 0x7f7c390a2d08>

File "/home/ljh/Desktop/YOLOX/yolox/core/launch.py", line 147, in _distributed_worker
main_func(*args)
│          └ (╒══════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════...
└ <function main at 0x7f7c2d6ace18>

File "/home/ljh/Desktop/YOLOX/tools/train.py", line 110, in main
trainer.train()
│       └ <function Trainer.train at 0x7f7c2f04b6a8>
└ <yolox.core.trainer.Trainer object at 0x7f7c2d6c4d30>
File "/home/ljh/Desktop/YOLOX/yolox/core/trainer.py", line 72, in train
self.train_in_epoch()
│    └ <function Trainer.train_in_epoch at 0x7f7c2e0b6950>
└ <yolox.core.trainer.Trainer object at 0x7f7c2d6c4d30>
File "/home/ljh/Desktop/YOLOX/yolox/core/trainer.py", line 81, in train_in_epoch
self.train_in_iter()
│    └ <function Trainer.train_in_iter at 0x7f7c2e0d2268>
└ <yolox.core.trainer.Trainer object at 0x7f7c2d6c4d30>
File "/home/ljh/Desktop/YOLOX/yolox/core/trainer.py", line 87, in train_in_iter
self.train_one_iter()
│    └ <function Trainer.train_one_iter at 0x7f7c2e0d22f0>
└ <yolox.core.trainer.Trainer object at 0x7f7c2d6c4d30>
File "/home/ljh/Desktop/YOLOX/yolox/core/trainer.py", line 93, in train_one_iter
inps, targets = self.prefetcher.next()
│    │          └ <function DataPrefetcher.next at 0x7f7c2f0b1bf8>
│    └ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7f7c3c811320>
└ <yolox.core.trainer.Trainer object at 0x7f7c2d6c4d30>
File "/home/ljh/Desktop/YOLOX/yolox/data/data_prefetcher.py", line 44, in next
self.preload()
│    └ <function DataPrefetcher.preload at 0x7f7c2f0b1b70>
└ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7f7c3c811320>
File "/home/ljh/Desktop/YOLOX/yolox/data/data_prefetcher.py", line 25, in preload
self.next_input, self.next_target, _, _ = next(self.loader)
│    │           │    │                        │    └ <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7f7c204cda20>
│    │           │    │                        └ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7f7c3c811320>
│    │           │    └ tensor([[[0., 0., 0., 0., 0.],
│    │           │               [0., 0., 0., 0., 0.],
│    │           │               [0., 0., 0., 0., 0.],
│    │           │               [0., 0., 0., 0., 0.],
│    │           │       ...
│    │           └ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7f7c3c811320>
│    └ tensor([[[[114., 114., 114.,  ..., 114., 114., 114.],
│                [114., 114., 114.,  ..., 114., 114., 114.],
│                [114., ...
└ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7f7c3c811320>
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
│    └ <function _MultiProcessingDataLoaderIter._next_data at 0x7f7c3c7b2a60>
└ <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7f7c204cda20>
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
return self._process_data(data)
│    │             └ <torch._utils.ExceptionWrapper object at 0x7f7bdf004ba8>
│    └ <function _MultiProcessingDataLoaderIter._process_data at 0x7f7c3c7b2b70>
└ <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7f7c204cda20>
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
│    └ <function ExceptionWrapper.reraise at 0x7f7c9ec342f0>
└ <torch._utils.ExceptionWrapper object at 0x7f7bdf004ba8>
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
│    │        └ 'Caught TypeError in DataLoader worker process 1.\nOriginal Traceback (most recent call last):\n  File "/home/ljh/.conda/envs...
│    └ <class 'TypeError'>
└ <torch._utils.ExceptionWrapper object at 0x7f7bdf004ba8>
TypeError: Caught TypeError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 83, in default_collate
return [default_collate(samples) for samples in transposed]
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 83, in
return [default_collate(samples) for samples in transposed]
File "/home/ljh/.conda/envs/ylx/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 61, in default_collate
raise TypeError(default_collate_err_msg_format.format(elem.dtype))
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <U7

can you help me to fix the problem, appreciate!

Maybe this PR #603 already fix your problem? Could you try our latest branch?

Maybe this PR #603 already fix your problem? Could you try our latest branch?

well， actuall  i seems have not changed the mosaic_prob value, and i have try the new verision of ylx, but it seems didnt work, i will try  it again.

2021-08-29 14:29:47 | INFO     | yolox.core.trainer:318 - Save weights to ./YOLOX_outputs/yolox_s
2021-08-29 14:29:48 | INFO     | yolox.core.trainer:188 - ---> start train epoch8
2021-08-29 14:29:48 | INFO     | yolox.core.trainer:191 - --->No mosaic aug now!
2021-08-29 14:29:48 | INFO     | yolox.core.trainer:193 - --->Add additional L1 loss now!
2021-08-29 14:29:48 | INFO     | yolox.core.trainer:318 - Save weights to ./YOLOX_outputs/yolox_s
2021-08-29 14:29:50 | INFO     | yolox.core.trainer:184 - Training of experiment is done and the best AP is 13.58
2021-08-29 14:29:50 | ERROR    | yolox.core.launch:147 - An error has been caught in function '_distributed_worker', process 'SpawnProcess-1' (24369), thread 'MainThread'<PHONE_NUMBER>86624):
Traceback (most recent call last):
File "", line 1, in
File "/home/ljh/.conda/envs/lix/lib/python3.6/multiprocessing/spawn.py", line 105, in spawn_main
exitcode = _main(fd)
│     └ 9
└ <function _main at 0x7fe50475a2f0>
File "/home/ljh/.conda/envs/lix/lib/python3.6/multiprocessing/spawn.py", line 118, in _main
return self._bootstrap()
│    └ <function BaseProcess._bootstrap at 0x7fe5048817b8>
└ <SpawnProcess(SpawnProcess-1, started)>
File "/home/ljh/.conda/envs/lix/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
│    └ <function BaseProcess.run at 0x7fe504871f28>
└ <SpawnProcess(SpawnProcess-1, started)>
File "/home/ljh/.conda/envs/lix/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
│    │        │    │        │    └ {}
│    │        │    │        └ <SpawnProcess(SpawnProcess-1, started)>
│    │        │    └ (<function _distributed_worker at 0x7fe49e00cbf8>, 0, (<function main at 0x7fe49260f950>, 2, 2, 0, 'nccl', 'tcp://<IP_ADDRESS>:3...
│    │        └ <SpawnProcess(SpawnProcess-1, started)>
│    └ <function _wrap at 0x7fe4a1aae730>
└ <SpawnProcess(SpawnProcess-1, started)>
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
│  │   └ (<function main at 0x7fe49260f950>, 2, 2, 0, 'nccl', 'tcp://<IP_ADDRESS>:33067', (╒══════════════════╤══════════════════════════...
│  └ 0
└ <function _distributed_worker at 0x7fe49e00cbf8>

File "/home/ljh/YOLOX/yolox/core/launch.py", line 147, in _distributed_worker
main_func(*args)
│          └ (╒══════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════...
└ <function main at 0x7fe49260f950>

File "/home/ljh/YOLOX/tools/train.py", line 110, in main
trainer.train()
│       └ <function Trainer.train at 0x7fe493fb3510>
└ <yolox.core.trainer.Trainer object at 0x7fe49261bbe0>
File "/home/ljh/YOLOX/yolox/core/trainer.py", line 72, in train
self.train_in_epoch()
│    └ <function Trainer.train_in_epoch at 0x7fe493021620>
└ <yolox.core.trainer.Trainer object at 0x7fe49261bbe0>
File "/home/ljh/YOLOX/yolox/core/trainer.py", line 81, in train_in_epoch
self.train_in_iter()
│    └ <function Trainer.train_in_iter at 0x7fe493038d08>
└ <yolox.core.trainer.Trainer object at 0x7fe49261bbe0>
File "/home/ljh/YOLOX/yolox/core/trainer.py", line 87, in train_in_iter
self.train_one_iter()
│    └ <function Trainer.train_one_iter at 0x7fe493038d90>
└ <yolox.core.trainer.Trainer object at 0x7fe49261bbe0>
File "/home/ljh/YOLOX/yolox/core/trainer.py", line 93, in train_one_iter
inps, targets = self.prefetcher.next()
│    │          └ <function DataPrefetcher.next at 0x7fe49401ba60>
│    └ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7fe446cf7ef0>
└ <yolox.core.trainer.Trainer object at 0x7fe49261bbe0>
File "/home/ljh/YOLOX/yolox/data/data_prefetcher.py", line 43, in next
self.preload()
│    └ <function DataPrefetcher.preload at 0x7fe49401b9d8>
└ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7fe446cf7ef0>
File "/home/ljh/YOLOX/yolox/data/data_prefetcher.py", line 25, in preload
self.next_input, self.next_target, _, _ = next(self.loader)
│    │           │    │                        │    └ <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7fe446cf7f28>
│    │           │    │                        └ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7fe446cf7ef0>
│    │           │    └ tensor([[[  3.0000, 218.2926, 379.3279,  74.1916,  58.1691],
│    │           │               [  3.0000, 449.9980, 291.3791,  92.9210,  69.9519],
│    │           │         ...
│    │           └ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7fe446cf7ef0>
│    └ tensor([[[[ 50.,  49.,  47.,  ..., 134., 132., 129.],
│                [ 53.,  48.,  46.,  ..., 130., 129., 130.],
│                [ 47., ...
└ <yolox.data.data_prefetcher.DataPrefetcher object at 0x7fe446cf7ef0>
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
│    └ <function _MultiProcessingDataLoaderIter._next_data at 0x7fe4a171b950>
└ <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7fe446cf7f28>
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
return self._process_data(data)
│    │             └ <torch._utils.ExceptionWrapper object at 0x7fe493593eb8>
│    └ <function _MultiProcessingDataLoaderIter._process_data at 0x7fe4a171ba60>
└ <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7fe446cf7f28>
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
│    └ <function ExceptionWrapper.reraise at 0x7fe503b9f2f0>
└ <torch._utils.ExceptionWrapper object at 0x7fe493593eb8>
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
│    │        └ 'Caught TypeError in DataLoader worker process 0.\nOriginal Traceback (most recent call last):\n  File "/home/ljh/.conda/envs...
│    └ <class 'TypeError'>
└ <torch._utils.ExceptionWrapper object at 0x7fe493593eb8>
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 83, in default_collate
return [default_collate(samples) for samples in transposed]
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 83, in
return [default_collate(samples) for samples in transposed]
File "/home/ljh/.conda/envs/lix/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 61, in default_collate
raise TypeError(default_collate_err_msg_format.format(elem.dtype))
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <U7
still have this problem, after close aug, this problem occurs.
because i used my custom coco dataset, the image_id is like 'u00001' type, use the original pycocotools cannot load image, so i change as follows:
def loadImgs(self, ids=[]):
"""
Load anns with the specified ids.
:param ids (int array)       : integer ids specifying img
:return: imgs (object array) : loaded img objects
"""
# return self.loadImgs(ids)
# print("==========",type(ids))
if _isArrayLike(ids):
# return [self.imgs[id] for id in ids]
return [self.imgs[ids]]
elif type(ids) == int:
return [self.imgs[ids]]
and here:
def load_anno_from_ids(self, id_):
im_ann = self.coco.loadImgs(id_)[0]
width = im_ann["width"]
height = im_ann["height"]
#anno_ids = self.coco.getAnnIds(imgIds=[int(id_)], iscrowd=False)
anno_ids = self.coco.getAnnIds(imgIds=[(id_)], iscrowd=False)
annotations = self.coco.loadAnns(anno_ids)
i will try to use the coco2017 to test whether is these place lead to error.

fix it.

i am getting similar error but before the start of training
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <U38
I am also using custom coco format dataset. Images and annotations seem to be loaded properly in coco.py .
Not sure where to look further!

i am getting similar error but before the start of training TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <U38 I am also using custom coco format dataset. Images and annotations seem to be loaded properly in coco.py .
Not sure where to look further!

would u mind show more details about that?

I created a new issue  here


Class 'App\Soap\Response\GetConversionAmountResponse' not found

When I try to reproduce your example, I get this message:
Class 'App\Soap\Response\GetConversionAmountResponse' not found
for this line:
use App\Soap\Response\GetConversionAmountResponse;
I could not find a Response file in this library. How could I solve this issue?
Thanks,
gezmi

Sorry, this was solved. However, I cannot get result apart from 0.
class SoapController extends Controller {
  /**
           * @var SoapWrapper
           */
          protected $soapWrapper;

          /**
           * SoapController constructor.
           *
           * @param SoapWrapper $soapWrapper
           */
          public function __construct(SoapWrapper $soapWrapper)
          {
            $this->soapWrapper = $soapWrapper;
          }

          /**
           * Use the SoapWrapper
           */
          public function show()
          {
            $this->soapWrapper->add('Currency', function ($service) {
              $service
                ->wsdl('http://currencyconverter.kowabunga.net/converter.asmx?WSDL')
                ->trace(true);
            });

            // Without classmap
            $response = $this->soapWrapper->call('Currency.GetConversionAmount', [
              'Anything' => 'something',
              'CurrencyFrom' => 'USD',
              'CurrencyTo'   => 'EUR',
              'RateDate'     => '2014-06-05',
              'Amount'       => '1000',
            ]);
            var_dump($this);
            dd($response->GetConversionAmountResult);

          }
}

The Anything line was added because it left out the first param (USD) otherwise.
Gives me this XML request:
"<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://tempuri.org/">
<SOAP-ENV:Body>
<ns1:GetConversionAmount/>
<param1>USD</param1> //this was missing without 'Anything'
<param2>EUR</param2>
<param3>2014-06-05</param3>
<param4>1000</param4>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
"

And the response:
"<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<GetConversionAmountResponse xmlns="http://tempuri.org/">
<GetConversionAmountResult>0</GetConversionAmountResult>
</GetConversionAmountResponse>
</soap:Body>
</soap:Envelope>"

What might be the problem?


Allow configuring idle_timeout Envoy parameter on HTTP Connection Manager

Please describe your use case / problem.
We currently have Ambassador deployed on-premise behind a firewall and fronted by a CDN. We encountered an issue where on low-traffic services, our clients were receiving some 503 first byte timeouts.
The issue is present because when the firewall chose to remove idle tcp sessions from its session table, it would not send a FIN or RST to the CDN. On the next request, the CDN thinks that it still has a tcp connection open and attempts to re-use the now-closed TCP connection, resulting in the request getting 'black-holed', and leads to an HTTP 503.
Because Envoy doesn't ever attempt to close idle tcp connections by default, we're left with Envoy thinking that sessions are open, but our firewall has closed them after a 30 minute timeout (without notifying the CDN that the connection is closed), and the CDN still thinks that the connection is open.
Our "options" for remediating this scenario were:

Configure tcp idle timeout on the CDN side to control connection idle time

This turned out to not be an option that we can readily control on our CDN. They have 'dynamic' connection times.

Make the firewall send a FIN or RST to the CDN when it removes a session from its session table

We investigated this as well, but on our firewall this is not an option that is configurable.

Have Envoy close idle tcp connections gracefully after a configurable timeout (in our case, some timer that is lower than our firewall idle session timeout)

We believe that this is the most appropriate way to solve the issue, and gives us control when tuning idle timeouts across each network device in the path, so this issue requests making the idle_timeout Envoy setting configurable on the Envoy http listener.

Describe the solution you'd like
Would like to be able to have the idle_timeout on the downstream HTTP Listener configurable for the reason mentioned above.
Describe alternatives you've considered
See description above.
Additional context
Looks like #1738 originally brought this up, but was closed as stale
I originally opened #2126 for this, but because of Github shenanigans the PR got closed when I was updating my fork. Created this issue for tracking, then will link PR's to this when I have one ready.

Closing - implemented in #2164


Envoy does not use all available cores with gRPC transcoder and produces low throughput

Title: Envoy does not use all available cores with gRPC transcoder and produces low throughput
Description:
I am running envoy with configuration that has a gRPC transcoder filter on a Kubernetes cluster with nodes that has 48 CPUs.
The envoy process is connected to a gRPC server in the upstream and I am using iperf to measure network performance in matters of throughput.
Two observations from my tests:

The throughput is very low compared to the saturated network throughput (compared to iperf test that bypasses the envoy process)
Probing the node CPU consumption, it seems that the envoy process is using only 2.5 CPU out of all the available
This is not a kubernetes configuration issue, as I tested the cluster with "vish/stress" image, and it reach full CPU utilization

Things I have tried doing the following in order to improve performance:

Configurting buffer and windows sizes per this example: https://www.envoyproxy.io/docs/envoy/latest/configuration/best_practices/edge
Using "ExactBalance", per this documentation: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/intro/threading_model#arch-overview-threading

The debug log is full of "onBelowWriteBufferLowWatermark" and "onAboveWriteBufferHighWatermark" messages, consistently enabling and disabling streams (hunders of times a second), Added an example in the logs below.
Will appreciate your help understanding why this is happening and how can I increase the performance.
 If so, Envoy will use the CPU resource of the Pod without --concurrency.
If you run Envoy as a single process, try with --concurrency.
Also, could you provide pprof data refer to https://github.com/envoyproxy/envoy/blob/main/bazel/PPROF.md, that will help us analyze.

It seems you run Envoy in a Pod without CPU resource limitation, right? so it will use full CPU resources by default. I also don't think it's a Kubernetes issue, there are many recv buffer over limit and send buffer over limit in the log, so the buffer size might be the bottleneck. Does it work if you increase the initial_stream_window_size and initial_connection_window_size?

@daixiang0 I am unable to use this tool because of internal compliance. I am running Envoy in a pod.
@gyohuangxin I did try to increase the buffers, but it didn't help.
Updating on my progress investigating this, as I think I found what caused this:
At first, I created a single HTTP/2 client to the envoy process and used multiplexing to multiplex multiple iperf client connections. I changed the implementation to create a new HTTP client per stream and when I did this, the throughput improved and also CPU consumption increased. What I think that happened is that at first, there was only a single HTTP connection with envoy, so there was a single gRPC transcoder filter that basically was the process bottleneck. After changing the implementation, a new filter was created per stream - enabling utilizing more of the CPU.
Even though this makes sense - what I expected to happen is that the gRPC filter will adjust to be robust against multiple streams that are multiplexed within a single HTTP/2 connection, otherwise (and this goes to all kind of filters), how do we leverage the multiplexing feature of HTTP/2 connection if in any way there will be a single filter for the entire connection?
Is there anything that can be configured that I have missed?

I met the problem of low throughput refer to the following Settings,but I didn't find the message about "onBelowWriteBufferLowWatermark" and "onAboveWriteBufferHighWatermark".

https://www.envoyproxy.io/docs/envoy/latest/configuration/best_practices/edge

I obtained the normal throughput by disabling the config delayed_close_timeout of type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager .
ref from file @ api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto

// The delayed close timeout is for downstream connections managed by the HTTP connection manager.
// It is defined as a grace period after connection close processing has been locally initiated
// during which Envoy will wait for the peer to close (i.e., a TCP FIN/RST is received by Envoy
// from the downstream connection) prior to Envoy closing the socket associated with that
// connection.
// NOTE: This timeout is enforced even when the socket associated with the downstream connection
// is pending a flush of the write buffer. However, any progress made writing data to the socket
// will restart the timer associated with this timeout. This means that the total grace period for
// a socket in this state will be
// <total_time_waiting_for_write_buffer_flushes>+<delayed_close_timeout>.
//
// Delaying Envoy's connection close and giving the peer the opportunity to initiate the close
// sequence mitigates a race condition that exists when downstream clients do not drain/process
// data in a connection's receive buffer after a remote close has been detected via a socket
// write(). This race leads to such clients failing to process the response code sent by Envoy,
// which could result in erroneous downstream processing.
//
// If the timeout triggers, Envoy will close the connection's socket.
//
// The default timeout is 1000 ms if this option is not specified.
//
// .. NOTE::
//    To be useful in avoiding the race condition described above, this timeout must be set
//    to at least +<100ms to account for
//    a reasonable "worst" case processing time for a full iteration of Envoy's event loop>.
//
// .. WARNING::
//    A value of 0 will completely disable delayed close processing. When disabled, the downstream
//    connection's socket will be closed immediately after the write flush is completed or will
//    never close if the write flush does not complete.
google.protobuf.Duration delayed_close_timeout = 26;

@sclei what scenario were you testing when you managed to obtain max throughput? I think that thr config you refered me to is more relevant to a case where a lot of tcp connections are starting and ending. In my case I have a small set of connections and each connection streams a long serie of json messaged to envoy.

Updating on my progress investigating this, as I think I found what caused this:
At first, I created a single HTTP/2 client to the envoy process and used multiplexing to multiplex multiple iperf client connections. I changed the implementation to create a new HTTP client per stream and when I did this, the throughput improved and also CPU consumption increased. What I think that happened is that at first, there was only a single HTTP connection with envoy, so there was a single gRPC transcoder filter that basically was the process bottleneck. After changing the implementation, a new filter was created per stream - enabling utilizing more of the CPU.

OK, I can understand it. The single HTTP/2 connection is the bottleneck.

Even though this makes sense - what I expected to happen is that the gRPC filter will adjust to be robust against multiple streams that are multiplexed within a single HTTP/2 connection, otherwise (and this goes to all kind of filters), how do we leverage the multiplexing feature of HTTP/2 connection if in any way there will be a single filter for the entire connection?

However, I cannot understand this very well. Can you give some details/examples of "being robust against multiple streams that are multiplexed within a single HTTP/2 connection"? IMO, from your scenario, I cannot see anything unreasonable about multiple stream in a single HTTP/2 connection.

@gyohuangxin Having multiple streams multiplexed in a single HTTP/2 connection is resonable. However with this architecture envoy will create only a single thread with a single gRPC transcoder which is what created the bottleneck. If envoy had created a filter per stream, and they would have computed in parallel, then this bottleneck wouldn't have been created since there were filter per stream, and it would still be possible to use HTTP/2 multiplexing. After figuring this, I am forced to change my architecture to having multiple HTTP connections, one per stream, instead of having a single connection that multiplexes all streams. Have I misunderstood something?

@ohadvano Actually we encountered the same issue when benchmarking GRPC, solved by adding more cores to envoy using --concurrency which mentioned by @daixiang0. We are using istio to config this value but I think it should be doable without istio.


Missing Map tests

Map:

[x] setZoom
[x] setZoomAround
[x] fitWorld
[x] #8365
[x] #8366
[ ] #8367
[x] setMinZoom
[x] setMaxZoom
[x] #8368
[x] #8369
[x] #8370
[x] #8371
[x] #8372
[x] createPane
[x] getPane
[x] getPanes
[x] getContainer
[x] getSize
[x] getPixelBounds
[x] getPixelOrigin
[x] getPixelWorldBounds
[ ] #8373
[x] #8374
[x] #8375
[x] #8376
[x] distance
[x] containerPointToLayerPoint
[x] layerPointToContainerPoint
[x] #8377
[x] #8378
[x] #8379
[ ] #8380
[x] #8381

Can you assign this to me @Falke-Design?

@stephenspol Done! 👍🏼

Hello, can you assign me a task. doesn't matter which one.

@NeverGiveUp23 maybe the easiest ones are:

containerPointToLatLng
latLngToContainerPoint
mouseEventToContainerPoint
mouseEventToLayerPoint
mouseEventToLatLng

It seems no one is assigned to this issue. Can I pick up this issue?

@abhi3315 Absolutely!

I've created some more granular issues so that we can track the changes better, feel free to sign up for any of them.

Closing this per the above comment / in favor of the more granular issues.

Let's keep this open as not all tasks inside of this tracking issue have been resolved.

Ok.

Hello! I am a first time contributor, are there any tasks I can be assigned and do you have any insight on how to best tackle them? Thank you!

@adeeb18 I'd recommend to take a look at the PRs for the issues that have been closed. They will give you a good impression on how to get started.

Oh, and if you are looking to sign up for any specific ones, let me know and I will assign you to them.

I would like to take mouseEventToLayerPoint, please.

Hello Falke-Design,can you assign me some task as I am new in open source contribution and I want some help to contribute and be a part of this amazing OPEN SOURCE Community

@jonkoops @Falke-Design Hi,I would like to contribute by adding a Map Test for distance. Kindly assign me the issue

@shreya024 please just create a PR and we will then check it. I will not assign you this issue, to keep it open for others too.

@jonkoops I am a new Contributor can you assign some Good First Issue to me


Support a variable Python base image name

Currently the base image is customisable by setting METAFLOW_BATCH_CONTAINER_IMAGE or passing image - if not specified this falls back to the users Python versions, e.g. python:3.9.2
This image name doesn't work for strategies using non-Dockerhub docker registries, where the Python image name may vary, e.g. when using AWS ECR pull through cache, the ecr-public Python image name is <account_id>.dkr.ecr.<aws_region>.amazonaws.com/public-ecr/docker/library/python:<version>. Using a registry such as this is useful to avoid being rate limited by Dockerhub without auth.
This requires the prefix /public-ecr/docker/library on the image to allow compatibility with non-pull-through-cache images in the same ECR registry.
I'd suggest making the default python image name customisable with a new config/environment variable to support these configurations.
Current workarounds
It's currently possible to workaround this by setting METAFLOW_BATCH_CONTAINER_REGISTRY=<account_id>.dkr.ecr.<aws_region>.amazonaws.com/public-ecr/docker/library and then using the full ECR URL of any non-default images. This will result in python using the ECR pull-through cache, and the registry being derived from the image URL for non-default images.

@Limess The workaround you referenced is indeed the solution we were going to recommend. For more involved use cases, we would ideally want to provide users with more fine-grained controls to determine the exact image to use. It's likely that users may want to compute a different image version compared to the one being used today. Let me think about what this user-specified translation layer can look like in practice and get back to you.