Spec filter loses ordering of operations

Ordering using the position attribiute in @ApiOperation annotations is lost after applying spec filters

Is this also relevant to 1.5?

I don't know. I haven't worked with this code in almost two years. Sorry.

On Mar 9, 2015, at 2:38 PM, Ron<EMAIL_ADDRESS>wrote:
Is this also relevant to 1.5?
—
Reply to this email directly or view it on GitHub.

No problem, I was actually asking @fehguy  ;)

@webron i don't think so.  feel free to close it

Great. Closing out.

Ah that's weird. The email made it look like it was a message to me.

On Mar 9, 2015, at 2:50 PM, Ron<EMAIL_ADDRESS>wrote:
No problem, I was actually asking @fehguy ;)
—
Reply to this email directly or view it on GitHub.


Is this typos?

isset($uploadedFile['tmp_name']) ? $uploadedFile['name'] : null
should it be
isset($uploadedFile['name']) ? $uploadedFile['name'] : null

No, see: https://github.com/slimphp/Slim/issues/1673

Looks like a bug to me. I don't understand from the manual page why it isn't. Input from @silentworks required!

This is what the manual page states.

$_FILES['userfile']['name']: The original name of the file on the client machine.
$_FILES['userfile']['tmp_name']: The temporary filename of the file in which the uploaded file was stored on the server.

To me this says that the tmp_name is greater to check against as this is an actual place the files is store on the server and if it wasn't this would return none from what the docs says, while there is no mention on if name would even be a thing at this stage.
But saying that, after reading this comment http://php.net/manual/en/features.file-upload.post-method.php#104349 it might make more sense to check name.

The reason I thought this was a typo is ...
If someone uses array data that name key is missing (maybe mocking data), this will result in PHP notice.


Using Vladmandic in Windows 11. My LORA menu is empty, my models and upscalers do populate and I get the SHOW error

Images generate, but I get this error, forcing me to click OK three times for it to go away and then generate the image finally...this is only when I start generating a new image on a canvas. After this, it doesn't appear as I keep clicking 'generating more txt2img'

(I had to use the unzip method to install the plug-in because my Photoshop v24.3.0 would not recognize the CCX file as a valid file for whatever reason)
Any idea how to fix these? Thank you!

I have resolved the ‘Show’ error message and plan to release the fix in version 1.2.6. However, it’s important to note that the error is harmless.
I haven’t tested the plugin with Vlad. I’ve heard that they have a different directory structure, which could be why you’re not seeing the Loras. Alternatively, it could be because you didn’t install the Auto-Photoshop-SD extension, which is responsible for scanning for Loras. I added this feature before Auto1111 exposed Lora from their API, so you may need to wait for the next update.

Photoshop v24.3.0 would not recognize the CCX file as a valid file for whatever reason)

You can’t install .ccx files with just Photoshop. You need Creative Cloud for that.


Git Large File Storage Problem: pickle.UnpicklingError: invalid load key, 'v'.

Hi,
I got this error when running the program. I know we are supposed to install git large file storage but I had some permission issue with the server. If it is possible, can you upload model weights and files that we can download and run the code? Thanks so much!

I think it's because my git file storage has a bandwidth limit. @gyNancy Please refer to this #2


Author/Write pages from the deployed app itself

I like writing at medium.com because it's very easy to write and publish, but it has many disadvantages. PardallMarkdown looks very good and with the LiveView functionality I believe it'd be possible to have a very streamlined write and publish workflow.
It'd need some sort of authentication and authorization for a user to log-in and start writing and messing with the server's filesystem.
I'm willing to work on this in my free time, this tool really fits my expectations, I'd like to know your opinions about this feature.
Example use case

I'm taking classes at my Uni and can just open any computer in the lab, log-in to my page and take notes there.

Hey @gbrls, this is completely doable with PardallMarkdown, as a matter of fact, it's the perfect use-case. For example, right now, the sample project https://pardall.xyz/ via https://github.com/alfredbaudisch/pardall-markdown-phoenix-demo, gets articles published via Markdown files pushed from a watched Git repo.
You could start with that as a base, then you add the CRUD interface to create and edit the Markdown content within the Phoenix application itself. You could start with Phoenix's default LiveView or even HTML CRUDs and expand upon that (Phoenix even has an auth generator built-in).


Compile from source on Skylake-X (Intel i9)

Hello,
Compiling tensorflow source on Skylake-X Intel i9 with "--config=opt" gives the following error in the snappy external module:

ERROR: /home/armafire/.cache/bazel/bazel_armafire/efbef35334c587b69e16a82829bb0e2d/external/snappy/BUILD:19:1: C++ compilation of rule '@snappy//:snappy' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command
(cd /home/armafire/.cache/bazel/bazel_armafire/efbef35334c587b69e16a82829bb0e2d/execroot/org_tensorflow &&
exec env -
CUDA_TOOLKIT_PATH=/usr/local/cuda
CUDNN_INSTALL_PATH=/usr/local/cuda-8.0
GCC_HOST_COMPILER_PATH=/usr/bin/gcc
PWD=/proc/self/cwd
PYTHON_BIN_PATH=/usr/bin/python
PYTHON_LIB_PATH=/usr/local/lib/python2.7/dist-packages
TF_CUDA_CLANG=0
TF_CUDA_COMPUTE_CAPABILITIES=6.1
TF_CUDA_VERSION=8.0
TF_CUDNN_VERSION=6
TF_NEED_CUDA=1
TF_NEED_OPENCL=0
external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++11' '-march=native' -MD -MF bazel-out/local_linux-opt/bin/external/snappy/objs/snappy/external/snappy/snappy.pic.d '-frandom-seed=bazel-out/local_linux-opt/bin/external/snappy/objs/snappy/external/snappy/snappy.pic.o' -fPIC -iquote external/snappy -iquote bazel-out/local_linux-opt/genfiles/external/snappy -iquote external/bazel_tools -iquote bazel-out/local_linux-opt/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -Wno-shift-negative-value -Wno-implicit-function-declaration -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE="redacted"' '-D__TIMESTAMP="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c external/snappy/snappy.cc -o bazel-out/local_linux-opt/bin/external/snappy/objs/snappy/external/snappy/snappy.pic.o)
cc1plus: warning: command line option '-Wno-implicit-function-declaration' is valid for C/ObjC but not for C++
external/snappy/snappy.cc: In member function 'void snappy::SnappySinkAllocator::Flush(size_t)':
external/snappy/snappy.cc:1403:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < blocks.size(); ++i) {
^~~~~~~~~~~~~~
In file included from external/snappy/snappy-internal.h:34:0,
from external/snappy/snappy.cc:30:
external/snappy/snappy.cc: In instantiation of 'bool snappy::SnappyScatteredWriter::AppendFromSelf(size_t, size_t) [with Allocator = snappy::SnappySinkAllocator; size_t = long unsigned int]':
external/snappy/snappy.cc:715:13:   required from 'void snappy::SnappyDecompressor::DecompressAllTags(Writer*) [with Writer = snappy::SnappyScatteredWritersnappy::SnappySinkAllocator]'
external/snappy/snappy.cc:799:3:   required from 'bool snappy::InternalUncompressAllTags(snappy::SnappyDecompressor*, Writer*, snappy::uint32) [with Writer = snappy::SnappyScatteredWritersnappy::SnappySinkAllocator; snappy::uint32 = unsigned int]'
external/snappy/snappy.cc:1460:78:   required from here
external/snappy/snappy.cc:1316:34: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (PREDICT_TRUE(offset - 1u < op_ptr_ - op_base_ && op_end <= op_limit_)) {
^
external/snappy/snappy-stubs-internal.h:80:25: note: in definition of macro 'PREDICT_TRUE'
#define PREDICT_TRUE(x) x
^
/tmp/ccxBWytY.s: Assembler messages:
/tmp/ccxBWytY.s:389: Error: no such instruction: kmovq %rdx,%k3' /tmp/ccxBWytY.s:391: Error: no such instruction: kshiftrq $32,%k3,%k2'
/tmp/ccxBWytY.s:394: Error: no such instruction: kmovq %k2,%rdx' /tmp/ccxBWytY.s:600: Error: no such instruction: kmovq %rsi,%k1'
/tmp/ccxBWytY.s:602: Error: no such instruction: kshiftrq $32,%k1,%k0' /tmp/ccxBWytY.s:605: Error: no such instruction: kmovq %k0,%rsi'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 34.595s, Critical Path: 14.66s
FAILED: Build did NOT complete successfully

The command I use is:
bazel build --config=opt -c opt //tensorflow/tools/pip_package:build_pip_package --verbose_failures -j 64
If I change to --config=mkl, then it compiles fine. Therefore, it seems that the problem is "--march=native" (generated by "--config=opt"), which forces the snappy module to generate AVX512, however, the generated assembly is not correct for Skylake-X Intel i9 (which supports AVX512). I tried "--march=skylake-avx512" and all other AVX512 options like "-mavx512f" and similar, and all result in the same error.
My goal is to compile tensorflow with Eigen AVX512. Any ideas how this can be done?

Could you share information about your operating system, distro, gcc version, clang version, and all that jazz? I ask because it looks like there might be something wrong with the compiler.

Operating System:
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.5 LTS
Release:	14.04
Codename:	trusty
GCC:
gcc (Ubuntu 5.4.1-2ubuntu1~14.04) 5.4.1 20160904
CLANG:
Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4)
Target: x86_64-pc-linux-gnu
Thread model: posix
It looks like the instruction "kmovq" (the one that fails to be identified) belongs to "AVX512BW", a category of AVX instructions not supported on Skylake-X i9. These are supported on KNL (Xeon Phi Knight Landings), but not on Skylake-X.

@armafire thanks for the report. Are you sure you are using the proper binutils? It looks like a gas error.

What should I do to verify the gas?

Maybe now the sources are more stable.
I've manage compile on i9-7900x:
https://github.com/thupalo/TF1.4_GPU_CUDA9_AVX512F
Could you please check if this works for you?

Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?

I just tested this on tags v1.5.0, v1.6.0-rc0, v1.6.0-rc1 and master -- -mavx512 is still broken on all of them.

Just a little housekeeping. Duplicate of #10220


StreamBuilder not updating during testWidget

Steps to Reproduce

main.dart:
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:flutter/material.dart';
import 'package:firebase_core/firebase_core.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp();
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  MyApp({Key key}) : super(key: UniqueKey());

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      debugShowCheckedModeBanner: false,
      title: 'My App',
      home: Scaffold(
        appBar: AppBar(
          title: Text('Tasks'),
          actions: [
            PopupMenuButton<String>(
              onSelected: (String value) async {
                if (value == 'Add Tasks') {
                  print('adding tasks');
                  for (Task task in mockTaskList) {
                    await globalFirestoreInstance.collection('tasks').doc(task.id).set(task.toJson());
                  }
                  print('tasks added');
                } else {
                  print('deleting tasks');
                  await globalFirestoreInstance.collection('tasks').get().then((snapshot) async {
                    for (DocumentSnapshot documentSnapshot in snapshot.docs) {
                      await documentSnapshot.reference.delete();
                    }
                  });
                  print('tasks deleted');
                }
              },
              itemBuilder: (BuildContext context) {
                return ['Add Tasks', 'Delete All Tasks'].map(
                  (String choice) {
                    return PopupMenuItem<String>(
                      value: choice,
                      child: Text(choice),
                    );
                  },
                ).toList();
              },
            ),
          ],
        ),
        body: Container(
          child: Directionality(
            textDirection: TextDirection.ltr,
            child: StreamBuilder<QuerySnapshot>(
              stream: globalFirestoreInstance.collection('tasks').snapshots(),
              builder: (BuildContext context, AsyncSnapshot<QuerySnapshot> snapshot) => (!snapshot.hasData)
                  ? Center(
                      child: CircularProgressIndicator(),
                    )
                  : ListView(
                      children: snapshot.data.docs
                          .map(
                            (DocumentSnapshot doc) => Directionality(
                              textDirection: TextDirection.ltr,
                              child: Card(
                                child: ListTile(
                                  title: Text(doc['name']),
                                ),
                              ),
                            ),
                          )
                          .toList(),
                    ),
            ),
          ),
        ),
      ),
    );
  }
}

final List<Task> mockTaskList = [
  Task(
    id: '0',
    dateCreated: Timestamp(1, 2),
    name: 'task 0',
    isComplete: false,
  ),
  Task(
    id: '1',
    dateCreated: Timestamp(3, 4),
    name: 'task 1',
    isComplete: false,
  ),
];

class Task {
  final String id;
  final String name;
  final Timestamp dateCreated;
  final bool isComplete;

  Task({@required this.id, @required this.name, @required this.dateCreated, @required this.isComplete});

  Map<String, dynamic> toJson() {
    return <String, dynamic>{
      'id': id,
      'name': name,
      'dateCreated': dateCreated,
      'isComplete': isComplete,
    };
  }
}

FirebaseFirestore globalFirestoreInstance = FirebaseFirestore.instance;

pubspec.yaml:
name: my_app
description: Github testing
version: 0.2.1+1

environment:
  sdk: ">=2.7.0 <3.0.0"

dependencies:
  flutter:
    sdk: flutter
  cloud_firestore: "1.0.1"
  firebase_core: "1.0.1"
  modal_bottom_sheet: "2.0.0"

dev_dependencies:
  flutter_driver:
    sdk: flutter
  flutter_test:
    sdk: flutter
  cloud_firestore_mocks: "0.7.0"
  pedantic: "1.11.0"
  screenshots: "0.2.1"
  test: "1.16.5"

flutter:
  uses-material-design: true

main_test.dart:
import 'package:cloud_firestore_mocks/cloud_firestore_mocks.dart';
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:lachies_life_planner/main.dart';

void main() {
  const String emptyFirestoreDump = '{\n'
      '  "tasks": {}\n'
      '}';

  const String populatedFirestoreDump = '{\n'
      '  "tasks": {\n'
      '    "0": {\n'
      '      "id": "0",\n'
      '      "name": "task 0",\n'
      '      "dateCreated": "1970-01-01T11:00:01.000",\n'
      '      "isComplete": false\n'
      '    },\n'
      '    "1": {\n'
      '      "id": "1",\n'
      '      "name": "task 1",\n'
      '      "dateCreated": "1970-01-01T11:00:03.000",\n'
      '      "isComplete": false\n'
      '    }\n'
      '  }\n'
      '}';

  Future<void> longPumpUntilSettle(WidgetTester tester) async {
    await tester.pump();
    await tester.pump();
    await tester.pump(const Duration(seconds: 1));
  }

  setUp(() {
    TestWidgetsFlutterBinding.ensureInitialized();
    globalFirestoreInstance = MockFirestoreInstance();
  });

  testWidgets('adding tasks to firestore will populate ListView', (WidgetTester tester) async {
    await tester.runAsync(() async {
      await tester.pumpWidget(MyApp());

      // initially no tasks in firestore
      expect((globalFirestoreInstance as MockFirestoreInstance).dump(), emptyFirestoreDump);
      expect(find.byType(Card), findsNothing);

      // Adding 2 mock tasks to firestore
      await tester.tap(find.byIcon(Icons.more_vert));
      await longPumpUntilSettle(tester);
      await tester.tap(find.text('Add Tasks'));
      await longPumpUntilSettle(tester);

      // should have added 2 widgets to ListView
      expect((globalFirestoreInstance as MockFirestoreInstance).dump(), populatedFirestoreDump);

      // this will fail
      expect(find.byType(Card), findsNWidgets(2));

      // but this would have passed
      // await tester.pumpWidget(MyApp());
      // expect(find.byType(Card), findsNWidgets(2));
    });
  });
}

Expected results:
2 Card widgets should have been added to the WidgetTester 'screen'
Actual results:
0 Card widgets are added to the WidgetTester 'screen', unless I completely rebuild the UI with await tester.pumpWidget(MyApp()); first

Logs

PS C:\Users\Lachie\MyProjects\FlutterProjects\lachies_life_planner> flutter test --verbose
[  +77 ms] executing: [C:\Users\Lachie\flutter/] git -c log.showSignature=false log -n 1
--pretty=format:%H
[  +68 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%H
[   +1 ms] c5a4b4029c0798f37c4a39b479d7cb75daa7b05c
[   +1 ms] executing: [C:\Users\Lachie\flutter/] git tag --points-at
c5a4b4029c0798f37c4a39b479d7cb75daa7b05c
[  +77 ms] Exit code 0 from: git tag --points-at c5a4b4029c0798f37c4a39b479d7cb75daa7b05c
[        ] 2.0.1
[  +38 ms] executing: [C:\Users\Lachie\flutter/] git rev-parse --abbrev-ref --symbolic @{u}
[  +43 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{u}
[        ] origin/stable
[        ] executing: [C:\Users\Lachie\flutter/] git ls-remote --get-url origin
[  +31 ms] Exit code 0 from: git ls-remote --get-url origin
[        ] https://github.com/flutter/flutter.git
[  +68 ms] executing: [C:\Users\Lachie\flutter/] git rev-parse --abbrev-ref HEAD
[  +33 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[   +1 ms] stable
[  +69 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping
update.
[   +1 ms] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping
update.
[   +1 ms] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[   +2 ms] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping
update.
[   +1 ms] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping
update.
[  +46 ms] Artifact Instance of 'MaterialFonts' is not required, skipping update.
[        ] Artifact Instance of 'GradleWrapper' is not required, skipping update.
[        ] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping
update.
[   +1 ms] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping
update.
[   +1 ms] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[        ] Artifact Instance of 'FlutterSdk' is not required, skipping update.
[        ] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping
update.
[   +1 ms] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping
update.
[   +1 ms] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FontSubsetArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'PubDependencies' is not required, skipping update.
[  +42 ms] Skipping pub get: version match.
[  +29 ms] Found plugin cloud_firestore at
C:\Users\Lachie\AppData\Roaming\Pub\Cache\hosted\pub.dartlang.org\cloud_firestore-1.0.1\
[   +9 ms] Found plugin cloud_firestore_web at
C:\Users\Lachie\AppData\Roaming\Pub\Cache\hosted\pub.dartlang.org\cloud_firestore_web-1.0.1\[  +12 ms] Found plugin firebase_core at
C:\Users\Lachie\AppData\Roaming\Pub\Cache\hosted\pub.dartlang.org\firebase_core-1.0.1\
[   +3 ms] Found plugin firebase_core_web at
C:\Users\Lachie\AppData\Roaming\Pub\Cache\hosted\pub.dartlang.org\firebase_core_web-1.0.1\
[ +135 ms] Found plugin cloud_firestore at
C:\Users\Lachie\AppData\Roaming\Pub\Cache\hosted\pub.dartlang.org\cloud_firestore-1.0.1\
[   +4 ms] Found plugin cloud_firestore_web at
C:\Users\Lachie\AppData\Roaming\Pub\Cache\hosted\pub.dartlang.org\cloud_firestore_web-1.0.1\[   +7 ms] Found plugin firebase_core at
C:\Users\Lachie\AppData\Roaming\Pub\Cache\hosted\pub.dartlang.org\firebase_core-1.0.1\
[   +6 ms] Found plugin firebase_core_web at
C:\Users\Lachie\AppData\Roaming\Pub\Cache\hosted\pub.dartlang.org\firebase_core_web-1.0.1\
[  +55 ms] Generating
C:\Users\Lachie\MyProjects\FlutterProjects\lachies_life_planner\android\app\src\main\java\io\flutter\plugins\GeneratedPluginRegistrant.java
[  +59 ms] running test package with arguments: [-r, compact, --timeout, 30s,
--concurrency=10, --,
C:\Users\Lachie\MyProjects\FlutterProjects\lachies_life_planner\test\main_test.dart]
00:00 +0: ...achie\MyProjects\FlutterProjects\lachies_life_planner\test\main_test.dart     [ +191 ms] test 0: starting test
C:\Users\Lachie\MyProjects\FlutterProjects\lachies_life_planner\test\main_test.dart
[  +18 ms] test 0: starting shell process
[   +7 ms] Stopping scan for flutter_test_config.dart; found project root at
C:\Users\Lachie\MyProjects\FlutterProjects\lachies_life_planner
[   +5 ms] Compiler will use the following file as its incremental dill file:
C:\Users\Lachie\AppData\Local\Temp\flutter_tools.e0dd7f72\flutter_test_compiler.ddfbb02c\output.dill
[   +1 ms] Listening to compiler controller...
[   +2 ms] Compiling
file:///C:/Users/Lachie/AppData/Local/Temp/flutter_tools.e0dd7f72/flutter_test_listener.e585d63c/listener.dart
[   +9 ms] C:\Users\Lachie\flutter\bin\cache\dart-sdk\bin\dart.exe --disable-dart-dev
C:\Users\Lachie\flutter\bin\cache\artifacts\engine\windows-x64\frontend_server.dart.snapshot--sdk-root C:\Users\Lachie\flutter\bin\cache\artifacts\engine\common\flutter_patched_sdk/
--incremental --no-print-incremental-dependencies --target=flutter --debugger-module-names
--experimental-emit-debug-metadata -DFLUTTER_WEB_AUTO_DETECT=true --output-dill
C:\Users\Lachie\AppData\Local\Temp\flutter_tools.e0dd7f72\flutter_test_compiler.ddfbb02c\output.dill --packages
C:\Users\Lachie\MyProjects\FlutterProjects\lachies_life_planner\.dart_tool\package_config.json -Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts --track-widget-creation
--initialize-from-dill
C:\Users\Lachie\MyProjects\FlutterProjects\lachies_life_planner\build\test_cache\build\c075001b96339384a97db4862b8ab8db.cache.dill.track.dill
[  +31 ms] <- compile
file:///C:/Users/Lachie/AppData/Local/Temp/flutter_tools.e0dd7f72/flutter_test_listener.e585d63c/listener.dart
00:01 +0: ...achie\MyProjects\FlutterProjects\lachies_life_planner\test\main_test.dart     [+1423 ms] <- accept
[   +3 ms] <- reset
[   +1 ms] Compiling
C:\Users\Lachie\AppData\Local\Temp\flutter_tools.e0dd7f72\flutter_test_listener.e585d63c\listener.dart took 1468ms
[   +6 ms] C:\Users\Lachie\flutter\bin\cache\artifacts\engine\windows-x64\flutter_tester.exe--disable-observatory --enable-checked-mode --verify-entry-points
--enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling
--non-interactive --use-test-fonts
--packages=C:\Users\Lachie\MyProjects\FlutterProjects\lachies_life_planner\.dart_tool\package_config.json
C:\Users\Lachie\AppData\Local\Temp\flutter_tools.e0dd7f72\flutter_test_listener.e585d63c\listener.dart.dill
[  +11 ms] Using this directory for fonts configuration:
C:\Users\Lachie\AppData\Local\Temp\flutter_tools.e0dd7f72\flutter_test_fonts.6214a3b9
[  +10 ms] test 0: awaiting initial result for pid 5444
[ +293 ms] test 0: process with pid 5444 connected to test harness
[   +1 ms] test 0: awaiting test result for pid 5444
00:02 +0: adding tasks to firestore will populate ListView
adding tasks
tasks added
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure object was thrown while running async test code:
  Expected: exactly 2 matching nodes in the widget tree
  Actual: _WidgetTypeFinder:<zero widgets with type "Card" (ignoring offstage widgets)>
   Which: means none were found but some were expected

When the exception was thrown, this was the stack:
#0      fail (package:test_api/src/frontend/expect.dart:155:31)
#1      _expect (package:test_api/src/frontend/expect.dart:150:3)
#2      expect (package:test_api/src/frontend/expect.dart:59:3)
#3      expect (package:flutter_test/src/widget_tester.dart:429:3)
#4      main.<anonymous closure>.<anonymous closure> (file:///C:/Users/Lachie/MyProjects/FlutterProjects/lachies_life_planner/test/main_test.dart:57:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
00:02 +0 -1: adding tasks to firestore will populate ListView [E]
  Test failed. See exception logs above.
  The test description was: adding tasks to firestore will populate ListView

[ +833 ms] test 0: process with pid 5444 no longer needed by test harness
[   +1 ms] test 0: cleaning up...
[   +1 ms] test 0: ensuring end-of-process for shell
[  +18 ms] test 0: deleting temporary directory
[   +4 ms] test 0: shutting down test harness socket server
[   +2 ms] test 0: finished
00:02 +0 -1: Some tests failed.
[   +9 ms] Deleting
C:\Users\Lachie\AppData\Local\Temp\flutter_tools.e0dd7f72\flutter_test_compiler.ddfbb02c...
[   +4 ms] killing pid 7496
[  +40 ms] Deleting
C:\Users\Lachie\AppData\Local\Temp\flutter_tools.e0dd7f72\flutter_test_fonts.6214a3b9...
[   +4 ms] test package returned with exit code 1
[   +9 ms] "flutter test" took 3,416ms.
[   +6 ms]
           #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
           #1      TestCommand.runCommand
(package:flutter_tools/src/commands/test.dart:305:7)
           <asynchronous suspension>
           #2      FlutterCommand.verifyThenRunCommand
           (package:flutter_tools/src/runner/flutter_command.dart:1157:12)
           <asynchronous suspension>
           #3      FlutterCommand.run.<anonymous closure>
           (package:flutter_tools/src/runner/flutter_command.dart:1009:27)
           <asynchronous suspension>
           #4      AppContext.run.<anonymous closure>
           (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #5      AppContext.run (package:flutter_tools/src/base/context.dart:149:12)
           <asynchronous suspension>
           #6      CommandRunner.runCommand (package:args/command_runner.dart:197:13)
           <asynchronous suspension>
           #7      FlutterCommandRunner.runCommand.<anonymous closure>
           (package:flutter_tools/src/runner/flutter_command_runner.dart:278:9)
           <asynchronous suspension>
           #8      AppContext.run.<anonymous closure>
           (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #9      AppContext.run (package:flutter_tools/src/base/context.dart:149:12)
           <asynchronous suspension>
           #10     FlutterCommandRunner.runCommand
           (package:flutter_tools/src/runner/flutter_command_runner.dart:234:5)
           <asynchronous suspension>
           #11     run.<anonymous closure>.<anonymous closure>
(package:flutter_tools/runner.dart:64:9)
           <asynchronous suspension>
           #12     run.<anonymous closure> (package:flutter_tools/runner.dart:62:12)
           <asynchronous suspension>
           #13     AppContext.run.<anonymous closure>
           (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #14     AppContext.run (package:flutter_tools/src/base/context.dart:149:12)
           #15     runInContext (package:flutter_tools/src/context_runner.dart:73:10)
           <asynchronous suspension>
           #16     main (package:flutter_tools/executable.dart:90:3)
           <asynchronous suspension>

[ +262 ms] ensureAnalyticsSent: 251ms
[   +2 ms] Running shutdown hooks
[        ] Shutdown hook priority 4
[   +3 ms] Shutdown hooks complete
[   +1 ms] exiting with code 1

 flutter analyze
Analyzing lachies_life_planner...
No issues found! (ran in 5.6s)

PS C:\Users\Lachie\MyProjects\FlutterProjects\lachies_life_planner> flutter doctor -v
[√] Flutter (Channel stable, 2.0.1, on Microsoft Windows [Version 10.0.18363.1440], locale
    en-AU)
    • Flutter version 2.0.1 at C:\Users\Lachie\flutter
    • Framework revision c5a4b4029c (3 weeks ago), 2021-03-04 09:47:48 -0800
    • Engine revision 40441def69
    • Dart version 2.12.0

[√] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at C:\Users\Lachie\Android\AndroidSDK
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = C:\Users\Lachie\Android\AndroidSDK
    • ANDROID_SDK_ROOT = C:\Users\Lachie\Android\AndroidSDK
    • Java binary at: C:\Users\Lachie\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

[√] Android Studio (version 4.0)
    • Android Studio at C:\Users\Lachie\Android\Android Studio
    • Flutter plugin version 51.0.1
    • Dart plugin version 193.7547
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] Connected device (2 available)
    • sdk gphone x86 (mobile) • emulator-5554 • android-x86    • Android 11 (API 30)
      (emulator)
    • Chrome (web)            • chrome        • web-javascript • Google Chrome 89.0.4389.90

• No issues found!

@lachiejames
Your code sample contains Firebase plugin which is 3rd party. Can you remove the relevant plugin code and try again to see if issue still persists or not ?
If so, please provide updated code sample that we can directly use to verify the issue.
Thanks.

I am unable to recreate the bug while using something other than FirebaseFirestore.instance.collection('tasks').snapshots() for the Stream source.  I tried using a different stream source, and the tests do not have the same issue.  Perhaps this needs to be raised on https://github.com/FirebaseExtended/flutterfire
For reference, this code (which does not use FirebaseFirestore) does not produce the same issue
main.dart:
import 'dart:async';

import 'package:flutter/material.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  int _counter = 0;
  StreamController<int> _events = StreamController<int>();

  @override
  initState() {
    super.initState();
    _events.add(0);
  }

  void _incrementCounter() {
    _counter++;
    _events.add(_counter);
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text("StreamBuilder test"),
        actions: [
          PopupMenuButton<String>(
            onSelected: (String value) => _events.add(0),
            itemBuilder: (BuildContext context) {
              return ['Reset'].map(
                (String option) {
                  return PopupMenuItem<String>(
                    value: option,
                    child: Text(option),
                  );
                },
              ).toList();
            },
          ),
        ],
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Text(
              'You have pushed the button this many times:',
            ),
            StreamBuilder(
              stream: _events.stream,
              builder: (BuildContext context, snapshot) {
                return Text(
                  snapshot.data.toString(),
                  style: Theme.of(context).textTheme.headline3,
                );
              },
            ),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _incrementCounter,
        tooltip: 'Increment',
        child: Icon(Icons.add),
      ),
    );
  }
}

widget_test.dart:
import 'package:flutter/material.dart';
import 'package:stream_test_bug/main.dart';
import 'package:flutter_test/flutter_test.dart';

void main() {
  testWidgets('adding tasks to firestore will populate ListView', (WidgetTester tester) async {
    TestWidgetsFlutterBinding.ensureInitialized();
    await tester.runAsync(() async {
      // Create MyApp
      await tester.pumpWidget(MyApp());
      await tester.pumpAndSettle();

      // Should initially be 0
      expect(find.text('0'), findsOneWidget);

      // Should go to 1 after tapping FloatingActionButton
      await tester.tap(find.byType(FloatingActionButton));
      await tester.pumpAndSettle();
      expect(find.text('1'), findsOneWidget);

      // Should reset to 0 after tapping 'Reset' on drop down menu
      await tester.tap(find.byIcon(Icons.more_vert));
      await tester.pumpAndSettle();
      await tester.tap(find.text('Reset'));
      await tester.pumpAndSettle();
      expect(find.text('0'), findsOneWidget);
    });
  });
}

Thanks for proactively filing in Flutterfire repo. Closing it from here based on your comment above.


Notes won't clear between recent notes from different hashtags

When browsing different hashtags via select input on recent notes screen, one can see notes from previous hashtag before notes from the next hashtag appear.
Steps to reproduce:

go to https://swarmstr.com/recent/asknostr
navigate to other hashtag using select menu in top left corner
verify that before notes from next tag are shown, notes from previous hashtag (asknostr) are still shown

Actual result
Notes from previous hashtag are shown before notes from next hashtag appear.
Expected result
Notes should be cleared between nagivations.

Fixed with latest release.


[QUESTION] How can I get access to @app in a different file from main.py?

Hello there,
Description
I have a middleware in my application, but, it is gigantic, and I want to move it to a different file. I would like to know if it is possible to access @app in a file that is not my main.py.
My main.py looks something like this:
import uvicorn
from fastapi import Depends, FastAPI, Header, HTTPException
from routers.supply_change import predict

app = FastAPI(title=title, description=description,
              version=version, docs_url=docs_url, redoc_url=redoc_url)

@app.middleware("http")
async def add_process_time_header(request: Request, call_next):
    # Do some stuff
    response = await call_next(request)
    # Do some stuff
    return response

app.include_router(predict.router)

but I want something like this:

main.py

import uvicorn
from fastapi import Depends, FastAPI, Header, HTTPException
from routers.supply_change import predict

app = FastAPI(title=title, description=description,
              version=version, docs_url=docs_url, redoc_url=redoc_url)

app.include_router(predict.router)

middleware.py

@app.middleware("http")
async def add_process_time_header(request: Request, call_next):
    # Do some stuff
    response = await call_next(request)
    # Do some stuff
    return response

You usually want to do it the other way around: Define a middleware function or class in a separate file, and then import that in your main where it will be attached to your app.
The decorator syntax for middlewares is equivalent to app.add_middleware(BaseHTTPMiddleware, dispatch=add_process_time_header), so what you should be doing is:
import uvicorn
from fastapi import Depends, FastAPI, Header, HTTPException
from routers.supply_change import predict

from starlette.middleware import BaseHTTPMiddleware
from .middleware import add_process_time_header

app = FastAPI(title=title, description=description,
              version=version, docs_url=docs_url, redoc_url=redoc_url)

app.add_middleware(BaseHTTPMiddleware, dispatch=add_process_time_header)

app.include_router(predict.router)

I did what you said, and it looks sooooo clean, thanks!

Thanks for the help here @sm-Fifteen ! :clap: :bow:
Thanks for reporting back and closing the issue @RenanAlonkin  :+1:


add semantic diff tool to git

https://difftastic.wilfred.me.uk/git.html

https://www.reddit.com/r/rust/comments/trcwix/difftastic_a_diff_that_understands_syntax/ has instructions for installing and configuring this!


can  jxl decode api be used in jpeg file?

it's said that the jxl library is compatible jpeg. so can it also used to decode jpeg?

Only after transcoding jpeg to jxl: https://gitlab.com/wg1/jpeg-xl/-/issues/232

This was mentioned as a possible idea for increasing JXL adoption in the JXL discord, as the JXL decoder often produces better looking output for the existing JPEGs on the internet.
From my understanding it is possible, but the code to do it has not been written.

It cannot at the moment, but I'd like to +1 this as a feature request: let libjxl decode jpeg directly (it already can to encode it to jxl, and it can decode a jpeg recompressed as jxl, so might as well make it handle it directly too).

Note that at the moment the decoder doesn't include the code to parse JPEG files (in lib/jxl/jpeg), only the encoder side uses it, so a decoder-only library with this feature would be larger. Perhaps this should be a build-time setting. There's also the question of API since the concepts between the two are different and that also affects the performance (like peak memory usage) that's possible to achieve. There's quite some work needed to do this FR properly.

I took an image and encoded it to q=20 and q=70 JPGs.  Then I losslessly transcoded them to JXL cjxl.  Then used djxl to convert them from JXL to PNG.  When I overlay the JPG and PNG with difference blending in GIMP and adjust levels, it looks like the differences are limited to the two least significant digits.  To decode the Q=70 JPG directly to PNG takes 7.792s.  To decode the JXL to PNG took 11.938s.  That's at least a 53% increased wait time, along with associated battery drain.

That's some big JPEG there...
Does this measure the full time taken by djxl? By far the biggest part of that is PNG compression, which is not really something that should be taken into account here :) (we use lodepng, which is not as fast as libpng IIRC)

It's a 24mp JPG from a digital camera.  I used the time command to get the timing.  I repeated the test with PPM and there's no significant difference in decode time.  I wasn't aware djxl was using a slow png library.
It would be nice to be able to decode JPG for the convenience of not having to use multiple libraries, but the differences in "image quality" seem minuscule.

Decoding directly a JPEG will likely be faster - as it is, (a) the first decode is about 2x slower than subsequent ones for internal reasons (you can try --num_reps=N and then dividing the time by N for a more accurate measurement) that would disappear when decoding an actual JPEG, and (b) about 50-60% of the time is spent entropy decoding and doing unnecessary-for-JPEG memmoves - entropy decoding a JPEG is significantly simpler, so this should account for a 30% or so reduction of the time you see today.
As for difference in image quality - you would likely be able to notice it more on HDR content, or more-than-8-bit monitors in general. I agree there is little difference on traditional displays, except for slow gradients of specific colors.

Note that at the moment the decoder doesn't include the code to parse JPEG files (in lib/jxl/jpeg), only the encoder side uses it, so a decoder-only library with this feature would be larger. Perhaps this should be a build-time setting. There's also the question of API since the concepts between the two are different and that also affects the performance (like peak memory usage) that's possible to achieve. There's quite some work needed to do this FR properly.

Compile-time option seems reasonable to me, at least for libjxl_dec. For the full libjxl I think it makes sense to always include such a feature. In terms of API, the main issue is probably streaming/progressive loading. If that's ignored for now, suppose we have the whole jpeg bitstream available, I think we can just parse it to internal representations (dc coeffs, ac coeffs, all 8x8 dctselect, etc) as if we're going to encode it, and then instead of starting tokenization etc, do the decode from there.
Doing progressive jpegs in a progressive way and sequential ones in an incremental/streaming way is more effort. For a typical application that isn't really needed though. If we at some point want to get rid of libjpeg-turbo in browsers, replacing it with libjxl, then that would be needed. Unless of course there would be magical consensus that it's OK to make a regression and let jpegs no longer render progressively, since you can use jxl instead if you care about progressive rendering (assuming that is made to work).
(But getting rid of libjpeg-turbo in browsers means we also need a jpeg encoder (because browsers are assumed to be able to encode jpeg), which is another thing we could do relatively easily (the full decoder kind of already has that for jpeg bitstream reconstruction), though it would again add more binary size to the libjxl build for browsers.)


Feature request: view module for hikari compositor

Hikari is a stacking Wayland compositor using wlroots.
It would be nice to have a Waybar module for hikari's views (the same as workspaces for river and sway)

It depends on itsvWayland compositor itself, sigh. Join the matrix room.


Cranelift: Using E-Graphs for Verified, Cooperating Middle-End Optimizations

Rendered
This RFC proposes to add a middle-end optimization framework to Cranelift based on e-graphs. A middle-end
optimization framework works to improve the target-independent IR (intermediate representation), CLIF, before Cranelift translates it to machine instructions. Cranelift already has some basic  ptimizations in this category: Global Value Numbering (GVN), Loop-Independent Code Motion (LICM), Constant folding, and most recently, a very basic form of alias analysis and redundant-load elimination. However, the current situation falls short in several ways.
The RFC will first describe how the current situation suffers from three problems: how to order and interleave these optimizations as we build more of them (the phase-ordering problem), how to ensure they are correct when they are all delicate hand-written code (the verification problem), and how to make it easier to build up a large body of known simplifications without tedious handwritten code. It will then describe how an e-graph-based framework could address all
three of these problems. It will describe some novel contributions in how to encode control flow in e-graphs (necessary in order to use e-graphs to optimize whole function bodies) developed during initial experimentation. Then finally it will discuss how we can use our existing rewrite-rule DSL, ISLE, to describe rewrites on the e-graph.

cc @fitzgen @jameysharp @elliottt @abrown @avanhatt @mlfbrown

Current prototype is bytecodealliance/wasmtime#4249; it implements the e-graph roundtripping but not (yet) the rewriting described in this RFC.

For example, the https://github.com/bytecodealliance/wasmtime/pull/1647 tooling included tools to "harvest" potential patterns to rewrite from real programs, and then use the superoptimization engine Souper to find rewrite rules offline. We could revisit such tooling and use it to build a significantly more effective body of rules semi-automatically.

FWIW, the harvester still exists and is independent of Peepmatic, it just extracts Souper IR from CLIF.
The only tool we are missing would be a Souper IR to ISLE(CLIF) translator for Souper's synthesized optimizations back to ISLE/this new framework.

This is incredibly cool work and I am very interested in following it! I also have appreciated your very thorough blog posts in the past. I'm seeing if it useful for people to have a place for egraph discussion, so I made a nascent but fairly quiet so far zulip. https://egraphs.zulipchat.com/join/q7bjayghbdw66xsnzgwc2hjb/

Super happy with how simple and obvious-after-the-fact your proposed e-graph destruction algorithm is (cycle breaking + scoped elaboration).

We will thus need a system of "optimization fuel" that limits rewrite steps overall and possibly per-rule or per-eclass. It is not yet certain what policies will work best; this is a question that we will need to experiment with. We simply wish to acknowledge now that it will be a problem that we will need to solve.

Note that egg already has a few knobs we can experiment with before trying to create our own custom thing here. In addition to perform-at-most-N-rewrite-iterations, you can also give it optional limits on the number of e-classes and e-nodes, and halt equality saturation once those are reached. I think the latter will be important for limiting the e-graph's memory overhead.

Hey all, egg author here. Very, very exciting stuff ahead! I will dive into the RFC soon and try to grok the details. I'm happy to help out as well!

How will e-graph-based optimization affect compile time?

Have you run spidermonke.wasm through sightglass with and without your prototype that just does round tripping through an e-graph? That could give us a tiny taste of what compile time overheads might look like (also I imagine you'd disable the old GVN when it is subsumed by the e-graph round trip). I'd also expect that, in theory, execution time of the generated code would be unchanged, but that would also be a good thing to empirically check.

How will e-graph-based optimization affect compile time?

Have you run spidermonkey.wasm through sightglass with and without your prototype that just does round tripping through an e-graph? That could give us a tiny taste of what compile time overheads might look like (also I imagine you'd disable the old GVN when it is subsumed by the e-graph round trip). I'd also expect that, in theory, execution time of the generated code would be unchanged, but that would also be a good thing to empirically check.

Nope, not yet, but that's next on my plate! Before doing measurements I want to do some basic optimizations to the scheme that I know how to do -- mainly, reuse existing CLIF insts rather than repopulating -- which should dramatically lower overhead. If with that we are "at parity" or "not much worse" than LICM (assuming LICM tweaks here) + GVN today, then I'd be very happy, but I guess how much overhead to accept there (in exchange for having an optimization framework) is another good discussion point.

I feel like this RFC is a little different from many of our other RFCs. Usually
we know we can implement the thing, and the RFC process is more about API design
and determining whether to implement the thing as described or in some
alternative way. Merging most RFCs is essentially committing to "yes we will do
this". This RFC, on the other hand, has relatively large open questions around
compilation overhead and whether we will actually solve phase ordering the way
we believe we should in theory. I think merging this RFC doesn't necessarily
mean "yes we will do this" because we still aren't 100% sure we can do
this. We think we can, but we need to actually build more to prove it out. So
I think merging this RFC should mean something more like "we will start landing
(cfg'd off) code in main and when we (hopefully) reach X, Y, and Z
acceptance criteria, we will stop hiding the code behind cfgs and remove the
old, non-egraph code paths."
And I would love this RFC to define and document our ahead-of-time consensus on
what exactly those acceptance criteria are so that we don't have moving goal
posts later on down the line when we are deciding whether to actually, fully,
100% commit to the e-graphs mid-end framework or not.

Just to spark discussion, I'll throw some stuff out there:

What kind of compile time overhead is acceptable for when we first turn this
on (with the expectation that we will have potential compile time wins down
the line as documented in the "Overall Cranelift Structure and Migration Path"
section)? 5% overhead? 10%?

What kind of memory overhead during compilation is acceptable? Unlike compile
time, I don't think we have any obvious possibilities for improving this in
the future other than limiting e-nodes/e-classes. (Please correct me if I'm
wrong!) 50% additional max RSS? 20%? 10%?

Finally, what should the acceptance criteria be for generated code quality?
How about: none of our Real World benchmarks in sightglass (SpiderMonkey,
markdown, bz2, meshoptimizer, and maybe intgemm-simd but I'm less familiar
with that one) get statistically significantly worse execution times and at
least one of them gets statistically significantly better execution
time. Additionally, we verify by hand that the alias analysis and GVN phase
ordering example you mention in the RFC actually gets resolved in practice by
our e-graphs framework.

What do other Cranelift stakeholders think our acceptance criteria should be?
Does compile time/space and code quality cover everything we need to keep an eye
on as far as acceptance quality goes or am I missing something else?

These are great questions, thanks @fitzgen -- I should've included more details around what I think the actual decisions should be (now and later)!
I actually am somewhat optimistic that we're close (a-few-weeks-to-a-month-ish?) to a point where we can evaluate the prototype plus basic rewriting rules that have functional parity to today's GVN, LICM, and simple_preopt, and make a decision with this RFC (merge or not) that means "we will actually do this". And to be clear I do want to get that far in the prototype before I personally advocate for merging; I want to ensure there are no unknown-unknowns in the rewrite step. I'd personally favor an acceptance criteria that is something like:

Replicates GVN, LICM, and simple_preopt (and clear path for alias analysis);
Is clearly an improvement in ease of contributing further optimizations;
Meets some compile-time threshold.

Open discussion how we balance items 2 and 3 above, but I hope we will have minimal overhead when we have only today's functionality. That would mean that additional compile-time cost is only a factor when actually adding interesting additional rules/optimizations.
On the flipside, I don't know if it necessarily makes sense to have a "something must speed up on initial merge" criterion, though I'm open to that if a bunch of other folks disagree! The reason is that I think once we have (i) a framework and (ii) a replica of today's rewrites, the future upside is clear enough that the work to come up with those additional optimizations can happen as a second step. But, I can see how we might want to require such a speedup if the compile-time overhead is also non-negligible.
Anyway all this is open to a bunch of discussion and, on that note, I'll put an item on the agenda for our next biweekly :-)

So I did some quick-and-dirty compile-time measurements. On SpiderMonkey.wasm, the overhead of roundtripping through the egraph adds about 23% to compile time (when opts are enabled). All current handwritten opts account for about 10% of compile time.
That's not terrible (not a doubling, for example) but it's not great either, so I dug into which parts were slow. It turns out that the extraction and elaboration take very little time (less than 10% of the egraph addition); most of the time is in the initial egraph build.
I suspect that we'll need to do some careful data-structure engineering to get things within acceptable overheads, but fortunately it looks like there's some relatively low-hanging fruit. Right now in egg the EClass struct keeps two Vecs, one for nodes and one for parents; the Id-to-EClass map is a HashMap while I would expect the Id space to be somewhat dense (maybe this is to clear out eclasses that have been merged?); and the Node interface forces us to keep args internally in separate lists (I use SmallVec currently for this). All of this could be refactored to use central Vecs in a pseudo-arena way (like Cranelift does for CLIF args/results currently), at the cost of passing around the "backing Vec".
I'll work on that for a bit, then return to hacking on the ISLE integration; once I prove out both of these, I'm happy to make a bunch more measurements!

@cfallin Awesome! At first glance, 23% isn't so bad, and I think there's you're right: a lot that could be done to get that down. Some quick points:

While the Id space is initially dense, you actually hope that it ends up sparse because many e-classes are getting merged.
I think you are right, we could get some perf back by using more arenas in EClass.

Also, the way the e-graph is being used in the prototype is basically a worst-case scenario for performance; for every Func, you build an e-graph, do e-graph stuff, extract and throw away. One big way we've improved performance in other egg applications is by batching many optimization queries at once. That would obviously require re-architecting the way egg and clif interact, which is probably something to do down the road.
Overall, I think this means that there are some serious "high-level" ways to improve performance as well as the lower-level ways you mentioned.

Thanks @mwillsey! By the way I'm planning to continue hacking on my fork of egg but I plan to upstream everything in due time, once plans are stable...
Re: this point:

Also, the way the e-graph is being used in the prototype is basically a worst-case scenario for performance; for every Func, you build an e-graph, do e-graph stuff, extract and throw away. One big way we've improved performance in other egg applications is by batching many optimization queries at once. That would obviously require re-architecting the way egg and clif interact, which is probably something to do down the road.

I want to dig a bit deeper: it seems you're saying that one way of tolerating overheads is to amortize the construction and extraction over more use of the egraph. I guess that's true in a "fraction of time spent in construction+extraction" sense, but I'm thinking in an absolute sense here: the time spent building the egraph and extracting from it, with no work between the two, is the bare minimum in additional overhead, no? At least as long as we have a separate egraph per function, and given that the overall problem we are solving is "compile this CLIF to machine code"? Or it's possible I'm missing some other way of batching or amortization that you mean to suggest...

Oh also a quick calibration point re:

At first glance, 23% isn't so bad

In Cranelift context this would be a huge deal actually: the whole regalloc rewrite (0.5 engineer-years) bought us about 20% compile-time improvement, and I fight to not give up 0.5% regularly. This is an open question above but I'll venture a ballpark number on this now just to calibrate (or at least give my own opinion): maybe 5% would be OK, but 20% would require a large increase in generated-code quality to justify, I think.

I'm planning to continue hacking on my fork of egg

Makes perfect sense. I'm happy to help hack on the fork if that's useful.

as long as we have a separate egraph per function

As long the e-graph optimization workflow is per-function, you can't really amortize. The batching I'm talking about is across functions: there's nothing stopping you from adding 1000 functions to the same e-graph (hopefully there's some sharing in there!) and optimizing them simultaneously. I suspect that even just adding 1000 functions to one e-graph is faster than adding 1000 functions to 1000 e-graphs because there is likely some sharing, but I could be wrong here.
Obviously one e-graph per function is cleaner for how the optimization workflow is right now. It just comes with the unfortunate overhead of basically copying your whole program twice: once into the e-graph and once out. Even outside of batching, another way of improving performance would be to eliminate one or both of these copies by increasing the part of the pipeline where you use e-graphs, ideally to some other "translation" boundary where you'd have to lower to another IR anyway. This is clearly a big task, though.

That's a really interesting idea; we can certainly play with it at some point!
We currently have a hard boundary between each function compilation, in order to allow for parallel compilation, but we'll need to break that boundary (optionally) eventually for inlining as well. And, actually, now that these two ideas are next to each other: putting the inlining and inlined function in the same egraph, linking up the dataflow, and then elaborating out of that is a really nice clean way of doing inlining. So, you've spawned some thoughts here, thanks!
One potentially complicating factor toward deduplicating efficiency between function bodies is that the "roots" (terminal nodes) will differ: both f and g may contain complex_expr(x, y, z) but where x, y, and z are different block params, or side-effecting ops (calls, loads), or whatever, in the two functions. I wonder if there is some sort of canonicalization that could help (in the extreme case this feels like a graph isomorphism problem though). In the egraph literature I mostly see expressions in terms of variable terminals x, y, etc and so this doesn't seem to be as big a deal. Any thoughts?

A brief update: I hacked up a custom egraph implementation designed to be more memory-efficient and allocation-light, and got the overhead down to 9%.
The key differences are in how it avoids ever cloning Nodes, using indexmap to allow both node-to-ID dedup and ID-to-node access and using these IDs in eclasses' list of nodes (the use of IDs to name nodes is itself different from egg too); and using Cranelift's EntityList to store lists of node IDs and eclass IDs in pooled storage; and using bumpalo (an arena allocator) to allocate the argument lists of eclass IDs. All of this is to implement the principles (i) O(1) memory allocations, i.e. we should grow single large containers rather than have nested containers per node or eclass (entity-component-system style) and (ii) avoid copies and store nodes only once. There's a little more we could do in theory with a custom hashmap that doesn't require Hash+Eq (namely, allows us to provide context for those; that would allow us to provide the backing pool separately and not store a fat-pointer slice for args and types in every Node, but only a u32 index).
I haven't built the merging or rule-application bits yet but the roundtripping (without any rule applications) actually never merges eclasses so this is enough to evaluate the overhead of this part of the pipeline.
Given this, and given that GVN+LICM add around 10% to compile time, I'm actually cautiously optimistic that we can slip into the hole left by removing these stages and be almost neutral when no rewrite rules are applied. That's the ideal case in my mind. Or at least have small enough overhead that this roundtrip is always on when opts are on. Later we can remove half the roundtrip by generating egraph nodes straight from the frontend/builder API, as noted in the last section of the RFC (and as you note as well @mwillsey) for further savings, but for now anchoring both ends of the transform in CLIF (which can be printed, manipulated, used for testing, etc) has substantial pragmatic benefits I think.
(I'm not sure if the path forward is to actually use cranelift-egraph or egg with a bunch of hacks; as a general principle I don't like to go all NIH and hack up my own versions of things; but the data structures and tradeoffs are different enough that this may be the right way, I'm not sure. Right now I just want to pathfind wrt "is this performance profile possible".)
After dropping this tidbit I'm now promptly going to disappear for a week of PTO (...sorry!) but wanted to get that data point out first. More later!

There's a little more we could do in theory with a custom hashmap that doesn't require Hash+Eq (namely, allows us to provide context for those; that would allow us to provide the backing pool separately and not store a fat-pointer slice for args and types in every Node, but only a u32 index).

hashbrown's RawTable might help with that.

There's a little more we could do in theory with a custom hashmap that doesn't require Hash+Eq (namely, allows us to provide context for those; that would allow us to provide the backing pool separately and not store a fat-pointer slice for args and types in every Node, but only a u32 index).

hashbrown's RawTable might help with that.

Indeed! I pushed a new implementation that does just that over here. (That could probably use a lot of cleanup, and is a bit lifetime-heavy, but proves out what I wanted to!) Thanks for the pointer to that.

This RFC is very interesting to me, i plan to use e-graphs too for a large part of the optimizer for a math-heavy language im writing. I considered using ISLE as well for writing patterns, i think it would work pretty well. However ran into a couple of open questions i could not resolve:

Egg by default uses exponential backoff for rules to avoid dominating rewrite time with rules like associativity, but if we just use a single gigantic pattern matching trie for the rules, how would we disable rules? This is probably doable by keeping some arrays for counting rule "hits", and checking those arrays before matching/falling through, but the matching and memory overhead for this needs to be explored. Using indirect jumps through a lookup table could probably also work (indirect-threaded interpreter style kinda). Would love to hear if @mwillsey has any ideas for this.
Is ISLE capable of expressing rules common to multiple types easily? for example, x * 1 => x can be applied to floats, ints, and uints of any size. I presume ISLE has a way to handle this, but i haven't explored it much.
Is it potentially possible to store e-class data separate to the actual language? I.e the Language enum would be 64/128-bit sized and only contain data required for matching, for example, Add([Id; 2]), To potentially match rules much faster by exploiting vectorized pattern matching.

But overall this approach seems promising, and it would be awesome to share rulesets/ideas with cranelift.

Hi @RDambrosio016,
egg doesn't use trie-based pattern matching, so you'd essentially have to re-implement e-matching to support this. I believe Z3 does do something like this (Efficient E-matching for SMT Solvers, section 4). We prototyped it at some point, and found it was slower because the rules we were using didn't significantly "overlap". In a high-performance setting, what I would try before tries is just running rules using the SimpleScheduler, but with a fixed, small number of iterations.
As far as using Analysis for tracking invariants, sure! Just be careful, an analysis is per e-class, so it depends on your notion of equivalence.

Interesting, i did a small experiment generating rules with my own DSL, and, while decently fast, it definitely seems like egg spends a lot of its time on rules that could never match if you have a ruleset that is somewhat larger as a consequence of duplicating rules for numerous numeric types. In this particular ruleset i have ~61 unique rules (including bidirectionality), which after being duplicated for multiple numeric types, result in 240 total rules, and this is how long egg takes on each step:
Number of rules: 240
Time from runner creation: 2.6857ms

Runner report
=============
  Stop reason: Saturated
  Iterations: 7
  Egraph size: 148 nodes, 23 classes, 177 memo
  Rebuilds: 4
  Total time: 0.0026048
    Search:  (0.82) 0.0021298000000000003
    Apply:   (0.17) 0.0004338
    Rebuild: (0.02) 0.0000397

2.0 - pow(x * 1.0 * -1.0, 2.0) + (y / z + 3.0) ----> fma(x, neg(x), fma(y, recip(z), 5))
Cost: 26.5 -> 12

This is on release with 1 CGUs and LTO on, on an i9-10850k. Search time making up the bulk of the time is not unexpected, but to me it seemed a little bit high considering how small the expression is. So i re-tried it but without duplicating rules, and i got these results:
Number of rules: 50
Time from runner creation: 1.0181ms

Runner report
=============
  Stop reason: Saturated
  Iterations: 7
  Egraph size: 148 nodes, 23 classes, 177 memo
  Rebuilds: 4
  Total time: 0.0009838
    Search:  (0.66) 0.0006475000000000001
    Apply:   (0.30) 0.0002992
    Rebuild: (0.04) 0.000035999999999999994

2.0 - pow(x * 1.0 * -1.0, 2.0) + (y / z + 3.0) ----> fma(x, neg(x), fma(y, recip(z), 5))
Cost: 26.5 -> 12

So it definitely seems like hierarchical matching of some sort would be required to avoid spending a ton of time searching the graph. It's odd that your experiment in trie matching ended up being slower, was it using generated rust code like ISLE creates, or was it dynamically matched at runtime?

@RDambrosio016 I'd be happy to talk more about sharing efforts once the Cranelift work here is more mature; for now there is still exploration going on. I suspect that a lot of what we do is going to be fairly Cranelift-specific: there is a large semantic gap between "x*1 == x" at the conceptual level, and the particular matchers, node formats, and idiosyncrasies of the framework. My experience in the past is that trying to paper over such details to give a nice high-level view can sometimes be counterproductive because it either creates subtle bugs, or sweeps efficiency concerns under the rug. But once we've built more, perhaps there's something to be used for your effort too :-)
In the meantime perhaps it's best if we keep this discussion thread scoped to the Cranelift effort? It's likely to get much longer in time anyway (I have some more updates cooking at the moment) but side-discussions about other projects might pull us a bit too far off course... thanks!

Absolutely 👍 , didnt mean to steer it into more minute details, i was just wondering if you considered/discussed this problem in the past, since i looked at the RFC but it did not seem to mention it.

For sure, performance is my main concern right now and I'm working to make the whole thing competitive with handwritten optimizations, so you'll here more about that in a bit!

Beating hand-written optimizations is definitely an achievable goal, considering the e-graph approach would have a much easier time knowing when to stop optimizing vs a standard pass-based optimizer. It will also be an interesting problem trying to figure out a good cost metric that maximizes generated code quality. Having a complete e-graph to choose from is great, but trying to select the ideal version from that will definitely prove to be a challenge.

As an update here, on the running prototype/experimental branch here, I've gotten overhead of Cranelift-with-egraphs (including elaboration that subsumes GVN and LICM, and rewrite rules using ISLE that encompass basic const propagation and algebraic simplifications) vs Cranelift-with-handwritten-opts down to ~1% slower on parallel compile time (and actually slightly faster in terms of total CPU time across cores):
cfallin@min:~/work/wasmtime % hyperfine --show-output --  "target/release/wasmtime compile --cranelift-set use_egraphs=true ../wasm-tests/spidermonkey.wasm"
Benchmark 1: target/release/wasmtime compile --cranelift-set use_egraphs=true ../wasm-tests/spidermonkey.wasm
  Time (mean ± σ):      1.693 s ±  0.098 s    [User: 10.747 s, System: 0.234 s]
  Range (min … max):    1.621 s …  1.951 s    10 runs

cfallin@min:~/work/wasmtime % hyperfine --show-output --  "target/release/wasmtime compile --cranelift-set use_egraphs=false ../wasm-tests/spidermonkey.wasm"
Benchmark 1: target/release/wasmtime compile --cranelift-set use_egraphs=false ../wasm-tests/spidermonkey.wasm
  Time (mean ± σ):      1.671 s ±  0.035 s    [User: 10.888 s, System: 0.236 s]
  Range (min … max):    1.606 s …  1.707 s    10 runs

(That's measuring on an M1 laptop with SpiderMonkey.wasm, a large Wasm module testcase.)
This is using a new acyclic-egraph approach that I'm calling the "aegraph" (or ægraph for those who love digraphs). I got the conventional egraph rewriting to work last week, and measured it and found compilation time was inflating significantly once rewrites occur -- something like 5x. I then had a series of realizations that the way that we're using egraphs here is more limited than perhaps a conventional full-equality-saturation approach, and had a series of realizations about simplifications we could make.
Basically an ægraph contains eclasses as nodes where each node contains an enode, an enode and a parent-eclass pointer, or two parent-eclass pointers. The union operation joins two eclasses together only by adding nodes (or as an optimization, replacing an enode node with an enode-and-parent node). An eclass node can refer only to earlier eclasses. We maintain the acyclicity invariant when adding enodes and unioning eclasses. Rule application occurs immediately when a new node is added. In a sense, eclasses are split into "levels" and a given level of an eclass refers only to the levels of argument eclasses that existed when it was created. It's thus a persistent immutable data structure. An eclass is represented by the id of its latest version (highest id), whereas the traditional union-find approach tends to push ids downward to the lowest id.
This data structure has some really compelling properties:

Because eclass nodes are immutable once created, we only need to apply rules to a node once. This "single pass" rule application is at the heart of its speed.
We don't re-canonicalize and re-intern enodes, so we don't need parent pointers and we don't need a rebuild step.
It plays really nicely with the scoped elaboration, in that we can keep a parallel scoped map with the "best node" (extraction information) and memoize; extraction doesn't need cycle removal any more.

The tradeoff is that it's not at all a full equality saturation (see above re: no recanonicalization). But I think that's OK for our use-case: we mainly (i) introduce new expressions then (ii) apply rules that introduce alternate forms of those expressions, and union them together.  In this case the ægraph is pretty ideal: it's like a streaming journal of nodes and we add a new node from the build, then add a few more nodes that are equivalent to it and union them in-place, then keep going, referring to the just-built expression by the latest id. So the ægraph is sort of a "multi-version IR" where every value can have multiple representations, combined with GVN.
(The case it misses that a traditional egraph with parent pointers, recanonicalization, and the rebuild step would get is where we introduce some node A that gets a new eclass, a rewrite rule adds a node B and unions it with A, and B happens to dedup to an already-existing eclass. The new ID for A will point to a union node that joins A and B's existing eclass, but other users of B will not see A because it's "newer". This seems to be of limited use for our specific rules because our rewrite rules tend to go from more expensive/complex expressions to simpler ones, so A is likely not better than B. That's not always the case and I'm sure there are cases where a full egraph does do better. But the multiple-version-representation and ability of rules to match on all versions seems like the key thing that we want from the data structure.)
Anyway, I'll write this up more properly but that's where I'm at now: a prototype that works and is fast. I'll add a bit more to it (integration with alias analysis! and other rules I haven't added yet) and then start to look at generated code quality too.

Very interesting! im curious to see if that specific limitation will end up limiting any optimization potential in the future. It may be worth it to try running generated code benchmarks both without rebuilding, and with. Although it's definitely possible that associativity rules will hide that limitation. Nevertheless, the performance on something as large as the spidermonkey wasm file is impressive! Did you happen to measure how much memory the e-graphs/program overall took?
Excited to try out this approach, i was thinking about perhaps working on algebraic identities for mul/div/rem mostly based on this paper. Would you accept a PR to your experimental branch to add such rules?

Sure; more opts are definitely welcome in a PR, thanks!
I haven't done any more detailed measurements; I'm polishing things wrt integrations now and once I have it in a state where I'm happy, I'll do a full comparison and set of measurements (with Sightglass) on compile- and runtime performance.

Hi all -- I have some updates. It's kind of a mixed bag -- tl;dr, code quality matches existing Cranelift (all optimizations have been reproduced), compilation speed is ~1-5% slower. Theoretical benefits of everything-in-one-fixpoint (GVN and alias analysis and LICM and algebraic simplifications all cooperating) work in filetests, but don't seem to do much to Wasm-based benchmarks in Sightglass.
So the current status of the WIP branch is:

"acyclic egraph" (ægraph) implementation is working;
producing and lowering out of this aegraph works fine;
ISLE-based rewrite rules operate on this aegraph, and cost-based extraction picks the best option for each node after rewrites are done;

rewrites in the aegraph apply immediately (because of acyclicity, there's no need to revisit a node);
ISLE itself has been extended to support "multiplicity" in extractors and constructors, so multiple rules can match and matchers on eclass IDs see all available enodes for each eclass;

The aegraph construction and "scoped elaboration" subsume GVN (global value numbering, a form of common-subexpression elimination) by means of the deduplication map;
The aegraph elaboration automatically does LICM (loop-invariant code motion) implicitly, by way of loop-nest awareness when traversing the domtree;
The alias analysis feeds into the rewrite rules and there are rules for redundant-load elimination and store-to-load forwarding;
Rewrite rules have been written for a bunch of constant-propagation and algebraic-simplification cases, and some other slightly more ad-hoc ones (e.g. reassociating loads to try to make LICM on partially-loop-invariant sums work)
A simple form of rematerialization has been implemented, where adds-with-one-immediate, and immediates themselves, are regenerated in the block they are used rather than maximally hoisted by GVN, in order to reduce register pressure.

All this works -- and IMHO the data structures and algorithms are pretty beautiful actually! -- but it's not any faster. It is nicer to update with more rewrite rules -- see cprop.isle and algebraic.isle -- and if it were completely at parity performance-wise, this would be an interesting benefit still IMHO. But there is still a small compile-time penalty, around 5% on average, with everything above incorporated.
I have some more thoughts in my running TODO list but most of the remaining ones are for compile speed, not code quality (runtime speed).
Given all that, I think I'm running up against a wall that's actually mostly supported by some other limits:

There is actually not a ton of remaining "low-hanging fruit" when running Wasm modules produced by an optimizing compiler. (Any strength reduction or other clever "instcombine"-type optimization I would expect to be done by LLVM before producing the .wasm.)
Lowering, once it occurs, disallows any further optimizations, because we can't edit VCode currently.

The first is just a fact of our benchmarking conditions (but we're oriented this way because we care very much about Wasm performance -- we can't just choose to focus on poorly-optimized CLIF instead and say "see look opts work!"). But the second is where a lot of my thoughts are going lately. In other words, from first principles, if we start with highly-optimized Wasm bytecode, where do we expect inefficiencies and redundancies to creep in? Probably one of the three areas:

The semantic gap between Wasm bytecode and CLIF, and how it is bridged (heap bounds checks, table accesses, and the like);
The semantic gap between CLIF and the machine, and how it is bridged (our lowering rules in our backends);
regalloc quality.

My forehead is already dented from banging against item 3 (regalloc quality) for a long time, and item 1 (Wasm to CLIF) is largely addressed already by designing CLIF semantics to be close to those of Wasm; but item 2 (lowering) presents unharvested opportunities IMHO.
In looking at disassemblies of "hot blocks" when profiling runtime I see a lot of not-great code, and it comes in several varieties: redundant flags computations (cmp / setnp / cmp / sete for float comparisons is a common one); redundant trap conditions that cprop could optimize away, or GVN could combine between multiple ops; address-mode junk produced by our "match up the operand tree and then regenerate an add tree for the parts we can't incorporate" logic in both x64 and aarch64; a bunch of "spill x+1, x+3, x+10, x+15 to separate spillslots for use later" (hence my attempt at remat above); and others. A lot of this comes from the inability to optimize code once we lower a CLIF op into multiple MachInsts, or one pseudo-MachInst.
(Those who have been around a while might be screaming: legalization and post-opt! Yes, you're right. I think we threw out some of the value of those in exchange for the speed and simplicity of the single-pass lowering, and the real value that we do get from a two-level IR. But we lost something too.)
So: I'm taking a little time to think about what might make sense as a way to optimize VCode. I don't think this means throwing out the aegraph and ISLE-based mid-end framework; I think instead there may be a way to write some mid-end legalizations that do a lot of expansion that we currently do in lowering, and then let lowering be simpler; or in the extreme, even have a Node::MachInst { ... } in the aegraph, though that has potential efficiency issues. More thought needed.
Anyway, I'd love to hear more thoughts on this, and we can talk about it more in the next Cranelift meeting, but there's my report on what I've learned in the last few weeks :-)

... and in a typical show of my luck, I discover just after writing the above that my remat was doing something wrong (LICM was overriding its decision). I now have the result:

on spidermonkey.wasm I see an 11.4% speedup, with 5% extra compile time (the latter I can optimize a bit more I think);
on bz2.wasm I see a 7.3% speedup, also with 5% extra compile time.

More measurements later but IMHO if I can get the compile time overhead a bit lower this should be compelling on its own (then the post-lowering optimization ideas above become the next step).

I did some optimization on compiler runtime tonight -- both low-level things like inline pragmas, refactoring some data structures to allow holding borrows rather than lookups, using dense rather than sparse maps in places, preallocating with correct sizes, and similar, and high-level things like algorithmic improvements to memoize more, trim the extraction search space (running upper bound and early exit/pruning), "subsume" all nodes in an eclass with one "final" version (for e.g. cprop), and the like.
In the end I got SpiderMonkey.wasm down to a ~1% compile-time regression (relative to baseline Cranelift), and bz2.wasm to a... 15% compile-time speedup (yeah, I know; it's wild; I was focusing entirely on SpiderMonkey and then just checked bz2 now, and would have fallen off my chair if gravity were slightly stronger):
[cfallin@xap]~/work/wasmtime% hyperfine  -L egraphs false,true 'target/release/wasmtime compile --cranelift-set use_egraphs={egraphs} ../wasm-tests/spidermonkey.wasm'
Benchmark 1: target/release/wasmtime compile --cranelift-set use_egraphs=false ../wasm-tests/spidermonkey.wasm
  Time (mean ± σ):      1.106 s ±  0.051 s    [User: 15.034 s, System: 0.505 s]
  Range (min … max):    1.043 s …  1.201 s    10 runs

Benchmark 2: target/release/wasmtime compile --cranelift-set use_egraphs=true ../wasm-tests/spidermonkey.wasm
  Time (mean ± σ):      1.115 s ±  0.033 s    [User: 15.915 s, System: 0.510 s]
  Range (min … max):    1.077 s …  1.162 s    10 runs

Summary
  'target/release/wasmtime compile --cranelift-set use_egraphs=false ../wasm-tests/spidermonkey.wasm' ran
    1.01 ± 0.05 times faster than 'target/release/wasmtime compile --cranelift-set use_egraphs=true ../wasm-tests/spidermonkey.wasm'

[cfallin@xap]~/work/wasmtime% hyperfine -r 200  -L egraphs false,true 'target/release/wasmtime compile --cranelift-set use_egraphs={egraphs} ../wasm-tests/bz2.wasm'
Benchmark 1: target/release/wasmtime compile --cranelift-set use_egraphs=false ../wasm-tests/bz2.wasm
  Time (mean ± σ):      82.1 ms ±  10.8 ms    [User: 203.2 ms, System: 58.4 ms]
  Range (min … max):    71.4 ms … 106.5 ms    200 runs

Benchmark 2: target/release/wasmtime compile --cranelift-set use_egraphs=true ../wasm-tests/bz2.wasm
  Time (mean ± σ):      71.4 ms ±  10.2 ms    [User: 191.0 ms, System: 58.7 ms]
  Range (min … max):    59.6 ms …  96.3 ms    200 runs

Summary
  'target/release/wasmtime compile --cranelift-set use_egraphs=true ../wasm-tests/bz2.wasm' ran
    1.15 ± 0.22 times faster than 'target/release/wasmtime compile --cranelift-set use_egraphs=false ../wasm-tests/bz2.wasm'

And for completeness, yesterday's runtime numbers were on my M1 laptop, while today's work is on a Linux/x86-64 desktop (Ryzen 3900X); on this system, SpiderMonkey.wasm sees a 13% speedup, and bz2.wasm a 3% speedup:
[cfallin@xap]~/work/wasmtime% target/release/wasmtime compile --cranelift-set use_egraphs=false ../wasm-tests/spidermonkey.wasm -o base.cwasm
[cfallin@xap]~/work/wasmtime% target/release/wasmtime compile --cranelift-set use_egraphs=true ../wasm-tests/spidermonkey.wasm -o egraphs.cwasm
[cfallin@xap]~/work/wasmtime% hyperfine -L file base,egraphs 'target/release/wasmtime run --allow-precompiled --dir=. {file}.cwasm ./fib.js'
Benchmark 1: target/release/wasmtime run --allow-precompiled --dir=. base.cwasm ./fib.js
  Time (mean ± σ):      1.345 s ±  0.023 s    [User: 1.338 s, System: 0.005 s]
  Range (min … max):    1.317 s …  1.384 s    10 runs

Benchmark 2: target/release/wasmtime run --allow-precompiled --dir=. egraphs.cwasm ./fib.js
  Time (mean ± σ):      1.195 s ±  0.010 s    [User: 1.189 s, System: 0.003 s]
  Range (min … max):    1.181 s …  1.205 s    10 runs

Summary
  'target/release/wasmtime run --allow-precompiled --dir=. egraphs.cwasm ./fib.js' ran
    1.13 ± 0.02 times faster than 'target/release/wasmtime run --allow-precompiled --dir=. base.cwasm ./fib.js'

[cfallin@xap]~/work/wasmtime% target/release/wasmtime compile --cranelift-set use_egraphs=false ../wasm-tests/bz2.wasm -o base.cwasm
[cfallin@xap]~/work/wasmtime% target/release/wasmtime compile --cranelift-set use_egraphs=true ../wasm-tests/bz2.wasm -o egraphs.cwasm
[cfallin@xap]~/work/wasmtime% hyperfine -L file base,egraphs 'target/release/wasmtime run --allow-precompiled {file}.cwasm'
Benchmark 1: target/release/wasmtime run --allow-precompiled base.cwasm
  Time (mean ± σ):     428.3 ms ±   9.7 ms    [User: 425.3 ms, System: 1.7 ms]
  Range (min … max):   416.6 ms … 441.6 ms    10 runs

Benchmark 2: target/release/wasmtime run --allow-precompiled egraphs.cwasm
  Time (mean ± σ):     414.5 ms ±  12.8 ms    [User: 411.5 ms, System: 1.9 ms]
  Range (min … max):   403.1 ms … 440.7 ms    10 runs

Summary
  'target/release/wasmtime run --allow-precompiled egraphs.cwasm' ran
    1.03 ± 0.04 times faster than 'target/release/wasmtime run --allow-precompiled base.cwasm'

I think we're at the point where we can discuss this in the next Cranelift meeting, then hopefully, if consensus exists, I'll start cleaning this up and preparing it for merging (in small, more easily-digestible pieces!).

@cfallin Amazing work! These are killer results!  I think specializing to the ægraph (great name) is definitely the right decision for this use case; the trade-offs seem to be worth it. Although, it does make me wonder what it would take to make "normal" equality saturation usable in this setting. Exciting future work!

This is so cool!
Would be really interested to see how much better results are with different tweaks. Does full egraph-style matching do much good? I'm thinking about the case where an enode creates a new match in a parent eclass.

Thanks @mwillsey! I'm definitely curious what other design points may be possible here. I suspect that benefits of more strongly normalizing approaches (full recanonicalize/rebuild as in a traditional egraph or egg) will become more apparent the more we add interesting rules. Right now I have rules I wrote with my traditional-compiler-author brain, so mostly constant prop and the like where the right-hand side is unambiguously better. (There are a few cases of reassociation where I let the extractor do the right thing though, and also scoped elaboration can make things interesting when another expression forces evaluation of some otherwise-expensive node, shorting its cost to 0 for subsequent uses and altering the best choice. (This is why extraction is done lazily and memoized.) In those cases I suspect having multiple versions around actually helps.)
There are definitely some experiments I'd like to run, if I were to put on my old grad-student hat (it has collected some dust!) -- among them, putting together a full rebuild step as @oflatt suggests, and maybe also a "rewrites always subsume" mode (ie no multi-version capability) with the rules. My day-job means I need to focus on getting this "makes compiler faster and better" magic into production shape first, of course, but I do want to experiment some as time allows :-)

Hello! Yet another egg contributor here. @mwillsey and I had a great chat with Nico Bruno, who was a main engineer of SQLServer. It turns out one main difference between equality saturation and cascade-style DB query optimization is that the latter maintains the acyclicity of the e-graph, just like aegraph. Acyclicity enables many powerful optimizations like branch-and-bound, and usually runs very fast. I highly recommend checking out some of the literature there, as well as some modern implementations like cockroach and orca.

@remysucre thanks for the pointers! (As with many other topics in systems work, the database folks got there first... why am I not surprised?!). I'm happy to hear there are similar approaches in use and I'll read up more.

Hi all -- I've updated the RFC to refer to ægraphs and include the numbers above. A full writeup of how ægraphs work is IMHO out-of-scope here but I will include a bunch of detail somewhere in cranelift/codegen/docs/ if this merges. The important decision points here, IMHO, are:

Should we have a mid-end?
Should it operate based on rewriting rules?
Is an implementation based on the prototype acceptable, given its numbers (with polish in a bunch of production details before merging)?

I'd like to kick off a round of discussion and (hopefully!) consensus-building on that. Happy to talk more about this in the next Cranelift meeting but let's see what folks think here first. Thanks!

Hi all -- after discussing this briefly yesterday it sounds like no one has any major objections; so given that, and the positive results we've found, I would like to make a
Motion to finalize with a disposition to merge
I've updated the list below to contain all stakeholders who have committed to Cranelift within the past three months, or participated in RFC discussions in the same timeframe.
Stakeholders sign-off
Arm

[ ] @akirilov-arm
[ ] @sparker-arm
[ ] @dheaton-arm

Cornell

[ ] @avanhatt

Embark Studios

[ ] @bnjbvr

Fastly

[ ] @alexcrichton
[ ] @cfallin
[ ] @fitzgen
[ ] @sunfishcode
[ ] @elliottt
[ ] @jameysharp
[ ] @iximeow

IBM

[ ] @uweigand

Intel

[ ] @abrown
[ ] @jlb6740

Unaffiliated

[ ] @bjorn3

Given approvals from multiple stakeholder groups, we now enter the 10-day final comment period.
If no objections are registered by 2022-09-03, we will merge this RFC. Thanks all for the discussion!

The FCP has now (more than) elapsed; with no objections, I will now merge this RFC! Thanks all for the spirited discussion and ideas. Hopefully soon now we can get a proper implementation into Cranelift; my next task is to clean up the prototype and start to submit pieces for review.


Strange behaviour with spring-boot (or jackson-databind:2.9.9)

Here is a simple example to reproduce the problem:
import java.util.Arrays;
import java.util.Collection;
import java.util.TreeSet;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.github.bohnman.squiggly.Squiggly;
import com.github.bohnman.squiggly.util.SquigglyUtils;

public class SquigglyTest {
	public static void main(String[] args) {
		Object object = new ListResponse(Arrays.asList(new Item[] {new Item("id")}));

		ObjectMapper mapper = Squiggly.init(new ObjectMapper(), "items[**]");
		System.out.println(SquigglyUtils.stringify(mapper, object));

		mapper = Squiggly.init(new ObjectMapper(), "items");
		// Should return set and id ... but returns only set when spring-boot-parent is in the pom!
		System.out.println(SquigglyUtils.stringify(mapper, object));
	}

	public static class ListResponse {
		private Collection<Item> items;

		public ListResponse(Collection<Item> items) {
			super();
			this.items = items;
		}

		public Collection<Item> getItems() {
			return items;
		}
	}

	public static class Item {
		private Collection<String> set;
		private String id;

		public Item(String id) {
			// Strange if we create a List instead of a Set, the problem vanishes
			this.set = new TreeSet<>();
			this.id = id;
		}

		public Collection<String> getSet() {
			return set;
		}

		public String getId() {
			return id;
		}
	}
}

The non filtered object is: {"items":[{"set":[],"id":"id"}]}
The program filters it with the "items[**]" and "items" filters.
If the only dependency in the pom is squiggly library (version 1.3.18), anything is fine and the output is the one expected:

{"items":[{"set":[],"id":"id"}]}
{"items":[{"set":[],"id":"id"}]}

Now, if we add the spring-boot-parent in the pom
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.1.7.RELEASE</version>
	</parent>

The output become:

{"items":[{"set":[],"id":"id"}]}
{"items":[{"set":[]}]}

The "items" filter doesn't work anymore.
After digging a little I found the problem is trigerred because the collection in the Item class is a TreeSet (with an ArrayList, the problem vanishes).
The important difference between with and without springboot parent is the version of jackson databind (2.9.9 in spring-boot dependency management instead of 2.6.4 declared in squiggly pom).
Removing the spring-boot parent and adding an explicit dependency to jackson-databind:2.9.9 makes the problem also occur.

@fathzer Sorry for the late reply.  This turned out to be a bug in Jackson. I have submitted the following issue in the project:
https://github.com/FasterXML/jackson-databind/issues/2475

Looks like this issue will be fixed starting in Jackson version 2.10.1

Hi,
Jackson 2.10.1 was released ... and it fixed it.
Thanks a lot :-)


Test Framework

Test framework has been created using Karma/Mocha/Chai/Sinon/PhantomJS (SauceLabs for CI is in the works). Tests should be added under the test/ directory. Any test fixtures that you may create should be placed under test/fixtures/. A sample test file has been created verifying some behaviors of the Checkbox component (this is intentionally not complete -- I wanted to get this out to the community so that we can all start contributing tests). Tests will be written in ES6 and transpiled to ES5.
Test coverage is handled by Istanbul. However, Istanbul has a bug that is currently preventing it from generating HTML reports so we'll need to depend upon something that can process the LCOV reports (travis). Test results (pass/fail) and coverage summaries are dumped to the console (a detailed text report is also generated under the coverage\ directory).
To have Karma watch your test files, run npm run test-watch. To perform a single run, execute npm test. If we want to integrate Karma with Gulp, we can do that. I chose not to because it is trivial to run the tests and I did not want to incur the overhead of spinning up/down Karma's server.
Note that a browser-polyfill is required for PhantomJS 1.9.X.
Resolves #396

Sweet - thanks @jkruder :+1:

Let there be testing!

Something I found out the hard way: engine.io-client is a dependency of socket.io-client and engin.io-client has a dependency on an archived git repo. So anyone trying to use material-ui behind a firewall/company intranet may run into some issues. There is a PR to resolve the engine.io-client dependency issue. I can't say when it will be resolved, but hopefully soon.
A workaround: Stand up nexus (>2.10) or some other NPM hosting product and download/obtain the source for engine.io-client and the github dependency. Modify the github dep's package.json and give it a version number (something in the 0.0.X range or even 0.0.0-beta) and then add it to your hosted npm registry. Update the engine.io-client package.json so that the github dep is the version number you selected in the previous step. Upload engine.io-client into your npm registry.  That should temporarily address the issue with Karma until it is resolved permanently


Add Support WebRTC-HTTP ingestion protocol (WHIP) muxer & WebRTC-HTTP egress protocol (WHEP) demuxer with an In-App WHIP/WHEP server

To enhance Moblin’s streaming capabilities, implement support for WHIP and WHEP protocols. This feature will enable super low latency streaming, allowing devices to connect with minimal delay. Additionally, it will facilitate seamless integration of participants by enabling them to join streams via web URL links, which can be added as SCENE > WHEP sources in the stream. Supporting WHIP and WHEP will improve real-time interaction, reduce latency, and provide a more flexible and scalable streaming experience for mobile users.
Resources:
https://github.com/software-mansion/react-native-whip-whep
https://github.com/nativewaves/FFmpeg/pull/1#ref-issue-2322209276

feel free to help out adding it to moblin


:bug: fix(config): Make etherscan api keys partial

Description
The old zod validator required every etherscan network to be defined.

use z.object().partial() instead of z.record()

Testing
100% test coverage for this package
Additional Information

[ ] I read the contributing docs (if this is your first contribution)

Your ENS/address:

Current dependencies on/for this PR:

main

PR #372

PR #373   👈

PR #371

This comment was auto-generated by Graphite.


[docs] Select placeholder prop is omitted

Related page
https://mui.com/material-ui/api/select/
Kind of issue
Missing information
Issue description
Following issue https://github.com/mui/material-ui/pull/44502, the placeholder prop has been removed from the Select component for having no effect. While I'm inclined to disagree (MUI could render a placeholder, and the Select documentation event suggests on how to render one) the API documentation should at least reflect that decision.
As of the time of writing, the Select API documentation makes no mention of the omission of the placeholder prop. In fact, it refers to OutlinedInput:

Props of the OutlinedInput component are also available.

Which is where it is rightfully listed, leading developers to believe it should be available in Select.
I was ready to file a bug report, but having found that issue, I discovered placeholder was omitted intentionally. At first I went to the documentation to figure out what's what with this prop, but no mention of it was made, which lead me to create the issue you're reading now.
Context
No response
Search keywords: select placeholder

@thany thanks for reporting! We should update the docs to reflect this.

https://github.com/mui/material-ui/pull/44856 updated the Select docs page with a callout to point out that the placeholder prop is not available in Select.
The Select API docs page still needs to be updated, but it probably requires some docs infra work as the info for what props from the component it inherits from are part of an abstraction: https://mui.com/material-ui/api/select/#props


Question about resample

Can we just change
https://github.com/kohya-ss/sd-scripts/blob/cd19df49cd512e13ac90db115c424d19c0e8868a/library/train_util.py#L2209
To
cv2.INTER_LANCZOS4
on this line in this file so that the lanczos4 algorithm is used to downscale the size of the training images?

Why? Lanczos4 is slower (which matters when datasets can contain tens of thousands of images) and OpenCV's own documentation says INTER_AREA is preferred for downscaling:
INTER_AREA
Python: cv.INTER_AREA

resampling using pixel area relation. It may be a preferred method for image decimation, as it gives moire'-free results. But when the image is zoomed, it is similar to the INTER_NEAREST method.