appbar and fab with poor performance

Hi, I run a very simple demo page, default appbar and default fab,  running on a device 765G SoC and 6G mem. As above image, in profile mode, it dropped frame, and appbar time cost is very high.  I tested more than 10 times, avg time more than 3ms. (appbar and fab build), I want to known my fault, why is it so poor ?
Thank you

[✓] Flutter (Channel unknown, 2.0.4, on Mac OS X 10.15.7 19H2 darwin-x64, locale zh-Hans-CN)
• Flutter version 2.0.4 at /Users/wangkai/flutter_infra/code/googleflutter/flutter
• Framework revision b1395592de (3 weeks ago), 2021-04-01 14:25:01 -0700
• Engine revision 2dce47073a
• Dart version 2.12.2
• Pub download mirror https://dart-pub.byted.org
• Flutter download mirror https://storage.flutter-io.cn
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
• Android SDK at /Users/wangkai/Library/Android/sdk
• Platform android-30, build-tools 30.0.2
• ANDROID_HOME = /Users/wangkai/Library/Android/sdk
• Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS
• Xcode at /Applications/Xcode.app/Contents/Developer
• Xcode 12.4, Build version 12D4e
• CocoaPods version 1.10.0
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 4.1)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
[✓] IntelliJ IDEA Ultimate Edition (version 2019.3)
• IntelliJ at /Applications/IntelliJ IDEA.app
• Flutter plugin version 45.1.2
• Dart plugin version 193.7547
[✓] VS Code (version 1.55.1)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.21.0
[✓] Connected device (2 available)
• Redmi K30 5G (mobile) • 93888b2a • android-arm64  • Android 10 (API 29)
• Chrome (web)          • chrome   • web-javascript • Google Chrome 89.0.4389.128

@kyleada
What does 765G SoC mean ? is it a different device that your flutter doctor -v shows Redmi K30 5G (mobile) • 93888b2a • android-arm64 • Android 10 (API 29) ?
Also, are you using devtools to measure the performance ?
Can you upgrade to latest stable or master and compare the performance to see if it persists or not ?
Thanks.

pub global activate devtools
Package devtools is currently active at version 2.1.1.

Installed executable devtools.
Activated devtools 0.9.7+3.

yes, the device is Redmi K30 5G (mobile) • 93888b2a • android-arm64 • Android 10 (API 29). I mean its cpu model is qualcomm 765G

Can you upgrade to latest stable or master and compare the performance to see if it persists or not ?

@kyleada Did you try this ? Also can you provide a minimal code sample that shows this behavior ?

Thank.
I surely use the stable channel. the test project is https://github.com/kyleada/flutter_app_profile
the operate way is : click the fab in the homepage, and got a new page, I checked the timeline.
the appbar and fab build cost time span is not long again.
But I alway can saw the drop frame, it seems the page transition. It's a so simple app, how can I optimize it?

Thanks for the details and sample code @kyleada. I see similar behavior for appbar wherein the performance seems to be shrinking. Keeping this issue open for further investigation.

flutter doctor -v
[✓] Flutter (Channel stable, 2.0.5, on Mac OS X 10.15.4 19E2269 darwin-x64,
    locale en-GB)
    • Flutter version 2.0.5 at /Users/dhs/documents/fluttersdk/flutter
    • Framework revision adc687823a (4 days ago), 2021-04-16 09:40:20 -0700
    • Engine revision b09f014e96
    • Dart version 2.12.3

[!] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    ! CocoaPods 1.9.3 out of date (1.10.0 is recommended).
        CocoaPods is used to retrieve the iOS and macOS platform side's plugin
        code that responds to your plugin usage on the Dart side.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/platform-plugins
      To upgrade see
      https://guides.cocoapods.org/using/getting-started.html#installation for
      instructions.

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] VS Code (version 1.55.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.20.0

[✓] Connected device (3 available)
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729 • ios
      • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                               •
      web-javascript • Google Chrome 89.0.4389.128

! Doctor found issues in 1 category.

any update ? expect

still no update @darshankawar ?

Hello @kyleada. I have tested this issue on current stable and master channels. I am using a Vivo V21, with a MediaTek Dimensity 800U processor, 8 core and 8GB RAM, 90Hz display for reference.
On stable, there were a couple of frames that reached up to 23ms. Though going through the timelines, they are not from the AppBar as per your original issue, but rather from the AnimatedBuilder used internally by the page navigation animation. The specific screenshot I have shows a 44ms frame that shows this
(Stable) AnimatedBuilder being the cause of a 40ms+ frame

On master, we see much better performance. Flutter is able to keep up keeping the frames under 11ms (should be under 11ms for 90FPS). I've seen one frame go up to 16.6ms (should be under 16ms for 60FPS), otherwise it has been under 11ms at all times.
(Master) Flutter keeping up at 90FPS

If you can confirm on your device whether the performance issues have been addressed, I can close this issue as fixed
flutter doctor -v
[✓] Flutter (Channel master, 2.13.0-0.0.pre.942, on macOS 12.3.1 21E258 darwin-arm, locale en-US)
    • Flutter version 2.13.0-0.0.pre.942 at /Users/nabeelparkar/fvm/versions/master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 20fd55737d (3 hours ago), 2022-05-06 06:04:06 -0400
    • Engine revision a73b884510
    • Dart version 2.18.0 (build 2.18.0-91.0.dev)
    • DevTools version 2.13.0

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0-rc1)
    • Android SDK at /Users/nabeelparkar/Library/Android/sdk/
    • Platform android-32, build-tools 32.0.0-rc1
    • ANDROID_SDK_ROOT = /Users/nabeelparkar/Library/Android/sdk/
    • Java binary at: /opt/homebrew/opt/openjdk@11/bin/java
    • Java version OpenJDK Runtime Environment Homebrew (build 11.0.15+0)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • CHROME_EXECUTABLE = /Applications/Brave Browser.app/Contents/MacOS/Brave Browser

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).

[✓] IntelliJ IDEA Ultimate Edition (version 2022.1)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin version 67.0.4
    • Dart plugin version 221.5588

[✓] VS Code (version 1.66.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.40.0

[✓] Connected device (3 available)
    • V2050 (mobile)  •<PHONE_NUMBER>000NM • android-arm64  • Android 12 (API 31)
    • macOS (desktop) • macos           • darwin-arm64   • macOS 12.3.1 21E258 darwin-arm
    • Chrome (web)    • chrome          • web-javascript • Brave Browser <IP_ADDRESS>

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

Code sample used
lib/main.dart
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
import 'package:flutter_app_profile/movie/movie_list_page.dart';

void main() {
  debugProfileBuildsEnabled = true;
  debugProfilePaintsEnabled = true;
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
          // This is the theme of your application.
          //
          // Try running your application with "flutter run". You'll see the
          // application has a blue toolbar. Then, without quitting the app, try
          // changing the primarySwatch below to Colors.green and then invoke
          // "hot reload" (press "r" in the console where you ran "flutter run",
          // or simply save your changes to "hot reload" in a Flutter IDE).
          // Notice that the counter didn't reset back to zero; the application
          // is not restarted.
          primarySwatch: Colors.blue,
          pageTransitionsTheme: PageTransitionsTheme(builders: {
            TargetPlatform.android: CupertinoPageTransitionsBuilder(),
            TargetPlatform.iOS: CupertinoPageTransitionsBuilder(),
          })),
      home: MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key? key, required this.title}) : super(key: key);

  // This widget is the home page of your application. It is stateful, meaning
  // that it has a State object (defined below) that contains fields that affect
  // how it looks.

  // This class is the configuration for the state. It holds the values (in this
  // case the title) provided by the parent (in this case the App widget) and
  // used by the build method of the State. Fields in a Widget subclass are
  // always marked "final".

  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  int _counter = 0;

  @override
  Widget build(BuildContext context) {
    // This method is rerun every time setState is called, for instance as done
    // by the _incrementCounter method above.
    //
    // The Flutter framework has been optimized to make rerunning build methods
    // fast, so that you can just rebuild anything that needs updating rather
    // than having to individually change instances of widgets.
    return Scaffold(
      appBar: AppBar(
        // Here we take the value from the MyHomePage object that was created by
        // the App.build method, and use it to set our appbar title.
        title: Text(widget.title),
      ),
      body: Center(
        // Center is a layout widget. It takes a single child and positions it
        // in the middle of the parent.
        child: Column(
          // Column is also a layout widget. It takes a list of children and
          // arranges them vertically. By default, it sizes itself to fit its
          // children horizontally, and tries to be as tall as its parent.
          //
          // Invoke "debug painting" (press "p" in the console, choose the
          // "Toggle Debug Paint" action from the Flutter Inspector in Android
          // Studio, or the "Toggle Debug Paint" command in Visual Studio Code)
          // to see the wireframe for each widget.
          //
          // Column has various properties to control how it sizes itself and
          // how it positions its children. Here we use mainAxisAlignment to
          // center the children vertically; the main axis here is the vertical
          // axis because Columns are vertical (the cross axis would be
          // horizontal).
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Text(
              'You have pushed the button this many times:',
            ),
            Text(
              '$_counter',
              style: Theme.of(context).textTheme.headline4,
            ),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: () {
          Navigator.push(
              context,
              MaterialPageRoute(
                  builder: (context) => new EventMovieListPage()));
        },
        tooltip: 'Increment',
        child: Icon(Icons.add),
      ), // This trailing comma makes auto-formatting nicer for build methods.
    );
  }
}

lib/movie/movie_item_model.dart

class MovieItem {
  String name;
  bool _selected = false;

  MovieItem(this.name);

  bool get selected => _selected;

  set selected(bool value) {
    _selected = value;
  }
}

lib/movie/movie_list_page.dart
import 'dart:convert';
import 'package:flutter/material.dart';
import 'package:flutter/widgets.dart';
import 'package:flutter_app_profile/movie/movie_item_model.dart';

class EventMovieListPage extends StatefulWidget {
  @override
  State<StatefulWidget> createState() {
    return EventMovieListPageState();
  }
}

class EventMovieListPageState extends State<EventMovieListPage> {
  List<MovieItem> movieList = [];

  void _fetchData() async {
    movieList = await Future.delayed(Duration(milliseconds: 5), (){
      List<MovieItem> temp = [];
      String body = '{ "list": [{ "name": "Hello, shanghai"}, {"name": "hello, beijing"},{"name": "hello, tianjin"}, {"name": "hello, chengdu"}]}';
      Map<String, dynamic> parsedJson = json.decode(body);
      for (int i = 0; i < parsedJson['list'].length; i++) {
        Map<String, dynamic> movieItemJson = parsedJson['list'][i];
        MovieItem result = MovieItem(movieItemJson['name'] as String);
        temp.add(result);
      }
      return temp;
    });
    setState(() {
    });
  }

  @override
  void initState() {
    super.initState();
    _fetchData();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
        appBar: AppBar(
          title: Text('Popular Movies'),
        ),
        body: buildList(),
        floatingActionButton: FloatingActionButton(
          child: Icon(Icons.list),
          onPressed: () async {
          },
        ));
  }

  Widget buildList() {
    if(movieList.isEmpty) {
       return Text("nothing");
    }
    return ListView.builder(
        itemCount: movieList.length,
        itemBuilder: (BuildContext context, int index) {
          MovieItem movieItem = movieList[index];
          return ListTile(
            title: Text(movieItem.name),
            trailing: new MovieItemWidget(movieItem),
          );
        });
  }

}

class MovieItemWidget extends StatefulWidget {

  MovieItem movieItem;

  MovieItemWidget(this.movieItem);

  @override
  State<StatefulWidget> createState() {
    return MovieItemWidgetState(movieItem);
  }
}

class MovieItemWidgetState extends State<MovieItemWidget> {

  MovieItem movieItem;

  MovieItemWidgetState(this.movieItem);

  @override
  Widget build(BuildContext context) {
    return GestureDetector(
      onTap: () {
        movieItem.selected = !movieItem.selected;
        // TODO
        setState(() {});
      },
      child: Icon(movieItem.selected ? Icons.star : Icons.star_border),
    );
  }
}


Update dependencies and workflows

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (53e481b) to head (148011d).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #338   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           96        96
  Lines         3228      3228
  Branches       242       242
=========================================
  Hits          3228      3228

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Refactor Memory methods to handle Temporary segment cases

Merge after #471
Refactor Memory methods to handle Temporary segment cases

Refactor Memory.get method to get values from Temporary Segments
Refactor Memory.insert method to insert values in Temporary Segments
Refactor fn from_relocatable_to_indexes
Add Memory.relocation_rules field

Checklist

[ ] Linked to Github Issue
[x] Unit tests added
[ ] Integration tests added.
[ ] This change requires new documentation.

[ ] Documentation has been added/updated.

Codecov Report

Merging #482 (e88b01d) into relocatable_segment_index_to_isize (056c887) will increase coverage by 0.53%.
The diff coverage is 100.00%.

@@                          Coverage Diff                           @@
##           relocatable_segment_index_to_isize     #482      +/-   ##
======================================================================
+ Coverage                               91.16%   91.70%   +0.53%
======================================================================
  Files                                      61       48      -13
  Lines                                    9648     2964    -6684
======================================================================
- Hits                                     8796     2718    -6078
+ Misses                                    852      246     -606

Impacted Files
Coverage Δ

src/utils.rs
100.00% <100.00%> (+0.47%)
:arrow_up:

src/vm/vm_memory/memory.rs
95.12% <100.00%> (-1.94%)
:arrow_down:

src/vm/vm_memory/memory_segments.rs
95.45% <100.00%> (-3.35%)
:arrow_down:

src/cairo_run.rs
72.91% <0.00%> (-14.19%)
:arrow_down:

src/serde/deserialize_program.rs
79.72% <0.00%> (-13.67%)
:arrow_down:

src/vm/runners/builtin_runner/bitwise.rs
68.57% <0.00%> (-13.10%)
:arrow_down:

src/vm/runners/builtin_runner/ec_op.rs
78.33% <0.00%> (-12.95%)
:arrow_down:

src/vm/runners/builtin_runner/output.rs
78.57% <0.00%> (-10.72%)
:arrow_down:

src/vm/trace/trace_entry.rs
83.33% <0.00%> (-10.42%)
:arrow_down:

src/serde/deserialize_utils.rs
81.53% <0.00%> (-9.64%)
:arrow_down:

... and 47 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


fix int64 bug (for older numpy versions)

Fixes #149

Coverage increased (+0.007%) to 68.054% when pulling 413593bde7288e53b2464a3806652dbac36b2ab4 on gertjanvanzwieten:fixint64bug into f97312af8a5ec0b0c0a43ed48b40aaab410dde98 on nutils:master.


any suggestion on converting shufflenet structure in TF to pytorch

Hi! thanks for your work! I'm currently facing an issue that I want to convert a shufflenet pretrained model in TF to pytorch format, as the network contains group convolution and depth convolution as well as concat op, also it's in NCHW format, could you give some suggestion or if it is currently possible in MMdnn? Thanks !

Hi, @MrWanter could you provide the TF model files (ckpt ckpt.data ckpt.meta or .pb)?


add step for installing visual studio preview

currently the preqreqs say user must have Visual Studio version later than 17.3.0 Preview 2.
They need to have Visual Studio Preview installed, which is a separate installation from Visual Studio. So, removing the prereq and adding installing Visual Studio Preview as the first step.

Looks like I don't have permission to request reviewers so @savannahostrowski @alexwolfmsft would y'all mind taking a look at this?

@alexwolfmsft

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


2 issues and an enhancement project

Hi Anthony,
First of all, thanks for the amazing work done with ContentTools. It's really a nice library.
I am currently using it on a Laravel based website, linked with SQL db. I'm gonna need to develop a way to add custom containers calling for external content types within the edited content so I'll keep you posted if I manage to achieve something (or if I encounter problems) and if I can release something you could reuse.
Right now I have two remaining bugs on my integration. One I simply can't figure out (after hours trying) and the other I haven't yet dived too deeply into.

First, on the crop function, I pass the "crop" array to my Controller and try to use it with Intervention Image (which is a pain, since it's handling crop in a totally different manner). The problem is I just can't figure out how the crop array works. From my tests, it seems to have 7 values (identified from 0 to 6), they can be an integer (between 0 and 9), a colon or a coma... And depending on the position of the cropmarks, some of the values become comas and other become integer. I have been completely unable to find a logic behind it, and from your tutorial, I would expect to find only 4 non-integer values between 0 and 1 (corresponding to top, left, bottom and right normalized values).
I should be able to convert the normalized values to the inputs expected by Intervention Image using simple maths... But I would need your help in understanding how the 'crop' output of the imageUploader.save function is built.

Second, and it might be a simple thing since I haven't looked into it too much, I have problems when reusing the editor a second time right after a first edit. For example, if I save the text, then reopen and save again, the editor delete the entire content. Same for the image: if I rotate an image, everything goes fine, but if I do it a second time right afterwards, it doesn't work. Also, the edit panel doesn't disappear when I save. Somehow it feels like everything's working fine, but something stays open after saving successfully and fucks things up afterwards. Maybe my controller should return a value when the AJAX request has been treated? But I've found nothing about it on your tutorials.

Below, the code for my editor.js (it's pretty much standard from your tutorials). Thanks in advance for your help.
Simon
/*! ContentTools v0.1.0, initialisation scripts */

window.addEventListener('load', function() {
	var editor;
	ContentTools.StylePalette.add([
		new ContentTools.Style('Author', 'author', ['p'])
	]);
	ContentTools.IMAGE_UPLOADER = imageUploader;
});

// Patch the editor's start/stop methods to support auto-save
var editorCls = ContentTools.EditorApp.getCls();

editor = ContentTools.EditorApp.get();
editor.init('*[data-editable]', 'data-name');

// Listen for save events
editor.bind('save', function (regions) {
    var name, payload, xhr;

    // Set the editor as busy while we save our changes
    this.busy(true);

    // Collect the contents of each region into a FormData instance
    payload = new FormData();
    for (name in regions) {
        if (regions.hasOwnProperty(name)) {
            payload.append(name, regions[name]);
        }
    }

    // Send the update content to the server to be saved
    function onStateChange(ev) {
        // Check if the request is finished
        if (ev.target.readyState == 4) {
            editor.busy(false);
            if (ev.target.status == '200') {
                // Save was successful, notify the user with a flash
                new ContentTools.FlashUI('ok');
            } else {
                // Save failed, notify the user with a flash
                new ContentTools.FlashUI('no');
            }
        }
    };

    xhr = new XMLHttpRequest();
    xhr.addEventListener('readystatechange', onStateChange);
    xhr.open('POST', '/www/'+contenttype+'/'+chapterid);
    xhr.setRequestHeader('X-CSRF-TOKEN', $("#token").attr('content'));
    xhr.send(payload);
});

editor.start = function () {

    // Call save every 30 seconds
    function autoSave() {
        editor.save(true, 'autoSave');
    };
    this.autoSaveTimer = setInterval(autoSave, 30 * 1000);

    // Start the editor
    editorCls.prototype.start.call(this);
};

editor.stop = function () {
    // Stop the autosave
    clearInterval(this.autoSaveTimer);

    // Stop the editor
    editorCls.prototype.start.call(this);
}

function imageUploader(dialog) {
     var image, xhr, xhrComplete, xhrProgress;

    // Set up the event handlers

     dialog.bind('imageUploader.cancelUpload', function () {
        // Cancel the current upload

        // Stop the upload
        if (xhr) {
            xhr.upload.removeEventListener('progress', xhrProgress);
            xhr.removeEventListener('readystatechange', xhrComplete);
            xhr.abort();
        }

        // Set the dialog to empty
        dialog.state('empty');
    });

	 dialog.bind('imageUploader.clear', function () {
        // Clear the current image
        dialog.clear();
        image = null;
    });

	dialog.bind('imageUploader.fileReady', function (file) {
        // Upload a file to the server
        var formData;

        // Define functions to handle upload progress and completion
        xhrProgress = function (ev) {
            // Set the progress for the upload
            dialog.progress((ev.loaded / ev.total) * 100);
        }

        xhrComplete = function (ev) {
            var response;

            // Check the request is complete
            if (ev.target.readyState != 4) {
                return;
            }

            // Clear the request
            xhr = null
            xhrProgress = null
            xhrComplete = null

            // Handle the result of the upload
            if (parseInt(ev.target.status) == 200) {
                // Unpack the response (from JSON)
                response = JSON.parse(ev.target.responseText);

                // Store the image details
                image = {
                    size: response.size,
                    url: response.url
                    };

                // Populate the dialog
                dialog.populate(image.url, image.size);

            } else {
                // The request failed, notify the user
                new ContentTools.FlashUI('no');
            }
        }

        // Set the dialog state to uploading and reset the progress bar to 0
        dialog.state('uploading');
        dialog.progress(0);

        // Build the form data to post to the server
        formData = new FormData();
        formData.append('image', file);

        // Make the request
        xhr = new XMLHttpRequest();
        xhr.upload.addEventListener('progress', xhrProgress);
        xhr.addEventListener('readystatechange', xhrComplete);
		xhr.open('POST', '/www/upload-image');
		xhr.setRequestHeader('X-CSRF-TOKEN', $("#token").attr('content'));
        xhr.send(formData);
    });

	function rotateImage(direction) {
        // Request a rotated version of the image from the server
        var formData;

        // Define a function to handle the request completion
        xhrComplete = function (ev) {
            var response;

            // Check the request is complete
            if (ev.target.readyState != 4) {
                return;
            }

            // Clear the request
            xhr = null
            xhrComplete = null

            // Free the dialog from its busy state
            dialog.busy(false);

            // Handle the result of the rotation
            if (parseInt(ev.target.status) == 200) {
                // Unpack the response (from JSON)
                response = JSON.parse(ev.target.responseText);

                // Store the image details (use fake param to force refresh)
                image = {
                    size: response.size,
                    url: response.url + '?_ignore=' + Date.now()
                    };

                // Populate the dialog
                dialog.populate(image.url, image.size);

            } else {
                // The request failed, notify the user
                new ContentTools.FlashUI('no');
            }
        }

        // Set the dialog to busy while the rotate is performed
        dialog.busy(true);

        // Build the form data to post to the server
        formData = new FormData();
        formData.append('url', image.url);
        formData.append('direction', direction);

        // Make the request
        xhr = new XMLHttpRequest();
        xhr.addEventListener('readystatechange', xhrComplete);
		xhr.open('POST', '/www/rotate-image');
		xhr.setRequestHeader('X-CSRF-TOKEN', $("#token").attr('content'));
        xhr.send(formData);
    }

    dialog.bind('imageUploader.rotateCCW', function () {
        rotateImage('CCW');
    });

    dialog.bind('imageUploader.rotateCW', function () {
        rotateImage('CW');
    });

	dialog.bind('imageUploader.save', function () {
        var crop, cropRegion, formData;

        // Define a function to handle the request completion
        xhrComplete = function (ev) {
            // Check the request is complete
            if (ev.target.readyState !== 4) {
                return;
            }

            // Clear the request
            xhr = null
            xhrComplete = null

            // Free the dialog from its busy state
            dialog.busy(false);

            // Handle the result of the rotation
            if (parseInt(ev.target.status) === 200) {
                // Unpack the response (from JSON)
                var response = JSON.parse(ev.target.responseText);

                // Trigger the save event against the dialog with details of the
                // image to be inserted.
                dialog.save(
                    response.url,
                    response.size,
                    {
                        'alt': response.alt,
                        'data-ce-max-width': image.size[0]
                    });

            } else {
                // The request failed, notify the user
                new ContentTools.FlashUI('no');
            }
        }

        // Set the dialog to busy while the rotate is performed
        dialog.busy(true);

        // Build the form data to post to the server
        formData = new FormData();
        formData.append('url', image.url);

        // Set the width of the image when it's inserted, this is a default
        // the user will be able to resize the image afterwards.
        formData.append('width', 500);

        // Check if a crop region has been defined by the user
        if (dialog.cropRegion()) {
            formData.append('crop', dialog.cropRegion());
        }

        // Make the request
        xhr = new XMLHttpRequest();
        xhr.addEventListener('readystatechange', xhrComplete);
		xhr.open('POST', '/www/insert-image');
		xhr.setRequestHeader('X-CSRF-TOKEN', $("#token").attr('content'));
        xhr.send(formData);
    });
}
	`
```

Hi @simonmuys thanks for the kind comments and it's great to hear investing time in CT :)
So re your first question, you are correct in assuming that the output of cropRegion should be a 4 item array, so these are the 2 relevant snippets of code that return crop marks.
ContentTools.ImageDialog.cropRegion: () ->
    # Return the defined crop-region (top, left, bottom, right), values are
    # normalized to the range 0.0 - 1.0. If no crop region is defined then
    # the maximum region will be returned (e.g [0, 0, 1, 1])
    if @_cropMarks
        return @_cropMarks.region()

    return [0, 0, 1, 1]

CropMarksUI.region: () ->
    # Return the crop region (top, left, bottom, right), values are
    # normalized to the range 0.0 - 1.0.
    return [
<EMAIL_ADDRESS>/ @_bounds[1],
<EMAIL_ADDRESS>/ @_bounds[0],
<EMAIL_ADDRESS>/ @_bounds[1],
<EMAIL_ADDRESS>/ @_bounds[0]
        ]

https://github.com/GetmeUK/ContentTools/blob/master/src/scripts/ui/dialogs/image.coffee#L41
https://github.com/GetmeUK/ContentTools/blob/master/src/scripts/ui/dialogs/image.coffee#L326

As you can see you should be returned 4 numbers between 0.0 and 1.0 in the order (top, left, bottom, right). If this isn't what's being returned then there's an issue, could you console.log(dialog.cropRegion()) and post the result here so I can see what you're being returned.
Re. your second question this is definitely not the expected behaviour and you should certainly be able to edit more than once, this sounds like an issue is being raised during the saving process - can you post any output in the JS console of your browser for me to review please.

Hi Anthony, just a heads up on this issue:
First, you're entirely right saying all the problems come from my poor handling in back-end :).
I've solved the problem of the double crop quite easily: I was using pathinfo but since your JS adds a parameter to the URL, I needed to parse it first to escape the parameter. That didn't take long.
Second problem, on the crop, logging the $crop array returned the correct values, as you'd have guessed. Back-end, I logged everything little by little and it resulted the problem was I retrieve an array of characters instead of a proper array of the four values. So using $crop[0] or $crop[1] only returning the corresponding character and not the whole value. For example, if my crop array was [0.234, 0.5, 1, 1], $crop[0], $crop[1], $crop[2], $crop[3] were returning respectively '0', '.', '2' and '3'.  I guess Laravel Request façade must have something wrong with floats being passed through a formData format.
What I did is to explode the string into a proper array. It's dirty work, but it made the trick.
Afterwards, the integration with Image Intervention library was headache but nothing maths can't solve. I'll publish the full Laravel Controller so feel free to use it in a tutorial if need be (it might not be perfect but it works for me!).
Now, for the last problem, I've had time to look a bit more into it and the problem was not on second save but on saving while no modification had been made. But you had provided a piece of code for that in your tutorial so it's not been too hard to solve.
I have now just one remaining problem with the edit toolbar that remained active when editor was shut or saved was performed. I tried to solve that using widget.hide() but it seems each time I use it it breaks the reste of the saving process.
I'll now concentrate on building a proper stylesheet and a tool for adding token calling for external content.
Thanks again for your help!
Simon
PS: this is the current Laravel Controller
`<?php namespace App\Http\Controllers;
use File;
use Validator;
use Redirect;
use App\User;
use Intervention\Image\ImageManagerStatic as Image;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Input;
use Illuminate\Support\Facades\Auth;
use App\Http\Controllers\Controller;
class UploadsController extends Controller {
/**
 * Upload by AJAX.
 *
 * @param $request
 * @return Response
 */
public function upload(Request $request)
{
	$user = User::find(Auth::user()->id);
	$imgid = count(glob(public_path().'/temp/'.$user->slug.'-image-*.*'))+1;

	$image = $request->file('image');
	$imagename = $user->slug.'-image-'.$imgid.'.'.$image->getClientOriginalExtension();
	$request->file('image')->move(public_path().'/temp/', $imagename);

	$size = getimagesize(public_path().'/temp/'.$imagename);
	$url = asset('temp/'.$imagename);

	return response()->json(['size' => $size, 'url' => $url]);
}

/**
 * Rotate image by AJAX.
 *
 * @param $request
 * @return Response
 */
public function rotate(Request $request)
{
	$origin = $request->input('url');

	$user = User::find(Auth::user()->id);
	$imgid = count(glob(public_path().'/temp/'.$user->slug.'-image-*.*'))+1;
	$imagename = $user->slug.'-image-'.$imgid.'.'.pathinfo(parse_url($origin)['path'], PATHINFO_EXTENSION);

	if ($request->input('direction') === 'CW') {
		$img = Image::make($origin)->rotate(-90);
	}
	elseif ($request->input('direction') === 'CCW') {
		$img = Image::make($origin)->rotate(90);
	}
	$img->save(public_path().'/temp/'.$imagename);

	$size = getimagesize(public_path().'/temp/'.$imagename);
	$url = asset('temp/'.$imagename);

	return response()->json(['size' => $size, 'url' => $url]);
}

/**
 * Insert image by AJAX.
 *
 * @param $request
 * @return Response
 */
public function insert(Request $request)
{
	$origin = $request->input('url');

	$user = User::find(Auth::user()->id);
	$imgid = count(glob(public_path().'/media/'.$user->slug.'-image-*.*'))+1;
	$imagename = $user->slug.'-image-'.$imgid.'.'.pathinfo(parse_url($origin)['path'], PATHINFO_EXTENSION);

	if ($request->input('crop') !== null) {
		$crop = explode(",", $request->input('crop'));

		$width = Image::make($origin)->width();
		$height = Image::make($origin)->height();

		$cropparam1 = round(($crop[3]-$crop[1])*$width);
		$cropparam2 = round(($crop[2]-$crop[0])*$height);
		$cropparam3 = round($crop[1]*$width);
		$cropparam4 = round($crop[0]*$height);

		$img = Image::make($origin)->crop($cropparam1, $cropparam2, $cropparam3, $cropparam4);
	}
	else {
		$img = Image::make($origin);
	}

	$img->save(public_path().'/media/'.$imagename);

	$size = getimagesize(public_path().'/media/'.$imagename);
	$url = asset('media/'.$imagename);
	$alt = '';

	return response()->json(['size' => $size, 'url' => $url, 'alt' => $alt]);
}

}
`

Hi Anthony,
I'm struggling with creating new tools with your API.
Right now, I'm trying nothing more fancy than creating a div wrapper with a class around the current paragraph. Since you have no tool that works this way, I wonder if it's possible using your Tool creation tutorial. But from what I see of your API, it should be possible.
I've been able to create a div component using ContentEdit.Element, but since I want it as a wrapper, I guess I should use ElementCollection instead.
Yet with the code below, I don't view any new div created.
I know I haven't yet added the element text to the supposedly created node but since I have erase everything that should not be useful for the div to appear.
Could you point me to what I'm doing wrong again?
Thanks in advance,
Simon
class ContentTools.Tools.Dialogue extends ContentTools.Tool

    # Convert the current text block to a dialogue block (e.g
foo

        # Return true if the tool can be applied to the current
        # element/selection.
        return element != undefined

    @apply: (element, selection, callback) ->
        # Apply the tool to the current element

        # Create a new dialogue element using the current elements content
        collection = new ContentEdit.ElementCollection(
            'div', {class:'dialogue'}
            )
        paragraph = new ContentEdit.Text('p')
        collection.attach(paragraph)

        # Get the element region or its closest one
        if element.parent().type() != 'Region'
            element = element.closest (node) ->
            return node.parent().type() is 'Region'
        region = element.parent()
        region.attach(collection)

        element.focus()

        callback(true)

ContentTools.DEFAULT_TOOLS[0].push('dialogue')

Hi @simonmuys,
The element collection class is actually designed to be inherited from rather than used directly (e.g the List element or Table element), having said that this does work there just a few minor issues with your code (you're very close):

    class ContentTools.Tools.Dialogue extends ContentTools.Tool

        # Convert the current text block to a dialogue block (e.g
foo

            # Return true if the tool can be applied to the current
            # element/selection.
            return element != undefined

        @apply: (element, selection, callback) ->
            # Apply the tool to the current element

            # Create a new dialogue element using the current elements content
            collection = new ContentEdit.ElementCollection(
                'div', {class:'dialogue'}
                )
            paragraph = new ContentEdit.Text('p', {}, 'My custom wrapped p tag')
            collection.attach(paragraph)

            # Get the element region or its closest one
            if element.parent().type() != 'Region'
                element = element.closest (node) ->
                    return node.parent().type() is 'Region'

            region = element.parent()
            region.attach(collection, element.parent().children.indexOf(element))

            paragraph.focus()

            callback(true)

    ContentTools.DEFAULT_TOOLS[0].push('dialogue')

So here's what I changed:

I updated the region.attach call so that the new element is inserted before the selected element (you were inserting it at the end of the region (which is I assume why you couldn't see it).
I changed the focus to event to be called against the paragraph (rather than the selected element when the tool is applied).

Hope that helps - Ant

Thanks a lot Anthony, this works perfectly. I've changed a bit the code now for attaching the initial text element and returning the selection.
I've started working on the token system. It's basically an insert of a  element with a given class (depending on the tool). I'm using the video modal to give the user the opportunity to select the right content to call. So far everything's working fine so I should be able to do what I need.
I'm still struggling to pass the array of options from my Laravel DB to the coffee script of the modal and I still have to verify I can easily create a selectbox with options but I'm quite confident on both points.
I'll share the code once it's done (or if I'm blocked somehow).
Thanks again for the plugin and congrats man. ContentTools is really an amazing library to work with.

Hi Anthony,
I've almost managed to finalize the token system.
Here is how I've done it. First, I've created a new tool:
class ContentTools.Tools.Character extends ContentTools.Tool
 # Create a character block (e.g <token class="character"></token) that will be parsed and replace by the proper content

    ContentTools.ToolShelf.stow(@, 'character')

    @label = 'Character'
    @icon = 'character'
    @tagName = 'token'

    @canApply: (element, selection) ->
        # Return true if the tool can be applied to the current
        # element/selection.
        return true

    @apply: (element, selection, callback) ->

        # If supported allow store the state for restoring once the dialog is
        # cancelled.
        if element.storeState
            element.storeState()

        # Set-up the dialog
        app = ContentTools.EditorApp.get()

        # Modal
        modal = new ContentTools.ModalUI()

        # Dialog
        dialog = new ContentTools.CharacterDialog()

        # Support cancelling the dialog
        dialog.bind 'cancel', () =>
            dialog.unbind('cancel')

            modal.hide()
            dialog.hide()

            if element.restoreState
                element.restoreState()

            callback(false)

        # Support saving the dialog
        dialog.bind 'save', (characterID) =>
            dialog.unbind('save')

            if characterID
                # Create a new dialogue element using the current elements content
                collection = new ContentEdit.ElementCollection(
                    'token', {class:'character', tokenType:'character', data:characterID}
                )
                collection.can('drag', true)
                collection.can('remove', true)

                # Find insert position
                [node, index] = @_insertAt(element)
                node.parent().attach(collection, index)

            else
                # Nothing to do restore state
                if element.restoreState
                    element.restoreState()

            modal.hide()
            dialog.hide()

            callback(characterID != '')

        # Show the dialog
        app.attach(modal)
        app.attach(dialog)
        modal.show()
        dialog.show()

ContentTools.DEFAULT_TOOLS[2].push('character')

I've also created a specific dialog tool with a select box for passing the right content to the token (based on the video dialog tool). I've escaped the preview for simplification:
class ContentTools.CharacterDialog extends ContentTools.DialogUI

    # A dialog to support inserting a character

    constructor: ()->
        super('Insert character')

    mount: () ->
        # Mount the widget
        super()

        # Update dialog class
        ContentEdit.addCSSClass(@_domElement, 'ct-character-dialog')

        # Update view class
        ContentEdit.addCSSClass(@_domView, 'ct-character-dialog__preview')

        # Add controls
        domControlGroup = @constructor.createDiv(['ct-control-group'])
        @_domControls.appendChild(domControlGroup)

        # Input
        @_domInput = document.createElement('select')
        @_domInput.setAttribute('id', 'selectcharacter')
        @_domInput.setAttribute('class', 'ct-video-dialog__input form-control')
        @_domInput.setAttribute('name', 'url')
        domControlGroup.appendChild(@_domInput)

        characters = jsvar.characters
        for i in characters
            do ->
                for j in i
                    domOption = document.createElement('option')
                    domOption.setAttribute('value', j['id'])
                    domOption.innerHTML = j['name']
                    document.getElementById('selectcharacter').appendChild(domOption)

        # Insert button
        @_domButton = @constructor.createDiv([
            'ct-control',
            'ct-control--text',
            'ct-control--insert'
            ])
        @_domButton.textContent = ContentEdit._('Insert')
        domControlGroup.appendChild(@_domButton)

        # Add interaction handlers
        @_addDOMEventListeners()

    save: () ->
        # Save the character. This method triggers the save method against the
        # dialog allowing the calling code to listen for the `save` event and
        # manage the outcome.

        # Attempt to parse a character embed URL
        characterID = @_domInput.value.trim()
        @trigger('save', characterID)

    show: () ->
        # Show the widget
        super()

        # Once visible automatically give focus to the link input
        @_domInput.focus()

    unmount: () ->
        # Unmount the component from the DOM

        # Unselect any content
        if @isMounted()
            @_domInput.blur()

        super()

        @_domButton = null
        @_domInput = null

    # Private methods

    _addDOMEventListeners: () ->
        # Add event listeners for the widget
        super()

        # Add support for saving the character whenever the `return` key is pressed
        # or the button is selected.

        # Input
        @_domInput.addEventListener 'keypress', (ev) =>
            if ev.keyCode is 13
                @save()

        # Button
        @_domButton.addEventListener 'click', (ev) =>
            ev.preventDefault()

            # Check the button isn't muted, if it is then the character ID field
            # isn't populated.
            cssClass = @_domButton.getAttribute('class')
            if cssClass.indexOf('ct-control--muted') == -1
                @save()

I then have a JS parser for adding content to the token through a JQuery load control:
$(function() {

	/*---------------------------------------*/
    /*  AJAX PARSER FOR TOKENS
    /*---------------------------------------*/
	$(document).ready(function(){
		$( "token" ).each(function() {
			var tokenType = $( this ).attr('tokenType');
			var tokenID = $( this ).attr('data');
			$( this ).load( "/tokenize/"+tokenType+"/"+tokenID );
		});
	});
});

The parser calls a Laravel controller that adds a blade view with the proper content inside the token div (I don't show the code for that since it's both pretty straightforward and very specific to my website).
I have a lot of work to do on the editor to be entirely what I need it to be but most of the difficulties are now behind since it's mostly replication of functionalities.
I still have two core issues I haven't figured out:

first, I just can't figure out how to have the drag/drop and remove behaviors on my token divs... I'm using an ElementCollection instead of an Element so it should be why the behavior is not set from the beginning and also why the collection.can('drag', true) method does not work. But when I switch to a regular Element, using the exact same syntaxe for the attributes object, it doesn'twork. Also, since ElementCollection inherits from Element, shouldn't it receive also the "can" method?

second, I still have a problem reediting the newly created div from the dialogue tool you've helped me build last time (I guess it will be the same for the token part). I can edit, toggle the behavior that creates a new div with the proper classes and appending the selected content, I can save and everything is fine. But when I try to edit again, I can't select the element anymore. My guess would be to say it's because they are ElementCollections instead of regular Elements, but I don't know where or how to specify these div can be selected and edited. Could you point out to the thing I've missed?

I hope the code I'm sharing can give you some ideas for new implementations on the core of your plugin. I do believe that adding tokens with customized data that can be parsed to add external content is an interesting feature to have in the API.
Thanks again,
Simon

@simonmuys drop behaviour must be specified for your new element, for example look at how this is done for the Text element:
https://github.com/GetmeUK/ContentEdit/blob/master/src/scripts/text.coffee#L451
Have you done this? If not your element can't be dropped or dropped onto by another element.
Might be a pain but is is it possible to see this up and running somewhere and your full code? It's difficult to grasp what you may be missing without.

Hi @anthonyjb .
No pb to give you an access to the WIP. Can you send me your email address at<EMAIL_ADDRESS>I'll send you the details for you to see what I've been doing and how it looks so far.
Best regards,
Simon

@simonmuys done, sorry for the late reply I've been away.


Merge feature/pip install to master

Move Python projects to pyproject.toml. Removes the legacy
behavior of calling python on setup.py, which is no longer the
advised way of creating python packages, per PEP 517.
Signed-off-by: Tim Whisonant<EMAIL_ADDRESS>

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 65.535%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
13141

Relevant Lines:
20052

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 65.535%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
13141

Relevant Lines:
20052

💛  - Coveralls


Setup Lock for HA upcomming versions

Hello,
in the logs I get the following warning that can cause problems in later versions:
Logger: homeassistant.helpers.frame
Source: helpers/frame.py:155
First occurred: 14:17:21 (1 occurrences)
Last logged: 14:17:21

Detected code that calls async_forward_entry_setup for integration ampere_modbus with title: Ampere.StoragePro_Com and entry_id: 01JAQK961QYMYY4AS6SWX634Z4, during setup without awaiting async_forward_entry_setup, which can cause the setup lock to be released before the setup is done. This will stop working in Home Assistant 2025.1. Please report this issue.

Thank you for this hint.
I have already removed the obsolete function. Coming with the next version.


error on line 447 and no GALILEO constellation definition

Hi, there is no definition of the Galileo constellation type on the top (CONSTELLATION_GALILEO = 6) and on line 447 there is an error. I guess it should be: svid = 'J{0:02d}'.format(prn)

Hi,
This issue has been fixed. Could you try again


Data collection practices - can we control them for Teams?

Teams/data-collection-practices.md - It mentions Teams, but really just talks about controlling them for Skype for Business on-premises. Are there controls for Teams?

@StainlessSteele thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


Add RollbackUnlessCommitted method to prevent leaking transactions

Describe the feature
Add a method to DB which can rollback a transaction only in the case it has not already been committed called RollbackUnlessCommitted

Motivation
I think this is a very much needed feature in the case of this code:
   tx := DB.Begin()

   err1 := someFunc1()
   if err1 != nil {
      tx.Rollback()
      return err1
   }

   err2 := someFunc2()
   if err2 != nil {
      tx.Rollback()
      return err2
   }

   err3 := someFunc3()
   if err3 != nil {
                           // <------- oops forgot to call tx.Rollback() here
      return err3
   }

   tx.Commit()
   return nil

It would be VERY helpful to have this method:
   tx := DB.Begin()
   defer tx.RollbackUnlessComitted() // <---- Only need to call once

   err1 := someFunc1()
   if err1 != nil {
      return err1
   }

   err2 := someFunc2()
   if err2 != nil {
      return err2
   }

   err3 := someFunc3()
   if err3 != nil {
      return err3
   }

   tx.Commit()
   return nil

Related Issues
This was previously mentioned in this PR but was rejected. I think the motivation behind it wasn't so clear & that's why it was rejected. This would be a very helpful feature and would prevent end users of Gorm from accidentally leaking transactions by forgetting to call tx.Rollback() (has happened to me several times).

Just wanted to understand why this was closed/rejected? Is it a personal preference that you don't think this method is needed? If so, that's fine just wondering.

won't add this, and this has been discussed many times already.


Cliffwalking should terminate when running into cliff

Description
The Cliffwalking environment as inspired from Richard and Sutton's RL book does not currently terminate when the agent walks into the Cliff. However, this is not the intended behaviour as outlined by Sutton. Running into the cliff should be considered a terminal state with a -100 reward.
I know this to be the case because I tried re-creating the graphs in the RL book. Only by terminating at the cliff can I reproduce the results by Sutton.
Type of change

[x] Bug fix (non-breaking change which fixes an issue)

Checklist:

[x] I have run the pre-commit checks with pre-commit run --all-files (see CONTRIBUTING.md instructions to set it up)
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

Thanks for the PR, good spot.
For the future, what page number in the book or link to the website page where it says this?

Hi @pseudo-rnd-thoughts,
This is in page number 132 of Sutton's RL book. Here's a screenshot of the page:

Technically the environment does what the text says

"Reward is −1 on all transitions except those into the region marked “The Cliff.” Stepping into this region incurs a reward of −100 and sends the agent instantly back to the start."

And on line if the new position is in the _cliff then the new position is self.start_state_index therefore, "sending the agent instantly back to the start"
Therefore, I'm closing the issue, feel free to message if you spot an issue I missed


MM-11318: Expands shortened links inline.

Summary
Displays preview for shortened image links.
Ticket Link
MM-11318
Device Information

Android 7.1.1 emulator
iPhone 6, iOS 11.4 emulator

Screenshots

we are not using fetch, we are using RNFetchBlob, are you sure it does not handle 302? I really want to avoid adding yet another library for network requests

another question, is this for 1.11 or 1.12?

Opened https://mattermost.atlassian.net/browse/MM-11785 to get non-markdown images loading via the proxy on mobile apps.

@enahum @hmhealey Note that since opened this PR I created an API endpoint that proxies the HEAD request to the shortened link for the webapp, which was required because the request could not be made directly by the browser due to CORS policy.
0/5, but does anyone have a preference whether we continue to use the getShortenedImageLink function or whether we use the new Redux action I added (like this https://github.com/mattermost/mattermost-webapp/pull/1601/files#diff-564aa388e5c66ad77190526b786fbe89R97)?

I think you should keep this one and add the option to use the new endpoint if it is available, remember that the mobile app needs to be backwards compatible, meaning it should work with the current ESR version

I just noticed shortened YouTube links are not previewing on mobile. I'll put this back to WIP to solve.

@hmhealey ping


Hi. May I ask why when I save the live edit and clear the cache, the page will be reloaded to the original interface, when f5 the page is normal again

tks, How to handle now?. Implemented the project

Hello. I got the same problem. I use live edit but sometimes, my website's appearance turn to original design. Please let me know when you release your update.
Thanks.

Hi, this bug is fixed in the dev branch, we will release update soon

This bug also happends when i use clearcache() function but sometimes, i have to use clearcache(). How to use clearcache() appropriately? There is another function named cache_clear() with $cache_group, How to use it ? I can't find any instruction about using those functions. Please help me.

Hi @peter-mw can you please share me a commit link of this problem solution.

hi @SAKIL2037 , the fix is on this commit https://github.com/microweber/microweber/commit/c81835cc398381146f8d9099b0051dd81257a5c1

thanks @peter-mw

Hi @peter-mw can you help me how to delete cache in a specific folder (by domain) ?

Hi @peter-mw I want to set Default cache time longer for all data to reduce showing original template issue. How can i do that and do you think can it work ?


Browse Videos

Great work!
Would be nice to have also videos multiselect

+1 for this; that would be really, really helpful!

+1

The video files are not shown. Are there any options for this?

I am not sure why this is closed. Seems to be still impossible to do !?

This would be really, super, incredibly, helpful.

but really really really helpful. :P


CHANGELOG.md: Prepare v0.2.0

Anything else folks would like to see included in v0.2.0?
//CC @laurentsenta

@mxinden thanks for triggering the release, lgtm!

Tagged and published.


Parsing Hebrew

Hi, I replaced the RSS link to http://www.ynet.co.il/Integration/StoryRss2.xml
And then instead of hebrew i got Gibbrish.
10x in advanced
Benny

I had a similar issue. it's not really an issue with Hebrew. ynet encode their RSS feeds incorrectly. (They included a windows-1255 header, while the actual content is in UTF-8)....
Try:
  // YNET FIX (wrong windows-1255 in header) (http://www.ynet.co.il)
  if ([[_feed.url substringToIndex:21] isEqual:@"http://www.ynet.co.il"]) {
    NSString *content = [[NSString alloc]  initWithBytes:[response.data bytes]
                                                  length:[response.data length]
                                                encoding: NSASCIIStringEncoding];

    content = [content stringByReplacingOccurrencesOfString:@"Windows-1255" withString:@"UTF-8"];
    encodedData = [content dataUsingEncoding:NSUTF8StringEncoding];
  } else {
    encodedData = response.data;
  }


CardStack doesn't properly remount in 1.0.0-rc.1

I've got the following setup (pseudocode):
Switch
    Route1 render={() =>
        Navigation1
            Card1
            Card2
    Route2 render={() =>
        Navigation2
            Card3
            Card4

When I navigate to Route1 is works just fine. But when I try to navigate from Route1 to Route2 it throws with this error:
Element type is invalid: expected a string (for built-in components) or
a class/function (for composite components) but got: null.

Check the render method of `SceneView`.

What happens is React does not remount the Navigation, so constructor from CardStack doesn't run and componentWillReceiveProps does, but it sets an empty array to state.cards.
Versions of libs:

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

As a workaround I render Navigations with unique keys, I believe it should by default.

This should be fixed in 1.0.0-rc.2


pulley: Refactor conditional branches and table codegen

This commit first fixed an issue with table access codegen to disable
spectre mitigations on Pulley targets like how spectre is disabled for
memory accesses as well. This unblocked many tests related to tables
which then led to a different error about a trapnz with an 8-bit value
not being supported.
In fixing trapnz with 8-bit values this PR went ahead and did a
general-purpose refactoring for how conditional branches are managed.
Previously conditional traps and conditional branches had some
duplicated logic and the goal was to unify everything. There is now a
single Cond which represents the condition of a conditional jump which
is used uniformly for all locations such as select, brif, and
trap[n]z. This new type represents all the sorts of conditional
branches that can be done in Pulley, for example integer comparisons and
whether or not a register is zero. This Cond type has various helpers
for printing it, inverting it, collecting operands, emission, etc.
The end result is that it's a bit wordy to work with Cond right now
due to the size of the variants but all locations working with
conditional traps are deduplicated and now it's just repetitive logic
rather than duplicated logic.
Putting all of this together gets a large batch of spec tests working.
I'll note that this does remove a feature where trapnz was turned into
nothing or an unconditional trap if the argument was a constant, but
that feels like an optimization perhaps best left for the middle-end
rather than doing it in the backend.
cc https://github.com/bytecodealliance/wasmtime/issues/9783

Note this is currently based on https://github.com/bytecodealliance/wasmtime/pull/9793


Best way to upgrade installed packages?

Hi,
What's the best way to upgrade required packages installed via the 'require'?

The node downloads the packages every time you restart node-red and open flow. If you have specified the version in your require command it will download the latest version automatically. If you specify the version it will download the specified version. See the excerpt below from readme
//syntax to install a specific version
var lowerCase =<EMAIL_ADDRESS>
//if no version specified the latest version is installed
var upperCase = require('upper-case');

Closing issue due to no activity


add initial content for release doc

@lobziik i tried to clean things up based on your suggestions, wdyt?

Yep, great! Thanks a lot @elmiko!
LGTM


Does not support Chinese

Overlapping when mangaro system can not input Chinese and display Chinese

The real problem here is that I don't know enough about how Chinese is meant to work in a terminal to actually go fix the Chinese support in Extraterm. I also don't know much about how Chinese input works either. Someone with the right knowledge needs to help first otherwise there is little I can do.

Thank you for your great software.
Unfortunately, this is the same for Korean. Korean, Chinese and Japanese have similar input system. When you find the problem, that would fix it for these languages.
I'm using KDE Neon(Kubuntu based Distro.), but typing Korean is somewhat weird.
Other Electron based application, for example, Atom, has no problem like this, so I hope you could find a solution.

@nemonein I've done some testing with Japanese on Windows 10 and macOS, but I haven't been able to figure out how to get IME working on a Kubuntu (which I use). Can you explain or point me to a guide which can tell me how to set this up? All the stuff I could find looked very out of date.

@sedwards2009 Thank you for your response. I don't know exactly what you want, and I have no idea about Japanese input system. However, I guess using Japanese with fcitx would be the same as Korean.
Here's how to do on Kubuntu.
sudo apt install fcitx fcitx-mozc
im-config -n fcitx

fcitx is input system, and fcitx-mozc is a Japanese engine for fcitx. You could get the info here.
If you don't have im-config on your system, then install it by sudo apt install im-config.
Reboot, and you could see the fcitx(keyboard) indicator on the system tray.
You can change the language by clicking the indicator from English(or your default language) to Japanese(or vice versa), or just use Ctrl-Space.
Later, you can remove all of them by,
im-config -n none
sudo apt remove fcitx fcitx-mozc
sudo apt autoremove

I hope this could help you.

@nemonein Thanks. I've been able to get it installed, and after a bit of playing around I've got mozc set up and I can activate it for testing.
I assume it works the same as the other languages.

I had a look at this issue last night on Linux. The biggest problem I see is the position and size of the textarea holding the temporary characters when you are busy composing with the menu system. It was misaligned and simply not big enough to show its contents before you accept the characters.
To fix this properly I need to go into my fork of Ace editor and fix a bunch of layout and positioning problems. I have to do this anyway to fix a number of other bugs I see. It might be a while before I get time to fix this issue. There are a number of "low hanging fruit" features that I want to get into Extraterm first.

I'm glad you've found what causes this.
However, there's similar issues in Simplenote.
I think extraterm and Simplenote use the same engine(electron, Ace engine), so you might find something useful there.
Good luck!

Now Chinese has been displayed normally, thank you

@johncyuan  Cool. Let me know if there is something else blocking Chinese language output or input.


xash3d: init at unstable-2023-06-13

Description of changes

Xash3D is a GoldSrc reimplementation capable of running Half-Life.
Closes https://github.com/NixOS/nixpkgs/issues/231696.
Also closes https://github.com/NixOS/nixpkgs/pull/166081 which was targetting the old Xash3D engine. This pull requests targets the FWGS fork. I just named the package xash3d and not xash3d-fwgs since the old Xash3D is deprecated and never made it into nixpkgs anyway.
My only big issue with this is that it needs to modify the assets of Half-Life to properly run on 64bit. I wrote a wrapper for this that will make the required symlinks from the libraries in hlsdk to the assets. I don't know if there is a better way, but that's what the AUR does (https://aur.archlinux.org/packages/xash3d-git and the old https://aur.archlinux.org/packages/xash3d-fwgs-git).
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Since this pull request is sitting here for a month already, I took the opportunity to update it. Initially I just wanted to update the version and do some small fixes as I saw there were some problems, but I did a lot more in the end.
 That was one big thing bugging me.
I found that there was yet someone else who tried to package xash3d: https://github.com/vs49688/nix-repo/tree/master/pkgs/xash3d-fwgs. With that i discovered that actually xash3d on 64bit can not only run half-life, but a lot of other games. The issue though is that each game needs its own version of the hlsdk and its own wrapper.  If anyone wants more games I will do it later on request, but for now I want this to be merged.
I just need help on one point. Right now my packages work, but I've put a lot of things in top-level/all-packages.nix and I feel it doesn't belong there. Moreover for each package I must specify the hlsdk and the wrapper, which ends up in duplications. I don't know how to deduplicate this though. So if anyone has a way, please tell me.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/2445

I took again the opportunity to make some changes:

Finished incorporating all the suggested changes of your review. Github hit some of them so I missed those ... Didn't incorporate them all though, see comments
Added some documentation
Removed exposing the hlsdk of each game. No one cares anyway, it's just a dependency needed by the games.
Added desktop icons for the games. I didn't test them though so can't say if it really works, but I do see the .desktop file in the game folders.

I've just learned about the existence of meta.mainProgram so I added that one.

Since this PR is staling I'm pinging those people who were trying too to package xash3d, in the hope that you guys don't mind:
@barinov274
@chayleaf
@vs49688
Also @necrophcodr you were interested in FreeHL, so you might be interested in xash3d too ?

I know everything runs fine, I've completed all of the 3 games valve, bshift and gearbox, I just need someone who can confirm the desktop items work and some review on the nix file so that this PR gets merged.

@TheBrainScrambler I never attempted to upstream mine for two reasons:

The licensing situation of Xash3d is uncertain.
It depends of custom tarballs of game data I've extracted from Steam and put in a private cache.

That being said, I'm not sure I like the idea of having a per-game xash3d-$game launcher. It makes more sense to have individual derivations of xash3d, each with a different "set" of base games in the Nix store, with user modifications in XASH3D_BASEDIR.
That's what I do in mine, and I create a new derivation with:
  xash3d-fwgs-full = self.xash3d-fwgs.withGames (g: [
    g.valve
    g.bshift
    ...
  ]);

@TheBrainScrambler I never attempted to upstream mine for two reasons:
1. The licensing situation of Xash3d is uncertain.

Just set the license to unfree, that's what I did. I also put a comment there pointing to the related issue.

2. It depends of custom tarballs of game data I've extracted from Steam and put in a private cache.

That being said, I'm not sure I like the idea of having a per-game xash3d-$game launcher. It makes more sense to have individual derivations of xash3d, each with a different "set" of base games in the Nix store, with user modifications in XASH3D_BASEDIR.
That's what I do in mine, and I create a new derivation with:
  xash3d-fwgs-full = self.xash3d-fwgs.withGames (g: [
    g.valve
    g.bshift
    ...
  ]);

That's why to package this I avoided 2. Instead, you can build Xash3D without any problems, then you can put your assets in $XDG_DATA_HOME and then run it the right game with the right wrapper.
I don't understand why you would want sets of games instead of just allowing to install each game individually. So what you want is for example this:

xash3d-set1 (with just valve
xash3d-set2 (with valve, bshift, gearbox)
...

Instead of just exposing valve, bshift and gearbox as individual packages, and letting the user choose which ones to install ? What's the point behind that ?
Besides, you need to have one launcher per game. The key is this line in the wrapper:
exec ${xash3d-unwrapped}/bin/xash3d -game ${gamedir} $flags_dll $flags_clientlib "$@"

with the flags:
    if [ ! -f "$XASH3D_BASEDIR/valve/dlls/${dllname}_amd64.so" ]; then
      flags_dll="-dll ${hlsdk}/${gamedir}/dlls/${dllname}_amd64.so"
      flags_clientlib="-clientlib ${hlsdk}/${gamedir}/cl_dlls/client_amd64.so"
    fi

(in retrospect this if is just useless. Will remove it)
See, instead of copying the hlsdks dlls in the user's assets and thus putting garbage in there, instead I specify to xash3d the location of the dlls when I launch it. And since every game has a different hlsdk, you need one launcher per game.
This part is how I avoid your problem 2.

So @vs49688 what do you think ?

@TheBrainScrambler I never attempted to upstream mine for two reasons:
1. The licensing situation of Xash3d is uncertain.

Just set the license to unfree, that's what I did. I also put a comment there pointing to the related issue.

Fair. But iirc this means it won't be built by Hydra, which to me is one of the main benefits of upstreaming.

2. It depends of custom tarballs of game data I've extracted from Steam and put in a private cache.

That being said, I'm not sure I like the idea of having a per-game xash3d-$game launcher. It makes more sense to have individual derivations of xash3d, each with a different "set" of base games in the Nix store, with user modifications in XASH3D_BASEDIR.
That's what I do in mine, and I create a new derivation with:
  xash3d-fwgs-full = self.xash3d-fwgs.withGames (g: [
    g.valve
    g.bshift
    ...
  ]);

That's why to package this I avoided 2. Instead, you can build Xash3D without any problems, then you can put your assets in $XDG_DATA_HOME/xash3d and then run it the right game with the right wrapper.
I don't understand why you would want sets of games instead of just allowing to install each game individually. So what you want is for example this:
* xash3d-set1 (with just valve)

* xash3d-set2 (with valve, bshift, gearbox)

* ...

Instead of just exposing valve, bshift and gearbox as individual packages, and letting the user choose which ones to install ? What's the point behind that ?
Besides, you need to have one launcher per game. The key is this line in the wrapper:
exec ${xash3d-unwrapped}/bin/xash3d -game ${gamedir} $flags_dll $flags_clientlib "$@"

with the flags:
    if [ ! -f "$XASH3D_BASEDIR/valve/dlls/${dllname}_amd64.so" ]; then
      flags_dll="-dll ${hlsdk}/${gamedir}/dlls/${dllname}_amd64.so"
      flags_clientlib="-clientlib ${hlsdk}/${gamedir}/cl_dlls/client_amd64.so"
    fi

(in retrospect this if is just useless. Will remove it)
See, instead of copying the hlsdks dlls in the user's assets and thus putting garbage in there, instead I specify to xash3d the location of the dlls when I launch it. And since every game has a different hlsdk, you need one launcher per game. This part is how I avoid your problem 2.

From a purist perspective, 2 isn't a problem - I personally want the assets system-wide, and not in my home directory.
From a realist perspective, due to the assets being proprietary (boo), there's not really a nice way to do it without user intervention (i.e. putting it in  $XDG_DATA_HOME/xash3d), so I guess I'm okay with this.

So um can I ask for this to be merged ?

I don't have merge permission unfortunately.
But since I've approved it, you can post it here: https://discourse.nixos.org/t/prs-already-reviewed/2617/1089

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1091

I've tested it and it currently seems to work fine with the OpenGL renderer, however the software renderer crashes with no real stack trace (other than originating from /nix/store/zns2rxbqx6fx9qi8x6kk9dlkdpnyivsj-xash3d-unwrapped-unstable-2023-07-02/lib/xash3d/libref_soft.so which appears to be/be related to the software renderer.
I've not been able to further debug the issue, and I don't see any open issues on the source repository about this, so I'm inclined to believe that this is a bug of the derivation being built, rather than a bug in the source code.
@TheBrainScrambler did you try running this under different settings, such as using the software renderer?

I've tested it and it currently seems to work fine with the OpenGL renderer, however the software renderer crashes with no real stack trace (other than originating from /nix/store/zns2rxbqx6fx9qi8x6kk9dlkdpnyivsj-xash3d-unwrapped-unstable-2023-07-02/lib/xash3d/libref_soft.so which appears to be/be related to the software renderer.
I've not been able to further debug the issue, and I don't see any open issues on the source repository about this, so I'm inclined to believe that this is a bug of the derivation being built, rather than a bug in the source code.
@TheBrainScrambler did you try running this under different settings, such as using the software renderer?
To be clear, the software renderer crashes upon loading any level. It does not crash in the menu itself.

To be honest I didn't test it until now, forgot that one defaults to true ...
But I've just did and apart from the fact that it looks ugly it works fine for me. I've tested this with valve and bshift. So this must be an issue on your side I guess ?
I should also document that I didn't test the other renderer, those that default to false. Will do this when I can.

I've tested it and it currently seems to work fine with the OpenGL renderer, however the software renderer crashes with no real stack trace (other than originating from /nix/store/zns2rxbqx6fx9qi8x6kk9dlkdpnyivsj-xash3d-unwrapped-unstable-2023-07-02/lib/xash3d/libref_soft.so which appears to be/be related to the software renderer.
I've not been able to further debug the issue, and I don't see any open issues on the source repository about this, so I'm inclined to believe that this is a bug of the derivation being built, rather than a bug in the source code.
@TheBrainScrambler did you try running this under different settings, such as using the software renderer?
To be clear, the software renderer crashes upon loading any level. It does not crash in the menu itself.

To be honest I didn't test it until now, forgot that one defaults to true ... But I've just did and apart from the fact that it looks ugly it works fine for me. I've tested this with valve and bshift. So this must be an issue on your side I guess ? I've tried for both to load the hazard course, and for valve I loaded the final boss fight too.
I should also document that I didn't test the other renderers, those that default to false. Will do this when I can.
EDIT: added a small comment above those flags that I didn't test. Thus people can still use them if they want, but it's now explicit that I didn't test those.

That's all possible. I did try running it with LD_DEBUG=libs and when it crashed, the following lines were output. If they're entirely related, I do not know, but it would definitely be a good indicator of the issue. If this is caused by this build process or not, I can't say for sure of course.
Additionally, I'd like to point out that if it works for you, then it should surely also work on my end, bar any hardware related issues (which I do not believe this to be).
Anyway, here's the output for posterity:
     42034:     calling init: /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/cl_dlls/client_amd64.so
     42034:
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/cl_dlls/client_amd64.so: error: symbol lookup error: undefined symbol: InitVGUISupportAPI (fatal)
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/cl_dlls/client_amd64.so: error: symbol lookup error: undefined symbol: GetClientAPI (fatal)
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/cl_dlls/client_amd64.so: error: symbol lookup error: undefined symbol: F (fatal)
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/cl_dlls/client_amd64.so: error: symbol lookup error: undefined symbol: HUD_ChatInputPosition (fatal)
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/cl_dlls/client_amd64.so: error: symbol lookup error: undefined symbol: HUD_GetRenderInterface (fatal)
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/cl_dlls/client_amd64.so: error: symbol lookup error: undefined symbol: HUD_ClipMoveToEntity (fatal)
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/cl_dlls/client_amd64.so: error: symbol lookup error: undefined symbol: IN_ClientTouchEvent (fatal)
     42034:
     42034:     calling fini: /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/dlls/hl_amd64.so [0]
     42034:
     42034:
     42034:     calling init: /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/dlls/hl_amd64.so
     42034:
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/dlls/hl_amd64.so: error: symbol lookup error: undefined symbol: GetNewDLLFunctions (fatal)
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/dlls/hl_amd64.so: error: symbol lookup error: undefined symbol: Server_GetBlendingInterface (fatal)
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/dlls/hl_amd64.so: error: symbol lookup error: undefined symbol: Server_GetPhysicsInterface (fatal)
     42034:     /nix/store/ngpwaaqf8vr94ql5bmzqz839a1ggkrw6-hlsdk-valve-unstable-2023-07-17/valve/dlls/hl_amd64.so: error: symbol lookup error: undefined symbol: SV_SaveGameComment (fatal)
Ver: Xash3D FWGS 0.20 (build -12874-notset, linux-amd64)
Crash: signal 11 errno 0 with code 1 at 0x78 (nil)
Ver: Xash3D FWGS 0.20 (build -12874-notset, linux-amd64)
Crash: signal 11 errno 0 with code 1 at 0x78 (nil)
Stack backtrace:
Stack backtrace:
 1: 0x7f603040f005 (/nix/store/zns2rxbqx6fx9qi8x6kk9dlkdpnyivsj-xash3d-unwrapped-unstable-2023-07-02/lib/xash3d/libref_soft.so)
 1: 0x7f603040f005 (/nix/store/zns2rxbqx6fx9qi8x6kk9dlkdpnyivsj-xash3d-unwrapped-unstable-2023-07-02/lib/xash3d/libref_soft.so)

@TheBrainScrambler I've looked into the issues for a bit, not being very familiar with the codebase I was unable to find the root of the issue. If anyone manages to run it just fine on a clean system with the software renderer, then I'd definitely say keep it as is, otherwise it might be better to default that part to off?
If you're able to test this on a machine that preferably hasn't had any related work done, or is a fresh install, and it works fine there, that's by far good enough of a confirmation that it is related to my setup, although I'm not aware of what exactly it could be when it seems to issue errors relating to the currently-not-built VGUI support.

@necrophcodr I've tried running Xash3D with LD_DEBUG=libs too, and I get the exact same symbol lookup errors that you have with the software renderer, except that I don't get any crash. So there errors are unrelated to your crash.
It's weird because Nix after all revolves about sandboxing and reproducible builds, normally I shouldn't have to get a clean install to test this ... I've built xash3d using nix-build so normally there should be no leak eh ?
I'm going to default the software renderer to false, because there is a merge conflict I have to solve now anyway, and thinking about it I don't want to retest it if there is an update. I first thought about turning it back on by default if someone else can test it and confirm it works, but I don't think it's worth maintaining it in the long run. I don't think anyone will actually care about this. And if someone does he can just enable it with an override.
If someone wants any of the renderers other than opengl enabled by default, let me know, I'm open to that. It shouldn't matter too much though since Hydra won't build this package anyway.

@necrophcodr I've tried running Xash3D with LD_DEBUG=libs too, and I get the exact same symbol lookup errors that you have with the software renderer, except that I don't get any crash. So there errors are unrelated to your crash.
It's weird because Nix after all revolves about sandboxing and reproducible builds, normally I shouldn't have to get a clean install to test this ... I've built xash3d using nix-build so normally there should be no leak eh ?
I'm going to default the software renderer to false, because there is a merge conflict I have to solve now anyway, and thinking about it I don't want to retest it if there is an update. I first thought about turning it back on by default if someone else can test it and confirm it works, but I don't think it's worth maintaining it in the long run. I don't think anyone will actually care about this. And if someone does he can just enable it with an override.
If someone wants any of the renderers other than opengl enabled by default, let me know, I'm open to that. It shouldn't matter too much though since Hydra won't build this package anyway.

Personally I much prefer the software renderer, because it's closer to the original experience. But that's my own experience, so I approve of this decision! I've got nothing else to add, it looks decent to me.

Personally I much prefer the software renderer, because it's closer to the original experience. But that's my own experience, so I approve of this decision! I've got nothing else to add, it looks decent to me.

If you want it I can re-enable it by default, but for that we need that someone else tests the software renderer to see if it's just you or just me for which it works/doesn't.

Oh no since it works for you and not for me, and the reason so far has not been discerned, I'd much prefer to keep it behind an easy-to-override flag instead. Those that care or want it supported, will be able to enable and test for it. And those that don't, probably won't miss it.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1180

Since this program showed up on hacker news today I went looking in nixpkgs! Can you explain why the SDK is needed to build? I wasn't really able to understand it. Is that only for 64 bit or non-x86 builds? I don't have any of the other games to test it with at  the moment. I would suggest splitting it into a 32-bit and 64-bit package (kinda like wine).
I was able to get a 32 bit build running against the latest git:
nix-shell -E 'let pkgs=(import <nixpkgs> {}).pkgsi686Linux; in with pkgs; stdenv.mkDerivation { name="xash3d-fwgs"; buildInputs = [ python3 pkg-config wafHook SDL2 libopus freetype fontconfig cmake ]; } '
$ wafConfigurePhase
$ wafBuildPhase
$ ./waf install --destdir="$PWD/destdir"

A couple issues I ran into:

for 32 bit build, I needed to copy (prebuilt?!) vgui.so from 3rdparty/vgui_support/vgui-dev/lib
libopus needs to be >= 1.4 (not yet in nixos 23.05), but this is not important

I tested your package on nixos 23.05. I had to change wafInstallFlags to installFlags but it still installed into the wrong place! (out/nix/store/.... instead of out/bin/), so I had to tweak it a bit:
# wafConfigureFlags ... "--prefix=/"
installFlags = [ "--destdir=${placeholder "out"}" ];

The command I used to build in its own folder outside of nixpkgs: NIXPKGS_ALLOW_UNFREE=1 nix-build -E 'with import <nixpkgs> {}; callPackage ./. {}'

Oh, and TIL another way to avoid the zip timestamp problem is to put ensureNewerSourcesForZipFilesHook in nativeBuildInputs!

Since this program showed up on hacker news today I went looking in nixpkgs! Can you explain why the SDK is needed to build? I wasn't really able to understand it. Is that only for 64 bit or non-x86 builds? I don't have any of the other games to test it with at the moment. I would suggest splitting it into a 32-bit and 64-bit package (kinda like wine).
I was able to get a 32 bit build running against the latest git:
nix-shell -E 'let pkgs=(import <nixpkgs> {}).pkgsi686Linux; in with pkgs; stdenv.mkDerivation { name="xash3d-fwgs"; buildInputs = [ python3 pkg-config wafHook SDL2 libopus freetype fontconfig cmake ]; } '
$ wafConfigurePhase
$ wafBuildPhase
$ ./waf install --destdir="$PWD/destdir"

A couple issues I ran into:
* for 32 bit build, I needed to copy (prebuilt?!) vgui.so from 3rdparty/vgui_support/vgui-dev/lib

* libopus needs to be >= 1.4 (not yet in nixos 23.05), but this is not important

I tested your package on nixos 23.05. I had to change wafInstallFlags to installFlags but it still installed into the wrong place! (out/nix/store/.... instead of out/bin/), so I had to tweak it a bit:
# wafConfigureFlags ... "--prefix=/"
installFlags = [ "--destdir=${placeholder "out"}" ];

*EDIT: I think you might also want to add dontAddPrefix for wafHook to not automatically set the prefix. That way I can get it to install to relative paths better.
The command I used to build in its own folder outside of nixpkgs: NIXPKGS_ALLOW_UNFREE=1 nix-build -E 'with import <nixpkgs> {}; callPackage ./. {}'

Why did you test against 23.05?

Why did you test against 23.05?
Yeah that was dumb. Hopefully that means some of my comments are wrong!

It's been a month since I last checked on this package, since no one has merged it yet. So there might be breakage perhaps, it wouldn't be the first time. I also don't remember everything so well.
But clearly looking at https://nixos.org/manual/nixpkgs/unstable/#waf-hook-variables-controlling (note: it's the manual for unstable) you have wafInstallFlags, and I know that it worked when I updated it last month. Also if you look at pkgs/development/tools/build-managers/waf/setup-hook.sh you indeed have a wafInstallPhase.
So make sure to test with unstable to see if you're still having problems building it.

Oh, and TIL another way to avoid the zip timestamp problem is to put ensureNewerSourcesForZipFilesHook in nativeBuildInputs!

Ah nice. It's not documented in the nixpkgs manual though :p but I will try this.

Can you explain why the SDK is needed to build?

For 64bit you need it for sure. For 32bit you don't need it, because then it will just use the one that is bundled with the game you play. For example Half-Life has in its directory dlls/hl.so and cl_dlls/client.so . Those bundled libraries are compiled for 32bit since the original games are for 32bit, that's why for 64bit you can't use them and have to build the SDK. But you can also just build the SDK for 32bit and it works too. Or so I think. I've never tested it since I'm on 64bit.

for 32 bit build, I needed to copy (prebuilt?!) vgui.so from 3rdparty/vgui_support/vgui-dev/lib

Honestly I've never really understood the business with VGUI.
Also using nixpkgs-review might make things easy for you to build this repo, but as with most things, I'm not sure. I will tell you the command I use to build xash3d-valve once I'm building it.

Nice, ensureNewerSourcesForZipFilesHook works, thanks !
I'm building with NIXPKGS_ALLOW_UNFREE=1 nix-build /path/to/my/nixpkgs -A xash3dGames.xash3d-valve where /path/to/my/nixpkgs is where I have my fork of nixpkgs, which has among other things the nix files for xash3d.

Thanks for the help/pointers and also the explanations about how the engine works. I will try to take a look next week.

[ensureNewerSourcesForZipFilesHook] where did you find it ? I can't even find it on the internet

I think grepping through nixpkgs. I was looking for similar date problems and if other packages had the issue for waf. It still it took me awhile to notice it even though it is even in the waf package, hah!

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1342

I am no longer using Nix. Someone else will have to take maintainership if this ever gets merged

I am no longer using Nix. Someone else will have to take maintainership if this ever gets merged

I think the lack of momentum on this is due to the license - it's unfree, meaning it won't be cached by Hydra, and people will have to rebuild it every time anyway so there's not really much point, as much as it saddens me.

My comment about dropping maintainership is not really related with this PR being stale, it's just that I'm not using Nix anymore, for other reasons.
I suspect more that it's just the huge number of PRs in nixpkgs that make this package ignored even though I've been posting about it 2 or 3 times in the "Already reviewed" thread on discourse. That issue is known and folks are trying to solve this.
But the point of this package was 1) while you have to build it, at least you don't have to write any recipe for it because the recipe is already written, and somehow figure out how to get Nix use your local packages, which combined with a rather bad documentation makes this a bit hard, and 2) the way I made this package was specifically done so that you don't touch your game assets, while all other methods I had seen so far told you to copy some files into your assets which I didn't like.
With that recipe you could just install the package, drop your assets into ~/.local/share/xash3d and be done with it


Run torchdynamo+nvfuser on PyTorch's test suite

@ezyang suggested running TorchDynamo+nvFuser on the PyTorch test suite to get a better sense of corner cases we might be missing.
This should be doable using the unittest setUp/tearDown hooks.

Before you even get here, should first just run TorchDynamo+Eager.
TorchDynamo will print a bunch of scary "failure" messages but as long as fallback to eager works OK this is not a problem (just check if the test suite passes). These failures should get deduplicated and triaged and used to inform bugfixing work that may need to be done. (We will need to turn off this errors printing by default when we actually do a release).
Although you can use setUp/tearDown, overriding run is easier:
diff --git a/torch/testing/_internal/common_utils.py b/torch/testing/_internal/common_utils.py
index 0b072919c4..549c1e9942 100644
--- a/torch/testing/_internal/common_utils.py
+++ b/torch/testing/_internal/common_utils.py
@@ -1794,7 +1794,11 @@ class TestCase(expecttest.TestCase):
             failures_before = 0 if result is None else len(result.failures)  # num tests marked as failed before starting
             errors_before = 0 if result is None else len(result.errors)  # num tests marked as errored before starting

-        super().run(result=result)
+        import torchdynamo
+        torchdynamo.config.trace = True
+        with torchdynamo.optimize("eager"):
+            super().run(result=result)
+
         # Early terminate test if necessary.
         if self._should_stop_test_suite():
             if result.wasSuccessful():

you may also consider just turning torchdynamo at the top level entry point run_tests, since this is supposed to work.
You might want to do https://github.com/facebookresearch/torchdynamo/issues/69 first so you don't rabbithole into useless code (and make your execution time take a long time)

The warnings @ezyang shared via chat are:

"/data/users/ezyang/torchdynamo/torchdynamo/variables/torch.py", line 151, in call_function
    return TensorVariable.create(
  File "/data/users/ezyang/torchdynamo/torchdynamo/variables/tensor.py", line 120, in create
    assert (
AssertionError: torch.* op returned non-Tensor Generator call_function <class 'torch._C.Generator'>

Fix:
should add the op returning a RNG generator to _disallowed_function_ids():
https://github.com/jansel/torchdynamo/blob/033d2b0183b2c09bf0860095d5e932bcf972d029/torchdynamo/allowed_functions.py#L18

"/data/users/ezyang/torchdynamo/torchdynamo/variables/misc.py", line 262, in call_function
    return self.obj.call_method(tx, self.name, args, kwargs).add_options(self)
  File "/data/users/ezyang/torchdynamo/torchdynamo/variables/tensor.py", line 228, in call_method
    constant_result = ConstantVariable(product(self.size), **options)
TypeError: reduce() of empty sequence with no initial value

Fix:
Our product() should default to 1 on an empty list of sizes.  Test case would be something like tensor(1).numel()

"/data/users/ezyang/torchdynamo/torchdynamo/variables/builtin.py", line 347, in call_isinstance
    arg_type = arg.python_type()
  File "/data/users/ezyang/torchdynamo/torchdynamo/variables/base.py", line 133, in python_type
    raise NotImplementedError(f"{self} has no type")
NotImplementedError: NumpyVariable() has no type

Fix:
We can add python_type() method to NumpyVariable() (since we will guard on the type of numpy tensors anyway, so it just a constant).
All those should be quick 1-line-ish fixes.  Though, definitely demonstrates we can get some good data from this test suite!

There are TONS of warnings, there isn't any particular reason to prioritize these, I think some triage will be necessary first

Note, status updates this effort are over in #69
https://github.com/pytorch/torchdynamo/issues/69#issuecomment-1126591497

Closing this as this is not so relevant right now.


Adding oauth2client #70

Changes Unknown when pulling 8d7f502000b5cd78203baf7cd6aa7ad6828b1b62 on WolfBerwouts:develop into ** on opengridcc:develop**.