Unable to call a platform channel method from another isolate

When I try to invoke a platform channel method from a custom spawned isolate, the app crashes badly (both on iOS and Android). I'm trying to figure out whether this is expected or not.
If it's not, it's probably worth to mention that somewhere.
Anyways I think that this can potentially be a strong limitation. Is there a way to be able to call platform plugins from a secondary isolate?

I got the same problem, any news about this issue?

@mravn-google Do you have any update on this? I ran across the same problem ( I'm in a need of generating e.g. RSA 2048 key pair on the platform side which takes a while on older devices). I'd rather avoid creating threads or services as that would double my work on the platform side since it is to be implemented for both Android and iOS platforms. Do we have any easy way to run those platform specific operations asynchronously from Dart? Your workaround seems to be fine, however it seems that I have some issues with implementing that as I am quite new to Flutter and Dart (I'm not sure how to configure a handler that forwards messages for an Isolate and so on), sorry about that.

I got the same issue.
I need to generate a PDF on the platform

I got the same issue. our product base on CPU-bound. we need to use platform method in isolate. Could you please tell us another way to solve this problem？

@Thomson-Tsui I had a similar problem, and based on a distillation of the work by @bkonyi and his FlutterGeofencing sample I managed to get something to work in an isolate that I am using with flutter_blue and other plugins. Its pretty untested but please give it a try.

@Thomson-Tsui I had a similar problem, and based on a distillation of the work by @bkonyi and his FlutterGeofencing sample I managed to get something to work in an isolate that I am using with flutter_blue and other plugins. Its pretty untested but please give it a try.

I got the same issue, i think it's important

Has anyone solved this problem?

I am using https://pub.dartlang.org/packages/flutter_isolate

I need to play audio at varying intervals in the background. The simplest way to do that would be to have a separate Isolate managing the audio, but without this feature it will take some really weird work around with ports

I have an assumption that when you run an isolate from dart, the native libraries do not link.
For example, when running flutter code from java, this method is called:
   private native void nativeRunBundleAndSnapshotFromLibrary (
       long nativePlatformViewId,
       @NonNull String [] prioritizedBundlePaths,
       @Nullable String entrypointFunctionName,
       @Nullable String pathToEntrypointFunction,
       @NonNull AssetManager manager
   );

In which all the magic happens)
This is verified by the fact that if you create several FlutterNativeView, and in each run:
  public void runFromBundle (FlutterRunArguments args)

then we get a few "dart isolates"

We also face this issue. We need to collect data from background isolates (e.g. location, noise, etc.) which all rely on platform channels. @mravn-google -- any news on an update to the API which could address this?

This really a serious problem in tool or media likes apps.
Platform thread isn't an easy way.

I have also faced the similar issue and stuck right now.

+1 (my project also badly needs platform channels in isolates)

I need to render images in background. It have to be done in an isolate because each image takes a few seconds and blocks ui if done in the main thread. I am stuck at the first native call of dart:ui.PictureRecorder and assume each and every graphical call (which uses native functions) will not work either. Since I am using a lot of functions in the canvas it would be a pain to work with callback ports.
A solution is highly appreciated.

+1 I got this issue

It seems like @mravn-google has stopped working for Google in Aarhus. I wonder if there are anyone else from the Google Flutter team who is looking into this. IMO, this is a major show stopper for serious use of Flutter for building resilient and robust Flutter applications....
Maybe @sethladd can give a status?

+1

+1

any updates on this?

+1

The day you met a bug that you do know how to solve by your own but only wait is the last hope before I become flutter engineer.

@sooxiaotong start a branch and pr?

We need a more detailed explanation of the workaround for new flutter/dart users. I only started using isolates two days ago, and flutter/dart last week, and it's looking like I'm going to have to use the main thread for some seriously labor-intensive tasks, which makes learning flutter a bad solution. Without usable multithreading, I'm better off learning Kotlin and building my apps twice. I hope y'all can get something together that is intelligible for beginners, I'd really like to be able to justify learning flutter to my employers so I can use it at work.

+1 semoga cepet beres

+1 My project needs platform channels in isolates.

I ran into this same issue and made a package (Isolate Handler) using a workaround similar to the one posted by @mravn-google. However, Isolate Handler does allow you to use channels both from the main isolate alongside other isolates. It supports direct MethodChannel calls from within isolates, but currently EventChannel streams are not supported. I will look into adding support for them if it turns out to be something people need.
One downside of the workaround is the need to supply the isolate handler with channel names. It is not a problem in my project as I am writing my native code myself, but otherwise you have to look through the source code of any plugin you are using to find them.
Another option that was already posted above is FlutterIsolate, which uses a different approach and is also worth looking into.
Hopefully the Flutter devs will provide a full and proper solution to this problem soon.

+1 any update here ?

+1

Interesting thing I am noticing here:
Several plugins are written with the assumption that the method call is necessarily passing messages across threads; relatedly, there are some mightily confused people who followed that bg geofencing example and now are wondering how to get back on the main thread...
That plugin code will not cause "jank" if they are performing a short task, but they are by default running on the platform thread, what used to be called "the UI thread" on android; this can absolutely cause gestures to be dropped during high workload, and I would not be surprised if it took resources from code returning Futures to the FutureBuilder.
You then are at the mercy of the plugin author's engineering; only if they moved their work to another thread in the native code will you be getting work done in a separate thread than the rest of your Dart code; those method calls have to run on the platform thread.
While I do understand the rationale the good chinmaygarde relates here, agreeing that:

most platform API calls are fast and often need to be made on the main thread;
it makes more sense to account for threads in the native code
I feel that the design decision opened up a significant anti-pattern as a default for plugin authors; on first look, I think even the firebase ml plugin uses the main thread thread for its processing.

There's no free lunch...

I ended up implementing heavy workload with the native platform threads using Java and Swift. Found no workaround.

+1 need this

Unfortunately - Flutter is not ready for primetime for it to lack something so very basic yet so very critical. Not all Flutter apps can get away with simple async/await. For the heavy lifting Isolates are essential and plugins that cannot allow platform channel method calls from another Isolate make them pointless and redundant. Xamarin had this  built into the framework from day 1 - surely this should be up voted for a solution now.

+1 need this

+1

I need to call a platform channel method from another isolate too!

+1

+1

+1

+1 This is a must have

+1
I have a native SDK that I must call. It would be a nice thing to have something like BackgroundFlutterMethodChannel

+1

+1

Does anyone have any information on what crashes badly actually describes? I'm working on a proof-of-concept that implements a native library and I'm running into watchdog timeouts (0x8badf00d) and other weird errors.
I suspect this is caused by some methods [and results] being invoked from other threads but I have not been able to make any progress in pinpointing the exact issue.

+1

+1

+1

+1

+1 need this

+1 need this also!

I think this issue should be prioritized as at least P2.
The uses of the platform channels that I can think of are

To call platform-specific APIs
To run platform-specific heavy tasks in the background and report when done

And this is a major blocker for using Platform channels.

The flutter_downloader plugin handles background isolate code. I think it's worth looking at it.

i was hoping this to be resolved in the upcoming version, but the bug still remains in 1.12 version!
our client side code is heavily on the other side of channel and is written in kotlin and the app almost stucks for each single http connection
Gooooooooooooooogle?!!!!!!!!!!!!?!?!?!?

This plugin might be helpful flutter_isolate

+1 we need this...come on

+1
The biggest problem for me is that fact the communication between Dart and the native platforms happens on the main thread - sending significant chunks of data is impossible without lagging UI or writing cumbersome code to page data.

I got the same issue, i think it's very important

@YaredTaddese
Depending on the complexity of what your ISOLATE does and the inter-communication between your UI and ISOLATE - It is possible to workaround this issue in your UI code by structuring what data the ISOLATE requires to do its job up front prior to invoking the ISOLATE. I have rich PDF generation in my current Flutter app - a user can select multiple PDF's to generate.
Each can generate from 1 to 10,000 pages . My UI is responsive , when each job completes a TAB is constructed showing the report - the user gets notified when a job completes via a TOAST and the UI does not grind to a halt - the user is unaware that there are services in the background busy generating rich PDF's.
Note, my PDF generation is an ISOLATE. In the ISOLATE -, I also need to download images from CLOUD FIRESTORE and embed in a PDF - this all happens seamlessly.

Are you calling a platform-specific code?

@YaredTaddese
Depending on the complexity of what your ISOLATE does and the inter-communication between your UI and ISOLATE - It is possible to workaround this issue in your UI code by structuring what data the ISOLATE requires to do its job up front prior to invoking the ISOLATE. I have rich PDF generation in my current Flutter app - a user can select multiple PDF's to generate.
Each can generate from 1 to 10,000 pages . My UI is responsive , when each job completes a TAB is constructed showing the report - the user gets notified when a job completes via a TOAST and the UI does not grind to a halt - the user is unaware that there are services in the background busy generating rich PDF's.
Note, my PDF generation is an ISOLATE. In the ISOLATE -, I also need to download images from CLOUD FIRESTORE and embed in a PDF - this all happens seamlessly.

Are you calling a platform-specific code?

Yes

Has anyone got a workaround for calling platform channels from an isolate ?it defeats the idea to do so from the UI thread.

I stucked on the same thing.
I am calling a methodchannel  with a kotlin code, and getStream.io API, but that is slow as hell, and my Flutter UI freeze, I wanted to add compute to it, but I got the same errors.
How can I fix this?

I just ran across this as well. I am prototyping a new app, and really this issue might be a showstopper for me and cause me to look into other non-Flutter solutions, which will make me sad.
As the volume of responses to this issue indicates, this is a MAJOR problem and pretty much defeats the main use case of isolates in the context of a Flutter app.

I just ran across this as well. I am prototyping a new app, and really this issue might be a showstopper for me and cause me to look into other non-Flutter solutions, which will make me sad.
As the volume of responses to this issue indicates, this is a MAJOR problem and pretty much defeats the main use case of isolates in the context of a Flutter app.

yeah this is a huge issue, need to fix ASAP

I've been learning flutter for 10 days, and I love it so far, but after a while experimenting, you start to come into problems, and see the downsides, and THIS issue, opened since 5 jan 2018, oof...

Please fix this bug.

Hope to support soon.

*@jpsarda, @klaszlo8207 Don't hold your breath. It doesn't seem like a bug, it's a major limitation of the system. And yes, there was a solution to the problem, until today: https://pub.dev/packages/isolate_handler. The programmer had to pull the plug on it today because a very recent change in Flutter (only a few days old) made it impossible to follow that route any more.
You have to know, however, that this didn't really mean parallelism. It might have appeared that way and looks are important in mobile apps but the platform code was running on the main thread, anyway, so delegating a long task back from the isolate to the main thread didn't really accomplish anything.

Would be nice to have dart handling the multithreading by default any time you use an async function.

@spiderion Async never was meant to be multithreading, just like in many other platforms. Don't misunderstand me, I won't say the platform channel limitation is not a big problem because it is (my app had been working with the plugin I mentioned up until yesterday and now I have to look for workarounds), but async/await was never about multithreading from the very first day.

@zoechi We would love to know if there is any update on this issue or if there is any plan to fix it?
Thank you.

@spiderion They can't really fix it, it follows from the way isolates work. They have no UI engine behind them, hence there is no platform channel communication. However, there are two plugins to help you out:

https://pub.dev/packages/flutter_isolate provides a replacement isolate that can communicate to plugins because it creates its own UI backing (nothing that you see or have to deal with, just technically),

https://pub.dev/packages/isolate_handler that we have just modified to rely on the package above because the previous way it used was made impossible by a recent Flutter change. The advantage in using this package rather than flutter_isolate itself is that this adds handling capabilities, you can start several isolates, keep track of them and you don't have to set up your own communication between the isolate and the main thread (something you have to with both the original stock Isolate and FlutterIsolate manually) because it's abstracted away and readily available. (This package might not yet be fully updated on Pub but it's already recent on GitHub).

I use the second with perfect success. Actually, I have been using it for quite some time and when the Flutter breaking change came, I helped its programmer to move on to the new solution just because my app broke, too. :-)

Hi @deakjahn  thank you for your quick answer.
The solutions you have provided look to be very useful. However, I don't know if it could solve my problem. I currently have a situation in the app where the user creates a story lets say similar to Instagram. The app, in that case, needs to upload on the firebase storage a list of videos and files which might take a very long time to upload (around 15 minutes if the connection is slow) then after the files have been uploaded the app needs create a document on the firebase cloud, Firestore. The problem that I encounter is that if the user kills the app while the files are uploading then the complete task is not fully executed.

That's not really the area of the isolate, it just does what you tell it to do. Firebase can resume stopped unloads, as far as I can see, although I never used it myself.

I am trying to run the getBatteryLevel example:
https://flutter.dev/docs/development/platform-integration/platform-channels?tab=android-channel-java-tab
and it crashes everytime I try to run it on an android device. Can someone please help? I am not even sure how to check the error, since there are no logs and it just keeps on running in an infinite loop and I am using the exact same code.

Labeling this issue for P5 ??? 🥵

Please, check out the earlier comments. It seems very unlikely that this will be changed in the framework because it requires mechanisms that most isolates don't need, so there's no point in adding that overhead. For the cases when you do need it, there are packages to solve it.

Users don't care about the mechanism; how hard it is or not. They just need results. This might be impossible to do. But if the users need it, Flutter should provide it...
I understand that the implementation has limitations and might take tons of time to add this feature to the existing isolate or to create a new API with this feature. I just think this should have a higher priority than P5.
Also, I read all of your comments and it feels to me like you are just promoting the package you created.

No, it wasn't me who created it, but I did accept its ownership a few weeks ago, yes. I don't think that was kept secret in this thread. In addition to that, I mentioned two packages that are both able to provide what you look for and I have no affiliation with the second one other than relying on its excellent work done so far.
If you read it, the only reason I used these in the first place myself was that I needed the very same functionality that you asked for. And my apps have been relying on it for more than a year now, without problems. Besides, becoming more familiar with the inner workings of isolates than an occasional developer just using them would be (plus that I started to work on their counterpart in Flutter Web), makes me think that what I wrote was true: this functionality will not be incorporated in the framework because there are quite a few arguments against its inclusion. Not because it would rake tons of time, not at all. It would take relatively little time. Simply because it would add a backend complexity to all isolates than only a few of them would actually need and use.

@deakjahn thanks for these tips, will using  isolate_handler help me invoke a method and send data to the dart side whenever I get a notification on the native side? (I'm using twilio native sdks and receiving push notifications on the native side)

If this means you have a patform plugin that you use to communicate through a platform channel, then yes, any of the two packages would help. If it works from regular code, then these packages will make it work from an isolate as well.

@deakjahn Thanks for the quick response! My problem is that I couldn't do it with a regular platform channel, I'm not sure but I think the twilio sdk handles listening to push notifications in a seperate isolate and not on the main thread, and I think that's my problem. I wanted to know if it would work if I invoked the method each time I get a notification using one of the 2 packages ?thanks a lot.

I don't even know what Twilio is (OK, I googled it :-) ), so I really can't be sure. But at any rate, if you want to interface any native code, you have to use a platform channel, anyway. Either you use an already existing plugin somebody made for using this Twilio or you author it yourself (in which case you either create practically the same plugin, just don't publish it and refer to it locally), or you simply copy the relevant plugin code into your own app (it won't be an external dependency but other than that, it will have practically the same code, anyway).
And, if you do have a platform channel, than the previous answer applies: if you already can use the platform channel and the plugin from the main thread of a regular Flutter app, then you can use the same from an isolate using any of the two packages.

So this issue exists Since 5. Jan. 2018 and it's still not possible with extra effort. The solutions provided are nice - but there is nothing working on desktop yet. So for anyone who wants to try how well flutter is working on desktop and see if it's possible for a future product will get totally stuck here.
Almost any App needs isolates, because there are always some huge computations, how is it possible that there is nothing happening on such important Api design flaws?

I have added a way out for this for my plugin System Alert Window
The solution is mentioned here Isolate communication.
Although the example provided talks specific to the system alert window plugin, it can be easily replicated for other plugins. And it requires no other fancy plugins to make it work!

This can still be quite problematic. It means that there is no way to call a plugin directly from a isolate that is running on the background / foreground, because you would have to send a message to the app scope in order to do anything with a plugin. So if your app isn't running there is no way to do anything with a plugin in the backgroud.
So for instance you can't call a plugin if you would like to do anything with shared preferences in the backgroundMessageHandler of the FCM plugin when the app is closed will throw a MissingPluginException. I think this needs to be reopened.

@michael-ottink the scenarios you describe are precisely the ones where there shouldn't be a problem. Plugins that execute code in the background generally instantiate a whole new FlutterEngine with its own plugin registry and the isolate that runs within it will be able to directly communicate over platform channels to those plugins.
Incidentally, that means that if you have a need for an isolate to use plugins, the easy way is to just wrap that isolate in a FlutterEngine and it will magically inherit those powers. This is what the flutter_isolate package does for you on Android and iOS (although I agree it would be better to have a proper fix rather than a workaround. The workarounds available today have bottlenecks, overheads or otherwise may be under-maintained or lack support across all platforms.)

@ryanheise so flutter_isolate does it for me? How I can't really tell from the docs. What do I need to do to make it work for my FCM onBackgroundMessage?

Your problem is that firebase_messaging hasn't been updated in a long while and is lagging far behind the latest background execution APIs. If they updated their plugin, you would no longer have this problem. Just to clarify again, the kind of scenario you describe is precisely the one where there shouldn't be a problem because background isolates should already have access to plugins if implemented correctly. the firebase_messaging plugin is not implemented according to the latest APIs and that is why it does not work for you. You can submit a bug report with that project.

I am in dire need for help with this. Is there possibly any way I can do this. I have my first serious dealine tommorow and I just can't figure it out.

Do everything to ensure that plugins are being loaded. I don't know if you have followed the instructions in the README on how to enable background messages, but you need to create a custom Application class that hooks into the initialisation of the plugin's background FlutterNativeView. If you haven't done that, none of the plugins will be available for you to use. If that's still not working, you can try downgrading your project to the old v1-style plugin architecture (and risk breaking other plugins that require v2).

I setup my Application.kt like this
PACKAGE_NAME

import `in`.jvapps.system_alert_window.SystemAlertWindowPlugin
import io.flutter.app.FlutterApplication
import io.flutter.plugin.common.PluginRegistry
import io.flutter.plugin.common.PluginRegistry.PluginRegistrantCallback
import io.flutter.plugins.firebasemessaging.FlutterFirebaseMessagingService

import android.os.Build
import android.app.NotificationManager
import android.app.NotificationChannel

public class Application: FlutterApplication(), PluginRegistrantCallback {

   override fun onCreate() {
     super.onCreate()
     FlutterFirebaseMessagingService.setPluginRegistrant(this)
     createNotificationChannels()
     SystemAlertWindowPlugin.setPluginRegistrant(this)
   }

   override fun registerWith(registry: PluginRegistry) {
     FirebaseCloudMessagingPluginRegistrant.registerWith(registry)
     SystemAlertWindowPlugin.registerWith(registry.registrarFor("in.jvapps.system_alert_window"))
   }

   fun createNotificationChannels() {
     if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
        val name = "groupChannel"
        val descriptionText = "This is the group channel"
        val importance = NotificationManager.IMPORTANCE_HIGH
        val mChannel = NotificationChannel("59054", name, importance)
        mChannel.description = descriptionText
        val notificationManager = getSystemService(NOTIFICATION_SERVICE) as NotificationManager
        notificationManager.createNotificationChannel(mChannel)
    }
  }
}

Best to post this on the firebase_messaging issues page because it is unrelated to this issue.

I can make a PR, which will have a new 'spawnIsolate' method for 'SchedulerBinding'.
Then it will be possible to call platform methods in this isolate.
This will only help you if you need to call a platform method and get a response.
import 'dart:async';
import 'dart:isolate';

import 'package:flutter/scheduler.dart';
import 'package:flutter/widgets.dart';
import 'package:path_provider/path_provider.dart';

Future<void> _test(SendPort sendPort) async {
  final dir = await getTemporaryDirectory();
  sendPort.send(dir);
}

void main() async {
  WidgetsFlutterBinding.ensureInitialized();

  final Completer<Object> completer = Completer<Object>();
  final RawReceivePort receivePort = RawReceivePort(completer.complete);

  final Isolate isolate = await SchedulerBinding.instance.spawnIsolate(
    _test,
    receivePort.sendPort,
  );

  print(await completer.future);

  receivePort.close();
  isolate.kill();
}

Log
Performing hot restart...
Syncing files to device Pixel 4...
Restarted application in 722ms.
I/flutter (11705): Directory: '/data/user/0/com.example.bug/cache'

I can make a PR, which will have a new 'spawnIsolate' method for 'SchedulerBinding'.
Then it will be possible to call platform methods in this isolate.
This will only help you if you need to call a platform method and get a response.
SchedulerBinding.instance.spawnIsolate

Any chance to test this? If it fits my needs etc ..

@Nailik
Replace this file (Actual for version 1.17.5)

binding.dart
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'dart:async';
import 'dart:isolate';
import 'dart:typed_data';
import 'dart:ui' as ui;

import 'package:flutter/foundation.dart';

import 'asset_bundle.dart';
import 'binary_messenger.dart';
import 'system_channels.dart';

/// Listens for platform messages and directs them to the [defaultBinaryMessenger].
///
/// The [ServicesBinding] also registers a [LicenseEntryCollector] that exposes
/// the licenses found in the `LICENSE` file stored at the root of the asset
/// bundle, and implements the `ext.flutter.evict` service extension (see
/// [evict]).
mixin ServicesBinding on BindingBase {
  @override
  void initInstances() {
    super.initInstances();
    _instance = this;
    _defaultBinaryMessenger = createBinaryMessenger();
    window.onPlatformMessage = defaultBinaryMessenger.handlePlatformMessage;
    initLicenses();
    SystemChannels.system.setMessageHandler(handleSystemMessage);
  }

  /// The current [ServicesBinding], if one has been created.
  static ServicesBinding get instance => _instance;
  static ServicesBinding _instance;

  /// The default instance of [BinaryMessenger].
  ///
  /// This is used to send messages from the application to the platform, and
  /// keeps track of which handlers have been registered on each channel so
  /// it may dispatch incoming messages to the registered handler.
  BinaryMessenger get defaultBinaryMessenger => _defaultBinaryMessenger;
  BinaryMessenger _defaultBinaryMessenger;

  /// Creates a default [BinaryMessenger] instance that can be used for sending
  /// platform messages.
  @protected
  BinaryMessenger createBinaryMessenger() {
    return const _DefaultBinaryMessenger._();
  }

  /// Handler called for messages received on the [SystemChannels.system]
  /// message channel.
  ///
  /// Other bindings may override this to respond to incoming system messages.
  @protected
  @mustCallSuper
  Future<void> handleSystemMessage(Object systemMessage) async { }

  /// Adds relevant licenses to the [LicenseRegistry].
  ///
  /// By default, the [ServicesBinding]'s implementation of [initLicenses] adds
  /// all the licenses collected by the `flutter` tool during compilation.
  @protected
  @mustCallSuper
  void initLicenses() {
    LicenseRegistry.addLicense(_addLicenses);
  }

  Stream<LicenseEntry> _addLicenses() async* {
    // We use timers here (rather than scheduleTask from the scheduler binding)
    // because the services layer can't use the scheduler binding (the scheduler
    // binding uses the services layer to manage its lifecycle events). Timers
    // are what scheduleTask uses under the hood anyway. The only difference is
    // that these will just run next, instead of being prioritized relative to
    // the other tasks that might be running. Using _something_ here to break
    // this into two parts is important because isolates take a while to copy
    // data at the moment, and if we receive the data in the same event loop
    // iteration as we send the data to the next isolate, we are definitely
    // going to miss frames. Another solution would be to have the work all
    // happen in one isolate, and we may go there eventually, but first we are
    // going to see if isolate communication can be made cheaper.
    // See: https://github.com/dart-lang/sdk/issues/31959
    //      https://github.com/dart-lang/sdk/issues/31960
    // TODO(ianh): Remove this complexity once these bugs are fixed.
    final Completer<String> rawLicenses = Completer<String>();
    Timer.run(() async {
      rawLicenses.complete(rootBundle.loadString('LICENSE', cache: false));
    });
    await rawLicenses.future;
    final Completer<List<LicenseEntry>> parsedLicenses = Completer<List<LicenseEntry>>();
    Timer.run(() async {
      parsedLicenses.complete(compute(_parseLicenses, await rawLicenses.future, debugLabel: 'parseLicenses'));
    });
    await parsedLicenses.future;
    yield* Stream<LicenseEntry>.fromIterable(await parsedLicenses.future);
  }

  // This is run in another isolate created by _addLicenses above.
  static List<LicenseEntry> _parseLicenses(String rawLicenses) {
    final String _licenseSeparator = '\n' + ('-' * 80) + '\n';
    final List<LicenseEntry> result = <LicenseEntry>[];
    final List<String> licenses = rawLicenses.split(_licenseSeparator);
    for (final String license in licenses) {
      final int split = license.indexOf('\n\n');
      if (split >= 0) {
        result.add(LicenseEntryWithLineBreaks(
          license.substring(0, split).split('\n'),
          license.substring(split + 2),
        ));
      } else {
        result.add(LicenseEntryWithLineBreaks(const <String>[], license));
      }
    }
    return result;
  }

  @override
  void initServiceExtensions() {
    super.initServiceExtensions();

    assert(() {
      registerStringServiceExtension(
        // ext.flutter.evict value=foo.png will cause foo.png to be evicted from
        // the rootBundle cache and cause the entire image cache to be cleared.
        // This is used by hot reload mode to clear out the cache of resources
        // that have changed.
        name: 'evict',
        getter: () async => '',
        setter: (String value) async {
          evict(value);
        },
      );
      return true;
    }());
  }

  /// Called in response to the `ext.flutter.evict` service extension.
  ///
  /// This is used by the `flutter` tool during hot reload so that any images
  /// that have changed on disk get cleared from caches.
  @protected
  @mustCallSuper
  void evict(String asset) {
    rootBundle.evict(asset);
  }

  Future<Isolate> spawnIsolate<T>(
    FutureOr<void> entryPoint(T message),
    T message, {
    bool paused = false,
    bool errorsAreFatal,
    SendPort onExit,
    SendPort onError,
    String debugName,
  }) {
    assert(
      _isMainIsolate,
      'Can\'t make multiple levels of isolates',
    );

    final RawReceivePort messageReceiver = RawReceivePort(
      (Object receivedMessage) async {
        if (receivedMessage is SendPort) {
          receivedMessage.send(
            _IsolateStarter<T>(
              ui.PluginUtilities.getCallbackHandle(entryPoint),
              message,
            ),
          );
        } else if (receivedMessage is _Message) {
          final ByteData result = await defaultBinaryMessenger.send(
            receivedMessage.channel,
            receivedMessage.message,
          );
          receivedMessage.sendPort.send(result);
        }
      },
    );
    RawReceivePort onExitReceiver;
    onExitReceiver = RawReceivePort(
      (Object message) {
        onExit?.send(message);

        onExitReceiver.close();
        messageReceiver.close();
      },
    );

    return Isolate.spawn(
      _startIsolate,
      messageReceiver.sendPort,
      paused: paused,
      errorsAreFatal: true,
      onExit: onExitReceiver.sendPort,
      onError: onError,
      debugName: debugName,
    );
  }
}

Future<void> _startIsolate<T>(SendPort sendPort) async {
  _sendPortToMainIsolate = sendPort;
  _IsolateBinding();

  final Completer<_IsolateStarter<T>> completer =
      Completer<_IsolateStarter<T>>();

  final RawReceivePort receivePort = RawReceivePort(
    (Object isolateStarter) {
      assert(isolateStarter is _IsolateStarter<T>);
      completer.complete(isolateStarter as _IsolateStarter<T>);
    },
  );

  sendPort.send(receivePort.sendPort);

  final _IsolateStarter<T> isolateStarter = await completer.future;

  receivePort.close();

  final Function function =
      ui.PluginUtilities.getCallbackFromHandle(isolateStarter.callbackHandle);

  await function(isolateStarter.message);
}

SendPort _sendPortToMainIsolate;

bool get _isMainIsolate => _sendPortToMainIsolate == null;

class _IsolateStarter<T> {
  _IsolateStarter(this.callbackHandle, this.message);

  final ui.CallbackHandle callbackHandle;
  final T message;
}

class _Message {
  _Message(
    this.channel,
    this.message,
    this.sendPort,
  );

  final String channel;
  final ByteData message;
  final SendPort sendPort;
}

class _IsolateBinding extends BindingBase with ServicesBinding {}

/// The default implementation of [BinaryMessenger].
///
/// This messenger sends messages from the app-side to the platform-side and
/// dispatches incoming messages from the platform-side to the appropriate
/// handler.
class _DefaultBinaryMessenger extends BinaryMessenger {
  const _DefaultBinaryMessenger._();

  // Handlers for incoming messages from platform plugins.
  // This is static so that this class can have a const constructor.
  static final Map<String, MessageHandler> _handlers =
      <String, MessageHandler>{};

  // Mock handlers that intercept and respond to outgoing messages.
  // This is static so that this class can have a const constructor.
  static final Map<String, MessageHandler> _mockHandlers =
      <String, MessageHandler>{};

  Future<ByteData> _sendPlatformMessage(String channel, ByteData message) {
    final Completer<ByteData> completer = Completer<ByteData>();

    if (_isMainIsolate) {
      // ui.window is accessed directly instead of using ServicesBinding.instance.window
      // because this method might be invoked before any binding is initialized.
      // This issue was reported in #27541. It is not ideal to statically access
      // ui.window because the Window may be dependency injected elsewhere with
      // a different instance. However, static access at this location seems to be
      // the least bad option.
      ui.window.sendPlatformMessage(channel, message, (ByteData reply) {
        try {
          completer.complete(reply);
        } catch (exception, stack) {
          FlutterError.reportError(FlutterErrorDetails(
            exception: exception,
            stack: stack,
            library: 'services library',
            context:
                ErrorDescription('during a platform message response callback'),
          ));
        }
      });
    } else {
      RawReceivePort receivePort;
      receivePort = RawReceivePort(
        (Object message) async {
          assert(message is ByteData);
          completer.complete(message as ByteData);
          receivePort.close();
        },
      );
      _sendPortToMainIsolate.send(
        _Message(channel, message, receivePort.sendPort),
      );
    }

    return completer.future;
  }

  @override
  Future<void> handlePlatformMessage(
    String channel,
    ByteData data,
    ui.PlatformMessageResponseCallback callback,
  ) async {
    ByteData response;
    try {
      final MessageHandler handler = _handlers[channel];
      if (handler != null) {
        response = await handler(data);
      } else {
        ui.channelBuffers.push(channel, data, callback);
        callback = null;
      }
    } catch (exception, stack) {
      FlutterError.reportError(FlutterErrorDetails(
        exception: exception,
        stack: stack,
        library: 'services library',
        context: ErrorDescription('during a platform message callback'),
      ));
    } finally {
      if (callback != null) {
        callback(response);
      }
    }
  }

  @override
  Future<ByteData> send(String channel, ByteData message) {
    final MessageHandler handler = _mockHandlers[channel];
    if (handler != null)
      return handler(message);
    return _sendPlatformMessage(channel, message);
  }

  @override
  void setMessageHandler(String channel, MessageHandler handler) {
    if (handler == null)
      _handlers.remove(channel);
    else
      _handlers[channel] = handler;
    ui.channelBuffers.drain(channel, (ByteData data, ui.PlatformMessageResponseCallback callback) async {
      await handlePlatformMessage(channel, data, callback);
    });
  }

  @override
  void setMockMessageHandler(String channel, MessageHandler handler) {
    if (handler == null)
      _mockHandlers.remove(channel);
    else
      _mockHandlers[channel] = handler;
  }
}

I have flutter 1.21 so i tried to update all the code to compile.
Now there's one Problem left because _IsolateBinding is missing a lot of implementations of BindingBase Methods...
I get sth like
Error: The non-abstract class '_IsolateBinding' is missing implementations for these members:
 - BindingBase with ServicesBinding.SchedulerBinding.addPersistentFrameCallback

around 30 times.
At the end the console prints
/C:/flutter/packages/flutter/lib/src/services/binding.dart:341:7: Error: 'BindingBase' doesn't implement 'SchedulerBinding' so it can't be used with 'ServicesBinding'.
 - 'BindingBase' is from 'package:flutter/src/foundation/binding.dart' ('/C:/flutter/packages/flutter/lib/src/foundation/binding.dart').
 - 'SchedulerBinding' is from 'package:flutter/src/scheduler/binding.dart' ('/C:/flutter/packages/flutter/lib/src/scheduler/binding.dart').
 - 'ServicesBinding' is from 'package:flutter/src/services/binding.dart' ('/C:/flutter/packages/flutter/lib/src/services/binding.dart').
class _IsolateBinding extends BindingBase with ServicesBinding {}

I'm not 100% sure what the call of _IsolateBinding() in _startIsolate does but without i get the common ServicesBinding.defaultBinaryMessenger was accessed before the binding was initialized. error.
I tried to fix this but i think my knowledge of mixins is not that good yet.
Any idea how to fix it?

@Nailik
At the moment, I cannot update flutter to help you

I'm wondering why the other flutter tools implemented by the flutter team (like say devtools) never hit this blocker.
They seem to be fixing bugs only if they get blocked by them.
This is not a severe new feature it's like a P1 framework blocker.

I'm wondering why the other flutter tools implemented by the flutter team (like say devtools) never hit this blocker.
They seem to be fixing bugs only if they get blocked by them.

To be honest, because i think the impact of this issue in real world apps is overrated. I'm just watching the issue because it would make things easier, but there are "workarounds". And many apps and plugins in production are using background functionality.
I'm also not sure if some of the +1s just don't quite understand what the issue is about. (Like https://github.com/flutter/flutter/issues/13937#issuecomment-635683123 ). Imo the official documentation for background isolates & headless runtime (and how to manage it) is still a bit lacking and the firebase messaging stuff is a nightmare, but this has nothing to do with the framework, engine 🤷♂
Maybe just stop bashing the flutter team, there is amazing progress, and this is not something which can't be worked around... And if there is a PR, why not just submit it for the flutter team to consider 🤔

There is a problem with implementing this behavior for isolates.
Example:
There are a main isolate and an engine.
If the isolate needs something, it asks the engine.
If the engine needs something, it asks the isolate.
Now our situation is:
There are 2 isolates and an engine.
If the isolate needs something, it asks the engine.
If the engine needs something, what should it do?
Ask both? Then the whose answer to take?
Or ask just one isolate? Then which one?

@hpoul I have to agree to disagree as many people here are beginners to android/ios programming and they chose their first framework for app development to be flutter.
So it's important that someone realises this and fixes these kind of issues where you're blocked and forced by the framework to do complex "workarounds" or post an official workaround with an example as obviously beginners won't know what to learn or even where to start because there's no well explained workarounds in the comments here or anywhere on stackoverflow or other blogs. There's only stuff pointing to some packages which may or may not have implemented this correctly.
People who have tons of experience working with services, background task queues etc., in both android and ios, who can easily work around this are not the target demographic for flutter.
And coming to off topic comments like that I'm sure there are false +1s in every other issue, which doesn't change the order of the issue priorities by much.

Hi @phanirithvij
flutter_isolate was linked above and has been around since February of 2019. It is also not complex. I'll copy the README which seems straightforward:

FlutterIsolate allows creation of an Isolate in flutter that is able to use flutter plugins. It creates the necessary platform specific bits (FlutterBackgroundView on android & FlutterEngine on iOS) to enable the platform channels to work inside an isolate.

Android
iOS
Description

FlutterIsolate.spawn(entryPoint,message)
:white_check_mark:
:white_check_mark:
spawns a new FlutterIsolate

flutterIsolate.pause()
:white_check_mark:
:white_check_mark:
pauses a running isolate

flutterIsolate.resume()
:white_check_mark:
:white_check_mark:
resumed a paused isoalte

flutterIsolate.kill()
:white_check_mark:
:white_check_mark:
kills a an isolate

I say straightforward since these methods have the same names as the ones in the "original" Isolate class so if you already know how to use isolates from the official documentation, it shouldn't be difficult to understand how to use this as a drop-in replacement, at least for those methods listed above.
(Note: I, too, would like this issue fixed officially for reasons I have already stated in an earlier comment, some of which I share with you, but I also wouldn't call this workaround "complex".)
In response to @nikitadol 's comment, flutter_isolate creates a separate engine for each isolate so it doesn't run into the same problem. Of course, it's not ideal to have to create a new engine per isolate, that is one reason why I'm sure the Flutter team could do a better job. Similarly, the alternative solution of piping all plugin method calls through the main isolate would create a bottleneck that runs counter to the goals of isolates in the first place, so I'm sure the Flutter team could do a better job of that, too.
I agree with those who have said the Flutter team might have higher priority issues to address first, but based on what we have seen before with the plugin architecture, it may also be that fixing this requires a major architectural change, and several other open issues also require a major architectural change, and weighing all of these different requirements could take time before they can even get around to addressing this particular issue.

@ryanheise

flutter_isolate creates a separate engine for each isolate

You say the Flutter team can do this, but how?
The questions I asked above remain valid.
If the solution is to create a new engine, then you can simply use your plugin

(Note, flutter_isolate is not my plugin, but since I also needed this feature I decided to contribute to the project.)
What I meant by "a better job" is that it is probably not necessary to spin up ALL of the machinery in a FlutterEngine just to communicate with plugins, but at the moment, the necessary infrastructure is all tied to the engine so that's why currently flutter_isolate has to spin up a whole engine just to get access to the plugins.

(Note, flutter_isolate is not my plugin, but since I also needed this feature I decided to contribute to the project.)

Sorry, I didn't notice that this is a fork

flutter_isolate has to spin up a whole engine just to get access to the plugins.

If you specify that 'is_background_view = true', then the engine does not start rendering - which means that the engine does not start completely

The precise API for running headless is different for iOS and Android v1 and Android v2, but essentially this is what flutter_isolate already does. There's still a difference in overhead between a normal isolate and a FlutterEngine, though, so getting the overheads down further is only something that the Flutter team would be able to do. It's not just an overhead in startup time, it's also an overhead in memory usage.

So you just suggest optimizing the creation of a new engine to work in the background?
Then this probably does not apply to this issue

I'm not suggesting that, I'm just pointing out that the flutter_isolate workaround has this limitation because of the overheads which can't be avoided. I'm not suggesting that the Flutter team should try to make the workaround more efficient, I think that ideally there would be an architectural change to the way that plugins are loaded so that it wouldn't be necessary to instantiate a FlutterEngine.

If the engine is 1, and isolates are greater than 1, then we return to this question:
https://github.com/flutter/flutter/issues/13937#issuecomment-667314232

I'm not trying to actually come up with a solution for this new lightweight architecture, but if you're asking me, I can think of two variations of lightweight solution:

Make method channels a bit like server sockets by making them accept connections from multiple clients. So a method channel in the platform side of the plugin could accept connections from multiple isolates, and then once connected could send messages to the right isolate.
Create a new plugin registry etc. for each isolate BUT within the same FlutterEngine,

if you're asking me

Yes, I ask

could send messages to the right isolate.

I think this is a bad solution since 'to the right isolate' is a relative concept

Create a new plugin registry etc. for each isolate BUT within the same FlutterEngine,

This could be a good option, but only if the engine always registers plugins by itself.
But this only happens if all plugins in the GeneratedPluginRegistrant.
https://github.com/flutter/engine/blob/f7d241fd8a889fadf8ab3f9d57918be3d986b4be/shell/platform/android/io/flutter/embedding/engine/FlutterEngine.java#L330-L344
Creating additional callbacks is not an obvious behavior.

I think this is a bad solution since 'to the right isolate' is a relative concept

It could be a bad solution, but it can't be bad for that reason. The "right isolate" is unambiguous in the same way a server socket handles client connections in an unambiguous way.

This could be a good option, but only if the engine always registers plugins by itself.

Which would be a good thing to ensure always happens. Currently the mechanism is a bit flaky.

It could be a bad solution, but it can't be bad for that reason. The "right isolate" is unambiguous in the same way a server socket handles client connections in an unambiguous way.

Then each plugin must take into account that it can be accessed from different isolates - bad

Yep.

Let me add my two cents to this discussion... Having commercially worked with Flutter for around two years now, the most important thing that I learnt was that Flutter is a UI toolkit. There wouldn't be anything surprising in this statement - it even says that on the main Flutter website - if people didn't forget about this.
Flutter is a UI toolkit, and what it does well is UI. If you're trying to create a CPU-heavy application that relies on expensive data processing or uses complicated algorithms and consumes larga amounts of data, then you should NOT use a UI toolkit for this purpose. Business Logic should not be mixed with the UI.
Of course, with Dart we can do much more than just UI, but let's not forget the main purpose fo this toolkit.
I've tried a number of approaches, including using headless Flutter - which proved to be a total disaster. Not because it doesn't work, but because we have problems explaining to other developers how it works. The maintenance burdon it adds up is just not worth it.
If your application really does need the processing power, IMHO, you have three options:

move your logic to the server;
move your logic to the native platform - you can use Kotlin Native for this - and use Event/Method channels to only provide the View Models to your UI (I think that's what OLX did: https://tech.olx.com/fast-prototypes-with-flutter-kotlin-native-d7ce5cfeb5f1);
don't use Flutter and look for another framework (Xamarin/Blazor might be a good fit for this as C# offers a more sophisticated multithreading environment than Dart).

All of the above have their shortcomings, but trying to stretch a library/framework to do what it wasn't really designed for isn't ideal either.

Had not the isolate_handler solved this problem？

@hasonguo It uses Ryan's solution in the background, it just adds an extra layer to make handling isolates (any isolates, actually) easier by adding the boilerplate for communication so that you don't need to set it up manually every time.
We're going in circles. Yes, there are solutions, both plugins actually. As Ryan pointed out, there is an overhead involved in making the channel communications possible. More likely than not, it's exactly this overhead that made the Flutter team avoid adding it automatically to the original Isolate, as you don't need the support in many (most?) cases.
Still, it's not as clean a solution as it could get. If the Flutter team would be able to provide channel communication without the overhead, directly and automatically, right out of the box with any basic Isolate, the whole discussion would be moot. However, the opposite also holds true: they are not really pursuing change for the sake of change. If there is a viable solution in a package or plugin, why spend time and add weight to the core?
So, yes, in a broader sense, you're completely right. Being the current maintainer of isolate_handler, I don't feel right to advertise it all the time (I just mentioned it once or twice during this discussion, and also received negative reactions, which were certainly not warranted at that time, but I don't want to overdo it, either). But both these plugins solve the issue all right, as far as I can tell. And, if I am correct in my assumption that the core team isn't really interested in changing the situation because of what I just described, it will stay that way in the future.

Create a new plugin registry etc. for each isolate BUT within the same FlutterEngine,

This could be a good option, but only if the engine always registers plugins by itself.
But this only happens if all plugins in the GeneratedPluginRegistrant.
https://github.com/flutter/engine/blob/f7d241fd8a889fadf8ab3f9d57918be3d986b4be/shell/platform/android/io/flutter/embedding/engine/FlutterEngine.java#L330-L344
Creating additional callbacks is not an obvious behavior.

Another improvement to this second idea is to make have plugins loaded lazily in the new isolate. Normally, if a project depends on many plugins, the GeneratedPluginRegistrant will be quite large and depending on how efficient a plugin's initialisation routine is, this may contribute to some of the overheads. Exactly how to implement this is another story. Platform channels aren't owned by plugins, so currently sending a message over a channel can't be used to trigger the platform side of a plugin to be instantiated. So either method channels would need to be associated with plugins, or there would need to be an additional mechanism that the Dart side of a plugin could call on to initialise itself lazily before the first method channel is created, and that mechanism could trigger the instantiation of the platform side of that plugin within the isolate.

code sample
import 'dart:async';
import 'dart:isolate';

import 'package:flutter/material.dart';
import 'package:flutter/services.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Material App',
      theme: ThemeData.dark(),
      home: Home(),
    );
  }
}

class Home extends StatefulWidget {
  static const platform = const MethodChannel('samples.flutter.dev/battery');

  @override
  _HomeState createState() => _HomeState();
}

class _HomeState extends State<Home> {
  String _batteryLevel = 'Unknown battery level.';

  Isolate _isolate;
  bool _running = false;
  static int _counter = 0;
  String notification = "";
  ReceivePort _receivePort;

  Future<void> _getBatteryLevel() async {
    String batteryLevel;
    try {
      final int result = await Home.platform.invokeMethod('getBatteryLevel');
      batteryLevel = 'Battery level at $result % .';

      print('result $result');
    } on PlatformException catch (e) {
      batteryLevel = "Failed to get battery level: '${e.message}'.";
    }

    setState(() {
      _batteryLevel = batteryLevel;
    });
  }

  void _start() async {
    _running = true;
    _receivePort = ReceivePort();
    _isolate = await Isolate.spawn(_checkBattery, _receivePort.sendPort);
    _receivePort.listen(_handleMessage, onDone: () {
      print("done!");
    });
  }

  void _stop() {
    if (_isolate != null) {
      setState(() {
        _running = false;
        notification = '';
      });
      _receivePort.close();
      _isolate.kill(priority: Isolate.immediate);
      _isolate = null;
    }
  }

  void _handleMessage(dynamic data) {
    print('RECEIVED: ' + data);
    setState(() {
      notification = data;
    });
  }

  void _checkBattery(SendPort sendPort) async {
    final int result = await Home.platform.invokeMethod('getBatteryLevel');
    print(result);
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
        appBar: AppBar(
          title: Text('Platform Channel in Isolate'),
        ),
        body: Center(
          child: Material(
            child: Center(
              child: Column(
                mainAxisAlignment: MainAxisAlignment.spaceEvenly,
                children: [
                  RaisedButton(
                    child: Text('Get Battery Level'),
                    onPressed: _running ? _stop : _start,
                  ),
                  Text(_batteryLevel),
                ],
              ),
            ),
          ),
        ));
  }
}

flutter doctor -v
[✓] Flutter (Channel master, 1.22.0-10.0.pre.451, on Mac OS X 10.15.7 19H2
    x86_64, locale en-GB)
    • Flutter version 1.22.0-10.0.pre.451 at
      /Users/tahatesser/Code/flutter_master
    • Framework revision 79400b2462 (46 minutes ago), 2020-10-01 09:06:46 -0400
    • Engine revision 612acf349e
    • Dart version 2.11.0 (build 2.11.0-180.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /Users/tahatesser/Code/sdk
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = /Users/tahatesser/Code/sdk
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.0.1, Build version 12A7300
    • CocoaPods version 1.9.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 50.0.1
    • Dart plugin version 193.7547
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.49.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.15.0

[✓] Connected device (4 available)
    • RMX2001 (mobile) • EUYTFEUSQSRGDA6D • android-arm64  • Android 10 (API 29)
    • macOS (desktop)  • macos            • darwin-x64     • Mac OS X 10.15.7
      19H2 x86_64
    • Web Server (web) • web-server       • web-javascript • Flutter Tools
    • Chrome (web)     • chrome           • web-javascript • Google Chrome
      85.0.4183.121

• No issues found!

@TahaTesser
In your example, the error is not related to this issue
E/flutter ( 7814): [ERROR:flutter/lib/ui/ui_dart_state.cc(177)] Unhandled Exception: Invalid argument(s): Isolate.spawn expects to be passed a static or top-level function

Hi @nikitadol
Thanks, updated example, it doesn't throw an exception, is that correct? (I never needed to use before)
Can you please provide a complete reproducible minimal code sample
Thank you

@TahaTesser

code sample
import 'package:battery/battery.dart';
import 'package:flutter/foundation.dart';
import 'package:flutter/widgets.dart';

Future<void> main() async {
  await printBatteryLevel();
  await compute(printBatteryLevel, null);
}

Future<void> printBatteryLevel([dynamic message]) async {
  WidgetsFlutterBinding.ensureInitialized();
  print(await Battery().batteryLevel);
}

logs
Launching lib/main.dart on Pixel 4 in debug mode...
Running Gradle task 'assembleDebug'...
✓ Built build/app/outputs/flutter-apk/app-debug.apk.
Installing build/app/outputs/flutter-apk/app.apk...
Waiting for Pixel 4 to report its views...
Debug service listening on ws://<IP_ADDRESS>:50709/-SPs_6AmL2Q=/ws
Syncing files to device Pixel 4...
I/flutter ( 8708): 39
E/flutter ( 8708): [ERROR:flutter/lib/ui/ui_dart_state.cc(166)] Unhandled Exception: Exception: UI actions are only available on root isolate.
E/flutter ( 8708): #0      Window._nativeSetNeedsReportTimings (dart:ui/window.dart:1003:86)
E/flutter ( 8708): #1      Window.onReportTimings= (dart:ui/window.dart:996:29)
E/flutter ( 8708): #2      SchedulerBinding.addTimingsCallback (package:flutter/src/scheduler/binding.dart:272:14)
E/flutter ( 8708): #3      SchedulerBinding.initInstances (package:flutter/src/scheduler/binding.dart:209:7)
E/flutter ( 8708): #4      ServicesBinding.initInstances (package:flutter/src/services/binding.dart:27:11)
E/flutter ( 8708): #5      PaintingBinding.initInstances (package:flutter/src/painting/binding.dart:23:11)
E/flutter ( 8708): #6      SemanticsBinding.initInstances (package:flutter/src/semantics/binding.dart:24:11)
E/flutter ( 8708): #7      RendererBinding.initInstances (package:flutter/src/rendering/binding.dart:32:11)
E/flutter ( 8708): #8      WidgetsBinding.initInstances (package:flutter/src/widgets/binding.dart:257:11)
E/flutter ( 8708): #9      new BindingBase (package:flutter/src/foundation/binding.dart:59:5)
E/flutter ( 8708): #10     new _WidgetsFlutterBinding&BindingBase&GestureBinding (package:flutter/src/widgets/binding.dart)
E/flutter ( 8708): #11     new _WidgetsFlutterBinding&BindingBase&GestureBinding&SchedulerBinding (package:flutter/src/widgets/binding.dart)
E/flutter ( 8708): #12     new _WidgetsFlutterBinding&BindingBase&GestureBinding&SchedulerBinding&ServicesBinding (package:flutter/src/widgets/binding.dart)
E/flutter ( 8708): #13     new _WidgetsFlutterBinding&BindingBase&GestureBinding&SchedulerBinding&ServicesBinding&PaintingBinding (package:flutter/src/widgets/binding.dart)
E/flutter ( 8708): #14     new _WidgetsFlutterBinding&BindingBase&GestureBinding&SchedulerBinding&ServicesBinding&PaintingBinding&SemanticsBinding (package:flutter/src/widgets/binding.dart)
E/flutter ( 8708): #15     new _WidgetsFlutterBinding&BindingBase&GestureBinding&SchedulerBinding&ServicesBinding&PaintingBinding&SemanticsBinding&RendererBinding (package:flutter/src/widgets/binding.dart)
E/flutter ( 8708): #16     new _WidgetsFlutterBinding&BindingBase&GestureBinding&SchedulerBinding&ServicesBinding&PaintingBinding&SemanticsBinding&RendererBinding&WidgetsBinding (package:flutter/src/widgets/binding.dart)
E/flutter ( 8708): #17     new WidgetsFlutterBinding (package:flutter/src/widgets/binding.dart)
E/flutter ( 8708): #18     WidgetsFlutterBinding.ensureInitialized (package:flutter/src/widgets/binding.dart:1229:7)
E/flutter ( 8708): #19     printBatteryLevel (package:bug/main.dart:11:25)
E/flutter ( 8708): #20     _IsolateConfiguration.apply (package:flutter/src/foundation/_isolates_io.dart:83:34)
E/flutter ( 8708): #21     _spawn.<anonymous closure> (package:flutter/src/foundation/_isolates_io.dart:90:65)
E/flutter ( 8708): #22     Timeline.timeSync (dart:developer/timeline.dart:163:22)
E/flutter ( 8708): #23     _spawn (package:flutter/src/foundation/_isolates_io.dart:87:35)
E/flutter ( 8708): #24     _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:304:17)
E/flutter ( 8708): #25     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)
E/flutter ( 8708):

flutter doctor -v
[✓] Flutter (Channel stable, 1.20.4, on Mac OS X 10.15.6 19G2021, locale en-BY)
    • Flutter version 1.20.4 at /Users/nikitadold/development/flutter
    • Framework revision fba99f6cf9 (2 weeks ago), 2020-09-14 15:32:52 -0700
    • Engine revision d1bc06f032
    • Dart version 2.9.2

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.0)
    • Android SDK at /Users/nikitadold/Library/Android/sdk
    • Platform android-30, build-tools 30.0.0
    • Java binary at: /Users/nikitadold/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/193.6626763/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.0.1, Build version 12A7300
    • CocoaPods version 1.9.3

[!] Android Studio (version 4.0)
    • Android Studio at /Users/nikitadold/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/193.6626763/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] IntelliJ IDEA Ultimate Edition (version 2020.2.2)
    • IntelliJ at /Users/nikitadold/Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app
    • Flutter plugin installed
    • Dart plugin version 202.7319.5

below you can find an alternative example for this behavior,
please not that, to my knowledge *this is expected*
since only one isolate is meant to be able to reach the platform
(that isolate is the one  running main)
I'll try to find the reference for my statement

Click to expand
import 'dart:async';
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:shared_preferences/shared_preferences.dart';

/// please add [shared_preferences: ^0.5.12] in [pubspec.yaml]

typedef Future<int> CounterCallback(int counter);

void main() => runApp(const MyApp());

class MyApp extends StatelessWidget {
  const MyApp() : super(key: const Key('MyApp'));
  @override
  Widget build(BuildContext context) => MaterialApp(home: MyHomePage());
}

class MyHomePage extends StatefulWidget {
  MyHomePage() : super(key: const Key('MyHomePage'));

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage>
    with SingleTickerProviderStateMixin {
  TabController _controller;

  @override
  void initState() {
    super.initState();
    _controller = TabController(length: 2, vsync: this);
  }

  @override
  void dispose() {
    _controller?.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) => SafeArea(
        child: FutureBuilder<SharedPreferences>(
          future: SharedPreferences.getInstance(),
          builder:
              (BuildContext context, AsyncSnapshot<SharedPreferences> prefs) =>
                  !prefs.hasData
                      ? const CircularProgressIndicator()
                      : Scaffold(
                          appBar: AppBar(
                            title: TabBar(
                              controller: _controller,
                              tabs: const <Widget>[
                                Icon(Icons.error_outline),
                                Icon(Icons.done),
                              ],
                            ),
                          ),
                          body: TabBarView(
                            controller: _controller,
                            children: [
                              SharedPrefsWidget(
                                instance: prefs.data,
                                title: 'with Isolate',

                                /// [here is the only difference!]
                                callback: (int counter) async =>
                                    compute((int i) => i + 1, counter),
                                key: const Key('isolate example'),
                              ),
                              SharedPrefsWidget(
                                instance: prefs.data,
                                title: 'vanilla',
                                callback: (int counter) async => counter + 1,
                                key: const Key('vanilla example'),
                              ),
                            ],
                          ),
                        ),
        ),
      );
}

class SharedPrefsWidget extends StatefulWidget {
  const SharedPrefsWidget({
    this.instance,
    this.callback,
    this.title,
    Key key,
  }) : super(key: key);
  final SharedPreferences instance;
  final CounterCallback callback;
  final String title;

  @override
  _SharedPrefsWidgetState createState() => _SharedPrefsWidgetState();
}

class _SharedPrefsWidgetState extends State<SharedPrefsWidget> {
  StreamController<int> _controller;

  @override
  void initState() {
    _controller = StreamController<int>();
    super.initState();
  }

  @override
  void dispose() {
    _controller?.close();
    super.dispose();
  }

  void _add(int counter) async =>
      _controller.sink.add(await widget.callback(counter));
  @override
  Widget build(BuildContext context) => StreamBuilder<int>(
        stream: _controller.stream,
        initialData: 0,
        builder: (BuildContext context, AsyncSnapshot<int> counter) => Scaffold(
          appBar: AppBar(
            title: Text(widget.title),
          ),
          body: Center(
            child: counter.hasError
                ? const Icon(Icons.error)
                : Text('counter: ${counter.data}'),
          ),
          floatingActionButton: FloatingActionButton(
            onPressed: () => counter.hasError ? {} : _add(counter.data),
            tooltip: 'Increment',
            child: const Icon(Icons.add),
          ),
        ),
      );
}

@iapicca
In your example, the error is not related to this issue
E/flutter (23174): [ERROR:flutter/lib/ui/ui_dart_state.cc(166)] Unhandled Exception: Invalid argument(s): Illegal argument in isolate message : (object is a closure - Function '<anonymous closure>':.)

@iapicca
In your example, the error is not related to this issue
E/flutter (23174): [ERROR:flutter/lib/ui/ui_dart_state.cc(166)] Unhandled Exception: Invalid argument(s): Illegal argument in isolate message : (object is a closure - Function '<anonymous closure>':.)

you are right mine is not a good example, please ignore it

If the isolate is spawned via a new FlutterEngine on the host platform this will work.  The next version of Flutter will have a new API available that is called FlutterEngineGroup which allows one to create FlutterEngines at a fraction of the cost they used to be (~180Kb).  The host platform can then be used as a dispatcher of messages between multiple isolates.  We do something similar here with add-to-app: https://github.com/flutter/samples/tree/master/add_to_app/multiple_flutters   The same approach would work for a plugin I believe.  I haven't tried setting the entry point for a spawned flutter engine to a function in a plugin, yet.

Thanks @gaaclarke - this is basically what the flutter_isolate plugin does at the moment (just with the "old" approach of instantiating a new FlutterEngine), however a more lightweight approach would definitely be welcome. It looks like the "new" FlutterEngineGroup API isn't yet available, you mentioned "the next version", does that mean there's a targeted version for release, or just "it's in the pipeline"?
Also in case others are interested, I've created a repository to track progress reimplementing the flutter_isolate plugin functionality to take advantage of this new API:
https://github.com/nmfisher/flutter_isolate_engine_group
Obviously it doesn't offer any benefit at the moment, but I can come back to it when the new API is released.

Also in case others are interested, I've created a repository to track progress reimplementing the flutter_isolate plugin functionality to take advantage of this new API:

Nice work! I have been maintaining flutter_isolate for the past year or so since I needed it as a dependency for the audio_service plugin. The original author is very open to assigning rights to someone else who is interested in taking over the project, so you should definitely consider contacting him. If I had the time, I would probably take flutter_isolate in the same direction to take advantage of this new API but we probably don't need to duplicate the effort here so it may be something worth thinking about.

And, just another support, with isolate_handler we actively rely on the functionality provided by flutter_isolate, watching closely any progress in this area. :-)

@deakjahn I was also reminded of your package when I wrote the previous comment, and perhaps you might also be interested in getting access to the flutter_isolate repository. I have just posted this issue on flutter_isolate repo to see if we can find someone interested in continuing with the flutter_isolate name.

@nmfisher @ryanheise

Nice work! I have been maintaining flutter_isolate for the past year or so since I needed it as a dependency for the audio_service plugin. The original author is very open to assigning rights to someone else who is interested in taking over the project, so you should definitely consider contacting him

More than happy if you want to push to this repo. Although I think @ryanheise has full control to add anyone to this repo too.

It looks like the "new" FlutterEngineGroup API isn't yet available, you mentioned "the next version", does that mean there's a targeted version for release, or just "it's in the pipeline"?

FlutterEngineGroup has been released in Flutter 2.0.  Sample code can be found here: https://github.com/flutter/samples/tree/master/add_to_app/multiple_flutters

Sorry @gaaclarke, that was lazy phrasing on my part - I meant to ask when it will be stable (since it's marked as "experimental" at https://flutter.dev/docs/development/add-to-app/multiple-flutters).

Sorry @gaaclarke, that was lazy phrasing on my part - I meant to ask when it will be stable (since it's marked as "experimental" at https://flutter.dev/docs/development/add-to-app/multiple-flutters and the source says "Do not ship production code with this".

Sure, it doesn't say "Do not ship production code with this."  It tries to communicate the risk of using it.  The reason is that it is a very complicated feature that depends on an experimental Dart feature.  We have tested it locally and have had a few customers ship products using it already.  Those customers have in some cases had to work with us to fix bugs they found.  Also, it's a warning about the API changing, which should be a slight nuance at worst.
If someone wants to use the feature in production they should first test the feature mimicking their use-case.  Then they should be willing to work with our team to fix any issues they run into.  Depending on the resources / skill of your team that might not be feasible honestly.
So basically, just make sure you do the correct calculus that factors in the risk / reward before using it.  That will be the case until more customers ship with it.  Until then we've decided to error on the side of caution.  If no one uses it, then a year later we say, "Well, no one has had any issues.  Let's remove the warning," then people start using it; that defeats the purpose of the warning obviously.  I hope that makes it clear.

Thanks for taking the time to explain @gaaclarke, that's clearer now.

And, just another support, with isolate_handler we actively rely on the functionality provided by flutter_isolate, watching closely any progress in this area. :-)

For now, the isolate_handler is a good solution:
https://pub.dev/packages/isolate_handler

All the solutions above seem not to have support  for Desktop.
Any ideas where to go from here?

There is a design doc out for FFI Platform Channels, they wouldn't have the same limitation about which isolate can call them. They, however, are a bit lower-level and require that you handle your own thread-safety and respond synchronously.

blank screen on Android(((
    <uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
    <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION"/>
    <uses-permission android:name="android.permission.INTERNET" />

   <application
        android:label="CoffeesUP"
        android:icon="@mipmap/ic_launcher"
        android:requestLegacyExternalStorage="true"
        android:usesCleartextTraffic="true">

E/flutter ( 1580): #1      MethodChannel._invokeMethod
package:flutter/…/services/platform_channel.dart:156
E/flutter ( 1580): <asynchronous suspension>
E/flutter ( 1580): #2      AndroidViewController.create
package:flutter/…/services/platform_views.dart:771
E/flutter ( 1580): <asynchronous suspension>
E/flutter ( 1580):
E/flutter ( 1580): [ERROR:flutter/shell/common/shell.cc(93)] Dart Unhandled Exception: PlatformException(channel-error, Unable to establish connection on channel., null, null), stack trace: #0      WebSettingsHostApi.dispose
package:webview_flutter_android/src/android_webview.pigeon.dart:879
E/flutter ( 1580): <asynchronous suspension>
E/flutter ( 1580):
E/flutter ( 1580): [ERROR:flutter/shell/common/shell.cc(93)] Dart Unhandled Exception: PlatformException(channel-error, Unable to establish connection on channel., null, null), stack trace: #0      WebViewHostApi.dispose
package:webview_flutter_android/src/android_webview.pigeon.dart:180
E/flutter ( 1580): <asynchronous suspension>
E/flutter ( 1580): #1      WebViewHostApiImpl.disposeFromInstance
package:webview_flutter_android/src/android_webview_api_impls.dart:109
E/flutter ( 1580): <asynchronous suspension>

Hi all,
We are using ffmpeg_kit_flutter to encode large videos in mobile clients before uploading them to the server.
Being able to do this in an isolate will largely increase UI performance of the app. the isolate will let us know when the encoding is finished and will return the saved file path.
However because of the root cause of this issue, this can not work now.
Do you have any suggestions? I thought the new flutter isolate mechanisms allowing to create flutter engines with a fraction of the cost would have helped. Is it something else?

@aytunch Considering that nothing has changed since, I won't refrain from suggesting two plugins (one of which I'm a maintainer of but it actually relies on the other one to do that part of the heavy lifting):
https://pub.dev/packages/flutter_isolate
https://pub.dev/packages/isolate_handler

@aytunch if I were making that project today I'd either:

Issue the information to encode via a platform channel to the native code and call ffmpeg from there.
Use ffmpeg_kit_flutter from a background isolate but use dart ports to communicate with that isolate instead of platform channels.

@gaaclarke  thank you for the insight into how to handle this currently. we are all looking for the ability to do background processing (to keep our FPS at 60+). IMO all these limitations with isolates will cause more bugs than just giving us typical threads. why? we cannot do all background work in elegant steps. instead I have to shuffle data between isolates and maintain the state of my process. the limitation of shared memory is a mistake. the limitation of not using plugins is even more painful. so instead of keeping our background processing clean, I'm now forced to do several work arounds.
i'm building a messaging app. I need to be able to encrypt and send over the network. I need to be able to receive messages, decrypt, and present. I created an isolate pool to receive messages. I'm now re-designing the app to work around all the isolate limitations. I need extra queues, extra state, and moving data from isolates to main.
Good luck everyone! Cheers.

IMO all these limitations with isolates will cause more bugs than just giving us typical threads. why? we cannot do all background work in elegant steps.

@j3g IMO I would contend that you are avoiding alot of potential bugs by not having to do all the required work yourself to safely use shared mutable state from multiple threads.
I'm also not clear as to how the limitation of not being able to call platform channels from non-main isolates affects your use case of needing to decrypt messages off the main isolate?

Thread-safe data types are your friend. I've never had threading issues, because I follow thread-safe design patterns. Because I'm a season developer I don't need artificial limitations for "safety". I would argue that because of isolate limitations, I am forced to add a lot of extra code to get things done. All that extra code gives more probability of bugs. My code is clean and sequential with threads, but more complex with isolates.
For Example. Why am I on this discussion? Because my flutter app crashes from the generic error “Exception: Null check operator used on a null value”. I figured out it happens when I tried to call my plugin within the isolate. I spent many hours troubleshooting thinking that it was my error. Now I discover it is yet another isolate limitation. I didn’t see this limitation in the official docs. So after today I will redesign my elegant code to something more complex...
@maks is there a forum I can go and discuss/learn further? i'm not finding quality on stackoverflow.

@j3g John, still, the question arises that why should each any ever isolate make it possible to call platform channels when it comes with a cost. The cost is smaller with the changes than it was before, yes, but it's still a cost. If many isolates don't need that extra functionality, why would we accept the incurred costs?
In every single part of Flutter development we accept the obvious fact that many of the important functionality comes in the form of plugins and packages, not in the core, for two reasons. One is to keep the engine as lean as possible (and it still isn't what I'd call lean, of course, in this day and age we and our users don't care that much whether an app is 1 MB or 20, but still). The other is to make sure a much larger community can build useful parts of the jigsaw puzzle and we don't need to wait for the engine team to deliver everything.
I came to be included in this channel-isolate story exactly for these reasons: I needed a solution to call a plugin. It didn't work out of the box, I looked around for solutions and found out that solutions do exist. It just happened that the original creator of one of the plugins mentioned above asked me whether, for her lack of time to maintain it, I'd be interested in taking it over and I did. It wasn't my original idea, I don't mention it from time to time because I want to take credit for it or brag about it but simply, as in many other areas of Flutter development, there is a solution with the usual addition of a single line in pubspec. Only those need to use it who need it, the rest of the isolates will not suffer from the additional cost.

I don't know why you @ replied me. You didn't respond to my talking points. I honestly cannot follow your points. It seems we can agree that plugins are important. As a senior super duper elite coder I need something more than isolates. And I would appreciate if my isolate didn't crash when I call a plugin. I've already moved on and solved this issue with many elite coder hacking skills. I'm hoping dart will add more advanced threads in the future. Cheers.

@j3g Simply because I mentioned a solution to your problem. There are two plugins currently that make isolates able to call plugins. Without crashes. It's up to you to use them and have a perfect, functioning solution, or not.
I simply tried to explain why it isn't probably necessary to have this functionality in the core isolates and why it's enough to have plugins fill the gap.

@deakjahn one (of the many) but a very strong one, for which it could be useful to call platform channel methods in another isolate is image compression.
I may want to use the native code to have the fastest possible image compression, and I would like to do that in an isolate.
With so many likes and people requesting this my humble humble humble opinione is that P5 does not reflect well the issue and the demand.
Best regards

@mariopepe I agree there are valid reasons for using platform channels in a non-main isolate, but I would suggest that for the use case of image compression, a much better solution would be to use Dart FFI, which does work well within any Isolate.

@mariopepe Yes, I know. I use isolates for similar purposes all the time. It's very easy to do and it works fine.
It's an awkward situation. I don't want to constantly advertise that I'm affiliated with the solution. It wasn't my invention, it's not part of my ego :-), I wast just an active user that needed it badly around the time when the original maintainer wanted to move on to other projects. There's no active development, it's on the back burner, it just works, that's all.
So, I don't want to advertise it actively. I just have the feeling, every time another post appears in this thread, that many who desperately ask for this functionality are not aware of the fact that it is readily available, just that it isn't in core. There are many such things in the Flutter world, tons of packages that are readily availabe but not in core. Why is this one any different?

I think isolate and background thread  are different instances one for dart and one for native.

@deakjahn Can we use the packages you have suggested for running ffmpeg_kit?

I can't see why not. I never tried that package myself but I used many other plugins that have channel calls without a single issue.

Update: There are no current plans to implement this at this time (sorry), mostly just because we're focused on higher-priority issues like improving graphics performance.
There exist a few workarounds, which have been mentioned several times in this thread: isolate_handler and flutter_isolate are the main ones.
If you don't need to talk to Android or iOS code, but can talk to C code instead, then dart:ffi may be an interesting alternative as well; that should work fine from a normal Dart isolate.
If anyone is interested in working on implementing this feature directly in the engine, the first step would be to create a design doc to describe the desired approach. Alternatively, to help this issue be considered in a coming planning cycle, it would be very useful to collect the use cases for this issue in one place, e.g. a Google Doc (you can use the same template and techniques as for design docs) or a comment on this issue. There are many use cases listed in this issue but they are spread among 150+ comments so it is hard to see at a glance the whole picture. Details on why specific use cases aren't sufficiently handled by the workarounds above would be most useful (e.g. Sqlite is mentioned as one use case, but the best way to use Sqlite from an isolate is dart:ffi, not message channels).
In either case, please feel free to reach out on our Discord (link in the repo's contributing guidelines, start with the #hackers-new channel) to learn more about our processes if you are interested in contributing.
Finally, I would like to remind everyone that +1'ing an issue does nothing, but using the Thumbs-Up emoji on the first comment help surface the issue in our searches of high-demand issues, as described on our wiki.

I haven't thought much about this but if I were to implement this I'd probably create a static function that can setup a BinaryMessenger on a background isolate which will listen to a dart port to wake up with a integer message_id and check against its stored Completers for one associated with the message_id.  On the host side the engine will store the result's binary data and send the message_id to the port so the target isolate could grab ownership of the results binary data without copying it.

I've created a design document that solves this issue.  Please give it a look if you are interested in this issue to make sure it addresses your needs: Isolate Platform Channels.

Support for iOS has just landed if anyone wants to play with it.  Support for Android and Desktop has been written and hopefully will land next week.

I wrote sample code for the feature if you want to see the new feature working: https://github.com/gaaclarke/background_isolate_channels_sample
The sample code is still in flux until I can get it landed in the official samples repo.

I'm marking this closed as per our guidelines that issues are closed when fixes hit master, not stable.  The only outstanding tasks are documentation tasks that can't land until this feature hits stable.

I haven't thought much about this but if I were to implement this I'd probably create a static function that can setup a BinaryMessenger on a background isolate which will listen to a dart port to wake up with a integer message_id and check against its stored Completers for one associated with the message_id. On the host side the engine will store the result's binary data and send the message_id to the port so the target isolate could grab ownership of the results binary data without copying it.

It all starts with an idea. Thanks for this @gaaclarke Can't wait for it to be on stable.


Bring in new datasets

We need to test-run RiboViz on some new datasets to check that it can actually work. These will not be incorporated into the automated testing, but will provide a larger informal test. And also move towards asking new substantial biological questions with RiboViz.
Ideas:

Muzzey 2014 Candida albicans (2 reps)
Weinberg 2016 Saccharomyces cerevisiae (1 rep, "standard")
Polymenis 2018 Saccharomyces cerevisiae mitosis (3 reps in many conditions)
Duncan and Mata Schizosaccharomyces pombe (up to 7 reps, useful for stats)

@ewallace will share the .yaml file for Mussey 2014, and track down the references and accession numbers for the others so that @FlicAnderson can start attacking them.

Sent to @FlicAnderson and @mikej888 info inc yaml files and transcriptome annotation for Muzzey et al, https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE52236

Muzzey D, Sherlock G, Weissman JS. Extensive and coordinated control of allele-specific expression by both transcription and translation in Candida albicans. Genome Res 2014 Jun;24(6):963-73. PMID: 24732588

Weinberg & Shah 2016 dataset is
Improved Ribosome-Footprint and mRNA Measurements Provide Insights into Dynamics and Regulation of Yeast Translation.
https://www.ncbi.nlm.nih.gov/pubmed/26876183
https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE75897
I do not have a .yaml file for them. Saccharomyces cerevisiae annotation files (250nt padding) available in RiboViz/docs/prepare-data.md

@shahpr and/or @lianafaye, can you reommend one dataset with the new inline barcodes?

https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE124428
On Jul 22, 2019, at 2:08 PM, Edward Wallace <notifications<EMAIL_ADDRESS>wrote:
@shahprhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fshahpr&data=02|01|premal.shah%40rutgers.edu|eeec9b3fba79466d661a08d70ecf98df|b92d2b234d35447093ff69aca6632ffe|1|0|636994157096159274&sdata=X6rYYD4xUkOCpdEVZgY%2FGlKtl0yzyjK2lhZobUoaG1k%3D&reserved=0 and/or @lianafayehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flianafaye&data=02|01|premal.shah%40rutgers.edu|eeec9b3fba79466d661a08d70ecf98df|b92d2b234d35447093ff69aca6632ffe|1|0|636994157096169271&sdata=JlfoOVy94wDphuINfOC4VRrugYHtf0YhTrE2dzpywBc%3D&reserved=0, can you reommend one dataset with the new inline barcodes?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Friboviz%2FRiboViz%2Fissues%2F29%3Femail_source%3Dnotifications%26email_token%3DAAZCTVH4MZWXKSADDZNQ3YDQAXZRRA5CNFSM4IF2S5BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2QWFZA%23issuecomment-513893092&data=02|01|premal.shah%40rutgers.edu|eeec9b3fba79466d661a08d70ecf98df|b92d2b234d35447093ff69aca6632ffe|1|0|636994157096169271&sdata=Mhgq8xYL1i5o8%2FxaWVAR7JHJ9EZbduJoj9XQLKlc9Ls%3D&reserved=0, or mute the threadhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAZCTVHIMUH27VILQJUJCA3QAXZRRANCNFSM4IF2S5BA&data=02|01|premal.shah%40rutgers.edu|eeec9b3fba79466d661a08d70ecf98df|b92d2b234d35447093ff69aca6632ffe|1|0|636994157096179267&sdata=xhPNz1UZh6nQTkoX8jJ0gMas0Wq7JOVLeccb0LMVi2k%3D&reserved=0.

Also we can extend to a larger part of the Guydosh & Green 2014 dataset, that we subsampled for the vignette, described in #31.

Relevant Duncan & Mata dataset is from this paper: General amino acid control in fission yeast...
Data are on ArrayExpress E-MTAB-5810
Note structure of the read includes UMIs, described in Data Preprocessing and Read Alignment.

For all other ribosome profiling samples, the structure of the reads is as follows: RRRR(NNNN…NNNN)RRRR-adaptor-, where R represents random nucleotides; N corresponds to the sequence of the RNA protected fragment; and the adaptor sequence is TGGAATTCTCGGGTGCCAAGG.

The pipeline as currently conceived does not deal with random nucleotide UMIs. Also, the structure differs for 2 samples which also have sample barcodes RRRRRRRRRRRR(NNNN….NNNN)BBB so we may wish to exclude them (wt.noAT.ribo.2 / wt.AT.ribo.2) from the initial analysis.
These are the kinds of problems we need to extend the pipeline to deal with. We could ignore the UMIs by trimming the random nucleotides at the ends, but that still requires another step to the pipeline. This dataset could be a good test case for flexible way to deal with UMIs.

28 August update:
Flic has taken a full dataset and configured RiboViz to run on this. The next steps will be to try other datasets in the same pattern. Edward will inspect the data to validate it.
This spawned discussion on Quality Assurance #56 and also logging #55.

Bring Muzzey 2014 Candida albicans as a full dataset; then downsample as per https://github.com/riboviz/RiboViz/issues/38

@FlicAnderson: when feasible, please update the ticket with current progress.

Update of the datasets I'm working on getting through RiboViz:
Note: for my own sanity and own notes I'm using a 'codename format' of:
[First Author's First Surname Letter]-[Genus-species first letters]_[publication year]

[X] G-Sc_2014 (Guydosh & Green; Saccharomyces cerevisiae; no UMIs, no barcodes) :+1:
[ ] [in progress] M-Ca_2014 (Muzzey et al; Candida albicans; no UMIs, no barcodes, 2 reps)
[ ] W-Sc_2016 (Weinberg & Shah 2016; Saccharomyces cerevisiae; no UMIs, no barcodes, 1 rep, "standard")
[ ] P-Sc_2018 (Polymenis 2018; Saccharomyces cerevisiae mitosis; with UMI, no barcodes, 3 reps in many conditions)
[ ] D-Sp_2018 (Duncan & Mata 2018; Schizosaccharomyces pombe; with UMI, no barcodes, up to 7 reps, useful for stats)
[ ] G-Sc_2019 (Gupta et al 2019; Premal's with UMI & with barcodes) (#39 )

G-Sc_2014 Guydosh & Green data was successfullly run late August (will shortly re-run with newly re-factored prep_riboviz.py)
M-Ca_2014 Muzzey data was run semi-successfully 09th Sept; majority of the pipeline ran, but due to different organism and lack of features_file (yeast_features.tsv does not apply to Candida albicans), the pipeline broke down when calling the missing value into generate_stats_figs.R.  This has been set up as issue #66 and I've created a branch to fix this in (handle-missing-features-file-66) and will work on that fix to complete the data run
W-Sc_2016 has not been run yet but should hopefully pose less of an immediate challenge :crossed_fingers:, and after running that I can work on creating downsampled dataset from M-Ca_2014 as agreed in issue #38
After that, I'll be able to work on the UMI-containing datasets and work on the Schizosaccharomyces pombe data which hasn't been run through RiboViz previously (while the first 3 datasets have previously been successfully run through previous versions).

Via Liana during meeting 02 Oct:

Weinberg reads are 8 random nucleotides, then the footprint, and then the adapter is TCGTATGCCGTCTTCTGCTTG

Update:
M-Ca_2014 dataset being re-run with latest develop to test fix #66
W-Sc_2016 dataset turns out to be a UMI dataset as per comment "8 random nucleotides", so will need UMI handling capabilities

We want to see how long W-Sc_2016 takes to run; then we create a vignette from it.
To then test on one of the files of D-Sp_2018.

Progress update on datasets:

[x] G-Sc_2014 (Guydosh & Green; Saccharomyces cerevisiae; no UMIs, no barcodes) +1
[x] M-Ca_2014 (Muzzey et al; Candida albicans; no UMIs, no barcodes, 2 reps)
[x] W-Sc_2016 (Weinberg & Shah 2016; Saccharomyces cerevisiae; 8-NT 5' UMIs, no barcodes, 1 rep, "standard")
[ ] next target >> D-Sp_2018 (Duncan & Mata 2018; Schizosaccharomyces pombe; with UMI, no barcodes, up to 7 reps, useful for stats)
[ ] P-Sc_2018 (Polymenis 2018; Saccharomyces cerevisiae mitosis; with UMI, no barcodes, 3 reps in many conditions)
[ ] G-Sc_2019 (Gupta et al 2019; Premal's with UMI & with barcodes) (#39)

G-Sc_2014 Guydosh & Green data initially run late August, will shortly re-run to obtain new plot outputs from ribogrid-plot-68 feature (now in develop)
M-Ca_2014 Muzzey data run successfully on 09 Oct on branch develop, last commit db73b2e81, run time: ~2 hours (8 processors)
W-Sc_2016 Weinberg data run successfully 20 Oct on branch: use-umi-tools-dedup-58, last commit 632911e3, run time: 3 hours (8 processors)
Next dataset: Setting up and finding read (1 sample of 7 total samples) and genome files for D-Sp_2018, as agreed will contact Edward re: any missing or non-obvious files; set up yaml. Run with use-umi-tools-dedup-58 or develop, depending on whether these changes are merged into develop yet or not, as this is UMI dataset
Next downsampled dataset: Can start creating downsampled dataset from M-Ca_2014 as agreed in issue #38, next downsampled dataset AFTER M-Ca_2014 will include UMIs, but will be something with more than one sample file (ie not W-Sc_2016 as this is single read file).

@ewallace asked @lianafaye and @shahpr to look at the outputs from W-Sc_2016 Weinberg.
Agreed that D-Sp_2018 Schizosaccharomyces pombe is a good choice to be the next dataset just now; important dataset that does not require demultiplexing (which we don't yet have).

@ewallace has found some issues with the above outputs and @FlicAnderson has addressed them.
We postpone the review until after we have closed #58.

Also we should add the 14-replicate dataset from C-Sc_2017: Chou et al 2017 Transcriptome-wide Analysis of Roles for tRNA Modifications in Translational Regulation

We have made huge amounts of progress on this in riboviz/example-datasets. Let's go through this ticket, create a separate issue for each dataset in riboviz/example-datasets, then close it.

Gupta et al 2019 is being addressed in example-datasets#14 by @acope3.

We have made huge amounts of progress on this in riboviz/example-datasets. Let's go through this ticket, create a separate issue for each dataset in riboviz/example-datasets, then close it.

All datasets have a separate issue ticket in example-datasets and can be followed up on there, so I'm now closing this ticket.