Split CookieAuthenticationEvents "ApplyRedirect" into multiple events

Split the current "ApplyRedirect" into multiple events depending on what is being redirected to. New names shouldn't have the word "Apply" in them, they should use the pattern RedirectToXyz.

:+1:
I have just been bitten by this design change (initiated when CookieMiddleware started to always redirect in b883920).
A few comments:
Although I'm watching the Notifications repo, I didn't see an announcement about that and beta-7 was a breaking (at runtime) change for me because redirect would still happen when options.LoginPath was set to null (until beta-6 that's how you prevented redirects). Unless I missed the announcement, I think this is worthy of more communications.
Configuring the redirects for not happening is a valid if not common scenario in Web API (an AJAX call cannot use a login HTML page...).
I found out that the new way to do this is to change OnApplyRedirect in options.Notifications (beta-7, now renamed to options.Events). That's fine but it could be better for 2 reasons:

OnApplyRedirect is very generic and the reason "why" is lost, which makes coding this handler more difficult (was it 401, 403, something else)? I think it should be split in more specific cases and this is what this ticket is about.

OnApplyRedirect is not part of the ICookieAuthenticationNotifications (renamed post beta-7) interface, which is what options.Notifications exposes. This means that overriding the behavior requries an unwieldy cast: ((CookieAuthenticationsNotifications)options.Notifications).OnApplyRedirect = .... It is also hard to discover when you look for a way to configure redirects in the IDE.
Maybe you should extend the interface, or if you don't want to expose the OnXYZ to interface consumer, expose CookieAuthentificationNotifications to options only.

@jods4 as for 2, the recommended way to override it is:
options.Notifications = new CookieAuthenticationNotificatoins()
{
 OnApplyRedirect = ...
}

@Tratcher thanks for the tip!

BTW it would be nice if the default behaviour when OnApplyRedirect does nothing was to return 401 / 403. I mean: currently the default Events.OnApplyRedirect sends an HTTP redirect to LoginPath but if you remove it then the server returns a plain 200 with no content.
My thinking is: the request was unauthorized, so it should start as a 401. That makes sense.
Now if ASP.NET wants to set a default OnApplyRedirect that transforms 401 into a LoginPath redirect I'm fine with that, but when I remove it I would like to have the 401 behaviour pass through.
My observation is that if you put an empty method in OnApplyRedirect you get a 200 OK, which makes little sense in this context by default. You currently (beta-7) have to hack context.RedirectUri to decide whether it's a 401 or 403 and actually set the context.Response.StatusCode to that.

Cookie/forms auth only defines redirects and should not produce 401s or 403s, those are defined by standard HTTP auth (e.g. basic, bearer, etc.).
The guideline for WebApi scenarios is to challenge for something specific like bearer, bypassing cookie auth entirely.

The guideline for WebApi scenarios is to challenge for something specific like bearer, bypassing cookie auth entirely.

I am not talking public internet-style web API here. I am talking private, secure, intranet web API that is back-end for a web application. Users have to log in the application using username and password. Server calls are limited by their personal rights and must be revokable when they log out or close their browser.
If by bearer you mean (from RFC 6750):

Bearer Token
A security token with the property that any party in possession of the token (a "bearer") can use the token in any way that any other party in possession of it can.  Using a bearer token does not require a bearer to prove possession of cryptographic key material (proof-of-possession).

Then I think it is absolutely not what my web API needs. I am curious if you have other ideas, but as far as I know cookie auth. seems to be the most adequate auth. primitive that ASP.NET offers me.
I also think that in this context, a request denied because of insufficient privileges should return 401 or 403 to the browser. Returning an empty 200 OK is just wrong and returning a redirect to an HTML login form is inappropriate for AJAX calls. I don't mind that the latter is the default, but .NET had a philosophy of being a "pit of success" where doing things the obvious way would "just work". Removing the redirect code ends up in a situation that does not make sense to me.
Before  b883920, when you set LoginPath to null you would disable the redirect and get a 401 instead, it seems to me that it made sense.
I don't think this is an uncommon scenario. I work at a large IT company and have been for a decade. We built lots of web apps or fat clients back-ends with just web services. Web API is remarkably well suited for that.

username and password API login is commonly handled by 'Basic' auth or Windows auth (NTLM).

Windows auth is great for that kind of scenarios and I use it whenever I can. But sometimes (like my current case) it is not feasible.
Basic auth is an option but it's not great. Username and password must be sent and verified on each requests, logging out is hacky. The web app must manage the credentials (and keep the username in JS memory) because it has to manipulate each request to include auth. If you let the browser do that, I believe you have little control over the credential caching time (can be as short as 15 minutes) and no control over the log on UI? (I might be wrong on the last one)

@blowdart Want to pick this one up?

For pit of success we try to steer away from cookie auth for apis because of CSRF.
Maybe the old effect could be configurable though, for back compat if you really want and understand the risk.

@Tratcher @blowdart Please note that the discussion went off on a tangent a bit.
At first I was just commenting on the new APIs design to configure cookie auth. middleware.
In the end, I can make everything work as I want to, so no blocker here.
The point I was making initially was:

Yes, splitting ApplyRedirect or making its cause easily identifiable is required (this ticket basically);
It is a bit disconcerting that putting a no-op in OnApplyRedirect ends up with a 200 OK request, since (a) this is not 200 OK obviously and (b) previously you could get the 401 back when you put null into LoginPath.
The API for configuring this is not very discoverable, as the option can not be found by IntelliSense (you have to know which class to instantiate (or cast to) to get the actual option).

The old effect is already achievable by setting the status code in OnApplyRedirect.

@blowdart
A web API for an internal application can easily include a CSRF token, can't it? I intend to do that but haven't configured ASP.NET 5 to achieve that. I was thinking of sending the CSRF token to the application at startup and then put it in every AJAX request header.

BTW thanks for the tips, I will think about Basic Auth and maybe reconsider a bit.
It is true that it removes CSRF requirements and I could just keep the user+password base64 string in the app and send it in a header with every http request.
Only downsides I see is 3rd party components who make their own requests and having to validate the password each time, which is (by design) expensive.

CSRF for webapi involves embedding a form, yanking the token out of it and sending it with every request. At that point you may as well have switched to bearer :)

(And basic is just as vulnerable to CSRF, as once the browser has logged it it'll flow automatically like cookies)

@blowdart

At that point you may as well have switched to bearer

Are you serious?
Bearer does not identify the user. Username + password does.
The point of CSRF token is not sending it in the URL to prevent crafting links or img sources or whatever.
Did I miss something here?

And basic is just as vulnerable to CSRF, as once the browser has logged it it'll flow automatically like cookies)

Indeed. I was mislead  when Tratcher said recommendation for APIs were Win or Basic and you said

we try to steer away from cookie auth for apis because of CSRF

I kinda assumed that Basic was better than Cookie in this regard, but indeed if the browser sends it automatically from its cache then it's no better.

I'm not saying bearer identifies the user (although if you used a jwt it could, validate signature instead of password). In middleware you'd validate and exchange the bearer token for a principal.
CSRF tokens go into a form and are linked to the user, or the browser by a secondary cookie. They are there to stop someone crafting a form and hosting it on another web site, then submitting the form to yours (and tricking your users into doing something on your site they didn't initiate)

In middleware you'd validate and exchange the bearer token for a principal.

That's more or less what the cookie is/does, isn't it? Except cookies are managed by the browser.

CSRF tokens go into a form and are linked to the user, or the browser by a secondary cookie. They are there to stop someone crafting a form and hosting it on another web site, then submitting the form to yours (and tricking your users into doing something on your site they didn't initiate)

I think that I understood your previous statement incorrectly:

At that point you may as well have switched to bearer

I thought you meant that sending a CSRF token in a HTTP header with every request was no more secure than authentication with a bearer token. Did you in fact mean that in the situation I describe sending a constant bearer token in a HTTP header was good enough to prevent CSRF on the web API? That the whole CSRF token thing was more complicated because it protects different scenarios?

That's more or less what the cookie is/does, isn't it?

Yes, that's exactly what the cookie middleware does.

That the whole CSRF token thing was more complicated because it protects different scenarios?

ah no, what I'm saying is the effort to write script to pull a CSRF token out, put it into a header, and then check it on the other end is close to the effort needed to write a bearer token issuance and parser that with very little effort you could go the bearer token route.

I see.
Thanks again, I'll stop here because I think we have moved rather far away from the issue topic. <_>

:)

https://github.com/aspnet/Security/pull/482


make error

thanks for your source code.
I try to make it, but /usr/bin/ld: cannot find -lippicv  error occured
what is it？ And what should I do to solve it ? please help me !
Thank you

Hi, it's an opencv error.
Try to recompile opencv with this flags:
cmake -DWITH_IPP=ON -DINSTALL_CREATE_DISTRIB=ON
More info at these links:
https://stackoverflow.com/questions/34401117
https://github.com/pjreddie/darknet/issues/290


Enhance error report for fuzzing

What have you changed? (mandatory)
Change the error report when user run fuzz testing before install corresponding fuzzer. See issue #4585.
The original error report:
error: no such subcommand: `afl`

	Did you mean `doc`?

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ErrorMessage { msg: "error building afl instrumented binary, exit code Some(101)" }', src/libcore/result.rs:999:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

And now:
Running fuzzer failed: cannot run afl subcommand, please run `cargo install afl` before testing
Error: 1

The change to honggfuzz is similar.
What are the type of the changes? (mandatory)
Improvement.
How has this PR been tested? (mandatory)
It is unrelated to test cases.
Does this PR affect documentation (docs) or release note? (mandatory)
No
Does this PR affect tidb-ansible update? (mandatory)
No
Refer to a related PR or issue link (optional)
#4585

PTAL @breeswish

Thanks for your contribution!


[Bug]: Permission denied inside running Apache Pulsar container

Module
Pulsar
Testcontainers version
1.17.3
Using the latest Testcontainers version?

We use Test Containers to run the Apache Pulsar docker container for testing purposes, but after updating to the latest version (Test Containers to 1.17.3 and Apache Pulsar to 2.10.0), we started getting permission errors inside the Apache Pulsar docker container to required standalone.conf file.
Also, these bugs are not reproducible in test containers 1.17.2 or Apache Pulsar 2.8.1.
Relevant log output
Traceback (most recent call last):
  File "/pulsar/bin/apply-config-from-env.py", line 104, in <module>
    f = open(conf_filename, 'w')
PermissionError: [Errno 13] Permission denied: '/pulsar/conf/standalone.conf'

### Additional Information

Sample project code to start Apache Pulsar:

PulsarContainer pulsar = new PulsarContainer(DockerImageName.parse("apachepulsar/pulsar:2.10.0"))
		.withCopyFileToContainer(MountableFile.forClasspathResource("standalone.conf"), "/pulsar/conf/standalone.conf")
		.withCommand("bin/pulsar standalone")
		.waitingFor(Wait.forListeningPort());
pulsar.start();
pulsar.execInContainer("bin/pulsar-admin", "namespaces", "set-retention", "public/default", "--size", "-1", "--time", "-1");
pulsar.execInContainer("bin/pulsar-admin", "namespaces", "set-schema-compatibility-strategy", "--compatibility", "ALWAYS_COMPATIBLE", "public/default");
pulsar.execInContainer("bin/pulsar", "initialize-transaction-coordinator-metadata", "-cs", "<IP_ADDRESS>:2181", "-c", "standalone")

@dmytro-rusnak thanks for raising the issue! in 1.17.3, pulsar image was updated to 2.10. I wonder if you need that upgrade too? Otherwise, you can use the previous version you were using in the meantime.

@eddumelendez hi. Thanks for your reply. Previously, we used to test containers 1.17.2 and pulsar 2.10.0 and everything worked fine, but after updating the test containers to the new version 1.17.3, we started to catch the permission denied issue.
In our situation, we have two options:

Downgrade the test containers to 1.17.2 and keep using pulsar 2.10.0.
Keep using the 1.17.3 test containers, but downgrade pulsar to 2.9.3 and below (because this issue only reproduces 2.10.x).

I hope this information helps you :)

thanks for sharing!

This change was introduced in the last release. The code below will fix the current issue
.withCopyFileToContainer(MountableFile.forClasspathResource("standalone.conf", 0777), "/pulsar/conf/standalone.conf"))

but  now you will be able to override config file given the prefix PULSAR_PREFIX_<property-name> which I would suggest as a much better way
PulsarContainer pulsar = new PulsarContainer(PULSAR_IMAGE)
            .withEnv("PULSAR_PREFIX_defaultNumPartitions", "2")

Hi @eddumelendez . I know the issue is closed, but the fix you provided didn't work for us.
For example, it fixed the permission denied, but the container still couldn't start with our standalone file and its configurations.
Do I need to create a separate ticket for this or provide additional information in this ticket?

@dmytro-rusnak For now, you can provide the additional information (setup code, logs) in this issue and we will look into it, thanks.

Hi @kiview . Thanks for replying me. I created a separate project as an example, but with configurations and dependencies like a real project (due to NDA, I can't share specific codes).
Also, I have shared the stack trace and logs from the docker container.
Let me know if you need some additional information :)


Remove log shipping from Flask docs, start deprecation

What does this pull request do?
Starts the deprecation process for the log shipping in Flask.

@elasticmachine, run elasticsearch-ci/docs


Add support for translation keys with dot syntax

Add support for translation keys with dot syntax. E.g.:
{
  "component": {
    "this.is.my.translation.key": "My Value"
  }
}

Great library, by the way!!

Hi!
That's a good point, I'll take some time to improve the lib and add this too.
Stay tuned ;)

Hi @nicodemuz very very sorry for the extremely late reply! I've been quite busy with work. If you are still interested in adding this feature, the place you are looking for is https://github.com/psyycker/react-translation/blob/master/src/utils/translation-utils.ts
getTranslationWithKey you'd need to change that so that if you are not able to find the key, then try to join it with the next one and keep trying until you are able to move forward (or not!)

This feature has been planned for the version 0.8.0 of the package (see roadman)
There is no release date planned at the moment.


Ec2 Driver Instance tagging on creation

Enhancement Request
Description

Opennebula only allows you to create ec2 instance tags after the creation of the instance.
Should be useful to have a mechanism to add instance tags on creation.
Use case

You have organized your ec2 portal with policies to allow or not allow users to create instances.
You need to attach tags on instance creation if you want this behaviour to work.
Progress Status

[ ] Branch created
[ ] Code committed to development branch
[ ] Testing - QA
[ ] Documentation
[ ] Release notes - resolved issues, compatibility, known issues
[ ] Code committed to upstream release/hotfix branches
[ ] Documentation committed to upstream release/hotfix branches

Public cloud integration has been deprecated in flavor of oneprovision, closing.


Underline as GIF shows

You include this GIF in the README

How do you configure VIM to do that curly underline and also not break syntax highlighting? Notice that let stays blue, and doesn't turn orange like the curly underline is.
I've tried setting ALEError and ALEWarning, but can't seem to come close to what you have.
I tried running :help ale-highlights to see if it explains it, but I just get the following error: E149: Sorry, no help for ale-highlights.

Use silent! helptags ALL to generate help tags, then you'll see the documentation. I recommend putting that in vimrc after loading the plugin in the README.
I've been using the darkspectrum color scheme for GVim for years now. You could try using that theme, or just some other theme.

Yeah, I use GVim because the highlighting looks better than in terminals, among a few other things. I'll try and add a tooltip to the image explaining which theme is used.

Okay, that should be there now.


Start remembering on login?

Hiya!
I see that from https://github.com/jasonjmcghee/rem/pull/9, its now possible to start Rem from login more easily (though I see that that release hasn't been made yet).
Are there any plans to enable "start remembering" on login too?
I'd help myself (as i do often for open source libraries), but I don't actually know any swift.

I'll make that release!  it starts remembering on login.

Fixed by the new release https://github.com/jasonjmcghee/rem/releases


Windows 10 - Whispering 5.0.0 Unresponsive When Cursor Outside Application

The application does not respond on Windows 10 when the cursor is outside the application.
I wanted to install the current version of Whispering 5.0.0 on a friend's laptop. Everything works, but when the cursor is within another application, Whispering no longer responds. I have tried all possible keyboard shortcuts to activate Whispering. Whispering does not react.
After installing the old version of Whispering 3.4.0, everything works normally.
Info
Device Specifications

HP Spectre x360 Convertible 15-bl0XX
Device name: DESKTOP-RHJ1HV7
Processor: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz 2.90 GHz
Installed RAM: 16.0 GB (15.9 GB usable)
System type: 64-bit operating system, x64-based processor
Pen and touch: Support for pen and touch input with 10 touch points

Windows Specifications

Edition: Windows 10 Home
Version: 22H2
Installed on: 09/2024
OS build: 19045.4894
Performance: Windows Feature Experience Pack 1000.19060.1000.0

v.5 works for me, both in Windows 10 and 11. Are you sure that you have the hotkey syntax correct?

Here's the English translation of your message:
I didn't know that the hotkey syntax only accepts uppercase letters, as I only use lowercase letter combinations on macOS.
Does this mean I have to press e.g.  "String" and "Shift" and then the desired letter? I can only try it out the next time I meet my friend with his laptop.

No, but I believe you must enter uppercase letters into the hotkey box of Whispering. You still can use the corresponding lowercase letter when hitting the hotkey on the keyboard.

I had entered lowercase letters as keyboard shortcuts, and as long as the cursor was within the application "Whispering," the shortcuts worked with the lowercase letters. However, there was no response from Whispering once the cursor was outside the application and in a different one.
In any case, I will try again using uppercase letters as keyboard shortcuts.

@rkhrkh Hopefully this should work now in the latest release!
I've been ironing out the kinks of Tauri's new shortcut manager, but it should solve a whole host of issues. The old shortcut manager had a lot of issues where it wouldn't fire when the window wasn't in focus. The latest manager seems to have fixed this problem. Could you let me know? If this issue still persists, could you create a new issue?


Release/0.6.0

New features

Added --force flag (#141). Without it Schema Guru will not override any table definitions of migrations if there was any manual changes (ignoring comments and white spaces)
SQL migrations within REVISION (#134)
Preserved column order (#135). Tightly coupled with migrations

Internal changes

For CLI, argot was replaced with scopt (#124). This resulted in massive refactoring, which should allow us to easily achieve feature parity for CLI and WebUI. Also  input argument is no more required be last, now we can add options after it (run schema --schema-by "$.path" {{input}} --enum 3). As a downside, Spark job now receives just a flag --enum-sets instead of multiple options (https://github.com/scopt/scopt/issues/103)
Schema segmentation now works with sane JsonNode-to-JValue converting (#125)

Very minor tweaks - please make those and then I will get the rc1 into acceptance testing. I'll also ask @fblundun to do full code review - Fred please go for it!
Also please link in the branch with the blog post when it's ready...

@alexanderdean typos has been fixed.

==> default: TASK: [aws-emr-cli | Ensure EMR CLI is extracted] *****************************
==> default: failed: [<IP_ADDRESS>] => {"changed": true, "cmd": " unzip elastic-mapreduce-ruby.zip -d /vagrant/emr-cli/ ", "delta": "0:00:00.366176", "end": "2016-03-28 13:18:58.342546", "rc": 1, "start": "2016-03-28 13:18:57.976370"}
==> default: stderr: replace /vagrant/emr-cli/net/http/connection_pool.rb? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL
==> default: (EOF or read error, treating as "[N]one" ...)
==> default: stdout: Archive:  elastic-mapreduce-ruby.zip
==> default:
==> default: FATAL: all hosts have already failed -- aborting
==> default:
==> default: PLAY RECAP ********************************************************************
==> default:            to retry, use: --limit @/home/vagrant/aws-tools.retry
==> default:
==> default: <IP_ADDRESS>                  : ok=17   changed=12   unreachable=0    failed=1
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

Solution: ticket to remove aws-tools from the vagrant.up

@alexanderdean can we fix a playbook? Or do you think it shouldn't even belong here?

Actually, yes, removing it makes sense. We need to run fatjar from a machine with keys and credentials anyway.

Yep, the one in questions uses the long-deprecated EMR CLI: https://github.com/snowplow/ansible-playbooks/blob/master/aws-tools.yml#L6
So even if we need to replace it with something else in the future, I think we should remove it...

When I attempt vagrant push I get:
[info] Compiling 32 Scala sources to /vagrant/target/scala-2.10/classes...
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:27: imported `Schema' is permanently hidden by definition of object Schema in package schemaguru
[error] import Common.{ TextFile, Schema, splitValidations }
[error]                           ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:54: type mismatch;
[error]  found   : x$1.type (with underlying type com.snowplowanalytics.schemaguru.Schema)
[error]  required: ?{def self: ?}
[error] Note that implicit conversions are not applicable because they are ambiguous:
[error]  both method any2stringfmt in object Predef of type (x: Any)scala.runtime.StringFormat
[error]  and method any2stringadd in object Predef of type (x: Any)scala.runtime.StringAdd
[error]  are possible conversion functions from x$1.type to ?{def self: ?}
[error]   implicit val schemaOrdering = implicitly[Order[Int]].contramap[Schema](_.self.version.addition)
[error]                                                                          ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:54: value self is not a member of com.snowplowanalytics.schemaguru.Schema
[error]   implicit val schemaOrdering = implicitly[Order[Int]].contramap[Schema](_.self.version.addition)
[error]                                                                            ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:68: value revisionCriterion is not a member of com.snowplowanalytics.schemaguru.Schema
[error]     schemas.groupBy(_.revisionCriterion)
[error]                       ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:80: value data is not a member of com.snowplowanalytics.schemaguru.Schema
[error]     val flatSource = flattenJsonSchema(sourceSchema.data, splitProduct = false).map(_.elems)
[error]                                                     ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:81: value data is not a member of com.snowplowanalytics.schemaguru.Schema
[error]     val flatSuccessive = successiveSchemas.map(s => flattenJsonSchema(s.data, splitProduct = false)).sequenceU
[error]                                                                         ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:81: Implicit not found: scalaz.Unapply[scalaz.Applicative, A]. Unable to unapply type `A` into a type constructor of kind `M[_]` that is classified by the type class `scalaz.Applicative`. Check that the type class is defined by compiling `implicitly[scalaz.Applicative[type constructor]]` and review the implicits in object Unapply, which only cover common type 'shapes.'
[error]     val flatSuccessive = successiveSchemas.map(s => flattenJsonSchema(s.data, splitProduct = false)).sequenceU
[error]                                                                                ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:211: type mismatch;
[error]  found   : source.type (with underlying type com.snowplowanalytics.schemaguru.Schema)
[error]  required: ?{def self: ?}
[error] Note that implicit conversions are not applicable because they are ambiguous:
[error]  both method any2stringfmt in object Predef of type (x: Any)scala.runtime.StringFormat
[error]  and method any2stringadd in object Predef of type (x: Any)scala.runtime.StringAdd
[error]  are possible conversion functions from source.type to ?{def self: ?}
[error]     } yield (source.self, buildMigration(source, targets))
[error]              ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:211: value self is not a member of com.snowplowanalytics.schemaguru.Schema
[error]     } yield (source.self, buildMigration(source, targets))
[error]                     ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/Migrations.scala:212: value _1 is not a member of Nothing
[error]     migrations.groupBy(_._1)
[error]                          ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/SchemaCodecs.scala:20: imported `Schema' is permanently hidden by definition of object Schema in package schemaguru
[error] import Common.{ Schema, SchemaDescription, SchemaVer }
[error]                 ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/SchemaCodecs.scala:60: type mismatch;
[error]  found   : com.snowplowanalytics.schemaguru.Common.SchemaDescription
[error]  required: com.snowplowanalytics.schemaguru.schema.JsonSchema
[error]             Schema(desc, JObject(cleanSchema))
[error]                    ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/SchemaCodecs.scala:66: type mismatch;
[error]  found   : x.type (with underlying type com.snowplowanalytics.schemaguru.Schema)
[error]  required: ?{def self: ?}
[error] Note that implicit conversions are not applicable because they are ambiguous:
[error]  both method any2stringfmt in object Predef of type (x: Any)scala.runtime.StringFormat
[error]  and method any2stringadd in object Predef of type (x: Any)scala.runtime.StringAdd
[error]  are possible conversion functions from x.type to ?{def self: ?}
[error]       case x: Schema => JObject(JField("self", Extraction.decompose(x.self)) :: x.data.obj)
[error]                                                                     ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/SchemaGuru.scala:23: imported `JsonConvertResult' is permanently hidden by definition of object JsonConvertResult in package schemaguru
[error] import Common.{ JsonConvertResult, DerivedSchema, SchemaGuruResult }
[error]                 ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/SchemaGuru.scala:23: imported `SchemaGuruResult' is permanently hidden by definition of object SchemaGuruResult in package schemaguru
[error] import Common.{ JsonConvertResult, DerivedSchema, SchemaGuruResult }
[error]                                                   ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/SchemaGuru.scala:83: type mismatch;
[error]  found   : com.snowplowanalytics.schemaguru.Common.DerivedSchema
[error]  required: com.snowplowanalytics.schemaguru.schema.JsonSchema
[error]     SchemaGuruResult(DerivedSchema(schema, None), jsonConvertResult.errors, Some(PossibleDuplicatesWarning(duplicates)))
[error]                                   ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/cli/DdlCommand.scala:79: type mismatch;
[error]  found   : List[com.snowplowanalytics.schemaguru.JsonFile]
[error]  required: List[com.snowplowanalytics.schemaguru.Common.JsonFile]
[error]           if (rawMode) transformRaw(someJsons)
[error]                                     ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/cli/DdlCommand.scala:80: type mismatch;
[error]  found   : List[com.snowplowanalytics.schemaguru.JsonFile]
[error]  required: List[com.snowplowanalytics.schemaguru.Common.JsonFile]
[error]           else transformSelfDescribing(someJsons)
[error]                                        ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/cli/DdlCommand.scala:105: type mismatch;
[error]  found   : List[com.snowplowanalytics.schemaguru.Common.Schema]
[error]  required: List[com.snowplowanalytics.schemaguru.Schema]
[error]     val migrationMap = Migrations.buildMigrationMap(schemas)
[error]                                                     ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/cli/SchemaCommand.scala:148: value describe is not a member of com.snowplowanalytics.schemaguru.SchemaGuruResult
[error] possible cause: maybe a semicolon is missing before `value describe'?
[error]       .describe(vendor, name, schemaver)
[error]        ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/utils/FileSystemJsonGetters.scala:130: type mismatch;
[error]  found   : List[scalaz.Validation[String,com.snowplowanalytics.schemaguru.Common.JsonFile]]
[error]  required: com.snowplowanalytics.schemaguru.ValidJsonFileList
[error]     (which expands to)  List[scalaz.Validation[String,com.snowplowanalytics.schemaguru.JsonFile]]
[error]     else if (file.isDirectory) for { f <- listAllFiles(file) } yield getJsonFile(f)
[error]                                        ^
[error] /vagrant/src/main/scala/com.snowplowanalytics/schemaguru/utils/FileSystemJsonGetters.scala:131: type mismatch;
[error]  found   : scalaz.Validation[String,com.snowplowanalytics.schemaguru.Common.JsonFile]
[error]  required: scalaz.Validation[String,com.snowplowanalytics.schemaguru.JsonFile]
[error]     else getJsonFile(file) :: Nil
[error]                            ^
[error] 22 errors found
[error] (schema-guru/compile:compileIncremental) Compilation failed

Weird. I had similar errors when I compiled it on local machine and after that tried to recompile in vagrant VM without cleaning target dir.

I'll try cleaning the target dir

Also, is it possible that you have some files from old version not modified? It tries to find com.snowplowanalytics.schemaguru.Schema which was defined in package.scala previously, but now moved to Common module.

Cleaning the target dir fixed it. The next error:
java.io.IOException: Cannot run program "gulp" (in directory "webui/src/main/resources/web"): error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
	at scala.sys.process.ProcessBuilderImpl$Simple.run(ProcessBuilderImpl.scala:68)
	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang(ProcessBuilderImpl.scala:112)
	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:128)
	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:101)
	at WebuiBuildSettings$$anonfun$1.apply$mcV$sp(WebuiBuildSettings.scala:60)
	at WebuiBuildSettings$$anonfun$1.apply(WebuiBuildSettings.scala:58)
	at WebuiBuildSettings$$anonfun$1.apply(WebuiBuildSettings.scala:58)
	at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
	at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
	at java.lang.ProcessImpl.start(ProcessImpl.java:130)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
	at scala.sys.process.ProcessBuilderImpl$Simple.run(ProcessBuilderImpl.scala:68)
	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang(ProcessBuilderImpl.scala:112)
	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:128)
	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:101)
	at WebuiBuildSettings$$anonfun$1.apply$mcV$sp(WebuiBuildSettings.scala:60)
	at WebuiBuildSettings$$anonfun$1.apply(WebuiBuildSettings.scala:58)
	at WebuiBuildSettings$$anonfun$1.apply(WebuiBuildSettings.scala:58)
	at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
	at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

On which exact step it fails? assemble_fatjar webui? Are you sure all playbooks were executed? Do you have a gulp command in vagrant box?

Yep assemble_fatjar webui. I have destroyed the Vagrant environment and am bringing it up again.

Everything's been pushed now!

@chuwy the release is working well for me - just waiting on the blog post...

@chuwy it looks good to me - I don't have any suggestions besides the file overwriting thing.


Using Claude in LangChain

Does anyone know if it is possible to use Claude for document question and answering in LangChain?
The Claude LLM model is integrated to LangChain. What is missing: LangChain-compatible Claude embeddings.
Does anyone know how I can generate them? Or alternative routes I could take to get Claude to work with Langchain document question and answering?

We have a LLMModel and ChatModel (coming this weekend) in LangChain.
We do not have embeddings, but that's because we do not currently expose a server API for embeddings as of April.
You're welcome to sub in any other embedding model you like for now. I believe langchain separates Embeddings from LLMModel and lets you plug-n-play, so you should be able to use any of the Embeddings modules supported by langchain.


Adds support for setting default "sendgrid_disable" options at mailer level, not just emails

Then you can have:
class AdminMailer < ActionMailer::Base
include SendGrid
sendgrid_disable :subscriptiontrack
def this_has_subscriptiontrack_disabled ; end
def this_has_subscriptiontrack_disabled_too ; end
def this_has_subscriptiontrack_enabled
sendgrid_enable :subscriptiontrack
end
end

I may be misunderstand this request, but this isn't the default option for things like :clicktrack or :opentrack. It depends on your settings at the account level, but you may have a mailer where you want those disabled.

+1 from me for the same reason as @chrisnicola.  DRY code is good :)

@stephenb The default behaviour when setting up a new account with Sendgrid is that click and open tracking is enabled on all emails unless a header is set explicitly disabling it. This feature would be very useful as it avoids having to disable click/open tracking in your Sendgrid account.


fixed android button issue

I fixed #201 by following the suggestion in the comments, i'll be shipping an app soon and it would be best if i didn't have to edit anything in my node modules.

@himanshu-satija Please review this.

This fails if I pass capitalize prop with the Button.
In the original issue, it failed due to this.props.children being an array
<Button onPress={this.props.turnOff}> {close ${this.props.id}} </Button>
This causes the children prop to be an array[' ', 'close 1', ' '] because of the extra spaces outside curly braces.
This should work fine.
<Button onPress={this.props.turnOff}>{close ${this.props.id}}</Button>
Actually, we should handle this case in a different way. Feel free to send another PR for that. Or we can fix that in the next version.

@himanshu-satija where is that code you put? i grepped my fork and couldn't find it.

This is not in your code. It is the Button usage which is causing this issue
https://github.com/GeekyAnts/NativeBase/issues/201#issuecomment-250681199

So what do you propose we do? We could do a filter for anything that isn't a component at the beginning for the method.

Yes, that would be the best approach

Awesome, I'll make the necessary changes and submit another PR

Thanks :)

Is there anything else I should filter for other than a space, such as null, undefined, etc?

Sure, we can filter undefined and null elements

Are there any automated tests for the lib?

No, but it is there on our roadmap.


Work in Progress: Camera effects

This is something I need for my application. Android side works well but iOS side seems more tricky..
Is that something you guys would be interested in? If not, I might just do the iOS quick and dirty, otherwise, I'll make sure it's mergeable.. :-)

Thats a good idea, I'll merge it.


[DOC,MNT] Documentation and testing for downsample.py

Describe the issue
transformations/downsample.py should have proper documentation and a unit test.
Suggest a potential alternative/fix
A unit test should be written in transformations/tests/test_downsample.py which tests the output of the contained function is as expected.
The file should pass the pre-commit checks.
Additional context
No response

I would like to start with this.
Aeon-Assign bot assign @futuer-szd


NextJS Conf 2021

Do you plan on updating the repository to have the look and feel of NextJS Conf 2021?

This design will continue from the 2020 conference 👍


test examples - endpoints do appear, seems to connect. but audio never starts to play back

Hi there,
Have been trying to investigate an issue in failure to playback audio firefox 100. The symptom is that a connection appears on the graph, and it tries to play. However the process hangs, the timer stays at 0:00. And after a while firefox stops responding and locks up.
This problem was previously here: https://bugzilla.mozilla.org/show_bug.cgi?id=1730499
And claims to have been fixed a while ago. However I myself am now on firefox 100 and the issue still persists.
So to try to debug this problem further I downloaded and build this repo here (mozilla/cubeb) from the latest git. About an hour ago.
The project builds successfully, and it generates all of the test-* executables. However unfortunately after launching such a test program (for example test_tone or test_audio)...
So it is still doing the same bug here (as in firefox browser). The connection looks to be initiated and seemingly becomes established to the pulse audio server. Which in my case is pipewire, (but this bug has also been reported on pulse audio server too). So the connection shows up on the graph in GUI (qpwgraph). So this is the point when playback is supposed to start. But that never happens.
The stdout of the test program prints a couple of line saying it has started playback. However the playback never occurs. And the test program just sits there. Here is an example:
[id:~/.builds/firefox/cubeb-build] 130 $ ./test_audio
Running main() from gtest_main.cc
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from cubeb
[ RUN      ] cubeb.run_volume_test_short

^C
[id:~/.builds/firefox/cubeb-build] 130 $

Not so sure how to proceed past this point... any advice or assistance would be greatly appreciated. Since I am just a regular firefox user. And have no particular expertise in this. Am completely lost as for what to do next. Thank you.

IDK if this could be any help. But in gdb i have to ctrl+c while its running to interrupt the program. And then the backtrace shows that it's still in the cube_stream_init ?
[id:~/.builds/firefox/cubeb-debug] $ gdb --directory ../cubeb ./test_tone
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test_tone...
(gdb) continue
The program is not being run.
(gdb) run
Starting program: /home/id/.builds/firefox/cubeb-debug/test_tone
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Running main() from gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from cubeb
[ RUN      ] cubeb.tone
[New Thread 0x7ffff71e6640 (LWP 2418886)]
pause

halt
^C
Thread 1 "test_tone" received signal SIGINT, Interrupt.
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555555f24bc) at ./nptl/futex-internal.c:57
57	./nptl/futex-internal.c: No such file or directory.
(gdb) stack
Undefined command: "stack".  Try "help".
(gdb) backtrace
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555555f24bc) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5555555f24bc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5555555f24bc, expected=expected@entry=0, clockid=clockid@entry=0,
    abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7ab2ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5555555f2460, cond=0x5555555f2490) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x5555555f2490, mutex=0x5555555f2460) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff79fa25c in pa_threaded_mainloop_wait () from /lib/x86_64-linux-gnu/libpulse.so.0
#6  0x0000555555562f9a in wait_until_io_stream_ready (stream=0x5555555fa220, mainloop=0x5555555f2310) at /home/id/.builds/firefox/cubeb/src/cubeb_pulse.c:439
#7  0x0000555555562fdf in wait_until_stream_ready (stm=0x5555555fa1b0) at /home/id/.builds/firefox/cubeb/src/cubeb_pulse.c:448
#8  0x0000555555565644 in pulse_stream_init (context=0x5555555f21e0, stream=0x7fffffffd388, stream_name=0x5555555ab254 "Cubeb tone (mono)", input_device=0x0,
    input_stream_params=0x0, output_device=0x0, output_stream_params=0x7fffffffd3f0, latency_frames=4096,
    data_callback=0x55555555ccab <data_cb_tone(cubeb_stream*, void*, void const*, void*, long)>,
    state_callback=0x55555555ce56 <state_cb_tone(cubeb_stream*, void*, cubeb_state)>, user_ptr=0x5555555f4b20)
    at /home/id/.builds/firefox/cubeb/src/cubeb_pulse.c:1013
#9  0x000055555555f7eb in cubeb_stream_init (context=0x5555555f21e0, stream=0x7fffffffd388, stream_name=0x5555555ab254 "Cubeb tone (mono)", input_device=0x0,
    input_stream_params=0x0, output_device=0x0, output_stream_params=0x7fffffffd3f0, latency=4096,
    data_callback=0x55555555ccab <data_cb_tone(cubeb_stream*, void*, void const*, void*, long)>,
    state_callback=0x55555555ce56 <state_cb_tone(cubeb_stream*, void*, cubeb_state)>, user_ptr=0x5555555f4b20) at /home/id/.builds/firefox/cubeb/src/cubeb.c:377
#10 0x000055555555d24d in cubeb_tone_Test::TestBody (this=0x5555555e6fe0) at /home/id/.builds/firefox/cubeb/test/test_tone.cpp:109
#11 0x000055555559c85c in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x5555555e6fe0,
    method=&virtual testing::Test::TestBody(), location=0x5555555b074b "the test body") at /home/id/.builds/firefox/cubeb/googletest/src/gtest.cc:2090
#12 0x0000555555596b01 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x5555555e6fe0,
    method=&virtual testing::Test::TestBody(), location=0x5555555b074b "the test body") at /home/id/.builds/firefox/cubeb/googletest/src/gtest.cc:2126
#13 0x0000555555580c28 in testing::Test::Run (this=0x5555555e6fe0) at /home/id/.builds/firefox/cubeb/googletest/src/gtest.cc:2161
#14 0x0000555555581432 in testing::TestInfo::Run (this=0x5555555e5ed0) at /home/id/.builds/firefox/cubeb/googletest/src/gtest.cc:2338
#15 0x0000555555581a09 in testing::TestCase::Run (this=0x5555555e6360) at /home/id/.builds/firefox/cubeb/googletest/src/gtest.cc:2445
#16 0x0000555555587073 in testing::internal::UnitTestImpl::RunAllTests (this=0x5555555e5fb0) at /home/id/.builds/firefox/cubeb/googletest/src/gtest.cc:4237
#17 0x000055555559de51 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x5555555e5fb0,
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x555555586dda <testing::internal::UnitTestImpl::RunAllTests()>,
    location=0x5555555b1208 "auxiliary test code (environments or event listeners)") at /home/id/.builds/firefox/cubeb/googletest/src/gtest.cc:2090
#18 0x0000555555597d61 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x5555555e5fb0,
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x555555586dda <testing::internal::UnitTestImpl::RunAllTests()>,
    location=0x5555555b1208 "auxiliary test code (environments or event listeners)") at /home/id/.builds/firefox/cubeb/googletest/src/gtest.cc:2126
#19 0x0000555555585dac in testing::UnitTest::Run (this=0x5555555d3920 <testing::UnitTest::GetInstance()::instance>)
    at /home/id/.builds/firefox/cubeb/googletest/src/gtest.cc:3871
#20 0x000055555557b152 in main (argc=1, argv=0x7fffffffd888) at /home/id/.builds/firefox/cubeb/googletest/src/gtest_main.cc:38
(gdb)

can see in vscode here the data in the request...

then it goes into the thread loop, and gets stuck in state PA_STREAM_CREATING forever...

tried pausing the debugger and manually forcing the state of that variable to be PA_STREAM_READY. but then the program says test case failed and exits -1
not really sure what to do next. as no multi threading experience here

when i manually set the sample rate to 96000 in the test example. It does not make any difference and the thread is still stuck waiting. So i think perhaps my 96khz setup is a bit of a red herring here?
is this more of a thread locking issue? (because that was indeed the problem before, on the previous firefox 92 issue fixed by alan in the rust layer, we can see in the patch fix for that)
Can see docs for it here, the thread locking / unlocking
https://freedesktop.org/software/pulseaudio/doxygen/threaded_mainloop.html
hmm :/ i'm just not sure how much further i could possibly get. at least now without some extra assistance

ok some real progress! (finally)...
this bug occurs for me in pipewire. but only when connecting the output of this node, as a new pulse audio stream... then connecting it to a jack application. which in my case is the program bitwig studio. but potentially it could occur with other jack programs
if the connection is made, then playback does not begin because:

the receiving pulse audio source never received a reply on the threaded callback to move into state READY coming from the jack application
in this case here the pulse audio source is the cubeb test_tone, but it could potentially be other pulse audio source applications, for example: chromium, or discord, etc.

however:

if i keep this hung / waiting for READY state pulse audio source connected to the jack sink
then ALSO subsequently connect it to an additional sink...
when this 2nd sink is NOT another jack application, but something else...
typically either an alsa hardware device, or
an alsa loopback device I have tried those today
but potentially perhaps also just another regular pulse audio node could also work too (not tested personally but in theory)

THEN the pulse source (here cubeb) WILL

start to commence playback
that playback will be received and work for the original 1st jack connection
even if you then subsequently disconnect the 2nd triggering (not jack) sink device

and playback will continue until finished. it will work for as long as that connection persists. until the next time the pulse audio based source is re-created
So this behaviour explains why:

the bug is not typically seen (by default) in pipewire
since the default setting is: node.autoconnect = true
and that then typically will automatically connect newly created nodes to the default ALSA sink device (or whatever the default device is set in pavucontrol etc.)
you have to manually set node.autoconnect = false in the pipewire settings, so as to avoid the automatic (but unwanted) connections that then works around the bug here

Therefore this bug report corresponds to the following 2 pipewire bug reports:
https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/2265
and
https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/2168

Not sure if it's helpful, but the official cubeb support on Linux is cubeb-pulse-rs instead of cubeb_pulse.c now.

thanks, i have since been made aware. actually it turns out this c++ version of the library i nice in the sense that it is very easy to fire up the debugger and observe what is happening. in part because all the pulse is itself also C. so not as much in the way of extra translation layers / hiding what is underneath. quite an education
thanks for working on these projects. my own issue was solved this morning on pipewire side. with virtual devices as a workaround. so nothing to worry about here (or in firefox or rust side either). please consider this matter closed now, but many thanks and appreciation for being able to use your library. it really helped a lot for getting to that solution


Fresh install of extension does not have defaults set

Hi, thanks for a very useful extension!!
Not a big deal personally, but I thought I'd mention that I installed this extension on a new user profile in Chrome on MacOS 10.11.6, and the options were not the default, I had to click "reset to defaults" to get the settings described in the README.
Attached is a screenshot of the settings after a fresh installation. Perhaps it's also worth noting: these aren't the options from my other profile on the same installation of Chrome, either.

Thanks for reporting. I can reproduce this bug, that be default it's showing the wrong default options.

Fixed in 0.3.2


2 x IS4 and nginx load balancing

Hey guys,
first of all, awesome job with IS, have been developing my single sign-on system for my company and this has been fundamental
Now, I have the following scenario
Server A:
A.IdentityServer
A.WebApp
Server B:
B.IdentityServer
B.WebApp
1 - user connects to WebApp,

nginx redirects him to A.WebApp
then A.WebApp redirects him to IdentityServer(A)
user signs in, everything is fine

2 - we shut down Server A

nginx now redirects the user to B.WebApp
B.WebApp has no session cookie, therefore redirects to IdentityServer(B)
B.IdentityServer has no session cookie (or the cookie is not for this app), therefore prompts the sign in form
:(

.NETCore session cookies are not valid for A.IdentityServer and B.IdentityServer, so we need to re-sign in
Any idea how to keep session across servers?
Sticky session is not useful here (since we kill the original server)
Reference tokens have the same issue
Could really use some pointer here,
Thanks in advance
Alex

Closing. If you still have issues, feel free to reopen.


Bool attribute as string?

http://plnkr.co/edit/YW1lSiy3J5c6fSdnISmj?p=preview
<my-tag boolvalue="false"></my-tag>

<my-tag>
   This is a {typeof opts.boolvalue}, and not bool
</my-tag>

Result:
"This is a string, and not bool"

That was an expected behaviour.
https://muut.com/riotjs/guide/#boolean-attributes
Here's a list of boolean attributes.
https://github.com/muut/riotjs/blob/master/lib/shared/compiler.js#L1-L5

Ok, thanks.


Feature proposal: spaces

🚀 Feature
The problem

Often datasets are composed of many different images that do not share a coordinate space (i.e: no reason/way to spatially or temporally relate them along a dimension); here I'm calling these separate world "spaces".
One often wants to have access to all the spaces without creating/deleting layers every time (and reading from disk every time)
It's often useful to quickly switch between spaces to compare features between samples.

Current approaches

simply load all the layers in the layerlist. This has 3 major downsides:

you have to toggle layer visibility to switch between spaces, which is a nightmare (especially if multiple layers share the same space, and linking them is out of question). This can be simplified with a widget (I currently do that in blik) but it's not ideal.
the layerlist is basically useless once many layers are loaded, cause it's impossible to find what you need and gives no useful visual cuues to the state of the viewer. (this last part will be partially solved by layer groups, but not the general problem of having many spaces)
If you have many layers (> hundreds), opening the whole dataset can take minutes. Time taken is no longer exponential, but still unusable in case of large datasets. This probably won't be addressed by the async efforts, because the issue is not with data loading but with vispy node creation and hooking up and firing of events (afaik).

create a widget that creates/inserts layers into the layer list on demand. This used to be my previous solution in blik.

does not have the problems above, because only the layers that are actually viewed are added to the viewer, making it easier to see what's what and faster to load
however, it's not a napari native solution, which means that

we can't use reader plugins to create the layers because they woudl all be added at once
the user/plugin is forced to keep track of everything that does not stay in the viewer (which means holding a reference to some sort of "dataset" object
bye bye serialization

The Solution
Make some form of point 2. a first class citizen in napari.

The viewer should be aware of spaces (or workspaces/scenes/worlds/whatever) which are collections of layers that are dimensionally unrelated to one-another but live in the same viewer
Make it easy to see what spaces exist and to switch between them

Potentially, once layer groups are a thing, spaces could simply be the nodes at depth 1 in the layer graph.

This would be good for for datasets with different modality and scales, e.g., if you have 3D CT scan and SEM of the same sample.
I also see this as a precursor to a 'project' system.

bye bye serialization

:rofl:
Nice issue @brisvag!
Although I see the need, I wonder why you would do this vs multiple viewers? Then spaces are serialisations of that particular viewer?
It would be good to have a concrete example where you have multiple spaces in one file, for example.
I also do see layergroups as a prerequisite here.

Although I see the need, I wonder why you would do this vs multiple viewers? Then spaces are serialisations of that particular viewer?
It would be good to have a concrete example where you have multiple spaces in one file, for example.
I also do see layergroups as a prerequisite here.

My use case is hundreds of spaces. Here's an example where I'm using trick number 1. (toggle visibility through a widget). In this example I only have one image per space, but If I had particle picking data I would have 4 more layers per space.
https://user-images.githubusercontent.com/23482191/165916904-93abe064-cee1-4c87-9efb-d41dbc586899.mp4
I don't think layergroups are a prerequisite; rather an enhancement. They can work together, and they solve slightly different problems. What about my proposal above you think requires layegroups?

If you have many layers (> hundreds), opening the whole dataset can take minutes. Time taken is https://github.com/napari/napari/pull/1945, but still unusable in case of large datasets. This probably won't be addressed by the async efforts, because the issue is not with data loading but with vispy node creation and hooking up and firing of events (afaik).

Are vispy nodes only created/attached when layers are added to the layerlist? nice!

I don't think layergroups are a prerequisite; rather an enhancement. They can work together, and they solve slightly different problems. What about my proposal above you think requires layegroups?

I agree here - this feels separate issues to layer groups, high value added for potentially low effort?
As an API, how about something like?
Viewer.spaces: EventedDict[LayerList]
Viewer.active_space: str

where Viewer.active_space indexes viewer.spaces

super basic prototype on a plugin widget
https://github.com/alisterburt/napari-spaces/blob/fc4840aaec2265af427a6600f0a35de94070feaa/src/napari_spaces/_widget.py#L16-L50
https://user-images.githubusercontent.com/7307488/166587193-6c5e1417-93a9-4a53-8709-c3308d9ad1ff.mp4
potentially a better API would be:
class Space(BaseModel):
    name: str
    layers: List[Layer]

Viewer.spaces: SelectableEventedList[Space]

def replace_layers():
    viewer.layers.clear()
    viewer.layers.extend(viewer.spaces.selection.active)

viewer.spaces.selection.events.active.connect(replace_layers)

cc @junxini this is the 'dataset navigation' we discussed - the UI is obviously not great and should probably be replaced by some of @Czaki's work over at napari/superqt#80
It could be that 'workflows' are defined in one space then applied over many spaces

cc @junxini this is the 'dataset navigation' we discussed - the UI is obviously not great and should probably be replaced by some of @Czaki's work over at napari/superqt#80

thank you for tagging me Alister, seeing the prototype really helps bring it to life!

It could be that 'workflows' are defined in one space then applied over many spaces

I imagined 'workflows' would be applied across multiple images/files and can definitely see a use case for having it applied across multiple spaces. Excited to see how spaces will improve the user experience

I imagined 'workflows' would be applied across multiple images/files and can definitely see a use case for having it applied across multiple spaces. Excited to see how spaces will improve the user experience

totally - there are lots of potential use cases where the data in a 'space' is not well represented by a single file on disk. I think having workflows start from layers in napari spaces is probably more flexible than assuming batch processing will always work over a set of files 🙂
This would separate the 'batch processing' project you're thinking about into

develop simple ways to load batches of data into spaces
develop the infrastructure for executing workflows on a per space basis

the UX would be quite nice I think, mirrors quite nicely what the deep learning community have done in terms of separation of concerns (pytorch DataLoaders are totally separate from the part which actually does the computation)
Let's keep this line of thought open 🙂

In the exception of loading time, I still do not understand why layer groups do not solve this problem? There is no problem that the writer could read multiple files from the disc, and return single layer groups.
Maybe above some number of layers, the layer list should have a search box? and the option Show this with hiding all others?
Or do you mean about situations where you would like to use a single layer in multiple spaces?

@Czaki I totally agree that layergroups can solve this problem - getting the GUI right feels really important though. Building on what you said, what do you think of the proposal below

In the image above, the root nodes of the layer group tree are 'spaces', navigated with your combo/search box. Only one of these root nodes can be visible at a time. The layers in each 'space' (branch of the tree) are controlled with controls as normal

So spaces will be a list of possible views for fast switching between them?
Something like multiple files panel from PartSeg https://partseg.readthedocs.io/en/latest/interface-overview/interface-overview.html#multiple-files-panel but with search (and maybe a better option for renaming elements?

While layer groups will help in general they don't actually address the same problem. My proposal for spaces is intended to solve the problem of data that does not belong to the same coordinates system. I think this comment above explains well why layergroups are not a solution. Indeed, PartSeg's file panel is basically targeting the same issue, but would be limited to files. We can generalize that to any collection of layers.

So spaces will be a list of possible views for fast switching between them?

I guess... which makes me think: maybe we should also allow for a layer to exist in multiple spaces at once? This would allow people to use spaces in a different way: to quickly load "presets" of combinations of layers in order to look at different visualizations. This would play well into our future efforts towards multi-canvas (you could simply activate more than one space, and boom).

Indeed, PartSeg's file panel is basically targeting the same issue, but would be limited to files. We can generalize that to any collection of layers

Of course, this cannot be used directly because it is pinned to the inner PartSeg data representation (which is still different from napari even after some migrations). I would link it as an example of an idea/interface.

but would be limited to files
PartSeg uses a single entry point file load so a single entry could represent multiple files. Also, the code of the widget does not assume that it needs to be a real file path but a unique name.

My proposal for spaces is intended to solve the problem of data that does not belong to the same coordinates system. I think this comment above explains well why layergroups are not a solution.

For me, your comment does not explain this. This comment suggest that this is something like change layergroups and reset view to adjust view to data.
As data does not share a coordinate system, the only reason I see to use "spaces" is the fast switch between data points. If yes, then such spaces should have subspaces for different interpretations (labels, points, shapes).
Maybe this is  wording things but base on your description it is better for me to think about this as "states" not "spaces". It may be correct?
My biggest wories about such feature are possible MemoryErrors. In PartSeg I need to add compressing labels data to reduce such errors when user store multiple segmentaions. But this of course slowdown state switch.
To address my worries it will be nice to attach information about this feature usage to error report about memory problem for simplify understand user issues/bug reports.

I think I wasn't very clear; when I say that they "do not share a coordinate systm,", I don't mean simply that they are "in different places" or that they use different transforms; what I mean is that the "world" of layer 1 is not the same as the "world" of layer 2.
 They are images from two different samples, whose feature you want to analyse, but that do not belong to the same plate/grid/support or (that is, there is no specific (or relevant) spatial or temporal relationship between them).
 You can indeed group them in two groups (i.e: i1+ and i2+). Now, what's the point of having the two groups in the viewer? You probably want to something with the collective data; however, you don't want to show them at the same time. You might want to show them side by side for visual comparison, sometimes even overlap them for some quick morphological observation, but in most cases you will simply switch between showing one group, to showing another group.
In my case, this happens all the time, multiplied by a lot: there is almost no reason for them to be on the canvas together, but I want them somewhere "in the viewer" so I can access their data quickly (and this is what I was showing in the example above). I also don't want the layerlist to be a huge mess where I'm forced to scroll (or search) the layer that I'm looking at anyways. Even better, it would be nice if there was a shortcut to quickly switch between these spaces so one doesn't have to make visible/invisible specific layer groups every time.
So in short, while I agree that layergroups are still the way forward and will make interacting with the layerlist smarter and easier, I still think that some higher-level separation of "spaces" (or workspaces, or even "states" as you say, depending on how the uses uses it) would be a nice addition.

So in short, while I agree that layergroups are still the way forward and will make interacting with the layerlist smarter and easier, I still think that some higher-level separation of "spaces" (or workspaces, or even "states" as you say, depending on how the uses uses it) would be a nice addition.

I agree. :)
But before implementing this I would love to see multiple columns docking areas for a clear interface.
And I think that It could be simply implemented as a plugin.

Counterpoint from discussion with @sofroniewn - if the spaces are really separate then why would the camera/dims be the same when changing spaces?
Because of that I'm now convinced that some ergonomic 'multi viewer' situation is actually what's most correct/flexible
People have discussed 'tabbed' viewers in the past as an implementation of this - I think for large numbers of viewer states a searchable combobox containing viewer states to be switched in and out is the way to go (probably situated somewhere around the layer list)
Switching spaces in the same viewer and tabbed viewers aren't necessarily mutually exclusive - imagine if clicking a viewer state applied it on the current viewer, open in new tab or open in new window could be available through a context menu :)

I think for large numbers of viewer states a searchable combobox containing viewer states to be switched in and out is the way to go (probably situated somewhere around the layer list)

Searchable list widget or even searchable QTreeWidget. Searchable combobox is not a good choice for 100 entries with names generated by some reader.

@Czaki don't those two take up a lot more space?

don't those two take up a lot more space?

Yes, they take but are much more readable. And allow to easily understand the current state.
Maybe the proper solution is to have QComboBox by default and a larger widget after clicking some button.

@Czaki sounds great to me!

And when you think about the size you need to remember that this is not only the size of a selector, but there needs to be an interface for managing the state. "Create space", "Delete space", "Rename space", "Update space" etc. And I'm not sure if such interface should be only in right-click menu

Right - I think it's because I subconsciosuly was already assuming that viewer state should be preserved when going back to a space, but I guess I never explicitly stated that. So yeah, I think we agree that ultimately we want that :P
However, one thing I think is very important: this should be something contained by the ViewerModel. I don't want to juggle 100 different Viewer objects. I Imagine this being something that sits just above the layerlist, but below the viewer. In that case, I suppose a different dims and camera (and potentially canvases...) should exist for each space.
Alternatively (and this is easier to implemen wihtout big structural changes), we just make it quick way to save and then serialize/deserialize viewer states. This way we just have one viewer object to deal with.

Totally agree :) especially on the juggling multiple viewers!
yay, seems like we have some consensus

However, one thing I think is very important: this should be something contained by the ViewerModel. I don't want to juggle 100 different Viewer objects. I Imagine this being something that sits just above the layerlist, but below the viewer. In that case, I suppose a different dims and camera (and potentially canvases...) should exist for each space.

Just catching up here - this is where I might disagree. If dims & camera etc are changing then you really have different viewers as a viewer contains a dims and camera.
Some of this was discussed in #3956 and #3955.
We could have a top level API like app and then something like
app.viewers # list of all viewers
app.viewer # current viewer model
app.window # the singleton window now lives on the app rather than the viewer

If you only have one viewer you'd just be working with viewer = app.viewer as before?
What does everyone think of this? It would probably be a 0.5.0 feature. I know @Carreau @jni @tlambert03 might have thoughts here too.
Is this such a major API change we might want to write a NAP?

Just catching up here - this is where I might disagree. If dims & camera etc are changing then you really have different viewers as a viewer contains a dims and camera.

No. You just could have to serialize/deserialize operations for state and use it to save/restore state with information about dims and camera positions.
This of course will be slower than having multiple viewers with all data in GPU memory but maybe nicer for less powerful computers user.

Just catching up here - this is where I might disagree. If dims & camera etc are changing then you really have different viewers as a viewer contains a dims and camera.

This is the way I see it: if you think of the viewer + dims + camera as your photography room/light setup/camera, layers are the objects and people that you're imaging, and spaces encode the way you arrange those subjects in your room (and which ones). At any times you should eb able to pick up an object from your storage and stick it in the room. You should also be able to reconstruct the room as it was two days ago. Anopther viewer would juyst be a completely new room, that has no shared storage or history with the other one.

Is this such a major API change we might want to write a NAP?

Either way, probably yes :P As soon as we have collected enough thoughts, I might do that!

Adding serialization/deserialization layers spaces will allow to save it in one viewer and load in another.

Trying to catch up here, so I'm going to try to summarize what I understand from the discussion above. Overall, I understand the motivation for the feature and think there are some solid ideas already.

This is the way I see it: if you think of the viewer + dims + camera as your photography room/light setup/camera, layers are the objects and people that you're imaging, and spaces encode the way you arrange those subjects in your room (and which ones).

This is effectively how I think about slicing/sampling the layers too. We're using the current state of (dims, camera, layerlist) as input and generating the sliced/sampled data + other stuff that we want to present in the canvas (e.g. that we send to vispy).
In the feature described here we have multiple sets of (dims, camera, layerlist) that we can switch between and one layer only belongs to one set. In what I imagine multi-canvas to be, we can present many of those sets of (dims, camera, layerlist) simultaneously and a layer can be presented in multiple canvases.
So I think this feature makes a nice precursor to multi-viewer/canvas, as it supports multiple viewers in the background, but doesn't work out all the details of how to present all of those in the GUI simultaneously. I think it also doesn't depend on removing the slice/sample state from the layers (because one layer belongs to exactly one space), whereas multi-canvas probably does.

However, one thing I think is very important: this should be something contained by the ViewerModel. I don't want to juggle 100 different Viewer objects.

Currently the Viewer is the ViewerModel due to inheritance. The Viewer also contains Window and thus QtViewer, some of which will need to at least be aware of the current space (e.g. QtDims needs to update to reflect the current space's Dims). Again, I think multi-canvas/viewer is a harder problem that requires major surgery on QtViewer because some widgets may need to be per-viewer (e.g. QtDims) whereas others probably don't (e.g. QtConsole).

No. You just could have to serialize/deserialize operations for state and use it to save/restore state with information about dims and camera positions.

I think we absolutely want to support full serialization of the whole viewer state to disk, but I see this as more akin to saving and loading files.

This of course will be slower than having multiple viewers with all data in GPU memory but maybe nicer for less powerful computers user.

I can see how memory limits could become a concern, but maybe we can address them by not sending over data from viewers that you're not looking at.

Is this such a major API change we might want to write a NAP?
Either way, probably yes :P As soon as we have collected enough thoughts, I might do that!

Great, I think what I'd like to see are API proposals and architecture diagrams here.
I'm still most excited about the API I proposed above https://github.com/napari/napari/issues/4419#issuecomment-1129443846

We could have a top level API like app and then something like
app.viewers # list of all viewers
app.viewer # current viewer model
app.window # the singleton window now lives on the app rather than the viewer

If you only have one viewer you'd just be working with viewer = app.viewer as before?
What does everyone think of this? It would probably be a 0.5.0 feature. I know @Carreau @jni @tlambert03 might have thoughts here too.

I really like this. One thing I would like is to maintain our current Viewer API as the primary creation API in script world and having an 'accessor' for the current app if it is needed
something like
import napari

viewer = napari.Viewer()
app = napari.get_app()

rather than
import napari

app = napari.Application()
viewer = app.viewer

these are details that can be worked out as we go, overall I'm super excited about this!!

I haven't read everything carefully... so forgive me if this was covered,  but:

my first thoughts are with @jni: "isn't this just multiple viewers?"

it seems like @brisvag's primary comment in response to @jni saying "isn't this just multiple viewers?" was "yes, but I have hundreds" of them.  Which doesn't necessarily mean that thinking of this as multiple viewers is incorrect, but just that we need to have better ways of hiding/showing/grouping Viewers.  Something like a ViewerList, akin to a layerlist, where each viewer can have some visibility and other state that determines how computationally "active" it is.

my second thought is that we have generally under-utilized the LayerList itself as a swappable object.  We think of a viewer as having one permanent/indelible layerlist... but there's no conceptual reason that you shouldn't be able to change the layerlist with viewer.layers = other_space.  (certainly, we'd have a lot of cleanup to do to make this doable, since we've generally been rather lazy about disconnecting things... which is why things like #3184 are so hard to do).  But really, the isolation is mostly in place to let a viewer change out its layerlist.  The viewer is mostly just a view onto a layerlist model.  So inasmuch as we can't do this it's kind of an oversight/bug.

in the case of a swappable layerlist, one then has to ask, what is the viewer beyond a set of layers, which comes back to @sofroniewn's comments about dims and cameras.  So, in the case of spaces here, is there really anything else about the viewer (beyond just the layerlist) that needs/wants to be synchronized across spaces?  Do you want the camera to stay constant when you switch between spaces?  And if not, doesn't that just bring us back to multiple viewers?

app.viewers ... we do essentially have this already (it's napari.Viewer._instances).  I'm not at all opposed to another level of abstraction by creating an app API, but I sort of see that as a separate issue (i.e. until we have something other than a set of Viewers that we want to track, discussion of that API is a bit orthogonal to this here).

(side note: I also do think that the side question/problem of reader plugins not being able to create multiple viewers/layerlists is a bit interesting, certainly as datasets get massive...)

Part of layers should be sharable between spaces/states. Something like data with differents interpretations.

Something like data with different interpretations.

currently the layer is the interpretation of the data... (i.e. it's a tight coupling between data and some display properties) but once we abstract out the concept of a "data source", then it would be much easier to accomplish this, as multiple layers (i.e. "interpretations") could share the same data source.

Which doesn't necessarily mean that thinking of this as multiple viewers is incorrect, but just that we need to have better ways of hiding/showing/grouping Viewers. Something like a ViewerList, akin to a layerlist, where each viewer can have some visibility and other state that determines how computationally "active" it is.

Perfect summary! It feels like we've reached consensus here on 'spaces' being multiple viewers.

we have generally under-utilized the LayerList itself as a swappable object.

in the case of a swappable layerlist, one then has to ask, what is the viewer beyond a set of layers, which comes back to @sofroniewn's comments about dims and cameras.

I agree that being able to swap the layerlist in and out would be great and a good example of isolation being well handled but I can't see when this would be necessary/more appropriate than switching viewers in the case where we have great viewer switching functionality - do you agree here?
I think being to do either will likely create a situation where some plugin developers use the multiple viewers API, some the multiple layerlists... It feels important to handle this idea of  'data in multiple spaces'  uniformly within the plugin ecosystem. I still really like the idea of

It could be that 'data processing workflows' are defined in one space then applied over many spaces

from https://github.com/napari/napari/issues/4419#issuecomment-1126353589, replacing spaces with viewers 🙂

i.e. until we have something other than a set of Viewers that we want to track, discussion of that API is a bit orthogonal to this here)

Agreed, the obvious candidate for something which would best be stored at app level is data sources accessible in different viewers but it's good to know that this can be thought of separately

(side note: I also do think that the side question/problem of reader plugins not being able to create multiple viewers/layerlists is a bit interesting, certainly as datasets get massive...)

Yeah, this is absolutely the motivating case for some incarnation of 'spaces' (multiple viewers) and is one reason why reader plugins haven't worked for us (you previously mentioned handling hdf5 being limited by similar issues)

I agree that being able to swap the layerlist in and out would be great and a good example of isolation being well done but I can't see when this would be necessary/more appropriate than switching viewers in the case where we have great viewer switching functionality - do you agree here?

It would be more appropriate when (and only when) you wanted to keep the other items in the ViewerModel (i.e. axes, camera, dims, grid, etc...) constant, but only change the set of layers. (not sure if that's the case here... but that's when)

I still really like the idea of batching analysis workflows across multiple spaces (viewers) from https://github.com/napari/napari/issues/4419#issuecomment-1126353589

In the past, @sofroniewn and I have enjoyed thinking about the LayerList as the (view-independent) object that might serve as a good unit to pass around in workflows.  (I think we should try to keep gui objects like the viewer out of workflow discussion as much as possible)

It would be more appropriate when (and only when) you wanted to keep the other items in the ViewerModel (i.e. axes, camera, dims, grid, etc...) constant, but only change the set of layers. (not sure if that's the case here... but that's when)

I'm with you, this has got my mind whirring and I think there's a useful nugget here. It feels like we need to support

have a single viewer with relatively constant depiction (i.e. only swap data in and out of a single viewer/set of layers)
have multiple viewers with different depictions of (potentially different, or not) data

Once we have the separate DataSource concept, both become much easier

In the past, @sofroniewn and I have enjoyed thinking about the LayerList as the (view-independent) object that might serve as a good unit to pass around in workflows. (I think we should try to keep gui objects like the viewer out of workflow discussion as much as possible)

This is interesting and really good to keep in mind - it feels somewhat incompatible with the separation of DataSource

In the past, @sofroniewn and I have enjoyed thinking about the LayerList as the (view-independent) object that might serve as a good unit to pass around in workflows. (I think we should try to keep gui objects like the viewer out of workflow discussion as much as possible)

This is interesting and really good to keep in mind - it feels somewhat incompatible with the separation of DataSource

If we end up with a list of DataSources that can be placed into different layers then that might become a more natural unit to pass around workflows - I think key idea was we wanted what get's passed around workflows to view independent. This is maybe a side discussion though, best for another issue.

my second thought is that we have generally under-utilized the LayerList itself as a swappable object ... The viewer is mostly just a view onto a layerlist model. So inasmuch as we can't do this it's kind of an oversight/bug.

Completely agree with this - and if this isn't working this is something that we should fix regardless of the outcome of this discussion.

app.viewers ... we do essentially have this already (it's napari.Viewer._instances). I'm not at all opposed to another level of abstraction by creating an app API, but I sort of see that as a separate issue (i.e. until we have something other than a set of Viewers that we want to track, discussion of that API is a bit orthogonal to this here).

I guess for me this is just a super concrete way to understand and expose the usage through a public API. While it might not be strictly necessary I find it helpful as it can give a clear publicly documentable way for folks to move between different viewers and elevates the behavior to being first class. I wonder what @jni thinks

I really like this. One thing I would like is to maintain our current Viewer API as the primary creation API in script world and having an 'accessor' for the current app if it is needed

I completely agree with this @alisterburt.

I think this thread is getting quite long now - lots of great ideas and great discussion, but I want to make sure that it doesn't fizzle out / become too hard for folks to follow. I wonder if @brisvag and others have enough input here to begin writing a NAP with an implementation proposal for whatever they think is best. If not, then we can keep discussing here more too

I think my main initial confusion about this discussion boils down to this: what do we mean by a viewer?

A literal ViewerModel with all the stucck that comes with it right now, a completely independent object that can spawn its own GUI (either as a separate windows, or as a tab, or whatelse)?
Or just an object that "serializes" a layerlist and info about axes, camera, and dims? (I might be using the serialize term improperly here, judging by some of the answers above; I simply mean "describe and remember some info so they can be retrieved and reconstructed as needed")

If it is the former, then I disagree, and that's my fear when I talk about "juggling a million objects". Will the console belong to a viewer, then, or to the app?
If it is the latter (as it seems, from the last few comments), then a viewer is a space. I'm fine with that, I'm not particularly attached to the terminology; I found it confusing because viewer is very much tied to the window, in my mind. You say "open the napari viewer", not "open the napari window, which is displaying the contents of a viewer" :P
I'm pretty sure everyone seems to be leaning towards the latter (correct me if I'm wrong!). I'll try to summarize some thoughts:

while swapping layerlists in and out of the viewer would be fine for many cases (In fact, I did this before, with some ugly workarounds to the current bugs due to lack of disconnections mentioned by @tlambert03), the ability to switch viewer state as well seems pretty important, so something smarter seems useful
DataSource will definitely be useful to make this able to look at different interpretations of the data; in the meantime, I don't think this possibility precludes any of the ideas we proposed, so that's good.
I like the app idea, but I agree with @alisterburt that the primary interface should probably be the Viewer. In general, I think that the key issue to solve here is to make it easy to find the viewer/data you want to display. (searchable dropdown, zoommable graph, voice-activated assistant...)

I will try to write this stuff up in a NAP soon. Thanks everyone for the input, very cool discussion :)

Will the console belong to a viewer, then, or to the app?

I'm not sure bringing in the scope of the console in to this discussion is terribly illuminating.  It too is an independent entity.  It's scope is actually the "process" (neither the viewer nor the app)... and if we wanted to, we could probably make it appear to have a scope/context that was more limited.  but again, I see that also as an orthogonal discussion.

what do we mean by a viewer?

well... I think your first description is the Viewer and the second description is the ViewerModel :joy: ...   Viewer is simply a ViewerModel with a window.  Though perhaps it's semantic: the Viewer is really not much of anything.  It's a composition of a ViewerModel and a Window (where the Window is a view on to the ViewerModel).

the ability to switch viewer state as well seems pretty important, so something smarter seems useful

this comes back to my question above of "is there really anything else about the viewer (beyond just the layerlist) that needs/wants to be synchronized across spaces?"  If you have a set of things beyond the set of layers that you want to swap in/out (like camera, dims, etc...) , then yeah, we need something like a full ViewerList (beyond just a swappable LayerList).

This is now a NAP!

#4684

Closing this, since the discussion has been moved to the NAP.