ConnectionError while loggin in

I installed the netflix addon following the procedure found on Kodi forum post 4 on OSMC (Kodi version 18). I am based in cambodia. When i try to log in, i got the following:
Exception: Connection error
Error details:
HTTPSConnectionPool(host='www.netflix.com',port=443):Max retries exceeded with url: /login (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x6038b450>: Failed to establish a new connection: [Errno-2] Name or service Not known',))
Is it an issue ? How can it be fixed ?

Please find attached the log
kodi.log

I have the same issue.

Same issue here. 443 errors where there weren’t any before...but port 443 is https, which suggests a networking problem, yet every other device on my network can play Netflix. (Current xbian on RPi4).

Happens when I try to login with both email/password and authentication key

the error mentioned is a network error
could be you have a broken/unstable DNS address in your network
or something similar that prevent a good communication
i can not do nothing here

No, it is an addon problem:  I can update other add ons from within kodi AND I can watch Netflix on other devices on my network. DNS isn’t the issue. I suspect it’s crypto-related. Will test and update. Which version of pycryptodomex do you test with?
Sent from nowhere near my ready room with my new tricorder

On Oct 19, 2020, at 10:38, Stefano Gottardo<EMAIL_ADDRESS>wrote:
Happens when I try to login with both email/password and authentication key
the error mentioned is a network error
could be you have a broken/unstable DNS address in your network
or something similar that prevent a good communication
i can not do nothing here
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

cryptodome? in what sense?
/login endpoint mentioned before do not use crypto it is a simple http request
anyway if you want to test something:
On windows/android: crypto package is embedded on kodi
On linux there are three different crypto packages that can be used:
PyCrypto
PyCryptodome
PyCryptodomex
when more packages are installed by default the addon give priority to PyCryptodomex

In my head, Netflix would want a secure https connection on 443, with some sort of key exchange between host (them) and client (us). A failure or issue there would result in a closed connection, wouldn’t it? And that response returned as the error.
I’m probably wrong, off somewhere in my logic, but it would be good to know where and why.
Sent from nowhere near my ready room with my new tricorder

On Oct 19, 2020, at 11:44, Stefano Gottardo<EMAIL_ADDRESS>wrote:
cryptodome? in what sense?
/login endpoint mentioned before do not use crypto it is a simple http request
anyway if you want to test something:
On windows/android: crypto package is embedded on kodi
On linux there are three different crypto packages that can be used:
PyCrypto
PyCryptodome
PyCryptodomex
when more packages are installed by default the addon give priority to PyCryptodomex
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

https by default use only the port 443 and is good

with some sort of key exchange between host (them) and client (us)

if it is what i think, this is the standard behaviour of the https protocol, but this is managed automatically by Request package otherwise you do not estabilish the communication
if you discover something let us know

I that can helps, i tried to connect netflix from raspi OS desktop Chromiun browser and it worked.
I had exactly the same hardware and network configuration as the one i used to try netflix.

This error could be SELinux-related.  In essence it is either DNS (i.e. not working properly), security policy (SELinux), firewall/network connectivity, or perhaps a broken IPv6 network (i.e. not routable).
DNS you can rule out by running "host www.netflix.com" via the OSMC SSH login.  For SELinux, perhaps run "journalctl" via the OSMC SSH login and see if there are any messages around the same time.
I have had zero issues with the Netflix plugin on OSMC Vero4K+ (initially running the Stretch build, now using Buster build w/ 4.9 kernel).

Ok, problem fixed, and not related to the plugin itself. I had a DNS issue on my network.
I fixed the problem by doing:
My OSMC > Network > Connection > Configure manually > change primary dns IP address to <IP_ADDRESS>.
For user-friendliness, this quick fixe could be suggested tu users when this error show up.

thanks i will add in the wiki

I'm getting the same error for /browse call on Windows.
kodi.log

I'm getting the same error for /browse call on Windows.
kodi.log

Me too. Check DNS configuration, no problems.
Maybe someone got a clue.


Set char check limit aria-live to assertive when over limit

What is the context of this PR?
This PR changes char check limit aria-live to assertive when over character limit in inputs and textfields
How to review
Check char check limit changes to assertive when over character limit and back to polite when back under

 MEDIUM: Char limit reached audio feedback (pg.41) >> Add aria-live alert to limit warning

Weren't we going to add aria-describedby to text areas too?

Weren't we going to add aria-describedby to text areas too?

Oh yeah forgot about that bit, Ill add it


Negative start dates break database parsing

start = datetime.fromtimestamp(int(show['start']), tz=utc_zone).replace(tzinfo=None) fails with an OSError: [Errno 22] Invalid argument for negative start dates.
I added some logging
                                try:
                                    start = datetime.fromtimestamp(int(show['start']), tz=utc_zone).replace(tzinfo=None)
                                except OSError:
                                    logger.error("Error parsing show %s: start=%s", title, show['start'])
                                    continue

which showed lots of shows with negative dates
                    ERROR    Error parsing show Europlast: start=-127635900
                     ERROR    Error parsing show Bundespostminister Stücklen eröffnet automatische Vermittlungsstelle für Auslands-Ferngespräche: start=-127473420
                     ERROR    Error parsing show Nerz und Brillanten unterm Weihnachtsbaum: start=-127549260
                     ERROR    Error parsing show Museen ohne Mittel? Von 35 Millionen Lottogewinn gingen 5 Millionen an staatliche Museen.: start=-127645680

The question probably is: Why are there negative values in the first place?
EDIT: Because the Filmliste is originally created by Java, it probably means

A negative number indicates the number of milliseconds before January 1, 1970, 00:00:00 GMT.
https://docs.oracle.com/javase/7/docs/api/java/sql/Date.html

Thank you for the report.
How can I reproduce this (what cmd do I need to run)?

I ran py -m mtv_dl download "channel=ZDF topic=heute-show duration+20m" using Python 3.7.7 on Win x64, mtv_dl==0.15 (pip installed)
But actually any database download and subsequent read should trigger this.

With Python 3.7.3 on macOS (via pyenv) it works like a charm and also the negative dates are interpreted correctly.
./mtv_dl.py list title=Europlast

  hash        │ channel │ title     │ topic      │ size │ start                     │ duration │ age                 │ region │ downloaded
 ═════════════╪═════════╪═══════════╪════════════╪══════╪═══════════════════════════╪══════════╪═════════════════════╪════════╪════════════
  2a15e227e08 │ ARD     │ Europlast │ Abendschau │ 9    │ 1965-12-15T18:35:00+01:00 │ 38s      │ 54y 10mm 16d 2h 47m │ DE     │ None
  135bdf1fc4c │ SWR     │ Europlast │ Abendschau │ 9    │ 1965-12-15T18:35:00+01:00 │ 38s      │ 54y 10mm 16d 2h 47m │ DE     │ None

Initially I thought the dates are obviously wrong in the Filmliste file, but the show start time is correct.
Unfortunately I do not have a Windows system available to test it, but i think we are facing this bug: https://bugs.python.org/issue36439
I would offer to catch and ignore this exception (what basically means ignoring such old shows on Windows) if this would help you? But I think I cannot really solve the bug.

One idea: Is this working on your machine?
>>> from datetime import *
>>> datetime(1970, 1, 1, 1) + timedelta(seconds=-25200)
datetime.datetime(1969, 12, 31, 18, 0)

i think we are facing this bug: https://bugs.python.org/issue36439

Yes, this seems to be the problem. Only positive values are documented as supported though. So even if it works on Linux/MacOS it probably should be replaced with a solution which does not rely on undocumented features.

>>> from datetime import *
>>> datetime(1970, 1, 1, 1) + timedelta(seconds=-25200)
datetime.datetime(1969, 12, 31, 18, 0)

Works perfectly on my system. So I would suggest to replace the fromtimestamp method with a function based on this snippet. I am not sure if there are any weird details like leap seconds which would need to be taking into account, but I guess it really doesn't matter if the start time is off by a few seconds.

Please have a look into the branch issue34 (https://github.com/efenka/mtv_dl/tree/issue34). I tried to implement a workaround there.

Now the error is with the timestamp() method...
2020-10-15T15:59:11  CRITICAL <class 'OSError'>: [Errno 22] Invalid argument
                                File "C:\Program Files\Python37\lib\runpy.py", line 193, in
                              _run_module_as_main
                                  "__main__", mod_spec)
                                File "C:\Program Files\Python37\lib\runpy.py", line 85, in
                              _run_code
                                  exec(code, run_globals)
                                File "C:\Program Files\Python37\lib\site-packages\mtv_dl.py",
                              line 1265, in <module>
                                  main()
                                File "C:\Program Files\Python37\lib\site-packages\mtv_dl.py",
                              line 1208, in main
                                  history=cw_dir / HISTORY_DATABASE_FILE)
                                File "C:\Program Files\Python37\lib\site-packages\mtv_dl.py",
                              line 434, in __init__
                                  self.initialize_filmliste()
                                File "C:\Program Files\Python37\lib\site-packages\mtv_dl.py",
                              line 386, in initialize_filmliste
                                  """, self._get_shows())
                                File "C:\Program Files\Python37\lib\site-packages\mtv_dl.py",
                              line 546, in _get_shows
                                  'hash': self._show_hash(channel, topic, title, size, start),
                                File "C:\Program Files\Python37\lib\site-packages\mtv_dl.py",
                              line 467, in _show_hash
                                  h.update(str(start.timestamp()).encode())

Well, at least one step further :)
Anyhow, I think I should not fight against this implementation detail of the python interpreter. Sorry. I will add a catch / ignore and skip these shows.
Of course, patches for a better solution are welcome.

Instead of skipping them it would be nicer to just assume a date of 1970-01-01. Then they could still be downloaded at least.


Submission Cleanup

Is this pull request associated with an issue(s)?
No.
Description
This PR should do the final cleanup tasks for the repository before submission.
TODOs

[x] Remove paper/outline.md.
[ ] Make sure the license headers are not appearing in the diffs.
[ ] @ryanmrichard Should the workflow changes related to the paper stay?

Yeah, keep the changes associated with the workflow


(test)cluster upgrade: use correlation id for ansible playbooks(test)

Create a random 10 hex-digit string to use as a correlation id
when starting the cluster upgrade playbook.  The correlation id
will be used by all of the progress events and by the other tasks
run by the ansible role.
Once the playbook has started, push the user to the events grid
with a filter set for the correlation id.  This will show the progress
of the upgrade as it happens.  Any failures encountered by the role
should also be visible.
Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=2040474
Change-Id: Ibce8d71da169634d3ab5578ac39b80ac2846a533
Signed-off-by: Scott J Dickerson<EMAIL_ADDRESS>

/ost


The oid of 'bf_cbc' is incorrect

The '<IP_ADDRESS>.4.1.3029.1.2' represents 'cryptlib public-key encryption' in https://oid-rep.orange-labs.fr/cgi-bin/display?oid=<IP_ADDRESS>.4.1.3029.1.2&a=display.
The correct OID of bf_cbc should be '<IP_ADDRESS>.4.1.3<IP_ADDRESS>'.

This OID has been set that way and unchanged since 1999. It seems to originate from the original Blowfish IETF draft:
https://datatracker.ietf.org/doc/html/draft-schneier-blowfish-00
This gives the OID definition as:

blowfishCBC ALGORITHM PARAMETER CBCParameter ::=
{iso(1) org(3) dod(6) internet(1) private(4) enterprise(1)
dds(3029) symmetric-encryption(1) 2}

This matches the definition in OpenSSL.
The '<IP_ADDRESS>.4.1.3029 arc is  for Peter Gutmann's cryptlib. It appears that at some point after the publication of that draft in 1999, Peter changed the OID definition to be <IP_ADDRESS>.4.1.3<IP_ADDRESS>
It's actually unclear to me what this OID is actually useful for - although you can fetch the BF-CBC algorithm by OID name if you so wish:
https://github.com/openssl/openssl/blob/c07a34e18b098b77ce7ecb14273b7c75f59b5871/providers/implementations/include/prov/names.h#L178
Changing this value therefore represents a potential backwards compatibility break (although possibly a theoretical one since it seems unlikely anyone is actually using this OID for anything).
Given this is a dead algorithm, I'm not sure "fixing" it now actually solves any real problems.

This mostly indicates that no one is using blowfish in a way where the OID is important.
I'd suggest correcting it or removing it.  Letting it continue to be wrong visavi cryptlib's arc (which I haven't been able to find documentation on, btw) would be wrong.
Side note: looking at cryptlib, the latest release doesn't contain blowfish.  Food for thought...

FWIW, Java seems to have the updated OID: https://github.com/slachiewicz/keystore-explorer/blob/c1ae00dc67ed905c703b467f486bd29451b51540/kse/src/net/sf/keystore_explorer/utilities/oid/ObjectIdUtil.java#L1154
See also: https://wiki.strongswan.org/issues/740#note-1

It's a bit surprising that the news reached up this late, especially considering that the strongswan issue odes refer to OpenSSL...

We cannot change the OID without breaking backwards compatibility with existing (albeit wrongly) encoded data. We could add the correct OID as an alias for the algorithm IMO but we cannot drop the existing incorrect OID association with bf_cbc.

@liwei3013 are you willing to submit a PR to add the correct OID?


modify setup.py, fix no package installed in site-packages

Add "packages=['flask_acl']" in setup.py , make it avaliable for pip install via "git+https://github.com/mikeboers/Flask-ACL"

+1 for this fix.

I seriously never pulled this? I'm sorry folks!


Terraform AVDI update

Create readme for the AVD insights module
Updated description to variables file
Add sample terraform.tfvars file for deployment

#sign-off


Still have an issue with purchase date wrapping prematurely on bottle detail page

see https://my-whiskies.online/charlie/7a604b7b-c4bd-42a4-9ac8-7c545cf77f5e

This is done and I am thrilled. Not just the "purchase date". The whole thing. Booyah.


Build SAS summary file parser for DAXA

That can then be used to determine what processed files we expect ahead of time, rather than pattern matching like I do now

To be clear this isn't going to parse absolutely everything from the summary files into a Pythonic format, because we absolutely do not need all of that information, but it will grab things like exposure IDs, the modes of those sub-exposures, the filter of those exposures etc.
That way we can make informed decisions as to what to attempt to process before starting.


Fixed line-breaks missing

Hi Mike, it appears that the current version is missing line breaks after conversion. I am attempting to change innerText to value, and this seems to be working well.
Here is a sample test.
<svg width="26" height="7" viewBox="0 0 26 7" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M26 3V0H0V3H6.45454C7.74219 3 8.95469 3.60625 9.72727 4.63636C11.3636 6.81818 14.6364 6.81818 16.2727 4.63636C17.0453 3.60625 18.2578 3 19.5455 3H26Z" fill="#D9D9D9"/>
</svg>

Thanks @rijieli!


rm "unexpected response" grep from system test

remove unneeded "unexpected response" inverse grep from system test that was readded in #925

LGTM


Error -13 EACCES permission denied Kestrel. Ubuntu 14.04 LTS

In my ubuntu server 14 LTS (Azure VM). I've installed the KVM following the instructions at aspnet home page and I've configured the libuv... but when I run k kestrel command I got the following error:
nicolocodev@nicolocoubt:~/dev/tesa$ k kestrel
System.Exception: Error -13 EACCES permission denied
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.Check (Int32 statusCode) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.tcp_bind (Microsoft.AspNet.Server.Kestrel.Networking.UvTcpHandle handle, Microsoft.AspNet.Server.Kestrel.Networking.sockaddr& addr, Int32 flags) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.UvTcpHandle.Bind (System.Net.IPEndPoint endpoint) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Http.Listener+<>c__DisplayClass0.<StartAsync>b__1 (System.Object _) [0x00000] in <filename unknown>:0

If i try using sudo I get the message: sudo: k: command not found

Oh! I'm so sorry :( Was my mistake. My problem was with Azure VM's Endpoints (Public/private ports) I changed that configs and everithing works well
Again, sorry :'(

I get the exact same error when running the HelloMvc example on normal Ubuntu 14.04 LTS, but only when I use port 80 in project.json. With port 5004 it works as expected. I'm not an expert in Linux and new to Asp.Net5. Any hint on what I'm doing wrong here?

A normal user is not allowed to bind sockets to TCP ports < 1024, you need more permissions than the normal user has (read: 'root'-access). The easiest and quickest solution probably is using sudo to start the process, but that's also the most frowned upon solution. The reason is that the process then has every access right on the system.
The cleanest solution and the one that for which Kestrel is developed is using a front-end server / load balancer on port 80 (for example nginx, or endpoints in Azure) that forwards the requests to an unprivileged Kestrel on a private port.

Thanks for the quick reply and the great explanation!

Yep! that was my way... defined a public port (80) and a private port for the application
Thanx @Tragetaschen for your great Explanation!

Hello,
it is possible to sudo dnx but sudo expects full path to dnx which can be obtained by running "which dnx" on Linux. It is not that I am saying that sudoing dnx is best practice from the security point of view but it is certainly possible in the way described above.
P. S. So happy that after much trial and error I was finally able to get Beta 6 bits running on Ubuntu 14 LTS on AWS.


Resolves critical security vulnerabilities

As part of #5, this PR upgrades dependencies with critical security vulnerabilities as detected by Snyk:
https://security.snyk.io/vuln/npm:constantinople:20180421
https://security.snyk.io/vuln/SNYK-JS-KNEX-471962
https://security.snyk.io/vuln/SNYK-JS-HANDLEBARS-534988
My client builds, unit tests, integration tests, and Docker containers are all working locally. I've played a practice game to completion to confirm everything is working.
Fixes #43

Unit tests failed after merging due to a transient network issue:

error An unexpected error occurred: "https://registry.yarnpkg.com/express/-/express-4.17.3.tgz: Request failed "503 Service Unavailable"".


downloader: hash files while they're downloading

This avoid rereading the entire file from disk once it's written.

(made a tiny bugfix)


Grid Autosport game.dat

Hello,
will it be possible to save NEFS version 1.6 the one used by Grid Autosport? Thanks!

See https://github.com/victorbush/ego.nefsedit/issues/20.


Query and update animals table

In this pull request I have:

Used database transactions.
Modified and deleted data in SQL.
Prepared complex queries that answered analytical questions using aggregate functions and GROUP BY.
Added species column

Screenshots for queries:

Thanks @Davidosky007


ENS Ropsten -- Registry requiring owner does not show account information

Hello,
I made a bid on ropsten with myetherwallet and went thru. On contract address 0x112234455c3a32fd11230c42e7bccd4a84e02010 when I request the owner of 0x8338ce36960756c347537badd974579a9e627fa8febe6523b9e426f40008d165 via https://www.myetherwallet.com/#contracts I get address 0x0000000000000000000000000000000000000000.
On the contrary when I do the same request via https://ropsten.etherscan.io/address/0x112234455c3a32fd11230c42e7bccd4a84e02010#readContract I have the response I wish, i.e. 0xa32cf44585bc9891586734fc2ee87c5ef09d7a8c, the bidding account.
By chance this account is the second ETH account of a ledger nano S (I don't think this last precision is of any interest but...)

@Arachnid Do we have outdated Ropsten information somewhere?
Configs Folder: https://github.com/kvhnuke/etherwallet/tree/mercury/app/scripts/ensConfigs
Ropsten: https://github.com/kvhnuke/etherwallet/blob/mercury/app/scripts/ensConfigs/ROPConfig.json
{
    "public": {
        "resolver": "0x4c641fb9bad9b60ef180c31f56051ce826d21a9a",
        "reverse": "0xdb6cead81ce14a63c284728eed17738a81327ff0",
        "ethAuction": "0xc19fd9004b5c9789391679de6d766b981db94610"
    },
    "registry": "0x112234455c3a32fd11230c42e7bccd4a84e02010"
}

Ropsten node was waaaaaay behind, it caught up now. Problem solved :+1:


Attribution?

Reading through the code, it seems like this library is almost a direct port of https://github.com/andreas/ocaml-graphql-server to Reason, but I didn't find the original library's copyright notice or license anywhere in the repo.
Am I missing something?

It's in the first paragraph of readme. The repo is also MIT just like the original OCaml repo.

but I didn't find the original library's copyright notice or license anywhere in the repo.

I'm not trying to be difficult but saying that the library is inspired by the other is not abiding by the terms of the license. Having been bitten by this once before, I'm just trying to help here.

@anmonteiro I will change the wording, thanks for looking out.

@anmonteiro done!


Snappy compression doesn't seem to compress anything

Version: Parquet.Net v<IP_ADDRESS>
Runtime Version: .Net Framework v4.5.2
OS: Windows
Background
I looked at this sometime ago for a prototype and I'm picking this back up again as work to evolve our data formats.  We have an old application with a proprietary columnar format and we need to decide whether to maintain that or modernise.  I see the code base has moved on somewhat since I last got stuck in, which is great.
Write performance is a big deal, we run at considerable scale (typically tens of thousands of cores creating data which feeds into a downstream big data platform).  Our experience elsewhere lead us to the conclusion that Snappy CPU consumption & compression ratio is a lot cheaper than paying the extra for GZip (the compute cost takes a long time to be paid back given a few extra compression % points).  However, I'm not seeing what experience tells me I should see using Parquet.Net and Snappy.
As a side, compression has a huge impact as our data has a lot of repeating dimension values.  It seems there is no dictionary / RLE encoding of repeating string values, all the data reduction comes from the chosen codec.  This would be huge for us as dictionary encoding is cheap and will have benefits on the read side.  I see this issue (https://github.com/elastacloud/parquet-dotnet/issues/279), is there any movement on a way forward?
Expected behavior
Changing the CompressionMethod to Snappy results in a file which is larger than I would expect.  File result using GZip is ~27MB but it takes 00:06:52 to translate into Parquet, using Snappy the file is 183MB and takes 00:01:36 to translate.  If I take the parquet produced using CompressionMethod = Snappy and run that through a snappy compression stream of my own I get 30MB MB (no time overhead, snappy is bullet fast).
Actual behavior
I expect using Snappy CompressionMethod to yield a file in 30MB range not 183MB range.
Steps to reproduce the behavior
I have a custom binary format file I'm converting into Parquet, when I run this with CompressionMethod = GZip I get compression, when I use Snappy I don't.  I can't share the real code but this example replicates the problem

_path = Path.Combine(Environment.CurrentDirectory, Guid.NewGuid() + ".parquet");

WriteFile(CompressionMethod.Snappy);
_snappyLength = new FileInfo(_path).Length;
System.Console.WriteLine(_snappyLength);

WriteFile(CompressionMethod.Gzip);
_gzipLength = new FileInfo(_path).Length;
System.Console.WriteLine(_gzipLength);

System.Console.WriteLine(_snappyLength / _gzipLength);

static void WriteFile(CompressionMethod compressionMethod)
{
    using (var file = File.Create(_path))
    {
        var schema = new Schema(new DataField<string>("field"));
        using (var parquetWriter = new ParquetWriter(schema, file) {CompressionMethod = compressionMethod})
        using (var rowGroup = parquetWriter.CreateRowGroup())
        {
            var data = Enumerable.Range(0, 1000).Select(x => "Value " + x).ToArray();
            rowGroup.WriteColumn(new DataColumn((DataField) schema.Fields[0], data));
        }
    }
}

Console output is:
5130
2253
2

This looks good to me now!


Installation barfs on fresh Mac OS X High Sierra - ld: warning: directory not found for option '-L/Users//grpc/libs/opt/c-ares'

What operating system (Linux, Windows, …) and version?
Mac OS X High Sierra 10.13.2
What did you do?

Installed Xcode and Unix tools

sudo xcode-select --install

Installed via ports

sudo port selfupdate
sudo port upgrade outdated
sudo port install autoconf automake libtool

Cloned gRPC

mkdir ~/install
cd ~/install
git clone -b $(curl -L https://grpc.io/release) https://github.com/grpc/grpc
cd grpc
git submodule update --init
LIBTOOL=glibtool LIBTOOLIZE=glibtoolize make

Steps 1 , 2 - Based on https://github.com/google/protobuf/blob/master/src/README.md Note for Mac users, based on guidance in https://github.com/grpc/grpc/tree/master/src/objective-c#install-protoc-and-the-grpc-plugin-without-using-homebrew
Step 3 - The LIBTOOL and LIBTOOLIZE is based on https://github.com/grpc/grpc/blob/master/INSTALL.md#macos.
What did you expect to see?
No warnings
What did you see instead?
Warnings for all *_plugin.cc
[HOSTCXX] Compiling src/compiler/cpp_plugin.cc
[HOSTLD]  Linking /Users/<user>/install/grpc/bins/opt/grpc_cpp_plugin
 ld: warning: directory not found for option '-L/Users/<user>/install/grpc/libs/opt/c-ares'
<output snipped for brevity>

Warnings for all lib*.dylib
[LD]      Linking /Users/<user>/install/grpc/libs/opt/libgpr.dylib
ld: warning: directory not found for option '-L/Users/<user>/install/grpc/libs/opt/c-ares'
<output snipped for brevity>

Anything else we should know about your project / environment?
Fresh and updated installation of Mac OS X High Sierra 10.13.2. No protoc installed. Protoc being built as part of the gRPC installation. release.dms from grpc.io contains v1.8.x.
Initial investigation
Based on https://github.com/grpc/grpc/blob/master/templates/Makefile.template, there is no mkdir cp for c-ares. There is one present for protobuf - https://github.com/grpc/grpc/blob/master/templates/Makefile.template#L902-L905
  	$(Q)mkdir -p $(LIBDIR)/$(CONFIG)/protobuf
  	$(Q)mkdir -p $(BINDIR)/$(CONFIG)/protobuf
  	$(Q)cp third_party/protobuf/src/.libs/libprotoc.a $(LIBDIR)/$(CONFIG)/protobuf
  	$(Q)cp third_party/protobuf/src/.libs/libprotobuf.a $(LIBDIR)/$(CONFIG)/protobuf

The lines which cause the warning to appear - https://github.com/grpc/grpc/blob/master/templates/Makefile.template#L592-L598,
ifeq ($(EMBED_CARES),true)
  CARES_DEP = $(LIBDIR)/$(CONFIG)/libares.a
  CARES_MERGE_OBJS = $(LIBARES_OBJS)
  CARES_MERGE_LIBS = $(LIBDIR)/$(CONFIG)/libares.a
  CPPFLAGS := -Ithird_party/cares -Ithird_party/cares/cares $(CPPFLAGS)
  LDFLAGS := -L$(LIBDIR)/$(CONFIG)/c-ares $(LDFLAGS)
  else

General searching does not lead to a definitive solution. Installing c-ares externally via ports seems to have been discouraged - https://groups.google.com/forum/#!topic/grpc-io/eVdtmWufE5w.

Tried adopting the "DISABLING ARES" line from @timburks' grpc-swift repository https://github.com/grpc/grpc-swift/blob/master/vendor-grpc.sh#L75-L76 like so
mkdir ~/install
cd ~/install
git clone -b $(curl -L https://grpc.io/release) https://github.com/grpc/grpc
cd grpc
git submodule update --init
perl -pi -e 's/#define GRPC_ARES 1/#define GRPC_ARES 0/' include/grpc/impl/codegen/port_platform.h
LIBTOOL=glibtool LIBTOOLIZE=glibtoolize make

No success. The installation continues to barf with the same aforementioned warnings.
Further, running
grep -r "GRPC_ARES" .

within a freshly cloned gRPC repository suggests that GRPC_ARES seems to be set to 0 everywhere.
./config.w32:    "/D_HAS_EXCEPTIONS=0 /DNOMINMAX /DGRPC_ARES=0 /D_WIN32_WINNT=0x600 "+
./include/grpc/impl/codegen/port_platform.h:#ifndef GRPC_ARES
./include/grpc/impl/codegen/port_platform.h:#define GRPC_ARES 0
./include/grpc/impl/codegen/port_platform.h:#define GRPC_ARES 1
./gRPC-Core.podspec:  s.compiler_flags = '-DGRPC_ARES=0'
./config.m4:    -D_HAS_EXCEPTIONS=0 -DNOMINMAX -DGRPC_ARES=0)
./grpc.gyp:      'GRPC_ARES=0',
./setup.py:                    ('GRPC_ARES', 0), ('NTDDI_VERSION', 0x06000000),)
./bazel/grpc_build_system.bzl:    defines = select({"//:grpc_no_ares": ["GRPC_ARES=0"],
./templates/gRPC-Core.podspec.template:    s.compiler_flags = '-DGRPC_ARES=0'
./templates/config.w32.template:      "/D_HAS_EXCEPTIONS=0 /DNOMINMAX /DGRPC_ARES=0 /D_WIN32_WINNT=0x600 "+
./templates/grpc.gyp.template:        'GRPC_ARES=0',
./templates/config.m4.template:      -D_HAS_EXCEPTIONS=0 -DNOMINMAX -DGRPC_ARES=0)
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_ev_driver_posix.cc:#if GRPC_ARES == 1 && defined(GRPC_POSIX_SOCKET)
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_ev_driver_posix.cc:#endif /* GRPC_ARES == 1 && defined(GRPC_POSIX_SOCKET) */
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.h:#ifndef GRPC_CORE_EXT_FILTERS_CLIENT_CHANNEL_RESOLVER_DNS_C_ARES_GRPC_ARES_WRAPPER_H
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.h:#define GRPC_CORE_EXT_FILTERS_CLIENT_CHANNEL_RESOLVER_DNS_C_ARES_GRPC_ARES_WRAPPER_H
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.h:#endif /* GRPC_CORE_EXT_FILTERS_CLIENT_CHANNEL_RESOLVER_DNS_C_ARES_GRPC_ARES_WRAPPER_H \
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc:#if GRPC_ARES == 1 && !defined(GRPC_UV)
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc:#endif /* GRPC_ARES == 1 && !defined(GRPC_UV) */
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_ev_driver.h:#ifndef GRPC_CORE_EXT_FILTERS_CLIENT_CHANNEL_RESOLVER_DNS_C_ARES_GRPC_ARES_EV_DRIVER_H
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_ev_driver.h:#define GRPC_CORE_EXT_FILTERS_CLIENT_CHANNEL_RESOLVER_DNS_C_ARES_GRPC_ARES_EV_DRIVER_H
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_ev_driver.h:#endif /* GRPC_CORE_EXT_FILTERS_CLIENT_CHANNEL_RESOLVER_DNS_C_ARES_GRPC_ARES_EV_DRIVER_H \
./src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc:#if GRPC_ARES == 1 && !defined(GRPC_UV)
./src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc:#else /* GRPC_ARES == 1 && !defined(GRPC_UV) */
./src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc:#endif /* GRPC_ARES == 1 && !defined(GRPC_UV) */
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper_fallback.cc:#if GRPC_ARES != 1 || defined(GRPC_UV)
./src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper_fallback.cc:#endif /* GRPC_ARES != 1 || defined(GRPC_UV) */

Is there something obvious that I am missing?

Can you please help me understand what could I be doing wrong? I am attempting installation from source on a Mac. This should not feel like a visit to the dentist, except that I am running out of ideas.
@muxi @timburks

@ncteisen @vjpai @ctiller @nathanielmanistaatgoogle @jcanizales
Apologies for looping you folks in. However, I am unable to get installation from source to work on a Mac for some time now. In case you are not the correct folks for this, could you please share the correct folks/mechanism/channel to communicate this? I am hoping to not post this issue again on the gRPC google group, in order to avoid duplication.
Additionally, I am unable to understand the rationale for c-ares, given that GRPC_ARES seems to be set to 0 for most, if not all, build mechanisms (bazel et al). This is based on my limited knowledge of grepping and absolutely no understanding of the underlying code or what c-ares does. It is purely from a perspective of someone trying to get gRPC to build for the last 2 weeks, albeit unsuccessfully.
./gRPC-Core.podspec:  s.compiler_flags = '-DGRPC_ARES=0'
./config.m4:    -D_HAS_EXCEPTIONS=0 -DNOMINMAX -DGRPC_ARES=0)
./grpc.gyp:      'GRPC_ARES=0',
./setup.py:                    ('GRPC_ARES', 0), ('NTDDI_VERSION', 0x06000000),)
./bazel/grpc_build_system.bzl:    defines = select({"//:grpc_no_ares": ["GRPC_ARES=0"],

The above snippet is from the full output of
grep -r "GRPC_ARES"

which is present in the comment preceding the previous comment.

c ares is the async DNS resolver we use.
Can you try installing c ares first? Something like:
cd third_party/cares
make
[sudo] make install

(or something like that, there may be a ./configure step somewhere. Can give a more detailed answer next week)


Creating a visual script virtual function after minimizing crashes

Godot version:
3cf3c4137a4e6a5e25d7950ff727ec2157e40a37
OS/device including version:
Windows 10, gtx 1080
Issue description:
Using the test case from https://github.com/godotengine/godot/issues/23495 I was able to get the editor to crash.
Steps to reproduce:

Open a scene without a script (or not a loaded, empty scene)
Switch to script interface
Open res://Tscn/MainScene.tscn file with script
Need https://github.com/godotengine/godot/pull/23535 patch or crash
Minimize and then maximize the editor
Click the new function button.
Crash

Minimal reproduction project:
Same as https://github.com/godotengine/godot/issues/23495

Fixed by #23537.


in screener() if quantity = 1, returns an empty list rather than the 1st element

fixed by:
if sequence - 20 <= quantity < sequence:
instead of:
if sequence - 20 <= quantity <= sequence:


User authentication - Epic

access restriction based on user identity, login, logout

This story is split into multiple stories.

This story is split into multiple stories.


Remove graphene

graphene is now in its own repo.
cc @GuillaumeGomez @sdroege

What do you mean with it being in its own repo? Wasn't the goal to have the -sys part here and the other part in gtk-rs/graphene?

@sdroege
@alatiera suggested moving graphene-sys to the graphene repo for easier maintenance, and @GuillaumeGomez agreed. graphene-sys now lives at https://github.com/gtk-rs/graphene/tree/master/graphene-sys.

I didn't agree. I said it was an acceptable solution just like doing nothing and took sourceview as an example. You'll note that sourceview is the only "main" gtk-rs repository which owns its own sys part.

You'll note that sourceview is the only "main" gtk-rs repository which owns its own sys part.

cairo too :)
I don't mind either way. Just keep it in the sys repo for the other 3 crates for consistency with the gtk3 versions.

I am also fine either way, since its somewhat standalone module, I think its easier to have both crates in the same repo. Would be fine otherwise as well if the other way around is preferred.

As that's how the gtk-rs/graphene repo is now, let's get this PR merged I guess!


Pin CoreFX version

I think this is blocking building runtime package store on the rel/2.0.0-preview1 branch on the CI.
\NuGet.targets(97,5): error : Unable to resolve 'System.Runtime.CompilerServices.Unsafe (>= 4.4.0-preview2-25225-02)' for '.NETStandard,Version=v1.3'.

@kichalla is working on fixing this

We removed all the preview2 packages from the aspnetcore-ci-release feed, and instead uploaded all the preview1 packages. We did this probably 1 hour ago. Was this PR attempting to work around that?

@Eilon I just found out that there are some more packages which need to be deleted and updated. The CoreFxLab packages were depending on the delete packages transitively.

Ok let me know what to delete and what to upload.

Cool. @kichalla please give me a heads up when this is fixed.

This is no longer needed


VPN Gateway Packet Capture page is not available on VPN Gateway Resource

Please add details explaining why the "VPN Gateway Packet Capture" option specified in documentation wouldn't be available when viewing a VPN Gateway.
Under the Key Considerations topic, the only that that would seems to prevent this options availability hints that Packet captures aren't supported on policy-based gateways, but in my scenario it's a route-based gateway, but the option is not available.  Please explain why or how to configure Azure to get this option.
Thank you for your time. 🙏

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 85090bbe-facd-f493-9817-b639df44f345
Version Independent ID: 29832cb6-6676-147d-5d5f-fccbd55b2fba
Content: Configure packet capture for VPN Gateway - Azure VPN Gateway
Content Source: articles/vpn-gateway/packet-capture.md
Service: vpn-gateway
GitHub Login: @cherylmc
Microsoft Alias: cherylmc

@S-Berens
Thank you for reaching out.
Can you please let us know which VPN SKU you are using?
Based on the feedback shared here
If you are using the Basic SKU. You can do a packet capture on Basic VPN gateway SKU via PowerShell/CLI.

@S-Berens Thank you for reaching out. Can you please let us know which VPN SKU you are using? Based on the feedback shared here  If you are using the Basic SKU. You can do a packet capture on Basic VPN gateway SKU via PowerShell/CLI.

Generation: Generation1
SKU: Basic

Gateway: VPN
VPN type: Route-based

I can confirm I was able to use PowerShell to Start/Stop Traces.  At the time I was having issues getting the Az modules installed (I still had AzureRM installed and they're not compatible) so I was looking for a shortcut but couldn't find a reason the option wasn't present in the Portal.
Hopefully the UI team will identify the issue and make it available.

@S-Berens
Thank you for sharing the update above and glad to know you were able to start/stop the traces. If it helps you can use Azure Cloud Shell from Azure Portal to run any PowerShell or CLI Commands.
Meanwhile, I will highlight this issue internally to the team. Thank you!

@S-Berens Thank you for reaching out. Can you please let us know which VPN SKU you are using? Based on the feedback shared here  If you are using the Basic SKU. You can do a packet capture on Basic VPN gateway SKU via PowerShell/CLI.

So in which SKU, VPN Gateway packet capture is available in portal of azure?


Configure Business Rules, Workflow and Process Visibility

Tutorials: https://developers.sap.com/tutorials/cp-starter-ibpm-employeeonboarding-2-configure.html
After running the Workflow Management Booster successfully, I cannot get into "Manage Rule Projects".  I get a blank page entitled "Authorization Issue".
291491cftrial

Seems to work now!


update: add Dockerfile, fix PORT env variable name for distinction with FILECOIN_DBS_PORT on DBS side

Fixes #3
Changes proposed in this PR:

Adds a Dockerfile
Change the PORT env variable name to ARWEAVE_DBS_PORT to avoid collisions with other services when using the DBS docker-compose config

For now I am still getting local errors when running the containers together, following error specific to Arweave:
arweave         | API Server is running at http://localhost:8081/
arweave         | /usr/src/app/node_modules/elliptic/lib/elliptic/curve/short.js:415
arweave         |   return this.x.fromRed();
arweave         |   TypeError: Cannot read properties of null (reading 'fromRed')
arweave         |     at Point.getX (/usr/src/app/node_modules/elliptic/lib/elliptic/curve/short.js:4

Any idea ?
Any review and remarks appreciated.

For now I am still getting local errors when running the containers together, following error specific to Arweave:
arweave         | API Server is running at http://localhost:8081/
arweave         | /usr/src/app/node_modules/elliptic/lib/elliptic/curve/short.js:415
arweave         |   return this.x.fromRed();
arweave         |   TypeError: Cannot read properties of null (reading 'fromRed')
arweave         |     at Point.getX (/usr/src/app/node_modules/elliptic/lib/elliptic/curve/short.js:4

Any idea ?

No clue.

@balessan @alexcos20 The tests fail because #6

@balessan @alexcos20 The tests fail because #6

@MantisClone ok, first step of #6 is:

Create two test accounts.

Which test accounts are we talking about ? I mean, on which application ? I guess I have issue with my local version not running because of that too, would help understand the dependency on other solutions, if any :-)

For now I am still getting local errors when running the containers together, following error specific to Arweave:
arweave         | API Server is running at http://localhost:8081/
arweave         | /usr/src/app/node_modules/elliptic/lib/elliptic/curve/short.js:415
arweave         |   return this.x.fromRed();
arweave         |   TypeError: Cannot read properties of null (reading 'fromRed')
arweave         |     at Point.getX (/usr/src/app/node_modules/elliptic/lib/elliptic/curve/short.js:4

Any idea ?

No clue. Does this actually prevent the Arweave DBS from operating?

Yes it does, it just fails on startup and the container stops.

For now I am still getting local errors when running the containers together, following error specific to Arweave:
arweave         | API Server is running at http://localhost:8081/
arweave         | /usr/src/app/node_modules/elliptic/lib/elliptic/curve/short.js:415
arweave         |   return this.x.fromRed();
arweave         |   TypeError: Cannot read properties of null (reading 'fromRed')
arweave         |     at Point.getX (/usr/src/app/node_modules/elliptic/lib/elliptic/curve/short.js:4

Any idea ?

No clue. Does this actually prevent the Arweave DBS from operating?

@MantisClone Yes it does, it just fails on startup and the container stops.

My best guess is that it's because the PRIVATE_KEY env var is not set.
References:

https://github.com/connext/chaindata/issues/1
https://github.com/connext/chaindata/pull/3
https://github.com/web3/web3.js/issues/1527#issuecomment-441684939


Add docker.build_args to game config

Changes

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Current dependencies on/for this PR:

main

PR #303

PR #304   👈

This stack of pull requests is managed by Graphite.


flutter build ios --release. Only works if i run flutter clean first (from successful debug)

doctor
✓] Flutter (Channel master, 1.19.0-2.0.pre.142, on Mac OS X 10.14.5 18F132, locale en-ID)

[!] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    ! Some Android licenses not accepted.  To resolve this, run: flutter doctor --android-licenses
[✓] Xcode - develop for iOS and macOS (Xcode 11.3)
[!] Android Studio (version 3.5)
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
[✓] VS Code (version 1.45.1)
[✓] Connected device (1 available)

So the problem is :

Flutter debug to ios, works fine
When try to build release, it errors like

Error output from Xcode build:
↳
    ** BUILD FAILED

...
Command PhaseScriptExecution failed with a nonzero exit code
    note: Using new build system
    note: Planning build
    note: Constructing build description
...
Encountered error while building for device.

run flutter clean
run flutter build ios --release
works

...
Xcode build done.                                           153.6s
...

Now when I try to debug : error

...
Could not build the application for the simulator.
Error launching application on iPhone 11 Pro Max.
Exited (sigterm)

run flutter clean
debug works

Hi @x4080
Can  you please provide flutter build ios --release with crash?
Thank uoi

@TahaTesser Sorry for late reply, I try again with empty new project and it works as expected, I'll try to copy my lib to this new project and try again

After above, I tried again in my old project release (without flutter clean after flutter debug) and here's the error
The domain/default pair of (/Users/zro/Documents/flutter
    projects/brix/build/ios/Release-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/ios/Flutter/App.framework/Info.plist, CFBundleExecutable) does not
    exist
    fatal error: lipo: can't map input file: /Users/zro/Documents/flutter
    projects/brix/build/ios/Release-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/ios/Flutter/App.framework/ (Invalid argument)
    fatal error: lipo: can't map input file: /Users/zro/Documents/flutter
    projects/brix/build/ios/Release-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/ios/Flutter/App.framework/ (Invalid argument)
    Failed to extract armv7 for /Users/zro/Documents/flutter
    projects/brix/build/ios/Release-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/ios/Flutter/App.framework/. Running lipo -info:
    fatal error: lipo: can't map input file: /Users/zro/Documents/flutter
    projects/brix/build/ios/Release-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/ios/Flutter/App.framework/ (Invalid argument)
    Command PhaseScriptExecution failed with a nonzero exit code
    note: Using new build system
    note: Planning build
    note: Constructing build description

Encountered error while building for device.

Any hints ?

Hi @x4080
This issue related to your plist file
Please check this solution
Thank you

Hi @x4080
Did you manage to solve the issue?
Thank you

@TahaTesser Sorry but I'm away from my computer right now, I'll let you know when I get into it, you can close it first if you like, I'll re open it if I still have problem, btw (havent tried it yet) why flutter clean seems to temporarily solve the issue if the problem is the plist ?

Duplicate of https://github.com/flutter/flutter/issues/57210.


Strip  where the cursor is

As I mentioned here: https://twitter.com/fr32c/status/625519766611083264
I am looking for an extension to get a shortcut for something frequent in my cleaning code work : manually removing useless span tags. (Manually because some of the tags are useful…)
This extension, with the desired keyboard shortcut, would strip tags where the cursor is:
If my text looks like this:

my bld txt

(I put a fake Cursor/pipe in closing span tag)
Then pressing the shortcut would result in this:

my bld txt

To give you a more real world example, I'm regularly using the html/epub export from inDesign, and the result can be:

mytext (AnotherText)

Expected text, with just 2 clicks and 2 times the shortcut:

mytext (AnotherText)

By the way, this selection already exists, since Brackets is highlighting the whole span set when I'm placing the cursor there, i'd love to remove evrything that's "green" (if this can help).

Closing this issue as that's our normal process for extension ideas. Developers looking for an idea will still be able to find it.

Hi, just wanted to say I just published an extension for this called Tag Deletion.

@qtbeee Hey, I just wanted to say i am using your tag deletion extension and its been great. I am wondering if you can improve it by adding the following features. if possible.
let say the line text is as follows "title"
i am wondering if you can do it so that it still select the same line after first deletion because sometimes i just want the "title" and not the p and strong tags.
also if you can do that for multiple cursor selections that would be wonderful.
Cheers, keep up the great work.

@hass0302
I've updated the extension to work for multiple cursor selections. Let me know if you find any problems. I'm planning on making it so the selections stay selected over the weekend.


how to convert local mp3 file to blob in react,

how to convert local mp3 file to blob in react,

Hey @liubobo, this is a question perfectly suited for StackOverflow: http://stackoverflow.com/questions/tagged/react-native
Many people from the community hang out there and will be able to see and answer your question, you can vote on answers and mark the question an answered, run bounties etc. It's a system that works really really well for questions. Keeping questions there lets us keep the list of bug reports and feature requests on github.

Close this for Stack Overflow or  Product Pains.


[BUG][API]Fix format2Duration when parameter has one is null

ThIs Close #4803
When querying a workflow instance, a process instance is running and the query will fail。

当查询工作流实例时，一个流程实例正在运行，查询将会失败
The reason is that the end time is currently empty when calculating the process duration.
原因是因为，计算流程持续时间时，结束时间目前还是空。
So the query failed.

所以导致查询失败。

Codecov Report

Merging #4802 (831ebc6) into dev (68de43a) will decrease coverage by 0.07%.
The diff coverage is 50.00%.

@@             Coverage Diff              @@
##                dev    #4802      +/-   ##
============================================
- Coverage     45.24%   45.17%   -0.08%
+ Complexity     3402     3400       -2
============================================
  Files           538      538
  Lines         23052    23054       +2
  Branches       2686     2686
============================================
- Hits          10431    10414      -17
- Misses        11692    11710      +18
- Partials        929      930       +1

Impacted Files
Coverage Δ
Complexity Δ

...pache/dolphinscheduler/common/utils/DateUtils.java
81.30% <50.00%> (-0.52%)
33.00 <2.00> (+1.00)
:arrow_down:

...olphinscheduler/server/registry/HeartBeatTask.java
37.50% <0.00%> (-46.88%)
2.00% <0.00%> (-1.00%)

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
3.00% <0.00%> (-1.00%)

...inscheduler/service/zk/CuratorZookeeperClient.java
60.97% <0.00%> (-4.88%)
7.00% <0.00%> (-1.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68de43a...831ebc6. Read the comment docs.

hi, let me worry things is if users use this return value (value is  "") calculation，maybe happen error, so, you should check any use this method's method.

hi, let me worry things is if users use this return value (value is "") calculation，maybe happen error, so, you should check any use this method's method.

In three cases, this function is used to calculate the duration of the query.

+1


Expand cellpose options

Closes #649
Closes #401
Checklist before merging

[x] I added an appropriate entry to CHANGELOG.md

I tested this new version of the task on the UZH servers. But the Enum part of Fractal web isn't deployed there yet. And it doesn't solve the actual issue we had with sparse images (see comments in #649 )
=> No immediate hurry on this, I'll go over it again in the coming days hopefully

I added a more complex set of normalization options now.
Remaining todos:

[ ] Figure out why the docstrings for the attributes in my CellposeCustomNormalizer do not get added to the manifest (do I need to specify the model somewhere? It reads the rest of it.
[ ] Test the task in an actual Fractal server & web instance: Is the manifest valid? Does the interface work?
[ ] Can I find a way to constrain the lower_percentile & upper_percentile to be min 0.0 and max 100.0 in the pydantic model? Would that be used correctly in the Fractal manifest afterwards?
[ ] Add tests for the new normalization functionality

Testing in local deployment: The task works in a local Fractal server. But the normalize entry is not shown in the interface. It's supposed to come between Flow-Threshold & Anisotropy. Instead, I get a double line there now

And seeing some validator errors in the console:

Maybe it's an issue with having Optional floats with default value None?

I updated the manifest creation scripts to include my models, but things still show up weirdly in the interface.
I now looked at the manifest to try and see if the issue could be there.
Other external pydantic models are referenced like this:
          "channel": {
            "$ref": "#/definitions/ChannelInputModel",
            "title": "Channel",
            "description": "Primary channel for segmentation; requires either `wavelength_id` (e.g. `A01_C01`) or `label` (e.g. `DAPI`)."
          },

But my new entry has a setup like that:
          "normalize": {
            "title": "Normalize",
            "default": {
              "default_normalize": true,
              "lower_percentile": null,
              "upper_percentile": null,
              "lower_bound": null,
              "upper_bound": null
            },
            "allOf": [
              {
                "$ref": "#/definitions/CellposeCustomNormalizer"
              }
            ],
            "description": "By default, data is normalized so 0.0=1st percentile and 1.0=99th percentile of image intensities in each channel. This automatic normalization can lead to issues when the image to be segmented is very sparse. You can turn off the automated rescaling and either provide your own rescaling percentiles or fixed rescaling upper and lower bound integers"
          },

I can see this making sense. I don't fully understand the "allOf" references, but maybe that's how this is supposed to be specified?
But I still get the following validator error in the console:

@tcompa Do you have any idea how to debug this to figure out why it fails?

As of https://github.com/fractal-analytics-platform/fractal-tasks-core/pull/650/commits/a9185ba9cbda4529260408015eb7da30d55890b8, this is the new rendering:

My first guess is that creating an instance of the normalizer and using it as default for the argument cannot be transformed into a nice default within the JSON schema. In general, it'd be better to say normalize: Optional[CellposeCustomNormalizer] = None, and then creating a default instances from within the task. I haven't tested this yet, I'll dig deeper later.

Ah, that makes sense! And it gets a default object (with default_normalize = True) in your screenshot above, so that's the behavior I was trying to ensure with setting a model. Thanks for that fix!
I'll test a few more things with it now! :)

Ok, it works as expected for me now in the interface as well! :) Thanks @tcompa !
On the remaining list:

[x] Test the task in an actual Fractal server & web instance: Is the manifest valid? Does the interface work? => Done
[x]  Can I find a way to constrain the lower_percentile & upper_percentile to be min 0.0 and max 100.0 in the pydantic model? Would that be used correctly in the Fractal manifest afterwards? => used Pydantic Field constraints, mostly works, will report a minor issue on the Fractal web side later
[ ]  Add tests for the new normalization functionality => WIP

I now added the tests for the new functionality. They can always be improved further of course, but should be a decent basis to start off from.
I have added the 0.14.2a2 version to the Pelkmans lab cluster and will run some tests on real-life data with some edge cases next.

Please don't merge this PR yet, because there is still a glitch on the JSON-schema side.
You can already test the current version, as we'll just have to provide a better way of reaching the same functionality.

Oh, what's the glitch?

The following JSON schema (a simplified version of the current one)
{
  "title": "CellposeSegmentation",
  "type": "object",
  "properties": {
    "channel": {
      "$ref": "#/definitions/ChannelInputModel",
      "title": "Channel"
    },
    "channel2": {
      "$ref": "#/definitions/ChannelInputModel",
      "title": "Channel2"
    },
    "normalize": {
      "$ref": "#/definitions/CellposeCustomNormalizer",
      "title": "Normalize"
    }
  },
  "required": [
    "channel"
  ],
  "additionalProperties": false,
  "definitions": {
    "ChannelInputModel": {
      "title": "ChannelInputModel",
      "type": "object",
      "properties": {
        "wavelength_id": {
          "title": "Wavelength Id",
          "type": "string"
        },
        "label": {
          "title": "Label",
          "type": "string"
        }
      }
    },
    "CellposeCustomNormalizer": {
      "title": "CellposeCustomNormalizer",
      "type": "object",
      "properties": {
        "default_normalize": {
          "title": "Default Normalize",
          "default": true,
          "type": "boolean"
        },
        "lower_percentile": {
          "title": "Lower Percentile",
          "type": "number"
        }
      }
    }
  }
}

is rendered as follows

Note the difference:

ChannelInputModel has no default values, and then it's not present in the formData panel (unless we explicitly set some of the channel or channel2 attributes);
CellposeCustomNormalizer has the true default for default_normalize, and then it's always included in the formData.

The consequence of this is that when using fractal-web, we can never run the task with normalize=None. In other words, we never enter this if branch
    # Set default normalizer
    if normalize is None:
        normalize = CellposeCustomNormalizer(default_normalize=True)

which is problematic for at least two reasons:

It's highly counterintuitive, since the call signature says normalize: Optional[CellposeCustomNormalizer] = None
It may be plain wrong, in case we slightly modify the content of that if branch, e.g. to change some default.

One solution is readily available, though.
As far as we understand (debugged with @mfranzon), the CellposeCustomNormalizer.default_normalize attribute is redundant. If any of the other four values is set, this is automatically false. But there is no use case where we may want to simultaneously set CellposeCustomNormalizer.default_normalize and e.g. CellposeCustomNormalizer.lower_bound (and that's why the validator has to check this condition, and raise an error if appropriate).
This also suggests a solution: default_normalize should not be an attribute, but rather a property. Something like:
class CellposeCustomNormalizer(BaseModel):
    lower_percentile: Optional[float] = Field(None, ge=0, le=100)
    upper_percentile: Optional[float] = Field(None, ge=0, le=100)
    lower_bound: Optional[int] = None
    upper_bound: Optional[int] = None

    @property
    def default_normalize(self) -> bool:
        if self.lower_percentile is not None:
            return False
        elif self.lower_bound is not None:
            return False
        elif self.upper_percentile is not None:
            return False
        elif self.upper_bound is not None:
            return False
        else:
            return True

I think this will work, but I've not yet tested it.

Higher-level comments:

Once again, we realize that mapping Python call signatures to JSON schemas has plenty of interesting edge cases;
If we want to keep the JSON Schemas "clean" (for any definition of "clean"), we can push a lot of the "custom" logic to places which are not then converted in JSON schemas, e.g. validators and properties.

The consequence of this is that when using fractal-web, we can never run the task with normalize=None

I can see how this is a general problem. For this task, that's actually fine.

As far as we understand (debugged with @mfranzon), the CellposeCustomNormalizer.default_normalize attribute is redundant. If any of the other four values is set, this is automatically false. But there is no use case where we may want to simultaneously set CellposeCustomNormalizer.default_normalize and e.g. CellposeCustomNormalizer.lower_bound (and that's why the validator has to check this condition, and raise an error if appropriate).

Unfortunately, there is one case:
One can set CellposeCustomNormalizer.default_normalize=False and set all the other elements to None. That is supported and has a specific behavior in Cellpose (though not a very typically useful one in my tests)

As of our call:
There are mainly four branches to be covered:

Do not apply any normalization at all;
Apply the default CP normalization;
Apply a user-provided bound-based normalization;
Apply a user-provided percentiles-based normalization.

And the expected default behavior, when there is no user input, is case number 2 (apply default CP normalization).

Here is a naive, untested, example:
from typing import Optional, Literal
from pydantic import BaseModel, Field

class CellposeCustomNormalizer(BaseModel):
    type: Optional[Literal["custom", "no_normalization"]] = None
    lower_percentile: Optional[float] = Field(None, ge=0, le=100)
    upper_percentile: Optional[float] = Field(None, ge=0, le=100)
    lower_bound: Optional[int] = None
    upper_bound: Optional[int] = None

    @property
    def apply_normalization(self) -> bool:
        # returns False when we do not want to apply any normalization
        # returns True when _some_ normalization has to be applied (CP-default, bounds, or percentiles)
        return self.type != "no_normalization"

    def validator_TBD(self):
        # This should enforce some relevant constraints, like:
        # if type=None, then no other attribute can be set
        # if type="no_normalization", then no other attributes can be set
        # if type="custom", then either the bounds or the percentiles attributes must be set, but not both
        pass

def task(
    ...
    normalize: Optional[CellposeCustomNormalizer] = None,
    ...):

In this version, we'd have

Do not apply any normalization at all; -> type="no_normalization"
Apply the default CP normalization; -> type=None (i.e. no input)
Apply a user-provided bound-based normalization; -> type="custom"
Apply a user-provided percentiles-based normalization. -> type="custom"

Something I'm not happy about is that I think that if you modify the default once you may not be able to get back to the original situation. That is, I don't think that Optional[Literal] translates into an enum with four options (None included). I'll iterate on this later.
This preliminary example is in the direction of coming up with a model with no default attributes - which would then "fix" the JSON-schema glitch.
The other option is to actually provide a default in the function call signature, and make sure that it's transformed into a JSON Schema that is easily understood from fractal-web.
A third option is to extend fractal-web to handle this "new" kind of JSON schema.

Very interesting thinking & the direction looks promising, thanks Tommaso!
The other thing to consider: We want to make sure users understand what they are running by default. e.g. "if I don't modify anything here, it will apply default Cellpose normalization"

I have a better-defined proposal (based on the previous one) which consists in using the model and call signature below. I'll document in two other comments the Python behavior and the JSON-Schema behavior.
Call-signature (simplified) and task function
@validate_arguments
def cellpose_segmentation(
    *,
    # Fractal arguments
    channel: ChannelInputModel,
    channel2: Optional[ChannelInputModel] = None,
    normalize: Optional[CellposeCustomNormalizer] = None,
    normalizeV2: Optional[CellposeCustomNormalizerV2] = None,
) -> dict[str, Any]:

    if normalizeV2 is None:
        normalizeV2 = CellposeCustomNormalizerV2()

Model for argument

class CellposeCustomNormalizerV2(BaseModel):
    """
    Validator to handle different normalization scenarios for Cellpose models

    If `type` is unset or set to `default`, Cellpose default normalization is
    used and no other parameters can be specified.
    If `type` is set to `no_normalization`, no normalization is used and no
    other parameters can be specified.
    If `type` is set to `custom`, either percentiles or explicit integer
    bounds can be applied.

    Attributes:
        type:
            One of `default` (Cellpose default normalization), `custom`
            (using the other custom parameters) or `no_normalization`.
        lower_percentile: Specify a custom lower-bound percentile for rescaling
            as a float value between 0 and 100. Set to 1 to run the same as
            default). You can only specify percentiles or bounds, not both.
        upper_percentile: Specify a custom upper-bound percentile for rescaling
            as a float value between 0 and 100. Set to 99 to run the same as
            default, set to e.g. 99.99 if the default rescaling was too harsh.
            You can only specify percentiles or bounds, not both.
        lower_bound: Explicit lower bound value to rescale the image at.
            Needs to be an integer, e.g. 100.
            You can only specify percentiles or bounds, not both.
        upper_bound: Explicit upper bound value to rescale the image at.
            Needs to be an integer, e.g. 2000.
            You can only specify percentiles or bounds, not both.
    """
    type: Optional[Literal["default", "custom", "no_normalization"]] = None
    lower_percentile: Optional[float] = Field(None, ge=0, le=100)
    upper_percentile: Optional[float] = Field(None, ge=0, le=100)
    lower_bound: Optional[int] = None
    upper_bound: Optional[int] = None

    @root_validator
    def validate_conditions(cls, values):

        # Replace type=None with type="default"
        type = values.get("type")
        if type is None:
            type = "default"
            values["type"] = type

        # Extract values of custom parameters
        lower_percentile = values.get("lower_percentile")
        upper_percentile = values.get("upper_percentile")
        lower_bound = values.get("lower_bound")
        upper_bound = values.get("upper_bound")

        # Verify that custom parameters are only provided when type="custom"
        if type != "custom":
            if lower_percentile is not None:
                raise ValueError(
                    f"Type='{type}' but {lower_percentile=}. "
                    "Hint: set type='custom'."
                    )
            if upper_percentile is not None:
                raise ValueError(
                    f"Type='{type}' but {upper_percentile=}. "
                    "Hint: set type='custom'."
                    )
            if lower_bound is not None:
                raise ValueError(
                    f"Type='{type}' but {lower_bound=}. "
                    "Hint: set type='custom'."
                )
            if upper_bound is not None:
                raise ValueError(
                    f"Type='{type}' but {upper_bound=}. "
                    "Hint: set type='custom'."
                )

        # The only valid options are:
        # 1. Both percentiles are set and both bounds are unset
        # 2. Both bounds are set and both percentiles are unset
        are_percentiles_set = (
            lower_percentile is not None,
            upper_percentile is not None,
            )
        are_bounds_set = (
            lower_bound is not None,
            upper_bound is not None,
            )
        if len(set(are_percentiles_set)) != 1:
            raise ValueError(
                "Both lower_percentile and upper_percentile must be set "
                "together."
                )
        if len(set(are_bounds_set)) != 1:
            raise ValueError(
                "Both lower_bound and upper_bound must be set together"
                )
        if lower_percentile is not None and lower_bound is not None:
            raise ValueError(
                "You cannot set both explicit bounds and percentile bounds "
                "at the same time. Hint: use only one of the two options."
            )

        return values

Pydantic-model behavior:
>>> from fractal_tasks_core.tasks.cellpose_transforms import CellposeCustomNormalizerV2

>>> CellposeCustomNormalizerV2()
CellposeCustomNormalizerV2(type='default', lower_percentile=None, upper_percentile=None, lower_bound=None, upper_bound=None)

>>> CellposeCustomNormalizerV2(type="default")
CellposeCustomNormalizerV2(type='default', lower_percentile=None, upper_percentile=None, lower_bound=None, upper_bound=None)

>>> CellposeCustomNormalizerV2(lower_bound=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for CellposeCustomNormalizerV2
__root__
  Type='default' but lower_bound=1. Hint: set type='custom'. (type=value_error)

>>> CellposeCustomNormalizerV2(type="default", lower_bound=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for CellposeCustomNormalizerV2
__root__
  Type='default' but lower_bound=1. Hint: set type='custom'. (type=value_error)

>>> CellposeCustomNormalizerV2(type="no_normalization")
CellposeCustomNormalizerV2(type='no_normalization', lower_percentile=None, upper_percentile=None, lower_bound=None, upper_bound=None)

>>> CellposeCustomNormalizerV2(type="no_normalization", lower_bound=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for CellposeCustomNormalizerV2
__root__
  Type='no_normalization' but lower_bound=1. Hint: set type='custom'. (type=value_error)

>>> CellposeCustomNormalizerV2(type="custom")
CellposeCustomNormalizerV2(type='custom', lower_percentile=None, upper_percentile=None, lower_bound=None, upper_bound=None)

>>> CellposeCustomNormalizerV2(type="custom", lower_percentile=2, upper_percentile=90)
CellposeCustomNormalizerV2(type='custom', lower_percentile=2.0, upper_percentile=90.0, lower_bound=None, upper_bound=None)

>>> CellposeCustomNormalizerV2(type="custom", lower_bound=1, upper_bound=10)
CellposeCustomNormalizerV2(type='custom', lower_percentile=None, upper_percentile=None, lower_bound=1, upper_bound=10)

>>> CellposeCustomNormalizerV2(type="custom", lower_percentile=2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for CellposeCustomNormalizerV2
__root__
  Both lower_percentile and upper_percentile must be set together. (type=value_error)

>>> CellposeCustomNormalizerV2(type="custom", lower_bound=2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for CellposeCustomNormalizerV2
__root__
  Both lower_bound and upper_bound must be set together (type=value_error)

>>> CellposeCustomNormalizerV2(type="custom", lower_percentile=2, upper_percentile=90, lower_bound=2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for CellposeCustomNormalizerV2
__root__
  Both lower_bound and upper_bound must be set together (type=value_error)

JSON Schema (simplified call signature):
{
  "title": "CellposeSegmentation",
  "type": "object",
  "properties": {
    "channel": {
      "$ref": "#/definitions/ChannelInputModel",
      "title": "Channel",
      "description": "Primary channel for segmentation; requires either `wavelength_id` (e.g. `A01_C01`) or `label` (e.g. `DAPI`)."
    },
    "channel2": {
      "$ref": "#/definitions/ChannelInputModel",
      "title": "Channel2",
      "description": "Second channel for segmentation (in the same format as `channel`). If specified, cellpose runs in dual channel mode. For dual channel segmentation of cells, the first channel should contain the membrane marker, the second channel should contain the nuclear marker."
    },
    "normalize": {
      "$ref": "#/definitions/CellposeCustomNormalizer",
      "title": "Normalize",
      "description": "By default, data is normalized so 0.0=1st percentile and 1.0=99th percentile of image intensities in each channel. This automatic normalization can lead to issues when the image to be segmented is very sparse. You can turn off the automated rescaling and either provide your own rescaling percentiles or fixed rescaling upper and lower bound integers"
    },
    "normalizeV2": {
      "$ref": "#/definitions/CellposeCustomNormalizerV2",
      "title": "Normalizev2",
      "description": "By default, data is normalized so 0.0=1st percentile and"
    }
  },
  "required": [
    "channel"
  ],
  "additionalProperties": false,
  "definitions": {
    "ChannelInputModel": {
      "title": "ChannelInputModel",
      "description": "A channel which is specified by either `wavelength_id` or `label`.",
      "type": "object",
      "properties": {
        "wavelength_id": {
          "title": "Wavelength Id",
          "type": "string",
          "description": "Unique ID for the channel wavelength, e.g. `A01_C01`."
        },
        "label": {
          "title": "Label",
          "type": "string",
          "description": "Name of the channel."
        }
      }
    },
    "CellposeCustomNormalizer": {
      "title": "CellposeCustomNormalizer",
      "description": "Validator to handle different normalization scenarios for Cellpose models",
      "type": "object",
      "properties": {
        "default_normalize": {
          "title": "Default Normalize",
          "default": true,
          "type": "boolean",
          "description": "Whether to use the default Cellpose normalization approach (rescaling the image between the 1st and 99th percentile)"
        },
        "lower_percentile": {
          "title": "Lower Percentile",
          "type": "number",
          "description": "Specify a custom lower-bound percentile for rescaling as a float value between 0 and 100. Set to 1 to run the same as default). You can only specify percentils or bounds, not both."
        },
        "upper_percentile": {
          "title": "Upper Percentile",
          "type": "number",
          "description": "Specify a custom upper-bound percentile for rescaling as a float value between 0 and 100. Set to 99 to run the same as default, set to e.g. 99.99 if the default rescaling was too harsh. You can only specify percentils or bounds, not both."
        },
        "lower_bound": {
          "title": "Lower Bound",
          "type": "integer",
          "description": "Explicit lower bound value to rescale the image at. Needs to be an integer, e.g. 100. You can only specify percentils or bounds, not both."
        },
        "upper_bound": {
          "title": "Upper Bound",
          "type": "integer",
          "description": "Explicit upper bound value to rescale the image at. Needs to be an integer, e.g. 2000 You can only specify percentils or bounds, not both."
        }
      }
    },
    "CellposeCustomNormalizerV2": {
      "title": "CellposeCustomNormalizerV2",
      "description": "Validator to handle different normalization scenarios for Cellpose models",
      "type": "object",
      "properties": {
        "type": {
          "title": "Type",
          "enum": [
            "default",
            "custom",
            "no_normalization"
          ],
          "type": "string",
          "description": "One of `default` (Cellpose default normalization), `custom` (using the other custom parameters) or `no_normalization`."
        },
        "lower_percentile": {
          "title": "Lower Percentile",
          "minimum": 0,
          "maximum": 100,
          "type": "number",
          "description": "Specify a custom lower-bound percentile for rescaling as a float value between 0 and 100. Set to 1 to run the same as default). You can only specify percentiles or bounds, not both."
        },
        "upper_percentile": {
          "title": "Upper Percentile",
          "minimum": 0,
          "maximum": 100,
          "type": "number",
          "description": "Specify a custom upper-bound percentile for rescaling as a float value between 0 and 100. Set to 99 to run the same as default, set to e.g. 99.99 if the default rescaling was too harsh. You can only specify percentiles or bounds, not both."
        },
        "lower_bound": {
          "title": "Lower Bound",
          "type": "integer",
          "description": "Explicit lower bound value to rescale the image at. Needs to be an integer, e.g. 100. You can only specify percentiles or bounds, not both."
        },
        "upper_bound": {
          "title": "Upper Bound",
          "type": "integer",
          "description": "Explicit upper bound value to rescale the image at. Needs to be an integer, e.g. 2000. You can only specify percentiles or bounds, not both."
        }
      }
    }
  }
}

Behavior in fractal-web sandbox

Behavior in react-jsonschema-form

Something I'm not happy about is that I think that if you modify the default once then you may not be able to get back to the original situation. That is, I don't think that Optional[Literal] translates into an enum with three options (None, custom and no_normalization). I'll iterate on this later.

This is still present, but mitigated by the fact that now there is "default" entry in the enum, and then one could easily decide to use that one and provide a non-null value for normalize={"type": "default"}. In my view that's a good-enough trade-off.

@jluethi I'll let you evaluate if this option fits with the expectation. If so, in principle you should just replace the current Pydantic model with the new one, and apply minor changes to the parts of the task where this object is used.

Ref for the fact that Pydantic uses allOf:

https://github.com/pydantic/pydantic/issues/2592

I like the logic of the new Enum with the 3 options. And the checks look good.
=> I like the new logic, thanks for revisiting it!
One detail that I'm not 100% happy with yet:

This is still present, but mitigated by the fact that now there is "default" entry in the enum, and then one could easily decide to use that one and provide a non-null value for normalize={"type": "default"}. In my view that's a good-enough trade-off.

The thing that confuses me somewhat: We now avoid setting defaults twice:

The parameter of the Cellpose model doesn't have a default, but we set it in an if normalizeV2 is None: normalizeV2 = CellposeCustomNormalizerV2() check
The type doesn't have a default, and we set it the same way (in such an if check).

Now this achieves the desired behavior of running default normalization if the user doesn't set anything. But it's even less transparent than the one before (where we also discussed that the first default = None => if check in the function) was suboptimal.
If such a setup is necessary to do this without revisiting the whole pydantic schemas for this, I'm ok with that and find it an acceptable trade-off.
But do we gain anything from
type: Optional[Literal["default", "custom", "no_normalization"]] = None

vs.
type: Literal["default", "custom", "no_normalization"] = "default"

?
Because it worked fine (as far as I could judge) with
default_normalize: bool = True

And that has the benefit that the user interface displays what default is being used in the web. Is there a reason that type couldn't be a literal with a default (instead of an optional literal)?

Also, neat new validation approaches!

Briefly

And that has the benefit that the user interface displays what default is being used in the web. Is there a reason that type couldn't be a literal with a default (instead of an optional literal)?

I fully agree, the best (-> most transparent) option is
type: Literal["default", "custom", "no_normalization"] = "default"

My previous understanding, however, was that this immediately leads to the anyOf issue as in your first version (ref https://github.com/pydantic/pydantic/issues/2592), and then it was easier to avoid it. I'm now trying to verify this assumption, and it turns out it's likely wrong. More on this shortly. If it's true that we hit no anyOf issue, then I fully agree with switching to the more explicit version.

Provisional information: the allOf issue only shows up in the presence of the default:
from pydantic.decorator import ValidatedFunction
from typing import Literal
from devtools import debug
from fractal_tasks_core.dev.lib_args_schemas import _validate_function_signature
from fractal_tasks_core.dev.lib_args_schemas import _remove_args_kwargs_properties
from fractal_tasks_core.dev.lib_args_schemas import _remove_pydantic_internals
from typing import Literal, Optional

from pydantic import BaseModel

class Normalizer(BaseModel):
    type: Literal["A", "B"] = "A"

def task_function_1(normalizer: Normalizer):
    pass

def task_function_2(normalizer: Optional[Normalizer] = None):
    pass

def task_function_3(normalizer: Normalizer = Normalizer()):
    pass

# GENERATE JSON SCHEMA
for task_function in [task_function_1, task_function_2, task_function_3]:
    _validate_function_signature(task_function)
    vf = ValidatedFunction(task_function, config=None)
    schema = vf.model.schema()
    schema = _remove_args_kwargs_properties(schema)
    schema = _remove_pydantic_internals(schema)
    debug(schema)

Output:
    schema: {
        'title': 'TaskFunction1',
        'type': 'object',
        'properties': {
            'normalizer': {
                '$ref': '#/definitions/Normalizer',    # <--------- this is OK
            },
        },
        'required': [
            'normalizer',
        ],
        'additionalProperties': False,
        'definitions': {
            'Normalizer': {
                'title': 'Normalizer',
                'type': 'object',
                'properties': {
                    'type': {
                        'title': 'Type',
                        'default': 'A',
                        'enum': ['A', 'B'],
                        'type': 'string',
                    },
                },
            },
        },
    } (dict) len=6

    schema: {
        'title': 'TaskFunction2',
        'type': 'object',
        'properties': {
            'normalizer': {
                '$ref': '#/definitions/Normalizer',    # <--------- this is OK
            },
        },
        'additionalProperties': False,
        'definitions': {
            'Normalizer': {
                'title': 'Normalizer',
                'type': 'object',
                'properties': {
                    'type': {
                        'title': 'Type',
                        'default': 'A',
                        'enum': ['A', 'B'],
                        'type': 'string',
                    },
                },
            },
        },
    } (dict) len=5

    schema: {
        'title': 'TaskFunction3',
        'type': 'object',
        'properties': {
            'normalizer': {
                'title': 'Normalizer',
                'default': {
                    'type': 'A',
                },
                'allOf': [
                    {
                        '$ref': '#/definitions/Normalizer',      # <--------- this is NOT OK
                    },
                ],
            },
        },
        'additionalProperties': False,
        'definitions': {
            'Normalizer': {
                'title': 'Normalizer',
                'type': 'object',
                'properties': {
                    'type': {
                        'title': 'Type',
                        'default': 'A',
                        'enum': ['A', 'B'],
                        'type': 'string',
                    },
                },
            },
        },
    } (dict) len=5

Option 3 from https://github.com/fractal-analytics-platform/fractal-tasks-core/pull/650#issuecomment-1935965918 is now also supported, as of fractal-web 0.9.2 (thanks @zonia3000!), therefore let's go with this one right away, since it's the "best" option (i.e. it doesn't have any known unintended effect - especially re: possible unexpected behaviors when the argument isn't provided).

The mocked version is then
from pydantic.decorator import ValidatedFunction
import json
from typing import Literal
from devtools import debug
from fractal_tasks_core.dev.lib_args_schemas import _validate_function_signature
from fractal_tasks_core.dev.lib_args_schemas import _remove_args_kwargs_properties
from fractal_tasks_core.dev.lib_args_schemas import _remove_pydantic_internals
from typing import Literal, Optional

from pydantic import BaseModel

class Normalizer(BaseModel):
    type: Literal["default", "custom"] = "default"

def task_function_3(normalizer: Normalizer = Normalizer()):
    pass

# GENERATE JSON SCHEMA
_validate_function_signature(task_function_3)
vf = ValidatedFunction(task_function_3, config=None)
schema = vf.model.schema()
schema = _remove_args_kwargs_properties(schema)
schema = _remove_pydantic_internals(schema)
print(json.dumps(schema, indent=2))

producing this schema
{
  "title": "TaskFunction3",
  "type": "object",
  "properties": {
    "normalizer": {
      "title": "Normalizer",
      "default": {
        "type": "default"
      },
      "allOf": [
        {
          "$ref": "#/definitions/Normalizer"
        }
      ]
    }
  },
  "additionalProperties": false,
  "definitions": {
    "Normalizer": {
      "title": "Normalizer",
      "type": "object",
      "properties": {
        "type": {
          "title": "Type",
          "default": "default",
          "enum": [
            "default",
            "custom"
          ],
          "type": "string"
        }
      }
    }
  }
}

and this UI in fractal-web 0.9.2:

While the actual model (untested!) should look like

class CellposeCustomNormalizerV2(BaseModel):
    """
    Validator to handle different normalization scenarios for Cellpose models

    If `type="default"`, then Cellpose default normalization is
    used and no other parameters can be specified.
    If `type="no_normalization"`, then no normalization is used and no
    other parameters can be specified.
    If `type="custom"`, then either percentiles or explicit integer
    bounds can be applied.

    Attributes:
        type:
            One of `default` (Cellpose default normalization), `custom`
            (using the other custom parameters) or `no_normalization`.
        lower_percentile: Specify a custom lower-bound percentile for rescaling
            as a float value between 0 and 100. Set to 1 to run the same as
            default). You can only specify percentiles or bounds, not both.
        upper_percentile: Specify a custom upper-bound percentile for rescaling
            as a float value between 0 and 100. Set to 99 to run the same as
            default, set to e.g. 99.99 if the default rescaling was too harsh.
            You can only specify percentiles or bounds, not both.
        lower_bound: Explicit lower bound value to rescale the image at.
            Needs to be an integer, e.g. 100.
            You can only specify percentiles or bounds, not both.
        upper_bound: Explicit upper bound value to rescale the image at.
            Needs to be an integer, e.g. 2000.
            You can only specify percentiles or bounds, not both.
    """
    type: Optional[Literal["default", "custom", "no_normalization"]] = None
    lower_percentile: Optional[float] = Field(None, ge=0, le=100)
    upper_percentile: Optional[float] = Field(None, ge=0, le=100)
    lower_bound: Optional[int] = None
    upper_bound: Optional[int] = None

    @root_validator
    def validate_conditions(cls, values):

        # Extract values
        type = values.get("type")
        lower_percentile = values.get("lower_percentile")
        upper_percentile = values.get("upper_percentile")
        lower_bound = values.get("lower_bound")
        upper_bound = values.get("upper_bound")

        # Verify that custom parameters are only provided when type="custom"
        if type != "custom":
            if lower_percentile is not None:
                raise ValueError(
                    f"Type='{type}' but {lower_percentile=}. "
                    "Hint: set type='custom'."
                    )
            if upper_percentile is not None:
                raise ValueError(
                    f"Type='{type}' but {upper_percentile=}. "
                    "Hint: set type='custom'."
                    )
            if lower_bound is not None:
                raise ValueError(
                    f"Type='{type}' but {lower_bound=}. "
                    "Hint: set type='custom'."
                )
            if upper_bound is not None:
                raise ValueError(
                    f"Type='{type}' but {upper_bound=}. "
                    "Hint: set type='custom'."
                )

        # The only valid options are:
        # 1. Both percentiles are set and both bounds are unset
        # 2. Both bounds are set and both percentiles are unset
        are_percentiles_set = (
            lower_percentile is not None,
            upper_percentile is not None,
            )
        are_bounds_set = (
            lower_bound is not None,
            upper_bound is not None,
            )
        if len(set(are_percentiles_set)) != 1:
            raise ValueError(
                "Both lower_percentile and upper_percentile must be set "
                "together."
                )
        if len(set(are_bounds_set)) != 1:
            raise ValueError(
                "Both lower_bound and upper_bound must be set together"
                )
        if lower_percentile is not None and lower_bound is not None:
            raise ValueError(
                "You cannot set both explicit bounds and percentile bounds "
                "at the same time. Hint: use only one of the two options."
            )

        return values

This PR should be ready now. I switched to using Option 3 described above: Using the CellposeCustomNormalizer() as default input for the argument.
Therefore, this version of the tasks will require Fractal web 0.9.2 (because of the allOf in the json schema).


feat: burrito

Current dependencies on/for this PR:

main

PR #144   👈

This comment was auto-generated by Graphite.


[SCOOP] (Enter your description here) remove permanent data not working in WingetUI 2.0.0 beta 2

Hello,
Does the latest beta also have this issue? https://github.com/marticliment/WingetUI/releases/tag/2.0.0-beta3


InfoBar: Some colors don't update when the theme is changed

Describe the bug
When InfoBar is inside a popup control (e.g. Flyout), some colors don't update when the theme changes from Dark to Light.
Steps to reproduce the bug

Create a new project (Blank app) in UWP for Windows 10 (not tested in Windows 11).
Add WinUI to the project.
On MainPage.xaml, replace the Grid element with this:

    <StackPanel x:Name="Root"
                Background="{ThemeResource ApplicationPageBackgroundThemeBrush}"
                Padding="16"
                RequestedTheme="Dark"
                Spacing="16">

        <Button x:Name="ThemeButton"
                Click="ThemeButton_Click"
                Content="Toggle theme" />

        <Button Content="Open info bar">
            <Button.Flyout>
                <Flyout>
                    <Grid x:Name="FlyoutRoot">
                        <muxc:InfoBar IsOpen="True"
                                      Severity="Error"
                                      Title="Title" />
                    </Grid>
                </Flyout>
            </Button.Flyout>
        </Button>

    </StackPanel>

On MainPage.xaml.cs, add the following event handler:

private void ThemeButton_Click(object sender, RoutedEventArgs e)
{
    Root.RequestedTheme = Root.RequestedTheme == ElementTheme.Light ? ElementTheme.Dark : ElementTheme.Light;
}

Start debugging.
Click on the button with the label "Open info bar".
Close the flyout.
Click on the button with the label "Toggle theme".
Click on the button with the label "Open info bar".
Notice how the background and the icon colors haven't changed.

Expected behavior
All colors are updated whenever the theme changes no matter if the InfoBar control is part of a popup or not.
Screenshots

NuGet package version
Microsoft.UI.Xaml 2.8.0
Windows app type

[ ] UWP
[ ] Win32

Device form factor
Desktop
Windows version
Windows 10 (20H2): Build 19042
Additional context
No response

Is there any estimated date for this bug to be fixed? I appreciate the update. Thank you.


Don't call it a sieve

Apologies, but I'd like to head off more posts like this: https://news.ycombinator.com/item?id=9541478
I hate bikeshedding nomenclature but this could prevent some objections.

looks good to me. see also #87 and #46 for other discussion on this code. I wonder if there's a variant of wadler's law for language homepages...


PositionwiseFeedForward operation order

Hello, thank you for providing this repository.
I was wondering if in the following code the activation function should be applied to x_fc1?
return self.fully_connected2(self.activation(x_fc1)) instead of return self.activation(self.fully_connected2(x_fc1))
original code lin 326:
https://github.com/goruck/nilm/blob/main/ml/transformer_model.py
class PositionwiseFeedForward(tf.keras.layers.Layer):
    """PositionwiseFeedForward layer.

    This is an implementation of position wise feed-forward as described in the
    paper "Attention is all you Need" (Vaswani et al., 2017)."""

    def __init__(self, d_model, d_ff, **kwargs):
        super().__init__(**kwargs)
        self.fully_connected1 = tf.keras.layers.Dense(d_ff)  # First fully connected layer
        self.fully_connected2 = tf.keras.layers.Dense(d_model)  # Second fully connected layer
        self.activation = GELU()

    def call(self, inputs, *args, **kwargs):
        # The input is passed into the two fully-connected layers, with GELU activation.
        x_fc1 = self.fully_connected1(inputs)

        return self.activation(self.fully_connected2(x_fc1))

@benghl yes you are right, the order should be reversed. I changed it to return self.fully_connected2(self.activation((x_fc1))).
Separately but related, the Attention is all you need paper calls for ReLU activation in between the dense layers. I used GeLU here which seemed to get better results but will try ReLU to see how well the model works with the corrected feed forward layer. Thank you for calling this out.


[ArcAppliance] Add CLI version 0.1.48 SCVMM bugs

This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your PR is merged into master branch, a new PR will be created to update src/index.json automatically.
The precondition is to put your code inside this repo and upgrade the version in the PR but do not modify src/index.json.

Feel free to merge when you get the chance.

ArcAppliance


Provide kmz or kml export of the walk

export the kmz file

Looks like that kmz is simple a zipped version of a kml file. Here is some info about kml (Keyhole Markup Language): https://en.wikipedia.org/wiki/Keyhole_Markup_Language

The file is accessible at https://treewalks.github.io/locations.kml


Qingwei/fixes

Coverage remained the same at 31.734% when pulling febbca48dba8ba5991023dd567f0a4dc1a4cb4a5 on qingweibinary:qingwei/fixes into d4bc74b8f5b2ab91e459520981b701556b350813 on binary-com:master.