Externalize registry backends

Fabio is heavily reliant on Consul for service registration, discovery and heartbeat checking. This is a PULL situation. It would be nice if Fabio could complete it's RESTful API as a generic registry interface that could provide a PUSH function and remove any internal dependencies.
This is outside of API request in the Issues list of Fabio.
How would this work?
Let's go to the PowerPoint:

An external registry (whatever it might be) would call the fabio api and register a service. The request would contain all the useful information for the "Add route for service svc from src to dst and assign tags" command but in json format. Included in the json request would be an additional callback attribute string to either a URL or WS. For example:
"serviceCheck":"http://<IP_ADDRESS>:80/is_service_up/19382493"
or
"serviceCheck":"ws://<IP_ADDRESS>:1234/19382493"
or generically
"serviceCheck":"http://<IP_ADDRESS>:80/health"

Fabio doesn't care too much about the value.

Fabio would register this request in it's table.
Fabio would ws attach or use the serviceCheck URL to periodically see if the registry or service is still alive. (in case the registry itself went down or the registry failed in it's attempt to issue an unregister call to Fabio)
An external registry would call fabio occasionally to deregister a service that was no longer available.  The request would contain all the useful information for the route del    command.
Fabio would deregister this service from the Table.

"route weight" could also be provided as apart of the API.
This would provide a means for any external registry to utilize Fabio instead of having a dependency on only Consul or having to compile in another pkg with the type Backend interface to support an interface. Services themselves might even be able to register to Fabio directly without middleware as long as they met the API requirement.

I can do that but it misses the point of having a service registry. :)
fabio is conceptually not dependent on consul. The current implementation is and that is a difference. Your services should depend on a service registry for service registration, discovery and centralized health checks. fabio is a mere consumer of that data to simplify your inbound http traffic routing.
If I wanted to do what you propose then I could stop using consul altogether and provide some generic mysql backend. Also, it would make your services dependent on the fabio API and I would have to replicate the behavior of the APIs that already exist. You couldn't use existing libraries for service registries.
My suggestion is to embrace consul instead of fighting it. If you want to use a different service registry backend then that is fine but then we/you/I would have to spend some time implementing it. A friend of mine has implemented one for the Google Compute Platform. It wasn't that difficult.

Please feel free to comment to keep the discussion going since I am curious why you are suggesting this in the first place but unless I am missing something really fundamental this is not going to happen.

Fun-damentally, Fabio is still just a consumer. What I am proposing, albeit a bit flawed, is a way for us to register any registry by way of API. You have an interface right now which requires a compilation. If you provided an API for us to wire up an external registry (or multiple ones), this would make it easy for any 3rd party to build their own.
Perhaps I might propose this api strategy
/register - register a registry with Fabio
/unregister - unregister a registry with Fabio
The register would give any metadata that Fabio needs to poll from the registry service.
Embracing Consul is only one half the equation since for Docker we need a registrator service as well. This propagates more points of failure in the system. Trying to eliminate this. Could easily provide a etcd solution or a Docker API solution or a proxy to another DNS. Whatever we dream. Could do all these things if you provided an API so we could easily tell Fabio what registry to use. As long as the API is available to us and you, we don't need Fabio to be compiled for every registry need out there. NOR do we have to clone or create any basturdization of your efforts for each one. We can continue to take advantage of your strong solution while plugging in our own registry service whatever it may be. You can continue as is and not provide state (even though you are providing state for each Route/Target).

The current registry is implemented via an interface which could be externalized. https://github.com/eBay/fabio/blob/master/registry/backend.go contains the interface definition.
OK, I get it - finally :) I'll think about it. Would make integration simpler but deployment more complex since you now have to build at least two components. Fabio and the registry adapter. But I could provide the current consul implementation as a sample implementation.

Just as a note for myself:
The idea is to decouple the registry backend from fabio itself via an API to make development of registry backends simpler since they don't need to be integrated into fabio itself.
Although, it makes the development of the registry backend simpler fabio no longer is the one binary that you run and it just works. This will increase operational complexity and create another source for potential errors. The benefit of this approach largely depends on how difficult it is to add support for additional registries. I think the idea has merit but there is a hidden cost in additional integration testing. Then users will have to debug both fabio and the registry plugin to determine where the cause of a misbehavior is.
So I'm not sold yet since lowering the operational complexity is one of the key goals of fabio.

I counter. You've just increased it by forcing only one provider down our throats. Your interface is useless internally. If you add another one we still have a complexity issue of having infrastructure complexity OUTSIDE of Fabio.
I recommend:
Continue to provide Consul built in. No change.
Create a generic registry using your interface.
But you set the API rules. This is a contract which we, as registry providers, must adhere to.

how to register
how to unregister
how fabio will call to check the registry and what the registry must provide back as a response. This can be same format as what you expect for services and k/v interaction from Consul. We, the providers, must strictly follow it and give you the info back exactly, or whatever you expect Consul would have done.

This is NOT hard for us and opens up a whole world of tools.
////
How to register might be nothing more than:

an api token (set in config)
a registry name string
an IP address and a path, or it might be a IP address and a port for WS
returns back a unique token (optional)

How to unregister:

api token
unique token from before (optional) or a name

Fabio callback format (when Fabio calls this endpoint above)

This can be the same layout as what you expect from Consul for all features you are using now.

I am not worried about the implementation since this is rather simple but what this introduces is a dependency problem. It means that this now becomes a binding contract which I must support forever since I have no control over the registries. If I want to support an additional functionality or refactor the API I have to support both APIs and you will have registry plugins which work with either the one or the other API. I also have no leverage of forcing registry providers to upgrade nor do I have any way for users to upgrade.
I would essentially give up the ability to quickly fix bugs or provide workarounds in implementations and so would the users.

I'm also wondering how many of these service registries are out there. I didn't focus on adding support for them since this wasn't my initial use case but with the internal api and the GCP proof of concept this has shown not to be too difficult.

I'd also like to make one comment on your language: I didn't force anything down anybodies throat. We use consul internally and that's what I've developed fabio for. I'm open for suggestions but I want the discussion to be solution oriented.

Your concerns are understood. But I would not be concerned about API refactoring or enhancement. Your functionality is not so dynamic that it should be dynamic. Yes, previous features do need to be continued to be supported but that (at least in my experience) has never prevented an API to continue to respond to new features. You can either simply add additional attributes or features in the current set (w/o disrupting the interface -- for example extra json attributes) or if you have a dramatically different set, simply version them e.g. /v1.0, /v2.0 in the standard way API's do when big changes occur.
But serious, how many api routes do you really anticipate? Usually there are almost hundreds when versioning is ever needed, or the app is very close to "business requirements" where that happens (like a MVC web site) but not pure engineering lb.
Be comfortable, you _aren't _responsible for supporting other providers needs. You set that boundary with the API interface. It should be a black box for us.
How many registry's are there out there? It grows all the time with imagination. Its not the registry behind the scene, but the proxy between fabio and the registry source. That source could be anything. You shouldn't care or worry:
Basic:
Fabio <= api => CustomProxy <= tbd => Registry
Classic:
Fabio <= yourapi => CustomProxy <= network => SkyDNS
Fabio <= yourapi => CustomProxy <= api => etcd
Fabio <= yourapi => CustomProxy <= network => zookeeper
Fabio <= yourapi => CustomProxy <= api => Docker
Fabio <= yourapi => CustomProxy <= api => Consul =)
Imagination:
Fabio <= yourapi => CustomProxy <= network => MicrosoftDNS <= app
Fabio <= yourapi => CustomProxy <= network => MySQL <= app
Fabio <= yourapi => CustomProxy <= internet => CustomProxy <= MaraDNS <= customregistrator
whatever you desire.

Another idea instead of api
https://github.com/hashicorp/go-plugin

This could indeed be a better approach than an API since otherwise people will write plugins in all kinds of languages which then pulls in those dependencies.

The go-plugin approach has the appeal of keeping the deployment simple since you would have to deploy at most two binaries: fabio and the registry plugin. However, testing the plugin would require a fabio instance or a mock that correctly implements that RPC interface. Ideally, the plugins would live as sub-directories in the fabio repo so that you don't have to assemble a working distribution. I'm just wondering how that is any different from the approach that I am taking right now with the only exception that not all dependencies are compiled in.
While the API approach provides great flexibility it will make integration testing, bugfixing and triaging, especially deployment more complex since people will write plugins in other languages which then pull in those dependencies. Stability of fabio then becomes directly related to the stability of that plugin with little option for me to fix it.
So right now I'm leaning towards getting a couple more registry backends into fabio first to see that a) the interface is sufficient and b) integration cost into fabio is high enough to justify a split. If the Docker API is an important part for you why don't you take the plunge of providing an implementation for it?
Adding a generic registry plugin which just externalizes the API is something that can be achieved at any time. Turning that clock back seems a bit harder to do.

There is one additional aspect which is the KV store. Right now this is used for manual overrides and I've got issue #27 which asks for storing certificates there. Storing this information so that it is available for all fabio instances requires consensus so that they all have the same view of the routing table. In the current setup consul solves that. Once consul becomes optional I would have to provide another alternative for these features. etcd and zookeeper provide alternatives but what about the Docker API or the various DNS services you mentioned? This probably requires decoupling the service discovery from the KV management in the API but still leaves me with the question on how to actually implement it.

Any features you are dependent on now from Consul should not change. The registry should be a black box as far as Fabio is concerned. I thought I had done enough to highlight this importance.
The proxy layer (or whatever you want to call it. perhaps driver is a better word) is responsible for keeping these dependent features intact.  You should not be testing it.
Another example to look at is how docker allows for custom engines.
https://docs.docker.com/engine/extend/plugins_network/
Do you think Docker needs to be responsible for testing third party plugins?  I think not. It's the developers responsibility. Why do you keep returning to that idea fixee that you are responsible??  I'm baffled.
The docker interface is something I want to do, but I dont want to compile it into your code, hence why an API of sorts would be a help.

I am not responsible for the quality of third party plugins but those plugins will make the deployment and more complex and with that fabio itself. The fact that it is a single binary which you just copy and run and don't have to configure is not a coincidence.
I am running a fairly complex setup right now and also maintain a public API which I have to keep backwards compatible. I prefer solutions that have "batteries included". Engineering is about tradeoffs and I'm looking for holes in your argument. In your mind the problem is solved but I just don't see it. The fact that you don't see any downsides but only benefits I something that in my experience doesn't work out.
fabio is not dependent on consul. It is dependent on a registry that provides the addresses of active services and the routes they serve. Because of the automatic nature there should be a way to modify the generated routing table in a consistent and persistent way (overrides).
You and others already mentioned that consul an unnecessary dependency. It is one more piece of infrastructure that has to be installed, maintained, backed up, configured, monitored, scaled, secured, upgraded and understood. So how would the Microsoft DNS registry plugin work if there is no consul and you want to store and maintain overrides for a cluster of fabio instances distributed over a couple of datacenters?
As for the Docker API: From what I can tell this is for a single host setup with multiple containers which can be queried fairly simple via the GET /containers/json call which provides addresses and ports of the containers and ENV vars could contain the paths to be registered. That and some authentication, maybe listen to the event API instead of polling should do the trick and can be implemented with the standard http lib. The problem is only where to store manual overrides but maybe that isn't an issue in a single host deployment.

After thinking about it for a while I think the simplest thing that could possibly work is along the lines of what you suggest. Fabio could just watch multiple KV entries in consul for changes instead of one for the overrides. By watching /fabio/registry/* and merging them all together you could support any external system through any means since all you have to do is to manage that entry with an HTTP POST request on an already existing API.
e.g.
/fabio/registry/consul
/fabio/registry/static
/fabio/registry/SkyDNS
/fabio/registry/Docker

The benefit of this is that I wouldn't have to think about a consistent persistence layer, authentication, API and so forth since consul already provides this. The current implementation of the consul registry could also use the same mechanism to be consistent with the rest and it could either become a plugin or remain internal. Order would not matter since the registry plugins are expected to issue only route add commands and the route del commands would remain in the manual overrides which would still be appended last. What do you think?

Sorry, been tied up here with security patching and other fires.
I'm sort of confused myself here so I will first sum up what I was thinking.
Where your argument breaks down for me is that you state you are not dependent on consul. We are mixing things here a bit. The fact that someone has to have another registry compiled into your code to be able to use it indicates there is a dependency issue - on you officiating and making available these alternatives for us. We, in the meantime, wait around. Also, you are linking in third party libraries for interfaces we don't need. That would mean bloat and overhead.
What I'm suggesting is to provide a plugin framework of some sort that is totally abstract and has no dependencies on any other services running via the Fabio Codebase. No consul compiled in drivers. Fabio is a black box to the registry services through a dynamic plug interface; the registry service is a black box to fabio. In fact, I would externalize your Consul registry code into a plugin and create another repo for it.
When I say plugin I mean either API  or code. I will let you decide. Its just an external service similar to a database -- say MySQL -- running through JDBC.  Only in this case, JDBC is really some fabio specification you came up with that developers must meet in order for fabio to work.
When fabio boots, you tell fabio what plugins to utilize and any config information it might need to utilize the plugin.
K/V overrides are responsible by whatever plugin is being called. Fabio should not care as long as it gets the informatino back.  For example, Foo.fabreg might store that in a local yaml file, while Bar.fabreg might utilize redis.  Fabio shouldn't care as long as it gets back the info it needs through an API/plugin standard that Fabio is expecting.

concerning authentication.
Let the configs for both the running registry and fabio contain a simple Bearer like token No need for fancy dancy stuff. We just want two services running close together on the same box anyway.
This is what I would code out. All as Docker images and containers...
Fabio <== api/rpc ==> RegistryPlugConsul <= api => Consul
and
Fabio <== api/rpc ==> RegistryPlugDocker <= api => Docker
||
<======> Local Map Storage (for K/V request handing from Fabio)
Fabio doesn't care. Just another IP address + port similar to a DB

This would require that every plugin developer has to solve the problem of the KV store. Then the KV store connection should be a separate API as well. One for consul, file, redis, ...
While this makes it indeed very flexible it would also mean that you now have to install, run, monitor and upgrade three things instead of just one. Maybe written in different languages, requiring different incompatible runtimes. Different combinations of plugins will work while others wont. Failure of any of these components leads to failure of fabio. People less equipped with knowledge of these things will plug things together and this will create support questions which someone has to pick up and respond to. Since fabio sits in the critical path of any application this worries me.
That is one of the reasons I like Go and the Hashicorp approach. Building a single "batteries included" binary which just works. You move the complexity from the deployment into the development. The plugin approach makes it easier for the developers to integrate but harder for the users to use.
Yes, code has to be compiled into fabio and a library might have to be added but the registry code is usually very simple, based on either HTTP, file or DNS and can fit into a couple of lines of code since the systems you are interfacing with already have an API.
However, I do acknowledge that another method for integrating with other registries is beneficial. I think watching multiple endpoints in consul might be a balanced first step since you can then write a plugin for another registry which just pushes the routing table into consul. With that step fabio becomes actually dependent on consul.

Another thought is that fabio actually only needs a versioned KV store and could provide an API to that to avoid that registry plugins would have to implement different KV store mechanisms. Then the fabio instances only need to be notified when the KV store has changed.

Also, the KV store is also important for custom error pages and certificates for the whole cluster.

consul as the KV store also provides one additional function which is essential to fabio clusters: change notification. Without that you have to revert to polling.
If I assume that the KV store and the coordination are the critical functions for fabio and that there are any number of providers which push routing information into that store then I could do the following:
fabio provides an API endpoint which allows uploading a routing table. Each routing table has a name, any number of routing tables is possible, all routing tables are merged in no particular order with the manual overrides merged last.
In the first iteration fabio would continue to use consul for the KV store and the coordination but other options are possible. To support any KV store (e.g. redis, MySQL, memcache, ...) I would have to factor out the cluster management (i.e. change notification).
So adding these endpoints together with merging all routing tables should provide the desired functionality.
PUT /routes/<name>
Content-Type: text/plain

route add service ...

DELETE /routes/<name>

This would allow you to write a plugin for the Docker API which just pushes its routing table into fabio whenever it detects that there is an update.
Initially I would support the current config language but support JSON in a subsequent iteration. Then I'll refactor the built-in consul registry to also just push to KV store. But there has to be a leader election so that not all fabio instances of a consul cluster push the same changes to the KV store at the same time.
This different routes should also be visible in the UI.
I could provide a number of "provided" registry plugins which get built together with fabio and which I run from fabio as a child process providing the API endpoint as a startup parameter. This way I still have a single code base, one set of deployables and you still have to start only fabio to get things rolling.

consul as the KV store also provides one additional function which is essential to fabio clusters: change notification. Without that you have to revert to polling.

I don't see the issue. I have no problem with adding a WS connection with the driver for callbacks as long as you provide a payload spec that Fabio expects. Long polling is also possible but any registry proxy/driver can easily implement a WS.
I'm confused where you are going with this but at least you are going. =D
My only gripe is having any hard boiled/compiled in libraries. As long as a metadata is defined and followed and you use a consistent fast transfer mechanism (such as RPC, WS) to that driver/proxy layer (which handles the key features that Fabio needs e.g. redis, etcd, consul, dockerapi for registry) then it will work.
Ideally, looking at fabio code should be very vanilla without references to any specific implementation of the kv/registry. Simple code.
Then, separate repos for drivers/proxy or whatever the name should be.
e.v.
eBay/fabio     - base router
eBay/fabio-consul - driver for consul
eBay/fabio-docker - driver for docker
eBay/fabio-etcd - driver for etcd.
etc...

In the current implementation consul has two completely independent functions:

consistent KV store
service registry

With that in mind:

Docker API is a service registry
etcd is a KV store
zookeeper is a KV store
redis is a KV store
GCP is a service registry
SkyDNS is a (probably) a service registry
...

To really follow your argument fabio would require two plugins:

a KV driver (to store manual overrides, 404/503 pages, certificates, cluster configuration)
a service registry driver (to generate the routing table)

Factoring these out as external plugins simplifies the development of external integrations but complicates deployment since now three independent executables are necessary for fabio to function independent of how they actually communicate.
Also, when you run a cluster of multiple fabio instances (as we do in our 16 node clusters) how do they all get notified that the manual overrides were changed by a user or that a 404 page changed? Again, in the current setup consul provides this function by providing a consistent storage and change notification in between the fabio instances.
Ease of deployment is most likely one design goal that the Hashicorp developers had when developing consul and nomad for example. Nomad has integration points but provides most functions out of the box (from the single binary) without the use of external plugins - unlike mesos/marathon which is way more flexible and more difficult to setup and control.
My issue is that I most likely don't have enough bandwidth to build integrations for all possible combinations into a single product even though I find that preferable since the library bloat in Go is minimal. I also don't think that a large plugin eco system is always the preferable solution. (again see Hashicorp)
What I do think though is that it would be sufficient if fabio supports a selection of common KV store backends natively (as in built-in) and provides an external API for service registries. I think supporting consul, etcd, redis and zookeeper as KV stores should provide enough coverage to get started.

You don't have to provide any plugin except one for Consul. We, out here, can do the rest as we need them. Just create an plugin protocol for us to follow.
Also you don't have to deal with two -- one of kv and one for services.  Assume the plugin, whatever we develop will handle both requests. ONE plugin for both.
Just to show you that it's possible, we could do this now without your coding anything. All we have to do is create a MOCK of Consul...make a server that looks like Consul to Fabio but really its just a hash of structures for the KV and using Docker API. Fabio would never know. It would just make Consul calls to a IP/Port/ Consuls API become the standard we implement. It would SPOOF Fabio by making it think it was a Consul instance.
But is this a solution? I don't think so. Removing your internal consul libraries and building a RPC/plugin would be the more mature way, using the Consul commands you are using as a standard to call other things.
I still don't believe you grasp this elegence yet.

and so...
https://traefik.io

Reading through this thread, I was thinking of Traefik.  It is very agnostic about the underlying config plane.
Fabio seems much more complete, though.
Our distributed config and service registry is converging around etcd.  It has certainly crossed my mind to build an etcd/consul gateway to get Fabio to play nice.
I totally understand the author's indifference to this if consul is entrenched with his employer.
We are also looking at building a supervisory daemon around HAProxy as another option.  Something like vamp.io but a bit simpler.   This would couple a newfangled control plane with a battle tested runtime.
The most interesting thing about all of this is how dynamic (and unsolved) this problem space is.  Lots of good stuff out there.  No obvious long term winner.
Like what you have built here though.   I think that maybe that accounts for the strong opinions in this thread.
Fabio could totally be the load balancer for the next 10 years.

@if6was9 i haven't given up on that plan but I think i need to provide a KV store solution and Fabio should retain its zero conf approach. This isn't related to eBay using consul. We just happen to pick it. Kubernetes is an obvious target.


[bazel] Remove deprecated config_setting usage

In a future version of bazel using cpu in config_setting is
deprecated, for these cases we can instead use platform constraints,
which actually simplifies this use case as well. Fixes:
WARNING: /.../external/grpc+/BUILD:192:15: in config_setting rule @@grpc+//:mac_arm64: select() on cpu is deprecated. Use platform constraints instead: https://bazel.build/docs/configurable-attributes#platforms.

Thanks for your contribution!


Fix pyodide runner

install pydantic==2.7.0 instead of 2.7.1 which is not available on Pyodide registry,
install pygments separately as for some reason it's not pulled as a dep
run multiqc.run with ClConfig
update pyodide itself *

Nice one!


New Link added- Set Up SAP Mobile Cards

Open https://developers.sap.com/tutorials/cp-mobile-cards-setup.html#b3213ab1-702d-4891-bf8d-e011f21ea8bf
New link added- Use a Cloud foundry service

Hi @Mukesh-50,
Thank you for raising the issue. We will update the screenshot.
Warm Regards
Sandeep T D S

Thank you for your feedback. The issue seems to be resolved, so I am closing the issue.
If you still have questions, feel free to reopen the issue.


leave the map when you jump from large height

Hi, thank you for this great job.
Can leave the map when you jump from large height;
Maybe Rigidbody Collision settings.

This is caused by the lack of interpolation.

I've moved active development to my https://www.patreon.com/tarodev.


The error message (& output) for NETSDK1148 could be MUCH more helpful

A referenced assembly was compiled using a newer version of Microsoft.Windows.SDK.NET.dll. Please update to a newer .NET SDK in order to reference this assembly.

This error message is really vague.
Which referenced assembly?
What are the different versions that have been detected?
I ended up here after getting this message but being unable to determine which dependency had the issue.
In fact, the only discrepancy in version numbers I could see was with the referenced assembly having a lower/older version of the SDK.
I now don't know if the error message is wrong or if I'm looking at the wrong dependency.
I'm left in a situation where the error message is essentially saying "there's something wrong, I'm not going to tell you exactly what, but it's probably related to SDK versions."
That's not very helpful!

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 84bd87ce-62bc-500a-6d55-3f76787ae850
Version Independent ID: 617440e5-3596-7bf3-8cbf-0cfa0d477f5a
Content: .NET SDK error list - .NET CLI
Content Source: docs/core/tools/sdk-errors/index.md
Product: dotnet-fundamentals
Technology: dotnet-cli
GitHub Login: @tdykstra
Microsoft Alias: tdykstra

The message implies that you just need to update the SDK that you're using to compile. Is it possible you don't have the latest version installed or you have a global.json file specifying an earlier version?

The message implies that you just need to update the SDK that you're using to compile. Is it possible you don't have the latest version installed or you have a global.json file specifying an earlier version?

"Just"?
As you point out, there are multiple places it could be looking for this, or it could be specified.
If the error message told me which versions were being used by which projects/references (which it definitely knows as it's comparing them), that could greatly help in debugging this.
By looking at the referenced version numbers in the projects, it's not clear where the problem is.
"Just" updating the SDK might not even do anything or risk breaking other compatibility issues. Plus, I already have the latest version installed, so that's not it.
I'm not looking for a solution to my specific issue (I found a workaround). I'm looking to make this easier for others who encounter this message in the future. (Or myself if I hit it again.)

Please transfer it to the SDK repo. The error lives in the SDK though is triggered by RAR returning  ResolveAssemblyReferenceUnresolvedAssemblyConflicts so CC @Forgind for a comment.


[Video Conference] virtual background in Zoom doesn't work

switching back to the original camera resolves this issue.

@dosoft
beside the virtual background, does the virtual camera functionality work for you?

Tracking the issue in https://github.com/microsoft/PowerToys/issues/6246


Changelog

It would be great if you could provide a changelog which lists especially all breaking changes.
Its not so easy to find out what happened from 0.11.2 to 1.x or from 1.x to the current 2.0.1 npm release.
And, by the way, bower and npm versions mismatch. Bower is 1.1.0, npm is 2.0.1.

 The breaking change was how the distributed files are generated and published for angular-cli support. It should actually currently be 0.13.1 but that's what happens when you code tired! I'll add a changelog, but that's the changes, just as an fyi.


cake midje double-loads files after changes

Cake 0.6 reloads changed source files before running any command. So far, so good. However...
To check facts, midje cake loads all the src and test files in the project, including any that have changed. So...
Some files may be loaded twice, and some test failures may be printed twice. The summary counts will still be correct.
Note: when this is fixed, fix the text at the end of https://github.com/marick/Midje/wiki/Cake-midje

Marking "won't fix" since cake seems to be dieing out.  If no one disagrees, I will close this issue.


The type or namespace name 'Xcode' does not exist in the namespace 'UnityEditor.iOS'

I was using a plugin in my project which has a build post processor.
When VSCode loaded the project, it showed an error on the line
using UnityEditor.iOS.Xcode;
saying:

The type or namespace name 'Xcode' does not exist in the namespace 'UnityEditor.iOS' (are you missing an assembly reference?) [Assembly-CSharp-Editor]

However, in Unity, build was successful. Is there any way to let VSCode recognize this namespace?
I was using Unity 5.4.0b15 on El Capitan with latest VSCode and VSCode plugin.

I only have b14 installed on this computer, but i had a peak and it doesn't look like there is a class or namespace under UnityEditor.iOS. Also online docs seem to support that, of course it could have been added in b15, ill probably install it tomorrow.
Are you sure that the precompile conditions aren't causing an ignore of that line?
Can anyone confirm/deny that is added in beta 15? Is there a library adding it outside of the standard unity libraries.

So, to elaborate, the code looks like this in the file:
#if UNITY_5 && (UNITY_IOS || UNITY_TVOS)
using UnityEditor.iOS.Xcode;
#endif

The building target was set as iOS so this precompile conditions included this. If the build target is set as Android this error disappears.
Actually there is a class in the code that seems to rely on that namespace:
#if UNITY_IOS
#if UNITY_5
...
PBXProject project = new PBXProject();
...
#endif
#endif

Actually I did a google search and I found this class description located in the online scripting reference. Looks like the Xcode.iOS namespace is also shown there.
(http://docs.unity3d.com/540/Documentation/ScriptReference/iOS.Xcode.PBXProject.html)

Interesting, mainly because i don't find it in auto complete either ... with b15 installed.
So I did some digging ... specifically looking at the oddities of that namespace, since it is not part of the core Unity it is actually located in the playback libraries specifically for iOS (UnityEditor.iOS.Extensions.Xcode). That being said, it looks like you might have to manually include that into your project?
Does Unity maybe include it during its compile? Does it show up in a mlnodevelop project if you look there? Have a search in your project file to see if its in there ... if its not that will give me a place to start ie am i cutting it out by accident, or is unity just not giving it to us.

Unfortunately, I tried "Build All" in MonoDevelop, and it also gave out the "namespace 'Xcode' does not exists" error, even though the C# file is in Assembly-CSharp-Editor.csproj.
Another thing I found is that IntelliSense in both VSCode and MonoDev could not even show "iOS" namespace in the auto complete when I enter"using UnityEditor."

Ya, it sounds like Unity includes that manually in their own compilation pass.
One possible option is that I could detect the platform and include it based on the relative path to Unity, but that seems 'backwards'. Unity should be including it on their own project creation. I'm going to send this along the chain.

Unity has acknowledged this as a problem and has passed it on for resolution within their development team. Unfortunately, no-one knows when a resolution will be in place :(

It is good to know that it is a bug, not something Unity intentionally did to restrict accesses. 😆 Thanks for doing that.
Do they have a bug report of this to track progresses??

793624
I would paste the actual link, but the whole back history is visible for my account with them at that point and sadly, thats not something I wish to share :) its been quite a few years LOL


test that the id is encrypted and not stored in raw format

Add test that the id is stored encrypted and not in raw format in the utility emissions emissions records.

Hi @sichen1234 ,
Is it for the utilityId ?
If that's, then I guess we already have the test for it:
let entry = response.body[0];           assert(entry.utilityId == "USA_EIA_11208", "GET request returned incorrect utilityId");           assert(entry.partyId == "1234567890", "GET request returned incorrect partyId");

It's for the partyId.  This tests that we get back the original partyId.
But how do we know that it has been encrypted in Fabric?
Si Chen
Open Source Strategies, Inc.
New Video: Creating Climate Solutions with Blockchain and Open Source
https://youtu.be/gtO3COq8crQ
On Sun, Jun 27, 2021 at 9:17 AM Pranamika Pandey @.***>
wrote:

Hi @sichen1234 https://github.com/sichen1234 ,
Is it for the utilityId ?
If that's, then I guess we already have the test for it:
let entry = response.body[0]; assert(entry.utilityId == "USA_EIA_11208",
"GET request returned incorrect utilityId"); assert(entry.partyId ==
"1234567890", "GET request returned incorrect partyId");
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hyperledger-labs/blockchain-carbon-accounting/issues/187#issuecomment-869188628,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANAS4JPH3ATPIGHKDJHSALTU5FIPANCNFSM464BZW3Q
.

Hi @sichen1234 ,
There can be two methods for it:

I guess, testing if the partyId is in raw format.
Using bcrypt.compare method to test a plaintext partyId against a hash, as mentioned here: https://stackoverflow.com/questions/42561968/using-chai-to-test-if-post-users-password-was-encrypted

Could you try adding test to compare the output of partyId from the REST
API?  Is it in the encrypted or unencrypted format right now?  If it is
encrypted then you can compare against the hash.  If the rest api is
already returning unencrypted then we'll have to find a way to get the
records directly on Fabric or in Couchdb.
Si Chen
Open Source Strategies, Inc.
New Video: Creating Climate Solutions with Blockchain and Open Source
https://youtu.be/gtO3COq8crQ
On Mon, Jun 28, 2021 at 8:58 AM Pranamika Pandey @.***>
wrote:

Hi @sichen1234 https://github.com/sichen1234 ,
There can be two methods for it:

I guess, testing if the partyId is in raw format.
Using bcrypt.compare method to test a plaintext partyId against a
hash, as mentioned here:
https://stackoverflow.com/questions/42561968/using-chai-to-test-if-post-users-password-was-encrypted

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hyperledger-labs/blockchain-carbon-accounting/issues/187#issuecomment-869805237,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANAS4L6S5CX2U4T4RFHSVLTVCL2BANCNFSM464BZW3Q
.

@sichen1234 do you want to store the encrypted partyId with AES or a similar cipher algorithm (https://cryptojs.gitbook.io/docs/#the-cipher-algorithms) or do you just want to check if the partyId is stored as a hash value?
Why do we need to encrypt the partyId?

We should already be encrypting them.  We did it because sometimes ID's
have sensitive information.  See
https://github.com/hyperledger-labs/blockchain-carbon-accounting/issues/135
Si Chen
Open Source Strategies, Inc.
New Video: Creating Climate Solutions with Blockchain and Open Source
https://youtu.be/gtO3COq8crQ
On Mon, Jul 5, 2021 at 5:03 AM Robin Klemens @.***>
wrote:

@sichen1234 https://github.com/sichen1234 do you want to store the
encrypted partyId with AES or a similar cipher algorithm (
https://cryptojs.gitbook.io/docs/#the-cipher-algorithms) or do you just
want to check if the partyId is stored as a hash value?
Why do we need to encrypt the partyId?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hyperledger-labs/blockchain-carbon-accounting/issues/187#issuecomment-874058809,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANAS4IISBV64BPPRN2DBT3TWGNSNANCNFSM464BZW3Q
.

@sichen1234, thanks for the information.
How much does the partyId vary for the different records? If we just use SHA256 to calculate a hash value of the partyId and if the partyIds are somehow similar or follow certain rules it could be easy to get the raw data via brute force hashing of the partyId schema.
To increase privacy we should consider using AES to really encrypt the data or calculate a hash value from the partyId and a random number we store privately in a database.
The following link explains my concerns: https://hyperledger-fabric.readthedocs.io/en/release-2.2/private-data-arch.html#protecting-private-data-content

The partyId is used for lookup of records.  If it is stored this way, how
will all members still be able to look up emissions records for a
particular party?

Si Chen
Open Source Strategies, Inc.
New Video: Creating Climate Solutions with Blockchain and Open Source
https://youtu.be/gtO3COq8crQ
On Tue, Jul 6, 2021 at 1:31 AM Robin Klemens @.***>
wrote:

@sichen1234 https://github.com/sichen1234, thanks for the information.
How much does the partyId vary for the different records? If we just use
SHA256 to calculate a hash value of the partyId and if the partyIds are
somehow similar or follow certain rules it could be easy to get the raw
data via brute force hashing of the partyId schema.
To increase privacy we should consider using AES to really encrypt the
data or calculate a hash value from the partyId and a random number we
store privately in a database.
The following link explains my concerns:
https://hyperledger-fabric.readthedocs.io/en/release-2.2/private-data-arch.html#protecting-private-data-content
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hyperledger-labs/blockchain-carbon-accounting/issues/187#issuecomment-874568705,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANAS4JSWXM7R47QWUI3673TWK5NPANCNFSM464BZW3Q
.

@sichen1234, as @petermetz highlighted in #230, hashing != encryption.
If we just hash the partyId and store the hash value someone could get the raw data via brute force hashing.
If we encrypt the partyId only people with the passphrase could decrypt the partyId and get the raw data. However, the encrypted partyId will stay the same as long as the passphrase doesn't change. So you could just query the encrypted partyId to look up all the emissions records. This can then be considered as a pseudonymous identifier.
Do I get this right? If so should we use encryption or hashing?

You're right, hashes could be broken like that, but @petermetz had said in a discord chat that sha256 was stronger than md5sum.  See attached:

Is it strong enough?
On the other hand, if you encrypt the party id, will a different app going through a different node be able to query for the same party, if they just know what the encryption algorithm is?  Or will they need the encryption passphrase as well?

@udosson  @sichen1234 Apologies for not going into more details earlier. SHA256 is much more secure than MD5, yes. Whether it's secure enough, it depends on the key space (maximum length of the account numbers) and a few other things.
What I meant by the hashing != encryption advice was just to not refer to it as encryption in documentation or anywhere else because it will confuse people who would assume based on that wording that the account numbers are in fact encrypted with an scheme such as AES-256.
It is perfectly valid to say that the SHA256 hashing is secure enough for the use-case/parameters if that's how it is.
More details below (if you are interested)
For example (back of the envelope calculations) if the account numbers are all 10 digit long numbers then you could task a consumer grade GPU to pre-calculate the hashes of all possible (10 billion) accounts numbers between 1 .. 999999999 and it would take it maybe 10 seconds to finish (based on this benchmark 1) with that task.
The resulting set of account number+hash pairs could be then used to quickly reverse look up which hash belongs to which account number - you'd call this data structure a rainbow table.
On the other hand if an account numbers are 20 digits long and they can contain alphanumeric characters as well then the key space is 36^20 meaning that computing the rainbow table as explained above would require much more time than the planet has left before the Sun goes cold a few billion years from now (not to mention that storing that rainbow table would be also practically impossible).
Another thing to consider is the allocation algorithm of the account numbers. If they are fully random within the key space that's good, but if it's sequential assignment (0001,0002,0003,00004) then it doesn't actually help if the key space itself is large because an attacker would only need to pre-compute the first N values for the rainbow table where N is the number of customers in total.
Yet another thing to consider is that you can protect against rainbow tables with salt & pepper 2 which can further strengthen your hashes making them almost similar to having encryption (Cryptographers will yell at me for even suggesting that, but I'm trying to keep it short)

Thanks for clearing that up.
I will check with @udosson about this.

@udosson If we could encrypt rather than just has the party Id then that would be a good thing to do.  Would you like to create a separate task for that?

@sichen1234, I think that encryption is the better option if we want to protect private data. Please, take a look at #263.

That sounds good.  Let's leave this task as an implementation of the
encryption then.
Si Chen
Open Source Strategies, Inc.
New Video: Creating Climate Solutions with Blockchain and Open Source
https://youtu.be/gtO3COq8crQ
On Tue, Jul 20, 2021 at 1:31 AM Robin Klemens @.***>
wrote:

@sichen1234 https://github.com/sichen1234, I think that encryption is
the better option if we want to protect private data. Please, take a look
at #263
https://github.com/hyperledger-labs/blockchain-carbon-accounting/issues/263
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hyperledger-labs/blockchain-carbon-accounting/issues/187#issuecomment-883206768,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANAS4ONY6KA3ZOBUHI64K3TYUX7PANCNFSM464BZW3Q
.


MAINT: Translation of hyp2f1 for complex numbers into Cython, part 2

Reference issue

This PR does not close any issues known to me.
What does this implement/fix?

This PR is the third in an ongoing project to translate the Gaussian hypergeometric function (exposed through scipy.special.hyp2f1) into Cython. To recap: the original Fortran implementation has many defects but the difficulty of working with and reviewing many lines of dense and sparsely documented Fortran 77 code has thwarted previous efforts to correct them. The goal is to move parts of the implementation in Cython piece by piece while correcting the defects until at last a solid and maintainable implementation is completed.
This PR handles the following cases

Unital argument for c - a - b > 0 through DLMF 15.4.20.
z = -1 and c = 1 + a - b through DLMF 15.4.26.
c - a or c - b a negative integer by applying an Euler hypegeometric transformation DLMF 15.8.1 after which the computation reduces to evaluation of a polynomial.
Re(z) >= 0 and |z| < 0.9 through the Maclaurin series for hyp2f1.

For cases 1 and 2, where the function can be evaluated as a ratio of products of gamma functions, the behavior has been updated to compute with logs and then exponentiate if an underflow or overflow occurs with the standard method.
We've also updated the max_degree argument of hyp2f1_series to be a uint64. This gets rid of platform dependent behavior for the cases where computation reduces to polynomial evaluation.
Additional information

Interesting, one of the special cases is failing on 32 bit Linux.
scipy/special/tests/test_hyp2f1.py:298: in test_special_case_z_near_minus_1
    assert_allclose(hyp2f1(a, b, c, z), expected, rtol=rtol)
E   AssertionError:
E   Not equal to tolerance rtol=1e-07, atol=0
E
E   Mismatch: 100%
E   Max absolute difference: 1.11460442e+27
E   Max relative difference: 2.4690097e+19
E    x: array(-1.114604e+27+0.j)
E    y: array(45143784.467839+0.j)
        a          = -102.1
        b          = -20.3
        c          = -80.8
        expected   = (45143784.46783885+0j)
        hyp2f1_test_case = Hyp2f1TestCase(a=-102.1, b=-20.3, c=-80.8, z=(-1+0j), expected=(45143784.46783885+0j), rtol=1e-07)
        rtol       = 1e-07
        self       = <scipy.special.tests.test_hyp2f1.TestHyp2f1 object at 0xd7f2c418>
        z          = (-1+0j)

The relevant code is
        # Try again with logs if there has been an overflow.
        if zisnan(result) or zisinf(result) or result == 0.0:
            result = exp(
                lgam(c) - lgam(c - a) +
                lgam(c - a - b) - lgam(c - b)
            )
            result *= gammasgn(c) * gammasgn(c - a - b)
            result *= gammasgn(c - a) * gammasgn(c - b)
        return result

What's interesting is that the imaginary part of the incorrect computed answer is equal up to tolerance with the real part of the expected answer. I'll have to think about this. If I can't get it to work on 32 bit I think I should just scrap this idea to handle over/underflow by taking logs so that behavior can at least be consistent across platforms.

I was able to reproduce the error in a 32-bit Linux docker container. I might have time on Monday to try to get to the bottom of this.

Why not use the boost package to replace cdflib ? What do you think @rgommers

Why not use the boost package to replace cdflib ? What do you think @rgommers

No particular reason. When I need low level access to a special function that's already in scipy.special I tend to default to whatever implementation is provided as a ufunc through spipy/special/functions.json. Note that replacing Gamma with the boost package will not fix the error. What's going on is that the correct branch isn't even being taken. DLMF 15.4.26 is applied when zabs(z + 1) < 1e-15 and fabs(1 + a - b - c) < 1e-15. On 32 bit linux, fabs(1 + a - b - c) is evaluating to ~ 3.56e-15.
One possible fix is just changing the tolerance for fabs(1 + a - b - c), but I'd like to see that it doesn't hurt precision for other parameter values first.

Why not use the boost package to replace cdflib ? What do you think @rgommers

No particular reason. When I need low level access to a special function that's already in scipy.special I tend to default to whatever implementation is provided as a ufunc through spipy/special/functions.json. Note that replacing Gamma with the boost package will not fix the error. What's going on is that the correct branch isn't even being taken. DLMF 15.4.26 is applied when zabs(z + 1) < 1e-15 and fabs(1 + a - b - c) < 1e-15. On 32 bit linux, fabs(1 + a - b - c) is evaluating to ~ 3.56e-15.
One possible fix is just changing the tolerance for fabs(1 + a - b - c), but I'd like to see that it doesn't hurt precision for other parameter values first.

I am sorry, my question wasn't well asked. thanks to @mckib2, Scipy can now use boost C++ functions as in gh-13328. Since hyp2f1 is currently not the only function in cdflib to not worry properly, I think it might be a better idea to replace the whole submodule by the boost implementation of those functions.

I am sorry, my question wasn't well asked. thanks to @mckib2, Scipy can now use boost C++ functions as in gh-13328. Since hyp2f1 is currently not the only function in cdflib to not worry properly, I think it might be a better idea to replace the whole submodule by the boost implementation of those functions.

I see. hyp2f1 for complex numbers actually isn't implemented in cdflib, it's in specfun.f, so I thought you were referring to something else. I don't think hyp2f1 for complex z is implemented in boost actually.

@person142 I decided to punt on the error and mark that particular test as an xfail conditionally for 32 bit systems. That test doesn't even pass on any system with the original implementation so this is still an improvement.

I see. hyp2f1 for complex numbers actually isn't implemented in cdflib, it's in specfun.f, so I thought you were referring to something else. I don't think hyp2f1 for complex z is implemented in boost actually.

As far as I'm aware, Boost distributions don't accept complex-valued inputs unfortunately

@person142 I played around with using the Lanczos approximation on a different branch. It works well except for when any of c, c - a, or c - b are negative. I guess cases like that be can handled with the reflection formula? I haven't checked how the cancellation works out, but the case logic seems like it will be a little complicated. I'll think I'll hold off on this for a little bit and for the next few PRs start translating the different transforms and finish working on the fix for gh-8054 (which is almost done).

@person142 I noticed a bug. I made max_degree in hyp2f1_series a uint64, but the loop variable k is just an int.

It works well except for when any of c, c - a, or c - b are negative

Yes, the approximation doesn't work for negative real part; as you have observed you do need to use the reflection formula there.

Thanks @steppi!

Before merging, I want to make max_degree signed after all. Since it's being mixed with doubles, the conversion from signed to double should be faster than unsigned to double in most cases.

Thanks @person142! I should be ready to submit a PR fixing gh-8054 next weekend. For the next PR, I think I may also change max_degree to a signed int64 instead of unsigned. From what I've read, I think the conversion from signed int to double may be faster on many processors, although I'm not sure if this almost 12 year old stackoverflow answer is now out of date. I'll benchmark first though.

Sounds good. For a complex function like hpy2f1 I'd be surprised if the cast ended up dominating, but it never hurts to look.


Error in cancel command implementation

2021-08-07 20:21:50    IO
||||||||###o#v###e#####o###################|| 458.325 (viterbi)

||||||||###################################|| 456.474
||||||||###o#n###e#########################|| 454.019
||||||||###a#####i#r#######################|| 453.536
||||||||###o#n###e##|##o###################|| 452.781
  [reject]

2021-08-07 20:21:51    IO DECODED
2021-08-07 20:21:51 ERROR cb error topic="pre:phrase" cb=<function pre_phrase at 0x7f2608cbb310>
   25:       lib/python3.9/threading.py:912* # cron thread
   24:       lib/python3.9/threading.py:954*
   23:       lib/python3.9/threading.py:892*
   22:                    talon/cron.py:155|
   21:                    talon/cron.py:103|
   20:          talon/scripting/rctx.py:233| # 'cron' main:<lambda>()
   19:                     talon/vad.py:21 |
   18: talon/scripting/speech_system.py:331|
   17:             talon/engines/w2l.py:931|
   16:      talon/scripting/dispatch.py:105|
   15:      talon/scripting/dispatch.py:144|
   14:      talon/scripting/dispatch.py:133|
   13:          talon/scripting/rctx.py:233| # 'phrase' main:_redispatch()
   12: talon/scripting/speech_system.py:61 |
   11:      talon/scripting/dispatch.py:105|
   10:      talon/scripting/dispatch.py:144|
    9:      talon/scripting/dispatch.py:133|
    8:          talon/scripting/rctx.py:233| # 'phrase' main:engine_event()
    7: talon/scripting/speech_system.py:387|
    6:      talon/scripting/dispatch.py:105|
    5:      talon/scripting/dispatch.py:143|
    4:      talon/scripting/dispatch.py:113|
    3: ------------------------------------# cron thread
    2:          talon/scripting/rctx.py:233| # 'pre:phrase' user.knausj.code.cancel:pre_phrase()
    1:       user/knausj/code/cancel.py:9  | before, after = d["text"][:-n], d['text'][-n:]
KeyError: 'text'

Probably just need to check whether the "text" key is present or not. Not sure though, could ask aegis.

@rntz I believe this is when you making just noises as that actually triggers pre:phrase but without any text. Just check if the key is there or add a try.
My own solution
https://github.com/AndreasArvidsson/andreas-talon/blob/master/misc/abort/abort.py

Fixed by #575


setup CI for NextJS template app

Ticket
WMDP-31
Changes

Added ci.yml to run github action for pass/fail checks when pushing code to remote. Ignores merges (push) to main.
Added .prettierignore, which ignores generated files and list of config files see below.

Context for reviewers

Created initial CI.

Ensures application successfully builds.
Passes all lint checks.
Passes format-check checks.
Passes test.

I decided to ignore configuration files for prettier to allow someone up the chain to decide whether to format them or not.
Left out security scanning, it appears that was handled in WMDP-32
Note branch protections need to be configured separately. Three options:

Setup manually through website
Create 1 shot git cli command using bash
Manage branch protections via terraform (my preference)

"All of these should run again after “merge” is clicked and any failures should block merging into main." I can set CI to run after merge is clicked, but the merge has already occurred so it wont be able to stop a merge, requiring the status checks to pass before merging should be sufficient for CI.

Testing

Testing was done using tst-ci-final in order to keep this repos history clean. Whoever ultimately reviews this just ping me and I will add you as a contributor so you can look at the git histories.

🎉 Thank you!


Use transparent color for scrollbar

The current way the scroll bar hides all the decorations in the right margin.
Please use a transparent color like all other themes

Same problem with minimap in insiders edition.

Thanks for your report 👍
This has already been fixed in #13, I'll try to deploy it next weekend.

@arcticicestudio thanks. In the meantime I will just steal your colors and put them as custom colorization in my workbench settings
"scrollbarSlider.activeBackground": "#434c5e99",
"scrollbarSlider.background": "#3b4252aa",
"scrollbarSlider.hoverBackground": "#434c5e99",

Sorry for the delay!
It's shipped with #13.


Fix migration bug where using -- as an actual string value gets lost

Fixes https://github.com/kriasoft/node-sqlite/issues/27

@koistya

thanks @BeLi4L for the review - was really busy at work since then, will change now

@theogravity what do you think about detecting comments by using ^-- .* (having a space after two dashes)?

I'd rather do that appraoch instead

@koistya @BeLi4L updated review

I'm going to merge this in - it's been sitting around so long and the tests are not breaking.


Remove link to edit profile page when already editing profile

This link is pointless as it displays "EDIT PROFILE" button on the edit profile page :)

Thanks!


Replace wrapper functions with macros

There are a number of Vensim functions in vensim.c that are implemented as simple wrappers around other system functions, e.g. _ABS simply calls fabs.  Some others, such as _IF_THEN_ELSE cause their arguments to be eagerly evaluated.
For both types of functions, we can replace the wrapper functions with simple macros, which will reduce function call overhead and provide some modest performance gains.

Here are some performance numbers, relative to other recent performance work.
MacBook Pro (2019) | 2.4 GHz 8-core i9, 32 GB RAM, macOS 10.15

Issue
C run (ms)
Wasm run (ms)
Wasm init (ms)
JS mem (MB)
Page mem (MB)

baseline
45.8
87.5
38.0
94
685

SDE 18
46.0
85.6
18.0
39
672

SDE 19
42.8
49.4
15.0
38
25

SDE 22
34.8
44.8
15.0
38
21

iPhone 8 | A11, iOS 13

Issue
C run (ms)
Wasm run (ms)
Wasm init (ms)
JS mem (MB)
Page mem (MB)

baseline
39.9
187.0
165.0
39
645

SDE 18
40.3
219.0
86.0
38
724

SDE 19
40.1
81.6
83.0
38
41

SDE 22
35.5
74.6
86.0
40
40

iPad Air (2013) | A7, iOS 12

Issue
C run (ms)
Wasm run (ms)
Wasm init (ms)
JS mem (MB)
Page mem (MB)

baseline
151.0
1372.2
30146.0
77
331

SDE 18
166.0
1408.0
4416.0
42
395

SDE 19
151.0
837.6
1291.0
45
41

SDE 22
137.0
771.6
1484.0
44
40

Size

Issue
Wasm size (bytes)

baseline
1,084,036

SDE 18
773,968

SDE 19
776,851

SDE 22
737,028

Merged to develop in 7c71a0f.


Clarification on method "public void InvokeAction (string data)" in class "HybridWebView"

On this line: if (action == null || data == null) {
What is the reasoning behind it not being string.IsNullOrWhiteSpace(data)
Is an empty string an expected data to pass into the action, if yes, why is a null data not expected as well.
Thank you.

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9c1bd30c-6e7a-df38-8c20-dff7361e901f
Version Independent ID: 013c6027-db9a-6039-3404-2e4f717133a9
Content: Implementing a HybridWebView - Xamarin
Content Source: docs/xamarin-forms/app-fundamentals/custom-renderer/hybridwebview.md
Product: xamarin
GitHub Login: @davidbritch
Microsoft Alias: dabritch

Hi @gsauzet
string.IsNullOrWhiteSpace applies here. The example has changed over time, and didn't originally use string data.


NgEntityService Loader Misses First Emission When using Async Pipe & NgOnInit

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report
[ ] Performance issue
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request
[ ] Other... Please describe:

Current behavior
Based on documentation*, I should be able to call an NgEntityService .get() in my components ngOnInit() function and subscribe to an NgEntityServiceLoader observable in the component template using async pipe.
[1] Docs: https://datorama.github.io/akita/docs/angular/entity-service#entity-service-loader
However, since the dispatcher in NgEntityServiceLoader is a subject, and the call to NgEntityService .get() occurs before the NgEntityServiceLoader observable is subscribed to via async pipe, the first emission is always missed.
Expected behavior
Subscribing to an NgEntityServiceLoader observable should return the current state of the observable / loader.
Minimal reproduction of the problem with instructions
I created an example close to the documentation. Caveat, I added an interceptor to delay the HTTP request to clearly show this behavior.
[2] Reproduction: https://stackblitz.com/edit/akita-loader-async-pipe-problem?file=src/app/hello.component.ts
What is the motivation / use case for changing the behavior?
The use case is the same as the documentation. Subscribe to .get() in your components ngOnInit, and subscribe to one of the loader's observables in the component's template via async pipe.
Today from a client perspective (no changes to lib), you could remedy this problem by creating your async pipe outside your component and passing the loaders value in as an input, passing the value of the loader to a Replay/BehaviorSubject, subscribing directly in NgOnInit, etc...  None of these methods are as convenient / idiomatic as the example in the documentation.
Environment

@datorama/akita: 6.1.3
@datorama/akita-ng-entity-service: 6.0.0
Angular version: 11.2.12

Browser:

[x] Chrome (desktop) version 90.0.4430.93 (Official Build) (64-bit)
[ ] Chrome (Android) version XX
[ ] Chrome (iOS) version XX
[ ] Firefox version XX
[ ] Safari (desktop) version XX
[ ] Safari (iOS) version XX
[ ] IE version XX
[ ] Edge version XX

For Tooling issues:
Others:
Can I make a PR for this? -- I believe it could be remedied by updating NgEntityServiceLoader's dispatcher to a ReplaySubject. Although I'm not sure if there's a case where that would cause negative regression. Could this possibly be toggled (ReplaySubject or Subject) behind a configuration property?

I see what you mean. Let's make it configurable, and will make it the default in the next major version.

Sounds good! I have changes on a feature branch but I'm unauthorized when I try to push. Do I need permissions or am I missing something on my end?
unable to access 'https://github.com/datorama/akita.git/': The requested URL returned error: 403

https://jarv.is/notes/how-to-pull-request-fork-github/


Make sure drive letter in path will match with actual system drive

What's the Problem?
In properties window under Customize tab in the path to icon file, it doesn't use drive letter where my system is actually installed which in my case is S: but the path that is there by default is C:\Windows\System32\SHELL32.dll and it results for nothing to show there until I select correct path (S:\Windows\System32\SHELL32.dll)

Solution/Idea
This window in regular explorer has path %SystemRoot%\System32\shell32.DLL, so an idea is basically having this path in this window in Files as well, which will result for icons to show up when you open that window without a need to select icon file in case if you have system installed not on C: drive.
Alternatives
I don't think there can be any alternatives to it
Priorities

Capability
Priority

Make sure drive letter in path will match with actual system drive
Could

Files Version
Version: <IP_ADDRESS> OS Version: 10.0.19044.1739
Windows Version
Windows 10 21H2 19044.1739
Comments
No response

@yaichenbaum

Asking @yaichenbaum for approval.

This issue should be resolved with #9748


matcha: 2019-06-22 -> 2019-07

Motivation for this change
Update to version 2019-07
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build matcha


images router wrong when baseUrl has subpath like http://x.x.x./me

images in markdown must write like ![alt](/me/images/language/x.x.x.x.png), but /me should not be here

It's fixed now!
I added render-hooks for the sub-path(/me).
You can find the hooks file at root/themes/zdoc/layouts/_default/_markup.
I did just a minimum usage. So, you can customize it as much as you want.
like this
{{ .Text | safeHTML }}

You can just use default hooks. It works well anyway


Postgres 12 + Postgis 3

I have realised there's a branch with that stack and as of today CARTO is using those versions within.
How safe is to use that branch? Why is the master branch using old Postgres10?

The https://github.com/sverhoeven/docker-cartodb/tree/py3pg12postgis3 branch is unable to create users, it dies due to non existing raster_overviews table. So it is not usable at the moment.
Carto has https://github.com/CartoDB/cartodb/pull/15479 open PR that will fix the raster_overviews error. Waiting for that PR to be merged.
The master branch is using most of the installation instructions of Carto themselves. They still point to Postgres10

Cool! I will watch that CARTO issue and wait. I was mistaken about the current prod versions used by CARTO.
On the other hand, the installation instructions say it requires PostgreSQL 10+ and as of today, AFAIK, CARTO is deployed with 11.x version and I am pretty sure it's pg12 compatible. The raster tables problems might be PostGIS 3.x to blame as vector and raster are managed by two different extensions now.

The raster_overviews PR has been merged.
The branch now can be build an usable Carto instance. I have tested import, analysis and visualizations, but geocoding is broken at the moment.
The image of py3pg12postgis3 branch has been build and has been pushed to Docker Hub, it can be pulled with docker pull sverhoeven/cartodb:py3pg12postgis3.

:top:
Thanks a lot!


Pretty-print JSON name values

When displaying name values that are valid JSON, we should optionally pretty-print them.

Blocked by https://github.com/namecoin/nmc-rpc-explorer/issues/10


Bug: [2.1.0] DMatrix creation from Arrow-backed pandas Dataframes can trigger (ArrowInvalid: Zero copy conversions not possible with boolean types)

We have a a issue with upgrading XGBoost from 2.0.3 to 2.1.0. We use arrow backed types for our pandas dataframes, and if there are boolean columns (1 bit per element, saves memory for our pandas manipulations), we can't create DMatrix form the dataframe.
Repro code:

import pandas as pd
import pyarrow as pa
import xgboost

tab1 = pa.table({"a": pa.array([1,2,3,4]), "b":([True, False, False, True])})
df1 = tab1.to_pandas(types_mapper=pd.ArrowDtype)

xgboost.DMatrix(df1)

In 2.0.3, this constructs a DMatrix, but in 2.1.0, this raises an error ( last few frames of the stack trace):
    605     raise ValueError(f"DataFrame for {meta} cannot have multiple columns")
    607 feature_names, feature_types = pandas_feature_info(
    608     data, meta, feature_names, feature_types, enable_categorical
    609 )
--> 611 arrays = pandas_transform_data(data)
    612 return PandasTransformed(arrays), feature_names, feature_types

File <install_path>/python3.11/site-packages/xgboost/data.py:540, in pandas_transform_data(data)
    538     result.append(cat_codes(data[col]))
    539 elif is_pa_ext_dtype(dtype):
--> 540     result.append(pandas_pa_type(data[col]))
    541 elif is_nullable_dtype(dtype):
    542     result.append(nu_type(data[col]))

File <install_path>/python3.11/site-packages/xgboost/data.py:468, in pandas_pa_type(ser)
    461 zero_copy = chunk.null_count == 0
    462 # Alternately, we can use chunk.buffers(), which returns a list of buffers and
    463 # we need to concatenate them ourselves.
    464 # FIXME(jiamingy): Is there a better way to access the arrow buffer along with
    465 # its mask?
    466 # Buffers from chunk.buffers() have the address attribute, but don't expose the
    467 # mask.
--> 468 arr: np.ndarray = chunk.to_numpy(zero_copy_only=zero_copy, writable=False)
    469 arr, _ = _ensure_np_dtype(arr, arr.dtype)
    470 return arr

File <install_path>/python3.11/site-packages/pyarrow/array.pxi:1587, in pyarrow.lib.Array.to_numpy()

File <install_path>/python3.11/site-packages/pyarrow/error.pxi:91, in pyarrow.lib.check_status()

ArrowInvalid: Zero copy conversions not possible with boolean types

Environment:
Python 3.11.6
OS                    : Darwin
machine               : arm64
pandas                : 2.2.2
numpy                 : 1.24.4
pyarrow               : 16.1.0

If I try to create a DMatrix directly from the pyarrow table, we get the same "ArrowInvalid: Zero copy conversions not possible with boolean types" in 2.1.0, but in 2.0.3 we get an error
File <installpath>/python3.11/site-packages/xgboost/data.py:1118, in dispatch_data_backend(data, missing, threads, feature_names, feature_types, enable_categorical, data_split_mode)
   1114     return _from_pandas_series(
   1115         data, missing, threads, enable_categorical, feature_names, feature_types
   1116     )
   1117 if _is_arrow(data):
-> 1118     return _from_arrow(
   1119         data, missing, threads, feature_names, feature_types, enable_categorical
   1120     )
   1121 if _has_array_protocol(data):
   1122     array = np.asarray(data)

File <installpath>/python3.11/site-packages/xgboost/data.py:737, in _from_arrow(data, missing, nthread, feature_names, feature_types, enable_categorical)
    732 import pyarrow as pa
    734 if not all(
    735     pa.types.is_integer(t) or pa.types.is_floating(t) for t in data.schema.types
    736 ):
--> 737     raise ValueError(
    738         "Features in dataset can only be integers or floating point number"
    739     )
    740 if enable_categorical:
    741     raise ValueError("categorical data in arrow is not supported yet.")

ValueError: Features in dataset can only be integers or floating point number

Pyarrow bools are packed, and they need to be unpacked from 1-bit bools to 1 byte bools for numpy

Thank you for sharing.
This test https://github.com/dmlc/xgboost/blob/09d32f1f2b37e13f0d56e025ccecddf1fa9db76d/python-package/xgboost/testing/data.py#L206
https://github.com/dmlc/xgboost/blob/09d32f1f2b37e13f0d56e025ccecddf1fa9db76d/tests/python/test_with_pandas.py#L512
is not doing what it's supposed to do.


Open service mesh lab

Description
Added lab for Open Service Mesh
This PR fixes #28
Notes for Reviewers
kubectl annotate osm-system <osm-system> openservicemesh.io/sidecar-injection=enabled found this command in the OSM documentation, if someone could please confirm if this is right?
Signed commits

[ ] Yes, I signed my commits.

@warunicorn19  I will get back to this. Till then you can add someone for review who can provide inputs related to osm?

Sure no problem @suhaniii, I've added @leecalcote as well, let's see what he says.


Web: Add Session handling

ATM we can't handle session ( user-storage ) ... so sad we can't handle errors ;)
Using: https://github.com/gorilla/sessions we could ;)
but it need some work to make it available wherever we might want. (guess its in conjunction with cookies as well ;p )

https://github.com/antonlindstrom/pgstore

Means, new dependency ;)

added wiki: https://github.com/kluthen/upsilon_cities_go/wiki/session_cookies


plugin is not compatible with gradle 8.x

java.outputDir was deprecated in gradle 7.6.x versions and now its removed from 8.x versions.
Current code is using this property, hence plugin is not compatible with 8.x

I am away for an extended time and I am not sure when I can perform the new
release ceremony. I will attempt it remotely although it is a bit complex.
I did not bring a laptop to perform work like this with me.

Resolved as release 3.0.6

Resolved as release 3.0.6

3.0.6 is not available yet. Latest version on https://mvnrepository.com/artifact/com.bmuschko.clover/com.bmuschko.clover.gradle.plugin is still 3.0.3

The Maven Central publication happens asynchonously by the plugin portal.
See latest here
https://plugins.gradle.org/plugin/com.bmuschko.clover

The Maven Central publication happens asynchonously by the plugin portal.
See latest here
https://plugins.gradle.org/plugin/com.bmuschko.clover

Thank you.


fix #1893

Possible null pointer dereference due to return value of called method [TcpMessageProtocolResolver]

Fixes #.
Motivation
located at
org/apache/eventmesh/protocol/cloudevents/resolver/tcp/TcpMessageProtocolResolver.java 65 75
Modifications
Add NPE to check for exceptions.
Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Optimize code


calling methods is not allowed in v-model

Version
3.1.2
Reproduction link
Vue 3  - not working:
sfc.vuejs.org
Vue 2 - working:
https://codepen.io/jkarczm/pen/ZEeNNPW?editors=1010
Steps to reproduce
<input v-model="whatever()">
What is expected?
No compilation error
What is actually happening?
v-model value must be a valid JavaScript member expression.

some improvements were made   in (bc100c5c48b98b6e2eabfa1d50e0d3099ea2a90d )
to cover more valid expressions, but it does not cover all valid member expressions.
like the one you encountered.
<input v-model="what().ever">

if you function does not have dynamic arguments, you can wrap the function in a computed
what = computed( ()=> whatever())

<input v-model="what.ever">

if you do have dynamic arguments, it will complicate things to work around that limitation.

if you function does not have dynamic arguments, you can wrap the function in a computed

Yeah, but if it has and they come from v-for (like in the example in my comment in the mentioned commit) i can't use computed.
I wonder why do we even need this check, vue doesn't check if you have proper expression in v-bind:foo or in mustaches, you're just getting a syntax or runtime error, why it's not enough in v-model?

and here is a possible workaround  if you really need dynamic arguments
https://sfc-vue3.netlify.app/#/gist/c1242a00f99d8a3ed26cb002303dfcca
I originally recommended reducing that to a development warning and not an error.
due to incomplete validation.

Using the return value of a function in v-model used to work in vue 2.
I encountered this problem after upgrading my app to vue 3

Same problem here:

    <input
      type="text"
      v-model="getValuesMarginMin()[0]"
      :class="'entry '+getUserInputStatusClass(getValuesMarginMin()[3])"
      disabled
    />
</td>

,getValuesMarginMin() {       return this.values[0];     },


Added aws s3 data node

Added AWS S3 DataNode

Fixes Avaiga/taipy#536

[x] Implemented a new predefined storage type "s3" in DataNodeConfig, to model data stored on an S3 bucket.
[x] List and design the configuration attributes required.
-    Create S3 datanode.
-    Read stream object from s3 bucket.
-    Write  stream object  to s3 bucket.
[x] Implemented the corresponding S3DataNode inheriting from the parent DataNode class.
[x] Add reference manual documentation.

Todo

[ ] Add reference manual documentation.
[ ] Write file to s3 bucket

Hello @toan-quach  and @jrobinAV , please permit me to tag you directly. For some reason,  I can't seem to request for a review.
Please have a look at my PR.

@Forchapeatl awesome!! Thanks for tagging us! We will review as soon as we can :D Thanks again for your help!

Hello @Forchapeatl.
Thank you very much for your contribution. For information, we are currently working on a significant repository refactoring. We are merging sub-repositories into a single repository. For instance, the taipy-core repository is about to be merged into the taipy repository.
This should not impact your work.
But just so you are not surprised, instead of merging your PR, we will move your code from this branch (in taipy-core) into a new branch of another repository (taipy). We will let you know and let you review the code.
Thank you for your understanding.

Hello @jrobinAV thank you for your kind review. The requested changes have been made. The S3ObjectDataNode has been implemented  and tested with the _read and _write fuctions in thier expected signatures.
The issue now is, the _append method I had a chat with @trgiangdo concerning this  method.  I saw how this was possible when we were adding files to a single bucket.  Now our DataNode is making reference to a single object not bucket. Handling this method should be possible  by copying all the data to a new object, appendng the new content and writing back to S3 , but I don't think it's worth the effort in cases where the object data (file) is very large ( I may be wrong ) . Please for your opinion.
Reference: Append Data to S3 Object

Thank you for your information. _append() is an optional method, so if S3 does not support appending, then it's totally fine.

About the DataNodeConfig, you probably want to take a look at this documentation page.
Then, you can take a look at the src/taipy/core/config/data_node_config.py file, where we define different configuring methods for different data node types, and implement a new configure_s3_data_node() method.
For now, the todo list:

[ ] Implement configure_s3_data_node() method to the DataNodeConfig class
[ ] Add _STORAGE_TYPE_VALUE_S3 attribute and add that attribute to the _ALL_STORAGE_TYPES list.
[ ] Update DataNodeConfig docstring.
[ ] Add configuration tests for S3DataNode that created from an s3 DataNodeConfig.

Hello @jrobinAV thank you for your kind review. The requested changes have been made. The S3ObjectDataNode has been implemented and tested with the _read and _write fuctions in thier expected signatures.
The issue now is the _append method. I had a chat with @trgiangdo concerning this method. I saw how this was possible when we were adding files to a single bucket. Now our DataNode is making reference to a single object not bucket. Handling this method should be possible by copying all the data to a new object, appendng the new content and writing back to S3 , but I don't think it's worth the effort in cases where the object data (file) is very large ( I may be wrong ) . Please for your opinion.
Reference: Append Data to S3 Object

Hello,
Yes, I agree with you. Let's forget abt the append method. It is an optional method anyway.
Thank you so much.

Hello everyone, the requested changes have been made. This includes.

[x]  Implement configure_s3_object_data_node() method to the DataNodeConfig class
[x] Add _STORAGE_TYPE_VALUE_S3 attribute and add that attribute to the _ALL_STORAGE_TYPES list.
[x]  Add configuration tests for S3ObjectDataNode that created from an s3_object DataNodeConfig.
[x] Update DataNodeConfig docstring.( I don't really know what this means but i believe it's been covered ).

Please let me know if more changes are required.

#tests3.py
import taipy as tp
import boto3
from taipy import Config, Core

def build_message(s3_object,name):
    return f"Hello {name}!"

 ,WHEN! Who?")
    ans = myS3ObjectDataNode.read()
    print(ans)

    hello_scenario.input_name.write("Taipy")
    hello_scenario.submit()

Hello everyone, things seem to be woring fine now. The S3ObjectDataNode does get created. However it seems to be failing storage type test.

Is this a file or variablecaplog.text ? Please for more information on this.
My tests failes on these lines:
https://github.com/Forchapeatl/taipy-core/blob/57b29061626186aa4839528f8eaf81baac15cf5e/tests/core/test_core.py#L35
https://github.com/Forchapeatl/taipy-core/blob/57b29061626186aa4839528f8eaf81baac15cf5e/tests/core/config/test_data_node_config.py#L132
https://github.com/Forchapeatl/taipy-core/blob/57b29061626186aa4839528f8eaf81baac15cf5e/tests/core/config/checkers/test_data_node_config_checker.py#L94
Please bear with me on the tests3.py code sample. The docs seems advanced for me.

The failed tests are because you updated the error message with new storage_type, but the test expected_error_message on the tests are not updated.
A small update should fix the problem.

Wow, thankyou @trgiangdo

I believe it has the same problem. A small update on the expected_error_message should fix it.

Passed all 1108 tests. I belive s3ObjectDataNode has been succesfully added.

The tests are failing:

On installation tests:

File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/taipy/core/data/aws_s3.py", line 13, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'

On backend tests:

>       assert expected_error_message in caplog.text
E       assert '`storage_type` field of DataNodeConfig `data_nodes` must be either csv, sql_table, sql, mongo_collection, pickle, excel, generic, json, parquet, s3_object, or in_memory. Current value of property `storage_type` is "bar".' in 'ERROR    Taipy:config.py:237 `storage_type` field of DataNodeConfig `data_nodes` must be either csv, sql_table, sql, ..., pickle, excel, generic, json, parquet, in_memory, or s3_object. Current value of property `storage_type` is "bar".\n'
E        +  where 'ERROR    Taipy:config.py:237 `storage_type` field of DataNodeConfig `data_nodes` must be either csv, sql_table, sql, ..., pickle, excel, generic, json, parquet, in_memory, or s3_object. Current value of property `storage_type` is "bar".\n' = <_pytest.logging.LogCaptureFixture object at 0x118d6c970>.text

The parquet, s3_object, or in_memory is different from parquet, in_memory, or s3_object
Can you check again?

The tests are up and running. That's great 😁
Can you take a look at the linter error in https://github.com/Avaiga/taipy-core/actions/runs/7102562575/job/19333215991?pr=828?
This is so that we have consistent code style in our project.

The tests are up and running. That's great 😁

Thank you. The code should be working fine now.

Well :) this is unexpected , @trgiangdo , I did run the pylint test as you said . I am abit reluctant to fix this error because , I am unable to reproduce the error on my environment , let me know what you think.

About the error
Running: pycodestyle --max-line-length=120 --ignore=E121,E123,E126,E226,E24,E704,W503,W504,E203 .
./tests/core/data/test_aws_s3_data_node.py:56:62: E231 missing whitespace after ','
./tests/core/data/test_aws_s3_data_node.py:76:61: E231 missing whitespace after ','

You just need to add a space after the comma in line 56 and line 76 and it should works

Hello @Forchapeatl.
FYI, this taipy-core repository is merged into the taipy repository and is going to be archived soon.
So, instead of merging your PR, can you create a new PR in the main taipy repository, so you can be counted as an official Taipy contributor by Github?
The content of the PR should be very similar.

Migrated to https://github.com/Avaiga/taipy/pull/585


SkiaSharp, SVG & Windows

Hi,
I have a project targeting macOS, windows, and eventually iOS & Android.
I needed to display some svg icons and was looking for a possible solution.
Therefore I ended up trying to use SkiaSharp.
I managed to get a basic drawing works on Android but when I run the windows project I get an exception stating that it cannot find the assembly SkiaSharp.Views.Forms even though I added it through nuget.
I started to gather clues that it might not work on Windows applications, but reading through it I found this article from Matthew saying:

"SkiaSharp can be used to render images on many platforms, including iOS, macOS, tvOS, Android, and Universal Windows Platform (UWP). Not only does SkiaSharp support native mobile apps, it also supports .NET Standard 1.3 and .NET Core on macOS and Windows, as well as Classic Windows Desktop and WPF applications."
(https://devblogs.microsoft.com/xamarin/drawing-with-skiasharp/)

This is an article dated back from 2017 for Xamarin, but I don't see why it wouldn't apply here as this is supposed to be just a wrapper around it.
My questions:

What is it? Should I continue trying to make it work or is this pointless?
Is there any other alternative to display svg images on all platforms?

I don't think I've tried SkiaSharp in WPF so I'll try to see what's going on.
Tagging @lachlanwgordon who did the original SkiaSharp work in case he has any idea.

Hi @PI2R, this should work but I haven't tested it so I'll take a look.
Are you using UWP or WPF?

If it's WPF I think I've found the issue, you also need to add the package SkiaSharp.Views.Forms.WPF to your WPF project, similar to how you need to have Xamarin.Forms.Platform.WPF in the WPF project even though you don't need the equivalent on other platforms.

Hi @lachlanwgordon
It is the default one created by the template, so unless I am going crazy (or crazier) this should be WPF :-)
Looking at the issue I already found on some Xamarin issues the fact that a special WPF package was need for SkiaSharp.
Unfortunately I installed SkiaSharp.Views..WPF instead of SkiaSharp.Views.Forms.WPF...
I tried to install the one you suggested and indeed it worked! Thanks a lot

I've updated the docs on this to add a note for WPF projects: https://github.com/MicrosoftDocs/mobile-blazor-bindings/pull/33. It should go live at https://docs.microsoft.com/mobile-blazor-bindings/walkthroughs/skiasharp shortly.
@PI2R  thank you for reporting this!
@lachlanwgordon thank you for assisting!

I've updated the docs on this to add a note for WPF projects: https://github.com/MicrosoftDocs/mobile-blazor-bindings/pull/33. It should go live at https://docs.microsoft.com/mobile-blazor-bindings/walkthroughs/skiasharp shortly.
@PI2R  thank you for reporting this!
@lachlanwgordon thank you for assisting!

Hi @Eilon
Thank you for updating the docs.
There was a test I wanted to do and that I just did when I saw you updated the doc.
In fact it seems I was half right and that SkiaSharp.Views.WPF is also required.
Indeed when @lachlanwgordon suggested to add the nuget package I did it without removing the one I added before.
When I just tried to remove it it stopped working again.

Hi @Eilon
Thank you for updating the docs.
There was a test I wanted to do and that I just did when I saw you updated the doc.
In fact it seems I was half right and that SkiaSharp.Views.WPF is also required.
Indeed when @lachlanwgordon suggested to add the nuget package I did it without removing the one I added before.
When I just tried to remove it it stopped working again.

Ah interesting I will update the docs to reflect that, thank you for the info!

Ah interesting I will update the docs to reflect that, thank you for the info!

Also while you're at it, I just tried on macOS, and adding package SkiaSharp.Views.Forms allowed me to draw the same canvas.
You might want to add macOS to your bullet iOS and Android.

Also while you're at it, I just tried on macOS, and adding package SkiaSharp.Views.Forms allowed me to draw the same canvas.
You might want to add macOS to your bullet iOS and Android.

Thank you once again @PI2R , the docs are updated now: https://github.com/MicrosoftDocs/mobile-blazor-bindings/pull/34 (will be live soon)

Thank you once again @PI2R , the docs are updated now: https://github.com/MicrosoftDocs/mobile-blazor-bindings/pull/34 (will be live soon)

I saw you added WIndows as well to the list of project types that should have SkiaSharp.Views.Forms installed.
It does not hurt and I don't think it is worth you waste time on this, but FYI my tests worked without it. Only the 2 "SkiaSharp.*.WPF" packages were needed.
Just to be sure I installed this extra package to the windows project and it did not seem to cause any more troubles to have it.
And maybe you're right it could be needed for more complex SkiaSharp scenarios.

I saw you added WIndows as well to the list of project types that should have SkiaSharp.Views.Forms installed.
It does not hurt and I don't think it is worth you waste time on this, but FYI my tests worked without it. Only the 2 "SkiaSharp.*.WPF" packages were needed.
Just to be sure I installed this extra package to the windows project and it did not seem to cause any more troubles to have it.
And maybe you're right it could be needed for more complex SkiaSharp scenarios.

Yeah I chose to include it "just in case" even if it might not be necessary. At worst it's harmless 😄

Yeah I chose to include it "just in case" even if it might not be necessary. At worst it's harmless 😄


Partial slashing, take 1

This is still fairly early stage and may be bugged, though @karlfloersch would probably be a much better position than me to test that at this point.
Basically, it keeps track of the portion of validators that have misbehaved at each link, and slashes based on that percentage.

If we keep track of total_slashed_penalities in eth for example, we could consider adding something like:
@public
@constant
def get_current_health() -> decimal:
    return total_curdyn_deposits / (total_curdyn_deposits + total_slashed_penalties)

@vbuterin Is this PR still under consideration or should we close?

Partial slashing spec has changed since this PR.
Closing for now.


The 3D printing file size is not suitable for the I3 frame printer, ha ha!

I really like this project and want to DIY one, but my I3 frame 3D printer cannot print OPEBOT files. I have a headache!
I just read the related articles and found that the "speed sensor" has no model. I found it on the Internet and there is no one. Can you tell me an alternative model? Can all modules of this signal be used?
"The slot-shaped through-beam photoelectric, as long as the non-transparent object passes through the slot, it can trigger to output 5V TTL level. Using Schmitt trigger to de-jitter pulse"

Could you please elaborate why you cannot print?
I assume the speed sensor is an unrelated issue. What do you mean by model of the speed sensor? It is an optical encoder and the disk attached to the motor shaft, interrupts the optical signal. By counting the interrupts, the number of rotations can be measured.

您能否详细说明为什么无法打印？
我认为速度传感器是一个无关紧要的问题。速度传感器的型号是什么意思？它是一个光学编码器，并且光盘固定在电机轴上，可中断光学信号。通过计算中断次数，可以测量转数。

Hahaha, what I mean by unprintable means that the size of the object exceeds the size that the I3 printer can print.
Can the speed sensor not be used?

You can try the glueable body. As for the speed sensor, yes it should work. Are you asking about a specific model that's different from the one in the BOM?

您可以尝试可胶粘的身体。至于速度传感器，是的，它应该可以工作。您是否要询问与BOM表中的模型不同的特定模型？

Yes, after your reminder, I noticed that there are close-up models in the BOM! Thank you very much, so you can print, I am preparing electronic components, ready to assemble one by hand, thank you again for sharing!


I want to see all my windows in the exposé

even when I'm a poweruser and have more than 25 windows open. A title for some windows would be enough for me to have an overview of what I am currently working with.

Thanks for your input! Starting with WA 3.3, we introduced paged in Exposé view enabling you to see all open windows even more comfortably :)


Support for Base16 and maybe later also Base64 share secret keys

It can be with a TOTP that a seed is a string of random characters, usually 16-32 characters long.
Currently, as I understand it now only Base32 strings are used as shared secret keys supported for TOTP. Wouldn't it make sense to implement 16-32 random characters and maybe even later Base64 strings as shared secret keys (seeds) as well?
I think this would be helpful too!

Sorry I completely missed your issue.
The secret keys are commonly stored in encoded fashion using Base32, which I don't believe is in the RFC but that is the standard practice when looking at the various OTP generators online.  The only RFC guideline about the secret key itself is that it is must be at least 128 bits in length, but they also recommend it to be 160 bits.


Update tests to use sysmon

Coverage introduced in Coverage.py 7.4.0 the support for sys.monitoring.
While it should not work well with the branch coverage used within warehouse, tests on our computers with @woodruffw showed a significant improvement:
Tests results

With sysmon:

DarkaMaul  : 26.7s
William: 29.36s

Baseline:

DarkaMaul: 56.38s
William: 50.08s

The relevant issue in coverage is https://github.com/nedbat/coveragepy/issues/1812
We would like to gather more feedback before eventually pushing this option forward.
NOTE: The other tracers are ctrace or pytrace
/cc @woodruffw

Definitely cool speedup!
Looks like this has some impact on how contexts are measured?
https://github.com/pypi/warehouse/actions/runs/10685632141/job/29619230896#step:5:236
/opt/warehouse/lib/python3.12/site-packages/coverage/html.py:123: CoverageWarning: No contexts were measured
  self.coverage._warn("No contexts were measured")

We measure contexts here:
https://github.com/pypi/warehouse/blob/6ce90e994a0f6fb7298cb500f5d5c41c8be75661/pyproject.toml#L3
This provides visual indicators in the HTML report to show which test functions are hitting a given line of code, making it easier to figure out which tests are calling what.
Is there anything on coverage.py tracking that issue, so we can leave ourselves a link to check in on it later on?

Is there anything on coverage.py tracking that issue, so we can leave ourselves a link to check in on it later on?

I think we should simply monitor coverage CHANGELOG to see when they are supported.
This PR can be either put on hold until coverage changes, or we can use the new setting for CI runs (with the possibility for interested developers to use it locally too).

This PR can be either put on hold until coverage changes, or we can use the new setting for CI runs (with the possibility for interested developers to use it locally too).

I think we'd likely benefit from the speedups where possible.
Can you provide a hook to allow the developer to pass along a flag to revert the behavior? I don't think the current patch allows setting the envvar that passes to docker compose command in the Makefile, so giving the end user some pay of passing down that the COVERAGE_CORE should revert to to ctrace monitor would be good to have.