[RFC] Create Autonomous Lattice Controller

Create Autonomous Lattice Controller
This RFC submits for comment the proposal that we create an autonomous lattice controller responsible for managing and reconciling declarative, lattice-wide deployments.
Summary
The lattice control interface is an imperative interface. Manifest files, as they were originally introduced in the pre-0.5/OTP versions of wasmCloud, are also imperative. Manifest file components and lattice control interface commands are all imperative--they instruct a single host to perform a single action. With this API, we can tell hosts to start actors and to stop actors, to start and stop providers, and we can even perform an auction where we ask the entire lattice for a list of hosts containing a set of constraints. The purpose of this RFC is to propose a layer of autonomous operation above the imperative control interface.
Design Detail
The following are preliminary design details for the autonomous lattice controller, which will hereafter simply be called the lattice controller. The proposed design includes the creation of an application that is deployed into a single lattice. This application will be responsible for monitoring and interrogating the observed state of a lattice and, through the use of imperative APIs and other controls at its disposal, issuing the appropriate commands to reconcile the gap between observed state and desired state.
In the case of the lattice controller, the desired state is declared through a set of deployments. As each new deployment is submitted to the lattice controller, it will validate the deployment declaration and then begin managing that deployment through a control loop that consists of comparing the observed state against the desired state and issuing the appropriate commands to reconcile.
Observing the Lattice
Generating a cohesive view of observed state involves a number of resources, including:

Lattice Event Stream - events emitted by hosts on the wasmbus.evt.{prefix} subject contain a number of events that, when applied to aggregates, can produce state. For example, events like actor_started and actor_stopped can be applied to an actor aggregate to update the actor's current state
Heartbeats - all hosts within a lattice emit heartbeats to the lattice. Each of these heartbeats contains an "inventory" of everything that's running within each host. This information can be used as "snapshot" style data that can be compared against computed state to compensate for things like message loss of individual events.
Lattice Cache - every time a link definition, OCI reference map, or claims token is published into the durable lattice cache stream, consumers can be notified of these changes and thus compute the state of the distributed cache.
Interrogations - periodically, the lattice controller can interrogate hosts or the lattice at large in order to confirm or refute observations about the state of managed deployments.

Deployment Definitions (Desired State)
A deployment definition contains the following key pieces of information:

A unique name
A list of desired actors and their instance counts

A description of the desired deployment spread of those actors

A list of desired capability providers and their instance counts

A description of the desired deployment spread of those providers

Deployments explicitly do not contain link definitions. Link definitions are entities created by operations for runtime actor configuration, and that configuration persists regardless of the number of instances of entities like actors and providers.
Deployment Spreads
A deployment spread is a definition for how a deployment should spread a given entity across the available hosts. Spreads are defined by ratios and a set of label-value pairs that define the constraint for that ratio. For example, if you wanted to ensure that your lattice always had 3 copies of the HTTP Server Provider running, and that 66% of those instances must be running on hosts tagged with zone1, and 33% of them must be running on hosts tagged with zone2 (a weighted failover scenario), you might define your spread as follows:

0.66 - zone == zone1
0.34 - zone == zone2

With such a spread definition, and an instance requirement of 3, the lattice controller would always attempt to ensure that you have 1 instance of the provider running in zone2 and 2 instances of the provider running in zone1. If the lattice controller determined that the available resources in the lattice don't support the desired spread (e.g. there's only one host listed with the zone:zone1 label), then a fallback policy would be used to either choose a random host (which could also fail if that host is already running that provider) or give up and only partially satisfy the deployment, which would leave the deployment in a failed state.
The various ratios within a spread must always sum to 1. We may come up with more complex ways of defining spreads in the future, but the core definition of a spread is a ratio applied to a set of constraints. Spreads can be applied to the deployment definitions for actors or providers, though actor spreads are more easily satisfied since more than one instance of the same actor can run within a single host.
Scope of a Deployment
Multiple deployments will co-exist within a single lattice, and the resources used by those deployments can co-exist with "unmanaged" resources. Actors and providers that are started manually will be left alone by the lattice controller (with some potential exceptions, discussed next).
As mentioned, a deployment describes a set of actors and providers. Actors and providers that are deployed by a lattice controller will be tagged as such, and the controller will manage only those resources that are part of a deployment.
Dealing with Overlapping Resources
As a general rule, deployments will never interfere with resources that either belong to different deployments or are unmanaged. However, there are a few exceptions that stem from the fact that capability providers are designed to be reused by many actors and a single host cannot contain multiple instances of the same provider (at least not until we get WASI-based providers). These exceptions are:

If, in order to satisfy a provider spread, the only option is to reuse an existing provider from another deployment, the controller will tag it for the additional deployment. Note that this does not violate multi-tenancy rules or security.
If, in order to satisfy a provider spread, the controller wants to reduce the number of running instances, it will first look for claimed providers from other deployments and simply un-tag those before requesting providers are terminated.
The controller will never request the termination of a capability provider for which it is not the sole manager (that provider is both managed and is only tagged with the given deployment)
If the reason that a provider is solely tagged with a given deployment is that it was previously claimed by that deployment, when instance reduction is required, the provider will be untagged rather than terminated. In other words, a previously unmanaged provider that was claimed to satisfy a deployment will be unclaimed, not terminated, when the deployment is done with it.

Controllers will never claim or reuse unmanaged actors or actors tagged with a different deployment.
Managing Scale (Large Lattices/Many Deployments)
If the lattice controller is ever deployed into a lattice where the rate of change events becomes so high that it cannot process the state change computations and desired imperative command outputs fast enough, then one of two solutions is recommended:

Split the lattice. If the throughput of the event stream from a single lattice is so high that a single lattice controller cannot keep up with it, then the lattice itself might be too large. Examine the communication patterns between components and see if a lattice split makes sense.
Scale the controller. The lattice controller is designed to be run as a BEAM cluster. In the 99% case, it's essentially deployed as a singleton, or a "cluster of 1", but if lattice volume becomes incredibly high, one solution might be to simply run more clustered instances of the lattice controller. The controller will distribute deployment management components across the cluster and will never issue duplicate instructions to the lattice control interface API as a result of being clustered. In other words, it's safe to scale the lattice controller and it will still manage its work queues in an idempotent fashion.

Interacting with the Lattice Controller
The lattice controller will expose an API through which deployment definitions can be submitted. Deployments are immutable, and as such, every time you publish a new deployment with a given name, that deployment is given a monotonically increasing revision number.
You can also use the lattice controller's API to query the observed state of the lattice, which can be very handy when building third-party tooling or simply performing routine troubleshooting tasks while doing development and testing.
Deployment Rollbacks
The lattice controller API will also support the ability to roll back a deployment, which is done by specifying the revision number to roll back to.
Drawbacks
The following is a list of identified drawbacks to implementing this RFC as described herein.
Is this just k8s for wasm?
Very early on, wasmCloud drew a line in the sand saying that it would not be responsible for scheduling the host process. It is entirely up to the wasmCloud consumer to figure out how and where and when they would like to start the host process. Once the process is running, we do indeed provide a way to manipulate the contents of the host (through the lattice control interface API).
One possible drawback to this that comes to mind is are we just re-inventing Kubernetes, but for wasm? While Kubernetes does allow the scheduling of far more resources than wasmCloud deployments (which only schedule 2 resource types), a devil's advocate could suggest that we're reinventing a wheel here.
Rationale and Alternatives
The main rationale for this approach is that we want to combine our desire for declarative, simple, "self-managing" deployments with our desire to remain compatible with, not competing against an entire ecosystem of tooling and available to the entire CNCF ecosystem without lock-in. The following is an itemization of some alternative approaches that were considered.
Tight Coupling with Kubernetes
One alternative to building our own lattice controller would be to simply bundle up all of the state observance, reconciliation, and deployment storage logic and stuff it inside a Kubernetes operator. While this approach might leverage more of the existing functionality of Kubernetes as a platform, this approach also prevents this capability from being used by anything other than kubernetes. Does our desire to enable first and third party tooling to manage declarative deployments offset the "isn't this just k8s for wasm" argument? We feel that we can accomplish much more by creating the lattice controller and then exposing the controller's API to thin veneer tooling like a Kubernetes operator than by embedding all of the functionality inside the black box of a "fat operator"

Multiple deployments will co-exist within a single lattice

YES! We identified that we would need this if we wanted to run a wasmcloud operator in each kubernetes cluster. I was hoping that this would be included. 😁

Deployment Spreads
How should this interact with external services like http load balancers?

If the reason that a provider is solely tagged with a given deployment is that it was previously claimed by that deployment, when instance reduction is required, the provider will be untagged rather than terminated. In other words, a previously unmanaged provider that was claimed to satisfy a deployment will be unclaimed, not terminated, when the deployment is done with it.

I haven't quite wrapped my head around overlapping resources yet (I hadn't even considered it as a possibility when thinking though the requirements for a wash ctl apply.
As a worked example, What happens here?:
Deployment A.1 => 10 providers plz => 10 providers claimed by A
Deployment B.1 => 10 providers plz => 10 providers claimed by A and tagged by B
Deployment A.2 => 0 providers plz => ??? 10 providers claimed by nobody and tagged by B
Deployment B.2 => 0 providers plz => ??? 10 zombie providers?

Deployments explicitly do not contain link definitions. Link definitions are entities created by operations for runtime actor configuration, and that configuration persists regardless of the number of instances of entities like actors and providers.

 I recognise that things are improved in -otp, but it feels like we will still need a way to declarative way to specify links.
Could the resource tagging strategy that's being proposed for capability providers also be applied to links?
I also have an impression in my head that links can be constrained to capability providers with a given set of tags, but I suspect I might have dreamed that one. I think you want to be able to specify these two types of link:

make a link between these actors that I control and these capability providers that I control
make a link between all actors of this type and all capability providers of this type (would probably need some kind of reference counting, so that it gets removed when the last controller asking for it goes away?)

I also realise that it's the things you don't build that make a product successful, so I'm happy for this bit to be dumped into the river of dreams if it helps to get the rest out of the door more quickly.

:rocket:
that looks exactly like what we're intending to support and is precisely the scenario I had in mind when using host labels in lattice auctions!
essentially, providers are started when none match a deployment spec. If the spread definition for providers requires more be started, more will be started. If the lattice essentially runs out of room for new providers, then (and only then) will the lattice controller look for reusable providers to claim. Claims are relinquished on providers as the instance count of providers drops/spread specification changes. Only providers that were started by the controller for a given deployment are terminated by that deployment.
but it feels like we will still need a way to declarative way to specify links. Link definitions are already declarative.. they're actually one of the few things you can do with the control interface that is (lattice) global and declarative. For now, until we find a need, the link definition is defined (keyed) as "apply this set of configuration values between all actor instances with public key X and provider instances with contract ID y and link name z"

I think your comment implies that we want to enhance the spread configuration so that you can easily specify not just a ratio by which instances are pulled from a fixed instance count, but a way to define a spread that dictates that you want 1 per matching host, regardless of the number of instances that means. This doesn't make much sense for actors, but, for server-class capability providers, makes total sense.

not have kuberntes kill my pods at random because of failed http health checks, because they're not running the capability provider.

The OTP version has /livez and /readyz endpoints, both of which reflect the liveness and readiness of the host, and so will not get your hosts/pods killed because they're not running the HTTP provider.

This is implemented by https://github.com/wasmCloud/wadm and further iteration will be tracked there


chore: update Twitter link and labels to X branding

Changed href in GetConnectedButton from Twitter to X URL (https://x.com/base)
Updated title and aria-label to reflect "Join us on X" instead of "Join us on Twitter"
Kept iconName and eventName as "twitter" to ensure compatibility with existing icons and events

🟡 Heimdall Review Status

Requirement
Status
More Info

Reviews

🟡
0/1

Denominator calculation

Show calculation

1 if user is bot
0

1 if user is external
0

From .codeflow.yml

1

Additional review requirements

Show calculation

Max
0

0

From CODEOWNERS

0

Global minimum
0

Max

1

1

1 if commit is unverified
0

Sum
1


The apiserver refuses to work when first etcd-server is not available.

Bug Report
What happened:
As the issue's title!
What you expected to happen:
The etcd cluster can be work, the apiserver should be work too!
How to reproduce it (as minimally and precisely as possible):
I have a etcd cluster of three node, and i stoped the first node etcd-0:
[root@master-0 service-catalog]# alias
alias etcdctl='ETCDCTL_API=3 etcdctl --cacert /etc/etcd/pki/ca.pem --cert /etc/etcd/pki/client.pem --key /etc/etcd/pki/client-key.pem --endpoints=[etcd-0:2379,etcd-1:2379,etcd-2:2379]'

[root@master-0 service-catalog]# etcdctl member list -w table
+------------------+---------+--------+---------------------+---------------------+
|        ID        | STATUS  |  NAME  |     PEER ADDRS      |    CLIENT ADDRS     |
+------------------+---------+--------+---------------------+---------------------+
|  f0f92d99677353f | started | etcd-0 | https://etcd-0:2380 | https://etcd-0:2379 |
| 1a147ce6336081c1 | started | etcd-1 | https://etcd-1:2380 | https://etcd-1:2379 |
| ed2c681b974a3802 | started | etcd-2 | https://etcd-2:2380 | https://etcd-2:2379 |
+------------------+---------+--------+---------------------+---------------------+

[root@master-0 service-catalog]#  etcdctl endpoint status -w table
Failed to get the status of endpoint etcd-0:2379 (context deadline exceeded)
+-------------+------------------+---------+---------+-----------+-----------+------------+
|  ENDPOINT   |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-------------+------------------+---------+---------+-----------+-----------+------------+
| etcd-1:2379 | 1a147ce6336081c1 |  3.2.18 |   20 MB |      true |        19 |    2028364 |
| etcd-2:2379 | ed2c681b974a3802 |  3.2.18 |   20 MB |     false |        19 |    2028364 |
+-------------+------------------+---------+---------+-----------+-----------+------------+

And then, apiserver refuses to work:
[root@master-0 service-catalog]# kubectl get clusterservicebrokers.servicecatalog.k8s.io
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get clusterservicebrokers.servicecatalog.k8s.io)

[root@master-0 service-catalog]# kubectl get pods -n ns-support
NAME                                                READY   STATUS    RESTARTS   AGE
catalog-catalog-apiserver-8696574c4c-gn5rf          1/1     Running   0          7m52s
catalog-catalog-controller-manager-89f68bf9-flrpp   0/1     Running   0          6d1h

[root@master-0 service-catalog]# kubectl get cs
NAME                 STATUS      MESSAGE                                                                                ERROR
etcd-0               Unhealthy   Get https://etcd-0:2379/health: dial tcp <IP_ADDRESS>:2379: connect: connection refused
scheduler            Healthy     ok
controller-manager   Healthy     ok
etcd-2               Healthy     {"health": "true"}
etcd-1               Healthy     {"health": "true"}

Anything else we need to know?:
Ref: https://github.com/kubernetes/kubernetes/issues/72102
Environment:

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

service-catalog version:
catalog-0.1.38.tgz, quay.io/kubernetes-service-catalog/service-catalog:v0.2.0

Cloud provider or hardware configuration:

[root@master-1 ~]# kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
master-0   Ready    master   6d19h   v1.13.0   <IP_ADDRESS>   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
master-1   Ready    master   6d19h   v1.13.0   <IP_ADDRESS>   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
master-2   Ready    master   6d19h   v1.13.0   <IP_ADDRESS>   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
worker-0   Ready    <none>   6d19h   v1.13.0   <IP_ADDRESS>   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
worker-1   Ready    <none>   6d19h   v1.13.0   <IP_ADDRESS>   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0

Do you have api aggregation enabled?

Do you see the configmap in kube-system?
Does it have all the necessary fields?

kubectl get cm -n kube-system extension-apiserver-authentication -o yaml and look for requestheader-XXX fields

[root@master-0 service-catalog]# kubectl get cm -n kube-system extension-apiserver-authentication -o yaml | grep requestheader
  requestheader-allowed-names: '["front-proxy-client"]'
  requestheader-client-ca-file: |
  requestheader-extra-headers-prefix: '["X-Remote-Extra-"]'
  requestheader-group-headers: '["X-Remote-Group"]'
  requestheader-username-headers: '["X-Remote-User"]'

Install tools:

Did you use helm? What were the helm arguments? Did you --set any extra values?

imagePullPolicy: IfNotPresent
apiserver:
  storage:
    etcd:
      useEmbedded: false
      tls:
        enabled: true
        clientCertSecretName: catalog-etcd-secret
      servers: https://<IP_ADDRESS>:2379,https://<IP_ADDRESS>:2379,https://<IP_ADDRESS>:2379
      imagePullPolicy: IfNotPresent
      persistence:
        enabled: true
        storageClass: glusterfs-storage-class
      resources:
        requests:
          cpu: 100m
          memory: 256Mi
        limits:
          cpu: 500m
          memory: 512Mi
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 1000m
      memory: 1024Mi
controllerManager:
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 1000m
      memory: 1024Mi

Are you trying to use ALPHA features? Did you enable them?

We use the same apimachinery, so it's up to upstream to fix. We'll pull it in some time after the official release.
kubernetes/kubernetes#72102
/assign @jberkhahn

@MHBauer Yes, thank you for your commented!
For now, I can used a VIP(generated via keepalived) to around this issue.


Create directories when a path is specified in the --output option

Fixes #512

Coverage remained the same at 81.649% when pulling c56fc4ab9db9a675924e58e3bbb9425f89aefa6c on yieldlab:create-directories into bf57c4d14b3cd71937c69bdeb1056acecf80ebb4 on Rebilly:master.

The Travis build failed because a record key for cypress is missing:
Opening Cypress...
You passed the --record flag but did not provide us your Record Key.
You can pass us your Record Key like this:
  cypress run --record --key <record_key>
You can also set the key as an environment variable with the name CYPRESS_RECORD_KEY.
https://on.cypress.io/how-do-i-record-runs
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>e2e-ci: `cypress run --record`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>e2e-ci script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!

LGTM 👍
Thanks! I will release a new version of CLI later today


Doc markdown needs to use "bash" for the code type

The doc/troubleshooting/ts_security.md doc contains the following snippet:

``` sh-session
2012-02-29 10:03:56,305 INFO   session.revalidate -
```

The "sh-session" code descriptor isn't recognizable by Jekyll. But "bash" is. I believe this is also recognizable by GitHub. From https://help.github.com/articles/github-flavored-markdown/:

You can find out which keywords are valid by perusing the languages YAML file.

pull request: https://github.com/kaazing/gateway/pull/140


Slow performance

Do some benchmarking to see what COG settings have an effect on the performance and show how can we prepare a COG to have the best performance possible.

When a cog has not enough overviews, rio-tiler/rasterio needs to request multiple tiles which can get out of hand and can result in requests taking minutes to be handled.
With this commit:
https://github.com/sogelink-research/ctod/commit/c435d06e55fbed71a5cfbcb1ba5375986930be1a we introduce a check if it's safe to load a tile based on the requested tile and dataset downsampling. When it is not safe to load a tile it will be skipped and return an empty tile and a warning will be displayed to generate more overviews.
2024-01-22 13:00:52 | WARNING | Skipping https://data.bev.gv.at/download/ALS/DSM/20220915/ALS_DSM_CRS3035RES50000mN2650000E4750000.tif (6, 69, 16), tile is not safe to load, generate more overviews

Impact of overviews/projection/tms in COG is currently being tested.

any recommendations for a known-good gdaladdo argument list?
I used
gdaladdo -r average input.tif 2 4 8 16

which worked - anything to improve here?

We are still testing some things but projecting to 4326 and using the same tiling schema as used by Cesium seems to performs nicely.
Warp to 4326
gdalwarp -t_srs EPSG:4326 R_09BN2.TIF R_09BN2_4326.TIF

Create COG
rio cogeo create R_09BN2_4326.TIF R_09BN2_4326_cog.tif --cog-profile deflate --blocksize 256 --overview-blocksize 256 --tms ./WGS1984Quad.json

However when we profile a tile request using (a slightly modified) tilebench we expect 2 GET request: header+data but it reports 4 requests. This is possibly because the internal tiles do not perfectly align correctly with the tile bounds.
tilebench profile http://localhost:3000/R_09BN2_4326_cog.tif --tile 17-134586-26855 --add-kernels | jq

{
  "HEAD": {
    "count": 1
  },
  "GET": {
    "count": 4,
    "bytes": 632003,
    "ranges": [
      "0-65535",
      "259176040-259439830",
      "265677443-265980118"
    ]
  },
  "WarpKernels": [],
  "Timing": 0.025652170181274414
}

We also tried a big VRT with 1000's of cogs which doesn't work at the moment.

Opened issue at rio-cogeo: https://github.com/cogeotiff/rio-cogeo/issues/275

I fetched two adjacent ALS DSM's from https://bev.gv.at:
wget https://data.bev.gv.at/download/ALS/DSM/20220915/ALS_DSM_CRS3035RES50000mN2650000E4750000.tif
wget https://data.bev.gv.at/download/ALS/DSM/20220915/ALS_DSM_CRS3035RES50000mN2650000E4700000.tif
reprojected with gdalwarp -t_srs EPSG:4326 <in> <out>
created cog's:
rio cogeo create ALS_DSM_CRS3035RES50000mN2650000E4700000-4326.tif ALS_DSM_CRS3035RES50000mN2650000E4700000-4326-cog.tif  --cog-profile deflate  --blocksize 256 --overview-blocksize 256 --tms  WGS1984Quad.json
The individual *4326-cog.tif files render in the viewer, and perform nicely
however, they are still unsafe?
(venv) BigM1:ctod mah$ python app.py --tile-cache-path ./ctod_cache --logging-level info --port 5001 --unsafe
2024-01-24 10:12:40 | INFO | -----------------------
2024-01-24 10:12:40 | INFO | CTOD Started
2024-01-24 10:12:40 | INFO | -----------------------
2024-01-24 10:12:40 | INFO | Port: 5001
2024-01-24 10:12:40 | INFO | Caching: enabled
2024-01-24 10:12:40 | INFO | Caching Path: ./ctod_cache
2024-01-24 10:12:40 | INFO | -----------------------
2024-01-24 10:13:05 | WARNING | Loading unsafe tile ./ctod/files/ALS_DSM_CRS3035RES50000mN2650000E4750000.tif (2, 4, 0), consider generating more overviews
2024-01-24 10:13:05 | WARNING | Loading unsafe tile ./ctod/files/ALS_DSM_CRS3035RES50000mN2650000E4750000.tif (2, 3, 0), consider generating more overviews
...

create VRT:
(venv) BigM1:files mah$ gdalbuildvrt stmk.vrt *-4326-cog.tif
0...10...20...30...40...50...60...70...80...90...100 - done.

rendering the VRT causes an exception:
ValueError: max() arg is an empty sequence
2024-01-24 10:37:26 | ERROR | Task exception was never retrieved
future: <Task finished name='Task-198' coro=<TerrainFactory._process_completed_cog_request() done, defined at /Users/mah/Ballon/src/ctod/ctod/core/factory/terrain_factory.py:84> exception=ValueError('max() arg is an empty sequence')>
Traceback (most recent call last):
  File "/Users/mah/Ballon/src/ctod/ctod/core/factory/terrain_factory.py", line 85, in _process_completed_cog_request
    await future  # Wait for the completion of the future
    ^^^^^^^^^^^^
  File "/Users/mah/Ballon/src/ctod/ctod/core/cog/cog_request.py", line 56, in download_tile_async
    reader = await self.cog_reader_pool.get_reader(self.cog, self.tms)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mah/Ballon/src/ctod/ctod/core/cog/cog_reader_pool.py", line 39, in get_reader
    reader = CogReader(self, cog, tms, self.unsafe)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mah/Ballon/src/ctod/ctod/core/cog/cog_reader.py", line 21, in __init__
    self._set_safe_level()
  File "/Users/mah/Ballon/src/ctod/ctod/core/cog/cog_reader.py", line 89, in _set_safe_level
    pixels_per_tile_downsampled = 256 * max(self.rio_reader.dataset.overviews(1))

is the ordering correct: gdalwarp -> rio cogeo create -> gdalbuildvrt ?
Versions:
(venv) BigM1:ctod mah$ rio cogeo --version
5.1.1

(venv) BigM1:ctod mah$ rio --version
1.3.9

(venv) BigM1:ctod mah$ gdalwarp --version
GDAL 3.8.3, released 2024/01/04

I can upload artefacts if needed

When not supplying --overview-level to rio cogeo your COG at overview 8 does not fit in 1 tile (259 slightly larger than 1 tile in width) when you request low zoom levels, something between 0-7.
1_4326_cog.tif overviews

    Id      Size           BlockSize     Decimation
    0       66304x45312    256x256       0
    1       33152x22656    256x256       2
    2       16576x11328    256x256       4
    3       8288x5664      256x256       8
    4       4144x2832      256x256       16
    5       2072x1416      256x256       32
    6       1036x708       256x256       64
    7       518x354        256x256       128
    8       259x177        256x256       256

There was also a small error in the unsafe check and i loosened the check a bit, it is fine that a COG request internally requests a few more tiles but without the check it can potentially request 100's of tiles internally. With the update you should not get the warnings anymore for the COG you created without --overview-level
VRT Test
The steps i took to download/warp/create cog and build a vrt file.
wget https://data.bev.gv.at/download/ALS/DSM/20220915/ALS_DSM_CRS3035RES50000mN2650000E4750000.tif
wget https://data.bev.gv.at/download/ALS/DSM/20220915/ALS_DSM_CRS3035RES50000mN2650000E4700000.tif

mv ALS_DSM_CRS3035RES50000mN2650000E4750000.tif 1.tif
mv ALS_DSM_CRS3035RES50000mN2650000E4700000.tif 2.tif

gdalwarp -t_srs EPSG:4326 1.tif 1_4326.tif
gdalwarp -t_srs EPSG:4326 2.tif 2_4326.tif

rio cogeo create 1_4326.tif 1_4326_cog.tif --cog-profile deflate --blocksize 256 --overview-blocksize 256 --tms ./WGS1984Quad.json -w --overview-level 10 --aligned-levels 9 --no-in-memory

rio cogeo create 2_4326.tif 2_4326_cog.tif --cog-profile deflate --blocksize 256 --overview-blocksize 256 --tms ./WGS1984Quad.json -w --overview-level 10 --aligned-levels 9 --no-in-memory

gdalbuildvrt dataset.vrt 1_4326_cog.tif 2_4326_cog.tif

The vrt seems to work fine with everything running locally http://localhost:5000/?cog=http://localhost:3000/dataset.vrt I can see data from both cogs and it perfoms pretty great. Warning: I'm not sure how optimal above creation options are and still looking into many cogs in the vrt file.
both cogs loaded using the vrt

thanks, really appreciated!

I converted all 55 DSM tiles for Austria from source above, and made them available here - total of 292GB, plus a relative VRT - feel free to test, has CORS headers (NB: I observed temp file sizes above 250GB during the conversion with rio cogeo create)
There's something I cant quite explain:

running ctod on an M1 Mac with 64GB: superfast, very quick zooming in and out
running ctod on a 16GB x86_64 linux box - same config: zoom in/out and the working set of the ctod process goes through the roof (use this link for now, I'll have to disable it eventually)
I dont have a 64GB x86_64 at hand - memory might be the cause but not sure

can you give these cogs a spin on x86_64?

Thanks for testing some things out. I have no idea what is happening internally when using a VRT, when I try to profile getting a tile using tilebench
tilebench profile http://localhost:3000/mhaberler.vrt --tile 19-563250-124861 --add-kernels --tms ./WGS1984Quad.json| jq

It runs for quite a while and I see my network going up to 12MB/s during the whole request and ends in an error. Trying to load it into QGIS freezes my whole system.
rio-tiler mentions: Map Tile reading from VRT might not be efficient if overviews are not present, because GDAL will try to open a lot of files. The vrt reports overviews and there are overviews available in the referenced COG's, so I'm not sure why it tries to load a huge amount of data.
With rio-tiler we should also be able to use https://cogeotiff.github.io/rio-tiler/mosaic/ The thing is that the creation of a mosaic json uses Quadkey which does not work with the TMS we are using. In theory this should still work but is not the most optimal.
I want to try to create a simple bash script to create a json with info on the datasets and in the reader part do my own selection of datasets and feed it to the mosaic reader and see how that works out.

thanks!
re overviews - the current tifs were created with
....
for fn in $*
do
    base=`basename $fn .tif`
    echo warping $base

    gdalwarp -t_srs EPSG:4326 $fn tmp.tif
    echo create cog for $base
    time rio cogeo create tmp.tif $target/${base}-cog.tif  --cog-profile deflate --blocksize 256 --overview-blocksize 256 --tms $wgsquad  -w --overview-leve
l 10 --aligned-levels 9  --no-in-memory
    rm -f tmp.tif
done
gdalbuildvrt bev-als-dsm.vrt $target/*.tif

prepared to retry with other suggestions - all of Austria takes over a day though

@mhaberler , it seems that gdal->rasterio doesn't treats the tiffs as COGS when you don't add  /vsicurl/ in front of every url, and hence tries to read every tiff in its entirety before producing tiles.
So the fix for a slow or non-performing vrt might be to make every https://etcetera into /vsicurl/https://etcetera
I tested this for your Austrian data and it works for me at reasonable speed now with CTOD.

wow.. great find! where did you dig that out?
so I assume in that case cog's overviews are ignored - that explains what I see
in that case the more interesting topic will be why this really zips on the Mac M1.. will have a look
should have results to report in a few days.
ps: plus my server being added an 8TB ssd shortly so there's leeway for experimentation :)

in that case the more interesting topic will be why this really zips on the Mac M1.. will have a look
sounds like it could be different versions of gdal. Would be interesting to find out, you might dig into gdal changelogs...

will check and report

step1:
upgraded GDAL by selecively installing from testing, as it's a bookworm/stable production box
GDAL now at  3.8.3
vsicurl change as suggested above not yet applied
result:
working set signifcantly reduced, usability sort of ok in in-cache regions
fluidity still far off from Mac M1 experience
vsicurl change is next

@TomvanTilburg-geodan do you have a syntax example how to change the .vrt file ?
or a command line example for gdalbuildvrt ?
my current (relative) .vrt has references like so:
      <SourceFilename relativeToVRT="0">/junk/www/static.mah.priv.at/cors/terrain/DSM/AT-1m/ALS_DSM_CRS3035RES50000mN2550000E4650000-cog.tif</SourceFilename>

@TomvanTilburg-geodan do you have a syntax example how to change the .vrt file ?

I just used a search and replace in my favourite text editor ;-)
My plan is to automize this via a bash script.

In the latest CTOD version I check if a given VRT should be loaded from an URL and if so add url's to the referenced files automatically, it also adds /vsicurl/ in front of the URL. So creating a vrt with relative paths should automatically work if the .vrt file is hosted in the same location as the files. When requesting a tile with the file locally (?cog=/home/user/dataset.vrt) it only changes the file references if it references a dataset with http/https by adding /vsicurl/. This way you won't have to do anything special when creating a vrt file.
I was hoping to see some fast country wide terrain using a vrt with these changes but unfortunately I don't see much improvement on my machine, I can still spot heavy network traffic. I also tried updating GDAL from 3.4.1 to 3.6.4. VRT needs still needs some more digging....
I also added support for mosaic. The mosaic should be more performant than the vrt because I'm reusing the readers for the different cogs. Later on mosaic also can be used to calculate on multiple tiff tiles when they overlap, like: highest/lowest/median/average or add priority. CTOD has it's own format for the JSON and does not support mosaicjson-spec since this only works with quadkeys.
a json can for now be created using the script: scripts/create_mosaic.py the extension can be .ctod or .json for it to be treated as a mosaic dataset.
python create_mosaic.py -i /home/user/cogs -o /home/user/cogs/mosaic.ctod --ignore-nodata

In the reader I check which cogs overlap with the requesting tile (this can be multiple files) and rio mosaic combines the data if necessary. When you create cogs and align the tiles it's possible that there will be large bounds around the actual data with nodata values, using the --ignore-nodata the 'real' bounds of the dataset will be used in the json file. When you don't add --ignore-nodata it still works but it's possible 2 tiles are requested where only 1 has data, which lowers the performance a bit.
It looks very promising but I still have to test it out on a large dataset.
To be continued.

I just created a json (mosaic) for the dataset of @mhaberler but since I don't have the files locally i cannot use --ignore-nodata so it uses it's reported bounds from gdalinfo. I can see it trying to use data from 11 different files for 1 tile when zoomed in where it should probably use 1 file. Due to the overlapping data it is pretty slow.
We should check if this is also a reason the vrt file is slow and what is causing the huge nodata around the files? overviews, aligned levels or a combination.
I Downloaded 1 file  with reported extent: 9.84375, 46.40625, 12.65625, 47.8125 and visualized the nodata with red in qgis.

@tebben mail me your public ssh key at haberlerm at gheemail;)

trying the  create_mosaic.py  suggestion now, looks like this will run a while

it's online:

(venv) mah@mah2:~/src/ctod$ python scripts/create_mosaic.py -i /junk/www/static.mah.priv.at/cors/terrain/DSM/AT-1m -o /junk/www/static.mah.priv.at/cors/terrain/DSM/AT-1m/mosaic.ctod --ignore-nodata
Calculating extent excluding nodata, this can take some time: /junk/www/static.mah.priv.at/cors/terrain/DSM/AT-1m/ALS_DSM_CRS3035RES50000mN2650000E4300000-cog.tif
/home/mah/src/ctod/scripts/create_mosaic.py:56: DeprecationWarning: Calling nonzero on 0d arrays is deprecated, as it behaves surprisingly. Use atleast_1d(cond).nonzero() if the old behavior was intended. If the context of this warning is of the form arr[nonzero(cond)], just use arr[cond].
indices = np.where(~data_array_masked.mask)
Calculating extent excluding nodata, this can take some time: /junk/www/static.mah.priv.at/cors/terrain/DSM/AT-1m/ALS_DSM_CRS3035RES50000mN2650000E4650000-cog.tif

....
should we bother about that warning
I restarted the service, not sure how to use this?

Commited some changes
Passing resampling to rio-tiler can lead to requesting much more data behind the scenes and slowing down the performance. Resampling now defaults to None instead of bilinear, this can lead to the terrain being blocky when passing the max zoom of the dataset. Consider setting the correct max zoom for your dataset or use the resamplingMethod with caution.
Max zoom now defaults to 18, most of the datasets don't go past 18 so this is a better default setting.
The geodetic surface normals for empty tiles were incorrect causing dark tiles, should be fixed now.
When loading a vrt or json/ctod dataset from url the refered datasets are now updated with the full path of the url instead of only the domain.
create_mosaic.py script should not throw the deprecated warning anymore.
creation of layer.json for mosaic can now use a local filepath.
@mhaberler
If you update CTOD on your server to the latest version and make the paths in the mosaic absolute, you should be able to use the mosaic file like this:
https://ctod.mah.priv.at/?cog=/junk/www/static.mah.priv.at/cors/terrain/DSM/AT-1m/mosaic.ctod
When you supply a local filepath, the refered datasources are not updated and loaded from the local file system based on the path parameter. When the mosaic file is given as an url, the refered datasources are updated to be loaded over the internet automatically https://ctod.mah.priv.at/?cog=https://static.mah.priv.at/cors/terrain/DSM/AT-1m/mosaic.ctod
Since you have all the data locally on your server you should update the paths in the .ctod file to be relative to the ctod server or use the absolute path. I should update the create_mosaic script to accept some more parameters to set the correct path more easily. For now just do a find and replace: "path": " to "path": "/junk/www/static.mah.priv.at/cors/terrain/DSM/AT-1m/ to make the paths absolute.
A new safe check was added for mosaic, when Cesium is requesting low zoom level tiles, it is possible terrain tiles are composed of multiple cogs, for now this cannot pass the 10 datasets per tile. If you want to work around this you can add --unsafe when starting CTOD or ignore low zoom levels by setting minZoom a bit higher.

followed your instructions, and the link you gave above works as given -  give it a try!
I noted some log messages like so:
2024-02-12T16:05:21.974730+01:00 ssl python[1072853]: 2024-02-12 16:05:21 | WARNING | Too many datasets intersecting with requested tile (7, 138, 30), 13
2024-02-12T16:05:21.980428+01:00 ssl python[1072853]: 2024-02-12 16:05:21 | WARNING | Too many datasets intersecting with requested tile (7, 137, 30), 11
2024-02-12T16:05:21.982170+01:00 ssl python[1072853]: 2024-02-12 16:05:21 | WARNING | Too many datasets intersecting with requested tile (7, 139, 30), 11
2024-02-12T16:05:42.343902+01:00 ssl python[1072853]: 2024-02-12 16:05:42 | WARNING | Too many datasets intersecting with requested tile (5, 34, 7), 45
2024-02-12T16:05:42.345005+01:00 ssl python[1072853]: 2024-02-12 16:05:42 | WARNING | Too many datasets intersecting with requested tile (5, 33, 7), 11
2024-02-12T16:05:44.385049+01:00 ssl python[1072853]: 2024-02-12 16:05:44 | WARNING | Too many datasets intersecting with requested tile (3, 8, 1), 55
2024-02-12T16:05:44.386258+01:00 ssl python[1072853]: 2024-02-12 16:05:44 | WARNING | Too many datasets intersecting with requested tile (2, 4, 0), 55
2024-02-12T16:05:53.691169+01:00 ssl python[1072853]: 2024-02-12 16:05:53 | WARNING | Too many datasets intersecting with requested tile (3, 8, 1), 55
2024-02-12T16:05:53.691976+01:00 ssl python[1072853]: 2024-02-12 16:05:53 | WARNING | Too many datasets intersecting with requested tile (2, 4, 0), 55
2024-02-12T16:05:54.682014+01:00 ssl python[1072853]: 2024-02-12 16:05:54 | WARNING | Too many datasets intersecting with requested tile (1, 2, 0), 55
2024-02-12T16:05:55.054027+01:00 ssl python[1072853]: 2024-02-12 16:05:55 | WARNING | Too many datasets intersecting with requested tile (5, 34, 7), 45
2024-02-12T16:05:55.054226+01:00 ssl python[1072853]: 2024-02-12 16:05:55 | WARNING | Too many datasets intersecting with requested tile (5, 33, 7), 11
2024-02-12T16:05:55.056152+01:00 ssl python[1072853]: 2024-02-12 16:05:55 | WARNING | Too many datasets intersecting with requested tile (4, 17, 3), 46
2024-02-12T16:05:55.056566+01:00 ssl python[1072853]: 2024-02-12 16:05:55 | WARNING | Too many datasets intersecting with requested tile (4, 16, 3), 11

I suspice ctod has a memory leak - vm size grows despite settling after idle periods

I added a wiki page on how to optimise COG's. Later on I will  look into other solutions to boost the performance as stated in the README
Closing this issue for now.


[e2e]: delegator-lowperms tag main default slsa3

Repo: https://github.com/slsa-framework/example-package/tree/v47.0.459
Run: https://github.com/slsa-framework/example-package/actions/runs/11351779052
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.delegator-lowperms.tag.main.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.delegator-lowperms.tag.main.default.slsa3.yml
Trigger: push
Branch: v47.0.459
Date: Tue Oct 15 18:07:20 UTC 2024

Repo: https://github.com/slsa-framework/example-package/tree/v47.0.460
Run: https://github.com/slsa-framework/example-package/actions/runs/11371268300
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.delegator-lowperms.tag.main.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.delegator-lowperms.tag.main.default.slsa3.yml
Trigger: push
Branch: v47.0.460
Date: Wed Oct 16 18:15:41 UTC 2024
Tests are passing now. Closing this issue.


EMA-150168

Getting building on Xcode 13 by renaming await to then_await

Oh, cool, I can use awaitPromise instead.

Also note how in the upstream they just deprecated the old name instead of deleting it.  Not sure if we wan to do that for now?


Webgl style : the src parameter not working with a conditional operator

I have to display a lot of point features (up to 500-600k), so I am using Webgl.
When I use the case operator for color or size properties, I have no problem.
But when I use it for the symbol style source (base64) the operator is not working and it returns a black square instead of the image.
let srcTypeA = 'data:image/png;base64,iVBORw0 ...',
    srcTypeB = 'data:image/png;base64,iVBORw0 ...',
    srcTypeDeafault = 'data:image/png;base64,iVBORw0 ...';

style = {
    symbol: {
        symbolType: 'image',
        src: [
            'case',
            ['==', ['get', 'type'], 'A'],
            srcTypeA ,
            ['==', ['get', 'type'], 'B'],
            srcTypeB,
            srcTypeDeafault
        ],
        color: [
            'case',
            ['<', ['get', 'speed'], 5], '#ff14c3',
            ['between', ['get', 'speed'], 5, 10], '#ffed02',
            '#00ff67'
        ],
        size: 14
    }
};

I have also tried to set the src as a feature property and get it with the get operator, but no more success..
style = {
   ...,
   src: ['get', 'imageSrc'],
   ...
}

How could I do it?

You can only use a single source, not an expression. Here is the code reading your src.
https://github.com/openlayers/openlayers/blob/c0009d5985daf821f38c6014c6945923bc3083c9/src/ol/webgl/ShaderBuilder.js#L547-L549
You can however use subregions from one image as in this example:
https://openlayers.org/en/latest/examples/icon-sprite-webgl.html


Log LLM queries and responses directly to files, to debug the evaluaion logic

Part of #204

Closing since we'll change the way we do logging

Closing since we'll change the way we do logging


deployment failed at "REDIS_PASSWORD" step, see image

This error is consistent across different regions I tried

Thanks for this! This should be fixed by https://github.com/linkedin/feathr/pull/233


refa: migrate to koishi 4.8.1

Sorry I didn't see this before.
Superseded by 942c00e59c78c321b78ef00101d32aebb8714552.


Use the scipy version of linalg.cholesky which consistently raises LinAngError with linearly dependent columns

Description
On MacOS at least, Samples.Gelman_Rubin() does not consistently raise np.LinAlgError when the samples have linearly dependent columns, as was pointed out by @htjb in #285 Using the scipy version appears to fix this
Checklist:

[x] I have performed a self-review of my own code
[x] My code is PEP8 compliant (flake8 anesthetic tests)
[x] My code contains compliant docstrings (pydocstyle --convention=numpy anesthetic)
[x] New and existing unit tests pass locally with my changes (python -m pytest)
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have appropriately incremented the semantic version number in both README.rst and anesthetic/_version.py

Sorry, this PR slipped me by.

On MacOS at least, Samples.Gelman_Rubin() does not consistently raise np.LinAlgError when the samples have linearly dependent columns, as was pointed out by @htjb in #285 Using the scipy version appears to fix this

Yeah, the LinAlgError was always a bit finicky, which is why I ended up repeating the same line three times when one time should have been enough... Do the tests still pass when we reduce it to a single repeated line?

I remember having the opposite issues with the PolyChord macOS CI, where I could run things locally fine but not pass the same tests on github.
I've just tested pythons 3.8-11 locally (M1 Pro macbook pro, macOS 13.3) and this test fails on all of them, and all four are fixed with this numpy->scipy changed.

@lukashergt It doesn't look like there is a macos test with python 3.8? Although I get the error locally with python3.10 on my mac and it passes on github.

This was a deliberate change a few months ago in #249. Tests on windows and macOS take much longer as they are running on virtual machines, so I think it was decided a range of python versions on linux and a single macOS and windows test would be sufficient.

Do the tests still pass when we reduce it to a single repeated line?

Well, that didn't work (see the failed tests after https://github.com/handley-lab/anesthetic/pull/287/commits/da96aea26c2c4a32a4423e131184467a289cd6bc)... That's annoying, now I'm less confident that this is truly resolved. If anybody has an idea how to reliably test for LinAlgErrors, let me know. For now, let's just leave it as is. If a similar issue pops up again, then maybe it's best to get rid of the entire try-except block...
I put the try-except block in there for confused people (aka my past self) that tried this with MCMC samples from a Gibbs chain, where some parameters didn't get updated as often as others, and I initially didn't question the samples but the G-R implementation...

Do the tests still pass when we reduce it to a single repeated line?

Well, that didn't work... It worked for me locally but as you can see the CI failed after da96aea... That's annoying, now I'm less confident that this is truly resolved. If anybody has an idea how to reliably test for LinAlgErrors, let me know. For now, let's just leave it as is. If a similar issue pops up again, then maybe it's best to get rid of the entire try-except block...
I put the try-except block in there for confused people (aka my past self) that tried this with MCMC samples from a Gibbs chain, where some parameters didn't get updated as often as others, and I initially didn't question the samples but the G-R implementation...

Just checking we're on the same page: the test is working correctly, just np.linalg.cholesky successfully computes something and doesn't throw an error. I couldn't get to the bottom of why numpy does this

Yes, numpy (and scipy) don't reliably produce a LinAlgError where they should. It appears that OS, python version, and platform all play a role in this. This should not affect the user, but is annoying when checking for code coverage, because it is difficult to reliably get the LinAlgError.

@lukashergt still happy for me to merge?

@lukashergt still happy for me to merge?

Seems to resolve things for now. If that's the end of it, that's great. My lengthy tangent was more for future reference.
So yes, please feel free to squash and merge. Many thanks.


[i18n] translation to French

Hi !
This is an initial translation to French for Fedidevs Starter Pack
I included a French NOTES-fr.txt containing explanations of the choices I made for some terms or techicalities (like non-breakable space or gender-neutral language)
Also, I think there is a bug in the code ;)
#: mastodon_auth/views.py:300 msgid "Unothorized" #really? ;)

Fantastic work! Thank you so much, @vincib! I'll merge this, but I still need to build a language switcher before the translation shows up on the live site. I'll do my best to get this done over the weekend.
Thanks again! ❤


Authentication and user interface

Is it possible to see the actions that I deployed with wsk from the console with the user interface, how will I do this?
and although I have authenticated myself and entered username: guest, password: auth key according to the information given, my login is not accepted. Where can I get the correct information?

I'm sure you have moved onto something else, but if you look at that whisk auth you see in the middle a colon (:) character.  The part before that is the username and the part after it is the password.


Issue: Maximum call stack size exceeded

Select Folder -> Export to PDF -> .... wait for dom to load forever -> Check console to find the below error.

Outcome: Cant export the folder.
Expectation: Export whole folder into pdf

But rest everything seems to work fine. Notes folder is attached below
TestNotes.zip - There are unlinked pictures in this zip file since its going over github limit of 25MB i removed the pictures.

I added file log, can you replace the local plugin, first locate which note file caused:
better-export-pdf.zip

I dont know how to install the plugin from .zip. I attached my obsidian notes which gave me the error in the first post.
If you try to export this folder (folder), you can reproduce the error on your machine.

@l1xnan It looks like a lot of btoa data in the encodedEmbeds function that was recently merged (possibly caused by multiple nested embeds, or a single large embed with many nodes). An interim fix would be splitting the btoa encoded strings if exceeding a certain length.
Although I think a more permanent fix would be loading the printDoc in the webview without serialising the HTML generated by the MarkdownRenderer.

@l1xnan It looks like a lot of btoa data in the encodedEmbeds function that was recently merged (possibly caused by multiple nested embeds, or a single large embed with many nodes). An interim fix would be splitting the btoa encoded strings if exceeding a certain length or storing the nested nodes in a weakmap.
Although I think a more permanent fix would be loading the printDoc in the webview without serialising the HTML generated by the MarkdownRenderer.

The issue lies in the String.fromCharCode function. When the content is excessively long, it triggers the RangeError: Maximum call stack size exceeded error. Currently, focusing on optimizing the btoa_utf8 function should resolve the issue.

Thanks for the fix @l1xnan . Can you release a minor version ? :3

Thanks for the fix @l1xnan . Can you release a minor version ? :3

I have released a fixed version. Since I couldn't reproduce the issue using your note files, I'm not sure if there are any other issues.

Yup, updated and it works.
But a little slow, i guess.

I have also noticed this slow problem, thought of a few areas that can be improved, and expect to complete it in the next minor release.

Sure, Thankyou. But its unrelated to this thread i guess. This particular issue is resolved :3


ACME Client building on worker instead of master nodes

Do you want to request a feature or report a bug?
Bug
What did you do?
I'm not quite sure if this is a bug but it seems weird to me.
From my understanding only the leader should create an LE Account and try to get certs. In fact as I read it, its currently implemented the other way arround:
https://github.com/containous/traefik/blob/v1.7/acme/acme.go#L131
	listener := func(object cluster.Object) error {
		account := object.(*Account)
		account.Init()
		if !leadership.IsLeader() {
			a.client, err = a.buildACMEClient(account)
			if err != nil {
				log.Errorf("Error building ACME client %+v: %s", object, err.Error())
			}
		}
		return nil

but maybe I'm completely wrong.

What did you expect to see?
Only the leader tries to create ACME account
What did you see instead?
Every node except the Leader is creation an Account. (At least that's what I believe to see).
Output of traefik version: (What version of Traefik are you using?)

(paste your output here)

Hi and thank you for your efforts in tracking down potential problems.
What you missed is that the leader has its own leadershipListener (https://github.com/containous/traefik/blob/v1.7/acme/acme.go#L179) in charge of dealing with certificates (onHost / acme.domains / ...) with its own ACME Client.
The ACME Client that exist in the "non leaders" is to handle the deprecated (but still supported) onDemand option (since we have no guarantee that the leader will be the first to handle the request targeting a new onDemand host).
I will let you confirm that it fixes your concern and close the issue.


vendor: update dependency opencontainers/runc

Hi,
This updates openconatainer/runc (and its deps). Ref. https://github.com/opencontainers/runc/pull/2065, running a version before that fix will result in some strange behavior on Aarch64 (Linux 5.0.X+).
Also related to https://github.com/kata-containers/tests/issues/1726, since because of the same problem (in k8s) kubelet, the node process, is unable to start on AArch64.
Runc diff here: https://github.com/opencontainers/runc/compare/f56b4cbeadc407e715d9b2ba49e62185bd81cef4...f4982d8

Hi @odinuge - Thanks for raising! This looks good - the only thing left to do is to raise an issue and reference it in your commit with Fixes: #XXX as shown here:

https://github.com/kata-containers/community/blob/master/CONTRIBUTING.md#main-patch

Don't worry if the issue is not assigned to you (we don't appear to be able to massage GitHub into allowing new contributors to self-assign issues).

Interestingly, the CI checks didn't catch this since three of the re-vendor commits summarised in your commit message specify a Fixes #XXX of their own :smile:

@odinuge any updates here?

Hi @egernst! I've been quite busy with work the last few days, but I will try do fix it today or tomorrow.

This fixes #596

/test


feat: add TypeScript type definitions

Description:
Adds a tsconfig.json file and some TS type definition packages for improved IDE experience. No changes to build steps are required.
Motivation and Context:
Improve IDE experience with better autocompletion and type hinting.
How Has This Been Tested?
No code was changed.
Checklist:

[x] My code follows the code style of this project.
[x] I have reviewed and updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes where needed.
[x] All new and existing tests passed.
[x] My commits have been squashed into a concise set of changes.

Coverage remained the same at 22.324% when pulling 8d81bebac40ca841b8705fb0d8cf458c0444fea9 on bolatovumar:typescript into 13dcf424c8cd336d9de3bd6337c343964d517dde on LN-Zap:next.

Decided to merge and work with it for a while so that we can better gauge the pros vs the cons

@mrfelton cool, thanks.


Scala RPC Server fully tested with TS RPC Client

@Shanedell has reported that the search function is returning no matches when matches are expected.
I think the best way forward is to run the Scala server with the RPC TC client tests as we do with the C++ server.  Search is being exercised by the TS client tests.  We need to make sure it's working there, then we have a good chance of getting it working on the front-end layer.
The tests will be a combination of sbt run to run the Scala server, then npm test to run the TS client tests.  I can script that in build.sh.

@arosien, now with the release of v0.9.14, we can successfully run sbt run and it will pull the necessary artifacts.  Now when trying to run a test against the Scala RPC server, the server is not staying up.  It binds to port 9000, then exits right away.  Will you look into this?

The scala server shut down because the current code only binds the port and then... doesn't wait for anything else, so it shuts down. Experimenting to instead wait for a signal to shutdown.

I'm not sure if there's an application-level "shut down" signal, so here's a patch that has the Scala rpc server wait for an enter key on stdin to shutdown. We could instead just block and wait for an OS-level signal to shutdown.
diff --git a/src/rpc/server/scala/build.sbt b/src/rpc/server/scala/build.sbt
index f4f71da..1efd6dc 100644
--- a/src/rpc/server/scala/build.sbt
+++ b/src/rpc/server/scala/build.sbt
@@ -46,6 +46,9 @@ startYear := Some(2021)
 publishConfiguration := publishConfiguration.value.withOverwrite(true)
 publishLocalConfiguration := publishLocalConfiguration.value.withOverwrite(true)

+fork := true
+run / connectInput := true
+
 libraryDependencies ++= Seq(
   "com.ctc" %% "omega-edit" % omegaVersion,
   "com.ctc" %% "omega-edit-native" % omegaVersion classifier s"${arch.id}",
diff --git a/src/rpc/server/scala/src/main/scala/com/ctc/omega_edit/grpc/boot.scala b/src/rpc/server/scala/src/main/scala/com/ctc/omega_edit/grpc/boot.scala
index 3551d8f..e02cdaf 100644
--- a/src/rpc/server/scala/src/main/scala/com/ctc/omega_edit/grpc/boot.scala
+++ b/src/rpc/server/scala/src/main/scala/com/ctc/omega_edit/grpc/boot.scala
@@ -19,12 +19,23 @@ package com.ctc.omega_edit.grpc
 import akka.actor.ActorSystem

 import scala.concurrent.ExecutionContext
+import scala.concurrent.duration._
+import scala.io.StdIn

 object boot extends App {
   implicit val sys: ActorSystem = ActorSystem("omega-grpc")
   implicit val ec: ExecutionContext = sys.dispatcher

-  EditorService.bind().foreach { binding =>
-    println(s"gRPC server bound to: ${binding.localAddress}")
-  }
+  for {
+    binding <- EditorService.bind()
+    _ = println(s"gRPC server bound to: ${binding.localAddress}")
+
+    _ = StdIn.readLine("press a key to shut down...\n")
+
+    _ = println(s"shutting down...")
+    _ <- binding.terminate(1.second)
+    _ = println("http terminated")
+    _ <- sys.terminate()
+    _ = println("actor system terminated")
+  } yield ()
 }

@arosien  On v0.9.14 I am also getting this error when trying to create a viewport
gRPC server bound to: /<IP_ADDRESS>:9000
[ERROR] [07/28/2022 20:49:14.846] [omega-grpc-akka.actor.default-dispatcher-4] [akka://omega-grpc/user/$a/L1VzZXJzL3NkZWxsL3dvcmtzcGFjZXMvZGFmZm9kaWwvc2FtcGxlV29ya3NwYWNlLy52c2NvZGUvbGF1bmNoLmpzb24=] 1024 is not a member of ValueEnum (0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512)
java.util.NoSuchElementException: 1024 is not a member of ValueEnum (0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512)
        at enumeratum.values.ValueEnum.$anonfun$withValue$1(ValueEnum.scala:58)
        at scala.Option.getOrElse(Option.scala:201)
        at enumeratum.values.ValueEnum.withValue(ValueEnum.scala:58)
        at enumeratum.values.ValueEnum.withValue$(ValueEnum.scala:57)
        at com.ctc.omega_edit.api.SessionEvent$.withValue(SessionEvent.scala:25)
        at com.ctc.omega_edit.api.SessionCallback.invoke(SessionCallback.scala:40)
        at com.ctc.omega_edit.api.SessionCallback.invoke$(SessionCallback.scala:30)
        at com.ctc.omega_edit.api.SessionCallback$$anonfun$apply$2.invoke(SessionCallback.scala:65)
        at jnr.ffi.provider.jffi.NativeClosureProxy$$impl$$1.invoke(Unknown Source)
        at com.kenai.jffi.Foreign.invokeArrayReturnLong(Native Method)
        at com.kenai.jffi.Invoker.invokeLong(Invoker.java:1970)
        at com.kenai.jffi.Invoker$LP64.invokeAddress(Invoker.java:2223)
        at com.ctc.omega_edit.FFI$jnr$ffi$0.omega_edit_create_viewport(Unknown Source)
        at com.ctc.omega_edit.SessionImpl.viewCb(SessionImpl.scala:116)
        at com.ctc.omega_edit.grpc.Session$$anonfun$receive$1.applyOrElse(Session.scala:123)
        at akka.actor.Actor.aroundReceive(Actor.scala:537)
        at akka.actor.Actor.aroundReceive$(Actor.scala:535)
        at com.ctc.omega_edit.grpc.Session.aroundReceive(Session.scala:98)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
        at akka.actor.ActorCell.invoke(ActorCell.scala:547)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

[ERROR] [07/28/2022 20:49:15.844] [omega-grpc-akka.actor.default-dispatcher-10] [akka.grpc.scaladsl.GrpcExceptionHandler(akka://omega-grpc)] Unhandled error: [Ask timed out on [Actor[akka://omega-grpc/user/$a#-211926991]] after [1000 ms]. Message of type [com.ctc.omega_edit.grpc.Editors$SessionOp]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.] (akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://omega-grpc/user/$a#-211926991]] after [1000 ms]. Message of type [com.ctc.omega_edit.grpc.Editors$SessionOp]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.)

The reason you're seeing that traceback is because I believe the Scala server does not include all the session event kinds.   1024 is SESSION_EVT_CREATE_VIEWPORT.  The events are enumerated in fwd_defs.hpp and duplicately defined in omega_edit.proto in the SessionEventKind enum.  SessionEvent.scala needs to match.

@arosien, as a server, we need some kind of non-interactive way to shut it down.  Maybe it can emit its process ID to a file that we can send that process a SIGHUP signal or something when we want it to shutdown.  That would work on Mac and Linux at least.

@arosien, as a server, we need some kind of non-interactive way to shut it down. Maybe it can emit its process ID to a file that we can send that process a SIGHUP signal or something when we want it to shutdown. That would work on Mac and Linux at least.

Using the sbt-native-packager that we already use, we can package up a daemon server that has a standard unix-y server script for start and stop. I'll whip it up.

That would be excellent.  Instead of a diff, can you create a branch off of main, push, and create a PR?  I'll link this issue to that PR.

With a nice wrapper script for starting and stopping, the TS/JS client code could start it up and shut it down on demand.  Would be very helpful to make the binding interface and port configurable on the CLI as well to avoid conflicts on port 9000 if we want to run more than one at time or if something else is already bound to that port on the same interface.

With a nice wrapper script for starting and stopping, the TS/JS client code could start it up and shut it down on demand. Would be very helpful to make the binding interface and port configurable on the CLI as well to avoid conflicts on port 9000 if we want to run more than one at time or if something else is already bound to that port on the same interface.

@scholarsmate so i thought there was already a plugin to supply such scripts, but so far i can only find a launch script, not a daemon-ization one. i have my minions on it...

I made #325 close this... should it be reopened, @scholarsmate, to capture additional testing steps, or would that be a new issue or something offline?

Reopening this because the test cases for the Scala RPC server are not fully working yet.

#334 fixes these!

There are still 2 tests that are failing: searching binary data and floating viewports.

There are still 2 tests that are failing: searching binary data and floating viewports.

looking into it...

There are still 2 tests that are failing: searching binary data and floating viewports.

looking into it...

@arosien One issue I think would be inside of src/bindings/scala/api/src/main/scala/com/ctc/omega_edit/SessionImpl.scala both calls to i.omega_edit_create_viewport hard code the param for is_floating to false. This I imagine causes the floating viewports issue.

@arosien @scholarsmate PR https://github.com/ctc-oss/omega-edit/pull/353 fixes the floating viewport support test. Now will just need to focus on fixing the binary data test

@Shanedell @scholarsmate
1) Searching
       Should work with replace on binary data:

      AssertionError: expected Uint8Array[ 123, 6, 5, 4, 7, 8, …(2) ] to deeply equal Uint8Array[ 123, 6, 5, 4, 7, 8, 9 ]
      + expected - actual

      -{"0":123,"1":6,"2":5,"3":4,"4":7,"5":8,"6":9,"7":0,"8":254,"9":255}
      +{"0":123,"1":6,"2":5,"3":4,"4":7,"5":8,"6":9}

      at /Users/arosien/nteligen/omega-edit/src/rpc/client/ts/tests/specs/search.spec.ts:112:71
      at Generator.next (<anonymous>)
      at fulfilled (tests/specs/search.spec.ts:23:58)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)

This fails because the requested segment is passed a length of 7, so the response is correct according to the inputs. The size of 7 comes from omega_session_get_computed_file_size, so I'm wondering if that method isn't returning the correct size.

It's also likely that the overwrite change is treating the data as null-terminated (in the Scala code), which would make the computed length incorrect. Checking that.

@arosien I'll add an expect test to check the return value of omega_session_get_computed_file_size to make sure that correct.  The same exact tests works with the C++ server which is why I'm skeptical that it's something other than the Scala RPC server.

I just pushed a change that tests the return value of omega_session_get_computed_file_size to make sure that it's correct.  The correct value is 10 and that is indeed what's getting returned with the C++ RPC server.  If it's returning 7 in the Scala RPC server, that means that it's very likely that the insert into the session is getting truncated because the Scala code is treating as a string and terminating it prematurely.


my side of the story

Added continuation to main story.

Hi! Thanks for your submission.
Unfortunately it seems that there were changes in the main that conflict with yours.
Could you please perform a merge as described in Tutorial 5 slides 82-96 and integrate those changes?
We do not want to lose any of them, so you should keep what others have written and add yours.
You're free to reorganize the content blocks so that it still makes a story, but please do not modify
the content of what others have written.
Please do as soon as possible. If you don't do anything I will assume you are not interested in improving your current grade.

Closing this as I failed to add in time.


Issue with cloudflare Plugin

Hi,
I got this issue when I activated CloudflarePlugin (in below).
2016/11/05 13:23:54 [error] 5143#5143: *21 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught TypeError: Argument 1 passed to CF\Integration\DefaultIntegration::__construct() must implement interface CF\Integration\ConfigInterface, instance of Magento\Framework\ObjectManager\ObjectManager given, called in vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php on line 93 and defined in vendor/cloudflare/cloudflare-plugin-backend/src/Integration/DefaultIntegration.php:20
Stack trace:
#0 vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(93): CF\Integration\DefaultIntegration->__construct(Object(Magento\Framework\ObjectManager\ObjectManager))
#1 vendor/magento/framework/ObjectManager/Factory/Compiled.php(88): Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject('CF\Integration\...', Array)
#2 vendor/magento/framework/ObjectManager/Factory/Compiled.php(130): Magento\Framewo" while reading response header from upstream
I followed the steps given to install the module but nothing changed:
From the magento2 root directory run the following commands:
composer require cloudflare/cloudflare-magento
composer update
bin/magento setup:upgrade
bin/magento setup:di:compile
Thanks in advance

Hi,
What version of the plugin are you using?
Thank,
John

Hi,
I am using I think the last version 1.0.9.
Thanks in advance for your reply.
Ferdinand
2016-11-07 1:27 GMT+01:00 John Wineman<EMAIL_ADDRESS>
Hi,
What version of the plugin are you using?
Thank,
John
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cloudflare/Cloudflare-Magento/issues/34#issuecomment-258723324,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF_wjLswOyeGq61MroadmeXYVRrKBBlOks5q7nBigaJpZM4KqR4y
.

--
Ferdinand KABORE
*Engineer in Salesforce & Web *
Port : 06 59 11 27 56

Still looking into this - what version of Magento are you running?

I am using Magento version 2.1.2
2016-11-08 17:57 GMT+01:00 John Wineman<EMAIL_ADDRESS>
Still looking into this - what version of Magento are you running?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cloudflare/Cloudflare-Magento/issues/34#issuecomment-259193747,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF_wjJAwlENw7m7PiAahcHoAgFRugM2eks5q8KnrgaJpZM4KqR4y
.

Hi,
My dev environment is 2.0, looks like this is a compatibility issue with 2.1.x.  Not sure what the root cause is but I'm setting up a 2.1 environment and working on a fix now!  I'll update this issue when I know more.
Thanks,
John

Hi John,
Please let me know if you find a fix for 2.1.x.
Courage
Thanks.
2016-11-09 0:18 GMT+01:00 John Wineman<EMAIL_ADDRESS>
Hi,
My dev environment is 2.0, looks like this is a compatibility issue with
2.1.x. Not sure what the root cause is but I'm setting up a 2.1 environment
and working on a fix now! I'll update this issue when I know more.
Thanks,
John
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cloudflare/Cloudflare-Magento/issues/34#issuecomment-259290427,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF_wjL_xxJ2m9_A8yyDAdVxP1WH2dONOks5q8QMpgaJpZM4KqR4y
.

Hi,
I'm able to reproduce the issue now and am working on a fix.
Thanks,
John

Hi,
after running
rm -rf var/*
bin/magento cache:flush

I was able to load the plugin, does this work for you?

Hello, @jwineman!
This plugin works fine only on development mode.
When you go to production, you need to compile all dependency injections.
You have a problem with Magento DI, and this Exception gives you a good notification about your problem.
With best regards!

I think:
rm -rf var/**
is very bad idea, because on production environment you will clear logs, preprocessed staff etc, DI compiled files etc.
And this move will get you additional problems.

Hi @hawok,
Can you please go into more detail?  Before developing this plugin I'd never used magento before so my domain knowledge is lacking.
Doesn't bin\magento setup:di:compile compile the dependency injection graph?  We have this as a step after installing the plugin.  Is there another command I should be aware of?
Thanks,
John

Thanks for your fast feedback, @jwineman!
I think you have a problem in di.xml file.
In this file you defined references for CF objects and inject custom objects from Backend.
When Magento processed this file (bin\magento setup:di:compile) and anybody go to the site they gets this Exception.
Maybe this issue depends on latest Magento changes to 2.1.x branch.
I think this is a start point of resolving this issue.

The issue is definitely between Magento CE 2.0.x and 2.1.x.  However if there is a problem with di.xml I'm not aware what it is.  The config in question is:
<type name="\CF\Integration\DefaultIntegration">
        <arguments>
            <argument name="config" xsi:type="object">\CF\Integration\DefaultConfig</argument>
            <argument name="integrationAPI" xsi:type="object">\CloudFlare\Plugin\Backend\MagentoAPI</argument>
            <argument name="dataStore" xsi:type="object">\CloudFlare\Plugin\Backend\DataStore</argument>
            <argument name="logger" xsi:type="object">\Psr\Log\LoggerInterface</argument>
        </arguments>

The first argument specified is \CF\Integration\DefaultConfig but Magento passes Magento\Framework\ObjectManager\ObjectManager instead.  I don't know why Magento isn't injecting the class I specified.  It looks like bin\magento cache:flush clears the generated class and forces it to be regenerated correctly though.  I'm not sure if rm -rf var/generation is needed.

If you will run
"bin/magento deploy:mode:set production"
You will see a process of static and DI compilation.
After that U will have this issue again.
When you don't have a compiled DI, Magento will go to all DI.xml and make all injections on the fly.
But I don't understand, why without compiled DI all works fine.
Maybe problems with sort, maybe defects on their compile process.

Okay this is helpful - thank you!  I'll debug further and report back.

Thank you too!
Would like to help you, but I can debug this problem tomorrow.
If you want to get some understanding of Magento 2 object management you can check http://alanstorm.com/category/magento-2/#magento-2-object-system.
Best regards!

Hello,
Thanks @jwineman John for your help to debug the issue. I agree with you @hawok. Effectively if I used the command line rm -rf var/*, I'll loose all the cache...
Let me know please, if you find another idea about that :-)

Still looking into this.  This issue seems similar but not root cause:
http://magento.stackexchange.com/a/101577
When I run bin/magento deploy:mode:set production it does error like you would expect but not specific enough to debug.  Running it a second time it completes successfully.  I'm confused.

Hello, @jwineman!
Yep, this problem depends on var/generation folder. Magento 2 still have this problem on the last release version. Thank you for your work!

@hawok are you saying this is something Magento, not Cloudflare needs to fix?

This problem depends on Magento 2 DI mechanics. I think you need update README and wait fix from Magento 2.
Thanks,
Dima.

I'll submit a bug with Magento and see what happens.

Hi, is there an update on this issue? any update from magento?

I just wanted to let you know I have this problem on Magento 2.0.7

Hi Guys,
Sorry I've been slow to reply.  I still don't have a root cause for this bug, another work around appears to be running bin/magento deploy:mode:set production twice.  Something about having ran setup:di:compile beforehand breaks.

Alright I've resolved this issue.  Magento allows you to specify a <preference/> in di.xml which tells Magento which implementation to use when a constructor specifies an interface.
http://devdocs.magento.com/guides/v2.0/extension-dev-guide/build/di-xml-file.html#abstraction-implementation-mappings
<preference for="CF\Integration\ConfigInterface" type="CF\Integration\DefaultConfig" />

Fixes the issue.  Working on a patch now.

Fixed in https://github.com/cloudflare/Cloudflare-Magento/pull/47
Can you try upgrading to 1.1.1 and let me know if that fixes the issue?


Executing powershell script with bool parameter doesnt work

Steps to reproduce
Run from CMD the following command
powershell -ExecutionPolicy Unrestricted -File .\somescript.ps1 -someParameter $True
In the script itself i have the following line
[Parameter(Mandatory=$true)] [bool] $installSQL
I tried playing with that line and the arguments in many ways, all fail.. i have to turn this into a string and make the string into a bool :\
Expected behavior
Script starting to work.. :)
Actual behavior
Error Message Cannot process
argument transformation on parameter 'installSQL'. Cannot convert value "System.String" to type "System.Boolean".
Boolean parameters accept only Boolean values and numbers, such as $True, $False, 1 or 0.
+ CategoryInfo          : InvalidData: (:) [bootstrapImageScript.ps1], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : ParameterArgumentTransformationError,bootstrapImageScript.ps1
Environment data
Name                           Value

PSVersion                      5.1.14393.1198
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.1198
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>

Workaround:
powershell -Command .\somescript.ps1 -someParameter $True

Related https://github.com/PowerShell/PowerShell/issues/4024 where parameter parsing for -File and -Command differ

@SteveL-MSFT So is this a workaround as this is a known issue or is it working as expected?

@pixelicous expectation is that -Command and -File should behave similarly but that is not the case today and should be investigated

@SteveL-MSFT Ok, so i think this issue should be kept live, another thing that might be related.. I am writing a PS script that calls a function from a PSM1 file in the same directory.
When I test in PS ISE i remember to remove-module and extract/reimport the function..
The issue is that a boolean parameter from the main script doesn't get sent to the function correctly.
The main script receives a bool parameter when executed, i set it to $false, i pass that to the function as boolean but it always returns $true, no matter how i input ($false, $False, false, 0) or how i try to read that variable, such as:
if ($boolParam)
if ($boolParam -eq $false)
etc..
it always returns $true!!
The only thing that made it work is switching the bool parameter to switch parameter in the main script, the function the script is calling from module still with "BOOL", and it works now.. just didnt want to open another issue, this all seems to be related
thanks in advance

@pixelicous: We'd need to know more about how you pass that Boolean and how the function is defined in order to diagnose the problem.
On a general note, whenever a nonempty string is cast to [bool], the result is $True - irrespective of the content of the string; e.g., [bool] 'False' yields $True.
However, with a parameter that is properly typed [bool], passing a string is prevented altogether.

@mklement0 All is written in the first post. I wrote how i declare my parameter and how do i call that ps1 script. The string is not empty. i tried running:
"-myParam $true"
"-myParam $True"
"-myParam true"
"-myParam True"
"-myParam 1"
On all i received that same error, when i change that parameter to SWITCH, it goes away (when i specify "-myParam"

Your original problem is explained in #4024 (in short: whatever value you pass is interpreted as a string, whose conversion to [bool] then fails - a known problem).
In a follow-up comment you were complaining about a different issue that you feel is related (" pass that to the function as boolean but it always returns $true"), and that is the one I've requested clarification on.
Generally, using a [switch] parameter is preferable to a [bool] anyway, assuming it has a reasonable default value.

@mklement0 hey, sorry im on a vacation and will only return back in around 2 weeks time, thus i cannot answer your exact question.
The reason why i do not use -myParam as a switch is because i'm passing that value between a few functions and tools from a control script. If i use this as a switch ill have to build the execution string with the argument or without, when i use $true or $false, the execution string is always the same with a parameter that changes, easier to read and understand..
Eventually.. i moved things around, i used the SWITCH parameter for the main control script, and BOOL for anything else, and it works now.. except for invoking powershell with -file, that still doesn't work.. but i still think it is related, if i wouldn't have used a SWITCH parameter in the main control but a BOOL one, it just doesnt get passed through.. i think it is looking whether that BOOL variable is filled with a value or not, thus sending back $true everytime although it should have sent back $false

@pixelicous:
No worries re delay.
As an aside (it doesn't solve the -File issue):
There actually is a way to use your command-building technique with [switch] parameters too, but the syntax is not often seen:

-Foo:$True is the same as just -Foo

-Foo:$False is the same as not specifying -Foo.

I wonder if we should support -switch [bool] syntax as I agree the current -switch:[bool] syntax is not obvious.  Should be separate issue we anyone wants to pursue.  I'll try to fix this and the -File arg issue next week.

@SteveL-MSFT:
The -switch:[bool] syntax is awkward (presumably because it was tacked on later), but necessary for disambiguation:
Consider & { param([switch] $Switch)  $Switch } vs. & { param([switch] $Switch, [bool] $Bool) $Switch, $Bool  }  - how would you distinguish between the two with arguments -switch $true, for instance?
Note that some Unix utilities have solved this problem by adopting the convention of allowing any switch (option) to be prefixed with no to signal negation:  for instance, -noswitch would then take the place of -switch:$false.

@mklement0 positional parameters make it problematic for sure.  Perhaps a non-breaking change would be to utilize the suggestion feature to suggest the colon syntax when it's not a valid set of parameters

@mklement0 The -switch: [bool] syntax is deliberately awkward as there is really only one scenario for passing an actual parameter to a switch: commands calling commands (which you described nicely in an earlier comment.) If you expect the user to pass an actual parameter, then you should be using boolean. If your cmdlet requires the user to pass arguments to a switch parameter then your cmdlet should be redesigned.
@SteveL-MSFT No we can't support -switch [bool]. The current behavior is fundamental to how the parameter binder works.
-Command and -File are very different operations. -Command gathers all of the arguments into a string and then processes that string as a PowerShell script. On the other hand, -File points at a script and passes the remaining arguments to the process. The application receives these arguments through argv[] as strings. Process creation simply doesn't allow for the parent to pass anything but strings to the child process. The PowerShell runtime does do some interpretation on these strings (e.g. treating strings that start with '-' as parameters instead of arguments) but this is necessarily limited by the lack of type information.

@BrucePay:
The immediate issue here is the current fundamental inability to pass a Boolean value (whether as an actual [bool] parameter of with the -switch:<bool> syntax) when using -File, because such a value is invariably treated as a string (for the reasons you describe), and therefore rejected.
I think that needs to be addressed, such as by accepting string values $true and $false as well (just as it is possible to pass a integer as a string).
-File and -Command are indeed currently very different, but I think they shouldn't be - see https://github.com/PowerShell/PowerShell/issues/4024#issuecomment-311541803

The current implementation makes the two different, but I agree with @mklement0 that there doesn't seem to be a reason from a user experience why it should be different.  Using the same code for -Command to collect the args and pass it as a string works for -File with .ps1 scripts, just need some additional code to handle scripts without .ps1 extension.

@PowerShell/powershell-committee the concern of supporting $true and $false is resolved with #4178, the concern regarding the difference between -file and -command in general is by design per #4024


Set up a "Hello world" project with two apps

In this milestone I have covered the following points :

Created a new Rails API app called 'hello-rails-back-end'.
Initialized my project with Git.
Project has a Postgres database set up.
Created a table for storing your messages and creating 5 different greetings.
Created an API endpoint that selects a random greeting from your table.

Below is the link to the pull request for the front-end part of my project
Front End

STATUS: INVALID CODE REVIEW REQUEST 🛑 🛑
Hi @ndikumanaisaie,
While you made a great effort in this project, unfortunately, I cannot proceed to review your code.
Invalid Code Review Request
The frontend README should contain a link to the backend as well.
Your Code Review Request will be marked as invalid in your Dashboard, so please submit a new one once you are ready 🙏
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear. And please, remember to tag me in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

Invalid Code Review Request does not count into the code reviews limit.

Dear @TedLivist Thank you for the feedback. The link to the backend pull request from the frontend have been included.


Feature/pazuzu registry impl

Missing elements:

links.
tests.
role base limitation.

This PR probably needs some rebasing :-)  Really tricky to follow the history.


Pinctrl amd

Description of changes
Backport to re-enable AMD touchpads with kernel 5.19
ref: https://github.com/NixOS/nixpkgs/pull/185672#issuecomment-1210335202 and #185884
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

I'm also experiencing this on 5.18.18.
AMD laptop touchpad stopped working on update, and:
CONFIG_PINCTRL=y
#CONFIG_PINCTRL_AMD is not set


How to add some function in topi/nn/?

I am trying add some function in topi.nn, But after I add one my function, i cannot run the test files.
python '/home/aries/tvm/topi/recipe/conv/copydepth.py'
Traceback (most recent call last):
File "/home/aries/tvm/topi/recipe/conv/copydepth.py", line 197, in
test_depthwise_conv2d_nchw()
File "/home/aries/tvm/topi/recipe/conv/copydepth.py", line 38, in test_depthwise_conv2d_nchw
DepthwiseConv2d = topi.nn.gaidepthwise_conv2d_nchw(Input, Filter, Stride, padding)
AttributeError: 'module' object has no attribute 'gaidepthwise_conv2d_nchw'
So I tried to install the tvm and topi again. But not make them.
But now I cannot run any test files.
TVM: Initializing cython mode...
Traceback (most recent call last):
File "/home/aries/tvm/topi/recipe/conv/depthwise_conv2d_test.py", line 2, in
import tvm
File "/home/aries/.local/lib/python2.7/site-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/init.py", line 5, in
from . import tensor
File "/home/aries/.local/lib/python2.7/site-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/tensor.py", line 6, in
from . import make as _make
File "/home/aries/.local/lib/python2.7/site-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/make.py", line 130, in
_init_api("tvm.make")
File "/home/aries/.local/lib/python2.7/site-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/_ffi/function.py", line 268, in _init_api
for name in list_global_func_names():
File "/home/aries/.local/lib/python2.7/site-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/_ffi/function.py", line 229, in list_global_func_names
check_call(_LIB.depthwise_conv2d_nchwTVMFuncListGlobalNames(ctypes.byref(size),
File "/usr/lib/python2.7/ctypes/init.py", line 375, in getattr
func = self.getitem(name)
File "/usr/lib/python2.7/ctypes/init.py", line 380, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/aries/.local/lib/python2.7/site-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/libtvm.so: undefined symbol: depthwise_conv2d_nchwTVMFuncListGlobalNames
What can I do now? And how to add some function in the topi/nn module

I deleted all files I added， and it works again.
But I would like to add some functions in tvm/python/tvm and tvm/topi/python/topi,
What should I do after added them into folders?
Thank you in advance for ur help.

Depend on how you do your development. If you do insource development (setup PYTHONPATH), then it would be totally fine

I just copy the tvm/topi/python/topi/nn/depthwise_conv2d.py and rename its function names, But It cannot be used.
I have installed again and setup pythonpath in bashrc file ( export PYTHONPATH=...)
What should I do? @tqchen
Traceback (most recent call last):
File "/home/aries/testtvm/tvm/topi/recipe/conv/depthwise_conv2d_test.py", line 212, in
test_depthwise_conv2d_nchw()
File "/home/aries/testtvm/tvm/topi/recipe/conv/depthwise_conv2d_test.py", line 55, in test_depthwise_conv2d_nchw
DepthwiseConv2d = topi.nn.depthcope_conv2d_nchw(Input, Filter, Stride, padding)
AttributeError: 'module' object has no attribute 'depthcope_conv2d_nchw'

you will need to import the function you defined in topi/nn/init.py

It works.


[CM-699] Web like rating view

Summary
Added feedback icons to show rating labels

Also, add a Changelog entry.


⚠ Bofrost.fr has degraded performance

In aa097db, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 846 ms

Resolved: Bofrost.fr performance has improved in f6cd0ac after 11 minutes.


Import fails with swagger.json

The swagger file from: https://ergebnis.radball-bw.de/swagger.json fails with schema errors. Is there any kind of debug log available to find the reason? the file works perfectly in swagger-ui

any idea?

@hoegertn
I get:
20/11/2015 11:19:35.480 Paw[234]: LMImporterExtension+LMImporter: Import: JavaScript Exception: Error: Invalid Swagger file (invalid schema or schema version < 2.0) ({
    column = 24;
    line = 181;
})

Yes, but the file is a valid swagger json. Swagger UI is able to parse it correctly

http://editor.swagger.io/#/
if you load it here i get an error:
✖ Swagger Error
Missing required property: info
Jump to line 0
Details
 Object
code:  "OBJECT_MISSING_REQUIRED_PROPERTY"
message:  "Missing required property: info"
 path: Array [0]
level: 900
type:  "Swagger Error"
description:  "Missing required property: info"
lineNumber: 0

Ok then I will have a look at the file and search the bug in the swagger generator

as a work around for know you can load it there and then click FILE->download json PAW and one this.

Ok I fixed some fields and now it works. I will create further issues for some improvements, but I will have a look at the source and try to create PullRequests. This issues was causes by the generator creating invalid spec file

@hoegertn great! we will try to improve the error reporting in the Paw importer, it would be nicer to show you precisely why it's failing.