[Request][Feature] Deploy to 1 target with multiple destination dirs

Currently I have a sandbox server with multiple web environments (for many testers). Now I need to define multiple targets with the same login info for each folder and deploy it separately. Something like this:
{
    "type": "sftp",
    "name": "Sandbox Server",
    "description": "A SFTP folder",
    "dirs": [
        "/home/mgsys/dev_envs/dev2",
        "/home/mgsys/dev_envs/dev3",
        "/home/mgsys/dev_envs/dev27"
    ],
    "host": "sandbox_srv", "port": 22,
    "user": "mgsys",
    "privateKey": "/users/admin/.ssh/id_rsa"
}

@CasperPas
Thanks for the feature request!
I will implment this in version 8.0 with a new target type. I think that it will be finished until the weekend.
Meanwhile you can try to use a batch target:
[
    {
        "type": "batch",
        "name": "Sandbox Server",
        "description": "A SFTP folder",
        "targets": [
            "Sandbox Server (dev2)",
            "Sandbox Server (dev3)",
            "Sandbox Server (dev27)"
        ]
    },

    {
        "type": "sftp",
        "name": "Sandbox Server (dev2)",
        "description": "Deploys to dev2 folder",
        "dir": "/home/mgsys/dev_envs/dev2",
        "host": "sandbox_srv", "port": 22,
        "user": "mgsys",
        "privateKey": "/users/admin/.ssh/id_rsa",

        "isHidden": true
    },

    {
        "type": "sftp",
        "name": "Sandbox Server (dev3)",
        "description": "Deploys to dev3 folder",
        "dir": "/home/mgsys/dev_envs/dev3",
        "host": "sandbox_srv", "port": 22,
        "user": "mgsys",
        "privateKey": "/users/admin/.ssh/id_rsa",

        "isHidden": true
    },

    {
        "type": "sftp",
        "name": "Sandbox Server (dev27)",
        "description": "Deploys to dev27 folder",
        "dir": "/home/mgsys/dev_envs/dev27",
        "host": "sandbox_srv", "port": 22,
        "user": "mgsys",
        "privateKey": "/users/admin/.ssh/id_rsa",

        "isHidden": true
    }
]

I know that this is not a very "handy" solution. This is why I will implement the new target type.
BTW: isHidden property means, that the underlying targets are not shown in the GUI. If you want to show then, simply remove them or set them to (false).

that's awesome. you're the best. I'm looking forward to that 👍 😉

@CasperPas
I released version 8.0.0 with a new target type, called each.

Your settings should look like that:
{
    "deploy": {
        "targets": [
            {
                "name": "Sandbox Server",
                "type": "each",

                "from": [
                    "/home/mgsys/dev_envs/dev2",
                    "/home/mgsys/dev_envs/dev3",
                    "/home/mgsys/dev_envs/dev27"
                ],

                "to": [ "dir" ],

                "targets": [ "Template of Sandbox Server" ]
            },

            {
                "name": "Template of Sandbox Server",
                "type": "sftp",

                "host": "sandbox_srv", "port": 22,
                "user": "mgsys",
                "privateKey": "/users/admin/.ssh/id_rsa",

                "isHidden": true
            }
        ]
    }
}

What happens is:

the Sandbox Server target interates over the 3 values in from
it sends all these values to "real" target Sandbox Server template
as defined in to property of Sandbox Server, the values are written to dir setting of Sandbox Server template (this is why you do not see a defined dir here, because Sandbox Server does this job for you)

You also can do this with any other properties, like the host, e.g.:
{
    "deploy": {
        "targets": [
            {
                "name": "Sandbox Server",
                "type": "each",

                "from": [
                    "dev2.example.com",
                    "dev3.example.com",
                    "dev27.example.com"
                ],

                "to": [ "host" ],

                "targets": [ "Template of Sandbox Server" ]
            },

            {
                "name": "Template of Sandbox Server",
                "type": "sftp",

                "dir": "/home/mgsys/dev_env",
                "user": "mgsys",
                "privateKey": "/users/admin/.ssh/id_rsa",

                "isHidden": true
            }
        ]
    }
}

BTW: I also added another feature, that makes it possible to quickly access examples, templates, wiki articles and other resource ... simply press F1 and select Deploy: Open example / template command:

More information can be found here.

Wow, you're awesome! I've just updated and it works great.
However, I also want to ask that, do you feed those each values to each "targets" then each time, it needs to establish the connection everytime? ex, if we have 3 "from" and 2 "targets" => 6 different sessions?

Thats right. This is what you have to keep in mind.
The job of this target type has to be as simple and generic as possible.
That means it has to work with all target types, not only with the existing onces, even with targets that will be implemented in the future and with targets that do not need a session or network connection, like local.
On the other hand the "each" cannot decide if "dir" or "host" setting properties are things that can be "put" into one session or not.
It only works as a simple for loop with all its "consequences".

@CasperPas
This might be also interesting for you:
With version 8.7.0 I released a similar target type, called map:
{
    "deploy": {
        "targets": [
            {
                "name": "Sandbox Server",
                "type": "map",

                "from": [
                    {
                        "dir": "/home/mgsys/dev_envs/dev2"
                    },
                    {
                        "dir": "/home/mgsys/dev_envs/dev3"
                    },
                    {
                        "dir": "/home/mgsys/dev_envs/dev27"
                    }
                ],

                "targets": [ "Template of Sandbox Server" ]
            },

            {
                "name": "Template of Sandbox Server",
                "type": "sftp",

                "host": "sandbox_srv", "port": 22,
                "user": "mgsys",
                "privateKey": "/users/admin/.ssh/id_rsa",

                "isHidden": true
            }
        ]
    }
}

As you can see: The difference is, that you can define a list of value repositories, which contain a list of properties (and their values) that should be written to Template of Sandbox Server.


Update Blog “test1”

Automatically generated by Netlify CMS

👷 Deploy Preview for phenomenal-custard-76e941 processing.

Name
Link

🔨 Latest commit
ef6e1c4c7696ff96834ba193c2e93000cf1f6419

🔍 Latest deploy log
https://app.netlify.com/sites/phenomenal-custard-76e941/deploys/62f4b00e6a25800009ffedff


Support for Workflow definitions

Kaleo Workflow defs are, strictly speaking, not using DDM at all. Wouldn't it still be cool to support them, however?

:zzz:


feat: add redis liveness probe

Description
Check the health of redis instance and expose it trough the API endpoint /health/ready
Look & Feel
Request
[GET] http://localhost:5000/api/v3_0/health/ready

Response
{
	"database_redis": false,
	"database_sql": true,
	"status": 503
}

Testing Steps

Configure credentials

echo 'FLEXMEASURES_REDIS_PASSWORD="MyPassWord"' >> ~/.flexmeasures.cfg

Run FM Server

flexmeasures run

Get and check status

curl localhost:5000/api/v3_0/health/ready

:gear: Check the response (status=503 and database_redis=false)
{
	"database_redis": false,
	"database_sql": true,
	"status": 503
}

Create Redis instance with Docker

sudo docker run --name my-redis -p 6379:6379 -d redis --requirepass "MyPassWord"

Get and check status

curl localhost:5000/api/v3_0/health/ready

:gear: Check the response (status=200 and database_redis=true)
{
	"database_redis": true,
	"database_sql": true,
	"status": 200
}

Clean up
sudo docker stop my-redis && sudo docker rm my-redis

Closes #663

Maybe we should be able to tell the endpoint what we define as health with some GET variables, e.g. http://localhost:5000/api/v3_0/health/ready?expect_redis=no

Good idea! Following Issue #663, we could also check if FLEXMEASURES_REDIS_PASSWORD  is an empty string, although I like the idea of passing the parameter because we could have a worker and a server working under the same environment.

For now, checking FLEXMEASURES_REDIS_PASSWORD seems to be the easiest solution. If the environment has a value here, we should be able to find it.
Unless these nodes operate in different access contexts (e.g. in Kubernetes). Of course, that is rather rare probably, and an be solved by not only using one environment configuration for all nodes. (giving a FM instance access to a credential it shall not be using is not a good security approach, either :)
So I'd argue to go ahead with the simple check.
What do you think?

@victorgarcia98 Can you add a changelog entry? And then let's merge this PR!


[enhancement]:  Alphabetical board sorting.

Is there an existing issue for this?

[X] I have searched the existing issues

Contact Details
No response
What should this feature add?
In my personal humble opinion there should be an alphabetical order for boards as an addition to sorting by time created. It will add an ability to customise boards order by adding leading number or letter. To someone like me with 30+ boards it will greatly improve board management.
Alternatives
No response
Additional Content
No response

I will add to this maybe also board pinning


Expose MongoDB Replica Set

Question
I have only recently started playing with kubernetes and helm and I'm having trouble with the mongodb-replicaset chart.
My goal was to create this cluster and then be able to access it on other servers that I have that need MongoDB.
I have tried to expose the Replica Set with a LoadBalancer but, it doesn't always return the primary node and, I have tried using ingresses but that also got me nowhere.
I feel like there has to be something I am missing. I am just trying to create a MongoDB replica set that is externally available to other sites that I run and connect to with a Mongo URI
Thanks in advance for the help!

this was the answer I got from the chart owner.
but to be honest, I was a little overstrained with that answer :)
quote:
The quick solution is to wrap an operator that knows to switch the service tags based on which pod is the current master.
If you're using the other members of the mongodb-repset as read slaves, then it makes sense to have a single service over all replicas for reads, and another service to handle writes via the master. That's one way to handle this use-case.

The chart has a governing service for the statefulset which is stateless. If you want to expose the replica set, you need additional services, one per replica, which could be of type NodePort or LoadBalancer. In your connect string you would then add all the service URLs. You don't connect to one individual replica. Let the client figure that out.
https://docs.mongodb.com/manual/reference/connection-string/
/cc @foxish

would it be feasible to have a setting for that on the helm chart?
something like replicaServiceType=NodePort

Care to create a PR?

After creating a NodePort Service Type I am getting errors that the local dns names cannot be resolved.
For example
MongoError: failed to connect to server [mongodb-mongodb-replicaset-1.mongodb-mongodb-replicaset.default.svc.cluster.local:27017] on first connect [MongoError: getaddrinfo ENOTFOUND mongodb-mongodb-replicaset-1.mongodb-mongodb-replicaset.default.svc.cluster.local mongodb-mongodb-replicaset-1.mongodb-mongodb-replicaset.default.svc.cluster.local:27017]
@unguiculus @adrianliechti

@alexwine36
$ helm install stable/mongodb-replicaset
==> name wise-owl
after a while, some pods starting coming up:
$ kubectl get pods
wise-owl-mongodb-replicaset-0        1/1       Running   0          1m
wise-owl-mongodb-replicaset-1        1/1       Running   0          42s
wise-owl-mongodb-replicaset-2        0/1       Running   0          20s
now create a service for each of the pod
$ kubectl expose pod wise-owl-mongodb-replicaset-0 --type=NodePort
$ kubectl expose pod wise-owl-mongodb-replicaset-1 --type=NodePort
$ kubectl expose pod wise-owl-mongodb-replicaset-2 --type=NodePort
if you list your services, you now see the individual ports
wise-owl-mongodb-replicaset-0   <IP_ADDRESS>           27017:32151/TCP   30s
wise-owl-mongodb-replicaset-1   <IP_ADDRESS>          27017:30616/TCP   25s
wise-owl-mongodb-replicaset-2   <IP_ADDRESS>           27017:30167/TCP   22s
now the connection string would look like this:
mongodb://{IP-OF-THE-KUBE-NODE-1}:32151,{IP-OF-THE-KUBE-NODE-2}:30616,{IP-OF-THE-KUBE-NODE-3}:30167/
in therms of minikube, you get the ip with
$ minikupe ip
and on a single node system, a connection string would be
(feel free to distribue the ips as you want, since the node port is available on every node and gets routed by k8s)
mongodb://<IP_ADDRESS>:32151,<IP_ADDRESS>:30616,<IP_ADDRESS>:30167/

@unguiculus
yeah we are willing. but creating multiple svc for a dynamic number of replica-pods seems to be a little tricky on the first sight. I will read a little deeper into the helm template engine... :)
in my special case, we built a service broker to interact between a PaaS and a detached k8s cluster (using helm and kubectl) - and it might be easier to invoke kubectl directly for this case...

@adrianliechti  After the pods are exposed as NodePort there is still an issue connecting through a mongodb uri. All of the individual mongodb instances that are connected to show that the replicaset's config has an incorrect hostname. I have tried to rewrite these values but, there seems to be something overriding them back to these original values.
{
"_id" : "rs0",
"version" : 3,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "replica-mongodb-replicaset-0.replica-mongodb-replicaset.default.svc.cluster.local:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
		},
		"slaveDelay" : NumberLong(0),
		"votes" : 1
	},
	{
		"_id" : 1,
		"host" : "replica-mongodb-replicaset-1.replica-mongodb-replicaset.default.svc.cluster.local:27017",
		"arbiterOnly" : false,
		"buildIndexes" : true,
		"hidden" : false,
		"priority" : 1,
		"tags" : {

		},
		"slaveDelay" : NumberLong(0),
		"votes" : 1
	},
	{
		"_id" : 2,
		"host" : "replica-mongodb-replicaset-2.replica-mongodb-replicaset.default.svc.cluster.local:27017",
		"arbiterOnly" : false,
		"buildIndexes" : true,
		"hidden" : false,
		"priority" : 1,
		"tags" : {

		},
		"slaveDelay" : NumberLong(0),
		"votes" : 1
	}
],
"settings" : {
	"chainingAllowed" : true,
	"heartbeatIntervalMillis" : 2000,
	"heartbeatTimeoutSecs" : 10,
	"electionTimeoutMillis" : 10000,
	"catchUpTimeoutMillis" : 60000,
	"getLastErrorModes" : {

	},
	"getLastErrorDefaults" : {
		"w" : 1,
		"wtimeout" : 0
	},
	"replicaSetId" : ObjectId("59839e61e5f61909903aabfb")
}

}

@alexwine36
at least we get closer :)
the hostname within the mongodb replicaset config seems to be okay to me. that's the endpoints of the members in the container network - and it's good if the master<->slave traffic is in that network. the mongodb cluster should not need to know about the port forwarding/external network stuff
which client or sdk do you to connect to the replicaset? and how does your connection string looks like? does it contains all the single nodes and is the replicaset "rs0" specified?
mongodb://..../?replicaSet=rs0

mongodb://..../?replicaSet=rs0

Yes, that would be the connect string with all the hosts in it.

@adrianliechti @unguiculus
As far as Clients I use two different ones
The first is using Studio 3T
The Data URI is
mongodb://<IP_ADDRESS>:30039,<IP_ADDRESS>:30087,<IP_ADDRESS>:32085/?readPreference=nearest&replicaSet=rs0
The error that it gives me is this

Connection
failed.
SERVER [replica-mongodb-replicaset-1.replica-mongodb-replicaset.default.svc.cluster.local:27017] (Type: UNKNOWN)
|/ Connection error (MongoSocketOpenException): Exception opening socket
|___/ Unknown host: replica-mongodb-replicaset-1.replica-mongodb-replicaset.default.svc.cluster.local
SERVER [replica-mongodb-replicaset-2.replica-mongodb-replicaset.default.svc.cluster.local:27017] (Type: UNKNOWN)
|/ Connection error (MongoSocketOpenException): Exception opening socket
|___/ Unknown host: replica-mongodb-replicaset-2.replica-mongodb-replicaset.default.svc.cluster.local
Details:
Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=replica-mongodb-replicaset-1.replica-mongodb-replicaset.default.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.UnknownHostException: replica-mongodb-replicaset-1.replica-mongodb-replicaset.default.svc.cluster.local}}, {address=replica-mongodb-replicaset-2.replica-mongodb-replicaset.default.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.UnknownHostException: replica-mongodb-replicaset-2.replica-mongodb-replicaset.default.svc.cluster.local}}]

The other connection I have tried is through Meteor.js which gives me an error like this
For this one I also tried this URI
mongodb://<IP_ADDRESS>:30039,<IP_ADDRESS>:30087,<IP_ADDRESS>:32085/den?replicaSet=rs0
and
mongodb://<IP_ADDRESS>:30039,<IP_ADDRESS>:30087,<IP_ADDRESS>:32085/?readPreference=nearest&replicaSet=rs0

W20170814-11:35:13.829(-6)? (STDERR) MongoError: failed to connect to serv
er [replica-mongodb-replicaset-1.replica-mongodb-replicaset.default.svc.cl
uster.local:27017] on first connect [MongoError: getaddrinfo ENOTFOUND rep
lica-mongodb-replicaset-1.replica-mongodb-replicaset.default.svc.cluster.l
ocal replica-mongodb-replicaset-1.replica-mongodb-replicaset.default.svc.c
luster.local:27017]

@alexwine36 please take a look at #1743, I think it will solve the issue you are facing.
I am still getting these frequent errors
2017-08-15T15:41:31.008+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] getaddrinfo("messy-hydra-mongodb-replicaset-1.messy-hydra-mongodb-replicaset.default.svc.cluster.local") failed: Name or service not known
2017-08-15T15:42:01.011+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] getaddrinfo("messy-hydra-mongodb-replicaset-1.messy-hydra-mongodb-replicaset.default.svc.cluster.local") failed: Name or service not known
2017-08-15T15:42:01.013+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] getaddrinfo("messy-hydra-mongodb-replicaset-2.messy-hydra-mongodb-replicaset.default.svc.cluster.local") failed: Name or service not known

But I think they are related to Kubernetes DNS not to the Helm chart.

Hey @alexwine36, I ran into the same issue as you.  Did you find any resolution?
I can reach the cluster from my mongo client, but once I'm connected, it appears to be trying to use member hosts that I cannot resolve.. names internal to the cluster.
Mongo docs say:

Always use resolvable hostnames for the value of the members[n].host field in the replica set configuration to avoid confusion and complexity.

https://docs.mongodb.com/v3.2/tutorial/change-hostnames-in-a-replica-set/

So if I read this correct you guys are saying it should give my client all the replicaset nodes in the connection url? This feel dirty, this would make scaling impossible without editing the config of all my clients.
auto_mongodb_url=mongodb://${CI_ENVIRONMENT_SLUG}-mongodb-replicaset:27017/${MONGODB_DB}?replicaSet=rs0

would become
auto_mongodb_url=mongodb://${CI_ENVIRONMENT_SLUG}-mongodb-replicaset-0.${CI_ENVIRONMENT_SLUG}-mongodb-replicaset.${KUBE_NAMESPACE}.svc.cluster.local:27017,${CI_ENVIRONMENT_SLUG}-mongodb-replicaset-1.${CI_ENVIRONMENT_SLUG}-mongodb-replicaset.${KUBE_NAMESPACE}.svc.cluster.local:27017,${CI_ENVIRONMENT_SLUG}-mongodb-replicaset-2.${CI_ENVIRONMENT_SLUG}-mongodb-replicaset.${KUBE_NAMESPACE}.svc.cluster.local:27017/${MONGODB_DB}?replicaSet=rs0

Secondly I can't pass this along to Helm easily
Error: failed parsing --set data: key "local:27017" has no value (cannot end with ,)

apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: mongodb
  labels:
    release: mongodb
  annotations: {}
spec:
  ports:
  - protocol: TCP
    port: 27017
    targetPort: 27017
  selector:
    release: mongodb
  type: LoadBalancer

Using labels can handle this automatically.

@carldanley That is what we tried but there is know way to force the load balancer to the master it will randomly select servers. The loadbalancer will only seem to work if you have a single mongodb instance. Example where <IP_ADDRESS> is the ip given from our loadbalancer endpoint:
awilhelm@MBP ~$ mongo --host <IP_ADDRESS> --port 27017
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:27017/
MongoDB server version: 3.6.3
MainRepSet:PRIMARY> ^C
bye
awilhelm@MBP ~$ mongo --host <IP_ADDRESS> --port 27017
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:27017/
MongoDB server version: 3.6.3
MainRepSet:SECONDARY> ^C
bye
awilhelm@MBP ~$ mongo --host <IP_ADDRESS> --port 27017
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:27017/
MongoDB server version: 3.6.3
MainRepSet:PRIMARY> ^C
bye
awilhelm@MBP ~$ mongo --host <IP_ADDRESS> --port 27017
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:27017/
MongoDB server version: 3.6.3
MainRepSet:SECONDARY>

Also the exposing of NodePort doesn't work on a distributed cluster because it appears that the client and server try to communicate using the k8s internal dns names that are not routeable.
I expose the Stateful set pods:
service "mongod-0" exposed
$ kubectl expose pod mongod-1 --type=NodePort
service "mongod-1" exposed
$ kubectl expose pod mongod-2 --type=NodePort
service "mongod-2" exposed```

I then attempt to connect using the IP of a node in my cluster from my laptop:

```$ mongo mongodb://<IP_ADDRESS>:30446,<IP_ADDRESS>:32234,<IP_ADDRESS>:31456/?replicaSet=MainRepSet
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:30446,<IP_ADDRESS>:32234,<IP_ADDRESS>:31456/?replicaSet=MainRepSet
2018-03-07T14:26:12.753-0500 I NETWORK  [thread1] Starting new replica set monitor for MainRepSet/<IP_ADDRESS>:30446,<IP_ADDRESS>:32234,<IP_ADDRESS>:31456
2018-03-07T14:26:12.903-0500 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to <IP_ADDRESS>:30446 (1 connections now open to <IP_ADDRESS>:30446 with a 5 second timeout)
2018-03-07T14:26:12.912-0500 I NETWORK  [thread1] Successfully connected to <IP_ADDRESS>:31456 (1 connections now open to <IP_ADDRESS>:31456 with a 5 second timeout)
2018-03-07T14:26:13.160-0500 I NETWORK  [thread1] Successfully connected to <IP_ADDRESS>:32234 (1 connections now open to <IP_ADDRESS>:32234 with a 5 second timeout)
2018-03-07T14:26:13.239-0500 I NETWORK  [thread1] changing hosts to MainRepSet/mongod-0.mongodb-service.default.svc.cluster.local:27017,mongod-1.mongodb-service.default.svc.cluster.local:27017,mongod-2.mongodb-service.default.svc.cluster.local:27017 from MainRepSet/<IP_ADDRESS>:30446,<IP_ADDRESS>:31456,<IP_ADDRESS>:32234

2018-03-07T14:26:17.979-0500 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] getaddrinfo("mongod-1.mongodb-service.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-03-07T14:26:18.239-0500 I NETWORK  [thread1] getaddrinfo("mongod-2.mongodb-service.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-03-07T14:26:22.982-0500 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] getaddrinfo("mongod-0.mongodb-service.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-03-07T14:26:28.486-0500 I NETWORK  [thread1] getaddrinfo("mongod-2.mongodb-service.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-03-07T14:26:33.490-0500 I NETWORK  [thread1] getaddrinfo("mongod-1.mongodb-service.default.svc.cluster.local") failed: nodename nor servname provided, or not known
^C2018-03-07T14:26:37.292-0500 E -        [main] Error saving history file: FileOpenFailed: Unable to open() file : No such file or directory
2018-03-07T14:26:37.292-0500 I CONTROL  [main] shutting down with code:0```

The connection happens but then it tries to switch over to using internal DNS names and my connection dies. This works fine from inside a pod in the k8s cluster:

$ kubectl exec -it mongod-0 bash
root@mongod-0:/# mongo mongodb://<IP_ADDRESS>:30446,<IP_ADDRESS>:32234,<IP_ADDRESS>:31456/?replicaSet=MainRepSet
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:30446,<IP_ADDRESS>:32234,<IP_ADDRESS>:31456/?replicaSet=MainRepSet
2018-03-07T19:29:42.722+0000 I NETWORK  [thread1] Starting new replica set monitor for MainRepSet/<IP_ADDRESS>:30446,<IP_ADDRESS>:32234,<IP_ADDRESS>:31456
2018-03-07T19:29:42.724+0000 I NETWORK  [thread1] Successfully connected to <IP_ADDRESS>:31456 (1 connections now open to <IP_ADDRESS>:31456 with a 5 second timeout)
2018-03-07T19:29:42.725+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to <IP_ADDRESS>:30446 (1 connections now open to <IP_ADDRESS>:30446 with a 5 second timeout)
2018-03-07T19:29:42.727+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to <IP_ADDRESS>:32234 (1 connections now open to <IP_ADDRESS>:32234 with a 5 second timeout)
2018-03-07T19:29:42.727+0000 I NETWORK  [thread1] Successfully connected to mongod-1.mongodb-service.default.svc.cluster.local:27017 (1 connections now open to mongod-1.mongodb-service.default.svc.cluster.local:27017 with a 5 second timeout)
2018-03-07T19:29:42.728+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] changing hosts to MainRepSet/mongod-0.mongodb-service.default.svc.cluster.local:27017,mongod-1.mongodb-service.default.svc.cluster.local:27017,mongod-2.mongodb-service.default.svc.cluster.local:27017 from MainRepSet/<IP_ADDRESS>:30446,<IP_ADDRESS>:31456,<IP_ADDRESS>:32234
2018-03-07T19:29:42.730+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to mongod-2.mongodb-service.default.svc.cluster.local:27017 (1 connections now open to mongod-2.mongodb-service.default.svc.cluster.local:27017 with a 5 second timeout)
2018-03-07T19:29:42.731+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to mongod-0.mongodb-service.default.svc.cluster.local:27017 (1 connections now open to mongod-0.mongodb-service.default.svc.cluster.local:27017 with a 5 second timeout)
MongoDB server version: 3.6.3
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
MainRepSet:PRIMARY>
Any help here would be great!

What I tend to do is if I need access to the db, you can port foward to the pod..
kubectl port-forward mongodb-0 27017
Then you can connect via localhost:27017

Ya that works for temporary things and testing but for a more robust production system it isn't going to work.
We ended up implementing shards + mongoS and this actually solved the problem as we can create a LB end point to MongoS and connect through that externally to the sharded backend.
I went through the following guide to help prove this concept:
pauldone.blogspot.com/2017/07/sharded-mongodb-kubernetes.html

Instead of exposing each replicaset individually you can also just create a second (congruent) public service using the kube-proxy as usual (if you need it)
{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "mongodb-mongodb-replicaset-public",
    "namespace": "default",
    "labels": {
      "app": "mongodb-replicaset",
      "chart": "mongodb-replicaset-3.1.0",
      "heritage": "Tiller",
      "release": "mongodb"
    },
    "annotations": {
      "service.alpha.kubernetes.io/tolerate-unready-endpoints": "true"
    }
  },
  "spec": {
    "ports": [
      {
        "name": "peer",
        "protocol": "TCP",
        "port": 27017,
        "targetPort": 27017
      }
    ],
    "selector": {
      "app": "mongodb-replicaset",
      "release": "mongodb"
    },
    "nodePort": "30300",
    "type": "NodePort",
    "sessionAffinity": "None"
  }
}

@Flowkap  This doesn't solve the problem. Because of the way you need to access a replica set in mongodb to determine the master you can't connect through a NodePort exposed service. This is because once you connect through the connection gets switch over to using internal kubernetes namespaces:
example
{code}
$ mongo mongodb://<IP_ADDRESS>:30080/?replicaSet=rs0
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:30080/?replicaSet=rs0
2018-04-27T13:47:28.009-0400 I NETWORK  [thread1] Starting new replica set monitor for rs0/<IP_ADDRESS>:30080
2018-04-27T13:47:28.152-0400 I NETWORK  [thread1] Successfully connected to <IP_ADDRESS>:30080 (1 connections now open to <IP_ADDRESS>:30080 with a 5 second timeout)
2018-04-27T13:47:33.227-0400 I NETWORK  [thread1] getaddrinfo("sonar-mongodb-replicaset-0.sonar-mongodb-replicaset.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-04-27T13:47:38.233-0400 I NETWORK  [thread1] getaddrinfo("sonar-mongodb-replicaset-2.sonar-mongodb-replicaset.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-04-27T13:47:38.233-0400 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] getaddrinfo("sonar-mongodb-replicaset-1.sonar-mongodb-replicaset.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-04-27T13:47:38.233-0400 W NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Unable to reach primary for set rs0
2018-04-27T13:47:43.318-0400 I NETWORK  [thread1] getaddrinfo("sonar-mongodb-replicaset-0.sonar-mongodb-replicaset.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-04-27T13:47:48.321-0400 I NETWORK  [thread1] getaddrinfo("sonar-mongodb-replicaset-2.sonar-mongodb-replicaset.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-04-27T13:47:53.327-0400 I NETWORK  [thread1] getaddrinfo("sonar-mongodb-replicaset-1.sonar-mongodb-replicaset.default.svc.cluster.local") failed: nodename nor servname provided, or not known
2018-04-27T13:47:53.327-0400 W NETWORK  [thread1] Unable to reach primary for set rs0
2018-04-27T13:47:53.327-0400 E QUERY    [thread1] Error: Could not find host matching read preference { mode: "primary", tags: [ {} ] } for set rs0 :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
{code}
If you connect to mongodb without the replica set URL you get put on a random mongodb instance and can not be sure you are on master:
example:
{code}
awilhelm@MBP ~ $ mongo --host <IP_ADDRESS> --port 30080
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:30080/
MongoDB server version: 3.6.4
rs0:SECONDARY> ^C
bye
awilhelm@MBP ~ $ mongo --host <IP_ADDRESS> --port 30080
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:30080/
MongoDB server version: 3.6.4
rs0:PRIMARY> ^C
bye
awilhelm@MBP ~ $ mongo --host <IP_ADDRESS> --port 30080
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:30080/
MongoDB server version: 3.6.4
rs0:SECONDARY> ^C
bye
awilhelm@MBP ~ $ mongo --host <IP_ADDRESS> --port 30080
MongoDB shell version v3.6.3
connecting to: mongodb://<IP_ADDRESS>:30080/
MongoDB server version: 3.6.4
rs0:PRIMARY> ^C
bye
{code}

I am facing the same issue here trying to expose my replica set. Has anyone gotten ant better results or at least some auto ip discovery?

We did a workaround to be able to connect Studio3T to the replica set in our Network.

Expose the NodePort of each POD
Create a TCP Frontend in haproxy (our external balancer) for each k8s node and bind it to 27017 ( or whatever the port you configured Mongo)
Create a backend for each k8s node with a single server that connect to the Exposed port
Create a record in our internal DNS of the hostname of each POD
Modify the Replica Set in mongo to remove the "FQDN" and keep only the hostname
Make sure the computer that need to connect to the replica have a "domain search" of the DNS zone

I also use external-dns Helm to update the DNS zone (hosted in CloudFlare) to bind to the right Node. It should not change but if we recreate all POD and container start on another Node they will get updated in the DNS. Could be great to develop some thing to update the exposed port too but it's a great workaround for now.

@adrianliechti How would you get the external IP for nodes? Thanks in advance

@PavelYarysh If you are asking about my mongoS solution above and are running on a hardware based system. We use https://metallb.universe.tf/ to give external IP address. MetalLB is easy to setup and works like a charm!

I had an excellent suggestion from a Jody (can't see how to link slack name to github name for direct @) on the kubernetes slack channel.
The suggestion is to fix this issue with deploying Replica Sets in kubernetes by deploying a side car system along with the replica set that will query the MongoDB replica's to see which is master, take that information and update a kubernetes service that will always resolve to the ReplicSet master so you can write as we need.
This is a bit of a tall order but it does seem to give an elegant solution to the problem.
I do not have the time to attempt a PR but I will hopefully be able to set some aside in the coming weeks to investigate this solution.

A possible solution that I have been visualizing would be to use a Network Plugin with BGP support (for example, Calico to ensure external connectivity of each of the Replica Set.

you can create a service per mongodb replica with this trick. Creates a LoadBalancer service in this example
{{- $this := . -}}
{{- $count := int (index $this.Values "mongodb-replicaset" "replicas") -}}
{{ range $i, $e := until $count}}
apiVersion: v1
kind: Service
metadata:
  name: "{{ $this.Release.Name }}-mongodb-lb-{{ $i }}"
  labels:
    chart: {{ $this.Chart.Name }}-{{ $this.Chart.Version | replace "+" "-"  }}
    release: {{ $this.Release.Name | quote }}
    heritage: {{ $this.Release.Service | quote }}
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: {{ index $this.Values "mongodb-replicaset" "port" }}
    targetPort: {{ index $this.Values "mongodb-replicaset" "port" }}
  selector:
    app: mongodb-replicaset
    release: {{ $this.Release.Name | quote }}
    statefulset.kubernetes.io/pod-name: circleci-mongodb-{{ $i }}
---
{{ end}}

so any elegant solution to this ?

Here's how we made this work, without making any changes to the mongodb-replicaset chart. This should work also for the mongodb chart.
The relevant issue is clearly stated here:

Thus, replica set members must be reachable from the client by the hostnames listed in the replica set config.

There is no ready way that hostnames from one Kubernetes cluster will be resolvable from outside that cluster, not least from another cluster. Since this is a DNS resolution issue, we used DNS to solve it. YMMV and there may be other solutions that are better suited to your situation.
We run apps in one cluster and MongoDb in another cluster, so our client apps need to connect to a MongoDb replica set running a different cluster.
MongoDb cluster steps:

Create a Service for each Pod in the MongoDb StatefulSet.
Create a multi-service Ingress, with an external hostname for each internal Pod Service. The Ingress Controller must support multiple TCP services. We used Voyager, since it has this capability. However, each MongoDb Service needs to listen on a different port. We used: 27017, 27018, and 27019, with our 3 node replica set.

We wrote some helm charts to generate all these manifests, including the appropriate hostnames and port allocations.
Client cluster steps:

Edit the cluster DNS resolution mechanism to rewrite requests for the internal hostnames to the external hostnames. The internal hostnames will be the ones returned by the replica set config.
Configure client apps to use the external hostnames to connect to MongoDb. You can also create an ExternalName Service to simplify this.

We use CoreDNS for cluster DNS, which provides an easy way to do this using the rewrite plugin.
Here is an example of a modified CoreDNS configuration for the client cluster:
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
data:
  Corefile: |-
    .:53 {
        errors
        log
        health

        rewrite stop {
          name regex mongodb.svc.cluster.local mongodb.example.com
          answer name mongodb.example.com mongodb.svc.cluster.local
        }

        kubernetes cluster.local. in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        reload
    }

Please note that mongodb.svc.cluster.local and mongodb.example.com are not real DNS names. In reality, these are regular expressions that will match all possible internal hostnames for Pods in the MongoDb StatefulSet and use capture groups to generate the relevant request and answer sections of the DNS query. See the rewrite plugin docs for plenty of examples.
From a Pod running in the client cluster, using nslookup or dig on the internal hostnames  shows that these hostnames actually map to IP addresses associated with the external hostnames used in the Ingress resources created earlier, thus allowing clients to connect (assuming the two external networks are peered in some way). The rewrite rule is applied before any local cluster DNS resolution takes place. This approach works great for us!
If your clients are not running in a Kubernetes cluster, you should be able to use whatever DNS resolver mechanism at hand to do something similar.

I had an excellent suggestion from a Jody (can't see how to link slack name to github name for direct @) on the kubernetes slack channel.
The suggestion is to fix this issue with deploying Replica Sets in kubernetes by deploying a side car system along with the replica set that will query the MongoDB replica's to see which is primary, take that information and update the pod that is primary with a label denoting it as the primary. Subsequently cleaning the old pod of the label.

I have followed this nice suggestion and wrote some simple app in golang to do the job. You can find it here: https://github.com/combor/k8s-mongo-labeler-sidecar
Docker container can be found here: https://hub.docker.com/r/combor/k8s-mongo-labeler-sidecar

The best solution I found was to use a Network Plugin with BGP support (for example, kube-router) to ensure external connectivity to each node in the Replica Set.

We can solve this if we make one small change to the helm chart and create a custom service. First, the helm chart just needs to set the 0th pod to have a higher priority in on-start.sh so he is always elected as the primary. You can also do this manually by connecting to MongoDB and using rs.reconfig(). Second, a custom service can be created to target just this primary pod like so:
apiVersion: v1
kind: Service
metadata:
  name: mongodb-primary
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: mongodb-replicaset
    release: my
    statefulset.kubernetes.io/pod-name: my-mongodb-replicaset-0
  ports:
  - protocol: TCP
    port: 27017

The externalTrafficPolicy=Local avoids an extra hop and skips nodes that don't have the pod. The statefulset.kubernetes.io/pod-name will target a specific pod in the statefulset by its ordinal number. Does this sound reasonable @unguiculus ?

@selmison did you get the kube-router to work with this particular mongodb-replicaset helm chart?
if so I was curious as to how exactly

Hi, @kwill4026
I did it for advertised Node's POD cidr to the BGP peers with the args:

--advertise-pod-cidr=true
--cluster-asn=
--peer-router-asns=
--peer-router-ips=

With this I have access to the replica-set-members's ips.

@selmison can you provide me with your email address if you don't mind. I wanted to ask you some other questions to make this easier.

@selmison which daemonset yaml file are you using in the daemonset directory to configure kube-router.

Hy, @kwill4026
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
  labels:
    k8s-app: kube-router
    tier: node
  name: kube-router
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-router
      tier: node
  template:
    metadata:
      labels:
        k8s-app: kube-router
        tier: node
    spec:
      containers:
      - args:
        - --cluster-asn=64512
        - --peer-router-asns=64513
        - --peer-router-ips=10.x.X.X
        - --run-router=true
        - --run-firewall=true
        - --run-service-proxy=false
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: cloudnativelabs/kube-router:v0.2.5
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 20244
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 3
          successThreshold: 1
          timeoutSeconds: 1
        name: kube-router
        resources:
          requests:
            cpu: 250m
            memory: 250Mi
        securityContext:
          capabilities: {}
          privileged: true
          procMount: Default
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
        - mountPath: /etc/cni/net.d
          name: cni-conf-dir
        - mountPath: /var/lib/kube-router/kubeconfig
          name: kubeconfig
          readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      initContainers:
      - command:
        - /bin/sh
        - -c
        - set -e -x; if [ ! -f /etc/cni/net.d/10-kuberouter.conf ]; then TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;
          cp /etc/kube-router/cni-conf.json ${TMP}; mv ${TMP} /etc/cni/net.d/10-kuberouter.conf;
          fi
        image: busybox
        imagePullPolicy: Always
        name: install-cni
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/cni/net.d
          name: cni-conf-dir
        - mountPath: /etc/kube-router
          name: kube-router-cfg
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kube-router
      serviceAccountName: kube-router
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      - effect: NoSchedule
        key: node.kubernetes.io/not-ready
        operator: Exists
      volumes:
      - hostPath:
          path: /lib/modules
          type: ""
        name: lib-modules
      - hostPath:
          path: /etc/cni/net.d
          type: ""
        name: cni-conf-dir
      - configMap:
          defaultMode: 420
          name: kube-router-cfg
        name: kube-router-cfg
      - hostPath:
          path: /var/lib/kube-router/kubeconfig
          type: ""
        name: kubeconfig
  updateStrategy:
    type: OnDelete

thanks @selmison. how did you determine the values for

--cluster-asn=64512
--peer-router-asns=64513

I know for  peer-router-ips that is the ip address for the external router which in my case would be the k8s service I set up to be a nodePort or LoadBalancer.


Add gomobile

Description
It opens the door for generating Java and Obj-C bindings. Ultimately my goal is to continuously provide feedback about the usability of the SSI-SDK on mobile.
My next steps would be a greenfield app with examples that demonstrate all of the standards contained by the SSI-SDK (for ex. resolving identifiers, encrypting/decrypting, signing, etc...)
Here's an example using gomobile bind to convert the crypto package  to Java and Obj-C

Great start @nrobi144. Do you think it's worth adding a Mage script (e.g. mage mobile) to run the generation command?
And perhaps adding a doc or section to the README about using this functionality?

Codecov Report

Merging #178 (89d8628) into main (bee622c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #178   +/-   ##
=======================================
  Coverage   55.72%   55.72%
=======================================
  Files          38       38
  Lines        4163     4163
=======================================
  Hits         2320     2320
  Misses       1449     1449
  Partials      394      394

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Great start @nrobi144. Do you think it's worth adding a Mage script (e.g. mage mobile) to run the generation command? And perhaps adding a doc or section to the README about using this functionality?

@decentralgabe 💯 will do that today

@decentralgabe I've added the ios, android and mobile commands to Mage. This is my first exposure to Go and Mage, so any suggestions are welcome (especially on the error handling part).
FYI: I'm currently building the crypto, cryptosuite and did packages (those seemed good candidates for the API). So far only crypto seems to have exported names, is that because of the early stage of the SDK?

@decentralgabe I've added the ios, android and mobile commands to Mage. This is my first exposure to Go and Mage, so any suggestions are welcome (especially on the error handling part).
FYI: I'm currently building the crypto, cryptosuite and did packages (those seemed good candidates for the API). So far only crypto seems to have exported names, is that because of the early stage of the SDK?

I'm open to making more of the methods exported! I tried to keep the public API minimal and useful; however if there is utility to be had by making more public methods let's do it.

@nrobi144 Thank you so much! This is awesome!

That makes sense. I'd start with mirroring the ssi-sdk example package on mobile with a simplified UI. @decentralgabe what do you think? I assume examples will demo the API layer of the SSI-SDK so that could guide us what needs to be exported? 🤔

That makes sense. I'd start with mirroring the ssi-sdk example package on mobile with a simplified UI. @decentralgabe what do you think? I assume examples will demo the API layer of the SSI-SDK so that could guide us what needs to be exported? 🤔

yes I agree with that, and if you run into issues in your work, please open an issue here outlining which APIs you'll need to be made public and for which purpose and I can help get that done.


Types using variable-font-lock-face in if-let bindings

I noticed that after updating rust-mode types in if-let bindings now uses the variable font lock instead of the proper type font lock. Is this intended behavior? If so, for what reason? Cheers!

It's probably just a bug.  Can you post an example here?

Here's a screenshot:
http://i.imgur.com/zWlfD3J.png
The first binding is highlighted correctly, but in the if-let statement the Some should match highlighting of the first one.


Support for xp.linalg.lu (not yet part of the Array API spec)

Context adding LU factorization to the Array API spec:

https://github.com/data-apis/array-api/issues/627

Since numpy.linalg.lu does not exist yet (but scipy.linalg.lu does), @rgommers suggested working out an API candidate as part of array-api-compat first by reviewing the API choices and options implemented in various libraries targeting Array API support.
A short term implementation for numpy could probably delegate the work to scipy.linalg.lu in a first iteration.
Based on this work, numpy.linalg could subsequently be extended to directly implement the correct API to be submitted for a future revision of the spec.
Motivation: this is needed to add Array API support to the randomized linear algebra solver for the Principal Component Analysis estimator in scikit-learn:

https://github.com/scikit-learn/scikit-learn/pull/26315

Non exhaustive list of available implementations:

scipy.linalg.lu
cupyx.scipy.linalg.lu
torch.linalg.lu
jax.scipy.linalg.lu
dask.array.linalg.lu
...

Related methods in scipy:

scipy.linalg.lu_factor
scipy.linalg.lu_solve

Thanks @ogrisel! It looks like you only have a single usage in scikit-learn (here), with lu(x, permute_l=True), so if that is covered you should be good for now, right?
I think for the API design decisions, it's best to put them on https://github.com/data-apis/array-api/issues/627. And then let's implement it here. I'll try to post some thoughts there within the next few hours.

It looks like you only have a single usage in scikit-learn (here), with lu(x, permute_l=True), so if that is covered you should be good for now, right?

Yes, this is also what I found out via git grep.

That sounds like the right thing to do to me, we don't want this package to have a direct dependency on scipy.

Based on the scope of array-api-compat, lu should be added first to a draft version of the standard. Then we can add it here. It's better to discuss design specifics of it in the array-api repo. For actual implementations, it would be better to discuss that with specific libraries.
Regarding the scipy question, if lu were added, I wouldn't see an issue with making only that function (in the numpy wrapper) depend on scipy.

Closing this as it's out of scope here as long as lu isn't in the standard. The better place for it to go as long as it isn't is the array-api-extra package https://github.com/data-apis/array-api-extra


Next: Supporting jest v30

Jest 30 is coming soon and is having some breaking changes.
Aiming to support it, I'm going to make the arrangements for the future v17.

Due to https://github.com/jestjs/jest/pull/14460 requires at least Node 18.12

coverage: 100.0%. remained the same
when pulling b802ee54335634182bd9100830374abbabdc5729 on try-jest-30
into 1831f521c3363e99f113607cece4ac29a26b2459 on prep-v17.

Or, I can make it non-breaking by testing compatibility only on certain Node versions.

non-breaking now


Question: how to verify the extension is working properly?

Hi!
I'd like to verify how the extension is working, because I already installed it and after browsing for a while, I haven't seen any requests arriving to postwoman.io
First question probably, do I need to be logged in Github or Google to make it work properly?
Best,

Hi @cougil ,
Well, you can verify that extension is hooking with Postwoman by running Postwoman and opening up the log.
There should be a log message Connected to Postwoman Browser Extension vX.x indicating a successful hook.
No, you don't need to be logged in for the extension to work.
I didn't understand what you meant by "I haven't seen any requests arriving to postwoman.io", when the extension is installed, Postwoman routes the Network Requests to the extension and the extension makes the actual request, hence it won't be visible in the Network Tab in DevTools.
If you are getting Network Errors or are not able to see requests even after installing the extension, then please report that.
Thank you!

Thank you @AndrewBastin for your answer.
Just to clarify: I downloaded the Extension (Chrome) and I was expecting to catch/see the requests my browser could make when I browse around the web. I understand that the extension should grab all the requests and therefore I should be able to see all of them appearing there. Is this what this extension does or only records whatever request I send but only when I'm connected to "postwoman.io"? Because it is the last one, then it is not exactly what I was looking for ;-)
Best,

Well, yes, as a security measure, the extension from the Web Store and AMO only hooks to the official Postwoman URLs. (check manifest.json). If you want it to work with the other URLs, you can add your own URLs to manifest.json and use the extension that way.


Error: The getter 'qualityLabel' isn't defined for the class 'Muxed

: Error: The getter 'qualityLabel' isn't defined for the class 'MuxedStreamInfo'.
../…/controllers/pod_video_quality_controller.dart:101

'MuxedStreamInfo' is from 'package:youtube_explode_dart/src/videos/streams/muxed_stream_info.dart' ('../../../.pub-cache/hosted/pub.dartlang.org/youtube_explode_dart-1.9.10/lib/src/videos/streams/muxed_stream_info.dart').
package:youtube_explode_dart/…/streams/muxed_stream_info.dart:1
Try correcting the name to the name of an existing getter, or defining a getter or field named 'qualityLabel'.
        quality: int.parse(element.qualityLabel.split('p')[0]),
                                   ^^^^^^^^^^^^

If You are using youtube_explode_dart package then  please upgrade it to the latest version
or
delete pubspec.lock and run flutter pub get

@newtaDev I used only pod player

@newtaDev I used only pod player I tried delete .pub cache clean project, but still it's same error

delete pubspec.lock file from your project folder and run flutter pub get .
This should work


Cherry leaves turn black with this mod installed

hello! Having an odd issue where installing this mod turns cherry leaf blocks completely black in-world. The item texture is unaffected.

Minecraft Fabric 1.20.1, mod version 0.0.20

Need a full log and mod list

Sorry for the delay!
Log: https://mclo.gs/haY5OgD
mod list is listed on the log.

update: this seems to affect all of Photosynthesis' leaves

hmm. log doesn't say anything is wrong with my mod. like anywhere... however it does spew errors for other mods. there might be some other mods conflicting with each other which might cause this.

I’ll look into it via binary search, though I didn’t see any errors
relating to photosynthesis leaves when looking through the logs. This is
definitely an odd one.
On Sun, 18 Feb 2024 at 7:02 PM Martin Floden @.***>
wrote:

hmm. log doesn't say anything is wrong with my mod. like anywhere...
however it does spew errors for other mods. there might be some other mods
conflicting with each other which might cause this.
—
Reply to this email directly, view it on GitHub
https://github.com/Zuiron/Photosynthesis/issues/227#issuecomment-1951524777,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXU7W7RBOL4QJP2UB4GQ73LYUKQAXAVCNFSM6AAAAABCQFPWMSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJRGUZDINZXG4
.
You are receiving this because you authored the thread.Message ID:
@.***>

can you try with fancy graphics? when you can, I'm curious if that might be the cause xD

Unfortunately, that doesn't fix the issue :/
It doesn't seem to be the primary suspects (leaf culling via MoreCulling, or some kind of breakage from PassableLeaves), so I'll do a proper binary search tomorrow to find the suspect.

alright, if there is an issue on my end i would really like to know and fix it asap :)

I'm so so sorry for the delay- life got busy and I forgot to do this!
I can confirm that the issue happens even if Photosynthesis and its dependencies are the only mods loaded.

The log doesn't seem to show any issues save for some texture, sound, and recipe errors

Ah- alright, when Photosynthesis is the only mod, setting graphics to 'fancy' does fix the issue. Hopefully that helps! If it could be fixed for people with weaker computers that need the graphics on Fast, that would be excellent :D

This seems to only partially affect some leaves, and fully affect others. In all cases, it makes the normally transparent sections completely black.

From left to right: Apple, Banana, Cinnamon, Orange, Pear, Coconut, Avocado, Cherry (all-black), Lemon

ok i know exactly why this is happening now. its because of the custom leaves block with overlay. not sure why its blacked out but. i will definately investigate and see if it can be resolved. thanks for reporting this! its definitely on my end issue.

you know what. its actually not possible to fix this per say.
the only way to actually fix this. is to bundle an "resource pack" with the season support leaves separately. because the season overlay block ONLY works on fancy leaves.

I will actually do that because the season leaves are quiet resource heavy. thanks for this!

oh and just to mention it. why its only happening to a few and not others is because the season leaves thing i have only applied to a couple leaves, mainly vanilla and a couple of my own trees just to test it out :P

No worries! Thank you sm for the fix :D

issue now resolved with latest release. can download now at modrinth: https://modrinth.com/mod/photosynthesis/version/1.20.1-0.0.21

oh and. just dont use the optional builtin resoucepack on FAST graphics. you can now use fancy without issues :)


feat: Create dependant models for asset model

This PR creates models which are depended by the Asset model.
Created Models

Business Unit Model
Asset Type Model
Location Model
Memory Model

LGTM


Instructions for setting up local environment for contributors

Would it be possible to add instructions to the CONTRIBUTING or README doc to advise developers/potential contributors on how to run pronto using the latest code from the pronto repo?

@eliotsykes thanks for suggestion! 🙇


Create a policy to automatically import discovered ROSA clusters

This policy monitors DiscoveredCluster resources for ROSA clusters and configures matching discovered clusters to be automatically imported. Status is shown for any that have been configured to be automatically imported.  Filtering by the discovered cluster display name can be updated in the policy by editing the ConfigMap.
Refs:

https://issues.redhat.com/browse/ACM-10304

I added the filtering for the annotation that indicates when an import has already been attempted and therefore should now be skipped.

The related jira work item: https://issues.redhat.com/browse/ACM-11385

Putting a hold to get this approved now, but will keep it in progress until we get closer to releasing the implementation used by the policy.
/hold

/unhold
Making this available for QE testing.


HasDynamicArrays function

Not sure how to test this? I can see it returns true on my machine, as it should.

The only way to test this is:

Detect the Excel version in the unit test, and run a different assertion based on that.
Run the test on multiple Excel versions (e.g. multiple VMs with different Excel versions).

The code looks fine. Using static here is clever.

The only way to test this is:

Detect the Excel version in the unit test, and run a different assertion based on that.
Run the test on multiple Excel versions (e.g. multiple VMs with different Excel versions).

But for (1), we use this function to detect the Excel version :)

No, I'm talking about a unit test written in Python, which will have to detect it in some other way.


Index arguments are mandatory, causing problems with typings

What version of Effect is running?
2.0.3
What steps can reproduce the bug?
Create this kind of code:
import * as S from "@effect/schema/Schema";
import * as F from "effect/Function";
import * as A from "effect/ReadonlyArray";

const isMyThing = F.pipe(S.string, S.is);

A.takeWhile([false, "this-will-pass"], isMyThing);

What is the expected behavior?
I expect no compilation error.
What do you see instead?
You will get a compilation error No overload matches this call. for 2nd argument of A.takeWhile.
Additional information
This problem stems from the fact that takeWhile and a bunch of other functions have mandatory index parameter added to them in #1850. However this conflicts with optional AST.ParseOptions accepted by function returned by S.is.
On the other hand, this error makes sense, since takeWhile passes number as 2nd parameter, and not AST.ParseOptions. While making sense technically, the actual real-life usage scenarios of combining is and e.g. takeWhile become cumbersome, as then there is a need to add one extra import and extra explicit typing:
import * as S from "@effect/schema/Schema";
import * as F from "effect/Function";
import * as A from "effect/ReadonlyArray";
import * as P from "effect/Predicate"; // Extra import

// Add explicit typing to isMyThing to make compiler error go away
// (still not nice as takeWhile invokes callback with number as 2nd argument)
const isMyThing: P.Refinement<unknown, string> = F.pipe(S.string, S.is);

A.takeWhile([false, "this-will-pass"], isMyThing);

Is it possible to change S.is to NOT have 2nd optional parameter, and then introduce S.isWithOptions which does? Or maybe S.isPlain which does NOT have 2nd optional parameter, and keep S.is as-is? Or maybe make all index number parameter for takeWhile and friends optional?

Tacit usage is never advised as it can lead to bugs, in the above code while TS is fine you actually have a bug, given that isMyThing is not actually a simple predicate.
The right way to use this is:
import * as S from "@effect/schema/Schema"
import * as F from "effect/Function"
import * as A from "effect/ReadonlyArray"

const isMyThing = F.pipe(S.string, S.is)

A.takeWhile([false, "this-will-pass"], (a) => isMyThing(a))

Note that generally doing whatever.map(f) is considered a bad practice across effect, we should always make the call explicit, like whatever.map(x => f(x)) this will also improve significantly stack traces when things go wrong

I see this issue has been already resolved, but I still just add one thing:

Note that generally doing whatever.map(f) is considered a bad practice across effect,

I see this point, and it is fine if you guys have such rule. However, I am not writing code for effect libraries, but writing code for my own things, and I don't see any problems with passing function directly instead of making custom callback for every simple invocation, considering how much verbosity (and thus cognitive overload) it adds to the code. Also, imposing such "bad/good practice" things on users of the library is perhaps a bit off-putting for many developers.

I see this issue has been already resolved, but I still just add one thing:

Note that generally doing whatever.map(f) is considered a bad practice across effect,

I see this point, and it is fine if you guys have such rule. However, I am not writing code for effect libraries, but writing code for my own things, and I don't see any problems with passing function directly instead of making custom callback for every simple invocation, considering how much verbosity (and thus cognitive overload) it adds to the code. Also, imposing such "bad/good practice" things on users of the library is perhaps a bit off-putting for many developers.

This is generally not a good idea, not just for effect user code. It can introduce bugs all around due to implicitly passing dangling arguments to your callback function.
So in general, it is advisable to avoid tacit usage of function callbacks across api boundaries.

It can introduce bugs all around due to implicitly passing dangling arguments to your callback function.

Hmm, do you have any reference link to post or something with more information and specifics about this? 🤔


'cleancss' not recognized in command line

I installed cleancss with npm, but unlike everything else I install with npm when I try 'cleancss' in the command line it says it is an unrecognized program. I am on Windows 10.

It seems like your PATH may be missing an entry to ./node_modules/.bin, but that's just how it works on *nix systems, see http://stackoverflow.com/questions/29167575/nodejs-cant-run-package-windows?lq=1 and relevant for more.

Oh yes, I forgot about the -g option to install globally. The install instructions should mention that for cleancss. Why does it work fine for gulp and other things without -g but not cleancss?

Unfortunately I don't know. Have you installed cleancss and gulp via npm install ... or via package.json?


Upgrading to Babel6

TODO

[ ] ./bin/load_fixtures requires code from src/graphql
[ ] ./bin/update_schema requires code from src/graphql
[ ] wait for https://github.com/babel/babel/pull/3160

Upgrade instructions
After merging ripster with babel6 support you should review all your codebase:

Replace babel/polyfill to babel-polyfill.
Replace babel-core/register to babel-register.
Replace await* to await Promise.all().
Replace require() to require().default.

I consider it's better to use Promise.map (bluebird) instead of (Promise.all + map)

@yankouskia I don't like to use external libraries in cases when it could be done by standard API.

@vslinko are there conflicts with packages which prevent from using babel6.

Waiting for https://github.com/facebook/relay/pull/889 to release

yup. I made that (2 lines) change locally to try out.
in babel-relay-plugin/lib/babelAdapter.js
add line 18:  var babel = require('babel-core');
change line 22:   if (Plugin == null || /^6./.test(babel.version)) {


Slot props as variables

@lnr0626 I still need to implement the last changes to the <Context> API, which are renaming set to put, set to use and add the variable syntax to them. I'll work on that later. In the meantime, if you want to start taking a look at the changes, feel free to jump in. Pay attention that since there's no ContentHandler anymore, there might be more code to be removed.

@lnr0626 after updating the context's  API to use variables and playing around with use, I had the feeling it would be more intuitive to just keep get. So I changed set to put but left get unchanged.

There's definitely additional code we can remove - the first that comes to mind is the validate_interpolation code in Surface.Compiler.Helpers - that was intended to validate users were calling @inner_content correctly; however with the render_live change they no longer call it as a function. I'm sure there's other code, but nothing else jumps to mind right now


Add global switch to disable word wrapping and HTML labels

Is the proposed behavior of the global switch to "disable by default" or "disable unconditionally?" We'd like to prevent the existing default behavior of word wrapping and HTML labels by default, but still allow users to enable these features selectively.

Disbable by default.

Under Extras, Configuration enter {"simpleLabels": 1} or click here.


Upgrade marshmallow and marshmallow-sqlalchemy

User Story - Business Need
While working on #1222, upgrading marshmallow and marshmallow-sqlalchemy to the latest major versions resulted in successful build followed by runtime errors in multiple container.  The tracebacks concluded with ValueError: Couldn't import '<varies by container>': Invalid fields for <UserSchema(many=False)>: {'user_to_organisation', 'verify_codes_identity_provider_user_id', 'user_to_service'}.

[x] discuss w/ QA

User Story(ies)
As a VA Notify developer
I want to upgrade marshmallow and marshmallow-sqlalchemy
So that we minimize vulnerabilities and future upgrade pains.
Additional Info and Resources

marshmallow changelog
marshmallow-sqlalchemy changelog

Appears to be up to 9 classes/objects that would need tweaking:
Private Zenhub Image
There are numerous instances of this SAWarning which is coming from marshmallow-sqlalchemy; check to either make sure these aren't seen or refactor to remove these warnings.
/usr/local/lib/python3.10/site-packages/marshmallow_sqlalchemy/convert.py:170: SAWarning: relationship 'ScheduledNotification.notification' will copy column notifications.id to column scheduled_notifications.notification_id, which conflicts with relationship(s): 'Notification.scheduled_notification' (copies notifications.id to scheduled_notifications.notification_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards.   To silence this warning, add the parameter 'overlaps="scheduled_notification"' to the 'ScheduledNotification.notification' relationship. (Background on this error at: https://sqlalche.me/e/14/qzyx)

Engineering Checklist

[ ] Upgrade the effected packages in requirements-app.txt
[ ] Build a new Docker image
[ ] Run the app locally.  Observe failures.  Fix them.
[ ] Update requirements.txt as described in README.md
[ ] Update the Slide deck with the key win and a demo slide

Acceptance Criteria

[ ] The image builds
[ ] All unit tests pass with the latest versions of the effected packages

QA Considerations

[ ] QA regression passes

Potential Dependencies
Leave blank if n/a
Out of Scope
Do not upgrade Celery or SQLAlchemy.  Separate tickets exist for those upgrades.

Hey team! Please add your planning poker estimate with Zenhub @cris-oddball @justaskdavidb2 @k-macmillan @kalbfled @ldraney @nikolai-efimov

Please add your planning poker estimate with Zenhub @EvanParish

While working on #1225 @kalbfled realized we may need to also upgrade marshmallow to get past certain errors

Please see #1225 for comments relevant to this ticket.

There are currently 63 test failing -
FAILED<EMAIL_ADDRESS>- TypeError: 'NoneType' object is not subscriptable
FAILED tests/app/notifications/rest/test_send_notification.py::test_should_not_allow_template_from_another_service[email] - assert 400 == 404
FAILED tests/app/notifications/rest/test_send_notification.py::test_should_reject_email_notification_with_bad_email - TypeError: 'NoneType' object is not subscriptable
FAILED<EMAIL_ADDRESS>- TypeError: 'NoneType' object is not subscriptable
FAILED<EMAIL_ADDRESS>- TypeError: 'NoneType' object is not subscriptable
FAILED<EMAIL_ADDRESS>- TypeError: argument of type 'NoneType' is not iterable
FAILED tests/app/dao/test_templates_dao.py::test_get_template_versions - marshmallow.exceptions.ValidationError: {0: {'_schema': ['Invalid input type.']}, 1: {'_schema': ['Invalid input type.']}}

FAILED tests/app/service/test_rest.py::test_update_service_sets_volumes[None-200-None-volume_email] - assert 400 == 200
FAILED tests/app/service/test_rest.py::test_update_service_sets_volumes[None-200-None-volume_sms] - assert 400 == 200
FAILED tests/app/test_schemas.py::test_services_schema_includes_providers - KeyError: 'email_provider_id'
FAILED tests/app/template/test_rest.py::test_update_does_not_create_new_version_when_there_is_no_change - assert 2 == 1
============================= 63 failed, 2812 passed, 58 skipped, 17 xfailed in 63.87s (0:01:03) ==============================
Unit tests failed

PR approved and merged. All non-pinned dependencies were updated, but no Twistlock issues resolved. 1 dependabot issue was resolved.
Sending to Perf.


[Doc] Fix the broken markdown syntax in the user guide

The second section of the user guide shows the code snippet in a strange way, which appears to be not intended.
https://shader-slang.com/slang/user-guide/conventional-features.html

Also some of item-list doesn't look right.

This section should be deleted and replaced with text that is consistent with SP004 after its implementation lands.


Update the form "Update user Balance"

Can we add the "name of the user" in the message instead of (this is the user balance.......) so it will be (this is the "name" balance.......)
Can we remove all the text.
Can we add location/office at the top of the form.

For the ' office admin feature' it needs to be done in the backend, I'm working in it.

Should be done


refactor: Pull latest changes from main to Gapic update

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

[ ] Make sure to open an issue as a bug/issue before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
[ ] Ensure the tests and linter pass
[ ] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕
If you write sample code, please follow the samples format.

Reviewed the Tracing bits. Please run ITTracingTest.java and ITE2ETracingTest.java before merging.

Yes these tests have passed already: https://btx.cloud.google.com/invocations/0f4f7f69-09bf-4030-bca4-1832bfd4e2b0/targets/github%2Fjava-datastore%2Fgoogle-cloud-datastore%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.datastore.it.ITE2ETracingTest;config=default/details
https://btx.cloud.google.com/invocations/0f4f7f69-09bf-4030-bca4-1832bfd4e2b0/targets/github%2Fjava-datastore%2Fgoogle-cloud-datastore%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.datastore.it.ITTracingTest;config=default/details


Data get lost on heavy load via KafkaEngine

We have created Yandex tank scenario in order to test the database on heavy load.
We use KafkaEngine to populate the database. The problem is that the database starts loosing data at 100rps.
How to reproduce

Create kafka group named labeling_olap__kiz__current_state__all___test_shard_localhost
Create kafka topic named labeling_olap__kiz__current_state__all
Create the database schema defined in DB_SCHEMA.txt
Install Yandex tank
Use tank python script and configution files attached to this issue
Run tank via the following command:
sudo docker run     -v $(pwd):/var/loadtest     -v $SSH_AUTH_SOCK:/ssh-agent -e SSH_AUTH_SOCK=/ssh-agent     --net host     -it direvius/yandex-tank -c ch_kafka_conf.yaml

Expected behavior
The tank executes 100 requests, 30000 records each. So you should end up having 3000000 records via select count(), but you'll get less due to data loss.
To make sure that Kafka works as expected we used standard Kafka console consumer and it consumes exactly 3000000 messages.
Error message and/or stacktrace
There is nothing in server log file that corresponds to this issue.
Additional context
We do use the following versions of SW:

Kafka 2.11
Clickhouse 19.3.4 and the latest

ya_tank.zip
DB_SCHEMA.txt

ENGINE = VersionedCollapsingMergeTree(ch_sign, kiz_version)
PARTITION BY producer_inn__for_partition
ORDER BY (gtin, batch, kiz__sign, kiz_version)

I thought @den-crane helped yu to find the root of the problem in telegram chat?
As I understand the reason of "missing" rows was wrong order by columns in  VersionedCollapsingMergeTree? Or it was not the only problem?

VersionedCollapsingMergeTree is a very specific table engine. Do you have a reason to use it?
Does the issue persist if you use MergeTree?

I thought @den-crane helped yu to find the root of the problem in telegram chat?
As I understand the reason of "missing" rows was wrong order by columns in VersionedCollapsingMergeTree? Or it was not the only problem?

probably you've confused two different cases. It's another issue.
And probably it's not related to VersionedCollapsingMergeTree.
I advised to try TSV. @aegorov05 agreed to try.

VersionedCollapsingMergeTree is a very specific table engine. Do you have a reason to use it?
Does the issue persist if you use MergeTree?

Changing engine to standard MergeTree does not make any difference.

I thought @den-crane helped yu to find the root of the problem in telegram chat?
As I understand the reason of "missing" rows was wrong order by columns in VersionedCollapsingMergeTree? Or it was not the only problem?

probably you've confused two different cases. It's another issue.
And probably it's not related to VersionedCollapsingMergeTree (though I am not sure, because I don't really know how yaTank generates unique ORDER BY (gtin, batch, kiz__sign
I advised to try TSV. @aegorov05 agreed to try.

We have also tried using TSV format instead of JSONEachRow. Same result.
TSV Test.zip

VersionedCollapsingMergeTree is a very specific table engine. Do you have a reason to use it?
Does the issue persist if you use MergeTree?

Changing engine to standard MergeTree does not make any difference. Instead of 3 million records we only got 2941414.

So the difference is 2941414 - 3000000 = 58586  and it's smaller that 65535 (by default kafka_max_block_size = max_block_size = 65536). May be your last block with messages was not flushed yet? Can you recheck the result after 10 seconds of inactivity (by default  stream_flush_interval_ms = 7500 ms)?

I think kafka engine waits for {kafka_max_block_size} rows in blocked read and after that flushes by max_insert_block_size or 7500ms.

VersionedCollapsingMergeTree is a very specific table engine. Do you have a reason to use it?
Does the issue persist if you use MergeTree?

Changing engine to standard MergeTree does not make any difference. Instead of 3 million records we only got 2941414.

So the difference is 2941414 - 3000000 = 58586 and it's smaller that 65535 (by default kafka_max_block_size = max_block_size = 65536). May be your last block with messages was not flushed yet? Can you recheck the result after 10 seconds of inactivity (by default stream_flush_interval_ms = 7500 ms)?

I checked result after 10 second and more time. Count is not changed.

@aegorov05 try to write (3000 * 65536 + 1) = 196608001 rows

Why so many? :) Any any multiple of 65536 would be ok,  so for example 100 requests, 32768 messages each. It would be 3276800 messages in the table which is 50 blocks of size 65536.

I checked result after 10 second and more time. Count is not changed.

Somebody was reporting some issues with flushing by timeout. Need to check the code. Please ensure that after you fill last block to the required block size whole data appears.

@aegorov05 try to write (3000 * 65536 + 1) = 196608001 rows

Why so many? :) Any multiple of 65536 would be ok, so for example 100 requests, 32768 messages each. It would be 3276800 messages in the table which is 50 blocks of size 65536.

I checked result after 10 second and more time. Count is not changed.

Somebody was reporting some issues with flushing by timeout. Need to check the code. Please ensure that after you fill last block to the required block size whole data appears.

I used default kafka-consumer-groups for check current offset. Offset is committed, LAG = 0.

@abyss7 Is there any estimate/milestone when this bug is going to be fixed?

I had similar problems: a small amount of messages was lost.
It turned out that the messages were recorded in Kafka with incorrect timestamp of messages, instead of milliseconds, seconds were written.
Maybe you also write seconds to timestamps?
To check this, use kafkacat:
> kafkacat -b broker:9092 -t topic -o end -c1 -f 'timestamp: %T, message: %s'

I had similar problems: a small amount of messages was lost.
It turned out that the messages were recorded in Kafka with incorrect timestamp of messages, instead of milliseconds, seconds were written.
Maybe you also write seconds to timestamps?

No, we write ms to timestamp
But I think CH Kafka Consumer shouldn't rely on message timestamps, because  there are a lot messages with one timestamp (ms or even ns) on heavy load

@abyss7 Is there any estimate/milestone when this bug is going to be fixed?

I will investigate this next week - no estimation on the fix for now.

We ran into the same issue and managed to temporarily resolve it by reducing the kafka_max_block_size to 1. Could it be that the kafka wrapper is failing to successfully poll messages using a larger batch size but those failed messages are not being retried, leading to the data loss?

We have also encountered this issue. Some messages are silently lost without any logs. We have tried setting kafka_max_block_size  to 1, but the speed of loading data from Kafka drops below this solution being viable.
I think at this state, this issue prevents using of Kafka engine in production environment.

https://gist.github.com/k0st1an/7767a7584751a3cca8e9511b86e54465
Since version ' 19.3.3` a problem has appeared:
In Clickhouse v<IP_ADDRESS>:
953770

In Clickhouse v<IP_ADDRESS>:
996993

In Clickhouse v<IP_ADDRESS>:
955197

In Clickhouse v<IP_ADDRESS>:
981035

In Clickhouse v19.4.0:
974905

In Clickhouse v19.3.9:
977697

In Clickhouse v19.3.8:
979060

In Clickhouse v19.3.7:
996120

In Clickhouse v19.3.6:
934678

In Clickhouse v19.3.5:
962754

In Clickhouse v19.3.4:
945722

In Clickhouse v19.3.3:
983144

In Clickhouse v19.1.9:
1000000

https://gist.github.com/k0st1an/7767a7584751a3cca8e9511b86e54465
Since version 19.3.3 a problem has appeared:
In Clickhouse v<IP_ADDRESS>:
953770

In Clickhouse v<IP_ADDRESS>:
996993

In Clickhouse v<IP_ADDRESS>:
955197

In Clickhouse v<IP_ADDRESS>:
981035

In Clickhouse v19.4.0:
974905

In Clickhouse v19.3.9:
977697

In Clickhouse v19.3.8:
979060

In Clickhouse v19.3.7:
996120

In Clickhouse v19.3.6:
934678

In Clickhouse v19.3.5:
962754

In Clickhouse v19.3.4:
945722

In Clickhouse v19.3.3:
983144

In Clickhouse v19.1.9:
1000000

I discovered that about 1 row per million gets lost in Clickhouse v19.1.9.

@alexm93 you may be right, but I couldn't beat it:
In Clickhouse v<IP_ADDRESS>:
3916937

In Clickhouse v<IP_ADDRESS>:
3898375

In Clickhouse v<IP_ADDRESS>:
3878460

In Clickhouse v<IP_ADDRESS>:
3880372

In Clickhouse v<IP_ADDRESS>:
3886615

In Clickhouse v19.4.0:
3907914

In Clickhouse v19.3.9:
3910847

In Clickhouse v19.3.8:
3873083

In Clickhouse v19.3.7:
3871377

In Clickhouse v19.3.6:
3855757

In Clickhouse v19.3.5:
3947924

In Clickhouse v19.3.4:
3877515

In Clickhouse v19.3.3:
3839619

In Clickhouse v19.1.9:
4000000

In Clickhouse v19.1.8:
4000000

In Clickhouse v19.1.7:
4000000

In Clickhouse v19.1.5:
4000000

In Clickhouse v18.16.1:
4000000

In Clickhouse v18.16.0:
4000000

In Clickhouse v18.14.19:
4000000

In Clickhouse v18.14.18:
4000000

In Clickhouse v18.14.17:
4000000

In Clickhouse v18.14.15:
4000000

In Clickhouse v18.14.14:
4000000

In Clickhouse v18.14.13:
4000000

In Clickhouse v18.14.12:
4000000

In Clickhouse v18.14.11:
4000000

In Clickhouse v18.14.10:
4000000

In Clickhouse v18.14.9:
4000000

The test is awesome! Adding that all builds on 19.1 branch (incl <IP_ADDRESS>) is working correctly.

can anyone confirm that the problem is fixed in 19.7? My test shows that data is still being lost in 19.7

can anyone confirm that the problem is fixed in 19.7? My test shows that data is still being lost in 19.7

Can you please provide the setup of your testing stand? I want to know the scenario to reproduce it.

Setup is very simple.
I had a large topic in kafka with jsons.
I have created 4 identical tables in clickhouse (1 standalone server) with corresponding kafka tables and mat views that feed from mentioned kafka topic.
After consumption finished I observed gaps in data inserted into those 4 tables in random places. (-XXX) means difference from largest value for this time slot.
Note that some data misses exactly 10000 messages which I believe should be some internal block size.
toStartOfInterval(event_time, INTERVAL 2 hour)    browser_clicks.data1    browser_clicks.data2    browser_clicks.data3    browser_clicks.data4
------------------------------------------------  ----------------------  ----------------------  ----------------------  ----------------------
2019-05-27 11:00:00                               16721458                16721458                16721458                16721458
2019-05-27 13:00:00                               44402325                44392325 (-10000)       44392325 (-10000)       44402325
2019-05-27 15:00:00                               49636872                49636872                49636872                49636872
2019-05-27 17:00:00                               35820136 (-14117)       35834253                35834253                35824253 (-10000)
2019-05-27 19:00:00                               43301844                43294694 (-7150)        43297684 (-4160)        43301844
2019-05-27 21:00:00                               39652534 (-4482)        39654166 (-2850)        39657016                39657016
2019-05-27 23:00:00                               16238829                16238829                16238829                16238829
2019-05-28 01:00:00                               4989495                 4989495                 4989495                 4989495
2019-05-28 03:00:00                               2643050                 2643050                 2643050                 2643050
2019-05-28 05:00:00                               6543845                 6543845                 6543845                 6543845
2019-05-28 07:00:00                               30924932 (-1401)        30926333                30923877 (-2456)        30926333
2019-05-28 09:00:00                               49297574                49292770 (-4804)        49287877 (-9697)        49289918 (-7656)
2019-05-28 11:00:00                               39687344 (-581)         39683783 (-4142)        39687925                39687925
2019-05-28 13:00:00                               43923416 (-7768)        43931184                43929293 (-1891)        43929894 (-1290)
2019-05-28 15:00:00                               48901709                48897484 (-4225)        48899138 (-2571)        48901709
2019-05-28 17:00:00                               34616002                34610704 (-5298)        34612315 (-3687)        34616002
2019-05-28 19:00:00                               41605188 (-1979)        41603572 (-3595)        41607167                41599980 (-7187)
2019-05-28 21:00:00                               34555756 (-4430884)     38939622 (-47018)       38986640                38939883 (-46757)
2019-05-28 23:00:00                               14378553 (-2052894)     16426578 (-4869)        16431447                16431092 (-355)

First attempt to reproduce was unsuccessful - no data lost.

After consumption finished
How did you make sure that consumption is finished on all tables? Can you provide tail (~1000 lines) of the logs to the moment when you do a check on number of messages?
Can you try the same scenario with kafka setting kafka_commit_every_batch=0 ?
Also, how many rows do you have per message?

Closing for now, since there were no other reports.

I also noticed that kafka engine lost data approximately 1%. If create another kafka engine and read data from beginning offsets again all data is written to table untli lag is not disappeared. Once lag is disapeared kafka engine lost data again.
Error log contains many
2019.08.29 13:20:27.778190 [ 33 ] {}  StorageKafka (queue_debug): Re-joining claimed consumer after failure
2019.08.29 13:20:27.778950 [ 33 ] {}  void DB::StorageKafka::streamThread(): std::exception. Code: 1001, type: cppkafka::HandleException, e.what() = Local: Timed ou
t, version = <IP_ADDRESS>

@borismol can you please check it on 19.11.9 (just released, it has important Kafka fixes)?

@filimonov i reproduced issue with <IP_ADDRESS>

@filimonov i reproduced issue with <IP_ADDRESS>

It means nothing.
The only stable release having latest Kafka fixes is 19.11.9. It is pull requests
#6805 and #6597, merged 3 days ago.
You can also pick latest 19.14 version, for example v<IP_ADDRESS>4-prestable, or wait few days till Kafka fixes will be ported to 19.13 too.


WebApiToSwaggerGenerator.GenerateForController() NullReferenceException with no helpful message when type not found in assembly

The following should (imho) throw an exception with an error message like $"Type {type} not found in assembly ${assemblyPath}" (after line 77 of WebApiToSwaggerGenerator.cs) , but instead throws a generic NullReferenceException at line 81 (because controllerType is null):
        var settings = new WebApiAssemblyToSwaggerGeneratorSettings
        {
            AssemblyPath = @"value/path/to/assembly.dll",
            DefaultUrlTemplate = "api/{controller}/{action}/{id}"
        };

        var generator = new WebApiAssemblyToSwaggerGenerator(settings);
        var swaggerService = generator.GenerateForController("NonExistingClass");

I couldn't figure out the cause of the NullReferenceException by using the NSwag nuget package; I had to download the source code and include it in my solution so that I could debug the code. Hope this helps. Thanks for the great project.

Now throwing a TypeLoadException. Thanks for using the project.
https://github.com/NSwag/NSwag/commit/34a924b1c66133183bba6a8fbee204f48bcca52f

Awesome, thanks!
On Thu, Feb 4, 2016 at 1:47 PM, Rico Suter<EMAIL_ADDRESS>wrote:

Now throwing a TypeLoadException. Thanks for using the project.
34a924b
https://github.com/NSwag/NSwag/commit/34a924b1c66133183bba6a8fbee204f48bcca52f
—
Reply to this email directly or view it on GitHub
https://github.com/NSwag/NSwag/issues/35#issuecomment-180020563.

Pushed to nuget => https://www.nuget.org/packages/NSwag.CodeGeneration/1.34.5878.35657


Segmentation fault when a JSON string with whitespaces is passed to boost::json::parse() function

Description
As described in the title.
Version of Boost: 1.80.0
Operating system: Ubuntu 22.04.3 LTS (64-bit)
Compiler: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
CMake: 3.22.1
Link to the ready-to-build project with reproduction example (just run ./run.sh):
https://github.com/Elnee/minimal-reproduction-for-boost-issue
Minimal reproduction example:
#include <bits/stdc++.h>
#include <boost/json.hpp>

int main() {

  const std::string json = R"({"example": "foo"})"; // NOTE: The whitespace here between colon and quotation mark.
  const auto &object = boost::json::parse(json).as_object();

  try {
    // Crash appears on the next line while executing the `at()` function.
    std::cout << object.at("example").as_string().c_str() << std::endl;
  } catch (const std::exception &ex) {
    std::cout << "Exception: " << ex.what() << std::endl;
  }

  return 0;
}

Output:
Segmentation fault (core dumped)

json::parse returns a json::value, which you don't store anywhere. You instead get a reference to its subobject of type json::object. At the end of that statement the value goes out of scope and destroys its internal object. After that you use a dangling reference. In order to fix this you should store either the value or the object:
const auto value = boost::json::parse(json);
const auto& object = value.as_object();
// or
const auto object = boost::json::parse(json).as_object();

@grisumbras Thank you, it works. But I don't understand why using a const-ref doesn't expand the lifetime of that object.
As I see boost::json::value has a union returned by as_object(). Is it because its lifetime is bound to the instance of boost::json::value?

And also, isn't it a bit misleading that I should think about as_object() method in such a way? I thought about it as getting the same object but in a different representation (due to the as prefix). It turned out to return some internals to me.
Doesn't it force me to know some internal library implementation details aside from the interface as a user?

@Elnee Per [class.temporary] p4, temporary objects are destroyed after evaluating the full-expression in which they appear (which in this case is the initialization of object). When a temporary object is bound to a reference, it's lifetime will be extended to that of the reference if, and only if, one of the conditions specified by  [class.temporary] p6 are met. Binding a reference to the result of calling a non-static member function of a temporary object does not satisfy any of these conditions, so the lifetime is not extended.

@sdkrystian Many thanks for the detailed explanation.


Implemented drawDecHL and drawDecACIX

Its not ready to merge quite yet, I still want to clean up drawDecACIX.

Code style and such looks perfect. Let me know when you're ready to have this merged.

Its ready to merge

Thanks!


add geometry type

Build Failed!
The last (useful) 2 log lines are shown below:
View Full Log
Cloning into 'mysqldump'...
fatal: reference is not a tree: a7f3ad8e63304f4f946f4a7b8b66f588e99cb696

Unfiltered log

[Container] 2018/10/14 19:13:59 Waiting for agent ping
[Container] 2018/10/14 19:13:59 Waiting for DOWNLOAD_SOURCE
[Container] 2018/10/14 19:13:59 Phase is DOWNLOAD_SOURCE
[Container] 2018/10/14 19:13:59 CODEBUILD_SRC_DIR=/codebuild/output/src461964318/src/github.com/assignar/ci
[Container] 2018/10/14 19:13:59 YAML location is /codebuild/output/src461964318/src/github.com/assignar/ci/buildspec.yml
[Container] 2018/10/14 19:13:59 Processing environment variables
[Container] 2018/10/14 19:13:59 Decrypting parameter store environment variables
[Container] 2018/10/14 19:13:59 Moving to directory /codebuild/output/src461964318/src/github.com/assignar/ci
[Container] 2018/10/14 19:13:59 Registering with agent
[Container] 2018/10/14 19:13:59 Phases found in YAML: 1
[Container] 2018/10/14 19:13:59  BUILD: 1 commands
[Container] 2018/10/14 19:13:59 Phase complete: DOWNLOAD_SOURCE Success: true
[Container] 2018/10/14 19:13:59 Phase context status code:  Message:
[Container] 2018/10/14 19:14:00 Entering phase INSTALL
[Container] 2018/10/14 19:14:00 Phase complete: INSTALL Success: true
[Container] 2018/10/14 19:14:00 Phase context status code:  Message:
[Container] 2018/10/14 19:14:00 Entering phase PRE_BUILD
[Container] 2018/10/14 19:14:00 Phase complete: PRE_BUILD Success: true
[Container] 2018/10/14 19:14:00 Phase context status code:  Message:
[Container] 2018/10/14 19:14:00 Entering phase BUILD
[Container] 2018/10/14 19:14:00 Running command node ./test.js
Cloning into 'mysqldump'...
fatal: reference is not a tree: a7f3ad8e63304f4f946f4a7b8b66f588e99cb696
child_process.js:644
    throw err;
    ^

Error: Command failed: git checkout a7f3ad8e63304f4f946f4a7b8b66f588e99cb696
    at checkExecSyncError (child_process.js:601:13)
    at Object.execSync (child_process.js:641:13)
    at execSync (/codebuild/output/src461964318/src/github.com/assignar/ci/test.js:7:18)
    at Object.<anonymous> (/codebuild/output/src461964318/src/github.com/assignar/ci/test.js:54:1)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)

[Container] 2018/10/14 19:14:02 Command did not exit successfully node ./test.js exit status 1
[Container] 2018/10/14 19:14:02 Phase complete: BUILD Success: false
[Container] 2018/10/14 19:14:02 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: node ./test.js. Reason: exit status 1
[Container] 2018/10/14 19:14:02 Entering phase POST_BUILD
[Container] 2018/10/14 19:14:02 Phase complete: POST_BUILD Success: true
[Container] 2018/10/14 19:14:02 Phase context status code:  Message:

Click here to retry the build

Build Failed!
The last (useful) 2 log lines are shown below:
View Full Log
Cloning into 'mysqldump'...
fatal: reference is not a tree: a4480f7b08916657e268d067178275cf6be1d5f8

Unfiltered log

[Container] 2018/10/23 20:40:45 Waiting for agent ping
[Container] 2018/10/23 20:40:45 Waiting for DOWNLOAD_SOURCE
[Container] 2018/10/23 20:40:45 Phase is DOWNLOAD_SOURCE
[Container] 2018/10/23 20:40:45 CODEBUILD_SRC_DIR=/codebuild/output/src240065650/src/github.com/assignar/ci
[Container] 2018/10/23 20:40:45 YAML location is /codebuild/output/src240065650/src/github.com/assignar/ci/buildspec.yml
[Container] 2018/10/23 20:40:45 Processing environment variables
[Container] 2018/10/23 20:40:45 Decrypting parameter store environment variables
[Container] 2018/10/23 20:40:45 Moving to directory /codebuild/output/src240065650/src/github.com/assignar/ci
[Container] 2018/10/23 20:40:45 Registering with agent
[Container] 2018/10/23 20:40:45 Phases found in YAML: 1
[Container] 2018/10/23 20:40:45  BUILD: 1 commands
[Container] 2018/10/23 20:40:45 Phase complete: DOWNLOAD_SOURCE Success: true
[Container] 2018/10/23 20:40:45 Phase context status code:  Message:
[Container] 2018/10/23 20:40:45 Entering phase INSTALL
[Container] 2018/10/23 20:40:45 Phase complete: INSTALL Success: true
[Container] 2018/10/23 20:40:45 Phase context status code:  Message:
[Container] 2018/10/23 20:40:46 Entering phase PRE_BUILD
[Container] 2018/10/23 20:40:46 Phase complete: PRE_BUILD Success: true
[Container] 2018/10/23 20:40:46 Phase context status code:  Message:
[Container] 2018/10/23 20:40:46 Entering phase BUILD
[Container] 2018/10/23 20:40:46 Running command node ./test.js
Cloning into 'mysqldump'...
fatal: reference is not a tree: a4480f7b08916657e268d067178275cf6be1d5f8
child_process.js:644
    throw err;
    ^

Error: Command failed: git checkout a4480f7b08916657e268d067178275cf6be1d5f8
    at checkExecSyncError (child_process.js:601:13)
    at Object.execSync (child_process.js:641:13)
    at execSync (/codebuild/output/src240065650/src/github.com/assignar/ci/test.js:7:18)
    at Object.<anonymous> (/codebuild/output/src240065650/src/github.com/assignar/ci/test.js:54:1)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)

[Container] 2018/10/23 20:40:48 Command did not exit successfully node ./test.js exit status 1
[Container] 2018/10/23 20:40:48 Phase complete: BUILD Success: false
[Container] 2018/10/23 20:40:48 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: node ./test.js. Reason: exit status 1
[Container] 2018/10/23 20:40:48 Entering phase POST_BUILD
[Container] 2018/10/23 20:40:48 Phase complete: POST_BUILD Success: true
[Container] 2018/10/23 20:40:48 Phase context status code:  Message:

Click here to retry the build

I should probably polish this up a bit I did this for a company I work for and I thought I would just be a matter of pushing to the fork and be done with it.
I'll make the necessary changes and push again

@pbassut is this merge going to be ready soon?
cheers!


[PAGOPA-1755] fix-company-name-banca-del-veneto-centrale-and-cassa-primiero-belluno

List of Changes
It's necessary to fix the company name related to two PSP as requested by the following ticket:

https://pagopa.atlassian.net/browse/PTPO-135
https://pagopa.atlassian.net/browse/PTPO-170

Motivation and Context
Fix requested by PSP in order to generate pdf receipt with the right company name in the footer
How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

@aferracci we need update this cfg too