add armhf/raspberry pi example stack

This is a working example app for the armhf platform, including docker stack support for a cluster of Raspberry Pis. I tried to stick to the "official" armhf/ images on the Docker Hub.
Currently, some images are pulled from my private Docker Hub repo (gesellix/sample-*) to make the docker-stack work out of the box. I guess it would be nice to provide those via the official dockersamples/ repo, but that can obviously be fixed later.
Please note:

The .Net worker hasn't been converted, yet.
I changed the building of the Java worker to be a two step process now (1. building the .jar, 2. wrapping it in a JRE-image). The good thing is that it shows how to split build and runtime images, the bad thing is that multiple steps are probably harder to grasp for newbies?

@gesellix hi, are you also planning on creating a .NET core worker image to complete the stack?

@ManoMarks I'd also recommend reducing the parallelism factor to 1 for all the containers for the Raspberry Pi. I am not even sure all the containers plus the .NET core code can fit inside the RAM of all Raspberry Pis with a full OS and Docker.
With the initial test (docker stack deploy) I didn't see the votes being incremented - probably because of the worker?
Btw.. the worker's docker ps logs says:
Waiting for redis
Waiting for redis
Waiting for redis
Waiting for redis
Waiting for redis
Waiting for redis
Waiting for redis
Waiting for redis

I can test across a multi-node cluster once we've iterated on a few of the comments.

We should also call these files armhf in the same way the docker/docker repo does. Someone will probably help us out with an aarch64 / arm64 version in the future. Maybe @vielmetti? :-)

It strikes me that this is a good reason for multiarch style images, so that docker-compose works independent of architecture, and the porting necessary is just in building the containers and not stitching them together.
Is there a query you can run on docker hub that returns only multiarch manifests?

@vielmetti like mentioned in my initial thoughts (#63) and seen in the popular portainer project, I'd also wish for an easier way of producing multiarch images. As far as I understood, we need to build native images on their respective platforms before combining them into a multiarch "view", right? Not sure whether this is currently possible in an automated way.

@alexellis I'll check if the .NET worker is possible. I haven't tried, yet.
I'll also check what happened to the Java worker waiting for redis. I swear it worked on my machine/cluster... but not anymore.

@gesellix - I was thinking of Portainer too as a model for this as far as multiarch goes.

So where are we on this? @alexellis do we have an LGTM?

@manomarks I'm working on some suggestions by the reviewers.
The most notable blocker seems to be the fact that referenced images should be available in an official repo. Did I get this right?

@ManoMarks several comments we need to work with. We need to iterate over it.

Update:

[x] Renamed tags and file infixes to armhf
[x] The docker-compose file uses local build contexts
[x] The Java worker Dockerfile is now available as combined builder+runtime file - I'll delete the others (builder, runtime) then, since we settle on the existing approach?

Not addressed/fixed, yet:

[ ] Postgres not available as official image, pending https://bugs.alpinelinux.org/issues/6372
[ ] The other example images need to be available on a docker or dockersample repo
[ ] I'm debugging the bug @alexellis found where votes aren't written to the db.

@tianon - can you open a new bug with stack trace from the core dump to replace
https://bugs.alpinelinux.org/issues/6372 ? Timo suggests that it's a different issue than the one I opened in October that was resolved.

https://bugs.alpinelinux.org/issues/6795 :+1:

Is there another relational database that we can use to replace PostGres on Alpine? Alternately, can we use a non-alpine version of Postgres?

@ManoMarks - there is https://github.com/kiasaki/docker-alpine-postgres which is a Dockerized Alpine Postgres - you might find that useful.
However, the bug 6795 reported above is now marked as fixed so this might be a sufficiently solved problem to make progress.

@alexellis, @gesellix is this moving forward?

@ManoMarks yes, I'm still working on it. I got a flu, though, and I also stumbled on issues like this one: https://github.com/rsyslog/rsyslog/issues/35
Sorry for the delay!

@gesellix No worries! I hope you feel better! I just wanted to make sure this wasn't getting lost.

Is there a postgres or mysql version in Raspbian aka (Debian) repo(s)?
https://wiki.debian.org/PostgreSql

local images for the stack

You mean without central registry? Would that work with multiple nodes, then?
I would hope for official images to be available at some point in time. The dockersamples repo might be considered as trusted repos?
Local images would keep bandwidth usage low, though. Hm.

Minor update: I'm struggling with the error below, leading to a 100% cpu usage of the kworker process. It seems like that error only appears after deploying the example stack (not instantly, but after a few minutes), so maybe it's a Docker on Raspberry Pi issue, or something triggered by one of the services. If some of you have an idea how to find the root cause, I'd be very glad :-)
/cc @alexellis @StefanScherer @DieterReuter
[ 1328.937557] BUG: using smp_processor_id() in preemptible [00000000] code: node/7528
[ 1329.030770] caller is debug_smp_processor_id+0x18/0x24
[ 1329.122947] CPU: 3 PID: 7528 Comm: node Not tainted 4.4.43-hypriotos-v7+ #1
[ 1329.214980] Hardware name: BCM2709
[ 1329.300230] [<800193e4>] (unwind_backtrace) from [<800149e0>] (show_stack+0x20/0x24)
[ 1329.390209] [<800149e0>] (show_stack) from [<8033613c>] (dump_stack+0xbc/0x108)
[ 1329.478209] [<8033613c>] (dump_stack) from [<80350c94>] (check_preemption_disabled+0x104/0x134)
[ 1329.565476] [<80350c94>] (check_preemption_disabled) from [<80350cdc>] (debug_smp_processor_id+0x18/0x24)
[ 1329.651826] [<80350cdc>] (debug_smp_processor_id) from [<7f3fd50c>] (ip_vs_schedule+0x180/0x738 [ip_vs])
[ 1329.739200] [<7f3fd50c>] (ip_vs_schedule [ip_vs]) from [<7f40b590>] (tcp_conn_schedule+0x118/0x22c [ip_vs])
[ 1329.827818] [<7f40b590>] (tcp_conn_schedule [ip_vs]) from [<7f3fef78>] (ip_vs_in.part.2.constprop.9+0x6e4/0x75c [ip_vs])
[ 1329.918360] [<7f3fef78>] (ip_vs_in.part.2.constprop.9 [ip_vs]) from [<7f3ff074>] (ip_vs_local_request4+0x40/0x44 [ip_vs])
[ 1330.010632] [<7f3ff074>] (ip_vs_local_request4 [ip_vs]) from [<8050dca8>] (nf_iterate+0x80/0x90)
[ 1330.104027] [<8050dca8>] (nf_iterate) from [<8050dd38>] (nf_hook_slow+0x80/0xec)
[ 1330.194149] [<8050dd38>] (nf_hook_slow) from [<8051a654>] (__ip_local_out+0xb4/0xc0)
[ 1330.284487] [<8051a654>] (__ip_local_out) from [<8051a684>] (ip_local_out+0x24/0x4c)
[ 1330.377252] [<8051a684>] (ip_local_out) from [<8051a9ac>] (ip_queue_xmit+0x144/0x3c0)
[ 1330.471462] [<8051a9ac>] (ip_queue_xmit) from [<80532dd8>] (tcp_transmit_skb+0x4d0/0x918)
[ 1330.566692] [<80532dd8>] (tcp_transmit_skb) from [<80534a14>] (tcp_connect+0x584/0x798)
[ 1330.664182] [<80534a14>] (tcp_connect) from [<80537b90>] (tcp_v4_connect+0x2fc/0x468)
[ 1330.760989] [<80537b90>] (tcp_v4_connect) from [<8054fab0>] (__inet_stream_connect+0x1a4/0x330)
[ 1330.859171] [<8054fab0>] (__inet_stream_connect) from [<8054fc80>] (inet_stream_connect+0x44/0x58)
[ 1330.960026] [<8054fc80>] (inet_stream_connect) from [<804bdec8>] (SyS_connect+0x74/0xa4)
[ 1331.059682] [<804bdec8>] (SyS_connect) from [<8000fc20>] (ret_fast_syscall+0x0/0x1c)

hmmm, looks similar: https://github.com/docker/docker/issues/27833

@gesellix Regarding building multiarch images. I've created a Travis matrix build with a deploy script that waits for the other builds and drafts and pushes the multiarch image.

@StefanScherer Nice, thanks for the link, I'll give it a try!

After the findings in https://github.com/docker/docker/issues/27833#issuecomment-282675416 I tried to deploy the stack on Raspbian instead of Hypriot, but now I hit this error:

"starting container failed: subnet sandbox join failed for "<IP_ADDRESS>/24": error creating vxlan interface: operation not supported"

Seems to be a known issue:

https://github.com/docker/libnetwork/issues/1166
https://github.com/docker/machine/issues/2753

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: active
 NodeID: ffvx1pznkxxx0345t5128xmbm
 Is Manager: true
 ClusterID: 09k04xtm1rq8lgbp5vsx59kjq
 Managers: 1
 Nodes: 5
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: <IP_ADDRESS>
 Manager Addresses:
  <IP_ADDRESS>:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Kernel Version: 4.4.38-v7+
Operating System: Raspbian GNU/Linux 8 (jessie)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 925.5 MiB
Name: pi1
ID: HFBQ:H5X6:VW6K:MNLM:JYJR:KLB5:PJXF:Y2L6:5LSV:64KE:XSOO:FGCF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpuset support
Labels:
 os=linux
 arch=arm
Experimental: true
Insecure Registries:
 <IP_ADDRESS>/8
Live Restore Enabled: false

Using Hypriot 1.4.0 with a recent Kernel works for @StefanScherer (https://github.com/docker/docker/issues/27833#issuecomment-288163486), and myself without any of the mentioned Kernel issues. 🎉
Some votes still aren't handled by the worker, as it seems. I'll try to find out what happens in such cases. I suspect this is an issue of the example stack in general, and no specific issue of the armhf stack. Multiple redis containers with only a single worker (watching a single redis instance) might be the reason...
The images still need to be pushed to an official repo. I'm happy to help with that, but any pointers how to start that process would be nice. I'll have to talk to @tianon I guess?

Now I got the error messages again (via dmesg):
[ 2102.873233] BUG: using smp_processor_id() in preemptible [00000000] code: node/3348
[ 2102.873250] caller is debug_smp_processor_id+0x18/0x24
[ 2102.873269] CPU: 2 PID: 3348 Comm: node Not tainted 4.4.50-hypriotos-v7+ #1
[ 2102.873279] Hardware name: BCM2709
[ 2102.873306] [<80019468>] (unwind_backtrace) from [<80014a14>] (show_stack+0x20/0x24)
[ 2102.873331] [<80014a14>] (show_stack) from [<803362dc>] (dump_stack+0xbc/0x108)
[ 2102.873358] [<803362dc>] (dump_stack) from [<80350e34>] (check_preemption_disabled+0x104/0x134)
[ 2102.873381] [<80350e34>] (check_preemption_disabled) from [<80350e7c>] (debug_smp_processor_id+0x18/0x24)
[ 2102.873443] [<80350e7c>] (debug_smp_processor_id) from [<7f40bb90>] (ip_vs_in.part.2.constprop.9+0x2fc/0x75c [ip_vs])
[ 2102.873573] [<7f40bb90>] (ip_vs_in.part.2.constprop.9 [ip_vs]) from [<7f40c074>] (ip_vs_local_request4+0x40/0x44 [ip_vs])
[ 2102.873656] [<7f40c074>] (ip_vs_local_request4 [ip_vs]) from [<8050e014>] (nf_iterate+0x80/0x90)
[ 2102.873699] [<8050e014>] (nf_iterate) from [<8050e0a4>] (nf_hook_slow+0x80/0xec)
[ 2102.873735] [<8050e0a4>] (nf_hook_slow) from [<8051a9d8>] (__ip_local_out+0xb4/0xc0)
[ 2102.873777] [<8051a9d8>] (__ip_local_out) from [<8051aa08>] (ip_local_out+0x24/0x4c)
[ 2102.873821] [<8051aa08>] (ip_local_out) from [<8051ad30>] (ip_queue_xmit+0x144/0x3c0)
[ 2102.873854] [<8051ad30>] (ip_queue_xmit) from [<80533178>] (tcp_transmit_skb+0x4d0/0x918)
[ 2102.873890] [<80533178>] (tcp_transmit_skb) from [<80533730>] (tcp_write_xmit+0x170/0xe9c)
[ 2102.873924] [<80533730>] (tcp_write_xmit) from [<80534774>] (__tcp_push_pending_frames+0x44/0xb0)
[ 2102.873946] [<80534774>] (__tcp_push_pending_frames) from [<80522f78>] (tcp_push+0x130/0x158)
[ 2102.873966] [<80522f78>] (tcp_push) from [<805265c8>] (tcp_sendmsg+0xc8/0xa6c)
[ 2102.873987] [<805265c8>] (tcp_sendmsg) from [<80550170>] (inet_sendmsg+0xa8/0xd0)
[ 2102.874010] [<80550170>] (inet_sendmsg) from [<804bd220>] (sock_sendmsg+0x24/0x34)
[ 2102.874033] [<804bd220>] (sock_sendmsg) from [<804bd2c4>] (sock_write_iter+0x94/0xc4)
[ 2102.874056] [<804bd2c4>] (sock_write_iter) from [<8016ae18>] (__vfs_write+0xb8/0xe8)
[ 2102.874077] [<8016ae18>] (__vfs_write) from [<8016b658>] (vfs_write+0xa0/0x1a8)
[ 2102.874096] [<8016b658>] (vfs_write) from [<8016bf78>] (SyS_write+0x4c/0xa0)
[ 2102.874122] [<8016bf78>] (SyS_write) from [<8000fc40>] (ret_fast_syscall+0x0/0x1c)

The current Raspbian (2017-03-02-raspbian-jessie-lite) works without Kernel errors.
Kernel Version: 4.4.50-v7+
Operating System: Raspbian GNU/Linux 8 (jessie)

 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:32:23 2017
 OS/Arch:      linux/arm
 Experimental: true

I created an issue for Hypriot: https://github.com/hypriot/image-builder-rpi/issues/166
Let's focus on Raspbian and the other steps to move forward with this PR.

Any update on this? Should I close this PR?

I'm going to close this PR. Feel free to re-open if you want to try again.


Multus interface is missing if openshift cluster is powercycled

In OKD 3.11 I have created deployment which integrated multus-cni annotation. It was observed that if we do powercycle of cluster the multus interface is missing OR simply gets deleted whenever POD comes up after powercycle. Is this expected behaviour ? POD definitions still had annotations defined however multus interface was missing. And after explicit delete of pod would get interface created.
Steps to Reproduce -

oc exec -it test-pod -- ifconfig [multus/2nd interface is present]
Powercycle complete cluster
oc exec -it test-pod -- ifconfig [multus/2nd interface is missing]

Multus Version - 3.4.1
OpenShift Version -
oc v3.11.0+4986039-380
kubernetes v1.11.0+d4cacc0

Hi @RiteshWalia -- unfortunately, Kubernetes v1.11 is below our supported version for Multus, so I don't believe we're likely to dig in to find the core issues here. However, it's plausible Multus will work with it, but, it would be usage at your own risk.
I do have some good news for you, though. With OKD 4.x, Multus CNI is installed by default and configured by OKD for you.
Something that I might recommend is that you enable logging: https://github.com/intel/multus-cni/blob/master/doc/configuration.md#logging
This might give you some insight as to what's happening. If there's no logging after the reboot, it may be that somehow Multus is not enabled. I have some previous experience OKD 3.11, and I believe that the openshift-sdn configuration will probably want to own the primary on-disk CNI configuration on each node, and it won't likely have Multus configured. So it's possible that the pods come up too early, before Multus is configured, and then Multus isn't executed during the cluster reboot.

Thanks @dougbtv . I'll mark this close. However if there is some interesting findings I'll update in the thread.


Do not use node types

e.g. in https://github.com/vega/svelte-vega/blob/26045afad2ecbeda19dde3c61fc42eba1745dfb4/packages/storybook/stories/ChangingDimensionDemo.svelte#L16
Should be ReturnType<typeof setTimeout>. Might have to add the DOM lib (https://www.typescriptlang.org/tsconfig#lib)

:tada: This issue has been resolved in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


PoS initialization / SmesherService does not work properly (got stuck?)

Description
If you run the Node with valid smeshing opts in the node config (aka it will start smeshing on run) and you don't have a PoS data yet — then your Node will start generating it. However, it won't respond on SmesherService.StopSmeshing (and keep sending you PostSetupStateStream updates, with PostStatus === In_progress).
Such behavior makes Smapp GUI looks like "it is not responding". For example, if you click on "Pause smeshing" or "Delete PoS data" buttons (actually both of them calls SmesherService.StopSmeshing with a different deleteFiles flag).
Another symptom is that if you try to close the Node it will be ending the process for too long (Smapp kills it by timeout)...
Steps to reproduce

Clean ~/post
Add such smeshing-settings into your node config:{
    "smeshing-coinbase": "stest1qqqqqqp8r9d6yzz8kqc3e9t0rgtlu8jwhlhyc4gm2xaut",
    "smeshing-opts": {
        "smeshing-opts-datadir": "~/post/data",
        "smeshing-opts-maxfilesize":<PHONE_NUMBER>,
        "smeshing-opts-numunits": 4,
        "smeshing-opts-provider": 1,
        "smeshing-opts-throttle": false
    },
    "smeshing-start": true
}

Run the node
Callgrpcurl --plaintext localhost:9092 spacemesh.v1.SmesherService.PostSetupStatusStream

And you'll get something like:{
 "status": {
   "state": "STATE_IN_PROGRESS",
   "opts": {
     "dataDir": "/Users/username/post/data",
     "numUnits": 4,
     "maxFileSize": "2147483648",
     "computeProviderId": 1
   }
 }
}

Pay attention, that the response does not contain numLabelsWritten, but it should contain it. So it might be a clue :)
Call in another shell:grpcurl -d "{\"deleteFiles\":false}" --plaintext localhost:9092 spacemesh.v1.SmesherService.StopSmeshing

It is expected to have an answer with empty status obj, but actually, this request gets stuck...
Wait for a while (I didn't find what exactly changes, but seems like Node goes to another "state" and then everything becomes fixed). It takes about 10-20 minutes on my machine. Also, I've noticed that usually, it starts sending numLabelsWritten, and my PoS file jumps from zero bytes to 16.8 MB :)

Interesting moment, that if you repeat everything, but set smeshing-start: false on the second step, and then call StartSmeshing with the same opts — everything will work well (but I'm not sure it is actually generating PoS). Let's call it a "workaround" just for reference :)
MAYBE it is happening because I'm using CPU as PoS data provider (go-spacemesh is constantly using about 99-103% CPU on my machine).
Actual Behavior

Requests to SmesherService.StopSmeshing does nothing
There is no numLabelsWritten in PostSetupStateStream
But after some time (actually, generated at least some PoS data? It's my assumption) — it fixes

Expected Behavior

StopSmeshing stops it at any time or at least reacts well (and stops sending "Progress" state)
numLabelsWritten should proceed :)

Environment
Please complete the following information:

OS: macOS 12.3 (but I'm afraid this issue valid for other platforms as well)
Node Version: 0.3.0-beta.0

Additional resources
Application logs (while it "stuck"):
15:44:17.369 › SmesherService, in grpc PostDataCreationProgressStream, output: {"state":2}
15:44:22.728 › SmesherService.StopSmeshing called:   {"deleteFiles":false}
15:44:40.867 › SmesherService.StopSmeshing called:   {"deleteFiles":false}
15:44:47.369 › SmesherService, in grpc PostDataCreationProgressStream, output: {"state":2}

Application logs (how it should be, got it using "workaround" described above):
15:48:34.961 › SmesherService.StartSmeshing called:   {"coinbase":{"address":"stest1qqqqqqp8r9d6yzz8kqc3e9t0rgtlu8jwhlhyc4gm2xaut"},"opts":{"dataDir":"/Users/brusher/post/eeb970d3","numUnits":4,"maxFileSize":2147483648,"computeProviderId":1,"throttle":false}}
...
15:48:35.193 › SmesherService, in grpc PostDataCreationProgressStream, output: {"state":2,"numLabelsWritten":{"low":1048576,"high":0,"unsigned":true}}
15:48:35.917 › SmesherService, in grpc PostDataCreationProgressStream, output: {"state":2,"numLabelsWritten":{"low":1048576,"high":0,"unsigned":true}}
15:49:06.185 › SmesherService, in grpc PostDataCreationProgressStream, output: {"state":2,"numLabelsWritten":{"low":1048576,"high":0,"unsigned":true}}
15:49:06.909 › SmesherService, in grpc PostDataCreationProgressStream, output: {"state":2,"numLabelsWritten":{"low":1048576,"high":0,"unsigned":true}}
...
15:49:45.574 › SmesherService.StopSmeshing called:   {"deleteFiles":false}
15:49:45.908 › SmesherService, in grpc PostDataCreationProgressStream, output: {"state":3,"numLabelsWritten":{"low":1048576,"high":0,"unsigned":true}}
15:49:46.184 › SmesherService, in grpc PostDataCreationProgressStream, output: {"state":3,"numLabelsWritten":{"low":1048576,"high":0,"unsigned":true}}
...

@brusherru actually the smeshing can be stopped when batch is finished. When batch is in progress then you cannot stop it (that's how gpu-post works)
What GPU was used for the test?
Cc @piersy

@brusherru were you able to check that with smaller batch size? Does it behave better then?

On my machine using a smaller batch size is fixing this. I used the smallest batch size of 8 and smeshing seemed to stop almost immediately.

yes makes sense :)


Upgrades not Working

In the upgrades code in main.js line 65 to line 149. The upgrades and not the buildings are not showing on the screen. Checking the console returns no error as an error would break the game. Fixing this would be splendid.

I think the reason why is that when you got the image files in the JS object "image" you didn't use the correct file path, and when getting the images I think you have to use something like "img/imagename.svg" or something like that. If you use the wrong file path then the images dont show

The img tag is not even showing though. :/

@PANDASARECOOOL I have confirmed the source is correct.

@PANDASARECOOOL FIXED ISSUE


Compile issue with Fortran bindings using IBM XL

See xsdk-project/xsdk-issues#94

@dalg24 or @aprokop : I'm making time to debug this on Summit, but the build documentation seems out of date -- even the note in the docs about the configure possibly being out of date is out of date 😆 ("You can always refer to scripts/docker_cmake" which no longer exists.)
Can I replicate this with trilinos as a TPL? Is there an existing trilinos%xl installation on summit? (If so, I can't find it.) Thanks!

@dalg24 or @aprokop : I'm making time to debug this on Summit, but the build documentation seems out of date -- even the note in the docs about the configure possibly being out of date is out of date 😆 ("You can always refer to scripts/docker_cmake" which no longer exists.)
Can I replicate this with trilinos as a TPL? Is there an existing trilinos%xl installation on summit? (If so, I can't find it.) Thanks!

I don't think any of us have built trilinos/dtk/arborx with xl on Summit. I'm pretty sure current version of arborx cannot even be built with xl (see arborx/ArborX#132). There's simply lack of interest in working around the fact that XL does not properly support C++14.
If we want to move forward, however, we need to figure out first which version of DTK was compiled in the xSDK issue. Judging by the Trilinos version there (12.18.1) and the dtk variants resources in the Spack's Trilinos, it should be dtk 3.1-rc1 (edfa050cd46e2274ab0a0b7558caca0079c2e4ca).

I don't think any of us have built trilinos/dtk/arborx with xl on Summit. I'm pretty sure current version of arborx cannot even be built with xl (see arborx/ArborX#132). There's simply lack of interest in working around the fact that XL does not properly support C++14.
If we want to move forward, however, we need to figure out first which version of DTK was compiled in the xSDK issue. Judging by the Trilinos version there (12.18.1) and the dtk variants resources in the Spack's Trilinos, it should be dtk 3.1-rc1 (edfa050cd46e2274ab0a0b7558caca0079c2e4ca).

So if Trilinos and DTK C++ won't build with XL, why is the complaint about the fortran wrapper failure?

So if Trilinos and DTK C++ won't build with XL, why is the complaint about the fortran wrapper failure?

So if Trilinos and DTK C++ won't build with XL, why is the complaint about the fortran wrapper failure?

Trilinos may build. And it's possible that that version of DTK may also build, but not certain. I cannot say anything for certain.

So if Trilinos and DTK C++ won't build with XL, why is the complaint about the fortran wrapper failure?

Trilinos may build. And it's possible that that version of DTK may also build, but not certain. I cannot say anything for certain.

I would just ask @luszczek for his configuration with spack.

I would just ask @luszczek for his configuration with spack.

DataTransferKit is going to be archived.


Avoid discarding error when failing to parse a path

If I make a mistake in the path in the argument of import_token_attr like this:
diff --git a/tests/test_macros/src/lib.rs b/tests/test_macros/src/lib.rs
index cacd071..faebae8 100644
--- a/tests/test_macros/src/lib.rs
+++ b/tests/test_macros/src/lib.rs
@@ -58,7 +58,7 @@ pub fn some_other_macro(tokens: TokenStream) -> TokenStream {
     .into()
 }

-#[import_tokens_attr(middle_crate::export_mod::sub_mod::macro_magic)]
+#[import_tokens_attr(middle_crate:export_mod::sub_mod::macro_magic)]
 #[proc_macro_attribute]
 pub fn distant_re_export_attr(attr: TokenStream, tokens: TokenStream) -> TokenStream {
     let imported_item = parse_macro_input!(attr as Item);

then the attribute is simply discarded without warning or error.
Having the attribute ignore is confusing and people can wonder why macro_magic is not found at the path they are expected because they made a typo in the path.

This has been published in the v0.3.5 release


Handle schema mismatch repairs, RTS txn and queue ops retries, queue flush side effects

This PR implements the following changes:

In the DB client API, in addition to the main return status (computed from the majority replies based on each RPC's semantics), also returns to client, for queue ops and txn commits, the minority failure status, if such a status exists, in variable called minority_status. This is useful to advise the client (RTS) to re-create missing or mismatching schema on minority nodes as they re-join the quorum.

This addresses the following problem: In the current RTS/DB implementation, while schema tables are automatically created on servers upon a DB node startup, schema mismatches can still happen in the form of missing actor input queues on DB servers that freshly joined a quorum (since the set of actors is dynamic, not part of the static schema). This caused the issue that newly restarted DB nodes were not able to re-join the quorum properly, as they missed this schema.
By the changes in this PR, the RTS now monitors the returned minority_status from all relevant operations (queue operations and txn commits) and triggers schema recreation on minority (or even majority) nodes if schema mismatches are observed.

In the RTS, we also handle DB ops and / or txn retries when either:
2.1) Transactions are aborted by the servers (e.g. due to serializability conflicts with other txns)
2.2) There is no quorum temporarily for a RPC (servers do not respond)
2.3) There is a quorum, but there are schema mismatches on a majority of servers (in this case operations are re-tried after schema recreation as well)

Also in relation to 2.1), this PR refactors the function FLUSH_outgoing(), which is flushing an actor's outgoing messages after running a continuation to all destination actors affected, in a single txn (that also includes the actor's state update and input queue consume). In the previous form, the function could produce side effects in the local RTS in-queues of the other actors, if the DB txn containing the update is aborted by the system (e.g. due to concurrency failures), then retried until success (effects on local queues were not rolled back upon a txn abort). To solve this, this PR refactors FLUSH_outgoing in 2 separate functions, one that retries the DB transaction to the global queues first until success (FLUSH_outgoing_db), producing no effects on the local queues. Only after the global txn succeeds, the local queues are updated via the second function (FLUSH_outgoing_local).

Fixes a segfault in handle_txn_commit() due to unknown txn_ids when nodes rejoin a quorum with no knowledge of an ongoing txn

Note: In the current form, only the DB operations that could actually produce schema mismatches in the current implementation are handled (minority_status set in the DB client API and schema repairs + retried triggered in the RTS). These are the queue operations (read_queue, consume, enqueue, subscribe etc), as well as the txn commit. Schema inconsistencies would only happen for these operations in the current implementation, because they stem from actor input queues being unknown to DB servers that freshly join a quorum (the set of actors is learned dynamically only once at RTS start up).
The other DB operations the RTS uses (e.g. inserts) would not currently produce schema inconsistencies, since the table schema is automatically created at server startup.
However, for completeness, I will update this PR before merging to also handle the remaining DB operations, like inserts, similarly (add the minority_status variable in the API, and also handle the other types of retries in the RTS, e.g. lack of quorum, for them).
Marking this PR as draft until this is completed.

Very cool! While I will read your code (not today though) I think I should actually spend more time in improving the test case to include various error scenarios, like partitioning db servers.
Have you manually tested stuff or do you have some test script or some process you could share?

If you add handling of schema errors for inserts, doesn't that mean we could start db servers with no schema and we would then create the schema from RTS as part of error handling for the first failed insert? That sounds pretty appealing to me! Only creating schema on startup is brittle and places heavy assumptions on environment. No assumptions and dealing with errors in generic ways make for robust programs!

Very cool! While I will read your code (not today though) I think I should actually spend more time in improving the test case to include various error scenarios, like partitioning db servers.
Have you manually tested stuff or do you have some test script or some process you could share?

Yes, it would be very cool to have regression tests include partition scenarios, including some sequences of partitions which could cause issues.
To test, I used the test/count.act actor, and generated partitions by killing servers and/or the RTS in some sequences of failures, while making sure state from the actor is correct throughout (counter increases without skipping any numbers), and there are no crashes on either servers or RTSs.
Also, that the RTS blocks when a quorum is lost and it resumes when a quorum is re-gained.
And also, that servers and RTSs re-join the views in the gossip protocol when they come up and are marked as dead in gossip messages when they die. The gossip views and messages are printed in server logs, as well as RTS logs currently.
Some interesting sequences of failures to try are:

Killing one DB server (quorum remains live)
Bringing that server back (should re-join view as live, as well as quorum - respond to queries)
Killing the RTS and bringing it back (should continue counting from where it left off)
Killing one DB server, killing the RTS, bringing the RTS up, bringing the DB server up (restore state from an incomplete replication set)
Killing one DB server, killing the RTS, bringing the DB server up, bringing the RTS up (restore state from a replication set that includes servers that have been restarted)
Scenarios 4) and 5) but with killing 2 servers, along with the RTS (here the RTS should block while there is no quorum, but recreate schemas on re-joining servers, unblock and resume counting from where it left off once there is a quorum again)
Scenarios 4) and 5) but with killing all 3 servers, but not killing the RTS. Here, the RTS should block while there is no quorum and resume counting after there is. However, if in addition we kill the RTS everything, it will currently resume from the beginning, since there is no disk persistence.

If you add handling of schema errors for inserts, doesn't that mean we could start db servers with no schema and we would then create the schema from RTS as part of error handling for the first failed insert? That sounds pretty appealing to me! Only creating schema on startup is brittle and places heavy assumptions on environment. No assumptions and dealing with errors in generic ways make for robust programs!

Right, since lack of schema errors will be propagated for inserts now, it would be possible to use that to trigger tables for complete schema creation as well. Just to note that right now what the RTS triggers when is started up is only the creation of input queues for actors (since the set of actors is dynamic and only known by the RTS). The "static" parts of the schema (tables for actor state, table of messages etc) are independently created by DB servers when they start up (similarly to a pre-installed schema server-side), so they currently involve no communication between RTS and DB servers at initial wire-up. We can discuss further options during the meeting.


Set up nightly runs of window fuzzer test

Description
This is the parent issue of setting up nightly run of velox_window_fuzzer_test in GitHub Actions.
Error Reproduction
No response
Relevant logs
No response

This is blocked by https://github.com/facebookincubator/velox/issues/9375.


"Brazii de Sus", "Bătești", "Brazii de Jos", "Negoiești", "Popești", "Stejaru" are listed in the drop-down menu

Description:
When user writes "Brazii de Sus", "Bătești", "Brazii de Jos", "Negoiești", "Popești" and "Stejaru" the locations appear in the drop-down menu next to the comune and the county witch is a part according to the law
Precondition:
The website is up an running.

Step 1
Write in the search bar "Brazii de Sus"
Expected results
The location appears in the drop-down menu as "Sat Brazii de Sus, PRAHOVA (Brazi)"

Step 2
Press "x" button.
Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Bătești"
Expected results
The location appears in the drop-down menu as "Sat Bătești, PRAHOVA (Brazi)"

Step 4
Press "x" button.
Expected results
The location was deleted from the search bar.

Step 5
Write in the search bar "Brazii de Jos"
Expected results
The location appears in the drop-down menu as "Sat Brazii de Jos, PRAHOVA (Brazi)"

Step 6
Press "x" button.
Expected results
The location was deleted from the search bar.

Step 7
Write in the search bar "Negoiești"
Expected results
The location appears in the drop-down menu as "Sat Negoiești, PRAHOVA (Brazi)"

Step 8
Press "x" button.
Expected results
The location was deleted from the search bar.

Step 9
Write in the search bar "Popești"
Expected results
The location appears in the drop-down menu as "Sat Popești, PRAHOVA (Brazi)"

Step 10
Press "x" button.
Expected results
The location was deleted from the search bar.


Bracketing comparison

Having the ability to set the viewer in a "bracketing mode" which would allow the hash pairs (selectable? all?) to be bracketed by a given range. This way, on each generation you could preview what things look like to either side of a given set of features or parameters at once without needing to adjust sliders manually.
An artist on ArtBlocks developed this in their own local test environment...

This is too much work for the potential benefit it would gain. Much easier to just manually map a slider to the feature and click around to see the results.


⚠ Bofrost.fr has degraded performance

In b9f50af, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1361 ms

Resolved: Bofrost.fr performance has improved in 27e7f39 after 6 minutes.


Replace indy sdk with indy vdr

Indy VDR is replacing Indy SDK, does it have all the features needed for AFJ? can/should we start moving to VDR?

To be a bit more precise, Indy SDK is being replaced by all three of Indy VDR / Indy Shared RS and Aries Askar.

Yes @swcurran is correct. Work was done and Askar and vdr are done. However credx is not done yet (very very close). However we did not have the time yet to fix it and integrate it into AFJ. When everything is modularised it should be easier to integrate so it might still take some time for full integration of the shared components.


README: Link Contributors

This is more or less a proposal, to promote the contributors of the server a bit more.

...no


Model layout broken with Django 2.1

Django 2.1 prevents models from having more than one primary key. It seems it only checks this in the system check:
examples.LdapGroup: (models.E026) The model cannot have more than one field with 'primary_key=True'.

This breaks not only the composite RDN feature but django-ldapdb as a whole, because django-ldapdb always has two primary keys on the model (dn and the selected RDN field).

The good news is: This can be circumvented by adding this to settings:
SILENCED_SYSTEM_CHECKS = ['models.E026']

The bad news is: This does not make me comfortable as a fix ;).
The awkward thing is that apparently, multi0column primary keys are long planned to be supported, and now they go and declare them as an error: https://code.djangoproject.com/wiki/MultipleColumnPrimaryKeys

Another approach would be to use a new, own attribute to designate RDN fields, but I'm not keen in such a major rewrite.


Add authors.md to the packages (UPM and UnityPackage)

With the recent changes to our documentation process, we would like to add the authors list to our package distributions.
This should be easily doable in a potential 2.7.1 release.

Fixed by #9922


Please update DBeaver to 6.2

https://dbeaver.io/2019/09/08/dbeaver-6-2/

It's being worked on over at https://github.com/lukesampson/scoop-extras/pull/2831.

Sorry didnt see that


Alias Import Duplication

Describe the bug
After exporting to bulk edit aliases quick and reimporting it has created a duplicate of most aliases (for me around 2,500)
To Reproduce
Steps to reproduce the behavior:

export alias csv
edit alias colum
import cdv

Expected behavior
only alias name to be updated, not new ones created, only edited 50 aliases but duplicated over 2,000
Environment information:

OS: Rasberry Pi

Additional context
Currently can't select then shift to scroll down and select last alias to bulk select either for quick fix in the alias page and have to select individually to delete

I haven't seen this one before, any chance i can get a copy of your export and import csv's to compare and see why it's not picking them up?
My gut feeling is you edited them in excel and dropped the leading 0 off the from of the capcode...

I haven't seen this one before, any chance i can get a copy of your export and import csv's to compare and see why it's not picking them up?
My gut feeling is you edited them in excel and dropped the leading 0 off the from of the capcode...

Replicated it for interest's sakes
DanrwAU is correct-
The export CSV when opened in Excel drops the leading zero/s. This results in the import process thinking it's a new record on re-import due to the primary key being different and inserting a new record without the leading zero/s.

Replicated it for interest's sakes
DanrwAU is correct-
The export CSV when opened in Excel drops the leading zero/s. This results in the import process thinking it's a new record on re-import due to the primary key being different and inserting a new record without the leading zero/s.

closing this task, thankyou @DanrwAU @J1GGLYPUFFS
For anyone that finds this for future reference and imported already and created new records just edit the database table and delete everything after a certain line (the last manual addition before trying the import) and this will clean up your alias list

closing this task, thankyou @DanrwAU @J1GGLYPUFFS
For anyone that finds this for future reference and imported already and created new records just edit the database table and delete everything after a certain line (the last manual addition before trying the import) and this will clean up your alias list


Review comments on s3fs lib code

[x] reginClass -> change name location constraint
[ ] iam endpoint get through parameter
[ ] error out if no endpoint is set
[ ] check aws from iks cluster
[ ] https://github.ibm.com/alchemy-containers/armada-storage-s3fs-plugin/wiki/Access-AWS-S3-from-IKS-Cluster
[ ]   params := req.GetParameters() not valid in DeleteVolume method
https://pkg.go.dev/github.com/container-storage-interface/spec/lib/go/csi#DeleteVolumeRequest

locationConstraint name change made
iam endpoint is required in session creation. Since GetParameter is not defined method for DeleteVolume, this need to be passed from secret
Error out if no cos-endpoints are set in secret
cos endpoints can be set from secrets similar to iam endpoint


Add percentile information to data analysis object?

Issue Description
Running DataAnalysis now takes a number for histogram buckets.
In the case of highly skewed/unbalanced data (say percentages where most values are between 99.00 and 100.00 and very sparse elsewhere) most of the data ends up in one bucket.
Perhaps we can add in something that will also give us percentile information.
As an eg.
Data: 0,10,10,20,50,99,100,100,101,102
percentilePointInterval = 10;
DataAnalysis dataAnalysis = AnalyzeSpark.analyze(schema, parsedInputData, maxHistogramBuckets, percentilePoint);
And the dataAnalysis information contains the information for the 10th 20th .. 100th percentile
Version Information
Please indicate relevant versions, including, if relevant:

Deeplearning4j version - 0.8.1-SNAPSHOT

Contributing
Not urgent. Functionality in place with boolean conditioning etc.

Added long ago via TDigest: https://github.com/deeplearning4j/DataVec/blob/ec4c2418367a7bf9a50a72ae1efec81896fc5b08/datavec-api/src/main/java/org/datavec/api/transform/analysis/columns/NumericalColumnAnalysis.java#L49


Persist key-value-store TTLs to disk in flush()

Turns out the TTL functionality I added needs the ability to persist to disk.  Do this by piggybacking on the existing persistence code; writing an optional absolute TTL to disk and then converting it back to relative form for the ttl_manager to use.

/retest

/retest

/retest

/retest

/retest

/retest


Need a way to deal with objects changing state (i.e. arrays)

For example
define_state(:foo) { ['joe', 'fred'] }

def add_friend(friend)
  foo << friend
end

This does not work as foo << friend updates the array object, but there is no way to know that the foo state has been updated.
I have implemented a couple of solutions, and am interested in which people would prefer:

define foo! method that will return an object that wrap the current value of foo, and observe any changes to that object.   So you can write

def add_friend(friend)
  foo! << friend
end

define foo! method that simple does foo = foo to indicate a state change.  This would return self so that the methods could be chained

def add_friend(friend)
  foo << friend; foo!
end

or
def add_friend(friend)
  foo << friend
ensure
  foo!.bar!
end

2b) In addition the Proc class can be updated so this works
lambda { |new_friend| foo << new_friend }.foo!

Just looking for people's input, preferences, or other ideas!

Thanks for pointing this out, I think this could be a common mistake given that the original setState is wrapped inside the setter.
Currently, you can write as below,
def add_friend(friend)
  self.foo = foo << friend; self
end

To get the benefit of brevity and also make it chainable.

I understand that you do as you said...
My intention is to add a solution that will read cleaner and be shorter.
I have implemented the solutions above, but feel that just adding the foo! method is the easiest and cleanest.
I.e. foo! << friend is equal to foo << friend; self.foo = foo
If I do a pull request with this will you accept it?

I think I get your point now, looks like a nice to have feature. Feel free to open up a pull request :+1:

i've made some more improvements to this... expect a pull request soon, but first I am integrating with some real code to see how everything feels.
basically there is now a foo! method that can be either used as foo!('new value') (equivilent to self.foo = 'new value')
or if no parameter is provided then foo!() returns an object that will update after any method calls on foo.
this shortens things up nicely.
In addition I added two class methods require_param and optional_param that will declare a parameter. The difference is these methods will also create methods that can read the parameters directly.
I.e. require_param :foo, type: String will require a foo parameter and will create a foo method that returns that parameter.
Finally if the type of parameter is Proc then the created method will call the proc.  So
require_param :foo, type: Proc
then
foo 1,2,3
will send 1,2,3 to whatever Proc foo got bound too.
Altogether these changes clean things up so that things read really nicely.
Oh yeah two other things:
You can say class MyComponent < React::Component::Base instead of including React::Component
And any class that responds to 'render' can be used in the rendering dsl.  So you can say
div do
  my_component param_1: 'Joe', param_2: 'Fred'
end

As long as an instance of MyComponent responds to render...
How does all that sound :-)

Hey, thanks for taking time making this suggestion.

or if no parameter is provided then foo! returns an object that will update after any method calls on foo.

How is this going to be implemented? A wrapper object which will delegate all method invocations?

In addition I added two class methods require_param and optional_param that will declare a validated parameter. The difference is these methods will also create methods that can read the parameters directly.

We currently got a params class methods, it becomes complicated for me to choose which one to use if another require_params, optional_params is introduced. So maybe add a new parameter, that's say accessor: true. So you can write like this,
params do
    requires :foo, type: Proc, accessor: true
end

Finally if the type of parameter is Proc then the created method will call the proc.

This is actually quite neat,  :+1:  on this.

You can say class MyComponent < React::Component::Base instead of including React::Component

Would you mind elaborating more on why having a base class rather than current module approach? The initial design decision made is to allow every Ruby class got render defined could be valid component. React::Component only provide helper methods. And there is a lot of refactoring work for React::Component on #17 , so I will suggest postponing this until it's done.

And any class that responds to 'render' can be used in the rendering dsl.

:+1:  on this.
Last but not least, would you mind open separate pull requests for these features? Thus we can review and merge them faster. :smiley:

Thanks for the feedback... will discuss more on the pull request.
As I am using this in a real app, I am making adjustments to the API as I go

Okay.. I setup a WIP pull request... your comments on these items would be great.

@catmando This is implemented as part of the state observer (! bang method) API.

This issue was moved to reactrb/reactrb#24


Extensibility of TreeBuilders

These changes allow subclassing the TreeBuilder (focus is especially on the XmlTreeBuilder, not so much on the HtmlTreeBuilder, where I didn't change any of the package-private methods). This includes changing methods of the Token class from package-private to public.
Also, I made outerHtmlHead and outerHtmlTail from Node public, to allow building custom toString representations of a Document from custom code.
See related tickets here:

https://github.com/jhy/jsoup/issues/600
https://github.com/jhy/jsoup/issues/943

My specific use case: I wanted to process HTML documents for pretty-printing them, but don't want jsoup to modify the existing page structure. Therefore I went with XmlTreeBuilder instead of HtmlTreeBuilder. The problem: Multiline <script> and <style> tags would be transformed to a single line with XmlTreeBuilder, that's why I needed to create my own customized subclass which would handle those tags the same way as HtmlTreeBuilder.

Not using this lib any longer. Closing.


[WIP] Add the scaleup/scaledown option to swift-ring

This PR depends on following Swifttool PR https://github.com/blueboxgroup/swifttool/pull/9

This looks good to me now. The subject still says WIP, but are you ready for this to be merged?

@j2sol I would like this to be merged after swifttool changes are merged.. Even though defaults are setup so that normal run should be fine...

PR depends on https://github.com/blueboxgroup/swifttool/pull/11 .. 9 was closed in favor of this one

@j2sol this can be merged now.. Swiftool pr is merged

retest


Installing SAP HANA, express edition with Docker - container wont start again (2)

more often than not the container can not be started again. It fails a few seconds after a docker start DKHXE202
the startup log:
...

StartService
OK
OK
Starting up HANA ...

18.03.2018 13:16:58
Start
OK

18.03.2018 13:17:10
StartWait
FAIL: process hdbdaemon HDB Daemon not running

docker ps -a shows for the container:
NAMES               IMAGE                                              CREATED ago          STATUS}
DKHXE202            store/saplabs/hanaexpress:<IP_ADDRESS>.20171211.1   8 days ago ago       Exited (2) 11 hours ago}

in the trace dir there is a file
it contains 2 lines with e:
...
[1139]{-1}[-1/-1] 2018-03-18 13:43:37.859636 e Daemon           NetworkListener.cpp(00796) : Failed to resolve empty hostname
...
[1164]{-1}[-1/-1] 2018-03-18 13:43:38.006257 e Daemon           MainImpl.cpp(00340) : Lock file<EMAIL_ADDRESS>with pid 1139 already exists. Stop the daemon process
...

The first seems to be unimportant, it is there even after a successful run / start.
But the second seems to be the pointer to the problem:
and indeed after the exit there still is a .pid file in the locks dir:
<EMAIL_ADDRESS>containing<PHONE_NUMBER>
I wonder what is the right procedure to stop and start a HXE container?

A simple docker stop DKHXE202 had sometimes led to such fails when trying to start it again.
Opening a shell within the container and a command HDB stop and thereafter a docker stop gave a start again fail too.
Shutdown and reboot and thereafter a docker start ... - same.

@JCurrie27 @aplata-sap Can you take a look here?

may be this problem is obsolete now with the new SPS 03?
on docker store we have now:
docker pull store/saplabs/hanaexpress:<IP_ADDRESS>.20180403.2

my problems happened with the SPS 02.
If I find some spare time the next days I will give the new docker container image a try.

Shutdown and reboot and thereafter a docker start ... - same.

means: shutdown and reboot of the docker host, while HXE container is running.

Results so far:
At first SPS 03 (with XSA) looked good. Clean stop can be at least achieved by issuing:
docker stop -t 320 container. According to the help this extends the time before the docker stop sends a kill signal (=hard stop) to the HXE container from the default 10 secs to 310. Within that time frame a safe HDB shutdown seems to work.
But I was able to get back to a not any more startable container with a similar  pid exist failure I had posted before:
from the trc:
Lock file<EMAIL_ADDRESS>with pid 4113 already exists. Stop the daemon process
this is the last entry there.
Seems to be caused by a deep suspend (normal suspend but for more than a few hours).
At wake up the container was unresponsive. Tried xs apps within. Result minutes of 100% CPU load on all cores, nothing else.
Had to stop it by a docker stop -t 300. Even within that grace time the container did not do a clean shutdown.
docker ps -a shows a 137 exit code for it.
Restart not possible, see above.

sorry closed by error, reopen

sorry closed by error, reopen

Finally I can state that there are still similar problems with not (re)startable HXE containers where the .pid file as leftover from a hard shutdown is the reason for that behavior.
The problem can be reproduced easily:
either after a longer suspend to RAM or after a hibernate to disk of the host, HXE will run wild (100% CPU on all cores) at wakeup.
This unresponsive hdb has to be stopped (killed) hard.
Restart will fail:  (from the stdout2):
An instance of this service is already running.
You should consider checking if the process with the id 413 is running
and of type hdbdaemon. If you are sure that there is no instance alive
remove the file<EMAIL_ADDRESS>and restart.

.pid file can be removed when the container had been setup with the additional parameter: --dont-exit-on-error by a simple rm<EMAIL_ADDRESS>within the container bash.
The next start of the same container will (normally) succeed then.
My suggestions for the tutorials:

Describe how to include the optional parameter --dont-exit-on-error as way to troubleshoot more easily.
in a troubleshooting section mention the removal of the .pid file as a (may be) remedy.

Thanks.

Question:
should I open a new issue for the running wild HXE here or is there an other way to semi-officially report that problem?

the grace period can be extended for an instance when the instance is first created by the docker run command.
Additional parameter should be:
--stop-timeout 330
see: https://docs.docker.com/engine/reference/commandline/run/#stop-container-with-timeout---stop-timeout
With that setup the shutdown sequence of the HDB should have finished within the grace period. No more SIGKILL from the docker stop to the HXE container.
if you agree please include this option into the tutorial

Hello gsgxnet: the development team is evaluating these stop/start scenarios and we'll respond as soon as we can. In the meantime, please continue monitoring the Community (https://www.sap.com/developer/topics/sap-hana-express.community.html#community) in case other users have found workarounds to this issue.

Hi JCurrie27,
I had developed my own workarounds soon after I had posted the scenarios. A bit complicated and a bit difficult to describe and implement - in summary I have now systemd stop jobs to safely stop hdb in case of system shutdown or system sleep.
The shutdown systemd would not be needed if the docker stop would be set up for the extended grace period, see above. If you think my a bit clumsy  workaround could be useful for others I can post it here.
At the moment I do not see any other option to handle the sleep/wakeup issue other than stopping the HXE before sleep. Could post my workaround too. OTOH I doubt HXE and docker are designed for laptop use.
I do not like to fill issue conversions with one time solutions which are useless for others besides myself. I had hit too many such workarounds which only were a major waste of time in other spaces in the past.

Customer was able to work around stop/start problem. Development is evaluating docker stop grace period enhancements. Closing this GitHub issue.


Terms Aggregation: Support for sorting on multiple criteria

https://github.com/elastic/elasticsearch/pull/7662

Started on this here: https://github.com/elastic/elasticsearch-net/commit/9e4551de121b085e69fad0acb0fea11b20be40c3 but as it stands now we need to write our own serializer for the terms aggregation in order to choose either the old or new ordering property..
letting this bake in the hopes we can come up with something cleverer during as the day progresses :)

This is fixed in 2.0.  Don't think we should do anything for the 1.x branch.


#2428 Admin refresh - Improvements to Exercises Live page

What's included?

Remove the "This is a new service..." line
Capitalise initial 'e' in page title, i.e. Live Exercises
Amend column heading to single word titles, i.e. Reference, Name, Open, Close, Status, Applications
Make the default sort by Reference Descending, i.e. lowest number first
Have the entire row be the link, i.e. no underlined hyperlink, click anywhere in the row to go to the relevant exercise, grey highlight when rolled over
Sign out link at top right becomes Display name, i.e. Addy, Nick
Sign out link has a Downward arrow symbol to the left (to signify submenu)
When Addy, Nick is clicked, a sub menu appears containing the options Sign out, My favourites - could be on a grey bar that stretches across the entire screen (this could either 'float' on top of text or push text down the screen when open
Exercises link in top navigation has a Downward arrow symbol to the left (to signify submenu)
When Exercises is clicked, a sub menu appears containing the items: Live Exercises, Create exercise, View archived exercises (in that order)

Closes #2428
Who should test?
✅ Product owner
✅ Developers
✅ UTG
How to test?

Go to the following page (and login if necessary): https://jac-admin-develop--pr2465-feature-2428-refresh-hxenxrk4.web.app/
Ensure the "This is a new service..." line isnt displayed
Ensure the title says 'Live Exercises'
Ensure column headings are: Reference, Name, Open, Close, Status, Applications
Ensure the data initially sorts by Reference Descending, i.e. lowest number first
Ensure the entire row is a link, i.e. no underlined hyperlink, click anywhere in the row to go to the relevant exercise
Ensure when you hover over a row it highlights the entire row in grey
Ensure the Sign out link at top right becomes Display name, i.e. Addy, Nick
Ensure the Sign out link has a Downward arrow symbol to the left (to signify submenu)
When the Display Name is clicked, a sub menu appears containing the options Sign out, My favourites
Ensure the Exercises link in top navigation has a Downward arrow symbol to the left (to signify submenu)
When Exercises is clicked, a sub menu appears containing the items: Live Exercises, Create exercise, View archived exercises (in that order)

Risk - how likely is this to impact other areas?
🟢 No risk - this is a self-contained piece of work
Related permissions
Have permissions been considered for this functionality?

No permission changes required

PREVIEW:DEVELOP
can be OFF, DEVELOP or STAGING

Ok those changes are done @nickaddy

Looking good, @drieJAC 👍

@drieJAC Can you pls incorporate the following feedback from UT?

Change Exercises submenu item View Archived exercises to Archived exercises
Move My favourites to Exercises submenu (last item in the menu)
Remove View live exercises button that appears when you click My favourites – this can now be accessed via the Exercises submenu

Those changes have been completed @nickaddy


Migrate the Internal Representation of Queries to a AST

Is your feature request related to a problem or challenge?
Currently, the relay network propagates queries using raw SQL strings + metadata as the internal representation. sqlparser-rs is used to validate and enforce constraints on the SQL strings, but for actually mapping queries, string substitution on raw SQL strings is the current method.
Describe the solution you'd like
Use DataFusion to parse the SQL strings down to a logical query plan. Replace instances of string substitutions with operations on this logical plan.
This logical plan can be converted to Substrait and executed directly on FlightSQL engines which support Substrait.
For all other execution engines (namely the custom Trino QueryRunner), the logical plan will need to be mapped back to raw SQL string, which could entail significant effort.
Describe alternatives you've considered
Leave the current Query template based internal representation as-is.
Additional context
No response

We should be able to accept plain SQL (no template + metadata), parse to sqlparser-rs AST, manipulate the AST directly, and format back to SQL string for execution.
The AST can always be converted to a logical plan by datafusion if desired.

FYI I filed to track https://github.com/apache/arrow-datafusion/issues/9495 to support going from Expr --> back to SQL


USER network mode can't register to consul

I'm using USER mode network with calico and try to run mesos-consul but
I see the logs:

INFO[0218] Syncing services finished. Stats, registerd: 1 (failed: 0), deregister: 0 (failed: 0).
but can't find the service on consul

HOST network mode is ok

Can you show more logs? It should contain information what is registered. There is a chance healtchceck is not passing so service is not available.

hi, here is log when i start an application
DEBU[0126] Asking Marathon for apps                      Location="marathon:8080"
DEBU[0126] Sending GET request to marathon               Location="marathon:8080" Protocol=http Uri="/v2/apps?embed=apps.tasks&label=consul"
DEBU[0126] Task is not healthy. Not Registering          ExpectedRegistrations=1 HasRegistrations=0 Id=redis.72c8ea4a-1bcf-11e8-a167-024222acc227 Sync=true
INFO[0126] Syncing services finished. Stats, registerd: 0 (failed: 0), deregister: 0 (failed: 0).
DEBU[0126] Leader detection disable
INFO[0126] Got HealthStatusEvent                         Id=redis.72c8ea4a-1bcf-11e8-a167-024222acc227
DEBU[0126] Asking Marathon for /redis                    Location="marathon:8080"
DEBU[0126] Sending GET request to marathon               Location="marathon:8080" Protocol=http Uri="/v2/apps//redis?embed=apps.tasks"
DEBU[0127] Forcing sync
INFO[0127] Syncing services started
DEBU[0127] Asking Marathon for apps                      Location="marathon:8080"
DEBU[0127] Sending GET request to marathon               Location="marathon:8080" Protocol=http Uri="/v2/apps?embed=apps.tasks&label=consul"
INFO[0127] Syncing services finished. Stats, registerd: 1 (failed: 0), deregister: 0 (failed: 0).
DEBU[0128] Forcing sync
INFO[0128] Syncing services started
DEBU[0128] Asking Marathon for apps                      Location="marathon:8080"
DEBU[0128] Sending GET request to marathon               Location="marathon:8080" Protocol=http Uri="/v2/apps?embed=apps.tasks&label=consul"
INFO[0128] Syncing services finished. Stats, registerd: 1 (failed: 0), deregister: 0 (failed: 0).
DEBU[0129] Forcing sync

the app pass health check but i can't see it on consul
There is some info about app:
    "Env": [
                "HOST=marathon",
                "MARATHON_APP_DOCKER_IMAGE=redis",
                "MARATHON_APP_ID=/redis",
                "MARATHON_APP_LABELS=CONSUL",
                "MARATHON_APP_LABEL_CONSUL=",
                "MARATHON_APP_RESOURCE_CPUS=1.0",
                "MARATHON_APP_RESOURCE_DISK=0.0",
                "MARATHON_APP_RESOURCE_GPUS=0",
                "MARATHON_APP_RESOURCE_MEM=100.0",
                "MARATHON_APP_VERSION=2018-02-27T15:04:03.012Z",
                "MESOS_CONTAINER_NAME=mesos-a89b8766-a1e5-47fd-9444-f9482545df1c",
                "MESOS_SANDBOX=/mnt/mesos/sandbox",
                "MESOS_TASK_ID=redis.72c8ea4a-1bcf-11e8-a167-024222acc227",
                "PORT0=6379",
                "PORTS=",
                "PORT_6379=6379",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.10",
                "REDIS_VERSION=4.0.8",
                "REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-4.0.8.tar.gz",
                "REDIS_DOWNLOAD_SHA=ff0c38b8c156319249fec61e5018cf5b5fe63a65b61690bec798f4c998c232ad"
            ],
            "Cmd": [
                "redis-server"
            ],
            "ArgsEscaped": true,
            "Image": "redis",
            "Volumes": {
                "/data": {}
            },
            "WorkingDir": "/data",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "MESOS_TASK_ID": "redis.72c8ea4a-1bcf-11e8-a167-024222acc227"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "33678def4b9cbec6f80b71edf1cf788d21f7c62bd26769d8356630041c21b5b2",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/33678def4b9c",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "net1": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "73fee30f8c88"
                    ],
                    "NetworkID": "62cac9c5ea1a3b0085527b937a7ab00fced17fe35aa0476dde743c973d2be69b",
                    "EndpointID": "975c656aeb81b4210e77464727d15708ac0ac31a13d15d5a106de1556bb084f8",
                    "Gateway": "<IP_ADDRESS>",
                    "IPAddress": "<IP_ADDRESS>",
                    "IPPrefixLen": 32,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "ee:ee:ee:ee:ee:ee"
                }
            }
        }
    }
]

I tested with HOST network mode and see it works as expected.

Could you post Minimal, Complete, and Verifiable example of marathon application JSON so I can test it? What versions are you using (Marathon, Consul, Marathon-Consul)?

marathon: Version 1.6.335
Consul:1.07
Marathon-Consul：1.4.2
Docker: 17.05.0-ce
[root@node1 ~]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
a96e392429fe        bridge                bridge                 local
ff4464cb6052        host                    host                    local
c512d25f75c4        my_net              calico                  global
df9ccac92157        none                  null                     local
marathon application JSON
{
"id": "/nginx-net",
"cmd": null,
"cpus": 0.1,
"mem": 64,
"disk": 0,
"instances": 3,
"constraints": [
[
"hostname",
"UNIQUE"
]
],
"acceptedResourceRoles": [
"*"
],
"container": {
"type": "DOCKER",
"docker": {
"forcePullImage": false,
"image": "nginx",
"parameters": [],
"privileged": false
},
"volumes": [],
"portMappings": [
{
"containerPort": 80,
"labels": {},
"protocol": "tcp",
"servicePort": 10004
}
]
},
"healthChecks": [
{
"gracePeriodSeconds": 300,
"ignoreHttp1xx": false,
"intervalSeconds": 60,
"maxConsecutiveFailures": 3,
"path": "/",
"portIndex": 0,
"protocol": "HTTP",
"ipProtocol": "IPv4",
"timeoutSeconds": 20,
"delaySeconds": 15
}
],
"labels": {
"consul": "nginx-net",
"HAPROXY_GROUP": "proxy",
"HAPROXY_0_VHOST": "node5.test"
},
"networks": [
{
"name": "my_net",
"mode": "container"
}
],
"portDefinitions": []
}
[root@node1 ~]# docker network inspect my_net
[
{
"Name": "my_net",
"Id": "c512d25f75c4e93b579559d1c39d08f85c39956916d900b62b8700633d81a472",
"Created": "2018-04-21T23:13:42.947360347+08:00",
"Scope": "global",
"Driver": "calico",
"EnableIPv6": false,
"IPAM": {
"Driver": "calico-ipam",
"Options": {},
"Config": [
{
"Subnet": "<IP_ADDRESS>/24"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"Containers": {
"3835c3c37b64cf4688fce64d6abb42587335aed72894da56dacf0e13067f9440": {
"Name": "mesos-88ab805c-4a85-44c0-bd2f-c6826be5413e",
"EndpointID": "533ea9f8685bbf75c9cb4aa08fa5603b3733b0474e36607d52076537052bcbdc",
"MacAddress": "ee:ee:ee:ee:ee:ee",
"IPv4Address": "<IP_ADDRESS>/32",
"IPv6Address": ""
},
"ep-f50a38d45064c24d1a21ea469eb99e7fd1df7114274844c85999609f78fce782": {
"Name": "mesos-1ca57deb-3502-483f-aa67-3e95ce41dbcd",
"EndpointID": "f50a38d45064c24d1a21ea469eb99e7fd1df7114274844c85999609f78fce782",
"MacAddress": "ee:ee:ee:ee:ee:ee",
"IPv4Address": "<IP_ADDRESS>/32",
"IPv6Address": ""
},
"ep-f5b2b0adfcdc7cc2b1b376ae67dad9d4509ffc715775862f10008363bd692bd4": {
"Name": "mesos-fb3ee9b9-7b1a-4802-8f96-1dbcf5603a9e",
"EndpointID": "f5b2b0adfcdc7cc2b1b376ae67dad9d4509ffc715775862f10008363bd692bd4",
"MacAddress": "ee:ee:ee:ee:ee:ee",
"IPv4Address": "<IP_ADDRESS>/32",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]


JSDoc code fixes for arrow-function initialized const duplicates comment

In a TypeScript file:
/**
 *
 * @param {string} y
 * @returns
 */
export const foo = (x: string) => x;

On y, request code actions to either rename or delete y.
If you rename, the result is this:
/**
 *
 * @param {number} y
 * @returns
 */
export /**
 * @param {number} x
 * @returns
 */
 const foo = (x: string) => x;

@a-tarasyuk The rename code fix has the same problem.

Originally posted by @zhoudaxia2016 in https://github.com/microsoft/TypeScript/issues/52276#issuecomment-1401646584

Thanks @a-tarasyuk!


Slow CI builds

I think that's an issue with travis ci. After all, the .travis.yml only installs a few packages and runs the tests.

If it is just travis CI, then fair enough. But running pytest locally took around seven minutes for me (internet connection around 30mps)

Ran for 3 min 43 sec. I believe that's the actual testing. Everything else is setup.

ok I guess 3 mins isnt so bad. i just noticed there were quite a few videos being downloaded as part of the test. would be good to know if such a number is strictly necessary / smaller files can be downloaded

Yeah, the videos downloaded are a bit arbritrary (besides having specific streams available)

someone submitted a test that actually downloads videos (doesn't not use mock) I'm going to delete those for now.

had a go at using mock - https://github.com/nficano/pytube/pull/297


Edit Step name to PUBLISH instead of ACTIVATE - Fiori App ABAP

Hello,
Scenario URL: https://developers.sap.com/tutorials/abap-environment-data-model.html
step 4.4: Activate your service binding.
change step and screenshot:
now we have to PUBLISH, not activate.
Please refer to screenshot:

Labels to be added:
Sprint Label: SCPTest-2011A
Landscape label: trail1, trail2
Scenario label: SCPTest-fabap
Prio: low-prio
Thanks,
Shubham

Hi @i353590 ,
I created following pull request: #6347 for this issue.
@MichaelCzcz  we can close this ticket.
Best regards,
Merve


NYT CSS + dependencies

Hi @jeremybmerrill!
The templates are loading NYT stylesheets and templates. Have you considered getting these under source control? And document what they do? Here's what I found:
https://github.com/newsdev/foialawya/blob/master/foias/templates/base.html#L6
https://github.com/newsdev/foialawya/blob/master/foias/templates/base.html#L65
https://github.com/newsdev/foialawya/blob/master/foias/templates/admin/base.html#L9
There's also a mention to the NYT here:
https://github.com/newsdev/foialawya/blob/master/foias/views.py#L25
that I haven't tested yet, but believe will be displayed on "social" login error

Hi @lfbrandao: Thanks for taking a look!
Those stylesheets are all from an NYT library that offers administrative components; a slightly more full-featured version of https://github.com/newsdev/adcom, that I didn't have time to factor out. I don't remember exactly what it does, maybe it can be factored out easily... I'll take a look.
I'll figure out how to get that @nytimes.com mention out of there. I was gonna work a little on this app today or tomorrow, so I can probably do that too.

Wonderful, thank you! Not super urgent, but figured it was worth mentioning

fixed teh second issue here: https://github.com/newsdev/foialawya/commit/8b95edee77995c02b8dacb654697c91fece46f93


feat: enhance the process topology graph to support dragging nodes

Enhance the process topology graph to support dragging nodes.
Video
https://user-images.githubusercontent.com/20871783/190152152-3617aa96-959f-4c13-85a7-83ed353f7400.mov
Signed-off-by: Qiuxia Fan<EMAIL_ADDRESS>

Is the hexagon boundary still working? Meaning, that you can't drag an out-of-pod node inside the hexagon.

Is the hexagon boundary still working? Meaning, that you can't drag an out-of-pod node inside the hexagon.

No, there is no limitation here.

Is it possible to add? Because the boundary is physical, we should not allow that.

Let me try to implement this.

Please recheck.
https://user-images.githubusercontent.com/20871783/190362957-e2553c29-9f1a-4778-a065-370e5aa40a9c.mov

Let's do a sync.


Fix tests 27, 28, and 30

Fix 3 tests that are failing due to a missing tmp directory.
Enhance run-test.perl to track the total number of tests and the number of tests that passed.

LGTM


make status_justify configurable

The default for status-justify is left, would it not be better to just remove the set line for it and let users set it?
You can also just overwrite it.

I would only do that if I were confident that this theme works well with every justify setting. Otherwise an existing tmux config may look broken. Making this setting explicit moves the responsibility to the user. Otherwise, totally agree! 👍


Building with bazel:  name 'json' is not defined

System information

OS Platform and Distribution: PYNQ - Linux Ubuntu 22.04 aarch64
Compiler version: gcc version 11.2.0 (Ubuntu 11.2.0-19ubuntu1)
Programming Language and version: Python 3.10.4
Bazel version: bazel 3.7.2

Hi!
I am trying to build the wheel for aarch64 because it is not provided in the PyPi repo. To do that, I cloned the mediapipe repository and lanched the build step with
python3 setup.py gen_protos && python3 setup.py bdist_wheel

The problem is that, after a few minutes of downloading/extracting stuff, bazel outputs a series of errors:

ERROR: /home/xilinx/.cache/bazel/_bazel_xilinx/dbcb6bd62247f0988afcb64139fd29a7/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:483:28: name 'json' is not defined
ERROR: /home/xilinx/.cache/bazel/_bazel_xilinx/dbcb6bd62247f0988afcb64139fd29a7/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:594:20: name 'json' is not defined
ERROR: /home/xilinx/.cache/bazel/_bazel_xilinx/dbcb6bd62247f0988afcb64139fd29a7/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:625:19: name 'json' is not defined
ERROR: error loading package '': in /home/xilinx/.cache/bazel/_bazel_xilinx/dbcb6bd62247f0988afcb64139fd29a7/external/build_bazel_rules_nodejs/index.bzl: Extension 'internal/npm_install/npm_install.bzl' has errors

Why it happens and how can I solve it? Thank you!

I also encountered this problem when compiling under the desktop environment. Can I ask you how to solve it?

@flySheldon,
Could you please raise a new fresh issue with the template  and complete steps to reproduce the issue from our end? Thank you!

I also encountered this problem when compiling under the desktop environment. Can I ask you how to solve it?

I used bazelisk instead of bazel. In this way, the mediapipe build process can use the bazel version it wishes

I also encountered this problem when compiling under the desktop environment. Can I ask you how to solve it?

I used bazelisk instead of bazel. In this way, the mediapipe build process can use the bazel version it wishes
I solved the problem with your method, thank you

I also encountered this problem when compiling under the desktop environment. Can I ask you how to solve it?

I used bazelisk instead of bazel. In this way, the mediapipe build process can use the bazel version it wishes
I solved the problem with your method, thank you

@thegabriele97 I solved the problem with your method, thank you


Add to the docs some information about opening relevant ports when doing clustering

Would point to the following doc link from mk8s from one of the sections talking about setting up clustering that you need to make sure any of the listed "host interfaces" in this list need to be allowed through the firewalls and such. As the list is quite lengthy and wasn't entirely obvious what all ports are getting used until we dug into the documentation of microk8s, so this would save someone else the same headaches, because the errors thrown by mk8s doesn't make it super clear it is a port issue.
Might also be worth starting a "troubleshooting" section as well with this as one of the things to double check.
https://microk8s.io/docs/ports

Good point, I believe your deployment was on RHEL7, on Ubuntu and Rhel 8 the install was able to open ports on its own I but I don't think this works on rhel 7

Yes, this was RHEL 7.

Added in documentation, so far it is in develop branch but when merge happen to main it will be also in main: https://splunk.github.io/splunk-connect-for-snmp/develop/bestpractices/#installing-splunk-connect-for-snmp-on-linux-redhat


Bump Dotnet.Script.DependencyModel to version 0.50.0

This PR bumps the Dotnet.Script.DependencyModel and Dotnet.Script.DependencyModel.NuGet packages to version 0.50.0 adding support for .Net Core 3.0 based scripts.

. Net Core 3.0 compilation deps needs more work

@mholo65 any thoughts here?
@david-driscoll you OK with bumping Nuget deps to 5.2.0?

@filipw yeah I'm fine with nuget 5.2, we need to stay more up to date on these things anyway.


SQLite support

By default, Nextcloud uses SQLite as its database. It has two benefits:

It's the only DB backend that allows you to switch to a different one later
It's easy to put the SQLite file into an arbitrary directory

The last point is especially relevant if you're hosting in a cloud. With SQLite, you can plug in a "block volume", mount it to /srv, and configure datadirectory => '/srv'. Doing so is a good idea regardless of the DB configuration.
With MariaDB or PostgreSQL, you have two options:

Change their configurations to store data in the mount point
Use managed databases

Both options require additional setup and are more error-prone.
So, I would go as far as suggesting SQLite be the default database backend. It's fit for individuals, and for more advanced use cases, one can always switch to a DB of their choice.

Hello there !
Thank you @bemyak  for the suggestion.
I agree that adding support for sqlite will make the role more complete regarding supporting the same backends than nextcloud.
Regarding the point to put the DB data into another location is probably out of the scope of this role, as a system config subject.
There are lots of ways to implement this; some requires more or less changes and/or system knowledge of the system.
for example, mounting an external block storage volume to the default data directory of the DB server, before installation, would also work without changing configuration
.. or binding the DB data folder to another one located inside a external volume, mounted somewhere else in the system.
etc.
Also as you said, wanting to do anything than default requires additional setup, testing and fixing.
On a devops perspective, this is the way. :smiley:

So, I would go as far as suggesting SQLite be the default database backend. It's fit for individuals, and for more advanced use cases, one can always switch to a DB of their choice.

That's an interesting point. @staticdev @wiktor2200  what do you think ?
On my side, I'm referring to Nextcloud's doc stating that sqlite is a low performance, non prod focused DB backend. It recommends to use mysql/mariadb, that is the role's default and probably the most widely used DB backend for nextcloud. ( it would be nice to have the server survey result to get a better estimation)
Regards.
Aal.

Regarding the point to put the DB data into another location is probably out of the scope of this role, as a system config subject.

I completely agree. This is out of the scope of this role. My point is that SQLite makes it easier to achieve this setup in its own role.
I would leave any database installation out of scope, ship SQLite as default and if a user wants something else, they should install it beforehand the way they want and supply this role with endpoint and credentials. Kind of like we do with SSL certs at the moment.
But I understand the vision of having everything out of the box, just expressing my humble opinion.

That's interesting point.  Nextcloud's usage generates lots of data in DB, so even small instance with few GB of data can grow to hundreds of MBs in DB.
I would leave MySQL as it is right now, because it's the most common solution.
But to have it as complete solution we could try to add SQLite as option to role in the future.

It's pretty easy to get it working already. I had to install php{{ php_ver }}-sqlite3 package and set nextcloud_db_backend: sqlite role variable.

SQLite is extremely slow

Slow in what sense? It shouldn't affect the transfer rate, right? Only UI responsiveness.

It's pretty easy to get it working already. I had to install php{{ php_ver }}-sqlite3 package and set nextcloud_db_backend: sqlite role variable.
It sounds great :)

Slow in what sense? It shouldn't affect the transfer rate, right? Only UI responsiveness.
I meant that usage of Nextcloud is very limited with SQLite and as documentation said it's  good only for testing and lightweight single-user setups without client synchronization, so mostly for testing purposes, because one of the main features is client sync. :) What is more adding any app (calendar, contacts, media etc.) probably will make whole instance too slow to use.

But still I agree that it would be good to add it as option for those who would be interested :)

I have nothing against having the option, the support will be proportional to the usage. @bemyak do you have interest in creating a PR maybe? Looks like you already figured out most of the job.

I don't know how to install an unpublished ansible collection. I can submit a PR, but I'll only test in on 1.8.0.

@bemyak we are in a transition phase. I think 2.0.0 will come soon enough so you can use it as base.


std::vector mutated during iteration

There are several places where items are removed from a std::vector during iteration - this invalidates the iterator, so subsequent operations may be invalid.
One example is History::remove_all_changes() - it creates an iterator and loops through the vector of changes, but calls remove_change() which removes an entry. In my case, it would then segfault when trying to continue removing changes based on the invalid iterator.
My local fix for it is something like:
while (!m_changes.empty()) {
  remove_change(m_changes.front());
}

But there may be a better option that is easier to reuse in other scenarios.
It's probably worth doing an audit of the project for this pattern, and resolving other instances as well.

@liamstask Thanks for the heads up. We've fixed it and checked we've not missed this somewhere else in version 1.2.0 which is already being uploaded.


[BUG] Upgrade: after draining the first node, the upgrade is stuck

Describe the bug

This was observed after https://github.com/harvester/harvester-installer/pull/229.
After pre-hook and drain of the first node are done, the upgrade is stuck because post-hook annotation is not set on plan secret.
The cluster has this condition:
  - lastUpdateTime: "2022-03-08T16:13:27Z"
    message: provisioning bootstrap node(s) custom-9441cf1c7106
    reason: Waiting
    status: Unknown
    type: Provisioned

And from rancher log:
2022/03/08 16:13:16 [INFO] Stopping metadata watch on apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2022/03/08 16:13:16 [INFO] Stopping metadata watch on rbac.authorization.k8s.io/v1, Kind=ClusterRole
2022/03/08 16:13:16 [INFO] Stopping metadata watch on rbac.authorization.k8s.io/v1, Kind=Role
2022/03/08 16:13:16 [INFO] Stopping metadata watch on rbac.authorization.k8s.io/v1, Kind=RoleBinding
2022/03/08 16:13:16 [INFO] Stopping metadata watch on admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2022/03/08 16:13:17 [INFO] [planner] rkecluster fleet-local/local: waiting: provisioning bootstrap node(s) custom-9441cf1c7106: waiting for probes: kube-controller-manager
2022/03/08 16:13:17 [INFO] [planner] rkecluster fleet-local/local: waiting: provisioning bootstrap node(s) custom-9441cf1c7106: waiting for probes: kube-controller-manager
2022/03/08 16:13:27 [INFO] [planner] rkecluster fleet-local/local: waiting: provisioning bootstrap node(s) custom-9441cf1c7106
2022/03/08 16:13:27 [INFO] [planner] rkecluster fleet-local/local: waiting: provisioning bootstrap node(s) custom-9441cf1c7106
2022/03/08 16:15:02 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022/03/08 16:20:02 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022/03/08 16:21:38 [INFO] [planner] rkecluster fleet-local/local: waiting: provisioning bootstrap node(s) custom-9441cf1c7106
2022/03/08 16:21:41 [INFO] [planner] rkecluster fleet-local/local: waiting: provisioning bootstrap node(s) custom-9441cf1c7106
2022/03/08 16:21:47 [INFO] [planner] rkecluster fleet-local/local: waiting: provisioning bootstrap node(s) custom-9441cf1c7106

To Reproduce
Steps to reproduce the behavior:

Create a 1.0.0 cluster
Upgrade with master ISO.

Expected behavior

Support bundle

Environment:

Harvester ISO version:
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630):

Additional context
Add any other context about the problem here.

Rancher team is aware of this and is fixing it.

According to Donnie, this will be addressed in https://github.com/rancher/rancher/issues/36782 as well.

After upgrading Rancher from v2.6.3-harvester1 to v2.6-5bb08b3692ac3c18ec1361694a095e2c25eb0346-head, I can confirm. I can trigger the RKE2 upgrade cycle with drain hooks enabled.

The Rancher version is bumped and the issue is supposed to be fixed.
It can be tested once the upgrade feature is ready.

Verified fixed on one click upgrade to v1.0.1-rc2. Close this issue.
Result

Can upgrade the 3 nodes harvester to v1.0.1-rc2 successfully without stuck

Can proceed to upgrade the second node after draining the first node

Test Information

Test Environment: 3 nodes harvester on provo bare machine
Harvester version: v1.0.0 GA to v1.0.1-rc2

Verify Steps

Prepare v1.0.0 three nodes harvester cluster on Provo machines
Create a ubuntu and SLES JeOS image
Create a ubuntu and SLES virtual machine
Enable network to harvester-mgmt
Create a virtual network vlan1 with id 1
Setup back target to AWS S3
Create two backups for the ubuntu and SLES VM
Shutdown all VMs
Follow the steps to upgrade to v1.0.1-rc2
https://github.com/harvester/docs/blob/42598035cfe79856f7e159d4c47c4d7d29178900/docs/upgrade/automatic.md
Check the one click upgrade completely finished
SSH to management node
Run sudo -i kubectl get managedcharts harvester -n fleet-local -o yaml | yq e '.spec.values' -
Check kube-vip.config not exists
Check longhorn.defaultDataPath: /var/lib/harvester/defaultdisk exists

Additional Context

Support Bundle
supportbundle_138129b2-b170-4944-bcae-bc40ff88707e_2022-04-07T05-48-23Z.zip


fix: upgrade to version 3.2.64

from: https://github.com/jenkins-x/jx

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign hferentschik
You can assign the PR to them by writing /assign @hferentschik in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jenkins-x-bot-test: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gsm-kapp
dff027ada134abea1703ee2a2038556c2a7e5113
link
/test gsm-kapp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

/retest

/test gsm-kapp

@jenkins-x-bot-test: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gsm-kapp
dff027ada134abea1703ee2a2038556c2a7e5113
link
/test gsm-kapp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.


Change optimization level for presubmit to Os

This will ensure that warnings produced by code that can be further
optimized will show up during presubmit. This will keep the code from
being submitted if it can be optimized for size further.
Resolves #418

Coverage remained the same at 72.52% when pulling 0c1041677df2faddb7f23a6c877fbb5f174ca51b on optimize-size-ci into c198eb8229ae1a637a69394b33ec4092860c8a3c on v2.0-beta.


Communication Contact annotation not working on object page header

The communication contact annotation shown below does not work in the object page header:
   FieldGroup #HeaderGeneralInformation : {
        $Type : 'UI.FieldGroupType',
        Data : [
            {
                Value : priority_code
            },
            {
                Value : incidentStatus_code
            },
            {
                Value : category_code
            },
            {
                $Type  : 'UI.DataFieldForAnnotation',
                Target<EMAIL_ADDRESS>                Label  : '{i18n>AssignedContact}'
            }
        ]
    },

but if I add the same piece to a facet on the main page it works fine (still shows yellow squigglies in the editor though):
    FieldGroup #GeneralInformation : {
        $Type : 'UI.FieldGroupType',
        Data : [
            {
                $Type : 'UI.DataField',
                Value : priority_code,
            },
            {
                $Type : 'UI.DataField',
                Value : category_code,
            },
            {
                $Type : 'UI.DataField',
                Value : incidentStatus_code,
            },
            {
                $Type  : 'UI.DataFieldForAnnotation',
                Target<EMAIL_ADDRESS>                Label  : '{i18n>AssignedContact}'
            }
        ],
    },

Any ideas?

Hi Jason,
thanks for pointing this out.
Actually I have no explanation why the contact is not shown anymore on the OP header. My assumption is, that it´s a bug in the current UI5 environment available in the cloud platform. My local environment using the latest UI5 libraries do not show the issue. Also, this must have worked before since the screenshots taken for the tutorial do also show the contact correctly. I have uploaded a fix showing the contact within a separate ReferenceFacet of the HeaderFacet. Of course this is not the real solution for the issue. I would just wait for some time to check whether this issue will be gone within the next days due to an upload of a new UI5 version into the cloud environment.
Best regards
Andreas


Unit test cases are failing because of exceljs

Unit test cases are failing because of exceljs and error is :
Error: Zone.js has detected that ZoneAwarePromise (window|global).Promise has been overwritten.
Most likely cause is that a Promise polyfill has been loaded after Zone.js (Polyfilling Promise api is not necessary when zone.js is loaded. If you must load one, do so before loading zone.js.)
at Function.Zone.assertZonePatched (webpack:///./node_modules/zone.js/dist/zone.js?:57:23)
at new NgZone (webpack:///./node_modules/@angular/core/fesm5/core.js?:3993:14)
at TestBed._initIfNeeded (webpack:///./node_modules/@angular/core/fesm5/testing.js?:1008:22)
at TestBed.execute (webpack:///./node_modules/@angular/core/fesm5/testing.js?:1093:14)
at UserContext.eval (webpack:///./node_modules/@angular/core/fesm5/testing.js?:1246:45)
at ZoneDelegate.invoke (webpack:///./node_modules/zone.js/dist/zone.js?:390:26)
at ProxyZoneSpec.onInvoke (webpack:///./node_modules/zone.js/dist/zone-testing.js?:288:39)
at ZoneDelegate.invoke (webpack:///./node_modules/zone.js/dist/zone.js?:389:52)
at Zone.run (webpack:///./node_modules/zone.js/dist/zone.js?:149:43)
at runInTestZone (webpack:///./node_modules/zone.js/dist/zone-testing.js?:509:34)
HeadlessChrome 83.0.4103 (Windows 10.0.0): Executed 1 of 1602 (1 FAILED) (0 secs / 0.063 secs)
HeadlessChrome 83.0.4103 (Windows 10.0.0) LoaderService should be created FAILED

Similar to #1008.
This may help

Similar to #1008.
This may help

Thanks for reply but the thread you mentioned, i have already gone through but no luck. Issue still persist.
Also i have tried multiple ways as mentioned in different blogs but no luck till now.

This issue is because of ExcelJS only, because i tried to run test cases in one of old branch before implementing Excel JS , on that branch, unit test cases are running successfully without any issue.

Error: node_modules/exceljs/index.d.ts:1466:22 - error TS2307: Cannot find module 'stream' or its corresponding type declarations.
1466  read(stream: import('stream').Stream): Promise;
~~~~~~~~
Error: node_modules/exceljs/index.d.ts:1487:23 - error TS2307: Cannot find module 'stream' or its corresponding type declarations.
1487  write(stream: import('stream').Stream, options?: Partial): Promise;
~~~~~~~~
Error: node_modules/exceljs/index.d.ts:1574:22 - error TS2307: Cannot find module 'stream' or its corresponding type declarations.
1574  read(stream: import('stream').Stream, options?: Partial): Promise;
~~~~~~~~
Error: node_modules/exceljs/index.d.ts:1579:63 - error TS7016: Could not find a declaration file for module 'events'. 'C:/Users/HM5176/Documents/GitLab/FrontPDCA-1/node_modules/events/events.js' implicitly has an 'any' type.
Try npm i --save-dev @types/events if it exists or add a new declaration (.d.ts) file containing declare module 'events';
1579  createInputStream(options?: Partial): import('events').EventEmitter;
~~~~~~~~
Error: node_modules/exceljs/index.d.ts:1594:23 - error TS2307: Cannot find module 'stream' or its corresponding type declarations.
1594  write(stream: import('stream').Stream, options?: Partial): Promise;
~~~~~~~~
Error: node_modules/exceljs/index.d.ts:1898:19 - error TS2307: Cannot find module 'stream' or its corresponding type declarations.
1898    stream: import('stream').Stream;
~~~~~~~~
Error: node_modules/exceljs/index.d.ts:1942:25 - error TS2503: Cannot find namespace 'NodeJS'.
1942    dictionary: Buffer | NodeJS.TypedArray | DataView | ArrayBuffer; // deflate/inflate only, empty dictionary by default
~~~~~~
Error: node_modules/exceljs/index.d.ts:1994:39 - error TS2307: Cannot find module 'stream' or its corresponding type declarations.
1994    constructor(input: string | import('stream').Stream, options: Partial);
~~~~~~~~
Error: node_modules/exceljs/index.d.ts:2003:18 - error TS2307: Cannot find module 'stream' or its corresponding type declarations.
2003    entry: import('stream').Stream;
same here
any help?


Animesh's profile pic missing in member's page

@toastypixels  github username changed ?

@coderpradp can you look into this too. :)

I mistakenly closed the issue. 😥


Allow the "RTE" folder name to be configurable

Adds optional rtedir attribute to /cprj/target/output element
e.g.
<output ... rtedir="CustomRTEFolder" />

Modified RteTestM3 project to use rtedir="CONFIG_FOLDER".
Renamed RTE to CONFIG_FOLDER in the test structure to reflect this
Builds on proposal from https://github.com/Open-CMSIS-Pack/Open-CMSIS-Pack/issues/30#issuecomment-935589272
Contributed by STMicroelectronics

The most important problem is that test is modified. Please leave the existing test as is (uses default RTE directory), but add a new one with custom RTE directory

Thanks for the review @edriouk I'll look into addressing the new test case, and avoiding the extra parameter for RteTarget hopefully today

@slhultgren I would like to put this enhancement/feature on to the agenda for our next project meeting on Dec. 14th 2021. The reason I do not take this change lightly is the fact that new projects with a custom "rte" folder are no longer compatible with today's projects and we need to understand how a migration path should/would look like in case we agree we need one. In case there is further extensions that you have in the works or planned already it maybe good to learn about that, too.
Does that work for you?

@jkrech  should we follow-up on this item ?
I think it is a good example of how "external contributors" can collaborate on the project.

Maybe it should be reflected in #77?