Make Rancher ARM compatible

Main components

[ ] cattle [rancher/cattle#1739]
[ ] strongswan-package [rancher/strongswan-package#1]
[ ] rancher-net [rancher/rancher-net#13]
[ ] swarm-agent
[ ] cadvisor-package [rancher/cadvisor-package#12]
[ ] rancher-metadata [rancher/rancher-metadata#18]
[ ] websocket-proxy [rancher/websocket-proxy#38]
[ ] rancher-catalog-service [rancher/rancher-catalog-service#48]
[ ] rancher-dns [rancher/rancher-dns#32]
[ ] agent-instance [rancher/agent-instance#18]
[x] rancher-compose
[x] python-agent (no changes needed, so no PR)
[ ] rancher [#5217]
[ ] go-machine-service [rancher/go-machine-service#103]
[ ] host-api [rancher/host-api#77]
[ ] go-rancher-metadata
[ ] machine-package
[ ] docker/machine (no PR. https://github.com/imikushin/machine/releases/tag/v0.7.0_arm)
[ ] vmware/govmomi (no PR. https://github.com/imikushin/govmomi/releases/tag/v0.2.0_arm)
[ ] rancher-docker-api-proxy
[ ] runc (share folder)
[ ] agent-binaries (This is kind of needed: not if ubuntu:16.04 is the base image for agent and agent-instance)

Additional Features Added

[ ] convoy
[ ] convoy-agent
[ ] kubernetes
[ ] kubectld
[ ] kubernetes-agent
[ ] kubernetes-model
[ ] rancher-ingress
[ ] swarm
[ ] vm
[ ] giddyup (no ARM-specific PR. https://github.com/imikushin/giddyup/releases/tag/v0.10.1_arm)
[ ] cluster-manager (no PR. https://github.com/imikushin/cluster-manager/releases/tag/v0.1.4_arm)

That's one awe-inspiring checklist :).

Can't wait for this 😄

Just assembled a 4 unit ODROID C2 cluster and can't wait to get Rancher going on it.

@teleki you can! Just build rancher/server and rancher/agent for ARM from #4704, rancher/agent-instance from rancher/agent-instance#18

@imikushin Thank you for the advice. I tried building rancher/server, but I am getting errors. Here is the output of running as docker ./build-image.arm.sh in the rancher/server folder:
++ dirname ./build-image.arm.sh
+ cd .
+ . ../.docker-env.arm
./build-image.arm.sh: line 6: ../.docker-env.arm: No such file or directory
+ echo 'Need to source .docker-env.arm to access Docker on ARM'
Need to source .docker-env.arm to access Docker on ARM
+ '[' '!' -e target/arm/.done ']'
+ mkdir -p target/arm
++ cat /dev/urandom
++ env LC_CTYPE=C tr -cd a-f0-9
++ head -c 16
+ S6_BUILDER=0484087bb24a011b
+ trap 'docker rm -fv 0484087bb24a011b 2>/dev/null || :' EXIT
+ docker run --name=0484087bb24a011b rancher/s6-builder:v<IP_ADDRESS>_arm /opt/build.sh
Unable to find image 'rancher/s6-builder:v<IP_ADDRESS>_arm' locally
Pulling repository docker.io/rancher/s6-builder
docker: Tag v<IP_ADDRESS>_arm not found in repository docker.io/rancher/s6-builder.
See 'docker run --help'.
+ docker rm -fv 0484087bb24a011b
+ :

Here is the output of git status:
On branch pr/4704
Your branch is up-to-date with 'origin/pr/4704'.
nothing to commit, working directory clean

docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.6.1
 Git commit:   20f81dd
 Built:        Wed, 20 Apr 2016 14:19:16 -0700
 OS/Arch:      linux/arm64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.6.1
 Git commit:   20f81dd
 Built:        Wed, 20 Apr 2016 14:19:16 -0700
 OS/Arch:      linux/arm64

Here is the output of docker images:
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
<none>                    <none>              a4ed6a02234b        About an hour ago   122 MB
ubuntu                    16.04               2fa927b5cdd3        27 hours ago        122 MB
armv7/armhf-ubuntu_core   latest              2c357551076b        7 months ago        109.5 MB

The Linux version message is:
Welcome to Ubuntu 16.04 LTS (GNU/Linux 3.14.29-56 aarch64)

This is my first attempt to venture into this type of stuff, so I am rather green. I wonder if I am missing some additional configuration settings... Your help is appreciated.

@imikushin: update: Successfully built rancher/agent and rancher/agent-instance from the respective PRs. However, rancher/server still has the same error as mentioned above.

@imikushin Succrddfully built rancher/agent and rancher/server.
@teleki You require to:
`

git clone https://github.com/imikushin/s6-builder
./build.arm.sh
`
Then you should have the required s6-builder docker image :) and everything should build just fine.
I can run the rancher/server on my Raspberry Pi 3, but it takes hours to start. One cpu is up to 100% all the time (java process), while starting.

@imikushin  Could it be, that the process still uses OpenJDK instead of OracleJDK?

@imikushin Thank you for the help; it is much appreciated. I have successfully built s6-builder.
However, now I am getting a different error as I am trying to build rancher/server. The error says that there is no "/usr/bin/java" which is probably true, since based on the logs, it seems that Java is in "/usr/lib/jvm". Here are the relevant snippets of the output:
....
Connecting to download.oracle.com (download.oracle.com)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 78321279 (75M) [application/x-gzip]
Saving to: 'jdk-8u91-linux-arm64-vfp-hflt.tar.gz'
...
2016-06-01 02:48:01 (1.51 MB/s) - 'jdk-8u91-linux-arm64-vfp-hflt.tar.gz' saved [78321279/78321279]

Download done.
Removing outdated cached downloads...
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/java to provide /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/jjs to provide /usr/bin/jjs (jjs) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/pack200 to provide /usr/bin/pack200 (pack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/policytool to provide /usr/bin/policytool (policytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/unpack200 to provide /usr/bin/unpack200 (unpack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/appletviewer to provide /usr/bin/appletviewer (appletviewer) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jar to provide /usr/bin/jar (jar) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jarsigner to provide /usr/bin/jarsigner (jarsigner) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javac to provide /usr/bin/javac (javac) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javadoc to provide /usr/bin/javadoc (javadoc) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javah to provide /usr/bin/javah (javah) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javap to provide /usr/bin/javap (javap) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jcmd to provide /usr/bin/jcmd (jcmd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jdb to provide /usr/bin/jdb (jdb) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jdeps to provide /usr/bin/jdeps (jdeps) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jinfo to provide /usr/bin/jinfo (jinfo) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jmap to provide /usr/bin/jmap (jmap) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jps to provide /usr/bin/jps (jps) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jrunscript to provide /usr/bin/jrunscript (jrunscript) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jstack to provide /usr/bin/jstack (jstack) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jstat to provide /usr/bin/jstat (jstat) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jstatd to provide /usr/bin/jstatd (jstatd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/rmic to provide /usr/bin/rmic (rmic) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/serialver to provide /usr/bin/serialver (serialver) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode
Oracle JDK 8 installed

...
 ---> 7dd5b4837334
Removing intermediate container d1461dd50c46
Step 33 : RUN cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract
 ---> Running in 437df28b035c
/usr/share/cattle/cattle.sh: line 197: /usr/bin/java: No such file or directory
The command '/bin/sh -c cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract' returned a non-zero code: 1
+ docker rm -fv bd1ba640d63ce3d8
bd1ba640d63ce3d8

Apparently somehow java is not in the container. However, if I look at the log is seems to show that Oracle Java was installed.
Any ideas where I'm going wrong?

@teleki Could you show us your current used server Dockerfile.arm and are you using the build-image.arm.sh for building?

@alphawolf1988 The Dockerfile.arm is the one I got from PR 4704 referenced a few messages above by @imikushin. I only have added one file, ".docker-env.arm", because it was not in the original code, so I got that from here: https://github.com/rancher/os-images/blob/master/.docker-env.arm.sample but changed the IP to my machine's IP.
Here is the my rancher/server/Dockerfile.arm:
FROM armhf/ubuntu:16.04

ENV TERM xterm
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends openjdk-8-jre-headless \
                    curl \
                    mysql-server \
                    xz-utils \
                    keychain \
                    unzip \
                    openssh-client \
                    iptables \
                    iproute2 \
                    git \
                    redis-server \
                    zookeeper \
                    spiped
RUN mkdir -p /var/run/mysqld && chmod -R 777 /var/run/mysqld

RUN apt-get install -y --no-install-recommends software-properties-common
RUN add-apt-repository -y ppa:webupd8team/java && apt-get update
RUN echo 'yes' | DEBIAN_FRONTEND=Teletype apt-get install -y --no-install-recommends oracle-java8-installer

ADD https://github.com/imikushin/giddyup/releases/download/v0.10.1_arm/giddyup /usr/bin/
ADD https://github.com/imikushin/cluster-manager/releases/download/v0.1.4_arm/cluster-manager /usr/bin/
RUN chmod +x /usr/bin/giddyup /usr/bin/cluster-manager
COPY bin/* /usr/bin/

ENV CATTLE_HOME /var/lib/cattle
ENV DEFAULT_CATTLE_API_UI_INDEX //releases.rancher.com/ui/1.0.5
ENV CATTLE_DB_CATTLE_DATABASE mysql
ENV CATTLE_USE_LOCAL_ARTIFACTS true
ENV no_proxy localhost,<IP_ADDRESS>,localaddress,.localdomain.com
ADD artifacts /usr/share/cattle

ADD service /service
ENV S6_SERVICE_DIR /service

COPY target/arm/*static.tar.gz /s6-statics/

EXPOSE 8080
ENV CATTLE_HOST_API_PROXY_MODE embedded
ENV CATTLE_RANCHER_SERVER_IMAGE v1.0.1_arm
ENV CATTLE_RANCHER_COMPOSE_VERSION v0.7.4
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_LINUX_URL   https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-linux-amd64-${CATTLE_RANCHER_COMPOSE_VERSION}.tar.gz
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_DARWIN_URL  https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-darwin-amd64-${CATTLE_RANCHER_COMPOSE_VERSION}.tar.gz
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_WINDOWS_URL https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-windows-386-${CATTLE_RANCHER_COMPOSE_VERSION}.zip
ENV DEFAULT_CATTLE_CATALOG_URL="library=https://github.com/rancher/rancher-catalog.git,community=https://github.com/rancher/community-catalog.git"

EXPOSE 3306
ENV CATTLE_CATTLE_VERSION v0.159.7-arm-4
ADD https://github.com/imikushin/cattle/releases/download/${CATTLE_CATTLE_VERSION}/cattle.jar /usr/share/cattle/

RUN cd /usr && for i in $(ls /s6-statics/*static.tar.gz);do tar -zxvf $i;done && rm -rf /s6-statics/*static.tar.gz
RUN cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract
RUN curl -sLf https:${DEFAULT_CATTLE_API_UI_INDEX}.tar.gz | tar xvzf - -C /usr/share/cattle/war --strip-components=1
RUN /usr/share/cattle/install_cattle_binaries && \
    cd $CATTLE_HOME && export IFS="," && \
    for i in $DEFAULT_CATTLE_CATALOG_URL; do rancher-catalog-service -validate -catalogUrl=$i;done

VOLUME /var/lib/mysql /var/log/mysql /var/lib/cattle

ENV DEFAULT_CATTLE_MACHINE_EXECUTE true
ENV DEFAULT_CATTLE_COMPOSE_EXECUTOR_EXECUTE true
ENV DEFAULT_CATTLE_CATALOG_EXECUTE true

CMD ["/usr/bin/s6-svscan", "/service"]

Thank you!

@teleki please try (sorry it wont format as code :/ ):
FROM armhf/ubuntu:16.04
ENV TERM xterm
RUN apt-get update && apt-get install -y --no-install-recommends software-properties-common
RUN add-apt-repository -y ppa:webupd8team/java && apt-get update
RUN echo 'yes' | DEBIAN_FRONTEND=Teletype apt-get install -y --no-install-recommends oracle-java8-installer
RUN java -version
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends
curl
mysql-server
xz-utils
keychain
unzip
openssh-client
iptables
iproute2
git
redis-server
zookeeper
spiped
RUN mkdir -p /var/run/mysqld && chmod -R 777 /var/run/mysqld
ADD https://github.com/imikushin/giddyup/releases/download/v0.10.1_arm/giddyup /usr/bin/
ADD https://github.com/imikushin/cluster-manager/releases/download/v0.1.4_arm/cluster-manager /usr/bin/
RUN chmod +x /usr/bin/giddyup /usr/bin/cluster-manager
COPY bin/* /usr/bin/
ENV CATTLE_HOME /var/lib/cattle
ENV DEFAULT_CATTLE_API_UI_INDEX //releases.rancher.com/ui/1.0.5
ENV CATTLE_DB_CATTLE_DATABASE mysql
ENV CATTLE_USE_LOCAL_ARTIFACTS true
ENV no_proxy localhost,<IP_ADDRESS>,localaddress,.localdomain.com
ADD artifacts /usr/share/cattle
ADD service /service
ENV S6_SERVICE_DIR /service
COPY target/arm/*static.tar.gz /s6-statics/
EXPOSE 8080
ENV CATTLE_HOST_API_PROXY_MODE embedded
ENV CATTLE_RANCHER_SERVER_IMAGE v1.0.1_arm
ENV CATTLE_RANCHER_COMPOSE_VERSION v0.7.4
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_LINUX_URL   https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-linux-amd64-${CATTLE_RANCHER_COMPOSE_VERSION}.tar.gz
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_DARWIN_URL  https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-darwin-amd64-${CATTLE_RANCHER_COMPOSE_VERSION}.tar.gz
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_WINDOWS_URL https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-windows-386-${CATTLE_RANCHER_COMPOSE_VERSION}.zip
ENV DEFAULT_CATTLE_CATALOG_URL="library=https://github.com/rancher/rancher-catalog.git,community=https://github.com/rancher/community-catalog.git"
EXPOSE 3306
ENV CATTLE_CATTLE_VERSION v0.159.7-arm-4
ADD https://github.com/imikushin/cattle/releases/download/${CATTLE_CATTLE_VERSION}/cattle.jar /usr/share/cattle/
RUN cd /usr && for i in $(ls /s6-statics/*static.tar.gz);do tar -zxvf $i;done && rm -rf /s6-statics/*static.tar.gz
RUN cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract
RUN curl -sLf https:${DEFAULT_CATTLE_API_UI_INDEX}.tar.gz | tar xvzf - -C /usr/share/cattle/war --strip-components=1
RUN /usr/share/cattle/install_cattle_binaries &&
cd $CATTLE_HOME && export IFS="," &&
for i in $DEFAULT_CATTLE_CATALOG_URL; do rancher-catalog-service -validate -catalogUrl=$i;done
VOLUME /var/lib/mysql /var/log/mysql /var/lib/cattle
ENV DEFAULT_CATTLE_MACHINE_EXECUTE true
ENV DEFAULT_CATTLE_COMPOSE_EXECUTOR_EXECUTE true
ENV DEFAULT_CATTLE_CATALOG_EXECUTE true
CMD ["/usr/bin/s6-svscan", "/service"]

Successfully registered agent :)

@alphawolf1988 Thank you for the help. Still no love. Here is what I get when I run the build-image.arm.sh:
...
Oracle JDK 8 installed
 ---> 2bbfa3297100
Removing intermediate container<PHONE_NUMBER>ff
Step 6 : RUN java -version
 ---> Running in b2a6a91defee
/bin/sh: 1: java: not found
The command '/bin/sh -c java -version' returned a non-zero code: 127

As I was Googling around, some messages were pointing me to think that the container may have a mixture of 32-bit and 64-bit libraries somehow. However, I am unable to assess if this is true or not.
I logged into the container:
docker run -ti 2bbfa3297100 /bin/bash
root@7c150d68f7f9:/# java -version
bash: /usr/bin/java: No such file or directory

Then I navigated to the Java directory. Here is what I got there:
root@7c150d68f7f9:/# cd /usr/lib/jvm/java-8-oracle/bin/
root@7c150d68f7f9:/usr/lib/jvm/java-8-oracle/bin# ls -la java*
-rwxr-xr-x 1 root root 7791 Jun  2 02:27 java
-rwxr-xr-x 1 root root 1809 Jun  2 02:27 java-rmi.cgi
-rwxr-xr-x 1 root root 8006 Jun  2 02:27 javac
-rwxr-xr-x 1 root root 8014 Jun  2 02:27 javadoc
-rwxr-xr-x 1 root root 8006 Jun  2 02:27 javah
-rwxr-xr-x 1 root root 8006 Jun  2 02:27 javap
root@7c150d68f7f9:/usr/lib/jvm/java-8-oracle/bin# ./java
bash: ./java: No such file or directory

Here is the puzzle: I see the file, it is there and when I run "file" on it, this what I get:
root@7c150d68f7f9:/usr/lib/jvm/java-8-oracle/bin# file ./java
./java: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=c46954c0ae919fe0f71b35f6716d88a811eaa92e, not stripped

Any more ideas?

@teleki Are you using a raspberry pi 3 or 2 and what OS distribution (and version) is used ?

@alphawolf1988 I am attempting to build a 4 unit ODROID C2 cluster. The specs are:

Amlogic S905 ARM® Cortex®-A53 (ARMv8) **2Ghz quad core CPU**

**2GB DDR3 SDRAM**

**32 GB eMMC 5.0 HS400 Flash Storage**

ARM Mali™-450 MP3 GPU (OpenGL ES 2.0/1.1 for Linux)

HDMI 2.0 4K/60Hz display

H.265 4K/60FPS and H.264 4K/30FPS capable VPU

Gigabit Ethernet

Ubuntu 16.04 LTS based on Kernel 3.14 LTS (GNU/Linux 3.14.65-66 aarch64)

@alphawolf1988 @imikushin unfortunately, still no love. I started from scratch again. It stopped with the following error:
...
Step 35 : RUN cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract
 ---> Running in 03035cafc513
/usr/share/cattle/cattle.sh: line 197: /usr/bin/java: No such file or directory
The command '/bin/sh -c cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract' returned a non-zero code: 1

Attached the entire output. build-image.arm.out.txt
Also attached the Dockerfile.arm file. Dockerfile.arm.txt
It seems that the problem is that java won't run, possibly due to a mixture of 64 bit and 32 bit code.
Any more ideas? Thanks in advance!

@teleki did you manage to get it run on the C2

Or trying other OS like Debian

@lavvy not yet. I am stuck for now. I have only tried it with the default OS that came on C2.

@lavvy not yet. I am stuck for now. I have only tried it with the default OS that came on C2.

This comment comes from a sadly underinformed place, but overall I've got
to observe that it seems that Java is the enemy of this quest.
Jacob Gadikian
E-mail<EMAIL_ADDRESS>SKYPE: faddat
Phone/SMS: +84<PHONE_NUMBER>
On Tue, Jun 14, 2016 at 9:36 PM, teleki<EMAIL_ADDRESS>wrote:

@lavvy https://github.com/lavvy not yet. I am stuck for now. I have
only tried it with the default OS that came on C2.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/4703#issuecomment-225901581,
or mute the thread
https://github.com/notifications/unsubscribe/AGz6iSZvdFvcmF7oRrJSK3OZGI4UmanSks5qLrx6gaJpZM4IYe7D
.

@faddat I'm afraid you are right. I installed Oracle Java directly onto the C2 and that worked fine. For some reason it doesn't work the way it is installed in the container. So far I have not had a chance to figure it out why.

For anyone who wants to give this a go... I have prebuilt images and instructions on my blog ... even with the incredible work that's been done so far, there's still a fair bit of manual work to bring up a node -- I didn't build rancher/server, just the things to run rancher/agent on arm while rancher/server is on amd64.

Have you attempted the Ur-Power heterogeneous cluster yet?  Clearly on the
server/agent level you have, but how might we make it work with some x86
agents and some ARM agents?
I'm thinking labels and careful rules, maybe?  (obviously anything using
ARM at this stage is a ways from production ready, but... I think
specialized compute is one of few lifelines Moore's Law has remaining.
:)
-Jake
Jacob Gadikian
E-mail<EMAIL_ADDRESS>SKYPE: faddat
Phone/SMS: +84<PHONE_NUMBER>
On Sun, Jun 26, 2016 at 3:06 AM, Rob Landers<EMAIL_ADDRESS>wrote:

For anyone who wants to give this a go... I have prebuilt images and
instructions on my blog
https://www.withinboredom.info/post/rancher-on-pi/ ... even with the
incredible work that's been done so far, there's still a fair bit of manual
work to bring up a node -- I didn't build rancher/server, just the things
to run rancher/agent on arm while rancher/server is on amd64.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/4703#issuecomment-228568071,
or mute the thread
https://github.com/notifications/unsubscribe/AGz6iSoNilX1Vc81w1J3cY15mu79629wks5qPYo4gaJpZM4IYe7D
.

@faddat Eventually (one day) rancher might read the metadata on the image and see what architecture it's built for and only deploy to the architecture its built for ;) Docker is kinda already working on that where pulling ubuntu:16.04 on x86 will pull the appropriate image, and will pull the equivelant on an arm architecture.
Until then, complex rules will be the norm in heterogeneous clusters.

+1

what is the progress of this issue, is there any other place we can be tracking it from, or is it in a little halt.
what about the agent , can't we be able to add an arm node at least, before the server comes.

following @withinboredom  blog I am trying to run add a Raspberry Pi 2 as a host, but it is failing to me
INFO: ENV: RANCHER_AGENT_IMAGE=rancher/agent:v1.0.2
Template parsing error: template: :1:8: executing "" at <.Volumes>: map has no entry for key "Volumes"
dirname: missing operand
Try 'dirname --help' for more information.
dirname: missing operand
Try 'dirname --help' for more information.
dirname: missing operand
Try 'dirname --help' for more information.
docker: Error response from daemon: Invalid volume spec ":": volumeinvalid: Invalid volume specification: ':'.
See 'docker run --help'.

Which distribution do you suggest me to use in the raspberry pi?

Hi, I'm having the same issue that @jgato is having. I am using Hypriot as the host OS:
pirate@black-pearl:~$ uname -a; docker version
Linux black-pearl 4.4.15-hypriotos-v7+ #1 SMP PREEMPT Mon Jul 25 08:46:52 UTC 2016 armv7l GNU/Linux
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:31:15 2016
 OS/Arch:      linux/arm

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:31:15 2016
 OS/Arch:      linux/arm

Doing a little digging and I found the line causing the error message @jgato and I were seeing:
https://github.com/rancher/rancher/blob/207b67ef6d5ff91db0e6dbde33a82a1c0288eb44/agent/run.sh#L193
When I run the command, I see the same error, but when I run the following, it seems to return fine:
pirate@black-pearl:~$ echo $(dirname $(dirname $(dirname $(docker inspect -f '{{index .Config.Volumes "/var/lib/cattle"}}' rancher-agent-state))))
.

@zerodivide1 how could I try it? Do I need to build my own agent images changing this line?

@jgato the problem with the arm image is that the docker client in it is a new version which has a different output format than the old version which run.sh is compatible with.
there are other issues even with the work around by @zerodivide1
https://www.adelton.com/docs/docker/docker-inspect-volumes-mounts#zero-value
The pulling request to build arm agent images is still in progress.

Any progress in this task?

Same question as @natami. Any status to when we can expect to get working Rancher ARM agent binaries?

@teleki : did you finally succeed to install rancher (server & agent) on your odroid-c2 cluster ? I would like to do the same thing, so I'm interested...

@mflor35 and I are also trying this with a PicoCluster running PINE64 boards.  We didn't discover this issue until after we started.  I'll post my results here.

@fredVT Sorry. I haven't had a chance to get back to this since last year, so no progress on it.

Yes, Rancher ARM agent is highly appreciated. Lots of RPi 3 idling ;-)

I subscribe on this issue as well.

I am new to Rancher and trying to test things on a Pi cluster therefore I am also looking for the rancher server docker image for Pi 3 in order to run an independent Pi cluster so that it is mobile and not dependent on a linux VM running the server. It sounds like it would be good to have a lightweight version of rancher server which does not overload the Pi 3.

I managed to execute the rancher/agent:v1.2.2 image on a Raspberry Pi but ran into the next problem. The server runs on a AMD64 machine and it seems like the image is downloading the agent from there!?This is the output of docker logs of the container:
INFO: Starting agent for DCA8E1DACFF85C6C62F2 INFO: Access Key: DCA8E1DACFF85C6C62F2 INFO: Config URL: http://rancher-server:8080/v1 INFO: Storage URL: http://rancher-server:8080/v1 INFO: API URL: http://rancher-server:8080/v1 INFO: IP: <IP_ADDRESS> INFO: Port: INFO: Required Image: rancher/agent:v1.2.2 INFO: Current Image: rancher/agent:v1.2.2 INFO: Using image rancher/agent:v1.2.2 INFO: Downloading agent http://rancher-server:8080/v1/configcontent/configscripts INFO: Updating configscripts INFO: Downloading http://rancher-server:8080/v1//configcontent//configscripts current= INFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh INFO: Sending configscripts applied 1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319 INFO: Updating pyagent INFO: Downloading http://rancher-server:8080/v1//configcontent//pyagent current= INFO: Running /var/lib/cattle/download/pyagent/pyagent-1-9e3dcab12585149a65d818082279708965cd826df1937a55b24f48428f8f0c0644fc1033b5be15c88dcf9540b1c04c67/apply.sh INFO: Updating host-api INFO: Downloading http://rancher-server:8080/v1//configcontent//host-api current=1-9608ead804f8cebbfe95b9f114701d8a5ac4244d603d196068043c902b1a0bdf INFO: Already up to date INFO: Sending host-api applied 1-9608ead804f8cebbfe95b9f114701d8a5ac4244d603d196068043c902b1a0bdf INFO: Updating python-agent INFO: Downloading http://rancher-server:8080/v1//configcontent//python-agent current=1-ba59beaa71a9d74e52708cff1beb0a6fd138b11c050cbdd33def3e7391bd696f INFO: Already up to date INFO: Sending python-agent applied 1-ba59beaa71a9d74e52708cff1beb0a6fd138b11c050cbdd33def3e7391bd696f INFO: Sending pyagent applied 1-9e3dcab12585149a65d818082279708965cd826df1937a55b24f48428f8f0c0644fc1033b5be15c88dcf9540b1c04c67 INFO: Starting agent /var/lib/cattle/pyagent/apply.sh INFO: Executing /var/lib/cattle/pyagent/agent INFO: Updating host-config INFO: Downloading http://rancher-server:8080/v1//configcontent//host-config current=host-config-1-0501df8804d09befe0190816322288c7413dfbd9631634a133aa87c2ae2fccc4 INFO: Already up to date INFO: Sending host-config applied 1-0501df8804d09befe0190816322288c7413dfbd9631634a133aa87c2ae2fccc4 /var/lib/cattle/pyagent/apply.sh: line 82: /var/lib/cattle/pyagent/agent: cannot execute binary file: Exec format error /var/lib/cattle/pyagent/apply.sh: line 82: /var/lib/cattle/pyagent/agent: Success
These are the Dockerfiles of
agent-base:
`# VERSION               0.3.1
FROM armv7/armhf-ubuntu:16.04
LABEL Version="0.3.1"
COPY ./rancher-entrypoint.sh /
ENV SSL_SCRIPT_COMMIT 98660ada3d800f653fc1f105771b5173f9d1a019
RUN apt-get update &&
apt-get install --no-install-recommends -y
arptables
bridge-utils
ca-certificates
conntrack
curl
ethtool
git
iproute2
iptables
iputils-ping
jq
kmod
openssl
passwd
pkg-config
psmisc
python2.7
tcpdump
vim-tiny
install cattle
RUN rm -rf /var/lib/apt/lists/*
RUN mkdir -p /var/lib/cattle /var/lib/rancher
RUN ln -s /usr/bin/python2.7 /usr/bin/python
RUN curl -s https://bootstrap.pypa.io/get-pip.py | python2.7
RUN pip install cattle
add the pre-compiled share-mnt to the image
ADD share-mnt /usr/bin/share-mnt
RUN  chmod +x /usr/bin/share-mnt
setup rancher/weave
RUN curl -sL https://github.com/rancher/weave/releases/download/r-v0.0.4/r > /usr/bin/r
RUN chmod +x /usr/bin/r
setup docker arm 17.03.0-ce
RUN curl -fsSLO https://get.docker.com/builds/Linux/armel/docker-17.03.0-ce.tgz && tar --strip-components=1 -xvzf docker-17.03.0-ce.tgz -C /usr/bin/
RUN chmod +x /usr/bin/docker
RUN rm /var/run
RUN mkdir /var/run
RUN curl -sLf https://raw.githubusercontent.com/rancher/rancher/${SSL_SCRIPT_COMMIT}/server/bin/update-rancher-ssl > /usr/bin/update-rancher-ssl
RUN chmod +x /usr/bin/update-rancher-ssl
ENTRYPOINT ["/rancher-entrypoint.sh"]
`
agent:
`FROM rancher/agent-base:v0.3.1
COPY register.py resolve_url.py run.sh /
RUN apt-get update &&
apt-get install --no-install-recommends -y
build-essential
git
wget
RUN wget https://storage.googleapis.com/golang/go1.7.5.linux-armv6l.tar.gz
RUN tar -xvf go1.7.5.linux-armv6l.tar.gz -C /usr/local
ENV PATH /usr/local/go/bin:$PATH
RUN mkdir /go
ENV GOROOT /usr/local/go
ENV GOPATH /go
ENV PATH $GOPATH/bin:$GOROOT/bin:$PATH
RUN go get github.com/rancher/agent
RUN cd $GOPATH/src/github.com/rancher/agent && go build && go install
ENTRYPOINT ["/run.sh"]
LABEL "io.rancher.container.system"="rancher-agent"
ENV RANCHER_AGENT_IMAGE rancher/agent:v1.2.2
`
I am not sure where to put the go build agent from the last image to not trigger the download from the server.
if you want to try this out, you have to compile share-mnt on your Raspberry Pi. Just check out the rancher/runc repository ,switch to the jazzhands branch and compile.

We have been doing some work on bringing newer versions of rancher-agent to aarch64 systems, it is still quite experimental but has been tested in a Cattle environment with rancher server 1.5.x in a raspberry pi 3.
For anyone interested to give this a try, these images and instructions to set it up are available on dockerhub.

@brunograz Any links to the sources, i could do with armhf. Asus Tinkerboard.

@LarsVader666 Sorry for the late reply. The approach I followed might not be the best practice for this case - as I did not use any of the dapper stuff - but it involved building binaries and adding them directly to the Dockerfile for each of the services (modifications in the Dockerfile were also required to make sure it is compatible with the aarch64 architecture).

First I built the agent-base container (the base container for most of the services)

Then you can build the agent container and the rest of the services. (healthcheck, scheduler, network-manager, dns, metadata, net...)

The biggest issue with dapper imo is that it does not download the proper binaries inside the Dockerfile of each service. I might be wrong about this, just don't have enough experience with it.
@dominictayloruk We don't have the source code published yet (my bad!) but will do some cleaning in code and publish it sometime soon.

What's the status?

Once the images are built, manifest-tool and/or manifest-list could be, so it'll download the image based on the architecture

I think some images have already been built for arm but it seems like the list in the first post has not been updated, for example:
host-api [rancher/host-api#77]
There also still is at least one open pull request about the issue and I wonder why this hasn't been merged yet:  agent-instance [rancher/agent-instance#18]

anyone have a copy of the docker-env.arm sample?  it is not available at the link above anymore and can't seem to find one.  thx

Any news on that? ARM support, with IoT world, would be a killer feature for Rancher. Imagine upgrading images on dozens of Raspberries agents... With Kubernetes supporting ARM64 natively, I think it could be a quick win for Rancher.

Anybody had any luck getting rancher-server/rancher-agent containers to work on Rancher 2.0?

@jmreicha I have, its actually fairly easy to do, there is a gotcha in that you have to use the View in API option to change a cluster setting in Rancher.  https://gist.github.com/ags131/7bdde11c932ef7a54f44c6decbfd88b8

@deniseschannon @smw355 could this be reopened?

deniseschannon closed this 8 hours ago
So we can state this is never going to happen, or...?

No it's an automatic close


Disable/Re-Enable rate limit for erc20 tokens through a bool flag

closes #309

commits to review

https://github.com/FuelLabs/fuel-bridge/pull/310/commits/a35b4d75654317da8af94c02378dde0d1fdac1e0
https://github.com/FuelLabs/fuel-bridge/pull/310/commits/b252a0f34d437e4916fcfeda31109fbb74ef3158


remove tm rollback command

current sdk version contains it's own rollback version which does tm rollback + sdk store rollback

Codecov Report

Merging #590 (723b899) into main (cb4deb2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #590   +/-   ##
=======================================
  Coverage   10.34%   10.34%
=======================================
  Files          55       55
  Lines        7770     7770
=======================================
  Hits          804      804
  Misses       6908     6908
  Partials       58       58

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb4deb2...723b899. Read the comment docs.


Bug issue

Hi, I got a bug when reproducing the translation task. Could you let me know if there are any possible solutions? It seems the problem is from dynamic_batching in petl. Thank you in advance!
TypeError: Wrong key type: '1609' of type '<class 'numpy.int64'>'. Expected one of int, slice, range, str or Iterable.
Main packages:
petl                     1.7.12
datasets                 1.11.0
fairseq                  0.12.2
hydra-core               1.0.7
jedi                     0.18.2
multiprocess             <IP_ADDRESS>
numpy                    1.24.2
pandas                   1.5.3
sacrebleu                1.5.1
sacremoses               0.0.53
tokenizers               0.10.3
torch                    1.9.0+cu111
torchaudio               0.9.0
torchvision              0.10.0+cu111
tornado                  6.2
tqdm                     4.65.0
transformers             4.9.0.dev0   /unify-parameter-efficient-tuning
Errors:
***** Running training *****
  Num examples = 3000
  Num Epochs = 30
  Instantaneous batch size per device = 10
  Total train batch size (w. parallel, distributed & accumulation) = 40
  Gradient Accumulation steps = 4
  Total optimization steps = 210
Traceback (most recent call last):
  File "/ivi/ilps/personal/stan1/unify-parameter-efficient-tuning/examples/pytorch/translation/run_translation.py", line 718, in <module>
    main()
  File "/ivi/ilps/personal/stan1/unify-parameter-efficient-tuning/examples/pytorch/translation/run_translation.py", line 630, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/ivi/ilps/personal/stan1/unify-parameter-efficient-tuning/src/transformers/trainer.py", line 1275, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/home/stan1/anaconda3/envs/adapter/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/stan1/anaconda3/envs/adapter/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/stan1/anaconda3/envs/adapter/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/stan1/anaconda3/envs/adapter/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/ivi/ilps/personal/stan1/unify-parameter-efficient-tuning/./petl/dynamic_batching.py", line 167, in __getitem__
    return self.dataset.__getitem__(item)
  File "/home/stan1/anaconda3/envs/adapter/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2709, in __getitem__
    return self._getitem(key)
  File "/home/stan1/anaconda3/envs/adapter/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2693, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
  File "/home/stan1/anaconda3/envs/adapter/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 583, in query_table
    _raise_bad_key_type(key)
  File "/home/stan1/anaconda3/envs/adapter/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 45, in _raise_bad_key_type
    raise TypeError(
TypeError: Wrong key type: '1609' of type '<class 'numpy.int64'>'. Expected one of int, slice, range, str or Iterable.

It seems the issue can be solved by changing some code in the torch even though it's a bit nasty.
change data = [self.dataset[idx] for idx in possibly_batched_index] to data = [self.dataset[idx.item()] for idx in possibly_batched_index]


Cannot downoald VS-CODE stable build from the website

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.74.3
OS Version: Windows 10 Pro 19045

Steps to Reproduce:

Go to https://code.visualstudio.com/
Click the Download for Windows Stable Build

Your stable build download link is broken or something else. Because I tried every way to download it but no success. I can download the insiders version but I need the stable one.
When I click download stable vesion button it redirects me to > https://az764295.vo.msecnd.net/stable/97dec172d3256f8ca4bfb2143f3f76b503ca0534/VSCodeUserSetup-x64-1.74.3.exe
and then page crashes with "This site can’t be reached, ERR_TIMED_OUT" error.
I have tried everything: phone, different pc, different network, VM, VPN on/off, extensions off still no success. Do you have any other official mirrors or how can I solve this?

I solved this by using VPN inside VM. But why? I still have no idea what's blocking it.

Seems to be an issue with your system's network and/or networking environment. I suggest following up with your sysadmin.


⚠ Bofrost.lu has degraded performance

In 3a57f16, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1466 ms

Resolved: Bofrost.lu performance has improved in aecd9cf.


What is exactly factor k?

Sorry for stupid question, but I understand that k multiplies the number of feature MAPs in cnn layers. So in the first layer, you use 16 feature planes, the second will be 16*k feature planes and so on. But what is the difference to original deep residual net? I read original one https://arxiv.org/pdf/1512.03385.pdf, and see that authors also increase number of feature planes (Figure 3). Please point out for me the difference?

@saix47 this line should explain it https://github.com/szagoruyko/wide-residual-networks/blob/master/models/wide-resnet.lua#L93
original resnet increases #planes only at each new group of blocks, wide resnet has k times more #planes in each 3x3 convolution

Oh, thanks. I got it.


Focus should be moved back to the input when an image attachments are removed

Testing #229263

Attach an image
Tab to the "remove" button
Hit enter
🐛 the focus is kind of hidden. Hitting tab brings me to the footer actions. I would instead expect focus is moved back to the input box when the last attachment is removed

This applies to quick chat as well


[Azure Functions] Add documentation for remote debugging

Follow up item from issue https://github.com/Azure/azure-functions-host/issues/5344
As of now there is only documentation for debugging azure functions locally.
https://docs.microsoft.com/en-us/azure/azure-functions/functions-develop-local
Need to add sections on remote debugging for each language. For languages where remote debugging is not supported, need to state that explicitly.
cc @ggailey777

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ad301acd-cfb9-5413-b882-e20e22632215
Version Independent ID: f1738d3d-e847-6500-4a40-59f3abd8517c
Content: Develop and run Azure functions locally
Content Source: articles/azure-functions/functions-develop-local.md
Service: azure-functions
GitHub Login: @ggailey777
Microsoft Alias: glenga

Thank you for your feedback! This issue is now assigned to the content author, @ggailey777, to review further and take the right course of action.


Add option for root node name on Import 3D scene window

Fix #6822

Added option for root node name, not just taking filename as @reduz commented https://github.com/godotengine/godot/issues/6822#issuecomment-253791761


[Masthead]: A11y - Content is not within a landmark element

Description
===== DEFECT DESCRIPTION =====
All content must reside within an element with a landmark role. Landmark roles provide programmatic access to sections of a web page, making it easy for the user to understand the structure of the page and easily navigate to different sections. They also enable users to skip over blocks of content that are repeated on multiple pages.
===== EXPECTED RESULT =====

Enclose the content within an HTML5 sectioning element with an implicit landmark role (e.g. Content);

Or, use the "role" attribute on an enclosing element to specify an appropriate ARIA landmark value (e.g.
...
).

Browser
Chrome, Safari, Firefox, Microsoft Edge
Carbon for IBM.com version
1.11.0
Severity
Severity 2 = Aspects of design is broken, and impedes users in a significant way, but there is a way to complete their tasks. Affects major functionality, has a workaround.
Application/website
cloud.ibm.com
Package
@carbon/ibmdotcom-web-components
CodeSandbox example
https://codesandbox.io/s/github/carbon-design-system/carbon-for-ibm-dotcom/tree/main/packages/react/examples/codesandbox/components/Masthead?file=/src/App.js
Steps to reproduce the issue (if applicable)
===== ENVIRONMENT =====
IBM CI 162 Checkpoint Violation: 1.3.1 Info and Relationships
IBM Guidance: https://www.ibm.com/able/requirements/requirements/?version=7_1#1_3_1
Test Tool Type: IBM Equal Access Accessibility Checker
=====STEPS TO REPRODUCE =====

Open https://dev.console.test.cloud.ibm.com
Login with IBM ID & password
Click Help ->Docs
Click All product docs
Click Discovery
Check Accessibility

Release date (if applicable)
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I checked the current issues for duplicate issues

@kennylam Is there someone from your team that can help take a look at this issue? Our innovation team is booked for Sprint 05 and this issue looks to be a high priority of the Cloud Platform team.

@rohitrajmishra the version of ibmdotcom/web-components on that page is quite old. The current version is 1.16.0. Here is the production version of the masthead, which does not have the landmark violation.
https://www.ibm.com/standards/carbon/web-components/iframe.html?id=components-masthead--default

From what I can see, this is an issue with the page, not the masthead. Using semantic elements (or roles where necessary) to define landmark page regions is the responsibility of page authors. You can fix the issue by replacing
 with either:

 preferred

The landmark error does not appear in the masthead Storybook preview, where the masthead is a descendant of an element with role="main".

The fix is to place the masthead in landmark region with role=main or role=banner. Converting the parent of the masthead to a semantic element () could've been used as well but the role approach is sufficient.


feat: add piano board to project

Hi Arash and Thanks for your contribution
I have been thinking about making a GUI for Keyano for a long time
your changes is not bad but it only work in some keys. Anyway I accepted your pull request
Bravo 😉🔥


Convert src/controllers/utils to TS

Is your feature request related to a problem? Please describe.
Convert src/controllers/utils to TS
Describe the solution you'd like
Conver src/controllers/utils to TS
Avoid making logical changes to these files. Some of these files will need new types or interfaces, add them whenever necessary. Prettier will format these files which is okay.
Describe alternatives you've considered
N/A
Additional context
N/A

There are 3 files in .js format so all the 3 files should be converted to TS?

@Harshal662 that's correct
i assigned to both of you @Dhaxor @Harshal662


Integrate GraphQL into the Project

Description
GraphQL will be used as a query language for our API, allowing us to define what data we need and retrieve it efficiently. This integration will enable us to send queries and mutations, making data interactions with our backend more flexible and efficient.
Requirements

Set Up GraphQL in the Project:

Install necessary GraphQL dependencies (e.g., Apollo Server, GraphQL modules).
Ensure dependencies are listed in the project’s package.json for future installs.

Define Basic Schema:

Create a schema file to outline the data structures, including types for Device, Location, DeerSighting, Alerts, and Users.
Each type should have basic fields; for instance, Device might have id, name, and status fields.

Implement Queries and Mutations:

Write basic queries to fetch data (e.g., get a list of devices or recent sightings).
Implement mutations for adding new entries, such as creating a new deer sighting or alert.

Test GraphQL Integration:

Use a tool like GraphQL Playground or Postman to test queries and mutations.
Verify that data returned matches the expected schema.

Acceptance Criteria

[x] GraphQL dependencies are installed and added to package.json.
[x] Basic schema for each entity is defined and documented.
[ ] Queries and mutations are set up and tested for all required types.
[ ] Integration is verified through testing, with documented examples of successful queries and mutations.

Added the query to retrieve from a database, and reviewed documentation to help understanding


How to specify private key password?

How do you specify the private key password?

This is more a question for lsyncd than this container.
In theory you'd specify it in the same way you specify it to rsync, which is used under the hood. If rsync has the option to provide, then you can specify it in the config. See https://axkibe.github.io/lsyncd/manual/config/layer4/ for more config details.
Given this isn't something the container deals with, i'm going to close the issue.


Publish to Maven Central

Are there any plans to put this on Maven? It'd be a lot easier to JIJ that way.

Unfortunately, Jitpack (with a release's commit hash being included in the release description) is the only solution that I have for having this mod hosted on a Maven repository; I don't have any kind of web server so I can't host there, I attempted to maintain one in the Git repository itself, but it was a terrible solution that wouldn't last for a long time; If there was a good solution, I'd use it, but yeah, for now, I'm stuck with Jitpack

You don't actually need to host a maven.
You can use Maven Central to host artifacts for free, in an easily accessible manner (almost everyone has it in their build already, and fabric loom adds it automatically), or if you want your own repository, JFrog has a service with free, managed Artifactory instances.

Alright, I'll see if I can get this published to Maven Central

Alright, so, it took a long while, I fought against ancient software, got a pair of keys and this mod will be available on Maven Central soon!
Also, one question, did Jitpack not work? Because I used it for Ok Zoomer and it worked just fine

Awesome! On the Jitpack note, it's functional but my understanding is that it's hard for Fabric to version correctly because it's based on commit hashes and not semver. @kb-1000 can correct me if I'm wrong.

Awesome! On the Jitpack note, it's functional but my understanding is that it's hard for Fabric to version correctly because it's based on commit hashes and not semver. @kb-1000 can correct me if I'm wrong.

Oh wait, I think I know exactly what you mean; Ever since I began to use a + instead of a - for the Minecraft version part of the version (it conflicted with the alpha/beta part), Jitpack stopped accepting releases since apparently, the + breaks it; I attempted to work around it by providing commit hashes for Jitpack usage in each GitHub release, but yeah, it's definitely not a real solution
Anyway, yeah, enjoy not having to use commit hashes!


SL-19707 Support formatting and parsing llsd objects of arbitrary depths.

Change the formatting code to use iteration instead of recursion to build the resultant data.  This prevents the stack overflow errors we were getting with depths of over 300 or so.
Also, improve performance in python3 formatting by using the python3 str.translate functionality to do some of the key, string, and uri modification in one step instead of many.

I have read the CLA Document and I hereby sign the CLA


Theming

I'd like to start expanding theme support, and it seems like the best way to do that is via a pref window.
Scope expanded to full theming. Just makes sense.

Pane is implemented, ThemePark pretty well integrated now. Just have to work out the Preview extension theming.

I think we can call this done enough to close!


fix(db): db seed should try all known config file names by default

Fixes #602

Oh sure. I considered it but thought it might be overkill – happy to though.

Hey @mcollina, want to make sure I'm setting off with the right philosophy here. The current test of an acceptable config file name seems to rely on packages/db/fixtures/sqlite/platformatic.db.json. For a meaningful end to end'ish test one would need to try each of the supported files in turn, so I could see swapping this out for a file containing the source data and writing each out one in turn, with a teardown after each (doing this is a temp directory would be sensible I think). The other way to go would be mocking, but I can see this is only used for undici usages rn and there's no general purpose mocking library in the project, which I'm going to guess is a deliberate decision. While I'm here, I'm a bit puzzled by the repetition of ourConfigFiles here, what's the reason for the hof supplying its own value? Thanks!

Yes exactly, don't mock unless it's absolutely necessary.

While I'm here, I'm a bit puzzled by the repetition of ourConfigFiles here, what's the reason for the hof supplying its own value? Thanks!

Where is that repeated?

It's first defined here.

They are explicitly different, it's not repeated.

Gaaah yes of course they are 🤦 . This might be related to the fact that I broke my glasses a week ago and haven't received the replacements yet 🙃 . Thanks!

Can you please fix linting?
/home/runner/work/platformatic/platformatic/packages/db/lib/seed.mjs
Error:   1:10  error  'resolve' is defined but never used  no-unused-vars

✖ 1 problem
 ELIFECYCLE  Test failed. See above for more details.
Error: Process completed with exit code 1.

Thanks for your patience here @mcollina , between learning a new test framework, getting my head round a largeish project, the day job and the dog it's taking a while.

I don't understand why this is tripping the DCO check as I rewrote the first commit to include the signoff line (I now see that this is the "least preferred option" but muscle memory).

I've checked all commits and they are all correctly signed off, not sure where the DCO app got stuck

👀

The test failure is in another package and I'm reasonably certain is unrelated to the change here (actually, I'm struggling to identify exactly which test has failed, I had to grep the logs for "not ok" and then the most helpful clue seems to be tapError: no plan.

Thanks for re-triggering the tests. Unfortunately I don't have access to a Windows machine so I can't try to debug the continued failure.

unfortunately we have a few flaky tests on windows


The player replaces the poster image with the first frame of the video for particular URIs

The poster image gets replaced by the first frame of the video for specific URIs. Please check this JS Fiddle for a repro: https://jsfiddle.net/17cgays4/11/
We've generated the URIs using Azure Media Services. The URIs generated for a resource works like it should. However when using a filter on the same asset, the player replaces the poster image with the first frame of the video. Everything else seems to work like it should.

Have you read the FAQ and checked for duplicate open issues?: Yes
What version of Shaka Player are you using?: 2.4.4
Can you reproduce the issue with our latest release version?: Yes
Can you reproduce the issue with the latest code from master?: Haven't tried.
Are you using the demo app or your own custom app?: Custom, doesn't matter though.
If custom app, can you reproduce the issue using our demo app?:
What browser and OS are you using?: Google Chrome 69.0.3497.100 (Official Build) (64-bit) / MacOS 10.14
What are the manifest and license server URIs?:
Working URI: https://amplilymedia-usea.streaming.media.azure.net/38df02d2-ae86-449f-9bab-05a570e77758/5d3997b5-396c-40fd-a074-5ab0909a.ism/manifest(format=mpd-time-csf)
Failing URI: https://amplilymedia-usea.streaming.media.azure.net/38df02d2-ae86-449f-9bab-05a570e77758/5d3997b5-396c-40fd-a074-5ab0909a.ism/manifest(format=mpd-time-csf,filter=10MinuteTi-filter-1257a65813ef4e5aa5f2f352c94261f1)
(Clear streaming -- no licenses)
What did you do?
What did you expect to happen? expected the poster image to stay
What actually happened? the poster image got replaced

When the poster is and isn't drawn is a behavior of the browser, not Shaka Player. Specifically, the poster is drawn if the video is on the first frame and has not yet been played. The filtered manifest you provide doesn't feed in the first few segments, but the video element knows that they are there due to the timing data in the mp4 files... so the poster never draws.
That's on-spec behavior, by the way; see the w3 spec on video elements. I don't think they were thinking of your specific use case when they were designing it.
As far as what we can do about this...

We could re-package the incoming media files to change all of their timing data, so that the first segment passed in is now the actual first segment. That would be an impractically large change for this problem, though, and also might require preloading the entire video before playing (depending on how we do the re-packaging).
We could handle drawing the poster manually, instead of leaving it to the video element. This is probably the easiest solution, but adding new elements to the DOM is outside the purview of the core Shaka Player library.

Option number one there is almost definitely out of the question. Option number two... maybe. We'll consider it. For now, though, I'll mark this issue as external.

The reason the poster stops being displayed in that asset is because there is a gap at the start.  When we start playing, we see the gap at the start and seek over it.  Because the time is no longer 0, the browser stops displaying the poster.

Closing this stale external (non-Shaka) issue. If you disagree, please reply with @shaka-bot reopen, and it will be automatically reopened.


Make shortcuts keyboard layout independent

background in #878

Shortcuts (e.g. switching tabs, open settings, ...) are hard coded to en-US keyboard layout key codes.
On other keyboard layouts (e.g. de-DE) some characters have different key codes, thus rendering the shortcuts unusable or even blocking some basic functionality (like in #878).

Investigate, if UI Events can be used to solve this:
http://www.w3.org/TR/DOM-Level-3-Events/#keys
@stephankn thanks for the hint.

Partly solved by only allowing tab shortcuts for 1to 6.
Aiming for a proper solution in a future release.


Add HomematicIP Cloud dimmer light device

Description:
Add support for dimmer light device.
Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass

If the code communicates with devices, web services, or third-party tools:

[x] New dependencies have been added to the REQUIREMENTS variable (example).
[x] New dependencies are only imported inside functions that use them (example).
[x] New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[x] New files were added to .coveragerc.

@MartinHjelmare Thanks for the review, I had to close the other PR as in the 3th party lib are more changes to be considered e.g. enums for state flags. This will come in a separated PR.
Ciao Mattias


Possible to deal with East Asian wide character?

In East Asian languages like Chinese, Jpanese, etc., some wide characters are twice the width. For example, given this CSV:
"C1","C2","C3"
"吃饭","睡觉","打豆豆"
"","睡觉","打人"

CSVLook draws the following table:
|-----+----+------|
|  C1 | C2 | C3   |
|-----+----+------|
|  吃饭 | 睡觉 | 打豆豆  |
|     | 睡觉 | 打人   |
|-----+----+------|

I'd love to find a way to support these languages better, but I'm guessing this is impossible to fix without going char by char and testing where it falls in the unicode set. That sounds... very slow.
If any has a suggestion for a performant (and ideally simple) way to fix this I'm all ears.

According to Google, the standard library already has a solution.
unicodedata.east_asian_width(unichr)

Returns the east asian width assigned to the Unicode character unichr as string.
This StackOverflow post covers a lot more details:
http://stackoverflow.com/questions/23058564/checking-a-character-is-fullwidth-or-halfwidth-in-python

Right, but you still have to iterate over, call that method and reformat every single character you write out, which could be extremely cost prohibitive. Then again, if we're talking mainly about csvlook that might be a reasonable trade-off, since its only meant for displaying a subset of rows anyway...

I do not think the utilities other than csvlook would need to consider this special case, as it is only for rendering. This would add a constant factor to the runtime.
I just did some simple test.
s = "今天你吃饭饭了么"

def f1():
    for i in xrange(10000000):
        len(s)

def f2():
    for i in xrange(10000000):
        for c in s:
            unicodedata.east_asian_width(c)

f1() would take around 600 ms, while f2() would take around 17 sec. However, in real situation, a LRU cache (with the cell content as key and its on-screen length as value) may make some improvement, as there would usually be duplicate cell contents.
I think maybe we can add an argument to handle this special case, while not lose the performance for general cases.

Yeah, I'd be happy to have a flag for this, at least for csvlook and maybe for other utilities (csvstat, etc.)
If you want to send a pull request that would be great!

Sure. I will see what I can do. BTW, it seems csvkit is under heavy refactoring right now. When will the code be relatively stable?

I'm glad you said that, because actually this change should be made in agate and then #515 should pull it into csvkit. Agate is pretty stable, so if you wanted to look at making the change there, it'll get pulled into csvkit when we refactor csvlook.

Sure. I am just learning agate now. It is a quite interesting package. I will contribute the code to agate first. Thanks for the help.

Cool - closing as the issue should be re-created on agate.


Header to metadata filter (#2436)

Add support for extracting dynamic metadata from requests. This can then
be used as static metadata would be used (e.g.: for subset load balancer
metadata matches, logging, etc).
Risk Level: Low
Testing: unit-test
Docs Changes:
Basic docs, more later.
Release Notes:
N/A
Signed-off-by: Raul Gutierrez Segales<EMAIL_ADDRESS>

Hi @rgs1 ! Very nice! Let me know when you think it's ready to be tested. I see you're still adding commits. I'm happy to test it and confirm that it works for me too.

@bmetzdorf yep sorry, still ironing out a few things... Will let you know!

@bmetzdorf @ggreenway ok, I think it's going through now... Sorry for the noise. Let me know if this goes along with what you are thinking. Thanks!
cc: @zuercher who filed #2436.

@bmetzdorf sorry, had to rebase plus i added one more commit to avoid string copies.

Btw, I am not sold on calling this filter header-to-metadata... That's just what we called it internally. Something more descriptive would be better.

Hi @rgs1.
So, the simple request_rules use case that I'm interested in works fine for plain HTTP, but as you already mentioned in #2436 it is missing regex extraction support. In addition to that though, it seems as if currently (and I know there's some work in progress to overhaul this) this (or any) http filter does not get called for websocket connections, which unfortunately is what I'd require (sigh..).
So I'm wondering if we could provide some functionality of this in the route matcher (which does work for websocket connections) instead:
    - match:
        prefix: "/"
        headers:
        - name: x-something
          value: abc([0-9]+)
          regex: true
      route:
        use_websocket: true
        cluster: mycluster
        metadata_match:
          filter_metadata:
            envoy.lb:
              digits_from_x_something: \1

We could also have two different implementations, until the websocket stuff gets refactored.
What do you think?

@bmetzdorf we've got many many route matchers, so if we implement it there we'd need to be able to configure this at the RouteConfiguration [0] or Route [1] level — to avoid the copy/pasta. But yeah, I think it would work for us.
cc: @derekargueta
[0] https://github.com/envoyproxy/envoy/blob/master/api/envoy/api/v2/rds.proto#L39
[1] https://github.com/envoyproxy/envoy/blob/master/api/envoy/api/v2/route/route.proto#L116

@rgs1 If you have many route matchers, would the Route level still be ok for you?
Long term I'd think that the HTTP filter is the right approach. The filter should work with the upcoming websocket changes. Until then the limited functionality I require could be provided in the Route level, which would be referencing a regex capture group inside metadata_match. Regex matching is already supported.

@bmetzdorf yeah, Route level is good I think.

@zuercher thanks for the review! I've addressed most of your comments in 471e25fc5163596a31829648b8a488344a83e855, the commit description has details and questions about the things I am not so sure about... LMK what you think. Thanks!

@zuercher i addressed your latest comments, lmk if i missed anything — thanks!

Hmm, we should probably emit some stats (e.g.: total count of  metadata keys added, total per key, etc)? We haven't had a need for them internally (yet), but I actually think they could be useful for users to validate things are working as expected. Thoughts?

@zuercher lmk if you think we should have stats for this iteration... or maybe that's something that can be addressed later?

@rgs1 I think you could leave stats for another pass. It'll keep this simpler and then the stats change will be easy to review as well.

@zuercher addressed, thanks! Couldn't find the annotation to ensure that at least one of on_header_present or on_header_missing is set. I only see oneof:
https://developers.google.com/protocol-buffers/docs/proto#oneof
I'll keep looking. In the meanwhile, it's handled via code — we should probably also
log an error if we find a rule without any action (on present or on missing).

@zuercher we could change the protobuf defs to make it easier to validate things at that level. If we reworked the Config message to look something like:
message Config {
  enum ValueType {
    STRING = 0;
    NUMBER = 1;
  }

  // Trigger an action if a header is either present or missing.
  enum HeaderTrigger {
    present = 0;
    missing = 1;
  }

  message Action {
    // What triggers this action.
    HeaderTrigger trigger = 1 [(validate.rules).enum.defined_only = true];

    // The namespace — if this is empty, the filter's namespace will be used.
    string metadata_namespace = 2;

    // The key to use within the namespace.
    string key = 3 [(validate.rules).string.min_bytes = 1];

    // The value to pair with the given key.
    //
    // When used with `HeaderTrigger.present`, if value is non-empty it'll be used
    // instead of the header value. If both are empty, no metadata is added.
    //
    // When used with `HeaderTrigger.missing`, a non-empty value must be provided
    // otherwise no metadata is added.
    string value = 4;

    // The value's type — defaults to string.
    ValueType type = 5;
  }

  // Apply to requests or replies?
  enum RuleType {
    requests = 0;
    replies = 1;
  }

  // A Rule defines what metadata to apply when a header is present or missing.
  message Rule {
    // What to apply this rule to.
    RuleType type = 1 [(validate.rules).enum.defined_only = true];

    // The header that triggers this rule — required.
    string header = 2 [(validate.rules).string.min_bytes = 1];

    // The action to apply to this header.
    repeated Action actions = 3 [(validate.rules).repeated .min_items = 1];

    // Whether or not to remove the header after a rule is applied.
    //
    // This prevents headers from leaking.
    bool remove = 4;
  }

  // The list of rules to apply to requests/responses.
  repeated Rule rules = 1 [(validate.rules).repeated .min_items = 1];
}

it would make it easier to express validations.... Thoughts?

I think the version that feels most natural to configure is probably the version to use, as long as the validation occurs somewhere.

@zuercher lets go with what's there then. I've pushed a commit that rejects Config objects with rules that have neither on_header_present nor on_header_missing set, lmk what you think. Thanks!

I think the h1_capture_fuzz_test failure in unrelated... will look in a bit.

Lets see if master fixes the h1_capture_fuzz_test failure...

@zuercher thanks for the review!
@ggreenway @mattklein123 @alyssawilk any of you with cycles to review this filter?

I think per https://github.com/envoyproxy/envoy/blob/master/GOVERNANCE.md#extension-addition-policy
If you add a CODEOWNERS with you and zuercher (assuming he volunteers to own) you're good to go with his approval.  And fixed format :-)

Yup per @alyssawilk that's fine with me. I don't have time to review this right now. Please make sure that all the docs are in place though. Thank you!!

@alyssawilk ah nice, will do!
@zuercher are you ok with being added to CODEOWNERS for this? :-) (no pressure)

@zuercher done — i went with Header to Metadata though, since the IPTagging filter also interchanges IP Tagging and IPTagging.... though I'll switch back to Header-To-Metadata if you have strong feelings about it :-)
Output is here: https://65121-65214191-gh.circle-artifacts.com/0/source/generated/docs/api-v2/config/filter/http/header_to_metadata/v2/header_to_metadata.proto.html

Hi @rgs1 . Unfortunately my use case requires websockets with early body, which violates the websocket spec. While #3301 / #3376 might still allow this for now (I haven't tested it yet) it will be a problem in the future when strict websocket support will be enforced, in which case the http_filter approach for extracting metadata won't work for me. @alyssawilk @ggreenway Do you see any chance that we might allow early websocket bodies even in the long term?

We've got regression tests for early websocket bodies for both the old style and new style websockets so hopefully you'll be set.
When you say "websockets with early body" do you mean explicit content-length != 0, or websocket payload arriving with the upgrade request?  I could see Envoy rejecting GET-with-body but I don't think anyone plans on killing off early data handling, especially if you comment on that regression test that there are users using it.

@alyssawilk I use both, upgrade request coming in with body payload as well as a upgrade response with content-length > 0. Is that something we could entertain to support long term (maybe hidden behind a use_relaxed_websockets if needed)?

More likely a more generic allow_disallowed_bodies or something but I suspect we could leave it config available.   As I'd likely be the person implementing any adhere-to-the-spec feature and @mattklein123 or I would likely be the ones to review if someone else writes it, we'll keep your traffic in mind :-)

That sounds great! Thanks @alyssawilk !

@rgs1 In regards to regex, what do you think about something like this?
http_filters:
  - name: envoy.filters.http.header_to_metadata
    config:
      request_rules:
        - header: x-version
          match: "^[0-9]+\.([0-9]+)$"
          on_header_present:
            metadata_namespace: envoy.lb
            key: version
            type: STRING
          on_header_match:
            metadata_namespace: envoy.lb
            key: minor_version
            value: "$1"
            type: STRING
          on_header_missing:
            metadata_namespace: envoy.lb
            key: default
            value: 'true'
            type: STRING
          remove: false

@bmetzdorf generally lgtm, lets try to make it consistent with how exact match vs regex match is implemented in other places... Something like this perhaps:
https://github.com/envoyproxy/envoy/blob/master/api/envoy/api/v2/route/route.proto#L941
What do you think?

Yeah that would work.

@bmetzdorf that looks good

@rgs1 While thinking about this more I'm realizing that it might be easier to create a new filter and retire the current one later. I think it could be a little bit messy trying to cram both types of configuration and logic into the existing one. What do you think?

@bmetzdorf sure that's fine too


Update "PrivilegesDemoter_PrivilegesApp-2.0.pkg" to include Privileges.app version 1.5.3

A new version of Privileges.app was released 28 days ago, this has been the first update to Privileges.app in almost 2 years. Bumping the current version to 1.5.3 from 1.5.2 with some bug fixes and feature additions.
https://github.com/SAP/macOS-enterprise-privileges
Would you be able to update the .pkg for PrivilegesDemoter_PrivilegesApp-2.0 to include this latest release of Privileges.app?
Thanks so much!

Thanks so much @sgmills!


"sui start" panic (main branch 1.29)

Steps to Reproduce Issue
(1) Build sui binary from main branch (currently 1.29.0-f6f2584c5072 )
(2) env RUST_LOG=error sui start --network.config /home/olet/suibase/workdirs/localnet/config/network.yaml
network.yaml is attached ( network.yaml.txt )
Expected Result
"sui start" to normally start and keep running.
Actual Result
A panic

This problem is not happening with devnet branch 1.28.1 (and many older versions).
The error message shows a path that includes the file "network.yaml" which is not a directory (it is the file specified with --network-config), could this be the problem?
System Information
ubuntu, rustc 1.75
Tests with 1.29 is my own cargo build, while 1.28.1 was the precompiled Mysten binaries ( sui-devnet-v1.28.1-ubuntu-x86_64.tgz )

Thanks for reporting this. I've made some changes to sui start that might be causing this. I'll look into it now, but would help me to know:

where is your genesis blob stored (folder relative to localnet/config/network.yaml). I assume it is in home/olet/suibase/workdirs/localnet
where are the previous DBs stored: are they in the home/olet/suibase/workdirs/localnet? Do you see a full_node_db there?

When testing, my script delete and regenerate "everything" from scratch. So in the case of testing 1.29, it does not reach the point of creating full_node_db.
(1) The blob is in /home/olet/suibase/workdirs/localnet/config

(2) Will follow up in a few minutes about where full_node_db is created when testing with 1.28.1

I see, in this case, I think you might just get away with passing the folder instead of the actual network file
sui start --network.config ~/suibase/workdirs/localnet/config.

But I will fix it asap so that it accepts the network.yaml config file as it used to.

Thanks. No rush to fix for me... as long it works as used to in devnet/testnet branch :smile:
Just FYI, specifying just the directory did not work:

Going on a tangent about full_node_db (can create another issue as needed?)
With 1.28.1, it is created in ~/.sui/sui_config ( it might have always work that way and never noticed).
This is inconsistent with all the rest being created in ~/suibase/workdirs/localnet/config

Thanks for the detailed info! I'll try to put up a fix today.

More info related to full_node_db:

thanks @mario4tier. I am not fully sure how suibase does stuff, particularly, why it creates another config-default folder to store the DBs.
I have a PR up that should fix the issue you have. If you have a chance to build the sui binary from that branch https://github.com/MystenLabs/sui/pull/18469 and try it locally, let me know if you still encounter issues.
Thanks for your patience here!

It works (tested with latest 1.29 branch + changes to sui_command.rs)
About DBs location
One key design choice for Suibase is to not interfere with the Mysten Labs official installation. So the Suibase localnet should not depend or even "touch" ~/.sui
That design allows to support a distinct set of binaries/repos/config/keystore for each network. The users/app can then operate localnet/devnet/testnet/mainnet concurrently... there is no "switch env" for Suibase users.
To keep things clean, all files related to a network are kept in a single place (e.g. ~/suibase/workdirs/testnet). full_node_db is the only exception right now...
I tried to modify the fullnode.yaml with an absolute path like done in network.yaml, but it did not work (still created in ~/.sui/sui_config ).
In short, ideally, Suibase would benefit from being able to specify the full_node_db path (but it is not breaking anything right now).


GET_ANCHOR function

Hey there ! 👋🏼
I really like that I can provide people with a clickable link that takes them to a specific row of a document, however I cannot find a way to create such link from a Reference object except with a fairly hacky formula.
Would you be interested in us developing a GET_ANCHOR function that would get a Record object (either returned by lookupOne or stored in a Reference column) as a parameter and return the anchor URL for said record ?

Seems useful @vviers. Could be made to rhyme with SELF_HYPERLINK, or perhaps be another optional argument for that method. What was your thinking about which widget/view to link to?

I would like this!

My plan was to :

add an anchor attribute to Record objects that contains the anchor url
create a function that would simply return this attribute

For example :
> MyTable.lookupOne(foo=$bar).anchor
'https://myorg.getgrist.com/o/docs/cy6oghugqQ8D/Document/p/2#a1.s3.r1.c12'

# equivalent to

> GET_ANCHOR(MyTable.lookupOne(foo=$bar))
'https://myorg.getgrist.com/o/docs/cy6oghugqQ8D/Document/p/2#a1.s3.r1.c12'

I'm happy to reuse SELF_HYPERLINK if you think that's more appropriate, although I think that would make this already quite complex function (for most users) even more complicated, so maybe it would be worth spending a bit of time redesigning this function ?

which widget/page to link to

That's a great question 🤔 I guess I'd be happy with any of those two options :

anchoring to the record in the source data (which is how I think of it when I make a Reference column or use the lookupOne method) so the anchor would have /p/data instead of /p/2 (and idk if s and c are required or can be ommited?)
have the widget / page be specified as parameters in the GET_ANCHOR function (maybe still defaulting to the source data ?)

wdyt ? I haven't really dug into the technical bits of this proposal yet so might be missing something

Riffing off of this idea : something else that would be quite cool is a right-click option to See in source data for any value in a Reference column that would take the user to the specific line that is referenced (though that would likely be more work)

s refers to the ID of the view (aka "page widget") -- since the same row may be present in multiple widgets (tables, cards) on the same page. And in fact, which table the row is from is determined by this parameter.
c is the colRef of the column where to place the cursor. It has to be one of the available columns in the given widget, though we could relax it to be optional, and place the cursor in the first column when omitted.
a is always 1, I think, it specifies the type of anchor link, and this is the only type we have currently.

So s is the biggest question in my mind -- what to do when a page has multiple widgets.

s refers to the ID of the view (aka "page widget") -- since the same row may be present in multiple widgets (tables, cards) on the same page. And in fact, which table the row is from is determined by this parameter.
[...]
So s is the biggest question in my mind -- what to do when a page has multiple widgets.

If we choose to always link to the p/data/ page (which again is the way I think about it in the context of Reference objects), could we compute a sensible default for the s parameter ? could we maybe even get rid of the s parameter ?

Riffing off of this idea : something else that would be quite cool is a right-click option to See in source data for any value in a Reference column that would take the user to the specific line that is referenced (though that would likely be more work)

We're currently working on a feature that might get at your desire here (if I am understanding it correctly) and may cover some other use cases in this thread.
Quick description of what we're working on:

Every table has a default card view.
By default, all referenced records in Reference and Reference List columns can be opened in this default card view in a lightbox (no need to configure or add any more widgets to pages, just works).
In addition, any row in a table can be opened in this default card view.
That default card view can be configured per table, by document editors with schema permissions. Each table's default card configuration is global for the doc, which means that opening records in the default card (rows or referenced records) from any page/widget applies the table's configured card widget. If you need something different, you can always add a card using the current method.
Default card view can be disabled for specific tables. There's sometimes helper tables in documents that would be confusing if opened in a card.

Additional parameters for SELF_HYPERLINK() is still useful. The feature I described above addresses the most common pattern and makes it easier, but cannot address all patterns/desires.

@anaisconce I love the sound of that ! I think that completely covers my need (and then some!)

Additional parameters for SELF_HYPERLINK() is still useful.
I can do this sometime soon :)


Added Lock Stale Issues flag

Fixes https://github.com/probot/stale/issues/122
This is only half of the implementation as this probably causes issues when "unmark"-ing stale issues as locked issues are filtered out of the results returned by the API.
Not sure what the correct way forward is to fix that as I'm not entirely sure why locked issues are filtered in the first place...
Thoughts welcome 👍

Once it is clear what to do with the res.data.items.filter(issue => !issue.locked) used on getStale and getClosable results I'll add an actual functional test around the lockStale flag...

Actually now that I'm thinking about it maybe the config should be lockInsteadOfClose of something along those lines...
This way the res.data.items.filter(issue => !issue.locked) bit is no longer a problem...
I'm not sure what is preferred in this case (maybe both are useful?), thoughts anyone?

I closed the original issue for this with a link to https://probot.github.io/apps/lock/ which is another github app for locking threads. These two things can be done in conjunction with one another, so I don't see a need for this additional functionality in stale.


naproche: fix build with GHC 9

Description of changes
Currently this prevents users from installing isabelle
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Done

A couple points:

In general, Haskell packages (even packages with only an exectuable) should be uploaded to Hackage.  That way, Haskellers can get them through their normal channels (like cabal, stack, Nixpkgs, haskell.nix, other Linux distros, etc).  Uploading your Haskell programs to Hackage means that you mostly don't have to do any work to keep it up-to-date in Nixpkgs.  It gets automatically updated to the latest release on Hackage.  If you're not the maintainer, I'd highly recommend creating an issue upstream urging the maintainer to release to Hackage.  (Although it appears that it wouldn't have helped in this case, since you're using an un-released version.  But in general it still holds true.)

Is there somewhere I need to register naproche so it will get caught here in the next round of updates

You can add naproche to the list in https://github.com/NixOS/nixpkgs/blob/0bbb65673c0ba31047c9ba6c4cd211556b534a4e/pkgs/top-level/release-haskell.nix#L140.  This makes sure that naproche will appear in the build reports for the haskell-updates PRs.  Here's an example build report:
https://github.com/NixOS/nixpkgs/pull/168049#issuecomment-1102643457

It is generally easier for us to have Haskell-related PRs targetting the haskell-updates branch instead of master.  In the future, you can also feel free to ping @-NixOS/haskell to get your Haskell-related PRs reviewed/merged.

Thanks.

Yeah, I am not the maintainer, and I don't think they will upload it to hackage any time soon. I just need it for isabelle
Thanks, added it now there
ok, i will target haskell-updates in the future

Thanks for the quick response!

Thanks for the quick reply :)


Add HTML5 player for testing

Need a HTML5 player like JW player for testing and demo

Mayank- I assinged to you to find out if we have JW player license to use here or not


Use one build in the appveyor matrix (to improve CI test time)

This changes appveyor to use just one variation in the build matrix, which should save 20min off overall build time (since appveyor is using one serialized build queue)

There is a 60min build limit per job.
Build Matrix make it possible to not fail build if take a bit longer than usual also

This one wasn't for merging (since it hit the 1 hour limit) but it is fixed by #948


[Feature] (documentation) Indicate which version of Create is supported

Describe the solution you'd like
Including which specific version(s) of Create that is currently supported to the mod description would be very helpful. ;)

Yes I will do that


Update hash of linux gstremer deps.

The uploaded file was updated since the PR merged.

This change is

r=me, but let's wait for Travis to be green.

@bors-servo r+

:pushpin: Commit b8e6963 has been approved by ferjm

:hourglass: Testing commit b8e69636424354fcecb385a9980f34b2a1fe31cf with merge 68b89f2729ef0a0e9f5fdc4bed7fdabf2a3e3b9b...

:sunny: Test successful - status-travis
Approved by: ferjm
Pushing 68b89f2729ef0a0e9f5fdc4bed7fdabf2a3e3b9b to master...


A11y: removeSelection button has no text.

Reproduction
https://formkit.link/e8d5b9df550843dd62e3b1a7604ccf9c
Describe the bug
The autocomplete features a reset button (optionally). There are two accessibility issues, of which one is serious:

If you point your screenreader cursor to this button, you will hear no explanation whatsoever. There's no text at all. Now people might argue "no problem, its tabindex is set to "-1", so no keyboard user will focus that button anyway. But that ignores that a screen reader user is not only tabbing, but can move to any "aria visible" element. I also think it would be wrong to remove that element from the accessibility tree by setting aria-hidden to true, because we show that button to make it clear to the user how they can reset the value, and if we feel that's necessary, it should also be necessary for the non-sighted user. Long story short: this button needs a label text (I personally would put it into the title attribute, so sighted users profit as well).
This is less critical, but from a screenreader user point of view, you could argue that the reset button is hard to get to. You can't tab to it, which you expect from buttons. Personally I think that's a good compromise, as it makes navigating faster and less interruptive. But an accessibility audit might point that out as an issue.

Environment
• OS: Mac
• Browser: Chrome
• Version 121

@DEPT-ch Hey there! I've gone ahead with your suggestion and added a title attribute to the resetSelection section of the autocomplete. This has been addressed at v0.121.5 of FormKitPro (you should see this reflected in your playground example if you hard reload, clear cache and make sure you are pulling down the latest version of Pro). I've also went ahead and set the removeSelection tabindex back to 0.
While we plan to have our own formal accessibility audit at some point of all of our FormKit Pro inputs (hopefully in the near future), we really appreciate getting feedback. Thank you for taking the time to let us know.

Fantastic, thanks for the quick fix!


custom lb policy support

(This is used to request new product features, please visit https://github.com/istio/istio/discussions for questions on using Istio)
Describe the feature request
The envoy has provided an extension point to lb policy to let developers to develop their own lb algorithm. But the DR cannot support to configure the custom lb policy now.
Describe alternatives you've considered
EnvoyFilter do everything.
Affected product area (please put an X in all that apply)
[ ] Ambient
[] Docs
[ ] Dual Stack
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Additional context

How could a DR provide a custom LB? i would think you need a custom build for envoy to do so, which puts you in "use EnvoyFilter" category IMO

How could a DR provide a custom LB?

I am thinking could the DR provide an extendable field with type TypedExtensionConfig in the xds/envoy-api (or similar things), then we can use any lb policy the data-plane provided, subset lb, client-weighted-round-robin, or any new policies in the future.

which puts you in "use EnvoyFilter" category IMO

EnvoyFilter always works fine. But I think it would be more friendly to configure the lb in the DR directly.

I don't think Istio APIs should support arbitrary extension points of unknown types. The APIs exist to form an abstraction oflver Istio behaviors, not as a passthrough to opaque envoy fields (which is the entire purpose of EnvoyFilter)

A lb policy is either...

not in envoy and in a fork: non goal for Istio to cover (but envoy filter can still cover it)
in envoy and useful for everyone: worth being first class
in envoy but not that useful: the exact intent of envoy filter

Okay, although in practices there are too much features that require to use EnvoyFilter.

Any undertermined config type should be discouraged from istio API,it's hard for istio to gurantee the correctness.  Turn to envoyfilter.
In your case, customized lb extension should be configurred easily with your own controlplane.


reset password stops postgres/solr containers?

I started my site, then used the password reset and the db and solr containers went away. Might we be triggering an old datacats bug?

  File "/home/ian/boxkite/git/ckan-multisite/ckan_multisite/api.py", line 40, in wrapper
    return func(*f_args)
  File "/home/ian/boxkite/git/ckan-multisite/ckan_multisite/api.py", line 61, in _check
    return func(*f_args)
  File "/home/ian/boxkite/git/ckan-multisite/ckan_multisite/api.py", line 96, in wrapper
    return func(environment)
  File "/home/ian/boxkite/git/ckan-multisite/ckan_multisite/api.py", line 160, in change_admin_pw
    environment.create_admin_or_set_password(request.form['password'])
AttributeError: 'Environment' object has no attribute 'create_admin_or_set_password'

70aaf54 fixes this

fixes the traceback but still losing db/solr containers

843ed4f32ec6b5d529574cdfac1918ce5e251779 should fix the db/solr issue

looks good


Suggest

Is it possible you ad a folder with precompiled images for each port like (https://github.com/loboris/MicroPython_ESP32_psRAM_LoBo) did. It would be easier for those who can not build an image.
Thanks for your answer in advance.

Who are these people who "can not build an image"? Why can't they build an image?
Anyone can, it just requires reading some documentation and putting a little effort.
And when someone like that, who "can't build an image", tries a precompiled image as you suggest and runs into some problem - he certainly also can't debug the image or have a clue where the problem could be, so he will come to us for help.
I certainly don't have the capacity to maintain and test multiple precompiled images and then support clueless users who try to use them.
The number of precompiled images is large. Each port would probably need several images. For example on ESP32 we have a version with spi-ram and without spi-ram, a version with color depth of 32 and a version with 16, etc.
That said, a possible limited approach could be to rely on CI artifacts.
Today we have RP2 image and STM32 image artifacts that are automatically generated every time the repo is updated (but not really tested). It's possible to create artifacts for other ports as well.
You can also create a new CI task to collect all these artifacts and present them nicely on a web page as you suggest.
If you would like to contribute by implementing that - feel free to open a Pull Request!


Async loading orchestration

Need a way to do something like this:
    Texture : ILoadable
    AudioTrack : ILoadable

    ...

    class OsuLogo
    {
        override Load() {
            Texture t1, t2, t3;
            AudioTrack sfx1;
            Game.Async.Load(delegate
            {
                t1 = Game.Textures.Load(@"Menu/osu");
                t2 = Game.Textures.Load(@"Menu/osuRipple");
                t3 = Game.Textures.Load(@"Menu/osuOverlay");
                sfx = Game.Audio.Track.Load(@"Menu/heartbeat");
            }, delegate {
                //On load completion...
                OsuLogo.Texture = t1;
                OsuLogoRipple.Texture = t2;
                OsuLogoOverlay.Texture = t3;

                startBeat(sfx1);
                FadeIn(500);
            });
        }
    }

Open to feedback on the semantics in this sample.

Considering:

async/await pattern
design of ILoadable

Will comment later if idea got.

async/await will be hard to use in the scenario i believe. but you're welcome to try.

class Loadable<T>
{
    Loadable(Task<T> loadTask) { ... }
    Loadable(Func<T> loadDelegate) { ... }
    bool IsCompleted { get { ... } }
    static implicit operator T(Loadable<T> value) { ... }
}

...

Loadable<Texture> t1 = new Loadable<Texture>(Game.Textures.LoadAsync(@"Menu/osu"));
Loadable<Texture> t2 = new Loadable<Texture>(() => Game.Textures.Load(@"Menu/osuRipple"));

This pattern needs no change of the resource class, but some changes of consumer class.
Considering another pattern.

public override async void Load()
{
    Task<Texture> t1, t2, t3;
    Task<AudioTrack> sfx1;
    t1 = Game.Textures.LoadAsync(@"Menu/osu");
    t2 = Game.Textures.LoadAsync(@"Menu/osuRipple");
    t3 = Game.Textures.LoadAsync(@"Menu/osuRipple");
    sfx1 = Game.Textures.LoadAsync(@"Menu/heartbeat");

    //if no same-time requirement
    OsuLogo.Texture = await t1;
    OsuLogoRipple.Texture = await t2;
    OsuLogoOverlay.Texture = await t3;
    startBeat(await sfx1);

    //if same-time required
    Task loadTask = Task.WhenAll(t1, t2, t3, sfx1);//pass if need to trace
    await loadTask;
    OsuLogo.Texture = t1.Result;
    OsuLogoRipple.Texture = t2.Result;
    OsuLogoOverlay.Texture = t3.Result;
    startBeat(sfx1.Result);
}

if there are several group of things to do, can make each into separated LoadXXX().

yeah don't see that usage of async/await being beneficial at all. we need to be able to background schedule, not just delay stuff inside Load

so difficult to make changes to current Scheduler model
maybe ILoadable is a better choice

:+1: to @huoyaoyuan's proposal

Could be simplified though
public override async void Load()
{
    Texture t1 = await Game.Textures.LoadAsync(@"Menu/osu");
    Texture t2 = await Game.Textures.LoadAsync(@"Menu/osu");
    Texture t3 = await Game.Textures.LoadAsync(@"Menu/osu");
    AudioTrack sfx1 = await Game.Textures.LoadAsync(@"Menu/heartbeat");
    startBeat(sfx1);
}

that code is pointless. the idea is that the completion action happens at a much later time (possibly seconds); not in that load method itself

Well, by using the Task<> we can still get a simliar result to what you're looking for, but we have the flexibility to also do something like this.

As ppy said, what we need is not just delay loading, but custom scheduling. async/await uses the default ThreadPool, but we are using our custom task scheduler.
To integrate with current scheduler model, my idea is make a "task with preconditions" class.

A new semantic idea:
Usage:
override Load()
{
    ScheduledDelegate<Texture> t1, t2, t3;
    ScheduledDelegate<AudioTrack> sfx1;
    t1 = Game.Textures.LoadBackground(@"Menu/osu");
    t2 = Game.Textures.LoadBackground(@"Menu/osuRipple");
    t3 = Game.Textures.LoadBackground(@"Menu/osuOverlay");
    sfx1 = Game.Audio.Track.LoadBackground(@"Menu/heartbeat");
    Game.BackgroundWorker.Work(() => //may be another name
    {
        OsuLogo.Texture = t1.Result;
        OsuLogoRipple.Texture = t2.Result;
        OsuLogoOverlay.Texture = t3.Result;
        startBeat(sfx1.Result);
        this.FadeIn(500);
        //and other works
    },//other options
    , t1, t2, t3, sfx1);
}

Prototypes:
ScheduledDelegate<TResult> : ScheduledDelegate
LoadBackground: returns a not started, not scheduled ScheduledDelegate<TResult>
BackgroundWorker.Work(Action completion, /*other options*/, params ScheduledDelegate[] preconditions)
The Work will receive parameters and tasks, and schedule itself.

peppy's code is not bad, but from my point of view, it can be improved:
//On load completion...
OsuLogo.Texture = t1;
OsuLogoRipple.Texture = t2;
OsuLogoOverlay.Texture = t3;

As a consumer, I don't want to write continuation code just for variables assignment. I think it should be done inside of TextureStore or elsewhere.
t1 = Game.Textures.Load(@"Menu/osu");
t2 = Game.Textures.Load(@"Menu/osuRipple");
t3 = Game.Textures.Load(@"Menu/osuOverlay");
sfx = Game.Audio.Track.Load(@"Menu/heartbeat");

As far as I understand, these resources are loaded sequentially using 1 thread. I think we should at least try to load them in parallel.

psoposed solution @ https://github.com/ppy/osu-framework/pull/253

Got async loading now.