RPI CM3 64 Bits

Description
I am trying to write a configuration file for the cm3 64 bits, but I keep getting errors regarding KERNEL_DEVICETREE. I am trying to make the machine similar to the raspberrypi3-64 and add the KERNEL_DEVICETREE for the CM, but from what I understand, they are not shipped.
During compilation, it fails, not finding some dtb.It seem KERNEL_DEVICETREE points to some, but those are not generated. If not linked to KERNEL_DEVICETREE paramater, how can I ship the right dtb?
Here is my conf file:
#@TYPE: Machine
#@NAME: RaspberryPi Compute Module 3 (CM3-64)
#@DESCRIPTION: Machine configuration for the RaspberryPi Compute Module 3 (CM3) in 64 bits mode

MACHINEOVERRIDES = "raspberrypi3:${MACHINE}"

require conf/machine/include/arm/arch-armv8.inc
include conf/machine/include/rpi-base.inc

KERNEL_DEVICETREE = " \
    broadcom/bcm2710-rpi-3-b.dtb \
    broadcom/bcm2837-rpi-3-b.dtb \
    broadcom/bcm2708-rpi-cm.dtb \
    broadcom/bcm2710-rpi-cm3.dtb \
    \
    overlays/hifiberry-amp.dtbo \
    overlays/hifiberry-dac.dtbo \
    overlays/hifiberry-dacplus.dtbo \
    overlays/hifiberry-digi.dtbo \
    overlays/i2c-rtc.dtbo \
    overlays/iqaudio-dac.dtbo \
    overlays/iqaudio-dacplus.dtbo \
    overlays/lirc-rpi.dtbo \
    overlays/pitft22.dtbo \
    overlays/pitft28-resistive.dtbo \
    overlays/pitft35-resistive.dtbo \
    overlays/pps-gpio.dtbo \
    overlays/rpi-ft5406.dtbo \
    overlays/w1-gpio.dtbo \
    overlays/w1-gpio-pullup.dtbo \
    overlays/pi3-disable-bt.dtbo \
    overlays/pi3-miniuart-bt.dtbo \
    overlays/vc4-kms-v3d.dtbo \
    "

SERIAL_CONSOLE ?= "115200 ttyS0"
VC4_CMA_SIZE ?= "cma-256"

UBOOT_MACHINE = "rpi_3_config"
MACHINE_FEATURES_append = " vc4graphics"

I am conscious this is not an issue itself, but once I understand how it works, I can make a PR here.

Steps to reproduce the issue:

Add the upper configuration file in meta-raspberrypi/conf/machine
Add KERNEL_DEFCONFIG_raspberrypi-cm3-64 ?= "bcmrpi3_defconfig" in recipes-kernel/linux/linux-raspberrypi.inc
bitbake linux-raspberrypi

Describe the results you received:
|   DTC     arch/arm64/boot/dts/broadcom/bcm2837-rpi-3-b.dtb
| make[3]: *** No rule to make target `arch/arm64/boot/dts/broadcom/bcm2708-rpi-cm.dtb'.  Stop.
| make[2]: *** [broadcom/bcm2708-rpi-cm.dtb] Error 2
| make[1]: *** [sub-make] Error 2
| make: *** [__sub-make] Error 2

Describe the results you expected:
Compilation done with dtb from CM.
Additional information you deem important (e.g. issue happens only occasionally):
Additional details (revisions used, host distro, etc.):

Looking in the linux-raspberrypi kernel sources, there isn't any *-cm.dtb files in https://github.com/raspberrypi/linux/tree/rpi-4.9.y/arch/arm64/boot/dts/broadcom.
I'd recommend asking on the linux-raspberrypi repository what the correct dts file is to for the RaspberryPi CM3 to be used in 64-bit mode.

You are right, that's where I mostly working. It's a WIP on the kernel and will update you asap. I'd like to merge it here once it's done

Ok that's good news, it'd be nice to have the CM3 working with a 64-bit operating system.
Do you have a kernel branch that does have the dts files you need?

Yes, here. For now, it is more experiments since it's the first time I work with this.

Ok. For development you can modify recipes-kernel/linux/linux-raspberrypi_4.9.bb in the meta-raspberrypi layer, change SRC_URI to point to your kernel repository and branch. That should allow you to test your changes. Once they're working and have been merged in to the upstream linux-raspberrypi repository you can send us a pull request to add the raspberrypi-cm3-64 MACHINE configuration.

Yes, thanks for the informations. I'm really new to the kernel side, but am used to Yocto. This is pretty interesting, but I have to say, it lacks documentation, especially on some dts files. One needs to dig to understand why it's needed. e.g, why is the bcm2837 needed for the 64 bit rpi3.

Here as an update. I manage to boot, the kernel log seems really promising. My only issue is it seems device tree overlays are ignored, for a reason I ignore. Because they are ignored, I cannot use pi3-disable-bt enabling to log in. Hence the message console [ttyS0] enabled is not displayed
I think dtoverlays are ignored since adding overlay to use uart1 doesn't change any behavior.
The setup:

rpi-basic-image
raspberrypi-cm3-64.conf
kernel 4.9.39. I also tested 4.8.16 and have same results
I don't use KERNEL_IMAGETYPE=uImage. U-boot lacks the config file. I wrote it, but I got stuck trying to edit Makefiles in the kernel source to create the uImage. From what I read, there's a way to list device tree list in uboot. It would be helpful.
cmdline.txt: dwc_otg.lpm_enable=0 console=tty1 console=serial0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait cma=128M@64M
config.txt:

dtparam=spi=on
dtparam=i2c=on
dtparam=i2c_arm=on
enable_uart=1
dtoverlay=pi3-disable-bt
dtoverlay=vc4-kms-v3d
gpu_mem=16
device_tree=bcm2837-rpi-cm3.dtb (WIP)
core_freq=250

Here is the complete log:
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.39 (dbensoussan@ip-whatever) (gcc version 5.4.0 (GCC) ) #1 SMP PREEMPT Thu Aug 3 15:28:23 UTC 2017
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 128 MiB at 0x0000000004000000
[    0.000000] percpu: Embedded 22 pages/cpu @fffffff9fdf8c000 s52504 r8192 d29416 u90112
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 845719
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 249984
[    0.000000] Kernel command line: 8250.nr_uarts=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ea00000 vc_mem.mem_size=0x3f000000  dwc_otg.lpm_enable=0 console=tty1 console=ttyAMA0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait cma=128M@64M drm.debug=0x1f
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Memory: 852612K/1015808K available (6780K kernel code, 754K rwdata, 2548K rodata, 2688K init, 828K bss, 32124K reserved, 131072K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffffff8000000000 - 0xffffff8008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffffff8008000000 - 0xffffffbebfff0000   (   250 GB)
[    0.000000]       .text : 0xffffff9a55480000 - 0xffffff9a55b20000   (  6784 KB)
[    0.000000]     .rodata : 0xffffff9a55b20000 - 0xffffff9a55db0000   (  2624 KB)
[    0.000000]       .init : 0xffffff9a55db0000 - 0xffffff9a56050000   (  2688 KB)
[    0.000000]       .data : 0xffffff9a56050000 - 0xffffff9a5610ca00   (   755 KB)
[    0.000000]        .bss : 0xffffff9a5610ca00 - 0xffffff9a561dba94   (   829 KB)
[    0.000000]     fixed   : 0xffffffbefe7fd000 - 0xffffffbefec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffffffbefee00000 - 0xffffffbeffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffffffbf00000000 - 0xffffffc000000000   (     4 GB maximum)
[    0.000000]               0xffffffbfe7000000 - 0xffffffbfe7f80000   (    15 MB actual)
[    0.000000]     memory  : 0xfffffff9c0000000 - 0xfffffff9fe000000   (   992 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] Failed to get local register map. FIQ is disabled for cpus > 1
[    0.000000] arm_arch_timer: Architected cp15 timer(s) running at 19.20MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns:<PHONE_NUMBER>67 ns
[    0.000006] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every<PHONE_NUMBER>078ns
[    0.000186] Console: colour dummy device 80x25
[    0.001397] console [tty1] enabled
[    0.001443] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=19200)
[    0.001507] pid_max: default: 32768 minimum: 301
[    0.001842] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.001883] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.002833] Disabling cpuset control group subsystem
[    0.003018] ftrace: allocating 23498 entries in 92 pages
[    0.064943] ASID allocator initialised with 65536 entries
[    0.072885] EFI services will not be available.
[    0.081879] Detected VIPT I-cache on CPU1
[    0.081942] CPU1: Booted secondary processor [410fd034]
[    0.088988] Detected VIPT I-cache on CPU2
[    0.089032] CPU2: Booted secondary processor [410fd034]
[    0.096108] Detected VIPT I-cache on CPU3
[    0.096149] CPU3: Booted secondary processor [410fd034]
[    0.096254] Brought up 4 CPUs
[    0.096447] SMP: Total of 4 processors activated.
[    0.096488] CPU features: detected feature: 32-bit EL0 Support
[    0.096591] CPU: All CPU(s) started at EL2
[    0.096652] alternatives: patching kernel code
[    0.097888] devtmpfs: initialized
[    0.112872] DMI not present or invalid.
[    0.113258] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns:<PHONE_NUMBER>275000 ns
[    0.113353] futex hash table entries: 1024 (order: 5, 131072 bytes)
[    0.119398] pinctrl core: initialized pinctrl subsystem
[    0.120318] NET: Registered protocol family 16
[    0.132696] cpuidle: using governor menu
[    0.133119] vdso: 2 pages (1 code @ ffffff9a55b27000, 1 data @ ffffff9a56054000)
[    0.133186] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.135360] DMA: preallocated 256 KiB pool for atomic allocations
[    0.135515] Serial: AMBA PL011 UART driver
[    0.139366] bcm2835-mbox 3f00b880.mailbox: mailbox enabled
[    0.199242] bcm2835-dma 3f007000.dma: DMA legacy API manager at ffffff800804b000, dmachans=0x1
[    0.201861] SCSI subsystem initialized
[    0.202106] usbcore: registered new interface driver usbfs
[    0.202238] usbcore: registered new interface driver hub
[    0.202419] usbcore: registered new device driver usb
[    0.202761] dmi: Firmware registration failed.
[    0.203702] raspberrypi-firmware soc:firmware: Attached to firmware from 2017-04-05 11:51
[    0.205456] clocksource: Switched to clocksource arch_sys_counter
[    0.277578] VFS: Disk quotas dquot_6.6.0
[    0.277715] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.278024] FS-Cache: Loaded
[    0.278344] CacheFiles: Loaded
[    0.293700] NET: Registered protocol family 2
[    0.294779] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[    0.294960] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[    0.295238] TCP: Hash tables configured (established 8192 bind 8192)
[    0.295439] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    0.295545] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    0.295899] NET: Registered protocol family 1
[    0.296514] RPC: Registered named UNIX socket transport module.
[    0.296551] RPC: Registered udp transport module.
[    0.296582] RPC: Registered tcp transport module.
[    0.296613] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.300305] workingset: timestamp_bits=46 max_order=18 bucket_order=0
[    0.323323] FS-Cache: Netfs 'nfs' registered for caching
[    0.324593] NFS: Registering the id_resolver key type
[    0.324672] Key type id_resolver registered
[    0.324708] Key type id_legacy registered
[    0.327803] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.328038] io scheduler noop registered
[    0.328077] io scheduler deadline registered
[    0.328161] io scheduler cfq registered (default)
[    0.431527] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[    0.433682] bcm2835-rng 3f104000.rng: hwrng registered
[    0.433934] [drm] Initialized
[    0.434205] Unable to detect cache hierarchy from DT for CPU 0
[    0.454385] brd: module loaded
[    0.465704] loop: module loaded
[    0.465753] Loading iSCSI transport class v2.0-870.
[    0.466547] usbcore: registered new interface driver smsc95xx
[    0.466599] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[    0.493122] dwc2 3f980000.usb: Configuration mismatch. dr_mode forced to host
[    0.544090] dwc2 3f980000.usb: DWC OTG Controller
[    0.544160] dwc2 3f980000.usb: new USB bus registered, assigned bus number 1
[    0.544240] dwc2 3f980000.usb: irq 41, io mem 0x00000000
[    0.544599] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    0.544646] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.544702] usb usb1: Product: DWC OTG Controller
[    0.544741] usb usb1: Manufacturer: Linux 4.9.39 dwc2_hsotg
[    0.544781] usb usb1: SerialNumber: 3f980000.usb
[    0.545792] hub 1-0:1.0: USB hub found
[    0.545869] hub 1-0:1.0: 1 port detected
[    0.546836] usbcore: registered new interface driver usb-storage
[    0.547183] mousedev: PS/2 mouse device common for all mice
[    0.548835] bcm2835-wdt 3f100000.watchdog: Broadcom BCM2835 watchdog timer
[    0.549247] bcm2835-cpufreq: min=600000 max=1275000
[    0.549870] sdhci: Secure Digital Host Controller Interface driver
[    0.549911] sdhci: Copyright(c) Pierre Ossman
[    0.552205] sdhost-bcm2835 3f202000.sdhost: could not get clk, deferring probe
[    0.552546] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.553213] ledtrig-cpu: registered to indicate activity on CPUs
[    0.553362] hidraw: raw HID events driver (C) Jiri Kosina
[    0.553632] usbcore: registered new interface driver usbhid
[    0.553671] usbhid: USB HID core driver
[    0.553960] Initializing XFRM netlink socket
[    0.554023] NET: Registered protocol family 17
[    0.554211] Key type dns_resolver registered
[    0.554688] Registered cp15_barrier emulation handler
[    0.554739] Registered setend emulation handler
[    0.555759] registered taskstats version 1
[    0.564587] 3f201000.serial: ttyAMA0 at MMIO 0x3f201000 (irq = 72, base_baud = 0) is a PL011 rev2
[    1.387549] console [ttyAMA0] enabled
[    1.396723] sdhost: log_buf @ ffffff8008111000 (c4041000)
[    1.446492] mmc0: sdhost-bcm2835 loaded - DMA enabled (>1)
[    1.452215] of_cfs_init
[    1.455009] of_cfs_init: OK
[    1.459082] uart-pl011 3f201000.serial: no DMA platform data
[    1.465158] Waiting for root device /dev/mmcblk0p2...
[    1.632228] mmc0: new high speed MMC card at address 0001
[    1.638745] mmcblk0: mmc0:0001 4FPD3R 3.64 GiB
[    1.643813] mmcblk0boot0: mmc0:0001 4FPD3R partition 1 4.00 MiB
[    1.650321] mmcblk0boot1: mmc0:0001 4FPD3R partition 2 4.00 MiB
[    1.656808] mmcblk0rpmb: mmc0:0001 4FPD3R partition 3 512 KiB
[    1.664648]  mmcblk0: p1 p2
[    1.685777] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[    1.694017] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[    1.701940] devtmpfs: mounted
[    1.708665] Freeing unused kernel memory: 2688K (fffffff9c09b0000 - fffffff9c0c50000)
INIT: version 2.88 booting
Error opening /dev/fb0: No such file or directory
Starting udev
[    2.178478] udevd[130]: starting version 3.2
[    2.206533] random: udevd: uninitialized urandom read (16 bytes read)
[    2.214623] random: udevd: uninitialized urandom read (16 bytes read)
[    2.221195] random: udevd: uninitialized urandom read (16 bytes read)
[    2.227789] random: udevd: uninitialized urandom read (16 bytes read)
[    2.265043] udevd[131]: starting eudev-3.2
[    2.398370] random: udevd: uninitialized urandom read (16 bytes read)
[    2.428769] random: udevd: uninitialized urandom read (16 bytes read)
[    2.658507] vc4_hdmi 3f902000.hdmi: vc4-hdmi-hifi <-> 3f902000.hdmi mapping ok
[    2.665850] random: fast init done
[    2.678638] vc4-drm soc:gpu: bound 3f902000.hdmi (ops vc4_hdmi_ops [vc4])
[    2.686677] vc4-drm soc:gpu: bound 3f806000.vec (ops vc4_vec_ops [vc4])
[    2.693754] vc4-drm soc:gpu: bound 3f400000.hvs (ops vc4_hvs_ops [vc4])
[    2.701010] vc4-drm soc:gpu: bound 3f206000.pixelvalve (ops vc4_crtc_ops [vc4])
[    2.709097] vc4-drm soc:gpu: bound 3f207000.pixelvalve (ops vc4_crtc_ops [vc4])
[    2.717115] vc4-drm soc:gpu: bound 3f807000.pixelvalve (ops vc4_crtc_ops [vc4])
[    2.755979] vc4-drm soc:gpu: bound 3fc00000.v3d (ops vc4_v3d_ops [vc4])
[    2.765084] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    2.771824] [drm] Driver supports precise vblank timestamp query.
[    2.809488] Console: switching to colour frame buffer device 90x30
[    2.829525] vc4-drm soc:gpu: fb0:  frame buffer device
[    2.923954] i2c /dev entries driver
[    2.987411] EXT4-fs (mmcblk0p2): re-mounted. Opts: data=ordered
Populating dev cache
Fri Aug  4 11:38:07 UTC 2017
INIT: Entering runlevel: 5
Configuring network interfaces... ifconfig: SIOCGIFFLAGS: No such device
ifconfig: SIOCSIFADDR: No such device
ifconfig: SIOCGIFFLAGS: No such device
ifconfig: SIOCSIFADDR: No such device
Starting Dropbear SSH server: Generating key, this may take a while...
Public key portion is:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvOlVjfx+9TKwtsIbWJ8fBcdyt+4aqNtAEFQWkKu6Ccz62N44xgO7Ha5hNZ/btUOPM6ydCPwhAo9Eg4UlUPFRkJ9XkcCiNQ0Dv2FTMMQWgvcsSOmSY1RO4Ho1GUY36IWvW2lcKJ7JT5zopjBncHdj0QPWMqUXw7YoLAOsn4GFRvR3D7yAMs+Kjv6cObOPonlwpuELYhG2nhHSJBD65V6u1zj5mwAJ48V8RPAEuUqLobKyw2rsVUNm7QN8eK5xDBsmbG6Ed94DxnzVPE7g16SjHtx+M8EjImPQoyckQI+IAEg7A+TM0VLfuM7P00a6U8Pxati3HK2MKE9nc53BzBgGl root@raspberrypi-cm3-64
Fingerprint: md5 8f:6f:2c:1c:f1:8b:9e:dd:72:42:d5:5a:4c:b8:b1:62
[    9.269227] NET: Registered protocol family 10
dropbear.
Starting syslogd/klogd: done
INIT: Id "S0" respawning too fast: disabled for 5 minutes

Why don't you use the default serial console? And figure it out from there.

Hey @agherzan, I worked on it since last update, but I was not working for almost 3 weeks or so. I'll update the status in the next week

I am almost done but I still cannot make have the HDMI working. I found out vc4 driver was not loaded. I then tried with a raspberrypi3-64 (not compute module) machine but no success, I have exactly the same issue, videocore is not loaded.I'm using the minimum layers for creating the machine and here is my setup.

Config.txt:
#Enable VC4 Graphics
dtoverlay=vc4-kms-v3d
gpu_mem=16
device_tree=bcm2710-rpi-3-b.dtb
core_freq=500
sdram_freq=600
arm_freq=1150
enable_uart=1

cmdline.txt:
net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait cma=8M

This is my local.conf:
MACHINE ?= "raspberrypi3-64"
DISTRO ?= "poky"

EXTRA_IMAGE_FEATURES = " debug-tweaks package-management tools-profile "
BB_NUMBER_THREADS ?= "16"
PARALLEL_MAKE ?= "-j 16"

I am using meta-raspberrypi@8527a396d38aec7776182caca8b9a237e03c3965 rpi-basic-image

This should work. I should have the videocore loaded at least during boottime, which I don't. Is there anything I'm missing?

@Minipada - HDMI output not working is a known issue with raspberrypi3-64, it's now in the documentation. We're looking at get it fixed though. Khem's latest patches may help here and these have been merged to master - could you re-try with the latest master? If that doesn't work, I've got another possible solution in progress as well which we could try.

@betafive Thanks for the hints, I'll try after the current tests. I managed to use the screen using the fkms, I get a warning but this is enough to debug the dts properly. I have made some progress after understanding the device tree wiki and answers from the pelwell, the rpi DT master.
Thanks to the screen (I still don't have ttyS0 access), I see the firmware loads by default the device tree starting with the trailer bcm27, so I have modified my config.txt to force the new bcm2837.
I discovered it today, some parts of the bcm27 dts are not loaded on the CM3 - 64 bits. I saw this reading the device tree in /proc/device-tree.
I will push my changes tomorrow morning and will post here where to get them.

Please excuse me @betafive for the very late answer. Here is the WIP with a description of the current status.

@Minipada, are you planning to upstream this to the linux-raspberrypi kernel or carry this as a patch in meta-raspberrypi? HDMI output in 64-bit mode should definitely be working now so it would be good to get support for raspberrypi-cm3-64 as a MACHINE if possible.

Hello @betafive I have actually made it work and updated my PR on the raspberryi/linux repository but got no answer. I am going to push the tests in some days, the OS will be used by the team and I will have more assurance of the quality of the work done
Here is the link: https://github.com/raspberrypi/linux/pull/2147

Ok, let's see how it goes upstream. Give them a reminder if needed.

pelwell is usually quite responsive so I think he will reply soon.

@Minipada, looks like upstream haven't responded :( Could you rebase and try submitting it again? If upstream is still unresponsive we could carry the patches here for now.

@Minipada is this still needed ? if so then the upstream pull branch needs refresh

@Minipada Friendly ping on this issue.

@agherzan thanks and sorry I didn't answer. I'll take a look at it tomorrow

The PR on the kernel repo looks more like a draft, I got requested some changes and am working on them and doing some cleanup.

@Minipada More people are asking about this. No pressure :)

I updated my pull request on the linux repository and have a PR ready for sumo as soon as they review and merge it (code here)

Cheers @Minipanda.

My PR is still without answer since some time, but I just saw this got merged https://github.com/raspberrypi/linux/commit/248e0f711ec517aff69b815bac7b884bd3a04679
Is it hope?

Issue 304 is related to this issue.
https://github.com/agherzan/meta-raspberrypi/issues/304
As you have discovered,  the dts file now exists in the raspberrypi linux repo, and the Makefile has also been fixed up.
Once the meta-raspberry recipes can leverage this change to raspberrypi linux, then the meta-raspberrypi kernel device tree could be updated to include the cm3.

Issue 304 is related to this issue.
https://github.com/agherzan/meta-raspberrypi/issues/304
As you have discovered,  the dts file now exists in the raspberrypi linux repo, and the Makefile has also been fixed up.
Once the meta-raspberry recipes can leverage this change to raspberrypi linux, then the meta-raspberrypi kernel device tree could be updated to include the cm3.

For now, if you wish to have a functional CM3 dtb built and deployed to the boot folder for your raspberrypi3-64 target, simply add the following to your local.conf:
KERNEL_DEVICETREE_append = "broadcom/bcm2710-rpi-cm3.dtb"
It's working well for me at head of sumo, where the linux-raspberry pi kernel recipe fetches a recent enough version where the cm3 dts is present, so the build of the dtb succeeds now.

I'll try asap, and close this.

Note, I am building the 4.14 flavor in meta-raspberrypi (4.14.68), sumo branch, and that worked.
I have not built for a 4.9 flavor in some time.

@stevep-korg I couldn't boot the image.
This is my local.conf:
MACHINE = "raspberrypi3-64"

EXTRA_IMAGE_FEATURES = " \
  allow-empty-password \
  debug-tweaks \
  empty-root-password \
  package-management \
  ssh-server-openssh \
"

BB_NUMBER_THREADS = "32"
PARALLEL_MAKE = "-j 32"

DL_DIR = "/home/build/yocto/yocto_topdir/downloads"
SSTATE_DIR = "/home/build/yocto/yocto_topdir/sstate-cache"
TMPDIR = "/home/build/yocto/yocto_topdir/tmp"

KERNEL_DEVICETREE_append = "broadcom/bcm2710-rpi-cm3.dtb"

The module is there. I have nothing either on the hdmi, nor on serial. I tried using rpi-basic-image and core-image-base (as I noticed the former indicated it was deprecated)
My layers:

poky master
meta-openembedded master
meta-raspberrypi sumo

OK great, glad you could boot it.
In terms of the UART communication stopping, we have not seen that.  We use the UART for MIDI.  We set the following in config.txt
configure uart0 for serial midi (/dev/ttyAMA0)
enable_uart=1
midi-uart0: fakes out tty driver so that baudrate 38400 yields 31250
dtoverlay=midi-uart0
BTW, I also put all repos in branch sumo that can be in sumo.  I find master to be too unstable for regular development purposes in my organization.
I build rpi-hwup-image.   I don't care right now that it's deprecated; it has what I need for now  My organization will probably update our recipes to continue to support the rpi-hwup-image. target, even if it is retired from meta-raspberry pi.  Also, beware of core-image-minimal if you need kernel modules; it doesn't package them into the image!

I confirm that everything works perfectly. Thanks @stevep-korg for the simple and efficient patch. I'm now closing it


Implement MNAIO post hook for saving VM images

In order to be able to re-use MNAIO VM images for other
projects to reduce test times when using the MNAIO build
we implement a change to the MNAIO tooling to use a file
backing store for the VM's, the post hook to prepare them
for uploading, and the component_metadata.yml file to
upload them to Cloud Files using the standard RE hook for
uploading file artifacts.
JIRA: RE-1992
(cherry picked from commit<PHONE_NUMBER>97d28b6964cabf5c451fa1c19109e8)
Issue: RE-1561

Thank you for submitting a pull request to this project!
Once all check jobs have passed and the pull request has received a
sufficient number of reviews, the repository's gate jobs can be triggered by
adding a comment of :shipit: to this pull request.
When the gate jobs have completed successfully, this pull request will get
merged automatically.

@dealako The RE_* env vars are set as part of standard jobs by jenkins. See Environment Variables in https://rpc-openstack.atlassian.net/wiki/spaces/RE/pages/457965626/Check+Job.
And yes, the mixed notation is a bit of a thing which can irk one. ;) This is a backport though, so please feel free to push a patch to the master branch to do some cleaning up if it irks you enough. :)

recheck_xenial_no_artifacts_swift_deploy

Testing done, please feel free to merge when you feel this should be.


path_provider getDownloadsDirectory documentation doesn't say when it might return null

The getDownloadsDirectory function is defined as potentially returning null. The documentation talks about it potentially throwing an UnsupportedError if this is not supported on the current platform, and talks about the returned Directory maybe not existing, but doesn't say when it might return null. So it's not clear if this is a failure mode that one should handle, or what to do when it happens.
https://pub.dev/documentation/path_provider/latest/path_provider/getDownloadsDirectory.html

The intent is that UnsupportedError is for platforms that don't implement this at all (e.g., a platform that has no concept of a download directory), while null would mean that the implementation understands the concept, but that there's no such directory available.
That distinction is probably not worth handling two different ways, especially since the null case should be very rare, so we could consider changing all the currently nullable methods to be non-nullable, and convert nulls from the platform interface to MissingPlatformDirectoryException (and then documenting that).

The docs also refer to the possibility of the path being valid but not existing, that's distinct from the case of the path not being available?

Yes; for instance on Linux you could configure your XDG settings to say your Download directory is ~/thisdirectoryhasneverexisted, and then path_provider would return that valid but non-existent directory.

What's an example of the path not being available? Something like Linux when no downloads directory has been configured at all?

Right now you could get it by running on a Linux without xdg-user-dir, or where that exists but fails when called.
In theory iOS/macOS could return an empty array from the search, but I would be pretty surprised if that's possible in practice.

Cool. Those details would be great to include in the docs. Thanks for the explanations!


Firefox- bashlike plugin error parsing as a table

The behavior of bash-like seems to not be consistent with Chrome, here is an example of one problem that I see when using a cli that routes through bash-like and adopts the same visualizations as kubectl:

The cli being used returns raw text as the output with the errors above, in Chrome the output is formatted in the Carbon styling.

what command did you issue here? thanks!

we probably should remove, for now, the use of the (awesome) flatMap. i do see that firefox 62 clais support for flatMap. you must be using an older version of firefox?
and the latest (chrome-based) Edge also supports flatMap. i think the only gaps are older Firefox, and the pre-chrome Edge.

@starpit
Looks like I'm on the Firefox Extended Support Release channel, so that is only on 60.8.0.  I think I can be agnostic on this one.  There might be government customers that don't allow for regularly updating their work approved browsers on their office computers, that might also apply to Edge.

yes, i think this one is important. it should be a straightforward matter either to use a flatMap polyfill npm, or just to edit the 3 or so places that call flatMap by hand.

@starpit in these cases where a JavaScript feature is not supported by a particular browser, I've fallen back on lodash.  I see that they have a flatMap function:  https://lodash.com/docs/4.17.15#flatMap


bug(poolmanager): define split routes messages on the message server

Background
The messages weren't added. Add them

merged


Adding network metrics to GCM sink.

Added TODO for the remaining two types: diskIO and FS. DiskIO we may
not need yet and FS requires more infrastructure.

Looks like the minions have a full disk :P free'd up some space and kicked off the build again. Still investigating...

LGTM! Let me know if you need help fixing the e2e test machine.


Cryptography module, integrity checking commands and small fixes

This PR adds a "cryptography" module in order to support more advanced security features, such as integrity checking, and later on, digitally Signed Packs. The implementation is based on the ongoing discussion for a Signed Packs feature, started in #8. Checksum support has also been requested before (#91).
Currently only the "sha256" digest is supported, which is the industry standard for most cases. The new commands are:

cpackget checksum-create takes a local .pack and creates a .checksum file, similar to the typical digest files (like .sha256). It contains a list of all the pack files and their digest, whose cryptographic hash function is provided in the filename (e.g: TheVendor.ThePack.1.2.3.sha256.checksum).
cpackget-verify takes that .checksum file and verifies the integrity of a local .pack, assuming it was generated with the other command.

It also includes small fixes regarding issues #85 and #93.

It looks good! I left a few nit-pick comments and one major concern about speed (regarding decompressing etc). Tests are also missing, but I talked to Luis earlier today and he's working on that.
Good work!

Cleaned up function descriptions a bit, but should be good to go.


Duplicate model function call in User test

https://github.com/jedireza/frame/blob/master/test/server/models/user.js#L142

It's not a duplicate. The first one uses username and the second one uses email.
:+1: Thanks for bringing this up. We can clean this up and make it more obvious though (especially since we're already using async right there).

ahh! sorry, didn't catch that.

If you'd like I'll take this one and clean it up.

:blush:


Style changes in ClusterEnsemble.py

Couple of things to do in ClusterEnsemble:

Figure out which attributes we actually need and how to define those attributes initially while still using the init of BaseUniverse.
Make verbosity of cluster_analysis() optional.
Refine plotting functions for histograms of clusters

This is out of date, will be addressed in a Refactor.


Update Percona Signing Key

Description
On December 28th 2018, Percona changed their signing key for new packages.
This means packages cannot be downloaded via Yum unless the key URL is updated.
There are still packages available on the repository that are signed with the old key,
so we need to have both keys available.
See https://www.percona.com/blog/2019/02/05/new-percona-package-signing-key-requires-update-on-rhel-and-centos/
I've struggled to run the tests for this project - chefspec fatal errors with the following message but that directory doesn't exist:
An error occurred in a `before(:suite)` hook.
Failure/Error: raise NotACookbook.new(path)

Berkshelf::NotACookbook:
  The resource at '/Users/kevans/Repositories/chef/percona/test/fixtures/cookbooks/test' does not appear to be a valid cookbook. Does it have a metadata.rb?

Similar PR raised at https://github.com/chef-cookbooks/yum-percona/pull/3 and it works there :)
Issues Resolved
None raised so far
Contribution Check List

[ ] All tests pass.
[x] New functionality includes testing.
[x] New functionality has been documented in the README if applicable

@teknofire or @kierenevans any chance you can look at running this project in CircleCI?

Hey @kierenevans looks like this needs a rebase.
I'll try and get this mopped up in this weeks sous-chefs meeting

If you can fix the merge issue, we can get this merged :D

          1 Error

:no_entry_sign:
Please include a CHANGELOG entry.

          1 Warning

:warning:
This Pull Request is probably missing tests.

   Apologies for the lack of contact.
Thank you to @josephholsten for rebasing this in #397, I've included the CHANGELOG.md added in that PR here to satisfy danger and adjusted the Dangerfile so it looks at the added files as well (though that's only relevant for this initial adding of the file).

Hey, we're just waiting for percona to get transferred to sous-chefs on the supermarket.
After that we'll have a 0.16.5 release with this fix in it.
Thanks for the fix!
Dan


Help needed - Importing all avatar images

Hello,
I am working on a social media research project and want to use for Dicebear for getting profile images 'avatars'. I need all the avatar images so that I can let user select from any one of them. Is that possible?
Thanks, and I love Dicebear!

You mean "avataaars"? The avatars are created dynamically. There are thousands of combinations. Maybe this is what you are looking for: https://getavataaars.com/


Move to androidx

This is a breaking change to support Androidx and requires react-native 0.60+. I've assumed a major version bump of 3.0.0 upon merge, but please let me know if you'd like me to use something else.
Need to:

[x] Update README

@mvayngrib this is ready for review. Thanks!

@chris-griffin thanks! will release as 3.0.0


[WIP] Removed XClarity Client gem requirement for tests

The rationale for this is Physical Infra specs seem to be the only ones with a gem requirement, but it doesn't appear that this is needed for the tests, but if it is, the tests should be refactored.

@miq-bot assign @jrafanie

@jrafanie what do you think? The tests still pass without the require 'xclarity_client' and so that's one less requirement in the code that's not needed.

@jrafanie can we merge? 🍰 ?

yup, I'm not sure why I didn't already merge it.  😕

yup, I'm not sure why I didn't already merge it. 😕

Just to keep me in suspense!


Look into different CI setup

Travis times out a lot lately for some of the fsm properties if they don't generate any output.
Maybe consider other CIs (Circle CI, ...?) or add a workaround script that outputs a character every minute or so..?

I think Github Actions will be more than enough here. What do you think?

I think so too, I don't think github actions already existed (or were mature enough) when I created this issue.
If you want, you could create a PR 😄  but there's not many changes nowadays so it's not super critical.

If you want, you could create a PR smile but there's not many changes nowadays so it's not super critical.

https://github.com/luc-tielen/lua-quickcheck/pull/27

That was fast 😲 . I will look into your changes later. Thanks!


The picture has to be pre-trained

The picture has to be pre-trained, how do I change the parameters of the 2D convolution into the parameters of the 3D convolution

Please check
https://github.com/Sense-X/UniFormer/blob/f92e423f7360b0026b83362311a4d85e448264d7/video_classification/slowfast/models/uniformer.py#L387-L421


[5.6] Fix JSON queries with table names on SQL Server

JSON queries on SQL Server don't yet support columns with table names:
DB::table('users')->where('users.options->language', 'en')->get();
DB::table('users')->whereJsonContains('users.options->languages', 'en')->get();

Table names have been broken because JSON queries work differently on SQL Server: While MySQL and PostgreSQL use the arrow operator (column->path), SQL Server uses JSON functions.
This led to incorrect SQL: where [users].json_value([options], '$."language"') = ?.
Unfortunately, the fix requires some refactoring. We have to move the use of wrapJsonSelector() from wrapValue() up to wrap(). We could integrate it into Database\Grammar::wrap(), but I think that it rather belongs into Database\Query\Grammars\Grammar since it doesn't concern Schema grammars.
I also used the opportunity to improve the SQL Server grammar by extracting wrapJsonPath(). It's a better solution than replacing the names of JSON functions in SQL strings.
I added table name tests for all databases.
The PR is also a preparation for supporting JSON queries on MariaDB (https://github.com/laravel/ideas/issues/811):
When replacing MySQL's arrow operator with the equivalent JSON_EXTRACT(), we would have run into the same wrapping problems.

Do you use Sql Server professionally on a daily basis?

I use it for one project, the rest is MySQL. So rather weekly than daily.

It looks like this changes the behavior of MySQL wrapping as well?

Do you mean the code changes in MySqlGrammar? Or that the generated SQL changes?

Well you change existing MySQL tests here:

I added table name tests for the other databases to catch bugs like this in the future.


⚠ ClamAV has degraded performance

In 6266a73, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 89 ms

Resolved: ClamAV performance has improved in fb1b3ed.


Any support for GPU ?

I notice that doing inference with language model on large amount of texts can be quite slow. In particular, it took me 11 minutes to decode around 4600 lines of text. I wonder if pyctcdecode supports using GPU for acceleration ? At the moment, I see that the model's using CPU only, making it hard for batch decoding and such.

@lopez86 Could you help me ?


Fix bugs

[ ] Allow running without VR
[ ] Fix camera orientation at start of scene
[ ] Check default settings (e.g., IPD)
[ ] Disable screen mirroring
[ ] Update readme
[ ] Add release
[ ] POV toggle
[ ] Remove bloom from UI

Keep up the good work! Let us know if you need some patreon $ to boost morale


LCD Screen Blinking after boot

I can get in via SSH, so the system is working, it seems after every update I battle with the LCD screen. I had saved the commands to install the driver and used it the last 3 updates and it worked fine once rebooted, this one not so much. Ideas??

Sounds like you are not using the LCD screen recommended in the readme and supported by the project with drivers installed in the SDcard image.
Can only  suggest to follow the manufacturer's instructions to set it up.
Can post the exact type and maybe pictures including the display's back, someone might have used it.

https://ibb.co/LrggmXm
I have a 3.5 inch display so ill try that, but it is a XPT2046. I also tried the following commands to install drivers which always worked before.
sudo rm -rf LCD-show
git clone https://github.com/goodtft/LCD-show.git
chmod -R 755 LCD-show
cd LCD-show/
sudo ./MPI4008-show

sudo raspi-config
7 Advanced options
A6 Screen Blanking
Select "No"

sudo raspi-config
7 Advanced options
A6 Screen Blanking
Select "No"

Unsubscribe

Unsubscribe

Please close this issue if it was solved.

@rootzoll Please close this


When delete a dialog, server throw an error

Describe the bug

Version

Browser

[ ] Chrome
[ ] Safari
[ ] Firefox
[ ] Edge

OS

[ ] macOS
[ ] Windows
[ ] Ubuntu

To Reproduce
Steps to reproduce the behavior:

Go to 'Design page'
Click on 'Delete a dialog' -> delete a dialog
See  Server error

Expected behavior

Screenshots

Additional context

And delete are not always successfully

same issue #2783


very large tutorial page added with tables of RESQML class attributes

I'm not sure this is a good idea, due to the size of the page – any thoughts on how this information should best be presented?

Codecov Report

Merging #251 (066da43) into master (9028acb) will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #251      +/-   ##
==========================================
+ Coverage   57.39%   57.41%   +0.01%
==========================================
  Files          53       53
  Lines       26142    26111      -31
==========================================
- Hits        15004    14991      -13
+ Misses      11138    11120      -18

Impacted Files
Coverage Δ

resqpy/olio/simple_lines.py
40.65% <0.00%> (-0.48%)
:arrow_down:

resqpy/olio/wellspec_keywords.py
66.66% <0.00%> (-0.31%)
:arrow_down:

resqpy/olio/keyword_files.py
41.80% <0.00%> (-0.13%)
:arrow_down:

resqpy/olio/factors.py
97.14% <0.00%> (-0.08%)
:arrow_down:

resqpy/lines.py
70.22% <0.00%> (-0.04%)
:arrow_down:

resqpy/olio/triangulation.py
92.62% <0.00%> (-0.04%)
:arrow_down:

resqpy/grid_surface.py
80.68% <0.00%> (-0.03%)
:arrow_down:

resqpy/olio/transmission.py
84.28% <0.00%> (-0.03%)
:arrow_down:

resqpy/model.py
67.69% <0.00%> (ø)

resqpy/grid.py
47.03% <0.00%> (+0.01%)
:arrow_up:

... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9028acb...066da43. Read the comment docs.

I've added /docs/scripts/schema.py
That file is geared up to be used as a module for programmatically interrogating the RESQML schema, or to be run as a script to generate a single text file holding rst-format tables showing the elements of each obj_* class, as used in the tutorial page.


Example Usage

It would be great to have an example of loading this as a plug-in to a project and calling it as a sub-task from a command.  If you all provide the general process, I gladly do the work here.
Basically my goal is to have something along the lines of this within the dodo.py file:
def task_deps():
    return {
        'task_dep':[
            'graph'
        ],
        'targets':[
          'tasks.png'
        ],
        'actions': ['dot -Tpng %(dependencies)s -o %(targets)s'],
    }

When. this is executed, doit deps, the following output is yielded:
doit deps
ERROR: deps. Task dependency 'graph' does not exist.

This being said, doit-graph has been installed into the environment.  Strange enough, if the command, doit graph is executed from the same terminal, the dot file is successfully generated.

This plugin provides a command not a task,  so you can not use it as a task_dep...
Although you could easily create a task that executes the command!

Thanks @schettino72.  What is the best way to do that?  I am pretty new to doit in general.


Investigate - /nllg does not list all groups

nllg pulls them straight from the db, so thats kinda weird

Seems like it's not telling other shards when a group was created and it doesnt appear on there until there is another reason to recache it, could that be the issue?

@BlackXnt Is this still exhibiting?

Safe to close this one.


Delete button for branches in confusing

Presently, the "Delete" button for branches is a trash bin icon:

This is unambiguous and provides satisfactory warning that what you're going to do is pretty severe.
However, on the page to confirm the deletion, the "No" button is ALSO a trash bin icon:

To me, this is confusing. I simply do not want to click on the "No" button even though I know it is innocuous because it uses the same icon. With the trash bin icon being used for "No" the logic seems to be that you are going to "trash" or delete your request to delete the branch.
I think an X icon might be preferred in this case, while the "Yes" button could be the trash bin icon. In any case this gave me pause and I wanted to see if anyone else had similar thoughts on this.

Just confirmed it on try.gitea.io and a custom 1.15.6 server. That problem is present in both servers.
I agree that this is confusing, and a potential solution should be backported to both 1.14 and 1.15. (Although I don't know whether 1.14 has the same problem but I would guess so.)

Made a fix.
Using the trash icon on the yes button would have been far more complicated, so I opted for the simple solution.


Added possibility to configure your API region via ServerOptFunc

Added possibility to configure your API region via ServerOptFunc.
Corrected typo for "WithHttpClient".

Example:
regionFunc := paapi5.WithRegion("eu-west-1")

//Create client
client := paapi5.New(regionFunc).CreateClient("MYTAG", "MYKEY", "MYSECRET")

Many thanks!


rsyslog crash on bosh-google-kvm-ubuntu-jammy-go_agent/1.18

Summary
I suggest to consider and evaluate rsyslog-openssl as replacement for rsyslog-gnutls.
Background
We had an rsyslog crash on one of our bosh managed VMs recently. This was not the first time the rsyslog daemon crashed, but unfortunately, I do not have more details on these older incidents.
# journalctl | grep "Oct 23 05:00:41" | head -1
Oct 23 05:00:41 <vm guid> rsyslogd[263401]: omrelp[<IP_ADDRESS>:10515]: error 'TLS record reception failed [gnutls error -54: Error in the pull function.]', object  'conn to srvr <IP_ADDRESS>:10515' - action may not work as intended [v8.2112.0 try https://www.rsyslog.com/e/2353 ]

I looked up OMRELP Manual which says that

If you have problems with the default setting, we recommend to switch to “openssl”.

This is the Reference for Error 2353.

@Benjamintf1 Would you have a problem with the Jammy stemcell switching to openssl as a tls backend for rsyslog?

cc @ctlong

The most immediate and quick answer is...I don't know?
The longer version is, off the top of my head and through immediate research, I don't have an inclination to say which is better or worse. I think I experimented with it and had trouble when trying to fix jammy, but I fixed it by fixing other settings on the receiver which seemed to have changed.
The easiest fix would be to setup a option to switch to openssl as the backend, but I'm not sure if that's sufficient to solve the problem or if it would require larger fixes. We have a pipeline setup with tests running against xenial, bionic, and jammy, so we should be able to run tests against any pr to insure it works on all of those stemcells at the moment.
Broadly speaking, I'm pretty amiable to different solutions, as long as it passes the existing ci.

We looked into it a little bit deeper, not only would you likely have to change quite a bit of the configuration to match the new network driver in syslog-release, but our current way of using the obsolete legacy directives in syslog-release are not even documented to work in the omrelp documentation.
While we have not made changes to remove relp, it has gotten a lot less support as a usecase compared to tcp/tls. We don't even have a test for relp at the moment. Broader industry support for relp is pretty low, and it doesn't seem to bring much to the table as a protocol. We should probably consider removing relp as a protocol from syslog-release.
That said, have you considered switching to tcp/tls?

@Benjamintf1 TCP is not 100% reliable, which might be required for audit relevant informations
https://rainer.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html
https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html
In addition big OS vendors have dedicated help / support for relp because of those requirements:
e.g. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-using_relp
On top of that rsyslog is aware of the problems with gnutls and heavily recommends to use openssl: https://github.com/rsyslog/rsyslog/issues/3915
The issue needs to be splitted into 2 dedicated "problems", the connection abort problem and the actual crash caused by the connection abort.
The connection abort was caused by bad TCP keepalive settings by the remote server and frequent connection aborts. There are even connection aborts with openssl without crash. Connection aborts can happen any time and therefore this is unrelevant here and needs to be fixed by the remote server.
This issue fix is about the crash of the rsyslog and the used implementation, the crash will be fixed with https://github.com/cloudfoundry/bosh-linux-stemcell-builder/pull/254

https://github.com/cloudfoundry/bosh-linux-stemcell-builder/pull/254/files#diff-b8c04108127281571c1e72921b036f853d2e481fe88c5b43a189a454eecac841R18
In so far as this, are you sure this functions as expected?
specifically in combination with the tls parameters, and the pre-existing import within syslog release(weird that this line exists at all given apparently the  line has been within the stemcell builder for 6 years, this is probubly where having a relp test would help perhaps?).

@Benjamintf1

In so far as this, are you sure this functions as expected?
yes we tested this in SAP for nailing down the issue.

specifically in combination with the tls parameters
short: yes, as we have 2 use cases inside SAP. One use case is the TLS shipment of syslog messages.

the pre-existing import within syslog release
The PR just adds additional openssl support for omrelp and once the module is loaded, the modload is a no-op. But i will remove this statement to avoid confusion as the rsyslog.conf is loaded first (and then subsequent rsyslog.d/*.conf)

On top of that the new rainerscript and the old syntax will co-exist and even ubuntu upstream uses old/new syntax in parallel: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/pull/254#issuecomment-1305313685

PR for bionic as well: @ramonskie https://github.com/cloudfoundry/bosh-linux-stemcell-builder/pull/255

Thank you Ruben, Benjamin, Carson for your input and discussions.
The PR provided by @h0nIg for Jammy has been use to create an AWS stemcell and tested in one of our environments.
The configuration works, the process rsyslogd is able to load the correct shared libs and metrics get properly shipped using RELP.
Do you require any additional input from our side?
I would be happy if you could merge the two PRs, that would perfectly resolve the present issue.

Thanks for merging the two PRs 254 and 255


Import trace error when adding option components

Hi,
Added the following components from /react-notion-x/src/third-party:

And included them in Components:

But I get this error import trace module not found:

Thoughts?

Follow the example: https://github.com/transitive-bullshit/nextjs-notion-starter-kit/blob/main/components/NotionPage.tsx#L47
Most of the collection-related imports you're trying to use aren't exported. Just use Collection.

Thanks for the quick reply.
Wanted to add in different database views. That’s why I added them
On Sun, Mar 27, 2022 at 2:26 AM Travis Fischer @.***>
wrote:

Closed #243
https://github.com/transitive-bullshit/nextjs-notion-starter-kit/issues/243
.
—
Reply to this email directly, view it on GitHub
https://github.com/transitive-bullshit/nextjs-notion-starter-kit/issues/243#event-6313441876,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKHJD4FS2EVJ6XWMEYIYG5DVCASVHANCNFSM5RYEJYYQ
.
You are receiving this because you authored the thread.Message ID:
<transitive-bullshit/nextjs-notion-starter-kit/issue/243/issue_event/6313441876
@github.com>

--
Magical regards,
Pooria Arab
Web3 enthusiast
www.pooriaarab.com
British Columbia, Canada
It is normal for me to take 2 days to read my emails and 2 more to reflect
on the matter and respond calmly. The culture of immediacy and the constant
fragmentation of time are not very compatible with the kind of life I lead
[image: facebook] https://www.facebook.com/Pooriarabb
[image: twitter] https://twitter.com/pooria_arab
[image: linkedin] https://www.linkedin.com/in/pooriaarab/
[image: instagram] https://www.instagram.com/pooria.arab/

Yayy 😀😀
Managed to show database views by changing this

Only thing is got to figure out how to change CSS of other views:


Inconsistencies in the sizes of components rendered within the dialog modal compared to those on full view pages

There are inconsistencies observed in the sizes of components rendered within the dialog modal compared to those on full view pages. The components rendered within the dialog modal are smaller than the ones rendered within a full view
Steps to reproduce the behavior:

Go to 'https://function-publicportal-test.shesha.dev/dynamic/TestModule/public-table'
Click on 'Public Portal' menu item
Click on 'Basic CRUD'
Click on 'Add' button
Check the components sizes
Navigate to the details view and check the size of the components
Actual Results: There are inconsistencies between the sizes

Expected behavior
Components rendered within the dialog modal and full view pages should maintain consistent sizes and proportions throughout the application

@Lukeybooi, Alex suggested we keep the same used on the Admin Portal. Before implementing, we need to discuss with the rest of the team

Created a separate item to address the discrepancies that we still have within the component labels and components sizes


oracle_user does not check if password needs to be updated

oracle_user module and oradb-manage-users role has update_password: ['always','on_create']. on_create does not care if password got changed and effectively does not allow to apply password changes against a database at all.
using always does however always try to ALTER USER ... IDENTIFIED BY ... which results in problems when user profile is set to not reuse x old passwords.
There was a basic functionality to check whether old password hash matches provided password hash, but this was commented out (lines 295-300 of python module) but when providing the new password as cleartext no kind of check was ever executed.
What is expected would be update_password: when_differ that will only then alter the password when it really needs to be changed.
This can be implemented simply for password hashes (by uncommenting mentioned lines and comparing given hash against the one retrieved from database).
It is a bit more difficult when provided password is in plaintext, as it would require validating it against the stored hash from the database. This is however also doable with the S:xxxxxxxx enrypted form which is available since 11gR2 afair and is used also in 19c. for code see https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/changes-in-oracle-database-12c-password-hashes/ in particular "S: part"
This change would allow changing password when really changes for accounts with profiles that do not allow password re-usal.

finally I did not use dbms_metadata as it would require changing some session settings that may crash other queries within the same cursor/session. Easiest was to extract the current hash from spare4 field where it is normally stored. Also - if this would fail, it would always report that password does not match hash and in worst case would trigger password change at every playbook run, which would not cause much problems.
I also changed the role not to define separate defaults for update_password (which were different that the one from module) but omit it if not defined thus falling back to module defaults. If they are not optimal, then this is where it should be changed.


Develop

Proposed changes
Merged bugfix for compilation error when trying to use DetailsInputView in Interface Builder.
Types of changes
What types of changes does your code introduce to Frames Ios?
Put an x in the boxes that apply

[X] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[ ] Lint and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)

Further comments
If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

Codecov Report

Merging #36 (02ffcd3) into master (383504d) will increase coverage by 0.51%.
The diff coverage is 93.93%.

@@            Coverage Diff             @@
##           master      #36      +/-   ##
==========================================
+ Coverage   92.29%   92.81%   +0.51%
==========================================
  Files          60       62       +2
  Lines        2805     2827      +22
==========================================
+ Hits         2589     2624      +35
+ Misses        216      203      -13

Impacted Files
Coverage Δ

Source/CardViewController.swift
95.16% <77.77%> (+7.05%)
:arrow_up:

Source/DetailsInputView.swift
100.00% <100.00%> (ø)

Tests/CardViewControllerTests.swift
99.19% <100.00%> (+0.05%)
:arrow_up:

Tests/StubCardViewControllerDelegate.swift
100.00% <100.00%> (ø)

Tests/StubCheckoutAPIClient.swift
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 383504d...02ffcd3. Read the comment docs.

Codecov Report

Merging #36 (02ffcd3) into master (383504d) will increase coverage by 0.51%.
The diff coverage is 93.93%.

@@            Coverage Diff             @@
##           master      #36      +/-   ##
==========================================
+ Coverage   92.29%   92.81%   +0.51%
==========================================
  Files          60       62       +2
  Lines        2805     2827      +22
==========================================
+ Hits         2589     2624      +35
+ Misses        216      203      -13

Impacted Files
Coverage Δ

Source/CardViewController.swift
95.16% <77.77%> (+7.05%)
:arrow_up:

Source/DetailsInputView.swift
100.00% <100.00%> (ø)

Tests/CardViewControllerTests.swift
99.19% <100.00%> (+0.05%)
:arrow_up:

Tests/StubCardViewControllerDelegate.swift
100.00% <100.00%> (ø)

Tests/StubCheckoutAPIClient.swift
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 383504d...02ffcd3. Read the comment docs.


Connection error. When start containers

[webmaster@localhost docker-puppeteer]$ docker-compose -f docker-compose.integration-tests.yml build --pull
app uses an image, skipping
Building tests
Step 1/3 : FROM buildkite/puppeteer:latest
Trying to pull repository docker.io/buildkite/puppeteer ...
sha256:e5b161ca16d91782549b58976cfe6ebebaaa76f76a2b1c92680f91ce0e7542a0: Pulling from docker.io/buildkite/puppeteer
Digest: sha256:e5b161ca16d91782549b58976cfe6ebebaaa76f76a2b1c92680f91ce0e7542a0
Status: Image is up to date for docker.io/buildkite/puppeteer:latest
---> d8855571c3f2
Step 2/3 : RUN npm i mocha
---> Using cache
---> 3fe3dd0e414b
Step 3/3 : ENV PATH "${PATH}:/node_modules/.bin"
---> Using cache
---> 208126a81ee6
Successfully built 208126a81ee6
[webmaster@localhost docker-puppeteer]$ docker-compose -f docker-compose.integration-tests.yml run tests
Creating docker-puppeteer_app_1 ... done
Started HeadlessChrome/72.0.3617.0
App
1) renders
0 passing (331ms)
1 failing

App
renders:
Error: net::ERR_CONNECTION_REFUSED at http://app/
at navigate (/node_modules/puppeteer/lib/FrameManager.js:107:37)
at process._tickCallback (internal/process/next_tick.js:68:7)
-- ASYNC --
at Frame. (/node_modules/puppeteer/lib/helper.js:144:27)
at Page.goto (/node_modules/puppeteer/lib/Page.js:624:49)
at Page. (/node_modules/puppeteer/lib/helper.js:145:23)
at Context.it (/integration-tests/index.test.js:37:33)

But when I use curl http://app/ in tests container, I have web page content from app container

It actually works for me but I thought the example looks cryptic and it's very hard to tell what's going on - and I still don't understand it to be honest.
Maybe just update the index.test.js test to something simpler in order to get your test to pass.
For example:
describe('App', () => {
  it('renders', async () => {
    const response = await page.goto('http://google.com')
    assert(response.ok())
  })
})```

I added wait-for-it.sh and cleaned up the examples in #55 — I think that should resolve this, but let me know if you're still seeing problems @alpody!


v1.4

Tables

[x] Add support to HTML tables.

"Credits" Table now displays properly.
Before

Table columns and rows etc., were all displayed as images.
Background for these images were white, which did not fit the dark theme.

After

Made custom images to use instead.
Converted those images to Base64 to use rather than hosting them. Cannot host on imgur as it is against TOS. CDN services were mostly paid. Hence using Base64 images was more convenient.


SAK-40247 Even though a user can edit a poll, they can't see the results.

@ern, Codacy has been addressed here, care to re-review?

@bjones86 Done

@maramar7, looks like the tests are still failing:
Tests in error:
  PollListManagerTest.testDeletePoll:198 Â» NumberFormat For input string: "null"
  PollListManagerTest.testSavePoll:123 Â» NumberFormat For input string: "null"

Tests run: 4, Failures: 0, Errors: 2, Skipped: 0

@bjones86 Done

There are still some tests failing here...
Running org.sakaiproject.poll.logic.test.PollListManagerTest
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 6.234 sec <<< FAILURE! - in org.sakaiproject.poll.logic.test.PollListManagerTest
testSavePoll(org.sakaiproject.poll.logic.test.PollListManagerTest)  Time elapsed: 0.131 sec  <<< FAILURE!
java.lang.AssertionError
	at org.sakaiproject.poll.logic.test.PollListManagerTest.testSavePoll(PollListManagerTest.java:130)

testDeletePoll(org.sakaiproject.poll.logic.test.PollListManagerTest)  Time elapsed: 0.016 sec  <<< ERROR!
java.lang.IllegalArgumentException: Poll id can't be null
	at org.sakaiproject.poll.logic.test.PollListManagerTest.testDeletePoll(PollListManagerTest.java:235)

Running org.sakaiproject.poll.logic.test.PollListManagerImplTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.608 sec - in org.sakaiproject.poll.logic.test.PollListManagerImplTest

Results :

Failed tests:
  PollListManagerTest.testSavePoll:130
Tests in error:
  PollListManagerTest.testDeletePoll:235 Â» IllegalArgument Poll id can't be null


How do I get the error num returned by sqlite, I need to handle the exception according to the error num

How do I get the error num returned by sqlite, I need to handle the exception according to the error num

I think they should be accessible using getErrorCode() and getExtendedErrorCode().
An example of that usage can be found in the database test file here

Yes, that's true, thanks @patrick-- for answering!


Mention documentation and roadmap in README

Coverage remained the same at 46.882% when pulling e22afdaaa0eb9aa0f78e3ac54b89ed3564cfeb2a on documentation into d53601534c526fc70ba3e82155b409e3cd508f8b on master.


Support turbine-js package build, remove temp dir build

Description of change

Support separated packages from https://github.com/meroxa/turbine-js/pull/115

meroxa-js-cli
meroxa-js-framework

Remove temporary directory building from turbine-js apps. Turbine-js apps now build just like turbine-go minus creating a binary

Meroxa cli calls turbine-js-cli to copy dockerfile from package to app directory
Meroxa cli tar/uploads app directory
Meroxa cli cleans up dockerfile

Type of change

[ ]  New feature
[ ]  Bug fix
[x]  Refactor
[ ]  Documentation

How was this tested?

[ ]  Unit Tests
[x]  Tested in staging
[ ]  Tested in minikube

Demo

Additional references
https://github.com/meroxa/turbine-js/pull/115
Documentation updated

Confirmed with @jayjayjpg that locally I can run apps init, apps run and apps deploy with the new changes

@raulb no worries at all!


change default max conns per pool

Changes the default max connections per pool to 10. This was previously System.coreCount which can create issues on systems with only one core as some complex Vapor routes require 2 connections at once to run successfully.
Fixes #45.

Great minds think alike: https://github.com/skelpo/database-kit/blob/master/Sources/DatabaseKit/ConnectionPool/DatabaseConnectionPoolConfig.swift#L6
:D
I did not see this PR yet.

I would start with a lower value. PostgreSQL connections are expensive and many instances only support 50-100 of them (see eg.
https://cloud.google.com/sql/docs/postgres/quotas#limits), at which point we could quickly run into problems with such a high constant (especially if the service is replicated). I would suggest something between 3 and 5; it still can be increased manually if need be.

@MrMage manually how?

By manually providing a ConnectionPoolConfig object to your service config.

@MrMage I think I would rather err on the side of too many connections than too few, given that debugging too few seems to be very difficult.

Hey @tanner0101, you just merged a pull request, have a coin!
You now have 875 coins.


No way to pass stdin to Container.execute

I recently added LXD backend support to launchpad-buildd.  I was able to use pylxd for almost all of it, but we need to be able to pipe standard input into lxc exec in some cases, and pylxd's Container.execute can't do that.  Could this feature be added?

How would you see that working?  Can you give an example of the types of things you are piping into lxc exec?
At the moment the execute command on a container (https://github.com/lxc/pylxd/blob/master/pylxd/models/container.py#L207) takes a list as the command (like subprocess.check_call() and friends).  It then uses the lxd RESTful API to run the command (async) and uses a websocket to get the results.
So, how might what you'd like to achieve fit into that model?

One example is where we want to add trusted keys to a build.  We do this by running the rough equivalent of lxc exec NAME apt-key add -, which takes key material on stdin.
Whether the command is passed as a list or not is immaterial for this purpose, although of course I agree that that's generally the best way to do things.  I'm not particularly familiar with websockets, but ideally I'd like to be able to pass something like stdin=fileobj to Container.execute and have it forward bytes from that file object to the stdin websocket returned by POST /1.0/containers/NAME/exec.  The API docs seem to indicate that this is supported, since they say "so that users can pass stdin and read stdout".

Okay, I think I understand.  I think we could do something where an object could be passed that has a duck-typed read() method which work similar to io.BufferedReader type class (i.e. read bytes until EOF) which would block if it was still waiting for bytes, etc.)  It would run in a separate thread in the ws4py websocket library.
As sys.stdin works like a file object, and thus, (IIRC io.TextIOWrapper) this would work if sys.stdin was passed to it, but equally, piping from files would also work (e.g. the file object from open()).
Is this what you were thinking?

This is pretty important missing feature.  Not easily worked around at all and seemingly somewhat basic functionality.
@cjwatson how did you work around this?


Questions

Hey @lox,
A few questions:

how would you feel about using https://github.com/urfave/cli (or something similar) for CLI arg parsing, to allow for args and/or env vars to be used, instead of flag?
would test coverage be of interest to you?

Yup, I'd be ok with urfave/cli or https://github.com/alecthomas/kingpin. The latter is probably better out of the box these days.

I've been planning to add some tests, this was a weekend prototype that ended up not needing much follow up till now!

Yup, I'd be ok with urfave/cli or https://github.com/alecthomas/kingpin. The latter is probably better out of the box these days.

kingpin looks reasonably good, does it mandate using = between flag names and values? Or does it take both --flag value + --flag=value syntax? I tend to default to --flag value when typing something these days, obviously depends on the end user.

I've been planning to add some tests, this was a weekend prototype that ended up not needing much follow up till now!

fair call :) as they always start. I'd be open to contributing some of my mock work potentially from dockerd-ci-proxy here... time permitting.

kingpin looks reasonably good, does it mandate using = between flag names and values? Or does it take both --flag value + --flag=value syntax? I tend to default to --flag value when typing something these days, obviously depends on the end user.

Nope, either works I believe.

Nope, either works I believe.

nice, sounds good then :)

Gonna close this one for now @CpuID!

:+1:


Write automated tests: Android Basics with Compose

There is an error when accessing to a node with the text "Cost of Service".
This text was changed to "Bill Amount".

This should be fixed now.
@JohnTrawolta let us know if you still face any isssues.


add test and remove unused warn log

to confirm #868

Codecov Report

Merging #886 into master will increase coverage by 0.06%.
The diff coverage is 39.17%.

@@            Coverage Diff             @@
##           master     #886      +/-   ##
==========================================
+ Coverage   53.55%   53.62%   +0.06%
==========================================
  Files          40       41       +1
  Lines        7527     7534       +7
==========================================
+ Hits         4031     4040       +9
+ Misses       2983     2979       -4
- Partials      513      515       +2

Impacted Files
Coverage Δ

engine_cond.go
54.96% <0%> (+0.96%)
:arrow_up:

error.go
0% <0%> (ø)

session.go
56.69% <39.72%> (+0.34%)
:arrow_up:

xorm.go
64.06% <0%> (-1.57%)
:arrow_down:

engine.go
58.62% <0%> (+0.45%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c8ae6fa...65c7bc1. Read the comment docs.


Codeunit 99000854 OnBeforeForecastConsumption Add Parameters

In Codeunit 99000854 OnBeforeForecastConsumption from #4746 can we add CurrForecast, ExcludeForecastBefore, UseParm, and LineNo to the function.
local procedure ForecastConsumption(var DemandInvtProfile: Record "Inventory Profile"; var Item: Record Item; OrderDate: Date; ToDate: Date) UpdatedOrderDate: Date
var
    ForecastEntry: Record "Production Forecast Entry";
    ForecastEntry2: Record "Production Forecast Entry";
    NextForecast: Record "Production Forecast Entry";
    CustomCalendarChange: Array[2] of Record "Customized Calendar Change";
    TotalForecastQty: Decimal;
    ReplenishmentLocation: Code[10];
    ForecastExist: Boolean;
    NextForecastExist: Boolean;
    ReplenishmentLocationFound: Boolean;
    ComponentForecast: Boolean;
    ComponentForecastFrom: Boolean;
    IsHandled: Boolean;             // NEW
begin
    // NEW+
    OnBeforeForecastConsumption(DemandInvtProfile,Item,OrderDate,ToDate,UpdatedOrderDate,IsHandled,CurrForecast,ExcludeForecastBefore,UseParm,LineNo);
    if IsHandled then
        exit;
    // NEW-

// NEW +
[IntegrationEvent(false, false)]
local procedure OnBeforeForecastConsumption(var DemandInvtProfile: Record "Inventory Profile"; var Item: Record Item; OrderDate: Date; ToDate: Date; var UpdatedOrderDate: Date; var IsHandled: Boolean; var CurrForecast: Code[10]; var ExcludeForecastBefore: Date; var UseParm: Boolean; var LineNo: Integer)
begin
end;
// NEW -

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


versionName should be defined in one location, then referenced everywhere else

Right now it's in app's build.gradle.kts, data-android and data:musicbrainz.

bdf8287d4943541b5736c0a945b9440c4eb8a3e6


Update Company Communicator 4.1.0 to 4.1.3

Hi =),
How to update this application from 4.1.0 to 4.1.3? I need go to : resource groups > settings > deployments and check the template and redeploy?
Anyone can confirm ? Thx for helping.

@nigelzin you need to go to: resource groups > app service > deployment center > then click on sync. This has to be repeated for all the 3 functions and make sure to restart the app after sync. Thanks!


Method to get all available countries

Description
It would be nice with a function in DateSystem to fetch all available countries, so we can produce country selectors, based on what is supported in Nager.Date
I.E.
public static List<CountryInfo> GetAvailableCountries()
{
    return new List<CountryInfo>
    {
        new CountryInfo
        {
            CommonName = "Sweden",
            OfficialName = "Kingdom of Sweden",
            LocalName = "Sverige",
            CountryCode = "SE",
            AlternativeSpellings = new Dictionary<string, string>
            {
                { "SE", "Sverige" },
                ...
            },
            Region = "Europe"
        },
        ...
    }
}

I just saw that Nager.Country is also an available package.

Yes i have split this packages not all projects need both.


Modify project completion metric to use classification counts

As per the comment from PFE - https://github.com/zooniverse/Panoptes-Front-End/issues/4093#issuecomment-336896345
Perhaps we should summate the workflow metrics according to the prefs now set instead of using the default retired ratio.

I'm absolutely behind this 100%. If I was the kind of person who wrote in all caps, I would be right now.

The current calculation doesn't take into account the proportion of each subjects subject counts, instead we just average out the the sum of the active workflow percentages. https://github.com/zooniverse/Panoptes/blob/fddc6e38dd4d1d61f3b767b2b3904d5496286a8a/app/workers/calculate_project_completeness_worker.rb#L42
If i have 2 active workflows:

workflow 1 = 1 retired / 5 subjects = 0.2
workflow 2 = 9 retired / 10 subjects = 0.9

The current calc is (0.2 + 0.9) / 2 = 0.55, it is actually 10/15 = 0.66. This needs to change to either just the classification counts or we weight the percentages coming from each workflow, e.g. 0.2 of 5/15 and 0.9 of 10/15.

Also noting that classification counts may be impacted by early retirement from caesar as well, though it's probably more accurate than what we have.

The current set-up continues to confuse researchers and volunteers alike. Got this message from Jules earlier today:
"Just a quick query. The stars on the Solar Stormwatch II homepage has always remained at 0%. It's currently at 37% if you click on the link but not everyone does and a demoralising 0% is all they see. Is this fixable? If not could it be removed just leaving the link? I've had a look to see if the SSW team could do anything but it doesn't look like it."
We have a progress bar by classification count option in the stats page, we really should at lest allow the option for the homepage!

This still continues to confuse researchers. Today it was Ed Hawkins, the PI of the Weather Rescue projects. He has released three projects and knows his stuff, but still couldn't figure out why his progress bar was not increasing.

Another volunteer complaining that this was making them lose heart on Muon Hunter today. Would really love to make this change at some point soon.
The fact we can show a progress bar based on classifications on our stats page but not the homepage is so frustrating!

Not saying we shouldn't modify the completeness metric API side but whatever we change too will have trade-offs as well, especially with external retirement systems (e.g. caesar).
Interestingly the fact this information exists API side for the stats page to work means this change can happen in the UI as well.

Yeah, I totally accept it won't makes sense for projects with other retirement rules, however at the moment it makes no sense for the majority of projects.

What change can happen UI side? Could you open an issue so we at least don't replicate it in the rewrite?

The fact we can show a progress bar based on classifications on our stats page but not the homepage

Apologies, that sent early. The issue is the API project resource has a non-optimal calculation for the completion metric whereas the stats pages calculate workflow completeness differently (configurable) based on expected classification counts against the retirement count.
The UI could use that information to calculate different progress metrics for presentation to the user.
Perhaps the API project resource should have both metrics as both have trade offs:

Subject retirement is a lagging indicator - it confuses folks as it only starts changing after a long time
Classification counts may never be reached if caesar is retiring data early vs the exepcted counts and projects will never finish.

One option might be to track the classifications per subject for retired subjects - This average, times the number of active subjects is a measure of the total classifications needed to retire all subjects regardless of the manner (Caesar, Bayesian pipeline, custom rules) retirement is determined.  EDC = (total expected classifications required - classifications done)/average daily classifications.
I will look at trying to get a average classifications to retirement from looking at the Subjectworkflowstatus. This will be slow but it is not necessary to keep a frequently updated value since it is only used as an average. Perhaps there is some means to determine if there is a major change to the workflows or retirement means that would affect the number, otherwise run with a weekly or even less frequently updated value.
Even better would be a direct calculation of the classifications required to complete unretired subjects, based on the average classifications to retirement, and the current number of classifications for each subject, but this number would have to be calculated every time an EDC was to be updated.
These EDC values also need to be cleaned of the increasing number of "training/practice " workflows with very high retirement limits and never any completeness  eg Earthquake detective runs 3 practice  workflows and rotates through small active subject sets of real data - the ECD numbers for the active subjects are be skewed because the the practice workflow subjects are lumped in...  Using subject retired metrics the error is fairly small on the current set (43 vs 2426 subjects), but some past subject sets are were very small (<100 subjects) and the error was large.  There are also many more accumulated classifications for the practice sets than for the current active "real" data at any time, so classification count metrics are also skewed.

This continues to baffle even the most experienced Zooniverse researchers. The latest victims are our very own Helen Spiers and Cliff Johnson!

I'd second Grant's thoughts here - the progress bar is counterintuitive (I'm sure there's some sort of counter pun there...)

The new project home page shows completion for each workflow, which I think fixes this.

And none of the competed classification metrics deal with Caesar customized retirement rules in the EDC calculation, nor deal with "training" workflows with extremely high retirement rules on small subject sets.
What is needed is the average classifications to retire a subject for the active data for each workflow.  With that number and the average classifications per day for that workflow, one can give good completion estimates that reflect the work done and remaining in a linear fashion.
To get that number, one must look at the SubjectWorkflowStatus  attribute classification count for all retired subjects divided by the number of retired subjects.  This number would reflect the actual rules in place, and the actual experience of the project.
This number is very stable for a given subject set,  workflow, and the project rules; it does not change rapidly once a significant number of subjects have retired. Hence it would not have to be updated frequently.  Early on, when a subject set has been changed or is new, and there is no data,  the retirement limit would serve as a proxy.

I have tabulated the retirement completeness and the current classification completeness based on the current retirement limit, total subjects and total classifications for all active workflows for all live launched projects.
As of today there were 34 active workflow where either the classification completeness or the retired subject completeness was 100%  Of these only 17 show both completeness as 100% indicating that the classification metric is wrong about 50% of the time.
There are 5 cases where the retired completeness is 100% (workflow is finished) but the classification completeness is <100%, and 12 cases where the classification completeness  is >=100% but the retirement completeness is showing <100% and the workflow is not finished.
In addition there are at least 11 more situations (out of 171workflows) where the mismatch between the two metrics is such that there has to be an issue with the classification metric - many more could be inaccurate to a lesser degree.
There are three common ways the classification metric for a workflow gets messed up - changes to the retirement limit during the processing of a subject set, changes to the contents of active subject sets, and the activity of early retirement rules (Caesar).
To get a good classification based metric we would need to tabulate the current number of classifications needed to complete all unretired subjects vs those required to complete all subjects given the current subject list, current retirement limit and a correction for early retirement for the current subject mix.   This would require working at the subject level and tallying up classifications needed to complete each subject with some sort of allowance for Caesar based on its affect on the retirement of subjects up to that point.
This would be very resource costly and slowwww!  I am wondering if we can use the statistics of the underlying process to estimate a completeness metric based on modeling the process.

Currently looking into this. (Excuse my rookie-ness and possible rant) I used https://www.zooniverse.org/projects/achmorrison/steelpan-vibrations/ to help me visualize and understand what the issue is. So from my understanding, the stats on the project's stats page is inconsistent with the completeness stat on the project's home page. (Due to different calculations for each)
Again, using https://www.zooniverse.org/projects/achmorrison/steelpan-vibrations/ as an example.
On the project's /stats page, completeness in this scenario is
= classifications / (subjects * retirement limit) (at this point in time its  83678/(13119 * 15)=83678/196785 = approx 43%)
But on the project's home page, completeness is calculated as the average of retired subjects / all subjects for all  workflows.  In this case since there is 1 workflow, it would just be retired / all. (at this point in time, thats 3528/15119 = approx 26%) which looks less complete than 43% on /stats page.
The one issue I have here is that by the /stats pages calculations (again using numbers of steelpan vibrations at this point in time as an example), we have 83678 calculations, if we divide that by the retirement limit of 15, we technically should have approx 5578 retired subjects instead of 3528 retired subjects.
So the question is, is  classifications /  (classifications needed to retire a subject) true "completeness" or is retired / all subjects in a workflow true "completeness". I dont know.
One thing, i was thinking we could do (which might be what this issue is asking for?) is have a separate calculation similar to that of the /stats page and maybe have the project team decide which they would like to use as a calculation for completeness for their project's home page with the 'default' being the current way we are calculating on home page and adding maybe a disclaimer that numbers may be different due to multiple reasons (unretirement, early retirement, updating retirement limit, etc). (Or maybe it's enough to have a disclaimer that numbers may be different?)

Yuenmichelle1,
The situation is much more complicated than your comment above:

subjects are selected for classification pseudo randomly, so the retirement of a subject depends on chance – when that subject finally get selected enough times. Certainly total classification/retirement limit does not equal retired subjects until very near the end of the process.  The pseudo random process can be modeled, and surprisingly the relation between classification completeness and retirement completeness throughout the process depends only on the retirement limit – (though the smaller the number of subjects in a subject set, the more “noise” there is)  I generate a linearized completeness from the retirement completeness in my daily dashboard reports https://github.com/PmasonFF/Project_Health_Dashboard. by modeling the selection process for different retirement limits.

How to state a project completeness is difficult – many projects have multiple workflows, some of which have no or very high retirement limits meant for training – including those into any completeness metric distorts the picture – for a project we really need a measure of how much work ( ie classification count) is needed to complete the research workflow(s) / the total work (ie classification count to retire all research subjects) with training workflows exempted.

Caesar and early retirement make this exercise very difficult.  Early retirement rules eliminate an unknown set of subjects early in the process. This results in retired subject counts being higher than “normal” for the same classification count with normal retirement.  Because the percentage of subjects that will retire early is unknown, the process can not even be modeled accurately.

The current project completeness is the average of the retirement completeness over all active workflows, and does not account for one or more workflows having larger active subject sets and hence require more work.

What could be done to define a more useful project completeness:
First determine which workflows are to be included in the completeness metric in the project builder “show stats”, including only those workflows selected for visibility on the detailed stats page.
Then calculate the number of classifications required to complete all remaining subjects that are not retired assuming the retirement limit is required in each case.  Think of this as the work remaining for a workflow.  Note this in an expensive calculation and must be done by querying every unretired subject in the active subject sets to see how many classifications have been done and how many remain.
Finally a calculation of the total classifications done plus the work remaining gives a total work for the workflow/active subject sets.
Completeness for the workflow is (total work - work remaining)/total work. .  As the workflow progresses this metric self-corrects for the early retirement issues. (ie early retirement knocks out some subjects, those remaining are those where the retirement limit is a more accurate measure of the work remaining for that subject, and  the total work calculation correctly reflects the effect of early retirement on those subjects already retired.)
Completeness for the project would be the (sum of total work for all included workflows – sum of the work remaining for all included workflows)/ sum of total work for all included workflows.

Adding to the discussion from my experience on calculating an estimate of remaining classifications for Features where I need estimated days to completion.
Pros of classification completion.

Motivating for volunteers/classifiers as progress bar will start moving much earlier eg. on a flat retirement limit of 15, subject completion doesn't start until around 45-50% of classification completion.
It MIGHT be easier to understand - caveat: the change to a straight % of Total project subjects completed has made progress bars cleared than before, given that around 60% of projects have multiple workflows during their lifespan.

Cons

The complexity of calculation - as explained by Pmason above, it's difficult.
I use the basic calculation for each workflow of total subj x ret limit - classifications done
but the problems encountered are
Removal of Practice workflows
Workflows with no retirement limit (currently 8% of projects) so I use an estimate or if genuinely no retirement I have to exclude.
Caesar, Group transcription and other retirement rules - I use estimates of average retirement limit from previous workflows/seasons/experience. Retirement limits are often non-linear over time within a workflow, with lower retirement classifications at the start for human images or blank images/pages to final subjects being more difficult requiring more classifications.
Workflows where the retirement limit changed at some point in the past - I set max/min points to detect these workflows and then use the rate of subject ret/classifications to estimate classifications on remaining subjects. It's a very rough calculation which isn't reliable until subjects start retiring but it's the best I can do with reasonable efficiency.
Workflows which in the past have received a great many classification on retired subjects, usually when in Paused status, and then have new subjects added without removing the old batched (Steelpan vibrations is an example of this). This gets worse over time when projects are never 'zero-ed" and my solution is the same as for 4. above. The problem with a progress bar on these is that it would look like far more progress has been made than true and at some point the bar would track down/back as it adjusted for the error. Not a good for motivation or simplicity. (If I can suggest to projects to zero for new data sets I do, but with limited success)

I worked through today's active projects and conditions 3 ,4 and 5 are true for at least 37% of projects so cannot be ignored.