SingleUtteranceGmmDecoder.feature_pipeline() causes segmentation fault

Kaldi:  compiled from master (d366a93aad)
PyKaldi: pykaldi-cpu 0.1.3 py37h14c3975_1 pykaldi
Python: 3.7.11
OS: Manjaro Linux VM
When calling feature_pipeline() from a SingleUtteranceGmmDecoder object twice, a segmentation fault occurs.
I am trying to run online GMM based decoding. I translated a similar c++ example that can be found here: https://kaldi-asr.org/doc/online_decoding.html#GMM-based, to PyKaldi. But when I try to feed the OnlineFeaturePipeline instance with audio data, it crashes with a segmentation fault.
I could enclose the error to being thrown when trying to get the feature pipeline twice and created following minimal working example:
#!/usr/bin/env python

from kaldi.online2 import (
    SingleUtteranceGmmDecoder,
    OnlineGmmAdaptationState,
    OnlineFeaturePipelineCommandLineConfig,
    OnlineGmmDecodingConfig,
    OnlineFeaturePipelineConfig,
    OnlineFeaturePipeline,
    OnlineGmmDecodingModels,
)
from kaldi.fstext import read_fst_kaldi

import subprocess, sys
from os.path import expanduser

base_path = expanduser("~/speech/kaldi/asr")
kaldi_root = expanduser("~/speech/kaldi/kaldi")

subprocess.run(
    f"{kaldi_root}/src/bin/matrix-sum --binary=false scp:{base_path}/data/train/cmvn.scp - >/tmp/global_cmvn.stats", shell=True
)

feature_cmdline_config = OnlineFeaturePipelineCommandLineConfig()
feature_cmdline_config.feature_type = "mfcc"
feature_cmdline_config.mfcc_config = f"{base_path}/conf/mfcc.conf"
feature_cmdline_config.global_cmvn_stats_rxfilename = "/tmp/global_cmvn.stats"
feature_config = OnlineFeaturePipelineConfig.from_config(feature_cmdline_config)

decode_config = OnlineGmmDecodingConfig()
decode_config.faster_decoder_opts.beam = 11.0
decode_config.faster_decoder_opts.max_active = 7000
decode_config.model_rxfilename = f"{base_path}/exp/mono/final.mdl"

gmm_models = OnlineGmmDecodingModels(decode_config)
pipeline_prototype = OnlineFeaturePipeline(feature_config)
decode_fst = read_fst_kaldi(f"{base_path}/exp/mono/graph/HCLG.fst")
adaptation_state = OnlineGmmAdaptationState()

decoder = SingleUtteranceGmmDecoder(
    decode_config, gmm_models, pipeline_prototype, decode_fst, adaptation_state
)

# this one does not crash, but using pipe.accept_waveform crashed for me
pipe = decoder.feature_pipeline()
# the next line crashes with "segmentation fault (core dumped)"
pipe = decoder.feature_pipeline()

You're using an old version of PyKaldi. You need a matching Kaldi installation from https://github.com/pykaldi/kaldi, the master version there is compatible with the master version of PyKaldi.
If you really want to use an old v0.1 PyKaldi, you'd need https://github.com/pykaldi/kaldi/tree/pykaldi_01
I really recommend to use the latest version though since you're otherwise likely to run into compatibility issues with newer models.


Roboflow App Integration

This is a demo app for creating datasets in an automated way using OAK camera with Mobilenet-SSD object detector and roboflow app.
Usage details are described in README.md Since the last review I have added an option for uploading images in arbitrary resolution using the synchronized NV12 frames.

Thanks a lot @tomek-l ! Will give it a try and circle back

Hi @tomek-l,
I tried to run the script and it failed with the following issue
INFO: Uploaded image with ID: WXhyfRvH8o7idrl3T3SR
INFO: Auto-uploading grabbed frame with 1 annotations!
INFO: Uploaded image with ID: WXhyfRvH8o7idrl3T3SR
ERROR: failed annotation for image ID: WXhyfRvH8o7idrl3T3SR
{'error': {'message': 'Image not found in dataset.', 'type': 'InvalidParameterException', 'hint': 'The image ID is returned by the upload API.'}}
ERROR: failed annotation for image ID: WXhyfRvH8o7idrl3T3SR
{'error': {'message': 'Image not found in dataset.', 'type': 'InvalidParameterException', 'hint': 'The image ID is returned by the upload API.'}}

Ran with the following parameters - --upload_res 640x360 --autoupload_interval 0.5 --autoupload_threshold 0.7 - and I don't see these images uploaded to roboflow at all
I also prepared a patch to use blobconverter and e.g. face detection network (same input/output dims, just detects faces for simplicity), feel free to use it to ease the setup (with blobconverter, we can download blob dynamically)
Index: roboflow-integration/models/download.sh
===================================================================
diff --git a/roboflow-integration/models/download.sh b/roboflow-integration/models/download.sh
deleted file mode 100644
--- a/roboflow-integration/models/download.sh	(revision 7f43e206f0e005879bd611875ac36d1c0ac64ed2)
+++ /dev/null	(revision 7f43e206f0e005879bd611875ac36d1c0ac64ed2)
@@ -1,1 +0,0 @@
-wget https://artifacts.luxonis.com/artifactory/luxonis-depthai-data-local/network/mobilenet-ssd_openvino_2021.4_5shave.blob
Index: roboflow-integration/main.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/roboflow-integration/main.py b/roboflow-integration/main.py
--- a/roboflow-integration/main.py	(revision 7f43e206f0e005879bd611875ac36d1c0ac64ed2)
+++ b/roboflow-integration/main.py	(date<PHONE_NUMBER>413)
@@ -1,41 +1,18 @@
 import time
-import json
 import argparse

-from pathlib import Path
 from collections import deque, OrderedDict
 from concurrent.futures import ThreadPoolExecutor

+import blobconverter
 import cv2
 import depthai as dai
 import numpy as np

 from utils.roboflow import RoboflowUploader

-BLOB_PATH = "models/mobilenet-ssd_openvino_2021.4_5shave.blob"
-LABELS = [
-    "background",
-    "aeroplane",
-    "bicycle",
-    "bird",
-    "boat",
-    "bottle",
-    "bus",
-    "car",
-    "cat",
-    "chair",
-    "cow",
-    "diningtable",
-    "dog",
-    "horse",
-    "motorbike",
-    "person",
-    "pottedplant",
-    "sheep",
-    "sofa",
-    "train",
-    "tvmonitor",
-]
+BLOB_PATH = blobconverter.from_zoo(name="face-detection-retail-0004", shaves=5)
+LABELS = ["background", "face"]

 def make_pipeline():
Index: roboflow-integration/requirements.txt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/roboflow-integration/requirements.txt b/roboflow-integration/requirements.txt
--- a/roboflow-integration/requirements.txt	(revision 7f43e206f0e005879bd611875ac36d1c0ac64ed2)
+++ b/roboflow-integration/requirements.txt	(date<PHONE_NUMBER>723)
@@ -1,9 +1,11 @@
 certifi==2021.10.8
 charset-normalizer==2.0.9
-depthai==<IP_ADDRESS>.dev0+c0d306db16bd76a98ca9b96297e21668c3176277 --extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/
+--extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/
+depthai==<IP_ADDRESS>.dev0+c0d306db16bd76a98ca9b96297e21668c3176277
 idna==3.3
 numpy==1.21.5
 opencv-python==<IP_ADDRESS>
 Pillow==8.4.0
 requests==2.26.0
 urllib3==1.26.7
+blobconverter==1.2.8
diff --git a/roboflow-integration/models/.gitkeep b/roboflow-integration/models/.gitkeep
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
GIT binary patch
literal 0
Hc$@<O00001

I signed up for another account with Roboflow to repeat the process fully from scratch,

Inferring from the logs this is what happens:

We get HTTP 200 response from image upload
Roboflow API responds with a valid id for the image (WXhyfRvH8o7idrl3T3SR)
When we try to make the annotation call for that same image WXhyfRvH8o7idrl3T3SR literally 500ms later, roboflow API is like "I have never met this man in my life" and throws HTTP 404 response.

This issue came up for me twice during development, and then went away after several minutes without any changes. What I suspect is that there might be a minimum time delta between the image getting uploaded, and the image being ready for annotation. I'm unsure whether there's a good way to address that. A dumb, but perhaps an effective way of solving that would be to add time.sleep(5) between upload and annotation request.

On a side note, what's even more strange is that Roboflow API would create the same id (WXhyfRvH8o7idrl3T3SR) for two consecutive images. Seems like this shouldn't happen (I get different ids even if I upload the same image twice).

I found that sometimes the images require a page refresh to show up.  are you able to see anything under:
(https://app.roboflow.com/tl-ryah5/stuff-blpdm/images/?

For debugging, could you try the same set of parameters on my dataset + API key and show the resulting console output? The command below seems to be working fine on my machine:

python3 main.py --dataset stuff-blpdm --api_key mwRWRi5DtIYZUqzdYlxj  --upload_res 640x360 --autoupload_interval 0.5 --autoupload_threshold 0.7

I guess this must have been something on the Roboflow side, I didn't see any images being uploaded but as I run it now, it works fine with both mine and your arguments.
Will merge this PR and add my fixes later (mostly blobconverter usage).
Thanks a lot!


Dynamic Function Calling

The annotation approach for function calling is really clean when you have a well defined set of functions that you can hard code into java classes.
It would be great if we could also define function parameters/attributes dynamically.

@jwclark Take a look at #339 , it solves this problem.

@TheoKanning - was this closed because #339 will be merged shortly?


sse2: implements last remaining intrinsics

sse2 is complete.

I don't know why _mm_loadr_pd is failing on Travis. It is not impacted by this PR !
I am going to fix _mm_store_pd on Windows.

@gwenn this is failing on 32 bit builds:
test x86::i586::sse2::tests::_mm_loadr_pd ... thread 'x86::i586::sse2::tests::_mm_loadr_pd' panicked at 'attempt to add with overflow', coresimd/src/x86/i586/sse2.rs:4055:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.
FAILED

Very similar code used to break the i686-apple-darwin (incrementing the offset in the while loops overflowed). IIRC the way I fixed it was by using repr(align) (see here ).
We should probably remove all similar looking code from the test suite.

I hope I will understand when I will see your patch how I made _mm_loadr_pd failed without modifying it.

@gwenn You did not break it, it was already broken, but the issue was not surfacing before. Either a new version of the compiler broke it (there was a change related to simd and repr(transparent) merged this week), or adding new tests/functions to the module did :/

@gnzlbg Thanks for your explanation.

@gwenn I've submitted a PR (#245) that might fix this.

@gwenn I've merged the test fixes, maybe the fix the issues here (I've restarted the build but it might be picking the old master commit so you might have to rebase).

Now things look way better, the i586/i686-unknown-linux-gnu builds fail with:
x86::sse2::assert__mm_shuffle_pd_shufpd' panicked at 'too many instructions in the disassembly', stdsimd-test/src/lib.rs:322:8

You probably want to enable the assert_instr of __mm_shuffle_pd only for x86_64 platforms but the tests should be green afterwards.

Sorry,
I don't have any windows machine so I can't investigate.

This is all green :)


Add ayu.land

Hi! I’d like to add my personal wiki/memex site to the webring ^ ^
I’m a China-based graduate student that dabble in interaction/multimedia design, musical composition, as well as computing in general. 100r & XXIIVV have both been a major inspiration for me, and a month or two ago I also started to put together a home for all my creations and scribbles with a bespoke site generator. English is my second language, but I’ve tried my best to make all content available as bilingual.
The webring icon is at the bottom of each page, but it relies on webfont to display properly (and falls back to an emoji), as I’d like the colour to be consistent with surrounding text; please specify if that poses a problem, in which case I shall go back to ordinary images. A link is also placed on the “Friends” page.
Thanks in advance and wish you a very pleasant lunisolar new year of the Rabbit 🐇

Hi Ayu :) Your website is wonderful <3 I don't mind for the webring icon, my browser doesn't seem to have half the emoji needed to see everything as a lot of it displays as little boxes, but the ring does show up. If you ever want to try, since the webring icon is a svg, you could also theme it with the rest by changing the stroke color based on the site's applied css theme. But in any case, it's not a problem because the rest is so well put together
Welcome to the webring!

Thank you for the appreciation, I’m delighted to join you!
Have a nice day 🧡


Avoid scroll on first load

close #55

@snewcomer this looks more readable too. Would you mind adding a patch version bump?


How to remove Transpose Layer after transfer to onnx

I have succesfully transfered my tf.keras built .h5 model to .onnx file. Furthermore, I need to transfer this .onnx file to .caffemodel and .prototxt. It turns out that the onnx2caffe project shows
TypeError: ONNX node of type Transpose is not supported.
I went back to check where the transpose comes from, but there is no Transpose layer in my tf.keras net structure.
May I ask how can I remove the Transpose Layer?

The generated onnx model may have Transpose node, because the input must be NCHW format for many ops, but keras model can have both NCHW or NHWC as input. So you should not remove Transpose node from onnx model. Actually Transpose is a common op, why onnx2caffe does not support it?

I get the your point. But I just stuck in here. The onnx2caffe project I used is as following.
https://github.com/MTlab/onnx2caffe
Is there any way to transfer keras model to caffemodel directly ?

I use the tensorflow as the backend to build the tf.keras sturcture, so in default for tensorflow, it will use NHWC, which is channel last.

This issue seems out of scope of keras2onnx, because (1) onnx model has to introduce Transpose after conversion. (2) ideally keras to caffe is a different topic. One work around (although not convenient) may be that we train the tensorflow model using NCHW, and then convert to onnx.

I tried set the trained data as channel first type by adding
tf.keras.backend.set_image_data_format('channels_first')
But it turns out more Transpose Layers show up, :(
I may try using keras2caffe to transfer my model.
Very appreciate for your reply!

Training with channel_first does not remove transpose layers

@avinash31d , Hi , Have you figured how to get rid of the transpose layer. I've tried using keras2caffe to transfer my tf.keras .h5 model to caffemodel , but the prediction results are not same.

Hi,@DanielXu123.
Whether your problem has been solved?I come across the same problem.
There is a transpose operation before the output.


Is there an option to encrypt the local secrets file?

Given it is not a good idea to store secrets in cleartext anywhere, is there a way to encrypt the locally stored secrets file?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ba5deb43-08e3-09fe-1696-29f49764580b
Version Independent ID: 0c910667-1d39-47dd-c70b-ab40999ea160
Content: Securely saving secret application settings for a web application - Azure Key Vault
Content Source: articles/key-vault/vs-secure-secret-appsettings.md
Service: key-vault
GitHub Login: @CawaMS
Microsoft Alias: cawa

@JeffGSparks
Thanks for your feedback! We will investigate and update as appropriate.

@JeffGSparks thank you for your feedback. We thought about this and the thing is Visual Studio needs to be able to decrypt the file to display content in VS. If that's the case, encrypting the file won't make the information more secure because if user has access to the file he will have access to Visual Studio and will be able to view the secret values anyway.
If there is more comment or feedback, please do share.
Thanks a lot!

@JeffGSparks We hope the clarification provided by @CawaMS Helped . Should you have any further query , please tag any of us to your reply or you an alternatively always open a new request. WE will close this issue now.
Thank you .


Re: The Puppet Race: On your mark

I suggest linking more directly to the next blog post in the five-part series (forward linking). There is a link, but it could be more obvious ("get an account" may be perceived as something else).
The Puppet Race
More generally, I think a link (to the next page in the series) at the end of each page would be expected, e.g. near "Next time".

That's a really good suggestion. I've added a little forward linking at the end of each article. "Next time" is a great link text too.
Thanks!


Some changes to make the files compile with gcc on Linux.

I'm planning on using this library for my bachelor thesis. It did not compile right away using gcc on Xubuntu 14.04, so I made some adjustments and thought they might be useful to you too.
Thanks a lot for providing the code!

Great that you're using this in your thesis, don't forget to reference me!
This library is very much a WIP, so prepare for bugs :)

Yes of course, will do, I'll be very grateful if your code helps me out the way I hope it will. :)


Why is pip install called with --ignore-installed

This makes it hard to prepare an environment with conda and have pip packages with C dependencies in the requirements.txt
Details

Project URL: https://github.com/TUW-GEO/gldas
Build URL (if applicable): https://readthedocs.org/projects/gldas/builds/5774270/
Read the Docs username (if applicable): cpaulik

Expected Result
pygrib should not be installed with pip since it is already installed via conda
Actual Result
pygrib installation failed because of missing system libraries for compiling the c code.

I worked around it by using python setup.py install instead.

Using setup_py_install: true and pip_install: false in readthedocs.yml works, but it requires that the conda environment file lists all dependency packages that should be installed through pip. As an example:
name: py36
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.6
  - anaconda
  - pytest-cov
  - pytest-xdist
  - coverage
  - pip:
      - better-apidoc
      - trajectorydata
      - sphinx-autobuild
      -<EMAIL_ADDRESS>

This seems it can be closed. Our yaml config might soon have more options to configure these steps, but currently this case is a bit of an edge case.


Segmentation Deletes lowercase letters

Problem
When the *.pseg.png is generated all the letters which are below written Just like y,g,p, are not detected in the file need the solution to that
See the word Players where lowercase "y" is missing. Need solution to this.
tried changing the --vscale but no changes in the file is being observed. Tried changing the scale but that too failed. any other option for changing the scale.
Input File

Output File

Steps to Reproduce (for bugs)

1../run-test
Your Environment

Python version: 2.7
Git revision of ocropy:1
Operating System and version: Ubuntu 16.04 LTS

I think i found the problem but can't think of a solution that is easy to implement by now.
The Problem:
Your idea to change the value of the scale parameter was fine, since the characters are removed in the function remove_hlines.
With an estimated scale of ~31, all objects are deleted that have a width >~310 px. These objects are detected based on connected black pixels.
As I took a closer look at the removed characters I recognized that they are connected to the long horizontal lines. Because of that each horizontal line and its connected characters are interpreted as a single object which is then removed by the remove_hlines function since the objects width is greater than ~310px.


[Q] run on Android ?

Is there any chance that this project is able to run on android TV (Nvidia Shield Pro, which has a Tegra X1) ?

You mentioned Tegra X1 that is NVIDIA's chip, which I take to mean that you want to use CUDA on it.
First let's ignore annoying drivers, performance, etc. According to this document, we can notice that this chip uses the Maxwell architecture（Compute Capability is 5.x). But the minimum Compute Capability required for this project is 6.0.
So I don't think it's a good idea, at least not yet.

Yep. Also, given the computational requirements of the model, running on the Tegra X1 would not be a very pleasant experience, I think.

Thanks for your answers @moyix @Frederisk
And for your work (I've not tested yet because of GPU requirement).
It's amazing to see open-source project that ease the use of alternative to Github Copilot.


Croatia

Added sponsors for Croatia Event. Please approve

Hi Damir! It looks like you added the ".vs" folder. Can you please remove it? Just delete the folder and commit + push on this same branch.
Thanks!

Hello,
i did delete it.
Thanks,
Damir
On 1.4.2021. 14:41:11, Gianluca Sartori @.***> wrote:
Hi Damir! It looks like you added the ".vs" folder. Can you please remove it? Just delete the folder and commit + push on this same branch.
Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub [https://github.com/sqlcollaborative/DataSaturdays/pull/179#issuecomment-811879832], or unsubscribe [https://github.com/notifications/unsubscribe-auth/AOK2VTSDCFDBLU2CZ6O5WWTTGRSWPANCNFSM42GKITAQ].

.gitignore added and .vs removed. Can you check now

Inclusive language warning refers to a comment in .gitignore so it should be fine


links no longer supported in newer version of Jenkins?

Can confirm? I'm using Jenkins 2.107.3 and pipelines

My mistake. Updated Jenkins and realized I forgot to re-configure Jenkins for base url.


rquickshare: 0.8.2 -> 0.10.2

Description of changes
https://github.com/Martichou/rquickshare/compare/v0.8.2...v0.10.2
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

welcome to nixpkgs!


Get-RunspaceData error: Collection was modified; enumeration operation may not execute

I am leveraging Invoke-Parallel to process VM actions in parallel in an Azure Automation runbook:
        $AzureRMVMs | Invoke-Parallel -ScriptBlock {

            $ARMVM = $PSItem

            $VMStatus = $ARMVM | Get-AzureRmVM -Status | Select-Object Name, @{n = 'PowerState'; e = {($PSItem.Statuses | Where-Object Code -Like "*PowerState*").DisplayStatus}}

            switch ($using:VMAction) {

                'Start' {

                    if ($VMStatus.PowerState -ne 'VM running') {

                        Write-Output "VM $($ARMVM.Name) is not running - starting"

                        $ARMVM | Start-AzureRmVM

                    } else {

                        Write-Output "VM $($ARMVM.Name) is already running - no action triggered"

                    }

                }
                'Stop' {

                    if ($VMStatus.PowerState -eq 'VM running') {

                        Write-Output "VM $($ARMVM.Name) is running - stopping"

                        $ARMVM | Stop-AzureRmVM -Force

                    } else {

                        Write-Output "VM $($ARMVM.Name) is already stopped - no action triggered"

                    }

                }
                'Default' {Write-Output "VM Action $($using:VMAction) not defined in runbook, no action taken"}

            }

        } -RunspaceTimeout 300 -NoCloseOnTimeout

This works as expected, however, the following error is generated in the error stream:
Get-RunspaceData : Collection was modified; enumeration operation may not execute.
At Invoke-Parallell 542 char:13

        Get-RunspaceData -wait

        ~~~~~~~~~~~~~~~~~~~~~~

CategoryInfo          : CloseError: (:) [Write-Error], InvalidOperationException
FullyQualifiedErrorId : Get-RunspaceData

Any idea what might be causing this?

Nice catch! Indeed there is a collection modification inside a foreach loop
The simplified version of this code is:
Foreach($runspace in $runspaces) {
...
                    $temphash = $runspaces.clone()
                    $temphash | Where-Object { $_.runspace -eq $Null } | ForEach-Object {
                        $Runspaces.remove($_)
                    }
...
}

https://github.com/RamblingCookieMonster/Invoke-Parallel/blob/master/Invoke-Parallel/Invoke-Parallel.ps1#L325-L328


Added support for pipenv and pythonbytes with a make wrapper

Thanks for the PR! Are you willing to work with me on this one or would you mind if I finish it myself?

Sure, what's up?

I review-ed your work, let me know if there are parts you like to do and which should I.

I hope you don't mind, I implemented downloading from https://pythonbytes.fm .


Additional fields

New code optionally adds FORMAT fields for B allele frequency (BAF) and Log R ratio (LRR) if the command line flag "--include-attributes" is included with the ID of each of the attributes you would like (i.e. "--include-attributes BAF" for only BAF or "--include-attributes BAF LRR" for both).
One potential limitation is the aggregation across multiple BPMRecords. Currently, the code only takes the first record. Is there a better way to do this?

@brianhill11, is this ready to be merged? Should we issue a pull request to Ilumina and start a conversation?

@nalinigans still going back and forth via this issue here
It looks like they're open to a PR once we figure this out.


minor fix in makefile

@BenjiLee I needed to add this in case the directory doesn't exist (sometimes the case with AMIs)

👍

Current coverage is 18.95% (diff: 100%)

Merging #884 into master will decrease coverage by 0.01%

@@             master       #884   diff @@
==========================================
  Files           470        470
  Lines         19978      19978
  Methods           0          0
  Messages          0          0
  Branches       2922       2922
==========================================
- Hits           3790       3787     -3
- Misses        15737      15741     +4
+ Partials        451        450     -1

Powered by Codecov. Last update a0d50ba...dbbf4a9


Add second part

Signed-off-by: Matt Jarvis<EMAIL_ADDRESS>
What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

WIP, just checking formatting ...

Ready for review

Changes as requested @ANeumann82


[Update Request]: MilosParipovic.OneCommander 3.65.2

What type of update are you requesting?
A new version of an existing package
Current Package Identifier
MilosParipovic.OneCommander
Package Version
3.65.2
Please describe the changes you would like to see
There is a new Release avaliable, 3.65.2 at https://www.onecommander.com/OneCommanderSetup<IP_ADDRESS>.msi

Closed With https://github.com/microsoft/winget-pkgs/pull/135820


[WIP] Make shell scripts POSIX-compatible

I used ShellCheck and checkbashisms to find and get rid of Bash-specific syntax and features in shell scripts.
This PR is not about refactoring or fixing the scripts. The intention of this PR is to make the scripts work in any POSIX-compliant shell.
This will allow developers to contribute to PHP-CS-Fixer using environments that don't have Bash, e. g. Alpine Linux.
Fixes #3544.

Looks like I don't have enough expertise in POSIX to make ci-integration.sh POSIX-compatible.

I can make ci-integration.sh POSIX-compatible after all, I just can't figure out how to test it in tests/Smoke/CiIntegrationTest.php.

how to test what in CiIntegrationTest?

@keradus CiIntegrationTest tests the ci-integration.sh script, but the tests use Bash arrays to verify results. I think I can rewrite ci-integration.sh to use strings instead of Bash arrays, but I can't figure out how to rewrite tests in CiIntegrationTest.

If this gets merged it might be worthwhile adding this check to the CI process

@ntzm You mean ShellCheck and checkbashisms?

oh, i understood you already made changes in ci-integration.sh ;)
then my comment is not valid

note:
should target LTS 2.2 branch, not master

Closed in favor of #3548.

for future, you could change target branch of PR without opening it as new PR


refactor: make codegen result compatible with older browsers.

📝 Description
The result of codegen is a file that has enough browser compatibility except it has some replaceAll methods in helpers.mjs which prevents the codegen results from great browser compatibility

NOTE: with this minor improvements, the codegen result now is compatible with iOS <= 12

⛳ Current behavior (updates)
It has some replaceAll methods in two different shared utility.
🚀 New behavior
I've replaced the two replaceAll methods with pure replace, split, and join methods.
💣 Is this a breaking change (Yes/No):
NO
📝 Additional Information

@theMasix Thanks for this. Could you add some tests for this change?

@theMasix Thanks for this. Could you add some tests for this change?

Is there any test for these methods already?

After thinking about this, I'd recommend installing a polyfill in your app instead.

Install core-js
import 'core-js/features/string/replace-all

Alternatively, you can run the codegen:prepare hook to modify these methods.
Thanks for understanding.


Add godoc to the templates

Add godoc to client templates
Closes: #5

Just 1 typo / LGTM

Please rebase on top of main after #9 merges 😄

Included in #11


TOC missing when viewing multiple pages in the Marketplace & Extensibility section

Some of the pages have the TOC missing when I clicked into the page from the TOC, including the page in the details here. A few more examples:
https://docs.microsoft.com/en-us/azure/devops/service-hooks/events?view=azure-devops
And basically everything under the Service hooks tree
https://docs.microsoft.com/en-us/azure/devops/extend/develop/contributions-overview?view=azure-devops
And some but not all pages under the Develop extensions tree.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4dc61b61-ed67-8313-99d4-d736e7f48234
Version Independent ID: 3e0fcf87-9744-cec8-24f8-9f68371959c6
Content: Integrate with service hooks - Azure DevOps
Content Source: docs/service-hooks/overview.md
Product: devops
Technology: devops-collab
GitHub Login: @chcomley
Microsoft Alias: chcomley

@felixtsem5 -- thank you for your feedback, and apologies for the delayed response.  This TOC is working for me now.


Kong 1.2.2 to Kong 1.3 Migrations up informationals/errors:

Summary
I am building from src and compiling for various reasons so I may be closing this on my own fairly soon if I didn't do certain steps right, migrations up throws these two things of interest:
Warning:
  | 2019/08/21 07:48:03 [warn] 8#0: *2 [lua] _G write guard:12: __newindex(): writing a global lua variable ('luarocks') which may lead to race conditions between concurrent requests, so prefer the use of 'local' variables
-- | --
  | stack traceback:
  | /usr/local/share/lua/5.1/luarocks/loader.lua:46: in main chunk
  | [C]: at 0x7f14ce518540
  | [C]: in function 'pcall'
  | /usr/bin/kong:3: in function 'file_gen'
  | init_worker_by_lua:48: in function <init_worker_by_lua:46>
  | [C]: in function 'xpcall'
  | init_worker_by_lua:55: in function <init_worker_by_lua:53>, context: ngx.timer

Error:
Error:
  | /usr/local/share/lua/5.1/kong/pdk/service.lua:107: module 'resty.kong.tls' not found:No LuaRocks module found for resty.kong.tls
  | no field package.preload['resty.kong.tls']
  | no file './resty/kong/tls.lua'
  | no file './resty/kong/tls/init.lua'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls.ljbc'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls/init.ljbc'
  | no file '/usr/local/openresty/lualib/resty/kong/tls.ljbc'
  | no file '/usr/local/openresty/lualib/resty/kong/tls/init.ljbc'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls.lua'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls/init.lua'
  | no file '/usr/local/openresty/lualib/resty/kong/tls.lua'
  | no file '/usr/local/openresty/lualib/resty/kong/tls/init.lua'
  | no file './resty/kong/tls.lua'
  | no file '/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/resty/kong/tls.lua'
  | no file '/usr/local/share/lua/5.1/resty/kong/tls.lua'
  | no file '/usr/local/share/lua/5.1/resty/kong/tls/init.lua'
  | no file '/usr/local/openresty/luajit/share/lua/5.1/resty/kong/tls.lua'
  | no file '/usr/local/openresty/luajit/share/lua/5.1/resty/kong/tls/init.lua'
  | no file '/.luarocks/share/lua/5.1/resty/kong/tls.lua'
  | no file '/.luarocks/share/lua/5.1/resty/kong/tls/init.lua'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls.so'
  | no file '/usr/local/openresty/lualib/resty/kong/tls.so'
  | no file './resty/kong/tls.so'
  | no file '/usr/local/lib/lua/5.1/resty/kong/tls.so'
  | no file '/usr/local/openresty/luajit/lib/lua/5.1/resty/kong/tls.so'
  | no file '/usr/local/lib/lua/5.1/loadall.so'
  | no file '/.luarocks/lib/lua/5.1/resty/kong/tls.so'
  | no file '/usr/local/openresty/site/lualib/resty.so'
  | no file '/usr/local/openresty/lualib/resty.so'
  | no file './resty.so'
  | no file '/usr/local/lib/lua/5.1/resty.so'
  | no file '/usr/local/openresty/luajit/lib/lua/5.1/resty.so'
  | no file '/usr/local/lib/lua/5.1/loadall.so'
  | no file '/.luarocks/lib/lua/5.1/resty.so'
  | stack traceback:
  | [C]: in function 'require'
  | /usr/local/share/lua/5.1/kong/pdk/service.lua:107: in function 'new'
  | /usr/local/share/lua/5.1/kong/pdk/init.lua:294: in function 'new'
  | /usr/local/share/lua/5.1/kong/global.lua:116: in function 'init_pdk'
  | /usr/local/share/lua/5.1/kong/cmd/migrations.lua:90: in function 'cmd_exec'
  | /usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:87>
  | [C]: in function 'xpcall'
  | /usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:44>
  | /usr/bin/kong:7: in function 'file_gen'
  | init_worker_by_lua:48: in function <init_worker_by_lua:46>
  | [C]: in function 'xpcall'
  | init_worker_by_lua:55: in function <init_worker_by_lua:53>

Some of the Docker file to give you an idea of how I am doing things for the build:
##### ---- From Source with Custom PATCH Approach ----- #####
RUN apk add --no-cache --virtual build-deps wget tar  ca-certificates \
        && apk add  --update --no-cache readline-dev outils-md5 linux-headers bsd-compat-headers m4 yaml-dev build-deps make gcc g++ libgcc zlib-dev perl pcre pcre-dev unzip tzdata wrk luarocks git

#CONFIG ENV
ENV LUAROCKS_KONG_VERSION=1.3.0rc2-0
ENV RESTY_VERSION=<IP_ADDRESS>
ENV RESTY_OPENSSL_VERSION=1.1.1c
ENV RESTY_LUAROCKS_VERSION=3.1.3

#Pull Openresty and patch
RUN wget https://openresty.org/download/openresty-$RESTY_VERSION.tar.gz \
        && tar -zxvf openresty-$RESTY_VERSION.tar.gz -C /tmp \
        && wget https://github.com/Kong/openresty-patches/archive/master.tar.gz \
        && tar -zxvf master.tar.gz -C /tmp \
        && cd /tmp/openresty-$RESTY_VERSION/bundle \
        && wget https://github.company.com/raw/Repo/company-kong-docker/dev/patches/ngx/nginx-1.15.8-cve_2019_9511_cve_2019_9513_cve_2019_9516.patch -P /tmp/openresty-patches-master/patches/$RESTY_VERSION/ \
	&& cd /tmp/openresty-$RESTY_VERSION/bundle && for i in /tmp/openresty-patches-master/patches/$RESTY_VERSION/*.patch; do patch -p1 < $i; done

#DROP in 1.15.8 files manually with Kong and Upstream Keepalive Changes (no patching)
RUN rm /tmp/openresty-$RESTY_VERSION/bundle/nginx-1.15.8/src/http/modules/ngx_http_upstream_keepalive_module.c \
    && rm /tmp/openresty-$RESTY_VERSION/bundle/nginx-1.15.8/src/http/ngx_http_upstream.c
COPY patches/ngx/nginx-1.15.8-ngx_http_upstream.c /tmp/openresty-$RESTY_VERSION/bundle/nginx-1.15.8/src/http/ngx_http_upstream.c
COPY patches/ngx/nginx-1.15.8-ngx_http_upstream_keepalive_module.c /tmp/openresty-$RESTY_VERSION/bundle/nginx-1.15.8/src/http/modules/ngx_http_upstream_keepalive_module.c

#Dowload OpenSSL
RUN cd /tmp \
    && curl -fSL https://www.openssl.org/source/openssl-$RESTY_OPENSSL_VERSION.tar.gz -o openssl-$RESTY_OPENSSL_VERSION.tar.gz \
    && tar xzf openssl-$RESTY_OPENSSL_VERSION.tar.gz

#Dowload 1.3 Kong dependency https://github.com/Kong/lua-kong-nginx-module
RUN mkdir /tmp/kong \
    && cd /tmp/kong \
    && git clone https://github.com/Kong/lua-kong-nginx-module

#Build OpenResty from source
RUN cd /tmp/openresty-$RESTY_VERSION \
        && ./configure -j2 --with-openssl=/tmp/openssl-$RESTY_OPENSSL_VERSION --add-module=/tmp/kong/lua-kong-nginx-module  --with-pcre-jit   --with-http_ssl_module --with-http_realip_module --with-http_stub_status_module --with-http_v2_module \
        && make -j2 \
        && make install

#Build Luarocks from source
RUN cd /tmp \
        && wget https://luarocks.org/releases/luarocks-$RESTY_LUAROCKS_VERSION.tar.gz \
        && tar zxpf luarocks-$RESTY_LUAROCKS_VERSION.tar.gz \
        && cd luarocks-$RESTY_LUAROCKS_VERSION \
        &&  ./configure --lua-suffix=jit  --with-lua=/usr/local/openresty/luajit  --with-lua-include=/usr/local/openresty/luajit/include/luajit-2.1 \
        && make build \
        && make install

#Needed for luarocks dependency builds to force https vs ssh
RUN git config --global url.https://github.com/.insteadOf git://github.com/

#Install Kong via luarocks
RUN luarocks install kong $LUAROCKS_KONG_VERSION OPENSSL_DIR=/tmp/openssl-$RESTY_OPENSSL_VERSION OPENSSL_LIBDIR=/tmp/openssl-$RESTY_OPENSSL_VERSION CRYPTO_DIR=/tmp/openssl-$RESTY_OPENSSL_VERSION/.openssl

#Get the Kong executable resty script
RUN wget https://raw.githubusercontent.com/Kong/kong/master/bin/kong -P /usr/bin
RUN chmod 777 /usr/bin/kong
RUN mv /usr/local/openresty/bin/resty /usr/bin

######------ END OF FROM SOURCE ------######

Maybe you can spot an issue, or can I just do a luarocks install resty.kong.tls to fix this?
Additional Details & Logs

Kong version 1.3

Interesting, so the rest.kong.tls is actually the module that comes from here:
#Dowload 1.3 Kong dependency https://github.com/Kong/lua-kong-nginx-module
RUN mkdir /tmp/kong \
    && cd /tmp/kong \
    && git clone https://github.com/Kong/lua-kong-nginx-module

So that must mean my --add-module=/tmp/kong/lua-kong-nginx-module failed:
#Build OpenResty from source
RUN cd /tmp/openresty-$RESTY_VERSION \
        && ./configure -j2 --with-openssl=/tmp/openssl-$RESTY_OPENSSL_VERSION --add-module=/tmp/kong/lua-kong-nginx-module  --with-pcre-jit   --with-http_ssl_module --with-http_realip_module --with-http_stub_status_module --with-http_v2_module \
        && make -j2 \
        && make install

Looks right to me though.

Hi @jeremyjpj0916,
Such a coincidence, we just opened https://github.com/Kong/kong/pull/4944 to provide more information for how to setup Kong development environment manually. In short, for compiling the OpenResty suite the recommended method is to use the new openresty-build-tools repository which will automatically include any required module and apply patches for you automatically, so the resulted OpenResty will have all the required libraries already included.
In the case for Kong 1.3, the following command should produce a usable OpenResty build:
./kong-ngx-build -p build \
        --openresty <IP_ADDRESS> \
        --openssl 1.1.1c \
        --luarocks 3.1.3 \
        --pcre 8.43

Let me know if it works for you!
Datong

@dndx appreciate the reply, about 10 minutes ago I was just saying that I should scratch the way we do it and used yalls official repo against my alpine 3.10 base for building from source. I will have to change a few things and include more nginx patchwork I run custom with Kong but that's a good approach for my base install process. Will hijack that shell script and invoke it with the args (and add a few of my own things to it). Will report back if changing over works. I still am scratching my head slightly because what I did do should work and worked for 1.2.1(and added mentioned 1.3 dependencies). Could very well be a path or permissions problem in containers potentially for the new stuff I changed for 1.3 so far.

I also got this problem. It seems like make install does not always install tls.lua. I needed to manually install that even though I had --add-module. I was sure this was going to bite us.

Will hijack that shell script and invoke it with the args (and add a few of my own things to it). Will report back if changing over works.

@jeremyjpj0916 if you need any tweaks to the script to make kong-ngx-build work properly in your environment with your customizations (e.g. loading extra patches from a third-party path), please let us know — or even better, send a PR adding those as optional flags and we'll be happy to take them!

I am likely running into . what Mr. Bugle here has described hah. Regardless I am excited to try yalls build tools script so I will give it a go.
@hishamhm Certainly would be cool to think up a pattern that will enable 3rd party patching into the flow with an optional arg(and how does that work when someone has 3rd party patches that all touch the same source c files? Can it?). I am not as git versed as the Kong squad on the whole using patch git diff files for doing code merging yet 😆, I do it newb old school drop and replace files, so what I have are two files that take what kong 1.3 needs AND what I need and I intend to find a spot in the shell script post patch work to simply drop these in as c file replacements prior to the compile:
nginx-1.15.8-ngx_http_upstream.c.txt
nginx-1.15.8-ngx_http_upstream_keepalive_module.c.txt
I believe this much needed upstream ssl keepalive logic I and a few others I have seen need will eventually make its way into OpenResty from what yah fearless leader mentioned awhile back so I am pretty content with just file dropping until it makes it in.
Hopefully will have the build tools tested in my dockerfile by EOD and maybe even a 1.3 deployed in our dev 👍 !

@hishamhm found the first problem with the build script running in a standalone fresh container. Seems to expect a localized /work directory to exist but it does not in our container. Maybe folks that have been running it for awhile already had a /work directory present on their laptop. Continuing to work it into our build process but thought to mention that bug.

Getting closer, seems to always fail on luarocks installing of Kong now, like so many dependencies github gets tired of all the luarocks pulls lol. Always hangs specifically on pgmoon for me, tried 3 separate builds and it failed same way, time to 1 off and install pgmoon prior to luarocks kong install and see if that helps:
kong-lapis <IP_ADDRESS>-1 depends on pgmoon (not installed)
Installing https://luarocks.org/pgmoon-1.10.0-1.rockspec

Cloning into 'pgmoon'...
fatal: unable to connect to github.com:
github.com[0: <IP_ADDRESS>]: errno=Operation timed out

Error: Failed installing dependency: https://luarocks.org/kong-lapis-<IP_ADDRESS>-1.src.rock  - Failed installing dependency: https://luarocks.org/pgmoon-1.10.0-1.rockspec  - Failed cloning git repository.
Removing intermediate container 5e5744dca170
error: build error: The command '/bin/sh -c luarocks install kong $LUAROCKS_KONG_VERSION OPENSSL_DIR=/openresty-build-tools/work/openssl-$RESTY_OPENSSL_VERSION OPENSSL_LIBDIR=/openresty-build-tools/work/openssl-$RESTY_OPENSSL_VERSION CRYPTO_DIR=/usr/local/kong/openssl' returned a non-zero code: 1

@dndx Another issue when specifying a /usr/local/kong thats not a singular folder is output like this:
    && ./kong-ngx-build -p /usr/local/kong \

Come build time watch the output of the luarocks 3.1.3 installation:
Initializing project 'luarocks-3.1.3' for Lua 5.1 ...
  | -----------------------------------------------------
  |
  | Checking your Lua installation ...
  | Adding entries to .gitignore ...
  | Preparing ./.luarocks/ ...
  | Wrote .luarocks/config-5.1.lua
  | Preparing ./lua_modules/ ...
  | ./luarocks already exists. Not overwriting it!
  | Preparing ./lua for version 5.1...
  | cp ./build/config-5.1.lua .luarocks/config-5.1.lua
  | mkdir -p "//usr/local/kong/luarocks/bin"
  | install "build/luarocks" "//usr/local/kong/luarocks/bin/luarocks"
  | mkdir -p "//usr/local/kong/luarocks/bin"
  | install "build/luarocks-admin" "//usr/local/kong/luarocks/bin/luarocks-admin"
  | mkdir -p "//usr/local/kong/luarocks/etc/luarocks"

Notice the double // going on? it ends up being simply /usr/local/kong/luarocks in the end but still a problem. Need something in shell script to handle -p build vs -p /full/path as I also tried running as -p full/path and that fails for us.

Getting close now, hitting path issues:

ERROR: /usr/bin/kong:9: module 'kong.cmd.init' not found:
--
  | no field package.preload['kong.cmd.init']
  | no file './kong/cmd/init.lua'
  | no file './kong/cmd/init/init.lua'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init.ljbc'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init/init.ljbc'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init.ljbc'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init/init.ljbc'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init.lua'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init/init.lua'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init.lua'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init/init.lua'
  | no file './kong/cmd/init.lua'
  | no file '/usr/local/kong/openresty/luajit/share/luajit-2.1.0-beta3/kong/cmd/init.lua'
  | no file '/usr/local/share/lua/5.1/kong/cmd/init.lua'
  | no file '/usr/local/share/lua/5.1/kong/cmd/init/init.lua'
  | no file '/usr/local/kong/openresty/luajit/share/lua/5.1/kong/cmd/init.lua'
  | no file '/usr/local/kong/openresty/luajit/share/lua/5.1/kong/cmd/init/init.lua'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init.so'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init.so'
  | no file './kong/cmd/init.so'
  | no file '/usr/local/lib/lua/5.1/kong/cmd/init.so'
  | no file '/usr/local/kong/openresty/luajit/lib/lua/5.1/kong/cmd/init.so'
  | no file '/usr/local/lib/lua/5.1/loadall.so'
  | no file '/usr/local/kong/openresty/site/lualib/kong.so'
  | no file '/usr/local/kong/openresty/lualib/kong.so'
  | no file './kong.so'
  | no file '/usr/local/lib/lua/5.1/kong.so'
  | no file '/usr/local/kong/openresty/luajit/lib/lua/5.1/kong.so'
  | no file '/usr/local/lib/lua/5.1/loadall.so'
  | stack traceback:
  | /usr/bin/kong:9: in function 'file_gen'
  | init_worker_by_lua:48: in function <init_worker_by_lua:46>
  | [C]: in function 'xpcall'
  | init_worker_by_lua:55: in function <init_worker_by_lua:53>

I see Kong files luarocks installs when pulling kong actually lives in:
/usr/local/kong/luarocks/share/lua/5.1/kong/*

And I tried adjusting the docker-entrypoint.sh file to accommodate that(KONG_LUA_PATH_OVERRIDE):
#!/bin/sh
set -e

# Disabling nginx daemon mode
export KONG_NGINX_DAEMON="off"

# Setting default prefix (override any existing variable)
export KONG_PREFIX="/usr/local/kong"

export KONG_LUA_PATH_OVERRIDE="/usr/local/kong/luarocks/share/lua/5.1/kong"

# Prepare Kong prefix
if [ "$1" = "/usr/local/kong/openresty/nginx/sbin/nginx" ];
	kong prepare -p "/usr/local/kong"
#fi

exec "$@"

Maybe export KONG_LUA_PATH_OVERRIDE="/usr/local/kong/luarocks/share/lua/5.1/kong" is not correct or maybe since I only set that at build time. Hope that gives a glimpse into the outside looking in view of struggles around building from src and pulling in Kong, should have it figured out tomorrow.

Figured out a way to fix it, LUA_PATH helps out with it, random googling of old kong issue solved that.

Well could not find documentation directly for what was wrong, but by setting the LUA_PATH to where the luarocks drops everything in this openresty build suit it started working again, managed to run migrations up from 1.2 to 1.3 successfully(seemingly), need to run a finish once I get 1.3 nodes setup.

Fixed as far as original issue at this point(as long as migrations finish goes smoothly heh), kong tls module somehow does start making it in when using yalls shell script logic.

Figured I would add some of the logic we used, if anyone else public net runs into problems building from source, hope maybe one of these bits will help:
######------ KONG BUILD TOOLS APPROACH ------######

ENV LUAROCKS_KONG_VERSION=1.3.0-0
ENV RESTY_VERSION=<IP_ADDRESS>
ENV RESTY_OPENSSL_VERSION=1.1.1c
ENV RESTY_LUAROCKS_VERSION=3.1.3
ENV PCRE_VERSION=8.43
ENV KONG_PATH=/usr/local/kong

RUN apk add --no-cache --virtual build-deps wget tar  ca-certificates \
        && apk add  --update --no-cache readline-dev outils-md5 linux-headers bsd-compat-headers m4 \
	yaml-dev build-deps bash bash-completion automake bash-doc make gcc g++ libgcc zlib-dev curl perl pcre pcre-dev unzip tzdata wrk luarocks git

RUN git clone https://github.com/kong/openresty-build-tools
RUN chmod -R 777 /openresty-build-tools
RUN cd openresty-build-tools \
    && mkdir work \
    && ./kong-ngx-build -p $KONG_PATH \
        --openresty $RESTY_VERSION \
        --openssl $RESTY_OPENSSL_VERSION \
        --luarocks $RESTY_LUAROCKS_VERSION \
        --pcre $PCRE_VERSION \
	--force

#Symlink to luarocks executable and nginx executable
RUN ln -s $KONG_PATH/luarocks/bin/luarocks /usr/bin/luarocks
RUN ln -s $KONG_PATH/openresty/nginx/sbin/nginx /usr/bin/nginx

#Needed for luarocks dependency builds to force https vs ssh as most corp block git:// proto
RUN git config --global url.https://github.com/.insteadOf git://github.com/

#Install Kong via luarocks
RUN luarocks install kong $LUAROCKS_KONG_VERSION OPENSSL_DIR=/openresty-build-tools/work/openssl-$RESTY_OPENSSL_VERSION OPENSSL_LIBDIR=/openresty-build-tools/work/openssl-$RESTY_OPENSSL_VERSION CRYPTO_DIR=$KONG_PATH/openssl

#Get the Kong executable resty script
RUN wget https://raw.githubusercontent.com/Kong/kong/master/bin/kong -P /usr/bin
RUN chmod 777 /usr/bin/kong
RUN mv $KONG_PATH/openresty/bin/resty /usr/bin

######------ END KONG BUILD TOOLS APPROACH ------######

Another Important point we got stuck on is to set the lua_path, build from source docs lacked info on that env:
export LUA_PATH="${luarocks-prefix}/share/lua/5.1/?.lua;;"

Had to change something recently too, thought to updatte thread here but pcre having issues with the <IP_ADDRESS> of openresty causeing the use of ngx/re problems finding some ffi binding. Looks like thibaultcha added a way to dynamically link so I change modified the build to this:
ENV LUAROCKS_KONG_VERSION=1.3.0-0
ENV RESTY_VERSION=<IP_ADDRESS>
ENV RESTY_OPENSSL_VERSION=1.1.1c
ENV RESTY_LUAROCKS_VERSION=3.1.3
ENV PCRE_VERSION=8.43
ENV KONG_PATH=/usr/local/kong

RUN apk add --no-cache --virtual build-deps wget tar  ca-certificates \
        && apk add  --update --no-cache readline-dev outils-md5 linux-headers bsd-compat-headers m4 \
	yaml-dev build-deps patch bash bash-completion automake bash-doc make gcc g++ libgcc zlib-dev curl perl pcre pcre-dev unzip tzdata wrk luarocks git

RUN git clone https://github.com/kong/openresty-build-tools
RUN chmod -R 777 /openresty-build-tools
RUN cd openresty-build-tools \
    && mkdir work \
    && ./kong-ngx-build -p $KONG_PATH \
        --openresty $RESTY_VERSION \
        --openssl $RESTY_OPENSSL_VERSION \
        --luarocks $RESTY_LUAROCKS_VERSION \
        --pcre $PCRE_VERSION \
	--force

Another thing that was interesting was the latest alpine 3.10 i had to install patch explicitly because the patch alpine came with wasn't fully featured and one of the CVE patches failed due to unknown operator "forward" .


url prefix is required to start with /

System Info
Infinity verion: 0.0.63
Output logs:
AssertionError: Routed paths must start with '/'

Fix:
use /v1 as prefix and the server starts as intended.
Possible solutions:

Do an input check on the input and update the help text
Prepend / to the input if not provided

Information

[ ] Docker
[X] The CLI directly via pip

Tasks

[X] An officially supported command
[ ] My own modifications

Reproduction
poetry install -E all
poetry run infinity_emb v2 --url-prefix v1

Expected behavior
Server starts with v1 prefix

Good catch


Added namespaces and folders

This library is awesome but lack of namespace, so I wanted to provide them also make some PSR code styling and fixed Composer autoload

Please if you'll accept this PR please tag version.

Seems reasonable.


Unexpected inconsistencies in intersected never overloaded functions

TypeScript Version:  2.1.4
Code
namespace SocketIO {
	export interface Socket {
		on(event: string, fn: Function): void;
		emit(event: string, fn: Function): void;
	}
}

interface X extends SocketIO.Socket {
	on(event: never, fn: Function): void;
}

interface Y extends SocketIO.Socket {
	emit(event: never, fn: Function): void;
}

type Z = X & Y;

var x: X;

x.on('bar', () => { }); // error
x.emit('baz', () => { });

var y: Y;

y.on('bar', () => { });
y.emit('baz', () => { }); // error

var z: Z;

z.on('bar', () => { }); // expected to throw compiler error
z.emit('baz', () => { }); // expected to throw compiler error

Expected behavior:
I was thoroughly confused when these two extended types that each overload the on and emit functions respectively with a never overload are allowed in type Z. I would expect Z intersection to maintain both compiler errors seen in X and Y both.
Actual behavior:
Neither calls to Z are marked as compiler errors.
Cause
I believe the reason this is happening is due to the intersection causing a different overwriting of declarations.
I imagine X is merged like so:
on(event: never, fn: Function): void;
on(event: string, fn: Function): void;

Since never overload is merged above string, never is ordered first and it works.
Y is similar:
emit(event: never, fn: Function): void;
emit(event: string, fn: Function): void;

So, then when X & Y are intersected something happens to the order of declarations that I'm confused about or is incorrect. Conceptually, to me it seems like if never overloads string in both instances, intersecting them both should work because precedence has been set. But I could easily be missing something.
For example, this works because the base interfaces overloads the SocketIO overloads on both merged X and Y types:
namespace SocketIO {
	export interface Socket {
		on(event: string, fn: Function): void;
		emit(event: string, fn: Function): void;
	}
}

// Base interface that blocks original overloads
interface W extends SocketIO.Socket {
	on(event: never, fn: Function): void;
	emit(event: never, fn: Function): void;
}

interface X extends W {
  // X-specific methods
}

interface Y extends W {
  // Y-specific methods
}

type Z = X & Y;

var x: X;

x.on('bar', () => { }); // error
x.emit('baz', () => { }); // error

var y: Y;

y.on('bar', () => { }); // error
y.emit('baz', () => { }); // error

var z: Z;

z.on('bar', () => { }); // error
z.emit('baz', () => { }); // error

My original use case is whitelisting the event overloads on a SocketIO interface, so that I can ensure the API is used correctly throughout the application--the never type was a clever way to accomplish this. However, I needed to split up the single interface into an Emitter and Listener, both which I had overload their respective methods with a never type--but I discovered once I intersected both those types, it didn't work as a single interface did.
Is this expected from the compiler's POV? Because it definitely wasn't from my POV as a user.

Perhaps I am misunderstanding what your are trying to accomplish, but I do not believe that overload order is responsible. I think the reason is that the intersection contains an overload that matches and so it picks that ones and is satisfied.

This is exactly as expected: If it's legal to do something to something of type X, then it's supposed to be legal to do that same thing to something of type X & Y. Both the on and emit calls were shown to be legal for X and Y respectively, so both calls succeeding on Z is the intended behavior.


Change colors used to highlight commiter names?

Simply put, I'd like to be able to customize the colors that represent various commiters in the Commit window.

Are these using highlight groups or are these hardcoded somewhere? It definitely clashes with my colorscheme.

You can change colors for lazygit using their official colorscheme or configuration files. Does that help?

Ah, so the highlighting for that specific element is handled in the config file and not by a highlight group?

Yes, lazygit is opened in a floating terminal window, and everything you see is either lazygit or neovim configuration.


Proposal Template Fork URL Not properly structured

When trying to fork an existing template, it sends you to the Opps page. Looking closing at the URL. it appears the the url isn't being constructed properly where as a second query is seperated by a &.
following the ?dao="" query it is currently "~?templateHash=""versus~&templateHash=""`

Fixed via #1527


feat(GridEngine): charAdded subject and public characterShifted obs

Adds an observable emitting when either a new character was added or an existing deleted
Related issue https://github.com/Annoraaq/grid-engine/issues/265

I wasn't sure whether I should modify your existing charRemoved subject and instead opted for adding a new one for the addition of new characters.
Also naming wise I was going against characterModified, as that might imply updates as well.

i'm not sure those are actually needed cause they should in theory be handled by the default export. I'll try  the branch out tomorrow and if it doesnt work I'll add em. 👍

yep I totally forgot CharacterShiftAction became a string enum, which is compiled to a dictionary, and as such needs dedicated re-exports in contrast to the previous union string type. I guess you're preferring string-enums for JS consumers so they have a point.
What I really do wonder though is if there is a way to automate, or at least warn, if a new string enum is introduced, that it should be exported in main-esm as well. Going for a custom ts-eslint rule sounds like quite an overkill :)

I do prefer enums over union string types because you can iterate over them and also refactoring is easier if you want to rename something or find all usages.
For JS consumers it should not matter because they have no static type safety anyway. I mean they theoretically COULD use an exported enum, because it gets transpiled into a JS object, but that won't help them much.
I think I didn't understand the part that string enums need automatic export and union types do not. Do you refer only to JS users? The reason I had in mind for exporting is that TS users can use the enums. For example when calling the move method they can call move(Direction.LEFT) instead of move('left')`. Following that idea I think that everything that is exported from the GridEngine.ts file should also be exported in main-esm.ts.

if you use Direction.Left its the same as var Direction = { Left: "Left" } so you need to import said construct as its actual transpiled JS code, no matter if JS or TS consumer. JS intellisense on a dictionary though is far better than nothing, thats for granted ;)
As for iterability,
const directions = [ "Left", "Right", ...] as const;
type Directions = typeof directions[number];

brings the same benefits of dynamically constructed strung unions.
Pure TS types that transpile into nothing, do not need to be explicitely exported innthe esm as they are available in the d.ts file and would result inside the ESM, well in nothing ;) String enums, aka Objects after transpile on the other hand need to be there

Yes it is a dictionary eventually at runtime. If you import it you can also use it as a type for a TS consumer:
const dir: Direction = ge.getFacingDirection();

My assumption (for some reason that I don't remember anymore) was that it was not possible to use it if it is not explicitly exported in the esm file (independent of whether it is an enum or let's say an interface). But when I was testing it before, both were correctly exported (enum and other types). But maybe my test was screwed up?
So if we don't export it in main-esm.ts, would there be a consequence for TS users or only for JS users?
Enums vs union types
The const assertion approach you suggest works for iteration. However, you have to define 2 things. And also you have to import 2 things (one for iteration and one for typing). That always felt clumpsy to me.
Considering the refactoring: When I have an example code like this:
const directions = [ "Left", "Right"] as const;
type Direction = typeof directions[number];|

const differentTypes = [ "Left", "Loft"] as const;
type DifferentType = typeof differentTypes[number];|

const testDir: Direction = "Left";
const test: DifferentType = "Left";

And now I want to rename Left of directions to Left2.
With VSCode I can't go and use the rename refactoring on "Left" of directions. So the only option I have is doing a search and replace, also accidentally replacing the "Left" of DifferentType.
Is that because the feature is not yet implemented in VSCode?

i cant see a reason why an interface would need to be additionally explicitely exported in the esm. I can access the type info and even do a type-import although there is no explicit definition. same applies to string literal types. Enum, both numeric and strings, on the other hand get transpiled and are needed, which I've replicated in my game, hence added the esm export as you mentioned but only for the string enum not CharacterShiftAction interface.
So if we dont export string enums, there are consequences for both consumers (TS, JS) in that they can only use pure strings Left but not e.g. Direction.Left
as for refactoring. tbh no idea, I'd have to give it a try. I typically only extend enums and rarely modify them, which is why the as const approach suites me well.

Alright I've tried it and the following is happening for refactor-rename:
// REFACTOR-RENAME LEFT doesn't work
const foo = ["LEFT", "LEFT2"] as const;
type Foos = typeof foo[number];

function test(a: Foos) {
  console.log(a);
}

test("LEFT");

// REFACTOR RENAME on A -> X works and the function call at the bottom is changed to X as well
type A = "A" | "B";
function test2(a: A) {
  console.log(a);
}
test2("A");

Ok I have tested it again. I think I now understood how it works and that is probably what you were trying to tell me.
The enum type is exported correctly without explicitly exporting it. So you could use it in a TS consumer in the following way:
const dir: Direction = gridEngine.getFacingDirection();

However, you could not do
const dir: Direction = Direction.LEFT;

It makes sense because Direction.LEFT needs to have a Direction JS object at runtime, which is not the case without explicitly exporting it because it is not part of the transpiled code.
That means that I can remove all the interfaces from the esm file.
So I think when it comes to the question whether we should use enums or union types (with const assertions) the pros and cons are mainly refactoring vs explicit exports. (For the refactoring I assume it means in VSCode so far it works for union types but not for const assertions yet.)
We have to stick with enums until the next major release anyway to stay backward compatible. And since the enums are already used in the codebase I think it is easiest to just stick with them for now. To tackle the explicit export problem I think I found a solution. We could just use export * from "./GridEngine"; in main-esm.ts. WDYT?

The enum type is exported correctly without explicitly exporting it. So you could use it in a TS consumer in the following way:
const dir: Direction = gridEngine.getFacingDirection();

However, you could not do
const dir: Direction = Direction.LEFT;

Exactly

That means that I can remove all the interfaces from the esm file.

Yep

So I think when it comes to the question whether we should use enums or union types (with const assertions) the pros and cons are mainly refactoring vs explicit exports. (For the refactoring I assume it means in VSCode so far it works for union types but not for const assertions yet.)

From the consumer perspective it's whether you can assign just a string in type-safe way (string unions) or need to import the string enum and then use it. So the import statement is the difference for the consumer.

We have to stick with enums until the next major release anyway to stay backward compatible. And since the enums are already used in the codebase I think it is easiest to just stick with them for now.

Whichever way tbh is totally fine for me. I've just started this conversation so that I know how to proceed for future PRs, so I'll stick to string enums as well.

To tackle the explicit export problem I think I found a solution. We could just use export * from "./GridEngine"; in main-esm.ts. WDYT?

Yep, that works. At least as long as everything is re-exported from GridEngine.ts, which might be a gotcha, but seems to be already the way you're handling things. Also hiding exports only from ESM doesn't make sense either so the * Export is absolutely ok.

Btw, sorry I've started this convo in the PR, definitely not the best place for it ;)
Where would you see questions like those fit? A new issue? A GitHub discussion?

From the consumer perspective it's whether you can assign just a string in type-safe way (string unions) or need to import the string enum and then use it. So the import statement is the difference for the consumer.

Oh yes. Good point. I think with version 3 of GridEngine we should reconsider if using enums is still a good choice.

Btw, sorry I've started this convo in the PR, definitely not the best place for it ;)
Where would you see questions like those fit? A new issue? A GitHub discussion?

No worries. Where to place it depends a bit on whether it is a pure question or a suggestion/concern. At least that is my opinion. However, I don't think it is a big difference.


Range#size returns an Union instead of an Int32

The following snippet:
range = 1_u8..5_u8
size = range.size
p! typeof(size)

returns

typeof(size) # => (Int32 | UInt8)

instead of the expected

typeof(size) # => Int32

Quoting PR comments to bring the general discussion back here:
@straight-shoota https://github.com/crystal-lang/crystal/pull/14588#issuecomment-2107541871

Hm, actually this can overflow on ranges of Int types that are bigger than Int32 🤔
Should we only cast to Int32 on smaller types?

@beta-ziliani https://github.com/crystal-lang/crystal/pull/14588#issuecomment-2130168028

I'm afraid there's no good solution here. The good part of raising on overflow is that if it fits, it works. The bad part is that the failure is at runtime...
An alternative, as good or bad, is to statically fail on > Int32 types and have people do the math themselves 🤷

That's also not good because (0_u128..1_u128).size is perfectly valid 🤷
I think casting to Int32 is fine. If you use bigger number types, it will raise but that's probably fine.
The type of #size for all collection types is Int32 because it's meant for collections that can be feasibly represented in memory.
We'll also need to increase that size type in order to support bigger collections. But while Int64 should be entirely sufficient for the use case related to actual memory storage, Range#size is not materialized and can reach arbitrary scales (with BigInt etc.).
#size is simply not a good fit for this.
I think we should explain that explicitly in the API docs and that's it. #size returns Int32 and if it doesn't fit, it will raise.
We should however consider adding an alternative method that unconditionally returns range.end - range.begin.
This would even be much more versatile because it would work not just with Int but any type that implements subtraction (e.g. 0_f32..1_f32, Time.utc..Time.utc).

Of course range.end - range.begin in general only works with an inclusive range. Perhaps we can have a special case for Int.
I don't follow what you mean in the rest of the comment about the type of 0 value. Why is that an issue?

If you do:
if bla
  0
else
  a - b
end

the result is the union of Int32 and whatever a and b are. This is why we get UInt8 | Int32. If we have a way to say "return 0, but casted to typeof(b)", then we drop the union and get the specific type.

Okay, gotcha. But the problem is not the union alone. Returning anything but Int32 breaks the contract of Enumerable#size : Int32.
So there's no point in converging the return type on UInt8. It needs to be Int32.
(having 0 typed identically to the difference should be possible though with something like typeof(a - b).new(0); it just doesn't help anything)

@beta-ziliani We can do typeof(n).zero, but since we fallback to Enumerable#size for other cases then the returned type is still an union. We can use indeed use macros and the returned type (when B and E are integers) will be typeof(B).
{% if B < Int && E < Int %}
  ...
  n < 0 ? typeof(n).zero : n
{% else %}
  ...
{% end %}

The return type of Range(B, E)#size still won't match Enumerable#size, and it will be any integer type that B happens to be. At the very least it won't be an Union, and we could cast it to be at least an Int32.
I think range.begin - range.end is basically #14740.

ah yes, when I wrote this comment I was aware of the problem with the interface in Enumerable#size and now I forgot 🙈  Sorry for the noise.

TBH: I'm not sure which solution is best. I think both choices are pros and cons; stable type but overflows vs unstable type but no overflow :shrug:

IMO there is no choice. The return type of any implementation of Enumerable#size must be Int32.
If we want a method to return anything else, it needs a different name.


Adding statistics to Raster

Resolves #602 GlacioHack/xdem#660
Description:
This pull request introduces enhancements to the statistics calculation methods in the project. The goal is to provide users with more flexible and comprehensive statistics functionality for raster data. The get_stats method has been created to:

Return a list of statistics in a dict containing mean, median, max, mean, sum, sum of squares, 90th percentile, nmad, rmse, std.
Support single statistic: User can request one statistic.
Support multiple statistics: Users can request multiple statistics in one call.
Support custom callables: Users can pass any callable function to calculate custom statistics.

A list of aliases has been drawn up in order to be as flexible as possible when users request statistics.
Changes:

NMAD in geoutils.stats
note : nmad can't be called directly from a raster because of circular imports, you have to go through an array like.
_statistics method: Calculates common statistics (mean, median, max, min, sum, RMSE, etc.) for a specified band of raster data.
get_stats method: Retrieves requested statistics by their name (either from a predefined list or user-defined), with support for passing a callable function.
_get_single_stat method: Handles fetching a single statistic by matching it with its alias or function name. Prints a warning and return NaN if the statistic required is not in the list
Support for callable statistics: Users can now provide any custom function that reduces the raster data (such as custom percentiles).

Usage Examples:

Get all statistics:
stats = raster.get_stats()

Get a single statistic (e.g., 'mean'):
mean_value = raster.get_stats("mean")

Get multiple statistics:
selected_stats = raster.get_stats(["mean", "max", "rmse"])

Using a custom callable statistic:
def custom_stat(data):
    return np.nansum(data > 100)  # Count the number of pixels above 100
custom_value = raster.get_stats(custom_stat)

Tests
The following tests ensure that the new functionality in the get_stats method works as expected:

Full Statistics:

The test retrieves all available statistics (mean, median, max, min, sum, etc.) for a raster and ensures each statistic is present and not None.

Single Statistic:

A test is included to fetch a single statistic, "Average" (which is an alias for "Mean"), and verifies that the returned value is a float.

Selected Statistics:

The test checks the retrieval of a selected set of statistics: mean, maximum, and std. Additionally, it includes the use of a custom statistic function (percentile_95), which computes the 95th percentile of the data.
This validates that both built-in and user-defined statistics can be fetched successfully.

Unrecognized Statistics:

Ensure that get_stats return NaN and a warning is raised if the statistic requested by the users is not in the list of aliases

NMAD tests in test_stats:

Ensure the good behavior of the scale factor
Check if the returned nmad is the same than scipy.stats.median_abs_deviation

Amazing! :smile:
I only have small comments on structure and documenting the alias names above.
Also had to approve the test run (as this is your first PR on this repo, I think), all passing!

@adebardo I had a section about get_stats in the raster class documentation.
I did not manage to get it running on xdem yet, I am not sure I can without geoutils be released with the update because of circular import conflicts.

@vschaffn To coordinate the changes between GeoUtils and xDEM, I typically do a pip install -e . in geoutils/ after installing xdem-dev with mamba. Then, if you are on the right branches for both packages locally, you should be able to run everything (tests, doc) with both versions where you want them to be.
If you want to run the tests/docs on xDEM's CI too, you can uncomment this line here in your xDEM PR: https://github.com/GlacioHack/xdem/blob/29a8cf4c8979d51723fe8b5a5a86e1fd5ba7cb96/environment.yml#L23. And potentially  specify the GeoUtils branch at the end of the github link following https://pip.pypa.io/en/stable/topics/vcs-support/ :wink:
This is especially useful if you know something might be behaving differently in CI than locally or is exclusive to CI, for example testing a dev change of the CI itself (otherwise can usually wait for GeoUtils release)


Allow nodes to dynamically join a running cluster

We want to allow nodes to dynamically join a kubernetes cluster after the cluster has been created. For version 1.0, we will implement "Static Clustering" as defined in clustering.md.
For reference, in #3168 (securing node → master communication), @roberthbailey proposes an implementation plan, step 2 of which entails this.

cc-ing @jszczepkowski who is working on autoscaling

I'm currently working on updating external load balancers (target pools) for external services when cloud nodes are changed. I'm just adding an extra call in NodeController.SyncCloudNodes method to reconcile external services. I think this is a part of this issue.

@mbforbes @roberthbailey @tkulczynski
Issue #5246 (Node Pod's IP ranges configured by nodecontroller) also seems to be a part of this issue.

Yes, definitely, thanks for linking here. #5246 is hopefully the last issue needed to make this possible; afterwards, @roberthbailey will be making it explicit and secure.

This is theoretically complete and upgrades are unblocked, though this needs to be verified.

I just created a GCE cluster, resized my MIG to add 2 new nodes, and then watched the nodes go from Unknown to Ready and verified that I could schedule work on the new nodes.

Yay, that's awesome. The other thing to do here would be to document this procedure. Should we leave this open or file a new bug to track that?

Well, right now this only really works on GCE. Maybe we should leave it open until #6949 goes in (which is a more general solution).

+1 for both documentation and leaving this open until #6949 is ready. Awesome work on this, very excited to try it out.

seconded.
+1 to leave open + docs.

#6949 is now merged (and the rollback of the rollback is also merged) so I think all that remains here is to add the appropriate documentation.

I think now that PR #8982 is merged (thanks @erictune !) we now have the doc fixes -- closing.


Added RCHLO and MIDWAY new adopters.

Hey, I just made a Pull Request!

:heavy_check_mark: Checklist

[ ] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[ ] All your commits have a Signed-off-by line in the message. (more info)

Welcome and thanks for adding yourselves to the adopters file! I noticed that non of the two websites work and I'm presented with this error message 😱


Dashes in included file names fail the scan

I have the following template:
{{> stored-credit-card-4321}}

However that causes the scanning of the mustache files to fail with an error.
(HummingbirdMustache.HBMustacheLibrary.ParserError error 1.)
Printing description of error:
▿ ParserError
  - filename : "tokens.json"
  ▿ context : Context
    - line : "      {{> stored-credit-card-4321}},"
    - lineNumber : 13
    - columnNumber : 17
  - error : HummingbirdMustache.HBMustacheTemplate.Error.unfinishedName

I've been using dashes in a number of templates up until now and they scan just fine. If I change the name of the file to storedCreditCard4321.mustache the scanning is fine so it appear that using a dash in the name of the file trips things up.

Ah that's probably just an over sight. I'll check it out

Should be fixed in v1.0.2


Different camera poses between dataset and blender

Hi, thanks for your amazing work!
When I try to run inference, I have some problems in pre_render. According to the code in "uncond.py", images in "gt" are rendered by the RayTracer module, while images in "depth", "normal" and "lights" are rendered by blender. It seems that "gt" have the different camera poses with the others. Is there any bug in c2w computing?
These are some examples rendered from the cat.obj given in run_examples.sh. The first one is in the "gt" and the second in the "normal". I think the "gt" is the right one.

Got it! The problem is that I used bpy==3.6, since I cound not install bpy==3.2 by pip. If I use the local blender 3.2.2 as the writer said, images rendered by blender are correct. This is the new normal aligned with gt:


packaging: RPM .spec cleanup

This pull request cleans up several small things in the RPM spec file.
It removes extraneous variables and unneeded RPM tags, adds a description for the package, and removes older RPM conventions that are no longer relevant on RHEL 6 and newer.

Can one of the admins OK this pull request so I can run a build?

OK to test


Too few arguments to function Illuminate\Console\GeneratorCommand::__construct(), 0 passed

Versions:

ide-helper Version: 2.10.0
Laravel Version: 8.75.0
PHP Version:7.4.26
jq/cray Version: 3.2.0-beta3

Description:
I'm the maintainer of a package called Cray.
When this package is installed, ide-helper would throw this error:
Too few arguments to function Illuminate\Console\GeneratorCommand::__construct(), 0 passed in /var/www/uot-web.test/vendor/jq/cray/src/CrayServiceProvider.php on line 82 and exactly 1 expected

Since Cray extends its own version of GeneratorCommand which in turn extends \Illuminate\Console\GeneratorCommand.
The tool works well otherwise, but when I run php artisan ide-helper:generate, it throws this exception.
I've opened an issue  on my project as well.
Steps To Reproduce:

Install ide-helper
Install jq/cray
Run

php artisan ide-helper:generate

@JunaidQadirB your instantiation of your Cray command is wrong
        // Register the main class to use with the facade
        $this->app->singleton('cray', function () {
            return new Cray();
        });

it aborts on the line new Cray(), Cray extends from \JunaidQadirB\Cray\Console\Contracts\GeneratorCommand which extends from \Illuminate\Console\GeneratorCommand which has this constructor definition:
    public function __construct(Filesystem $files)
    {
        parent::__construct();

        if (in_array(CreatesMatchingTest::class, class_uses_recursive($this))) {
            $this->addTestOptions();
        }

        $this->files = $files;
    }

I.e. it requires the Filesystem argument.
The reason why ide-helper trigger this is because the generate command introspects the Facades and it exposes that your Facade does not work due to the incorrect definition in your service provider.
Couple of roads leading to Rome:

properly pass the required argument:        // Register the main class to use with the facade
        $this->app->singleton('cray', function (Container $container) {
            return new Cray($container->get('files'));
        });

don't use the 'cray' alias at all: in your Facade just reference the Cray::class    protected static function getFacadeAccessor()
    {
        return \JunaidQadirB\Cray\Console\Commands\Cray::class;
    }

and remove the ->singleton() part entirely.
This leaves resolving the class to Laravel itself and it will correctly auto-inject the expected argument.

HTH!

@mfn Thanks a lot for your time to explain in detail the issue at my end. 🙏


Convert iteration based STF load test playlist targets to time-based

Convert iteration based STF load tests to time-based tests.
Related to https://github.com/AdoptOpenJDK/openjdk-tests/issues/2104

Note: The following System tests have not been converted to time-based in this pass, as they are not really load tests:

LockingLoadTest (Not really a load test; it runs a minimix load and runs lock contention tests).
HCRLateAttachWorkload (Not a load test It launches process 'A' with a mini-mix load and then runs process 'B' that attaches an agent to 'A' and runs various unit tests repeatedly).
JDITest (Not a load test; it starts and monitors various relevant attaching or listening processes).
All tests under jlm (Not really load tests. They do stuff like launching sub-process A with a load, while launching sub-process B that issues JLM based commands to test JLM API).
All tests under sharedClasses (Not really load tests. These tests run a mini-mix load in a sub-process and runs various sharedclasses chache verification tests, softmx tests, etc).
All tests under modularity (not really load tests; these tests are mostly for unit-testing various modularity features).

FYI @pshipton
Signed-off-by<EMAIL_ADDRESS><EMAIL_ADDRESS>

Tests:

extended.system on Windows: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6063/

sanity.system on Windows: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6064/

extended.system on x64 Linux: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6060/

sanity.system on x64 Linux: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6061/

Tests:

extended.system on Windows: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6063/

sanity.system on Windows: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6064/

extended.system on x64 Linux: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6060/

sanity.system on x64 Linux: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6061/

This PR is to be delivered After the following 3 PRs have been merged:

https://github.com/AdoptOpenJDK/openjdk-systemtest/pull/396
https://github.com/AdoptOpenJDK/stf/pull/100
https://github.com/eclipse/openj9-systemtest/pull/125

This PR is to be delivered After the following 3 PRs have been merged:

https://github.com/AdoptOpenJDK/openjdk-systemtest/pull/396
https://github.com/AdoptOpenJDK/stf/pull/100
https://github.com/eclipse/openj9-systemtest/pull/125

We have the path limitation issue on windows. Does it make sense to assume the default is 5mins and avoid adding _5min into every test name?

We have the path limitation issue on windows. Does it make sense to assume the default is 5mins and avoid adding _5min into every test name?

re: https://github.com/AdoptOpenJDK/openjdk-tests/pull/2203#issuecomment-766370942 when someone wants to add a fleet of test targets for 30min or 2hrs you won't have a symmetric naming strategy (which may be okay, but then automation to parse names for some other purpose must know the default case).  Just want to mention it now, as I expect to see some longer running targets showing up in the special level at some point.

re: https://github.com/AdoptOpenJDK/openjdk-tests/pull/2203#issuecomment-766370942 when someone wants to add a fleet of test targets for 30min or 2hrs you won't have a symmetric naming strategy (which may be okay, but then automation to parse names for some other purpose must know the default case).  Just want to mention it now, as I expect to see some longer running targets showing up in the special level at some point.

I agree with Shelley, we would most probably be introducing some other time variants of these tests in future.
About the path limit, please note that this PR does contain a few test name updates with that in mind as well - I've converted "OpenJ9" to "J9", "Hotspot" to "HS", ConcurrentScavenge" to "CS", etc. in test names in this PR.

I agree with Shelley, we would most probably be introducing some other time variants of these tests in future.
About the path limit, please note that this PR does contain a few test name updates with that in mind as well - I've converted "OpenJ9" to "J9", "Hotspot" to "HS", ConcurrentScavenge" to "CS", etc. in test names in this PR.

Thanks @ShelleyLambert . That is a good point. We may want to use other programs to parse and triage it.
In this case, instead of adding the time in the middle of the test name, can we have it at the end? For example, DaaLoadTest_daa1_5m, DaaLoadTest_daa1_2h, DaaLoadTest_daa1_30m

Thanks @ShelleyLambert . That is a good point. We may want to use other programs to parse and triage it.
In this case, instead of adding the time in the middle of the test name, can we have it at the end? For example, DaaLoadTest_daa1_5m, DaaLoadTest_daa1_2h, DaaLoadTest_daa1_30m

can we have it at the end and make it shorter?

That's a great suggestion! Done!

can we have it at the end and make it shorter?

That's a great suggestion! Done!

From the Grinder result, this PR will increase the system tests execution time. For example, https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_sanity.system_x86-64_linux/ takes ~2hrs and the above Grinder took ~3hrs.
We will monitor the test builds and tune the tests if needed.

From the Grinder result, this PR will increase the system tests execution time. For example, https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_sanity.system_x86-64_linux/ takes ~2hrs and the above Grinder took ~3hrs.
We will monitor the test builds and tune the tests if needed.

We could also increase parallelization from 2 to 3.

We could also increase parallelization from 2 to 3.


Can't load aergia

index.js is in wrong location and needs to be moved to root dir. Also update references in package.json and webpack.onfg.js.

Branch: cant-load-aergia


jupyterlab_sparkmonitor is not a valid npm package errore

While trying to load jupyterlab extension manager I'm getting this error (*)
Do you have any clue?
 Updating nodejs npm helped fixed this issue for me in the past. The following thread may help:

Please try again with version 3.0.1 and report back if this issue persists


Fix slowdowns when navigating

Continued from https://github.com/fsprojects/Fabulous/issues/373#issuecomment-480486017
I just tried the AllControls sample on my Android device (release mode) and WPF on my beefy desktop computer. There are fairly noticeable slowdowns when navigating on Android, and less so (sub-second, but still noticeable) on WPF. After a cold start on Android, navigating to TabbedPage 3 takes around 2 seconds. I have a hard time imagining clients/employers (or users) accepting that, regardless of maintenance/agility benefits of an Elmish architecture.

What is the root cause of the navigation slowdown, particularly on Android?
Which factors influence it? (Devices? Model size? etc.)
Can anything be done about it? What?

I tried profiling using AllControls.WPF in VS. According to "Application Timeline", the majority of the time is spent in Application Code (as opposed to rendering). Unfortunately I couldn't get "Instrumentation" to work, so I can't see which parts of Fabulous are hotspots.

@cmeeren #241 was open for this problem as well.
I suspect Fabulous spends to much time browsing the whole UI tree to determine whether or not create new controls.
This is especially bad for low end devices.
I think some simple fixes should give big boost in the first time.
We have yet to profile it.

I think some simple fixes should give big boost in the first time.

Would be great if there was some low-hanging fruit.
For inspiration, check out the React docs on Reconciliation

Regarding profiling, if you need R# you can probably apply for a free OS license.

I suspect Fabulous spends to much time browsing the whole UI tree to determine whether or not create new controls.

I have taken a (very) brief look at some of the updating code in ViewConverters.fs. I might have overlooked something, or not looked carefully enough, but initially it seems to me that Fabulous follows the React heuristics I linked to: It re-uses the control if it's the same type, or skips any reconciliation and reconstructs the subtree if it's another type.
Does this match your understanding of Fabulous's reconciliation? Are there any places you know or suspect there might be fairly low-hanging performance improvements?

Just want to say that I'd like to help if there's anything I can do. I'm not here to just complain. 😊 But as I explained in the comment above, I can't immediately see something obviously wrong (which would be surprising, given the Fabulous authors).
Do let me know if there's anything I can do, places I can look, things I can look for, etc. I really want to use Fabulous but need better performance, and can't figure out how to scratch this itch. 😛

Note that the AllControls app has not been performance optimized, i.e. there are relatively few dependsOn calls which are pretty much necessary to get good performance for incremental view update.
A more general question is how we might best guide the user as to the best place to insert dependsOn calls.
For example, could there be a mode (or Debug mode) that reports when large parts of the View tree re being re-created (as objects) but are in fact identical (and perhaps even relate this to exact source line locations or stacks where the uppermost invariant ViewElements were created?)
Note also I don't think dependsOn is the only way to avoid recomputation of view sub-trees under invariant sub-models., it's just one relatively simple way.  I'd like to hear of other suggestions and hacks that do this.

Note that the AllControls app has not been performance optimized, i.e. there are relatively few dependsOn calls which are pretty much necessary to get good performance for incremental view update.

Does that explain the poor performance when navigating, but much better performance at the page after navigation?

A more general question is how we might best guide the user as to the best place to insert dependsOn calls.
For example, could there be a mode (or Debug mode) that reports when large parts of the View tree re being re-created (as objects) but are in fact identical (and perhaps even relate this to exact source line locations or stacks where the uppermost invariant ViewElements were created?)

I like the idea. The question is how to set the thresholds so the warnings are real and not spammy.
Also, I think clear documentation can get you far on its own. :)

Note also I don't think dependsOn is the only way to avoid recomputation of view sub-trees under invariant sub-models., it's just one relatively simple way. I'd like to hear of other suggestions and hacks that do this.

Some kind of Key property might be good for collections. It would allow you to do smarter differential update of collections. AFAIK it's an important optimization in React (they throw warnings at you if you don't supply it in collections). Not sure how easy it is though, given that you can add any attribute to any HTML DOM element, but here we are working with objects with a limited set of properties.

Just leaving this here: I've built a quite simple application which consists of basically one navigation page and one child page. Whenever there's another child page pushed from that, I can literally watch it freeze for maybe half a second. For no obvious reason, as I run only once through the app's message loop to trigger the navigation. I took Program.withConsoleTrace out of the equation, but with no noticeable effect.
That's on the iOS Simulator right now in DEBUG mode. Will check again as soon as I can get it deployed to some hardware device.
We even have a word for that in German: Gedenksekunde ;-).

Does reflection  slow down the performance?
For example,
1 model containers 10 models,
1 model container 10 properties,
Then every model comparision will compare 100 properties from reflection
It's a bit slow when get property from reflection first time,
So we will get a slowdowns when we navigate to a new page
May be we should done the comparision of models by pre-compiling
Some related work may be helpful:
https://github.com/MoiraeSoftware/myriad
https://github.com/Fody/

Does reflection slow down the performance?

Yes, reflection can slow down performance.
But you will only notice performance issue if you make intensive use of it.
Individually, it's not that bad.
That's why we tried to limit reflection to a minimum in Fabulous.XamarinForms.
The line you're referencing is one of the few (if not the only one) to use reflection.
So I don't think it's the source of the slowdowns...
Instead, I think we're a bit too heavy on the memory.
Every time I noticed slowdowns, a corresponding output in the console indicated the GC triggered.
GC pauses the app for several milliseconds, enough to make it noticeable sometimes.
So we either don't clean up enough memory, or we're instantiating too many small objects that clutter up the memory forcing the garbage collector to pass.
This seems to fit because iOS is not plagued by those slowdowns, while Android takes it fully.
On iOS, it's fully native no GC (only reference counting) - on Android, there's 2 of them (.NET GC and JVM GC)

As an aside, if/when we have to use reflection, its always worth considering if we can make use of Delegate.CreateDelegate for better performance - see Jon Skeet's article

Slowdowns were a result of Fabulous doing too many allocations on the heap, and GC triggering.
v2 fixes that


Fix LinkedIn social using incorrect default path and Update GitHub and Twitter to use canonical URL

Similar to how YouTube uses /channel, LinkedIn uses /in as a prefix for profiles.

@MrGreensWorkshop I also noticed that GitHub and Twitter were using www. which isn't the canonical url (they'll redirect to github.com and twitter.com respectively). All the other social links are correct however (they all use www.).

Related issues, #68 , #69,
Thank your for your contribution,
Next time, could you please create issue per PR? Thank you for your understanding.

Next time, could you please create issue per PR?

Out of curiosity, why create a redundant issue for the PR? It seems like extra work for not much gain, but it could be I'm missing something.

Yes, there are cases where there is extra work, but at some points it is necessary. For those who are new to the project, it is more understandable to look at the issue rather than PR's. Sometimes I wondered why we did such a thing after the changes made in PR. I'm in favor of managing the problem and solution separately. If you look at my previous PR's, you can  see what I suggest in each PR for each issue there. That's why I added those issue and PR templates.
I know it's not needed in situations like your PR's. But if there is no rule, people will prefer not to file any issues. I hope I could explain what I mean.