Failure to build BFLOAT16 functions on Windows

Hello,
It appears the pre-built binaries do not contain the BFLOAT16 functions declared in cblas.h (Windows 64bit)
Also, when I try to build on Windows, carefully following the instructions (https://github.com/xianyi/OpenBLAS/wiki/How-to-use-OpenBLAS-in-Microsoft-Visual-Studio) the build fails.
So far, two types of error occur, both for CooperLake.
S:/code/BLAS/OpenBLAS/OpenBLAS-develop/kernel/x86_64/sbgemm_ncopy_16_cooperlake.c(138,2): error: unknown type name 'u_int64_t'; did you mean 'uint64_t'?
        u_int64_t permute_table2[] = {
        ^~~~~~~~~
        uint64_t
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\stdint.h(24,28): note: 'uint64_t' declared here

and the second manifests in a macro which appears to be using the AVX512 incorrectly for CooperLake. Hence, this error occurs many times.
C:/code/BLAS/OpenBLAS/OpenBLAS-develop/kernel/x86_64/sbgemm_kernel_16x4_cooperlake.c(297,4): error: Register k0 can't be used as write mask
                        MASK_STORE_4X(0, 0, 0); MASK_STORE_4X(0, 0, 1); MASK_STORE_4X(0, 0, 2);

This GitHub link seems to provide some relevant info:  https://github.com/golang/go/issues/29893
Would it be possible for you to investigate why Open BLAS fails to build for BFLOAT16 using the Windows build instructions?
I am currently using the following cmake line:
cmake .. -G "Ninja" -DCMAKE_CXX_COMPILER=clang-cl -DCMAKE_C_COMPILER=clang-cl -DCMAKE_Fortran_COMPILER=flang -DCMAKE_MT=mt -DBUILD_WITHOUT_LAPACK=yes -DNOFORTRAN=1 -DDYNAMIC_ARCH=ON -DBUILD_BFLOAT16=ON -DCMAKE_BUILD_TYPE=Release
I have also tried with the cmake line provided in the build instructions, the errors are the same.
I have tried using Visual Studio 2015 and also 2019 (deleting the build folder each time, and starting afresh), the errors are the same.
Any help or guidance you could provide would be very much appreciated.
The work you people do is truly impressive.
Thank you

First looks to be a "harmless" typo that will not show up on platforms that alias the two spelling variants (Linux, Mac, probably BSD as well), seems MSVC headers do not alias u_int64_t to uint64_t.
Second one I'm less sure how to handle - almost looks like an assembler error (picking k0 of all thing when the instruction only says k). Found something in the llvm-dev archive where the suggestion was to use "Yk" instead of k ( that would be in all statements of the _MASK_STORE_C_2nx16 macro starting from line 116, and in _MASK_STORE_C_16 from line 156).
Unfortunately I do not have Cooperlake hardware to test.

Many thanks for the excellent feedback Martin, it's very much appreciated.
I altered the file sbgemm_kernel_16x4_cooperlake.c with the fixes you provided. It compiled.
That allowed the compiler to continue past its limit of 20 errors to reveal more.
The next errors were in the file sbgemm_microk_cooperlake_template.c they all appear to be due to calling intrinsic shuffles, e.g. the function _mm512_shuffle_epi32() which expects a constant integer rather than an integer.
Here's the first error (the others are very similar):
C:/code/BLAS/OpenBLAS/OpenBLAS-develop/kernel/x86_64/sbgemm_microk_cooperlake_template.c(396,35): error: argument to '__builtin_ia32_pshufd512' must be a constant integer                 arrayB_512[i]   = _mm512_shuffle_epi32(arrayB_512[i], SHUFFLE_MAGIC_NO);
In the file the variable SHUFFLE_MAGIC_NO is declared as an integer in various functions.
I altered it to be defined as a constant integer thus:
#define  SHUFFLE_MAGIC_NO (const int)0x39
I then deleted the build folder and started again. This allowed the compilation to continue.
Unfortunately I then obtained link errors;
`[71/72] Linking C static library lib\Release\openblas.lib
[72/72] Linking C executable utest\openblas_utest.exe
FAILED: utest/openblas_utest.exe
cmd.exe /C "cd . && C:\Users\tim\anaconda3\Library\bin\cmake.exe -E vs_link_exe --intdir=utest\CMakeFiles\openblas_utest.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~1.0\VC\bin\amd64\link.exe /nologo utest\CMakeFiles\openblas_utest.dir\utest_main2.c.obj  /out:utest\openblas_utest.exe /implib:lib\Release\openblas_utest.lib /pdb:utest\openblas_utest.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console  lib\Release\openblas.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK: command "C:\PROGRA~2\MICROS~1.0\VC\bin\amd64\link.exe /nologo utest\CMakeFiles\openblas_utest.dir\utest_main2.c.obj /out:utest\openblas_utest.exe /implib:lib\Release\openblas_utest.lib /pdb:utest\openblas_utest.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console lib\Release\openblas.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:utest\openblas_utest.exe.manifest"
failed (exit code 1120) with the following output:
openblas.lib(setparam_PRESCOTT.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_PRESCOTT
openblas.lib(setparam_NEHALEM.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_NEHALEM
openblas.lib(setparam_CORE2.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_CORE2
openblas.lib(setparam_BARCELONA.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_BARCELONA
openblas.lib(setparam_SANDYBRIDGE.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_SANDYBRIDGE
openblas.lib(setparam_BULLDOZER.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_BULLDOZER
openblas.lib(setparam_PILEDRIVER.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_PILEDRIVER
openblas.lib(setparam_HASWELL.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_HASWELL
openblas.lib(setparam_STEAMROLLER.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_STEAMROLLER
openblas.lib(setparam_EXCAVATOR.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_EXCAVATOR
openblas.lib(setparam_ZEN.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_ZEN
openblas.lib(setparam_SKYLAKEX.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_SKYLAKEX
openblas.lib(setparam_COOPERLAKE.c.obj) : error LNK2019: unresolved external symbol sbgemm_r referenced in function init_parameter
utest\openblas_utest.exe : fatal error LNK1120: 13 unresolved externals
ninja: build stopped: subcommand failed.`
I did notice that you have added the directive "Fix Compilation for CooperLake on Windows/clang". I am not used to
the etiquette of how this forum/issues WiKi works. Should I stop providing feedback, or is this additional message useful?
Thank you again.

This is extremely useful - the issue ticket was automatically closed when I merged the fixes we had come up with yesterday. I will reopen it and try to find out where the missing references come from - possibly something in the cmake build files not matching what the gmake build does

Many thanks Martin, very much appreciated.

Bisected to #3364 , which apparently missed defining sbgemm_r for DYNAMIC_ARCH builds. The "const int" issue on the other hand appears to be a problem not seen with gcc.

Hi Martin, unfortunately the link errors are still present:
192 warnings generated.
[9354/9355] Linking C static library lib\Release\openblas.lib
[9355/9355] Linking C executable utest\openblas_utest.exe
FAILED: utest/openblas_utest.exe
cmd.exe /C "cd . && C:\Users\tim\anaconda3\Library\bin\cmake.exe -E vs_link_exe --intdir=utest\CMakeFiles\openblas_utest.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1429~1.300\bin\Hostx64\x64\link.exe /nologo utest\CMakeFiles\openblas_utest.dir\utest_main2.c.obj  /out:utest\openblas_utest.exe /implib:lib\Release\openblas_utest.lib /pdb:utest\openblas_utest.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console  lib\Release\openblas.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK: command "C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1429~1.300\bin\Hostx64\x64\link.exe /nologo utest\CMakeFiles\openblas_utest.dir\utest_main2.c.obj /out:utest\openblas_utest.exe /implib:lib\Release\openblas_utest.lib /pdb:utest\openblas_utest.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console lib\Release\openblas.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:utest\openblas_utest.exe.manifest" failed (exit code 1120) with the following output:
openblas.lib(setparam_PRESCOTT.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_PRESCOTT
openblas.lib(setparam_NEHALEM.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_NEHALEM
openblas.lib(setparam_CORE2.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_CORE2
openblas.lib(setparam_BARCELONA.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_BARCELONA
openblas.lib(setparam_SANDYBRIDGE.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_SANDYBRIDGE
openblas.lib(setparam_BULLDOZER.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_BULLDOZER
openblas.lib(setparam_PILEDRIVER.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_PILEDRIVER
openblas.lib(setparam_HASWELL.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_HASWELL
openblas.lib(setparam_STEAMROLLER.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_STEAMROLLER
openblas.lib(setparam_EXCAVATOR.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_EXCAVATOR
openblas.lib(setparam_ZEN.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_ZEN
openblas.lib(setparam_SKYLAKEX.c.obj) : error LNK2001: unresolved external symbol sbgemm_small_kernel_b0_tt_SKYLAKEX
utest\openblas_utest.exe : fatal error LNK1120: 12 unresolved externals
ninja: build stopped: subcommand failed.

I altered the kernel/setparam-ref.c as specified in https://github.com/xianyi/OpenBLAS/pull/3547/files, I added the following code after line 1826
#if BUILD_BFLOAT16==1
  TABLE_NAME.sbgemm_r = (((BUFFER_SIZE -
			       ((TABLE_NAME.sbgemm_p * TABLE_NAME.sbgemm_q *  4 + TABLE_NAME.offsetA
				 + TABLE_NAME.align) & ~TABLE_NAME.align)
			       ) / (TABLE_NAME.sbgemm_q *  4) - 15) & ~15);
#endif

I have previously altered the const integer(s) in kernel/x86_64/sbgemm_microk_cooperlake_template.c using
#define SHUFFLE_MAGIC_NO (const int)0x39
Initially I simply added the const in front of the int declaration. However, that still produced the compile errors.
I therefore used the compiler directive above to force the values to be constant integers. This removed the compiler errors and the openblas.lib was built.
I still get the link errors above. Please forgive me if I am being overly enthusiastic, and thank you for this tremendous help.

On a more detailed look at the link errors, it appears that one of the previous errors:
openblas.lib(setparam_COOPERLAKE.c.obj) : error LNK2019: unresolved external symbol sbgemm_r referenced in function init_parameter
Has been fixed. Which is not surprising since you added the code to deal with it...

Sorry, I am still working on the fix as I find time today besides my regular work commitments.

No need to be sorry Martin, I thank you very much for everything you have done/are doing. I really do appreciate it. Hope work goes well.

There's a stray dollar sign in line 681 of kernel/CMakeLists.txt that causes empty definitions for the sources of the various missing sbgemm_small_kernel_b0_tt functions. I've updated the PR accordingly

Wow, excellent, it builds without link errors now:
192 warnings generated.
[9366/9367] Linking C static library lib\Release\openblas.lib
[9367/9367] Linking C executable utest\openblas_utest.exe

Yipee, now I can run tests...I can't thank you enough Martin.

Thank you very much for confirming - my tests so far had been on Linux.

Hi Martin.
I have run some tests on the following Windows CPU's:
Alder Lake: 12th Gen Intel i7-12700K 3.61 GHz (approx. 15 times slower)
Westmere EP: Intel® Xeon® Processor X5670 (approx. 6 times slower)
Coffee Lake: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz   2.59 GHz (approx. 13 times slower)
Sandy Bridge: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz   3.40 GHz (approx. 5 times slower)
AMD Ryzen Threadripper 3970X 32-Core Processor 3.69 GHz (approx. 14 times slower)
Unfortunately using BFLOAT16 representation produces slower GEMM than single precision. I have put the approximate slow-down factor in brackets after the CPU in the above list.
I used very simply code (shown below, using a single thread only).
I hope this is useful, and thank you again for all the excellent work/help.
#include "windows.h"
#include <iostream>
#include <cblas.h>
#include <chrono>

using namespace std;
using namespace std::chrono;

////////////////////////////////////////////////////////////////////////////////
// high resolution timer class
class TestTimer
{
  protected:
    high_resolution_clock::time_point m_start; // saves time point in nanoseconds

  public:
    TestTimer(void)
    {
      m_start = high_resolution_clock::now();
    }

    void Reset(void) { m_start = high_resolution_clock::now(); };

    double nano(void)
    {
      high_resolution_clock::duration lapsed = high_resolution_clock::now() - m_start;
      return (double)lapsed.count();
    }

    double micro(void)
    {
      return nano() * 0.001;
    }

    float ms(void)
    {
      return (float)(micro() * 0.001);
    }

    float s(void)
    {
      return (float)(micro() * 0.000001);
    }

};
////////////////////////////////////////////////////////////////////////////////

/////////////////////////////////////////////////////////
void SetFloatArray(float* p, blasint n, float val)
{
  for (blasint i = 0; i < n; i++) p[i] = val;
}
/////////////////////////////////////////////////////////

/////////////////////////////////////////////////////////
void ToBFLOAT(const float* A, bfloat16* B, blasint n)
{
  cblas_sbstobf16(n, A, 1, B, 1);
}
/////////////////////////////////////////////////////////
void FromBFLOAT(float* A, const bfloat16* B, blasint n)
{
  cblas_sbf16tos(n, B, 1, A, 1);

}
/////////////////////////////////////////////////////////

/////////////////////////////////////////////////////////
void TestBLAS_bFLOAT()
{
  openblas_set_num_threads(1);

  blasint m = 128;
  blasint k = 27;
  blasint n = 64;

  blasint lda = k;
  blasint ldb = n;
  blasint ldc = n;

  // A matrix has (m, k) = (rows, cols)
  blasint A_size = m * k;
  float* A = (float*)malloc(A_size * sizeof(float)); SetFloatArray(A, A_size, 2.0f);

  // B matrix has (k, n) = (rows, cols)
  blasint B_size = k * n;
  float* B = (float*)malloc(B_size * sizeof(float)); SetFloatArray(B, B_size, 2.0f);

  // C matrix has (m, n) = (rows, cols)
  blasint C_size = m * n;
  float* C = (float*)malloc(C_size * sizeof(float)); SetFloatArray(C, C_size, 1.0f);

  // FBLOAT16 matrices
  bfloat16* Ab = (bfloat16*)malloc(A_size * sizeof(bfloat16));
  bfloat16* Bb = (bfloat16*)malloc(B_size * sizeof(bfloat16));
  ToBFLOAT(A, Ab, A_size);
  ToBFLOAT(B, Bb, B_size);

  TestTimer timer;
  double t = 0.0;
  blasint nn = 10000;
  blasint i;

  // do one first to allow lib to find CPU architecture
  cblas_sbgemm(CBLAS_ORDER::CblasRowMajor, CBLAS_TRANSPOSE::CblasNoTrans, CBLAS_TRANSPOSE::CblasNoTrans,
    m, n, k, 1.0f, Ab, lda,
    Bb, ldb,
    1.0f, C, ldc);

  // TEST half precision GEMM
  cout << endl;
  cout << " BFLOAT GEMM" << endl;
  t = 0.0;
  openblas_set_num_threads(1);
  for (i = 0; i < nn; i++)
  {
    SetFloatArray(C, C_size, 1.0f);

    timer.Reset();

    cblas_sbgemm(CBLAS_ORDER::CblasRowMajor, CBLAS_TRANSPOSE::CblasNoTrans, CBLAS_TRANSPOSE::CblasNoTrans,
      m, n, k, 1.0f, Ab, lda,
      Bb, ldb,
      1.0f, C, ldc);

    t += timer.nano();
  }
  t /= (double)nn;

  //cout << "C:" << C << endl;
  cout << "Av time for BFLOAT GEMM = " << t << " nano" << endl;

  // GEMM
  /*
    C := alpha*op(A)*op(B) + beta*C
    where:
      op(X) is one of op(X) = X, or op(X) = XT, or op(X) = XH,
      alpha and beta are scalars,
      A, B and C are matrices:
      op(A) is an m-by-k matrix,
      op(B) is a k-by-n matrix,
      C is an m-by-n matrix.
  */

  cout << endl;
  cout << " GEMM" << endl;
  openblas_set_num_threads(1);
  t = 0.0;
  for (i = 0; i < nn; i++)
  {
    SetFloatArray(C, C_size, 1.0f);

    timer.Reset();

    cblas_sgemm(CBLAS_ORDER::CblasRowMajor, CBLAS_TRANSPOSE::CblasNoTrans, CBLAS_TRANSPOSE::CblasNoTrans,
      m, n, k, 1.0f, A, lda,
      B, ldb,
      1.0f, C, ldc);

    t += timer.nano();
  }
  t /= (double)nn;

  //cout << "C:" << C << endl;
  cout << "Av time for GEMM = " << t << " nano" << endl;

  free(A); free(B); free(C); free(Ab); free(Bb);

}
/////////////////////////////////////////////////////////

/////////////////////////////////////////////////////////
int main()
{
  TestBLAS_bFLOAT();
}
/////////////////////////////////////////////////////////

Unfortunately this is to be expected on your range of non-AVX512 hardware - right now, optimized x86_64 SBGEMM kernels are only provided for Intel Cooper Lake and Sapphire Rapids cpus (which offer dedicated instruction set support). Minimal BF16 support for older cpus is provided by the "generic" 2x2 blocked SGEMM kernel with added conversions to/from regular float32, so guaranteed to be slower than the respective optimized SGEMM counterparts.
The only other architecture for which actual performant BFLOAT16 support is currently implemented is IBM POWER10.

Sorry for the delay Martin, many thanks for the feedback, much appreciated.

Yes, OpenBLAS should recognize it as Haswell. No advantage for BFLOAT16 though, if that is what you are after

Thanks Martin, much appreciated


playing freeze

google speech live subtitle can not smoothly work while playing movies, what is the problem

I've wrote about that in README
You should use VLC to play the video/audio, then you can use this app to display its SUBTITLES
If you want to watch directly from browser, you can try VOSK LIVE SUBTITLE
https://github.com/botbahlul/VOSK-Powered-Live-Subtitle-V3
It has CONTINUOS FULL DUPLEX SPEECH RECOGNIZER but ONLY SUPPORT 21 LANGUAGES.
Free Google Speech Recognition support 132 languages, but doesn't support full duplex speech recognition.


Memory usage increase

Hi Mr Bank,
I'm using PNGdec in my ESP-IDF project (ESP32-S3) with direct callback rendering.
With latest master code (a0f4b16) I noticed an increase of about 1.5 KiB in static memory consumption compared to commit 8abf6be (31 jan 2022).
Is that to be expected?
Thanks.

You can see why in the commit you referenced. I increased the line buffer to allow decoding of up to 480-wide 32-bit pixels. You're welcome to change your local copy to buffer fewer pixels if you're working with small images.


Feature/21-student centreexit

This PR adds all the CRUDs methods to the Student and CenterExit Models. It also includes some fixes for other issues.

As advice, in the studentSerializer, in the meta class, I would set it up as follows:
* fields: "**all**"

So that if there is any change in the model in the future, it will not be affected.

Thank you for the great recommendation! I have it taken into account in the last commit in this PR, and it would be great to implement it in all the models. I am very much looking forward to your review.

I tried to POST a Student with a Family but no EducationCenter and got the error:
{
    "education_center": [
        "This field may not be null."
    ]
}

Please fix this.

Now, it should be solved in the last commit. Thanks for noticing!


Work with ECMA-376 format files

Please, add ability to work with ECMA-376 files like .docx

Hello? Anybody here? 🕸

Hi Vasily,
This project is meant to provide the complement UI for the RMS SDK for iOS. It is not the repository for the actual RMS SDK. Currently we do not support Office files natively within the RMS SDK. It is up to the application to abstract out the RMS artifacts (license, rights etc) and embed it into those files.
If you are looking for ECMA-376 support as a new feature, or any other RMS related questions, please contact us at<EMAIL_ADDRESS>describing your needs.
Thanks
Shyam

Thank you! I'll write on email, that you provided.

Can you help me with another question (about sample app)?
Sample work fine when I pass simple files like pjpg or ptxt via "Open In", but whey I tryong open that file from apps Documents directory, method "+[MSProtectedData protectedDataWithProtectedFile:..." returns error "ERROR: main thread: An error occurred: There was an unexpected error when connecting to the Rights Management service. Contact the application support for further assistance.". How properly I should pass path to method's parameter?

Error Domain=RMSDomain Code=-1000 "There was an unexpected error when connecting to the Rights Management service. Contact the application support for further assistance." UserInfo={NSLocalizedFailureReason=(
0   RMSSampleApp                        0x000e950e +[ErrorFactory createErrorWithLocalizedMsg:englishMsg:code:contextLogger:] + 298
1   RMSSampleApp                        0x000e93ad +[ErrorFactory createErrorWithDict:code:contextLogger:] + 173
2   RMSSampleApp                        0x000e9263 +[ErrorFactory createErrorWithCode:contextLogger:] + 666
3   RMSSampleApp                        0x000c2d0a +[MSProtectedData completeWithInputInvalidError:completionBlock:] + 78
4   RMSSampleApp                        0x000c18f5 __120+[MSProtectedData protectedDataWithProtectedFile:userId:authenticationCallback:consentCallback:options:completionBlock:]_block_invoke + 37

@anivaros I am seeing the same issue when running [MSProtectedData protectedDataWithProtection..] Error Domain=RMSDomain Code=-1000 "There was an unexpected error when connecting to the Rights Management service. Contact the application support for further assistance."  Did you have a resolution when you closed this? Thanks, Kari

@kariohlsen solution is not call SDK's methods in "didFinishLaunchingWithOptions". It's looks like SDK needs more time to start. Try in "didBecomeActive" or somewhere else.

@anivaros Thanks.  Also wondering did you ever get answer on MSRightsManagement SDK support of docx, xlsx, pptx files? I sent them query but haven't heard back.

The SDK on iOS just like Android works on streams. Whether the file is Docx, pptx does not impact the it ability to open a document.

@kariohlsen : This question has been answered above in this thread
The RMS SDK does not know about file structures (other than PFiles).
So the application is responsible to extract the serialized publishing license and send it to the SDK to get the decryption keys (MSUserPolicy). Then you need to send in the buffer/stream that you want to decrypt using that MSUserPolicy object. We do not own the file format for Office files, so you will need to parse the format to extract the encrypted data yourself.
Hope this helps. If you have any other RMS related questions, please contact us at<EMAIL_ADDRESS>describing your needs.
Thanks
Shyam

Thanks @shiswara for the above information. I have treated the docx file as a custom protected file, extracting the PL and ended with the below error while instantiating the MSUserPolicy object with the extracted PL data. Could you please confirm that we should files such as docx, pptx, xslx and pdf as custom protected files and extract the publishing license to instantiate the MSUserPolicy Object and decrypt the file.
Below is console log
2018-12-31 16:24:44.854296+0530 Docs@Work[566:90112] INFO: work thread: Using database at path file:///var/mobile/Containers/Data/Application/0CA876D0-D65F-47DB-85FE-2D4D609963AE/Library/Caches/MSRightsManagement.framework.database.V2, context: MainMOC 2018-12-31 16:24:44.859229+0530 Docs@Work[566:89709] WARNING: work thread: Cached policy was not found in the database for key: OI09nowNoR0jtJg+TjrruFtGAljPvSl0VQ7OUQinJpM= and user<EMAIL_ADDRESS>2018-12-31 16:24:44.860409+0530 Docs@Work[566:89755] [ACT]:[SEMANTIC]:[Info]:Stage Receive: event name=cacheop, event priority=2, initId=29725485-EBAF-48AB-82AE-CC91C558A1D2, timestamp=1546253684859 2018-12-31 16:24:44.860763+0530 Docs@Work[566:89755] [ACT]:[TIMEDQUEUE]:[Info]:1 event put into timed queue. 2018-12-31 16:24:44.861306+0530 Docs@Work[566:89709] INFO: work thread: Could not find a cached policy for<EMAIL_ADDRESS>2018-12-31 16:24:44.863326+0530 Docs@Work[566:89709] INFO: work thread: Cached key for the policy not found 2018-12-31 16:24:44.865552+0530 Docs@Work[566:89709] INFO: work thread: Cached key for the policy is not found. 2018-12-31 16:24:44.866211+0530 Docs@Work[566:89294] Error parsing the publish license! No XrML element was found 2018-12-31 16:24:44.866899+0530 Docs@Work[566:89294] [ACT]:[SEMANTIC]:[Info]:Stage Receive: event name=parsexrmllicenseop, event priority=2, initId=29725485-EBAF-48AB-82AE-CC91C558A1D2, timestamp=1546253684866 2018-12-31 16:24:44.867134+0530 Docs@Work[566:89709] INFO: work thread: Getting the challange for PL. 2018-12-31 16:24:44.867138+0530 Docs@Work[566:89294] [ACT]:[TIMEDQUEUE]:[Info]:1 event put into timed queue. 2018-12-31 16:24:44.868818+0530 Docs@Work[566:89709] ERROR: work thread: An error occurred: There was an unexpected error when connecting to the Rights Management service. Contact Microsoft for further assistance. 2018-12-31 16:24:44.873429+0530 Docs@Work[566:89709] ERROR: work thread: failed to parse the PL with error : Error Domain=RMSDomain Code=-1 "There was an unexpected error when connecting to the Rights Management service. Contact Microsoft for further assistance." UserInfo={NSLocalizedDescription=There was an unexpected error when connecting to the Rights Management service. Contact Microsoft for further assistance., NSLogsPath=/var/mobile/Containers/Data/Application/0CA876D0-D65F-47DB-85FE-2D4D609963AE/Library/Application Support/MSProtection_LOG.txt, NSEnglishDescriptionKey=There was an unexpected error when connecting to the Rights Management service. Contact Microsoft for further assistance., NSLocalizedFailureReason=( 	0   Core                                0x000000010c709700 +[ErrorFactory createErrorWithLocalizedMsg:englishMsg:code:contextLogger:] + 292 	1   Core                                0x000000010c709598 +[ErrorFactory createErrorWithDict:code:contextLogger:] + 180 	2   Core                                0x000000010c709450 +[ErrorFactory createErrorWithCode:contextLogger:] + 1064 	3   Core                                0x000000010c6ef9b4 +[PublishLicenseParser injectFakeRootElement:error:] + 108 	4   Core                                0x000000010c6efaec +[PublishLicenseParser parsePublishLicense:error:] + 192 	5   Core                                0x000000010c6efe10 +[PublishLicenseParser extract:scenarioId:error:] + 480 	6   Core                                0x000000010c7197c0 -[IdentityStoreKey initWithPublishLicense:email:scenarioId:] + 184 	7   Core                                0x000000010c6ca2f0 +[MSAuthInfo authInfoWithPublishLicense:userId:asyncControl:consentCallback:completionBlock:] + 384 	8   Core                                0x000000010c6c1478 __122+[MSUserPolicy userPolicyWithSerializedPolicy:userId:authenticationCallback:consentCallback:asyncControl:completionBlock:]_block_invoke + 232 	9   libdispatch.dylib                   0x000000010d8c51dc _dispatch_call_block_and_release + 24 	10  libdispatch.dylib                   0x000000010d8c519c _dispatch_client_callout + 16 	11  libdispatch.dylib                   0x000000010d8d1200 _dispatch_queue_override_invoke + 968 	12  libdispatch.dylib                   0x000000010d8d67c8 _dispatch_root_queue_drain + 604 	13  libdispatch.dylib                   0x000000010d8d6500 _dispatch_worker_thread3 + 136 	14  libsystem_pthread.dylib             0x000000018438bfac _pthread_wqthread + 1176 	15  libsystem_pthread.dylib             0x000000018438bb08 start_wqthread + 4 )} 2018-12-31 16:24:44.875141+0530 Docs@Work[566:89709] INFO: work thread: +[MSAuthInfo authInfoWithPublishLicense:userId:asyncControl:consentCallback:completionBlock:]_block_invoke completion block is called. [line 78] 2018-12-31 16:24:44.876114+0530 Docs@Work[566:89294] [ACT]:[SEMANTIC]:[Info]:Stage Receive: event name=getauthinfoclientop, event priority=2, initId=29725485-EBAF-48AB-82AE-CC91C558A1D2, timestamp=1546253684875 2018-12-31 16:24:44.876386+0530 Docs@Work[566:89294] [ACT]:[TIMEDQUEUE]:[Info]:1 event put into timed queue. 2018-12-31 16:24:44.876733+0530 Docs@Work[566:89709] ERROR: work thread: An error occurred: The publishing license for the protected content has not been set correctly. Contact the application support for further assistance. 2018-12-31 16:24:44.877490+0530 Docs@Work[566:89294] [ACT]:[SEMANTIC]:[Info]:Stage Receive: event name=createuserpolicyconsumeclientop, event priority=2, initId=29725485-EBAF-48AB-82AE-CC91C558A1D2, timestamp=1546253684877 2018-12-31 16:24:44.877739+0530 Docs@Work[566:89294] [ACT]:[TIMEDQUEUE]:[Info]:1 event put into timed queue. 2018-12-31 16:24:44.878167+0530 Docs@Work[566:89709] INFO: work thread: Error: while getting the userPolicy for PL Printing description of error: Error Domain=RMSDomain Code=-7 "The publishing license for the protected content has not been set correctly. Contact the application support for further assistance." UserInfo={NSLocalizedDescription=The publishing license for the protected content has not been set correctly. Contact the application support for further assistance., NSScenarioId=3998EB25-743B-4EEB-ADBC-910DC5158532, NSLogsPath=/var/mobile/Containers/Data/Application/0CA876D0-D65F-47DB-85FE-2D4D609963AE/Library/Application Support/MSProtection_LOG.txt, NSEnglishDescriptionKey=The publishing license for the protected content has not been set correctly. Contact the application support for further assistance., NSLocalizedFailureReason=( 	0   Core                                0x000000010c709700 +[ErrorFactory createErrorWithLocalizedMsg:englishMsg:code:contextLogger:] + 292 	1   Core                                0x000000010c709598 +[ErrorFactory createErrorWithDict:code:contextLogger:] + 180 	2   Core                                0x000000010c709450 +[ErrorFactory createErrorWithCode:contextLogger:] + 1064 	3   Core                                0x000000010c6ca628 __93+[MSAuthInfo authInfoWithPublishLicense:userId:asyncControl:consentCallback:completionBlock:]_block_invoke + 100 	4   libdispatch.dylib                   0x000000010d8c51dc _dispatch_call_block_and_release + 24 	5   libdispatch.dylib                   0x000000010d8c519c _dispatch_client_callout + 16 	6   libdispatch.dylib                   0x000000010d8d1200 _dispatch_queue_override_invoke + 968 	7   libdispatch.dylib                   0x000000010d8d67c8 _dispatch_root_queue_drain + 604 	8   libdispatch.dylib                   0x000000010d8d6500 _dispatch_worker_thread3 + 136 	9   libsystem_pthread.dylib             0x000000018438bfac _pthread_wqthread + 1176 	10  libsystem_pthread.dylib             0x000000018438bb08 start_wqthread + 4 )}


Cant access data or models through Shell script files

Hi. I am not able to download the data or model files through shell script. It logins fine but does not download zip files and then throws error while trying to unzip no file.

Issue resolved by ccessing FTP site in sh file in Browser. Seems like the problem was with wget

hello, I'm trying to emulate your experiment and test it with my own data. But I don't know what form your raw data is in. In other words, how do you convert your data into pickle files for use? I hope you can give me your advice.  Thank you.


Release Zebra v1.0.1

Close #7021
Prepare for the Release

[ ] Make sure the PRs with the new checkpoint hashes and missed dependencies are already merged.
(See the release ticket checklist for details)

Summarise Release Changes
These steps can be done a few days before the release, in the same PR:
Change Log
Important: Any merge into main deletes any edits to the draft changelog.
Once you are ready to tag a release, copy the draft changelog into CHANGELOG.md.
We use the Release Drafter workflow to automatically create a draft changelog. We follow the Keep a Changelog format.
To create the final change log:

[x] Copy the latest draft changelog into CHANGELOG.md (there can be multiple draft releases)
[x] Delete any trivial changes

[x] Put the list of deleted changelog entries in a PR comment to make reviewing easier

[x] Combine duplicate changes
[x] Edit change descriptions so they will make sense to Zebra users
[x] Check the category for each change

Prefer the "Fix" category if you're not sure

README
README updates can be skipped for urgent releases.
Update the README to:

[x] Remove any "Known Issues" that have been fixed since the last release.
[x] Update the "Build and Run Instructions" with any new dependencies.
Check for changes in the Dockerfile since the last tag: git diff <previous-release-tag> docker/Dockerfile.
[x] If Zebra has started using newer Rust language features or standard library APIs, update the known working Rust version in the README, book, and Cargo.tomls

You can use a command like:
fastmod --fixed-strings '1.58' '1.65'

Create the Release PR

[x] Push the updated changelog and README into a new branch
for example: bump-v1.0.0 - this needs to be different to the tag name
[x] Create a release PR by adding &template=release-checklist.md to the comparing url (Example).
[ ] Freeze the batched queue using Mergify.
[ ] Mark all the release PRs as Critical priority, so they go in the urgent Mergify queue.

Update Versions and End of Support
Update Zebra Version
Choose a Release Level
Zebra follows semantic versioning. Semantic versions look like: MAJOR.MINOR.PATCH[-TAG.PRE-RELEASE]
Choose a release level for zebrad. Release levels are based on user-visible changes from the changelog:

Mainnet Network Upgrades are major releases
significant new features, large changes, deprecations, and removals are minor releases
otherwise, it is a patch release

Zebra's Rust API doesn't have any support or stability guarantees, so we keep all the zebra-* and tower-* crates on a beta pre-release version.
Update Crate Versions

If you're publishing crates for the first time:

[ ] Install cargo-release: cargo install cargo-release
[ ] Make sure you are  an owner of the crate or a member of the Zebra crates.io owners group on GitHub

[ ] Update crate versions and do a release dry-run

[ ] cargo clean (optional)
[ ] cargo release version --verbose --execute --workspace --exclude zebrad beta
[ ] cargo release version --verbose --execute --package zebrad [ major | minor | patch ]
[ ] cargo release publish --verbose --dry-run --workspace

[ ] Commit the version changes to your release PR branch using git: cargo release commit --verbose --execute --workspace

Update End of Support
The end of support height is calculated from the current blockchain height:

[ ] Find where the Zcash blockchain tip is now by using a Zcash explorer or other tool.
[ ] Replace ESTIMATED_RELEASE_HEIGHT in end_of_support.rs with the height you estimate the release will be tagged.

Optional: calculate the release tagging height

Add 1152 blocks for each day until the release
For example, if the release is in 3 days, add 1152 * 3 to the current Mainnet block height

Update the Release PR

[ ] Push the version increments and the release constants to the release branch.

Publish the Zebra Release
Create the GitHub Pre-Release

[ ] Wait for all the release PRs to be merged
[ ] Create a new release using the draft release as a base, by clicking the Edit icon in the draft release
[ ] Set the tag name to the version tag,
for example: v1.0.0
[ ] Set the release to target the main branch
[ ] Set the release title to Zebra  followed by the version tag,
for example: Zebra 1.0.0
[ ] Replace the prepopulated draft changelog in the release description with the final changelog you created;
starting just after the title ## [Zebra ... of the current version being released,
and ending just before the title of the previous release.
[ ] Mark the release as 'pre-release', until it has been built and tested
[ ] Publish the pre-release to GitHub using "Publish Release"
[ ] Delete all the draft releases from the list of releases

Test the Pre-Release

[ ] Wait until the Docker binaries have been built on main, and the quick tests have passed.
(You can ignore the full sync and lightwalletd tests, because they take about a day to run.)
[ ] Wait until the pre-release deployment machines have successfully launched

Publish Release

[ ] Publish the release to GitHub by disabling 'pre-release', then clicking "Set as the latest release"

Publish Crates

[ ] Run cargo login
[ ] Run cargo clean in the zebra repo (optional)
[ ] Publish the crates to crates.io: cargo release publish --verbose --workspace --execute
[ ] Check that Zebra can be installed from crates.io:
cargo install --locked --force --version 1.0.0 zebrad && ~/.cargo/bin/zebrad
and put the output in a comment on the PR.

Publish Docker Images

[ ] Wait for the the Docker images to be published successfully.
[ ] Un-freeze the batched queue using Mergify.

Release Failures
If building or running fails after tagging:

Tag a new release, following these instructions...

Fix the bug that caused the failure
Start a new patch release
Skip the Release Preparation, and start at the Release Changes step
Update CHANGELOG.md with details about the fix
Follow the release checklist for the new Zebra version

Trivial changes:
$ cat trivial-issues.txt | sort -u
 - add(ci): Run release builds and production Docker image tests on pull requests ([#7055](https://github.com/ZcashFoundation/zebra/pull/7055))
 - add(tests): Add snapshot tests for sprout database formats ([#7057](https://github.com/ZcashFoundation/zebra/pull/7057))
 - build(deps): bump Swatinem/rust-cache from 2.4.0 to 2.5.0 ([#7002](https://github.com/ZcashFoundation/zebra/pull/7002))
 - build(deps): bump baptiste0928/cargo-install from 2.0.0 to 2.1.0 ([#6903](https://github.com/ZcashFoundation/zebra/pull/6903))
 - build(deps): bump bitflags from 2.3.1 to 2.3.2 ([#6943](https://github.com/ZcashFoundation/zebra/pull/6943))
 - build(deps): bump clap from 4.3.3 to 4.3.4 ([#6957](https://github.com/ZcashFoundation/zebra/pull/6957))
 - build(deps): bump clap from 4.3.4 to 4.3.5 ([#7038](https://github.com/ZcashFoundation/zebra/pull/7038))
 - build(deps): bump clap from 4.3.5 to 4.3.6 ([#7059](https://github.com/ZcashFoundation/zebra/pull/7059))
 - build(deps): bump clap from 4.3.6 to 4.3.8 ([#7066](https://github.com/ZcashFoundation/zebra/pull/7066))
 - build(deps): bump docker/build-push-action from 4.1.0 to 4.1.1 ([#6942](https://github.com/ZcashFoundation/zebra/pull/6942))
 - build(deps): bump docker/metadata-action from 4.5.0 to 4.6.0 ([#6941](https://github.com/ZcashFoundation/zebra/pull/6941))
 - build(deps): bump hyper from 0.14.26 to 0.14.27 ([#7082](https://github.com/ZcashFoundation/zebra/pull/7082))
 - build(deps): bump insta from 1.29.0 to 1.30.0 ([#7051](https://github.com/ZcashFoundation/zebra/pull/7051))
 - build(deps): bump itertools from 0.10.5 to 0.11.0 ([#7050](https://github.com/ZcashFoundation/zebra/pull/7050))
 - build(deps): bump log from 0.4.18 to 0.4.19 ([#6919](https://github.com/ZcashFoundation/zebra/pull/6919))
 - build(deps): bump peter-evans/dockerhub-description from 3.4.1 to 3.4.2 ([#7023](https://github.com/ZcashFoundation/zebra/pull/7023))
 - build(deps): bump reviewdog/action-actionlint from 1.37.0 to 1.37.1 ([#7003](https://github.com/ZcashFoundation/zebra/pull/7003))
 - build(deps): bump sentry from 0.31.3 to 0.31.4 ([#6958](https://github.com/ZcashFoundation/zebra/pull/6958))
 - build(deps): bump sentry from 0.31.4 to 0.31.5 ([#6988](https://github.com/ZcashFoundation/zebra/pull/6988))
 - build(deps): bump serde\_json from 1.0.96 to 1.0.97 ([#6987](https://github.com/ZcashFoundation/zebra/pull/6987))
 - build(deps): bump tj-actions/changed-files from 36.1.0 to 36.2.1 ([#6973](https://github.com/ZcashFoundation/zebra/pull/6973))
 - build(deps): bump tj-actions/changed-files from 36.2.1 to 36.3.0 ([#6986](https://github.com/ZcashFoundation/zebra/pull/6986))
 - build(deps): bump tj-actions/changed-files from 36.3.0 to 36.4.0 ([#7004](https://github.com/ZcashFoundation/zebra/pull/7004))
 - build(deps): bump tj-actions/changed-files from 36.4.0 to 36.4.1 ([#7024](https://github.com/ZcashFoundation/zebra/pull/7024))
 - build(deps): bump tj-actions/changed-files from 36.4.1 to 37.0.3 ([#7065](https://github.com/ZcashFoundation/zebra/pull/7065))
 - build(deps): bump w9jds/firebase-action from 11.30.1 to 12.4.0 ([#7005](https://github.com/ZcashFoundation/zebra/pull/7005))
 - change(commands): Ignore error from loading config if running the 'generate' or 'download' commands ([#7014](https://github.com/ZcashFoundation/zebra/pull/7014))
 - change(docs): Add `cargo clean` step to crate publishing steps ([#6959](https://github.com/ZcashFoundation/zebra/pull/6959))
 - change(docs): Explicitly invoke --execute when bumping crate versions ([#6949](https://github.com/ZcashFoundation/zebra/pull/6949))
 - change(network): Configurable maximum connections per IP ([#7013](https://github.com/ZcashFoundation/zebra/pull/7013))
 - change(release): Add deny.toml update details to release-checklist.md ([#7042](https://github.com/ZcashFoundation/zebra/pull/7042))
 - change(release): Add more cargo clean to the release checklist ([#6964](https://github.com/ZcashFoundation/zebra/pull/6964))
 - change(release): Change network upgrade wording in release-checklist.md ([#7010](https://github.com/ZcashFoundation/zebra/pull/7010))
 - change(rename): Update missed tower-batch-control renames ([#7011](https://github.com/ZcashFoundation/zebra/pull/7011))
 - change(state): Use `OrderedUtxo` in `CheckpointVerifiedBlock` ([#6971](https://github.com/ZcashFoundation/zebra/pull/6971))
 - cleanup(test): Make test debugging output more readable ([#7027](https://github.com/ZcashFoundation/zebra/pull/7027))
 - cleanup(workflow): Remove an outdated TODO in release-binaries.yml ([#6978](https://github.com/ZcashFoundation/zebra/pull/6978))
 - fix(build): Suppress warnings about doc links pointing to private items ([#6944](https://github.com/ZcashFoundation/zebra/pull/6944))
 - fix(cd): Rename a CD job with the same name as a CI job ([#7063](https://github.com/ZcashFoundation/zebra/pull/7063))
 - fix(changelog): Fix a broken link in CHANGELOG.md ([#6979](https://github.com/ZcashFoundation/zebra/pull/6979))
 - fix(cleanup): redundant configuration feature ([#6929](https://github.com/ZcashFoundation/zebra/pull/6929))
 - fix(deps): Replace openssl with rustls in tests and experimental features ([#7047](https://github.com/ZcashFoundation/zebra/pull/7047))
 - fix(doc): Add `fastmod --hidden` to mass-renames.md ([#6913](https://github.com/ZcashFoundation/zebra/pull/6913))
 - fix(docker): Improve Dockerfile cache use by making build commands match exactly ([#6933](https://github.com/ZcashFoundation/zebra/pull/6933))
 - fix(docker): Stop resetting the `cargo-chef` cache in the Dockerfile ([#6934](https://github.com/ZcashFoundation/zebra/pull/6934))
 - fix(logs): Avoid grouping logs for separate tasks ([#6923](https://github.com/ZcashFoundation/zebra/pull/6923))
 - fix(net): Clean up licensing, closure `move`, log typos, tracing spans ([#6995](https://github.com/ZcashFoundation/zebra/pull/6995))
 - fix(release): Use correct cargo release manifest key name ([#7028](https://github.com/ZcashFoundation/zebra/pull/7028))
 - fix(state): Replace a chain length assertion with a NotReadyToBeCommitted error ([#7072](https://github.com/ZcashFoundation/zebra/pull/7072))
 - refactor(app): De-duplicate and fix version handling code ([#6996](https://github.com/ZcashFoundation/zebra/pull/6996))
 - refactor(consensus): Rename router\_verifier to block\_verifier\_router ([#6998](https://github.com/ZcashFoundation/zebra/pull/6998))
 - rename(state): do additional renaming for clarification purposes ([#6967](https://github.com/ZcashFoundation/zebra/pull/6967))
- Refactor the structure of finalizable blocks, to make validation clearer ([#7035](https://github.com/ZcashFoundation/zebra/pull/7035), [#7025](https://github.com/ZcashFoundation/zebra/pull/7025))

If Zebra has started using newer Rust language features or standard library APIs, update the known working Rust version in the README, book, and Cargo.tomls

This step was done in PR #7032.

This is the draft changelog as of commit caf9a0925f80cd93003a90cad6394ded1bdc1b15 at Wednesday 28 June 0100 UTC.

$ cargo install --locked --force --version 1.0.1 zebrad
Downloaded zebrad v1.0.1
Downloaded 1 crate (231.9 KB) in 1.72s
Updating crates.io index
Installing zebrad v1.0.1
Updating crates.io index
warning: package `x25519-dalek v2.0.0-rc.2` in Cargo.lock is yanked in registry `crates-io`, consider running without --locked
Updating crates.io index
Compiling proc-macro2 v1.0.60
Compiling quote v1.0.28
...
   Compiling zebrad v1.0.1
   Compiling zcash_primitives v0.11.0
   Compiling zebra-chain v1.0.0-beta.27
   Compiling zcash_proofs v0.11.0
   Compiling zebra-node-services v1.0.0-beta.27
   Compiling zebra-network v1.0.0-beta.27
   Compiling zebra-script v1.0.0-beta.27
   Compiling rocksdb v0.21.0
   Compiling zebra-state v1.0.0-beta.27
   Compiling zebra-consensus v1.0.0-beta.27
   Compiling zebra-rpc v1.0.0-beta.27
    Finished release [optimized + debuginfo] target(s) in 3m 00s
   Replacing /home/dev/.cargo/bin/zebrad
    Replaced package `zebrad v1.0.0` with `zebrad v1.0.1` (executable `zebrad`)

$ ~/.cargo/bin/zebrad


chore: align image caption in columns

Please always provide the GitHub issue(s) your PR is for, as well as test URLs where your change can be observed (before and after):
Fix #836 and #838
Test URLs:

Before: https://main--moleculardevices--hlxsites.hlx.page/products/clone-screening
After: https://836--moleculardevices--hlxsites.hlx.page/products/clone-screening

Also tested on:
https://836--moleculardevices--hlxsites.hlx.page/products/cellular-imaging-systems/acquisition-and-analysis-software/in-carta-image-analysis-software


Build fail

Attempting build on OS X 10.11.5 (15F34).
Followed all steps without warnings or errors, until make.
make failed with the following output.
$make
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-recursive
Making all in src
Making all in sparkey
make[3]: Nothing to be done for `all'.
./libpostal_data download all /applications/libpostal
Checking for new libpostal data file...
libpostal data file up to date
Checking for new libpostal geodb data file...
New libpostal geodb data file available
Downloading multipart: http://libpostal.s3.amazonaws.com/geodb.tar.gz, size=486746092, num_chunks=7
Downloading part 1: filename=/applications/libpostal/geodb.tar.gz.1, offset=0, max=67108863
Downloading part 2: filename=/applications/libpostal/geodb.tar.gz.2, offset=67108864, max=134217727
Downloading part 3: filename=/applications/libpostal/geodb.tar.gz.3, offset=134217728, max=201326591
Downloading part 4: filename=/applications/libpostal/geodb.tar.gz.4, offset=201326592, max=268435455
Downloading part 5: filename=/applications/libpostal/geodb.tar.gz.5, offset=268435456, max=335544319
Downloading part 6: filename=/applications/libpostal/geodb.tar.gz.6, offset=335544320, max=402653183
Downloading part 7: filename=/applications/libpostal/geodb.tar.gz.7, offset=402653184, max=486746092
make[3]: *** [all-local] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

I googled the error and found identical output in openvenues/node-postal#6(closed). @jeffriesen 'got it running' on Aug 25. In this repo, the last four lines of my output match #13(closed). I followed chamnap's solution in #5(closed), i.e. I did brew upgrade, with result pkg-config 0.29.1; then, I ran make again, but had the same output, except Downloading part 3 and Downloading part 4 were reversed -  Downloading part 4: filename=/applications/libpostal/geodb.tar.gz.4, offset=201326592, max=268435455 Downloading part 3: filename=/applications/libpostal/geodb.tar.gz.3, offset=134217728, max=201326591.

Does the dir you specified for --datadir when running configure have enough space? It needs approximately 2.2G free to build the current version of libpostal (that number may change in subsequent updates to the models).
That what it sounds like to me. At that point in the build it's just trying to download chunks of a file in parallel using curl, so probably either lack of disk space or some kind S3 connection weirdness.

The disk (Macintosh HD) that houses the dir I specified for --datadir had approximately 4G free. I freed up an order of magnitude more space and ran it again; It generated the same output.

Ok, and the datadir you want is /applications/libpostal, correct? That's what's being used, so if you moved it somewhere along the way, you'll have to re-run configure. That's also probably a root-owned directory, might need to chown it to your user. That doesn't seem likely though because It looks like it did download one of the files (libpostal_data.tar.gz, which is about 10M) successfully but failed on geodb.tar.gz, which is ~400M.
You can also try manually downloading the files. If you have awscli installed, it's aws s3 cp s3://libpostal/geodb.tar.gz $LIBPOSTAL_DATA_DIR. Otherwise it's curl http://libpostal.s3.amazonaws.com/geodb.tar.gz -O $LIBPOSTAL_DATA_DIR/geodb.tar.gz. If downloading that way works, then maybe there's a problem with the data download script, but I very much suspect it's space/permissions.

Yes, the libpostal dir. For me it's /desktop/libpostal.
I ran curl http://libpostal.s3.amazonaws.com/geodb.tar.gz -O $LIBPOSTAL_DATA_DIR/geodb.tar.gz
Output:
0curl: (56) Recv failure: Connection reset by peer
curl: (3) <url> malformed

Oh, my typo. The curl command is -o lowercase, not -O. Will want to set LIBPOSTAL_DATA_DIR=/desktop/libpostal before running that.
Also, I can see in the make output that the dir being used is /applications/libpostal, so you'll need to run configure --datadir=/desktop/libpostal to set it to the correct location as far as make is concerned.

curl http://libpostal.s3.amazonaws.com/geodb.tar.gz -o /desktop/libpostal/geodb.tar.gz succeeded.
I moved the libpostal dir to /Applications, and ran make there.  As discussed, make fails at downloading part 7 of geodb.tar.gz. I tried:
/Applications/libpostal$ src/address_parser
dyld: Library not loaded: /usr/local/lib/libpostal.0.dylib
  Referenced from: /Applications/libpostal/src/.libs/address_parser
  Reason: image not found
Trace/BPT trap: 5

Ok, I'm not clear on where you want the datadir to be, /desktop or /Applicaitons. When you say "I moved the libpostal dir" is that with mv or did you run ./configure again?
Try nuking the entire libpostal checkout and start over following the exact steps in the Readme using whichever datadir you chose.

I apologize for the possibly unnecessary length.
I moved the libpostal dir with mv and then ran ./configure again:
/users/samuelputnam/desktop$ mv libpostal /applications
/users/samuelputnam/desktop$ cd /applications/libpostal
/applications/libpostal$ ./bootstrap.sh
glibtoolize: putting auxiliary files in '.'.
glibtoolize: copying file './ltmain.sh'
glibtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac,
glibtoolize: and rerunning glibtoolize and aclocal.
glibtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
configure.ac:8: installing './compile'
configure.ac:6: installing './missing'
src/Makefile.am: installing './depcomp'

/applications/libpostal$ ./configure --datadir=/applications/libpostal
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
# I am leaving out output lines to save space in this comment
config.status: executing depfiles commands
config.status: executing libtool commands

/applications/libpostal$ make
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-recursive
Making all in src
Making all in sparkey
make[3]: Nothing to be done for `all'.
./libpostal_data download all /applications/libpostal/libpostal
Checking for new libpostal data file...
libpostal data file up to date
Checking for new libpostal geodb data file...
New libpostal geodb data file available
Downloading multipart: http://libpostal.s3.amazonaws.com/geodb.tar.gz, size=486746092, num_chunks=7
Downloading part 1: filename=/applications/libpostal/libpostal/geodb.tar.gz.1, offset=0, max=67108863
Downloading part 3: filename=/applications/libpostal/libpostal/geodb.tar.gz.3, offset=134217728, max=201326591
Downloading part 4: filename=/applications/libpostal/libpostal/geodb.tar.gz.4, offset=201326592, max=268435455
Downloading part 2: filename=/applications/libpostal/libpostal/geodb.tar.gz.2, offset=67108864, max=134217727
Downloading part 5: filename=/applications/libpostal/libpostal/geodb.tar.gz.5, offset=268435456, max=335544319
Downloading part 6: filename=/applications/libpostal/libpostal/geodb.tar.gz.6, offset=335544320, max=402653183
Downloading part 7: filename=/applications/libpostal/libpostal/geodb.tar.gz.7, offset=402653184, max=486746092
^Z
[3]+  Stopped                 make

I stopped it because it did not continue for an amount of time similar to that when it had failed at this step.
Then, nuking the entire libpostal checkout and start over following the exact steps in the Readme using whichever datadir you chose.
/applications/libpostal$ cd ..
/Applications$ rm -rf libpostal
/Applications$ brew uninstall snappy autoconf automake libtool pkg-config
Uninstalling /usr/local/Cellar/snappy/1.1.3... (20 files, 411.9K)
Uninstalling /usr/local/Cellar/autoconf/2.69... (70 files, 3.0M)
Uninstalling /usr/local/Cellar/automake/1.15... (130 files, 2.9M)
Uninstalling /usr/local/Cellar/libtool/2.4.6... (70 files, 3.7M)
Uninstalling /usr/local/Cellar/pkg-config/0.29.1... (10 files, 627.2K)

/Applications$ brew install snappy autoconf automake libtool pkg-config
==> Downloading https://homebrew.bintray.com/bottles/snappy-1.1.3.el_capitan.bot
Already downloaded: /Library/Caches/Homebrew/snappy-1.1.3.el_capitan.bottle.tar.gz
==> Pouring snappy-1.1.3.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/snappy/1.1.3: 20 files, 411.9K
==> Downloading https://homebrew.bintray.com/bottles/autoconf-2.69.el_capitan.bo
Already downloaded: /Library/Caches/Homebrew/autoconf-2.69.el_capitan.bottle.4.tar.gz
==> Pouring autoconf-2.69.el_capitan.bottle.4.tar.gz
==> Caveats
Emacs Lisp files have been installed to:
  /usr/local/share/emacs/site-lisp/autoconf
==> Summary
🍺  /usr/local/Cellar/autoconf/2.69: 70 files, 3.0M
==> Downloading https://homebrew.bintray.com/bottles/automake-1.15.el_capitan.bo
Already downloaded: /Library/Caches/Homebrew/automake-1.15.el_capitan.bottle.2.tar.gz
==> Pouring automake-1.15.el_capitan.bottle.2.tar.gz
🍺  /usr/local/Cellar/automake/1.15: 130 files, 2.9M
==> Downloading https://homebrew.bintray.com/bottles/libtool-2.4.6.el_capitan.bo
Already downloaded: /Library/Caches/Homebrew/libtool-2.4.6.el_capitan.bottle.tar.gz
==> Pouring libtool-2.4.6.el_capitan.bottle.tar.gz
==> Caveats
In order to prevent conflicts with Apple's own libtool we have prepended a "g"
so, you have instead: glibtool and glibtoolize.
==> Summary
🍺  /usr/local/Cellar/libtool/2.4.6: 70 files, 3.7M
==> Downloading https://homebrew.bintray.com/bottles/pkg-config-0.29.1.el_capita
Already downloaded: /Library/Caches/Homebrew/pkg-config-0.29.1.el_capitan.bottle.tar.gz
==> Pouring pkg-config-0.29.1.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/pkg-config/0.29.1: 10 files, 627.2K

/Applications$ git clone https://github.com/openvenues/libpostal
Cloning into 'libpostal'...
remote: Counting objects: 21231, done.
remote: Compressing objects: 100% (24/24), done.
error: RPC failed; curl 56 SSLRead() return error -365.00 KiB/s
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
/Applications$

Trying git clone again:
/Applications$ git clone https://github.com/openvenues/libpostal
Cloning into 'libpostal'...
remote: Counting objects: 21231, done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 21231 (delta 6), reused 0 (delta 0), pack-reused 21207
Receiving objects: 100% (21231/21231), 32.36 MiB | 328.00 KiB/s, done.
Resolving deltas: 100% (12741/12741), done.
Checking connectivity... done.

Continuing:
/Applications$ cd libpostal
/Applications/libpostal$ ./bootstrap.sh
glibtoolize: putting auxiliary files in '.'.
glibtoolize: copying file './ltmain.sh'
glibtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac,
glibtoolize: and rerunning glibtoolize and aclocal.
glibtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
configure.ac:8: installing './compile'
configure.ac:8: installing './config.guess'
configure.ac:8: installing './config.sub'
configure.ac:6: installing './install-sh'
configure.ac:6: installing './missing'
src/Makefile.am: installing './depcomp'
parallel-tests: installing './test-driver'

Applications/libpostal$ ./configure --datadir=/applications/libpostal
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
# I am leaving out output lines to save space in this comment
config.status: executing depfiles commands
config.status: executing libtool commands

/Applications/libpostal$ make
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-recursive
Making all in src
Making all in sparkey
depbase=`echo buf.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
	/bin/sh ../../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I../..     -I/usr/local/include -O2 -Wall -Wextra -Wfloat-equal -Wshadow -Wpointer-arith -Werror -pedantic -MT buf.lo -MD -MP -MF $depbase.Tpo -c -o buf.lo buf.c &&\
	mv -f $depbase.Tpo $depbase.Plo
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I../.. -I/usr/local/include -O2 -Wall -Wextra -Wfloat-equal -Wshadow -Wpointer-arith -Werror -pedantic -MT buf.lo -MD -MP -MF .deps/buf.Tpo -c buf.c  -fno-common -DPIC -o .libs/buf.o
# I am leaving out output lines to save space in this comment
language_classifier_test-string_utils.o language_classifier_test-file_utils.o utf8proc/language_classifier_test-utf8proc.o language_classifier_test-unicode_scripts.o  -L/usr/local/lib ./.libs/libscanner.a -lsnappy
./libpostal_data download all /applications/libpostal/libpostal
Checking for new libpostal data file...
New libpostal data file available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10.1M  100 10.1M    0     0   300k      0  0:00:34  0:00:34 --:--:--  313k
x address_expansions/
x address_expansions/address_dictionary.dat
x numex/
x numex/numex.dat
x transliteration/
x transliteration/transliteration.dat
Checking for new libpostal geodb data file...
New libpostal geodb data file available
Downloading multipart: http://libpostal.s3.amazonaws.com/geodb.tar.gz, size=486746092, num_chunks=7
Downloading part 2: filename=/applications/libpostal/libpostal/geodb.tar.gz.2, offset=67108864, max=134217727
Downloading part 3: filename=/applications/libpostal/libpostal/geodb.tar.gz.3, offset=134217728, max=201326591
Downloading part 1: filename=/applications/libpostal/libpostal/geodb.tar.gz.1, offset=0, max=67108863
Downloading part 4: filename=/applications/libpostal/libpostal/geodb.tar.gz.4, offset=201326592, max=268435455
Downloading part 5: filename=/applications/libpostal/libpostal/geodb.tar.gz.5, offset=268435456, max=335544319
Downloading part 6: filename=/applications/libpostal/libpostal/geodb.tar.gz.6, offset=335544320, max=402653183
Downloading part 7: filename=/applications/libpostal/libpostal/geodb.tar.gz.7, offset=402653184, max=486746092

This time, make moved beyond the Downloading part 7: filename=/applications/libpostal/libpostal/geodb.tar.gz.7, offset=402653184, max=486746092 step after 30 mins and failed at Downloading part 10: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.10, offset=603979776, max=672728402.
x geodb/
x geodb/geodb.spi
x geodb/geodb.spl
x geodb/geodb_feature_graph.dat
x geodb/geodb_features.trie
x geodb/geodb_names.trie
x geodb/geodb_postal_codes.dat
Checking for new libpostal parser data file...
New libpostal parser data file available
Downloading multipart: http://libpostal.s3.amazonaws.com/parser.tar.gz, size=190250553, num_chunks=2
Downloading part 1: filename=/applications/libpostal/libpostal/parser.tar.gz.1, offset=0, max=67108863
Downloading part 2: filename=/applications/libpostal/libpostal/parser.tar.gz.2, offset=67108864, max=190250553
x address_parser/
x address_parser/address_parser.dat
x address_parser/address_parser_vocab.trie
x address_parser/address_parser_phrases.trie
Checking for new libpostal language classifier data file...
New libpostal language classifier data file available
Downloading multipart: http://libpostal.s3.amazonaws.com/language_classifier.tar.gz, size=672728402, num_chunks=10
Downloading part 1: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.1, offset=0, max=67108863
Downloading part 2: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.2, offset=67108864, max=134217727
Downloading part 3: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.3, offset=134217728, max=201326591
Downloading part 4: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.4, offset=201326592, max=268435455
Downloading part 5: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.5, offset=268435456, max=335544319
Downloading part 6: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.6, offset=335544320, max=402653183
Downloading part 7: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.7, offset=402653184, max=469762047
Downloading part 8: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.8, offset=469762048, max=536870911
Downloading part 9: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.9, offset=536870912, max=603979775
Downloading part 10: filename=/applications/libpostal/libpostal/language_classifier.tar.gz.10, offset=603979776, max=672728402
make[3]: *** [all-local] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Based on the content of libpostal_data,
fi

LIBPOSTAL_S3_BUCKET_NAME="libpostal"
LIBPOSTAL_S3_KEY="s3://$LIBPOSTAL_S3_BUCKET_NAME"
LIBPOSTAL_S3_BUCKET_URL="http://$LIBPOSTAL_S3_BUCKET_NAME.s3.amazonaws.com"
LIBPOSTAL_DATA_FILE="libpostal_data.tar.gz"
LIBPOSTAL_GEODB_FILE="geodb.tar.gz"
LIBPOSTAL_PARSER_FILE="parser.tar.gz"
LIBPOSTAL_LANG_CLASS_FILE="language_classifier.tar.gz"

COMMAND=$1

it looks like language_classifier.tar.gz could be the final download during make, so:
/Applications/libpostal$ curl http://libpostal.s3.amazonaws.com/language_classifier.tar.gz -o /applications/libpostal/language_classifier.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  641M  100  641M    0     0   204k      0  0:53:37  0:53:37 --:--:--  134k
/Applications/libpostal$ open language_classifier.tar.gz

Trying it out:
/applications/libpostal$ src/language_classifier "One-hundred twenty E 96th St"
ERR   transliteration table is NULL. Call libpostal_setup() or transliteration_module_setup()
   at transliterate (transliterate.c:674) errno: None
Languages:

en (0.999715)

And address_parser:

/applications/libpostal$ src/address_parser
Loading models...

Welcome to libpostal's address parser.

Type in any address to parse and print the result.

Special commands:

.language [code] to specify a language
.country [code] to specify a country
.exit to quit the program

> "Quatre vingt douze Ave des Champs-Elysees"

Result:

{
  "house": "\" quatre vingt",
  "road": "douze ave des",
  "house": "champs-elysees \""
}

>

Seems all is good here. Thank you.

What are the steps if I want to remove it from my system ?


Ensure dtype consistency for activation functions

This PR includes the following:

Replace hard_swish with hard_silu to follow the naming convention of silu
Export hard_silu as hard_swish, following the pattern of silu and swish
Add tanh activation (it is currently missing in the codebase)
Add dtype tests for all existing activation functions

Codecov Report
Attention: 24 lines in your changes are missing coverage. Please review.

Comparison is base (f0b7062) 75.07% compared to head (4c234d6) 55.93%.
Report is 3 commits behind head on master.

Files
Patch %
Lines

keras/backend/torch/nn.py
0.00%
13 Missing :warning:

keras/backend/tensorflow/nn.py
14.28%
6 Missing :warning:

keras/ops/nn.py
75.00%
3 Missing and 1 partial :warning:

keras/backend/jax/nn.py
50.00%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #18876       +/-   ##
===========================================
- Coverage   75.07%   55.93%   -19.14%
===========================================
  Files         336      336
  Lines       34846    34863       +17
  Branches     6854     6852        -2
===========================================
- Hits        26162    19502     -6660
- Misses       7149    13889     +6740
+ Partials     1535     1472       -63

Flag
Coverage Δ

keras
55.93% <66.19%> (-19.02%)
:arrow_down:

keras-jax
?

keras-numpy
55.93% <66.19%> (?)

keras-tensorflow
?

keras-torch
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Support for Optional members in interfaces followed by non-optional ones

tB applies this language restriction to interfaces as well as normal subs/functions; I'm not sure if it's possible for the latter, but for interfaces, there's currently a lot of interfaces where I'm having to drop 'optional' attributes so the COM spec clearly supports this... how practical would it be to support this in tB and is it something that should be added?
I'd say it should be added because it aids in clarity, not having to create and pass empty parameters that might be interpreted as 0.

(quoting @WaynePhillipsEA  on Discord)
WaynePhillipsEA — Today at 4:08 PM
Ahem...  you might want to try passing a pointer value to byref-UDTs in tB ;). It's an undocumented, but perhaps rather useful, feature.  Though note that we may later refine the "feature".
[4:09 PM]
I'm particular, I find it comes in very handy when working with union types.
(/quoting @WaynePhillipsEA  on Discord)


MIDI in p5.js

Hi,
I was wondering what is the p5.js stance on MIDI? MIDI is such a crucial part of music production and art, so it seems natural that p5.js should have some support for it. From what I can see, p5.js-sound has some functions to convert MIDI note numbers <-> pitch frequencies, but is not yet able to handle MIDI messages in any way - is that right?
As I understand it, the Web MIDI API works well but apparently is supported only on Chrome, which I'm guessing is the main reason that MIDI is not yet supported on p5.js. (There seem to be workarounds for other browsers, but I have not explored this)
Even if we cannot handle MIDI messages, we could still have the feature to load SoundFonts and synthesize them in the browser, right? This would be great for users (like myself) who wish to produce music in p5.js without having to define my own oscillators, not to mention that synthesizing natural-sounding instrument voices using oscillators is impractical in most cases.
Inspiration (with links to many awesome demos): MIDI.js

Good idea to think about MIDI!
Web MIDI is exciting, but as you mentioned, MIDI itself isn't super accessible via the web—only in Chrome or with special plugins. It's a nice bonus if a user is able to hook up a MIDI keyboard to a project, or able to trigger MIDI messages in a DAW so that they can just focus on using p5 to generate the notes. It could be worth providing some examples for how to do that, but it limits our audience if we depend on that functionality.
MIDI can be a useful framework for thinking about music as numbers. So we use freqToMidi and midiToFreq to make it easier to work with pitch as numbers, without reinventing the wheel. Sometimes we already have decent numbers in Web Audio, which I think is a better reference—for example, there's not really a reason to think of velocity between 0 and 127, so we can normalize that range to 0-1 to match the Web Audio API gain node.
MIDI time representation was designed to be read more by machines than humans. For example, you can't have a note duration, but instead need to do that with velocity changes (note on/off) over time. So I think we'd need higher level abstractions around time that are more human-friendly. Tone.js MIDI Convert is a cool example of translating MIDI scores into a JSON format that is a bit more human-readable. It integrates well with Tone.js structures for working with pitch / duration / velocity over time, which are really well thought out.
midi.js is inspiring, too! Porting soundfonts to the web is a cool idea. Would we add these soundfonts to the library or point to some separate source(s)? I don't know how many users we could expect to load soundfonts, let alone generate them, which I recall being a bit of a deep process with midi.js. If the goal is to make it quick for users to get decent sounds, maybe there are simpler ways to accomplish this, like the incorporation of presets into p5 effects and synths (which started last summer)

about sound fonts,  I used this :  https://github.com/danigb/soundfont-player
example with p5js (press keyboard from 's' to 'l' :
https://b2renger.github.io/Introduction_p5js/07_audio_01_soundfont/index.html
code : https://github.com/b2renger/Introduction_p5js/tree/master/07_audio_01_soundfont

@b2renger that's very cool, and pretty much exactly what I had in mind! If we can support this kind of behaviour in p5.js, I would be a happy cookie! But before getting into how exactly to bring SoundFonts in, I think there's more groundwork we need to do.
@therewasaguy
Fair enough, I agree that we shouldn't exclude some of our audience. In any case, we don't need Web MIDI to produce standalone music in the browser, which at least for me is the more important goal before we think about connecting hardware and native DAWs.
That said, I believe it is quite important to maintain MIDI-compatibility beyond the level of note-numbers (just so that users can do things like load MIDI files into their programs, save their compositions as MIDI files, also so we don't close off the possibility of eventual hardware/DAW integration.
Since you brought up the point of representation, I would also like to push for a compact way to handle notes in p5.js, such as a new p5.Note class which captures information about pitch, velocity and timing in one package. This will be incredibly useful if we want to bring more advanced compositional capabilities to p5.js.
I completely agree that MIDI messages in general are not designed for humans, and p5.js definitely needs to be more human-centric! I like the Tone.js JSON representation you pointed at. I picked out this from their example, it seems like they use this representation for a single note:
// Tone.js JSON note representation
midi: Number,               // midi number, e.g. 60
time: Number,               // absolute time from song start, in seconds
note: String,               // note name, e.g. "C4"
velocity: Number,           // normalized 0-1 velocity
duration: Number,           // duration between noteOn and noteOff

which could be a great basis for a Note class in p5.js. This differs from how other notes are encoded in MIDI files, which uses
// MIDI message note representation
midi: Number,               // midi number, e.g. 60
time: Number,               // time delay from the previous note event in seconds
velocity: Number,           // normalized 0-1 velocity

and requires a second note with velocity = 0 (noteOff) to end the note, since there is no duration parameter. These are very different representations, and at first glance the Tone.js version seems far more intuitive. However, each has its pros and cons:
Tone.js

[PRO] The duration parameter is very useful and much more intuitive than having a separate noteOff event.
[CON] The duration of a note is not known in advance during a real-time scenario (when a player starts a note, for example in a browser keyboard application, we don't have information about how long he will hold it)
[CON] The use of absolute time makes the notes less portable - it only works in the context of a global song, so for instance we cannot just grab a random note from the middle of the song and play it "right now".

MIDI

[PRO] The use of relative time delay from previous note is more local, and we can handle notes without worrying about where it fits into our whole song. This allows for use cases like storing an array of notes, and sampling from the array randomly to make a song - which is a crucial way that I expect people will want to interact with notes.
[CON] Relative time delay FROM the previous note is far less intuitive than relative time delay TO the next note. (The reason MIDI does this is for the same reason about not knowing how long the delay is in a real-time scenario)
[CON] We have to remember to produce the corresponding noteOff event as well.

I will also note that our own synth classes also use the same absolute time representation, but while this is fine for note playback, I feel that individual note storage needs to be local. I would also point out that despite an absolute time representation, even the current documentation shows the user thinking in terms of time delays rather than absolute times:
monosynth.play(45,1,x=0,1);
monosynth.play(49,1,x+=1,0.25);

Unfortunately, the time parameter in general is not very intuitive to think about, but it is crucial to have it defined separately from duration, otherwise we cannot have polyphony. Based on this, I can see one possible option which has the best of both worlds - locality, intuitiveness and ease-of-use.
// Proposed note representation
midi: Number,               // midi number, e.g. 60
time: Number,               // relative time delay to the next note event
note: String,               // note name, e.g. "C4"
velocity: Number,           // normalized 0-1 velocity
duration: Number,           // duration between noteOn and noteOff

To me, this seems like the perfect scenario for a generative application, but it does not lend well to the idea of real-time performance recording (but I think there might be ways to work around this, such as filling the time and duration parameters with a special flag to indicate that the values are not known - in which case the synth should hold the note indefinitely?).

Great analysis here.
I think a p5.note class that draws inspiration from MIDI (where it is helpful) is an interesting idea. Though I also worry about adding too many new classes to the API, which can make it feel intimidating.
Time is definitely something to try to figure out, and something that I hope we can tackle as part of algorithmic examples.
Currently time is represented as "timeFromNow" across p5.sound. This is different from MIDI as you've pointed out which uses relative time delay to the next note event. It's also slightly different from Web Audio that uses AudioContext Time. In hindsight it would've been nice to be consistent with Web Audio time, similar to the way millis can be useful for visualization in p5.
The p5.soundLoop was meant to allow for experimentation with relative time delay. We do this by changing the interval until the next iteration of the sound loop. Here's an example.
I think it's important to be consistent with our representation of time within the library. So I hesitate to introduce new ways to represent time in the p5 sound API, unless it's part of a major update (and breaking change) across the entire library.

Thanks for the example, @therewasaguy ! This helps clarify the options we have. I think I'll focus on getting a simple algorithmic composition example working now, this ought to make clear what we have that works well, and where we might need new features to improve the programmer's experience.
Quick question, you mentioned something about presets for synths - is there an example for this? I still would like to try to incorporate pre-rendered soundfonts like @b2renger did simply because it provides a wealth of sound options quickly and easily, but I'd be interested to know if there are easier ways to get instrument sounds using the existing synth classes.

you mentioned something about presets for synths - is there an example for this?

@jvntf made great progress on this idea in this branch https://github.com/processing/p5.js-sound/commits/presets for example https://github.com/processing/p5.js-sound/commit/c4493b20a454219d07f5d5919a9903bb90e3977f#diff-559e243463d4dac174c5eee32f469a81R174

Thanks guys for the discussion! I'm going to close this issue for now, I plan to carry on some of these ideas in more focused threads.

Native support would be cool.  Not a lot of steps to integrate with browser built-in midi.
Example:
https://gist.github.com/eclewlow/96d9cc8628157842d56962880bbefdd3
References:
https://www.keithmcmillen.com/blog/making-music-in-the-browser-web-midi-api/
https://www.midi.org/17-the-mma/99-web-midi

Sorry this might all already be implemented with p5.PolySynth()


Unspecified error on production crashed server

We've a running app today it crashed while uploading a file.
on('error')
event was called but it doesn't give any info, just file and store info.
Log shows this:
TypeError: Uncaught, unspecified "error" event.
at TypeError (<anonymous>)
at EventEmitter.emit (events.js:74:15)
at [object Object].<anonymous> (packages/cfs:collection/common.js:94:1)
at [object Object].emit (events.js:106:17)
at Writable.<anonymous> (packages/cfs:storage-adapter/storageAdapter.server.js:203:1)
at Writable.emit (events.js:117:20)
at Response.<anonymous> (packages/cfs:s3/s3.upload.stream2.js:152:1)
at Request.<anonymous> (/opt/myapp/app/programs/server/npm/cfs:s3/node_modules/aws-sdk/lib/request.js:350:18)
at Request.callListeners (/opt/myapp/app/programs/server/npm/cfs:s3/node_modules/aws-sdk/lib/sequential_executor.js:100:18)
at Request.emit (/opt/myapp/app/programs/server/npm/cfs:s3/node_modules/aws-sdk/lib/sequential_executor.js:77:10)

Any idea on what could be the problem?
Any way to avoid this?
I've tried but can't find a way to reproduce it.
Thanks!

Its on the trello as a high priority - we need better error handling / in order to debug


[directx-dxc] Fix missing dxil.dll install

[x] Changes comply with the maintainer guide
[x] SHA512s are updated for each updated download
[x] The "supports" clause reflects platforms that may be fixed by this new version
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

Please run: . /vcpkg x-add-version {portname} and commit the changes.

@microsoft-github-policy-service agree

@microsoft-github-policy-service agree

Usage test pass with following triplets:
x86-windows
x64-windows
x64-windows-static

Is there anything to do on this PR before it can be merged?


class for js

@ben-scully
1, please give 'new game' button a class "newGame"
2, when user click 'new game' button,  JS will add a class 'hidden' to the button, so please style the button to disapear with the 'hidden' class
3, also when the result is ready to show, JS will add a class 'showResult' to the result element (like :div or whatever)
and @oliver-jk-redding  for your reference maybe

Also make note of the class names in the result template (it's been pushed and merged) for your styling :)


yarn remove does not trigger rebuild of webpack dll

yarn add works as expected. yarn remove does not trigger the rebuild.

In the scripts field of package.json, you can add "postuninstall": "npm run build:dll" to run rebuild again after remove a package.

Known issue with yarn about postuninstall scripts


Error: Calendar is read only

I get an error from the Alfred debugger
    NSLocalizedDescription = "The calendar is read only.";
}] (-10025)

Any idea how I can fix this?

Have you tried giving Alfred full access to the calendar?

Alfred and Terminal both have full access.
Here is the debugging from a test.
[10:56:08.099] NLP Calendar[Script Filter] Queuing argument 'meeting 5pm'
[10:56:08.275] NLP Calendar[Script Filter] Script with argv 'meeting 5pm' finished
[10:56:08.280] NLP Calendar[Script Filter] {"items": [{"title": "meeting", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 5:00 PM", "arg": "meeting 5pm", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:10.972] NLP Calendar[Script Filter] Queuing argument 'm'
[10:56:11.077] NLP Calendar[Script Filter] Script with argv 'm' finished
[10:56:11.083] NLP Calendar[Script Filter] {"items": [{"title": "m", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 10:56 AM", "arg": "m", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:11.092] NLP Calendar[Script Filter] Queuing argument 'me'
[10:56:11.193] NLP Calendar[Script Filter] Script with argv 'me' finished
[10:56:11.198] NLP Calendar[Script Filter] {"items": [{"title": "me", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 10:56 AM", "arg": "me", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:11.237] NLP Calendar[Script Filter] Queuing argument 'mee'
[10:56:11.359] NLP Calendar[Script Filter] Script with argv 'mee' finished
[10:56:11.366] NLP Calendar[Script Filter] {"items": [{"title": "mee", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 10:56 AM", "arg": "mee", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:11.481] NLP Calendar[Script Filter] Queuing argument 'meet'
[10:56:11.582] NLP Calendar[Script Filter] Script with argv 'meet' finished
[10:56:11.589] NLP Calendar[Script Filter] {"items": [{"title": "meet", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 10:56 AM", "arg": "meet", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:11.678] NLP Calendar[Script Filter] Queuing argument 'meeti'
[10:56:11.780] NLP Calendar[Script Filter] Script with argv 'meeti' finished
[10:56:11.787] NLP Calendar[Script Filter] {"items": [{"title": "meeti", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 10:56 AM", "arg": "meeti", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:11.916] NLP Calendar[Script Filter] Queuing argument 'meetin'
[10:56:12.021] NLP Calendar[Script Filter] Script with argv 'meetin' finished
[10:56:12.028] NLP Calendar[Script Filter] {"items": [{"title": "meetin", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 10:56 AM", "arg": "meetin", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:12.045] NLP Calendar[Script Filter] Queuing argument 'meeting'
[10:56:12.147] NLP Calendar[Script Filter] Script with argv 'meeting' finished
[10:56:12.153] NLP Calendar[Script Filter] {"items": [{"title": "meeting", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 10:56 AM", "arg": "meeting", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:12.318] NLP Calendar[Script Filter] Queuing argument 'meeting '
[10:56:12.420] NLP Calendar[Script Filter] Script with argv 'meeting ' finished
[10:56:12.426] NLP Calendar[Script Filter] {"items": [{"title": "meeting", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 10:56 AM", "arg": "meeting ", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:12.658] NLP Calendar[Script Filter] Queuing argument 'meeting 5'
[10:56:12.778] NLP Calendar[Script Filter] Script with argv 'meeting 5' finished
[10:56:12.785] NLP Calendar[Script Filter] {"items": [{"title": "meeting 5", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 5:00 AM", "arg": "meeting 5", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:13.186] NLP Calendar[Script Filter] Queuing argument 'meeting 5p'
[10:56:13.313] NLP Calendar[Script Filter] Script with argv 'meeting 5p' finished
[10:56:13.319] NLP Calendar[Script Filter] {"items": [{"title": "meeting 5p", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 10:56 AM", "arg": "meeting 5p", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:13.606] NLP Calendar[Script Filter] Queuing argument 'meeting 5pm'
[10:56:13.720] NLP Calendar[Script Filter] Script with argv 'meeting 5pm' finished
[10:56:13.726] NLP Calendar[Script Filter] {"items": [{"title": "meeting", "subtitle": "\ud83d\udcc5 Calendar \u2022 Today at 5:00 PM", "arg": "meeting 5pm", "valid": true, "icon": {"path": "icon.png"}}]}
[10:56:14.861] NLP Calendar[Script Filter] Processing complete
[10:56:14.872] NLP Calendar[Script Filter] Passing output 'meeting 5pm' to Run Script
[10:56:15.634] NLP Calendar[Run Script] Processing complete
[10:56:15.642] NLP Calendar[Run Script] Passing output 'Error: 585:675: execution error: Calendar got an error: Failed to save event [meeting] with error [{
    NSLocalizedDescription = "The calendar is read only.";
}] (-10025)
' to Post Notification

Hi, this issue occurred because calendar_nlp.py and calendar_profile.py fail to detect the default calendar. In your case, it defaults to the calendar profile with read-only access.
I've updated it with the new release. Please check it and try selecting your default calendar profile first using clprofile.


Comments in NAF seem to be (partially) unsupported

I get this error when adding a comments=True flag to the function naf_from_doc:
ValueError: Comment may not contain '--' or end with '-'

Should be fixed now! Could you try with the same data?


Error deploying with compose.

I can manually deploy with "Services" no problem. I then export the configs (ymls). I rebuild the cluster and use rancher compose to deploy and I get a ton of container creation errors like :
Message : 1bb798b6-69e4-47b3-9b04-aaa17465f6ac : Image [clemenko/demo_app1:latest] failed to pull : Error pulling image (latest) from clemenko/demo_app1, Driver overlay failed to create image rootfs 87de57de6955d5e098c7ae30ce27368be101a109d03dacabce20608e17808985: lstat /var/lib/docker/overlay/bf84c1d84a8fbea92675f0e8ff61d5b7f484462c4c44fd59f0fdda8093620024: no such file or directory
I am using Google Compute Engine, coreos, docker 1.6.2, rancher v0.25-rc2.
My scripts are located at : https://github.com/clemenko/rancher_deploy
Specifically gce.sh

Could you provide the docker-compose.yml, rancher-compose.yml and the commmand that you ran? I haven't been able to reproduce.
Also, could you try running it with debug on? --debug and paste the full results?

Script that I ran on a Mac :
server=<IP_ADDRESS>
api_string=$(curl -s 'http://'$server':8080/v1/apikey?projectId=1a5' -X POST -H 'Accept: application/json' --data {"type":"apikey"})
access_key=$(echo $api_string|sed 's/\\////g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/":"/|/g' | sed 's/[,]/ /g' | sed 's/"//g'|grep publicValue|awk -F"|" '{print $2}')
secret_key=$(echo $api_string|sed 's/\\////g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/":"/|/g' | sed 's/[,]/ /g' | sed 's/"//g'|grep secretValue|awk -F"|" '{print $2}')
./rancher-compose --url http://:8080$server/v1/projects/1a5 --access-key $access_key --secret-key $secret_key --debug --project-name Default up
scripts and ymls : https://github.com/clemenko/rancher_deploy
Nothing new outputs from rancher compose.
I am seeing the same errors.
I can build the services correctly individually.
Actually it appears that using rancher-compose it is failing to deploy to the first two rancher-agents.

I will leave the environment up at : http://<IP_ADDRESS>:8080

Can you try logging into one of those hosts (that are having the error) to see if you can do a docker pull of the image?

6612f074f2ff: Error pulling image (latest) from clemenko/demo_app1, Driver overlay failed to create image rootfs 87de57de6955d5e098c7ae30ce27368be101a109d03dacabce20608e17808985: lstat /var/lib/docker/overlay/bf84c1d84a8fbea92675f0e8ff61d5b7f484462c4c44fd59f0fdda8093620024: no such file or directory f0fdda8093620024: no such file or directory mplete
FATA[0005] Error pulling image (latest) from clemenko/demo_app1, Driver overlay failed to create image rootfs 87de57de6955d5e098c7ae30ce27368be101a109d03dacabce20608e17808985: lstat /var/lib/docker/overlay/bf84c1d84a8fbea92675f0e8ff61d5b7f484462c4c44fd59f0fdda8093620024: no such file or directory

If you can't do a docker pull on the host, then it looks like there is something wrong with the host (outside of Rancher). Maybe Docker related?
Rancher appears to be working fine in the sense that it will try to deploy on all the hosts, it will eventually find the hosts that are working and deploy the number of containers in the service.

Does Rancher Compose write anything to /var/lib/docker/tmp? That was full on the first two servers.

Not that I could tell, but I'd have to defer to @ibuildthecloud for the correct answer.

I run my build script. Run docker pull on the nodes and it works ok. When I don't pre-pull the images it fails ONLY on the first two nodes. I will investigate more.

I spoke with @ibuildthecloud. Docker uses /var/lib/docker/tmp, but Rancher does not. Unfortunately, I won't be able to help with the docker issue. :)

Odd that native docker commands work. Even the rancher GUI works. But rancher compose causes this error. It looks like /var/lib/docker/tmp may not be a problem. It becomes a problem from all the failed pulls.

Doing a docker run on the same host doesn't have an issue?

Nope. The errors only show up when running rancher-compose.

I have confirmed the error shows up on BOTH CoreOS and CentOS. I have more testing.

The latest test is to build 4 agent nodes. Usually the error shows up on the first two nodes. I successfully ran docker pull on the first node. I am still seeing the error on the second node. I am testing again to see if it is isolated to the host "1h1", 1h2, 1h3 naming.

rebuilt, 1h2, and 1h3 were the errored servers.

My work around is to run docker pull before the rancher compose command. NO issues now. Check out my github repo/scripts.

@clemenko We just released a beta version of rancher-compose. I know how much you like bleeding edge. :) I still don't understand why you wouldn't be able to do a docker pull on the images on the 2 agents, but if it's happening again, please re-open.

Error still exists. I even created brand new docker-compose.yml and rancher-compose.yml

@clemenko Could you try with ubuntu instead of CoreOS? Also, not sure if you'd want to try Docker 1.7.1 that just came out.

I tested with an ubuntu box for rancher-compose and ubuntu hosts and got the same issue.

here is my docker-compose : app2:
tty: true
image: clemenko/demo_app2
links:

redis:redis
stdin_open: true
app1:
tty: true
image: clemenko/demo_app1
links:
redis:redis
stdin_open: true
redis:
restart: 'no'
tty: true
image: redis
stdin_open: true
lb1:
ports:
'80'
tty: true
image: rancher/load-balancer-service
links:
app1:app1
stdin_open: true

rancher-compose :
app2:
scale: 4
app1:
scale: 4
redis:
scale: 1
lb1:
scale: 4
load_balancer_config:
name: lb1 config

I have seen similar errors with smaller images (nginx/httpd) :
app2:
tty: true
image: nginx
links:

redis:redis
stdin_open: true
app1:
tty: true
image: httpd
links:
redis:redis
stdin_open: true
redis:
restart: 'no'
tty: true
image: redis
stdin_open: true
lb:
ports:
'80'
tty: true
image: rancher/load-balancer-service
links:
app1:app1
stdin_open: true

All these errors seem like odd Docker errors with graphdrivers.  Are you sure the exact same docker pull command works on these servers?

Yup it works. Check out the function "demo"  I am using to curl the api for the keys and rancher-compose command. https://github.com/clemenko/rancher_deploy/blob/master/gce_core_rancher.sh
With the docker pull command it runs every time. Without it fails randomly.

@clemenko I am planning on closing the issues in this repo to consolidate into the rancher repo. I know there is a lot of history in this issue, but once I close the issues, it won't be visible. I believe this is a random docker issue with pulling issues. I don't believe it's a rancher-compose issue. If you still have issues, please open up in rancher.


[CARE-5792] Fix - Rework Table element to not rely on context

Hey @yuriyyakym, I've reworked the approach to use the transformation as suggested. Looks much better now I think.


add note about null AZs on Azure [#157055189]

Signed-off-by: Robert Sullivan<EMAIL_ADDRESS>

This looks good.


Node-Before-After files to be added to the codebase

Only two requests.

My bad. Previously made changes did not apply on this branch. Code updated. Thanks!


Merge projects

I'm also have a side project for io_uring and have run into many of the same issues. I was wondering if you wanted to create a Github organization and work together on it. Let me know what you think, really cool project!

hey @hodgesds . my original goal was to make a proposal for integration with golang runtime.
this library was a playground/showcase in order to understand if it is possible to use uring efficiently with goroutines and in general to learn more about uring io specifics.
beside several surprises (e.g serialized writes to same fd, slower buffered reads) i am satisfied with an experiment, and want to move forward with POC for runtime integration. maybe proposal won't be even accepted, and it definitely won't be generic enough to cover every use-case. so there is still a place for a good library, but at this time i don't plan any big work on external library (maybe some perf improvements, if i will find them).

That makes sense, I sort of ran into similar feelings as well. Without integration into the runtime I think it will be hard to be as efficient as other uring implementations. I think the right path forward is mapping a ring per runtime "M". The question becomes how to you interact/configure the ring with the runtime without exposing too much of the internals. I'm guessing the Go team aren't really looking to expose everything needed to configure the rings appropriately or even what that API would look like (environment vars, runtime functions, etc).

What kind of things you would like to configure?
The thing is that IOPOLL/SQPOLL are out the picture if this will be integrated per cpu or per golang processor.
SQ size can be small, because it will be reused after each enter.
CQ size can be large by default, and double if necessary with some dance.
The only other flag that matters if for sharing worker pool, but it will be on by default.

I was mostly concerned about queue sizes because if the mlock value is too small then setting up the ring might fail, which I guess it could always fall back? When I get some more time I think I might try to make a netpoller and see if I can make that work.


docs(examples): supplychain microk8s deployment files

Description
As a contributor/maintainer I want to be able to have an easy way of deploying the supplychain example to a microk8s Kubernetes cluster so that I can see a good example of how projects using Cactus can be operated in a data center environment.
We start with microk8s because that's the default Ubuntu k8s distro and our CI pipeline is built on top of Ubuntu already so this should eliminate a few extra complexities compared to focusing on other k8s distros instead since microk8s could be tested on a Windows computer as well via WSL 2
While working on this issue it is likely that other shortcomings will be unearthed in the code-base, things that could warrant their own user stories/bugs as well and therefore the classification of this issue is "400-expert" because there's no major change in the code, but you'll need to be an expert in a few different things to progress with this.
Acceptance Criteria
There's either a Helm chart defined or a KNative file defined and a top level npm script that performs the deployment to a microk8s Kubernetes cluster
The process of performing the deployment/testing it out is documented to the extent that it should be workable from the point where someone has an Ubuntu machine fresh out of the oven so to speak. For setting up the k8s cluster itself it's sufficient to reference the documentation of microk8s itself, but the specific addons we need have to be documented of course.
cc: @takeutak @sfuji822 @hartm @jonathan-m-hamilton @AzaharaC @jordigiam @kikoncuo @petermetz  @arnab-roy

Linking the spike issue with the same contents #911


The HTML report has visual glitches (stairs effect) with the .NET 8 SDK

Description
The VSTest HTML report renders with visual stairs glitch, better described with this screenshot.

Steps to reproduce

Checkout https://github.com/serilog-contrib/serilog-formatting-log4net/commit/3b1bdae0c71c4ec373d95e6fa8d53f42eb27c099

Update the global.json file from 6.0.400 to 8.0.100

Run dotnet test

Open the coverage/net6.0/results/TestResults-net6.0.html file, which should be equivalent to the above screenshot.

Expected behavior
The HTML report renders nicely. Note that this is what happens withe the .NET 6 SDK (i.e. without touching the global.json file).

Wow, did not know we have an opposite Star Wars mode.

Reverting this commit fixes the issue:
https://github.com/microsoft/vstest/pull/4576/files

The issue was that with the new writer the settings from xslt were not copied, and it output a self-closing  tag when there was no content in error-info. This is a valid XML syntax but not valid HTML syntax.

Same issue solved on SO: https://stackoverflow.com/questions/71357056/c-sharp-xslt-transformer-uses-self-closing-html-tags-which-are-invalid