May be a BOM issue?

Issue summary
Hi,
I have been playing with importing a csv file. It has came to my notice that there is some sort of issue with μ character. May be it is something else I am wrong.
System informations

Information
Description

League\Csv version
8.1.1

PHP/HHVM version
PHP 7.0.8-0ubuntu<IP_ADDRESS> (cli) ( NTS )

OS Platform
ubuntu<IP_ADDRESS>

Standalone code, or other way to reproduce the problem
You can download the csv file from http://www.data-explorer.com/content/data/standard-model-of-particle-physics-csv.zip
<?php
require __DIR__ . '/vendor/autoload.php';

use League\Csv\Reader;

$file = "/path/to/physics/file.csv";

$reader = Reader::createFromPath($file);
$reader->stripBom(true);

echo "<pre>";

//get the first row, usually the CSV header
$headers = $reader->fetchOne(0);
print_r($headers);

$reader->addFilter(function ($row, $index) {
    return $index > 0;
});

$rows = $reader->fetch();

foreach ($rows as $row) {
    $content = array_combine($headers, $row);
    $json = json_encode($content);

    if ($json === FALSE) {
        echo var_dump($content);
        exit;
    }
}

Expected result
I am not sure what would be the appropriate character for μ .
Actual result
array(21) {
  ["ID"]=>
  string(3) "9"
  ["Name"]=>
  string(9) "Muon"
  ["Symbol"]=>
  string(3) "�"
 ....
}

There is actually more on the same csv which have similar issues.
Thank you.

@harikt there's nothing wrong with the library. By default the library thinks you are using a UTF-8 document. In your case the document is not UTF-8. Here's what I did:
require 'vendor/autoload.php';

use League\Csv\Reader;

$file = __DIR__."/file.csv";

$reader = Reader::createFromPath($file);
$input_bom = $reader->getInputBOM();
//in your case $input_bom === Reader::BOM_UTF16_LE
//so I've just added this following line (I assume that the iconv extension is installed on your server)
$reader->appendStreamFilter('convert.iconv.UTF-16/UTF-8'); //converting to UTF-8
$reader->stripBOM(true);
echo "<pre>";
//get the first row, usually the CSV header
$headers = $reader->fetchOne(0);
$rows = $reader->setOffset(1)->fetchAssoc($headers); //Simpler than using the Reader::addFilter method
foreach ($rows as $row) {
    echo json_encode($row, JSON_PRETTY_PRINT), PHP_EOL;
}

@nyamsprod Thank you for your quick help.
A few things I noticed are
require 'vendor/autoload.php';

use League\Csv\Reader;

$file = __DIR__."/file.csv";

$reader = Reader::createFromPath($file);
$reader->appendStreamFilter('convert.iconv.UTF-16/UTF-8');
$reader->stripBOM(true);
$headers = $reader->fetchOne(0);

Adding $reader->appendStreamFilter('convert.iconv.UTF-16/UTF-8'); creates a warning as below.
PHP Notice:  Uninitialized string offset: 0 in /var/www/projects/vendor/league/csv/src/Modifier/QueryFilter.php on line 225

Notice: Uninitialized string offset: 0 in /var/www/projects/vendor/league/csv/src/Modifier/QueryFilter.php on line 22

Also closely looking the id keys got missing. See below.
{
    "": "17",

Thank you

@harikt strange I failed to reproduce this bug and I'm also using the 16.04 LTS
<?php

error_reporting(-1);
ini_set('display_errors', '1');

require 'vendor/autoload.php';

use League\Csv\Reader;

$file = __DIR__."/test.csv";

$reader = Reader::createFromPath($file);
$reader->appendStreamFilter('convert.iconv.UTF-16/UTF-8');
$reader->stripBOM(true);

echo "<pre>";
foreach ($reader->fetchAssoc(0) as $row) {
    echo json_encode($row, JSON_PRETTY_PRINT), PHP_EOL;
}

I hope you could re-produce this now. This is when you call $reader->getInputBOM(); .
<?php

error_reporting(-1);
ini_set('display_errors', '1');

require 'vendor/autoload.php';

use League\Csv\Reader;

$file = __DIR__."/test.csv";

$reader = Reader::createFromPath($file);
$input_bom = $reader->getInputBOM();
if ( $input_bom === Reader::BOM_UTF16_LE || $input_bom === Reader::BOM_UTF16_BE ) {
    echo "BOM UTF-16 " . PHP_EOL;
    $reader->appendStreamFilter('convert.iconv.UTF-16/UTF-8');
}
$reader->stripBOM(true);
foreach ($reader->fetchAssoc(0) as $row) {
    // echo json_encode($row, JSON_PRETTY_PRINT), PHP_EOL;
}

@harikt thanks I found the issue ... I'll release a patch to fix this when I have time :+1:

Good to hear you was able to reproduce the same.
And thank you very much for the valuable support you have provided.
I wasn't able to find the reason to send a PR. Sorry about that :( .

Also in the case as I mentioned when calling $reader->getInputBOM() the ID was null on json_encode data.

Also in the case as I mentioned when calling $reader->getInputBOM() the ID was null on json_encode data.

I've looked into this issue and in fact again there's no issue. The thing is if you use stream filter you don't need to explicitly call the stripBOM method.
The conversion is done prior to generate the array row from the CSV document so the BOM sequence is already removed.  (I didn't know this, since I've never really used the stream filter on BOM document).
So the code should be:
<?php

error_reporting(-1);
ini_set('display_errors', '1');

require 'vendor/autoload.php';

use League\Csv\Reader;

$file = __DIR__."/file.csv";

$reader = Reader::createFromPath($file);
$input_bom = $reader->getInputBOM();
if ($input_bom === Reader::BOM_UTF16_LE || $input_bom === Reader::BOM_UTF16_BE) {
    $reader->appendStreamFilter('convert.iconv.UTF-16/UTF-8');
}
foreach ($reader->fetchAssoc(0) as $row) {
    echo json_encode($row, JSON_PRETTY_PRINT), PHP_EOL;
}

The stripBOM method should only be used if stream filter can not be used to convert your CSV document to UTF-8.
I should document this into the library documentation.. or you could do it 👍

Thank you @nyamsprod .
I will look into sending a PR regarding the same. Please give me some time.
Thank you.

version 8.1.2 is released with the fix


"Quick Open Files"  is slow

it got slower with the 1.18 version. There is a delay when searching a file.

What OS?
Can you tell me about your workspace? Is it very large? Are there symlinks?

Ubuntu 16.04
Yes it is a large workspace, but i am comparing the same workspace with 1.17

hw. specs;
i-7,  32g ram, ssd

vscode118.mp4.zip
here is a little cast

vscode117.mp4.zip
And this one is with 1.17, hope these help

I think the difference might be that in the 1.18 recording the cache wasn't populated yet.
After opening VS Code the first search will be slower because the cache is still empty, subsequent searches are faster because they can get results from the cache. This hasn't changed between 1.17 and 1.18.
Could you confirm that both versions show some delay on the first search, but not on the second?

Nope, 1.17 is faster on both first and second searches. Here is another cast comparing both.
vscode118vs117.mp4.zip

btw. i just tried insider and it is fast again. I hope it will be so in 1.18.2

Are you on 64- or 32-bit Linux?

64

I'm not aware of any changes in 1.19-insiders that would affect QuickOpen performance. (Since you mention it is fast there.)

I've excluded some folders and it is faster. But, still i've no explanation why it is faster on a fresh installation without any config.
Anyway, since no one else had the same problem. I'm closing the issue. Thanks.


Use StringLiteral For Discriminator When Generating TypeScript Client

I'm using JsonInheritanceConverter (as described here) in order to support inheritance with my service definition and generate a TypeScript client.
With the client side generated code, I see the discriminator field type is a simple string which doesn't provide type checking and error validation (considering service type has changed, no code will be break on the client side).
Is it possible to generate the client in a way the discriminator possible values will be StringLiterals?

In case of generating classes you should not set this manually but it's set via the used class.
So i think this is about the interface output?

Thanks for replying.
Yes, the type output type set to interface.

I think discriminated unions can be leveraged here.
So instead of generating
export interface Event {
    discriminator: string;
}

export interface FooEvent extends Event {
    foo: number
}

export interface BarEvent extends Event {
    bar: string;
}

we could generate
export type Event = FooEvent | BarEvent;

export interface FooEvent {
    discriminator: 'FooEvent',
    foo: number
}

export interface BarEvent {
    discriminator: 'BarEvent',
    bar: string;
}

I've modified the Interface template to try this out, and it seems to work. It doesn't deal with nested inheritance, or with shared properties on the abstract type. Do you think we should look further into this method?
{%- if HasDescription -%}
/** {{ Description }} */
{%- endif -%}
{% if IsAbstract -%}
{% if ExportTypes %}export {% endif %}type {{ ClassName }} =
    {{ DerivedClasses | map: "ClassName" | join: "
    | " }};
{% else -%}
{% if ExportTypes %}export {% endif %}interface {{ ClassName }} {
{% if HasBaseDiscriminator -%}
    {{ BaseDiscriminator }}: "{{ DiscriminatorName }}";
{% endif -%}
{%- for property in Properties -%}
{%-     if property.HasDescription -%}
    /** {{ property.Description }} */
{%-     endif -%}
    {% if property.IsReadOnly %}readonly {% endif %}{{ property.InterfaceName }}{% if property.IsOptional %}?{% endif %}: {{ property.Type }}{{ property.TypePostfix }};
{%- endfor -%}
{%- if HasIndexerProperty -%}

    [key: string]: {{ IndexerPropertyValueType }};
{%- endif -%}
}
{% endif -%}

I want to use discriminated unions. The code from @stijnherreman is a very good start. We had shared properties on our base abstract type, so we needed to handle those. We also had some abstract types that were not discriminated unions, so we also need to handle that. I think that we need more information in the template to handle all cases, meaning NJsonSchema might need to be updated.
This is what we ended up with. The _Interface hack will not work if a type implements multiple interfaces. It also doesn't handle nested inheritance.
{%- if HasBaseDiscriminator %}{% if IsAbstract %}{%- assign IsDiscriminatedUnion = true -%}
{%- else -%}{% assign IsDiscriminatedUnionMember = true %}{%  endif %}{%- endif -%}
{%- if HasDescription -%}
/** {{ Description }} */
{%- endif -%}
{% if IsDiscriminatedUnion -%}
export type {{ ClassName }} = {{ DerivedClasses |  map: "ClassName" |  join: " | " }};
{%- endif -%}
{%- if ExportTypes %}export {% endif %}interface {{ ClassName }}{% if IsDiscriminatedUnion %}_Interface{% endif %}{{ Inheritance }}{% if IsDiscriminatedUnionMember %}_Interface{% endif %} {
{%- if IsDiscriminatedUnionMember -%}
    {{ BaseDiscriminator }}: '{{ DiscriminatorName }}';
{%- endif -%}
{%- for property in Properties -%}
{%-     if property.HasDescription -%}
    /** {{ property.Description }} */
{%-     endif -%}
    {% if property.IsReadOnly %}readonly {% endif %}{{ property.InterfaceName }}{% if property.IsOptional %}?{% endif %}: {{ property.Type }}{{ property.TypePostfix }};
{%- endfor -%}
{%- if HasIndexerProperty -%}

    [key: string]: {{ IndexerPropertyValueType }};
{%- endif -%}
}
{%- if GenerateTypeCheckFunctions and HasInheritance -%}

{% if ExportTypes %}export {% endif %}function is{{ ClassName }}(object: any): object is {{ ClassName }} {
    return object && object['{{ BaseDiscriminator }}'] === '{{ DiscriminatorName }}';
}
{%- endif -%}

example result:
export type MyResponse = Success | Failure;
export interface MyResponse_Interface {
  type: string;
  code: number;
}

export interface Success extends MyResponse_Interface {
  type: 'success';
  result: string;
}

export interface Failure extends MyResponse_Interface {
  type: 'failure;
  reason: string;
}


ImageMagick err

[Error: Could not execute GraphicsMagick/ImageMagick: gm "identify" "-ping" "-format" "%wx%h" "./headp.png" this most likely means the gm/convert binaries can't be found]
/Users/xxx/node_modules/js2image/utils/image-to-struct.js:19
var content = converted.replace(/\S[0m/g,"").replace(/\n/g,"",\n"");

没有安装 GraphicsMagick 依赖


add: Dominik Gronkiewicz

Description
[Include a brief description of the changes you've made]
Related Issues
[Include any related issues or pull requests that this PR addresses or is related to. Use GitHub's shorthand syntax to link to them, like #1234.]
Changes Proposed
[Describe the changes you've made in detail. Be specific and include any relevant code snippets.]
Checklist

[ ] I have read and followed the Contribution Guidelines.
[ ] All new and existing tests passed.
[ ] I have updated the documentation to reflect the changes I've made.
[ ] My code follows the code style of this project.
[ ] The title of my pull request is a short description of the requested changes.

Screenshots
Note to reviewers

@dominikgronkiewicz Thank you so much for your contribution! Your efforts are greatly appreciated and will go a long way in improving our project. Please feel free to share it with others and Star the repo to help us grow even further!


Add a command to run whole file through a shell script

Eh, I don't think this is actually worth doing considering you can easily just back out to the shell and run whatever you need.  Maybe I can revisit this later.


Issue in the documentation invalid file paths

I searched for the file path and it didnt exist

Paths should be correct now. Thanks!

You are welcome sir
On Sat, Jul 29, 2017 at 1:56 PM, Rüdiger Lunde<EMAIL_ADDRESS>wrote:

Paths should be correct now. Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aimacode/aima-java/issues/327#issuecomment-318813591,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AYXHtUqcrWqzZLk6J-Cf2k9yFlLyKwdEks5sSuzJgaJpZM4Oilyh
.


remove references to v1beta3 in phabricator example

#9286 #9635
@nikhiljindal @bgrant0607

Will fix the failed Shippable.

@k8s-bot test this please


makefile fails to detect NDK directory

Hi,
The Makefile fails to detect the NDK directory on my machine (Linux Mint 19 Cinnamon), with Android Studio 3.6.1 . The NDK when installed via Android Studio SDK Manager, is located in "~ /Android/Sdk/ndk-bundle/", not "~/Android/Sdk/ndk/". Ran into some other issues as well, but they seem to be fixed already yesterday.
Awesome work by the way, keep up the good work!

Thought this was awesome too.
I only  have a one plus one running android 6 still. So api level 23. But got it working.
I needed to change the api level to 23 (and download the sdk) otherwise the app would build but crash upon startup. However after going to level 23 I also needed to remove this in the AndroidManifest.xml. After that a make push run indeed starts the app on good ol android marshmallow and works fine.
I don't have time to create a proper pull request. This is my makefile that works on latest macos 10.15.4 and android studio v3.6.3:
Makefile (hardcoded some paths and set api level to 23 to make it work on my setup)
`
➜  rawdrawandroid git:(master) ✗ ls /Users/wschrep/Library/Android/sdk/
build-tools    cmdline-tools  emulator       extras         fonts          licenses       ndk            patcher        platform-tools platforms      sources        tools
➜  rawdrawandroid git:(master) ✗ ls /Users/wschrep/Library/Android/sdk/ndk
21.1.6352462
➜  rawdrawandroid git:(master) ✗ ls /Users/wschrep/Library/Android/sdk/ndk/21.1.6352462/platforms/android-/Users/wschrep/Library/Android/sdk/
➜  rawdrawandroid git:(master) ✗ ls
AndroidManifest.xml       README.md                 android_native_app_glue.h errors2.txt               rawdraw
LICENSE                   Sources                   android_usb_devices.c     makecapp                  temp.apk
Makefile                  aligned.apk               android_usb_devices.h     makecapp.apk              test.c
Makefile.back             android_native_app_glue.c error.txt                 my-release-key.keystore
➜  rawdrawandroid git:(master) ✗ vi Makefile
➜  rawdrawandroid git:(master) ✗ adb devices
List of devices attached
1f234385	device
➜  rawdrawandroid git:(master) ✗ make
➜  rawdrawandroid git:(master) ✗ vi Makefile
➜  rawdrawandroid git:(master) ✗ make push run
echo "Installing" sitweb.eu.rawdraw
Installing sitweb.eu.rawdraw
/Users/wschrep/Library/Android/sdk/platform-tools/adb install makecapp.apk
Performing Push Install
makecapp.apk: 1 file pushed, 0 skipped. 4.8 MB/s (20294 bytes in 0.004s)
pkg: /data/local/tmp/makecapp.apk
Failure [INSTALL_FAILED_ALREADY_EXISTS]
/Users/wschrep/Library/Android/sdk/platform-tools/adb shell am start -n sitweb.eu.rawdraw/android.app.NativeActivity
Starting: Intent { cmp=sitweb.eu.rawdraw/android.app.NativeActivity }
➜  rawdrawandroid git:(master) ✗ make clean
rm -rf temp.apk makecapp.apk makecapp aligned.apk
➜  rawdrawandroid git:(master) ✗ make
mkdir -p makecapp/lib/armeabi-v7a
/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi23 -clang -ffunction-sections -Os -fdata-sections -Wall -fvisibility=hidden -Os -DANDROID -DAPPNAME="rawdraw" -I./rawdraw -I/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462/sysroot/usr/include -I/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462/sysroot/usr/include/android -fPIC -I. -m32 -o makecapp/lib/armeabi-v7a/librawdraw.so test.c android_native_app_glue.c -Wl,--gc-sections -s -lm -L/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/23  -lGLESv3 -lEGL -landroid -llog -shared -uANativeActivity_onCreate
make: /Users/wschrep/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi23: No such file or directory
make: *** [makecapp/lib/armeabi-v7a/librawdraw.so] Error 1
➜  rawdrawandroid git:(master) ✗ ls /Users/wschrep/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi2
➜  rawdrawandroid git:(master) ✗ vi Makefile
➜  rawdrawandroid git:(master) ✗ make
mkdir -p makecapp/lib/armeabi-v7a
/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi23-clang -ffunction-sections -Os -fdata-sections -Wall -fvisibility=hidden -Os -DANDROID -DAPPNAME="rawdraw" -I./rawdraw -I/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462/sysroot/usr/include -I/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462/sysroot/usr/include/android -fPIC -I. -m32 -o makecapp/lib/armeabi-v7a/librawdraw.so test.c android_native_app_glue.c -Wl,--gc-sections -s -lm -L/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/23 -lGLESv3 -lEGL -landroid -llog -shared -uANativeActivity_onCreate
mkdir -p makecapp/assets
echo "Test asset file" > makecapp/assets/asset.txt
rm -rf temp.apk
/Users/wschrep/Library/Android/sdk/build-tools/29.0.3/aapt package -f -F temp.apk -I ~/Library/Android/sdk/platforms/android-23/android.jar -M AndroidManifest.xml -S Sources/res -A makecapp/assets -v --target-sdk-version 23
Found 1 custom asset file in makecapp/assets
Configurations:
(default)
Files:
assets/asset.txt
Src: () makecapp/assets/asset.txt
mipmap/icon.png
Src: () Sources/res/mipmap/icon.png
values/strings.xml
Src: () Sources/res/values/strings.xml
AndroidManifest.xml
Src: () AndroidManifest.xml
Resource Dirs:
Type values
values/strings.xml
Src: () Sources/res/values/strings.xml
Type mipmap
mipmap/icon.png
Src: () Sources/res/mipmap/icon.png
Including resources from package: /Users/wschrep/Library/Android/sdk/platforms/android-23/android.jar
applyFileOverlay for drawable
applyFileOverlay for layout
applyFileOverlay for anim
applyFileOverlay for animator
applyFileOverlay for interpolator
applyFileOverlay for transition
applyFileOverlay for xml
applyFileOverlay for raw
applyFileOverlay for color
applyFileOverlay for menu
applyFileOverlay for font
applyFileOverlay for mipmap
Processing image: Sources/res/mipmap/icon.png
(processed image Sources/res/mipmap/icon.png: 100% size of source)
(new resource id icon from mipmap/icon.png #generated)
AndroidManifest.xml:4: error: No resource identifier found for attribute 'requestLegacyExternalStorage' in package 'android'
make: *** [makecapp.apk] Error 1
➜  rawdrawandroid git:(master) ✗ grep requestLegacyExternalStorage *
AndroidManifest.xml:
grep: Sources: Is a directory
grep: makecapp: Is a directory
grep: rawdraw: Is a directory
➜  rawdrawandroid git:(master) ✗ vi AndroidManifest.xml
➜  rawdrawandroid git:(master) ✗ cp AndroidManifest.xml AndroidManifest.xml.back
➜  rawdrawandroid git:(master) ✗ vi AndroidManifest.xml
➜  rawdrawandroid git:(master) ✗ make
mkdir -p makecapp/assets
echo "Test asset file" > makecapp/assets/asset.txt
rm -rf temp.apk
/Users/wschrep/Library/Android/sdk/build-tools/29.0.3/aapt package -f -F temp.apk -I ~/Library/Android/sdk/platforms/android-23/android.jar -M AndroidManifest.xml -S Sources/res -A makecapp/assets -v --target-sdk-version 23
Found 1 custom asset file in makecapp/assets
Configurations:
(default)
Files:
assets/asset.txt
Src: () makecapp/assets/asset.txt
mipmap/icon.png
Src: () Sources/res/mipmap/icon.png
values/strings.xml
Src: () Sources/res/values/strings.xml
AndroidManifest.xml
Src: () AndroidManifest.xml
Resource Dirs:
Type values
values/strings.xml
Src: () Sources/res/values/strings.xml
Type mipmap
mipmap/icon.png
Src: () Sources/res/mipmap/icon.png
Including resources from package: /Users/wschrep/Library/Android/sdk/platforms/android-23/android.jar
applyFileOverlay for drawable
applyFileOverlay for layout
applyFileOverlay for anim
applyFileOverlay for animator
applyFileOverlay for interpolator
applyFileOverlay for transition
applyFileOverlay for xml
applyFileOverlay for raw
applyFileOverlay for color
applyFileOverlay for menu
applyFileOverlay for font
applyFileOverlay for mipmap
Processing image: Sources/res/mipmap/icon.png
(processed image Sources/res/mipmap/icon.png: 100% size of source)
(new resource id icon from mipmap/icon.png #generated)
Creating 'temp.apk'
Writing all files...
'AndroidManifest.xml' (compressed 62%)
'assets/asset.txt' (not compressed)
'res/mipmap/icon.png' (not compressed)
'resources.arsc' (not compressed)
Generated 4 files
Included 0 files from jar/zip files.
Checking for deleted files
Done!
unzip -o temp.apk -d makecapp
Archive:  temp.apk
inflating: makecapp/AndroidManifest.xml
extracting: makecapp/assets/asset.txt
extracting: makecapp/res/mipmap/icon.png
extracting: makecapp/resources.arsc
rm -rf makecapp.apk
cd makecapp && zip -D9r ../makecapp.apk .
adding: resources.arsc (deflated 73%)
adding: res/mipmap/icon.png (deflated 8%)
adding: AndroidManifest.xml (deflated 62%)
adding: lib/armeabi-v7a/librawdraw.so (deflated 53%)
adding: assets/asset.txt (stored 0%)
jarsigner -sigalg SHA1withRSA -digestalg SHA1 -verbose -keystore my-release-key.keystore -storepass password makecapp.apk standkey
adding: META-INF/MANIFEST.MF
adding: META-INF/STANDKEY.SF
adding: META-INF/STANDKEY.RSA
signing: resources.arsc
signing: res/mipmap/icon.png
signing: AndroidManifest.xml
signing: lib/armeabi-v7a/librawdraw.so
signing: assets/asset.txt

Signer
X.509, CN=example.com, OU=ID, O=Example, L=Doe, ST=John, C=GB
[trusted certificate]

jar signed.
Warning:
The signer's certificate is self-signed.
rm -rf aligned.apk
/Users/wschrep/Library/Android/sdk/build-tools/29.0.3/zipalign -v 4 makecapp.apk aligned.apk
Verifying alignment of aligned.apk (4)...
50 META-INF/MANIFEST.MF (OK - compressed)
409 META-INF/STANDKEY.SF (OK - compressed)
836 META-INF/STANDKEY.RSA (OK - compressed)
2007 resources.arsc (OK - compressed)
2362 res/mipmap/icon.png (OK - compressed)
2524 AndroidManifest.xml (OK - compressed)
3412 lib/armeabi-v7a/librawdraw.so (OK - compressed)
19524 assets/asset.txt (OK)
Verification succesful
ls -l aligned.apk
-rw-r--r--  1 wschrep  staff  20208 May 18 22:12 aligned.apk
➜  rawdrawandroid git:(master) ✗ vi Make
➜  rawdrawandroid git:(master) ✗ vi Makefile
➜  rawdrawandroid git:(master) ✗ make push run
echo "Installing" sitweb.eu.rawdraw
Installing sitweb.eu.rawdraw
/Users/wschrep/Library/Android/sdk/platform-tools/adb install makecapp.apk
Performing Push Install
makecapp.apk: 1 file pushed, 0 skipped. 22.3 MB/s (20206 bytes in 0.001s)
pkg: /data/local/tmp/makecapp.apk
Failure [INSTALL_FAILED_ALREADY_EXISTS]
/Users/wschrep/Library/Android/sdk/platform-tools/adb shell am start -n sitweb.eu.rawdraw/android.app.NativeActivity
Starting: Intent { cmp=sitweb.eu.rawdraw/android.app.NativeActivity }
➜  rawdrawandroid git:(master) ✗ adb uninstall sitweb.eu.rawdraw
Success
➜  rawdrawandroid git:(master) ✗ adb uninstall makecapp.apk
Failure [DELETE_FAILED_INTERNAL_ERROR]
➜  rawdrawandroid git:(master) ✗ make push run
echo "Installing" sitweb.eu.rawdraw
Installing sitweb.eu.rawdraw
/Users/wschrep/Library/Android/sdk/platform-tools/adb install makecapp.apk
Performing Push Install
makecapp.apk: 1 file pushed, 0 skipped. 15.0 MB/s (20206 bytes in 0.001s)
pkg: /data/local/tmp/makecapp.apk
Success
/Users/wschrep/Library/Android/sdk/platform-tools/adb shell am start -n sitweb.eu.rawdraw/android.app.NativeActivity
Starting: Intent { cmp=sitweb.eu.rawdraw/android.app.NativeActivity }
➜  rawdrawandroid git:(master) ✗ make push run
echo "Installing" sitweb.eu.rawdraw
Installing sitweb.eu.rawdraw
/Users/wschrep/Library/Android/sdk/platform-tools/adb install makecapp.apk
Performing Push Install
makecapp.apk: 1 file pushed, 0 skipped. 35.1 MB/s (20206 bytes in 0.001s)
pkg: /data/local/tmp/makecapp.apk
Failure [INSTALL_FAILED_ALREADY_EXISTS]
/Users/wschrep/Library/Android/sdk/platform-tools/adb shell am start -n sitweb.eu.rawdraw/android.app.NativeActivity
Starting: Intent { cmp=sitweb.eu.rawdraw/android.app.NativeActivity }
➜  rawdrawandroid git:(master) ✗ makecapp.apk
➜  rawdrawandroid git:(master) ✗ diff AndroidManifest.xml AndroidManifest.xml.back
4c4
<

➜  rawdrawandroid git:(master) ✗ make push run
echo "Installing" sitweb.eu.rawdraw
Installing sitweb.eu.rawdraw
/Users/wschrep/Library/Android/sdk/platform-tools/adb install makecapp.apk
Performing Push Install
makecapp.apk: 1 file pushed, 0 skipped. 9.0 MB/s (20206 bytes in 0.002s)
pkg: /data/local/tmp/makecapp.apk
Failure [INSTALL_FAILED_ALREADY_EXISTS]
/Users/wschrep/Library/Android/sdk/platform-tools/adb shell am start -n sitweb.eu.rawdraw/android.app.NativeActivity
Starting: Intent { cmp=sitweb.eu.rawdraw/android.app.NativeActivity }
➜  rawdrawandroid git:(master) ✗ less Make
Make: No such file or directory
➜  rawdrawandroid git:(master) ✗ less Makefile
    mkdir -p makecapp/lib/x86
    $(CC_x86) $(CFLAGS) $(CFLAGS_x86) -o $@ $^ $(LDFLAGS)

makecapp/lib/x86_64/lib$(APPNAME).so : $(ANDROIDSRCS)
mkdir -p makecapp/lib/x86_64
$(CC_x86) $(CFLAGS) $(CFLAGS_x86_64) -o $@ $^ $(LDFLAGS)
#We're really cutting corners.  You should probably use resource files.. Replace android:label="@string/app_name" and add a resource file.
#Then do this -S Sources/res on the aapt line.
#For icon support, add -S makecapp/res to the aapt line.  also,  android:icon="@mipmap/icon" to your application line in the manifest.
#If you want to strip out about 800 bytes of data you can remove the icon and strings.
#Notes for the past:  These lines used to work, but don't seem to anymore.  Switched to newer jarsigner.
#(zipalign -c -v 8 makecapp.apk)||true #This seems to not work well.
#jarsigner -verify -verbose -certs makecapp.apk
makecapp.apk : $(TARGETS) $(EXTRA_ASSETS_TRIGGER)
mkdir -p makecapp/assets
echo "Test asset file" > makecapp/assets/asset.txt
rm -rf temp.apk
$(AAPT) package -f -F temp.apk -I $(ANDROIDSDK)/platforms/android-$(ANDROIDVERSION)/android.jar -M AndroidManifest.xml -S Sources/res -A makecapp/assets -v --target-sdk-version $(ANDROIDVERSION)
unzip -o temp.apk -d makecapp
rm -rf makecapp.apk
cd makecapp && zip -D9r ../makecapp.apk .
jarsigner -sigalg SHA1withRSA -digestalg SHA1 -verbose -keystore $(KEYSTOREFILE) -storepass $(STOREPASS) makecapp.apk $(ALIASNAME)
rm -rf aligned.apk
$(BUILD_TOOLS)/zipalign -v 4 makecapp.apk aligned.apk
ls -l aligned.apk
uninstall :
($(ADB) uninstall $(PACKAGENAME))||true
push : makecapp.apk
echo "Installing" $(PACKAGENAME)
$(ADB) install makecapp.apk
run : push
$(eval ACTIVITYNAME:=$(shell $(AAPT) dump badging makecapp.apk | grep "launchable-activity" | cut -f 2 -d"'"))
$(ADB) shell am start -n $(PACKAGENAME)/$(ACTIVITYNAME)
clean :
rm -rf temp.apk makecapp.apk makecapp aligned.apk
`
And patched AndroidManifest.xml:
`

`
Totally cool we can draw fast and fullscreen with app thats only 53k in size ;)

Thought this was awesome too.
I only have a one plus one running android 6 still. So api level 23. But got it working.
I needed to change the api level to 23 (and download the sdk) otherwise the app would build but crash upon startup. However after going to level 23 I also needed to remove this in the AndroidManifest.xml. After that a make push run indeed starts the app on good ol android marshmallow and works fine.
I don't have time to create a proper pull request. This is my makefile that works on latest macos 10.15.4 and android studio v3.6.3:
Makefile diff versus repo version:
diff Makefile Makefile.macos_level_23
4c4
< all : makecapk.apk
---
> all : makecapp.apk
10,11c10,11
< APPNAME?=cnfgtest
< PACKAGENAME?=org.yourorg.$(APPNAME)
---
> APPNAME?=rawdraw
> PACKAGENAME?=sitweb.eu.$(APPNAME)
16c16,20
< ANDROIDVERSION?=29
---
> #ANDROIDVERSION?=24
> #ANDROIDVERSION?=29
> # good for marshmallow android v6.0
> ANDROIDVERSION?=23
>
21c25
< ADB?=adb
---
> ADB?=/Users/wschrep/Library/Android/sdk/platform-tools/adb
42c46
< SDK_LOCATIONS += $(ANDROID_HOME) ~/Android/Sdk $(HOME)/Library/Android/sdk
---
> SDK_LOCATIONS += $(ANDROID_HOME) ~/Library/Android/sdk $(HOME)/Library/Android/sdk
47,48c51,54
< NDK?=$(firstword $(wildcard $(ANDROIDSDK)/ndk/*) )
< BUILD_TOOLS?=$(firstword $(wildcard $(ANDROIDSDK)/build-tools/*) )
---
> #NDK?=$(firstword $(wildcard $(ANDROIDSDK)/ndk/*) )
> NDK?=/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462
> #BUILD_TOOLS?=$(firstword $(wildcard $(ANDROIDSDK)/build-tools/*) )
> BUILD_TOOLS?=/Users/wschrep/Library/Android/sdk/build-tools/29.0.3
68c74,75
< TARGETS?=makecapk/lib/arm64-v8a/lib$(APPNAME).so #makecapk/lib/armeabi-v7a/lib$(APPNAME).so makecapk/lib/x86/lib$(APPNAME).so makecapk/lib/x86_64/lib$(APPNAME).so
---
> #TARGETS?=makecapp/lib/arm64-v8a/lib$(APPNAME).so #makecapp/lib/armeabi-v7a/lib$(APPNAME).so makecapp/lib/x86/lib$(APPNAME).so makecapp/lib/x86_64/lib$(APPNAME).so
> TARGETS?=makecapp/lib/armeabi-v7a/lib$(APPNAME).so #makecapp/lib/x86/lib$(APPNAME).so makecapp/lib/x86_64/lib$(APPNAME).so
71c78,79
< CFLAGS_ARM32:=-mfloat-abi=softfp
---
> #CFLAGS_ARM32:=-mfloat-abi=softfp
> CFLAGS_ARM32:=-m32
85,88c93,96
< 	mkdir -p makecapk/lib/arm64-v8a
< 	mkdir -p makecapk/lib/armeabi-v7a
< 	mkdir -p makecapk/lib/x86
< 	mkdir -p makecapk/lib/x86_64
---
> 	mkdir -p makecapp/lib/arm64-v8a
> 	mkdir -p makecapp/lib/armeabi-v7a
> 	mkdir -p makecapp/lib/x86
> 	mkdir -p makecapp/lib/x86_64
90,91c98,99
< makecapk/lib/arm64-v8a/lib$(APPNAME).so : $(ANDROIDSRCS)
< 	mkdir -p makecapk/lib/arm64-v8a
---
> makecapp/lib/arm64-v8a/lib$(APPNAME).so : $(ANDROIDSRCS)
> 	mkdir -p makecapp/lib/arm64-v8a
94,95c102,103
< makecapk/lib/armeabi-v7a/lib$(APPNAME).so : $(ANDROIDSRCS)
< 	mkdir -p makecapk/lib/armeabi-v7a
---
> makecapp/lib/armeabi-v7a/lib$(APPNAME).so : $(ANDROIDSRCS)
> 	mkdir -p makecapp/lib/armeabi-v7a
98,99c106,107
< makecapk/lib/x86/lib$(APPNAME).so : $(ANDROIDSRCS)
< 	mkdir -p makecapk/lib/x86
---
> makecapp/lib/x86/lib$(APPNAME).so : $(ANDROIDSRCS)
> 	mkdir -p makecapp/lib/x86
102,103c110,111
< makecapk/lib/x86_64/lib$(APPNAME).so : $(ANDROIDSRCS)
< 	mkdir -p makecapk/lib/x86_64
---
> makecapp/lib/x86_64/lib$(APPNAME).so : $(ANDROIDSRCS)
> 	mkdir -p makecapp/lib/x86_64
108c116
< #For icon support, add -S makecapk/res to the aapt line.  also,  android:icon="@mipmap/icon" to your application line in the manifest.
---
> #For icon support, add -S makecapp/res to the aapt line.  also,  android:icon="@mipmap/icon" to your application line in the manifest.
112,113c120,121
< #(zipalign -c -v 8 makecapk.apk)||true #This seems to not work well.
< #jarsigner -verify -verbose -certs makecapk.apk
---
> #(zipalign -c -v 8 makecapp.apk)||true #This seems to not work well.
> #jarsigner -verify -verbose -certs makecapp.apk
117,119c125,127
< makecapk.apk : $(TARGETS) $(EXTRA_ASSETS_TRIGGER)
< 	mkdir -p makecapk/assets
< 	echo "Test asset file" > makecapk/assets/asset.txt
---
> makecapp.apk : $(TARGETS) $(EXTRA_ASSETS_TRIGGER)
> 	mkdir -p makecapp/assets
> 	echo "Test asset file" > makecapp/assets/asset.txt
121,125c129,133
< 	$(AAPT) package -f -F temp.apk -I $(ANDROIDSDK)/platforms/android-$(ANDROIDVERSION)/android.jar -M AndroidManifest.xml -S Sources/res -A makecapk/assets -v --target-sdk-version $(ANDROIDVERSION)
< 	unzip -o temp.apk -d makecapk
< 	rm -rf makecapk.apk
< 	cd makecapk && zip -D9r ../makecapk.apk .
< 	jarsigner -sigalg SHA1withRSA -digestalg SHA1 -verbose -keystore $(KEYSTOREFILE) -storepass $(STOREPASS) makecapk.apk $(ALIASNAME)
---
> 	$(AAPT) package -f -F temp.apk -I $(ANDROIDSDK)/platforms/android-$(ANDROIDVERSION)/android.jar -M AndroidManifest.xml -S Sources/res -A makecapp/assets -v --target-sdk-version $(ANDROIDVERSION)
> 	unzip -o temp.apk -d makecapp
> 	rm -rf makecapp.apk
> 	cd makecapp && zip -D9r ../makecapp.apk .
> 	jarsigner -sigalg SHA1withRSA -digestalg SHA1 -verbose -keystore $(KEYSTOREFILE) -storepass $(STOREPASS) makecapp.apk $(ALIASNAME)
127c135
< 	$(BUILD_TOOLS)/zipalign -v 4 makecapk.apk aligned.apk
---
> 	$(BUILD_TOOLS)/zipalign -v 4 makecapp.apk aligned.apk
134c142
< push : makecapk.apk
---
> push : makecapp.apk
136c144
< 	$(ADB) install makecapk.apk
---
> 	$(ADB) install makecapp.apk
139c147
< 	$(eval ACTIVITYNAME:=$(shell $(AAPT) dump badging makecapk.apk | grep "launchable-activity" | cut -f 2 -d"'"))
---
> 	$(eval ACTIVITYNAME:=$(shell $(AAPT) dump badging makecapp.apk | grep "launchable-activity" | cut -f 2 -d"'"))
143c151
< 	rm -rf temp.apk makecapk.apk makecapk aligned.apk
---
> 	rm -rf temp.apk makecapp.apk makecapp aligned.apk

AndroidManifest diff:
diff AndroidManifest.xml AndroidManifest.xml.back
4c4
<   <application android:debuggable="true" android:hasCode="false" android:label="@string/app_name" android:theme="@android:style/Theme.NoTitleBar.Fullscreen" tools:replace="android:icon,android:theme,android:allowBackup,label" android:icon="@mipmap/icon" >
---
>   <application android:debuggable="true" android:hasCode="false" android:label="@string/app_name" android:theme="@android:style/Theme.NoTitleBar.Fullscreen" tools:replace="android:icon,android:theme,android:allowBackup,label" android:icon="@mipmap/icon" android:requestLegacyExternalStorage="true">

Oops.  Didn't mean to close.  I added the other NDK search path.  See if that fixes it, @GitHubIsToxic GitHubIsToxic

Also, @w-A-L-L-e you should update the Makefile and try again.  I've added a lot better support for older SDKs and different NDK/SDK configurations.

Also even api level 23 also works. If you just change the xml file and remove the android:requestLegacyExternalStorage="true" which only works on latest android v10 I think. Most likely this is needed for all levels lower than 29...

Nope just did this (is it on other branch maybe?):
➜  rawdrawandroid git:(master) ✗ rm Makefile
➜  rawdrawandroid git:(master) ✗ git checkout Makefile
Updated 1 path from the index
➜  rawdrawandroid git:(master) ✗ make
mkdir -p makecapk/lib/arm64-v8a
/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android29-clang -ffunction-sections -Os -fdata-sections -Wall -fvisibility=hidden -Os -DANDROID -DAPPNAME=\"cnfgtest\" -I./rawdraw -I/sysroot/usr/include -I/sysroot/usr/include/android -fPIC -I. -DANDROIDVERSION=29 -m64 -o makecapk/lib/arm64-v8a/libcnfgtest.so test.c android_native_app_glue.c -L/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/29 -Wl,--gc-sections -s -lm -lGLESv3 -lEGL -landroid -llog -shared -uANativeActivity_onCreate
make: /toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android29-clang: No such file or directory
make: *** [makecapk/lib/arm64-v8a/libcnfgtest.so] Error 1
➜  rawdrawandroid git:(master) ✗

Anyway don't have time now. I'll just drop it here in case it might be handy for someone.
This works for me:
➜  rawdrawandroid git:(master) ✗ cp Makefile.macos_level_23 Makefile
➜  rawdrawandroid git:(master) ✗ adb uninstall sitweb.eu.rawdraw
Success
➜  rawdrawandroid git:(master) ✗ make push run
echo "Installing" sitweb.eu.rawdraw
Installing sitweb.eu.rawdraw
/Users/wschrep/Library/Android/sdk/platform-tools/adb install makecapp.apk
Performing Push Install
makecapp.apk: 1 file pushed, 0 skipped. 14.3 MB/s (20206 bytes in 0.001s)

	pkg: /data/local/tmp/makecapp.apk
Success
/Users/wschrep/Library/Android/sdk/platform-tools/adb shell am start -n sitweb.eu.rawdraw/android.app.NativeActivity

Starting: Intent { cmp=sitweb.eu.rawdraw/android.app.NativeActivity }
➜  rawdrawandroid git:(master) ✗

The entire tweaked makefile:
all : makecapp.apk

.PHONY : push run

# WARNING WARNING WARNING!
APPNAME?=rawdraw
PACKAGENAME?=sitweb.eu.$(APPNAME)
RAWDRAWANDROID?=.
RAWDRAWANDROIDSRCS=$(RAWDRAWANDROID)/android_native_app_glue.c
SRC?=test.c
#We've tested it with android version 24 and 29.
#ANDROIDVERSION?=24
#ANDROIDVERSION?=29
# good for marshmallow android v6.0
ANDROIDVERSION?=23

#Default is to be strip down, but your app can override it.
CFLAGS?=-ffunction-sections -Os -fdata-sections -Wall -fvisibility=hidden
LDFLAGS?=-Wl,--gc-sections -s
ANDROID_FULLSCREEN?=y
ADB?=/Users/wschrep/Library/Android/sdk/platform-tools/adb
UNAME := $(shell uname)

ANDROIDSRCS:= $(SRC) $(RAWDRAWANDROIDSRCS)

#if you have a custom Android Home location you can add it to this list.
#This makefile will select the first present folder.

ifeq ($(UNAME), Linux)
OS_NAME = linux-x86_64
endif
ifeq ($(UNAME), Darwin)
OS_NAME = darwin-x86_64
endif

# Search list for where to try to find the SDK
SDK_LOCATIONS += $(ANDROID_HOME) ~/Library/Android/sdk $(HOME)/Library/Android/sdk

#Just a little Makefile witchcraft to find the first SDK_LOCATION that exists
#Then find an ndk folder and build tools folder in there.
ANDROIDSDK?=$(firstword $(foreach dir, $(SDK_LOCATIONS), $(basename $(dir) ) ) )
#NDK?=$(firstword $(wildcard $(ANDROIDSDK)/ndk/*) )
NDK?=/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462
#BUILD_TOOLS?=$(firstword $(wildcard $(ANDROIDSDK)/build-tools/*) )
BUILD_TOOLS?=/Users/wschrep/Library/Android/sdk/build-tools/29.0.3

testsdk :
        echo $(BUILD_TOOLS)

CFLAGS+=-Os -DANDROID -DAPPNAME=\"$(APPNAME)\"
ifeq (ANDROID_FULLSCREEN,y)
CFLAGS +=-DANDROID_FULLSCREEN
endif
CFLAGS+= -I$(RAWDRAWANDROID)/rawdraw -I$(NDK)/sysroot/usr/include -I$(NDK)/sysroot/usr/include/android -fPIC -I$(RAWDRAWANDROID)
LDFLAGS += -lm -L$(NDK)toolchains/llvm/prebuilt/$(OS_NAME)/sysroot/usr/lib/aarch64-linux-android/$(ANDROIDVERSION) -lGLESv3 -lEGL -landroid -llog
LDFLAGS += -shared -uANativeActivity_onCreate

CC_ARM64:=$(NDK)/toolchains/llvm/prebuilt/$(OS_NAME)/bin/aarch64-linux-android$(ANDROIDVERSION)-clang
CC_ARM32:=$(NDK)/toolchains/llvm/prebuilt/$(OS_NAME)/bin/armv7a-linux-androideabi$(ANDROIDVERSION)-clang
CC_x86:=$(NDK)/toolchains/llvm/prebuilt/$(OS_NAME)/bin/x86_64-linux-android$(ANDROIDVERSION)-clang
CC_x86_64=$(NDK)/toolchains/llvm/prebuilt/$(OS_NAME)/bin/x86_64-linux-android$(ANDROIDVERSION)-clang
AAPT:=$(BUILD_TOOLS)/aapt

# Which binaries to build? NOTE: If you want to add to the number of supported releases, add to this line.
  You should probably use resource files.. Replace android:label="@string/app_name" and add a resource file.
#Then do this -S Sources/res on the aapt line.
#For icon support, add -S makecapp/res to the aapt line.  also,  android:icon="@mipmap/icon" to your application line in the manifest.
#If you want to strip out about 800 bytes of data you can remove the icon and strings.

#Notes for the past:  These lines used to work, but don't seem to anymore.  Switched to newer jarsigner.
#(zipalign -c -v 8 makecapp.apk)||true #This seems to not work well.
#jarsigner -verify -verbose -certs makecapp.apk

makecapp.apk : $(TARGETS) $(EXTRA_ASSETS_TRIGGER)
        mkdir -p makecapp/assets
        echo "Test asset file" > makecapp/assets/asset.txt
        rm -rf temp.apk
        $(AAPT) package -f -F temp.apk -I $(ANDROIDSDK)/platforms/android-$(ANDROIDVERSION)/android.jar -M AndroidManifest.xml -S Sources/res -A makecapp/assets -v --target-sdk-version $(ANDROIDVERSION)
        unzip -o temp.apk -d makecapp
        rm -rf makecapp.apk
        cd makecapp && zip -D9r ../makecapp.apk .
        jarsigner -sigalg SHA1withRSA -digestalg SHA1 -verbose -keystore $(KEYSTOREFILE) -storepass $(STOREPASS) makecapp.apk $(ALIASNAME)
        rm -rf aligned.apk
        $(BUILD_TOOLS)/zipalign -v 4 makecapp.apk aligned.apk
        ls -l aligned.apk

uninstall :
        ($(ADB) uninstall $(PACKAGENAME))||true

push : makecapp.apk
        echo "Installing" $(PACKAGENAME)
        $(ADB) install makecapp.apk

run : push
        $(eval ACTIVITYNAME:=$(shell $(AAPT) dump badging makecapp.apk | grep "launchable-activity" | cut -f 2 -d"'"))
        $(ADB) shell am start -n $(PACKAGENAME)/$(ACTIVITYNAME)

clean :
        rm -rf temp.apk makecapp.apk makecapp aligned.apk

@w-A-L-L-e I urge you to still test master, as I believe I have it all working.  If it's still an issue, open an issue.

I might be forgetting something or doing something wrong. But I do git pull and I've got latest version and still get the error wit the master makefile (maybe I'm missing some export) anyway doesn't matter you can close this, my tweaked makefile works good enough for me. Thanks for sharing this.
➜  rawdrawandroid git:(master) ✗ git checkout Makefile
Updated 1 path from the index
➜  rawdrawandroid git:(master) ✗ git pull
Already up to date.
➜  rawdrawandroid git:(master) ✗ make push run
mkdir -p makecapk/lib/arm64-v8a
/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android29-clang -ffunction-sections -Os -fdata-sections -Wall -fvisibility=hidden -Os -DANDROID -DAPPNAME=\"cnfgtest\" -I./rawdraw -I/sysroot/usr/include -I/sysroot/usr/include/android -fPIC -I. -DANDROIDVERSION=29 -m64 -o makecapk/lib/arm64-v8a/libcnfgtest.so test.c android_native_app_glue.c -L/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/29 -Wl,--gc-sections -s -lm -lGLESv3 -lEGL -landroid -llog -shared -uANativeActivity_onCreate
make: /toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android29-clang: No such file or directory
make: *** [makecapk/lib/arm64-v8a/libcnfgtest.so] Error 1
➜  rawdrawandroid git:(master) ✗

I'm really hoping to make this work for everyone.  It looks like it might not be finding your SDK.
NDK?=/Users/wschrep/Library/Android/sdk/ndk/21.1.6352462

Cause it's definitely not finding your NDK.
Can you add
echo $(ANDROIDSDK)

to the first compile function, so we can see if it found the SDK?

It outputs this:
➜  rawdrawandroid git:(master) ✗ make
echo ~/Android/Sdk
/Users/wschrep/Android/Sdk
mkdir -p makecapk/lib/arm64-v8a
/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android29-clang -ffunction-sections -Os -fdata-sections -Wall -fvisibility=hidden -Os -DANDROID -DAPPNAME=\"cnfgtest\" -I./rawdraw -I/sysroot/usr/include -I/sysroot/usr/include/android -fPIC -I. -DANDROIDVERSION=29 -m64 -o makecapk/lib/arm64-v8a/libcnfgtest.so test.c android_native_app_glue.c -L/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/29 -Wl,--gc-sections -s -lm -lGLESv3 -lEGL -landroid -llog -shared -uANativeActivity_onCreate
make: /toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android29-clang: No such file or directory
make: *** [makecapk/lib/arm64-v8a/libcnfgtest.so] Error 1
➜  rawdrawandroid git:(master) ✗

Which does indeed not exist:
ls  ~/Android/Sdk
ls: /Users/wschrep/Android/Sdk: No such file or directory

however this path does:
ls ~/Library/Android/sdk/
build-tools    cmdline-tools  emulator       extras         fonts          licenses       ndk            patcher        platform-tools platforms      sources        tools

Ok so yes, you got me curious. So if I do this export :
export ANDROIDSDK=~/Library/Android/sdk
then the makefile works (apart from changes to make it work for level23).
Maybe you can add a uname == "Darwin" test and auto set ANDROIDSDK to ~/Library/Android/sdk ?
Then it will work out of the box on macos without needing to do exports... ;)

Is $(HOME) not set in the makefile?

# Search list for where to try to find the SDK
SDK_LOCATIONS += $(ANDROID_HOME) ~/Android/Sdk $(HOME)/Library/Android/sdk

#Just a little Makefile witchcraft to find the first SDK_LOCATION that exists
#Then find an ndk folder and build tools folder in there.
ANDROIDSDK?=$(firstword $(foreach dir, $(SDK_LOCATIONS), $(basename $(dir) ) ) )

Should automatically handle all searching.  No need to check for darwin or anything.
I am worried about this line:
SDK_LOCATIONS += $(ANDROID_HOME) ~/Android/Sdk $(HOME)/Library/Android/sdk
Specifically the $(HOME).  Can you see if replacing that with a ~ on your system does the trick?

Noope didn't help, same behaviour (also $HOME is set on my system by default to same dir as ~) only when I have ANDROIDSDK set in env vars (like in prev message) it works, when I do unset $ANDROIDSDK it fails : /toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android29-clang: No such file or directory.
Other issue found. When I run this using api level 23  the app runs fine with about 38fps and when rotating in horizontal plane we see the gyro works fine. But when I rotate phone and trigger  landscape mode it sometimes gives a nice alternate view but 90% of the time it crashes. This is what I see in the logs using logcat:
05-19 10:35:41.076 22319 22333 D rawdraw : FPS: 38
05-19 10:35:41.124 11666 11903 I InputReader: Reconfiguring input devices.  changes=0x00000004
05-19 10:35:41.124 11666 11903 I InputReader: Device reconfigured: id=5, name='synaptics-rmi-ts', size 1080x1920, orientation 1, mode 1, display id 0
05-19 10:35:41.144 22319 22333 D rawdraw : Pause: 0xb4cfc360
05-19 10:35:41.181 22319 22333 D rawdraw : activityState=13
05-19 10:35:41.182 22319 22333 D rawdraw : event not handled: 13SaveInstanceState: 0xb4cfc360
05-19 10:35:41.210 22319 22333 D rawdraw : event not handled: 12APP_CMD_SAVE_STATE
05-19 10:35:41.210 22319 22333 D rawdraw : Stop: 0xb4cfc360

05-19 10:35:41.243 22319 22333 D rawdraw : activityState=14
05-19 10:35:41.243 22319 22333 D rawdraw : event not handled: 14NativeWindowDestroyed: 0xb4cfc360 -- 0xaeb05f08
05-19 10:35:41.263 13115 13714 D PhoneInterfaceManager: [PhoneIntfMgr] getDataEnabled: subId=5 phoneId=0
05-19 10:35:41.263 13115 13714 D PhoneInterfaceManager: [PhoneIntfMgr] getDataEnabled: subId=5 retVal=false
05-19 10:35:41.263 17292 22115 D TelephonyManager: getDataEnabled: retVal=false
05-19 10:35:41.277 22319 22333 D rawdraw : APP_CMD_TERM_WINDOW
05-19 10:35:41.277 22319 22333 D rawdraw : event not handled: 2APP_CMD_TERM_WINDOW
05-19 10:35:41.277 22319 22333 D rawdraw : InputQueueDestroyed: 0xb4cfc360 -- 0xb4cfe020
05-19 10:35:41.277 22319 22333 D rawdraw : APP_CMD_INPUT_CHANGED
05-19 10:35:41.277 22319 22333 D rawdraw : Destroy: 0xb4cfc360
05-19 10:35:41.277 22319 22333 D rawdraw : event not handled: 0APP_CMD_DESTROY
05-19 10:35:41.277 22319 22333 D rawdraw : Destroying
05-19 10:35:41.372 11666 14763 I WindowState: WIN DEATH: Window{c1fcb54 u0 sitweb.eu.rawdraw/android.app.NativeActivity}
05-19 10:35:41.373 11666 13776 I qti_sensors_hal: L3GD20 Gyroscope: activate=0
05-19 10:35:41.387 11456 11456 I Zygote  : Process 22319 exited cleanly (10)
05-19 10:35:41.396 11666 11679 I ActivityManager: Process sitweb.eu.rawdraw (pid 22319) has died
05-19 10:35:41.396 11666 11679 W ActivityManager: Force removing ActivityRecord{32b06db u0 sitweb.eu.rawdraw/android.app.NativeActivity t52}: app died, no saved state

A few notes.
(1) Please try to debug the Makefile (use echo to print out variables) The Makefile should select from a list of SDKs and find the first one that exists on your system.  If this is not happening something is wrong with the makefile's selection process.
(2) I have seen an issue one off with the display rotation.  It appears to be related to creating and destroying the window, which happens frequently, and happens by necessity when rotating the screen. Perhaps code somewhere is executing which is trying to work with the screen while the texture is invalid?
(3) I don't know, but I would imagine it wouldn't be too hard.  https://developer.android.com/ndk/reference/group/camera There's a whoooole site on it!

Idd a whoooole site and not a single clear simple example to be found there or by googling ;).
Something like this simple java project https://github.com/justinjoy/native-camera2/blob/master/app/src/main/java/org/freedesktop/nativecamera2/NativeCamera2.java would be a nice addition (but then a C version) and same for audio. Just saying you've got a nice demo and sources for simple use of screen with double buffer, gyro reading, keyboard input and multi touch. All we need next is audio, video, pictures and some way to send/receive them and you can pretty much use this for any hack/project ;)

@cnlohr: with the latest Makefile, it almost works, but I had to change this:
SDK_LOCATIONS += $(ANDROID_HOME) ~/Android/Sdk $(HOME)/Library/Android/sdk
To:
SDK_LOCATIONS += ~/Android/Sdk $(ANDROID_HOME) $(HOME)/Library/Android/sdk
I.e. prefer sdk folder in home directory over ANDROID_HOME.
It seems it fails on my machine because ANDROID_HOME points to "/usr/lib/android-sdk" for some reason, and it doesn't contain any binaries (or symlinks) there.
My Android Studio / NDK install is pretty recent/fresh (few months old), and I haven't used it, so I don't understand why it's in such a deplorable state already. Google keeps moving folders around, I think I have at least 3 folders containing partial SDK/NDK duplicates on my hard disk with the SDK (about 70 GB probably). It is likely most users will have borked installs as well. Anyway, with the above change, the Makefile works for me out-of-the box, with updated Android Studio + NDK.

Hmm... That's really rough... I feel like the "correct" behavior is to honor ANDROID_HOME first and foremost. That's frustrating it's broken in your case.  Perhaps the best thing for you would be on your system to override that variable in your bashrc, or profile so that it points where you want instead of making any changes to rawdrawandroid?

I feel like the "correct" behavior is to honor ANDROID_HOME first and foremost.

I would concur, but it looks like it's apparently not considered a definitive/required path for Android SDK?
Perhaps there's a small chance Android Studio may update the ANDROID_HOME env variable if I perform some action in it (e.g. a build) perhaps, and wasn't set properly by an update. But since one of the aims of rawdrawandroid is to avoid the use of Android Studio...
I should note that there's no "ndk" or "ndk-bundle" folder inside my "/usr/lib/android-sdk", so that'd be one way to check to prefer the actual home folder over ANDROID_HOME.
tox@ryzen:/usr/lib/android-sdk$ ls
build-tools  licenses  platforms  platform-tools  tools

tox@ryzen:~/Android/Sdk$ ls
build-tools  cmake  emulator  licenses  ndk  ndk-bundle  patcher  platforms  platform-tools  skins  system-images  tools

(And note that ~/Android/Sdk/ndk/ lacks some essential build tools, but ~/Android/Sdk/ndk-bundle/ works).
As I said, I have a fairly fresh install, and haven't used it in the mean while, and only updated it once (a few days ago when I first tried rawdrawandroid). It's likely other users will run into the same issue.
I suppose we could always try and check if the actual tool binaries are present before choosing a directory, to be absolutely sure.

On a sidenode, how feasible do you think it would be to create a package containing only the bare essentials for rawdrawandroid to build? I.e. distill Android Studio/SDK + NDK down to a small stand-alone toolkit (ignoring whether licensing would allow that)?

Ok, it looks like ANDROID_HOME is deprecated by Google:
https://developer.android.com/studio/command-line/variables.html#envar
Which could explain why it wasn't updated on my machine by the Android Studio update. But the suggested alternative (ANDROID_SDK_ROOT) is also not set on my machine, so I think we're in unknown territory anyhow.
FYI, the relevant env vars on my machine:
ANDROID_HOME=/usr/lib/android-sdk
ANDROID_NDK_HOME=/usr/lib/android-ndk
PATH=/usr/lib/android-sdk/platform-tools:~/Android/android-sdk-linux/tools:/usr/lib/android-ndk:/home/tox/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

So, your suggestion would be to place ~/Android/Sdk as the first path to check?

Err, ok, I had a comment, but now thinking back I'm still confused.  It feels like the current search path is still "best" ... I'm really not sure what to do with this new data.

From following the recomendations from the link I posted above, I would suggest:

check if ANDROID_HOME is defined, use that it it contains /ndk/ or /ndk-bundle/
check if ANDROID_SDK_ROOT is defined, use that if it contains /ndk/ or /ndk-bundle/
check if ~/Android/Sdk/ndk-bundle/ exists, use that as first fallback
check if ~/Android/Sdk/ndk/ exists, use that as second fallback

That would work on my machine, and wouldn't break anything for other users.
I would also echo a message if the NDK dir is still not found, so it isn't burried in error messages from subsequent invokations in the makefile.

On mac this is the path (notice no capital S in sdk):
~/Library/Android/sdk
ndk is then at /Library/Android/sdk/ndk/21.1.6352462...

@w-A-L-L-e  - Looking at the Makefile, ~/Library/Android/sdk is one of the search paths.
SDK_LOCATIONS += $(ANDROID_HOME) ~/Android/Sdk $(HOME)/Library/Android/sdk

The NDK is saerched as such:
NDK?=$(firstword $(wildcard $(ANDROIDSDK)/ndk/*) $(wildcard $(ANDROIDSDK)/ndk-bundle/*) )

I am pretty sure based on everything you're saying this is correct.  Find SDK first.  Then once SDK is found, find NDK.
@GitHubIsToxic I don't actually know how to check for a lack of NDK or other folder... I'm actually really new to this syntax for firstword.  Would you (stat BLAH||fail BLAH) kind of thing?

@cnlohr: I'm myself pretty new to anything beyond the most basic Makefile routine, although I've learned a lot tinkering with this one. I did find this: https://stackoverflow.com/questions/20763629/test-whether-a-directory-exists-inside-a-makefile
Also, while on the subject, may I suggest to use lastword instead of firstword on this line:
BUILD_TOOLS?=$(firstword $(wildcard $(ANDROIDSDK)/build-tools/*) )
On my machine there are two sub dirs in /home/tox/Android/Sdk/build-tools:  /28.0.3/ and /29.0.2/. Using lastword it will select the latest (29.0.2). Though I'm not actually sure the directory order handled by wildcard is actually alphabetically sorted, or dependent on underlying file system behavior (kind of random like readdir()).

Now, THAT is a change I think that makes sense for!  Making the change now there.  Does that get you moving again?

I believe in almost all situations it is alphabetical.  I think this is the only change I want to make... Any issues closing this thread?

Will do a pull request with all my suggestions so far. I'll try to come up with an elegant solution that works on my machine, separate from that.

I did find a good and clean solution, see https://github.com/cnlohr/rawdrawandroid/pull/16 .
You can probably close this issue.

Your changes look great!  I think you are a fluke, but I don't mind supporting it.  I have reviewed and will approve your PR.


Kafka-Node: Producer never get notified when a partition moved to new broker on auto rebalance.

Scenario,
I have two brokers (broker 1 & broker 2) with  auto.leader.rebalance.enable=true.
Create TEST topic with replica: 2, partitions 8.
Use a producer to push message on TEST topic.
when i stopped a broker1, broker2 became leader for all the partitions. few mins later broker1 comes online. after leader.imbalance.check.interval.seconds, all partitions are distribute among two brokers but my producer never gets notified to refresh the metadata.
there after some msgs are getting failing because of payload was sending to broker w/ invalid or non-exist partition.
Is it right behavior of Producer? or am i doing wrong?

Which producer are you using: low level or high level?
The low level producer, I think, doesn't do this detection job. Not sure, thought. But the high level producer may handle this scenario.
The desirable behavior of the high level producer, in my opinion, should be as follows:

The producer sends messages to a non-existing partition which might have moved to a different broker.
The producer receives an error from the server.
If the error says that the partition is not found, then the producer should try to refresh its metadata.
Then try sending again to the same partition found in another broker.
If the partition is not found after refreshing the metadata, then should escalate the error to the client.

@haio what do you say?

Thanks for the response @kadishmal ...
i tried with both Producers (Low & High Level). unfortunately i couldn't see any error in the send callback. however kafka broker2 started getting below error on auto rebalance.
2015-03-16 10:22:48,370] WARN [KafkaApi-1] Produce request with correlation id 14878 from client slam.local on partition [TESTING,4] failed due to Leader not local for partition [TESTING,4] on broker 1 (kafka.server.KafkaApis)

[2015-03-16 10:22:52,394] WARN [KafkaApi-1] Produce request with correlation id 14882 from client slam.local on partition [TESTING,6] failed due to Leader not local for partition [TESTING,6] on broker 1 (kafka.server.KafkaApis)

[2015-03-16 10:22:55,416] WARN [KafkaApi-1] Produce request with correlation id 14885 from client slam.local on partition [TESTING,6] failed due to Leader not local for partition [TESTING,6] on broker 1 (kafka.server.KafkaApis)

[2015-03-16 10:22:57,436] WARN [KafkaApi-1] Produce request with correlation id 14887 from client slam.local on partition [TESTING,4] failed due to Leader not local for partition [TESTING,4] on broker 1 (kafka.server.KafkaApis)

I could see event notification when i initiate manual preferred replica election for rebalance from CLI. this scenario Kafka-Node/lib/Producer works fine. but auto rebalance needs to be fixed i guess.
please share your thoughts on this…

@lsampathreddy the Producer(Low and High Level) will not refresh metadata when broker change so far, I think this feature can be added in next version.
@kadishmal the current implementation for refreshing metadata is watching the broker metadata in zookeeper, if it changes then emit a brokersChanged event, then refresh the metadata.
In your advices above, I don't know why should  do this: The producer sends messages to a non-existing partition which might have moved to a different broker.

@haio a response to your last question:

The producer sends messages to a non-existing partition which might have moved to a different broker.

Imagine a producer continuously sending messages.
At some point rebalancing occurs.
However, producer keeps sending before the client notices the brokerChanged event. In such case, the producer will send messages which will fail.

I've checked your PR, the fix you've committed looks very simple. I doubt it fixes this issue. @lsampathreddy would you please fetch the latest version of kafka-node and see if it fixes your problem?

As far as I remember, re-balancing can be detected via watches set by Zookeeper#listBrokers(), so brokersChanged is never emitter when re-balancing occurs.
Instead, after brokers re-balance topics between each other, ZooKeeper nodes located in /brokers/topics path are changed. I tried to solve this issue in my fork's branch (but still have got issues): watch-topics-zookeeper

@estliberitas , @kadishmal & @haio  Thanks for your responses.
@kadishmal
#176 works perfectly when broker added or deleted. zookeeper emit brokerChanged to its watchers. but my problem still not resolved completely...
Somehow i debug the Kafka scala code, KafkaController never notify the changes to zookeeper when auto-rebalance is enabled, i feel condition not required at KafkaController.scala . if Kafka itself notifies the preferred election changes to Zookeeper via Path(/admin/preferred_replica_election or /admin/reassign_partitions) then Kafka-Node will work perfectly in auto rebalance.
i reported an issue at Kafka Jira [ISSUE] (https://issues.apache.org/jira/browse/KAFKA-2027). waiting for their response.
@estliberitas if i do manual re-balance with below command Kafka-Node was able to identify the changes.
bin/kafka-preferred-replica-election.sh

@lsampathreddy great! Maybe my 0.8.0 setup was different. We had an issue making simple test: stopping 1 broker, then starting it and waiting for re-balancing.

The client watch /brokers/ids in zookeeper to check whether the brokers change, this works when broker die, relive or add new broker, but it can't handle the case when re-balancing occurs but the data in /brokers/ids didn't change. @estliberitas mentioned another way: watch /brokers/topics, I'm not sure it can work, since the client support multiple topics and a topic may has multiple partitions, this way can be more complicate.
@lsampathreddy In what case you still have this issue?

Hi @Haio,
My issue here is client should watch leadership imbalance on brokers.
This imbalance could happen when a broker dies and relive. Kafka 0.8.1 feature auto rebalance periodically check for any partition imbalance on brokers and tries to elect and balance the partition among the brokers.
now this change was not determined/detected by Kafka-Node client watchers. This is tricky. i am looking for appropriate solution.
@haio, @estliberitas & @kadishmal please share your thoughts on this...

I believe it should detect the rebalanced partitions. At least the high producer and consumer should handle this.

hi @lsampathreddy, how can we reproduce the imbalance on brokers?

@haio
here is the steps in detail for reproducing partition imbalance…

start zookeeper (should run on 2181 port)
start 2 kafka servers with individual server.properties. make sure these properties file have below properties

auto.leader.rebalance.enable=true
leader.imbalance.check.interval.seconds=100

create topic ‘TESTING’ using shell command. replica:2 & partition: 8
send message to TESTING topic from test file.

var kafka = require('kafka-node'),
    Producer = kafka.Producer,
    KeyedMessage = kafka.KeyedMessage,
    client = new kafka.Client('<IP_ADDRESS>:2181', 'clientId'),
    producer = new Producer(client);

var buildPayload = function (msg) {
		var payload = [];
		payload.push({topic: 'TESTING', messages: msg+" ", partition: Math.floor(Math.random() * 8)});
		return payload;
};

producer.on('ready', function () {
  console.log('producer is ready...');
});

var counter = 0;
setInterval(function(){
  counter ++;
  var payloads = buildPayload(counter);
  console.log(payloads);
  producer.send(payloads, function (err, data) {
    //if(err) console.log('error'+ err);
  });
}, 1000);

producer.on('error', function (err) {
  console.log('err at produer'+err);

})

start kafka-console-consumer.sh to see the msgs sent from producer
check the partition information for TEST topic by running script

slam:kafka_2.9.2-<IP_ADDRESS> s.lambu$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic TESTING

Topic:TESTING	PartitionCount:8	ReplicationFactor:2	Configs:

Topic: TESTING	Partition: 0	Leader: 0	Replicas: 0,1	Isr: 0,1
Topic: TESTING	Partition: 1	Leader: 1	Replicas: 1,0	Isr: 0,1
Topic: TESTING	Partition: 2	Leader: 0	Replicas: 0,1	Isr: 0,1
Topic: TESTING	Partition: 3	Leader: 1	Replicas: 1,0	Isr: 0,1
Topic: TESTING	Partition: 4	Leader: 0	Replicas: 0,1	Isr: 0,1
Topic: TESTING	Partition: 5	Leader: 1	Replicas: 1,0	Isr: 0,1
Topic: TESTING	Partition: 6	Leader: 0	Replicas: 0,1	Isr: 0,1
Topic: TESTING	Partition: 7	Leader: 1	Replicas: 1,0	Isr: 0,1

stop broker 1 by Ctrl + C. now Kafka will elect new leader for all partition. now broker 0 becomes leader for all partitions. we can verify it by below command.
also verify producer, it should refresh the metadata on 'brokersChanged'.

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic TESTING
Topic:TESTING	PartitionCount:8	ReplicationFactor:2	Configs:

Topic: TESTING	Partition: 0	Leader: 0	Replicas: 0,1	Isr: 0
Topic: TESTING	Partition: 1	Leader: 0	Replicas: 1,0	Isr: 0
Topic: TESTING	Partition: 2	Leader: 0	Replicas: 0,1	Isr: 0
Topic: TESTING	Partition: 3	Leader: 0	Replicas: 1,0	Isr: 0
Topic: TESTING	Partition: 4	Leader: 0	Replicas: 0,1	Isr: 0
Topic: TESTING	Partition: 5	Leader: 0	Replicas: 1,0	Isr: 0
Topic: TESTING	Partition: 6	Leader: 0	Replicas: 0,1	Isr: 0
Topic: TESTING	Partition: 7	Leader: 0	Replicas: 1,0	Isr: 0

start the broker 1 again. now Kafka will expand the ISR to 2 nodes but leader is broker 0. we can verify it with same above command.
here we can see leader imbalance, two brokers available but still single broker (0) is leader to all the partitions.

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic TESTING
Topic:TESTING	PartitionCount:8	ReplicationFactor:2	Configs:
	Topic: TESTING	Partition: 0	Leader: 0	Replicas: 0,1	Isr: 0,1
	Topic: TESTING	Partition: 1	Leader: 0	Replicas: 1,0	Isr: 0,1
	Topic: TESTING	Partition: 2	Leader: 0	Replicas: 0,1	Isr: 0,1
	Topic: TESTING	Partition: 3	Leader: 0	Replicas: 1,0	Isr: 0,1
	Topic: TESTING	Partition: 4	Leader: 0	Replicas: 0,1	Isr: 0,1
	Topic: TESTING	Partition: 5	Leader: 0	Replicas: 1,0	Isr: 0,1
	Topic: TESTING	Partition: 6	Leader: 0	Replicas: 0,1	Isr: 0,1
	Topic: TESTING	Partition: 7	Leader: 0	Replicas: 1,0	Isr: 0,1

after leader.imbalance.check.interval.seconds each broker runs it own process to check whether any imbalance exist. attaching the controller.log one of the kafka broker.
here we can see how partition 5 moved from 0 to 1.

[2015-03-17 15:11:36,419] checking need to trigger partition rebalance (kafka.controller.KafkaController)
[2015-03-17 15:11:36,419] DEBUG [Controller 0]: preferred replicas by broker Map(1 -> Map([TESTING,5] -> List(1, 0), [TESTING,3] -> List(1, 0), [TESTING,1] -> List(1, 0), [TESTING,7] -> List(1, 0)), 0 -> Map([TESTING,4] -> List(0, 1), [TESTING,6] -> List(0, 1), [TESTING,2] -> List(0, 1), [TESTING,0] -> List(0, 1))) (kafka.controller.KafkaController)
[2015-03-17 15:11:36,419] DEBUG [Controller 0]: topics not in preferred replica Map([TESTING,5] -> List(1, 0), [TESTING,3] -> List(1, 0), [TESTING,1] -> List(1, 0), [TESTING,7] -> List(1, 0)) (kafka.controller.KafkaController)
>>> [2015-03-17 15:11:36,420] TRACE [Controller 0]: leader imbalance ratio for broker 1 is 1.000000 (kafka.controller.KafkaController)
[2015-03-17 15:11:36,420] DEBUG [Controller 0]: controllerContext.liveBrokerIds Set(1, 0) (kafka.controller.KafkaController)
[2015-03-17 15:11:36,420] DEBUG [Controller 0]: leaderBroker 1 (kafka.controller.KafkaController)
[2015-03-17 15:11:36,420] INFO [Controller 0]: Starting preferred replica leader election for partitions [TESTING,5] (kafka.controller.KafkaController)
[2015-03-17 15:11:36,421] INFO [Partition state machine on Controller 0]: Invoking state change to OnlinePartition for partitions [TESTING,5] (kafka.controller.PartitionStateMachine)
>>> [2015-03-17 15:11:36,426] INFO [PreferredReplicaPartitionLeaderSelector]: Current leader 0 for partition [TESTING,5] is not the preferred replica. Trigerring preferred replica leader election (kafka.controller.PreferredReplicaPartitionLeaderSelector)
>>> [2015-03-17 15:11:36,428] DEBUG [Partition state machine on Controller 0]: After leader election, leader cache is updated to Map([TESTING,5] -> (Leader:1,ISR:0,1,LeaderEpoch:3,ControllerEpoch:2), [TESTING,4] -> (Leader:0,ISR:1,0,LeaderEpoch:2,ControllerEpoch:1), [TESTING,6] -> (Leader:0,ISR:1,0,LeaderEpoch:2,ControllerEpoch:1), [TESTING,3] -> (Leader:0,ISR:0,LeaderEpoch:2,ControllerEpoch:2), [TESTING,2] -> (Leader:0,ISR:1,0,LeaderEpoch:2,ControllerEpoch:1), [TESTING,0] -> (Leader:0,ISR:1,0,LeaderEpoch:2,ControllerEpoch:1), [TESTING,1] -> (Leader:0,ISR:0,LeaderEpoch:2,ControllerEpoch:2), [TESTING,7] -> (Leader:0,ISR:0,LeaderEpoch:2,ControllerEpoch:2)) (kafka.controller.PartitionStateMachine)
>>> [2015-03-17 15:11:36,430] INFO [Controller 0]: Partition [TESTING,5] completed preferred replica leader election. New leader is 1 (kafka.controller.KafkaController)

verify our test producer,  it should still sending msg to TESTING topic with out any error. if any msg sending to partition 5 that msg never read by consumer. because our metadataHash having old values, partition5 will maps to broker 0 not broker 1. since zookeeper never send 'borkerChanged' or 'anyOther' event to refresh the metadata in this scenario.

hence msg lost… :(
please update me if I’m doing wrong…

Thanks, will check this.

@haio @lsampathreddy what I offered seems a bit tricky to implement because /brokers/topics has 2-level tree of nodes inside: /brokers/topics/TOPIC_ID/PARTITION_NUM. In case of many topics and many partitions kafka-node can produce a watch-hell for ZK. But maybe there is no need in watching partition states, topic nodes. I did not make enough experiments.

I have made a fix for this.  When the client get NotLeaderForPartitionerror, that means the leader for the partition has changed, so it emit a brokersChanged event(just for consistency), consumer and producer listen this event and refresh topic metadata.
@lsampathreddy can you test it with this commit 704ae54981d5f4778d781b72f17fc3f6ab72f654

@haio  Thank you for your response.
I tested with your changes.. its working perfectly now. :+1:
as soon as preferred-election complete in Kafka, next msg gets fail because of partition moved. new code emits brokersChanged event when it receive 'NotLeaderForPartition' error, so client can refresh the metadata. Its perfect...
please let me know which version you are going include these changes.

@lsampathreddy check<EMAIL_ADDRESS>

Thanks you @haio


Docker :: Python 3.7 :: which base image to use?

I would like to put my app into Docker container. And now I faced the problem...
Python 3.7 - which Docker base image for azure speech services?
python:3.7 -> does not work, it is missin libssl.1.0.0
python:3.7-alpine -> seems that python package can not be installed there...

Hi @madkote, we're working on lifting this in future releases. For the python:3.7 container the underlying distro is Debian 9, which we do not yet support.
So right now, please use Ubuntu 16.04 or 18.04 as underlying distro and make sure to install the prerequisites.

thx @mahilleb-msft !
ok, good to know that this issue is known.<
indeed, I had to go with ubuntu 18, but with python 3.6.

Hi @madkote, with version 1.5.0 of the Speech SDK, Debian 9 based images like python:3.7 should also be supported. libssl1.0.2 should be already pre-installed, but libasound2 is still a dependency on top.
I'm closing this issue here, but please re-open if you run into problems using python:3.7 with Speech SDK 1.5.0.


Bug: Emoji in message be shown as unreadable codes when sending or receiving messages though miniapp

What happened?
Emoji in message be shown as unreadable codes when sending or receiving messages though miniapp.
The reason maybe is that there is no TextEncoder/TextDecoder in weixin miniapp .
What did you expect to happen?
please repair the Encoder/Decoder implemented in sdk for emoji
How can we reproduce it (as minimally and precisely as possible)?
send text message with emoji though miniapp sdk
Anything else we need to know?
No response
version

open-im-sdk 3.4.1

Cloud provider

OS version

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

Install tools

fixed in 3.5.1-alpha.7, pleasle update open-im-sdk and oimws.


simple exporting models

Brought over the core logic of exporting models into a single file under extra
Kept the logic for clang and webgpu, only returning the logic needed for the model to run.
Added a third option to export out to a json file with everything needed to run the model enabling the ability to bring these models into any language of choice provided it can access the backened used to export (OPENCL, CUDA, etc).
As a example of the import ability I created a golang lib tinyrnnr that can import this json file and run efficientnet without any tensor logic.

Why is this draft? Ready to merge?

Wanted to add a swift/Metal example as well but can't get the image properly fitted. Will just skip that and rebase with master and mark ready for review

Sorry, can you rebase. I shouldn't have merged the editor config PR.

They just timed out here, I'll rerun it.


instructions unclear

The final test is seeking "friends" as a property, not "enemies", yet the instructions do not ask for those specific object properties - as a matter of fact I assumed to use the same properties as cat - all are correct save friends vs enemies
Challenge http://beta.freecodecamp.com/challenges/i-object has an issue. Please describe how to reproduce it, and include links to screenshots if possible.

Closing as stale


Proper Docker Images

Would it be possible to publish official docker images? I've been wanting to set this up but can't get it to run nicely in my Coolify Setup (it runs just fine when i do docker compose up -d locally)

Yeah it should be possible to whip up a quick CI job for that. Happy to accept a patch if this is needed at short notice!

This is now available: https://github.com/icyphox/legit/pkgs/container/legit

Thank you so much!


Do I need to install the nexmon patch first?

Should we install the nexmon patch first or just use the installation script directly?

No, all you need to do it run the script on a freshly setup pi. Make sure you use the right raspi-os version.

Thank you.


Slack > Mattermost Attachments

Is your feature request related to a problem? Please describe.
As far as I know not, looked into the issues that already exist.
Describe the solution you'd like
Solution doesn't exist, it's a feature request.
Describe alternatives you've considered
Alternative doesn't exist, it's a feature request.
Additional context
Add support to send the whole attachment through the message. Now a day if the message doens't have text it is looking to the text in the attachment part. I have tested it with a webhook and user and doens't work atm.

If I understand the problem correct, you want the slack message attachments (https://api.slack.com/docs/message-attachments) to be sent to mattermost? (we're not talking about file attachments?)
Are you getting attachments which are missing the fallback property so that you don't get a message on mattermost?
My issue with this is that the only bridge that supports slack attachments beside slack is mattermost.
I want to keep stuff as generic as possible between the different bridges, so I'm not going to add this option at the moment.

Yes, I want that the whole slack attachments part is being sent through the bridge. And yes we are not talking about file attachments. The message has got the fallback property filled in, I'm getting a message but not the message that I got in slack.
I can understand your issue about this, it's easier the get everything nearly the same.

@42wim I have done some more research and it looks like the bot is sending only the Title and Text in the attachment part through, is this correct?
I also have a question on your answer, is it not possible to add an argument in the config file that sounds like SlackAttachments. If this argument is set to true is sends the whole attachment part through and if it is set to false it stays the same as now a day. Is this an option?

Correct, if the Text is not empty it uses the Text and Title. If the Text is empty it uses Fallback
Yes, everything is possible, it just takes work :-)

I don't know if this is the issue for me, but some of the info/messages from github-bot on slack wont cross the bridge to mattermost:
Slack message:
[osmc/osmc] New comment by samnazarko on issue #534: [Installer] An error occurred while writing the image! Please consult logfile
I've updated to Catalina's .1 point release, so can test this shortly.
Mattermost post:
I've updated to Catalina's .1 point release, so can test this shortly.
Missing all the data on who posted and on which issue wher issue number and description is a link to issue on github.
We prefer the Slacks Github-bot to the Mattermost Github-bot.

@42wim I'm have no experiense with go at all, luckily VS-code has a go-lang plugin, forked this repository and cloned it localy and started to read through the code to the best of my general programming knowledge. I think i might have identified whats my problem. In short mattermost should convert the slack <url|linkdescription> but slack bridge never gives the bridge the data,
I send this in slack:
Okey [nl] http://google.com|Link-descrtiption, text after desction-text that is a url [alot of other formating text testing]
what is in debug log:
"[0144] DEBUG slack:        <= Message is &config.Message{Text:"Okey\nhttp://google.com/ , text after desction-text that is a url\n"
mb-sdsebug.txt

Been reading and playing around with the code, still have to Google 50% of the code to understand it somewhat. But from what I've gathered, Matterbridge was built with Matter most v. 4 api. Been reading up on v 5 and it's now possible to sent attachments via Api using post.prop.
Anyway, what I' wondering about is that if I write a code draft, which enable most slack-attachments in bridge  out on mattermost, enabled via a config key in [matter most]. It will probably never pass code-checks for a matterbridge pr. Due to my complete lack of code optimization in golang.
Is there someone that can help a golang newbie to go from newbie draft to actual code that can be PR'ed?

After some more tinkering I did have a semi working solution: In bridge/slack/handlers.go HandleAttachments, if msg.extra != "", I added code that parsed attachments and contructed a msg.text with data derived from attachments, drawback any linebreak added to msg.txt to part header (attachment pretext + \n +title+title_link)) \n original msg.text/attachemt.txt(handle the links to mattermost-links) \n footer (attachment footer+footer_link). This was never gonna be accepted as a pr, and since it didn't accept my \n I scrapped the ide.
Next try, in bridge/mattermost/mattermost.go Send-function. Trying convert slackattachments from msg.Extra["slack_attachment" , converting for example AuthorName to author_name and change any slack-link to a mattermost link in fallback, pretext, text and footer. Ran into a problem two folded problem, no PostMessageProp in matterclient/messages.go(which i fixed, but late scrapped) and  ranging msg.extra["slack_attachment"] din't populate properly (PEBKAC). Realized again that this wasn't the proper place for the code.
Last try, and this is where I am right now, in bridge/helper.go func HandleSlackAttachments i thought it was a good idea to do the work needed:

check if Slack2MMattachment is set in toml  (cant use IsKeySet, since i don't have a bridge to reference)
2.extract the need and parse info,

change the needed key-names,
link rewriting in attachment.text , attachment.pretext, attachment.fallback and attachment.footer

create a "footer-string" in msg.txt, by extracting attachment.footer attachment.footericon then set msg.text = attachment.text/msg.txt + footer-string


feature request: add a bracket or brackets automatically to completions

Hi! It seems to me that it would be convenient to automatically add to completions:

brackets if no arguments are expected (as in glimpse())
or a single bracket if arguments are expected (as in mutate().

Please consider adding this feature.

Hi @maxim-uvarov this looks like a good improvement, I'll try to add it.


FR: Change order of custom regex replacement

Is Your Feature Request Related to a Problem? Please Describe.
The custom regex replacement is a great addition (thank you so much!). Often times you'd want to have multiple of these replacements run and the order in which they're run can affect the outcome. It would be great if the order could be changed from the interface. There is of course a workaround: re-add rules to get the order that you want.
Describe the Solution You'd Like
I would like to be able to rearrange the order in which regexes are run.
Describe Alternatives You've Considered
Re-adding a rule to get the order I want is a good workaround, but makes it a bit difficult to manage as the rule set grows.
Additional Context
Here's an example of what I'm doing with this. I've set up the day planner plugin and use it for the pomodoro time-boxing method. My template has divided my whole day in half hour chunk tasks like so:
Prep the day

[ ] 9:00
[ ] 9:25 BREAK
[ ] 9:30 Meeting example
...

Finalize the day

[ ] 16:30
[ ] END

During the day I fill out what I've been doing, but not for all of these tasks. Day planner automatically checks all of these tasks when the time has passes, so at the end of the day I end up with this:
Prep the day

[x] 9:00
[x] 9:25 BREAK
[x] 9:30 Meeting example
...

Finalize the day

[x] 16:30
[ ] END

Note that the last one is never checked (not sure if this is a bug in day planner, but doesn't matter). I use the regex replacement to remove all the tasks that are checked + (empty after the time or 'BREAK'). However, the last line (- [] END) I wanted to remove as well, which I can only do if the previous line is checked. Now, when that previous line is empty, it is removed and the regex doesn't remove the last line. The order in which the regexes run makes the difference. If I remove the last line first, and then the others, it works. Sorry for the wall of text, but it's a real life example :)

Oh lol, I didn't realize my example would show up as actual tasks in this ticket, sorry about that!

Sorry about the delay. I just went and checked how bad the logic change would be to do this and realized it would be relatively easy. I just had to make some CSS changes to make it work.
This change should go out with the next release and it is on master now. Please let us know if this does not work.


First and last IPs from CIDR block is ignored

I have block of 4 IPs as cidr-global, but kube-vip-cloud-provider is throwing error after using 2nd and 3rd addresses:
Error syncing load balancer: failed to ensure load balancer: no addresses available in [envoy-gateway-system] cidr [{IP}/30]

It's introduced by this PR https://github.com/kube-vip/kube-vip-cloud-provider/pull/69/files#diff-a166480b55dac4f8c491356fb0bd1930e511f677c59a01453e9dbb6493c8a5e8R26-R43
It assumes the first ip and last ip would be occupied by network ip and broadcast ip regardless how large the network range is. Previously we only skip .0 and .255.
I think we could use at least below 2 options:

reverse to the original option, and if the user wants to ignore some IP, we add skip-ip-range-<namespace> option, if the user want to skip some ip, just add it here, then it could work with all kinds of customization of network if they want to use cidr
Add a flag that skip-cidr-end-ips which set default to false, if we need to skip end ips then we ask the user to enable that. otherwise we won't skip the end ips, unless it's .0 and .255 (original behavior)

On the second thought, it make more sense to exclude the first ip and last ip in cidr. If you just want a specific range to be allocated, you can use ip range instead. Let me update the doc

On the second thought, it make more sense to exclude the first ip and last ip in cidr. If you just want a specific range to be allocated, you can use ip range instead. Let me update the doc

Can you explain why does it makes sense? I just want to understand. It's not a problem configuring by range, but sometimes you want to write it cleaner.)
Aren't these correct for CIDR Blocks with:

up to 24bit:  always start with 0 and end with 255 with inbetween  0,255 addresses which also does need skipping.

24bit: starts and ends with 0 and 255

25 to 30: only these can use id or broadcast adresses other than 0 or 255.

I didn't read the code, but in my opinion it is only targeted for 25-30bit adresses and only if they are required to setup broadcast and id adresses

MM, your point also make sense, when we specify cidr for allocating IP, who knows whether that cidr contains broadcast and id ip or not. Maybe we could have a flag to explicitly skip the end ips. And switch back to the original behavior.
Hi @wusendong, since you made this change https://github.com/kube-vip/kube-vip-cloud-provider/pull/69/files#diff-a166480b55dac4f8c491356fb0bd1930e511f677c59a01453e9dbb6493c8a5e8R26-R43, are you ok with above suggestion?

MM, your point also make sense, when we specify cidr for allocating IP, who knows whether that cidr contains broadcast and id ip or not. Maybe we could have a flag to explicitly skip the end ips. And switch back to the original behavior.
Hi @wusendong, since you made this change https://github.com/kube-vip/kube-vip-cloud-provider/pull/69/files#diff-a166480b55dac4f8c491356fb0bd1930e511f677c59a01453e9dbb6493c8a5e8R26-R43, are you ok with above suggestion?

I had a similar confusion when I wrote that PR, when using range-global it should be up to the user to ensure that it's full of usable IPs, and we shouldn't take it upon ourselves to remove .0 and .255.
So I agree more with Option 2 in your previous comment, let the user specify the list of IPs to skip, but remove the code that automatically skips .0 and .255 so that the semantics are intuitively expected by the normal user.

Thanks for the input, what about option 3, user might not always they have a networkid or broadcast ip when using cidr

Thanks for the input, what about option 3, user might not always they have a networkid or broadcast ip when using cidr

Although option 3 makes new users confused, I think option 3 is acceptable because it maintains the compatibility of the original options.

In the newest 0.0.10 release I reverted the default behavior to previous one and took option 3 as a workaround for now. Just a head up


Add food views

In this pull request...

display a list of food as in the wireframe.
food list view leads to a form that allows users to add new food.
authorized user can delete food

Hi @see-why ,
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉
Good luck on the next ones
Cheers and Happy coding!👏👏👏


Goals/principles doc

Just dropping a markdown file to start collecting them and writing them down in one place. Feel free to comment on the couple i added :)

Warnings

:warning:
There are no updates provided to CHANGELOG. Ensure there are no publicly visible changes introduced by this PR.

  Generated by :no_entry_sign: dangerJS

Closing for now as it's obviously outdated.


[Bug]: Segfault when sending concurrent requests to chroma in server mode

What happened?
Using chroma in server mode in a docker container. Connecting to it using a custom asynchronous client.
-> Many, many segmentation faults and container restarts. Also had multiple collections with the same name, even though my client always uses get_or_create. Barely usable at all.
#570 mentions that the code is not threadsafe, so I assume the reason for this bug is that FastAPI puts chroma into a thread pool
Using async def instead of def seems to fix the problem on my side. PR incoming
https://fastapi.tiangolo.com/async/#path-operation-functions
Versions
main branch as of today, docker container, tilt.dev, WSL2
Relevant log output
2023-06-06 11:07:13 INFO     uvicorn.access  <IP_ADDRESS>:39558 - "GET /api/v1/collections/3E3SBTKO4G631XV8 HTTP/1.1" 500
2023-06-06 11:07:13 INFO     uvicorn.access  <IP_ADDRESS>:39570 - "POST /api/v1/collections/ HTTP/1.1" 307
2023-06-06 11:07:13 INFO     uvicorn.access  <IP_ADDRESS>:39572 - "POST /api/v1/collections/ HTTP/1.1" 307
2023-06-06 11:07:13 INFO     uvicorn.access  <IP_ADDRESS>:39584 - "POST /api/v1/collections HTTP/1.1" 200
2023-06-06 11:07:13 INFO     uvicorn.access  <IP_ADDRESS>:39588 - "POST /api/v1/collections HTTP/1.1" 200
/docker_entrypoint.sh: line 5:    68 Segmentation fault      uvicorn chromadb.app:app --workers 1 --host <IP_ADDRESS> --port 8000 --proxy-headers --log-config log_config.yml

@alfink thank you for looking into this, digging in now

PR appears to fix this hanged state and error message below
chroma-server-1      | munmap_chunk(): invalid pointer
chroma-server-1      | 2023-06-20 08:02:40 INFO     uvicorn.error   Waiting for application startup.
chroma-server-1      | 2023-06-20 08:02:40 INFO     uvicorn.error   Application startup complete.
chroma-server-1      | 2023-06-20 08:04:23 INFO     uvicorn.access  <IP_ADDRESS>:55000 - "GET /api/v1/collections HTTP/1.1" 200
chroma-server-1      | 2023-06-20 08:04:23 ERROR    chromadb.server.fastapi list index out of range
chroma-server-1      | Traceback (most recent call last):
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/anyio/streams/memory.py", line 98, in receive
chroma-server-1      |     return self.receive_nowait()
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
chroma-server-1      |     raise WouldBlock
chroma-server-1      | anyio.WouldBlock
chroma-server-1      |
chroma-server-1      | During handling of the above exception, another exception occurred:
chroma-server-1      |
chroma-server-1      | Traceback (most recent call last):
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 43, in call_next
chroma-server-1      |     message = await recv_stream.receive()
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/anyio/streams/memory.py", line 118, in receive
chroma-server-1      |     raise EndOfStream
chroma-server-1      | anyio.EndOfStream
chroma-server-1      |
chroma-server-1      | During handling of the above exception, another exception occurred:
chroma-server-1      |
chroma-server-1      | Traceback (most recent call last):
chroma-server-1      |   File "/chroma/./chromadb/server/fastapi/__init__.py", line 57, in catch_exceptions_middleware
chroma-server-1      |     return await call_next(request)
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 46, in call_next
chroma-server-1      |     raise app_exc
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 36, in coro
chroma-server-1      |     await self.app(scope, request.receive, send_stream.send)
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 75, in __call__
chroma-server-1      |     raise exc
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 64, in __call__
chroma-server-1      |     await self.app(scope, receive, sender)
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
chroma-server-1      |     raise e
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
chroma-server-1      |     await self.app(scope, receive, send)
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 680, in __call__
chroma-server-1      |     await route.handle(scope, receive, send)
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 275, in handle
chroma-server-1      |     await self.app(scope, receive, send)
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 65, in app
chroma-server-1      |     response = await func(request)
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 231, in app
chroma-server-1      |     raw_response = await run_endpoint_function(
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 162, in run_endpoint_function
chroma-server-1      |     return await run_in_threadpool(dependant.call, **values)
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
chroma-server-1      |     return await anyio.to_thread.run_sync(func, *args)
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
chroma-server-1      |     return await get_asynclib().run_sync_in_worker_thread(
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
chroma-server-1      |     return await future
chroma-server-1      |   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
chroma-server-1      |     result = context.run(func, *args)
chroma-server-1      |   File "/chroma/./chromadb/server/fastapi/__init__.py", line 194, in delete_collection
chroma-server-1      |     return self._api.delete_collection(collection_name)
chroma-server-1      |   File "/chroma/./chromadb/api/local.py", line 230, in delete_collection
chroma-server-1      |     self._db.delete_collection(name)
chroma-server-1      |   File "/chroma/./chromadb/db/duckdb.py", line 143, in delete_collection
chroma-server-1      |     collection_uuid = self.get_collection_uuid_from_name(name)
chroma-server-1      |   File "/chroma/./chromadb/db/duckdb.py", line 83, in get_collection_uuid_from_name
chroma-server-1      |     return self._conn.execute(
chroma-server-1      | IndexError: list index out of range
chroma-server-1      | 2023-06-20 08:04:23 INFO     uvicorn.access  <IP_ADDRESS>:55016 - "DELETE /api/v1/collections/project_1 HTTP/1.1" 500
chroma-server-1      | 2023-06-20 08:04:23 INFO     uvicorn.access  <IP_ADDRESS>:55020 - "POST /api/v1/collections HTTP/1.1" 200
chroma-server-1      | 2023-06-20 08:04:23 INFO     uvicorn.access  <IP_ADDRESS>:55000 - "POST /api/v1/collections/c76cfef2-cf4c-4cb5-ad4b-7c5523e3d61a/get HTTP/1.1" 200
chroma-server-1      | 2023-06-20 08:04:26 INFO     chromadb.db.duckdb collection with name project_1 already exists, returning existing collection
chroma-server-1      | 2023-06-20 08:04:26 INFO     uvicorn.access  <IP_ADDRESS>:55024 - "POST /api/v1/collections HTTP/1.1" 200
chroma-server-1      | 2023-06-20 08:04:26 INFO     uvicorn.access  <IP_ADDRESS>:55036 - "GET /api/v1/collections/c76cfef2-cf4c-4cb5-ad4b-7c5523e3d61a/count HTTP/1.1" 200
chroma-server-1      | 2023-06-20 08:04:26 DEBUG    chromadb.db.index.hnswlib Index saved to /index_data/index/index.bin
chroma-server-1      | 2023-06-20 08:04:26 DEBUG    chromadb.db.index.hnswlib Index saved to /index_data/index/index.bin
chroma-server-1      | 2023-06-20 08:04:26 INFO     uvicorn.access  <IP_ADDRESS>:55024 - "POST /api/v1/collections/c76cfef2-cf4c-4cb5-ad4b-7c5523e3d61a/add HTTP/1.1" 201
chroma-server-1      | 2023-06-20 08:04:26 INFO     uvicorn.access  <IP_ADDRESS>:55000 - "POST /api/v1/collections/c76cfef2-cf4c-4cb5-ad4b-7c5523e3d61a/get HTTP/1.1" 200
chroma-server-1      | 2023-06-20 08:04:48 INFO     uvicorn.access  <IP_ADDRESS>:45350 - "GET /api/v1/collections HTTP/1.1" 200
chroma-server-1      | munmap_chunk(): invalid pointer

@fullstackwebdev which PR?

I want to say that adding async seemed to fix my concurrent requests.
The concurrency worked for a small project, but when I added more documents, then the server stopped working(no crash but unresponsive) if I queried in quick succession. I can guess that this was also the problem when adding documents in quick succession.
With this change, there are some logs in chroma about knn query, and it waits about 10s to return answers, but it does in the end.
chroma-server-1      | 2023-07-11 09:40:15 DEBUG    chromadb.db.index.hnswlib time to run knn query: 0.0009410381317138672
chroma-server-1      | 2023-07-11 09:40:17 DEBUG    chromadb.db.index.hnswlib time to pre process our knn query: 0.0026051998138427734
chroma-server-1      | 2023-07-11 09:40:17 DEBUG    chromadb.db.index.hnswlib time to run knn query: 0.0006129741668701172
chroma-server-1      | 2023-07-11 09:40:19 DEBUG    chromadb.db.index.hnswlib time to pre process our knn query: 0.002637147903442383

Related to the last message, I want to add that for longer sequences, the chroma seems to hang, and then return a batch of responses in a single go. This is with the async fix.
Here is how the responses come in if I send queries after 1s delay. (| = query sent, - = response)
||||-|-|||---|-||||---||--|||||----|||---||||----||||----|||||----||||----|||||-----||||||-----||||||------|||||-----||||||||-------||||||------|||||-------------

It's obvious that queries get in some kind of queue and then return at the same time in batches.
I would expect that queries get handled in a more sequential manner. Currently there is possibly a 10s lag between a request and a response.

I am not sure how to explaining the batching, other than in a RAM constrained environment, you may be running into a high number of page swaps. The queries are not batched and run. Could you share your data shape, scale and machine parameters @tonisives ?

Ok I found the problem, it is because of my where clause. It is quite long:
metadata format:
  "metadatas": [
    {
      "source": "595dda75",
      "platform": "platform-1",
      "c_date":<PHONE_NUMBER>,
      "severity": 3,
      "c_name": "name-1",
      "loc": {
        "lines": {
          "from": 17,
          "to": 62
        }
      }
    },

where:
  let dateWhere = {
    c_date: {
      $gt: 0
    }
  }

  let severityWhere = [2, 3].map((it) => {
    return {
      severity: {
        $eq: it
      }
    }
  })

  let where = {
    $and: [
      dateWhere,
      {
        $or: severityWhere
      }
    ]
  } as any

query
  let result = (await collection.query({
    nResults: 10,
    queryTexts: [query],
    where: where
  }))

Without this where, the queries return instantly

@tonisives we are looking into adding indexes here which will help a lot!

@tonisives we are looking into adding indexes here which will help a lot!

Thanks! Looking forward to it!

I ran my program with 0.4.3, and the problem seems to be gone. Even with fast queries and with the where clause. This is without the async fix (clean 0.4.3 tag).

So for me, this issue is resolved.

Glad to hear it @tonisives - 0.4 shipped with changes to make concurrent adds/queries safe as well as switched to sqlite which helped with where performance. I'll close this out for now but feel free to reopen if it comes up again!