Bug: Query Performance Discrepancy: KuzuDB CLI vs. Rust Code

Kùzu version
rust - kuzu 0.6.0 kuzu_cli - Kuzu 0.6.0
What operating system are you using?
mac os somona (also tasted in winodws 11 23h2 with same result)
What happened?
We encountered a performance issue when testing a large query. Initially, the query took around 3.5 seconds to execute in our Rust code, which raised concerns about performance.
To investigate, we ran the same query in the KuzuDB CLI, where it surprisingly took only 252 ms on average. Suspecting a problem in our code, we simplified the test by writing a basic program that only runs the query and measures the time. In this case, the query took 3.46 seconds.

Thinking the problem might be related to unoptimized code, we compiled the Rust code in release mode. While the execution time dropped significantly, but it was still 368 ms—more than twice the time taken by the CLI.

We are looking for insights into why there’s such a large performance gap between the CLI and the Rust implementation, and how we can optimize the Rust execution further.
Are there known steps to reproduce?

Run a sample query using Rust in embedded mode and note the execution time.
Run the same query in the KuzuDB CLI and compare the execution time.

I have not been able to reproduce any significant difference in runtime between the CLI and the Rust API (when both are built in release mode), however there does seem to be a noticeable discrepancy between what the query result reports as its compilation/ execution time and the total time spent running the query. I think the solution is probably to add more timers to cover the parts of which currently aren't included.
That said, I haven't tried this on very many queries, and I only noticed a significant difference on small queries, mostly because what I'm observing appears to be a small amount of overhead that's not really scaling with the query time (e.g. for a simple query where the compilation+execution time was reported as only ~3ms the total time was ~20ms, but on a comparably simple query with a compiliation+execution time of ~550ms the total time was ~560ms).
To double-check that this is the case for your workload, could you try comparing the results of QueryResult::get_compiling_time and get_executing_time to the results from the CLI? I suspect that the issue is just that this particular query is spending some extra time doing work when the timers aren't running.

Thank you letting us know about this functions. You are correct their is no discrepancy with compile time or execution time
here is the result

[Time taken in rust sample code]

[Time taken in kuzu cli]
Execution time + compile time (in kuzu cli) = 373.36
we can see this nearly 250ms time overhead (Total time - (Execution time + compile time)) leaving compile time and execution time  every time we run the same query mentioned above
Can we get more insight on why their is overhead and anyway if wecan reduced it

Please refer to the image below, where we attempted to find the same node using two different queries: a larger query (path finder) and one using the UUID (primary key).

In the second case (using UUID), the overhead is minimal, just around 4 ms, and both queries return the same node.
We would like to understand where the time is being spent in the first case (beyond execution and compilation time), as the difference is significant.

Hi @biplab5464, we may need some more information on the query you're running.

we can see this nearly 250ms time overhead (Total time - (Execution time + compile time)) leaving compile time and execution time every time we run the same query mentioned above
Can we get more insight on why their is overhead and anyway if wecan reduced it

If you could paste the query itself, removing any sensitive parts and replacing with placeholders, that could help us understand better what the query is doing and suggest some ideas.

In the second case (using UUID), the overhead is minimal, just around 4 ms, and both queries return the same node.

for this one, shouldn't this be expected? We construct a hash index on the primary key column of every table, so query performance is bound to be faster than if you were trying to lookup properties on a node that didn't have a hash index. There would be multiple additional intermediate steps in the query plan generated in the latter case.

Here is a replica query, so the db is like Tree data structure,  we have 3 schemas Root, Node and Leaf and 1 Group relationships called 'CONTAINS' that are  FROM Root To Node , FROM Node TO Node  and  FROM Node To Leaf
MATCH (n:Root)-[:CONTAINS]->({name:'Groups'})
        -[:CONTAINS]->(Node:{name:'Group 1'})
        -[:CONTAINS]->(:Node{name:'jack'})
        -[:CONTAINS]->(:Node{name:'smith'})
        -[:CONTAINS]->(:Node{name:'denath'})
        -[:CONTAINS]->(:Node{name:'child'})
        -[:CONTAINS]->(:Node{name:'child of child'})
        -[:CONTAINS]->(:Node{name:'grandchild'})
        -[:CONTAINS]->(:Node{name:'1'})
        -[:CONTAINS]->(:Node{name:'2'})
        -[:CONTAINS]->(:Node{name:'3'})
        -[:CONTAINS]->(:Node{name:'4'})
        -[:CONTAINS]->(:Node{name:'5'})
        -[:CONTAINS]->(:Node{name:'6'})
        -[:CONTAINS]->(:Node{name:'7'})
        -[:CONTAINS]->(:Node{name:'8'})
        -[:CONTAINS]->(x {name:'leaf'})

for this one, shouldn't this be expected? We construct a hash index on the primary key column of every table, so query performance is bound to be faster than if you were trying to lookup properties on a node that didn't have a hash index. There would be multiple additional intermediate steps in the query plan generated in the latter case.

Here we are comparing the overhead Time (Total time - (Execution time + compile time)). This logic set true for (Execution time + compile time) which happens during compile and execution of the query but we are talking the extra time that is happening when we get the Same Node but from different query, that is path finder : 257.712 ms and from uuid : 3.917 sec ( Please refer to picture in my previous comment ). Please let me know me if i am wrong.

just wanted to confirm, are you both using Windows (for Ajay's case it's clear based on the terminal screenshots).

We have both Windows and macOS, and the issue is the same on both platforms
In a single comment, we are either using Windows or macOS, not mixing both platforms.


fix: solve not correct value displayed when max btn is used

[x] solve not correct value displayed when max btn is used

Code seems legit and looks that fix the issue, only need to fix my Pali build to test it first before I can say it can be merged.

I test it, found an error about decimals coming as undefined but already fixed it. Now is working and can merge it.


habitat model - WCT habitat (CW model) in ELKR is incorrect

No habitat is modelled throughout the majority of stream in the ELKR watershed group, with odd exception of the Fording River.
CW and gradient seem to be fine on other streams, it appears to be a processing issue.

Seems to be the same result with the discharge based model, and in other watershed groups.

https://github.com/smnorris/bcfishpass/blob/main/model/02_habitat_linear/sql/habitat_wct.sql#L47
👎  🤯


CheckpointFunction with Flex-attn BlockMask

Hi there, thanks for the wonderful project!
I have one question related to the checkpoint function and wonder if flex-attn can work with this feature. In Megatron-LM I replaced the original attn_mask with the BlockMask for the transformer, and it yields an error related the the checkpoint function:
# megatron checkpoint funciton impl
class CheckpointFunction(torch.autograd.Function):
    """Checkpoint Function

    This function is adapted from torch.utils.checkpoint with two main changes:
    1) torch.cuda.set_rng_state is replaced with `_set_cuda_rng_state`
    2) the states in the model parallel tracker are also properly tracked/set/reset.
    """

    @staticmethod
    def forward(ctx, run_function, distribute_saved_activations, *args):
        ctx.run_function = run_function
        ctx.distribute_saved_activations = distribute_saved_activations

        # Copy the rng states.
        ctx.fwd_cpu_rng_state = torch.get_rng_state()
        ctx.fwd_cuda_rng_state = torch.cuda.get_rng_state()
        ctx.fwd_cuda_rng_state_tracker = get_cuda_rng_tracker().get_states()

        with torch.no_grad():
            outputs = run_function(*args)

        # Divide hidden states across model parallel group and only keep
        # the chunk corresponding to the current rank.
        if distribute_saved_activations:
            ctx.input_0_shape = args[0].data.shape
            safely_set_viewless_tensor_data(
                args[0], split_tensor_into_1d_equal_chunks(args[0].data, new_buffer=True)
            )

        # Store everything.
        ctx.save_for_backward(*args)

        return outputs

[rank4]:   File "/home/tiger/code/Megatron-MLLM/megatron/core/tensor_parallel/random.py", line 307, in checkpoint
[rank4]:     return CheckpointFunction.apply(function, distribute_saved_activations, *args)
[rank4]:   File "/home/tiger/.local/lib/python3.9/site-packages/torch/autograd/function.py", line 575, in apply
[rank4]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[rank4]: TypeError: save_for_backward can only save variables, but argument 1 is of type BlockMask

The error msg is that the attn_mask(BlockMask for flex-attn) is not a tensor and thus cannot put in *args for ctx.save_for_backward. Later i found the checkpoint function impl.  of Pytorch, which may solve the ctx.save_for_backward problems by only saving tensors at ctx.save_for_backward:
class CheckpointFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, run_function, preserve_rng_state, *args):
        check_backward_validity(args)
        ctx.run_function = run_function
        ctx.preserve_rng_state = preserve_rng_state
        # Accommodates the (remote) possibility that autocast is enabled for cpu AND gpu.
        ctx.device = _infer_device_type(*args)
        ctx.device_autocast_kwargs, ctx.cpu_autocast_kwargs = _get_autocast_kwargs(
            ctx.device
        )
        if preserve_rng_state:
            ctx.fwd_cpu_state = torch.get_rng_state()
            # Don't eagerly initialize the cuda context by accident.
            # (If the user intends that the context is initialized later, within their
            # run_function, we SHOULD actually stash the cuda state here.  Unfortunately,
            # we have no way to anticipate this will happen before we run the function.)
            ctx.had_device_in_fwd = False
            device_module = _get_device_module(ctx.device)
            if getattr(device_module, "_initialized", False):
                ctx.had_device_in_fwd = True
                ctx.fwd_devices, ctx.fwd_device_states = get_device_states(*args)

        # Save non-tensor inputs in ctx, keep a placeholder None for tensors
        # to be filled out during the backward.
        ctx.inputs = []
        ctx.tensor_indices = []
        tensor_inputs = []
        for i, arg in enumerate(args):
            if torch.is_tensor(arg):
                tensor_inputs.append(arg)
                ctx.tensor_indices.append(i)
                ctx.inputs.append(None)
            else:
                ctx.inputs.append(arg)

        ctx.save_for_backward(*tensor_inputs)

        with torch.no_grad():
            outputs = run_function(*args)
        return outputs

    @staticmethod
    def backward(ctx, *args):
        if not torch.autograd._is_checkpoint_valid():
            raise RuntimeError(
                "When use_reentrant=True, torch.utils.checkpoint is incompatible"
                " with .grad() or passing an `inputs` parameter to .backward()."
                " To resolve this error, you can either set use_reentrant=False,"
                " or call .backward() without passing the `inputs` argument."
            )
        # Copy the list to avoid modifying original list.
        inputs = list(ctx.inputs)
        tensor_indices = ctx.tensor_indices
        tensors = ctx.saved_tensors
        device_module = _get_device_module(ctx.device)

        # Fill in inputs with appropriate saved tensors.
        for i, idx in enumerate(tensor_indices):
            inputs[idx] = tensors[i]

        # Stash the surrounding rng state, and mimic the state that was
        # present at this time during forward.  Restore the surrounding state
        # when we're done.
        rng_devices = []
        if ctx.preserve_rng_state and ctx.had_device_in_fwd:
            rng_devices = ctx.fwd_devices
        with torch.random.fork_rng(
            devices=rng_devices, enabled=ctx.preserve_rng_state, device_type=ctx.device
        ):
            if ctx.preserve_rng_state:
                torch.set_rng_state(ctx.fwd_cpu_state)
                if ctx.had_device_in_fwd:
                    set_device_states(ctx.fwd_devices, ctx.fwd_device_states)
            detached_inputs = detach_variable(tuple(inputs))     <------------------

            device_autocast_ctx = torch.amp.autocast(
                device_type=ctx.device, **ctx.device_autocast_kwargs
            ) if torch.amp.is_autocast_available(ctx.device) else contextlib.nullcontext()
            with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
                outputs = ctx.run_function(*detached_inputs)

        if isinstance(outputs, torch.Tensor):
            outputs = (outputs,)

        # run backward() with only tensor that requires grad
        outputs_with_grad = []
        args_with_grad = []
        for i in range(len(outputs)):
            if torch.is_tensor(outputs[i]) and outputs[i].requires_grad:
                outputs_with_grad.append(outputs[i])
                args_with_grad.append(args[i])
        if len(outputs_with_grad) == 0:
            raise RuntimeError(
                "none of output has requires_grad=True,"
                " this checkpoint() is not necessary"
            )
        torch.autograd.backward(outputs_with_grad, args_with_grad)
        grads = tuple(
            inp.grad if isinstance(inp, torch.Tensor) else None
            for inp in detached_inputs
        )

        return (None, None) + grads

And another question comes: since it detach input for tensor (<----  in code block), which operates on attn_mask if it is a tensor, I wonder if skipping this detach operation is doable when the attn_mask is a BlockMask used for flex-attn, or is there other safe ways to use flex-attn with the checkpoint function?
Thanks!

cc @soulitzer autograd function should compose with HOPs right?

Yeah it looks like the checkpoint function is not handiling the inputs correctly, it was probably written with the assumption that it would only be applied to a function that takes only tensor args
The tensors composing the BlockMask should never require grad / interact with the autograd graph so I think that the detach is a noop

@drisspg Thank you for your prompt reply. According to your answer, I can use the pytorch implemented checkpoint function with BlockMask :)


create a login page using neumorphism

create a login page using neumorphism which takes email id and password as input

I want to contribute to this please assign me under hacktoberfest tag.

I would like to work on this. Kindly Assign me the issue.

Please see PR #12
Thanks and Regards

You pr is merged successfully. Thanks for your contribution


Firebase.initializeApp() crashes chrome.

Discussed in https://github.com/FirebaseExtended/flutterfire/discussions/7017

Originally posted by komineyuki September 18, 2021
I'm debugging on chrome, and when Firebase.initializeApp() called, chrome crashes.
I installed Firebase on my flutter project;
//terminal
 flutter pub get firebase_auth
 flutter pub get firebase_core
 flutter pub get

And I enabled Firebase in pubspec.yaml.
//pubspec.yaml
  firebase_auth: ^3.1.1
  firebase_core: ^1.6.0

I saw the Firebase console screen and copied the CDN code to the <body> of index.html.
Then initialize Firebase with main.dart.
//main.dart
import 'package:flutter/material.dart';
import 'package:firebase_core/firebase_core.dart';
import 'package:firebase_auth/firebase_auth.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp();
  runApp(MaterialApp(
  ....

When await Firebase.initializeApp() called, all time chrome freezes so long time and VScode appears.
There is a yellow mark in the ui.webOnlyInitializePlatform(); part of web_entrypoint.dart.
image : https://i.stack.imgur.com/I7g30.png
Also I couldn't find any errors in debug console.
The following code will work fine.
//main.dart
    void main() {
      runApp(MaterialApp(
      ....

Addition;
I created a new Flutter project and installed flutter_core.
And I wrote the same code as above but it didn't work.
Thank you.

I re-wrote a part of index.html, like this;
<script src="https://www.gstatic.com/firebasejs/8.6.1/firebase-app.js"></script>

<script>
  const firebaseConfig = {
    apiKey: "**********************",
    authDomain: "***********",
    projectId: "***********",
    storageBucket: "***************",
    messagingSenderId: "**************",
    appId: "************",
    measurementId: "****************S"
  };

  // Initialize Firebase
  firebase.initializeApp(firebaseConfig);
  const analytics = getAnalytics(app);
</script>

It works.
I referred here
https://firebase.flutter.dev/docs/installation/web/


Best way to implement a simple jump function.

What is the best way to implement a simple jump function ?

Sorry for the delay in response, see Demo.manipulation, the code for it should help.


Add remote config for disabling the session start message

Goal
Adds a remote config field that prevents the session start message from being sent. We don't really use the start message & it adds bandwidth when making requests + complexity to cache failed requests on disks. Therefore we will add a remote config that disables sending it so that we can experiment with disabling it by default.
Testing
Added unit test coverage. Confirmed in an example app that sessions still show on the dashboard.

@fnewberg tagging you for review as this adds a new remote config field

if we decide to keep the start message (see earlier comment) i'll add a config for this when i'm back


FormBuilderChipsInput

Not able to set the data into chipsBuilder after returning from route. Can you help out with the FormBuilderChipsInput which on click goes to another route and on response gets the data and fills in form of Chips. Current Code

FormBuilderChipsInput(
attribute: "ABC",
decoration: InputDecoration(
contentPadding:
EdgeInsets.all(8),
labelText: "Label"),
validators: [],
initialValue:
widget.values,
chipBuilder: (context, state,
selectedOptions) {
return InputChip(
deleteIcon: Icon(null),
elevation: 5,
key: ObjectKey(selectedOptions),
label: Text(
selectedOptions.toString()),
onDeleted: () {
state.deleteChip(
selectedOptions);
},
materialTapTargetSize:
MaterialTapTargetSize
.shrinkWrap,
);
})

So in this case if I setState the widget.values then the FormBuilderChipsInput isn't updating, I am purposely taking another bool to manage the thing, which I don't want to do. Any another way?

This issue should be fixed. Feel free to reopen if not


The DATABASE_URL environment variable couldn't found in shell

What happened?
/app # python manage.py shell_plus
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/environ/environ.py", line 273, in get_value
value = self.ENVIRON[var]
File "/usr/local/lib/python3.6/os.py", line 669, in getitem
raise KeyError(key) from None
KeyError: 'DATABASE_URL'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 30, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/init.py", line 371, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.6/site-packages/django/core/management/init.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/init.py", line 204, in fetch_command
settings.INSTALLED_APPS
File "/usr/local/lib/python3.6/site-packages/django/conf/init.py", line 56, in getattr
self._setup(name)
File "/usr/local/lib/python3.6/site-packages/django/conf/init.py", line 43, in _setup
self._wrapped = Settings(settings_module)
File "/usr/local/lib/python3.6/site-packages/django/conf/init.py", line 106, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/local/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/app/config/settings/local.py", line 1, in
from .base import *  # noqa
File "/app/config/settings/base.py", line 41, in
'default': env.db('DATABASE_URL'),
File "/usr/local/lib/python3.6/site-packages/environ/environ.py", line 204, in db_url
return self.db_url_config(self.get_value(var, default=default), engine=engine)
File "/usr/local/lib/python3.6/site-packages/environ/environ.py", line 277, in get_value
raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the DATABASE_URL environment variable
What should've happened instead?
I found the below command in ./compose/production/django/entrypoint that "export DATABASE_URL" is only valid for the current shell, when you start a new shell, there is no DATABASE_URL environment variable.
export DATABASE_URL="postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}"
why not place this variable in ./.env/.local/.django? It always seems to be valid in the container.
Steps to reproduce
I started a project powered by latest cookiecutter-django version using docker-compose, and when I executed "docker-compose -f local.yml up -d", the app had been successful launched, and the http://<IP_ADDRESS>:8000/ can be visited successfully.
But when I executed "docker exec -it bila_django_1(my container name) sh",  and when  I want to do anything like "python manage.py shell_plus", there is a error like above.

In the entrypoint file you mentioned, there's an explicit explanation to as to why: # N.B. If only .env files supported variable expansion... Keep in mind, we make extensive use of .env files which, by nature, do not support shell variable expansion, therefore, entrypoint is the closest place to have such complex variables set.
You might wanna create a Makefile or something where the DATABASE_URL (and any other complex env var for that matter) would be provided in docker compose's -e argument while pre-emptively constructed from the underlying environment-specific envs. There's, by the way, an ongoing (proposed) Makefile effort already: https://github.com/pydanny/cookiecutter-django/pull/1879.
If you've got a better idea, feel free to send us a PR :)
Closing this one for now; suit yourself in re-opeining it should a PR from you to follow.

So how can we do if we want to run python manage.py in  a new shell? What I have for now is use docker exec -it -e DATABASE_URL=database_url python manage.py，but that is not convenient and sometimes not secure. Any other better way?

@zmrenwu see my comment at #1879

@zmrenwu I just had the same problem and what works for me is to source the entrypoint file so that the env variables become available:
Within container shell: source compose/production/django/entrypoint

You can echo the environment variables into ~/.bashrc file for the user in the container. See the below changes I've made to the django entrypoint file.
`
#!/bin/sh
set -o errexit
set -o pipefail
set -o nounset
echo "export CELERY_BROKER_URL=${REDIS_URL}" >> ~/.bashrc
if [ -z "${POSTGRES_USER}" ]; then
base_postgres_image_default_user='postgres'
echo "export POSTGRES_USER=${base_postgres_image_default_user}" >> ~/.bashrc
fi
echo "export DATABASE_URL=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}" >> ~/.bashrc
postgres_ready() {
python << END
import sys
import psycopg2
try:
psycopg2.connect(
dbname="${POSTGRES_DB}",
user="${POSTGRES_USER}",
password="${POSTGRES_PASSWORD}",
host="${POSTGRES_HOST}",
port="${POSTGRES_PORT}",
)
except psycopg2.OperationalError:
sys.exit(-1)
sys.exit(0)
END
}
until postgres_ready; do

&2 echo 'Waiting for PostgreSQL to become available...'
sleep 1
done
&2 echo 'PostgreSQL is available'

exec "$@"
`

You can echo the environment variables into ~/.bashrc file for the user in the container. See the below changes I've made to the django entrypoint file.
`
#!/bin/sh
set -o errexit
set -o pipefail
set -o nounset
echo "export CELERY_BROKER_URL=${REDIS_URL}" >> ~/.bashrc
if [ -z "${POSTGRES_USER}" ]; then
base_postgres_image_default_user='postgres'
echo "export POSTGRES_USER=${base_postgres_image_default_user}" >> ~/.bashrc
fi
echo "export DATABASE_URL=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}" >> ~/.bashrc
postgres_ready() {
python << END
import sys
import psycopg2
try:
psycopg2.connect(
dbname="${POSTGRES_DB}",
user="${POSTGRES_USER}",
password="${POSTGRES_PASSWORD}",
host="${POSTGRES_HOST}",
port="${POSTGRES_PORT}",
)
except psycopg2.OperationalError:
sys.exit(-1)
sys.exit(0)
END
}
until postgres_ready; do

&2 echo 'Waiting for PostgreSQL to become available...'
sleep 1
done
&2 echo 'PostgreSQL is available'

exec "$@"
`

Would there by any security risks by using this approach?

After you are in django container run
source /entrypoint

After you are in django container run
source /entrypoint

Do you think this should be included at some step of the deployment? I think it's not possible to be placed at ENTRYPOINT step in Dockefile.
Btw, do you know why does it happen?

Sourcing /entrypoint, while giving us the desired env. variables, also brings its own issues.
The script contains
set -o errexit
set -o pipefail
set -o nounset

which make sense as a Docker entrypoint, but render the shell session unsuitable for interactive use.
For instance, a simple env|grep zzz exits the terminal.
I'd thus suggest unsetting these shell options after the sourcing:
source /entrypoint && set +euo pipefail


Select2 4.0 compability problem

Select2 4.0 announced "no more hidden input tags" for advanced usage https://select2.github.io/announcements-4.0.html#hidden-input . It's using select tag instead for type=hidden

Thanks but I haven't really looked at Select2 version 4.  I don't have much spare time right now but I hope to get around to making it version 4 compatible.

https://github.com/tetranz/select2entity-bundle/pull/4


getx GetConnect onInit not work

getx GetConnect onInit not working causing this error. Unhandled Exception: Invalid argument(s): No host specified in URI albums/1
Here is a simple example
https://github.com/giantss/flutterGetxTest/tree/main/testapp
example:
home_binding.dart
import 'package:get/get.dart';
import 'package:testapp/app/modules/home/providers/home_provider.dart';

import '../controllers/home_controller.dart';

class HomeBinding extends Bindings {
  @override
  void dependencies() {
    Get.lazyPut<HomeController>(
      () => HomeController(),
    );
    Get.lazyPut<HomeProvider>(
          () => HomeProvider(),
    );
  }
}

home_provider.dart
import 'package:get/get.dart';
import '../home_model.dart';

class HomeProvider extends GetConnect {
  @override
  void onInit() {    // no call
    httpClient.defaultDecoder = (map) {
      if (map is Map<String, dynamic>) return Home.fromJson(map);
      if (map is List) return map.map((item) => Home.fromJson(item)).toList();
    };
    httpClient.baseUrl = 'https://jsonplaceholder.typicode.com/';
  }

  Future<Home?> getHome(int id) async {
    final response = await get('albums/$id');  // error is thrown here
    return response.body;
  }

  Future<Response<Home>> postHome(Home home) async => await post('home', home);
  Future<Response> deleteHome(int id) async => await delete('home/$id');
}

pubspec.yaml
name: testapp
version: 1.0.0+1
publish_to: none
description: A new Flutter project.
environment:
  sdk: '>=2.17.6 <3.0.0'

dependencies:
  cupertino_icons: ^1.0.2
  get: 4.6.5
  flutter:
    sdk: flutter

dev_dependencies:
  flutter_lints: 2.0.1
  flutter_test:
    sdk: flutter

flutter:
  uses-material-design: true

@jonataslaw
Can you help take a look? This problem has been bothering me for a while.

@giantss You have to use Get.put instead of Get.lazyPut.

@giantss You have to use Get.put instead of Get.lazyPut for the provider.

@pratamatama The demo I provided is replaced with the following code and it still doesn't work.
class HomeBinding extends Bindings {
  @override
  void dependencies() {
    Get.lazyPut<HomeController>(
      () => HomeController(),
    );
    Get.put(HomeProvider);
    // Get.lazyPut<HomeProvider>(
    //       () => HomeProvider(),
    // );
  }
}

@giantss put it above your lazyput, that should work.
also, you might want to include the parenthesis on the HomeProvider so it doesn't treat it as a Type instead of an instance.
like so

Get.put(HomeProvider());

Get.lazyPut<HomeController>(
    () => HomeController(),
);

@giantss put it above your lazyput, that should work.
also, you might want to include the parenthesis on the HomeProvider so it doesn't treat it as a Type instead of an instance.
like so
Get.put(HomeProvider());

Get.lazyPut<HomeController>(
    () => HomeController(),
);

@jonataslaw Tried it all, still doesn't work. I have a demo example above, can you clone it and help? thank you very much。

I can initialize getconnect's onInit in two ways.
extending GetxService and using the init() implementation, injecting with
await Get.putAsync(() => MyApi().init());

and also with GetxService's onInit injecting
Get.put(MyApi())

in this way, the problem at the moment is that even initializing and managing to retrieve the assignments I make in httpClient such as baseUrl being equal to the value I entered in the previous line, and using httClient .<method> or directly get methods like post()or get() it doesn't recognize this variable outside of inits. Hope this helps, although it's not the complete answer.
And I'm following

@kauemurakami
Thanks a lot for your answer.
This is just a very simple demo, and I didn't even write any logic. In theory, this kind of problem should not occur. Is this an official bug? @jonataslaw

big bug

Find this website explained as DI functionality, it has some auxiliary parameters that can help the put not die: the-flutter-getx-ecosystem-dependency-injection-

I use this code and worked well

@giantss put it above your lazyput, that should work.
also, you might want to include the parenthesis on the HomeProvider so it doesn't treat it as a Type instead of an instance.
like so
Get.put(HomeProvider());

Get.lazyPut<HomeController>(
    () => HomeController(),
);

@jonataslaw Tried it all, still doesn't work. I have a demo example above, can you clone it and help? thank you very much。

I think I see the problem. You are putting a GetConnect provider into the binding, but then directly instantiating it inside the controller. You can't do that.
Instead of doing var provider = HomeProvider(), try to use final provider = Get.find<HomeProvider>().
It will use the instance of HomeProvider that was previously being injected from the binding.
OR... If you prefer, refactor your code to be more like this.
home_bindings.dart
class HomeBinding extends Bindings {
  @override
  void dependencies() {
    Get.lazyPut(
      () => HomeController(
        provider: Get.put(HomeProvider()), // HERE
      )
    );
  }
}

home_controller.dart
class HomeController extends GetxController {
  HomeController({ required HomeProvider provider }) : _provider = provider; // HERE
  final HomeProvider _provider;

  final homeData = Home().obs;

  @override
  void onInit() {
    fetchHomeData();
    super.onInit();
  }

  void fetchHomeData() {
    homeData.value = _provider.getHome(1) as Home;
  }
}

@giantss

I was able to use it this way #894, I hope I can still help, in this case the problem was set baseUrl, but also apply same modifiers.


crash when using arrow keys to navigate query results

2023-06-02.txt
Flow Launcher version: 1.15.0
OS Version: 22631.1830
IntPtr Length: 8
x64: True
Python Path: C:\Users\12069\AppData\Roaming\FlowLauncher\Environments\Python\PythonEmbeddable-v3.8.9\pythonw.exe
Node Path:
Date: 06/02/2023 14:27:50
Exception:
OneNote Exception:
Websites: https://github.com/Odotocodot/Flow.Launcher.Plugin.OneNote
Author: Odotocodot
Version: 1.1.1
Flow.Launcher.Core.ExternalPlugins.FlowPluginException: Unable to cast COM object of type 'Microsoft.Office.Interop.OneNote.ApplicationClass' to interface type 'Microsoft.Office.Interop.OneNote.IApplication'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{452AC71A-B655-4967-A208-A4CC39DD7949}' failed due to the following error: The RPC server is unavailable. (0x800706BA).
---> System.InvalidCastException: Unable to cast COM object of type 'Microsoft.Office.Interop.OneNote.ApplicationClass' to interface type 'Microsoft.Office.Interop.OneNote.IApplication'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{452AC71A-B655-4967-A208-A4CC39DD7949}' failed due to the following error: The RPC server is unavailable. (0x800706BA).
at System.StubHelpers.StubHelpers.GetCOMIPFromRCW(Object objSrc, IntPtr pCPCMD, IntPtr& ppTarget, Boolean& pfNeedsRelease)
at Microsoft.Office.Interop.OneNote.ApplicationClass.FindPages(String bstrStartNodeID, String bstrSearchString, String& pbstrHierarchyXmlOut, Boolean fIncludeUnindexedPages, Boolean fDisplay, XMLSchema xsSchema)
at ScipBe.Common.Office.OneNote.OneNoteProvider.<>c__DisplayClass4_0.b__0(Application oneNote)
at ScipBe.Common.Office.OneNote.OneNoteProvider.CallOneNoteSafely[T](Func2 action)    at ScipBe.Common.Office.OneNote.OneNoteProvider.FindPages(String searchString)    at Flow.Launcher.Plugin.OneNote.OneNotePlugin.Query(Query query)    at System.Threading.Tasks.Task1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Flow.Launcher.Core.Plugin.PluginManager.<>c__DisplayClass22_0.<b__0>d.MoveNext() in C:\projects\flow-launcher\Flow.Launcher.Core\Plugin\PluginManager.cs:line 194
--- End of stack trace from previous location ---
at Flow.Launcher.Infrastructure.Stopwatch.DebugAsync(String message, Func`1 action) in C:\projects\flow-launcher\Flow.Launcher.Infrastructure\Stopwatch.cs:line 40
at Flow.Launcher.Core.Plugin.PluginManager.QueryForPluginAsync(PluginPair pair, Query query, CancellationToken token) in C:\projects\flow-launcher\Flow.Launcher.Core\Plugin\PluginManager.cs:line 216
--- End of inner exception stack trace ---
at Flow.Launcher.Core.Plugin.PluginManager.QueryForPluginAsync(PluginPair pair, Query query, CancellationToken token) in C:\projects\flow-launcher\Flow.Launcher.Core\Plugin\PluginManager.cs:line 216
at Flow.Launcher.ViewModel.MainViewModel.<>c__DisplayClass135_0.<g__QueryTask|2>d.MoveNext() in C:\projects\flow-launcher\Flow.Launcher\ViewModel\MainViewModel.cs:line 836
--- End of stack trace from previous location ---
at Flow.Launcher.ViewModel.MainViewModel.QueryResults() in C:\projects\flow-launcher\Flow.Launcher\ViewModel\MainViewModel.cs:line 837
at System.Threading.Tasks.Task.<>c.b__128_0(Object state)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

Hey is this error constant throwing? and were you using the notebook explorer when it happened?

Crashed twice when I first installed flow launcher and tried the plugin, I just tested a few times moments ago, all good now. Prob just because I didn't restart after I installed flow launcher. Nice plugin BTW

Crashed twice since I installed flow launcher and tried the plugin, I just tested a few times moments ago, all good now. Prob just because I didn't restart after I installed flowlaucher.


willow interface with WIS

I have setup a WIS on an older AMD motherboard with a GTX 1080ti. The response from willow to the WIS is super fast - but....not accurate.
When I use the exact same configuration on willow for the ESPBOX and for instance want to control the "Foyer Light" or "Foyer Lamp" it respond back with "Fire light or "Fire lamp". Half the time it is an 'unrecognized command".
Also, I have lamps on both sides of my bed, defined in HA as a group "bedsides" - when connected to WIS it either sees it a "bed side" or "beside" - but I cannot get it to recognize "bedsides". When I revert back to Multinet, all the above problems goes away.
I have even tried different settings for VAD timeout (reduced it when I went to WIS) and DET_MODE_3CH-95 changed to 3CH_90 - but it still does not make any change.
Any ideas what settings I can try to get a better accuracy with WIS?
Or is is just my luck again?
Thanks

You can add ?model=large&beam_size=5 to your wis url in your config to make the model more precise. Given you are using a 1080 ti you should still easily get sub 1 second inference but have higher accuracy. 🙂

Just to be clear: the wis url should be:  https://<IP_ADDRESS>:19000/api/willow?model=large&beam_size=5
The TTS URL I leave the same (https://<IP_ADDRESS>:19000/api/tts) ?
Will try and report back.
Nope - does not make any difference at all. Still get the wrong responses: "Fire Light" and now...wait for it...... "both sides"

It almost seems to me that it has something to do with the volume.
When I raise my voice about 40% louder - it picks up the right command. I realized that "bedsides' are more prune to errors that "bedside". So I renamed the group in HA.
But now I got it to work now with WIS.
No other changes made to the audio/mic willow settings:
PCM, DET_MODE_3CH_95, VAD_MODE_3, Mic db: 37.5dB
Any other ideas?

neah - I think it is volume related.
I got it to work now with bedsides, but saying it louder.

I give up - there is no consistency in the accuracy of these commands. It does not make sense.
My only solution is to define device-groups in HA. For "Foyer Light" and "Foyer Lamp" I created "Fire Light" and "Fire Lamp" groups, which point to the respective device. I did the same for "bedsides" - I greated groups called "Both Sides", "Bedside" and "Bed side".
I do not understand why it works perfectly with Multinet, but not with WIS.
One should expect better accuracy, right, as it does not go out on the WWW?

Multinet is a restricted model that contains only your devices turned into on/off commands, whereas WIS isn't looking at your devices and is instead simply translating your speech into text with zero context. I've noticed sometimes speaking a bit slower with some of these less common words can help it.

I have noticed this problem before, there is definitely an accuracy problem....I had HA entities that were called "smart plug" and I was having issues, I end up renaming my entities for just one word like "plug" and apparently that works a bit better.
Assuming that we cannot improve the accuracy, maybe we could have an hybrid approach in WIS when we want to turn ON/OFF devices. We could send some context (names of the devices in HA) and similar names would still trigger them....

I'm not sure exactly what's going on with your really poor (nearly random results).
A lot of this comes down to an NLP/NLU approach. The problem (as you note) with general STT is that the responses more-or-less have to be 100% accurate to match with Home Assistant. HA does a little bit of "fuzzy" matching but not nearly enough.
As mentioned, Multinet works because it has predefined commands and just returns a probability that your speech matched one of them (and it's often pretty low if you look at the console). It then sends the exact pre-defined text to Home Assistant.
With the Willow Application Server we will be adding a much more capable NLU/NLP engine that will bring significantly better "fuzzy" matching for grammar and user intent to HA command - the best blend between free-form speech input with STT and probability matching to a command. Of course this doesn't address one of the fundamental issues you seem to be having (straight up garbage STT) but it will greatly help for everything else once the issue you're having gets sorted. One of the fundamental issues here is that Whisper kind of isn't intended for very short commands - it gets the WER scores it does with longer speech segments because it has context. When you feed it a few words (single seconds of audio) it has significantly less to go on to generate a sensical response. We've been looking at other approaches for this as well.
On that note, what versions of Willow and WIS are you running? Have you tried multiple ESP BOX devices? You're experience is pretty far from typical and I'd really like to investigate this further as we prepare for our first stable release.

Kristian
Thanks for the response. YEs, I have tried this with three ESP-Boxes and
one ESP-Box=lite.
I have tried it with the latest version of the software that was
available as of last Saturday.
Are the commands I can issue to show the versions of willow and WIS?
Thanks
On Mon, Jul 3, 2023 at 8:02 AM Kristian Kielhofner @.***>
wrote:

I'm not sure exactly what's going on with your really poor (nearly random
results).
A lot of this comes down to an NLP/NLU approach. The problem (as you note)
with general STT is that the responses more-or-less have to be 100%
accurate to match with Home Assistant. HA does a little bit of "fuzzy"
matching but not nearly enough.
As mentioned, Multinet works because it has predefined commands and just
returns a probability that your speech matched one of them (and it's often
pretty low if you look at the console). It then sends the exact pre-defined
text to Home Assistant.
With the Willow Application Server we will be adding a much more capable
NLU/NLP engine that will bring significantly better "fuzzy" matching for
grammar and user intent to HA command - the best blend between free-form
speech input with STT and probability matching to a command. Of course this
doesn't address one of the fundamental issues you seem to be having
(straight up garbage STT) but it will greatly help for everything else once
the issue you're having gets sorted. One of the fundamental issues here is
that Whisper kind of isn't intended for very short commands - it gets the
WER scores it does with longer speech segments because it has context. When
you feed it a few words (single seconds of audio) it has significantly less
to go on to generate a sensical response. We've been looking at other
approaches for this as well.
On that note, what versions of Willow and WIS are you running? Have you
tried multiple ESP BOX devices? You're experience is pretty far from
typical and I'd really like to investigate this further as we prepare for
our first stable release.
—
Reply to this email directly, view it on GitHub
https://github.com/toverainc/willow/issues/219#issuecomment-1618359117,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEN5KACI4ISX3VW2QR6F2XLXOLGHTANCNFSM6AAAAAAZ24X2ME
.
You are receiving this because you authored the thread.Message ID:
@.***>

The best way (currently) to get "version" for both Willow and WIS is to execute git describe --always --dirty --tags in root of the Willow and WIS directories.

I had to rebuild the "willows" ubuntu machine I used before to flash ESPBOX (and -lite). Everything is rebuilded, configure willow and flash the ESP-BOX-lite. When I run it, I get these errors, constanly:

I redownloaded willow and went through all the steps and then flashed again - same problem.
This is my config as it was before:

Do I miss something on the new "willow/ubuntu" machine side?
Or did something change on the source (Github) side?

I missed saying that it's the same Ubuntu distro I used before.

Is WIS up and running? Are you able to reach /api/docs on the WIS instance from a browser?

That was it. When I did a docker ps - the container was up and running, but had to stop and rm it. Restarted it and I connect again.
Thanks for your help and direction.


Fix alignment of progress-bar text

When the height of a .progress/.progress-bar is changed, the text within it does not appear to vertically align properly. Adding vertical-align: middle to the  elements within the .progress-bar should fix alignment issues.

Beyond this not solving the issue for the recommended label approach for the progress component, generic selectors targeting just elements like this are discouraged. Pushing up a proper fix for this based on what we had in v3.


Background overlay missing on first load

I found this behaviour very strange. Upon first page load it totally disappear background overlay. screenshot attached see whole white background; environment chrome

Expected behaviour: there should dark gray overlay on background of swal modal is popup

@mumairali could you please make jsfiddle which shows this issue?

@limonte Thank you for quick response. Unfortunately I wouldn't able to do this on fiddle but problem which I found is here
highlighted div with styles are automatically generated display: none One thing more if I try once more time then overlay works

I can't reproduce this issue, most probably it's up to your project-related code.

I think you'r right. I am digging into my project :+1:

@mumairali Did you figure out the source of this bug? I'm also experiencing it atm.
Thanks!

Shalom @mumairali ! Can you please make jsfiddle (here's an example) with code you're using so I can reproduce this bug and fix it.


timeout <= 0 not working for deleteTopics function

Describe the bug
In the deleteTopics docs, the description for timeout field is missing. It only says it is a number. There is no clear description of whether it is the max time taken in deleting all topics or the timeout of sending a deletion request (connecting to broker and all).
To find out, how this timeout is working under the hood, I jumped into the code and found that admin.deleteTopics is calling broker.deleteTopics under the hood and it has this doc (jsdocs comments). According to this doc, if timeout <= 0 then it will trigger the topic deletion and will return but when I provide timeout value as 0 or 1, it throws REQUEST_TIMED_OUT error.
Are these comments up to date? If no, is there any way I can trigger topic deletion asynchronously?
The asynchronous part is important for some applications. In my case, I am writing a scheduler that will clean up stale topics daily (a lambda function) but I don't want to pay for lambda execution time until all topics are deleted. I just want to send an async deletion request and if something goes wrong, I will pick those topics in the next run.
To Reproduce
Please provide either a link to a:

[failing test in a KafkaJS fork]
Sorry, I am not running KafkaJS locally.
[repository with an example project reproducing the issue]
Not possible

If none of the above is possible to provide, please write down the exact steps to reproduce the behavior:

Create a Kafka client.
Create an admin using const admin = client.admin() and connect to the admin using await admin.connect().
Call await admin.deleteTopics({topics: ['an-existing-topic'], timeout: 0 or -1}).

Expected behavior
The delete request should be sent successfully and I should get an ok response from deleteTopics call.
Observed behavior
The request failed with the REQUEST_TIMED_OUT exception when passing timeout as <= 0 value.
Environment:

OS: MacOS 11.4
KafkaJS version: 1.14.0
Kafka version: 2.7.0
NodeJS: node:12.13.1

Hi @aditya81070 , the comments are correct. I think this is a bug, the protocol layer for deleteTopics is probably not taking this scenario in consideration.

@tulios Thanks for quick response. Are there any plans to fix this one in upcoming release?
If it is a easy fix , I can raise a PR. I went deep down in the code to the some Encoder thing but couldn't understand fully. Some guidance on this will be helpful.

I wasn't able to find any mention of timeout values <= 0 making topic deletion async. There's no KIP mentioning it and nothing in the Java client docs or anywhere else, so I think this is incorrect. I'm removing that comment in #1427 and updating the docs to include the default value.


Improve phone number authentication

Description of change

Alter validation of mobile_number in users table so that it

Can include white spaces, -, (, ) or +.
Excluding the above characters contains only digits and is between 8 and 15 characters long.

Apply these validations when the user submits the mobile number during the login process.

NB I've copied the accepted characters and length criteria from Notify. It's safe to be pretty generous because if Notify rejects the number the user will see an error at the next step. The important thing at our end is to apply the same validations when the user submits the mobile number as we do when we save the number to the database (we weren't doing this, which is what was causing the bug).
Story Link
https://trello.com/c/igCiPkfe/1257-some-valid-mobile-number-formats-arent-accepted
Screenshot

Decisions
I've opted not to use a gem for now, since in a sense the phone number is being validated by the log in process (if the request to notify succeeds the user gets a code etc). But we explore the gem option? phonelib looks like a good candidate. It uses google's libphonenumber which is recommended by GOV.UK Design System.

Worth re-testing with the changes merged from rubocop? 👮♂

Worth re-testing with the changes merged from rubocop? 👮♂

💯 %!


Modify a set of genes for a set of cells from real scRNA-seq data to create a synthetic dataset

Hello!
I was wondering if it would be possible to use scDesign3 to modify a set of genes for a set of cells from a real scRNAseq dataset to create a synthetic dataset? If so, how? Thanks very much!

Dear Siyao,
I did not fully get your question. I think you can select some genes and cells and only fit the model on this submatrix, and then alter the model. Please check our step-by-step mode in our vignittee.
Best regards,
Dongyuan

@SONGDONGYUAN1994 Thanks for your quick response. Sorry if I didn't describe my question clear enough for you. What I really want to do is to create a new cell type in the simulated data by modifying a set of genes from the existing real dataset. Do you think scDesign3 can do that? Thank you! Following your tutorial here https://songdongyuan1994.github.io/scDesign3/docs/articles/scDesign3-cellType-vignette.html, it seems that scDesgin3 can only take a real dataset and create a synthetic with the same cell types or varying their cell type proportions. Could scDesign create a new cell type from real data? Thank you!

Hi Siyao,
It depends on your definition of a new cell type. In https://songdongyuan1994.github.io/scDesign3/docs/articles/scDesign3-introduction-vignette.html , we describe the step-by-step mode. Eventually, you will input your own parameter matrix in function simu_new. Of course you can create new cell types and add them into the simu_new.
Best,
Dongyuan


Fix packedge requirements

Please follow the guide below

Issues submitted without this template format will be ignored.
You will be asked some questions and requested to provide some information, please read them carefully and answer completely.
Put an x into all the boxes [ ] relevant to your issue (like so [x]).
Use the Preview tab to see how your issue will actually look like.

Before submitting an issue, make sure you have:

[x] Updated to the latest version
[ x Read the README and docs
[x] Searched the bugtracker for similar issues including closed ones

Purpose of your issue?

[x] Bug report (encountered problems/errors)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections requests more details for particular types of issues, you can remove any section (the contents between the triple ---) not applicable to your issue.

For a bug report, you must include the Python version used, code that will reproduce the error, and the error log/traceback.
Paste the output of python -V here:
Code:
# Example code that will produce the error reported
from instabot import Bot

bot = Bot()
bot.login(username="user", password="pass")
followers = bot.get_user_followers(bot.user_id)

Error/Debug Log:
[GCC 4.8.2] on linux
Collecting instabot
  Downloading https://files.pythonhosted.org/packages/bd/7a/5e37bd6e68d9c464415d55e46a351319136d58839a5e9fb2e8e799890507/instabot-0.8.0.tar.gz (49kB)
Building wheels for collected packages: instabot
  Running setup.py bdist_wheel for instabot: started
  Running setup.py bdist_wheel for instabot: finished with status 'done'
  Stored in directory: /home/runner/.cache/pip/wheels/51/88/b7/1f2e94145b6bc4695e64e705cb1bcb1da08f6be70582a7e650
Successfully built instabot
Installing collected packages: instabot
Successfully installed instabot-0.8.0
You are using pip version 9.0.1, however version 19.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Repl.it: package installation success

 Traceback (most recent call last):
  File "main.py", line 4, in <module>
    from instabot import Bot
  File "/home/runner/.site-packages/instabot/__init__.py", line 1, in <module>
    from .api import API
  File "/home/runner/.site-packages/instabot/api/__init__.py", line 1, in <module>
    from .api import API
  File "/home/runner/.site-packages/instabot/api/api.py", line 19, in <module>
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

Describe your issue
In last commits you broke package requirements. Sometimes its tqdm( used for progress-bars), my collegue  got another traceback willing to installinstabot`:
>>> from instabot import Bot
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/evgeny/code/PyCourses/smm-test-5/env/lib/python3.6/site-packages/instabot/__init__.py", line 1, in <module>
    from .api import API
  File "/home/evgeny/code/PyCourses/smm-test-5/env/lib/python3.6/site-packages/instabot/api/__init__.py", line 1, in <module>
    from .api import API
  File "/home/evgeny/code/PyCourses/smm-test-5/env/lib/python3.6/site-packages/instabot/api/api.py", line 16, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/evgeny/code/PyCourses/smm-test-5/env/lib/python3.6/site-packages/instabot/__init__.py", line 1, in <module>
    from .api import API
  File "/home/evgeny/code/PyCourses/smm-test-5/env/lib/python3.6/site-packages/instabot/api/__init__.py", line 1, in <module>
    from .api import API
  File "/home/evgeny/code/PyCourses/smm-test-5/env/lib/python3.6/site-packages/instabot/api/api.py", line 18, in <module>
    import six.moves.urllib as urllib
ModuleNotFoundError: No module named 'six

>>> from instabot import Bot
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/evgeny/code/PyCourses/smm-test-5/env/lib/python3.6/site-packages/instabot/__init__.py", line 1, in <module>
    from .api import API
  File "/home/evgeny/code/PyCourses/smm-test-5/env/lib/python3.6/site-packages/instabot/api/__init__.py", line 1, in <module>
    from .api import API
  File "/home/evgeny/code/PyCourses/smm-test-5/env/lib/python3.6/site-packages/instabot/api/api.py", line 19, in <module>
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

pip install tqdm


fix(ecma/parser): throw error when function body has use strict and paramaters is not simple

Description:
throw SyntaxError: Illegal 'use strict' directive in function with non-simple parameter list,
when function body has a 'use strict' and function paramaters is not simple
Related issue (if exists):
fix #3236

I don't understand, what is span test?
btw I just found out that tsc doesn't emit this error

I meant something like https://github.com/swc-project/swc/blob/d396c32fe5139fdfa04934e3f29297ac1a8e99f7/crates/swc_ecma_parser/tests/typescript-errors/issue-1209/input.ts.stderr


UI/UX improvements + fixed typo in rate-limit verification

Fixed bug that forced it to focus on one trigger forever
Fixed typo that allowed minimum rate-limit of 640ms instead of 650ms
Changed output to use spinners
Added check to create a trigger_list.txt if not present
Ran codebase through cargo fmt

Using past-tense in your commit messages smh.


Fix science build to handle nest file names.

Previously a flat name like "science.pyz" would work without a --path
mapping if the file was in the cwd, but a name like "dist/science.pyz"
would not even if it the file was at that path relative to the cwd.
Fixes #2

@sureshjoshi you've got an invite to collaborate in this repo, but even if you don;t want to accept that, perhaps you want to review the test. The fix is exactly what you suggested with a new test that fails without the fix.


[RI-437] Backport pike fixes

This backports a couple of fixes into pike which may help resolve RI-437.
Commit: 199c2f78be47bf2b098715993d8c5813632c97a4
Remove functions source in deploy.sh
Sourcing the functions script in deploy.sh may result in failure on a
deployment where the base OS has been installed using a minimal image.
The functions script will be sourced by the install script so any and
all dependencies will be taken care of in the order in which they're
used.
Commit: 085d24b206aab3f7bb4a7c5ed74fc257769be026
add default option to the variable DEPLOY_AIO
Issue: RI-437

recheck_issue_link


Session is not cleared after config-cli run

Given Using keycloak config cli version v0.8.0
When Run the following command in the shell
Then After successful execution, There are active sessions in User -> username (admin) -> Sessions which is created by config cli.
java -jar ./keycloak-config-cli.jar \
  --keycloak.url=http://localhost:8080/auth \
  --keycloak.user=admin \
  --keycloak.password=admin \
  --import.file=realm.json

Expected behavior
After running the command, the config cli should logout the session which is used. There should be no session from admin-cli client.
Environment:

Keycloak Version: 7.0.0
keycloak-config-cli Version: v0.8.0-7.0.0
Java Version: 1.8.0_292

Hi,
thanks for reporting this. Currently there is no builtin method inside the keycloak client to terminate a sessions.
See: https://github.com/keycloak/keycloak/pull/8258

Ok.
@jkroepke Actually, we can logout the current session using refresh token. Here is a curl example
# logout
curl -s -H "Authorization: bearer $access_token" \
-d "client_id=admin-cli" \
-d "refresh_token=$refresh_token" \
"http://localhost:8080/auth/realms/master/protocol/openid-connect/logout"

This may not be possible in keycloak-admin-client. Just for your information.

If there is a way to do a request manually (using the RESTeasy java library), I could implement it.
Otherwise, I hope they implement the PR soo.


Postgresql requires version bump

What version of UAA are you running?
74.22.0
How are you deploying the UAA?
I am deploying the UAA

using a bosh release I downloaded from bosh.io

What did you expect to see? What goal are you trying to achieve with the UAA?
PostgreSQL JDBC Driver 42.2.13
What did you see instead?
PostgreSQL JDBC Driver 42.2.12
Notes:
PostgreSQL JDBC Driver versions < 42.2.13 is prone to XXE vulnerability.
https://jdbc.postgresql.org/documentation/changelog.html#version_42.2.13
https://cheatsheetseries.owasp.org/cheatsheets/XML_External_Entity_Prevention_Cheat_Sheet.html

Thanks for the report! Indeed this needs to be bumped.  👍

@shamus  re-opened it because uaa.war does not include an updated version of https://jdbc.postgresql.org/documentation/changelog.html but still version x.12 - mentioned in this ticket, therefore add the version label to the gradle definition


Chip changes appear to have broken an l10n test in a pending g3 change

Steps to Reproduce
This was found in a presubmit test for a g3 change.  More detail is available there.
The broken test was testing the value of a tooltip assigned to a Chip.  The test was passing on April 3, but after the g3 roll of Flutter to d6d874474b21b512aac03f1dcd0d3b88835cdcdd on April 6, the test is failing.  The failure seems to be comming from Chip's animation controllers, which I notice were changed substantially since the previous g3 roll (bffae2157102dd75a625153d3dc866d3de69c853 on March 29).
Logs
══╡ EXCEPTION CAUGHT BY ANIMATION LIBRARY ╞═════════════════════════════════════════════════════════
The following assertion was thrown while notifying status listeners for AnimationController:
'package:flutter/src/widgets/animated_child_switcher.dart': Failed assertion: line 107 pos 16:
'_children.contains(entry)': is not true.

Either the assertion indicates an error in the framework itself, or we should provide substantially
more information in this error message to help you determine and fix the underlying cause.
In either case, please report this assertion by filing a bug on GitHub:
  https://github.com/flutter/flutter/issues/new

When the exception was thrown, this was the stack:
#2      _AnimatedChildSwitcherState.addEntry.<anonymous closure> (package:flutter/src/widgets/animated_child_switcher.dart)
#3      _AnimationController&Animation&AnimationEagerListenerMixin&AnimationLocalListenersMixin&AnimationLocalStatusListenersMixin.notifyStatusListeners (package:flutter/src/animation/listener_helpers.dart:161:19)
#4      AnimationController._checkStatusChanged (package:flutter/src/animation/animation_controller.dart:531:7)
#5      AnimationController._animateToInternal (package:flutter/src/animation/animation_controller.dart:396:7)
#6      AnimationController.reverse (package:flutter/src/animation/animation_controller.dart:346:12)
#7      _AnimatedChildSwitcherState.addEntry (package:flutter/src/widgets/animated_child_switcher.dart:86:34)
#8      _AnimatedChildSwitcherState.build (package:flutter/src/widgets/animated_child_switcher.dart:131:7)
#9      StatefulElement.build (package:flutter/src/widgets/framework.dart:3713:27)
#10     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3625:15)
#11     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#12     StatefulElement.update (package:flutter/src/widgets/framework.dart:3782:5)
#13     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#14     _RenderChipElement._updateChild (package:flutter/src/material/chip.dart:1566:30)
#15     _RenderChipElement.update (package:flutter/src/material/chip.dart:1583:5)
#16     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#17     SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:4644:14)
#18     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#19     SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:4644:14)
#20     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#21     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#22     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#23     StatelessElement.update (package:flutter/src/widgets/framework.dart:3685:5)
#24     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#25     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#26     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#27     StatefulElement.update (package:flutter/src/widgets/framework.dart:3782:5)
#28     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#29     SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:4644:14)
#30     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#31     SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:4644:14)
#32     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#33     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#34     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#35     StatefulElement.update (package:flutter/src/widgets/framework.dart:3782:5)
#36     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#37     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#38     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#39     StatelessElement.update (package:flutter/src/widgets/framework.dart:3685:5)
#40     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#41     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#42     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#43     StatefulElement.update (package:flutter/src/widgets/framework.dart:3782:5)
#44     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#45     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#46     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#47     ProxyElement.update (package:flutter/src/widgets/framework.dart:3892:5)
#48     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#49     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#50     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#51     StatefulElement.update (package:flutter/src/widgets/framework.dart:3782:5)
#52     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#53     SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:4644:14)
#54     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#55     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#56     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#57     StatelessElement.update (package:flutter/src/widgets/framework.dart:3685:5)
#58     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#59     SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:4644:14)
#60     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#61     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#62     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#63     StatelessElement.update (package:flutter/src/widgets/framework.dart:3685:5)
#64     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#65     SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:4644:14)
#66     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#67     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#68     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#69     StatefulElement.update (package:flutter/src/widgets/framework.dart:3782:5)
#70     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#71     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#72     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#73     StatefulElement.update (package:flutter/src/widgets/framework.dart:3782:5)
#74     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#75     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#76     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#77     StatefulElement.update (package:flutter/src/widgets/framework.dart:3782:5)
#78     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#79     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#80     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#81     StatelessElement.update (package:flutter/src/widgets/framework.dart:3685:5)
#82     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#83     ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3636:16)
#84     Element.rebuild (package:flutter/src/widgets/framework.dart:3478:5)
#85     StatelessElement.update (package:flutter/src/widgets/framework.dart:3685:5)
#86     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#87     SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:4644:14)
#88     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#89     RenderObjectElement.updateChildren (package:flutter/src/widgets/framework.dart:4362:32)
#90     MultiChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:4752:17)
#91     Element.updateChild (package:flutter/src/widgets/framework.dart:2682:15)
#92     _LayoutBuilderElement._layout.<anonymous closure> (package:flutter/src/widgets/layout_builder.dart:118:18)
#93     BuildOwner.buildScope (package:flutter/src/widgets/framework.dart:2205:19)
#94     _LayoutBuilderElement._layout (package:flutter/src/widgets/layout_builder.dart:107:11)
#95     RenderObject.invokeLayoutCallback.<anonymous closure> (package:flutter/src/rendering/object.dart:1666:58)
#96     PipelineOwner._enableMutationsToDirtySubtrees (package:flutter/src/rendering/object.dart:740:15)
#97     RenderObject.invokeLayoutCallback (package:flutter/src/rendering/object.dart:1666:13)
#98     _RenderLayoutBuilder.performLayout (package:flutter/src/widgets/layout_builder.dart:205:5)
#99     RenderObject.layout (package:flutter/src/rendering/object.dart:1570:7)
#100    RenderFlex.performLayout (package:flutter/src/rendering/flex.dart:738:15)
#101    RenderObject.layout (package:flutter/src/rendering/object.dart:1570:7)
#102    RenderPositionedBox.performLayout (package:flutter/src/rendering/shifted_box.dart:381:13)
#103    RenderObject._layoutWithoutResize (package:flutter/src/rendering/object.dart:1445:7)
#104    PipelineOwner.flushLayout (package:flutter/src/rendering/object.dart:709:18)
#105    AutomatedTestWidgetsFlutterBinding.drawFrame (package:flutter_test/src/binding.dart:613:23)
#106    _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&GestureBinding&RendererBinding._handlePersistentFrameCallback (package:flutter/src/rendering/binding.dart:208:5)
#107    _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding._invokeFrameCallback (package:flutter/src/scheduler/binding.dart:990:15)
#108    _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding.handleDrawFrame (package:flutter/src/scheduler/binding.dart:930:9)
#109    AutomatedTestWidgetsFlutterBinding.pump.<anonymous closure> (package:flutter_test/src/binding.dart:578:9)
#112    TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:69:44)
#113    AutomatedTestWidgetsFlutterBinding.pump (package:flutter_test/src/binding.dart:566:27)
#114    WidgetTester.pumpWidget.<anonymous closure> (package:flutter_test/src/widget_tester.dart:195:22)
#117    TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:69:44)
#118    WidgetTester.pumpWidget (package:flutter_test/src/widget_tester.dart:192:27)
#119    _WidgetTestController.pumpWidget <SNIP>
The AnimationController notifying status listeners was:
  AnimationController#3e210(⏮ 0.000; paused)

Flutter Doctor
This was found after the roll of Flutter to d6d874474b21b512aac03f1dcd0d3b88835cdcdd in g3.

@gspencergoog Do you know anything that might cause an animation to break when changing a Chip's delete tooltip?

Well, there's currently a bug in the AnimatedChildSwitcher, for which I have a PR that is almost ready for merging: https://github.com/flutter/flutter/pull/16250
That could cause this sort of failure, but I'd have to look at the source for the test.

I believe this is fixed, since the AnimatedChildSwitcher fix in #16250 landed.


Add Manifest Factorization through .flux.yaml config files

Want to test this? Go to https://github.com/weaveworks/flux-kustomize-example and follow the instructions.
This feature (behind the flag --manifest-generation) is now available in our prereleases repo: https://hub.docker.com/r/weaveworks/flux-prerelease/tags
This PR implements Generators and Updaters as specified in Generalizing Manifest Factorization in Weave Flux.
~It is meant as a Proof of Concept and will only be merged if/when there is positive feedback.~
One major wart is that Kubernetes doesn't offer Ephemeral containers yet. It's been in the works for quite a while ( https://github.com/kubernetes/kubernetes/pull/59416 ) but until then there is no simple and (reasonably) secure way to run generators and releasers. For now, we will simply assume that the command executables must live in the flux container.

[x] Abstract out manifest file operations (done by using an adapter of Manifests: ResourceStore)
[x] Implement Generators and Updaters
[x] Add the most popular factorization tools to the Flux container image (only Kustomize for now)
[x] Create a comprehensive example (https://github.com/2opremio/flux-kustomize-demo)
[x] Address FIXMEs TODOs and performance issues
[x] Hide behind an experimental flag
[x] Add more unit tests
[x] Circulate concept through the community to gather feedback

Fixes #943 #1261 https://github.com/weaveworks/flux/issues/1811 (and #1420 to certain extend)

Some top-level (and top-down) observations on the code:

I think cluster.Manifests, which was already a bit of a fudge, could just be removed in favour of ResourceStore/updateableResourceStore. Other things need very little from Manifests so far as I can tell, and those can be described as interfaces at the point of consumption, with just the methods they need.
PolicyTranslator seems misplaced -- it's the Kubernetes implementation of manifests that uses annotations, not any abstraction.

I think cluster.Manifests, which was already a bit of a fudge, could just be removed in favour of ResourceStore/updateableResourceStore. Other things need very little from Manifests so far as I can tell, and those can be described as interfaces at the point of consumption, with just the methods they need.

I see a problem/difficulty with this.
Decoupling ResourceStore from the cluster implementation. The ResourceStore has cluster-agnostic code which I would like to keep out of the kubernetes (or cluster) packages. The cluster-specific parts are supplied by Manifests (i.e. right now, adding a new cluster would only entail providing a new Manifests implementation).

PolicyTranslator seems misplaced -- it's the Kubernetes implementation of manifests that uses annotations, not any abstraction.

That makes sense. Do you have any alternatives in mind? Ideally decoupling ResourceStore from the cluster specifics.

I was waiting for the PR to be approved to add the docs, but I guess I can adapt the design document to documentation.

Also, should I move https://github.com/2opremio/flux-kustomize-demo to the weave org? Something like https://github.com/weave/flux-kustomize-example

Post-merge checklist:

[ ] Edit the PR description to update the Latest image to the latest at https://hub.docker.com/r/weaveworks/flux-prerelease/tags
[ ] Add a comment to #1261 explicitly indicating it's been merged in master (pointing users to https://hub.docker.com/r/weaveworks/flux-prerelease/tags ), that it now requires --manifest-generation=true and encourage them to test it before it's released.
[ ] (optional) Update https://github.com/weaveworks/flux-kustomize-example so that it doesn't link to this PR and mention it's already in master (linking to https://hub.docker.com/r/weaveworks/flux-prerelease/tags )
[ ] (optional) Enable this feature in the Weave Cloud dev environment and tell the Weave Cloud team about it, since they may be interested in refactoring the dev and prod manifests (using jk or kustomize).

Thanks @squaremo !!


Preference Pane Sidebars Truncating Text

Describe the bug
A clear and concise description of what the bug is.
To Reproduce

Run latest master, built by GH Actions.
Open up the preferences

Screenshots
See left sidebar:

What version of macOS?
10.14.6
What version of Quicksilver?
Lastest Master
@skurfer @n8henrie - are you able to reproduce this on your version of macOS? If you build from master, do you also get this issue? If I build myself, I don't get the issue.
If this is just an issue for me (on 10.14) then I suggest I look into this for 2.1.0, don't have it affect our 2.0 release.

I don’t see this with the version I built locally.


microsoft 365 defender: bugfix - variable typing issue for encryption/decryption

For whatever reason, sometimes this function sends an integer instead of a string, which is required in encryption helper. This fixes this issue.
Please ensure your pull request (PR) adheres to the following guidelines:

Please refer to our contributing documentation for any questions on submitting a pull request, link: Contribution Guide

Pull Request Checklist
Please check if your PR fulfills the following requirements:

[x] Testing of all the changes has been performed (for bug fixes / features)
[x] The readme.html has been reviewed and added / updated if needed (for bug fixes / features)
[x] Use the following format for the PR description: <App Name>: <PR Type> - <PR Description>
[x] Provide release notes as part of the PR submission which describe high level points about the changes for the upcoming GA release.
[x] Verify all checks are passing.
[x] Do NOT use the next branch of the forked repo. Create separate feature branch for raising the PR.
[x] Do NOT submit updates to dependencies unless it fixes an issue.

Pull Request Type
Please check the type of change your PR introduces:

[ ] New App
[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation
[ ] Other (please describe):

Security Considerations (REQUIRED)

[x] If you are exposing any endpoints using a REST handler, please document them in the readme.html. Not Applicable/No Changes
[x] If this is a new connector or you are adding new actions Not Applicable/No Changes

[ ] Please document in the readme.html all methods (eg, OAuth) used to authenticate
with the service that the connector is integrating with.
[ ] If any actions are unable to run on SOAR Cloud, please document this in the readme.html.

[x] Are you introducing any new cryptography modules? If yes, please elaborate their purpose: Not Applicable/No Changes
[x] Are you are accessing the file system? If yes, please verify that you are only accessing paths returned through
the Vault API. Not Applicable/No Changes
[x] Are you are marking code to be ignored by Semgrep with nosemgrep? If yes, please provide justification in an additional comment next to the ignored code. Not Applicable/No Changes

Release Notes (REQUIRED)

Fixed typing error where asset ID integer type could cause errors during encryption/decryption of the state file.

What is the current behavior? (OPTIONAL)

Sometimes saving/loading the state file will cause a type error due to the asset ID being an integer and not a string.

What is the new behavior? (OPTIONAL)

The variable typing is fixed so the error no longer occurs.

Other information (OPTIONAL)

Any other information that is important to this PR such as screenshots of how the component looks before and after the change.

Pay close attention to (OPTIONAL)

Any specific code change or test case points which must be addressed/reviewed at the time of GA release.

Screenshots (if relevant)

Thanks for contributing!

Hello @supergnaw, Is this issue occurring when the app editor is used? Because we already have this platform bug reported of this issue being present when anything is run from app editor.
Analysis ticket: https://splunk.atlassian.net/browse/SOARHELP-1799
Platform bug: https://splunk.atlassian.net/browse/PSAAS-13415

Hello @supergnaw, Is this issue occurring when the app editor is used? Because we already have this platform bug reported of this issue being present when anything is run from app editor.
Analysis ticket: https://splunk.atlassian.net/browse/SOARHELP-1799
Platform bug: https://splunk.atlassian.net/browse/PSAAS-13415

Hello @supergnaw, Did you get a chance to look into this: https://github.com/splunk-soar-connectors/microsoft365defender/pull/8#issuecomment-1592410058?

Hello @supergnaw,
This issue of asset_id not being of str type is something that needs to be fixed at a platform level instead of app level because it is used in almost every app of Splunk SOAR. The app correctly fetches the asset_id and uses it as it should and so everything is passing at the container level. But the app editor functionality should be fixed as it manipulates the asset_id fetched.
Hence, we still insist on not making this a code change but a platform change and we request you to use the container action run as it will also help you to see the action_result data and everything that any user would want to check. For debugging, I would suggest you to check the spawn.log and other files on that instance and not make this code change for the same and wait for the platform fix.

Thanks for concluding this issue @supergnaw.
According to our discussion(as mentioned in the screenshot below), I am closing this PR.


Ruby: add example for connection termination detection from server handler

Using Ruby, I'm trying to implement a service that returns an infinite stream of events Watch() Stream.
Looking at the documentation I wasn't able to understand how I can detect that a client connection is closed from my handler. I need this to release resources/threads so they don't grow forever.
I attached to this issue an example code where I start a thread for each call to Watch(). I would like to be able to stop the thread after the client disconnects.
grpc_stream_thread.tar.gz

Does the current route chat example satisfy?
Otherwise one idea on this:
The first parameter to a server-side stream handler is a ruby Enumerable. Reads on this are blocking, and one possibility is that the client could send the sentinal of some sort of Enumerable (e.g. https://github.com/grpc/grpc/blob/master/examples/ruby/route_guide/route_guide_client.rb#L148) when it's done. On the server side one could basically make a blocking read on the input stream by enumerating through it. This completes once the client has sent the end of it's Enumerable


Some pops should convert slower

This was discussed in the context of African pops, but most colonial pops should probably convert slower (so that all colonies don't just become states early in Vic2).
Previous discussions
http://forum.paradoxplaza.com/forum/showthread.php?707362-The-EU4-to-V2-Converter-Project&p=17930796&viewfull=1#post17930796
http://forum.paradoxplaza.com/forum/showthread.php?707362-The-EU4-to-V2-Converter-Project&p=17934163&viewfull=1#post17934163
http://forum.paradoxplaza.com/forum/showthread.php?707362-The-EU4-to-V2-Converter-Project&p=17934346&viewfull=1#post17934346

I suspect the new mechanics making lower-class pops convert slower will achieve this. If not, we can reopen it.


(intermediate value)(...) is not a function error in code immediately following index.js from object-path

Since upgrading object-path to 0.11.8 this error started appearing.
I think it's because of the removed ; in index.js on https://github.com/mariocasciaro/object-path/commit/4f0903fd7c832d12ccbe0d9c3d7e25d985e9e884#diff-e727e4bdf3657fd1d798edcd6b099d6e092f8573cba266154583a746bba0f346L306

I am only using index.js in my project, could be an issue in other files as well.

Where that (function() { at the end is coming from? It's not from object-path right?

No, it's from another package I have. But the problem originates from object-path, because it's not properly closed with a ; (at least in index.js which I'm using)
See this issue: https://stackoverflow.com/questions/42036349/uncaught-typeerror-intermediate-value-is-not-a-function
Downgrading object path to 0.11.7 fixes the error.
The file you see has many plugins concatenated in a single app.js file, one of which is object-path's index.js

I think just adding a ; at https://github.com/mariocasciaro/object-path/blob/4f0903fd7c832d12ccbe0d9c3d7e25d985e9e884/index.js#L320 would stabilize the package.

I realized that the problem only occurs because I minify my app.js file. Object-path works when included separately and not minified: https://jsfiddle.net/hajdi_k/Lge47nwv/
I see a lot of resources on the net for issues with semicolons and minification. Also see this comment (on a completely different repo but similar issue) https://github.com/dalelotts/angular-date-time-input/issues/21#issuecomment-249773398 .
I guess I can separate all files I concatenate with ; or a new line, before minifying.. My error now occured because index.js was followed by (function() { from another package.
It would be nice to not have to do this, because this occurs only on this package (so far). Others might be experiencing this issue as well and it's a bit difficult to debug.
Closing, since I can handle it when minifying.


replace .charCodeAt() with .startsWidth()

What kind of change does this PR introduce?
lib/NormalModuleFactory.js
replace .charCodeAt() with .startsWidth()
code is more understandable
Did you add tests for your changes?
yes
Does this PR introduce a breaking change?
no
What needs to be documented once your changes are merged?
nothing

Tests are failed

charCodeAt is used for performance reasons.
Could you measure the performance difference between both versions.

https://hasty.dev/HHHum1ZQlt0XwKsvGiFZAO

Yeah, looks like charCodeAt is faster, so let's close, anyway thank you fro your PR


[SPARK-26192][MESOS][2.4] Retrieve enableFetcherCache option from submission for driver URIs

What changes were proposed in this pull request?
Retrieve enableFetcherCache option from submission conf rather than dispatcher conf. This resolves some confusing behavior where Spark drivers currently get this conf from the dispatcher, whereas Spark executors get this conf from the submission. After this change, the conf will only need to be specified once.
How was this patch tested?
With (updated) existing tests.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Thank you for making a new PR, @mwlon .

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11240/
Test PASSed.

Test build #105990 has started for PR 24750 at commit acbace0.

Test build #105990 has finished for PR 24750 at commit acbace0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105990/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11271/
Test PASSed.

Test build #106022 has started for PR 24750 at commit f8759fa.

Test build #106022 has finished for PR 24750 at commit f8759fa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106022/
Test PASSed.


Cleanup

Compiler options
Disable one line functions
Add partial flag (Issue #21)

Debugging travis issues was painful. Note, do not set the language to cpp, because that overwrites $CC and $CXX.


Investigate improving Kryo serialisation for classes from DataSketches

It should be possible to register the classes from the DataSketches library that we have serialisers for in the Kryo Registrator. Their existing serialisers should just be wrapped in Kryo serialisers. This should improve the performance of Spark jobs which use them. This may not be simple though because some of the classes, e.g. DoublesUnion, are abstract which seems to cause Kryo problems in finding the correct serialiser. The classes that implement the abstract classes (e.g. HeapDoublesUnion) are not visible outside the DataSketches library so serialisers for them cannot be provided.

Merged into develop.


Calibration OOM

The LM, BFGS, LBFGS solvers are running out of memory. I must verify why and implement our own solvers. Probably there is some initialiser that takes up too much memory. We could write our own LM quickly and verify on controlled cases. It should be easily able to do calibration for LWA on my laptop.

[x] #98

Ready to retest

So far so good. Now only compile time issue is a problem. And possibly also the predict is taking too long.

Addressed, but still some closure issues to sort out.


Service Feasibility -

For the list of attributeParameterValues, we need to set the default value based on what the service is providing.  Also need to display both the attribute name and parameter name.

@MikeMillerGIS,
We can set the default values for attributeParameterValues based on what the service is providing.
However, we feel it does not make sense to display both the attributeName and parameterName.
We investigated this sample routing app that you shared with us initially. The approach followed in the sample app is:

In the overall logic that is used in sample routing app, it never makes use of attributes' datatypes
The service info provides datatypes only for attributeNames (Height Restriction). It does not provide datatypes for parameterNames (Vehicle Height).
It displays the lookup values  (prohibited, avoid_low,...) when parameterName == Restriction Usage for all attribute parameter names
It displays a number spinner when parameterName != Restriction Usage
For multiple occurrences of same attributeName,

the parameterName 'Restriction Usage' is displayed with a dropdown of lookup values
parameterNames other than 'Restriction Usage' is displayed with a number spinner and are displayed indented to the same attributeName.
Please see the below graphic that illustrates this.

Please let us know your thoughts on this.

So the second parameter is based on selecting a value for the first it seems, So first you would need to set height restriction to "Yes" and then you can apply a height restriction value.  We should use the same logic as shown in this app.

@MikeMillerGIS That is correct. You would need to set height restriction to any one of the lookup values (Prohibited, Avoid,...), then you can apply a vehicle height value.
We see two approaches to implement this.
Approach A: The ideal and recommended approach to handle this gracefully is to derive the logic used in the sample app and then modify it to handle cases like LengthToBusinessRatio, where there are no Restriction Usage parameters. We also need to display Vehicle Height as a sub-item of Height Restriction in config UI as well as widget UI. Dont need to display both attributeName and parameterName in config UI.
Approach B: A quick fix is to show min max value boxes in config for all parameters other than Restriction Usage. We can also display both, the attributeName and parameterName in config and widget. However, this will lead to the parameters Height Restriction: Restriction Usage and Height Restriction: Vehicle Height (meters) behaving independently and not related to each other.
We will provide a detailed description of Approach A in the next comment update shortly.
Implementing Approach B will take us 4 days approx. Implementing Approach A will take us longer, about two weeks.
Please confirm. Let us know if you prefer a short call where we could walk you through the approaches.

@MikeMillerGIS Here is a description of Approach A.
We have worked out a diagram to illustrate our understanding of how attribute parameter values should work in the widget. Please have a look.

The config UI would look like this.

Considering the configuration settings in the image above, the widget UI will not display Height Restriction as well as Vehice Height, as it is defaulting to value. It would display Vehicle Weight (with a slider) as a sub-item of Weight Restriction (displayed as dropdown with default value pre-selected).
We also think that "Not Used" is not required in the list of lookup values, since the checkbox is achieving the same effect and the default value will be pre-selected in the dropdown.

I feel approach A is the correct approach.

@MikeMillerGIS, We see that there three types of parameters on restriction attributes from the documentation

Default Restriction Usage parameter (e.g., Oneway, Driving a Bus)
Parameters for modeling vehicle characteristics (e.g., Height Restriction & Vehicle Height)
Parameters on cost attributes (e.g. LengthToBusinessRatio)

We will be able to support all three types in the following manner in config UI for 'default to value' and 'allow user input' cases. (Please click to view image in full size)

Looks good.

@MikeMillerGIS
We have implemented this and will be incorporating in 6/10 release.

verified


make page fetching asynchronous

Just came across this plugin—really like the spirit of it, but was looking at the performance, and noticed that in Live mode, the requests for the pages happen serially:

The performance could be dramatically improved if the requests were performed asynchronously, and I think that conversion would be relatively straightforward. Thanks!

So...no? 😇

I'm currently working on a new release, so maybe. :-)


Merge drawBubbles and fill to drawScheme

I'm not sure if this makes sense, but it might be a clearer naming convention.


Remove redundant metadata synchronization (#1424)

Interesting, I assume this is a perf issue?

@spencergibb yeah a user reported contention under load, and I honestly can't see any evidence a thread safe map is required (per contract or per actual usage). LMK what your thoughts are on this.

I haven't either and can't guess any future issues