A positional error occurred while running “ bash run_learner.sh ”

Hello everyone, I am deploying “SERL” in the real world and I fully follow the instructions on the webpage for hardware and software deployment.
I first want to reproduce task 1: peg insertion
Here is my operating procedure, please read it to see if there are any errors. Thank you:
I followed the instructions on the webpage and opened a terminal, running: "python serl_robot_infra/robot_servers/franka_server.py --gripper_type=Robotiq --robot_ip=<IP_ADDRESS> --gripper_ip=/dev/ttyUSB0"
Follow the instructions to update the TARGETPOSE in peg_env/config.py with the measured end effector pose. Then Record 20 demo trajectories with the spacemouse. Then Edit demo_path and checkpoint_path in run_learner.sh and run-actor.sh
The above steps seem to be fine and everything is normal.
Then run two terminals separately:
“bash run_learner.sh“
“bash run_actor.sh”
The "bash run_learner. sh" node crashes during runtime, and shortly after, "bash run_actor.sh" also crashes. I can't find out what caused it.
I will paste the information images displayed on the three terminals.
The terminal running in the top left corner of the image is: "bash run_actor.sh"
The terminal running in the upper right corner of the image is "bash run_learner. sh"
The terminal running under the image is: python serl_robot_infra/robot_servers/franka_server.py --gripper_type=Robotiq --robot_ip=<IP_ADDRESS> --gripper_ip=/dev/ttyUSB0
Thank you all for your help

Of course, I can paste all the information of the "bash run_learner. sh" terminal below for your reference. Thank you:
(serl) aml@aml-NUC11PHi7:~/serl-main/examples/async_peg_insert_drq$ bash run_learner.sh
2024-07-01 16:12:52.652413: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
I0701 16:12:54.338428<PHONE_NUMBER>83456 _schedule.py:74] A polynomial schedule was set with a non-positive `transition_steps` value; this results in a constant schedule with value `init_value`.
I0701 16:12:54.338671<PHONE_NUMBER>83456 _schedule.py:74] A polynomial schedule was set with a non-positive `transition_steps` value; this results in a constant schedule with value `init_value`.
I0701 16:12:54.338775<PHONE_NUMBER>83456 _schedule.py:74] A polynomial schedule was set with a non-positive `transition_steps` value; this results in a constant schedule with value `init_value`.
/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/flax/core/lift.py:305: RuntimeWarning: kwargs are not supported in vmap, so "train" is(are) ignored
  warnings.warn(msg.format(name, ', '.join(kwargs.keys())), RuntimeWarning)
The ResNet-10 weights already exist at '/home/aml/.serl/resnet10_params.pkl'.
Loaded 5.418792M parameters from ResNet-10 pretrained on ImageNet-1K
replaced conv_init in pretrained_encoder
replaced norm_init in pretrained_encoder
replaced ResNetBlock_0 in pretrained_encoder
replaced ResNetBlock_1 in pretrained_encoder
replaced ResNetBlock_2 in pretrained_encoder
replaced ResNetBlock_3 in pretrained_encoder
entity: null
exp_descriptor: serl_dev_drq_rlpd10demos_peg_insert_random_resnet_097
experiment_id: serl_dev_drq_rlpd10demos_peg_insert_random_resnet_097_20240701_161301
group: null
project: serl_dev
tag: serl_dev_drq_rlpd10demos_peg_insert_random_resnet_097
unique_identifier: '20240701_161301'

wandb: Currently logged in as: 345915750 (zpb345915750). Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.17.2
wandb: Run data is saved locally in /tmp/tmpy80et43_/wandb/run-20240701_161304-serl_dev_drq_rlpd10demos_peg_insert_random_resnet_097_20240701_161301
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run serl_dev_drq_rlpd10demos_peg_insert_random_resnet_097_20240701_161301
wandb: ⭐ View project at https://wandb.ai/zpb345915750/serl_dev
wandb: 🚀 View run at https://wandb.ai/zpb345915750/serl_dev/runs/serl_dev_drq_rlpd10demos_peg_insert_random_resnet_097_20240701_161301
demo buffer size: 576
 starting learner loop
Filling up replay buffer: 204it [01:09,  2.95it/s]
 sent initial network to actor
learner:   0%|                                                            | 0/1000000 [00:00<?, ?it/s]/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/flax/core/lift.py:305: RuntimeWarning: kwargs are not supported in vmap, so "train" is(are) ignored
  warnings.warn(msg.format(name, ', '.join(kwargs.keys())), RuntimeWarning)
/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/flax/core/lift.py:305: RuntimeWarning: kwargs are not supported in vmap, so "train" is(are) ignored
  warnings.warn(msg.format(name, ', '.join(kwargs.keys())), RuntimeWarning)
/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/flax/core/lift.py:305: RuntimeWarning: kwargs are not supported in vmap, so "train" is(are) ignored
  warnings.warn(msg.format(name, ', '.join(kwargs.keys())), RuntimeWarning)
/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/flax/core/lift.py:305: RuntimeWarning: kwargs are not supported in vmap, so "train" is(are) ignored
  warnings.warn(msg.format(name, ', '.join(kwargs.keys())), RuntimeWarning)
/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/flax/core/lift.py:305: RuntimeWarning: kwargs are not supported in vmap, so "train" is(are) ignored
  warnings.warn(msg.format(name, ', '.join(kwargs.keys())), RuntimeWarning)
/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/flax/core/lift.py:305: RuntimeWarning: kwargs are not supported in vmap, so "train" is(are) ignored
  warnings.warn(msg.format(name, ', '.join(kwargs.keys())), RuntimeWarning)
/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/flax/core/lift.py:305: RuntimeWarning: kwargs are not supported in vmap, so "train" is(are) ignored
  warnings.warn(msg.format(name, ', '.join(kwargs.keys())), RuntimeWarning)
I0701 16:16:54.329327<PHONE_NUMBER>83456 checkpoints.py:574] Saving checkpoint at step: 0
I0701 16:16:54.329805<PHONE_NUMBER>83456 checkpoints.py:662] Using Orbax as backend to save Flax checkpoints. For potential troubleshooting see: https://flax.readthedocs.io/en/latest/guides/training_techniques/use_checkpointing.html#orbax-as-backend-troubleshooting
W0701 16:16:54.331048<PHONE_NUMBER>83456 type_handlers.py:222] SaveArgs.aggregate is deprecated, please use custom TypeHandler (https://orbax.readthedocs.io/en/latest/custom_handlers.html#typehandler) or contact Orbax team to migrate before August 1st, 2024.
I0701 16:16:54.335811<PHONE_NUMBER>83456 checkpointer.py:157] Saving checkpoint to /home/aml/serl-main/examples/async_peg_insert_drq/peg_insert_20/checkpoint_0.
learner:   0%|                                                           | 0/1000000 [02:29<?, ?it/s]
Traceback (most recent call last):
  File "/home/aml/serl-main/examples/async_peg_insert_drq/async_drq_randomized.py", line 415, in <module>
    app.run(main)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/aml/serl-main/examples/async_peg_insert_drq/async_drq_randomized.py", line 394, in main
    learner(
  File "/home/aml/serl-main/examples/async_peg_insert_drq/async_drq_randomized.py", line 313, in learner
    checkpoints.save_checkpoint(
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/flax/training/checkpoints.py", line 697, in save_checkpoint
    orbax_checkpointer.save(
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/orbax/checkpoint/checkpointer.py", line 164, in save
    raise ValueError(f'Destination {directory} already exists.')
ValueError: Destination /home/aml/serl-main/examples/async_peg_insert_drq/peg_insert_20/checkpoint_0 already exists.
wandb: - 0.033 MB of 0.033 MB uploaded
wandb: Run history:
wandb:             actor/actor_loss ▁
wandb:                actor/entropy ▁
wandb:            actor/temperature ▁
wandb:                     actor_lr ▁
wandb:           critic/critic_loss ▁
wandb:          critic/predicted_qs ▁
wandb:             critic/target_qs ▁
wandb:                    critic_lr ▁
wandb: temperature/temperature_loss ▁
wandb:               temperature_lr ▁
wandb:         timer/sample_actions ▁
wandb:   timer/sample_replay_buffer ▁
wandb:               timer/step_env ▁
wandb:                  timer/total ▁
wandb:                  timer/train ▁
wandb:          timer/train_critics ▁
wandb:
wandb: Run summary:
wandb:             actor/actor_loss 0.26329
wandb:                actor/entropy -6.81702
wandb:            actor/temperature 0.01
wandb:                     actor_lr 0.0003
wandb:           critic/critic_loss 1.82864
wandb:          critic/predicted_qs -0.2719
wandb:             critic/target_qs -1.03865
wandb:                    critic_lr 0.0003
wandb: temperature/temperature_loss -0.03989
wandb:               temperature_lr 0.0003
wandb:         timer/sample_actions 0.05981
wandb:   timer/sample_replay_buffer 0.05218
wandb:               timer/step_env 0.1329
wandb:                  timer/total 0.20114
wandb:                  timer/train 59.3275
wandb:          timer/train_critics 30.11736
wandb:
wandb: 🚀 View run serl_dev_drq_rlpd10demos_peg_insert_random_resnet_097_20240701_161301 at: https://wandb.ai/zpb345915750/serl_dev/runs/serl_dev_drq_rlpd10demos_peg_insert_random_resnet_097_20240701_161301
wandb: ⭐ View project at: https://wandb.ai/zpb345915750/serl_dev
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 1 other file(s)
wandb: Find logs at: /tmp/tmpy80et43_/wandb/run-20240701_161304-serl_dev_drq_rlpd10demos_peg_insert_random_resnet_097_20240701_161301/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information.
Exception in thread Thread-4 (run):
Traceback (most recent call last):
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/aml/agentlace-main/agentlace/zmq_wrapper/req_rep.py", line 49, in run
    res = self.impl_callback(message)
  File "/home/aml/agentlace-main/agentlace/trainer.py", line 117, in __callback_impl
    return request_callback(_type, _payload) if request_callback else {}
  File "/home/aml/serl-main/examples/async_peg_insert_drq/async_drq_randomized.py", line 237, in stats_callback
    wandb_logger.log(payload, step=update_steps)
  File "/home/aml/serl-main/serl_launcher/serl_launcher/common/wandb.py", line 94, in log
    wandb.log(data, step=step)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 449, in wrapper
    return func(self, *args, **kwargs)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 400, in wrapper_fn
    return func(self, *args, **kwargs)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 390, in wrapper
    return func(self, *args, **kwargs)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 1877, in log
    self._log(data=data, step=step, commit=commit)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 1641, in _log
    self._partial_history_callback(data, step, commit)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 1513, in _partial_history_callback
    self._backend.interface.publish_partial_history(
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/interface/interface.py", line 618, in publish_partial_history
    self._publish_partial_history(partial_history)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py", line 89, in _publish_partial_history
    self._publish(rec)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._sock_client.send_record_publish(record)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self.send_server_request(server_req)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._send_message(msg)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/home/aml/anaconda3/envs/serl/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

Hi @iu777, thanks for trying SERL. Your steps for reproducing the peg insertion task looks correct to me. I think the error is due to the save checkpoint path /home/aml/serl-main/examples/async_peg_insert_drq/peg_insert_20 already existing, and the checkpoint saving code in the learner is set to non-overwriting by default. This causes the learner to first crash, then the actor hangs as the learner node is not available to receive the data.

Just delete that ckpt and retry

Thank， @jianlanluo @charlesxu0124
After deleting the CKPT folder and the trajectory path file from the previous training, I re taught the points and 20 spatial rules, and then ran the actor and learner nodes. Currently, the machine is undergoing normal training.
But I have a question: I have observed a slow increase in the progress bar between the actor terminal and the learner terminal, and my "peg" task has been trained for half an hour without successfully inserting it once. And my Franka has a stronger impact force. May I ask which step I set incorrectly or have an issue with? How to solve it? I hope to receive a detailed answer, as I am still exploring machine learning. Thank you very much!
(The terminal on the left of the following image is the Learner node, and the output "5555" and "6666" are added by me to observe the program running. Please ignore them. Thank you.)
(The terminal in the upper right corner of the following image is: actor node)

Looking at your learner speed, it looks like you are running the model on CPU. Also make sure you have all the functions JIT'ed as in our repo. Try to reinstall the whole conda environment with jax[cuda] using the updated instructions on the main branch. It's the least finicky if you start cleaning and reinstall the whole environment from scratch rather than trying to reinstall jax by itself. FYI, we get around 4it/s on a RTX 4090.

Also make sure you read the paper & understand it,  so that you can set the reference limiting correctly

Thank you. I will recheck the environment according to your suggestion and check if the program is using CPU for training instead of GPU.
In the current training, I have encountered a problem. During the training process of my Franka robotic arm, it was originally in a well vertical posture and twisted to a singular point, causing it to be stuck in mid air and unable to perform peg insertion training.
Franka was stuck up and down in mid air for a long time, but the program was not interrupted. Is this phenomenon normal? How can I avoid or handle it?
Thank you
The distorted Franka image is below
@jianlanluo @charlesxu0124

Yes the joint singularity is a common issue with the Franka arm after repeated motion. Here's a few tips to alleviate the problem:

Before you start the run, make sure the robot is in a neutral position; especially make sure the first (bottom) joint is centered. This slows down the joint configuration change
Increase the joint1_nullspace_stiffness parameter in the serl_franka_controllers
Use the joint reset feature every N episodes

Hope this helps


Request for comment: How about supporting interface style

For resource cleanup (and also initialization), I think following interface style is an useful option.
How do you think about this idea? @jooohn
Example code
Functional style that is currently supported (copied from README.md):
Design.bind(
    'resource1',
    async () => {
        const resource1 = new Resource1();
        console.log('initializing resource 1');
        await resource1.initialize();
        return resource1;
    },
    async resource1 => {
        console.log('closing resource 1');
        await resource1.close();
    },
)

Proposal - interface style (equivalent code with the functional style):
import { initialize, close } from "typesafe-di";

class Resource1 {
   async [initialize]() {
      console.log('initializing resource 1');
  }
   async [close]() {
      console.log('closing resource 1');
   }
}

Design.bind('resource1', () => new MyResource());  // Design will check existence of `initialize`, `close` members, and invoke them if exist.

Benefits

To users who use class to define DI managed object, interface style provides simpler way to define DI managed object
We can access to private fields directly from initialize and close.

Functional style cannot access to private members of the class. Thus we need to delegate to class method from the callback function.

Please note that I am not trying to deny current functional style. Just proposing another option.
Detail of the proposal
Type of initialize and close constants
Proposal is to use unique symbol for initialize and close.
As like as Symbol.iterator, symbol named method has following benefits:

No risk of method name conflict
Does not pollute JSON.stringify output and most of other debugging utilities

As far as I know, there are no strong de facto name of the cleanup / initialization methods. Thus I propose to use unique symbol name for this typesafe-di library.

Additional proposal: If Design.bind method supports following overload, it may more useful:
// New overload of the Design.bind
bind(key: K, klass: {
  new(): {
    [initHandler]?(): Promise<void>
    [closeHandler]?(): Promise<void>
  }
})

To use it:
Design.bind('resource1', MyResource);  // Assume Design will call `new` of the `MyResource` class.

Another discussion point: current Injector design is not class friendly because Injector returns Promise but class's constructor (and also getter/setter) cannot be async.
Current functional-style has following pain point in combination with class:
class MyResource {
  constructor(private injector: Injector<...>) {
    // We cannot get injected object here. Because `injector.something` returns Promise but constructor cannot await.
  }

  async doSomething() {
     // We always have to write `(await injector.something)` to use injected objects.
     (await injector.something).doAnything();
  }
}

If we pass Injector to initialize method, it remove the pain point. This is why I came up with this interface style proposal, I remembered.
class MyResource {
  private something: Something;

  async [initialize](injector: Injector<...>) {
      this.something = await injector.something;
  }

  /* no need to be async */ doSomething() {
     // We do not need to await injected object every time!
     this.something.doAnything();
  }

  get something() {
    // We can use injected object from getter/setter also.
    // Note that getter/setter cannot be async function.
    return this.something.anything;
  }
}

Design.bindClass('resource1', MyResource);

Thanks for the proposal! I have a few questions.

We can access to private fields directly from initialize and close.

It seems same as calling resource => resource.close() as finalizer in functional style, but what's the difference?

Current functional-style has following pain point in combination with class:

Taking injector as a constructor argument looks painful. I recommend you to await injected values to be resolved before passing it to constructor.
class A {
  // take pure values
  constructor(age: number) {}
}

Design.bind('a', async (injector: Injector<{ age: number }>) => new A(await injector.age));

Anyway, adding another utility method for convenience sounds good. I personally prefer initialize-before-construct style, so how about the style like the following?
// pseudo code
bindClass<
  K extends string,
  D extends { [key: string]: any },
  V extends {
    finalize()?: Promise<void>
  }
>(key: K, resource: { initialize(injector: Injector<D>): Promise<V> })

class Sample {
  public static async initialize(injector: Injector<{ age: number }>) {
    return new Sample(await injector.age);
  }

  constructor(age: number) {
    this.age = age;
  }

  async finalize() {}
}

This is more generic than requiring class - actually it just reorganized functional style signature a bit - but it works well with class style. Plus, I guess we can use overload with this style.
What do you think about this? Does this solve your pain point?

Thank you for response!

We can access to private fields directly from initialize and close.

It seems same as calling resource => resource.close() as finalizer in functional style, but what's the difference?

directly or indirectly is the difference.
class MyResource {
   #myHiddenField: any;

   async finalize() {
     await this.#myHiddenField.close();
   }
}

const resource = new MyResource();
Design.bind('resource1', () => resource, async () => resource.finalize());  // Must write delegate function

class MyResource {
   #myHiddenField: any;

   async [finalize]() {
     await this.#myHiddenField.close();
   }
}

Design.bindClass('resource1', MyResource);

Current functional-style has following pain point in combination with class:

Taking injector as a constructor argument looks painful. I recommend you to await injected values to be resolved before passing it to constructor.

I can understand your example code. But it makes our real code redundant (in our use case) as below, even if using modern JS/ES syntax:
class MyResource {
  constructor(private foo: FooFooFoo, private bar: BarBarBar, private baz: BazBazBaz, private jane: JaneJaneJane, private doe: DoeDoeDoe) {}
}

Design.bind('a', async (injector: Injector<{ foo: FooFooFoo, bar: BarBarBar, baz: BazBazBaz, jane: JaneJaneJane, doe: DoeDoeDoe }>) => new MyResource(await injector.foo, await injector.bar, await injector.baz, await injector.jane, await injector.doe));

OR
//
// --- Code in src/something/my_resource.ts ---
//
type MyResourceDeps = { foo: FooFooFoo, bar: BarBarBar, baz: BazBazBaz, jane: JaneJaneJane, doe: DoeDoeDoe };

class MyResource {
  #deps: MyResourceDeps;  // Non-promise type, private field.

  constructor(deps: MyResourceDeps){ this.#deps = deps; }
}

//
// --- Code in src/something/index.ts ---
//
export const somethingDesign = Design.bind('a', async (injector: Injector<MyResourceDeps>) => new MyResource(
  Object.fromEntries(await ["foo", "bar", "baz", "jane", "doe"].map(async (name) => [name, await injector[name]]))
)).bind('b', async (..............) => new MyAnotherResource(
  ................
)). // and so on...

If we can pass Injector to the class method, it makes our code slightly DRY:
//
// --- Code in src/something/my_resource.ts ---
//
type MyResourceDeps = { foo: FooFooFoo, bar: BarBarBar, baz: BazBazBaz, jane: JaneJaneJane, doe: DoeDoeDoe };

class MyResource {
  #deps: MyResourceDeps;  // Non-promise type, private field.

  async [initialize](injector: Injector<MyResourceDeps>) {
      this.#deps = Object.fromEntries(await ["foo", "bar", "baz", "jane", "doe"].map(async (name) => [name, await injector[name]]));
  }

  doSomething() {
    this.#deps.foo.doAnything();
  }
}

//
// --- Code in src/something/index.ts ---
//
export const somethingDesign = Design.bindClass('a', MyResource).bindClass('b', MyAnotherResource)...

so how about the style like the following?

Seems to be good option, I think it almost solve our pain point.
What I thought in the original proposal is to reduce boilerplate code as possible.
But now I think both initialize-before-construct style and class method style requires same amount of boilerplate code..
//  initialize-before-construct style

type MyResourceDeps = { foo: FooFooFoo, bar: BarBarBar, baz: BazBazBaz, jane: JaneJaneJane, doe: DoeDoeDoe };

class MyResource {
  public static async initialize(injector: Injector<MyResourceDeps>) {
    return new MyResource(Object.fromEntries(await ["foo", "bar", "baz", "jane", "doe"].map(async (name) => [name, await injector[name]])));
  }

  // deps is non promise-type. So that safe to use from non-async methods (include getter/setter)
  constructor(private deps: MyResourceDeps) {}

  // Must write both static method and constructor everytime...
}

// class method style

type MyResourceDeps = { foo: FooFooFoo, bar: BarBarBar, baz: BazBazBaz, jane: JaneJaneJane, doe: DoeDoeDoe };

class MyResource {
  private deps: MyResourceDeps;  // Non promise-type. So that safe to use from non-async methods (include getter/setter)

  [initialize](injector: Injector<MyResourceDeps>) {
    this.deps = Object.fromEntries(await ["foo", "bar", "baz", "jane", "doe"].map(async (name) => [name, await injector[name]])));
  }
}

I realized that what I really want is to eliminate Object.fromEntries(await ["foo", "bar", "baz", "jane", "doe"].map(async (name) => [name, await injector[name]]))) boilerplate.
Do you have any good idea?

We can access to private fields directly from initialize and close.

It seems same as calling resource => resource.close() as finalizer in functional style, but what's the difference?

directly or indirectly is the difference.

I'm afraid I still don't get the point..
class MyResource {
   #myHiddenField: any;

   async finalize() {
     await this.#myHiddenField.close();
   }
}

const resource = new MyResource();
Design.bind('resource1', () => resource, async () => resource.finalize());  // Must write delegate function

class MyResource {
   #myHiddenField: any;

   async [finalize]() {
     await this.#myHiddenField.close();
   }
}

Design.bindClass('resource1', MyResource);

bindClass in the second example implicitly depends on finalize method, but we still need finalize to be public, right? Whether it touches private fields or not, it reduces boilerplates anyway. I still don't get why the bindClass style is better to access to private fields. Maybe you just wanted to reduce boiler plates?

I realized that what I really want is to eliminate Object.fromEntries(await ["foo", "bar", "baz", "jane", "doe"].map(async (name) => [name, await injector[name]]))) boilerplate.

Yeah I know your pain point.. That's one of the disadvantages to adopt Promise as its interface.
If we can use mechanisms to determine implementation from types (e.g. type classes), we can achieve both supporting asynchronous object construction and pure value injection. But unfortunately there's no such feature..
I would write utility function like the following - though it won't eliminate all of the pains..
function pick<T extends { [key: string]: any }, K extends keyof T>(
    obj: { [P in keyof T]: Promise<T[P]> },
    ...keys: K[]
): Promise<Pick<T, K>> {
  ...
}

const injector: { a: Promise<number>, b: Promise<string>, c: Promise<boolean> } = ...
const aAndB = await pick(injector, 'a', 'b');

Please let me know if you find a great solution for this issue ;)

Maybe you just wanted to reduce boiler plates?

Yes yes yes! My interest is to just reduce redundant / non-essential code.

I would write utility function like the following - though it won't eliminate all of the pains..

Yeah, pick function seems to be an option. Still we need to call pick function explicitly, but we do not have to implement pick-like function for each project if typesafe-di provides the pick utility.
To summarize, I merged your counterproposal as below example:
//
// --- In typesafe-di ---
//
bind<     // Or `bindClass` method if there are overloading problem
  K extends string,
  D extends { [key: string]: any },
  V extends {
    finalize()?: Promise<void>
  }
>(key: K, resource: { initialize(injector: Injector<D>): Promise<V> })

//
// --- In user's code ---
//
import { Injector, Design, pick } from "typesafe-di";

type MyResourceDeps = { foo: FooFooFoo, bar: BarBarBar, baz: BazBazBaz, jane: JaneJaneJane, doe: DoeDoeDoe };

// Improvement #1: We don't have to write two `initialize`
Design.bind('sample', MyResource);

class MyResource {
  public static async initialize(injector: Injector<MyResourceDeps>) {
    // Improvement #2 : We can await multiple objects with the `pick` utility function.
    return new MyResource(await pick(injector, 'foo', 'bar', 'baz', 'jane', 'doe');

    // In my understanding, static method can access to private members (such as `#something: any;`, `private something: any;`).
    // So that we can write any initialization code even if it have to read/writes private members.
  }

  constructor(private deps: MyResourceDeps) {}

  anyMethod() {
    this.deps.foo.doSomething();
  }

  async finalize() {
    // ...
  }
}

Does this example code is same as that you imagine? @jooohn
I think above code is common ground. Due to lack of runtime type information of TypeScript, I agree above style is current feasible & best option.
By the way, do you want implement this, @jooohn ? Or may I write PR ?

I have time for this work, but it would be great to welcome you as a contributor ;)

Anyway, I might find a better way.

This inject function creates another function which is perfect fit for bind's argument. TS can infer required types from class constructor, so we don't even need to extract constructor argument type as XxxDeps.
Sometimes I use typesafe-di to build a function, so I suggest implementing injectFunction too.

With these helper functions, maybe we don't need bindClass style anymore because the usage looks succinct enough (for me). The injection of MyResource class would look like the following.
// Map constructor value directly.
Design.bind('myResource', inject(MyResource, ['foo', 'bar', 'buz', 'jane', 'doe']), resource => resource.finalyze());

// Do some initialization task inside `initialize` function. (`injectFunction` needs to accept async function, but it's easy)
class MyResource {
  static async initialize(params: { pure: number }) {
    ...
  }
}
Design.bind('myResource', injectFunction(MyResource.initialize, ['foo', 'bar', 'buz', 'jane', 'doe']), resource => resource.finalyze());

And we can have MyResource completely independent from typesafe-di.
@saiya What do you think?

inject wrapper function is interesting!
But do we need to write resource => resource.finalyze() everytime? It seems to be a little redundant & easy to forget.
In the your 3rd example code, inject method seems to be initialize method aware (inject(MyResource). How about making it finalizer aware also?
Design.bind('myResource', inject(MyResource, ['foo', 'bar', 'buz', 'jane', 'doe']), resource => resource.finalyze());

↓
Design.bind('myResource', inject(MyResource, ['foo', 'bar', 'buz', 'jane', 'doe']));

// `inject` returns something like { resolvable, finalize }
// Add bind(key: K, inject: { resolvable, finalize }) overload

But do we need to write resource => resource.finalyze() everytime? It seems to be a little redundant & easy to forget.

Yes. Registering initializer and finalizer at the same time makes sense imo. I don't think it's too redundant, but are there that many resources to handle lifecycle? Maybe it's not a common case.

In the your 3rd example code, inject method seems to be initialize method aware (inject(MyResource).

No, inject doesn't aware initializer method (check the signature). It depends on the class constructor. If you just maps injector values to the class constructor like the MyResource example does, we can use this.

I think as a library, it's worth supporting inject and injectFunction since they're common pain points imo. If you need to support additional implicit dependencies, maybe you can define your own utility function for your project.
// this may not work
function resourceDesign<K extends string, V extends { finalize(): Promise<void> }, D>(key: K, Class: { initialize(deps: D): Promise<V> }) {
  return Design.bind(key, Class, instance => instance.finalize());
}

Design
  .bind('xxx', yyy)
  .merge(resourceDesign('aaa', MyResource));

Registering initializer and finalizer at the same time makes sense imo. I don't think it's too redundant, but are there that many resources to handle lifecycle? Maybe it's not a common case.

at the same time makes sense yes. But must write 2 callbacks is redundant.
In my project there are over 20 resources to handle lifecycle. Most of 20 is AWS's API related classes in my case (using so many AWS services in the project). In general, most SaaS's API client must be closed. For large project, I think it is normal usecase.

No, inject doesn't aware initializer method (check the signature). It depends on the class constructor. If you just maps injector values to the class constructor like the MyResource example does, we can use this.

Sorry, /initializer/new/.
What I want to say is: Despite inject method allows us to specify class (inject(MyResource) and DI container create instance of the class, why do we need to specify finalizer callback (instance => instance.finalize()) every time? Why DI container should not / do not want to call .finalizer method of the class instance which is created by DI container?

maybe you can define your own utility function for your project.

Yes we can define it. Why typesafe-di should not provide such utility?

Or again, it's worth supporting another method like 'bindResource' which REQUIRES 'finalize' method. it's vulnerable to let finalize method optional because it can cause resource leak when typo.

In my original proposal, I proposed Symbol named method. It has following benefits:

No risk of name conflict
No risk of typo

Also, checking missing finalizer is easy. For example, in my usecase, all Gateway (API client wrapper) object should implement finalizer. Thus iterate all Gateway instances and check existence of the [finalize] method is enough.

What I want to say is: Despite inject method allows us to specify class (inject(MyResource) and DI container create instance of the class, why do we need to specify finalizer callback (instance => instance.finalize()) every time? Why DI container should not / do not want to call .finalizer method of the class instance which is created by DI container?

There's no SHOULD nor SHOULD NOT, but it's the matter of the design decision. Because constructor is a special method, it deserves special treatment imo.

Yes we can define it. Why typesafe-di should not provide such utility?

The followings are the reasons why I'm hesitating to support this new feature.

I want to keep this library simple enough - we have finalizer mechanism already.
I prefer explicit style over implicit style.
I prefer the way that bound value is independent from typesafe-di. Having them to implement finalize will create an implicit dependency to typesafe-di - users may implement finalize function even though they actually want to name it close or shutdown.

However I know it's convenient to support class finalizer, so my proposal is to have another bind method which expects special finalizer method (By the reason 3, I prefer finalize to be a general string method rather than symbol, which makes explicit dependencies to typesafe-di).

It's explicit to use different method for different constraints (finalize implementation)
bindResource never take the 3rd argument (finalizer) of bind method.

The usage would look like the following (btw I would rename inject -> injectClass, injectFunction -> inject because class injection is more specific).
class Class {
  constructor(params: { foo: string, bar: string }) {}
  static initialize(params: { foo: string }) {}
}

Design
  .bind('class1', (injector: Injector<HasFoo & HasBar>) => new Class({ foo: await injector.foo, bar: await injector.bar })))
  .bind('class2', inject(Class.initialize, ['foo']))
  .bind('class3', injectClass(Class, ['foo', 'bar']))
  // compile error: bindResource expects finalize function
  .bindResource('class4', injectClass(Class, ['foo', 'bar']))

class Resource {
  constructor(params: { foo: string, bar: string }) {}
  static initialize(params: { foo: string }) {}
  async finalize() {}
}
Design
  // current implementation
  .bind('resource1', (injector: Injector<HasFoo & HasBar>) => new Resource({ foo: await injector.foo, bar: await injector.bar })), resource => resource.finalize())
  .bindResource('class2', inject(Resource.initialize, ['foo']))
  .bindResource('class2', injectClass(Resource, ['foo', 'bar']))

It looks we have sufficiently less boiler plates now.

I want to keep this library simple enough - we have finalizer mechanism already.
I prefer explicit style over implicit style.

Yes, I can understand there are are always trade off between...

Simplicity of the library/framework VS User's burden to write something by themselves
Explicit & obvious coding style VS User do not want to think/know about detail to keep focus on own business logic

so my proposal is to have another bind method which expects special finalizer method

It seems great common ground.
I love your bindResource + injectClass (or inject + static initialize) + finalize counterproposal. It reduces redundant code and pain to defined DI managed classes so much !

By the reason 3, I prefer finalize to be a general string method rather than symbol, which makes explicit dependencies to typesafe-di

Does explicit dependencies to typesafe-di is problem in this case?
For example, sometime I want to write cleanup method as finalize(timeoutMills: number): Promise<void> to specify timeout by caller. But, in typesafe-di, there are no arbital "timeout" support for resource cleanup. So that I implement it as finalize(): Promise<void> and add set finalizeTimeout(mills: number) in the class. I think this design example "depends" on the typesafe-di.
In general, if user uses any DI implementation, object lifecycle management "depneds" on the DI library. And also, object should be owned by only one owner in general. Thus finalize() (or similar) method should be called by only typesafe-di (if user decide to use typesafe-di). This is why I think Symbol named method is okay in this case.

It seems great common ground.
I love your bindResource + injectClass (or inject + static initialize) + finalize counterproposal. It reduces redundant code and pain to defined DI managed classes so much !

That's great! I'll work on this after some adjustment.

Does explicit dependencies to typesafe-di is problem in this case?

Not a problem, but just my preference. We can safely remove typesafe-di (I hope you won't ;)) without removing import { finalizeSymbol } from 'typesafe-di', which may reside everywhere in the codebases.
Plus, we already have finalize term in the return value of Design.resolve, so having the same method seems consistent.
What do you think?

That's great! I'll work on this after some adjustment.

Thank you very much!!
I will watch typesafe-di update, but if there are anything I can help, please tell me :-)

Not a problem, but just my preference. We can safely remove typesafe-di (I hope you won't ;)) without removing import { finalizeSymbol } from 'typesafe-di', which may reside everywhere in the codebases.

When user remove typesafe-di, user MUST take care finalize method by any other way otherwise it causes resource leak. If it is symbol named method, it is easy to list up finalize methods (search the symbol usage with IDE/editor) that is not called anymore by the removed typesafe-di library.

Plus, we already have finalize term in the return value of Design.resolve, so having the same method seems consistent.

I see. Actually Symbol naming is not strong opinion for me. So that using normal string naming is also okay for me.

(Just a chat)
Apparently typesafe-di was handling its creation (which depends on the finalizer), so there are no resources to leak when user removes typesafe-di designs ;)
IDE friendliness is a good point, but I'll go with normal method style now!

Apparently typesafe-di was also handling objects creation (which depends on the finalizer), so there are no resources to leak when user removes typesafe-di designs ;)

Resource leak may occur when user replace typesafe-di with any other library (or manually new) and the library / way does not call finalize method. For example, DI library may expect "close" as the method name and just ignore "finalize" method.

I'll go with normal method style now!

No problem. I think typesafe-di can support also Symbol method name when it is needed. So that current dicision does not restrict future capability in my understanding.

Feature added in https://github.com/m3dev/typesafe-di/pull/16


(PUP-5199) Add clear_paths feature and parameter for package provider

Allow gem paths reset if clear_paths packaage parameter is true

To make sure I understand what's going on:

A provider requires a gem
That provider is marked as unsuitable because the gem isn't available

resources depending on it are deferred

Package resource installs gem
Provider is never marked as suitable and the resources fail

Is that correct? If so, I think it's the provider suitability/deferral logic that is busted, and we should be trying to fix it there. Making the package provider clear puppet's gem settings feels super super dirty to me.
We'll investigate what's going on here and update the Jira ticket with our findings. You might also be interested in PUP-3502

Yes, the workflow is very much accurate, but I'm not sure what "provider is marked as suitable/unsuitable" practically means. I get LoadError ruby exception and catalog execution ends there.
Fully agree on the solution being ugly, but I didn't really know how to attack it in any other way.
Thank you for the PUP-3502 reference, I'll give Josh Cooper's suggestion a try and get back with the results.

Hi @i11, I'm curious about under what circumstances the LoadError occurs. Do you have two resources, one package resource that installs the gem, and another resource that relies on the gem, e.g. exec? Or do you have a custom fact, type or provider that is trying to require 'rest-client'? Could you run with --trace and include the backtrace in the ticket when the issue occurs?

Hi Josh,
The first scenario with 2 resources.
Here is a snippet from my class
 package { 'mime-types':
   ensure => '1.25.1',
   provider => gem,
 } ->
 package { 'rest-client':
  ensure   => '1.6.7',
   provider => gem,
 } ->
nexus_repository { 'hosted':
  label => 'Hosted',
  provider_type => 'maven2',
  type => 'hosted',
  policy => 'release',
  exposed => true,
  write_policy => 'allow_write_once',
  browseable => true,
  indexable => true,
  not_found_cache_ttl => 1440,
}

nexus_repository comes from https://forge.puppetlabs.com/atlassian/nexus_rest. I did modify it a bit while testing to make it autoload rest-client instead of requiring it to make catalog compile.
 So there is definitely a problem with the module. In the unmodified case, the provider does require 'config.rb'. And config.rb then tries to require 'rest-client'. The problem is that puppet will try to require rest-client anytime the provider code is required, which will even happen on the master when trying to figure out the default provider for a type!
The puppet way of handling this is to confine the provider to be suitable based on a feature. So create a feature in the module (lib/puppet/feature/restclient.rb) containing:
  Puppet.features.add(:restclient, :libs => %{rest-client})

And then confine the provider based on the feature:
confine :feature => :restclient

Each time puppet evaluates whether the rest-client resource is suitable, it will clear the gem path. Once the gem is successfully loaded, puppet's ruby process will "see" the newly loaded lib directory for the gem.
I would recommend filing an issue with the author of the module, and verifying that the above work as expected.

Hi @joshcooper, thank you very much for the suggestion!
We tried and it worked, will create pull request from module side instead.

Thanks guys for your help! I'm closing this pull request.


Parsing GC Logs can be slow

For some logs, parsing is slow.  There are a myriad of factors involved in this, including the order or rules and how they are applied.  The ZGC log attached with permission from AliBaba is an example of this.
slow_zgc_log.log

PR #295 gives about a 28% improvement, but I wouldn't consider this problem solved.


Changed : Move fse_decompress.c from common/ to decompress/

For consistency.

ah, unfortunately, it was not a mistake.
"fse_decompress.c" is needed for compression,
since it's part of the dictionary module.
So it's really "common", from a zstd perspective.
"common" regroup files required for both compression and decompression.

Oh no, my bad, I didn't realize that. :)
Could the common parts be extracted to a new "fse_common.c" or would there not be anything left?

There would be nothing left.
The entire decompression module is required.
Maybe we could rename fse_decompress.c into fse_common.c,
but we also try to keep in sync with FSE directory
and from within FSE context, it's really a decompression module, so it makes sense to keep this name.


Installation with Expo requires config

Current Behavior
https://tamagui.dev/docs/intro/installation suggest that config on TamaguiProvider is not required, but it is when using Expo.
Expected Behavior
No response
Tamagui Version
"@tamagui/config": "^1.99.1",
"tamagui": "^1.99.1"

Platform (Web, iOS, Android)
iOS (Expo GO)

Reproduction

npx create-expo-app repro
cd repro
npm i tamagui
npm run reset-project
Wrap app/_layout.tsx in TamaguiProvider imported from tamagui

You'll get an error:
TypeError: Cannot read property 'state' of undefined

This error is located at:
    in Theme (created by ThemeProvider)
    in ThemeProvider (created by TamaguiProvider)
    in Provider (created by TamaguiProvider)
    in TamaguiProvider (created by TamaguiProvider)
    in TamaguiProvider (created by RootLayout)
    in RootLayout

System Info
No response

TamaguiProvider  always requires config. It just depends whether you're using a custom one imported from your local tamagui.config.ts or from tamagui import { config } from '@tamagui/config/v3'
I've updated docs to make it more clear :)


I can't build sql queries with dynamic column sort and sort order.

Due to pagination I neet to make queries with different sort orders and even different column, but currently the @vercel/postgres clean my query, deleting my dynamic order.
// Works
await sql`SELECT * FROM metagame_cards ORDER BY occurrences DESC LIMIT ${query.pageSize} OFFSET ${query.page * query.pageSize}`;

// Doesn't work
await sql`SELECT * FROM metagame_cards ORDER BY ${columnSort} DESC LIMIT ${query.pageSize} OFFSET ${query.page * query.pageSize}`;

// Doesn't work
await sql`SELECT * FROM metagame_cards ORDER BY ${columnSort} ${sortOrder} LIMIT ${query.pageSize} OFFSET ${query.page * query.pageSize}`;

Is there a way to (at least in a dangerous way) do this?

We currently have a PR providing this functionality:
const orderbyColumn = identifiers('occurrences');
const orderbyDirection = sortOrder === 'DESC' ?  fragment`DESC` : fragment`ASC`;
await sql`SELECT * FROM metagame_cards ORDER BY ${orderByColumn} ${orderByDirection} LIMIT ${query.pageSize} OFFSET ${query.page * query.pageSize}`;

However, we don't know if that's going to be merged yet -- there's a good bit of security auditing to do and probably some discussions around API!
For now, you can use sql.query directly. It means you'll have to do some manual string-concatenation, which isn't the best, but it's the best workaround until something better like the above is merged!

I got same issues. I'm curious why is it happened? 🤔 Dynamic values is work properly in other things.

Is there any update on if/when that PR will be accepted? I just updated Next and Vercel to latest but am still not able to build ORDER BY queries dynamically.

For now, you can use sql.query directly. It means you'll have to do some manual string-concatenation, which isn't the best, but it's the best workaround until something better like the above is merged!

how does one get the sql.query approach to function properly? The following is the gist of what I'm trying to run in my code and it still seems to ignore the ORDER BY like it did when I was trying to use the template strings to build the query dynamically.
const orderBy = sortBy + ' ' + sortDirection;
  const params = [orderBy, ITEMS_PER_PAGE, offset];

  const fullQuery = `
    SELECT
      recipes.id,
      recipes.user_id,
      recipes.name,
      ingredients.name AS primary_ingredient,
      recipe_types.name AS recipe_type,
      recipes.calories_per_serving,
      recipes.servings,
      recipes.preparation_time_mins,
      recipes.cooking_time_mins,
      recipes.taste_rating,
      recipes.instructions,
      recipes.image_url,
      recipes.date
    FROM recipes
    JOIN ingredients ON recipes.primary_ingredient = ingredients.id
    JOIN recipe_types ON recipes.recipe_type = recipe_types.id
    ORDER BY $1
    LIMIT $2 OFFSET $3
  `;

  const recipeData = await sql.query(fullQuery, params);

If I hard code the ORDER BY line, it works fine, so I'm assuming that means I'm not using this sql.query correctly. (I concatenated the 'orderBy' string in it's own variable because I was getting a syntax error if I kept sortBy and sortDirection as separate params and then put 'ORDER BY $1 $2' - it didn't seem to like having two params next to each other).


Stack trace when adding first role after message

Repro:

Clear AuthMatrix state
Send request to AuthMatrix
Create New Role

Stacktrace
Traceback (most recent call last):
File "/AuthMatrix/AuthMatrix.py", line 337, in getInputRoleClick
self._db.getOrCreateRole(newRole)
File "/AuthMatrix/AuthMatrix.py", line 629, in getOrCreateRole
self.arrayOfMessages[messageIndex].addRoleByIndex(userIndex)
UnboundLocalError: local variable 'userIndex' referenced before assignment

This is a major bug that affects the results and message table checkbox UI when a user adds new roles after messages have already been added.  Yikes!
Fixed in v0.5.2, commit: 16af1bd


OpenTelemetryLayer to make use of Filtered

Feature Request
Motivation
It seems that OpenTelemetryLayer may be able to make use of Filtered to allow Registry to avoid creating spans in some cases.
For the vast majority of cases users will set their otel sampling rate to something low, but this doesn't have an effect on the tracing side of things, and seems to have quite a performance hit.
Otel is complicated, and I am very aware that I may have missed the reason as to why this is not feasible while remaining compatible with the otel spec.
Proposal
Add a new function OpenTelemetryLayer::prefiltered(self) that returns the layer wrapped via with_filter.
Move the logic for deciding if a span is of interest into the Filter. Only apply the logic for root spans.
The downside is that the tracer would have to be moved to an Arc so that it can be shared between the filter function and the layer.
I can create a PR POC if the above seems reasonable.
Alternatives
To avoid the arc, modify the FilterLayer to allow self filtering if a layer implements Filtered.

Hmm, I'm not entirely sure why this strictly needed as opposed to writing tracing_opentelemetry::layer().with_filter(filter).with_tracer(tracer).
(I'm genuinely unsure, so I'd love to get clarification!)

I'm not really explaining this well.
I'm assuming that we are on the same page regarding the intent, that the filter would take on the work that is currently inside the otel layer to call sampled_context. Calling of sampled_context would no longer be in the remit of the otel layer itself.
Then from the user point of view I'd like to be able to call:
tracing_opentelemetry::layer().with_tracer(tracer)
And have it internally sort out the details of wrapping the otel layer in a filter for performance.
However this would probably be a breaking change, so maybe some different api or prefiltered method on otel layer could be in order.

Looks like https://docs.rs/tracing-subscriber/latest/tracing_subscriber/layer/trait.Layer.html#method.on_layer can be used to add the filter so no new APIs would be needed.


Call Dlib in C++ library from Java app using JNA

I try to use my C++ library with dlib code in Java project via JNA. So...
Expected Behavior
I want to execute some Dlib C++ code(e.g. get_frontal_face_detector()) in C++ library from Java app.
Current Behavior
I get Java exception:

Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library '/home/user/QtProjects/build-dlibTest-Desktop_Qt_5_10_1_GCC_64bit-Release/libdlibTest.so': Native library (home/user/QtProjects/build-dlibTest-Desktop_Qt_5_10_1_GCC_64bit-Release/libdlibTest.so) not found in resource path ([file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar, file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/cldrdata.jar, ...
at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:303)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:427)
at com.sun.jna.Library$Handler.(Library.java:179)
at com.sun.jna.Native.load(Native.java:591)
at com.sun.jna.Native.load(Native.java:565)
at Main$CppLibrary.(Main.java:12)
at Main.main(Main.java:20)

Steps to Reproduce
Java app code:
public class Main {

    public interface CppLibrary extends Library {

        CppLibrary INSTANCE = (CppLibrary)
                Native.load("/home/user/QtProjects/build-dlibTest-Desktop_Qt_5_10_1_GCC_64bit-Release/libdlibTest.so",
                        CppLibrary.class);

        void dlibTest();
    }

    public static void main(String[] args){
        CppLibrary.INSTANCE.dlibTest();
    }
}

C++ library code:
#include <dlib/image_processing/frontal_face_detector.h>

using namespace dlib;
using namespace std;

extern "C"
{
    void dlibTest(){
        frontal_face_detector detector = get_frontal_face_detector();
        cout<<"dlib test"<<endl;
    }
}

Please note that if I delete frontal_face_detector detector = get_frontal_face_detector(); line then all works fine.

You apparently aren't linking your shard library to dlib. This isn't a dlib issue.
I will however point out that you can use CMake to build a jar file (see http://dlib.net/other.html#Java) and cmake will do the right thing and it will work.

Thanks, I successfully built library with dlib using swig. But now I can't build some code expressions. I changed example files for testing:
swig_api.h:
#ifndef EXAMPLE_SWIG_ApI_H_
#define EXAMPLE_SWIG_ApI_H_

#include <dlib/java/java_array.h>
#ifdef SWIG
%include <dlib/java/java_array.h>
#endif
// ----------------------------------------------------------------------------------------

#endif // EXAMPLE_SWIG_ApI_H_

java_array.h:
#ifndef DLIB_SWIG_JAVA_ARRAY_H_
#define DLIB_SWIG_JAVA_ARRAY_H_

#include <dlib/image_processing/frontal_face_detector.h>
#include <dlib/dnn.h>
#include <dlib/data_io.h>
#include <dlib/timer.h>
#include <dlib/misc_api.h>

namespace java
{

template <template <int,template<typename>class,int,typename> class block, int N, template<typename>class BN, typename SUBNET>
using residual = dlib::add_prev1<block<N,BN,1,dlib::tag1<SUBNET>>>;

}

#endif // DLIB_SWIG_JAVA_ARRAY_H_

Output:

./run_test.sh
mkdir: cannot create directory ‘build’: File exists
-- Using CMake version: 3.5.1
-- Compiling dlib version: 19.16.99
-- C++11 activated.
-- JAVA_HOME environment variable not set, trying to guess it...
-- Using JAVA_HOME OF /usr/lib/jvm/java-8-openjdk-amd64
-- C++11 activated.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/user/Documents/dlib-master/dlib/java/build
[ 97%] Built target dlib
[ 98%] Swig source
/home/user/Documents/dlib-master/dlib/java/../../dlib/java/java_array.h:18: Error: Syntax error in input(1).
CMakeFiles/myproject.dir/build.make:61: recipe for target 'lib/java_src/net/dlib/swigJAVA_wrap.cxx' failed
make[2]: *** [lib/java_src/net/dlib/swigJAVA_wrap.cxx] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/myproject.dir/all' failed
make[1]: *** [CMakeFiles/myproject.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

But I can get dlib and build this function in java_array.h:
std::vector<dlib::matrix<dlib::rgb_pixel>> jitter_image(
    const dlib::matrix<dlib::rgb_pixel>& img
);

I know that The 'using' keyword in template aliasing is not fully supported yet in swig, so maybe I should somehow change template expression, but what exactly should I do?
Also, using namespace dlib; occurs to Error: Nothing known about namespace 'dlib'.

Java doesn't understand C++ templates.  Write a class without templates,
something simple java can understand. Also, this is a swig/java question,
not a dlib question.  Consult the swig documentation.

Put things that upset swig into #ifndef SWIG blocks.  Or put them in a different file.  There are just limitations on what swig can parse in its main interface file.


Excluding users from CombinedAudio

General Troubleshooting

[X] I have checked for similar issues.
[ ] I have updated to the latest JDA version.
[ ] I have checked the branches or the maintainers' PRs for upcoming features/bug fixes.

Issue
Issue Type

[ ] Bug Report
[X] Feature Request

Description
The equivalent of muting a user client-side. Basically a setting to exclude specific users from the CombinedAudio received in AudioReceiveHandler.handleCombinedAudio(CombinedAudio).
Additionally, a setting to automatically exclude other bot users would be neat.

An alternative to this would be setting up a user filter each time JDA combines audio data.

Probably the simplest implementation of this would be modifying AudioReceiveHandler interface to have a getExcludedUsers should return List<User> which would be used to determine whether handleCombinedAudio and handleUserAudio received audio from a user when they user spoke.

I suggest giving AudioReceiveHandler it's own filter method like the following:
boolean filter(User user)
This would make it easier for example to automatically filter out bot users instead of having to add them to a list whenever they join the VoiceChannel.


Auvidea Carrier Support

Hi,
I am looking at supporting a new carrier for the jetson TX2 with an Auvidea Carrier Board.
They provide the firmware but I am struggling to see how to integrate it. The page for the firmware update
Below the readme of the firmware update.
My question is should I reproduce that with a recipe or is there a better way ?
Best Regards
How to install the auvidea kernel the nvidia sdkmanager
####################################################################
This guid is writen for the J120 and the TX2

1. prepare the sdk
ONLY NEEDED IF THE IF NO TX2 WAS FLASHED BEFORE
-start the sdkmanager
-select the Jetson TX2(P3310) as target Hardware
-start the installation until the sdk asks you for either use automatic or manual setup
 -> at this point you can choose to "skip" the rest of the installation and contine with step 2 of this instructions

2. copy the contents of the auvidea packet to the nvida_sdk folder
-> cp -r ~/J120_kernel/* /home/user/nvidia/nvidia_sdk/Jetpack_4.2_Linux_P3310/Linux_for_Tegra/

3. switch into the Linux_for_Tegra folder
-> /home/user/nvidia/nvidia_sdk/Jetpack_4.2_Linux_P3310/Linux_for_Tegra/

4. apply the changes
-> sudo ./apply_binaries.sh

5. start the sdkmanger and follow the normal installation process

From a quick glance and the downloads page and the instructions, they're just overlaying some files on top of the L4T/JetPack contents.  What you need to do is to dig into the source code of those changes (which it sounds like they provide).  It's likely there are some device tree modifications in the kernel source, at the very least.
For some hints, have a look at #67 - some folks did some work on supporting the ConnectTech carrier boards, and I would expect this to be similar.

Yes I have matched what they did in their PR and with the source provided by Auvidea I was able to have it running.

Hi @madisongh or others - A bit on the side here: For a legacy system, I am trying to have the ConnectTech/Orbitty carrier board supported by a Yocto build. Did any1 have success with having BSPs successfully running a CTI boards that you are aware of?
BR, Atle

You might want to mine the issues list where it's been mentioned in the past


Not working in VSCode

None of the snippets are working in js or jsx files. Any help is highly appreciate.
OS - Windows 10 Pro

What is the vscode version you are using

pls check the below issue, a solution is provided in that
https://github.com/tmarun/vscode-material-ui-snippets/issues/1


Refine the group management dialog

In add group dialog there should be more tips to inform user that the group name is optional, if it's left empty the gid will be used as group name.
Should refine the labels and messages to explicitly tell user the group is to be imported, in contrast to added.

I don't think we need a signpost to help the user understand that the name is optional. It should get filled in automatically when the Group DN is filled in. the only required field is designated by the red asterisk (*)

I will create a REST API to check the GroupDN is valid and retrieve its group name in LDAP. see
#5145


Update sawtooth_app_state.go

Missing the sign off in your commit.
Additionally, it would be good if you would provide some more context in the commit message. Please follow message format described in this blog post.
Both can be accomplished by using git commit -s --amend and force-push the changes.


Suggestion of Making a CLI for FastApi

actually there is no issue i contacted mister Sebastián to propose for him an idea about making a CLI for the framework  to create a project structure  and managing our projects so that we developers have  start point that facilitate the hard work of making an architecture to the project therefore stays flexible for  expanding the project and adding more features and  automate the boring stuff like creating CRUD from the ORM that will make the project 1000% time faster than before .
locking dependencies by creating virtual environment add README file for documentation the project etc...
i really hope some one help me making this become true because i dont know how to make a good cli but i managed to create something for tornado i called it  " cycloneCLI " for creating my own projects  then i found something called torn for tornado framework the problem is its a very very good structure and a starter project template that no one uses it it may be because the framework is starting to die but it helped me a lot to understand my needs better and work on different parts  of my project with more efficiency and good quality code so why we dont make fastapi better . and thank you
.
"med amine fouzai aka John constantine"

Is the goal just initial project generation? If so, there are quite a few cookiecutter templates available to create projects with varying levels of features. There are some official ones at https://fastapi.tiangolo.com/project-generation/ and a bunch of third party ones under https://fastapi.tiangolo.com/external-links/#projects . Cookiecutter is a pretty standard way to go for this sort of thing.
If the goal is not project generation, what sort of other features are you looking for?

On a second read, I see there are two pieces mentioned that project generation wouldn’t get you.

Locking dependencies and virtual environment- I suggest Poetry for this though there are some other options.

Generating CRUD functions from ORM models. This is an interesting idea but kind of a giant topic.  You could generate things for super simple access patterns (e.g. get by primary key) but most ORMs already have a clean way to do the simple things. You have any examples of the sort of things you’d like to generate?

mr dbanty  thank you for responding  ,what i'm looking is a start point way that all the developers using fastapi can start with for example the django-admin for creating apps and the rest-framework for REST APIS what i wont is smothing like that to enhance fastapi productivity twice the time i tried a lot of simller frameworks and i dont actully wont to see a third party module that generate my stuff i wont the core of the framework to handle this stuff thats why it called a framework
and at the same time it will be awesome to see a new fresh tech having a lot of promised feature
what i'm saying is the python devs don't have a lot of features in the frameworks side this is a first time where i see a framework that supports graphql from the start its true that it uses graphene as a thired party module but its better than nothing also this makes me thnik twice revew my dev stack .

On a second read, I see there are two pieces mentioned that project generation wouldn’t get you.

Locking dependencies and virtual environment- I suggest Poetry for this though there are some other options.
Generating CRUD functions from ORM models. This is an interesting idea but kind of a giant topic.  You could generate things for super simple access patterns (e.g. get by primary key) but most ORMs already have a clean way to do the simple things. You have any examples of the sort of things you’d like to generate?

its true i use Anaconda to handle my dependencies and virtual environment but its able to use virtualenv inside the framework to handel it and lock dependencies ,and i  agree with  you in the Generating CRUD functions from ORM models it may be  kind of a giant topic but its doable .
look at symfony(php) for example it it has doctrine for making controllers out of view and if you see this as i do believe me the amount of speed gained from this uncredible projects will end in less than 10 lines of code

Yeah, as @dbanty says, you can check the project generators, that would be the first and simplest use case. :rocket:
Also, about a CLI to generate FastAPI parts, FastAPI is designed to reduce the boilerplate and code duplication as much as possible. So, most or all of the code you write using FastAPI is actually related to your own app, not to required boilerplate, like what is the path, the HTTP operation, the request body, etc. A CLI tool to generate that would probably have to take a lot of arguments in the CLI to generate the same things in code. So, it would probably end up meaning that users need to learn a new CLI app, with a lot of CLI arguments and options, that would generate very little code, mostly just copying the data passed in those CLI arguments and options. But in that case, it's actually simpler to just write it in your code directly, instead of passing through the extra layer of abstraction with a CLI.
On the other side, for something more complex, it would depend on each different use case, for example, it would probably depend on each specific ORM, database, deployment setup, etc. So it wouldn't belong to FastAPI itself. But of course, if you need something more complex than the Cookiecutter project generators for your own projects, you can always build your own CLI. For example, with Typer: https://github.com/tiangolo/typer

thnak you mister Sebastián Ramírez for your reply ,i 'll go with your
Suggestion  and i  will create a thierd party libary that generates a fastapi project for the community using typer  for any one needs a simple and helpfull structure to begin with and i will add other features in the way  and thank you again.

Hey, @MedAmineFouzai i don't know what you have done, but just randomly saw this issue. I'm working on manage-fastapi  i would be happy to hear your thoughts and ideas :)


fix distinct count bug

fix https://github.com/meilisearch/meilisearch/issues/2152
I think the issue was that we didn't take off the excluded candidates from the initial candidates when returning the candidates with the search result.

https://github.com/meilisearch/milli/pull/490 should be merged before this PR since a new check is Required to merge the milli PRs: Specify breaking

bors merge

bors merge

bors merge-

bors mertge

bors merge


fix: handle non iso-8859-1 in the index document name

TODO: this uses deprecated function unescape which has been deprecated. We should find out alternative

🐝 PR preview in Swarm
Preview URL: https://bah5acgzaup4mtzen4xfk742ni5ukxij653ukxg3zhx5cdmkxkabl3wbxj3ea.bzz.link
Swarm Hash: a3f8c9e48de5caaff34d4768aba13eeee8ab9b793dfa21b1575002bdd8374ec8
Commit Hash: 07e3475e7f3bb8e707460e079677c5993f1664df
Commit Message: fix: handle non iso-8859-1 in the index document name


do not remove minecraftArguments for legacy override versions

discussion on discord
resolves https://github.com/PrismLauncher/PrismLauncher/issues/1580
when noapplet is used with legacyLaunch, AbstractLauncher::gameArgs is always empty (or just contains --demo) because when generating the meta, minecraftArguments are removed, which causes the there to be no params entry for it. There is some legacy code in MojangVersionFormat.cpp that might be meant to handle this but at current it is almost entirely unused. This pr simply removes the code that nullified the default arguments.
I tested this change in a couple versions, "noapplet" with a1.2.6 causes some IndexOutOfBounds exception with or without minecraftArguments set, and with this fix b1.7.3 and 1.3.2 get sessionIds and usernames passed through with noapplet, which in turn lets legacyFixes find skins, whereas without minecraftArguments set it behaves how I laid out in the original issue.

Thanks!


BUG/API: Categorical doesn't support categories with tz

Creating Categorical from DatetimeIndex with tz results in GMT.
import pandas as pd
idx = pd.date_range('2011-01-01', periods=5, freq='M', tz='US/Eastern')
idx
# DatetimeIndex(['2011-01-31 00:00:00-05:00', '2011-02-28 00:00:00-05:00',
#                '2011-03-31 00:00:00-04:00', '2011-04-30 00:00:00-04:00',
#                '2011-05-31 00:00:00-04:00'],
#               dtype='datetime64[ns]', freq='M', tz='US/Eastern')

pd.Categorical(idx)
# [2011-01-31 05:00:00, 2011-02-28 05:00:00, 2011-03-31 04:00:00, 2011-04-30 04:00:00, 2011-05-31 # 04:00:00]
# Categories (5, datetime64[ns]): [2011-01-31 05:00:00, 2011-02-28 05:00:00, 2011-03-31 04:00:00
#                                 , 2011-04-30 04:00:00, 2011-05-31 04:00:00]

@sinhrks this is only the repr that is incorrect, the actual categories are still tz aware:
In [95]: pd.Categorical(idx)
Out[95]:
[2011-01-31 05:00:00, 2011-02-28 05:00:00, 2011-03-31 04:00:00, 2011-04-30 04:00
:00, 2011-05-31 04:00:00]
Categories (5, datetime64[ns]): [2011-01-31 05:00:00, 2011-02-28 05:00:00, 2011-
03-31 04:00:00
                                , 2011-04-30 04:00:00, 2011-05-31 04:00:00]

In [96]: cat = pd.Categorical(idx)

In [97]: cat.categories
Out[97]:
DatetimeIndex(['2011-01-31 00:00:00-05:00', '2011-02-28 00:00:00-05:00',
               '2011-03-31 00:00:00-04:00', '2011-04-30 00:00:00-04:00',
               '2011-05-31 00:00:00-04:00'],
              dtype='datetime64[ns]', freq='M', tz='US/Eastern')

@jorisvandenbossche Thanks. I think I understood the issue. Also, PeriodIndex seems to show incorrect category values (or intended?).
import pandas as pd
idx = pd.period_range('2011-01-01 09:00', freq='H', periods=5)
c1 = pd.Categorical(idx)
# [2011-01-01 09:00, 2011-01-01 10:00, 2011-01-01 11:00, 2011-01-01 12:00, 2011-01-01 13:00]
# Categories (5, int64): [359409, 359410, 359411, 359412, 359413]

@sinhrks https://github.com/pydata/pandas/blob/master/pandas/core/categorical.py#L1328
should be self.categories (the .get_values()) converts to a baser form that's why tz and period are messed up.
Also pls check on the Series repr as well

Thanks. I've prepared #10718, and will look into Series with Categorical also.


Differentiate running a function with test=True or mock=True

When running a function in test mode (for example salt <target> state.highstate test=True), we get metrics reported as if the function was executed successfully/failing according to the result:
salt_expected_responses_total{function="state.highstate",state="highstate"} 1
# HELP salt_function_responses_total Total number of response per function processed
# TYPE salt_function_responses_total counter
salt_function_responses_total{function="state.highstate",state="highstate",success="false"} 1
# HELP salt_function_status Last function/state success, 0=Failed, 1=Success
# TYPE salt_function_status gauge
salt_function_status{function="state.highstate",minion="myminion.company.com",state="highstate"} 0
# HELP salt_new_job_total Total number of new job processed
# TYPE salt_new_job_total counter
salt_new_job_total{function="state.highstate",state="highstate",success="true"} 1
# HELP salt_responses_total Total number of response job processed
# TYPE salt_responses_total counter
salt_responses_total{minion="myminion.company.com",success="false"} 1

It would be benefitial to differentiate running states in test mode, mock mode or "production".

Hello @julioz,
Thanks for your insights.

we get metrics reported as if the function was executed successfully/failing according to the result

This is by design, respecting Salt "success" result.
A failure in dry-run (test=True) is usually considered as an issue. I am not familiar with the usage of mock=True.
Why do you want to exclude or differentiate them? Can you elaborate on your usecase?
Note: adding more labels to identify dry-run from mock from prod will quickly increase the metric cardinality, especially when health-minion feature is enabled.

Why do you want to exclude or differentiate them? Can you elaborate on your usecase?

While keeping track of success/failure on automatic job runs (e.g highstate) we still would like to give engineers to possibility to iterate on their Salt states while developing, while not poisoning our metric reporting, especially because we have alerts set for when failure rate is above a certain threshold.
So the use-case would be for an engineer iterating on a new state (or refactor) and potentially using test=True to dry-run states from the salt-master node.

Note: adding more labels to identify dry-run from mock from prod will quickly increase the metric cardinality, especially when health-minion feature is enabled. So I would like to perfectly understand the need to find the most appropriate implementation.

Totally understand - what I was thinking of was some sort of flag we can pass to the exporter so that it ignores dry-run executions, or alternatively expose a separate metric with potentially less granularity (i.e no minion labels, etc) counting how many dry-run executions were read from the event bus.
In other words, passing that flag would allow for us to split what's a dry-run and what's not, in terms of success/failed state executions.

Thanks for the details, crystal clear.
I did not thought about this usecase because we do dev on dev/preprod instances, so production metrics are never poisoned by development.
I'll probably go for an option to ignore dry-run and/or mock.
FYI, I am working on a revamp of the flag to simplify the configuration of the exporter.
To be transparent, I won't probably be able to work on this until beginning of June as I am on vacation :)
But I'll keep you updated as soon as I make progress on it.

hi @julioz,
I have added the option to ignore test and mock. See the dry_run_mock_exclusion branch.
It is not merged yet as I am working on a rework of the configuration system of the exporter.

Awesome, thank you very much. Looking forward to trying this out when the release is ready 👏

Hi @julioz,
The feature is included in the new v0.6.0 release! :tada:


chore: Bump operatorpkg with latest metrics

Fixes #N/A
Description
Bump github.com/awslabs/operatorpkg to latest to contain changes for operator_status_condition_transitions_total and operator_status_condition_current_status_seconds metrics
How was this change tested?
make presubmit
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>6
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 77.998%

Totals

Change from base Build<PHONE_NUMBER>1:
0.02%

Covered Lines:
8852

Relevant Lines:
11349

💛  - Coveralls

/lgtm


Fix NullPointerException in MuxBaseSDKTheoPlayer.handleAdBreakStarted when release has already been called

We noticed a crash in the SDK due to a race condition. When the release method is called before handleAdBreakStarted, player == null and an exception is thrown when accessing player.get().
We also applied this fix to the seeking method, since a similar scenario seems possible.

We appreciate your contribution! I'll release this today.
Thanks again


chore: remove redundant .editorconfig properties

Codecov Report
Base: 94.11% // Head: 94.11% // No change to project coverage :thumbsup:

Coverage data is based on head (64276ae) compared to base (b881954).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff            @@
##               main      #19   +/-   ##
=========================================
  Coverage     94.11%   94.11%
  Complexity      160      160
=========================================
  Files            16       16
  Lines           425      425
  Branches         41       41
=========================================
  Hits            400      400
  Misses           17       17
  Partials          8        8

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.