TF2.0 leaking memory when input_shape is specified in keras layer

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): there is a custom MWE code snippet included in the issue
OS Platform and Distribution: Linux Elementary Loki (Ubuntu 16.04)
TensorFlow installed from (source or binary): binary
TensorFlow version: 2.0.0-alpha0
Python version: 3.7.1

Describe the current behavior
Generating several models with no input shape specified results in normal memory occupancy:
import time

from tensorflow import keras

for _ in range(100):
    model = keras.models.Sequential()
    model.add(keras.layers.Dense(120, activation='relu'))
    model.compile(loss='binary_crossentropy',
                  optimizer=keras.optimizers.SGD())
    time.sleep(0.1)

This is the behavior expected when the input shape is specified. However, when multiple models are generated within the same program life cycle, the specification of an input shape seems to produce a memory leak.
If the input shape is specified when the models are created, they pile up in memory, without being destroyed. Also the execution time seems to increase quite a lot.
import time

from tensorflow import keras

for _ in range(100):
    model = keras.models.Sequential()
    model.add(keras.layers.Dense(120, activation='relu', input_shape=(10, 10)))
    model.compile(loss='binary_crossentropy',
                  optimizer=keras.optimizers.SGD())
    time.sleep(0.1)

Describe the expected behavior
Model generation should behave the same from a memory point of view regardless of whether the input shape is specified or not.
Code to reproduce the issue
It is enough to execute the code snippets included in the issue. In order to generate the memory occupancy, the memory_profiler package can be used. Assuming the code snippet is pasted in a file called test.py, run:
$ mprof run --include-children python test.py
$ mprof plot

Other info / logs
I generated a list of all the objects for the 2 code snippets, using Pympler.
from pympler import muppy
from pympler import summary

...

all_objects = muppy.get_objects()
occupancy = summary.summarize(all_objects)
summary.print_(occupancy)

For the models generated without input shape, here is the result:
                                                                   types |   # objects |   total size
======================================================================== | =========== | ============
                                                             <class 'str |       79278 |     14.02 MB
                                                            <class 'dict |       14468 |      6.93 MB
                                                            <class 'code |       25252 |      3.49 MB
                                                            <class 'type |        2582 |      2.57 MB
                                                            <class 'list |        8573 |    944.06 KB
                                                           <class 'tuple |       12079 |    796.21 KB
                                                             <class 'set |         732 |    466.12 KB
                                                         <class 'weakref |        4475 |    349.61 KB
                                                     <class 'abc.ABCMeta |         341 |    346.27 KB
                     <class 'tensorflow.core.framework.op_def_pb2.ArgDef |        3822 |    328.45 KB
  <class 'google.protobuf.pyext.cpp_message.GeneratedProtocolMessageType |         369 |    320.16 KB
                                                            <class 'cell |        5921 |    277.55 KB
                                                     function (__init__) |        1694 |    224.98 KB
                                                        <class 'property |        2466 |    192.66 KB
                                              <class 'wrapper_descriptor |        2353 |    183.83 KB

And here is the snapshot for the models generated with the input shape.
                                                           types |   # objects |   total size
================================================================ | =========== | ============
                                                   <class 'tuple |      296304 |     26.77 MB
                                                    <class 'dict |       71296 |     16.14 MB
                                                     <class 'str |       84828 |     14.54 MB
                                                     <class 'int |      395032 |     10.55 MB
                                                    <class 'list |       98027 |      9.89 MB
                                                    <class 'code |       25281 |      3.49 MB
                                                    <class 'type |        2587 |      2.58 MB
                                                     <class 'set |        2944 |    975.00 KB
               <class 'tensorflow.python.framework.ops.Operation |       14100 |    771.09 KB
    <class 'tensorflow.python.framework.ops.Operation._InputList |       14100 |    771.09 KB
                  <class 'tensorflow.python.framework.ops.Tensor |       14100 |    771.09 KB
  <class 'tensorflow.python.pywrap_tensorflow_internal.TF_Output |       13200 |    721.88 KB
                                            <class 'SwigPyObject |       14100 |    660.94 KB
                                 <class 'collections.OrderedDict |        1262 |    622.22 KB
                                                 <class 'weakref |        6127 |    478.67 KB

Edit: typo in keras.models
Edit: typo in keras.layers

@b3by I ran the code, but did not see any output, does this need any modifications in the code, if not please suggest me to reproduce the issue

Unless you look at your system manager or htop, the way you can see the memory leak is by plotting the memory consumption using memory_profiler. Did you execute the script by running the following?
mprof run --include-children python test.py && mprof plot

It should open up a matplotlib plot with the memory taken by the script.

This is expected behavior, as the model has to be rebuilt if no input shape is available. You can use clear_session to clear the current Keras session.

I just tried to include
tf.keras.backend.clear_session()
into the test script (both before and after the sleep), but no luck with that, the memory leak is still there. Now if this was really an expected behaviour, that would mean the bug is actually in the clear_session method. I understand your point, but I just can't wrap my mind around the idea that old model instances are not destroyed just because the new ones have to be rebuilt.

To follow up on what @karmel said:
When you make a Sequential model without input shape specified there isn't actually enough information at compile time to create the weights. (For instance in the dense layer above, there's no way to know that the kernel needs to be shape [10, 120] and it can't allocate a [???, 120] Variable.) So Sequential defers weight creation until it sees the first batch, at which point it can create variables. So the lack of memory growth is simply reflecting the fact that the example doesn't reach the point to which allocation is deferred.
On the other hand, when input shape is provided then Sequential actually can allocate weights before seeing the first batch. The reason that those weights aren't freed when the model goes out of scope is that unless otherwise specified, all tensors used by keras share the same global graph. It has the nice property of allowing mixing and matching of models (this is why you can compose models from other models, for instance), but it also means that this graph holds references to weights and therefore blocks their garbage collection even beyond the life of the python Model object.
tf.keras.backend.clear_session destroys the underlying global graph, and therefore allows variables to be freed. The reason you're seeing different behavior is that there was a bug where clear_session didn't work in 2.0, and the fix (https://github.com/tensorflow/tensorflow/commit/5956c7e9c44e23cd1a006df872ae468201fdb600) was added after the alpha cut. If you try tf-nightly-2.0-preview it should do the right thing.
I've replicated your example (which was stellar, by the way. top notch repro.) and confirmed that clear_session works as intended in the latest nightly build: https://gist.github.com/robieta/474aa37a76ba242d9f5ea2e5284c9fab
We're also actively looking into the broader issue of making keras more gc friendly, so stay tuned.

@robieta spot on. Brilliant explanation and excellent notebook, thank you very much. The shared global graph explains things, I was just unlucky to stump in a bug (fixed in nightly) that was concealed in a normal behaviour that I took for a bug. I already patched my code so that I spawn a new process every time I train a new fold, but I might try with the nightly version and see if my whole project breaks.

will this fix work in beta1?


GEM_HOME is set in my shell when tmuxinator runs

Describe the bug
warning: I'm a complete ruby noob, but I've got bitten multiple times by this.
The GEM_HOME environment variable is set in my shell when tmuxinator runs. This creates various undesirable side effects when running Ruby scripts.
To Reproduce

Create the ~/.config/tmuxinator/foobar.yml configuration file (see below)
Create a tm executable file with content somewhere in your PATH.

#!/bin/zsh
tmuxinator start foobar

Run tm in your shell.

Expected behavior
I would like tmuxinator to not set the GEM_HOME environment variable, so that it doesn't mess with any gem install. Or, alternatively, I would like my gem install not messed up when running inside tmuxinator.
Environment

tmuxinator Version: 1.1.5
tmux Version: 3.1
Ruby Version: ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin19]
OS: macOS Catalina

Additional information
Content of my tmuxinator configuration file:
# /Users/marc/.config/tmuxinator/foobar.yml

name: marc
root: ~/

# Specifies (by name or index) which window will be selected on project startup. If not set, the first window is used.
startup_window: bf:root

windows:
  - dashboard:
      layout: main-vertical
      panes:
        - wtfutil

  - tools:
      layout: d4f0,178x45,0,0{89x45,0,0,9,88x45,90,0[88x22,90,0,14,88x22,90,23,15]}
      panes:
        - org ls
        -
        -

  - general:
      layout: even-horizontal
      panes:
        -
        -

  - bf:root:
      layout: even-horizontal
      root: "~/foo/bar"
      panes:
        -
        -
  - bf:private:
      layout: e703,178x48,0,0{89x48,0,0[89x24,0,0,16,89x23,0,25,18],88x48,90,0,17}
      root: ~/foo/bar
      panes:
        - 'cd baz'
        - 'cd baz'
        - 'cd baz'

  - bf:python:
      root: ~/foo/bar

  - bf:go:
      root: ~/foo/bar

  - bf:ruby:
      root: ~/foo/bar

Hi, @marcw.
I don't believe this is tmuxinator's doing, per se, but is possibly caused by some piece of the Ruby toolchain, which might possibly be invoked when you run tmuxinator. (FWIW, I just attempted to simulate this on a MacBook running High Sierra and didn't see the issue.)
I'd still like to help you if I'm able, though. Could you do the following and confirm that $GEM_HOME is still being set unexpectedly? (This shouldn't have any negative effects, since you don't want this variable set, but be careful.)
unset GEM_HOME
tmuxinator list
echo $GEM_HOME

If the variable wasn't set, you could try starting your project to see if that makes a difference, but I can't imagine it would.
Also, you said you were running into issues with where things are being installed from within a tmux/inator session: You can control where gem install libraries and their dependencies using the -i flag.
For example, until you address the undesired behavior, you could use: gem install -i ./gem-install-test colorls within tmux/inator sessions.

Thank you very much for your help @ethagnawl.
Here is the result.
Last login: Tue Jun  2 09:36:52 on ttys001
You have mail.
~|0|$ echo $GEM_HOME

~|0|$ unset GEM_HOME
~|0|$ tmuxinator list
tmuxinator projects:
blackfire
~|0|$ echo $GEM_HOME

~|0|$

I just realized I didn't specify the content of $GEM_HOME, so here it is:
# when inside a tmux session launched by tmuxinator
$ echo $GEM_HOME
/usr/local/Cellar/tmuxinator/1.1.5/libexec

and now I understand:
$ cat `which tmuxinator`
#!/bin/bash
GEM_HOME="/usr/local/Cellar/tmuxinator/1.1.5/libexec" exec "/usr/local/Cellar/tmuxinator/1.1.5/libexec/bin/tmuxinator" "$@"

It'd seem to be a Homebrew issue. I don't use Homebrew and the "formula" is maintained by the Homebrew project. It might be worth opening an issue there, but I have no idea what the solution would look like.
I'm going to close this issue for now, but let me know if you think there's anything else worth discussing.

Many thanks for your time.


make clearer how to use a prism theme in 11ty

the original text points to a git theme repository. in my case i also reached to prism official site and enjoyed one of their default themes. at first i thought i should grab the css from the git repository mentioned be 11ty, but the default themes are not there. then i struggled to discover the cdn approach to include the css. after a while i was able to solve everything, but kept the thought that an example of how to include a prism theme could help other noobs like me 😃.
i love 11ty by the way. thank you.

my first open-source pull request merge. so happy :)

Awesome @joaomelo!


Cline doest work with o1 Preview

Which API Provider are you using?
OpenAI
Which Model are you using?
o1 Preview
What happened?

this is throught the latest update inside the Vs Code extension on my Mac
Steps to reproduce

Add open ai api key
2.Choose o1 Preview
Ask anything

Relevant API REQUEST output
No response
Additional context
No response

Fix in v2.0.15! Please give it a try and re-open if you're still running into issues


Support web sockets in async-http-client

This isn't totally mature yet, but I think it's far enough along to be worth discussing, especially while we're thinking about how to support this in blaze-client.

Sorry for not looking at this for a while. I think the API is on the right track. One thing I don't like about the Stream focused API is that it makes it a bit more awkward (IMO) to make endpoints where the input and output are dependent on one another and not in a trivial way such as mapping.
This is the PR that initially got me on the 'why not just a Task[A] and A => Task[Unit] soapbox.


Update namespace from OpenFeature -> OpenFeatureSDK

It is recommended not to share a namespace and class name to avoid import issues and code like OpenFeature.OpenFeature.Instance.*
Add test case for FeatureClient constructor that tests null provider

Codecov Report

Merging #23 (5725c07) into main (4446810) will increase coverage by 0.33%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #23      +/-   ##
==========================================
+ Coverage   99.32%   99.66%   +0.33%
==========================================
  Files          15       15
  Lines         297      297
  Branches       22       22
==========================================
+ Hits          295      296       +1
+ Partials        2        1       -1

Impacted Files
Coverage Δ

src/OpenFeature/Constant/Reason.cs
100.00% <ø> (ø)

src/OpenFeature/Error/FeatureProviderException.cs
100.00% <ø> (ø)

src/OpenFeature/Extension/EnumExtensions.cs
100.00% <ø> (ø)

...enFeature/Extension/ResolutionDetailsExtensions.cs
100.00% <ø> (ø)

src/OpenFeature/Hook.cs
100.00% <ø> (ø)

src/OpenFeature/Model/ClientMetadata.cs
100.00% <ø> (ø)

src/OpenFeature/Model/EvaluationContext.cs
100.00% <ø> (ø)

src/OpenFeature/Model/FlagEvaluationOptions.cs
100.00% <ø> (ø)

src/OpenFeature/Model/FlagEvalusationDetails.cs
100.00% <ø> (ø)

src/OpenFeature/Model/HookContext.cs
100.00% <ø> (ø)

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4446810...5725c07. Read the comment docs.


Integration with sip-ua

Hi @cloudwebrtc
What are the recommended callbacks to set as per the example app if using sip-ua to manage the call?
I've now got displayIncomingCall triggering on a push from FCM in onBackgroundMessage.
Thanks.

I mean, can you attached sip-ua/webrtc and keep the native dialler screen?

So far, I only ported the plugin. It seems that I need a test environment that can wake up the app from APNS or FCM and perform RFC8599 testing.

As soon as we get the contact headers sorted on dart sip UA in
https://github.com/cloudwebrtc/dart-sip-ua/issues/89 I can test.
I'll be working on that this week then I just need
https://github.com/cloudwebrtc/dart-sip-ua/issues/105 and I can go live on
app stores.
On Wed, 9 Sep 2020, 07:34 CloudWebRTC<EMAIL_ADDRESS>wrote:

So far, I only ported the plugin. It seems that I need a test environment
that can wake up the app from APNS or FCM and perform RFC8599 testing.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/flutter-webrtc/callkeep/issues/5#issuecomment-689338376,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABG66FKWN5MKGQISN5BWS3SE4OVRANCNFSM4RAOB3FQ
.

Hi @cloudwebrtc
So I have a working RFC8599 env. I just need to understand how to hook an Accept / Deny key press to _handleAccept
and put this into Android for handling handling a call: https://github.com/doneservices/flutter_callkeep/wiki/Native-android-code-that-handles-incoming-calls as I want to do self managed:
https://developer.android.com/guide/topics/connectivity/telecom/selfManaged
So push comes in, we call CallKeep and then present our own local notification with Actions using above example from the other call keep github repo.
My background pushes are working nicely on Android. Outbound calls are no problem.
Thanks.

So https://github.com/flutter-webrtc/dart-sip-ua/issues/89 is done.

Cool, nice work.

Cool, nice work.

For our use anyway. I wouldn't say dart-sip-ua is a 100% RFC8599 compliant User Agent, but it works well against OpenSIPS 3.1 with RFC8599 on that side. Very close and the compliance is in your own hands.
Thanks.

Got quote far with this now, but I think callkeep does its own audio and BT?
https://github.com/flutter-webrtc/callkeep/blob/0502639460b8398255727367f030ef8f0e1b2156/android/src/main/java/io/wazo/callkeep/VoiceConnection.java#L73

I/TelecomFramework( 7878): VoiceConnectionService: onAudioStateChanged TC@7_1 [AudioState isMuted: false, route: SPEAKER, supportedRouteMask: SPEAKER, activeBluetoothDevice: [null], supportedBluetoothDevices: []]: (NCSSF.aC->ICFG.sF->ICFG.sF->ICFG.sF->CARSM.pM_UPDATE_SYSTEM_AUDIO_ROUTE)->CS.cASC->H.CS.cASC(cast)@E-CFs
I/TelecomFramework( 7878): VoiceConnectionService: onAudioStateChanged TC@7_1 [AudioState isMuted: false, route: SPEAKER, supportedRouteMask: SPEAKER, activeBluetoothDevice: [null], supportedBluetoothDevices: []]: (NCSSF.aC->ICFG.sF->ICFG.sF->ICFG.sF->CAMSM.pM_2002->CARSM.pM_SWITCH_FOCUS)->CS.cASC->H.CS.cASC(cast)@E-CFs
I/TelecomFramework( 7878): VoiceConnectionService: answer TC@7_1: (ICA.aC->CSFM.rF)->CS.an->H.CS.an(cgad/cast)@E-CIM
D/RNCK:VoiceConnection( 7878): onAnswer called
D/RNCK:VoiceConnection( 7878): onAnswer executed

...

D/RTCAudioManager( 7878): --- updateAudioDeviceState: wired headset=false, BT state=HEADSET_UNAVAILABLE
D/RTCAudioManager( 7878): Device status: available=[], selected=NONE, user selected=NONE
D/RTCAudioManager( 7878): setAudioDeviceInternal(device=SPEAKER_PHONE)
D/RTCAudioManager( 7878): New device status: available=[SPEAKER_PHONE, EARPIECE], selected=SPEAKER_PHONE
D/FlutterWebRTCPlugin( 7878): onAudioManagerDevicesChanged: [SPEAKER_PHONE, EARPIECE], selected: SPEAKER_PHONE
D/RTCAudioManager( 7878): --- updateAudioDeviceState done
D/RTCAudioManager( 7878): AudioManager started
I/flutter ( 7878): [2020-10-05 16:05:15.447] Level.debug rtc_session.dart:1572 ::: emit "peerconnection"
I/FlutterWebRTCPlugin( 7878): getUserMedia(audio): mandatory: [], optional: [googNoiseSuppression: true, googEchoCancellation: true, echoCancellation: true, googEchoCancellation2: true, googDAEchoCancellation: true]
D/FlutterWebRTCPlugin( 7878): MediaStream id: 25425f5d-e23b-4c88-8f06-f9213d7ce135
D/FlutterWebRTCPlugin( 7878): addStreamcom.cloudwebrtc.webrtc.utils.AnyThreadResult@7872a1a
I/flutter ( 7878): [2020-10-05 16:05:15.465] Level.debug rtc_session.dart:625 ::: emit "sdp"
D/FlutterWebRTCPlugin( 7878): peerConnectionSetRemoteDescription() start
D/FlutterWebRTCPlugin( 7878): onAddTrack
D/FlutterWebRTCPlugin( 7878): peerConnectionSetRemoteDescription() end

so I do this:
class CallController extends GetxController implements SipUaHelperListener {
  final logger = Logger(printer: PrettyPrinter(printTime: true));
  final _helper = Get.find<SipHelperService>().helper;
  Call _call;
  CallState _callState;

  Call get call => this._call;
  CallState get callState => this._callState;
  SIPUAHelper get helper => this._helper;
  FlutterCallkeep get callKeep => this._callKeep;

...

  @override
  void callStateChanged(Call call, CallState callState) {
    logger.d(callState.state);
    // TODO Check outbound calls from DialpadScreen()
    //if (callState.state == CallStateEnum.CALL_INITIATION) {
    this._call = call;
    this._callState = callState;
    //}
  }

...

  void _callKeepCallBacks() {
    _callKeep.on(CallKeepPerformAnswerCallAction(), answerCall);
    _callKeep.on(CallKeepDidPerformDTMFAction(), didPerformDTMFAction);
    _callKeep.on(
        CallKeepDidReceiveStartCallAction(), didReceiveStartCallAction);
    _callKeep.on(CallKeepDidToggleHoldAction(), didToggleHoldCallAction);
    _callKeep.on(
        CallKeepDidPerformSetMutedCallAction(), didPerformSetMutedCallAction);
    _callKeep.on(CallKeepPerformEndCallAction(), endCall);
  }

  Future<void> answerCall(CallKeepPerformAnswerCallAction event) async {
    final String callUUID = event.callUUID;
    logger.d('[answerCall] $callUUID');
    _callKeep.setCurrentCallActive(callUUID);
    call.answer(helper.buildCallOptions());
  }

Got quote far with this now, but I think callkeep does its own audio and BT?
https://github.com/flutter-webrtc/callkeep/blob/0502639460b8398255727367f030ef8f0e1b2156/android/src/main/java/io/wazo/callkeep/VoiceConnection.java#L73

I/TelecomFramework( 7878): VoiceConnectionService: onAudioStateChanged TC@7_1 [AudioState isMuted: false, route: SPEAKER, supportedRouteMask: SPEAKER, activeBluetoothDevice: [null], supportedBluetoothDevices: []]: (NCSSF.aC->ICFG.sF->ICFG.sF->ICFG.sF->CARSM.pM_UPDATE_SYSTEM_AUDIO_ROUTE)->CS.cASC->H.CS.cASC(cast)@E-CFs
I/TelecomFramework( 7878): VoiceConnectionService: onAudioStateChanged TC@7_1 [AudioState isMuted: false, route: SPEAKER, supportedRouteMask: SPEAKER, activeBluetoothDevice: [null], supportedBluetoothDevices: []]: (NCSSF.aC->ICFG.sF->ICFG.sF->ICFG.sF->CAMSM.pM_2002->CARSM.pM_SWITCH_FOCUS)->CS.cASC->H.CS.cASC(cast)@E-CFs
I/TelecomFramework( 7878): VoiceConnectionService: answer TC@7_1: (ICA.aC->CSFM.rF)->CS.an->H.CS.an(cgad/cast)@E-CIM
D/RNCK:VoiceConnection( 7878): onAnswer called
D/RNCK:VoiceConnection( 7878): onAnswer executed

...

D/RTCAudioManager( 7878): --- updateAudioDeviceState: wired headset=false, BT state=HEADSET_UNAVAILABLE
D/RTCAudioManager( 7878): Device status: available=[], selected=NONE, user selected=NONE
D/RTCAudioManager( 7878): setAudioDeviceInternal(device=SPEAKER_PHONE)
D/RTCAudioManager( 7878): New device status: available=[SPEAKER_PHONE, EARPIECE], selected=SPEAKER_PHONE
D/FlutterWebRTCPlugin( 7878): onAudioManagerDevicesChanged: [SPEAKER_PHONE, EARPIECE], selected: SPEAKER_PHONE
D/RTCAudioManager( 7878): --- updateAudioDeviceState done
D/RTCAudioManager( 7878): AudioManager started
I/flutter ( 7878): [2020-10-05 16:05:15.447] Level.debug rtc_session.dart:1572 ::: emit "peerconnection"
I/FlutterWebRTCPlugin( 7878): getUserMedia(audio): mandatory: [], optional: [googNoiseSuppression: true, googEchoCancellation: true, echoCancellation: true, googEchoCancellation2: true, googDAEchoCancellation: true]
D/FlutterWebRTCPlugin( 7878): MediaStream id: 25425f5d-e23b-4c88-8f06-f9213d7ce135
D/FlutterWebRTCPlugin( 7878): addStreamcom.cloudwebrtc.webrtc.utils.AnyThreadResult@7872a1a
I/flutter ( 7878): [2020-10-05 16:05:15.465] Level.debug rtc_session.dart:625 ::: emit "sdp"
D/FlutterWebRTCPlugin( 7878): peerConnectionSetRemoteDescription() start
D/FlutterWebRTCPlugin( 7878): onAddTrack
D/FlutterWebRTCPlugin( 7878): peerConnectionSetRemoteDescription() end

so I do this:
class CallController extends GetxController implements SipUaHelperListener {
  final logger = Logger(printer: PrettyPrinter(printTime: true));
  final _helper = Get.find<SipHelperService>().helper;
  Call _call;
  CallState _callState;

  Call get call => this._call;
  CallState get callState => this._callState;
  SIPUAHelper get helper => this._helper;
  FlutterCallkeep get callKeep => this._callKeep;

...

  @override
  void callStateChanged(Call call, CallState callState) {
    logger.d(callState.state);
    // TODO Check outbound calls from DialpadScreen()
    //if (callState.state == CallStateEnum.CALL_INITIATION) {
    this._call = call;
    this._callState = callState;
    //}
  }

...

  void _callKeepCallBacks() {
    _callKeep.on(CallKeepPerformAnswerCallAction(), answerCall);
    _callKeep.on(CallKeepDidPerformDTMFAction(), didPerformDTMFAction);
    _callKeep.on(
        CallKeepDidReceiveStartCallAction(), didReceiveStartCallAction);
    _callKeep.on(CallKeepDidToggleHoldAction(), didToggleHoldCallAction);
    _callKeep.on(
        CallKeepDidPerformSetMutedCallAction(), didPerformSetMutedCallAction);
    _callKeep.on(CallKeepPerformEndCallAction(), endCall);
  }

  Future<void> answerCall(CallKeepPerformAnswerCallAction event) async {
    final String callUUID = event.callUUID;
    logger.d('[answerCall] $callUUID');
    _callKeep.setCurrentCallActive(callUUID);
    _call.answer(helper.buildCallOptions());
  }

How do you then listen to the CallKeep events because they are not firing off

Hello @georgenhari @ghenry
how you integrated callkeep and sip-ua library together
can you share some code  call connection related
thank you in advance

Hello @georgenhari @ghenry how you integrated callkeep and sip-ua library together can you share some code call connection related thank you in advance

bro any solution?

Hello @georgenhari @ghenry how you integrated callkeep and sip-ua library together can you share some code call connection related thank you in advance

bro any solution?

No,bro...i stopped this  task


Content Update Requested

I am a Senior Manager with Support Services at CSC Corptax. I would like to request a couple of updates to this article. https://docs.microsoft.com/en-us/azure/active-directory/saas-apps/corptax-tutorial
The official spelling of Corptax is Corptax not CorpTax. Please correct.
The link to work with the Corptax Support Team should not direct to an email as that person may not always be with Corptax, instead it should direct them to https://connect.corptax.com the Customer Portal for our software. Thanks!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 37c0311b-2d85-c5d5-4fd6-ea179a429ee2
Version Independent ID: 40654940-8392-bc02-3ad8-2fd6f63b7364
Content: Tutorial: Azure Active Directory integration with CorpTax
Content Source: articles/active-directory/saas-apps/corptax-tutorial.md
Service: azure-active-directory
GitHub Login: @jeevansd
Microsoft Alias: jeedes

@gtidman Thank you for bringing this to our attention. We are investigating this and will update you shortly.

@gtidman
Thank you for the updates.
@jeevansd
Jeevan, The "CorpTax" wording would need to be changed. And the contact under the section - Create CorpTax test user to be changed as the user requested ?

@chetansriv To fix this issue.

@neeleshray This is update as requested. Please review

#please-close


Add struct mirrors to use external types

This PR adds an attribute marker (called mirror) to use external structs/enums. Since frb needs to know the fields in a struct, external types need to be redefined in the crate, with the marker. Once frb has read the placeholder types to generate the glue code, they are not used (they are really just used to know the "model" of an external type), so they should not even make it to runtime. Despite this minor inconvenience, it still makes the experience of dealing with external types way better.
Usage is demonstrated in the pure_dart example.
Mirrored structs/enums need to be wrapped in a newtype (invisible to users) to work around Rust's orphan rule. That's what "wrapper" refers to in the codegen.
Enum struct variants are not supported, I may work on adding them in a future PR.

Good job! Just a question: What happens if the user accidentally write down the wrong definition, i.e. a definition different from the actual mirrored type?
For example, for the external type:
#[derive(Debug, Clone)]
pub struct ApplicationSettings {
    pub name: String,
    pub version: String,
    pub mode: ApplicationMode,
}

What if we have the following problems, e.g.:
#[frb(mirror(ApplicationSettings))]
pub struct _ApplicationSettings {
    pub name: String,
    pub version: i32, // 1. wrong type? (can be even more sophisticated)
    // pub mode: ApplicationMode, // 2. missing field
    pub a: i32, // 3. extra field
}

This can happen frequently, when, for example, the external library upgrades and add/remove/change a field.

Can we guarantee that this is a compile-time error? If so then we are safe.

Can we guarantee that this is a compile-time error? If so then we are safe.

@all-contributors bot please add @Unoqwy for code

What happens if the user accidentally write down the wrong definition, i.e. a definition different from the actual mirrored type?
Hm I think this should be considered undefined behavior because it may act differently depending on the types.

A quick search reveals https://docs.rs/static_assertions/latest/static_assertions/macro.assert_type_eq_all.html . Can we use that to assert each and every field?
Well, this would assert static types, it would still require to figure out how to get a field's type at compile time.

So this sounds quite challenging given we'd need to get the type of fields at compile time but if we can do that then there is no need to have mirror in the first place since we could infer the field types.

Seems not that difficult. We do not need to "get the type of fields at compile time". We only need to generate some assertion code for that.
I get an idea: What about generate the following code in the generated_bridge.rs. Notice we generate it inside flutter_rust_bridge_codegen, so it is quite easy to do following the existing code.
const _: () = {
  // this function is NEVER called, and we NEVER need to really construct a `ApplicationSettings` object. But it is sufficient for static checker!
  fn dummy(a: ApplicationSettings) {
    let _: String = a.name; // we ensure a.name field is a string!
    let _: String = a.version;
    let _: ApplicationMode = a.mode;
  }
}

Yeah, that would indeed work.
I was just testing something similar:
const _: fn() = || {
        fn field_0() -> i32 {
            None::<Test>.unwrap().name
        }
};```
Your way of doing it is cleaner though, let's go with it.

Ok so this is working great. By using the type name as var the errors are actually meaningful. (e.g. https://imgur.com/IVaxOCo)
However, I was trying this out in an actual scenario but nested structs are not a thing apparently? The examples seem to all use Vec<> when showcasing struct nesting, is there any technical reason behind it or could it be implemented?

However, I was trying this out in an actual scenario but nested structs are not a thing apparently? The examples seem to all use Vec<> when showcasing struct nesting, is there any technical reason behind it or could it be implemented?

struct A {
  b: Box<B>
}

need a Box and it should work.
maybe we should add it to doc?

Oh I did not realize this pr is ready for review (again)... Please ping me if I do not reply after one or two hour!

LTGM

Oops a problem: https://github.com/fzyzcjy/flutter_rust_bridge/issues/357

doc added


Sync validation of array?

I've been digging through the example app and docs but I can't see an example of how to validate an array of objects?
class Item {
  name: string; // Needs to be required
  description: string;
}

class Template {
  name: string;
  items: Item[];
}

this is 'stripped out' snippet from our app, where we are validating invoice.payments elements (latter example from MrWolfZ answer):
export interface State {
    form: FormGroupState<Invoice>;
    dateTimeSettings: DateTimeSettings;
}

export interface Invoice extends Entity {
    payments: Payment[];
}

export interface Payment extends Entity {
    invoiceId: number;
    description: string | null;
    amount: number;
}

const FORM_NAME = 'INVOICE_DETAIL_FORM';

const initialForm = createForm(newInvoice(defaultDateTimeSettings), FORM_NAME);

export const initialState: State = {
    form: initialForm,
    dateTimeSettings: defaultDateTimeSettings,
};

const formReducerFn = (dateTimeSettings: DateTimeSettings) =>
    createFormGroupReducerWithUpdate<Invoice>({
        payments: updateArray<Payment>(
            updateGroup<Payment>({
                description: validate(customValidations.required),
                amount: validate([
                    customValidations.required,
                    customValidations.double,
                ]),
            }),
        ),
    });

Pretty much what @nihique said, validate the items in the array but also that there's at least 1 which I managed to figure out myself.

I've tried to copy your example and get:
[ts]
Argument of type '{ name: ProjectFn<AbstractControlState<string>>; blah: ProjectFn<AbstractControlState<string | any...' is not assignable to parameter of type 'FormGroupState<TemplateRole>'.
  Object literal may only specify known properties, and 'name' does not exist in type 'FormGroupState<TemplateRole>'.

with the model:
iterface Template {
  name: string;
  roles: TemplateRole[];
}

interface TemplateRole {
  name: string;
}

with code:
createFormGroupReducerWithUpdate<Template>({
  name: validate<string>(required),
  roles: updateArray<TemplateRole>(
    updateGroup<TemplateRole>({
      name: validate<string>(required)
    })
  )
});

Yes, the typing for those update functions is a bit sub-optimal. This should do the trick:
createFormGroupReducerWithUpdate<Template>({
  name: validate<string>(required),
  roles: roles => updateArray<TemplateRole>(
    cast(roles),
    updateGroup<TemplateRole>({
      name: validate<string>(required)
    }),
  )
});

With the next version (which I'll release as soon as TypeScript 2.8 is finally released) your version will work due to improved type inference for child controls. Also, I will clean up the update functions in general to be more consistent.

That works thanks.
But.
if you add another property to validate inside the updateArray, updateGroup or add another updateGroup you get the error again. I can't seem to find the right combination :(

Sorry for the late response. I hope you were able to find the right combination in the meantime. If not, to work around these typing issues (which I am in the process of fixing properly) you can always make the updateGroup or updateArray call explicit by providing an arrow function (Just as I did in my example for the updateArray call. Basically, in my previous example you would adjust the reducer like this:
createFormGroupReducerWithUpdate<Template>({
  name: validate<string>(required),
  roles: roles => {
    roles = validate<TemplateRole[]>(minLength(1), roles);
    return updateArray<TemplateRole>(
      cast(roles),
      // by making the `role` state parameter explicit we ensure that the
      // TypeScript compiler chooses the right overload
      role => updateGroup<TemplateRole>(cast(role), {
        name: validate<string>(required)
      }),
    );
  }
});

Alright, so in the new and shiny TypeScript 2.8 world the following will type check:
interface Template {
  name: string;
  roles: TemplateRole[];
}

interface TemplateRole {
  name: string;
}

createFormGroupReducerWithUpdate<Template>({
  name: validate(required),
  roles: compose(
    validate<TemplateRole[]>(minLength(1)),
    updateArray(
      updateGroup<TemplateRole>({
        name: validate(required),
      }),
    ),
  ),
});

As far as I know Angular plans to drop TypeScript 2.8 support sometime in May at which point I will also release version 3.0.0 of ngrx-forms. I'll close this issue once that happens.

I have just released version 3.0.0. The code above should now properly type check.


can not find how to make prediction for keras ML translation

I am trying to predict the pretrained model of keras Machine translation
but i get the error and i could not solve or find suitable answer
and it seems everyone gets this error
code:
model = load_model('model.h5')

single_x_test = ['how are you']

model.predict(np.array(single_x_test, ndmin=0))

Error:

ValueError: Error when checking input: expected embedding_1_input to have shape (4,) but got array with shape (1,)

Yeah, I've bumped into this issue for models trained using a distributed strategy (as I imagine is the case for most pre-trained models).
Instead of:
results = model.predict(inputs) # shape error

Try:
results = model(inputs, training=False)

@BlackSnow76
Moving this issue to closed status as there has been no activity, in case you still face the error please create a new issue.


🚀 Multi-tab file editor

[ ] Allow opening multiple files
[ ] Highlight open files in File Manager
[ ] Reuse File Manager: Switch between file editors using by selecting files
[ ] Indicate unsaved files inf File Manager
[ ] Bonus: do not reload main file (any files) from device if it has unsaved changes

Hello, I can see the file manager implemented in app.js. So then would app.js and app.css be the relevant files to make changes to here?


Fixed stream read blocking signals

this code demo blpop will block pcntl_signal
<?php
declare(ticks=1);
# I want ctrl + c terminal this process
# but that unavailable because int signal is blocking by fgets
include __DIR__ . '/../autoload.php';
pcntl_signal(SIGINT, function($signal) {
    echo("INT signal\n");
    exit(0);
});
$client = new \Predis\Client();
$client->blpop('test', 10);

https://github.com/imcj/predis/blob/demo-blocking/examples/stream_connection_blocking_signals.php
I fixed that. Add select in read, select not blocking signals.when signal arrived , select will raise an exception.and then dispatch signals again.
now, i can listen SIGINT, save process state. and quit.
why did that?
because i use predis in the imcj/mqk , that library start multi processes consume redis list message, and mapping to the event.i want use signal SIGINT graceful all processes, but predis hold all signals, i cann't listen SIGINT signal.
thx for all contributors.
    public function read()
    {
        $socket = $this->getResource();
        while (true) {
            try {
                $this->select([$socket], [], []);
            } catch (\Exception $e) {
                if ($e == 'Interrupted system call') {
                    pcntl_signal_dispatch();
                    continue;
                }
            }
            $chunk = fgets($socket);
            break;
        }

Hi, I'm currently investigating possibilities of non-blocking reads, since I want a long-running SUBSCRIBE-based script, which should also be able to respond to signals properly. Current implementation blocks in a hard way and there is no way around.

sorry,my english so bad.need some times to think what's do you say.maybe
few hours, becuase my country is am 11:28.
Michael Moravec<EMAIL_ADDRESS>上午10:25写道：

@Majkl578 commented on this pull request.
In src/Connection/StreamConnection.php
https://github.com/nrk/predis/pull/467#discussion_r172079541:

@@ -308,7 +308,18 @@ protected function write($buffer)
public function read()
{
$socket = $this->getResource();

   $chunk = fgets($socket);

   while (true) {

       try {

           $this->select([$socket], [], []);

Imho you can use non-blocking stream even without event loop. You just
need to loop synchronously, just like you did here with while( true), and
check fgets(...) !== false. The point is that in such synchronous loop,
signals are dispatched properly, unlike blocking stream.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/nrk/predis/pull/467#discussion_r172079541, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAhrOOAZFQuKWni9CfH34dqDaKprElSRks5tbKIIgaJpZM4P297M
.

@Majkl578 it's definitely true that in the last 12 months I didn't have much time to work on my open-source projects but this library is definitely not abandoned... on hiatus would probably be a better definition of its status, after all it doesn't have major issues (I can't reproduce most of those currently opened TBH) and there are a few improvements for the current stable branch that would be nice to have but not urgent. It doesn't explicitly come with a profile for Redis 4.x and this is probably the most striking issue right now, but it's not like it won't work and devs can still inject their own implementation of commands if they really need them.
That said I'd like to work on it again very soon but since I don't have a good track record on my promises in this regard I won't dare to give a timeframe. By the way good contributions would be a great help, unfortunately in the past, aside from very few exceptions, I didn't really get much help beyond minor fixes which are greatly appreciated anyway, but they sstill don't change the fact that this basically remains a one-man project.
As for support for old PHP releases, the fact that it still maintains backwards compatibility with PHP 5.x is quite a given for versions in the 1.x branch... I mean there's no real need to shake-up the internals of this library just for the fun of it when it's stable and such, these new features are for the most part not really required for Predis to work fine. Obviously it's a completely different matter for the 2.x branch: as I've stated before it's pretty much decided that Predis 2.x will drop support for PHP 5.x exactly because v1.x, with minor fixes and maintenance, will still do fine for past versions of PHP for many years to come.
I hope this clarifies a bit the current state of Predis.


Add drag and drop to 'Manage Environments'

Expected Behavior

You should be able to use drag and drop to re-order environments (Manage Environments)
The 'Sub Environments' ordering within 'Manage Environments' should match the 'Activate Environment' order (at the top of the sidebar)

Actual Behavior

Environments within 'Manage Environments' cannot be reordered
The 'Sub Environments' ordering within 'Manage Environments' does not match the 'Activate Environment' order

Reproduction Steps
No response
Is there an existing issue for this?

[X] I have searched the issue tracker for this problem.

Additional Information
Likely related to https://github.com/Kong/insomnia/issues/5101
Insomnia Version
2022.7.0-beta.4
What operating system are you using?
Ubuntu
Operating System Version
20.04.1
Installation method
deb download from github releases
Last Known Working Insomnia version
No response

Hi @darcyrush this is indeed a known issue. We've temporarily removed the option to drag and drop within the Environment Editor in order to fix other bugs and do other improvements.
We'll be improving and re-adding drag and drop as soon as possible.

@filfreire if possible, I personally would like to add/have the feature to be able to group the sub-environments into folders as well.

@darcyrush we've implemented this change, and it will return in the next patch release / beta release


can we have one liners?

like snake("foo-bar")?

I made this library so I could chain multiple methods like snake case a string and make all letters upper or lower so I didn't add a feature as you mentioned.


Rules for version number validation ?

Hello,
I am having issues with importing a new device update. It seems that the version number is validated against some rules. We make rc versions of our updates and there version number lookis like X.Y.Z-rc which gives me this error : Cannot parse import manifest. (Error parsing version string: 0.0.1-rc. Path 'updateId.version', line 5, position 25.)
Are those rules specified somewhere ? I couldn't find them anywhere.
Thanks for your help

@NicolasRiou, only numerical version number is supported. The rules are specified in our import manifest schema. Hope this helps.

It does help, thank you !


@typings for @ngrx/store

Hi, where can i find typings for @ngrx/store, @ngrx/db, etc... ?

sorry, found it my bad.

@gouravcoder please close :)

@gouravcoder A hint were you found it would be great. Thank you!

@denyo there isn't a special for the typings. They are included when you install the packages

@brandonroberts Thanks for the quick reply.
I already had them but didn't really notice because I was looking for more detailed comments on each function like I am used to from DefinitelyTyped.


[K12] Pushing fix to allow TSO automation for K12 1.18

Fix package.xml and layout file  to deploy 2 additional layouts and fields.
Critical Changes
Changes
Issues Closed
New Metadata
Deleted Metadata
Testing Notes

@jofsky - reviewing now.

@spinkelman - Do you mind reviewing the TSO updates once @jofsky has made the changes called out in the TSO update WI?


Fix/faq relative links

This PR fixes the issues with relative links from https://github.com/corona-warn-app/cwa-website/issues/2611 "Multiple broken links in FAQs (approx. 65)".
Two points are addressed:

non-working links using ../ construction are corrected
different ways of writing links to other sections are harmonized

Harmonization:

For language dependent directories, with a /en and a /de variant, the construct ../../directory is used. This applies to  '../../screenshots, '../../eventregistration and  '../../blog
The simpler construct '/assets is used, since this is a language independent directory, instead of ../../../assets.

This is partly a duplicate of https://github.com/corona-warn-app/cwa-website/pull/2610.

I think neither my PR #2610 nor this PR covers all links. There are some changes here I forgot in #2610 but also some from #2610 are missing here.

@Ein-Tim

This is partly a duplicate of https://github.com/corona-warn-app/cwa-website/pull/2610.

That is correct. I submitted this PR in draft status because I want to hear feedback from the Open Source Team about how they want to handle the issues. If they want to go your route and correct only the blog links first, that is OK with me. I would modify this PR accordingly to remove the blog changes.

I think neither my PR https://github.com/corona-warn-app/cwa-website/pull/2610 nor this PR covers all links. There are some changes here I forgot in https://github.com/corona-warn-app/cwa-website/pull/2610 but also some from https://github.com/corona-warn-app/cwa-website/pull/2610 are missing here.

Again, I suggest to wait for feedback from the Open Source Team before putting in more effort into either of the two PRs. I believe I fixed all the bad blog links in the FAQs in this PR. I didn't compare the two PRs so far.

This PR is now ready for review and merging.
The full (updated) Cypress suite passes with no remaining errors.


Reduces logging of discovery of entity proxiy to warning instead of e…

…rror
Signed-off-by: Graham Wallis<EMAIL_ADDRESS>Fixes #4151

Thanks @grahamwallis Can we also do it for o.o.o.a.r.g.r.GraphOMRSMetadataCollection --- isRelationshipKnown
Example:
2020-11-25 21:23:32.109 ERROR [ForkJoinPool.commonPool-worker-2] o.o.o.a.r.g.r.GraphOMRSMetadataCollection --- isRelationshipKnown relationship with GUID 98772b3f-23c3-4323-aa6b-73c93bc0ef4f:data_file_record<EMAIL_ADDRESS>does not exist in repository omas-server

@lpalashevski Sure - have just added it.

Awesome, much appreciated!


Performance Issues with Lemmatization

I'm encountering a very slow processing time on a Pandas DataFrame of ~40k sentences (generally short to mid-length extracted from TV dialog).
It appears to be doing 4k sentences/10mins on a GPU/High-Memory Pro Google Colab notebook.  This may be normal or infact good. It seems to be 2x the speed of the SpaCy pipeline specs here:
https://stackoverflow.com/questions/51372724/how-to-speed-up-spacy-lemmatization
Any guidance on speeding up this processing step would be appreciated. Could this be accelerated with multi-core? Are any config changes required? Any guidelines on breakpoints on memory size of Google Cloud instances that may help?
Great package, long-overdue and very needed. - J

Hi J,
This package is leveraging WordNetLemmatizer from the nltk package, as you can see from the source code here. If you only need the lemmatization function, you can directly use the nltk package.
To process large amount of data, you might want to leverage big data solution, such as Spark. Below is an example how you can wrap the code, and run in Spark environment such as Azure Databricks.
I have no experience with Google cloud, hence I cannot answer your questions about it.
from text_preprocessing import preprocess_text
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType
from pyspark.sql import DataFrame as SparkDataFrame

def preprocess_text_spark(df: SparkDataFrame,
                          target_column: str,
                          preprocessed_column_name: str = 'preprocessed_text'
                         ) -> SparkDataFrame:
    """ Preprocess text in a column of a PySpark DataFrame by leveraging PySpark UDF to preprocess text in parallel """
    _preprocess_text = udf(preprocess_text, StringType())
    new_df = df.withColumn(preprocessed_column_name, _preprocess_text(df[target_column]))
    return new_df

Closed issue due to inactivity.


load tensorrt engine file in c++

Hello. Thanks for your work. I'm new in tensorrt c++. I worked with your detectNet example and successfully used your pre-trained ssd_mobilenet_v2.
I have nvidia trafficcamnet engine file. How can i load this engine file in detectNet?

Hi @Maximal4028  did you have found any solution related to loading engine files in C++?


[BUG] When converting to an emoji, cursor is shifted one character back.

Screencast from 2023-09-09 12-33-03.webm
In my conversion rules, each result has the cursor placed after the space. However, for emojis after conversion, the cursor is place in between the emoji and the space, not after it.

LONG TIME!
Indeed, the code was not correctly handling characters with multiple Unicode units. I am currently addressing this issue.


Added friendly snippets plugin to LuaSnip

The friendly snippets plugin is super helpful and does not bloat the config very much, since we are adding only two lines.
As a beginner, it took me some time to figure out how to add this to the Kickstart config. So, I think this will be helpful for other users like myself who start with ThePrimegeans config video and then convert to Kickstart. I like this config a lot and plan on sticking with it, but this was something I immediately missed, and think that due to the audience overlap others may run into this too. Granted it is very simple, but I hate to admit it took me a long time to figure out how to add it, so maybe other beginners may struggle as well.
Feel free to disregard this if you don't want it! This is my first-ever PR so I apologize if I have done something incorrectly.

Hiya! Thanks for your contribution. I struggled with this too.
This project walks a difficult to manage line between providing a good starting point but not making too many assumptions around what people want.
There are conflicts to resolve between this PR and the current master, let's think on this and figure out what makes sense.

@feoh I think this is a good idea to add this. But seems like he wrote it when it was packer based. I tried it lazy but it keeps failing. If it is added here it will be helpful. Seems many people loved this feature

@DanielBrittainJr have you had a chance to rewrite this in lazyvim .seems you wrote that during kickstart was using packer as package manager.

Closing this in favor of #296


Enable LGTM.com analysis on PRs

Hi fonttools folks,
LGTM (AKA Semmle) is a free-for-open-source code analysis service, recently acquired by GitHub. It has already been running an analysis on the fonttools codebase (among many other open source projects), and has identified lots of alerts of varying severity. In adobe-type-tools-land, we've been running LGTM on AFDKO and psautohint Pull Requests  for a few months now, and have been very pleased with the results. It has identified many potential issues that would not likely have been picked up by our test suite.
I propose enabling LGTM's analyze-on-PR for fonttools/fonttools repo as well. It's a fairly unobtrusive check, not too chatty, and does not block merging (it's informational-only). And addressing the existing backlog of alerts might be a good way for folks to contribute to the project and get familiar with the codebase.
See https://lgtm.com/ for more info.

I think this LGTM bot is a bit too noisy. I'm constantly distracted by its notifications or when I am reading the PR messages. Would it be possible to only have its checks in the PR's "Checks" tab but not also have it spam the main PR's message board?

I agree, it's too noisy.

Comments on PRs can be disabled here by repo owners (which I'm currently not). If you can make me owner long enough to disable it I'll change the setting, or an owner can do it.

I just deactivated them. Thanks Josh


Fix signal handling

signal handling seems broken ... or context cancelation is not always picked up properly ... needs more work

not only nats neets to wait 3 sec. every service if run in single server mode (as in kubernetes) needs to wait to let the service deregister itself. -> a global sleep instead of the exit call.
we need to find a way to make the registry 'forget' services. no matter what we do to shutdown gracefully ... kubernetes might decide to OOM kill us before we can act ...

@fschade This is far from what we need to do. But it will at least give all reva services 3sec to deregister themselves.
However, there as still a problem with the natsjskv registry: we are using a TTL and nats will forget the service nodes. Unfortunately, it does not emit an event for the delete, causing the caches in our go micro clients to never forget the nodes.
Currently nodes are only forgotten if the service properly deregisters itself (which this PR fixes). If a node is OOM killed ... we don't forget it. I'll create a subseqent PR to put the TTL into the registry values and ignore them if they timed out.

regarding the code duplication, I wonder if we can use generics to reduce the amount of boilerplate we have ...


Keep the sources-controller, since 0.12 still has it

Issue to be fixed
Fixes #116
Proposed Changes

This PR fixes an overlooked error in 0.12, which is that 0.12 still has the deployment named sources-controller.

Release Note

/lgtm


Documentation correct but index.js needs updating

The documentation is correct, however the source code for index.js needs updating.
loginWithAppServiceMSI function needs resource option passed in for the key vault.
i.e.
function getKeyVaultCredentials(){
  if (process.env.APPSETTING_WEBSITE_SITE_NAME){
    return msRestAzure.loginWithAppServiceMSI({resource: 'https://vault.azure.net'});
  } else {
    return msRestAzure.loginWithServicePrincipalSecret(clientId, secret, domain);
  }
}

This should be resolved.  Please reopen if you are able to reproduce.  Thanks


NC | NSFS | Versioning | Delete of partial directory of nested key results in AccessDeniedError

Environment info

NooBaa Version: 5.18.0 (current master)
Platform: NC

Actual behavior

When a bucket with versioning enabled and there is a nested key inside and the client tries to delete the parent directory of the file without the last slash '/' (let's say it was by accident) will result in an error AccessDenied and in FS layer it is Error: Operation not permitted, code: 'EPERM'

Expected behavior

Create a delete marker without an error.

Steps to reproduce

Create an account with the CLI: sudo node src/cmd/manage_nsfs account add --name <account-name> --new_buckets_path /tmp/nsfs_root1 --access_key <access-key> --secret_key <secret-key> --uid <uid> --gid <gid>
Note: before creating the account need to give permission to the new_buckets_path: chmod 777 /tmp/nsfs_root1, chmod 777 /tmp/nsfs_root2.
Start the NSFS server with: sudo node src/cmd/nsfs --debug 5
Notes:

I Change the config.NSFS_CHECK_BUCKET_BOUNDARIES = false; //SDSD because I’m using the /tmp/ and not /private/tmp/.

Create the alias for S3 service:alias nc-user-1-s3=‘AWS_ACCESS_KEY_ID=<access-key> AWS_SECRET_ACCESS_KEY=<secret-key> aws --no-verify-ssl --endpoint-url https://localhost:6443’.
Check the connection to the endpoint and try to list the buckets (should be empty): nc-user-1-s3 s3 ls; echo $?

More information - Screenshots / Logs / Other output
Might be that this issue is with the same root cause of the issue - #8320
Oct-10 16:13:38.960 [nsfs/79233]  [WARN] core.sdk.namespace_fs:: NamespaceFS._delete_latest_version error: retries=10 latest_ver_path=/tmp/nsfs_root1/bucket-v/a/b/c [Error: Operation not permitted] { code: 'EPERM', context: 'SafeLink _link_from.c_str()=/tmp/nsfs_root1/bucket-v/a/b/c _link_to.c_str()=/tmp/nsfs_root1/bucket-v/a/b/.versions/c_null _link_expected_mtime=1728566008914826496 _link_expected_inode=152582414 ' }
Oct-10 16:13:38.961 [nsfs/79233] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error>AccessDenied<Message>Access Denied</Message><Resource>/bucket-v//a/b/c</Resource><RequestId>m23bksck-csss5c-135r</RequestId></Error> DELETE /bucket-v//a/b/c {"host":"localhost:6443","accept-encoding":"identity","user-agent":"aws-cli/2.17.11 md/awscrt#0.20.11 ua/2.0 os/macos#24.0.0 md/arch#arm64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3api.delete-object","x-amz-date":"20241010T131338Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=<>/20241010/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4dc8ae840909447ab17ef08ce5c6d25c62a422ed1ba7fa79900a210ac1cfa3b5","content-length":"0"} Error: Operation not permitted - context: SafeLink _link_from.c_str()=/tmp/nsfs_root1/bucket-v/a/b/c _link_to.c_str()=/tmp/nsfs_root1/bucket-v/a/b/.versions/c_null _link_expected_mtime=1728566008914826496 _link_expected_inode=152582414

@romayalon same behaiour is tested in version enabled bucket and its working with out creating any issue

put key to object

s3-api put-object --bucket naveen-noobaa-bucket --key x/y/z/lala.txt
{
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "ServerSideEncryption": "AES256",
    "VersionId": "xoWDrLSfwBFc4ByeDT1hD3VIPambJblb"
}

listing item

3-api list-object-versions --bucket naveen-noobaa-bucket
{
    "Versions": [
        {
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Key": "x/y/z/lala.txt",
            "VersionId": "xoWDrLSfwBFc4ByeDT1hD3VIPambJblb",
            "IsLatest": true,
            "LastModified": "2024-11-04T10:10:26+00:00",
            "Owner": {
                "DisplayName": "pnt-aws-root+255735097469",
                "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
            }
        }
    ],
    "RequestCharged": null
}

delete parial dir key and its retun delete marker

s3-api delete-object --bucket naveen-noobaa-bucket --key  x/y/z
{
    "DeleteMarker": true,
    "VersionId": "5OKNiYVfBRTsyZzQhEV9WMQcpq3H8Gn0"
}

list bucket after deleting partial dir

naveenpaul:noobaa-core$ s3-api list-object-versions --bucket naveen-noobaa-bucket
{
    "Versions": [
        {
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Key": "x/y/z/lala.txt",
            "VersionId": "xoWDrLSfwBFc4ByeDT1hD3VIPambJblb",
            "IsLatest": true,
            "LastModified": "2024-11-04T10:10:26+00:00",
            "Owner": {
                "DisplayName": "pnt-aws-root+255735097469",
                "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
            }
        }
    ],
    "DeleteMarkers": [
        {
            "Owner": {
                "DisplayName": "pnt-aws-root+255735097469",
                "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
            },
            "Key": "x/y/z",
            "VersionId": "5OKNiYVfBRTsyZzQhEV9WMQcpq3H8Gn0",
            "IsLatest": true,
            "LastModified": "2024-11-04T10:11:33+00:00"
        }
    ],
    "RequestCharged": null
}

cc : @shirady

@romayalon both noobaa and aws behaving similer, When I try to get object using dir path such as (a/b/c or  a/b/c/) its returning The specified key does not exist. error for both noobaa and AWS
Local noobaa behavior

put object

s3-local put-object --bucket bucket4 --key a/b/c/lala.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "ETag": "\"mtime-d5ddydsepvcw-ino-byd1o\"",
    "VersionId": "mtime-d5ddydsepvcw-ino-byd1o"
}

try to get using dir path both failed

s3-local get-object --bucket bucket4 --key a/b/c ./src/cmd/text1.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

s3-local get-object --bucket bucket4 --key a/b/c/ ./src/cmd/text1.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

try with object key path and its working

naveenpaul:noobaa-core$ s3-local get-object --bucket bucket4 --key a/b/c/lala.txt ./src/cmd/text1.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "AcceptRanges": "bytes",
    "LastModified": "2024-11-04T12:16:30+00:00",
    "ContentLength": 0,
    "ETag": "\"mtime-d5ddydsepvcw-ino-byd1o\"",
    "VersionId": "mtime-d5ddydsepvcw-ino-byd1o",
    "ContentType": "text/plain",
    "Metadata": {}
}

AWS Behavior

list object items

naveenpaul:noobaa-core$  s3-aws list-objects --bucket naveen-noobaa-bucket
{
    "Contents": [
        {
            "Key": "i/j/k/lala.txt",
            "LastModified": "2024-11-04T11:50:33+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "pnt-aws-root+255735097469",
                "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
            }
        },
        {
            "Key": "x/y/z/lala.txt",
            "LastModified": "2024-11-04T10:10:26+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "pnt-aws-root+255735097469",
                "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
            }
        }
    ],
    "RequestCharged": null
}

try to get using dir path (i/j/k or i/j/k/) both failed.

s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k ./src/cmd/text1.txt
An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k/ ./src/cmd/text1.txt\
An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

get object with object complete path and its working

noobaa-core$ s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k/lala.txt ./src/cmd/text1.txt
{
    "AcceptRanges": "bytes",
    "LastModified": "2024-11-04T11:50:33+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "VersionId": "MnMY1mGNXquvYLicPaEeqgFb8gmNJ_Ff",
    "ContentType": "binary/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}

@romayalon both noobaa and aws behaving similer, When I try to get object using dir path such as (a/b/c or a/b/c/) its returning The specified key does not exist. error for both noobaa and AWS
Local noobaa behavior

put object

s3-local put-object --bucket bucket4 --key a/b/c/lala.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "ETag": "\"mtime-d5ddydsepvcw-ino-byd1o\"",
    "VersionId": "mtime-d5ddydsepvcw-ino-byd1o"
}

try to get using dir path both failed

s3-local get-object --bucket bucket4 --key a/b/c ./src/cmd/text1.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

s3-local get-object --bucket bucket4 --key a/b/c/ ./src/cmd/text1.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

try with object key path and its working

naveenpaul:noobaa-core$ s3-local get-object --bucket bucket4 --key a/b/c/lala.txt ./src/cmd/text1.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "AcceptRanges": "bytes",
    "LastModified": "2024-11-04T12:16:30+00:00",
    "ContentLength": 0,
    "ETag": "\"mtime-d5ddydsepvcw-ino-byd1o\"",
    "VersionId": "mtime-d5ddydsepvcw-ino-byd1o",
    "ContentType": "text/plain",
    "Metadata": {}
}

AWS Behavior

list object items

naveenpaul:noobaa-core$  s3-aws list-objects --bucket naveen-noobaa-bucket
{
    "Contents": [
        {
            "Key": "i/j/k/lala.txt",
            "LastModified": "2024-11-04T11:50:33+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "pnt-aws-root+255735097469",
                "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
            }
        },
        {
            "Key": "x/y/z/lala.txt",
            "LastModified": "2024-11-04T10:10:26+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "pnt-aws-root+255735097469",
                "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
            }
        }
    ],
    "RequestCharged": null
}

try to get using dir path (i/j/k or i/j/k/) both failed.

s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k ./src/cmd/text1.txt
An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k/ ./src/cmd/text1.txt\
An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

get object with object complete path and its working

noobaa-core$ s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k/lala.txt ./src/cmd/text1.txt
{
    "AcceptRanges": "bytes",
    "LastModified": "2024-11-04T11:50:33+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "VersionId": "MnMY1mGNXquvYLicPaEeqgFb8gmNJ_Ff",
    "ContentType": "binary/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}

On NooBaa you tested it on the branch that contains your fix https://github.com/noobaa/noobaa-core/pull/8480 ?

yes @romayalon , I did the test with my changes in branch delete_partial_dir


Update PT-BR translation

This fixes typos, improve some strings and finish the translation.

Thanks for the contribution.
I'll be available in the next release.


[url_launcher] documentation - example code fix

Description
Converted return type of _launchURL from void to Future. It greeted me with a red screen once. Even though it had gone away later, thought it could help with reducing the confusion.
Related Issues
Nope
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
Pull request directly to Master, since it was a small, direct edit from the browser.

[ ] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See [Contributor Guide]).

[ ] All existing and new tests are passing.

[ ] I updated/added relevant documentation (doc comments with ///).

[ ] The analyzer (flutter analyze) does not report any problems on my PR.

[ ] I read and followed the [Flutter Style Guide].

[x] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]

[ ] I updated pubspec.yaml with an appropriate new version according to the [pub versioning philosophy].

[ ] I updated CHANGELOG.md to add a description of the change.

[x] I signed the [CLA].

[ ] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

@googlebot  I signed it!

Closing, it need further changes making the example complex, sorry.


[BUG] Close Button Not Working Properly on Installation Options Dialog

Please confirm these before moving forward

[X] I have searched for my issue and have not found a work-in-progress/duplicate/resolved issue.
[X] I have tested that this issue has not been fixed in the latest (beta or stable) release.
[X] I have checked the FAQ section for solutions.
[X] This issue is about a bug (if it is not, please use the correct template).

UniGetUI Version
3.1.0
Windows version, edition and architecture
Windows 11 Pro 22631.3810
Describe your issue
Clicking the X (close icon) on red box does not close the dialog box in the Installation Options dialog box, only clicking a small portion at the lower side of the X icon closes it. But works perfectly in fullscreen.
Steps to reproduce the issue
Open UniGetUI in normal state (not fullscreen)
Go to the Software Updates tab.
Open the Installation Options dialog box via icon.
Click the red X (close icon) at the top right.
UniGetUI Log
[03/07/2024 7:06:45 am]
                           __  __      _ ______     __  __  ______
                          / / / /___  (_) ____/__  / /_/ / / /  _/
                         / / / / __ \/ / / __/ _ \/ __/ / / // /
                        / /_/ / / / / / /_/ /  __/ /_/ /_/ // /
                        \____/_/ /_/_/\____/\___/\__/\____/___/
                            Welcome to UniGetUI Version 3.1.0
[03/07/2024 7:06:45 am]
[03/07/2024 7:06:45 am] Version Code:  3.1
[03/07/2024 7:06:46 am] An error occurred while migrating chocolatey
[03/07/2024 7:06:46 am] System.ArgumentException: The path is empty. (Parameter 'relativeTo')
                           at System.IO.Path.GetRelativePath(String relativeTo, String path, StringComparison comparisonType)
                           at UniGetUI.PackageEngine.Managers.ChocolateyManager.Chocolatey.LoadManager() in C:\SomePrograms\WingetUI-Store\src\UniGetUI.PackageEngine.Managers.Chocolatey\Chocolatey.cs:line 278
[03/07/2024 7:06:46 am] Lang files were updated successfully from GitHub
[03/07/2024 7:06:46 am] Downloaded new icons and screenshots successfully!
[03/07/2024 7:06:47 am] Command scoop was not found on the system
[03/07/2024 7:07:03 am] Backup saved to C:\Users\73djj\Documents\UniGetUI\GUY_FAWKES installed packages 2024-07-03 07-07-03.json

Package Managers Logs
Manager Winget with version:
Naive WinGet CLI Version: v1.8.1791
Microsoft.WinGet.Client PSModule version:
Using Native WinGet helper (COM Api)

——————————————————————————————————————————

Logged native task on manager Winget. Task type is ListSources
Process start time: 03/07/2024 7:19:07 am
Process end time:   03/07/2024 7:19:07 am

-- Task information
 ...

The task reported success

——————————————————————————————————————————

Logged subprocess-based task on manager Winget. Task type is RefreshIndexes
Subprocess executable: "C:\Users\73djj\AppData\Local\Microsoft\WindowsApps\winget.exe"
Command-line arguments: " source update --disable-interactivity"
Process start time: 03/07/2024 7:19:11 am
Process end time:   03/07/2024 7:19:14 am

-- Process STDOUT
 ...

Return code: SUCCESS (0)

——————————————————————————————————————————

Logged subprocess-based task on manager Winget. Task type is ListPackages
Subprocess executable: "C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe"
Command-line arguments: "-ExecutionPolicy Bypass -NoLogo -NoProfile"
Process start time: 03/07/2024 7:19:11 am
Process end time:   UNFINISHED

-- Process STDIN
 ...

-- Process STDOUT
 ...

Return code: Process has not finished yet

——————————————————————————————————————————

Logged native task on manager Winget. Task type is ListSources
Process start time: 03/07/2024 7:19:12 am
Process end time:   03/07/2024 7:19:12 am

-- Task information
 ...

The task reported success

——————————————————————————————————————————

Logged subprocess-based task on manager Winget. Task type is ListUpdates
Subprocess executable: "C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe"
Command-line arguments: "-ExecutionPolicy Bypass -NoLogo -NoProfile"
Process start time: 03/07/2024 7:19:14 am
Process end time:   UNFINISHED

-- Process STDIN
 ...

-- Process STDOUT
 ...

Return code: Process has not finished yet

——————————————————————————————————————————

Relevant information
This issue does not affect all dialog boxes.
The problem is specific to Installation Options dialog box.
Screenshots and videos

What happens if you resize the UniGetUI window so it is a little bit taller?

I suspect it is interfering with the title bar

This issue was fixed on AppSdk 1.6.
Latest UniGetUI beta uses AppSdk 1.6


Add support for Angular libraries to Keys Detective

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report
[ ] Performance issue
[x] Feature request
[ ] Documentation issue or request
[ ] Support request
[ ] Other... Please describe:

Current behavior
Angular's suggested folder structure is the following:

Root

node_modules
projects

lib1
lib2

src

app
assets

It's very common to need translation support in files inside the libraries. In our case, we have one unique file per language for all libraries and app inside the assets folder.
There's no problem with the keys extractor, but our problem is with the keys detective, since:

Running it with -i src will detect keys existing in projects as extra keys.
Running it with -i projects will detect keys existing in src/app as extra keys.
Running it with -i . will scan node_modules which is less than ideal and causes errors.

Expected behavior
We think either one of the following functionalities will solve the problem:

Add an option to exclude one or several folders, so folders like node_modules aren't scanned.
Add the option for adding multiple inputs folder.

Minimal reproduction of the problem with instructions

Create a new project using Angular CLI.
Add a new library using Angular CLI.
Add some translation keys in both the library and the app folder.
Try to use the Keys Detective.

What is the motivation / use case for changing the behavior?
Having the need for translation support in both an Angular application and libraries at the same time.

@potray just to make sure I understand, you also have keys in your src folder? the src contains your main app and projects you supporting libs??

@shaharkazaz Yes, we have keys in both root/projects/ and in root/src/app/. Sadly scanning in root also scans node_modules, so it's not an option.

@potray How about we add support for an array of input paths?
Would you like to create a PR for it?

@shaharkazaz That would be perfect.
I cannot promise a PR since lately I'm having difficulties finding time to do stuff, but if I can and nobody is on it I can try :)

@potray try v2.2.0 and LMK how it goes 🙂

Thanks for the new functionality! I updated but sadly I'm experiencing the following
/home/yarrick/bee/client/node_modules/@ngneat/transloco-keys-manager/helpers/updateScopesMap.js:1
"use strict";Object.defineProperty(exports,"__esModule",{value:!0});const toCamelCase_1=require("./toCamelCase"),readFile_1=require("./readFile"),glob=require("glob"),scopes_1=
require("../keysBuilder/scopes"),tsquery_1=require("@phenomnomnominal/tsquery"),base="ObjectLiteralExpression:has(PropertyAssignment > Identifier[name=TRANSLOCO_SCOPE]) Propert
yAssignment:has(Identifier[name=/useValue|useFactory/])",useStringQuery=`${base} > StringLiteral`,useObjectQuery=`${base} > ObjectLiteralExpression`;function updateScopesMap({i
nput:e,files:s}){const t=s||e.map(e=>glob.sync(`${e}/**/*.ts`)).flat(),r={};for(const e of t){const s=readFile_1.readFile(e);if(!1===s.includes("TRANSLOCO_SCOPE"))continue;let
t={};const o=tsquery_1.tsquery.ast(s),i=tsquery_1.tsquery(o,useStringQuery);if(0===i.length){const e=tsquery_1.tsquery(o,useObjectQuery);for(const s of e)for(const e of s.prope
rties)if(e.initializer){const s=e.name.text;

TypeError: e.map(...).flat is not a function
    at Object.updateScopesMap (/home/yarrick/bee/client/node_modules/@ngneat/transloco-keys-manager/helpers/updateScopesMap.js:1:593)
    at Object.resolveConfig (/home/yarrick/bee/client/node_modules/@ngneat/transloco-keys-manager/helpers/resolveConfig.js:1:1099)
    at Object.findMissingKeys (/home/yarrick/bee/client/node_modules/@ngneat/transloco-keys-manager/keysDetective.js:1:491)
    at Object.<anonymous> (/home/yarrick/bee/client/node_modules/@ngneat/transloco-keys-manager/index.js:2:832)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)

Sadly before I updated to v2.2.0 I was using v2.1.0 without trying the find script, and before that we were using v1 (yes, we totally forgot to update it for very long 😅) so I cannot provide further information.

@potray can you please provide more information? node version, the configuration used, and every env related info.

Sure:

npm v6.14.4
node v10.21.0
The script I ran: transloco-keys-manager extract --unflat --sort
Environment (all running inside a WSL 2 instance):

Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal

Anything else that could be useful?

@potray The issue is with your node version, array flat is supported starting Node 11.
I'll add a polyfill and release a fix.

@potray can you please try v2.2.1?

@shaharkazaz works like a charm!!

@potray I need you to isolate the issue.
Please try each input path alone and see which one reproduces this error.
Also, try to downgrade back to 2.1.0 and re-run on each path.
Thanks

@shaharkazaz I was able to isolate the issue.
We have a file with the following content:
//noinspection ES6UnusedImports
import {} from '@ngneat/transloco'; // for key extractor to work

/**
 * t(action.collection.bigImpact)
 * t(action.collection.quickWins)
 * t(action.collection.triedAndTested)
 * t(action.collection.groundwork)
 * t(action.collection.allActions)
 */

Somehow removing one of the lines makes the error disappear. This is very weird and it makes little sense to me, but it's what I experience 😕

@potray which line are you removing?
And why do you have such a file? Shouldn't it be part of the template or an actual component file?

@shaharkazaz Removing any of the 5 lines in the comment make the script work.
We have more content in that file (models to be more exact), but I removed everything until I managed to find the culprit code, which is that comment.

@potray you say the first run works as expected? Re-runing triggers the error?
Have you tried to combine all the comment values under a single t(...)?

@shaharkazaz Sorry, that was very misleading on my part. What happened exactly was the script ran, I identified extra keys because the comment was missing, I added the comment I wrote, and after that, the script failed.
If you mean combining them like this
/**
 * t(action.collection.bigImpact,action.collection.quickWins,action.collection.triedAndTested,action.collection.groundwork,action.collection.allActions)
 */

It also produces the same error.

@potray thanks for clarifying.
I'll try to reproduce but if you can provide a reproduction that would be great 👍

@shaharkazaz Alright, this is crazy, but this is what happened:
I tried to recreate the issue in another Angular project and I was unable, so I thought something is wrong with our en.json file. I couldn't find anything so out of frustration I deleted the entire content of the en.json file, I still got the error, I undid, and now the error is gone 🤯
I can only bet of an extraneous character somewhere in the file, but so far I've run both the extract and find scripts, and both work flawlessly in the two folders we needed.
Thanks a lot for your time!!

@potray glad to hear! If you ever do find the reason, do share :)


Dependabot alerts

Remove webpack-log package to resolve security vulnerability
https://issues.redhat.com/browse/COST-2269

Codecov Report

Merging #2331 (2752e27) into master (6f92520) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2331   +/-   ##
=======================================
  Coverage   69.25%   69.25%
=======================================
  Files         410      410
  Lines        7755     7755
  Branches     1584     1584
=======================================
  Hits         5371     5371
  Misses       2348     2348
  Partials       36       36

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f92520...2752e27. Read the comment docs.


2.0.0

Added: Visually hidden element style [hidden][aria-hidden="false"]
Added: Currently updating element style [aria-busy="true"]
Added: Trigger element style [aria-controls]
Added: Color style for <html>
Added: CHANGELOG.md
Removed: Standards-breaking visually hidden style [hidden~="screen"]
Removed: Standards-breaking IE-proprietary style [unselectable="on"]
Removed: Prefix-less properties and the use of Autoprefixer
Updated: Form styling
Updated: Support for the latest Chrome, Edge, Firefox, and Safari
Updated: Licensing reference in package.json
Updated: Development dependencies
Updated: README.md and code documentation

The ARIA styles are brilliant.


Consolidate logic between acall_module_method and call_module_method.

call_module_method.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#1150  👈
#1149
#1144 : 1 other dependent PR (#1145 )
#1140
#1138
#1137
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @rohinb2 and the rest of your teammates on  Graphite


[FEAT] Dynamically parallel local parquet reader

Implement a dynamically parallel local streaming parquet reader.
Background
The current streaming local parquet reader, while fast and streaming, has some problems:

It reads and deserializes ALL row groups and ALL columns in parallel.
It does not respect downstream back-pressure (the crossbeam channels are all bounded by max chunks, it's free to fill it up).

This leads to unnecessarily high memory usage, and it potentially starves downstream tasks.
Solution
Instead of launching all tasks at once, we can cap the number of parallel tasks based on certain factors:

Number of CPUs
Number of Columns.

Results
Most glaringly, the benefits of these are in memory usage of streaming queries, for example:
next(daft.read_parquet("data/tpch-dbgen/1_0/1/parquet/lineitem").iter_partitions()) # read lineitem tpch sf1

The new implementation hits a peak of 300mb, while the old goes over 1gb.

Another example, where we stream the entire file, but the consumption is slow:
for _ in daft.read_parquet("/Users/colinho/Desktop/Daft/z/daft_tpch_100g_32part_64RG.parquet").iter_partitions():
    time.sleep(0.1)

The new implementation hits a peak of 1.2gb, while the old goes over 3gb.

To maintain perfomance parity, I also wrote some benchmarks for parquet files with differing rows / cols / row groups, the results show that the new implementation is pretty much on par, with some slight differences.

On reading a tpch sf-1 lineitem table though: the results are pretty much the same: (~0.2s)

Tested this new implementation on TPCH SF1:


fix(Enterprise Usage Reports): regen service to get Pager

PR summary
This commit contains a re-gen of the service to
leverage the new Pagers emitted by the SDK generator.
Signed-off-by: Phil Adams<EMAIL_ADDRESS>PR Checklist
Please make sure that your PR fulfills the following requirements:

[x] The commit message follows the Angular Commit Message Guidelines.
[x] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

Current vs new behavior

Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information

Looks good, but it seems you should run the Detect Secrets tool.

Just making sure our scanning job is running correctly :joy:

:tada: This PR is included in version 0.26.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Invalid Fantasy font resolving on Linux

~> fc-match fantasy
impact.ttf: "Impact" "Regular"

But font-kit returns Arial for some reasons.

Looks like this is because this font has Condensed width (75%), so a matching algorithm skips it. But Chrome, Firefox, pango, Qt are selecting Impact and not Arial.


Add documentation for metrics and entropy

It adds documentation for qutip.core.metrics and qutip.entropy.
Note:

Please suggest a better name for the file.
A better way to change to default backend(?)

@Ericgig I have made the required changes.


proof: Add introspection into the ephemeral node

Examples of when information about the ephemeral node can be used:

Storage engines for Merkle trees that contain ephemeral nodes as
well as immutable ones. In this case Ephem can be used to replace
IDs[begin:end] before making a query, and then Rehash is not
necessary.
Merkle tree drawing tools. The ephemeral node needs to be displayed,
and sometimes with a style distinct from other nodes.
Code that needs to extract individual hashes from inclusion and
consistency proofs. The Nodes type, with the introduction of Ephem
method, provides a complete reflection of the proof.

Codecov Report

Merging #15 (567c83b) into main (fd7bbd8) will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #15      +/-   ##
==========================================
+ Coverage   97.88%   97.93%   +0.04%
==========================================
  Files           6        6
  Lines         378      387       +9
==========================================
+ Hits          370      379       +9
  Misses          6        6
  Partials        2        2

Impacted Files
Coverage Δ

proof/proof.go
97.16% <100.00%> (+0.26%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd7bbd8...567c83b. Read the comment docs.

Wait before #17, I will rebase this PR on it, because it will become more straightforward.

@AlCutter @smeiklej This PR is now ready for review.


feat: add typescript definition

The v2 API is so simplified I think it shouldn't be a huge maintenance burden to keep the definition in this repo.
Types are based on the types in the README, plus the missing cloned event from #286. Constants were copied from the header file.
I'm unsure about the return type from watch. I based it on https://github.com/fsevents/fsevents/blob/e468f15134a1d19943f7c5d1d8d62ce1e60401c2/fsevents.js#L26-L30, but I don't know what the promise resolves to? Also, should it always return a Promise regardless of instance === null or not? Happy to tweak that part of the code.
EDIT: I've opened up a separate PR with the change to always return a Promise, and you can choose to accept that or not. I can tweak the type in this PR based on that conclusion: #296

Thank you

Thanks @paulmillr!


test-branch/execute-job/quoted-authorization/667eb782-1353-4dac-a821-74d49c1791ca

PR for testing add65409eb81a384d547814139f8a5be213b2d8b

/canonical/self-hosted-runners/run-workflows add65409eb81a384d547814139f8a5be213b2d8b


[1.20.1 Forge] Doesn't work

I have tested it on vanilla crops (wheat, carrot) and it doesn't work.
Canola crops do harvest, but I'm not sure whether the Ultimate Car Mod has that functionality built in for their own crops.

You probably need to change ./config/rightclickharvest.json or use the in game config editor.
requireHoe should be false for what you are expecting. Feel free to reopen if necessary


Adding Swagger Middleware prior to UseFastEndpoints

I'm experiencing an issue with swagger UI due to our current application utilizing an Authorization fallback policy to require an authenticated user by default. For normal MVC controllers I'm able to bypass this for the swagger UI by adding the middleware for swagger prior to the .UseAuthorization call.
This fails when doing something similar for FastEndpoints due to the Static Config.ServiceResolver not being set yet right here:
https://github.com/FastEndpoints/FastEndpoints/blob/1c5db30c632abf53280a0a8ca1ea5e2f5b57bc53/Src/Swagger/ValidationSchemaProcessor.cs#L45
Currently my only workaround is to have the following extension method to set the property via reflection
public static IApplicationBuilder UseFastEndpointServiceResolverWorkaround(this IApplicationBuilder app)
{
    var ep = ((app as IEndpointRouteBuilder) ?? throw new InvalidCastException("Cannot cast [app] to IEndpointRouteBuilder"));
    var type = typeof(Config);
    var field = type.GetField("_resolver", BindingFlags.NonPublic | BindingFlags.Static);
    field.SetValue(null, ep.ServiceProvider.GetRequiredService<IServiceResolver>());
    return app;
}

and with that I'm able to then order the middleware properly
app.UseFastEndpointServiceResolverWorkaround();
app.UseSwaggerGen();
app.UseAuthentication();
app.UseAuthorization();
app.UseFastEndpoints();

I'm not sure how common my scenario is & I'm happy to make a PR for this depending on feedback.
It looks like we can really simply use this same line of code in the UseSwaggerGen() extension:
https://github.com/FastEndpoints/FastEndpoints/blob/1c5db30c632abf53280a0a8ca1ea5e2f5b57bc53/Src/Library/Main/MainExtensions.cs#L65
or perhaps update the ValidationSchemaProcessor to take the services it depends on in the constructor

pls check out v<IP_ADDRESS>-beta.
should be sorted now.
thanks!

Yes, it is working in that version. Thanks for the quick turnaround!


fix(mempool): Extend ErrInvalidTx with CheckTxResponse info

When calling the RPC endpoints broadcast_tx_*, the mempool returns a reqRes object with a potential error. Before #4040, reqRes didn't have an error. An error on the ABCI CheckTx call would be returned encoded in the fields Code, Log, etc, of ResultBroadcastTx. Any other internal error of the mempool would be discarded.
The problem is that now when the transaction is invalid, broadcast_tx_* return ErrTxBroadcast{Source: ErrCheckTxFailed, ErrReason: err} where err  is simply ErrInvalidTx, without any other information. This PR adds all the fields in ResultBroadcastTx (Code, Data, Log, Codespace, and Hash) to ErrInvalidTx, so that this information is available gain to the broadcast_tx_* caller.

PR checklist

[x] Tests written/updated
[x] Changelog entry added in .changelog (we use unclog to manage our changelog)
[ ] Updated relevant documentation (docs/ or spec/) and code comments

Finally, regarding the "update docs/spec" in the check list, we might inform which information is returned by the caller (tx submitter) under https://docs.cometbft.com/v1.0/spec/abci/abci++_methods#checktx.


Intellisense does not recognize new Pandas dataframe variable as such when it is assigned with Pandas concat function

VS Code version: VS Code 1.47.3
Extension version (available under the Extensions sidebar): Python v2020.7.96456
OS and version: Windows 10, x64, 10.0.18363
Python version (& distribution if applicable, e.g. Anaconda): Python 3.8 (Windows store installation)
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
Relevant/affected Python packages and their versions: Pandas v1.0.4
Relevant/affected Python-related VS Code extensions and their versions: Python (Microsoft) v2020.7.96456
Value of the python.languageServer setting:
When I search settings for "python language server", I only get "Python: Lanuage Server" with the value "Microsoft"
There is no setting python.languageServer. Could it be that the language server is now included in the general Microsoft Python extension since intellisense does work except for the specific problem reported below.
I installed the PyLance language server but kept getting this issue:
https://github.com/microsoft/pylance-release/issues/192.

Thus, I have removed the PyLance extension and the problem reported here exists with the "Microsoft" python language server. I suggest that you release your new language server in the same place as the existing one and let it be exclusively setting based, which one to choose.
Steps to reproduce:
Code which demonstrates the problem of the issue reported here:
import pandas as pd

df_1 = pd.DataFrame()
df_2 = pd.DataFrame()
df_3 = pd.concat([df_1,  df_2])

df_3.rein

Expected behaviour
When typing the last line, intellisense should suggest the reindex function of the DataFrame class.
Actual behaviour
No functions are suggested. Forcing Intellisense lookup with Ctrl+Space gives the message: "No suggestions."

This doesn't reproduce in Pylance 2020.8.0.

Note that languageServer not accepting Pylance is #16, and will be fixed in the next Python extension release.


Obscure error fired when numerical values are set to CSS custom properties via this.customStyles

Description

An obscure exception is thrown (property.indexOf is not a function) when numerical values  (or other non string values) are set to CSS custom properties via this.customStyles
This makes it problematic to recover from or to understand and so on.
Live Demo
http://codepen.io/lozandier/pen/8ce0995ea6c5b174b3147f66d7450def
Steps to Reproduce

Create a custom element & append it to page
Optionally set custom variables in its designated stylesheet
In its ready/attached state, attempt to change an existing or
-->

Expected Results

When style-centric statements are ran with a nonstring value— such as this.customStyle['--baz'] = 2;  the method should fire an exception like the other core methods do when they encounter an invalid intent by end users that their intent is invalid and steps to provide explicitly a string value.
Alternatively, the setter should scrutinize the set value (i.e. using typeof) and correctly change it to string internally for common primitives sensible for style declarations such as Number.

Actual Results
this.customStyle['--baz'] = 2; causes an TypeError exception be thrown: VM3064 style-properties.html:127 Uncaught TypeError: property.indexOf is not a function
Browsers Affected

[x] Chrome
[x] Firefox
[x] Edge
[x] Safari 9
[x] Safari 8
[x] IE 11

Versions

Polymer:  all versions
webcomponents: n/a

The error is no longer thrown. Closing in favor of #3695


Update redirect policy to clear auth header

Update redirect policy to clear auth header when creating the redirect request.

One other thing that came up as part of ACR investigation was that any time a service call returned a redirect return code - we lost the headers of the original request(which is intentional) but ACR needs to intercept one of those headers.  As of now I am handling this via an internal ACR policy. I am not sure if this is a common enough scenario to be in the core? I am curious what others think?


Build for Python 3.6

Is it possible to build simplekml for Python 3.6 ?
https://anaconda.org/conda-forge/simplekml/files

It is noarch and you can install it with:
conda create --name TEST python=3.6 simplekml