NLog Memory Leak with slow log writer

Type (choose one):

Bug

NLog version: (e.g. 4.2.3)
4.4.12 and 4.5 (started with 4.4.12, tried an upgrade to 4.5)
Platform:  .Net 4.6.1 (Azure Web App)
Current NLog config (xml or C#, if relevant)
    <extensions>
      <add assembly="NLog.AzureBlobStorage"/>
    </extensions>
    <targets>
      <target type="AsyncWrapper" name="logger" overflowAction="Grow">
        <target type="AzureAppendBlob" name="bloblog" connectionString="logs" container="logs"
          layout="${date:format=yyyy-MM-dd HH\:mm\:ss.fffff},${threadid},${logger},${message},${all-event-properties}"
          blobName="catalog-api\${date:universalTime=true:format=yyyy}/${date:universalTime=true:format=MM}/${date:universalTime=true:format=yyyy-MM-dd}/${date:universalTime=true:format=HH}:00.log"/>
      </target>
      <target type="AsyncWrapper" name="publisher" overflowAction="Grow">
        <target type="AzureAppendBlob" name="blobexception" connectionString="logs" container="logs"
          layout="${date:format=yyyy-MM-dd HH\:mm\:ss.fffff},${threadid},${exception:format=type}${newline}${exception:format=message}${newline}${event-properties:item=RootId},${event-properties:item=ParentId},${event-properties:item=Id}${newline}${exception:format=data}${newline}${newline}${exception:format=stacktrace}${newline}"
          blobName="catalog-api\${date:universalTime=true:format=yyyy}/${date:universalTime=true:format=MM}/${date:universalTime=true:format=yyyy-MM-dd}/${date:universalTime=true:format=HH}:00-exception.log"/>
      </target>
    </targets>
    <rules>
      <logger name="*" minlevel="Trace" writeTo="logger"/>
      <logger name="exception" minlevel="Error" writeTo="publisher" final="true"/>
    </rules>
  </nlog>

In case of a BUG:

What is the current result?
As load increases on the site, AppPool increases in memory usage. This memory is never released.

What is the expected result?
AppPool to increase in memory as load increases (due to buffering of logs) but then release this as load descreases

Did you checked the Internal log?
yes

Please post full exception details (message, stacktrace, inner exceptions)
n/a

Are there any workarounds? yes/no
no

Is there a version in which it did work?
no

Can you help us by writing an unit test?
n/a

I know there have been a number of NLog memory leak issues raised before (#2369 #2220 #2343 #1846 #316) and these have been closed. However, I don't think that they have been closed with a proper resolution, more like "a workaround has been found that works, I'll close it".
The workaround appears to be finding a way to increase throughput of the file logger. So, I guess, the logger could then keep pace with the logs so objects did not start to grow in the async logger buffer, therefore the problem goes away.
In our case, we're using a custom Azure Blob logger and the existing workarounds for the async wrapper suggested by @snakefoot  did not make any difference. I suspect that our logging throughput is high enough, and the write performance of the Blob Logger is slow enough, so that no matter what happens we cause the async buffer to grow, and therefore increase memory usage.
This would be fine (and expected behaviour) as long as when the site cools down the buffer eventually gets drained and memory is released. This does not appear to be happening, as although all log writes are completed, memory remains high - potential culprit is the LogEventInfo object which is reported in #2220

I did some digging and then found this fork: https://github.com/jkowalski/NLog/pull/19/files that appears to explicitly dispose of the object that may be causing all the trouble. As far as I can see, this code isn't implemented in the official NLog code base.
I've checked this Azure Blob Logger code and it appears pretty simple so I don't think the reason is this extension, besides, even if we could increase it's performance, I feel we are still masking an underlying issue with the async wrapper.

The PR (https://github.com/jkowalski/NLog/pull/19) is indeed interesting. We could release a test package to check if that resolves your problem. @snakefoot what do you think?

That the PR is broken, and should not be used.
Could be nice if your memory profiler could show what object chain keeps the LogEventInfo alive. Maybe there is an array that requires manual clear.
Sent from my Samsung device
-------- Original message --------
From: Julian Verdurmen<EMAIL_ADDRESS>Date: 10/04/2018 12:07 (GMT+01:00)
To: NLog/NLog<EMAIL_ADDRESS>Cc: Rolf Kristensen<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [NLog/NLog] NLog Memory Leak with slow log writer (#2655)
The PR (jkowalski/NLog#19https://github.com/jkowalski/NLog/pull/19) is indeed interesting. We could release a test package to check if that resolves your problem. @snakefoothttps://github.com/snakefoot what do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/2655#issuecomment-380046504, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AK-fnEI0NF-LfjodS0iGOG9olaBeh5eAks5tnIR5gaJpZM4TN4kh.

Just to update this; one thing @snakefoot suggested in #2220 helps,

batchSize="1000"

This helps in that memory increases at a much lower rate, but it's still growing.
This:

"timeToSleepBetweenBatches="0"

does not make any difference.

If you are using the AzureAppendBlob from nuget. Then you can try and configure this on the blob-target (Not the AsyncWrapper)
OptimizeBufferReuse="true"
It can be used on targets that are very simple (Like the blob-target). Or on smart targets like file-target  (Already enabled by default)
Sent from my Samsung device
-------- Original message --------
From: grahambunce<EMAIL_ADDRESS>Date: 10/04/2018 13:46 (GMT+01:00)
To: NLog/NLog<EMAIL_ADDRESS>Cc: Rolf Kristensen<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [NLog/NLog] NLog Memory Leak with slow log writer (#2655)
Just to update this; one thing @snakefoothttps://github.com/snakefoot suggested in #2220https://github.com/NLog/NLog/issues/2220 helps,
batchSize="1000"
This helps in that memory increases at a much lower rate, but it's still growing.
This:
"timeToSleepBetweenBatches="0"
does not make any difference.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/2655#issuecomment-380070125, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AK-fnMEMciaLbOs3q7a75YR6ytUC9xDqks5tnJusgaJpZM4TN4kh.

@grahambunce Here is an updated objected chain if it helps?

I'm pretty sure that the System.Collections.Generic.Queue clears the reference after Dequeue. So it seems you have over 300.000 logevents pending waiting to get through NLog.AzureBlobStorage.
Thought your problem was that when the system was idle, then there was still a high memory pressure. Looks like you are shown the numbers of when the system is fully active?
You could also improve the NLog.AzureBlobStorage to handle batches of logevents, instead of just writing one at a time.

With Queue, List, and other such data structures in the .NET Framework, when you remove an element from the collection, the collection internally wipes out its reference to the stored element,

https://blogs.msdn.microsoft.com/pfxteam/2012/05/08/concurrentqueuet-holding-on-to-a-few-dequeued-elements/

Is it the repo:
https://github.com/heemskerkerik/NLog.AzureAppendBlob
?

@snakefoot Yes, even though the system is idle, the memory dump still shows nlog objects in memory. For example, I ran a soak test to increase memory, waited 60 minutes, confirmed that nothing was being written to the logs, and then took this memory dump.

I've still got the NLog objects in memory. I would have expected the GC to kick them out by then as the site wasn't doing anything at all. I took a few other snapshots too and, surprisingly, the NLog objects were slightly different (which I find odd) but still in the same ballpark.
@304NotModified Yes, I am using that repro.  There is another https://github.com/JDetmar/NLog.Extensions.AzureStorage/blob/master/src/NLog.Extensions.AzureStorage/BlobStorageTarget.cs that seems to have a more advanced NLog Target that supports this method:
protected override void Write(IList<AsyncLogEventInfo> logEvents)
I can't tell from the NLog framework how it decides to call the list overload when compared with the single overload but if as @snakefoot suggests this is a better option for performance and mitigating the issue I will try it.
I still wonder if even if I improve throughput it's still masking an underlying issue?

Made a small test program that uses the file-target to write against an non-existing path (KeepFileOpen=false, ConcurrentWrites=true).
The test program writes 1 million logevents, and reaches 1 Gigabyte in memory usage (Started from 100 Megabytes)
When I try to force GC.Collect(2) then it drops 100 MegaBytes. If I force one more then it drops 100 MegaBytes again. This continues until it reaches the starting point again of 100 MegaBytes.

@snakefoot thanks for creating the test. Is this agreeing or disagreeing with me? :) I guess one view is that, yes, the memory footprint increases as expected but that a forced GC will eventually drop the memory used. The other view is that, what happens if you don't force the GC?
In my case the app pool ran low on memory, requests backed up and then the site crashed... so any "auto" GC execution rules didn't pick up the problem...

@snakefoot >yes I can reproduce your issue, and the application doesn't reduce its memory usage automatically when idle, but needs "help".
Ok. Is this something that can be looked into within NLog (i.e. why is memory not being released without help) or do you consider this a hosting app issue?

This is not an issue that can be fixed in the NLog core engine. It had to be fixed in the custom Nlog target.
Sent from my Samsung device
-------- Original message --------
From: grahambunce<EMAIL_ADDRESS>Date: 12/04/2018 12:44 (GMT+01:00)
To: NLog/NLog<EMAIL_ADDRESS>Cc: Rolf Kristensen<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [NLog/NLog] NLog Memory Leak with slow log writer (#2655)
@snakefoothttps://github.com/snakefoot >yes I can reproduce your issue, and the application doesn't reduce its memory usage automatically when idle, but needs "help".
Ok. Is this something that can be looked into within NLog (i.e. why is memory not being released without help) or do you consider this a hosting app issue?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/2655#issuecomment-380760693, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AK-fnNfNsJ4R2VsI6cb7GOz0HXXmN1kOks5tnzATgaJpZM4TN4kh.

Maybe try and use the RoundRobin group and have multiple blob targets inside this. Each wrapped with their own AsyncWrapper
Sent from my Samsung device
-------- Original message --------
From: grahambunce<EMAIL_ADDRESS>Date: 12/04/2018 12:44 (GMT+01:00)
To: NLog/NLog<EMAIL_ADDRESS>Cc: Rolf Kristensen<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [NLog/NLog] NLog Memory Leak with slow log writer (#2655)
@snakefoothttps://github.com/snakefoot >yes I can reproduce your issue, and the application doesn't reduce its memory usage automatically when idle, but needs "help".
Ok. Is this something that can be looked into within NLog (i.e. why is memory not being released without help) or do you consider this a hosting app issue?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/2655#issuecomment-380760693, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AK-fnNfNsJ4R2VsI6cb7GOz0HXXmN1kOks5tnzATgaJpZM4TN4kh.

Based on comments by @snakefoot I tried the different blob writer https://github.com/JDetmar/NLog.Extensions.AzureStorage/blob/master/src/NLog.Extensions.AzureStorage/BlobStorageTarget.cs instead of my original. This writer had a method to handle multiple log events (I don't know how the NLog framework chooses to use this method over the single log event method) and then combine these into a single write to the blob.
This significantly improved the issue so I think we're back to "if we can write fast enough, the problem goes away".
I don't know if this means this issue should be closed, or kept as a known issue that can be investigated further - I'll leave that up to you.

I don't know how the NLog framework chooses to use this method over the single log event method

The NLog framework enables batching using the AsyncWrapper (Performance) or the BufferingWrapper (Throttle)

I don't know if this means this issue should be closed, or kept as a known issue that can be investigated further - I'll leave that up to you.

Close it the best IMO. it's now inactive.


fix: detect sharepoint folder URL with RootFolder param

:tada: This PR is included in version 1.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


How to run Function App on different port than default port 80

Hi, I am able to change the binding port, run & test func app locally by using,
"Host": {
"LocalHttpPort": 8080
}
in local.settings.json
How to achieve the same when running from container? I do not want use "docker run -p" command. I have tried to use,
ENV "Host.LocalHttpPort"=8080
in my Dockerfile.. But it is not working.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5d505ab9-d351-6650-6d0c-bb65d2daa229
Version Independent ID: 2a1d3487-a93d-2206-7ef9-a91aa275ec4e
Content: Create Azure Functions on Linux using a custom image
Content Source: articles/azure-functions/functions-create-function-linux-custom-image.md
Service: azure-functions
GitHub Login: @ggailey777
Microsoft Alias: glenga

@githubavi Thank you for posting your question. We will review this and provide an update as appropriate.

@githubavi The way you are trying will not work because the LocalHttpPort sets the default port used when running the local Functions host (func host start and func run) and the docker container has only function runtime and not func core tools hence it will not understand LocalHttpPort
You can try setting the port in your Dockerfile as shown below:
ENV ASPNETCORE_URLS=http://*:8080
https://github.com/Azure/azure-functions-docker/blob/master/host/2.0/stretch/amd64/appservice/start.sh

Thanks Ketan,
I will try and confirm.
-Avijit

From: KetanChawda-MSFT<EMAIL_ADDRESS>Sent: 21 March 2019 03:22
To: MicrosoftDocs/azure-docs
Cc: Avijit Samanta; Mention
Subject: Re: [MicrosoftDocs/azure-docs] How to run Function App on different port than default port 80 (#27508)
@githubavihttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgithubavi&data=02|01||796660492ce842617ec608d6ade722b6|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636887605560176841&sdata=4D8w%2Ft2RlzkXphP9TBiyXvmaBC3croSN6Oh26KI9sFs%3D&reserved=0 The way you are trying will not work because the LocalHttpPort sets the default port used when running the local Functions host (func host start and func run) and the docker container has only function runtime and not func core tools hence it will not understand LocalHttpPort
You can try setting the port in your Dockerfile as shown below:
ENV ASPNETCORE_URLS=http://*:8080
https://github.com/Azure/azure-functions-docker/blob/master/host/2.0/stretch/amd64/appservice/start.shhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-functions-docker%2Fblob%2Fmaster%2Fhost%2F2.0%2Fstretch%2Famd64%2Fappservice%2Fstart.sh&data=02|01||796660492ce842617ec608d6ade722b6|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636887605560186852&sdata=mW041dBssj4dlWocZp9DwqGQzxpGR7Xd%2BKGu3%2BG2rNI%3D&reserved=0
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F27508%23issuecomment-475176399&data=02|01||796660492ce842617ec608d6ade722b6|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636887605560206868&sdata=AhLb7E%2BMf84tpIakchkfAoMhjtMhJU3SPE%2B1f%2BXXyrc%3D&reserved=0, or mute the threadhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADz4vt7vge-gdyp_FdVMQ__Y8lLxZIXXks5vY11qgaJpZM4b798S&data=02|01||796660492ce842617ec608d6ade722b6|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636887605560216873&sdata=pCQxCOGhR%2FEybOelKmF9Nw0ZfShnWQuoyEZM0rm7Q2Y%3D&reserved=0.

@githubavi Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

Hi, sorry for my late response. It is working now.
Thanks for the help.
-Avijit
Get Outlook for Androidhttps://aka.ms/ghei36

From: KetanChawda-MSFT<EMAIL_ADDRESS>Sent: Monday, March 25, 2019 9:35:12 AM
To: MicrosoftDocs/azure-docs
Cc: Avijit Samanta; Mention
Subject: Re: [MicrosoftDocs/azure-docs] How to run Function App on different port than default port 80 (#27508)
Closed #27508https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F27508&data=02|01||ccaf2c0426694edf9fe008d6b0d7159c|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636890835152162457&sdata=KhENFBVRifUWYACg6OObl5kdVYNDUABX6HRWS4TvE%2FQ%3D&reserved=0.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F27508%23event-2225379913&data=02|01||ccaf2c0426694edf9fe008d6b0d7159c|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636890835152172465&sdata=SZEeiJPLcMtkjf0GhhJbfpc2e83yRa5oprtRFbr%2FlyE%3D&reserved=0, or mute the threadhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADz4vvEjmf0iIRpqneLJlSB3qYxObrLaks5vaEr4gaJpZM4b798S&data=02|01||ccaf2c0426694edf9fe008d6b0d7159c|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636890835152172465&sdata=T4YwmoSryljRxZ6Ea0X32K5meZc08m5mxHmSsrlTQgw%3D&reserved=0.

@githubavi Thank you for the confirmation. I am glad that it worked.

You can check this on Running Multiple Azure Functions Locally.


Fix advertising policy for release

Overview (Required)

https://github.com/DroidKaigi/conference-app-2018/pull/678

Let me merge 🙏


How to reduce memory when decoding on CPU?

With zipformer I can get good performance.
Currently, when I decode on CPU one by one (not using batch), the memory cost will go to 3.5G. The token size is 5000 and using greedy_search to decode. I try to reduce it to 4000, but the memory cost seem not decrease much.
Any idea to reduce it without obvious performance degrade?
Some model configurations below:
num-encoder-layers=2,2,2,3,2,2
downsampling-factor=1,2,4,8,4,2
feedforward-dim=256,384,512,768,512,384
num-heads=4,4,4,4,4,4
encoder-dim=192,256,256,384,256,256
query-head-dim=24
value-head-dim=8
pos-head-dim=4
pos-dim=24
encoder-unmasked-dim=192,192,256,256,256,192
cnn-module-kernel=31,31,15,15,15,31
decoder-dim=256
joiner-dim=256

Could you tell us which script you are using?
Have you changed any code or just used the original code from us?
Also, please tell us whether you are using a streaming or a non-streaming model and what is the typical wave duration of your test file.
It would be great if you can post the complete decoding command.

Could you tell us which script you are using?
Have you changed any code or just used the original code from us?
Also, please tell us whether you are using a streaming or a non-streaming model and what is the typical wave duration of your test file.
It would be great if you can post the complete decoding command.

Hi @csukuangfj,
I have export the model using torch.jit.export and write my one decode code, which is used in offline scenario. However, the core decoding code is from icefall.
Actually my wave is usually long than 1 minite, but the max duration of wave file for asr is 20s (which is force cut by energy vad).
I have attached code below.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import logging
import math
from argparse import ArgumentParser
from pathlib import Path

import kaldifeat
import sentencepiece as spm
import torch
import torchaudio
from beam_search import greedy_search_batch
from icefall.lexicon import Lexicon
from pydub import AudioSegment
from torch.nn.utils.rnn import pad_sequence

_LOGGER = logging.getLogger(__name__)

class Decoder:

    def __init__(self, lang_dir):
        self._sp = spm.SentencePieceProcessor()
        self._sp.load(str(lang_dir / 'bpe.model'))

        self._lexicon = Lexicon(lang_dir)

        self.blank_id = self._lexicon.token_table['<blk>']
        self.vocab_size = max(self._lexicon.tokens) + 1
        _LOGGER.info('Decoder Init Succeed.')

    @property
    def lexicon(self):
        return self._lexicon

    def decode_tokens(self, tokens):
        token_table = self._lexicon.token_table
        return self._sp.decode([token_table[idx] for idx in tokens])

    def decode(self, model, encoder_out, encoder_out_lens):
        pred = list()
        hyp_tokens = greedy_search_batch(
            model=model,
            encoder_out=encoder_out,
            encoder_out_lens=encoder_out_lens,
        )
        for i in range(encoder_out.size(0)):
            pred.append(self.decode_tokens(hyp_tokens[i]))
        return pred

class Asr:

    def __init__(self, model_dir):
        torch.set_num_threads(1)
        torch.set_num_interop_threads(1)
        model_dir = Path(model_dir)
        self._device = torch.device('cpu')
        self._feat_extractor = self.__get_feat_extractor()
        self._model = self._init_model(model_dir)
        self._decoder = Decoder(model_dir / 'lang',)
        _LOGGER.info('Asr Model init succeed.')

    def __get_feat_extractor(self):
        opts = kaldifeat.FbankOptions()
        opts.device = self._device
        opts.frame_opts.dither = 0
        opts.frame_opts.snip_edges = False
        opts.frame_opts.samp_freq = 8000
        opts.mel_opts.num_bins = 80
        feat_extractor = kaldifeat.Fbank(opts)
        _LOGGER.info('Fbank feat extractor init succeed.')
        return feat_extractor

    def _init_model(self, model_dir):
        model = torch.jit.load(model_dir / 'model.pt')
        model.eval()
        model.to(self._device)
        _LOGGER.info('ZipFormer Load succeed.')
        return model

    @staticmethod
    def _read_wav_files(wav_files):
        waves = list()
        for wav_file in wav_files:
            wave, sample_rate = torchaudio.load(wav_file)
            waves.append(wave[0].contiguous())
        return waves

    def _get_feature(self, wav_files):
        waves = self._read_wav_files(wav_files)
        waves = [w.to(self._device) for w in waves]

        features = self._feat_extractor(waves)
        features = pad_sequence(
            features,
            batch_first=True,
            padding_value=math.log(1e-10)
        )

        feature_lengths = torch.tensor(
            [f.size(0) for f in features], device=self._device
        )
        return features, feature_lengths

    def _encode(self, features, feature_lengths):
        """编码.

        Args:
            features: 特征.
            feature_lengths: 特征长度.

        Returns:
            编码输出和编码输出长度.
        """
        encoder_out, encoder_out_lens = self._model.encoder(
            features=features,
            feature_lengths=feature_lengths
        )
        return encoder_out, encoder_out_lens

    def recognize(self, wav_files):
        features, feature_lengths = self._get_feature(wav_files)
        encoder_out, encoder_out_lens = self._encode(features, feature_lengths)
        texts = self._decoder.decode(self._model, encoder_out, encoder_out_lens)
        return texts

def _main():
    parser = ArgumentParser('recognize')
    parser.add_argument('model_dir', type=Path, help='model directory')
    parser.add_argument('wav_dir', type=Path, help='wav directory')
    args = parser.parse_args()

    asr = Asr(args.model_dir)
    for wav_file in args.wav_dir.iterdir():
        duration = AudioSegment.from_file(wav_file).duration_seconds
        text = asr.recognize([wav_file])[0]
        _LOGGER.info(f'[{wav_file}]: [{duration:.3f}s] [{text}]')

if __name__ == '__main__':
    LOGGER_FORMAT = ('%(asctime)s.%(msecs)03d - %(name)s:%(lineno)s '
                     '- %(funcName)s() - %(levelname)s - %(message)s')
    logging.basicConfig(format=LOGGER_FORMAT, level=logging.INFO)
    _main()

I see.
Please add
@torch.no_grad()
as we are doing in decoding.

Could you post your updated code?

@csukuangfj
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import logging
import math
from argparse import ArgumentParser
from pathlib import Path

import kaldifeat
import sentencepiece as spm
import torch
import torchaudio
from beam_search import greedy_search_batch
from icefall.lexicon import Lexicon
from pydub import AudioSegment
from torch.nn.utils.rnn import pad_sequence

_LOGGER = logging.getLogger(__name__)

class Decoder:

    def __init__(self, lang_dir):
        self._sp = spm.SentencePieceProcessor()
        self._sp.load(str(lang_dir / 'bpe.model'))

        self._lexicon = Lexicon(lang_dir)

        self.blank_id = self._lexicon.token_table['<blk>']
        self.vocab_size = max(self._lexicon.tokens) + 1
        _LOGGER.info('Decoder Init Succeed.')

    @property
    def lexicon(self):
        return self._lexicon

    def decode_tokens(self, tokens):
        token_table = self._lexicon.token_table
        return self._sp.decode([token_table[idx] for idx in tokens])

    def decode(self, model, encoder_out, encoder_out_lens):
        pred = list()
        hyp_tokens = greedy_search_batch(
            model=model,
            encoder_out=encoder_out,
            encoder_out_lens=encoder_out_lens,
        )
        for i in range(encoder_out.size(0)):
            pred.append(self.decode_tokens(hyp_tokens[i]))
        return pred

class Asr:

    def __init__(self, model_dir):
        torch.set_num_threads(1)
        torch.set_num_interop_threads(1)
        model_dir = Path(model_dir)
        self._device = torch.device('cpu')
        self._feat_extractor = self.__get_feat_extractor()
        self._model = self._init_model(model_dir)
        self._decoder = Decoder(model_dir / 'lang',)
        _LOGGER.info('Asr Model init succeed.')

    def __get_feat_extractor(self):
        opts = kaldifeat.FbankOptions()
        opts.device = self._device
        opts.frame_opts.dither = 0
        opts.frame_opts.snip_edges = False
        opts.frame_opts.samp_freq = 8000
        opts.mel_opts.num_bins = 80
        feat_extractor = kaldifeat.Fbank(opts)
        _LOGGER.info('Fbank feat extractor init succeed.')
        return feat_extractor

    def _init_model(self, model_dir):
        model = torch.jit.load(model_dir / 'model.pt')
        model.eval()
        model.to(self._device)
        _LOGGER.info('ZipFormer Load succeed.')
        return model

    @staticmethod
    def _read_wav_files(wav_files):
        waves = list()
        for wav_file in wav_files:
            wave, sample_rate = torchaudio.load(wav_file)
            waves.append(wave[0].contiguous())
        return waves

    def _get_feature(self, wav_files):
        waves = self._read_wav_files(wav_files)
        waves = [w.to(self._device) for w in waves]

        features = self._feat_extractor(waves)
        features = pad_sequence(
            features,
            batch_first=True,
            padding_value=math.log(1e-10)
        )

        feature_lengths = torch.tensor(
            [f.size(0) for f in features], device=self._device
        )
        return features, feature_lengths

    def _encode(self, features, feature_lengths):
        """编码.

        Args:
            features: 特征.
            feature_lengths: 特征长度.

        Returns:
            编码输出和编码输出长度.
        """
        encoder_out, encoder_out_lens = self._model.encoder(
            features=features,
            feature_lengths=feature_lengths
        )
        return encoder_out, encoder_out_lens

    def recognize(self, wav_files):
        features, feature_lengths = self._get_feature(wav_files)
        encoder_out, encoder_out_lens = self._encode(features, feature_lengths)
        texts = self._decoder.decode(self._model, encoder_out, encoder_out_lens)
        return texts

@torch.no_grad()
def _main():
    parser = ArgumentParser('recognize')
    parser.add_argument('model_dir', type=Path, help='model directory')
    parser.add_argument('wav_dir', type=Path, help='wav directory')
    args = parser.parse_args()

    asr = Asr(args.model_dir)
    for wav_file in args.wav_dir.iterdir():
        duration = AudioSegment.from_file(wav_file).duration_seconds
        text = asr.recognize([wav_file])[0]
        _LOGGER.info(f'[{wav_file}]: [{duration:.3f}s] [{text}]')

if __name__ == '__main__':
    LOGGER_FORMAT = ('%(asctime)s.%(msecs)03d - %(name)s:%(lineno)s '
                     '- %(funcName)s() - %(levelname)s - %(message)s')
    logging.basicConfig(format=LOGGER_FORMAT, level=logging.INFO)
    _main()

Does the memory grow linearly from 0 to 2.1GB and then keep at 2.1 GB?

_LOGGER.info(f'[{wav_file}]: [{duration:.3f}s] [{text}]')

Could you give the output of the above log? What is the max value of duration?

Does the memory grow linearly from 0 to 2.1GB and then keep at 2.1 GB?

_LOGGER.info(f'[{wav_file}]: [{duration:.3f}s] [{text}]')

Could you give the output of the above log? What is the max value of duration?

The max duration is 20s in my wav files.
The memory first grow to around 1.5G for the first few wavs, then grow slowly to 2.1G and keep at 2.1G.
The first few wavs (~5 wavs) is extremely slow, may cost 1~2 minute to finish decode. I'm not sure if it's normal that the warm up for this asr model need this time.


Unit test using the wrong database

Hi Jimmy,
When running the unit tests they seem to be connecting to ContosoUniversityDotNetCore-Pages database instead of ContosoUniversityDotNetCore-Pages-Test?
Is this a know issue?

This was resolved in the switch over to using psake as the new build script. I noticed the testing csproj file was not doing the right thing. @jbogard you can close this issue.


Parallelize importer

The importer tool converts one map at a time. Easy opportunity to make use of all CPUs. Simple change in one layer of the code. Makes logging harder to read, but eh. Only trick is that ensure_popdat_exists has to be called before the parallelization starts.

This is the primary relevant file for this issue right?
abstreet/importer/src/main.rs

Are there any others I should be mindful of as well? @dabreegster

That's the main one. The other modules in importer are worth a look too, but hopefully they're pretty simple. I think the only easy opportunity for parallelization is the for name in names loop.
The only potential race conditions would be attempting to produce the same file at the same time. This could happen by two callers trying to do seattle::ensure_popdat_exists, since this first fully builds the huge_seattle map and then produces a file from it. I guess within each of the per-city modules, there's an input() function that downloads all input files needed -- we also don't want to duplicate that work at the same time. Ah, and the utility download always produces tmp_output before renaming; probably have to deal with that. Maybe this is a little more complex than I first estimated.
(By the way, if you know of a simple way to express a dependency graph of tasks and be idempotent / detect work that's already been done, there may be a much better way to express everything in importer.)

Okay, so from looking at it seems like the only jobs that can be run in parallel within for name in names would be job.raw_to_map,job.scenario, and job.scenario_everyone. So, in line with that, I should only spawn threads for those jobs and leave job.osm_to_raw to run sequentially given those concerns you've expressed about the input() function. So, with that in mind, the solution I am considering is the following:
    let mut was_ensure_popdate_exists_called = false;
    for name in names {
        //Leave job.osm_to_raw alone
        if job.osm_to_raw {
            match job.city.as_ref() {
                "austin" => austin::osm_to_raw(&name),
                "los_angeles" => los_angeles::osm_to_raw(&name),
                "seattle" => seattle::osm_to_raw(&name),
                x => panic!("Unknown city {}", x),
            }
        }

        //Spawn a thread to manage this
        if job.raw_to_map {
            utils::raw_to_map(&name, job.use_fixes);
        }

        //Spawn a
        if job.scenario {
            assert_eq!(job.city, "seattle");
            //make sure ensure_popdat_exists is only called once
            if !was_ensure_popdat_exists{
                seattle::ensure_popdat_exists(job.use_fixes);
                was_ensure_popdat_exists = true;
            }

            //Spawn a thread to manage this
            let mut timer = abstutil::Timer::new(format!("Scenario for {}", name));
            let map = map_model::Map::new(abstutil::path_map(&name), job.use_fixes, &mut timer);
            soundcast::make_weekday_scenario(&map, &mut timer).save();
        }

        if job.scenario_everyone {
            assert_eq!(job.city, "seattle");
            if !was_ensure_popdat_exists_called{
                seattle::ensure_popdat_exists(job.use_fixes);
            }

            //Spawn a thread to manage this
            let mut timer = abstutil::Timer::new(format!("Scenario for {}", name));
            let map = map_model::Map::new(abstutil::path_map(&name), job.use_fixes, &mut timer);
            soundcast::make_weekday_scenario_with_everyone(&map, &mut timer).save();
        }
    }
//Wait for all threads to complete

This should almost work. The only problem is that ensure_popdat_exists may call raw_to_map, so there may be a case when two threads are both doing raw_to_map(huge_seattle) at the same time. That's technically correct, provided writing the output file is atomic.
I haven't done much parallelism in Rust before, but it could be nice to use the type system to statically enforce correctness. What if we had some marker type that does not implement Sync to prevent accidentally calling osm_to_raw concurrently?
Also, the generic dependency graph executor I was thinking of is https://github.com/salsa-rs/salsa. Probably overkill for this tool, though.

Agreed that the dependency graph executor is probably a bad move. Plus, it seems unstable and I imagine adding unstable dependencies wouldn't be healthy for this project. I'll take a stab at a marker type but other than I'll look to execute what I've said already.

Maybe I'm missing something but I keep getting the following compile error on the latest revision of the repo:
error[E0599]: no associated item named `MAX` found for type `f64` in the current scope
  --> geom/src/bounds.rs:16:25
   |
16 |             min_x: f64::MAX,
   |                         ^^^ associated item not found in `f64`
   |
   = help: items from traits can only be used if the trait is in scope
   = note: the following trait is implemented but not in scope; perhaps add a `use` for it:
           `use rand::distributions::weighted::alias_method::Weight;`
help: you are looking for the module in `std`, not the primitive type
   |
16 |             min_x: std::f64::MAX,
   |                    ^^^^^^^^^^^^^

error[E0599]: no associated item named `MAX` found for type `f64` in the current scope
  --> geom/src/bounds.rs:17:25
   |
17 |             min_y: f64::MAX,
   |                         ^^^ associated item not found in `f64`
   |
   = help: items from traits can only be used if the trait is in scope
   = note: the following trait is implemented but not in scope; perhaps add a `use` for it:
           `use rand::distributions::weighted::alias_method::Weight;`
help: you are looking for the module in `std`, not the primitive type
   |
17 |             min_y: std::f64::MAX,
   |                    ^^^^^^^^^^^^^

error[E0599]: no associated item named `MIN` found for type `f64` in the current scope
  --> geom/src/bounds.rs:18:25
   |
18 |             max_x: f64::MIN,
   |                         ^^^ associated item not found in `f64`
   |
help: you are looking for the module in `std`, not the primitive type
   |
18 |             max_x: std::f64::MIN,
   |                    ^^^^^^^^^^^^^

error[E0599]: no associated item named `MIN` found for type `f64` in the current scope
  --> geom/src/bounds.rs:19:25
   |
19 |             max_y: f64::MIN,
   |                         ^^^ associated item not found in `f64`
   |
help: you are looking for the module in `std`, not the primitive type
   |
19 |             max_y: std::f64::MIN,
   |                    ^^^^^^^^^^^^^

error[E0599]: no associated item named `MAX` found for type `f64` in the current scope
  --> geom/src/bounds.rs:96:27
   |
96 |             min_lon: f64::MAX,
   |                           ^^^ associated item not found in `f64`
   |
   = help: items from traits can only be used if the trait is in scope
   = note: the following trait is implemented but not in scope; perhaps add a `use` for it:
           `use rand::distributions::weighted::alias_method::Weight;`
help: you are looking for the module in `std`, not the primitive type
   |
96 |             min_lon: std::f64::MAX,
   |                      ^^^^^^^^^^^^^

error[E0599]: no associated item named `MAX` found for type `f64` in the current scope
  --> geom/src/bounds.rs:97:27
   |
97 |             min_lat: f64::MAX,
   |                           ^^^ associated item not found in `f64`
   |
   = help: items from traits can only be used if the trait is in scope
   = note: the following trait is implemented but not in scope; perhaps add a `use` for it:
           `use rand::distributions::weighted::alias_method::Weight;`
help: you are looking for the module in `std`, not the primitive type
   |
97 |             min_lat: std::f64::MAX,
   |                      ^^^^^^^^^^^^^

error[E0599]: no associated item named `MIN` found for type `f64` in the current scope
  --> geom/src/bounds.rs:98:27
   |
98 |             max_lon: f64::MIN,
   |                           ^^^ associated item not found in `f64`
   |
help: you are looking for the module in `std`, not the primitive type
   |
98 |             max_lon: std::f64::MIN,
   |                      ^^^^^^^^^^^^^

error[E0599]: no associated item named `MIN` found for type `f64` in the current scope
  --> geom/src/bounds.rs:99:27
   |
99 |             max_lat: f64::MIN,
   |                           ^^^ associated item not found in `f64`
   |
help: you are looking for the module in `std`, not the primitive type
   |
99 |             max_lat: std::f64::MIN,
   |                      ^^^^^^^^^^^^^

error: aborting due to 8 previous errors

For more information about this error, try `rustc --explain E0599`.
error: could not compile `geom`.

I upgraded to Rust 1.43 on April 23, which made those imports unnecessary. Try rustup update stable


"Stealth" Deletion method

Hi there! Amazing library, absolutely loved it. I've noticed there's no mention of DeletionEvents nor how they behave in the wiki, however?
Simply requesting the registerDeletionEvent method and the DeletionEvent Functional Interface's behaviors to be explained.

ah yes! I forgot to update the wiki... I've have it up within the hour (hopefully not in two)

I have updated the wiki to V:0.1.0. Feel free to reopen this issue if there are any other issues


Write acceptance criteria for U01

@mortenwh commented on Fri Mar 10 2023

GK: First attempt (please add/adjust)

sub-hourly observations of at least the 3 project members (FMI, MetNo and KNMI) are available within 10? minutes after the observation has been made
the sub-hourly observations can be retrieved via an OGC compliant API
all features of the API data retrieval are documented in publicly available HTML pages
The API endpoint has at least the following query parameters:
[ToDo]
The user can specify the observations for a certain area via a bounding box


Softdevice.run_with_callback never calls the callback

I've been tryingg to get SocEvents by calling run_with_callback. However, not event  arrives. I've been able to debug, and in events#run_soc the error seems to be NotFound (5). I'm testing this with nrf52840 and S140.

NotFound just means there are no SOC events available at that point in time. You do need to configure the softdevice to generate the events you're interested in. For example, to get USB power events you need to call:
            use nrf_softdevice::raw;
            raw::sd_power_usbremoved_enable(1);
            raw::sd_power_usbdetected_enable(1);
            raw::sd_power_usbpwrrdy_enable(1);


Include git commit info in version number

To indicate that the deployed versions can differ between different environments, even if the version number can be the same, we should include the git hash in the version number, and even link to the commit in this repo.
To bake it into the app, we need to update the Dockerfile to use an argument like so:
docker build --build-arg COMMIT=$(git rev-parse HEAD)

In the Dockerfile, we expose it to the app with NEXT_PUBLIC_COMMIT, so it gets inlined during build
Then we can display it in the app like so:

v3.1.1 (https://github.com/visualize-admin/visualization-tool/commit/8e64d532cdac3992d8dd927d3ca0d77805a5d20f)

And we can link to github directly from the commit sha1 :)

@ptbrowne yes, that's what I suggested ;) The only question to me is if we should do this regardless of env (dev/int/prod) but I'd say yes, to reduce complexity

Yes, I do not think it causes a problem right now.


Trading bots integrated into SE-2

Created new package trading-bots.
The trading bots now use the deployed contracts data, the scaffold config file and the tokens config file from the nextjs package.
The bots can be run with yarn trading:setup and yarn trading:trade [tokenName].
More info at the README file.

Looks good


How does LGM support fp16 training for diff-gaussian-rasterization?

Hi! I wonder which code you revised to support fp16 training in the original diff-gaussian-rasterization?

Hi, have you figure it out?

In fact, their diff-gaussian-rasterization does not support fp16. they use float32 in this module and fp16 in their neural networks.


Create account roles - hide hosted-cp flag

Hide the flag until AWS managed policies are in place.
Related: SDA-8498

@zgalor @tbrisker PTAL

/lgtm


Exposing actual test failure in case of UNDERTOW-884

Original test was passing whether there was or wasn't included fix for
UNDERTOW-884 in io.undertow.security.impl.FormAuthenticationMechanism.
This change enhances the test to fail in case of wrong location header
@stuartwdouglas if you are OK with this change, please put it also to 1.4.x and 1.3.x branches.

Windows Build 2005 outcome was FAILURE using a merge of b760d97b91d6248d3557ebae0fbf957bb540114e
Summary: Execution timeout; tests failed: 1, passed: 1640, ignored: 518 Build time: 00:45:04
Failed tests
io.undertow.server.handlers.ReceiverTestCase.testAsyncReceivePartialBytes: java.lang.RuntimeException: test failed with i equal to 6
	at io.undertow.server.handlers.ReceiverTestCase.doTest(ReceiverTestCase.java:233)
	at io.undertow.server.handlers.ReceiverTestCase.testAsyncReceivePartialBytes(ReceiverTestCase.java:200)
Caused by: java.net.SocketTimeoutException: Read timed out
	at io.undertow.server.handlers.ReceiverTestCase.runTest(ReceiverTestCase.java:246)
	at io.undertow.server.handlers.ReceiverTestCase.doTest(ReceiverTestCase.java:231)
	at io.undertow.server.handlers.ReceiverTestCase.testAsyncReceivePartialBytes(ReceiverTestCase.java:200)

The windows failures doesn't seem to be related to this PR. The test failure is known failure: https://issues.jboss.org/browse/UNDERTOW-989


Numpy version is too old

Hey @jeff-regier , you limited Numpy version in #209 (https://github.com/YosefLab/scVI/blob/66cc63ead0d1a219f8d0ba1018d6f610f2b91679/setup.py#L13) but there are useful new functionalities in 1.15 and 1.16
We should require at least numpy 1.16 in setup.py

Sure, that's fine. Versions later than 1.14 printed out a lot of warnings back when I put that limit on numpy, but perhaps that won't happen any more


Integrating vis-timeline

It would great to integrate vis-timeline into this package.
https://github.com/visjs/vis-timeline

I just released the Streamlit Timeline component with bidirectional communication.
https://github.com/giswqs/streamlit-timeline


Application message push support fcm, HMS push

Is your feature request related to a problem? Please describe.
Log management statistics
Describe the solution you'd like
Optimize application push
Describe alternatives you've considered
Android version integrated fcm push
Android version integrated HMS push
App push to reach the RBI
App users report the model and whether they support FCM push when logging in
Background push strategy modification:
Huawei models give priority to hms push
Mi models give priority to MI PUSH
Other overseas Android models are given priority in FCM push
Additional context
Add any other context or screenshots about the feature request here.

Issue range is too large, split into # 36, # 37 two issues


sap.m.ScrollContainer & .Page: missing focus indicator for scroll content

URL (minimal example if possible)
E.g.: https://openui5nightly.hana.ondemand.com/entity/sap.m.Page/sample/sap.m.sample.PageFloatingFooter
Steps to reproduce the problem

Open the above sample page
With the keyboard, Tab through elements until it reaches the page

What is the expected result?
Since the scroll content is an interactive element (scrollable with arrows, Home, End, etc.), the focus indicator should be visible there too, according to WCAG 2.1 Focus Visible SC 2.4.7 (Level AA):

Any keyboard operable user interface has a mode of operation where the keyboard focus indicator is visible.

What happens instead?

UI5 outline style is not applied to the scroll content. Instead, we see UA's default focus indicator.
When focused with a mouse, there is no focus indicator at all which is inconsistent with https://github.com/SAP/openui5/issues/1289#issuecomment-271235357.

Any other information? (attach screenshot if possible)
In Chrome, for example, we see a blue outline:

A similar issue can be observed with sap.m.ScrollContainer.

Another case:

Open https://openui5nightly.hana.ondemand.com/controls (The focus is set on the search field).
Press Tab three times.

Now the user needs to guess where the focus went.

Never mind. Just realized that I had a flag enabled in Chrome which caused this issue. Sorry!

Ok, the issue is actually reproducible in clean-installed stable Firefox (72.0, Windows). FF behaves like Chromium-based browsers with #enable-experimental-web-platform-features, just without the blue focus indicator when the scroll area is focused.

Hi boghyon,
The Page from the sample is a container that does not receive the focus. Like many other containers, it is just a div in the DOM, not focusable by default. Same as the page here, many posts, but the body does not receive the focus, or have an indicator. It reacts to the arrow down events, because they bubble through all non-scrollable children. So we could say that it does not need the focus to be interactive.
E.g. The focus is on the 'Comment' button when I hold arrow up and the page scrolls.
When the focus is on the text area the cursor moves inside and the page does not scroll.
A sap.m.ScrollContainer, on the other hand, is a container designed to receive the focus on demand. Its API includes a property 'focusable'.
Regards,
Alex

Hi Alex, thanks for the response.
It's true that divs shouldn't be focusable technically. But what's weird is that Firefox requires an additional Tab press to reach the [Toggle floating footer] button whereas other browsers don't.
This leads to the issue mentioned in https://github.com/SAP/openui5/issues/2785#issuecomment-571585675 where FF users lose the track of focus indicator when tabbing through, since the scroll area itself gets the focus beforehand without any indicator. In other browsers, it's always clear where the focus indicator is.
I guess this is a FF issue. Not sure if UI5 should do anything about it.

There is some more info. I think we need to workaround this table, to make the experience the same in all browsers:
https://allyjs.io/data-tables/focusable.html
Our element is an overflowing
It's documented as 'tabbable' in FF.

Hello @boghyon ,
We've created an internal incident<PHONE_NUMBER> to investigate this problem. The status of the issue will be updated here in GitHub.
Regards,
Iliana

Hello @boghyon ,
Seems like the issue is fixed by the linked above change. Therefore I am closing the issue.
Best regards,
Tereza


Enhancement: run builds on docker

This should provide more resources and a little speed-up.

Do they start faster? Running time seems about same

@mroderick for reference: http://blog.travis-ci.com/2014-12-17-faster-builds-with-container-based-infrastructure/


Crash with Svelte 5

So, Svelte 5 is now at RC. then I tried some taste of it.
I add v5 my project:
λ npm install --save-dev svelte@next
λ npm list
<EMAIL_ADDRESS>SvelteNEXT
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>
then Sublime Text triggers:

Then the LSP log :
LSP-svelte: Error: Cannot find module 'D:\xxxx\_SvelteNEXT\node_modules\svelte\compiler.cjs

And yes, Svelte 5 deosnt have that compiler.cjs file.
Svelte 4 running just fine.
ENV:

WIndows x64
Sublime Text 4
LSP Svelte v3,10

I will try to find time to see how to address this.
Thanks for the report.

this PR is related https://github.com/sveltejs/language-tools/pull/2327

I created a release which includes the fix https://github.com/sublimelsp/LSP-svelte/releases/tag/v3.1.1
it will be available in ~2h or less on Package Control

wohoo, Svelte 5 support works like a charm . Thank you @predragnikolic


Update auth, dynamodb, s3, sqs to 2.17.268

Updates

software.amazon.awssdk:auth
software.amazon.awssdk:dynamodb
software.amazon.awssdk:s3
software.amazon.awssdk:sqs

from 2.17.193 to 2.17.268.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "software.amazon.awssdk" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "software.amazon.awssdk" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #954.


All done.


build: update node versions

prevent release job from running on forks
cache npm modules

:tada: This PR is included in version 1.11.0 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


on funding reuse, Bob never creates Lock transaction

if you abort a swap post-funding and then restart with reuse of the funding address, on the subsequent attempt, it won't move to Lock. This appears to be due to a race condition:

When swapd receives FundingUpdated, it checks whether the pending request already exists
the pending request is only added after Msg::Reveal(Reveal::Proof(_)) has been received.

Solution: syncer should correctly ignore funding txs that were created prior to it being instantiated.

there is an option on the syncer, for the earliest block to be greater than some block, and its passed, but the syncer seems to ignore it @TheCharlatan

I just added the option today in monero-rpc-rs to select transfer by block height. Am patching the code now to make use of it. Will then work on something similar for Bitcoin as well.

reminder: on swapd there is some lookback weirdness i added at some point that probably should be removed after @TheCharlatan completes his work

As of #167 , the height argument for the WatchAddress task is now respected. I also opened #174 to save the counter to the node secrets file. This way, we won't derive the same address twice. #174 needs more follow-up work though, to increment only where appropriate.


Feature in memory repository data protected

Why do you need this change?

sorry for long response...
I've inherited this class and changed it for working with auto increment id (maybe I should not use auto increment id at all, but ...), with 'protected' I should not duplicate methods that are valid in this case.
If it's contradict with your private/protected agreement, just cancel this PR

@vladylen I'm closing this one.


[Events] Add missing actor label in eventActorShow.js

I was looking at the code to get some understanding of GBVM and noticed eventActorShow was missing a label as opposed to eventActorHide. I just added it back.

Good spot, thanks!


Update Travis Config to Follow Stable and LTS only

Was local issue w/ deps and running test from inside Ubuntu BASH on Windows 10; resolved.


Pickup dispatch sample app when logout option given as no it navigates back to samples login page

How to reproduce:

Set up pickup dispatch sample and deploy the war in tomcat
Register a oidc service provider with callback url http://localhost:8080/pickup-dispatch/oauth2client
Add the client id and secret to dispatch.properties file
Access http://localhost:8080/pickup-dispatch/oauth2client
Login as admin:admin or with a multi attribute login (mobile)
Click on Logout
From the popup to confirm logout select No

Expected behavior:
You should be landed back to home page
Actual behavior
You will get navigated back to login page
Environment information (Please complete the following information; remove  any unnecessary fields) :
IS 5.12.0 alpha 16
https://user-images.githubusercontent.com/31848014/159537854-3190fc32-a75f-40ba-aadf-2a049c73572e.mp4

@ShanikaWickramasinghe , when the app initiates logout request, the app session gets invalidated. When the user denies the consent for logout, only the IdP session will not be terminated. so the user will be redirected to the post-logout-redirect-uri back. so this is expected.


Add min/max to regex keyword

Consider:
map:
  regex;(^db\.type):
    min: 1
...

Would be nice if we could specify the min/max constraints under a regex.  These would specify the minimum and maximum number of matches for that regex.  Before I take a stab at it, any initial thoughts?

I think this could be the syntax:
map:
  regex;(^foo\.):
    matches:
      min: 1
      max: 5
    ...

@cloudtoad Before i take a stab at this, i would like to have a small explanation whenever this feature is usefull. I do not see a good real world case where this would make sense to have. Do you have any real world schema/data files where you want to use this feature? I am on the fence of implementing this feature, please convince me :)

@cloudtoad Pushing this one into the future and release 1.7.0 becuase i have some new ideas on how to implement regex schema syntax that will require some restructuring but it could open up for implmenting this. I do not want to add this tot he next release becuase that one is due to be released very shortly.

@cloudtoad I am closing this one. Please open a new feature request with the new issue templates i have created that will add in all the text/questions i had above that has not been answered.


Issue with VisNER

first one it says it requires updated Nodejs. I resolved it.
after that I tried in Firefox, Edge browser it is not working.
but in chrome the fronted opened.
but when I input some text it is not giving any output. please check

Its working now, Thanks


API versioning

Right now, we aren't considering the Odoo version at all.
Because of that:

We can't use the type system to guarantee that a particular API method is available.
We can't have multiple versions of an API endpoint (e.g., if Odoo changed the call_kw method slightly)

I'm not actually sure this would be a desirable feature. A few downsides would be:

Much more rigid - users would need to store a version-typed Client object, or else use match {} blocks to cover all possible Odoo versions
More difficult to maintain
How would we handle SaaS versions (like 16.3 and 16.4 which are actually closer to 16.0 and 17.0 respectively)

It does seem like the Odoo APIs are very stable (perhaps because they're used internally), so maybe this is all fine.
One option might be some runtime-checks - that would eliminate issues 1 & 3 above.

If anyone has thoughts/ideas here, I'm keen to hear them!


[network] Split TransportManager and PeerManager #9032 - (105)

Motivation
To make some of the coding around TransportManager easier to work with, I've split out as much as I can out of PeerManager mod.rs without having to refactor the PeerManager code. This has reduced the mod.rs line size from 1300 to 800 lines. I plan on refactoring more pieces to make this a more manageable codebase and reduce some of the confusion.
###Test Plan
CI/CD tests are covered.

/canary

:sunny: Canary successful

/land


Migrate tests form JUnit asserts to Google Truth

Have started the implementation in the branch "https://github.com/asus200786/AIF2/tree/%23233-Migrate-tests-form-JUnit-asserts-to-Google-Truth"


at_c: cram onboarding

Communicate registrar to get a new atSign and cut new set of .atKeys
Requires #74

Thank you for completing this @srieteja +1


yosys: 0.38 -> 0.40

Description of changes
https://github.com/YosysHQ/yosys/releases/tag/yosys-0.39
This update breaks yosys-synlig. I'm marking this PR as draft in the hopes that it can be fixed soon @hzeller @nbdd0121.
https://github.com/YosysHQ/yosys/releases/tag/yosys-0.40
Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[x] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Let me have a look, I think upstream synlig dealt with this issue; I'll update this thread in a few hours.

Ok, I confirmed that latest synlig will not yet work with Yosys 0.39.
To not hold up this Yosys release, I am fine if synlig is marked broken for now while I work out what needs to be done to get it compatible with 0.39.

I marked yosys-synlig as broken.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/3752

Updated to Yosys 0.40.

This builds and passes tests on x86_64-linux and aarch64-darwin. Found that glasgow does not build on macOS due to a test failure due to yosys synthesizing slightly less efficiently. It probably needs to be upgraded, and/or that test skipped.

I've created #318592 so that yosys upgrades are not shackled by the slow update of that plugin.
Need someone to review and merge that one.

This review should get finished. In the meantime Yosys is already at 0.42.
I am sorry if the Synlig confusion slowed this down.

Please update to the latest Yosys and I will review.

I can merge when this is ready. The new amaranth bump i have in the works requires at least yosys version 0.40

i ended up playing around with this, and made this updater for abc:
  # updates to the latest tag, but the tags in this fork are unofficial
  # and the git ref is needed by yosys
  passthru.updateScript = writeShellScript "update-abc" ''
    set -euo pipefail
    repo=${finalAttrs.src.owner}/${finalAttrs.src.repo}
    rev="$(${lib.getExe gh} api repos/$repo/tags --jq '.[0].commit.sha')"
    date="$(${lib.getExe gh} api repos/$repo/commits/$rev --jq '.commit.committer.date' | cut -dT -f1)"
    ${lib.getBin common-updater-scripts}/bin/update-source-version ''${UPDATE_NIX_ATTR_PATH:-abc-verifier} 0-unstable-$date --rev=$rev "$@"
  '';

superseeded by https://github.com/NixOS/nixpkgs/pull/322306


Allow konflux images with oci index, be used in osbs

STONEBLD-2896

Maintainers will complete the following section

[ ] Commit messages are descriptive enough
[ ] Code coverage from testing does not decrease and new code is covered
[ ] Python type annotations added to new code
[ ] JSON/YAML configuration changes are updated in the relevant schema
[ ] Changes to metadata also update the documentation for the metadata
[ ] Pull request has a link to an osbs-docs PR for user documentation updates
[ ] New feature can be disabled from a configuration file

mypy is failing

mypy is failing

well that is ci issue

mypy is failing

fixed

Commit message could describe better what is changed and why


interact with chess binary

i found your BU Nim package, which might be useful for my coding idea : a terminal script to communicate with 'stockfish', a well-known chess engine binary, using a thread (or 2) .. i tried several solutions, but i can't get it to work : just give a UCI (protocol) command by stdin and catch its output by stdout.
i'd not expect this to be hard ..
can you supply a basic script for this ? Or point me to some function / example ?
i use Nim 2.0.6
i'm on Xubuntu 22.04

I am really unsure how much I can help you.  I am a little familiar with chess but haven't played in decades.  I'm not very familiar with UCI.  What you describe seems more like a text protocol with simple parsing & printing (& whatever else you are doing on the outside of your stockfish).  So, I agree this should not be hard, and Nim does make things like parsing / printing simple text things easy, just with the Nim stdlib.
However, what kind of data structure / type you want to load from stdin & print to stdout (or maybe vice-versa) probably depends on the UI or analysis or whatever you are trying to build and you haven't said much about that.    So, this is not really the best place to discuss this or general Nim programming.  I am also traveling right now. So, my ability to reply in detail is limited.  The basic stub of your logic might be something like:
type
  Extra = enum Castle # , etc., etc.
  BoardCoord = tuple[x,y: int]
  ChessMove = tuple[frm, to: BoardCoord; aux: enum Extra]
parseCoord(s: string) = discard # fill me in
var chMv: ChessMove
for line in lines stdin:
  if not parseCoord(chMv.frm, line, 0):
    stderr.write "bad line: ", line, "\n"
  if not parseCoord(chMv.to, line, 2):
    stderr.write "bad line: ", line, "\n"
  #... parseExtra etc., etc.
  # At some point the line is fully parsed and maybe do something else

Since this is not really some kind of "Unix" or "statistical data analysis" or other tool of broader generality than the game of Chess, I am going to close this issue.  But you can ask more questions if you like.

i will give you the basics of running a chess engine binary (on linux here) with the UCI protocol : giving commands by stdin and catch its output by stdout.
Info UCI : see https://www.chessprogramming.org/UCI and https://github.com/fsmosca/UCIChessEngineProtocol
UCI has many features & functions, but in practice it simply goes like this (i use the open source stockfish (SF) engine, stable and very strong), see terminal display below.
first set usage of the UCI protocol by 'uci' : SF returns its '@@@' info lines (my indication) and these should end with 'uciok'.
then 'ucinewgame' initiates a new game, no response will be given.
then we set the starting position, no response will be given.
then we let the engine run with some max-time-per-game (w/b=white/black / 'inc'=time-INCrement-after-each-move) and it shows its continues output until the line 'bestmove [..]' is given.
that's all, this type of command sequence repeats for every move : the next position is derived by a command like 'position startpos move1 move2 move3 ...' - so, until the current position is reached.

$ stockfish
@@@ Stockfish 14.1 by the Stockfish developers (see AUTHORS file)
uci
@@@ id name Stockfish 14.1
@@@ id author the Stockfish developers (see AUTHORS file)
@@@ option name Debug Log File type string default
@@@ option name Threads type spin default 1 min 1 max 512
@@@ option name Hash type spin default 16 min 1 max 33554432
@@@ option name Clear Hash type button
@@@ option name Ponder type check default false
@@@ option name MultiPV type spin default 1 min 1 max 500
@@@ option name Skill Level type spin default 20 min 0 max 20
@@@ option name Move Overhead type spin default 10 min 0 max 5000
@@@ option name Slow Mover type spin default 100 min 10 max 1000
@@@ option name nodestime type spin default 0 min 0 max 10000
@@@ option name UCI_Chess960 type check default false
@@@ option name UCI_AnalyseMode type check default false
@@@ option name UCI_LimitStrength type check default false
@@@ option name UCI_Elo type spin default 1350 min 1350 max 2850
@@@ option name UCI_ShowWDL type check default false
@@@ option name SyzygyPath type string default <empty>
@@@ option name SyzygyProbeDepth type spin default 1 min 1 max 100
@@@ option name Syzygy50MoveRule type check default true
@@@ option name SyzygyProbeLimit type spin default 7 min 0 max 7
@@@ option name Use NNUE type check default true
@@@ option name EvalFile type string default nn-13406b1dcbe0.nnue
@@@ uciok

ucinewgame
position startpos
go wtime 70000 btime 70000 winc 10000 binc 10000
@@@ info string NNUE evaluation using nn-13406b1dcbe0.nnue enabled
@@@ info depth 1 seldepth 1 multipv 1 score cp 38 nodes 20 nps 10000 tbhits 0 time 2 pv d2d4
@@@ (--more--)
@@@ info depth 18 seldepth 21 multipv 1 score cp 39 nodes 552434 nps 525127 hashfull 232 tbhits 0 time 1052 pv d2d4 g8f6 g1f3 d7d5 c2c4 e7e6 b1c3 f8e7 c1g5 b8d7 e2e3 e8g8 a1c1 c7c6 d1c2 f6e4 g5e7 d8e7
... sinfo depth 19 seldepth 19 multipv 1 score cp 39 nodes 588356 nps 530050 hashfull 240 tbhits 0 time 1110 pv d2d4 g8f6 g1f3 d7d5 c2c4 e7e6 b1c3 f8e7 c1g5 b8d7 e2e3 e8g8 a1c1 c7c6 d1c2 f6e4 g5e7 d8e7 c4d5
... tinfo depth 20 seldepth 27 multipv 1 score cp 46 nodes 924634 nps 530180 hashfull 376 tbhits 0 time 1744 pv d2d4 d7d5 c2c4 e7e6 b1c3 g8f6 c1g5 f8b4 g1f3 h7h6 g5f6 d8f6 d1a4 b8c6 e2e3 e8g8 a1c1 b4c3 c1c3 e6e5
... opinfo depth 21 seldepth 27 multipv 1 score cp 45 nodes 1042162 nps 532258 hashfull 417 tbhits 0 time 1958 pv d2d4 d7d5 c2c4 e7e6 b1c3 g8f6 c1g5 f8e7 e2e3 e8g8 g1f3 b8d7 a1c1 h7h6 g5h4 c7c5 c4d5 f6d5 h4e7 d5e7 d4c5
@@@ info depth 22 seldepth 27 multipv 1 score cp 45 nodes 1338001 nps 532643 hashfull 519 tbhits 0 time 2512 pv d2d4
@@@ bestmove d2d4 ponder d7d5

go wtime 70000 btime 70000 winc 10000 binc 10000
@@@ info string NNUE evaluation using nn-13406b1dcbe0.nnue enabled
@@@ info depth 1 seldepth 1 multipv 1 score cp 45 nodes 31 nps 15500 tbhits 0 time 2 pv d2d4
@@@ (--more--)
@@@ info depth 23 seldepth 27 multipv 1 score cp 39 nodes 552434 nps 525127 hashfull 232 tbhits 0 time 1052 pv d2d4 g8f6 g1f3 d7d5 c2c4 e7e6 b1c3 f8e7 c1g5 b8d7 e2e3 e8g8 a1c1 c7c6 d1c2 f6e4 g5e7 d8e7
stop
@@@ info depth 24 seldepth 29 multipv 1 score cp 44 nodes 1920301 nps 513724 hashfull 714 tbhits 0 time 3738 pv e2e4
@@@ bestmove e2e4 ponder e7e5

in the above listing you can see me entering the command 'stop', best shown in the second run : i paste 'stop[enter]' in terminal and its feeded into stdin -- Note : the first run shows me entering 's','t','op' (the '...' lines) but then SF also gets the 'stop' command by stdin. Btw. SF is our reference, its logic and code are great.
('ponder' means the expected opponent reaction move)
creating my script idea in Python is not too hard, many examples exist, and even complete UCI modules / APIs .. but first i wanted to create a Nim GUI for the SF process (expecting the communication itself to be easy with threads), using the Owlkettle Widget builder for Nim : i opened an Issue there but my idea gave trouble along the way for 'the experts', pointing to some bug or short-coming of their Owlkettle or Nim or both .. also depending on versions, i don't remember exactly, but it isn't solved yet .. so, i might eventually use another "Widget kit" to build a GUI for it .. here's the Issue :
Text View widget : thread process can not change text buffer
https://github.com/can-lehmann/owlkettle/issues/145
i also had contact with Stefan Salewski, he's the one who wrote a very nice PDF about Nim programming, but lately he changed his mind (due to some bad experience in the Nim community) and turned to Rust, so he couldn't help me either !?
last thing for now : here are 2 "code pieces" i composed, which don't work, but they can give you the idea, i think .. honestly, ChatGPT helped here, but i'm stuck, being fully aware of this AI short-comings ..

import osproc, strutils, streams, os

# Function to run the Stockfish engine as a separate process
proc runStockfish() =
  let stockfishCmd = "stockfish"  # Assuming Stockfish is in the PATH
  var stockfishProcess = startProcess(stockfishCmd, options={poUsePath, poStdErrToStdOut})

  defer:
    close(stockfishProcess)

  # Send the initial UCI command
  stockfishProcess.input.writeLine("uci")
  stockfishProcess.input.flush()

  # Main loop to handle user input and engine output
  while true:
    if stdin.ready():  # Check if the user entered a command
      let userCommand = stdin.readLine().strip()
      if userCommand == "quit":
        stockfishProcess.input.writeLine("quit")
        stockfishProcess.input.flush()
        break
      stockfishProcess.input.writeLine(userCommand)
      stockfishProcess.input.flush()

    # Check and display the engine's output
    while stockfishProcess.output.peek() > 0:
      let engineOutput = stockfishProcess.output.readLine().strip()
      if engineOutput.len > 0:
        echo "Engine output: ", engineOutput

# Run the Stockfish engine
runStockfish()

import osproc, strutils, os, weave, selectors

# Task to read from Stockfish
proc readFromStockfish(stockfish: Process) =
  var selector = newSelector()  # Create a selector to manage non-blocking I/O
  selector.register(stockfish.outputStream)

  while stockfish.running:
    if selector.waitFor(1000):  # Wait for data to be ready, timeout in milliseconds
      let output = stockfish.outputStream.readLine()
      echo "Stockfish Output: ", output
      if output.contains("uciok"):
        echo "Stockfish is ready in UCI mode"
      elif output.contains("bestmove"):
        echo "Best move: ", output

# Task to write commands to Stockfish
proc writeToStockfish(stockfish: Process) =
  # Send UCI initialization command and other commands in a loop
  let commands = @["uci", "position startpos", "go depth 10"]
  for command in commands:
    stockfish.inputStream.writeLine(command)
    stockfish.inputStream.flush()  # Ensure the command is sent
    echo "Sent Command: ", command
    sleep(1000)  # Wait for a second between commands

# Main function to launch Stockfish and interact with it in parallel
proc main() =
  # Start the Stockfish engine
  let stockfish = startProcess("/path/to/stockfish", options={poUsePath},
                               stdin=stdinPipe, stdout=stdoutPipe)

  # Spawn tasks to interact with Stockfish
  spawn readFromStockfish(stockfish)
  spawn writeToStockfish(stockfish)

  # Run the weave event loop to process tasks
  runLoop()

  # Close the Stockfish process after tasks are done
  stockfish.close()

# Run the main function
main()

and what about the Nim package Weave, can it be of any use for me ?
Weave: A flexible parallelism framework in Nim that allows you to schedule and manage tasks and data-parallel work efficiently : https://github.com/mratsim/weave
btw. ChatGPT mentioned a few other Nim modules like this, but i couldn't get them working ..

I don't know anything about Owlkettle. This all sounded familiar to me and I guess I already gave you some advice in this Nim Forum thread about a year ago which I still think is the simplest way to go for a beginning programmer.  Though, as mentioned in the Forum, optimization to run stockfish as a coprocess rather than re-starting stockfish all the time is possible, it is also tricky/advanced and it sounds like you are struggling with basics.
To run the engine in parallel with the GUI, you could just send the output of stockfish to a file and then check every 10ms or so for the file growing to greater than zero size.  Then you don't need threads.  On Unix it could look as easy as popen("stockfish>/tmp/o &", "w") and then some writes to init & load the board state.  All you need is a way for your GUI to do something every some-number-of-milliseconds.  As I said, I don't know Owlkettle (nor do I want to learn right now), but this is pretty common in GUIs.  Relevant keywords here are "alarms", "timers", and maybe "timeout" (the last often means to cancel an operation like pondering rather than "continue to wait", but it can be the right API to use since the timeout might apply to "waiting for user-keyboard/mouse input" not the "waiting for stockfish" side).
Relevant for the doing "something" of my above paragraph would be std/os.getFileInfo("/tmp/o").size to check if stockfish is done pondering.  And then you need to read & parse its output into some Nim types that are part of your GUI.  I don't know UCI (and sorry, I am not interested in learning right now), but I cannot imagine they have no command to "initialize the whole board to foo" or a command to "print out the new board".  That is all you need.  (Well, and the Nim types to represent, Nim code to parse & Nim code to format for stockfish/UCI, and all that other Nim code for the GUI.)
Your program must maintain that board state anyway (e.g., to not let users drag pieces from an empty square to another empty square).  So, in addition to the types I mentioned above (or nearby variants), you will probably want (at least!) some kind of type Piece = enum King="K", Queen="Q"... and also type BoardState = array[8, array[8, Piece]] to update and render {or I don't know, maybe the two array index slots would be enums like array[A..H, array[1..8, Foo]] for some other enum with A..H as enum name tags }.
I really can only give a sketch and vague pointers on how to keep your project simple or within scope for you.  In fact, I've almost run out of such pointers.  I cannot write your program for you, though I encourage you to persevere and get to some nice usability point. :-)  { And ChatGPT almost always disappoints me. }
As for the Nim destructor bugs -- yeah.. They happen.  It sounds like the Owlkettle guy worked around it, but you may run into others.


Grafana 7.0 compatibility

Some elements are not themed properly after update to Grafana 7.0 such as tables:

The edit-screen of panels is not themed at all:

Select dashboard is also not themed:

I guess there is more as much of the design was changed in 7.0

relates to https://github.com/gilbN/theme.park/issues/45
Will see if I have time this weekend

Should be good now. Let me know if you find more.


the hidden_states lost when training on voxceleb1

Hi developers,
I tried to use wave2vec2/hubert to train a speaker identification model and the error is shown in the above figure. The command which makes this situation is:
python3 run_downstream.py -n exp -m train -u wav2vec2 -d voxceleb1 -f
After checking the features produced by the upstream model, I found that the lost hidden_states is "hidden_state_12". But it is fine if I remove the "-f" option or use other pre-train models (e.g. tera).

I also tried this:

If I didn't set no_grd(), the hidden_states seems to be missed.


Question of all questions: Will Station be continued?

It's sad reading all these issues without getting any answers. So: Are there plans with Station?

https://github.com/getstation/desktop-app/commit/f4c2de9c1c7e44cc0e522e7e19a518134f2fa088

This is sad. This is an awesome APP.
Also, I thought all the problem could be fixed by upgrading the Electron version, so I cloned it and try.
But things didn't go well like there were so many error when upgrading.
I wish I knew how Electron works :(


Do not remove installed plugins after upgrade

Hello, I use SSHJ extension and every time I update the app I need to reinstall that plugin.
Is it possible to keep installed plugins after an upgrade so we do not need to reinstall them afterwards ?
Thanks.

Hello @krilim,
Please provide information about your OS and installation method you're using (e.g. exe installer, zip archive, etc.)


Consider alternatives to poetry

The pypoetry project are not fixing breaking bugs. We should look for an alternative. Perhaps Flit?
https://github.com/sdispater/poetry/issues/536

Also https://github.com/ofek/hatch

Note that this issue is currently quite important, as our docs simply list the tools you need to install.
For all of these tools except poetry, simply going to the tool's website and following their guidance will get you a working install in the vast majority of cases (unless you have a really funky system, in which case you should know what to do).
Currently, following the recommended method of installing poetry (which by the way is curl piped into python, which is pretty nasty already) will give you a broken install on standard Ubuntu (!!) and any other system which follows the Python standard laid out in PEP 394 whereby python is Python 2 and python3 is Python 3. Poetry helpfully doesn't bother mentioning this anywhere except in the issue above, and no clear and obvious fix is available due to poetry's obscure method of installation.
We are in a position where we either need to document how to get poetry working on PEP 394 compliant systems (which, in my humble opinion, shouldn't be our problem), or we need to find an alternative to poetry where the default behaviour on a standards-compliant system is not horrifically broken.

I did do a lot of research into this, and it is unfortunate the poetry is the only tool with the feature-set that we need.
To clarify this further, we should start enumerating features that we need.
A start:

Easy to Install for new contributors
Installs a locked version of development dependencies
Ideally uses pyproject.toml
Dependency Resolution
Project build and publishing
Supports Python 3.6+

I'm inclined to agree, but the reason I think we should keep looking for alternatives is precisely that poetry catastrophically fails at point number 1: "Easy to Install for new contributors". I still haven't managed to get it working outside of docker on Ubuntu 18.04.

Poetry will pick up the currently activated Python version to create or use the virtualenvs.
Apparently using pyenv is a solution. If this is the case and we stick with poetry, we should definitely document that.

Anything that overrides the parent process's PATH to make python point to the intended Python version will "work", but this will almost always be iffy and require local configuration. poetry really needs to understand that invoking plain old python is always going to cause problems for people trying to use Python 3 (if you're using Python 2.x, you may not notice). The issue essentially makes poetry incompatible with almost all Python 3 distributions.

After some discussion, the poetry maintainers have shown that they will fix this before the 1.0 release. See https://github.com/sdispater/poetry/issues/536#issuecomment-541625301 for details.
I think that as I've been able to find a suitable alternative, we can consider this issue done. There does not seem to be another option with the featureset of poetry.


Make the Cache API deal with request's redirect mode not being "follow"

I think it would make sense for the Cache API to take request's redirect mode into account. Currently the Cache API always returns the redirected response. IT would make sense that when redirect mode is not "follow", you'd get an error instead.
See also https://github.com/whatwg/fetch/issues/79.

If we do this, how can a site offline a navigation that redirects to a cross-origin resource without CORS headers?

cache.add(redirectingNavUrl, { mode: 'cors', redirect: 'follow' }) fails because redirect does not have CORS header.
cache.add(redirectingNavUrl, { mode: 'no-cors', redirect: 'follow' }) succeeds, but the resulting opaque result cannot be passed to respondWith() for a navigation.

Today, however, you can offline the original site during install and offline the cross-origin redirect destination using a read-through-cache approach.  You can do:

cache.add(redirectingNavUrl, { mode: 'same-origin', redirect: 'manual' }) to store the opaqueredirect.
The navigation will then follow the redirect and get another service worker for the cross-origin destination.  (This doesn't happen in the cases above because skip-serviceworker-flag.)
That other service worker can offline the final destination.

If we can't store the opaqueredirect in Cache, though, then the original site always has to go to network which will fail while offline.

We should be able to store opaque redirects, and we don't say you can't so it should be fine.

F2F resolution: cache.match(request) should reject if the redirect mode is manual and the matched response had a redirect.

To be clear, the resolution was to fix both of those in Fetch.

Having said that, the Cache API needs to be updated to store response's url list to make this work. So that does need to be updated as part of this.

Having said that, the Cache API needs to be updated to store response's url list to make this work. So that does need to be updated as part of this.

I think the spec does this already, but it wasn't previously exposed to script in any way.  So the URL list or equivalent needs to be added to implementations.


Mute Microphone

Hello and sorry for asking this again.
Is juts that I could not find something to close your microphone apart from those methods about the steam, but I don't want to mute the stream, just your own audio. I'm usign your "Join Channel Audio" system.
I see that I can set the volume using the class "device manager", do you think that this would be the best option?

are you try "muteLocalAudioStream"?

oh wait is not that for the share stream or is also for normal voice channels?
I tried to use one of those functions, but it was crashing, I'd try again if that's what I'm looking for.

muteLocalAudioStream works for any stream that you publish to the channel.

just to make sure when we say "stream" are we talking about any kind of voice connection or just for the screen share stream?

are you try this code？

this also can use in joinchannel，you can see joinchannel api have option too.

https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_irtcengine.html#api_irtcengine_mutelocalaudiostream

I created a function in the audio widget class to call the Rtc function "muteLocalAudioStream", calling this first function is working but is not doing anything, is this way the right way?

I think that's right. If the problem is solved, can you close the issue?

Is not, is not doing anything, but I don't see the logic to mute the audio in this function apart from that if

ohh sorry, the value has to be true, is working


electron_21: init at 21.0.1

Description of changes
Add Electron 21 and make it the default electron package.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Please add an empty line before electron_21 = mkElectron "21.0.1" {

Missed that. Sorry.

@GrahamcOfBorg build electron_21

You also need to add electron_21 on line 15901 of all-packages.nix


Proof of concept

As the developer, I want to get the game to a complete, "proof of concept" state.

AC:

All basic parts of the game are present and working as expected, including

the user-controlled paddle,
the ball, and
the bricks (which break upon contact with the ball as expected).

The game properly detects the win and lose conditions.
The game can be restarted.
Optional:

An in-game quit option.
Basic scoring system is included.

#4

In game quit option not implemented, as it is not immediately clear how to do so with Electron, and I may change bundlers.


feat(CosmosPackagesStore): implement service/describe in store

~⚠  Depends on #2404 (provides the actions)~
Epic: Store support for Edit Configuration Settings.
Summary: Adds support to cosmos store for service/describe endpoint.
Quick description: expose a new method .fetchServiceDescription() on the cosmos packages store. A successful result of this will be just like .fetchPackageDescription() - it will set the packageDetails field in the store. The component (not built yet) can then use this the populate its form.
Closes DCOS-18200.
Checklist

[x] Did you add a JIRA issue in a commit message or as part of the branch name?
[x] Did you add new unit tests?
[ ] Did you add new integration tests?
[ ] If this is a regression, did you write a test to catch this in the future?

Alfred please test this

@MatApple I only grepped for descriptionSuccess and descriptionError within dcos-ui and dcos-ui-plugins-private. There is no other repo that would use these events, right?

@ahoskins no other repos 👍

💯


Long Load Times For Large Datasets - Possible Solution?

I'm a relative novice to EF and EF Core, but one of the things that has been really bothering me is the amount of time it takes me to load a large dataset. Googling this, it seems to bother quite a few people. It may be clear to everyone developing the project what's causing this slowdown, but I wanted to start a discussion in case it isn't, and hopefully work towards a solution that makes the project better.
Reading about common frustrations with EF Core, you'll find that most people suggest that to decrease load time for large datasets that you don't intend on changing, you should set:
Context.ChangeTracker.AutoDetectChangesEnabled = false;
Context.ChangeTracker.QueryTrackingBehavior = QueryTrackingBehavior.NoTracking;

After trying to load a dataset 1.6M rows deep with these options set as listed, I still found the performance to be less than stellar. SQL Server Express will return these rows to me in a mere 20 seconds, but the total time to load them using EF was 428 seconds.
So, I cloned the most current version of the code and added it to my project. After stepping through everything, I found that there is a reason: a relatively hidden function in EFCore, Microsoft.EntityFrameworkCore.Query.Internal.WeakReferenceIdentityMap.CollectGarbage()
Since I've not been around this project for a while, I can't speak to this function's necessity. But because I did load up the code, I was able to gather some empirical statistics surrounding my particular dataset with and without this function call. Below, you'll find two images illustrating the cumulative and incremental time taken inside the entity framework code to load the entire dataset into memory. Each image shows two series: one with CollectGarbage() enabled, and one with the call to CollectGarbage() commented out.

As you can plainly see, this function seems to be adding O(N^2) complexity to basic loading of entities. An inspection of the offending function yields a little insight. Every 500 rows loaded will cause an inspection of the full array of already-loaded objects.
        private const int IdentityMapGarbageCollectionThreshold = 500;
...
        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public void CollectGarbage()
        {
            if (++_identityMapGarbageCollectionIterations == IdentityMapGarbageCollectionThreshold)
            {
                var deadEntries = new List<TKey>();

                foreach (var entry in _identityMap)
                {
                    if (!entry.Value.TryGetTarget(out _))
                    {
                        deadEntries.Add(entry.Key);
                    }
                }

                foreach (var keyValue in deadEntries)
                {
                    _identityMap.Remove(keyValue);
                }

                _identityMapGarbageCollectionIterations = 0;
            }
        }

That's basically my report. I don't know if anything can be done about this, because simply I don't understand the ramifications of removing it or disabling it. If there are few ramifications and we could have the option to disable this feature at the very least, I am certain many users would appreciate it.
Further technical details
EF Core version: 2.2.0-preview2-35143
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Win 10 Pro
IDE: (e.g. Visual Studio 2017 15.9.0 Preview 3.0)

@cculver Also, thanks for the epic filing. The graphs are compelling. :clap:

@ajcvickers I have a question regarding your statement  "For 3.0, we will stop doing identity resolution for no-tracking queries.".
Does this mean that you plan to completely remove identity resolution in combination with AsNotTracking in 3.0, without an option to enable it?
I would not be happy if this would be removed without an option to enable it.
The documentation of QueryTrackingBehavior.NoTracking clearly states that:
Identity resolution will still be performed to ensure that all occurrences of an entity with a given key in the result set are represented by the same entity instance.
Please consider adding a new Option QueryTrackingBehavior.NoTrackingNoIdentityTracking.
Benefits:

No breaking change, usage of NoTracking and identity resolution is a good thing for some scenarios
Still allow users to disable identity tracking, if they see a performance improvements.
Can reduce memory usage + GC-Time if specific instances are very common in a query (ex. a company query where most companies reference the same country).

@davidroth Tracking queries will still do full identity resolution. In essence, no tracking queries were trying to behave like tracking queries in this respect, since normally the tracking is what provides the identity resolution.

@ajcvickers So to recap, NoTracking in EF 3.0 will not perform any identity resolution, right? Will we have at least identity resolution within each materialisation phase (per query) when using NoTracking?

@davidroth Current plan is to go back to the EF6 way of doing things for this, so no.

@davidroth after this change is effective, if you want a query to execute with identity resolution within its results, but without interfering with the context’s state, you can still do so by executing the query on a separate DbContext instance. I wonder if we can make that easier than it is now. I am also curious what kind of queries you are concerned about returning duplicate instances.

@divega  Nothing specific in my mind. I was just thinking, that there might be usages in disconnected scenarios, where one would like to have the combination of "NoChangeTracking + IdentityTracking" (Better query performance because no ChangeTracking is involved but still have unique instances in the resulting graph).
Sure, spinning up a new context with Tracking enabled, will give you Identity tracking. But this always comes with the price of populating EntityEntries with Old/NewValues etc.

@davidroth That's essentially what we try to do now. But it's added complexity, which leads to bugs like this one, for something that we don't think is especially useful. We could always revisit this in the future, but for now we think it's more pragmatic to go back to very simple semantics and implementation.

The query pipeline no longer does identity resolution for no-tracking queries, so closing this.

Re-opening: breaking change documentation is missing.


Incorrect Results when using OR Operator in WHERE Clause with Pinot Connector on Trino

Trino version: v419.
Pinot version: v0.12.1
I am using Trino to query Pinot, and I have encountered an issue with the results when using the OR operator in the WHERE clause. The specific query is as follows:
WITH fd AS (
  SELECT user_id, affiliate
  FROM (
    SELECT user_id, affiliate, row_number() OVER (PARTITION BY user_id ORDER BY count(0) DESC) AS rn
    FROM sensitive_table
    GROUP BY user_id, affiliate
  ) ac
  WHERE rn = 1
)

SELECT fd.user_id, s2.account_user_id, fd.affiliate
FROM sensitive_table_2 s2
RIGHT JOIN fd ON fd.user_id = s2.account_user_id
WHERE fd.user_id = 'sensitive_user_id' OR s2.account_user_id = 'sensitive_user_id'

When I modify the condition to use the OR operator
WHERE fd.user_id = 'sensitive_user_id' OR s2.account_user_id = 'sensitive_user_id'

, the query fails to retrieve any data. However, when I use the AND operator
WHERE fd.user_id = 'sensitive_user_id' AND s2.account_user_id = 'sensitive_user_id'

, the query returns the expected results.
This behavior seems to indicate an issue with the Pinot connector in Trino when handling OR conditions in the WHERE clause. The connector might not be correctly evaluating the OR operator in this scenario, leading to incorrect results.
I kindly request the Trino community to investigate and address this issue with the Pinot connector, as it hinders the ability to construct queries with flexible conditions. Thank you for your attention to this matter.

cc @elonazoulay @xiangfu0

I think the valid query should be
  SELECT user_id, affiliate
  FROM (
    SELECT user_id, affiliate, row_number() OVER (PARTITION BY user_id ORDER BY count(0) DESC) AS rn
    FROM sensitive_table
    GROUP BY user_id, affiliate
  ) ac
  WHERE rn = 1
)

SELECT fd.user_id, s2.account_user_id, fd.affiliate
FROM sensitive_table_2 s2
RIGHT JOIN fd ON fd.user_id = s2.account_user_id
WHERE fd.user_id = 'sensitive_user_id'

For RIGHT JOIN, you don't need fd.user_id = 'sensitive_user_id' OR s2.account_user_id = 'sensitive_user_id'

Hello Xiang Fu,
We have discovered that the issue was caused by exceeding the default limit of 100,000 for pinot.server.query.executor.num.groups.limit when grouping the query results. This resulted in incorrect results.
Thank you for your previous response!


Support Volume Migration to OpenEBS Localpv Hostpath Storage Class

Support data migration path from rook/ceph to OpenEBS local volumes.
When migrating to OpenEBS local provisioner, local volumes will be created on the node where the consumer (pod) was originally scheduled on prior to the migration.
Verified migration paths for:

Rook-1.0.4 to OpenEBS on a single node Kubernetes cluster
Rook-1.0.4 to OpenEBS on a three node Kubernetes cluster
Rook-1.0.4 to longhorrn on a three node Kubernetes cluster to ensure there weren't any regressions

internal tracking issue [sc-58504]


Update quick-start-guide.md

second hospital accepts a POST. Would confuse the reader and lose him/her at first glance.
Purpose
Correct the quick start documentation

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Add Icon PortfolioItem subresource

https://projects.engineering.redhat.com/browse/SSP-129
THIS PR IS A CLEANUP OF THIS PR: https://github.com/ManageIQ/catalog-api/pull/112
This PR adds support for retrieving icons through the service portal API. It is a pass through for the topology icons service (in the future, if there is no overridden icon in the service portal database we will return that instead). The meat of reaching out to Topology for the icon is in ServiceOffering::Icons
It provides the following new subresource:
GET /portfolio_items/{portfolio_item_id}/icon

cc @syncrou

@mkanoor - Could you please take a look at this?

@miq-bot remove_label unmergeable

@mkanoor would be great if you could take a look at this.

@miq-bot remove_label unmergeable


feat(single-spa-angular): add possibility to have a zone-less application

This looks good to me - are you planning on adding an e2e test to this PR? Or should I merge this one in first

Released in https://github.com/single-spa/single-spa-angular/releases/tag/v4.1.0


Work around - add {{end}} at the end.

This was mainly added for historic purposes, should be fixed now.


Added onnx model gRPC protocol test on triton

Added Automation Test for RHOAIENG-11565

Verified with Jenkins  Build 397


Add readme

This repo must have a readme to explain module purpose and api.

Well, I think there is a pending pull request for that, which is ready since... August 2016


Fixes #5593 : Allow disabling HtmlEncode on DynamicForms

If we want to default encode form values, an alternative to this PR would be a checkbox to opt-out HtmlEncoding.
More, HtmlEncoded form values are not correctly diplayed in Form Submissions view.

We talked about it during the triage, and these kind of options should not exist, as each element should define the behavior. A ScripElement would not encode, a TextElement, would.

I agree for defining this behavior at element level rather than on the form. It allows par field rules.
However, we must be able to choose to encode or not TextElement's values, because there are many cases in which you don't want an encoded string (most cases for me).