Kernel Memory is broken with latest nugets

Using the 0.8 release of LlamaSharp and Kernal-Memory with the samples there is an error because the LlamaSharpTextEmbeddingGeneration doesn't implement the Attributes property.
I took the source and created my own and added this:
public IReadOnlyDictionary<string, string> Attributes => new Dictionary<string, string>();
So it wouldn't error.
But no matter what model I use I get "INFO NOT FOUND." (I've tried kai-7b-instruct.Q5_K_M.gguf, llama-2-7b-32k-instruct.Q6_K.gguf, llama-2-7b-chat.Q6_K.gguf and a few others)
I've tried loading just text, an html file, and a web page to no avail.

Hi, which version of kernel-memory did you install?

I also notice that there is no way to add grammar to this right now so AskAsync can't force good output of JSON as an example. Can this be added?
And I'm noticing, now that I have it hacked to work(ish) that a memory.AskAsync works, but if you do it again (i.e. memory.AskAsync) for a second question against the data, I just get a ton of \n characters and no valid response. I'm not sure if that's kernel-memory doing that, or if it's this that's causing the issue. (If it helps, I'm in GPU only mode with dual nvidia 8 gb cards.)

I also notice that there is no way to add grammar to this right now so AskAsync can't force good output of JSON as an example. Can this be added? Same with MainGPU and the other settings on the context. And there are a few optoins missing (TOP_K) for the executer as well that you can't get to with Microsoft's With(new TextGenerationOptions). Perhaps an extended one for LlamaSharp that has all of the rest of the settings that are available?

Yes, the current version of Kernel Memory does not provide more customizable options.
However, there are currently two ways to format the output: one is by reformatting the Ask Result through LLM, and the other is by using the Search method to obtain search results from documents and then organizing prompts to generate desired results.

And I'm noticing, now that I have it hacked to work(ish) that a memory.AskAsync works, but if you do it again (i.e. memory.AskAsync) for a second question against the data, I just get a ton of \n characters and no valid response. I'm not sure if that's kernel-memory doing that, or if it's this that's causing the issue. (If it helps, I'm in GPU only mode with dual nvidia 8 gb cards.)

It seems to be related to the state management of the model. @AsakusaRinne any idea?

On the first issue I was able to take your code, and manually add a grammar to it. Is there a way that we can just expose Grammar in the LLamaSharpConfig for right now that will get passed through?  Same with MainGPU, and TOP_K?

On the first issue I was able to take your code, and manually add a grammar to it. Is there a way that we can just expose Grammar in the LLamaSharpConfig for right now that will get passed through? Same with MainGPU, and TOP_K?

👍 Good idea, I seem to have a solution to the issue #289. 😃  Thank you for your suggestion.

On the issue with getting the endless \n on the second AskAsync, it also does it with embeddings. If you import 2 documents in a row, the second one will fail.

Update: It isn't the context, and the native handle shows that it's still valid.  I've tried to replace all of the shared versions of the context and model with new versions spun up per request, and there was no difference.

Yes, I used WithlllamaSharpDefaults.
I'm using a console app
var modelPath = Path.GetFullPath(Path.Combine(Assembly.GetEntryAssembly()!.Location, "..", "..", "..", "..", @"openhermes-2.5-mistral-7b-16k.Q5_0.gguf"));

var builder = new KernelMemoryBuilder()
        .WithQdrant(new QdrantConfig
        {
            APIKey = "",
            Endpoint = "http://localhost:6333",

        })
        .WithLLamaSharpDefaults(new LLamaSharpConfig(modelPath)
        {
            ContextSize = 16_384,
            Seed = 1337,
            GpuLayerCount = 20,
        })
        .With(new TextPartitioningOptions
        {
            MaxTokensPerLine = 100,
            MaxTokensPerParagraph = 300,
            OverlappingTokens = 30,
        })
        .With(new TextGenerationOptions
        {
            MaxTokens = -1,
            TopP = 0.95F,
            Temperature = 0.0,
            FrequencyPenalty = 1.1F,
            StopSequences = ["\n"],
            ResultsPerPrompt = 1
        });

    var memory = builder
                    .BuildServerlessClient();

await memory.ImportTextAsync("The quick brown fox jumps over the lazy dog.");

await memory.AskAsync("Who jumps over the lazy dog?");

await memory.AskAsync("What kind of dog is it?");

The second AskAsync will return a ton of \n\n no matter the model at least when using CUDA.
The second part about EmbeddingMode is just in the setup of the WithLLamaSharpDefaults it doesn't split out the parameters so they both use the same params for text creation and embedding instead of optimizing out the embedding params.

@xbotter I think we should watch the prompt put into our model in the second run with context size reduced to 4096. Could you please take a look? I'm on duty this weekend.

@xbotter I think we should watch the prompt put into our model in the second run with context size reduced to 4096. Could you please take a look? I'm on duty this weekend.

This can only be resolved from the kernel memory . I have already submitted an issue https://github.com/microsoft/kernel-memory/issues/164 and waiting for further updates.

If it matters I’m not changing the context length and it still happens.

KernelMemory author here, let me know if there's something I can do to make the integration better, more powerful, easier, etc :-)

Not related to this, but since you offered, the biggest single issue with kernal memory (and LLMs in general) is context length. In kernal memory it's worse because you're trying to analyse documents which can be very long and if you Ask on many documents the context runs out and causes issues.
I'd love to see KernelMemory port or otherwise adopt some of the winding techniques of memgpt natively and expose methods for private LLMs to impliment incremental analysis etc.  I.e. keeping the system message and query in context, and then taking part by part of documents in a loop to pull facts out of and accumulate answers.

Thanks for the feedback, we merged a PR today that allows to configure and/or replace the search logic, e.g. defining token limits.
And this PR https://github.com/microsoft/kernel-memory/pull/189 allows to customize token settings and tokenization logic. Would appreciate someone to take a look.
This snippet shows how we could add LLama to KernelMemory:
public class LLamaConfig
{
    public string ModelPath { get; set; } = "";

    public int MaxTokenTotal { get; set; } = 4096;
}

public class LLamaTextGenerator : ITextGenerator, IDisposable
{
    private readonly string _modelPath;

    public LLamaTextGenerator(LLamaConfig config)
    {
        this._modelPath = config.ModelPath;
        this.MaxTokenTotal = config.MaxTokenTotal;
    }

    /// <inheritdoc/>
    public int MaxTokenTotal { get; }

    /// <inheritdoc/>
    public int CountTokens(string text)
    {
        // ... count tokens using LLama tokenizer ...
        // ... which can be injected via ctor as usual ...
    }

    /// <inheritdoc/>
    public IAsyncEnumerable<string> GenerateTextAsync(
        string prompt,
        TextGenerationOptions options,
        CancellationToken cancellationToken = default)
    {
        // ... use LLama backend to generate text ...
    }
}

This looks good from what I can see!
Is there a roadmap for memgpt style functionality? Really missing this from python.

Is there a roadmap for memgpt style functionality? Really missing this from python.

you mean a chat UI?
Things on the roadmap are supporting chat logs and some special memory views. I always wanted to create a simple UI for demos, maybe as a side project, no timeline though

About LLamaSharp, could you point me to how to count the tokens for a given string? is there some example?

If you have a LLamaContext:
LLamaContext context;
int[] tokens = context.Tokenize("this is a string");
int count = tokens.Length;

If you only have a model and no context (i.e. LLamaWeights):
LLamaWeights weights;
int[] tokens = weights.NativeHandle.Tokenize("this is also a string");
int count = tokens;

dluc: Generally I see this when:

The tokens to keep doesn't include the original prompt and the question.
The context_length is too short for the total of everything.
The max_tokens (i.e. the maximum response length) is too short.

Generally what you want is the context_length to be the same as the model's length.  And you want max_tokens to either be short but long enough for the answer you're expecting because LLAMA has a bad habit of repeating itself, or System Message + all user messages + assistant responses including max_tokens <= context_length.
I use TikSharp to calculate the number of tokens for all prompts, add 10 or so just to be safe and subtract that from context_length and make that the max token length, then set antiprompts to AntiPrompts = ["\n\n\n\n", "\t\t\t\t"] which gets rid of 2 of the cases (especially when generating json with the grammar file) of repetition instead of ending.
This technique also works when using ChatGPT 3.5+ so you don't get errors since it hard refuses and costs you money, to produce more than the context_length so you have to do this math or risk it blowing up and running up your bill.

PS: Kernel-Memory is broken again with the latest update to Kernel-Memory.

@dluc , as you are developer of kernel memory, can you provide some sample of MemoryServerless based on LLamaSharp? I am trying to make it work (by getting code for text generator from https://github.com/microsoft/kernel-memory/pull/192), but don't have much luck.AskAsync goes to infinite state
Here is my code:
`
open System
open LLama
open LLama.Common
open LLamaSharp.KernelMemory
open Microsoft.FSharp.Core
open Microsoft.KernelMemory
open Microsoft.KernelMemory.AI
open Microsoft.KernelMemory.Handlers
open Microsoft.KernelMemory.MemoryStorage.Qdrant
let memoryBuilder = KernelMemoryBuilder()
let inferenceParams = new InferenceParams(AntiPrompts = [|"<|end_of_turn|>"|])
let llamaConfig = new LLamaSharpConfig("/Users/codechanger/llama/openchat_3.5.Q5_K_M.gguf")
llamaConfig.DefaultInferenceParams <-inferenceParams
llamaConfig.ContextSize<- 4096u
type Generator(config: LLamaSharpConfig) =
let modelParams  = new ModelParams(config.ModelPath)
do modelParams.ContextSize<- config.ContextSize
let weights = LLamaWeights.LoadFromFile(modelParams);
let embedder= new LLamaEmbedder(weights, modelParams)
let context = weights.CreateContext(modelParams)
interface ITextEmbeddingGenerator with
member this.CountTokens(text) = context.Tokenize(text).Length
member this.GenerateEmbeddingAsync(text, cancellationToken) =
task{
let  embeddings = embedder.GetEmbeddings(text)
return Embedding(embeddings)
}
member this.MaxTokens =int (modelParams.ContextSize.GetValueOrDefault())
// member this.(data, kernel, cancellationToken) =
//     }
// member this.Attributes = Dictionary<string,obj>()

type TextGenerator(config: LLamaSharpConfig) =
let modelParams  = new ModelParams(config.ModelPath)
do modelParams.ContextSize<- config.ContextSize
let weights = LLamaWeights.LoadFromFile(modelParams)
interface ITextGenerator with
member this.CountTokens(text) =
use context = weights.CreateContext(modelParams)
context.Tokenize(text).Length
member this.GenerateTextAsync(prompt, options, cancellationToken) =
let parameters = InferenceParams()
parameters.Temperature <-float32 options.Temperature
parameters.AntiPrompts <-  options.StopSequences |> Seq.toArray
parameters.TopP <- float32 options.TopP
parameters.PresencePenalty <- float32 options.PresencePenalty
parameters.FrequencyPenalty <- float32 options.FrequencyPenalty
parameters.MaxTokens <- options.MaxTokens.GetValueOrDefault()
let executor = new StatelessExecutor(weights, modelParams)
executor.InferAsync(prompt)
member this.MaxTokenTotal = int (config.ContextSize.GetValueOrDefault())
let mb = memoryBuilder
.WithCustomEmbeddingGenerator(Generator(llamaConfig))
.WithCustomTextGenerator(TextGenerator(llamaConfig))
.With(new TextPartitioningOptions(MaxTokensPerParagraph = 300, MaxTokensPerLine = 100, OverlappingTokens = 30))
let kernelMemory = mb.Build()
task{
let! doc = kernelMemory.ImportDocumentAsync("/Users/codechanger/test.txt")
Console.WriteLine(doc)
let! res = kernelMemory.AskAsync("autumn")
Console.WriteLine(res.Result)
}|> ignore
Console.ReadLine() |> ignore
`
test.txt itself is very simple:
Autumn is sad season
I am using Bloke model for openchat 3.5

Update: LLamaSharp 0.8.1 is now integrated into KernelMemory, here's an example: https://github.com/microsoft/kernel-memory/blob/main/examples/105-dotnet-serverless-llamasharp/Program.cs
There's probably some work to do for users, e.g. customizing prompts for LLama and identifying which model works best. KM should be sufficiently configurable to allow that.

@dluc Thank you a lot for this great work! Since kernel-memory has supported integration of LLamaSharp internally, I think we'll give up maintaining LLamaSharp.kernel-memory package since the next version. If there's any issue about the LLamaSharp integration in kernel-memory, please feel free to ping me or @xbotter. I'd like to help with it. :)

The example is missing A LOT and it appears that it doesn't handle stuff properly either. I.e. it uses Azure openai to do tokenization/embeddings instead of LLama like the LLamaSharp.kernel-memory does and there isn't even any code to make it work.
And even then, I'm not sure how the sample would even work, because OpenAI uses a different tokenization length than LLama so the results will be vastly different if it works at all.
I'd suggest before LLamaSharp.kernel-memory is legacied, that the sample needs to be updated to use LLamaSharp for the entire roundtrip.

@AsakusaRinne I would take the opportunity to thank you all for LlamaSharp, making it so straightforward to integrate Llama into SK and KM.
Before removing KM from LlamaSharp, I'd just highlight that we didn't add LLamaEmbeddings in KM, because text comparison tests were failing, because cosine similarity for similar strings is off. For example comparing the similarity of these strings, using embeddings:
const string e1 = "It's January 12th, sunny but quite cold outside";
const string e2 = "E' il 12 gennaio, c'e' il sole ma fa freddo fuori";
const string e3 = "the cat is white";

Results with OpenAI Ada 2
e1 <--> e2: 0.9077211022377014     OK
e1 <--> e3: 0.7831940054893494     OK
e2 <--> e3: 0.7444156408309937     OK

Results with llama-2-7b-chat.Q6_K.gguf
e1 <--> e2: -0.09282319992780685   FAIL
e1 <--> e3: 0.655430018901825      OK
e2 <--> e3: -0.10494938492774963   OK

Results with llama-2-13b.Q2_K.gguf
e1 <--> e2: 0.4025874137878418     FAIL
e1 <--> e3: 0.6390143632888794     OK
e2 <--> e3: 0.33123183250427246    OK

This is kinda required.
And AskAsync is REALLY slow.

This is kinda required.
And AskAsync is REALLY slow.

[Maybe we can move this conversation to KM repo if it's out of scope here]
@JohnGalt1717 could you help understanding if AskAsync is slower than the underlying model?

What hardwarre and drivers are you testing on?
Is LlamaSharp InferAsync faster? if so, how much?

From my tests:

on a PC with 64GB RAM, Nvidia T600, CUDA 12: Llama is slow, and AskAsync reflects the same performance. E.g. 3 minutes to answer a question with ~2000 tokens.
on a MacBook M1 with 32GB RAM: Llama is quite fast and the same for AskAsync. E.g. 12 seconds to answer a question with ~2000 tokens.

The example is missing A LOT and it appears that it doesn't handle stuff properly either. I.e. it uses Azure openai to do tokenization/embeddings instead of LLama like the LLamaSharp.kernel-memory does and there isn't even any code to make it work.
And even then, I'm not sure how the sample would even work, because OpenAI uses a different tokenization length than LLama so the results will be vastly different if it works at all.
I'd suggest before LLamaSharp.kernel-memory is legacied, that the sample needs to be updated to use LLamaSharp for the entire roundtrip.

It appears there may be a bit of confusion regarding the distinction between text indexing and RAG text generation. The process of text indexing operates independently and does not necessarily require the use of the same model utilized for text generation. It's perfectly reasonable to use one model for embedding and indexing, while employing a different model for text generation.
As for the test in the repo, we have executed it on numerous occasions with both text and files, using a blend of OpenAI/Azure OpenAI for embeddings and LLama for text generation, which has consistently resulted in expected outcomes. If you're able to provide the steps to reproduce any errors you're experiencing, we'd certainly appreciate the opportunity to investigate them further. Could you kindly share some additional details about your scenario?

@AsakusaRinne I would take the opportunity to thank you all for LlamaSharp, making it so straightforward to integrate Llama into SK and KM.
Before removing KM from LlamaSharp, I'd just highlight that we didn't add LLamaEmbeddings in KM, because text comparison tests were failing, because cosine similarity for similar strings is off. For example comparing the similarity of these strings, using embeddings:
string e1 = "It's January 12th, sunny but quite cold outside";
string e2 = "E' il 12 gennaio, c'e' il sole ma fa freddo fuori";
string e3 = "the cat is white";

Results with OpenAI Ada 2
e1 <--> e2: 0.9077211022377014     OK
e1 <--> e3: 0.7831940054893494     OK
e2 <--> e3: 0.7444156408309937     OK

Results with llama-2-7b-chat.Q6_K.gguf
e1 <--> e2: -0.09282319992780685   FAIL
e1 <--> e3: 0.655430018901825      OK
e2 <--> e3: -0.10494938492774963   OK

Results with llama-2-13b.Q2_K.gguf
e1 <--> e2: 0.4025874137878418     FAIL
e1 <--> e3: 0.6390143632888794     OK
e2 <--> e3: 0.33123183250427246    OK

May I ask for a way to reproduce it? I'm not sure if it's because the model does not support the language in the second sentence.

This is kinda required.
And AskAsync is REALLY slow.

[Maybe we can move this conversation to KM repo if it's out of scope here]
@JohnGalt1717 could you help understanding if AskAsync is slower than the underlying model?

What hardwarre and drivers are you testing on?
Is LlamaSharp InferAsync faster? if so, how much?

From my tests:

on a PC with 64GB RAM, Nvidia T600, CUDA 12: Llama is slow, and AskAsync reflects the same performance. E.g. 3 minutes to answer a question with ~2000 tokens.
on a MacBook M1 with 32GB RAM: Llama is quite fast and the same for AskAsync. E.g. 12 seconds to answer a question with ~2000 tokens.

I have a machine that I've run with and without GPU (2070 super). If I use the Kai 7b 5_S_M model from hugging face, it's 3:30 and 1:40 respectively run directly from the text of the partitions that a search returns.
if I use AskAsync it has been running for over 45 minutes and still hasn't returned.

Thanks for all of the hard work integrating KernelMemory with LLamaSharp, I have been able to get a working RAG prototype going in Godot with these two libraries. I noticed that for some reason, the IKernelMemoryBuilder doesn't actually load any layers onto the GPU, no matter how many layers I have specified. This happens with my desktop that has an RTX 3080 12GB and my laptop with an eGPU (RTX 3060 12GB).
I can confirm that I only have the Cuda12 backend added, and when I am using just LLamaSharp to ChatAsync it loads the model just fine onto the GPU (for both setups).  Perhaps I'm doing something wrong, but if other users can look at their Task Manager to see if the IKernelMemoryBuilder actually utilizes the GPU or not that could explain the slow speed. It runs my 5800X3D at ~66% for all file embeddings and queries.
I am also trying to figure out if there is a way to interact with the loaded model in a standard chat fashion, but I do not see any methods in IKernelMemory that interact with the model without searching the database. From what I can tell, using an IKernelMemoryBuilder utilizes the standard LLamaSharp weights/context/executor/embedder but does not make them available after creation of the IKernelMemory. It would be great to have access to the standard LLamaSharp session.ChatAsync method when utilizing a KernelMemory, if possible.


Need more tables in models/interactions

All marked by TODO

We used ENUM type to control vocabulary instead of reference tables.


Language Server - error filter

Look for BadType & co before making new errors. Since bad nodes are only generated from errors, it means we already made one.

This all looks good but I think this in particular needs a lot of tests. There are so .any situations to test that we don't double error and while you might have written it well now, tests ensure that it's written well when others work with it in the future.

I don't really understand why you wrote the tests like this - I was expecting to see more tests like testErrorFilter. This may test the functionality but I'm not sure how clearly it defines the behaviour. What do you think?
Also presumably we can get rid of "we currently can't read past the..." comments now?

Also presumably we can get rid of "we currently can't read past the..." comments now?

Incorrect. We indeed still can't read past the <x> before making the error.
Currently, we still can't because doing so could create other errors before the one we wanted to make.
Later on, we'll be able to read the full aggregate before making the error, while still making sure that said error will always have priority over any thrown during the parsing of said aggregates.

I don't really understand why you wrote the tests like this - I was expecting to see more tests like testErrorFilter. This may test the functionality but I'm not sure how clearly it defines the behaviour. What do you think?

The first test clearly shows a script throwing an error.
The second one is a modification to the script that results in another error being thrown earlier. Additionally, the earlier that happened in the first script does not show up in any of the errors thrown in the second script, showing that it was effectively filtered.
The main issue I faced was that it was hard to convey how the modified script "should" have made the error, and that the filter is the only reason it didn't do so.

I don't really understand why you wrote the tests like this - I was expecting to see more tests like testErrorFilter. This may test the functionality but I'm not sure how clearly it defines the behaviour. What do you think?

The first test clearly shows a script throwing an error. The second one is a modification to the script that results in another error being thrown earlier. Additionally, the earlier that happened in the first script does not show up in any of the errors thrown in the second script, showing that it was effectively filtered.
The main issue I faced was that it was hard to convey how the modified script "should" have made the error, and that the filter is the only reason it didn't do so.

So its just running two different tests, but linking them because they're conceptually similar? Sounds like those should be split up for more unity unit tests. Why do you not just have one test per instance added in this PR that checks if a child isBad and thus doesn't throw?

Why do you not just have one test per instance added in this PR that checks if a child isBad and thus doesn't throw?

But how would I check that said child "isBad", and how can I then prove that the lack of an error is specifically due to that?

Why do you not just have one test per instance added in this PR that checks if a child isBad and thus doesn't throw?

But how would I check that said child "isBad", and how can I then prove that the lack of an error is specifically due to that?

And... how is that different from what I did (the expected behaviour being "not throwing one particular error")

So its just running two different tests, but linking them because they're conceptually similar? Sounds like those should be split up for more unity unit tests. Why do you not just have one test per instance added in this PR that checks if a child isBad and thus doesn't throw?

I can't do that because if I was to do it, these tests wouldn't be unit tests. They would need to test more than what they are supposed to, only to prove the absence of something
testErrorFilter was really the simplest example I could make (and even then, that test was stuck having to imitate ScriptData), but they all aren't that simple (in that they all don't make a single error), and these additional errors are unrelated to what these 'unit tests' are meant to test.
This is why the assertion made by the test needs to be limited to "none of the errors made by this script is 'that one'".
However, if those would then become individual tests, the issue I then thought of was that it would be too easy to submit to those tests an error message that just can't happen at all.
For example, what if someone changes the message of an error, and only fixes tests based on which fail?
They would miss those tests, because they wouldn't fail.
This is because even if the error we thought was filtered out was still thrown, it still wouldn't match what we thought its message was (since it was changed).
This is why I paired the tests I made with another. The tests need to prove that the error they assert isn't being thrown is an error that can be thrown, which here is done by pairing them with a script that just so happens to throw them. Otherwise, you could just make a typo in the error message, and it would pass.

I've found with you Fred that when you post a wall of text I only need to read the last paragraph. It makes interacting with you much easier :)

Can we get a little bit more documentation of invalid, perhaps even a JSDoc with each parameter describes, and then I will merge


jasper route path not matched

change root path "src/reports/dailyincome.jasper" to "reports/dailyincome.jasper"

issue was fixed in root path.


[Documentation] Boring to Beautiful codelab hasn't been updated for go_router 4

I am having difficulty running step_01 in boring_to_beautiful.
When I run the app, I get an error:
 This GoRouteInformationParser needs to be used with GoRouteInformationProvider, did you forget to pass in GoRouter.routeInformationProvider to the Router constructor?
'package:go_router/src/go_route_information_parser.dart':
Failed assertion: line 148 pos 13: 'routeInformation is DebugGoRouteInformation'

if you see go router object you will have 1 property called routeInformationProvider
you need to pass that property in MaterialApp.router()
example:
MaterialApp.router(
        routeInformationParser: router(context).routeInformationParser,
        routeInformationProvider: router(context).routeInformationProvider,
        routerDelegate: router(context).routerDelegate,
        title: 'My App',
)

Please note in my case i have created function called router in another file
function signature
GoRouter router(BuildContext);

Hi @sammic297, thanks for catching that.
The bot updated the go_router dependency to 4.0.0 in https://github.com/flutter/codelabs/commit/5a584831fe8e15922e37067610b23a912c8a8cb0 but it looks like the code wasn't updated for go_router 4. There's a PR in progress to address this issue in https://github.com/flutter/codelabs/pull/1004
/cc @domesticmouse

Thanks @sammic297! Just merged flutter/codelabs#1004 which should resolve this issue. Please let me know if you're still running into issues!

Per our triage policy, I'll be closing this issue since the linked PR has been merged.
@sammic297, if you do run into this issue after checking out the latest source, kindly leave a comment below and this issue will be reopened.
Thank you

Thanks, everyone.
I will update the clone shortly.
Namaste 🙏🏾🙏🏾🙏🏾


docs/readme.md: Added decpendency installation instructions

I don't know much about other distributions, feel free to add
instructions if your distro isn't listed here :^)

Might be better if it links to https://go.dev/dl/.

I did consider that, though it doesn't have any real instructions for how to install it, it just hands you a .tar.gz, and even the install page doesn't give what I'd consider clear instructions…
Besides, on Linux, most people should probably be installing software through their package manager, unless manually installing it is the recommended way for go? Though I don't see a reason for that… I'm not a go developer, sorry…
I will probably link the official install docs too though, but keeping a simple copy-pastable command is probably a good thing for lesser-experienced users

Thank you, @zeGolem!


Text on top of first post (name of poster) gets cut off in default window

The text on top of first post (name of poster of first post in feed) gets cut off in default window. It becomes visible only on maximizing Ramme. Changing window size while NOT maximized, does not fix the problem.

You shouldn't be able to maximize  the window :(

I want full screen!?!

@fleamour say again, you don't seem pretty sure about that! 🤣

Well deb installs Linux Mint 18.1 & fullsize maximise works. Tumbleweed KDE needs to be set through right clicking on window list then checking fullscreen .

Yes, just noticed this error.

Fixed


Is there a conflict between responsive_framework and flutter_screenutil?

My app runs on mobile, and uses both responsive_framework and flutter_screenutil.
In the app, there are two different designSize values for flutter_screenutil in two widgets (home page and live page).
when switching back from the screen with a different designSize (live page),
I reset the previous screen's (home page) designSize using ScreenUtil.init.
However, some elements in certain areas(user id part) of the home page interface are partially reduced in size.

I suspect that some of the size judgments in responsive_framework may have been affected.
Is there a method in responsive_framework similar to ScreenUtil.init for resetting the screen size?
App:
  @override
  Widget build(BuildContext context) {
    return ScreenUtilInit(
      useInheritedMediaQuery: true,
      rebuildFactor: RebuildFactors.all,
      designSize: Size(390, 844),
      builder: (context, child) => MaterialApp(
        builder: (context, widget) => ResponsiveWrapper.builder(
          widget,
          maxWidth: 1200,
          minWidth: 428,
          defaultScale: true,
          breakpoints: [
            const ResponsiveBreakpoint.resize(428, name: MOBILE),
            const ResponsiveBreakpoint.autoScale(800, name: TABLET),
            const ResponsiveBreakpoint.resize(1000, name: DESKTOP),
          ],
        ),
        home: HomePage(),
      );
  }

HomePage -> LivePage
LivePage:
  @override
  Widget build(BuildContext context) {
    return ScreenUtilInit(
          designSize: const Size(750, 1334),
          minTextAdapt: true,
          splitScreenMode: true,
          builder: (context, child) {
            return Container();
          },
        );
  }

  @override
  void dispose() {
    super.dispose();

   ScreenUtil.init(...);
 }

Can you try changing the context variable? I suspect the name shadowing might be passing the wrong context without the updated MediaQuery from ScreenUtils to the ResponsiveWrapper.
 @override
  Widget build(BuildContext context) {
    return ScreenUtilInit(
      useInheritedMediaQuery: true,
      rebuildFactor: RebuildFactors.all,
      designSize: Size(390, 844),
      builder: (context, child) => MaterialApp(
        builder: (context2, widget) => ResponsiveWrapper.builder(
          widget,
          maxWidth: 1200,
          minWidth: 428,
          defaultScale: true,
          breakpoints: [
            const ResponsiveBreakpoint.resize(428, name: MOBILE),
            const ResponsiveBreakpoint.autoScale(800, name: TABLET),
            const ResponsiveBreakpoint.resize(1000, name: DESKTOP),
          ],
        ),
        home: HomePage(),
      );
  }


load_array document need update.

need to update document and examples for load_array
see ref and code snippets (outdated example):
https://mapdldocs.pyansys.com/user_guide/mapdl.html?highlight=load_array
from ansys.mapdl.core import launch_mapdl
import numpy as np
mapdl = launch_mapdl()
arr = np.random.random((5, 3))
mapdl.load_array(arr, 'MYARR')

Thanks, Alex!

Resolved in #391


What's the status of this project?

Hi,
I was looking for examples of WinForms apps and came across NUnit. By chance I see it hasn't been touched in 2 years and suddenly a large commit was made 2 days ago.
I'm wondering what the status of the project is, what the plans are for the future, what was involved in the update? (Too much code to read all the changes).

NUnit 3.0 was recently released, including a console runner. We decided the gui runner should be separate. I've been working on the runner for a few years and I'm now in the process of getting it up on GitHub. This project was created a few years back, but has not had source  code until now. We used it to accumulate gui-related issues.
The current code can be built and run but is not really usable for real work. Too many features are still absent. Several other contributors are getting started now on helping with it and I expect to do an 0.1 release in January.

Thanks a lot

An Alpha in January would be great :+1:

We are still aiming for 0.1 in January, but it won't be even an alpha release by our standards - just work in progress to get some feedback.


Update contact page

First of all sorry as my original branch got some issues so have to delete it
Although I have made all changes which were suggested

Hi, Please note that, I'll be able to review and merge the PR Tomorrow. Not today because of travelling.

@aslams2020 Thankyou for letting me know,
I hope that you review and merge it as early as possible.

Thank you so much @aslams2020 for guiding me properly to work on the project.
And so it was my first contribution it was an amazing experience working with you.


Checking if client exists in taskbar preview

There is probably an easier way of doing this, but if you are hovering on a client in the taskbar and close the client you get an error. Additionally, since closing removes the item from the taskbar the preview does not disappear. This is easily solved by making a check for if c.content is a valid field. Interestingly you cannot simply check if the client exists as I guess there is a timing issue where the client exists but client content does not.

Adding an if statement at the start of draw_widget with pcall(function () return type(c.content) end) seems to fix the problem completely for me. Might be a nice addition


IPython 3 Release

[x] Update requirements.txt
[x] Update setup.py
[x] Finalize Docs

Closed by #10


Exclude test pack from index

This allows us to use full pack CI in this test pack without actually adding it to the index.
The full pack CI includes testing changes to the deploy step.
Depends on StackStorm-Exchange/ci#108

No. That would break the weekly test runs which we need to ensure PATs stay active. Also, that relies on an obscure side effect instead of making skipping the index explicit.

Do we support deploying to Exchange from a non-master branch? If so, we should keep this, and just make sure that we add a deploy key to the CircleCI config for this repo, since that actually prevents the deploy step from pushing updated changes to the index.
If we do not support deploying to Exchange from a non-master branch, then we should add some comments to the deploy script to let future developers (but really, future me) know why that's hard-coded.
This pack and the test2 pack both exist to test installing non-master default branch, but I truly don't know if that includes deploying these packs to Exchange. I don't know what the intent has been in the past.

I want to use test and test2 for the purpose listed in the readme:
https://github.com/StackStorm-Exchange/stackstorm-test2/blob/a82739da92113b401d09192743cf2eb7296e2b7b/README.md#L3
To test changes to the pack CI system itself, including changes to the deploy step. But I can't do that if we [purposely] break the CI config.

I think, possibly in addition to or instead of using a flag to disable write operations for the deploy step of the test packs, that we should just ensure that they do not ever have any valid credentials that give them write/push access. That way, if we mess up our script logic and accidentally try to write/push to the index, the lack of a valid write creds will still help prevent it.

I think we could use read-only PATs for test packs: https://github.com/StackStorm-Exchange/ci/pull/107/commits/6b9d512c34463285c7293ef82466945454a5448c
I'm not sure if that will affect the ability to add tags to the pack repo.

If we do not support deploying to Exchange from a non-master branch, then we should add some comments to the deploy script to let future developers (but really, future me) know why that's hard-coded.

I agree. There should be a hint somewhere.
Looks like these test repos never ran the deploy script and should not be deployed to Exchange index. We also don't support any non-master branches deployments to Exchange which complements that story.
Because previously CircleCI config defined master for deployment while repository used aaa the deployment script never ran.
The behavior that changed it 6 days ago is:
https://github.com/StackStorm-Exchange/stackstorm-test/commit/b684a69747b3828ff7f3563393779234fb209a50#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R95
which was probably best intent.

Closing this in favor of a variable added via CircleCI itself.
https://github.com/StackStorm-Exchange/ci/pull/108#issuecomment-777746129


L2/R2 Back Touch Pad flipped with L3/R3 in The Last of Us

Recently got a hotspot so now I’ve been using my Vita more for remote play. I noticed that some games like God of War have horrible remote play controls which was fixed when I installed the plugin.
However, I noticed for The Last of Us, for some odd reason, my L2 & R2 are being mapped to the bottom parts of the back touch pad rather than the top parts of the back touch pad where my “trigger grip” lies. So now my L3 and R3 positions appear to be flipped.
Every other game I’ve tested (Uncharted collection, God of War, KH3, FF7 Demo) all seem to have no problems. It just appears to be The Last of Us.
I have not changed the settings of the plugin at all. I only installed it, and went straight to remote play. Removing the plugin lets me play The Last of Us without issue.
Is there a way to resolve this? In addition, is there a way to set certain control schemes from the plugin to certain games? (For example, I like the front touch pad corners being used to reload in the Uncharted collection, but I prefer the default layout for the plugin in God of War)
EDIT: Just found out that the same issue persists with Uncharted 4/Lost Legacy.

The controller type 2 should fix it...
But it breaks the touchpad button


Fix remote task submission could not find the jar

Fix remote task submission could not find the jar

Thanks very much for your contribution. Before submitting a pull request, here should be have issue description or email discussion. Please explain the background of the issue clearly, so that the reviewer reviewing the pull request has enough information to better progress the work.

Ok, thank you

This pr is used to fix the error that the jar cannot be found when a user submits a remote task


backport: merge bitcoin#22278, #24103, #24235, #24177, #24299, #24917, #25349, #25571, #17487, #26999, partial bitcoin#22564 (blockstorage backports: part 2)

Additional Information

Dependent on https://github.com/dashpay/dash/pull/6097
Dependency for https://github.com/dashpay/dash/pull/6067
Test failures in linux64_multiprocess-build (build) and linux64_tsan-test (build) do not stem from this PR but are pre-existing failures in develop (build, build). A fix for the build failures has been opened as a separate PR.

Breaking Changes
None observed.
Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas (note: N/A)
[x] I have added or updated relevant unit/integration/functional/e2e tests
[x] I have made corresponding changes to the documentation (note: N/A)
[x] I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

Rebasing on develop

 -:  ---------- >  1:  5394d63e18 feat: improve merge-check action to leave a comment and label to PRs
 -:  ---------- >  2:  35c76705d1 feat: implement basic Github Actions based CI, which reuses underlying logic from GitLab CI
 -:  ---------- >  3:  d1addb27aa fix: change fallback download path to be an s3 link which includes a few packages
 -:  ---------- >  4:  ed8ffa7fb4 feat: have cppcheck linter respect CACHE_DIR env variable
 -:  ---------- >  5:  4dbdecdd1e refactor: rename builder-image -> build-image and builder as image name to dashcore-ci-runner
 -:  ---------- >  6:  acf1315270 Merge bitcoin/bitcoin#25091: test: Remove extended lint (cppcheck)
 -:  ---------- >  7:  2237b4266b refactor: use `=` with ENV in dockerfiles
 -:  ---------- >  8:  1b139e4837 feat: automatically run guix-build on all tags pushed
 -:  ---------- >  9:  101a31555f refactor: simplify caching setup, add a restore key to actually cache besides 1 run
 -:  ---------- > 10:  580bbe6d1c feat: improve guix building; run always, save artifacts
 -:  ---------- > 11:  770651aa15 set hosts in guix-check
 -:  ---------- > 12:  92f82f6fc6 fix: group the ENV in quotations
 -:  ---------- > 13:  aafded67d9 fix: compilation error due to rebase error between bitcoin#22937 and ipc/process
 -:  ---------- > 14:  540f6871d3 fix: lock `::cs_main` before accessing `ChainstateManager::m_best_header`
 -:  ---------- > 15:  2777d30caa fix(ci): only run some GH actions on PR approval, not on every comment
 -:  ---------- > 16:  c2bd834f3b Merge bitcoin/bitcoin#21169: fuzz: Add RPC interface fuzzing. Increase fuzzing coverage from 65% to 70%.
 -:  ---------- > 17:  4dd36f603a Merge bitcoin/bitcoin#21798: fuzz: Create a block template in tx_pool targets
 -:  ---------- > 18:  813993d44a Merge bitcoin/bitcoin#21892: fuzz: Avoid excessively large min fee rate in tx_pool
 -:  ---------- > 19:  2398283ff6 Merge bitcoin/bitcoin#21922: fuzz: Avoid timeout in EncodeBase58
 -:  ---------- > 20:  3bfefde53b Merge bitcoin/bitcoin#21931: ci: Bump cirrus fuzz CPUs to avoid timeout
 -:  ---------- > 21:  82a6aa5907 Merge bitcoin/bitcoin#21810: fuzz: Various RPC fuzzer follow-ups
 -:  ---------- > 22:  d89847f43e Merge bitcoin/bitcoin#22004: fuzz: Speed up transaction fuzz target
 -:  ---------- > 23:  06ea87040b Merge bitcoin/bitcoin#20773: refactor: split CWallet::Create
 -:  ---------- > 24:<PHONE_NUMBER> fix: follow-up bitcoin#20773 - for collateral lock/unlock coins
 -:  ---------- > 25:  9ad26f1664 fix: follow-up bitcoin#20773 - coinjoin loader can be nullptr too
 1:  c0a5f3c08c = 26:  3fddb84ff4 test: simplify `pow_test`'s `get_next_work block` index construction
 2:  f161a14ac9 = 27:  66490224a5 merge bitcoin#22278: Add LIFETIMEBOUND to CScript where needed
 3:  4b275bd5e3 = 28:  5d82ed7476 merge bitcoin#24103: Replace RecursiveMutex m_cs_chainstate with Mutex, and rename it
 4:  a231d04e3f = 29:  0d457178b4 merge bitcoin#24235: use stronger EXCLUSIVE_LOCKS_REQUIRED()
 5:  ec720b865f = 30:  7a0f22e20a merge bitcoin#24177: add missing thread safety lock assertions
 6:  4be170c323 = 31:  12d6d03e09 merge bitcoin#24299: UnloadBlockIndex and ChainstateManager::Reset thread safety cleanups
 7:  93f06ee075 = 32:  4ce6aa5bcb merge bitcoin#24917: Make BlockManager::LoadBlockIndex private
 8:  94d529e58d = 33:  ea01f62b81 partial bitcoin#22564: Move mutable globals cleared in ::UnloadBlockIndex to BlockManager
 9:  591c837a0c = 34:  464edf8c5d merge bitcoin#25349: CBlockIndex/CDiskBlockIndex improvements for safety, consistent behavior
10:  31afcc41cb = 35:  88f407439a merge bitcoin#25571: Make mapBlocksUnknownParent local, and rename it
11:  7e798d9808 = 36:  5d852c3e23 merge bitcoin#17487: allow write to disk without cache drop
12:  e86dce77a2 = 37:  b119bd4e0d merge bitcoin#26999: A few follow-ups to bitcoin#17487


time keeper component

Fixed test container so the component actually works.
Refactored css and added scalable variables so it can easily scale.
Also tests for functions utils

closes #70


Support for LCID strings (en_us/en-us) on plural forms

Hi,
Is it possible to support LCID strings for the plural forms? we make use of en_us and en_ca rather than the en form, which poses problems when wanting to access the file we want, as getPluralForms function in the util.ts only supports the language codes.
Unless there is an alternative we can employ to circumvent that?
Many thanks

I suggest to use Matice: https://github.com/GENL/matice


Discrepancies between isValid and normalizeEmail

So there are situations where isEmail will return true, but normalizeEmail will return false.
For example:
<EMAIL_ADDRESS>-> isEmail returns true, normalizeEmail returns false.
Really this comes down to a behavior that I think the library should enforce:

If isEmail returns true, normalizeEmail should be able to return a value

Currently normalizeEmail can return false, which seems wrong. It feels like if normalizeEmail can't do what it needs to it should return the original string or null?
Extend isEmail?
Currently normalizeEmail is actually doing domain-specific email validation as well as normalizing the email. For example<EMAIL_ADDRESS>is syntactically valid, but invalid for gmail. Maybe isEmail should also be extended to contain this domain-specific logic to bring it into line with what normalizeEmail is doing.

Agreed, this is confusing behavior of normalizeEmail(). It should always succeed, provided the input email is valid. It should be isEmail() that rejects those particular emails (starting with +). We could add domain-specific validation to isEmail(), but I wonder if<EMAIL_ADDRESS>is even valid (in the general case)?

@tux-tn<EMAIL_ADDRESS>is an invalid gmail address. I think we should add domain-specific validation given how ubiquitous gmail addresses are.

According to [this post|https://www.quora.com/Is-it-currently-possible-to-establish-a-gmail-email-account-shorter-than-6-characters], there are gmail addresses out in the wild that have a username part less than six characters. It's an edge case, but it's possible we ran into one of these today, and now I am faced with having to disable email validation altogether because it is trying to be too clever.


httpCookies option

Now there is. Yesterday I tried to write one in js. But phantomjs doesn't allow access to document.cookie

I hate that we only have pseudo tests.
We would have to read the pdf again and check what's rendered.

Yep have run into that myself writing the renderDelay tests...


Added null check

Added null check

The repo will never return null - if no courses are found, it will return an empty collection.  Is there a reason I might be missing for you to suggest an additional null check?

Ah, ok. This clarifies so much for me. Thanks! Ur courses are awesome btw.

Tnx! :)


Potential bug preventing app from syncing from scratch

Description
while investigating #3167 and going through historical commits using git bisect it output the first commit that supposedly broke syncing a52ed2612d9070af7133dd1c8b3a519d1b9eefaa introduced by this pr https://github.com/celestiaorg/celestia-app/pull/2583

I'm going to start working on this. My hunch is that the app hash mismatch is caused by the fact that main doesn't have an upgrade module in it's v1 yet v1.x has. This is because the upgrade module was extended to include the signalling component for coordination. My plan is to actually separate these out into two modules: an x/upgrade which just includes the ibc stuff and should look the same as v1, and x/signal for coordinating upgrades


Groups

Group 1:
some urls array to get response
Group 2:
some urls array to get response
Group 3:
some urls array to get response
All groups should work in parallel mode.
How to put response from each Group when it will be complete in a separate variables ($group1, $group2, $group3?

Possibly something like this
$groups = array();

$multi_curl = new MultiCurl();
$multi_curl->success(function ($instance) use (&$groups) {
    $groups[$instance->group] = $instance->response;
});

$get_1 = $multi_curl->addGet('https://www.example.com/');
$get_1->group = 'a';

$get_2 = $multi_curl->addGet('https://www.example.net/');
$get_2->group = 'b';

$get_3 = $multi_curl->addGet('https://www.example.org/');
$get_3->group = 'c';

$multi_curl->start();

var_dump($groups);

That's what i search, thanks.
But how to use array of urls in this example instead one url like this:
$get_1 = $multi_curl->addGet('https://www.example.com/');


Updated Readme File

Changed Backslash to forward slash in readme file where you discussed about how to install required dependencies.
Hence Code was not working with backslash in it.

I approve this merge, thank you for your contributions.


Withdraw Shares Tests

Gets us close to parity for withdraw share tests.  The short side is still needed and will land next.  The solidity tests have fuzzing, which are not included in these tests, but can/should be added later.

Codecov Report
Patch coverage: 50.98% and project coverage change: -0.80 :warning:

Comparison is base (cd0e32b) 77.69% compared to head (a7e8208) 76.90%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #324      +/-   ##
==========================================
- Coverage   77.69%   76.90%   -0.80%
==========================================
  Files          26       27       +1
  Lines        2031     2100      +69
==========================================
+ Hits         1578     1615      +37
- Misses        453      485      +32

Flag
Coverage Δ

unittests
76.90% <50.98%> (-0.80%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

elfpy/agents/wallet.py
91.86% <0.00%> (ø)

elfpy/markets/hyperdrive/assets.py
0.00% <0.00%> (ø)

elfpy/utils/apeworx_integrations.py
0.00% <0.00%> (ø)

elfpy/markets/hyperdrive/hyperdrive_actions.py
77.87% <60.95%> (-4.35%)
:arrow_down:

elfpy/markets/hyperdrive/hyperdrive_market.py
73.20% <64.70%> (+0.98%)
:arrow_up:

elfpy/time/time.py
85.41% <100.00%> (+0.97%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Add PEP484 Typing + typing to setup.py

Add in Py2/3 compatible PEP484 typing
Add in typing as a install_requires in setup.py

Fixes coming

No worries. Thanks 👍


Two prompts when uninstalling an extension pack

Testing #14291

install the '999 Martins Favourite Extensions' extension pack
uninstall the extension pack. You are prompted with:

Are you sure to uninstall 'Martins Favourite Extensions'

Press ok

Would you like to uninstall 'Martins Favourite Extensions' only or its dependencies also.

A single prompt would do

Duplicates #14414


"Slow" network issue.

Network was transferring tracks very slowly, then q always didn't reply with a normal response, resulting in connection drop.
Was on Debian 6.0.3 on Thinkpad.

Github weirdly duplicated this...


Document still inserted/removed if my before-hook returns false

In the documentation and in this change (https://github.com/matb33/meteor-collection-hooks/commit/2c986b3aeae7ad3ce3e146669675fd515db346b9) I saw that it shouldn't be possible to insert a document if the hook for before.insert returns false.
As I've tried this in my project, I was able to still insert or remove documents. Here's a very short sample for a test:
const MyCollection = new Mongo.Collection('my-collection');
MyCollection.before.insert(() => false);
MyCollection.insert({});
// MyCollection.find().count() === 0 --> is false

What works is throwing an error ...
const MyCollection = new Mongo.Collection('my-collection');
MyCollection.before.insert(() => { throw new Meteor.Error(0, 'not-allowed'); });
MyCollection.insert({}); // throws an error

But then not all before-hooks are run, which isn't much of a hassle to me, but maybe to others. What I specially dislike about this way is that I now both have to check for the outcome of the method as well as catching the error - and this on every method where I do something on this collection.
EDIT: I could confirm that updates stop, but not insert or remove calls ...

It seems like something on my setup here is ruining it for some collections ... I'm closing it until further notice.


Add policy enforcement to enable/disable logging

Security policy has been extended to support "logging" flag,
which will enable logging inside GCS when set to "true".
By default the logging will be disabled once security policy
becomes available, unless otherwise specified.
Although GCS shouldn't be logging any sensitive information
this change adds extra guarantees that nothing is leaked.
The logging is disabled by setting the logrus output to
io.Discard.
The only drawback is that in SNP scenario the gcs command
is hardcoded and there's a window between the initial
logging configuration inside gcs main and when it's
disabled during setting the security policy. In this case
the vsock dial messages and UVM boot sequence might end up
in the logs, but that shouldn't expose anything.
Signed-off-by: Maksim An<EMAIL_ADDRESS>

@SeanTAllen @KenGordon FYI

@anmaxvl final overall comment that I forgot to leave earlier. I don't feel qualified to comment on the how of turning off logging. I could spend time to get up to speed if you want input on that, otherwise, I will stick to the "policy engine specific" bits that I've commented on.


Update TS Text Processor Example

Summary
Updates the Expression Measurement Typescript Text Processor Example project to use the latest version of the Hume TS SDK (0.8.1).

Successfully ran the example.


[Event Request] Page 7117 "Sales Analysis Report" - protected var

Please add the following Variables ItemAnalysisView, AnalysisLine to protected var in Page 7117 "Sales Analysis Report"
    protected var
        CurrentReportName: Code[10];
        CurrentColumnTemplate: Code[10];
        CurrentLineTemplate: Code[10];
        //>> Begin
        ItemAnalysisView: Record "Item Analysis View";
        AnalysisLine: Record "Analysis Line";
        //<< End

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 12340.


[Figma] Update Modal doc

Dup of this.
https://app.zenhub.com/workspaces/design-system-61e8a2ae77c3a60012e5003c/issues/gh/metamask/design-tokens/441


Added-ML-Cheatsheet

What does this PR do?
Machine Learning Cheatsheet.
For resources
Websites and Pdf
Description
Machine Learning Guide
Why is this valuable (or not)?
Machine Learning Guide
How do we know it's really free?
Its a PDF And Website.
For book lists, is it a book? For course lists, is it a course? etc.
Its a PDF And Website.
Checklist:

[x] Read our contributing guidelines.
[x] Search for duplicates.
[x] Include author(s) and platform where appropriate.
[x] Put lists in alphabetical order, correct spacing.
[x] Add needed indications (PDF, access notes, under construction).
[x] Used an informative name for this pull request.

Follow-up

Check the status of GitHub Actions and resolve any reported warnings!

Thanks, and welcome to free-programming-books!


Too Many Open files error with more remote FTP inputs than ulimit

Snakemake version
latest as of today: 5.19.3+1.g7026550f
Describe the bug
A workflow with too many FTP remote files will fail. Too many is defined by ulimit -n. Using HTTP remotes for the same files does not fail. It looks like maybe the FTP remote mechanism isn't closing file handles after checking remote files.
Logs
[jmeppley@tyrosine test_ftp_bug]$ snakemake -n --config limit=15 protocol=FTP
Building DAG of jobs...
Updating job 0 (all).
Traceback (most recent call last):
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/host.py", line 885, in _FTPHost_dir_command
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/ftplib.py", line 575, in dir
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/ftplib.py", line 468, in retrlines
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/ftplib.py", line 399, in transfercmd
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/ftplib.py", line 361, in ntransfercmd
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/socket.py", line 724, in create_connection
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/socket.py", line 708, in create_connection
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/socket.py", line 144, in __init__
OSError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/__init__.py", line 703, in snakemake
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/workflow.py", line 668, in execute
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/dag.py", line 1179, in update_checkpoint_dependencies
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/dag.py", line 1357, in replace_job
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/dag.py", line 705, in update
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/dag.py", line 788, in update_
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/dag.py", line 705, in update
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/dag.py", line 776, in update_
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/io.py", line 334, in exists
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/io.py", line 174, in wrapper
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/io.py", line 355, in exists_remote
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/snakemake/snakemake/remote/FTP.py", line 206, in exists
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/path.py", line 68, in exists
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/host.py", line 921, in lstat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 835, in _lstat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 801, in __call_with_parser_retry
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 719, in _real_lstat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/path.py", line 145, in isdir
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/path.py", line 125, in _is_file_system_entity
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/host.py", line 937, in stat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 846, in _stat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 801, in __call_with_parser_retry
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 768, in _real_stat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 719, in _real_lstat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/path.py", line 145, in isdir
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/path.py", line 125, in _is_file_system_entity
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/host.py", line 937, in stat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 846, in _stat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 801, in __call_with_parser_retry
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 768, in _real_stat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 719, in _real_lstat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/path.py", line 145, in isdir
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/path.py", line 125, in _is_file_system_entity
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/host.py", line 937, in stat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 846, in _stat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 801, in __call_with_parser_retry
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 768, in _real_stat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 729, in _real_lstat
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 620, in _stat_results_from_dir
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/stat.py", line 613, in _host_dir
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/host.py", line 889, in _dir
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/host.py", line 610, in _robust_ftp_command
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/host.py", line 885, in _FTPHost_dir_command
  File "/mnt/histidine/scratch2/jmeppley/tmp/snakemake_bug/test.env/lib/python3.6/site-packages/ftputil/error.py", line 172, in __exit__
ftputil.error.FTPOSError: [Errno 24] Too many open files
Debugging info: ftputil 4.0.0, Python 3.6.10 (linux)

Minimal example
The snakefile for the above error is:
from snakemake.remote.FTP import RemoteProvider as FTPRemoteProvider
from snakemake.remote.HTTP import RemoteProvider as HTTPRemoteProvider

protocol = config.get('protocol', 'FTP').upper()
if protocol == 'FTP':
    RP = FTPRemoteProvider()
elif protocol == 'HTTP':
    RP = HTTPRemoteProvider()
else:
    raise Exception("Unknown protocol " + protocol)

# remote loaction
REMOTE_TEMPLATE = \
    'ftp.ncbi.nlm.nih.gov/refseq/release/complete/{fragment}protein.gpff.gz'
LOCAL_TEMPLATE = 'download/{fragment}protein.gpff.gz'

# find files to download
fragments, = RP.glob_wildcards(REMOTE_TEMPLATE)

# limit download so test doesn't take forever
limit = config.get('limit', 2500)
rule all:
    input: expand(LOCAL_TEMPLATE, fragment=fragments[:limit])

rule dl_proteins:
    input: RP.remote(REMOTE_TEMPLATE)
    output: LOCAL_TEMPLATE,
    shell: """ ln -s ../{input} {output} """

Note: you get to the error faster if you shrink your ulimit first: ulimit -n 10 and run with a small file limit `snakemake --config limit=15 -n'.

I just found the immediate_close option.


{Type: VarInt, Source #0: us.myles.ViaVersion.protocols.protocol1_9to1_8.packets.PlayerPackets$17 (Anonymous), Data: [], Packet ID: 10}

output link of /viaversion dump
Plugins: ( a lot self-coded )
AsyncWorldEditInjector, WorldEdit, CommandBlockerUltimate, Votifier, WorldGuard, ReloadWarning, AntiTNT, AsyncWorldEdit, MaxBans, Team-Chat, McModknowerLib, ChatChanneler, EasyFriede, ViaVersion, Vault, CombatLogX, Lores, CreativeItemControl, Essentials, McModknowerSpielmodusManager, FirstSpawnMessage, PlotSquared, Discord, Multiverse-Core, Holograms, Modul_CMDBlocker, WorldProtect, Shopkeepers, McModTechVote, PermissionsEx, Kein_Fallschaden_Modul, ChatEx, Tablist, Clearlagg, Respawnkit_Modul, ArmorStandTools, Lobby, SkyPvP
Console output that told me to create the issue:
[16:53:12] [Server thread/INFO]: Crazygamer263 lost connection: Internal Exception: io.netty.handler.codec.DecoderException: us.myles.ViaVersion.exception.InformativeException: Please post this error to http://github.com/MylesIsCool/ViaVersion/issues
{Type: VarInt, Source #0: us.myles.ViaVersion.protocols.protocol1_9to1_8.packets.PlayerPackets$17 (Anonymous), Data: [], Packet ID: 10}

Is there more to this error? If not could you provide more via debug=true in server.properties?
Is there a cause for this error? Does it happen when you equip armour etc with a plugin?

I just saw the line while i looked through the latest.log file and it happend only once.

I can't really help apart from the above, this will be down to plugins.
The most common one is people sending bad armour equip packets.
(Best idea is if you can consistently replicate, remove plugins + test)


Is it possible observe connection with client?

Is it possible observe connection with client?  When we turn off wifi on client - socket didnt close. 'setReadTimeout' works like reconnection with timeout.

Problem was resolved, with ping-pong messages


removing emannotationschemas code from package

merging to get casey up to speed... will do this in the future though with review


Merge asset manifests on install/uninstall

We obviously need to copy the manifests in a directory on install to
make sure we keep them. Note that on install, if we see the barclamp
doesn't have a manifest, we ensure that there's no old manifest
installed for that barclamp.

Please use Pathname, it makes multiple file handling much easier like join, or read without calling seperate File.

def manifests_path
  @manifests_path ||= Pathname.new(CROWBAR_PATH).join("barclamps", "manifests")
end

@tboerger actually, I'd prefer to avoid this: this makes a totally different style than what exists in the same code path (in bc_install_layout_1_ap), which makes it hard to read. I'm happy to switch to Pathname, but really, we should switch everything to it in that case. And I'd prefer another PR for this.

I already mentioned that on the other mgmt lib improvement (related to migrations) opened by you. IMHO the Pathname approach makes it much better readable because you can simply call nearby all File, FileUtils and more methods on the Pathname object.

@tboerger I don't disagree it's a good move; I disagree here because if you look a few lines earlier, we have the same code already not using Pathname. So we should do the move, but switching everything together, to avoid having two similar code paths look totally different.

So maybe we need to solve that later on...

At this point, this has been tested as a package patch for quite some time => merging.


Adapts ZEP2 to codec

Makes changes to the current ZEP2 draft to switch sharding from the storage transformer model to a codec.

@joshmoore @MSanKeys963 @rabernat Could one of you merge this PR? This also reflects the changes from https://github.com/zarr-developers/zarr-specs/pull/223

Thanks for sending the PR, @normanrz.
Merging this now.


Update tarjan_test.cpp

std::sort fix

Hi @hxm-cpp, thanks for opening this PR!
It is indeed preferred to explicitly include the headers which are used, as opposed to including these transitively. Out of interest, did you have any issues building the tests without this include?

Hey @bobluppes since you never recieved an awnser for this, at least for me, using cmake --build . did give me an error because id didn't find std::sort, after including algorithm in tarjan.h (i did that in #217 since I didn't see this merge request) the project built without issues.

Hi @hxm-cpp, thanks for opening this PR!
It is indeed preferred to explicitly include the headers which are used, as opposed to including these transitively. Out of interest, did you have any issues building the tests without this include?

Hey @bobluppes since you never recieved an awnser for this, at least for me, using cmake --build . did give me an error because id didn't find std::sort, after including algorithm in tarjan.h (i did that in #217 since I didn't see this merge request) the project built without issues.

Hi @FlorianXXIV, thanks for confirming! I was now able to reproduce this locally as well, not sure why this was broken before though. I will merge this PR after which I can rebase yours 👍🏻


CVAT-3D Milestone1

CVAT-3D : Create a 3D task - Support to upload PCD/BIN files in archived zip.
Changes include:
Added Bin as accepted MIME type under Image Set
Added Dimension Validation Class to verify PCD vs PhotoCD and BIN to PCD conversion.
Added Support for Point Cloud and BIN Files for 3D Tasks.
Added an extra attribute dimension across API's and Formats.
Modified MPEG and Zip Chunk Writer to support PCD
Test Cases:
Manual Unit testing done locally.
Existing test cases work as expected.

[x ] I submit my changes into the develop branch

[ x] I have added tests to cover my changes

[ x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.

I will review the patch  tomorrow. Please resolve conflicts.

@dvkruchinin , could you please help us to investigate problems with tests in the PR? Please put here comments ASAP.
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
9793  │ Tests:        6                                                                                │
9794  │ Passing:      0                                                                                │
9795  │ Failing:      2                                                                                │
9796  │ Pending:      0                                                                                │
9797  │ Skipped:      4                                                                                │
9798  │ Screenshots:  2                                                                                │
9799  │ Video:        false                                                                            │
9800  │ Duration:     56 seconds                                                                       │
9801  │ Spec Ran:     actions_tasks_objects/case_24_delete_unlock_lock_object.js                       │
9802  └────────────────────────────────────────────────────────────────────────────────────────────────┘
9803
9804
9805  (Screenshots)
9806
9807  -  /home/travis/build/openvinotoolkit/cvat/tests/cypress/screenshots/actions_tasks_     (1280x720)
9808     objects/case_24_delete_unlock_lock_object.js/Prepare to testing (failed).png
9809  -  /home/travis/build/openvinotoolkit/cvat/tests/cypress/screenshots/actions_tasks_     (1280x720)
9810     objects/case_24_delete_unlock_lock_object.js/Delete unlocklock object -- Create
9811     and delete object via Delete shortcut -- before all hook (failed).png
9812
9813
9814────────────────────────────────────────────────────────────────────────────────────────────────────
9815
9816  Running:  actions_tasks_objects/case_25_canvas_bitmap_feature.js                        (18 of 72)
9817
9818
9819  1) Prepare to testing
9820  Canvas bitmap feature
9821    2) "before all" hook for "Bitmap not visible."
9822
9823
9824  0 passing (57s)
9825  2 failing
9826
9827  1) Prepare to testing:
9828     AssertionError: Timed out retrying: expected 'http://localhost:8080/auth/login' to include '/tasks'
9829      at Context.eval (http://localhost:8080/__cypress/tests?p=cypress/support/index.js:678:12

@manasars , please investigate problems with tests in parallel. All tests should be passed to merge a PR into our repo.

Please put here comments ASAP.

@nmanovic
All tests are not successful in this PR due to an error when creating django superuser
django.db.utils.ProgrammingError: relation "auth_user" does not exist
LINE 1: INSERT INTO "auth_user" ("password", "last_login", "is_super...
As a result, it was not possible to log in to the application under the superuser.

In this case, an error occurred in the test issue_1568_cuboid_dump_annotation.js
Due to the lack of opening submenu for Menu - > Dump annotations.
How to reproduce:

Create a task
Go to menu
Try to move cursor to "Dump annotations" or "Upload annotations" or "Export as a dataset". Submenu should appear.
But it doesn't appear.

Hi @nmanovic,
We have generated synthetic data for pcd, bin and img test files.
Resolved build errors, 1 test case shows as pending and notified the same on mail.
Kindly review PR.
Thank You.

@manasars ,
Could you please inspect logs from travis-ci next time before ask me to review the patch again?
raceback (most recent call last):
  File "/home/django/utils/cli/tests/test_cli.py", line 41, in setUpClass
    super().setUpClass()
  File "/opt/venv/lib/python3.8/site-packages/django/test/testcases.py", line 1123, in setUpClass
    cls.setUpTestData()
  File "/home/django/utils/cli/tests/test_cli.py", line 54, in setUpTestData
    create_db_users(cls)
  File "/home/django/cvat/apps/engine/tests/test_rest_api.py", line 45, in create_db_users
    user_admin = User.objects.create_superuser(username="admin", email="",
  File "/opt/venv/lib/python3.8/site-packages/django/contrib/auth/models.py", line 157, in create_superuser
    return self._create_user(username, email, password, **extra_fields)
  File "/opt/venv/lib/python3.8/site-packages/django/contrib/auth/models.py", line 140, in _create_user
    user.save(using=self._db)
  File "/opt/venv/lib/python3.8/site-packages/django/contrib/auth/base_user.py", line 67, in save
    super().save(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 753, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 790, in save_base
    updated = self._save_table(
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 933, in _do_insert
    return manager._insert(
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1249, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed: auth_user.username
----------------------------------------------------------------------
Ran 132 tests in 45.361s
FAILED (errors=30)
Destroying test database for alias 'default'...
The command "docker-compose -f docker-compose.yml -f docker-compose.ci.yml run cvat_ci /bin/bash -c 'coverage run -a manage.py test cvat/apps utils/cli && mv .coverage ${CONTAINER_COVERAGE_DATA_DIR}'" exited with 1.

Hi @nmanovic ,
The ci/cd build has passed, latest code pushed to PR.

@manasars , an extra level of directories + temporary files inside archives. Could you please fix that?
test_pointcloud_pcd.zip\test_pointcloud_pcd\test_pointcloud_pcd\related_images\000001_pcd

@manasars , an extra level of directories + temporary files inside archives. Could you please fix that?
test_pointcloud_pcd.zip\test_pointcloud_pcd\test_pointcloud_pcd\related_images\000001_pcd

The same problem with another archive.

Coverage increased (+0.2%) to 62.962% when pulling 6879381e82f5788fa52625045752f21dca8c9c8e on manasars:cvat-3D-M1 into 9bce3d6f4aa4e537d16b2981c4d2a2dacd755082 on openvinotoolkit:develop.

@bsekachev , do you have any questions to the client or server part? Could you please review?

@nmanovic
I do not have critical comments about the patch. Looks like my comments and the most comments of other guys have been resolved. There are several unresolved conversations above opened not by me.
Also I've noticed newly appeared client-side code issues (not critical) and the most of them are expected to be highlighted or even fixed automatically by static linters on precommit hook. Apparently it doesn't work or just ignored. And I believe we should clarify this moment with @manasars because getting rid of such the issues on code review was exact purpose of the linters integration.

Hi @bsekachev ,
The pre-commit hook had stopped working earlier due to some git config.
We have modified the same and now the lint changes are checked.
Fixed all the mentioned lint errors.


Make installed system boot experience more like Live ISO

Make installed system boot experience more like Live ISO, in part by utilizing the mechanism discussed in https://github.com/helloSystem/ISO/issues/312.

[ ] Splash screen
[ ] No text consoles unless booted in verbose mode
[ ] rcorder optimization to start Xorg and helloDesktop as early in the boot process as possible

"Splash screen" has been resolved.


Remove experimental redacting logger filter

Changes
This PR removing experimental basic, best-effort, logging filter to redact secrets from accidental logging. sys.stdout, sys.stderr, and print() statements are not handled.
This won't work anymore:

Tests

[x] make test run locally
[x] make fmt applied
[x] relevant integration tests applied

Codecov Report
Patch coverage has no change and project coverage change: -0.09 :warning:

Comparison is base (c578001) 52.80% compared to head (d05ed86) 52.72%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #92      +/-   ##
==========================================
- Coverage   52.80%   52.72%   -0.09%
==========================================
  Files          29       29
  Lines       17103    17065      -38
==========================================
- Hits         9032     8998      -34
+ Misses       8071     8067       -4

Impacted Files
Coverage Δ

databricks/sdk/dbutils.py
92.48% <ø> (+0.66%)
:arrow_up:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


1.14 Jenny Generation causes compilation errors when the project is opened with Visual Studio for Mac

Jenny Generation causes compilation errors when the project is opened with Visual Studio for Mac
Steps to reproduce the behavior:

Open Match-One in a mac with Visual Studio for Mac
Generate with dotnet Jenny/Jenny.Generator.Cli.dll gen
Go to Unity and see that there are compilation errors about duplicated and missing classes

Expected behavior
No compilation errors after code generation
Screenshots

Entitas Version

Version Entitas 1.14

Additional context
I think it happens with Visual Studio for Mac and with Visual Studio Code but for sure not with Rider
Zip with the project
The zip contains all the project files after opening it with Visual Studio for Mac and after Jenny Generation was executed so it has compilation errors. Also, I had to remove the Library folder because with it the zip was too big to update to GitHub issues.
Match-One_IDE_VisualStudioForMac_GeneratedBroken 2.zip

@AlberGanAvi I can confirm and reproduce.
Unity generates a different csproj file depending on Rider or VS code, which I find surprising. The Roslyn code generator plugin uses standard practices and no special customization, so the VS code produced csproj seems off. I will investigate and check the differences.

Among other things, this is missing when not using Rider. Adding those fixes the issue for me. Unfortunately, those lines will be removed when csproj is regenerated. Will investigate more
<_TargetFrameworkDirectories>non_empty_path_generated_by_unity.rider.package</_TargetFrameworkDirectories>
<_FullFrameworkReferenceAssemblyPaths>non_empty_path_generated_by_unity.rider.package</_FullFrameworkReferenceAssemblyPaths>
<DisableHandlePackageFileConflicts>true</DisableHandlePackageFileConflicts>

Will fix using a Jenny PreProcessor

Fixed in Jenny.Plugins 1.1.0
Reattached new Jenny.zip to Entitas 1.14.1
https://github.com/sschmid/Entitas/releases/tag/1.14.1
You can add a new pre processor Jenny.Plugins.ReferenceAssemblyPathsPreProcessor to Jenny.properties like in Match-One
https://github.com/sschmid/Match-One/commit/ad34a99ba17e8cd95b698e21384bd7292bc2e5ff


Handle reserved mixpanel properties and GeoIP for profiles

Description

Adds special handling for $ip.
Motivation and Context

Mixpanel is using many properties which have special meaning. We can use them to make created profile better.
Unfrotunatelly this information isn't anywhere in API specification and I found it when I was solving what to do with server side tracking and Geoip.
https://help.mixpanel.com/hc/en-us/articles/115004708186
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have read the CONTRIBUTING document.
[x] I haven't repeated the code. (DRY)
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

It is customProperty because it is not usual property tracked by CRMs.


Fix key error

Main change is if we can't find a saved setting on whether the class is enabled or not, just make it enabled. The other place this could be wrong is in the Settings instance declaration Possessor is not anywhere to be found. Regardless, I like having the default.
Invert the if statement for less nesting, and put the named into a variable since that's an expensive grab.
Don't clear the new dictionary. It doesn't do anything since a new dictionary is ALWAYS empty unless a different initializer is used.

Accepted and update in v<IP_ADDRESS>


SubViewport display a pink/blue square

Godot version
Godot 4 alpha 13
System information
MacOS 11.6
Issue description
I was unable to use a SubViewport as a texture:

Steps to reproduce
Add a SubViewport and add the Godot icon inside it  and check the preview.
Minimal reproduction project
No response

Can you reproduce this in previous 4.0 alphas? You can download them here;

@Calinou Yes in alpha 13-12-4-2 and the master, I also tried #63306 but doesn't fix the issue.

I tried #62938 #63091 they don't help either.

same issue on windows with same scene and with 3d camera (on alpha 13, 11, 10, 9)

(alpha 13)
Setting size_2d_override_stretch to true and render_target_update_mode to Always solves this problem on my end.

@mieldepoche thanks! Indeed put render_target_update_mode to always solve the issue.
The issue seems to be that the Viewport is not detected as being visible (although it is).

This seems to be a duplicate of #55471, I have a try in #62854.
Refer to https://github.com/godotengine/godot/issues/55471#issuecomment-1179535202, https://github.com/godotengine/godot/pull/62854#issue-1299673343.

Duplicate of #55471.


R2DBC API: Expose product name

ConnectionFactory should expose a product name, ideally as part of driver-metadata. Optionally, the metadata could also expose server-metadata.
The product name is a useful detail when configuring SQLErrorCodeSQLExceptionTranslator.

What would you think this should look like?  Something like Map<String, String> getMetadata() or more specific Metadata getMetadata()?

Let's stick with a Metadata getMetadata() to give value objects a meaningful name.

Sounds good.  Pushed through with a static name on the PostgreSQL implementation.  I'd expect to extend both the interface and PostgreSQL class with new, strongly named and typed methods based on feedback.


Add coverage for some API subcollection actions

Moves the existing tags subcollection tests (which were based on vms) under the vms specs
Adds some tests for tags subcollections of groups, users and templates
Adds a test for service request subcollection of service templates

These tests describe the behavior affected by https://github.com/ManageIQ/manageiq/pull/11694
@miq-bot assign @abellotti
@miq-bot add-label api, test

should we backport to Euwe release since it's related to #11694 ?
(helping minimizing conflicts in other backports to E-release)

@abellotti I'm thinking yes, since it should really have been part of the same revision. But not sure what hte protocol is :smile:

usually both together, but we need that fix quick.  I'll mark as Euwe-yes.  Thank you!!.

Euwe Backport details:
$ git log
commit 80309ecdcf182161d0d15b382b02d56cea55be21
Author: Alberto Bellotti<EMAIL_ADDRESS>Date:   Thu Oct 6 10:10:23 2016 -0400

    Merge pull request #11722 from imtayadeway/api/test/subcollection-actions

    Add coverage for some API subcollection actions
    (cherry picked from commit 34894fcc411b47afea491b587608a559c9357572)


:sparkles: Inventory policy

Summary

support k8s-policy
update to remove the creationtimestamp. see https://github.com/kubernetes-sigs/controller-runtime/blob/master/FAQ.md#q-how-do-i-have-different-logic-in-my-reconciler-for-different-types-of-events-eg-create-update-delete

Related issue(s)
Fixes #
Tests

[x] Unit/function tests have been added and incorporated into make unit-tests.
[ ] Integration tests have been added and incorporated into make integration-test.
[ ] E2E tests have been added and incorporated into make e2e-test-all.
[ ] List other manual tests you have done.

/test test-integration

/cherry-pick release-2.12

/hold

/test test-e2e

/unhold

/lgtm


set explicit protobuf package names

This PR is fixing a problem with namespacing protobuf message types with proto.RegisterType which registers protobuf messages globally under names that contain protobuf package name and message name. Currently we are using the same name for protobuf package to get the same go package name pb. This PR changes that to have different name for protobuf and its namespacing and an explicit one for go package name.
This issue was discovered on the same named (but different) message Delivery in retrieval and pushsync. The mistake is from the start how package names are referenced and this PR fixes that for all.

Wow. This is pretty serious. How did you catch this and how did this not come up in the tests?

Tests are not registering all protocols, and the runtime error is just logged by protobuf library, not panicked. Also this did not come up as pushsync protocol was not added as protocol to p2p service (another pr).


Implement change note and update type feature

This implements the change note and update type feature.

Users can now add a change note and select an update type in the edit screen.
When a document is first created, it has a "major" update type by default because it'll be new. ~Users can still change this.~ This is unchangeable.
The update type and change note are sent to the publishing-api, so change notes for major updates will be displayed on the site. The update type is no longer part of the publish call.
The change note is displayed on the summary page as well.
After a document is published, we clear the update type and change note.
I've added a test for the creation of a document, because there wasn't a feature test that tests all the things that need to be set up when the document is first created.

Known issues

The layout of the new form labels will be fixed by @alex-ju and https://github.com/alphagov/govuk_publishing_components/pull/508.
If users don't select an update type, the publishing-api will reject the update. This needs a new validation rule, which will be part of https://trello.com/c/K52YqwPK.

Preview

https://trello.com/c/NEQWHpoU

@kevindew updated with behaviour to not allow update type change for the first edition.

Yep. I've added this to https://trello.com/c/K52YqwPK to the acceptance criteria. Can't be a nil column though, because we'll still allow users to save the page without a note (just not publish).

Oh yes good point

Mwoah, I think having a named state is preferable in this case because it better matches the UI and mental model of users. Do you feel strongly about this?

No not too strongly, it's consistent with our other states - yet another enum situation :( . Might be prudent to have a similar validation in: https://github.com/alphagov/content-publisher/blob/master/app/models/document.rb#L24-L25

@kevindew updated with behaviour to not allow update type change for the first edition.

Awesome, do we need this to then have a default one that it does on first edition here: https://github.com/alphagov/content-publisher/pull/236/files#diff-9ebabcf3d75c8d23c983f107afd49258R24 ?

@kevindew the default for the first edition is done here:
https://github.com/alphagov/content-publisher/blob/54abab93688968ee23ba6d623b624a07c4ca96bf/app/controllers/new_document_controller.rb#L41-L50
And tested here:
https://github.com/alphagov/content-publisher/blob/54abab93688968ee23ba6d623b624a07c4ca96bf/spec/features/create_a_document_spec.rb#L38-L40

@tijmenb Ah sorry missed that - should that have a change note of "First published." too?

@kevindew updated, it now has the correct defaults. Squashed the commits.


Not possible to choose a specific server

It´s not possible to choose a specific server. If you edit Results > server it will be overwritten after next update of Top 5 servers.

Thats as designed :-) this state Shows the Last Server tested
I can implement e new state, how would you like ?
Scan by id or url ?
Just for one run or permernantly

implement in 0.1.3 https://github.com/iobroker-community-adapters/ioBroker.web-speedy/commit/c83b39d9e6e8dec52a4a971a049efb7b3ecdc6c6

URL and permernantly. Please. ;)

url added in 0.1.6 https://github.com/iobroker-community-adapters/ioBroker.web-speedy/commit/091f7b66d71b809a82087aeddc119fcb23e9edd0


Parameters saving like empty string in DB

Hello!
Sending MenuItem data with empty string ""

In my db (MySQL 8.0.14) it saving like empty string too:

And after, i have problem with methods getDisplayValue() and getValue() in my custom linkable models
I resolved for my models (remove type for parameter)
public static function getDisplayValue($value = null, $parameters = null)

But i have problem with MenuItemStaticURL becouse $parameters must be array or null
May be default value for parameters in NovaMenu.vue should be null?
Or i have another resolve for this problem?

That's really weird. I can't reproduce this at all.
With this request payload:

When I dd() the result out in the controller, I receive:

Where parameters is null. I don't really have a clue why it's loading as '' for you.
Can you try adding this to right before line 63 in MenuController.php?
// Add fail-safe due to https://github.com/optimistdigital/nova-menu-builder/issues/41
$data['parameters'] = empty($data['parameters']) ? null : $data['parameters'];

Like so:

And let me know if this would fix it for you?

@Tarpsvo, yes, this fix my problem. Thanks, i hope it will be in new release.

Fix released in version 2.2.2, good luck!

@Tarpsvo Hello again :D
I have same problem, now when update menu items. I think this
$data['parameters'] = empty($data['parameters']) ? null : $data['parameters'];

need add to MenuController.php 94 line


❓ Preprocessing of Reddit TIFU for BART/BART-RXF

Hello,
thanks for providing the code for the BART-RXF model!
I am trying to reproduce the results from the paper "Better Fine-Tuning by Reducing Representational Collapse" (https://arxiv.org/pdf/2008.03156.pdf) with the BART-RXF/BART on Reddit_tifu.
For now I am using the provided dataset from the paper "Extractive Summarization as Text Matching" (https://github.com/maszhongming/MatchSum. The authors provide train, dev and test splits.
Now I have two questions:

Is the same data split used for the Representational Collapse paper or did you use a "custom" split?
Is any preprocessing like BPE applied to the reddit data?

Thank you very much.
@ArmenAg

@PhMeier Hey! Apologies for the late response. We use the same split that is available in the huggingface RedditTIFU dataset, specifically the short version. And yes you will need to apply the same pre-processing that is described here: https://github.com/pytorch/fairseq/blob/main/examples/bart/README.summarization.md
Let me know if you need any more help!

Dear Armen,
thank you very much for your kind reply, i really appreciate that!
Great to hear about the preprocessing.
Unfortunately I only find a "train" split on the huggingface website (https://huggingface.co/datasets/reddit_tifu) and no "test" split, maybe I missed something.

@ArmenAg  Hi Armen,
Any update on this? I see that there is a PR for the RXF model on the sentence prediction task, is it useable for summarization (BART) as well? I can't find a similar script for summarization...

Another question, @ArmenAg the huggingface doesn't provide splits for the Reddit_TIFU dataset. It only has the training split, for both long and short versions. I assume you have used the 80-10-10 split suggested by Pegasus authors for the purpose of training and validation, right?


Speed up layout of text boxes

Processing some PDFs takes a lot of time in layout.py; in particular (for one PDF I've tried) in LTLayoutContainer.group_textboxes().
This PR has two changes:

Remove utils.csort() and replace all uses with native list.sort()
In group_textboxes(), make dists a sortedcontainers.SortedListWithKey() instead of a list, which removes the need to re-sort dists repeatedly

Note that replacing csort() (which uses "Decorate-Sort-Undecorate" to ensure the sort is stable) with the native sort doesn't change the behaviour, since Python sorting is stable since Python 2.2.
For the test PDF I have (which motivated these changes), running pdf2txt.py now takes around 7 seconds instead of 17 seconds.
(I also tested these changes with the file in #128, but no speed up was seen for that input.)

For now, I am merging this. Might take a look at the requested changes later.