Entitas with Roslyn Code Generation via dotnet IIncrementalGenerator

[ EDIT by @sschmid ]

I edited the issue description because this issue is now the main issue for the new code generator

dotnet's source generators, specially IIncrementalGenerator may be a valid alternative to the current approach with Jenny
Learn about dotnet Incremental Generators:

https://github.com/dotnet/roslyn/blob/main/docs/features/incremental-generators.md

While migrating to dotnet source generators, I will use the opportunity to update and improve the generated code:

add namespaces support for components and contexts
support multiple projects and multiple Unity asmdef

I already made good progress but it's still in research state. Any help from the community is greatly appreciated! Please feel free to engage and help in the conversation if you can! 🙏

Tasks
Generators

[x] Component ContextApi Generator (Unique, Normal, Flag)
[x] Component EntityApi Generator (Normal, Flag)
[ ] ? Component EntityApiInterface Generator (Multiple Contexts)
[x] OBSOLETE Component Generator
[x] Component Lookup Generator
[x] OBSOLETE Component MatcherApi Generator
[x] ContextAttribute Generator
[x] Context Generator
[x] ContextMatcher Generator
[x] OBSOLETE Contexts Generator
[x] Entity Generator
[ ] EntityIndex Generator
[ ] Event EntityApi Generator
[ ] Event ListenerComponent Generator
[ ] Event ListenerInterface Generator
[ ] Event System Generator
[ ] Event Systems Generator

Attributes

[ ] CleanupAttribute
[x] ContextAttribute
[ ] CustomEntityIndexAttribute
[ ] EntityIndexAttribute
[ ] EntityIndexGetMethodAttribute
[ ] EntityIndexType
[ ] EventAttribute
[x] OBSOLETE PostConstructorAttribute
[ ] PrimaryEntityIndexAttribute
[x] UniqueAttribute

original message from issue author:
Hi,
I have a suggestion and would want to contribute.
As in official code generation guide by Unity - https://docs.unity3d.com/Manual/roslyn-analyzers.html - we can use separate project and create dll which unity will use right in it's code compilation steps.
We already have a separate project for it - Jehny, all we need is to make sure that code gen is done by Microsoft Roslyn Source Generators, and put a label “RoslynAnalyzer”  for the DLL inside the release branch (create a Unity package)
This way we don't need to use Jehny Server for constant monitoring of changes, and it will help clean up the workflow.
Here's some code that we need to use for Roslyn Source Generators
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Text;
using System.Text

[Generator]
public class ExampleSourceGenerator : ISourceGenerator
{
}

Hope it will be helpful.

@studentutu thanks! Roslyn Source Generators is definitely sth I would like to research at some point! That might be useful for Entitas. Jenny however follows a different idea where the input can be anything, not just source code or assemblies. It more general purpose for any kind of code generation

One downside is

Note: Roslyn analyzers are only compatible with the IDEs that Unity publically supports, which are Visual Studio and JetBrains Rider.

But I'll have a look

Looks like we would need to switch to Unity when we make changes in order to recompile, which would take too long.

Ok, building in the IDE works too 👍

yep, definitely better with IIncrementalGenerator

One thing of note though: I'm having severe problems with AdditionalFiles in Unity 2021.3 - that's files you want to pass along to source generators outside of source code. So for example if you had some csv files you would want to use as a source to generate from.
The Unity docs on this are kind of sparse - and the one page that does mention this describes an implementation that:
a) does not work - at all - as described
b) is of questionable design, given that one needs to rename files to somename.[Gernerator].additionalfile
So if that is something that one wants to support I'd wait for Unity to actually switch to permanent, modern csproj files and using MsBuild, as outlined in their roadmap talk. Although it is a bit bold to assume that this will just work then. They usually manage to do stuff that is not in line with what .NET devs are used to ;).

Hi, I started testing IIncrementalGenerator and started a new branch. I added Entitas.Generators and Entitas.Generators.Tests projects to get started.
https://github.com/sschmid/Entitas/tree/roslyn-source-generators/src/Entitas.Generators
https://github.com/sschmid/Entitas/tree/roslyn-source-generators/tests/Entitas.Generators.Tests
So far I'am happy with IIncrementalGenerator performance and will look more into it.

If all goes well I can imagine that it can replace Jenny and setting up Entitas will be much easier.
I started with snapshot testing to verify the output of the source generators, it's pretty cool. A test looks like this
[Fact]
public Task Test() => TestHelper.Verify(
    GetFixture("NamespacedComponentWithOneField"),
    new ComponentGenerator());

More about snapshot testing:
https://andrewlock.net/creating-a-source-generator-part-2-testing-an-incremental-generator-with-snapshot-testing/

I made some progress using IIncrementalGenerator. It's great. But now I was testing it with the latest Unity 2022.3 LTS and it looks like IIncrementalGenerator is not yet supported. Can this be true? I hope I'm wrong!

Ok, got it to work!
Unity docs say you should use this specific version: Microsoft.CodeAnalysis 3.8
https://docs.unity3d.com/Manual/roslyn-analyzers.html
This version does not contain IIncrementalGenerator.
The current version is 4.6.0, but that one does not work in Unity.
I manually checked each version to find the latest one that works, which is
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.1.0" />
And you can use netstandard2.1
<TargetFramework>netstandard2.1</TargetFramework>

Currently stuck, because it seems like you cannot resolve generated attributes, e.g.
similar to the current approach I wanted to generate a convenience context attribute for each context, so you can add this attribute to components as usual
[MyApp.Main.Context, Other.Context]
partial MovableComponent : IComponent { }

But since those attributes are generated, the don't seem to be part of the compilation for looking up symbols 😭
This was easily possible with Jenny...
Any ideas how to solve this?

To be more specific:
Testing with non-generated attributes, I can easily get the attribues like this
var attribute = symbol.GetAttributes().First();

With the generated ones the same code returns ErrorTypeSymbol instead of the attributes

@sschmid  you can use following

Create a list of ICustomGenerators with Execute(SourceProductionContext cx, Compilation compilation, other params if needed)

For each generator create a separate class:

///
///   Generates systems for step ..... for all components.
///
[Generator(LanguageNames.CSharp)]
public class SystemsDescriptorGenerator : IIncrementalGenerator, ICustomGenerators
{
void ICustomGenerators.Execute(SourceProductionContext cx, Compilation compilation, other params)
{
// Do work.
cx.AddSource($"{cx}.{Attribute}CustomStep{component.Name}.generated", GeneratedCode(cx, component));
}

Don't forget to add to the YourComponents.Generators.csproj

<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.0.1" PrivateAssets="all" />
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.3" PrivateAssets="all" />

And for the actual game /Runtime .csproj:
<ItemGroup>
<ProjectReference Include="..\YourComponents.Generators.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />

afaik step 3. and 4. are already part of the current setup.
Can you provide documentation for ICustomGenerators interface?
Is it part of roslyn or a concept you introduced yourself?
what's calling ICustomGenerators.Execute?
I don't see how this makes generated code of other incremental generators available to the input compilation of this generator. Can you explain how your approach solves/bypasses this?

@Ants Aare
ICustomGenerators  is just a custom interface, for use with a single generator
Basically, you begin with a normal Generator, that includes a list of other generators (list of ICustomGenerators ).
Then in the main Generator, you simply execute each custom one by providing with GeneratorExecutionContext  and Compilation into them.
So by using a predefined ordering, you will get a correct compilation of source generators.

Maybe I'm understanding it wrong, but I don't think this solves the problem(?) What you're describing is just a way to call multiple methods one after each other through an interface inside a regular sourcegenerator. First of all this doesn't include recompilation steps inbetween the calls to ICustomGenerators, which would be neccessary to get the symbols as INamedTypeSymbols using GetAttributes(). Secondly this would downgrade the current solution to a normal ISourceGenerator instead of an IIncrementalGenerator.
Btw: The order in which we call cx.AddSource is irrelevant as all files will be added to the compilation in the same pass once the generator finishes executing. No matter if we're using SourceProductionContext or GeneratorExecutionContext

right, missed the part of the "generated source code with new attributes".

Hi, a quick update and some thoughts:
I have a working proof of concept using IIncrementalGenerator.
Main goal of this new generator is easy of use (no Jenny setup needed anymore) and support for namespaces in components and contexts + support for Unity asmdefs.
As previously explained in other GitHub issues, it's necessary to change the generated code to not used partial on the Entity classes (and others), but instead use static C# extension methods. This way it's possible to get a similar and familiar API as we have now, but at the same time support asmdefs. We also cannot pre-generate all component indexes anymore, as new components may be introduced from other dlls. So there are still a few challenges ahead, but here's a quick sample of how it could like based on my current generators:
Contexts
Now with namespace support!
Works with or without a namespace.
You can define contexts in your code like this:
// MainContext.cs
namespace MyApp
{
    partial class MainContext : Entitas.IContext { }
}

// OtherContext.cs
partial class OtherContext : Entitas.IContext { }

Components
Now with namespace support!
Works with or without a namespace.
You can define components in your code like this:
// MovableComponent.cs
namespace MyFeature
{
    [MyApp.Main.Context, Other.Context]
    partial class MovableComponent : Entitas.IComponent { }
}

// PositionComponent.cs
namespace MyFeature
{
    [MyApp.Main.Context, Other.Context]
    partial class PositionComponent : Entitas.IComponent
    {
        public int X;
        public int Y;
    }
}

The generated component extensions work for all specified contexts and can be chained:
MainContext mainContext = new MyApp.MainContext();
MyApp.Main.Entity mainEntity = mainContext.CreateEntity()
    .AddMovable()
    .ReplaceMovable()
    .RemoveMovable()
    .AddPosition(1, 2)
    .ReplacePosition(3, 4)
    .RemovePosition();

OtherContext otherContext = new OtherContext();
Other.Entity otherEntity = otherContext.CreateEntity()
    .AddMovable()
    .ReplaceMovable()
    .RemoveMovable()
    .AddPosition(1, 2)
    .ReplacePosition(3, 4)
    .RemovePosition();

Matchers
I currently generate component indexes for each context, e.g.
MyFeaturePositionComponentIndex. They also include the namespace.
I might use them later to assign component indexes ones the app starts.
Matcher.AllOf(stackalloc[]
{
    MyFeaturePositionComponentIndex.Value,
    MyFeatureMovableComponentIndex.Value
});

More updates:
Yay, added component deconstructors
var (x, y) = entity.GetPosition();
x.Should().Be(1);
y.Should().Be(2);

Also, since the only purpose of ContextAttributes is for code generators, I added a [Conditional] attribute, so all those attributes are stripped from components once compiled.

Fyi, for those who are interested about the changes, see branch: roslyn-source-generators
https://github.com/sschmid/Entitas/tree/roslyn-source-generators

More updates:
I'm currently testing alternatives to the currently generated ComponentLookup. The current approach doesn't allow Unity's asmdefs or multiple projects.
The new approach should work with multiple assemblies per solution. At some point however, you would need to assign an index to each component. With the following idea you can build up your game with multiple separate assemblies and the main project that consumes them can implement a partial method per context and add the ContextInitializationAttribute to it:
public static partial class ContextInitialization
{
    [MyApp.Main.ContextInitialization]
    public static partial void Initialize();
}

This will be picked up by the generator and it will generate everything that used to be in ComponentLookup, e.g.:
namespace Entitas.Generators.IntegrationTests
{
public static partial class ContextInitialization
{
    public static partial void Initialize()
    {
        MyFeatureMovableComponentIndex.Index = new ComponentIndex(0);
        MyFeaturePositionComponentIndex.Index = new ComponentIndex(1);

        MyApp.MainContext.ComponentNames = new string[]
        {
            "MyFeature.Movable",
            "MyFeature.Position"
        };

        MyApp.MainContext.ComponentTypes = new System.Type[]
        {
            typeof(MyFeature.MovableComponent),
            typeof(MyFeature.PositionComponent)
        };
    }
}
}

More updates:
I improved the overall caching performance by using multiple pipelines. This means, only affected files should be regenerated leaving most of the files untouched.
I can recommend this cookbook for incremental generators:
https://github.com/dotnet/roslyn/blob/main/docs/features/incremental-generators.md
Next problem: if something in the generator fails, nothing will be generated. This can easily be reproduced by declaring the same component twice which will break everything. I tried to wrap all spc.AddSource() calls in a try catch block, because they fail when filenames are not unique, but this didn't help either.
Does anyone know how to make handle exceptions in a source generator?

@sschmid you can use tests, and check snapshots? https://andrewlock.net/creating-a-source-generator-part-2-testing-an-incremental-generator-with-snapshot-testing/
Otherwise - you can even use simpler diagnostic as a log inside the compilation
See https://github.com/dotnet/roslyn-sdk/blob/main/src/Microsoft.CodeAnalysis.Testing/README.md
await new CSharpAnalyzerTest<SomeAnalyzerType, XUnitVerifier>
{
TestState =
{
Sources = { @"original source code" },
ExpectedDiagnostics = { Verify.Diagnostic().WithLocation(1, 7) },
AdditionalFiles =
{
("File1.ext", "content1"),
("File2.ext", "content2"),
},
},
}.RunAsync();

As @studentutu hinted it's probably a good idea to emit diagnostics for these problems if you run into generation failures.
On top of that, you can also be proactive and write additional implementations of DiagnosticAnalyzer for common problem scenarios. Those can also offer code fixes if the class of error has an easy way to fix it. That's the lightbulb fixes that are popping up in Visual Studio / Rider.
In my ECS prototype I used those to validate Queries (so impossible queries produce errors).

have you thought about using enums perhaps? you can just collect all component names together into one file and make an enum out of it like this
public enum AppContextComponents
{
    Test1Component,
    Test2Component,
    Test3Component,
}

I feel like doing both the
MyFeatureMovableComponentIndex.Index = new ComponentIndex(0);
and
        MyApp.MainContext.ComponentNames = new string[]
feels like it's basically the same thing as an enum. (ints in the lower level code that also have a string version if you need it). Plus you can just declare the enum using the component names without worrying about the actual numbers etc. the only thing that matters is that they're unique, right? enums should guarantee that. Maybe I'm understanding something wrong though.

Is there a use case for interfaces that derive from Entitas.IComponent?
I.e.:
ICustomInterface : Entitas.IComponent{}

PositionComponent : ICustomInterface{}

If not I would recommend to make Entitas.IComponent a sealed interface. This way we can filter out more syntax trees in the predicate, thereby improving performance. The incremental generators re-run the transform every time, even if the predicate didn’t result in any new or changed syntax trees.(documentation). Which means that on every recompile the transform will create & cache empty immutable arrays for every single class declaration in the solution. If we filter out only relevant syntaxes using this:
&& candidate.BaseList.Types.Any(x => x.Type switch
                                                      {
                                                          IdentifierNameSyntax identifierNameSyntax => identifierNameSyntax.Identifier is { Text: "IComponent" },
                                                          QualifiedNameSyntax qualifiedNameSyntax =>
                                                              qualifiedNameSyntax.Left is IdentifierNameSyntax { Identifier.Text: "Entitas" } &&
                                                              qualifiedNameSyntax.Right is IdentifierNameSyntax { Identifier.Text: "IComponent" },
                                                          _ => false
                                                      })

we would not create as many duplicate immutablearrays

I also noticed an inefficiency in the newly made improvements you posted. I don't think it makes sense for the first transform to return an array of arrays. If we do this we will have multiple duplicate copies of ComponentDeclaration with the same data in all fields except one (the context). If you have 20 contexts attributes on one component you will get 20 strings for Namespace, 20 strings for FullName, etc etc etc. soooo much duplicate data. Why not make one struct and have it hold an array of contexts. this way you only have 1 string for each + 20 entries in the array inside of the ComponentDeclaration. You can do the rest with custom equality comparers etc. and the pipeline shouldn't be affected too much

@sschmid did you check out the PR I did? I added Diagnostics to some methods using something similar to a Result pattern. capturing diagnostics properly can sometimes be a bit weird and you kind of need to split off the pipeline to it's own source output. I checked out the way the MVVM source generator is doing it after I saw this comment, their setup is nice and they're doing lots of things right imo. here's a link to the comment

Hi @Ants-Aare,
about the enums:
I plan to remove ComponentNames like MyApp.MainContext.ComponentNames and update the ContextInfo to only contain the component types.
Also I'm affraid this won't work when you have multiple assemblies:
Generated code like AddPosition() depends on an index, which will potentially be assigned in another assembly. Therefore you cannot use an enum which contains all component indexes. Instead I generate a readonly struct ComponentIndex for each component, which can be assigned later in the main assembly.
Thanks for the perf improvments tips, that's always great to talk about! Your suggestion with the sealed component interface makes sense! I will try it soon! Thanks
I switched to one ComponentDeclaration per context with the hope to improve caching, as only the added or removed ones are affected. One component might result in mutliple generated files, like extension methods for entity, context, matcher etc, so I tried to optimize for that and reduce the need to regen unaffected ones. We can discuss this if you think I'm doing it wrong, because I might actually :D
Yes, thanks again for the PR. For now I was foccussing on getting a working protoype that works in Unity, Rider and VSCode + multiple asmdefs. During this research the ComponentGenerator changed heavily basically every day, and I wasn't yet sure if I can make it work, so no need for diags :D But now that that works, I can now also work diags and will have a look at your PR again

ah, okay! I understand where you're going with this 🤔 I agree that enums would be a bit too simplistic.
I'll try and update the PR so there's not that many conflicts with the current state of the branch. Can you add me to the Entitas Repo perchance? would be a bit easier to work on stuff

@Ants-Aare fyi, your suggestion with IComponent is is:
https://github.com/sschmid/Entitas/commit/951893400108352a1499d2e41c21187b72a05d2c
btw, turns out there is no such thing as a sealed interface :D

I have an issue, does someone know how to handle this?
Description:
I added a new project Entitas.Generators.Attributes which contains 1 attribute:

ContextAttribute

I wanted to try to use that one on one of the test components:

SomeNamespacedComponent

using Entitas;
using Entitas.Generators.Attributes;
using MyApp;

namespace MyFeature
{
    [MyApp.Main.Context]
    [Context(typeof(MainContext))]
    public sealed class SomeNamespacedComponent : IComponent { }
}

Problem:
The ComponentGenerator does not correctly identify this attribute. Inseatd of Entitas.Generators.Attributes.ContextAttribute it resolves it to UnderlyingSymbol: {ErrorType Entitas.Context}.
I tried to add the Entitas.Generators.Attributes project to the tests run by adding it to the references:
IEnumerable<PortableExecutableReference> references = new[]
{
    MetadataReference.CreateFromFile(typeof(object).Assembly.Location),
    MetadataReference.CreateFromFile(typeof(IComponent).Assembly.Location),
    MetadataReference.CreateFromFile(typeof(Attributes.ContextAttribute).Assembly.Location), // <------- added
    MetadataReference.CreateFromFile(typeof(MyApp.Library.ContextAttribute).Assembly.Location)
};

Reproduce

do a breakpoint in Entitas.Generators.ComponentGenerator:75
debug test NamespacedComponent in Entitas.Generators.Tests.ComponentGeneratorTests
ignore the diff windows if they popup
checking symbol.GetAttributes().Last().AttributeClass reveales UnderlyingSymbol: {ErrorType Entitas.Context}

When changing the usage of the attribute in SomeNamespacedComponent from [Context(typeof(MainContext))] to [ContextAttribute(typeof(MainContext))], it's still an ErrorType, but at least the fullName is now correct: Entitas.Generators.Attributes.ContextAttribute
No idea how to solves this atm. I think the dependency to Entitas.Generators.Attributes should be correct by using MetadataReference.CreateFromFile() (maybe this is wrong?). At least that's how I do it for the other dependencies

To me it sounds like there's both a issue with the compilation and an issue with correctly resolving the typesymbols.
It seems to be assuming that if you write [Context] you want to resolve it into the Entitas namespace instead of in the Entitas.Generators.Attributes namespace. If you write [ContextAttribute] then it resolves that correctly into the Entitas.Generators.Attributes namespace. Maybe there's a class inside Entitas that is also called Context or something like that, or it's just using the first namespace it finds. Maybe I'm understanding it wrong though.
I'm not 100% sure what the cause for the faulty compilation could be, in my mind it should be able to find the Entitas.Generators.Attributes.ContextAttribute correctly. What happens when you try getting syntaxContext.SemanticModel.Compilation.GetTypeByMetadataName("Entitas.Generators.Attributes.ContextAttribute")? is that one an ErrorType too?
In any case, do we really want it to only work when it can resolve the type? We're not using the attribute type for anything as far as I understand. Maybe checking if the Identity is equal to the context attribute is enough. We should follow the programmer's intent, and if we can verify that intent without depending on other parts of the system compiling correctly then I say we should go for it.
PS: does this issue occur only in the test case or when built too? I haven't checked that yet, but if it works then it's probably just an issue with creating a correct compilation in tests.

Yes, it finds it correctly with syntaxContext.SemanticModel.Compilation.GetTypeByMetadataName("Entitas.Generators.Attributes.ContextAttribute")
I'll investigate more...

Looking at my tests I think I ran into similar problems... I had this in there:
internal struct ForceAssemblyLoadComponent : IComponent
{
}

public static class GeneratorTestHelper
{
    public static string GetGeneratedOutput<TGenerator>(string source, ITestOutputHelper helperOutput)
        where TGenerator : IIncrementalGenerator, new()
    {
        var trick17 = (IComponent) new ForceAssemblyLoadComponent();
        var syntaxTree = CSharpSyntaxTree.ParseText(source);

        var references = new List<MetadataReference>();
        Assembly[] assemblies = AppDomain.CurrentDomain.GetAssemblies();
        foreach (var assembly in assemblies)
        {
            if (!assembly.IsDynamic && !string.IsNullOrWhiteSpace(assembly.Location))
            {
                references.Add(MetadataReference.CreateFromFile(assembly.Location));
            }
        }

        var compilation = CSharpCompilation.Create(
            "foo",
            new SyntaxTree[] { syntaxTree },
            references,
            new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary));

        // TODO: Uncomment this line if you want to fail tests when the injected program isn't valid _before_ running generators
        // var compileDiagnostics = compilation.GetDiagnostics();
        // Assert.False(compileDiagnostics.Any(d => d.Severity == DiagnosticSeverity.Error), "Failed: " + compileDiagnostics.FirstOrDefault()?.GetMessage());

        IIncrementalGenerator generator = new TGenerator();

        var driver = CSharpGeneratorDriver.Create(generator);
        driver.RunGeneratorsAndUpdateCompilation(compilation, out var outputCompilation, out var generateDiagnostics);
        Assert.False(generateDiagnostics.Any(d => d.Severity == DiagnosticSeverity.Error), "Failed: " + generateDiagnostics.FirstOrDefault()?.GetMessage());

        string output = outputCompilation.SyntaxTrees.Last().ToString();

        helperOutput.WriteLine(output);

        return output;
    }
}

@rubenwe 😍😍😍
I just added this and concatenated it with my existing references, and now it works 😍😍😍
Assembly[] assemblies = AppDomain.CurrentDomain.GetAssemblies();
foreach (var assembly in assemblies)
{
    if (!assembly.IsDynamic && !string.IsNullOrWhiteSpace(assembly.Location))
    {
        references.Add(MetadataReference.CreateFromFile(assembly.Location));
    }
}

Now I'll try to figure out why and learn from it :D
Thanks for the fix!

...of course, cannot use Unsafe in Unity 😭
error CS0122: 'Unsafe' is inaccessible due to its protection level

I already tries to allow unsafe code in the player settings, but no success yet... let's see if there's a solution.
Does anyone happen to know?

There's also an option to allow unsafe code per asmdef, unfortunately this also doesn't help. However, Unity seems to offer their own impl of Unsafe.As()
var indexes = global::Unity.Collections.LowLevel.Unsafe.UnsafeUtility.As<ComponentIndex[], int[]>(ref indices);

I'll probably add sth like this:
    public static global::Entitas.IAllOfMatcher<Entity> AllOf(params ComponentIndex[] indices)
    {
#if NON_UNITY
        var indexes = global::System.Runtime.CompilerServices.Unsafe.As<ComponentIndex[], int[]>(ref indices);
#else
        var indexes = global::Unity.Collections.LowLevel.Unsafe.UnsafeUtility.As<ComponentIndex[], int[]>(ref indices);
#endif
        return global::Entitas.Matcher<Entity>.AllOf(indexes);
    }

Optimizing usablilty for Unity, as I expect most people will use Entitas with Unity. Non Unity C# project must add NON_UNITY compiler flag

I think it would make sense to test this on a couple of devices before pushing these changes out. How's the performance on this? is it actually casting it or is it a trick so the compiler won't complain? On another note... What do you think about using a bit array? If we know the number of components we can make a bit array. if there's 20 components in a context it will always pass around a bit array of length 20, etc. and then do a bitwise comparison for each component. I'm not 100% sure about the implementation of this since I haven't thought it through but my thinking was that when requesting a matcher it creates the full bit array and on each additional matcher call it will just modify the bit array in place.
would look kinda like this then:
boolean value that would match on FooComponent=> myContextBitArrayMatcher[0]
boolean value that would match on BarComponent=> myContextBitArrayMatcher[1]
boolean value that would match on FooBarComponent=> myContextBitArrayMatcher[2]
since it would be super fast to iterate over, take up super little memory and if we can figure out a way to modify it in place while still keeping usability good it'd be even faster and better.
On usability it would look something like this (idk if it's gonna work)
MyContext.CreateMatcher() //Allocates the bit array once
    .WithFooComponent() // a bitarray extension method type of deal that's specific to the namespace of the MyContext (?)
    .WithBarComponent() // Would set the bit at index 1 to true

idk maybe i'm missing something here and it wouldn't work as well as it's in my head.

@Ants-Aare I'm totally down to discuss these kinds of improvements. I was also recently fantasizing about migrating Entitas to struct arrays at some point etc. I also met again with Maxim and talked about more efficient ways to store components.
But to stay focussed I will need to stick to my current roadmap which looks like this:

new Entitas release with new code generator that replaces Jenny and supports namespaces + Unity asmdefs, with a focus not to break tooooo much. New generated code will already break a lot of code.
follow up with a spring cleaning release: go over the complete code base and do "post dotnet 6" spring cleaning. Delete Entitas.Blueprints, Entitas.Migration and other old sub project, merge all Unity related sub projects into Entitas.Unity (rip Entitas.VisualDebugging). Also try to remove all dependencies to e.g. DesperateDevs, Sherlog, TCPeazy. Update code to official C# coding conventions. This release will all about modernizing to have a modern and fun to work code base which is ready for dotnet 8.0 and whatever comes next
Create completely automated build pipeline that builds commits and release Entitas to nuget and openupm, similar to what I already have done with DesperateDevs, Jenny, Sherlog, TCPeasy, etc
everything else, starting with addressing open issue, brainstoriming about things like what you suggested, etc

I'd suggest to use the UNITY_ENGINE compiler flag instead of needing a new one NON_UNITY

@PereViader I was hoping that sth like this exists, but unfortunately I don't think it does. I checked the docs https://docs.unity3d.com/Manual/PlatformDependentCompilation.html and I also had a look at the generated Unity csproj, they don't contain UNITY_ENGINE
But Unity includes UNITY_X_Y_OR_NEWER , e.g. UNITY_2020_1_OR_NEWER, so I will try using that instead of `NON_UNITY . I hope they will never remove those in the future :D

@Ants-Aare About you Unsafe.As() question:

How's the performance on this?

Here are the benchmark results from https://github.com/sschmid/Entitas/blob/roslyn-source-generators/bench/Entitas.Benchmarks/ComponentIndexBenchmarks.cs
|                          Method |     Mean |     Error |    StdDev | Rank |   Gen0 | Allocated |
|-------------------------------- |---------:|----------:|----------:|-----:|-------:|----------:|
|  ComponentIndexUnsafeToIntArray | 3.305 ns | 0.0161 ns | 0.0151 ns |    1 | 0.0057 |      48 B |
|    SpanComponentIndexToIntArray | 6.592 ns | 0.0436 ns | 0.0408 ns |    2 | 0.0057 |      48 B |
| RefSpanComponentIndexToIntArray | 7.753 ns | 0.0161 ns | 0.0134 ns |    3 | 0.0057 |      48 B |
|        ComponentIndexToIntArray | 8.640 ns | 0.0518 ns | 0.0484 ns |    4 | 0.0115 |      96 B |

Also interesting that ref Span was always slower than normal Span in my runs...

why does the compilationprovider not work?
in my experience you should be able to just do something like this:
var components = context.CompilationProvider.Select((compilation, token) => compilation
            .GetSymbolsWithName(x=> x.EndsWith("Component"))
            .ToImmutableArray());

this will give you a value provider with all symbols that end with Component. Obviously if we want to enable components that can be called anything you can do something else.
I'd also recommend to filter it etc:
var components = context.CompilationProvider.Select((compilation, token) =>
    compilation.GetSymbolsWithName(x=> x.EndsWith("Component"))
            .Where(x => x is NonErrorNamedTypeSymbol {Kind: NamedTypeSymbol, etc etc} namedTypeSymbo
                && namedTypeSymbol.Attributes.Any(etc etc))
            .ToImmutableArray());

You should probably also filter out all those that are already generated through other means or something like that so you don't accidentally have duplicate pipelines, etc.

@Ants-Aare

why does the compilationprovider not work?

:D exactly, that's the question.
For example, when debugging the test with this simple fixture which contains MyFeature.SomeNamespacedComponent and MyApp.ContextInitialization using this simple general purpose CompilationProvider
var components = initContext.CompilationProvider
    .Select((compilation, _) => compilation
        .GetSymbolsWithName(_ => true, SymbolFilter.Type)
        .ToImmutableArray());

I get the expected symbols:

MyFeature.SomeNamespacedComponent
MyApp.ContextInitialization

but only those, not the HealthComponent which is also part of the compilation from another assembly

Probably worth repeating that I can successfully get the HealthComponent manually in this slightly modified CompilationProvider with a breakpoint
var components = initContext.CompilationProvider
    .Select((compilation, _) =>
    {
        var array = compilation
            .GetSymbolsWithName(_ => true, SymbolFilter.Type)
            .ToImmutableArray();

        return array;
    });

// this works
compilation.GetTypeByMetadataName("MyOtherFeature.HealthComponent");

I found a solution, but I hope that's not the recommended way, because it feels like a workaround.
Explanation:
I added a CompilationProvider. Afaik, CompilationProviders are triggered every time you type no matter which class you're in. So in general I think you need to be careful with these in order to not slow down the users machine with unnecessary computations after each keystroke.
Please see: ComponentGenerator.cs
and GetOrderedComponentsFromAllAssemblies
In this method I more or less recursively check all namespace from ALL assemblies, get their members and add them to a list if they are components, potentially iterating over many many thousands of types. Then I order them and the FullNameAndContextCompilationComparer checks equality using SequenceEqual, checking all components and if the order changed (used for order index assignment for per context).
If nothing changes, no code will be generated. The caching works.
However, in order to check if nothing changes I have to always check all components from all assemblies. Feels heavy... This is one of the reasons I liked Jenny, because the dev is in charge to generate the code.
Does anyone have a smarter way of checking components and their order, maybe not every keystroke? That feels too much work for something that changes rarely (compared to the numer of keystrokes)

Ah, maybe I can do this step only on build! And not when files change. Because in this case nothing of that specific part of the generated code is necessary to develop, only for runtime. Remember, the generated code only basically initializes the context for you, e.g.
using global::MyApp.Main;

namespace MyApp
{
    public static partial class ContextInitialization
    {
        public static partial void InitializeMain()
        {
            global::MyFeature.MyAppMainLoadingComponentIndex.Index = new ComponentIndex(0);
            global::MyFeature.MyAppMainMovableComponentIndex.Index = new ComponentIndex(1);
            global::MyFeature.MyAppMainPositionComponentIndex.Index = new ComponentIndex(2);
            global::MyFeature.MyAppMainUserComponentIndex.Index = new ComponentIndex(3);

            MyApp.MainContext.ComponentTypes = new global::System.Type[]
            {
                typeof(global::MyFeature.LoadingComponent),
                typeof(global::MyFeature.MovableComponent),
                typeof(global::MyFeature.PositionComponent),
                typeof(global::MyFeature.UserComponent)
            };
        }
    }
}

I will see if I can make it on build only

doing it at build time means that all references and methods etc to these generated files will create errors in the console until it's built... and it would mean that editor play mode would be different than on device code. maybe it would make more sense to do it with initContext.RegisterImplementationSourceOutput or something similar

@Ants-Aare

generated files will create errors in the console until it's built

not quite. As soon as you create the InitializeMain all necessary things will be generated. Even if not, all code + other generated code will work, because this just generates a method body, so nothing will break if it exists or not.
I can also do the heavy task of getting all components every time I work on that method. But I would like not to do it when I work on other code.
I'll have a look what initContext.RegisterImplementationSourceOutput is, thanks!

RegisterImplementationSourceOutput works in the same way as RegisterSourceOutput but declares that the source produced has no semantic impact on user code from the point of view of code analysis. This allows a host such as the IDE, to chose not to run these outputs as a performance optimization. A host that produces executable code will always run these outputs.

😍😍😍 I will try!!!

Btw, I'd like to take a moment an thank everyone for your suggestions and help! It's a lot of fun to work on Entitas like that again! 🫶 Together we can make it 💪

I never used source generator in big projects. Do they scale well with the codebase size?
I just did a simple test which confirms predicates and transforms are ran with every keystroke, no matter if the changes are done in a related or unrelated file, even if it's just whitespace such as spaces or newlines.
initContext.SyntaxProvider.CreateSyntaxProvider(
    Predicate,
    Transform
);

The only caching optimization seems to happen when the transform method produces the same result; then it skips the generation part.
That sounds kind of scary to me, as a simple class itself has multiple child nodes, each triggering the predicate. So 1 class with a few fields and methods might trigger the predicate 10x. Scale that up to a massive codebase and 🤯
Does anyone have experience with larger codebases and source generators?
Will Jenny be a better choice in those cases?

Given that dotnet itself is increasingly using source generators, I wouldn't worry too much. Having early exits and a meta model that can easily be created, cached and compared in early stages of the source generator pipeline is absolutely relevant. And so is respecting cancellation requests.
The discussion here is quite relevant I think.
I'm not sure if the generators are currently called in a multi-threaded way, but given their Single-Pass nature and conceptual order independence I wouldn't be surprised.

@sschmid  using source generators on the large project doesn't impact is that much, whole compilation lasts 1-2 minutes, as we are using a separate dll and a separate project outside of Unity. In unity we then simply use that dll in conjunction with Unity view layer logic.

@rubenwe Yeah, I might worry too much :D
I remember that feeling made me stick to Jenny another year, where I can opt in and generate once sth relevant changed. I feel better now that I tested it in the IntegrationTests project: the part I was worried about takes 5ms... it's alright I guess :D
And yes, it's not blocking the IDE experience and runs in the background.
@studentutu yeah, separat projects will help to improve the generator, but I was mostly worried about how busy the machine is during coding, not just compiling. Every keystroke will invoke the generator pipeline, and that always felt like a waste of resources to me. But I guess I need to learn to live with it :D

Followup performance test result:
2000 components in 29ms. I'll take that :D

Bummer: I tried the current state in older Unity LTS versions, only 2022.3 worked 😭 I also downgraded to netstandard2.0 to support VS code. The irony that new stuff works in Rider but not VS 🤪
I can imagine it's gonna be a tough sell for many to upgrade their project to Unity 2022.3. I'll see if I can find a solution to support older versions too

I can share a little bit about the version drama based on my current testing:
1st version clash:

I typically use netstandard2.1 as a default, it also works with Unity (kind of)
testing the genrstors in Rider worked well, but VS code requires netstandard2.0 💥
so I downgraded back to netstandard2.0

2st version clash:

IIncrementalGenerator have been add in Microsoft.CodeAnalysis.CSharp 4.0 (I need those, must have)
Unity docs tell we must use Microsoft.CodeAnalysis 3.8 💥
that seems to be true until Unity 2022.3 LTS (I only test with LTS versions)

luckily starting from this version I was able to use Microsoft.CodeAnalysis.CSharp 4.0

however, latest is Microsoft.CodeAnalysis.CSharp 4.6, which doesn't work in Unity

I tested all versions in between and found Microsoft.CodeAnalysis.CSharp 4.1 is the latest you can use

So my current results:

use netstandard2.0, not latest
use Microsoft.CodeAnalysis.CSharp 4.1, not latest
only works in Unity 2022.3 LTS and later

🤷♂ but fair enough

Followup performance test result: 2000 components in 29ms. I'll take that :D

When you get closer to completion or after it's in I can also take a look if I can help and find a few extra places to tweak - but honestly 29ms is probably good enough ;D

Nullables would also allow me to remove throwing exceptions in the generated (e.g. when you try to get a component that has not been set) and move the responsibility to the consumer.

Please do that! This would allow us to use proper pattern expressions as filters in reactive systems and would simplify code in a lot of places!
So something like this:
    protected override bool Filter(GameEntity entity) =>
        entity.hasEquipmentChestType &&
        entity.equipmentChestType.Value is ChestType.Small or ChestType.Big &&
        entity.isInstantOpen;

Could at least become something like this:
protected override bool Filter(GameEntity entity) => entity is
{
    equipmentChestType: { Value: ChestType.Small or ChestType.Big },
    isInstantOpen: true
};

@rubenwe omg that last code snippet feels fresh, I like it a lot. This entitas version is breaking anyways, so it makes sense to include multiple changes and the cost of implementing it right now is fairly low.

More updates and infos on the route to source generators:
In a simple setup with source generators and Unity, the source generator will be active in each asmdef. This is convenient because you can split up your game in multiple sub projects with asmdefs and all works as expected.
Example:
Split up the app into Contexts, FeatureOne and FeatureTwo asmdefs. FeatureOne and FeatureTwo are completely decoupled from each other, but both depend on Contexts, because they need MainContext in the systems but also to add the context attribute to their components.
MyApp
└─ `Contexts` asmdef with `MainContext`
   ├─ `FeatureOne` asmdef (with some components and systems using `MainContext`)
   └─ `FeatureTwo` asmdef (with some components and systems using `MainContext`)

Contexts could contain context definitions like
namespace MyApp
{
    partial class MainContext : Entitas.IContext { }
}

The source generator will generate the rest of the classes, like the other part of the partial MainContext, but also the context specific classes like ComponentIndex, Entity, Matcher. All of those will be in the resulting Contexts.dll.
FeatureOne and FeatureTwo could contain component definitions like
using Entitas;
using Entitas.Generators.Attributes;
using MyApp;

namespace MyFeature
{
    [Context(typeof(MainContext))]
    public sealed class PositionComponent : IComponent
    {
        public int X;
        public int Y;
    }
}

The source generator will generate extensions like AddPosition(), ReplacePosition() and RemovePosition and all the other things we're familiar with, like Matchers etc.

However, there are also some files that have been generated by Jenny so far, that only exists once in the app, like the Feature class that is a sub class of Systems. This class has been generated, because it contains compiler flags like ENTITAS_DISABLE_VISUAL_DEBUGGING, UNITY_EDITOR, DEVELOPMENT_BUILD in various combinations. The Unity project needs this file as source code, so it can compile this class based on the current compiler flags and therefore could not be part of Entitas.dll.
As described above, the source generator is in every asmdef, so it cannot generate the Feature class anymore, because each sub project would generate this class. But it must be unique.
I actually moved the Feature class to Entitas now because I plan to distribute Entitas using openupm which allows me to distibute packages with the Entitas source code instead of dlls. If you prefer to consume Entitas as a dll, you would need to add the Feature class manually to you game.

On another note, the Contexts class that you probably know from Contexts.sharedInstance is probably going away. It was always meant as a convenience class for easy access, but every codebase is different, and some people might prefer other solutions like dependency injection. Since you can define your contexts in code now instead of the Jenny.properties file, you can decide how you want you access the context. If you like a static instance you can simply implement a context like this
namespace MyApp
{
    partial class MainContext : Entitas.IContext
    {
        public static MainContext Instance
        {
            get => _instance ??= new MainContext();
            set => _instance = value;
        }

        static MainContext _instance;
    }
}

and later use it like this
var entity = MainContext.Instance.CreateEntity();

You're in full control of your contexts now. Of course, you can also just recreate the Contexts class yourself, if you prefer that.
The Contexts class also used compiler flags to add ContextObserver when running in Unity, so you can see contexts in the hierarchy. Since the Contexts class in gine now, I added a new extension method for contexts that you can call yourself:
context.CreateContextObserver();

#if ENTITAS_DISABLE_VISUAL_DEBUGGING || !UNITY_EDITOR
[System.Diagnostics.Conditional("false")]
#endif
public static void CreateContextObserver(this IContext context)
{
    Object.DontDestroyOnLoad(new ContextObserver(context).gameObject);
}

This call will be stipped out when compiling using the ConditionalAttribute. Only when UNITY_EDITOR or !ENTITAS_DISABLE_VISUAL_DEBUGGING the code remains and a ContextObserver will be created.

And finally, to follow up on previous thoughts on nullables and other changes, I will try to keep the new generated code as similar to the current version as possible to not break too much (see https://github.com/sschmid/Entitas/issues/1069#issuecomment-1628777173).

I just finished event component and system generation, and finally fixed a subtle bug, where listener entities might leak.
Example of the old code:
https://github.com/sschmid/Match-One/blob/7d490c6d236c478f62b83a579724d730d2c9b88f/Assets/Generated/Game/Components/GamePositionListenerComponent.cs#L76-L84
Example of the new code:
https://github.com/sschmid/Entitas/blob/roslyn-source-generators/tests/Entitas.Generators.Tests/snapshots/ComponentGeneratorTests.EventNamespacedComponent%23MyFeature.MyAppMainAnyEventNamespacedAddedListenerComponent.g.verified.cs#L36-L51
Basically, when unsubscribing from event the listener can be auto cleanup up when empty. But it only removed the component, leaving the entity without any components. Now the listener entity will be destroyed when there are no components left on the entity

Hello!
Does anyone know how to set and use global options for generators (or analyzers)?
I described my previous solution here: #1069
but I have some open question:
What I'm currently doing:

add some key-value pairs to .editorconfig see code

please irgnore the fact that they are commented in this commit

create an OptionsProvider: see code
use it to check if a generator should run or be skipped: see code

this uses a helper class EntitasAnalyzerConfigOptions which is calling this: see code

optionsProvider.GetOptions(syntaxTree).TryGetValue(key, out var value)

Question
Why do I need to pass in a syntaxTree?
I initially tried to use the GlobalOptions, but this didn't work and the gobal options did not contain the keys from .editorconfig
optionsProvider.GlobalOptions.TryGetValue(key, out var value)

It sounds like GlobalOptions would be the correct place to check? How can I add custom key-value pairs to GlobalOptions? Seems like .editorconfig is not the correct place?
Also, I don't always have a syntaxTree that I can pass in. I pass one in where ever I can, and it works, but e.g. when woking with the compilation like with initContext.CompilationProvider, I cannot pass a syntaxTree.

I'm referring to this that I found in the docs: https://github.com/dotnet/roslyn/blob/main/docs/features/source-generators.cookbook.md#access-analyzer-config-properties

A generator is free to use a global option to customize its output. For example, consider a generator that can optionally emit logging. The author may choose to check the value of a global analyzer config value in order to control whether or not to emit the logging code. A user can then choose to enable the setting per project via an .editorconfig file:

mygenerator_emit_logging = true

[Generator]
public class MyGenerator : ISourceGenerator
{
    public void Execute(GeneratorExecutionContext context)
    {
        // control logging via analyzerconfig
        bool emitLogging = false;
        if (context.AnalyzerConfigOptions.GlobalOptions.TryGetValue("mygenerator_emit_logging", out var emitLoggingSwitch))
        {
            emitLogging = emitLoggingSwitch.Equals("true", StringComparison.OrdinalIgnoreCase);
        }

        // add the source with or without logging...
    }

    public void Initialize(GeneratorInitializationContext context)
    {
    }
}

Unfortunately, context.AnalyzerConfigOptions.GlobalOptions.TryGetValue doesn't work for me, only when passing a syntaxTree

One benefit you get with keeping the syntaxTree is that you could potentially configure the generator per component. Not sure if that's ever useful, but you can do it:
.editorconfig
[PositionComponent.cs]
entitas_generator.component.component_index = false
entitas_generator.component.context_extension = false
entitas_generator.component.context_initialization_method = false
entitas_generator.component.entity_extension = false
entitas_generator.component.entity_index_extension = false
entitas_generator.component.events = false
entitas_generator.component.event_systems_extension = false
entitas_generator.component.matcher = false

entitas_generator.context.component_index = false
entitas_generator.context.context = false
entitas_generator.context.entity = false
entitas_generator.context.matcher = false

Yay, all generators are complete. I updated my local MatchOne repo to give it a test run. Works great! I will share more soon!

Completed all task from above. Will close.
Development continues on branch entitas-2.0-beta, which includes the new code generator + more

Update:
entitas-2.0-beta is now merged into main and I updated Match-One, so you can test and play around with Entitas 2.0-beta and the new dotnet source generators:
https://github.com/sschmid/Match-One/tree/entitas-2.0-beta
Development is not complete, things may change, but it's the first working version that I wanted to share to get feedback. Once Entitas 2.0 gets closer to release I will share more docs and upgrade guides. For now Match-One is the only "documentation" in form of a working project. Try it, break it and have fun :D


Defition of a type should not contain namespaces in API reference

The definition of a type in API reference is too verbose because it includes all namespaces.
For example, Dictionary:
[System.Runtime.InteropServices.ComVisible(false)]
public class Dictionary<TKey,TValue> :
System.Collections.Generic.ICollection<System.Collections.Generic.KeyValuePair<TKey,TValue>>,
System.Collections.Generic.IDictionary<TKey,TValue>,
System.Collections.Generic.IEnumerable<System.Collections.Generic.KeyValuePair<TKey,TValue>>,
System.Collections.Generic.IReadOnlyCollection<System.Collections.Generic.KeyValuePair<TKey,TValue>>,
System.Collections.Generic.IReadOnlyDictionary<TKey,TValue>, System.Collections.IDictionary,
System.Runtime.Serialization.IDeserializationCallback, System.Runtime.Serialization.ISerializable

This would have been much easier to understand if it was just:
[ComVisible(false)]
public class Dictionary<TKey,TValue> : ICollection<KeyValuePair<TKey,TValue>>,
IDictionary<TKey,TValue>, IEnumerable<KeyValuePair<TKey,TValue>>,
IReadOnlyCollection<KeyValuePair<TKey,TValue>>, IReadOnlyDictionary<TKey,TValue>,
IDictionary, IDeserializationCallback, ISerializable

We do have an internal bug tracking this already. /cc @dend

Issue moved to https://github.com/MicrosoftDocs/feedback/issues/225


Carica/PhpCss VS Symfony/CssSelector

I do appreciate for this excellent package. May you please provide some info regarding choosing between Carica/PhpCss and Symfony/CssSelector in the docs?

I don't know the current state of the Symfony package. Not sure if it makes any difference which package you use now. You can test the PHPCss converter at http://xpath.thomas.weinert.info.


add API calls to get category and videos

This PR will add code for the API calls to get all the categories and videos
Please review this PR, and give your feedback

https://dekho-video-72pqgi60o-hsnice16.vercel.app/

Appreciation: Your code is clean and readable. The folder structure is also awesome. Liked that you have used constants for dispatch values.
Just a suggestion on UI, on clicking somewhere else on the screen the settings component should go away.

Thanks for making the suggestion completely clear by attaching the SS. Yeah, I will do the changes.


My comments

It is hard to get good feedback, so here are my comments.
A description of why this design is better than other designs would be helpful.
The image is great, but it took a while to understand it.  Are those separate RAM blocks?  In that case, maybe the strength of this design is high bandwidth to the memory blocks.  A better introduction would have helped me understand the image the first time I looked at it.
Lots of great details.  It would be nice to have a simple higher level view of the instruction set.  What were you thinking when you designed this?
With all of the different memory access options, you may want to justify it using the data presented here and in the appendix.
https://users.ece.cmu.edu/~koopman/stack_computers/sec6_3.html
It is not clear which board/fpga this is running on.
It is not clear how your memory is structured.  One block ram, multiple ones???
After 3 reads, I am still not sure where the wright address comes from.
It would most helpful to see the FPGA resources required by this design.
Two memory accesses per clock cycle is interesting.
Do any real Forth applications really need a 15 or 16 bit program address space.  I thought that 14 bits would be plenty.  Maybe I am wrong.
What frequency does this run at?  Which path limits the frequency?
What happens during interrupts?
How much does this board cost?
When you define the bits such as

. These three bits define the RAM access operation.

Please specify what the options are.  This cpu does a great job of it.
https://github.com/CPUNexus/Tridora-CPU-mirror/blob/main/doc/tridoracpu.md
I now see that you did this quite well with the ucode table.  I am still not sure what a ucode is.
I hope that the feedback helps.

Hi, thank you for your questions about uf16. I know that the documentation is very poor, and I am committed to improve it in the future.
I try to answer to your questions:
In my implementation, I use a single 8kbyte RAM block and a single 8kbyte ROM block. Both are accessed through the same single bus (von newman architecture).
The FPGA board I used is the TinyFPGA-BX. Here there are 16 EBR blocks for the 8kbyte RAM and 16 EBR blocks for the 8kbyte ROM. You can find the FPGA resources used in the iCEcube2 log file.
These resources are used to implement the full SoC:

uf16
USB interface
USB to UART interface
Vectored interrupt controller
Timer
gpio

There is a single bus to access both RAM and ROM, so there is only a single access per clock cycle.
Cycle by cycle the address is generated to access the ROM for an instruction fetch, or to access the RAM for a read/write, or for a push/pop on data/return stack.
The maximum address space is 64kbyte. In the TinyFPGA-BX there are only a total of 16kbyte EBR available, so I divided them in two, 8kbyte for the ROM with the ansi 94 FORTH (almost complete, only the LEAVE word is missing) and 8kbyte for the RAM. The RAM is used for additional word definitions and for the stacks. The uf16 allows preemptive multitasking, so multiple stacks can be put in RAM, a data/return stack couple for each task.
When an interrupt is requested on irq_i input then the uf16 program counter jumps immediately to the address provided by the irqaddr_i input.
I hope that it is clearer now. Please let me know.

Thank you.
There is now a channel for discussing Forth Hardware on the Forth Discord server.  (Soft core forth machines).
https://discord.gg/fbBf46ay


Solved issue #395 Navbar 4

#395 Issue solved

Good job @AmitDeka
Congrats 🎉, your PR got merged. To be eligible for the swags from ClueLess. Follow this steps.

Star this repository. ⭐
SignIn / SignUp in [ClueLess Website](https://www.clueless.tech/) .
Go to profile section from the top right hand corner.

Comment your profile URL under this comment tagging the person who merged your PR.

That’s it. Follow the leaderboard !
And don’t forget to share your achievement with your friends.

https://www.clueless.tech/profile/40214392 @Rajdip019


Add logic to build cluster.Spec from cluster

Issue #, if available: part of https://github.com/aws/eks-anywhere/issues/1090 and https://github.com/aws/eks-anywhere/issues/1109
Description of changes:
Builds (fully) a cluster.Spec using a kubernetes client:

Builds the cluster.Config with the default builder
Retrieves Bundles and the eksd release from the cluster
Some data massaging to complete the Spec

Please, look at comments in PR for more localized explanations.
Testing (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/hold

/unhold


CiviCRM 5.x compatibility

I noticed that there have not been any commits to the project in a while.  Is the extension still actively developed?  Do you know if it's compatible with CiviCRM 5.x?

Hello @TechnologyClassroom and thanks for checking.
I would say this extension is not actively developed any longer. Active development was originally funded :). I have been spending my limited volunteer dev time on core issues.
I am, however, more than happy to respond to bug reports, feature requests, etc in the form of issues. If you can contribute funding or provide a PR for your requested changes, I will definitely review and help it get merged. If you cannot, I can't guarantee any sort of timeline that I can do the work but i'm happy to receive issues and be aware of them. It's possible that I will find time to take care of them. I'm not 100% sure what happened to the other PR from Monish to support more forms but that was definitely and interesting idea.
I just gave this a reaaally quick and dirty test in CiviCRM 5.18.alpha1 and the basic features all seem to be functioning. I had to clear caches before the settings page would load up.
I'll close this for now, I think it is answered. Please feel free to open other issues if you end up using it!


Autocomplete: hide suggestions on blur

Please make sure these boxes are checked before submitting your PR, thank you!

[ ] Make sure you follow Element's contributing guide (中文 | English).
[ ] Make sure you are merging your commits to dev branch.
[ ] Add some descriptions and refer relative issues for you PR.

#6249


Filtering duplicates

It is possible to add a second identical filter. Clicking on one opens the dropdown for both. I think there needs to be a check to not allow adding duplicate filters.

This only applies to some filters

Is it happening at the deployed server or localhost? Do you still remember the steps to reproduce the error?

On the prod deployment. For example: add postmortem interval, then go back to the dropdown list. Postmortem interval is still on the list and still be added.

And this is without hitting apply

OK. Found the issue you mentioned. Basically, it will not happen if user only use mouse click. Here are the steps to reproduce the issue:

Add any filters
Clear the input box
When the input box is empty, hit "backspace" again
Filters will reappear in the dropdown, and can be added again
I will work on that.

Issue addressed in recent PR #21


Reference to cp in documentation

This is possibly a little too picky, but the documenation blurb often makes reference to a cp function (example).
However, the S3FileSystem implements a get function in place of cp. Might it be clearer to replace references to cp with get - or even add a dummy cp that implements get?
If a correction is desirable I'll happily push a PR :-)

Forgive me for being daft, how do you propose we implement cp (or copy) with the current API? For clarity, I would expect behaviour like AWS CLI cp: https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

Copies a local file or S3 object to another location locally or in S3.

... without streaming data to a local machine and then back up to AWS

There are three methods, copy which duplicates a remote file (without streaming locally) and get, put which do remote-local-remote. We do not have a single method which attempts to determine whether you meant a local or remote file in each path given.
fsspec would give the opportunity to implement a general cp along the lines of AWS's, streaming locally when copying across file-systems, or using whatever method the backend has when copying on a single file-system. Not all FSs necessarily have a native copy function.


ARM Template not available

I've recieved an error after clicking on "Deploy to Azure" The error suggests that the ARM template isn't available.
"There was an error loading the template from URI 'https://raw.githubusercontent.com/dgkanatsios/AzureContainerInstancesManagement/master/deploy.json'. Ensure that the template is publicly accessible and does not have any comments."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 697e5b68-adbf-e20d-cb28-a2651891c82f
Version Independent ID: a5beac2b-eedc-e696-f38b-59e43f059504
Content: Multiplayer Game Server Hosting Using ACI - Azure Gaming
Content Source: azure-gaming-pr/reference-architectures/multiplayer-synchronous-aci.md
Product: azure-gaming
GitHub Login: @BrianPeek
Microsoft Alias: brpeek

@dgkanatsios The deploy.json is in your repo and I'm getting the same error.  But, the file exists at that link.  Any idea why it's failing?

weird. @ShinraCorpMixer can you try copying/pasting the template to an Azure Template Deployment? Just to make sure something hasn't changed in the ARM interpreter

Hello @dgkanatsios I've tried copying the deploy.json file, however it brought up an error during validation:
"{"telemetryId":"89cbfe79-ad63-44bf-9a10-8c3731c49bc5","bladeInstanceId":"Blade_127527ee6cf1496eae622faf3ea8394e_0_0","galleryItemId":"Microsoft.Template","createBlade":"DeployToAzure","code":"InvalidTemplateDeployment","message":"The template deployment 'Microsoft.Template' is not valid according to the validation procedure. The tracking id is '1aa823b2-ab5a-47a4-bc20-4d34021d1ec4'. See inner errors for details.","details":[{"code":"ValidationForResourceFailed","message":"Validation failed for a resource. Check 'Error.Details[0]' for more information.","details":[{"code":"ServerFarmNotFound","message":"The specified app service plan was not found."}]}]}"

Small update, I've forked the repo and modified the template URIs to point it to the forked one, seems to be working fine except for the error that I've posted earlier. I'm scratching my head as to why it says the the app service plan was not found. I'm still a novice when it comes to ARM templates so I'm not sure why it's not finding the App Service information even if there's a dependsOn for it at the bottom of the deploy.function.json file.
"details": [
{
"code": "ServerFarmNotFound",
"message": "The specified app service plan was not found."
}

Looks like this was fixed but the issue was never closed.  Thanks!

Looks like this was fixed but the issue was never closed.  Thanks!


[Feature Request]: "Add New Quote"

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
Adding new quote
Record

[X] I agree to follow this project's Code of Conduct
[X] I want to work on this issue
[X] I'm willing to provide further clarification or assistance if needed.

@JayShukla8 please assign me this issue under hacktoberfest.

@JayShukla8 Please add hacktoberfest tag before merging.


How to get the complete trace log

It seems like the current external trace interface doesn't provide the execution result of each instruction. Is there a way to get the full trace information including the architectural state changes?
https://github.com/westerndigitalcorporation/swerv_eh1/blob/master/design/swerv.sv#L34

This repository is deprecated. Please use the Chip Alliance repository to post questions/issues.

Hi, I have pushed (to the Chip Alliance repo) an update to the testbench that dumps a very simple/basic execution trace (time, pc, opcode, regnum, value) to exec.log. You can use this to extend to a more detailed trace for your TB.
Please continue to post queries on the Chip Alliance site.


How to configure Platypus-generated app to accept links dragged out of webpages?

I'm developing a Droplet on OS X 10.11.3 that accepts links (urls) dragged directly out of webpages. The underlying script works fine when run from Terminal, since Terminal accepts everything that's dropped onto it: I simply type myUrlWrangler.pl followed by a space in Terminal, and then drop the url into Terminal.
However as a Platypus app it doesn't accept urls that I drag from a webpage or from the address bar onto the droplet. I've tried quite a few Uniform Type Identifiers:
public.item
public.data
public.content
public.html
public.url
public.file-url
public.url-name
com.apple.webarchive

including no UTI at all, just the Suffix * and enabled Accept dropped text, to no avail.
The Platypus docs say,

You can drag files from the Finder into the suffix or UTI list to add their respective suffix/UTI.

However, when I drop a URL from a webpage onto the UTI list, nothing happens (which may be a hint why it doesn't work with the generated app either).
I found, by adding the UTI com.apple.web-internet-location that at least URLs dragged from the address bar are accepted.
There's an article where the author analysed Info.plt for the Dropbox app to reveal how it allows to receive everything, namely with the UTIs public.dataand public.content. Yet, even this combo does not enable my Platypus app to receive URLs.
How can I configure my Platypus-generated app to accept links dragged out of webpages?

Which browser are you dragging out of? They offer different representations of addresses. You can debug using ClipboardViewer.

You should be able to fix this by enabling the app as a Service under Drop Settings. Let me know if that works.


Client running on a Raspberry Pi Model 4. Support for how to do FOTA updates.

How would one go about doing a firmware over-the-air (FOTA) update if the wakaama client is running on a Pi model 4 and some Leshan demo instance? I've tried using Thingsboard and simply transfering a bootcode.bin file is not sufficient enough. I assume some sort of logic needs to happen on the client side for the firmware update to happen correctly. Is there support for how to do this?

LWM2M only helps you transfer the data from server to your Pi. The rest of it you need to implement them by yourself.


Insert into returning alternative

Alternative implementation of #8030

Note: I am unable to run unit tests for Postgre or Oracle.
Someone should confirm this code is actually working.

@nineinchnick, can you confirm the unit tests are passing for Postres and Oracle?

@klimov-paul  I only ran AR tests but they do pass. I can run the rest tomorrow.

Why returning an empty array in this case? Shouldn't it return the primary key value(s) as described in the comment?

Fixed.

:+1:


DeOSS looks like not support to upload huge file

When i use the function 'UploadFileChunks' in go-sdk try send file to Deoss,but always get 524 errorcode.Pls check the reason.

@democ98 The 524 is limited due to cloudflare, please do not over rely on the public gateway and use it for your generation environment, we cannot guarantee the stability of the public gateway and recommend you run your own gateway.


feat: add barretenberg wasm build

Please read contributing guidelines and remove this line.

#6174
#6173  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alexghr and the rest of your teammates on  Graphite

Folded into #6174


pose user guided caching

What happened (or feature request):
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else do we need to know:
Related source links or issues:

Welcome @ashooner - I imagine you're filling this in with an edit as we speak, but we're looking forward to your issue.


Slack integration for critical medium high low mute alerts

New feature motivation
Slack integration including the list of findings instead of just the aggregates.
Solution Proposed
When it scans EC2 suppose, it should send critical alerts which EBS volume is not encrypted and it should the points also
like this all the services.
Describe alternatives you've considered
N/A
Additional context
Hey there :wave:
I'm Prowler, the handy multi-cloud security tool :cloud::key:
I have just finished the security assessment on your AWS Account<PHONE_NUMBER>73 with a total of 262 findings.
- 206 Passed findings (78.63%)
- Failed findings (21.37%)
- 81 Scanned Resources

<Add list of findings>

Hey there :wave:
I'm Prowler, the handy multi-cloud security tool :cloud::key:
I have just finished the security assessment on your AWS Account<PHONE_NUMBER>73 with a total of 262 findings.
206 Passed findings (78.63%)
Failed findings (21.37%)
81 Scanned Resources
i need which failed findings list example

Hey @jfagoagas
While I'm waiting to get approval from @pedrooot to start working on the ingress/egress issue, can I work on this one?
Thanks,
Amogh

Sure! @abant07 go for it

Hey @jfagoagas @pedrooot
Here is my plan to go about solving this issue.
Firstly, I want @codepipe to know that I believe the reason why Prowler doesn't list out all the findings for the passed and failed checks is because there are some prowler users who will have thousands of findings, and if we were to display all those findings without some sort of sorting mechanism, it will get really messy and long, and hinder the user experience. (Prowler team correct me if I am wrong on this).
However, I do believe it's important to get some insight to what findings went wrong and which ones went right. So, what I will do is I will select 20 random findings that passed and 20 random findings that failed, and list them in the slack report.
As per how I will change the code. There are two files I will be altering, the slack.py file and the outputs.py file. In the slack.py file there is a method called the __create_message_blocks__() and this method is responsible for creating the UI of the slack report, and this method also takes in a parameter called stats, which is a dictionary that is generated in the ouputs.py file.
In the outputs.py file, there is a method called extract_findings_statistics which is generating the statistic reports for each finding and then displaying it in the slack report. Currently, it generates information like total failed, total passed, total muted failed, total muted passed, resource count.
So, what I will need to do is add another 2 key-value pair in the dictionary that the extract_findings_statistics method is returning called stats["findings_passed/failed"] = finding_names, where finding_names is a list of 20 random findings that were passed/failed. Then in the slack.py file, I will reference in these keys and display them as a list in the slack report.
However, I did have one question, do "findings" basically mean the check names (ie. ec2_client_vpn_endpoint_connection_logging_enabled). Because if so, in the outputs_test.py file, we are currently using MagicMock() to simulate these findings, but there are no "name" fields, just resource_id and status, or muted.
for example, here is what I mean
finding_1 = mock.MagicMock()
finding_1.status = "PASS"
finding_1.resource_id = "test_resource_1"
finding_2 = mock.MagicMock()
finding_2.status = "FAIL"
finding_2.resource_id = "test_resource_2"
findings = [finding_1, finding_2]

After thinking about it. What I can do is instead of doing 20 random passes and fails, I will only list checks that have the highest severity.


Docs for citation functionality

Provides documentation for the cite-brainglobe tool in https://github.com/brainglobe/brainglobe-utils/pull/28.
Also related to https://github.com/brainglobe/BrainGlobe/issues/61.

Can this be merged now?

I guess so


fix(sql): /ServiceAccounts fails when sql is only active provider (#839)

Cherry pick of 225688b
This was a bit of a weird error. After we turned off the S3 provider in Front50, the /serviceAccounts endpoint stopped working, failing with the following error message: "Configured storage service does not support service account permissions".
This happens when there's no ServiceAccountsService bean (which it really wasn't). This bean is autoconfigured using @ConditionalOnBean(ServiceAccountDAO.class), and according to Spring there's no ServiceAccountDAO in the context. But when inspecting the context, it actually is a ServiceAccountDAO present. I think that for some reason the ServiceAccountDAO bean is created after Spring tries to wire up ServiceAccountsService, but this doesn't happen with the other providers. I have no idea why. The only thing I can think of is that SqlConfiguration.kt is the first provider config written in Kotlin, but I don't really understand why that should matter. I'd appreciate any insight into the matter.
I have fixed the issue by creating a new @ConditionalOnAnyProviderExceptRedisIsEnabled annotation that returns true if any of the providers except Redis are enabled. I've tested running the app with only SQL enabled, only Redis enabled*, and both SQL and Redis enabled. All works now.
* This didn't work before, as @robzienert correctly theorised.

Add back @ConditionalOnBean and fix visibility of two functions

some minor Java style fixes

Added new conditional annotation and applied it in relevant places

The new @ConditionalOnAnyProviderExceptRedisIsEnabled will be true if any of the providers except Redis are enabled. Whether Redis is enabled or not does not matter.

Conditionally wire up ServiceAccountsController

Co-authored-by: Cameron Fieber<EMAIL_ADDRESS>

I think that instead of merging this, we should strongly consider rolling back #803.  That was labeled as a fix and cherry-picked into 1.19 despite being a pretty large change, and it looks like it is the cause of this regression here.
I'm surprised that #803 was cherry picked in the first place as it seemed like a reasonably large and risky change (which this bug supports), and am not overly excited about continuing to patch/fix bugs against it in the 1.19 branch instead of just rolling it back.

Rolling back #803 in the 1.20 branch sounds good to me. @adriengentil WDYT?

While I agree that #803 never should have been cherry-picked, it went out with the 1.20.1 release. So now it's been in four releases. Rolling it back to me feels potentially as worrisome as merging this... I think we're kinda stuck in a hard situation now and I don't know what the right move is...

I agree, #803 shouldn't have been merged. I was misguided by the fix label (shame on me and should have dug in more thoroughly.
Like @plumpy said, I think we're in in between two not so great solutions. I think the right thing to do is to merge this and "fix the fix" rather than rolling back #803, however.

Merging. We can’t keep the release broken.

I don't know why the branch protection is still looking for a Travis status, but I think mergify needs a little help here...

I believe it's waiting for approval by a release manager; since @ethanfrogers is release manager this cycle, deferring to him on approval/merge.

It looks like it's waiting for this:
status-success=continuous-integration/travis-ci/pr

based on this:
https://github.com/spinnaker/front50/pull/848/checks?check_run_id=752800211

I merged it with admin privileges. I am not/wasn't sure how to adjust Mergify's rules to exclude Travis from the requirements.

hey, I guess this would be part of 1.20.6. Any updates when 20.6 will be released?

@ethanfrogers Any possibility of a 1.20.6 soon™?

fyi for anyone looking at this from the future wonder if it got in. It did in 1.20.6! 🙌 🙌


Feature Request: Add ability to specify delimiter for slice flags

When specifying a slice flag:
Strs []string `short:"s" long:"strs" description:"..."

there doesn't appear to be a way to specify a delimiter to use to split the provided strings into the string slice. For instance, if the above flag is specified:
./program -s one,two -s three

Results in:
0: one,two
1: three

Instead of:
0: one
1: two
2: three

The typical way to do this is to specify -s multiple times. Adding the ability to split the arguments by a delimiter would be a big usability improvement, particularly when you need to specify many arguments into that slice.
There currently is the env-delim option which appears to have a similar functionality. I propose to add a delim option that will split the flag values by the given delimiter.

Also, I would personally add a size tag that would enforce slice (or maybe additionally, string size) arguments. Could be a range or a single integer.
e.g.
size:"3"
size:"1,-"
size:"-,4"
size"3,5"


Support for crossorigin attribute on injected link tag

Hello,
I need support for crossorigin="use-credentials" on injected <link> tag.
Correct me if I'm wrong, but such option is not available.
Thx

Hope it gets merged soon!
https://github.com/arthurbergmz/webpack-pwa-manifest/pull/62

This was fixed in #62

Works well, thank you!


Updates to have more Functions and for better usage with the new API

I believe the latest branch failed due to a connection error, which is no problem with my code.

I think changing the utils import breaks when the package is installed
Please write meaningful commit messages
Please keep number commits low (1-3), per pull request
Follow existing conventions when naming methods, e.g. prefer get_something over getSomething
Avoid breaking the API
Avoid duplicating methods (login vs logina)
Keep the number of changes per PR low

Sorry, not merging this.

I think changing the utils import breaks when the package is installed
Please write meaningful commit messages
Please keep number commits low (1-3), per pull request
Follow existing conventions when naming methods, e.g. prefer get_something over getSomething
Avoid breaking the API
Avoid duplicating methods (login vs logina)
Keep the number of changes per PR low
Sorry, not merging this.

Thanks for the information, I will attempt to clean it up. Logina logs the user in with an auth_token rather than a password fyi.


switch to memlab for memory leak e2e testing

Type of PR: enhancement
PR checklist:

[ ] Addresses an existing issue: fixes #
[x] Includes tests
[ ] Documentation update

Overview of change:
Switches the memory leak e2e testing to use memlab.

@kirchet, @eugene-korobko this is something we thought about recently when facing loads of issues using Puppeteer.
Looks like a better replacement for it.
What do you guys think?

@kirchet, @eugene-korobko this is something we thought about recently when facing loads of issues using Puppeteer. Looks like a better replacement for it. What do you guys think?

I do not mind for this replacing if it works better

@kirchet, @eugene-korobko this is something we thought about recently when facing loads of issues using Puppeteer. Looks like a better replacement for it. What do you guys think?

I do not mind for this replacing if it works better

Cool we'll merge that one then and observe over the next few PR how it behaves vs Puppeteer


speedup headless getTextContent

I am using PDF.JS as a server-side parser to get pdf text content only.
Everything works fine but processing is quite slow (10s for 20MB/120pages pdf file).
I wondering if it is possible to disable some unneeded facilities to speed up the process.
A the moment I defined my own PDFDataRangeTransport (with rangeChunkSize:4096) in order to load only the necessary amount of data (I also disableStream:true and disableAutoFetch:true)
Is it possible to go further ?

Everything works fine but processing is quite slow (10s for 20MB/120pages pdf file).

That would put the average text-extraction speed at ~83 ms/page, which really doesn't seem bad at all considering that 20 MB is a fairly large PDF file. (Just for comparison, the PDF specification weighs in at 8.58 MB for a 756 page document.)
Besides, what constitutes as slow is highly subjective, meaning that there's no proper criteria for when this issue would be considered "fixed".

I wondering if it is possible to disable some unneeded facilities to speed up the process.

Assuming that you're only calling getDocument, followed by getPage and finally getTextContent, then I'm not sure what that would be!?
The PDF format is quite complex and having to parse large files will simply take time, there's really no way around that.

Is it possible to go further?

Sorry, but this doesn't seem very actionable unfortunately; please also consider ISSUE_TEMPLATE.md and CONTRIBUTING.md when opening as issue.


Interface fix library

Interface fixes for OAE library. Currently under redesign to accommodate a mosaic view and a better display of tables.

Tasks left:

add iron icon for more options
populate interface with mosaics


Add GridModel.showGroupRowCounts config

Default true (maintains current behavior) but set to false to suppress group member row counts within full-width group rows.
Fixes #647 (more)

Hoist P/R Checklist
Pull request authors: Review and check off the below. Items that do not apply can also be
checked off to indicate they have been considered. If unclear if a step is relevant, please leave
unchecked and note in comments.

[x] Caught up with develop branch as of last change.
[x] Added CHANGELOG entry, or determined not required.
[x] Reviewed for breaking changes, added breaking-change label + CHANGELOG if so.
[x] Updated doc comments / prop-types, or determined not required.
[x] Reviewed and tested on Mobile, or determined not required.
[x] Created Toolbox branch / PR, or determined not required.

If your change is still a WIP, please use the "Create draft pull request" option in the split
button below to indicate it is not ready yet for a final review.

Pull request reviewers: when merging this P/R, please consider using a squash commit to
collapse multiple intermediate commits into a single commit representing the overall feature
change. This helps keep the commit log clean and easy to scan across releases. PRs containing a
single commit should be rebased when possible.

@haynesjm42 and @lbwexler - would we be able to merge this in? Any concerns? It's not essential but would be happy to have it unless there are blockers of any kind. Thanks much.


Fail to build dp-xgboost

        copying dp_xgboost/LICENSE -> build/temp.linux-x86_64-cpython-39/dp_xgboost
        INFO:DP-XGBoost build_ext:Building from source. /tmp/lib/libxgboost.so
        INFO:DP-XGBoost build_ext:Run CMake command: ['cmake', 'dp_xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=0', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_LZ4=0', '-DPLUGIN_DENSE_PARSER=0']
        error: [Errno 2] No such file or directory: 'cmake'
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for dp-xgboost
  Failed to build dp-xgboost
  ERROR: Could not build wheels for dp-xgboost, which is required to install pyproject.toml-based projects

  at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/env.py:1476 in _run
      1472│                 output = subprocess.check_output(
      1473│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1474│                 )
      1475│         except CalledProcessError as e:
    → 1476│             raise EnvCommandError(e, input=input_)
      1477│
      1478│         return decode(output)
      1479│
      1480│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:

  PoetryException

  Failed to install /home/gerimedica/.cache/pypoetry/artifacts/21/3b/77/255d6ff4df584892ff965f6b01c4e187c0da7d2c53796debe8f3e11532/dp-xgboost-0.2.8.tar.gz

  at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/pip.py:51 in pip_install
       47│
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│

error was resolved by installing cmake.
However, this has never happened with other packages, maybe the author of this package could upload a wheel such that user do not have to make it themselves?


Add Shifter support to NJS

This adds the ability to use Shifter (github.com/nersc/shifter) to launch containers.  This is needed to run at NERSC.
To trigger this, USE_SHIFTER must be set.  There are additional variables that can be set to specify the callback interface and ref data location.  This also add a mydocker wrapper script.  This is need to work around some constraints imposed by Shifter.  While this could have been done in the java side, the wrapper script simplifies things a great deal.
Please review and provide feedback.

So the PR doesn't seem to break anything, which is great. So I can merge it in on that basis if you'd like. Otherwise, we can hold off on it until we can actually test these features. Let me know what you think @scanon


Add public wiki to Cosmos Ansible

A lot of this info is in here https://github.com/hyphacoop/cosmos-configurations-private/wiki#worklogs
need to clean up and remove private info

Please review:
https://github.com/hyphacoop/cosmos-configurations-private/wiki/Multi-Node-Testnet-Setup
https://github.com/hyphacoop/cosmos-ansible/blob/three-node-example/examples/README.md

I'd add a sentence here like this:

We will use a scenario in which several hosts need to be set up on the cosmostest.network domain. This is an example domain, you should replace this with your own.

In here, https://github.com/hyphacoop/cosmos-ansible/blob/three-node-example/examples/README.md#start-a-three-node-testnet shouldn't we be using the public facing mnemonics?
Could we also make folders for these? Perhaps like this:

validator-keys/<validator-name>/node_key.json
validator-keys/<validator-name>/priv_validator_key.json
validator-keys/<validator-name>/self-delegation-wallet-mnemonic.txt

Make the guides for multi-node testnet setup and monitoring part of the repo in a docs folder.


What does the files attribute do in a JavaScript sandbox?

I have something like:
<script id="inline" type="text/plain">
    console.log("running inline script");
</script>

<codapi-snippet engine="browser" editor="basic" sandbox="javascript" output-mode="dom" files="#inline" >
</codapi-snippet>

is files="#inline" supposed to run the script inside that tag? Thta console is not executed
Additional question: I presume this does document.querySelector("#inline")
can this be a Web Component shadowDOM reference?

files are only supported by the codapi engine.

can this be a Web Component shadowDOM reference

Nope.


feat: I/O safety for 'sys/uid' & 'sched'

What this PR does:
Adds I/O safety for modules:

sys/uio
sched

The Ci failure, again, is a network issue:(
    Updating crates.io index
    Updating git repository `https://github.com/rust-lang/libc`
warning: spurious network error (2 tries remaining): SecureTransport error: (null); class=Net (12)
warning: spurious network error (1 tries remaining): SecureTransport error: (null); class=Net (12)
error: failed to get `libc` as a dependency of package `nix v0.26.1 (/private/var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build)`

Caused by:
  failed to load source for dependency `libc`

Caused by:
  Unable to update https://github.com/rust-lang/libc?rev=15d27952bfa93e5e4f419c603f275486f15a050c

Caused by:
  failed to clone into: /Users/admin/.cargo/git/db/libc-d1297d136f47e864

Caused by:
  network failure seems to have happened
  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  SecureTransport error: (null); class=Net (12)

Exit status: 101


Expose plugin API version and semver compatible check

Motivation
/kind feature
Consumers might need to print the supported plugin API version :point_down:
https://github.com/falcosecurity/libs/blob/de277bbc1e7a92edc84db9a925fb3e12c7b1cb07/userspace/libscap/plugin_info.h#L30-L32
/kind bug
Moreover, we can assume that using semver for version checking is a common expectation of our users.
However, the current implementation checks the MAJOR number only :point_down:
https://github.com/falcosecurity/libs/blob/de277bbc1e7a92edc84db9a925fb3e12c7b1cb07/userspace/libsinsp/plugin.cpp#L434-L455
Feature

Expose the current plugin framework API version (ie. by adding get_plugin_version to scap or sinsp)
Perform a proper SemVer version check when checking for the plugin compatibility

Alternatives
Considering the simplicity of this solution and assuming semver is a de facto standard, I don't see any other real alternative.
Additional context

/assing @FedeDP


Vendoring strategy for pop/sqlx

Since pop is a wrapper for sqlx and pop doesn't vendor sqlx, what are your recommendation for keeping in sync with x version of sqlx?

That's the age old question in Go: "How does a library vendor/lock a dependency?". I don't have an answer, since it's a hotly debated topic through out the community. I had thought of vendoring, but a lot of people feel that a library shouldn't vendor, only an application. I'm open to suggestions.


Feature Request: [[Brief Description of Feature]] moving the project to vite

the current project is taking a lot of time before starting.
we can move it to the vite which is a build tool that aims to provide a faster and leaner development experience for modern web projects.
a lot more here https://vitejs.dev/guide/

I know but this project is full based on Docusaurus. So, read all about this


Linear mode with adjustable speeds and sync

Not all signals are going to produce pretty pictures. I think this would make an excellent regular scope given its attention to detail and aesthetics. But we need a way to switch to that mode and a slider for setting speed.
Optionally, on real scopes, you can set a different signal to sync to. That'd be nice. Say, use the left channel as sound source and right channel as sync, or vice versa.

speed adjustment is there in the code, just not released yet.
can you explain the sync more? (you mean a trigger?)
this is meant as a xy scope, so a trigger isn't really necessary.

Yes, after looking up the term, I mean a trigger. From using it, I can tell that this is an XY scope. That much is obvious. What I'm asking though is... could it also be a traditional oscilloscope? Otherwise the name of the app and the screenshot on the home page are kind of misleading.

yes, you have a point there. maybe i'll add trigger support one day (there is mono channel support if you compile it yourself), maybe i'll rename it to xy-oscilloscope. who knows what the future will bring :)

mono mode is relased now (happens automatically when you have only one channel in the audio file).
i'll close this bug for now, everything else you mentioned is a bit out of scope for a bugfix (code contributions as PRs are welcome)


Denseclus on large data set

Hi, hi. 😊
I would  kindly ask for a little bit of help.
I have ta large data set (over 6mil rows and 20 columns ) with mixtured datatypes.
The idea was to implement the DenseClus.
(mix of umap embeddings for different data types with HDBSCAN).
I've implemented on subset oh 100K.
But the OOM error popped up really soon.
My core questions would be?

Is there option of fitting DanseClaus
on this kind of dataset?
Recommendations related to prams?
There is no possibility of transform the unseen data/ how to apply on entire data set?
As I work on DataBricks, some recommendations regarding the memory-performance of indtance?
The strategy of dealing with this kind of task?

Additional advices are welcoming. 🤗

@jovanaarsic In your case, I would recommend not using it if the dataset size is so large that it cannot fit on single machine. Instead looking it to other algorithms for clustering support by Spark MLLib. Alternatively, you could sub-sample to get a dataset that fits on single machine and run.


Docs for kong.tools.responses.send() reference removed 'raw' param

Summary
https://getkong.org/docs/0.9.x/lua-reference/modules/kong.tools.responses/ includes a method signature that no longer exists.
send (status_code, body, raw, headers)

It appears the 'raw' param was removed 5 months ago.
Is there a way to send a raw string response using this method?  For instance, I'd like to return pretty-printed JSON, so:
content
not
cjson.encode {message = content})
Steps To Reproduce

Additional Details & Logs

Kong version (0.9)
Kong debug-level startup logs ($ kong start --vv)
Kong error logs (<KONG_PREFIX>/logs/error.log)
Kong configuration (registered APIs/Plugins & configuration file)
Operating System

Thanks for the report.
You could use the plain ngx.say() method to send your responses as-is for now.

Thanks!  I'm a casual user and don't have the backstory on the removal of 'raw' but my naive feeling is it would be nice to have that option.

It is mostly that I am not a fan of necessary abstractions... Until we come up with a proper plugin development API promising backwards compatibility, we'd rather avoid such abstractions.


Add draft text (current version) for methods and results/discussion of F&F section

Hi Dilay, I'll merge this but maybe save up future PRs until it's ready to go?
Also, when you merge the final version, please could you update your markdown to use the right style for references, figures, links, etc.?
https://comob-project.github.io/snn-sound-localization/organising#referencing