Introduced an Optimizable protocol and made all layers optimizable.

This should be merged after #319 and #320 as I have already merged their changes into this.
You can see the changes introduced in this PR in isolation here.

I don't think introducing new protocols (Optimizable or ParameterlessLayer) is the right direction here. Adding dummy protocols is sort of an anti-pattern in Swift. Ultimately we just want certain constraints to be able to trigger derived conformances so that TangentVector gets VectorProtocol conformance. Have you tried adding an explicit struct TangentVector to each layer?

Sorry, Optimizable is not a new protocol, but rather a type alias grouping multiple constraints together. The issue with struct TangentVector is that it needs to be explicitly added to all parameter-less layers which is not very user friendly. Ideally we want users to be able to create parameter-less layers without worrying about TangentVector explicitly. Also, just adding that struct without the extra EmptyParameterSet struct I added, results in compiler errors. For EmptyParameterSet I had to add explicit implementations for VectorProtocol because the compiler was unable to derive them otherwise.
The nice thing about this solution is that the only change that needs to be made to support parameter-less layers is that they need to declare conformance to ParameterlessLayer instead of Layer and everything else works out fine on the compiler side.

The Optimizable type alias is defined here.

I really like the approach you propose and also feel we are not far from that. However, I'm not very familiar with how swift evolution works and how easy it would be to change Numeric depending on how often it's used in existing code.
Note that to completely remove the need for KeyPathIterable from optimizers we would also need to support abs(_:), which I don't think is in ElementaryFunctions.

I removed Optimizable and so the remaining changes simply make all layers usable with any optimizer we want.

However, I'm not very familiar with how swift evolution works and how easy it would be to change Numeric depending on how often it's used in existing code.

ABI stability for Numeric is the biggest issue that's blocking this from happening right now AFAIK.

Note that to completely remove the need for KeyPathIterable from optimizers we would also need to support abs(_:), which I don't think is in ElementaryFunctions.

Good point. If Magnitude can get removed from Numeric, it means that we can derive both Numeric and SignedNumeric. abs(_:) sounds like something that can be added to SignedNumeric.
It would be great to ask a quick question about removing Magnitude from Numeric on the evolution forum. Are you interested in starting a discussion there?

I can do that, but I first need to understand a bit more about how magnitude is being used. It sounds to me like the semantics of magnitude are equivalent to abs(_:)?
Also, just to get a bit of understanding about the evolution process,  how do we go about proposing to remove some functionality? I imagine we need a good reason to do so and in this case I don't necessarily see why it is necessary to remove it. Couldn't we simply make Magnitude equal to Self for tensors and set the getter to be the abs(_:) function? This way we also don't need to add that to SignedNumeric and we don't break compatibility with the current Numeric as defined in the standard library.

The nice thing about this solution is that the only change that needs to be made to support parameter-less layers is that they need to declare conformance to ParameterlessLayer instead of Layer and everything else works out fine on the compiler side.

The issue is that it is a public API that does not guarantee the semantics of conforming types. A type can totally declare a ParameterlessLayer and have parameters or its own TangentVector, in which case the ParameterlessLayer conformance would contradict the fact that such layer has parameters. Any public protocol would change the user model quite significantly, and should be considered thoroughly. In this particular case, instead of providing a default empty TangentVector to conforming types, I don't see ParameterlessLayer can provide any useful information to a generic algorithm that declares a conformance constraint to this protocol.
I'd recommend against directly dropping in public, user-visible protocols, but instead adding a custom TangentVector structure to any such parameterless layer. This won't add any cognitive burden on the user in looking at the protocol hierarchy. After this, a natural next step would be to figure out a default VectorSpaceScalar for empty structs. Perhaps we can define an empty struct that conforms to Numeric and can act as the default VectorSpaceScalar that the compiler derives.

Oh I see your point. Yes I also dislike that it does not guarantee the semantics of the conforming type. Let me see if I can work out something simpler with the empty struct.

I can do that, but I first need to understand a bit more about how magnitude is being used. It sounds to me like the semantics of magnitude are equivalent to abs(_:)?

That is a good point! Any thoughts, @stephentyrone? It would be great to remove even the Comparable constraint from Magnitude. The Comparable constraint is what's really making Numeric non-derivable.

Yes I agree. I don't think Comparable is necessary there. This actually makes me wonder where that constraint is being used.

I just did a quick search through the code and it's only being used in very few places. In principle, we can add the Comparable constraint there separately.

Yeah. In the case that the Comparable constraint can't be removed, we can technically still derive Numeric but it will require deriving Comparable for aggregates (comparison followed by reduction), and I'm not sure if it's the right thing.

Do you think an evolution pitch would be easier if I first try to remove that constraint and see how many changes this results into in the standard library?

Do you think an evolution pitch would be easier if I first try to remove that constraint and see how many changes this results into in the standard library?

I suspect there will be some pushback against this on the forum, so it'd be nice to see the reaction from the community first. If there's a serious ABI concern, it's probably just not possible.

I created a discussion topic here. I'm not sure if what I said is sufficient to start this discussion, but please feel free to add any additional comments you may have.
In the meantime, do you think that the changes introduced in this PR are acceptable so we can support optimizers with all layers as long as I remove ParameterlessLayer and add TangentVector structs to all parameter-less layers? I'll try to push these changes in a little bit.

Nice post! Just a note: ElementaryFunctions wasn't designed as part of Swift for TensorFlow. We only made it derivable.

In the meantime, do you think that the changes introduced in this PR are acceptable so we can support optimizers with all layers as long as I remove ParameterlessLayer and add TangentVector structs to all parameter-less layers? I'll try to push these changes in a little bit.

Yep!

Thanks and good catch! :) I edited the post.

So, if I don't define allDifferentiableVariables and move(along:) as I was doing here:
public extension ParameterlessLayer {
    var allDifferentiableVariables: EmptyParameterSet {
        get { return EmptyParameterSet() }
        set {}
    }

    mutating func move(along direction: EmptyParameterSet) {}
}

I get the following compiler error:
Assertion failed: (Storage.hasVal), function getPointer, file /Users/eaplatanios/Development/GitHub/swift-source-temp/llvm/include/llvm/ADT/Optional.h, line 180.
Stack dump:
0.      Program arguments: /Library/Developer/Toolchains/swift-tensorflow-LOCAL-2019-06-30-a.xctoolchain/usr/bin/swift -frontend -c /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Bindings/EagerExecution.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Bindings/RawOpsGenerated.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Bindings/TFTensorOperation.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Context.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/ArrayOps.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/DataTypes.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/DifferentialOperators.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Execution.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/LazyTensorOperation.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/LazyTensorTFFunctionBuilder.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/LazyTensorTrace.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/PythonConversion.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Runtime.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/ShapedArray.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/StringTensor.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Tensor.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/TensorGroup.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/TensorHandle.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/TensorProtocol.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/TensorShape.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Threading.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Utilities.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Initializers.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layer.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/BackwardsCompatibility.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Convolutional.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Core.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Embedding.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Initialization.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Normalization.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Pooling.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Recurrent.swift -primary-file /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Upsampling.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Loss.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/Basic.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/Comparison.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/Dataset.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/Math.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/NN.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Optimizers/MomentumBased.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Optimizers/Optimizer.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Optimizers/SGD.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Random.swift -emit-module-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Upsampling~partial.swiftmodule -emit-module-doc-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Upsampling~partial.swiftdoc -emit-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Upsampling.d -emit-reference-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Upsampling.swiftdeps -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -I /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug -I /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib -F /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks -enable-testing -g -module-cache-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/ModuleCache -swift-version 4.2 -Onone -D SWIFT_PACKAGE -D DEBUG -color-diagnostics -enable-anonymous-context-mangled-names -parse-as-library -module-name TensorFlow -o /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Upsampling.swift.o -index-store-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/index/store -index-system-modules
1.      Swift version 5.1-dev (LLVM 7d54a7c711, Swift f62bd5687d)
2.      While type-checking 'UpSampling3D' (at /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Upsampling.swift:73:8)
3.      While type-checking 'repeatingElements(_:alongAxis:count:)' (at /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Upsampling.swift:89:13)
4.      While validating 'repeatingElements(_:alongAxis:count:)' (at /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Upsampling.swift:89:13)
0  swift                    0x00000001128d3955 llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 37
1  swift                    0x00000001128d2998 llvm::sys::RunSignalHandlers() + 248
2  swift                    0x00000001128d3f48 SignalHandler(int) + 264
3  libsystem_platform.dylib 0x00007fff759bcb5d _sigtramp + 29
4  libsystem_platform.dylib 0x0000000000004250 _sigtramp +<PHONE_NUMBER>
5  libsystem_c.dylib        0x00007fff758766a6 abort + 127
6  libsystem_c.dylib        0x00007fff7583f20d basename_r + 0
7  swift                    0x0000000112d5e583 swift::AnyFunctionType::getAutoDiffAssociatedFunctionType(swift::AutoDiffParameterIndices*, unsigned int, unsigned int, swift::AutoDiffAssociatedFunctionKind, llvm::function_ref<llvm::Optional<swift::ProtocolConformanceRef> (swift::CanType, swift::Type, swift::ProtocolDecl*)>, swift::GenericSignature*, bool) (.cold.25) + 35
8  swift                    0x000000010fad941c swift::AnyFunctionType::getAutoDiffAssociatedFunctionType(swift::AutoDiffParameterIndices*, unsigned int, unsigned int, swift::AutoDiffAssociatedFunctionKind, llvm::function_ref<llvm::Optional<swift::ProtocolConformanceRef> (swift::CanType, swift::Type, swift::ProtocolDecl*)>, swift::GenericSignature*, bool) + 3676
9  swift                    0x000000010f6b7d07 swift::ASTVisitor<(anonymous namespace)::AttributeChecker, void, void, void, void, void, void>::visit(swift::DeclAttribute*) + 17655
10 swift                    0x000000010f6b37f1 swift::TypeChecker::checkDeclAttributes(swift::Decl*) + 49
11 swift                    0x000000010f6f4988 swift::TypeChecker::validateDecl(swift::ValueDecl*) + 6008
12 swift                    0x000000010f6fd1c0 (anonymous namespace)::DeclChecker::visitFuncDecl(swift::FuncDecl*) + 48
13 swift                    0x000000010f6efde3 (anonymous namespace)::DeclChecker::visit(swift::Decl*) + 243
14 swift                    0x000000010f6fb5ab (anonymous namespace)::DeclChecker::visitStructDecl(swift::StructDecl*) + 139
15 swift                    0x000000010f6f005d (anonymous namespace)::DeclChecker::visit(swift::Decl*) + 877
16 swift                    0x000000010f6efce6 swift::TypeChecker::typeCheckDecl(swift::Decl*) + 38
17 swift                    0x000000010f790bdc swift::performTypeChecking(swift::SourceFile&, swift::TopLevelContext&, swift::OptionSet<swift::TypeCheckingFlags, unsigned int>, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int) + 940
18 swift                    0x000000010eec9d71 swift::CompilerInstance::parseAndCheckTypesUpTo(swift::CompilerInstance::ImplicitImports const&, swift::SourceFile::ASTStage_t)::$_1::operator()(swift::SourceFile&) const + 81
19 swift                    0x000000010eec574b swift::CompilerInstance::parseAndCheckTypesUpTo(swift::CompilerInstance::ImplicitImports const&, swift::SourceFile::ASTStage_t) + 507
20 swift                    0x000000010eec504a swift::CompilerInstance::performSemaUpTo(swift::SourceFile::ASTStage_t) + 618
21 swift                    0x000000010ecb2e92 performCompile(swift::CompilerInstance&, swift::CompilerInvocation&, llvm::ArrayRef<char const*>, int&, swift::FrontendObserver*, swift::UnifiedStatsReporter*) + 1730
22 swift                    0x000000010ecb17f1 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 3025
23 swift                    0x000000010ec59bf9 main + 729
24 libdyld.dylib            0x00007fff757d13d5 start + 1

which seems to me like a bug.
An example of how I define a parameter-less layer now, giving me the above error, is this:
public struct GlobalMaxPool2D<Scalar: TensorFlowFloatingPoint>: Layer {
    public typealias TangentVector = EmptyParameterSet

    /// Creates a global max pooling layer.
    public init() {}

    /// Returns the output obtained from applying the layer to the given input.
    ///
    /// - Parameter input: The input to the layer.
    /// - Returns: The output.
    @differentiable
    public func callAsFunction(_ input: Tensor<Scalar>) -> Tensor<Scalar> {
        return input.max(squeezingAxes: [1, 2])
    }
}

Side comment about Magnitude. As a backup option we can always define it as Float or Double and then have it be the sum of the magnitudes of all things in the aggregate type. For tensors this could be defined as the Frobenius norm of the tensor. This would give us reasonable semantics.

What if you define both
public struct AllDifferentiableVariables: Differentiable {}

and
public struct TangentVector: Differentiable, VectorProtocol {
    public typealias VectorSpaceScalar = Float
}

?

This results in a lot of errors indicating that no conformances were derived:
/Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Convolutional.swift:616:15: error: type 'ZeroPadding3D<Scalar>.TangentVector' does not conform to protocol 'VectorProtocol'
public struct TangentVector: Differentiable, VectorProtocol {
              ^
Swift.VectorProtocol:3:10: note: protocol requires function 'adding' with type '(ZeroPadding3D<Scalar>.TangentVector.VectorSpaceScalar) -> ZeroPadding3D<Scalar>.TangentVector'; do you want to add a stub?
    func adding(_ x: Self.VectorSpaceScalar) -> Self
         ^
Swift.VectorProtocol:5:10: note: protocol requires function 'subtracting' with type '(ZeroPadding3D<Scalar>.TangentVector.VectorSpaceScalar) -> ZeroPadding3D<Scalar>.TangentVector'; do you want to add a stub?
    func subtracting(_ x: Self.VectorSpaceScalar) -> Self
         ^
Swift.VectorProtocol:7:10: note: protocol requires function 'scaled(by:)' with type '(ZeroPadding3D<Scalar>.TangentVector.VectorSpaceScalar) -> ZeroPadding3D<Scalar>.TangentVector'; do you want to add a stub?
    func scaled(by scalar: Self.VectorSpaceScalar) -> Self
         ^
/Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Convolutional.swift:614:15: error: type 'ZeroPadding3D<Scalar>.TangentVector' does not conform to protocol 'ElementaryFunctions'
public struct ZeroPadding3D<Scalar: TensorFlowFloatingPoint>: Layer {
              ^
/Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Convolutional.swift:614:15: error: type 'ZeroPadding3D<Scalar>' does not conform to protocol 'Differentiable'
public struct ZeroPadding3D<Scalar: TensorFlowFloatingPoint>: Layer {
              ^
Swift.Differentiable:2:16: note: candidate has non-matching type 'Self'
    public var allDifferentiableVariables: Self { get set }
               ^
Swift.Differentiable:2:26: note: candidate would match if 'ZeroPadding3D<Scalar>' was the same type as 'ZeroPadding3D<τ_0_0>.TangentVector'
    public mutating func move(along direction: Self.TangentVector)
                         ^
Swift.Differentiable:4:9: note: protocol requires property 'allDifferentiableVariables' with type 'ZeroPadding3D<Scalar>.AllDifferentiableVariables'; do you want to add a stub?
    var allDifferentiableVariables: Self.AllDifferentiableVariables { get set }
        ^
Swift.Differentiable:5:19: note: protocol requires function 'move(along:)' with type '(ZeroPadding3D<Scalar>.TangentVector) -> ()'; do you want to add a stub?
    mutating func move(along direction: Self.TangentVector)
                  ^

Is there anything similar to the final keyword from Java/Scala in Swift? I.e., a way to declare that a type/property cannot be overridden by conforming types?

This probably means derived conformances are currently hard coded to not trigger derive if there are no members. We should fix that.

Is there anything similar to the final keyword from Java/Scala in Swift? I.e., a way to declare that a type/property cannot be overridden by conforming types?

There is final, but final does not apply to protocols. All protocol requirements are final once implemented.

This probably means derived conformances are currently hard coded to not trigger derive if there are no members. We should fix that.

I looked a bit into this earlier and it seems that this assertion is failing (at least when the compiler was crashing). Maybe this is also related to what happens when there are no members.

Actually never mind, this line seems to be where the check happens. Or at least one such check.

Yeah, looks like that should be fixed.
Are you also interested in implementing a @memberwise attribute for triggering AdditiveArithmetic, VectorProtocol, ElementaryFunctions, and Differentiable derived conformances? This is described here.

Unfortunately I don't have time to look into it this week because I'm rushing with some research projects and will also be traveling Wednesday-Sunday. I may be able to look into it when I get back but I can't know yet.

Quick observation: removing the condition in this line does not fix the problem.

Side comment about Magnitude. As a backup option we can always define it as Float or Double and then have it be the sum of the magnitudes of all things in the aggregate type. For tensors this could be defined as the Frobenius norm of the tensor. This would give us reasonable semantics.

I believe we can just go with this as it's technically not wrong given the loose semantics that magnitude already has.

@rxwei could we have the automatic derivation for VectorProtocol derive VectorSpaceScalar as an empty struct itself, for empty structs? I don't why an empty struct cannot conform to AdditiveArithmetic.

Also, I realized that the ParameterlessLayer approach I original implemented for this does guarantee the semantics of the conforming type as it forces AllDifferentiableVariables to be equal to EmptyParameterSet. No matter what properties the users of the protocol define in conforming structs, these properties will not be differentiating and will thus not be treated as model parameters. So, I don't think semantics is a problem here. What do you think?

Also, I realized that the ParameterlessLayer approach I original implemented for this does guarantee the semantics of the conforming type as it forces AllDifferentiableVariables to be equal to EmptyParameterSet.

I don't think that's the case. The user can always declare a custom TangentVector and go against the "parameterless" promise as specified by such a protocol. Protocols that do not declare new requirements are kind of an anti-pattern in Swift because they do not provide useful information for generic algorithms. There are only extremely few cases where we considered carefully and used these empty protocols, namely TensorFlowFloatingPoint, but it was only motivated by the fact that typealias doesn't support where-clauses yet.

My recommendation is going a different route:

Phase 1: Make parameterless layers compatible with optimizers (as we discussed upstream).

Fix derived conformances for VectorProtocol so that it can be derived for empty structs so long as they declare a VectorSpaceScalar.
Declare an empty TangentVector and AllDifferentiableVariables in each of the parameterless layers.

Phase 2: Remove special cases in TangentVector conformance derivation.

Fix the bug where one cannot extend a compiler-synthesized TangentVector.
Remove the special case that "if all members conform to VectorProtocol/ElementaryFunctions, conform TangentVector to VectorProtocol/ElementaryFunctions" from Differentiable derived conformances.
Extend TangentVectors to conform to VectorProtocol/ElementaryFunctions for each layer.

Phase 3: Pitch VectorProtocol to Swift Evolution.

Carefully examine whether VectorProtocol should be derived for empty structs.  If so, consider adding an empty field type to stdlib that can act as the default VectorSpaceScalar (I feel it's unlikely, but just something for discussion).
Constrain TangentVector on a conformance to VectorProtocol in addition to AdditiveArithmetic.

That plan sounds good, although phase 2 seems like it might require a lot
of work. Do you think you’ll be able to tackle phase 1, fixing the derived
conformances?
Regarding the ParameterlessLayer protocol approach, note that there is a
AllDifferentiableVariables == TangentVector constraint on Layer and the
additional constraint TangentVector == EmptyParameterSet on
ParameterlessLayer so I still don’t think it’s possible to override the
expected behavior. Could you please describe how that would be possible? I
understand the hesitation with respect to introducing a new protocol, but I
also want to understand why it cannot guarantee the desired functionality.
On Mon, Jul 1, 2019 at 2:27 PM Richard<EMAIL_ADDRESS>wrote:

My recommendation is going a different route:

Phase 1: Make parameterless layers compatible with optimizers (as we
discussed upstream).

Fix derived conformances for VectorProtocol so that it can be
derived for empty structs so long as they declare a
VectorSpaceScalar.
Declare an empty TangentVector and AllDifferentiableVariables in
each of the parameterless layers.

Phase 2: Remove special cases in TangentVector conformance
derivation.

Fix the bug
https://forums.swift.org/t/synthesized-member-types-and-type-checking/26386
where one cannot extend a compiler-synthesized TangentVector.
Remove the special case that "if all members conform to
VectorProtocol/ElementaryFunctions, conform TangentVector to
VectorProtocol/ElementaryFunctions" from Differentiable derived
conformances.
Extend TangentVectors to conform to VectorProtocol/
ElementaryFunctions for each layer.

Phase 3: Pitch VectorProtocol to Swift Evolution.

Carefully examine whether VectorProtocol should be derived for
empty structs. If so, consider adding an empty field type to stdlib that
can act as the default VectorSpaceScalar (I feel it's unlikely, but
just something for discussion).
Constrain TangentVector on a conformance to VectorProtocol in
addition to AdditiveArithmetic.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/swift-apis/pull/322?email_source=notifications&email_token=AAJ4EXDPR42Q22HEZBYE5K3P5JEBJA5CNFSM4H4OVI32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY66UZQ#issuecomment-507374182,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJ4EXHVRADO5M7Z7JUGLADP5JEBJANCNFSM4H4OVI3Q
.

(Note: I filed TF-632 to track phase 2. I'm interested in working on it but won't start for a while.)

That plan sounds good, although phase 2 seems like it might require a lot of work. Do you think you’ll be able to tackle phase 1, fixing the derived conformances?

Unfortunately I don't be able to tackle the derived conformances fix this week. Feel free to tackle it when you get a chance. It should be a chance of synthesis condition plus a bunch of tests.

Could you please describe how that would be possible? I understand the hesitation with respect to introducing a new protocol, but I also want to understand why it cannot guarantee the desired functionality.

Sorry I omitted some details here. AllDifferentiableVariables is going to be deprecated and removed from the Differentiable protocol. This means that we will not be able to have this sort of mutually recursive constraint when the removal is implemented.

I might tackle it if I get a chance in the next few days, or later today.
Regarding your second comment, even if AllDifferentiableVariables is
deprecated, ParameterlessLayer adds the constraint that TangentVector == EmptyParameterSet which the user cannot override, so doesn’t this
guarantee the desired behavior?
On Mon, Jul 1, 2019 at 3:05 PM Richard Wei<EMAIL_ADDRESS>wrote:

That plan sounds good, although phase 2 seems like it might require a lot
of work. Do you think you’ll be able to tackle phase 1, fixing the derived
conformances?
Unfortunately I don't be able to tackle the derived conformances fix this
week. Feel free to tackle it when you get a chance. It should be a chance
of synthesis condition plus a bunch of tests.
Could you please describe how that would be possible? I understand the
hesitation with respect to introducing a new protocol, but I also want to
understand why it cannot guarantee the desired functionality.
Sorry I omitted some details here. AllDifferentiableVariables is going to
be deprecated and removed from the Differentiable protocol. This means
that we will not be able to have this sort of mutually recursive constraint
when the removal is implemented.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/swift-apis/pull/322?email_source=notifications&email_token=AAJ4EXCOLNFLPDKCKU3RSHTP5JIN3A5CNFSM4H4OVI32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY7BWEY#issuecomment-507386643,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJ4EXG5PKRMJUT4RPLI6ALP5JIN3ANCNFSM4H4OVI3Q
.

Regarding your second comment, even if AllDifferentiableVariables is deprecated, ParameterlessLayer adds the constraint that TangentVector == EmptyParameterSet which the user cannot override, so doesn’t this guarantee the desired behavior?

Yeah, you are right that TangentVector is property constrained. The other reservation I had is that adding a new public API EmptyParameterSet is pretty suboptimal. Perhaps we can call it EmptyTangentVector or something.

I could make that change. Also, when we fix the derived conformances for
empty structs we can completely remove that struct and set TangentVector
to be an empty struct.
On Mon, Jul 1, 2019 at 3:32 PM Richard Wei<EMAIL_ADDRESS>wrote:

Regarding your second comment, even if AllDifferentiableVariables is
deprecated, ParameterlessLayer adds the constraint that TangentVector ==
EmptyParameterSet which the user cannot override, so doesn’t this
guarantee the desired behavior?
Yeah, you are right that TangentVector is property constrained. The other
reservation I had is that adding a new public API EmptyParameterSet is
pretty suboptimal. Perhaps we can call it EmptyTangentVector or something.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/swift-apis/pull/322?email_source=notifications&email_token=AAJ4EXCOPG5IFYNGVHUSQDLP5JLTLA5CNFSM4H4OVI32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY7DXXY#issuecomment-507395039,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJ4EXC46JITXVHSLSCT4NLP5JLTLANCNFSM4H4OVI3Q
.

Done. This should be ready for review now.

Also, is it ok if I make a separate PR to add a ParameterScalar associated type to Layer? I believe it's safe to assume that all parameters of a layer are of a single type and it makes my life easier with some stuff I'm working on which "collect" all parameters using KeyPathIterable. In this case, it's nice to be explicit about the type of the parameters so that "unconsidered" types are not silently ignored (as it happens with the key-path-based optimizers, for example).
I attempted this change and only 2 lines of code change in all of the TensorFlow module.

Also, is it ok if I make a separate PR to add a ParameterScalar associated type to Layer? I believe it's safe to assume that all parameters of a layer are of a single type and it makes my life easier with some stuff I'm working on which "collect" all parameters using KeyPathIterable. In this case, it's nice to be explicit about the type of the parameters so that "unconsidered" types are not silently ignored (as it happens with the key-path-based optimizers, for example).

This direction was specifically considered, implemented, and rejected in the Parameterized/ParameterGroup effort. We've decided not to hard-code any scalar type in the Layer protocol because we'd like to keep APIs future-proof (supporting multiple-precision training, for example).

The PR title should be corrected to "Make all layers' tangent vectors conform" to reflect what's actually going on.

The PR title should be corrected to "Make all layers' tangent vectors conform" to reflect what's actually going on.

Good catch, thanks Richard! I made all the requested changes so I'll run tests and merge if they pass.

This direction was specifically considered, implemented, and rejected in the Parameterized/ParameterGroup effort. We've decided not to hard-code any scalar type in the Layer protocol because we'd like to keep APIs future-proof (supporting multiple-precision training, for example).

Is it possible to add a recursivelyAllWritableKeyPaths(conformingTo:) function that accepts a protocol that the traversed key path types must conform to?

It is only possible for non-PAT protocols. Protocol metatypes cannot be of PATs, for example:
test.swift:1:9: error: protocol 'Differentiable' can only be used as a generic constraint because it has Self or associated type requirements
let x = Differentiable.self
        ^

Thanks!

@rxwei @dan-zheng I just bumped into one more requirement for aggregates over tensors that may be good to think about. Say we want to add support for clipping gradient based on the global norm. That requires us to compute the norm of each tensor in the aggregate (norm[t]) and then compute:
globalNorm = sqrt(sum([norm[t].squared() for t in tensors]))

Say we can compute sqrt and squared() using a conformance to ElementaryFunctions. How do we go about the sum reduction over the aggregate?

For reference, this is the current hack solution I'm using that I don't particularly like since layers don't have a ParameterScalar type:
extension KeyPathIterable {
  public mutating func clipByGlobalNorm<Scalar: TensorFlowFloatingPoint>(clipNorm: Scalar) {
    let clipNorm = Tensor<Scalar>(clipNorm)
    var globalNorm = Tensor<Scalar>(zeros: [])
    for kp in self.recursivelyAllWritableKeyPaths(to: Tensor<Scalar>.self) {
      globalNorm += self[keyPath: kp].squared().sum()
    }
    globalNorm = sqrt(globalNorm)
    for kp in self.recursivelyAllWritableKeyPaths(to: Tensor<Scalar>.self) {
      self[keyPath: kp] *= clipNorm / max(globalNorm, clipNorm)
    }
  }
}

@rxwei @dan-zheng with respect to the protocols, why do we want to related VectorProtocol to Numeric? Can't we let Numeric be and design completely separate protocols that are defined over tensor structures?

I think this PR is raising a bunch of interesting points about tradeoffs in the API design and the direction we're going with compiler synthesis and the core language protocols.  Could one of you bring this to the swift@tensorflow mailing list, and/or a design meeting?


Not an issue. Letting you know I used it on an ad8400 and it works fine.

I do have a couple of questions as I am trying to use it with a circuitpython project and there is no library for it. I looked through your library and I'm a newb with libraries. Is the baudrate 16000000?
Do you set the mode and the phase?
What is the format of the data sent to the chip?
I'm close to having it work but it's hit or miss if the chip reacts to the command or not
Thanks!!

@muttstang
Sorry for the extreme late reaction, this issues of this repo slipped through somehow
Thanks for testing the AD8400, I update the readme with this info

Is the baudrate 16000000?

The max baudrate differs per board, e.g. the UNO can only do 4 Mbit.
Setting it to 16M gives most boards max performance.

Do you set the mode and the phase?

These are set in begin()
  _spi_settings = SPISettings(_SPIspeed, MSBFIRST, SPI_MODE1);

What is the format of the data sent to the chip?

MSB_FIRST
Hope this answers your questions,
Again sorry for the late reply.
PS, I have worked on a dual channel command today in the develop branch.
These wont work for the AD8400 of course as it has one channel.

If questions are answered you may close the issue.

No worries. I ended up running a different (i2c) chip and just using basic commands.


re-deploy holesky native staking strategy

re-deploy holesky native staking strategy
Should be merged before this PR https://github.com/OriginProtocol/origin-dollar/pull/2037 is merged

@naddison36 I'll just merge this in, since there isn't much to code review. Still nice that it is a separate PR IMO


How to troubleshoot when it has no effect?

When I throw an error, the stack trace line points to the built file, not the original one. (It makes no difference if I register source-map-support or not.)
I think everything is set up as per the instructions: I have a source map next to the built file, and a source mapping comment at the bottom of the built file that points to that source map. And the source map correctly points back to the original file. At the top of my script, I have require('source-map-support').install();. But it seems to have no effect.
Or maybe there is something wrong with my setup, but I can't tell. Is it possible to put source-map-support into some kind of 'strict' mode, so it errors noisily if it fails to find/parse a source map?

I just tested my sourcemap using a SourceMapConsumer, and it worked fine (I was able to find correct source position for a given position in the generated file). So I'm guessing the problem is that source-map-support can't actually find the sourcemap?
But I can't see any problems... I have a generated file foo.js with this comment at the end of it:
//# sourceMappingURL=foo.js.map

And foo.js.map is a valid sourcemap which is in the same directory as the generated file. So it all looks correct to me.

I worked it out... I was testing it by doing throw 'oops', i.e. I was throwing a string not an error. This gave a kind of stack trace, but I guess it wasn't a 'real' stack trace so source-map-support couldn't hook into it. Hope this helps someone.


lastUpdated inconsistently rounded

Sometimes lastUpdated is returned with millisecond precision, and sometimes rounded to the second.
Easily reproduced on MySql with Toolbox when doing updates with RestController and JsonBlobService.  The update typically returns the Blob with millisecond precision, but subsequent listing, will return the same value rounded to the second.

@lbwexler - it sounds like the column in your local database might be of type datetime(0) - that's second precision. On my local DB and deployed toolbox DBs, we see datetime(6) - microsecond precision, and I cannot replicate the issue.
I'm pretty confident that is how GORM will auto-create the schema - can you check your local DB to confirm?

Not sure how your schema ended up in that state - potentially manually created or adjusted at some point in the past?

Eureka -- that is indeed the state of several (but not all) of my datetime columns!

BUT -- look here -- SYBASE, for example, only seems to store datetimes down to 1/300 of a second -- NOT a full millisecond.  So my strong guess is that this is still going to be an issue in general for code that we write that compares a ms timestamp before and after its round-tripping in the DB.

I am going to close this, as I don't think there is an actionable bug here, but we do need to be very careful about storing timestamps that we generate in javascript and java, and expecting to round-trip them in the DB and get back exact values.
Disappointing -- surprised this has not bitten us before.  I suppose if we really want to get the accuracy we could use the datetime2 (7) or a long to store a timestamp,

We should absolutely be using datetime2 for all Date/Instant columns in SQL server world.  This should be the default assuming the correct dialect is specified - you can see the SQLServer2008Dialect (and later) map Types.TIMESTAMP onto datetime2.
https://github.com/hibernate/hibernate-orm/blob/d7e8a69df5981cd8ad93c6819799c4cbca0b2e70/hibernate-core/src/main/java/org/hibernate/dialect/SQLServer2008Dialect.java#L29
Java Date is mapped to that TIMESTAMP type here:
https://github.com/hibernate/hibernate-orm/blob/d7e8a69df5981cd8ad93c6819799c4cbca0b2e70/hibernate-core/src/main/java/org/hibernate/type/descriptor/sql/JdbcTypeJavaClassMappings.java#L131


Slack action: "Push Slack thread to website"

For those who want to move out of Slack communities, abilities to push a thread from Slack via an action would be a great way to lower the barrier to entry

In a way, we now have this!
In another way, a Slack Action for this would be awesome, so you can pull more historical posts.


enable blob

About the PR
pro never uncommented it so weve just never been able to play it
Why / Balance
how do you think its meant to get tested???
Changelog
:cl:

fix: Fixed blob never spawning.

So, since yesterday, there was... Only 1 Blob? Spawned as an admeme after the update?!

there are no admins

speed merge please


Update jquery-ui-timepicker-el.js

Hi,
I corrected millisecText and microsecText translations.
Thank you.

Thank you for the fix.  I have added it to the dev branch.


[CLOUD-1600] chore: snyk code scanning

Weird, this is a renamed job

I think this PR is just borked (circle check that can never possibly return). I'll open a new one


Creating a multilevel dropdown menu

I had few requests to add multilevel dropdown menu to the sidebar menu. Well, good news! It's already included in the initial release and here is how to do it:

          Multilevel Menu

                First level

                        Second level

                            Third level

The javascript in the app.js file will automatically render the tree view! Enjoy :+1:

Nice!! Thank you very much!!

How use it in JSF? I can not make it work... Please help. Thank you in advance.

you can use jquery smartmenu
https://github.com/vadikom/smartmenus,
it is even devices friendly, here is my implementation under ng2:
http://plnkr.co/edit/wLqLUoBQYgcDwOgSoRfF?p=preview https://github.com/Longfld/DynamicaLoadMultiLevelDropDownMenu

Very nice, but how can I do it if using the top navigation layout?
Thanks in advance

Hi. I have the exact question of @fw-it : how can I do it if using the top navigation layout?
Can we use tree aswell in conjuction with downdown?
Thanks in advance


Import error: No module named theme

There is a problem with the instantiation of a theme app, according to the documentation, in the following way:
INSTALLED_APPS = (
    # Material apps
    'material.theme.blue',
    'material',
    'material.admin',
    'material.frontend',
    ...

It seems the file material/theme/__init__.py in the "theme" package definition is missing.

yeah, py 2.7  bug

I've released the fixed version 0.12.2 and pro.


Updating to add a new feature

New feature allow admins to perform forensic extraction and analysis of malware infected files. Also fixing a few point for article clarity.

@JohanFreelancer9

@AdwoodMicrosoft Gentle reminder

@AdwoodMicrosoft Please implement the suggested changes to proceed for the PR merge. Thanks


Create an EdDSA scheme using Ristretto and Blake3

EdDSA provides a lot of tunable parameters, and when Ed25519 was initially created, it used SHA-512 with the note that the hash function could be updated to SHA-3 or other systems later on. Ed25519 still uses SHA-512 in standard contexts, but with the introduction of the Ristretto255 encoding as an alternative to splitting up keys into two algorithms (X25519 and Ed25519), we can provide a signature scheme to match the signcryption scheme that Ristretto support was initially introduced to enable. Instead of using SHA-3, we'll use Blake3, and improved version of Blake2s which was a finalist in SHA-3 anyways. As Blake3 provides 128-bit security, this matches fairly well with Curve25519.
In order to unify the key format used with EdDSA, this scheme will need to modify the previous simplicity assumed by using a scalar/element pair as with signcryption. Instead of using the scalar as the private key, a random 32-byte seed or secret is used as the private key as in EdDSA, then that key is hashed to get the scalar and a prefix key used for creating challenge hashes. In Ed25519, this works by taking the SHA-512 hash of the seed. In this scheme, we have two options: take the Blake3 hash of the seed and finalize its output to 64 bytes, or use the seed as the key for a keyed hash, then finalize 64 bytes of output. The plain hash method matches the EdDSA standard more exactly, though the availability of a keyed hash mode, if properly secure, can simplify the design here a little bit.
When calculating the (r, s) signature pair in an EdDSA scheme, s is still a scalar and is encoded in the usual way. However, r is now a Ristretto element rather than a normal extended Edwards coordinate in that the serialization algorithm should follow the Ristretto encoding instead of the traditional Edwards point compression.
It might be relevant to find out more about including a key id like in signcryption and whether it's useful here, too.

After experimenting a bit and reading more about EdDSA and Blake3, it seems like the adjustments to Ed25519 can be the following:

Replace SHA-2 hashes with keyed Blake3 hashes. Instead of hashing the key first in each of the hash steps, these can put the first 32 bytes in the key. Specifically, this changes the following places:

Use the private key (seed) as the key to a Blake3 keyed hash. Finalize this output to 64 bytes with the first 32 bytes being clamped and parsed as the secret scalar and the second 32 bytes being used to initialize another keyed Blake3 hash which will be later used as the "challenge" hash for generating the R-half of signatures and is known as the "prefix" part of the expanded key.
When signing a message, instead of hashing the prefix, we start with this challenge hash, update with the message to be signed, then finalize the output to 64 bytes. Those bytes are reduced to a scalar r the same way as in Ed25519. Using this reduction step simplifies implementations modifying an existing Ed25519 library similar to other things Ristretto255 optimizes for.
This scalar is multiplied by the Ristretto base point which is the same as the Ed25519 base point, but its compressed output (used for the R-half of the signature) uses the Ristretto point compression encoding rather than the usual compressed Edwards coordinates.
The compressed point R is then used as the key for another Blake3 hash. This is then updated with the Ristretto encoding of the public key (earlier secret scalar multiplied by Ristretto base point) followed by the message contents. This is similar to the Ed25519 step but with R used as the hash key rather than as part of the initial hash followed by the public key and message as usual. This hash is finalized to 64 bytes and reduced to a scalar k using the same scalar reduction as before.
This scalar k is multiplied by the secret scalar, then their product is added to the scalar r. This resulting scalar s is serialized as the S-half of the signature.
Verification works as the inverse which uses the R-half of the signature to initialize a Blake3 hash, updates with the public key and message, then finalizes 64 bytes and reduces them to calculate k.
The verification equation can be validated by multiplying the negated public key by k and adding that to the product of s and the Ristretto base point, then checking if that resulting Ristretto element is equivalent to the point encoded in R.

There is one outstanding design choice to make here that might make things less "neat" but more correct: the value of R may be attacker-controlled during verification, but it has to satisfy the algebra and the hash output, so it might or might not benefit from using a plain Blake3 hash rather than a keyed hash for this step.
The nice thing about this algorithm is that it meshes better with the Ristretto signcryption scheme and Ristretto key exchange scheme rather than requiring special-purpose algorithms for different concerns (a common point of confusion by developers; see for example misuse of RSA when exposed improperly).

Switching the R value to initialize as hash input makes sense. After getting a proof of concept version working in both Java and C, there are a couple open issues:

Hash-to-scalar: can we just use the default 256-bit Blake3 output and reduce from there, or do we finalize a 512-bit Blake3 output and reduce that value to a scalar the same way Ed25519 does to the SHA-512 output? Another algorithm may need to be added to the C code to support the small reduction without the added overhead of a full 512-bit reduction.
Re-using the double-scalar-vartime algorithm from this paper which is already used in the Ed25519 code. The current version results in projective coordinates which will need to be math-checked to see how it can adapt to the ristretto equality check (might be a drop-in since projective and extended coordinates have a bit of overlap).

Possible red herring, but CFRG is looking at KangarooTwelve as a keyed XOF which, like Blake3, reduces the rounds of a more conservative design (in this case SHAKE from SHA-3) to something still provably secure.

For hash-to-scalar, I think this should use the 512-bit variant as there's a defined algorithm for mapping a uniform 64-byte string to a scalar, but for a 32-byte string, this only mentions that it should be validated to be within the proper scalar field range and rejected otherwise (in the case of uniform strings; serialization of scalars needs to reduce to within the range first).

Just curious if you were able to get ed25519 to use BLAKE3...

Yeah, though there are options on how to implement it. The simplest approach is to take the RFC and swap SHA-2 with Blake3 in hash mode. Another alternative I considered was to use it in keyed hash mode using the private key seed as the key there and its hash output as basis for the private key and signing half (I forget the exact details; it's been a few months).


Composer create homestead files

@citricsquid suggested the following to have composer setup homestead rather than using the example file.
php vendor/bin/homestead make --ip=<IP_ADDRESS>", "sed -i '' 's/homestead.app/pterodactyl.local/g' Homestead.yaml

The best way that I've found to do this is to define your own "setup" command within your composer configuration, then encourage people to run that command after cloning your project instead of composer install. For example, a command named setup might do the following:

Install all the composer dependencies
Copy the configuration file
Set up the Homestead environment
Generate the application key
??? Anything else you need

And you can create such a command in your composer.json like so:
"scripts": {
    "post-install-cmd": [
        "php artisan clear-compiled",
        "php artisan optimize"
    ],
    "pre-update-cmd": [
        "php artisan clear-compiled"
    ],
    "post-update-cmd": [
        "php artisan optimize"
    ],
    "setup": [
        "composer install",
        "php -r \"copy('.env.example', '.env');\"",
        "php vendor/bin/homestead make --ip=<IP_ADDRESS>",
        "sed -i '' 's/homestead.app/pterodactyl.local/g' Homestead.yaml",
        "php artisan key:generate"
    ]
}

Then someone might do the following:
$ git clone https://github.com/Pterodactyl/Panel.git
$ composer setup

That's the best way I've uncovered so far, I'll update this issue if I ever find something better. Unfortunately it's not possible to do this automatically on composer install because there's no "on first composer install", but there's no danger to someone running composer install and then composer setup after.

Thanks for the note about .app, I was unaware of that. I like the idea of using a setup command as well. It might be possible to use post-create-project-cmd, although I haven't looked into it closely.

unfortunately the post-create-project-cmd event is only fired when you first create a project via composer (composer create-project --prefer-dist laravel/laravel blog), so it's not a viable option.

Yeah, it seems like having a custom setup command is the way to go. I could potentially include a separate bash script that the install calls each time that checks if it has ready been run, but that is excessive. Plus, if it is its own command they can always redeploy if for some reason it is needed.

Discovered that the command included for changing the hostname in the Homestead configuration won't work cross platform. Essentially sed does not have a cross platform way to rename in place without a backup file (Stackoverflow discussion of the problem here). The cross platform solution is to enable the backup file and then remove it after, so the command looks like:
sed -i.bak 's/homestead.app/pterodactyl.local/g' Homestead.yaml && rm Homestead.yaml.bak

Still need to provide documentation for editing their hosts file as well, I could echo it in probably, but that would require sudo privileges.

I probably should have a setup-dev and setup command, but I'll come back to that when we figure out how we will be shipping the final panel.


503 Service Unavailable Error.

https://metasmoke.erwaysoftware.com/ gives me a "503 Service Unavailable" error page. Would be nice if it worked :-)

OK, it seems to work again now, but it seems this is a recurring issue, so would be good for it to be fixed.

Wait a sec, is it not working for you right now? What's dig metasmoke.erwaysoftware.com show?

Oh. Yeah, recurring symptom, this time it was a different issue. Still digging.


Issues with post install script

I am currently trying to install nodegit v0.15.1 via npm but it is failing to complete the post-install script with the following error:

Error: The specified module could not be found.
\?\C:\Code...\temp\electron-projects\node_modules\nodegit\build\Release\nodegit.node
at Error (native)
at Object.Module._extensions..node (module.js:568:18)
at Module.load (module.js:456:32)
at tryModuleLoad (module.js:415:12)
at Function.Module._load (module.js:407:3)
at Module.require (module.js:466:17)
at require (internal/module.js:20:19)
at Object. (C:\Code..\electron-projects\node_modules\nodegit\dist\nodegit.js:11:12)
at Module._compile (module.js:541:32)
at Object.Module._extensions..js (module.js:550:10)
at ChildProcess.exithandler (child_process.js:202:12)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:191:7)
at maybeClose (internal/child_process.js:850:16)
at Socket. (internal/child_process.js:323:11)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at Pipe._handle.close [as _onclose] (net.js:492:12)
killed: false,
code: 1,
signal: null,
cmd: 'node C:\Code\...\temp\electron-projects\node_modules\nodegit\dist\nodegit.js' }

Tracking up a few lines though and this file is reported to have downloaded ok:

[nodegit] Running install script
[nodegit] Success: "C:\Code...\temp\electron-projects\node_modules\nodegit\build\Release\nodegit.node" is installed via remote
[nodegit] Completed installation successfully.

I have confirmed on that it is indeed on the disk. Therefore I am baffled as toi why it is erroring out as it is.
Incase it is important I am running the following node and npm versions:
Node: v6.1.0
NPM: v3.8.6
Please let me know if any more information is needed and thanks in advance.

This is fixed as of 0.16.0. It's because it's compiled for electron, not node, so when we try to load it up in node it dies. The check is bypassed when being built from electron in newer versions.

Hi @maxkorp. I did as you suggested and it did solve the issue I outlined so thank you for the help with that. However, I am now encountering a new problem. When it is downloading the pre-built electron it is attempting to download Electron v0.37.8 that, I assume from the message I get, is not available (it is very old version of Electron now so I understand this being the case). Error in console I get:

[nodegit] Running pre-install script
[nodegit] Running install script
node-pre-gyp
ERR! Tried to download: https://nodegit.s3.amazonaws.com/nodegit/nodegit/nodegit-v0.16.0-electron-v0.37-win32-x64.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for<EMAIL_ADDRESS>and<EMAIL_ADDRESS>(electron-v0.37 ABI) (falling back to source compile with node-gyp)
gyp

However, in all my package.json files I am referencing the very latest version of Electron, v1.4.0. Indeed if I go back to nodegit 0.15.1 it then downloads Electron v1.4.0 without a problem, but not when I switch back to 0.16.0.
It seems that something is locking it to Electron v0.37.8 when I use nodegit v0.16.0. I can't find anything in my code away from nodegit that is locking it down though.


"kid" parameter in header does not have to be specified if only one key exists

According to the OpenID Connect spec, the kid parameter must be present if the JWK Set document has more than one key. It is silent on whether the parameter needs to be present if only one key exists.
The auth server we are using does not sent the kid parameter, which causes oidc-client to always reject its tokens without this patch.

Wasn't a PR for this done just yesterday: https://github.com/IdentityModel/oidc-client-js/pull/102

Hah, did not notice that. Nevermind!

And apologies for not checking the existing issues/PRs thoroughly

no prob -- i was just confused and checking. thx


User should be able to raise appeal request on behalf of other user.

Summary
As a user, they want to raise an appeal for another user in Guardian. This could help the manager/supervisor to raise all access-request for a user and sometimes this access request is for integration account to other systems like metabase, tableau etc.
Proposed solution
Allow the appeal-request flow to get another user's email as input. Will trigger the approval flow for giving another user and send appeal notification to both users(in case it's a non-human account).

@rahmatrhd @ravisuhag currently guardian doesn't allow the creation of appeal for another user and this is checked here.
IMO, this should be part of policy where an org can define they should/shouldn't create an appeal on behalf of another user. The sample policy will look like this:

id: allow_create_appeal_for_other_user
steps:

name: create_appeal_for_other_user_check
description: appeal will be rejected if its creating for other individual
strategy: auto
approve_if: $appeal.account_type == "user" && $appeal.created_by == $appeal.account_id

Guardian will check whether it's creating an appeal for another individual based on created_by == account_id and handle the communication for both users.
@rahmatrhd @ravisuhag wdyt?

@singhvikash11 Can you document how it will change the contract of appeal? Are you proposing that create_appeal_for_other_user_check this step will only be needed when admins want to disable the appeal creation on behalf of other users?

@ravisuhag yes, As an admin they should be able to add this check as part of the approval flow instead of its default check-in guardian appeal service.

id: sample_policy
steps:

name: bigquery_resource_check
description: appeal will be rejected if its for other than bigquery
strategy: auto
approve_if: $appeal.resource.provider_type == "bigquery"

name: create_appeal_for_other_user_check
description: description: appeal will be rejected if its creating for other individual
strategy: auto
approve_if: $appeal.account_type == "user" && $appeal.created_by == $appeal.account_id

name: manager_approval
description: approval from manager
strategy: manual
approvers:
$appeal.creator.manager_email

Instead, we can just have an enable/disable flag in the policy to allow users to create an appeal for another user to make it more straightforward.
Also other things to note:

user details in creator field right now is based on the created_by field. Need to make it based on account_id if the appeal is created by a different user
notifications target

  id: my-policy
  version: 1
  steps: []
+ appeal:
+   allow_cross_individual_user: true

we can think of a better name

@rahmatrhd IMO, user details in the creator field should be based on the created_by field. Creator owns the lifecycle of this appeal. We only need to notify another user about the appeal creation, appeal approval, or appeal revoke.
In this case, the changes are only on the notification side. The notification module needs to check whether an appeal is for another individual and if yes then It'll send a notification to that account id(email field).

Yes, Creator i think should always be the person who created appeal and created_by id should be of appeal creator. We can think of a different of capturing on whose behalf the appeal is being raised.

@rahmatrhd @ravisuhag This one is finalized then, we can start the dev on this. The approach is:

Create a config in policy to enable/disable cross-user appeal creation.
If the cross-user appeal is enabled and the creator and appeal's account_id are different then Guardian will allow creation of appeal just like service account.
In case of cross-user appeal, Guardian will send an additional notification to appeal's account_id.
These other individuals should be able to see these requests and be able to renew the appeals.

should the account_id user be allowed to cancel the appeal as well?
need to extend duplicate appeal logic as well. example: if user A created an appeal for user B, and then user B also created a new appeal to the same resource himself, user B should receive a duplicate error
the policy documentation also needs to be made clear that $appeal.creator might not always be the person who accesses the resource

Update:
Creator will own the entire appeal lifecycle and appeal will not show in account_id's appeals.
Later Guardian will provide a context of resources which user have access to provide information to user who is getting access. This will also provide history of past access and revocation to the user on a given resource.


documentation fixes

added -U nphs_lps_waters option for prepare_receptor to prevent Zn removal
note to explain situation when no TZ atoms are added

Thank you @rczerminski-valo
Looks like prepare_receptor removes chains if all residues in that chain are non standard. In example/docking_with_zinc_metalloproteins/data/proteinH.pdb the zinc is in the same chain as the protein residues of the protein, which are standard residues, so zinc was not removed. Now, if zinc is the only residue in its own chain, it is removed. I'll add this info and merge. Thanks again.


Flutter iOS app won't run on a simulator

What have I tried

Flutter clean
Delete PodFiles and podfile.lock
Delete and recreate ios folder
Change deployment version

Actual results:
This:
FlutterRunOutput.txt
The app works on Android so far, and I have successfully published it on the Play Store to test (if that helps at all)
Unfortunately I am not sure how to provide a reproducible issue since this is a pretty big app that I have been developing for android mostly and now just wanted to implement iOS as well. If there are any tips on how to reproduce the issue please let me know

Logs
Flutter run verbose:
FlutterRunVerbose.txt
Flutter Analyse:

Analyzing balkan_streak...

   info • The value of the local variable 'children' isn't used • lib/screens/progress_screen.dart:74:32 • unused_local_variable
   info • The operand can't be null, so the condition is always false • lib/screens/registration_screen.dart:324:54 • unnecessary_null_comparison
   info • Name non-constant identifiers using lowerCamelCase • lib/widgets/coupon_redeem.dart:39:19 • non_constant_identifier_names
   info • The operand can't be null, so the condition is always true • lib/widgets/coupon_redeem.dart:58:34 • unnecessary_null_comparison
   info • This class (or a class that this class inherits from) is marked as '@immutable', but one or more of its instance fields aren't final:
          CustomCheckbox.isChecked • lib/widgets/custom_checkbox.dart:4:7 • must_be_immutable
   info • This class (or a class that this class inherits from) is marked as '@immutable', but one or more of its instance fields aren't final:
          GameCardWidget.lockVisibleLeft, GameCardWidget.lockVisibleRight, GameCardWidget.colorLeft, GameCardWidget.colorRight •
          lib/widgets/game_card.dart:7:7 • must_be_immutable
   info • This class (or a class that this class inherits from) is marked as '@immutable', but one or more of its instance fields aren't final:
          SettingsCard.text • lib/widgets/settings_card.dart:3:7 • must_be_immutable

7 issues found. (ran in 5.9s)

Flutter Doctor -v:
[✓] Flutter (Channel stable, 2.8.1, on macOS 11.4 20F71 darwin-x64, locale en-ME)
    • Flutter version 2.8.1 at /Users/danijelpantovic/Developer/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 77d935af4d (5 weeks ago), 2021-12-16 08:37:33 -0800
    • Engine revision 890a5fca2e
    • Dart version 2.15.1

[✗] Android toolchain - develop for Android devices
    ✗ Unable to locate Android SDK.
      Install Android Studio from: https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK components.
      (or visit https://flutter.dev/docs/get-started/install/macos#android-setup for
      detailed instructions).
      If the Android SDK has been installed to a custom location, please use
      `flutter config --android-sdk` to update to that location.

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.2

[✗] Chrome - develop for the web (Cannot find Chrome executable at
    /Applications/Google Chrome.app/Contents/MacOS/Google Chrome)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.

[!] Android Studio (not installed)
    • Android Studio not found; download from
      https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/docs/get-started/install/macos#android-setup for
      detailed instructions).

[✓] VS Code (version 1.63.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.32.0

[✓] Connected device (1 available)
    • iPhone 8 Plus (mobile) • 5BE9C34C-E0BA-49C7-A726-D49938A7F2AC • ios •
      com.apple.CoreSimulator.SimRuntime.iOS-14-2 (simulator)

! Doctor found issues in 3 categories.

Podfile:
ENV['COCOAPODS_DISABLE_STATS'] = 'true'

project 'Runner', {
  'Debug' => :debug,
  'Profile' => :release,
  'Release' => :release,
}

def flutter_root
  generated_xcode_build_settings_path = File.expand_path(File.join('..', 'Flutter', 'Generated.xcconfig'), __FILE__)
  unless File.exist?(generated_xcode_build_settings_path)
    raise "#{generated_xcode_build_settings_path} must exist. If you're running pod install manually, make sure flutter pub get is executed first"
  end

  File.foreach(generated_xcode_build_settings_path) do |line|
    matches = line.match(/FLUTTER_ROOT\=(.*)/)
    return matches[1].strip if matches
  end
  raise "FLUTTER_ROOT not found in #{generated_xcode_build_settings_path}. Try deleting Generated.xcconfig, then run flutter pub get"
end

require File.expand_path(File.join('packages', 'flutter_tools', 'bin', 'podhelper'), flutter_root)

flutter_ios_podfile_setup
platform :ios, '12.0'
target 'Runner' do
  use_frameworks!
  use_modular_headers!
  pod 'Firebase/Core'
  pod 'Firebase/Firestore'
  pod 'Firebase/Analytics'
  flutter_install_all_ios_pods File.dirname(File.realpath(__FILE__))
end

post_install do |installer|
  installer.pods_project.targets.each do |target|
     flutter_additional_ios_build_settings(target)
   target.build_configurations.each do |config|
    config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '12.0'
   end
  end
 end

Pubscpec:
name: montenegro_streak_project
description: A new Flutter project.

# The following line prevents the package from being accidentally published to
# pub.dev using `pub publish`. This is preferred for private packages.
publish_to: 'none' # Remove this line if you wish to publish to pub.dev

# The following defines the version and build number for your application.
# A version number is three numbers separated by dots, like 1.2.43
# followed by an optional build number separated by a +.
# Both the version and the builder number may be overridden in flutter
# build by specifying --build-name and --build-number, respectively.
# In Android, build-name is used as versionName while build-number used as versionCode.
# Read more about Android versioning at https://developer.android.com/studio/publish/versioning
# In iOS, build-name is used as CFBundleShortVersionString while build-number used as CFBundleVersion.
# Read more about iOS versioning at
# https://developer.apple.com/library/archive/documentation/General/Reference/InfoPlistKeyReference/Articles/CoreFoundationKeys.html
version: 1.0.0+2

environment:
  sdk: '>=2.14.0 <3.0.0'

dependencies:
  flutter:
    sdk: flutter

  # The following adds the Cupertino Icons font to your application.
  # Use with the CupertinoIcons class for iOS style icons.
  cupertino_icons: ^1.0.2
  cloud_firestore: ^3.1.0
  firebase_core: ^1.2.0
  http: ^0.13.3
  firebase_database: ^8.1.0
  firebase_auth: ^3.2.0
  provider: ^5.0.0
  persistent_bottom_nav_bar: any
  cached_network_image: ^3.0.0
  material_dialogs: ^1.1.1
  lottie: ^1.2.1
  url_launcher: ^6.0.15
  cloud_functions: ^3.1.1
  firebase_messaging: ^11.1.0
  flutter_local_notifications: ^9.1.0
  email_validator: any
  flutter_pw_validator: ^1.3.0
  flutter_launcher_icons: ^0.9.2
  change_app_package_name: ^1.0.0
  auto_size_text: ^3.0.0
  marquee: ^2.2.0
  shared_preferences: ^2.0.9
  step_progress_indicator: ^1.0.1
  dotted_line: ^3.1.0
  screenshot: ^1.2.3
  connectivity_plus: ^2.2.0

dev_dependencies:
  flutter_test:
    sdk: flutter

flutter_icons:
  android: true
  ios: true
  image_path: "images/new_logo_balkan_streak.png"

# For information on the generic Dart part of this file, see the
# following page: https://dart.dev/tools/pub/pubspec

# The following section is specific to Flutter.
flutter:

  # The following line ensures that the Material Icons font is
  # included with your application, so that you can use the icons in
  # the material Icons class.
  uses-material-design: true

  # To add assets to your application, add an assets section, like this:
  assets:
    - images/

  # An image asset can refer to one or more resolution-specific "variants", see
  # https://flutter.dev/assets-and-images/#resolution-aware.

  # For details regarding adding assets from package dependencies, see
  # https://flutter.dev/assets-and-images/#from-packages

  # To add custom fonts to your application, add a fonts section here,
  # in this "flutter" section. Each entry in this list should have a
  # "family" key with the font family name, and a "fonts" key with a
  # list giving the asset and other descriptors for the font. For
  # example:
  fonts:
    - family: SofiaProMedium
      fonts:
        - asset: fonts/sofiapromedium.OTF
    - family: SofiaProLight
      fonts:
        - asset: fonts/sofiaprolight.OTF
    - family: SofiaProSemiBold
      fonts:
        - asset: fonts/sofiaprosemibold.OTF
    - family: SofiaProBold
      fonts:
        - asset: fonts/sofiaprobold.OTF
    - family: Trueno
      fonts:
        - asset: fonts/trueno.otf
  #
  # For details regarding fonts from package dependencies,
  # see https://flutter.dev/custom-fonts/#from-packages

@alt-DooM
From the log you provided, it seems to be failing due to:
`GeneratedPluginRegistrant.h:10:9: error: could not build module 'Flutter'
    #import <Flutter/Flutter.h

<unknown>:0: error: failed to emit precompiled header    `

Can you attach your ios/Runner.xcodeproj/project.pbxproj ?

@alt-DooM From the log you provided, it seems to be failing due to:
`GeneratedPluginRegistrant.h:10:9: error: could not build module 'Flutter'
    #import <Flutter/Flutter.h

<unknown>:0: error: failed to emit precompiled header    `

Can you attach your ios/Runner.xcodeproj/project.pbxproj ?

Here it is:
RunnerProject.txt

@alt-DooM
Do you see Run Script inside Targets in Xcode  or have Install builds only checked ?
Also, can you confirm if you by any chance have Anaconda or ruby or python installed ?

@alt-DooM
Do you see Run Script inside Targets in Xcode or have Install builds only checked ?
Also, can you confirm if you by any chance have Anaconda or ruby or python installed ?

This is what is in the Run Script:

I do have only python installed

@alt-DooM
Does it run properly through Xcode ? If not, can you provide the xcode error log too ?

@alt-DooM Does it run properly through Xcode ? If not, can you provide the xcode error log too ?

I get 38 build errors on it, I'm not sure where can I view the logs for the issues ?

@alt-DooM
Can you try to create a fresh app and run it on simulator and see if it still gives you same error or not ?

@alt-DooM
Can you try to create a fresh app and run it on simulator and see if it still gives you same error or not ?

Fresh app works without issues, but this one that I have been working on for a couple months now gives me a heap of errors for some reason

@alt-DooM
In you ios/podfile, try adding below inside post_install section and see if it helps:
    target.build_configurations.each do |config|
      config.build_settings['ONLY_ACTIVE_ARCH'] = 'NO'
    end

e.g:
post_install do |installer|
   installer.pods_project.targets.each do |target|
     flutter_additional_ios_build_settings(target)
    target.build_configurations.each do |config|
      config.build_settings['ONLY_ACTIVE_ARCH'] = 'NO'
    end

@alt-DooM In you ios/podfile, try adding below inside post_install section and see if it helps:
    target.build_configurations.each do |config|
      config.build_settings['ONLY_ACTIVE_ARCH'] = 'NO'
    end

e.g:
post_install do |installer|
   installer.pods_project.targets.each do |target|
     flutter_additional_ios_build_settings(target)
    target.build_configurations.each do |config|
      config.build_settings['ONLY_ACTIVE_ARCH'] = 'NO'
    end

If above doesn't work, try below:
config.build_settings['EXCLUDED_ARCHS[sdk=iphonesimulator*]'] = 'arm64 x86_64'

@alt-DooM In you ios/podfile, try adding below inside post_install section and see if it helps:
    target.build_configurations.each do |config|
      config.build_settings['ONLY_ACTIVE_ARCH'] = 'NO'
    end

e.g:
post_install do |installer|
   installer.pods_project.targets.each do |target|
     flutter_additional_ios_build_settings(target)
    target.build_configurations.each do |config|
      config.build_settings['ONLY_ACTIVE_ARCH'] = 'NO'
    end

First solution:
firstSolution.txt
Second solution:
secondSolution.txt

@alt-DooM In you ios/podfile, try adding below inside post_install section and see if it helps:
    target.build_configurations.each do |config|
      config.build_settings['ONLY_ACTIVE_ARCH'] = 'NO'
    end

e.g:
post_install do |installer|
   installer.pods_project.targets.each do |target|
     flutter_additional_ios_build_settings(target)
    target.build_configurations.each do |config|
      config.build_settings['ONLY_ACTIVE_ARCH'] = 'NO'
    end

If above doesn't work, try below:
config.build_settings['EXCLUDED_ARCHS[sdk=iphonesimulator*]'] = 'arm64 x86_64'

Tried this as well as the second solution but no luck
Is it worth it to create a new project and copy files over to it or would that be too much of a hassle ?

@alt-DooM
From the error screenshot you shared earlier, it seems to be pointing to objective C files. Is your project developed in Objective C ?

Is it worth it to create a new project and copy files over to it or would that be too much of a hassle ?

You may try that which will be generated in Swift.

@alt-DooM From the error screenshot you shared earlier, it seems to be pointing to objective C files. Is your project developed in Objective C ?

Is it worth it to create a new project and copy files over to it or would that be too much of a hassle ?

You may try that which will be generated in Swift.

Nope, everything is in Dart
Not sure what else to try honestly, are there any xcode settings I should be aware of ?

No @alt-DooM he meant ios language must be set to swift. And try recreating project

I've same problem with this plugin.
Build is ok, but when running we got this error:
Symbol not found: _OBJC_CLASS_$_Reachability
PS: The app is using OBJ-C, we need this because of some native SDK's.

No @alt-DooM he meant ios language must be set to swift. And try recreating project like this

Oh okay, so just to make sure we are on the same page, I can create a project and it will run fine, but when trying to run a project I have transferred from Windows to Mac I get these errors

@bantunes22
That sounds like a different issue, so please file a new issue using template.
@alt-DooM

And try recreating project like this

@bantunes22 That sounds like a different issue, so please file a new issue using template.
@alt-DooM

And try recreating project like this

Okay so just to make sure we are on the same page, you suggest I create a new project following the steps from the link you provided, then transfer all the needed files from my old project that we are currently trying to fix ?
Can that new project be the same as this one or will I run into publishing issues on Google Play or iOS Store if package names are different and such ?

If package names are different then it will make an issue. So, make sure your package names are the same, and make sure you have implemented your native functionalities if you have any. The code change won't make any effect AFAIK.

@bantunes22 That sounds like a different issue, so please file a new issue using template.
@alt-DooM

And try recreating project like this

The issue has been fixed by creating a new project on my Mac device then transferring all the code to it.
So I guess some paths are hardcoded into the project it seems since transferring it from Windows to Mac via File Transfer or Dropbox and stuff like that does not work

@alt-DooM I don't think so. Cause whenever you move files from mac to windows, You will be creating a separate folder with mac code. And the rest folder is original code that will be automatically migrated AFAIK(Correct me if I'm wrong).  So That might not be an issue too.

@yahu1031 Thanks for your input.
@alt-DooM Considering your last comment, I am going ahead and closing this as resolved. If you face similar behavior again, feel free to open a new issue with all required details.
Thanks.


Use Label() instead of specifying the repo name

This allows dependents to use another repo name
and allows to avoid specifying transitive dependencies with Bzlmod.

This looks good, but could you provide a concrete example of the benefit you mentioned in the description? I'd like to see how it works in practice.

The benefits are only visible with Bzlmod which has separate repo namespaces per module.
So when you don't set the repo_name for grpc to com_github_grpc_grpc, it currently fails to resolve these targets.


Add domain for University of Siedlce

University official website:  https://www.uws.edu.pl/
Official Website for Institute of Computer Science: https://ii.uws.edu.pl/

@Balltiless Thank you for your patience while waiting for a review and processing your request. Pull request merged.


Add a way to disable the new leaves on palm trees, banana trees, etc.

The new leaves on palm, banana, and other trees are just ugly to me. They simply don't fit in.
It would be nice to have a config option to just use normal leaves on all trees.

This is now doable with treepacks


Internationalize

Use one of the below node modules such that one can easily select a prefered language. Once selected, all appropriate texts should display with the preferred language.
The Hamburger menu should have a menu item labeled Language or  an appropriate icon
that gives the user the option to select at least one of the below languages in alphabetical order. If we can support more languages, please add it to this list.

Amharic
English
Oromigna
Tigrayan

https://github.com/rse/vue-i18next
https://github.com/panter/vue-i18next
https://github.com/bluelovers/vue-i18next2
https://kazupon.github.io/vue-i18n/

Here is a PR.
https://github.com/Ethiopia-COVID19/Covid19.ET/pull/27


User settings and Lora/Model Issue when loading

Your question
I thought SwarmUI would be great...but it's not.  Namely, i cant tell it where i want the generations to be placed(SAVED).  Also, i have THOUSANDS of loras/embeddings/models that have spaces in them, and they wont load and crashes.  I MEAN WOW...  Why do i have to rename thousands of files when ever other UI can use them just fine no matter how they are named.  PLUS, when it was loading the folders i placed to manually point to them, the program was pulling from and comparing my database to one from Civitai!!!  thats a serious issue when i didn't tell it to do this.  I DONT want my database compromised and it looks like that this UI was doing it.  WHY was it verifying my models/loras/embeddings?
Logs
No response
Other
I am rally hopeful that these issues will be addressed as this UI looks REALLY PROMISING! thanks for all your hard work!

Apologies, if i came across rude!  NOT MY INTENTIONS.
I have used ComfyUI with spaces in my embedding names/folders/etc.. and have never had an error.  It is throwing errors for loras and emb's.  Here is one example out of hundreds-thousands that it is throwing out.
[Warning] Error handling metadata database for model Q:\ComfyUI\ComfyUI_windows_portable\ComfyUI\models\loras/PONY\Concept Art Style SDXL_LoRA_Pony Diffusion V6 XL.safetensors: System.IO.IOException: Insufficient system resources exist to complete the requested service. : 'Q:\ComfyUI\ComfyUI_windows_portable\ComfyUI\models\loras\PONY\model_metadata.ldb'
at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func2 fn)    at LiteDB.LiteCollection1.Upsert(T entity)
at SwarmUI.Text2Image.T2IModelHandler.LoadMetadata(T2IModel model) in T:\Swarm_UI\SwarmUI\src\Text2Image\T2IModelHandler.cs:line 322
hundreds of these types for the emb's:
21:27:05.126 [Warning] Embedding model 'PONY TI/eznegativexl.safetensors' contains spaces in its name, which will cause it to not parse properly on the backend. Please rename the file or folder to remove spaces.
I am seeing no spaces, but i do have spaces in the folders, and i have MANY folders that some do have spaces.
I am running i9-14900ks / RTX-4090 / 128GB DDR5 RAM

That LoRA error is unrelated to spaces. I think your data drive is full?
The embedding errors are real errors - ComfyUI doesn't give an error message for it, it just silently fails to load the embedding :( that's why I added a warning in Swarm, so you wouldn't think you're using the embedding but actually it's doing nothing. For things like negative embeds it can be really hard to tell whether it's working or not since they're designed to make subtle changes, thus the need for a special warning message to let you know.
And yes the folder name PONY TI having a space means... there's a space.
For reference, in the ComfyUI input, your prompt box will have something like prompt words here words words embedding:PONY TI/eznegativexl.safetensors words words words, and Comfy splits that by spaces and compares each option for if it starts with embedding: and if so, looks for a file with the name after that part, so it will find embedding:PONY and fail to load an embedding named PONY, and then it will see the word TI/eznegativexl.safetensors and treat that as just prompt text, not an embedding at all.
If you just did eg embedding:eznegativexl.safetensors it will see the embedding: but the filename eznegativexl.safetensors will be invalid, as you don't have a file named that inside the embeddings folder, you have a file in a subfolder that it doesn't know to look for unless it's written explicitly as part of the name.

I use M2's for my drives 4-8TB ones, and they have enough space (at least 400GB-1TB left on most of them).  for the reading of emb's, when it does that in comfy, i found that just removing the subfolder it actually works as intended even though it's in a sub folder.  i just delete the PONY TI/ in front of the embedding.  it does give the error if i forget to remove it, but they work as intended if i just remove the subfolder reference!!  Odd but at least it works, so i just kept them in the subfolder in Comfy so they are easier to find for each different model usage.
For the reset metadata, (the warning reference you give) it isn't going to change anything on my drives right?  like delete loras/emb info etc, and or the images i have next to the (named exactly as the lora/emb) for reference.
ALSO, i always have a same-named Txt file next to it as i place important info, like triggers, prompts, and extra data the creator uses.  In comfy when i have a txt file next to it (same name as the item) it pulls it up in a secondary box that i could copy and paste the info i needed.  VERY convenient!!
I know you cant do that for triggers (multiple ones to pick from as you have stated this would be difficult) unless you could somehow make your own reference file (like .info or similar) that it could "auto place triggers" when it loads the lora/emb?
ORRR more simply, have it load an area in the info section that you already have for each item where as long as the txt file is named the exact same as the item then it will place it into a BOX (at the bottom maybe?) and then the user can just paste what they want from the .txt (.info) file.
This could be a solution to many issues or requests i've seen, and for my own item id love to see in your product.  this would give the user at least an option to manually place items where they want, like active prompts/triggers and the like.
2nd:  Is there a way to utilize the COMFY yaml file that points to all the locations for better setup?  It would be great once you select COMFY as a backend, it would PULL the yaml file automatically and fill in the locations, or copy them into your yaml I saw in data folder in Swarm.  Also i didn't see a location pointing to upscalers.  Might have missed it somewhere, but didn't see it in the list, but did see it in the yaml file.
Thanks again for your help.  So far i am liking what I'm seeing from your product.  I'm pretty good with Comfy, but this gives us a plug and play feel!

For the reset metadata, (the warning reference you give) it isn't going to change anything on my drives right? like delete loras/emb info etc, and or the images i have next to the (named exactly as the lora/emb) for reference.
That won't delete any of your files, that will delete and rebuild Swarm's model metadata database, the file it uses to load model lists quicker (ie without having to read every file again to see what's in it every time you launch swarm)
ORRR more simply, have it load an area in the info section that you already have for each item where as long as the txt file is named the exact same as the item then it will place it into a BOX (at the bottom maybe?) and then the user can just paste what they want from the .txt (.info) file.
This is called model metadata, Swarm supports it better than any other UI, just click "Edit Metadata" on the model and type in whatever you want in whatever part you want.
The automated trigger limitation is just that if you set an automatic trigger, you have to pick one. If you have multiple... well it's not automatic anymore is it? it's manual. Bit redundant to have an automatic system that just automatically... tells you to go do it manually.
2nd: Is there a way to utilize the COMFY yaml file that points to all the locations for better setup?
Model paths are configured in the UI under Server Configuration

Thanks for the reply.  I have thousands of Models! lol. it would take forever to copy it all over in the Edit Metadata,  couldn't it have the ability to just read it from the already made txt file i created for them all?
Sorry, might have not explained the Comfy yaml file.  I have multiple locations on separate drives, so i manually edited the yaml file for Comfy to manually point to these locations so Comfy can find them.  As it is, i have to re-enter each location.  This is not hard to do but wanted a simple solution; as i still use comfy for crazy templates and sometimes i re-edit the yaml file to point to a new location.  So if Swarm could just PULL those changes from the Comfy yaml, it would be more convenient is what i'm trying to convey.

Just for lora's i have over 3500!

The space in the embed path thing is another you say potato I say potahhhto issue.
Many people consider filename and filepath different things, and so the error being Embedding model 'PONY TI/eznegativexl.safetensors' contains spaces in its name leads to confusion.
Changing "name" to "name or path" (or something similar) will help understanding.


Should a disabled button trigger a Popover or tooltip

If it truly is disabled, then this would only occur on hover (which is not a11y and therefore bad). The only other option I can think of is have it fake disabled and focusable or put the popover somewhere else. Like on a label connected to the button.

Disabled buttons are not tabbable/focusable and therefore cannot trigger popover or tooltip.


[CORE]  Distinguish between native scan and vanilla spark scan in plan tree string

Description
Currently, the nodeName of navtive scan are the same as vanilla spark scan, making it impossible to distinguish them in planTreeString.
For example, for FileSourceScanExec:
update nodeName from "Scan **" to "NativeScan **"
update nodeNamePrefix from "NativeFile" to "File"

@zhztheplayer Do you think it is necessary to modify？

Not a big issue. We may align with other operator name like ScanTransformer


Having a open bracket breaks how link display is managed by jira plugin

as you can see here, I accidentally hit enter when only "[webapp" was written in the title having the jira integration display the link incorrectly.

Original Jira Ticket: https://mattermost.atlassian.net/browse/MM-27890

Closing issues due to inactivity. This issue can be re-opened with more interest from our community.


Task: Investigate risks around incorrect calculations and data quality

Describe the task
Do a holistic review of the end to end process of the data pipeline and try to see if there are any risks of calculation errors and understand the impact of various potential issues.
Acceptance Criteria

[ ] Documentation of data pipeline risks and impact
[ ] New issues capturing actions we want to take from this discovery

Additional context

TODO

https://github.com/usds/justice40-tool/issues/946#issuecomment-989155252 is somewhat related, if something goes wrong in an upstream data source. I think the relevant question there is, do we detect that case?

There are two questions here:

If an upstream data source is returning incorrect data, can we detect this?
If we have introduced a logic error that results in incorrect data, can we detect this?

I'm not sure how much we can do about the first case. https://github.com/usds/justice40-tool/issues/946#issuecomment-989155252 is very related here, because then if we do see something wrong, at least we've saved a previous version we can roll back to. I think anything else we do would have to be dataset specific sanity checks as part of the ETL job that pulls it down, which isn't quite the holistic solution we'd hope for but might be the best we can do.
For the second case, there are issues for more test coverage in general: https://github.com/usds/justice40-tool/issues?q=is%3Aopen+is%3Aissue+label%3Atesting-backend and https://github.com/usds/justice40-tool/issues/931 is about running the full end to end score generation.
So this is related to the mitigations in https://github.com/usds/justice40-tool/issues/946#issuecomment-989155252. If we have a setup to leave previous versions of the data around, we could write a github action to run on a specific version of the data that will not change, and compare the results to the last run to make sure they're identical. This will at least tell us if we've introduced a bug that will change the output. The github action can be granular enough to tell us which step has a delta, so that we know the true scope of a change to the pipeline.
The question there is: Do we ever expect to change the pipeline in a way where the point isn't to change the output. If that's not the case, this might have less value. Doing this with a smaller dataset that we could commit into version control could still help though, because we can see a delta of what's changed.

Requires some deeper knowledge of our data and calculations.


videos automatically fastforwards when played on a playlist

if i play a playlist the videos will autoskip after 2-3 seconds and it will play the next video on the playlist this keeps on happening

Yeah same issue here.


fix(material/select): disabled state out of sync when swapping form group with a disabled one

Fixes the disabled state of a mat-select falling out of sync with its form control if the control's group is swapped out with one that is disabled on init.
Fixes #17860.

Is there a reason this little PR hasn't been merged yet?


Test report randomly appears under different workflow

With v8 of the action, we've noticed that sometimes the test reports will appear not under the workflow they were created from, but another workflow that runs in parallel on the same trigger.
The two workflows:

build-and-test: This runs test-reporting for .NET builds. Two jobs run two builds (matrix builds, under Ubuntu and Windows) in parallel. The workflow utilizes an external reusable workflow, and where in the end test-reporting is called is here.
validate-pull-request: This doesn't use test-reporting.

What we see:

This build-and-test run is correct, the test reports appear there. This is the corresponding validate-pull-request run and this is also correct, no test reports there.
This build-and-test run happened 5 minutes before the above one, without changing to the workflows. As you can see, the test reports are not there. They are, instead, under this is the corresponding validate-pull-request run.

This seems to be a bug to me. Note that the two workflows are unrelated apart from having an overlap of triggers.
Before we had the validate-pull-request workflow, we noticed test reports randomly missing, which is perhaps the same issue.

So, this actually relates to a long-standing github issue, where the requesting run id isn't linked to the check suite id.
It's discussed here in detail:
https://github.com/dorny/test-reporter/issues/67

I see, thanks for explaining. How about then, as a stopgap workaround, do what's mentioned here? It would even be enough to include the "Check run HTML" link in the run summary.

https://github.com/phoenix-actions/test-reporting/pull/21 is about something similar.

Thank you!