Creating two instances of Oxygen

I have a project that requires two distinct instances for Oxygen - one for public API and another for admin access, which must be served only on a localhost. I can't combine them because public API is by default a plain HTTP, which is not secured for serving the admin panel. Therefore, I need a second access point.
The problem is that Oxygen has a single internal global state, but I need two separate states for different ports. To illustrate the issue, let's consider a ModuleA that is defined as:
module ModuleA

using HTTP: Request
using Oxygen

function __init__()

    @get "/inside" function(req::Request)
        return "Hello from Inside"
    end

end

end

The Oxygen in the runtime has only a single state, as shown in this example:
using ModuleA

using HTTP: Request
using Oxygen

const ROUTER = Oxygen.Core.getrouter()

@get "/outside" function(req::Request)
    return "Hello from Outside"
end

@show ROUTER(Request("GET", "/outside"))
@show ROUTER(Request("GET", "/inside"))

where ROUTER now has both inside and outside routes attached to it.
I cannot run those two routes on separate processes as I need to access the same global state, which requires a lock for certain operations. The actor model is not suitable here as the concurrency is simple, but communication is complex.
I can think of two ways to tackle the issue. One option is to only serve routes with a matching tag. However, I haven't explored this approach in depth yet. The downside of this option is that all the routes would need to be tagged, which would make the code less legible.
Another approach I would much prefer is that the state could be initialised locally with a macro along with stateful operations. From the API standpoint, that could work as follows:
using Oxygen
using HTTP

const ROUTER = HTTP.Router()
@oxidise ROUTER

where the @oxidise macro defines a custom @put, @post, @get, serve etc. methods for the ROUTER along with the additional hidden state. This could work without issues as the methods would overshadow those exported by Oxygen. This would also address the issue #115 properly.
I spent a few hours attempting to refactor Oxygen by limiting the usage of global state only to exported methods. The idea was to inject the global state into subsequent methods, making it easier to supersede them with a macro. Unfortunately, I got overwhelmed by the amount of work needed here, as the codebase contains a significant amount of global variables. The major challenge I faced was the autodoc.jl custommiddleware variable, which is referenced by the hasmiddleware method. I suspect that there are more couplings with the autodoc.jl library like this one.
I also noticed that you have include(“util.jl”) in multiple files. This is a little bad as the compiler needs to compile every of those methods separately. Alternatively, you can put include(“util.jl”) in the top module Oxygen.jl and then refer to it internally with using ..Util.
Anyway, this is a cool project, and many find it useful as I do. I just wish it would have been organised in a layered way where globals would have been put at the outmost layer, and thus, the state initialisation could be simulated with a macro where needed.

Hi @JanisErdmanis,
Thanks for the detailed write up, you bring up a lot of good points. Since needing access to the same global state is a hard requirement, it does limit the options a bit.
For now I'd recommend going with option 1, where you use the router function to split up your app into two main sections (internal and public). This will allow your app to access the same global memory while properly segmenting the api to protect it form unauthenticated users.
Routers help keep your code DRY, by having a single place to configure whole sections of your api. The router is just a function, so it can be exported and shared across multiple files.
It's worth mentioning that you can't nest routers within routers at the moment. If you need more fine grained control I'd recommend just defining another router function with a mostly similar prefix
using Oxygen
using HTTP

function AdminMidleware(handler::Function)
    function(req::HTTP.Request)
        # do some authentication here
        handler(req)
    end
end

# all "admin" routes will be prefixed with "/internal" and have the AdminMidleware applied to each request
admin = router("/internal", tags=["internal"], middleware=[AdminMidleware])

"""
    Define the internal routes
"""

@get admin("/") function()
    text("welcome to the online version of the app")
end

@get admin("/custom") function()
    text("Here's a custom admin endpoint")
end

"""
    Define the public facing routes
"""

@get "/public" function()
    text("welcom to the public version of the app")
end

serve()

You're absolutely right, this package leans super heavily on the internal states for the simple api. The global state has to be stored somewhere, but I like the idea of having a macro so something to allow for multiple instances of Oxygen. Granted, that's going to take a lot of work to rework things to be more composable.
Moving forward:

Enable router nesting
Refactor Oxygen to be more layered
Enable mulitple Instances

I have some other ideas in the road map that I believe will benefit from these changes

The AdminMiddleware looks interesting. I could block requests for the public API going in there. However, in my situation, the admin API also serves the webpage and thus, putting internal GET  in front of every request is visible in the address bar. It could be solved by modifying request targets in the middleware, which is something to consider.
For now, I have found that:
include(Base.find_package("Oxygen"))
using .Oxygen

works reasonably well. However, I don’t want to keep Oxygen dependencies in my Project.toml and for the past few hours, I have been trying to find a workaround to switch the project load to the Oxygen dependencies and switch back, but I haven’t figured out that yet.
I suggest that we tackle the task of moving out global states incrementally. My proposal is to first move all global states to the Oxygen.jl scope and add corresponding methods that interact with that state there. We can start by moving the ROUTER outside with corresponding macro definitions. Internally, we should parameterise as many methods as possible with the router.
Then we could move on to the autodoc.jl module, focusing on mountedfolders, taggedroutes, custommiddleware. Other variables look like defaults and could be later added as keyword arguments for the serve method. Perhaps repeat tasks and cronjobs as well as timers could already work without refactoring. Can you have a look at whether it could be so? This could significantly reduce the effort involved.
From metrics.jl, only a history object would need to be put in the global scope. Meanwhile, from streamutil.jl, parametrisation with HANDLER seems easy. So, in total, there seem to be the following variables that need to be taken into most outside layer:

ROUTER from core.jl
server from core.jl
mountedfolders from autodoc.jl
taggedroutes from autodoc.jl
custommiddleware from autodoc.jl
history from metrics.jl
HANDLER from streamutil.jl

Do you have any thoughts on if this list is exhaustive?
I could do the implementation of parameterisations for each component individually with corresponding pull requests. Would you be open to accepting such contributions that involve parameterizing internal methods and relocating those globals to the topmost layer where the exported convenience methods also would be added? I believe that with a bit of guidance, I could successfully execute this plan within a week or at most two.

Could you share a simple example of what the API would look like for a basic server?

The exported API would not change after the changes, and only the internal API would be affected. The @oxidise ROUTER would be entirely optional and only useful for situations like mine where one needs two different routers or would like a bit cleaner approach when defining the router in the module. The API would remain the same concerning exported methods. There are a few details on how to expose the internal state of the oxidised router, which we can discuss when we get to this point.
Implementing the @oxidise macro can be made relatively simple. Let's say that the exported convenience methods of Oxygen could be defined in the methods.jl file included at the topmost scope. The macro in the expanded form then would only do the following:
using Oxygen
using HTTP

router = HTTP.Router()

### Macro @oxidise router

ROUTER[] = router
server = ...
mountedfolders = ...
taggedroutes = ...
custommiddleware = ...
history = ...
HANDLER = ...

include(joinpath(dirname(Base.find_package("Oxygen")), "methods.jl"))

###

But I just realised it is a little more complicated than that. Here, the methods.jl would require explicit referencing with an Oxygen module in front for the right side. Thus, the macro would need to load the methods.jl into AST and add Oxygen to every method available within the topmost Oxygen scope. Nevertheless, this is totally doable.
An advantage of this approach is that it doesn't add complexity to the codebase. The only difference is that internal methods are parameterised, while the exported methods are defined in methods.jl, which binds them to the globals at the highest level.

So I was doing a little work on this idea, and kinda stumbled into a solution. While we were talking about the API and what it would look like going forward I wanted to take another stab at emulating the FastAPI interface from python.
I had some trouble at first, but eventually was able to dynamically generate a module and dump all of the exported symbols from core.jl into the newly created module. Julia really tries hard to prevent people from doing this, but when done it ends up perfectly sandboxing the package.
Down below I have an example where I create two completely separate instances with their own unique globals, routers and functions. It uses the exact same default API that oxygen uses and integrating it was a breeze.
Here's the file I added to make this possible:
https://github.com/OxygenFramework/Oxygen.jl/blob/feature/multi-instance/src/instances.jl
module MultiInstanceDemo

include("../src/Oxygen.jl")
using .Oxygen
using HTTP

# Setup the first app
app1 = oxidize()

app1.get("/") do
    "welcome to server #1"
end

app1.@get "/subtract/{a}/{b}" function(req, a::Int, b::Int)
    a - b
end

# Setup the second app
app2 = oxidize()

app2.get("/") do
    "welcome to server #2"
end

app2.@get "/add/{a}/{b}" function(req, a::Int, b::Int)
    a + b
end

# start both servers together
app1.serve(async=true)
app2.serve(port=8081)

# clean it up
app1.terminate()
app2.terminate()

end

And here's a video of me running it:
Screen Recording 2024-02-06 at 11.14.51 PM.webm

I will have a look at the implementation a bit later. I do, however, think that using app1 and app2 in front of methods and emulating OOP this way is not the way to go. Julia has an excellent namespace system, so I suggest a following API instead, which accomplish the same goal:
module MultiInstanceDemo

module APP1
# Setup the first app

using Oxygen
using HTTP
@oxidise

get("/") do
    "welcome to server #1"
end

@get "/subtract/{a}/{b}" function(req, a::Int, b::Int)
    a - b
end

end

module APP2
# Setup the second app

using Oxygen
using HTTP
@oxidise

get("/") do
    "welcome to server #2"
end

@get "/add/{a}/{b}" function(req, a::Int, b::Int)
    a + b
end

end

# start both servers together
APP1.serve(async=true)
APP2.serve(port=8081)

# clean it up
APP1.terminate()
APP2.terminate()

end

I looked into implementation and I find to be the same as doing the following:
module app1
include(Base.find_package("Oxygen"))
import .Oxygen: get, @get # and all other symbols
end

module app2
include(Base.find_package("Oxygen"))
import .Oxygen: get, @get
end

app1.get("/") do
    "welcome to server #1"
end

app1.@get "/subtract/{a}/{b}" function(req, a::Int, b::Int)
    a - b
end

# Setup the second app
app2 = oxidize()

app2.get("/") do
    "welcome to server #2"
end

app2.@get "/add/{a}/{b}" function(req, a::Int, b::Int)
    a + b
end

# start both servers together
app1.serve(async=true)
app2.serve(port=8081)

# clean it up
app1.terminate()
app2.terminate()

This approach suffers from the need to include all Oxygen dependencies in the app project. Isn't the same issue present in your approach?

That's close, but not the same. In this approach, we're including all the files at runtime and reinitializing everything (new internal globals for each module).
I've since made some changes so that the generated modules look similar like this:
module app1
include("Oxygen.jl") # the absolute path to this file is used
using .Oxygen
end

module app2
include("Oxygen.jl") # the absolute path to this file is used
using .Oxygen
end

To your last point, I believe this is the case. Each generated module has to be initialized to prevent them from using the same internals.
I did find a much cleaner way to generate the modules like the example below, but this has the same issue we're encountering now where the same internal globals are getting used between instances.
module app2
using Oxygen
end

Regardless, I don't see this getting in the way of your planned refactor. If anything they'll complement each other, and allow people to define servers either statically or dynamically since the external API won't change. I'd also change the name of this function so that it has a different one than the one you're planning to avoid confusion.
I'm excited about this approach because it allows people to dynamically create and stop instances (in the same parent module). I had some other ideas around performance and introducing the ability to easily spin up and manage multiple instances of oxygen, which now became much more plausible.
Being able to encapsulate them dynamically opens up a bunch of new possibilities.

Ok. I will start with parametrising the internal code with the ROUTER  and bringing it out at the topmost scope. This will likely determine if I will be able to accomplish the goal.
In retrospect, I have already thought that all globals could be systematically injected as the first argument in the following approach:
function some_internal_function(ctx::@NamedTupple{router::Router, server::Server}, arg1, arg2)
    # Some internal function
end

which has the benefit of not polluting internal interfaces and also allows to do this refactor incrementally.

I made good progress yesterday and managed to put the router at the top scope and run all tests. I forked and added a new branch, feature/oxidise, available here. I had only an option to push the commit to master. Perhaps you could create a branch where I could push commits.
I had a lot of warnings about overwritten methods during tests. Thus, I fixed those by changing the way the module is imported to a more orthodox approach. However, in the process, I broke templatingtests.jl. I noticed you were using Requires; I thought it got deprecated in favour of the new Julia extension system. Perhaps you can comment on those things for context.
I prefer to use capital case for global variables, much like you have used for ROUTER and HANDLER; thus, I plan to capitalise all other global variables as well if you have no objections here. I also did not extend Core functions but referred to them explicitly. This has the benefit that it is actually very easy to make the oxidise macro, which in the expanded form now looks like could be:
using Oxygen: Core

const ROUTER =  ...
const server = ...
const mountedfolders = ...
const taggedroutes = ...
const custommiddleware = ...
const history = ...
const HANDLER = ...

include(joinpath(dirname(Base.find_package("Oxygen")), "methods.jl"))

I have finished refactoring and implemented the @oxidise method. It works as follows:
module Example

using Oxygen; @oxidise

@get "/inside" function(req::Request)
    return "Hello from Inside"
end

end

and allows the use of the module in the following manner:
using Example
Example.serve(port=3424)

The @oxidise macro is fully optional, and after the refactor, all tests pass. I have deprecated enabledocs, disabledocs and isdocsenabled, which now show deprecation warnings pointing to the docs keyword in the serve function.
During the refactoring, I have improved tests with the following changes:

Fixed warnings with using Oxygen where previously a local import was used;
Hiding the errors printed from a server by introducing the show_errors keyword argument in serve, which passes it down to DefaultSerializer;
Hiding errors for internalrequest with @suppress macro at the call site when errors are expected;
I changed the port used in the tests from 8080 to 7070 because it is already being used frequently by applications on desktop systems.

Apart from refactoring globals into a context object, I have fixed the duplicate include statements for Util. I also added 'HTTP.Request` to the export list for Oxygen.
The history object at the global scope could, in a practical sense, be created within the serve method at runtime.  I, however, did not made such a change as it is also passed to internalrequest function. I would suggest that internalrequest exported to the user by default work with an empty history object, and only if it is explicitly passed to internalrequest would it be used instead. Would you find that change appropriate?
The remaining globals are those that are used at runtime. I have sketched that these could be created within a serve method with the following struct:
struct Runtime
     run::Ref{Bool}
     jobs::Set
     timers::Vector{Timer}
     history::CircularDeque{HTTPTransaction}}(CircularDeque{HTTPTransaction}(1_000_000)
     streamhandler::Union{Nothing, StreamUtil.Handler}
end

which is then passed downstream. It could be returned with the serve method for async=true along the HTTP.Server in the following struct:
struct Service
    context::Context
    runtime::Runtime
    server::HTTP.Server
end

To the service object, we could attach terminate,  stopcronjobs, startcronjobs, clearcronjobs, resetcronstate, stoptasks, starttasks so those would not need to be defined in methods.jl.

Thanks for all your hard work, I've been checking out the individual commits you've been pumping out all weekend.

To your question around internalrequest() function, I think what you described makes sense internally. As long as the publicly exported interface doesn't allow people to inject or modify the history object.  The metrics keyword should be an all-or-nothing kind of flag that determines if we maintain a history or not.

Breaking down the Context object into smaller structs is a great idea. Creating a monolith would make it more difficult to modify and grow later down the line. Techinically, all the values are used at runtime, but I see your point. Here's another structure we could think about.
struct Service end
struct Documenation end
struct CronManagement end
struct RepeatTasks end

struct Context
     service::Service
     docs::Documenation
     cron::CronManagement
     tasks::RepeatTasks
end

You don't have to follow these naming conventions exactly, but It'd be helpful to have high cohesion between related variables & functionality. When having big context objects it's important that we know exactly where the right variables are located so that there's no confusion ( it also makes it very readable )

I've also enabled running the ci/cd pipeline for this pr, so you should be able to adjust the code to improve the coverage. It's currently failing for julia version 1.6.6 but this compat entry was from when I first started the project. Feel free to adjust the target version to 1.6 to let it automatically select the latest TLS version.

I updated the pull request and tested that the code works with Julia 1.6. Code coverage has decreased only slightly, and it is only because I had to modify code that had not been covered before, where we can look into streamutil.jl

To your question around internalrequest() function, I think what you described makes sense internally. As long as the publicly exported interface doesn't allow people to inject or modify the history object. The metrics keyword should be an all-or-nothing kind of flag that determines if we maintain a history or not.

You may have misunderstood me. What I am proposing is the following change:
internalrequest(req::Oxygen.Request; middleware::Vector=[], metrics::Bool=true, serialize::Bool=true, catch_errors=true) = Oxygen.Core.internalrequest(CONTEXT[], HISTORY[], req; middleware, metrics, serialize, catch_errors)

To a modified public API as follows:
function internalrequest(req::Oxygen.Request; middleware::Vector=[], metrics::Bool=false, serialize::Bool=true, catch_errors=true, history::Union{Nothing, History}=nothing)

    if metrics && isnothing(history)

        error("History object needs to be specified")

    else

        history = History(0)

    end

    return Oxygen.Core.internalrequest(CONTEXT[], history, req; middleware, metrics, serialize, catch_errors)

end

Would you find such a change appropriate?
This is necessary as I would like to instantiate the history struct in the serve method at runtime. This is desirable because it reduces the number of global variables at the global scope while separating compile time constants (those listed in the Context) and those populated during a serve method during runtime.
The distinction between Context and Runtime is important because it lets one quickly glance through the codebase to see whether a particular function is run at runtime or compile time. When a function accepts a runtime object, one treats the context object as immutable, leading to safer and more predictable code. Explicit immutability, where possible, is a great thing.
At the moment, I don’t see a Context object as defined to be excessively large for outweighing the benefit of accessing a variable directly as ctx. docspath vs ctx.docs.docspath. It is pretty clear from naming of the variables what are their purpose in the codebase. Additional documentation can be provided for more clarity. When browsing the codebase, the developer can find all variables from ctx by simply searching in the editor. KISS.

I've updated your branch with a couple of commits. Currently, I have some work in a local branch were I've migrated the cron module to use the new Context struct. The default module is working and tests are passing - but the @cron macro isn't defined when trying to use the new @oxidise macro.
I have this code in a separate local branch since these changes are breaking the new macro. Feel free to try migrating the cron stuff as well, or let me know if you want me to push directly to your branch.

Currently, I have some work in a local branch were I've migrated the cron module to use the new Context struct

Putting jobs and run into Context object will require you to add startcronjobs and stopcronjobs at methods.jl. Nevertheless, I urge you to consider the following organisation of variables:
struct Context
    router::Router
    docs::DocsContext
    cron::CronContext
    tasks::TasksContext
end

struct Runtime
    cron::CronRuntime
    tasks::TasksRuntime
    history::History
    streamhandler::Union{Nothing, StreamUtil.Handler}
end

struct Service
    context::Context
    runtime::Runtime
    server::HTTP.Server
end

where the Context is initialised at compile time and Runtime is populated at runtime. The Service is returned from a serve method:
serve(ctx::Context, kwargs...) -> Service
serve(ctx::Context, kwargs...) -> Service

Then, one can attach to the objects the following methods:
startcronjobs(::Union{CronContext, Context, Service}) -> CronRuntime
stopcronjobs(::Union{CronRuntime, Runtime, Service})
clearcronjobs((::Union{CronContext, Context, Service})

starttasks(::Union{TasksContext, Context, Service}) -> TasksRuntime
stoptasks(::Union{TasksRuntime, Runtime, Service})

terminate(::Union{CronRuntime, TasksRuntime, Runtime, Service})
resetstate(::Union{Runtime, CronRuntime, TasksRuntime, Service})

This clearly separates concerns and promotes immutability. It does not mean that the API needs to change in a breaking way; both options could be supported and eventually see if the Julia community prefers using existing API coupled to a global state or uses these API functions with return value from serve.
I encountered a particular issue yesterday where the separation was important. When the serve method is called repeatedly from a compiled module, resetting the compile-time constants should not be possible because otherwise, you end up with a clean state, and there is no use to that. This was easy to fix with the following change:
if (@__MODULE__) == Oxygen
    CONTEXT[] = Oxygen.Core.Context()
end

If all variables were being put in Context instead, we would have to go through every context field and reset them individually, knowing which are created at runtime and which are at compile time.

Ok great, it looks like we're on the same page. I didn't reorganize the variables into those categories based on an older comment. I really like how it keeps things tight and organized. I won't be able to work on those changes till later tonight, so you don't have to worry about any rogue commits changing your progress.
Next Steps

Reorganize the context struct
Finish migrating the cron and tasks globals
Ensure API is working with and without the oxidise macro
Write a documentation block in the readme describing this new macro

Let me know if you can think of anything else missing from this list

Sounds good, let me know what's still pending (if you commit anything that's still a work in progress) and I can take a look at it later tonight to keep things moving.

I have found three additional things which would be great to do with this refactoring session:

move docspath and schemapath as keyword arguments to serve;
move cronjobs out by adding the job_definitions during register;
refactor the docs registration on a router created at runtime and couple it with a middleware.

The last two tasks are needed to make Context object constant, which is necessary to avoid potential bugs when the server is restarted within the same Julia session.

So, I have come to the conclusion of the refactoring. The last changes have moved docspath and schemapath out to serve as keyword arguments and now registers documentation to a separate router at runtime, which is coupled to the main router with a middleware. This change also prevents HTTP.register from modifying already present routes, and thus, when restarting a serve, no longer warnings by HTTP.serve! are shown, allowing to refactor out Suppressor. This is great because now the user gets a warning if it has duplicate routes registered, preventing confusion.
The second change I made was refactoring out cronjobs from the Context and making registration of cron jobs at the route registration time. The result is that the Context is now fully immutable during the runtime; thus, the service would work exactly the same when restarted. Both of these changes required splitting the register function to add direct methods that work with the router, where I introduced parse_route and parse_func_params.
Yesterday, I finished the migration of cron and tasks globals, which are stored at the service object created during runtime. I also cleaned up the global scope. Now there are only two globals, CONTEXT and SERVICE. From the latter one, the history object is taken when it is available, and this is how internalrequest works. No tests were broken with this change. Also, I made it so that internal request by default has metrics=false and docs=false as now there are docs middleware instantiation involved, which in most instances can be skipped.
I also noticed a peculiar behaviour in the error handling in the middleware. When an error is thrown by the router, it is handled by the serialiser, but if it is thrown by a middleware, it is handled by middleware if metrics=true. Is this intended?
I have no more changes planned. I added a simple test for @oxidise macro. So, I see that the only task remaining is writing documentation for the macro. This can also mention the workflow of working with Revise, as I described in #112. A small announcement on discourse may also be suitable as the tool has become more versatile, and the Revise support is neat.
It would be cool if we could merge it on master, as incorporating merges with this refactor will be super painful. A release of a new version may be delayed for a little to see if there are any incurring bugs.
To conclude, I am very glad that this package had such detailed test cases. It was a pleasure to do this refactor and be able to find the bugs exactly when they occur!

Awesome, I'm looking forward to reviewing and merging these changes in.
Regarding this point:

I also noticed a peculiar behaviour in the error handling in the middleware. When an error is thrown by the router, it is handled by the serialiser, but if it is thrown by a middleware, it is handled by middleware if metrics=true. Is this intended?

Not really, at the moment any errors thrown inside a middleware function aren't caught on purpose. I think this is just a byproduct of the Metrics being at the "top" of the middleware chain and including some error handling. I'd like people to manually handle their errors for middleware functions since adding any default behavior can mess with certain flows.


Upgrade jackson-databind to 2.12.0

Fixes #:
Description of changes:
Upgrade jackson-databind to 2.12.0 according to CVE-2020-25649
Update THIRD-PARTY
Tests:
If new tests are added, how long do the new ones take to complete
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Build is failing because of dependency conflict. Can you check and resolve the conflict?

Build is failing because of dependency conflict. Can you check and resolve the conflict?

Build is failing because of dependency conflict. Can you check and resolve the conflict?

The dependency conflict in updateShas step was executed in PA dir. I think this is the conflicts between RCA and PA on jackson-databind.

Build is failing because of dependency conflict. Can you check and resolve the conflict?

The dependency conflict in updateShas step was executed in PA dir. I think this is the conflicts between RCA and PA on jackson-databind.


Change startup code to be monolithic main.c rather than clutter up the inc/src dirs.

Explored by pull request #14. Not a good idea in the end.


Update hooks-state.md

In the chapter API REFERENCE  React.Component setState()
https://reactjs.org/docs/react-component.html#setstate
I read :
If the next state depends on the current state, we recommend using the updater function form, instead:
this.setState((state) => {
return {quantity: state.quantity + 1};
});
Isn't that this case?

React will make sure that the state updates are flushed after each click, and we're intentionally don't want to overcomplicate this example, so it's good as-is. Thanks for asking!


ci: limit max parallel runs for TPU

What does this PR do?
quite often se saturation on resources so lets limit them to run at most two at the same time

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
[x] Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

[x] Is this pull request ready for review? (if not, please submit in draft mode)
[x] Check that all items from Before submitting are resolved
[x] Make sure the title is self-explanatory and the description concisely explains the PR
[x] Add labels and milestones (and optionally projects) to the PR so it can be classified

:books: Documentation preview :books:: https://pytorch-lightning--18707.org.readthedocs.build/en/18707/

cc @carmocca @borda @JackCaoG @Liyang90 @gkroiz

Let's suffer further...


How do I find the sampling rate for the extraction of esp32, where on menuconfig? (AEGHB-152)

What do you mean "sampling rate for the extraction of esp32", can you describe it in an other way?

What do you mean "sampling rate for the extraction of esp32", can you describe it in an other way?

I basically want to know what the sampling rate is, is it 100Hz, 10Hz? How do i find this out?

Therocally the sampling rate of ESP32 can reach millions per second.
The main bottleneck of sampling rate is sending rate, we used to sample CSI data at 3KHz.

Oh, i believe that i get you now, there was a misguide in what i said.

Therocally the sampling rate of ESP32 can reach millions per second. The main bottleneck of sampling rate is sending rate, we used to sample CSI data at 3KHz.

I mean, therocally the sampling rate of ESP32 is millions per second. How many CSI data you receive per second mainly depend on the sending rate and how many frames you lose when transmitting.

Hello, @olliec28
Closing this issue, feel free to reopen with more updates. Thanks.


Feat(eos_designs)!: Enhance inband management configuration options

Change Summary
Provide support for the following new keys:

inband_management_vrf
inband_management_description
inband_management_mtu

Related Issue(s)
Fixes #2336
Component(s) name
arista.avd.eos_designs
Proposed changes
How to test
New molecule tests:

inband_mgmt
inband_mgmt_defaults

Checklist
User Checklist

N/A

Repository Checklist

[x] My code has been rebased from devel before I start
[x] I have read the CONTRIBUTING document.
[x] My change requires a change to the documentation and documentation have been updated accordingly.
[x] I have updated molecule CI testing accordingly. (check the box if not applicable)

Are you planning to include inband_mgmt_vrf as part of this change?

Are you planning to include inband_mgmt_vrf as part of this change?

Hi @pvinci-arista yes, it is already included.
Check the molecule vars here for different examples. Please test it out with your use case:
ansible_collections/arista/avd/molecule/eos_designs_unit_tests/inventory/group_vars/INBAND_MGMT_TESTS.yml


Fixed code block language

@nataz77 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@nataz77 - thank yo for your contribution.
#sign-off


Cannot read property 'async' of undefined (jest testing, migrating to RN37)

This happens when running jest tests, after migrating from RN35 to RN37. On RN36 it works fine.
There goes my package.json
{
  "name": "Evosphere",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "start": "node node_modules/react-native/local-cli/cli.js start",
    "buildandroid": "react-native bundle --platform=android --entry-file=index.android.js --bundle-output='android/app/src/main/assets/index.android.bundle' --dev=false",
    "test": "jest --watch"
  },
  "dependencies": {
    "axios": "^0.15.*",
    "dpath": "^2.0.2",
    "evosphere-router": "0.1.9",
    "immutable": "^3.8.1",
    "intl": "^1.2.5",
    "lodash": "^4.16.4",
    "node-uuid": "^1.4.7",
    "normalizr": "^2.2.1",
    "react": "15.3.2",
    "react-native": "0.36.0",
    "react-native-drawer": "^2.3.0",
    "react-native-i18n": "^0.1.1",
    "react-native-image-picker": "^0.22.12",
    "react-native-router-flux": "^3.35.0",
    "react-native-tabs": "^1.0.9",
    "react-redux": "^4.4.5",
    "redux": "^3.6.0",
    "redux-immutable": "^3.0.8",
    "redux-logger": "^2.7.0",
    "redux-thunk": "^2.1.0",
    "string-template": "^1.0.0",
    "tcomb-form-native": "^0.6.1"
  },
  "devDependencies": {
    "babel-core": "^6.14.0",
    "babel-jest": "^16.0.0",
    "babel-plugin-transform-object-rest-spread": "^6.8.0",
    "babel-plugin-transform-react-jsx": "^6.8.0",
    "babel-plugin-transform-remove-console": "^6.8.0",
    "babel-polyfill": "^6.13.0",
    "babel-preset-latest": "6.14.0",
    "babel-preset-react-native": "^1.9.0",
    "babel-preset-react-native-stage-0": "^1.0.1",
    "babel-register": "^6.14.0",
    "enzyme": "^2.5.1",
    "faussaire": "^0.1.8",
    "faussaire-util": "^0.1.0",
    "jest": "^16.0.1",
    "jest-enzyme": "^2.0.0",
    "jest-react-native": "^16.0.0",
    "react-native-fs": "^2.0.1-rc.2",
    "react-native-mock": "^0.2.7",
    "react-test-renderer": "^15.3.2",
    "updtr": "^0.2.1"
  },
  "jest": {
    "scriptPreprocessor": "<rootDir>/node_modules/babel-jest",
    "moduleFileExtensions": [
      "js",
      "json",
      "es6"
    ],
    "preset": "jest-react-native"
  }
}

Closing this issue because it has been inactive for a while. If you think it should still be opened let us know why.


StaveNote.wait is undefined.. related to trackFromVoice

I'm not very well versed in this package but I believe there's a situation where wait is undefined whereas it should be zero.
This is causing an error in some cases when calling get TickDuration there is an error

In the NoteOnEvent function, there is a call to Object.assign with some default values, including wait: 0
...
// Set default fields
fields = Object.assign({
    channel: 1,
    startTick: null,
    velocity: 50,
    wait: 0
}, fields)
...

However, in trackFromVoice the value of wait is returned as undefined for most notes of noteType==='n', because in the first if clause wait is not set for the first loop...
...
if (tickable.noteType === 'n') {
    tickable.keys.forEach(function (key) {
    // build array of pitches
    pitches.push(_this.convertPitch(key));
    });
} else if (tickable.noteType === 'r') {
    // move on to the next tickable and use this rest as a `wait` property for the next event
    wait = _this.convertDuration(tickable);
    return;
}

track.addEvent(new NoteEvent({
    pitch: pitches,
    duration: _this.convertDuration(tickable),
    wait: wait
}));

// reset wait
wait = 0;
...

This results in the defaults not actually being applied because the key is already present and Object.assign doesn`t overwrite it.
Object.assign({wait: 0}, {wait: undefined})
// {wait: undefined}

The value of wait should either be set at the top, or omitted from the returned object, or maybe the addEvent should be defaulted to zero e.g:
track.addEvent(new NoteEvent({
    pitch: pitches,
    duration: _this.convertDuration(tickable),
    wait: wait || 0 // Set to 0 by default or
}));

OK I see this is related to the whole trackFromVoice discussion.. I'm not 100% sure if this covers all the cases, but this is an implementation which gets a bit closer to the target I believe
/**
 * Support for converting VexFlow voice into MidiWriterJS track
 * @return MidiWritier.Track object
 */
trackFromVoice(voice) {
  var track = new Track();
  var wait = [];
  var pitches = [];

  voice.tickables.forEach(tickable => {
    pitches = [];

    if (tickable.noteType === 'n') {
      tickable.keys.forEach(key => {
        // build array of pitches
        pitches.push(this.convertPitch(key));
      });
      track.addEvent(new NoteEvent({pitch: pitches, duration: this.convertDuration(tickable), wait: wait}));
      wait = [];

    } else if (tickable.noteType === 'r') {
      // move on to the next tickable and use this rest as a `wait` property for the next event
      wait.push(this.convertDuration(tickable));
      return;
    }
  });

  // Rests at the end of the track, add a ghost note..
  if(wait.length > 0) {
    track.addEvent(new NoteEvent({pitch: '[C4]', duration: '0', wait, velocity: '0'}));
  }

  return track;
}

Hey @artsyca,
Thanks for documenting this!  I will more closely look at this as soon as I can.  PRs always welcome!
-Garrett


openpose

https://github.com/CMU-Perceptual-Computing-Lab/openpose

https://github.com/lgarithm/stdnn-ops/pull/3


REFPLTB-1931: Reimplementation of ethsw_thread_main function

Reason for change: To avoid memory allocation
Test procedure: TBD
Risks: None
Signed-off-by: Manigandan Gopalakrishnan<EMAIL_ADDRESS>

https://jenkins.cmf.code.rdkcentral.com/job/github-triggered/16776/ started

https://jenkins.code.rdkcentral.com/job/github-triggered/24144/ started

https://jenkins.code.rdkcentral.com/job/build-component/59894/ started.

Smoke tests for ['build-tur-bb'] added.
https://jenkins.cmf.code.rdkcentral.com/job/test-components-flow/130167/


Make the replaced results appear in the matches counter

Once again, a tight fit, but now the error section is completely free to do only what's supposed to be doing: Showing errors.

As this changes a translatable string, I pushed this one to 4.0.

Cherry-picked for 3.2.1.


added axiom for ring chromosomes

addresses #3507
Still need to do:
'ring chromosome' should not be in the "autosomal" branch.

@sabrinatoro is this ready to merge?


Cloud Events Support

Cloud Events v0.3 must be supported over the kafka channels inbound HTTP endpoint and outbound http request to subscriber.  The CloudEvents SDK should be evaluated for use https://github.com/cloudevents/sdk-go .  The Knative good_client https://github.com/knative/eventing/blob/master/pkg/kncloudevents/good_client.go should be evaluated for potential benefits on top of the standard.
Kafka CE bindings were not a part of the v0.3 spec, but were added for 1.0.  There is no hard requirement to begin using this approach now, but it might be convenient when already trying to meet the requirements of not altering the event as it passes through kafka.
Cloud event sent in to the channel must be have the same data and all other CE attributes as the cloud event sent to subscribers (except for tracing header manipulation)
See the following for further reference of conformance requirements addressed in this story (not all requirements are implemented in this story):
https://github.com/knative/eventing/blob/master/docs/spec/channel.md#input
https://github.com/knative/eventing/blob/master/docs/spec/channel.md#output
(5)

Addressed in PR #25


particle delta calculations are constraints/(w1+w2) but should be constraints*w1/(w1+w2)?

Hi warp team, I noticed while testing xpbd in a simple system of particles with contrasting radius and masses, that unexpected results relating to delta calculations were occurring.
I note that in the xpbd integrator, for particle-ground, particle-shape, and particle-particle kernels, the deltas are                 delta += (delta_f - delta_n) / denom    or of a similar nature.
If I make the following change to the kernels:  delta += (delta_f - delta_n) / denom * w1.
Then both gravity segregation and the brazil nut effect emerge. Without this, the high w particles have deltas that are too low and vice versa.
EDIT: I forgot to also mention that without this modification, when w<1.0 for any given system, there is a tendency for particles to explode outwards. This also appears fixed with this change.

Hi @yingDaWang-UNSW,
Thank you for reporting this issue! We have recently updated the XPBD integrator with these particle contact constraint modifications.

I believe this was fixed by @eric-heiden in 302b79e37725ad46eed4b49b0ba6d4d5160e0dda. Please reopen this issue if this is not the case. Thanks!


⚠ Bofrost.fr has degraded performance

In 069f79e, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1454 ms

Resolved: Bofrost.fr performance has improved in bdd993c after 6 minutes.


Trinary: Test with multiline strings.

Many trinary solutions use a regex: /^[012]+$/ to check for trinary
validity which can be 'fooled' by strings which contain newlines.
This adds a test for that case.
(It also fixes the example solution which had that bug.)

Brought it in under #380


fix(controller): Maybe bug with nil woc.wfSpec. Fixes #3121

Checklist:

[x] Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
[x] The title of the PR is (a) conventional, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
[x] I've signed the CLA.
[ ] I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
[ ] My builds are green. Try syncing with master if they are not.
[x] My organization is added to USERS.md.

Fixed #3160

closes #3158


A new simple trick to mitigate the codebook collapse

paper: https://arxiv.org/abs/2411.02038
repo: https://github.com/youngsheen/SimVQ

@BakerBunker have you tried the rotation trick from Fifty et al. yet?

Not yet, just discovered this paper from arxiv. Maybe it can be used with rotation trick and further improve the codebook utilization.

@BakerBunker the rotation trick brought my code utilization up to 100% nearly instantaneously

@BakerBunker your opinion counts a lot as a citizen scientist, give it a try and let me know what you see!

I've been banging my head on the ResidualVQ collapse and, it appears that changing the optimizer from AdamW to Adagrad did the trick.
RE: rotation trick, it would be nice to add an example. Happy to take a stab

@guillaumeguy it is already added by the first author himself here

Here are my numbers (the metric is the dot product between the original x and the reconstructed_x using 5 codes):
AdamW: Failed to converge in both cases (with low codebook utilization)

0.65 - No trick
0.65 - Rotation trick

AdaGrad

0.817 - No trick
0.821 - Rotation trick (slightly better)

@guillaumeguy thanks for sharing these results
what are you training on that only has 5 codes?  that seems unrealistic

I am creating semantic tokens. They use 3 codes in the paper FWIW.

@guillaumeguy oh I see.. they use different amounts of codes per depth. does the repo even support that?

let me read this TIGER paper in more detail tomorrow

@guillaumeguy have you heard of anyone successfully deploying something like TIGER?

Youtube said they had some success. Here is the paper.

@guillaumeguy ok, I'll get out the ability for different codebook sizes tomorrow. think you need to test it out in that fashion

@guillaumeguy ok, this should be available now here

Thanks @lucidrains ! Such a quick turnaround time.
Do you mind pointing me where in the paper you're seeing different codebook size? I'm reading 256 size x 3 codebooks:

@guillaumeguy ah my bad, i misread that section.. ok, so the repo could support it by default. where were you getting 5 codes from?

@guillaumeguy think i'll keep the ability for different amount of codes across layers though, may come in handy

Mostly my experiment. The level 3 the paper chose is arbitrary and only capture the high level features of the input embeddings. I suspect that production systems need better granularity (Youtube went the other way by increasing the cardinality but I like the idea of just increase the # of codebooks). Will report back

@guillaumeguy oh i understand now, you are using 5 codebooks, each of size 256 codes?

@guillaumeguy ok, then i think your results are legit

@BakerBunker went through the paper and changed my mind, let me play around with it tomorrow morning and see if i can cleanly integrate it

ok, i've gone through the paper in detail. not 100% on this paper, but i think it is worth trying, and will probably redo this in a separate file. if all of a sudden everyone is clamoring over it, i can slowly port over working features from all the research of year's past in VQ

unfortunately, not seeing it for this paper

@lucidrains Hi, I'm the first author of the paper SimVQ and I appreciate your interest in our group’s work!
In our paper, ALL main results for both image and audio tasks are conducted with the codebook frozen, which is sampled from a Gaussian distribution. You can find the core code here. We also conducted an ablation study with an unfrozen codebook in the paper (Trainable = Yes), where we observed slightly lower performance compared to the frozen variant. Notably, the unfrozen version can still achieve 100% codebook utilization.

Please feel free to reach out if you have any questions. I would be grateful if you could consider adding our method to your excellent vector-quantize-pytorch repository.

@youngsheen hi Yongxin, thanks for reaching out and for the interesting paper
i tried out your formulation but it performed poorly. i could have made a mistake though, so let me retry. i noticed you add noise to the targets (std = 0.01). where exactly is this noise applied? are you doing stochastic sampling?

@youngsheen do you see some overlap between your work and FSQ and LFQ? i see no mention of that in your paper, but i'm sure you thought about it

@lucidrains  Hi, thank you for trying out our formulation. Just to clarify, we only add noise to the targets in the toy examples to simulate small variations in the encoder outputs. This noise is not included in the main experiments. If you’d like, I’d be glad to help troubleshoot further—feel free to share your implementation details here, and we can take a look together.

@youngsheen ok thank you Yongxin. your offering to help is encouraging
i will circle back to this tomorrow and try it again

@youngsheen hi Yongxin, revisiting this this morning
do you see any issues in this section of the code? that should capture the core idea right?

@lucidrains Hi,
In vector_quantize_pytorch/sim_vq.py
Line 64: The standard deviation of the Gaussian distribution should be small, as is common in model initialization methods. For our experiments, we set std=e_dim**-0.5
Line 99-108: We did not apply normalization or the rotation trick. Instead, we directly used the STE in line 112.
Everything else looks good to me.

@youngsheen yes, i added the rotation trick because it was performing really poorly
let me try with the smaller init

@youngsheen no dice, the code utilization quickly collapses down to < 1 %
here is the new init code

it seems like you forgot to uncomment the STE code.

@youngsheen ah oops, ok yea it is uncommented
still the same performance
you can try this yourself by checking out the branch and running pip install . && python examples/autoencoder_sim_vq.py

@lucidrains Hi, I ran the code you provided and identified an issue with the commitment loss calculation. After correcting it, the codebook utilization quickly reaches nearly 100%.
In vector_quantize_pytorch/sim_vq.py, Line 73, commit_loss = 0.25 * ((x - quantized.detach()) ** 2).mean() + ((x.detach() - quantized) ** 2).mean()
Additionally, the current calculation of codebook utilization is incorrect. You should sum the activated codes over the training process or on the validation set for utilization tracking.

sum_code = [0] * num_codes

def train(model, train_loader, train_iterations=1000, alpha=1.0):

    for _ in (pbar := trange(train_iterations)):

        for ind in indices.unique():
            sum_code[ind] = 1

        pbar.set_description(
            + f"active %: {sum(sum_code) / num_codes * 100:.3f}"
        )

@youngsheen hey, indeed it is a bit better, sitting around 30% code utilization (old method). it would also seem the 0.25 is crucial, as it suffers if i make it the same weight
i'm not sure i agree with your deeming a code is active across the entire training run. surely you need to have some time window?

@youngsheen in any case, it does seem to be an improvement over base VQ. let me play around with it some more
thank you for this exchange

can't believe it takes 7 years from Oord's initial paper until now for these solutions to emerge. have to play the long game with research

@BakerBunker you can try it out by invoking from vector_quantize_pytorch import SimVQ
i'll document this in the readme once i get out the residual flavor later this week

Super cool work -- congrats @youngsheen! I bid quite strongly on this paper, but unfortunately, was not assigned to review it.
@lucidrains appreciate the tag and shoutouts to the rotation trick :)
re:codebook utilization -- my understanding was that codebook utilization is computed on a single batch (i.e. # of codes used in this batch divided by total # of codes) and then this value is averaged across all batches. ViT-VQGAN has a good description, and this is how we computed codebook utilization in the rotation trick paper:

Recently, it seems like there's a more liberal interpretation. FSQ computes codebook utilization as "The fraction of the codewords that are used at least once when encoding the validation set."

My take-away from this is to be very clear which measure is used in the manuscript and highlight that different measures are being used across different works.

@cfifty ah i see! thanks for clearing up the two ways code utilization is being measured :pray:

@guillaumeguy ok, i got the residual flavor of sim vq built here. give it a test drive and let me know what you see on your end

@guillaumeguy also, i'm curious if you buy into the hand wavy arguments that residual VQ induces hierarchical reps in that recommendation paper

Thanks @lucidrains ! Why did the expected input dimension change? ResidualVQ was happy with (B,dim) but simVQ isn't.
from vector_quantize_pytorch import ResidualSimVQ

B = 24
dim = 256
input = torch.randn(B,dim)

sim_vq = ResidualSimVQ(
            dim=dim,
            num_quantizers=4,  # specify number of quantizers
            codebook_size=256,
            rotation_trick=True,
        )

sim_vq(input)

Will get you:
Equations:
    [c] d = 256 256
    b n = 24
Reason: Expansion '(2,)' of expression 'b n' does not match expansion '(1,)' of expression '24'

@guillaumeguy can you try the latest version?

@guillaumeguy seeing anything? 👀

it’s running as we speak (with performance slowly increasing). Should have some results tomorrow.

ok, the results are a bit disappointing (keeping everything else constant).

Method
Similarity (100=perfect)
active (%, perfect=100)

SimVQ
82.3
4.0

ResidualVQ
87.7
65.9

I will try to tune the lr. This will probably help simVQ a bit

@guillaumeguy that doesn't seem too bad at all, 5% improvement, although code utilization for plain sim vq seems low
is this with Adam or adagrad? Curious if the collapse issue got mitigated or not

oh nvm, noticed you were comparing only with residual vq
both have rotation trick turned on?

you meant to write ResidualSimVQ, correct?

ok, I would say try the small MLP instead of single Linear that I have in the examples. Otherwise, if it still does not work, I'd recommend just sticking with vanilla residual VQ + rotation trick then

@guillaumeguy thank you for sharing this

it is using AdaGrad with lr=5e-3 and batch size of 1024. The encoder/decoder have 2 intermediary layers

@guillaumeguy I would be curious to see a sim vq + Adam experiment and see if it still collapses or not
that will be very telling
only if you have time or course!

Hold on ...  I used SimVQ instead of ResidualSimVQ. Let me try with ResidualSimVQ

how did you decide on using adagrad vs adam? was this from some paper or your own discovery?

Hold on ... I used SimVQ instead of ResidualSimVQ. Let me try with ResidualSimVQ

Lol

how did you decide on using adagrad vs adam? was this from some paper or your own discovery?

Own discovery.

how did you decide on using adagrad vs adam? was this from some paper or your own discovery?

Own discovery.

that's kinda cool! maybe worth a tweet

After a few iterations (~200 steps).  Looks like Ada works better and is less sensitive to learning rate.

description
lr
optimizer
Batch
eval_active (100=all)
eval_cmt_loss
eval_dot_prod (100=perfect)
eval_rec_loss
train_active  (100=all)

Ada vae 8 quantizers (dim 256)
0.005
ada
1024
52.25
0.00
75.00
0.02
55.71

SimVQ - Ada - vae 8 quantizers (dim 256)
0.005
ada
1024
0.59
0.00
67.56
0.02
0.59

SimVQ - Ada - vae 8 quantizers (dim 256)
0.4
ada
1024
0.78
9016951.00
18.86
0.03
0.93

Ada vae 8 quantizers (dim 256)
0.4
ada
1024
13.96
0.00
19.90
0.03
15.67

SimVQ - Adam - vae 8 quantizers (dim 256)
0.4
adam
1024
0.49
2223902208.00
19.69
0.03
0.39

adam vae 8 quantizers (dim 256)
0.4
adam
1024
0.39
0.00
18.27
0.03
0.39

adam vae 8 quantizers (dim 256)
0.005
adam
1024
49.76
0.00
65.03
0.02
62.40

SimVQ - Adam - vae 8 quantizers (dim 256)
0.005
Adam
1024
0.59
0.00
67.64
0.02
0.63

(SimVQ = ResidualSimVQ)
Both use the rotation trick. No shared codebook

thank you Guillaume, this is very helpful! I would say run one last experiment where you remove the rotation trick from SimVQ, just in case I broke it by removing one half of the commit loss

Without rotation trick:

RE: 200 steps, yes, the TIGER paper runs for 20k steps but I found it useful to run a lot more (it plateaued at around 150k)

aw ok, well I guess vanilla and rotation trick it is

Strange results—it seems like SimVQ is not working at all. I believe there must be an issue somewhere. Could you provide the code for me to test?

@youngsheen

@youngsheen Hi, congrats on the great work🎉!
I noticed a difference: should the bias for the codebook projection set to be false as in here? https://github.com/lucidrains/vector-quantize-pytorch/blob/3e4ce165774d3f5944f12ffb5ccd02848bb38df6/vector_quantize_pytorch/sim_vq.py#L62
In your implementation you keep the default value which is bias=True

@ehsan-ami hey Ehsan! thanks for adding your voice
have you tried it yet? curious what your experience has been

@lucidrains Hi Phil! Thanks for your great repo!
Yes, I just did some experiments with both your implementation (w/ rotation trick) and Yongxin's implementation.
With both of them I am observing is that codebook utilization starts from a low value and slowly increase up to some cap. value (3% utilization); after that it fluctuates . Here are my codebook utilization (entropy with the base of 2) plot:
The magenta color shows your implementation and the orange one is based on original implementation

(I discontinued the orange experiment as it seemed that I am getting (almost) similar training stats)
SimVQ seems to be a promising method for training VQ-VAEs; though I am suspicious that it is sensitives to hyperparameter choice (optimizer, the ratio of embedding size to codebook size, codebook projection matrix initialization method, etc.).  At least for my model I couldn't make it to get good performance in terms of reconstruction loss and codebook utilization.
PS 1: I am using SimVQ for time series quantization
Codebook size is 2048 (~=entropy 11), embedding_dim: 192, optimizer: AdamW with default betas
PS 2: I tried FSQ, which has no extra hyperparameter; it could get good utilization and reconstruction performance.

@ehsan-ami thank you Ehsan!


relevant / non-relavant ratio

What is the ratio of Relevant/Non-Relevant (positive/negative) in the large training set (Train Triples Large)? 400/1?

I dont know specifics but I think its ~1000/1. The logic was we could create a triples that was large enough that not all of it would be used.


Channels closing without stacktrace from iconv

Issue description
I'm randomly getting channel closed errors with no useable stacktrace when running ts-node-dev like this:
Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
    at process.target.send (internal/child_process.js:705:16)
    at Object.exports.send (/PATH_TO_PROJECT/node_modules/ts-node-dev/lib/ipc.js:17:14)
    at /PATH_TO_PROJECT/node_modules/ts-node-dev/lib/wrap.js:88:9
    at Object.nodeDevHook [as .js] (/PATH_TO_PROJECT/node_modules/ts-node-dev/lib/hook.js:61:17)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.getCodec (/PATH_TO_PROJECT/node_modules/iconv-lite/lib/index.js:65:27)
    at Object.getDecoder (/PATH_TO_PROJECT/node_modules/iconv-lite/lib/index.js:127:23)
    at getDecoder (/PATH_TO_PROJECT/node_modules/raw-body/index.js:45:18)
    at readStream (/PATH_TO_PROJECT/node_modules/raw-body/index.js:180:15)
    at getRawBody (/PATH_TO_PROJECT/node_modules/raw-body/index.js:108:12)
    at read (/PATH_TO_PROJECT/node_modules/body-parser/lib/read.js:77:3)
    at jsonParser (/PATH_TO_PROJECT/node_modules/body-parser/lib/types/json.js:135:5)
    at Layer.handle [as handle_request] (/PATH_TO_PROJECT/node_modules/express/lib/router/layer.js:95:5)

I know this error is just a symptom of real problem in my code, but it's really hard to debug with no stacktrace or error message beyond this.
I'm guessing a type is going unchecked somewhere which is resulting in iconv erroring out rather than the error being caught + handled earlier.
Context
OS version (is it docker or host?), ts-node-dev version
macOS 11.2,  ts-node-dev v1.1.1
Did you try to run with ts-node?
Yup, not had the same thing happen (yet). Not been able to find the core problem yet either.
Did you try to run with --files option enabled?
Yes
Did you try to run with --debug option enabled?
Yes
Do you have a repro example (git repo) with simple steps to reproduce your problem?
No - random

I also have this problem when I start webpack-dev-server with ts-node-dev. For me, I restarted my computer and it worked. I guess it could be that the thread port is occupied by some program, which causes the ts-node-dev thread to fail to start.


Hover text needed for lozenges for class, attribute etc.

Can the E for data element be changed to DE please too?

This is two separate issues so needs to be split.

changing the styling is a CSS issue (and trivial).  I'll pick this up outside of this ticket.  @AngelaFaulding has final say on the e / de choice.
For the shortdesc for the map entries this feeds into the breaking up of the maps into bite-size chunks.  I'm minded to have dedicated topics for these and hang the detailed maps off those.

We have had a few comments about the styling and would like e changing to de please.

Nicholas decided it should stay as e so issue closed.


CAMEL-12665 - Apache Pulsar integration via component

Similar to the other Pull Request however based on master rather than 2.x

I merged this on master, but I had to change some stuff.

Can you please update the documentation too? What is there actually, it's autogenerated, but you can add example and explanations. We can add the same to 2.x, once you aligned.


"Be specific" ending still gets question mark added

See, for example, https://pbeprep.com/questions/7101/edit
The automatic addition of the question mark is very useful. However, the "be specific" specifier at the end might need to either be detected in a special way, or perhaps turned into a flag that adds a language-specific ending.

This doesn't have a very thorough check or non-English check, you are correct. This would be a good opportunity to enhance the system with a more clever check.
The question you linked should be covered, though. See the below checks that are currently in the system.
https://github.com/Deadpikle/pathfinder-bible-experience-prep-quiz-master/blob/208d87c6f4413a39ef6516bbe0901c3fd6cccc67/views/user/quiz/take-quiz.twig#L453-L456
https://github.com/Deadpikle/pathfinder-bible-experience-prep-quiz-master/blob/master/app/Services/PDFGenerator.php#L18-L21

Oh..there is a separate check before it messing with the logic, here. 🤦🏻
https://github.com/Deadpikle/pathfinder-bible-experience-prep-quiz-master/blob/208d87c6f4413a39ef6516bbe0901c3fd6cccc67/views/user/quiz/take-quiz.twig#L450-L457

Fixed, and bonus: finally got PHPUnit in the project for some better testing!
Thank you for the report.


2 issues

Expected behavior
When I was in the replies section, I clicked Discover More People on the right, it showed more people, my replies section should have stayed intact. Instead, it says, Oops... empty feed.
Actual behavior
I am in REPLIES section....

I  hit the DISCOVER MORE PEOPLE and it changed to...

Two Issues
1.) It should have kept my Replies in the middle.
2.) In addition, from her.... after I hit Discover More People one time, and hit it again, it doesn't give me a new list of new people to review. In fact, nothing happens.
How to reproduce

Browser: chrome
Operating system: win 10

"Discover More People" is not implemented yet. Related to #812


700016 Not Documented

700016 is not documented. Message I'm receiving is "Application with identifier <GUID> was not found in the directory <DIRECTORY GUID>. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fa732de6-2cab-d780-aa21-dfec80e0a8fe
Version Independent ID: 1a11b9b6-cf4f-3581-0864-0d5046943b6e
Content: Sign-in activity report error codes in the Azure Active Directory portal
Content Source: articles/active-directory/reports-monitoring/reference-sign-ins-error-codes.md
Service: active-directory
Sub-service: report-monitor
GitHub Login: @MarkusVi
Microsoft Alias: markvi

@threesixfour Thank you for your feedback . We will engage the content authors for getting this request reviewed and actioned.

@MarkusVi Could you please help getting the error details documented in the article after verifying with engineering?

I have staged an update for this.
#please-close


Add Fault with BaseError and BaseException

All updates made here will be merged in the last PR (#211), in this nested chain of collection execution PRs.


feat: Removed support for launch configuration and replace count with for_each

Description

Add support for cluster addons
Add support for cluster identity provider configuration
Create separate, standalone sub-modules for:

User data
EKS Managed Node Group
Self Managed Node Group
Fargate Profile

Drop support for launch configurations - only launch template will be supported going forward
Drop support for managing aws-auth configmap

Drop requirement of requiring aws-iam-authenticator
Drop requirement of Kubernetes terraform provider

Replace list/count usage with maps/for_each
Update user data usage to allow users to provide their own template in addition to the various settings and configurations provided for the module provided user data templates
Update to allow cohesive suite of settings that are uniform across the node pool types supported and continued support of multiple node group default settings under the root module
Update to allow more control over customizations/configurations of (not limited to):

Security groups
IAM roles
Tags
User data
Launch template

More details provided under the UPGRADE-18.0.md document

Motivation and Context

Closes #858
Closes #1105
Closes #1280
Closes #1302
Closes #1385
Closes #1443
Closes #1469
Closes #1488
Closes #1498
Closes #1505
Closes #1539
Closes #1558
Closes #1566
Closes #1579
Closes #1582
Closes #1590
Closes #1616
Closes #1624
Closes #1626
Closes #1630
Closes #1644
Closes #1645
Closes #1658
Closes #1661
Closes #1662
Closes #1675
Closes #1676
Closes #1677
Closes #1679
Closes #1682
Closes #1685
Closes #1686
Closes #1688
Closes #1689
Closes #1692
Closes #1693
Closes #1694
Closes #1695
Closes #1699
Closes #1701
Closes #1702
Closes #1705
Closes #1707
Closes #1710
Closes #1711
Closes #1717
Closes #1720
Closes #1721
Closes #1724
Closes #1726
Closes #1727
Closes #1729

Breaking Changes

Yes! Please see the UPGRADE-18.0.md document for details

How Has This Been Tested?

[x] I have tested and validated these changes using one or more of the provided examples/* projects

All examples under examples/ have been updated and validated

@bryantbiggs I think this feature should be moved to wubmodule
similar to node groups

@daroga0002 / @antonbabenko this is getting close for review if you have time to check it out. I still need to populate the upgrade document and comprehensive list of changes, plus a few other design decisions (i.e. - using the default blocks, security group usage, etc.). I've made some more changes today to align variable usage and descriptions so I need to run through all of the examples yet again tomorrow to re-validate but 🤞🏽 they are working as intended (if not, please let me know).
Let me know if you see any glaring/obvious errors - I'll be working on the upgrade doc and I would like to avoid having to make multiple iterations on it since I know its going to be quite involved.
cc - @barryib / @max-rocket-internet / @dpiddock / @lisfo4ka for any input you may have as well 🙏🏽

I think it is a good idea to have an SVG diagram describing bootstrap logic - https://raw.githubusercontent.com/bryantbiggs/terraform-aws-eks/refactor/breaking-changes/.github/images/user_data.svg

I think it is a good idea to have an SVG diagram describing bootstrap logic -<EMAIL_ADDRESS>(raw)

yes, agreed! the images will be in the final PR but right now I have both a relative link and a fully qualified link. the relative link allows the image to be rendered in this PR just to make sure everything shows up as expected but it won't render on the remote Terraform docs for the module; this is what the fully qualified URL is for. so I was just putting a TODO reminder to remove the relative links before merging so that we have the fully qualified URLs that will show the images on both GitHub and on Terraform's module docs

ok this is finally ready for final review. I tried to work out some terraform state mv ... steps but in the end the module wants to replace nearly everything that is moved to the new name due to the extensive use of name_prefix (IAM roles, security groups, launch templates, autoscaling group, EKS node group, etc.). Therefore, I have left out any state mv recommendations from the upgrade guide. Let me know if there are any additional changes/feedback/final validation - @antonbabenko / @terraform-aws-modules/triage-supporters

@bryantbiggs  https://www.terraform.io/docs/language/modules/develop/refactoring.html this can be helpful for migration
Terraform 1.1 introducing this 🎉

Are we raising minimum requirements from Terraform 0.13 to 1.1 already? :) It will be a surprise for many users, plus refactoring features released yesterday are not battle-tested yet. I think we can come back to this and start using something like this in half a year but not before.

thank you for the feedback @daroga0002 ; to answer some of your questions posed:

do we remove users/roles feature in aws-auth config? what will be desired way to make it?

this change set has removed all management of the aws-auth configmap. there is one output that is provided that renders a configmap containing all of the roles created by the module, but how users utilize/manage the configmap is now completely up to them. https://github.com/terraform-aws-modules/terraform-aws-eks/pull/1680#discussion_r765859051

aws-auth seems be now required to be done outside module what is causing that in case of self nodes it will always require handling this externally (so module can be threated as not sufficient in terms of creating cluster

I think this is similar to above (apologies, not quite following the question). yes; aws-auth is out of scope for the module in this proposed change

EKS silently create in background SG which is assigned by default to node groups, can we also assign this by default to self nodes so then we will have unified approach and will dont require adding those SGs for DNS and etc.

yes, I believe you are correct that EKS managed node groups do create a security group by default that connects the node group to the control plane. could we somehow integrate and tap into that node group in this module - TBH I'm not sure. just off the top of my head I see a few concerns:

It looks very precarious and ripe for cyclical dependencies which are problematic
It is only made available when an EKS managed node group is added into the mix (maybe Fargate profiles, I can't remember?), but if you only use self-managed node groups then this is not present and you are back to no network connectivity

I'm not saying its not a good idea, I just don't know what that would look like other than adding in quite a bit of complexity and weird edge cases that will once again introduce more complexity back into the module

what will happen with legacy module? should it be archived in seperate branch?

Nothing, just git business as usual. The 17.x (and all prior) are eternally locked into git/github/and the Terraform registry. we always strongly recommend users fully pin their module versions so they don't get any unexpected surprises so even if this does go forward, people using 17.x today will not know anything different. However, if they wish to utilize new features/fixes/changes, then they will have to evaluate the upgrade to 18.x and make that change when they see fit

ok - the last change is in, this should be ready to go at any time @antonbabenko

@bryantbiggs awesome work on writing all this! 👏 ❤
I got excited seeing this, so I tried using this new version of the module! Unfortunately, I hit an issue. Chances are I am doing something wrong, but when I try to pass in dynamic resources to this new version of the module, I get an error: The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each depends on. I opened a PR with a minimal example of failing code here: https://github.com/bryantbiggs/terraform-aws-eks/pull/1
I think this may be due to Terraform's limitations for for_each? Or I am doing something wrong. Both are possible 😅

@bryantbiggs Sorry for the delay. Is it ready for final review & merge (I remember there were some Q in your fork)?

Drop support for managing aws-auth configmap

@bryantbiggs what will be replaced with ?

Drop support for managing aws-auth configmap

@bryantbiggs what will be replaced with ?

@DanyC97 it is left up to users as to how they want to manage the aws-auth configmap. This change set does provided a rendered aws-auth configmap template that can be used (see examples)

Drop support for managing aws-auth configmap

@bryantbiggs what will be replaced with ?

@DanyC97 it is left up to users as to how they want to manage the aws-auth configmap. This change set does provided a rendered aws-auth configmap template that can be used (see examples)

hmm, i see. If before we had "batteries included" now the user will need to know that "hey, you need one more thing to do outside/ in addition to current module".  Hopefully this note will be somehow written in bold ;)

Drop support for managing aws-auth configmap

@bryantbiggs what will be replaced with ?

@DanyC97 it is left up to users as to how they want to manage the aws-auth configmap. This change set does provided a rendered aws-auth configmap template that can be used (see examples)

hmm, i see. If before we had "batteries included" now the user will need to know that "hey, you need one more thing to do outside/ in addition to current module". Hopefully this note will be somehow written in bold ;)

This is all captured in the extensive UPGRADE-18.0.md guide
I will also caution, your statement above rings true for the current "batteries included" configuration. Users are required to have aws-iam-authenticator installed locally to where they are running Terraform; this is one of the many reasons for removing. This module is for provisioning AWS infrastructure resources for EKS, while cluster configuration and management is left out of scope in order to reduce the burden on this module (meaning whatever changes sent to the cluster API, those should not be managed here as that is mixing concerns)

Drop support for managing aws-auth configmap

@bryantbiggs what will be replaced with ?

@DanyC97 it is left up to users as to how they want to manage the aws-auth configmap. This change set does provided a rendered aws-auth configmap template that can be used (see examples)

hmm, i see. If before we had "batteries included" now the user will need to know that "hey, you need one more thing to do outside/ in addition to current module". Hopefully this note will be somehow written in bold ;)

This is all captured in the extensive UPGRADE-18.0.md guide
I will also caution, your statement above rings true for the current "batteries included" configuration. Users are required to have aws-iam-authenticator installed locally to where they are running Terraform; this is one of the many reasons for removing. This module is for provisioning AWS infrastructure resources for EKS, while cluster configuration and management is left out of scope in order to reduce the burden on this module (meaning whatever changes sent to the cluster API, those should not be managed here as that is mixing concerns)

fair @bryantbiggs  thanks for taking detailed info

@bryantbiggs Great Job! When the version 18.0 will be released? Thanks so much

@bryantbiggs awesome work on writing all this! 👏 ❤
I got excited seeing this, so I tried using this new version of the module! Unfortunately, I hit an issue. Chances are I am doing something wrong, but when I try to pass in dynamic resources to this new version of the module, I get an error: The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each depends on. I opened a PR with a minimal example of failing code here: bryantbiggs#1
I think this may be due to Terraform's limitations for for_each? Or I am doing something wrong. Both are possible 😅

workaround is to precreate the iam roles

@bryantbiggs I am not sure why this PR didn't trigger a major release action, do you know?
https://github.com/terraform-aws-modules/terraform-aws-eks/runs/4714285326?check_suite_focus=true

@bryantbiggs I renamed the PR title to include ! to make sure that this should be a breaking release but it didn't seem to help. Weird. https://www.conventionalcommits.org/en/v1.0.0/#commit-message-with--to-draw-attention-to-breaking-change

@antonbabenko it's probably because none of the commit (not the PR title) contains a !.

PR title was "feat!: ...", so it was used as a commit ee9f0c646a45ca9baa6174a036d1e09bcccb87b1, too.

oh yes, you're right

well thats really annoying! let me see what I can find out @antonbabenko

I've opened a PR to fix this here https://github.com/terraform-aws-modules/terraform-aws-eks/pull/1736 based on some testing I did today to dive into this issue. tl;dr - default of angular preset does not respect the ! and requires BREAKING CHANGE in the commit footer to trigger a breaking change (feat! is not valid either under this preset which is why nothing was released)

This PR is included in version 18.0.0 :tada:

I am trying to find out why docs/iam-permissions.md is missing from master. Because it is now missing from master, the there is a broken link here: https://learn.hashicorp.com/tutorials/terraform/eks


Update README.md

Great work so far! Feature engineering will become more intuitive with practice. For bicycling consider the factors that may effect the decision to ride, such as too hot, too cold, too windy, or maybe a high pollen count? Considerations like these may create very predictive features. Keep up the good work!


Design Choice: one file or X files?

All the code in one file, a la John Zelle's graphics.py package is quite ungainly with over 3000 lines.
Do you want to keep everything in one file, or are you considering splitting the code into various modules?

I was considering splitting it up especially because it would make it
easier for the user to access different functions for different purposes.
It made it a bit easier when developing to have everything in one place,
but I see your point. I'll work on splitting the file down.
And thanks so much for reaching out! I'm pretty excited.
[image: Mailtrack]
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&
Sender
notified by
Mailtrack
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&
07/03/20,
10:04:46 PM
On Fri, Jul 3, 2020 at 10:00 PM FredInChina<EMAIL_ADDRESS>wrote:

All the code in one file, a la John Zelle's graphics.py package is quite
ungainly with over 3000 lines.
Do you want to keep everything in one file, or are you considering
splitting the code into various modules?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/BhavyeMathur/goopy/issues/44, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQE6XJ5KNNBXCNEEIVTIBI3RZYBT3ANCNFSM4OP6CEHA
.

--
Regards,
Bhavye Mathur 8B
MYP Innovation & Technology Captain
Indus International School Pune
<EMAIL_ADDRESS>+91 97398 69101

So I was trying to split the library down into multiples files and it seemed like a lot of work... and I asked myself why? Why are so many libraries split into multiple files when 1 will just do? There must be a good reason for Manim, Keras, Tkinter and basically every other professional library to split themselves into multiple files.
The first thing that comes to mind is just that it becomes more organized but is that really an issue when using an IDE that can fold everything down? The main thing I see is that users can quickly reach functions which they need - need colours? use goopy.colour._____. So do you have any tips for splitting the file?

Clearly, you have not thought that through, here is an article to get you started:
https://www.quora.com/Why-do-you-want-to-split-your-program-code-into-multiple-files?share=1
For compiled languages, it is to avoid having to recompile the entire program when recompiling only one file suffices.
John Zelle's graphics.py package is designed to be lightweight, and make it easy for novice programmers to get started. It has, if I remember, in the order of 700-900 lines of code, which is a lot, but still "somewhat" manageable in one file.

It's been done and I have also renamed everything to follow the Python Convention rather than Camel Casing.


Native Amp Support

Native automatic mixed precision support (torch.cuda.amp) is finally merged:
https://pytorch.org/docs/master/amp.html
https://pytorch.org/docs/master/notes/amp_examples.html
Apex Amp has many known pain points (extension builds, forward/backward compatibilty, DataParallel support, flaky checkpointing, i don’t even know if it can be hacked to handle double backward/gradient penalty, others…). torch.cuda.amp repairs all these, the interface is more flexible and intuitive, and the tighter integration brings more future performance optimizations into scope.
If you want to talk about adding torch.cuda.amp to Lightning, with an eye towards it becoming the true source of mixed precision and replacing Apex, message me on Pytorch slack anytime. I pinged you there as well, but I’m not sure if you monitor it habitually.

Closing and starting a duplicate issue to get #1337.


Add Arrangement attribute for RadioButtonGroup

Codecov Report

Merging #20 into master will decrease coverage by 0.05%.
The diff coverage is 21.42%.

@@            Coverage Diff             @@
##           master      #20      +/-   ##
==========================================
- Coverage   45.19%   45.13%   -0.06%
==========================================
  Files         101      101
  Lines        1956     1965       +9
==========================================
+ Hits          884      887       +3
- Misses       1072     1078       +6

Impacted Files
Coverage Δ

...orm_element_parsers/radio_button_group_parser.dart
7.69% <0%> (-3.42%)
:arrow_down:

...orms/lib/src/form_elements/radio_button_group.dart
17.64% <60%> (+17.64%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd64a09...7741b98. Read the comment docs.


Defined new revisions for golang dep: github.com/google/certificate-transparency-go

Ran component through the scanners since I noticed it had not been harvested yet. PR matches curation site. Closing.


Final v3 fixes

This fixes the location of required in params.  It also fixes the format of params that use custom type mappings.
After this, I think the swagger v3 support is complete.  For the project I'm using it on.  I'm now able to get a swagger.json that has no validation problems.

Coverage increased (+0.4%) to 89.939% when pulling c8d7ad6d60724859ec1cb7151366bef3dcf60c25 on asdcdow:final-v3-fixes into 57177830e9732777fd1bf095f59f815f0ed7cb14 on iheartradio:master.

Coverage increased (+0.4%) to 89.939% when pulling ee11041c081c2da0fe340a452aad446639e68bb2 on asdcdow:final-v3-fixes into 57177830e9732777fd1bf095f59f815f0ed7cb14 on iheartradio:master.

Do you have any idea when you will put out a new release with the swagger v3 support?

Thanks very much it's just released. which Play version and Sbt version are you using?

Awesome! I'm currently using Play 2.6.5 and SBT 0.13.16

@asdcdow hmm, only play 2.6 + SBT 0.13.16 + scala 2.12 is updated as of now, since we don't have cross build set up on the play2.6 branch.  I believe it should be easy but I don't have the bandwidth to do that any time soon.

Do you you mean only SBT 1.0?  If so, that's ok.  I should be able to update without too much trouble.  However, there seems to be a merge issue with the 0.7.2 release.  It looks like the plugin is only passing 4 arguments to the runner when it should be passing 5.  As a result I'm getting match errors.

@asdcdow yeah only sbt 1.0. For 0.7.2 Oddly the tests passed. I'll take a look.

just released 0.7.3 with the fix.

Thanks, that seems to have solved my problem


nixos/hqplayerd: init

Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Still struggling with a mysterious segfault at runtime that I can't reproduce on Fedora 34.
Might have to revert to using a FHSEnv for this one. TBD.

Alright, segfault tackled, will test this today and then it should be good to go :D

it works :partying_face:

Had andi take a look at it and he verbally OK'd it. Merging since it had been previously approved as well.


How to DI interface instead using AWSXRayRecorder.InitializeInstance(configuration);

Hi,
I would like to ask everyone a question. I want to use DI for AWSXRayRecorder instead WSXRayRecorder.InitializeInstance(configuration); in startup.cs
Thanks

Hi @lethienhoang
Can you explain your use case or what you are trying to achieve? Some example code snippet would be very helpful.

Hi @lethienhoang ,
I am closing this issue due to inactivity. Please feel free to reopen if your question is still unanswered.


Find and coalesce .jshintrc files

Please check if the PR fulfills these requirements

[x] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

[X] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Other... Please describe:

What is the current behavior? (You can also link to an open issue here)
jshint-loader uses the first .jshintrc file it finds and ignores the extends property.
What is the new behavior?
Brings jshint-loader to parity with jshint cli with respect to its ability to find and coalesce nested .jshintrc files.
Does this PR introduce a breaking change?

[ ] Yes
[ ] No
I don't know whether this is a breaking change. Seems ok to me.

Looks like part of the travis build was cancelled.

@akeating - Rebase current master. It has the loader utils fix as well as some travis improvments so it doesn't take 20 minutes to run.

Replaced by #38


Token are stored in plain-text

Both GitHub and GitLab tokens are stored un-encrypted in ~/.config/tsrc.yml.
We should use something like keyring instead.

No longer relevant


[bug] Temperature value null in some edge cases

          > Thanks for sharing this. I am looking into it.

btw
  File "/config/custom_components/dual_smart_thermostat/climate.py", line 949, in _async_control_climate
    await self._async_control_cooling(time, force)
  File "/config/custom_components/dual_smart_thermostat/climate.py", line 1133, in _async_control_cooling
    too_cold = self._is_too_cold()
               ^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/dual_smart_thermostat/climate.py", line 1470, in _is_too_cold
    return target_temp >= self._cur_temp + self._cold_tolerance
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '>=' not supported between instances of 'NoneType' and 'float'

Originally posted by @10der in https://github.com/swingerman/ha-dual-smart-thermostat/issues/210#issuecomment-2207187511
Also here:
https://github.com/swingerman/ha-dual-smart-thermostat/pull/210#issuecomment-2143971749

closed by #242


Text out of error notification border/area

Bug
Text is out of notification area
Screenshots

(if it's something wrong screenshot let me know)

LBRY Daemon version: 0.81.0
LBRY App version: 0.48.0-1 AUR package
LBRY Installation ID: 7BsAHfDrqNEEoAxow942FdzDmRP48NrgGYgLkXioumdG2aYvwj4PtUcpAYt6Mkb5VZ
Operating system: Manjaro Linux

Regards :slightly_smiling_face:

Thanks for filing the issue!

Hi. I have an update and maybe it will be useful :slightly_smiling_face:
I've seen other error notification but it displayed text properly. Maybe because link was shorter? I don't know. Notification disappeared too fast and I don't know what was written there.
Thanks and have a good day :slightly_smiling_face:


Get electricity usage monitors

A Kill A Watt is about $20 -- should we just get two?
@moorepants are the Meijo teams planning to be able to use a 120V NEMA outlet?

Can you find out from petros the electricity needs for them? I think Japan is on the same voltage, just needs an adapter for plug shape.
Two is fine.
The should be charged to my EME 185 account, which I don't know the number. Ask Jen Young.

Haven't heard back yet, so I'll send a follow-up. In the meantime, I looked into it a bit and it seems the sockets in the US and Japan are essentially physically the same. Japan outlets provide 100V with some at 50Hz and some at 60Hz. A proper 120V to 100V converter is around $40, but I don't think it's necessary because most AC adapters accept some range of inputs like 100-240V at either 50/60Hz.

Haven't heard back from Petros, Jen Young, or both? FYI, Petros is coming by here on Friday. I'll schedule so we can both meet with him.

Was referring to Petros -- sorry, I hadn't re-read the previous message.

For the record, we're moving forward assuming the Meijo teams will be able to use US power without issue.
Order for the electricity usage monitor has been placed.

Picked these up yesterday. I can hang on to them and bring them to the competition.

Once one of the machine is running here, try them out and see if it can actually measure what we want.

I had the bottlecap team run a bottle through with the meter today. A couple things:
I think what we want to know is the energy consumption per bottle cap removal (in Watt-seconds). The kill-a-watt monitor displays instantaneous power in Watts (at about 2 Hz), and it provides an integrated power in kilowatt-hours. The issue is that this kWh monitor is meant for long-term monitoring and its resolution is 10 Watt-hours. If we wanted to accurately measure the energy consumption per bottle cap removal, you'd have to run the machine continuously until the kWh meter increments and count the number of bottles processed. At 60 Watts (approximate power consumption of UCD bottlecap team's design), this will take 10 minutes to increment the kWh meter. I'm guessing 60 Watts would be on the high end compared to the Meijo bottlecap design and both paperplane designs.
What we can do is take video of the Watt meter while the machine processes N bottles, watch it back hat half speed to take down the output each time it updates, integrate, and divide by N. The monitor might just be a zero order hold, but it's possibly sampling more rapidly and averaging to produce a somewhat smooth meter output (I played around with it a bit and I think it is the latter). Either way, I think it's reasonable to do the test this way despite the chance of peak power consumption occurring between samples. I was surprised how steady the power consumption of the UCD bottlecap team's design was, so it shouldn't be an issue unless the Meijo team's cap removal happens extremely quickly.
All this should hold for the paperplane projects as well.
In the future, it may be worthwhile to look around for a power/energy monitor that can be hooked up to a DAQ. I'm guessing that would be in a totally different price range compared to a consumer product like the kill-a-watt though.

For the record, Professor Linke let us borrow a Watts up? PRO, which looks like it has 0.001 Wh increments, though it only updated in 0.1 Wh increments (which was fine). The Watts up? standard model is about $100 [link].


correct path for violation in list

use string representation instead if getName to create the path

{
   "list":[
      "invalidElement"
   ]
}

the path in case if a constraint violation exception for the given example:
before: "#/list"
now: "#/list[0]"

Coverage remained the same at 95.968% when pulling cf25fb7160620a6c3a1122437aa3b01250f1bb4d on validation_error_list into 2248a9a15b67b6a884c5fc8b6edc7494fde0c6dc on master.


question

I really like the idea of html/js/etc to desktop app.
I really do NOT like the idea of a desktop app being dependent on listening to tcp/ip ports.
It looks like this one might not be...? is it?   If so, then what's the alternative for getting input/data from the ui to the code[since ajax & not tcp/ip].
Are there any like that[besides vuido]?

I don't understand this question. There is no need to use any TCP/IP ports in order to create a GUI.

I think by "tcp/ip ports" you mean running a local web server and then opening the page in your browser.
If you want to use HTML and Javascript for a desktop app, take a look at Electron.


Typo in EncodedPayload->decreaseSize

https://github.com/rollbar/rollbar-php/blob/c2f8be87983ecdd3c46ceb3c459de02376af4707/src/Payload/EncodedPayload.php#L26
$this->size =- $amount

@ramapitecus thanks for bringing this to our attention (as well as #427 and #428).
@ArturMoczulski can you take a look when you get a chance?

I'm not sure I understand what the problem is here

I'm sorry I wasn't clear. The method is called decreaseSize so I assume that if $this->size is 10, and you call $obj->decreaseSize(2) you'ld want $this->size === 8 but there is a typo there, the =- is flipped. And $this->size === -2 after that call.

Fixed in master. Will be included in the next release.

Released in v1.7.3


[FAQ] CQRS queries

Hi,

Need to know how CQRS is implemented here ?
How is ES helping to build CQRS ?
From casual code walk through, my understanding is, The commands and query both hit the same database, am I right ?
How to use two DBs with eventual consistency for CQRS ?

Thanks.

We use Commands to trigger actions on domain model, domain model publishes events that are handled by denormalizers to build a read model. All reads are from denormalized read model, all "writes" are through execution of command. Reads & writes separated.
ES does not help, it is just a way to implement it - you could build a perfect CQRS application without Event Sourcing
Yes. Even more - currently RailsEventStore supports only synchronous subscription handlers - however it is quite easy to implement asynchronous flow with Resque or any other background jobs lib.
See point 3. Use asynchronous handlers would be nice. However even now I could imagine that instead of storing read models in SQL database any document store could be used to store serialized order / order list (ie. Redis)

@kirantpatil sorry for very late response.


Automated Package Upgrade

:tada: This PR is included in version 25.6.103 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


update sign in flow

Pre-Submission Checklist

[ ] Your pull request targets the staging branch of FreeCodeCamp.
[x] Branch starts with either fix/, feature/, or translate/ (e.g. fix/signin-issue)
[ ] You have only one commit (if not, squash them into one commit).
[x] All new and existing tests pass the command npm run test-challenges. Use git commit --amend to amend any fixes.

Type of Change

[ ] Small bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds new functionality)
[ ] Breaking change (fix or feature that would change existing functionality)
[ ] Add new translation (feature adding new translations)

Checklist:

[x] Tested changes locally.
[x] Closes currently open issue (replace XXXX with an issue no): Closes #10804

Description

This adds a /welcome view which it directs new signups to. This view explains what to do if a duplicate account has been created (this happens all the time). It also gives campers a chance to skip our onboarding if they're in a hurry.
It also removes the notice about verifying their email address, since this is something they can do later.

@BerkeleyTrue please take a look at this and QA. Will this be easy to apply to staging?

@QuincyLarson Is it supposed to aim at backup/master instead of staging?

@hallaathrad yes - I was hoping we could merge this into master so it could reduce confusion among new users while waiting for React to ship.