proposal: a faster C-call mechanism for non-blocking C functions

I'm not sure this is the place to submit this, the golang-dev list might also have been a good candidate.
The cgo FFI mechanism is very general, and makes some pessimistic assumptions about the function being called - specifically it has to handle the case where the called function may block, e.g. in a blocking syscall. Having to handle the worst case adds a lot of overhead to each call into C, including just calling a math function or other small function that does no syscalls or does not block.
The problem is for well-behaved calls that neither block nor call back into Go, if the called function does not do much work then the cgo overhead dominates the runtime and causes poor performance. This is a well known limitation (and often complained about on golang-nuts and elsewhere, google "cgo overhead" returns about 45,000 hits.) Some of this can be mitigated by making the C function do more work, e.g. exporting a chunkier vs chattier API - but that's not always feasible or desirable.
For some classes of problems this overhead is just not admissible, and people resort to doing bad things like calling C from assembly on a Go stack, which just isn't built for it. The Go runtime itself finds the C call overhead too much at times, and Dmitry Vyukov had to work around it when integrating the TSAN race checker with Go. A look at https://golang.org/src/runtime/race_amd64.s?m=text shows a custom C call mechanism that just puts the arguments into the right registers, switches stacks from the Go stack to the C stack, and calls the TSAN runtime function.
Since this seems to be a requirement that cannot always be worked around, and it's a requirement of both Go users and the Go runtime itself, and the workarounds are difficult, fraught, and fragile - I would like to see a solution for this in Go itself. Such a solution should very closely mirror the functionality in race_amd64.s, ideally because that approach is about as low overhead as possible, and because it can be used to replace that code. It could take any form, but since it needs to be backwards compatible and ideally one does not introduce new syntax - something like an attribute on the function declaration, or a cgo directive would be nice. It is up to the programmer to ensure that C functions called in this way do not block, and don't call back into Go.

e.g. __attribute__ ((nonblocking)) in the declaration,
or // #cgo nonblocking directly preceeding the callsite or the declaration

Is this a reasonable feature for a future version of Go?

This would probably be more usefully discussed on golang-dev rather than on the issue tracker.
In any case I think the first step is straightforward: measure the overhead of a cgo call and get a good estimate of how much could be saved by adopting this proposal.

Sorry for posting this in the wrong place then.
Overhead of cgo is about 170ns and up (600-800 cycles). Calling Go functions is about 100 times faster. Using the alternative call mechanism, Dmitry cites a real-world improvement of up to 50% of programs running under the race checker - that's for the entire program, presumable not all of which is spent calling into TSAN.
Again not all programs will benefit from this, but there are a class of programs which do - including Go itself - and more importantly for which there are no practical alternatives.

Note that the performance of cgocall has steadily decreased in the past 2 years. The penalty for a cgo call was around 30ns in go1, now (go1.6) we're at ~200ns.
Maybe the easiest thing to do would be to first investigate whether we can recover at least some of the go1 performance for cgocall ?
Reference for this problem is issue #9704.

I like the idea of nullifying the large overhead of cgo which does affect a class of programs. I'm a little concerned on how hard it would be for a programmer to prove that a C function is safe for the #cgo nonblocking behavior. Trivial functions are easy, but what about a moderately complex C library? How  can I convince myself that there is no codepaths that call a blocking syscall? In fact, how can I know which standard library functions do call blocking syscalls? I find GODEBUG=cgocheck very useful, so maybe something similar could be attempted (thought I can't see how)?

I'm not convinced that #cgo nonblocking is a good idea, because, as you say, it's very easy to get wrong.  In particular it's very easy to get wrong for unlikely error cases, leaving you with a program that normally works but occasionally deadlocks.
We could detect a deadlock easily enough in the sysmon thread: if a supposedly non-blocking cgocall doesn't return within some period of time, we can declare an error.  We can't recover, though; all we can do at that point is crash.
I don't know of a way to know which standard library functions do blocking syscalls.  That is a somewhat different path than a cgo call, and should be more efficient--for example, it doesn't have to switch stacks, as syscalls don't use any stack space.  Still it's true that there is some overhead.  It's also true that the syscall package uses a more efficient call path for syscalls that don't block; they are the ones annotated with sysnb comments rather than sys.
It's worth noting that we've been surprised in the past by which syscalls can not block.  For example, when people use a FUSE file system, Dup can block (see issue #10202).  This is the kind of problem that could vex #cgo nonblocking in unusual cases.

 As far as I understand it it's non-optimal, but not going to cause deadlocks. What would cause deadlocks is if the code did something that blocks on another goroutine which won't run now that we're blocking the scheduler. That's still a problem but a lot more specific than just it needs to be a function that never blocks. It needs to be a function that usually doesn't block, and that if it does it never blocks on another thread in the process - e.g. with a mutex, or something. That's a lot easier to guarantee. I can call functions from the C standard library not sure that they would block or not, but I can be sure if they will or won't block on another goroutine in my program - especially since any goroutine blocked in the scheduler cannot be in the middle of another C non blocking call. Like the mutex(s) in malloc aren't a problem because a goroutine can't get unscheduled in the middle of a malloc. Any mutexes only used inside C also won't be a problem because it's not possible to put a goroutine to sleep while it's in the middle of a C call. Of course my understanding may well be flawed, but if it's not, then it seems like this is not an unreasonable burden to place on the programmer for use of an opt-in, advanced feature.
Remember that the alternatives when cgo performance is not enough, are more dangerous. They carry these exact same risks, plus are fragile to the Go implementation changing, plus involve assembly and access to go runtime internals. Worse yet, if one avoids doing it right by digging into the go runtime to change stacks, and just do the call on the goroutine stack - now you're making assumptions about how much stack space your C program needs - and that's an extremely unsafe assumption that can't be easily made. Who knows which standard library functions allocate a big buffer on the stack or use recursion.

The most efficient implementation of #cgo nonblocking--and if we don't do an efficient implementation, why bother--can easily deadlock.  The scheduler assumes that it can preempt any goroutine, in order to do a garbage collection or simply because the goroutine has been running for a long time.  However, a goroutine blocked in C code can not be preempted.  That is, a goroutine running in C code is essentially equivalent to a nonpreemptible goroutine, which leads us to the kinds of problems we see in #10958.  Those problems are quite bad, but we live with it because it's hard to write a nonpreemptible goroutine.  But with an efficient #cgo nonblocking, it's easy.
To avoid that problem, we need to notify the scheduler that we are entering a different regime--the steps taken by entersyscall in runtime/proc.go.
No matter what, we need to switch to a new stack.  A goroutine stack is often very small--as small as 128 bytes.  No C function can be expected to run on a stack that small.  So we must always change stacks.
And, of course, not matter what, we must pass the arguments using the C ABI.
But at this point--notify the scheduler, change stacks, pass C ABI arguments--we're pretty much doing everything that cgo does today.  So #cgo nonblocking isn't saving us much time, if any.

Ok, now I understand the implications better. It's clear blocking could cause serious performance problems for the entire process - however, it's still not clear to me that it can deadlock, except in very unusual situations that don't arise in a typical Go program - I can live with that.
If #cgo nonblocking is implemented it needs to be performant, as you say. Changing stacks is unavoidable and passing arguments with the C ABI is unavoidable (but a lot easier for the compiler to do efficiently.) The interaction with the scheduler is the part that would have to be dropped.
I do know that the Go runtime makes exactly this tradeoff when it calls the TSAN runtime functions. They're known not to block, so that's considered safe enough and worth it for the 50% speedup. I have similar needs with an in-memory database, and I'm confident that those functions don't block, except maybe in a once-in-a-blue-moon scenario. I can live with that if it doesn't cause deadlocks in any real scenario.

Basically, the main problem is that calling C functions from Go is slow, and has only become slower. This unfortunately makes Go less suitable for graphical applications and games that have to call into a C OpenGL or Vulkan API many times per second. I don't have any great ideas on how to solve this problem, but I insist that it is something that needs to be improved.

@aclements mentioned him (or somebody) making defers allocated on the stack in Go 1.8. That is #14939. That will help with this bug too, since each cgo call does 2+ defers, IIRC.

There seems to be two things that can be done here.

Speed up existing CGO calls
Add an additional way of calling a well-behaved restricted subset of C functions

This issue should be about the second solution only.
There are definitely things that can be done for (1). That will shrink the set of programs which need solution (2). However (1) will always leave performance on the floor because it has to do extra work to handle not well-behaved C calls.
Now maybe CGO can be sped up enough that the performance difference is so small that the set of programs that would benefit from (2) is too small to be worth the effort of implementing and maintaining it. Or maybe it is deemed too dangerous to be used correctly - but then I question why the Go runtime uses it for the race detector.

I don't think one can base any arguments on the race detector, which is a special case.  It calls into the support library for every memory read or write.  That is vastly more calls than even the most active cgo using program can possibly do.

I'll grant that, but I'm still not content to leave that much performance on the floor. I'm going to try the race detector C calling approach and see how much of a difference it makes in my use case. I'll report back in a couple weeks with some numbers hopefully.
A faster CGO mechanism allows calling C functions that do less work. e.g. if the C function takes 200ns and we speed up the CGO mechanism to 30ns (which seems to have been true in past Go versions), then the total calling time for that function goes from 400 to 230ns, almost twice as fast. If the C function itself takes 30ns, then it goes from 230ns to 60ns, which is a four fold improvement. That opens up more options when designing APIs.
I just spent two days implementing a function in assembly in Go because it needed popcnt, prefetch, and bsr. It could have been implemented in two hours in C, but the CGO overhead makes it a non-starter. In other places I duplicate code between C and Go (using very unidiomatic Go with lots of unsafe) to avoid the CGO overhead.

A C function that took 30ns is, optimistically, at most 50 machine
instructions, assuming the code cache is hot and the code in question
performs only memory accesses that are already in the L1 data cache.
What work can realistically be done in that few number of cycles that
cannot be rewritten in Go?
On Fri, 1 Jul 2016, 10:15 Daniel Eloff<EMAIL_ADDRESS>wrote:

I'll grant that, but I'm still not content to leave that much performance
on the floor. I'm going to try the race detector C calling approach and see
how much of a difference it makes in my use case. I'll report back in a
couple weeks with some numbers hopefully.
A faster CGO mechanism allows calling C functions that do less work. e.g.
if the C function takes 200ns and we speed up the CGO mechanism to 30ns
(which seems to have been true in past Go versions), then the total calling
time for that function goes from 400 to 230ns, almost twice as fast. If the
C function itself takes 30ns, then it goes from 230ns to 60ns, which is a
four fold improvement. That opens up more options when designing APIs.
I just spent two days implementing a function in assembly in Go because it
needed popcnt, prefetch, and bsr. It could have been implemented in two
hours in C, but the CGO overhead makes it a non-starter. In other places I
duplicate code between C and Go (using very unidiomatic Go with lots of
unsafe) to avoid the CGO overhead.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/16051#issuecomment-229823348, or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA864gdwx_X3mtM1OEZFUon2m5hBaks5qRFwogaJpZM4I0hlW
.

@eloff, there's been some discussion (though no concrete proposals to my knowledge) to add functions for things like popcount and bsr that would be compiled as intrinsics where supported (like math.Sqrt is today). With 1.7 we're trying this out in the runtime, which now has a ctz intrinsic available to it on amd64 SSA. Obviously that doesn't solve the overall problem, but it would chip away at one more reason to use cgo in a low-overhead context.

@aclements That sounds very interesting, I'm looking forward to that! You're right, it would eliminate another reason for calling lightweight C functions from Go.

Well, it would be useful, but it's a C world, and certainly for programming
graphically intensive games, I need to call into C Apis very frequently.
All that stack swapping adds up a lot.
I think both approaches are needed, cgo calls need to be made faster in
general, but a way to mark certain C functions as special cases could
equally be useful.
To throw out a wild idea if myself, maybe consecutive calls into C could be
grouped together and use the same C stack without swapping back. This would
probably require a cgo directive...
On 1 Jul 2016 9:19 pm, "Daniel Eloff"<EMAIL_ADDRESS>wrote:

@aclements https://github.com/aclements That sounds very interesting,
I'm looking forward to that! You're right, it would eliminate another
reason for calling lightweight C functions from Go.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/16051#issuecomment-230027505, or mute
the thread
https://github.com/notifications/unsubscribe/AAEWeUDyOm9kqAgwrqBzfLHRe45-llHRks5qRWg4gaJpZM4I0hlW
.

To throw out a wild idea if myself, maybe consecutive calls into C could be grouped together and use the same C stack without swapping back.

How is that different from writing a function in C that makes those calls in sequence?

The difference is that I would like to do it in go, not C. Sure, I could
write a low level game engine in C and call into that from go. But that
defeats the purpose of using go in the first place.
I tried to write a 2d game engine in go a few years ago, but the cgo
overhead was too much. So I was forced to go back to C, and since all the
logic was in C, there wasn't much reason left to use go for that project.
Such a C block in go as I propose, would be a bit like extern "C" in C++,
you could still use a subset of go while handling the lower level API.
On 1 Jul 2016 9:45 pm, "Bryan C. Mills"<EMAIL_ADDRESS>wrote:

To throw out a wild idea if myself, maybe consecutive calls into C could
be grouped together and use the same C stack without swapping back.
How is that different from writing a function in C that makes those calls
in sequence?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/16051#issuecomment-230032467, or mute
the thread
https://github.com/notifications/unsubscribe/AAEWeTUDLzs0Fs-SmywjjLMualwbqWF9ks5qRW5ggaJpZM4I0hlW
.

Another gamedev here supporting this proposal. As it stands now, Go would be really a good candidate for a gamedev language, the GC is getting really fast and the language has a lot to offer to the industry.
Sadly the cgo performance make it a no starter at the moment. I keep wondering how this cost is also impacting other areas like server applications with heavy UDP traffic. I am using Go for the server in my game and looking at the code it eventually ends up calling various WSA* functions on Windows.. so there you go, that price is there even when developing non strictly games software.
Perhaps on Linux there is a difference between a syscall and a cgo call, on Windows it does not seem to be the case tho.
I wonder if, instead of tagging cgo calls we could tag the goroutine to use some kind of "special" stack and get ad-hoc treatment from the scheduler to be able to call into C with as close to 0 cost as possible.
I do think this could improve the performance of the entire tech, I agree with rewriting as much as possible in Go, that's fine, but it's still a world of C based operating systems (and it will be for the foreseeable future)  and, eventually, you'll have to call C stuff in order to make things happen.. be it a triangle on the screen or a packet over the network.

Our current plan is to start by optimizing the existing cgo path and see how far we can get with that before we consider introducing a new mechanism (particularly one that could introduce new classes of bugs).

I keep wondering how this cost is also impacting other areas like server applications with heavy UDP traffic.

I'm not sure, but it might be possible to further optimize Windows system calls beyond the regular cgo path. Most other platforms don't use cgo for syscalls, and there's much less overhead. OTOH, Windows syscalls can have callbacks, so the generality may be necessary.

As the OP, I applaud that strategy. I still think a specialized mechanism
may be needed, but it seems relatively easy to roll my own. If I supply my
own C stacks then I don't need any fragile hooks into the go runtime or
scheduler, just a little assembly code. So if it's still a problem
afterward, there is an option that just requires a little effort - and I
think for a potentially unsafe and advanced feature that's even desirable.
On Sat, Jul 9, 2016 at 3:57 PM, Austin Clements<EMAIL_ADDRESS>wrote:

Our current plan is to start by optimizing the existing cgo path and see
how far we can get with that before we consider introducing a new mechanism
(particularly one that could introduce new classes of bugs).
I keep wondering how this cost is also impacting other areas like server
applications with heavy UDP traffic.
I'm not sure, but it might be possible to further optimize Windows system
calls beyond the regular cgo path. Most other platforms don't use cgo for
syscalls, and there's much less overhead. OTOH, Windows syscalls can have
callbacks, so the generality may be necessary.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/16051#issuecomment-231560365, or mute
the thread
https://github.com/notifications/unsubscribe/AABrtk08THykI6Q2OnEp9wXn7RlUcgkoks5qUCdagaJpZM4I0hlW
.

I keep wondering how this cost is also impacting other areas like server applications with heavy UDP traffic.

I think syscall cost is not significant here. But I might be wrong. If you have some real numbers to show one way or other, we could try and make it faster.

Windows syscalls can have callbacks, so the generality may be necessary

Very few Go windows apps use callbacks - the ones who do call syscall.NewCallback. We could optimize the app if it doesn't calls syscall.NewCallback.
Alex

Actually I did some tests right now, calling a function from a dll, using CGO ends up being faster than using syscall.Syscall to call the same function.. on Windows.
You are right tho about the UDP thing, I did some actual tests/calculations and with a rate of about 20K packets/second I am looking at a 2-3ms overhead to call into Winsock so nothing to be worried about at all.

calling a function from a dll, using CGO ends up being faster than using syscall.Syscall to call the same function.. on Windows.

That is surprising to me. Do you mind providing instructions to reproduce what you see? Thank you.
Alex

Sure, this is the relevant part of the code:
/*
#include <windows.h>
#include <mmsystem.h>
#cgo LDFLAGS: -lwinmm
*/
import "C"
var (
winmm, _        = syscall.LoadLibrary("winmm.dll")
ftimeGetTime, _ = syscall.GetProcAddress(winmm, "timeGetTime")
)
func tgt() uint32 {
ret, _, _ := syscall.Syscall(ftimeGetTime, 0, 0, 0, 0)
return uint32(ret)

}
func testCGOCall() {
v := uint32(0)
for i := 0; i < 20000; i++ {
	v += uint32(C.timeGetTime())
}

}
func testSyscallCall() {
v := uint32(0)
for i := 0; i < 20000; i++ {
	v += tgt()
}

}
And this is the result on my PC:
Syscall: 4.0056ms
CGO: 3.0032ms

And this is the result on my PC:
Syscall: 4.0056ms
CGO: 3.0032ms

Thank you for the source. But I don't think you run test for long enough. I did
c:\dev\src\issues\issue16051>type foo.go
package issue16051

import (
        "syscall"
)

/*
#include <windows.h>
#include <mmsystem.h>

#cgo LDFLAGS: -lwinmm
*/
import "C"

func Call_timeGetTime_CGO() uint32 {
        return uint32(C.timeGetTime())
}

var (
        winmm        = syscall.MustLoadDLL("winmm.dll")
        ftimeGetTime = winmm.MustFindProc("timeGetTime")
)

func Call_timeGetTime_Syscall() uint32 {
        ret, _, _ := syscall.Syscall(ftimeGetTime.Addr(), 0, 0, 0, 0)
        return uint32(ret)
}

c:\dev\src\issues\issue16051>type foo_test.go
package issue16051_test

import (
        "testing"

        "issues/issue16051"
)

func BenchmarkTimeGetTimeSyscall(b *testing.B) {
        for i := 0; i < b.N; i++ {
                issue16051.Call_timeGetTime_Syscall()
        }
}

func BenchmarkTimeGetTimeCGO(b *testing.B) {
        for i := 0; i < b.N; i++ {
                issue16051.Call_timeGetTime_CGO()
        }
}

c:\dev\src\issues\issue16051>go test -run=none -bench=.
testing: warning: no tests to run
BenchmarkTimeGetTimeSyscall     10000000               228 ns/op
BenchmarkTimeGetTimeCGO         10000000               229 ns/op
PASS
ok      issues/issue16051       5.087s

c:\dev\src\issues\issue16051>

and I see not much difference between syscall and cgo.
Alex
PS: On Windows you cannot even measure it if it is less than 15ms

Well I get the same results increasing the loop count:
Syscall: 2.5059244s
CGO: 2.2626265s
Perhaps I have a weird PC.
And no, using the right timer on Windows you can get reasonable results even for <1ms .

I prepared a self-contained benchmark that highlights real-world cgo performance issues here: https://github.com/rasky/tdb-cgo-bench. It's an example that uses the cgo wrapper of TrailDB, which highlights a real-world case where the overhead of cgo is very high. The included benchmark shows a 6x performance regression when executing the same basic code in Go vs C. I hope this can be useful while working on cgo performances in the 1.8 cycle.

@rasky, that's very helpful. Thanks! I was able to reproduce the 6x slowdown locally with your benchmark. I took a glance at the profile; much like other high-overhead cgo benchmarks (#9704), it spends a lot of time dealing with defers (~28%). We have a plan to fix that (#14939). Much of the remaining overhead goes into entersyscall/exitsyscall, which, unfortunately, are much trickier to optimize.

Let's revisit this once the work to speed up defer is complete.

We are going to decline this specific proposal.  Marking some cgo calls as non-blocking can fail in too many subtle ways that are hard to understand.  We are certainly interested in speeding up cgo calls in general, but this approach is not the one we will take.
Note that we have sped up cgo calls in 1.8 in https://golang.org/cl/30080.

@ianlancetaylor what about seperating C and Go in two processes which connected with pipe or mmap, and make a C-call is translate to send a msg to C-process and waiting for a response.
This way can make C-code run faster and keep Go runtime happy.

@linkerlin this proposal was declined eight months ago. Please do not continue the conversation on closed issues. Please see https://golang.org/wiki/Questions for good places to ask. Thanks.

@linkerlin, putting them in separate processes would make it hard to pass pointers, which is one of the main features of cgo. But even putting them in separate threads in the same process wouldn't help with the overhead: the transition would still have to interact with the Go scheduler and then it would also have to interact with the OS scheduler. Nonetheless, if an application wanted to split up its Go and C components this way, it could certainly do so without runtime help.

I think it will be better if we can control blocking/nonblocking at each invoking point.
For example, say the write system call, if it is writing to a disk file, it will not block, but if it writes to a socket/pile, it may be blocked.
Maybe we can use some syntax like this:
C.write(fd, data, size)  // will switch thread
C.nonblocking.write(fd, data, size)  // will not switch thread

@chen3feng This issue is closed.  Comments on closed issues are not tracked.  Please open a new issue or use a forum; see https://golang.org/wiki/Questions .  Thanks.
I think it is extremely unlikely that we would ever let the programmer pick whether a cgo call blocks or not, as that is very easy to get wrong, and getting it wrong may cause the entire program to freeze.  That said, please do not reply here.


Basic e2e server & delete function added to models

Coverage decreased (-1.07%) to 88.679% when pulling 57e3baa26ecc9dd1a1017e2ab267c21466e59774 on e2e into c092a80747a197a6d6c73f03882f93c48046033f on master.

Coverage decreased (-0.2%) to 89.572% when pulling d98da082dc0fd473442be72af863316c26bd169b on e2e into c092a80747a197a6d6c73f03882f93c48046033f on master.


Be clear about the error when the request is missing required data

I tried to word it so that its not just missing data from the body
but should also cover cases where the URL is missing requird path
components (like instance_id)
Closes #274
Signed-off-by: Doug Davis<EMAIL_ADDRESS>

Looks good but Travis has failed.

weird, I could have sworn I ran the checker - must not have. Anyway, fixed now - thanks for the poke.

LGTM

lgtm

2 more reviews needed

LGTM

One more @pmorie @fmui

@shalako @pmorie @zrob @vaikas-google one more review needed

LGTM


Fix  wrong recognition for  "from three thirty to four thirty"

Fix cases like "from three thirty to four thirty"  which would be wrong extracted in DatePeriodExtractor.

This PR is a temporary fix, and that MatchYearPeriod will be refactored later.


[stable/prometheus-operator] Bump version dependency

Signed-off-by: Ryan Orth<EMAIL_ADDRESS>What this PR does / why we need it: Updates the grafana dependency of the prometheus-operator helm chart
Which issue this PR fixes
Bump version dependency charts:
grafana
Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] title of the PR contains starts with chart name e.g. [stable/chart]

/assign @gianrubio

/ok-to-test
/assign

hey @gianrubio or @davidkarlsen -  would you mind approving this review when you get the chance? thanks!

/lgtm

hey @davidkarlsen , could I get another lgtm? someone else's changes got merged in before mine for the prometheus-operator chart so  I needed to do another version bump. thanks again!

/lgtm


Allow (reduced) test suite also without MATLAB

Maybe through switching to c++ shared library?

closed by c35f0d84d4a23842c14bfc8a6111c4987f5481ed


Add Game using Terminal.gui called "Capital and Cargo" to showcase.md

My 12 year old son and I built a game using Terminal.gui.
I believe it would be a good candidate to be on the Showcase list of this project.
If you agree, please approve this pull request, but if you don't, that is also fine.
Changes
Add https://github.com/dhorions/Capital-and-Cargo to showcase.md
Pull Request checklist:

[ ] I've named my PR in the form of "Fixes #issue. Terse description."
[X] My code follows the style guidelines of Terminal.Gui - if you use Visual Studio, hit CTRL-K-D to automatically reformat your files before committing.
[X] My code follows the Terminal.Gui library design guidelines
[ ] I ran dotnet test before commit
[ ] I have made corresponding changes to the API documentation (using /// style comments)
[X] My changes generate no new warnings
[X] I have checked my code and corrected any poor grammar or misspellings
[X] I conducted basic QA to assure all features are working

WOW!
I love this. I only played for a few minutes, but it's addicting.
Kudos to your son (and you)!
It would be killer if you could port this to v2 asap so we have another real-world test case of all the new APIs.

It would be killer if you could port this to v2 asap so we have another real-world test case of all the new APIs.

I haven't looked into the differences with v2, will it be a lot of rework you think?
Is there some kind of conversion documentation?

It would be killer if you could port this to v2 asap so we have another real-world test case of all the new APIs.

I haven't looked into the differences with v2, will it be a lot of rework you think? Is there some kind of conversion documentation?

Actually, if you're interested maybe what I'll do is submit a PR to your repo and do the porting myself.
This will help me flush out the docs!

Love it!

It would be killer if you could port this to v2 asap so we have another real-world test case of all the new APIs.

I haven't looked into the differences with v2, will it be a lot of rework you think? Is there some kind of conversion documentation?

Actually, if you're interested maybe what I'll do is submit a PR to your repo and do the porting myself.
This will help me flush out the docs!

That is a great idea. What do you think @dasmehe

I

It would be killer if you could port this to v2 asap so we have another real-world test case of all the new APIs.

I haven't looked into the differences with v2, will it be a lot of rework you think? Is there some kind of conversion documentation?

Actually, if you're interested maybe what I'll do is submit a PR to your repo and do the porting myself.
This will help me flush out the docs!

That is a great idea. What do you think @dasmehe

I like it!


inlay hints toggle option

As it stands right now editor.inlayHints.enabled has the options on, onUnlessPressed, offUnlessPressed and off.
I think there should also be a toggle option that changes between on and off every time the hotkey (ctrl+alt) is pressed. Or maybe a command like rust-analyzer's (now gone) toggle inlay hints.

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!

@Calcoph
try this way
https://marketplace.visualstudio.com/items?itemName=rebornix.toggle
    {
        "key": "ctrl+alt+h",
        "command": "toggle",
        "when": "editorTextFocus",
        "args": {
            "id": "inlayHints",
            "value": [
                {
                    "editor.inlayHints.enabled": "on"
                },
                {
                    "editor.inlayHints.enabled": "off"
                }
            ]
        }
    },


fixing appenine logic

Signed-off-by: barinek<EMAIL_ADDRESS>Fixes #12

Codecov Report

Merging #147 (98a87ce) into main (5f249e9) will increase coverage by 0.19%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #147      +/-   ##
==========================================
+ Coverage   91.61%   91.81%   +0.19%
==========================================
  Files          38       38
  Lines        1539     1539
==========================================
+ Hits         1410     1413       +3
+ Misses         86       85       -1
+ Partials       43       41       -2

Impacted Files
Coverage Δ

...orchestratorproviders/googlecloud/google_client.go
100.00% <100.00%> (+4.47%)
:arrow_up:

...atorproviders/googlecloud/google_cloud_provider.go
96.36% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f249e9...98a87ce. Read the comment docs.


spec/dummy/README missed package name change

This change is

Coverage remained the same at ?% when pulling 9c3392669c9abca6f5bb049355d9bb3f2a86413a on ened:patch-2 into d364ebf5f69afd6a228976303e1d1f76021cbf8e on shakacode:master.

Coverage remained the same at ?% when pulling 9c3392669c9abca6f5bb049355d9bb3f2a86413a on ened:patch-2 into d364ebf5f69afd6a228976303e1d1f76021cbf8e on shakacode:master.

https://github.com/shakacode/react_on_rails/blob/master/spec/dummy/client/package.json#L70
Let's mention this is done in the postinstall script.

Thanks @ened. We can apply my suggestion in another PR if you like.


DS install broken: Drush terminated due to New Relic PHP Library missing

bin/ds build --install andbin/ds build --intl --install  fails with the following error:
New Relic PHP Library missing. Check Install New Relic for PHP for more information. in install_verify_requirements() (line 768 of /var/www/vagrant/html/includes/install.core.inc).
Drush command terminated abnormally due to an unrecoverable error.

/CC: @DeeZone @aaronschachter

Easy quickfix here could be to remove the New Relic module from the dosomething.info file, so we don't install it by default.

@aaronschachter  But why can't it find it? Isn't it in drupal-org.make?

No, the make file is not adding the PHP Library: https://docs.newrelic.com/docs/agents/php-agent/getting-started/new-relic-php
@mshmsh5000 manually added it for production.

Ok, I will install this and include into salt-platform.

@sergii-tkachenko Is this closed by https://github.com/DoSomething/salt-platform/pull/11 ?

@aaronschachter Yes, it's closed. Thank you!


I can't acces to session from template twig

Perhaps this problem is caused by my environnement, but I can't acces to my session from my template.
When I add
{{dump(app.session)}}
I got "null" response...
Is it normal on sulu ? Or you think that I have a problem in my configuration ?
Thanks !
Yoann

On a default installation on the website no session is started. Also you should keep in mind that all pages are response default as public and will be cached.
If you are on a page template I recommend to keep the page cache and use an ESI (render_esi). In the esi controller you could start or access the session and set its response to private ($response->setPrivate();) . More about esi's in symfony http://symfony.com/doc/current/book/http_cache.html#using-esi-in-symfony.

@Yoann-TYT Maybe the session is not started else I cant help you because on my installation it works without problem also by accessing it directly in twig, but I would not recommend you to pass the whole session to twig, I would just pass the parameters so your template can be reuse. And pls use the github issue template next time!
if (!$request->getSession()->isStarted()) {
    $request->getSession()->start();
}

Trank you Alexander ;-)
I will use the github issue template next time !
And for my problem, I think it's a configuration problem... I tell you if I solve it. ( We can close this ticket if you want ! )

Ok thx let us know how you solved it.


Unable to register host on Ubuntu 16.04 LTS

Rancher Version: 1.0.1
Docker Version: 1.10.3
OS: Ubuntu 16.04 LTS Desktop
Steps to Reproduce:

Launch rancher server,
Launch rancher host on the same machine with properly configured IP.

Results:
Unable to launch the agent to register a host, it fails fast with the following logs:
Error response from daemon: 400 Bad Request: malformed Host header
Expected:
It should register the local host properly.

It appears the Ubuntu-packaged Docker was compiled with Go 1.6, for no obvious reason.  This breaks communication with older clients, Docker switched and there was a whole thread that resulted in them reverting back: https://github.com/docker/docker/issues/20865
So you will need to use the docker binaries from Docker, not Ubuntu, until they fix it.

Thanks Vincent for the prompt answer! Going to see how to install Docker binaries, as their get.docker.com script is not working for some unknown reason.

Unfortunately, it seems like there is an issue with installing Docker on Ubuntu Xenial using their APT repository:
https://github.com/docker/docker/issues/22182
The issue should be fixed in Docker 1.11.1.

The solution can be to switch the repository to testing (instead of main) in docker.list, it installs 1.11.0-rc5 at the moment.

@jloisel sorry for the issue, it's fixed now on our end.

Thanks @tiborvass

@tiborvass Thanks!

but this means docker 1.11 installation which is as far I know not supported my rancher?

"supported" is relative.. the main thing that doesn't work is cAdvisor/stats graphs, but we can't make Docker add 1.10 to their Xenial repo.
Also I put in a bug with Ubuntu and they pulled in a PR based on what @ibuildthecloud suggested to support the Host header in Go 1.6, so the packaged 1.10.3 should work now:  https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1574904

i hit the same bug.
I'm using gentoo distribution.
docker 1.11.0
go 1.6
in gentoo we only have go 1.6.
https://packages.gentoo.org/packages/dev-lang/go
what i should do? wait?
by the way, your product is awesome. I tried in windows docker-toolbox and is amazing.
thanks for your work.

@mercuriete If you can't downgrade to Docker 1.10.3, then you can follow this issue on when we add in Docker 1.11 support for Rancher. It will definitely be available for v1.1, which is scheduled for mid June.
https://github.com/rancher/rancher/issues/4491

@deniseschannon thank you very much for the answer. I am only testing your product so i can wait until june or test it on a virtual machine.
PS: i cant downgrade to 1.10.3 because on gentoo we only have docker 1.11.0
https://packages.gentoo.org/packages/app-emulation/docker
your product is awesome :)


failed to start satellite.service - Bitfocus Companion Satellite.

Hey there,
I've got the following Problem.
I'v flashed the Imagepi to an SD Card and run it on an Raspberry Pi 3 Model B.
it seems to run, but have got a Problem.
Can you help me to solve the Problem?
Thank you very much and best Regads

This should be fixed in the v1.6.1 image

Thank you very much!


Youtube: Only downloading last 2 hours of 3 hour video. (with hack as a workaround)

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2020.07.28
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log
$ youtube-dl --verbose https://www.youtube.com/watch?v=7q2E_dMf-PA -f 160+139 -o test.mp4
[debug] System config: ['--prefer-free-formats']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.youtube.com/watch?v=7q2E_dMf-PA', '-f', '160+139', '-o', 'test.mp4']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.05.29
[debug] Python version 3.8.5 (CPython) - Linux-5.7.12-200.fc32.x86_64-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4
[debug] Proxy map: {}
[youtube] 7q2E_dMf-PA: Downloading webpage
[youtube] 7q2E_dMf-PA: Downloading m3u8 information
[youtube] 7q2E_dMf-PA: Downloading MPD manifest
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1597463882/ei/6Qg3X-PLPOTpiwTDgZGYCw/ip/[REDACTED]/id/7q2E_dMf-PA.0/source/yt_live_broadcast/requiressl/yes/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/beids/23886217/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhAIYKJYy_vhL6FN2sN8JmwtZUzthfIEC1uyLBsWLFRF1-AiBSi_MHyF2FME9Q96eTwBDWUYjupTJnp_MoVJcxOVW74Q%3D%3D'
[dashsegments] Total fragments: 3600
[download] Destination: test.f160.mp4
[download] 100% of 104.01MiB in 29:48
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1597463882/ei/6Qg3X-PLPOTpiwTDgZGYCw/ip/[REDACTED]/id/7q2E_dMf-PA.0/source/yt_live_broadcast/requiressl/yes/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/beids/23886217/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhAIYKJYy_vhL6FN2sN8JmwtZUzthfIEC1uyLBsWLFRF1-AiBSi_MHyF2FME9Q96eTwBDWUYjupTJnp_MoVJcxOVW74Q%3D%3D'
[dashsegments] Total fragments: 3600
[download] Destination: test.mp4.f139
[download] 100% of 64.95MiB in 30:09
[ffmpeg] Merging formats into "test.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i file:test.f160.mp4 -i file:test.mp4.f139 -c copy -map 0:v:0 -map 1:a:0 file:test.temp.mp4
Deleting original file test.f160.mp4 (pass -k to keep)
Deleting original file test.mp4.f139 (pass -k to keep)

Description
On that page,the youtube player in firefox shows the video is 3:01:49 in length, and plays fully at that length. While youtube-dl downloads and exits cleanly, it only downloads the last 2:00:00 of it. This happens with both the dash and hls manifests I've tried.
$ ffmpeg -i test.mp4 -f null -
...
frame=216000 fps=8600 q=-0.0 Lsize=N/A time=02:00:00.00 bitrate=N/A speed= 287x
...

Is there a workaround or proper method to grab the full video which is available to play on the site?
This is likely related to issue #26290.
Edit: the output is from an older youtube-dl, but the results are the same on my boox with v. 2020.07.28
Edit 2: streamlink also gets only 2 hours from hls. How does youtube player do it?!
Edit 3: Well, I'm not sure what the youtube player is doing, but I was able to hack the missing fragments into place...
First, I wrote out the info.json for the url.
Then I edited the json and found the first fragment in the stream:
      "fragments": [
        {
          "path": "sq/1855/lmt/1597197498473364",
          "duration": 2
        },

Then I generated all the fragments from 0 to 1854:
$ for F in {0..1854} ; do echo -e "\t{\n\t  \"path\": \"sq/${F}\",\n\t  \"duration\": 2\n\t}," ; done > missing.txt
$ head -n 10 missing.txt
	{
	  "path": "sq/0",
	  "duration": 2
	},
	{
	  "path": "sq/1",
	  "duration": 2
	},
	{
	  "path": "sq/2",

I then inserted these missing fragments before fragment 1855 in the json and saved it.
Then I was able to run youtube-dl against this .json with:
youtube-dl  -f 139 --load-info-json test.info.json -o test.m4a

And got the full length stream saved, as the fragments are on the server, just not enumerated in the manifest.
It would seriously suprise me if this is what the web player is doing to get the full stream.
Maybe the youtube plugin can check to see if fragments start at 0, if not check if 0 is on the server, and compensate for the incomplete manifest as a work around?

Youtube is definitely up to something new as this phenomenon is something I haven't seen before and it started about 5-6 days ago? As most of you know livestreams used to only display the last 2 hours even on the webpage for X amount of time until they got processed into a VOD.
Now it seems like the videos are instantly viewable on the webpage from start to finish with some missing features(Chat replay). However Youtube-dl only seems to grab the last 2 hours regardless.
I might be repeating what has already been said but since the "Processing into a VOD" could be different from person to person depending on your region, I wanted to throw out that I'm experiencing the exact same thing, which likely everybody is.

Same issue here.
$ command youtube-dl --version
2020.07.28

$ command youtube-dl --verbose 'https://www.youtube.com/watch?v=Mm0KCzYpMhQ'
[debug] System config: []
[debug] User config: ['--ignore-errors', '--no-mtime', '--console-title']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.youtube.com/watch?v=Mm0KCzYpMhQ']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.8.3 (CPython) - Linux-5.7.9-arch1-1-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] Mm0KCzYpMhQ: Downloading webpage
[youtube] Mm0KCzYpMhQ: Downloading m3u8 information
[youtube] Mm0KCzYpMhQ: Downloading MPD manifest
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1597700011/ei/S6M6X_zOHpOcs8IPi7SF-Ak/ip/50.999.999.85/id/Mm0KCzYpMhQ.1/source/yt_live_broadcast/requiressl/yes/tx/23908007/txs/23908006%2C23908007/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Ctx%2Ctxs%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIgaBMCXhRtrGe2SeAHys3agvoV10DHyPqygiCOG-_PfjcCIQCBWI3JoZSHDc-Eu5xkAU2Xi_Jpll4aYzVP1m4HEE0g3A%3D%3D'
[dashsegments] Total fragments: 3600
[download] Destination: 【APEX耐久】ダイアモンドになるまで終わらないラストバトル！！【湊あくあ】-Mm0KCzYpMhQ.f299.mp4
[download] 100% of 3.01GiB in 10:12
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1597700011/ei/S6M6X_zOHpOcs8IPi7SF-Ak/ip/50.999.999.85/id/Mm0KCzYpMhQ.1/source/yt_live_broadcast/requiressl/yes/tx/23908007/txs/23908006%2C23908007/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Ctx%2Ctxs%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIgaBMCXhRtrGe2SeAHys3agvoV10DHyPqygiCOG-_PfjcCIQCBWI3JoZSHDc-Eu5xkAU2Xi_Jpll4aYzVP1m4HEE0g3A%3D%3D'
[dashsegments] Total fragments: 3600
[download] Destination: 【APEX耐久】ダイアモンドになるまで終わらないラストバトル！！【湊あくあ】-Mm0KCzYpMhQ.f140.m4a
[download] 100% of 151.43MiB in 06:39
[ffmpeg] Merging formats into "【APEX耐久】ダイアモンドになるまで終わらないラストバトル！！【湊あくあ】-Mm0KCzYpMhQ.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:【APEX耐久】ダイアモンドになるまで終わらないラストバトル！！【湊あくあ】-Mm0KCzYpMhQ.f299.mp4' -i 'file:【APEX耐久】ダイアモンドになるまで終わらないラストバトル！！【湊あくあ】-Mm0KCzYpMhQ.f140.m4a' -c copy -map 0:v:0 -map 1:a:0 'file:【APEX耐久】ダイアモンドになるまで終わらないラストバトル！！【湊あくあ】-Mm0KCzYpMhQ.temp.mp4'
Deleting original file 【APEX耐久】ダイアモンドになるまで終わらないラストバトル！！【湊あくあ】-Mm0KCzYpMhQ.f299.mp4 (pass -k to keep)
Deleting original file 【APEX耐久】ダイアモンドになるまで終わらないラストバトル！！【湊あくあ】-Mm0KCzYpMhQ.f140.m4a (pass -k to keep)

The result was the exactly last 2 hours of the 6:54:07 video. Using Firefox for Linux or YouTube app for Android, the video can be played without any problems though Chat Replay hasn't been made available yet.

 No matter bitrate, both 139 and 140 have a duration of 2:00:00 while the video is exactly 3:17:51 of length.

Same problem, but it seems like the problem disappears after up to 2 days since the live stream has been finished (please mind it when trying to reproduce the issue).

OP of the linked issue here (on a new account, don't worry about it I'm just dumb...). Yes, they are basically the same issue. Or rather, your issue is more current because YouTube has literally changed how they handle livestream recordings in the past week. When I posted my issue, livestream recordings were clipped on the YouTube site just like what youtube-dl is giving until the processing finishes. Now the full video is available to browser users but youtube-dl is still somehow finding the clipped one. So I agree this now looks like a genuine bug.
(As an aside, this is not the first time the full recording has been available to regular desktop browser users. There was a period of about a week a month or two ago where I could consistently get the full video, but only in chrome, not firefox. That was possibly a trial run for what we're seeing now. After that period it went back to the 2 hour clipped recording that has been standard for years.)
The work-around you posted by editing the json to include the missing fragments is very neat but without trying a bunch of cases I'm not sure how robust it is. If you would rather just filter these videos and wait for processing to occur (for people with relatively automated workflows), I've had success with using filesize after duration broke. Specifically, the filesize shows as null or NA for fragmented videos but will give the real size after processing. This might break some random old video somewhere though so use it with caution.
As a word of warning for anyone else this comes up for, I've run into some very weird cases experimenting with youtube-dl on unprocessed livestreams over 2 hours in the past week. In one case, the video and audio were both downloaded normally (well, slowly, since downloading fragmented files is slow, but no errors or anything), but when they were combined I was left with just a 15 second long video from somewhere in the middle and the audio was gibberish. I also saw a case where when I called youtube-dl, there were literally 0 formats available, like it saw there was a video but it couldn't get any data at all, but subsequent calls couldn't reproduce that result.

I went to grab this livestream 5 hours after it was finished, https://www.youtube.com/watch?v=VMfaFRPGvvM. It downloaded complete with the dash_manifest using youtube-dl.
Interested to see if any changes occurred in the manifest, I wrote out the info.json and inspected it.
The manifest was still in fragments form, but instead of 2 second fragments, it listed 5 second fragments.
This is a change on youtube's end. and might be youtube's way of fixing this problem for us? The previous manifests were showed 2 second fragments with a limit of 3600 fragments (2 hours) before it lost the starting ones.
With 5 second fragments, 3600 fragments would be 5 hours, making the bug only show up on livestreams over 5 hours in length.

@bittaurus

making the bug only show up on livestreams over 5 hours in length?

No. Here is a live archive which has just finished being streamed minutes ago: https://www.youtube.com/watch?v=Esl7kGD5FdE. Although the web player has no problem with it, youtube-dl gives the last 2 hours of 3:26:08, using dash manifest. It occurs even when I use hls manifest:
url=$(curl --silent 'https://www.youtube.com/watch?v=Esl7kGD5FdE' | grep -o 'https:\\/\\/[^"]\+googlevideo.com\\/[^"]\+' | grep 'hls_variant' | sed 's/\\//g')
ffmpeg -i "${url}" out.mkv #=> The same result. The problem is not specific to youtube-dl.

@bittaurus
This behavior existed before; in fact the 2 hour duration is new and was added when YouTube made several improvements to their livestream platform like increasing supported max resolution and adding low latency options. Depending on the settings for the livestream, the fragments can be 1s, 2s, 5s, and maybe other amounts. I have seen the clipped version of the video be a maximum of 2 hours, 4 hours, 5 hours, or maybe more (and it can be less than these if the livestream goes offline and back online again; only the final X hours of the stream, as it was shown in realtime, will be available. So you may see 1:58:30 clipped recordings if there were connection issues, for example). The settings that matter in determining this are (most importantly) the latency setting, the (source) resolution, framerate, and possibly bitrate. Higher settings lead to shorter durations but I do not know exactly what formula YouTube is using. It is not just a fixed number of fragments; you will find 2 hour videos with 3600 fragments or 7200 (2s vs 1s), and a 4 hour clip with 5s fragments is 2880 which I'm pretty sure I've seen.
I have no way of knowing for sure, but it is likely the stream you were looking at was not using low latency settings, and while it is 1080p, it is only 30fps. So being able to get 4 or 5 hours of the video rather than 2 (which was enough for the full video in this case) is not a surprise. That is what you would have seen in a browser as well up until about a week ago.

Here's a quick bash script to insert the fragments into a .info.json file.  It assumes the same number of fragments are missing from every fragmented stream (which is true for every case I've seen so far.)  It relies on gnu seq and gnu sed.
#!/bin/bash
#
# fixfragx [file.info.json]

[ ! -s "${1}" ] && echo -e "Usage: fixfragx [filename.info.json]\n\nInserts missing begining youtube fragments into youtube-dl --write-info-json file." && exit

FRAGS=$(sed -e 's/^.*"fragments": \[{"path": "sq\///' -e 's/\/.*$//' "${1}")
[ "${FRAGS}" == "" ] || [ "${FRAGS}" == "0" ] && echo "no missing fragments foun
d." && exit

((--FRAGS))
echo "missing fragments {0..${FRAGS}} found."

INSERT=$(for F in $(seq 0 ${FRAGS}) ; do echo -n "{\"path\": \"sq/${F}/\"}, " ; done)
sed -i 's_"fragments": \[_"fragments": \['"${INSERT}"'_g' "${1}"

[ "$?" == "0" ] && echo "missing fragments have been inserted." && exit
echo "error with insertion of missing fragments."
exit

Good luck.

Here is a batch script to generate the missing fragments (you first have to replace the first fragment number, the last fragment number and the duration):
@echo off
for /L %%g in (0,1,6222) do (
echo:        {>>fragments.txt
echo:          "path": "sq/%%g",>>fragments.txt
echo:          "duration": ^2>>fragments.txt
echo:        },>>fragments.txt
)

First, I wrote out the info.json for the url.
Then I edited the json and found the first fragment in the stream:
      "fragments": [
        {
          "path": "sq/1855/lmt/1597197498473364",
          "duration": 2
        },

Then I generated all the fragments from 0 to 1854:
$ for F in {0..1854} ; do echo -e "\t{\n\t  \"path\": \"sq/${F}\",\n\t  \"duration\": 2\n\t}," ; done > missing.txt
$ head -n 10 missing.txt
	{
	  "path": "sq/0",
	  "duration": 2
	},
	{
	  "path": "sq/1",
	  "duration": 2
	},
	{
	  "path": "sq/2",

I then inserted these missing fragments before fragment 1855 in the json and saved it.
Then I was able to run youtube-dl against this .json with:

Where you get this data?
Where you get info.json?
Where you found the first fragment?

Checklist

[x]  I'm reporting a broken site support
[x]  I've verified that I'm running youtube-dl version 2020.07.28
[x]  I've checked that all provided URLs are alive and playable in a browser
[x]  I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x]  I've searched the bugtracker for similar issues including closed ones

Verbose log
$ youtube-dl --verbose https://www.youtube.com/watch?v=7q2E_dMf-PA -f 160+139 -o test.mp4
[debug] System config: ['--prefer-free-formats']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.youtube.com/watch?v=7q2E_dMf-PA', '-f', '160+139', '-o', 'test.mp4']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.05.29
[debug] Python version 3.8.5 (CPython) - Linux-5.7.12-200.fc32.x86_64-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4
[debug] Proxy map: {}
[youtube] 7q2E_dMf-PA: Downloading webpage
[youtube] 7q2E_dMf-PA: Downloading m3u8 information
[youtube] 7q2E_dMf-PA: Downloading MPD manifest
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1597463882/ei/6Qg3X-PLPOTpiwTDgZGYCw/ip/[REDACTED]/id/7q2E_dMf-PA.0/source/yt_live_broadcast/requiressl/yes/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/beids/23886217/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhAIYKJYy_vhL6FN2sN8JmwtZUzthfIEC1uyLBsWLFRF1-AiBSi_MHyF2FME9Q96eTwBDWUYjupTJnp_MoVJcxOVW74Q%3D%3D'
[dashsegments] Total fragments: 3600
[download] Destination: test.f160.mp4
[download] 100% of 104.01MiB in 29:48
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1597463882/ei/6Qg3X-PLPOTpiwTDgZGYCw/ip/[REDACTED]/id/7q2E_dMf-PA.0/source/yt_live_broadcast/requiressl/yes/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/beids/23886217/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhAIYKJYy_vhL6FN2sN8JmwtZUzthfIEC1uyLBsWLFRF1-AiBSi_MHyF2FME9Q96eTwBDWUYjupTJnp_MoVJcxOVW74Q%3D%3D'
[dashsegments] Total fragments: 3600
[download] Destination: test.mp4.f139
[download] 100% of 64.95MiB in 30:09
[ffmpeg] Merging formats into "test.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i file:test.f160.mp4 -i file:test.mp4.f139 -c copy -map 0:v:0 -map 1:a:0 file:test.temp.mp4
Deleting original file test.f160.mp4 (pass -k to keep)
Deleting original file test.mp4.f139 (pass -k to keep)

Description
On that page,the youtube player in firefox shows the video is 3:01:49 in length, and plays fully at that length. While youtube-dl downloads and exits cleanly, it only downloads the last 2:00:00 of it. This happens with both the dash and hls manifests I've tried.
$ ffmpeg -i test.mp4 -f null -
...
frame=216000 fps=8600 q=-0.0 Lsize=N/A time=02:00:00.00 bitrate=N/A speed= 287x
...

Is there a workaround or proper method to grab the full video which is available to play on the site?
This is likely related to issue #26290.
Edit: the output is from an older youtube-dl, but the results are the same on my boox with v. 2020.07.28
Edit 2: streamlink also gets only 2 hours from hls. How does youtube player do it?!
Edit 3: Well, I'm not sure what the youtube player is doing, but I was able to hack the missing fragments into place...
First, I wrote out the info.json for the url.
Then I edited the json and found the first fragment in the stream:
      "fragments": [
        {
          "path": "sq/1855/lmt/1597197498473364",
          "duration": 2
        },

Then I generated all the fragments from 0 to 1854:
$ for F in {0..1854} ; do echo -e "\t{\n\t  \"path\": \"sq/${F}\",\n\t  \"duration\": 2\n\t}," ; done > missing.txt
$ head -n 10 missing.txt
	{
	  "path": "sq/0",
	  "duration": 2
	},
	{
	  "path": "sq/1",
	  "duration": 2
	},
	{
	  "path": "sq/2",

I then inserted these missing fragments before fragment 1855 in the json and saved it.
Then I was able to run youtube-dl against this .json with:
youtube-dl  -f 139 --load-info-json test.info.json -o test.m4a

And got the full length stream saved, as the fragments are on the server, just not enumerated in the manifest.
It would seriously suprise me if this is what the web player is doing to get the full stream.
Maybe the youtube plugin can check to see if fragments start at 0, if not check if 0 is on the server, and compensate for the incomplete manifest as a work around?

i did the same but have onli audio.
How can i get full stream with video?

It seems YouTube on browser access the video meta from completely different endpoint currently (most likely directly embedded in the page).
The desktop YouTube web itself did not use the dash manifest nor the hls manifest at all.
The page only contains a the base url and duration of the segment.
And it use JavaScript to compute the segment index against current video time.
And sign it with the same method it used to sign the download url in manifest.
That is probably the reason youtube is able to play the whole vod on browser normally even the dash manifest it returned is incomplete

It seems YouTube on browser access the video meta from completely different endpoint currently (most likely directly embedded in the page).
The desktop YouTube web itself did not use the dash manifest nor the hls manifest at all.
The page only contains a the base url and duration of the segment.
And it use JavaScript to compute the segment index against current video time.
And sign it with the same method it used to sign the download url in manifest.
That is probably the reason youtube is able to play the whole vod on browser normally even the dash manifest it returned is incomplete

Issue is still present =(

Based on the workaround above, I wrote a little Node script to add the missing fragments for each format:
addMissingFragments.zip
To use:

Install a recent version of Node.js
Extract addMissingFragments.js somewhere and open a command window in that location
Run the following commands, substituting the youtube video URL for [youtube-url]:

youtube-dl --output video --skip-download --write-info-json [youtube-url]
node addMissingFragments video.info.json
youtube-dl --load-info-json video.info.json

If you want to download a specific format or add other arguments that affect the output file, add them to the 3rd line (the one that loads the modified json).
Note: This script is very simple and wouldn't be able to handle changes to the fragment path structure.

@ehoogeveen-medweb Thanks, I was able to download a complete 7+ hour MPD livestream by adding the missing fragments using your script.

Many thanks to @ehoogeveen-medweb for the script to add missing segments.
Wrote a simple .bat file to automate the upload process, you need to put the addMissingFragments.js file next to the batch file, run MissingFragments-dl and insert the link
MissingFragments-dl.zip

Thanks for this report. I faced the same issue recently with this live stream.
I would not have noticed the problem if I had not checked the resulting file generated by youtube-dl.
The logs show a very low number of total segments: e.g. [dashsegments] Total fragments: 7159 (which should be around 29000).
It would be nice if Youtube-dl could warn the user, and perhaps allow to force downloading from segment number 0 by implementing the hacks mentioned above.
That video is probably still being processed by YT at the time of writing, so I'll wait for YT to fix itself on this one but that is not ideal for "emergency archivists" (ie. streams that suddenly get deleted by their author without warning).

I guess compute the segment number and insert id yourself isn't technically a "workaround"?
Because youtube actually never generate a full manifest for it.
Even the youtube web page itself compute that by itself.
If you watch it on youtube before fully processed.
The debug information actually says it is operated under a "manifest-less" mode.

youtube-dl --load-info-json video.info.json --format "bestvideo[protocol=http_dash_segments]+bestaudio"

Thank you, this worked!

sss_dg_****

2: Sometimes fragments use ranges instead of indices. Not much can be done about those, so skip those formats. Also try to select the most common duration if there are multi

I tried this command a few times and it doesn't seem to download audio. There's no metadata for audio bitrate or anything, it just isn't there. Is there a way I can specify an audio format with this command? Downloading them separately isn't even working for me right now.

Same problem, but it seems like the problem disappears after up to 2 days since the live stream has been finished (please mind it when trying to reproduce the issue).

That seems very straightforward. But otherwise we need to back-port https://github.com/yt-dlp/yt-dlp/pull/5091.


help: dad third song on hard not working

what does it mean by this

you did a bad (probably not you though) and the file doesnt exist so just rename the file to that and you should be good

i fixed it, was trying to get b-sides working.
had to delete the old data folder in assets and put the kade one

PR #763 fixed this

reminder to close this issue


Bump Rails Translation Manager

⚠ This repo is Continuously Deployed: make sure you follow the guidance ⚠

There's a failing test on this one, @1pretz1


fix(trading): positions tooltips update

Related issues 🔗
Closes #4278
Description ℹ
Improve wording for position resolution indicator tooltip.
Demo 📺

Technical 👨🔧
Details of technical implementation that reviewers may need to be aware of, if applicable.

@barnabee are you good with the text here?

Looks great 👍


VSC loses resolution while I'm working

I'm having a strange problem where my code becomes barely visible while I'm working. I've been having this issue for more than a month already. I've tried updating the graphic drivers but it doesn't work. The program keeps switching from normal to very bad resolution, making the code unreadable.

VSCode Version: 1.44.1
OS Version: Windows 10 Pro

Steps to Reproduce:

Open VSC
Start working with it.

Does this issue occur when all extensions are disabled?: Yes

Does the problem also happen if you launch with the --disable-gpu option?

Does the problem also happen if you launch with the --disable-gpu option?

I haven't tried it but I've found that the following solution worked for me: https://stackoverflow.com/questions/36108515/how-to-reset-settings-in-visual-studio-code
I still don't know what has caused the problem but it seems that I'm no longer experiencing it so I'm closing the issue.


[Bug]: Document shows incorrect diagram

Installation method
Own AWS account
What happened?
https://build-ee0efb6.eksworkshop.com/docs/introduction/getting-started/first screenshot shows activemq instead of rabbitmq
What did you expect to happen?
expect document (including screenshots to be reflecting code repo)
How can we reproduce it?
not a code bug, document needs update
Anything else we need to know?
No response
EKS version
1.23

Diagram was updated in the latest version


Velocity does not work when used in pure SVG context

Hello! Velocity is really nice. Thank you very much for all the effort!
I think I found a small issue. I'm trying to use Velocity inside an SVG image (no HTML context). I've placed a small representative example below (you can also see the image here):
<?xml version="1.0" encoding="utf-8" ?>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" >

  <text id="make_me_move" x="20" y="20">Test</text>

  <script type="text/javascript" xlink:href="http://cdnjs.cloudflare.com/ajax/libs/velocity/1.2.2/velocity.min.js"></script>

  <script type="text/javascript">
    <![CDATA[
      Velocity(document.getElementById("make_me_move"), {y: 200}, 2000);
    ]]>
  </script>
</svg>

It fails with an error in the console, like this (when run without the minified version):

The problem is in line 1509, where it uses the prefix element to check for browser support. When used in a pure SVG context, the style property of the prefix element is null.
I've verified that checking for null before doing that makes it work, in SVG context. Like so:
if (Velocity.State.prefixElement.style !== null && Type.isString(Velocity.State.prefixElement.style[propertyPrefixed])) {

I realize that there are probably cache implications by doing this, and from what I've read, a major reason for Velocity to exist is its caching and performance. However, in SVG context, it does not work without this. So, do you think this is a valid fix? I can submit a PR, if you think so.
PS: Unfortunately, I cannot use CodePen to show this, because CodePen runs all code in HTML context (surrounded by HTML). The div used in that case has the style property.

Thank you so much for this very thoughtful post, Aravind. Much appreciated. Will look into this shortly.


How to redirect back to sign-up form

Hi, I want to use login with facebook in my sign-up form. After authentication from facebook, I do not want to login user, instead I want to take some more input field from user and then user will be able to login. So how can I implement callback function after authentication with facebook pop-up box get closed.

Please use Gitter chat to ask questions.


prettify runner output

Applies formatting to the output of the runner, and prettifies headers and json bodies.

Nice one!


Support externalized webapp URLs in tenant qualified URL mode - Phase 1

In Identity Server, we have the option of externalizing user-facing web apps such as the login portal, recovery portal, and connector endpoints such as TOTP, email OTP and SMS OTP portals.
Currently, we can only define some of these externalized URLs at the server level.
Externalized URL can be absolute URLs residing outside of IS and also custom contexts residing within IS. For example, our default authentication endpoint is https://localhost:9443/authenticationendpoint/login.do
This can be externalized as either,

https://localhost:9443/security/login.do
https://login.somedomain.com/login.do

When it comes to tenant qualified URLs feature, currently we do not have the capability to support externalized web apps. In order to support this,
Phase I
When the tenant qualified URLs feature is enabled

If the URL externalized (or customized) is a relative one (ie. hosted within IS with a different context than default), then build the URL with tenant qualified path (/t/{tenantDomain} in the path)

If the URL externalized is an absolute one, then we do not try to tenant qualify it.

If no URLs are defined from configs, then build the tenant qualified URLs with the default context

A few identified points to fix

[ ] ConfigurationFacade in Identity Framework that provides login, retry and recovery URLs
[x] TOTP https://github.com/wso2/product-is/issues/10900
[x] Email OTP https://github.com/wso2/product-is/issues/10931
[ ] SMS OTP https://github.com/wso2/product-is/issues/10947
[ ] Other connectors that involve redirecting users to a input page

Phase II

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Fix typo in Readme.rst

Thanks for finding this. It is now fixed in the dev branch and will be merged to master soon.


Page setting: Derive name from video or series doesn't (always) work

It works for me,
https://github.com/elan-ev/tobira/assets/1752342/54274f5b-6e92-44d8-a210-dbe995ffd099
but not for my colleague:

Can someone please check whether re-naming a page works reliably?

I've just seen, that this problem only exists at "my page". On sub pages ist works properly.
On my page i add some videos and then i like to derive name from one of these videos. This is not working. "Name directly" is working.

Ahh ok, thanks for the details. This bug is already solved by #860. This is released as v2.1 and I assume you are testing on v2.0. Testing on tobira.opencast.org works for example. So I will close this issue. If you still have problems after updating to 2.1, feel free to reopen.


move cmake finder config files to root dir

Fixes https://github.com/leapmotion/autowiring/issues/525

[ ] only tested on Mac OSX, need to test on other platforms

I'll test on Windows.

Don't forget your labels.

+1


feat: support passing watchfiles.BaseFilter instance

I think the user should be able to create their own reload filter and pass it this way. The current way of customizing reload filters is very limited, while this way removes all limitations.
import watchfiles
from granian import Granian

reload_filter_instance = watchfiles.PythonFilter(extra_extensions=(".html",))
# Let's call it reload_filter_instance to avoid making breaking changes
Granian(..., reload_filter_instance=reload_filter_instance).serve()

I am willing to submit a PR if you approve this change.

For CLI users, we can add the ability to pass the path to the instance by --reload-filter-instance some_package.some_module:my_reload_filter.

Sounds good to me in regards of the Python interface, but for the CLI maybe an alternative approach could be to add the --reload-only-* counterparts of the existing --reload-ignore-* params?
At that point the usage would be smth like --reload-only-patterns *.py --reload-only-patterns *.html..
WDYT @monosans?


ES6 super call issues (because of typescript version used in this plugin) How can I change it?

So this bug has been fixed: https://github.com/Microsoft/TypeScript/issues/4118
But this plugin uses 1.5.3 before it was fixed. How can I use typescript@next with the plugin?

You can override the version of typescript used by this plugin by specifying an overrides entry in your package.json:
    "devDependencies": {
      "css"<EMAIL_ADDRESS>      "ts"<EMAIL_ADDRESS>      "typescript"<EMAIL_ADDRESS>    },
    "overrides": {
<EMAIL_ADDRESS>{
        "dependencies": {
          "typescript"<EMAIL_ADDRESS>        }
      }
    }

thanks!

FYI - version 2.0.5 of the plugin now uses<EMAIL_ADDRESS>


[Bug]: StockHistoricalDataClient produces a wrong URL

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
client.get_stock_bars(StockBarsRequest(
symbol_or_symbols=["AAPL"],
timeframe=TimeFrame(amount=1,unit=TimeFrame.Day),
start=datetime(2023, 9, 7)))
does not succeed. It creates an invalid timeframe by appending an additional digit:
https://data.alpaca.markets/v2/stocks/bars?start=2023-09-07T00%3A00%3A00Z&timeframe=11Day&symbols=AAPL
Expected Behavior
client.get_stock_bars(StockBarsRequest(
symbol_or_symbols=["AAPL"],
timeframe=TimeFrame(amount=1,unit=TimeFrame.Day),
start=datetime(2023, 9, 7)))
Shall produce the URL:
https://data.alpaca.markets/v2/stocks/bars?start=2023-09-07T00%3A00%3A00Z&timeframe=1Day&symbols=AAPL
SDK Version I encountered this issue in
alpaca_py-0.11.0-py3-none-any.whl
Steps To Reproduce
from alpaca.data.historical import stock
from alpaca.data.requests import StockBarsRequest
from alpaca.data.timeframe import TimeFrame
from datetime import datetime

historical = stock.StockHistoricalDataClient(...)

request_bars = StockBarsRequest(
    symbol_or_symbols=["AAPL"],
    timeframe=TimeFrame(amount=1,unit=TimeFrame.Day),
    start=datetime(2023, 9, 7))

historical.get_stock_bars(request_bars)

Filled out the Steps to Reproduce section?

[X] I have entered valid steps to reproduce my issue or have attached a minimally reproducible case in code that shows my issue happening; and understand that without this my issue will be flagged as invalid and closed after 30 days.

Anything else?
No response

@peters-r try whith the TimeFrameUnit:
from alpaca.data.timeframe import TimeFrame
from alpaca.data.timeframe import TimeFrameUnit
StockBarsRequest(symbol_or_symbols=[SYMBOL],timeframe=TimeFrame(1, TimeFrameUnit.Day), start=START, end=END)

This is expected TimeFrame is not an Enum, but TimeFrameUnit is.

So it's not a bug, but still I think the API is highly confusing in that regard. Maybe could be worth to at least make sure that TimeFrameUnit and TimeFrame don't have the same 'Day' property or to make it more explicit that one is an enum and the other isn't. Or to update the docs to make it more obvious. I probably wouldn't have filed a bug if I found it in the docs.
It works for me now. Thanks for your quick support!

@peters-r I agree with you that having the same name can be confusing, but changing the API would be a breaking change that I would prefer to avoid right now.
The TimeFrame Day class property was actually imagined as a shortcut, please check out the implementation here.
So in your case you can just use timeframe=TimeFrame.Day
Hope this helps.


Blocks experiment stuff

The tests were passing locally.

Do you think we should tidy this up before merging it?

Don't know, the history already has the odd WIP commit in it. If we do want to tidy it I think we'll have to do that together, I don't feel confident rewriting your bits alone.


eventwatch: add controller

This controller can be used to filter interesting event reasons and take action on them.
For example, this allow to register handler, that will update Prometheus counter when an interesting event is observed (or update operator conditions, etc..).
Example:
eventwatch.New().WithEventHandler("kube-apiserver", "Reason", func(*corev1.Event) error {
  // prometheusCounter++
  return nil
}).ToController(informers, eventClient, eventRecorder)

/lgtm
/approve

@deads2k this will provide generic bridge between events and metrics


VR new

I noticed that the player position is not the headset position, so player-position based things like loading a dynamic world do not work.


compat with angular 1.4.5

require('angular') yields empty object with bower/browserify
Look at this file: the angular module define window.angular. When requireing angular, it would return an empty object instead of what we want

@alexbeletsky Can you have a look at this PR since you are the one who introduced this block of code?

I'm not sure why those changes are needed, it's a UMD wrapper and should not be changed. Would like to hear @m-vdb some explanations. I recently switched to angular 1.4 and angucomplete-alt works without any issues.

@alexbeletsky Thanks for your feedback.
@m-vdb Please give us your reasoning.

hi guys, the reasoning is the following:

initial setup: browserify + bower
when using browserify, we enter in this branch of code

if (typeof module !== 'undefined' && module.exports)

here is the file that is installed when using bower: https://github.com/angular/bower-angular/blob/master/angular.js. I cannot link to it (too big), so here is a snippet:

// you'll find that inside the code
angular           = window.angular || (window.angular = {}),

there is no trace of module.exports = in this module
the result is that when doing require("angular") like in your code (I did the same mistake in my app) you'll get an empty object (but window.angular will be set)

I think the main issue is that angular doesn't use module.exports but not sure they'll want to change it.

I'm not a browserify expert but why do you use it with bower?
I just created a small example with browserify with angular 1.4.5 and angucomplete-alt.
It worked fine.
@alexbeletsky Could you verify it too?

I use browserify because it's way better than including each of your files individually. Anyway I'm not using this module anymore. fix it or don't :)

Ok. I'm closing this.


Optimize the default value of replication_num when create table

In the current code, if we do not specify replication_num when creating OlapTable, the default value of 3 will be used, but for some test environments and small clusters, the number of backends is likely to be less than 3, which will cause confusion for users
Therefore, when the number of backends is less than 3 When the default replication_num should be equal to the number of backends, the default value should be 3 when the number of backends is greater than 3.

put creating-table on a WebUI, hide the default replication_num logic.
or doris users should know one be in test-env, so replication_num must 1, not defualt


using JWT get Error  : Import named 'getSchemaValidator' not found in module

i follow you tutorial this https://elysiajs.com/plugins/jwt.html
mycode :
.use(jwt({
        name:"jwtBego",
        secret:"myfist"
    }))
    .get("/masuk",async({jwt,cookie,setCookie,params})=>{
        setCookie('auth', await jwt.sign(params), {
            httpOnly: true,
            maxAge: 7 * 86400,
        })

        return `Sign in as ${cookie.auth}`
    })

but get error like this
SyntaxError: Import named 'getSchemaValidator' not found in module '/home/minto/belajar/elysyaPro/elyapp/manelysia/node_modules/elysia/build/es/index.js'.
[nodemon] app crashed - waiting for file changes before starting...
whats solution

What version of Elysia are you using?
This error likely happens if you're using elysia =< 0.1.0


youtube static address work no longer

i tryed with docker
Before I could make m3u with
https://www.youtube.com/channel/ChnnelID/live
ex) https://www.youtube.com/channel/UCJUCcJUeh0Cz2xyKwkw5Q1w/live
but it does not work now
can you give me a hint?

try build newer youtube-dl in docker image?

try build newer youtube-dl in docker image?

I tryed
in host
$sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o ~/Download/youtube-dl
$docker cp ~/Download/youtube-dl livetv(my container):/usr/local/bin/youtube-dl
$docker exec -it livetv /bin/sh
in container
#chmod a+rx /usr/local/bin/youtube-dl
and it still doesn't work

try build newer youtube-dl in docker image?

not only youtube-dl but streamlink do not work also.
is there any can do to get it?

我也有同样情况，是不是youtube-dl不用一直启动进程？同时，手动执行youtube-dl 提示如下：
I have the same situation. Isn't it necessary for youtube-dl to start the process all the time? At the same time, manually execute the youtube-dl prompt as follows:
~ # /usr/bin/youtube-dl -f best -g https://youtu.be/J8KVk6rBJ1E
ERROR: J8KVk6rBJ1E: YouTube said: Unable to extract video data

刚刚更新最新的youtube-dl , 执行后，提示如下：
Just update the latest youtube-dl, after execution, the prompt is as follows:
env: can't execute 'python': No such file or directory

最后OK了，但是貌似针对有些直播，解析为m3u8后，很快就无效了。再解析，就再有效了。这个该如何应对？
Finally, it was OK, but it seemed that for some live broadcasts, after parsing to m3u8, it soon became invalid. If you analyze it again, it will be effective again. How to deal with this?


[Bug] Dropdown menu not closing when user navigates away / off focus

Version
Angular
Reproduction link
https://www.creative-tim.com/learning-lab/tailwind/angular/overview/notus?ref=na-index-navbar
Operating System
Windows/ Mac
Device
All Devices
Browser & Version
All Browsers
Steps to reproduce
We are trying the basic version for a demo site and still under development.
One major issue we have is that, if the user clicks on Menu (dropdown/ pull down)
https://demos.creative-tim.com/notus-angular/admin/maps
ex: click on User Icon and navigate away from the menu and click on any other action links - You will see the top right corner menu open all the time.
Same issue here - https://demos.creative-tim.com/notus-angular/
click on Demo Pages and navigate way,
How to close the menu when user navigates away?
please help
What is expected?
Dropdown menu should be closed when user navigates away
What is actually happening?
Dropdown always open

Solution
Additional comments

Hey, @trovend ,
Thank you for your issue.
We know about this, and at the moment we have not figured out a way of solving the issue.
Please, if you have any ideas on how to solve this, let us know.
Best,
Manu


cmd/mito: add package godoc

Please take a look.

Tagging this as v1.12.3 purely to make it visible at latest. I won't bump the beats go.mod.


Support configuration for place templates

What type of feature request is this?
Mantle doesn't support a Roblox feature I would like to use
I'd like mantle to support configuring "Allow this place to be copied as a template using the Create Place API in your experience." and "Allow this place to be updated using the Save Place API in your experience."

These settings are used to create games that create brand new Roblox places from a "template" place, which is a feature used in a few building games to persist players builds

I took a quick look into this and it seems like currently the place configuration API Mantle uses does not support these fields. The one which the website uses is a www endpoint which I try to avoid using. I'm guessing that those properties will get ported over to the new creator dashboard at some point and then it will be easier to add.


Support the latest RxJS 6.5.3 in the latest Angular CLI 8.3.20

🚀 Feature request
Description
Update to the latest RxJS package in Angular CLI 8 line.
Describe the solution you'd like
If there are no breaking changes (not aware of any), update RxJS to the latest 6.5.3 version in the Angular CLI 8 line, actually in 8.3.20.

RxJS 6.4.x is the recommended version to use with 8.x.  Angular 9.0 (currently in release candidate stage) will use 6.5.x as the minimum version.


Fixed ReadOnlyPasswordHashWidget's template for RTL languages.

Following from #15977, this fixes bidirectional support in a part of the user admin:
Before this change,

It may not be obvious to a LTR-only user why this is wrong, so I marked the problems here:

Note how the 'salt' field, whose name is currently untranslated, is completely broken (the field name is thrown in the middle of the value; marked in red), and how the value of the 'hash' field (cyan) is also turned around.
After this change, it looks like so:

The field name is always to the right of the field value (since this is RTL, this is "before"), and the field value itself is presented properly -- you may compare with the LTR presentation:

Note: The use of dir="auto" on the  tag is for bidi isolation; ... is equivalent to <bdi>...</bdi>. I thought doing it this way is a little more elegant, but I can see an argument that using <bdi> in both cases is more consistent and expresses the intent more clearly.

@shaib Thanks for this patch :+1:

Note: The use of dir="auto" on the  tag is for bidi isolation; ... is equivalent to <bdi>...</bdi>. I thought doing it this way is a little more elegant, but I can see an argument that using <bdi> in both cases is more consistent and expresses the intent more clearly.

Do we need dir="auto"? :thinking: I don't see a difference without it, and dir is already set on HTML tag:

You're right that it makes no difference with the current set of properties and their Hebrew translations, and further, it's a little hard to come up with a likely example where it would, here.
In general, the addition changes dir from the inherired rtl (or ltr) to auto; this would matter if the property names were affected by directionality. For example, if instead of salt it were called salt+, without the dir="auto" it would look like this

and with it, it would be correct:

but this may very well turn out to be a YAGNI.

@shaib Thanks :+1:

Thank you! :prince:


Hydrogen Mapping

I'm working with Hydrogen tracing  so I need the atom map for hydrogen, but I found Chem.CanonSmiles will remove it in some cases.
Please see examples below:
rsmi = '[H][O:1][H:2]'
canon_rsmi = Chem.CanonSmiles(rsmi)

'[OH2:1]' # miss the hydrogen with atom number 2

rsmi = '[H:1][H:2]'
canon_rsmi = Chem.CanonSmiles(rsmi)

'[H:1][H:2]' # still keep hydrogen number

What I expect is can still keep the atom map number of hydrogen and it should be explicit rdkit.Mol.
Screenshots
If applicable, add screenshots to help explain your problem.
Configuration (please complete the following information):

RDKit version:  2024.3.5
OS: Fedora 37
Python version (if relevant): 3.11
Are you using conda? Yes
If you are using conda, which channel did you install the rdkit from? pip
If you are not using conda: how did you install the RDKit? pip

I'm closing this because it is a question, not a bug report or enhancement request.
Please ask questions using the "Discussions" tab.


Edit json schema

Description
Some trial changes to improve json schema. Mostly being able to specify "additional properties": false on every object.

Codecov Report

Merging #840 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #840   +/-   ##
=========================================
  Coverage     66.48%   66.48%
  Complexity     1749     1749
=========================================
  Files           286      286
  Lines          6081     6081
  Branches        700      700
=========================================
  Hits           4043     4043
  Misses         1789     1789
  Partials        249      249

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30c70ff...f424f18. Read the comment docs.

I didn't realise we were going to remove the BNF too. Does it help or hinder us by having it?


Series constructor does not accept TypedDict

Describe the bug
When creating a series from a TypedDict, Pyright reports
No overloads for "__new__" match the provided arguments. Argument of type (TypedDict) cannot be assigned to parameter "data" of type "Scalar | _ListLike[Unknown] | dict[HashableT1@__new__, Any] | BaseGroupBy[Unknown] | NaTType | NAType | None" in function "__new__"
To Reproduce
from typing import TypedDict
import pandas as pd
class MyDict(TypedDict):
    a: str
    b: str

my_dict = MyDict(a="", b="")
pd.Series(my_dict)

Pyright yields
error: No overloads for "__new__" match the provided arguments (reportCallIssue)
error: Argument of type "MyDict" cannot be assigned to parameter "data" of type "Scalar | _ListLike[Unknown] | dict[HashableT1@__new__, Any] | BaseGroupBy[Unknown] | NaTType | NAType | None" in function "__new__"
    Type "MyDict" is incompatible with type "Scalar | _ListLike[Unknown] | dict[HashableT1@__new__, Any] | BaseGroupBy[Unknown] | NaTType | NAType | None"
      "MyDict" is incompatible with "ExtensionArray"
      "MyDict" is incompatible with "NAType"
      "MyDict" is incompatible with "NaTType"
      "MyDict" is incompatible with "Timedelta"
      "MyDict" is incompatible with "Timestamp"
      "MyDict" is incompatible with "bool"
      "MyDict" is incompatible with "bytes"
    ... (reportArgumentType)

Please complete the following information:

OS: Linux
OS Version: 22
python version: Python 3.12.4
version of type checker: Pyright 1.1.374
version of installed pandas-stubs: <IP_ADDRESS>603

Additional context
Seems like TypedDict is incompatible with dict, but I don't see why it matters in this specific example. Changing overload 8 (series.pyi(332, 9))
@overload
    def __new__(
        cls,
        data: (
            Scalar
            | _ListLike
            | dict[HashableT1, Any]
            | BaseGroupBy
            | NaTType
            | NAType
            | None
        ) = ...,
        index: Axes | None = ...,
        *,
        dtype: Dtype = ...,
        name: Hashable = ...,
        copy: bool = ...,
    ) -> Series: ...

to
@overload
    def __new__(
        cls,
        data: (
            Scalar
            | _ListLike
            | Mapping[HashableT1, Any]
            | BaseGroupBy
            | NaTType
            | NAType
            | None
        ) = ...,
        index: Axes | None = ...,
        *,
        dtype: Dtype = ...,
        name: Hashable = ...,
        copy: bool = ...,
    ) -> Series: ...

fixes this specific example but I don't know if it breaks something elsewhere.

I think that if you changed all the Series.__new__() overloads to use Mapping instead of dict, it would work and handle your use case.
PR with tests welcome.


Configured so ordering are using created_at

When using UUID's, the active record methods 'first' and 'last' doesn't
work as expected, given that it's expecting an auto-incremented
id to sort by.

👍


grape routes with rake

How i can mount one class to different routes
my code
module Inter
class API < Grape::API
mount ::API::V1::InterV1 => "/api"
mount ::API::V2::InterV2 => "/api/v2"
mount ::API::V2::InterV2 => "/api/software"
end
end
but it generate /api/v2/api/software

It's not currently possible to mount one API under two different routes. See https://github.com/ruby-grape/grape/issues/570 for more details and a workaround.

Closing this as a dup of #570.


@JuliaRegistrator register v0.1.3

@JuliaRegistrator register
Patch release v0.1.3
As mentioned in #11 the solver is now Optimizer Agnostic! Currently supports GLPK and Gurobi

Can not support SCIP, it doesn't support Lazy Constraints Callbacks
Solver remaining to test: CPLEX and Xpress

According to JuMP manual, the supported solver for Lazy Constraints Callbacks are:

CPLEX, GLPK, Xpress, Gurobi

Error while trying to register: Version 0.1.2 already exists

@JuliaRegistrator register
Patch release v0.1.3
As mentioned in #11 the solver is now optimizer-agnostic! Currently supports GLPK and Gurobi

Can not support SCIP, it doesn't support Lazy Constraints Callbacks
Solver remaining to test: CPLEX and Xpress
Adding Documenter.jl
JET tests are passing!

According to JuMP manual, the supported solver for Lazy Constraints Callbacks are:

CPLEX, GLPK, Xpress, Gurobi and SCIP

Registration pull request created: JuliaRegistries/General/111227
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.3 -m "<description of version>" 91d972fec9e8b165677724df910b5919a3115586
git push origin v0.1.3

@JuliaRegistrator register
Releaste notes:
RingStarProblems v0.1.3 patch release
As mentioned in #11 the solver is now optimizer-agnostic! Currently supports GLPK and Gurobi

Can not support SCIP, it doesn't support Lazy Constraints Callbacks
Solver remaining to test: CPLEX and Xpress
Adding Documenter.jl
JET tests are passing!

According to JuMP manual, the supported solver for Lazy Constraints Callbacks are:

CPLEX, GLPK, Xpress, Gurobi and SCIP

Registration pull request updated: JuliaRegistries/General/111227
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.3 -m "<description of version>" 6d121bb30b7686284b95ffd1fd968f29fd6f2f95
git push origin v0.1.3

Registration pull request updated: JuliaRegistries/General/111227
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.3 -m "<description of version>" 6d121bb30b7686284b95ffd1fd968f29fd6f2f95
git push origin v0.1.3

Registration pull request updated: JuliaRegistries/General/111227
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.3 -m "<description of version>" 6d121bb30b7686284b95ffd1fd968f29fd6f2f95
git push origin v0.1.3

@JuliaRegistrator register
Release notes:
Patch release v0.1.4

Solving a small bug when calling rspoptimize

Registration pull request created: JuliaRegistries/General/111232
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.4 -m "<description of version>" b67ecd338869244d21b00803209d4561527b9ce6
git push origin v0.1.4


Don't include the filepicker.io JS when using inhouse file upload

This prevents a small information leak where filepicker.io gets some information about mamute users (IP, browser, etc) even when the inhouse upload is enabled.

@xdarklight nice catch
@csokol ok :ship: