cmd/go: "fork/exec foo.test: text file busy" flakes

While trybotting https://go-review.googlesource.com/c/go/+/70151, I got this flake https://storage.googleapis.com/go-build-log/c8965fb8/openbsd-amd64-60_817d939c.log;
fork/exec /tmp/go-build452069409/b118/jsonrpc.test: text file busy
FAIL	net/rpc/jsonrpc	0.003s

Maybe recent cmd/go work?
/cc @rsc

A similar one from a linux/386 trybot:
https://storage.googleapis.com/go-build-log/a1da169f/linux-386_2279a6a1.log
fork/exec /tmp/go-build989750233/b052/ecdsa.test: text file busy
FAIL	crypto/ecdsa	0.001s

Almost certainly related to the recent cmd/go changes. Russ is looking into it.
$ greplogs -dashboard -e "text file busy" -l | findflakes -paths
First observed cdbc363 11 Oct 19:03 2017 (24 commits ago)
Last observed  c37647f 12 Oct 07:12 2017 (2 commits ago)
30% chance failure is still happening
45% failure probability (11 of 23 commits)
Likely culprits:
   45% cdbc363 cmd/go: record both build ID and content ID in archives and binaries
   25% 85f93c8 cmd/compile: fix some plive comments
   14% e130dcf cmd/compile: abort earlier if stack frame too large
    7% 624630b cmd/compile: fold constant comparisions into SETxxmem ops.
No known past failures

It's not really clear how cdbc363 could actually be causing this, though.

Makes no sense. I did have a missing f.Close, but on a file open for reading, not writing. The close of the file open for writing seems obviously there:
w, err := os.OpenFile(target, os.O_WRONLY, 0)
if err != nil {
	return err
}
err = buildid.Rewrite(w, matches, newID)
if err != nil {
	w.Close()
	return err
}
if err := w.Close(); err != nil {
	return err
}

I am wondering if a child process is inheriting the file descriptor and so it stays open in the child after we close it, but os.OpenFile adds O_CLOEXEC, which should make that impossible.
Trying to reproduce. I could just bypass that code path entirely for binaries we are going to run and discard. I assume that would fix the problem, but I'd like to understand the actual problem first.

Reproduced with
while go test -short encoding/...; do true; done

on my Mac. The code in question is not actually needed for ephemeral binaries, like the ones that go test runs, so I'll probably just disable it without understanding it and move on.

It occurs to me that this would happen if os.File.Close returned before the fd had actually been closed. @ianlancetaylor and I have been talking about things like this related to net.Conns, but since the use of the writable fd here is single-threaded, I don't see how the close would not be complete at the moment the test runs.

If anyone can make this happen reliably - like five times in a row in a given time period reliably - then please patch in CL 70390 and see how that changes the frequency at which it happens.
I've somehow lost the ability to reproduce this on my Mac in an unmodified client, so I can't see whether CL 70390 makes the situation better. I did, however, see the problem happen once on my Mac even with CL 70390 applied, so it doesn't appear to be a complete fix regardless.
It should be safe, though, so if anyone can demonstrate proof that it helps, it's fine to submit.

I got this to happen once on Linux using 'perf trace record'. Here is the trace - trace1.txt.gz - truncated right after the exec that got ETXTBSY. Literally every open call in the trace has CLOEXEC set.
But maybe the problem is that there is a separate fork+exec happening in parallel, and while that separate child's exec will close the writable fd, it hasn't exec'ed yet, so it still holds the fd in its forked copy of the fd tables. That would explain the problem. It would not explain why I saw this even after removing the "open binary for write" code from my Mac, but maybe there is a second problem on the Mac.
Now that I got a trace on Linux with the old code I will try with my pending CL.

Backing up a step. On systems where close(2) actually does real work and can return a meaningful error, does that only apply to the final close of the underlying file? That is, if I do:
fd = open()
write(fd)
fd2 = dup(fd)
close(fd)
close(fd2)

Does the close(fd) do any real work flushing writes, or is it only the close(fd2)? Because if it's only the close(fd2), then any time you have a file open for write and have a different goroutine doing fork+exec, there's a race where you might not see write errors because your close is not the last. The other goroutine can do a fork (essentially creating fd2 as above), and then close(fd) does nothing - and succeeds - because fd2 is still open, and then the new child process either execs or exits and closes fd2 implicitly.
Are things really this bad?
The same logic means that cmd/go basically can never open an executable for write. It must always invoke a subprocess to do that, because if it opens an executable for write there is no guarantee about a timely close of the fd relative to a future attempted exec.
Are things really this bad?
/cc @ianlancetaylor @aclements

My understanding is the first close does serious work. I base this on the fact that the first close will release any process-wide file system locks, even when there are other descriptors. (I went and looked in the linux kernel sources to confirm, and did find an unconditional call to a flush op on the close path, but I couldn't follow it into a file system implementation.)
Unfortunately, I can't nail down what serious work is. It is definitely FS-specific, but I haven't found any notes about what is conventional for "fast" file systems. It sounds like it flushes to the kernel write buffers. The syscall documentation explicitly says it does not push the changes to physical media:

   A successful close does not guarantee that the data has been
   successfully saved to disk, as the kernel uses the buffer cache to
   defer writes.  Typically, filesystems do not flush buffers when a
   file is closed.  If you need to be sure that the data is physically
   stored on the underlying disk, use fsync(2).  (It will depend on the
   disk hardware at this point.)

But that doesn't say anything about the visibility of the changes to other processes.
There is also this alarming note:

   It is probably unwise to close file descriptors while they may be in
   use by system calls in other threads in the same process.  Since a
   file descriptor may be reused, there are some obscure race conditions
   that may cause unintended side effects.

http://man7.org/linux/man-pages/man2/close.2.html
It is not clear if this applies to close-on-exec descriptors. If it does, then syscall.Close should grab a read lock on runtime.execLock.

It is not clear if this applies to close-on-exec descriptors. If it does, then syscall.Close should grab a read lock on runtime.execLock.

That's an easy enough experiment to see if it helps, though, assuming this is reproducible enough.

If a process has a file open for write access, and while the file is open the process calls fork, then there are two processes that have the file open for write access.  An execve will fail with ETXTBSY if any process, even a completely unrelated process, has the file open for write access.  This doesn't really have anything to do with when close flushes buffers.  The execve will fail if the file is open for write access, even if nothing has been written.
So, yes, if the go tool is opening executables for writing in parallel with exec'ing other programs, and the go tool then wants to exec the executable that it just wrote, there is a race condition.

thread 1: open EXE1 for writing
thread 2: call StartProcess to execute EXE2
thread 2: fork, creating child 1
child 1: starts, now has EXE1 open for writing
thread 1: writes to EXE1 and closes it
thread 1: call StartProcess to execute EXE1
thread 1: fork, creating child 2
child 2: starts
child 2: calls execve to execute EXE1
child 2: execve fails with ETXTBSY because child 1 has EXE1 open for writing

The usual fix is to not write EXE1 in the first place.  Instead, write TMP1.  Write to TMP1 as much as you like.  Fork off other processes that temporarily hold TMP1 open for writing--it's OK.  When you're done writing to TMP1, call rename(TMP1, EXE1).  Now you are safe to exec EXE1; people might have TMP1 open for writing, but nobody has EXE1 open.

Unfortunately, neither of the current fork/exec-related locks is applicable here. runtime.execLock is about calling Exec directly, to replace the entire Go process, which we're not doing.
syscall.ForkLock is about making sure a child process does not inherit a temporarily not-close-on-exec fd. That is, consider:
1. T1: fd = open()
2. T2: fork, creating P2 (inherits fd)
3. T1: setCloseOnExec(fd)
4. P2: exec new program, goes on its way having inherited fd _not_ CLOEXEC.

Now P2 has a copy of fd, because the fork happened before fd was marked close-on-exec, so it didn't get marked close-on-exec in P2. The ForkLock protects this kind of sequence by keeping 2 from splitting the combination of 1+3.
That's not what's happening here. For one thing, we're using open with O_CLOEXEC, so that 1+3 is actually just 1. What's happening here is:
1. T1: fd = open("test.exe", O_CLOEXEC|O_WRONLY)
2. T2: fork, creating P2 (inherits fd cloexec)
3. T1: close(fd)
4. T1: fork, creating P3
5. P3: exec("test.exe"): ETXTBSY because of P2's fd
6. P2: exec("other.exe") - closes fd

syscall.ForkLock guards against P2 inheriting fd non-close-on-exec. Here, P2 has inherited fd close-on-exec, and when it gets to the exec in step 6, it will close fd. But that doesn't do us any good at step 5 when we attempt the exec. Right now we acquire ForkLock before the fork and release ForkLock after the fork, before waiting for the exec to complete. (The signal that exec has completed is that we get EOF on a close-on-exec pipe fd we passed to it.) We could potentially keep holding ForkLock until the exec completes. That would prevent the sequence above by making sure that two different fork+exec cannot be interleaved.
In effect linux/amd64 already does this, because it has been changed to use vfork, which doesn't let the parent run again (and release ForkLock) until the exec completes. And that seems OK, so maybe extending ForkLock in general is OK too.

@ianlancetaylor I don't understand why rename would help. Changing the name of the directory entry pointing at the open file doesn't make it less open.
$ echo first > file1; (sleep 1; echo hi) >>file1 & mv file1 file2; cat file2; wait; cat file2
[1] 93060
first
[1]+  Done                    ( sleep 1; echo hi ) >> file1
first
hi
$

or maybe more to the point:
$ cp /bin/ls ls1; sleep 1 >>ls1& ./ls1; mv ls1 ls2; ./ls2
[1] 19089
-bash: ./ls1: Text file busy
-bash: ./ls2: Text file busy
$

Bother, you're clearly right.  I was thinking of a different problem.

Why don't we have the linker handle the build ID, rather than cmd/go itself?

Hey look! We've been here before - #3001, which turned into this very long comment 5+ years ago:
nbusy := 0
for {
	var buf bytes.Buffer
	cmd := exec.Command(cmdline[0], cmdline[1:]...)
	cmd.Stdout = &buf
	cmd.Stderr = &buf
	cmd.Dir = dir
	// TODO: cmd.Env
	err := cmd.Run()

	// cmd.Run will fail on Unix if some other process has the binary
	// we want to run open for writing.  This can happen here because
	// we build and install the cgo command and then run it.
	// If another command was kicked off while we were writing the
	// cgo binary, the child process for that command may be holding
	// a reference to the fd, keeping us from running exec.
	//
	// But, you might reasonably wonder, how can this happen?
	// The cgo fd, like all our fds, is close-on-exec, so that we need
	// not worry about other processes inheriting the fd accidentally.
	// The answer is that running a command is fork and exec.
	// A child forked while the cgo fd is open inherits that fd.
	// Until the child has called exec, it holds the fd open and the
	// kernel will not let us run cgo.  Even if the child were to close
	// the fd explicitly, it would still be open from the time of the fork
	// until the time of the explicit close, and the race would remain.
	//
	// On Unix systems, this results in ETXTBSY, which formats
	// as "text file busy".  Rather than hard-code specific error cases,
	// we just look for that string.  If this happens, sleep a little
	// and try again.  We let this happen three times, with increasing
	// sleep lengths: 100+200+400 ms = 0.7 seconds.
	//
	// An alternate solution might be to split the cmd.Run into
	// separate cmd.Start and cmd.Wait, and then use an RWLock
	// to make sure that copyFile only executes when no cmd.Start
	// call is in progress.  However, cmd.Start (really syscall.forkExec)
	// only guarantees that when it returns, the exec is committed to
	// happen and succeed.  It uses a close-on-exec file descriptor
	// itself to determine this, so we know that when cmd.Start returns,
	// at least one close-on-exec file descriptor has been closed.
	// However, we cannot be sure that all of them have been closed,
	// so the program might still encounter ETXTBSY even with such
	// an RWLock.  The race window would be smaller, perhaps, but not
	// guaranteed to be gone.
	//
	// Sleeping when we observe the race seems to be the most reliable
	// option we have.
	//
	// http://golang.org/issue/3001
	//
	if err != nil && nbusy < 3 && strings.Contains(err.Error(), "text file busy") {
		time.Sleep(100 * time.Millisecond << uint(nbusy))
		nbusy++
		continue
	}

	return buf.Bytes(), err
}
panic("unreachable")

There's a specific problem: cmd/go writes a binary and then runs it, and a race in fork+exec makes that fail with ETXTBUSY sometimes.
There's a more general problem: any Go program that writes a binary and then runs it can hit that same race.
The first question to answer is which problem we want to solve. Do we want to protect other Go programs from this race, or just work around it in cmd/go, like we did in #3001?
If we want to solve the general problem, then holding ForkLock until exec closes the old fds seems like the best solution, provided we're sure that seeing the pipe close means all the other close-on-exec fds are closed too.
If we want to solve the cmd/go problem only, I think it suffices to just stop calling updateBuildID for binaries we write+run+remove. The other case is binaries we install+run, but we only do that during 'go test' with an implicit install of the binary (like 'go test -cpuprofile'), and in that case the final link+updateBuildID+run is not running in parallel with any other possible fork+execs. We can also drop the cmd/cgo hack that I found, because we no longer install+run cmd/cgo.
Putting the updateBuildID into the linker would be an alternative way to solve the cmd/go problem only, but the code would have to be in gccgo's linker too, and we'd still need to keep the code in cmd/go for updating the build ID in package archives. It seems easier to have it in one place instead of three.

If we want to solve the general problem, then holding ForkLock until exec closes the old fds seems like the best solution, provided we're sure that seeing the pipe close means all the other close-on-exec fds are closed too.

I think seeing the pipe close only ensures (on Linux) that lower-numbered FDs are closed, but on the implementation of do_close_on_exec.

provided we're sure that seeing the pipe close means all the other close-on-exec fds are closed too

I'd be surprised if that's a POSIX guarantee. If there were thousands of FDs to CLOEXEC and the pipe was early amongst them, I wouldn't be surprised if the reader saw EOF whilst other close were still to happen.

I have been trying to work out why the linux/amd64 builders don't flake with "text file busy", while it is quite common on linux/386. With some help from @aclements I think I know why.
As of https://golang.org/cl/37439 we set CLONE_VFORK linux/amd64. This means we don't get to the parent's call of ForkLock.Unlock() until after the close-on-exec FDs are closed in the child.
So we could resolve this by moving the other Unix ports to a vfork-style syscall.

I don't think it's necessary to solve the general problem.  It's quite unusual for a program to write an executable.  There is no problem if the executable is written by a child process, and that is exactly how most executables are written, the child process being the linker.

I'm not sure I understand quite how you are using build IDs now, but it looks like you are creating a build ID that is a hash of the executable.  gccgo's linker already has a way to do that: --build-id=sha1.

I opened #22315 for the general problem. I'm less convinced that it's unnecessary to fix. It's easy to imagine an AppEngine-like service that runs code on demand by fetching a zip file, unpacking it, and running the program inside, and that would get exactly these kinds of flakes. Yes, it's not really our fault, but at the same time I wasted a day debugging this last week (and probably a similar amount of time five years ago), and there's an open Java bug report about the same problem, so people do hit this. If there's something simple we can do, we probably should do it. But let's move that discussion to #22315.
I sent CL 71570 to stop writing the build ID for binaries we will run. That eliminates the race as currently being experienced by cmd/go today and should fix this issue (cmd/go flakes).


Shape inspections incorrectly occuring on type cast shapes

Environment

name
version

IDEA version
2020.2

Luanalysis version
1.03

OS
macOs 10.15.6

What are the steps to reproduce this issue?

Run Inspect Code against his file:

---@alias VArgs std__Packed<any>

local result = --[[---@type std__Packed<any> ]] {n = 0}
local result = --[[---@type VArgs ]] {n = 0}

What happens?
The cast on the last line appears to have no effect, and and error is shown
What were you expecting to happen?
The aliased type should behave the same as the original

The description for this issue was a bit off as the type cast was correctly occurring:

It was a bit more complicated, and to-do with the functionality that provides inspections on table literals being assigned to a shape type. Although, the alias was involved in the example above, it'd occur in other circumstances too.


Add ListTask feature

Add ListTask feature with GUI updates

Codecov Report
Base: 66.16% // Head: 64.91% // Decreases project coverage by -1.24% :warning:

Coverage data is based on head (3ac2659) compared to base (72abd81).
Patch coverage: 3.12% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #136      +/-   ##
============================================
- Coverage     66.16%   64.91%   -1.25%
  Complexity      409      409
============================================
  Files            77       79       +2
  Lines          1383     1411      +28
  Branches        143      145       +2
============================================
+ Hits            915      916       +1
- Misses          429      456      +27
  Partials         39       39

Impacted Files
Coverage Δ

...rc/main/java/seedu/address/logic/LogicManager.java
71.42% <0.00%> (-3.58%)
:arrow_down:

.../seedu/address/logic/commands/ListTaskCommand.java
0.00% <0.00%> (ø)

.../seedu/address/logic/parser/AddressBookParser.java
80.00% <0.00%> (-4.22%)
:arrow_down:

src/main/java/seedu/address/model/Model.java
100.00% <ø> (ø)

src/main/java/seedu/address/ui/MainWindow.java
0.00% <0.00%> (ø)

src/main/java/seedu/address/ui/PersonCard.java
0.00% <0.00%> (ø)

...rc/main/java/seedu/address/ui/PersonListPanel.java
0.00% <0.00%> (ø)

src/main/java/seedu/address/ui/TaskListCard.java
0.00% <0.00%> (ø)

...rc/main/java/seedu/address/model/ModelManager.java
94.11% <25.00%> (-5.89%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


HWKINVENT 45 - Deleting metrics before deleting the resource from inventory

also when creating and deleting the metrics, now we are doing it in parallel with the $q.all

LGTM.


FirebaseFirestoreSwift is not working when installed via Swift Package Manager

My Environment

Xcode version: 12.5.1
Firebase SDK version: 8.7.0
Installation method: Swift Package Manager
Firebase Component: FirestoreSwift

Describe the problem

When I installed FirebaseFirestoreSwift via Swift Package Manager, Type Firestore has no member Dedocer error is shown.

Steps to reproduce:

Inatall Firebase via Swift Package Manager, and select Firestore and FirebaseFirestoreSwift

This may be an Xcode cache issue. Please try cleaning and rebuilding.


Keyboard key press and key release events

Add a way to detect a key press event and a key release event.
In my particular problem I am trying to toggle the display of debug information on screen. I have a global var debug bool. It is being toggled when I press D:
if ebiten.IsKeyPressed(ebiten.KeyD) {
    debug = !debug
}

the debug information is displayed:
if debug == true {
    ebitenutil.DebugPrint(…)
}

Problem here is that at each iteration of the main loop debug is toggled at about 60 Hz… A key press and release events would help to solve that.

I believe this is covered in the typewriter example
Relevant code:

// repeatingKeyPressed return true when key is pressed considering the repeat state.
func repeatingKeyPressed(key ebiten.Key) bool {
const (
delay    = 30
interval = 3
)
d := inpututil.KeyPressDuration(key)
if d == 1 {
return true
}
if d >= delay && (d-delay)%interval == 0 {
return true
}
return false
}

Specifically I believe you could use inpututil.KeyPressDuration(ebiten.KeyD) for exactly your needs.

Many thanks!
On Sat, 21 Jul 2018 at 03:39, atolVerderben<EMAIL_ADDRESS>wrote:

I believe this is covered in the typewriter example
https://hajimehoshi.github.io/ebiten/examples/typewriter.html
Relevant code:
// repeatingKeyPressed return true when key is pressed considering the
repeat state.
func repeatingKeyPressed(key ebiten.Key) bool {
const (
delay = 30
interval = 3
)
d := inpututil.KeyPressDuration(key)
if d == 1 {
return true
}
if d >= delay && (d-delay)%interval == 0 {
return true
}
return false
}
Specifically I believe you could use
inpututil.KeyPressDuration(ebiten.KeyD) for exactly your needs.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/hajimehoshi/ebiten/issues/648#issuecomment-406757574,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgvKsoqWfxaLfQCkNdqweVpFd4U2cJKks5uIng6gaJpZM4VZPzf
.

Oh boy. I didn't realize that there's even more under the child directories.
I've just checked and there are two functions that seem to provide my wanted functionality:
https://godoc.org/github.com/hajimehoshi/ebiten/inpututil#IsKeyJustPressed
https://godoc.org/github.com/hajimehoshi/ebiten/inpututil#IsKeyJustReleased
I have checked inpututil.IsKeyJustPressed and sometimes when I release the pressed button, the event is fired again. That is once when in press and once when I release. If the time between press and release is small, the event is fired only when I press.

Oh, you mean that the even for release is not fired when you press a key too quickly? That might be a bug. Which environment are you using? Thanks!

$ gopherjs version
GopherJS 1.10-4
$ go version
go version go1.10.3 linux/amd64

Git's HEAD at my $GOPATH/src/github.com/hajimehoshi/ebiten points at the
commit 9047e54ae6541ceb6a5415f0e5ecc563dbaa04dc.
The problem only appears in the go version, not the gopherjs web version.
Please ask me for more information as needed.
On Sat, 21 Jul 2018 at 07:11, Hajime Hoshi<EMAIL_ADDRESS>wrote:

Oh, you mean that the even for release is not fired when you press a key
too quickly? That might be a bug. Which environment are you using? Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/hajimehoshi/ebiten/issues/648#issuecomment-406769042,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgvKn2BFtAqQTAaLY3_9mJxIv7EcUB5ks5uIqnbgaJpZM4VZPzf
.

package main

import (
        "github.com/hajimehoshi/ebiten"
        "github.com/hajimehoshi/ebiten/inpututil"
)

func update(screen *ebiten.Image) error {
        if inpututil.IsKeyJustPressed(ebiten.KeyA) {
                println("A is just pressed")
        }
        if inpututil.IsKeyJustReleased(ebiten.KeyA) {
                println("A is just released")
        }
        return nil
}

func main() {
        if err := ebiten.Run(update, 320, 240, 2, "foo"); err != nil {
                panic(err)
        }
}

I couldn't reproduce this on my MacBook Pro.

I've tried reproducing using your code and it did reproduce:
$ go run reproduce.go
# github.com/go-gl/glfw/v3.2/glfw
In file included from /home/yuval/gopath/src/github.com/go-gl/glfw/v3.2/glfw/c_glfw_linbsd.go:24:0:
/home/yuval/gopath/src/github.com/go-gl/glfw/v3.2/glfw/glfw/src/linux_joystick.c: In function ‘_glfwInitJoysticksLinux’:
/home/yuval/gopath/src/github.com/go-gl/glfw/v3.2/glfw/glfw/src/linux_joystick.c:224:42: warning: ‘%s’ directive output may be truncated writing up to 255 bytes into a region of size 9 [-Wformat-truncation=]
             snprintf(path, sizeof(path), "%s/%s", dirname, entry->d_name);
                                          ^~~~~~~
In file included from /usr/include/stdio.h:862:0,
                 from /usr/include/X11/Xcursor/Xcursor.h:26,
                 from /home/yuval/gopath/src/github.com/go-gl/glfw/v3.2/glfw/glfw/src/x11_platform.h:39,
                 from /home/yuval/gopath/src/github.com/go-gl/glfw/v3.2/glfw/glfw/src/internal.h:169,
                 from /home/yuval/gopath/src/github.com/go-gl/glfw/v3.2/glfw/glfw/src/x11_init.c:28,
                 from /home/yuval/gopath/src/github.com/go-gl/glfw/v3.2/glfw/c_glfw_linbsd.go:19:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:64:10: note: ‘__builtin___snprintf_chk’ output between 12 and 267 bytes into a destination of size 20
   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        __bos (__s), __fmt, __va_arg_pack ());
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

here I press a and quickly release it
A is just pressed
A is just released

here I press it and release only after a while:
A is just pressed
A is just released
A is just pressed
A is just released

$ uname -a
Linux yankel-Inspiron-5567 4.15.0-24-generic #26-Ubuntu SMP Wed Jun 13 08:44:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04 LTS
Release:        18.04
Codename:       bionic

So the problem is that only one time pressing and releasing a key print out four messages like
A is just pressed
A is just released
A is just pressed
A is just released

?

That's right.

Thanks.
Hmm, this might be a hardware or a driver problem on your keyboard. If you pressing A, is "A" shown only once? http://en.key-test.ru/ (as browser implementation might be different from GLFW, this might hide the problem though).

The problem does not appear in the browser.

Ahh, the problem itself doesn't happen on GopherJS version as you mentioned, so testing the website doesn't make sense. Sorry for bothering you.
I appreciate your reporting. I'll take a look this on my Linux machine later.

This sounds like this issue in glfw. It's been fixed for 3.3 but 3.3 is still in development :(

Dang, I think I'm running into this, too - ebiten.IsKeyPressed() is returning true even after I release the key. Is there some way around this, I wonder?

I also wonder, but have not come up with it.

I tested on my Linux laptop (DELL Inspiron 3162), but could not reproduced. Hmm

Ooh, ok, I could reproduce this by holding the key for long time. I'll think an ad-hoc way to fix this, or forking glfw.

@hajimehoshi Do you plan to contribute patch upstream? Probably maintaining fork indefinitely is not a very sustainable solution :)

I didn't think so, but it depends on go-gl/glfw's policy. As keeping go-gl/glfw should wrap the original GLFW as it is, I think I should NOT apply the change, but I should have asked the authors before I determined.
@shurcooL Hi, we would like to fix the bug in GLFW 3.2 by applying a patch to GLFW before having GLFW 3.3 which we are not sure when it is released, but I am not sure I can do that since go-gl/glfw might have to be just a simple wrapper to wrap the original GLFW. What do you think?

Created a branch to use hajimehoshi's GLFW https://github.com/hajimehoshi/ebiten/tree/issue-648-glfw-fork

@shurcooL I posted https://github.com/go-gl/gl/issues/111, so let's discuss there. Sorry for the confusion.

The fix was merged into go-gl/glfw.
@kibertoad Hi, thank you for pointing it out!

Woohoo!

Note: if you don't use Go modules, the below command should work:
go get -u github.com/hajimehoshi/ebiten/...
go get -u github.com/go-gl/glfw/v3.2/...
# go get -u github.com/go-gl/glfw/... failed since v3.0 cannot be compiled :-(

If you have a trouble, try clearing the cache:
go clean -i -cache github.com/go-gl/glfw/v3.2/...


orders list added during serialise to prevent RuntimeError

closes #232

Coverage increased (+0.04%) to 94.027% when pulling 1b1f7da316432372e4e414a3e6a0bdf0b33388dc on bugfix/232-dictionary-changed-size into b2d60b0c30f6dd9be0fc348903a9d766c2734157 on master.


Fixing "AzCopy from S3 to Blob fails with 401 code after update"

Fix for https://github.com/Azure/azure-storage-azcopy/issues/1551

So this was a regression and we don't have a test, can the e2e package accommodate such a test right now?

I will look into adding tests for this scenario!

So this was a regression and we don't have a test, can the e2e package accommodate such a test right now?

We currently don't have support for both S3 and GCP in our E2E test framework. Shall we pick it up for 10.14 or next cleanup week? @zezha-msft


refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template

REQUIRED:

Removed boilar code
Changed REGION variable name to LOCATION
Hardcode tensorflow version to 2.15.1
Removed os.getenv(""IS_TESTING"") while cleanup bucket
Colab enterpise logo added
Written bullet points into different line. As currently they written on single line
Different places changed text from first person to second person
Placed hyperlink at appropriate place where its present with some unwanted things

Adapted notebook with new template and removed boilerplate code

REQUIRED: Fill out the below checklists or remove if irrelevant

If you are opening a PR for Official Notebooks under the notebooks/official folder, follow this mandatory checklist:

[x] Use the notebook template as a starting point.
[x] Follow the style and grammar rules outlined in the above notebook template.
[x] Verify the notebook runs successfully in Colab since the automated tests cannot guarantee this even when it passes.
[x] Passes all the required automated checks. You can locally test for formatting and linting with these instructions.
[ ] You have consulted with a tech writer to see if tech writer review is necessary. If so, the notebook has been reviewed by a tech writer, and they have approved it.
[ ] This notebook has been added to the CODEOWNERS file under the Official Notebooks section, pointing to the author or the author's team.
[x] The Jupyter notebook cleans up any artifacts it has created (datasets, ML models, endpoints, etc) so as not to eat up unnecessary resources.

If you are opening a PR for Community Notebooks under the notebooks/community folder:

[ ] This notebook has been added to the CODEOWNERS file under the Community Notebooks section, pointing to the author or the author's team.
[ ] Passes all the required formatting and linting checks. You can locally test with these instructions.

If you are opening a PR for Community Content under the community-content folder:

[ ] Make sure your main Content Directory Name is descriptive, informative, and includes some of the key products and attributes of your content, so that it is differentiable from other content
[ ] The main content directory has been added to the CODEOWNERS file under the Community Content section, pointing to the author or the author's team.
[ ] Passes all the required formatting and linting checks. You can locally test with these instructions.

/gcbrun

/gcbrun

suggested edits

We did all the required changes.

/gcbrun


Info window doesn't work

From directory:

I face a problem: I use a marker but I can't show the info window on both cases:
when a click on it,
or through setInfoWindowVisible() method
I can reproduce it on your official demo: changing tab from 'Google Maps Demo' to another one (e.g. 'Add Markers Demo') and then back again to 'Google Maps Demo', when I do click on the marker the info window does not appear again (neither the icons on the polygon).

Same issue as #45. Info window is set by calling executeJs instead of setting a property.


Resolve keyof and index operations instead of their targets.

Fixes #58752

Seems reasonable. Can you just address the style nits?

@typescript-bot cherry-pick to release-5.5


Add ability to delete

Add operation-admins to delete their operations
Adds the ability for users to delete an operation and corresponding data. This will remove both database records as well as content store data.
Addresses Issue: #10
Details
Who can use this
This is restricted to admins of the operation in question, as well as super admins, via the admin/operations list menu
Back End Changes

Adds a new service and endpoint to trigger the delete (this is a synchronous process, as it is expected that operations won't be sufficiently large to justify offloading this to a separate worker.)
Updates the delete evidence endpoint to also delete content
Works for all evidence types, including no-content evidence
Adds some new test helpers

Front End Changes

Adds to the Operation settings a new Delete Operation button that kicks off the delete process
Delete operation prompts the user with a simple challenge, as is done in the admin section (the requesting user will need to type in the operation slug to verify that they actually want to delete the operation)
Returns the user to the list operations page upon successfully deleting an operation

Meeting License Requirements

I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.

Note that this solution does everything within the same request. This should be fine, as it's unlikely that remote deletes will take an excessive amount of time, however this could be revised to support huge operations as follows:

Upon starting delete, record all evidence and add this to a separate table in the database as a queue of keys to remove from the content store
Develop a worker that will read this queue and progressively remove items

The upside to this process is that delete operations should become much quicker, as it instead becomes about 10ish database queries, executed in a transaction. It also allows easy removal of orphaned files (though this really shouldn't be necessary). It should also allow us to remove the warning on redeployment, since it'll all be transaction based.
The downside is that another process handle this removal, and also that content would remain around longer. This could be done as a long running goroutine within the existing backend, or an entirely separate application that can be managed separately.


/stats  command

Hi guys,
it would be great to develop the /stats admin command!
This command should use as input the messages collected in the DB and then show the number of request per day to the bot.
Is someone interested? Some volunters in the develop of this feature? ;)

Hi, I have an outline of the team that bypassing the main DB class accomplishes this . In the coming days I'm going to add this functionality to a class of DB, and can be published as soon as it is completed yet .

Very good!

Hi. What is mean of ['text' => 'callback', 'callback_data' => 'identifier']? what is identifier? where it is defined? Why when i press the this button, bot show popup that say "Hello world!".
Please describe.

Because you're using example commands. You should go through all of them and read the code and understand it.
Then start creating own commands, you can literally copy all SystemCommands to your project dir and set     $telegram->addCommandsPath($commands_path); pointing to it, thats where you can start developing your bot commands!

@skmohammadi

src/Commands/SystemCommands/CallbackqueryCommand.php

yes, i found that. thank you @antimech

I'll close this for now, as this would be included in some analytics feature, if that becomes a thing at some point.


Extension is unable to find my remote fork

Extension version: 0.23.1
VSCode Version: 1.53.2
OS:

Steps to Reproduce:
1.Try to create a PR with a remote named fork
2. Notice how the extension says it can't figure out where my fork is:

As a workaround, I think adding fork to the githubPullRequests.remotes should fix this

As a workaround, I think adding fork to the githubPullRequests.remotes setting should fix this

Closing as this is intentional.


Add progress.Writer to init/deinit

Hey there, really like your workerpool package. I am fiddling around with it to use the init/deinit feature to create a progress bar for each worker.

An init function create a tracker:
func initWrk(workerID int) (tr *progress.Tracker, err error) { // worker init function
	tr = &progress.Tracker{
		Message: fmt.Sprintf("wrk%d", workerID),
		Units:   progress.UnitsDefault,
	}
	return tr, nil
}

The handler can increment the tracker each time it handles something...
func handler3(job workerpool.Job[Payload], workerID int, tracker *progress.Tracker) (Result, error) {
	r := Result{}
	r.Sqrt = math.Sqrt(float64(job.Payload.Num))
	time.Sleep(100 * time.Millisecond)
	tracker.Increment(1)
	return r, nil
}

My problem is that I'd need the tracker in the main function as I need to add it to the progress.Writer.
pw.AppendTracker(tracker)

Any chance I can get the init result from each worker handled up? SImilar to range p.Results?
for tracker := range p.Connections {
    pw.AppendTracker(tracker)
}

}

Thanks for the suggestion.
Keep in mind that workers can start and stop at any time. Only the active workers have a connection (progress.Tracker in your case). When a worker stops the connection is set to nil and when it starts again a new connection is created. So whatever progress you saved in the previous connection object is lost so your tracker starts from 0 again.
So the solution is to save your trackers in a slice outside the initWrk function. The slice should have size equal to the maximum number of workers. In init() you check if there is an existing tracker for that worker and use it, otherwise you create a new one. like this:
func initWrk(trackers []*progress.Tracker) func(workerID int) (*progress.Tracker, error) {
	return func(workerID int) (*progress.Tracker, error) {
		existingTracker := trackers[workerID]
		if existingTracker != nil {
			// reusing existing tracker if it exists
			// so you don't lose the progress if the worker stops and then starts again
			return existingTracker, nil
		}
		tr := &progress.Tracker{
			Message: fmt.Sprintf("wrk%d", workerID),
			Units:   progress.UnitsDefault,
		}
		trackers[workerID] = tr
		return tr, nil
	}
}

func main() {
	pw.Trackers := make([]*progress.Tracker, 5)
	pool, err := workerpool.NewPoolWithResultsAndInit(5, handler3, initWrk(pw.Trackers), deinitWrk(pw.Trackers))
}

or make the initWrk a closure in main() (so it has access to pw.Trackers) to avoid the function returning a function.
You don't need to protect the trackers slice with a mutex because each worker reads and writes to a different position of the slice, and my library won't call init for the same worker concurrently (unless there is a bug, I will check).
I don't know if I can add some method that would help you.
Adding a method in Pool to return a slice with the connections of all the currently active workers (and nil for the inactive) wouldn't help at all.
Adding a p.Connections channel "SImilar to range p.Results" as you said sounds better but I don't think it can help you. Because when a worker is stopped and then restarts it will create a new tracker that starts from 0. So this has to be done in init(). Unless I have misunderstood your requirements.
I'm open to suggestions though.

Maybe I can add an option to keep the connection and not reset it whenever a worker stops.
In that case adding a p.Connections channel will work. Whenever a worker starts it will send it's connection to this channel. but then you would accumulate lots of inactive connections. So the worker should send a struct with the workerID and the connection so you save each connection in a slice at position workerID. So instead of pw.AppendTracker(tracker) as you proposed you will do pw.Trackers[c.WorkerID] = c.Connection.
But this isn't much simpler that my solution. Let me know what you think.

Thanks a bunch. I'll give it a spin. I did not know that providing an argument like that is possible. 🤔
func initWrk(trackers []*progress.Tracker) func(workerID int) (*progress.Tracker, error)

I created a little example here: gitlab.com/qnib-golang/scripts/workpool. I reckon a reconnecting worker is not handled correctly as the tracker is marked done at deinit. But that's fine for my use-case I suppose.
Thanks for your input.

I created a little example here: gitlab.com/qnib-golang/scripts/workpool. I reckon a reconnecting worker is not handled correctly as the tracker is marked done at deinit. But that's fine for my use-case I suppose.
Thanks for your input.

@ChristianKniep I looked at your code. You should submit jobs and call p.StopAndWait() in main() not in a goroutine. because otherwise your program doesn't wait until all jobs have been processed. Also I believe you can call tracker.MarkAsDone() once after p.StopAndWait() and not in deinit().
Here is my fix. I have also added  workerpool.StopWorkerAfterNumOfJobs(2) when creating the pool just to test if it causes any issues and it seems to work fine. Obviously you should remove that.
diff --git a/pkg/lib/pool/pool.go b/pkg/lib/pool/pool.go
index 36f50c4..9590314 100644
--- a/pkg/lib/pool/pool.go
+++ b/pkg/lib/pool/pool.go
@@ -33,8 +33,6 @@ func NewTrackers(pw progress.Writer, cnt int) *Trackers {

 func deinitWrk(trackers *Trackers) func(workerID int, tr *progress.Tracker) error { // worker deinit function
 	return func(workerID int, tr *progress.Tracker) error {
-		t := trackers.List[workerID]
-		t.MarkAsDone()
 		return nil
 	}
 }
@@ -61,7 +59,7 @@ func WorkerPool(cmd *cobra.Command, args []string) {
 	wrkCnt := 4
 	wrkTracker := NewTrackers(r.Writer, wrkCnt)

-	p, err := workerpool.NewPoolWithResultsAndInit(4, Handler, initWrk(wrkTracker), deinitWrk(wrkTracker))
+	p, err := workerpool.NewPoolWithResultsAndInit(4, Handler, initWrk(wrkTracker), deinitWrk(wrkTracker), workerpool.StopWorkerAfterNumOfJobs(2))
 	if err != nil {
 		panic(err)
 	}
@@ -75,23 +73,23 @@ func WorkerPool(cmd *cobra.Command, args []string) {
 	}()
 	// Submit half the jobs with delay
 	go func() {
-		time.Sleep(5 * time.Second)
-		for i := 0; i < total/2; i++ {
-			pl := Payload{
-				Num: i,
-			}
-			p.Submit(pl)
+		for result := range p.Results {
+			tracker.Increment(1)
+			_ = result
+			//tracker.(result.Value.Sqrt)
+			//fmt.Println("result:", result.Value)
 		}
-		p.StopAndWait()
+		time.Sleep(1 * time.Second)
 	}()
-	for result := range p.Results {
-		tracker.Increment(1)
-		_ = result
-		//tracker.(result.Value.Sqrt)
-		//fmt.Println("result:", result.Value)
+
+	time.Sleep(5 * time.Second)
+	for i := 0; i < total/2; i++ {
+		pl := Payload{
+			Num: i,
+		}
+		p.Submit(pl)
 	}
-	time.Sleep(1 * time.Second)
+	p.StopAndWait()
 	tracker.MarkAsDone()
 	r.Stop()
-
 }

Also I believe you can call tracker.MarkAsDone() once after p.StopAndWait() and not in deinit().

Actually this is not enough because this the Overall tracker and you have other trackers for each worker. so I believe you should add a for loop after p.StopAndWait() marking every tracker as done.

Ok, back on the laptop. What I am struggling with is to get it to wait correctly I suppose.
What is the correct mechanism to signal to the pool that I am done submitting jobs?
Or do I just wait until all jobs in the system are completed? I might do that by matching the results with what I submitted in the first place?

I am waiting for all results to come back before I finish.
https://gitlab.com/qnib-golang/scripts/workpool/-/merge_requests/1/diffs#b8a9c8aedc539a0e25040f3565c0893987a14203_92_110

Sorry I wrote my previous fix in a hurry and didn't think properly. It was my mistake to call StopAndWait() in main(). Reading from p.Results in main is correct. so your code was pretty good. it just needed 2 minor fixes:
diff --git a/pkg/lib/pool/pool.go b/pkg/lib/pool/pool.go
index 36f50c4..155fc84 100644
--- a/pkg/lib/pool/pool.go
+++ b/pkg/lib/pool/pool.go
@@ -33,8 +33,6 @@ func NewTrackers(pw progress.Writer, cnt int) *Trackers {

 func deinitWrk(trackers *Trackers) func(workerID int, tr *progress.Tracker) error { // worker deinit function
 	return func(workerID int, tr *progress.Tracker) error {
-		t := trackers.List[workerID]
-		t.MarkAsDone()
 		return nil
 	}
 }
@@ -90,8 +88,10 @@ func WorkerPool(cmd *cobra.Command, args []string) {
 		//tracker.(result.Value.Sqrt)
 		//fmt.Println("result:", result.Value)
 	}
-	time.Sleep(1 * time.Second)
 	tracker.MarkAsDone()
+	for _, tr := range wrkTracker.List {
+		tr.MarkAsDone()
+	}
 	r.Stop()

 }

Your new code seems more complicated for no reason. After the loop for result := range p.Results { ends you have received all results so you can mark all trackers as done and exit. Your previous code with my fixes are good I believe. Is something not working properly?


Generate typescript types and query/mutation hooks

Tasks

[ ] Generate typescript types using the introspected GraphQL schema
[ ] Generate typescript query/mutation hooks using parametrized queries and mutations using .graphql files

Resources

https://www.graphql-code-generator.com/

This may have more relevance since the Dynamic Schema (#18) is basically working now. With the code generator, it's possible to create higher level APIs on top of the dynamic graph.


apply go fmt

Proposed changes
I propose that we go fmt the code to increase formatting consistency, increase readability, and reduce whitespace noise in hactoberfest and future contributions.
Context
The golang community tends to align around the gofmt tool to provide consistent formatting for Go source code. (see: go fmt your code on The Go Blog)
Possible Implementation
go fmt could be run against the code on the main branch
this would possibly cause merge conflicts with any open PRs
Other information
while running this once on main would address formatting today, it would be a good idea to bake this into the development/contribution workflow, either through build scripts (e.g. Makefiles are pretty common in Go repos) or github actions in a PR workflow (e.g. this gofmt-action action seems to run gofmt and fail a PR if it detects any changes) or even suggesting a pre-commit hook script (see: e.g. this example)

closing as #13 seemed to resolve this.


make language controller

also fix constant inserting of languages upon restart by putting condition in data.sql to ignore if it already exists, cut off auto generated id for languages for that cause

tested via postman


WIP: replace bootstrap layout with css grid

CSS Grid header is visually identical to bootstrap version in production. Still working on the rest.

https://learnwhileyoupoopcom-ngjsjantgi.now.sh/


Improve seeking controls with keyboard

Show media control when any key that is associated with an event is pressed is a new feature and were added more options to seek with keyboard. Now left and right arrows skips -5 or 5 seconds instead of -15 or 15, shift + right or left arrow skips 10 and shift + ctrl + left or right skips 15 seconds.

Coverage decreased (-0.03%) to 71.743% when pulling 89246693e5ad122c3923842f738fa36ddd589677 on jhonatangomes:master into bff717832691dcee9313cc0dd6560697d7366232 on clappr:master.


Cookbook examples that demonstrate Vue + Client Side storage

This should maybe be 2 issues, but what about cookbook entries for "Vue + LocalStorage" and "Vue + IDB." In general it isn't a big deal, but I could see people being interested in seeing it in action anyway. As I said, this could be one CB entry or two. With IDB being a bit complex, it may make sense for there to be two distinct entries.

I think this would be great. It would be excellent if it could be one entry, but I understand why you might need to break it up as well.

What I'll do is start off with LocalStorage and get a feel for how long it is and make a judgement call then.

Hey! I just made an example with Vue and backgroundSync- I think that's different enough from using IDB for a different entry. I would also cover messaging back from sw with client.postMessage. Just to give you the heads up so we don't cover the same thing :) But this space is deep enough for a few entries, as you say.

Thank you for reminding me that I need to do this. ;) Honestly thought it may be a few weeks.

Just FYI, began work on this today. Hope to have first draft by end of next week.


prettier v3 and apply fixes to existing files

Closes https://github.com/kamilkisiela/graphql-hive/pull/2901 https://github.com/kamilkisiela/graphql-hive/pull/2173

doesn't this break the vscode plugin?

doesn't this break the vscode plugin?

seems to work after restarting VSCode. Config loading works fine, I guess because we are still using CJS.


render very low

bad performance when my list 30 item

you try to set animated={false}?

Hi @nppull, a workaround for your issue is @paolosantarsiero's suggestion. Or maybe some Flatlist related rendering methods. Can you please share a bit of your code for testing and reproducing?

Hi @nppull and @paolosantarsiero I've implemented a default loader that will appear meanwhile all heights are being calculated, and then it disappears, also the "customLoader" prop, for implementing your own loader. Can you please try using the newest version?


Timer seems bugged

Whenever I drop a clue, the infoboxes aren't rendering at all. I'm on a jagex account > which plays runelite. I've uninstalled and re-installed the plugin. No luck. The infoboxes simply aren't rendering at all.
Also, a clan member of mine mentioned that when they drop their clue(s), there is a time shown for 60 minutes, but when they logout and log back in, the timer drops to 16 minutes. No matter how many times they pick up the clue and drop it again, the clue timer remains at 16 minutes.

does it happen on <IP_ADDRESS>? the newest version?


Support Go remote module import syntax for catalog locations

Feature Suggestion
Standardize on supporting the Golang way of importing remote modules for catalog locations.
The syntax is more or less: <host>/<repo-path>/<file-path>. For example github.com/spotify/backstage/app-config.yaml, where github.com is the repo hosting provider, spotify/backstage is the repo path, and app-config.yaml is the file path. We could support this syntax without having to declare an explicit type for the location as well, and just determine it based on the host.
Possible Implementation
We probably end up needing custom logic for each hosting provider, at least it makes sense to start out there. So to begin with we'd only support github.com as the host part, and we'd expect the 2 first path components to be the repo path.
I haven't dug into the Go logic for determining the location, but it's possible that there's something there that we can re-use.
Context
Discussion in https://github.com/spotify/backstage/pull/1276#discussion_r439740243

With "standardize", do you mean that this would be the normalized form that we always convert to and store, no matter what form the user used originally?
And maybe I'm reading too much into it, but did you mean to stop with the type field entirely and instead deduce it based on the target? Because if so, I'm wondering if it'll really work out when a large number of (even close source custom) types of location are added as time goes on.

Main use-case for this would be to allow it to be entered in manually created Location/LocationRef entities, as the full URLs are a bit tedious to handle. I do think we probably need some normalized form to store too, it's possible that this might be a good candidate. And nah I still think we should have type, but I guess this could be put under a repo/remote/url type or something generic like that. It could also be the default type, so a Location entity would often just have a targets field with a couple of easy to parse (for the eyes) URLs.
I do think it could also be a good starting point for handling different types of input in the register component input field. Atm we assume it's github, but I think we'll have to deduce that based on the host, which will probably use logic similar to this.

I think the main way of entering the target value - both in the gui and in files - will be to copy the url from a browser.

Would be about making reading existing files nicer, as well as mixing different types of hosts.
Closing this for now anyway, don't think it's worth adding atm.


Using nginx this way is a kubernetes anti-pattern

The proper was to get outside access to your cluster apps is to use an ingress controller like ingress-nginx and define ingress resources per application. For SSL the proper way to add certificates is to use cert-manager.
Once cert-manager is configured with the dns01 challenge you can put a common annotation on the ingresses to request, apply and use SSL certificates.

+1

Feel free to read my mission statement on the containers I build:
https://github.com/onedr0p/containers#mission-statement

Personally I would not use Traefik and instead use ingress-nginx.
See my comments here:
https://old.reddit.com/r/kubernetes/comments/137u79r/ingress_controller_for_vanilla_k8s/jiy8r89/

@aleksasiriski I'm curious about the traefik recommendation - any particular reason to use it over nginx? Since most of the config is abstracted away by the ingress config anyways.

Personal preference, but go with either one. Some of the statements onedr0p mentions I can agree with but other not so much. So, again, it's personal preference for me but both are fine.

The author of this project is aware of this now and is up to them to wether or not to update their project, closing...


Clean up test_parser.py, use xFail instead of skip where appropriate

This is something I did while working on the Lark.scan implementation. I wanted to use xfail and decided that it should be quite useful for many other tests as well. I also deleted a few old comments/completely skipped tests suggesting features that will probably never be implemented.
I adjusted a few of the conditions based on the actual behavior of the code, for example cyk does actually support the tree less transformer argument. It doesn't have quite as big of a benefit, but it does work.

Why change the skip to xfail? xfail implies the test should be fixed in the future, i.e. failing for now but should work. (I'm sure you know pytest supports skip too)
Also, I'm wondering if this PR won't cause too many collisions with the lark.lark PR..

xfail implies the test should be fixed in the future, i.e. failing for now but should work

I somewhat disagree: While this is a possible meaning, I would in this case interpret it as these should fail, and if they don't the tests aren't actually testing for what they should, without any promises about the future. E.g. a test case that is designed to specifically need the dynamic lexer should not function without it. But if you are strongly against this interpretation, that's fine by me.

Also, I'm wondering if this PR won't cause too many collisions with the lark.lark PR..

I don't think so? That PR hasn't touched test_parser.py at all.

Oh right, sorry my bad.
As for xfail, I don't think it means that the test won't function, but rather that the test is correct but the source code isn't. So for example xfailing ambiguity on lalr would mean that we expect lalr to support ambiguity, and it's temporarily broken.
If it's not a bother, could you please change xfail into skip?

As for xfail, I don't think it means that the test won't function, but rather that the test is correct but the source code isn't. So for example xfailing ambiguity on lalr would mean that we expect lalr to support ambiguity, and it's temporarily broken.

This is a possible interpretation, and probably the more common one. But I don't think the feature itself is all too prescriptive in this regard.
As a concrete example, consider the tests that use the tree less transform parameter. These only work for lalr (and cyk) and should produce an error message for earley, meaning that the tests using this should not pass, i.e. they are expected to fail. We currently have no plans to ever implement this feature (AFAIK), but it's still an ok usecase of xfail IMO. If you think this is to confusing, I will revert that part of this PR.

Yes, using Earley in that case should produce an error, but usually that would be caught in an AssertRaises and not with the xfail mechanism.
But actually, if you feel strongly about it that's fine by me. I usually just use xfail as a loose TODO marking, so not much loss there.

Edit: I take it back. I just ran it in the console and got a huge wall of xfail warnings.. Let's just use skip, okay?

Sure, did that. Now it's a huge wall of skip warnings instead.


Fix the Let's Go button color

PR Type: ___
website update
FIX #41
Fix #40
Fix The "Let's go" button which is not visible in normal mode
Fix the Contributors Heading to the center

Affiliation: N/A

Ticket: N/A

Checklist:

[X] I've read Contributing Guidelines
[ X] I agree to follow the Code of Conduct

This looked good, why did you close the PR?

actually i made the PR in main branch that's why i closed this


Missing await keyword

The async scripts works very well, thank you!
One note though, is that it forgets to add the await  keyword on lines like this one:
const inputClassic = $(selectors.inputClassic);
where it should transform those into:
const inputClassic = await $(selectors.inputClassic);.
Do note that it does add the await  keyword in lines like that:
expect($(selectors.elementP1).getText()).toEqual('2.140%');
(to: expect(await $(selectors.elementP1).getText()).toEqual('2.140%');)

Thanks for reporting, this should indeed be changed, ideally to:
await expect($(selectors.elementP1)).toHaveText('2.140%');


Make work with new binding syntax

https://canjs.com/doc/can-stache-bindings.html

I'm pretty sure they work: https://canjs.com/doc/can-stache-bindings.event.html#on_SCOPE_PROP_EVENT_by_SCOPE_PROP__CALL_EXPRESSION_


Merge Release/1.6.3 to master

Summary

Concisely describe the issue and your solution:
Link to existing issue (type "#" to check):
Specify the tests done so far:

Solutions (Optional)
Please describe the details of your solution if it might be too complicated for reviewers.

Codecov Report

Merging #221 into master will decrease coverage by 2.21%.
The diff coverage is 27.44%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e45e94...5be9247. Read the comment docs.


Support executable compilation

A common configuration error when using rust-openssl is to reference headers and libraries from different versions which results in weird and hard to track down breakage. I want to handle this by compiling and running a little executable that compares the values of the version encoded in a #define in the headers and a function in the library.
Unfortunately, gcc-rs will currently only build staticlibs.

This is actually pretty intentional today because it's not often possible to actually run the executable that would be generated (e.g. almost all cross compilation scenarios).
The "most robust" thing I think would perhaps be something like gcc -E and then parsing the output if you just want to learn about various #define constants.

That's a fair point, but the check I want to make isn't necessary for compilation to continue, and since the vast majority of compilation is not crossed I'm fine with gcc-rs saying "no".
The gcc -E approach isn't going to work because I need one #define out of a header and the result of calling SSLeay() at runtime.

So this is one reason I added the get_compiler API where you could get the tool itself and add your own arguments and such, perhaps that would be good enough for now rather than adding a first-class API?

I also just wanted to suggest this feature. If you think cross-compilation is really a problem, you could offer the API to only compile executables to the host architecture (not to the cross-compiled architecture), which is the exact use case of @sfackler and me (dynamic build configuration).

I'm going to close this as it's not currently my intention to support this in this library as this crate is primarily for build scrips and building native code, especially handling the cross-compile cases.


fix(synchronise_tracking_and_event_data): fix check for missing playe…

closes #78

Maybe worth considering making -999 a global variable for missing data defined somewhere, as currently it's "hardcoded" in a bunch of places. But that's out of the scope of the issue at hand.

Tests added @Alek050


[FLUME-3473] fixed testMinimumRequiredSpaceTooSmallforPut test by changing buffer size dynamic

Current state: Since the buffer size is limited by the size
byte[] buffer = new byte[64 * 1024];

This was causing while loop to execute limited by buffer size. This was causing test failure since "checkpointDir.getUsableSpace() > minimumRequiredSpace" was still satisfied even after the byte size is exhausted.
The change in PR introduces dynamic size of buffer, so that always we get checkpointDir.getUsableSpace() <= minimumRequiredSpace post while loop and hence test always passes.
Tested it in pipeline many times and it passed always with this fix.
A snippet of TestLog-output.txt with extra logs to check if the condition is satisfied.
2023-05-21 19:40:50,774 (main) [INFO - org.apache.flume.channel.file.TestLog.doTestMinimumRequiredSpaceTooSmallForPut(TestLog.java:245)] checkpointDir.getUsableSpace():<PHONE_NUMBER>28
2023-05-21 19:40:50,774 (main) [INFO - org.apache.flume.channel.file.TestLog.doTestMinimumRequiredSpaceTooSmallForPut(TestLog.java:246)] minimumRequiredSpace:<PHONE_NUMBER>48
2023-05-21 19:40:50,791 (main) [INFO - org.apache.flume.channel.file.Log.put(Log.java:656)] usableSpace449352470528
2023-05-21 19:40:50,791 (main) [INFO - org.apache.flume.channel.file.Log.put(Log.java:657)] minimumRequiredSpace449358696528
2023-05-21 19:40:50,791 (main) [INFO - org.apache.flume.channel.file.Log.put(Log.java:659)] usableSpace <= requiredSpace

@rgoers please review it


Update aws-core, http-client-spi, ... to 2.17.112

Updates

software.amazon.awssdk:aws-core
software.amazon.awssdk:http-client-spi
software.amazon.awssdk:netty-nio-client

from 2.17.104 to 2.17.112.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "software.amazon.awssdk" } ]

labels: library-update, early-semver-patch, semver-spec-patch

Superseded by #26.


create virtual node edits

Preview here:

https://nadine.preview.docs.r3.com/en/platform/corda/5.0/application-networks/creating/mgm/virtual-node.html
https://nadine.preview.docs.r3.com/en/platform/corda/5.0/application-networks/creating/members/virtual-node.html

@yarivus in fairness, Joseph's edits came in late on Friday evening after regular hours. Nadine's first commit on the associated corrections was 9am this morning, as can be seen from commits above (before your prompt). It doesn't get much faster than that.


Dropdown Item: should be able to expand to multi-line if content is too wide

Info
Currently our Dropdown component does not expand to multi-line if the item given is too wide for the component. Breaking one of our tenants in the Figma Designs:

Acceptance Criteria:

If an item in the Dropdown is too long for the width of the Dropdown, it should be converted into multi-line, rather than adding an ellipsis.

This is completed in this PR: https://github.com/GovAlta/ui-components/pull/2296


fix #1193 - unified joint copyright notice in LICENSE file

resolves #1193 by amending the Nuxt.js copyright

What's the reason for adding a year? Basically, each year maintainers will need to update.
No point to keep Paul in license, his name is already in package.json file.

There is no use in adding a year indeed, even the whole notice doesn't change the fact that it is copyrighted as it is merely a hint and clarifies the obvious. If you're sure that the previous maintainer's notice can and should be removed, then I'll apply the changes RN. I simply kept both in there in descending order by date because I saw that somewhere in such a format (I believe it was a cryptocurrency fork).

I'm not a lawyer but this license specifically mentions that:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

That to me means that everything from the original license should be retained, including the original author.

I'm not a lawyer but this license specifically mentions that:

I know many modules were transferred previously to the nuxt community, and my suggestion was to keep it the same as in other repositories under same organization.

I'll just ping @paulgv in case he has something to say about that.

My other suggestion would be to add you too to the license and readme file 😉
Thanks!

So, we're likely gonna revert the suggestions one by one again? Where would we be without squash merging :slightly_smiling_face:

Thanks


Add feature for measuring duration of arbitrary code

Currently we have a scoped tracepoint, which allows us to measure an time of executing the code:
{
  ht_feature_callstack_start_string(timeline, "label");
  // some code
  ht_feature_callstack_start_string_stop(timeline);
}

Unfortunately we can't measure time of code which is overlapping; e.g. we can't measure time for code1+code2 and code2+code3 at the same time using the same timeline:
// code1
// code2
// code3

We should design and implement timeline feature, which would allow us to measure time of any code; something like:
HT_DurationId id1 = ht_feature_duration_start(timeline);
// code1
HT_DurationId id2 = ht_feature_duration_start(timeline);
// code2
HT_DurationNs duration1 = ht_feature_duration_stop(timeline, id1);
// code3
HT_DurationNs duration2 = ht_feature_duration_stop(timeline, id2);

I'm going to look in to this later today 👍

Hey @andrewrch any progress on that issue? :)

Hey 😊 I made a start on it on the train back on Tuesday. Started with the implementation of feature_callstack and worked from there. Getting some issues at the moment accessing events I put in a bag. Will spend a bit more time on it tonight/tomorrow and let you know if I need a hand.

Thanks for the update :) No rush, just wanted to know if you had a chance to look into this and if there's anything I could help you with.


Secondary modes not working

Somehow the secondary modes for the GoPro Hero 4 are not working... Also the Protune Settings are only working for the video mode and not for Photo or MultiShot :/ tried to find the settings but no succes

what firmware?

HD<IP_ADDRESS>.00

It seems that the urls are incomplete for current firmware. I sniffed the GP Hero4 urls with the latest firmware, please refer to the latest issue.

Is there any progress on solving that issue?

You can try this out: https://github.com/ztzhang/GoProWifiCommand

It's working, although you got some typos submode instead of sub_mode

It seems to be a firmware issue. Some firmware uses sub_mode instead.

I updated the repo with the latest URLs from the latest FW (HERO4 Black 3.00 July 30, 2015)


If DEV_MODE flag is set to true, bring up db as a container, else bring it up as a service

https://github.com/Shippable/pm/issues/10359

this is done, closing


Drop outdated and broken ClusterImageSets.

rc.3+ will work better, but quay rate limiting is affecting the ability
to install these releases.

/lgtm


AWQ and VILA dependency compatible issue

Hi Team,
I encountered some issues while trying to run AWQ and VILA. It appears that the dependencies for these two projects, based on the latest commits, have conflicts and compatibility issues. Specifically, AWQ seems to use older dependencies compared to VILA.
I would suggest moving to a Docker container and building these projects via Docker images. This approach is likely to help us avoid such dependency conflicts in the future.
Please advise on the best way to proceed.
Thank you for your consideration.

Seconding this. Running the instructions as posted leads to installation problems.

Can confirm this. VILA and AWQ have some dependencies issues if following installation instructions. Has someone found a solution yet?

I encountered the same problem, the installation of VILA now need updating

docker is 1 million time better than conda. Just provide a docker image, everything is done.


Add initial throughput and latency benchmark

For https://github.com/istio/ztunnel/issues/99
This introduces the first benchmarks. We have 3 throughput tests: direct (no proxy, baseline), TCP, and HBONE.
Example results:
$ cargo bench --profile release
   Compiling ztunnel v0.0.0 (/usr/local/google/home/howardjohn/go/src/istio.io/ztunnel)
    Finished release [optimized] target(s) in 1m 45s
     Running benches/throughput.rs (out/rust/release/deps/throughput-1d814c184fd554d8)
tcp/direct              time:   [27.686 ms 28.092 ms 28.529 ms]
                        thrpt:  [3.4230 GiB/s 3.4763 GiB/s 3.5273 GiB/s]
                 change:
                        time:   [+16.629% +19.708% +22.844%] (p = 0.01 < 0.05)
                        thrpt:  [-18.596% -16.463% -14.258%]
                        Performance has regressed.
tcp/hbone               time:   [337.63 ms 345.95 ms 354.01 ms]
                        thrpt:  [282.48 MiB/s 289.06 MiB/s 296.18 MiB/s]
                 change:
                        time:   [-36.935% -35.396% -33.782%] (p = 0.01 < 0.05)
                        thrpt:  [+51.016% +54.788% +58.567%]
                        Performance has improved.
tcp/tcp                 time:   [134.12 ms 138.60 ms 143.24 ms]
                        thrpt:  [698.13 MiB/s 721.49 MiB/s 745.58 MiB/s]
                 change:
                        time:   [-23.810% -21.124% -18.050%] (p = 0.01 < 0.05)
                        thrpt:  [+22.026% +26.781% +31.251%]
                        Performance has improved.

The echo server implementation is a bit slow; we can use a better impl once https://github.com/istio/ztunnel/issues/158 is done.

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@howardjohn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

test_ztunnel
e8a534c5186ff5b2ebe5dfeec4889a999ed1db81
link
true
/test test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

test_ztunnel
2fd04b1fafe17f49b45c0e6801d8c558a2a510e3
link
true
/test test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

this is broken for the same reason hbone e2e test is, I have a fix ill merge in another PR

/retest

@howardjohn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

test_ztunnel
2fd04b1fafe17f49b45c0e6801d8c558a2a510e3
link
true
/test test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


asyncio: make db reads non-blocking

kiss: db writings must be blocking.
readings, instead, can be ran in executors to avoid the rpc calls to block the event loop

there's no evidence this is worth the effort right now.


⚠ Bofrost.at has degraded performance

In 0b21e87, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1610 ms

Resolved: Bofrost.at performance has improved in c25b313.


Merge pull request #2 from google/master

Aligning
Checklist

[ ] Use a separate branch in your local repo (not master).
[ ] Test coverage is 100% (or you have a story for why it's ok).

Coverage remained the same at 100.0% when pulling ae1242851d081f61faa36fafe442ef3b01bd85f9 on sedobrengocce:master into 78d67b12af0f137c9691e8899da1d549e11e8691 on google:master.


Issue with orientation

When device is rotated with resets to 0 index.
Suppose i am at index 100 in portrait mode and when i rotate the device it shows the image at 0 index.
Does anyone face the same issue ?
Please provide the solution to it.

@rk28286 actually same issue for me.. :(


Funding yDaemon Team

Scope
yDaemon is an API, written in go, to provide an easy access to processed data from Yearn's ecosystem. It's now used in various projects and is becoming a milestone for our products:

yBribe
yCRV
Vaults at Yearn
Seafood
ySync
yData
Yearn Watch
Yearn SDK
Some other projects

yDaemon provide us with a full scale API, ready to compute various elements, from the vault list, to the harvest, from the prices to the meta elements and much more.
A full documentation is available here: https://ydaemon.yearn.farm
While relying mostly on the subgraph with the V1, we are now trying to bring a new version which would use mostly (target 90%) a direct connection with a set of node to compute the relevant data. The idea is to be able to bootstrap yDaemon in a few minutes, grab all the stuff we need to at least mimic what we have from the subgraph, while bringing more customisation, more routes, more.
Scope includes:

yDaemon API
yDaemon documentation
Risk framework
Vaults/Tokens/Protocols/Strategies metadata

Plan
The API will evolve with our needs, but here are some stuff we plan to work on

[ ] Merge the experimental branch to be the default one
[ ] Write an historical APY route based on all available events
[ ] Write a route to get, for a specific vaults, all the earnings for all strategies with the fee breakdown
[ ] Write "scoped-app", which could be started via env variable or argument, to be able to start only some process (ex: yBribe system only).
[ ] Build a load balancer system and some try-catch for multiple node architecture
[ ] Bring more indexers to catch new events as they are emited by the nodes and process them.
[ ] Fix the various issues, warnings and errors (prices, missing meta files, missing risk framework).
[ ] Build new routes for our needs, and new daemons to get the required data.
[ ] Write more tests
[ ] Much more

Deadline
2023-02-27
People

Major (no grant)
Dark (no grant)

Possible contributors:

Borrito
Murderteeth

Money
We would like to request a 9000 DAI grant for a 3 month initial period which will be used to pay contributors.
We plan to distribute a grant for each person in charge of a project and use bounties to fund other contributions linked to projects in maintenance mode or isolated contributions.
Amount
9000
Wallet address
0x0
Reporting
Monthly

Monthly Report for yDaemon - December
The experimental branch has been merged to the main branch, and furthermore, the indexers for vaults and strategies have been implemented. As mentioned in the above request, we plan to provide the historical APY and the breakdown of earnings for a given vault/strategy from the indexed events. To this end, we started with adding a simple pricing system using external APIs.
Many other PRs were done this month for maintaining the data integrity of yDaemon, namely the risk framework and the metadata of strategies and vaults. We share some examples for these maintanence efforts in the following.
Risk Framework

https://github.com/yearn/ydaemon/pull/153
https://github.com/yearn/ydaemon/pull/164
https://github.com/yearn/ydaemon/pull/167
https://github.com/yearn/ydaemon/pull/171

We discovered that there are a number of discrepancies in our perception of the risk scores and the actual rubric (e.g. high scores for Curve and Balancer pools), which kept us from adding the risk scores to the new frontend.
They will be handled together with the missing scores (e.g. StrategyCurveyCRV) this month.
Metadata

https://github.com/yearn/ydaemon/pull/162
https://github.com/yearn/ydaemon/pull/168
https://github.com/yearn/ydaemon/pull/174
https://github.com/yearn/ydaemon/pull/175


Could you please publish a new release?

Our builds broke since pinnacle updated to 0.3.5 yesterday. It'd be great if you could republish.
Also please consider taking the ^ away from the dependency definition to prevent future issues:
dependencies:
  pinenacl: ^0.3.5

Hi,
Sorry, I have just published the v0.2.2 and bumped the v0.3.1 codebase to the v0.4.0 due to the API break changes, to keep API breaks out of the minor releases.
I will keep the caret symbol as there should not be any API breaks in patch releases.
Even in minor releases but major, though it would be very inconvenient to have API breaks only in major releases.
Also, I retracted the relevant pinenacl-dart v0.3.5 and the bip32-ed25519-dart v0.3.1 packages from pub.dev.
So, pls use the v0.2.2 until you migrate (highly recommended) your code to the v0.4.0.
So, sorry for any inconvenience caused by this.
Cheers,
Pal


Disabled underline in diffs

Is there a way to disable the underline in the diffs? For newly added lines it underlines the whole line with a green line. I find this too distracting and can't read diffs properly because of it.
Sidenote, am loving the syntax highlighting in diff!

See also, #526

You can edit your personal stylesheet to change it. The CSS classes are .markup.added.diff and .markup.removed.diff.


zero position

I bought a few motors on aliexpress with modified versions of your controller. I discovered that the zero position on the motors is set at whatever the position it's at when the controller is powered on. In other words the zero position is forgotten once powered off. Is that the case with your original controller too?

nevermind, I just read through the position sensing of your thesis which explains how position is known at turn-on. My controller is using the same encoder so I guess there's something different in the firmware then.

nevermind, I just read through the position sensing of your thesis which explains how position is known at turn-on. My controller is using the same encoder so I guess there's something different in the firmware then.

Hi? Did you find how to solve the zero position problem?


Exclude React from a bundle

Hello,
I started to use create-react-app and it is simply amazing!
I use it for both creating apps and creating components bundle. In later case I was wondering if it is possible to exclude reacts lib from the bundle itself. I am including react using script tag in the main app.

Yes ,I also has the problem.

Yes ,I also has the problem.

No, this is currently not a supported use case. This project is mostly for building standalone apps rather than for building integrations into existing apps. For existing apps, setting up your own build pipeline (possibly based on this one) might be better anyway since there are usually more configuration needs in such projects.


RuntimeError: operator torchvision::nms does not exist

Prerequisites

[X] I have read the documentation.
[X] I have checked other issues for similar problems.

Backend
Colab
Interface Used
UI
CLI Command
No response
UI Screenshots & Parameters
No response
Error Logs
INFO     | 2024-06-28 13:00:09 | autotrain.cli.autotrain:main:58 - Using AutoTrain configuration: conf.yaml
INFO     | 2024-06-28 13:00:09 | autotrain.parser:post_init:133 - Running task: lm_training
INFO     | 2024-06-28 13:00:09 | autotrain.parser:post_init:134 - Using backend: local
INFO     | 2024-06-28 13:00:09 | autotrain.parser:run:194 - {'model': 'meta-llama/llama-2-7b-chat-hf', 'project_name': 'devmansur', 'data_path': 'data/', 'train_split': 'train', 'valid_split': None, 'add_eos_token': True, 'block_size': 1024, 'model_max_length': 2048, 'padding': 'right', 'trainer': 'default', 'use_flash_attention_2': False, 'log': 'tensorboard', 'disable_gradient_checkpointing': False, 'logging_steps': -1, 'eval_strategy': 'epoch', 'save_total_limit': 1, 'auto_find_batch_size': False, 'mixed_precision': 'fp16', 'lr': 0.0002, 'epochs': 1, 'batch_size': 1, 'warmup_ratio': 0.1, 'gradient_accumulation': 4, 'optimizer': 'adamw_torch', 'scheduler': 'linear', 'weight_decay': 0.01, 'max_grad_norm': 1.0, 'seed': 42, 'chat_template': None, 'quantization': 'none', 'target_modules': 'all-linear', 'merge_adapter': False, 'peft': True, 'lora_r': 8, 'lora_alpha': 32, 'lora_dropout': 0.05, 'model_ref': None, 'dpo_beta': 0.1, 'max_prompt_length': 128, 'max_completion_length': None, 'prompt_text_column': None, 'text_column': 'text', 'rejected_text_column': None, 'push_to_hub': False, 'username': 'abc', 'token': '', 'unsloth': False}
Saving the dataset (1/1 shards): 100% 4/4 [00:00<00:00, 422.24 examples/s]
Saving the dataset (1/1 shards): 100% 4/4 [00:00<00:00, 1860.83 examples/s]
INFO     | 2024-06-28 13:00:09 | autotrain.backends.local:create:8 - Starting local training...
INFO     | 2024-06-28 13:00:09 | autotrain.commands:launch_command:400 - ['accelerate', 'launch', '--num_machines', '1', '--num_processes', '1', '--mixed_precision', 'fp16', '-m', 'autotrain.trainers.clm', '--training_config', 'devmansur/training_params.json']
INFO     | 2024-06-28 13:00:09 | autotrain.commands:launch_command:401 - {'model': 'meta-llama/llama-2-7b-chat-hf', 'project_name': 'devmansur', 'data_path': 'devmansur/autotrain-data', 'train_split': 'train', 'valid_split': None, 'add_eos_token': True, 'block_size': 1024, 'model_max_length': 2048, 'padding': 'right', 'trainer': 'default', 'use_flash_attention_2': False, 'log': 'tensorboard', 'disable_gradient_checkpointing': False, 'logging_steps': -1, 'eval_strategy': 'epoch', 'save_total_limit': 1, 'auto_find_batch_size': False, 'mixed_precision': 'fp16', 'lr': 0.0002, 'epochs': 1, 'batch_size': 1, 'warmup_ratio': 0.1, 'gradient_accumulation': 4, 'optimizer': 'adamw_torch', 'scheduler': 'linear', 'weight_decay': 0.01, 'max_grad_norm': 1.0, 'seed': 42, 'chat_template': None, 'quantization': 'none', 'target_modules': 'all-linear', 'merge_adapter': False, 'peft': True, 'lora_r': 8, 'lora_alpha': 32, 'lora_dropout': 0.05, 'model_ref': None, 'dpo_beta': 0.1, 'max_prompt_length': 128, 'max_completion_length': None, 'prompt_text_column': 'autotrain_prompt', 'text_column': 'autotrain_text', 'rejected_text_column': 'autotrain_rejected_text', 'push_to_hub': False, 'username': 'abc', 'token': '', 'unsloth': False}
Traceback (most recent call last):
File "/usr/local/bin/accelerate", line 5, in
from accelerate.commands.accelerate_cli import main
File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 19, in
from accelerate.commands.estimate import estimate_command_parser
File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/estimate.py", line 34, in
import timm
File "/usr/local/lib/python3.10/dist-packages/timm/init.py", line 2, in
from .layers import is_scriptable, is_exportable, set_scriptable, set_exportable
File "/usr/local/lib/python3.10/dist-packages/timm/layers/init.py", line 8, in
from .classifier import ClassifierHead, create_classifier, NormMlpClassifierHead
File "/usr/local/lib/python3.10/dist-packages/timm/layers/classifier.py", line 15, in
from .create_norm import get_norm_layer
File "/usr/local/lib/python3.10/dist-packages/timm/layers/create_norm.py", line 14, in
from torchvision.ops.misc import FrozenBatchNorm2d
File "/usr/local/lib/python3.10/dist-packages/torchvision/init.py", line 6, in
from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils
File "/usr/local/lib/python3.10/dist-packages/torchvision/_meta_registrations.py", line 164, in
def meta_nms(dets, scores, iou_threshold):
File "/usr/local/lib/python3.10/dist-packages/torch/library.py", line 440, in inner
handle = entry.abstract_impl.register(func_to_register, source)
File "/usr/local/lib/python3.10/dist-packages/torch/_library/abstract_impl.py", line 30, in register
if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist
INFO     | 2024-06-28 13:00:12 | autotrain.parser:run:199 - Job ID: 22375
Additional Information
No response

in colab, you need to update torch. run this before everything:
pip install -U torch

in colab, you need to update torch & torchvision. run this before everything:
pip install -U torch torchvision

Thank you,
I tried, Same error response.

sorry, it needs to be done after.

After all the block? Or the firstblock?

which colab notebook are you using? :)

which colab notebook are you using? :)

https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_LLM.ipynb

change first cell to:
#@title 🤗 AutoTrain LLM
#@markdown In order to use this colab
#@markdown - upload train.csv to a folder named `data/`
#@markdown - train.csv must contain a `text` column
#@markdown - choose a project name if you wish
#@markdown - change model if you wish, you can use most of the text-generation models from Hugging Face Hub
#@markdown - add huggingface information (token) if you wish to push trained model to huggingface hub
#@markdown - update hyperparameters if you wish
#@markdown - click `Runtime > Run all` or run each cell individually
#@markdown - report issues / feature requests here: https://github.com/huggingface/autotrain-advanced/issues

import os
!pip install -U autotrain-advanced > install_logs.txt 2>&1
!pip install -U torch torchvision
!autotrain setup --colab > setup_logs.txt
from autotrain import __version__
print(f'AutoTrain version: {__version__}')

change first cell to:
#@title 🤗 AutoTrain LLM
#@markdown In order to use this colab
#@markdown - upload train.csv to a folder named `data/`
#@markdown - train.csv must contain a `text` column
#@markdown - choose a project name if you wish
#@markdown - change model if you wish, you can use most of the text-generation models from Hugging Face Hub
#@markdown - add huggingface information (token) if you wish to push trained model to huggingface hub
#@markdown - update hyperparameters if you wish
#@markdown - click `Runtime > Run all` or run each cell individually
#@markdown - report issues / feature requests here: https://github.com/huggingface/autotrain-advanced/issues

import os
!pip install -U autotrain-advanced > install_logs.txt 2>&1
!pip install -U torch torchvision
!autotrain setup --colab > setup_logs.txt
from autotrain import __version__
print(f'AutoTrain version: {__version__}')

It works. Thank you.

change first cell to:
#@title 🤗 AutoTrain LLM
#@markdown In order to use this colab
#@markdown - upload train.csv to a folder named `data/`
#@markdown - train.csv must contain a `text` column
#@markdown - choose a project name if you wish
#@markdown - change model if you wish, you can use most of the text-generation models from Hugging Face Hub
#@markdown - add huggingface information (token) if you wish to push trained model to huggingface hub
#@markdown - update hyperparameters if you wish
#@markdown - click `Runtime > Run all` or run each cell individually
#@markdown - report issues / feature requests here: https://github.com/huggingface/autotrain-advanced/issues

import os
!pip install -U autotrain-advanced > install_logs.txt 2>&1
!pip install -U torch torchvision
!autotrain setup --colab > setup_logs.txt
from autotrain import __version__
print(f'AutoTrain version: {__version__}')

Hey, unfortunately, I have the same problem. I have already tried to install the package manually, but I still get the same error. Is the Colab still working for you or is the problem recurring?
Thanks for your help :)

Still have the same issue.
On Sat, 6 Jul 2024, 05:37 Nicklas Matzulla, @.***>
wrote:

change first cell to:
@.*** 🤗 AutoTrain LLM
@.*** In order to use this colab
@.*** - upload train.csv to a folder named data/
@.*** - train.csv must contain a text column
@.*** - choose a project name if you wish
@.*** - change model if you wish, you can use most of the text-generation models from Hugging Face Hub
@.*** - add huggingface information (token) if you wish to push trained model to huggingface hub
@.*** - update hyperparameters if you wish
@.*** - click Runtime > Run all or run each cell individually
@.*** - report issues / feature requests here: https://github.com/huggingface/autotrain-advanced/issues
import os
!pip install -U autotrain-advanced > install_logs.txt 2>&1
!pip install -U torch torchvision
!autotrain setup --colab > setup_logs.txt
from autotrain import version
print(f'AutoTrain version: {version}')
Hey, unfortunately, I have the same problem. I have already tried to
install the package manually, but I still get the same error. Is the Colab
still working for you or is the problem recurring?
Thanks for your help :)
—
Reply to this email directly, view it on GitHub
https://github.com/huggingface/autotrain-advanced/issues/692#issuecomment-2211523669,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACARJUJC5YB6DG3STKPIZFDZK4YMFAVCNFSM6AAAAABKB337QKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJRGUZDGNRWHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

#@title 🤗 AutoTrain LLM
#@markdown In order to use this colab
#@markdown - upload train.csv to a folder named `data/`
#@markdown - train.csv must contain a `text` column
#@markdown - choose a project name if you wish
#@markdown - change model if you wish, you can use most of the text-generation models from Hugging Face Hub
#@markdown - add huggingface information (token) if you wish to push trained model to huggingface hub
#@markdown - update hyperparameters if you wish
#@markdown - click `Runtime > Run all` or run each cell individually
#@markdown - report issues / feature requests here: https://github.com/huggingface/autotrain-advanced/issues

import os
!pip install -U autotrain-advanced > install_logs.txt 2>&1
!autotrain setup --colab > setup_logs.txt
!pip install -U torch==2.4.0 torchvision
from autotrain import __version__
print(f'AutoTrain version: {__version__}')

This works for me, the pip install line needs to be after the autotrain setup line. And for some reasons newer versions of torch seems not working for me.


[FEATURE] Allow for source to be specified for JobTaskSqlTaskFile

Problem Statement
We should be able to specify the source location of a sql task file via JobTaskSqlTaskFile class. Right now source attribute is missing.
Proposed Solution
Modify the class as follows
class JobTaskSqlTaskFile(BaseModel):
    """
    Job Task SQL Task File specifications

    Attributes
    ----------
    path:
        SQL filepath
    source:
        Location type of the notebook, can only be WORKSPACE or GIT. When set to WORKSPACE, the notebook will be
        retrieved from the local Databricks workspace. When set to GIT, the notebook will be retrieved from a Git
        repository defined in git_source. If the value is empty, the task will use GIT if git_source is defined and
        WORKSPACE otherwise.
    """

    path: str = None
    source: Literal["WORKSPACE", "GIT"] = None

Additional Context
No response

Fixed in https://github.com/okube-ai/laktory/pull/279