proposal: Go 2: goroutine collector or closer to prevent goroutine leaks

Memory management in Go is implicit and this is considered to be a good thing. I think this needs to be kept in mind when discussing language features that are implicit vs explicit.
Memory leaks are one of the issues that an implicit garbage collector is able to prevent. All memory is accounted for in Go.
However, goroutines leaks have replaced memory leaks and just like manual memory-managment, goroutine-management is a pain. Just like in C where developers must think about the lifetime of any memory they allocate. In Go, developers need to think about the lifetime of any goroutine they create. In both cases, this is done explicitly. In C with free and in Go with context.Context.  Both are optional and can result in leaks.
@networkimprov notes this about goroutine leaks.

for stalled I/O syscalls, the leak encompasses an OS thread, which is typically a lot more memory than a goroutine.

The question would be then, if Go has an implicit garbage collector, why doesn't it have an implicit goroutine collector.

    // 2. subsequent blocking IO operations inside that goroutine panic with "end of goroutine".
    // 3. blocking channel operations panic with "end of goroutine".
    job = nil

    //Prevent unused variable error.
    fmt.Println(job) //Output: nil
}

To extend the lifetime of a goroutine, assign it to a global variable or appropriately scoped value (ie a http.Request).
To cancel a goroutine, remove any references to it that are held by your program.
This proposal would also reduce the need of context.Context being added to every function signature and addresses some of the issues of #28342
If backwards compatabillity is desired, go calls made as a statement should not have their goroutines collected. As for future compatabillity, the goroutine type is flexible and other functionality for dealing with goroutines (ie return values, recovering from panics, and/or priority) could be added in the future without breaking compatabillity.
Granted, this wouldn't prevent all goroutines leaks ie (infinite loops without blocking operations or error handling) but I believe it would catch most real-world cases.
This proposal is similar to #27982 however it doesn't have the same objections as that proposal.
Namely, there is no parent-child relationship between goroutines and this is not exactly syntactic sugar for contexts.
Closable goroutines
An alternative to having a goroutine collector, is to have closable goroutines which are closed with the builtin close() function.
When closed they act as if they were collected by a goroutine collector as described above.
See comment below.
var job = go DoSomething()

//when a value of the goroutine type is closed.
// 1. blocking IO operations inside that goroutine return an "goroutine closed" error.
// 2. subsequent blocking IO operations inside that goroutine panic with "goroutine closed".
// 3. blocking channel operations panic with "goroutine closed".
close(job)

FAQ
What blocking operations return an error or panic?
The only blocking IO operations that raise an error or panic are channel operations and system-call backed functions that can return an error such as the methods on os.File or net.Conn, all other blocking operations block as usual.

The question would be then, if Go has an implicit garbage collector, why doesn't it have an implicit goroutine collector.

I don’t understand this statement, go does have such a facility — when a goroutine exits any memory references that are now unreachable are freed. This includes the goroutines stack and it’s associated runtime data structures.
What I think you’re asking for is explicit goroutine collection, that is, a way to cause a goroutine to exit this freeing its resources. Is that what you are asking for?

go does have such a facility — when a goroutine exits any memory references that are now unreachable are freed.
@davecheney that is garbage collection, unreachable memory is automatically freed by the runtime. I am referering to cleaning up 'zombie' goroutines that are no longer completing any useful function because whatever created them has dropped every reference to them presumably because the work is complete, cancelled or there was a panic.

I suppose it's also addressing #5308 in some form.

Interesting idea.
If a collected goroutine panics due to a blocking channel operation, that will most likely terminate the program.  Is that what we want?
What if the goroutine is holding a lock when it is collected, then it is quite possible that the lock will not be released, and the program will deadlock.  Is that what we want?  It seems hard to debug.
There are quite a few ways that a goroutine can block.  It can be executing a blocking system call, or it can be blocked in cgo code.  It can be blocking trying to acquire a mutex.  It can be sleeping due to time.Sleep.  We would have to identify each possibility and figure out how to handle it.  That is a lot of library interaction for a language feature.

Thanks for confirming. I have two additional questions;

What are your criteria for identifying zombie goroutines?
What is preventing those zombie goroutines from exiting?

@ianlancetaylor

If a collected goroutine panics due to a blocking channel operation, that will most likely terminate the program. Is that what we want?

That's not ideal, I think this type of panic could be automatically recovered by the goroutine before exiting.

What if the goroutine is holding a lock when it is collected, then it is quite possible that the lock will not be released, and the program will deadlock. Is that what we want? It seems hard to debug.

You would expect locks to be released with defer, on errors, or after recovering from a panic.

There are quite a few ways that a goroutine can block.

The proposal only applies to channel operations and system IO that can return an error such as methods on os.File and net.Conn. Any other blocking operations do not panic and will continue to block as usual.
@davecheney

What are your criteria for identifying zombie goroutines?

This proposal adds goroutine reference types that are returned by go invocations, a goroutine is considered to be a 'zombie' when no references to it exist. If I want a goroutine to exist, I keep a reference to it somewhere. If I have no references to the goroutine then it should stop.

What is preventing those zombie goroutines from exiting?

Some 'zombie' goroutines may exit just fine. Others may be blocked on a IO operation that is taking a very long time or are waiting to recieve on a channel that has no senders anymore. In other words, they think they still have work to do.
Collecting a goroutine doesn't terminate it, the runtime just moves it along so to speak because what it's doing is no longer important. It does this by preventing IO & channel operations so that the goroutine eventually 'gives up' and exits.

This proposal adds goroutine reference types that are returned by go invocations, a goroutine is considered to be a 'zombie' when no references to it exist. If I want a goroutine to exist, I keep a reference to it somewhere. If I have no references to the goroutine then it might as well not exist to me

So this is the flow of logic
ref := go something()
ref = nil // kills goroutine

Yea? Something like this?

Others may be blocked on a IO operations that are taking a very long time or are waiting to recieve on a channel that has no senders anymore. In other words, they think they still have work to do.

This doesn't sound like the reference to a goroutine model you described earlier. It sounds different.

Collecting a goroutine doesn't terminate it, the runtime just moves it along so to speak because what it's doing is no longer important.

Could you be more precise about "just moves it along". Do you mean send some kind of signal too it?

To leverage this, we'd probably keep a map of active goroutines, and therefore name them. Terminating a goroutine then looks like delete(grMap, "process1"). Do you agree, or envision a different scheme?
Wouldn't it be essential in some cases to discover what "process1" is blocked on before deleting it?
The proposal doesn't reference any of the recent discussion re interrupting blocking file ops and/or syscalls (e.g. #41054), but seems to incorporate that goal. Is that accurate?

runtime just moves it along ... by preventing IO & channel operations so that the goroutine eventually 'gives up' and exits.

That could become a CPU-intensive loop if any code is retrying on error. And you can't continue past a mutex that didn't lock.

@davecheney

So this is the flow of logic

Yes, this is what I was illustrating with the example:
//goroutine is a new builtin 'non-comparable reference type' that is returned by go calls.
var job goroutine = go DoSomething()

//when a value of the goroutine type is *collected* by the garbage collector,
// 1. blocking io operations inside that goroutine return an "end of goroutine" error.
// 2. blocking channel operations panic with "end of goroutine".
job = nil

This doesn't sound like the reference to a goroutine model you described earlier. It sounds different.

I gave some examples of why a goroutine wouldn't exit on its own. Whether or not a goroutine will exit is a halting problem and does not have anything to do with being a 'zombie' in this context.
The goroutine is classified as a 'zombie' or 'collectable' strictly when there are zero references to it.

Could you be more precise about "just moves it along". Do you mean send some kind of signal too it?

Not a signal. Once a goroutine has been 'collected' or marked as a 'zombie' under this proposal, it is not allowed to perform IO operations such as read/write on files or connections. It is also not allowed to block during a send or recieve channel operation.
Any IO operations that the goroutine makes will return an error such as "end of goroutine".
Any blocking channel operations (read or write) that the goroutine makes will panic with a message such as "end of goroutine".
The rationale here is that the goroutine is able to handle these errors/panics by cleaning up and exiting.
Whoever has the reference to the goroutine effectively has control over its lifetime and when references are freed, goroutines will be able to stop without any extra coordination or channels having to be passed throughout the call stack.
It also addresses the common kinds of goroutine leaks that developers make when working with goroutines.
Here is an example from https://www.ardanlabs.com/blog/2018/11/goroutine-leaks-the-forgotten-sender.html
// [Adapted from Listing 1 in the article]
// leak would normally be a buggy function. It launches a goroutine that
// blocks receiving from a channel. Nothing will ever be
// sent on that channel and the channel is never closed so
// that goroutine would be blocked forever.
func leak() {
	ch := make(chan int)

	//The returned reference is not stored anywhere so the goroutine will be collected.
	//The channel operation will panic and the goroutine will exit, preventing the goroutine leak.
	_ := go func() {
		defer recover() //Recover is added to prevent the panic from bringing down the entire program.
		val := <-ch
		fmt.Println("We received a value:", val)
	}()
}

@networkimprov

To leverage this, we'd probably keep a map of active goroutines, and therefore name them. Terminating a goroutine then looks like delete(grMap, "process1"). Do you agree, or envision a different scheme?

Yes for globally scoped routines that could make sense, or a map to a slice of goroutine values so that you can associate a job  with all the goroutines that are related to that job. If you wanted to cancel the job, you delete it from the map and all the goroutines would get collected and if written correctly would cleanup and exit.

Wouldn't it be essential in some cases to discover what "process1" is blocked on before deleting it?

I think you need to provide an example of such a case.

The proposal doesn't reference any of the recent discussion re interrupting blocking file ops and/or syscalls (e.g. #41054), but seems to incorporate that goal. Is that accurate?

Yes, this proposal can address situations where a goroutine is blocked for a long time on a resource without any SetTimeout methods, set a timeout in the map of active jobs for deletion and the goroutine will return an error on the blocking resource and the goroutine should exit.

That could become a CPU-intensive loop if any code is retrying on error. And you can't continue past a mutex that didn't lock.

Well, if the goroutine repeats a blocking IO operation after an error was returned, then it should panic. A "end of goroutine" needs to be handled correctly the first time.

Wouldn't it be essential in some cases to discover what "process1" is blocked on before deleting it?

For example, a process that alternately waits on two resources, one which blocks indefinitely, the other which blocks for short times, but can become stalled due to external failure. You only want to "collect" this goroutine in the latter case.
You could keep a status variable alongside the goroutine reference, and update it before & after every blocking op, or the runtime could let you inspect what any goroutine is currently blocked on.

Wouldn't it be essential in some cases to discover what "process1" is blocked on before deleting it?

Process the resource that could stall inside of a timeout and pass a buffered status channel so that you can detect if the goroutine was ended due to timeout.
package main

import (
	"fmt"
	"io/ioutil"
	"os"
	"time"
)

//UnreliableResource represents a function that could block for a long time.
func UnreliableResource(status chan error) {
	f, err := os.Open("/path/to/unreliable/resource")
	if err != nil {
		status <- err
		return
	}

	b, err := ioutil.ReadAll(f)
	if err != nil {
		status <- err
		return
	}

	status <- err
	return
}

//timeout ends the given goroutine after the given duration
//timeout must be given the only reference to g, ie go timeout(time.Second, go DoSomething())
//otherwise this function does nothing.
func timeout(duration time.Duration, g goroutine) {
	time.Sleep(duration)
}

func main() {
	var status = make(chan error, 1)
	go timeout(30 * time.Second, go UnreliableResource(status))

	//if there is a timeout, error will be "end of goroutine"
	if err := <-status; err != nil {
		fmt.Println(err)
	}
}

Compare #19702.

I don't think this proposal is semantically appropriate. The purpose of garbage collection is to “[simulate] a computer with an infinite amount of memory.” On such a system, the blocked goroutines would never be collected, and therefore would never panic or return an error.
We violate that abstraction today using finalizers, but finalizers introduce a lot of other problems (such as #9442, #41362, #7546). I think we should seek to minimize their use, not double-down on them as a mechanism for cancellation.
From that standpoint, #19702 was a stronger proposal: it sought to make the behavior of the program more like a computer with infinite memory, not less so.

@bcmills
I understand the concern, it's related to #38057
To avoid losing the abstraction, goroutines could be explicitly closed with close (then the developer is free to catch leaking goroutines with runtime.SetFinalizer) but this starts to look like a different proposal.
var job = go DoSomething()

//when a value of the goroutine type is closed.
// 1. blocking IO operations inside that goroutine return an "goroutine closed" error.
// 2. subsequent blocking IO operations inside that goroutine panic with "goroutine closed".
// 3. blocking channel operations panic with "goroutine closed".
close(job)

This reads nicely and a timeout function is easier to reason about.
//timeout closes the given goroutine after the given duration
func timeout(duration time.Duration, job goroutine) {
	time.Sleep(duration)
 	close(job)
}
go timeout(time.Second, go UnreliableWork())

Then there is no need for a special goroutine type, the go invocation could instead return a 'returning channel' that will return the result of the function. Closing this channel would close the goroutine. Although you would need multi-value channels for multiple return values which is a little odd.
func DoSomething() (*Result, error) {
 	return nil, errors.New("failed to do anything")
}

var status chan(*Result, error) = go DoSomething()

result, err <- status

This is more like a builtin context.Context.
Should we continue the disccusion here, in #28342 or is it worth creating a new proposal?

Why not to use context ?
ctx, cancel := context.WithCancel(context.Background())
go DoSomething(ctx)

time.Sleep(timeout)
cancel()

(btw, DoSomething can be easily found in context package documentation, coincidence? https://godoc.org/context)

Why not to use context?

See #28342, there are a number of challenges with context, for example:

It creates two classes of functions, ones that know about context and ones that don't.
Every function that needs to be cancellable needs to have a ctx context.Context as an argument.
Unlike with errors, it has no builtin support for its usecase.
ctx context.Context stutters.
Needs to be handled and used correctly otherwise there will be goroutine leaks.
The name can be confusing and doesn't imply that any kind of cancellation is involved.
Is easy to abuse as a defacto untyped goroutine local-storage.

Inexerienced Go developers can easily leak goroutines. If DoSomething is leaky, context will not preven this type of leak:
func DoSomething(ctx context.Context) {
    var leak chan struct{}
    <- leak
}

Closing or collecting the goroutine will.

Is the rough goal to ensure that you don't get goroutine leaks?
If that's the case, then you could have approaches to solving it. For example https://github.com/egonelbre/antifreeze implements a runtime checker for goroutines that were stuck waiting for too long.
In a similar fashion you could have something that monitors goroutines to finish in time, e.g. https://play.golang.org/p/SbBEi4OYOH7.

Wouldn't it be essential in some cases to discover what "process1" is blocked on before deleting it?
For example, a process that alternately waits on two resources, one which blocks indefinitely, the other which blocks for short times, but can become stalled due to external failure. You only want to "collect" this goroutine in the latter case.

Process the resource that could stall inside of a timeout and pass a buffered status channel so that you can detect if the goroutine was ended due to timeout.

I expect synchronous code like we'd write today. Besides that, before you deliberately terminate a goroutine, you may want to log a stack trace. You could also use that info to decide whether to terminate at all.
grMap["process1"] = go func(a, b tResource) {
   for {
      v, err := waitForever(a)
      if err != nil { break }
      err = waitaMoment(v)
      if err != nil { return }
      v, err = waitForever(b)
      ...
      err = waitaMoment(v)
      ...
   }
}(x, y)

t := time.NewTicker(d)
for {
   <-t.C
   for key, proc := range grMap {
      blkr := proc.Blocker()
      if blkr.Time > max && blkr.Top != waitForever { // .Top is function at top of stack
         delete(grMap, key)
      }
      proc = nil
   }
}

@egonelbre, note that a runtime check for permanently-blocked goroutines was proposed (and, IMO unfortunately, rejected) in #37681.

@Splizard you could also note in the issue that for stalled I/O syscalls, the leak encompasses an OS thread, which is typically a lot more memory than a goroutine.

The proposal here seems too subtle to me.
It seems odd to have a different behavior based on whether the result of the go statement is assigned to a variable.  Presumably if the result is not assigned to a variable it is as though it is assigned to a variable that is never changed, rather than having the value discarded (and therefore garbage collected).  That seems like the opposite of the natural default.
It seems confusing to have the behavior change specifically for channel operations and for I/O syscalls.  The latter isn't all that well defined in any case.
There may well be ways to improve goroutine cancellation and leaks.  But this specific proposal seems to me to be tweaking the language to support a particular use case, rather than providing a general and orthogonal mechanism.

Based on the discussion above, this is a likely decline.  Leaving open for four weeks for final comments.

It seems odd to have a different behavior based on whether the result of the go statement is assigned to a variable.

This is addressed by explicitly closing goroutines instead of implcitily collecting them, so if the collector option of this proposal is a likely decline, should I create a new proposal about explicitly closeable goroutines?
var job = go DoSomething()

//when a value of the goroutine type is closed.
// 1. blocking Read/Write operations inside that goroutine return an "goroutine closed" error.
// 2. subsequent blocking Read/Write operations inside that goroutine panic with "goroutine closed".
// 3. blocking channel operations panic with "goroutine closed".
close(job)

It seems confusing to have the behavior change specifically for channel operations and for I/O syscalls.

The rationale is that a closed goroutine no longer has permission to block and it should terminate as soon as possible.
Readers and Writers should return an error if they are running on a closed goroutine. Channel operations should panic.
This is the bare minumum that is required to be able to prevent the most common goroutine leaks.

to support a particular use case

What is the particular use case?

The rationale is that a closed goroutine no longer has permission to block and it should terminate as soon as possible.
Readers and Writers should return an error if they are running on a closed goroutine. Channel operations should panic.
This is the bare minumum that is required to be able to prevent the most common goroutine leaks.

What should happen if a goroutine is blocked waiting for a lock? What should happen if a goroutine currently holds this lock, but the author has chosen not to use defer to free that lock?

What should happen if a goroutine is blocked waiting for a lock?

It remains blocked. Unless the lock implementation uses channels, or blocking Read/Write calls in which case a panic or error should be raised and the lock will not be aquired.

What should happen if a goroutine currently holds this lock, but the author has chosen not to use defer to free that lock?

If they are reading or writing to an unbuffered channel while they are holding a lock, then the goroutine panics and the lock is leaked. Much like present Go behaviour, if a goroutine writes to a closed channel whilst holding a lock, it panics and the lock is leaked. The proposal extends this risk to reading from unbuffered channels too.
If you are holding a lock and you perform blocking channel operations, you must defer the release of the lock or catch any potential panics so that the lock doesn't leak.

Thank you for explaining. The restrictions on locks appear to undermine the utility of this proposal — goroutines waiting for a lock will continue to “leak”.

This is addressed by explicitly closing goroutines instead of implicitly collecting them, so if the collector option of this proposal is a likely decline, should I create a new proposal about explicitly closeable goroutines?

Let me first say clearly that I think that any proposal around closeable goroutines is unlikely to be accepted.
That said, if you want to pursue this, you should also address the other concerns, not just this one.

The rationale is that a closed goroutine no longer has permission to block and it should terminate as soon as possible.
Readers and Writers should return an error if they are running on a closed goroutine. Channel operations should panic.
This is the bare minumum that is required to be able to prevent the most common goroutine leaks.

Thanks, I think I understand that.  The problem is that it is very specific and is not orthogonal.  Certain cases work, other cases fail, and the distinction is based on the implementation rather than on any fundamental aspect of the language.  There is no deep reason why we say a goroutine will return an error from a Reader but not from, say, a sync.Once that is waiting for the function to complete.  I believe the effect will be confusing for users of the language, who will have to remember specific circumstances rather than a simple rule.

What is the particular use case?

Goroutines that only block for I/O or channel operations.

Further discussion, but no change in consensus.

Whoops, sorry, jumped the gun there.

No change in consensus.  This time for sure.


Upgrade to payment methods and API version 2019-03-14

Move card payments to payment methods: https://stripe.com/docs/api/payment_methods
Upgrade to latest API version: https://stripe.com/docs/upgrades#2019-03-14

Some nits, otw lgtm

looks great!

🎉


No error when using uninitialized value in noImplicitAny & strictNullChecks mode

From master as of right now (7b34b612beda66b0812462a3feeabc63852cd842)
var x;
var y = 100 + x;

Compile with strictNullChecks and noImplicitAny
Expected: Some sort of error for using x.
Actual: No error at all?

This is actually by design. Your var x is an implicit any variable that is tracked by control flow analysis (#11263). Because it hasn't been initialized, its control flow type in the 100 + x expression is undefined, and the + operator permits undefined as an operand value. If anything, we should discuss whether the numeric operators should be stricter with respect to undefined and null, but that's a different discussion.

Sure, that issue definitely sounds like something we should look into; is the type of x still going to be undefined under noImplicitAny?


Remove IREE solver

Removal of the IREE solver, which is not shipped with the wheels

I am going to wait to merge this until the removal is done in PyBaMM


Unable to click links

I am trying to click the logout link test the functionality.
I have tried a few things:
$("#log").click(function(e){
e.stopPropagation()
...
})
$("#log")[0].click()
Not sure what else to try. Any suggestions?

I found a workaround on the "Can't find variable: jQuery" error. Here in my jquery.go fork (yura415/jquery.go.js) I added "ensureJqueryIsLoaded" method, that checks jQuery availability and used it internally in waitForPage. See my examples/example6.js.


K.Dᴀᴠɪs: Watch dad(?:dy)?.*(?:fricc|r8p)

K.Dᴀᴠɪs requests the watch of the watch_keyword dad(?:dy)?.*(?:fricc|r8p). See the Metasmoke search here and the Stack Exchange search here.

dad(?:dy)?.*(?:fricc|r8p) has been seen in 0 true positives, 0 false positives, and 0 NAAs.

!!/approve

Coverage remained the same at 64.657% when pulling 9a903f6f9afa2f2e6737c0a6c381f336c56fe098 on auto-blacklist-1521524018.3332274 into 4d6fb16e817bf344cb7186b3e78603fc0980de4e on master.


ci: Disable binaries for Android task explicitly

In this repo GUI test are disabled implicitly as they make no sense for now.
Therefore, the test_bitcoin_qt target in the src/qt/Makefile is unavailable, that breaks CI Android task.
This PR fix this issue by explicitly defining a set of binaries via configure options, that allows to avoid relying on the test_bitcoin_qt target from the src/qt/Makefile.

@MarcoFalke
Is it reasonable to suggest this PR in the main repo?


Make sure we also test with django 2.1 and 2.0

This is a follow up to #19

@Normos @zsoldosp Test setup should now be correct, testing 2.0 and 2.1 separately.

LGTM - I would probably bump the minor version, though the actual app hasn't changed  so might be ok to republish, but not sure the release process supports that (plus tagging)


Prevent banning trusted peers for a long time

Describe the feature
ref #7052
with #7052 we provided a bit more leeway when penalizing misbehaving trusted peers, but we would still endup banning them, which basically puts them into a timeout for:
https://github.com/paradigmxyz/reth/blob/54449917b1dc31e996bbcf5dfbb0dc169eed18b8/crates/net/network/src/peers/manager.rs#L1217-L1218
TODO

ban misbehaving trusted peers for shorter durations:

https://github.com/paradigmxyz/reth/blob/54449917b1dc31e996bbcf5dfbb0dc169eed18b8/crates/net/network/src/peers/manager.rs#L299-L301
we should use the default uban interval here so trusted peers get unbanned with the next interval
Additional context
No response

I'd like to fix it


Recents widget with username colour (from usergroup)

As asked on ThemeForest, suggestion for changing the recents widget to include the username colour from the usergroup.
modify file:
ext/eles/coreframe/inc/widgets_service.php
alter (line 98):
u.username
to:
u.username,
after (line 98):
u.username,
new line (line 99):
g.group_colour
alter (line 104):
. USERS_TABLE . ' u ' . '
to:
. USERS_TABLE . ' u, '
For the following edit I'm not sure, as I used bb_groups g there since I don't know if the variable GROUPS_TABLE exists.
after (line 104):
. USERS_TABLE . ' u, '
new line (line 105);
. GROUPS_TABLE . ' g ' . '
after (line 114):
AND p.poster_id = u.user_id
new line (line 115):
AND u.group_id = g.group_id
alter (line 126):
$author = $row['username'];
to:
$author = "".$row['username']."";
after (line 126):
$author = "".$row['username']."";
new line (line 127):
$author_clean = $row['username'];
after (line 139):
'AUTHOR' => $author,
new line (line 140):
'AUTHOR_CLEAN' => $author_clean,
modify file:
ext/eles/coreframe/styles/eles/templates/widgets/recents.html
alter (line 7):
title="{L_POST_BY_AUTHOR} {c_widget_recents.AUTHOR} » {c_widget_recents.TIMESTAMP}"
to:
title="{L_POST_BY_AUTHOR} {c_widget_recents.AUTHOR_CLEAN} » {c_widget_recents.TIMESTAMP}"
That should be all. With the exception of using GROUPS_TABLE as commented this is working for me.

Hi!
Many thanks for your detailed guide!
I integrated this and will have it available in the next release.
Cheers


NIDAQmx

Hi,
I am not very experienced with C. I was wondering if you could give me some instruction on what I am doing wrong please?
 Gave same error. Win7.
cgogen -ccdefs C:\goworksp\src\local\NIDAQmx\nidaqmx.yml
Thank you for responding

Sorry to hear that it did not help. Possibly related cznic/cc#58 which has a pending PR cznic/cc#60.

@5k3105 Hi! A really quick fix would be to define __linux__ so the header won't be relying on compiler-specific types. Using compiler defined types for this case is like using a sledgehammer for cracking nuts.
PARSER:
  SourcesPaths: ["NIDAQmx.h"]
  Defines:
    __linux__: 1

Please close the issue if this helps. @cznic probably right, borrowing defines from compiler not yet available on windows.

@5k3105 also, your translator rules look messed up a bit, check out these:
TRANSLATOR:
  ConstRules:
    defines: expand
  Rules:
    global:
      - {action: accept, from: "^DAQmx"}
      - {action: replace, from: "^DAQmx"}
      - {transform: export}
    post-global:
      - {action: replace, from: _$}
      - {load: snakecase}

works beautifully

Wow guys! Thanks a lot!
 I think you may define something, there is two options:

Run go install this way: CGO_CFLAGS="-D__linux__=1" go install
Add platform-specific FlagGroups to cgogen config (see https://github.com/xlab/cgogen/wiki/Generator-config-section#flaggroups)

@5k3105 There are some bad news by the way. There is a lot, A LOT of methods that use ... variadic argument lists that are not supported in CGO. Normally there is two options to handle them: 1) ignore, 2) write a wrapper. Given their count, 2) is out of question, and 1) is probably too.
So for now, the quickest way to deal with it is to ignore:
      - {action: ignore, from: "^DAQmxGet"}
      - {action: ignore, from: "^DAQmxCreate"}
      - {action: ignore, from: "^DAQmxSet"}

You may add rules explicitly for variadic methods only, this should help you to get things working at least for anything else.
Note that ... variadics are not supported, but wrapper auto-generation can be done, this is scheduled on January holidays I guess. You can track progress in https://github.com/xlab/cgogen/issues/9

C:\goworksp\src\local\telem
λ set CGO_CFLAGS="-D__linux__=1"

C:\goworksp\src\local\telem
λ go install
# local/NIDAQmx
gcc: error: "-D__linux__=1": Invalid argument

These are the only calls I will use right now:
	nidaqmx.SelfTestDevice() // int32 __CFUNC     DAQmxSelfTestDevice            (const char deviceName[]);

	nidaqmx.CreateTask() // int32 __CFUNC     DAQmxCreateTask                (const char taskName[], TaskHandle *taskHandle);

	nidaqmx.CreateAIThrmcplChan() // int32 __CFUNC     DAQmxCreateAIThrmcplChan       (TaskHandle taskHandle, const char physicalChannel[], const char nameToAssignToChannel[], float64 minVal, float64 maxVal, int32 units, int32 thermocoupleType, int32 cjcSource, float64 cjcVal, const char cjcChannel[]);

	nidaqmx.ValDegC
	nidaqmx.ValTTypeTC
	nidaqmx.ValBuiltIn
	nidaqmx.ValGroupByScanNumber

	nidaqmx.StartTask() // int32 __CFUNC     DAQmxStartTask                 (TaskHandle taskHandle);

	nidaqmx.ReadAnalogF64() // int32 __CFUNC     DAQmxReadAnalogF64             (TaskHandle taskHandle, int32 numSampsPerChan, float64 timeout, bool32 fillMode, float64 readArray[], uInt32 arraySizeInSamps, int32 *sampsPerChanRead, bool32 *reserved);

	nidaqmx.StopTask() // int32 __CFUNC     DAQmxStopTask                  (TaskHandle taskHandle);

	nidaqmx.ClearTask() // int32 __CFUNC     DAQmxClearTask                 (TaskHandle taskHandle);

Ok, I think the env var took.

Using:
GENERATOR:
  PackageName: nidaqmx
  PackageDescription: "Package nidaqmx provides Go bindings for National Instruments NIDAQmx"
  PackageLicense: "THE AUTOGENERATED LICENSE.
  Includes: ["NIDAQmx.h"]

PARSER:
  SourcesPaths: ["NIDAQmx.h"]
  Defines:
    __linux__: 1

TRANSLATOR:
  ConstRules:
    defines: expand
  Rules:
    global:
      - {action: ignore, from: "^DAQmxGet"}
      - {action: ignore, from: "^DAQmxCreate"}
      - {action: ignore, from: "^DAQmxSet"}
      - {action: accept, from: "^DAQmx"}
      - {action: replace, from: "^DAQmx"}
      - {transform: export}
    post-global:
      - {action: replace, from: _$}
      - {load: snakecase}

Lots of lines like this:
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x1673d): undefined reference to `DAQmxStartNewFile'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16793): undefined reference to `DAQmxStartTask'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x167e9): undefined reference to `DAQmxStopTask'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16846): undefined reference to `DAQmxSwitchCloseRelays'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x168ae): undefined reference to `DAQmxSwitchConnect'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x1690b): undefined reference to `DAQmxSwitchConnectMulti'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16969): undefined reference to `DAQmxSwitchCreateScanList'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x169d1): undefined reference to `DAQmxSwitchDisconnect'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16a2e): undefined reference to `DAQmxSwitchDisconnectAll'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16a8b): undefined reference to `DAQmxSwitchDisconnectMulti'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16b06): undefined reference to `DAQmxSwitchFindPath'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16b79): undefined reference to `DAQmxSwitchGetMultiRelayCount'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16bec): undefined reference to `DAQmxSwitchGetMultiRelayPos'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16c4a): undefined reference to `DAQmxSwitchGetSingleRelayCount'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16ca8): undefined reference to `DAQmxSwitchGetSingleRelayPos'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16d05): undefined reference to `DAQmxSwitchOpenRelays'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16d63): undefined reference to `DAQmxSwitchSetTopologyAndReset'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16db9): undefined reference to `DAQmxSwitchWaitForSettling'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16e16): undefined reference to `DAQmxTaskControl'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16e6c): undefined reference to `DAQmxTristateOutputTerm'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16ec2): undefined reference to `DAQmxUnreserveNetworkDevice'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16f30): undefined reference to `DAQmxWaitForNextSampleClock'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x16f93): undefined reference to `DAQmxWaitUntilTaskDone'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x1703a): undefined reference to `DAQmxWriteAnalogF64'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x170be): undefined reference to `DAQmxWriteAnalogScalarF64'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17165): undefined reference to `DAQmxWriteBinaryI16'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x1720c): undefined reference to `DAQmxWriteBinaryI32'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x172b3): undefined reference to `DAQmxWriteBinaryU16'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x1735a): undefined reference to `DAQmxWriteBinaryU32'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17413): undefined reference to `DAQmxWriteCtrFreq'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x174a7): undefined reference to `DAQmxWriteCtrFreqScalar'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17560): undefined reference to `DAQmxWriteCtrTicks'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x175eb): undefined reference to `DAQmxWriteCtrTicksScalar'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x176a4): undefined reference to `DAQmxWriteCtrTime'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17738): undefined reference to `DAQmxWriteCtrTimeScalar'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x177df): undefined reference to `DAQmxWriteDigitalLines'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17861): undefined reference to `DAQmxWriteDigitalScalarU32'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17908): undefined reference to `DAQmxWriteDigitalU16'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x179af): undefined reference to `DAQmxWriteDigitalU32'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17a56): undefined reference to `DAQmxWriteDigitalU8'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17af2): undefined reference to `DAQmxWriteRaw'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17b65): undefined reference to `DAQmxWriteToTEDSFromArray'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17bcd): undefined reference to `DAQmxWriteToTEDSFromFile'
C:\Users\clott\AppData\Local\Temp\go-build291329218\local\NIDAQmx\_obj\nidaqmx.cgo2.o:nidaqmx.cgo2.c:(.text+0x17c2e): undefined reference to `DAQmxXSeriesCalAdjust'
collect2.exe: error: ld returned 1 exit status

@5k3105
Yeah, sure, you should link against a library. You're pretty close.
Depending on the environment, you must link against .dll, .so or .dylib
If you library is located under name libnidaqmx.so in /usr/local/lib then specify CGO_LDFLAGS="-L/usr/local/lib -lnidaqmx" I hope you see the pattern in these.
Another option is to fill the FlagGroups in generator's config that will specify LDFLAGS for your platform, there are many other options but they're much more advanced (and time consuming).
TIP! If you have pkg-config installed, you can take advantage of that, see example:
https://github.com/xlab/vorbis-go/blob/master/vorbis.yml#L6

@xlab Thank you for all your help!
λ go install
# local/NIDAQmx
..\NIDAQmx\const.go:2067: ReadWaitMode redeclared in this block
        previous declaration at ..\NIDAQmx\const.go:1519
..\NIDAQmx\const.go:2441: ValPulseTime redeclared in this block
        previous declaration at ..\NIDAQmx\const.go:2417
..\NIDAQmx\const.go:2445: ValPulseTicks redeclared in this block
        previous declaration at ..\NIDAQmx\const.go:2419

I was initially going to just delete these extra lines but it turns out they are indeed redeclared but with different values sometimes. So I just put a '2' behind the second instances of these and it stopped complaining.
My CGO_LDFLAGS looks like this: -L/goworksp/src/local/NIDAQmx -lnicaiu
But I get:
λ go install
# local/telem
C:\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:/TDM-GCC-64/bin/../lib/gcc/x86_64-w64-mingw32/5.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lnicaiu
collect2.exe: error: ld returned 1 exit status

I feel like I've tried every permutation of the above. It's a .dll file - "nicaiu.dll" I placed in the nidaqmx folder. I found an old post refering to this: http://forums.ni.com/t5/Multifunction-DAQ/How-Can-I-Use-The-NI-DAQmx-ANSI-C-Function-Library-With-GCC-on/td-p/195781
Funny thing is, even when I remove the '-lnicaiu' from the flags:
CGO_LDFLAGS =-L/goworksp/src/local/NIDAQmx
It still responds with cannot find -lnicaiu:
λ go install
# local/telem
C:\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:/TDM-GCC-64/bin/../lib/gcc/x86_64-w64-mingw32/5.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lnicaiu
collect2.exe: error: ld returned 1 exit status

That's strange indeed, ld must always be capable of finding nicaiu.dll in paths defined by -L.
:(

In the second link it mentioned:

Qt 4.7 with included mingw can use NIDAQmx.lib "as is" with LIBS += "\NIDAQmx.lib" statement in pro file.

Not sure what this means but I gave qt a try:
λ go install
# local/telem
C:\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:/Qt/Qt5.7.0/Tools/mingw530_32/bin/../lib/gcc/i686-w64-mingw32/5.3.0/../../../../i686-w64-mingw32/bin/ld.exe: unrecognised emulation mode: i386pep
Supported emulations: i386pe
collect2.exe: error: ld returned 1 exit status

You're going too deeply in the rabbit hole.
I think it's pretty solvable with plain MinGW64 + msys shell, with clean env variables. Just clean the mess.

Hi,
Got it to compile! (complete go build log at bottom)
host link: "gcc" "-m64" "-gdwarf-2" "-mconsole" "-o" "C:\\msys64\\tmp\\go-build984789194\\local\\telem\\_obj\\exe\\a.out.exe" "C:\\msys64\\tmp\\go-link-612005067\\go.o" "C:\\msys64\\tmp\\go-link-612005067\\000000.o" "C:\\msys64\\tmp\\go-link-612005067\\000001.o" "-L\\\\goworksp\\\\src\\\\local\\\\NIDAQmx" "-lnicaiu" "-g" "-O2" "-lmsvcrt" "-lm" "-mthreads" "-lwinmm" "-lws2_32" "-lntdll" "-ladvapi32" "-lkernel32"

Meaningful line:
"-L\\\\goworksp\\\\src\\\\local\\\\NIDAQmx"

Which I could not get to change through any of my env vars (.profile, .bash_profile, windows env vars). It should be /c/goworksp/src/local/NIDAQmx. Since it was missing the c, it couldn't find the .dll. Wound up putting dll in /c/usr/local/lib and it compiled.
So :) thank you!
I now have two problems. The SelfTestDevice which takes a string and returns an int (error code) is coming back with 'Device identifier is invalid.' Pretty sure this is the correct string I am sending it since I am just translating some python over - so same device name. Is there any way that string (const char deviceName[]) could get changed?
I am wondering if the way I changed the constants may be disturbing things (https://github.com/xlab/cgogen/issues/11#issuecomment-262314019). I just did a search and replace on the names but any of the functions using the old names will surely take the first value and not the second value I renamed to 'value2'.
Second - nidaqmx.CreateTask() // int32 __CFUNC     DAQmxCreateTask                (const char taskName[], TaskHandle *taskHandle);
This is the first function used in the code. It does not have a variadic argument but is also not generated because it starts with 'Create'. So there is a way I can include that single function? (- {action: accept, from: "^DAQmxCreateTask"}):
  Rules:
    global:
      - {action: accept, from: "^DAQmxCreateTask"}
      - {action: ignore, from: "^DAQmxGet"}
      - {action: ignore, from: "^DAQmxCreate"}
      - {action: ignore, from: "^DAQmxSet"}
      - {action: accept, from: "^DAQmx"}
      - {action: replace, from: "^DAQmx"}
      - {transform: export}

go build log:
clott@N-04711 MINGW64 /c/goworksp/src/local/telem
$ go build -v -x -ldflags "-L /c/goworksp/src/local/NIDAQmx -v"
WORK=C:\msys64\tmp\go-build984789194
local/telem
mkdir -p $WORK\local\telem\_obj\
mkdir -p $WORK\local\telem\_obj\exe\
cd C:\goworksp\src\local\telem
"C:\\Go\\pkg\\tool\\windows_amd64\\compile.exe" -o "C:\\msys64\\tmp\\go-build984789194\\local\\telem.a" -trimpath "C:\\msys64\\tmp\\go-build984789194" -p main -complete -buildid 6c79789db6d79a20215dc936c2f622a43fbe87f4 -D _/C_/goworksp/src/local/telem -I "C:\\msys64\\tmp\\go-build984789194" -I "C:\\goworksp\\pkg\\windows_amd64" -pack "C:\\goworksp\\src\\local\\telem\\main.go"
cd .
"C:\\Go\\pkg\\tool\\windows_amd64\\link.exe" -o "C:\\msys64\\tmp\\go-build984789194\\local\\telem\\_obj\\exe\\a.out.exe" -L "C:\\msys64\\tmp\\go-build984789194" -L "C:\\goworksp\\pkg\\windows_amd64" -extld=gcc -buildmode=exe -buildid=6c79789db6d79a20215dc936c2f622a43fbe87f4 -L /c/goworksp/src/local/NIDAQmx -v "C:\\msys64\\tmp\\go-build984789194\\local\\telem.a"
# local/telem
HEADER = -H10 -T0x401000 -D0x0 -R0x1000
searching for runtime.a in $WORK\runtime.a
searching for runtime.a in C:\goworksp\pkg\windows_amd64\runtime.a
searching for runtime.a in \c\goworksp\src\local\NIDAQmx\runtime.a
searching for runtime.a in C:\Go\pkg\windows_amd64\runtime.a
 0.00 deadcode
 0.02 pclntab=285276 bytes, funcdata total 44156 bytes
 0.02 dodata
 0.02 reloc
 0.03 reloc
 0.03 asmb
 0.03 codeblk
 0.04 datblk
 0.04 sym
 0.04 dwarf
 0.04 headr
 0.04 symsize = 0
host link: "gcc" "-m64" "-gdwarf-2" "-mconsole" "-o" "C:\\msys64\\tmp\\go-build984789194\\local\\telem\\_obj\\exe\\a.out.exe" "C:\\msys64\\tmp\\go-link-612005067\\go.o" "C:\\msys64\\tmp\\go-link-612005067\\000000.o" "C:\\msys64\\tmp\\go-link-612005067\\000001.o" "-L\\\\goworksp\\\\src\\\\local\\\\NIDAQmx" "-lnicaiu" "-g" "-O2" "-lmsvcrt" "-lm" "-mthreads" "-lwinmm" "-lws2_32" "-lntdll" "-ladvapi32" "-lkernel32"
C:\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:/TDM-GCC-64/bin/../lib/gcc/x86_64-w64-mingw32/5.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lnicaiu
collect2.exe: error: ld returned 1 exit status

Regarding strings: you need to form an 0-ending string like "hello\x00" otherwise you hand a full memory chunk to C land, otherwise it reads your passwords from a browser tab here too.
Take these snippets:
var end = "\x00"
var endChar byte = '\x00'

func safeString(s string) string {
	if len(s) == 0 {
		return end
	}
	if s[len(s)-1] != endChar {
		return s + end
	}
	return s
}

func safeStrings(list []string) []string {
	for i := range list {
		list[i] = safeString(list[i])
	}
	return list
}

Hi,
I don't understand why TaskHandle passed into DAQmxCreateTask as *taskHandle is represented in go as []unsafe.Pointer.

So cgogen assumes by default that every function argument or struct field being a pointer means it's a slice (and it's correct, because you can work with the first slice item there if it's not an array). And return values assumed to be pointers (just more likely be a pointer rather than an array, due to unknown length).
check out the docs about this behaviour and how to deal with it using PtrTips:
https://github.com/xlab/cgogen/wiki/Translator-config-section#ptrtips
you can override the default behaviour by providing a line that covers everything. And work out the cases when you actually have arrays, not pointers. Or maybe make a rule making every first argument of any function be always a pointer, useful for receivers. By the way, you can find a plenty of legit examples here: https://github.com/xlab/cgogen/wiki/Examples
Good luck

Ok, I will take a look at that.
I will do this temporarily then:
var ths []unsafe.Pointer

nidaqmx.CreateTask("task1", ths)

th := ths[0]

nidaqmx.CreateAIThrmcplChan(th, safeString(DAQMX_DEVICE_NAME + "/ai1"), safeString("T0 - LiDAR box right side"), -60.0, 100.0, nidaqmx.ValDegC, nidaqmx.ValTTypeTC, nidaqmx.ValBuiltIn, 25.0, safeString(" "))

@5k3105

nidaqmx.CreateTask("task1", ths)

you forgot safeString or \x00 in string.

Thank you ;)

Hi,
I went ahead and updated the yml and generated the lib again, but instead of giving
TaskHandle *unsafe.Pointer in CreateTask, it gave TaskHandle unsafe.Pointer
So I changed it in the file to *unsafe.Pointer (which is probably wrong, right?).
It did compile bu I getting nothing in response so I imagine this is wrong?
Later I ran into:
func ReadAnalogF64(TaskHandle unsafe.Pointer, NumSampsPerChan int32, Timeout float64, FillMode uint32, ReadArray []float64, ArraySizeInSamps uint32, SampsPerChanRead []int32, Reserved []uint32) int32 {
SampsPerChanRead needed to be *int32 so I updated the yml:
  PtrTips:
    function:
      - {target: ^DAQmxCreateTask$, tips: [0,ref]}
      - {target: ^DAQmxReadAnalogF64$, tips: [0, 0, 0, 0, 0, 0, ref, ref]}

Another point that might be making things misbehave:
For ReadAnalogF64, parameter ReadArray size is dependent on the following parameter called ArraySizeInSamps.
Since ReadArray expects a slice in Go, I can't:
var readarray [16]float64

when ArraySizeInSamps = 16
In the other code I look at the array size is statically defined before it's use:
data = numpy.zeros((1000,), dtype=numpy.float64)

DAQmxReadAnalogF64(taskHandle,1000,10.0,DAQmx_Val_GroupByChannel,data,1000,byref(read),None)

or in C:
 float64     data[1000];

DAQmxReadAnalogF64(taskHandle,1000,10.0,DAQmx_Val_GroupByChannel,data,1000,&read,NULL)

(examples from: https://pythonhosted.org/PyDAQmx/usage.html)

Hi,
I also get
Error -200492	Reserved parameter must be NULL.
for the last parameter of DAQmxReadAnalogF64. I have tried to put nil there and also
var reserved uint32
reserved = 0

f64 := nidaqmx.ReadAnalogF64(th, 1, 100.0, nidaqmx.ValGroupByScanNumber, readarray, 16, &samplesperchanread, &reserved)

which gives me the error as well.
How do I specify a C NULL in Go when it expects a *uint32 ?

How do I specify a C NULL
As usual really, nil.

instead:
, &reserved)

do:
, nil)

Hi. Thank you for your help.
original error code was:
Error -200492	Reserved parameter must be NULL.
after the change to nil for 'reserved' parameter, error code is:
Error -200230	NULL pointer was passed for a required parameter.
I think this now refers to the  'readarray' parameter.
(TaskHandle taskHandle, int32 numSampsPerChan, float64 timeout, bool32 fillMode, float64 readArray[], uInt32 arraySizeInSamps, int32 *sampsPerChanRead, bool32 *reserved)

Hi,
Do you know of any way I can fix this:
..\NidaqMonitor\nidaqMonitor.go:57: cannot use readarray (type [16]float64) as type []float64 in argument to nidaqmx.ReadAnalogF64

Use arr = make([]float64, 16) instead of [16]float64. That seems to be a Go error.
By the way, as the issue is resolved, I propose to move the discussion to email<EMAIL_ADDRESS>


[BUILD] Test closing stale PRs

What changes were proposed in this pull request?
Here I'm seeing if we can close stale PRs via a PR message, as I'd expect.
See thread<EMAIL_ADDRESS>Closes #9354
Closes #9451
Closes #10507
Closes #10486
Closes #10460
Closes #10967
Closes #10681
Closes #11766
Closes #9907
Closes #10209
Closes #10379
Closes #10403
Closes #10842
Closes #11036
Closes #13003
Closes #10887
How was this patch tested?
(No changes)

Test build #58375 has started for PR 13052 at commit b81422f.

+1

Do you mind adding #13003 maybe?

Test build #58375 has finished for PR 13052 at commit b81422f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58375/
Test FAILed.

Merged build finished. Test FAILed.

@HyukjinKwon that's already in the description. I'm ignoring the failure since there is no change to anything at all here.

Merged to master. Let's see if it worked.

PS for anyone who tries this -- it does work, but to make the merge script work with an empty commit, its git commit command needs --allow-empty. I didn't actually suggest changing the script since we generally don't want to allow that, I think.


Can't add corners to the screenshot of the the HostViewController when menu is opened.

@AlaeddineOuerteni
By corners, do you mean to round the corners?  Can you explain any further?

Yes @enmiller.
I want to change the cornerRadius of the layer of the screenshot imageView.
Is it possible ?

v2.3.0 now has a property on nSideMenuItemOptions that will allow you to set the corner radius of the opened content item.

Great job @enmiller ;)


Tab plugin should use a class selector instead of a ul tag selector

Hello. I want to use <ol> instead of <ul>, but in the Tab plugin tag name is hardcoded.
https://github.com/twbs/bootstrap/blob/master/js/tab.js#L28
var $ul      = $this.closest('ul:not(.dropdown-menu)')

I want to use <ol> instead of <ul>

Why?

Because I have an ordered list.

Because I have an ordered list.

And you have an ordered list because...?

And you have an ordered list because...?

Because it is a ordered dataset from database and it should be displayed as an orderd list of tabs. It is semantic, isn't it?

Bootstrap 3 is no longer being officially developed or supported.
All work has moved onto our next major release, v4. As such, this issue or pull request is being closed as a "won't fix." For additional help and support, we recommend utilizing our community resources. Thanks for your understanding, and see you on the other side of v4!
<3,
@mdo and team


Min & Max Values for Bar Visualization

Is your feature request related to a problem? Please describe.
It is currently not possible to see the value range a bar visualization has.
Describe the solution you'd like
Additional texts on the bar visualization displaying the min and max value the range of the bar is representing.
Describe alternatives you've considered
Maybe also display that range in the information card.

Added in #39


Win symlinks

This PR changes the 'yotta link' implementation from junctions to
symbolic links, which has a number of advantages:

symbolic links can point to pretty much anything (including network paths)
it seems to be harder to mess things up using symbolic links. Junctions
can behave weirdly when they're deleted (read: in some circumstances,
deleting a junction will also delete whatever directory the junction is
pointing to. Happened to me and others a number of times). Directory symbolic
links can safely be deleted using "rmdir" (the implementation of 'rmF' was
changes in fsutils_win.py to reflect this)

Also, "yotta unlink" works in Windows now.
Tested on Win7 64 bit, NTFS file system.

I lost couple of repositories with staged changes doing unlinking :-)
:dancer: unlink for windows will be back !

For me, yt unlink never worked under Windows, failing with an access denied error or something similar to that. However, when I wanted to "fix" that and tried to remove the linked module by hand from the command line, it'd sometimes (not all the time!) delete both the junction and the directory it was pointing to. Fun fun fun.

Do not merge yet, I believe I can make it less ugly.

ack

It seems like someone should have encapsulated this into a separate python module – maybe we should do that?

OK, so apparently I was wrong. Symlinks don't have a "better" behaviour when it comes to deleting stuff. I did a little test which involved creating three types of links (symbolic link, hard link, junction) to a directory, then trying to delete the link the Windows way (rd /s /q <linkname>) and the Linux way (rm -rf <linkname>, using rm from https://github.com/bmatzelle/gow/ (IIRC)). Results:

rm -rf
rd /s /q

Symlink
Deletes everything
Deletes just the link

Hard link
Access denied
Access denied

Junction
Deletes everything
Deletes just the link

I was unable to create any hard links at all (Access is denied), but other than that, it looks like my problem with deleted files came from my attempt to use rm instead of rd (it's concievable that the Windows port of rm doesn't handle links very well, since the link semantics are different in Windows and Linux). With that in mind, the only thing this PR really fixes for me is yotta unlink, which can probably be fixed easier in the original junction-based implementation.
That said, did other people experience this problem? @0xc0170, do you remember how you managed to get your repositories deleted? Again, yotta unlink didn't delete anything for me in Windows, because it never worked; everytime I used yotta unlink I got an Access is denied exception.

I'll have to spend some time to do this - I am using a commander based on git bash. I have tried a minute ago to link local repo, and once I did yt link, the repo is gone. This is the console output (yt link within cmsis-core-k64f is not shown as it was in another console tab).
C:\Code\git_repo\github\yotta_thing\ksdk-a
λ yt link cmsis-core-k64f
info: C:\Code\git_repo\github\yotta_thing\ksdk-a\yotta_modules\cmsis-core-k64f -> C:\Python27\Lib\yotta_modules\cmsis-core-k64f -> c:\Code\git_repo\github\yotta_thing\cmsis-core-k64f

C:\Code\git_repo\github\yotta_thing\ksdk-a
λ yt ls
mbed-hal-ksdk-mcu 0.1.4
|_ mbed-hal 0.2.0
| |_ mbed 3.0.2 yotta_modules\mbed
| | \_ cmsis-core 0.1.2 yotta_modules\cmsis-core
| |   \_ cmsis-core-freescale 0.1.1 yotta_modules\cmsis-core-freescale
| |     \_ cmsis-core-k64f ~0.1.0 missing
| \_ mbed-hal-freescale 0.1.4 yotta_modules\mbed-hal-freescale
\_ mbed-hal-k64f 0.1.7
  |_ uvisor-lib 0.6.4 yotta_modules\uvisor-lib
  \_ mbed-hal-frdm-k64f 0.1.3 yotta_modules\mbed-hal-frdm-k64f

C:\Code\git_repo\github\yotta_thing\ksdk-a
λ yt build
info: get versions for cmsis-core-k64f
info: download<EMAIL_ADDRESS>from https://yotta-private.herokuapp.com

Once I did yt link, the repo is gone and reported as missing. I'll find more time next week to find exact commands when this happens on my machine with the logging, so I can share as above which was just a quick check.

Let me guess - that repo was linked before? Cause I also found the perfect way to reproduce this:

link a module in your current module
yt ls in the current module will show you the linked module propely
linked the same module again
yt ls will show missing and the directory of the module being linked in is gone :sob:

I've figured out the cause too. When linking, yotta will first try to rmRf the directory that's being linked. On POSIX, this ends up calling unlink, which is the right thing to do. On Windows, it goes ahead and deletes everything :facepunch: So I'm gonna go ahead and create a patch for this issue first.
Of course, this means that this PR shouldn't be merged. Sorry for the noise.


Update pdf2svg.js

Changing file write to sync solves the problem.
Looks like promise chaining with a file write at the end with out waiting for the write to happen causes the issue.

I wonder if there is a logical explanation to this fix -- node.js people recommend to use async callbacks.

Looks like promise chaining with a file

callback chaining

I'm also wondering about the explanation for this. I'm fine with the sync approach if that works, but I do not understand why the async approach does not work properly.

I think the problem is not with the async calls but probably with when the call gets executed.
The "End Document" print statement gets printed before the write happens, which I think might be the cause.
By blocking the promise till the write happens this solves the problem.
There might be a way to do it with async calls but I did not figure it out yet.
Will continue working on it.

I tried different different things but only the sync file write works. I think we need to consult a node export here to tell us what the issue is. From the previous link it seems like a memory leak problem.

By blocking the promise till the write happens this solves the problem.

I tried adding callback to block a promise -- the same problem.

I tried different different things but only the sync file write works.

I'm worry it's just coincidence and the patch can be just masking the real problem, which can surface later.

Blocked by https://github.com/nodejs/node/issues/13337

node.js experts say it is a memory leak, so switching to sync might only fix the issue only for this case and can reappear in the future. I recommend to wait until leak will be found.

@yurydelendik take a look at this https://github.com/petkaantonov/bluebird/issues/1068 it seems like an inherit problem in promises in node.
The solution presented requires bluebird lib, I can try it out and see if it works.

@yshahin okay, pdf.js also have polyfill in compatibility.js, see if "forcing" it will help

@yashsriv the https://github.com/yurydelendik/nodesegfault/commit/2bb6f26c09309d942901c3d66fca3e6d3fdaf6f4 fixes the issue, so you are right it's due to Promises implementation.

Blocked by https://github.com/nodejs/help/issues/652

@yurydelendik how do you want to handle this?
Use the your fix? or change it to sync calls?

If it works with our promise polyfill and not with the native Node.js one, I'm not sure why the Node.js people claim the memory leak is on our side. Even if there is a memory leak, it shouldn't matter if you use sync/async writing.
Personally I'm fine with the sync approach here to solve this issue, but I'll let @yurydelendik make the call.

I'm closing this PR in favor of mine at #8540, because moving from async to sync is unnecessary.
The reason that moving from async to sync "solves" the problem is that it forces Node.js to suspend execution until the file writing finishes. Consequently, pages are effectively processed sequentially.
My patch makes also changes the pdf2svg code to process the pages sequentially, but without blocking execution.
In this thread, it has been said that using our Promise polyfill solves the issue as well. I think that this is a mere coincidence. Normally promises are scheduled as a microtask, but with our polyfill it is scheduled using setTimeout, which causes the promise to resolve less quickly. Consequently, when the SVG generation process (with many "promises") finishes, some file writing requests have likely finished as well, allowing Node.js to reclaim the memory for the string that is passed to fs.writeFile (in pdf2svg.js).


Numpy clip dtype

Numpy.clip at dtype 'bool' is different from ubuntu and Windows.

jenkins, test this please


Support generating symbols.zip for Android projects

See https://docs.unity3d.com/ScriptReference/EditorUserBuildSettings-androidCreateSymbolsZip.html.
Note that this is a bool property in Unity 2019.4 - 2020.3 and then an AndroidCreateSymbols property in 2021.1+

Added in 0fe678774488ae6762f46074f9a66c529a9c056d


added pending application page

After an anonymous user applies as a student, adviser, or admin, they will be redirected to the pending application page. This is simply a page with a text, 'Your application is being processed,' and a spinner component. The goal is to show that their application was successful and that they just need to wait for their application to be either approved or rejected.
I also made the route guard cleaner. Before approving this PR, the route guard should be checked properly. This is just a suggestion and doesn't need to be approved. Everything seems to be working so far, but I might have overlooked something, so a thorough review is needed to ensure it doesn't break the application or cause any other issues.

reverted the pull request, as the developer will need further consultation


Add mirror config for PuChuangSida CT log

It was recently accepted by Chrome (see https://crbug.com/667663).

LGTM


Adapter creation error

Casdoor version
Installed via docker

Error

@satishviswanathan use latest version higher than v1.377.1: https://github.com/casdoor/casdoor/releases/tag/v1.377.1

@hsluoyz can I install that built via docker ?
It would be helpful, if you can share the same

@satishviswanathan yes, install this image: https://hub.docker.com/layers/casbin/casdoor/v1.378.0/images/sha256-ea0fe44f06aab550b3d27d39b9fa8c4a91b0e3f22fc79236a2676692cf3b668e?context=explore

@hsluoyz  installed version v1.378.0

Now when I create an Organization, it always saves with the default name and display name and I'm unable to update the name in edit mode.

@satishviswanathan if this is a new issue, please create new github issue


[Feature Request] Sound effect when changing volume level

First and foremost ... Thanks for this awesome app!
This is totally a nice to have feature, but any chance we'll see the ability to get the sound effect when changing volume levels like you get when changing volume levels natively? It's just nice because it gives you a quick indication of how loud the volume actually is via your ear sensors instead of eye sensors :)

Hi @ammmze,
That's an excellent idea !

Wow...that was pretty quick 🙂 can't wait to try it next time I'm at my computer! Thank you


time-comp does not stop after first match

time-comp and time-spam selection of output doesn't seems to work, don't stop after first match

fixed


Serializing enums / reserializing file not 1:1

Hi @cleftheris,
I've followed the tutorial to serialize/deserialize, using the provided sample file "tradacoms.utilitybill.edi". Deserializing that file provides me a POCO wherein the VatRateCategoryCode gets set to L. However, serializing the same POCO using same grammar yields me a vastly different .edi file compared to the original- and the enum is serialized to its integer equivalent instead of the text.
var grammar = EdiGrammar.NewTradacoms();
var interchange = default(Interchange);

using (var stream = new StreamReader(@"C:\edi testing\tradacoms.utilitybill.edi"))
{
	interchange = new EdiSerializer().Deserialize<Interchange>(stream, grammar);
}

using (var tw = new StreamWriter(File.Open(@"C:\edi testing\tradacoms UtilityBill test1.edi", FileMode.Create)))
{
	using (var twEDI = new EdiTextWriter(tw, grammar))
	{
		new EdiSerializer().Serialize(twEDI, interchange);
	}
}

Why does the enum get serialized as an integer? And why does the reserialized file look dramatically different from the sample?
Thanks!

Hi @shinji144 and thanks for your interenst in Edi.Net.
The reason behind the different serialization behavior has todo with the sample.
First of all this usecase shows how someone needing to read an Edi transmission could decide to deserialize characters as well as letters into CLR enums. This feature unfortunately does not work bothways at the moment. It is however a nice enhancement since we could infer the serialization preference from the Picture (format) on the value attribute and decide whether we need to convert to the numeric counterpart underneath (which is the default behavior) or use the name instead.
On the other hand there is no complete parity between the sample nodels and the transmission. In other words models may be incomplete and ignore parts of the data hence the difference in serialized output. The example is meant to showcase the power that the library has in mappind edi to a more dotnet friendly poco class. In case you need to have a one to one parity you need to tweak the example models to suit your needs.


feat(storybook): Co-habitating/Upgrading Storybooks to v7 (dependency madness ensues)

SUMMARY

This PR does the following:

Temporarily removes the deny-list for license types. For some reason it's blocking a goofily-licensed submodule, which is perfectly license compatible (MIT). I'll flip this back on once I'm done with this effort. Resolved.
Upgrades BOTH the Superset Storybook and superset-ui-demo to Storybook 7
Migrates all the Stories to use Controls instead of Knobs
Includes all the Stories from superset-ui-demo in the main Superset storybook (you can still run superset-ui-demo on its own as well, though)
Migrates mdx stories (overviews) to tsx - these might need touchups
Fixes a bunch of little random bugs along the way
Removes a bunch of package vulnerabilities (1 critical, 17 high, 11 moderate)
Bumps ESLint and Prettier by necessity, and for the sake of making this PR mergable, disables the new offences - we'll get those in follow-up PRs, I promise.
Touches many other moving parts - it's all a house of cards... but things seem to be running stably.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

Run the root storybook and look at all the bells and whistles!
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

MERGE QUICK #yolo

Just need a final code-owner check from @michael-s-molina @kgabryje or @geido (because I touched Select, MetadataBar, DropdownContainer) and we'll be good to smash this thing in!

Follow-up PRs to address all of the above coming soon! I was spot checking stories as I ported them to the new format, but knew that some would get screwed up along the way. Let the polishing phase begin!

Thank you for all the hard work @rusackas!
While testing the PR, I found the following problems:
Some components are not rendering anymore:

Some components are rendering but their content is not:
Some controls that used to work are not rendering. In this example, sizes and styles were a select:
Some plugins are overflowing the render area:
This list is not exhaustive, as I only tested a sample of the stories. I can confirm that these problems don't happen on master. Given the size of the PR and the possibility of conflicts, I'm ok with merging it as is and tackling the problems in a follow-up given that the Storybook is an isolated area.
Amazing work 👏🏼

Tracking this here https://github.com/orgs/apache/projects/315/views/1?pane=issue&itemId=53539080


Returning the first match

Is it possible to sift through a list, but only return the first match?
users = [
  { id: 1, name: 'moe' },
  { id: 2, name: 'larry' },
  { id: 3, name: 'curly' },
  // ...1000 more
]

I'd like to do sift({ id: 2 }, users) but not needlessly run through the entire set.

nevermind—I see sift.indexOf does pretty much this. thank you!


Allow extension of LoggingEventSouce

Is it possible to implement some extension points in the EventSource logging framework?
For instance, it would be very useful if one can implement its own EventSource class with different name (not just "Microsoft-Extensions-Logging") and pass it to the EventSourceLoggerProvider.

@yodasad Can't you wrap/inherit LoggingEventSource and change the name in your own implementation already?

 It is simple as that...
thnx


Are there some $ keyword for Function overloading partial specialization?

Assume have a overloading Function like this:
type Fn = ( (v:'a', fn:(iv: boolean)=>void )=>void )&  // 1
          ( (v:'b', fn:(iv: number)=>void )=>void )&   // 2
          ( (v: string, fn: Function)=>void);          // 3

Literal 'a' and 'b' are more specialization than a string. Some language allow this kind overloading.(allow 'a' exclude from a const string), and cause in JavaScript, there is no Function overloading indeed, the args will be checked in runtime inside a Function, it is legal to exclude 'a' from string here, do not need a const string.
How to make this overloading work?
Currently when  use 'a' , flow will notify case 1 and case 3 will both work.
function ts3(f: Fn) {
  const v = f('a', iv => {}); // fail
}

How can i exclude 'a' and 'b' from a string?
flow try => here

This would be really useful, especially for stuff like addEventListener; I end up having to annotate almost all of my event listeners to deal with the "...case 3 looks promising too"

@erikdesjardins  Is it i missed something or a feature lacked?

You didn't miss anything (AFAIK); I have the same problem as you.

@erikdesjardins Try to use this overloading to restrict the passed in parameters.
Ideally if v is a, the passed in fn must be (iv: boolean)=>void, if v is string, the passed in fn could be a Function.
so i hope this should be checked by Flow. manual-annotating is useless to me,
But im a bit nervous about this issue, half month passed, it is still not confirmed to  a bug, so seems it is me missed something?    So that a and b could be excluded from a const string rightly.

@iamchenxin a and b can't be excluded from a type that includes all possible strings which the string type is. You'll need a way to say that my type is a set of all possible strings excluding a and b.

@vkurchatkin @alexeygolev After think this again, I find something interesting, cause JavaScript does not really have Function overloading, in truth, it is legal to exclude a string literal from string（no need a const string, cause inside a function, the args are checked in runtime.）, this is a bit different from a language which has a Function overloading(args are checked in compile time, so 'a' and 'b' can only exclude from a const string, not a string)
Maybe in JavaScript, we should not call this feature Function overloading, they are just a description of possible args combinations.

Closing since this issue hasn’t seen a lot of activity. Sorry about that 😣
If we see better patterns for function overloading in the future from JavaScript then we’ll implement it 👍


Classify Java ops in API definitions

This PR classifies the Java operators so that they do not all end up in the main (core) package, e.g. the Sqrt operator is now accessed from ops.math.sqrt(...) instead of ops.sqrt(...).
The classification is following pretty the one used in Python V2 API. Here's the list of additional packages that are generated using it and a few exceptions to make it work in Java:
org.tensorflow.op.audio
org.tensorflow.op.bitwise
org.tensorflow.op.collective
org.tensorflow.op.core
org.tensorflow.op.data
org.tensorflow.op.dtypes
org.tensorflow.op.image
org.tensorflow.op.io
org.tensorflow.op.linalg
org.tensorflow.op.math
org.tensorflow.op.nn
org.tensorflow.op.quantization
org.tensorflow.op.random
org.tensorflow.op.signal
org.tensorflow.op.sparse
org.tensorflow.op.strings
org.tensorflow.op.summary
org.tensorflow.op.train

Exceptions for Java:

Assert is renamed to AssertThat to avoid conflict with assert keyword
Switch is renamed to SwitchCond to avoid conflict with switch keyword

CC: @asimshankar , @annarev , @martinwicke

Thanks!
@annarev : Does it make sense to collapse many of these endpoints into the "base" api_def so they are automatically shared across languages, instead of having to duplicate them between Python and Java?

@asimshankar : ok found the case, both classes are generated: Conv3DBackpropFilter and Conv3dBackpropFilter (the former is from Conv3DBackpropFilter op and the later is from Conv3DBackpropFilterV2 op).
I don't know why this specific test it was failing since in both files the classes was named correctly but I'll rename them to avoid this conflict. I suggest to drop all ops that has a V2 version at this point, what do you think?

all right, I pushed a new version using lowercase accronyms exclusively and renamed the conflicting op class to Conv3dBackpropFilterV2.
@asimshankar , if you think we should also get rid of any version that has been succeeded, just let me know (e.g. if there is MyOp, MyOpV2 and MyOpV3 available, we only generate a class for the last one that we rename MyOp).

all right, I pushed a new version using lowercase accronyms exclusively and renamed the conflicting op class to Conv3dBackpropFilterV2.
@asimshankar , if you think we should also get rid of any version that has been succeeded, just let me know (e.g. if there is MyOp, MyOpV2 and MyOpV3 available, we only generate a class for the last one that we rename MyOp).

@asimshankar   Any update on the approval ? Please keep us posted.

@karllessard : Yeah, makes sense to only generate for the latest version and drop the older ones.

@karllessard : Yeah, makes sense to only generate for the latest version and drop the older ones.

@asimshankar I'm having some doubts now and would like to have your advice: if we delete previous versions, how will we handle the next ones?
For example, if we are at MyOpV2 right now, we'll delete MyOp and rename endpoint of V2 as MyOp... but when V3 appears, whose gonna be MyOp? V3? If so, for backward compatibility, I guess we'll have to keep MyOpV2 but with its original name?

@karllessard : Yeah, makes sense to only generate for the latest version and drop the older ones.

@asimshankar I'm having some doubts now and would like to have your advice: if we delete previous versions, how will we handle the next ones?
For example, if we are at MyOpV2 right now, we'll delete MyOp and rename endpoint of V2 as MyOp... but when V3 appears, who's gonna be MyOp? V3? If so, for backward compatibility, I guess we'll have to keep MyOpV2 but with its original name?

We'll do exactly what we do in Python - where we keep the endpoint name the same but change the implementation so that it invokes the new operation. For example, consider this really old change where a GatherV2 operation was added and we set things up so that the Python tf.gather function adds the GatherV2 operation (see change to array_ops.py in this commit)
For the example you provided, yes, this amounts to the Java class org.tensorflow.op.mypkg.MyOp adding the MyOpV3 operation to the graph instead of the MyOpV2 operation. Does that make sense?

In Java, the endpoints are automatically generated from the signature of their op, so it will work only if V3 has a signature backward compatible with V2. To support other cases, we’ll need to create a wrapper class that will “hide” the original ops. That shouldn’t be a problem though and will probably be exceptional.
Ok, I’ll clean up the ops tonight to keep only the last version available, thanks!

@asimshankar done!

@asimshankar is it too late to update this PR once again? I want to make visible those ops that are hidden by default from the base_api defs, since there is no wrapper in Java for them (~140 ops, ex: WriteSummary).
Otherwise I'll just create another PR for this, thanks

I wouldn't override the hidden ops. Ops are generally marked hidden because they typically require some additional outside-of-graph pre-processing before being useful, i.e., we'd want a hand-crafted Java class instead of having clients use the op directly. Similar to some Python API endpoints.

@harshini-gadige , @asimshankar : is there anything left to be done for this PR to get merged? I thought the review was completed

@harshini-gadige , @asimshankar : is there anything left to be done for this PR to get merged? I thought the review was completed

@karllessard  If  the reviewer approves it, I can proceed further to help this get merged. Before that please make sure you made all changes as suggested by the reviewer(if any).

I think I did, @asimshankar?

@harshini-gadige sorry for bugging you again with this but can we target to merge this PR for 1.13 release? Thanks

Something got stuck in the tooling. I'm looking into it.

Really sorry for missing all the questions here earlier!
@asimshankar Yep, definitely makes sense to collapse into base api_defs if all languages use the same namespaces. Is it true for C++ and Go?
@karllessard: "For example, if we are at MyOpV2 right now, we'll delete MyOp and rename endpoint of V2 as MyOp... but when V3 appears, who's gonna be MyOp? V3? If so, for backward compatibility, I guess we'll have to keep MyOpV2 but with its original name (forcing the clients to update their code)?"
Are you asking about op name at C++ layer or at Java API layer? At C++ layer, we don't deprecate ops as far as I know, therefore MyOp in V3 would need to be named MyOpV3. At Java layer, we can break backwards compatibility at a major release. That is, when we release TF 3.0, we can name V3 version MyOp. Older versions could either be removed or renamed.
On Python side, we provide a tf.compat.v1 module that keeps V1 names available in V2. For e.g. tf.compat.v1.foo would still point to 1.* version of foo even though tf.foo points to the 2.* version. May be something like this can be done on Java side too.
We also provide a script for Python API that renames endpoints for 2.0 upgrade. For e.g. that script could auto rename 'MyOp' to 'MyOpV2' if 'MyOpV2' is corresponding 2.* name.

At Java layer, we can break backwards compatibility at a major release.

Ok that make sense, thanks Anna


Question: why build most of api from XML in build.rs?

As per the title, I want to know why this library is designed as such to build most of api from XML spec.

There's a lot of code that would be duplicated and it's not uncommon to need to update how a value is read/written, so rather than updating hundreds of lines, it's easier to update an XML file and a little bit of the codegen.
As a quick example, HeaderVariablesSpec.xml is 325 lines long, and the generated file is 3335, so there's a whole order of magnitude difference.  EntitiesSpec.xml is 1341 lines, generated is 4737.

A good example is recent PR #56.  It changes a lot of how a test-only JSON parsable version is generated.  Rather than manually change thousands of lines, that PR is only 150 lines long.

Understood. That explains it then. Gonna play around with macros in maybe replacing build script.


iOS - Update Course Dates endpoint

Description
LEARNER-2924
iOS - Update Course Dates endpoint

Codecov Report

Merging #1029 into master will increase coverage by 0.1%.
The diff coverage is 0%.

@@           Coverage Diff            @@
##           master   #1029     +/-   ##
========================================
+ Coverage   48.29%   48.4%   +0.1%
========================================
  Files         406     406
  Lines       25692   25692
========================================
+ Hits        12407   12435     +28
+ Misses      13285   13257     -28

Impacted Files
Coverage Δ

Source/CourseDatesViewController.swift
0% <0%> (ø)
:arrow_up:

Source/VideoBlockViewController.swift
48.36% <0%> (+10.65%)
:arrow_up:

Source/UIViewController+CommonAdditions.swift
29.41% <0%> (+11.76%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e81bdae...a961cbd. Read the comment docs.

@saeedbashir I have remove the protocol which was not using. The webview is looking fine as i checked which device are you using to run the app?

@salman2013 It seems you have posted an old image. I have checked it again and same padding issue exists.

@saeedbashir the code is updated and the padding is correct as we are giving the custom css at application side.
The course date fragment is updating at server side and when it will done we will remove our custom css from application side.
Ref Ticket : https://openedx.atlassian.net/browse/LEARNER-3206

@salman2013 Got it but you still need to fix some comments added in AuthenticatedWebViewController

@saeedbashir ready for another pass, thanks


undefined method `port_range' for Chef::Resource::FirewallRule

Cookbook version
[3.1.1]
Chef-client version
[12.18.31]
Platform Details
[ec2 AWS Ubuntu 16.04]
Scenario:
[Open range of ports]
Steps to Reproduce:
name "test"
description "ufw port range test"
run_list [
"recipe[ufw]"
]
override_attributes(
:firewall => {
:rules => [
{"allow to TCP ports 32765-32768 (NFS) from 10.x.x.x/24" => {
"port_range" => "32765..32768",
"source" => "10.x.x.x/24",
"protocol" => "tcp"
}
}
]
}
)
Expected Result:
Open range of ports defined in firewall rules.
Actual Result:
Fails.
Logs:
Recipe Compile Error in /var/chef/cache/cookbooks/ufw/recipes/default.rb
NoMethodError
undefined method `port_range' for Chef::Resource::FirewallRule
Cookbook Trace:
/var/chef/cache/cookbooks/ufw/recipes/default.rb:62:in block (3 levels) in from_file'   /var/chef/cache/cookbooks/ufw/recipes/default.rb:53:in block (2 levels) in from_file'
/var/chef/cache/cookbooks/ufw/recipes/default.rb:34:in each'   /var/chef/cache/cookbooks/ufw/recipes/default.rb:34:in block in from_file'
/var/chef/cache/cookbooks/ufw/recipes/default.rb:32:in each'   /var/chef/cache/cookbooks/ufw/recipes/default.rb:32:in from_file'
Relevant File Content:
/var/chef/cache/cookbooks/ufw/recipes/default.rb:
55:        protocol params['protocol'].to_sym if params['protocol']
56:        direction params['direction'].to_sym if params['direction']
57:        interface params['interface'] if params['interface']
58:        logging params['logging'].to_sym if params['logging']
59:        port params['port'].to_i if params['port']
60:        if params['port_range']
61:          ends = params['port_range'].split('..').map { |d| Integer(d) }
62>>         port_range ends[0]..ends[1]
63:        end
64:        source params['source'] if params['source']
65:        destination params['destination'] if params['destination']
66:        dest_port params['dest_port'].to_i if params['dest_port']
67:        position params['position'].to_i if params['position']
68:        action act
69:      end
70:    end
71:  end
Platform:
x86_64-linux
Running handlers:
[2018-03-07T12:09:44+00:00] ERROR: Running exception handlers
Running handlers complete
[2018-03-07T12:09:44+00:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 07 seconds
[2018-03-07T12:09:44+00:00] INFO: Sending resource update report (run-id: 942f3034-e1ab-4f39-8e98-17ac8f286915)
[2018-03-07T12:09:44+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2018-03-07T12:09:44+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-03-07T12:09:44+00:00] ERROR: undefined method `port_range' for Chef::Resource::FirewallRule
[2018-03-07T12:09:44+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Wrong cookbook


add  POST with json type equals to [String:AnyObject]

Otherwise everything is encoded as a string and we can't call the api like this
NSURLConnection.POST(url, JSON: ["foo": 1, "bar": "..."])

Yes, but AnyObject isn't valid JSON. You can have strings, numbers, bools, arrays or dictionaries of the same. I was kind of waiting on some Pod that does JSON better and would then use that.
Still, practically, this is what you need in the interim so I guess it should be committed.

That would be great, unless you see another work around?


Make useCustomSlot obsolete

Currently, the property useCustomSlot is only ever defined by users if the element has slot content anyway, for example:
<vue-dropzone :options="dropzoneOptions" :useCustomSlot=true>

...

</vue-dropzone>

Automatically detecting if the component has defined children (via Vue's built-in $slots) would make the usage both more convenient and more in line with how components are usually built.
To use a slot:
<vue-dropzone :options="dropzoneOptions">

...

</vue-dropzone>

To not use a slot:
<vue-dropzone :options="dropzoneOptions" />

@WillGoldstein from my knowledge it's still being maintained, but it lacks maintainers, as seen in the readme file :)


[SB] Remove msrest dependency

closes #26571
Replaced msrest with the azcore equivalent, we should not need msrest as part of setup.py either for SB.

/azp run python - servicebus - tests

API change check
API changes are not detected in this pull request.


WIP: CLB to SLICE split

This PR is a continuation of work started in #537. It should be merged afterwards.
There are two new tables in the SQL database:

"tile_former_site" Relates new SLICE tiles (instances) to concrete site instances of a CLB which were used to create them.
"pip_in_tile_instance" Relates new SLICE tiles (instances) to pips from the pip_in_tile table.

The whole import flow works until the handle_direction_connections in prjxray_assign_tile_pin_direction.py. Right now it fails there as it cannot find SLICEL tile in the edge_assignments map.

#665 is superceeding this PR


Feature request: Locking movement along axes

Hi,
I am working on a dynamic character controller, and it would be great to be able to lock linear/rotational movement along certain axes for a given object. Something along the line of
body->setAngularFactor(rp3d::Vector3(0.0, 0.0, 1.0));
body->setLinearFactor(rp3d::Vector3(1.0, 0.0, 1.0))

Would be really, really handy.
Thanks!

Good idea ! I will try to add this in the next release.

I am trying to set a body to a fixed rotation as well, something like body:setFixedRotation(bool) in box2d.
Is it possible to do something similar in current reactphysics3d?
Thank you.

I hope this feature will be implemented one day :)

It's already implemented in the 'develop' branch. I will release the next version of the library in the next weeks with this feature.

I glad to announce that this feature is now implemented in version v0.9.0 that has just been released.
There is now a RigidBody::setLinearLockAxisFactor() method to lock the translational movement of a body along the world-space x, y and z axes and a RigidBody::setAngularLockAxisFactor() method to lock the rotational movement of a body around the world-space x, y and z axes.
Do not hesitate to take a look at the documentation for more information.


/var/lib is mounted as tmpfs

From @petrosagg on October 9, 2016 6:6
There is state that should be persisted in /var/lib that is currently lost during reboot due to the tmpfs mount. For example the systemd random-seed and the systemd clock file that helps maintain a monotonic clock.
./systemd/clock
./systemd/random-seed
./NetworkManager/dhclient6-9fc92dad-29a7-4de8-acf6-c09e95341029-wlan0.lease
./NetworkManager/dhclient6-wlan0.conf
./NetworkManager/seen-bssids
./NetworkManager/timestamps
./NetworkManager/dhclient-9fc92dad-29a7-4de8-acf6-c09e95341029-wlan0.lease
./NetworkManager/secret_key
./NetworkManager/dhclient-wlan0.conf
./NetworkManager/NetworkManager-intern.conf
./dropbear/authorized_keys

Copied from original issue: resin-os/meta-resin#406
 Front conversations

From @petrosagg on October 9, 2016 6:13
Also /var/lib includes /var/lib/systemd/coredump which could easily fill up RAM if enough programs crash

ping @agherzan @willnewton on this

The systemd part was implemented as part of https://github.com/resin-os/meta-resin/pull/662 .
dropbear - is currently persistent on disk and bindmounted over with copy as tmpfs at boot. I would say we can keep it as is for now.
NetworkManager - needs to be worked on because it can get pretty big due to old leases. On my host I'm around 1M already used by 198 file from which 188 are leases.
I'll say this needs to be completed in the future but the high priority subtask was done.

At least NetworkManager/secret_key needs to be persisted for stable MAC randomization to work with NetworkManager. The other stuff might need to be kept (I guess seen-bssids and timestamps) might be better to keep. The rest, hm....

The NetworkManager part here should be fixed by https://github.com/resin-os/meta-resin/pull/850


Continue training from restored model

Hi, thanks for the repo. I have been testing it and so far it works fine for me.
I played with the save and restore functions, and I can restore a model and change the arguments         n_warmup, max_steps, test_interval, evaluate so that it actually evaluates using the restored model. I set a very small test_interval and max_steps for the evaluation to be executed.
However, if I want to continue training, it seems to not use the restored model, i.e with a usual max_steps and test_interval; It goes to be as bad as starting from scratch.
is there a way load the trained model before starting the training so the learned parameters are used from the beginning?

Hi @cambel, I could not reproduce your problem with SAC on Pendulum-v0 environment.
$ python examples/run_sac.py --env-name Pendulum-v0 --n-warmup 500 --max-steps 50000 --memory-capacity 10000 --gpu -1
$ ls results/20200108T191338.721946_SAC_/
...
checkpoint
ckpt-5.data-00000-of-00001
ckpt-5.index
...
$ python examples/run_sac.py --env-name Pendulum-v0 --n-warmup 500 --max-steps 50000 --memory-capacity 10000 --gpu -1 --model-dir results/20200108T191338.721946_SAC_/

The orange line is trained from scratch, and blue line is trained from restored model. As you can see from SAC/actor_loss, SAC/critic_Q_loss, and SAC/critic_V_loss, learning curves immediately converges nearly to the same values on trained model from scratch.

The reason why initial loss is not as low as converged value is that initial sample contains random values, where distribution mismatch occurs (random values are sampled from uniform distribution).
If you can provide an example to reproduce your problem, please let me know. I will look into the problem :)

Hi @keiohta, thank you, I am working on a custom environment so it was harder to see if the agent was converging to the previously learned model but I think is working.