Missing a way to get access to wasm_instance_t from wasi_instance_t for C-API

For the new wasi.h we can create own wasi_instance_t* wasi_instance_new. But there is no away to access the wasm API using the wasi_instance_t handle. E.g. for getting the export table with wasm_instance_exports.
Seems like we need an API to get the wasi_instance_t from the wasi_instance_t. To avoid having to replicate all wasm APIs as wasi APIs.

It's possible if you know the specific export you want from the WASI instance, but it requires using wasm_importtype_new to create an import type to pass to wasi_instance_bind_import and that's super kludgy.
The reason that wasi_instance_new simply doesn't return a wasm_instance_t* is that there's currently no way in the C API to get an instance's module and from the module its name so you can resolve imports against a WASI instance manually (one needs to know the "name" of the WASI module that wasi_instance_new instantiated).
The wasi_instance_new and wasi_instance_bind_import functions abstract away the name of the WASI module (or modules) from the instance.  It did it this way for simplicity, but also to punt on how best to support WASI as it evolves standard snapshot to snapshot (the WASI instance might represent a single specific snapshot or potentially multiple WASI for backwards compatibility; the API user doesn't know).
Rather than treating the WASI instance like a normal module instance, I think perhaps having void wasi_instance_exports(const wasi_instance_t*, own wasm_extern_vec_t*) would make sense to get at all the exports of the instance for now.

Yes, wasi_instance_exports would be fine for now.

As I mentioned in #1221, the problem with adding a wasi_instance_exports is that wasm_extern_t won't give any names, so you'd only get a set of functions and their types, without knowing which function is which.
If we were to properly model this, we would need to represent a WASI module with wasm_module_t so you could call wasm_module_exports to get the symbolic information.
So maybe have wasm_module_t* wasi_module_new(wasm_store_t*, const char* name) where name must be a supported WASI version.  From there the usual functions would work.
This is a bunch of work, but it would unify the two APIs (Wasm C API and the WASI extensions) more.

So maybe have wasm_module_t* wasi_module_new(wasm_store_t*, const char* name) where name must be a supported WASI version. From there the wasm_instance_* functions would work.

It would be great from my perspective to be able to treat wasi as regular module and not have to special case it.

I believe the wasi implementation now integrates seamlessly with wasmtime_*. Does it still make sense to keep this issue open?


Implement basic framework

The base framework for SQL Buddy should be built using:

Symfony config
Symfony http
Symfony routing
Doctrine DBAL

Completed in https://github.com/deliciousbrains/sqlbuddy/commit/504ba5343568f7220595d7740d38ee70580eca40 and https://github.com/deliciousbrains/sqlbuddy/commit/1d8f20dfb45df7dc4acba9e070759c7a990a73ac


All host(s) tried for query failed (no host was tried)

I'm getting the issue in the title quite often. The drivers discovers my C* ring correctly and then fails this this "no host was tried".   Any idea what might cause this?
Sample exception:
14/10/17 12:47:36 INFO com.datastax.driver.core.Cluster: New Cassandra host /xxx.xxx.xxx.120:9042 added
14/10/17 12:47:36 INFO com.datastax.driver.core.Cluster: New Cassandra host /xxx.xxx.xxx.108:9042 added
14/10/17 12:47:36 INFO com.datastax.driver.core.Cluster: New Cassandra host /xxx.xxx.xxx.42:9042 added
14/10/17 12:47:36 INFO com.datastax.spark.connector.cql.CassandraConnector: Connected to Cassandra cluster: development_cassandra
14/10/17 12:47:36 INFO com.datastax.driver.core.Cluster: New Cassandra host /xxx.xxx.xxx.3:9042 added
14/10/17 12:47:36 INFO com.datastax.driver.core.Cluster: New Cassandra host /xxx.xxx.xxx.:9042 added
14/10/17 12:47:36 INFO com.datastax.driver.core.Cluster: New Cassandra host /xxx.xxx.xxx.10:9042 added
14/10/17 12:47:37 ERROR jobs.actualvalues.ActualValues: Unable to create cassandra model
jobs.common.CassandraModelCreateException: Unable to create model for: abc
	at jobs.common.CassandraModelCreator$.loop$1(CassandraModelCreator.scala:18)
	at jobs.common.CassandraModelCreator$.createModel(CassandraModelCreator.scala:21)
	at jobs.actualvalues.ActualValues$$anonfun$6.apply(ActualValues.scala:32)
	at jobs.actualvalues.ActualValues$$anonfun$6.apply(ActualValues.scala:31)
	at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:99)
	at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:98)
	at com.datastax.spark.connector.util.IOUtils$.closeAfterUse(IOUtils.scala:16)
	at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:98)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
	at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65)
	at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:256)
	at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:172)
	at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
	at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)

Scratch that - we have code that is returning an iterator to ResultSet and seemingly reading paged data for a long period of time. During our reading the connection is getting timed out. This is an error in our usage and not a bug.


Clarification of poisson_multinomial (possibly returns wrong shape?)

Hey there,
thanks a lot for the repo - it's super informative and has been a great help for me in understanding and writing my own code!
While doing a PyTorch implementation of poisson_multinomial I noticed the following: poisson_multinomial returns shape [2,2,10] for shape [2, 200, 10] as input.
My interpretation of the input shape is: batch size 2, sequence length 200, tracks 10. As far as I understand, the poisson multinomial loss has one loss per track which would result in a shape of [2, 1, 10] (or some shape with 20 losses). Instead it returns losses of shape [2,2,10]. I'm sure I'm missing something here, but why is there twice as many losses as I expect?
Here's a script to reproduce the behavior (mostly copied from this repo except for everything after the loss function):
import tensorflow as tf
import torch
import numpy as np

def poisson(yt, yp, epsilon: float = 1e-7):
    """Poisson loss, without mean reduction."""
    return yp - yt * tf.math.log(yp + epsilon)

def poisson_multinomial(
    y_true,
    y_pred,
    total_weight: float = 1,
    weight_range: float = 1,
    weight_exp: int = 4,
    epsilon: float = 1e-7,
    rescale: bool = False,
):
    """Possion decomposition with multinomial specificity term.

    Args:
        total_weight (float): Weight of the Poisson total term.
        epsilon (float): Added small value to avoid log(0).
        rescale (bool): Rescale loss after re-weighting.
    """
    seq_len = y_true.shape[1]
    pos_start = -(seq_len / 2 - 0.5)
    pos_end = seq_len / 2 + 0.5
    sigma = -pos_start / (np.log(weight_range)) ** (1 / weight_exp)

    positions = tf.range(pos_start, pos_end, dtype=tf.float32)
    position_weights = tf.exp(-((positions / sigma) ** weight_exp))
    position_weights /= tf.reduce_max(position_weights)
    position_weights = tf.expand_dims(position_weights, axis=0)
    position_weights = tf.expand_dims(position_weights, axis=-1)

    y_true = tf.math.multiply(y_true, position_weights)
    y_pred = tf.math.multiply(y_pred, position_weights)

    # sum across lengths
    s_true = tf.math.reduce_sum(y_true, axis=-2, keepdims=True)
    s_pred = tf.math.reduce_sum(y_pred, axis=-2, keepdims=True)

    # total count poisson loss, mean across targets
    poisson_term = poisson(s_true, s_pred)  # B x T
    poisson_term /= tf.reduce_sum(position_weights)

    # add epsilon to protect against tiny values
    y_true += epsilon
    y_pred += epsilon

    # normalize to sum to one
    p_pred = y_pred / s_pred

    # multinomial loss
    pl_pred = tf.math.log(p_pred)  # B x L x T
    multinomial_dot = -tf.math.multiply(y_true, pl_pred)  # B x L x T
    multinomial_term = tf.math.reduce_sum(multinomial_dot, axis=-2)  # B x T
    multinomial_term /= tf.reduce_sum(position_weights)

    # normalize to scale of 1:1 term ratio
    loss_raw = multinomial_term + total_weight * poisson_term
    if rescale:
        loss_rescale = loss_raw * 2 / (1 + total_weight)
    else:
        loss_rescale = loss_raw

    return loss_rescale

torch.manual_seed(42)
np.random.seed(42)
bs, tracks, seq_len = 2, 10, 200
y_true = torch.randint(low=0, high=1000, size=(bs, seq_len, tracks)).float()
y_pred = torch.randint(low=0, high=1000, size=(bs, seq_len, tracks)).float()

loss_tf = poisson_multinomial(y_true=y_true, y_pred=y_pred, weight_range=1.1)

print(f"{loss_tf.shape=}")

Thanks a lot for your help!
Best
Timon

Sorry, this is code under active development. I pushed a modified version that should clarify the shapes.

Thanks for the quick reply! Makes sense now.


[SPARK-13049] Add First/last with ignore nulls to functions.scala

This PR adds the ability to specify the ignoreNulls option to the functions dsl, e.g:
df.select($"id", last($"value", ignoreNulls = true).over(Window.partitionBy($"id").orderBy($"other"))
This PR is some where between a bug fix (see the JIRA) and a new feature. I am not sure if we should backport to 1.6.
cc @yhuai

@hvanhovell Thanks for the PR. Do you know why expr/callUDF does not work?

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50228/
Test FAILed.

Merged build finished. Test FAILed.

@yhuai expr("last(r, true)") would return an UnresolvedFunction(UnresolvedAttribute(r), Literal(true)). The problem is that the WindowSpec does not recognize UnresolvedFunction's.
This is the cleaner fix. We could also add a match to the WindowSpec function for unresolved functions.

retest this please

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50230/
Test FAILed.

retest this please

Test build #50231 has started for PR 10957 at commit defcc02.

Why might this be a bug fix?

A user is trying to get this working on 1.6 using the dataframe api. That doesn't work directly because functions.scala misses  the functions implemented in this PR. The indirect approach using expr(...) doesn't work because WindowSpec does not support UnresolvedFunctions.
I guess this is more a feature than a bug fix....

Test build #50231 has finished for PR 10957 at commit defcc02.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50231/
Test PASSed.

Actually can you update the Python API as well?

Test build #50462 has started for PR 10957 at commit b002d60.

Test build #50462 has finished for PR 10957 at commit b002d60.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50462/
Test FAILed.

Merged build finished. Test FAILed.

Test build #50463 has started for PR 10957 at commit 6e4da4f.

Test build #50463 has finished for PR 10957 at commit 6e4da4f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50463/
Test PASSed.

Merged build finished. Test PASSed.

Thanks - only some minor comment on the documentation to make it more clear.

Test build #50464 has started for PR 10957 at commit 809c999.

Test build #50464 has finished for PR 10957 at commit 809c999.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50464/
Test PASSed.

Merged build finished. Test PASSed.

Thanks - merging this in master.


Add artifical information filter

Closing in favour of #280. This branch has a few newer and older xbitinfo implementations mixed.


docs: improve the documentation of SparseLabelOp.simplify

Summary
Closes #1260
Details and comments

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 86.73%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
8706

Relevant Lines:
10038

💛  - Coveralls

In FermionicOp there is a note in index_order method that pertains to simplify. Would it be better in conjunction with this to more have a note in simplify so if you look at the subclass simplify docs you have an idea of the other methods that can be called first to potentially allow simplify to further reduce things. At present, both it and VibrationalOp inherit the SparseLabelOp simplify docstring that was updated here - so it would mean adding something for them - which BosonicOp already seems to do as it has further details there pertinent to that sub-class.

I tried to find a solution which does not require duplicating the simplify docstring into the subclasses in order to append to it.
If you feel this solution is not sufficient, we can do that though.

I certainly would not suggest duplicating the entire thing - of course the args bit one has too and a sentence describing it. This is pretty much what BosonicOp.simplify() has plus some extra specific notes for that operator - it does not inherit the parent class docstring unlike FermionicOp or VibrationalOp. It is a bit extra work to do this for simplify but it could save the user going around other methods in the class to see if they would help - like index_order for example may do. I'll leave it to you - I'll approve this since in improves what it states - anything further like I was suggesting can always be done later since its "just docs".

I actually had overlooked that BosonicOp.simplify already has a custom docstring. I will take another look on how to improve things :+1:


Partial does not solve React.setState

Traditionally in TS, the absence of value or an explicit undefined value was exactly the same thing.
Perhaps it still even makes sense for Partial, but I thought Partial was meant to solve the React setState problem; and I believe it's still not solved today.
TypeScript Version:  2.1.4
Code
type State = { veryImportant: number }

// This compiles fine with all the mandatory flags (strictNullChecks, etc)
// if setState uses Partial<State>
// This is a huge invariant violation and can happen very easily for instance
// by setting `veryImportant` to a nullable variable.
this.setState({ veryImportant: undefined })

Expected behavior:
A non nullable property should not be updatable with null/undefined with strictNullChecks.
We need to be able to say "give me an object that

Either do not declare a property I have
Or if it does, must have the same exact type (not nullable)

By the way, I don't even use React, but this is a fairly common and useful behavior to have.

I believe this is what you're looking for:
function setState<T, K extends keyof T>(obj: T, state: Pick<T, K>) {
    for (let k in state) {
        obj[k] = state[k];
    }
}

interface Foo {
    a: string;
    b: number;
}

let foo: Foo = { a: "hello", b: 42 };
setState(foo, { a: "test", b: 43 })
setState(foo, { a: "hi" });
setState(foo, { b: 27 });
setState(foo, { });
setState(foo, { a: undefined });  // Error
setState(foo, { c: true });  // Error

Looks good! Thanks.

@ahejlsberg Thanks for the post. However, the following situation then creates an issue:
interface FooState {
  bar: string;
  foo?: string;
}

const defaultFooState: FooState = {
  bar: "Hi",
  foo: undefined,
};

class Foo extends React.Component<{}, FooState> {
  public doStuff() {
    this.setState(defaultFooState);
  }
}

And updating setState as follows:
        setState<K extends keyof S>(f: (prevState: S, props: P) => Pick<S, K>, callback?: () => any): void;
        setState<K extends keyof S>(state: Pick<S, K>, callback?: () => any): void;

You get the error:
severity: 'Error'
message: 'Argument of type 'FooState' is not assignable to parameter of type 'Pick<FooState, "bar" | "foo">'.
  Property 'foo' is optional in type 'FooState' but required in type 'Pick<FooState, "bar" | "foo">'.'
at: '37,19'
source: 'ts'

Updating the state to explicitly allow the optional to be undefined also produces the same error.
interface FooState {
  bar: string;
  foo?: string | undefined;
}

@ericanderson question: in your opinion, should setState({ foo: undefined }) be an error, or not?

Ive been kicking it around in my head. If foo is allowed to be undefined (either because its optional or explicitly allowed), then that should not be an error for setState(), ignoring how TS is implemented.
If you implement setState with Partial, it works as intended for this case but breaks because it allows you to set undefined where its not allowed.

interface TState = {  foo?: string; }
let state = { foo: "initial" };
Now I want state to become { foo: undefined }, how am I to do that if I can't call setState({ foo: undefined })?
I prefer the Partial to the Pick in this case. Partial allows some invalid behavior, but Pick prohibits some valid expressions.
If indexed types had the concept of optionality, then keyof FooState would return "foo?" | "bar", and Pick would become a perfect expression of the actual type of setState.
(Then I think for full completeness, if indexed types gain the concept of optionality they should also gain the concept of requiredness, so that we could undo optionality.
type Required = {
[K in keyof T]!: T[K]
})

@masonk if you define your state as type TState = { foo: string | undefined} you can still use it as you wish with Pick.
But if we use Partial, and you have type State = { foo: string}, then it will not be an error to have setState({foo: undefined}).

@RyanCavanaugh If there is a bug here, I would suggest that any parameter who is an optional type is also an implicit "T | undefined" so that when you convert it in something like Pick, the ability to assign undefined does not get lost.

Sure, but the idiomatic way to say { foo: string | undefined } is { foo?: string }. As I understand the TypeScript, those two declarations ought to behave identically throughout TypeScript.

@masonk Correct. I think we could say that there is currently a bug in TypeScript that the undefined get lost in the case of Pick

@masonk Also do note that they arent always assignable.
type A = { foo?: string };
type B = { foo: string | undefined};

let a: A = { foo: "bar" }
let b: B = { foo: "bar" }

a = b; // okay
b = a; // not okay

I think the problem is that the symmetry between { foo: string | undefined } and { foo?: string } is a lie -- spreading in {} to an object has different effects than spreading in { foo: undefined }.
In an ideal world there would be a difference
interface A {
  foo?: string;
}
interface B {
  foo: string | undefined;
}
interface C {
  foo?: string | undefined;
}
let a: A = { }; // OK
let b: B = { }; // Error, missing 'foo'
let c: C = { }; // OK
a.foo = undefined; // Error, undefined is not string
b.foo = undefined; // OK
c.foo = undefined; // OK

setState(a, { }); // OK
setState(a, { foo: undefined }); // Error
setState(b, { }); // Error, foo missing
setState(b, { foo: undefined}); // OK
setState(c, { }); // OK
setState(c, { foo: undefined}); // OK

We've sort of swept this under the rug, but with Object spread and Object.keys, there really is a difference between "missing" and "present with value undefined". It ought to be possible to express this difference to indicate your intent.

@RyanCavanaugh I agree with you, however the example for setState(c, { foo: undefined}); was not working for me on my other laptop (while it does work on the playground right now). If I was imaging the issue for C, then I agree that everything is working as it should be.

@ericanderson  I think the playground uses strictNullChecks: false

Ok, those examples from @ericanderson  and @RyanCavanaugh  are helpful. I now see that there are clear differences between "no key" and "key: undefined".
Damnit, you know what this means?
It means almost all of the interfaces in all of the code I've ever written are maltyped.
Pretty much wherever I have said, { foo?: T }, I meant to say { foo?: T | undefined }.  Because {} is assignable to { foo?: T }, I have to treat foo as T | undefined at runtime. With strictNullChecks I'm actually required to. Since I'm already doing that, I usually - read almost always - want to allow myself to assign undefined to foo.
I can't think of a single place in my code where I've made an optional property but wish for my compiler to insist that if the property exists then it must not be undefined, but allow non-existence. But that's my problem, not TS's problem. TS is just following JS here, and JS clearly makes a distinction.

Okay, I found my repro, it fails with strictNullChecks true or false., this feels like it should work @RyanCavanaugh:
interface C {
  foo?: string | undefined;
}
let c: C = {}; // OK
let c2: C = {
    foo: undefined,
} // OK

function setState<T, K extends keyof T>(obj: T, state: Pick<T, K>) {
    for (let k in state) {
        obj[k] = state[k];
    }
}

setState(c, c2); // !OK =  ERR Argument of type 'C' is not assignable to parameter of type 'Pick<C, "foo">'.

Nevermind. I withdraw my complaint. In my above example, there is no way for TS to know that c2.foo, in general, was given a value or was left off. Thus if it promises that it returned a Pick<C, "foo">, then foo MUST be defined, which we can't guarantee. While this isn't my ideal, it seems to be correct, and to go back to the source of this (https://github.com/DefinitelyTyped/DefinitelyTyped/pull/13155), Pick is the right choice and the side effect of State objects having no optionals is the most rational thing.
Thanks again @ahejlsberg & @RyanCavanaugh

I'm confused again. How could that possibly be right?
It shouldn't matter to TS whether "c2.foo, in general, was given a value or was left off", because we've said that both are allowed.  C is a supertype of Pick. Anything of type C should be assignable to something of type Pick.

C cannot be assigned to Pick<C, "foo"> because later when you use the Pick version it MUST have a param of foo that is on the object. C isn't guaranteed to have that

Yes, but how could that possibly be the correct behavior?
keyof C isn't "foo", it's "foo"?, only the type system doesn't know about that yet.

Yeah, that can't be right. You need to be able to reset some state, for instance the id of a timeout when it's over/cancelled should be put back to undefined.
Since a React State is encapsulated, it's not overly helpful to make some of its keys optional, but just removing the ? still doesn't make your example compile.
  interface C {
    foo: string | undefined;
  }
  let c: C = {}; // OK
  let c2: C = {
      foo: undefined,
  } // OK

  function setState<T, K extends keyof T>(obj: T, state: Pick<T, K>) {
      for (let k in state) {
          obj[k] = state[k];
      }
  }

  setState(c, c2);
// error TS2322: Type '{}' is not assignable to type 'C'.
 // Property 'foo' is missing in type '{}'.

@AlexGalays
  let c: C = {}; // OK

Does that actually work? @RyanCavanaugh's example above that didn't work.

Just to be clear, my comment above was normative not descriptive. Today we considered { x?: number } to be the same as { x: number | undefined } for the purposes of reading x. But we are thinking of changing this to more explicitly model "present but undefined" and "missing"

If we are considering {x?: number} to be the same as {x: number | undefined} currently, then currently there is a bug, given the example I pasted above, no?

@masonk  Hugh, you're right, the error is at this line rather than the last.

In the playground right now, here is what actually happens. Unexpected* things happens when I pick optional keys.
interface A {
    foo?: string;
}
interface B {
    foo: string | undefined;
}
interface C {
    foo?: string | undefined;
}

let a: A = {};
let a2: A = { foo: "ok" }
let a3: A = { foo: undefined };

let b: B = { }; // (As expected) Error: Property "foo" is missing
let b2: B = { foo: "ok" };
let b3: B = { foo: undefined };

let c: C = {};
let c2: C = { foo: "ok" }
let c3: C = { foo: undefined }

function setState<T, K extends keyof T>(obj: T, state: Pick<T, K>) {
    for (let k in state) {
        obj[k] = state[k];
    }
}
setState(a, {}); // Ok (As expected)
setState(a, a2); // (Unexpected) Error: foo is optional in type 'A' but required in type Pick<A, "foo">
setState(a, a3); // (Unexpected) Error: Argument of type 'A' is not assignable to parameter of type Pick<A, "foo">

setState(b, {}); // Ok (As expected; I have picked 0 keys from B)
setState(b, b2); // Ok (As expected)
setState(b, b3); // Ok (As expected)

setState(c, {}); // Ok
setState(c, c2); // (Unexpected) Error: foo is optional in type 'C' but required in type Pick<C, "foo">
setState(c, c3); // (Unexpected) Error: Argument of type 'C' is not assignable to parameter of type Pick<C, "foo">

To me.

We've been discussing this and have come to the consensus that Pick<T, K> should copy the modifiers from T which it currently doesn't do. More specifically, for a mapped type { [P in keyof T]: X } we currently copy the modifiers from T, and we want to do the same for a mapped type { [P in K]: X } where K is a type parameter with the constraint K extends keyof T.

Thanks @ahejlsbeeg
Will this be in 2.2 or the next 2.1

You guys are the best. Thank you. (By the way, mapped types are an amazing feature; I love them. In fact, 2.1 is an amazing release over all and ya'll are killing it over there).

I just want to remark that the rapidity of this feedback loop is awesome! An issue comes up, is discussed and gets resolved in less than 2 days, and this is a programming language we are talking about. Great work!

sure thing mapped types got some love, let's not skip the legs day tho

I found another problem with using Pick for typing setState, which happens when updating state conditionally.
declare class Component<P, S> {
    setState<K extends keyof S>(f: (prevState: S, props: P) => Pick<S, K>, callback?: () => any): void;
}

declare const component: Component<{}, { foo: number; bar: string; }>;
component.setState(() => 0 === 0 ? { foo: 0 } : { bar: '' });

The last line raise an error, which says
error TS2345: Argument of type '() => { foo: number; } | { bar: string; }' is not assignable to parameter of type '(prevState: { foo: number; bar: string; }, props: {}) => Pick<{ foo: number; bar: string; }, "foo...'.
  Type '{ foo: number; } | { bar: string; }' is not assignable to type 'Pick<{ foo: number; bar: string; }, "foo" | "bar">'.
    Type '{ foo: number; }' is not assignable to type 'Pick<{ foo: number; bar: string; }, "foo" | "bar">'.
      Property 'bar' is missing in type '{ foo: number; }'.

Is it possible to make Pick<T, K1> assignable to Pick<T, K2> if K1 is assignable to K2 to resolve the above problem?

Yeah, typescript has a lot of issues inferring from function returns. I don't think it's an issue with Pick especially.
It infers that your Pick must have the full set of keys.

The compiler should probably infer the return value to be 'Pick<T, "foo"> | Pick<T, "bar">' instead. Then it could infer that both of those are assignable to 'Pick<T, K>' I believe.

@ericanderson
Ah, yes, you are right.
I was misunderstanding Pick and this comment is incorrect.

I think if K1 is assignable to K2, then Pick<T, K1> should be assignable to Pick<T, K2>.

Well, certainly the issue I described is not specific to Pick.
function f(s: string): void;
function f(n: number): void;
function f(args: string | number): void {}

f(''); // no error
f(0); // no error
f(0 === 0 ? '' : 0); // error: Argument of type '"" | 0' is not assignable to parameter of type 'number'. Type '""' is not assignable to type 'number'.

It is likely that although you can define an argument of a function as one of multiple types, it must have a single type per use.

I don't totally understand the @ahejlsberg solution - we're supposed to redefine setState on every component to get TS to settle down about setState with a partial object?

I agree with @lukecwilliams's not getting how this is solved, especially given the title of this issue. Partial still won't work with setState, since setState uses Pick. Please reopen @ahejlsberg or describe how Partial< T > is compatible with Pick< T, ... >, or conclude that setState should have another signature. Anything but leaving it closed when it's still an issue.
The following does not work, and is a not-so-uncommon way to partially build up a state somewhere, to then apply it with setState. Please consider foo to be a lot more complex and not as trivial as this example, where alternative logic would obviously suffice.
interface State
{
	a: string;
	b: number;
}

class Comp extends React.Component< { }, State >
{
	foo( )
	{
		const state: Partial< State > = { };
		if ( !0 ) // obviously apply some useful logic here
			state.a = "foo";
		this.setState( state );
	}
}

It fails with:
TS2345: Argument of type 'Partial<State>' is not assignable to parameter of type 'Pick<State, "a" | "b">'.
  Property 'a' is optional in type 'Partial<State>' but required in type 'Pick<State, "a" | "b">'.

Yes, one can Hawaii-cast things to make tsc happy, à la:
this.setState( state as State );

or, pick your poison:
const state = { } as State;

but this is not a solution. Or is it? I'm personally resentful to blatantly lie about types this way. It's a bad habit.

I have another example of some strangeness around the Pick infrastructure as used in React types specifically. Unfortunately I don't fully understand it but this is something I ran into:
typescript version is 2.6.2 and using "@types/react": "^16.0.31"
export type ExportProperties = {
    hpid: boolean,
    gender: boolean,
    age: boolean,
    diagnosisCode: boolean,
    medicationCode: boolean,
    procedureCode: boolean
};

export class StudyExportComponent extends React.Component<{ study: Study }, ExportProperties> {

    constructor(props: { study: Study }) {
        super(props);
        this.state = {
            hpid: true,
            gender: false,
            age: false,
            diagnosisCode: false,
            medicationCode: false,
            procedureCode: false
        };
        this.handleCheck = this.handleCheck.bind(this);
        this.handleCheck2 = this.handleCheck2.bind(this);
        this.handleCheck3 = this.handleCheck3.bind(this);
    }

    handleCheck(name: keyof ExportProperties) {
        return (event: any) => {
            let state = {};
            state[name] = event.target.checked;
            this.setState(state);
        };
    }
    handleCheck2(name: 'hpid' | 'gender' | 'age' | 'diagnosisCode' | 'medicationCode' | 'procedureCode') {
        return (event: any) => {
            let boolValue: boolean = event.target.checked;
            this.setState({ [name]: boolValue });
        };
    }
    handleCheck3(name: keyof ExportProperties) {
        return (event: any) => {
            let boolValue: boolean = event.target.checked;
            this.setState({ [name]: boolValue });
        };
    }
// Clipped rendering stuff

With the following errors:
This first one is for handleCheck2
src/components/study/study-export.tsx(41,27): error TS2345: Argument of type '{ [x: string]: boolean; }' is not assignable to parameter of type 'ExportProperties | ((prevState: Readonly<ExportProperties>, props: { study: Study; }) => ExportPr...'.
  Type '{ [x: string]: boolean; }' is not assignable to type 'Pick<ExportProperties, "hpid" | "gender" | "age" | "diagnosisCode" | "medicationCode" | "procedur...'.
    Property 'hpid' is missing in type '{ [x: string]: boolean; }'.

41             this.setState({ [name]: boolValue });
                             ~~~~~~~~~~~~~~~~~~~~~

And this one for handleCheck3:
src/components/study/study-export.tsx(47,27): error TS2345: Argument of type '{ [x: string]: boolean; }' is not assignable to parameter of type 'ExportProperties | ((prevState: Readonly<ExportProperties>, props: { study: Study; }) => ExportPr...'.
  Type '{ [x: string]: boolean; }' is not assignable to type 'Pick<ExportProperties, "hpid" | "gender" | "age" | "diagnosisCode" | "medicationCode" | "procedur...'.
    Property 'hpid' is missing in type '{ [x: string]: boolean; }'.

47             this.setState({ [name]: boolValue });
                             ~~~~~~~~~~~~~~~~~~~~~

My work-around is clearly in handleCheck which does not have any TS errors but as you can see is clearly working around types in this case (at least I think it is).
I also just noticed that this compiles just fine:
this.setState({ [name]: boolValue } as Pick<ExportProperties, keyof ExportProperties>);

Now that I look at this a little more maybe that's the appropriate thing to do...
Now just to be clear none of this is failing the most common use-case as shown below:
this.setState({ hpid: true });

Hopefully this post is helpful to others even if not actionable to the Typescript language.

I think you're types could be improved. Specifically to prevent the problem, change the type of name in your functions to keyof ExportProperties. I bet that solves your problem (and also means you dont have to update that function everytime you add something to ExportProperties).
Unrelated but tip, if you use => notation for your methods, you dont have to manually bind them, they are bound for you.


chat commands: add commonly used aliases

This PR...

Adds the temp alias for Tempoross.

Currently Tempoross only has somewhat obscure aliases like fishtodt. Everyone I know refers to it as temp.

Adds the graar alias for General Graardor.

This is probably the second most used alias for him (after bandos). Feel free to correct me if I'm crazy.

Removes the gauntlet alias for Gauntlet.

An alias that's the same as the longBossName except for its capitalization is superfluous.

I looked at the wiki page visit analytics for a few aliases:

Alias
Views

bando
298

bandos
17268

graardor
3542

graar
106

garage door
206

so I don't think graar is needed.


Reimplement Chess logic using Bitboards technique

This is a very early PR. I created for the pure purpose of not being lazy.
The goal is implementing Biboards technique for scalachess which will (hopefully) improve scalachess performance.
This is a port from my experimental library Schack which in turns is inspired by shakmaty and compression.
Previous effort: #210
blocked by #336
TODO

[x] validate chess position
[x] support variants
[x] port tests
[x] integrate more with the rest of the library

Current benchmarks comparison between this bitboard implementation vs master:
jmh visualization

14.0.0-RC2 published on https://github.com/lichess-org/lila-maven

14.0.0-RC3 published on https://github.com/lichess-org/lila-maven

Latest benchmark result: https://jmh.morethan.io/?sources=https://gist.githubusercontent.com/lenguyenthanh/6c31a1060242d72b94d1fabe37f7adfb/raw/2daebf95248a12e671052b0c357e34bca154ac0e/master.json,https://gist.githubusercontent.com/lenguyenthanh/6c31a1060242d72b94d1fabe37f7adfb/raw/e3e9a3299c1c88002970d6f025ac291ecb02a36c/bitboard-d628b9b.json


MPCF: Connection is not established after 21 virtual connections

So I ran Can shift data via parallel connections test using version that contains fix for zombies, on iOS devices, multiple times and it turned out, that it failed after 8 test. What was interesting is that it failed exactly at the same time when I ran it multiple times, so every time 8 tests were passing, and 9 one was failing due to timeout.
Enrico investigated the issue and it turned out, that the problem occurs after 21 virtual connections have been used, at the 22nd connection the peer will disconnect. He tested it using 1, 2, 3, and 4 parallel connections and test failed after 21, 10, 7 and 5 run, always.
So it looks like we don't do something right when we close the virtual sockets
What might be interesting that it might be connected to the issue with replication on iOS, when it works only for some short period of time and then nothing happens, see #1890.

Fixed in https://github.com/thaliproject/thali-ios/commits/master-enrico-error-handling-tmp


Works fine locally (NPM/Webpack) but breaks when in production

Firstly, thanks for creating such a great plugin. Seems to work brilliantly when I run it locally. In production, however, it breaks. No console errors and when you click an image you are just taken to the source in a new window. I am using Bootstrap 5.
Am I doing something wrong?
App.js (entry file)
import 'bootstrap';
import 'bs5-lightbox';

File.html

When I view the source js file in production there is no reference to data-toggle but there is in development mode.
Appreciate any advice/guidance.

Seems I have to initialise even with a data-lightbox
import Lightbox from 'bs5-lightbox';
document.querySelectorAll('[data-lightbox]').forEach(el => el.addEventListener('click', Lightbox.initialize));


Date field allow nullable save

CURD Field type date is not allowing to save null value.
I have a column in database which is nullable in migration definition. Until I don't save value it's working as expected but after saving value i don't have any option to save it NULL again.
CRUD::field('expiry')->type('date')->allows_null(true);
I was expecting by passing allows_null(true) it should allow me to save null again in database but it's neither throwing any error nor allowing me to empty pre filled value.
Please let me know how can i do this ?

Hey @raviroshanmehta
The Date field uses the browser's date picker. Some browsers give clear options, while some don't.
For example, Google Chrome:


Running control refactor

Pull Request Test Coverage Report for Build 725514122

0 of 12 (0.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.05%) to 25.487%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

utils/running_controller.go
0
12
0.0%

Totals

Change from base Build 725382659:
-0.05%

Covered Lines:
1713

Relevant Lines:
6721

💛  - Coveralls


klustakwik2 settings not all making it properly from the prm file to the execution of kk2

It seems that use_mua_cluster doesn't get set correctly, subset doesn't get set at all, whereas points_for_cluster_mask and num_starting_clusters do get set correctly.
My .prm file has this:
klustakwik2 = {
    'drop_last_n_features': 1,
    #'max_split_iterations': None,
    'full_step_every': 1,
    #'num_changed_threshold': 0.05,
    #'penalty_k': 0.0,
    'split_every': 50,
    #'break_fraction': 0.0,
    'always_split_bimodal': True,
    'consider_cluster_deletion': 1,
	'subset': 50,
    #'subset_break_fraction': 0.01,
    #'max_quick_step_candidates': 100000000,
    'split_first': 20,
    #'mua_point': 2,
    #'max_quick_step_candidates_fraction': 0.4,
    #'dist_thresh': 9.21034037198,
    'max_iterations': 1000,
    'max_possible_clusters': 2001,
    #'penalty_k_log_n': 1.0,
    'points_for_cluster_mask': 150,
    #'fast_split': False,
    #'prior_point': 1,
    #'noise_point': 1,
    #'use_noise_cluster': True,
    'use_mua_cluster': False,
    'num_starting_clusters': 2000,
    'save_clu_every': 10
}

My .log file shows this:
2015-07-23 18:02:24  [D]  logging:110      Running SpikeDetekt with the following parameters: {'consider_cluster_deletion': 1, 'use_single_threshold': 1, 'connected_component_join_size': 2, 'probe_adjacency_list': {[omitted]}, 'sample_rate': 25000, 'points_for_cluster_mask': 150, 'subset': 50, 'always_split_bimodal': 1, 'n_excerpts': 50, 'num_starting_clusters': 2000, 'n_features_per_channel': 3, 'chunk_size_seconds': 1.0, 'split_every': 50, 'raw_data_files': '/home/nick/data/Ehrlich/20150626/20150626_all_shifted.dat', 'waveform_dc_offset': '', 'threshold_weak_std_factor': 2.0, 'full_step_every': 1, 'extract_s_before': 10, 'filter_high_factor': 0.47499999999999998, 'split_first': 20, 'chunk_overlap_seconds': 0.014999999999999999, 'filter_butter_order': 3, 'extract_s_after': 24, 'dtype': 'int16', 'use_mua_cluster': 0, 'waveform_scale_factor': '', 'detect_spikes': 'negative', 'filter_low': 500.0, 'probe_channels': [etc]

But my kk2 output shows this:
18:22:46 [I] Running klustakwik...
INFO     klustakwik.initial_parameters: consider_cluster_deletion = 1
INFO     klustakwik.initial_parameters: split_first = 20
INFO     klustakwik.initial_parameters: mua_point = 2
INFO     klustakwik.initial_parameters: noise_point = 1
INFO     klustakwik.initial_parameters: max_quick_step_candidates = 100000000
INFO     klustakwik.initial_parameters: max_quick_step_candidates_fraction = 0.4
INFO     klustakwik.initial_parameters: always_split_bimodal = 1
INFO     klustakwik.initial_parameters: num_starting_clusters = 2000
INFO     klustakwik.initial_parameters: dist_thresh = 9.21034037198
INFO     klustakwik.initial_parameters: penalty_k = 0.0
INFO     klustakwik.initial_parameters: full_step_every = 1
INFO     klustakwik.initial_parameters: use_noise_cluster = True
INFO     klustakwik.initial_parameters: max_split_iterations = None
INFO     klustakwik.initial_parameters: fast_split = False
INFO     klustakwik.initial_parameters: use_mua_cluster = True
INFO     klustakwik.initial_parameters: break_fraction = 0.0
INFO     klustakwik.initial_parameters: prior_point = 1
INFO     klustakwik.initial_parameters: points_for_cluster_mask = 150
INFO     klustakwik.initial_parameters: penalty_k_log_n = 1.0
INFO     klustakwik.initial_parameters: split_every = 50
INFO     klustakwik.initial_parameters: subset_break_fraction = 0.01
INFO     klustakwik.initial_parameters: max_iterations = 1000
INFO     klustakwik.initial_parameters: max_possible_clusters = 2001
INFO     klustakwik.initial_parameters: num_changed_threshold = 0.05
INFO     klustakwik.initial_parameters: use_noise_cluster = True
INFO     klustakwik.initial_parameters: use_mua_cluster = 0

Note that there are evidently two "use_mua_cluster" parameters that kk2 is trying to use. cc @thesamovar .

so weird that the parameter is present twice in the initial parameters! @thesamovar any ideas?

OK I think this should be fixed now.

You want me to do another release of KK2?

OK, I did a 0.2.3 release with this fixed in it.

thanks @thesamovar !
@nsteinme let us know if that fixed it, if so you can close this issue

Sorry for taking a while to test this. I believe it does not seem to be fixed. First - it's a little difficult to answer this because there is no log file produced of the KK2 output anymore, is that true? If there is I'm not sure where to look for it. So I just have to watch the terminal carefully...
But I think it's not working because my parameters file has 'subset': 50, and 'full_step_every': 3, but it's doing neither of those things. "subset" does not show up in the list of klustakwik.initial_parameters at all (and indeed it is trying to use all 2.5million spikes), and full_step_every = 1. num_starting_clusters =2000 despite that my prm file specifies 500, however points_for_cluster_mask=150, correctly, which I think is not the default value.
Is it possible that it is using some values that were embedded in the kwik file during the initial run, and that it is not looking back into the prm file for the updated parameters I would like to use? It should look back to the prm on each call I guess since otherwise you cannot do different runs of KK with different parameters without re-doing everything incl. spike detection from the top, yeah?
This is the call I make:
$ phy cluster-auto params20150626_phy.prm
This is using phy on pyramidal, phy -v gives:
0.2.2.dev0-git-v0.2.1-200-g7f91b41c
KK2 reports:
KlustaKwik2 version git-0.2.4-1-g2dac4576

I'm removing "done" unless you think I did something wrong in my test. Let me know, thanks -

can you close this and open 2 new issues:

kk2 log
take params from PRM instead of KWIK in phy cluster-auto

thanks

Sure


Investigate eliminating circe from core

We want to cut a 1.0.0 release, at least for the core modules, however to respect semantic versioning that means we can't have any unstable dependencies. Currently the core module depends on circe, which hasn't reach 1.0.0 yet.
Replacing circe in core might be desirable for performance reasons: it's only used for constructing the query result value, and it's quite likely that it can be replaced by something both stable and more performant. the Json type does show through in top level public signatures, but that's probably a mistake anyway and should be changed before Grackle's 1.0.0.
This needn't affect the circe module which can track circe's version number.

Consensus is that we'll keep the core dependency on circe. This means we'll need to bump the major version if we make a binary incompatible update to circe.


Add tourism page

For more details, open the Copilot Workspace session.

@CasP0 I don't understand what this PR is about, is it to test the GitHub copilot workspace?


Train the UFF problem

Hello author, I encountered this error while training UFF. May I know how to resolve it, thank you!!
root@autodl-container-b9e411b050-3a77f4bb:~/autodl-tmp/M3DM# OMP_NUM_THREADS=1 python3 -m torch.distributed.launch --nproc_per_node=1 fusion_pretrain.py --accum_iter 16 --lr 0.003 --batch_size 16 --data_path datasets/patch_lib
/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects --local_rank argument to be set, please
change it to read from os.environ['LOCAL_RANK'] instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions
warnings.warn(
| distributed init (rank 0): env://, gpu 0
[10:57:46.999771] job dir: /root/autodl-tmp/M3DM
[10:57:46.999924] Namespace(accum_iter=16,
batch_size=16,
blr=0.002,
data_path='datasets/patch_lib',
device='cuda',
dist_backend='nccl',
dist_on_itp=False,
dist_url='env://',
distributed=True,
epochs=3,
gpu=0,
input_size=224,
local_rank=0,
log_dir='./output_dir',
lr=0.003,
min_lr=0.0,
num_workers=0,
output_dir='./output_dir',
pin_mem=True,
rank=0,
resume='',
seed=0,
start_epoch=0,
warmup_epochs=1,
weight_decay=1.5e-06,
world_size=1)
[10:57:47.000363] <dataset.PreTrainTensorDataset object at 0x7f2d6f6a46d0>
[10:57:47.000448] Sampler_train = <torch.utils.data.distributed.DistributedSampler object at 0x7f2c9833a700>
[10:57:47.002473] base lr: 3.00e-03
[10:57:47.002497] actual lr: 3.00e-03
[10:57:47.002515] accumulate grad iterations: 16
[10:57:47.002534] effective batch size: 256
[10:57:47.132814] AdamW (
Parameter Group 0
amsgrad: False
betas: (0.9, 0.95)
eps: 1e-08
lr: 0.003
weight_decay: 0.01
)
[10:57:47.132945] Start training for 3 epochs
[10:57:47.133200] log_dir: ./output_dir
[W reducer.cpp:1303] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
Traceback (most recent call last):
File "fusion_pretrain.py", line 201, in
main(args)
File "fusion_pretrain.py", line 171, in main
train_stats = train_one_epoch(
File "/root/autodl-tmp/M3DM/engine_fusion_pretrain.py", line 48, in train_one_epoch
loss /= accum_iter
RuntimeError: Output 0 of _DDPSinkBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 6865) of binary: /root/miniconda3/bin/python3
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
elastic_launch(
File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
fusion_pretrain.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time      : 2023-11-08_10:57:50
host      : autodl-container-b9e411b050-3a77f4bb
rank      : 0 (local_rank: 0)
exitcode  : 1 (pid: 6865)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Changing  loss /= accum_iter (line 58 in engine_fusion_pretrain.py) into  loss = loss / accum_iter fixes the problem.


made some ui enhancements, corrected grammatical errors in data

in data, there were grammatical errors and and letter case incorrect so fixed that
In the search input which we see on first page, py was bit large so fixed that too
also some other ui enhancements to make the site better

mention issue here !

mention issue here !

Incorrect letter casing in data and py on the home page is bit large #55

Great @mohit200008  Looking forward for more contribution !


Support channels

It would be great if channels could be yielded as an alternative to utilizing promises and node style callbacks. In general I would prefer to utilize generators for all async behavior to avoid common pitfalls with promises and callbacks. Channels also the obvious benefit of handling multiple updates and supporting functional mutation out of box.

First, I'm glad you opened this issue
In the csp branch you'll find I'm already experiencing with this
Code
https://github.com/yelouafi/redux-saga/blob/csp/src/internal/channel.js
Test
https://github.com/yelouafi/redux-saga/blob/csp/test/channels/channel.js
redux-saga is not quite csp. the take model is more close to eventEmitter (or hot observables in Rx) because it doesn't buffer events (principally for technical reasons)
My goal is not to add a feature for itself. Until now I tried to adopt a pragmatic approach, I only add features needed by the users (cancellation, select). Actually what I think the lib lacks is some way to connect to external event sources (websockets, obervables ...) I'm more willing for an event iterator concept : An iterator that connects to the event source and buffer events in order for sagas to take them. It means Sagas will not miss those events even if they were blocked while those events happened: i.e. they will be processed in order. It's like a half of csp channels (just the take side of a channel).
CSP channels adds support for synchronous puts: A saga can 'put'  a message for another/others saga(s), and the put will block until one Saga takes that message, The model can handle inter-process communication and synchronisation using the famous rendez-vous mechanism, but I'm not sure if this use case is really needed by redux/saga users, plus it's not easy to grok by people (especially with buffered channels: dropping, sliding ...). I don't want to provide some complicated feature that no one or only few persons will use.
I'm also exploring some similar ideas like the linda programming model. Which is similar to csp but uses fewer semantics (a global channel called the tuple space). You can find some of my experiments here
https://github.com/yelouafi/linda/blob/master/examples/gen/hello.js
I don't plan to add this model to redux-saga. Just trying to find some inspiration

Thank you for the detailed explanation.
I could see you getting the same issue opened for Rx, bacon, etc. What may be a better solution is to provide a facility for decorating call. Alternatively, I like the idea of actually implementing a pure eventEmitter since adapters already exist for many observable implementations. If you consider the approach of calling Promises or node style callbacks, the current philosophy seems to be one of interoperability with popular async strategies. With that said, I am very interested to see what you come up with as an alternative observable mechanism.
If I rephrased the issue to not be channel specific I think it may go something like...
"I would like to use my observable based framework that my code base already depends on with sagas".

We have a use-case for this as well since we're using websockets (via pusher.js). We thought about using the deferred pattern found in #51 but it is more code than we feel is necessary and we're worried that we might drop events.

see API proposal #254


Error in building of topology graph in solr loading

From<EMAIL_ADDRESS>on November 03, 2014 16:36:15
Test case:
OWLTools-Solr/src/test/resources/topology-test.obo
See also:
<a href="http://jira.geneontology.org/browse/GO-626" rel="nofollow">http://jira.geneontology.org/browse/GO-626

The topology graph edge between

GO:0030414 (peptidase inhibitor activity
and
GO:0008233 (peptidase activity)

Should not be part_of.

The paths are:

P, NR
I, R (which reduces to R)

This is calculated directly by the OGW. For the topology graph, all that needs to be done is a straightforward translation of the list returned by OGW, with reduced paths of length>1 discarded.

The problem here may reside in getEdgeLabel in OGWE, which makes a dangerous call to getSingleQuantifiedProperty.

Original issue: http://code.google.com/p/owltools/issues/detail?id=105

From<EMAIL_ADDRESS>on November 04, 2014 12:51:56
Correction: transitivity graph

From<EMAIL_ADDRESS>on November 04, 2014 15:05:58
There is now some better documentation for purpose and structure of the GOlr graphs (via BBOP JS): https://github.com/kltm/bbop-js/wiki/Graph https://github.com/kltm/bbop-js/wiki/TopologyGraph https://github.com/kltm/bbop-js/wiki/TransitivityGraph
Cc<EMAIL_ADDRESS>

From<EMAIL_ADDRESS>on November 06, 2014 14:21:40
Pushed a fix for the issue in r2400 . The correct results are now calculated using the ExpressionMaterializingReasoner for Amigo/Solr.
See also the test case OWLTools-Solr/src/test/java/owltools/solrj/TransitivityGraphTest.java
Status: Started
Labels: -Priority-Medium Priority-High

From<EMAIL_ADDRESS>on November 21, 2014 11:07:03
A failing integration test related to GAF Solr loading revealed some issues on the memory requirements (and unnecessary computations) for the setup of the ExpressionMaterializingReasoner.
The strategy is now to only create expressions for a requested subset of properties.


Generating properties with transformations

Hello again,
I have another newbie question if you please.
Is it possible to generate properties (not only properties' values) in the values set by transformations?
Something like that:
hull:
  object:
    deployment:
      my-cool-deployment:
        pod:
          _HT_ADD_PROPERTIES: _HT!{{ something_that_generates_a_dictionary }}

The _HT_ADD_PROPERTIES could be some code word to let HULL know that we're going to add some properties.
Thank you for your patience and, again, for a really nice tool!

Hi @khmarochos,
if I understand correctly you want to create an arbitrary number of properties via a transformation and add them to an (arbitrary) dictionary having the possibility to specify additional keys as well?
If so, that is not possible directly but can be done I think. I'll try to explain using the pod dictionary from above:
Let's assume you have a deployment like this:
hull:
  objects:
    deployment:
      example:
        pod:
          containers:
            main:
              image:
                repository: somerepo
                tag: "latest"

and you want to add some additional properties dynamically to the pod. At the moment there is no way to do that on the pod itself, transformations normally fully replace values and not add key-values to dicts where they are used.
But you can utilize the sources feature for this I think. If you put a transformation for the pod value in another object you don't render but reference as a source, this transformation is evaluated first and afterwards merged with the object which references it as a source under sources:
hull:
  objects:
    deployment:
      pod-values:
        enabled: false
        pod: |-
          _HT!{
            "replicas": 5
          }
      example:
        sources:
        - pod-values
        pod:
          containers:
            main:
              image:
                repository: somerepo
                tag: "latest"

So pod in pod-values is evaluated to replicas: 5 and this replicas: 5 is merged to the definition of example. pod-values deployment is disabled and will not be rendered at all.
In the end you have in the example deployment:
...
    spec:
      containers:
      - env: []
        envFrom: []
        image: somerepo:latest
        name: main
        ports: []
        volumeMounts: []
      imagePullSecrets:
      - name: release-name-hull-test-example-registry
      - name: release-name-hull-test-local-registry
      initContainers: []
      replicas: 5
      serviceAccountName: release-name-hull-test-default
      volumes: []

Hope it brings the idea across, more information on sources is here.
Quite late now so can't add more info at the moment. If you need more support please let me know and I will get back to you right after christmas days. Thanks.

I'd like to appreciate the detailed documentation you provided for the project. If I had been more careful, I'd have noticed the sources parameter before. Thank you!


tests: put (back) plugins tests in a single linkFarm

Thanks for testing this out. I'll try and dig up the command we were using to check memory usage.
I've marked it as a draft because we'll also need to re-work the tests devshell command and probably revert 71126bfebeae32cd731d70705295804f3da3f158 too.

If we need to, how complex do you think it'll be to split the tests over a handful of link farm derivations? Either based on test-sources/* paths or arbitrarily distributed evenly?

With our new hardware a single linkfarm by itself won't OOM but for the couple of minutes it takes to evaluate it won't leave much for everyone else. It is an option but only just, and it can't really afford to get much bigger.

I'll try and dig up the command we were using to check memory usage.

https://github.com/nix-community/nixvim/issues/1878#issuecomment-2240793881

split the tests over a handful of link farm derivations? Either based on test-sources/* paths or arbitrarily distributed evenly?

The command I gave was only using a single worker as sequential eval made it easier to see problem locally. On the old hardware we were using 16 workers for eval, now we are using 32. Only splitting the linkfarm up a bit (e.g. into 4) likely won't help much as they will still end up being evaluated at the same time but by more than that or by path should work.
I'm hoping we'll have some improvements land in buildbot-nix over the next couple of weeks, my short term solution to speed thing up would be to reduce the amount of tests by platform, e.g. full tests on x86_64-linux / aarch64-darwin and only a smoke test on aarch64-linux / x86_64-darwin.

With our new hardware a single linkfarm by itself won't OOM but for the couple of minutes it takes to evaluate it won't leave much for everyone else. It is an option but only just, and it can't really afford to get much bigger.

Thanks for clarifying! In that case I don't think we want to go back to a single link farm until buildbot is able to handle it without (potential) issue.

I'll try and dig up the command we were using to check memory usage.

#1878 (comment)

Thanks!

split the tests over a handful of link farm derivations? Either based on test-sources/* paths or arbitrarily distributed evenly?

The command I gave was only using a single worker as sequential eval made it easier to see the problem locally. On the old hardware we were using 16 workers for eval, now we are using 32. Only splitting the linkfarm up a bit (e.g. into 4) likely won't help much as they will still end up being evaluated at the same time but by more than that or by path should work.

Currently we split the tests up one-per file (i.e. test-sources/**), this reduces eval memory a lot, however the quantity of builds it results in leads to a number of other issues:

nix flake show takes around 45s and produces a lot of output.
the buildbot web UI becomes sluggish and struggles to deal with the quantity of builds it needs to get metadata for
buildbot is unable to run all queued builds simultaneously, leading to much slower builds where many builds are sat waiting for an available builder

Only splitting the linkfarm up a bit (e.g. into 4) likely won't help much

Are you saying we'd need in the region of 32 items being built by nix flake check?
test-sources/* won't help much there:
$ ls -ld tests/test-sources/*
drwxr-xr-x 1 matt users  20 Aug 13 13:31 tests/test-sources/example-configurations/
-rw-r--r-- 1 matt users 298 Aug 13 13:31 tests/test-sources/examples.nix
-rw-r--r-- 1 matt users 429 Aug 13 13:32 tests/test-sources/extended-lib.nix
drwxr-xr-x 1 matt users 348 Aug 13 13:32 tests/test-sources/modules/
drwxr-xr-x 1 matt users 258 Aug 13 13:31 tests/test-sources/plugins/

However ls -ld tests/test-sources/*/* prints 32 lines. For contrast ls -ld tests/test-sources/** prints 306 lines!
We also have (a handful of) other checks that aren't under test-sources, so we wouldn't have to hit the number exactly.

I'm hoping we'll have some improvements land in buildbot-nix over the next couple of weeks, my short term solution to speed things up would be to reduce the amount of tests by platform, e.g. full tests on x86_64-linux / aarch64-darwin and only a smoke test on aarch64-linux / x86_64-darwin.

Personally, I'd like a way to indicate to buildbot whether a PR is ready for a "full" check or should do a "light" check (e.g. only on x86_64-linux).
This would fit well with mergify's "two-step CI" strategy, where lightweight checks are run on all PR pushes, but heavy checks are only run on PRs queued for merging.
Queued PRs could be identified with a label, for example.

Sorry, just realised that you were asking about this on matrix and I didn't reply. If you don't get a response after a couple of days feel free to ping me.

nix flake show takes around 45s and produces a lot of output.

Yeah, using #checks is not really ideal for a project of the size of nixvim, I'd open an upstream issue about this.

buildbot is unable to run all queued builds simultaneously, leading to much slower builds where many builds are sat waiting for an available builder

Smarter scheduling will help but we only have limited darwin capacity so that will be a problem regardless. Also we still have another two CI systems so other builds could be running even if there isn't a buildbot queue.

Only splitting the linkfarm up a bit (e.g. into 4) likely won't help much

Are you saying we'd need in the region of 32 items being built by nix flake check?

However ls -ld tests/test-sources/*/* prints 32 lines.

We also have (a handful of) other checks that aren't under test-sources, so we wouldn't have to hit the number exactly.

This PR has 72 items in total but the 4 linkfarms (1 per platform) end up in memory at the same time as the small jobs are processed quicker. 16 linkfarms (4 per platform) probably won't change much, they'll still be the slowest jobs that'll accumulate in memory.

Personally, I'd like a way to indicate to buildbot whether a PR is ready for a "full" check or should do a "light" check (e.g. only on x86_64-linux).

This would fit well with mergify's "two-step CI" strategy, where lightweight checks are run on all PR pushes, but heavy checks are only run on PRs queued for merging.

I'd open an upstream issue about this as well.

Thank you very much for those insights @zowoq.
One solution could be to fall back to some imperative shell script which would force the CI process to happen somehow sequentially.
This would artificially limit the amount of eval/builds that can happen simultaneously.
Of course, I would rather not have to do that and keep the exhaustive .#checks approach.
This turns out to be a not so easy problem !


Better Instruction visibility

Better visibility for instruction
When adding your name, don't add it at the beginning or end of the file. Put it anywhere in between.

Do rebase your local version of repository after your PR has been merged

Yeah sorry, realised this after making the PR.


GEODE-2988: Add a security example.

@pdxrunner @jaredjstewart @jinmeiliao @sbawaska @karensmolermiller
I don't like that I don't have an ExampleTest to go with the Example, but the example itself is doing a lot of security-testing in itself.  I couldn't think of a meaningful test that wouldn't be trivialized by mocking.  Otherwise, I tried to touch both the gfsh and the client side of things.  Far from comprehensive, but touches on a bit of everything.

When executing the superUser script, I received the following error on Step 12, the 'change loglevel' command:

Executing - change loglevel --loglevel=DEBUG --member=server1

Sep 11, 2017 2:28:32 PM org.springframework.shell.core.SimpleParser parse
WARNING: Option 'member' is not available for this command. Use tab assist or the "help" command to see the legal options

@davebarnes97 Yeah, it looks like my environment alias of gfsh points at the 1.3-SNAPSHOT, and the option aliasing (GEODE-2818) will be included in that.
I don't think anything else should be broken in these tests between v1.2 and v1.3, so there's certainly no harm it making it work for the older version.

Just a general comment: I thought the example would be mostly java client example, not the gfsh commands scripts. Probably I was mistaken?

Streamlined much of the example, per suggestions from @jaredjstewart.  I have not yet pruned many of the permissions from the example_security.json.
On the one hand, many roles are still used in the suggested gfsh scripts.  On the other hand and as @metatype points out, they may grow stale without being caught by the gradle script, since they would have to be explicitly run by the person playing with the example.
I think there is value in demonstrating security both in client code and in gfsh, but these extra scripts could be removed or spun out into a separate example if we think a more concise scope would be better.

I've split the gfsh and client side examples into separate examples.  I will close this PR for the time being, as the respective README.mds need to be updated and other considerations must be addressed before the work is finished.


Abbility to update cached data without deleting whole cache

Problem: I have created AWS Lambda function for preprocessing data. The lambda is triggered every time new file arrives in a S3 bucket, preprocess data and writes data into a new file on another S3 bucket. The problem occurs when two files arrive in the S3 bucket at an interval of ~ 30 seconds. Since the s3 = s3fs.S3FileSystem(anon=False) is initialized when lambda is triggered for the first file, it lists all files inside that S3 bucket and put that information inside cache, so when second file arrives, it throws FileNotFoundError error since the second file is not listed in the cache. This problem does not occur when files arrive at an interval of more than 5 minutes, because in that period the instance of the first Lambda function ends, so a new instance is initialized for the second file.
Also, the s3.invalidate_cache() does not resolve problem, because it erases the entire cache, which also contains the credentials for accessing the S3 bucket and after that it throws PermissionError: Access Denied Traceback error.
Current Solution: I was forced to call the s3.ls(refresh=True) command within the lambda function (with the refresh argument set to True), because I noticed in the source code that the cache is updated as part of this function.
New Feature: Creating a function that would be able to update cache or some part of the cache passed through the arguments of the function, so that we would not be forced to use functions we don't really need inside our code, just to change the state of the cache

Also, the s3.invalidate_cache() does not resolve problem, because it erases the entire cache, which also contains the credentials for accessing the S3 bucket

This should definitely not happen! I would say that you were doing the right thing, and any loss of credentials is due to something else that should be investigated.

Creating a function that would be able to update cache or some part of the cache passed through the arguments of the function, so that we would not be forced to use functions we don't really need inside our code, just to change the state of the cache

The listings cache is a fiddly thing. I'm not sure how your proposal would look exactly, but you should be aware that you can turn off directory caching altogether with use_listings_cache=False or set a time-to-live with listings_expiry_time=.


implemented support for prism monitor mode header

Thanks for the commit!  Just a few minor comments, and I think we'll be good to go.

Our Wireshark friends helped me out again, apparently there are two types (type1 and type2). I was looking at an old (or different spec) Implemented both the type1 and type2 Prism DID types.


Image Unification

This is the corresponding PR for https://github.com/mattgodbolt/compiler-explorer/pull/667
I open this so we don't forget about it, as the only change I've made so far is to remove the lang parameter from the run script.

Looks good: obviously will hold on merging this for now :)

Failure is:
  1) Paths Linux -> Windows path:
     TypeError: Cannot read property 'substring' of undefined
      at Compile.compile.filename (lib/compilers/WSL-CL.js:49:52)
      at Context.<anonymous> (test/win-path-tests.js:66:22)


plots of backtesting results

After running backtesting with algorithm such as MACDTrendAlgorithm, I got statistics on terminals, seems log files are  in
LEAN/Lean/Launcher/bin/Debug/
MACDTrendAlgorithm-log.txt
MACDTrendAlgorithm-order-events.json
MACDTrendAlgorithm.json
Is there any tool to plot these log files and get report? Thanks.

Hey!
Not at the moment sorry, the GUI was deprecated. We've recently migrated the whole cloud to now run VSCode -- and we're preparing the same VSCode technology to run locally in your desktop.

Is there any update on the VSCode GUI running on the desktop?


[Translation] Release MicroProfile 6.0 - Release Blog Post German Translation

Once third task in #726 about: Draft a release announcement blog post using template (technical person) is complete, add release blog post in German

I'll pick this :)

@emecas @dalexandrov Where are we at one this? Thank you :)

MicroProfile 6 is getting pretty stale. If we really don't have time for this, it may be best to just drop this if it gets too far past February.

@rstjames Please publish

Picture:
MicroProfie Version 6.0 ist veröffentlicht!

done

wonderful, @rstjames. When will the media post go out?

I will re-open this ticket and the other b/c it is helpful to at least add the Twitter post to help with closing the task.
This is part of the new guidelines on how to make Release translations that Mitia wrote, which I still  need to review and for that, all the examples must exist. Thanks!

Tweet : https://twitter.com/MicroProfileIO/status/1650436969686609921

@emecas, I added you as a contributor to the blog. :)


To support an input for RG name

Our environment does not allow users to create new RGs. Therefore, the RG needs to exist and be provided as an input for the temporary storage account location.

@tamirkamara can you review?

I'll will take care of this week

@ryholt @carpenike ping for updates

@ryholt I'm ok with your changes but the CI is failing and blocks my approval

Yup going to try to get it fixed this morning. The lint is failing because the line is too long.


Include ca-cert and cert for the admin node explicitly

It is included as a side effect because velum/init.sls includes them,
so make it explicit on the top file for the admin too.
feature#deployment-stability

We could also completely get rid of ca-cert and cert here. In any case I think it's good to also be consistent on the admin here.


Extensible variants reformat to Ocaml

module Attr = {
    type t = int;
};
type tag 'props = ..;
type ttype = Type;
type description 'props = {
  ttype: ttype,
  props: 'props
};
type element = Native (tag 'props) (description 'props): element;

type attrProps = { attr: Attr.t };

type tag 'props += Attr: tag attrProps;

module Attr = {
  type t = int;
};

type tag 'props = ..;

type ttype =
  | Type;

type description 'props = {ttype: ttype, props: 'props};

type attrProps = {attr: Attr.t};

type 'props tag +=
  | Attr: attrProps tag;

Notice how the extension of the variant gets reformatted to something halfway Ocaml, halfway Reason.

Whoever fixes this, please mark yourself as the assignee. Please add test cases as well.

@IwanKaramazow are you on master? If not, which version.

I'm on master. My package.json includes "reason": "facebook/reason".
Will fix this myself, is blocking me too much on RED

Okay thank you. I'll assign it to you.

This is done. @jordwalke cough up the money! lol

This is done. @jordwalke cough up the money! lol

Let me know if bounty source doesn't do its thing correctly.


Add VSCode tasks example and documentation

vs-code-containers sounds interesting. What would be the advantages over communicating with a Microfab container?

The real advantage of dev containers is anyone can pick up a codebase and develop in the same environment without any error prone manual setup, or clashes between tools for different pieces of work.


Cannot load playlist

It can load a playlist that includes a little like https://www.youtube.com/playlist?list=PL4C2OaC1jQqR3ICDBf4j1dH1Fk4uIo-Lx this URL.
But it cannot load a playlist that includes many things like https://www.youtube.com/playlist?list=PLzkuLC6Yvumv_Rd5apfPRWEcjf9b1JRnq
https://pastebin.com/raw/18eFR5hA this is an error log

Please see https://github.com/sedmelluq/lavaplayer/issues/259 as NoCopyrightSounds is one of the urls that causes this error. When this issue is resolved in Lavaplayer we will include it in a new release here.


TIDAL download support

Requested Feature
I'd love to be able to download using TIDAL while not having to convert playlists around. Of course it would have to be a paid or trial account the user would have to link somehow.
Possible implementation
No response

Tidal Media Downloader is a Python project we could integrate as a backend for downloading tracks from Spotify in high quality.
See: https://github.com/yaronzz/Tidal-Media-Downloader
If you reopen this issue I could give it a shot or even just test it.


Can I setup with local flutter sdk?

The problem is flutter over the internet is too slow. If Is there a way to use locally installed flutter SDK? It will save huge time saver.

Are you running on a self-hosted runner?
On self-hosted that have Flutter SDK installed locally you can use the CLI directly without using this action.


MIP102c2-SP35 amendment

Originating forum post: https://forum.makerdao.com/t/mip102c2-sp35-mip-amendment-subproposal-formal-submission/24423
Do not merge unless Ratification Poll passes

Poll did not succeed. Closing
https://vote.makerdao.com/polling/QmZJjiPy


Add a marker in sidebar for files that has comments on GitHub from a PR that I created?

There already is a feature that automatically shows the comments made on a PR on github when the file is opened. Can we also add a marker of some sort in the sidebar > file tree to indicate that the same file has comments? May be a black dot or something?

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

This feature request has not yet received the 20 community upvotes it takes to make to our backlog. 10 days to go. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

:slightly_smiling_face: This feature request received a sufficient number of community upvotes and we moved it to our backlog. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!


Swiftify object customization methods

@tgoyne @segiddins

Maybe some of these methods belong more naturally on RLMObjectSchema?

This change will make it impossible to provide default values for array properties


Fix query with dot notation on nested objects

 You should sign our Contributor License Agreement in order to get your pull request merged.


Audit instructs to wrongly remove revision.

Ref: https://github.com/Homebrew/homebrew-core/pull/3799
> require "cmd/audit"
> formula = Formula["luajit"]
> fv = FormulaVersions.new(formula, :max_depth => 10)
=> #<FormulaVersions:0x007ff6f3981650 @name="luajit", @path=#<Pathname:/usr/local/Library/Taps/homebrew/homebrew-core/Formula/luajit.rb>, @repository=#<Pathname:/usr/local/Library/Taps/homebrew/homebrew-core>, @entry_name="Formula/luajit.rb", @max_depth=10>
> revision_map = fv.revision_map("origin/master")
=> {}
> revisions = revision_map[formula.version]
=> []
> revision_map[formula.stable.version].empty?
=> true

Apparently caused by https://github.com/Homebrew/brew/commit/0b8eab8dac9586288b23160a93da3a72536e380d, so perhaps we were relying on stderr for the audit check here.
CC @MikeMcQuaid

The broken aspect that causes the later two incorrect empty outputs is revision_map = fv.revision_map("origin/master").
That should be returning:
=> {#<Version::FromURL:0x007fa1b8891028 @version="2.0.4">=>[2, 2, 2, 1, 1, 1, 0, 0, 0, 0], #<Version::FromURL:0x007fa1b8890a38 @version="2.1.0-beta2">=>[2, 2, 2, 1], #<Version:0x007fa1b88de058 @version="2.1">=>[1, 1, 0, 0, 0, 0]}


Little typo on the testing page

In the section that begins...
We could split the handler function from its registration, like this:
(defn get-completed-count
  [app-db _]
  (reaction (completed-count (:todos @db))))

:todos @db should be :todos @app-db

Thanks!  You should be able to directly edit those Wiki pages.  At least I hope you can.

Oh yeh, duly edited.


Convert to adoc format

The final goal is convert this repo into the RVI document format, but I think it's at least a good start to convert markdown to adoc format.
Convert by kramdown

Thanks again for creating this PR!
As we went ahead with #52, let's close this one.


Update da.json

fixed few small words to be correct! ;)

another one has been corrected! ;)
https://github.com/sorry-app/theme-whitelabel/pull/91/commits/4cab124f7247e8bd1deb541dcb2265577bd8c412

@MikkelDK thanks so much for these changes. I'll look to get these merged in and released on Monday. Have a great weekend! 👍

@SirRawlins You too buddy! ;)


Tag v20.2.0-beta.3 release notes for GH issue creation

This change is

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/789d1151a130ee33fcd8e77d247311f0499f01d7/
Edited pages:

releases/v20.2.0-beta.3.md


Coordinator segment balancer max load queue fix

DruidCoordinatorBalancer will now respect config setting maxSegmentsInNodeLoadingQueue, filtering servers which cross the threshold from potential move targets.
Related to #5882

This fix as it is currently, will remove servers from consideration as both a destination to move segments to, as well as a source to move segments from. If it is more preferable to still allow moving segments off of these servers, the server list could be duplicated and pick segments from the full set, but on consider moving to the filtered set.

Cool, I think it'd be good to still allow moves from servers that have a full load queue

Updated to let segments still be moved from servers with a full load queue


Added score calculator on front

can also add calculated value for level: Low\Medium\High\Critical


add KeybindCard component

This PR:

Adds the KeybindCard component
Adds the KeybindCard story
Adds the SolidKeyboard icon
Adds a tiny button variant to Button component
Adds transparent option to colorClassname in Button component

NOTE:  the logic for updating the keybinds is not yet implemented.

Just pushed some changes, resolve conflicts and we merge

Alright. resolved


Out-HTMLView does not work in IE or Chrome for me

Is there something I need to enable in IE or Chrome to get all the features to work for Out-HTMLView?  I just get a plain old HTML table in IE and chrome - none of the fancy buttons, filters, exports, etc.
Just trying something simple like get-service | out-htmlview.
Thanks
NK

Hi,
Internet Explorer may not work. It's a bit old. As for Chrome, you must have JS enabled. Probably it's disabled on your machine for local files. By default, it's enabled but if it's company laptop maybe it's forced. JavaScript is responsible for all those bells and whistles.

Thank you.


[BUG] Error while resetting index

Describe the bug
Getting this while resetting the index:
Ignoring exception in on_application_command_error
Traceback (most recent call last):
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/commands/core.py", line 124, in wrapped
    ret = await coro(arg)
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/commands/core.py", line 978, in _invoke
    await self.callback(self.cog, ctx, **kwargs)
  File "/home/xxx/xxx/cogs/commands.py", line 182, in delete_all_conversation_threads
    await self.converser_cog.delete_all_conversation_threads_command(ctx)
  File "/home/xxx/gpt2/cogs/text_service_cog.py", line 855, in delete_all_conversation_threads_command
    await ctx.respond("All conversation threads in this server have been deleted.")
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/commands/context.py", line 286, in respond
    return await self.followup.send(*args, **kwargs)  # self.send_followup
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/webhook/async_.py", line 1745, in send
    data = await adapter.execute_webhook(
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/webhook/async_.py", line 219, in request
    raise NotFound(response, data)
discord.errors.NotFound: 404 Not Found (error code: 10008): Unknown Message

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/client.py", line 378, in _run_event
    await coro(*args, **kwargs)
  File "/home/xxx/xxx/gpt3discord.py", line 120, in on_application_command_error
    raise error
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/bot.py", line 1114, in invoke_application_command
    await ctx.command.invoke(ctx)
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/commands/core.py", line 375, in invoke
    await injected(ctx)
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/commands/core.py", line 124, in wrapped
    ret = await coro(arg)
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/commands/core.py", line 1312, in _invoke
    await command.invoke(ctx)
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/commands/core.py", line 375, in invoke
    await injected(ctx)
  File "/home/xxx/.local/lib/python3.9/site-packages/discord/commands/core.py", line 132, in wrapped
    raise ApplicationCommandInvokeError(exc) from exc
discord.errors.ApplicationCommandInvokeError: Application Command raised an exception: NotFound: 404 Not Found (error code: 10008): Unknown Message
Traceback (most recent call last):
  File "/home/xxx/xxx/models/index_model.py", line 177, in reset_indexes
    EnvService.find_shared_file(f"indexes/{user_id}_search")
  File "/home/xxx/xxx/services/environment_service.py", line 73, in find_shared_file
    raise ValueError(f"Unable to find shared data file {file_name}")
ValueError: Unable to find shared data file indexes/417185533564026882_search

To Reproduce
Steps to reproduce the behavior:

Add a index
Try resetting

Expected behavior
Reset to be successful, but seems not
Screenshots
N/A
Additional context
N/A

Fixed


Implement hotfix for depletion of homozygous alternate genotypes in subset code

Hot fix code will also need to be added to subset scripts that include genotype information

Please paste the solution that Julia coded that will be in the following PR: https://github.com/broadinstitute/gnomad_qc/pull/112
This should be a relatively quick piece of work, but we need to do this separately for the subsetting code


Bad idea:

Windows: simply copying bin\64\sciter.dll to c:\windows\system32 is just enough

That is extremely bad idea to be honest.
sciter.dll shall be placed into folder where the exeuctable resides.
And so each application will be able to use sciter.dll it was designed and tested for.
Alternatively you shold find some other way placing and loading sciter.dll from folder where your PHP files of the project reside. sciter.dll is a part of your application but not the whole OS.

Thank you for your advice. We'll correct it.


new(modern_bpf): add support for select syscall

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area API-version

/area build

/area CI

/area driver-kmod

/area driver-bpf

/area driver-modern-bpf

/area libscap-engine-bpf

/area libscap-engine-gvisor

/area libscap-engine-kmod

/area libscap-engine-modern-bpf

/area libscap-engine-nodriver

/area libscap-engine-noop

/area libscap-engine-source-plugin

/area libscap-engine-savefile

/area libscap-engine-udig

/area libscap

/area libpman

/area libsinsp

/area tests

/area proposals

Does this PR require a change in the driver versions?

/version driver-API-version-major

/version driver-API-version-minor

/version driver-API-version-patch

/version driver-SCHEMA-version-major

/version driver-SCHEMA-version-minor

/version driver-SCHEMA-version-patch

What this PR does / why we need it:
Add support for select syscall in modern bpf probe.
Which issue(s) this PR fixes:

Part of #723
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: loresuso
Once this PR has been reviewed and has the lgtm label, please assign lucaguerra for approval. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, loresuso
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: f0b1abcdb3a6291d21b6c19faccc938400aae964

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, loresuso
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, hbrueckner, loresuso
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


[Bug] Avoid overriding a query with a subset of it

When using the normalized cache the updates are made through this piece of code which replaces bigger types with smaller types.
For istance, if I have a list of posts where I get their titles
searchPosts(search: String!){
  post {
    title
  }
}

this would replace a bigger type, let's say a detail page which has this query
postById(id: ID!) {
  post {
    title
    subtitle
  }
}

and the subtitle field would be deleted.
The correct way would be to addAll the new object to the old object (if the old one exists).

Closed because of #156

:tada: This issue has been resolved in version 1.0.0-beta.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


CloudView should support all operations

From the cloud view we can set properties or add children

We should be able to perform all changes. To do that we should be able to:

Set references
Delete nodes
Perhaps reorder nodes (optional)

A master detail view would be nice. You select a node on the left side and then you see all the properties and references on the right side and you can directly edit properties in a text field and set references using drag and drop.


Add default signing key parameter to serve

⚠ This is my first attempt to change a Go project ever. I really don't know much about the language or its conventions. I'm leaving my own comments in the PR with doubts and questions. So, please, review thoroughly and be merciless when suggesting changes. ⚠

This PRs adds the ability to define a default signing key to be used when "sign" requests are received by the service that the serve command exposes.
I took as a reference the implementation of the same functionality in the sign command.
Please see my own comments in the code.

@samlown @dstotijn @agorer I opened this PR as a draft and I'm not sure if that sends you any notification (that's why I'm mentioning you explicitly). This is the time I write a single line of Go code, so I'm looking forward to your feedback and suggestions to improve.

Just pushed some amendments based on @samlown's feedback (Thanks a lot!).
I'm marking this PR as "Ready for review". I'd appreciate a second 👀 at the code and an answer to this not-yet-resolved question.


Added LogProcessors and implemented shutdown and forceFlush for LoggerProviders

Context
LogProcessors are a sort of middleman between LogRecords and LogExporters that allow for modification of LogRecords after creation. This PR creates a LogProcessor type that contains onEmit, shutdown, and forceFlush methods. It also creates shutdown and forceFlush functions for LoggerProviders that call the shutdown and forceFlush methods of internal LogProcessors.
Testing

stack build runs
Test suite updated so that old tests still work with updated LoggerProvider type, but no new test written for LogProcessors. It would probable make more sense to write individual tests for each implemented LogProcessor

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#132  👈
#131
#129
#128
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @evanlauer1 and the rest of your teammates on  Graphite

Approved pending the LogProcessor -> LogRecordProcessor rename


Agent does not pull used memory from containers on Raspberry Pi 4b

Thanks for a great piece of software!  I will be switching to this from the heavy Prometheus/Grafana/InfluxDB stack soon.
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.6.28+rpt-rpi-v8
Architecture: arm64
Docker version 20.10.24+dfsg1, build 297e128
Data retrieved from agent:
  {
    "c": 0.19,
    "m": 0,
    "n": "prometheus"
  },
  {
    "c": 0.1,
    "m": 0,
    "n": "adguard_exporter"
  },
  {
    "c": 0.08,
    "m": 0,
    "n": "glances"
  },
  {
    "c": 0.71,
    "m": 0,
    "n": "dockge_dockge_1"
  },
  {
    "c": 0.04,
    "m": 0,
    "n": "grafana"
  },
  {
    "c": 0.01,
    "m": 0,
    "n": "beszel-agent"
  },
  {
    "c": 0.13,
    "m": 0,
    "n": "node_exporter"
  }
]

The containers are obviously running because of the CPU usage in the response but there is no memory being reported back in the response.
I haven't found anything in the logs that indicate an obvious error retrieving data.  Let me know if there's any further info I can provide.

This is probably due to the specific docker version and differences in the API.
If you could do the following it would be very helpful:

Run docker ps to list your running containers, then choose a container ID.
Make a request to the API for the container resource usage (replace d07d19594b6e with your container ID):

curl --unix-socket /var/run/docker.sock -H "Content-Type: application/json" "http://localhost/containers/d07d19594b6e/stats?stream=0&one-shot=1"

Post the whole response here, or just the "memory_stats" section.

Thanks!

Thanks for the reply... After running the curl you sent using a container ID from my instance, it looks like this isn't an issue with your code.  Every value in the memory_stats object is a 0.  I will do some research on why this isn't populated in this scenario (RPi 4b + docker 20.10)
Here's the memory_stats object for posterity:
  "memory_stats": {
    "stats": {
      "active_anon": 0,
      "active_file": 0,
      "anon": 0,
      "anon_thp": 0,
      "file": 0,
      "file_dirty": 0,
      "file_mapped": 0,
      "file_writeback": 0,
      "inactive_anon": 0,
      "inactive_file": 0,
      "kernel_stack": 0,
      "pgactivate": 0,
      "pgdeactivate": 0,
      "pgfault": 0,
      "pglazyfree": 0,
      "pglazyfreed": 0,
      "pgmajfault": 0,
      "pgrefill": 0,
      "pgscan": 0,
      "pgsteal": 0,
      "shmem": 0,
      "slab": 0,
      "slab_reclaimable": 0,
      "slab_unreclaimable": 0,
      "sock": 0,
      "thp_collapse_alloc": 0,
      "thp_fault_alloc": 0,
      "unevictable": 0,
      "workingset_activate": 0,
      "workingset_nodereclaim": 0,
      "workingset_refault": 0
    }
  }

FYI, the changes to /boot/firmware/cmdline.txt to enable the memory cgroup works.  Thanks for the quick reply and the awesome software!

Awesome, good to hear.


update links

resolve https://github.com/jazzband/django-dbbackup/issues/470

Looks like we can get tests passing by removing the 3.6 tests, which is fine since Django no longer supports this Python version.

Quite annoyingly, no new PRs can be merged without fixing the following:

https://github.com/jazzband/help/issues/196


[WIP] Add per_file to ocaml flags

This tries to address https://github.com/janestreet/jbuilder/issues/61
It compiles and the backward compatibility tests that I've tried seem fine.
It is preliminary, but creating the PR could allow me to speed things up and get immediate opinions on the best way to cleanup the current changes.
TODO:

fix defaults left behind
add tests
add documentation

For the question of what per_file should match on, there are several solution possible. We can say that it matches on filenames, but then to be consistent we have to decide what to do of (preprocess (per_file ...)) which currently matches on module names.
Or we could say that it always matches on module names. And invent a syntax if we need to distinguish implementations from interfaces, for instance: (impl:foo intf:bar).
I guess both have their pros and cons. For instance if we add support for globs and you have a reason project with one .mll file, then to apply a flag to all the implementations, in one case you'd write: *.{ml,re}, in the other you'd write: impl:*.

I don't know what is the best approach but I think that in this case making them consistent is better and for preprocess it makes more sense to use the module for matching

I suggest doing the following: we rename per_file to per_module (but keep per_file as an alias in (preprocess ...)). If latter we need more fine grained filtering, i.e. we need to distinguish between .ml and .mli then we'll add (per_file ...) with the expected semantic and we'll mint a new jbuild version.

I am closing this for now. I am likely to restart working on it in a couple of weeks and will reopen it once the necessary changes are ready for review.

I've now completely replaced per_file by per_module, so you are free to reintroduce per_file with a new semantic

Awesome! Thanks.
I am leaving for holidays today, I'll get on preparing the new PR in two weeks when I am back