update Relationship model

By reading the API coverage, I decided to update the Relationship model.
I would have wanted to update also the Account model but I did not understand what is missing...

This likely closes #19?

@G10xy Please try to link existing issues that you’re fixing.
Ideally, please also assign (or ask others to assign to you) issues you plan to work on.
Otherwise, there might be unnecessary duplicate work because multiple people might be working on the same issue without knowing of the others working on them.

This likely closes #19?

Yes, I think so

@G10xy Please try to link existing issues that you’re fixing.
Ideally, please also assign (or ask others to assign to you) issues you plan to work on.
Otherwise, there might be unnecessary duplicate work because multiple people might be working on the same issue without knowing of the others working on them.

You are totally right, sorry


How to train multi-gpu on tensorflow using nccl library?

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux Ubuntu 16.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
Local custom build PC

TensorFlow installed from (source or binary):
binary

TensorFlow version (use command below):
1.10

Python version:
3.5

Bazel version (if compiling from source):
No

GCC/Compiler version (if compiling from source):
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609

CUDA/cuDNN version:
9.0/7.2

GPU model and memory:
Two GTX 1080 Ti each with 11G of memory

Exact command to reproduce:
IDK

Describe the problem:
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
I was wondering if you could provide a tutorial on how to train a simple CNN on multiple gpus on MNIST, or Cifar dataset that also explains how to use the nccl library.
The only tutorials available are the following: https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py#L101
http://blog.s-schoener.com/2017-12-15-parallel-tensorflow-intro/
https://proteusmaster.urcf.drexel.edu/urcfwiki/index.php/Job_Script_Example_09_TensorFlow_MNIST_Multi-GPU-CNN
None of which explains how to use nccl.
I also search for an answer on many websites but the only one i found is this from openai in Chinese:
http://openresearch.ai/t/nccl-efficient-tensorflow-multigpu-training/159
Although 1/6 of the population of earth does speak Chinese, the other 5/6 does not! So i was wondering if someone in tensorflow team that knows English and perhaps Chinese could help with this issue. Thank you. I would also appreciate if you do me a favor and do not refer or pass me to stackoverflow. Thanks again.

@Toby Boyd I am wondering if there is a way to do this without using the estimator api. Since in estimator api there is no flexibility and one cannot use it for deep research projects.
Based on the complaint I have seen about tensorflow, i guess people in the community do not have any voice and tensorflow does what it does regardless of the needs of the community.

From your comment linked below did not mention you could not use Estimator.  I understand your frustration. We spend a lot of time listening to the community and sometimes we fail to get it correct.  A lot of people want easy multi-gpu and the best way to do that was via Estimator or Keras that can then handle all the little details.  For some top researchers, mostly in performance, they would use the tf_cnn_benchmarks approaches and build from them to add custom logic to some some really neat tricks.  The issue with that is the code is hard to follow as it is a sandbox to test new ideas.  The site you linked is not OpenAI (Elan Musk) and while I did not take time to read it closely I think the code snippets are from tf_cnn_benchmarks.  Reasons we wrapped it:
@anj-s
Can you provide any info on using distribution strategies without Estimator or Keras?
Best of luck finding what you need.

@tfboyd Thank you so much for your answer. I certainly can write my code with Keras. But based on what i have read on Tensorflows latest documentation in order to be able to use all_reduce I have to go back to Estimators, which is what i would prefer to avoid. I would greatly appreciate if you can help me find a way to tune the code in https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py so that I could use the all_reduce functionality as well even if it might add some complicated code.
The Chinese link i shared above has the following code snippets:
[def allreduce_grads(all_grads, average=True):
from tensorflow.contrib import nccl
nr_tower = len(all_grads)
if nr_tower == 1:
return all_grads
new_all_grads = []  # N x K
for grads in zip(*all_grads):
summed = nccl.all_sum(grads)
    grads_for_devices = []  # K
    for g in summed:
        with tf.device(g.device):
            # tensorflow/benchmarks didn't average gradients
            if average:
                g = tf.multiply(g, 1.0 / nr_tower, name='allreduce_avg')
        grads_for_devices.append(g)
    new_all_grads.append(grads_for_devices)

# transpose to K x N
ret = list(zip(*new_all_grads))
return ret

](url)

The link is not Chinese (it's Korean).
I wrote the snippet above originally in tensorpack: https://github.com/tensorpack/tensorpack/blob/81a4fc332eeae7e230e0736958014a0958fca822/tensorpack/graph_builder/utils.py#L140-L170.
You can probably use it for multigpu training very easily: just do tf.gradients on each GPU, then call this function to reduce them. It is used in this file.

The link I provided also showed Keras, it was at the top of the document I linked for you.  @ppwwyyxx  Apologies for guessing the snippets were from the benchmark code.  Big thanks for sharing as always.

@tfboyd You can try using DistributionStrategy APIs directly and your code snippet should look something like this:
  with my_distribution.scope():
    iterator = my_distribution.distribute_dataset(
        dataset).make_one_shot_iterator()
    tower_train_ops = my_distribution.call_for_each_tower(
        tower_fn, iterator.get_next())
    train_op = tf.group(my_distribution.unwrap(tower_train_ops))
  ```
If you use MirroredStrategy for distributing your model on multiple GPUs on a single machine DistributionStrategy handles aggregating gradients using allreduce.

From the above conversation it seems like @kazemSafari wants to implement the allreduce functionality. Hence I am not sure if using DistributionStrategy API directly will help.

@tfboyd Thank you so much again. I saw MirroredStrategy is indeed available in Keras on its own. I will let you know if I have more questions.
@anj-s Hi thank you for your explanation. I was wondering if I you could provide a simple example using MirroredStrategy say on a simple dataset like mnist with a simple one layer network that doesn't use estimator or Keras and just using tensorflow.

@ppwwyyxx Thank you for sharing the code. Sorry about my comment. I was trying to follow the code you shared on mnist with a simple mlp:
import tensorflow as tf
import numpy as np

def get_shape(tensor):
    return tensor.get_shape().as_list()

# Source:
# https://stackoverflow.com/questions/38559755/how-to-get-current-available-gpus-in-tensorflow
def get_available_gpus():
    """
        Returns a list of the identifiers of all visible GPUs.
    """
    from tensorflow.python.client import device_lib
    local_device_protos = device_lib.list_local_devices()
    return [x.name for x in local_device_protos if x.device_type == 'GPU']

PS_OPS = [
    'Variable', 'VariableV2', 'AutoReloadVariable', 'MutableHashTable',
    'MutableHashTableOfTensors', 'MutableDenseHashTable'
]

# see https://github.com/tensorflow/tensorflow/issues/9517
def assign_to_device(device, ps_device):
    """Returns a function to place variables on the ps_device.

    Args:
        device: Device for everything but variables
        ps_device: Device to put the variables on. Example values are /GPU:0 and /CPU:0.

    If ps_device is not set then the variables will be placed on the default device.
    The best device for shared varibles depends on the platform as well as the
    model. Start with CPU:0 and then test GPU:0 to see if there is an
    improvement.
    """
    def _assign(op):
        node_def = op if isinstance(op, tf.NodeDef) else op.node_def
        if node_def.op in PS_OPS:
            return ps_device
        else:
            return device
    return _assign

def device_options():
    config = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)
    with tf.Session(config=config) as sess:
        # your code here
        pass

# Source:
# https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py#L101
def average_gradients(tower_grads):
    """Calculate the average gradient for each shared variable across all towers.
    Note that this function provides a synchronization point across all towers.
    Args:
    tower_grads: List of lists of (gradient, variable) tuples. The outer list ranges
        over the devices. The inner list ranges over the different variables.
    Returns:
            List of pairs of (gradient, variable) where the gradient has been averaged
            across all towers.
    """
    average_grads = []
    for grad_and_vars in zip(*tower_grads):

        # Note that each grad_and_vars looks like the following:
        #   ((grad0_gpu0, var0_gpu0), ... , (grad0_gpuN, var0_gpuN))
        grads = [g for g, _ in grad_and_vars]
        grad = tf.reduce_mean(grads, 0)

        # Keep in mind that the Variables are redundant because they are shared
        # across towers. So .. we will just return the first tower's pointer to
        # the Variable.
        v = grad_and_vars[0][1]
        grad_and_var = (grad, v)
        average_grads.append(grad_and_var)
    return average_grads

# Source
# https://github.com/tensorpack/tensorpack/blob/81a4fc332eeae7e230e0736958014a0958fca822/tensorpack/graph_builder/utils.py#L140-L170
#################################
def split_grad_list(grad_list):
    """
    Args:
        grad_list: K x N x 2
    Returns:
        K x N: gradients
        K x N: variables
    """
    g = []
    v = []
    for tower in grad_list:
        g.append([x[0] for x in tower])
        v.append([x[1] for x in tower])
    print("g: ")
    print(g)
    print("v: ")
    print(v)
    return g, v

def merge_grad_list(all_grads, all_vars):
    """
    Args:
        all_grads (K x N): gradients
        all_vars(K x N): variables
    Return:
        K x N x 2: list of list of (grad, var) pairs
    """
    all_towers = [list(zip(gs, vs)) for gs, vs in zip(all_grads, all_vars)]
    print("all_towers: ")
    print(all_towers)
    return all_towers

def allreduce_grads(all_grads, average=True):
    """
    All-reduce average the gradients among K devices. Results are broadcasted to all devices.
    Args:
        all_grads (K x N): List of list of gradients. N is the number of variables.
        average (bool): average gradients or not.
    Returns:
        K x N: same as input, but each grad is replaced by the average over K devices.
    """
    from tensorflow.contrib import nccl
    nr_tower = len(all_grads)
    if nr_tower == 1:
        return all_grads
    new_all_grads = []  # N x K
    for grads in zip(*all_grads):
        print('grads:')
        print(grads)
        summed = nccl.all_sum(grads)

        grads_for_devices = []  # K
        for g in summed:
            with tf.device(g.device):
                # tensorflow/benchmarks didn't average gradients
                if average:
                    g = tf.multiply(g, 1.0 / nr_tower)
            grads_for_devices.append(g)
        new_all_grads.append(grads_for_devices)

    # transpose to K x N
    ret = list(zip(*new_all_grads))
    return ret
###########################

# Source
# https://github.com/tensorpack/tensorpack/blob/81a4fc332eeae7e230e0736958014a0958fca822/tensorpack/graph_builder/training.py#L240
def apply_gradients(opt, grads):
    devices = get_available_gpus()
    # num_gpus = len(devices)
    raw_devices = ['/gpu:{}'.format(k) for k in devices]
    # optimizer using NCCL
    train_ops = []
    with tf.name_scope('apply_gradients'):
        for idx, grad_and_vars in enumerate(grads):
            with tf.device(raw_devices[idx]):
                # apply_gradients may create variables. Make them LOCAL_VARIABLES
                # with override_to_local_variable(enable=idx > 0):
                train_ops.append(opt.apply_gradients(
                    grad_and_vars, name='apply_grad_{}'.format(idx)))
    train_op = tf.group(*train_ops, name='train_op')
    return train_op

def create_parallel_optimization(X, Y, input_flattened_dim, num_classes, optimizer, controller="/cpu:0"):
    devices = get_available_gpus()
    num_gpus = len(devices)
    # Place all ops on CPU by default
    # with tf.device('/cpu:0'):
    tower_grads = []
    losses = []
    # tf Graph input
    # X = tf.placeholder(tf.float32, [None, num_input])
    # Y = tf.placeholder(tf.float32, [None, num_classes])

    # Split data between GPUs
    X_s = tf.split(X, num_gpus)
    Y_s = tf.split(Y, num_gpus)
    # optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
    # Get the current variable scope so we can reuse all variables we need once we get
    # to the second iteration of the loop below
    with tf.variable_scope(tf.get_variable_scope()) as outer_scope:
        for i, id in enumerate(devices):
            name = 'tower_{}'.format(i)
            # Use the assign_to_device function to ensure that variables are created on the
            # controller.
            with tf.device(assign_to_device(id, controller)), tf.name_scope(name):

                _x = X_s[i]
                _y = Y_s[i]

                logits = mnist_mlp(_x, input_flattened_dim, num_classes)

                # Define loss and optimizer (with train logits, for dropout to take effect)
                loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(
                    logits=logits, labels=_y))

                with tf.name_scope("compute_gradients"):
                    # `compute_gradients` returns a list of (gradient, variable) pairs
                    grads = optimizer.compute_gradients(loss)
                    tower_grads.append(grads)

                losses.append(loss)
                outer_scope.reuse_variables()

    print(tower_grads[0])

    all_grads, all_vars = split_grad_list(tower_grads)
    all_grads = allreduce_grads(all_grads)
    tower_grads = merge_grad_list(all_grads, all_vars)

    train_op = apply_gradients(optimizer, tower_grads)
    # return confusion_mat, equality_op, accuracy_op, train_op
    return train_op

def mnist_mlp(x, input_flattened_dim, num_classes):

    W = tf.get_variable(name='W1',
                        shape=[input_flattened_dim, num_classes],
                        initializer=tf.keras.initializers.glorot_uniform(seed=None),
                        dtype=tf.float32)
    b = tf.get_variable(name='b1',
                        shape=[num_classes],
                        initializer=tf.zeros_initializer(),
                        dtype=tf.float32)
    # Output layer, class prediction
    logits = tf.nn.relu(tf.matmul(x, W) + b)

    return logits

mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

print(x_train.shape)

x_train = np.reshape(x_train, [x_train.shape[0], -1])
print(x_train.shape)
print(y_train.shape)

input_flattened_dim = 784
num_classes = 10
learning_rate = .0001
max_steps = 100
batch_size = 200
X = tf.placeholder(tf.float32, [None, input_flattened_dim])
Y = tf.placeholder(tf.float32, [None, num_classes])

train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train)).repeat(None).batch(batch_size)
print(train_dataset)
iterator = train_dataset.make_one_shot_iterator()
# extract an element
next_element = iterator.get_next()

optimizer = tf.train.AdamOptimizer(learning_rate)
train_op = create_parallel_optimization(X, Y, input_flattened_dim, num_classes, optimizer, controller="/cpu:0")

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(max_steps):
        batch_x, batch_y = sess.run(next_element)
        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})

But I get the following error:
(60000, 28, 28)
(60000, 784)
(60000,)
<BatchDataset shapes: ((?, 784), (?,)), types: (tf.float64, tf.uint8)>
2018-10-04 17:02:35.991165: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-10-04 17:02:36.080553: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-04 17:02:36.080933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.607
pciBusID: 0000:01:00.0
totalMemory: 10.91GiB freeMemory: 10.13GiB
2018-10-04 17:02:36.154460: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-04 17:02:36.154821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.607
pciBusID: 0000:02:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2018-10-04 17:02:36.155558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1
2018-10-04 17:02:36.480714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-04 17:02:36.480738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1
2018-10-04 17:02:36.480743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N Y
2018-10-04 17:02:36.480746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   Y N
2018-10-04 17:02:36.480964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:0 with 9790 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-10-04 17:02:36.555074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:1 with 10403 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
[(<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>, <tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>), (<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>)]
g:
[[<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>, <tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>], [<tf.Tensor 'tower_1/compute_gradients/gradients/tower_1/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>, <tf.Tensor 'tower_1/compute_gradients/gradients/tower_1/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>]]
v:
[[<tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>], [<tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>]]
grads:
(<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>, <tf.Tensor 'tower_1/compute_gradients/gradients/tower_1/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>)
grads:
(<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>, <tf.Tensor 'tower_1/compute_gradients/gradients/tower_1/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>)
all_towers:
[[(<tf.Tensor 'Mul:0' shape=(784, 10) dtype=float32>, <tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>), (<tf.Tensor 'Mul_2:0' shape=(10,) dtype=float32>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>)], [(<tf.Tensor 'Mul_1:0' shape=(784, 10) dtype=float32>, <tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>), (<tf.Tensor 'Mul_3:0' shape=(10,) dtype=float32>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>)]]
2018-10-04 17:02:38.038636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1
2018-10-04 17:02:38.038690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-04 17:02:38.038695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1
2018-10-04 17:02:38.038699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N Y
2018-10-04 17:02:38.038701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   Y N
2018-10-04 17:02:38.038823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:0 with 9790 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-10-04 17:02:38.038988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:1 with 10403 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
Traceback (most recent call last):
  File "/home/kazem/PycharmProjects/Prasad/multi_gpu_example.py", line 270, in <module>
    train_op = create_parallel_optimization(X, Y, input_flattened_dim, num_classes, optimizer, controller="/cpu:0")
  File "/home/kazem/PycharmProjects/Prasad/multi_gpu_example.py", line 220, in create_parallel_optimization
    train_op = apply_gradients(optimizer, tower_grads)
  File "/home/kazem/PycharmProjects/Prasad/multi_gpu_example.py", line 164, in apply_gradients
    with tf.device(raw_devices[idx]):
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/kazem/tf1.10_p3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 4085, in device
    device_function = pydev.merge_device(device_name_or_function)
  File "/home/kazem/tf1.10_p3/lib/python3.5/site-packages/tensorflow/python/framework/device.py", line 284, in merge_device
    spec = DeviceSpec.from_string(spec or "")
  File "/home/kazem/tf1.10_p3/lib/python3.5/site-packages/tensorflow/python/framework/device.py", line 230, in from_string
    return DeviceSpec().parse_from_string(spec)
  File "/home/kazem/tf1.10_p3/lib/python3.5/site-packages/tensorflow/python/framework/device.py", line 165, in parse_from_string
    self.device_index = int(y[1])
ValueError: invalid literal for int() with base 10: ''

Could you provide me with a simple example of using all_reduce. Thanks again.

@ppwwyyxx Thanks. It works in the simple example. but i couldn't figure out how to use batch normalization or print accuracy. It also seems to scale poorly in performance. So I guess i have to go the route of using Keras. Thanks again.

There are other things to do to scale in addition to using nccl. That's why there are tensorflow/benchmarks, distribution strategy (and tensorpack as well) to do the other necessary work to scale.
Going back to the original issue, I also hope to see some examples that use distribution strategy without Estimator or Keras.

@ppwwyyxx Could provide me with some information about the things and factors i need to know to help scale up?
or maybe refer me to a place where i can learn more about them and how to implement them as well. Since i was training an RGAN for a company i was interning at this summer and we found out that it is very challenging to scale up performance even though we used tfrecords.

@kazemSafari   heads up there is an API being reviewed to give access to multi-gpu API without other wrappers.  You may want to look at it to see how it may fit your needs/wants.
https://github.com/tensorflow/community/pull/25


Systemrole promt in my chain.TS

Issue you'd like to raise.
Hi ,
I'm trying to see how I can put a system message for my chain to tell him that for example "Your name is XX".
I've tried a lot of things and saw a lot of issues resolved and documentations but they never worked... Any help will be appreciated. here is the code of my chain.ts:
`
import {OpenAI} from "langchain/llms/openai";
import {pinecone} from "@/utils/pinecone-client";
import {PineconeStore} from "langchain/vectorstores/pinecone";
import {OpenAIEmbeddings} from "langchain/embeddings/openai";
import {ConversationalRetrievalQAChain} from "langchain/chains";
import { PromptTemplate } from "langchain/prompts";
import { ChatOpenAI } from "langchain/chat_models/openai";
async function initChain() {
const model = new ChatOpenAI({
modelName: "gpt-3.5-turbo",
temperature: 0,
});
const pineconeIndex = pinecone.Index('canada');
const vectorStore = await PineconeStore.fromExistingIndex(
new OpenAIEmbeddings({}),
{
pineconeIndex: pineconeIndex,
textKey: 'text',
},
);
return ConversationalRetrievalQAChain.fromLLM(
model,
vectorStore.asRetriever(),
{returnSourceDocuments: true}
);
return ConversationalRetrievalQAChain.fromLLM(
model,
vectorStore.asRetriever(),
{returnSourceDocuments: true},
);
}
export const chain = await initChain(

The system prompt work now ,  Thank you! but that gave me another issue. Now it is only answering with the system role content and It seems to not use the contents retrieved from my pinecone index..

🤖
Hello,
From your code, it seems like you're trying to add a system message to your chain. You can achieve this by using templates with qaChainOptions. Here's how you can modify your code:
// Create templates for qaChainOptions to use as instructions for the chain and chat model
const qa_template = `Use the following pieces of context to answer the question at the end.

{context}

Question: {question}
Helpful Answer:`

const sys_template = `I want you to act as a document that I am having a conversation with. Your name is XX. You will provide me with answers from the given info. If the answer is not included, search for an answer and return it, and make a note that you searched outside of the given info. Never break character.`

// Create a chain that uses the OpenAI LLM and Pinecone vector store.
const chain =  ConversationalRetrievalQAChain.fromLLM(
  model,
  vectorStore.asRetriever(),
  {
    returnSourceDocuments: true,
    // SOLUTION
    qaChainOptions: {type: "stuff", prompt : PromptTemplate.fromTemplate(`${sys_template}\n${qa_template}`) }
  },
  );

This solution was found in a similar issue: [help] How to add SystemMessage to ConversationalRetrievalQAChain?
I hope this helps! If you have any other questions, feel free to ask.
Best, Dosu
Sources

[help] How to add SystemMessage to ConversationalRetrievalQAChain?
examples/src/chains/llm_chain_chat.ts
langchain/src/chains/conversational_retrieval_chain.ts

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.

The system prompt work now , Thank you! but that gave me another issue. Now it is only answering with the system role content and It seems to not use the contents retrieved from my pinecone index..

🤖
Hello,
From your code, it seems like you're trying to add a system message to your chain. You can achieve this by using templates with qaChainOptions. Here's how you can modify your code:
// Create templates for qaChainOptions to use as instructions for the chain and chat model
const qa_template = `Use the following pieces of context to answer the question at the end.

{context}

Question: {question}
Helpful Answer:`

const sys_template = `I want you to act as a document that I am having a conversation with. Your name is XX. You will provide me with answers from the given info. If the answer is not included, search for an answer and return it, and make a note that you searched outside of the given info. Never break character.`

// Create a chain that uses the OpenAI LLM and Pinecone vector store.
const chain =  ConversationalRetrievalQAChain.fromLLM(
  model,
  vectorStore.asRetriever(),
  {
    returnSourceDocuments: true,
    // SOLUTION
    qaChainOptions: {type: "stuff", prompt : PromptTemplate.fromTemplate(`${sys_template}\n${qa_template}`) }
  },
  );

This solution was found in a similar issue: [help] How to add SystemMessage to ConversationalRetrievalQAChain?
I hope this helps! If you have any other questions, feel free to ask.
Best, Dosu
Sources

[help] How to add SystemMessage to ConversationalRetrievalQAChain?
examples/src/chains/llm_chain_chat.ts
langchain/src/chains/conversational_retrieval_chain.ts

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.

The system prompt work now , Thank you! but that gave me another issue. Now it is only answering with the system role content and It seems to not use the contents retrieved from my pinecone index..

🤖
Hello,
Based on the code you've shared and the context provided, it seems like the issue might be related to how the ConversationalRetrievalQAChain is configured.
In your code, you've initialized the ConversationalRetrievalQAChain with the BufferMemory option, which is set to return the chat history (memoryKey: "chat_history"). However, it doesn't seem to be configured to use the contents retrieved from the Pinecone index.
To use the contents retrieved from your Pinecone index, you need to ensure that the outputKey in the BufferMemory options is set to the key where the retrieved contents are stored. If the retrieved contents are stored under a different key, you should update the outputKey accordingly.
Here's an example of how you might configure the BufferMemory options:
const chain =  ConversationalRetrievalQAChain.fromLLM(
  model,
  vectorStore.asRetriever(),
  {
    returnSourceDocuments: true,
    qaChainOptions: {
      type: "stuff",
      prompt : PromptTemplate.fromTemplate(`${sys_template}\n${qa_template}`),
      memoryOptions: {
        memoryKey: "chat_history",
        outputKey: "pinecone_contents" // replace with the key where the retrieved contents are stored
      }
    }
  },
);

As for the qaChainOptions, it is used to initialize the QA chain, which is responsible for question-answering tasks. The qaChainOptions is an object of type QAChainParams and is used to set the type and prompt of the QA chain. The prompt can be set directly or by using a QA template.
If this doesn't solve your issue, could you please provide more details about how you're adding a system message to your chain and how you're using the contents retrieved from your Pinecone index? This will help me understand your issue better and provide a more accurate solution.
Best, Dosu
Sources

Ignored vector store? (spoiler: I think it actually was the JSONLoader)
examples/src/chains/conversational_qa_built_in_memory.ts
langchain/src/chains/conversational_retrieval_chain.ts

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.

@dosu-bot, here is my code:
I just want to give a system role to my bot , it worked but when I'm continuing my conversation with him , he forget the context from the embeddings retrieved from the vector database.
Here is my code:
`
import {OpenAI} from "langchain/llms/openai";
import {pinecone} from "@/utils/pinecone-client";
import {PineconeStore} from "langchain/vectorstores/pinecone";
import {OpenAIEmbeddings} from "langchain/embeddings/openai";
import {ConversationalRetrievalQAChain} from "langchain/chains";
import { ChatOpenAI } from "langchain/chat_models/openai";
import { PromptTemplate } from "langchain/prompts";
async function initChain() {
const model = new ChatOpenAI({
modelName: "gpt-3.5-turbo",
temperature: 0,
});
const pineconeIndex = pinecone.Index('canada');
const sys_template = ``
const vectorStore = await PineconeStore.fromExistingIndex(
    new OpenAIEmbeddings({}),
    {
        pineconeIndex: pineconeIndex,
        textKey: 'text',

    },
);
return ConversationalRetrievalQAChain.fromLLM(
    model,
    vectorStore.asRetriever(),
    {returnSourceDocuments: true,
    qaChainOptions: {type: "stuff", prompt : PromptTemplate.fromTemplate(`${sys_template}`) }

    },
);

}
export const chain = await initChain()`

@dosu-bot
Hi, The system role works now , but it seems to not take in consideration the vectorstore as a retriever anymore. When I ask him question , he take in consideration the QAtemplate but not the vectorestore , how can we make them work together please

@dosu-bot:
How to do this in my typescript code:
import {pinecone} from "@/utils/pinecone-client";
import {PineconeStore} from "langchain/vectorstores/pinecone";
import {OpenAIEmbeddings} from "langchain/embeddings/openai";
import {ConversationalRetrievalQAChain} from "langchain/chains";
import { ChatOpenAI } from "langchain/chat_models/openai";
import { PromptTemplate } from "langchain/prompts";
async function initChain() {
const model = new ChatOpenAI({
modelName: "gpt-3.5-turbo",
temperature: 0,
});
const pineconeIndex = pinecone.Index('canada');
const vectorStore = await PineconeStore.fromExistingIndex(
    new OpenAIEmbeddings({}),
    {
        pineconeIndex: pineconeIndex,
        textKey: 'text',

    },
);

console.log(vectorStore)
const qa_template = `Use the following pieces of context to answer the question at the end.

Content: ${vectorStore}
Question: {question}
Helpful Answer:`
return ConversationalRetrievalQAChain.fromLLM(
    model,
    vectorStore.asRetriever(),
    {returnSourceDocuments: true,
    qaChainOptions: {type: "stuff", prompt : PromptTemplate.fromTemplate(`${qa_template}`) }

    },
);

}
export const chain = await initChain()


Vim9: cannot use :const in :def function

Describe the bug
In Vim9 script, we cannot use :const in a :def function.
To Reproduce
Run this shell command:
vim -Nu NONE -S <(cat <<'EOF'
    vim9script
    def Func()
        const g:NAME = 123
    enddef
    defcompile
EOF
)

E1016 is raised:
E1016: Cannot declare a global variable: g:NAME

Expected behavior
No error is raised.
Environment

Vim version: 8.2 Included patches: 1-2673
OS: Ubuntu 16.04.7 LTS
Terminal: xterm(366)

Additional context
:const works at the script level (since 8.2.2194):
vim9script
const g:NAME = 123

no error

It also works in the legacy context:
vim9script
fu Func()
    const g:NAME = 123
endfu
Func()

no error

Now that we can use :lockvar in compiled code (since 8.2.2672), I don't see any reason to keep disallowing :const (is there one?):
vim9script
def Func()
    g:NAME = 123
    lockvar g:NAME
enddef
defcompile

no error

vim9script
def Func()
    g:NAME = 123
    lockvar g:NAME
enddef
Func()
g:NAME = 456

E741: Value is locked: g:NAME

These 2 snippets show that we can assign a value to a global variable and lock it in compiled code, using :lockvar.  But it makes the code more verbose because we need 2 lines instead of 1:
# verbose
g:NAME = 123
lockvar g:NAME

# less verbose
const g:NAME = 123

Allowing :const in compiled code would not let us do something which we can't already perform.  It would just make it easier.

Also, if it is disallowed to lock a local variable:
vim9script
def Func()
    var NAME = 123
    lockvar NAME
enddef
defcompile

E1178: Cannot lock or unlock a local variable

Then, to be consistent, I think it should also be disallowed to use :const for a local variable:
vim9script
def Func()
    const NAME = 123
enddef
defcompile

no error

Then, to be consistent, I think it should also be disallowed to use :const for a local variable:

Ah no, it's working as intended:

:lockvar does not work on local variables.  Use :const and :final
instead.

https://github.com/vim/vim/blob/38a3bfa9a2931729a5e0c28dc087f745b68988ef/runtime/doc/vim9.txt#L301-L302


Add more usage examples

It is beneficial to have these use cases available in the examples folder as separate runnable files:

[ ] All supported data types in a single structure (could make the enums.rs example obsolete?)

All "basic" types, including:

Chrono DateTime (https://github.com/ClickHouse/clickhouse.rs/issues/1)
(U)Int256 with its workarounds (see https://github.com/ClickHouse/clickhouse.rs/issues/48)
Highest precision decimals

Advanced types - maps/arrays/tuples/arrays of all this, nested, etc. This will close:

https://github.com/ClickHouse/clickhouse.rs/issues/101
https://github.com/ClickHouse/clickhouse.rs/issues/114
https://github.com/ClickHouse/clickhouse.rs/issues/72 (not sure)

[ ] ClickHouse Cloud connection + DDL example (wait_end_of_query, select_sequential_consistency settings highlight)
[x] General ClickHouse settings usage via .with_option
[x] async_insert - with and without wait_for_async_insert, using just .with_option; if it works (it should), this closes https://github.com/ClickHouse/clickhouse.rs/issues/65 as no additional code is required
[ ] query_id, cancel and kill queries - similarly to https://github.com/ClickHouse/clickhouse-js/blob/main/examples/cancel_query.ts, outline the diff between TCP and HTTP behaviors
[ ] session_id, maybe with views; outline potential LB issues and lack of sticky sessions
[ ] Request timeout (using tokio)
[ ] Parameters binding
[ ] Escaped SQL example (https://github.com/ClickHouse/clickhouse.rs/issues/96)
[ ] Custom HTTP client (https://github.com/ClickHouse/clickhouse.rs/issues/47)
[ ] Advanced query usage: https://github.com/ClickHouse/clickhouse.rs/issues/53

Is there no example of how to use an HTTPS clienthouse URL it seems not and hard to find examples of this

@joshstevens19, you could use this example: https://github.com/ClickHouse/clickhouse-rs/blob/master/examples/clickhouse_cloud.rs, as CH Cloud is HTTPS.


Why is the default of removeOptionalTags set to true?

According to http://perfectionkills.com/experimenting-with-html-minifier/#remove_optional_tags this feature might change the behaviour of the result.
This also caused https://github.com/ampedandwired/html-webpack-plugin/issues/128
Why do you remove HTML, BODY and HEAD?

Thanks @chrisnicola that looks promissing :+1:

@hemanth - Also resolved with #43 Close me


UI Framework

Well done for your work, I really appreciate your effort towards DX.
Is it possible to add a CSS/UI framework of choice?
PrimeVue, ant-design-vue, etc!

Yes most definitely.
The neat thing here is that you control how your pages are rendered. This means you can use whatever tool you want.
Closing but let me know if you have any further doubt/question.

Thanks for your response!
However, I am trying to have more control on routes. So, I am using the router example, but I am getting the  error below:
TypeError: Cannot use 'in' operator to search for 'path' in undefined
at resolve (/home/folamy/Desktop/vue-router/node_modules/vue-router/dist/vue-router.cjs.js:2828:20)
at pushWithRedirect (/home/folamy/Desktop/vue-router/node_modules/vue-router/dist/vue-router.cjs.js:2931:51)
at Object.push (/home/folamy/Desktop/vue-router/node_modules/vue-router/dist/vue-router.cjs.js:2901:16)
at Object.render (/pages/all.page.server.js:11:10)
at renderHtmlDocument (/home/folamy/Desktop/vue-router/node_modules/vite-plugin-ssr/dist/renderPage.node.js:201:47)
at renderPageId (/home/folamy/Desktop/vue-router/node_modules/vite-plugin-ssr/dist/renderPage.node.js:116:36)
at async renderPage (/home/folamy/Desktop/vue-router/node_modules/vite-plugin-ssr/dist/renderPage.node.js:96:24)
at async /home/folamy/Desktop/vue-router/server/index.js:28:20
<EMAIL_ADDRESS>No _error.page.js found. We recommend creating a _error.page.js file. (This warning is not shown in production.)
How do I fix this?

FYI, https://github.com/brillout/vite-plugin-ssr#other-tools goes in depth. You'll have confidence you can use whatever tool you want.


Polars: add len, count, null_count, n_unique

Close #1740
Close #1719

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#2095  👈
#2091
#2089
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Shoeboxam and the rest of your teammates on  Graphite

I pulled variations of counting queries out into a separate notebook.

I added full context examples to all the methods now! It does read better.


Tracking: 7-2 Barrel Room and 8-2 Lava Room coins don't track

Since last apworld update (0.20) the coins ID have changed and are not able to be tracked anymore.
ID needs to be changed in the tracker too

Corrected in #17


Fixed previous lit review index; added litreview2

This fixes the issue mentioned in my last pull request.  Thanks!

:book: :recycle: :two: :three:


Update environment to install latest lightning when UPGRADE_PL=true

[sc-62209]

No worries on the changes proposed ... should we build the demos within this PR to ensure everything builds or is that okay to skip?

No worries on the changes proposed ... should we build the demos within this PR to ensure everything builds or is that okay to skip?

Seems like this update has revealed more issues with the latest lightning.


Natti offline support

gitlab git provider support:

gitlab "api" style and "natural" browser copy/paste style urls supported

Offline TF module:

option to copy local TF exe into working dir

Shell attr change:

DEPRECATED github specific attrs in favor of generic attrs
updated generic TF service shell to use new attrs.

Unit tests and live end to end integration test done with gitlab repo and local tf exe.

this can be closed - added PR from qualilab branch


Fix race during SoftLink initialization

Closes #8773. This PR fixes the thread race during the initialization.

/test mini


Feature - add openmw cfg overwrite

Support portable installs, add a -c arg

[ ] CLI
[ ] GUI

https://discord.com/channels/260439894298460160/262662689489158145/1216800422304550932
Something closer to a spec than the modding-openmw.com page:

Any option that's valid as thing=value in openmw.cfg is valid as --thing value on the command line.
The command line is always the highest-priority config source.
The local openmw.cfg (i.e. the one in the same directory as the OpenMW binary) is always loaded, if it exists. If it doesn't, the global openmw.cfg (i.e. one at a platform-specific location for things like this, e.g. /etc/openmw on Linux) is loaded instead. If neither are present, your installation's busted.
A config source can specify additional config sources with the config option. These will be loaded as a higher-priority (but don't beat the command line, as that's always the highest priority).
A config source can specify that values for a setting from all lower-priority config sources should be ignored with the replace option (e.g. specifying --replace config on the command line will make OpenMW not use any config sources specified anywhere other than the command line, which is how the launcher scripts in the modding-openmw.com guide work).
By default, the local/global openmw.cfg specify config=?userconfig?, which evaluates to a platform-specific directory, e.g. Documents\My Games\OpenMW on Windows, or ~/.local/share/openmw on Linux. One way people make portable installs is to just get rid of this line, and then the binary directory's the only config source, so everything goes there instead.


Bare metal example with cross compile

I'd like to build for a bare-metal crate for a different architecture. How do I pick the arch without having nix flip out about choosing which gcc to use?

How are you building?

        pkgs = import nixpkgs {
          inherit system;
          crossSystem = {
              config = "x86_64-unknown-linux-musl";
          };
          overlays = [
            (import "${cargo2nix}/overlay")
            rust-overlay.overlay
          ];
        };

Works in release-0.9.1 branch.  You can manually configure system & crossSystem instead of using flake-utils forEachSystem and get more control.  The cc and rustc will all be from that nixpkgs, so it's all ready to go up to the extent that the hostPlatform you're after is currently building successfully.

See #214 to get some ideas.
I would feel more comfortable if something broke.  It's a black box to me until I spend more time on all the host offsets.

ick the arch without having nix flip out about choosing which gcc to use?
As long as you don't need stdenv on your host, which only happens if you try to "cross" compile on the host for the host (there is no such meaningful offset), nix will not need a gcc and all that on the host.

You will need a stdenv on your build platfrom to build native stuff for depsBuildBuild and  one that can target the host if you need any gcc outputs linked into your binary outputs for your host.
If all of the dependencies on the host are rlibs, that means you don't need the depsBuildHost gcc, the one that would target your host.

#214 is just about complete.  If you can provide a bare metal example, I can probably get it to build with an arbitrary target.  Something I need some examples for is how the Rust toolchain might be selected from the overlay with one target but then configured to output host binaries for weird vendor targets.

How do I pick the arch without having nix flip out about choosing which gcc to use?

Don't depend on gcc for that host -> target.  This is really about categorizing all dependencies correctly.
We may need support for picking a different linker, one that is available on the build platform and can emit code for the host.  This is really just #232 though.
Using a rust toolchain with std available shouldn't cause issues?  I wouldn't expect it, but correct me if I'm wrong.
Closing for #232 since linker customization should cover most cases.


Destroyer Update strings.xml

100℅ was added to the Spanish language without errors.

Please update (modify) original strings.xml.
And use a pull request with #539 .


merge from template-node-app

.github/pr_labeler.yml

chore: (systemd)

.github/workflows/pr_labeler.yml

chore: (systemd)

:tada: This PR is included in version 5.0.79 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release
v5.0.79

Your semantic-release bot :package::rocket:


feat: Add support for WindiCSS

Support WindiCSS color format in tailwind.config.js.  Non-breaking change.

Moves the colors.js file into /colors/index.
Moves the colorValues variable into its own file.
Creates a new colors/windi.js file, which allows user to require(‘daisyui/colors/windi’)

More about this PR
Hi, @saadeghi!  I really appreciate your work with DaisyUI.  Actually, I appreciate it so much that I spent the day figuring out how to get it to work with WindiCSS (https://windicss.org/).  WindiCSS is the TailwindCSS compiler that inspired creation of the new TailwindJIT compiler that was announced in the last week or so.  The competition has definitely been supportive for the Tailwind community, and now we have options.
Tailwind and Windi support the same classes and utilities, but their plugin APIs are different.

The existing daisyui/colors object is not supported by Windi.  This PR adds support for windi at daisyui/colors/windi.
Windi already has a plugin that converts the daisyui plugin to work.

Earlier today, I submitted a PR to WindiCSS to add support for DaisyUI.
https://github.com/windicss/windicss/pull/198
Once it’s merged, I would absolutely love it if this PR could also be merged.  (I will submit a PR for the docs, too.)

Thanks @marshallswain for contribution.
I haven't worked with WindiCSS yet but I guess I should give it a try soon 😅
I will merge this when that PR is merged 👍

Awesome.  Thanks @saadeghi.
Oh.  Looks like it has been merged.  That was quick!


require isn't working when running from a zip file

Hi, congrats on your new release! Sadly I'm getting another error. lua's require isn't working when I try to run a game that's zipped. I get this error when I try to require "game.lua":
wificlient-10-201-59-198:build Nico$ ./clove curse.zip main.lua
Debug: Sdl version:  2.0.3
Debug: OpenGL version: 2.1 INTEL-10.14.73
Debug: GLSL version 1.20
Debug: Vendor: Intel Inc.
Debug: Renderer: Intel HD Graphics 5000 OpenGL Engine
2016-09-21 12:31:11.114 clove[5936:72834] 12:31:11.114 WARNING:  140: This application, or a library it uses, is using the deprecated Carbon Component Manager for hosting Audio Units. Support for this will be removed in a future release. Also, this makes the host incompatible with version 3 audio units. Please transition to the API's in AudioComponent.h.
Debug: OpenAL version:  1.1 ALSOFT 1.15.1
 Error loading from argv: [string "line"]:31: module 'game' not found:
	no field package.preload['game']
	no file 'game.lua'
	no file 'game/init.lua'
	no file './game.so'
	no file '/usr/local/lib/lua/5.1/game.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
Love version name:  TouchingTheLimits 0.5.2

PS: I also tried your release binary, but It errors too:
wificlient-10-201-59-198:~ Nico$ /Users/Nico/Desktop/clove
dyld: Library not loaded: /Users/vlad/Programming/build-CLove-Desktop-Debug/src/3rdparty/physfs/libphysfs.1.dylib
  Referenced from: /Users/Nico/Desktop/clove
  Reason: image not found
Trace/BPT trap: 5

Hi, thanks! Can you please probide the source code or the .zip file?
In the mean time I'll do some tests myself.

@npinochet, This bug and many many new features are about to come. After 3 days of hard work I can say that the bug is fixed but it comes with a little price which I'm going to explain once the new big commit comes out.

Everything was fixed in the last commit. Now I'm going to write the wiki for everyone to understand how to use the feature and what has been changed because of it


Prepare for package manager support with carthage shared scheme

Preparation for carthage support, once we get #13 merged  (remove all external dependcies) we can integrate our first package management solution

Closing, as I merged in aggregate share earlier


Duplicate Indexes

pghero says "These indexes exist, but aren’t needed. Remove them for faster writes."
On subgraph_deployment
attr_2_0_subgraph_deployment_id (id)

is covered by
subgraph_deployment_id_key (id)

Also it says that 1258 unused indices. Remove them for faster writes.

You are right, that duplicate index on subgraph_deployment(id) should be dropped.
The 1258 unused indexes are a little more tricky: they will be unused as long as there is no query traffic. And quite a few of them will stay unused. But the ones that do get used are crucial for any sort of performance under decent load - if we got rid of all those indexes, indexers would quickly run into performance issues under load, and would have to figure out how to get rid of them. Neither situation is ideal, but for now overindexing seems preferable to indexer performance falling apart with real traffic.

Could you please provide a query that we should run to get rid of it correctly?
Will it be fixed on the code level so new setups won't create it again?

Could you please provide a query that we should run to get rid of it correctly?
Will it be fixed on the code level so new setups won't create it again?

@akme that isn't how database indexes work, from my understanding, but @lutter can correct me if I'm wrong here.
Indexes are created in order to boost the read/query performance.
More indexes - faster query time / slower data writes
They're valuing data serving over data gathering, basically, because the current logic of the graph-node is to overindex (hence why you have a few thousand indexes that were never used - as noticed by pg_stats).
If you want, you can manually create indexes from your topmost slow queries, if they are missing... or needed 🙂

@trader-payne yes, I understand that, so I created an issue only about duplicates, that 100% redundant.

Could you please provide a query that we should run to get rid of it correctly?
Will it be fixed on the code level so new setups won't create it again?

The way to get rid of the duplicate index is with drop index subgraphs.attr_2_0_subgraph_deployment_id - I will make sure that the migration that drops it that we will ship one day will work regardless of whether that index is there or not.
As for indexes in general: as @trader-payne indicated, there's a tradeoff in terms of the usability of graph-node between  indexing everything and requiring every indexer to be very good at analyzing and fixing performance issues caused by missing indexes. For now, we are opting on the side of somewhat slower writes and requiring less in-depth knowledge from indexers. But there's definitely room for better tooling to make using fewer indexes feasible without a black belt in Postgres administration.

@lutter I think this issue is long solved on the one hand (duplicates) and being addressed on the other hand (unused indexes).  Therefore we might want to close this.


docs: svelte add {#each} key to examples

Svelte needs a key in an iteration to ensure the items are animated properly.

Thanks!


emacs always disappear(I think crash) in windows

Hi purcell,
Your emacs config is great, I use it from the beginning of this year. But I found my emacs always disappear when I 'm using.
emacs version is 24.3, it's installed by cygwin, launched by  XWin Server(also installed from cygwin).
When it's disappear, I can't find any sign or clue.
Could you get me some suggestion ?
e.. by the way, one more question, when emacs boot up, it always  connect melpa.org:443, it will take a long time. Can I disable it ?

Hi purcell,
Your emacs config is great, I use it from the beginning of this year. But I found my emacs always disappear when I 'm using.
emacs version is 24.3, it's installed by cygwin, launched by XWin Server(also installed from cygwin).
When it's disappear, I can't find any sign or clue.
Could you get me some suggestion ?

It sounds like some kind of platform-specific issue, and I don't have access to a Windows machine to test on. Perhaps try starting Emacs from a terminal window so that you can see if a message is printed when Emacs crashes. Alternatively, see if the XWin Server keeps a log. Or perhaps try a different Emacs version.

e.. by the way, one more question, when emacs boot up, it always connect melpa.org:443, it will take a long time. Can I disable it ?

This is usually because the config is trying to install required packages. Perhaps try adding (setq debug-on-quit t) to the top of init.el, then press C-g while Emacs is waiting to connect. You should then see a backtrace which will show which part of the config is causing a problem. Paste the backtrace here and I'll take a look. :-)

It sounds like some kind of platform-specific issue, and I don't have access to a Windows machine to test on. Perhaps try starting Emacs from a terminal window so that you can see if a message is printed when Emacs crashes. Alternatively, see if the XWin Server keeps a log. Or perhaps try a different Emacs version.
Thank you for your advice, I will try by using Emacs on terminal.

It sounds like some kind of platform-specific issue, and I don't have access to a Windows machine to test on. Perhaps try starting Emacs from a terminal window so that you can see if a message is printed when Emacs crashes. Alternatively, see if the XWin Server keeps a log. Or perhaps try a different Emacs version.
Thank you for your advice, I will try by using Emacs on terminal.

Any particular reason why you're not using vanilla emacs for w32? I've never had much luck running cygwin-emacs, but the GNU-provided emacs 24.5 win32 builds just worked out of the box for me.

I have updated to emacs 24.5, and will report whether hanging.
but I alse need wait long time when I boot up emacs,
the Message column show:
gnutls.el: (err=[-54] Error in the pull function.) boot: (:priority NORMAL :hostname melpa.org :loglevel 0 :min-prime-bits 256 :trustfiles (/etc/pki/tls/certs/ca-bundle.crt /usr/ssl/certs/ca-bundle.crt) :crlfiles nil :keylist nil :verify-flags nil :verify-error nil :callbacks nil)
Loading /home/Administrator/.emacs.d/.session...done
Wrote /home/Administrator/.emacs.d/.emacs.desktop.lock
Desktop: 2 frames, 0 buffers restored.
Desktop restored in 511.03ms
Loading a theme can run Lisp code.  Really load? (y or n) n [2 times]
For information about GNU Emacs and the GNU system, type C-h C-a.
the Warnings show:
Warning (initialization): An error occurred while loading `/home/Administrator/.emacs.d/init.el':
GnuTLS error: melpa.org, -54

Yep, not sure about that, sorry. Seems like that's a problem with Emacs on Windows, and I can't really investigate that myself.

It won't hang again after I updated to Emacs 24.5 in windows.
many thanks to your support purcell.

Yay, great!


Middleware documentation is not clear

It is not clear from the documentation that middleware that is provided when creating a server is executed before route-specific middleware.

Updated with PR #18


Replay issue reading a RW texture

Description
I have a capture where the reading of a RW texture whose format is R32_UINT incorrectly returns 0.
Steps to reproduce
Load the capture, inspect a certain thread for a certain compute shader, observe the load fail, even after inspecting the texture and making sure it's not 0. I can send you the capture for inspection and details on an email.
Update: If I treat them as read-only then I can make it work, but subsequently another RW R32G32_UINT texture I'm using manifests the same behavior. It doesn't return 0, but seems to be returning an entry that is incorrect for the integer coordinate I'm passing in.
Environment

RenderDoc version: 1.6 2019-11-26 (latest nightly as of bug report date)
Operating System: Windows 10
API: D3D11

I have tried the latest nightly because it was failing in a previous nightly.

Just to let you know your email with the repro hasn't come through. I checked my junk folder just in case (though I rarely get false positives there) and it's not there either.

Sorry, that was my bad, I didn't actually send it. It should be in your inbox now.

There was a swizzle not being applied, which meant the data ended up in the wrong component and didn't get through. That commit should fix the problem.

That's great, thanks a lot for the quick turnaround


chore(deps): update to Next 13.1.0

Summary

Test plan

Visit the Deploy Preview (insert link to specific page) ...

Relevant links (GitHub issues, Notion docs, etc.) or a picture of cute animal
Standard checks:

[ ] Check the Deploy Preview's Demo site for your PR's functionality
[ ] Add docs when necessary

🧪 Once merged, make sure to update the version if needed and that it was published correctly.

Updating as part of https://github.com/netlify/next-runtime/pull/1907


Custom Script Extension Error Code 14 - failed to start etcd - highly-available key value store

Is this a request for help?
Yes
Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE
What version of acs-engine?:
acs-engine version
Version: v0.24.3
GitCommit: ad007b644
GitTreeState: clean
Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.12.1
What happened:
Master VMSS scale fails with the custom script extension throwing an error 14.
{
  "status": "Failed",
  "error": {
    "code": "ResourceDeploymentFailure",
    "message": "The resource operation completed with terminal provisioning state 'Failed'.",
    "details": [
      {
        "code": "VMExtensionProvisioningError",
        "message": "VM has reported a failure when processing extension 'k8s-master-22668494-vmssCSE'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=14\n[stdout]\n\n[stderr]\n\"."
      }
    ]
  }
}

What you expected to happen:
Success setup of cluster
How to reproduce it (as minimally and precisely as possible):
I was trying to setup cluster across Availability Zones in Central US region.
The cluster is attached to an existing VNet and I had two agent types - one Windows and one Linux.
Here is my acs-engine config file
{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.12.1"
    },
    "masterProfile": {
      "count": 5,
      "dnsPrefix": "",
      "vmSize": "Standard_DS2_v2",
      "OSDiskSizeGB": 64,
      "storageProfile" : "ManagedDisks",
      "vnetSubnetId": "/subscriptions/[redacted]/resourceGroups/DevSub02_Network_RG/providers/Microsoft.Network/virtualNetworks/DevSub02-VNet-001/subnets/Kubernetes-Master",
      "agentVnetSubnetID": "/subscriptions/[redacted]/resourceGroups/DevSub02_Network_RG/providers/Microsoft.Network/virtualNetworks/DevSub02-VNet-001/subnets/Kubernetes",
      "firstConsecutiveStaticIP": "<IP_ADDRESS>",
      "availabilityProfile": "VirtualMachineScaleSets",
      "availabilityZones": [
        "1",
        "2"
      ]
    },
    "agentPoolProfiles": [
      {
        "name": "linuxagents",
        "count": 4,
        "vmSize": "Standard_DS2_v2",
        "OSDiskSizeGB": 64,
        "storageProfile" : "ManagedDisks",
        "vnetSubnetId": "/subscriptions/[redacted]/resourceGroups/DevSub02_Network_RG/providers/Microsoft.Network/virtualNetworks/DevSub02-VNet-001/subnets/Kubernetes",
        "availabilityProfile": "VirtualMachineScaleSets",
        "availabilityZones": [
          "1",
          "2"
        ]
      },
      {
        "name": "winagents",
        "count": 4,
        "vmSize": "Standard_DS2_v2",
        "OSDiskSizeGB": 64,
        "storageProfile" : "ManagedDisks",
        "vnetSubnetId": "/subscriptions/[redacted]/resourceGroups/DevSub02_Network_RG/providers/Microsoft.Network/virtualNetworks/DevSub02-VNet-001/subnets/Kubernetes",
        "availabilityProfile": "VirtualMachineScaleSets",
        "osType": "Windows",
        "availabilityZones": [
          "1",
          "2"
        ]
      }
    ],
    "windowsProfile": {
      "adminUsername": "manager",
      "adminPassword": "[redacted]",
      "windowsPublisher": "MicrosoftWindowsServer",
      "windowsOffer": "WindowsServerSemiAnnual",
      "windowsSku": "Datacenter-Core-1803-with-Containers-smalldisk",
      "imageVersion": "1803.0.20180912"
    },
    "linuxProfile": {
      "adminUsername": "manager",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "[redacted]"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "[redacted]",
      "secret": "[redacted]"
    }
  }
}

Anything else we need to know:
No NSG rules applied to the VNet/Subnet
Partial output from cluster-provision.log (full log attached)

'[' 1 -eq 0 ']'+ '[' 99 -eq 100 ']'+ sleep 5+ for i in '$(seq 1 $retries)'+ timeout 30 systemctl daemon-reload+ timeout 30 systemctl restart etcdJob for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details.+ '[' 1 -eq 0 ']'+ '[' 100 -eq 100 ']'+ return 1+ RESTART_STATUS=1+ systemctl status etcd --no-pager -l+ '[' 1 -ne 0 ']'+ echo 'etcd could not be started'etcd could

Output of etcd.log
● etcd.service - etcd - highly-available key value store
Loaded: loaded (/etc/systemd/system/etcd.service; disabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Fri 2018-11-02 02:29:07 UTC; 6ms ago
Docs: https://github.com/coreos/etcd
man:etcd
Process: 29284 ExecStart=/usr/bin/etcd $DAEMON_ARGS (code=exited, status=1/FAILURE)
Main PID: 29284 (code=exited, status=1/FAILURE)
Nov 02 02:29:07 k8s-master-22668494-vmss000000 systemd[1]: Failed to start etcd - highly-available key value store.
Nov 02 02:29:07 k8s-master-22668494-vmss000000 systemd[1]: etcd.service: Unit entered failed state.
Nov 02 02:29:07 k8s-master-22668494-vmss000000 systemd[1]: etcd.service: Failed with result 'exit-code'.
cluster-provision.log
systemctl status etcd.service:
Nov 02 02:28:26 k8s-master-22668494-vmss000000 etcd[27951]: found invalid file/dir lost+found under data dir /var/lib/etcddisk (Ignore this if you are upgrading etcd)
Nov 02 02:28:26 k8s-master-22668494-vmss000000 etcd[27951]: the server is already initialized as member before, starting as etcd member...
Nov 02 02:28:26 k8s-master-22668494-vmss000000 etcd[27951]: peerTLS: cert = /etc/kubernetes/certs/etcdpeer0.crt, key = /etc/kubernetes/certs/etcdpeer0.key, ca = , trusted-ca = /etc/kubernetes/certs/ca.crt, client-cert-auth = true
Nov 02 02:28:26 k8s-master-22668494-vmss000000 etcd[27951]: listening for peers on https://<IP_ADDRESS>:2380
Nov 02 02:28:26 k8s-master-22668494-vmss000000 etcd[27951]: listening for client requests on <IP_ADDRESS>:2379
Nov 02 02:28:26 k8s-master-22668494-vmss000000 etcd[27951]: listening for client requests on <IP_ADDRESS>:2379
Nov 02 02:28:26 k8s-master-22668494-vmss000000 etcd[27951]: --initial-cluster must include k8s-master-22668494-vmss000000=https://<IP_ADDRESS>:2380 given --initial-advertise-peer-urls=https://<IP_ADDRESS>:2380
Nov 02 02:28:26 k8s-master-22668494-vmss000000 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 02:28:26 k8s-master-22668494-vmss000000 systemd[1]: Failed to start etcd - highly-available key value store.
journalctl -xe | more
Subject: Unit etcd.service has failed
Defined-By: systemd
Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Unit etcd.service has failed.
The result is failed.
Nov 02 02:28:26 k8s-master-22668494-vmss000000 systemd[1]: etcd.service: Unit entered failed state.
Nov 02 02:28:26 k8s-master-22668494-vmss000000 systemd[1]: etcd.service: Failed with result 'exit-code'.
Nov 02 02:28:27 k8s-master-22668494-vmss000000 systemd[1]: etcd.service: Service hold-off time over, scheduling restart.
Nov 02 02:28:27 k8s-master-22668494-vmss000000 systemd[1]: Stopped etcd - highly-available key value store.
Subject: Unit etcd.service has finished shutting down
Defined-By: systemd
Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Unit etcd.service has finished shutting down.

@bjd145 How many repros have you experienced? This seems like a systemd edge case, though we'd love to discover root cause and deal with it better.

I've run the configuration at least 5-7 times and it has happened every time.  I tried last night with only having one AZ and 3 nodes for the master and I had the same error.

@ritazh @kkmsft Do we have known-working test scenarios for the above cluster config?

Thanks for reporting this @bjd145
Will try to reproduce this on our end.
One issue I see is you have specified
"firstConsecutiveStaticIP": "<IP_ADDRESS>" in your apimodel and the log is showing listening for peers on https://<IP_ADDRESS>:2380 as it is expecting the first master node to have <IP_ADDRESS>. I'm guessing your vnetcidr for master subnet starts with <IP_ADDRESS>.
From https://github.com/Azure/acs-engine/blob/master/docs/clusterdefinition.md

When MasterProfile is using VirtualMachineScaleSets, this value will be determined by an offset from the first IP in the vnetCidr. For example, if vnetCidr is <IP_ADDRESS>/16, then firstConsecutiveStaticIP will be <IP_ADDRESS>

@ritazh Thank you for looking into this and catching my mistake on the firstConsecutiveStaticIP.
I did try again by removing firstConsecutiveStaticIP and letting the value be determined but I am still having issues getting the cluster setup.
My template looks as following now:
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorVersion": "1.12.1"
},
"masterProfile": {
"count": 3,
"dnsPrefix": "bjdk8s06",
"vmSize": "Standard_DS2_v2",
"OSDiskSizeGB": 64,
"storageProfile" : "ManagedDisks",
"vnetSubnetId": "/subscriptions/[redacted]/resourceGroups/DevSub02_Network_RG/providers/Microsoft.Network/virtualNetworks/DevSub02-VNet-001/subnets/Kubernetes-Master",
"agentVnetSubnetID": "/subscriptions/[redacted]/resourceGroups/DevSub02_Network_RG/providers/Microsoft.Network/virtualNetworks/DevSub02-VNet-001/subnets/Kubernetes",
"availabilityProfile": "VirtualMachineScaleSets",
"availabilityZones": [
"1"
]
},
"agentPoolProfiles": [
{
"name": "bjdk8slinux",
"count": 2,
"vmSize": "Standard_DS2_v2",
"OSDiskSizeGB": 64,
"storageProfile" : "ManagedDisks",
"vnetSubnetId": "/subscriptions/[redacted]/resourceGroups/DevSub02_Network_RG/providers/Microsoft.Network/virtualNetworks/DevSub02-VNet-001/subnets/Kubernetes",
"availabilityProfile": "VirtualMachineScaleSets",
"availabilityZones": [
"1"
]
}
],
"linuxProfile": {
"adminUsername": "manager",
"ssh": {
"publicKeys": [
{
"keyData": "[redacted]"
}
]
}
},
"servicePrincipalProfile": {
"clientId": "[redacted]",
"secret": "[redacted]"
}
}
}
The Subnets are defined as following:

Kubernetes-Master - <IP_ADDRESS>/24
Kubernetes - <IP_ADDRESS>/23

There are not NSG or RouteTables assigned to the subnet yet. Each of the three masters can ping each other and connect to each other over tcp.
The CSE still fails with error code 14
Logging into any of the 3 masters:
manager@k8s-master-43851225-vmss000001:/var/log/azure$ systemctl status etcd
● etcd.service - etcd - highly-available key value store
Loaded: loaded (/etc/systemd/system/etcd.service; disabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Thu 2018-11-08 18:57:40 UTC; 43min ago
Docs: https://github.com/coreos/etcd
man:etcd
Process: 6288 ExecStart=/usr/bin/etcd $DAEMON_ARGS (code=exited, status=1/FAILURE)
Main PID: 6288 (code=exited, status=1/FAILURE)
Nov 08 18:57:39 k8s-master-43851225-vmss000001 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 18:57:39 k8s-master-43851225-vmss000001 systemd[1]: Failed to start etcd - highly-available key value store.
Nov 08 18:57:39 k8s-master-43851225-vmss000001 systemd[1]: etcd.service: Unit entered failed state.
Nov 08 18:57:39 k8s-master-43851225-vmss000001 systemd[1]: etcd.service: Failed with result 'exit-code'.
Nov 08 18:57:40 k8s-master-43851225-vmss000001 systemd[1]: etcd.service: Service hold-off time over, scheduling restart.
Nov 08 18:57:40 k8s-master-43851225-vmss000001 systemd[1]: Stopped etcd - highly-available key value store.
Nov 08 18:57:40 k8s-master-43851225-vmss000001 systemd[1]: etcd.service: Start request repeated too quickly.
Nov 08 18:57:40 k8s-master-43851225-vmss000001 systemd[1]: Failed to start etcd - highly-available key value store.
Nov 08 18:57:40 k8s-master-43851225-vmss000001 systemd[1]: etcd.service: Unit entered failed state.
Nov 08 18:57:40 k8s-master-43851225-vmss000001 systemd[1]: etcd.service: Failed with result 'start-limit-hit'.
syslog shows:
Nov  8 18:30:56 k8s-master-43851225-vmss000000 etcd[3918]: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag
Nov  8 18:30:56 k8s-master-43851225-vmss000000 etcd[3918]: recognized environment variable ETCD_DATA_DIR, but unused: shadowed by corresponding flag
Nov  8 18:30:56 k8s-master-43851225-vmss000000 etcd[3918]: setting maximum number of CPUs to 2, total number of available CPUs is 2
Nov  8 18:30:56 k8s-master-43851225-vmss000000 etcd[3918]: found invalid file/dir lost+found under data dir /var/lib/etcddisk (Ignore this if you are upgrading etcd)
Nov  8 18:30:56 k8s-master-43851225-vmss000000 etcd[3918]: peerTLS: cert = /etc/kubernetes/certs/etcdpeer0.crt, key = /etc/kubernetes/certs/etcdpeer0.key, ca = , trusted-ca = /etc/kubernetes/certs/ca.crt, client-cert-auth = true
Nov  8 18:30:56 k8s-master-43851225-vmss000000 etcd[3918]: listening for peers on https://<IP_ADDRESS>:2380
Nov  8 18:30:56 k8s-master-43851225-vmss000000 etcd[3918]: listening for client requests on <IP_ADDRESS>:2379Nov  8 18:30:56 k8s-master-43851225-vmss000000 etcd[3918]: listening for client requests on <IP_ADDRESS>:2379
Nov  8 18:30:56 k8s-master-43851225-vmss000000 etcd[3918]: --initial-cluster must include k8s-master-43851225-vmss000000=https://<IP_ADDRESS>:2380 given --initial-advertise-peer-urls=https://<IP_ADDRESS>:2380
Nov  8 18:30:56 k8s-master-43851225-vmss000000 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Nov  8 18:30:56 k8s-master-43851225-vmss000000 systemd[1]: Failed to start etcd - highly-available key value store.


DirDiff is broken after patch 8.1.2289

Describe the bug
The DirDiffNext command of the DirDiff plugin creates buffers that do not exist on the file system.  git bisect says that this happened after patch 8.1.2289: after :diffsplit closing the window does not disable diff.
A mail refering to this issue is being sent to the DirDiff maintainer.
To Reproduce
Content of the /path_to_testdir directory:
    $ cd /path_to_testdir; ls -R
    .:
    one/  two/

    ./one:
    a  b

    ./two:
    a  b

The content of file /path_to_testdir/two/a is AA. The content of the three other files is not relevant.
Run the following commands in the test directory (note that the call to DirDiffNext may be replaced by hitting the keys j and <Enter>:
    :DirDiff one/ two/
    :ls
      1 %a-  "/tmp/vnc0HK7/1"               line 7
      2  a   "/path_to_testdir/two/a" line 0
      3  a   "/path_to_testdir/one/a" line 0
    :DirDiffNext
    :ls
      1  a-  "/tmp/vnc0HK7/1"               line 7
      2 %a-  "/path_to_testdir/two/a" line 1
      4  a   "/b"                           line 0
      5  a   "AA/b"                         line 0

Problem: the new buffers 4 and 5 do not exist.
Environment (please complete the following information):

All Vim versions after (and including) 8.1.228.
Archlinux
DirDiff version 1.1.4, i.e. the latest version.

Investigation
The problem does not occur when the call to set_diff_option() in ex_diffoff() at the HEAD of the vim master branch is commented out.
DirDiffNext parses the buffer of the top right window instead of the one in the bottom window.  So it seems that either patch 8.1.2289 does not restore some context after calling ex_diffoff() or that the plugin makes some assumption that is not anymore valid after the patch.
Workaround
The following patch to dirdiff.vim prevents the call to ex_diffoff() and is a workaround (the plugin does not need the new closeoff option although it should be able to run with it).
diff --git a/dirdiff.vim b/dirdiff.vim
index b4a602e..03769e9 100644
--- a/dirdiff.vim
+++ b/dirdiff.vim
@@ -411,6 +411,7 @@ function! <SID>DirDiffOpen()

         " Go back to the diff window
         wincmd j
+        set diffopt=internal,filler
         " Resize the window
         exe("resize " . g:DirDiffWindowSize)
         exe (b:currentDiff)

Oups, s/DirDiff version 1.1.4/DirDiff version 1.1.5/ which is indeed the latest version.

On 2020-05-05, xdegaye wrote:

Describe the bug
The DirDiffNext command of the DirDiff plugin creates buffers that do not exist
on the file system. git bisect says that this happened after patch 8.1.2289:
after :diffsplit closing the window does not disable diff.
A mail refering to this issue is being sent to the DirDiff maintainer.
To Reproduce
Content of the /path_to_testdir directory:
$ cd /path_to_testdir; ls -R
.:
one/  two/

./one:
a  b

./two:
a  b

The content of file /path_to_testdir/two/a is AA. The content of the three
other files is not relevant.
Run the following commands in the test directory (note that the call to
DirDiffNext may be replaced by hitting the keys j and :
:DirDiff one/ two/
:ls
  1 %a-  "/tmp/vnc0HK7/1"               line 7
  2  a   "/path_to_testdir/two/a" line 0
  3  a   "/path_to_testdir/one/a" line 0
:DirDiffNext
:ls
  1  a-  "/tmp/vnc0HK7/1"               line 7
  2 %a-  "/path_to_testdir/two/a" line 1
  4  a   "/b"                           line 0
  5  a   "AA/b"                         line 0

Problem: the new buffers 4 and 5 do not exist.
Environment (please complete the following information):
• All Vim versions after (and including) 8.1.228.
• Archlinux
• DirDiff version 1.1.4, i.e. the latest version.
Investigation
The problem does not occur when the call to set_diff_option() in ex_diffoff()
at the HEAD of the vim master branch is commented out.
DirDiffNext parses the buffer of the top right window instead of the one in the
bottom window. So it seems that either patch 8.1.2289 does not restore some
context after calling ex_diffoff() or that the plugin makes some assumption
that is not anymore valid after the patch.
Workaround
The following patch to dirdiff.vim prevents the call to ex_diffoff() and is a
workaround (the plugin does not need the new closeoff option although it should
be able to run with it).
diff --git a/dirdiff.vim b/dirdiff.vim
index b4a602e..03769e9 100644
--- a/dirdiff.vim
+++ b/dirdiff.vim
@@ -411,6 +411,7 @@ function! DirDiffOpen()
     " Go back to the diff window
     wincmd j

   set diffopt=internal,filler
   " Resize the window
   exe("resize " . g:DirDiffWindowSize)
   exe (b:currentDiff)

On 2020-05-05, xdegaye wrote:

Oups, s/DirDiff version 1.1.4/DirDiff version 1.1.5/ which is
indeed the latest version.

I was able to reproduce the problem starting Vim as "vim --clean",
using Vim 8.2.587, and sourcing the plugin, version 1.1.5.
Interestingly, if I started Vim as just "vim", with all my
configuration, DirDiff behaved fine.
While the latest version on the Vim scripts page,
https://www.vim.org/scripts/script.php?script_id=102, is 1.1.5, that
page is no longer maintained.  The plugin is now available only from
the GitHub repo at https://github.com/will133/vim-dirdiff.  The
latest version is 1.1.6.
I didn't know all that until I started looking into this problem.
I discovered that my dirdiff plugin was out of date, so I updated to
the latest from the repo.  I no longer see the problem.
Regards,
Gary

This is related to will133/vim-dirdiff#27.

I suppose it works also when the closeoff argument from the diffopt settings is removed? Having said that, I am not sure what you expect us to do here. As apparently the issue is also fixed in the dirdiff plugin, I tend to close this issue

@chrisbra

I suppose it works also when the closeoff argument from the diffopt settings is removed?

It is indeed what is proposed in the workaround in the OP.
Not sure that this is fixed as issue will133/vim-dirdiff#27 is closed in DirDiff only because the problem does not occur anymore after an unrelated https://github.com/will133/vim-dirdiff/commit/478c1a20ff2612eda19afd8f965e35889a3499fe commit.
There is this french expression mostly used by developers that says tomber en marche (it is a reference to the tomber en panne common idiom that translates to to breakdown in english, tomber means to fall and en marche means working) and it means that something is working now but for no understandable reason. I would not call that a fix.

@xdegaye : sometimes a problem disappears for an unknown reason. On sites using the Bugzilla software to report and follow bugs and RFEs, it's the difference between RESOLVED FIXED (the problem has been identified, and the appropriate change has been implemented — the coffee machine had broken so it was repaired or replaced on such-and-such a day; or there was an error in the source and it was fixed at changeset abcd12349876fedc) and RESOLVED WORKSFORME (the problem does not happen anymore but we don't know why — maybe there was a race condition somewhere, and a refactoring of some apparently unrelated piece of code made the race condition disappear; or maybe the bug reporter switched to a faster machine and doesn't get hit by the problem anymore).
So we don't know what made the problem disappear, but as long as it doesn't come back we can't go hunting for it.


Add markdown-oxide language-server

looks like Helix enables this one by default (along marksman).
Unfortunately we have a bug where, if one language server is not installed, we disable all language servers for the current buffer (lsp-block-in-buffer)
We should fix that, then we can enable multiple servers by default (though that still is kind of hacky)


Merge present.py into run.py

Add --run-specs to helm-run
Change default arguments of --run-specs and --conf-paths to []

Fixes #1225

Yup, I ran the quick start and confirmed that it still works.


add patches for quest

nice work.
I think it might be better to split the patches into one file for each though.
That will keep it easier to manage in the long term, since we will probably get one patch for each newer quest, where the resources are missing some field.
maybe we can put them under Patch/Quest/[mainQuestId].json.
Maybe if we want to be able to categorise it even more in the future (maybe AQ,RQ... or based on the region) we could also allow subfolders, as long as the filename is [mainQuestId].json. But this is fully optional, since it would make the patcher a bit more complex.

Yeah, that's a better solution


replace all remaining uses of __x with __id

See discussion here for background

Hey @ericniebler Can you please say what i exactly need to do here, I'm ready to give it a try.

Sure. There are places that use __x to hide a template parameter's associated namespaces. You'll see this pattern in code:
template <class ReceiverId>
struct xyz_receiver {
  using Receiver = stdexec::__t<ReceiverId>;
  // ... receiver implementation here
};

// ... later

template <class Receiver>
xyz_receiver< stdexec::__x<Receiver> > make_xyz_receiver(Receiver rcvr) {...

With __x, the code needs to become:
template <class ReceiverId>
struct xyz_receiver {
  using Receiver = stdexec::__t<ReceiverId>;

  struct __t {
    using __id = xyz_receiver;
    // receiver implementation here...
  };
};

// ... later

template <class Receiver>
stdexec::__t< xyz_receiver< stdexec::__id<Receiver> > > make_xyz_receiver(Receiver rcvr) {...

Replacing __x with __id this way improves the quality of diagnostics by making nested sender types shorter and less noisy, and so easier to read.
Thanks for offering to help!

Woahh, that's neat !!!
Can you provide me with a head start? I simply want to know which files require modifications, then I'll get going.
Thank you.

You could start with include/exec/async_scope.hpp.

Okay It'll do.
Thanks


[GHSA-m3px-vjxr-fx4m] Filament Excel Vulnerable to Path Traversal Attack on Export Download Endpoint

Updates

Affected products
CVSS v3
Description

Comments
CVE: https://github.com/pxlrbt/filament-excel/security/advisories/GHSA-m3px-vjxr-fx4m, Repo: https://github.com/pxlrbt/filament-excel/releases/tag/v1.1.14

Hi there @pxlrbt! A community member has suggested an improvement to your security advisory. If approved, this change will affect the global advisory listed at github.com/advisories. It will not affect the version listed in your project repository.
This change will be reviewed by our Security Curation Team. If you have thoughts or feedback, please share them in a comment here! If this PR has already been closed, you can start a new community contribution for this advisory

Hi @RChutchev, I made sure to update the CVE record for CVE-2024-42485 in addition to updating the advisory. Thanks for the contribution and have a good week!


Added possibility to truncate a table cell

In the scenario having an app definition like with a real long cmd section, the command dcos marathon app list would be jumbled. Therefore it would be nice to have the possibility to truncate the cmd section.
   I would add a test though in cli/tests/unit/test_tables.py and/or cli/dcoscli/tables.py

unrelated failing test fixed in #885


Lot of heuristic antivirus detection with GOARCH=386 on Windows

What version of Go are you using (go version)?
go version go1.9.1 windows/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
set GOARCH=386
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\dev\go;C:\dev\neard\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set GO386=sse2
set CC=gcc
set GOGCCFLAGS=-m32 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\username\AppData\Local\Temp\go-build
520536943=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

My issue
Hi, i have made a small wrapper written in Go to make some apps portable on Windows system. For example slack-portable.
And i have an issue when the same code is compiled for 386 goarch. There are lot of heuristic antivirus detections (13 / 67), whereas if it's compiled in x64, there are far fewer (2 / 66).
Here is an example with slack-portable :

slack-portable-ia32.exe
slack-portable-x64.exe

Can you tell me the difference between compiling in 386 versus compiling in x64 and how antivirus scans this type of detection?
Thanks a lot!

An interesting observation.  However, this sounds like more of an issue with the antivirus than an issue with Go, directly at least.  We only use our bug tracker for tracking actionable bugs and tracking proposals going through the Proposal Process.
Please see https://golang.org/wiki/Questions for good places to ask questions like this.
You can come back and reopen or refile this bug if there is something actionable that needs to be done in the compiler/linker/whatever.


Players gain fly mode after creative.

A player in survival can gain essentials fly mode on by the following procedure.
Visit a creative world.
Teleport directly to a survival world into a fly allow region.
Teleport to a region without the fly flag.
At that point the player can fly and essentials /whois shows fly:true. Running the command /fly  will disable their fly mode.
On my server this can be achieved by visiting plot world, then spawn where I have the flag fly allow set, then tping home assuming home is in survival and not in a fly allow region.
WorldGuard Extra Flags 2.11.1
Paper-1004
WorldGuard 6.2

Fixed?

No, it's still an issue. I use 'tricks' to prevent them from tping from a creative world into a survival region with the fly allow flag.

Is this finally fixed? I can't seem to reproduce the error with old version but with new version the fly flag works more reliable. If I change up words and teleport around it works just fine but on old versions it had some issues.

It is fixed. I'm not sure when but a month or so ago I tested and found it no longer occurs. I just tested to verify. I going to hope it's not overstepping my bounds to close this issue.


How to implement filtering for each column in dropdown header element

this component is awesome, but is there any features like datatables.net dropdown header filtering?

No but the header supports custom components for each cell so it would be pretty easy to add a dropdown component that uses gtApplyFilter to filter the column based on selected values.


ignore item is not working

I installed the plugin to track my inventory
but i have expensive Anchovy pizza in my inventory. I added it to the ignore list box and it did not ignore the pizza.

had to hover over it (applies after banking) i feel like this should be in the title of the feature lol
ignore items: (applies after banking)
/*
Anchovy pizza
*/

Good idea. I will add it into the title. Eventually I can probably just get it to apply instantly.


Progressive web app removed?

#501 indicated that this was part of webamp once

I mostly built it to learn about PWAs. It introduced a lot of complexity which caused real bugs, and the value seemed minimal. What were you hoping to get out of having Webamp as a PWA?

I was using it for the pin to taskbar/start menu/app menu functionality, but Edge seems to offer this for apps that aren't actual PWAs. Not the other chromium distributions.

Yeah, I think offline support was the main value added, but I doubt many people actually need/use that, and it caused many issues where people would load an old version from cache rather than getting the newest version.


Moving tasks between projects

Hi. When i move a task to another project it always shows up on the first column. its there a way to change that. ex: pick which column you want it to go to.
Thanks

Until this improvement is implemented, can this behavior be documented?  This is the only place I've found that describes what will happen to the moved task.

It's now possible with the development branch, see http://kanboard.net/documentation/duplicate-move-tasks for more information.


Fix arm64 backend to compile

Some changes to the arm64 backend as discussed on irc

I have to make some changes and remove the config.lisp file which I included by mistake


Cannot get directive to compile under Angular 6

The program is very neat. However, I cannot get the directive to build under angular 6 / angular cli.
Does anybody have a working directive for angular 6?

well, looks like everybody has a working directive :)
What is exactly the issue? Is there any error message? What command do you run? What is the issue?

I apologize. It works. Great piece of coding. Thanks


[1.12.2] [suggestion] Barrels and Sieves out of different Woods

Please add Barrels and Sieves(and maybe Sifters, Grinders and Crucibles) for all 6 Vanilla Wood Types.

This is planned to be done in 1.13, with the re-write. But is unlikely to be done in 1.12.

I would realy like to have it in 1.12.2 because i think many Mods won't be available for 1.13.2.(atleast not before there is a Stable Forge 1.13.2 version)


permission issue to set up email when running test via Run-TestsInBcContainer

Describe the issue
test procedure includes code creating Test Email Account (using "Connector Mock" Codeunit).

while running test manually at BC, all test procedures reports success
while running test via powershell script, getting: failure message="Your user account does not give you permission to set up email. Please contact your administrator."
in both cases I am using the same credentials

Scripts used to run the test and cause the issue
Run-TestsInBcContainer     -companyName $companyName
-containerName $containerName     -extensionId $TESTAppId
-credential $credential `
-detailed
procedure causing the issue
procedure SetTestEmailAcc()
var
EmailAccount: Record "Email Account";
ConnectorMock: Codeunit "Connector Mock";
EmailAccounts: TestPage "Email Accounts";
begin
ConnectorMock.Initialize();
ConnectorMock.AddAccount(EmailAccount);
EmailAccounts.OpenView();
EmailAccounts.GoToRecord(EmailAccount);
EmailAccounts.MakeDefault.Invoke();
EmailAccounts.Close();
end;
User Account info
UserPermissionSetID: SUPER
Scope: System
Thank you,
Milos

Sorry for the delay.
Could you share a repro script and the output from that.


Add a Safe Check to template deployment values

fixes #3731
This checklist is used to make sure that common guidelines for a pull request are followed.
General Guidelines

[x] The PR has modified HISTORY.rst with an appropriate description of the change (see Modifying change log).

Command Guidelines

[x] Each command and parameter has a meaningful description.
[x] Each new command has a test.

(see Authoring Command Modules)

Codecov Report

Merging #3737 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master    #3737   +/-   ##
=======================================
  Coverage   72.09%   72.09%
=======================================
  Files         422      422
  Lines       26259    26259
  Branches     3995     3995
=======================================
  Hits        18932    18932
- Misses       6096     6099    +3
+ Partials     1231     1228    -3

Impacted Files
Coverage Δ

...azure-cli-core/azure/cli/core/commands/__init__.py
73.99% <0%> (ø)
:arrow_up:

...dback/azure/cli/command_modules/feedback/custom.py
31.25% <0%> (ø)
:arrow_up:

src/azure-cli-core/azure/cli/core/util.py
70.06% <0%> (ø)
:arrow_up:

...nent/azure/cli/command_modules/component/custom.py
16.23% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4739a4...a0ae427. Read the comment docs.

@oakeyc I believe this would still fail... Please repro this issue in https://github.com/Azure/azure-cli/issues/3731 yourself and verify that it does actually work as intended.
if deploy_values.get('status value', None) != 'Started':
    result = deploy_values['status value'] + ': ' + long_name
    result += ' (' + deploy_values.get('type', '') + ')'

If 'status value' doesn't exist, you get None and None != 'Started' would be True.
Then deploy_values['status value'] would fail.
cc: @tjprescott

@derekbekoe, you're right that it's does not work as intended. Should I revert the PR or just submit a new one with the fix

@oakeyc you can submit a new PR.


SyntaxError: f-string expression part cannot include a backslash

I am running the script on windows using Python 3.11
I am getting this error:
File "E:_Trading\RobinhoodAI\main.py", line 313
)
^
SyntaxError: f-string expression part cannot include a backslash

Hey! Let me check

Ok. try v2.5.2 - the issue should be fixed there. Let me know if it's not

That issue is fixed


Add favicon.ico and handler

Created a goreportcard favicon based on the original gopher favicon.

Haha, I like it, thanks!

@nstratos that is great, thank you!

Thank you too. I love this project!


⚠ Bofrost.be has degraded performance

In 4f2d2b0, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1132 ms

Resolved: Bofrost.be performance has improved in c1a689d.


[style]: Footer UI Hover Effect error

What's the style idea?
an unnecessary underline is coming on hover to the email
@SurajPratap10 pls assign me this under GSSOC23
Add screenshots

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Raise a new issue after completing the previous one, As it's not a good practice to assign so many issues to a single folks at a time. For now I am closing this @AmanSingh6574 👍

@SurajPratap10  sir can you pls open this issue now as the rest of the issues are completed


--help-config should be called --help-settings

Follow-on from #1105 rebranding exercise.

While I'm doing this I should rename this internal variable to avoid confusion in the future:
https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203


DS-2066-pipeline-versions initial draft of pipeline versions docs

Description
These changes contain documentation on data science pipeline versioning, to be available at RHOAI 2.7.
How Has This Been Tested?
These documentation changes have been tested against a build provided by the development team.
Merge criteria:

[ ] The commits are squashed in a cohesive manner and have meaningful messages.
[ ] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[ ] The developer has manually tested the changes and verified that the changes work

@jgarciao - could you please review, thanks.


Iterate through CSV field list, skip camel call on raw

Resolve #8

https://ownerrez.zendesk.com/agent/tickets/138698


Anyone have a pretrained model with default configuration and the great results?

Hi,
Did you obtain the get the great results with default configuration? If you succeed, could you please provide me the model? Thanks a lot!

solved

Hi @chansongoal , can you share your finding here?


Update

Updated background to new black2
Conformed to style-guide:
- background of selected item
- foreground of selected item
- inactive/disable text
Changed title color to Lavender to be different than the main text color
Changed highlight color for keyboard shortcut to Red to stand out more (as in the original BTop)
Fixed typo in theme[temp-mid]: "#f8bd95" -> "#f8bd96"

Thanks, looks good to me!
@Pocco81, what do you think of using maroon for the button accents? While blue admittedly doesn't stand out too much, I feel like maroon stands out a bit too much.

@lokesh-krishna would have to see first. Got any screenshots at hand?

closed as per v0.2 update of catppuccin