Operator generation from onnx

onnx as submodule

[x] freezed on latest version
[x] wrote Makefile handling onnx build

generator script

[x] operator header
[x] operator doxygen documentation
[x] resolve function for operator type
[x] global operator stub
[ ] resolve function for operator name
[x] stub for each type combination
[x] fallback to installed onnx instead of self-compiled
[x] operator subset
[x] domain subset
[x] version subset
[x] old operators
[ ] type subsets
[ ] return enum
[x] domain specific header directory
[x] domain in operator name
[x] version in operator name
[x] domain specific implementation directory
[x] domain specific header filename
[x] domain specific implementation filename
[ ] switch to replace weakref with hardcoded stub
[ ] generate headerfiles which include whole domains

Questions

how to manage generated files?

should be generated with main Makefile, implies building onnx
[x] simply add the generated files and update when onnx submodule is updated

my choice [alrevuelta: agree. Where do you plan to add the call to the python script?]

how to handle the different types?

[x] convert current stub into switch for type specific implementations and stub these type implementations [alrevuelta: My choice also. As a random idea, would be nice to have the possibility of not including all the types in the binary. I.e. if I want to run inference on a model that is using only the double implementation, maybe there is a way to "skip" the other types and have a smaller binary, which might be useful in some constrained devices. Just an idea though.]

allows partial implementations
stub can check if right types were supplied
how to resolve different types between inputs?
if possible, my choice

single implementation handles all types

should the operator return value be a custom enum? [alrevuelta: Will be scarify a lot of performance? Your suggestion seems fine to me (ENOSYS, ENOMEM) are important, but I don't really understand EDOM or ERANGE. If the onnx is correct we should never run into overflow issues, right?]

[x] yes, we have cases errno is not sufficient
no, we can do everything with errno

documents which return values are expected

should we respect onnx domains?

currently 3 existing domains: '', ai.onnx.ml', ai.onnx.training'
may produce name conflicts in the future
[x] rename operators to reflect domains

my choice [alrevuelta: Yep, totally agree. I would both rename and restructure dirs]

[x] restructure dirs to reflect domains

would allow to generate domain libraries, my choice

[x] rename files to reflect domains

not necessary if we split up dirs

should we integrate 'old' operators?

onnx redefined some operators
produces naming conflicts
what if our implementations become 'old'?
[x] rename operators to reflect version they were introduced

my choice [alrevuelta: Agree. Lets make this project as flexible and maintainable as possible]

Example of generated header

//this file was generated by ../../onnx_generator/OperatorHeader.py
# ifndef OPERATOR_OPERATOR__ONNX__UNIQUE__11_H
# define OPERATOR_OPERATOR__ONNX__UNIQUE__11_H

# include "operator.h"
# include "onnx.pb-c.h"

/**
 * onnx operator 'Unique' version 11
 *
 * @param[in]  n_input     Number of inputs (always 1)
 * @param[in]  input       Array of pointers to the inputs
 * @param[in]  n_attribute Number of attributes
 * @param[in]  attribute   Array of pointers to the attributes
 * @param[in]  n_output    Numper of outputs (1 to 4)
 * @param[out] output      Array of pointer to the outputs
 * @return                 Error code
 *
 * @retval     0        No Error
 * @retval     ENOSYS   Operator is stubbed
 * @retval     EINVAL   Invalid argument
 * @retval     ENOMEM   Out of Memory
 * @retval     EFAULT   Invalid addr
 * @retval     EDOM     Math argument out of domain
 * @retval     ERANGE   Math result not representable
 *
 * Find the unique elements of a tensor. When an optional attribute 'axis' is
 * provided, unique subtensors sliced along the 'axis' are returned.
 * Otherwise the input tensor is flattened and unique values of the flattened
 * tensor are returned.
 *
 * This operator returns the unique values or sliced unique subtensors of the
 * input tensor and three optional outputs.
 * The first output tensor 'Y' contains all unique values or subtensors of the
 * input.
 * The second optional output tensor 'indices' contains indices of 'Y'
 * elements' first occurance in 'X'..
 * The third optional output tensor 'inverse_indices' contains, for elements
 * of 'X', its corresponding indices in 'Y'. ".
 * The fourth optional output tensor 'counts' contains the count of each
 * element of 'Y' in the input.
 *
 * Outputs are either sorted in ascending order or optionally in the order of
 * the first occurrence of the values in the input.
 *
 * https://docs.scipy.org/doc/numpy/reference/generated/numpy.unique.html
 * Example 1:
 * input_X = [2, 1, 1, 3, 4, 3]
 * attribute_sorted = 0
 * attribute_axis = None
 * output_Y = [2, 1, 3, 4]
 * output_indices = [0, 1, 3, 4]
 * output_inverse_indices = [0, 1, 1, 2, 3, 2]
 * output_counts = [1, 2, 2, 1]
 * Example 2:
 * input_X = [[1, 3], [2, 3]]
 * attribute_sorted = 1
 * attribute_axis = None
 * output_Y = [1, 2, 3]
 * output_indices = [0, 2, 1]
 * output_inverse_indices = [0, 2, 1, 2]
 * output_counts = [1, 1, 2]
 * Example 3:
 * input_X = [[1, 0, 0], [1, 0, 0], [2, 3, 4]]
 * attribute_sorted = 1
 * attribute_axis = 0
 * output_Y = [[1, 0, 0], [2, 3, 4]]
 * output_indices = [0, 2]
 * output_inverse_indices = [0, 0, 1]
 * output_counts = [2, 1]
 * Example 4:
 * input_x = [[[1., 1.], [0., 1.], [2., 1.], [0., 1.]],
 *
 * [[1., 1.], [0., 1.], [2., 1.], [0., 1.]]]
 * attribute_sorted = 1
 * attribute_axis = 1
 *
 * intermediate data are presented below for better understanding:
 *
 * there are 4 subtensors sliced along axis 1 of input_x (shape = (2, 4, 2)):
 * A: [[1, 1], [1, 1]],
 *
 * [[0, 1], [0, 1]],
 *
 * [[2, 1], [2, 1]],
 *
 * [[0, 1], [0, 1]].
 *
 * there are 3 unique subtensors:
 *
 * [[1, 1], [1, 1]],
 *
 * [[0, 1], [0, 1]],
 *
 * [[2, 1], [2, 1]].
 *
 * sorted unique subtensors:
 * B: [[0, 1], [0, 1]],
 *
 * [[1, 1], [1, 1]],
 *
 * [[2, 1], [2, 1]].
 *
 * output_Y is constructed from B:
 * [[[0. 1.], [1. 1.], [2. 1.]],
 *
 * [[0. 1.], [1. 1.], [2. 1.]]]
 *
 * output_indices is to map from B to A:
 * [1, 0, 2]
 *
 * output_inverse_indices is to map from A to B:
 * [1, 0, 2, 0]
 *
 * output_counts = [2 1 1]
 *
 * Constraint T:
 *   Input can be of any tensor type.
 *   Allowed Types: tensor_bool, tensor_complex128, tensor_complex64,
 *                  tensor_double, tensor_float, tensor_float16, tensor_int16,
 *                  tensor_int32, tensor_int64, tensor_int8, tensor_string,
 *                  tensor_uint16, tensor_uint32, tensor_uint64, tensor_uint8
 * Input T X:
 *   A N-D input tensor that is to be processed.
 *   Allowed Types: tensor_bool, tensor_complex128, tensor_complex64,
 *                  tensor_double, tensor_float, tensor_float16, tensor_int16,
 *                  tensor_int32, tensor_int64, tensor_int8, tensor_string,
 *                  tensor_uint16, tensor_uint32, tensor_uint64, tensor_uint8
 *
 * Output T Y:
 *   A tensor of the same type as 'X' containing all the unique values or
 *   subtensors sliced along a provided 'axis' in 'X', either sorted or
 *   maintained in the same order they occur in input 'X'
 *   Allowed Types: tensor_bool, tensor_complex128, tensor_complex64,
 *                  tensor_double, tensor_float, tensor_float16, tensor_int16,
 *                  tensor_int32, tensor_int64, tensor_int8, tensor_string,
 *                  tensor_uint16, tensor_uint32, tensor_uint64, tensor_uint8
 *
 * Output tensor(int64) indices:
 *   A 1-D INT64 tensor containing indices of 'Y' elements' first occurance in
 *   'X'. When 'axis' is provided, it contains indices to subtensors in input
 *   'X' on the 'axis'. When 'axis' is not provided, it contains indices to
 *   values in the flattened input tensor.
 *   Allowed Types: tensor_int64
 *
 * Output tensor(int64) inverse_indices:
 *   A 1-D INT64 tensor containing, for elements of 'X', its corresponding
 *   indices in 'Y'. When 'axis' is provided, it contains indices to subtensors
 *   in output 'Y' on the 'axis'. When 'axis' is not provided, it contains
 *   indices to values in output 'Y'.
 *   Allowed Types: tensor_int64
 *
 * Output tensor(int64) counts:
 *   A 1-D INT64 tensor containing the count of each element of 'Y' in input
 *   'X'
 *   Allowed Types: tensor_int64
 *
 * Attribute INT axis (optional):
 *   (Optional) The dimension to apply unique. If not specified, the unique
 *   elements of the flattened input are returned. Negative value means
 *   counting dimensions from the back. Accepted range is [-r, r-1] where r =
 *   rank(input).
 *
 * Attribute INT sorted (optional):
 *   (Optional) Whether to sort the unique elements in ascending order before
 *   returning as output. Must be one of 0, or 1 (default).
 *
 * @since version 11
 *
 * @see ../../../third_party/onnx/onnx/onnx/defs/tensor/defs.cc:2500
 * @see https://github.com/onnx/onnx/blob/master/docs/Operators.md#Unique
 */
int operator__onnx__unique__11(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

onnx_operator operator__onnx__unique__11_resolve(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_bool(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_complex128(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_complex64(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_double(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_float(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_float16(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_int16(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_int32(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_int64(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_int8(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_string(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_uint16(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_uint32(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_uint64(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

__attribute__((weak, alias("operator_stub")))
extern
int operator__onnx__unique__11__T_tensor_uint8(
  size_t                  n_input,
  Onnx__TensorProto    ** input,
  size_t                  n_attribute,
  Onnx__AttributeProto ** attribute,
  size_t                  n_output,
  Onnx__TensorProto    ** output
);

# endif

Example of generated source

//this file was generated by ../../onnx_generator/OperatorImplementation.py
#include "operator__onnx__unique__11.h"
#include <inttypes.h>
#include <stdio.h>

int operator__onnx__unique__11(
    size_t                  n_input,
    Onnx__TensorProto    ** input,
    size_t                  n_attribute,
    Onnx__AttributeProto ** attribute,
    size_t                  n_output,
    Onnx__TensorProto    ** output
) {
    return operator__onnx__unique__11_resolve(
        n_input,
        input,
        n_attribute,
        attribute,
        n_output,
        output
    )(
        n_input,
        input,
        n_attribute,
        attribute,
        n_output,
        output
    );
}

onnx_operator operator__onnx__unique__11_resolve(
    size_t                  n_input,
    Onnx__TensorProto    ** input,
    size_t                  n_attribute,
    Onnx__AttributeProto ** attribute,
    size_t                  n_output,
    Onnx__TensorProto    ** output
){
    { // check if inputs and outputs have valid range
        if (n_input < 1) {
            fprintf(stderr,
                "mismatch of input tensors: "
                "found %" PRId64 ", "
                "expected at least 1\n",
                n_input
            );
            exit(1);
        }
        if (n_input > 1) {
            fprintf(stderr,
                "mismatch of input tensors: "
                "found %" PRId64 ", "
                "expected at most 1\n",
                n_input
            );
            exit(1);
        }
        if (n_output < 1) {
            fprintf(stderr,
                "mismatch of output tensors: "
                "found %" PRId64 ", "
                "expected at least 1\n",
                n_output
            );
            exit(1);
        }
        if (n_output > 4) {
            fprintf(stderr,
                "mismatch of output tensors: "
                "found %" PRId64 ", "
                "expected at most 4\n",
                n_output
            );
            exit(1);
        }
    }
    { // check if multiple tensors constrained by 'T' have same type
        size_t n_tensors = 0;
        Onnx__TensorProto *tensors[2];
        { // find inputs for constraint 'T'
        char *names = { "X" };
        n_tensors += operator_findTensors(
            tensors,
            names,
            sizeof(names)/sizeof(*names),
            input,
            1
        );
    }
        { // find outputs for constraint 'T'
        char *names = { "Y" };
        n_tensors += operator_findTensors(
            tensors,
            names,
            sizeof(names)/sizeof(*names),
            output,
            1
        );
    }
        if ( !operator_tensorsAreOfSameType( tensors, n_tensors ) ) {
            fprintf(stderr, "tensor type mismatch between: ")
            for (size_t i = 0; i < n_tensors; i++) {
                fprintf(stderr, "%s, ", tensors[i]->name);
            }
        fprintf(stderr, "\n");
        exit(1);
        }
    }
    { // check if tensor 'X' has valid type
        Onnx__TensorProto *tensor = NULL;
        const char *name = "X";
        size_t n_tensor = operator_findTensors(&tensor, &name, 1, input, n_input);
        if (n_tensor == 0) {
            fprintf(stderr,"tensor '%s' not found!", name);
            exit(1);
        }
        uint32_t types[] = {
            ONNX__TENSOR_PROTO__DATA_TYPE__COMPLEX64, ONNX__TENSOR_PROTO__DATA_TYPE__INT8, ONNX__TENSOR_PROTO__DATA_TYPE__UINT32, ONNX__TENSOR_PROTO__DATA_TYPE__BOOL, ONNX__TENSOR_PROTO__DATA_TYPE__INT16, ONNX__TENSOR_PROTO__DATA_TYPE__DOUBLE, ONNX__TENSOR_PROTO__DATA_TYPE__FLOAT16, ONNX__TENSOR_PROTO__DATA_TYPE__FLOAT, ONNX__TENSOR_PROTO__DATA_TYPE__UINT8, ONNX__TENSOR_PROTO__DATA_TYPE__STRING, ONNX__TENSOR_PROTO__DATA_TYPE__UINT64, ONNX__TENSOR_PROTO__DATA_TYPE__COMPLEX128, ONNX__TENSOR_PROTO__DATA_TYPE__INT32, ONNX__TENSOR_PROTO__DATA_TYPE__UINT16, ONNX__TENSOR_PROTO__DATA_TYPE__INT64
        };
        if (!operator_tensorIsOneOfTypes(tensor, types, 15)) {
            fprintf(stderr,
                "input tensor '%s' has unexpected type: %u\n",
                name,
                tensor->data_type
            );
            exit(1);
        }
    }
    { // check if tensor 'Y' has valid type
        Onnx__TensorProto *tensor = NULL;
        const char *name = "Y";
        size_t n_tensor = operator_findTensors(&tensor, &name, 1, output, n_output);
        if (n_tensor == 0) {
            fprintf(stderr,"tensor '%s' not found!", name);
            exit(1);
        }
        uint32_t types[] = {
            ONNX__TENSOR_PROTO__DATA_TYPE__COMPLEX64,
            ONNX__TENSOR_PROTO__DATA_TYPE__INT8,
            ONNX__TENSOR_PROTO__DATA_TYPE__UINT32,
            ONNX__TENSOR_PROTO__DATA_TYPE__BOOL,
            ONNX__TENSOR_PROTO__DATA_TYPE__INT16,
            ONNX__TENSOR_PROTO__DATA_TYPE__DOUBLE,
            ONNX__TENSOR_PROTO__DATA_TYPE__FLOAT16,
            ONNX__TENSOR_PROTO__DATA_TYPE__FLOAT,
            ONNX__TENSOR_PROTO__DATA_TYPE__UINT8,
            ONNX__TENSOR_PROTO__DATA_TYPE__STRING,
            ONNX__TENSOR_PROTO__DATA_TYPE__UINT64,
            ONNX__TENSOR_PROTO__DATA_TYPE__COMPLEX128,
            ONNX__TENSOR_PROTO__DATA_TYPE__INT32,
            ONNX__TENSOR_PROTO__DATA_TYPE__UINT16,
            ONNX__TENSOR_PROTO__DATA_TYPE__INT64
        };
        if (!operator_tensorIsOneOfTypes(tensor, types, 15)) {
            fprintf(stderr,
                "output tensor '%s' has unexpected type: %u\n",
                name,
                tensor->data_type
            );
            exit(1);
        }
    }
    { // check if tensor 'indices' has valid type
        Onnx__TensorProto *tensor = NULL;
        const char *name = "indices";
        size_t n_tensor = operator_findTensors(&tensor, &name, 1, output, n_output);
        if (n_tensor == 0) {
            fprintf(stderr,"tensor '%s' not found!", name);
            exit(1);
        }
        uint32_t types[] = {
            ONNX__TENSOR_PROTO__DATA_TYPE__INT64
        };
        if (!operator_tensorIsOneOfTypes(tensor, types, 1)) {
            fprintf(stderr,
                "output tensor '%s' has unexpected type: %u\n",
                name,
                tensor->data_type
            );
            exit(1);
        }
    }
    { // check if tensor 'inverse_indices' has valid type
        Onnx__TensorProto *tensor = NULL;
        const char *name = "inverse_indices";
        size_t n_tensor = operator_findTensors(&tensor, &name, 1, output, n_output);
        if (n_tensor == 0) {
            fprintf(stderr,"tensor '%s' not found!", name);
            exit(1);
        }
        uint32_t types[] = {
            ONNX__TENSOR_PROTO__DATA_TYPE__INT64
        };
        if (!operator_tensorIsOneOfTypes(tensor, types, 1)) {
            fprintf(stderr,
                "output tensor '%s' has unexpected type: %u\n",
                name,
                tensor->data_type
            );
            exit(1);
        }
    }
    { // check if tensor 'counts' has valid type
        Onnx__TensorProto *tensor = NULL;
        const char *name = "counts";
        size_t n_tensor = operator_findTensors(&tensor, &name, 1, output, n_output);
        if (n_tensor == 0) {
            fprintf(stderr,"tensor '%s' not found!", name);
            exit(1);
        }
        uint32_t types[] = {
            ONNX__TENSOR_PROTO__DATA_TYPE__INT64
        };
        if (!operator_tensorIsOneOfTypes(tensor, types, 1)) {
            fprintf(stderr,
                "output tensor '%s' has unexpected type: %u\n",
                name,
                tensor->data_type
            );
            exit(1);
        }
    }
    {
    uint32_t T = 0;
    { // get type of constraint 'T'
        const char name = "X";
        Onnx__TensorProto *tensor = NULL;
        size_t n_tensor = operator_findTensors(&tensor, &name, 1, input, n_input);
        if (n_tensor == 0) {
            fprintf(stderr,"tensor '%s' not found!", name);
            exit(1);
        }
        T = tensor->data_type;
    }
    switch ( T ) {
        case ONNX__TENSOR_PROTO__DATA_TYPE__BOOL: return &operator__onnx__unique__11__T_tensor_bool;
        case ONNX__TENSOR_PROTO__DATA_TYPE__COMPLEX128: return &operator__onnx__unique__11__T_tensor_complex128;
        case ONNX__TENSOR_PROTO__DATA_TYPE__COMPLEX64: return &operator__onnx__unique__11__T_tensor_complex64;
        case ONNX__TENSOR_PROTO__DATA_TYPE__DOUBLE: return &operator__onnx__unique__11__T_tensor_double;
        case ONNX__TENSOR_PROTO__DATA_TYPE__FLOAT: return &operator__onnx__unique__11__T_tensor_float;
        case ONNX__TENSOR_PROTO__DATA_TYPE__FLOAT16: return &operator__onnx__unique__11__T_tensor_float16;
        case ONNX__TENSOR_PROTO__DATA_TYPE__INT16: return &operator__onnx__unique__11__T_tensor_int16;
        case ONNX__TENSOR_PROTO__DATA_TYPE__INT32: return &operator__onnx__unique__11__T_tensor_int32;
        case ONNX__TENSOR_PROTO__DATA_TYPE__INT64: return &operator__onnx__unique__11__T_tensor_int64;
        case ONNX__TENSOR_PROTO__DATA_TYPE__INT8: return &operator__onnx__unique__11__T_tensor_int8;
        case ONNX__TENSOR_PROTO__DATA_TYPE__STRING: return &operator__onnx__unique__11__T_tensor_string;
        case ONNX__TENSOR_PROTO__DATA_TYPE__UINT16: return &operator__onnx__unique__11__T_tensor_uint16;
        case ONNX__TENSOR_PROTO__DATA_TYPE__UINT32: return &operator__onnx__unique__11__T_tensor_uint32;
        case ONNX__TENSOR_PROTO__DATA_TYPE__UINT64: return &operator__onnx__unique__11__T_tensor_uint64;
        case ONNX__TENSOR_PROTO__DATA_TYPE__UINT8: return &operator__onnx__unique__11__T_tensor_uint8;
        default: {
            fprintf(stderr, "no matching type for constraint 'T' found!\n");
            exit(1);
        }
    }
}
}

how to manage generated files?

[ ] should be generated with main Makefile, implies building onnx
[x] simply add the generated files and update when onnx submodule is updated

my choice [alrevuelta: agree. Where do you plan to add the call to the python script?]

@alrevuelta would integrate the onnx build and generation into the main Makefile as an optional target

how to handle the different types?

[x] convert current stub into switch for type specific implementations and stub these type implementations [alrevuelta: My choice also. As a random idea, would be nice to have the possibility of not including all the types in the binary. I.e. if I want to run inference on a model that is using only the double implementation, maybe there is a way to "skip" the other types and have a smaller binary, which might be useful in some constrained devices. Just an idea though.]

allows partial implementations
stub can check if right types were supplied
how to resolve different types between inputs?
if possible, my choice

[ ] single implementation handles all types

@alrevuelta good idea. I would do this via the linker by generating a library per domain and type
So we have the stub inside the binary, but no actual implementation. The stub size can be reduced to a simple return ENOSYS.

should the operator return value be a custom enum? [alrevuelta: Will be scarify a lot of performance? Your suggestion seems fine to me (ENOSYS, ENOMEM) are important, but I don't really understand EDOM or ERANGE. If the onnx is correct we should never run into overflow issues, right?]

[ ] yes, we have cases errno is not sufficient
[ ] no, we can do everything with errno

@alrevuelta I included these errors just to show what errors we could expect from math.h
An enum would cost nothing (both resolve to simple ints), but would allow us to specify an operator interface (only these codes may be produced and should be handled)

@alrevuelta good idea. I would do this via the linker by generating a library per domain and type
So we have the stub inside the binary, but no actual implementation. The stub size can be reduced to a simple return ENOSYS.

I could also introduce a global stub and redirect the linker to this single stub if the symbol isn't defined. This may even solve the problem of having a stub for each input type permutation.

@nopeslide I like the idea of adding onnx as a submodule but maybe we can do it in a simpler way. What if we specify the specific official release(s) that we support (i.e. v1.6.0) rather that a specific commit?
If we do it like that, we can avoid having the onnx makefile and building it. We can just use the power of Python and pip. So with pip we install whatever version we are using and then C.defs.get_all_schemas() will have all the data for that release. This version information could be for example in a requirements.txt.
Maybe its not needed but doing it this way it will be very simple to add it to the CI. One idea aside could be to autogenerate in CI all the operators status that you made

@nopeslide I like the idea of adding onnx as a submodule but maybe we can do it in a simpler way. What if we specify the specific official release(s) that we support (i.e. v1.6.0) rather that a specific commit?
If we do it like that, we can avoid having the onnx makefile and building it. We can just use the power of Python and pip. So with pip we install whatever version we are using and then C.defs.get_all_schemas() will have all the data for that release. This version information could be for example in a requirements.txt.

@alrevuelta this strips the option of being bleeding edge and moves the build process to the distributor, what may get us into problems regarding build options. Who guarantees the provided library is the one we need?

Maybe its not needed but doing it this way it will be very simple to add it to the CI. One idea aside could be to autogenerate in CI all the operators status that you made

@alrevuelta this information should be an easy query of the binary with nm and would be the next step ;D

Couple of things more:
* Lets say that we have few operators implemented. If you run the python script as it is it will regenerate all c/h files and destroy the already implemented ones.

@alrevuelta I'm rewriting the script to only generate headers (only possible with the global stub). So no implementation will be overwritten. The actual implementations should not even need the generated header, since the linker handles the replacement.

* Just curious. Any reason behind having one `.h` file per operator? Can't we have a huge common header file for all operators?

@alrevuelta mostly debugging reasons, so I can generate a single headers and see simple diffs. we could generate a massive header file (up to 100k loc), but this would also make it harder to see changes when we switch onnx versions.

@alrevuelta this strips the option of being bleeding edge and moves the build process to the distributor, what may get us into problems regarding build options. Who guarantees the provided library is the one we need?

@nopeslide Which kind of problems can we get regarding the build options? Who guarantees the provided library is the one we need? well if I'm not wrong and we set a dependancy to a specific onnx version in the requirements.txt (which btw is recognised by GitHub under "Insights" -> "Dependency graph") thats the guarantee that the library we are providing is the one we need. When calling C.defs.get_all_schemas() we should be calling the exact same code as the one you are calling. Quick example:
Lets say that we are working with commit 553df22c67bee5f0fe6599cff60f1afc6748c635. With your solution you would have under third_party/onnx the folder onnx @553....
That commit, corresponds to the latest release v1.6.0, so with my solution we would have the requirements.txt file with the content onnx==1.6.0, which will tell that whatever python script is run in our repo, needs to have that onnx version. Both runs should give the same results, with the difference that we target a release rather than a commit (which looks cleaner for me).
Just trying here to avoid compiling what has already been compiled by someone else. No need to clone the whole repo and easier to install in whatever platform with pip.
I'm also insisting on this because it might be useful when testing. There is a script in onnx.backend.test.cmd_tools called data generate_data that generates all the tests for all operators (the .onnx + .pb) that we have copy-pasted in our repo. Was your testing idea related to this?

@alrevuelta this strips the option of being bleeding edge and moves the build process to the distributor, what may get us into problems regarding build options. Who guarantees the provided library is the one we need?

@nopeslide Which kind of problems can we get regarding the build options? Who guarantees the provided library is the one we need? well if I'm not wrong and we set a dependancy to a specific onnx version in the requirements.txt (which btw is recognised by GitHub under "Insights" -> "Dependency graph") thats the guarantee that the library we are providing is the one we need. When calling C.defs.get_all_schemas() we should be calling the exact same code as the one you are calling. Quick example:
Lets say that we are working with commit 553df22c67bee5f0fe6599cff60f1afc6748c635. With your solution you would have under third_party/onnx the folder onnx @553....
That commit, corresponds to the latest release v1.6.0, so with my solution we would have the requirements.txt file with the content onnx==1.6.0, which will tell that whatever python script is run in our repo, needs to have that onnx version. Both runs should give the same results, with the difference that we target a release rather than a commit (which looks cleaner for me).
Just trying here to avoid compiling what has already been compiled by someone else. No need to clone the whole repo and easier to install in whatever platform with pip.

@alrevuelta pip itself builds the source each time you install the onnx package (at least on my platform). Further, the current version "1.6.0"  (what pip uses) is broken with the latest protobuf version. A fix exists upstream, the submodule tracks it, but I can't build onnx via pip on my platform since I have the latest protobuf version.

I'm also insisting on this because it might be useful when testing. There is a script in onnx.backend.test.cmd_tools called data generate_data that generates all the tests for all operators (the .onnx + .pb) that we have copy-pasted in our repo. Was your testing idea related to this?
@alrevuelta Would be the next thing, I limited the onnx build just to the things I need for the stub generation.

@alrevuelta I'm rewriting the script to only generate headers (only possible with the global stub). So no implementation will be overwritten. The actual implementations should not even need the generated header, since the linker handles the replacement.

@alrevuelta
how do we want to implement the choice of the right implementation (regarding type)?
Currently I'm writing the type-checker/type-switch of the generator script.
Should these checks run every time the operator is called? (kind of slow I assume)
Or should I write a "resolve" function which returns the right implementation for the specified input?
As long the input type doesn't change (which shouldn't, should it?) we could reuse the inital resolved function.

@alrevuelta pip itself builds the source each time you install the onnx package (at least on my platform). Further, the current version "1.6.0" (what pip uses) is broken with the latest protobuf version. A fix exists upstream, the submodule tracks it, but I can't build onnx via pip on my platform since I have the latest protobuf version.

@nopeslide  No sure what pip does under the hood but for me it takes just some seconds to install, more than cloning the repo and compiling.

how do we want to implement the choice of the right implementation (regarding type)?
Currently I'm writing the type-checker/type-switch of the generator script.
Should these checks run every time the operator is called? (kind of slow I assume)
Or should I write a "resolve" function which returns the right implementation for the specified input?
As long the input type doesn't change (which shouldn't, should it?) we do not need to resolve the type again

@nopeslide Yep, thats something that I don't really know how to solve, I guess a switch is the way to go. The input type can change along the model (see operator Cast). An example would be a model that uses fixed point representation, where the input is lets say double and the calculations are made in int16 and the output is returned again as a double.

Had a quick look, looks neat! Added my 5 cents and looking forward to see it in action.
Quick question regarding resolve function for operator type bullet. Where do you plan to do this? In .c files? I'm trying to understand where do you plan to resolve the type. Are you going to generate one .c file for each operator where the correct function type is called?

asked me the same thing, the first idea was exactly what you assumed:
Generate a resolver which returns the correct function pointer and generate a typeless operator function which always calls the resolver and executes the returned function.
So we have the option of just using the typeless operator or "cache" the resolved functions.
If you have a better proposal, let me hear it.

@alrevuelta please have look a the current state.
keep in mind that the generated code was never put through gcc: ignore syntax errors, just assume it's pseudo code :)
I'm not quite satisfied that the resolve function does all the sanity checking.
How about I split this functionality up into 2 different function? one resolving the the constraints and the other checking if the operator can work on the supplied parameters?
Can I assume this onnx statement holds true for connxr?
Input and outputs are positionally associated with operator inputs and outputs. Attributes are associated with operator attributes by name. source
If we take a look at the batchnormalization operator. this operator has a lot of optional outputs. does this mean if we want to get the second optional output_var output we have to specify a dummy for the first optional output_mean output or is just the order fixed (output_mean is skipped). I'm asking because the lookup of names is quite time-consuming and if we skip those optional inputs/outputs we always have to find the correct tensor.

@nopeslide Looks really nice, thanks a lot for this great work!

I'm not quite satisfied that the resolve function does all the sanity checking.
How about I split this functionality up into 2 different function? one resolving the constraints and the other checking if the operator can work on the supplied parameters?

I like having all these sanity checks, but sometimes I have second thoughts and think its too much. Shouldn't our code be more dummy? Can't we assume that the model is well defined and all the number of inputs/outputs and types are correct? Or maybe we can just run all the checks when the model is loaded and not on runtime? onnx provides onnx.checker.check_model(onnx_model), but haven't checked what it does. Lets say we are running inference in a real world scenario, some times per second. Do we really need to check that the inputs/outputs and types are correct for every inference?
But looking at what we have, I can see three types of checks:

Input and outputs sizes

Same tensor type

Tensor type

Isn't 2) contained in 3)?. If all tensors have the correct type, that would imply 2, wouldn't it?
Any reason why you are not checking also the number of attributes?

Can I assume this onnx statement holds true for connxr?
Input and outputs are positionally associated with operator inputs and outputs. Attributes are associated with operator attributes by name. source

Really good point. Currently I'm just searching for the tensor by its name but I agree its very time consuming. Will investigate further on this and give you an answer.

I like having all these sanity checks, but sometimes I have second thoughts and think its too much. Shouldn't our code be more dummy?

I completely concur with you, hence the question if I should separate check and resolve functionalities so we can run an optional check of the network at startup and then resolve each operator at will without the overhead. I would also split these two functions into different files, so we can easily select what tests to compile. I just included the testing, because I thought we could use it while debugging stuff.
The onnx wrapper is also capable of introducing new operator schemas without touching the onnx framework, therefore the checks make sense because onnx can't verify these extensions.

Isn't 2) contained in 3)?. If all tensors have the correct type, that would imply 2, wouldn't it?

in 2) I just check if they share the same type, no type validation happens
in 3) I check each input and output if they have a valid type
currently 3) checks all tensors, regardless of shared types, this is of course overkill if they share a type, but I liked the simplicity of non-dependent checks.

Any reason why you are not checking also the number of attributes?

Didn't come around for it, would be the next thing. I also ignore the optional flag of inputs/outputs at the moment.

@alrevuelta

Really good point. Currently I'm just searching for the tensor by its name but I agree its very time consuming. Will investigate further on this and give you an answer.

How about I generate a context struct for each operator? this would also simplify the actual operator implementation. sth like:
struct operator__onnx__unique__11_context {
  struct {
    Onnx__TensorProto *X;
  } in;
  struct {
    Onnx__TensorProto *Y;
    Onnx__TensorProto *indices;
    Onnx__TensorProto *inverse_indices;
    Onnx__TensorProto *counts;
  } out;
  struct {
    int axis;
    int sorted;
  } attr;
  onnx_operator run;
};

the resolve function could simply fill such a struct and specify the callback (run in this example).

I completely concur with you, hence the question if I should separate check and resolve functionalities so we can run an optional check of the network at startup and then resolve each operator at will without the overhead. I would also split these two functions into different files, so we can easily select what tests to compile. I just included the testing, because I thought we could use it while debugging stuff.

Yes, lets do that. Lets separate the checks from the resolver and run it at startup.

How about I generate a context struct for each operator? this would also simplify the actual operator implementation. sth like:

In that case we would have to define one context struct per operator, maybe to much. And I also liked the current implementation, where all operators share the same interface. With this approach each operator would have a different input type.
What about having a common one where optional fields are NULL and the order follows the spec? Using BatchNormalization as an example:

Inputs: X, scale, B, mean, var, training_mode(optional)
*Outputs: Y, output_mean(optional), output_var(optional), saved_mean(optional), saved_var(optional)
Attributes: momentum(optional), output_mean(optional)

struct operator__context {
    Onnx__TensorProto **inputs;
    Onnx__AttributeProto **attributes;
    Onnx__TensorProto **outputs;
    onnx_operator run;
};

That structures will be filled at startup, so we don't have to search anymore and inputs will always have size 6, outputs 5, and attributes 2. With this we wouldn't need n_input, n_attribute and n_output.
And I really like this idea "the resolve function could simply fill such a struct and specify the callback (run in this example)." If I have correctly understood, you want to resolve the function at startup son on run time we already know which function to call.

And I really like this idea "the resolve function could simply fill such a struct and specify the callback (run in this example)." If I have correctly understood, you want to resolve the function at startup son on run time we already know which function to call.

yes

In that case we would have to define one context struct per operator, maybe to much. And I also liked the current implementation, where all operators share the same interface. With this approach each operator would have a different input type.

how about we combine both approaches? :D
the operator context will be kept generic like you proposed:
struct operator__context {
    Onnx__TensorProto **inputs;
    Onnx__AttributeProto **attributes;
    Onnx__TensorProto **outputs;
    onnx_operator run;
};

the tensors in the list have a fixed position and are NULL if not specified.
Further I generate these custom structs for each operator:
  struct {
    Onnx__TensorProto *X;
  } operator__onnx__unique__11_input;

  struct {
    Onnx__TensorProto *Y;
    Onnx__TensorProto *indices;
    Onnx__TensorProto *inverse_indices;
    Onnx__TensorProto *counts;
  } operator__onnx__unique__11_output;

  struct {
    Onnx__AttributeProto *axis;
    Onnx__AttributeProto *sorted;
  } operator__onnx__unique__11_attribute;

struct {
  struct operator__onnx__unique__11_input *in;
  struct operator__onnx__unique__11_output *out;
  struct operator__onnx__unique__11_attribute *attr;
  onnx_operator run;
} operator__onnx__unique__11_context;

inside the implementation we can simply cast the generic context to the specific one:
int operator__onnx__unique__11(struct operator__context *generic_context) {
  struct operator__onnx__unique__11_context *specific_context = (void *) context;

// access input tensor 'X'
  specific_context->in->X // same as generic_context->inputs[0]
...
}

but what about variadic tensors?
we could NULL-terminate each pointer list to accommodate for variadic tensors,
but wouldn't it be practical to directly know the number of variadic arguments?

@nopeslide Looks good to me. And being able to do specific_context->in->X is quite convenient and improves readability.

but what about variadic tensors?
we could NULL-terminate each pointer list to accommodate for variadic tensors,
but wouldn't it be practical to directly know the number of variadic arguments?

Can you elaborate more on the variadic function approach? How is the function prototype if we have two different types, tensor and attribute. And how are the inputs handled within the function?

@alrevuelta

Can you elaborate more on the variadic function approach? How is the function prototype if we have two different types, tensor and attribute. And how are the inputs handled within the function?

I meant variadic tensors featured by onnx (i.e. see Adagrad)
We could just append all of these to the tensor pointer list and define them as variable array inside the specific struct:
struct operator__ai_onnx_training__adagrad__1_input {
  Onnx__TensorProto *R;
  Onnx__TensorProto *T;
  Onnx__TensorProto *inputs[];
};

by NULL-terminating the pointer list we could enable simple iteration over them, but I assume there are cases in which the number of tensors may be a nice information to have beforehand.

@nopeslide I see, so your suggestion is to combine the variadic tensor with the solution we already discussed? I like the idea, and its also a great way of supporting operators with an undefined number of i/o.
NULL-terminating the sequence seems good to me, but I agree that having the n_of_tensors might be good in some occasions. Maybe adding it is too much, so I would go with the NULL-terminated way.

Maybe adding it is too much, so I would go with the NULL-terminated way.

too much in what way?
memory-wise a null termination costs the same as a counter.
execution-wise there should be no difference in calling since we pass the context struct as parameter.

@nopeslide Right. In that case which value do you plan to store? The total number of input/outputs? Or only the number of non-fixed i/o? I would go with the second.

have a look at loop.
it features optional inputs and an additional variadic input.
onnx seems to solve the problem by forcing the user to insert default tensors for the optional ones.
I think we may overengineer this one. let us just do the NULL-termination and see if we get into any problems. the struct can always be extended without breaking anything.

However I can't come up with a good solution for the outputs. Some nodes inputs depend on previous nodes outputs to work. So some nodes inputs, will point some other nodes outputs. And if we do that at startup, it will point to a memory position that is empty at that moment, which is not problem ofc.
My point here is that for populating all inputs for a given operator, we need some knowledge on past operators, and this is something that I don't think is trivial.
Any ideas?

If I understand you correctly, you are worried about the initialization order?
If so, isn't this the same problem we are also facing when scheduling nodes with multiple threads?
To schedule multiple threads we would need a dependency graph telling us how the nodes depend on each other and which nodes may be executed in parallel.
And this dependency graph should also tell us in what order the need to be initialized.

If I understand you correctly, you are worried about the initialization order?
If so, isn't this the same problem we are also facing when scheduling nodes with multiple threads?
To schedule multiple threads we would need a dependency graph telling us how the nodes depend on each other and which nodes may be executed in parallel.
And this dependency graph could also tell us in what order the need to be initialized.

@nopeslide
Not exactly, so far we are assuming that once we reach a node, all its inputs have been previously computed. So that should be ok for now.
I will try to explain it with a quick patch, that we can also use for prototyping and discussions.

@nopeslide Created a quick PR #11 to discuss the implementation details. I think its better to have it working with some simple model like MNIST and everything that Python generates hardcoded, and then we can scale it up to Python and this PR. The patch is just pseudocode.

sorry I missed some thinks

@alrevuelta
It compiles and throws no warnings or errors for the minimal mnist generation. So we could try to use the generated code for #11

@nopeslide Sorry for such a late reply.

variadic support in the context generation
via variable array? (does only support one variadic tensor in input and output)
via pointer to second struct?
resolve function filling the context (makes no sense to generate this, because it's runtime code)

So far I don't have an opinion regarding variadic support.
Regarding the function filling the context, I assumed it was going to be autogenerated code. Are you saying we have to write the code to populate the structures that contain the context? That implies searching for the tensors, and having specific knowledge of i/o/attributes name. Either way, lets not add more to this PR.

It compiles and throws no warnings or errors for the minimal mnist generation. So we could try to use the generated code for #11 (just run make onnx_generator (with ONNX_CUSTOM= if you want to use pip onnx))

Yes, thats a great idea. How should we proceed?, my idea:

Generate all the files in this commit and merge it to master, since its not breaking anything.
In #11 use the context and start migrating the old operators to the new interface.
And the rest we will see, because as you said, this PR is huge and its becoming difficult to see whats going on.

So far I don't have an opinion regarding variadic support.

lets make the varaidic problem an issue so we have a reminder when we come around implementing operators with variadic tensors.

Regarding the function filling the context, I assumed it was going to be autogenerated code. Are you saying we have to write the code to populate the structures that contain the context? That implies searching for the tensors, and having specific knowledge of i/o/attributes name. Either way, lets not add more to this PR.

what I meant was: filling and resolving these structs is a generic algorithm. it makes no sense to generate a custom one for each operator.
the current algorithm (I had in mind) would look like this:

get onnx version from pb.
find matching operand set in generated set structure
iterate through nodes in pb

find resolve function for  each node in set (we should save the resolver in the context)
malloc/construct attribute, input and output list and fill with names from pb
set resolved tensor if it is an initializer (no need for each operator to resolve from pb)

iterate through input list of created contexts
if tensor is unresolved, find and set originating context
resolve output generating context

find all input tensors

if input tensor is unset, resolve and execute originating context
search in originating context output array the correct tensor

execute

this step no. 5 would be a recursive call on the output node, walking up to the networks input and executing them one after another on its way back (depth-first execution).

@nopeslide

what I meant was: filling and resolving these structs is a generic algorithm. it makes no sense to generate a custom one for each operator.

I don't think it is that generic. Populating the context will need some knowledge of the specific one. For example, if we are populating the attribute context of conv:
typedef struct operator_context_attribute__operator__onnx__conv__11 {
    size_t length;
    Onnx__AttributeProto *auto_pad;
Onnx__AttributeProto *dilations;
Onnx__AttributeProto *group;
Onnx__AttributeProto *kernel_shape;
Onnx__AttributeProto *pads;
Onnx__AttributeProto *strides;
} operator_context_attribute_operator__onnx__conv__11

Lets say we are working with a conv operator with auto_pad, kernel_shape and strides. So we would have in the attribute list for that node three attributes with those names. We can get the pointer to each on of them, but how do we fill the structure? Some of them can be missing (if they are optional) and without specific code for each structure I don't know how we can handle those gaps.
Its more or less what I was doing here: https://github.com/alrevuelta/cONNXr/pull/11/files#diff-dbd4019d3365b93edeaa89f8607d6d45R75

@nopeslide

what I meant was: filling and resolving these structs is a generic algorithm. it makes no sense to generate a custom one for each operator.

I don't think it is that generic. Populating the context will need some knowledge of the specific one. For example, if we are populating the attribute context of conv:
typedef struct operator_context_attribute__operator__onnx__conv__11 {
    size_t length;
    Onnx__AttributeProto *auto_pad;
Onnx__AttributeProto *dilations;
Onnx__AttributeProto *group;
Onnx__AttributeProto *kernel_shape;
Onnx__AttributeProto *pads;
Onnx__AttributeProto *strides;
} operator_context_attribute_operator__onnx__conv__11

Lets say we are working with a conv operator with auto_pad, kernel_shape and strides. So we would have in the attribute list for that node three attributes with those names. We can get the pointer to each on of them, but how do we fill the structure? Some of them can be missing (if they are optional) and without specific code for each structure I don't know how we can handle those gaps.
Its more or less what I was doing here: https://github.com/alrevuelta/cONNXr/pull/11/files#diff-dbd4019d3365b93edeaa89f8607d6d45R75

You are absolutely right


Make website to interface for ML inference

Have you started on this @AndersBensen?

@magn3141 Nope 😄

Done in #18


W9 Review Ticket

Video: https://www.youtube.com/watch?v=6vBow2BA2j4

4.5/5

Good edited video
Smart to put all repositories in one place
Code is a bit repetitive of Jokes API
Good Collaboration


Model::create() not working

Hello!
I 'm trying to do an insert using Model :: create ()  ( and factory(MyClass::class)->create();)and this is not working, I'm passing an array with the attributes and values ( this array == $ fillable in the model) and only created_at and updated_at is passed to save ...
As I said earlier the fields of the array are the same as $ fillable variable.
The code:
$conta = new Conta();
$conta->create([
    'pessoa_i' => 1,
    'vendedor_id' => 1,
    'titulo' => 'testing',
    'data_emissao' => date('d/m/Y'),
    'vlr_total' => 'teste',
    'vlr_restante' => 0,
    'qtd_parcelas' => 0,
    'tipo_operacao' => 'R',
    'tipo_documento' => 'TITULO'
]);

Error message:
SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails<PHONE_NUMBER>0133_db.contas_receber_pagar, CONSTRAINT contas_receber_pagar_pessoa_id_foreign FOREIGN KEY (pessoa_id) REFERENCES pessoas (id)) (SQL: insert into contas_receber_pagar (updated_at, created_at) values (2016-02-01 19:15:31, 2016-02-01 19:15:31))
My class :
class Conta extends Model {
use PresentableTrait;
protected $presenter = ContaPresenter::class;
protected $fillable = [
    'pessoa_id',
    'vendedor_id',
    'titulo',
    'data_emissao',
    'vlr_total',
    'vlr_restante',
    'qtd_parcelas',
    'observacao',
    'tipo_operacao',
    'tipo_documento',
    'nro_cheque',
    'nro_conta_cheque',
    'nro_agencia_cheque',
    'banco_cheque',
];

Can help me?

laravel 5.1

Please ask questions on the forums.

@GrahamCampbell
This is not a question , it is a potential problem , when you have foreign key the method Model :: create () does not work, see yourself!

Ah, right. Sorry. What exact laravel version please?

It's either a typo (you are trying to insert pessoa_i instead of pessoa_id) or you are generally missing a pessoa record with an id of 1.

@GrahamCampbell  5.1!
@acasar Really I was wrong, but I tested again with ' pessoa_id ' correct and happens the same error!
@GrahamCampbell  maybe this help you! in the debug process:
public static function create(array $attributes = []) {
$model = new static($attributes);  --- this line excluded the attributes
$model->save();
return $model;
}

@GrahamCampbell 5.1!

More specific please.

@mprandot Do you have any mutators? I still highly doubt that it's a bug in Laravel.

Feel free to ask on the forums, and post back here in a new issue if you have an isolated bug report for us.


Example Dockerfile using latest linux release

I spent an hour tweaking this and since I have docker experience and I saw 3 issues about it figured someone else might get some time saved. Issue #179.  Not really sure where the main repository maintainer would put the file, but here it is.
To use this right now:

download the linux source zip file and extract it to a folder.
Run:
docker build -t wexflow-5.2-linux .
docker run -i -t -p 8000:8000  wexflow-5.2-linux:latest

and then navigate in a web browser to http://localhost:8000

Thanks for the example! Did you also manage to access the Backend web application when hosting Wexflow in a docker container? When I access the container using port 8000 I end up in the Swagger UI http://localhost:8000/swagger-ui/doc.html:

All the examples I found about how to access the Backend UI mention opening the file directly.

Glad you figured out from the docs that that the Wexflow repository actually contains multiple web services/apps that need their own ports. As my dev machine is actually a Mac, I spun up a basic docker-compose.yml file to to this for myself since I did NOT want to run the Backend from a file.  Note that this setup doesn't actually use the Docker file I contributed here, but instead uses a downloaded copy of the latest linux release .zip of Wexflow to launch TWO containers: 1 super light, very basic http server, and 1 .NET Alpine backend.
If you wanted, you could build a whole 2nd image and Docker file based on the 'wexflow-backend'
service I have here, but it's kind of overkill.
docker-compose.yml (this setup assumes that you've placed the downloaded and unzipped file
wexflow-5.2-linux-dotnet-core.zip  in the same local directory as the docker-compose.yml file.
`
version: "3"
services:
wexflow-backend:
image: danjellz/http-server
ports:
- 8080:8080
volumes:
- ./wexflow-5.2-linux-dotnet-core/Backend:/public
networks:
- mynetworkname
command:
http-server --cors
wexflow:
image: mcr.microsoft.com/dotnet/core/sdk:3.1-alpine
ports:
- 8000:8000
volumes:
- ./wexflow-5.2-linux-dotnet-core/:/opt/wexflow/
working_dir: /opt/wexflow/Wexflow.Server
# The 2 lines below were used by me as well because I had a mongodb container, I did not use the built in SQLite support:
# depends_on:
#  - mongo
command:
sh -c "cd /opt/wexflow/Wexflow.Server && dotnet Wexflow.Server.dll"
networks:
- mynetworkname
`

@bh-adrienne Thanks for your feedback. I created a wiki page related to your pull request. Do you think we should update the Dockerfile and add wexflow-backend to it?

@pestevez I updated the documentation regarding accessing the backend from Docker as well. Tell me if this works for you.
@bh-adrienne Thanks for your contribution. I added a docker-compose.yml to the repo.

@pestevez Wexflow docker images are now on Docker Hub. Follow the instructions there to get the latest version of Wexflow running on your Linux machine. I also updated the documentation regarding Docker.

Awesome, thanks so much @aelassas.


Unify navigation links between local and remote pumps

When clicking on the user's name on the top-right corner, we get two different behaviours depending on whether this is the user's own pump, or a remote pump:

locale pump: links to activity, favourites, ...; settings, account; logout
remote pump: logout (only)

It would be good to unify those a bit, adding the links to activity, favourites, ... on the remote pump interface too (I'm assuming they can be derived from the user's information, correct me if I'm wrong). Settings and account might however not make sense on a remote pump, though.
Also, it would be good to add a link to get back to the user's home stream. On the local pump, they just have to click the network's name on the top-left, but this leads to the local server stream, rather than home stream, on remote servers.
A link reliably taking the user back to their home view, say, from the top-right menu, would probably ease navigation.

I think we have to ask ourselves how we want to present the web app. Do we want each site to present its web interface as a separate app? Do we want all web interfaces to look the same, so they feel like one web app? Seems to me this will be extremely hard to accomplish technically.

On Sun, Oct 04, 2015 at 06:52:24PM -0700, Alex Jordan wrote:

I think we have to ask ourselves how we want to present the web app.
Do we want each site to present its web interface as a separate app?
Do we want all web interfaces to look the same, so they feel like one
web app? Seems to me this will be extremely hard to accomplish
technically.

Hum, that's a fair point (thinking of, e.g., @jpope777's pump). I think Evan
has repeatedly said that the Webapp was just a bootstrap mechanisms, and
that he was expecting people to later use other applications.
Perhaps the webapp should consume streams from remote site, without ever
accessing the HTTP URLs themselves, so that everything would be
displayed within the home webapp.
That said, the current web frontend is still a default upon which other
web apps are tweaked (see @jpope777's again, or steve@hub.polari.us's),
so fixing this there is likely to trickle down to other forks.
--
Olivier Mehani<EMAIL_ADDRESS>PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE  F5F9 F012 A6E2 98C6 6655
Confidentiality cannot be guaranteed on emails sent or received unencrypted.

Yeah, I was thinking of jpope's pump, too. IIRC that felt like a separate application, just because it looked so different.
By this:

Perhaps the webapp should consume streams from remote sites, without ever accessing the HTTP URLs themselves, so that everything would be displayed within the home webapp.

I assume you mean doing something similar to what diaspora* does? It could be done, but it seems like a lot of work and also, doesn't feel very "webby" to me.

I think there's two things here:

How much we should "help" users on a remote pump using the web client.  I think if we can help users navigate back and around, that's good.  We don't want users getting lost.  On the other hand, in a sense this is slightly dangerous: a remote server can "lie" about what those buttons are and if users get used to this, this could be a great way to phish users into doing activities they didn't intend.  I worry about this somewhat.  If we start doing this, we need a very clear visual indicator that "you are in remote mode".  But helping users get back can help them from getting lost.  So maybe what we really want is a "you are in remote mode", and that indicator of remote mode is something clickable that takes you back to your "home" server.  That's simple enough... users can navigate to their other tabs and etc then.
The amount we should spend investing in the web client.  Yes, the web client is "just another client", but the value here is high; the web interface is the main way that users interact with the pumpiverse, and the low polish on the web application is something that pushes many users away.

@cwebber AFAIK, remote pumps wouldn't even have to lie about buttons, right? If the remote pump is malicious, it's game over, no matter what we do. Any indicator about "remoteness" will be patched out by the malicious actor.
Don't get me wrong, I'm worried about this too. I'd like to find a solution that was secure but also had the "webbiness" of the currrent scheme, but right now, I don't see one.

That being said, yeah, we should fix the web client. Even just fixing some of the obvious bugs would go a long way.


Reactions to make most chemicals in system

Based on the reagents/chemicals.yml list I recreated most of the recipes to create the chemicals therein. Instances of Acetone have been replaced with Silicon and Sugar to Glucose. Also, there is currently no heating, and plasma would be lost in the process of creating chems that normally use it as a catalyst.

Which chemistry system is this based on?

Correct me if I'm wrong, but in this PR you've added reactions to reactions/chemicals.yml and left reactions/medicine.yml unchanged. What I'm saying is that many of the reactions you added to reactions/chemicals.yml already reside in reactions/medicine.yml. The examples I provided are the first 3 reactions you added to reactions/chemicals.yml. They are in reactions/medicine.yml already. The links are to their positions in reactions/medicine.yml on master.

Ah I see what you are saying. I was just going down a list made via the reagents/medicine.yml and since those were in the reactions/medicine.yml I added them.

I went through them and they're all present in reactions/medicine.yml on master. Sorry I didn't notice this when you were working on this on discord. I thought you were adding new chemicals. Don't let this discourage you from contributing more. Check with us again next time you're working on something and we'll make sure it's not something that's already done / in progress. You're able to close the PR with the "Close pull request" button at the bottom of the chatbox.

I did these before you started doing it. But its okay.


Error: self is not defined

This is an amazon-ivs-web-broadcast issue. I'm posting here as I can't find the amazon-ivs-web-broadcast package on GitHub.com.
TL;DR:
When using amazon-ivs-web-broadcast in a Node.js environment, rather than in a browser, it throws Error: self is not defined. This can also happen when using Next.js.
Without digging into amazon-ivs-web-broadcast's minified JavaScript code I assume this is happening because amazon-ivs-web-broadcast accesses the window object.
Is there a plan to make amazon-ivs-web-broadcast isomorphic, or a workaround so it can be used in a Lambda function or EC2 instance?

Lambda function or EC2 instance use case:
I wish to use amazon-ivs-web-broadcast in a Lambda function, or an EC2 instance, to join a stage and stream to its participants, and to pipe participant streams to other AWS services.
Is there a better way to achieve this?
Next.js usage:
In a Next.js app I'm loading the amazon-ivs-web-broadcast package as a dependency using package.json as shown in the code below:
package.json Note: the code below has been shortened for brevity
{
  "dependencies": {
    "amazon-ivs-web-broadcast": "^1.9.0",
  },
}

amazon-ivs-web-broadcast is then imported into a client component as follows:
example.tsx
"use client";

import { Stage } from "amazon-ivs-web-broadcast";

export default function Example() {
  // ...
  // Create a new stage instance
  let stage: Stage | null = new Stage(/* ... */);
  // ...
}

When the app runs Error: self is not defined is thrown.

Without digging into amazon-ivs-web-broadcast's minified JavaScript code I assume this is happening because Next.js first runs client component code on the server before running it on the client.
I see that in this demo repo amazon-ivs-web-broadcast is being loaded using "next/script". Is this to work around this issue?

Hi @mattvick, sorry to hear you're having this issue! While I can't speak to your question regarding Lambda & EC2, I may be able to help with your Nextjs issue.
As a workaround to your self is not defined errors in Nextjs, consider dynamically importing the library. An example implementation is available in this web broadcast demo app as well. As noted, you can also load the library with next/script.
For a full list of known issues with the web broadcast sdk, you may refer to the following pages:

https://aws.github.io/amazon-ivs-web-broadcast/docs/low-latency-sdk-guides/known-issues
https://aws.github.io/amazon-ivs-web-broadcast/docs/real-time-sdk-guides/known-issues

Regarding your Lambda & EC2 questions, I would also recommend that you post on AWS re:Post where our developer advocates may provide additional insights.


Add support for HTTPS

Right now, BaGet is always running under http://localhost:5000. It would be great to have a switch in appsettings.json to run under HTTPS.

BaGet is working fine with https - depending on the hosting area.
(Azure WebApp/Kudu, Docker Environment...)
Appsettings.json is only the development default, you can/should define your requirements on runtime!
Yes, there are options to implement https inside Kestrel, but in my experience typically this the "part of the job" that should be implemented by the hosting environment and not inside BaGet.
If you provide more information what you try to do, we may be able to help you in some way...
Werner

In most of the cases, I agree with you, Werner. However, for internal purposes, I don't need a full-fledged web server. I want a simple 'download and run' NuGet server without the need to setup all the other stuff. That's what I love about BaGet!

I personally use the same approach that @WernerMairl describes and this works well for production workloads. That being said I completely agree with @mu88, BaGet should work well as a standalone app. I'll go ahead and a config for this, thanks for the suggestion! :)

@WernerMairl Can you explain how to make SSL work, since you mention that it already works? My environment is a Linux server with BaGet running in a docker container. In order to get SSL I have to use a reverse proxy that handles the SSL itself. However, BaGet builds absolute URLS based solely on the request URL (which means it becomes localhost:5555) so the /v3/index.json response comes back with nothing but localhost:5555 URLs, which obviously causes package submission to fail. There doesn't seem to be a way to override the hostname/port used.

@cabal95 As far as I can see the Urls generated by BaGet take there information from the Forward headers. Did you set them correctly. Also there is the PathBase setting if you are running in subdirectory.

Yes, it is. I just verified with a tcpdump trace that Apache is sending along the X-Forwarded- headers. I'm just using Apache's standard reverse proxy config:
	<Location />
		ProxyPass http://localhost:5555/
		ProxyPassReverse http://localhost:5555/
	</Location>

However, it is partially working now. Yesterday I found and added the ProxyPreserveHost On option (which from the docs means Apache sends the original Hostname requested instead of the "new" hostname) and that seemed to fix it from sending back localhost instead of the proper hostname.
The /v3/index.json is still sending back just http:// URLs instead of https:// which isn't great, because when I upload  a new package it will still send my API key unencrypted (and then Apache will redirect it to SSL), but at least it is functional now.
When I looked at the BaGet source code, it seems to be building the URLs solely based on the Request Scheme & Hostname and not looking for any X-Forwarded- headers like it would need to.

Probably relevant lines are:
https://github.com/loic-sharma/BaGet/blob/7b4a3be0fa763f2341b75fae89bc52c4bd7edfbd/src/BaGet.Core.Server/Configuration/ConfigureForwardedHeadersOptions.cs#L11
and
https://github.com/loic-sharma/BaGet/blob/7b4a3be0fa763f2341b75fae89bc52c4bd7edfbd/src/BaGet/Startup.cs#L58
They may or may not be relevant - I am not yet that familiar with the codes structure/architecture.

Aha.  Okay, based on this I was able to determine that it is honoring the X-Forwarded-Proto header - but Apache does not send it by default. When I modified the Apache config to send that header, then the URLs were indeed built with https://. However, the X-Forwarded-Host header is still ignored. Looking at the documentation for that ForwardedHeaders enum, it appears that the above code needs to be modified to also include the ForwardedHeaders.XForwardedHost bit in the mask. That is probably why it is being ignored.

Hi @cabal95,
I am having similar issue where /v3/index.json is sending back http instead of https. You mentioned you changed apache config to get https back. Can you please let me know what change you had to make ?

This is what I ended up using to get it to pass through everything I needed.
Obviously this isn't the entire config file, but this is the relevant section.
        <Location />
                ProxyPass http://localhost:5555/
                ProxyPassReverse http://localhost:5555/
                ProxyPreserveHost On
                RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
                RequestHeader set "X-Forwarded-SSL" expr=%{HTTPS}
        </Location>

Thank you. This worked.

Hi Guys,
Sorry for reopening this one, But I haven't figured it out yet, Currently I have the same issue, but in my case I'm using nginx as a reverse proxy. As @cabal95 did mention previously, I've added proxy_set_header Host $host; to my nginx configuration, it looks like this :
server {
          nameserver site.x.com;

          location /nuget/ {
             proxy_set_header Host $host;
             proxy_pass http://localhost:5000/;
          }
}

This is not working for me, I'm missing something ? , I'm currently using the latest docker hub image.
Thanks in advance,
Sincerely,
Hector

Hi Guys,
I was checking and just in case that anybody that implements Nginx as reverse proxy, you should add the following headers:
server {
          nameserver site.x.com;

   location / {
             proxy_set_header Host $host;
             proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
             proxy_set_header X-Forwarded-Proto $scheme;
             proxy_set_header X-Forwarded-Host  $host;
             proxy_pass http://localhost:5000/;

          }
}

Please close again this issue.
Thanks in advance,
Sincerely,
H
@cabal95  Thank you very much, you help me a lot. I did the same thing with tcpdump.

Adding HTTPS support has become more urgent today with the latest VS2020 17.3 edition, the following warning/error comes up:
error NU1803: You are running the 'restore' operation with an 'HTTP' source, 'http://nuget.mysite.com:5000/v3/index.json'. Non-HTTPS access will be removed in a future version. Consider migrating to an 'HTTPS' source.

So I was able to enable HTTPS just by modifying the appsettings.json. I installed my SSL certificate into the Windows Certificate Store, in Local Machine -> Personal. Then I added the following to the Kestrel -> Endpoints block in appsettings.json:
"HttpsInlineCertStore": {
  "Url": "https://*:443",
  "Certificate": {
    "Subject": "my-certificate-CN",
    "Store": "My",
    "Location": "LocalMachine",
  }
},

In my case, my certificate is a wildcard so it's CN was *.mydomain.com, which is what I put in the Subject field above.
I also had to make sure that BaGet was running under the SYSTEM account so that it could access the Local Machine certificate store.

I also previously had it configured to use a .pfx file with a password, but that's less secure since the password had to be included in the config file in plaintext.
See the Microsoft docs on how to configure Kestrel HTTPS for certificate files/stores/etc.

Would be cool if we had a solution that lets us just use the loicsharma/baget:latest image and mount in the cert/key files we got from certbot and it magically works.

This did not work for me with Apache on Ubuntu 22.04 with .NET 6 based BaGet. I had to do the following change in addition to the Apache configuration posted above:
diff --git a/src/BaGet/ConfigureBaGetOptions.cs b/src/BaGet/ConfigureBaGetOptions.cs
index b28a658..52c8a4c 100644
--- a/src/BaGet/ConfigureBaGetOptions.cs
+++ b/src/BaGet/ConfigureBaGetOptions.cs
@@ -69,7 +69,7 @@ public void Configure(FormOptions options)

         public void Configure(ForwardedHeadersOptions options)
         {
-            options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
+            options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedHost | ForwardedHeaders.XForwardedProto;

             // Do not restrict to local network/proxy
             options.KnownNetworks.Clear();

Source: https://soapfault.com/2020/02/24/asp-net-core-reverse-proxy-and-x-forwarded-headers/

I've found a solution to use baget with https. We use baget in docker, so a nginx reverse proxy was implemented.
docker-compose.yml:
version: "3"
services:
  baget:
	image: loicsharma/baget
	container_name: baget
	volumes:
	  - ./baget_data:/var/baget
	  - ./appsettings.json:/app/appsettings.json
	  - ./certs/:/certs:ro
	ports:
	  - "5555:80"
	restart:  unless-stopped
	environment:
	 #DEBUG: "true"
	  VIRTUAL_HOST: baget
	  VIRTUAL_PORT: 80
	env_file:
	 - baget.env

	logging:
	 driver: "json-file"
	 options:
	   max-size: "50m"

  baget_ssl:
   image: nginx
   volumes:
	- ./nginx.conf:/etc/nginx/conf.d/mysite.template
	- ./baget.crt:/etc/ssl/certs/nginx-selfsigned.crt
	- ./baget.key:/etc/ssl/private/nginx-selfsigned.key
	#- ./local-ca.crt:/usr/local/share/ca-certificates/local-ca.crt
   ports:
	- "5543:443"
	#- "9180:80"
   restart:  unless-stopped
   command: /bin/bash -c "cat /etc/nginx/conf.d/mysite.template > /etc/nginx/conf.d/default.conf && exec nginx -g 'daemon off;'"
   logging:
	driver: "json-file"
	options:
	 max-size: "50m"

nginx.conf:

server {
		listen 80;
		listen [::]:80 default_server;
		server_name  baget;
		listen 443 ssl http2 default_server;
		#listen [::]:443 ssl http2 default_server;
		ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
		ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
		#ssl_certificate /etc/ssl/certs/baget.crt;
		#ssl_certificate_key /etc/ssl/private/baget.key;

		client_max_body_size 1280m;
		proxy_read_timeout 600;

		location / {
			proxy_pass http://localhost:5555;
			proxy_set_header X-Forwarded-Proto $scheme;
			proxy_set_header X-Forwarded-Host  $host;
			proxy_set_header X-Forwarded-Port 5543;
			proxy_set_header Host $host:5543;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "Upgrade";
			access_log off;

		}
		# You may need this to prevent return 404 recursion.
		location = /404.html {
				internal;
		}
}


feat: add plugin system and experimentation plugin to the site

As discussed with @trieloff during our recent workshop, I'm adding the plugin system and the experimentation plugin to the website.
Description

Adds the minimal plugin system proposal from https://github.com/adobe/aem-boilerplate/pull/254
Adds the experimentation plugin from https://github.com/adobe/aem-experience-decisioning/tree/plugin-api
Instrument the existing performance.js as a plugin

Related Issue
https://github.com/adobe/aem-boilerplate/pull/254
Motivation and Context
The goal is to instrument a half-dozen sites so we can validate the current implementation and test it against real use cases in prod.
How Has This Been Tested?
The plugin system has been tested on 2 websites, and the experimentation plugin has a demo site to test the various use cases. I also added a demo experiment in the drafts: https://plugin-system--helix-website--adobe.hlx.page/drafts/ramboz/test-experiment/
It changes the label in the CTA.
Screenshots (if appropriate):

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have signed the Adobe Open Source CLA.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

@trieloff @kptdobe @davidnuescheler Let me know your thoughts on the PR.


DASHBUILDE-48: MVP & Unit testing

This PR it's a refactor of the UI layer in order to comply with the MVP best practices. In a nutshell:

Ensure all UI components follow the MVP (Model View Presenter) pattern in order to ensure a clear separation between "the view", "the presenter" and "the model".

In such model, The presenter must not contain any widget reference and the model must not "touch" the view as well. The presenter must provide the required logic to pass & update the view with the required information.

MVP applicability ensures the logic in the presenter can be properly tested using junit/mockito tests. Along with the refactor several unit tests over the presenter components are provided.

When creating a new dashboard, the contextual menu item for creating new dispalyers does not appear, so i cannot test the mvp refactor for the displayer widget, will try to fix it np

This might be or not a side-effect issue of the refactor. As a work-around, you can test the Displayer editor from the Perspective editor.

The contextual menu issue has nothing to do with the changes in this PR since it's also failing on master right now.  @romartin Please do file a JIRA.

Please, two issues to review:

Gallery tree does not appear
Contextual menu for new perspectives does not appear

The gallery tree does not seem an issue. Verified on a fresh installation.
The contextual menu issue has been reported: https://issues.jboss.org/browse/DASHBUILDE-82

Tested again. +1

Tested the two other PR that depend on this one:

The one from jbpm-console is now working in SDM, please see the PR link
The one from kie-wb-common works fine, please see the PR link

So let's wait for the +1 on jbpm-console-ng's PR...

Added +1 on the jbpm-console's PR, as the issue is not produced by this refactor.
+1 All three PR can be merged

+1

Thanks.
master: 4277c03e16c165585514aa863b842718ca7d406d


revamp(Collective): Move countryIso under location.country

See https://github.com/opencollective/opencollective/issues/1835

[x] Accept countryISO in LocationInputType
[x] Export countryISO in LocationType
[x] Deprecate countryISO in CollectiveInterface

Coverage decreased (-0.004%) to 67.216% when pulling 1a639c0af42d5853e32f58b0047f185467374b21 on revamp/country into 18510a3017f996feeac6a1fbeacf1771b5b86480 on master.


load_dataset uses out-of-date cache instead of re-downloading a changed dataset

Describe the bug
When a dataset is updated on the hub, using load_dataset will load the locally cached dataset instead of re-downloading the updated dataset
Steps to reproduce the bug
Here is a minimal example script to

create an initial dataset and upload
download it so it is stored in cache
change the dataset and re-upload
redownload

import time

from datasets import Dataset, DatasetDict, DownloadMode, load_dataset

username = "YOUR_USERNAME_HERE"

initial = Dataset.from_dict({"foo": [1, 2, 3]})
print(f"Intial {initial['foo']}")
initial_ds = DatasetDict({"train": initial})
initial_ds.push_to_hub("test")

time.sleep(1)

download = load_dataset(f"{username}/test", split="train")
changed = download.map(lambda x: {"foo": x["foo"] + 1})
print(f"Changed {changed['foo']}")
changed.push_to_hub("test")

time.sleep(1)

download_again = load_dataset(f"{username}/test", split="train")
print(f"Download Changed {download_again['foo']}")
# >>> gives the out-dated [1,2,3] when it should be changed [2,3,4]

The redownloaded dataset should be the changed dataset but it is actually the cached, initial dataset. Force-redownloading gives the correct dataset
download_again_force = load_dataset(f"{username}/test", split="train", download_mode=DownloadMode.FORCE_REDOWNLOAD)
print(f"Force Download Changed {download_again_force['foo']}")
# >>> [2,3,4]

Expected behavior
I assumed there should be some sort of hashing that should check for changes in the dataset and re-download if the hashes don't match
Environment info

datasets version: 2.15.0                                                   │
Platform: Linux-5.15.0-1028-nvidia-x86_64-with-glibc2.17                     │
Python version: 3.8.17                                                       │
huggingface_hub version: 0.19.4                                            │
PyArrow version: 13.0.0                                                      │
Pandas version: 2.0.3                                                        │
fsspec version: 2023.6.0

I meet a similar problem as using loading scripts. I have to set download_mode='force_redownload' to load the latest script.


Link error

Start section Link error

do you mean quick start: https://dubbo.apache.org/docs/v2.7/user/quick-start/
it had been fixed.

do you mean quick start: https://dubbo.apache.org/docs/v2.7/user/quick-start/
it had been fixed.


[✨] do not rely on "name" property from package json to scope the q-manifest.json file in tmp directory

Is your feature request related to a problem?
As part of this fix https://github.com/BuilderIO/qwik/pull/2398 there was added the logic to rely on the name property from package.json for the temporary manifest file. It appears to be not really convenient in project setups w\o package.json, for example in an Nx repo.
Describe the solution you'd like
Instead of relying on the name property from package.json it might be reasonable to use something more generic, maybe the hash of the project root path of a certain app or something like that
Describe alternatives you've considered
Additional context

we cant delegate on the absolute of the root, because it would not be stable

We have a response and the issue is stale. I close it.


why 100vh for a button?

Hello @jpmedeirosmorais Its applied to the main container floatingWhatsapp not to button in which to make sure that the chat box will open properly.
width: auto will keep the element width same as its parent so if the Component is nested to an element that has a fixed width, the chat box will not open.

Okay, but the floating button(when it is instanceated)makes a big space
between it and the next component.
Em sáb., 18 de set. de 2021 16:45, Ahmed Khalil @.***>
escreveu:

Hello @jpmedeirosmorais https://github.com/jpmedeirosmorais Its applied
to the main container floatingWhatsapp not to button in which to make
sure that the chat box will open properly.
width: auto
https://stackoverflow.com/questions/17468733/difference-between-width-auto-and-width-100-percent
will keep the element width same as its parent so if the Component is
nested to an element that has a fixed width, the chat box will not open.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/awran5/react-floating-whatsapp/pull/7#issuecomment-922363628,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARVHJOO3GXA76HCMWMSA6Z3UCTT7FANCNFSM5EIHEIDQ
.

Would you please show an example? You could use codesandbox

I had the same issue. I had to copy the css file and remove the full height.

Very good! it worked for me too. Thank you very much, @jorgemndoza!

Hey guys .. Thank you for help
@jpmedeirosmorais Indeed, you were right, I just released a new version v4.0.5 with cool new features, please try it out.

@jpmedeirosmorais You're welcome.
@awran5 Good job and thanks for creating this package.


After edit animal - values didn't change after changing values second time

Steps:

Choose an action: click on 'edit'
Choose an animal: click on 'Select an animal'
Click on animal name value
Fill in all values
Click on 'Edit animal'

✅result: Choosen animal's values changed

Change again few values
Click on 'Edit animal'

❌ result: Values didn't change
❔ expected: Values change when user change them second time and click 'Edit animal'
https://github.com/FilipParyz/TestingDoggos/assets/148799634/bf87d8eb-6590-4927-bd2d-08ce85ec9d7d

After clicking edit Animal (submit) the select animal drop down resets as it should as we don't know which one we want to edit ;) just select in that drop down Ĺadna again

We will discuss ideas at the meetings

After the arrangements, I've made a change -

when user chooses animal or food - data automatically fill in.
when user submits edition - data is cleared


[SPARK-40075][SQL][SS] Refactor kafka010.JsonUtils to replace Json4s with Jackson

What changes were proposed in this pull request?
This pr just refacotr kafka010.JsonUtils to replace Json4s with Jackson.
Why are the changes needed?
Weaken the dependence on json4s.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GitHub Actions

also close this one


Validate integer field range

Inputting an out-of-range number in an IntegerField results in a database error which causes an error page to be displayed. This pull request propose an active in-form validation to avoid this issue.
Limits according to: https://docs.microsoft.com/en-us/sql/t-sql/data-types/int-bigint-smallint-and-tinyint-transact-sql

Inputting an out-of-range number in an IntegerField results in a database error. This pull request propose an active in-form validation to avoid this issue.

I just realized that this applies only to MSSQL, and not to any other DB supported by Flask-Admin. My PR thus is not valid. Closing.


replace uses of iteritems with six.iteritems

re #5648 , #282
replace dict_instance.iteritems() with six.iteritems(dict_instance) to be python 2/3 compatible.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

I just rebased to resolve a conflict, but it doesn't seem to be queued on Jenkins

Can one of the admins verify this patch?

This is OK to test.

tests havent finished, removing the label.

@jtattermusch: LGTM


Allow for SLACK_RUBY_BOT_ALIASES to contain $ and others

Currently has to be a word.

Closed via https://github.com/slack-ruby/slack-ruby-bot/pull/166.


Implement enhancement requests from the original Repo's Issues

Adding copy to clipboard button on the dialogue - #17 and #8
Remember last cancelled dialog and not show it twice  #1
On successful scan, leave the dialogue open unless user dismisses it themselves #10
Cherry-Pick fix for uppercase http/https urls from https://github.com/BFlorry/obsqr/tree/uppercasefix

I've added more there.
Care to offer a preliminary release of whatever you have done already?

@Noitarud oops, thanks for the reminder. I'll take a look at it again this weekend and come up with a release


[#154249348] Use ACM for public-facing TLS certs

#154249348 Use ACM for public-facing TLS certs
What?
Following on from the spike (#142501669) we now want to proceed with the roll-out of Amazon Certificate Manager (ACM) for the public-facing certs.
In this PR we add the logic to use the ACM certificates using the terraform  ACM certificate datasource to automatically discover them. We also add
a temporary commit with logic to delete the old certificates, which can be reverted in the future.
Dependencies
IMPORTANT We added the logic of creating the ACM certificates in alphagov/paas-bootstrap#129. That PR must be merged and applied first.
How to review

Apply alphagov/paas-bootstrap#129
Run this pipeline.
Check that the certificates have been deleted:

DEPLOY_ENV=hector
aws iam list-server-certificates | grep ${DEPLOY_ENV}
aws s3 cp s3://gds-paas-${DEPLOY_ENV}-state/cf-certs.tfstate - # it shall fail
aws s3 cp s3://gds-paas-${DEPLOY_ENV}-state/cf-certs.tar.gz  - | tar -tvzf -  # It shall not include apps or system

You can try the code after reverting the temporary commit that deletes the old certs:

Create a tmp branch, revert the last commit and push
Rerun the pipeline.

Next steps.
After merge, wait some time and revert the temporary commit.
Who?
Anyone but @keymon or @bandesz

@alext

Can we now remove the CF_SKIP_SSL_VALIDATION variable set for dev in the Makefile?

That is a good question and I have mix feelings. The variable is all over the place and that feature can be really useful when not using ELBs. For instance, we want to run the deployment on bosh-lite. Because that, I was thinking to simple set CF_SKIP_SSL_VALIDATION=false in dev
On the other hand, if it is not being used and tested, the feature will eventually be broken, and it would become obsolete code.
Other thing to take into account: we must keep it for the availability tests to work until the certs are replaced.

On the other hand, if it is not being used and tested, the feature will eventually be broken, and it would become obsolete code.

👐

On the other hand, if it is not being used and tested, the feature will eventually be broken, and it would become obsolete code.

:open_hands:

OK, I would remove the CF_SKIP_SSL_VALIDATION then ;)
We can always review the git history if we want to add it back.

Other thing to take into account: we must keep it for the availability tests to work until the certs are replaced.

OK, I would remove the CF_SKIP_SSL_VALIDATION then ;)

Based on the issue with the availability tests, I think this should be done in a follow-up PR (it can be the same one that removes the cf-certs.tfstate cleanup.

Based on the issue with the availability tests, I think this should be done in a follow-up PR (it can be the same one that removes the cf-certs.tfstate cleanup).

I am preparing everything to be as easy as revert one commit.

@alext I fixed all those comments, but I am fixing now the cloudfront configuration that relies on the system cert.

One comment above about retaining the terraform config for the moment. I think we should retain that, and the scripts to manually run it until we remove these certs.
Also, this needs a rebase onto master.

Rebased.