ApproximateGradientSumFunction and SGFunction

Describe your changes

Implements the SubsetSumFunction based on the Hackathon-000-Stochastic-Algorithms script and removes the AverageSumFunction.
Implements the SGDFunction based on the Hackathon-000-Stochastic-Algorithms script. I include the SGDFunction in this PR because is the easiest stochastic function class.

 SubsetSumFunction
The SubsetSumFunction is defined as
$$ \frac{1}{n} \sum_{i=1}^{n}f_{i}(x)$$
where $n$ is the number of subsets and $f_{i}$ are smooth functions.
Example
$$ \frac{1}{n}\sum_{i=1}^{n} F_{i}(x) = \frac{1}{n}\sum_{i=1}^{n}||A_{i} x - b_{i}||^{2} $$
f = SubsetSumFunction([LeastSquares(Ai, b=bi)]*n for Ai,bi in zip(A_subsets, b_subsets))

Note
At the moment, there are 2 sampling methods: random and sequential.  For the random case, we have the following.

sampling=random, replacement=True : means thats the next_subset method can select the same subset_num
sampling=random, replacement=False : means thats the next_subset method selects subset_num uniquely.

Example
a = [1,2,3,4,5,6]
num_subset = 2

sampling=random, replacement=True: [3,1,2], [4,2,6]
sampling=random, replacement=False: [1,3,2], [6,4,5]
sampling=sequential, [1,2,3], [4,5,6].

 SGDFunction
Example
f = SGDFunction([LeastSquares(Ai, b=bi)]*n for Ai,bi in zip(A_subsets, b_subsets), sampling="random", replacement=False)

 GD vs SGD in CIL
We can solve the LeastSquares minimisation problem $$\min_{x}\|Ax - b\|^{2}$$

Deterministically using: GD: objective_function = LeastSquares
Randomly using  GD: objective_function = SGDFunction(list of LeastSquares)

 Proximal Gradient Algorithms in CIL
For non-smooth objectives we can solve  $$\min_{x>0}||Ax - b||^{2}$$

Deterministically using: ISTA/FISTA: f = LeastSquares, g=IndicatorBox(lower=0)
Randomly using : ISTA/FISTA: f = SGDFunction(list of LeastSquares) , g=IndicatorBox(lower=0)

G = IndicatorBox(lower=0)
F = LeastSquares(A, b = noisy_sino)
initial = ig.allocate(0)
ista = ISTA(initial=initial, f=F, g=G, update_objective_interval = 1,
            max_iteration = 50)
ista.run(verbose=1)

F_sgd1 = SGDFunction(f_subsets, sampling = "random")
G = IndicatorBox(lower=0)
num_epochs = 10
initial = ig.allocate(0)
s_ista1 = ISTA(initial=initial,
            f=F_sgd1,
            g=G, update_objective_interval = n_subsets,
            max_iteration = num_epochs * n_subsets)
s_ista1.run(verbose=1)

Describe any testing you have performed

Add tests for SubsetSumFunction
Add tests for SGDFunction [wip]

Demos

https://github.com/TomographicImaging/Hackathon-000-Stochastic-Algorithms/tree/main/contrib

Link relevant issues
Close #1341
Close #1343
From the Hackathon-000-Stochastic-Algorithms repository
Close 11
Close 10
Close 9
Checklist when you are ready to request a review

[x] I have performed a self-review of my code
[x] I have added docstrings in line with the guidance in the developer guide
[x] I have implemented unit tests that cover any new or modified functionality
[ ] CHANGELOG.md has been updated with any functionality change
[x] Request review from all relevant developers
[x] Change pull request label to 'Waiting for review'

Contribution Notes
Please read and adhere to the developer guide and local patterns and conventions.

[x] The content of this Pull Request (the Contribution) is intentionally submitted for inclusion in CIL (the Work) under the terms and conditions of the Apache-2.0 License.
[x] I confirm that the contribution does not violate any intellectual property rights of third parties

@KrisThielemans (could not add you in the reviewers list)

In your description you have 2 examples @epapoutsellis
f = SubsetSumFunction([LeastSquares(Ai, b=bi)]*n for Ai,bi in zip(A_subsets, b_subsets))

and
f = SGDFunction([LeastSquares(Ai, b=bi)]*n for Ai,bi in zip(A_subsets, b_subsets), sampling="random", replacement=False)

I think they should be as follows:
f = SubsetSumFunction(*[LeastSquares(Ai, b=bi) for Ai,bi in zip(A_subsets, b_subsets)])

and
f = SGDFunction(*[LeastSquares(Ai, b=bi) for Ai,bi in zip(A_subsets, b_subsets)], \
    sampling="random", replacement=False)

Is it true?

I think the *n has to be there to compensate for the 1/n in SubsetSumFunction (which is one reason one I think the latter is a bad idea)

I suggest some changes, but in particular to the design:

I really don't think using 1/n is a good idea. It means that an SGDFunction is not a direct replacement of the original, but a replacement of 1/num_subsets orig_function. This means that every derived algorithm needs to know this, and adjust step size etc.
I suggest a change to have a subset_gradient function (without next_subset())
There is no reason to put the preconditioner in SGDFunction. In principle, it could sit in SubsetSumFunction, but in fact, it could sit in Function. I'm not so sure you want to have it in this hierarchy at all though. A gradient preconditioner is something for gradient algorithms, not for functions.

 For the preconditioner
Short answer: Totally agree, the preconditioner should be a parameter for each of the algorithms, e.g., GD, ISTA, FISTA that will follow the f.gradient(x, out=out) step. This is something we discussed with @zeljkozeljko .
Long answer: However, we do not have this option atm for out proximal gradient algorithms. In addition, the preconditioner should be a callable method. With this setup, we can cover the cases where
$$ x_{k+1}= x_{k} - \gamma_{k}D(x_{k}, iteration) \nabla f(x_{k}) $$
a) preconditioner is fixed $\frac{1}{A^{T}\mathbb{1}}$
b) depends on the iterates  $\frac{x_{k}+\delta}{A^{T}\mathbb{1}}$
c) depends on the num_subsets, i.e., activate/deactivate it after some epochs or iterations.
d) None
Notice that in case c) the (deterministic) proximal gradient algorithms have no info about the number of subsets. Atm, this is known from the SubsetsSumFunction (SGDFunction, SAGFunction, etc). So if we have a callable method for the precond step that depends on (x, iteration) the user has all the information to define it.
self.objective_function.gradient(self.x, out=self.x_update)
self.x_update *= apply_precond(self.x, self.iteration)
self.x.sapyb(1.0, self.x_update, -self.step_size, out=self.x)

Notice that the same can be applied for adaptive step-size, e.g., decreasing step size in SGD that satisfy certain conditions. Unless we have a fix step size or another method armijo rule, it should be a callable method. See for instance SGDClassifier. Finally, all of the above should and can work in ISTA and FISTA.

I suggest some changes, but in particular to the design:

I really don't think using 1/n is a good idea. It means that an SGDFunction is not a direct replacement of the original, but a replacement of 1/num_subsets orig_function. This means that every derived algorithm needs to know this, and adjust step size etc.
I suggest a change to have a subset_gradient function (without next_subset())
There is no reason to put the preconditioner in SGDFunction. In principle, it could sit in SubsetSumFunction, but in fact, it could sit in Function. I'm not so sure you want to have it in this hierarchy at all though. A gradient preconditioner is something for gradient algorithms, not for functions.

 For the 1/n convention
Short answer: I think it is important to have the 1/n weight in the base class SubsetSumFunction (inherits from SumFunction) . In this way we will not have problems when comparing with other algorithms e.g., deterministic and will not have to adjust future algorithms found in the literature when the 1/n is considered for finding the argmin.
Long answer:  In the first hackathon, we implementedSAG based on this review paper. Then I think an AverageFunction class was created but we decided with @ClaireDelplancke @zeljkozeljko @Imraj-Singh @BillyTang to remove it and fix the scalings in the SubsetSumFunction.
Example: In CIL we can solve the following LS problem
$$ argmin_{x} f(x) = \frac{1}{2}||Ax - d||^{2} $$

using the LeastSquares function and the GD algorithm (Deterministic)
using the SumFunction function and the GD algorithm (Deterministic)
using the SGDFunction function and the GD algorithm (Stochastic)

For 3) the SGDFunction has the $\frac{1}{n}$ in the call method. Therefore to match the min for the deterministic cases we need $\frac{1}{n}$ in front of the LeastSquares and the SumFunction.
For 1) :
f = LeastSquares(Aop, b=bop, c = 0.5/n)
initial = ig.allocate()
gd = GD(initial = initial, objective_function = f, max_iteration=n_epochs,
       update_objective_interval=1)
gd.run(verbose=1)

The default step_size of GD is $\frac{2*0.99}{L}$ and we have
$$ x_{k+1} = x_{k} - \frac{2 * 0.99}{L} * \frac{1}{n}A^{T}(Ax_{k}-d) =  x_{k} - \frac{2 * 0.99}{\frac{||A^{T}A||}{n}} * \frac{1}{n}A^{T}(Ax_{k}-d) = x_{k} - \frac{2 * 0.99}{||A^{T}A||} * A^{T}(Ax_{k}-d)$$
For 2) :  Here, we assume that $$f(x) : = \frac{1}{n}\sum_{i=1}^{n} f_{i}(x), \quad f_{i}(x) = \frac{1}{2}||A_{i}x-d_{i}||^{2} $$
Assuming that we have split $A$ into $A_{i}$'s and $d$ into $d_{i}$'s, we can have
average_sum = (1./n)*SumFunction(*fi_cil)
average_sum_gd = GD(initial = initial, objective_function = average_sum,  max_iteration=n_epochs,
       update_objective_interval=1)
average_sum_gd.run(verbose=1)

Here we have,
$$ L = \frac{1}{n}\sum_{i=1}^{n} L_{i} = \frac{1}{n}\sum_{i=1}^{n} ||A_{i}^{T}A_{i}|| $$
$$ x_{k+1} = x_{k} - \frac{2 * 0.99}{L} * \frac{1}{n}\sum_{i=1}^{n}A_{i}^{T}(A_{i}x_{k}-d_{i}) =  x_{k} - \frac{2 * 0.99}{ \sum_{i=1}^{n} ||A_{i}^{T}A_{i}|| } \sum_{i=1}^{n}A_{i}^{T}(A_{i}x_{k}-d_{i})$$
Finally for the SGD case, we have
sgd_fun = SGDFunction(fi_cil)
step_size = 0.005
sgd = GD(initial = initial, objective_function = sgd_fun,
        step_size = step_size, max_iteration=n_epochs*n_subsets,
       update_objective_interval=n_subsets)
sgd.run(verbose=1)

and
$$ x_{k+1} = x_{k} - \gamma_{k} \nabla f_{i_{k}} = x_{k} - \gamma_{k} A_{i_{k}}^{T}(A_{i_{k}} x - d_{i_{k}}) $$
Notice that in cases 1) and 2), there is no dependence on the number of subsets. Therefore finding the argmin is ok. However comparing the objective values for the deterministic vs stochastic algorithms we need the $\frac{1}{n}$ in front of LeastSquares and SumFunction.

I'm trying to resume the conversation we had with @KrisThielemans @epapoutsellis @gfardell
With this PR we want to propose a structure in CIL for stochastic optimisation with gradient type algorithms. A number of choices have already been made, the most relevant being that we decided not to develop stochastic algorithms rather a special class of functions that, plugged in an algorithm like GD, will make it behave like SGD. Same applies for ISTA and FISTA.
As @epapoutsellis wrote we want to be able to solve optimisation problems of the form
$$
\text{argmin}_u F(u, b) + \alpha R(u)
$$
where $F$ is a data fitting term, $b$ is the measured data, $R$ is the regulariser and $\alpha$ the regularisation parameter. $F$ is supposed to be smooth, while the regulariser might not be, and in such case we'd use an appropriate algorithm.
At the bottom of this PR we rewrite the $F$ as a separable sum
$$
F(u,b) = \frac{1}{n} \sum^n f_i(u,b)
$$
The base class for this in CIL is the SubsetSumFunction. @KrisThielemans argued that the presence of the $1/n$ multiplicative factor makes the previous not exactly a separable sum, but an averaged separable sum, hence the name SubsetSumFunction would not be describing the object correctly, proposing to use something that contained average.
Most of the algorithms we are planning to implement, have been published following this convention. The optimisation problem becomes
$$
\text{argmin}_u \frac{1}{n} \sum^n f_i(u,b) + \alpha R(u)
$$
I quote @epapoutsellis from a private communication,

If you change the number of subsets, you change the strength of your fidelity term, then you need to adjust the strength of your regulariser, i.e., alpha, in order to have the same argmin in every optimisation problem.

Also, the objective value is not immediately comparable between 2 set ups with different number of subsets, but it needs to be scaled by it.
This means that with the convention
$$
F(u,b) = \frac{1}{n} \sum^n f_i(u,b)
$$
the user will need to be extra careful in scaling the regularisation parameter and the objective value with the number of subsets.
However, if we were to choose the convention for which the separable sum is a separable sum, i.e. it does not have the $1/n$ scaling:
$$
F(u,b) = \sum^n f_i(u,b)
$$

the name SubsetSumFunction makes sense (to @KrisThielemans )
we would not have any problem of comparison of algorithms with different amount of subsets

Conversely, it would add a burden on the developers team as:

each subset function needs to take into account the missing $1/n$ which seems to be the convention in the papers
comparison with literature results (following the $1/n$ convention) will need to be appropriately scaled

Further notes on the $1/n$ convention:

the spdhg paper just uses a sum.
if we use this for gated data, it will often happen that some gates have very few counts, while others more. Doing simple averaging then makes no sense. Summing does. It seems therefore to me that the sum convention is more general.
"online" methods probably use a running average as opposed to summing, as they don't know yet how many data will come in. (but I don't know, @zeljkozeljko might).

An ugly solution would be to provide 2 functions for every case: SAGASumFunction and SAGAAverageFunction. Comparisons with papers can use the latter, and I can use the former 😄.  But this would cause a lot of code proliferation, so is probably a bad idea.

If we were to develop the functions as sums, we would need to change how the function calculates the approximate gradient, but the comparison with the papers could be simply achieved by:
functions = [subset_f for i in range(n)]
f = (1/n)*SAGAFunction(*functions)

Could this be tried with the SAGAFunction @epapoutsellis ?

Having read this, it seems to me that we just multiply the approximate gradient with n in the implementation, remove the 1/n in __call__ etc, and we're done.  no?

We have a white smoke: Remove the 1/n in front of the sum and we consider the expectation of the $n*f_{i}$
$$ E(n*\nabla f_{i}(x)) = \sum_{i=1}^{n} P_{i} (n*\nabla f_{i}(x)) = \sum_{i=1}^{n} 1/n * n * \nabla f_{i}(x) := f(x) $$

I'm a bit confused by this usage
sgd = GD(initial = initial, objective_function = sgd_fun,
        step_size = step_size, max_iteration=n_epochs*n_subsets,
       update_objective_interval=n_subsets)

Specifically, I'm unsure about calling sgd_fun the objective_function. The naming sort of suggests that the (underlying) objective changes by changing the algorithm, whereas what we sort of mean there is that we are only specifying the method used to estimate the gradient of the objective

sgd_fun = SGDFunction(fi_cil)

Yes, if sgd_fun is a SubsetSumFunction (SGD, SAG, SAGA etc), then  the gradient method which is called in every (proximal) gradient based algorithm , is a stochastic gradient which is different than the standard GD/ISTA/FISTA where the functions are not expressed as a sum.

I'm a bit confused by this usage

@zeljkozeljko I guess this will always be slightly confusing. However, choosing to stick the stochastic functionality in the functions allows us to utilize all gradient based algorithms present in CIL, i.e. GD, ISTA and FISTA and be able to perform stochastic variance minimisation with the specific gradient estimator, i.e. by changing the function.
The algorithm stochastic gradient descent can be created by GD with the SGDFunction. The same function could be used in ISTA and FISTA to create the respective stochastic version of the algorithms for free.
Probably we could decide to rename SGDFunction to be more explicit of what the function does. I think removing the D would suffice: this function is a stochastic gradient function, SGFunction.

Describe your changes (Round 2)

Implements the SubsetSumFunction  without the the 1/n weight
The SGDFunction is renamed to SGFunction
There are two sampling strategies:

replacement=True in every iteration a subset is selected randomly.
replacement=False: we do not select a subset that was previously selected
-  random suffle: At the end of each epoch = after n subsets we resuffle the list of all functions.
- single suffle: When we instantiate SGFunction for example, we suffle the list of all functions only once and this will be use for the rest of epochs.
_full_gradient renamed to full_gradient

Unittests for SubsetSumFunction and  SGFunction
Remove preconditioning from the SubsetSumFunction class

I wonder if we can agree on how the user sets the sampling strategies, as these are also useful to SPDHG, see https://github.com/TomographicImaging/CIL/pull/1345#issuecomment-1290648685.
In particular, I do not like to have a binary selector replacement (whose meaning in my view is unclear), followed by another selector: random_shuffle and single_shuffle. This means that the user needs to add another selection after they selected True or no further selection otherwise.
I would suggest going in line with https://github.com/TomographicImaging/CIL/pull/1345#issuecomment-1290648685 where the sampling strategies are explicitly named and selected by strings: in that case random, random_permutation, uniform and stagger.
There random would be equivalent to replacement=True, random_permutation would be equivalent to replacement=True AND single_shuffle=True. Clearly the options do not suffice to describe the options that @epapoutsellis lists here and would need to be expanded, but the double selector doesn't seem to be a good idea to me.

@paskino I don't have a strong opinion on this, although string options don't allow for code completion and need more error checking (case conversion etc), but that probably doesn't matter. There seems something wrong with the link to the comment though. It doesn't do anything for me

By the way, I think the title of this PR should now be changed to mention SGFunction

I wonder if we can agree on how the user sets the sampling strategies, as these are also useful to SPDHG, see #1345 (comment).

Come to think of it, I think it might be better to isolate the "subset number generator" in a separate class. At some point, you'd want to use non-uniform sampling (in fact, already used in the spdhg paper if the penalty is a "subset" I believe). You don't really want people to have to modify SubsetSumFunction to enter their favorite sampler. It's probably also clearer if we just accept a Python discrete RNG in the constructor.

I'm afraid I now also have some doubts about the name SubsetSumFunction (!), and therefore the use of "subset" everywhere. As far as these algorithms goes, they are just a sum of functions, we just happen to use them for subsets. (Again, the example being of the penalty term is a subset). In typical STIR fashion, I'd call this SumFunctionWithApproximateGradient or so... Luckily, it's a base class that nobody will ever use...

@paskino I don't have a strong opinion on this, although string options don't allow for code completion and need more error checking (case conversion etc), but that probably doesn't matter. There seems something wrong with the link to the comment though. It doesn't do anything for me

@KrisThielemans I updated the link in my comment, which now points to https://github.com/TomographicImaging/CIL/issues/1344#issue-1341547476 which describes a  "subset number generator" class, in fact.
I'd prefer enums for the selection of the sampling strategy. However, we've not been very constant in using enums throughout CIL, I'm afraid.
To be honest, I find the STIR naming conventions, though very descriptive, ~a bit~ overwhelming.

updated the link in my comment, which now points to #1344 (comment) which describes a "subset number generator" class, in fact.

that class is somewhat different. It points to a "all  angles in all subset generator". That is something useful as well, but is related to the generation of the Ai etc. This isn't quite what we need here I think, as this is just about "which subset (or "i") do I choose next). This is just a random (or sequential or whatever) number from 0....N.

To be honest, I find the STIR naming conventions, though very descriptive, ~a bit~ overwhelming.

they can be... but maybe this one isn't so bad. Anyway, feel free to propose an alternative 😄

I wonder if we can agree on how the user sets the sampling strategies, as these are also useful to SPDHG, see #1345 (comment).

Come to think of it, I think it might be better to isolate the "subset number generator" in a separate class. At some point, you'd want to use non-uniform sampling (in fact, already used in the spdhg paper if the penalty is a "subset" I believe). You don't really want people to have to modify SubsetSumFunction to enter their favorite sampler. It's probably also clearer if we just accept a Python discrete RNG in the constructor.

Yes, this is what we discussed in the previous meeting. The new class (suggested name: SubsetGenerator) will implement the next_subset method. And the new next_subset method will have only
self.subset_num = SubsetGenerator ( num_subset, `sampling_method`)

I'm afraid I now also have some doubts about the name SubsetSumFunction (!), and therefore the use of "subset" everywhere. As far as these algorithms goes, they are just a sum of functions, we just happen to use them for subsets. (Again, the example being of the penalty term is a subset). In typical STIR fashion, I'd call this SumFunctionWithApproximateGradient or so... Luckily, it's a base class that nobody will ever use...

updated the link in my comment, which now points to #1344 (comment) which describes a "subset number generator" class, in fact.

that class is somewhat different. It points to a "all angles in all subset generator". That is something useful as well, but is related to the generation of the Ai etc. This isn't quite what we need here I think, as this is just about "which subset (or "i") do I choose next). This is just a random (or sequential or whatever) number from 0....N.

Yes,

updated the link in my comment, which now points to #1344 (comment) which describes a "subset number generator" class, in fact.

that class is somewhat different. It points to a "all angles in all subset generator". That is something useful as well, but is related to the generation of the Ai etc. This isn't quite what we need here I think, as this is just about "which subset (or "i") do I choose next). This is just a random (or sequential or whatever) number from 0....N.

Yes, the class in #1344 (comment) will be used to generate operators (Ai) and data (di) and is prior to running the algorithm. The next_subset method which has the two basic strategies (replacement=True or False) is related to which function (fi) is going to be selected and how in every iteration/epoch.

Why closing this?

Why closing this?

I reopened it

this is what we discussed in the previous meeting. The new class (suggested name: SubsetGenerator) will implement the next_subset method. And the new next_subset method will have only
self.subset_num = SubsetGenerator ( num_subset, `sampling_method`)

I think this is slightly wrong. I suggest something like
generator =  SubsetGenerator ( num_subsets, `sampling_method`)
subset_func = SGFunction(fi_cil, generator)
...

# somewhere in SGFunction
 subset_num = self.subset_num_generator()

@KrisThielemans , @paskino is the following ok?
import numpy as np

class SubsetGenerator(object):

    def __init__(self, num_subsets, sampling_method = "random", replacement = True, shuffle ="random"):

        self.num_subsets = num_subsets
        self.sampling_method = sampling_method
        self.replacement = replacement
        self.shuffle = shuffle
        self.subset_num = -1
        self.index = 0

        if self.replacement is False:

            # create a list of subsets without replacement, first permutation
            self.list_of_subsets = np.random.choice(range(self.num_subsets),self.num_subsets, replace=False)

            if self.shuffle not in ["random","single"]:
                raise NotImplementedError("Only {} and {} are implemented for the replacement=False case.".format("random","single"))

    def next_subset(self):

            if self.sampling_method=="random" :

                if self.replacement is False:

                    # list of subsets already permuted
                    self.subset_num = self.list_of_subsets[self.index]
                    self.index+=1

                    if self.index == self.num_subsets:
                        self.index=0

                        # For random shuffle, at the end of each epoch, we permute the list again
                        if self.shuffle=="random":
                            self.list_of_subsets = np.random.choice(range(self.num_subsets),self.num_subsets, replace=False)
                else:

                    # at each iteration (not epoch) a subset is randomly selected
                    self.subset_num = np.random.randint(0, self.num_subsets)

            elif self.sampling_method=="sequential":

                if self.subset_num + 1 >= self.num_subsets:
                    self.subset_num = 0
                else:
                    self.subset_num += 1
            else:
                raise NotImplementedError("Only {} and {} are implemented at the moment.".format("random","sequential"))

            return self.subset_num

and is used as
generator = SubsetGenerator(self.n_subsets)
self.F_SG = SGFunction(self.fi_cil, generator)

and the next_subset method of SubsetSumFunction is
    def next_subset(self):

        # Select the next subset using the subset_generator

        self.subset_num = self.subset_generator.next_subset()
        self.subsets_used.append(self.subset_num)

No problem to make it function/generator, something like
def subset_num_generator(num_subsets, sampling_method = "random", replacement = True, shuffle ="random"):

        num_subsets = num_subsets
        sampling_method = sampling_method
        replacement = replacement
        shuffle = shuffle
        subset_num = -1
        index = 0

        if replacement is False:

            # create a list of subsets without replacement, first permutation
            list_of_subsets = np.random.choice(range(num_subsets),num_subsets, replace=False)

            if shuffle not in ["random","single"]:
                raise NotImplementedError("Only {} and {} are implemented for the replacement=False case.".format("random","single"))

        while(True):

            if sampling_method=="random" :

                if replacement is False:

                    # list of subsets already permuted
                    subset_num = list_of_subsets[index]
                    index+=1

                    if index == num_subsets:
                        index=0

                        # For random shuffle, at the end of each epoch, we permute the list again
                        if shuffle=="random":
                            list_of_subsets = np.random.choice(range(num_subsets),num_subsets, replace=False)
                else:

                    # at each iteration (not epoch) a subset is randomly selected
                    subset_num = np.random.randint(0, num_subsets)

            elif sampling_method=="sequential":

                if subset_num + 1 >= num_subsets:
                    subset_num = 0
                else:
                    subset_num += 1
            else:
                raise NotImplementedError("Only {} and {} are implemented at the moment.".format("random","sequential"))

            yield subset_num

    sng = subset_num_generator(10, replacement=False)
    for i in range(20):

        if i>=1 and np.mod(i,10)==0:
            print("new epoch")
            print(next(sng))
        else:
            print(next(sng))

4
0
1
6
8
2
9
7
5
3
new epoch
5
9
4
8
7
0
1
3
6
2

@paskino is this ok? Where should we add this function?At the moment, I have created a new script inside Functions directory.

@KrisThielemans, @paskino, @jakobsj @zeljkozeljko
Changes after Hackathon meeting (17/10/2022) and CIL-Dev meeting (18/10/2022):

A utilities directory is added in the cil.optimisation that contains the FunctionNumberGenerator.py

Accepted/Implemented sampling_method for the FunctionNumberGenerator.py are : random, random_permutation, fixed_permutation.

random: every function is selected randomly, with replacement.

fng = FunctionNumberGenerator(10, sampling_method="random")
generated_numbers=[print(next(fng),end=' ') for _ in range(2*10)] # epochs = 2

0 8 3 0 3 0 1 0 9 6 0 8 6 4 0 3 6 0 1 7

random_permutation: every function is selected randomly without replacement. At the end of each epoch, i.e., after all the functions have been selected, the list of functions is permuted again.

fng = FunctionNumberGenerator(10, sampling_method="random_permutation")
generated_numbers=[print(next(fng),end=' ') for _ in range(2*10)]

8 1 2 6 9 3 7 0 4 5 0 6 1 9 3 8 5 2 7 4

fixed_permutation: the list of functions is permuted one time at the beginning (without replacement) and is used for all epochs

fng = FunctionNumberGenerator(10, sampling_method="fixed_permutation")
generated_numbers=[print(next(fng),end=' ') for _ in range(2*10)]

5 6 4 1 0 9 3 7 8 2 5 6 4 1 0 9 3 7 8 2

SubsetSumFunction is remaned to ApproximateGradientSumFunction

ApproximateGradientSumFunction can accept either a string for the selection of functions or instance of the FunctionNumberGenerator

The subset term is removed from ApproximateGradientSumFunction and SGFunction

Unittests for ApproximateGradientSumFunction and SGFunction are updated

Example code for the LS problem
import numpy as np
from cil.optimisation.algorithms import GD
from cil.optimisation.functions import ApproximateGradientSumFunction, LeastSquares, SGFunction
from cil.optimisation.utilities import FunctionNumberGenerator
from cil.optimisation.operators import MatrixOperator
from cil.framework import VectorData

# create LS problem (Ax = b)
np.random.seed(10)
n = 50
m = 500
n_subsets = 10

Anp = np.random.uniform(0,1, (m, n)).astype('float32')
xnp = np.random.uniform(0,1, (n,)).astype('float32')
bnp = Anp.dot(xnp)

Ai = np.vsplit(Anp, n_subsets)
bi = [bnp[i:i+n] for i in range(0, m, n)]

Aop = MatrixOperator(Anp)
bop = VectorData(bnp)
ig = Aop.domain
x_cil = ig.allocate('random')

fi_cil = []
for i in range(n_subsets):
    Ai_cil = MatrixOperator(Ai[i])
    bi_cil = VectorData(bi[i])
    fi_cil.append(LeastSquares(Ai_cil, bi_cil, c=1.0))

F_SG = SGFunction(fi_cil)  # default selection = "random"

# or
# F_SG = SGFunction(fi_cil, selection="random_permutation")

# or
# generator = FunctionNumberGenerator(n_subsets, sampling_method="fixed_permutation")
# F_SG = SGFunction(fi_cil, generator)

step_size = 1./F_SG.L
initial = ig.allocate()
epochs = 200
sgd = GD(initial = initial, objective_function = F_SG, step_size = step_size,
            max_iteration = epochs * n_subsets,
            update_objective_interval = 10*n_subsets)
sgd.run(verbose=1)

print("Function selected")
k=0
for i in range(0, epochs, 20):
    print("Epoch : {}, function used : {}".format(i, F_SG.functions_used[k:n_subsets+k]))
    k+=n_subsets

# import cvxpy
# u_cvxpy = cvxpy.Variable(ig.shape[0])
# objective = cvxpy.Minimize( 0.5*cvxpy.sum_squares(Aop.A @ u_cvxpy - bop.array))
# p = cvxpy.Problem(objective)
# p.solve(verbose=True, solver=cvxpy.SCS, eps=1e-4)

# np.testing.assert_allclose(p.value, sgd.objective[-1], atol=1e-1)
# np.testing.assert_allclose(u_cvxpy.value, sgd.solution.array, atol=1e-1)

@zeljkozeljko  thank you for your quick comments. In my latest commit, I have implemented as you suggested but I changed  them again today after the discussion we had with @paskino, @jakobsj. I have not spent time fixing docs.
@KrisThielemans , @paskino, @jakobsj @zeljkozeljko before starting reviewing this PR, could we agree on the following:

Name and signature of the FunctionNumberGenerator.
Name and signature of the ApproximateGradientSumFunction.

Thank you @KrisThielemans for your comments, before addressing your comments and @zeljkozeljko, I think we need to have a final  decision for the design. Please have a look #1377.

You might consider using the new interface of numpy.random which uses (quite confusingly named) Generators, see https://numpy.org/doc/stable/reference/random/index.html
rng = np.random.default_rng(optional_seed)
rints = rng.integers(low=0, high=10, size=3)

it would allow passing a seed such that the generated sequence is reproducible. (cannot do that with random.choice as you don't know who's going to call that as well). Of course, that could be done later.

Recording part of an email conversation here on adding capabilities for non-uniform random sampling. @epapoutsellis had a proposal but noted that algorithms might need to ask what those probablities are. I think that leads back to the need of having a class. At that point, a "generator" is probably not useful anymore and we can revert to having an iterator interface
def IndexGenerator:
  def __init__(lots of arguments here)
  def get_probabilities(self)
  def uses_replacement(self)
  # either this is an iteratable object
  def iter(self) # could return a generator as far as I know
  # or it is already an iterator
  def __iter__(self)
  def __next__(self)

and we’d have to decide to use a string+extra arguments
'''python
ig = IndexGenerator(num_functions, “random”, probabilities=[.1, .3, .2, …])
gen = ig.iter()
next(gen)
or use a hierarchy of derived classes, e.g.
```python
  def RandomIndexGenerator: IndexGenerator
     def __init__ (either_num_functions_or_list_of_probabilities, replacement=True)
     ...

 ig = RandomIndexGenerator (5)

where I think the latter design is more easily extendable.

@KrisThielemans @paskino @jakobsj  see below my proposal. I created a base class RandomSampling that creates random indices or batches (e.g. mini-batch SGD). Depending on the batch_size, we have control of which of the RandomIndex or RandomBatch classes will be created. The signature remains the same for these two classes and the __next__ method is different.  I also included some static methods in the base class that work for different batch_sizes. I guess for the names we need to decide. I have no objection.
import numpy as np
import random

class RandomSampling():

    def __new__(cls, num_indices, batch_size=1, prob = None, replace = True, shuffle=False, seed = None):

        cls.batch_size = batch_size

        if cls.batch_size == 1:
            return super(RandomSampling, cls).__new__(RandomIndex)
        else:
            return super(RandomSampling, cls).__new__(RandomBatch)

    def __init__(self, num_indices, batch_size=1, prob = None, replace = True, shuffle=False, seed = None ):

        self.num_indices = num_indices
        self.batch_size = batch_size
        self.equal_size_batches = self.num_indices%self.batch_size==0
        if self.equal_size_batches:
            self.num_batches = self.num_indices//self.batch_size
        else:
            self.num_batches = (self.num_indices//self.batch_size)+1
        self.prob = prob
        self.replace = replace
        self.shuffle = shuffle
        self.indices_used = []
        self.index = 0
        np.random.seed(seed)

        if self.replace is False:
            self.list_of_indices = np.random.choice(num_indices, size=self.num_indices, p=prob, replace=False)
        else:
            if shuffle is True and self.batch_size==1:
                raise ValueError("Shuffle is used only with replace=False")
            self.list_of_indices = np.random.choice(num_indices, size=self.num_indices, p=prob, replace=True)

        if self.batch_size>1:
            self.partition_list = [self.list_of_indices[i:i + self.batch_size] for i in range(0, self.num_indices, self.batch_size)]

    def show_epochs(self, epochs):

        total_its = epochs * self.num_indices
        for _ in range(total_its):
            next(self)

        if self.batch_size==1:
            k = 0
            for i in range(epochs):
                print(" Epoch : {}, indices used : {} ".format(i, self.indices_used[k:k+self.num_indices]))
                k+=self.num_indices
            print("")
        else:
            k=0
            for i in range(epochs):
                print(" Epoch : {}, batchs used : {} ".format(i, self.indices_used[k:k+num_batches]))
                k += num_batches

    @staticmethod
    def uniform(num_indices, batch_size = 1, seed=None):
        return RandomSampling(num_indices,  batch_size=batch_size, prob=None, replace = True, shuffle = False, seed = seed)

    @staticmethod
    def non_uniform(num_indices, prob, batch_size = 1, seed=None):
        return RandomSampling(num_indices, batch_size=batch_size, replace = True, prob=prob, shuffle = False, seed = seed)

    @staticmethod
    def uniform_no_replacement(num_indices, batch_size = 1, shuffle=False, seed=None):
        return RandomSampling(num_indices, batch_size=batch_size, prob=None, replace = False, shuffle = shuffle, seed = seed)

    @staticmethod
    def non_uniform_no_replacement(num_indices, prob, batch_size = 1, shuffle=False, seed=None):
        return RandomSampling(num_indices, batch_size=batch_size, prob=prob, replace = False, shuffle = shuffle, seed = seed)

    @staticmethod
    def single_shuffle(num_indices, batch_size = 1, seed=None):
        return RandomSampling(num_indices, batch_size = batch_size, replace = False, shuffle = False, seed = seed)

    @staticmethod
    def random_shuffle(num_indices, batch_size = 1, seed=None):
        return RandomSampling(num_indices, batch_size = batch_size, replace = False, shuffle = True, seed = seed)

class RandomBatch(RandomSampling):

    def __next__(self):

        tmp = list(self.partition_list[self.index])
        self.indices_used.append(tmp)
        self.index+=1

        if self.index==len(self.partition_list):
            self.index=0

            if self.shuffle is True:
                self.list_of_indices = np.random.choice(self.num_indices, size=self.num_indices, p=self.prob, replace=self.replace)
                self.partition_list = [self.list_of_indices[i:i + self.batch_size] for i in range(0, self.num_indices, self.batch_size)]

        return tmp

class RandomIndex(RandomSampling):

    def __next__(self):

        if self.replace is False:

            index_num = self.list_of_indices[self.index]
            self.index+=1

            if self.index == self.num_indices:
                self.index = 0
                if self.shuffle is True:
                    self.list_of_indices = np.random.choice(self.num_indices, size=self.num_indices, p=self.prob, replace=False)
        else:

            index_num = np.random.choice(self.num_indices, size=1, p=self.prob, replace=True).item()

        self.indices_used.append(index_num)

        return index_num

Examples
Our users can do, in the case of batch_size=1
rs = RandomSampling.uniform(10)
rs.show_epochs(epochs=3)

 Epoch : 0, indices used : [8, 1, 4, 8, 5, 9, 1, 9, 0, 3]
 Epoch : 1, indices used : [3, 4, 9, 2, 4, 0, 7, 9, 4, 1]
 Epoch : 2, indices used : [9, 6, 6, 8, 9, 6, 2, 0, 1, 5]

rs = RandomSampling.random_shuffle(10, seed=100)
rs.show_epochs(epochs=3)

 Epoch : 0, indices used : [7, 6, 1, 5, 4, 2, 0, 3, 9, 8]
 Epoch : 1, indices used : [7, 8, 5, 3, 4, 6, 0, 1, 9, 2]
 Epoch : 2, indices used : [9, 7, 1, 8, 5, 4, 3, 2, 6, 0]

tmp = [random.random() for i in range(10)]
tmp_sum = sum(tmp)
prob = [i/tmp_sum for i in tmp]
rs = RandomSampling.non_uniform(10, prob=prob)
rs.show_epochs(epochs=3)

 Epoch : 0, indices used : [2, 5, 7, 7, 2, 1, 6, 6, 2, 7]
 Epoch : 1, indices used : [7, 4, 9, 9, 7, 1, 6, 1, 4, 9]
 Epoch : 2, indices used : [6, 5, 6, 2, 5, 2, 8, 2, 2, 2]

And for the case where batch_size>1
rs = RandomSampling.uniform(10, batch_size=5)
rs.show_epochs(epochs=3)

 Epoch : 0, batchs used : [[5, 7, 9, 5, 9], [1, 4, 6, 9, 6]]
 Epoch : 1, batchs used : [[5, 7, 9, 5, 9], [1, 4, 6, 9, 6]]
 Epoch : 2, batchs used : [[5, 7, 9, 5, 9], [1, 4, 6, 9, 6]]

rs = RandomSampling.single_shuffle(10, batch_size=2, seed=100)
rs.show_epochs(epochs=3)

 Epoch : 0, batchs used : [[7, 6], [1, 5], [4, 2], [0, 3], [9, 8]]
 Epoch : 1, batchs used : [[7, 6], [1, 5], [4, 2], [0, 3], [9, 8]]
 Epoch : 2, batchs used : [[7, 6], [1, 5], [4, 2], [0, 3], [9, 8]]

Comparison ISTA, proxSGD, proxMBSGD
For the 3D walnut dataset , I reconstructed one 2D slice with
 Acquisition Geometry 2D: (1601, 392) with labels ('angle', 'horizontal')
 Image Geometry 2D: (392, 392) with labels ('horizontal_y', 'horizontal_x')

For the $optimal=x^{*}$, I used FISTA with 1000 iterations.

num_subsets=100 and n_epochs=40.
The subsets are generated using AcquisitionGeometrySubsetGenerator with random_permutation.
For both proxSGD and proxMBSGD, I use uniform sampling with replacement and batch_size=5 for the latter.
step_size is the same for both.

rs2 = RandomSampling(len(f_subsets), batch_size=5)
F2 = SGFunctionBase(f_subsets, selection=rs2)

For the proxMBSGD we need to add some lines in the approximate_gradient method as

    if isinstance(self.function_num, int):
        self.functions[self.function_num].gradient(x, out=out)
        out*=self.num_functions
    else:
        tmp = 0
        # function_num is a list of lists, prange???
        for i in self.function_num:
            tmp += self.functions[i].gradient(x)
        out.fill(tmp)
        out *= self.selection.num_batches

Superseded by #1550


Create table statement

System information:

Operating system (distribution) and version
DBeaver version
Latest community

Connection specification:

Database name and version
Poistgres

Describe your question:
Is there a way to generate a create table statement for a table already in the database?

Hi @BobInCT ,
you can do it using:

context menu for the table in the database navigator

table properties - DDL tab

Ahhh, somehow missed that, thanks.


adjusted styles and functionality of removing SVG

fixed SVG preview size
fixed button to upload SVG -- added a label element to control the text
added in confirmation for removing SVG
updated styles of upload and remove button

I'm not certain how to test this myself, but it looks like you incorporated my example and the @tpett PR as well. Will look for the update!


Separate tests from builds that generate artifacts

From @godofredoc on an internal doc on "Engine Goldens + Cherrypick Workflow":

Would it be possible to add a goal to separate tests from the builds generating artifacts?
The context of this request is that builds that generate release artifacts run on secure infrastructure that can only be triggered by new commits and manual runs with multiparty approvals. This would mean that if goldens get blocked on small differences that are manually approved the entire release process can be delayed. If all the tests are separated to its own builders then we would be able to retry from the dashboard without MPAs and huge delays because of the limited resources.
Happy to provide more details if needed.

As of right now, (I believe) @godofredoc is suggesting this should be a blocker to having a cherrypick workflow on the release branch, and I tend to agree with him that without this separation we'd be putting a lot of burden on release engineers.
Anyway, filed as an issue to discuss with the engine/web team. Feel free to chime in.
/cc @zanderso @gaaclarke @jonahwilliams @mdebbar for a start.

Seems reasonable, but what needs to be done here? I don't think we combined the builders intentionally, so much as we started with limited capacity (and tests) and so everything ran on one machine.
Is there a way to build everything on one shard an execute the tests on another (so we can avoid building things twice)?

@christopherfujino added in chat:

My understanding is what is needed here is @keyonghan's graph scheduler, is that right?

I added:

Maybe, but I am hoping we don't build a dependency graph around how to do the underlying work
That is, I am hoping the answer is "Separating the tests manually is not needed, carry on" or "Graph scheduler will take N quarters, so let's separate them manually for now"

The goal here is to separate tests from building artifacts as different builder targets.
Take Mac mac_host_engine for example

Now: the test Host Tests for host_debug is running within the same subbuild which builds the artifact.
After: a new .ci.yaml builder target to be added to focus on running the tests

Some TODO work:

[ ] Infra team

[ ] recipes support to have a way to reference artifacts generated in the build targets
[ ] add docs explaining how to add these targets in .ci.yaml

[ ] Engine team

[ ] separating the tests as a separate target in .ci.yaml

Long term: the graph scheduler should be able to handle the build and test scheduling efficiently enough that no new targets are needed. Say: the scheduler schedules build as target A, and schedules test as target B, without concatenating them inside a single target. This way, we can control rerun of the test separately. (@matanlurey there does exist some overlap considering long term:-))
/cc @godofredoc please correct me if I miss anything.

@keyonghan My understanding is the infra work is blocking the second part, the engine team work?
If so, let's re-assign this to team-engine. Thanks!

IIUC, we can do this today without any recipe changes with the downside that some builds will be duplicated. In particular, builds that both produce artifacts and run tests would be split into two builds. One build that only produces artifacts, and a second build that is only done for the tests. So for example, we'd split mac_host_engine.json into mac_host_engine_artifacts.json and mac_host_engine_tests.json. mac_host_engine_artifacts.json would not run any tests, and mac_host_engine_tests.json would not upload any artifacts.
As for some of these builds being duplicated: (1) The builds that produce artifacts are already not using goma or RBE on the release builders, but the since the builds for testing aren't producing artifacts, they could use goma/RBE, so the additional builds will be much cheaper. (2) The ideas mentioned above for recipe changes could eliminate the duplicate work as an optimization.

If we the capacity to do that, then we can definitely make the changes

but I guess I don't understand how that will help with the problems of reruns being slow on release branches: they'd still end up building all of the artifacts?

Ahh okay, so the problem is not just rerunning the builders, its that the secure builders are slow

Yeah, that's doable, but will build the artifacts twice in separated .json builders.
For release branch: the execution time of the test one will be reduced considering using rbe, but will consume extra (non dart-internal) bots
For main branch: same as release branch, this will consume more capacity for each run
The queue time in our pool has been well within SLO for a while. But this may potentially increase the queue. It would be good we incrementally refactor targets.
(Considering this is not blocked on recipes, routing back to engine team for config refactoring)


[Log in] No text placeholder<EMAIL_ADDRESS>in empty ‘email’ field while login

Environment: Win10 v21H2 64-bit, Chrome v 98.0.4758.82
Reproducible: always
Build found: frontend: gcp-0.0.53, backend: gcp-0.0.112
Preconditions

Go to OoS (https://oos.dmytrominochkin.cloud/#/)
Click on the ‘Увійти/Зареєструватися’ button

Steps to reproduce

Check that ‘Емейл’ field is filled with grey text placeholder<EMAIL_ADDRESS>
Actual result
No grey text placeholder<EMAIL_ADDRESS>
Expected result
Email field is empty with grey text<EMAIL_ADDRESS>by default
User story and test case links
User stories #213, #45
Test case OS-302
Labels to be added
"Bug", "pri: very low", "severity: trivial", Type "UI"

@Olya-web7  Verified, closed


IDE autocomplete

How does one make IDE understand Tpetry\PostgresqlEnhanced\Query\Builder methods?
:)

This is something I am unhappy with for a long time. But I don't know how to fix it 🤷♂
Recently, Laravel Idea shared that they will support this with their next update [0]. But I don't know any approach for VSCode or PHP Storm without the plugin.
[0] https://x.com/laravel_idea/status/1755279230210904084?s=20

I've added a .phpstorm.meta.php file to my project root with the following contents:
<?php

declare(strict_types=1);

namespace Illuminate\Database\Eloquent {
    /**
     * @mixin \Tpetry\PostgresqlEnhanced\Query\Builder
     */
    class Builder {}
}

namespace PHPSTORM_META {
    //metadata directives
}

Hmm, but that file would still need to be copied. And potentially conflicts with other packages wanting to do the same. Would be awesome if it could be at the root dir of every package in the vendor dir. So, every package could provide their own always-updated definitions.

Yup, this isn't a solution for packages and def feels hacky, but it's worked well for me. Be good when Laravel Idea supports this, tho!

@vatsake @boris-glumpler Are you using PhpStorm with Laravel Idea? In that case you could test the open Pull Request which should do a full integration with auto-suggestions and jump-to-code.

Yup, I do. I've just finished a hike in a pretty remote part of Nepal, tho, and it'll take me 2-3 days to get back to my laptop. Will try it out then.

Ok, then I'll test on the weekend again.

Autocomplete for PhpStorm with Laravel Idea plugin is now live with 0.35.0.
Other IDEs or PhpStorm without the plugin are not supported as there are no ways to do this currently. But I've requested the Laravel IDE helper package to make this possible in the future with a generic way: barryvdh/laravel-ide-helper#1529


If possible, deploy EKS pods to a private subnet, not public

We want to deploy our pods to a private subnet if possible. According to AWS, it seems that the instances in a public subnet can send outbound traffic directly to the internet, whereas the instances in a private subnet can't. Currently, we deploy our pods to a public subnet because we want Internet users to be able to reach the reverse proxy pods, web server pods, and database proxy pods. However, having the entire EKS cluster in public subnets exposes many other services with which we do not intend users to interact.
This AWS EKS networking guide states that only the load balancer need be in a public subnet; all other nodes can be in private subnets.

This is a good idea, but it does require the implementation of significant infrastructure to support. Because instances in a private subnet cannot reach the Internet, all configuration must happen before any resources are deployed. In particular, all apps must be containerized (which they already are in our setup), and those containers must be stored in an AWS container registry. In addition, we would also need to mirror certain packages or find other ways to make them installable, e.g., when we start the Wireguard EC2 node, it needs to apt install docker.io and its dependencies.
I am moving this to second release as possible future work.

Based on some research, I believe that the following microservices should be in the public subnets:

Web app
Database proxy
Reverse proxy
VPN

The reason is that the Flutter web app does not (and, as far as I can tell, cannot) perform any server-side computation. Flask apps perform server-side computation, so if I had used Flask for the web server, I could have made the request on the database without needing a proxy. Since Flutter cannot do that, I need a database proxy that a user can reach to execute code that will interface with the database from within the VPC. The reverse proxy is also necessary so that both the web app and the database proxy can be accessed from a single load balancer URL--the database proxy Kubernetes Service could have been a LoadBalancer Service instead, but then we would be incurring extra costs and have another entry point into our VPC. Finally, the VPN needs to be in a public subnet so that admins can reach it from the Internet.
All other microservices, including Prometheus and the SNS notifier, can be in private subnets.

In the eks-managed-node-group submodule, you can specify subnet IDs and node taints, e.g., "public" or "private". Then you could force pods to deploy to nodes with specific taints.


utils: add tidyAsync() which allows the fn to return a Promise.

The tidy() function disallow async function, which makes memory management with some asynchronous operations not too friendly.
But we still have to use this method with caution, because tfjs has also been implemented before, but it was later disabled in the following commit: https://github.com/tensorflow/tfjs/commit/5acd02003c3a00256f495ced52a0c1c85bde52fc.

Related PR from deeplearn(tfjs): https://github.com/tensorflow/tfjs-core/pull/763.

There is still an issue when multiple tidy functions are created.

Added a state to avoid multiple tidies are running, which is buggy, this might be the reason that tfjs disallow asynchronous calls, @jan-www @FeelyChau would you please have a look again?

Call tidy() after tidyAsync() just like:
tidyAsync(() => {
  // ....
  tidy(() => { ... });
  // ...
});

All variables are stack-based without side effects, however tidyAsync() still has no possibility to be running concurrently @FeelyChau @jan-www.


Errors when using V2 API with new or old Google Maps API keys

This script no longer works as-is with the existing V2 API (either the old API key produces a "development" version or the new API key produces errors when loading the pages.

Control is no longer supported in the Google Maps Javascript API v2. Please visit https://developers.google.com/maps/documentation/javascript/v2/v2tov3 to migrate your application to v3.

Using a new API Key provided by Google sometimes results in the following (and sometimes works, depending on the key)
Uncaught RangeError: Maximum call stack size exceeded
at common.js:1
at Object..yb (maps?file=api&v=2&key=key=AIzaSyDoak-H2Ga82-X7e3qkLAjfKwdEl7E8EA0:39)
at a (common.js:54)
at common.js:54
at Object..yb (maps?file=api&v=2&key=key=AIzaSyDoak-H2Ga82-X7e3qkLAjfKwdEl7E8EA0:39)
at a (common.js:54)
at common.js:54
at Object._.yb (maps?file=api&v=2&key=key=AIzaSyDoak-H2Ga82-X7e3qkLAjfKwdEl7E8EA0:39)
at a (common.js:54)
at common.js:54
It might be worth upgrading to the V3 API to resolve these issues, but these might also be a bug in the Google Maps API.

Upgraded to Google Maps v3 and the problems are resolved.


Allow setting refresh on datasource variables

This is a direct copy of the same feature on query variables.
Not setting refresh on a datasource variable that uses another
variable results in a dashboard that fails to load unless you
"re-select" the variable in question.

Hej,
From what I see in Grafana (v7.5.x here), there isn't a way to specify when to refresh "datasource" variables:

I think we shouldn't expose a  Refresh option for Datasource variables but instead only set the correct refresh value on the builder. Would you be willing to update your PR with that in mind? :)

Sure, I can do that. I'll try to look at it later today.

Merged, thanks :muscle:


When will the code and models be released?

I would like to also show my interest in this project. Looking forward to seeing what it can do.


Have global exports be compatible with Web Workers

Assigning to window only works in a normal browser environment; however, Web Workers don't have access to window. Instead, the global object inside Web Workers is self.
Browserify already ensures that global is present and dereferenced accordingly. It will equal to window unless self is defined.
This fixes Web Workers compatibility lost in a81e5558f086808b682b6d196b80893d2c8926f8 (v2.3.0).
/cc @jbnicolai @ndhoule

test/browser/index.html is failing now.

@dasilvacontin If you're talking about this failure, the same error is present on master:

It's not clear to me whether this error is expected or not. The browser test suite might have fallen out of date since it doesn't seem exercised under CI.

test/browser/index.html was fixed in the latest patch release. Confirmed that this PR fixes web worker compatibility. Thanks! :)


First word sometimes cut/incomplete

I noticed, that some generated prompts have a cut/incomplete first word, the first 2 characters are missing.
For example:
rait of julia garner as kate beckinsale wearing ...
rait of a man by Greg Rutkowski, wearing ...
ometric game asset on an island, ...
azing, hyperdetailed portrait ...
mmetry portrait of a woman in ...
mm photograph of a very cute ...
And it seems this happens only when the prompt is initially empty.
Not that big issue but sometimes it's overlooked when copying the generated prompt.

No one cares to check this ... ?

@0lm
I mostly use the promptgen-lexart, and it doesn't matter if I use the buildin or the external version.
I also have the gpt2-medium-650k-stable-diffusion-prompt-generator and promptgen-majinai-unsafe, but the two generate strange prompts of their own kind. :)


[QST] GDS, dlpack and cuCIM

Hi  cuCIM team,
I am deploying RAPIDS with cuCIM v21.06  on ppc64le architecture (Summit cluster) and in our NV-ARM Developer Kit cluster, and noted during compilation, it grabs GDS. So my questions are:
Is GDS a hard dependency ?, if not, is there any env. variables to disable it ?
Thanks,

Thanks @benjha for your interest in cuCIM!

Is GDS a hard dependency ?, if not, is there any env. variables to disable it ?

GDS is not a hard dependency.
GDS is already disabled for now and it would work in the compatible mode (has the same behavior but without libcufile (GDS)).
https://github.com/rapidsai/cucim/blob/45cf987e17e22acd84165e3dd3d6bd5ae72e19cd/CMakeLists.txt#L106
The current implementation uses a wrapper code for cufile and it load libcufile.so dynamically if libcufile is available (and libcufile.so won't be accessed if CUCIM_SUPPORT_GDS is OFF).
https://github.com/rapidsai/cucim/blob/048d53e6f99c2129b9febd08e0ae6d1b37d74451/gds/src/cufile_stub.cpp#L55-L59
In current implementation, even if GDS (libcufile.so) is not available/needed, GDS's header file (cufile.h) is needed:
https://github.com/rapidsai/cucim/blob/048d53e6f99c2129b9febd08e0ae6d1b37d74451/gds/include/cufile_stub.h#L19
To get cufile.h (for both with GDS and without GDS), a GDS package (v0.95 which is old now) is downloaded from the internet and used during building (https://github.com/gigony/cucim/blob/python_unittests/CONTRIBUTING.md#local-development-using-conda-environment-for-gcc-9x-and-nvcc-110x):
https://github.com/rapidsai/cucim/blob/45cf987e17e22acd84165e3dd3d6bd5ae72e19cd/run#L392
For this reason, I think it wouldn't be a problem though you will see GDS-related messages during the build.
cuCIM currently doesn't support non-X86-64 architecture(and non-Linux OS) and has not tested on such architecture/OS. Please let us know what kinds of errors do you see during the build.
We are also interested in your use cases with cuCIM in your project and wonder if which part (image loader or image processing) would be your focus area.
Thanks.
Note: Image processing part (scikit-image compatible APIs using CuPy) is pure Python code and no platform-dependent as long as CuPy is installed. We may want to provide an option to build only one of them (cucim.skimage and cucim.clara).

Thanks for the clarification. I agree, if we disable access to libcufile.so we'll be in the safe place.
I am supporting Rapids on Summit and I'd like to include cuCIM as part of the deployment. We have some users currently working with scikit-image and cuCIM will be a good addition so they can move their workflows to GPUs. Of course in the context of a HPC system while I/O is important for now we are focusing on image processing. I am also evaluating Rapids performance on ARM architecture, just to track performance is comparable to x86/ppc64le.
We also are evaluating image processing libraries in the context of Whole slide imaging in pathology. This will also include training/inference of tumor sites in a subsequent phase.

Btw,
Is cuCIM not meant to be build within an existent  built from source Rapids environment  ?
I noted cuCIM gets its own rmm and also uses its own dlpack (v0.2) which is different to the Rapids 21.06's supported version (dlpack>=0.5,<0.6.0a0), thus I am having this compilation error given that DLContext does not exist in dlpack 0.5
/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/ifd.cpp:262:25: error: 'struct DLTensor' has no member named 'ctx'
  262 |     out_image_container.ctx = DLContext{ static_cast<DLDeviceType>(out_device.type()), out_device.index() };
      |                         ^~~
/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/ifd.cpp:262:31: error: 'DLContext' was not declared in this scope; did you mean 'ucontext'?
  262 |     out_image_container.ctx = DLContext{ static_cast<DLDeviceType>(out_device.type()), out_device.index() };
      |                               ^~~~~~~~~
      |                               ucontext
/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/tiff.cpp: In member function 'bool cuslide::tiff::TIFF::read_associated_image(const cucim::io::format::ImageMetadataDesc*, const cucim::io::format::ImageReaderRegionRequestDesc*, cucim::io::format::ImageDataDesc*, cucim::io::format::ImageMetadataDesc*)':
/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/tiff.cpp:718:25: error: 'struct DLTensor' has no member named 'ctx'
  718 |     out_image_container.ctx = DLContext{ static_cast<DLDeviceType>(out_device.type()), out_device.index() };
      |                         ^~~
/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/tiff.cpp:718:31: error: 'DLContext' was not declared in this scope; did you mean 'ucontext'?
  718 |     out_image_container.ctx = DLContext{ static_cast<DLDeviceType>(out_device.type()), out_device.index() };

Thanks

Note: Image processing part (scikit-image compatible APIs using CuPy) is pure Python code and no platform-dependent as long as CuPy is installed. We may want to provide an option to build only one of them (cucim.skimage and cucim.clara).

I think this is already possible, we just don't have a pre-packaged version with only the cucim.skimage component. If you check out the repository, change to the python/cucim folder and run pip install . -v it should install the Python package, but without any of the compiled binaries. Thus cucim.clara would be unavailable, but cucim.skimage should still work since all compilation there is currently at runtime through CuPy. CuPy>=9 would be required at run time, but no other RAPIDS packages would be.
This currently seems to work, but I have not confirmed with the NVIDIA team whether we can continue to guarantee this going forward (we do not currently test this installation method)

@benjha Thanks for sharing the use cases!

Is cuCIM not meant to be build within an existent built from source Rapids environment ?

That's correct. Currently, cuCIM's build doesn't depend on the same version of other RAPIDS components (and other RAPIDS suites) -- That's why it uses specific versions of rmm and DLPack from external Git repositories.
Please refer to the following documents for building cuCIM locally:
https://github.com/gigony/cucim/blob/python_unittests/CONTRIBUTING.md
And thanks for the information regarding rmm/DLPack versions. I would update those versions later.

Closing as answered. Happy to discuss more questions in new issues as needed


Release 1.22

This will include all the post-1.21 features, like Logged Out views.

          2 Warnings

:warning:
Big PR

:warning:
Please provide a linked tracker story for this PR in the description

  Generated by :no_entry_sign: danger


Double nested theme property in tailwind.config.js

There's an issue with the tailwind.config.js file, the theme property is nested inside another theme property. This way customisations won't work.

PR welcome!


[CT-248] Implement retryable errors for Spark/Databricks

Describe the bug
When using dbt-spark, we randomly but somewhat frequently experience some errors that could be handled by detecting them as retryable errors.
Based on our logs, I believe at least the following errors could be considered retryable. The models usually always runs successfully the next time they're run.
They're listed in order from most frequently experienced to least:

HiveException: at least one column must be specified for the table
HiveException: Unable to alter table
A 503 response was returned but no Retry-After header was provided
Connection failed with error: Bad Status: No status code
ProtocolChangedException: The protocol version of the Delta table has been changed by a concurrent update. Please try the operation again

Ideally one could check whether the error message contains any of the above strings, and if so retry the query a number of times like in the BigQuery implementation - see comment from jtcohen6 linking to the implementation for BigQuery:
https://github.com/dbt-msft/dbt-sqlserver/issues/119#issuecomment-784911176
Steps To Reproduce
Seeing that these issues happen randomly, it's difficult to list a set of steps that will consistently reproduce the issues.
Expected behavior
If any of the listed errors happen, the connector should retry the given model X number of times, where ideally X is defined in the profile like for the BigQuery adapter (https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile/#retries).
Screenshots and log output
N/A
System information
The output of dbt --version:
installed version: 1.0.1
   latest version: 1.0.1

Up to date!

Plugins:
  - spark: 1.0.0

The operating system you're using:
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
The output of python --version:
Python 3.9.10
Additional context
N/A

Hey @grindheim, is there a reason this issue has been closed? I'm currently using dbt-spark and the lack of retry logic is definitely something I'd like to see addressed so wondering if there has been a discussion elsewhere that resulted in the closing of this issue.

@pgoslatara No, there was just no feedback at all for 26 days, so figured I'd just close it. But reopening it now.

Thanks @grindheim! I'd really like to see this implemented, I'm not sure I have the time or the knowledge to undertake this but keeping this issue open for now may allow someone else jump in with a solution.

@grindheim Thanks for opening the issue, and @pgoslatara thanks for the prompt to keep it open! Apologies for the delay in response from us. We're revamping the way we triage and maintain adapter plugin repositories. I think the topic is a good one; intermittent errors frustrate many users, and implementing retry on the adapter/connection level is the right way to go.
The question is whether these errors are cropping up while:

opening initial connections
executing queries

We already have "naive" retry implemented for initial connection opening, in two different ways, if connect_retries and connect_timeout are set in the connection profile:

A retry_all config, to retry all messages that raise errors (including
A set of retryable messages, to retry even when retry_all: False, which right now has handling only for terminated clusters while they resume:

https://github.com/dbt-labs/dbt-spark/blob/d7f1d38d0dcf272dc0e513db4eeada0c08c407f5/dbt/adapters/spark/connections.py#L534-L543
We don't have any retry implemented during query execution. I think adding a set of retryable errors is a good idea, if we can confirm that they are consistently intermittent for all users. If that guarantee of consistency proves impossible, we could also pursue the approach recommended in https://github.com/dbt-labs/dbt-core/issues/3303, whereby users can "bring their own" retryable error statuses (list of exceptions defined in profiles.yml).


Posibility to use requirements.txt and virtualenv

Hi!
First of all, congrats for this project. It seems pretty complex and the code is very readable.
Regarding the issue, you could add instructions to create a virtualenv and, with that, the use of requirements.txt, which is very practical for testing projects without messing your system installation.

Added Pipenv


Unable to configure ClamAV

It's not possible to configure ClamAV using the documented environment variable:
$ docker run -e CLAMD_SETTINGS_CSV="StreamMaxLength=100M" quay.io/ukhomeofficedigital/clamav:v1.5.0
sed: couldn't open temporary file /usr/local/etc/sedTrhZHX: Permission denied
/docker-entrypoint.sh: line 22: /usr/local/etc/clamd.conf: Permission denied

Due to /usr/local/etc/clamd.conf being owned by root instead of clamav.

Fixed by #36


Division by zero in overlap function

Relevant part:
  File ".../freki/readers/base.py", line 116, in overlap
    return (b.ury - a.lly) / b.height
ZeroDivisionError: float division by zero

This is probably because the line above has if b.height: return 0 instead of if not b.height: return 0

Closing as there is no longer any overlap function.


I can't Optimize the code within the target?

when I ran ic-cdk-optimizer target/wasm32-unknown-unknown/release/_rust_canister_.wasm -o target/wasm32-unknown-unknown/release/_rust_canister_-opt.wasm I got the following error.
thread 'main' panicked at 'Could not read the file.: Os { code: 2, kind: NotFound, message: "No such file or directory" }', /Users/apple/.cargo/registry/src/github.com-1ecc6299db9ec823/ic-cdk-optimizer-0.3.4/src/main.rs:34:27

Correct place to discuss this is here: https://github.com/dfinity/cdk-rs/issues/278


How to deploy the model on a non cuda device

HI,
i have trained the model on a gpu device, but will be deploying the model on a cpu device. The repo talks about compiling some layers on cuda devices only. Since cuda is not available for cpu, how to set up the environment to serve the model on cpu device?

I hope you are talking about pytorch-1.0 branch of this repo.
So, because this branch compile some layers only for cuda by default, you can adapt CPU kernels for other layers manually.
By default, setup.py compile kernels from lib/csrc folder, but there are another kernels (CPU and GPU) in lib/roi_align(_crop, _pooling) and lib/nms folders.

May be late, but If you work with PyTorch 1.x and python 3.x you can try my implementation of this work that full work on CPU.


WIP: Reactive refactoring

See https://github.com/cyclejs/collection/issues/6#issuecomment-225923187 + up xstream to 5.0.5.
The task runner example won't work until cycle HTTP will support current xstream.

I turned some doc snippets to single-backticks from triple-backticks just to overcome markdown-doctest, but there should be a better approach

I re-enabled the examples (by adding the sources stuff to globals in .markdown-doctest-setup.js). I also added a helpful error message to markdown-doctest so that other people should find this easier in future (thanks for the unintentional feedback :D).
This looks really good, and I'm consistently amazed by your quality contributions! I'll have a more detailed look over the code in the next couple of days and leave some feedback. Hopefully by that point HTTP will be updated.

Everything works with new Cycle HTTP, hooray!

This is really elegant, and cuts right down on boilerplate. My only concern is that it might be too magical, but I think we'll find the right balance there through real world use.
A++ PR 💯 👍

Published as v0.2.0 💃


New driver please! :)

Hey, dear developers of this tool.
Thanks a lot for this tool. It's nice for all db-developers.
1 question: can u add driver for connect to TERADATA DB? and how mach of time need for to do  this?
Neliubin Egor
Teradata developer
A1 Belarus
<EMAIL_ADDRESS>

Hey, dear developers of this tool.
Thanks a lot for this tool. It's nice for all db-developers.
1 question: can u add driver for connect to TERADATA DB? and how mach of time need for to do this?
Neliubin Egor
Teradata developer
A1 Belarus
<EMAIL_ADDRESS>
Hello, @mtxr hand is full now, maybe if you can create a connector for this plug in then it will be a great help for SQLTools and Teradata itself. Cheers!

@NEgorE so far I've been trying not to add new drivers to enhance the currently existing ones. The project is growing a lot, and that's awesome, but the worklaod has increased a lot for me to take over all issues.
But I've developed a tool to help developers to quick start in a new drivers. This tool generates some boiler plates stuff where you can start on.
I'll close this issue for to to keep track for the enhancements and bugs.
I'll be glad to merge a PR with this new driver if you are able to work on it. Reach me out if you need any help, I'll be happy to assist.


'show install inactive' encounters timeout

Received command timeout: 'show install inactive' on Fretta (NCS5500).
I tried the pattern manually. The pattern would work in this case without '[\r\n]' in the front.
Will email the full log. Below is part of the condoor.log that shows the issue:
2017-06-14 12:27:31,001 [   DEBUG] condoor.fsm:call_action(23): A=Update driver, config mode and hostname when received an expected prompt.
2017-06-14 12:27:31,001 [   DEBUG] condoor.device:update_driver(410): (Calvados): Prompt: 'RP/0/RP0/CPU0:WB_DCE_1#'
2017-06-14 12:27:31,001 [   DEBUG] condoor.device:driver_name(303): Driver change Calvados -> eXR
2017-06-14 12:27:31,001 [   DEBUG] condoor.device:make_driver(321): Make Device: telnet://root@<IP_ADDRESS>:23 with Driver: eXR
2017-06-14 12:27:31,001 [   DEBUG] condoor.drivers.generic:update_hostname(322): Hostname detected: WB_DCE_1
2017-06-14 12:27:31,002 [   DEBUG] condoor.drivers.generic:make_dynamic_prompt(302): Platform: eXR -> Dynamic prompt: ''[\r\n]RP\/[0-3]\/RS?P0-1?:?WB\_DCE\_1(\([^()]\))?#|[\r\n]RP\/[0-3]\/RS?P0-1?:ios#|[\r\n]rommon \d+ >|[\r\n]XML> ''
2017-06-14 12:27:31,002 [   DEBUG] condoor.drivers.generic:update_config_mode(314): Mode: global
2017-06-14 12:27:31,002 [   DEBUG] condoor.drivers.generic:update_hostname(322): Hostname detected: WB_DCE_1
2017-06-14 12:27:31,002 [   DEBUG] condoor.fsm:run(193): NS=-1,NT=60
2017-06-14 12:27:31,002 [   DEBUG] condoor.fsm:run(203): WAIT-4-STRING Stop at E=2,S=-1
2017-06-14 12:27:31,002 [    INFO] condoor.device:send(240): Command executed successfully: 'exit'
2017-06-14 12:27:31,003 [   DEBUG] condoor.device:send(228): Sending command: 'show install inactive'
2017-06-14 12:27:31,607 [   DEBUG] condoor.drivers.eXR:wait_for_string(74): Expecting: '[\r\n]RP\/[0-3]\/RS?P0-1?:?WB\_DCE\_1(\([^()]\))?#|[\r\n]RP\/[0-3]\/RS?P0-1?:ios#|[\r\n]rommon \d+ >|[\r\n]XML> '
2017-06-14 12:27:31,607 [   DEBUG] condoor.drivers.eXR:wait_for_string(75): Calvados prompt: 'sysadmin-vm:?(?:_node)?[0-3]_RS?P[0-1]:?.*#'
2017-06-14 12:27:31,607 [   DEBUG] condoor.fsm:run(152): WAIT-4-STRING Start
2017-06-14 12:28:31,661 [   DEBUG] condoor.fsm:run(176): E=5,S=0,T=60,RT=60.05
2017-06-14 12:28:31,661 [   DEBUG] condoor.fsm:run(182): A=Exception WB_DCE_1: Timeout waiting for prompt
2017-06-14 12:28:31,662 [   ERROR] condoor.device:execute_command(277): Command timeout: 'show install inactive'

@jamesjwei It must be an old condoor version installed. In 1.0.2 there are no such patterns.


Make emulation base images available

CDK users have reported an issue with the sam build images that causes building to fail if they are using btrfs or zfs storage drivers. The suggested resolution is to update the root mode which can only be done in the base image.
Are those base image Dockerfiles (i.e. public.ecr.aws/sam/emulation-python3.9) available anywhere?

Thanks @corymhall for raising this issue. The emulation images are managed by a different team. I will share this issue with them.

public.ecr.aws/sam/emulation-python3.9 are not managed within this repo but by the Lambda team. I will reach out internally.
Closing as this issues is not related to the build images.


Implement diffing for dirty states [phab:D1783]

Created by djrenren at 2017-07-18 22:04:19:
This change introduces diffing dirty states to allow for realigning of
references. In order to do this, the server maintains a copy of the original
source text and the dirty text. On requests for references and definitions,
the two files are diffed, and all references are given a mapped location.
Mapping happens again for all references located in open files so the reference
site can be realigned

djrenren wrote at 2017-07-18 22:30:43:

Remove dependence on vscode document sync

zarko wrote at 2017-07-19 17:47:59:
Looks like it's working well in VS Code.

zarko commented inline (context lost during migration) at 2017-07-19 17:54:15:
Please remove unnecessary whitespace.

zarko commented inline (context lost during migration) at 2017-07-19 17:54:15:
Please describe what members are used for.

zarko commented inline (context lost during migration) at 2017-07-19 17:54:15:
Long line; did you clang-format?

zarko commented inline (context lost during migration) at 2017-07-19 17:54:15:
Describe what this does (and maybe consider changing its name to be more descriptive; adjustDirtyLocations?)

zarko commented inline (context lost during migration) at 2017-07-19 17:54:15:
Comment?

djrenren wrote at 2017-07-19 18:29:34:

Fix formatting and nits

djrenren wrote at 2017-07-19 23:41:05:

Read dirty text on file open. Handle failed requests

djrenren wrote at 2017-07-19 23:44:14:

Read dirty text on file open. Handle failed requests

djrenren wrote at 2017-07-20 00:25:46:

Prevent double-opening files and ensure diffing on first open

schroederc commented inline (context lost during migration) at 2017-07-20 16:16:45:
Missing space.
Is there auto-formatting for Typescript yet?

djrenren wrote at 2017-07-20 19:58:18:

Fix bug where ref patching abandoned early

schroederc accepted this revision at 2017-07-20 20:03:06

schroederc commented inline (context lost during migration) at 2017-07-20 20:03:51:
Please don't forget to fix up formatting.

djrenren wrote at 2017-07-20 20:08:25:

Sometimes I forget to run format

djrenren closed via commit bc9520d77cd44fd3c76ff3ccbd50ffce3c1d6a9d at 2017-07-20 20:09:00


Console does not behave reasonably if address space is deleted

If I create a address space (brokered or standard) and then attach a browser to the console, then delete the address space, there is not visual indication to the user that the address space has gone.  Furthermore, the buttons in the console remain usable (that is, I can click between connections and addresses tabs), but there is still no indication to user that the address space is gone.
If I then re-create the address space, the console remains in the same non functional state. The user needs to know to logout and login again.
If I check the browser's console, I see messages like:
rhea.js:9323 WebSocket connection to 'wss://console-myproject-brokered.<IP_ADDRESS>.nip.io/websocket' failed: Error in connection establishment: net::ERR_NETWORK_CHANGED
connect @ rhea.js:9323
Connection._connect @ rhea.js:5307
Connection.reconnect @ rhea.js:5300
rhea.js:5531 [connection-1] error: [object Event]

My ideas for this:

on the server side, use a signal handler to gracefully close the websocket connection when the agent is closed.   This should allow the client to discover the loss it peer sooner.
on the client side, use a rhea disconnect handler to trigger the display of a message informing the user that the connection has been lost.  Unfortunately, the client is not going to be able to distinguish between an address space being deleted from a simple agent pod restart, so the message will have to be worded to cover both cases.

@k-wall I think detecting that address space is deleted or pod restart would is hard without tapping into the “global” state. IMO “Client disconnected” or “Connection to server lost” would be fine and a sufficient improvement for the lifetime of the per address space console.


Linting

Should wait for #39 so that all linting issues can be fixed.

I will try to button up #39 tomorrow.


reference: Add "Advanced Tagging" section

As factories and CI stories get more complex, we are finding the need
to introduce more powerful ways to do handle tagging. This change
adds a new document to help explain what's possible. Its based
on the issue defined here:
https://github.com/foundriesio/ci-scripts/issues/44
Signed-off-by: Andy Doan<EMAIL_ADDRESS>

Docs for e5d513ae9ef5b23d5f3ad53ef00f42a60df045ac are browesable at: https://ci.foundries.io/projects/fio-docs/builds/369/docs/artifacts/html/index.html

Rendered page is: https://ci.foundries.io/projects/fio-docs/builds/369/docs/artifacts/html/reference/advanced-tagging.html

Docs for 3ac971be47eeda60a5d89207b471e1d520e19202 are browesable at: https://ci.foundries.io/projects/fio-docs/builds/370/docs/artifacts/html/index.html

Docs for 34b6467bcdcd220ecdb723a40a9320ca94c2c4ed are browesable at: https://ci.foundries.io/projects/fio-docs/builds/371/docs/artifacts/html/index.html

Docs for bf36f7cfe829517d77b3b6db5121a70e3da3f014 are browesable at: https://ci.foundries.io/projects/fio-docs/builds/372/docs/artifacts/html/index.html

@EmbeddedAndroid @ricardosalveti  - just add a new "Scenario 3" to this PR.

Docs for 3c4674b35ca83fd265a5ba2ce0a0b6285854ea33 are browesable at: https://ci.foundries.io/projects/fio-docs/builds/373/docs/artifacts/html/index.html


Improve event debugging experience.

Two things make event debugging hard in Nu -

The chain monad pollutes the call stack really quickly because it's not tail-recursive.

It's too hard to identify from where an event originates when using the Observation arrow.

Fix issue 1 by making the chain monad tail-recursive.
Fix issue 2 by finding a good place to add metadata to subscription entries or somewhere.

#2 fixed for a while. Still trying to recruit @petricek to help me with #1 :)

Remainder of issue moved to here - https://github.com/bryanedds/Prime/issues/3


The most important pages are mobile friendly

As a user with a mobile phone, I want to use TeamRadion in my mobile phone browser to control a stations playlist. I don't need to see the video or hear music, but I want to be able to do anything necessary to participate in managing a playlist.
Acceptance criteria

The most important pages (for mobile use) are mobile friendly:
On the station page, the main video is disabled. Instead, I can access the playlist features (voting, history, favourites etc.) comfortable on a touch device with a small screen
When adding songs, I can preview the new song
I can login with my mobile phone comfortably

On the station page, the main video is disabled. Instead, I can access the playlist features (voting, -
history, favourites etc.) comfortable on a touch device with a small screen :heavy_check_mark:
When adding songs, I can preview the new song :x:
I can login with my mobile phone comfortably :heavy_check_mark:

Testing:

When adding song, I can preview the new song. But I cannot add some of songs with long content preview

On the station page, the main video is disabled. Instead, I can access the playlist features (voting, -
history, favourites etc.) comfortable on a touch device with a small screen :heavy_check_mark:
When adding songs, I can preview the new song :heavy_check_mark:
I can login with my mobile phone comfortably :heavy_check_mark:


Logo - Required dimensions?

Dear Georg,
I am currently migrating from github pages to the hugo academic theme. I totally enjoy the experience. However, when trying to include a logo instead of a title, the image is never displayed. Only a "broken image" icon. I am using a "logo.png" and it is saved in .\static\img
What could I do wrong? Are there any requirements regarding dimension?
Thanks a lot for your amazing theme.
René

See https://github.com/gcushen/hugo-academic/blob/master/.github/support.md .
The repo for the Academic documentation provides an example of using the logo.


Added timestamps to workCommand output.

So you can track progress on the Command line.
Failed: Illuminate\Queue\CallQueuedHandler@call
Processed: Illuminate\Queue\CallQueuedHandler@call
Processed: Illuminate\Queue\CallQueuedHandler@call
Processed: Illuminate\Queue\CallQueuedHandler@call
Processed: Illuminate\Queue\CallQueuedHandler@call
Processed: Illuminate\Queue\CallQueuedHandler@call
turns into
[2015-10-20 11:06:55] Failed: Illuminate\Queue\CallQueuedHandler@call
[2015-10-20 11:06:55] Processed: Illuminate\Queue\CallQueuedHandler@call
[2015-10-20 11:06:56] Processed: Illuminate\Queue\CallQueuedHandler@call
[2015-10-20 11:06:56] Processed: Illuminate\Queue\CallQueuedHandler@call
[2015-10-20 11:06:56] Processed: Illuminate\Queue\CallQueuedHandler@call
[2015-10-20 11:06:57] Processed: Illuminate\Queue\CallQueuedHandler@call
so you can see new lines appearing in the terminal :-)

Please send to 5.2.

Please send as one commit too.

@GrahamCampbell OK :-)  see #10669


[WIP] feat: adding support for reading authoritativeSource

adding support for reading authoritativeSource (camel case)
closes #74

@HassanAkbar I don’t see particular issues with this PR, but what could be the simplest test process for verifying this change?

This is not needed anymore because the issue is fixed in jekyll-geolexica in this PR -> https://github.com/geolexica/jekyll-geolexica/issues/12


Hax contributor updates

<If necessary, assign reviewers that know the area or changes well. Feel free to tag any additional reviewers you see fit.>
Details
Merging the copy edits to Master
Fixed Issues
https://github.com/Expensify/Expensify/issues/154933

I have read the CLA Document and I hereby sign the CLA

recheck


Show workers not working

In the live map when I hit the show workers button in the bottom left the page becomes unresponsive and then because I am in google chrome it crashes and gives me the aw snap error.
I have everything else working except that. I need to try to see where my workers are because it is not scanning very accurately compared to my main scanner app for smaller locations.

You have to many workers, and it takes a lot of effort for javascript to draw all that circles on your map. I had the same problem, and I updated the code a little to draw circles (show workers scanning areas) only for selected workers, check this out - https://github.com/modrzew/pokeminer/pull/100

I only have 9 workers is that too many?

Not at all, with only 9 workers it should work like a charm, for me it takes over 1 minute freeze to generate and display over 100 workers and their areas.
Are you trying to display workers and circles on mobile device? Did you try on other browser?

the number of workers doesnt really matter. Questions is how many points per cycle in total you have. If you have 9 workers with 1000 points per cycle this would be too much for javascript to handle

@Aiyubi That was it. My points per cycle was extremely high now it is about 40 and it worked immediately.
Another question but I might need to make another issue is that the circles are tiny and very far apart. what should I change to either make them closer together or larger circles?

@chromer90 there is no way to make the circles larger - its a limit of the game
and to make them be closer together you need to adjust the lat/long gain. This will give you more points again.
What I did: choose a small area, then adjust the lat/long gain, then increase to the big area.

Thanks that seemed to help alot! I will run it all day today and see if it collects data better. Just need to make more workers now lol


[mmlspark] allow allocating more than int max array

resolves issue https://github.com/microsoft/LightGBM/issues/2615 by introducing long versions of SWIG-generated array
ping @chris-smith-zocdoc @guolinke
validated jar in mmlspark

@imatiach-msft Sorry for the inconvenience, but can you please rebase to the latest master to fix CI checks?

@StrikerRUS done!

Hey @imatiach-msft,
Can you please resolve conflicts that appeared after merging #2850?

@StrikerRUS done!  Resolved conflicts, updated PR.

ping @guolinke @StrikerRUS thanks!

@imatiach-msft
The diff looks nice to me, but unfortunately I have no idea what is going on here, sorry! 😃
static TYPE *new_##NAME
%{}
...

@StrikerRUS this is just a slightly modified version of the SWIG file here which we were already using in the previous %array_functions call:
https://github.com/swig/swig/blob/master/Lib/carrays.i
note how the values with "int" type are now just replaced to have "int64_t" type.
Hope this helps clear up any confusion!

Hello @imatiach-msft,
If we're going to change that, then why not use an unsigned tpe like size_t?

And another point @imatiach-msft ,
SWIG's carrays.i, is NOT memory-allocation safe, which means your whole application can crash just because the OS kills your underlying C process.
Before talking with you, I posted an issue at SWIG's repo to see if they wanted to fix it but they never answered back: https://github.com/swig/swig/issues/1755.
So my suggestion is since we're changing the Java API anyways, just fix it ourselves. We could throw an exception or even easier, use new (nothrow). Anyone who wishes to handle that case just has to test in Java for the null pointer, all other users who are not concerned with such issues (non-production) can keep working like before.
What do you say?

@AlbertoEAF "then can we use an unsigned type like size_t"
I think long is way more than enough in terms of size for the use-case.  I don't see size_t used as much in the lightgbm code anyway (also I'm not sure how I would do this from the scala code whereas I've already tested this with int64_t).

@AlbertoEAF  "So my suggestion is since we're changing the Java API anyways, just fix it ourselves"
Agree that it should be fixed, but can this be part of a separate PR?  This issue is much more pressing and I really want to fix it so I can publish a new lightgbm jar for customers with very large data.  The additional change would require more testing on my side.

gently ping @guolinke for the review


Install stuff on the server

I just added a package called django-simple-captcha.
Install that and any other required dependencies on the server

Also need to install something related to filters for the bootstrap CSS to work on the forms


Support new React Native new architecture (TurboModules/Fabric)

See https://reactnative.dev/docs/new-architecture-library-intro
https://reactnative.dev/blog/2022/06/16/resources-migrating-your-react-native-library-to-the-new-architecture

Would love to make a review of pull request with this


Trade Announce is Broken Since Last Commit

Message: SendChatMessage(): Chat message limits exceeded
Time: Sat Sep  4 17:47:30 2021
Count: 3
Stack: SendChatMessage(): Chat message limits exceeded
[string "@Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua"]:761: in function <...Ons\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:717>
[string "=[C]"]: ?
[string "=[C]"]: ?
[string "=[C]"]: ?
[string "@Interface\AddOns\WIM\Modules\WhisperEngine.lua"]:746: in function SendChatMessage' [string "@Interface\AddOns\GogoLoot\trade.lua"]:124: in function PrintTrade'
[string "@Interface\AddOns\GogoLoot\trade.lua"]:255: in function `TradeEvent'
[string "@Interface\AddOns\GogoLoot\loot.lua"]:459: in function <Interface\AddOns\GogoLoot\loot.lua:445>
Locals: errMsg = "SendChatMessage(): Chat message limits exceeded"
isBugGrabber = nil
tsmErrMsg = nil
oldModule = nil
private =  {
globalNameTranslation =  {
}
FormatErrorMessageSection =  defined @Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:517
ErrorHandler =  defined @Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:134
origErrorHandler =  defined @Interface\SharedXML\SharedBasicControls.lua:296
hitInternalError = false
CreateErrorFrame =  defined @Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:536
num = 0
SanitizeString =  defined @Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:510
AddonBlockedHandler =  defined @Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:502
GetStackLevelInfo =  defined @Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:344
ParseLocals =  defined @Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:396
ignoreErrors = false
errorReports =  {
}
IsTSMAddon =  defined @Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:478
localLinesTemp =  {
}
GetStackInfo =  defined @Interface\AddOns\TradeSkillMaster\LibTSM\Service\ErrorHandler.lua:313
}
MAX_STACK_DEPTH = 50
OLD_TSM_MODULES =  {
1 = "TradeSkillMaster_Accounting"
2 = "TradeSkillMaster_AuctionDB"
3 = "TradeSkillMaster_Auctioning"
4 = "TradeSkillMaster_Crafting"
5 = "TradeSkillMaster_Destroying"
6 = "TradeSkillMaster_Mailing"
7 = "TradeSkillMaster_Shopping"
8 = "TradeSkillMaster_Vendoring"
9 = "TradeSkillMaster_Warehousing"
}

I am unable to replicate this, so far. Tried trading to some one, getting something traded, as well as both at the same time.
Any more info on this?

Let's move this back to Planning.
I think it's possible when you do like...
Greater Eternal Essence x10
Greater Eternal Essence x10
Greater Eternal Essence x10
Greater Eternal Essence x10
Greater Eternal Essence x10
Greater Eternal Essence x10
and 500g 50s
Likely an edge case we need to work through, given the 255 character limit.
TSM combines it where it can, so when you mail it would just say, "Greater Eternal Essence x60". Anyway I think this is something we'll have to fiddle with later.

Adding the same items to 1 message is going to help in some scenario's, but not in all.
There is a maximum of 5 links in the same message, so if either of the two trade-parts contains 6 different items (or 5 items and an enchant) the crash will occur. I am trying to split the message into two separated ones ATM, which so far seem promising.


[clkmgr] Design sign-off for M2.5

Description
This tracks open design tasks for M2.5.
See effort estimation docs for details.
CC @msfschaffner @matutem

Michael, since this is design I may need your help.

Hi @matutem - I am already at it for all *mgr design sign-offs.
It looks like there are no open design TODO's, so it's really just about creating the checklist docs and running through the checklists.

Checklist doc

Documentation update done in https://github.com/lowRISC/opentitan/pull/18105.


New methods and cleanup

Adding support for new methods introduced to v1.5
Adding makeCall method on client to DRY up code a bit
Documentation correction

Coverage decreased (-9.1%) to 59.192% when pulling 3e5ec0c7fef942ac94ee9810cf7a0b2b47ef7e3f on add-new-methods into ea1301c1aa390002c619a42671127b86f614cbcd on dev.


improvement: docs updates

The numbers in the names mess with AshHq. We could manually do the guides list in the DocIndex, but this was faster. Happy to walk it back. We also now support regexes in the code modules, so I updated it to include additional docs.

Once we get a release w/ the guide names improved, I'll add the libraries to the ash_hq db and it should index 👍🏻

If you're good with it, I'll merge it (or you can, all the same to me)


ccid: 1.4.27 -> 1.4.29

Also update the non-existent homepage and download link.
Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg  build ccid

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: ccid
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on aarch64-linux (full log)
Attempted: ccid
Partial log (click to expand)
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/build/ccid-1.4.29'
make[1]: Leaving directory '/build/ccid-1.4.29'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/fpf4wwqqmf3fcj1qfbby9p4gv252ms04-ccid-1.4.29
shrinking /nix/store/fpf4wwqqmf3fcj1qfbby9p4gv252ms04-ccid-1.4.29/pcsc/drivers/ifd-ccid.bundle/Contents/Linux/libccid.so
strip is /nix/store/zrs21zqcchgyabjf4xfimncdq16njizc-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/fpf4wwqqmf3fcj1qfbby9p4gv252ms04-ccid-1.4.29
checking for references to /build in /nix/store/fpf4wwqqmf3fcj1qfbby9p4gv252ms04-ccid-1.4.29...
/nix/store/fpf4wwqqmf3fcj1qfbby9p4gv252ms04-ccid-1.4.29

Success on x86_64-linux (full log)
Attempted: ccid
Partial log (click to expand)
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/build/ccid-1.4.29'
make[1]: Leaving directory '/build/ccid-1.4.29'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/1idr0fdbl5cpyvh0sv8pl8ny826lnj4k-ccid-1.4.29
shrinking /nix/store/1idr0fdbl5cpyvh0sv8pl8ny826lnj4k-ccid-1.4.29/pcsc/drivers/ifd-ccid.bundle/Contents/Linux/libccid.so
strip is /nix/store/1hi76hr87bd1y1q1qjk0lv8nmcjip1c8-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/1idr0fdbl5cpyvh0sv8pl8ny826lnj4k-ccid-1.4.29
checking for references to /build in /nix/store/1idr0fdbl5cpyvh0sv8pl8ny826lnj4k-ccid-1.4.29...
/nix/store/1idr0fdbl5cpyvh0sv8pl8ny826lnj4k-ccid-1.4.29

Thank you!