code of training model

@JuewenPeng Hi, could you provide the code for training the model, I would be very grateful!

I'm sorry, but we do not consider providing the training code now.

That's okay, thank you.

Thanks for your comprehension.

@JuewenPeng Hello, HUST Alumni, could you provide the training dataset, I would be very grateful!

I am uploading the data to Baidu Netdisk. It will take some time. BTW, you can refer to our another paper (MPIB: An MPI-Based Bokeh Rendering Framework for Realistic Partial Occlusion Effects) for the details of data generation.

Ok, thank you!

@JuewenPeng Hello, could you provide the K and disp_focus in the BLB dataset, I would be very grateful!

The detailed information is shown in the info.json of each scene directory. You can use it as follows.
file = open(os.path.join(scene_path, 'info.json'), 'r')
info_data = json.load(file)
Ks = info_data['blur_parameters']
focus_distances = info_data['focus_distances']

for i in range(5):
  for j in range(10):
    disp_focus = 1 / focus_distances[j]
    defocus = Ks[i] * (disp - disp_focus)

Briefly, we set 5 K and 10 disp_focus. If you normalize the disp to 0-1 (do the same with disp_focus), K is from 10 to 50, and disp_focus is from 0 to 1.

Training dataset: https://pan.baidu.com/s/1bTxgBn54kB4xJ4YFoOvcAA?pwd=df72

Briefly, we set 5 K and 10 disp_focus. If you normalize the disp to 0-1 (do the same with disp_focus), K is from 10 to 50, and disp_focus is from 0 to 1.
That means K = [10, 20, 30, 40, 50] and disp_focus =( (1 / focus_distances)-(1 / focus_distances).min)/((1 / focus_distances).max-(1 / focus_distances).min)?

That means
disp = (disp - disp.min()) / (disp.max() - disp.min())  # 0-1
disp_focus = (1/focus_distances[j] - disp.min()) / (disp.max() - disp.min())  # 0-1
K = Ks[i] * ((disp.max() - disp.min()))  # 10-50
defocus = K * (disp - disp_focus)

But if you just need defocus map, you don't need to do the normalization since the above code is equivalent to
disp_focus = 1/focus_distances[j]
K = Ks[i]
defocus = K * (disp - disp_focus)

Thank you very much!!!

@JuewenPeng Hello, could you provide the code to calculate the error map, I would be very grateful!

We provide the code below. Note that this implementation is a little different from what we did in the original paper. This one is more efficient, and you can directly use it without predicting an error map by ARNet. Plus, you can adjust the two parameters delta1 and delta2 freely.
import torch
import torch.nn as nn
import torch.nn.functional as F
import cupy
import re

def gaussian_blur(x, r, sigma=None):
    r = int(round(r))
    if sigma is None:
        sigma = 0.3 * (r - 1) + 0.8
    x_grid, y_grid = torch.meshgrid(torch.arange(-int(r), int(r) + 1), torch.arange(-int(r), int(r) + 1))
    kernel = torch.exp(-(x_grid ** 2 + y_grid ** 2) / 2 / sigma ** 2)
    kernel = kernel.float() / kernel.sum()
    kernel = kernel.expand(1, 1, 2 * r + 1, 2 * r + 1).to(x.device)
    x = F.pad(x, pad=(r, r, r, r), mode='replicate')
    x = F.conv2d(x, weight=kernel, padding=0)
    return x

kernel_Render_updateOutput = '''

    extern "C" __global__ void kernel_Render_updateOutput(
        const int n,
        const float delta1,
        const float delta2,
        const float threshold1,
        const float threshold2,
        const float* radius_max,     // max blur radius map
        const float* radius_min,     // min blur radius map
        int* error                   // error map
    )
    {
        for (int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x; intIndex < n; intIndex += blockDim.x * gridDim.x) {
            const int intN = ( intIndex / SIZE_3(error) / SIZE_2(error) / SIZE_1(error) ) % SIZE_0(error);
            // const int intC = ( intIndex / SIZE_3(error) / SIZE_2(error)                 ) % SIZE_1(error);
            const int intY = ( intIndex / SIZE_3(error)                                 ) % SIZE_2(error);
            const int intX = ( intIndex                                                 ) % SIZE_3(error);

            float fltRadiusMax = VALUE_4(radius_max, intN, 0, intY, intX);
            float fltRadiusMin = VALUE_4(radius_min, intN, 0, intY, intX);

            if ((fltRadiusMax < threshold1) || (fltRadiusMin/fltRadiusMax > threshold2)) {
                continue;
            }

            for (int intDeltaY = -(int)(fltRadiusMax); intDeltaY <= (int)(fltRadiusMax); ++intDeltaY) {
                for (int intDeltaX = -(int)(fltRadiusMax); intDeltaX <= (int)(fltRadiusMax); ++intDeltaX) {

                    int intNeighborY = intY + intDeltaY;
                    int intNeighborX = intX + intDeltaX;

                    if ((intNeighborY >= 0) && (intNeighborY < SIZE_2(error)) && (intNeighborX >= 0) && (intNeighborX < SIZE_3(error))) {

                        float fltDist = sqrtf((float)(intDeltaY)*(float)(intDeltaY) + (float)(intDeltaX)*(float)(intDeltaX));
                        if (fltDist < fltRadiusMax) {
                            float alpha = fltDist / fltRadiusMax;
                            float beta = fltRadiusMin / fltRadiusMax;
                            // float fltError = (1 - powf(alpha, delta1)) * (1 - powf(beta, delta2));
                            // float fltError = (1 - powf(alpha, delta1)) * (delta2 > beta); // (0.5 + 0.5 * tanhf(50 * (delta2 - beta)));
                            // float fltError = (delta1 > alpha) * (delta2 > beta); // (0.5 + 0.5 * tanhf(50 * (delta2 - beta)));
                            float fltError = (0.5 + 0.5 * tanhf(20 * (delta1 - alpha))) * (0.5 + 0.5 * tanhf(20 * (delta2 - beta)));
                            atomicMax(&error[OFFSET_4(error, intN, 0, intNeighborY, intNeighborX)], int(fltError * 1e8));
                        }
                    }
                }
            }
        }
    }

'''

def cupy_kernel(strFunction, objVariables):
    strKernel = globals()[strFunction]

    while True:
        objMatch = re.search('(SIZE_)([0-4])(\()([^\)]*)(\))', strKernel)

        if objMatch is None:
            break
        # end

        intArg = int(objMatch.group(2))

        strTensor = objMatch.group(4)
        intSizes = objVariables[strTensor].size()

        strKernel = strKernel.replace(objMatch.group(), str(intSizes[intArg]))
    # end

    while True:
        objMatch = re.search('(OFFSET_)([0-4])(\()([^\)]+)(\))', strKernel)

        if objMatch is None:
            break
        # end

        intArgs = int(objMatch.group(2))
        strArgs = objMatch.group(4).split(',')

        strTensor = strArgs[0]
        intStrides = objVariables[strTensor].stride()
        strIndex = ['((' + strArgs[intArg + 1].replace('{', '(').replace('}', ')').strip() + ')*' + str(
            intStrides[intArg]) + ')' for intArg in range(intArgs)]

        strKernel = strKernel.replace(objMatch.group(0), '(' + str.join('+', strIndex) + ')')
    # end

    while True:
        objMatch = re.search('(VALUE_)([0-4])(\()([^\)]+)(\))', strKernel)

        if objMatch is None:
            break
        # end

        intArgs = int(objMatch.group(2))
        strArgs = objMatch.group(4).split(',')

        strTensor = strArgs[0]
        intStrides = objVariables[strTensor].stride()
        strIndex = ['((' + strArgs[intArg + 1].replace('{', '(').replace('}', ')').strip() + ')*' + str(
            intStrides[intArg]) + ')' for intArg in range(intArgs)]

        strKernel = strKernel.replace(objMatch.group(0), strTensor + '[' + str.join('+', strIndex) + ']')
    # end

    return strKernel
# end

# @cupy.util.memoize(for_each_device=True)
@cupy.memoize(for_each_device=True)
def cupy_launch(strFunction, strKernel):
    return cupy.cuda.compile_with_cache(strKernel).get_function(strFunction)
# end

class _FunctionRender(torch.autograd.Function):
    @staticmethod
    def forward(self, radius_max, radius_min, delta1, delta2):
        # self.save_for_backward()

        threshold1 = min(radius_min.shape[2], radius_min.shape[3]) / 1000
        threshold2 = 0.9

        error = torch.zeros_like(radius_max, dtype=torch.int)

        if error.is_cuda == True:
            n = error.nelement()
            cupy_launch('kernel_Render_updateOutput', cupy_kernel('kernel_Render_updateOutput', {
                'delta1': delta1,
                'delta2': delta2,
                'threshold1': threshold1,
                'threshold2': threshold2,
                'radius_max': radius_max,
                'radius_min': radius_min,
                'error': error,
            }))(
                grid=tuple([int((n + 512 - 1) / 512), 1, 1]),
                block=tuple([512, 1, 1]),
                args=[
                    cupy.int(n),
                    cupy.float32(delta1),
                    cupy.float32(delta2),
                    cupy.float32(threshold1),
                    cupy.float32(threshold2),
                    radius_max.data_ptr(),
                    radius_min.data_ptr(),
                    error.data_ptr(),
                ]
            )

        elif error.is_cuda == False:
            raise NotImplementedError()

        # end

        return error.float() / 1e8
    # end

    # @staticmethod
    # def backward(self, gradBokehCum, gradWeightCum):
    # end

# end

def FunctionRender(radius_max, radius_min, delta1, delta2):
    error = _FunctionRender.apply(radius_max, radius_min, delta1, delta2)

    return error
# end

class ModuleGenError(torch.nn.Module):
    def __init__(self):
        super(ModuleGenError, self).__init__()
    # end

    def forward(self, defocus, delta1, delta2, short_size=384):
        b, _, h, w = defocus.shape
        if short_size:
            h_re = int(round(min(h, max(short_size, short_size * h / w))))
            w_re = int(round(min(w, max(short_size, short_size * w / h))))
            scale = (h * w / h_re / w_re) ** 0.5
            defocus = 1/scale * F.interpolate(defocus, size=(h_re, w_re), mode='bilinear', align_corners=True)
        else:
            h_re = h
            w_re = w

        radius = defocus.abs()
        size = 2
        radius = F.pad(radius, pad=(size, size, size, size), mode='replicate')
        radius = F.unfold(radius, kernel_size=2*size+1)
        radius = radius.reshape(b, -1, h_re, w_re)
        radius_max = radius.max(dim=1, keepdim=True)[0]
        radius_min = radius.min(dim=1, keepdim=True)[0]

        error = FunctionRender(radius_max, radius_min, delta1, delta2)

        error = gaussian_blur(error, 3)

        if short_size:
            error = F.interpolate(error, size=(h, w), mode='bilinear', align_corners=True)

        return error
    # end
# end

if __name__ == '__main__':
    import os
    # os.environ['CUDA_VISIBLE_DEVICES'] = '7'
    defocus = 20 * torch.rand(1, 1, 1080, 1920).cuda()
    module = ModuleGenError().cuda()
    error = module(defocus, delta1=0.9, delta2=0.8, short_size=384)
    print(error)

OK. Thank you very much!!!

Hello! I had some problems while training arnet and would like to ask you for some advice. Firstly, ARNet's losses are falling but fluctuating a bit. The loss function does not contain the error map loss term. Another problem is that the bokeh image from ARNet‘s output has  colour errors.

Hello! I had some problems while training arnet and would like to ask you for some advice. Firstly, ARNet's losses are falling but fluctuating a bit. The loss function does not contain the error map loss term. Another problem is that the bokeh image from ARNet‘s output has colour errors. I don't know if you have encountered these problems and would appreciate some ideas on how to solve them.

Maybe the model needs more time to train. Also, our training code is based on DeepFocus. You can check if there is something wrong.

Hello! In the training dataset, bokehme_syn_data, you provided, do I need to normalize the disparity after reading 'disparity.exr'? That is, does the second step below require?

disp = cv2.imread(item["disp"], -1).astype(np.float32)
disp = (disp - disp.min()) / (disp.max() - disp.min())

No, you don’t need to do that.

OK，thank you for your reply!

hi, your work is so great! I have downloaded the traindataset and found that the disparity.exr is 5123, since bokeh_gt and image is 512512*3. Could you please give a guidance on how to use it? Thank you very much!

You can try disp = cv2.imread(disp_path, -1).astype(np.float32).

You can try disp = cv2.imread(disp_path, -1).astype(np.float32).

Thanks a lot!!

@wzfsjtu I trained the model using my own implementation of the training code, but it didn't work very well. I wonder if you have trained the model and how well it works.

@JuewenPeng Hi! I have a question. If I want to be using a DSLR to create the dataset, how should I determine the parameter K and parameter gamma corresponding to each bokeh image taken by the DSLR camera?

I think it is really hard. Besides, it is hard to obtain the disparity map, and captured all-in-focus images and bokeh images exist color inconsistency and misalignment, such as EBB! dataset.

OK, I got it. Thank you!
---Original---
From: @.>
Date: Fri, Sep 16, 2022 20:25 PM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [JuewenPeng/BokehMe] code of training model (Issue #4)
I think it is really hard. Besides, it is hard to obtain the disparity map, and captured all-in-focus images and bokeh images exist color inconsistency and misalignment, such as EBB! dataset.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

@JuewenPeng Hi,brother! Have you carried out any pre-processing and post-processing operations to further improve the results? Could you provide the relevant codes? The results I have obtained, both from my individual replication and from using the model you provided, are still a bit short of the metrics in your paper. So I would like to ask you what pre-processing and post-processing operations can be used to further improve the results.

Do you mean the evaluation on the BLB dataset?

The pretrained model we provide in this repository is different from that in the original paper, but in our experiment, this one is much better. You can use the following code to evaluate the model on the BLB dataset. Remember to change the data path to your own.
# NOTE
# In the BLB dataset, the maximum values of the all-in-focus image and the bokeh ground truth may be larger than 1.
# The numerical ranges of the predicted bokeh image and the ground truth will be clipped in [0, 1] before evaluation.

import os

import cv2
import numpy as np

import time
import xlwt
import json
import warnings
import argparse

import torch
import torch.nn as nn
import torch.nn.functional as F

############# import your model #############
from neural_renderer import ARNet, IUNet
from classical_renderer.scatter import ModuleRenderScatter  # circular aperture
#############################################

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

def calc_psnr(pred, gt, data_range=1):
    if data_range == 1:
        pred = pred * 255
        gt = gt * 255
    mse = torch.mean((pred - gt) ** 2)
    if mse == 0:
        return float('inf')
    else:
        return 20 * torch.log10(255.0 / torch.sqrt(mse)).item()

def calc_ssim(X, Y, mask=None, data_range=1, size_average=True, win_size=11, win_sigma=1.5, win=None, K=(0.01, 0.03), nonnegative_ssim=False):
    r""" interface of ssim
    Args:
        X (torch.Tensor): a batch of images, (N,C,H,W)
        Y (torch.Tensor): a batch of images, (N,C,H,W)
        data_range (float or int, optional): value range of input images. (usually 1.0 or 255)
        size_average (bool, optional): if size_average=True, ssim of all images will be averaged as a scalar
        win_size: (int, optional): the size of gauss kernel
        win_sigma: (float, optional): sigma of normal distribution
        win (torch.Tensor, optional): 1-D gauss kernel. if None, a new kernel will be created according to win_size and win_sigma
        K (list or tuple, optional): scalar constants (K1, K2). Try a larger K2 constant (e.g. 0.4) if you get a negative or NaN results.
        nonnegative_ssim (bool, optional): force the ssim response to be nonnegative with relu

    Returns:
        torch.Tensor: ssim results
    """
    if not X.shape == Y.shape:
        raise ValueError("Input images should have the same dimensions.")

    for d in range(len(X.shape) - 1, 1, -1):
        X = X.squeeze(dim=d)
        Y = Y.squeeze(dim=d)

    if len(X.shape) not in (4, 5):
        raise ValueError(f"Input images should be 4-d or 5-d tensors, but got {X.shape}")

    if not X.type() == Y.type():
        raise ValueError("Input images should have the same dtype.")

    if win is not None:  # set win_size
        win_size = win.shape[-1]

    if not (win_size % 2 == 1):
        raise ValueError("Window size should be odd.")

    if win is None:
        win = _fspecial_gauss_1d(win_size, win_sigma)
        win = win.repeat([X.shape[1]] + [1] * (len(X.shape) - 1))

    ssim_per_channel, cs = _ssim(X, Y, mask=mask, data_range=data_range, win=win, size_average=False, K=K)
    if nonnegative_ssim:
        ssim_per_channel = torch.relu(ssim_per_channel)

    if size_average:
        return ssim_per_channel.mean()
    else:
        return ssim_per_channel.mean(1)

def _fspecial_gauss_1d(size, sigma):
    r"""Create 1-D gauss kernel
    Args:
        size (int): the size of gauss kernel
        sigma (float): sigma of normal distribution

    Returns:
        torch.Tensor: 1D kernel (1 x 1 x size)
    """
    coords = torch.arange(size).to(dtype=torch.float)
    coords -= size // 2

    g = torch.exp(-(coords ** 2) / (2 * sigma ** 2))
    g /= g.sum()

    return g.unsqueeze(0).unsqueeze(0)

def gaussian_filter(input, win):
    r""" Blur input with 1-D kernel
    Args:
        input (torch.Tensor): a batch of tensors to be blurred
        window (torch.Tensor): 1-D gauss kernel

    Returns:
        torch.Tensor: blurred tensors
    """
    assert all([ws == 1 for ws in win.shape[1:-1]]), win.shape
    if len(input.shape) == 4:
        conv = F.conv2d
    elif len(input.shape) == 5:
        conv = F.conv3d
    else:
        raise NotImplementedError(input.shape)

    C = input.shape[1]
    out = input
    for i, s in enumerate(input.shape[2:]):
        if s >= win.shape[-1]:
            out = conv(out, weight=win.transpose(2 + i, -1), stride=1, padding=0, groups=C)
        else:
            warnings.warn(
                f"Skipping Gaussian Smoothing at dimension 2+{i} for input: {input.shape} and win size: {win.shape[-1]}"
            )

    return out

def _ssim(X, Y, mask, data_range, win, size_average=True, K=(0.01, 0.03)):

    r""" Calculate ssim index for X and Y

    Args:
        X (torch.Tensor): images
        Y (torch.Tensor): images
        win (torch.Tensor): 1-D gauss kernel
        data_range (float or int, optional): value range of input images. (usually 1.0 or 255)
        size_average (bool, optional): if size_average=True, ssim of all images will be averaged as a scalar

    Returns:
        torch.Tensor: ssim results.
    """
    K1, K2 = K
    # batch, channel, [depth,] height, width = X.shape
    compensation = 1.0

    C1 = (K1 * data_range) ** 2
    C2 = (K2 * data_range) ** 2

    win = win.to(X.device, dtype=X.dtype)

    mu1 = gaussian_filter(X, win)
    mu2 = gaussian_filter(Y, win)

    mu1_sq = mu1.pow(2)
    mu2_sq = mu2.pow(2)
    mu1_mu2 = mu1 * mu2

    sigma1_sq = compensation * (gaussian_filter(X * X, win) - mu1_sq)
    sigma2_sq = compensation * (gaussian_filter(Y * Y, win) - mu2_sq)
    sigma12 = compensation * (gaussian_filter(X * Y, win) - mu1_mu2)

    cs_map = (2 * sigma12 + C2) / (sigma1_sq + sigma2_sq + C2)  # set alpha=beta=gamma=1
    ssim_map = ((2 * mu1_mu2 + C1) / (mu1_sq + mu2_sq + C1)) * cs_map

    if mask == None:
        ssim_per_channel = torch.flatten(ssim_map, 2).mean(-1)
        cs = torch.flatten(cs_map, 2).mean(-1)
    else:
        crop_size = int(win.shape[3] // 2)
        mask = mask[:, :, crop_size:-crop_size, crop_size:-crop_size]
        ssim_per_channel = torch.flatten(ssim_map * mask, 2).mean(-1) / torch.flatten(mask, 2).mean(-1)
        cs = torch.flatten(cs_map * mask, 2).mean(-1) / torch.flatten(mask, 2).mean(-1)
    return ssim_per_channel, cs

def style(bold=False, underline=False, italic=False, auto_warp=True, align_h='center', align_v='center'):
    style = xlwt.XFStyle()

    font = xlwt.Font()
    font.bold = bold
    font.underline = underline
    font.italic = italic
    style.font = font

    alignment = xlwt.Alignment()
    if align_h == 'center':
        alignment.horz = xlwt.Alignment.HORZ_CENTER
    elif align_h == 'left':
        alignment.horz = xlwt.Alignment.HORZ_LEFT
    elif align_h == 'right':
        alignment.horz = xlwt.Alignment.HORZ_RIGHT
    else:
        print('error')
        exit(0)
    if align_v == 'center':
        alignment.vert = xlwt.Alignment.VERT_CENTER
    elif align_v == 'top':
        alignment.vert = xlwt.Alignment.VERT_TOP
    elif align_v == 'bottom':
        alignment.vert = xlwt.Alignment.VERT_BOTTOM
    else:
        print('error')
        exit(0)
    alignment.wrap = int(auto_warp)
    style.alignment = alignment

    return style

########################## change to your rendering pipeline ##########################
def gaussian_blur(x, r, sigma=None):
    r = int(round(r))
    if sigma is None:
        sigma = 0.3 * (r - 1) + 0.8
    x_grid, y_grid = torch.meshgrid(torch.arange(-int(r), int(r) + 1), torch.arange(-int(r), int(r) + 1))
    kernel = torch.exp(-(x_grid ** 2 + y_grid ** 2) / 2 / sigma ** 2)
    kernel = kernel.float() / kernel.sum()
    kernel = kernel.expand(1, 1, 2*r+1, 2*r+1).to(x.device)
    x = F.pad(x, pad=(r, r, r, r), mode='replicate')
    x = F.conv2d(x, weight=kernel, padding=0)
    return x

def pipeline(classical_renderer, arnet, iunet, image, defocus, gamma, args):
    bokeh_classical, defocus_dilate = classical_renderer(image**gamma, defocus*args.defocus_scale)

    bokeh_classical = bokeh_classical ** (1/gamma)
    defocus_dilate = defocus_dilate / args.defocus_scale
    gamma = (gamma - args.gamma_min) / (args.gamma_max - args.gamma_min)
    adapt_scale = max(defocus.abs().max().item(), 1)

    image_re = F.interpolate(image, scale_factor=1/adapt_scale, mode='bilinear', align_corners=True)
    defocus_re = 1 / adapt_scale * F.interpolate(defocus, scale_factor=1/adapt_scale, mode='bilinear', align_corners=True)
    bokeh_neural, error_map = arnet(image_re, defocus_re, gamma)
    error_map = F.interpolate(error_map, size=(image.shape[2], image.shape[3]), mode='bilinear', align_corners=True)
    bokeh_neural.clamp_(0, 1e5)

    for scale in range(int(np.log2(adapt_scale))):
        ratio = 2**(scale+1) / adapt_scale
        h_re, w_re = int(ratio * image.shape[2]), int(ratio * image.shape[3])
        image_re = F.interpolate(image, size=(h_re, w_re), mode='bilinear', align_corners=True)
        defocus_re = ratio * F.interpolate(defocus, size=(h_re, w_re), mode='bilinear', align_corners=True)
        defocus_dilate_re = ratio * F.interpolate(defocus_dilate, size=(h_re, w_re), mode='bilinear', align_corners=True)
        bokeh_neural_refine = iunet(image_re, defocus_re.clamp(-1, 1), bokeh_neural, gamma).clamp(0, 1e5)
        mask = gaussian_blur(((defocus_dilate_re < 1) * (defocus_dilate_re > -1)).float(), 0.005 * (defocus_dilate_re.shape[2] + defocus_dilate_re.shape[3]))
        bokeh_neural = mask * bokeh_neural_refine + (1 - mask) * F.interpolate(bokeh_neural, size=(h_re, w_re), mode='bilinear', align_corners=True)

    bokeh_neural_refine = iunet(image, defocus.clamp(-1, 1), bokeh_neural, gamma).clamp(0, 1e5)
    mask = gaussian_blur(((defocus_dilate < 1) * (defocus_dilate > -1)).float(), 0.005 * (defocus_dilate.shape[2] + defocus_dilate.shape[3]))
    bokeh_neural = mask * bokeh_neural_refine + (1 - mask) * F.interpolate(bokeh_neural, size=(image.shape[2], image.shape[3]), mode='bilinear', align_corners=True)

    bokeh_pred = bokeh_classical * (1 - error_map) + bokeh_neural * error_map

    return bokeh_pred.clamp(0, 1), bokeh_classical.clamp(0, 1), bokeh_neural.clamp(0, 1), error_map
#######################################################################################

def main():
    gamma = 2.2

    ############################# change to your settings #############################
    method = 'BokehMe'            # method name
    root = '/data2/pengjuewen/Bokeh/Blender/data'        # path to BLB dataset
    save_root = os.path.join('./BLB', method)  # path to save

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    parser = argparse.ArgumentParser(description='Bokeh Rendering', fromfile_prefix_chars='@')

    parser.add_argument('--defocus_scale', type=float, default=10.)
    parser.add_argument('--gamma_min', type=float, default=1.)
    parser.add_argument('--gamma_max', type=float, default=5.)

    # Model 1
    parser.add_argument('--arnet_shuffle_rate', type=int, default=2)
    parser.add_argument('--arnet_in_channels', type=int, default=5)
    parser.add_argument('--arnet_out_channels', type=int, default=4)
    parser.add_argument('--arnet_middle_channels', type=int, default=128)
    parser.add_argument('--arnet_num_block', type=int, default=3)
    parser.add_argument('--arnet_share_weight', action='store_true')
    parser.add_argument('--arnet_connect_mode', type=str, default='distinct_source')
    parser.add_argument('--arnet_use_bn', action='store_true')
    parser.add_argument('--arnet_activation', type=str, default='elu')

    # Model 2
    parser.add_argument('--iunet_shuffle_rate', type=int, default=2)
    parser.add_argument('--iunet_in_channels', type=int, default=8)
    parser.add_argument('--iunet_out_channels', type=int, default=3)
    parser.add_argument('--iunet_middle_channels', type=int, default=64)
    parser.add_argument('--iunet_num_block', type=int, default=3)
    parser.add_argument('--iunet_share_weight', action='store_true')
    parser.add_argument('--iunet_connect_mode', type=str, default='distinct_source')
    parser.add_argument('--iunet_use_bn', action='store_true')
    parser.add_argument('--iunet_activation', type=str, default='elu')

    # Checkpoint
    parser.add_argument('--arnet_checkpoint_path', type=str, default='./checkpoints/arnet.pth')
    parser.add_argument('--iunet_checkpoint_path', type=str, default='./checkpoints/iunet.pth')

    # Input
    args = parser.parse_args()

    arnet_checkpoint_path = args.arnet_checkpoint_path
    iunet_checkpoint_path = args.iunet_checkpoint_path

    classical_renderer = ModuleRenderScatter().to(device)

    arnet = ARNet(args.arnet_shuffle_rate, args.arnet_in_channels, args.arnet_out_channels, args.arnet_middle_channels,
                  args.arnet_num_block, args.arnet_share_weight, args.arnet_connect_mode, args.arnet_use_bn,
                  args.arnet_activation)
    iunet = IUNet(args.iunet_shuffle_rate, args.iunet_in_channels, args.iunet_out_channels, args.iunet_middle_channels,
                  args.iunet_num_block, args.iunet_share_weight, args.iunet_connect_mode, args.iunet_use_bn,
                  args.iunet_activation)

    arnet.cuda()
    iunet.cuda()

    checkpoint = torch.load(arnet_checkpoint_path)
    arnet.load_state_dict(checkpoint['model'])
    checkpoint = torch.load(iunet_checkpoint_path)
    iunet.load_state_dict(checkpoint['model'])

    arnet.eval()
    iunet.eval()
    ###################################################################################

    os.makedirs(save_root, exist_ok=True)

    scene_lst = [name for name in sorted(os.listdir(root)) if '.' not in name]
    disp_focus_lst = [f'({i+1})' for i in range(10)]
    metric_lst = ['psnr', 'ssim', 'runtime']

    scene_num = len(scene_lst)
    disp_focus_num = len(disp_focus_lst)
    metric_num = len(metric_lst)

    with torch.no_grad():
        for K_idx in range(5):
            scene_metric_avg = np.zeros([metric_num, scene_num])
            disp_focus_metric_avg = np.zeros([metric_num, disp_focus_num])

            # initialize excel
            workbook = xlwt.Workbook(encoding='utf-8')
            worksheets = [workbook.add_sheet(ind, cell_overwrite_ok=True) for ind in metric_lst]

            standard_style = style()
            left_style = style(align_h='left')

            for worksheet in worksheets:
                worksheet.write_merge(0, 1, 0, 1, method, style=standard_style)
                worksheet.write_merge(0, 0, 2, 1 + disp_focus_num, 'Refocused Disparity', style=standard_style)
                for i, disp_focus in enumerate(disp_focus_lst):
                    worksheet.write(1, 2 + i, disp_focus, style=standard_style)
                worksheet.write_merge(2, 1 + scene_num, 0, 0, 'Scene', style=standard_style)
                worksheet.write_merge(2 + scene_num, 2 + scene_num, 0, 1, 'Average', style=standard_style)
                worksheet.write_merge(0, 1, 2 + disp_focus_num, 2 + disp_focus_num, 'Average', style=standard_style)

            for scene_idx in range(scene_num):
                scene_name = scene_lst[scene_idx]
                scene_path = os.path.join(root, scene_name)
                save_scene_path = os.path.join(save_root, scene_name)

                os.makedirs(save_scene_path, exist_ok=True)

                image = cv2.imread(os.path.join(scene_path, 'image.exr'), -1)[..., :3].astype(np.float32) ** (1/gamma)
                image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)  # Input RGB and output RGB by default

                ###### change to the name of corrupted depth map if necessary ######
                depth_name = 'depth.exr'
                ####################################################################

                depth = cv2.imread(os.path.join(scene_path, depth_name), -1)[..., 0].astype(np.float32)
                disp = 1 / depth

                ############# comment it if using tensorflow model #############
                image = torch.from_numpy(image).permute(2, 0, 1).unsqueeze(0).contiguous().to(device)
                disp = torch.from_numpy(disp).unsqueeze(0).unsqueeze(0).contiguous().to(device)
                ################################################################

                for metric_idx in range(metric_num):
                    worksheets[metric_idx].write(2+scene_idx, 1, scene_name, style=standard_style)

                file = open(os.path.join(scene_path, 'info.json'), 'r')
                info_data = json.load(file)
                Ks = info_data['blur_parameters']
                focus_distances = info_data['focus_distances']

                for df_idx in range(len(disp_focus_lst)):
                    K = Ks[K_idx]
                    disp_focus = 1 / focus_distances[df_idx]
                    defocus = K * (disp - disp_focus) / args.defocus_scale

                    gt_name = f'bokeh_{K_idx:0>2d}_{df_idx:0>2d}.exr'
                    gt = cv2.imread(os.path.join(scene_path, gt_name), -1)[..., :3].astype(np.float32) ** (1/gamma)
                    gt = cv2.cvtColor(gt, cv2.COLOR_BGR2RGB)
                    gt = torch.from_numpy(gt).permute(2, 0, 1).unsqueeze(0).contiguous().to(device)

                    # neglect the runtime of the first inference
                    if scene_idx + df_idx == 0:
                        pipeline(classical_renderer, arnet, iunet, image, defocus, gamma, args)

                    ###### comment it if using pytorch cpu or tensorflow model ######
                    torch.cuda.synchronize()
                    #################################################################
                    start = time.time()

                    pred = pipeline(classical_renderer, arnet, iunet, image, defocus, gamma, args)[0]

                    ###### comment it if using pytorch cpu or tensorflow model ######
                    torch.cuda.synchronize()
                    #################################################################
                    end = time.time()

                    ############ uncomment it if using tensorflow model ############
                    # pred = torch.from_numpy(pred).to(device)
                    ################################################################

                    # evaluation
                    psnr = calc_psnr(pred.clamp(0, 1), gt.clamp(0, 1))
                    ssim = calc_ssim(pred.clamp(0, 1), gt.clamp(0, 1))

                    # save results
                    pred = pred[0].cpu().clone().permute(1, 2, 0).numpy()
                    pred = cv2.cvtColor(pred, cv2.COLOR_RGB2BGR)

                    save_name = f'bokeh_{K_idx:0>2d}_{df_idx:0>2d}.jpg'
                    cv2.imwrite(os.path.join(save_scene_path, save_name), pred * 255)

                    runtime = end - start

                    scene_metric_avg[0, scene_idx] += psnr
                    scene_metric_avg[1, scene_idx] += ssim
                    scene_metric_avg[2, scene_idx] += runtime

                    disp_focus_metric_avg[0, df_idx] += psnr
                    disp_focus_metric_avg[1, df_idx] += ssim
                    disp_focus_metric_avg[2, df_idx] += runtime

                    # write to excel
                    ii = scene_idx + 2
                    jj = df_idx + 2
                    worksheets[0].write(ii, jj, float(psnr), style=left_style)
                    worksheets[1].write(ii, jj, float(ssim), style=left_style)
                    worksheets[2].write(ii, jj, float(runtime), style=left_style)

                    print(f'scene[{scene_idx+1}/{scene_num}]  disp_focus[{df_idx+1}/{disp_focus_num}]  '
                          f'PSNR:{psnr}  SSIM:{ssim}  Runtime: {runtime}')

            scene_metric_avg /= disp_focus_num
            disp_focus_metric_avg /= scene_num
            assert np.abs(scene_metric_avg.mean(axis=1)[0] - disp_focus_metric_avg.mean(axis=1)[0]) < 1e-5
            metric_avg = scene_metric_avg.mean(axis=1)

            for scene_idx in range(scene_num):
                for df_idx in range(disp_focus_num):
                    ii = scene_idx + 2
                    jj = df_idx + 2
                    for metric_idx in range(metric_num):
                        worksheets[metric_idx].write(ii, 2+disp_focus_num, float(scene_metric_avg[metric_idx, scene_idx]), style=left_style)
                        worksheets[metric_idx].write(2+scene_num, jj, float(disp_focus_metric_avg[metric_idx, df_idx]), style=left_style)

            for metric_idx in range(metric_num):
                worksheets[metric_idx].write(2+scene_num, 2+disp_focus_num, float(metric_avg[metric_idx]), style=left_style)

            xls_name = f'evaluation_K={int(10*(K_idx+1))}.xls'
            workbook.save(os.path.join(save_root, xls_name))

    print(f'"{method}" evaluation done!')

if __name__ == '__main__':
    main()

OK, thank you very much!

@JuewenPeng Hi! I have two questions about model training. First, since I see that you did not normalize ‘image.exr’ and 'bokeh.exr' after reading them when evaluating BLB, I would like to ask if I need to normalize  ‘image.exr’ and 'bokeh.exr' after reading them when training the model. The second query is whether the numerical ranges of the predicted bokeh image and the ground truth will be clipped in [0, 1] before calculating the loss. Looking forward to your answers!

@JuewenPeng Hi! Could you provide the test dataset EBB400 containing the corresponding disparity map and the refocused disparity?  I would be very grateful!

@JuewenPeng Hi! I have two questions about model training. First, since I see that you did not normalize ‘image.exr’ and 'bokeh.exr' after reading them when evaluating BLB, I would like to ask if I need to normalize ‘image.exr’ and 'bokeh.exr' after reading them when training the model. The second query is whether the numerical ranges of the predicted bokeh image and the ground truth will be clipped in [0, 1] before calculating the loss. Looking forward to your answers!

For the first question, you don't need to normalize the images during the training. Values of all RGB images are supposed to be in the range of [0, 1.5]. For the second question, we clip the values of bokeh images in [0, 1] for all methods since some of them cannot output values out of [0, 1] (our methods can do that), so we think this clipping operation can make the comparisons fairer.

@JuewenPeng Hi! Could you provide the test dataset EBB400 containing the corresponding disparity map and the refocused disparity? I would be very grateful!

Do you really need that? Honestly, I don't think this experiment is really necessary. Just to make the experiment more complete.

@JuewenPeng Hi! Could you provide the test dataset EBB400 containing the corresponding disparity map and the refocused disparity? I would be very grateful!

Do you really need that? Honestly, I don't think this experiment is really fair and necessary. Just to make the experiment more complete.

Here's the thing, I want to see the bokeh effect in a real scene. The EBB400 is just right for me for this need. However, generating disparity maps and determining refocused disparity values is a time-consuming task, and it would save me a lot of work if you could provide readily available data.

OK, let me upload it to the Baidu Netdisk.

@JuewenPeng Hi! I have two questions about model training. First, since I see that you did not normalize ‘image.exr’ and 'bokeh.exr' after reading them when evaluating BLB, I would like to ask if I need to normalize ‘image.exr’ and 'bokeh.exr' after reading them when training the model. The second query is whether the numerical ranges of the predicted bokeh image and the ground truth will be clipped in [0, 1] before calculating the loss. Looking forward to your answers!

For the first question, you don't need to normalize the images during the training. Values of all RGB images are supposed to be in the range of [0, 1.5]. For the second question, we clip the values of bokeh images in [0, 1] for all methods since some of them cannot output values out of [0, 1] (our methods can do that), so we think this clipping operation can make the comparisons fairer.

Thank you very much for your answers. For the second question, I would like to know if it is necessary to clip the numerical ranges of the predicted bokeh and the gt bokeh before calculating the loss when training the model.

Thank you very much

Thank you very much!!!

OK, let me upload it to the Baidu Netdisk.

Thank you very much!!!

I think it's optional, but in my practice, I didn't clip the predicted values during the training.

I want to know for the pretrained model you provide in this repository, did you train it using only the train dataset in bokehme_syn_data, and did you add some data from BLB? I use only the train dataset in bokehme_syn_data to train the model, and when testing the BLB dataset, the SSIM metric can only reach 0.97. I wonder if there is anything I need to pay attention to in the training process

We only trained our model on the synthetic dataset.

EBB400 Baidu Netdisk: [https://pan.baidu.com/s/1l3Rug16HEB2uUi3u366vLw?pwd=f7mp](https://pan.baidu.com/s/1l3Rug16HEB2uUi3u366vLw?pwd=f7mp](https://pan.baidu.com/s/1l3Rug16HEB2uUi3u366vLw?pwd=f7mp))
We conduct our experiment using the disparity maps in disparity directory, all of which are predicted by MiDaS. We also provide the disparity maps predicted by DPT in disparity_dpt directory. You can use them if expecting better bokeh rendering effects.

OK, thank you very much!

Could you provide your own email if you want some advice on training.

<EMAIL_ADDRESS>Thank you very much!

EBB400 Baidu Netdisk: https://pan.baidu.com/s/1l3Rug16HEB2uUi3u366vLw?pwd=f7mp
We conduct our experiment using the disparity maps in disparity directory, all of which are predicted by MiDaS. We also provide the disparity maps predicted by DPT in disparity_dpt directory. You can use them if expecting better bokeh rendering effects.

thanks for the ebb400. and there is another question. I can't download the entire EBB training set because the website
[(https://competitions.codalab.org/competitions/24716#participate) is not valid. Do you have a backup here? Can you share it

Sorry for that, but you'd better first register for the competition and then download the entire dataset.

Sorry for that, but you'd better first register for the competition and then download the entire dataset.

But it seems that i can't register for the competition because it  ends.

I remember that one can register for the competition any time.

I remember that one can register for the competition any time.

this ?

Yes, that's it. Just wait until the official give you the access to the dataset. By the way, how long have you been waiting?

about 4 days

Ok, in that case, I will upload the data and send the link to you. It also requires some time.

thanks a lot

Ok, in that case, I will upload the data and send the link to you. It also requires some time.

thans ~~~

Ok, in that case, I will upload the data and send the link to you. It also requires some time.

hi , i get the access to the dataset from official, thanks again.

You are welcome.

EBB400 Baidu Netdisk: https://pan.baidu.com/s/1l3Rug16HEB2uUi3u366vLw?pwd=f7mp
We conduct our experiment using the disparity maps in disparity directory, all of which are predicted by MiDaS. We also provide the disparity maps predicted by DPT in disparity_dpt directory. You can use them if expecting better bokeh rendering effects.

well, I found there is a folder names "focus_mask" in EBB400 , how did you make this ? thanks ~

We manually label the focus mask of each image.

Hi, this is a great work. I have some questions about the training: 1) My understanding is that during training the bn is set to True for both the networks. is this correct? 2) What about share_weight? Is this also set to True for both the models? 3) When training IUNet (while ARNet is frozen), is the adaptive resizing unused? If you consider adaptive resizing before ARNet for training IUNet,, how do you manage different input sizes to IUNet? 4) Is the "connect_mode" set to "distinct_source" for both models? 5) Do you start training from scratch or the models are initialized by weights driven for another task? 6) You have mentioned that "To improve the generalization, we additionally augment the input disparity map with random gaussian blur, dilation and erosion." Could you please elaborate more on this? What are the maximum values for blur, dilation, and erosion? I would really appreciate it if you could help me with these questions.


'"gcc"' is not recognized as an internal or external command, operable program or batch file.

Hello,
I've got MinGW in my PATH, and if I do:
gcc {filename}.c
in the command prompt on windows, it works fine.
However, if I try to compile a file in Atom, with F6, I get the following error:
'"gcc"' is not recognized as an internal or external command, operable program or batch file.
Why does this happen?
I am running Windows 10, with the latest version of Atom on a 64-bit computer.
Thanks,
MoonriseWarrior

Oh, no... My bad. Forgot to restart Atom after adding MinGW to PATH. Closing issue.


Fixes for #21, #15, #40

Hi,
Here are commits that resolves issues #21 (translations) and #15 (active items below tree leafs).
#15 is made as an option in order not to break existing code, but for us (as well for issue's author) it is desired feature to be able to highlight menu item for all pages visible under path starting with same path as menu item.

Thanks! Some of these issues are being addressed in other PRs and this PR is against master rather than develop. Keeping each PR focused on 1 issue will make it easier for us to evaluate them, in the future.


Add MerkleProof for 160 bit hashes

Pull Request Test Coverage Report for Build 68

0 of 0   changed or added relevant lines in 0 files are covered.
13 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-11.8%) to 40.789%

Files with Coverage Reduction
New Missed Lines
%

contracts/mocks/MerkleProofWrapper.sol
1
50.0%

contracts/MerkleProof.sol
12
40.74%

Totals

Change from base Build 66:
-11.8%

Covered Lines:
25

Relevant Lines:
66

💛  - Coveralls


119 support pypi sdist artifacts with reqstool files

TODO:

[] check with None or defaults for annotations and test_results in src/reqstool/requirements_indata/requirements_indata_paths.py (run tests)


Use NM.Device.get_applied_connection() to get applied config

Currently we are using NM.RemoteConnection which is on-disk profile. We should switch to NM.Device.get_applied_connection() which shows the applied configure.
It's a ASYNC call, should be perpared by plugin level before sending to lower level codes.

The DNS is using applied config.
The route will use nispor.
The interface is using nispor.


UI to surface Alpine.js warnings and errors

Alpine.js uses console.warn in the following scenarios:

x-if/x-for on a non-template https://github.com/alpinejs/alpine/blob/ee019cbc66d2adb3e5126bf3822d9d701a40daf8/src/utils.js#L29
template with more than 1 child node https://github.com/alpinejs/alpine/blob/ee019cbc66d2adb3e5126bf3822d9d701a40daf8/src/utils.js#L31
error during expression evaluation https://github.com/alpinejs/alpine/blob/5cc46c31b08e4b25821fb85fd0fad6903bc7e574/src/utils.js#L69

I believe some developers suppress console.warn, it would be nice to surface the relevant errors/warnings:

in a "warnings" tab in the devtools
highlight the offending components/elements (toggleable through settings)

For 1. and 2. we should probably re-scan the DOM and to figure out which are the offending template's.
For 3. the console.warn contains the element as well as the expression.

Any idea where we should add various tabs? We could remove the "Alpine v2.7.1 detected" and add them there. Just a quick idea:

<button class="bg-alpine-300 hover:bg-alpine-300 leading-none px-3 py-2 rounded-full">Components</button>

<button class="hover:bg-alpine-300 leading-none px-3 py-2 rounded-full">Events</button>

<button class="hover:bg-alpine-300 leading-none px-3 py-2 rounded-full">Warnings</button>

We could also add a count for each button when not selected, like "Warnings 2" Or grab something from tailwindui?
I also noticed a white gap bottom right of the right pane. I'll see what that is later

We could do tabs like you've done but to the right, before the GitHub/docs/settings icons?

Here's how that looks:

It could be moved over some more too. The "docs" icon has some extra padding too that should probably be removed to match the other icons.
Is it just me or does the "Alpine.js v2.7.3 detected" feel too overwhelming? It's not really that critical of information besides the version number, which is really only critical in a situation where you're trying to debug in support

Also, i'm not a great designer, so if someone has an opinion on the button styles, etc. please do share it. :)

I went with the right aligned tabs (not pill shapes though) in #129 (still a WIP). Probably makes more sense, I tend to naturally float to the right when searching for buttons.

Yeah maybe we can shorten the text from "Alpine.js v2.7.3 detected" to circle icon + "v2.7.3" and maybe use a lighter shade of orange?
We can also compress the pills/tabs for small breakpoints by switching to an icon + title attribute

Ah, I didn't check #129 yet. All that sounds good. Right side sounds good too. Choosing the right icons might be the challenge now :)
I think we also need room for a counter too. For example if there are 3 warnings or 5 events. Or if not a counter, just a colored dot might be enough to show there's some activity there.

We also need a loading state for the buttons. Or otherwise just make the toolbar always dark?

We also need a loading state for the buttons. Or otherwise just make the toolbar always dark?

yeah I think this looks fine, we can disable/fade or even hide them while loading

Not to put the cat amongst the pigeons 😉
Here's my take:

The right align feels right for the top nav items
Might be worth separating the secondary nav items to give a bit of visual/UX hierarchy
Not wanting to just go ahead and copy the VueJs guys, but an active tab being underlined in some way when it's in a header I reckon feels a bit cleaner
Also not wanting to completely rip off @KevinBatdorf 😉, but I really like the footer in his inline Alpine inspector tool*. Could be a better place to put version detection being mindful that the top bar is going to fill up pretty quick when using the panel in the sidebar view. Could also become a kind of live console e.g. seeing events come in if you're not currently in the events panel 'Click event caught from blah component` etc.

Don't get me started on those sweeeet looking themes 👌🏼😍

Or does the right align feel right?! 😬😆

I like it! And rip off anything :)
Maybe we can add a theming setup to this project eventually as well.
Looking at both, right-aligned seems nicer. Maybe we could move those three icons to the right footer as well.

I personally prefer the right aligned items, feels less crowded on the left hand side where the list of components, etc will be.
@stephenoldham is the design master mind though ;)

Ha! Not sure about that @ryangjchandler 😆
Big yes from me on the theming setup @KevinBatdorf 👍🏼 Would be a rad addition to the settings layer.
The only concern might be the settings getting lost on the footer but see what you think:

Here's a few footer vibes...you get the idea:

The 2nd one in is my favourite. The separator is nice, but I also like the last one with the different BG colour.

If somebody is going to implement the tabs, etc, I'll hold off on #129.

Ha! Not sure about that @ryangjchandler 😆
Big yes from me on the theming setup @KevinBatdorf 👍🏼 Would be a rad addition to the settings layer.
The only concern might be the settings getting lost on the footer but see what you think:

Here's a few footer vibes...you get the idea:

😍 could try with the status bar (Alpine.js vX.X.X detected) between the header and the main panes?
I was happy to settle for text links in the header 😄

Right then...the contenders...

Quite like both 😆 Subheader version would just need a bit of play with the padding on the right. Thoughts?

Subheader version will give use the opportunity to bring in a search UI maybe?
I guess they're very similar either way (plus I'm sure we can move it around if we decide on one and end up not liking it)

I'm gonna vote for the footer version.

I think in terms of the content hierarchy I prefer the footer. Also feels like there's more space in that layout with less at the top.
@HugoDF As you said we can totally change it if we don't vibe that layout in practise.

Yeah it's fine, I'm a bit weary of making big changes to the layout of the UI if people have come to rely on it.
Then again, better to make the change now and support future features rather than wait

Anybody volunteering to implement the new design?

Can do 👍🏼

cool, I would say just to watch out for the testid's otherwise tests will fail and also to hide tabs that don't have any content for production builds (the same way we hid the settings modal #111)

Additional things we can write our own "warning" for (see https://github.com/alpinejs/alpine/discussions/975)

x-init on an element without x-data
Alpine is loaded but there is no x-data on the page
an alpine directive is being used on an element that doesn't have x-data and has no element with x-data in its ancestors

Thinking about it further the following:

x-if/x-for on a non-template element
template with more than 1 child node

Don't need to/can't be done with console.warn interception, so it might be worth splitting as a separate issue.
Other note: I'm not sure how we're going to highlight template's that have issues (since they don't visually display)

Additional things we can write our own "warning" for (see https://github.com/alpinejs/alpine/discussions/975)

x-init on an element without x-data
Alpine is loaded but there is no x-data on the page
an alpine directive is being used on an element that doesn't have x-data and has no element with x-data in its ancestors

Thinking about it further the following:

x-if/x-for on a non-template element
template with more than 1 child node

Don't need to/can't be done with console.warn interception, so it might be worth splitting as a separate issue.
Other note: I'm not sure how we're going to highlight template's that have issues (since they don't visually display)

We might not need to instrument console.warn once https://github.com/alpinejs/alpine/pull/1000 gets released

We might not need to instrument console.warn once https://github.com/alpinejs/alpine/pull/1000 gets released


see xskillscore

Came across this repo. Just in case you aren't aware there is https://github.com/xarray-contrib/xskillscore which may share some similarities with this repo.

@raybellwaves Thank you for the note! I have seen that listed but hadn't clicked on it before. I agree that it would make sense to have the stats part of this package be done using xskillscore — have put that as a to do for the future when I'm working on this next. Thanks again!


Use of text-merge (custom fields) with Embedded Signing

The documentation says "coming soon": https://app.hellosign.com/api/textTagsWalkthrough
Section: Allowing a signer to edit pre-set values
Example: custom_fields:[{"name":"organization_name", "value":"Enter organization name", "editor":"1", "required":"true"}]

I believe this is a duplicate of issue #21 . Can you confirm?

@neillom This is not the same issue as #21. We will work towards showing more clear examples of both. We'll start with issue 21.

@neillom
curl 'https://api.hellosign.com/v3/signature_request/send'
-u 'API KEY:'
-F 'title=NDA with Acme Co.'
-F 'subject=The NDA we talked about'
-F 'message=Please sign this NDA and then we can discuss more.'
-F<EMAIL_ADDRESS>-F 'signers[1][name]=Jack'
-F 'custom_fields=[{"name":"organization_name", "value":"Enter organization name", "editor":"signer1", "required":"true"}]'
-F<EMAIL_ADDRESS>-F 'use_text_tags=1'
-F 'hide_text_tags=1'
-F 'test_mode=1'
The above call with the attached file called texttags.pdf
texttags.pdf
which has this on the pdf:
[text-merge|req|signer1|organization_name|api_id]
Causes the functionality that represents this issue.
more info can be found here:
https://app.hellosign.com/api/textTagsWalkthrough and this is different from #21. Both of these issues are implementations of custom fields but this issue is implementing custom fields using text tags and the other issue is implementing custom fields using hellosign end user templates. That is the main difference.
DESIRED RESULTS: Sending the following would have the same results as it does with the curl call.
signature_request = client.send_signature_request(
test_mode=True,
files=files,
use_text_tags=True,
hide_text_tags=False,
file_urls=None,
title="NDA with Acme Co.",
subject="The NDA we talked about",
message="Please sign this NDA and then we can discuss more.",
signing_redirect_url=None,
custom_fields=[{"name":"organization_name", "value":"Enter organization name", "editor":"signer1", "required":"true"}],
signers=signers,
client_id="CLIENT_ID_HERE",
signing_options = {"draw": True, "type": True, "upload": True, "phone": False, "default": "type"},
texttags.pdf
cc_email_addresses=cc_email_addresses)
print(signature_request)
Tested on version 3.8.9 and this has not been implemented yet.

A workaround in the meantime would be to use a requests library. Example code here:
buildTheRequest = 'https://' + API_KEY +<EMAIL_ADDRESS>data = {         'client_id': CLIENT_ID,         'file_url': 'FILE_URL_HERE',         'subject': 'emb sig req w text tags',         'signers[1][name]': 'SIGNER_NAME_HERE',         'signers[1][email_address]':'EMAIL_ADDRESS_HERE',         'custom_fields': '[{"name":"test1", "value":"Testing custom fields here"}]',         'use_text_tags': 1,         'hide_text_tags': 1,         'test_mode': '1'        } print(buildTheRequest) r = requests.post(buildTheRequest, data) print(r.text)

Adding another request to have this added to the SDK.
Internal reference number: 10930
Related ticket number: 11378488


jupyter extension flaky test fix - move unmount step until after app is ready

the end-to-end tests for the jupyter extension were flaky due to the unmount endpoint serving 404s sometimes. moving the unmount step until after the app is ready appears to fix this.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Kevin Yang seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

@kevyang Can you create a jira ticket for this?


Kids: implement design system

Was fininshed in https://github.com/bcc-code/bcc-media-app/commit/6917477a759965d114e244d9b0ebb3f4defbe503


zfsUnstable: 2.1.6 → 2.1.7-staging

With 5.19 kernels now being EoL and my laptop demanding at least 5.17 to function, I want to get ahead of the curve and add support for 6.0 kernels. The 2.1.6 tag was cut so late that it supports 6.0 but isn’t as well tested according to the conversations I read. There aren’t many commits in the 2.1.7-staging at this point, but this will help keep my system afloat until the real 2.1.7 release. I’ve booted my laptop into this and everything appears to be functioning normally.
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@ofborg test zfs.unstable

@hmenke I rebased with master a couple of times as there were updates that affected kernels. I also included the latest patch added as well as fixing the date year (I put "2021").


Refactors concurrent close tests

This refactors the concurrent close tests to use barriers, as opposed to time or offsets to implement phases. In doing so, it is easier to reason with and debuggable without changing the sleep value.

thanks for letting me interrupt your weekend @mathetake!


Using ANT patterns when uploading

We have a use case to upload parts of a directory tree to Artifactory by excluding and including files and directories using ANT patterns. We need to provide a way to our users to specify multiple ANT patterns to be excluded from their upload and multiple ANT patterns to be the only files/directories included in their upload.
I'd like to see an option on the upload JFrog CLI command similar to the --exclusions option but supporting ANT patterns instead of regular expressions. I'd also like to see a similar --inclusions option that works with ANT patterns.
At the moment we are doing pattern matching ourselves, before creating a compressed archive of the directory tree and passing that to the JFrog CLI upload command. However, we are seeing bad performance with this implementation and think this can be sped up if it was implemented as part of the JFrog CLI command.
This is related to issue #867 - that is the other feature request needed for the same use case.

@mariakoleva,
Thanks for sharing this requirement!
Have you tried using the jfrog rt u command without regular expressions, meaning without the --regexp option? If you don't use this option, the command expects patterns which are very similar to ANT patterns. I believe that if you send an ANT pattern to the command, it will work as expected. This is because the double asterisk syntax of ANT patterns works seamlessly with the wildcards patent accepted by JFrpg CLI.
Would you like to try it out and let us know?

Hi Eyal,
Thank you for your reply! I had not tried that. It is great to hear this might already be in place.
I tried a couple of tests cases for exclusion that we have and noticed the below:
When using this exclude pattern “/dir//*.txt” the following doesn’t get excluded: dir/a.txt. As far as I know, ** should match 0 or more directories.
I’ve also got a question on the inclusion. Is there a way to pass in multiple include patterns? So far I’ve been able to include by using a pattern in the first argument to the upload command. But I’m not sure how to do multiple include patterns.
Thank you,
Maria
From: Eyal Ben Moshe<EMAIL_ADDRESS>Sent: 07 December 2020 18:02
To: jfrog/jfrog-cli<EMAIL_ADDRESS>Cc: Koleva, Maria (Application Infrastructure<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [jfrog/jfrog-cli] Using ANT patterns when uploading (#918)
@mariakolevahttps://github.com/mariakoleva,
Thanks for sharing this requirement!
Have you tried using the jfrog rt u command without regular expressions, meaning without the --regexp option? If you don't use this option, the command expects patterns which are very similar to ANT patterns. I believe that if you send an ANT pattern to the command, it will work as expected. This is because the double asterisk syntax of ANT patterns works seamlessly with the wildcards patent accepted by JFrpg CLI.
Would you like to try it out and let us know?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/jfrog/jfrog-cli/issues/918#issuecomment-740082940, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARPCZCQLZBNJ4QSSOR7VZMTSTUKDBANCNFSM4UQVJYQQ.

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent required and/or permitted under applicable law, to monitor electronic communications, including telephone calls with Morgan Stanley personnel. This message is subject to the Morgan Stanley General Disclaimers available at the following link: http://www.morganstanley.com/disclaimers.  If you cannot access the links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you acknowledge that you have read, understand and consent, (where applicable), to the foregoing and the Morgan Stanley General Disclaimers.
You may have certain rights regarding the information that Morgan Stanley collects about you. Please see our Privacy Pledge https://www.morganstanley.com/privacy-pledge for more information about your rights.

Hi @mariakoleva,
For passing multiple include patterns for the jfrog rt upload command , I recommend using File Specs. See Example 4: Upload, which uploads files to Artifactory from two different local paths. You can then pass the file spec path to the command using the --spac option.
As for the example you showed, in which you provided the “**/dir/**/*.txt” pattern, this is actually one of the ways ANT patterns differ from plain wildcard patterns. Changing the pattern you provided to either “dir/*.txt” or “*dir/*.txt” will match "dir/a.txt". And if you need to be more specific, you can provide multiple patterns.
Please let me know if this helps.

Hi,
Thank you for your reply. I haven't tried file specs but they look useful for passing in multiple patterns. I suppose this feature request can be amended to a request to support ANT patterns in file specs.
Unfortunately, changing from ANT patterns to plain wildcard patterns is not an option for us. We have a large number of CI jobs using ANT patterns which we are switching over to Artifactory. It's not feasible to change all of them to wildcard patterns.
Thanks,
Maria

@mariakoleva,
The latest release of JFrog CLI adds a new --ant option to the jfrog rt u command.
Looking forward to your feedback for it.

Hi @eyalbe4,
Thank you for adding this! I finally had the chance to test the new --ant option. I tested 25 use cases with it, and only found a few issues. I've raised all of those as support tickets already but I'm happy to share the ticket numbers with you if that will be helpful.

Thank you for the feedback @mariakoleva!
We'll happily add improvements and fixes following the support investigation.

As am trying this, is there support for variable matching too, e.g.
{
  "files": [
    {
      "pattern": "target/build/**/{file:\\w}*.jar",
      "target": "some/path/{file}.jar,
      "ant": true
    }
}


Setting to disable preview pane

Is your feature request related to a problem? Please describe.
My preference would be to use the space currently being used for the preview pane to display longer file names instead.
Describe the solution you'd like
I'd prefer if I could fully disable the preview part of the popup
Describe alternatives you've considered
None

Use g:clap_open_preview = 'never' in #636 .

Wow that was fast.  Thank you!


CSS

CSS
User Story
As a boot camp student
I want the  prework notes to be structured on a webpage
So that I can easily find and read the information
Acceptance Criteria
GIVEN  a prework Study Guide website
WHEN I view the study guide
THEN I see a dark blue header and footer, and four boxes with a shadow

Color was changed as a "sandbox" experiment.
Notice that the topic cards on the webpage share a new look defined by a border and shadow. Try to change the color and values for the border and shadow to experiment with how the changes affect the webpage. Take a look at the MDN Web Docs on CSS Links to an external site.to see what other properties and values there are to try. Use this project as a sandbox to play in!


Invariant Violation: Native component for "VRTText" does not exist

environment: windows.
After the project is initialized, the exception is run.

@robinv8
see the issue 177

This's the problem that I'm having
Invariant Violation: requireNativeComponent: "VRT360Image" was not found in the UIManager.

i just create a react-viro project using

react-viro init helloword

and second i run project inside

react-native run-android

app run successfully and show below screen

when i click on AR button, i am showing below ERROR screen

and   on click VR button show below ERROR screen

what`s going wrong ?
please someone help me


[Contribution] Map users, tasks, and contribution interactions

Overview
Capture the framework that identifies users, all contribution tasks, and methods of interaction.
The details
Identify the following in a grid:

the users (who)
tasks (what)
how they are interacting/contributing

Use this to map out all the flow scenarios.

Photo from June 13 brainstorm with Lauren, Anna, and Connor


Create documentation for the Mainnet Smart Contract deployment

Now that the following issue has been completed in the Vega environments we need to have the smart contract migration guide

https://github.com/vegaprotocol/devops-infra/issues/898

I think this is in addition to the usual deployment instructions, however if not we can close this issue as a duplicate of:

https://github.com/vegaprotocol/documentation/issues/133

Closing as already completed:
https://docs.vega.xyz/docs/testnet/node-operators/migration-guides/smart-contracts-migration


⚠ Bofrost.de has degraded performance

In 89a819f, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1625 ms

Resolved: Bofrost.de performance has improved in 10381c2 after 305 days, 7 hours, 52 minutes.


NRT failure due to FieldInfo & File mismatch

Description
There has been a nasty test failure in ES for awhile: https://github.com/elastic/elasticsearch/issues/105122
The test simulates a document indexing failure. It turns out, that this test failure is caused by a series of strange conditions in Lucene. If we fail on indexing a field, but have points value field that comes AFTER the field that is indexing, things will blow up when opening a reader if the writer has soft-deletes enabled.
The failure description is as follows:

First, we have an IndexWriter configured with soft-deletes & no commits on closing
Index a document with fields as follows ["field that will throw, "nice point field"]
We update the FieldInfos eagerly here: https://github.com/apache/lucene/blob/0aa88910ca9a1032d288996d14203eac4953f2de/lucene/core/src/java/org/apache/lucene/index/IndexingChain.java#L592-L603
FieldInfos now indicate we have a point field
The field that will throw is handled, document indexing fails. Since this is a regular text field, it does not automatically close the indexer
A NRT reader is opened on the writer and attempts to flush, but the field info is incorrect given the fields that are there with soft-delete (e.g. don't delete the segment)

 Test that replicates the failure

  public void testExceptionJustBeforeFlushWithPointValues() throws Exception {
    Directory dir = newDirectory();
    Analyzer analyzer =
        new Analyzer(Analyzer.PER_FIELD_REUSE_STRATEGY) {
          @Override
          public TokenStreamComponents createComponents(String fieldName) {
            MockTokenizer tokenizer = new MockTokenizer(MockTokenizer.WHITESPACE, false);
            tokenizer.setEnableChecks(
                false); // disable workflow checking as we forcefully close() in exceptional cases.
            TokenStream stream = new CrashingFilter(fieldName, tokenizer);
            return new TokenStreamComponents(tokenizer, stream);
          }
        };
    DirectoryReader r = null;
    IndexWriterConfig iwc =
        newIndexWriterConfig(analyzer).setCommitOnClose(false).setMaxBufferedDocs(3);
    MergePolicy mp = iwc.getMergePolicy();
    iwc.setMergePolicy(
        new SoftDeletesRetentionMergePolicy("soft_delete", MatchAllDocsQuery::new, mp));
    IndexWriter w = RandomIndexWriter.mockIndexWriter(dir, iwc, random());
    Document newdoc = new Document();
    newdoc.add(newTextField("crash", "do it on token 4", Field.Store.NO));
    newdoc.add(new IntPoint("int", 17));
    expectThrows(IOException.class, () -> w.addDocument(newdoc));
    try {
      r = w.getReader(false, false);
    } catch (AlreadyClosedException ace) {
      // expected
    }
    dir.close();
  }

The exception thrown is:
        Caused by:
        java.io.FileNotFoundException: No sub-file with id .kdi found in compound file "_0.cfs" (fileName=_0.kdi files: [_Lucene99_0.tip, .nvm, .fnm, .tvd, _Lucene99_0.doc, _Lucene99_0.tim, _Lucene99_0.pos, .tvm, _Lucene99_0.tmd, .fdm, .nvd, .fdx, .tvx, .fdt])
            at org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.openInput(Lucene90CompoundReader.java:170)
            at org.apache.lucene.codecs.lucene90.Lucene90PointsReader.<init>(Lucene90PointsReader.java:63)
            at org.apache.lucene.codecs.lucene90.Lucene90PointsFormat.fieldsReader(Lucene90PointsFormat.java:74)
            at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:152)
            ... 55 more

Version and environment details
No response

I am having a difficult time figuring out how to fix this. It seems to me that if the segment is "hard deleted", we should reset all its FieldInfos as there isn't any data written in it at all.
But, I am not sure the individual processDoc action can do this as it only knows about the documents it added.

What makes matters worse, is that it doesn't even have to be ALL docs that failed, just some of them that had point values (or knn vector values, etc.). Anything that eagerly updates FieldInfos but don't actually get flushed could trigger this weird behavior when opening the NRT reader.


fixed spelling of "write-XY" commands to "Write-XY"

just a little code cleanup

Awesome. Thanks @mwallner


fix(Navigation): do not race with security error for navigation

Currently, NavigationWatcher listens to lifecycle events from Page
domain and security events from Security domain.
However, the events are dispatched from different processes in browser:

Page's lifecycle events are dispatched from renderer process
Security events are dispatched from browser process

This makes for the undefined order between events and results in
NavigationWatcher reporting different failuer messages, based on
the event order.
This patch stops relying on security errors in navigation watcher and
instead switches to request failure codes for the main resource.
Fixes #1195

@Janpot this should also address your comment here  https://github.com/GoogleChrome/puppeteer/issues/901#issuecomment-333005738


No Blur

Hi,
i'm not sure if I am missing something, but my form is only transparent. There is no blur at all.
My FormBorderStyle is set to None
This is the code I tested with:
using System;
using System.Drawing;
using System.Windows.Forms;
using WinBlur;
using static WinBlur.UI;

namespace WindowsFormsApp2
{
    public partial class Popup : Form
    {

        public Popup()
        {
            InitializeComponent();
            this.BackColor = Color.Black;
            SetBlurStyle(cntrl: this, blurType: BlurType.Acrylic, designMode: Mode.DarkMode);
        }

        private void Popup_Load(object sender, EventArgs e)
        {

        }
    }
}

The X in the middle is from the popup form, which should be transparent and blurred.

Unfortunately, the blur effect works ONLY on WindowTypes other than "None". There could be a fix, I am currently seeing what I can improve in this project or fix. But, it's likely that this is something that cannot be fixed and it's a feature by Windows, where only default WindowType/Style is going to be blurred.

Btw, by WindowTypes/Style I meant FormBorderStyle. My bad.

Here's a temporary fix!
(This should be in your form btw)
protected override CreateParams CreateParams
{
    get
    {
        int WS_DLGFRAME = 0x400000;
        CreateParams result = base.CreateParams;
        result.Style &= ~WS_DLGFRAME;
        return result;
    }
}

This should allow you to use the Effect while not seeing the default border/title and implementing your own border and top bar. Btw, the FormBorderStyle should not be None, so technically this isn't a fix, but it works. I will still be looking into a way to fix this.

Thanks for your reply! I have now set my FormBorderStyle to Sizeable but I don't quite understand where exactly I should put your fix. Can you please give me a full code example as in my original message?

Here's a code example:
using System;
using System.Windows.Forms;
using WinBlur;

namespace WinBlurTest
{
    public partial class Form1 : Form
    {

        protected override CreateParams CreateParams
        {
            get
            {
                int WS_DLGFRAME = 0x400000;
                CreateParams result = base.CreateParams;
                result.Style &= ~WS_DLGFRAME;
                return result;
            }
        }

        public Form1()
        {
            InitializeComponent();
            UI.SetBlurStyle(cntrl: this, blurType: UI.BlurType.Acrylic, designMode: UI.Mode.DarkMode);
        }

        private void Form1_Load(object sender, EventArgs e)
        {

        }
    }
}

It should be at the top of the class

I've just noticed that when the Form is focused, everything is black, but when I click anywhere else, it changes its BackColor.

Thing is, I noticed you have to minimize and reopen the form for it to work. Also, perhaps there's a backcolor that's tampering with the color. You might've set the backcolor of a Panel for example as Black. Anyways, if it still doesn't work, then like I said this is a temporary solution. I will be working on a better solution and I'll update the package once I have done it.

The problem now is that even if I remove your fix, the background is still not what it should be. Now it's not even just transparent. Minimising and reopening does not help.
using System;
using System.Drawing;
using System.Windows.Forms;
using WinBlur;
using static WinBlur.UI;

namespace WindowsFormsApp2
{
    public partial class Popup : Form
    {

        public Popup()
        {
            InitializeComponent();
            SetBlurStyle(cntrl: this, blurType: BlurType.Acrylic, designMode: Mode.DarkMode);
        }

        private void Popup_Load(object sender, EventArgs e)
        {

        }
    }
}

I just realized... Are you on Windows 10?
Dude 😭 This doesn't work on Windows 10. The Blur effects have been introduced in Windows 11 and only a very very few versions of Windows 10. Perhaps the final versions, and even then, you'd have to do some work to enable the Effects. I'm sorry, but it's your OS, not the package.
Your original issue is still valid though, and I am still gonna work on fixing it.

Damn. Thats sad. You should remove the "Windows 10" in your Readme then i guess.
Do you know anything similar that works on Win10?

Well. It does work on Win10, so It's not incorrect. It's just that, it doesn't work on all versions, and you have to enable it most of the time.
And unfortunately there are no blur effects supported on Versions below 10-11, unless you create it yourself via the Form using user controls and code.

Too bad. Good luck then


[BEAM-14322] [Website] update carousel cards

[BEAM-14322]

update structure of quote cards
update scss and html styles
change cards content and images

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

R: @nausharipov

LGTM. We only need merge the change from #17205 that removes the unused const swiper.

Changes from this PR are already in https://github.com/apache/beam/pull/17205. There is no need in this PR.


put bin/azk8stool to release page

can you put bin/azk8stool to release page ,so i can download it easily?
curl -Lo /usr/local/bin/azk8spull https://github.com/xxx/azk8spull && chmod +x /usr/local/bin/azk8spull

azk8s pull xxx

OK. Release v1.0.0 is added. Please enjoy.
curl -Lo /usr/local/bin/azk8spull https://github.com/xuxinkun/littleTools/releases/download/v1.0.0/azk8spull && chmod +x /usr/local/bin/azk8spull


Best initial secret bootstrapping pattern

Is your feature request related to a problem? Please describe.
I would like to ask for recommended pattern to a chicken and an egg problem of initial secret creation. If I am using AWS as my backend I still need to create initial secret by hand, by terraform (plaintext state file), by another secret solution like SOPS etc. It does seem a bit problematic.
Describe the solution you'd like
I would like an example how to solve it in a nicer way, for example using EKS IRSA or other solutions depending on the backend.

Hey @maxkokocom, particularly for EKS: yes, you should use IRSA :100: recommended.
There are slight differences on how to integrate ESO with IRSA, the different approaches are documented here: https://external-secrets.io/latest/provider/aws-secrets-manager/#eks-service-account-credentials

I just found an Article on the AWS blog describing the process: https://aws.amazon.com/blogs/containers/leverage-aws-secrets-stores-from-eks-fargate-with-external-secrets-operator/, there are many more out the on the interwebs. You can also check out our linked blog articles: https://external-secrets.io/latest/eso-blogs/

Thank you very much!


Merging master into live

OPS Build status updates of commit d5586b0:
:clock10: Preparing: average preparing time is 46 sec(s)

OPS Build status updates of commit d5586b0:
:clock10: Incremental building: average incremental building time is 19 sec(s)

OPS Build status updates of commit d5586b0:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/faq.md
:white_check_mark:Succeeded
View

docs/integrate-model.md
:white_check_mark:Succeeded
View

docs/mlgen.md
:white_check_mark:Succeeded
View

docs/tools-and-samples.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

OPS Build status updates of commit b3629a4:
:clock10: Preparing: average preparing time is 46 sec(s)

OPS Build status updates of commit b3629a4:
:clock10: Incremental building: average incremental building time is 19 sec(s)

OPS Build status updates of commit b3629a4:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/faq.md
:white_check_mark:Succeeded
View

docs/integrate-model.md
:white_check_mark:Succeeded
View

docs/mlgen.md
:white_check_mark:Succeeded
View

docs/tools-and-samples.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[BUG]: project names are being truncated

Creating the name (2) Q&A kbirk 240616 results in project called  (2) Q

Duplicate of https://github.com/DARPA-ASKEM/terarium/issues/3639


Add set for Laravel 7.x

[x] Method rename

[x] The Blade::component method has been renamed to Blade::aliasComponent. https://github.com/laravel/framework/commit/aece7d78f3d28b2cdb63185dcc4a9b6092841310

[x] The undocumented addHidden and addVisible methods have been removed. Instead, please use the makeHidden and makeVisible methods. https://github.com/laravel/framework/pull/30348, https://github.com/laravel/framework/pull/31463

[x] Parameter type changed

[x] The report, render, shouldReport, and renderForConsole methods of your application's App\Exceptions\Handler class should accept instances of the Throwable interface instead of Exception instances. https://github.com/laravel/framework/pull/30610

[x] Argument added

[x] The resolveRouteBinding method of the Illuminate\Contracts\Routing\UrlRoutable interface now accepts a $field argument. https://github.com/laravel/framework/pull/30471

[x] Class rename

[x] The deprecated Illuminate\Http\Resources\Json\Resource class has been removed. https://github.com/laravel/framework/pull/30619

[x] The Illuminate\Foundation\Testing\TestResponse class has been renamed to Illuminate\Testing\TestResponse. The Illuminate\Foundation\Testing\Assert class has been renamed to Illuminate\Testing\Assert. https://github.com/laravel/framework/pull/31050

Thank you :+1:


EIP721

Hey all.
Here's a WIP of EIP721 along with an extensive test suite. https://github.com/ethereum/EIPs/blob/master/EIPS/eip-721.md.
EIP721 is non-fungible token standard of the Ethereum community. It contains several interfaces for implementing NFT:

A Base interface: This is all normal functionality (transferring, approving, etc).
An Enumerable Interface: This interface makes it easier to enumerate over tokens [all or owned by specific owners].
A Metadata Interface: Used for storing off-chain & on-chain information about the NFT.

The goal was to implement without too many dependencies into one specific EIP721 contract. It borrows from OpenZeppelin's implementation with various stylistic changes.
There's some things that I want to point out:
Please Double Check Conformity to EIP721 standard
Please help double check if I got all functions & interfaces & recommendations.
Naming
I spent a lot of time thinking about different naming of variables. Having to keep track of tokens using index arrays makes it complicated and sometimes cumbersome. Are the variable naming clear? If not, are there any other recommendations? I feel it could be even more clearer, but had trouble thinking up appropriate names.
EIP721 itself has inconsistent naming: eg usage of approve to denote approving one owner per token ID vs setApprovalForAll which denotes setting a specific operator to be able to transfer & approve all tokens of a specific owner.
Other inconsistencies in the standard include:

Mismatch of camelcase usage: eg tokenId vs tokenURI.
Mismatch of dangling underscore: eg, last variable here does not have a dangling underscore  function safeTransferFrom(address _from, address _to, uint256 _tokenId, bytes data) external payable;. Event logs also hate dangling underscores since eslint shouts about it.

Even though there are inconsistencies in the standard, I'd err on not changing an this, but the things we can change, I want more advice on this.
No Function To Retrieve Token Arrays
EIP721 does not specify a way to retrieve arrays of tokens. This seems like a shortsight? In the enumerable interface, it would've been nice to retrieve batches of arrays. Is this an extension we want to include perhaps? eg, return all tokens or return an owner's tokens. Otherwise, currently, you'll have to enumerate yourself with several eth_calls vs one eth_call.
Default Payable?
EIP721 by default sets transfer functions to add payable to transfer & approve functions. This does not seem the most secure to me. Should we perhaps think of forcing it to not have this by default?
Blocking onERC721Received
The standard specifies that receiver should not consumer more than 50000 gas. I find this an odd restriction.
This function MUST use 50,000 gas or less.
What are thoughts on this?
Revert Reason
During the development of this EIP721, it became possible to add reasons for reverts (eg, using revert or require). Do we want to add this? I think it could help. > 0.4.22 of Solidity. I think for now, best to keep it simple, but in the future, perhaps upgrade to add in reasons. :)
Naming Implementation
The only unimplemented interface is naming & symbols. The reason being that's specified as external pure. This means you can't even read state, so the only way to implement naming is to hardcode it in the function, which I feel is stylistically awkward. eg
function name() external pure returns (string _name) {
    return 'NFT name';
}

So, I changed to public as visibility specifier. This however does not change the interface signatures.
This PR also contains another bump of Truffle 4.1.7.
For now that it is it. I will update if more comes to mind.
Here's a pic of a puppy. NOTE: This is not a collectible. Do not attempt to buy it. ;)

Fixed @simondlr

About the gas limit in the specs I actually think that limiting variance of the gas spent in different implementations is good! 👍

Thanks @skmgoldin for finding that bug. Thanks @maurelian @GNSPS @akuanti for inputs. I fixed the bug and added some stylistic changes.

Seems there's some tiny changes in the standard coming in. Notably, 50k gas is removed (we didn't include it). https://github.com/ethereum/EIPs/issues/721#issuecomment-393212619

There's been 3 changes to the spec. Minor changes.

As expected, the ERC721 spec changed the "pure" to "view" on name & symbol. Knew that didn't make sense. Even though it's not directly utilised like that in this implementation, I will change it to reflect that.
Transfer & Approval added indexed to tokenIDs.
The 50k gas requirement was removed. I will change the comments to reflect that.

Since the last call, some changes have been made.
Notably: the on receive function added an "operator" so that the recipient knows who a token belongs to if it was transferred to them.
https://github.com/fulldecent/EIPs/commit/27788131d5975daacbab607076f2ee04624f9dbb#diff-a87271f8186c59c075e229cd7421f205
There's also small stylistic changes to documentation to improve the language and remove ambiguity. Will need to add this.

Bumped it FINAL spec on ERC721 and tidied the PR some more. Ready for final review.

Here is the updated reference implementation / https://github.com/0xcert/ethereum-erc721


[Question] Is there a way to continue scanning for devices in doze mode?

Does the Plugin.BLE library support scanning for devices in doze mode?
I'm working on a .Net7 Maui app (just Android at this stage) to create an IoT solution and have managed to successfully detect, connect, get services, and read characteristics of a device. However, whenever my app loses focus the library stops returning discovered devices. As soon as the app returns to the foreground, detections start flowing in again.
I have tried calling the library from a Foreground Service but still had no luck there. So I then tried implementing a native Android BLE scanner and was able to receive detections when my app was not in the foreground.
Ultimately I'm hoping to have the app continue detecting devices while in doze mode, and I'm hoping this is achievable using this library before I have to go and build my own library from scratch.
Thanks in advance

Hi @tmacintosh,
I'm the friendly issue checker.
It seems like (100.00 %) you haven't used our issue template :cry: I think it is very frustrating for the repository owners, if you ignore them.
If you think it's fine to make an exception, just ignore this message.
But if you think it was a mistake to delete the template, please close the issue and create a new one.
Thanks!

It should be possible to continue scanning for devices when the app is in the background or when the screen is off (both on Android and iOS), but IIRC you need to use a scan filter.
Note that the scanning rate may be reduced when the app is not in the foreground.


How to train EfficientDet-D6

When I set Hyper parameter phi as 5 or 6 ,I found that I have to decrease the batchsize to a low value (because my gpu has only 11gb mem) .
But after I modify the batchsize to 2 or 4 ,the model can not be trained normally,the loss can not decline and the eval ap is almost zero all the time.
Can you give me some suggestions?

Add option --freeze-bn when the batch size is so small like 2 and 4.

Same issue here. Have not been able to train with phi>=3, the regression head does not converge. @xuannianz Were you able to train D3 or above on VOC2007?


fix(test): fix running tests & add a sample test

When trying to run the (empty) set of test of the plugin skeleton I ran into the following error:
$ gulp test
[14:59:03] Using gulpfile ~/git/skeleton-plugin/gulpfile.js
[14:59:03] Starting 'test'...
WARN [watcher]: Pattern "/home/erik/git/skeleton-plugin/jspm_packages/es6-module-loader.js" does not match any file.
WARN [watcher]: Pattern "/home/erik/git/skeleton-plugin/test/unit/**/*.js" does not match any file.
INFO [karma]: Karma v0.12.37 server started at http://localhost:9876/
INFO [launcher]: Starting browser Chrome
INFO [Chrome 48.0.2564 (Linux 0.0.0)]: Connected on socket 6wJVRtoW44MHuJHI-uO1 with id 56321510
Chrome 48.0.2564 (Linux 0.0.0) ERROR
  Uncaught TypeError: Incorrect configuration order. The baseURL must be configured with the first System.config call.
  at /home/erik/git/skeleton-plugin/jspm_packages/system.js:4

I have updated karma-jspm with this PR to fix this issue. Also, I've added a sample test to make Karma actually run something.

Thanks for the contribution!

Are you perhaps interested in running the tests automatically with Travis CI? I've got Karma & Chrome running on that as well.


Getting VoD stream total duration from the manifest

Have you read the Tutorials?
yes
Have you read the FAQ and checked for duplicate open issues?
yes
What version of Shaka Player are you using?
3.0.3
Please ask your question
Using getManifest()?.presentationTimeline.getDuration() to get total duration of the video from the manifest.
Getting this warning in the console:
player.js:3865 Shaka Player's internal Manifest structure is NOT covered by semantic versioning compatibility guarantees.  It may change at any time!  Please consider filing a feature request for whatever you use getManifest() for.
Is there a better way to do this or should I open a feature request to expose getDuration() on a higher level?

You should be able to use mediaElement.duration, where mediaElement is a reference to the <video> or <audio> tag you're using with Shaka Player.
Does this help?

You could also use player.seekRange() to determine the end of the video.

Thanks. Both options work. I assumed video element would have only the duration of the current buffer. Clearly I have a lot more to learn about this :slightly_smiling_face:
For future reference:
player.getManifest()?.presentationTimeline.getDuration() ⚠
mediaElement.duration ✅
player.seekRange().end ✅


os-tor exit settings, plus other tweaks

Currently we don't provide settings for the following:

[x] ExitRelay (0|1|auto), current default is auto, future default will be 0, better would be to set explicitly to 1 or 0, with 0 disabling all further exit settings
[x] IPv6 exit (0|1) defaults to 0, not being able to set this makes our current choice in ACL settings useless (IPv6 settings will be ignored, since we do not exit to IPv6 anyway without setting this to 1)

Our ExitPolicy settings should be improved:

[x] we /must/ allow wildcard network exits, either by not filling in the network statement or letting the user fill in 'any', this should get translated into an asterisk symbol in the config file.

[ ] we should transition away from reject6/accept6 statements: accept/reject cover both IPv4 and v6, if adresses are explicitly given they match only their network type, for wildcard adresses that we only want to match one network type the format should be as follows "accept *4:22" or "accept *6:22"
So the user should only be presented with the choice of v4/v6 when using wildcards (otherwise adress type already does selection) and should be presented the choice between v4 and v6 together or either one seperately.

[x] Furthermore we don't supply a function to upload a DirFrontPage nor do we supply the torproject default one. We should allow users to set these in case they want to run an exit and wish to display an exitnotice.
Furthermore currently tor won't start with dirport set to port 80, due to privilege issues, though port 80 is the recommended port when running an exit and displaying an exit notice.

[ ] Lastly, currently we restart Tor when we update it's configuration, due to the nature of Tor (with longstanding running connections being preferred) it would be better to send a HUP instead of a full restart of the daemon.

@gizahNL Two open issues in this old ticket... close because not needed or will you work on it?

I can see if I can make it work in the next few days. Otherwise you may also close it. After the first minor fixes tor exit relaying has been good enough for me for now ;)

I can see if I can make it work in the next few days. Otherwise you may also close it. After the first minor fixes tor exit relaying has been good enough for me for now ;)

@gizahNL well, that's good to hear. I just want to do a litte "spring clean" on old tickets and this one caught my attention... if still valid and being worked on no problem to leave it open :)


Error: " CIT Resewn requires Cloth Config to be able to show the config.

When i try to open config of mod i see error: " CIT Resewn requires Cloth Config to be able to show the config. But I have Cloth Config.

Full list of mods (if needed)

cloth-config-8.2.88-forge.jar
....
You downloaded forge cloth


Images failing to show - due to missing file extension (.png)

Since a day or two the spincord bot - as added to our server - no longer includes a file extension - hence the response becomes a download rather than an inline cover_image
!albumart ben liebrand styles xl

Still worked on March 21, 2021. Broken shortly thereafter.
Thank you for a great project!

oh, magically resolved today :)


Forum groups view counter is broken

Time ago I reported this issue, forum chats viewed messages counter got by UpdateChatReadInbox is always different from the official Telegram app

UpdateChatReadInbox contains the number of unread messages in the "View as Messages" mode. If the mode isn't enabled for the forum, then other Telegram apps preload 20 recently updated forum topics, combine their unread counters, and show the sum as unread counter of the forum. Currently, this isn't implemented in TDLib, but can be implemented on the app's side.


Wrong unprocessed keywords: ["type"] error when using propertyNames with $ref

In my example (https://github.com/zepatrik/schemasafe-propertyNames-example) schemasafe reports an unprocessed keyword error. The problem is there is no "type" keyword at the position it is reported. I tried several different things to circumvent this error but nothing worked.
The problem seems to be rooted at using "$ref" with "propertyNames", as moving the definition into the "propertyNames" schema works.
See also https://github.com/SchemaStore/schemastore/pull/1312 where this initially came up.

@zepatrik Thanks for the testcase! Investigating.

Fixed in aa8fd5797dba4297519746d483456647a25652cb.
Thanks again!

When this will be released @ChALkeR ? :) I need that bugfix as well :)

This issue is fix and closed. But for the user of this product the issue is still not fixed because it is not yet released!


update ownCloud instance name

Description
ownCloud instance name should start with ow not mm.

I renamed the deployment initials to "oc"

Verified on devnet.
ownCloud instance name now starts with oc.