MKL provider : OSX vs Windows performance

Hello,
I've started a simple bench. I need to do lots of vector computations for a spring-mass system. For a standard system, my algorithm will run about 1e5 iterations moving about 1e4 points (x,y,z) at each iterations.
I'm investigating how the MKL provider could speed up of my computations. I'm also investigating 2 different layouts for my data structure :

Array of Structure (AoS)
Structure of Arrays (SoA)

based on A Guide to Vectorization with Intel® C++ Compilers :

The most common and likely well known data structure is the array, which contains a  contiguous collection of data items that can be accessed by an ordinal index. This data can be organized as an Array Of Structures (AOS) or a Structure Of Arrays (SOA). While AOS organization is excellent for encapsulation it can be poor for use of vector processing. Selecting appropriate data structures can also make vectorization of the resulting code more effective.

I've remarked, using the sample of code given bellow, that performance is very different between Mono Mac and Windows with the MKL provider.
The bench does pointwise multiplication of double vectors of size N = 10000.

For the AoS layout, the data is organized in a jagged array double[1000][10].
For the SoA layout, the data is organized in a single array or mathnet.vector double[10000].

Here are the results. CPU timing is given in ns per elementary operation (= total CPU time in s x 1e9 / N).
Results for MAC MONO under Yosemite (macbook core i7)
AOS (naive loop) = 6,46 ns/elop
AOS (mathnet managed) = 24,31 ns/elop
AOS (mathnet mkl) = 32,32 ns/elop
SOA (naive loop) = 4,9 ns/elop
SOA (mathnet managed) = 4,26 ns/elop
SOA (mathnet mkl) = 21,36 ns/elop

Results for Windows 7 via VMWARE (macbook core i7)
AOS (naive loop) = 7,19 ns/elop
AOS (mathnet managed) = 19,2 ns/elop
AOS (mathnet mkl) = 14,49 ns/elop
SOA (naive loop) = 4,12 ns/elop
SOA (mathnet managed) = 4,92 ns/elop
SOA (mathnet mkl) = 0,52 ns/elop

Do you know why there's a huge difference between Mac Mono and Windows with the basic pointwise multiplication (x40) ?
Is this inherent to P/Invoke with monomac ?
Is this inherent to how I build the MKL provider on OSX ?
Thanks,
Lionel
Here's my code on the pointwise multiplication of 2 vectors :
using System;
using MathNet.Numerics.LinearAlgebra;
using MathNet.Numerics;
using System.Diagnostics;

namespace TestConsoleNummerics
{
	class MainClass
	{
		public static void Main (string[] args)
		{
			//test_Matrix(1000);
			int N = 10000;
			//test_Vmul(N);
			test_Vmul_AOSvsOAS(N);
			Console.Read();
		}

		static void test_Vmul_AOSvsOAS(int N)
		{
			// Pb definition
			int loop = 10000;
			var w = Stopwatch.StartNew();
			Random rnd = new Random();

			int n = 10;
			int ne = N / n;

			// AOS : naive loop
			var aos_x = new double[ne][];
			var aos_y = new double[ne][];
			for (int i = 0; i < aos_x.Length; i++)
			{
				aos_x[i] = new double[n];
				aos_y[i] = new double[n];
				for (int j = 0; j < aos_x[i].Length; j++)
				{
					aos_x[i][j] = rnd.NextDouble();
				}
			}

			w.Restart();
			for (int k = 0; k < loop; k++)
			{
				for (int i = 0; i < aos_x.Length; i++)
				{
					for (int j = 0; j < aos_x[i].Length; j++)
					{
						aos_y[i][j] = aos_x[i][j]*aos_x[i][j];
					}
				}
			}
			Console.WriteLine("AOS (naive loop) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

			// AOS : MathManged & MKL
			var aos_mathnet_x = new Vector<double>[ne];
			var aos_mathnet_y = new Vector<double>[ne];

			for (int i = 0; i < aos_x.Length; i++)
			{
				aos_mathnet_x[i] = Vector<double>.Build.Dense(n);
				aos_mathnet_y[i] = Vector<double>.Build.Dense(n);
				for (int j = 0; j < aos_x[i].Length; j++)
				{
					aos_mathnet_x[i][j] = aos_x[i][j];
				}
			}

			Control.UseManaged();
			w.Restart();
			for (int k = 0; k < loop; k++)
			{
				for (int i = 0; i < aos_x.Length; i++)
				{
					aos_mathnet_x[i].PointwiseMultiply(aos_mathnet_x[i], aos_mathnet_y[i]);
				}
			}
			Console.WriteLine("AOS (mathnet managed) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

			Control.UseNativeMKL();
			w.Restart();
			for (int k = 0; k < loop; k++)
			{
				for (int i = 0; i < aos_x.Length; i++)
				{
					aos_mathnet_x[i].PointwiseMultiply(aos_mathnet_x[i], aos_mathnet_y[i]);
				}
			}
			Console.WriteLine("AOS (mathnet mkl) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

			// SOA : naive loop
			var soa_x = new double[N];
			var soa_y = new double[N];
			for (int i = 0; i < aos_x.Length; i++)
			{
				for (int j = 0; j < aos_x[i].Length; j++)
				{
					soa_x[n * i + j] = aos_x[i][j];
				}
			}

			w.Restart();
			for (int k = 0; k < loop; k++)
			{
				for (int i = 0; i < soa_x.Length; i++)
				{
					soa_y[i] = soa_x[i] * soa_x[i];
				}
			}
			Console.WriteLine("SOA (naive loop) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

			// SOA : MathManged & MKL
			var soa_mathnet_x = Vector<double>.Build.Random(N);
			var soa_mathnet_y = Vector<double>.Build.Dense(N);

			for (int i = 0; i < soa_x.Length; i++)
			{
				soa_mathnet_x[i] = soa_x[i];
			}

			Control.UseManaged();
			w.Restart();
			for (int i = 0; i < loop; i++)
			{
				soa_mathnet_x.PointwiseMultiply(soa_mathnet_x, soa_mathnet_y);
			}
			Console.WriteLine("SOA (mathnet managed) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

			Control.UseNativeMKL();
			w.Restart();
			for (int i = 0; i < loop; i++)
			{
				soa_mathnet_x.PointwiseMultiply(soa_mathnet_x, soa_mathnet_y);
			}
			Console.WriteLine("SOA (mathnet mkl) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

		}

	}
}

Perhaps multi-threading isn't being used for MKL on Mac? Might want to check how many threads are being set and also make sure the MKL wrapper is linking to the correct version of MKL when built on Mac.
Only thing I can think of.
-----Original Message-----
From: "Lionel du Peloux"<EMAIL_ADDRESS>Sent: 18/11/2015 9:24 PM
To: "mathnet/mathnet-numerics"<EMAIL_ADDRESS>Subject: [mathnet-numerics] MKL provider : OSX vs Windows performance (#352)
Hello,
I've started a simple bench. I need to do lots of vector computations for a spring-mass system. For a standard system, my algorithm will run about 1e5 iterations moving about 1e4 points (x,y,z) at each iterations.
I'm investigating how the MKL provider could speed up of my computations. I'm also investigating 2 different layouts for my data structure :
Array of Structure (AoS)
Structure of Arrays (SoA)
based on A Guide to Vectorization with Intel® C++ Compilers :
The most common and likely well known data structure is the array, which contains a contiguous collection of data items that can be accessed by an ordinal index. This data can be organized as an Array Of Structures (AOS) or a Structure Of Arrays (SOA). While AOS organization is excellent for encapsulation it can be poor for use of vector processing. Selecting appropriate data structures can also make vectorization of the resulting code more effective.
I've remarked, using the sample of code given bellow, that performance is very different between Mono Mac and Windows with the MKL provider.
The bench does pointwise multiplication of double vectors of size N = 10000.
For the AoS layout, the data is organized in a jagged array double[1000][10].
For the SoA layout, the data is organized in a single array or mathnet.vector double[10000].
Here are the results. CPU timing is given in ns per elementary operation (= total CPU time in s x 1e9 / N).
Results for MAC MONO under Yosemite (macbook core i7)
AOS (naive loop) = 6,46 ns/elop
AOS (mathnet managed) = 24,31 ns/elop
AOS (mathnet mkl) = 32,32 ns/elop
SOA (naive loop) = 4,9 ns/elop
SOA (mathnet managed) = 4,26 ns/elop
SOA (mathnet mkl) = 21,36 ns/elop
Results for Windows 7 via VMWARE (macbook core i7)
AOS (naive loop) = 7,19 ns/elop
AOS (mathnet managed) = 19,2 ns/elop
AOS (mathnet mkl) = 14,49 ns/elop
SOA (naive loop) = 4,12 ns/elop
SOA (mathnet managed) = 4,92 ns/elop
SOA (mathnet mkl) = 0,52 ns/elop
Do you know why there's a huge difference between Mac Mono and Windows with the basic pointwise multiplication (x40) ?
Is this inherent to P/Invoke with monomac ?
Is this inherent to how I build the MKL provider on OSX ?
Thanks,
Lionel
Here's my code on the pointwise multiplication of 2 vectors :
using System;
using MathNet.Numerics.LinearAlgebra;
using MathNet.Numerics;
using System.Diagnostics;
namespace TestConsoleNummerics
{
class MainClass
{
public static void Main (string[] args)
{
//test_Matrix(1000);
int N = 10000;
//test_Vmul(N);
test_Vmul_AOSvsOAS(N);
Console.Read();
}
    static void test_Vmul_AOSvsOAS(int N)
    {
        // Pb definition
        int loop = 10000;
        var w = Stopwatch.StartNew();
        Random rnd = new Random();

        int n = 10;
        int ne = N / n;

        // AOS : naive loop
        var aos_x = new double[ne][];
        var aos_y = new double[ne][];
        for (int i = 0; i < aos_x.Length; i++)
        {
            aos_x[i] = new double[n];
            aos_y[i] = new double[n];
            for (int j = 0; j < aos_x[i].Length; j++)
            {
                aos_x[i][j] = rnd.NextDouble();
            }
        }

        w.Restart();
        for (int k = 0; k < loop; k++)
        {
            for (int i = 0; i < aos_x.Length; i++)
            {
                for (int j = 0; j < aos_x[i].Length; j++)
                {
                    aos_y[i][j] = aos_x[i][j]*aos_x[i][j];
                }
            }
        }
        Console.WriteLine("AOS (naive loop) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        // AOS : MathManged & MKL
        var aos_mathnet_x = new Vector<double>[ne];
        var aos_mathnet_y = new Vector<double>[ne];

        for (int i = 0; i < aos_x.Length; i++)
        {
            aos_mathnet_x[i] = Vector<double>.Build.Dense(n);
            aos_mathnet_y[i] = Vector<double>.Build.Dense(n);
            for (int j = 0; j < aos_x[i].Length; j++)
            {
                aos_mathnet_x[i][j] = aos_x[i][j];
            }
        }

        Control.UseManaged();
        w.Restart();
        for (int k = 0; k < loop; k++)
        {
            for (int i = 0; i < aos_x.Length; i++)
            {
                aos_mathnet_x[i].PointwiseMultiply(aos_mathnet_x[i], aos_mathnet_y[i]);
            }
        }
        Console.WriteLine("AOS (mathnet managed) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        Control.UseNativeMKL();
        w.Restart();
        for (int k = 0; k < loop; k++)
        {
            for (int i = 0; i < aos_x.Length; i++)
            {
                aos_mathnet_x[i].PointwiseMultiply(aos_mathnet_x[i], aos_mathnet_y[i]);
            }
        }
        Console.WriteLine("AOS (mathnet mkl) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        // SOA : naive loop
        var soa_x = new double[N];
        var soa_y = new double[N];
        for (int i = 0; i < aos_x.Length; i++)
        {
            for (int j = 0; j < aos_x[i].Length; j++)
            {
                soa_x[n * i + j] = aos_x[i][j];
            }
        }

        w.Restart();
        for (int k = 0; k < loop; k++)
        {
            for (int i = 0; i < soa_x.Length; i++)
            {
                soa_y[i] = soa_x[i] * soa_x[i];
            }
        }
        Console.WriteLine("SOA (naive loop) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        // SOA : MathManged & MKL
        var soa_mathnet_x = Vector<double>.Build.Random(N);
        var soa_mathnet_y = Vector<double>.Build.Dense(N);

        for (int i = 0; i < soa_x.Length; i++)
        {
            soa_mathnet_x[i] = soa_x[i];
        }

        Control.UseManaged();
        w.Restart();
        for (int i = 0; i < loop; i++)
        {
            soa_mathnet_x.PointwiseMultiply(soa_mathnet_x, soa_mathnet_y);
        }
        Console.WriteLine("SOA (mathnet managed) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        Control.UseNativeMKL();
        w.Restart();
        for (int i = 0; i < loop; i++)
        {
            soa_mathnet_x.PointwiseMultiply(soa_mathnet_x, soa_mathnet_y);
        }
        Console.WriteLine("SOA (mathnet mkl) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

    }

}

}
—
Reply to this email directly or view it on GitHub.

If it is a p/invoke problem then you could try putting your loop into C/C++ and call the MKL wrapper from there. Then there'd only be a single p/invoke call.
-----Original Message-----
From: "Lionel du Peloux"<EMAIL_ADDRESS>Sent: 18/11/2015 9:24 PM
To: "mathnet/mathnet-numerics"<EMAIL_ADDRESS>Subject: [mathnet-numerics] MKL provider : OSX vs Windows performance (#352)
Hello,
I've started a simple bench. I need to do lots of vector computations for a spring-mass system. For a standard system, my algorithm will run about 1e5 iterations moving about 1e4 points (x,y,z) at each iterations.
I'm investigating how the MKL provider could speed up of my computations. I'm also investigating 2 different layouts for my data structure :
Array of Structure (AoS)
Structure of Arrays (SoA)
based on A Guide to Vectorization with Intel® C++ Compilers :
The most common and likely well known data structure is the array, which contains a contiguous collection of data items that can be accessed by an ordinal index. This data can be organized as an Array Of Structures (AOS) or a Structure Of Arrays (SOA). While AOS organization is excellent for encapsulation it can be poor for use of vector processing. Selecting appropriate data structures can also make vectorization of the resulting code more effective.
I've remarked, using the sample of code given bellow, that performance is very different between Mono Mac and Windows with the MKL provider.
The bench does pointwise multiplication of double vectors of size N = 10000.
For the AoS layout, the data is organized in a jagged array double[1000][10].
For the SoA layout, the data is organized in a single array or mathnet.vector double[10000].
Here are the results. CPU timing is given in ns per elementary operation (= total CPU time in s x 1e9 / N).
Results for MAC MONO under Yosemite (macbook core i7)
AOS (naive loop) = 6,46 ns/elop
AOS (mathnet managed) = 24,31 ns/elop
AOS (mathnet mkl) = 32,32 ns/elop
SOA (naive loop) = 4,9 ns/elop
SOA (mathnet managed) = 4,26 ns/elop
SOA (mathnet mkl) = 21,36 ns/elop
Results for Windows 7 via VMWARE (macbook core i7)
AOS (naive loop) = 7,19 ns/elop
AOS (mathnet managed) = 19,2 ns/elop
AOS (mathnet mkl) = 14,49 ns/elop
SOA (naive loop) = 4,12 ns/elop
SOA (mathnet managed) = 4,92 ns/elop
SOA (mathnet mkl) = 0,52 ns/elop
Do you know why there's a huge difference between Mac Mono and Windows with the basic pointwise multiplication (x40) ?
Is this inherent to P/Invoke with monomac ?
Is this inherent to how I build the MKL provider on OSX ?
Thanks,
Lionel
Here's my code on the pointwise multiplication of 2 vectors :
using System;
using MathNet.Numerics.LinearAlgebra;
using MathNet.Numerics;
using System.Diagnostics;
namespace TestConsoleNummerics
{
class MainClass
{
public static void Main (string[] args)
{
//test_Matrix(1000);
int N = 10000;
//test_Vmul(N);
test_Vmul_AOSvsOAS(N);
Console.Read();
}
    static void test_Vmul_AOSvsOAS(int N)
    {
        // Pb definition
        int loop = 10000;
        var w = Stopwatch.StartNew();
        Random rnd = new Random();

        int n = 10;
        int ne = N / n;

        // AOS : naive loop
        var aos_x = new double[ne][];
        var aos_y = new double[ne][];
        for (int i = 0; i < aos_x.Length; i++)
        {
            aos_x[i] = new double[n];
            aos_y[i] = new double[n];
            for (int j = 0; j < aos_x[i].Length; j++)
            {
                aos_x[i][j] = rnd.NextDouble();
            }
        }

        w.Restart();
        for (int k = 0; k < loop; k++)
        {
            for (int i = 0; i < aos_x.Length; i++)
            {
                for (int j = 0; j < aos_x[i].Length; j++)
                {
                    aos_y[i][j] = aos_x[i][j]*aos_x[i][j];
                }
            }
        }
        Console.WriteLine("AOS (naive loop) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        // AOS : MathManged & MKL
        var aos_mathnet_x = new Vector<double>[ne];
        var aos_mathnet_y = new Vector<double>[ne];

        for (int i = 0; i < aos_x.Length; i++)
        {
            aos_mathnet_x[i] = Vector<double>.Build.Dense(n);
            aos_mathnet_y[i] = Vector<double>.Build.Dense(n);
            for (int j = 0; j < aos_x[i].Length; j++)
            {
                aos_mathnet_x[i][j] = aos_x[i][j];
            }
        }

        Control.UseManaged();
        w.Restart();
        for (int k = 0; k < loop; k++)
        {
            for (int i = 0; i < aos_x.Length; i++)
            {
                aos_mathnet_x[i].PointwiseMultiply(aos_mathnet_x[i], aos_mathnet_y[i]);
            }
        }
        Console.WriteLine("AOS (mathnet managed) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        Control.UseNativeMKL();
        w.Restart();
        for (int k = 0; k < loop; k++)
        {
            for (int i = 0; i < aos_x.Length; i++)
            {
                aos_mathnet_x[i].PointwiseMultiply(aos_mathnet_x[i], aos_mathnet_y[i]);
            }
        }
        Console.WriteLine("AOS (mathnet mkl) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        // SOA : naive loop
        var soa_x = new double[N];
        var soa_y = new double[N];
        for (int i = 0; i < aos_x.Length; i++)
        {
            for (int j = 0; j < aos_x[i].Length; j++)
            {
                soa_x[n * i + j] = aos_x[i][j];
            }
        }

        w.Restart();
        for (int k = 0; k < loop; k++)
        {
            for (int i = 0; i < soa_x.Length; i++)
            {
                soa_y[i] = soa_x[i] * soa_x[i];
            }
        }
        Console.WriteLine("SOA (naive loop) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        // SOA : MathManged & MKL
        var soa_mathnet_x = Vector<double>.Build.Random(N);
        var soa_mathnet_y = Vector<double>.Build.Dense(N);

        for (int i = 0; i < soa_x.Length; i++)
        {
            soa_mathnet_x[i] = soa_x[i];
        }

        Control.UseManaged();
        w.Restart();
        for (int i = 0; i < loop; i++)
        {
            soa_mathnet_x.PointwiseMultiply(soa_mathnet_x, soa_mathnet_y);
        }
        Console.WriteLine("SOA (mathnet managed) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

        Control.UseNativeMKL();
        w.Restart();
        for (int i = 0; i < loop; i++)
        {
            soa_mathnet_x.PointwiseMultiply(soa_mathnet_x, soa_mathnet_y);
        }
        Console.WriteLine("SOA (mathnet mkl) = " + w.ElapsedMilliseconds * 1e6 / (loop * N) + " ns/elop");

    }

}

}
—
Reply to this email directly or view it on GitHub.


Quadratic equation solver.

takes in the co-efficients of a quadratic equation and return the answers

@TosinAina Please re-run this and ascertain the correctness of the code.

Done @Precillieo.


Continue working with Nodejs Servers

Challenge Continue working with Nodejs Servers has an issue.
User Agent is: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) coc_coc_browser/55.2.126 Chrome/49.2.2623.126 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.

@thuyetbk, if this is a legitimate issue, please provide more details.


parallel upload chunks [SITL-248]

Running with one transfer:
$ time ESTHRI_XFER_COUNT=1 cargo run -- put 256MiB.bin --bucket esthri-test --key 256MiB.bin
...

real    0m13.340s
user    0m0.647s
sys     0m0.212s

Running with the default (16) transfers:
$ time cargo run -- put 256MiB.bin --bucket esthri-test --key 256MiB.bin
...

real    0m4.200s
user    0m0.767s
sys     0m0.359s

Running with the old code:
$ time cargo run -- put 256MiB.bin --bucket esthri-test --key 256MiB.bin
...

real    0m44.631s
user    0m0.567s
sys     0m0.318s

:partying_face:
How does it compare to the aws cli now?

On par with awscli now for upload, still need to look at download and sync speeds.


Remove charts directory

This was used by the chart generator script, but the Jupyter notebook uses the images directory
Missed in d0ad3f61

Confirmed by the rebuild-notebook CI step.


download update-from-repo.sh from my server

rawgit seems to do not like requesting files often, it blocks on my server.
So I will place latest version of update-from-repo.sh to my server
https://www.rrr.de/gnadelwartz/update-from-repo.sh

Oh, no, no. We shouldn't be doing it.

but it will not work, try from host not on your network where you not already logged in on github

I just did:
curl -s -o update-from-repo2.sh https://raw.githubusercontent.com/webmin/webmin/master/update-from-repo.sh

and
wget https://raw.githubusercontent.com/webmin/webmin/master/update-from-repo.sh


calico should remove the virtual interface when failing to enter pod netns in cmdDel.

Expected Behavior

calico should remove the virtual interface in the host side when failing to enter pod netns in cmdDel.
Current Behavior

diff --git a/cni-plugin/pkg/dataplane/linux/dataplane_linux.go b/cni-plugin/pkg/dataplane/linux/dataplane_linux.go
index 3f9b89655..c6c2154f3 100644
--- a/cni-plugin/pkg/dataplane/linux/dataplane_linux.go
+++ b/cni-plugin/pkg/dataplane/linux/dataplane_linux.go
@@ -17,6 +17,7 @@ package linux
 import (
        "context"
        "fmt"
+       k8sconversion "github.com/projectcalico/calico/libcalico-go/lib/backend/k8s/conversion"
        "io"
        "net"
        "os"
@@ -551,6 +552,13 @@ func (d *linuxDataplane) CleanUpNamespace(args *skel.CmdArgs) error {
                if nsErr != nil {
                        if _, ok := nsErr.(ns.NSPathNotExistErr); ok {
                                logCtx.Info("Workload's netns already gone.  Nothing to do.")
+                               hostVethName := k8sconversion.NewConverter().VethNameForWorkload(podNs,podName)
+                               link,err := netlink.LinkByName(hostVethName)
+                               if err != nil {
+                                       d.logger.WithField("virtual interface",hostVethName).Errorf("LinktByName return error: %v",err)
+                               } else {
+                                       d.logger.WithField("virtual interface",hostVethName).Infof("found interface %s exists",link.Attrs().Name)
+                               }
                                return nil
                        }
                        logCtx.WithError(nsErr).Error("Failed to enter workloads netns.")

2023-04-14 16:15:44.944 [INFO][2138204] dataplane_linux.go 504: Deleting workload's device in netns. ContainerID="75d3f57abfcc79c2985911f4426137f01a64fc89d033ef3d95fedd2dfe8b8a10" iface="eth0" netns="/proc/2137890/ns/net"
2023-04-14 16:15:44.944 [INFO][2138204] dataplane_linux.go 515: Entered netns, deleting veth. ContainerID="75d3f57abfcc79c2985911f4426137f01a64fc89d033ef3d95fedd2dfe8b8a10" iface="eth0" netns="/proc/2137890/ns/net"
2023-04-14 16:15:45.454 [INFO][2138204] dataplane_linux.go 542: Workload's veth was already gone.  Nothing to do. ContainerID="75d3f57abfcc79c2985911f4426137f01a64fc89d033ef3d95fedd2dfe8b8a10" iface="eth0" netns="/proc/2137890/ns/net"
2023-04-14 16:15:45.517 [INFO][2138204] dataplane_linux.go 558: found interface cali757eb1af9e1 exists ContainerID="75d3f57abfcc79c2985911f4426137f01a64fc89d033ef3d95fedd2dfe8b8a10" virtual interface="cali757eb1af9e1"
2023-04-14 16:15:45.832 [INFO][2138204] k8s.go 583: Releasing IP address(es) ContainerID="75d3f57abfcc79c2985911f4426137f01a64fc89d033ef3d95fedd2dfe8b8a10"
2023-04-14 16:15:46.366 [INFO][2138204] utils.go 196: Calico CNI releasing IP address ContainerID="75d3f57abfcc79c2985911f4426137f01a64fc89d033ef3d95fedd2dfe8b8a10"

In my local, I created a job, and the job created 50+ pods, When the status of the pod changed to Completed, I found that some virtual NICs were not recycled in time.  as shown above.
There is a case when the netns file does not exist, but the pod's netns are not deleted. I'm not sure why this happens,  A similar problem was encountered in our customer environment. It resulted in a large number of virtual NICs on the host and a very high CPU on the calico-node.
In this case, should we actively delete the virtual NIC on the host side？
Possible Solution

Steps to Reproduce (for bugs)

container runtime is docker

create a job, and set parallelism to 40

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl:5.34.0
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4
  parallelism: 50

Insert debug code in calico cni-plugin to check if the virtual NIC on the host side exists when netns is inaccessible
Wait for state of all pods to change to Completed, Check out calico cni's log(/var/log/calico/cni/cni.log)

Context

https://github.com/projectcalico/calico/pull/6356
Your Environment

Calico version

root@controller:~# calicoctl version
Client Version:    v3.23.3
Git commit:        3a3559be1
Cluster Version:   v3.23.3
Cluster Type:      k8s,kdd,kubespray,kubeadm

Orchestrator version (e.g. kubernetes, mesos, rkt):

root@controller:~# kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.0
Kustomize Version: v4.5.7
Server Version: v1.25.0

Operating System and version:

root@controller:~# uname -a
Linux controller 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Link to your project (optional):

It would be better if you could provide the k8s and calico version information.

We hit this issue in our product, and the version of the production is old, k8s version is 1.18, calico version is 1.16.
But I reproduced the problem on new version of k8s(1.24.7) and calico version(v3.24.5), so I opened this issue.

It is worth noting that this problem does not happen if containerd is used as a container runtime, only happened in docker as container runtime

@cyclinder please fill in all the details asked in the template so we can help. Cluster information and the steps for reproducing the issue are important.

I thought the veth was auto-deleted when the namespace it was in was deleted.  ISTR there is a reason for deleting hte veth from within the netns: it makes sure that we delete the right veth if the pod is being restarted or something like that.

@mazdakn Thanks, updated.

I thought the veth was auto-deleted when the namespace it was in was deleted.

Yes, I think so too, I don't found veth residue in my local. In a production environment, our customer creates AI pods at regular intervals of 1 minute via several cron jobs, After a period of time, a large number of veth residues were found on the node. This causes the calico node's cpu usage to reach over 90%.

ISTR there is a reason for deleting hte veth from within the netns: it makes sure that we delete the right veth if the pod is being restarted or something like that.

In scenarios with high concurrency, it is indeed possible to cause a large number of veth residues, and it is not clear what the root cause is. Should we remove the veth device on the host side when netns is inaccessible?

This causes the calico node's cpu usage to reach over 90%.

Why does the residual veths cause high CPU usage?

Should we remove the veth device on the host side when netns is inaccessible?

I think so. These veths have become dirty data when the netns was deleted.
containerd will be responsble for the lifecyle of pod's netns if your runtime is containerd.
I think it's better to use containerd instead of docker if your k8s version is higher than 1.24. Your problem may be caused by compatibility issues with Docker, I guess.

Why does the residual veths cause high CPU usage?

We found more than 1000 veth devices in the host, but in fact, only 100+ pods exist in the host.  I saw logs of calico-node and found calico-node always checking veth interface...

2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali8f52b63ed52"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali33b8dac7e81"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali03b5de578f2"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali09b03079c56"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali01ada96bea7"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali40325d8945d"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="calie95891e0b77"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali3fd3f9001a9"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali3317d7aa695"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali9383850f54b"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="calib6ecd0f1940"
2023-03-15 11:15:20.826 [INFO][86] monitor-addresses/reachaddr.go 55: Checking interface CIDRs Name="cali55fd3e317d7"

When we clean these dirty veth devices manually, the cpu usage is normal

FWIW, Calico v3.25.1 should fix the high CPU usage.  Then the question is: does Calico get told about the deletion of the pod?  Is kubelet issuing a CNI DEL operation to Calico?  If not, then it's a kubelet/CRI/dockershim bug.  Sicne this seems to eb fixed in containerd and dockershim is deprecated, it seems likely to be a k8s/dockershim bug, not telling Calico to clean up?

FWIW, Calico v3.25.1 should fix the high CPU usage.

Could you tell me which one PR fixed it?

Then the question is: does Calico get told about the deletion of the pod? Is kubelet issuing a CNI DEL operation to Calico?

Yes, kubelet has told calico about the deletion of the pod,  but calico failed to enter pod's netns, so it failed. So the veth on the node cannot be deleted. It looks like the deletion of netns and the deletion of /proc//net/ns files are not synchronized, causing this problem.
I know the veth was auto-deleted when the namespace it was in was deleted, but you can't be sure when namespace is actually deleted, it could be very fast or very slow. In our production (high concurrency), the namespace deletion took longer, so it resulted in veth residuals.
So I think we should remove the veth on host side when netns  path is unreachable.


Recurring task completion generates a Creation date on the new task when it shouldn't

Bug Report
IMPORTANT:
Please follow this template strictly when reporting bugs. Providing detailed and accurate information helps the development team to understand and address the issue effectively.
App Version: [2.0.0-dev13]
Platform: [Windows]
Installation Method: [Direct Download]
Bug Description:
The Settings dialog says "Append creation date when todo is created".  I have that turned off.  When I complete a recurring task, the newly created task has the creation date at the front of the task.
Steps to Reproduce:

Turn Append creation date off in settings
Create a daily recurring event
--> Verify the creation date wasn't prepended
Complete that task
--> Check the todo.txt file and you'll see the task now has a creation date prepended.

Expected Behavior:
When a recurring task is completed, the newly created future task should not have the creation date, since I have that setting turned off in Settings.
Actual Behavior:
All future tasks will have the creation date
Additional Information:

[Include any additional information that might be relevant to the bug, such as error messages or console logs.]

Screenshots:
[If applicable, include screenshots that demonstrate the bug.]
System Information:

OS Version: Windows 10 and 11
Processor: Intel
RAM: 16GB & 64GB
Graphics Card: Intel(R) Iris(R) Xe Graphics Family, Intel Corporation compatible

Reproducibility:

[Always]

Impact:

[Moderate]

Workaround:
None

@Piscean could you please check if this still occurs in https://github.com/ransome1/sleek/releases/tag/v2.0.0-dev15?

Okay, that makes sense.  Thank you.  Please consider this solved and close the bug.
Eric

From: ransome @.>
Sent: Tuesday, October 10, 2023 11:00 AM
To: ransome1/sleek @.>
Cc: Eric @.>; Mention @.>
Subject: Re: [ransome1/sleek] Recurring task completion generates a Creation date on the new task when it shouldn't (Issue #539)
However, now the completed task has the completion date listed twice.
That is not a double completion date, it is actually creation and completion date. Unfortunately the todo.txt parser (jsTodoTxt in the latest version) refuses to add a completion date, if there is no creation date. As a workaround, everytime when a todo is marked as complete, a creation date needs to be added by sleek. So in case a todo did not already have creation date, todays date will be added. As long as this behavior doesn't change upstream, we need to live with this workaround.
The issue has been raised herehttps://github.com/jmhobbs/jsTodoTxt/issues/50.
—
Reply to this email directly, view it on GitHubhttps://github.com/ransome1/sleek/issues/539#issuecomment-1755958178, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAQUJ3LTPUKG67KOCPQBMK3X6WEMHAVCNFSM6AAAAAA5OW4HBGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJVHE2TQMJXHA.
You are receiving this because you were mentioned.Message ID: @.***>


Lazy loader for zip images

Loading animations from zips can take a lot of time when there are a lot of frames inside a zip. In my tests around 1ms per frame. However, we only need the first frame initially.
This change provides a lazy loader for images from within a zip file. Currently, it will load images + textures on demand. Initially, I though about preloading the next one or two images but it turned out that this is fast enough for all our cases. I might still add it later.

@tito @akshayaurora are you still planning to review this or should i close it?


Redeem maker shares in continuation via queue

We may want to add a new system level timeout to the bet queue so that these redemption transactions wait as long as they need to in order to run. We probably also want to run the bettor's redeemShares in the continuation so their bet returns faster and the lock on the contract frees sooner.

@devloai -- please review this PR


New release "0.1.1"

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/113196
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.1 -m "<description of version>" bacc4b86a3dc7205abc531bf8add1499f6fc6c61
git push origin v0.1.1


Add clientTimeout to peer forwarder configuration, optimize CPF seria…

…lization
Signed-off-by: Chase Engelbrecht<EMAIL_ADDRESS>Description
This PR adds a new CPF configuration option for client timeout. This aimed at reducing the ResponseTimeoutExceptions received when the CPF server is overloaded with requests.
The server requestTimeout is set to 1s above the configured value to give overhead for the processing time outside of the buffer write.
The serialization on the client side is change from a String to byte[] for a slight performance boost and less memory consumption.
Check List

[ ] New functionality includes testing.
[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #2190 (0cab49a) into main (5f492a5) will increase coverage by 0.30%.
The diff coverage is 89.47%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #2190      +/-   ##
============================================
+ Coverage     93.78%   94.09%   +0.30%
- Complexity     1635     1699      +64
============================================
  Files           204      215      +11
  Lines          4747     4911     +164
  Branches        381      393      +12
============================================
+ Hits           4452     4621     +169
+ Misses          207      198       -9
- Partials         88       92       +4

Impacted Files
Coverage Δ

...pper/peerforwarder/PeerForwarderConfiguration.java
87.67% <75.00%> (-0.50%)
:arrow_down:

...arch/dataprepper/peerforwarder/PeerClientPool.java
100.00% <100.00%> (ø)

...pper/peerforwarder/PeerForwarderClientFactory.java
95.23% <100.00%> (ø)

...pper/peerforwarder/client/PeerForwarderClient.java
88.37% <100.00%> (ø)

...warder/server/PeerForwarderHttpServerProvider.java
85.71% <100.00%> (ø)

...peerforwarder/server/PeerForwarderHttpService.java
88.46% <100.00%> (ø)

...pensearch/dataprepper/model/metric/JacksonSum.java
100.00% <0.00%> (ø)

...nsearch/dataprepper/model/metric/JacksonGauge.java
100.00% <0.00%> (ø)

...earch/dataprepper/model/buffer/AbstractBuffer.java
100.00% <0.00%> (ø)

...earch/dataprepper/model/metric/JacksonSummary.java
100.00% <0.00%> (ø)

... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


::Mimemagic.by_magic() => nil ... always, but by_path returns a result

It appears by_magic does not work - it ALWAYS returns a nil no matter what the arguments are.

Is this still the case? Maybe your file does not match any magic. Did you test some common file types which are detected correctly by file?

Is this still the case? Maybe your file does not match any magic. Did you test some common file types which are detected correctly by file?


Explore using DocDb for warm exploration storage

We are going to start off using ElasticSearch, but we think that DocDb might be a good fit for this scenario.
We need to do some research and experimentation. and document our findinings.

Azure DocumentDB limits and quotas:
https://azure.microsoft.com/en-us/documentation/articles/documentdb-limits/
Maximum Request Units / second per collection: 2500

In order to scale up, it looks like you would need to partition the full data stream into multiple collections. This would complicate queries which would then need to run across collections.

We shoud verify what **Maximum Request Units ** really means.

Also, can this limit be overcome through batching?


SCT-1682 Queue Status Page

I used the kbaseCatalogStats.js and catalog_stats.js as a model for these edits.

Nice improvements.
Some general observations:
Should the queue status be available on the main catalog menu or via the jobs plugin (which really just steals parts of the catalog plugin)? As a regular user the "Index" menu option which is used to get to the big honking catalog index is not available. One CAN however get to the queue status by going to the jobs browser and navigating around a bit until the index menu reveals itself (surely a bug).
I think the expanded Catalog menu is triggered by the user being a registered developer. This is probably one reason the jobs browser exists at all -- since it is not easily available through the catalog ui for regular users.
Another issue is that the Jobs Status table is revealed for non-admins.
Just to check myself -- is it true that the queue status is for all kbase users and the jobs status is just for admins? I make that presumption since the jobs status header reads "(Administrator)"
Finally, at the moment there is an error in the DS call for condor stats (no such item for Cursor instance), but the ui just shows empty tables. Perhaps replace the tables with an error message?
Now a few comments about the specific tables.
generally, any ui issues with the admin tool are less important than the user facing one
Queue Status:

header could align better with columns

representation of queue with no available slots might call for something special- presumably it is an outage of some sort. Currently it shows as 100% utilization with a red background.

are there plans to define the queues and terms (column headers)? If so there could be tooltips or other help available on this page, and/or links to kbase.us documentation. I didn't find anything in kbase.us when searching for the queue names.

Job Status:

may not be necessary to show the precise timestamp for submission time. There are ways to represent past timestamps somewhat compactly and helpfully. E.g. "5 minutes ago", "3 days ago", "Feb 3", "Dec 10, 2018". The precise time of day down to the second is perhaps not helpful at a glance, although one could inspect it (mouseover, modal popup) to get more detail.

table header still breaks apart at about 1400px width; tables are indeed cranky!

Should the queue status be available on the main catalog menu or via the jobs plugin (which really just steals parts of the catalog plugin)? As a regular user the "Index" menu option which is used to get to the big honking catalog index is not available. One CAN however get to the queue status by going to the jobs browser and navigating around a bit until the index menu reveals itself (surely a bug). I think the expanded Catalog menu is triggered by the user being a registered developer. This is probably one reason the jobs browser exists at all -- since it is not easily available through the catalog ui for regular users.

I could use your advice on how to proceed with this one.

Another issue is that the Jobs Status table is revealed for non-admins.
Just to check myself -- is it true that the queue status is for all kbase users and the jobs status is just for admins? I make that presumption since the jobs status header reads "(Administrator)"

This is intentional.   Queue status and Jobs status should be exposed to all users.  Only other people's usernames (and another endpoint for the condor stats service) are exposed for catalog admins.

Finally, at the moment there is an error in the DS call for condor stats (no such item for Cursor instance), but the ui just shows empty tables. Perhaps replace the tables with an error message?

Great idea. I didn't think of this possibility until the service actually was busted.

generally, any ui issues with the admin tool are less important than the user facing one

Very true

Queue Status:
header could align better with columns

I'll try to work on this and try to figure out the css issues

representation of queue with no available slots might call for something special- presumably it is an outage of some sort. Currently it shows as 100% utilization with a red background.

Here's something a bit more special

are there plans to define the queues and terms (column headers)? If so there could be tooltips or other help available on this page, and/or links to kbase.us documentation. I didn't find anything in kbase.us when searching for the queue names.

That's a good idea.  I wonder what we should tell the user's about the queues.  Maybe they should be renamed for the user's consumption, like they are named here: https://kbase.github.io/kb_sdk_docs/references/execution_engine.html
Job Status:

may not be necessary to show the precise timestamp for submission time. There are ways to represent past timestamps somewhat compactly and helpfully. E.g. "5 minutes ago", "3 days ago", "Feb 3", "Dec 10, 2018". The precise time of day down to the second is perhaps not helpful at a glance, although one could inspect it (mouseover, modal popup) to get more detail.

good point, I will remove the seconds.. I  have the service set to update every minute or so, so I could instead change it to static text that says "Updated 1 minute ago"

table header still breaks apart at about 1400px width; tables are indeed cranky!

Do you have any recommendations on how to fix this? Ideas include having some javascript that runs and tries to set a CSS property for the table headers and makes them a fixed width? Maybe I can try to modify the dynamic table widget ?  Or maybe I try to forgo the dynamic table widget and use datatables on my own so I can configure the CSS my way?
@eapearson Thanks for the great feedback and suggestions

Should the queue status be available on the main catalog menu or via the jobs plugin (which really just steals parts of the catalog plugin)? As a regular user the "Index" menu option which is used to get to the big honking catalog index is not available. One CAN however get to the queue status by going to the jobs browser and navigating around a bit until the index menu reveals itself (surely a bug). I think the expanded Catalog menu is triggered by the user being a registered developer. This is probably one reason the jobs browser exists at all -- since it is not easily available through the catalog ui for regular users.

I could use your advice on how to proceed with this one.

I hate to bounce this back to others, but I think this is an issue for the product owner. There are subjective decisions here. One approach is to support separate sets of tools for different classes of user - e.g. regular user, developer, admin (leaving aside for the moment that each service defines its own class of users, and we have the auth service for roles, ...). Another is to have the tool adjust to the class of the user, disabling or enabling features. In this case we are actually doing both, and that may be unavoidable. I guess I would put more weight on the utility of the end product and the user experience the internal engineering concerns like code duplication, which can be addressed by various means.

Another issue is that the Jobs Status table is revealed for non-admins.
Just to check myself -- is it true that the queue status is for all kbase users and the jobs status is just for admins? I make that presumption since the jobs status header reads "(Administrator)"

This is intentional. Queue status and Jobs status should be exposed to all users. Only other people's usernames (and another endpoint for the condor stats service) are exposed for catalog admins.

Okay, cool. So the "(Administrator)" should probably be removed?

Finally, at the moment there is an error in the DS call for condor stats (no such item for Cursor instance), but the ui just shows empty tables. Perhaps replace the tables with an error message?

Great idea. I didn't think of this possibility until the service actually was busted.

I know - busted services are always a great time to work on error handling :)

generally, any ui issues with the admin tool are less important than the user facing one

Very true

Queue Status:
header could align better with columns

I'll try to work on this and try to figure out the css issues

representation of queue with no available slots might call for something special- presumably it is an outage of some sort. Currently it shows as 100% utilization with a red background.

Here's something a bit more special

Perfect!
It reminded me of the lyrics "nothing from nothing leaves nothing"...

are there plans to define the queues and terms (column headers)? If so there could be tooltips or other help available on this page, and/or links to kbase.us documentation. I didn't find anything in kbase.us when searching for the queue names.

That's a good idea. I wonder what we should tell the user's about the queues. Maybe they should be renamed for the user's consumption, like they are named here: https://kbase.github.io/kb_sdk_docs/references/execution_engine.html

Yeah, it would be great to tie the queues into defined policies like that. Regular kbase users will probably not have exposure to the sdk docs, and the policies of, resources for, etc. those queues will change over time -- so it would be nice to link to a document like the one you cited above, but oriented to users, and with terms aligned to wherever they may be exposed (narrative, apps, job viewer, queue viewer, etc.)

Job Status:

may not be necessary to show the precise timestamp for submission time. There are ways to represent past timestamps somewhat compactly and helpfully. E.g. "5 minutes ago", "3 days ago", "Feb 3", "Dec 10, 2018". The precise time of day down to the second is perhaps not helpful at a glance, although one could inspect it (mouseover, modal popup) to get more detail.

good point, I will remove the seconds.. I have the service set to update every minute or so, so I could instead change it to static text that says "Updated 1 minute ago"

table header still breaks apart at about 1400px width; tables are indeed cranky!

Do you have any recommendations on how to fix this? Ideas include having some javascript that runs and tries to set a CSS property for the table headers and makes them a fixed width? Maybe I can try to modify the dynamic table widget ? Or maybe I try to forgo the dynamic table widget and use datatables on my own so I can configure the CSS my way?

Well, at the least the dynamic table widget could use some love, because an issue like keeping column headers together, how to handle overflow within them, how to handle wide tables, etc. are some of the key issues with tabular displays. HTML tables are not especially good at this sort of application, much better for well behaved, static tables.

@eapearson Thanks for the great feedback and suggestions

Everything seems to work ok. It's possible to embed this view into the jobs browser by doing
             window.KBase.kBaseCatalogQueue(
                {
                    runtime: runtime,
                    usernames: [runtime.service('session').getUsername()],
                    includePublicStats: false,
                    includeUserRunSummary: false,
                    useUserRecentRuns: true,
                    dontShowBackButton: true
                },
                $(container)
            );

But the CSS is not quite right. Not sure how to fix that.
I wasn't able to fix the column headers, except by doing a minimum column length of 150. There could be something that can be done with some inline settings on the text and buttons, not I wasn't able to figure out it, even by live editing the css using a jquery selectory.
Otherwise, I believe all other points have been addressed

Is there a reason for the large font sizes? We mostly don't use them elsewhere except for headers.
Probably don't need to spell out "Percentage" in the utilization column. Could do "%" or just leave out (since the % is repeated in the cells)
Using H3 for the danger label is a bit weird - for one it applies the headline font, which is why it looks super bolded -- it is bold applied to an already bold font.
Also, usage of text shadow is, while not bad in and of itself, not a style we much if at all elsewhere.  Just mentioning.
I'll poke around some more in a bit.

Is there a reason for the large font sizes? We mostly don't use them elsewhere except for headers.
I changed it from x-large to large, let me know if I should change something else.

Probably don't need to spell out "Percentage" in the utilization column. Could do "%" or just leave out (since the % is repeated in the cells)
Fixed

Using H3 for the danger label is a bit weird - for one it applies the headline font, which is why it looks super bolded -- it is bold applied to an already bold font.
Thanks, I changed it to

Also, usage of text shadow is, while not bad in and of itself, not a style we much if at all elsewhere. Just mentioning.
I did this because otherwise the white text is hard to read when the progress bar is empty

re: the "here" link. Generally one doesn't use "click here" type of link, but make the concept or term itself the link. So, "Learn more about our queues" could just be a link. But this is somewhat subjective.
In either case, the link should open into a new window or tab by setting the attribute target="_blank".
Other than that, I'm good with just merging this in, getting it up on ci, and then letting the actual product owner (Paramvir?) play with it.
If that's what you want, just let me know here or in slack.

@eapearson I Fixed the link. Sounds good on merging it in.

ok.


Fix empty invite-only streams shown broken on subs page.

We now mark invite-only streams where the last user left as "deactivated" in the backend.
Now an invite-only stream that has no users left will be effectively deleted, and will not appear in the list of streams, even for an administrator (and thus one won't see the broken view as mentioned in the issue).
Fixes #3657

Supporting GIF:

@timabbott By the way, should we also have the feature taking effect without page reload? I'll have another commit for that feature if it is required.
Thanks!

that change seems plausible, but this needs tests.
Also, you should rebase to get rid of the merge commit :).
Check out our GitHub guide for advice on this :).

Yes. I am on it. Just trying to implement my first test properly. :)
I will rebase all the commits once done.

This'll be fixed in some time. Extremely sorry for the delay.


Documentation Returns -> Return :/

Documentatio, correction, Returns -> Return :/

I would have thought it was returns too.  I rarely use that field.


Auto Delete on Idle not working as intended

Related command
az servicebus topic subscription update --auto-delete-on-idle "PT5M"  --ids <xyz>
az servicebus topic subscription update --auto-delete-on-idle "00:05:00"  --ids <xyz>
Describe the bug
After calling any of these commands the operation seemingly succeeds but nothing gets updated.
The resulting JSON shows a random date:
{
  "accessedAt": "0001-01-01T00:00:00+00:00",
  **"autoDeleteOnIdle": "10675199 days, 2:48:05.477581",**
  "clientAffineProperties": null,
  "countDetails": {
    "activeMessageCount": 0,
    "deadLetterMessageCount": 0,
    "scheduledMessageCount": 0,
    "transferDeadLetterMessageCount": 0,
    "transferMessageCount": 0
  },
  "createdAt": "2022-06-10T15:38:19.880700+00:00",
  "deadLetteringOnFilterEvaluationExceptions": true,
  "deadLetteringOnMessageExpiration": false,
  "defaultMessageTimeToLive": "366 days, 0:00:00",
  "duplicateDetectionHistoryTimeWindow": null,
  "enableBatchedOperations": true,
  "forwardDeadLetteredMessagesTo": null,
  "forwardTo": "...",
  "id": "<xyz>",
  "isClientAffine": false,
  "location": "East US 2",
  "lockDuration": "0:01:00",
  "maxDeliveryCount": 10,
  "messageCount": 0,
  "name": "<abc>",
  "requiresSession": false,
  "resourceGroup": "mygroup",
  "status": "Active",
  "systemData": null,
  "type": "Microsoft.ServiceBus/Namespaces/Topics/Subscriptions",
  "updatedAt": "2022-06-10T15:38:19.880700+00:00"
}

To Reproduce
Run command above on an existing topic subscription
Expected behavior
After calling this command I would expect the referenced subscription Auto Delete on Idle time updated to 5 minutes.
Environment summary

azure-cli                         2.37.0

core                              2.37.0
telemetry                          1.0.6

Extensions:
arcdata                            1.4.1
containerapp                       0.3.5

Dependencies:
msal                            1.18.0b1
azure-mgmt-resource             21.1.0b1

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\xyz\.azure\cliextensions'

Python (Windows) 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 22:57:10) [MSC v.1929 32 bit (Intel)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

route to CXP team

@amoradiaz Apologies for the late reply. Thanks for reaching out to us and sharing this feedback. We are looking into this issue and we will provide an update.

@amoradiaz I wasn't able to reproduce this issue. While I ran the below command, I was able to set the autoDeleteOnIdle property successfully. See below:

az servicebus topic subscription update --resource-group MyRGName --namespace-name MyResource --topic-name MyTopic --name MySubName --auto-delete-on-idle PT7M

Output:

{
"accessedAt": "0001-01-01T00:00:00+00:00",
"autoDeleteOnIdle": "0:07:00",
"clientAffineProperties": null,
"countDetails": {
"activeMessageCount": 0,
"deadLetterMessageCount": 0,
"scheduledMessageCount": 0,
"transferDeadLetterMessageCount": 0,
"transferMessageCount": 0
},
"createdAt": "2022-06-14T07:50:42.039807+00:00",
"deadLetteringOnFilterEvaluationExceptions": false,
"deadLetteringOnMessageExpiration": false,
"defaultMessageTimeToLive": "14 days, 0:00:00",
"duplicateDetectionHistoryTimeWindow": null,
"enableBatchedOperations": true,
"forwardDeadLetteredMessagesTo": null,
"forwardTo": null,
"id": "XXXXX",
"isClientAffine": false,
"location": "South India",
"lockDuration": "0:00:30",
"maxDeliveryCount": 2,
"messageCount": 0,
"name": "XXXXX",
"requiresSession": false,
"resourceGroup": "XXXXX",
"status": "Active",
"systemData": null,
"type": "Microsoft.ServiceBus/Namespaces/Topics/Subscriptions",
"updatedAt": "2022-06-14T07:50:42.039807+00:00"
}

I am using the same Azure CLI version as that of your environment 2.37.0.
As a next action plan, Could you please run the below command check if that helps ? Awaiting your reply.

az resource update --ids /subscriptions/12345678910/resourceGroups/XXXXXX/providers/Microsoft.ServiceBus/namespaces/XXXXX/topics/XXXXX/subscriptions/XXXXX --set properties.autoDeleteOnIdle=PT10M

PS: Update the SubscriptionID, RG name, SB namespace, Topic Name, Subscription name and --auto-delete-on-idle value before running the command.

No dice, I tried what you used, and still the same issue
az servicebus topic subscription update --resource-group <xyz> --namespace-name <abc> --topic-name <def> --name <ghi> --auto-delete-on-idle PT7M
{
  "accessedAt": "0001-01-01T00:00:00+00:00",
  **"autoDeleteOnIdle": "10675199 days, 2:48:05.477581",**
  "clientAffineProperties": null,
  "countDetails": {
    "activeMessageCount": 0,
    "deadLetterMessageCount": 0,
    "scheduledMessageCount": 0,
    "transferDeadLetterMessageCount": 0,
    "transferMessageCount": 0
  },
  "createdAt": "2022-06-14T12:56:07.444982+00:00",
  "deadLetteringOnFilterEvaluationExceptions": true,
  "deadLetteringOnMessageExpiration": false,
  "defaultMessageTimeToLive": "00:10:00"
  "duplicateDetectionHistoryTimeWindow": null,
  "enableBatchedOperations": true,
  "forwardDeadLetteredMessagesTo": null,
  "forwardTo": "<jkl>",
  "id": "/subscriptions/<tuv>/resourceGroups/<xyz>/providers/Microsoft.ServiceBus/namespaces/<abc>/topics/<def>/subscriptions/<ghi>",
  "isClientAffine": false,
  "location": "East US 2",
  "lockDuration": "0:01:00",
  "maxDeliveryCount": 10,
  "messageCount": 0,
  "name": "<ghi>",
  "requiresSession": false,
  "resourceGroup": "<xyz>",
  "status": "Active",
  "systemData": null,
  "type": "Microsoft.ServiceBus/Namespaces/Topics/Subscriptions",
  "updatedAt": "2022-06-14T12:56:07.444982+00:00"
}

Then tried what you suggested with the resource update, and it gave me the same result.
az resource update --ids /subscriptions/<tuv>/resourceGroups/<xyz>/providers/Microsoft.ServiceBus/namespaces/<abc>/topics/<def>/subscriptions/<def> --set properties.autoDeleteOnIdle=PT10M
Then I tried with a different property just for fun and it worked:
az resource update --ids /subscriptions/<tuv>/resourceGroups/<xyz>/providers/Microsoft.ServiceBus/namespaces/<abc>/topics/<def>/subscriptions/<def> --set properties.defaultMessageTimeToLive=PT7M
{
  "accessedAt": "0001-01-01T00:00:00+00:00",
  "autoDeleteOnIdle": "10675199 days, 2:48:05.477581",
  "clientAffineProperties": null,
  "countDetails": {
    "activeMessageCount": 0,
    "deadLetterMessageCount": 0,
    "scheduledMessageCount": 0,
    "transferDeadLetterMessageCount": 0,
    "transferMessageCount": 0
  },
  "createdAt": "2022-06-14T12:56:07.444982+00:00",
  "deadLetteringOnFilterEvaluationExceptions": true,
  "deadLetteringOnMessageExpiration": false,
  **"defaultMessageTimeToLive": "0:07:00",**
  "duplicateDetectionHistoryTimeWindow": null,
  "enableBatchedOperations": true,
  "forwardDeadLetteredMessagesTo": null,
  "forwardTo": "<jkl>",
  "id": "/subscriptions/<tuv>/resourceGroups/<xyz>/providers/Microsoft.ServiceBus/namespaces/<abc>/topics/<def>/subscriptions/<ghi>",
  "isClientAffine": false,
  "location": "East US 2",
  "lockDuration": "0:01:00",
  "maxDeliveryCount": 10,
  "messageCount": 0,
  "name": "<ghi>",
  "requiresSession": false,
  "resourceGroup": "<xyz>",
  "status": "Active",
  "systemData": null,
  "type": "Microsoft.ServiceBus/Namespaces/Topics/Subscriptions",
  "updatedAt": "2022-06-14T14:50:55.551864+00:00"
}

It also worked with maxDeliveryCount
Interestingly enough, I don't see this property in the Portal UI, is this under some sort of extra permission?

@amoradiaz This is a quick follow-up to check if you had a chance to follow the above suggestions to isolate this issue. Awaiting your reply.

After following the steps that you detailed, the command successfully updated the auto-delete idle property, and I can even see it in the portal:

From the API perspective, what command sets the "Never auto-delete" property to true?  This is something I'm not doing consciously
az servicebus topic subscription create --name
                                        --namespace-name
                                        --resource-group
                                        --topic-name
                                        [--auto-delete-on-idle]
                                        [--dead-letter-on-filter-exceptions {false, true}]
                                        [--default-message-time-to-live]
                                        [--enable-batched-operations {false, true}]
                                        [--enable-dead-lettering-on-message-expiration {false, true}]
                                        [--enable-session {false, true}]
                                        [--forward-dead-lettered-messages-to]
                                        [--forward-to]
                                        [--lock-duration]
                                        [--max-delivery-count]
                                        [--status {Active, Disabled, ReceiveDisabled, SendDisabled}]

@amoradiaz Thanks for getting back. I am glad that my above suggestions helped.
The --auto-delete-on-idle property is optional and if not specified while creating the subscription using AzureCLI it will be set to Never.  The minimum duration that can be set for this property is 5 minutes.
You can tweak / update this property even after the subscription is created. If you would like to Turn Off the Auto delete (change it to Never). You can do it either from portal or AzCLI:
From portal:

You can enable the Checkbox to Never auto-delete:

From AzCLI:

az servicebus topic subscription update --name MySubscription --namespace-name MyNamespace --resource-group MYRG --topic-name MyTopic --auto-delete-on-idle P10675198DT2H48M5.477S

The above command will update the --auto-delete-on-idle to Never. Hope this answers.

@amoradiaz I hope you had a chance to look at my above comment. Please let me know if you need any further assistance on this or we can proceed with closure.

I can confirm that what you suggested checks out. I can do all of that as long as I don't add a forwarding queue/topic.
As soon as I do that, the Auto-delete option disappears from the portal view.
before:

after:

After running the update the command, it returns the "never" time span no matter what I had before.

This is the scenario that I'm currently in, where I wanted some of this subscriptions to expire on their own, even if they were tied to queues. Some of my queues are also auto expiring, which creates this scenario where the topic subscription remains forever (never autodelete), but the queue it was pointing to is already auto-deleted.
Is there any way to propagate the auto-delete from a queue to its subscription so that we don't leave subscriptions pointing to nowhere?

Involving Service Team to comment on this.
@Saglodha Please look into the above ask once you get a chance.

@navba-MSFT - Ack on above ask.
@amoradiaz - Thanks for sharing details about your setup. I would look into this use case of yours and share feedback soon.

Adding @EldertGrootenboer  for visibility here. @EldertGrootenboer , can you review this ask and confirm if this is expected behavior?

We are looking into this, and will come back when we have more information.

@amoradiaz this is expected behavior by design. The behavior has been documented here as well.
https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-auto-forwarding#using-autoforwarding
Unfortunately, today there is no mechanism to relay the inactiveness of destination entity on to the source entity and clean that up as well.
It has typically been our belief that Autoforwarding is more of a configuration set up by customer that is set to work whenever there is a message on the source entity. Therefore, when such a configuration is set up, we choose not to interfere with the lifetime of the entity through Autodelete.
Having said that, we did notice a documentation issue in the above link that the destination entity's lifetime will also be adjusted. That is incorrect. We have opened a workitem to correct that part of the documentation soon.


Improve the AutoReleaser explanation at the readme

Please guys, can you check this PR? The idea is to improve the AutoReleaser explanation to avoid people understanding that if the AutoReleaser is not set, the cache will not work as expected. Thanks! @Sloy @despinola

Coverage remained the same at 100.0% when pulling 5fce2786fa2b3783e78c7a738df816aac63afe53 on applying-rafa-and-david-suggestions-to-readme into 658e5fd6f54a155fb5a9ba8c75beaf740f591c0d on master.

👍 Much better! :)

Great! :·) Thanks thanks!


Would you like to challenge to create a pull up load more animation for ListView ?

Just like the youtube app?

@realcarlos This is not the kind of challenges we make. If want a pull to refresh you can use this class from Flutter Docs: RefreshIndicator


how to get the query line?

**Kibana version:**6.2.4
**Elasticsearch version:**6.2.4
**Server OS version:**red hat 7
**Browser version:**firefox 56
Browser OS version:
Original install method (e.g. release pkg link, gulp, from source, etc.):
pkg
Describe the bug:
add the query line to the payload.hits,and then  the alarm can show the query line?
"query": "Exception AND NOT  (exception test)"

@aabababb Please use the template provided so we can get some insight in what you want to accomplish 100%. If I understand you correctly, using script conditions may be the way to approach this.
In script conditions you can add a field to the payload, I don't know if the query can be extracted, but it's worth a shot.
In script conditions:
payload.hits.total > 100; payload.str = JSON.stringify(payload);

Something like that.

@aabababb Thanks for taking time to report an issue.
Please fill all the following fields to make sure we have all the data we need to help you.
Also, make sure you use the latest release we issued 3 days ago  https://github.com/sirensolutions/sentinl/releases
Tips:

If your watcher is not executed, try to query Elasticsearch manually first. Do you receive the results you want for your query?
For example, on Linux, you can query Elasticsearch using curl:

curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"match_all": {}
}
}
'
-->
System Details
Versions

Type
Version

Kibana

Elasticsearc

Server OS

Browser

Original Install Method

[ ] Release pkg link
[ ] Gulp
[ ] Source
Other:

Bug Details
Bug Description
Steps to Reproduce

Expected Behavior
Sample of data you have in Elasticsearch

Full Watcher JSON

Screenshots (if relevant)
Errors in Browser Console (if relevant)

Kibana logs and/or server output (attached if relevant)
Any additional information

Lucene query is not supported yet. We support only Elasticsearch DSL query in the watcher input. But we have will add the Lucene query too. Follow this issue for future updates https://github.com/sirensolutions/sentinl/issues/608


ARROW-1016: Python: Include C++ headers (optionally) in wheels

here's an appveyor build: https://ci.appveyor.com/project/xhochy/arrow/build/1.0.320


Support for CSW GetRecords pagination

At this stage, all GetRecords are performed as series of AJAX request promises, and joined together. For catalogues having a lot of datasets, this causes performance issues. A new solution should be investigated to bind to the pagination UI.


fix: AzureOpenAiChatOptions to handle null values for presencePenalty and frequencyPenalty

Thank you for taking time to contribute this pull request!
You might have already read the [contributor guide][1], but as a reminder, please make sure to:

Sign the contributor license agreement ✅
Rebase your changes on the latest main branch and squash your commits ✅
Add/Update unit tests as needed ✅
Run a build and make sure all tests pass prior to submission ✅

I discovered a null safety issue while working with spring-ai in Kotlin.
The AzureOpenAiChatOptions builder does not handle null values properly.
Specifically, the withFrequencyPenalty and withPresencePenalty methods call doubleValue() without checking if the parameter is null.
Here is the problematic code:
public Builder withFrequencyPenalty(Float frequencyPenalty) {
    this.options.frequencyPenalty = frequencyPenalty.doubleValue();
    return this;
}

public Builder withPresencePenalty(Float presencePenalty) {
    this.options.presencePenalty = presencePenalty.doubleValue();
    return this;
}

I managed to work around this issue in Kotlin by ensuring null safety as shown below:
val chatOptions = this.chatOptions?.let {
    AzureOpenAiChatOptions().apply {
        maxTokens = it.maxTokens
        temperature = it.temperature
        topP = it.topP
        logitBias = it.logitBias
        user = it.user
        n = it.n
        stop = it.stop
        presencePenalty = it.presencePenalty?.toDouble() // like this
        frequencyPenalty = it.frequencyPenalty?.toDouble() // like this
    }
}

...

However, I believe it would be beneficial to address this issue at its source.
Therefore, I am submitting this pull request with additional test cases to highlight and resolve this issue.

Thanks for the improvement @Hyune-c


Test compatible Ruby version and add Ruby versions to README and Github Action

My first contribute to the project, so to understand it, I first run tests and check the compatible Ruby version, then add the Compatibility part in README so that the user can setup the project to use the gem more easily.

For the time being, Ruby version lower than 3.1 is not Compatitable

@alchaplinsky can you review the pull request for me?


Openldap module declarative contents

This PR allows to declaratively set the contents of the LDAP database. I've tested it on my machine by replacing the stock module with the post-PR one, and it appears to work.

cc @domenkozar @Mic92 for the last few changes to this module, hopefully it can still get in before 18.03 :)

services = {
+    openldap = {
+      enable = true;
+      declarativeContents = ''
+        dn: dc=example,dc=org
+        objectClass: domain
+        dc: example
+      '';
+    };

results in:
Mar 03 17:51:50 turingmachine systemd[1]: Starting LDAP server...
Mar 03 17:51:50 turingmachine openldap-pre-start[28809]: Available database(s) do not allow slapadd
Mar 03 17:51:50 turingmachine systemd[1]: openldap.service: Control process exited, code=exited status=1
Mar 03 17:51:50 turingmachine systemd[1]: openldap.service: Failed with result 'exit-code'.
Mar 03 17:51:50 turingmachine systemd[1]: Failed to start LDAP server.

What else is required?

Is services.openldap supposed to work without a configuration file? (the one generated is empty if extraConfig is empty)
It works for me with this (ie. keeping the same extraConfig as before and adding the declarativeContents):
{
  services.openldap = {
    enable = true;
    urlList = [ "ldap://<IP_ADDRESS>:389/" ];
    extraConfig = ''
      include ${pkgs.openldap}/etc/schema/core.schema
      include ${pkgs.openldap}/etc/schema/cosine.schema
      include ${pkgs.openldap}/etc/schema/inetorgperson.schema
      include ${pkgs.openldap}/etc/schema/nis.schema

      database bdb
      suffix dc=gaspard,dc=ninja
      directory /var/db/openldap
      rootdn cn=root,dc=gaspard,dc=ninja
      rootpw [...]
      index cn,sn,uid pres,eq,approx,sub
      index objectClass eq
    '';
    declarativeContents = ''
      dn: dc=gaspard,dc=ninja
      objectClass: domain
      dc: gaspard

      dn: ou=users,dc=gaspard,dc=ninja
      objectClass: organizationalUnit
      ou: users

      # ...
    '';
  };
}

@GrahamcOfBorg test openldap

Failure on x86_64-linux (full log)
Partial log (click to expand)
while evaluating the attribute ‘drvPath’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/lib/customisation.nix:179:13:
while evaluating the attribute ‘drvPath’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/lib/customisation.nix:146:13:
while evaluating the attribute ‘buildCommand’ of the derivation ‘vm-test-run-dovecot’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘buildCommand’ of the derivation ‘nixos-test-driver-dovecot’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘buildCommand’ of the derivation ‘nixos-vm’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘buildCommand’ of the derivation ‘nixos-system-machine-18.03.git.ee2b145’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating ‘optionalString’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/lib/strings.nix:138:26, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/nixos/modules/system/activation/top-level.nix:40:9:
while evaluating the attribute ‘closure’ of the derivation ‘initrd’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating anonymous function at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/build-support/closure-info.nix:9:1, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/build-support/kernel/make-initrd.nix:33:13:
assertion failed at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/build-support/closure-info.nix:11:1

test worked locally.

Thanks! :)

Failure on aarch64-linux (full log)
Partial log (click to expand)
   Loaded: loaded (/nix/store/fr6ms7ks7i7bw7fxi9b77wxcj3rw7q54-unit-openldap.service/openldap.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
error: command `systemctl status openldap.service' did not succeed (exit code 3)
command `systemctl status openldap.service' did not succeed (exit code 3)
cleaning up
killing machine (pid 627)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/gp84f4znis7hrlqwg80yhmw4iv6a3034-vm-test-run-dovecot.drv' failed with exit code 255
error: build of '/nix/store/gp84f4znis7hrlqwg80yhmw4iv6a3034-vm-test-run-dovecot.drv' failed


Syntax highlighting not working

I have installed the plugin using Vundle, however I am not seeing the correct syntax highlighting.

I have tried disabling all other plugs, no change.

You need to set g:cpp_experimental_simple_template_highlight or g:cpp_experimental_template_highlight before loading after/syntax/cpp.vim file.
That's Because bit is checking those variables on creating syntax (when syntax file is parsed), not when source code is highlighted.
Simply add
let g:cpp_experimental_template_highligh =1

to your vimrc.

I added that line to my vimrc file, no effect, but I see you're right - if I do let g:cpp_experimental_template_highligh =1 then set syntax=cpp it works.
Why would it not be working when it's in the vimrc file?

I would say wrong order of assigning variable and loading plugin, but I am not sure. I don't know your vimrc so this is just a guess.

Glad the issue has been resolved. Closing.


Add protobuf

Official non-cmake

For package: https://github.com/ruslo/hunter/issues/79

In the meantime, protobuf seems to support cmake for Version 3. It is still in beta stage, but it might be worth to have another look at the package.
I would be very interested in hunter supporting protobuf.

protobuf seems to support cmake for Version 3

I've just noticed that there is a cmake directory :) Probably we just need to add a top CMakeLists.txt with one command add_subdirectory(cmake) and everything will just work.
@KerstinKeller Can you test this solution? Hints: https://github.com/ruslo/hunter/wiki/usr.adding.new.package

Yes, I will have a look. It seems not quite as easy though (at least for
VS2012):
To compile protobuf, you're required to manually check out goolemock/gmock
into the main directory. There is however a realease tar.gz which contains
this folder already, but I cannot compile because of the
"_VARIADIC_TEMPLATE_MAX" default 5 in VS2012 .
I also tried with VS2015 Community (which seems not to be supported by
Hunter yet - probably cmake\modules\hunter_setup_msvc_vcvarsall.cmake will
have to be adapted).
I will  investigate further and update the issue accordingly.
2015-09-30 12:02 GMT+02:00 Ruslan Baratov<EMAIL_ADDRESS>
protobuf seems to support cmake for Version 3
I've just noticed that there is a cmake directory :) Probably we just
need to add a top CMakeLists.txt with one command add_subdirectory(cmake)
and everything will just work.
@KerstinKeller https://github.com/KerstinKeller Can you test this
solution? Hints:
https://github.com/ruslo/hunter/wiki/usr.adding.new.package
—
Reply to this email directly or view it on GitHub
https://github.com/ruslo/hunter/issues/97#issuecomment-144344930.

VS2015 should work fine with Hunter. From what I've remember while fixing this bug: https://github.com/forexample/hunter-simple/issues/2

To compile protobuf, you're required to manually check out goolemock/gmock into the main directory

@KerstinKeller I think BUILD_TESTING=OFF might be helpful here?

Ok, I was able to successfully install with hunter and BUILD_TESTING=OFF. I changed this in the protobuf cmake. Is there also an option to pass this from protobuf/hunter.cmake file?
And what is the general procedure to make this available then? Need to fork the protobuf project to integrate the changes (mainly additional CmakeLists.txt with add_subdirectory(cmake)  in the main directory)?

Ok, I was able to successfully install with hunter and BUILD_TESTING=OFF. I changed this in the protobuf cmake. Is there also an option to pass this from protobuf/hunter.cmake file?

Yes, you can use hunter_cmake_args see example: https://github.com/ruslo/hunter/wiki/usr.adding.new.package#cmake-options

And what is the general procedure to make this available then? Need to fork the protobuf project to integrate the changes (mainly additional CmakeLists.txt with add_subdirectory(cmake) in the main directory)?

Yes, you can use this repository: https://github.com/hunter-packages/protobuf Use branch hunter and keep other branches equal to upstream. Though it make sense to try to apply this patch to the official repository.

Ok, thanks already. So I would probably create the hunter branch from the beta-release tag commit, because that's the version to be made available, make my changes and tag it with 3.0.0-beta-1-hunter.

Cool, thank you

What is the progress of this hunterization? As I see the the mentioned modifications is not in hunter-packages/protobuf repository (disabling test, and root CMakeLists.txt).
I plan to use this library (on the top of ZeroMQ) and if you give me rights to the repository I help to integrate it.
@ruslo When I tried some weeks ago to compile this project I cannot set the path of cmake/CMakeLists.txt. (In this case we can avoid the separated repository). Can I add this as parameter to ExternalParoject_Add command? As I remember I tried -Hcmake/ but it had no effect.

and root CMakeLists.txt

Should be a trivial task

disabling test

This can be done by setting default cmake arguments for package in hunter.cmake, no need to patch protobuf (see this wiki page).

if you give me rights to the repository I help to integrate it

Done. @tatraian You have permissions to push to branches directly but I think it make sense to make a pull request for review/discussion purpose.

Can I add this as parameter to ExternalParoject_Add command? As I remember I tried -Hcmake/ but it had no effect

As far as I know there is no way to specify this with simple option. The easiest way to do fixes is to add root CMakeLists.txt.

Can I update the forked repository to the current state of original? They released beta-2 in December.
After that I plan to create a branch from the release tag. After I'm going to clone the repository and send pull requests. Is this OK?

Is this OK?

@tatraian Sure, it sounds great.

Fixed by https://github.com/ruslo/hunter/pull/340


WIP: Test to remove our need for a rustls fork

Our fork of rustls has a function added to the CipherSuite enum to convert from a string to the enum.
This is a test to see how dirty it would be to do it in our code instead, obviating the need for our fork of rustls.

Since ring and rustls have now gone away, is this PR obsolete and can it be closed?


Update CCF version to 5.0.10

Update CCF version to 5.0.10 to consume latest fixes for service certificates

TODO: wait for CCF team to release version 5.0.10


fix Locale Picker and small picker styles

Details

Removed API call.
Fixed the Picker style so that it shows a blue border while focused.
Fixed picker style on Firefox browsers. https://expensify.slack.com/archives/C01GTK53T8Q/p1629962984011400

Fixed Issues

$ https://github.com/Expensify/App/issues/4513
Tests | QA Steps

Open the login page & sign in with some account.
Sign out of the app.
Switch a language.
Language should change.

Tested On

[x] Web
[x] Mobile Web
[x] Desktop
[x] iOS
[x] Android

Screenshots

Web | Desktop

Mobile Web

iOS

Android

🚀 Deployed to staging by @Beamanator in version: 1.0.88-3 🚀

platform
result

🤖 android 🤖
cancelled 🔪

🖥 desktop 🖥
cancelled 🔪

🍎 iOS 🍎
cancelled 🔪

🕸 web 🕸
success ✅

🚀 Deployed to production by @roryabraham in version: 1.0.90-2 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
failure ❌

🕸 web 🕸
success ✅

@parasharrajat just double checking here - what does the picker look like when it is not active? It seems like the styles are still a little off - I think when the picker is active, the border color should change instead of an additional blue border being added. Let me know if that makes sense.

When picker is not active it shows gray border like other input in app. There are two borders arround the picker. I changed the colour or outer border but didn't know that there were two  borders. So In a follow up I will remove the inner border.


#/ Showing with home url

#/ showing after the homepage url. it should removed

We'll follow up in a similar issue in the Cart's feedback repository 🙂
https://github.com/snipcart/snipcart-v3-feedback/issues/34


PoissonVI tutorial

Formatting

[x] My tutorial has only one top-level (#) header

Reproducibility

[x] My tutorial works on Google Colab
[x] My tutorial sets scvi.settings.seed = 0 at the beginning of the notebook
[x] My tutorial has been run and includes outputs (e.g. plots, tables)

Other

[x] Counts and normalized data should co-exist in the datasets, see the API overview for an example
[x] For scRNA-seq data, normalization should be counts per median library size and then log1p transformed -- if not, a reason should be given (not applicable here)

@martinkim0, fixed it!


Fix the error message for duplicate policies (#1601)

#1601
@kapilt - Oddly, this code path hadn't been touched in a long time.  I think we have a few different validation points in the code, but iirc they are hard to merge.  I made the simplest fix I could (I made the return value from schema.py:validate() more consistent).

@scotwk Won't this code also cause the same IndexError: list index out of range error?

I've convinced myself that that fallthru bit of code should be OK.

its rather unclear how this fixes the referenced issue, its just adding handling to the extant happy path that was already catching this issue.

ah. its more clear now the error interface is expecting a tuple of (error, policy_name) which this corrects the dupe detection to.


issue #216, Added Apple Park image to 06-02 WORKPLACE, English version

This is good, but please add the file to the fig folder and refer to it as in the code for other cases where we refer to files.  Then resubmit.

Hi Glen,
I used this method because I don't have access. I will need to be a
collaborator of the repository to gain push access to add image files to
the folder. Can you add me as a collaborator? I will resubmit once I gain
access.
Thanks,
Greg
[image: Screenshot 2024-02-28 at 2.25.41 AM.png]
On Tue, Feb 27, 2024 at 8:59 PM Glen Weyl @.***> wrote:

This is good, but please add the file to the fig folder and refer to it as
in the code for other cases where we refer to files. Then resubmit.
—
Reply to this email directly, view it on GitHub
https://github.com/pluralitybook/plurality/pull/336#issuecomment-1967494792,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AW567E2PVPL5LWZHLLROM7DYVY3J7AVCNFSM6AAAAABD4DKU2SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRXGQ4TINZZGI
.
You are receiving this because you authored the thread.Message ID:
@.***>


lib_manticore_columnar wrong path

Hello
Under Debian 10
After Compiling Manticore columnar lib install in /share/manticore/modules
But manticore not found it
Then I copy libs to
/usr/share/manticore/modules/lib_manticore_columnar.so
And everything worked
Manticore 5.0.3 cff178311@220703 dev (columnar 1.16.1 637ddc6@220727) (secondary 1.16.1 637ddc6@220727)
Maybe it's a bug with the library directory

@webigorkiev can you comment why you've closed the issue?

In new version Manticore 6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)
In my environment, the code build was successful.


Config - Clarify Aliasing Example

https://webpack.js.org/configuration/resolve/#resolve-alias:

alias: {
  xyz$: path.resolve(__dirname, 'path/to/file.js')
}

which would yield these results:
[...]
import Test2 from 'xyz/file.js'; // Error, /path/to/file.js/file.js is invalid

I'm not sure if I really got how the trailing $ works, but if so, I think the explanation for import Test2 from 'xyz/file.js' is wrong. As I understand it, 'xyz/file.js' is not an exact match for the alias xyz$, so webpack does not even consider this alias for resolving and therefore never even tries to find /path/to/file.js/file.js. Is that correct?
If yes, I would prefer:
alias: {
  xyz$: '/path/to/file.js'
}

import Test1 from 'xyz'; // Exact match, so /path/to/file.js is imported
import Test2 from 'xyz/file.js'; // Not an exact match, so normal module resolution takes place

I use the complete path (instead of path.resolve(...)) in the alias definition to clearly identify the file in the comment of the second code block.

I believe you are correct xyz$ should only match import Test1 from 'xyz'; and not throw an error for the second one. However, I would leave the path.resolve(...) in as developers will have to include the full path through their whole file system, not just the path within their directory (i.e. /Users/Me/Desktop/Project/Test/path/to/file.js not just /path/to/file.js). @bebraw do you agree?

@skipjack Yup, that's the way the match works.

@iTob191 so if you'd be interested in submitting a pr with this change you mentioned:
import Test1 from 'xyz'; // Exact match, so /path/to/file.js is imported
import Test2 from 'xyz/file.js'; // Not an exact match, so normal module resolution takes place

That would be great.


example generation

hi, thanks for release your code. i would like to ask why you concat all the instruction and input in the prompt to get the new prompt?
looking forward to your reply. thanks.

Thanks for your question. The function encode_prompt in src/generate_hard_instruction.py indeed supports concatenating several instructions to get the new prompt. But in our implementation, we set num_prompt_instructions=1 to use only one instruction for obtaining the new prompt.

thanks for your reply


invalid key format

Same issue as #17
https://github.com/SunshineStream/Sunshine/runs/6951924698?check_suite_focus=true#step:6:70
Importing private key
  mode of '/home/builder/.ssh/aur' changed from 0644 (rw-r--r--) to 0600 (rw-------)
  Load key "/home/builder/.ssh/aur": invalid format

AUR account - SSH Public Key:
ssh-rsa A...=

doesn't have any extra lines
github secrets - AUR_SSH_PRIVATE_KEY
b...
...
...
...
==

doesn't have any extra lines
I generated my key following these commands (https://phoenixnap.com/kb/generate-setup-ssh-key-ubuntu):
mkdir -p $HOME/.ssh
chmod 0700 $HOME/.ssh
ssh-keygen

public key came from aur.pub file
private key came from aur file
workflow step: https://github.com/SunshineStream/Sunshine/actions/runs/2522361975/workflow#L77
Maybe I did something wrong generating the key or adding it to my aur profile?

It is likely that you did something that corrupt your keys. I also personally experienced this issue before.
Make sure that the keys (after you paste it to GitHub secrets):

Don't have trailing newline.
Don't have leading newline.
Use LF as line breaker, not CRLF.

It is likely that you did something that corrupt your keys. I also personally experienced this issue before.
Make sure that the keys (after you paste it to GitHub secrets):
* Don't have trailing newline.

* Don't have leading newline.

* Use LF as line breaker, not CRLF.

Thanks for the reply... I meant to comment earlier, but I got this to work.
The issue was that I used a passphrase when generating my keys. I tried without and it worked. It would be a nice addition to support a passphrase if possible. Anyway, thanks!

passphrase is used to encrypt private keys, to prevent a malicious attacker from obtaining your private key even if they gain access to your computer, because even though they have access to your encrypted private key, they still don't know the passphrase you used to encrypt the key (because you didn't store the passphrase). To store passphrase in a GitHub secret would nullify its protection, rendering passphrase pointless.


There is no option to specify a path to scan.

We have a repository that contains multiple microservices but they do not need to scan at the same time.
Currently there is no way to specify what path the scan should be limited to. The Azure DevOp HCL Appscan task does allow this.

Thanks for the feedback. This is something we'll look to add to the GitHub Action. The only way to control this today is to use a configuration file and store it at the root of the repository. The file needs to be named appscan-config.xml and can be as simple as the following (see full documentation here):
<Configuration>
    <Targets>
        <Target path="<some relative path>"/>
    </Targets>
</Configuration>


Range Error: Invalid string length when download report

/project-sp-api/node_modules/amazon-sp-api/lib/request.js:23
body += chunk;
^
RangeError: Invalid string length
at IncomingMessage. (/project-sp-api//node_modules/amazon-sp-api/lib/request.js:23:17)
at IncomingMessage.emit (node:events:390:28)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)
at IncomingMessage.Readable.push (node:internal/streams/readable:228:10)
at HTTPParser.parserOnBody (node:_http_common:141:24)
at HTTPParser.execute ()
at TLSSocket.socketOnData (node:_http_client:487:22)
at TLSSocket.emit (node:events:390:28)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)
at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
at TLSWrap.onStreamRead (node:internal/stream_base_commons:199:23)
Hello, I am experiencing "Invalid string length" issue when downloading a report.
What does this error mean? That my report contains no data?
Using these options to call:
let report = await sellingpartner.download(report_document,              {               json: true,               unzip: true,               file:'/project-sp-api/report.json'             }            )
Much thanks!

Hi @shiowwj,
I would rather thinks its the opposite, maybe the report is too large which results in this error? We are already trying to better deal with large reports by implementing a stream mode (see #56). Which report is it you are requesting?

Hello, sorry for the long delay, as i did not encounter this error when i switched the MarketplaceID to Mexico.
I requested the GET_BRAND_ANALYTICS_SEARCH_TERMS_REPORT.
I've since then switched marketplaceid to USA and encountered this error again.

Hi @shiowwj,
pretty sure that the size of the report is the issue here. Its probably > 100 MB for US and quite smaller in Mexico. Implementing a stream mode should fix the problem, but due to very limited resources here I don't know how fast we can really implement it.

We're having the same issue :(
CustomError: Cannot create a string longer than 0x1fffffe8 characters
    at SellingPartner._decodeBuffer (C:\wamp\www\repricer\node_modul
es\amazon-sp-api\lib\SellingPartner.js:246:13)
    at SellingPartner.download (C:\wamp\www\repricer\node_modules\am
azon-sp-api\lib\SellingPartner.js:670:24)

@emilianmanolache This is most likely a problem with the report being so big that it can't be transformed to JSON directly. Please see this stackoverflow post for more information.
Currently there is no easy fix for this, but you could either try to not transforming the result to json (don't pass json:true to .download()) or, depending on the type of report, you could try to reduce the report size by i.e. requesting a smaller time frame.

Hi @amz-tools.
Thanks for your reply!
I've tried passing json: false to download(), but I get the same error.

@emilianmanolache I see, I am afraid the only solution is to implement a stream mode that will be able to deal with that, which is already on our enhancements list but we haven't been able to find the time for it yet.
@danielecr has already implemented a stream mode on top of this module, you may check that out and see if it helps you: https://www.npmjs.com/package/stream-amazon-sp-api


how to send you new domains of mac?

kmqhqhs02w.adobestats.io
wdyav7y3rf.adobestats.io

✅ Added Thank you for your contribution!


Try downgrading python to fix coverage issue

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 0.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
0

Relevant Lines:
0

💛  - Coveralls