Missing triangles in output

Hello, me again!
In the test scene I'm currently using there are a few triangles apparently missing. I was sure it was a problem with the scene file (exported from blender), with Assimp (the lib I use to load the scene) or my code. I've started to play a bit with the mesh in blender (simplify, remove vertices, triangulate etc), then by exporting to different file formats. Nothing changed. Then I tried to render the scene with my BIH, and, to my surprise, the scene was fine.
That's how it looks with my BIH

and with the BVH:

(sorry for the bad quality, hope will do anyway). Please note the fireplace, or the corner opposite to the point of view. There are a couple more, but you can't see it in the that image (I can of course provide you with better ones).
I understand that this is not much to work with, but thought to report it should you be interested in analyzing the issue. Let mw know if you want the collada/gltf/blend file of the scene.
That's how I create the bvh:
	void Preprocess()
	{
		//bvh::SweepSahBuilder<bvh::Bvh<real>> builder(bvh);
		bvh::LocallyOrderedClusteringBuilder<bvh::Bvh<real>, int>  builder(bvh);

		//Adapt data format
		bvh::BoundingBox<real>* bboxes = new bvh::BoundingBox<real>[shapes.size()];
		bvh::Vector3<real>* centers = new bvh::Vector3<real>[shapes.size()];

		qlaabox<real> worldbb;
		for (int i = 0; i < shapes.size(); ++i) {
			qlaabox<real> bb = shapes[i].origShape->GetBoundingBox();
			bboxes[i] = bvh::BoundingBox<real>(MasterBVHPrimitive::toVector3(bb.min), MasterBVHPrimitive::toVector3(bb.max));
			worldbb.Union(bb);
			centers[i] = MasterBVHPrimitive::toVector3(bb.Center());
		}

		builder.build(bvh::BoundingBox<real>(MasterBVHPrimitive::toVector3(worldbb.min), MasterBVHPrimitive::toVector3(worldbb.max)), bboxes, centers, shapes.size());
		bvh::LeafCollapser<bvh::Bvh<real>> collapser(bvh);
		collapser.collapse();
		std::cout << "Radius: " << builder.search_radius << std::endl;
		std::cout << "Shapes count : " << shapes.size() << std::endl;
		//std::cout << "Tree depth : " << TreeDepth(root) << std::endl;
		std::cout << "Node count : " << bvh.node_count << std::endl;
		//std::cout << "Leaf count : " << LeafCount(root) << std::endl;
		/*std::cout << "Average primitives/leaf : " << ((real)ShapesCount(root)) / LeafCount(root) << std::endl;
		std::cout << "Empty leaves : " << EmptyLeaves(root) << std::endl;
		std::cout << "Max shapes : " << MaxShapes(root) << std::endl;*/
	}

And the traversal code:
	bool Intersect(const ray<vector4<real>>& r, SurfacePoint& hitdesc) const
	{
		// Intersect a ray with the data structure
		Ray ray = MasterBVHPrimitive::toRay(r);

		bvh::ClosestPrimitiveIntersector<bvh::Bvh<real>, MasterBVHPrimitive> primitive_intersector(bvh, shapes.data());
		bvh::SingleRayTraverser<bvh::Bvh<real>> traverser(bvh);

		auto hit = traverser.traverse(ray, primitive_intersector);
		if (hit)
		{
			hitdesc = hit->intersection.hitdesc;
			return true;
		}
		else
			return false;
	}

Thank you and regards,
Alessandro

Hello, and thank you for the bug report! Do you observe the same problem with all builders, or just bvh::LocallyOrderedClustering? As a side note, I am pretty sure that using int instead of an unsigned integer type for the Morton code type like you do is going to produce errors. I am going to add some static assertions to catch that.

Hello, and thank you for the bug report! Do you observe the same problem with all builders, or just bvh::LocallyOrderedClustering? As a side note, I am pretty sure that using int instead of an unsigned integer type for the Morton code type like you do is going to produce errors. I am going to add some static assertions to catch that.

Thank you for the hint about the int issue, will check that for sure this afternoon. I will also check that my custom intersector is correct and doesn't lose triangles here and there :D

Thank you for the hint about the int issue, will check that for sure this afternoon. I will also check that my custom intersector is correct and doesn't lose triangles here and there :D

And of course I will also try with other builder and let you know...

And of course I will also try with other builder and let you know...

Note that if you have an OBJ file for the scene you can try it with the benchmark tool provided in the test directory:
benchmark scene.obj --builder locally_ordered_clustering --eye x y z --dir x y z --up x y z

Note that if you have an OBJ file for the scene you can try it with the benchmark tool provided in the test directory:
benchmark scene.obj --builder locally_ordered_clustering --eye x y z --dir x y z --up x y z

Hello,
I've been able to test with:
//bvh::SweepSahBuilder<bvh::Bvh> builder(bvh);
//bvh::LocallyOrderedClusteringBuilder<bvh::Bvh, unsigned int>  builder(bvh);
bvh::LinearBvhBuilder<bvh::Bvh, unsigned int> builder(bvh);
SweepSahBuilder better than LocallyOrderedClusteringBuilder, LinearBvhBuilder better than SweepSahBuilder. With SweepSahBuilder that's the result:

the lamp is never 100% correct (see my BIH render) and there are a couple missing triangles. I tried with BinnedSahBuilder but it was suspiciously. So slow, it might have been testing each triangle for each ray (pixels were slowing being printed, 1 for every second or so). I suppose I was using it the wrong way. I could not test SpatialSplitBvhBuilder because it requires an additional array of primitives and I have not been able to provide it (it's late + I don't use c++ very often since 10 years or so + I'm not familiar at all with 11). If needed, no problems in writing some code tomorrow to make it work (maybe a copy ctor for my triangle would do the trick).
I might as well try your benchmark, if I find some more time to compile it.
Let me know if you want some more test.
Regards,
Alessandro

Hello,
I've been able to test with:
//bvh::SweepSahBuilder<bvh::Bvh> builder(bvh);
//bvh::LocallyOrderedClusteringBuilder<bvh::Bvh, unsigned int>  builder(bvh);
bvh::LinearBvhBuilder<bvh::Bvh, unsigned int> builder(bvh);
SweepSahBuilder better than LocallyOrderedClusteringBuilder, LinearBvhBuilder better than SweepSahBuilder. With SweepSahBuilder that's the result:

the lamp is never 100% correct (see my BIH render) and there are a couple missing triangles. I tried with BinnedSahBuilder but it was suspiciously. So slow, it might have been testing each triangle for each ray (pixels were slowing being printed, 1 for every second or so). I suppose I was using it the wrong way. I could not test SpatialSplitBvhBuilder because it requires an additional array of primitives and I have not been able to provide it (it's late + I don't use c++ very often since 10 years or so + I'm not familiar at all with 11). If needed, no problems in writing some code tomorrow to make it work (maybe a copy ctor for my triangle would do the trick).
I might as well try your benchmark, if I find some more time to compile it.
Let me know if you want some more test.
Regards,
Alessandro

Can you provide the implementation for MasterBVHPrimitive? Can you also say if real is float or double?
If all the BVH builders exhibit this issue, chances are the problem stems from either the data fed to the BVH building algorithm, or from the primitive intersection routine, or both. I cannot reproduce this issue, even using your scene, which I suppose is the living room scene from the Mc Guire CG archive (https://casual-effects.com/g3d/data10/index.html#mesh21):

As an additional note, using the centers of the triangles gives better BVHs than using the centers of the bounding boxes like you do in the example above. I think the simplest option if you want to debug the issue is to just use bvh::Triangle, by copying your shapes into an array or std::vector of triangles. Then, you can work out which pieces are not working properly by replacing the bits of the bvh::Triangle data structure one by one.
As an added bonus, the data layout of bvh::Triangle is carefully crafted to speed up the intersection routine too, so it will most likely make things run faster too. Moreover, bvh::Triangle is compatible with bvh::compute_bounding_boxes_and_centers() and bvh::compute_bounding_boxes_union(), which you can use to compute the bounding box and centers of each triangle, and the global bounding box, in parallel.

Can you provide the implementation for MasterBVHPrimitive? Can you also say if real is float or double?
If all the BVH builders exhibit this issue, chances are the problem stems from either the data fed to the BVH building algorithm, or from the primitive intersection routine, or both. I cannot reproduce this issue, even using your scene, which I suppose is the living room scene from the Mc Guire CG archive (https://casual-effects.com/g3d/data10/index.html#mesh21):

As an additional note, using the centers of the triangles gives better BVHs than using the centers of the bounding boxes like you do in the example above. I think the simplest option if you want to debug the issue is to just use bvh::Triangle, by copying your shapes into an array or std::vector of triangles. Then, you can work out which pieces are not working properly by replacing the bits of the bvh::Triangle data structure one by one.
As an added bonus, the data layout of bvh::Triangle is carefully crafted to speed up the intersection routine too, so it will most likely make things run faster too. Moreover, bvh::Triangle is compatible with bvh::compute_bounding_boxes_and_centers() and bvh::compute_bounding_boxes_union(), which you can use to compute the bounding box and centers of each triangle, and the global bounding box, in parallel.

Hello, and thank you again! (beautiful render btw, just out of curiosity: how long did it take to render, and at which resolution?)
I will most try to test with your triangle (not sure about replacing my class though, I'd like to leave at least some of my code in my renderer XD XD). Real is float.
I will post my MasterBVHPrimitive implementation here, but don't want you to lose more time if the bug is in my code, I'll first test as you suggested. I don't think the problem is in the intersection routine, as it works when using BIH... well, we will see. I'll also use the primitive centers.
Thank you again

Hello, and thank you again! (beautiful render btw, just out of curiosity: how long did it take to render, and at which resolution?)
I will most try to test with your triangle (not sure about replacing my class though, I'd like to leave at least some of my code in my renderer XD XD). Real is float.
I will post my MasterBVHPrimitive implementation here, but don't want you to lose more time if the bug is in my code, I'll first test as you suggested. I don't think the problem is in the intersection routine, as it works when using BIH... well, we will see. I'll also use the primitive centers.
Thank you again

If you have some code online I'd be happy to have a look and even debug this myself. I don't think that's going to take too much of my time. I suggested using bvh::Triangle only for debugging: If it works with it then maybe you can determine the parts of the code that are faulty. To be clear, I am almost certain that this is a bug in your code given that I can render that same scene and that there are tests to verify the correctness of various parts of the library, but it still is technically possible that this is a bug in this library because of some unusual setup or scene input, so hopefully using bvh::Triangle will also help clearing that last doubt.
If you cannot share the code for MasterBVHPrimitive then I cannot really help debugging beyond giving general advice such as:

Trying a simpler scene (e.g. Cornell Box) to see if the problem appears there,
Making sure that the primitive data (and their order in the array) does not change between the call to build() and traverse(),
Making sure the conversion functions (to/from vector or bounding boxes of your program and this library) are correct,
Making sure your rays are correct (in particular, the tmin/tmax values).

The rendered picture I linked above took around 1min to render on my laptop, using Path Tracing with MIS, at a resolution of 1080x720. It's not completely converged as you can see from the remaining noise.

If you have some code online I'd be happy to have a look and even debug this myself. I don't think that's going to take too much of my time. I suggested using bvh::Triangle only for debugging: If it works with it then maybe you can determine the parts of the code that are faulty. To be clear, I am almost certain that this is a bug in your code given that I can render that same scene and that there are tests to verify the correctness of various parts of the library, but it still is technically possible that this is a bug in this library because of some unusual setup or scene input, so hopefully using bvh::Triangle will also help clearing that last doubt.
If you cannot share the code for MasterBVHPrimitive then I cannot really help debugging beyond giving general advice such as:

Trying a simpler scene (e.g. Cornell Box) to see if the problem appears there,
Making sure that the primitive data (and their order in the array) does not change between the call to build() and traverse(),
Making sure the conversion functions (to/from vector or bounding boxes of your program and this library) are correct,
Making sure your rays are correct (in particular, the tmin/tmax values).

The rendered picture I linked above took around 1min to render on my laptop, using Path Tracing with MIS, at a resolution of 1080x720. It's not completely converged as you can see from the remaining noise.

Hello,

If you have some code online I'd be happy to have a look and even debug this myself.
No code online, unfortunately :-( My code is relatively clean for the most part, but the project is a mess and with a lot of dependancies like assimp, FreeImage, boost and Oidn (is an old project I abandoned 10 years ago due little free time. I got back to it without really thinking to invest much time, but its just too entertaining to work on it XD). I will most certainly try with you Triangle as soon as possible and report here the results.

The rendered picture I linked above took around 1min to render on my laptop, using Path Tracing with MIS, at a resolution of 1080x720
Simply Amazing!
It's not completely converged as you can see from the remaining noise.
Oidn would turn it in a perfectly clean image. It does wonders, and it's straightforward to integrate...

I'll let you know...

Hello,

If you have some code online I'd be happy to have a look and even debug this myself.
No code online, unfortunately :-( My code is relatively clean for the most part, but the project is a mess and with a lot of dependancies like assimp, FreeImage, boost and Oidn (is an old project I abandoned 10 years ago due little free time. I got back to it without really thinking to invest much time, but its just too entertaining to work on it XD). I will most certainly try with you Triangle as soon as possible and report here the results.

The rendered picture I linked above took around 1min to render on my laptop, using Path Tracing with MIS, at a resolution of 1080x720
Simply Amazing!
It's not completely converged as you can see from the remaining noise.
Oidn would turn it in a perfectly clean image. It does wonders, and it's straightforward to integrate...

I'll let you know...

Hello again!
I had the following tests: first I switched to primitive centroid instead than bbox. Then I noticed that I could kind of reproduce similar issues with my BIH by setting my epsilon too small or too large and tried playing a bit with those values. Then I had a try with real = double. I found and fixed a bug in my MasterBVHPrimitive.
As these quick things did not work, I used your Triangle implementation, first by creating one on the fly in my Triangle intersection routine, delegate the intersection computation, and fill my intersection structure accordingly, and then by adding a new member variable of type Triangle to my QLTriangle. Nothing changed:

I've noticed that today (and well before I started playing with the code) the wrong features are different: in particular, the fireplace looks fine. I can't explain why.
My MasterBVHPrimitive is as follows:
`	using Vector3 = bvh::Vector3;
using Triangle = bvh::Triangle;
using Ray = bvh::Ray;
struct MasterBVHPrimitive {
	QLShape* origShape;

	struct Intersection {
		real t;
		SurfacePoint hitdesc;
		// Required member: returns the distance along the ray
		real distance() const { return t; }
	};

	// Required type: the floating-point type used
	using ScalarType = real;
	// Required type: the intersection data type returned by the intersect() method
	using IntersectionType = Intersection;

	MasterBVHPrimitive() : origShape(NULL) {}
	MasterBVHPrimitive(QLShape* origShape) : origShape(origShape) {}

	// Required member: returns the center of the primitive
	Vector3 center() const {
		return toVector3(((QLTriangle*)origShape)->GetTriangle()->GetCentroid());
	}

	// Required member: returns a bounding box for the primitive (tighter is better)
	bvh::BoundingBox<real> bounding_box() const {
		qlaabox<real> bb = origShape->GetBoundingBox();
		return bvh::BoundingBox<real>(toVector3(bb.min), toVector3(bb.max));
	}

	// Required member: computes the intersection between a ray and the primitive
	std::optional<Intersection> intersect(const Ray& r) const {
		ray<vector4<real>> qlr = toQLRay(r);
		Intersection intersection;
		if (origShape->Intersect(qlr, intersection.hitdesc))
		{
			intersection.t = intersection.hitdesc.point.distance;
			return std::make_optional<Intersection>(intersection);
		}
		else
			return {};
	}

	static const bvh::Vector3<real> toVector3(const vector4<real>& v) { return bvh::Vector3<real>(v.x, v.y, v.z); }
	static const vector4<real> toQLVector(const bvh::Vector3<real>& v) { return vector4<real>(v[0], v[1], v[2]); }
	static const bvh::Ray<real> toRay(const ray<vector4<real>>& r) {
		return 	bvh::Ray<real>(toVector3(r.origin), toVector3(r.direction), r.mindistance, r.maxdistance);
	}
	static const ray<vector4<real>> toQLRay(const bvh::Ray<real>& r) {
		return ray<vector4<real>>(toQLVector(r.origin), toQLVector(r.direction), r.tmin, r.tmax);
	}

};`

That's my intersection code, which I think is pretty much correct (it is also almost 15 years old :D):
`		bool Intersect(const ray<vector4>& r, T &ucoord, T &vcoord) const
{
assert(data != NULL);
		const vector4<T> b = VERTEX(_B) - VERTEX(_A);
		const vector4<T> c = VERTEX(_C) - VERTEX(_A);

		//Find the determinant
		vector4<T> pvec = r.direction.Cross(c);
		T det = b.Dot3(pvec);

		//IF CULL
	/*	if(det < T(Q_EPSILON)) return false;

		//U parameter
		vector4<T> tvec = r.origin - VERTEX(_A);
		T u = tvec.Dot3(pvec);
		if(u < T(0.0) || u > det) return false;

		//V parameter
		vector4<T> qvec = tvec.Cross(b);
		T v = r.direction.Dot3(qvec);
		if(v < T(0.0) || u + v > det) return false;

		//T parameter
		T t = c.Dot3(qvec);

		T inv_det = T(1.0) / det;
		t *= inv_det;
		u *= inv_det;
		v *= inv_det;

		/*ALTRIMENTI*/
		if(IsZero(det)) return false;

		T inv_det = T(1.0) / det;

		vector4<T> tvec = r.origin - VERTEX(_A);

		//U parameter
		T u =  tvec.Dot3(pvec) * inv_det;
		if(u < T(0.0) || u > T(1.0)) return false;

		//V parameter
		vector4<T> qvec = tvec.Cross(b);
		T v = r.direction.Dot3(qvec) * inv_det;

		if(v < T(0.0) || u + v > T(1.0)) return false;

		//T parameter
		T t = c.Dot3(qvec) * inv_det;

		/*END IF*/
		if(t < r.maxdistance && t > T(0))
		{
			r.maxdistance = t;
			ucoord = u;
			vcoord = v;
			return true;
		}
		return false;
	}`

In the next couple of days I'll try to test with your Triangle class as you describe.
Thank you again and best regards,

Hello again!
I had the following tests: first I switched to primitive centroid instead than bbox. Then I noticed that I could kind of reproduce similar issues with my BIH by setting my epsilon too small or too large and tried playing a bit with those values. Then I had a try with real = double. I found and fixed a bug in my MasterBVHPrimitive.
As these quick things did not work, I used your Triangle implementation, first by creating one on the fly in my Triangle intersection routine, delegate the intersection computation, and fill my intersection structure accordingly, and then by adding a new member variable of type Triangle to my QLTriangle. Nothing changed:

I've noticed that today (and well before I started playing with the code) the wrong features are different: in particular, the fireplace looks fine. I can't explain why.
My MasterBVHPrimitive is as follows:
`	using Vector3 = bvh::Vector3;
using Triangle = bvh::Triangle;
using Ray = bvh::Ray;
struct MasterBVHPrimitive {
	QLShape* origShape;

	struct Intersection {
		real t;
		SurfacePoint hitdesc;
		// Required member: returns the distance along the ray
		real distance() const { return t; }
	};

	// Required type: the floating-point type used
	using ScalarType = real;
	// Required type: the intersection data type returned by the intersect() method
	using IntersectionType = Intersection;

	MasterBVHPrimitive() : origShape(NULL) {}
	MasterBVHPrimitive(QLShape* origShape) : origShape(origShape) {}

	// Required member: returns the center of the primitive
	Vector3 center() const {
		return toVector3(((QLTriangle*)origShape)->GetTriangle()->GetCentroid());
	}

	// Required member: returns a bounding box for the primitive (tighter is better)
	bvh::BoundingBox<real> bounding_box() const {
		qlaabox<real> bb = origShape->GetBoundingBox();
		return bvh::BoundingBox<real>(toVector3(bb.min), toVector3(bb.max));
	}

	// Required member: computes the intersection between a ray and the primitive
	std::optional<Intersection> intersect(const Ray& r) const {
		ray<vector4<real>> qlr = toQLRay(r);
		Intersection intersection;
		if (origShape->Intersect(qlr, intersection.hitdesc))
		{
			intersection.t = intersection.hitdesc.point.distance;
			return std::make_optional<Intersection>(intersection);
		}
		else
			return {};
	}

	static const bvh::Vector3<real> toVector3(const vector4<real>& v) { return bvh::Vector3<real>(v.x, v.y, v.z); }
	static const vector4<real> toQLVector(const bvh::Vector3<real>& v) { return vector4<real>(v[0], v[1], v[2]); }
	static const bvh::Ray<real> toRay(const ray<vector4<real>>& r) {
		return 	bvh::Ray<real>(toVector3(r.origin), toVector3(r.direction), r.mindistance, r.maxdistance);
	}
	static const ray<vector4<real>> toQLRay(const bvh::Ray<real>& r) {
		return ray<vector4<real>>(toQLVector(r.origin), toQLVector(r.direction), r.tmin, r.tmax);
	}

};`

That's my intersection code, which I think is pretty much correct (it is also almost 15 years old :D):
`		bool Intersect(const ray<vector4>& r, T &ucoord, T &vcoord) const
{
assert(data != NULL);
		const vector4<T> b = VERTEX(_B) - VERTEX(_A);
		const vector4<T> c = VERTEX(_C) - VERTEX(_A);

		//Find the determinant
		vector4<T> pvec = r.direction.Cross(c);
		T det = b.Dot3(pvec);

		//IF CULL
	/*	if(det < T(Q_EPSILON)) return false;

		//U parameter
		vector4<T> tvec = r.origin - VERTEX(_A);
		T u = tvec.Dot3(pvec);
		if(u < T(0.0) || u > det) return false;

		//V parameter
		vector4<T> qvec = tvec.Cross(b);
		T v = r.direction.Dot3(qvec);
		if(v < T(0.0) || u + v > det) return false;

		//T parameter
		T t = c.Dot3(qvec);

		T inv_det = T(1.0) / det;
		t *= inv_det;
		u *= inv_det;
		v *= inv_det;

		/*ALTRIMENTI*/
		if(IsZero(det)) return false;

		T inv_det = T(1.0) / det;

		vector4<T> tvec = r.origin - VERTEX(_A);

		//U parameter
		T u =  tvec.Dot3(pvec) * inv_det;
		if(u < T(0.0) || u > T(1.0)) return false;

		//V parameter
		vector4<T> qvec = tvec.Cross(b);
		T v = r.direction.Dot3(qvec) * inv_det;

		if(v < T(0.0) || u + v > T(1.0)) return false;

		//T parameter
		T t = c.Dot3(qvec) * inv_det;

		/*END IF*/
		if(t < r.maxdistance && t > T(0))
		{
			r.maxdistance = t;
			ucoord = u;
			vcoord = v;
			return true;
		}
		return false;
	}`

In the next couple of days I'll try to test with your Triangle class as you describe.
Thank you again and best regards,

Not sure if the errors you see are related, but you use 0 instead of ray.mindistance in the intersection routine. This means you might report intersections when there are none. Additionally, if you use an epsilon value in isZero(), you will get artefacts for long and thin triangles. Here's what I get with your intersection routine and the following definition for isZero():
bool isZero(Scalar x) { return std::fabs(x) < Scalar(1.e-5); }

Adjusting the epsilon only changes the scale at which the problem appears. Here with 1e-7, zoomed in on the radio:

I recommend using an intersection routine that is designed to reject NaNs and does not need an epsilon value. The trick is to remember that any comparison between a and b (in floating point) returns false if either a or b is a NaN (assuming IEEE-754, which is more than reasonable). The intersection routine in this library follows this principle.
I don't think the artefacts you report are due to your intersection routine, though, since I only see them on small objects and not in the same places as you do. I still think something might be wrong during construction. I also thought that it could be related to ray offsets.  Do you offset your rays from surfaces when bouncing? If you don't use the tmin of the ray, I suppose you use the normal? Something could be wrong there.
Finally, are the artefacts you observe exactly the same for each run? If they differ, if only a bit, it might make sense to use valgrind to determine if you have any illegal memory access or uninitialized memory that could explain this behavior.

Not sure if the errors you see are related, but you use 0 instead of ray.mindistance in the intersection routine. This means you might report intersections when there are none. Additionally, if you use an epsilon value in isZero(), you will get artefacts for long and thin triangles. Here's what I get with your intersection routine and the following definition for isZero():
bool isZero(Scalar x) { return std::fabs(x) < Scalar(1.e-5); }

Adjusting the epsilon only changes the scale at which the problem appears. Here with 1e-7, zoomed in on the radio:

I recommend using an intersection routine that is designed to reject NaNs and does not need an epsilon value. The trick is to remember that any comparison between a and b (in floating point) returns false if either a or b is a NaN (assuming IEEE-754, which is more than reasonable). The intersection routine in this library follows this principle.
I don't think the artefacts you report are due to your intersection routine, though, since I only see them on small objects and not in the same places as you do. I still think something might be wrong during construction. I also thought that it could be related to ray offsets.  Do you offset your rays from surfaces when bouncing? If you don't use the tmin of the ray, I suppose you use the normal? Something could be wrong there.
Finally, are the artefacts you observe exactly the same for each run? If they differ, if only a bit, it might make sense to use valgrind to determine if you have any illegal memory access or uninitialized memory that could explain this behavior.

Thank you for the hints about the intersection method. As said, that one of the olders part of the program, I might gave skipped some details (i.e. checking distance against 0 instead than min). I'll consider different algorithms in the future,
Now, look at that, this is really, really weird and I could not come out with an explanation not involving exotic quantum effects:

That's how it looks in Blender:

The "holes" are not missing triangles. It looks like they are kind of a boolean operations (front triangle - back triangle) or something like that. Same thing for other 2 holes (which aren't holes, I've tried changing the "no hit/background" color, but they are always black. I'm trying to understand if they are shading issues i.e. normal or something like that)
In addition, the same exact problem, with the same holes, also when moving the pov:

I don't think this is something related to the intersection routine: as you said, something related to the bvh building.  I'll keep investigating...

Thank you for the hints about the intersection method. As said, that one of the olders part of the program, I might gave skipped some details (i.e. checking distance against 0 instead than min). I'll consider different algorithms in the future,
Now, look at that, this is really, really weird and I could not come out with an explanation not involving exotic quantum effects:

That's how it looks in Blender:

The "holes" are not missing triangles. It looks like they are kind of a boolean operations (front triangle - back triangle) or something like that. Same thing for other 2 holes (which aren't holes, I've tried changing the "no hit/background" color, but they are always black. I'm trying to understand if they are shading issues i.e. normal or something like that)
In addition, the same exact problem, with the same holes, also when moving the pov:

I don't think this is something related to the intersection routine: as you said, something related to the bvh building.  I'll keep investigating...

It looks like the indices are incorrect. I have seen this type of artefact when converting the quads of a scene into triangles and messing up the order of the indices.

It looks like the indices are incorrect. I have seen this type of artefact when converting the quads of a scene into triangles and messing up the order of the indices.

I'm starting from scratch, that is from the original obj file, imported in a new blender scene. I have still a few things I'd like to check, ie the geometric normals: my shading normals look fine, usually, but if I use the geometric ones (even only to calculate the offset from the hit point) I get a mess.

I'm starting from scratch, that is from the original obj file, imported in a new blender scene. I have still a few things I'd like to check, ie the geometric normals: my shading normals look fine, usually, but if I use the geometric ones (even only to calculate the offset from the hit point) I get a mess.

Hello, just wanted you to know that I could get some pretty decent results by starting from the obj file instead than from the collada one:

a few hours of debugging: the lib I use to load input files was passing me wrong materials for some of the meshes, some other meshes had no normals etc. Still, a few holes and if I use collada thing did not change much. I've noticed that a couple tris were missing even while using my BIH when I left the assimp lib generate normals and tangent space, while by skipping that prepass things are nice when using obj (not yet when using collada, but I think it's a matter of the wrong material. Maybe):

(noiser because waaaay slower).
Not sure what is happening. I'd say there are a couple thing playing at the same time (normals, materials, maybe something with the epsilon annoying the BIH traversal?).
I will have another look as soon as possible.
Regards,
Alessandro

Hello, just wanted you to know that I could get some pretty decent results by starting from the obj file instead than from the collada one:

a few hours of debugging: the lib I use to load input files was passing me wrong materials for some of the meshes, some other meshes had no normals etc. Still, a few holes and if I use collada thing did not change much. I've noticed that a couple tris were missing even while using my BIH when I left the assimp lib generate normals and tangent space, while by skipping that prepass things are nice when using obj (not yet when using collada, but I think it's a matter of the wrong material. Maybe):

(noiser because waaaay slower).
Not sure what is happening. I'd say there are a couple thing playing at the same time (normals, materials, maybe something with the epsilon annoying the BIH traversal?).
I will have another look as soon as possible.
Regards,
Alessandro

Hello! Do you have any update? Should I close the issue?

Hello! Do you have any update? Should I close the issue?

Hello, sorry for not having updated the status! Truth is, I could not
identify the true cause: sometimes I was sure it was something in the BVH,
because my BIH was working fine, sometimes I suspected a bug in the mesh
loader, and sometimes I could not help but wondering if there was something
wrong with my shading code.
I've currently disabled culling (now rays intersect triangles on both
sides) and with the current obj living room (and its collada converted
version) It looks like I no longer have these glitches. I did not have much
time to try with other scene/settings however. Please close the issue.
Should I ever see them again, I will report them here (if I can exclude my
code/external libraries).
Thank you again!
Il giorno gio 28 gen 2021 alle ore 12:48 Arsène Pérard-Gayot <
<EMAIL_ADDRESS>ha scritto:

Hello! Do you have any update? Should I close the issue?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/madmann91/bvh/issues/28#issuecomment-769000016, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ASLDGH7I5QOOCU5A6CKAFATS4FFHZANCNFSM4WLUGFLA
.

Hello, sorry for not having updated the status! Truth is, I could not
identify the true cause: sometimes I was sure it was something in the BVH,
because my BIH was working fine, sometimes I suspected a bug in the mesh
loader, and sometimes I could not help but wondering if there was something
wrong with my shading code.
I've currently disabled culling (now rays intersect triangles on both
sides) and with the current obj living room (and its collada converted
version) It looks like I no longer have these glitches. I did not have much
time to try with other scene/settings however. Please close the issue.
Should I ever see them again, I will report them here (if I can exclude my
code/external libraries).
Thank you again!
Il giorno gio 28 gen 2021 alle ore 12:48 Arsène Pérard-Gayot <
<EMAIL_ADDRESS>ha scritto:

Hello! Do you have any update? Should I close the issue?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/madmann91/bvh/issues/28#issuecomment-769000016, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ASLDGH7I5QOOCU5A6CKAFATS4FFHZANCNFSM4WLUGFLA
.

Alright, thank you for the update! Closing the issue. Do not hesitate to re-open if required.

Alright, thank you for the update! Closing the issue. Do not hesitate to re-open if required.


faster SqlParser

This requires a rebase now.
Please try to keep the changes isolated per topic
If you also want to fix up other topics, those should be separate PRs for quicker review and being able to merge them.


Proposed timeline for Iron Irwini

This PR proposes the timeline of the final days of Iron Irwini development. It is a compressed timeline compared with Humble, being only 6 weeks long. Please discuss.

Not sure if this is out of scope for this PR, but when does the EOL date get defined? It's still TBD here.

THis is relying on rolling being ready to fork and already targeting the appropriate platforms for Iron. I would suggest that we provide a few more deadlines earlier than the beginning of this cycle to make sure that people are aware of when to expect that. Such as "Estimated January: Rolling transition to Iron target platforms" we can't necessarily know exactly when the target platforms will be available though we should move to them earlier rather than later to find any issues. For Iron it's still going to be the Ubuntu LTS so we could switch that over much earlier. But in general we can't necessarily expect our target platforms available in beta until ~ 6 months ahead of the release.

I think that the two weeks shorter schedule is feasible. However, I see real advantages more with the next LTS release: Then, package maintainers have more time to adapt their packages within Rolling to the new Ubuntu LTS distro. This might be especially beneficial for packages with complex third-party dependencies.

THis is relying on rolling being ready to fork and already targeting the appropriate platforms for Iron. I would suggest that we provide a few more deadlines earlier than the beginning of this cycle to make sure that people are aware of when to expect that. Such as "Estimated January: Rolling transition to Iron target platforms" we can't necessarily know exactly when the target platforms will be available though we should move to them earlier rather than later to find any issues. For Iron it's still going to be the Ubuntu LTS so we could switch that over much earlier. But in general we can't necessarily expect our target platforms available in beta until ~ 6 months ahead of the release.

That's fair.  We haven't traditionally done that, but adding in some general guidelines for when we make platform choices is a good idea.  @gbiggs Please go ahead and make that change.

I added a couple of extra dates in 32bf286. Please take a look.

This pull request has been mentioned on ROS Discourse. There might be relevant details there:
https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-07-21-2022/26647/1


keyNotFound(code: "BFCL")

When running bclm v.0.0.3 there is an following error output:
sudo bclm write 75
keyNotFound(code: "BFCL")
Running on USB-C Macbook Air 2019. It seems that Macbooks with USB-C charging doesn't have BFCL in SMC.
BTW: Thank you for this great and dead simple tool.

Fixed in v0.0.4 now, can you tell I don't have a USB-C Mac? :)
Also thank you for the kind words! That was my exact goal when creating this.

Hi zackelia,
I am using Hackintosh on my T460s with v0.0.4, but I got the KeyNotFound(code: "BCLM") error.


[WIP] stats: add histogram for request headers size per upstream

Signed-off-by: Ranjith Kumar<EMAIL_ADDRESS>Commit Message: stats: add histogram for request headers size per upstream
Additional Description:
Risk Level: Low
Testing: Added test cases
Docs Changes: yes
Release Notes:
Fixes #10308

PTAL @nezdolik

All checks have passed. PTAL @nezdolik

After brief discussion with @jmarantz, proposed histogram should be enabled by config flag, as it is expensive to track histogram per cluster with large number of clusters. @ranjithkumar007 here is example of histogram controlled by config flag: https://github.com/envoyproxy/envoy/pull/9227

here is example of histogram controlled by config flag: #9227

Thanks @nezdolik . So, we need a new entry in cluster config.
I was wondering how to add such entry that would fit well for #11559 as well.
Should we have seperate entries like track_upstream_rq_headers_size, track_upstream_rq_body_size ? or a dict like track_upstream_rq_stats : {'headers_size': true, 'body_size' : false}?

Think the first approach is preferred.

Now that this stat is optional, there is no need to update memory limits in stats_integration_test.cc?

Does stats_integration_test.cc include optional stats as well while tracking memory consumption?

No, so you can back your change to stats_integration_test.cc, which should hopefully fix CI. Thanks!

/wait

you can back your change to stats_integration_test.cc

The memory consumption values(1, 2) from CI are exactly 256 bytes higher than the values on master(1, 2).
absl::optional<T>  occupied memory(so 256 bytes overhead) even if it was empty.

/wait

windows ci failure unrelated to PR

/wait

/wait

@repokitteh bot :rotating_light:, don't remove waiting label :neutral_face:

/wait

Ci tests passeD except Marcos which I guess is flaky and unrelated to Pr.
@jmarantz , weirdly stats_integration_test have passed without any modifications.

Can you look at the CI for docs? Failure looks real (I don't know much about it myself)

Can you look at the CI for docs? Failure looks real (I don't know much about it myself)

Yeah, I will update docs.

/wait

/wait

@ranjithkumar007 could you please update links to related issues in PR description? (as now we are addressing 2 issues in same PR)

/wait

/wait

/wait

/wait

Windows ci build failure looks unrelated to PR.

Please do not force-push in envoy github PRs -- it makes reviews difficult because comments get lost.
Just merge master.

Please do not force-push in envoy github PRs -- it makes reviews difficult because comments get lost.
Just merge master.

I force pushed on top of a single commit that I made after the review.
I will make sure to merge master every time from now on.  Apologies for this time.

@ranjithkumar007 dismissed jmarantz’s stale review via f88549a 9 minutes ago
I am not sure why/when did this happen :confused: ? I just clicked on resolve conversation.

Will fix ci.
/wait

/wait

CI tests are passing except coverage which looks like a flaky issue.
I read Breaking change policy, API versioning and tried to modify code following those. In the code, I deprecated track_timeout_budgets in v3, removed it and reserved its field number in v4alpha. The generated config for v4alpha still has track_timeout_budgets in the form of hidden_envoy_deprecated_track_timeout_budgets.  Currently, I updated the code to track timeout stats based on config.has_track_cluster_stats() or config.track_timeout_budgets() where config is v3 Cluster config . So, when transitioning from v3 to v4, is this updated to config.has_track_cluster_stats() or config.hidden_envoy_deprecated_track_timeout_budgets() and from v4 to v5, do we get rid of track_timeout_budgets completely?

@envoyproxy/api-shepherds kind ask for api review

@jmarantz There is a decline of 256 bytes in memory usage per cluster. On master it is 45003. On this PR it shows 44747.

Awesome; one last regold away from passing release tests. Since you changed an existing large structure from absl::optional to unique_ptr, the memory reduction makes sense to me.
Nice job! Fix CI and this LGTM.

/lgtm api

/azp run

hds_integration_test is timing out again. Possibly a flaky test.

/azp run
RE hds_integration_test timing out...not clear whether this is just death by 3 seconds/test-method * lots of test methods, or something is getting stuck. Hopefully we'll catch a faster machine on this run.

PR lgtm apart from the point of adding back docs for deprecated field.

retest

Looks like hds flaky test is the reason of failing CI, @ranjithkumar007 could you try to retrigger CI with /retest comment?

Yeah, I will do it once I update the docs.

Thanks everyone!  Good to have first contribution to envoy merged 😀 .


Add postcss-message to show postcss 4.1 message output ?

Related to #64
What do you think about that as an option?
https://github.com/postcss/postcss-messages

:+1:

I will go for a custom skin by default with the logo/color of cssnext. Do you think it's a good idea ?

I think yes. We will see that the error is raised by cssnext

Also for cli https://github.com/davidtheclark/postcss-log-warnings

How does it look ?

Woooot awesome.

Is cssnext actually parsing your error and suggesting an accurate fix for it? That's pretty cool.
Related to the error formatting: is it possible to use whitespace to better indicate the problem line and the corrected line?
Something like:
> cssnext:custom-selectors:missing-colon-fix: <css input>:1:30: Incorrect syntax for @custom-selector.

@custom-selector --h h1, h2;

Should be:

@custom-selector :--h h1, h2;

I also see the "Please adjust this..." sentence as maybe being unnecessary.
Anyway, those are just my 2¢ as a relative outsider who apparently has an opinion on this.

My question was more for the design that the content itself.
That said, i will probably adjust messages with some newlines, that's a good idea.
For the record some plugin currently use console.log to talk to users (eg. postcss-custom-properties might warns you if you try to use an undefined var). Since it's not really users friendly (especially when you are using postcss in a verbose env like webpack) you might miss those log. That's why I asked this API for postcss.
This way we will be able to properly emit warnings in the appropriate way depending in the env (webpack, gulp, cli...) + in the css itself using the famous hack ::before/content.

That said, I really like the design. I know the blue complements the logo, but it has the side effect of being calming even in the midst of the potential frustrations of learning these new specs. Bonus!

I think I will replace the logo by another icon: { ! }

Is this any better ?

Or with more space ?

I like them both. What's nice about the additional space is that it makes it easier to pick out individual issues if you miss the > cssnext:.

Or with the icon in the top left corner

I like that better.

I will go for a simpler icon
Which one ?

I vote for the second one: being filled makes it more visible, and it keeps the shape of the {!} logo

first one IMO

The downside being that it looks like and information instead of a warning/error :/

Note that we might have here info/debug message in the future.

I prefer the third one, but the second one is quite ok and mostly efficient. So the 2nd one. The first one is "an error", not a notice.

I like the third one best. Second favorite is the second one.

Updated the project I'm using cssnext on to see how this works in practice, and I love this feature so much.

Glad you enjoy it.


Visualization with GLVis- Probe

Hi,
I'm using GLVis for visualization of my results generated by ex2p.   Is there a way to probe the values from a specific node/element?
Thanks
Shani

Does miniapps/tools/get-values.cpp works for you?

@shani9383 did you build with GSLIB? This miniapp requires it. See INSTALL.

If you want to interactively check the values at specific points, you can do this in ParaView with, for example, the "Hover Points" functionality.
After saving your solution with ParaViewDataCollection (see e.g. example 9), in ParaView you can enable "Hover Points":

Then, hover over the desired point and it will display the solution value:

Thank you so much. Ill try it.
On Wed, Jul 5, 2023 at 8:12 AM Will Pazner @.***> wrote:

If you want to interactively check the values at specific points, you can
do this in ParaView with, for example, the "Hover Points" functionality.
After saving your solution with ParaViewDataCollection (see e.g. example
9), in ParaView you can enable "Hover Points":
[image: hover_points]
https://user-images.githubusercontent.com/11493037/251042463-e2b358d5-8004-42f9-b69e-ff4bce9f2b65.png
Then, hover over the desired point and it will display the solution value:
[image: solution]
https://user-images.githubusercontent.com/11493037/251042502-89140075-678c-47e6-84fe-bd83679f2b20.png
—
Reply to this email directly, view it on GitHub
https://github.com/mfem/mfem/issues/3686#issuecomment-1621037556, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/A3ENVWMXF6GY4C5OX6QRV3TXOTZR7ANCNFSM6AAAAAAYMASEPY
.
You are receiving this because you were mentioned.Message ID:
@.***>


Fish commands for multiwerf

This might be a minority shell https://fishshell.com/ but it would be nice for multiwerf to provide some way of working with it.
(testing multiwerf & werf in bash so far)

Hello! Thanks for idea. I think it is totally possible to add support for fish and we would like to have such support.


react-native

Is there a way to use this with react-native?

Hey, no not really. There are also no plans to port this repo.


.dev domain extension

Not really an issue but more of a query, with Google releasing the .dev domain extension it's got me thinking. Because the domains need to have SSL setup to access will win-acme be able to setup the certificate for the first time or will it need to be done in a different way. I generally use the simple mode for a single binding.
Only asking as I've had issues previously where ACME wasn't able to validate the certificate because the site was already redirecting to https.  Maybe it's already resolved and validated the domain differently now.
Thanks
James

Only asking as I've had issues previously where ACME wasn't able to validate the certificate because the site was already redirecting to https. Maybe it's already resolved and validated the domain differently now.

I don't expect .dev to behave any differently than .com if/when Let's Encrypt accepts it as a valid root. They work with the public suffix list ( https://publicsuffix.org/list/public_suffix_list.dat ) where dev is listed, so it should be no problem as long as port 80 is publicly accessible. Otherwise you have to use DNS-validation.

Tested it and not an issue..


OnLoadStart Not Executed when url changes

onLoadStart:
(InAppWebViewController controller, String url) async {
print("ONloadStart exected $url");
if (url.startsWith("https://mySite.com/")) {
print("url #####################################$url");
         RegExp regExp = new RegExp("access_token=(.*?)&");
                  this.token = regExp.firstMatch(url)?.group(1);

                  print(
                      "ACCESS TOKEN INAPP WEBVIEW TOKEN #####################################$token ");

I had a similar issue in yoururl.startsWith("https://mySite.com/" add www. before your site it should work, i think in app web view adds that by default to all urls
Example :  url.startsWith("https://www.mySite.com/"

@nikhilmufc7 Thank you!Will try your solution.This issue is currently only on ios,In Android its working perfectly

@sahad752 I too am facing the same issue. Did the @nikhilmufc7 solution solve for you? No luck for me yet.

@bsp003 nop !The above solution didnt work.


Fix Create Release Notes GH Action

This repo doesn't have any self-hosted runners, but we can just run on ubuntu-latest.

Codecov Report

Merging #20 (13c408e) into main (25cb1e6) will decrease coverage by 0.36%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #20      +/-   ##
==========================================
- Coverage   76.52%   76.16%   -0.37%
==========================================
  Files         160      160
  Lines        2773     2773
  Branches       72       72
==========================================
- Hits         2122     2112      -10
- Misses        651      661      +10

Impacted Files
Coverage Δ

...kafka/streams/KafkaStreamLifeCycleManagement.scala
64.04% <0.00%> (-11.24%)
:arrow_down:

...StreamsUpdatePartitionsOnStateChangeListener.scala
82.85% <0.00%> (-8.58%)
:arrow_down:

.../surge/internal/kafka/KafkaProducerActorImpl.scala
88.57% <0.00%> (+0.40%)
:arrow_up:

...a/surge/internal/persistence/PersistentActor.scala
96.73% <0.00%> (+0.65%)
:arrow_up:

...n/scala/surge/internal/streams/StreamManager.scala
76.25% <0.00%> (+1.25%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25cb1e6...13c408e. Read the comment docs.


mbedtls_ssl_setup returned -0x7f00 while sending multiple mails (IDFGH-7167)

Environment

Development Kit:      [ESP32-DevKit]

Kit version (for WroverKit/PicoKit/DevKitC): [v1]

Module or chip used:  [ESP32-WROOM-32|]

IDF version (run git describe --tags to find it): v4.3

Compiler version : xtensa-esp32-elf-gcc (crosstool-NG esp-2021r1) 8.4.0

Operating System:     [Linux]

Using an IDE?: [VS Code]

Power Supply:         [USB]

Problem Description
I am trying to send mail, again and again, using the smtp_client example but after sending mail for 3 or 4 times the code crashes with the following error mbedtls_ssl_setup returned -0x7f00. Any help related to it would be grateful.
Thanks and Regards,
Maaz Shaikh

Hello, @MaazSk!
Thank you for sending the issue report. I was investigating this issue and was able to reproduce it with the smtp_client example on the ESP-IDF v4.3 tag.
Could you try out this patch and let me know if it works for you?

I was able to resolve the issue by adding the following lines sorry didn't update you
```

mbedtls_x509_crt_free(&cacert);
mbedtls_ssl_config_free(&conf);
mbedtls_ctr_drbg_free(&ctr_drbg);
mbedtls_ssl_free(&ssl);

Thanks and regards,
Maaz Shaikh


[Bug]: Allow user to enter full name of known hotspot in "Search" box

What happened?
Not sure if this is really a bug so much as a feature request.  And it is definitely not urgent.
When looking to pull up a hotspot on Explorer, the user has to wait until a search of the names in use based on the characters typed returns a match and then has to select the correct/desired result from a list.  This behavior/series of steps is necessary even if the user knows the full name.  And if the user knows the full name and hits enter after typing it in, the field just goes blank and does not return anything.
From a UX perspective, it seems as though it would be heaps better to allow the user to type the full name of the hotspot in, hit enter, and get the desired result without any extra steps involved.  This might tweak how/when the search function queries the API for the correct hotspot name string.  And it likely would require some additional "error hanlding" be created for errant names/no results.  However, such functionality is very common and it, therefore, seems it would be worthwhile from a UX perspective.
URL
https://explorer.helium.com/ and https://explorer.helium.com/hotspots
What browsers are you seeing the problem on?
Brave
Do you see the issue on Mobile or Desktop?
Desktop
Relevant log output
No response

so because the "animal name" is derived from a hotspot's blockchain address, it's actually not possible to know a hotspot's blockchain address from just its full animal name, so we actually do have to wait for the API's response in order to direct the browser to that page, even if you know the full animal name
but you're right that the UX of using the search bar can definitely be improved in a handful of ways, so I've addressed the UX parts of your feature request in the PR here: #739

Makes sense.  I figured there must be a reason as I couldn't imagine Team
Helium allowing such a thing!
Thanks for the feedback and otherwise handling.
On Tue, Nov 2, 2021 at 5:25 PM Daniel James @.***>
wrote:

so because the "animal name" is derived from a hotspot's blockchain
address, it's actually not possible to know a hotspot's blockchain address
from just its full animal name, so we actually do have to wait for the
API's response in order to direct the browser to that page, even if you
know the full animal name
but you're right that the UX of using the search bar can definitely be
improved in a handful of ways, so I've addressed the UX parts of your
feature request in the PR here: #739
https://github.com/helium/explorer/pull/739
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/helium/explorer/issues/735#issuecomment-958279200,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF6SQHFRPGUBBRFLN4IY3ZDUKBXNVANCNFSM5GSSKPMA
.


Notification contains stale background when cover image isn't set

I noticed that the notification (visible when pulling down the status bar) was displaying a stale background image while playing a track with no cover image set.
At first I thought that my code was failing to clear something, but now I'm thinking that this may be either a bug or weird-but-deliberate behavior on Android's part. It seems like when the updated notification doesn't set a large icon, Android is displaying a pixelated version (??) of the previous icon in the background, tinted to blue.
This is annoying, since Android is clearly capable of displaying a notification without a large icon -- that's what it does before any songs have been queued, and also when I truncate the playlist.
NotificationCreator.kt has this line, but I still see the same behavior when I make the condition also check cover != null:
if (!mediaId.isEmpty()) builder.setLargeIcon(cover)

It's not obvious to me what's different after I truncate the playlist (which causes the desired behavior of the icon not being shown). Maybe the difference is coming from the MediaSessionCompat, which gets passed in via setContentIntent(controller.sessionActivity) on the NotificationCompat.Builder and also here:
        val style = MediaStyle()
        style.setMediaSession(mediaSession.sessionToken)
        builder.setStyle(style)

This gets stupider the more I look at it. If I make MediaSessionManager set MediaMetadataCompat.METADATA_KEY_ALBUM_ART to an empty 256x256 bitmap, the stale cover image gets shown. If I set the bitmap to red instead, it gets used.

This is all so dumb. As best as I can tell, the notification (via NotificationCompat.MediaStyle?) is just compositing the new bitmap (or absence of a bitmap) on top of the old bitmap using the "over" operator. If the new bitmap has an empty alpha channel or doesn't exist, it keeps using the old bitmap. If the new bitmap has 50% alpha, it gets blended with the old bitmap.


Move off of project.json onto .csproj

This is significant change that moves omnisharp-roslyn from project.json to csproj. This means that, if anyone is using Visual Studio to develop OmniSharp must use VS 2017, which became generally available just yesterday.
There are a lot of changes in the build script related to building on macOS/Linux. With the latest .NET Core SDK, building a .NET Framework TFM (like net46) on macOS/Linux is no longer supported. So, we'll use the msbuild that ships as part of Mono 4.8. Note that anyone building on Linux must also install the msbuild package.

do we still need https://github.com/OmniSharp/omnisharp-roslyn/blob/dev/NuGet.Config#L7?  Roslyn is pulled from the public nuget feed now

@filipw:

do we still need https://github.com/OmniSharp/omnisharp-roslyn/blob/dev/NuGet.Config#L7? Roslyn is pulled from the public nuget feed now

Good point. I'll pull that out.

OK. Feedback should be addressed.


How to run VGate project on android ?

I want to develop an L2TP IPsec VPN  android application, I searched and found that I can use VGate project for my application, but I don't know how and which IDE I should use run it?

Visual studio

@GildedHonour  Should I import all SortEther project or just I need VGate classes
https://github.com/SoftEtherVPN/SoftEtherVPN_Stable/tree/master/src/VGate
I need L2TP IPsec VPN would you please tell me which folders or classes I need ?

I don't know.

As far as I know the source code for VPNGate is not openly released, although it uses many headers, c files from SoftEther.
Also vpngate has nothing to do L2TP or OpenVPN. On android there are many L2TP and VPN clients that can be directly used with a SoftEtherVPN Server.
Closing this ticket


Support better search engines for indexing (elasticsearch, mysql/pgsql/sqlite fulltext or something else)

Gitea 1.7.6
Git 2.11
Operating system: Linux
Database: all
Can you reproduce the bug at https://try.gitea.io:

[x] Not relevant

Description
Bleve indexer is very inefficient: it uses a lot of disk space and a lot of memory. Also it keeps all index mmap'ed all the time which makes Gitea crash when I enable it on my 32-bit server with just 1.4gb of git repositories after generating 2GB of index.
There exist a lot of more popular and efficient full-text search engines, starting with ones built into Postgres / MySQL / SQLite (MySQL's one is not the most efficient one, but it still works). Then there's Elasticsearch and so on. Index sizes are much smaller in Elasticsearch and Postgres (compared to size of indexed data).

Hi I've also seen in the configuration files that there're two types of ISSUE_INDEXER_TYPE available. What is the differences between "db" and "bleve"? Is it safe to change from "bleve" into "db" in config files and then a simple restart?

@adamcavendish db will use database's Like to search issues. Your operations are safe. But both types are not inefficient.

I guess this could fix things like #5694 #5277 #3448 and #2967 if implemented properly.

Would love to help with the implementation for the elasticsearch code search backend! @lunny @jeblair

Just a note that SQLite has FTS indexing, and that it is quite efficient (I have gigabytes of plain text files indexed that way)

So no built-in, single binary SQLite search?

@rcarmo issues index support built-in SQLite search, code index support built-in bleve search.


A bug in highlighting YAML files when using multi-line strings

VSCode Version: 1.19.3
OS Version: macOS 10.13.3

Steps to Reproduce:

Try using a multi-line string starting with a vertical bar.
Write some text on the next line.
Add text in quotes.
The last quote should be on a new line.
The rest of the text gets highlighted like a string.

EXAMPLES:

This produce the bug (see the attached image):

field1: 12
field2: |
  bash -c "
  git init
  "
field3:
  - 1234

This too:

field1: 12
field2: |
  bash -c "git init
  "
field3:
  - 1234

But this does not:

field1: 12
field2: |
  bash -c "git init
  && git add ."
field3:
  - 1234

So, the whole thing looks like it gets messed when a multi-line statement initialized with a vertical bar has a line with only quote.

Does this issue occur when all extensions are disabled?: Yes

This issue was moved to textmate/yaml.tmbundle#21


Updates to data for May demo

Based on the April 17th meeting, we will want to do one update of the data before the next demo.
These updates will include:

fixed format of the existing dataset
including all 67 cohorts that are available from the IHCC website CSV

From the JSON provided, we had to fix a couple things:

JSON strucure must be consistent between all objects.  A category cannot be both an array and an object.  Example:
https://github.com/jamesaoverton/IHCC/blob/54947b2edec2e4fd93dc899afbcba3a6008877b0/data/cohort-data.json#L31
https://github.com/jamesaoverton/IHCC/blob/54947b2edec2e4fd93dc899afbcba3a6008877b0/data/cohort-data.json#L76
No special characters in field names & only underscores
Examples:
https://github.com/jamesaoverton/IHCC/blob/54947b2edec2e4fd93dc899afbcba3a6008877b0/data/cohort-data.json#L118
Change the available_data_types array to individual boolean fields at the top level; assume empty in the csv is false. Ex:

{
    "genomic_data": true,
    "clinical_data": true,
    "phenotype_data": false
....

Thanks! @beckyjackson will make those changes, then add all the cohorts from https://github.com/jamesaoverton/IHCC/blob/master/data/member_cohorts.csv
I guess there's 67 rows, for 66 cohorts.

@beckyjackson one additional request to change the available_data_types format (edited as No. 3 above)

@rosibaj - for the consistency, would something like this work for you, or do you want the lowest-level always to be a list?
"questionnaire/survey data": {
  "lifestyle and behaviours": {
    "alcohol": {},
    "nutrition": {},
    "sleep": {},
    "tobacco": {}
  }, ...

@beckyjackson No, what you have proposed above would not work, unless each of the     "alcohol": {}, has additional elements in the ontologies that are assigned afterwards. So i think the best assumption her is:
Yes,

the lowest level should always be a list
the lowest level needs to be consistent between all cohort objects

For many children of 'questionnaire/survey data', there are next-level children, like this example:
"questionnaire/survey data": {
  "lifestyle and behaviours": [
    "alcohol",
    "tobacco"
  ]
}

But the problem comes if a cohort only looks at signs and symptoms from questionnaire/survey data, because this one doesn't have any children:
"questionnaire/survey data": [
    "signs and symptoms"
]

How would you like the data to show up in the above case, where only 'signs and symptoms' shows up under 'questionnaire/survey data'?
Thanks for your help!

@beckyjackson Its imperative that the JSON structure between cohorts be exactly the same.  This means that we definitely cannot have "questionnaire/survey data" as both an array and an object.
This is a harmonization issue with these options as solutions that I can see for now:

ignore the values on cohorts with no children
harmonize a single structure that works for all cohorts

seems the better option to me.  We can do a structure like this:

Cohort 1 (has children)
"questionnaire/survey data": {
      "lifestyle_and_behaviours": [
         "alcohol",
         "tobacco"
       ],
      "physiological_measurements": [
          "height",
          "weight"
        ],
      "general_variables": null,
}

Cohort 2 (has no children) so populate the general variables array with those values.
"questionnaire/survey data": {
      "lifestyle_and_behaviours": null
       "physiological_measurements": null
       "general_variables": ["signs and symptoms"] <-- group all possible cases were there are no child
}

Cohort 3 ( also has no children)
"questionnaire/survey data": {
      "lifestyle_and_behaviours": null
       "physiological_measurements": null
       "general_variables": ["any_potential_value"] <-- group all possible cases were there are no child
}

Note that the structure should remain the same in all cases.  Its not needed to populate the null fields (as that can be taken care of automatically), but it is important that the basic structure in all cohort documents is identical.
If there are other cases of cohorts having non-child entities in other objects, this same methodology can be used.
Does this make sense?

I want to make sure I'm interpreting this correctly:
If a cohort looks at only signs and symptoms, they get:
"questionnaire/survey data": {
    "general_variables": ["signs and symptoms"]
}

If a cohort looks at just medication (not worrying about the sub-categories, therefore this gets null since it has children, just not applicable for this cohort) and signs and symptoms they get:
"questionnaire/survey data": {
    "medication": null,
    "general_variables": ["signs and symptoms"]
}

Finally, if a cohort were to look at posology from medication, they would get:
"questionnaire/survey data": {
    "medication": ["posology"]
}

@beckyjackson i think that looks correct based on your description of cohorts!

Great - following this pattern, I regenerated the cohort data for what we currently have. If you have a chance, can you take a look at it and make sure it's what you expect? https://github.com/jamesaoverton/IHCC/blob/data-update/data/cohort-data.json
Thanks so much!

@beckyjackson I have reviewed - This structure looks great but there is still one small issue in the naming of fields:
Can we please remove spaces/special characters from fields names? For examples:

basic cohort attributes <-- remove   (spaces)
socio-demographic and economic characteristics <-- remove - and

@beckyjackson I have reviewed - This structure looks great but there is still one small issue in the naming of fields:
Can we please remove spaces/special characters from fields names? For examples:

basic cohort attributes <-- remove   (spaces)
socio-demographic and economic characteristics <-- remove - and

@beckyjackson sorry for the multiple comments! was a result of the github incident!

No worries! I couldn't even respond haha. I updated the file in the PR to remove the spaces and special characters.

@rosibaj We're hoping that https://github.com/jamesaoverton/IHCC/blob/master/data/full-cohort-data.json is exactly what you want. If you have any trouble with it, please reopen this issue.

@jamesaoverton It seems i dont have the permission to repoen this issue.  There are 2 small things:  Is it possible to have these fixed?

tiny thing… cirulation_and_respiration is supposed to be circulation_and_respiration ? (misspelled circulation
Can all cohort documents contain all possible fields, se either to null if a key/value or and empty array if the data does not exist for that cohort?

Hi @rosibaj - thanks for catching that typo, that's an easy fix to make!
As for your second point, I'm not sure I understand what you're asking.
In current structure, if a cohort looks at a broad category that has children, but the cohort doesn't care about the children, we give that category a null to keep the structure the same. For example, if a cohort looks at weight:
"questionnaire_survey_data": {
    ...
    "anthropometry": ["weight"],
    ...
}

But if they just look at the broad category of "anthropometry", they get:
"questionnaire_survey_data": {
    ...
    "anthropometry": null,
    ...
}

If we were to include all fields, I worry you'd end up with two separate cases that get null values. I think you want all the top-level categories, correct? But if a cohort looks at just the broad category, it gets null. Then, it would also get null if it doesn't look at it at all.
Perhaps I'm not understanding your request correctly, so could you provide an example? Thank you so much!

@beckyjackson
I understand the distinction that you are making.  In the long-run, i think a better way to do this is to actually assign a value (ex Not gathered) to be explicit about this. However, thats a discussion for a later time!
For the sake of the demo, can we just have the data formatted like this for now:
"questionnaire_survey_data": {
    ...
    "anthropometry": ["weight"],
    ...
}

But if they just look at the broad category of "anthropometry" rather than putting null, make it an empty array.
"questionnaire_survey_data": {
    ...
    "anthropometry": [],
    ...
}

Sure, that's not a probelm. Then, do you want the upper-level broad categories to all be displayed, and have null if they are not collected?

@beckyjackson yes, but populated as empty data.  Using biosample as an example, i would expect to see this in the 'Genomics England / 100,000 Genomes Project"` cohort document:
https://github.com/jamesaoverton/IHCC/blob/96848a04b640e29d5e323c27f2502744d93324a5/data/cohort-data.json#L86
"biosample": {
         "sample_type": [ ]
},
```

I'm sorry, I'm still a bit confused. Going back to the antropometry data, say they don't collect any questionnaire/survey data, are you saying you would still want to display all the fields that appear in the CINECA structure, even though they don't have anything mapped to this:
"questionnaire_survey_data": {
    ...
    "anthropometry": [],
    ...
}

How does this differ from if they do collect anthropometry data, but don't have something mapped to the children (currently displayed as "anthropometry": null, but will be changed to "anthropometry": [])?

@beckyjackson
In your examples, you are making the assumption that     "anthropometry": [], and   "anthropometry": null mean different things.  In fact, in the way that you are translating this informatino they do mean different things.
In the faceted search display (based on Elasticsearch index) there is no functional difference between a null and empty field. Both would be an empty facet in the display (meaning the same thing: this data does not exist for this filter).  Inconsistent documents are however harder to work with, which is why we prefer to at least have the empty data state.
I think that a possible solution to this is to include the category as a value of itself (see below) for examples:

Cohort does not collect anything from questionaire and survey data:

"questionnaire_survey_data": {
    ...
    "anthropometry": [No Data],
    ...
}

Cohort collects anthropometry but no children:

"questionnaire_survey_data": {
    ...
    "anthropometry": [anthropometry],
    ...
}

Cohort collects anthropometry with children:

"questionnaire_survey_data": {
    ...
    "anthropometry": [weight, height],
    ...
}

However, this constitutes a larger data modelling question that is not in the scope of this demo and warrants more thought.
For the demo purposes, can we continue with the empty array display?

I changed the null values to empty arrays and fixed the typo - please see the data file here.

@rosibaj We've updated https://github.com/jamesaoverton/IHCC/blob/master/data/full-cohort-data.json with those changes.


path/filepath: EvalSymlinks of a directory which is mapped from volumeID (instead of drive letter) does not work

What version of Go are you using (go version)?
go 1.13

$ go version

Does this issue reproduce with the latest release?
Not tested
What operating system and processor architecture are you using (go env)?
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
go env Output
$ go env

What did you do?
Create a symlink from volume id
mklink /D \my\path \?\Volume{1b3b1146-4076-11e1-84aa-806e6f6e6963}\
Running filepath.EvalSymlinks against a symlink (\my\path) which is created from a volumeid returns the following error:
too many links!

What did you expect to see?
What did you see instead?

/cc @robpike @rsc

I found a way to use Powershell command to workaround this issue in kubernetes code.
https://github.com/kubernetes/kubernetes/pull/92328

cc @alexbrainman @bcmills
@gopherbot add OS-Windows

@ericwj can you look at fixing #39785 in your fix for this?
Also, if you're interested in fixing other GOOS=windows issues, there's a handful I could point you to :-)

Man I am so annoyed about this for years now that I learned Go just to fix this - I've written about 100 lines total now...

Consider weighing in on #40104 to see issues like this finally be resolved once and forever.

@jingxu97
I tried reproducing your problem on current tip (it will be part of go1.15 soon), and I cannot. In particular, I tried an existing Go library test, that does exactly what you are describing in this issue.
c:\>go version
go version devel +3a4322627e Fri Jul 10 11:32:36 2020 +0000 windows/amd64

c:\>go test -v -run=NTName path/filepath
=== RUN   TestNTNamespaceSymlink
--- PASS: TestNTNamespaceSymlink (0.06s)
PASS
ok      path/filepath   0.087s

c:\>

Can you, please, try to run this test yourself (you will have to be the Administrator) and tell us, if it fails or passes? If you think your situation is different, then, please, provide complete reproduction steps, and not just vague description.
Thank you.
Alex

This is the code Alex refers to: https://github.com/golang/go/blob/master/src/path/filepath/path_windows_test.go#L538
@alexbrainman according to the kubernetes issue linked above, there's no drive letter assigned to this volume.

@alexbrainman according to the kubernetes issue linked above, there's no drive letter assigned to this volume.

How do I reproduce this? The issue above asks me to use mklink command. So I did.
Alex

I believe you create a VHDX and don't assign it a drive letter. Then mklink /D with that volume as target.

Yes, @networkimprov is correct. This is a very common configuration on Windows that go should not pre-empt with bugs.
The PowerShell:
$vhd = New-VHD -Path Test.vhdx -SizeBytes 3MB -Fixed
$vhd | Mount-VHD
$vhd = Get-VHD Test.vhdx # refresh IsAttached etc
$vhd | Get-Disk | Initialize-Disk -PartitionStyle GPT
$part = $vhd | Get-Disk | New-Partition -UseMaximumSize -AssignDriveLetter:$false
$vol = $part | Format-Volume -FileSystem NTFS

cmd /c mklink /j Test $vol.Path
[System.IO.File]::WriteAllText("$PWD\Test\Dockerfile", "FROM scratch")
docker build Test

# Cleanup
Dismount-Vhd Test.vhdx
rmdir Test
del Test.vhdx

The output from docker is this time:
unable to prepare context: unable to evaluate symlinks in context path: CreateFile C:\tools\Volume{9d7e0c41-a29a-4cea-b0f8-f16693425a69}: The system cannot find the file specified.

This fails, I believe about exactly the same, configuring Storage Spaces and wanting to put docker on a VirtualDisk as opposed to a Hyper-V VHD. So now docker doesn't run on any kind of serious Windows Server farm.

Oh hell, I don't know if my go is actually patched with #40095... Your output will be different in that case.

Sorry @jingxu97, I didn't get any notifications from this somehow.
Anyway yes, the test passes, but obviously they are very incomplete:
go test -v -run=NTName path/filepath
=== RUN   TestNTNamespaceSymlink
--- PASS: TestNTNamespaceSymlink (0.10s)
PASS
ok      path/filepath   0.396s

I don't know if my go is actually patched with #40095...

It is and I tested my hello.go to correctly resolve symlinks, but doing that today I get in an endless loop in walkSymlinks which after 255 tries will still claim too many links in symlink.go#L99.
For one it by now has path: "C:\Docker\Mount\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\...+213 more" on the stack where it started with C:\Docker\Mount\Subdir.
So this fix is partial. Obviously I also did not add any tests, changing only one file.

This is the culprit
PS C:\Docker> dir

    Directory: C:\Docker

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           6/13/2020  1:50 AM                DockerWeb
l----           6/13/2020  4:20 AM                Mount -> Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\
d----           6/13/2020  1:27 AM                Other
-a---           7/11/2020  6:12 PM       37748736 Disk.vhdx

PS C:\Docker> (Get-VHD .\Disk.vhdx) | Get-Disk | Get-Partition | select -ExpandProperty AccessPaths
C:\Docker\Mount\
\\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\

I'm not sure how C:\Docker\Mount became an access path for the volume, but this means that in this setup the #40095 fix isn't applicable, because GetFinalPathNameByHandle(... VOLUME_NAME_DOS) will return "\\?\C:\Docker\Mount\" with a handle created from C:\Docker\Mount and hence GetFinalPathNameByHandle(... VOLUME_NAME_GUID) will not be tried.
After running
(Get-VHD .\\Disk.vhdx) | Get-Disk | Get-Partition | Remove-PartitionAccessPath -AccessPath C:\Docker\Mount\

I run in yet another problem, although I like the first part of it - symlink.go#L88 will say C:\Docker\Mount is not a symlink, followed by symlink.go#L83 Lstat yielding (null, 2) and the result hence becomes:
API server listening at: <IP_ADDRESS>:47251
Abs [<nil>] C:\Docker\Mount\Subdir => C:\Docker\Mount\Subdir
Dir [nul] C:\Docker\Mount\Subdir => C:\Docker\Mount
EvalSymlinks [CreateFile C:\Docker\Mount\Subdir: The system cannot find the file specified.] C:\Docker\Mount\Subdir => (empty string)
```

And still my fix isn't applicable. It is such a mess.

Removing the C:\Docker\Mount\ access path in fact made Mount a regular directory, so at least LStat was correct about that, but shouldn't EvalSymLinks in that case just return the original path?
Running
PS> rmdir Mount
PS> cmd /c mklink /j Mount \\?\Volume...

my fix becomes applicable.
API server listening at: <IP_ADDRESS>:12289
Abs [<nil>] C:\Docker\Mount\Subdir => C:\Docker\Mount\Subdir
Abs [nul] C:\Docker\Mount\Subdir => C:\Docker\Mount
EvalSymlinks [<nil>] C:\Docker\Mount\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir
IsAbs [<nil>] C:\Docker\Mount\Subdir => true
VolumeName [<nil>] C:\Docker\Mount\Subdir => C:
ResolveAndValidateContextPath [<nil>] C:\Docker\Mount\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\
ResolveAndValidateContextPath [<nil>] C:\Docker\Mount\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir
GetContextFromLocalDir [<nil>] C:\Docker\Mount\Subdir => (\\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir, Dockerfile)
Process exiting with code: 0

Finally, to reproduce against the original issue text above:
PS C:\Docker> cmd /c mklink /D DirectorySymbolicLink "\\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\"
symbolic link created for DirectorySymbolicLink <<===>> \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\
PS C:\Docker> hello
API server listening at: <IP_ADDRESS>:16921
Abs [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => C:\Docker\DirectorySymbolicLink\Subdir
Abs [nul] C:\Docker\DirectorySymbolicLink\Subdir => C:\Docker\DirectorySymbolicLink
EvalSymlinks [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir
IsAbs [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => true
VolumeName [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => C:
ResolveAndValidateContextPath [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\
ResolveAndValidateContextPath [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir
GetContextFromLocalDir [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => (\\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir, Dockerfile)
Process exiting with code: 0

So this fix is partial.

This fix is complete for the situation that is described.

@ericwj
You seems to be talking to yourself. I am still confused where are the repro steps. Could you point me to the steps, please?
Thank you.
Alex

Use this powershell script to make a vhdx and symlink, then pass the symlink to filepath.EvalSymlinks.
$vhd = New-VHD -Path Test.vhdx -SizeBytes 3MB -Fixed
$vhd | Mount-VHD
$vhd = Get-VHD Test.vhdx # refresh IsAttached etc
$vhd | Get-Disk | Initialize-Disk -PartitionStyle GPT
$part = $vhd | Get-Disk | New-Partition -UseMaximumSize -AssignDriveLetter:$false
$vol = $part | Format-Volume -FileSystem NTFS

cmd /c mklink /j Test $vol.Path

package main

import ("path/filepath"; "fmt")

func main() {
   p, err := filepath.EvalSymlinks("Test")
   if err != nil { panic(err) }
   fmt.Println(p)
}

Following up on the explanation in #40180, it might be worth considering not to take #40095 and leave EvalSymLinks completely unable to identify any volume on Windows forever. Having it translate UNC mappings and nothing else would then imho not break anyone, short of displeasing some people's feeling for aesthetic if they care about which DOS device name they see in the debugger.

In that case, should your patch adjust the error message given for a \\?\Volume... target?

If that is not considered a breaking change, that would be very useful.

Are you saying that I should do that? But how? I could leave it return a misleading error number 3 (path not found) and change only the text - but only if absolutely required, or change both the text and the type of error to make it at least not look like it is Windows' fault. Which error would that then be and how do I return it?

I think changing the error is an extension of #40180. If there's agreement that EvalSymlinks is wrong for Windows, then purposely failing for certain symlink targets is beneficial.

use this powershell script to make a vhdx and symlink, then pass the symlink to filepath.EvalSymlinks.

Thank you @networkimprov for the steps. Unfortunately,  I am not very good with PowerShell. First line of your instruction fails with:
PS C:\Users\Alex\dev\src\issue\go\39786> $vhd = New-VHD -Path Test.vhdx -SizeBytes 3MB -Fixed
New-VHD : The Hyper-V Management Tools could not access an expected WMI class on computer 'HoHoHo'. This may indicate
that the Hyper-V Platform is not installed on the computer or that the version of the Hyper-V Platform is incompatible
with these management tools.
At line:1 char:8
+ $vhd = New-VHD -Path Test.vhdx -SizeBytes 3MB -Fixed
+        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-VHD], VirtualizationException
    + FullyQualifiedErrorId : Unspecified,Microsoft.Vhd.PowerShell.Cmdlets.NewVhd

PS C:\Users\Alex\dev\src\issue\go\39786>

What did I do wrong?
Thank you.
Alex

Nothing is wrong Alex, that is expected on clean machines. The Hyper-V tools will install both the virtual disk provider service and the PowerShell modules that are needed.
So, install Hyper-V like this:
PS> Enable-WindowsOptionalFeature Hyper-V # on desktops
PS> Install-WindowsFeature Hyper-V # on servers

So, install Hyper-V like this:

Unfortunately I cannot install Hyper-V.  I tried installing it once in the past, and it refuses to install because I have https://www.virtualbox.org installed.
Is there a way to reproduce this without installing Hyper-V ?
Thank you.
Alex

Alex, can you install Hyper-V on a Windows VM builder instance?

Alex, can you install Hyper-V on a Windows VM builder instance?

I wouldn't now how to do that. Do you have another way to reproduce this?
Alex

Eric may know another way to demonstrate the error.
For help with trying this on a Windows builder instance, cc @dmitshur @jayconrod

Install Hyper-V in a machine running in VirtualBox. That can work if virtualbox can be made to passthrough the virtualization. I think it is unfortunate the hypervisor is needed - maybe it isn't but I have no idea how to get the vhd provider on the system any other way.
So if virtualbox can do something similar to
$vm | Set-VMProcessor -ExposeVirtualizationExtensions

Azure supports it. It is usually called nested virtualization. Also on the virtualbox site.

I find that the feature names differ for desktop and server. Also on server, the command I gave previously isn't enough, since the PowerShell functionality is not actually a subcomponent of Hyper-V itself.
For Desktops:
PS> Get-WindowsOptionalFeature -Online -FeatureName *Hyper-V* | ft
PS> Enable-WindowsOptionalFeature Microsoft-Hyper-V-All -Online # will install Microsoft-Hyper-V-Hypervisor

It could be possible that New-VHD starts working with just Microsoft-Hyper-V-Management-PowerShell and Microsoft-Hyper-V-Services such that the hypervisor isn't needed, but I haven't tried de-installing anything:
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Management-PowerShell, Microsoft-Hyper-V-Services

On server, the feature installation is unfortunately less detailed such that the provider services and Hyper-V cannot be selected separately, as mentioned.
PS> Install-WindowsFeature Hyper-V -Restart
PS> Install-WindowsFeature Hyper-V-PowerShell
PS> Get-WindowsFeature -Name *Hyper-V*

Display Name                                            Name                       Install State
------------                                            ----                       -------------
[X] Hyper-V                                             Hyper-V                        Installed
        [X] Hyper-V Management Tools                    RSAT-Hyper-V-Tools             Installed
            [X] Hyper-V Module for Windows PowerShell   Hyper-V-PowerShell             Installed

Install Hyper-V in a machine running in VirtualBox. That can work if virtualbox can be made to passthrough the virtualization.

I don't have operating system that runs inside of VirtualBox that supports Hyper-V. I only have Linux and Windows 7 VMs.
Alex

Alex, I think we can make a CL that runs on a Windows builder instance.
We just need to know how a Go test program (run by the builder) can call a local script. Where in the tree should the test program & script live, and what path should exec.Command() use for the script?
The test program looks something like:
// +build windows

package main

import (
   "os/exec"
   "fmt"
   "os"
   "testing"
)

const kPath = "dir"
var kArgs = []string{"."} // args in separate strings
   // re quoting, see https://golang.org/pkg/os/exec/#Command

func TestScript(i *testing.T) {
   aCmd := exec.Command(kPath, kArgs...)
   aCmd.Stdin, aCmd.Stdout, aCmd.Stderr = os.Stdin, os.Stdout, os.Stderr
   err := aCmd.Run()
   if err != nil {
      fmt.Fprintf(os.Stderr, "%v\n", err)
      i.Fail()
   }
}

To get Windows is really easy.

I suspect you are not a Windows Insider, but it is really easy to register - assuming you already have a Microsoft Account
Then you can download an ISO file of the Release Preview Channel from Windows Insider Preview Downloads

There is also always the option to download an evaluation version, for example Windows 10 Enterprise.
The evaluation version will be good for half a year. The insider previews also come with a version-specific expiration date, but I believe those are valid a little longer.

Is this still true? The post is from 2018.
How to enable nested virtualization on Google Cloud

Nested virtualization does not currently support Windows instances.

I'm not sure if that would be a blocker, since I have no clue whether that also means VM's inside such a machine can run Windows. That would at least be very unhandy, having to run some *nix with a Windows VM inside it with nested virtualization enabled twice just to create some VHD's.
But it would be very easy to just run go tests on Microsoft Azure.

How to enable nested virtualization in an Azure VM.

The script this program calls can be run with pwsh -File but

It is easiest to have PowerShell 7 already installed or it would have to invoke a PowerShell 5 script just to download and install that - I get all sorts of problems running my script on PowerShell 5 which I don't feel like fixing and the version of .NET it runs on has blocking problems which might never be fixed over compatibility concerns.
It should probably not call Install-WindowsFeature. To be able to depend on Hyper-V PowerShell being available (or crash over it), the machine should be setup with Hyper-V before testing go.

That said any number of disks attached to a regular VM would be sufficient to demonstrate the issues - as long as the machine uses GPT any and all partitions will have partition access paths and can be configured to use any drive letters or not, access paths and links and junctions.
The repro steps include this line
$part = $vhd | Get-Disk | New-Partition -UseMaximumSize -AssignDriveLetter:$false

But all that is needed is to initialize the $part variable with a partition object that is OK to be modified, e.g.:
$part = Get-Disk -Number 0 | Get-Partition | where Type -eq Basic | sort Size | select -First 1
$part = Get-Partition -DiskNumber 0 -PartitionNumber 3
$part = Get-Partition -DriveLetter C

Then these commands allow reproducing the issue against any partition on the machine:
$part.AccessPaths # list
$part | Add-PartitionAccessPath -AccessPath foofullpath
$part | Remove-PartitionAccessPath -AccessPath foofullpath
# directory junction
mklink /j foo ($part | Get-Volume).Path
rmdir foo
# hardlink
mkdir /d foo ($part | Get-Volume).Path
rmdir foo

although beware that the issues do not reproduce if the partition already has a drive letter or any other DOS device path with a drive letter.
Now to get a partition that is OK to be modified (or several), run:
$boot = Get-Partition -DriveLetter C
$size = $part.Size - 128MB
$boot | Resize-Partition -Size $size
$disk = $boot | Get-Disk
for ($i = 0; $i -lt 4; $i++) {
    $part = $disk | New-Partition -Size 8MB
    $part | Format-Volume -Label "TestPart$i" -AssignDriveLetter:$false
}

for example. I am not sure about minimum sizes and hence how much space is needed. But with these examples you can make it any way you like, on any disk you like.

Eric & Alex, let's focus on creating a VHDX on a Go builder for Windows, as any Go project member could run a builder.
Creating a CL with the repro script should be easy, we just need the info I listed above.

We just need to know how a Go test program (run by the builder) can call a local script. Where in the tree should the test program & script live, and what path should exec.Command() use for the script?

Normally I create temp directory, and put whatever I need there. And delete after test is finished.

I suspect you are not a Windows Insider, but it is really easy to register - assuming you already have a Microsoft Account

I don't have Microsoft Account. I don't want to create an account just to run this repo. If nothing else, I can create Windows VM on Google Cloud or something. But that would take me long time, because I need to install all appropriate tools. And I don't install Windows every day, so, I expect, it will take me awhile to figure out.

Is this still true?

I have no idea.

That said any number of disks attached to a regular VM would be sufficient to demonstrate the issues - as long as the machine uses GPT any and all partitions will have partition access paths and can be configured to use any drive letters or not, access paths and links and junctions.

I don't know what "machine uses GPT" means. If you give me steps to repro, I will try them. Thank you.
Alex

I don't know what "machine uses GPT" means.

GPT as opposed to MBR.
See also Hard Drives and Partitions | Microsoft Docs.
On Windows it is just massively much easier to do things like resizing partitions and add a few than on *nix. All the steps you need have been described here and here. You can also use pointy clicky Disk Management or command line tools that won't have you read massive amounts of man pages and then still have you try and hopefullly not die and hack the kernel: diskpart.

I have exactly the same issue and this problem is anoying for a better automation on Windows.
To decouple infra/software provisioning I try to apply a "unix style" mounts in C:. The idea is the use the feature of NTFS AccessPath (which are basicaly a simple Junction). Thanks to that the software provisioning automation can use more predictible path wich will work all the time, nevermind if the infra provisioning decided or not to add a dedicated volume in a subpath.
On windows Junctions and SymLinks are not the same thing and golang should not threat them as same !
A Symlink can be done on a file or folder and point to another filer or folder or even Junction. But it's virtual.
A Junction is really a "mount point", and it's what happen when you use the powershell command Add-PartitionAccessPath mantioned above.
The function EvalSymlink should not try to extrapolate a Junction, it's like if in unix you try to convert "/mnt/myvolume" to  "/dev/sda1"
A windows junction object has a Type "Junction", but the target should be an identifier to a VolumeID :
Exemple :
PS C:\var> Get-Item .\lib\ | Format-List

    Directory: C:\var

Name           : lib
CreationTime   : 7/12/2022 2:33:56 PM
LastWriteTime  : 7/12/2022 2:33:56 PM
LastAccessTime : 7/12/2022 2:33:56 PM
Mode           : d----l
LinkType       : Junction
Target         : {Volume{1566c3e6-85da-48d5-aba4-98a3a99f32b0}\}

PS C:\var> Get-Volume | Format-List

ObjectId             : {1}\\xxxxxxxx\root/Microsoft/Windows/Storage/Providers_v2\WSP_Volume.ObjectId="{ce6930b6-
                       fed1-11ec-838a-806e6f6e6963}:VO:\\?\Volume{1566c3e6-85da-48d5-aba4-98a3a99f32b0}\"
PassThroughClass     :
PassThroughIds       :
PassThroughNamespace :
PassThroughServer    :
UniqueId             : \\?\Volume{1566c3e6-85da-48d5-aba4-98a3a99f32b0}\
AllocationUnitSize   : 4096
DedupMode            : NotAvailable
DriveLetter          :
DriveType            : Fixed
FileSystem           : NTFS
FileSystemLabel      : applications
FileSystemType       : NTFS
HealthStatus         : Healthy
OperationalStatus    : OK
Path                 : \\?\Volume{1566c3e6-85da-48d5-aba4-98a3a99f32b0}\
Size                 :<PHONE_NUMBER>2
SizeRemaining        :<PHONE_NUMBER>2
PSComputerName       :

As mentioned above the volume ID is probably related to GPT format, but the overall idea should be the same with MBR.
The target should just contain another kind of identifier.
For now as workaround I added a letter to my volume to let docker service start but it also mess up the automations because in the case of docker the "configured root path" (in deamon.json) is different than the output in docker info --format "{{ .DockerRootDir }}" which is the result after EvalSymlinks, pointing on D:

Hello,
Any chance to move forward on this problem ?
It's really a conceptual bug in golang for windows and it cause issues in multiple golang softwares.
As exemples I have :

docker if the data files are configured inside a junction (eg: C:/var/lib is a junction and docker configured to strore data in C:/var/lib/docker/)

time="2023-03-29T14:32:23.347218600Z" level=info msg="Starting up"
Unable to get the full path to root (C:/var/lib/docker/): failed to canonicalise path for C:/var/lib/docker/: EvalSymlinks: too many links

opentelemetry collector when parsing log files in a junction (eg: configured to read C:/var/log/iis/*/*.log where C:/var/log is a junction. Other test case, read C:/var/lib/docker-sites/*/Logs/IIS/W3SVC1/*.log where C:/var/lib/docker-sites/*/Logs is a symlink to a folder in a junction => C:\var\log\docker-sites\xxx, C:/var/log is the junction)

2023-03-29T14:54:53.977Z        error   fileconsumer/reader_factory.go:117      resolve attributes: %!w(*errors.errorString=&{EvalSymlinks: too many links})    {"kind": "receiver", "name": "filelog/iis", "pipeline": "logs", "component": "fileconsumer"}

The solutions seems around :
https://github.com/golang/go/blob/master/src/path/filepath/symlink_windows.go evalSymlinks() function
and
https://github.com/golang/go/blob/e09bbaec69a8ff960110e13eabb3bef5331ecb0c/src/path/filepath/path_windows.go#L61
and
https://github.com/golang/go/blob/e09bbaec69a8ff960110e13eabb3bef5331ecb0c/src/cmd/vendor/golang.org/x/sys/windows/syscall_windows.go#L1457
But the logic in this functions seems kind of blur to me.

Or Docker should stop using EvalSymlinks. And crappy workarounds like isUNC. And support Windows properly in general. https://github.com/search?q=repo%3Adocker%2Fcli+evalsymlinks&type=code
This PR has been open for ages for one https://github.com/docker/cli/pull/2649