SafeInt

a integer wrapper that has a dedicated NaN value and utilises
core.checkedint to check for overloads
additionally, implicit value propagation safe integer comparison function are
added

Safe has a very specific meaning in D, so it's better to stay away from that word for anything not involving memory safety. CheckedInt works just as well IMO.

There are holes in the handling of NaN values, especially when mixing different types. For example:
SafeInt!short small;
writeln(small.isNaN); // prints true
SafeInt!long big = small;
writeln(big.isNaN); // prints false

Also, isn't it unnecessary for safeAssign to range check when the types of this and v are the same? There are several other combinations which don't require range checks either, although I could imagine some of these being optimized out automatically by a good compiler.
I don't know if the design of writeln makes this easy or not, but catching bugs in code using SafeInt would be much easier if writeln(SafeInt!short.nan) would print NaN instead of -32768.
Lastly, I cannot think offhand how no bitwise operations are implemented improves safety, except perhaps in the case of the shift operators >> and <<. Over/underflow is not possible with ~, &, |, or ^, is it? This restriction seems like an unnecessary inconvenience, but perhaps I'm missing something.

@tsbockman
overloading toString might be a good idea
I didn't include bit operations, as I consider them unsafe by design in a safe int type.

docu:
http://burner.github.io/phobos/phobos-prerelease/std_experimental_safeint.html

NaN handling seems better now; I have not yet found a way to break it with numeric operations. I might test more thoroughly later.
The following code fails to compile with a template error:
SafeInt!int safe;
int raw = 0;
safe = raw;

Also, you seem to have duplicated some code between opAssign and safeAssign. Shouldn't the body of opAssign just look like this now?
this.safeAssign(vIn);
return this;

TheNaN check and call to getValue seem redundant based on my reading of safeAssign.
As far as safeAssign itself goes, the current implementation appears correct.
However, I suggested some performance optimisations based on skipping unnecessary range checks for certain type combinations in my previous comment. Considering the frequency with which that function is likely to be called in tight loops, optimizing it seems worthwhile.
I would be willing to try and write said optimized version myself, if you agree that a runtime performance boost is worth some extra code complexity in the implementation of safeAssign.

I am no longer getting template errors.
The new safeAssign is better, but still contains unnecessary branching in some cases, including what will likely be the most common: where is(T == SafeIntType!V).
Here is what I believe to be an optimal safeAssign implementation, with no runtime tests except those actually needed in each case:
static if (is(T == SafeIntType!V))
{
	// If this and v have the exact same value type, no runtime checks are required at all.
	this.value = getValue(v);
}
else
{
	static if (isSafeInt!V && T.sizeof >= (SafeIntType!V).sizeof)
	{
		if (v.isNaN)
		{
			this.value = this.nan;
			return;
		}
	}

	auto vVal = getValue(v);
	static if (isSigned!(typeof(vVal))
		&& (isUnsigned!T || T.sizeof < vVal.sizeof))
	{
		if (less(vVal, this.min) || greater(vVal, this.max))
		{
			this.value = this.nan;
			return;
		}
	}
	else static if (isUnsigned!(typeof(vVal))
		&& ((isSigned!T && T.sizeof <= vVal.sizeof)
		|| (T.sizeof < vVal.sizeof)))
	{
		if (greater(vVal, this.max))
		{
			this.value = this.nan;
			return;
		}
	}
	this.value = cast(T) vVal;
}

A little bikeshedding: The encoding of the NaN state is an implementation detail, which value exposes as part of the public interface. I think value should be renamed to something scarier sounding like raw or bits.
Either way, the documentations should include a warning to the user that it may not always be correct to naively interpret that variable as an ordinary T.

I did some simple benchmarking with DMD2, in case anyone else is interested.
I wrote an algorithm that computes the first 40,000 primes and their (integer floor) square roots. The primes are found using trial division, and the roots with a binary search guess-and-check based method. This exercises operators =, ==, <, >, <=, ++, +, -, /, %, and *.
The algorithm is templated such that all math and array indexing is done using type parameter T. It does not use any heap memory.
On average, SafeInt is 7-10x slower:
int: 38 ms, 915 μs, and 3 hnsecs
SafeInt!int: 381 ms, 677 μs, and 6 hnsecs
long: 66 ms and 536 μs
SafeInt!long: 501 ms, 834 μs, and 9 hnsecs

Commenting out opBinary in SafeInt to allow cheap wrapping arithmetic yields:
int: 38 ms, 983 μs, and 8 hnsecs
SafeInt!int: 95 ms, 590 μs, and 7 hnsecs
long: 66 ms, 576 μs, and 1 hnsec
SafeInt!long: 124 ms, 120 μs, and 4 hnsecs

So opBinary is very slow, but everything else is reasonable. I tried playing with the code some, but wasn't able to meaningfully improve things. Perhaps GDC or LDC will be able to close the gap some once they upgrade to 2.067.1.
Regardless, it seems fast enough to be useful.

thank you for the effort. I will have a look at safeAssign.
could you post your benchmark, because of  https://github.com/D-Programming-Language/phobos/pull/2995

After some fiddling, I finally got SafeInt to compile under GDC (probably version 2.065? I haven't been able to figure out which version of the D front-end is shipping with Linux Mint right now).
Performance of GDC vs DMD (relative time required; lower is better):
       Type          |  GDC   |  DMD   | DMD / GDC
---------------------|--------|--------|-----------
              int    |   100% |   123% |    1.2x
   SafeInt!(  int  ) |   171% | 1,280% |    7.5x
CheckedInt!(  int  ) |   117% | 1,390% |   11.9x

             uint    |   100% |   136% |    1.4x
   SafeInt!( uint  ) |   161% | 1,080% |    6.7x
CheckedInt!( uint  ) |   145% | 1,420% |    9.8x

              long   |   100% |   111% |    1.1x
   SafeInt!(  long ) |   285% |   837% |    2.9x
CheckedInt!(  long ) |   122% |   743% |    6.1x

             ulong   |   100% |   119% |    1.2x
   SafeInt!( ulong ) |   290% |   826% |    2.8x
CheckedInt!( ulong ) |   132% |   789% |    6.0x

While DMD is almost as fast as GDC with the built-in integer types, there is no contest when using user-defined types: GDC is anywhere from 3 to 12 times faster. I suspect that DMD is not properly inlining user-defined operations.
Regardless of why DMD is so slow, the GDC numbers are quite encouraging: SafeInt is only 1.6 to 2.9 times slower than the built-in types, which is much more palatable than the ~10x hit performance takes under DMD.
Nevertheless, I got to thinking about the NaN encoding scheme used by SafeInt, and realized that while it certainly makes efficient use of memory, it has the detrimental effect of serializing arithmetic which could otherwise have been issued in parallel on superscalar CPU architectures.
So, I decided to take a crack at rewriting SafeInt as CheckedInt based on a different storage format:
struct CheckedInt(T) {
    bool isNaN;
    T value;
}

This format mostly prevents isNaN propagation from serializing with value calculations (I think). Even if I'm wrong about why, the benchmark still makes it unambiguously clear that it is quite a lot faster: with 64-bit integers on GDC, CheckedInt is over 2x faster than SafeInt. In fact, GDC's CheckedInt!int run was actually a tiny bit faster than DMD's int run!
The disadvantage is that the separate isNaN field requires a minimum of one extra byte of storage. This would hardly be worth mentioning, except that because of value's alignment requirements, in practice CheckedInt uses exactly twice as much memory as SafeInt.
The trick is that only variables being actively manipulated benefit from CheckedInt's extra speed, while only bulk data storage, such as arrays, benefit significantly from SafeInt's compact size. With a clever API, I think it should be possible to enable mixing the two different formats almost seamlessly, such that local variables and temporary values automatically use the fast CheckedInt style, while data structures use the compact SafeInt style. (Obviously a unified API would use a better naming scheme than that.)
Done right, I think this could become a popular feature of D; I know that I, personally, would seriously consider using SafeInt/CheckedInt by default, just to catch bugs, if the performance was less than 50% slower than the built-in unchecked types. Even people who aren't willing to give up that much speed in the release build could still use checked arithmetic in debug builds, like people do with array bounds checking now.
I will probably set up a separate repository later this week to refine this idea. (When I do so, I will make sure to include the benchmark code you wanted.)
I think it would be wise to iterate on the API a bit, and get some feedback from the community before setting it in stone by merging with Phobos.

@burner I put my CheckedInt code online at https://github.com/tsbockman/CheckedInt. It's in fairly good shape now, except that I haven't added your packed data storage format to it yet. (I plan to do so soon, though.)
The benchmark code is in main.d. I had an earlier version which tested your SafeInt as well, but I can't get your latest commit to compile at the moment.
Have a look and let me know what you think.
If you still think your version is a better base to move forward with, feel free to borrow anything you like from mine. Also, while CheckedInt was a nearly complete rewrite, I took much inspiration from SafeInt, so just tell me if you want to be credited somewhere.

After some more fiddling, I now have the latest SafeInt compiling again. Here are the latest benchmark results:
         Type          |  GDC   |  DMD   | DMD / GDC
-----------------------|--------|--------|-----------
                int    |   100% |   118% |    1.2x
SafeIntSmall!(  int  ) |   156% | 1,280% |    8.2x
 SafeIntFast!(  int  ) |   541% | 2,370% |    4.4x
  CheckedInt!(  int  ) |   115% | 1,380% |   12.0x

               uint    |   100% |   139% |    1.4x
SafeIntSmall!( uint  ) |   193% | 1,120% |    5.8x
 SafeIntFast!( uint  ) |   267% | 2,200% |    8.2x
  CheckedInt!( uint  ) |   170% | 1,450% |    8.5x

                long   |   100% |   111% |    1.1x
SafeIntSmall!(  long ) |   288% |   844% |    2.9x
 SafeIntFast!(  long ) |   268% | 1,190% |    4.5x
  CheckedInt!(  long ) |   115% |   738% |    6.4x

               ulong   |   100% |   106% |    1.1x
SafeIntSmall!( ulong ) |   286% |   842% |    2.9x
 SafeIntFast!( ulong ) |   276% | 1,250% |    4.5x
  CheckedInt!( ulong ) |   141% |   789% |    5.6x

It looks like SafeIntExplicit isn't achieving the kind of performance boost that my CheckedInt did relative to the original Inline format.
I believe this is at least partly because your algorithms are still written in Inline style, which serializes the actual arithmetic with the NaN checks, whereas the CheckedInt algorithms are written in a different style which takes advantage of the Explicit format to allow arithmetic to mostly run in parallel with the NaN propagation.
I don't think it's necessary to have two separate versions of every algorithm - but if the Inline and Explicit formats are to share code, they must use the faster Explicit style for calculations. This should not make the Inline format any slower, and it will make the Explicit format much faster.

I have now implemented the dual storage formats for CheckedInt. I also created a separate branch vs-safeint in my repository with SafeInt included in the benchmark.
After some more fiddling, I now have the latest SafeInt compiling again. I have also updated GDC to 2.066.1 and added LDC, as well.
               Type            |    GDC     |    GDC     |    LDC     |    DMD
                               | (inlineme) |   (core)   |   (core)   |   (core)
-------------------------------|------------|------------|------------|------------
                int            |     113%   |    126%    |     100%   |     147%
SafeIntSmall!(  int  )         |     203%   |    563%    |   1,520%   |   1,470%
  CheckedInt!(  int , "slim" ) |     165%   |    331%    |   1,330%   |   3,380%
 SafeIntFast!(  int  )         |     585%   |    631%    |   1,660%   |   2,780%
  CheckedInt!(  int , "fast" ) |     159%   |    301%    |   1,160%   |   2,740%
  CheckedInt!(  int  )         |     146%   |    300%    |   1,140%   |   2,720%

               uint            |     100%   |    100%    |     136%   |     121%
SafeIntSmall!( uint  )         |     144%   |    252%    |   1,008%   |   1,060%
  CheckedInt!( uint , "slim" ) |     159%   |    253%    |   1,100%   |   2,660%
 SafeIntFast!( uint  )         |     167%   |    247%    |   1,140%   |   2,050%
  CheckedInt!( uint , "fast" ) |     138%   |    230%    |     915%   |   2,210%
  CheckedInt!( uint  )         |     134%   |    233%    |     920%   |   2,190%

                long           |     112%   |    112%    |     100%   |     135%
SafeIntSmall!(  long )         |     294%   |    361%    |     888%   |     902%
  CheckedInt!(  long, "slim" ) |     293%   |    367%    |     880%   |   1,500%
 SafeIntFast!(  long )         |     301%   |    365%    |     901%   |   1,270%
  CheckedInt!(  long, "fast" ) |     124%   |    364%    |     782%   |   1,220%
  CheckedInt!(  long )         |     125%   |    361%    |     773%   |   1,220%

               ulong           |     103%   |    100%    |     131%   |     102%
SafeIntSmall!( ulong )         |     264%   |    312%    |     851%   |     813%
  CheckedInt!( ulong, "slim" ) |     293%   |    324%    |     845%   |   1,480%
 SafeIntFast!( ulong )         |     282%   |    313%    |     865%   |   1,200%
  CheckedInt!( ulong, "fast" ) |     134%   |    322%    |     758%   |   1,230%
  CheckedInt!( ulong )         |     124%   |    322%    |     761%   |   1,230%

The separate GDC (inlineme) and GDC (core) categories reflect a discovery I made somewhat by accident: importing the actual core.checkedint from D runtime yields noticeably slower code compared to copy-pasting the source of core.checkedint into the project under another name (inlineme.checkedint, in this case).

thank you for all the work.
I believe your mixin use helps a lot with performance. I think the selected function is not getting inlined.
Also the core.checkedint functions are not inline (at least that is was valgrind tells me with gdc). well unless you put them in the file you did ;-)
I will do some tinkering over the week
anyway compiler are strange
p.s. I think we need this https://github.com/D-Programming-Language/phobos/pull/2995 in phobos

Aren't the core.checkedint functions compiler-recognized intrinsics that should always be inlined? IT doesn't make sense that they wouldn't be.

it doesn't make sense then https://github.com/D-Programming-Language/druntime/blob/master/src/core/checkedint.d#L55

I don't claim to know what DMD actually does, that's just what I thought it did.

also gdc doesn't inline the function ....

@MetaLang From the documentation in core.checkedint, lines 18-20:
 * While the generic versions of these functions are computationally expensive
 * relative to the cost of the operation itself, compiler implementations are free
 * to recognize them and generate equivalent and faster code.

So, they're supposed to be intrinsics, but they're not required to be recognized as such by compilers.
Seeing as core.checkedint has only been around since D runtime 2.066, I suspect that the big three compilers simply haven't gotten around to implementing the intrinsics yet.
I would look into adding them myself, but I have no experience with x86(-64) assembly, nor with hacking any of the three compilers.

I would look into adding them myself, but I have no experience with x86(-64) assembly, nor with hacking any of the three compilers.

You don't need experience with assembly to hack on GDC or LDC.

@ibuclaw Taking a quick peek at the GDC code base... Is gcc/d/intrinsics.def all I would need to change to make core.checkedint intrinsics?

@tsbockman - for GDC to identify them as intrinsics, yes.  You still need to train it on what to do when it encounters the symbol.

I assume that core.checkedint will be mapped to these guys.  https://gcc.gnu.org/onlinedocs/gcc/Integer-Overflow-Builtins.html

@tsbockman - I've raised a PR so you don't have to.  https://github.com/D-Programming-GDC/GDC/pull/110

@ibuclaw Thanks. I figured you could do it way faster than me :-)
I've not even had time to look at this further today. Your (nearly) one-man quest to keep GDC 8x faster than DMD is appreciated.
I will try to find time to benchmark your PR some time in the next week or two, if I can figure out how to build GDC from source. It will also be interesting to see if @burner's force inline will allow DMD to close the gap at all...

Just added negs too.  Though I've just had a look at how you use core.checkedint here, and I don't think you'll get any benefits from my change, let me just check...

@ibuclaw I thought so to, and replaced it with a string mixin version. Same problem, no inlining.

Same problem, no inlining.

No, different problem, same exhibited behaviour.  ;-)

Agree with @MetaLang that "safe" has a specific meaning in D, i.e., memory-safety, and we should rename this type to something else to avoid misunderstandings, since this type doesn't have anything to do with memory safety, but more with handling numerical overflow/underflow and comparisons nicely between different integer sizes.

using mixins helped a lot, gdc:
            int   5755 msecs

SafeIntFast   5814 msecs
SafeIntSmall   6449 msecs
to find 200_000 prims and sum them up
ldc:
int  5741 msecs
SafeIntFast  5749 msecs
SafeIntSmall  6409 msecs
dmd:
int   5750 msecs
SafeIntFast  58188 msecs
SafeIntSmall 21425 msecs

The benchmark in SafeInt contains a bug which allows the compiler to potentially optimize out the prim!(SafeIntSmall!int)() call entirely: you store the result in i2, but print i3 again instead.
Amusingly, DMD is the only compile that actually caught this for me:
         int<PHONE_NUMBER>  4131 msecs
 SafeIntFast<PHONE_NUMBER> 42233 msecs
SafeIntSmall<PHONE_NUMBER>     0 msecs

So, DMD is faster at something. :-P
More generally, prim(T)() does not work type T hard enough to be a useful benchmark. A few problems I see:
1. By using size_t for the counter j in the first inner loop, you have ensured that about half the work there will be unchecked, regardless of the type of T.
2. Aside from comparisons (which my testing indicates are not a major bottleneck in either of our versions), the only checked operations being exercised are int adds() and int mods(). Of these, int adds() is among the cheapest checked operations, and your int mods() is actually not checked at all:
private auto modFunc(T)(T v1, T v2, ref bool overflow)
    if(isIntegral!T)
{
    return v1 % v2;
}

While I, personally, think that x % 0 should just return 0, DMD, LDC, and GDC all disagree:
enum n = 5 % 0; // Error: divide by 0

Even with a "divide by 0" check in place, int mods() is still as simple as a checked op can get. Compare that to long subs():
long subs(long x, long y, ref bool overflow)
{
    long r = cast(ulong)x - cast(ulong)y;
    if (x <  0 && y >= 0 && r >= 0 ||
        x >= 0 && y <  0 && r <  0 ||
        y == long.min)
        overflow = true;
    return r;
}

When I designed my simplePrimes() test, I was not trying to make an optimal prime finder, or even an optimal trial division implementation. (There are much better algorithms, even if simplicity is desired.) I included a call to floorSqrt() on every iteration not because it was the best way to find primes, but rather in order to exercise a wider range of checked operations (=, ==, <, >, <=, ++, +, -, /, %, and *) in a somewhat even mixture.
(Still, I should probably go find some standard integer benchmarks to convert, to verify that my ad-hock test is not itself somehow misleading.)
3. Since you never check isNaN anywhere, this could potentially lead to a smart compiler optimizing out the overflow checks. (I don't think this one is having a big effect at the moment, though.)
4. Overall, prim() is just too short and simple.
I've found that, in general, optimizing compilers often produce unrealistically good results with very simple code. There should be at least a small performance difference between SafeInt!int (or CheckedInt!int) versus int, because these checked types actually have more work to do. I kept adding complexity to my simplePrimes() test until a real difference appeared - it didn't take very much.
Running my own, slightly more complex benchmark, I see no major change in the performance of your latest SafeInt version:
               Type            |   GDC x64  |   GDC x64  |   LDC x64  |   DMD x64
                               | (inlineme) |   (core)   |   (core)   |   (core)
-------------------------------|------------|------------|------------|------------
                int            |     127%   |    126%    |     100%   |     147%
SafeIntSmall!(  int  )         |     185%   |    292%    |   1,230%   |     943%
  CheckedInt!(  int , "slim" ) |     165%   |    330%    |   1,400%   |   3,360%
 SafeIntFast!(  int  )         |     223%   |    324%    |   1,380%   |   2,310%
  CheckedInt!(  int , "fast" ) |     148%   |    294%    |   1,250%   |   2,720%
  CheckedInt!(  int  )         |     153%   |    294%    |   1,250%   |   2,705%

               uint            |     102%   |    100%    |     136%   |     119%
SafeIntSmall!( uint  )         |     143%   |    214%    |   1,000%   |     744%
  CheckedInt!( uint , "slim" ) |     160%   |    249%    |   1,180%   |   2,650%
 SafeIntFast!( uint  )         |     168%   |    231%    |   1,120%   |   1,780%
  CheckedInt!( uint , "fast" ) |     139%   |    230%    |   1,010%   |   2,180%
  CheckedInt!( uint  )         |     135%   |    232%    |   1,020%   |   2,180%

                long           |     113%   |    112%    |     100%   |     134%
SafeIntSmall!(  long )         |     296%   |    353%    |     704%   |     784%
  CheckedInt!(  long, "slim" ) |     296%   |    367%    |     772%   |   1,500%
 SafeIntFast!(  long )         |     299%   |    366%    |     801%   |   1,190%
  CheckedInt!(  long, "fast" ) |     127%   |    350%    |     666%   |   1,210%
  CheckedInt!(  long )         |     126%   |    349%    |     660%   |   1,210%

               ulong           |     100%   |    104%    |     122%   |     102%
SafeIntSmall!( ulong )         |     266%   |    327%    |     730%   |     695%
  CheckedInt!( ulong, "slim" ) |     296%   |    338%    |     780%   |   1,480%
 SafeIntFast!( ulong )         |     272%   |    322%    |     803%   |   1,120%
  CheckedInt!( ulong, "fast" ) |     135%   |    333%    |     667%   |   1,230%
  CheckedInt!( ulong )         |     124%   |    334%    |     667%   |   1,230%

SafeInt!int, specifically, has improved enough that it no longer looks like an outlier, but SafeInt!uint, SafeInt!long, and SafeInt!ulong are all essentially unchanged.

The built-in types are still unambiguously faster than any checked type.
Hybrid CheckedInt!T is still the fastest checked type on GDC (inlineme) -- and by quite a lot when T is a 64-bit type.
SafeIntSmall is still fastest on DMD.
SafeIntFast is still slower than SafeIntSmall (because it's still serializing on NaN checks).
Everything else is about the same, on a given compiler.

As you said, compilers are strange. ;-)

@burner That new test is better, in that it attempts to exercise more operations.
However, I don't think it does what you intend. Here's what I see when I track it:

The first iteration of the outer loop continues almost immediately, because i = 3 and 3 is already in buf[1].
The second iteration begins with i == 5. This quickly passes the primality test.
Having passed the primality test, it then proceeds to the // Collatz sequence loop.
The Collatz loop then takes two iterations to reach i == 4.
Thereafter it gets stuck in a cycle where 4 -> 2 -> 1 -> 4.
This cycle executes several hundred million times. At the end i == 4.
Thereafter, because the stride of the outer for loop is 2, every iteration continues immediately since i is always even and evens are never prime.

The causes of this are failure are two:

The termination condition for the Collatz sequence should be i != 1, rather than i != 0. This breaks the 4 -> 2 -> 1 -> 4 cycle.
The Collatz variable should be separate; reusing i messes up the prime testing sequence.

I would also suggest rearranging things somewhat so that the interesting parts don't get skipped when i is composite.
Here is a version that does something more reasonable:
private T exerciseInt(T)()
{
	enum upTo = 113_382; // Raising the limit above 113,382 will cause the Collatz sequence to overflow if is(T == int).
	T ret = 5;
	T[337] buf; // 337^2 > 113,382
	buf[0] = 2;
	buf[1] = 3;
	T bufIdx = 2;

con: for(T i = 3; i < upTo; i+=2)
	{
		// Collatz sequence
		T cI = i;
		T cN = 0;
		while (cI != 1)
		{
			if (cI % 2 == 0)
				cI /= 2;
			else
				cI = cI * 3 + 1;

			++cN;
		}

		// Digit Sum
		T d = cN + i;
		T dSum = 0;
		while (d > 0)
		{
			dSum += d % 10;
			d /= 10;
		}
		ret += dSum;

		// Trial division primality test
		for (T j = 0; j < bufIdx; ++j)
		{
			if (i % buf[cast(size_t)j] == 0)
				continue con;
		}
		// Save small primes for future trial division
		if (bufIdx < buf.length)
			buf[bufIdx++] = i;

		// Keep ret small so that it doesn't overflow
		if(i < ret)
			ret = ret - i;
		else
			ret = i - ret;
	}

	return ret;
}

Running this fixed test produces similar results to the last set I posted.
Also, you still have a typo on line 36, where i3 should be i2.

mixin -> alias
ldc --release -O3 safeint.d -ofsafeint && ./safeint
int!  int      21490  1776 msecs
SafeIntFast!  int      21490  2706 msecs
SafeIntSmall!  int      21490  2674 msecs
     int! uint      21490  2323 msecs

SafeIntFast! uint      21490  2709 msecs
SafeIntSmall! uint      21490  3074 msecs
     int! long      21490  5221 msecs

SafeIntFast! long      21490  5768 msecs
SafeIntSmall! long      21490  6096 msecs
     int!ulong      21490  4910 msecs

SafeIntFast!ulong      21490  5689 msecs
SafeIntSmall!ulong      21490  5914 msecs

Looks like need to rebase for merge.  BTW, has anyone tested a before and after with the new GDC checkedint intrinsics?

@ibulcaw I intend to test the new GDC intrinsics, but haven't had time to figure out the GDC build process yet. Hopefully I will get to it some time this week.
If you want to run my benchmark on your own system:

Download the latest version of my CheckedInt code.
Compile with -m64 -O3 -frelease without the new intrinsics support. Run it and copy-paste the console output for us here.
Delete inlineme.checkedint.d and change line 3 in preciseops.d to import core.checkedint.
Compile again, this time with the new intrinsics support. Run it and copy-paste the console output.

Under GDC, the optimized benchmark takes about 30 seconds.

gdc is building right now, but I had no luck in the past building it. @ibuclaw http://code.dlang.org/packages/safeint

gdc (GCC) 5.1.0
int!  int      21490  1853 msecs
SafeIntFast!  int      21490  2493 msecs
SafeIntSmall!  int      21490  2680 msecs
     int! uint      21490  2428 msecs

SafeIntFast! uint      21490  3128 msecs
SafeIntSmall! uint      21490  3291 msecs
     int! long      21490  5339 msecs

SafeIntFast! long      21490  6811 msecs
SafeIntSmall! long      21490  6803 msecs
     int!ulong      21490  4868 msecs

SafeIntFast!ulong      21490  6350 msecs
SafeIntSmall!ulong      21490  6712 msecs
gdc (GCC) 6.0.0 20150705 (experimental)
     int!  int      21490  1819 msecs

SafeIntFast!  int      21490  1843 msecs
SafeIntSmall!  int      21490  2288 msecs
     int! uint      21490  2341 msecs

SafeIntFast! uint      21490  2721 msecs
SafeIntSmall! uint      21490  2754 msecs
     int! long      21490  5200 msecs

SafeIntFast! long      21490  5346 msecs
SafeIntSmall! long      21490  5620 msecs
     int!ulong      21490  4896 msecs

SafeIntFast!ulong      21490  4978 msecs
SafeIntSmall!ulong      21490  5407 msecs
looks pretty good IMO

@burner I just realized that we have to implement opBinaryRight. Otherwise:
SafeInt!int s;

s = int.max;
s = s * 2;
writeln(s); // nan

s = int.max;
s = 2 * s;
writeln(s); // -2

I found out about this when I removed alias value this from CheckedInt; the alias was hiding the lack of opBinaryRight by having it fall back to unchecked arithmetic. Oops...

@ibuclaw I (eventually) got GDC master to compile. Initially, my code would not compile. I reduced the problem to the following:
module main;

import core.checkedint;
import std.stdio;

enum ICEit = true;

static if(ICEit) {
// This yields "internal compiler error: in expand_intrinsic_arith", etc.
    auto test(ref bool overflow) {
        return adds(1, 1, overflow);
    }
} else {
// This would compile fine.
    auto test(ref bool overflow) {
        bool overflow2 = overflow;
        auto ret = adds(1, 1, overflow2);
        overflow = overflow2;
        return ret;
    }
}

void main(string[] args)
{
    bool overflow = false;
    writeln(test(overflow));
    writeln(overflow);
    stdin.readln();
}

Incorporating that workaround into my CheckedInt code, I see good performance with the new GDC. I won't bother posting detailed numbers until you fix that bug, just in case it effects the results.
However, a quick summary is that the performance with real intrinsics is a little better than the (already very good) numbers I was getting with copy-paste induced inlining with GDC 2.066.1.

@tsbockman - raised a bug report for that.  Though it was trivial to adjust to handle ref parameters also.  https://github.com/D-Programming-GDC/GDC/commit/7ed8c84c285e3426175085a26acef5c134c861ac

However, a quick summary is that the performance with real intrinsics is a little better than the (already very good) numbers I was getting with copy-paste induced inlining with GDC 2.066.1.

Good, I can rest now then. ;-)

@tsbockman nice catch.

@ibuclaw I can confirm that ref parameters are working now.
Just LDC 32-bit left that needs the copy-paste of core.checkedint now... I should go raise a bug report with them.

@burner Sadly, not all binary operators are commutative:
writeln(8 / 4); // 2

SafeInt!int sa = 4;
writeln(8 / sa); // 0???

@tsbockman I felt so smart :(

@burner I have been finding keeping track of all the corner cases with this stuff quite difficult; you might want to have a look at my fuzz testing in the unit tests at the end of preciseops.d.
This method is probably too slow for the final Phobos submission, but in the mean time it has caught many bugs in my own code. It even revealed a couple of problems with D runtime - #14785 and #14786.

@burner I have submitted a tiny fix for druntime issue 14785, a bug in core.checkedint.
Would you mind reviewing it? No one else seems to be taking an interest at the moment: druntime pull request 1365.

Would love to see aliases sbyte, sshort, sint and slong to be standardized and aliased to the proper SafeInt wrapper. This will then facilitate use of safe int in places where built-ins are typically used today.

@DmitryOlshansky Yes, standard aliases are a planned feature.
At the moment, I am still optimizing performance and refining the interface in other ways here: CheckedInt.
Once the design settles down enough I hope to find time to fully document my work and turn it into an alternative pull request. It's clear that this won't be ready in time for 2.069. Maybe 2.070?

@DmitryOlshansky there are now S(byte|short|int|long), Su(byte|short|int|long), SF(byte|short|int|long) and SFu(byte|short|int|long)

@DmitryOlshansky there are now S(byte|short|int|long), Su(byte|short|int|long), SF(byte|short|int|long) and SFu(byte|short|int|long)

I belive that following built-in notation of everything in lowercase is in order here.

@DmitryOlshansky
I would argue that they are custom types -> uppercase first letter. how about this is being discussed in a review thread. do you want be the review manager?

@DmitryOlshansky
I would argue that they are custom types -> uppercase first letter. how about this is being discussed in a review thread. do you want be the review manager?

I'd take that role in a couple of days if nobody else would. Please assemble all relevant things: docs, code (this pull, other forks?) and a short description.  Thanks!

@DmitryOlshansky
code: is here
dub: could be found http://code.dlang.org/packages/safeint
doc: CyberShadow/AutoTest

I have written some questions concerning the design trade-offs involved in making a good SafeInt/CheckedInt type. (I will post this on the dlang.org forum, as well; discussion which is not specific to SafeInt's current design probably belongs there, rather than here.)
For the purposes of this poll please assume the following (based on my own extensive testing):

Code using checked operations will take about 1.5x longer to run than unchecked code. (If you compile with GDC, anyway; DMD and LDC are another story...)
The main design decision with a significant runtime performance cost, is whether to throw exceptions or not. With some optimization, the hit is modest, but noticeable.
Even if the API uses exceptions some places, it can still be used in nothrow @nogc code, at the cost of some extra typing.

Two further points I would ask the reader to consider:

A checked integer type is fundamentally semantically different from an unchecked type. The difference is of similar magnitude to that of floating-point vs fixed-point.
It might be wise to read the entire poll before answering it - the questions are all related in some way.

The poll results are here, if you wish to preview the questions:
http://polljunkie.com/poll/kzrije/checked-integer-type-behaviour/view
When you are ready, please take the poll yourself:
http://polljunkie.com/poll/cytdbq/checked-integer-type-behaviour
Thanks for your time.

@DmitryOlshansky I wrote a very short description about SafeInt in the post above. Docs are created through cybershadows tester, the code is here and tsbockman has not created a PR for his checkedint

@DmitryOlshansky I wrote a very short description about SafeInt in the post above. Docs are created through cybershadows tester, the code is here and tsbockman has not created a PR for his checked

Hm, okay let's ping @tsbockman and I'll try to wrap up a review thread later today.

thanks

@burner @DmitryOlshansky I said before that I don't expect to have anything ready in time for 2.069. Surely there is still plenty of time to get something ready for 2.070, seeing as 2.069 hasn't even been released yet?
One of the questions I asked on the survey, is how much of a hurry people are in to see SafeInt or something similar in Phobos. 90% selected this option: Take your time and make sure you get it right the first try.
If you look at the rest of the survey results, you will see that a clear (well, as clear as it can be with only 10 responses...) majority favour a design which is much more similar to CheckedInt than SafeInt.
What are you two looking to get out of a review thread right now:

You aren't willing to wait?
You think a review thread will reach a very different conclusion than the survey?
Something else?

I'm not trying to be a pain here; I just don't see what the rush is.
CheckedInt is probably about 95% complete now, as far as the API and implementation go. The main thing that is missing as this point (which I now acknowledge as necessary based on the survey feedback) is to make optional or hidden the bitwise operations somehow, preferably without introducing too much additional complexity or template bloat.
Beyond that, I need to document my work and streamline the test suite.
The hold-up on my end is mostly that I am quite ill at the moment, and unable to concentrate sufficiently to work much on this.

What are you two looking to get out of a review thread right now:

You aren't willing to wait?
You think a review thread will reach a very different conclusion than the survey?
Something else?

Bring to a wider audience? Polish up the rough edges of SafeInt? I don't mind competing proposals in fact that'd be great.
So...

The hold-up on my end is mostly that I am quite ill at the moment, and unable to concentrate sufficiently to work much on this.

Then let us postpone it a bit. What I'd want is head-to-head discussion (battle?) of the 2 proposals on NG review thread. One poll is hardly a good evidence, and BTW any stats on it? How many people voted, were they core developers or passers by?

If you look at the rest of the survey results, you will see that a clear (well, as clear as it can be with only 10 responses...) majority favour a design which is much more similar to CheckedInt than SafeInt.

how do you get to that conclusion?

@DmitryOlshansky

One poll is hardly a good evidence,

True, but at least it's something. My previous attempt at starting a discussion in the forums yielded basically no useful feedback other than some arguing about bitwise operators.

and BTW any stats on it?

You can view the poll results here: http://polljunkie.com/poll/kzrije/checked-integer-type-behaviour/view
At the upper right of each question is a tiny button whose tool tip says "View Detail" that will pop up more detailed statistics for the sorting-style questions.

How many people voted,

10 people voted. More would definitely be nice, but 10 is at least 8 more people than I had answers to most of those questions from before...

were they core developers or passers by?

The survey was anonymous, so I don't know.
What I do know about who took the survey is that:

They found it either here, or in the in the main dlang.org forum.
No one who took the survey selected the I probably won't use this option in response to the last question.

@burner

how do you get to that conclusion?

Leaving aside the final two "soft" questions on the survey, there are twelve questions to which the specific behaviour of a checked integer type may be objectively compared. The responses to two of those twelve are split 50/50, and implementing the majority answer on a third would require changes to D's operator overloading, so that leaves nine differentiating points to consider.
Of those nine, I believe that SafeInt currently implements the majority answer on only one. CheckedInt implements the majority answer on eight.
If you can't work out how I got those numbers, I can see about whipping up some example code to demonstrate the semantic differences between the two.

After rereading the questions/answers in the pool, I find that some are quite suggestive i.e.

When casting to an unchecked integer type T, a checked integer value which is greater than T.max or less than T.min should:

both answer are not idiomatic. the correct way is not present. use to!int(SafeInt!long(long.max));
@DmitryOlshansky I also believe a competition will not benefit either proposal because:

SafeInt and CheckedInt are actually not designed for the same thing. i.e. SafeInt does not allow shifting CheckedInt does.
tsbockman and I will be tempted to follow user request to selected. even though the user request may only be beneficial to that one user
cooperation is better than competition https://en.wikipedia.org/wiki/Prisoner's_dilemma

@tsbockman maybe CheckedInt ticks eight out of nine boxes. But who says that these nine boxes are the important ones.
To make this clear I will not join a competition over this. I joined the D development to cooperate not compete. SafeInt is designed in a way, I wanted a "secured" Integer type to work. I thought other people would like such an Integer impl., therefore I created this PR. I will not change any or the core design to win a competition.

Regarding your poll, I find the results for the penultimate question almost horrifying... Who answered this anyway? Mathematical correctness and ease of writing correct code are in dead last for desired features. This is why we want a checked int type in the first place, isn't it?

@burner

SafeInt and CheckedInt are actually not designed for the same thing.

You are quite correct. However, there is enough overlap that I don't think we will (or should) get support for putting both into Phobos.

i.e. SafeInt does not allow shifting CheckedInt does.

Please don't focus on that point in particular; based on the feedback I've gotten (not just in the survey), I will likely hide the bitwise operators such that they are no more easily accessed than they are on SafeInt.

tsbockman and I will be tempted to follow user request to selected. even though the user request may only be beneficial to that one user

That's why I've been trying to get feedback from a wider audience, to avoid tailoring CheckedInt exclusively to my own needs. Honestly, I was somewhat surprised at how closely the majority response came to my API on the survey; I really wish I could get about ten times as many responses, just to convince myself it's not a statistical accident.

maybe CheckedInt ticks eight out of nine boxes. But who says that these nine boxes are the important ones.

If you think there are more important boxes not represented in my survey, please tell us what they are, and consider doing your own follow-up survey.
I did try to design my survey to enable people to express a preference for something more like your design, if that's what they want. I also suggested some reasons why they might do so in the survey: it is (ideally) faster due to the lack of exceptions, and less verbose to work with in code that mixes checked and unchecked types.

SafeInt is designed in a way, I wanted a "secured" Integer type to work. I thought other people would like such an Integer impl., therefore I created this PR. I will not change any or the core design to win a competition.

This sounds like you're saying you will "follow user request to selected. even though the user request may only be beneficial to that one user" - the one user being you. What am I missing here?

@MetaLang "Mathematical correctness" and "Easy to write correct code" are first, not last.
The numbers on the left are the average rank given by respondents, with 1 being the highest possible priority. This is much more obvious when you actually take the poll, as opposed to looking a the somewhat poorly-formated results, afterwards.

@tsbockman

This sounds like you're saying you will "follow user request to selected. even though the user request may only be beneficial to that one user" - the one user being you. What am I missing here?

I think your sentence is missing a "be" -> ... follow user request to be selected ...
But no, If @DmitryOlshansky wants a competition, I will close this PR and not participate in such competition.
This is going totally off-topic. But IMO polling about an api is wrong in the first place. Because, again polling is competition and cooperation is just better.

@burner Polling is simply a tool to find out what a group thinks. Knowing what the people with whom you wish to cooperate think seems important to me...
I am not intending to make the poll authoritative.
However, you can expect the same issues raised by the poll to be raised in any formal review, because they are questions which naturally arise in the course of studying this design.

But no, If @DmitryOlshansky wants a competition, I will close this PR and not participate in such competition.

This is almost intimidating :) Anyway if we have 2 designs that are not converging there is a competition like it or not. The design that tries to be what 90% of (key) people expect typically wins. I have nothing to do with it, maybe "want" was misleading word.
Also - "take it or leave it" approach certainly won't pass the review. If you'd like to close this PR it's up to you but pointing at me b/c "I want competition" and somehow you hate competitions is ... strange.

cooperation is better than competition https://en.wikipedia.org/wiki/Prisoner's_dilemma

It's just one specific situation. And merging 2 incompatible designs hardly wins anything (if they are trully incompatible). On the contrary survival of the fittest works even if it's not aesthetically appealing to some of us.

@DmitryOlshansky I'm not trying to intimidate. Maybe I was not clear enough, there is competition between CheckedInt and SafeInt, no doubt. But I don't want to see a fight about what gets into phobos. Ultimately one author will not feel welcome anymore. I don't want that.
I think pooling is wrong as there is no open discussion about the design, only predefined meanings to select from.
@tsbockman I'm always talking about cooperating, so lets actually do that.

Lets each created a feature list and discuss the differences.
....

@DmitryOlshansky addition: ... and not wanting a fight about whats gets into phobos for me means, not participating in the competition about phobos. -> closing the PR

@tsbockman My mistake then, that's actually a relief.

@burner

Ultimately one author will not feel welcome anymore. I don't want that.

I can only speak for myself, but I will not take it personally if my design does not "win". There's always code.dlang.org...
This should be about what will best serve the long-term interests of Phobos users, not about our personal feelings.

I think pooling is wrong as there is no open discussion about the design, only predefined meanings to select from.

Polling is useful in this situation for several reasons:

There are decisions to be made for which no amount of discussion will produce agreement. Polling offers a vaguely objective means of picking a side. For example, the almost perfect 50/50 split on the bitwise operator issue is what convinced me that it is worth the extra complexity to make the feature optional/hidden (details to-be-determined).
Discussions can go off-topic; polls don't. When I tried to start a general design discussion in the forums, we just ended up arguing about the highly contentious, but not central to the design, issue of bitwise operators. The poll I posted got me some real feedback about the other major issues.
Answering a poll is much faster and much less work than writing thoughtful forum posts. Just because someone wouldn't spend hours reading and writing prose doesn't mean their opinion is without value. Polling taps a wider audience.

Moreover, both places the poll was posted were discussion threads. Anyone who wants to give feedback that doesn't fit the poll format is welcome to do so.

Lets each created a feature list and discuss the differences.

The poll itself is basically that list. There are, of course, some missing bits and pieces, but it is reasonably comprehensive.
At a higher level, I see the most important design decisions as lying on two axes:

Mathematical correctness versus conformance to the behaviour of the built-in integer types.
Enforcement of correct API use - particularly when mixing checked and unchecked types - versus minimizing verbosity of typical use.

At the moment, CheckedInt favours mathematical correctness and enforcement of correct API use, while SafeInt favours conformance to the behaviour of the built-in integer type and minimizing verbosity of typical use.
An example of mathematical correctness versus conformance to the behaviour of the built-in integer type:
const int n = 1_000_000_000;
const uint m = 3_000_000_000u;
writeln(n - m); // Prints "2294967296".

SafeInt!int sn = n;
SafeInt!uint sm = m;
writeln(sn - sm); // Prints "nan".

CheckedInt!int cn = n;
CheckedInt!uint cm = m;
writeln(cn - cm); // Prints "-2000000000".

The answer given by the unchecked types is, in this case, simply wrong. However, although SafeInt and CheckedInt give different answers, either one could be considered "correct" depending on your priorities.
If you want to use a checked type as a means of debugging in a program which you intend to work properly even with unchecked arithmetic, then SafeInt's nan answer is better. It says, "that operation will fail if you try it with unchecked arithmetic".
On the other hand, if you are committed to using checked types, even in release mode, then CheckedInt's -2000000000 is better: why wouldn't you want the mathematically correct answer, as long as it can be represented by the operation's return type?
An example of Enforcement of correct API use versus minimizing verbosity of typical use:
X binarySearch(X, A)(A[] a, A key) {
    X low = 0;
    X high = cast(X)(a.length - 1); // Ignore this bug; the one we're interested in is below

    while(low <= high) {
        X mid = (low + high) / 2;
        A midVal = a[mid];

        if (midVal < key)
            low = mid + 1;
        else if (midVal > key)
            high = mid - 1;
        else
            return mid; // key found
    }
    return X(-(low + 1));  // key not found.
}

int[] a = new int[2_000_000_000]; // all initialized to 0, by default
writeln(binarySearch!long(a, 1)); // Correct: "-2000000001".
writeln(binarySearch!int(a, 1)); // "Segmentation fault" with -noboundscheck

The above demonstrates a recurring real-world bug caused by integer overflow. Among the code impacted by this subtle mistake was Sun's java.util.Arrays.binarySearch() for about the first decade of its existence.
It's an integer overflow bug, so SafeInt will catch it, right?
writeln(binarySearch!(SafeInt!int)(a, 1)); // "Segmentation fault" with -noboundscheck

Well, sort of... SafeInt does detect the overflow of X mid = (low + high) / 2, but when we access a[mid] on the next line, the nan state is implicitly cast to some integer value. This still causes a segmentation fault (if we're lucky), just maybe one that hits a different invalid memory address than before.
This isn't exactly SafeInt's fault; the API of SafeInt makes no guarantees about the safety of a cast to the underlying unchecked type. The programmer is responsible for prepending each such unsafe conversion with if(mid.isNaN) .... But, SafeInt's design in this area makes it very easy to forget to do so.
On the other hand, with CheckedInt the faulty code above won't even compile:
A midVal = a[mid]; // cannot implicitly convert expression (mid) of type CheckedInt!(int, cast(StoreNaN)1) to ulong

We can fix it the same nothrow way we would with SafeInt:
if(mid.isNaN)
    // handle the failure somehow here...
A midVal = a[mid.value];

Or, we could instead add an explicit cast(ptrdiff_t)mid which will throw an exception if it fails:
writeln(binarySearch!(CheckedInt!int)(a, 1)); // Exception: long cannot represent nan

This comes with the usual helpful trace of the callstack, and if we wanted to, we could catch it at some higher level of code and handle it appropriately. Most importantly, invalid casts of CheckedInt values will reliably throw, whereas more complex variants of the kind of bugs which cause Segmentation faults sometimes just silently give wrong answers and/or corrupt memory.
SafeInt's approach is slightly more convenient if you don't care about checking for nan - but then, why are you using SafeInt instead of just int?
This kind of problem is not specific to array indices, but pops up any time you want to pass a SafeInt value as an argument to a function which only accepts unchecked parameters. For some types of code, a lot of manual checks are required to properly handle all possible nan states.
Knowing programmers (myself included), I expect that people will forget/be too lazy to insert all those checks everywhere they should. In this case, SafeInt may give a false sense of security.
CheckedInt's approach guarantees that you won't just forget, and is much less verbose if you choose to do things right, and are willing to allow exceptions. It could be made even more streamlined by adding an alias this opCast!T, but I think both for performance reasons, and for sanity's sake when writing nothrow code, it is better not to have an implicit cast which can throw.
Overall, my personal exploration of the design space leads me to believe that a good, safe, performant drop-in replacement of the unchecked types, for debugging purposes, is not possible without compiler support.
Since the other requirements impose an incompatible API anyway, I think conformance to the behaviour of unchecked integers should be sacrificed in favour of improved mathematical correctness, particularly in the area of mixing signed and unsigned types. There is no meaningful performance cost to doing so, it expands the range of valid inputs for checked algorithms, and it eliminates a lot of tedious casting when trying to write correct code which mixes signed and unsigned.
Concerning enforcement of correct API use, as @MetaLang said, if ease of writing correct code is not a top priority, why use a checked type in the first place? If you're smart enough to never omit a necessary isNaN check, you're probably smart enough to write correct code using the unchecked types directly.

I only speak for myself, but I will not take it personally if my design does not "win". There's always code.dlang.org...

Same here. I really do not see how hiding one of proposals in favor of the other will help the community. Ultimately what gets 90% of use cases work w/o loosing on typical things like preventing abuse, being not error-prone, fast and easy to use "wins".
To me there is only one rule - whatever gets to Phobos must be the best design we could come up with given the constraints.

I thought about SafeInt and its interface some more. I think it should behave something like this:

Bahave as integers do
1.1 except when integers behave stupid, then turn the SafeInt into an NaN value
Do not allow implicit conversion like float -> SafeInt -> double
Do not allow implicit conversions in ctors

auto s = SafeInt!int(1.337); // This should not compile
auto t = SafeInt!int(to!int(1.337)); // Force the explicit conversion

Make it hard for people to ignore conversion error.

auto s = SafeInt!int(256);

byte b;
if(s.get!byte(b)) {
    // handle conversion error;
}

No opBinaryRight, because it is unclear

auto s = 10 * SafeInt!byte(256); // What is the result type, and what is the result

Force it to be explicit
auto s = SafeInt!int(10) * SafeInt!byte(256); //typeof(s) == SafeInt!int, s == NaN

SafeInt!int is redundant Safe!int is enough

1 and 1.1 I assume people want to SafeInt to behave as integer except when interger behave wrong. There they want someway to find out that something went wrong.
2 If you don't have implicit conversion your conversion need to explicit a cast or std.conv.to. These explicit conversion are easy to spot. Which means that errors introduced are easy to spot.
3 See 2
4 If you need to move the value of a SafeInt into a normal integer people should have a hard time ignore possible error.
5 Make it explicit, so people do not assume something that might be wrong.
6 DRY
thoughts?

@burner

1 and 1.1 I assume people want SafeInt to behave as integer except when interger behave wrong. There they want someway to find out that something went wrong.

Bahave as integers do 1.1 except when integers behave stupid, then turn the SafeInt into an NaN value

The benefit of your proposal here, is that it makes SafeInt into a debugging tool for finding problems in code which is written to - theoretically - work just fine with unchecked integers. So, you could compile a function with is(T == SafeInt!int) to test it; if it didn't seem to be generating any nan states you could then recompile it with is(T == int) for better performance in release mode.
This would be a useful tool, and it was originally how I envisioned a checked integer type should work. However, there is a good reasons not to do it this way - or at least, not only this way:
Checked arithmetic is all about detecting failures in obscure corner cases. For many types of code, it is just not practical to test all of those corner cases to see if a nan is generated. (And frequently, if you can actually figure out all the cases you need to test in the first place, you don't need a checked type to find the bugs anyway.) In light of this, to provide much real benefit SafeInt will often need to be used in release mode, not just debug mode.
If you accept that SafeInt should always be used, even in release mode, then what benefit is there to making it fail in all the same cases that the underlying unchecked type does? At this point, it is a distinct numeric type, and free to extend correct behaviour to cover additional scenarios, such as signed/unsigned comparisons.
One of the key conclusions from my poll, is that a strong majority of people want a checked integer type which does the mathematically correct thing in as many cases as possible - regardless of what the unchecked types do. This is what CheckedInt tries to do.
I started moving in this direction, because when I tried to figure out how to make a drop-in checked replacement for the standard unchecked types, I couldn't figure out how to make something with a similar enough API to really accomplish that.
However, I am now reconsidering this earlier statement:

However, there is enough overlap that I don't think we will (or should) get support for putting both into Phobos.

I am beginning to think that we should design two distinct types, after all:

A DebugInt type whose sole design goal is to serve as a tool to help programmers write less buggy integer code, which can then be recompiled to use unchecked types in release mode, for speed. This would be based on the general principles outline in your last post.

A CheckedInt type which doesn't even try to be interchangeable with the built-in unchecked integer types, in favour of achieving mathematically correct behaviour in as many cases as possible, without sacrificing too much performance. This would be a refinement of my work, satisfying the desire expressed by the poll results.

If this split makes sense to you, then we can focus on the development of the DebugInt type for now, and I will open a separate pull request later for CheckedInt.

Clearly there are two different design goal.
Having Debug!int (even though I don't like the name) as a way to debug and tighten int processing is pretty much my current goal now.
Something like:
auto someMathFunctionWorkingOnInts(T)(T a) { .... }

Safe!int a;
int b;

a = someMathFunctionWorkingOnInts(a);
b = someMathFunctionWorkingOnInts(b);

would be nice
I think getting Checked!int into phobos as well as Debug!int is going to be difficult as the delta between Debug!int < Checked!int < BigInt would be rather small. Maybe adding a small value optimization to BigInt could fill a similar niche.

@burner

Debug!int (even though I don't like the name)

Yeah the name is just a place-holder, to distinguish (in my own mind) SafeInt's new direction from that of CheckedInt.
Safe!int does look nicer than SafeInt!int, but don't you think Safe is a bit too short and ambiguous for a type name? We already agreed to include compact aliases, anyway.

I think getting Checked!int into phobos as well as Debug!int is going to be difficult as the delta between Debug!int < Checked!int < BigInt would be rather small.

I hope not; there is clearly demand for something like CheckedInt, but I also think DebugInt would serve a distinct and useful purpose.
The way I envision it, people who really want safety will use CheckedInt all the time, while the availability of DebugInt will improve the quality of code on projects where the author is unwilling to accept even the 1.5 - 2x performance hit from using CheckedInt in release mode.
Regardless, as I said earlier, there is always dub.

Maybe adding a small value optimization to BigInt could fill a similar niche.

That's an interesting thought, but even if performance could be made acceptable (which I highly doubt), BigInt still uses potentially unbounded memory, and cannot easily be made @nogc.

@burner Addressing some of the details of your earlier ideas:

Do not allow implicit conversion like float -> SafeInt -> double
Do not allow implicit conversions in ctors

Implicit/unchecked conversions should be allowed for those type conversions where the new type can represent all possible values of the old.

Implicit unchecked integer -> SafeInt should be allowed in some cases:

template IsSafeConversion(From, To)
    if(isIntegral!From && isIntegral!To)
{
    enum isSafeConversion = From.sizeof <= To.sizeof && isSigned!To?
        (isSigned!From || From.sizeof != To.sizeof)
      : isUnsigned!From;
}

SafeInt -> unchecked integer must always be explicit and checked, if only because the SafeInt could be nan
Floating-point -> SafeInt must obviously be explicit and checked, as you say.
SafeInt -> floating-point does not always need to be checked, though:

template isSafeConversion(From, To)
    if(isIntegral!From && isFloatingPoint!To)
{
    enum isSafeConversion = (From.sizeof * 8 - (isSigned!From? 1 : 0)) <= To.mant_dig;
}

I see no benefit to blocking the safe conversions outlined above.

Make it hard for people to ignore conversion error.

Yes.

No opBinaryRight, because it is unclear
Force it to be explicit

I think opBinaryRight is clear enough, personally.

If you don't have implicit conversion your conversion need to explicit a cast or std.conv.to. These explicit conversion are easy to spot. Which means that errors introduced are easy to spot.

I need to study std.conv.to more; I am not familiar with it.
Depending on exactly what you have in mind here, in certain scenarios you may be turning what should be one explicit check into two, by separating the SafeInt.isNaN check from the bounds check. This should be a solvable problem, though.
One addition I would make to your list, in line with the DebugInt concept, is that mixed signed/unsigned operations, including comparisons, should be a compile-time error. When people mix signed and unsigned types, they usually want a signed operation, but D gives them an unsigned operation instead. Make them explicitly convert one operand or the other to clarify their intent.
SafeInt's current behaviour of fixing signed/unsigned comparisons is nice, but creates a dangerous inconsistency with the unchecked types. This feature belongs in a CheckedInt style type, but not a DebugInt one.

Re: signed/unsigned comparisons, it turns out that there is already a plan in place to (mostly) fix this for the built-in unchecked types: DMD PR #1913 fixes bug 259 according to the plan laid out by Andrei Alexandrescu in comment 39, which has been pre-approved by him and Walter Bright.
If DMD PR #1913 (or something similar) ever actually gets merged, this will allow SafeInt to keep its current enhanced comparison behaviour without causing bugs in unchecked release-mode code.
I had assumed that altering the coercion rules in this way for the unchecked types would be rejected out of hand, but now I am wondering: if we could get signed/unsigned arithmetic fixed, as well as signed/unsigned comparisons, perhaps the goals of DebugInt and CheckedInt could be satisfied by a single type, after all?
The desire to fix signed/unsigned mixed operations is a major driver of the differences between the DebugInt and CheckedInt approaches.

hm, interesting

@burner Since the original author seems to have moved on to other things, I decided to try and update that signed/unsigned comparison patch for DDMD, myself. I was semi-successful; see my recent comment on the PR.

@tsbockman nice, I'm not a (d)dmd hacker. But thank you for making an effort. But make sure that the powers that are like this change.

@burner

nice, I'm not a (d)dmd hacker.

OK. I will probably make a more serious effort to debug or rewrite the PR myself; I just figured it was worth asking around a bit first, in case the answer was obvious to someone else.

But make sure that the powers that are like this change.

The solution being attempted was designed and pre-approved by Walter and Andrei back in 2013. Andrei was still hoping someone would get it working as recently as the beginning of 2015.

The solution being attempted was designed and pre-approved by Walter and Andrei back in 2013. Andrei was still hoping someone would get it working as recently as the beginning of 2015.

that sound very goods

@burner I have started work on fixing DMD issue 259, but based on the response so far I don't think my patches will get accepted quickly. Moreover, to unify DebugInt and CheckedInt would also require a similar fix for signed/unsigned arithmetic, and getting that accepted would surely be a long process.
In light of this, I intend to proceed with the split design we discussed earlier:

A DebugInt type intended to be interchangeable with regular unchecked integer types.
A CheckedInt type that abandons compatibility with unchecked types in favour of greater mathematical correctness and a more convenient API.

I've been thinking some more about what the actual names should be. My solution:

The real name of DebugInt can be SafeInt.
The real name of CheckedInt can be SmartInt.
The Int postfix might be dropped, as you suggested earlier.
Place both types in module std.checkedint.

I think this captures the distinction between the two types, while avoiding the awkward DebugInt name. The module itself really should be called std.checkedint both to match core.checkedint, and to avoid unnecessarily biasing users' choice of checked type.
What do you think of the SmartInt name?

I do not like the Smart(Int) name. Smart sounds like something smart will be done I, as a user, have to understand to use it correctly. When actually, GOODNAME!int is a accurate portray of the mathematical integer domain. So maybe an alternative would be Mathematical!int.

@burner

Smart sounds like something smart will be done I, as a user, have to understand to use it correctly.

I intend the name to be reminiscent of "smart pointer", as described by the Wiki:

A smart pointer is an abstract data type that simulates a pointer while providing additional features, such as automatic memory management or bounds checking. These additional features are intended to reduce bugs caused by the misuse of pointers while retaining efficiency.

So, "smart" as in the data type itself is smarter than a standard pointer or integer - with the goal that the user doesn't have to think so hard to use it correctly.
Mathematical!int or MathInt!int is also a possibility; I will mull it over.
Although conceptually it feels like a better fit for a BigInt type, the name does have the distinct advantage of starting with a different letter than SafeInt, enabling a good alias scheme:
alias mint = MathInt!int;
alias sint = SafeInt!int;

version(Debug) {
    alias dint = SafeInt!int;
} else {
    alias dint = int;
}

I have uploaded a working draft of the split SafeInt/SmartInt design to my CheckedInt repository.
This is almost a complete re-write of my previous version, to allow the SafeInt and SmartInt types to share some code. It also includes many other small improvements.
Summary of the current status:

SafeInt and SmartInt themselves are largely complete (albeit in need of more polish in various areas).
Important features still on my TODO list (I do not expect any of these to be difficult):

An easy, safe, nothrow way to convert from checked to unchecked types.
Segregated bitwise operators
Concise aliases

The tests might be about 50% complete.
There is no documentation at all right now.
The only major technical risk I see with the current design, is template bloat.

Feedback is welcome.

The only major technical risk I see with the current design, is template bloat

If at some point the compiler gets smarter about instantiations it can discard (everything was inlined) then this becomes a non-issue.  Or at least I remember it being suggested as an argument being used for octal!xxx.

@ibuclaw

If at some point the compiler gets smarter about instantiations it can discard (everything was inlined) then this becomes a non-issue.

Yes, I have tried to design things in such a way that almost everything is inlined.
Still - even if all gazillion possible template instantiations don't actually end up in the final generated binary, they do slow down compilation significantly.

Progress update: Upon further testing, I realized that my previous approach to the SafeInt API was rather painful to use in nothrow code.
I am currently experimenting with a different design along the lines of IeeeFlags, rather than NaN. I believe this will be considerably less awkward, but won't know for sure until I get my nth rewrite to a usable state.
Note this would effect SafeInt only; SmartInt works well with @burner 's original sentinel NaN value concept.

I am currently experimenting with a different design along the lines of IeeeFlags, rather than NaN. I believe this will be considerably less awkward, but won't know for sure until I get my nth rewrite to a usable state.

I got this into a testable state today, and I like it much better. Using SafeInt is easy now, with or without nothrow.

My rewrite is up now: https://github.com/tsbockman/checkedint
You can also find it on DUB: http://code.dlang.org/packages/checkedint

merge conflict - please rebase.
@tsbockman what is the status of this? Does CheckedInt replace this PR?

@greenify

what is the status of this? Does CheckedInt replace this PR?

It is intended to, but @burner will have to judge for himself whether my work fulfills his goals well enough for him to close this.
As for checkedint:

I'm pretty happy with the basic design, feature set, and non-DMD performance now.
The tests are reasonably complete, although there is a lot more that could be added. In particular, I plan to write many more unittest blocks as part of the docs.
It has some real documentation, but I know that the docs need a lot more work:

They are incomplete (not all public symbols are covered yet).
They haven't been reviewed by anyone else yet, so there's probably stuff in there that seems clear to me (because I wrote checkedint, and don't need the docs myself) that will turn out to be confusing to others.
They're not in Phobos style yet. (It's currently a stand-alone DUB package, and I haven't got around to figuring out how to use the Phobos docs style sheet and DDOC macros there.)
Because of the way I structured the code, if I finish the docs as-is there will be a lot of near-duplicate entries. I'm not quite sure what (if anything) I should do about this issue yet.

So the docs are the main blocker for checkedint at this point. Still, I think they're already good enough that anyone who has been following this thread, and is willing to skim the source code, should be able to evaluate my work.
checkedint is not ready to be accepted into Phobos yet (because of the docs), but it could reasonably be rejected at this point if the community reviews the design and decides that it's not what people want.

They're not in Phobos style yet. (It's currently a stand-alone DUB package, and I haven't got around to figuring out how to use the Phobos docs style sheet and DDOC macros there.)

For Phobos style you can use dfmt, for the macros have a look what we do at mir - we want to preview the macros for upcoming modules too - see here.

They haven't been reviewed by anyone else yet, so there's probably stuff in there that seems clear to me (because I wrote checkedint, and don't need the docs myself) that will turn out to be confusing to others.

I am happy to make a pass if the docs are up ;-)

Because of the way I structured the code, if I finish the docs as-is there will be a lot of near-duplicate entries. I'm not quite sure what (if anything) I should do about this issue yet.

Do you know of /// ditto? What do you mean exactly?

It is intended to, but @burner will have to judge for himself whether my work fulfills his goals well enough for him to close this.

What is your opinion on this @burner?

@wilzbach

for the macros have a look what we do at mir - we want to preview the macros for upcoming modules too - see here.

Thanks! That should be very helpful.

I am happy to make a pass if the docs are up ;-)

You can easily build the docs yourself from the dub package using dub build -b docs. If you want to give some feedback, please do so by creating an issue or pull request against my GitHub repository:
https://github.com/tsbockman/checkedint

Do you know of /// ditto?

Yes. I use it, but it only helps with some types of near-duplication; not all.

What do you mean exactly?

Go look at my code and it should become clear.


Cyborgs can wear welding masks to become immune to flashes

Description
Read title.
Cyborgs are able to wear any type of welding mask in the provided head slot for cosmetics. They shouldn't be able to wear functional hats in general.
Reproduction

Open dev env
Spawn any cyborg
Spawn any welding mask
Place welding mask on cyborg
Spawn flash
Attempt to use flash on cyborg
Scream

Screenshots
https://github.com/user-attachments/assets/4194afec-c7e0-4dae-8eaf-6034c281cfe4
Additional context
They can also gain minor damage resistances from sec helmets or similar, but that doesn't entirely invalidate the largest form of counterplay against them.

Welding masks should restrict your vision so there is a downside to wearing them constantly.
This PR solves this #31999


[ISSUE-300] Make config type of RSS_CLIENT_TYPE as enum

What changes were proposed in this pull request?
move the ClentType.java from internal-client module to common module
change stringType to enumType for RSS_CLIENT_TYPE in the RssBaseConf.java
use name method convert ShuffleServerConf.RSS_CLIENT_TYPE to String(I am not sure if this is the right way)
Why are the changes needed?
Easy to add different extensions  #300
Does this PR introduce any user-facing change?
No
How was this patch tested?
Already added

Codecov Report

Merging #310 (3b9ccd6) into master (84f781f) will decrease coverage by 1.12%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #310      +/-   ##
============================================
- Coverage     61.01%   59.89%   -1.13%
+ Complexity     1489     1409      -80
============================================
  Files           185      177       -8
  Lines          9314     8782     -532
  Branches        900      864      -36
============================================
- Hits           5683     5260     -423
+ Misses         3326     3230      -96
+ Partials        305      292      -13

Impacted Files
Coverage Δ

...ain/java/org/apache/uniffle/common/ClientType.java
100.00% <ø> (ø)

...uniffle/storage/factory/ShuffleHandlerFactory.java
0.00% <ø> (ø)

.../org/apache/uniffle/common/config/RssBaseConf.java
93.42% <100.00%> (ø)

...a/org/apache/uniffle/server/RegisterHeartBeat.java
87.50% <100.00%> (ø)

...storage/handler/impl/DataSkippableReadHandler.java
83.78% <0.00%> (-2.71%)
:arrow_down:

...pache/spark/shuffle/writer/WriteBufferManager.java

...org/apache/spark/shuffle/RssSparkShuffleUtils.java

.../java/org/apache/spark/shuffle/RssSparkConfig.java

.../org/apache/spark/shuffle/writer/WriterBuffer.java

...che/spark/shuffle/writer/BufferManagerOptions.java

... and 5 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@selectbook Will you continue this pr?

@selectbook Will you continue this pr?

yeah, i will continue to fix it from this night

@selectbook Will you continue this pr?

yeah, i will continue to fix it from this night

Ok, take your time.

@selectbook Gently ping.

@selectbook Maybe you're busy now, I help you solve the code style and review issues. I hope you don't mind it. This pr is related to https://github.com/apache/incubator-uniffle/issues/133. I hope this pr can be merged quickly.

Merged. Thanks @selectbook @jerqi .

@selectbook Maybe you're busy now, I help you solve the code style and review issues. I hope you don't mind it. This pr is related to #133. I hope this pr can be merged quickly.

I'm sorry I've been really busy recently. Thanks for your patience


soapyplutosdr: init at 0.2.2

Description of changes
This is required in order to use a Pluto SDR in gqrx.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[x] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@FliegendeWurst Thank you for the feedback, adjusted accordingly!
@GrahamcOfBorg build soapyplutosdr


[Bug]  Kyuubi Spark Authz plugin can't sync the policy from Apache Ranger

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the bug
Hello, I used Apache Ranger to control the access policy of the Hadoop cluster, and kyuubi sync the policy of hive. When I config the properties using the instroduction of  "https://kyuubi.apache.org/docs/latest/security/authorization/spark/install.html", some problem arised:

The log shows that, It's not certified by Ranger Admin.
So I changed the code of org.apache.ranger.admin.client.RangerAdminRESTClient, add username and password  to be authenticated by Ranger Admin.

Both on client mode and cluster mode, kyuubi spark can successful sync the policy from Ranger Admin.
Affects Version(s)
1.6.0
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
No response
Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?

[ ] Yes. I can submit a PR independently to fix.
[ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
[ ] No. I cannot submit a PR at this time.

It does not sound like a Kyuubi side issue, and I'm even not sure if it's a bug or caused by the wrong usage. The 401 response indicates that there are some issues with authentication/authorization.

have you checked the Ranger server log?
what are your configurations of each component?

It does not sound like a Kyuubi side issue, and I'm even not sure if it's a bug or caused by the wrong usage. The 401 response indicates that there are some issues with authentication/authorization.

have you checked the Ranger server log?
what are your configurations of each component?

It may be a configuration isssue in configing Kyuubi Spark AuthZ Plugin.
At presenet, according to the instroduction of the document in "https://kyuubi.apache.org/docs/latest/security/authorization/spark/install.html#ranger-spark-security-xml", the property of 'ranger.plugin.spark.policy.source.impl' is set to 'org.apache.ranger.admin.client.RangerAdminRESTClient'.

When spark sync the hive policy from Ranger Admin, it shoud use the username and password to authenticate the url http://ranger-admin.org:6080.
But by default, if the property of 'ranger.plugin.spark.policy.source.impl' is set to 'org.apache.ranger.admin.client.RangerAdminRESTClient'. It doesn't authenticate with username and password. Therefore, I got the 401 response.
So, I change some code of 'org.apache.ranger.admin.client.RangerAdminRESTClient', and pass username and password to it so that spark can successful sync the policy from Ranger Admin and pass the authentication of the url of http://ranger-admin.org:6080.
Maybe the instroduction of the document in "https://kyuubi.apache.org/docs/latest/security/authorization/spark/install.html#ranger-spark-security-xml" is not so complete?

full content of ranger-spark-security.xml
<configuration>
    <property>
        <name>ranger.plugin.spark.service.name</name>
        <value>cluster_hive</value>
        <description>Name of the Ranger service containing policies for this SPARK instance</description>
    </property>
    <property>
        <name>ranger.plugin.spark.policy.source.impl</name>
        <value>org.apache.ranger.admin.client.RangerAdminRESTClient2</value>
        <description>Class to retrieve policies from the source</description>
    </property>
    <property>
        <name>ranger.plugin.spark.policy.rest.url</name>
        <value>http://host121:6080</value>
        <description>URL to Ranger Admin</description>
    </property>
    <property>
        <name>ranger.plugin.spark.policy.rest.url.username</name>
        <value>admin</value>
        <description>username to Ranger Admin</description>
    </property>
    <property>
        <name>ranger.plugin.spark.policy.rest.url.password</name>
        <value>admin123#</value>
        <description>password to Ranger Admin</description>
    </property>
    <property>
        <name>ranger.plugin.spark.policy.pollIntervalMs</name>
        <value>30000</value>
        <description>How often to poll for changes in policies?</description>
    </property>
    <property>
        <name>ranger.plugin.spark.policy.cache.dir</name>
        <value>/etc/ranger/cluster_spark/policycache</value>
        <description>Directory where Ranger policies are cached after successful retrieval from the source</description>
    </property>
</configuration>

When spark sync the hive policy from Ranger Admin, it should use the username and password to authenticate ...

WHY IT SHOULD use the username and password?

When spark sync the hive policy from Ranger Admin, it should use the username and password to authenticate ...

WHY IT SHOULD use the username and password?

So that spark can successful sync the policy. Like using curl tool, it's ok to get the policy use the username and password.

But I get nothing without the username and password.

So the key point here is the ranger does not expose the API to setup basic authentication information, then Kyuubi can not do that(maybe we can use reflection but it's may be a dirty way). Then it's not a Kyuubi bug, you'd better to push the Ranger community to expose the new API so that the downstream project can leverage.

Thank you too much for your reply, I will close this issue and raise it  in the Ranger community.

@pan3793 Hello, I test again and put
      <property>
      <name>ranger.credential.provider.path</name>
      <value>/etc/ranger/admin/rangeradmin.jceks</value>
    </property>

into ranger-spark-security.xml and spark works well to sync policies from ranger admin.
Maybe we can improve the document of "https://kyuubi.apache.org/docs/latest/security/authorization/spark/install.html" and add the property list before to ranger-spark-security.xml.

This configuration seems tightly depends on your environment.

I am having this issue too. Normally Ranger policy will be synced with GSSAPI (Kerberos authentication), not sure if Kyuubi plans to support that.

@risyomei the original reporter are talking about basic authentication but you are talking about GSSAPI(Kerberos) authentication, are you SURE you are meeting the same issue?

Even in a fully-kerberized cluster, we see authz is trying to use simple authentication when accessing the ranger admin and failed with the same error message.
Therefore, I think there are two ways of resolving this issue:

Using the credential store as @xiaozhch5 suggests in the comment (https://github.com/apache/kyuubi/issues/3791#issuecomment-1315249377).
Using the Kerberos (GSSAPI) to access the Ranger. <-- That's the thing I am not sure if Kyuubi plans to support.

... Using the Kerberos (GSSAPI) to access the Ranger. <-- That's the thing I am not sure if Kyuubi plans to support.

It already done, in Spark client mode.

Thank you very much for your confirmation, that's exactly what I wanted to know.
I was having problem with Spark cluster mode + Kyuubi LDAP Authentication.
Is there any way to use Spark cluster mode + Kyuubi LDAP Authentication + authz at the moment?

I explained the technical details on https://github.com/apache/kyuubi/discussions/3620, it also mentioned some workaround, and approach to support it directly(looking forward volunteer to send PR)

@pan3793
Thank you very much for the explanation.
Totally answered my question! 🙇

I got the same issue like you, but its not working when change some code to add username and password , I repleace the ranger-plugin-common-2.3.0.jar in spark classpath and then also got 401 ,but I test with curl like : curl -v -u admin:admin http://ip:6080/service/plugins/secure/policies/download/spark_service work fine.

I change the code and work fine.


there is no "override a container's labels on runtime"

File: config/labels-custom-metadata.md
looking into a way to change/update/add labels to running container
i bumped into this page with the link.
read the docs, and reading https://github.com/moby/moby/issues/21721 i concluded this
is not supported , so strange to see this link pointing to information  on how to set labels DURING CREATION , not how to set on a running container

The confusing part is "Overriding a container's labels at runtime" which would probably be easier to understand if "runtime" was changed to "docker run".


Allow pipeline uploads when no-command-eval is true

When disabling command-eval in the agent config we are unable to upload pipelines.
BUILDKITE_COMMAND=“buildkite-agent pipeline upload path/to/pipeline.yml”
Results in:

This agent is not allowed to evaluate console commands. To allow this, re-run this agent without the --no-command-eval option, or specify a script within your repository to run instead (such as scripts/test.sh).

Would it be possible to have pipelines in code and also the no-command-eval setting?

We're going to create a script at .buildkite/pipeline.sh that then just calls out to buildkite-agent pipeline upload


Lambda Functions Not Working in FlexBE Web UI

Title: Lambda Functions Not Working in FlexBE Web UI
Description:
I have been using ROS2 with the flexbe_app, and in that environment, lambda functions worked correctly. They also functioned as expected in ROS1. However, I am now using the flexbe_webui, and when I try to load behaviors that include lambda functions, I encounter several issues.
Steps to Reproduce:

Use flexbe_app with ROS2 to create a behavior that includes lambda functions.
Attempt to load the same behavior in flexbe_webui.

Expected Behavior:

The lambda functions should be correctly parsed and executed as they were in the flexbe_app environment for both ROS1 and ROS2.

Actual Behavior:

Initially, an error is thrown: TypeError: Cannot read properties of undefined (reading 'match').
If the lambda functions are fed as strings, the behaviors can be loaded, but the lambda functions do not work.

Example Case:
Here is a simple example demonstrating the issue:
In flexbe_app with ROS2:
OperatableStateMachine.add('IsOrder',
                           CheckConditionState(predicate=lambda x: x == 'sample-order'),
                           transitions={'true': 'SampleTimeout', 'false': 'CustomTimeout'},
                           autonomy={'true': Autonomy.Off, 'false': Autonomy.Off},
                           remapping={'input_value': 'order_type'})

When changing the lambda as a string it can load the states, and the behaviors, but the lambda function does not run as it should. This is how the code looks when feeding the lambda as a string:
OperatableStateMachine.add('IsOrder',
                           CheckConditionState(predicate="lambda x: x == 'sample-order'"),
                           transitions={'true': 'SampleTimeout', 'false': 'CustomTimeout'},
                           autonomy={'true': Autonomy.Off, 'false': Autonomy.Off},
                           remapping={'input_value': 'order_type'})

This issue only happens for webui.
Environment:

ROS2 Iron
FlexBE Web UI version: 0.0.0
OS: Ubuntu 22.04

I'll look into this tonight.  Thanks for the report.  Can you send me a screen shot of your entry box under the webui.

I believe this is fixed with the latest push to beta.  @RPRKLR please take a look and let me know.
We are trying to do more prevalidation on inputs, and provide notice of how types are interpreted (more in documentation coming this summer).  I have simplified the equation test, and it seems more flexible now.  Let me know if this is working for you now.

Does the lambda function needs to be loaded as a string?

No.   Type it in without quotes.

The lambdas are working, when I enter them without quotes. But when I save the behavior, when I try to load it, I still get the error that Code parsing failed: TypeError: Cannot read properties of undefinied (reading 'match')

The issue is that I have two types of lambda functions, it can load the first type, which looks like this:
lambda x : self.switch_off_timer

It can not load the lambda if it is checking a string, for example:
lambda x: x=='is_navigating'

Confirmed the issue.  I thought the issue was caused by something else, and in my late night testing I did not try to re-start and reload as I was focused on execution of behavior after saving.  I can confirm the error is happening on my end, but it will be tomorrow night before I have time to fully understand and fix.

Pushed change that addresses this issue in my tests.

Thanks for the support, I am still facing an issue. If the state only have one input, it works, but if the state have 2 inputs, like the FlexibleCheckConditionState, which have 2 inputs like predicate and input_keys,  get an error, which can be seen in the picture.

I tried to put lambdas with quotes and lambda functions without quotes like lambda x: x==self.is_waiting_for_service, and none of them work.

I have pushed some updates to flexbe_webui, and also flexbe_behavior_engine (ros2-devel) branch.  The BE changes include some tweaks to descriptions of parameters, and the webui includes some additional validation and clean up.
With these, I am able to use the the FlexibleCheckConditionState with complex lambdas.

I can create the FlexibleCheckConditionState, and I can set the parameters like on the picture:

And I can save and launch the behavior, and it works without a problem, but when I try to load the saved behavior, I get the following error:
The loaded behavior contains errors! Please fis and save:
parameter predicate of state /IfServiceWaiting has empty value
And also it can not load the transitions too, so it is not connected to other states.

@RPRKLR Please give the latest a try.

Now everything is working, as it should. Thank you for your support! The issue can be closed.

Great!  Thank you for your detailed error reports and helpful testing.  I appreciate the input.