Adding murmurhash3 digest to phobos

A proposal of addition of murmurhash3 to phobos.
Maybe it should go in experimental first. Please let me know what you think.

Thx for the comments @9il.
I still need to spend some time on the documentation.
I'll add some tests to check if there's an alignment issue when pushing consecutive blocks in the Piecewise struct.

LGTM from 30K feet. Any experts on board to vet this?

LGTM from 30K feet. Any experts on board to vet this?

MurmurHash implementation looks good. SMurmurHash3_XXX_XXx would be very useful for fast user defined hash functions (especially for containers or ndslice, where strides between elements can be greater than 1).

@9il I added the test about alignement you were worried about.
I still need to add some documentation.

I added the test about alignement you were worried about.

Your test/code works on X86 but it won't work on architectures which do not properly support unaligned loads. This includes older ARM devices so I'd say this is a blocker. Instead of this code:
hasher.putBlocks(cast(const(Block)[]) data[0 .. consecutiveBlocks]);

you'll have to copy n bytes into a block/ubyte union:
union
{
    Block block;
    ubyte[Block.sizeof] rawData;
} buf;

foreach(...)
{
    buf.rawData[] = data[n..n+Block.sizeof];
    hasher.putBlock(...);
}

If benchmarking shows that the copying makes the algorithm much slower you could do a runtime check whether data.ptr is properly aligned and then skip the copying.

Thx @jpf91 this is very helpful. I'll test different versions but I expect the performance to drop dramatically. According to the original C++ implementation: that's about 1.68 bytes per cycle, or about 9.5 cycles per 16-byte chunk. this allows to achieve 5GB/sec for MurmurHash3_x64_128.
The current implementation is not as fast but I achieved 4.3GB/sec (different computer so not necessarily comparable). I'll report here my findings but copying is not free so performance will suffer.
On an architecture note, the implementation is currently suboptimal for big-endian.
See comment on endianness on the wikipedia page. I need to fix this also.

I'll test different versions but I expect the performance to drop dramatically.

You can use version(X86) and others to determine if algorithm can use unaligned loads.

You can use version(X86) and others to determine if algorithm can use unaligned loads.

That certainly should be done. It would still be nice to have a fast-path for aligned data on architectures without unaligned loads as well. Here are some possible solutions:

Your put method isn't marked as safe or trusted. So you could probably argue that your function isn't memory safe and document that the programmer is responsible for passing only aligned data.

You can check data.ptr  and select the codepath for aligned or unaligned data at runtime. This will probably still have some performance impact (depending on the user code / data size passed to put). But you could combine this approach with version statements.

You could also add an put(const(uint)[]) overload. All data passed to this overload would be guaranteed to be aligned. But:

I'm not sure how well this actually works with range composition.
It doesn't work that well for the object-oriented interface. You could add it as an additional method but then it still couldn't be used with generic code using the Digest interface. Adding a new method to the interface could break user code. As it's safe to cast uint[] to ubyte[] the overload for aligned data could always pass the data to the overload for unaligned data. So we'd have to make Digest an abstract class instead of an interface and provide a default implementation.

Pinging some people who have worked on std.digest for more opinions: @schuetzm @CyberShadow @klickverbot

Your put method isn't marked as safe or trusted. So you could probably argue that your function isn't memory safe and document that the programmer is responsible for passing only aligned data.

I don't think @safe makes any alignment guarantees. But an in contract should be used here, if this solution is chosen.

You can check data.ptr and select the codepath for aligned or unaligned data at runtime. This will probably still have some performance impact (depending on the user code / data size passed to put). But you could combine this approach with version statements.

AFAIK even on x86, unaligned loads come with a performance penalty, although it's likely to be smaller for consecutive unaligned loads because of caching. Maybe provide a default implementation that can handle unaligned data, and an optimized one that requires aligned input?

I don't think @safe makes any alignment guarantees. But an in contract should be used here, if this solution is chosen.

Sorry, I should have added a more detailed explanation: Safe doesn't make any guarantees about alignment, but it guarantees memory safety. Unaligned loads can cause data and memory corruption on some architectures, therefore a function which might produce unaligned loads depending on input parameters could never be safe (or trusted). As this put function is not marked safe, it could be argued the API user is responsible for proper alignment (similar to the user being responsible for passing only zero terminated strings to C str functions). I don't think it's a good solution, but I think it's something that could be argued for. But you're right, an in contract is really necessary when using this approach.

AFAIK even on x86, unaligned loads come with a performance penalty, although it's likely to be smaller for consecutive unaligned loads because of caching.

That's correct, but articles on that always compare unaligned loads vs. aligned loads (of  pre-aligned data). But in this case we can't require pre-aligned data and the more interesting question is unaligned loads vs. first copying data to an aligned buffer + aligned loads. I haven't found any info on the performance of that. (In fact, a clever compiler might see that your just copying bytes to an aligned buffer and performing a load. So if the compiler concludes an unaligned load is faster it might actually rewrite the code.)
And as X86 uses the same instructions for unaligned vs. aligned loads IIRC there's shouldn't be any performance loss on X86 for aligned data when using the same codepath as for unaligned data. But this is just guessing. It's probably something that really needs benchmarks ;-)

Maybe provide a default implementation that can handle unaligned data, and an optimized one that requires aligned input?

That's basically my solution 3. I think it's the best solution but I don't know how well it works with range composition (e.g. is file.byChunk(1024) an aligned buffer? It probably needs to go through a map(a => cast(uint[])a stage, etc)).

Some numbers FYI based on code from this repo.
It's comparing the original C++ implementation with the optimized D version and finally the digest version with the copy for alignment. I tested with the 3 compilers (help me tweak compilation flags if necessary).
Looking at the fastest implementation MurmurHash3_x64_128:

For the optimized version: LDC runs at 93% of C++, GDC at 65%, DMD at 30%.
For the aligned version: LDC runs at 8% of C++, GDC at 13%, DMD at 13%.

DMD DMD64 D Compiler v2.069.2

 % make clean && make
rm -Rf build *.o
mkdir -p build
g++ -I/smasher/src -O3 -Wall -Werror -c smasher/src/MurmurHash3.cpp -o build/CMurmurHash3.o
dmd -O -inline -release benchmark.d murmurhash3.d build/CMurmurHash3.o -ofbuild/benchmark
./build/benchmark
Please wait while benchmarking MurmurHash3, running 4096*hash(256KiB) = 1GiB
C++ MurmurHash3_x64_128        - 19598 GiB/s
D SMurmurHash3_x64_128         - 5491 GiB/s
D digest MurmurHash3_x64_128   - 1561 GiB/s

C++ MurmurHash3_x86_128        - 12190 GiB/s
D SMurmurHash3_x86_128         - 3703 GiB/s
D digest MurmurHash3_x86_128   - 1352 GiB/s

C++ MurmurHash3_x86_32         - 9246 GiB/s
D SMurmurHash3_x86_32          - 3385 GiB/s
D digest MurmurHash3_x86_32    - 575 GiB/s

GDC gcc version 5.2.0 (crosstool-NG crosstool-ng-1.20.0-232-gc746732 - 20150830-2.066.1-dadb5a3784)

 % make clean && make
rm -Rf build *.o
mkdir -p build
g++ -I/smasher/src -O3 -Wall -Werror -c smasher/src/MurmurHash3.cpp -o build/CMurmurHash3.o
~/Downloads/gdc_x86_64-pc-linux-gnu/bin/gdc -O3 -frelease -Wall -Werror benchmark.d murmurhash3.d build/CMurmurHash3.o -obuild/benchmark
./build/benchmark
Please wait while benchmarking MurmurHash3, running 4096*hash(256KiB) = 1GiB
C++ MurmurHash3_x64_128        - 19787 GiB/s
D SMurmurHash3_x64_128         - 12840 GiB/s
D digest MurmurHash3_x64_128   - 2586 GiB/s

C++ MurmurHash3_x86_128        - 12047 GiB/s
D SMurmurHash3_x86_128         - 8000 GiB/s
D digest MurmurHash3_x86_128   - 2279 GiB/s

C++ MurmurHash3_x86_32         - 9288 GiB/s
D SMurmurHash3_x86_32          - 7394 GiB/s
D digest MurmurHash3_x86_32    - 705 GiB/s

LDC LDC - the LLVM D compiler (0.16.1): based on DMD v2.067.1 and LLVM 3.7.0

 % make clean && make
rm -Rf build *.o
mkdir -p build
g++ -I/smasher/src -O3 -Wall -Werror -c smasher/src/MurmurHash3.cpp -o build/CMurmurHash3.o
~/Downloads/ldc2-0.16.1-linux-x86_64/bin/ldc2 -O -release -inline benchmark.d murmurhash3.d build/CMurmurHash3.o -ofbuild/benchmark
./build/benchmark
Please wait while benchmarking MurmurHash3, running 4096*hash(256KiB) = 1GiB
C++ MurmurHash3_x64_128        - 18963 GiB/s
D SMurmurHash3_x64_128         - 17732 GiB/s
D digest MurmurHash3_x64_128   - 2609 GiB/s

C++ MurmurHash3_x86_128        - 12190 GiB/s
D SMurmurHash3_x86_128         - 10114 GiB/s
D digest MurmurHash3_x86_128   - 2346 GiB/s

C++ MurmurHash3_x86_32         - 9288 GiB/s
D SMurmurHash3_x86_32          - 8847 GiB/s
D digest MurmurHash3_x86_32    - 751 GiB/s

From your numbers, this should be 14%, right?

Good catch @klickverbot , I mismatch between LDC and DMD : )
For the aligned version: LDC runs at 13% of C++, GDC at 13%, DMD at 8%.

@gchatelet Can manual inlining of update and shufle improve performance? There is no reason why GDC is slower than G++, but inlining may be bad.

Use -fno-bounds-check and gdc is infinitely fast ;-) It seems without bounds checks all side effects in the benchmark gone and gdc removes the hasher.putBlocks and similar calls (That's what you get for marking everything pure ;-) Actually a quite nice result). So add something like this:
//dummy.d
extern(C) void foo(ubyte[])
{
}

//benchmark.d
extern(C) void foo(ubyte[]);

auto useHasher(H)() {
	H hasher;
	hasher.putBlocks(cast(const(H.Block)[])buffer);
	hasher.finalize();
	foo(hasher.getBytes()[]); //<=========================
	return hasher.getBytes();
}

$(BUILD_DIR)/benchmark: benchmark.d murmurhash3.d $(BUILD_DIR)/CMurmurHash3.o
	gdc dummy.d -c -o dummy.o
	gdc -O3 -frelease -Wall -Werror -fno-bounds-check dummy.o $^ -o$@

On my system results seem to be quite random for some reason, but -fno-bounds-check seems to improve performance for SMurmurHash3_*. So you might want to get rid of these boundchecks.
I'm disappointed the compiler can't optimize this though:
// gdc murmurhash3.d -fdump-tree-original -c
// cat murmurhash3.d.003t.original
;; Function putBlocks (_D11murmurhash320SMurmurHash3_x64_1289putBlocksMFNaNbNiNfMAxG2mXv)
;; enabled by -tree-original

{
  if (this != 0)
    {
      <<< Unknown tree: void_cst >>>
    }
  else
    {
      _d_assert_msg ({.length=9, .ptr="null this"}, {.length=13, .ptr="murmurhash3.d"}, 59);
    }
  {
    ulong block[2];
    ulong __key1526;
    struct const(ulong[2])[] __aggr1525;

    __aggr1525 = {.length=blocks.length, .ptr=blocks.ptr};
    __key1526 = 0;
    while (1)
      {
        if (!(__key1526 < __aggr1525.length)) break;
        block = *(__aggr1525.ptr + (__key1526 < __aggr1525.length ? __key1526 * 16 : _d_arraybounds ({.length=13, .ptr="murmurhash3.d"}, 61)));
        update ((ulong &) &this->h1, *(const ulong *) &block, this->h2,<PHONE_NUMBER>568883157,<PHONE_NUMBER>109919103, 31, 27,<PHONE_NUMBER>);
        update ((ulong &) &this->h2, *((const ulong *) &block + 8), this->h1,<PHONE_NUMBER>109919103,<PHONE_NUMBER>568883157, 33, 31, 944331445);
        __key1526 = __key1526 + 1;
      }
  }
  this->size = this->size + blocks.length * 16;
}

For the digest version it's definitely Piecewise.put which needs to be optimized. Changing put to only do hasher.putBlocks(cast(const(Block)[])data); for testing:
C++ MurmurHash3_x64_128        - 20898 GiB/s
D SMurmurHash3_x64_128         - 21904 GiB/s
D digest MurmurHash3_x64_128   - 21223 GiB/s

C++ MurmurHash3_x86_128        - 14949 GiB/s
D SMurmurHash3_x86_128         - 13128 GiB/s
D digest MurmurHash3_x86_128   - 15004 GiB/s

C++ MurmurHash3_x86_32         - 9225 GiB/s
D SMurmurHash3_x86_32          - 9288 GiB/s
D digest MurmurHash3_x86_32    - 9330 GiB/s

(As already said test results on my machine seem to be kinda weird...)

Unaligned loads can cause data and memory corruption on some architectures

I wasn't aware of that, I thought it would always trap.

Maybe provide a default implementation that can handle unaligned data, and an optimized one that requires aligned input?

That's basically my solution 3. I think it's the best solution but I don't know how well it works with range composition (e.g. is file.byChunk(1024) an aligned buffer? It probably needs to go through a map(a => cast(uint[])a stage, etc)).

I was thinking of MurmurHash3 vs MurmurHash3Aligned, both with the usual void put(scope const(ubyte)[]) methods.

Thx @jpf91 , I updated the benchmark but kept the unaligned version.
GDC even beats the C++ implementation (more inlining possibilities I guess)
 % make clean && make -j12
rm -Rf build *.o
mkdir -p build
g++ -I/smasher/src -O3 -Wall -Werror -c smasher/src/MurmurHash3.cpp -o build/CMurmurHash3.o
~/Downloads/gdc_x86_64-pc-linux-gnu/bin/gdc -O3 -frelease -fno-bounds-check -Wall -Werror benchmark.d murmurhash3.d build/CMurmurHash3.o -obuild/gdc_benchmark
~/Downloads/ldc2-0.16.1-linux-x86_64/bin/ldc2 -O5 -release -inline benchmark.d murmurhash3.d build/CMurmurHash3.o -ofbuild/ldc_benchmark
dmd -O -inline -release benchmark.d murmurhash3.d build/CMurmurHash3.o -ofbuild/dmd_benchmark
build/gdc_benchmark
Please wait while benchmarking MurmurHash3, running 4096*hash(256KiB) = 1GiB
C++ MurmurHash3_x64_128        -  94% - 18789 GiB/s
D SMurmurHash3_x64_128         - 100% - 19787 GiB/s
D digest MurmurHash3_x64_128   -  12% - 2535 GiB/s
C++ MurmurHash3_x86_128        -  61% - 12083 GiB/s
D SMurmurHash3_x86_128         -  63% - 12488 GiB/s
D digest MurmurHash3_x86_128   -  12% - 2377 GiB/s
C++ MurmurHash3_x86_32         -  46% - 9288 GiB/s
D SMurmurHash3_x86_32          -  45% - 8943 GiB/s
D digest MurmurHash3_x86_32    -   3% - 710 GiB/s
build/ldc_benchmark
Please wait while benchmarking MurmurHash3, running 4096*hash(256KiB) = 1GiB
C++ MurmurHash3_x64_128        - 100% - 19692 GiB/s
D SMurmurHash3_x64_128         -  89% - 17579 GiB/s
D digest MurmurHash3_x64_128   -  12% - 2368 GiB/s
C++ MurmurHash3_x86_128        -  60% - 12012 GiB/s
D SMurmurHash3_x86_128         -  51% - 10114 GiB/s
D digest MurmurHash3_x86_128   -  11% - 2173 GiB/s
C++ MurmurHash3_x86_32         -  47% - 9288 GiB/s
D SMurmurHash3_x86_32          -  44% - 8809 GiB/s
D digest MurmurHash3_x86_32    -   3% - 687 GiB/s
build/dmd_benchmark
Please wait while benchmarking MurmurHash3, running 4096*hash(256KiB) = 1GiB
C++ MurmurHash3_x64_128        - 100% - 19598 GiB/s
D SMurmurHash3_x64_128         -  27% - 5469 GiB/s
D digest MurmurHash3_x64_128   -   7% - 1560 GiB/s
C++ MurmurHash3_x86_128        -  62% - 12154 GiB/s
D SMurmurHash3_x86_128         -  18% - 3703 GiB/s
D digest MurmurHash3_x86_128   -   6% - 1371 GiB/s
C++ MurmurHash3_x86_32         -  47% - 9267 GiB/s
D SMurmurHash3_x86_32          -  17% - 3396 GiB/s
D digest MurmurHash3_x86_32    -   3% - 597 GiB/s

@schuetzm

Unaligned loads can cause data and memory corruption on some architectures

I wasn't aware of that, I thought it would always trap.

I had to debug such a problem for GDC once ;-) See for example: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0068b/Bcfihdhj.html

Address alignment for word transfers
In most circumstances, you must ensure that addresses for 32-bit transfers are 32-bit word-aligned.
[...]
If your system does not have a system coprocessor (cp15), or alignment checking is disabled:

For STR, the specified address is rounded down to a multiple of four. [effectively overwriting data at a lower address => data corruption]

The same behavior always happens on old ARMv5 chips. For loads it's more complicated. A load will not corrupt memory, but will load invalid data. If this data is then somehow used for memory address calculation you can still end up with corrupted memory.
If you want to read more about this, see http://www.heyrick.co.uk/armwiki/Unaligned_data_access and http://stackoverflow.com/a/16549366/471401 .

I was thinking of MurmurHash3 vs MurmurHash3Aligned, both with the usual void put(scope const(ubyte)[]) methods.

That might be a good idea. Some testing shows that on X86_64 alignment is less of a problem than data length though, and ubyte[] allows any data length. I think MurmurHash3Aligned with put(Block[]) is more useful. But then there's no big difference to SMurmurHash3_*.
@gchatelet

LDC and DMD would also probably perform better when using the noboundscheck option. However, for phobos the code will have to be compiled with boundschecks, so you'll have to change the code to workaround bounds checking code:

void putBlocks(scope const(Block)[] blocks...) pure nothrow @nogc @trusted
{
    const(Block)* start = blocks.ptr;
    const(Block)* end = blocks.ptr + blocks.length;

    for(auto ptr = start; ptr < end; ptr++)
    {
        update(h1, (*ptr)[0], h2, c1, c2, 31, 27, 0x52dce729);
        update(h2, (*ptr)[1], h1, c2, c1, 33, 31, 0x38495ab5);
    }
    size += blocks.length * Block.sizeof;
}

Aligned vs. unaligned data seems to perform more or less the same for me on x86_64 when using putBlocks
With GDC there's no slowdown on X86_64 when first reading into a buffer. This code is safe for architectures without unaligned loads. For DMD there's a 3% slowdown.

struct Piecewise(Hasher)
{
    union BufferUnion
    {
        Block block;
        ubyte[Block.sizeof] blockData;
    }
    BufferUnion buffer;

    void put(scope const(ubyte)[] data...) pure nothrow @trusted
    {
        // Buffer should never be full while entring this function.
        assert(bufferSize < Block.sizeof);
        const(ubyte)* start = data.ptr;

        size_t preprocessed = 0;
        /*
         * Do main work: process chunks of Block.sizeof bytes
         */
        size_t numBlocks = (data.length - preprocessed) / Block.sizeof;
        const(ubyte)* end = start + numBlocks * Block.sizeof;

        for(; start < end; start += Block.sizeof)
        {
            buffer.blockData = start[0 .. Block.sizeof];
            hasher.putBlock(buffer.block);
        }
        hasher.size += numBlocks * Block.sizeof;
    }
}

struct SMurmurHash3_x64_128
{
    void putBlock(scope const Block block) pure nothrow @nogc
    {
        update(h1, block[0], h2, c1, c2, 31, 27, 0x52dce729);
        update(h2, block[1], h1, c2, c1, 33, 31, 0x38495ab5);
    }
}

There is a 10-20% slowdown when supporting data buffers with lengths which are not multiples of the block size. Here's my code: Note: This code is untested and needs proof-reading as well as many tests, especially for corner cases. (data.length < Block.sizeof, data.length == Block.sizeof, ...). The if (data.length + bufferSize < Block.sizeof) statement seems to be the main slowdown source.

struct Piecewise(Hasher)
{
    union BufferUnion
    {
        Block block;
        ubyte[Block.sizeof] blockData;
    }
    BufferUnion buffer;

    void put(scope const(ubyte)[] data...) pure nothrow @trusted
    {
        // Buffer should never be full while entring this function.
        assert(bufferSize < Block.sizeof);
        const(ubyte)* start = data.ptr;

        /*
         * Check if we have some leftover data in the buffer.
         * Then fill the first block buffer.
         */
        if (data.length + bufferSize < Block.sizeof)
        {
            buffer.blockData[bufferSize .. $] = start[0 .. data.length];
            bufferSize += data.length;
            return;
        }
        auto preprocessed = Block.sizeof - bufferSize;
        buffer.blockData[bufferSize .. $] = start[0 .. preprocessed];
        hasher.putBlock(buffer.block);
        start += preprocessed;

        /*
         * Do main work: process chunks of Block.sizeof bytes
         */
        size_t numBlocks = (data.length - preprocessed) / Block.sizeof;
        const(ubyte)* end = start + numBlocks * Block.sizeof;

        for(; start < end; start += Block.sizeof)
        {
            buffer.blockData = start[0 .. Block.sizeof];
            hasher.putBlock(buffer.block);
        }
        // +1 for preprocessed Block
        hasher.size += (numBlocks + 1) * Block.sizeof;

        /**
         * Now add remaining data to buffer
         */
        bufferSize = data.length - preprocessed - numBlocks * Block.sizeof;
        buffer.blockData[0 .. bufferSize] = end[0 .. bufferSize];
    }
}

struct SMurmurHash3_x64_128
{
    void putBlock(scope const Block block) pure nothrow @nogc
    {
        update(h1, block[0], h2, c1, c2, 31, 27, 0x52dce729);
        update(h2, block[1], h1, c2, c1, 33, 31, 0x38495ab5);
    }
}

At least for X86_64 we can ignore the effects of unaligned data and the overhead caused by copying. So the copying version which will be safe for all architectures has no negative effects on X86_64. A much bigger problem is processing data buffers with lengths which are not multiples of the Blocksize. So maybe a MurmurHash3Aligned digest which only accepts put(ulong[Block.sizeof/8][]) is a good idea for those use cases where we have proper data sizes. (I'm not actually sure what type we should use for that. ubyte[n] would be nice, but ubyte[n].alignof == 1 so that's not what we really want)

Here are ARM results for ARMv5 (no unaligned load support) with linux kernel configured to fix up these accesses (very slow):

C++ MurmurHash3_x64_128, aligned data -  97% - 529 GiB/s
C++ MurmurHash3_x64_128, unaligned data -   6% - 33 GiB/s
D SMurmurHash3_x64_128, aligned data - 100% - 544 GiB/s
D SMurmurHash3_x64_128, unaligned data -   5% - 33 GiB/s
D digest MurmurHash3_x64_128, aligned data -  57% - 311 GiB/s
D digest MurmurHash3_x64_128, unaligned data -  53% - 293 GiB/s

Please note that the C++ MurmurHash3_x64_128 and SMurmurHash3_x64_128 generate invalid results for unaligned data with the standard linux kernel configuration. The digest version works fine. So an overload for a fast version for aligned data is even more important for ARM. I couldn't find a specific reason for the 40% slowdown for the digest version. Even simply forwarding the data by calling putBlocks is 10% slower. GCC probably can't optimize ARM code as well as X86_64 code.
My conclusion would be to use the unaligned-data-safe but slightly slower approach I've posted above for the default MurmurHash3_x86_32 std.digest digests. Then either introduce MurmurHash3_x86_32Aligned with a put function which only takes a Block[] parameter. Or even better: maybe make the SMurmurHash3_* structs compatible with the digest API (alias put => putBlocks, add start and finish functions).

Here are the numbers with the latest version. I used @jpf91 's suggestions and added more tests. dmd is still super slow but ldc and gdc are now as fast as the C++ version with bounds-checking. Also ldc is now benchmarked against C++ code compiled with clang++ to be fair. Last thing I fixed the benchmark's results to display correct throughput (should have been MiB instead of GiB).
I'm now working on improving the documentation, stay tuned !
Feedback is still welcome of course :)
 % make clean && make -j12
rm -Rf build *.o
mkdir -p build
clang++ -I/smasher/src -O3 -Wall -Werror -c smasher/src/MurmurHash3.cpp -o build/CMurmurHash3_clang.o
g++ -I/smasher/src -O3 -Wall -Werror -c smasher/src/MurmurHash3.cpp -o build/CMurmurHash3_g++.o
dmd -O -release -inline benchmark.d murmurhash3.d build/CMurmurHash3_g++.o -ofbuild/dmd_benchmark
~/Downloads/gdc_x86_64-pc-linux-gnu/bin/gdc -O3 -frelease -Wall -Werror -march=native benchmark.d murmurhash3.d build/CMurmurHash3_g++.o -obuild/gdc_benchmark
~/Downloads/ldc2-0.16.1-linux-x86_64/bin/ldc2 -O3 -release -inline benchmark.d murmurhash3.d build/CMurmurHash3_clang.o -ofbuild/ldc_benchmark
build/gdc_benchmark
Please wait while benchmarking MurmurHash3, running 4096*hash(256KiB) = 1GiB
x64_128 C++          - 100% - 19.51 GiB/s
x64_128 D            - 105% - 20.49 GiB/s
x64_128 D digest     -  89% - 17.53 GiB/s

x86_128 C++          - 100% - 12.02 GiB/s
x86_128 D            -  93% - 11.31 GiB/s
x86_128 D digest     -  89% - 10.75 GiB/s

x86_32 C++           - 100% - 9.26 GiB/s
x86_32 D             -  97% - 9.04 GiB/s
x86_32 D digest      - 102% - 9.46 GiB/s

build/ldc_benchmark
Please wait while benchmarking MurmurHash3, running 4096*hash(256KiB) = 1GiB
x64_128 C++          - 100% - 15.57 GiB/s
x64_128 D            - 113% - 17.64 GiB/s
x64_128 D digest     -  92% - 14.32 GiB/s

x86_128 C++          - 100% - 9.14 GiB/s
x86_128 D            - 110% - 10.11 GiB/s
x86_128 D digest     -  97% - 8.89 GiB/s

x86_32 C++           - 100% - 8.79 GiB/s
x86_32 D             -  99% - 8.74 GiB/s
x86_32 D digest      - 104% - 9.20 GiB/s

build/dmd_benchmark
Please wait while benchmarking MurmurHash3, running 4096*hash(256KiB) = 1GiB
x64_128 C++          - 100% - 19.74 GiB/s
x64_128 D            -  23% - 4.59 GiB/s
x64_128 D digest     -  23% - 4.66 GiB/s

x86_128 C++          - 100% - 12.15 GiB/s
x86_128 D            -  25% - 3.04 GiB/s
x86_128 D digest     -  25% - 3.06 GiB/s

x86_32 C++           - 100% - 9.28 GiB/s
x86_32 D             -  36% - 3.37 GiB/s
x86_32 D digest      -  36% - 3.36 GiB/s

@gchatelet That looks great! Thanks for incorporating the suggestions and for contributing this module.
Some small comments:

Maybe add a comment like this to putBlocks and Piecewise.put:

/*
 * Use pointer manipulation instead of foreach to avoid bounds checking (see @@@BUG@@@ 15581).
 * This function heavily affects the performance of [Piecewise/hash calculation] so make sure to
 * benchmark all changes!
 */

Maybe rename the module to murmurhash instead of murmurhash3. All other modules do not include the algorithm number. (If we add murmurhash2 at some point we could still put these into extra files and use package.d for std.digest.murmurhash).
You need to have another look at the windows Makefiles and add murmurhash.d to some more rules. Just search for crc.d and add the same rules.

I did a quick review and apart from these small issues and the missing documentation this module definitely gets a :+1: from me. The code looks quite nice. The checkResult unittest helper could be useful for other std.digest modules. Maybe also use the dmd -cov coverage check and remember to squash your commits when this pull request is ready for merging.

Can someone have a look on the last version of the documentation and give some feedback?
Also I've been told to squash the history to produce a single commit. Since a rebase with squashing rewrites the history I needed to push with --force which seems to confuse the auto-tester (search for 1d60afc on this page to pin down the rewritten commit). Was it the expected way of doing it? Or shall I create a new pull request when everything is good?
Finally I'd need a permission to merge if everything looks good.

seems to confuse the auto-tester

We don't worry about the results on old commits.  As long as https://auto-tester.puremagic.com/pull-history.ghtml?projectid=1&repoid=3&pullid=3916 is showing the right latest commit.
If you fetch and rebase on top of upstream/master (or edit the first commit) those commit hashes will all change and the old results will disappear anyway.

Was it the expected way of doing it?

Yes.

Or shall I create a new pull request when everything is good?

No, rebasing is correct.

I think you can move the
Implementation uses pointer manipulation instead of foreach to avoid bounds checking (see @@@BUG@@@ 15581) making the function @trusted instead of @safe. This function heavily affects the performance of hash calculation so make sure to benchmark all changes. Benchmark can be found on github.

documentation into a normal comment, it doesn't really need to appear in DDOC.
And the putBlock, putRemainder, finalize and get methods of the SMurmurHash3_* structs should be documented (a single summary sentence per function). Otherwise LGTM :+1:

Thx everyone for the advices. I think this is good to go.

Ping :)

Looks great! Ping @9il @jpf91 @schuetzm

Since D-Programming-Language/dmd#5470 is in I intend to simplify the code to have everything @safe.
I started but got side tracked I'll push a new version later today.

Some questions:

Why are the structs called SMurmurHash_... instead of just MurmurHash_...?
There should be an alias called (S)MurmurHash that is defined as the appropriate implementation for the current architecture.
If Piecewise is "not intended to be used directly" as the documentation says, it should be private.

Erf sorry I completely mess up the pull request :(
Let me try to fix this.

@schuetzm

Why are the structs called SMurmurHash_... instead of just MurmurHash_...?

SMurmurHash_ is the low level struct used by people knowing what they're doing.
MurmurHash_ is an alias to Piecewise!SMurmurHash_ it allows to push data that is not a multiple of block size and is suitable for use by the digest library.
I'm happy to change the naming to something better, suggestions welcome.

There should be an alias called (S)MurmurHash that is defined as the appropriate implementation for the current architecture.

I'm not sure to understand what you mean. Please see my comment above.

If Piecewise is "not intended to be used directly" as the documentation says, it should be private.

It is private in the sense that it can be instantiated only with types in the same module but it has to be public to be used by the digest library.

I was going by the documentation: http://dtest.thecybershadow.net/artifact/website-c30f31be957f108026d0684c62a1cda24b5d949f-fa589e0c6067cac4be3e0bffc3f9769b/web/phobos-prerelease/std_digest_murmurhash.html
It lists many architecture specific MurmurHash3_x86/x64_xyz variants, but it doesn't mention a generic version that can be used on all architectures. I don't think anything architecture specific should be exposed at all, what would be the use? It's only an implementation detail. Instead, there should be MurmurHash3_32 and MurmurHash3_128 aliases pointing to the right implementation.
As for Piecewise, are you sure it has to be public? In other words, did you try to make it private? If that doesn't work, I'd consider that a compiler bug. In any case, if it isn't supposed to be used outside the module, there should be no (ddoc) documentation.

Soooo...

Why S as a prefix to structs? We don't do that.
As a putative user, how do I compute murmur hash portably? All I see is names with _x86 in them so I worry it's not portable to other architectures.

I'd just expect some struct MurmurHash(uint size) with size restricted to 32, 64, or 128.

As a putative user, how do I compute murmur hash portably? All I see is names with _x86 in them so I worry it's not portable to other architectures.

Each variant would work on any architecture. This historical suffix, which refers to optimization pattern. Maybe we can change name to _opt32bit and _opt64bit

I'm abroad with very low connectivity until April 10.

Sure we can get rid of the S prefix. What do you suggest for naming to differentiate between the low level API and the digest API? I can't come up with a good name.
I would stick with the original naming since each variant produce different results but are optimized for different platforms. The JavaScript implementation as the same names for instance. We can still provide an alias that select the best version according to the platform but then people might expect the same result on x32 and x86_64 which will not be the case. Thoughts ?

We can still provide an alias that select the best version according to the platform but then people might expect the same result on x32 and x86_64 which will not be the case. Thoughts ?

I think if we provide such an alias, we have to make it very obvious that it produces different hashes on different systems. UnportableMurmurHash3 or LocalMurmurHash3 or something like that.

Rebase please

@9il just rebased.
FYI I'll be at DConf2016, it might be easier to discuss the last points of this PR IRL.

Please take another look.
I updated the documentation, changed the naming of the structure and improved the performance of DMD a bit by returning by value in update(T)(...) instead of passing by ref.
Let me know if there's anything else I can do to improve this PR.

LGTM! Thank you!

please add changelog info and update index

About that x86/x64 distinction: The Wikipedia page calls these variants MurmurHash64A and MurmurHash64B. I would much prefer these names. There's really nothing x86(_64) specific about them.
Also, the documentation says "MurmurHash3 optimized for x86 processors". Is this true? Or is it really optimized for 32bit architectures in general, rather than x86 specifically? The source code looks completely generic, so I suspect it's the latter. Please adjust the documentation accordingly.

Also, the documentation says "MurmurHash3 optimized for x86 processors". Is this true? Or is it really optimized for 32bit architectures in general, rather than x86 specifically? The source code looks completely generic, so I suspect it's the latter. Please adjust the documentation accordingly.
Hide all checks

Yes, implementation is generic. MurmurHash3 was optimized for x86 (A for 32 bit and B for 64 bit) as an algorithm.

@9il can you point me to relevant documentation for changelog info and update index ?
@schuetzm , as @aappleby mentioned I think we should have a generic test suite for compliance and speed for all the hashes in std.digest. It would be nice to track performance for algorithm/compiler/architecture.

@9il can you point me to relevant documentation for changelog info and update index ?

https://github.com/dlang/phobos/blob/master/changelog.dd
It's a bit tricky, but you need to ensure that the RELATIVE_LINK2 name that you add is the same - it will soon get easier to add stuff to the changelog.
I couldn't find an index page for digest - @9il?

think we should have a generic test suite for compliance and speed for all the hashes in std.digest. It would be nice to track performance for algorithm/compiler/architecture.

Or better: for entire Phobos ;-)

Or better: for entire Phobos ;-)

@wilzbach totally agree. We should provide a way to spot phobos performance regressions. But that's for a separate PR :-)

@9il can you point me to relevant documentation for changelog info and update index ?
https://github.com/dlang/phobos/blob/master/changelog.dd
It's a bit tricky, but you need to ensure that the RELATIVE_LINK2 name that you add is the same - it will soon get easier to add stuff to the changelog.
I couldn't find an index page for digest - @9il?

https://github.com/dlang/phobos/blob/master/index.d

Almost there. Before moving forward I would like to make sure the naming is correct.
Marc pointed out the x86/x64 part was confusing. I added some documentation about that but still.
The A and B variants are only for MurmurHash2 IIUC.
How about the following?

MurmurHash3_32
MurmurHash3_128
MurmurHash3_128_opt_x64

Suggestions are welcome.
I think the test suite / benchmark from SMHasher should be integrated part of std.digest so it should be a different PR.

Yes, x86 must go.

Well, almost. It still says x64, but isn't specific to x64. There are tons of other 64-bit architectures. And for symmetry reasons, maybe rename the 32-bit optimized version to MurmurHash3_128_opt_32, too?

MurmurHash3_32
MurmurHash3_128
MurmurHash3_128_opt_x64

MurmurHash3_32_opt32
MurmurHash3_128_opt32
MurmurHash3_128_opt64

@andralex can you take a final look?

@andralex can you take a final look?

You can't reach Andrei via Github (he get's way too many notifications), write him a short mail to ping him ;-)

Thanks for the email. Will get on this soon.

I added a few nits, and have a high-level question. So we have separate structures defined for "optimized for 32 bits systems" and "optimized for 64 bits systems". Why aren't those hidden and selected automatically depending on the platform? Would someone be interested in running the opt64 thing on a 32-bit system or vice versa?

Naming-wise, I think we should go with one only:
struct MurmurHash(uint size /* 32, 64, or 128 */, uint opt = size_t.sizeof == 8 ? 64 : 32);

Would someone be interested in running the opt64 thing on a 32-bit system or vice versa?

Yes, this is common for distributed databases.

@9il thanks, cool. Then it seems the template with the defaulted second argument is appropriate.

I posted a new version based on @andralex suggestions. It's more compact and clearer.
One drawback is that we can't distinguish between MurmurHash2 32 bit (unimplemented) and MurmurHash3 32 bit.
Maybe we just don't care...
Let's just hope there won't be a MurmurHash4 : )
Destroy!

The name should still be MurmurHash3, because as you said MurmurHash2 is a thing, too. The module can still be called murmurhash without a version number.

I have ideas for a MurmurHash4, but I don't know if I'll ever publish it. :)
On Fri, May 27, 2016 at 2:57 AM, Guillaume Chatelet <
<EMAIL_ADDRESS>wrote:

I posted a new version based on @andralex https://github.com/andralex
suggestions. It's more compact and clearer.
One drawback is that we can't distinguish between MurmurHash2 32 bit
(unimplemented) and MurmurHash3 32 bit.
Maybe we just don't care...
Let's just hope there won't be a MurmurHash4 : )
Destroy!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dlang/phobos/pull/3916#issuecomment-222107999, or mute
the thread
https://github.com/notifications/unsubscribe/AAvgn0V3BGiI-t5WE78tvlhmBgzF0WpHks5qFsAQgaJpZM4HB6m-
.

Changed MurmurHash to MurmurHash3, fixed the documentation rebased + squashed history.
@aappleby good to know :) I'm curious now!

@gchatelet please ignore my nitpicks - if we want we can still improve docs afterwards.
So can we go with murmurhash or does it need to go through the std.experimental hop?

@wilzbach about experimental: since the interface is already standardized I don't think it makes sense.
I'd just go for it and improve. This PR has been initiated 6 months ago, let's merge and iterate.
I don't see any show stopper any more.

Auto-merge toggled on

PR for Slice to use MurmurHash3 #4639


Fullscreen only a window without border on windows

I think fullscreen should fill the whole window! It shouldn't just be a normal window without borders, but it should fill the whole screen even if the resolution of the game is smaller than the resolution of the monitor -> fullscreen.
On Linux the fullscreen mode with the game's res smaller than the monitors is "okay" because it makes the rest of the monitor black.
On Mac I don't know the behavior of this (no Mac available :-()
Couldn't this be done by the OpenGL Window Context correctly?

This is only when having really small sizes.. smaller than 800x600 so I don't consider this to be a real issue..


Fix radio command sound.

fixes issue with Reporting In command having the wrong sound

Looks good, thanks!


[dev] Fix for blank screen from push notifications on iOS 11 #trivial

This should make sure forceSetViewController isn't being called so many times. I would really like some time to look at how our ARTopMenuViewController does routing, because this particular method is called so many times when it seems like it should only happen once, and it's not entirely clear why. For this sprint though, this fixes the blank screen issue and should go into the next release.
Fixes https://artsyproduct.atlassian.net/browse/IN-93
Related: https://artsyproduct.atlassian.net/browse/IN-40

          1 Error

:no_entry_sign:
Developer Specific file shouldn’t be changed

  Generated by :no_entry_sign: Danger

@orta  +1 to the Danger rule about not changing developer-specific files!


Issue with Microsoft Graph on-behalf-of user

Issue

Please provide us with the following information:

Following code
myMSALObj.loginPopup(loginRequest)
        .then(handleResponse)
        .catch(error => {
            console.error(error);
        });

throws error
ServerError: invalid_client: 7000218 - [2021-04-19 18:09:38Z]: AADSTS7000218: The request body must contain the following parameter: 'client_assertion' or 'client_secret'.
Trace ID: <trace-id>

    - [ ] 1-1) Sign-in with Azure AD
    - [ ] 1-2) Sign-in with Azure AD B2C
    - [ ] 2-1) Acquire a Token and call Microsoft Graph
    - [ ] 3-1) Protect and call a web API on Azure AD
    - [ ] 3-2) Protect and call a web API on Azure AD B2C
    - [x] 4-1) Call a web API that calls Microsoft Graph on-behalf-of user
    - [ ] 4-2) Call a web API that calls another web API on-behalf-of user
    - [ ]   5) Deploy to Azure Storage and App Service

This issue is for a

    - [x] bug report -> please search issues before submitting
    - [ ] question
    - [ ] feature request
    - [ ] documentation issue or request

Minimal steps to reproduce

Any log messages given by the failure

Expected/desired behavior

Library version

Browser and version

Chrome, Edge, Firefox, Safari?

Mention any other details that might be useful

Thanks! We'll be in touch soon.

@pra7eek can you share the your configuration? (masking client secret, if any). In particular, double check to make sure your SPA redirect URI is registered as Single page application (on Authentication blade):

Hi @derisen , thanks for the response. When I added the Redirect URI while creating the app, by default it was assigned to the 'Web' type of application. After seeing your screenshot I changed it to the SPA app, and now I'm able to run the example.

@pra7eek cheers! Closing this.


white background on dark mode with embed

Hi, is it possible to remove/fix the white background when using embed in dark mode?
thank you

SELECT 'card' AS component, 1 AS columns;
SELECT
    'https://player.vimeo.com/video/951425265?h=debe29aecf' AS embed,
    'autoplay; fullscreen; picture-in-picture; clipboard-write' AS allow,
    'iframe' AS embed_mode,
    '700' AS height;

Hello ! The white background you can see is displayed by vimeo, not by SQLPage.
Vimeo offers the option to make this background black if you want: https://help.vimeo.com/hc/en-us/articles/12426260232977-Player-parameters-overview
select 'shell' as component, 'dark' as theme;
SELECT 'card' AS component, 1 AS columns;
SELECT
    'https://player.vimeo.com/video/951425265?transparent=0' AS embed,
    'autoplay; fullscreen; picture-in-picture; clipboard-write' AS allow,
    'iframe' AS embed_mode,
    '700' AS height;

Thanks, it worked with transparent=0
SELECT
    'https://player.vimeo.com/video/951425265?transparent=0' AS embed,
    'autoplay; fullscreen; picture-in-picture; clipboard-write;' AS allow,
    'iframe' AS embed_mode,
    '700' AS height;

is it possible to pass the style="background-color: #FFFFFF" ?
Example embed code:
<iframe src="https://player.vimeo.com/video/951425265" style="background-color: #FFFFFF" width="1000" height="400" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>


Enabling 'Multi value headers' gives 502 by ALB

Description
In our use case, we configured ALB by pointing lambda as target. This lambda is deployed with AWS Lambda Web Adapter to run http service.
In one of our API endpoint, we had a requirement on query parameters with multiple values for the same key. It is working as expected when the web application is deployed locally for testing.
What is the problem?
When we deployed the same to AWS environment using AWS Lambda Web Adapter. We observe that the application logic can get only last value of the query parameter.
To resolve this, we come to know about this: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html#multi-value-headers
By default 'Multi value headers' is disabled. Based on this, we enabled 'Multi value headers' in the target group.
After this configuration, we get '502 Bad Gateway' as response for every API requests even without multiple values for the same key.
In the logs, we see following error:
thread 'main' panicked at 'failed to build request: http::Error(InvalidUri(InvalidFormat))', /root/.cargo/git/checkouts/aws-lambda-rust-runtime-bf4c93ddbea2f260/9ee217c/lambda-http/src/request.rs:230:10

@ashwinspg what's the LWA version you have this issue? Could you share one sample request as well?

This issue is associated with LWA version: 0.3.2.
Enabling 'Multi value headers' in target group of AWS console and using LWA itself is creating a problem (throwing 502) irrespective of with/without multi value query string parameter.
@bnusunny

0.3.2 is an old version. I tested with the latest version 0.8.3. It does not throw 502, but does not support "Multi value headers". I need to check the lambda-http crate in Lambda Rust Runtime. LWA uses that crate to convert incoming lambda invokes into http requests. We probably need to add "Multi value headers" support in lambda-http.

Updating the version to 0.8.3 resolves the problem.
Thank you for the support 👍
@bnusunny


running_in_mac_app() function doesn't work

I want to be able to run spyder as a .app under OSX to be able to "open with" from the finder.
But I don't want to bundle everything in an app, as I want to be able to organize my modules with brew and pip.
Therefore I created a minimal app bundle and copied the spyder executable inside.
My problem is that the function running_in_mac_app() doesn't detect that I am running an app.
the __file__ variable in https://github.com/spyder-ide/spyder/blob/master/spyderlib/config/base.py#L397 has a value of "/usr/local/lib/python3.5/site-packages/spyderlib/config/base.py"
(so no Spyder.app)
I therefore had to change the following two lines to sys.platform == "darwin" to make my app work:
https://github.com/spyder-ide/spyder/blob/master/spyderlib/app/spyder.py#L3054
https://github.com/spyder-ide/spyder/blob/master/spyderlib/utils/qthelpers.py#L73
While I understand that my use case is probably not representative of the majority of the users, is that something you would consider modifying?
I tried dropping the __file__ condition but then the OSX 10.7 hack makes the app crash:
https://github.com/spyder-ide/spyder/blob/master/spyderlib/app/spyder.py#L1246

Sorry, we build app bundles ourselves following these instructions:
https://github.com/spyder-ide/mac-application
and running_in_mac_app() works exactly for our use case.

I'd say you should follow those instructions too, but don't add to the app all the additional modules that we do (like pandas and matplotlib)

Closing because we don't plan to do anything about this.
@impact27, you can easily patch Spyder to have the behavior that you want :-)


Trigger typeahead manually? Is that possible?

In my app the user may send a request for his position. When the position information is retrieved the data is populated to an input field which has the typeahead directive
document.getElementById('city').value = input;

But I want the typeahead to start and to search if the city is available in the typeahead resource. Is this possible?

focus event was a overhead for me, and i used click event - because focus occurred also after selection..
(i also overloaded the 'typeahead' directive itself)
app.directive('typeahead', function () {
return {
require: 'ngModel',
link: function (scope, element, attr, ctrl) {
element.bind('click', function () {
ctrl.$setViewValue();
});
}
};
});

focus event was a overhead for me, and i used click event - because focus occurred also after selection..
(i also overloaded the 'typeahead' directive itself)
app.directive('typeahead', function () {
    return {
        require: 'ngModel',
        link: function (scope, element, attr, ctrl) {
            element.bind('click', function () {
                ctrl.$setViewValue();
            });
        }
    };
});

I tried above but could not make it work

For everyone having the same issue, the following example works for me.
angular.module('app').directive('typeahead', typeaheadWatchChanges);

  typeaheadWatchChanges.$inject = ["$timeout"];

  function typeaheadWatchChanges($timeout) {
    return {
      require: "ngModel",
      link: function (scope, element, attr, controller) {
        scope.$watch('vm.model', function (value) {
          if (value) {
            element.focus();
            var old = controller.$viewValue;
            controller.$setViewValue('');
            $timeout(function () {
              controller.$setViewValue(old);
            });
          }
        });
      }
    };
  }

I finally got this working after quite a bit of hassle with the aforementioned proposed solutions.  These were the scenarios that I had to have this working for:

user clicks inside input to trigger opening of dropdown for:

existing values on model
when model is undefined/blank
when existing model value is a number

Sorry this is in Coffeescript.... should be self explanatory however.
.directive 'typeaheadClickOpen', ($parse, $timeout)->
    dir =
        restrict: 'A'
        require: 'ngModel'
        link: ($scope, elem, attrs)->

            triggerFunc = (evt)->
                ctrl = elem.controller 'ngModel'
                prev = ctrl.$modelValue || ''
                ctrl.$setViewValue if prev then '' else ' '
                if prev then $timeout -> ctrl.$setViewValue "#{prev}"

            elem.bind 'click', triggerFunc

In order to trigger the dropdown when the value was blank/undefined I set to an empty space ' ' as the UI renders blank/undefined as '' already.  It's not ideal because ' ' != '' and not sure how that would present in validation cases.
Another thing to note is for those model values that were numbers, I had to cast the prev values as a string ergo the "#{prev}" bit on the timeout.
It's disappointing to see such a huge need for exposing an API of sorts to trigger the dropdown and after nearly 2 years there is still no library solution.

@jusopi thanks worked for me :+1:

@jusopi thanks for sharing!
And you know, this is open-source, and when people like you do not send a pull request with a proposed solution, then there won't be no library solution :dancer:

Here is decaffeinated directive by @jusopi just in case.
Tested on Angular 1.4.3 / UI Bootstrap 0.13.0
  .directive('typeaheadClickOpen', function($parse, $timeout) {
      return {
          restrict: 'A',
          require: 'ngModel',
          link: function($scope, elem, attrs) {
              triggerFunc = function(evt) {
                  var ctrl = elem.controller('ngModel'),
                      prev = ctrl.$modelValue || '';
                  if (prev) {
                      ctrl.$setViewValue('');
                      $timeout(function() {
                          ctrl.$setViewValue(prev);
                      });
                  } else {
                      ctrl.$setViewValue(' ');
                  }
              }
              elem.bind('click', triggerFunc);
          }
      }
  })

@kennethlynne I didn't consider sending a pull request because the issue is closed as of Aug 5th 2013.  That and the fact that there are a few other similar feature requests that have been largely ignored by the project owner(s), didn't figure it worth the effort.

@kazeno Thank you.
Your solution saved my time.

@kazeno thanks, nice solution. I'd also add this to the condition element.val().length && !element.is(':focus') so that if the input is already focused the fetch won't happen again.

@jusopi  Got one issue in IE, when used above directive, its being opened but when clicked on the item its doesnt select anything.

@jbmyid the project I developed my solution for does not support IE at all.  We're targeting Chrome and FF exclusively.  Sorry.

@jusopi np Thanks, we were also not supporting just started now so got the issue :)

Hi All,
I want to use Typeahead functionality to get ajax call on time interval but I do not want to show list of items's pop up. How can I prevent to show that popup?

@piyush-dholariya this is something you should ask on stackoverflow, not in the repo's issues.

Having http://stackoverflow.com/questions/24764802/angular-js-automatically-focus-input-and-show-typeahead-dropdown-ui-bootstra as a source of inspiration I created the following directive:
`
function typeaheadShowOnFocus() {
return {
require: 'ngModel',
link: function($scope, element, attrs, ctrl) {
element.bind('focus', function() {
ctrl.$setViewValue();
$(element).trigger('input');
$(element).trigger('change');
});
ctrl.$parsers.unshift(function(inputValue) {
var value = (inputValue ? inputValue : '');
ctrl.$viewValue = value;
          return value;
        });
        ctrl.$parsers.push(function(inputValue) {
          return inputValue === '' ? '' : inputValue;
        });
      }
    };
  }

`
I hope it helps someone

@jusopi Thanks for the directive, it works for me.

@robbse Thx for your code, it works for me.

The following works fine:
// Notice $element and $timeout is injected.
function promptTypeAhead(id, value) {
var element = document.getElementById(id);
$timeout(function () {
var control = $element.find(element).controller('ngModel');
control.$setViewValue(''); // Replicate an on change event
control.$setViewValue(value);
control.$render();
}, 0);
}
Credit: @kennethlynne

Hi - so i am using this to open the typeahead on click of a button (latest angular bootstrap ui)
Here's my Test Scenario:

select a value from drop down
Click on the button
Typehead opens up with the filtered value.

Problem is that to focus on dropdown..we are resetting the selected value to '' in setviewvalue..and that removed the selected value. is there a way around it?
Also, if we set the value to 'prev' the dropdown applies the filter and only show the selected value in dropdown.. Is there a way to get away from filter when the button is clicked?
but when the dropdown opens up it applies the currently selected
.directive('typeaheadClickOpen', function($parse, $timeout) {
return {
restrict: 'A',
require: 'ngModel',
link: function($scope, elem, attrs) {
triggerFunc = function(evt) {
var ctrl = elem.controller('ngModel'),
prev = ctrl.$modelValue || '';
if (prev) {
ctrl.$setViewValue('');
$timeout(function() {
ctrl.$setViewValue(prev);
});
} else {
ctrl.$setViewValue(' ');
}
angular.element('#inputId').trigger('focus');
}
elem.bind('click', triggerFunc);
}
}
})


Pattern not found on injection.

Since around September 11th multiple people have experience a "Pattern not found" error on injection, I'm not really sure what causes it, but restarting your PC usually fixes the issue entirely.

We could try to dump the module if the signature wasn't found. Maybe this will help in finding out what went wrong.

i can confirm

same

We could try to dump the module if the signature wasn't found. Maybe this will help in finding out what went wrong.

since we don't know how to recreate it yet this would either rely on user submission or us getting it out of luck, im not opposed to the idea though

Here dis shit happend yesterday to me >:)
Dere the Crash Dump :3
https://www.mediafire.com/file/q1yq0dohc1erzxw/Crash_FW_a2ab.dmp/file
https://www.mediafire.com/file/0d11c6jv78xtu8m/Crash_FW_a2d4.dmp/file

The crash dump will only include where the issue happened, but we know it happened in CSignature::Find()

 Here dis shit happend yesterday to me >:) Dere the Crash Dump :3 https://www.mediafire.com/file/q1yq0dohc1erzxw/Crash_FW_a2ab.dmp/file https://www.mediafire.com/file/0d11c6jv78xtu8m/Crash_FW_a2d4.dmp/file

Previous: 55 8B EC 83 EC 28 56 8B 75 10 57 8B 7D 14 8B C7 99 83 E7 03 83 E2 03
New: 55 8B EC 83 EC ? 56 8B 75 ? 57 8B 7D ? 8B C7
Another weird case... Is it possible that our sig scanner has issues with wildcards? But that would not explain why it only happens sometimes.

or why after it happens it won't stop happening on the same sig until you restart your PC.

can confirm

i had the same problem, restarting didnt help anything but moving steam to documents fixed it for me.

i have the same problem, same error, heres the crash dump
Crash_FW_54c6.zip

If this happens again, please create a full dump of the game using Process Explorer:

Solution! If you have Bakkesmod opened quit it, quit steam, load Fedoraware again and it should fix the problem

Happens to me too, the fix "load fedoraware again" doesn't work for me

still having the same issue


Keep screen on while active

Since the Soundscape web app doesn't need much interaction, the device tends to auto-lock even when the app is open and active. At least on iOS Safari, my understanding is that geolocation updates aren't reported to websites while the device is locked. Thus, callouts are stopped prematurely.
Is there a way to keep the screen on when a callout mode is enabled, e.g. by pretending that a video is playing?

I think another feasible solution is to use the Screen Wake Lock API , which seems to be supported by most common browsers, including iOS Safari Unfortunately, it does look like it won't work on iOS Safari if the user has the site installed as a home screen web app. If that's not an acceptable caveat, then the best workaround would be to have a silently repeating audio/video in the background, as you suggested. Either way, I can get started on this.

Thanks for the insight! I was not familiar with the Screen Web Lock API. That looks like the most straightforward and appropriate way to accomplish this, so I'd certainly welcome that implementation.


Directory for image

Sir,
I am working on my M.Sc. thesis on protoplanetary discs and I have obtained following issues.
When running the segment of code given in "docs-->example_dust-->Dust_model.ipynb"  to get an image of temperature gradient, the following error was shown.
No such file or directory: './dust_temperature.bdat'
Similar error has occurred for later code which gives the image of disc. The file is not found and hence the image is not obtained.
The segment of that code is given below
check temperature (northern emisphere)
Ts=np.fromfile('./dust_temperature.bdat', count=gridmodel.Nrgridmodel.Nphigridmodel.Nth+4, dtype=float)[4:].reshape( (gridmodel.Nphi, gridmodel.Nth, gridmodel.Nr))
plt.pcolormesh(gridmodel.redge, gridmodel.thedge[::-1], Ts[0,:gridmodel.Nth,:])
plt.xlabel('Radius [Rsun]')
plt.xscale('log')
plt.ylabel(r'Polar angle [rad]')
plt.colorbar(label='Temperature [K]')
plt.show()
Could you let me know that what file is used to obtain the data of the image. Since I have to obtain the images of sources in Orion Cluster, it will be helpful if I obtain general data file for all the images.
Thanking you in advance,

That means that the dust temperature file was not produced. This is likely because of an error earlier in your code. I recommend you check earlier messages in the terminal to see what could have gone wrong. If I had to guess, radmc3d probably complained that there are no opacity files (dustopac*.inp) or stars.inp file.

Error is also shown for, "dust.compute_opacities()" and this part of code comes before dust temperature file. The error shown for this is "FileNotFoundError: ./Tempkappa/dustkappa_temp_1.inp not found."
Detailed error
FileNotFoundError                         Traceback (most recent call last)
Cell In[22], line 1
----> 1 dust.compute_opacities()
File ~\Conda\lib\site-packages\disc2radmc\model.py:460, in dust.compute_opacities(self)
457     os.system('mv '+path+'dustkappa_'+self.lnk_file_p[:-4]+'.inp '+path+'Tempkappa/dustkappa_temp_'+str(i+1)+'.inp ')
459     # read opacity
--> 460     opcti=np.loadtxt('./Tempkappa/dustkappa_temp_'+str(i+1)+'.inp', skiprows=2)
461     opctj=opctj+weights[i]*opcti[:,1:]
463 ### write opacity file
File ~\Conda\lib\site-packages\numpy\lib\npyio.py:1338, in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack, ndmin, encoding, max_rows, quotechar, like)
1335 if isinstance(delimiter, bytes):
1336     delimiter = delimiter.decode('latin1')
-> 1338 arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter,
1339             converters=converters, skiplines=skiprows, usecols=usecols,
1340             unpack=unpack, ndmin=ndmin, encoding=encoding,
1341             max_rows=max_rows, quote=quotechar)
1343 return arr
File ~\Conda\lib\site-packages\numpy\lib\npyio.py:975, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)
973     fname = os.fspath(fname)
974 if isinstance(fname, str):
--> 975     fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
976     if encoding is None:
977         encoding = getattr(fh, 'encoding', 'latin1')
File ~\Conda\lib\site-packages\numpy\lib_datasource.py:193, in open(path, mode, destpath, encoding, newline)
156 """
157 Open path with mode and return the file object.
158
(...)
189
190 """
192 ds = DataSource(destpath)
--> 193 return ds.open(path, mode, encoding=encoding, newline=newline)
File ~\Conda\lib\site-packages\numpy\lib_datasource.py:533, in DataSource.open(self, path, mode, encoding, newline)
530     return _file_openers[ext](found, mode=mode,
531                               encoding=encoding, newline=newline)
532 else:
--> 533     raise FileNotFoundError(f"{path} not found.")
FileNotFoundError: ./Tempkappa/dustkappa_temp_1.inp not found.

Did you follow the instructions on how to compute the dust opacities at https://github.com/SebaMarino/disc2radmc/blob/main/docs/example_dust/Dust_model.ipynb just after cell 5?


(chore): Patch security vulnerability in loofah

Loofah is a dependency on rails-html-sanitizer which is a dependency of actionpack (rails). There is no patch for Rails at this time so we have to define it explicitly.
CVE-2018-3741

 Patch security vulnerability in Loofah

 Patch security vulnerability in Loofah


MAINT: Only use pip in travis.

Rather than a hodgepodge of pip and conda.  To make downloads not take
forever, download wheels from our wheelserver.

:+1:

@richafrank I'd save the :+1: for when I actually get this working... :sob:

You can count it as my cheering you on.  :tada:

Things I have learned tonight:

Travis' default intrastructure is based on Ubuntu 12.04.  Our wheels are built on a machine using Ubuntu 14.04.  This means that we can successfully install our numpy wheel, but at runtime it fails trying to find liblapack.so.3.  On 14.04, this shared object is provided by liblapack3 or libatlas3-base.  On 12.04, the analogous .so files are provided by liblapack3gf, but they're named liblapack.so.3gf.  The name mangling seems to be part of a historical migration strategy for Debian switching fortran compilers.  It's not clear to me whether it's safe or reasonable to simply copy over the .3gf shared objects to where our Ubuntu 14.04 wheels expect them to be.

Travis has beta support for building on Ubuntu 14.04.  You can enable this by setting sudo: required and dist: trusty in your .travis.yml.  However, the Python 2.7.10 installed on the new image appears to have been compiled in a different unicode configuration than the machine that builds our wheels.  When I try to use them on Travis' 14.04 infrastructure, I get failures trying saying undefined symbol: PyUnicodeUCS4_AsUnicodeEscapeString.  This appears to happen because our wheels are built against a Python installation that was compiled with --enable-unicode=ucs4, whereas Travis' Python is compiled with --enable-unicode=ucs2.

Closing this since I don't think this particular strategy will be worthwhile.


openelec switch to usbgrabber as a source

Hi guys, this is first time I installed latest openelec 4.2.1 and hyperion. Source for hyperion after installation is RPi desktop. But how can I switch my source to my usbgrabber?

@pete111 can be closeD?

Closed due to age. Please open a new one, if problems still exists.
Thank you


Building Pulsar (incl. docker images) from source with Java 9+ (Java 11, Java 15, etc.) causes silent NoSuchMethodErrors at runtime

Building Pulsar (incl. docker images) from source with Java 11 causes silent NoSuchMethodErrors at runtime
To Reproduce
Steps to reproduce the behavior:
Use Java 11.
Then, compile v2.6.2-candidate-1 docker images locally:
git clone --depth 1 -b v2.6.2-candidate-1 https://github.com/apache/pulsar pulsar-2.6
cd pulsar-2.6
mvn -DskipTests clean install
./docker/build.sh

Start pulsar in docker
docker run -p 6650:6650 pulsar:2.6.2 bin/pulsar standalone --no-functions-worker -nss

Compile & run this Pulsar code with required libraries:
import org.apache.pulsar.client.api.*;

public class PulsarJava11Problem {
    public static void main(String[] args) throws PulsarClientException {
        PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
        Producer<String> producer = pulsarClient.newProducer(Schema.STRING).topic("mytopic").create();
    }
}

The producer never gets created and it times out.
What happens on a server is a java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
This happens at https://github.com/apache/pulsar/blob/097108a7abae750a239abb606d2a31442319e741/pulsar-common/src/main/java/org/apache/pulsar/common/schema/LongSchemaVersion.java#L44 .
java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
	at org.apache.pulsar.common.schema.LongSchemaVersion.bytes(LongSchemaVersion.java:44)
	at org.apache.pulsar.common.protocol.Commands.newProducerSuccessCommand(Commands.java:377)
	at org.apache.pulsar.broker.service.PulsarCommandSenderImpl.sendProducerSuccessResponse(PulsarCommandSenderImpl.java:95)
	at org.apache.pulsar.broker.service.ServerCnx.lambda$handleProducer$19(ServerCnx.java:1137)
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$createSchemaLocator$28(BookkeeperSchemaStorage.java:504)
	at org.apache.bookkeeper.util.ZkUtils$1.processResult(ZkUtils.java:80)
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$10$1.processResult(ZooKeeperClient.java:737)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:636)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)

This exception never gets printed to the log, this stacktrace was captured using the debugger.
Expected behavior
Compiling with an unsupported Java version should result in an error and the NoSuchMethodError that happens in Pulsar shouldn't get "swallowed".
It should be possible to compile the code with Java 9+ without running into issues when running with Java 8.
There's an explanation of the issue with ByteBuffer at https://github.com/eclipse/jetty.project/issues/3244 . A solution is also described:

The solution is to cast the ByteBuffer to Buffer when calling those methods:
((Buffer)byteBuffer).position(0);

@eolivelli I noticed that you had done some changes to allow compiling with JDK 14+. Have you come across the ByteBuffer incompatibilities?

This problem should not only exist in 2.6.2, but it should also be a master problem, @zymap helped to verify the logic with JDK 1.8, it seems that everything is OK. It looks like we need to support different JDK versions to build Apache Pulsar.

@lhotari
my efforts about building with JDK14+ are about using JDK14 for development,
if you build it with JDK>8 you cannot run with JDK8.
We should add maven.compiler.release=8 in our pom.xml, it must be done in a profile activated when you are on jdk9+, it is a quite easy fix.
that's a very common problem:
java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;

We should add maven.compiler.release=8 in our pom.xml, it must be done in a profile activated when you are on jdk9+, it is a quite easy fix.

@eolivelli I think it would be valuable. It's easy to miss that you must compile Pulsar with Java 8 since the compilation passes without errors and even at runtime the errors are swallowed and not logged (at least in this particular case). Very hard to debug.

I guess that if you build with JDK14+ and you run everything on JDK14+ you won't hit problems.
probably you are not going to run Pulsar on JDK8, as newer JVMs are generally (I don't want to start a discussion now) more efficient and provide better performances.
In BK community we are switching to JDK11+ (but still next release will work on JDK8)

There a blog post with an easy solution: https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ .
This property should be defined in pom.xml:
<properties>
<maven.compiler.release>8</maven.compiler.release> </properties>

You have to add it in a profile, because if you add only this property the build will break on jdk8 because the 'release' option is not valid with jdk8


feat: support html-webpack-plugin@4

resolve #51

@numical Can this pr be merged?

This is super important. For many projects, can this PR be merged please ???


Remove double normalization from GTO radial basis

All tests are expected to fail since I have to also change the reference data in regression tests.


HDF5 pytables input/output from Containers

This is an initial implementation of a simple I/O  system that takes Containers and creates HDF5 tables where one row in the table contains the container. It is intended as a building-block to make a more fancy I/O system or just as a technology study (there may be better ways to do it).

elements of a Container are translated to columns in the output table automatically
the user can assign a transformation function to each column that is called on the value before writing it
it does not yet use the common Serializer class yet, since I didn't quite like the interface (needs some review)
it does not recurse into sub-containers, though this could be done.

Use Case
Pipeline needs a way to efficiently store data after calibration (DL1), DL2 event lists, etc.
Usage
    # make some dummy data in containers
    r0tel = R0CameraContainer()
    mc = MCEventContainer()
    mc.reset()
    r0tel.adc_samples = np.random.uniform(size=(50,10))
    r0tel.adc_sums = np.random.uniform(size=50)
    r0tel.num_samples = 10
    r0tel.meta['test_attribute'] = 3.14159
    r0tel.meta['date'] = "2020-10-10"

    # write the data to a file
    writer = SimpleHDF5TableWriter('test.h5', group_name='R0')

    for ii in range(100):
        r0tel.adc_samples[:] = np.random.uniform(size=(50,10))
        r0tel.adc_sums[:] = np.random.uniform(size=50)
        r0tel.num_samples = 10
        mc.energy = 10**np.random.uniform(1,2) * u.TeV

        writer.write("tel_001", r0tel)
        writer.write("tel_002", r0tel)  # write a second table too
        writer.write("MC", mc)

Which generates:

TODO

allow multiple containers to be joined into a table
either add event_id to each container, or make a way to add a primary key when calling write()
implement reading back into containers
figure out what to do with things like tels_with_data, which are var-length lists and not easily supported (convert to fixed-length masks?)
unify with Serializer base class, so we can implement other ones as well and speed test

Reading now works like this:
reader = SimpleHDF5TableReader("test.h5")
mc = MCEventContainer()

for cont in reader.read("/R0/MC", mc):
    print(cont)

{'alt': 0.0,
 'az': 0.0,
 'core_x': <Quantity 0.4060639535514794 m>,
 'core_y': <Quantity 0.5495818276293845 m>,
 'energy': <Quantity 77.34112211504227 TeV>,
 'h_first_int': 0.0,
 'tel': {}}
{'alt': 0.0,
 'az': 0.0,
 'core_x': <Quantity 0.2265416589346445 m>,
 'core_y': <Quantity 0.9771415161022963 m>,
 'energy': <Quantity 94.48982399127513 TeV>,
 'h_first_int': 0.0,
 'tel': {}}
{'alt': 0.0,
 'az': 0.0,
 'core_x': <Quantity -0.9818453098851436 m>,
 'core_y': <Quantity 0.8782971828013879 m>,
 'energy': <Quantity 29.2428288115323 TeV>,
 'h_first_int': 0.0,
 'tel': {}}

Right now both cont and mc above are pointers to the same structure (like in hessio_event_source, no copying is done), but we can discuss that. Also, multiple tables can be read at once by getting several generators.

Since this doesn't interfere with anything, and I want to use it during the next workshop, I will merge it. The API may of course change


Re-run Last launch configuration

VS Code version: 1.61
Summary
have a key binding/menu to rerun recently executed launch configuration without having the same file opened. In other words, implement https://github.com/microsoft/vscode/issues/25310 but for rerunning launch configurations instead of tasks (with the same support for "runOptions": {"reevaluateOnRerun": false}.
Description:
Please let me know if I'm mistaken, but one of the most common workflows in every project is to work on a single entry point for a while. For example, you want to fix a bug in your application and you're going to edit many files, but you always going to run the same entry point file to check the outcome. Also, I guess its typical to have a very dummy/boring entry point file that just launches some highlevel function/class, so you are not going to work on that file per see and you don't want to have it hanging in your editor.
In PyCharm/Eclipse and others, you're going to choose a file entry point (e.g. main.py) and press a keybinding  (ctrl+shift+f10) Edit Configurations to make this file a default thing to run. Then you might be editing a library.py but then by pressing another keybinding (ctrl+F10) you're going to rerun main main.py.
Workarounds
Currently, I have to keep main.py always opened and keep coming back to it after fixing a code in multiple library files which is a waste of precious clicks. I recently switched to VS Code from Intellisense family and I'm not sure if I'm doing it right, but I really tried to setup a similar workflow and I can't imagine how people are developing without it?
I tried Launch Config plugin. It allows making a task out of a launch configuration. Then I set up "runOptions": {"reevaluateOnRerun": false} flag for such task. But since my launch configuration also has preLaunchTask configured, the rerun keybinding is going to rerun that preLaunchTask since it was the last one (not the launch itself).
I tried to setup a task -> that runs input -> that runs vscode.debug.startDebugging command -> that runs a launch configuration (!), but it can't find configurations with preLaunchTask (probably related to https://github.com/microsoft/vscode/issues/134496).
I do not want to create a separate configuration for each entry point I have. There might be 2 or 3 different entry points that I work on given the project (e.g. main and a unit test). Its not productive to keep changing configurations every week and have multiple key bindings.

Just use or create a launch config with a fixed entrypoint file. Then use F5 to (re)run that launch config.

We have no architecture to take "snapshots" of a launch config that uses variables (like the ${file} variable that you are using).

A "launch configuration" is an important concept in the VS Code debug extension architecture: how it gets created, then resolved, and finally passed to debug adapters is a delicate but deterministic framework and all debug extensions are relying on that heavily. Introducing a "rerun last resolved task" concept would add a side entry point to this framework that bypasses the early stages and thus breaks some assumptions. That's the reason why we cannot easily introduce this concept in a transparent non-breaking way.
It would be possible to add the "rerun last resolved task" concept as an "opt-in" feature. But this requires considerable design and planning work on the debug extension API and maybe even on the Debug Adapter Protocol side.

"difference between launches and tasks"
VS Code initially had only "launch configurations" specifically targeted to debugging. "Launch configurations" have a handful of common properties defined by VS Code's debugging UI. All all other properties are contributed as JSON schemas by debug extensions and their semantic is not understood by VS Code.

When "tasks" got introduced, we did not consider to generalise the debug related "launch configurations" in order to cover "tasks" because we did not see enough benefits.

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!


Replacing multiple layers with images does not work; it only replaces the same layer multiple times

Describe the bug
I have an application that is replacing some set layers with images. I loop over every layer in the doc, and when i get to the layers I want, I replace them with a new image I just downloaded.
Here is a snippet of code.
with Session(rpath/to/file', action="open") as ps2:
        # Open the Photoshop document
        ps2.echo(ps2.active_document.name)
        for layer in ps2.active_document.layers:
            if layer.name.startswith('GK'):
                   layer_designator = layer.name.split()[1]
                         if layer_designator == 'CLUBLOGO':
                            name = layer.name
                            print('logolayer name is: ' + name)
                            url3 = str(players['Goalkeeper']['club_logo'])
                            destination_folder3 = r"path/to/save"
                            filename3 = url3.split("/")[-1]
                            destination_path3 = os.path.join(destination_folder3, filename3)

                            response3 = requests.get(url3)
                            if response3.status_code == 200:
                                with open(destination_path3, "wb") as f3:
                                    f3.write(response3.content)
                                print(f"Image downloaded and saved to {destination_path3}")

                                # Replace the image of the current active layer with the downloaded image
                                replace_contents = ps2.app.stringIDToTypeID("placedLayerReplaceContents")
                                desc = ps2.ActionDescriptor
                                idnull = ps2.app.charIDToTypeID("null")
                                desc.putPath(idnull, destination_path)
                                ps2.app.executeAction(replace_contents, desc)

                            else:
                                print("Failed to download the image")
                            layer.name = name
                        elif layer_designator == 'FLAG':
                            name = layer.name
                            print('flaglayer name is: ' + name)

                            url = str(players['Goalkeeper']['flag'])
                            destination_folder = r"path/to/file"
                            filename = url.split("/")[-1]
                            destination_path = os.path.join(destination_folder, filename)

                            response = requests.get(url)
                            if response.status_code == 200:
                                with open(destination_path, "wb") as f:
                                    f.write(response.content)
                                print(f"Image downloaded and saved to {destination_path}")

                                # Replace the image of the current active layer with the downloaded image
                                replace_contents = ps2.app.stringIDToTypeID("placedLayerReplaceContents")
                                desc = ps2.ActionDescriptor
                                idnull = ps2.app.charIDToTypeID("null")
                                desc.putPath(idnull, destination_path)
                                ps2.app.executeAction(replace_contents, desc)

                            else:
                                print("Failed to download the image")
                            layer.name = name

Expected behavior
I would expect the layers to appear and replace the smart objects they are pointing to. I have confirmed the URLs are separate, and they are downloading correctly. When i added in some sleeps, I could see the same layer being replaced multiple times. I can also confirm that the layers are named correctly and are printing out correctly to the terminal when accessed. Is there some caching issues I am unaware of with selected layers?
Desktop (please complete the following information):

OS: Windows 11
Photoshop Version: 2024, v25.4
Python Version: 3.11.0

let me know if there is any more info you need. I could have a workaround where I just create a new layer everytime and move it to the correct spot, but this is the beauty of smart objects to just replace them I would hope

a bit late but photoshop 2024 is not supported it seems.


Support JDK10

[x] The Docker image should contain JDK10.
[ ] The repository should be tagged as jdk-ocamorph-pyphen-10.0.0.

Waiting for the 10.0.0 OpenJDK Docker image before closing this issue and tagging the master branch.

Done, version 10.0.0 has been published successfully.


Tutorial Link on js.ipfs.io leads to 404 page

Behavior: when I click the link Tutorials on the js.ipfs.io page it leads me to https://github.com/ipfs/js-ipfs/tree/master/examples#js-ipfs-examples-and-tutorials which is a 404 page.
Expected Behavior: When I click the link Tutorials on the page it should lead me to a page of examples and tutorials.
Keep being awesome!

Thanks for reporting, @lindseymysse. Are you still having this issue? For me the page you've linked to loads correctly.

Looks like you fixed it! Thanks a ton.


openssl 3.0.1 TLS 1.1/1.0 negotiation failed

conf.dump.gz
Under openssl 3.0.1
Running with  openssl s_server -cert /tmp/.ossl3.pem -key /tmp/ossl3.key,
Then run openssl s_client -tls1_1, or openssl s_client -tls1 got error from s_server:
Using default temp DH parameters
ACCEPT
ERROR
80A19282E37F0000:error:0A000076:SSL routines:tls_choose_sigalg:no suitable signature algorithm:ssl/t1_lib.c:3331:
shutting down SSL
CONNECTION CLOSED

Then I try on openssl 1.1.1m acting as server, openssl3.0 as s_client, I got error of:
ACCEPT
ERROR
139869616887616:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1543:SSL alert number 80
shutting down SSL
CONNECTION CLOSED

TLS 1.2 and 1.3 work as expected.
The same commands with openssl 1.1.1m as server and client works fine for all protocol.
cert-key.tar.gz

add the configuration dump
conf.dump.gz

SHA1 is required for TLS1.0 and 1.1 and it is not available at the default SECLEVEL. You need to use -cipher DEFAULT:@SECLEVEL=0 with OpenSSL-3.0 to be able to use TLS1.0 or TLS1.1.

That works. Thanks.


ospdo adoption dataplane adoption

Add Director Operator dataplane_adoption scripts for adoption
Related CI patch and background:
https://gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/-/merge_requests/966
Jiras:
https://issues.redhat.com/browse/OSPRH-1891
https://issues.redhat.com/browse/OSPRH-2984

recheck


[volume] Add support for units of volume.

Coverage remained the same at 100.0% when pulling 8b53c8f3f14175500f1085f36e023b05c914806d on mvolk/volume into 497219f27369607bbf9b874780ab4df81eee2b8e on master.

Coverage remained the same at 100.0% when pulling bfe3359a3ab38969b9665f401c96f3206c4cf695 on mvolk/volume into 497219f27369607bbf9b874780ab4df81eee2b8e on master.


fix docker compose setting, make cello-api-engine depends_on cello-po…

make cello-api-engine depends_on cello-postgress

Follow https://github.com/hyperledger/cello/wiki/Contribution-Process to format the commit message.


Can you provide an example usage of this algorithm on an image?

I need this image enhancement algorithm to run on a set of images. An example of usuage would be appreciated.

I've written an article providing some details: https://www.toptal.com/algorithms/successive-mean-quantization-transform
But I don't have an example usage in the form of source code.


Seperate logic and parsing for each transaction type

ParseTransaction() was added, without bRPConly, to have "safe" read-only access to the base layer.

The interpretation checks from stepXYZ() were added directly to the logic methods.

Until now, some parts of the logic were directly part of interpretPacket(), which were moved into seperated logicMath_XXX() methods, for all transaction types.

Payloads for every transaction type are parsed in new interpret_XXX() methods.

The main motivation is to provide a way to parse/interpret payloads, without triggering any effects, and to depreciate the stepXYZ() methods. The one-for-all-types method for this is interpret_Transaction(), which then calls other parsing/interpretation methods, based on the transaction type.
This should pave the way and lay out a basis for finer transaction handling.

Wow - this looks like it could be really, really useful!

I'll definitely spend some time over the next day or two to give this a proper look over mate - really excited about this one!

Thanks @zathras-crypto. Any feedback is highly welcomed. I'm pretty much testing new waters and methologies here, and there is a good chance, that some parts may not make sense. :)

I'm surprised, as far as I can see it should go this path:

gettransaction_MP()
populateRPCTransactionObject() (valid added here?)
populateRPCTypeInfo()
populateRPCTypeSimpleSend()

However (regtest):
gettransaction_MP e39ecc03946c94e145342a6e97ef369a079a92ca56bc31eb70c8fb90965a63ae
{
  "txid": "e39ecc03946c94e145342a6e97ef369a079a92ca56bc31eb70c8fb90965a63ae",
  "sendingaddress": "mkoBjBRB8HKDqKESQt2Js4XDQGTo8pUKEA",
  "referenceaddress": "mq5sNzFKYW3VpSXcvnh86E5F93ZQuWYxZD",
  "ismine": true,
  "confirmations": 1,
  "fee": "0.00010000",
  "blocktime":<PHONE_NUMBER>,
  "version": 0,
  "type_int": 0,
  "type": "Simple Send",
  "type": "Crowdsale Purchase",
  "propertyid": 1,
  "divisible": true,
  "amount": "0.00000001",
  "purchasedpropertyid": 3,
  "purchasedpropertyname": "MDiv",
  "purchasedpropertydivisible": true,
  "purchasedtokens": "0.00000000",
  "issuertokens": "0.00000000"
}

The zero amounts are expected here, but the valid is missing.

That's interesting:
    // in populateRPCTransactionObject():
    txobj->push_back(Pair("value1", "1"));
    txobj->push_back(Pair("value2", "2"));

    // populate validity
    bool valid = getValidMPTX(txid);
    txobj->push_back(Pair("valid", valid));

    txobj->push_back(Pair("value3", "3"));
    txobj->push_back(Pair("value4", "4"));

gettransaction_MP e39ecc03946c94e145342a6e97ef369a079a92ca56bc31eb70c8fb90965a63ae
{
  "txid": "e39ecc03946c94e145342a6e97ef369a079a92ca56bc31eb70c8fb90965a63ae",
  "sendingaddress": "mkoBjBRB8HKDqKESQt2Js4XDQGTo8pUKEA",
  "referenceaddress": "mq5sNzFKYW3VpSXcvnh86E5F93ZQuWYxZD",
  "ismine": true,
  "confirmations": 1,
  "fee": "0.00010000",
  "blocktime":<PHONE_NUMBER>,
  "version": 0,
  "type_int": 0,
  "type": "Simple Send",
  "value1": "1",
  "value2": "2",
  "valid": true,
  "value3": "3",
  "type": "Crowdsale Purchase",
  "propertyid": 1,
  "divisible": true,
  "amount": "0.00000001",
  "purchasedpropertyid": 3,
  "purchasedpropertyname": "MDiv",
  "purchasedpropertydivisible": true,
  "purchasedtokens": "0.00000000",
  "issuertokens": "0.00000000"
}

Now "value4": "4" is missing.
It looks like it's overwritten by "type": "Crowdsale Purchase", probably because there is already a "type"`.

Just thinking, in light of #55..
What, if we further split logic checks and actually applying the effects? In this case it would be possible to check, whether a transaction could be valid, before sending one.
That's very doable for simple transaction types, but it seems more difficult for complex transactions such as STO or trading on either one of the exchanges.. :/
On the other hand, I guess the same applies to all checks done on the RPC layer, which face similar challenges, and in the end it probably means to take a close look at failure paths, and get a smarter/more flexible logic or checks.

I played around with restructuring the STO code, to get an idea, whether it's feasible to split checking of logic and executing effects.
It basically comes down to having two stages, in this particular case at least, where the outcome of the effect is determined (i.e. STO receivers gathered, including amounts and fees), and then executed. For STO there is no penality, because we already run a loop over the receivers two times.
I think it's worth to pursuit, and slowly move towards such an approach, however, I think it should not be part of this PR (since it's getting pretty huge). The much more motivating upside of this is that the code becomes more testable, and the second stage (executing the logic) could basically be plastered with assertions.
For example, let's take a look at the simple send code here:

src/omnicore/omnicore.cpp#L3697-L3727

If it passes the checks above the comment line, then updating the tally is certainly expected to be successful. A split would seperate the precondition checks above the line, and executing the logic below the line.

Another note:
I started to group and tag the members in tx.h, and I think we should move towards transaction type specific classes for all types at some point, which we already have (to some degree) for the MetaDEx and DEx. To compare:

src/omnicore/tx.h#L32-L83
src/omnicore/dex.h#L36-L43
src/omnicore/mdex.h#L89-L98

Once that's done, we may eventually consider persisting those transaction objects, which we already do for some via the file based persistence (which simply writes and reads all members).
My ultimate vision is to be able to describe and derive global state by executing a sequence of transactions. A simple example:

A purchases 20 MSC
A sends 5 MSC to B
A sends 3 MSC to B

A now has 12 MSC, and B has 8 MSC (obviously), however, instead of saying "A has 12 MSC", we can also say "A has 20 - 5 - 3 = 12 MSC". Hope it's clear where I'm going ... :)

Sorry took a little while to get around to this one dude but I like it - I like it a lot - excellent work :)
I didn't find anything that stood out as problematic - given Travis gave it the all clear is it safe to assume it passes all tests (OmniJ and unit tests in src/test?

Yup, tests pass, including the old Python tests and mainnet consensus.

Nice :)  This is a huge step forward for us I think - and opens the door for unconfirmed handling.

Alright, last update pushed to address the feedback. :)


Provide the ability to lazily yield rows from Table storage

This can be accomplished using the continuation token in Azure Table Storage.

Not sure how this would work in the context of Async though.

Something like this?
http://tomasp.net/blog/async-sequences.aspx/

Yeah, I had thought of that. I think it's probably overkill to add an entire CE just for this though. An alternative is to simply leaves things as they are or to only add it in for the synchronous version.


Updating documentation for version 1.2 of the library

Updating documentation for v1.2.0 of the library

@d45 @johnDance @seanmakesgames @xreid @jz-huang @LGraber

@nikhillakshman    - Looks good!


Examples are not working

Just wanted to run one of the examples but receive error on production mode.
Commands ran:
npm run build
npm run start

Result:
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/usr/testExample/dist/server.js' imported from /usr/testExample/node_modules/solid-start-node/index.js
Tried on /examples/bare, /examples/with-tailwindcss and some more

It should be working now, could you try again?

I think this can be closed

with-prisma doesn't work: Pls add
  ssr: {     external: ['@prisma/client'],   },
in vite.config.ts.
After that it works, but still this console msg: Uncaught Error: PrismaClient is unable to be run in the browser.

To get with-prisma working I had to change db/index.ts to:
export const db = () =>   server$(() => {     return new PrismaClient()   })

@Maxximus007 oh my god that fixed my issue with building my application.
The error message I got was really different from OP though and had me looking into different 'problems'. I'll share my error for others that might get it as well.
(Node version 16.18 with pnpm)
I got these warnings in pnpm build:
(...)
solid-start server built in: 934.499ms

Use of eval is strongly discouraged, as it poses security risks and may cause issues with minification
Use of eval is strongly discouraged, as it poses security risks and may cause issues with minification
Use of eval is strongly discouraged, as it poses security risks and may cause issues with minification
Use of eval is strongly discouraged, as it poses security risks and may cause issues with minification
Use of eval is strongly discouraged, as it poses security risks and may cause issues with minification

This was the error I got after  pnpm start:
file:///home/<USER>/<PROJECTROOT>/dist/server.js:40386
	import_path.default.join(__dirname, "../query-engine-darwin");
	                        ^

ReferenceError: __dirname is not defined in ES module scope
This file is being treated as an ES module because it has a '.js' file extension and '/home/home/<USER>/<PROJECTROOT>/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.
    at file:///home/<USER>/<PROJECTROOT>/dist/server.js:40386:27
    at file:///home/<USER>/<PROJECTROOT>/dist/server.js:51839:3
    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:526:24)

After adding     // ssr: { external: ['@prisma/client'], }, it works perfectly

Just setup solid-start with Prisma and I am getting Uncaught Error: PrismaClient is unable to be run in the browser. this seems to also be breaking basic features like onClick events. When I move createServerData$ out of the useUser and wrap the the useUser function call it fixes the uncaught error and fixes the onClick events.
index.jsx
export function routeData() {
  return createServerData$(async (_, { request }) => {
    return await useUser(request)
  })
}

The error seems to be with the useUser file being .js/.ts and not a .jsx/.tsx file.


[tasks] deeplink for drawthings make it conditional on model repo type

ok, ready for review!

cc @liuliu too in case you can review

will merge this one so we can already test in production, but feel free to open a new PR or suggest changes


v2 Fails to read 3rd column

Description
Attaching an excel file and I can see that library fails to read unit column values, instead just omits it entirely and shifts next columns to its value
Steps to reproduce the issue:

Try to parse the kmg_excel.xlsx

Describe the results you received: Unit column is omitted
Describe the results you expected: Unit column on row 34 and 35 should be parsed properly
Output of go version:
go version go1.20 linux/amd64

Excelize version or commit ID:
v2.6.1

Environment details (OS, Microsoft Excel™ version, physical, etc.):
Ubunut Linux 64bit

Thanks for your issue. This problem already has been fixed in v2.8.0, please upgrade to the lasted version or use the master branch code.

There is no feedback for this issue, so I've closed this, if you have a question, please let me know, and you can reopen this anytime.


ADD NEW TRADING RULE : RELATIVE STRENGTH INDEX (RSI)

DESCRIPTION : RELATIVE STRENGTH INDEX
CONDITION/PARAMETER : GIVE A BUY SIGNAL WHEN RSI VALUE IS BELOW 30 VICE VERSA GIVE A SELL SIGNAL WHEN THE RSI VALUE IS ABOVE 70.
if RSI value < 30:
position = max_investment
if RSI value > 70:
position = -max_investment
quick link to RSI
https://www.investopedia.com/terms/r/rsi.asp

Hi @bi-kash @fhashim for our common reference. thanks

Definition of RSI

formula for rsi

@Cal231995 can you confirm if the time period used to compute RSI is 14 days? thanks

@Cal231995 can you confirm if the time period used to compute RSI is 14 days? thanks

You can see the rule (coded by @bi-kash ) here: https://github.com/ta-oliver/infertrade/blob/5f74bdeb99eb26c15df0b5417de837466cefaee1/infertrade/algos/community/allocations.py#L522
We should match the PDF documentation with the Python code (and vice-versa!).

@Cal231995 can you confirm if the time period used to compute RSI is 14 days? thanks

yes , we used 14 as our default

What Is the Relative Strength Index (RSI)?
The Relative Strength Index (RSI), developed by J. Welles Wilder, is a momentum indicator that measures the speed and change of price movements. It is an extremely popular indicator that is used to indicate overbought and oversold signals. RSI is considered overbought when above 70 and oversold when below 30. RSI can also be used to identify the general trend of an asset. It can also be used for identifying bullish and bearish divergences.
@fhashim you can used this definition . This one's free of copyright issue.


SPI plugin load/save speed and sleep time calculation

This is a self reminder to fix the following:

save/load SPI speed parameter into the project workspace
fix usleep calculation as spotted by @janosvitok
https://github.com/mcallegari/qlcplus/blob/master/plugins/spi/spioutthread.cpp#L106

Fixed by d8a24c466342090af177ba30aee02ce57027386d
The SPI speed is still not saved into the project workspace, since it is a global setting and not per-line.
Might find a suitable solution in the future.


Update README.rst

Add quay.io status badge to readme.
Please ensure the following when opening a PR:

[ ] Added or updated tests
[x] Updated documentation
[ ] Applied style guidelines
[ ] Considered generalizability beyond our own use case

Thanks for the review @dshiga