[Imaging-369] Fix array bounds exception for TIFF with internal JPEG …

This pull request addresses an error introduced by Imaging-194 in which an edge-case could cause an array-bounds exception when reading an odd-sized TIFF file with internal JPEG compression (TIFF with JPEG is a new feature contributed by Imaging-194).
The pull request includes a code fix, expanded JUnit test, and a new test file.  It fixes the problem and slightly improves the JUnit test coverage. Code passed spotbugs and checkstyle checks.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (50e3dc5) 71.26% compared to head (d7ba95b) 71.28%.
Report is 2 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #335      +/-   ##
============================================
+ Coverage     71.26%   71.28%   +0.02%
- Complexity     3541     3543       +2
============================================
  Files           352      352
  Lines         17360    17360
  Branches       2690     2690
============================================
+ Hits          12371    12375       +4
+ Misses         3926     3924       -2
+ Partials       1063     1061       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


High memory usage even after test finishes

Environment

VerneMQ Version: 7ea2de2bd4db01cbe9e678ef2653a831d7e3cf9a
OS: CentOS 7
Erlang/OTP version (if building from source): 21.1
VerneMQ configuration (vernemq.conf) or the changes from the default

allow_anonymous = off
allow_register_during_netsplit = on
allow_publish_during_netsplit = on
allow_subscribe_during_netsplit = on
allow_unsubscribe_during_netsplit = off
allow_multiple_sessions = off
max_inflight_messages = 100
max_online_messages = 10000
max_offline_messages = 1000
max_message_size = 0
upgrade_outgoing_qos = off
listener.max_connections = infinity
listener.nr_of_acceptors = 10
systree_enabled = off
erlang.async_threads = 64
erlang.max_ports = 262144
leveldb.maximum_memory.percent = 8
erlang.distribution.port_range.minimum = 6000
erlang.distribution.port_range.maximum = 7999
erlang.distribution_buffer_size = 10MB
plumtree.outstanding_limit = 50000

We also set +hmqd off_heap on erlang VM.
Expected behavior
VerneMQ should need amount of memory which could be described in terms of parameters such as numer of clients, subscriptions, messages, nodes, etc.
Actual behaviour
Usually memory usage on our VerneMQ nodes is about 6 GB. However, sometimes we are observing excessive usage of RAM that gets worse over time. It happens that 16 GB RAM + 32 GB of swap is not enough for handling 80k connections and vernemq is killed by the OS.
Details
Our setup:

5 VerneMQ instances, 16GB RAM, 4 CPU each
haproxy as a load balancer (on a separate machine)
28 mzbench nodes serving as MQTT clients

Test scenario:

400k clients (let's call them "consumers") connect to the cluster over 9 minutes with a constant rate. Each of them subscribes to a topic which is unique per client.
1 special client ("producer") connects to the cluster. Once all the consumers are connected it starts to send messages to consumers' topics, one message per topic (client), 800 messages per second. The producer also subscribes to its own topic. This step of the test lasts for 15 minutes.

Once a consumer receives a message from the producer, it sends the message back to the producer.

When the exchange of the messages is finished, all the clients gradually disconnect.

Here are grafana charts showing memory and swap availability as well as network traffic and number of open tcp connections.

On the node which uses significantly more memory than the others, we could see the following log entries:
2018-11-29 13:26:42.655 [info] <0.163.0>@vmq_sysmon_handler:handle_event:95 monitor busy_dist_port <0.28189.38>  {#Port<0.240054>,unknown}
2018-11-29 13:26:52.437 [info] <0.163.0>@vmq_sysmon_handler:handle_event:95 monitor busy_dist_port <0.5512.32>  {#Port<0.240054>,unknown}
2018-11-29 13:27:04.524 [info] <0.163.0>@vmq_sysmon_handler:handle_event:95 monitor large_heap <0.232.0> [{name,plumtree_broadcast},{initial_call,{plumtree_broadcast,init,1}},{almost_current_function,{gen_server2,in,2}},{message_queue_len,58733}] [{old_heap_block_size,0},{heap_block_size,237288211},{mbuf_size,0},{stack_size,20},{old_heap_size,0},{heap_size,157902820}]
2018-11-29 13:27:55.891 [info] <0.163.0>@vmq_sysmon_handler:handle_event:95 monitor large_heap <0.232.0> [{name,plumtree_broadcast},{initial_call,{plumtree_broadcast,init,1}},{almost_current_function,{gen_server2,drain,1}},{message_queue_len,1002}] [{old_heap_block_size,0},{heap_block_size,237288211},{mbuf_size,0},{stack_size,18},{old_heap_size,0},{heap_size,157287251}]
2018-11-29 13:28:28.369 [warning] <0.232.0>@plumtree_broadcast:schedule_lazy_tick:697 Broadcast overloaded, 459580465ms mailbox traversal, schedule next lazy broadcast in 1464166ms, the min interval is 10000ms
2018-11-29 13:28:31.862 [info] <0.163.0>@vmq_sysmon_handler:handle_event:95 monitor large_heap <0.232.0> [{name,plumtree_broadcast},{initial_call,{plumtree_broadcast,init,1}},{almost_current_function,{erlang,send,2}},{message_queue_len,2987}] [{old_heap_block_size,0},{heap_block_size,197740176},{mbuf_size,0},{stack_size,4305},{old_heap_size,0},{heap_size,160192315}]
2018-11-29 13:28:34.401 [info] <0.163.0>@vmq_sysmon_handler:handle_event:95 monitor large_heap <0.232.0> [{name,plumtree_broadcast},{initial_call,{plumtree_broadcast,init,1}},{almost_current_function,{gen_server2,drain,1}},{message_queue_len,2754}] [{old_heap_block_size,0},{heap_block_size,164783480},{mbuf_size,0},{stack_size,20},{old_heap_size,0},{heap_size,160328783}]

(note that due to timezone offset, 13:00 in the log message corresponds to 14:00 in the charts)
Despite the actual test ended before 15:10 (grafana time), the .118 node remained very busy for 1.5h.
In this particular instance of the problem, the .118 node eventually recovered from the crisis. However, we have seen situations where a VerneMQ was killed and, in consequence, another VerneMQ node crashed:
x.y.z.118:
=erl_crash_dump:0.5
Fri Nov 23 14:58:03 2018
Slogan: eheap_alloc: Cannot allocate<PHONE_NUMBER>8 bytes of memory (of type "heap").
System version: Erlang/OTP 21 [erts-10.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64] [hipe]
Compiled: Fri Sep 28 12:44:50 2018
Taints: bcrypt_nif,mzmetrics,eleveldb,crypto
Atoms: 17648
Calling Thread: scheduler:0

x.y.z.95:
2018-11-23 14:58:03.125 [warning] <0.384.0>@vmq_cluster_mon:handle_info:122 cluster node<EMAIL_ADDRESS>DOWN
2018-11-23 14:58:03.143 [error] <0.209.0> Supervisor plumtree_sup had child plumtree_metadata_hashtree started with plumtree_metadata_hashtree:start_link() at <0.22939.11> exit with reason {{nodedown,'vernemq@x.y.z.118'},{gen_server,call,[{plumtree_metadata_manager,'vernemq@x.y.z.118'},{merge,{{vmq,subscriber},{[],<<"mzbench-pub-96935">>}},{metadata,{[{'vernemq@x.y.z.95',3,[]},{'vernemq@x.y.z.118',14,[{[{'vernemq@x.y.z.118',true,[{[<<"qwe">>,<<"asd">>,<<"mzbench-pub-96935">>],2}]}],{1542,982251,522645}}]},{'vernemq@x.y.z.204',3,[]},{'vernemq@x.y.z.144',6,[]}],[]}}},infinity]}} in context child_terminated
2018-11-23 14:58:03.144 [error] <0.2938.18> gen_fsm <0.2938.18> in state exchange terminated with reason: {{{nodedown,'vernemq@x.y.z.118'},{gen_server,call,[{plumtree_metadata_manager,'vernemq@x.y.z.118'},{merge,{{vmq,subscriber},{[],<<"mzbench-pub-96935">>}},{metadata,{[{'vernemq@x.y.z.95',3,[]},{'vernemq@x.y.z.118',14,[{[{'vernemq@x.y.z.118',true,[{[<<"qwe">>,<<"asd">>,<<"mzbench-pub-96935">>],2}]}],{1542,982251,522645}}]},{'vernemq@x.y.z.204',3,[]},{'vernemq@<IP_ADDRESS>',6,[]}],[]}}},infinity]}},{gen_server,call,[plumtree_metadata_hashtree,{compare,#Fun<plumtree_m..>,...},...]}}
2018-11-23 14:58:03.150 [error] <0.2938.18> CRASH REPORT Process <0.2938.18> with 0 neighbours exited with reason: {{{nodedown,'vernemq@x.y.z.118'},{gen_server,call,[{plumtree_metadata_manager,'vernemq@x.y.z.118'},{merge,{{vmq,subscriber},{[],<<"mzbench-pub-96935">>}},{metadata,{[{'vernemq@x.y.z.95',3,[]},{'vernemq@x.y.z.118',14,[{[{'vernemq@x.y.z.118',true,[{[<<"qwe">>,<<"asd">>,<<"mzbench-pub-96935">>],2}]}],{1542,982251,522645}}]},{'vernemq@x.y.z.204',3,[]},{'vernemq@x.y.z.144',6,[]}],[]}}},infinity]}},{gen_server,call,[plumtree_metadata_hashtree,{compare,#Fun<plumtree_m..>,...},...]}} in gen_fsm:terminate/7 line 609

It's worth mentioning that initially we have seen this problem erlang.distribution_buffer_size = 64MB and default values for plumtree.outstanding_limit and +hmqd. We started to experiment with those in order to mitigate the issue.
Is it a known issue? Are there any recommended workarounds? The reason we tried changing erlang.distribution_buffer_size and plumtree.outstanding_limit was that we observed high traffic (even when the test ended). Do you think tweaking these parameters can help?

Thanks for reporting this.
Can you tell us more about the OS configurations, especially TCP buffer tunings?
OOM kills that we've seen were often related to misconfigured LevelDB (not the case here) and/or oversized TCP buffers.

sysctl settings:
net.ipv4.tcp_rmem = 4096	87380	6291456
net.ipv4.tcp_wmem = 4096	16384	4194304

At the time when the test was conducted, our advanced.config specified:
{tcp_listen_options,[{sndbuf,1048576},{recbuf,1048576}]}

Now we re-run the test with smaller TCP buffers:
{tcp_listen_options,[{sndbuf,32768},{recbuf,32768}]}

We also restored erlang.distribution_buffer_size and plumtree.outstanding_limit to the default values. The problem occurred again:
$ sudo head  /var/log/vernemq/erl_crash.dump
=erl_crash_dump:0.5
Mon Dec  3 12:49:54 2018
Slogan: eheap_alloc: Cannot allocate<PHONE_NUMBER>8 bytes of memory (of type "heap").
System version: Erlang/OTP 21 [erts-10.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64] [hipe]
Compiled: Fri Sep 28 12:44:50 2018
Taints: bcrypt_nif,mzmetrics,eleveldb,crypto
Atoms: 17940
Calling Thread: scheduler:0
=scheduler:1
Scheduler Sleep Info Flags: SLEEPING | TSE_SLEEPING | WAITING

The crash was on the .118 machine. Here are some charts from grafana:

We lowered the setup rate from ~830 connects/subscribes per second to ~500. We also increased erlang.distribution_buffer_size from 512 MB to 1 GB. The test was running in a loop over night and eventually the problem appeared again -- after ~19 hours of testing. But it is a progress comparing to the former test results -- the cluster was operating well for a longer period.
Can you think of any configuration parameters that could help? Or, alternatively, is there a way to limit the subscription rate on the VerneMQ side? In other words, can we somehow configure VerneMQ to reject SUBSCRIBEs if they are too frequent?
By the way, shouldn't the "Broadcast overloaded" message above read "459580465us mailbox traversal"? timer:now_diff used in get_mbox_traversal_time function in plumtree_broadcast.erl returns time in microseconds, according to the documentation. The "ms" suffix is widely used to denote time in milliseconds.

After some investigation we found that the most probable reason for the excessive memory usage is that the plumtree's gen_server2 is too busy moving messages from the process' mailbox to the gen_server2's internal queue. As a result, there is no time (or too little time) to process the messages.
We mitigated the issue by implementing a mechanism which drops i_have messages when the process is under stress. The solution needs newer gen_server2. @dergraf , @larshesel , could you please review the commit referenced above as well as @p3we 's changes? If you like the idea, we would be happy to make PRs.
We made a build which incorporates the patch and ran a test with the original setup rate (~830 per sec) over night. Here are memory usage statistics:

wow, nice work!
Thanks so much for the time & effort going into this.

@szymonk I think this looks very good and makes sense to me... Awesome work! I'd be for merging this in one form or another, but we'd have to review the gen_server2 to see what changed there as well. A PR would be most welcome!

Great! Please have a look at the two PRs referenced above. Once these are merged, we can make another one to the vernemq project which will integrate the changes.

hi @szymonk we'll try to get this reviewed and hopefully merged today/tomorrow. Sorry for the delay!

hi @szymonk @p3we, we just merged both vernemq/plumtree#22 and erlio/gen_server2#1.
We're considering what the default value should be if we'd want it to be disabled by default or if we should enable it - and we're leaning towards enabling it - which value would you did you use in your tests? Would you like to do the PR for VerneMQ as well?

We found 1000000 (one million) to work best for us. However, the optimal value may depend on processing power, workload, use-case, etc. I'll make a PR in a second.

@szymonk As you might know we've bundled a new replication layer vmq_swc along the vmq_plumtree. We believe that your use case will heavily benefit from vmq_swc, so if you have time and resources it would be great to rerun your benchmark and compare the results. If you're interested I would wait until https://github.com/vernemq/vernemq/pull/1074 is merged, and then build from master.
All your tunings are still valid, it's also still eleveldb under the hood, so we'd really compare the replication performance.
Looking forward to your answer, please also don't hesitate to get in touch over Slack.
Best,
Andre


add authorization to api

Will need to add environment variables to the readme

Will need to add environment variables to the readme

Maybe it is good idea to disable auth for local/dev env?

Will need to add environment variables to the readme

Maybe it is good idea to disable auth for local/dev env?


Export Nitro Protocol deployed addresses in nitro-protocol

With this we will get
const addresses =<EMAIL_ADDRESS>
to be available, addressing https://github.com/statechannels/statechannels/issues/3010

This PR removes deploy artifacts from the repository. So we lose information like:

What PK deployed each contract.
What was the transaction of the deploy.

Are we sure we don't want to retain this information? If you find deploy artifacts too verbose, perhaps we can trim some of the keys.
One of the nice things about hardhat + CREATE2 is that if a contract has not changed since the last deploy and a redeploy is triggered, the contract is NOT redeployed. Have you checked that this behavior holds with deploy artifacts removed?

One of the nice things about hardhat + CREATE2 is that if a contract has not changed since the last deploy and a redeploy is triggered, the contract is NOT redeployed. Have you checked that this behavior holds with deploy artifacts removed?

I did try this and I also did find that that contract was not re-deployed

What PK deployed each contract.
What was the transaction of the deploy.

Are we sure we don't want to retain this information?

This information is available on etherscan, so we're not really losing it, just not storing it inside the repo explicitly; so I think it's fine


Compiler warns about "pure expression does nothing in statement position"

If you have a future foo: Future[Unit] and do a await(foo) the compiler will give a warning:
Warning:(258, 16) a pure expression does nothing in statement position; you may be omitting necessary parentheses
          await(foo)
               ^

+1

+1

In scala 2.11, I get a very similar error message for
def checkForException(codeBlock: Any, finallyBlock: Any = ()): Any = {
    try {
      codeBlock
    } catch {
      ...
    } finally {
      finallyBlock
    }
  }

the warning is:

a pure expression does nothing in statement position; you may be omitting necessary parentheses
[warn]       finallyBlock

Update: I fixed my warning by changing the function signature to be:
def checkForException(codeBlock: => Unit, finallyBlock: => Unit = ()): Any


Newbie How-To Contribute Tutorials

I really want to contribute to this project, but I'm just starting to learn how to use things like git, Visual Studio and other required tools, so it's a bit overwhelming. Therefore it would be extremely valuable to someone like me if more experienced developers out there could create some absolute beginner text / video tutorials on how to get the Entitas source setup so you can start contributing.
One weakness I see in most tutorials for projects like this is the focus on fellow developers of the same skill level or knowledge. However, what I really need are tutorials that don't assume any previous experience and treat me as completely new and ignorant on the topic. Now I like to think I'm not completely ignorant, but it's always best to assume the worse. 😄
Ideally I'd love a video tutorial that would go through the step-by-step process of setting up Entitas in Visual Studio on Windows and show how to debug your changes and test them in Unity. So essentially a much more detailed step-by-step video version of the CONTRIBUTING readme file. The Cloning Daggerfall Unity from GitHub Tutorial video is a good example of the kind of content I'm talking about here.
For experienced developers here this shouldn't be very difficult to create at all, because I basically just want you to show me how you go about setting up everything when you work with the Entitas source. Which I'm sure is second nature to you. 😉

These tuts are extremely time consuming to make, and this framework is open source and all. So we've been trying to contribute as a community to the docs and stuff - but we all have day jobs too, so it's a little slow.
We've added quite a few tuts already, but I assume you're struggling with them from your post. I'd be more than happy to answer questions you have if you PM me on gitter, maybe I can point you in the right direction. I am not an expert or even a professional programmer here, but I've been through the "learning to use entitas" process recently so I might be able to help here and there.

Perhaps if you clearly define what you want explained in this video tutorial I or someone else from the entitas community will make one and put it up on youtube. I wouldn't mind doing a video tutorial but wouldn't know what to cover. I've done live streaming on livecoding and twitch of my projects before but not made any youtube tutorials.

@FNGgames @T2RKUS Thanks for the reply. Here's my specific situation if that helps any:
I already have the Entitas source forked and cloned to my local machine through SourceTree, but I don't know how to build and test any modifications made in Visual Studio from Unity.
Simon has been trying to help me with this, but he's on Mac and I'm on Windows and so far I can't get it working, At the moment typing ./Scripts/update into the SourceTree Terminal just returns /c/GitHub/Entitas/Scripts/build: line 6: xbuild: command not found and I have no idea how to fix this.
I've been looking at other means of testing in Unity, but so far I don't even know which scripts or DLLs I should copy over from the Entitas souce to Unity in order to test my modifications manually.

I see, and what version of Unity are you currently using?

For starters, when I ran xbuild from the commandline the command didn't exist so I looked for xbuild in my Program Files(x86) on windows 10 It was in the Mono folder after doing a search for it. If you add xbuild.exe to your PATH then it should make that error go away.

Just found out that if you go to Task Manager and find Windows Explorer (The File Browser) and right click -> restart it will also reload the desktop and the environment variables as well. No restart needed.

@IsaiahKelly This might be fixed in the Future. Microsoft just release a new mono version last week where they actually made the switch from xbuild (mono) to msbuild (which also runs on windows). Currently I cannot update to msbuild, but I might in the future.
In general, you have to make sure that the paths in Scripts/consts are setup to work properly on windows. E.g. /c/GitHub/Entitas/Scripts/ doesn't look like a valid path to me
should more look like c:\GitHub\...

The build script worked for me under the source tree terminal but not cygwin

I wonder if that has to do with me adding xbuild to PATH. I didn't try it before that so I don't know.

@T2RKUS on windows? do you have mono installed? (xbuild)

I installed Mono yesterday but It wasn't added to my PATH.

The build script works as is, no need to change it on windows.

I will probably not be much help with regards to git I'm afraid, or even the entitas backend - i can help with front-end stuff though

Maybe part of the problem here is I don't know what xbuild and msbuild are, and probably also don't completely understand what you mean by mono,but I'm guessing your not talking about MonoDevelop? 😳
So do I need to install this?: http://www.mono-project.com/download/ and should it be the 32-bit or 64-bit version?
It would also seem that constant UNITY="/Applications/Unity/Unity.app/Contents/MacOS/Unity" is useless on Windows and should be changed.

@IsaiahKelly Now that I have it working on my machine I'll write a build and copy script in bash which uses the build script then copies everything to where it needs to go. basically a unity template demployement script if you like then upload it here.

@sschmid you could do a bash -version and change the UNITY variable based on whether the terminal returns "i686-pc-msys", "msys" or something for windows.

Yeah, I was already thinking there must be a way to make the scripts execute differently on Windows. That would help newbs like myself a lot.

hmm, Just wondering whether I should make use of the deploy script and call that after build then unzip that file into the assets folder defined in my build script.

#!/bin/bash -e
echo $0
source "$(cd $(dirname ${BASH_SOURCE[0]}) && pwd)/consts"

# User Configuration

TEMPLATE_ROOT="${ROOT}/EntitasTemplate"

# Script Configuration

ASSETS="${TEMPLATE_ROOT}/Assets"
ASSETS_LIBS="${ASSETS}/Libraries"
ASSETS_ENTITAS_ROOT="${ASSETS_LIBS}/Entitas"
ASSETS_ENTITAS="${ASSETS_ENTITAS_ROOT}/Entitas"
ASSETS_ENTITAS_EDITOR="${ASSETS_ENTITAS}/Editor"
ASSETS_ENTITAS_MIGRATION="${ASSETS_ENTITAS_ROOT}/Migration"
ASSETS_ENTITAS_EDITOR_IMAGES="${ASSETS_ENTITAS_EDITOR}/Images"

${SCRIPTS}/update

CP="cp -f"

# Copy Libraries  ---------------------------------------------------------

mkdir $ASSETS_LIBS
mkdir $ASSETS_ENTITAS_ROOT $ASSETS_ENTITAS $ASSETS_ENTITAS_EDITOR $ASSETS_ENTITAS_EDITOR_IMAGES
mkdir $ASSETS_ENTITAS_MIGRATION

# Entitas

CP "${ROOT}/${ES}/${BIN}/${ES}.dll" $ASSETS_ENTITAS
CP "${ROOT}/${ES_UTILS}/${BIN}/${ES_UTILS}.dll" $ASSETS_ENTITAS
CP "${ADDONS}/${ES_U}/${BIN}/${ES_U}.dll" $ASSETS_ENTITAS
CP "${ADDONS}/${ES_UE}/${BIN}/${ES_UE}.dll" $ASSETS_ENTITAS_EDITOR

# Entitas Code Generation

CP "${ADDONS}/${ES_CG}/${BIN}/${ES_CG}.dll" $ASSETS_ENTITAS_EDITOR
CP "${ADDONS}/${ES_CG_ATTR}/${BIN}/${ES_CG_ATTR}.dll" $ASSETS_ENTITAS
CP "${ADDONS}/${ES_CG_G}/${BIN}/${ES_CG_G}.dll" $ASSETS_ENTITAS_EDITOR
CP "${ADDONS}/${ES_CG_P}/${BIN}/${ES_CG_P}.dll" $ASSETS_ENTITAS_EDITOR
CP "${ADDONS}/${ES_CG_UE}/${BIN}/${ES_CG_UE}.dll" $ASSETS_ENTITAS_EDITOR

# Entitas Migration

CP "${ADDONS}/${ES_MIG}/${BIN}/${ES_MIG}.dll" $ASSETS_ENTITAS_MIGRATION
CP "${ADDONS}/${ES_MIG}/${BIN}/${ES_MIG}.dll" $ASSETS_ENTITAS_EDITOR
CP "${ROOT}/${ES_UTILS}/${BIN}/${ES_UTILS}.dll" $ASSETS_ENTITAS_MIGRATION
CP "${ADDONS}/${ES_MIG_CLI}/${BIN}/${ES_MIG_CLI}.exe" $ASSETS_ENTITAS_MIGRATION
CP "${ADDONS}/${ES_MIG_UE}/${BIN}/${ES_MIG_UE}.dll" $ASSETS_ENTITAS_EDITOR

# Entitas Visual Debugging

CP "${ADDONS}/${ES_VD_CG_P}/${BIN}/${ES_VD_CG_P}.dll" $ASSETS_ENTITAS_EDITOR
CP "${ADDONS}/${ES_VD_U}/${BIN}/${ES_VD_U}.dll" $ASSETS_ENTITAS
CP "${ADDONS}/${ES_VD_UE}/${BIN}/${ES_VD_UE}.dll" $ASSETS_ENTITAS_EDITOR

# Entitas Blueprints

CP "${ADDONS}/${ES_BP}/${BIN}/${ES_BP}.dll" $ASSETS_ENTITAS
CP "${ADDONS}/${ES_BP_CG_P}/${BIN}/${ES_BP_CG_P}.dll" $ASSETS_ENTITAS_EDITOR
CP "${ADDONS}/${ES_BP_U}/${BIN}/${ES_BP_U}.dll" $ASSETS_ENTITAS
CP "${ADDONS}/${ES_BP_UE}/${BIN}/${ES_BP_UE}.dll" $ASSETS_ENTITAS_EDITOR

# Fabl

CP "${ADDONS}/${ES_CG_G}/${BIN}/fabl.dll" $ASSETS_ENTITAS

# Copy Images  ---------------------------------------------------------

cp "${ADDONS}/${ES_UE}/${ES_UE}/Editor/Images/"* $ASSETS_ENTITAS_EDITOR_IMAGES
cp "${ADDONS}/${ES_MIG_UE}/${ES_MIG_UE}/Editor/Images/"* $ASSETS_ENTITAS_EDITOR_IMAGES
cp "${ADDONS}/${ES_VD_UE}/${ES_VD_UE}/Editor/Images/"* $ASSETS_ENTITAS_EDITOR_IMAGES

Try putting this script in the Scripts folder of the repo and change the user config to where your project is.

@T2RKUS Thanks for all your help on this! I installed Mono 64-bit (Preview, no GTK#) and added this PATH:

But get the same line 6: xbuild: command not found error again. Is this the wrong version of mono or am I setting the PATH incorrectly?

did restart your computer or Windows Explorer after you set the variable.

Duh!

It should look something like this: http://imgur.com/a/2F8Cy

Did the Task Manger restart trick, but that didn't work. So I restarted and that didn't help either. 😞
I've also checked the Scripts/consts definitions as Simon suggested, but UNITY is the only incorrect path I can see.  Can you tell/show me how you have that file setup? Here's mine at the moment:
#!/bin/bash -e
SCRIPTS="$(cd $(dirname ${BASH_SOURCE[0]}) && pwd)"
ROOT="${SCRIPTS}/.."

My script just calls the build script in the scripts directory so you should have it alongside
update
build
pack etc.
it loads the consts file, builds, then copies the dll/images to the folder specified in my script. you can create a new file called update_unity in that scripts folder and run it from there.

@T2RKUS I already placed your script there and named it "temp", but have now changed it to "update_unity"
Installed mono 32-bit and and changed PATH to that installation and removed "xbuild.exe" since your screenshot doesn't show that in the path. Still doesn't work.
$ ./Scripts/update_unity
./Scripts/update_unity
/c/GitHub/Entitas/Scripts/update
/c/GitHub/Entitas/Scripts/build
/c/GitHub/Entitas/Scripts/build: line 6: xbuild: command not found

I'm wondering if my issue here is with the git working directory path? Simon doesn't think  /c/GitHub/Entitas is a valid path, but I'm not sure if this is true or just the way SourceTree handles it? The literal path is C:\GitHub\Entitas\ Is this path correct and how would I change it if it isn't?

I was just using the MSYS shell (which is what you are using if you start the terminal from Source Tree as you say) from my downloads folder and it worked fine there. I don't see what the problem could be. It was actually pretty easy to get working on my windows machine with minimum effort.

Btw, just committed "Using msbuild instead of xbuild"
6df2aa55b2e1a5d2517b4fd4c86e75c4d551d7b4

sorry, have to revert since travis ci doesn't use mono 5.0 yet

Hey! It's me again 🙂. I recently reinstalled Windows 10, Unity 2017.1 and Visual Studio 2017 and decided to give this another shot. So I downloaded the Entitas source and opened it up again. Tried running Tests again, but I believe it gave me an error related to the build target or something, so I changed the target to x86 and it finally ran! However, now I'm getting an error message on line 17 of check_namespaces.cs with the following message:
NUnit.Framework.AssertionException: '  Expected: greater than 80

  But was:  0

No idea what this actual means, so I totally stuck again. 😞


RFM - quaternion

I am the developer of this application; however, I'm not a Windows user so cannot honestly be a maintainer of a Chocolatey package. I reached out to a previous maintainer (xdimquax) but he did not return. Getting a Chocolatey package wasn't a priority for me so this was lying in the backyard for almost 5 years... anyway:
I'm happy to work together with whoever is willing to pick this package up - despite the passed years, an update process should be straightforward as the means of building the package didn't change much. Ultimately, generation of new Chocolatey packages can be incorporated into my CI pipeline for even lower maintenance; but someone will have to be around to reinvigorate the package first, and validate the artifacts from time to time.
I DON'T Want To Become The Maintainer

[x] I have followed the Package Triage Process and I do NOT want to become maintainer of the package;
[x] There is no existing open maintainer request for this package;

Checklist

[x] Issue title starts with 'RFM - '

Existing Package Details
Package URL: https://community.chocolatey.org/packages/quaternion
Package source URL: https://github.com/xdimquax/choco
Date the maintainer was contacted (in YYYY-MM-DD): 2019-08-12 (as I said, almost 5 years ago)
How the maintainer was contacted: https://community.chocolatey.org/packages/quaternion/<IP_ADDRESS>-beta1/ContactOwners

As the last contact with the maintainer was 5 years ago, can you try again?

He deleted his account from GitHub, and his packages were up for grabs even then. But I can try once again as a formality...

If that's the case then it's reasonable to assume the maintainer has stopped maintaining their packages. There's no need, in my opinion, to contact them under these circumstances.

Thae package is ready, i've asked to be added as maintainer.
Have a nice day.

That's weird, as the other packages have been assigned as maintainer, this one, even with the contact admins on the 4th i still haven't received any answers.
@pauby , can you help ?

I don't know why that was missed. I see messages today, 7 and 4 March.
I've added you as a maintainer.

Thank you very much

Done and approved
Have a nice day


Errors using Java 8, Spigot 1.8.9.

[22:30:31] [Server thread/ERROR]: Could not pass event InventoryClickEvent to DecoHeads v1.3 org.bukkit.event.EventException 	at org.bukkit.plugin.java.JavaPluginLoader$1.execute(JavaPluginLoader.java:310) ~[spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:62) ~[spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at org.bukkit.plugin.SimplePluginManager.fireEvent(SimplePluginManager.java:502) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:487) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at net.minecraft.server.v1_8_R3.PlayerConnection.a(PlayerConnection.java:1630) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at net.minecraft.server.v1_8_R3.PacketPlayInWindowClick.a(SourceFile:31) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at net.minecraft.server.v1_8_R3.PacketPlayInWindowClick.a(SourceFile:9) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at net.minecraft.server.v1_8_R3.PlayerConnectionUtils$1.run(SourceFile:13) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72] 	at net.minecraft.server.v1_8_R3.SystemUtils.a(SourceFile:44) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at net.minecraft.server.v1_8_R3.MinecraftServer.B(MinecraftServer.java:715) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at net.minecraft.server.v1_8_R3.DedicatedServer.B(DedicatedServer.java:374) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at net.minecraft.server.v1_8_R3.MinecraftServer.A(MinecraftServer.java:654) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at net.minecraft.server.v1_8_R3.MinecraftServer.run(MinecraftServer.java:557) [spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72] Caused by: java.lang.NullPointerException 	at com.rayzr522.decoheads.DHListener.onInventoryClick(DHListener.java:39) ~[?:?] 	at sun.reflect.GeneratedMethodAccessor549.invoke(Unknown Source) ~[?:?] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_72] 	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_72] 	at org.bukkit.plugin.java.JavaPluginLoader$1.execute(JavaPluginLoader.java:306) ~[spigot-1.8.9.jar:git-Spigot-db6de12-07c3001] 	... 15 more

Sorry, this is actually caused by something I didn't know about at the time of writing the plugin (originally). I assume this happened while you were in creative?

Yes. :(

Sorry for the inconvenience. I will fix it tomorrow and then give you a dev build.
 I'll fix it up for ya :D

Oh okay, thanks ! :D And when do you think, the 1.4 will be released ? :D (yeah I'm so impatient xD)
Thanks a lot ! :D

Not sure, tbh I haven't even touched it in a month or two. Now that I have some stuff to fix / feature requests from you I will start working on it again.
So probably a week or two, I work pretty quick :wink:

Well I might not the categories system done at that point... it depends, I may or may not be able to modify the GUI to allow that functionality. We'll see :grin:

Oh yeahs thanks ! :D
Okay good luck ! :) Fantastic plugin, the best I saw.

Thank you :grin:
People like you are what make developing worth it :smile:

:D.

Any news ? :D

Sorry! I've been busy with my dev group, and the last few days I've been (shudder) cleaning out my hard drive. I used to have only 98GB of free space, now I have something like 240GB!
Anyways, I should have more free time, I'll try and get to this in the next 7 days. Thanks for your patience :P

Okay, no problem. :dagger:
Thaanks. You're welcome. :D

I've fixed this specific issue and localized the messages/text in the plugin into a config file. I still need to tweak a couple things, but after that I'll release v1.3.1. It should be out on Monday.
If you know how to use Maven, you can compile itself by simply cloning the project, going into the project directory with your terminal/command prompt, and running mvn clean package.
Do keep in mind that it's not ready for release yet, and I will most likely change some things with the localization config. I'm not a huge fan of the current formatting.
Also just one more note, I will be changing to use Java 8 after this release. I'm sorry if some people don't have Java 8, but it is ridiculous to ignore the huge benefit to me as a developer just for a few people, especially since nearly all server hosts are using Java 8 now and people with homebrew servers can upgrade in a matter of 15-Ish minutes.
Big wall o' text for ya :P

Hello,
I don't know how to use Maven. :( I will wait for the release so. ;)
I use Java 8 so this isn't a problem for me. :D
Thanks ! :)

Fixed in v1.3.1

Yep.


formatter Prop is executing twice with one value slider

formatter Prop is executing twice (and one of them is undefined) when you have single value slider. In my opinion it should run only once, when you have one value slider.
This pice of code:
                formatter: function (v) {
                     console.log(v);
                     return v + '$';
                 }

Prints in console:
924000
undefined

Fixed in the latest version


Add tooltips to browser bar UI

I tried to make the UI as intuitive as possible, and did initially add tooltips to the input labels. But my CSS skills aren't the best, and it messed up the layout.
I suppose this should be something fairly simple to anyone with frontend experience. Either to fix the CSS, or switch to using the title attribute (and removing the unnecessary CSS)
A suitable PR would add tooltips to the keywords and interval label documenting what they do.

Hi. The keyword textarea already has a description. Do you want to remove that and add a tooltip or something else?

Could you please assign this to me ?

@ameybhavsar24 You can leave the text in the textarea tag. It'll only show up on first use once #3 is implemented. But you can reuse the same text in the popup. Or use a summarized form of the description given in README.md
@ilkerunal Sorry, I'm going to assign on an FCFS basis


Option to remove "All Channels" category

Hi, is it possible to have the option to remove the "All Channels" category from menubar.

seems like it is the core driving the skin. would need to find source of the iptv core to be able to change it..i wish i can find it..lol

This option was implemented in IPTV Core 3.9.4. Please use the following intent extra to hide "All Channels" tab: intent.putExtra("hide_all_channels_tab", true)


Portal does (not) exist

According to your testing, does the issue occur with ONLY Immersive Portals mod and Farbic API?
Yes
Describe the issue
The portal does not exist but it also does.
Edit: For more clarification: I can see through the portal, but not enter it.
Post the log here
No response

What do you mean by "cannot enter it"? Crash when entering?

Well, if I try to enter it, the

If I try to enter it, I just go to the other face of the portal.

Does it happen to all nether portals? Does it happen to end portals?

Close because no reply.


Component errors, component auth

This fixes some error-handling on component authentication and adds "anonymous" and "ticket" authentication methods.

Hmm, I have no idea why the py3.3 builders are sad


Refactors VariableList to simplify code and use hooks.

When rewriting the docs client to use hooks, much of it was straight-forward, except for VariableList. The state management, with children having state managed by parents and propagating changes up, did not follow the model very well. It was also confusing for me to reason about. I think the biggest reason to have parents manage state is so when children are not rendered, the state stays - using CSS visibility instead of conditional rendering of children removes this problem and allows state to stay close to who owns it, the bean itself, and we don't need a global, indexed registry of bean state. If these were extremely huge amounts of DOM we may prefer conditional rendering, but these shouldn't see performance issues using CSS instead.
I also removed hasLocation prop since it can be derived from the variables list, which was simpler for me to reason about.
I'll send a follow-up PR with the rest of the migration, the rest is mechanical so wanted to separate out this more significant change.

@anuraaga Thanks! Looking forward to your hooks migration PR. :wink:


Can not resolve symbol LocalDataSource

Hello! I have just added the code from your Quick Start tutorial and my Idea PhpStorm (10.0.4) underscore LocalDataSource when i'm trying to import it from ng2-smart-table and shows tip "Can not resolve symbol LocalDataSource":
import { Ng2SmartTableModule, LocalDataSource } from 'ng2-smart-table';
Everything actually works good as aspected but this underscore is a little bit annoying. And because of it also not available autocomplete in Idea.
My environment:
typescript: 2.0.2,
angular:  2.0.0
angular-cli: 1.0.0-beta.14
node: 5.6.0
os: darwin x64
"ng2-smart-table": "^0.3.1"
P.s Thanks for this useful  component

In my project i have import { Ng2SmartTableModule } in my Module and I have LocalDataSource import statement in my component. If you do it like this it'll work.


Implement etcd3 adapter for config

Interface is defined here: model/config.go
Sample implementation: adapter/config/memory
Reusable platform code goes to platform/consul (see platform/kube)
The implementation should also provide a cache controller (similar to upcoming adapter/config/thirdpartyresource)

cc @louiscryan issue for etcd plugin

@geeknoid @rshriram
Given the design discussion yesterday we should get cracking on this :)

Seems to be pretty straight forward. I'll have a PR soon.

What's the adapter model we want to use here? In proc or RPC-based? In-proc
is convenient as it avoids the need to run a separate service, but it has
the downside of requiring rebuilding the storage server in order to support
a new storage backend.
Basically, I'd like to see an actual design articulated for our (unnamed)
storage component. What is its API surface, what is the API surface and
compositional model of adapters, what is the configuration story for the
storage component itself and its adapters, etc.
On Wed, Jun 14, 2017 at 12:07 PM, Isaiah Snell-Feikema <
<EMAIL_ADDRESS>wrote:

Seems to be pretty straight forward. I'll have a PR soon.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/pilot/issues/850#issuecomment-308528825, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVucHeX9kmTjiO9lDkSjwleUR6YkHnySks5sEC98gaJpZM4N1i4c
.

In process adapter model. There is a client cache involved, so adding RPC prevents sharing memory. This is not set in stone, but I don't see it wise to spend time designing RPC when we have only 1-3 storage adapters.

API surface is defined in Pilot based on the observed use of the storage component (see model/config.go. This is a minimal API and does not cater to Mixer. We can stack Pilot's API on top Istio config API.

We need aggregation over disjoint configuration types in Pilot. Ingresses are not owned by Pilot, and we need to compose them with route rules. See adapter/config/aggregate.

Configuration story for adapters: none. We'll need an opaque struct for bootstrapping the system (perhaps in the global mesh config), but it's just a blob, with no handles to manipulate it.

On Wed, Jun 14, 2017 at 3:13 PM Martin Taillefer<EMAIL_ADDRESS>wrote:

What's the adapter model we want to use here? In proc or RPC-based?
In-proc

is convenient as it avoids the need to run a separate service, but it has

the downside of requiring rebuilding the storage server in order to support
a new storage backend.
How about we start with in-proc as part of Galley, and then slowly
transition to out of process as need arises ?

Basically, I'd like to see an actual design articulated for our (unnamed)
storage component.

do we need another name and another service for this?

What is its API surface, what is the API surface and
compositional model of adapters, what is the configuration story for the
storage component itself and its adapters, etc.

My two cents: we start simple. Otherwise, this component is going to end up
blocking everything else.
~shriram

Pilot should not talk directly to the backend storage systems.
It should go thru the Galley watcher API.
When Galley is there, we will rip out all this storage stuff from Pilot.
Pilot will continue to access service discovery backends.

Can we keep this issue focused on pilot+etcd3 please? I'd like to avoid going astray towards the more general config design. We clearly need a direct etcd3 adapter in pilot for service registry, delta to make it work for config is minimal.

I don't think this is a good use of our resources.
Architecturally, we want there to be a single storage abstraction that all
components talk to. We should design this abstraction carefully and
deliberately for this purpose, as opposed to intentionally spending
resources designing and implementing something we know is insufficient for
the sake of very short term expediency.
I expect that we will have a stack where
For config:
Pilot -> Galley -> Storage Component -> Storage Adapter -> Storage Backend
Mixer -> Galley -> Storage Component -> Storage Adapter-> Storage Backend
For other storage needs:
Pilot -> Storage Component -> Storage Adapter -> Storage Backend
As I said, we need to design the API surface for the Storage Component and
the Storage Adapter, and I hope that we can do this design work once.
On Wed, Jun 14, 2017 at 1:52 PM, Kuat<EMAIL_ADDRESS>wrote:

Can we keep this issue focused on pilot+etcd3 please? I'd like to avoid
going astray towards the more general config design. We clearly need a
direct etcd3 adapter in pilot for service registry, delta to make it work
for config is minimal.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/pilot/issues/850#issuecomment-308554597, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVucHVBAjf2yMebKoy3jIHd2B0p2iAG1ks5sEEgAgaJpZM4N1i4c
.

You might be right, but config is integral to pilot and we need something before galley gets ready which doesn't seem to get any closer. We can't break config in Pilot since the rest 90% of code depends on it and doesn't receive enough attention with all the config discussion.

Do you really need to make any enhancement in this space until Galley?
Can't you just keep what you've been using to date?
On Wed, Jun 14, 2017 at 2:56 PM, Kuat<EMAIL_ADDRESS>wrote:

You might be right, but config is integral to pilot and we need something
before galley gets ready which doesn't seem to get any closer. We can't
break config in Pilot since the rest 90% of code depends on it and doesn't
receive enough attention with all the config discussion.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/pilot/issues/850#issuecomment-308569938, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVucHQofb_H24aAhEMkVa5v7S91nyOFuks5sEFcRgaJpZM4N1i4c
.

TPR are deprecated from 1.7. What we have will no longer work.

And we can't just stick with 1.6 for a month or two until Galley and/or the
storage components are up and running? Is there some urgency for us to
upgrade?
On Wed, Jun 14, 2017 at 2:59 PM, Kuat<EMAIL_ADDRESS>wrote:

TPR are deprecated from 1.7. What we have will no longer work.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/pilot/issues/850#issuecomment-308570497, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVucHUJdciBxri19pNUR-8MnqXD40y2Sks5sEFezgaJpZM4N1i4c
.

We still have time till 1.7
Etcd3 is not for service registry. Let's not have our service registry
stuff until we see a real need for it. So, if we don't need our own service
registry, we don't need direct access to data store. We can have readonly
service registry adapters at pilot. Starting with consul.
@geeknoid I didn't realize that this was pilot etcd3 we were talking about.
I agree we need the API layer for galley. We could do the simple API layer,
with adapters built into galley. And evolve from there. Adding multiple
meta layers (storage component for example) seems overkill for starters.
Galley's API surface can be stable for pilot and mixer while we can
independently evolve the storage interfaces since there is no consumer of
that interface. We are the ones who are going to be building adapters and
for the foreseeable future, I don't see us building any other adapter than
etcd3.
To kuats point, we could start with the config surface in pilot as a
starting point for galley API and go from there, so as to unblock work on
multiple fronts. We can tolerate smallish API changes. It's part of the dev
churn. It might seem adhoc but it would get us moving.
Thoughts ?
On Wed, Jun 14, 2017 at 5:59 PM Kuat<EMAIL_ADDRESS>wrote:

TPR are deprecated from 1.7. What we have will no longer work.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/pilot/issues/850#issuecomment-308570497, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH0qd7hWPmXjCwQ6v2avB-eTdGgCi53Fks5sEFeygaJpZM4N1i4c
.

--
~shriram

@geeknoid I will send a design doc for the Galley surface api and the storage backend.
There is a PR out for the api surface, I will send it in a design doc + storage back end.

Closing the issue, since the discussion diverged into galley again. Feel free to reopen when the time comes to build etcd adapters.

@rshriram I don't really care where adapters live. Give me working code.

no istio component should require the presence of a HA distributed store to function. they should all (pilot included) need to get config updates from time to time (and for service discovery have adapters to ingest the changes from the source of truth)

@geeknoid moving away from TPR is part of plan for next release for VM
support, hybrid etc. so as long as we are stuck with TPR, we can't make
progress on that front.
--
~shriram


Use a proper warning when a name is not found in a module

Closes #278
Let me know if this fix is OK. I had to add the log object to the DFSCollector class.
I haven't tried to update the scipy config as suggested in the issue. If we do that we need to make sure the docs all still get generated correctly. SciPy does a lot of weird dynamic stuff with its modules using module-level __dir__ and __getattr__.

Thanks !


Add a Wiki

This project needs explanation and how-tos, asap.

The initial content of the wiki has been added. See: Slope-ECS Wiki

The initial wiki has been completed.
Awaiting release.


⚠ Bofrost.at has degraded performance

In b629ed3, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1014 ms

Resolved: Bofrost.at performance has improved in ea20caa.


Python 3 compliant code

Since there is now a branch that has Python 2 compliant code, I believe that the master should hold Python 3 compliant code.

Check out PR https://github.com/Jamonek/Robinhood/pull/23 can't do MD at the moment. I'll edit


UI improvements

Sorting, searching, improved pause/resume, viewed/unviewed, and check boxes for performing actions on selected watches.

Looks like a failure in the tests/test_backend.py test

Thanks for the fast turnaround and feedback - will work the issues this week :-).

Made all the adjustments and solved the backend test failure.  The backend test checks to ensure the 'unviewed' class isn't present at a certain point with "assert b'unviewed' not in res.data", and my markup had the word unviewed in several href s and in visible text so it triggered it.  I changed it all to 'notviewed' and 'Mark as not viewed' so it's all good now even though it doesn't roll off the tongue as well, lol.

And of course I forgot to recheck the positioning of the search box close button, so please hold off until I fix it.  It's fine in firefox, but wraps to the next line in all the other major browsers :-/,

Reverted the search box clear button to simpler styling for browser compatibility.  That should be all for this PR.

Now that should be everything.

@ntmmfts nearly there :) can you check those changes i've asked for?

Looks like I've addressed all the changes and replied to them, but still shows 1 change requested.  Can you pls let me know if there's anything that I haven't completed yet?  Pushed "Requested adjustments" (c07a481) yesterday and I think it addressed everything.  Thanks!

Hey! OK a few things

Please edit your PR and aim it at the https://github.com/dgtlmoon/changedetection.io/tree/ui-improvements branch, otherwise it's almost impossible for me to also work with you on the improvements before committing them to master

2 .On lower resolutions there's a breakpoint to flow the tables into something more useful, but your checkboxes should flow more like "inline-block", or sit very far to the left (like in the first 5% of the table column)

move the 'search' box into the header bar, it wont hurt the layout as much then

Hide these until they are required (until you have checked one of the checkboxes, otherwise they dont need to be there)

Make them a different colour, they are too confusing to look at with your other organisational tabs

When you have more than a few tabs, your extra buttons get real weird, not sure how to solve this, but if you fix your branch to merge into my ui-improvements I'll find a way

I'm not sure about using an email/mail icon in this situation, since people would assume it already has something todo with email or notifications..

Awesome feedback - understand all and already starting to work on them. I knew the low-res/mobile needed work, thanks for reminding me.  Probably a few days to complete, and will will do the push/PR into a ui-improvements branch as requested.

Moved to ui-improvements branch and created a new PR.


[nodes] Add better error handling to processor and CLI

Adds better error information to graph execution, prints better errors in the CLI, and sends errors as events.

i would highly suggest making the change i .. suggested, but i'm not going to make it a blocker

Good call. Updated it to set_node_error.


[SharedModel] Skip date/time tests outside of Pacific TZ

Related Issue
Fixes #3609
Description
It's a bad idea to dynamically modify the timezone of the build machines where these unit tests sometimes run. To unblock us for now, I've added logic to skip running these specific tests when the hosting machine isn't in the Pacific timezone.
How Verified
Unit tests & debugger
Microsoft Reviewers: Open in CodeFlow

/azp run


handler import-cluster: timeout getting credential for cluster group, requeuing error in fleet-controller pods

Rancher Server Setup

Rancher version: 2.6head f40e542
Installation option (Docker install/Helm Chart): Helm

Information about the Cluster

Kubernetes version: 1.24.2
Cluster Type (Local/Downstream): local - 1.24.2 RKE1

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom) : Admin

Steps to reproduce:

RKE1 1.24 downstream cluster come sup Active, but timeout getting credential for cluster group seen in continuous Delivery page for this cluster
On cluster management page --> Cluster details --> related resources --> Cluster object is seen in Wait Applied/Error state
Errors seen in Rancher logs:

2022/07/06 03:38:02 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-mfhzj6w6: ClusterUnavailable 503: cluster not found, requeuing
2022/07/06 03:38:10 [INFO] cri-dockerd is enabled for cluster version [v1.24.2-rancher1-1]
2022/07/06 03:39:12 [INFO] cri-dockerd is enabled for cluster version [v1.24.2-rancher1-1]
2022/07/06 03:39:20 [ERROR] Failed to handle tunnel request from remote address <IP_ADDRESS>:34690 (X-Forwarded-For: <IP_ADDRESS>): response 401: failed authentication
2022/07/06 03:39:21 [INFO] cri-dockerd is enabled for cluster version [v1.24.2-rancher1-1]

And errors in fleet-controller pod:

time="2022-07-06T04:37:17Z" level=error msg="error syncing 'fleet-local/local': handler import-cluster: secrets \"local-cluster\" not found, requeuing"
--
Tue, Jul 5 2022 9:37:24 pm | time="2022-07-06T04:37:24Z" level=error msg="error syncing 'fleet-default/sowmya-linode-rke2': handler import-cluster: timeout getting credential for cluster group, requeuing"
Tue, Jul 5 2022 9:37:46 pm | time="2022-07-06T04:37:46Z" level=error msg="error syncing 'fleet-default/c-q69rl': handler import-cluster: timeout getting credential for cluster group, requeuing"
Tue, Jul 5 2022 9:37:56 pm | time="2022-07-06T04:37:56Z" level=error msg="error syncing 'cattle-fleet-system/fleet-controller': handler global-config: waiting on secret for service account cattle-fleet-system/fleet-controller-bootstrap, requeuing"

Fleet errors will be resolved separately via https://github.com/rancher/fleet/pull/850

Verified this is addressed in https://github.com/rancher/rancher/commit/cfba0bc4fe19177cef627d36ad86744b62e5e751.
ENVIRONMENT DETAILS

Rancher Install: Helm
Rancher version: v2.6-head https://github.com/rancher/rancher/commit/cfba0bc4fe19177cef627d36ad86744b62e5e751
Browser: Chrome

TEST RESULT
PASS
VERIFICATION STEPS

Setup Rancher and navigate to the UI in a browser.
Created a standard user and login to Rancher as that user.
Provisioned a downstream RKE1 1.24 node driver cluster and verified that it came up as Active:

Verified in the Rancher logs that the original error is not seen.


Adds empty into schema for node

Please check if the PR fulfills these requirements

[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Other... Please describe:

What is the current behavior? (You can also link to an open issue here)
Any { node: { fs: 'empty' } } will not allow Webpack to run
What is the new behavior?
Any { node: { fs: 'empty' } } will run as it did in beta.22
Does this PR introduce a breaking change?

[ ] Yes
[x] No

If this PR contains a breaking change, please describe the following...

Impact:
Migration path for existing applications:
Github Issue(s) this is regarding:

Other information:
Support for empty was broken through the schema, so I've added it back. This is exactly why Webpack should only be warning and not throwing errors as I now need to lock down to a version before the schema enforcement.

Thanks


Sliding Window logic fix and additional token length in larger candidates

I was trying to stress test and our sliding window logic seemed to be failing in irregular windows that might not align perfectly with the candidate set size (which is a majority of cases, besides our settings like (20, 10) on candidates of 20/100, etc.).
This should fix it.
Additionally, some non-breaking changes (for our (20, 10) experiments) when we are dealing with smaller windows (you don't need to fix a 300 token limit).

LGTM,! given that my last comment is not a blocker
Thank you!


Nargo fmt incorrectly formats nested structures

Aim
Format this code:
    assert_eq(
        result, [
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2
        ]
    );

Expected Behavior
Do nothing
Bug
nargo fmt removes one level of tabs incorrectly
    assert_eq(
        result, [
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2
    ]
    );

To Reproduce

Copy this code part
Run nargo fmt

Project Impact
Nice-to-have
Impact Context
Very low pri
Workaround
None
Workaround Description
No response
Additional Context
No response
Installation Method
Binary (noirup default)
Nargo Version
noirc version = 0.23.0+5be9f9d7e2f39ca228df10e5a530474af0331704
NoirJS Version
No response
Would you like to submit a PR for this Issue?
None
Support Needs
No response

This also shows in imports:
use crate::fixtures::{
    london::crypto_punks::header::{block_header_partial as london_block_header_partial, block_header_rlp as london_block_header_rlp},
    frontier::zero::header::{
    block_header_partial as frontier_zero_block_header_partial,
    block_header_rlp as frontier_zero_block_header_rlp
},
    frontier::first::header::{
    block_header_partial as frontier_first_block_header_partial,
    block_header_rlp as frontier_first_block_header_rlp
},
    paris::usdc::header::{block_header_partial as paris_block_header_partial, block_header_rlp as paris_block_header_rlp},
    paris::usdc::header::{hash, number, state_root, transactions_root, receipts_root}
};


Update how-to-label-images.md

@lobrien

@j-martens : Thanks for your contribution! The author(s) have been notified to review your proposed change.


USHIFT-1972: Fix service restart race condition for services that exit

When services are started synchronously using systemctl start or systemctl restart, they go through start and then running sub-state.
The race condition was on services that do not have a "long" running phase, i.e. greenboot. Those services run until completion and exit. So, there is a start, very short running and exited sub-states for those services.
To fix the race condition, services can be started and restarted using --no-block option. This allows for a quick return from the systemctl command and then status check for the start or running sub-states. If the service fails later, this needs to be checked using other service-specific checks, which are performed in the tests anyway.
The fixes include:

Servicectl keyword now uses the --no-block option to allow immediate return in case of start and restart commands. This allows for proper service status check in case of greenboot, etc. that can terminate and not be running.
Remove Systemctl With Retry keyword

/test microshift-metal-tests microshift-metal-tests-arm

/test metal-periodic-test

/test metal-periodic-test

/test metal-periodic-test

/test metal-periodic-test

/test metal-periodic-test

/test microshift-metal-tests microshift-metal-tests-arm

/lgtm

/cherry-pick release-4.15


Feedback from Shadow Client for MQTT Disconnect

Desired Feature Description
Because of the Device Shadow client consuming the MQTT callback, no feedback is provided to the application level if an MQTT disconnect occurs. The callback functionality only reaches back as far as prvShadowMQTTCallback. This function could/should report the disconnect back to the application so it can reconnect, if desired.
Describe the solution you would like
Shadow Client could contain reference to an application call-back to handoff the MQTT callback to if there is nothing of interest for the Shadow Client in the callback.
This would allow the application to respond to eMQTTAgentDisconnect events in an alternative way other than simply printing "Warning: got an MQTT disconnect message.".
Additional context
(I believe) Currently without a way to detect an MQTT disconnect, the application can not detect the disconnect without first failing to publish. If this is incorrect, I do not see another way the application can detect this failure.
Thank you! 😄

@tombresson: We do have a new version of MQTT and Shadow library which provide a finer level of control as far as the user callback is concerned. It exists in BLE-Beta branch and is currently work in progress: https://github.com/aws/amazon-freertos/blob/feature/ble-beta/lib/include/aws_iot_mqtt.h
I would suggest you adopting the new library, and please let us know of any positive/negative feedback.
Documentation is here: https://docs.aws.amazon.com/freertos/latest/userguide/freertos-mqtt-2.html
API reference is here: https://docs.aws.amazon.com/freertos/latest/lib-ref/html3/mqtt/index.html
We are very interested in your feedback.

By the way, I apologize, but this issue/feature request appears to already be mentioned in #322.

Hello, have you had a change to look at the latest implementation of the shadow library mention by lt72? I will be closing this issue in the meantime.


Migration fails

https://github.com/input-output-hk/cardano-db-sync/blob/9503f09a426243307706531f0d0e53989ce15377/schema/migration-2-0012-20210723.sql#L9
Running migration from to c334c12 to 9503f09 fails with error:
Running : migration-2-0012-20210723.sql
psql:/nix/store/cf686ccywfr6jinrmw3j5bb4bahdwka3-cardano-db-sync-schema/schema/migration-2-0012-20210723.sql:19: ERROR:  column "ex_unit_number" contains null values
CONTEXT:  SQL statement "ALTER TABLE "tx" ADD COLUMN "ex_unit_number" uinteger NOT NULL"
PL/pgSQL function migrate() line 7 at EXECUTE
ExitFailure 3

I think the problem is not setting default values explicitly in SQL, e.g.:
ALTER TABLE "tx" ADD COLUMN "ex_unit_number" uinteger NOT NULL DEFAULT 0;
ALTER TABLE "tx" ADD COLUMN "ex_unit_fee" lovelace NOT NULL DEFAULT 0;
ALTER TABLE "tx" ADD COLUMN "script_size" uinteger NOT NULL DEFAULT 0;

Same problem with https://github.com/input-output-hk/cardano-db-sync/blob/master/schema/migration-2-0013-20210802.sql#L9
Running : migration-2-0013-20210802.sql
psql:/nix/store/1gfbx5nccb95i9cbngl1kgg9mq1irls4-cardano-db-sync-schema/schema/migration-2-0013-20210802.sql:18: ERROR:  column "seconds" contains null values
CONTEXT:  SQL statement "ALTER TABLE "epoch_sync_time" ALTER COLUMN "seconds" SET NOT NULL"
PL/pgSQL function migrate() line 7 at EXECUTE
ExitFailure 3

Fixing all of the above with
diff --git a/schema/migration-2-0012-20210723.sql b/schema/migration-2-0012-20210723.sql
index 85e06a9..37439c8 100644
--- a/schema/migration-2-0012-20210723.sql
+++ b/schema/migration-2-0012-20210723.sql
@@ -6,9 +6,9 @@ DECLARE
 BEGIN
   SELECT stage_two + 1 INTO next_version FROM schema_version ;
   IF next_version = 12 THEN
-    EXECUTE 'ALTER TABLE "tx" ADD COLUMN "ex_unit_number" uinteger NOT NULL' ;
-    EXECUTE 'ALTER TABLE "tx" ADD COLUMN "ex_unit_fee" lovelace NOT NULL' ;
-    EXECUTE 'ALTER TABLE "tx" ADD COLUMN "script_size" uinteger NOT NULL' ;
+    EXECUTE 'ALTER TABLE "tx" ADD COLUMN "ex_unit_number" uinteger NOT NULL DEFAULT 0' ;
+    EXECUTE 'ALTER TABLE "tx" ADD COLUMN "ex_unit_fee" lovelace NOT NULL DEFAULT 0' ;
+    EXECUTE 'ALTER TABLE "tx" ADD COLUMN "script_size" uinteger NOT NULL DEFAULT 0' ;
     -- Hand written SQL statements can be added here.
     UPDATE schema_version SET stage_two = next_version ;
     RAISE NOTICE 'DB has been migrated to stage_two version %', next_version ;
diff --git a/schema/migration-2-0013-20210802.sql b/schema/migration-2-0013-20210802.sql
index abafa76..c0b51fb 100644
--- a/schema/migration-2-0013-20210802.sql
+++ b/schema/migration-2-0013-20210802.sql
@@ -6,6 +6,7 @@ DECLARE
 BEGIN
   SELECT stage_two + 1 INTO next_version FROM schema_version ;
   IF next_version = 13 THEN
+    EXECUTE 'UPDATE "epoch_sync_time" SET "seconds" = 0 WHERE "seconds" IS NULL;' ;
     EXECUTE 'ALTER TABLE "epoch_sync_time" ALTER COLUMN "seconds" SET NOT NULL' ;
     EXECUTE 'ALTER TABLE "epoch_sync_time" ALTER COLUMN "seconds" TYPE word63type' ;
     -- Hand written SQL statements can be added here.

results in:
Failed reading Alonzo genesis file "/nix/store/3ypkn16yiw7sajpp4ash3z4az70q4zs3-alonzo-genesis.json": There was an error parsing the genesis file: /nix/store/3ypkn16yiw7sajpp4ash3z4az70q4zs3-alonzo-genesis.json Error: Error in $.costModels['"PlutusV1"']: Error decoding Language: "\"PlutusV1\""

If you just want to run db-sync you should avoid the master branch and instead use tagged releases.
If you insist on running from the master branch, migrations will not always work. When they do not work, you only option is to drop the database and resync.

Thanks a lot for clarification.


Update Regions “2022-09-26-adsf”

Automatically generated by Netlify CMS

👷 Deploy Preview for gilded-sunburst-0982f4 processing.

Name
Link

🔨 Latest commit
b1b8295f37598715684ca13e52c1e88e3f398cae

🔍 Latest deploy log
https://app.netlify.com/sites/gilded-sunburst-0982f4/deploys/6331aee8d9b8710008ca628b


[Bug] Error when searching self join relationships.

Summary of problem or feature request
I have a problem when search data
Code snippet of problem
My Model
class Appl_menu extends Model
{
    protected $table = 'appl_menus';
    public $timestamps = false;

    protected $fillable = ['menu_id', 'menu_name', 'menu_module_id', 'menu_file_name', 'menu_id_parent',
        'menu_seqn', 'menu_enable_sts', 'created_by', 'created_date', 'updated_by', 'updated_date'];

    protected $primaryKey = 'menu_id';

    public function nullRelation()
    {
        return null;
    }

    public function menus ()
    {
        return $this->belongsTo(Appl_menu::class,'menu_id_parent');
    }

    public function menuss ()
    {
        return $this->hasMany(Appl_menu::class,'menu_id_parent','menu_id');
    }
}

my controller
public function index(Request $request, Builder $htmlBuilder)
    {
        if ($request->ajax()){
            $menuss = Appl_menu::with('modules', 'menus');
            //->orderBy('module_id');
            return Datatables::of($menuss)
                ->addColumn('action', function ($menu) {
                    return view('datatable._action', [
                        'model' => $menu,
                        'form_url' => route('menus.destroy', $menu->menu_id),
                        'edit_url' => route('menus.edit', $menu->menu_id),
                        'confirm_message' => 'Yakin mau menghapus menu ' . $menu->menu_name . '?'
                    ]);
                })->make(true);
        }

        $html = $htmlBuilder
            ->addColumn(['data' => 'menu_id', 'name' => 'menu_id', 'title' => 'ID'])
            ->addColumn(['data' => 'menu_name', 'name' => 'menu_name', 'title' => 'Nama Menu'])
            ->addColumn(['data' => 'modules.module_name', 'name' => 'modules.module_name', 'title' => 'Module'])
            ->addColumn(['data' => 'menus.menu_name', 'name' => 'menus.menu_name', 'title' => 'Parent', 'defaultContent' => ''])
            ->addColumn(['data' => 'action', 'name' => 'action', 'title' => '', 'orderable' => false, 'searchable' => false]);

        return view('menus.index')->with(compact('html'));
    }

Error output when search
SQLSTATE[42601]: Syntax error: 7 ERROR: syntax error at or near "as"
LINE 1: ...ST("appl_menus" as "laravel_reserved_0.menu_name" as TEXT)) ...
^ (SQL: select count(*) as aggregate from (select '1' as "row_count" from "appl_menus" where (LOWER(CAST("appl_menus"."menu_id" as TEXT)) LIKE %se% or LOWER(CAST("appl_menus"."menu_name" as TEXT)) LIKE %se% or (select count(1) from "appl_modules" where "appl_menus"."menu_module_id" = "appl_modules"."module_id" and LOWER(CAST("appl_modules"."module_name" as TEXT)) LIKE %se%) >= 1 or (select count(1) from "appl_menus" as "laravel_reserved_0" where "laravel_reserved_0"."menu_id" = "appl_menus"."menu_id_parent" and LOWER(CAST("appl_menus" as "laravel_reserved_0.menu_name" as TEXT)) LIKE %se%) >= 1)) count_row_table)

javascript output
<script type="text/javascript">(function(window,$){window.LaravelDataTables=window.LaravelDataTables||{};window.LaravelDataTables["dataTableBuilder"]=$("#dataTableBuilder").DataTable({"serverSide":true,"processing":true,"ajax":"","columns":[{"data":"menu_id","name":"menu_id","title":"ID","orderable":true,"searchable":true},{"data":"menu_name","name":"menu_name","title":"Nama Menu","orderable":true,"searchable":true},{"data":"modules.module_name","name":"modules.module_name","title":"Module","orderable":true,"searchable":true},{"data":"menus.menu_name","name":"menus.menu_name","title":"Parent","defaultContent":"","orderable":true,"searchable":true},{"data":"action","name":"action","title":"","orderable":false,"searchable":false}]});})(window,jQuery);
</script>

What is this error and why that is occurred?
System details

Operating System windows10
PHP Version 5.6.25
Laravel Version 5.2
Laravel-Datatables Version 6.22

@coolflames can you try returning the query if it works? I think I've seen this issue before and this might be related to self join handling of the DB driver.
return $menuss = Appl_menu::with('modules', 'menus');

If this works, then there must be a bug in the package when compiling self join tables. Thanks for reporting.

the query work well the first time populate data, problems appear when it starts input search

Hello,
i think i have the same issue on a n.n relation.
i have a count columns that works on init but when searching the query string is interpreted as a column name in the like clause, and it end up with a sql error.
I don't want that column to be searchable anyway, the searchable parameter is false when i'm doing a dump of $request->query. I don't understand why searchableColumnIndex() still return that column.
i'm on 6.24.4, do i need to update laravel to 5.4 and datatables to 7.*?
thanks

I have another example of failing self joins, where email is having recovery (one to many) email and forwarding to emails (many to many).
Here are the relevant parts of the Email model (relations):
    public function recoveryEmail()
    {
        return $this->belongsTo(EmailAccount::class);
    }

    public function forwardingToEmailAccounts()
    {
        return $this->belongsToMany(
            EmailAccount::class,
            'emails_have_forwarding_emails',
            'forwarding_to_email_id',
            'forwarded_from_email_id'
        );
    }

Here is the failing query, where recoveryEmail is referencing email_accounts table and I'm querying base table with the same name:
EmailAccount::select('email_accounts.*')
    ->with(['recoveryEmail' => function ($query) {
        $query->select(['id', 'email']);
    }]);

And the exception + failing MySQL query:
Exception Message:??SQLSTATE[42000]: Syntax error or access violation: 1066 Not unique table/alias: 'email_accounts' (SQL:
select `email_accounts`.*
from `email_accounts`
left join `email_accounts` on `email_accounts`.`recovery_email_id` = `email_accounts`.`id`
order by `email_accounts`.`email` desc limit 10 offset 0
)

The problem is similar with forwardingToEmails.
I guess this is more of a problem with Eloquent not allowing aliasing, that's why DataTables package doesn't know how to order the data (since both tables use the same name).
Was wondering if there may be a workaround for it?

Fixed via #2051.

Hi, I'm using v9.4.1 but still getting the unique table error on self joining for parent relation.
Should I have to modify my code?
My datatables index method and the relation:
    public function query(Topic $model)
    {
        $model = $model->newQuery();

        $model->with('parent')->select('topics.*');

        return $model;
    }

    public function parent()
    {
        return $this->belongsTo('App\Models\Topic', 'parent_id');
    }

Thanks!

@dhcmega the PR was reverted as it causes another issue. Self join is currently not yet supported again. Thanks!

So any suggestion on this yet? I do not even have a self-joined table, but I have same issue.


Make additional query options available through genquery.py

We have been working on an extended Python rule interface for
genquery, that exposes genquery options that we require but that are
not currently accessible via row_iterator and paged_iterator. Since
these features are very generic, we would be interested to hear your
thoughts on our implementation and whether this could be incorporated
in the rule engine plugin.
In order to expose the extra options without breaking compatibility, the
query interface is implemented as a new class, and row_iterator is
changed such that it calls that class.

Offset and limit options: To avoid having to fetch/process all
resultsets in the query when a page is requested, we add offset/limit
parameters that set relevant options on the genquery object.

Case insensitivity: We add a parameter for specifying that
UPPER_CASE_WHERE should be applied, so that queries can become case
insensitive.

Immediate query closing: For queries with a large number of
resultsets, row_iterator currently appears to fetch all remaining rows
when a query is aborted / limited. By setting AUTO_CLOSE on the query
during close, we only need a single extra fetch at the end of a query,
while still preventing statement table full errors.

Getting the total number of rows: We allow getting the total amount of
matching rows (when COUNT() on a column is not sufficient) via
genquery's RETURN_TOTAL_ROW_COUNT option. This can be done without
fetching all results beforehand.

Additionally, (very limited) testing shows consistent performance
improvements, both for queries with ~30 results and over 9000 results.
A very simple testcase is here: https://gist.github.com/cjsmeele/1f2b72b6584caf569c8ca208ed964e61 - to be run once with the current row_iterator, and once with this PR version.
(I admit that I haven't yet found the cause of this difference)
edit: after some investigation, #35 explains most of the performance difference

incorporated into https://github.com/irods/irods_rule_engine_plugin_python/pull/38
closing - thanks!


[SPARK-28023][SQL] Trim the string when cast string type to Boolean/Numeric types

What changes were proposed in this pull request?
This pr trim the string when cast string type to Boolean/Numeric types for 2 reasons:

Our Spark SQL itself is also inconsistent, such as:

spark-sql> select cast('  1 ' as int), cast(' 1 ' as float);
NULL	1.0

The mainstream database trims the string when cast string type to Boolean/Numeric types:
PostgreSQL:

Teradata:

Oracle:

Vertica:

SQL Server:

MySQL:

How was this patch tested?
unit tests

Test build #106506 has started for PR 24872 at commit 5ab6ccd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11751/
Test PASSed.

Test build #106506 has finished for PR 24872 at commit 5ab6ccd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106506/
Test FAILed.

Test build #106515 has started for PR 24872 at commit 9c28e6a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11757/
Test PASSed.

Test build #106515 has finished for PR 24872 at commit 9c28e6a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106515/
Test PASSed.

Just in case, how about to introduce new parameter for toInt, toLong and etc. (to existing or new methods with additional parameter to control trimming). Trimming inside of loops of those methods shouldn't impact on performance so much, and shouldn't produce additional garbage.

If this is a correctness issue, we can't put it under a flag, right?

Is it really correctness issue? Is the correct behavior described somewhere in Spark's docs?

we wouldn't want the behavior to vary with the flag.

The flag would be useful to restore performance in the cases when user's input doesn't contain spaces.

How would people generally find this, etc.

As any other flags - in docs and in the migration guide.

The overhead of trimming should be trivial compared to parsing.

The statement should be confirmed by benchmarks, isn't it?

As I can see, trim() always does a copy of the original string:
https://github.com/apache/spark/blob/17781d75308c328b11cab3658ca4f358539414f2/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java#L543

Taking a step back: is there a correct behavior? does Hive or a SQL standard suggest that " 3.0" should cast correctly to a double? if so, then there is no question that this is a fix, and we shouldn't offer a flag to make the behavior incorrect.
Is there not a clear correct behavior? then don't enforce it in Spark. Callers trim() input if needed, or don't if it isn't. No call for a flag there.
The perf question probably doesn't matter, then, either way.
Oh, this is UTF8String.trim(). Looks like there are some obvious optimizations here to avoid a copy, like String.trim(). It doesn't even optimize for the common case. I'll try that in a separate PR, as that will be worthwhile no matter what.

@dongjoon-hyun
Trim the string when casting string to date/timestamp is fixed by #22943.
Partition columns and non-partition columns are the same code path.
@srowen
Hive fixed cast string to numeric types with regards to leading/trailing spaces by HIVE-17782:

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11793/
Test PASSed.

Test build #106552 has started for PR 24872 at commit aff33b2.

I agree with @srowen 's opinion regarding performance. When we add a short-cut to avoid copy, we could minimize performance degradation.

Test build #106552 has finished for PR 24872 at commit aff33b2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106552/
Test PASSed.

When I talk about a flag, I meant a flag for toLong, toInt and etc. methods as well, see https://github.com/apache/spark/pull/24872#issuecomment-502394796 . Spark implementation is a copy of Hive's: https://github.com/apache/spark/blob/17781d75308c328b11cab3658ca4f358539414f2/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java#L1026
where input trimming is integrated to the parsing functions:
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyLong.java#L130-L137
I just want to say that the fix (or a feature if it is not defined by SQL standard) should be made in UTF8String instead of Cast.
Speaking about SQL config, let's consider 2 use cases:

User's input doesn't have spaces. In that case, we just introduce additional overhead of trimming strings.
User's input does have spaces.

And the user aware of them and call trim() explicitly. In that case, we just add additional implicit trimming which cause additional overhead in current implementation. No benefits too but if it is a critical place in user's code, the user has to spend time in troubleshooting the performance problems.
If trim() is not called explicitly, the behavior is going to be changed, actually. Here, I think, the SQL flag (in spark.sql.legacy namespace) could be useful to restore previous behavior.

Additional thoughts, if we wrap Cast's string input by the expression StringTrim, we could eliminate double trimming by folding StringTrim.

Presumably right now, all input to these functions doesn't have spaces -- otherwise it would fail.
If it's not clear whether the input should be trimmed by these functions from a standards perspective, then I'd say don't make this change at all. Just leave behavior without a compelling reason to change it.
If there is, then we need to enforce it. You're right, that leaves users with a possibly redundant trim() in their code. If they know enough to know this, they'd just remove the manual trim(), then -- not undo this 'fix' going forward for future usages. Most people won't know about the flag either way, anyway, if one were added.
What's the cost? I put together a crude benchmark of 90% strings that have no whitespace at the ends, and 10% that do. It's 20 nanoseconds per call or so. If I add one extra short-circuit to trim() for that common case, it's 6 nanoseconds. We can at least bring the overhead of the common case down a lot, but it's already very small. I'll propose that change separately anyway.
If this change is important, I think a flag isn't necessary. But it may just not be the right behavior change anyway.

We can at least bring the overhead of the common case down a lot, but it's already very small.

I worry mostly regarding the garbage the trim() produces. That could impact on GC pauses.

If it's not clear ... from a standards perspective, then I'd say don't make this change at all.

Let's confirm that this is required by the standard. At the moment, the trimming seems just a nice feature.

If we would introduce an optimization to return this when no space, trim() will generate no garbage for the common case.

See https://github.com/apache/spark/pull/24884

Test build #106600 has started for PR 24872 at commit e0ae14b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11836/
Test PASSed.

Test build #106600 has finished for PR 24872 at commit e0ae14b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CollapseCodegenStages(
case class AdaptiveSparkPlanExec(
case class StageSuccess(stage: QueryStageExec, result: Any) extends StageMaterializationEvent
case class StageFailure(stage: QueryStageExec, error: Throwable) extends StageMaterializationEvent
case class InsertAdaptiveSparkPlan(session: SparkSession) extends Rule[SparkPlan]
case class LogicalQueryStage(
case class PlanAdaptiveSubqueries(
abstract class QueryStageExec extends LeafExecNode
case class ShuffleQueryStageExec(
case class BroadcastQueryStageExec(
case class ReusedQueryStageExec(
class ParquetFilters(
class ParquetOutputWriter(path: String, context: TaskAttemptContext)
class ParquetReadSupport(val convertTz: Option[TimeZone],
  case class FileTypes(
class ParquetWriteSupport extends WriteSupport[InternalRow] with Logging
class ParquetDataSourceV2 extends FileDataSourceV2
case class ParquetPartitionReaderFactory(
case class ParquetScan(
case class ParquetScanBuilder(
case class ParquetTable(
class ParquetWriteBuilder(
case class ArrowEvalPythonExec(udfs: Seq[PythonUDF], resultAttrs: Seq[Attribute], child: SparkPlan,
case class SparkListenerSQLAdaptiveExecutionUpdate(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106600/
Test PASSed.

Benchmark and benchmark result.
package org.apache.spark.sql.execution.benchmark

import org.apache.spark.benchmark.Benchmark

/**
 * Benchmark trim the string when casting string type to Boolean/Numeric types.
 * To run this benchmark:
 * {{{
 *   1. without sbt:
 *      bin/spark-submit --class <this class> --jars <spark core test jar> <spark sql test jar>
 *   2. build/sbt "sql/test:runMain <this class>"
 *   3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
 *      Results will be written to "benchmarks/CastBenchmark-results.txt".
 * }}}
 */
object CastBenchmark extends SqlBasedBenchmark {

  override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {

    val title = "Benchmark trim the string when casting string type to Boolean/Numeric types"
    runBenchmark(title) {
      withTempPath { dir =>
        val N = 500L << 13
        val df = spark.range(N)
        val withoutWhitespace = "withoutWhitespace"
        val withWhitespace = "withWhitespace"
        val types = Seq("int", "long", "float", "double", "decimal", "boolean")

        df.selectExpr("cast(id as string) as str")
          .write.mode("overwrite").parquet(dir + withoutWhitespace)
        df.selectExpr(s"concat('${" " * 5}', id, '${" " * 5}') as str")
          .write.mode("overwrite").parquet(dir + withWhitespace)

        val benchmark = new Benchmark(title, N, minNumIters = 5, output = output)
        Seq(withoutWhitespace, withWhitespace).foreach { data =>
          Seq(false, true).foreach { isTrimStr =>
            val expr =
              types.map(t => s"cast(${if (isTrimStr) "trim(str)" else "str"} as $t) as c_$t")
            val name = s"$data ${if (isTrimStr) "with" else "without"} trim"
            benchmark.addCase(name) { _ =>
              spark.read.parquet(dir + withoutWhitespace).selectExpr(expr: _*).collect()
            }
          }
        }
        benchmark.run()
      }
    }
  }
}

Before this pr(after SPARK-28066):
[info] Java HotSpot(TM) 64-Bit Server VM 1.8.0_211-b12 on Linux 3.10.0-957.1.3.el7.x86_64
[info] Intel Core Processor (Broadwell)
[info] Benchmark trim the string when casting string type to Boolean/Numeric types:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] withoutWhitespace without trim                     8696          10753         NaN          0.5        2123.1       1.0X
[info] withoutWhitespace with trim                        9915          10251         545          0.4        2420.6       0.9X
[info] withWhitespace without trim                        6128           9433         NaN          0.7        1496.0       1.4X
[info] withWhitespace with trim                           9666          10162         628          0.4        2359.8       0.9X

After this pr(after SPARK-28066):
[info] Java HotSpot(TM) 64-Bit Server VM 1.8.0_211-b12 on Linux 3.10.0-957.1.3.el7.x86_64
[info] Intel Core Processor (Broadwell)
[info] Benchmark trim the string when casting string type to Boolean/Numeric types:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] withoutWhitespace without trim                     7379           9063         NaN          0.6        1801.4       1.0X
[info] withoutWhitespace with trim                        6260           7819        1505          0.7        1528.3       1.2X
[info] withWhitespace without trim                        5997           8175         NaN          0.7        1464.2       1.2X
[info] withWhitespace with trim                           6016           8252        1143          0.7        1468.8       1.2X

Based on the above discussions, let us keep it open. We can revisit it later.

Hi, @gatorsmile . Do you think we can make a decision for this? (There are some other minor PRs depending on this.)

I personally don't see a compelling reason to do this, even though trim() has been optimized somewhat.

The original purpose of this issue and PR is to follow PostgreSQL behavior in Apache Spark 3.0.0.
Some other PRs are pending because we are waiting for the decision.

If we are going to do this, the other PRs will rebase up on this and go further toward PostgreSQL.
If we are going to close this as Won't Fix, the other issues will ignore this difference and proceed in anyway.

I'm okay with both directions. The important thing in this PR is that we need PMC's decision to move forward.

We are unable to 100% match the semantics of PostgreSQL. Adding the extra trim for the Boolean/Numeric type casting will impact the perf of a common operation, although it is not high. Based on the above discussions, it might not be worth it to support this corner case.
I think, in the future, we can add the PostgreSQL-compatible mode for supporting all the corner cases. This effort will be big. If we decide to add it, it might need 1000+ JIRAs and PRs.

Thank you for the conclusion, @gatorsmile and @srowen !
@wangyum . Could you resolve the issue as Later and close this PR?
And, let's proceed the other PRs with this difference.

The problem I see is not about pgsql compatibility, but about internal consistency. Why casting to date/timestamp trims the spaces but casting to numeric does not?
BTW for parsing, we don't really need a "safe" trim that copies the data. We can do a "cheap" trim that only changes the UTF8String offsets. The trimmed UTF8String instance will only be used in the parsing method. I don't think it will have a big perf overhead.

The problem I see is not about pgsql compatibility, but about internal consistency. Why casting to date/timestamp trims the spaces but casting to numeric does not?

BTW for parsing, we don't really need a "safe" trim that copies the data. We can do a "cheap" trim that only changes the UTF8String offsets. The trimmed UTF8String instance will only be used in the parsing method and won't be shared by others. I don't think it will have a big perf overhead.

Yea, aggree. for the reason of type coercion changed between spark versions unexpectedly, now the binary comparator are affected.

@yaooqinn can you send a PR to fix it? We can reuse @wangyum 's benchmark: https://github.com/apache/spark/pull/24872#issuecomment-503827016  and see if "cheap" trim can help.

@yaooqinn can you send a PR to fix it? We can reuse @wangyum 's benchmark: #24872 (comment) and see if "cheap" trim can help.

OK


Login Activity

I am creating a login activity for the application for the personalization task in recommender systems.

Login page activity created in the application.

Firebase integration complete for the application

Register Activity Complete

Starting from Register Activity till the Login Activity is complete.

Login Functionality Completed.


Properly provide MessageBoxResult from Win32 as an enumeration instead of an integer.

Suggestion
Currently to use a MessageBoxResult you need to directly use the integer (by looking up on MSDN or otherwise the values).
This type properly is an enumeration and (I think) only used as such.
Current:
    unsafe {

        let user_result = MessageBoxA(HWND_DESKTOP, "", "", MB_YESNO);

        // If and only if the user hit Yes (Closing the window or hitting No are equivalent).

        return user_result.0 == 6;

    }

Proposed:

    unsafe {

        let user_result = MessageBoxA(HWND_DESKTOP, "", "", MB_YESNO);

        // If and only if the user hit Yes (Closing the window or hitting No are equivalent).

        return match MessageBoxA(HWND_DESKTOP, "", "", MB_YESNO) {

            MB_Result::Yes => true,

            _ => false

        }

Docs

You are looking at the docs for .NET which is unrelated to this crate or the Windows API. MessageBoxA is documented here:
https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-messageboxa
Following the correct docs, you can for example use it as follows in Rust:
match MessageBoxA(None, s!("Hello"), s!("World"), MB_YESNO) {
    IDYES => panic!("IDYES"),
    other => panic!("{:?}", other),
};

Thank you very much...
I see that now, should have been obvious, I've got too many tabs open
Didn't think about looking for literals to compare with either


Rename Structure to StructureEnum (or similar), and remove it as a first-class citizen

Structure stands out as the one "object" type which isn't a direct reference wrapper, and in general isn't at all consistent with other wrapper types.
While it is useful, I don't think it deserves it's place as the "default" type for representing a Structure.
What would you think of replacing Structure with a plain reference-wrapper (similar to RoomObject), and relegating the current thing named Structure to the side?
We could still make it reasonably nice to use by implementing From<Structure> for StructureEnum, and vice-versa.
One thing I think this could open up is implementing a few alternative StructureEnum-like things. Plain reference types deserve to exist because they're the simplest kind of structure, but I don't think there's a similar obvious reason that Structure being implemented as an enum is the best. If the default wasn't a "nice extension", but rather just the simplest thing, that would make it easier to experiment with other designs (in-crate or out-of-crate).
Other ideas could be like:

a whole set of enums, matching like TransferableThing, WithdrawableThing, HasStorageThing, etc. with fallible conversions inbetween
a reference-wrapper SomeStructure which implements all possible structure operations (type, my, etc.) and fails at runtime if the structure doesn't implement them
two wrapper around Box<dyn StructureProperties> and &dyn StructureProperties which allow casting to different dyn X things by checking the type at runtime
something like struct StructurePhantom<T: ?Sized>(Reference, PhantomData<T>) which could be typed to dyn StructureProperties, or dyn Withdrawable, etc.

I've got no idea how pheasible all of these would be, or if any would provide concrete advantages over our current Structure. But I think if the current "smart" solution was no longer a default but rather an add-on, it would be easier to add in other "smart" solutions as similar add-ons.
cc @ASalvail, @Dessix

A definite yes from me. Working with structure types is currently the most painful part of the API in my opinion.

Not saying no, but how about we start from actual examples of code we would like to use? If we gather some such examples, it'd give us a better idea of what/how we would like to implement.
@Dessix what is annoying you with the structures? I'm catching up, but I'm much further code-wise.

Not saying no, but how about we start from actual examples of code we would like to use? If we gather some such examples, it'd give us a better idea of what/how we would like to implement.

I think this is definitely a good idea. I should... use Structure more.
My main gripe with the current Structure implementation is just that it's kind of arbitrary that it's implemented as an enum, and that no other similar types are implemented that way - they're all plain reference wrappers.

There is the option of removing them entirely and expect people to use dyn StructureProperties. I always prefer to remove code rather than add more. 😛
I'd rather we solve a concrete problem when adding to the API elements that aren't strictly necessary.

I'm in favor of removing code - but not necessarily functionality? Like this can be useful, but I don't think it's a sane default. Maybe it could move to a separate repo?
With dyn StructureProperties, I like it in principle - but I think it'd be much more code than a reference wrapper in practice.
I mean the two places Structure is actually used is as a result from find::STRUCTURES and look::STRUCTURES, and it'd be super easy to switch those over to a reference wrapper. To get a dyn StructureProperties, though, we'd need code like what our current Structure enum does - where it first gets the type manually, then switches on that an

Oh, sure the reference wrapper makes sense. Actually, it's probably the easiest and most useful short-term.