Fix pathname canonicalization for input with non-standard protocols.

The test() and exec() methods also perform canonicalization on their
input on a component-by-component basis when the dictionary is passed.
In those cases we must be careful to use the correct type of pathname
canonicalization based on the protocol value.  Special schemes like
'https' get more typical encoding, but schemes like 'data' get a less
strict form of encoding.
This change also cleans up some fallback strings in test() and
exec() that are no longer needed.

This is part of #7.


AngelUMICH/Long-Time-Dynamics is no longer public

README.md:1440 has the following line. This project is no longer public.
[Long-Time-Dynamics](https://github.com/AngelUMICH/Long-Time-Dynamics): studies cascade overlap and defect annealing on damage and time scales up to 1 dpa and a few seconds, respectively, by Angel Chavira. Irradiation cascades are modeled by two methods: random frenkel pair insertion analogous to electron irradiation and actual ion damage events whose dynamics are driven by traditional MD simulation.

removed


BTI reader does not recognize EEG channels

thanks anne-so !

closing?

Closing, feel free to reopen if it's not actually fixed


node taint failed

Hi, I'm not really sure I fully understand how taint works. But I noticed that on v0.9 controllers node are not marked as unschedulable, nor in the apiserver parameters.
In the userdata, there is a command that failed :
https://github.com/coreos/kube-aws/blob/master/config/templates/cloud-config-controller#L479
+ echo tainting this node
24178] hyperkube[5]: tainting this node
24404] hyperkube[5]: + hostname=ip-10-0-2-7.eu-west-1.compute.internal
24615] hyperkube[5]: + kubectl='/kubectl --server=http://<IP_ADDRESS>:8080'
24906] hyperkube[5]: + taint='/kubectl --server=http://<IP_ADDRESS>:8080 taint node ip-10-0-2-7.eu-west-1.compute.internal'
25187] hyperkube[5]: + /kubectl --server=http://<IP_ADDRESS>:8080 taint node ip-10-0-2-7.eu-west-1.compute.internal node.alpha.kubernetes.io/role:master:NoSchedule
57690] hyperkube[5]: error: at least one taint update is required
59076] hyperkube[5]: + echo done.
59381] hyperkube[5]: done.
59673] hyperkube[5]: + echo uncordoning this node
59947] hyperkube[5]: uncordoning this node
60249] hyperkube[5]: + /kubectl --server=http://<IP_ADDRESS>:8080 uncordon ip-10-0-2-7.eu-west-1.compute.internal
33116] hyperkube[5]: node "ip-10-0-2-7.eu-west-1.compute.internal" already uncordoned
36197] hyperkube[5]: + echo done.
36576] hyperkube[5]: done.

From what I understand, taint should be a K=V type like in this issue : https://github.com/kubernetes/kubernetes/issues/33530
But apparently this not right either (https://github.com/kubernetes/kubernetes/issues/36272).
Again from what I understand there should be :

a label : node.alpha.kubernetes.io/role=master
a taint : node.alpha.kubernetes.io/role=master and not dedicated (that's the case in master branch)

Is that correct and if that's the case it just a missing "=" instead of ":"  ?

Hi @ArchiFleKs! You are correct it should be '=' and this has been patched by PR #199, combined with another fix for the taint-and-cordon task on controllers (Ref #215 #199 for controller, #231 for similar fix on workers).

@ArchiFleKs Thanks for your report 👍
As @whereisaaron explained, this should be fixed in v0.9.3-rc.3 released today. Would you mind trying it and reopen if the issue still persists?


Allow action to throw error

It would be convenient to add a helper chaining method (e.g. .throw()) so that the server action would throw the error instead of returning it.
Use cases: the error.tsx and <ErrorBoundary /> components would automatically work if the server action throws the error, now I have to manually re-throw it
// page.tsx
const [data, err] = await action()
if (err) {
 throw new Error(err)
}

if with the chaining method I could just do
const action2 = action.throw()
const data = await action2()

Hi, I am planning to improve error handling for the 1.0.0 release. Will make sure this functionality is available. In the meantime, you can try throwing the error in the shape error function on a base procedure.

Thanks for the amazing work!

Hey, I'm not really familiar with what errors can be.
In the experimental_shapeError callback, err is of type "unknown", whereas I thought all errors from zsa are of type TZSAError.
Can errors from experimental_shapeError be both an instance of the Error class, as well as TZSA?
"use server"

import { createServerActionProcedure, TAnyCompleteProcedure } from "zsa"

export const DALProcedure = (procedure: TAnyCompleteProcedure) => {
    return createServerActionProcedure(procedure)
        .experimental_shapeError(({ err }) => {
            const error = typeof err === "object" && err !== null && "data" in err && typeof err.data === "string" ? err.data : "Unknown database error"
            throw new Error(error)
        })
        .handler(() => {})
}


Bug: Unit test failed to execute after upgrading NUnit3-VS-Adapter from 4.3.1 to 4.4.0

Expected Behavior

While upgrading the nuget package, the tests should still work.
Current Behavior

NUnit failed to load the tests.
An issue is created at (https://github.com/nunit/nunit3-vs-adapter/issues/1070)[https://github.com/nunit/nunit3-vs-adapter/issues/1070]
Steps to Reproduce
N/A
Context (Environment)
Portal version: N/A
LoRaWAN Stack version:  N/A
Logs
Build started 02/2[7](https://github.com/CGI-FR/IoT-Hub-Portal/actions/runs/4286108441/jobs/7465242392#step:9:8)/2023 19:31:40.
     1>Project "/home/runner/work/IoT-Hub-Portal/IoT-Hub-Portal/src/AzureIoTHub.Portal.sln" on node 1 (VSTest target(s)).
     1>ValidateSolutionConfiguration:
         Building solution configuration "Debug|Any CPU".
Test run for /home/runner/work/IoT-Hub-Portal/IoT-Hub-Portal/src/AzureIoTHub.Portal.Tests.Unit/bin/Debug/net7.0/AzureIoTHub.Portal.Tests.Unit.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.5.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
NUnit Adapter <IP_ADDRESS>: Test execution started
Running all tests in /home/runner/work/IoT-Hub-Portal/IoT-Hub-Portal/src/AzureIoTHub.Portal.Tests.Unit/bin/Debug/net7.0/AzureIoTHub.Portal.Tests.Unit.dll
   NUnit failed to load /home/runner/work/IoT-Hub-Portal/IoT-Hub-Portal/src/AzureIoTHub.Portal.Tests.Unit/bin/Debug/net7.0/AzureIoTHub.Portal.Tests.Unit.dll
NUnit Adapter <IP_ADDRESS>: Test execution complete
No test is available in /home/runner/work/IoT-Hub-Portal/IoT-Hub-Portal/src/AzureIoTHub.Portal.Tests.Unit/bin/Debug/net7.0/AzureIoTHub.Portal.Tests.Unit.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Results File: /home/runner/work/IoT-Hub-Portal/IoT-Hub-Portal/src/AzureIoTHub.Portal.Tests.Unit/TestResults/_fv-az362-[8](https://github.com/CGI-FR/IoT-Hub-Portal/actions/runs/4286108441/jobs/7465242392#step:9:9)14_2023-02-27_1[9](https://github.com/CGI-FR/IoT-Hub-Portal/actions/runs/4286108441/jobs/7465242392#step:9:10)_31_47.trx

     1>Done Building Project "/home/runner/work/IoT-Hub-Portal/IoT-Hub-Portal/src/AzureIoTHub.Portal.sln" (VSTest target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:07.[23](https://github.com/CGI-FR/IoT-Hub-Portal/actions/runs/4286108441/jobs/7465242392#step:9:24)

Additional Information

Tested locally with the release 4.4.1 (pre-release). This works on my computer now.
We will have to wait until the official release to accept the bump.


Add background images

Added some background images to the Hero and GetStarted sections as shown in the print screens below. Not sure if this is what we want. Personally i think it is not too bad. But let me know what you think.
Hero:

GetStarted:

Looks soo epic! :heart:


Build error

First of all its great work.
I might be doing some silly mistake while building.
I am not able to build, here is the log:
gyp info it worked if it ends with ok
gyp info using<EMAIL_ADDRESS>gyp info using<EMAIL_ADDRESS>| linux | x64
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory `/home/gaganb/rev-geocode/nodesidx/build'
  CXX(target) Release/obj.target/libspatialindex/deps/libspatialindex/src/tools/rand48.o
In file included from ../deps/libspatialindex/src/tools/rand48.cc:18:0:
../deps/libspatialindex/include/spatialindex/tools/rand48.h:60:0: warning: "__THROW" redefined [enabled by default]
 #   define __THROW
 ^
In file included from /usr/include/features.h:375:0,
                 from /usr/include/math.h:27,
                 from ../deps/libspatialindex/src/tools/rand48.cc:16:
/usr/include/sys/cdefs.h:61:0: note: this is the location of the previous definition
 #   define __THROW throw ()
 ^
In file included from ../deps/libspatialindex/src/tools/rand48.cc:18:0:
../deps/libspatialindex/include/spatialindex/tools/rand48.h:61:0: warning: "__NTH" redefined [enabled by default]
 #   define __NTH(fct)  fct
 ^
In file included from /usr/include/features.h:375:0,
                 from /usr/include/math.h:27,
                 from ../deps/libspatialindex/src/tools/rand48.cc:16:
/usr/include/sys/cdefs.h:63:0: note: this is the location of the previous definition
 #   define __NTH(fct) __LEAF_ATTR fct throw ()
 ^
../deps/libspatialindex/src/tools/rand48.cc: In function ‘short unsigned int* seed48(short unsigned int*)’:
../deps/libspatialindex/src/tools/rand48.cc:80:31: error: declaration of ‘short unsigned int* seed48(short unsigned int*)’ has a different exception specifier
 seed48(unsigned short xseed[3]) __THROW
                               ^
In file included from ../deps/libspatialindex/src/tools/rand48.cc:18:0:
../deps/libspatialindex/include/spatialindex/tools/rand48.h:66:24: error: from previous declaration ‘short unsigned int* seed48(short unsigned int*) throw ()’
 extern unsigned short *seed48(unsigned short xseed[3]) __THROW;
                        ^
../deps/libspatialindex/src/tools/rand48.cc: In function ‘long int nrand48(short unsigned int*)’:
../deps/libspatialindex/src/tools/rand48.cc:98:32: error: declaration of ‘long int nrand48(short unsigned int*)’ has a different exception specifier
 nrand48(unsigned short xseed[3]) __THROW
                                ^
In file included from ../deps/libspatialindex/src/tools/rand48.cc:18:0:
../deps/libspatialindex/include/spatialindex/tools/rand48.h:68:13: error: from previous declaration ‘long int nrand48(short unsigned int*) throw ()’
 extern long nrand48(unsigned short xseed[3]) __THROW;
             ^
../deps/libspatialindex/src/tools/rand48.cc:118:28: error: declaration of ‘void lcong48(short unsigned int*)’ has a different exception specifier
 lcong48(unsigned short p[7]) __THROW
                            ^
In file included from ../deps/libspatialindex/src/tools/rand48.cc:18:0:
../deps/libspatialindex/include/spatialindex/tools/rand48.h:74:13: error: from previous declaration ‘void lcong48(short unsigned int*) throw ()’
 extern void lcong48(unsigned short p[7]) __THROW;
             ^
../deps/libspatialindex/src/tools/rand48.cc: In function ‘long int jrand48(short unsigned int*)’:
../deps/libspatialindex/src/tools/rand48.cc:129:32: error: declaration of ‘long int jrand48(short unsigned int*)’ has a different exception specifier
 jrand48(unsigned short xseed[3]) __THROW
                                ^
In file included from ../deps/libspatialindex/src/tools/rand48.cc:18:0:
../deps/libspatialindex/include/spatialindex/tools/rand48.h:76:13: error: from previous declaration ‘long int jrand48(short unsigned int*) throw ()’
 extern long jrand48(unsigned short xseed[3]) __THROW;
             ^
../deps/libspatialindex/src/tools/rand48.cc: In function ‘double erand48(short unsigned int*)’:
../deps/libspatialindex/src/tools/rand48.cc:136:32: error: declaration of ‘double erand48(short unsigned int*)’ has a different exception specifier
 erand48(unsigned short xseed[3]) __THROW
                                ^
In file included from ../deps/libspatialindex/src/tools/rand48.cc:18:0:
../deps/libspatialindex/include/spatialindex/tools/rand48.h:78:15: error: from previous declaration ‘double erand48(short unsigned int*) throw ()’
 extern double erand48(unsigned short xseed[3]) __THROW;
ake: *** [Release/obj.target/libspatialindex/deps/libspatialindex/src/tools/rand48.o] Error 1
make: Leaving directory `/home/gaganb/rev-geocode/nodesidx/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/home/gaganb/rev-geocode/nodesidx/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:204:12)
gyp ERR! System Linux 3.10.0-123.el7.x86_64
gyp ERR! command "/home/gaganb/.nvm/versions/node/v6.3.1/bin/node" "/home/gaganb/rev-geocode/nodesidx/node_modules/.bin/node-gyp" "build"
gyp ERR! cwd /home/gaganb/rev-geocode/nodesidx
gyp ERR! node -v v6.3.1
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok

I can see you are using linux, I can use an image and try and recreate this. What version are you running?

I tried on CentOS 7 and Ubuntu 16.04, error is same.
So you can try on any Linux dist.


Build a new mail server

hello, i have an issue when building a mail server, installation went good on Ubuntu 18 server, hosted at Digitalocean, but after creating organization, i can't build a mail server, the green button just keeps showing that it's loading, but nothing happens. Thanks for your time

You can't use DO for postal because their port 25 is closed and they will not open.

I see, thank you for your answer, any company you suggest ?

It may be worth asking digital ocean if they will allow you to send emails before changing providers.
Regarding your initial problem, it sounds like you are encountering a server problem so you should have a look at your logs to see what it is.

It may be worth asking digital ocean if they will allow you to send emails before changing providers.
Regarding your initial problem, it sounds like you are encountering a server problem so you should have a look at your logs to see what it is.

thanks for your answer, can you point me to the logs i should check ?

They are in /opt/postal/log, for your error I think you'll need the rails one but I could be mistaken.

Hi, I am having the same problem and I am locally hosting the server.  My port 25 is open and smtp is listening on it, however, I am still unable to build a new mail server.  Any assistance will be greatly appreciated as I am new to this.

@Xando235 as per my previous message, you will need to look at your own log files to see what is going wrong

Just thought I'd chime in. I ran into the same issue, in my case the error was in /opt/postal/logs/rails.log
[web.1:32242] [2020-11-26T00:34:02.640] INFO -- : Started POST "/org/example/servers" for <IP_ADDRESS> at 2020-11-26 00:34:02 -0800 [web.1:32242] [2020-11-26T00:34:02.641] INFO -- : Processing by ServersController#create as JSON [web.1:32242] [2020-11-26T00:34:02.641] INFO -- :   Parameters: {"utf8"=>"✓", "server"=>{"name"=>"Example Mail", "permalink"=>"", "mode"=>"Live"}, "commit"=>"Build server", "org_permalink"=>"example"} [web.1:32242] [2020-11-26T00:34:02.659] INFO -- : Completed 500 Internal Server Error in 18ms (ActiveRecord: 8.8ms) [web.1:32242] [2020-11-26T00:34:02.660] FATAL -- : [web.1:32242] [2020-11-26T00:34:02.660] FATAL -- : Mysql2::Error::ConnectionError (Access denied for user 'postaluser'@'localhost' to database 'postal-server-3'): [web.1:32242] [2020-11-26T00:34:02.660] FATAL -- : [web.1:32242] [2020-11-26T00:34:02.660] FATAL -- : lib/postal/message_db/database.rb:319:in query_on_connection'`
I checked and my passwords in /opt/postal/config/postal.yml are correct, I have access to mysql with
mysql -u postaluser -p

I fixed the issue. First thing I did was fix the mysql issues.
Next I double-checked my postal config: there was an issue. I was following two tutorials, one had "postaluser" and one had "postal" so I had been using the wrong user for the database! 🤦
Anyways, once that was sorted out, I reinitialized postal and everything works!

Glad you figured it out, which tutorial were you following? The quick install script should make all this easier to get going with rather than following individual steps

The quick install script is for Ubuntu 16.04, I'm running Ubuntu 20.04. I also had modified the tutorial a bit, but from what I can tell it's essentially the one provided here on github that someone modified a bit.
The biggest difference with the way I did it was that I didn't need to add any external repositories for the install. I did have to specify a particular version of something for a ruby package
I'm going to try modifying the current script to make it work on any standard 20.04 install and submitting it with a feature request for 20.04 support.

#1265

hi @Lyamc i am getting the same issue cant build the server, can you tell me clear steps , because i follow every tutorial cant work anything.

https://dba.stackexchange.com/a/115032
GRANT ALL PRIVILEGES ON postal\-%.* TO postal@'localhost';
what I did was add all privileges to any db with postal as prefix to my postal user on SQL.
If the postal does not have all privileges this will be necessary since it creates one db for each mailserver.


Still not working..

I followed all the steps (installed python and more...).But if I run the .vbs file a window pops up and closes instantly (one second or something after I ran it the file...) So.. then.. I thought thats not so import or just installing anything in the bgackground - so... I open the .py file and there is the menu where you can set up the config for the following build. So finally I think it's working. I tested the webhook three times and turned off "run on start up" and "vm protect". So. I click on build. Again a window pops up for one second or anything and tells me any error. But it instantly closes too fast, so I can't read what it's trying to tell me. Please give me some advice. Thanks!

Please contact me on discord, Blank?#0886

I already added you. But you have messages disabled. Accept my request.

I already added you. But you have messages disabled. Accept my request.

already done so


Row accumulator support update Scalar values

Which issue does this PR close?

Closes #6002 .
Rationale for this change
improve the Aggregator performance when group by high cardinality columns.

What changes are included in this PR?

add update_scalar() method to RowAccumulator  trait
Adaptively decide the accumulators update mode

Are these changes tested?
I had test this on my local Mac.
For TPCH-q17, there is at about 40% improvement.
Before this PR:
Running benchmarks with the following options: DataFusionBenchmarkOpt { query: Some(17), debug: false, iterations: 3, partitions: 1, batch_size: 8192, path: "./parquet_data", file_format: "parquet", mem_table: false, output_path: None, disable_statistics: false, enable_scheduler: false }
Query 17 iteration 0 took 1888.9 ms and returned 1 rows
Query 17 iteration 1 took 1851.8 ms and returned 1 rows
Query 17 iteration 2 took 1853.1 ms and returned 1 rows
Query 17 avg time: 1864.61 ms
After this PR:
Running benchmarks with the following options: DataFusionBenchmarkOpt { query: Some(17), debug: false, iterations: 3, partitions: 1, batch_size: 8192, path: "./parquet_data", file_format: "parquet", mem_table: false, output_path: None, disable_statistics: true, enable_scheduler: false }
Query 17 iteration 0 took 1377.0 ms and returned 1 rows
Query 17 iteration 1 took 1324.3 ms and returned 1 rows
Query 17 iteration 2 took 1317.7 ms and returned 1 rows
Query 17 avg time: 1339.68 ms.

Are there any user-facing changes?

@Dandandan @andygrove @yahoNanJing

TPCH 10g
2023-03-20(d77ccc25979e3a5e7d71ca86a6317fce50a4bff7)
2023-04-12(2c573105c3d3e4044b4ca74e4faa59d31e57f17b)https://github.com/apache/arrow-datafusion/pull/5866https://github.com/apache/arrow-datafusion/pull/5837
2023-04-14(bd705fe6bb6f018b71dc1f25409e45ec061fd6df)https://github.com/apache/arrow-datafusion/pull/5973
2023-04-14(5bfc1410d677b3da2b9a4adaa3f032562a14a113)https://github.com/apache/arrow-datafusion/pull/6003

q1
23665.52
20753.61
20891.52
21408.54

q2
5589.88
3617.40
3464.09
3464.95

q3
9777.48
5961.30
5785.48
5732.81

q4
5188.19
3226.53
3132.99
4091.51

q5
15309.18
12133.36
10995.68
14361.51

q6
6293.92
4114.41
4504.54
3529.13

q7
28215.12
25501.06
23664.61
24111.81

q8
11563.27
8585.95
10085.88
7257.50

q9
19719.88
16851.29
16703.78
16446.74

q10
14240.17
9811.16
9618.89
10092.34

q11
5774.94
4358.94
4326.38
4629.20

q12
7027.51
4771.94
4659.69
4712.21

q13
17182.12
15855.66
15594.57
16362.75

q14
7567.25
4924.66
4679.48
4458.82

q15
17523.50
11911.70
11484.73
8626.74

q16
4064.01
3719.74
3707.31
3667.75

q17
135964.77
117756.03
107820.88
83682.60

q18
89023.33
60923.75
65235.46
66134.85

q19
9244.44
8504.44
11795.41
8504.76

q20
25746.50
15681.54
15023.04
12232.71

q21
34265.73
33962.40
34681.59
34787.75

q22
8026.30
8000.52
8334.53
8487.91

Total
495383.13
400927.39
396190.53
366784.89

For Q5,  there is downgrade between the versions.  But I think it is  unrelated to my PRs, the aggregation is very lightweight.
=== Physical plan with metrics ===
SortExec: expr=[revenue@1 DESC], metrics=[output_rows=5, elapsed_compute=5.335µs, spill_count=0, spilled_bytes=0]
  ProjectionExec: expr=[n_name@0 as n_name, SUM(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@1 as revenue], metrics=[output_rows=5, elapsed_compute=583ns, spill_count=0, spilled_bytes=0, mem_used=0]
    AggregateExec: mode=Single, gby=[n_name@2 as n_name], aggr=[SUM(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)], metrics=[output_rows=5, elapsed_compute=3.773667ms, spill_count=0, spilled_bytes=0, mem_used=0]
      ProjectionExec: expr=[l_extendedprice@0 as l_extendedprice, l_discount@1 as l_discount, n_name@2 as n_name], metrics=[output_rows=72985, elapsed_compute=541ns, spill_count=0, spilled_bytes=0, mem_used=0]
        CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=72985, elapsed_compute=1.764µs, spill_count=0, spilled_bytes=0, mem_used=0]
          HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column { name: "n_regionkey", index: 3 }, Column { name: "r_regionkey", index: 0 })], metrics=[output_rows=1, input_batches=1, output_batches=1, build_input_batches=1, input_rows=1, build_input_rows=364380, build_mem_used=37529364, join_time=1.260917ms, build_time=3.941506121s]
            ProjectionExec: expr=[l_extendedprice@0 as l_extendedprice, l_discount@1 as l_discount, n_name@4 as n_name, n_regionkey@5 as n_regionkey], metrics=[output_rows=364380, elapsed_compute=1.041µs, spill_count=0, spilled_bytes=0, mem_used=0]
              CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=364380, elapsed_compute=1.767µs, spill_count=0, spilled_bytes=0, mem_used=0]
                HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column { name: "s_nationkey", index: 2 }, Column { name: "n_nationkey", index: 0 })], metrics=[output_rows=25, input_batches=1, output_batches=1, build_input_batches=13, input_rows=25, build_input_rows=364380, build_mem_used=31885080, join_time=7.462876ms, build_time=3.931845754s]
                  ProjectionExec: expr=[l_extendedprice@2 as l_extendedprice, l_discount@3 as l_discount, s_nationkey@5 as s_nationkey], metrics=[output_rows=364380, elapsed_compute=4.209µs, spill_count=0, spilled_bytes=0, mem_used=0]
                    CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=364380, elapsed_compute=8.226µs, spill_count=0, spilled_bytes=0, mem_used=0]
                      HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column { name: "l_suppkey", index: 1 }, Column { name: "s_suppkey", index: 0 }), (Column { name: "c_nationkey", index: 0 }, Column { name: "s_nationkey", index: 1 })], metrics=[output_rows=100000, input_batches=13, output_batches=13, build_input_batches=1046, input_rows=100000, build_input_rows=9103367, build_mem_used=1038498128, join_time=74.74725ms, build_time=3.853093588s]
                        ProjectionExec: expr=[c_nationkey@0 as c_nationkey, l_suppkey@3 as l_suppkey, l_extendedprice@4 as l_extendedprice, l_discount@5 as l_discount], metrics=[output_rows=9103367, elapsed_compute=215.36µs, spill_count=0, spilled_bytes=0, mem_used=0]
                          CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=9103367, elapsed_compute=31.512778ms, spill_count=0, spilled_bytes=0, mem_used=0]
                            HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column { name: "o_orderkey", index: 1 }, Column { name: "l_orderkey", index: 0 })], metrics=[output_rows=59986052, input_batches=7323, output_batches=7323, build_input_batches=262, input_rows=59986052, build_input_rows=2275919, build_mem_used=174938288, join_time=807.006477ms, build_time=702.906984ms]
                              ProjectionExec: expr=[c_nationkey@1 as c_nationkey, o_orderkey@2 as o_orderkey], metrics=[output_rows=2275919, elapsed_compute=48.584µs, spill_count=0, spilled_bytes=0, mem_used=0]
                                CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=2275919, elapsed_compute=18.668µs, spill_count=0, spilled_bytes=0, mem_used=0]
                                  HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column { name: "c_custkey", index: 0 }, Column { name: "o_custkey", index: 1 })], metrics=[output_rows=2275919, input_batches=262, output_batches=262, build_input_batches=184, input_rows=2275919, build_input_rows=1500000, build_mem_used=93390992, join_time=206.31563ms, build_time=49.711972ms]
                                    ParquetExec: limit=None, partitions={1 group: [[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/customer/part-0.parquet]]}, projection=[c_custkey, c_nationkey], metrics=[output_rows=1500000, elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0, page_index_rows_filtered=0, num_predicate_creation_errors=0, bytes_scanned=3006119, row_groups_pruned=0, pushdown_rows_filtered=0, predicate_evaluation_errors=0, time_elapsed_scanning_until_data=2.157083ms, time_elapsed_scanning_total=16.696249ms, time_elapsed_processing=15.288754ms, time_elapsed_opening=1.074375ms, page_index_eval_time=2ns, pushdown_eval_time=2ns]
                                    ProjectionExec: expr=[o_orderkey@0 as o_orderkey, o_custkey@1 as o_custkey], metrics=[output_rows=2275919, elapsed_compute=25.543µs, spill_count=0, spilled_bytes=0, mem_used=0]
                                      CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=2275919, elapsed_compute=2.349505ms, spill_count=0, spilled_bytes=0, mem_used=0]
                                        FilterExec: o_orderdate@2 >= 8766 AND o_orderdate@2 < 9131, metrics=[output_rows=2275919, elapsed_compute=19.347372ms, spill_count=0, spilled_bytes=0, mem_used=0]
                                          ParquetExec: limit=None, partitions={1 group: [[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/orders/part-0.parquet]]}, predicate=o_orderdate@4 >= 8766 AND o_orderdate@4 < 9131, pruning_predicate=o_orderdate_max@0 >= 8766 AND o_orderdate_min@1 < 9131, projection=[o_orderkey, o_custkey, o_orderdate], metrics=[output_rows=15000000, elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0, page_index_rows_filtered=0, num_predicate_creation_errors=0, bytes_scanned=94686185, row_groups_pruned=0, pushdown_rows_filtered=0, predicate_evaluation_errors=0, time_elapsed_scanning_until_data=4.577708ms, time_elapsed_scanning_total=613.686009ms, time_elapsed_processing=365.906294ms, time_elapsed_opening=1.525625ms, page_index_eval_time=2ns, pushdown_eval_time=2ns]
                              ParquetExec: limit=None, partitions={1 group: [[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/lineitem/part-0.parquet]]}, projection=[l_orderkey, l_suppkey, l_extendedprice, l_discount], metrics=[output_rows=59986052, elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0, page_index_rows_filtered=0, num_predicate_creation_errors=0, bytes_scanned=453645745, row_groups_pruned=0, pushdown_rows_filtered=0, predicate_evaluation_errors=0, time_elapsed_scanning_until_data=6.237208ms, time_elapsed_scanning_total=2.314007854s, time_elapsed_processing=1.391610509s, time_elapsed_opening=1.374334ms, page_index_eval_time=2ns, pushdown_eval_time=2ns]
                        ParquetExec: limit=None, partitions={1 group: [[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/supplier/part-0.parquet]]}, projection=[s_suppkey, s_nationkey], metrics=[output_rows=100000, elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0, page_index_rows_filtered=0, num_predicate_creation_errors=0, bytes_scanned=372287, row_groups_pruned=0, pushdown_rows_filtered=0, predicate_evaluation_errors=0, time_elapsed_scanning_until_data=1.480458ms, time_elapsed_scanning_total=76.493501ms, time_elapsed_processing=1.467501ms, time_elapsed_opening=1.877375ms, page_index_eval_time=2ns, pushdown_eval_time=2ns]
                  ParquetExec: limit=None, partitions={1 group: [[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/nation/part-0.parquet]]}, projection=[n_nationkey, n_name, n_regionkey], metrics=[output_rows=25, elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0, page_index_rows_filtered=0, num_predicate_creation_errors=0, bytes_scanned=544, row_groups_pruned=0, pushdown_rows_filtered=0, predicate_evaluation_errors=0, time_elapsed_scanning_until_data=790.959µs, time_elapsed_scanning_total=8.265166ms, time_elapsed_processing=122.376µs, time_elapsed_opening=1.493584ms, page_index_eval_time=2ns, pushdown_eval_time=2ns]
            ProjectionExec: expr=[r_regionkey@0 as r_regionkey], metrics=[output_rows=1, elapsed_compute=125ns, spill_count=0, spilled_bytes=0, mem_used=0]
              CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=1, elapsed_compute=1.336µs, spill_count=0, spilled_bytes=0, mem_used=0]
                FilterExec: r_name@1 = ASIA, metrics=[output_rows=1, elapsed_compute=6.208µs, spill_count=0, spilled_bytes=0, mem_used=0]
                  ParquetExec: limit=None, partitions={1 group: [[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/region/part-0.parquet]]}, predicate=r_name@1 = ASIA, pruning_predicate=r_name_min@0 <= ASIA AND ASIA <= r_name_max@1, projection=[r_regionkey, r_name], metrics=[output_rows=5, elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0, page_index_rows_filtered=0, num_predicate_creation_errors=0, bytes_scanned=230, row_groups_pruned=0, pushdown_rows_filtered=0, predicate_evaluation_errors=0, time_elapsed_scanning_until_data=289.083µs, time_elapsed_scanning_total=297.375µs, time_elapsed_processing=96.583µs, time_elapsed_opening=900.083µs, page_index_eval_time=2ns, pushdown_eval_time=2ns]

Q5
Before this PR
Running benchmarks with the following options: DataFusionBenchmarkOpt { query: Some(5), debug: false, iterations: 3, partitions: 1, batch_size: 8192, path: "./parquet_data10", file_format: "parquet", mem_table: false, output_path: None, disable_statistics: true, enable_scheduler: false }
Query 5 iteration 0 took 4093.0 ms and returned 5 rows
Query 5 iteration 1 took 4031.0 ms and returned 5 rows
Query 5 iteration 2 took 4008.9 ms and returned 5 rows
Query 5 avg time: 4044.29 ms
After this PR
Running benchmarks with the following options: DataFusionBenchmarkOpt { query: Some(5), debug: false, iterations: 3, partitions: 1, batch_size: 8192, path: "./parquet_data10", file_format: "parquet", mem_table: false, output_path: None, disable_statistics: true, enable_scheduler: false }
Query 5 iteration 0 took 4042.3 ms and returned 5 rows
Query 5 iteration 1 took 3978.7 ms and returned 5 rows
Query 5 iteration 2 took 3973.1 ms and returned 5 rows
Query 5 avg time: 3998.02 ms
Almost no difference.

Thanks, cool PR. Linking #5969 as potentially related

Filed https://github.com/apache/arrow-rs/issues/4095 .

the current aggregate process, when dealing with high-cardinality aggregations, suffers from low efficiency due to the frequent generation of small arrays from slicing.

Especially for high-cardinality aggregations, the query performance will be downgraded due to lots of slicing invoking

Thank you for your ongoing efforts in improving the performance of aggregate functions. Really appreciate it!
If I understand the core concept of this PR correctly: the current aggregate process, when dealing with high-cardinality aggregations, suffers from low efficiency due to the frequent generation of small arrays from slicing. To address this, you've changed the approach for high-cardinality aggregations by generating ScalarValue or Vec<ScalarValue> instead of creating arrays through slicing.
Nonetheless, generating ScalarValues could still lead to some overhead. As an alternative, I propose the following: we could add a set of methods to arrow-arith's aggregate functions, allowing them to accept a selection vector (also suggested by @alamb previously in #5944 (comment)). This way, we can avoid creating slices and generating ScalarValue at the same time.

Yes, you are right. I agree that generating ScalarValue will involve some overhead.  I'm not sure whether I can get rid from the ScalarValues and generate the rust native type and use the native type to update the accumulator states directly, let me do some experiment on this.  Regarding the other approach you have proposed, I'm open to that, but currently because the arrow-rs aggregation framework does not provide such capabilities, I think when it is ready, we can try this approach also and with the benchmark result and finally decide which way to go.

I will try the following items on the basis of the PR:

Get rid from generating the Scala values
Avoid type cast and null check in the hot path
Try to reuse the RowAccessor and void constructing it again and again.

I'm open to that, but currently because the arrow-rs aggregation framework does not provide such capabilities, I think when it is ready, we can try this approach also and with the benchmark result and finally decide which way to go.

Agree. We could always revisit it when https://github.com/apache/arrow-rs/issues/4095 get resolved or find another sound approach. Thanks again!

I had a comment for refactoring col_to_scalar function. It simplifies the code and removes duplication. Is there a specific reason for not doing so? Maybe I misunderstood what col_to_scalar does. Without this change we may have diverging versions of similar code in row_hash.rs and scalar.rs. One may change one of them, and corresponding change may be missing in other file.

I had a comment for refactoring col_to_scalar function. It simplifies the code and removes duplication. Is there a specific reason for not doing so? Maybe I misunderstood what col_to_scalar does. Without this change we may have diverging versions of similar code in row_hash.rs and scalar.rs. One may change one of them, and corresponding change may be missing in other file.

Yes, I will change accordingly per your comments.

@yjshen @mustafasrepo @Dandandan
I did some experiment by removing the Scalar values and using the rust native types to update the Accumulator state directly. It get some performance gain but not that obvious(reduced 50ms~ on my Mac).
In this PR, I will continue on the Scalar values update approach.
And I think after this PR, the major bottleneck of the high cardinality aggregations on DataFusion should be the Cache miss and Memory stall and our CPU instructions per cycle is much lower than DuckDB's (on tpch-q17).

@mustafasrepo
There is some performance downgrade after using the ScalarValue::try_from_array due to additional null check.
I will leave it for future improvement because we might remove the Scalar values creation here.

Hi @Dandandan, do you still have any concerns for this PR?

I'll merge this PR to unblock the further optimizations. If there's any further concerns, we can create a following issue.


Revert data migration, fix finding framework agreements

Revert "Migrate framework agreements to framework_agreements"
This reverts commit 54368e9.
We haven't been returning the correct framework agreement in all
cases, so we're reverting the migration so that we can release
the correct version of the code through to production.
Filter FrameworkAgreement by supplier_id and framework_id
Before, we were only looking for FrameworkAgreements by
supplier_id, which meant we might be getting back an agreement
for the right supplier but for the wrong framework.
This commit will fix this.
Others
In a couple of places, we've been using supplier.id where we should be using supplier.supplier_id.  Easy fixes.

All looks good :+1:

🚅 🚋 🚋


User defined types draft for CodeView.

Methods to emit enum type definitions were added.
DebugVarInfo and _variableToTypeIndex were changed and now store TypeDesc.
According ObjectWriter update is not published yet, so all tests are expected to fail.
Please review interface changes and my awful C# skills.

PTAL @MichalStrehovsky

I wrote small test  to check that it works as expected.
Some known problems:

Set enum value from the immediate window, e.q. colors = Blue. (fix is unknown).
All fields are shown as public (will be fixed soon).
Strings are not supported (will be fixed soon).
Polymorphism works as expected, it shows pointer type, as C++ does. (could not be fixed).
There are no methods descriptions, e.q. s.MethodName() doesn't work from the immediate window. (methods descriptions will be added in the future)
Generics doesn't work well.

PTAL @dotnet/corert-contrib .
You could take this PR before it is merged and report errors.

@davidwrighton Will be reviewing this

@davidwrighton I made the changes that we discussed.


HOSTEDCP-1483: update cloud-credential-operator in go.mod

What this PR does / why we need it: update cloud-credential-operator to newer version that doesn't use a deprecated import with a CVE
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes # HOSTEDCP-1483
Checklist

[x] Subject and description added to both, commit and PR.
[x] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/retest

/test e2e-aks

/test e2e-kubevirt-aws-ovn-reduced

/approve


GitHub action to install Kontrol on public runners

Per a request from Discord (relevant GitHub issue: https://github.com/open-dollar/od-contracts/issues/520), it would be nice to have a GitHub action that installs Kontrol on public runners for people to use on GitHub. I think the approach is to:

Call the install-nix and install-cachix actions to setup Nix and cachix caches on the runner.
Install kup tool.
Use kup to install the selected version of kontrol.

Done: https://github.com/runtimeverification/install-kontrol.


Allow chat to be recorded in server logs

I don't know if there's a server-side logging mod that can fix this, but running my own server, I can't see the chat messages sent while this mod is installed.

Yep, this is a bug and should be a simple fix. I just forgot to add the logging when I overwrote the chat system.


mul_elemwise() deprecated?

A quick question: I'm getting the error
'mul_elemwise' is not an exported object from 'namespace:CVXR'
I can see that multiply() seems to do the same thing. mul_elemwise() only exists in documentation but not in the package anymore, from a cursory search. Is it deprecated?

Yes, mul_elemwise has been renamed to multiply. We will update the documentation shortly.


Run MiqServer.status_update in server process

Fixes #20835
[Errno::ESRCH]: No such process ... when run from a worker.
In appliance/systemd, you can run status_update in the server's workers.
In podified, each process is isolated so we must run this, specifically
process_status, in the server process.
It didn't make sense to run this is in the worker processes anyway so
change all installations to run this in the server process.
Note, I'm looking at the surrounding lines in this file and seeing that we're logging stuff in random generic/priority workers and much of it doesn't make sense in pods or even appliances... I'll add :queue_name => 'miq_server' to them in a followup as they're not BROKEN but just weird.  I'll keep this separate since it's broken and more backportable.

@agrare I have a followup I'm working on to do the same for other logging here.  While this other logging is in the same place, it doesn't blow up so I though it's better to keep it separate so this surgical fix is easy to backport.  The other logging is just convenient to be logged in the server/orchestrator process in pods since you don't have mixed server/worker processes logging to the same file.


When pasting images, editing of text becomes very slow

Description
I am mostly using Trumbowyg on macOS Safari an have enabled the respective plugins to allow pasting images into the editor which are stored as base64 HTMLs.
However, when pasting an image, the text editing becomes very laggy and slow.
How to reproduce?
You can try it yourself on the trumbowyg demos webpage where after an image was pasted, text editing becomes very slow.
Any chance to speed this up, so it is still realistically usable for HTMLs with pasted images?

Commenting to note that I have experienced this too, and limiting the size of the images down to a quite small 500px maximum on either side has helped a lot, though it will still be an issue if your users fill the text box with a bunch of images. This can be done in the trumbowyg.base64.js file down around line 143 within the if (isValidImage(e.target.result)) { function, something like:
var image = new Image();
image.src = e.target.result;
image.onload = function() {   if(this.width > 500) {   alert('Image is too large. Sorry! Files pasted in the editor have a limit of 500px on either side to avoid high load times.');   return false; } else if(this.height > 500) {   alert('Image is too large. Sorry! Files pasted in the editor have a limit of 500px on either side to avoid high load times.');   return false; } else {   trumbowyg.execCmd('insertImage', fReader.result, false, true);   $(['img[src="', fReader.result, '"]:not([alt])'].join(''), trumbowyg.$box).attr('alt', values.alt);   trumbowyg.closeModal();   } };
That being said, the issue was not apparent until we launched with the editor and it's definitely a pain point!

Is it only on Safari?

From my perception this is not only a Safari issue but also seen in Chrome: it seems solely be an issue with slightly larger pictures. Question is why it slows it so much down, as while typing the image is simply "moved" and should ideally not consume so much CPU power.

Hi.
Faced with the same problem. The problem appears when pasting an image from the clipboard. An example image is attached. The problem also manifests itself in the editor on the site: https://alex-d.github.io/Trumbowyg/
Adding the "max-height: 300px;" style helps. The brakes continue, but the editor is working.


Please add ELLIO: IP Feed (Community version) to the feeds.

Hi,
First of all thanks for the awesome repo and the daily feed updates!
I was looking for a way to contribute in it too but couldn't find anything on that , so I think you can doing it only.
Please take a look at [https://cdn.ellio.tech/community-feed], I believe it will be a great addition to the Zeek-Intelligence Feeds.
Thank You.
More Info about Ellio Tech = https://ellio.tech
Let me know if you want to know anything else
Regards
Ayush Tomar.

Ayush Tomar,
Thank you for submitting the feed for the community!
It has been added!
https://github.com/CriticalPathSecurity/Zeek-Intelligence-Feeds/blob/master/ellio.intel
-PK

Thank you so much for adding Ellio Community feed, but I think there is a minor issue. The link for both the home page and the list URL is same, can you please update the list URL link to https://feed.ellio.tech/ ?
Thank You
-Ayush

I'm updating the URL now.
Thank you, Ayush!


Vein Dedicated Server Template Attempt

https://iceofwraith.github.io/GenericConfigGen/
Game Information:
Game Title: Vein
How to obtain: Steam Store
Suppoted OSs: Windows, Linux
Supports RCON/Console input? Yes
I confirm:

[x] that I have searched for an existing module request for this application.
[x] that the server is an official server distributed by the games publishers/developers.
[x] that the server is available to the general public.
[x] that the server can be run headless (without a GUI).
[x] I have already attempted to create a configuration myself using the configuration tool

My current attempt:
{"_availablePortOptions":["Custom Port","RCON Port"],"_compatibility":"None","Meta_DisplayName":"Vein","Meta_Description":"Vein Dedicated Server","Meta_Arch":"x86_64","_Meta_Author":"UserID10T","Meta_Author":"UserID10T - Made with AMP Config Generator","_Meta_GithubOrigin":"https://github.com/UserID10T - Made with AMP Config Generator/AMPTemplates.git","_Meta_GithubURL":"https://github.com/UserID10T - Made with AMP Config Generator/AMPTemplates","Meta_URL":"https://store.steampowered.com/app/1857950/VEIN/","Meta_MinAMPVersion":"<IP_ADDRESS>","Meta_SpecificDockerImage":"","Meta_DockerRequired":"False","Meta_ContainerPolicy":"Supported","Meta_ContainerPolicyReason":"","Meta_Prerequsites":"[]","Meta_EndpointURIFormat":"steam://connect/{ip}:{GenericModule.App.Ports.$SteamQueryPort}","_SupportsWindows":false,"_SupportsLinux":true,"App_AdminMethod":"STDIO","App_HasReadableConsole":true,"App_HasWritableConsole":true,"App_DisplayName":"Vein","App_CommandLineArgs":"{{$PlatformArgs}} {{$FormattedArgs}} -log","App_WindowsCommandLineArgs":"","App_CommandLineParameterFormat":"-{0} \"{1}\"","App_CommandLineParameterDelimiter":" ","App_RapidStartup":"false","App_ApplicationReadyMode":"Immediate","App_ExitMethod":"OS_CLOSE","App_ExitString":"stop","App_ExitTimeout":"30","App_ExitFile":"app_exit.lck","App_SupportsLiveSettingsChanges":"False","App_LiveSettingChangeCommandFormat":"set {0} \"{1}\"","App_ApplicationIPBinding":"<IP_ADDRESS>","App_AdminPortRef":"RemoteAdminPort","App_UniversalSleepApplicationUDPPortRef":"GamePort1","App_PrimaryApplicationPortRef":"GamePort1","App_UniversalSleepSteamQueryPortRef":"SteamQueryPort","App_MaxUsers":"8","App_UseRandomAdminPassword":"True","App_RemoteAdminPassword":"","App_AdminLoginTransform":"None","App_RCONConnectDelaySeconds":"30","App_RCONConnectRetrySeconds":"15","App_RCONHeartbeatCommand":"ping","App_RCONHeartbeatMinutes":"0","App_TelnetLoginFormat":"{0}","App_SteamUpdateAnonymousLogin":"True","App_SteamForceLoginPrompt":"False","App_SupportsUniversalSleep":"False","App_WakeupMode":"Any","App_TemplateMatchRegex":"{{(\\$?[\\w]+)}}","App_MonitorChildProcess":"False","App_MonitorChildProcessWaitMs":"1000","App_MonitorChildProcessName":"","App_Compatibility":"None","_App_SteamWorkshopDownloadLocation":"/Vein/Mods","App_SteamWorkshopDownloadLocation":"{{$FullBaseDir}}/Vein/Mods","Console_FilterMatchRegex":"","Console_FilterMatchReplacement":"","Console_ThrowawayMessageRegex":"(WARNING|ERROR): Shader.+","_Console_AppReadyRegex":"Server is ready\\.","_Console_UserJoinRegex":"User \\*{username} \\(\\*{userid}\\) connected from \\[\\*{endpoint}\\]","_Console_UserLeaveRegex":"User \\*{username} \\(\\*{userid}\\) disconnected\\. Reason: \\*{misc}","_Console_UserChatRegex":"\\*{username}: \\*{message}","Console_UpdateAvailableRegex":"^\\[\\d\\d:\\d\\d:\\d\\d\\] \\[INFO\\] A new server update is available! v[\\d\\.]+.$","Console_SuppressLogAtStart":"False","Console_ActivateLogRegex":"","Console_UserActions":"{}","Console_SleepMode":"False","Console_SleepOnStart":"False","Console_SleepDelayMinutes":"5","Console_DozeDelay":"2","Console_AutoRetryCount":"5","Console_SleepStartThresholdSeconds":"25","_PortMappings":[{"_Protocol":"0","Protocol":"Both","Port":"27021","_PortType":"Steam Query Port","_Name":"Steam Query Port","Name":"Steam Query Port","_Description":"","Description":"Port used for main game traffic","Ref":"SteamQueryPort"},{"_Protocol":"2","Protocol":"UDP","Port":"7780","_PortType":"Main Game Port","_Name":"Application Port","Name":"Main Game Port","_Description":"Port for main traffic","Description":"Port used for main game traffic","Ref":"MainGamePort"}],"_ConfigFileMappings":[{"ConfigFile":"Vein/Saved/Config/WindowsServer/GameUserSettings.ini","_ConfigType":"1","ConfigType":"ini","_AutoMap":true,"AutoMap":true}],"_UpdateSourceURL":"","_UpdateSourceGitRepo":"","_UpdateSourceUnzip":false,"_DisplayImageSource":"","_SteamServerAppID":"","_WinExecutableName":"","_LinuxExecutableName":"/Vein/Binaries/Linux/VeinServer-Linux-Test","_AppSettings":[{"DisplayName":"Server Name","Category":"Server Settings","Description":"The name displayed in the server title.","Keywords":"server,name","FieldName":"servername","InputType":"text","IsFlagArgument":false,"ParamFieldName":"servername","IncludeInCommandLine":true,"DefaultValue":"MyVeinServer","Placeholder":"MyVeinServer","Suffix":"","Hidden":false,"SkipIfEmpty":false,"_CheckedValue":"true","_UncheckedValue":"false","_EnumMappings":[],"EnumValues":{}},{"DisplayName":"Max Players","Category":"Server Settings","Description":"The maximum number of players","Keywords":"max,players","FieldName":"maxplayers","InputType":"text","IsFlagArgument":false,"ParamFieldName":"maxplayers","IncludeInCommandLine":true,"DefaultValue":"6","Placeholder":"6","Suffix":"","Hidden":false,"SkipIfEmpty":false,"_CheckedValue":"true","_UncheckedValue":"false","_EnumMappings":[],"EnumValues":{}}],"_UpdateStages":[{"UpdateStageName":"Vein Server Download","_UpdateSourcePlatform":"1","UpdateSourcePlatform":"Linux","_UpdateSource":"8","UpdateSource":"SteamCMD","UpdateSourceData":"2131400","UpdateSourceArgs":"2131400","UpdateSourceVersion":"public","UpdateSourceTarget":"","UnzipUpdateSource":false,"OverwriteExistingFiles":false,"_ForceDownloadPlatform":"1","ForceDownloadPlatform":"Linux","UpdateSourceConditionSetting":null,"UpdateSourceConditionValue":null,"DeleteAfterExtract":true,"OneShot":true}],"Console_AppReadyRegex":"^Server\\s+is\\s+ready\\\\\\.$","Console_UserJoinRegex":"^User\\s+\\\\(?<username>.+)\\s+\\\\\\(\\\\(?<userid>.+)\\\\\\)\\s+connected\\s+from\\s+\\\\\\[\\\\(?<endpoint>.+)\\\\\\]$","Console_UserLeaveRegex":"^User\\s+\\\\(?<username>.+)\\s+\\\\\\(\\\\(?<userid>.+)\\\\\\)\\s+disconnected\\\\\\.\\s+Reason:\\s+\\\\.*$","Console_UserChatRegex":"^\\\\(?<username>.+):\\s+\\\\(?<message>.+)$","Meta_OS":2,"Meta_ConfigManifest":"veinconfig.json","Meta_MetaConfigManifest":"veinmetaconfig.json","_Meta_PortsManifest":"veinports.json","_Meta_StagesManifest":"veinupdates.json","Meta_ConfigRoot":"vein.kvp","App_RootDir":"./vein/","_SteamCheck":"2131400","Meta_DisplayImageSource":"steam:2131400","App_BaseDirectory":"./vein/2131400/","App_WorkingDir":"2131400","_SteamClientAppID":"2131400","App_ExecutableWin":"2131400\\","App_ExecutableLinux":"2131400//Vein/Binaries/Linux/VeinServer-Linux-Test","_App_LinuxCommandLineArgsCompat":"","_App_LinuxCommandLineArgsInput":"","App_LinuxCommandLineArgs":"","App_Ports":"@IncludeJson[veinports.json]","App_UpdateSources":"@IncludeJson[veinupdates.json]","Errors":[],"isValid":true}

I take it this is a cry for help lol
Template in Greelan/AMPTemplates:dev
Some weirdness about it tho - eg the MaxPlayers setting is stripped from Game.ini when the server is shut down
Guess that is what you get with a game that is in its demo release phase lol

Yeeeeah, the maxplayers value was something I found when trying to obtain all of the usable in game values. My latest launch wasn't populating game.ini so I couldn't dig into them. I can probably manage getting the game.ini values so the template can be updated.
If you prefer, I can take the template back to get those game settings, or you're welcome to keep it as a live template for everyone. I just want to get used to this process so I can make more.
My next plan is Qanga, another early demo game. Heh.

There are a bunch of issues with your template - some due to the config generator - and this is not the way to make them available anyway
I've already seen the notion website with the server params. But it would be pushing shit uphill trying to maintain a template for such an early access server. Better to wait

To be perfectly honest, if you want to get into templates, better to look at existing ones and learn from those and use them as a base. For example there are lots of UE servers that already have templates

I tried looking through the .kvp files and all, which is exactly how I progressed since I first posted.
I looked at the Ark SE since it used the GenericModule.kvp.
I found the documentation confusing, so I just struggle bussed my way through understanding it, and so what I had was very chopped up from things I "thought" belonged in the kvp.

Lol, Ark is one of the worst ones possible to start with
Just look at my VEIN template. Tho I was lazy with the Game.ini creation since I couldn't be bothered having one downloaded from GH (which is what most templates do if the config file has to be created)

Yeah, I've learned that almost everything with Ark is awful.
I was already poking around the VEIN template and spotted a lot of what you were talking about. I reuploaded it to my repo, so I can mess with it(I have not gotten good with forking yet, so please tell me to remove it and done.)
I really appreciate the help and clarity. I'm hoping I can drum up more games and applications once I'm comfortable with the generator. I've been creating some python projects for content creators and Discord, which may be useful for others. and a deployable webserver would be convenient for that. I just like projects.

No disrespect to Ice, but I suggest not using the generator. It has some bugs and is incomplete. I only do stuff manually - after 150+ templates, it gets quicker (VEIN took me less than an hour). Every server, even those based on the same engine, has differences and quirks and the generator doesn't cater for them, or the tricks often needed

No disrespect taken. I agree it needs attention.

I could quickly knock up a basic Qanga template and you could play around with it

Your help with Qanga would be fantastic, and would give me a relevant comparison to better understand what is unique between templates when creating them, so I can more manually manage them. The generator was an awesome place to get started with it, and once I started exporting and manually editing it helped me a lot.

Heads up, that VEIN AppID you used is for the full unreleased VEIN. It needs the Vein Demo ID for now, 2082470.

Otherwise it screams because the steamappid.txt doesn't match the cert it should. Requires .kvp update and restart of game server, adding/replacing in steamappid.txt isn't enough it scrubs it periodically.

Fixed

Also found Server Branch is not populating with the server config field options.

Wdym?

{
"DisplayName":"Server Branch",
"Category":"Updates",
"Subcategory":"Steam Downloads:download",
"Description":"Sets the server branch to install. Update the server after switching branches",
"Keywords":"server,branch,depot",
"FieldName":"ServerBranch",
"InputType":"enum",
"ParamFieldName":"ServerBranch",
"DefaultValue":"public",
"EnumValues":{
"public":"Public (default)",
"hotfix":"Hotfix builds (hotfix)",
"canary":"Bleeding-edge builds (canary)"
}
}

Look under Updates ....

Oh lord you're right the tag is right there Category Updates.
Sorry about that. The server is up and running, so thank you for taking care of it.

Now I see how you can slap it under any menu, thank you!!!

QANGA is also now in my dev branch

Thank you!
Game Port having issues binding, but I'll update it and figure out how to make a pull request, if that was the right term. I'm green with gh too.

The server uses Steam sockets, so never opens a socket on the game port. They still recommend including the argument, so that is what I have done
If you have anything to add, pass them on to me


test: Un-Quarantine K8sUpdates on GKE

This removes the quarantine check for the K8sUpdates test on GKE. The
issues with upgrade tests had on GKE were addressed in #14187, #14294,
#14295, and #14393.
The last fix was merged 6 days ago. The quarantined GKE pipeline had
since no failures that look related to the issues with DNS that we
previously observed.
Fixes: #13833
Fixes: #13898

test-me-please

I removed the "needs-backport" label for now, as this caused issues on the v1.9 branch. Needs investigation first as to why.

I removed the "needs-backport" label for now, as this caused issues on the v1.9 branch. Needs investigation first as to why.


Question about https://tls.ulfheim.net/

Hi,
Thanks for this wonderful work! Made me understand so much about TLS 1.2
i'm confused about the part where you wrote CLIENT HANDSHAKE FINISHED
In the Encryption IV, The client has sent an initialization vector for decrypting this block. Because we have overridden the rand function it is a predictable sequence.
and the Encrypted Data as well.
Where does the IV come from? Also the "data" that you encrypt, can you specify exactly which one, I find all the things so clear but this part is not for me.
16 03 03 00 40 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f 22 7b c9 ba 81 ef 30 f2 a8 a7 8f f1 df 50 84 4d 58 04 b7 ee b2 e2 14 c3 2b 68 92 ac a3 db 7b 78 07 7f dd 90 06 7c 51 6b ac b3 ba 90 de df 72 0f
Decryption
14 00 00 0c cf 91 96 26 f1 36 0c 53 6a aa d7 3a
Is all this part also sent to the Server? Or just the encrypted version?
Thanks so much for the clarification :)

Hi!  To answer your questions:

the IV is generated out of nothing (it is supposed to be purely random data).  The IV is needed as an input for this particular encryption algorithm, and it is a requirement of the algorithm that the data is random so the output is permuted in a way that prevents certain attacks.  (Of course in our example the randomness is removed, thus the predictable IV of [0x40, 0x41, ... 0x4f])
The data that was encrypted is the string "14 00 00 0c cf 91 96 26 f1 36 0c 53 6a aa d7 3a" (marked in the document in green).  The first part "14 00 00 0c" is a type and length header, and the latter part "cf 91 96 26 f1 36 0c 53 6a aa d7 3a" is the encrypted payload that is sent.  This payload is a hash of the handshake conversation that was calculated by the client.
Only the encrypted version is sent to the server.  The server must decrypt the data using the keys that have been calculated in the handshake, and then it must look at the decrypted data and compare that to a hash of the handshake conversation that it calculated itself (in the same way that the client had calculated it).  If this data matches then it knows that both the client and server calculated the same encryption keys, and that they also agree on what data had been sent in the handshake conversation (no tampering took place).  If this doesn't match the connection is aborted.

I hope this helps!

Awesome thanks for the detail. I know have a way clearer vision of this part
To clarify, the IV is sent to the server too right?

@HandleMaser Yes, an IV is sent in the clear.  The only security requirement is that the client does not re-use it (this is ensured by randomness and length, it's not actually checked).

Thanks for the clarifications and your time!
Have a great day ahead


bug: Invalid Index on disabled default db and additional_databases

TL;DR
The latest v21 release introduced a bug regarding a value lookup, Invalid index.
Expected behavior
No response
Observed behavior
│ Error: Invalid index
│
│   on .terraform/modules/postgresql/modules/postgresql/main.tf line 47, in locals:
│   47:   database_name = var.enable_default_db ? google_sql_database.default[0].name : (length(local.databases) > 0 ? google_sql_database.additional_databases[0].name : "")
│     ├────────────────
│     │ google_sql_database.additional_databases is object with 10 attributes
│
│ The given key does not identify an element in this collection value. An object only supports looking up attributes by name, not by numeric index.
╵

Terraform Configuration
module "postgresql" {
  source  = "GoogleCloudPlatform/sql-db/google//modules/postgresql"
  version = "21.0.0"

  ...

  enable_default_db = false

  additional_databases = [{ name = "any_db", charset = "UTF8", collation = "en_US.UTF8" }]

  ...

Terraform Version
Terraform v1.9.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v5.39.0
+ provider registry.terraform.io/hashicorp/google-beta v5.39.0
...

Additional information
The condition was introduced in https://github.com/terraform-google-modules/terraform-google-sql-db/pull/612/files#diff-5af99345140d405f1b9683608683cd8e45b6cce368ba0446706651ca3a92f041R47

+1

+1
We are also experiencing this.

@q2w your PR #612 is causing this error. Can you plz check and send a fix?

Fixed in #625


update the hazelcast example to API version v1beta3

update the hazelcast example to API version v1beta3

@nikhiljindal

The example is not working properly as stated in #7731, no matter whether I use v1beta1 yaml file or v1beta3 yaml file. It seems to be a problem with the java code of hazelcast rather than Kubernetes.

Thanks @caesarxuchao LGTM.

Ref https://github.com/GoogleCloudPlatform/kubernetes/issues/6584


Add CI pipeline for quickstart

This pull request added a CI pipeline to ensure that all incoming pull requests which are relevant to quickstart folder are:

Terraform syntax error free && well formatted.
Can be applied directly without error, no change when execute terraform plan again.

This pipeline would execute e2e tests for all examples inside quickstart folder per week, once a test was passed, the Terraform core version and all providers' versions that were used in the test would be recorded as a TestRecord.md file so we could know with which versions this example has been tested with.

Need further polish, will reopen when it's ready.


Support for overriding .Chart.IsRoot

Is your feature request related to a problem? Please describe.
We have a few charts that are used as "generic charts". One simply imports them as a dependency, like so:
dependencies:
  - repository: https://repository.example.com/helm/
    name: simple-rest
    version: "0.1.2"
    alias: my-app

And fills the values:
my-app:
  image:
    repository: my-app

Simple enough!
We also have test coverage on these. However, we have a few functions that rely on .Chart.IsRoot that come from a library chart. These detect if they are being used in a subchart (like one of these generic charts) and they change their behavior accordingly, like slightly adjusting labels and such.
To test for these, we have to make a "dummy chart" to use our own chart as a dependency, making the process slightly more convoluted than it needs to be: one needs to make sure they ran helm dependency update before tests, slightly less generic CI, you get the idea.
Describe the solution you'd like
I am not sure if it is easy, or even viable, but we can already override some contents from the Chart object: version and appVersion. Being able to also override IsRoot might just be enough to "trick" the chart into complying with tests more easily, avoiding the necessity for a dummy parent chart, and making the tests as easy as running helm unittest.
However, I have no idea of the possible side effects of this change, or if it even is possible.
Describe alternatives you've considered
Only the dummy chart idea, above.

Is there a publicly available example somewhere? Could you share a full example with template, Chart.yaml and a suite?
I'm struggling with unittest and helm dependency update as well. This definitely needs a new flag.
But in regards .Chart.IsRoot not sure, there are probably ways to trick/fake call to some commands. Need some sort of investigation here


feat: add swift casbin idea

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Replaced by: https://github.com/casbin/Summer2023/pull/3


Add Networking

Adding networking to this game would be amazing! I am not sure if it is entirely possible without having to rewrite the entire game, but a possible suggestion would be to use socket.io. Is this possible? I would love to help, but I am fairly new to javascript.

I agree, networking would be a great feature, but it wouldn't be easy to implement.
Looking at socket.io, you could definitely use it to send game state data between players, but you'd need to write server side JavaScript to keep track of the game, remember whose turn it was, and make sure the players can't do anything against the rules.
Anyway, I don't have any plans to write networking, but if you design how the server part works and communicates to the clients, we can talk about modifying the existing JavaScript code to make that work.
Also, a great site to learn JavaScript is https://www.codecademy.com/.

I agree, networking would be a great feature, but it wouldn't be easy to implement.
Looking at socket.io, you could definitely use it to send game state data between players, but you'd need to write server side JavaScript to keep track of the game, remember whose turn it was, and make sure the players can't do anything against the rules.
Anyway, I don't have any plans to write networking, but if you design how the server part works and communicates to the clients, we can talk about modifying the existing JavaScript code to make that work.
Also, a great site to learn JavaScript is https://www.codecademy.com/.

Hi guys, have you started working on this feature?

No:(

You could potentially use PHP or Python but you'd definitely need some server-side component. You can do it without using sockets through the use of simple AJAX requests

You could potentially use PHP or Python but you'd definitely need some server-side component. You can do it without using sockets through the use of simple AJAX requests

I am not familiar with AJAX like at all, however I would be able to do the PHP end. Any idea on how the AJAX part would work?

I am not familiar with AJAX like at all, however I would be able to do the PHP end. Any idea on how the AJAX part would work?

It's also doable with node and you may be able to reuse some code on both sides.
I think we should use sockets since it enables us pushing statuses instead of (long) polling them, but that's a networking thing we can do after implementing this using Ajax.

It's also doable with node and you may be able to reuse some code on both sides.
I think we should use sockets since it enables us pushing statuses instead of (long) polling them, but that's a networking thing we can do after implementing this using Ajax.

Well, you'd need to send AJAX requests to the server in order to synchronize the state between multiple clients. So each time any user does anything significant in the game, you'd send one of these requests to the server. Now @arielsegura, using sockets is not a bad idea. HTML5 Server Sent Events can also be used and both of these options are better than polling. However, I do have somewhat of a personal bias against using Javascript for server-side code

Well, you'd need to send AJAX requests to the server in order to synchronize the state between multiple clients. So each time any user does anything significant in the game, you'd send one of these requests to the server. Now @arielsegura, using sockets is not a bad idea. HTML5 Server Sent Events can also be used and both of these options are better than polling. However, I do have somewhat of a personal bias against using Javascript for server-side code

We can also use spring-boot if you don't like node.
Sockets are supported there too.
Anyway, to start working on this we need to define what changes we need to do. What goes to the back and what stays at the front.

We can also use spring-boot if you don't like node.
Sockets are supported there too.
Anyway, to start working on this we need to define what changes we need to do. What goes to the back and what stays at the front.

Any of them work for me. I could look into HTML5 Server Sent Events if we wanted to go that route. Entirely up to us. What do you guys think? @arielsegura @admin96

Any of them work for me. I could look into HTML5 Server Sent Events if we wanted to go that route. Entirely up to us. What do you guys think? @arielsegura @admin96

Personally, I would prefer HTML5 SSE + PHP. It would be very simple to get started... all you need is WAMP/LAMP and you're ready to get started. PHP is kind of slow, so, I am open to using Python instead of PHP because it has better performance and the syntax is much better. Java might not be the best choice because writing ridiculously verbose OOP code for this web app would decrease productivity and increase the time needed to implement different functionality

I haven't used Python, but I'm sure I could figure it out. Did you want to get started? @admin96

It would be very simple to get started... all you need is WAMP/LAMP and you're ready to get started

You still need to download and configure the server. Node.js or spring-boot allows apps are by far easier to deploy. If someone only wants to download and run the app, they can do it w/o any server nor configurations.

What is the "develop" branch? gh-pages? So I fork the project

Its the same with python, you can do HTTP servers really easily without having apache!

gh-pages is the main branch. The game which can be played online is available through this branch!

Has somebody forked the project? Do we want to share emails or skype's to communicate outside of this?

join https://crazymonopoly.slack.com

Add me to the group<EMAIL_ADDRESS>

Joined! @arielsegura


TimeSpan Parameters Cannot be Deserialized for Activities

Repro: Sandbox.Functions.Worker.zip
I created a simple orchestration and activity, which I've attached, that passes a TimeSpan parameter. The orchestration is able to deserialize the value, but the activity throws an error:
[2023-09-25T19:08:33.116Z] Function 'EchoTimeSpanAsync', Invocation id '53fbfc18-c746-4d37-9fe6-723d025830bc': An exception was thrown by the invocation.
[2023-09-25T19:08:33.117Z] Result: Function 'EchoTimeSpanAsync', Invocation id '53fbfc18-c746-4d37-9fe6-723d025830bc': An exception was thrown by the invocation.
Exception: Microsoft.Azure.Functions.Worker.FunctionInputConverterException: Error converting 1 input parameters for Function 'EchoTimeSpanAsync': Cannot convert input parameter 'value' to type 'System.TimeSpan' from type 'System.String'. Error:System.Text.Json.JsonException: Invalid leading zero before '0'. Path: $ | LineNumber: 0 | BytePositionInLine: 1.
[2023-09-25T19:08:33.118Z]  ---> System.Text.Json.JsonReaderException: Invalid leading zero before '0'. LineNumber: 0 | BytePositionInLine: 1.
[2023-09-25T19:08:33.119Z]    at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
[2023-09-25T19:08:33.120Z]    at System.Text.Json.Utf8JsonReader.ConsumeZero(ReadOnlySpan`1& data, Int32& i)
[2023-09-25T19:08:33.121Z]    at System.Text.Json.Utf8JsonReader.TryGetNumber(ReadOnlySpan`1 data, Int32& consumed)
[2023-09-25T19:08:33.122Z]    at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
[2023-09-25T19:08:33.123Z]    at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
[2023-09-25T19:08:33.124Z]    at System.Text.Json.Utf8JsonReader.Read()
[2023-09-25T19:08:33.125Z]    at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
[2023-09-25T19:08:33.126Z]    --- End of inner exception stack trace ---
[2023-09-25T19:08:33.127Z]    at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)
[2023-09-25T19:08:33.127Z] Executed 'Functions.EchoTimeSpanAsync' (Failed, Id=53fbfc18-c746-4d37-9fe6-723d025830bc, Duration=68ms)
[2023-09-25T19:08:33.128Z]    at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
[2023-09-25T19:08:33.130Z] System.Private.CoreLib: Exception while executing function: Functions.EchoTimeSpanAsync. System.Private.CoreLib: Result: Failure

The message makes it sound like the value was serialized as a standard string (E.g. 00:01:00) while the deserializer expects JSON (E.g. "00:01:00").
Additionally, after failing the logs report the function invocation as having succeeded, while the DTFx reports the correct status of "Failed".
[2023-09-25T19:08:33.288Z] Executing 'Functions.EchoAsync' (Reason='(null)', Id=bbf776c1-e485-4dbf-97bd-fb2ae333208a)
[2023-09-25T19:08:33.414Z] Executed 'Functions.EchoAsync' (Succeeded, Id=bbf776c1-e485-4dbf-97bd-fb2ae333208a, Duration=128ms)

This same behavior occurs in versions 1.0.0, 1.0.2, and 1.0.3 of the package Microsoft.Azure.Functions.Worker.Extensions.DurableTask.

I have traced the issue down to how the WebJobs extension decides to serialize data to the worker. In the case of TimeSpan and other types which serialize to a JSON value type (ie: not an object { }), there is this code which this scenario falls into.
The end result is we erroneously unwrap the JSON input and send across as a strong. For TimeSpan we end up stripping the containing quotes from it.

Possibly related: https://github.com/Azure/azure-functions-durable-extension/issues/2504

Resolved by https://github.com/Azure/azure-functions-durable-extension/pull/2656


[CONTENT] Quickstart Basics

Hey @gdalle and fellow authors! I am beginning to draft a quickstart page that works for both new Julians as well as new programmers whose first language is Julia. I'll keep working on this Draft PR (feel free to take a look of course) as I continue to add content. If at any point content should be nixed, just let me know and I can remove it and put it maybe on Discourse or my blog.
Excited for this project! Thank you!

Hey @TheCedarPrince, thanks for the suggestions!
It's a good question whether or not this blog should target programming beginners. Since we do not cover Julia syntax, and we plan to dive pretty deep into type inference on page 3, it will probably never be a standalone resource, but that doesn't mean we can't try.
What do @adrhill and @jacobusmmsmit think?

I've taken into account the Windows-specific advice and integrated them myself since the file structure of the repo has changed. As for the first page, I'm still unsure so awaiting further opinions. Thanks a bunch!

Sounds good @gdalle ! Thanks for letting me know -- been kinda waiting on what you are thinking. I've been helping a friend of mine get started with Julia and have been using the Modern Julia Workflows guide to help him with getting started. It's been great so far!

Hey @gdalle -- anything you think you can take from here that would make sense to add into this PR? I wrote this blog post: https://jacobzelko.com/10082023195125-julia-learning-resources/ up for some students and friends of mine who want to get started with Julia. It has my own Cedarian spin on it so its not quite aligned perfectly with the project here, but was just curious if I can incorporate anything from it into this PR.

Thanks, lovely blog post! I spotted a duplicate link in item 3 of "what is a Julian" but otherwise it covers all the basics :)
I didn't see too many things that MoJuWo doesn't address, except the recommendations for specific tasks (data analysis, plotting) and the links to non-official tutorials. Both of those are out of scope for now


Make IO#skip IO#write returns the number of bytes it skipped/written

Closes #9185

IO#skip still allows any Int as the argument, but UInt64 is returned
Fix build (🤞) and added a couple of assertions w.r.t. #9185
Make IO#write_* : UInt64
Make IO#printf(...) : UInt64

I added a tiny amount of specs for write.
I am in doubt whether all write operations should return the byte counts or not.

If we limit to only  IO#write(Slice) it would seem rather limited.
If we allow all write_* methods there is a need to for a BytesCounter to allow nested writes accounting. Which is nice, but it adds some overhead clearly.
I added the same mechanism to count for IO#printf, but if that method returns the bytes written I feel that the same should happen with IO#puts and IO#print. I see puts and print as more high level operations, but I am not sure about were does printf fall.

It's a bit weird that write would return UInt64 but read Int32. I understand it is convenient, but it still feels odd.
I think this is ready for a round of feedback before going further.

Rebased and updated. I removed the wrapper and implement the accounting in the ByteFormat and Encoding. These are the only cases needed if we keep the printf, puts and other methods to return Nil. Making only the write_* return the number of bytes written.
@carlhoerberg , how does this sound for your use cases?

How's the performance for the nested byte accounting?

@carlhoerberg not good. It adds an extra layer. We also check that in ruby printf/puts return Nil.
I wanted to have a composable solution, hence I went on BytesCounter. But if the API is enough as it is right now, there is no need for it.

Maybe &+ should be used instead of +? I don't know if that's where the overhead comes from. Do we want the write to fail on overflow if exceeding UInt64 max?

Well, it can't overflow. There is not way to write more that UInt64 bytes in those method.
Since it can't overflow using &+ should be safe.
The overhead i refer was due to a IO wrapper, not the addition itself.

Ok, let's leave the ByteCounter out then, performance is way more important here. Let's just return the bytes where it's easy and cheap.

Why is the written length needed? What's a use case?

If we allow all write_* methods there is a need to for a BytesCounter

If the nested write calls return the size, why a wrapper is needed?

If we allow all write_* methods there

Also, what other write_ methods are there? I only found write_byte and write_bytes.

I only found write_byte and write_bytes

There is also write_utf8 and write(Slice). Those are the low level functions to write bytes, and where counting is more likely to be useful. If you do puts or printf the output is then more text oriented.
Note that IO#write in Ruby or Write in Go, also returns the number of bytes. For these functions is quite simple and direct to obtain this number.

@waj Oh, I see.
I'm still curious which methods needed a ByteCounter.

Also a note again that in Go for example the return value is an int, not an unsigned int. My guess is that if it works fine for Go, it will work fine for Crystal. But if you think UInt64 is better, then go ahead.

The return type of these methods should be UInt64. This will match #7660.
ref: (https://github.com/crystal-lang/crystal/pull/9185#pullrequestreview-400913944)

It bugs me a little the inconsistency with read operations. But it bugs me more if it would be with copy that is a write operation :-).

I think it's mainly low level so UInt64 is fine here.
@asterite making printf return number of bytes required (or was easier) with the wrapper. The implementation of String::Formatter would've be tainted with accounting of those bytes otherwise. And its implementation uses io <<, which returns the IO and not the amount of bytes written.

I changed the implementation to use &+ when possible.

Oh, I see. Yeah, for textual things maybe returning the number of bytes is not needed. In Go printf returns the number of bytes written, though. I just don't know what one does with that number anyway :-)

Is every write(slice) method supposed to return slice.bytesize.to_u64?  How much overhead if any does that add?

In Go Write returns the number of bytes written and an optional error. That means that you might get less bytes written in case of an error, and you know that.
Since in Crystal if write is successful it means all the slice was written, it's absolutely pointless to return slice.size. It's actually more job for the runtime for when you don't need it, and you can always know it by doing slice.size, as @didactic-drunk mentions.
What would be needed here is for exceptions to capture how many bytes were written before the exception happened.
That said, no use case has been provided for this change other than "avoiding to call size on the given slice instead of getting the value from the call".

@didactic-drunk @HankAnderson that's not true, consider this:
class CustomProtcol < IO
  def write(slice)
    bytes = 0_i64
    bytes += @io.write "prefix".to_slice
    bytes += @io.write slice
    bytes += @io.write "suffix".to_slice
  end
end

more than slice.size is written, so you can't just blindly return slice.size. Even more important in to_io, eg.
struct ShortString
  def initialize(@string)
  end

  def to_io(io)
    io.write_byte @string.bytesize.to_u8
<EMAIL_ADDRESS>+ sizeof(UInt8)
  end
end

socket = TCPSocket
bytes_written = socket.write_bytes ShortString.new("foo")
bytes_written # 4, not 3

It vastly simplifies position accounting.

I don't think a custom protocol should be an IO, really... you are using IO in a wrong way. For example if you print an object, or puts an object to that IO, you will get many prefixes and suffixes, which is probably not expected.
If you want to track written slice size in the custom protocol (which is not an IO!) then do it, but all IOs just write the full slice as a byte size.
For that matter, Flate, Gzip, etc. currently return the passed slice size, but in reality they might write less data (because it is compressed!). So already writer's return value is confusing. Is it returning how much data was actually written, or how much of the slice was written? If it's the latter, then your custom protocol should return the slice size, regardless of what else was written. If it's the former, then the stdlib is already broken.
If you check what Go does, https://golang.org/pkg/io/#Writer (maybe you wanted this is Crystal because of Go or Ruby?) you'll notice that the return value is always in bounds with the passed slice size. It's returning how many bytes were written from the slice, not to the actual device (this is hard or impossible to know when data is buffered and compressed).
So your custom protocol should return slice.size.
In fact all IOs on crystal will return that because if they are successful, all the slice will be written out. If they are not, an exception will be raised. The problem is that this exception doesn't  tell you how many bytes were written, and this is the only useful thing that we should add if we wanted to.
But right now, IO#write return value is completely broken and nonsense.

But right now, IO#write return value is completely broken and nonsense.

I agree. It's also completely lacking any documentation.
In what use case would it not return slice.size?

yes, it was a bad example, i'm not implementing a protocol with a custom IO, but the write_bytes is correct, you can't know how many bytes that was actually written if write_bytes/to_io doesn't return the actual number of written written, with prefixes/compressed or whatever. The "problem" it solves is that you don't have to have to repeatedly call IO#pos, which gets expensive on files and impossible on sockets, or riddle your code with implementation details (how many extra bytes does a ShortString add?)
But I agree, the lack of enforcing is problematic.

To me it's a nice to have, not a need to have, you can hack around it with manual position accounting. And its current form seems to cause a lot of confusion and is hard (impossible?) to enforce.

The "problem" it solves is that you don't have to have to repeatedly call IO#pos

Maybe you can show us your code to understand why you need to know the position at every point?

you can't know how many bytes that was actually written if write_bytes/to_io doesn't return the actual number of written written

Then those methods should return how many bytes were written, but there's no need to do it for IO#write.

We can enforce that the write returns always the slice size if we change the design of IO and add write_impl(io : IO) : Nil (or whatever name) that can be override. Then IO#write will take care of returning the slice.
But that will be half way. The write_bytes implementation is also subject to not behave as expected.
There are some contracts that can't be enforced by types. We can discuss a design that is less fragile, but it can always be broken in some way by the implementation.
I agree that protocols are better to be implemented by using and IO rather than inheriting one. But if we think of some transformation that can be activated or not to decorate an IO, inheriting make sense in the current state of the std-lib. If these decorators change the the amount of bytes written from the input maybe it is desired to still keep track of the underlying position count. I know is a maybe. But that is why I feel more comfortable with the proposed change.

I think the changes on the IO interface were a mistake and we should revert it. Let me explain why:

Trivial: the value that should be returned by IO#write is always slice.size, otherwise the implementation is incorrect. Returning a value that could be trivially calculated at the call site sounds bad to me.
Confusing: this could be mitigated with documentation, but the interface alone is confusing, I think mainly because of the previous reason. Is it the returned value the number of bytes meant to be write or the bytes actually written? Because the first option it's too trivial to be true, then it feels the answer should be the second one. (but note that is impossible in the general case)
Error prone: as we could see already, if someone extends IO but it doesn't return slice.size, then the implementation is wrong.

At the end we have something that makes programmers less happy without too much gain.
Now, for the cases where having this value is actually useful, those could be solved in a different way:

Make a (private) alternative method to write bytes, that also returns the slice.size
Create a AccountingIO that keeps count of bytes written so it always know the position without constantly calling pos.

I know going back and forward with changes like this is far from ideal. But since we're not at 1.0 yet it's a good opportunity to assume our mistakes and fix them before it's too late.

Just for reference, because the original discussion about adding byte count to IO#write isn't linked here: It's at #9186


Move instrumentation to be SpatialEntry field

Resolves part of #354
Now all SpatialEntrys can be search against instrumentation

@matt-bernstein, this is the last bit to resolve #354

Merging due to deliverable timelines


Optimisation for PassiveTreeView: cache the search results.

Simple but effective optimisation for the PassiveTreeView, caching the search results saves several thousand calls of string.match() which were quite expensive.
Storing the previous string used isn't the neatest solution but I think it's the most error resistant. Alternatives include:

A boolean searchStrNeedsChecking, which requires the anyone who sets searchStr to set that too.
The same flag with a SetSearchStr(str) function to set it for you, but you'd still need to remember to call the setter instead of setting it directly (Lua doesn't do private variables).
Use the function above and iterate over the nodes in that function rather than doing it later, which risks multiple calls in the same frame causing spikes, and you'd probably still have to store the string somewhere so you can save/load it so it has the same problem as the above.

Attempt 3: Now with more dev, less commits, more rebase, and less trees!

3rd time is indeed lucky. Thanks for the contribution!