Elassandra mappings with type coercing aren't behaving the same as elasticsearch

Elassandra version: <IP_ADDRESS>
JVM version (java -version): docker image
OS version (uname -a if on a Unix-like system): host system is OSx but I also tested it in GKE.
Description of the problem including expected versus actual behavior:
When using a mapping template, that enables type coercing, Elassandra doesn't behave the same as ElasticSearch.  Instead it returns a 500 and an exception.
Steps to reproduce:
In both of these examples we'll be inserting a document that has an integer mapping.  And in both cases the int field's value will be submitted as a string value.
Testing Elasticsearch 6.8
docker pull docker.elastic.co/elasticsearch/elasticsearch:6.8.6
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.8.6
export HOST=localhost
export INDEX=ents_type_site1
echo '{
  "index_patterns": "ents_*",
  "mappings": {
    "_doc": {
      "properties": {
        "_created": {
          "type": "date"
        }
      },
      "dynamic_templates": [
        {
          "integer_fields": {
            "match": "int_*",
            "mapping": {
              "coerce": true,
              "type": "long"
            }
          }
        }
      ]
    }
  }
}' | http PUT http://$HOST:9200/_template/ent_table_template
{
    "acknowledged": true
}

echo '{"int__eventcnt":"1"}' |   http POST http://$HOST:9200/$INDEX/_doc/a1bc=:1cd==
{
    "_id": "a1bc=:1cd==",
    "_index": "ents_type_site1",
    "_primary_term": 1,
    "_seq_no": 0,
    "_shards": {
        "failed": 0,
        "successful": 1,
        "total": 2
    },
    "_type": "_doc",
    "_version": 1,
    "result": "created"
}

Note that elasticsearch handled the casting of the string to an int.

Testing elassandra 6.8
docker run  -p 9200:9200 -p 9300:9300 --name node0 -d strapdata/elassandra:<IP_ADDRESS>
export INDEX=ents_type_site1
echo '{
  "index_patterns": "ents_*",
  "mappings": {
    "_doc": {
      "properties": {
        "_created": {
          "type": "date"
        }
      },
      "dynamic_templates": [
        {
          "integer_fields": {
            "match": "int_*",
            "mapping": {
              "coerce": true,
              "type": "long"
            }
          }
        }
      ]
    }
  }
}' | http PUT http://$HOST:9200/_template/ent_table_template
{
    "acknowledged": true
}

echo '{"int__eventcnt":"1"}' |   http POST http://$HOST:9200/$INDEX/_doc/a1bc=:1cd==
{
    "error": {
        "reason": "java.lang.String cannot be cast to java.lang.Number",
        "root_cause": [
            {
                "reason": "[<IP_ADDRESS>][<IP_ADDRESS>:9300][indices:data/write/bulk[s][p]]",
                "type": "remote_transport_exception"
            }
        ],
        "type": "class_cast_exception"
    },
    "status": 500
}

Please provide the following information:

elassandra logs (logs/system.logs or /var/lib/cassandra/system.log)

2020-01-10 06:01:40,015 WARN  [elasticsearch[<IP_ADDRESS>][http_server_worker][T#2]] org.elasticsearch.common.logging.DeprecationLogger.deprecated(DeprecationLogger.java:242) [types removal] The parameter include_type_name should be explicitly specified in put template requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions.
2020-01-10 06:01:42,691 INFO  [elasticsearch[<IP_ADDRESS>][generic][T#3]] org.elassandra.shard.CassandraShardStateListener.afterIndexShardStarted(CassandraShardStateListener.java:69) shard [ents_type_site1][0] started
2020-01-10 06:01:42,994 ERROR [elasticsearch[<IP_ADDRESS>][write][T#1]] org.elassandra.cluster.QueryManager.upsertDocument(QueryManager.java:842) [java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Number].[ents_type_site1] failed to parse field _doc=int__eventcnt
2020-01-10 06:01:43,011 WARN  [elasticsearch[<IP_ADDRESS>][write][T#1]] org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:133) path: /ents_type_site1/_doc/a1bc=:1cd==, params: {index=ents_type_site1, id=a1bc=:1cd==, type=_doc}
org.elasticsearch.transport.RemoteTransportException: [<IP_ADDRESS>][<IP_ADDRESS>:9300][indices:data/write/bulk[s][p]]
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Number
        at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$7.cqlValue(NumberFieldMapper.java:870)
        at org.elasticsearch.index.mapper.NumberFieldMapper$NumberFieldType.cqlValue(NumberFieldMapper.java:1057)
        at org.elasticsearch.index.mapper.MappedFieldType.cqlValue(MappedFieldType.java:423)
        at org.elassandra.cluster.Serializer.serialize(Serializer.java:230)
        at org.elassandra.cluster.Serializer.serialize(Serializer.java:223)
        at org.elassandra.cluster.QueryManager.upsertDocument(QueryManager.java:840)
        at org.elassandra.cluster.QueryManager.insertDocument(QueryManager.java:694)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:468)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:495)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:464)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:226)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:169)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:160)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:146)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:83)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1088)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1066)
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:105)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:445)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:387)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:327)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:314)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:696)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

elasticsearch cluster state (curl http://localhost:9200/_cluster/state)
cassandra schema (cqlsh>DESC KEYSPACE <your_keyspace>)
cassandra gossip state (run: nodetool gossipinfo)

Elassandra is not Elasticsearch, so don't force the discovery to single-node and follow the Elassandra quick-start guide .

@vroyer what?  This has nothing to do with discovery...  This about index mappings and string coercing, i.e. Elassandra can't handle coercing the term "100" into 100.

I only used docker in the reproduction steps to keep things simple.  And the same exception happens when you use the _bulk api.
For our evaluation we are testing Elassandra in Kubernetes, which I setup using Google's Application Market place.
The reason we use coercing is that when converting int64's to float64's for JSON, some precision is lose.  So, for large numbers we marshal the int64 as strings, and then let Elasticsearch coerce them back into number types.

@vroyer I think I figured out your confusion.  In the reproduction steps I had steps for starting up both an ElasticSearch node and steps for starting up a Elassandra node.  The command "discovery.type=single-node" was only used in the ElasticSearch example, please continue reading...  The important part is the results of inserting a single document into both ElasticSearch vs Elassandra.

Sorry for closing the ticker, i thought you were trying to run elassandra with the single-node discovery.
Effectively, quoted numbers are not interpreted as number by elassandra, while it is properly interpreted by elasticsearch.  Without quotes, elassandra works fine:
put "$INDEX/_doc/a1bc=:1cd==?pretty" '{"int__eventcnt":1}'
{
"_index" : "ents_type_site1",
"_type" : "_doc",
"_id" : "a1bc=:1cd==",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
}
}
We are going to fix that quickly. Thanks for reporting the issue.


git+ssh for private git repositories

Expected behavior
It should work for SSH key.
.netrc prevent you from using two-factor-authentication.
https://help.github.com/articles/requiring-two-factor-authentication-in-your-organization/
Actual behavior
It works for ~/.netrc https://github.com/tools/godep/issues/58
godep version output
godep v74 (darwin/amd64/go1.7rc1)
go version output
go version go1.7rc1 darwin/amd64
Contents of Godeps.json file
private repos

godep just shells out to git.
On Thu, Sep 15, 2016 at 9:40 AM Cícero Verneck Corrêa <
<EMAIL_ADDRESS>wrote:

Expected behavior
It should work for SSH key.
.netrc prevent you from using two-factor-authentication.
https://help.github.com/articles/requiring-two-factor-authentication-in-your-organization/
Actual behavior
It works for ~/.netrc #58 https://github.com/tools/godep/issues/58
godep version output
godep v74 (darwin/amd64/go1.7rc1)
go version output
go version go1.7rc1 darwin/amd64
Contents of Godeps.json file
private repos
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tools/godep/issues/514, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAAZ-bXC54PqVhhU1p87WZKizF74rYKks5qqXT3gaJpZM4J-FUI
.

I am closing this issue because the repo is going to be archived.


Feature Request: multiply keys from differend accounts, faster notifcation posting?

Is it possible to make so you can apply more than 1 key from differend accounts and have it fetch notifications more often
like 1 key = fetch ever 30 min, 2 keys = every 15 min, 3 keys every 10m
and have config options to define how often it should scan and how many keys is suppiled
Like
"keys" => "#", // Numbers of keys you have supplied
"interval" => "#" // how often in min it should try and fetch notifications

Mmm this is doable


[ZEPPELIN-3593] Change LuceneSearch's directory to file system from memory

What is this PR for?
Reducing a usage of memory by moving Lucene search's directory to the file system
What type of PR is it?
[Improvement]
Todos

[ ] - Replace RamDirectory to MMapDirectory

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-3593

How should this be tested?
It doesn't affect any current behaviors. Test search service
Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

Will zeppelin reindex all notes when zeppelin server is restarted ? If we store it in file system, we should be able to avoid this kind of reindex.

Yes, agreed at that point. But in case where users add new notes manually,
we cannot know them. It means we should read all of indexes and match them
to sync file system and indexes. So I suggest to reduce memory usage at
first and then think how to increase the performance of indexes later.
On Mon, 9 Jul 2018 at 5:37 PM Jeff Zhang<EMAIL_ADDRESS>wrote:

Will zeppelin reindex all notes when zeppelin server is restarted ? If we
store it in file system, we should be able to avoid this kind of reindex.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/zeppelin/pull/3058#issuecomment-403403584, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADcflpemFfrSJJ2mUONMzq-mLmAFSR05ks5uExZagaJpZM4VHIzc
.

--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Thanks @jongyoul  feel free to create followup ticket.


Removing path and line number from warnings.

Declutters the warning message so the message is easier to see
Removes the warnings.warn(message) from the message as well.
Difference between old and new shown below:

Previous:
/Users/DevinPetersohn/software_builds/modin/modin/error_message.py:32: UserWarning: `cov` defaulting to pandas implementation.
To request implementation, send an email to<EMAIL_ADDRESS>  warnings.warn(message)

This commit:
WARN: `cov` defaulting to pandas implementation.
To request implementation, send an email to<EMAIL_ADDRESS>

[x] passes git diff upstream/master -u -- "*.py" | flake8 --diff
[x] passes black --check modin/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/344/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/405/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/345/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/406/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/474/
Test PASSed.


Remove unnecessary null pointer checks

Extra null pointer checks are not needed in the destructor for the class "ServerData".

I appreciate that it is not necessary to check the pointer before calling delete, but I don't think this is worth fixing. The profile.cpp is quickly hacked up sample code for profiling the library and not part of the library or production code. -- cheers


Set a customisable rate limit to the logging rules

As discussed in #19, this patch adds an extra variable to the main interface allowing to customise the rate limiting policy applied to the rules that log discarded traffic. By default, a limit is configured as follows:
chain INPUT {
...
limit rate 3/minute log prefix "[nftables] FORWARD Rejected: " flags all counter packets 0 bytes 0
...
}
# same for OUTPUT and FORWARD

Closes #19.

@keachi PTAL


Support for indian number plate

Hi @SandroMachado
I am stuck on indian number plates.
Can you please help me to solve this problem.

Hi @Sumituppal-delhivery.
this project only supports US and EU licence plates.
Regards

Can you please me guide me how can i do it for indian licence plates.

You need to check the parent project: https://github.com/openalpr/openalpr


Help Wanted: Document usage of .nvmrc in plugin projects

Context of where this documentation be useful https://github.com/mattermost/mattermost-plugin-todo/issues/198#issuecomment-1441116706
Example

You'll need to install node 14, or install https://nvm.sh and run nvm install in the root of this repository

@cwarnermm This issue is marked as completed, but I'm not sure if it ever was worked on

@cwarnermm can I claim the issue and work on it?

@imsherlocked Absolutely! Note that the general message to convey is:

Different plugin projects may require specific versions of Node.js. The recommended version for the given project is always defined in a file in the root of the repository called .nvmrc. You can use the tool https://nvm.sh (Node Version Manager) to install and switch between node versions in your terminal. If you have nvm installed, you can run nvm install anywhere in the plugin repository, and it will automatically find the .nvmrc file in the root, and install and use that version. If you already have that version installed, you can run nvm use, though nvm install is easier since you don't need to check if you already have it installed.

@cwarnermm I would like to add this note and complete this ticket. Feel free to assign this to me.

Thanks, @Rutam21!


close() on input pipe does not cleanup child process

When close() is called on an input pipe, the close() builtin function does not .Wait for the exec.Cmd at the other end of the "pipe" and so the resources associated with the child process are not released. This leaves "zombie processes" which continue to be tracked in the parent, and maybe store their return value. close() will cause them to be forgotten by the interpreter so new processes will be spawned for the same command.
Example
You can easily reach reach the limit on the number of open processes, at which point the fork/exec call no longer works. Here, I set the limit to 512 (pretty low) and give it a go.
$ ulimit -u 512
$ goawk 'BEGIN { for(i=1;i<=512;i++){ "date"|getline; close("date") }}'
fork/exec /bin/sh: resource temporarily unavailable
fork/exec /bin/sh: resource temporarily unavailable
fork/exec /bin/sh: resource temporarily unavailable
...

I thought it would be nice to have interfaces for input/output streams and specific structs for file/pipe streams. I still have to make another pass to clean things up and also write tests. But the commit above fixed my problem here and the minor issue in #203.

This is interesting work, thanks @juster. Cleaning up the structure of handling file/pipe streams is something I've been hoping to do for a while -- I half suspected this was an issue but have never looked into it. So thanks. I'll look at this more closely (including your commit) in the next couple of weeks.

Out of interest, @juster: what are you using GoAWK for, and how did you stumble across this issue? I'm always interested to learn more about how it's being used.

I use goawk all the time for analyzing server logs. They can be in the GBs. A coworker suggested goawk because it parses CSV. I used to use a lot of Perl or python but I decided to experiment if awk was capable of more or less the same thing. Goawk is also faster than nawk and often even grep!
One weakness with awk is with time stamps. I found this bug because I was trying to parse time stamps with the date command! So I was opening quite a few commands.
Regardless of this bug it was too slow. But then I discovered I could provide custom awk functions written in Go and embed a script in a binary. I love it!


feat: Ignore health checks for tracing purposes

Health checks tend to pollute the trace list and often cause other traces to get dropped when the appserver rate limits automatic traces.

/gcbrun

/gcbrun


Relax script detection

Fixes #14
Tweaks the logic for detecting whether Stripe.js has already been loaded.
Previously it would expect Stripe.js to be loaded without a trailing slash on the URL (https://js.stripe.com/v3).
Now it will detect the script either with or without a trailing slash appended to this URL (https://js.stripe.com/v3 or https://js.stripe.com/v3/).

@dweedon-stripe and I were thinking of using the "starts with" [attr^=value] selector to support other future entrypoints.


Remote debug on Linux Arm Xilinx Petalinux (Yocto) gives broken pipe error

Issue Description
I am trying to remote debug a default console app from VSCode on Windows 10 to a Xilinx ARM Linux target.  The Linux is built using Xilinx's petalinux tools (Yocto based).  By default petalinux is 64-bit only, however I was able to alter it to be multilib and have installed the 32-bit ARM dotnet 3 preview 3 sdk, which seems to be working.  However I am not able to debug remotely.
Steps to Reproduce
Create new default console app.
Try and remote debug per Linux-arm instructions.
Expected Behavior
debugging
Actual Behavior
"The pipe program 'plink.exe' exited unexpectedly with code 0."
Logs
Starting: "c:\Program Files (x86)\PuTTY\plink.exe" -l root -pw <DELETED> <IPADDRESS> "~/vsdbg/vsdbg --interpreter=vscode"

-> (C) {"command":"initialize","arguments":{"clientID":"vscode","clientName":"Visual Studio Code","adapterID":"coreclr","pathFormat":"path","linesStartAt1":true,"columnsStartAt1":true,"supportsVariableType":true,"supportsVariablePaging":true,"supportsRunInTerminalRequest":true,"locale":"en-us"},"type":"request","seq":1}

-> (C) {"command":"launch","arguments":{"name":".NET Core Remote Launch - Framework Dependent (console)","type":"coreclr","request":"launch","program":"~/dotnet/dotnet","args":["~/armtestNet3/armtestNet3.dll"],"cwd":"~/armtestNet3","stopAtEntry":true,"console":"internalConsole","logging":{"engineLogging":false},"pipeTransport":{"pipeCwd":"C:\\Users\\chris\\source\\armtestNet3","pipeProgram":"c:\\Program Files (x86)\\PuTTY\\plink.exe","pipeArgs":["-l","root","-pw","<DELETED>","<IPADDRESS>"],"debuggerPath":"~/vsdbg/vsdbg"},"internalConsoleOptions":"openOnSessionStart","__sessionId":"fd809e3a-4f1f-47de-817d-fb15672e009c"},"type":"request","seq":2}

<- (E) {"seq":2,"type":"event","event":"output","body":{"category":"console","output":"-------------------------------------------------------------------\nYou may only use the Microsoft .NET Core Debugger (vsdbg) with\nVisual Studio Code, Visual Studio or Visual Studio for Mac software\nto help you develop and test your applications.\n-------------------------------------------------------------------\n"}}

-------------------------------------------------------------------
You may only use the Microsoft .NET Core Debugger (vsdbg) with
Visual Studio Code, Visual Studio or Visual Studio for Mac software
to help you develop and test your applications.
-------------------------------------------------------------------
<- (R) {"seq":3,"type":"response","request_seq":2,"success":true,"command":"launch"}

<- (E) {"seq":4,"type":"event","event":"initialized","body":{}}

-> (C) {"command":"setBreakpoints","arguments":{"source":{"name":"Program.cs","path":"C:\\Users\\chris\\source\\armtestNet3\\Program.cs","checksums":[{"algorithm":"SHA1","checksum":"92563e218acdd22b67550e2da721079f7d91a162"},{"algorithm":"SHA256","checksum":"5a7689ea23724d5c707ae53d842263c89c241b1941b37c251e71a673bae82681"},{"algorithm":"SHA1","checksum":"dd6ab0f04bf28a1ddbaedfccd61a199b7e661c82"},{"algorithm":"SHA256","checksum":"5fe112aa51807dc93df2eac8e6888a33bc0f9ebd25aff992bdcbc0bb5884c7f5"}]},"lines":[9],"breakpoints":[{"line":9}],"sourceModified":false},"type":"request","seq":4}

<- (R) {"seq":5,"type":"response","request_seq":4,"success":true,"command":"setBreakpoints","message":"","body":{"breakpoints":[{"id":1,"verified":false,"message":"The breakpoint is pending and will be resolved when debugging starts.","line":9}]}}

-> (C) {"command":"setExceptionBreakpoints","arguments":{"filters":["user-unhandled"]},"type":"request","seq":5}

<- (R) {"seq":6,"type":"response","request_seq":5,"success":true,"command":"setExceptionBreakpoints"}

-> (C) {"command":"configurationDone","type":"request","seq":6}

<- (ERROR) {"request_seq":6,"success":false,"command":"configurationDone","message":"The pipe program 'plink.exe' exited unexpectedly with code 0.","body":{"error":{"id":1000,"format":"The pipe program 'plink.exe' exited unexpectedly with code 0."}},"seq":7,"type":"response"}

OmniSharp log
Starting OmniSharp server at 4/4/2019, 1:15:22 PM
Target: c:\Users\chris\source\armtestNet3
OmniSharp server started.
Path: C:\Users\chris.vscode\extensions\ms-vscode.csharp-1.18.0.omnisharp\1.32.11\OmniSharp.exe
PID: 6504
    Starting OmniSharp on Windows 6.2.9200.0 (x64)

    DotNetPath set to dotnet

    Located 2 MSBuild instance(s)
        1: Visual Studio Professional 2019 16.0.28729.10 - "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin"
        2: StandAlone 15.0 - "c:\Users\chris\.vscode\extensions\ms-vscode.csharp-1.18.0\.omnisharp\1.32.11\msbuild\15.0\Bin"

    Registered MSBuild instance: Visual Studio Professional 2019 16.0.28729.10 - "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin"

    Detecting Cake files in 'c:\Users\chris\source\armtestNet3'.

    Could not find any Cake files

    Project system 'OmniSharp.DotNet.DotNetProjectSystem' is disabled in the configuration.

    No solution files found in 'c:\Users\chris\source\armtestNet3'

    Queue project update for 'c:\Users\chris\source\armtestNet3\armtestNet3.csproj'

    Detecting CSX files in 'c:\Users\chris\source\armtestNet3'.

    Could not find any CSX files

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.CSharpWorkspaceOptionsProvider

    Loading project: c:\Users\chris\source\armtestNet3\armtestNet3.csproj

    Configuration finished.

    Omnisharp server running using Stdio at location 'c:\Users\chris\source\armtestNet3' on host 2636.

    Successfully loaded project file 'c:\Users\chris\source\armtestNet3\armtestNet3.csproj'.

    Adding project 'c:\Users\chris\source\armtestNet3\armtestNet3.csproj'

    Update project: armtestNet3

    Queue project update for 'c:\Users\chris\source\armtestNet3\armtestNet3.csproj'

    Queue project update for 'c:\Users\chris\source\armtestNet3\armtestNet3.csproj'

    Loading project: c:\Users\chris\source\armtestNet3\armtestNet3.csproj

    Successfully loaded project file 'c:\Users\chris\source\armtestNet3\armtestNet3.csproj'.

    Update project: armtestNet3

    Queue project update for 'c:\Users\chris\source\armtestNet3\armtestNet3.csproj'

    Queue project update for 'c:\Users\chris\source\armtestNet3\armtestNet3.csproj'

    Loading project: c:\Users\chris\source\armtestNet3\armtestNet3.csproj

    Successfully loaded project file 'c:\Users\chris\source\armtestNet3\armtestNet3.csproj'.

    Update project: armtestNet3

    armtestNet3: Did not find 'Microsoft.NETCore.Platforms' in lock file.

    armtestNet3: Did not find 'Microsoft.WindowsDesktop.App' in lock file.

    armtestNet3: Did not find 'runtime.win-x64.Microsoft.NETCore.DotNetAppHost' in lock file.

C# log
NONE
Environment information
VSCode version: 1.32.3
C# Extension: 1.18.0
Dotnet Information
.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview3-010431
 Commit:    d72abce213
Runtime Environment:
OS Name:     Windows
OS Version:  10.0.17763
OS Platform: Windows
RID:         win10-x64
Base Path:   C:\Program Files\dotnet\sdk\3.0.100-preview3-010431\
Host (useful for support):
Version: 3.0.0-preview3-27503-5
Commit:  3844df9537
.NET Core SDKs installed:
2.1.602 [C:\Program Files\dotnet\sdk]
3.0.100-preview3-010431 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.0.0-preview3-19153-02 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.0.0-preview3-27503-5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.0.0-preview3-27504-2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

Visual Studio Code Extensions

Extension
Author
Version

csharp
ms-vscode
1.18.0

It looks like what is going on is that vsdbg is abruptly exiting (crashing?) once it tries to start debugging.
Two questions:

I am assuming you are running the latest version of vsdbg on your arm machine?
Can you try running this test on your arm machine to make sure this isn't a libicu compatibility problem?

Yes, fresh latest install.
Test passed both with and without DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1

I was wondering if maybe I'm missing some other 32-bit lib, but as far as I can tell so far I have what is needed.
root@xilinx-zcu106-2018_3:~/vsdbg# ldd32 vsdbg
	libvsdbg.so => /home/root/vsdbg/./libvsdbg.so (0xf7b95000)
	libvsdebugeng.impl.so => /home/root/vsdbg/./libvsdebugeng.impl.so (0xf7aa7000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xf7961000)
	libvsbaseservices.so => /home/root/vsdbg/./libvsbaseservices.so (0xf78cc000)
	libvsdebugeng.so => /home/root/vsdbg/./libvsdebugeng.so (0xf775a000)
	libdl.so.2 => /lib/libdl.so.2 (0xf7747000)
	libpthread.so.0 => /lib/libpthread.so.0 (0xf771e000)
	libm.so.6 => /lib/libm.so.6 (0xf769f000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf7672000)
	libc.so.6 => /lib/libc.so.6 (0xf7537000)
	/lib/ld-linux-armhf.so.3 (0xf7c9d000)

Yes, I also suspect a missing dependency might be the problem. I think the next thing I might suggest is taking a system call trace using strace and maybe we can see some missing file probes or other better indications of the problem. If that doesn't work, I can dig up how to run ulimit with vsdbg so that we can attempt to gather a coredump if it is crashing.

strace attached.  I haven't picked through it yet.  I thought I would throw it up here first and then start digging in.  BTW, handy script for starting strace before the process is started can be found here: http://konrness.com/linux/how-to-strace-a-process-that-has-not-started-yet/
straceVsdbg.txt

@bigredhdl It looks like this isn't a missing library problem but rather something in the vsdbg process is attempting to read or write to unaligned memory (at least if I understand the meaning of this signal code):
[pid  8647] 13:30:57 --- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRALN, si_addr=0xf880ae} ---

I will see if I can figure out how we can debug this.

Here are some troubleshooting steps: https://github.com/OmniSharp/omnisharp-vscode/wiki/Attaching-to-remote-processes#2-save-a-coredump-of-a-possible-vsdbg-crash

coredump attached.
vsdbg-2401.cor.zip

Backtrace:
root@xilinx-zcu106-2018_3:~/crash_reports# gdb ~/vsdbg/vsdbg ~/crash_reports/vsdbg-2401.cor
GNU gdb (GDB) 8.0
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-xilinx-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/root/vsdbg/vsdbg...done.
[New LWP 2401]
[New LWP 2404]
[New LWP 2402]
[New LWP 2405]
[New LWP 2408]
[New LWP 2406]
[New LWP 2403]
[New LWP 2407]

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/home/root/vsdbg/vsdbg --interpreter=vscode'.
Program terminated with signal SIGBUS, Bus error.
#0  ATL::CElementTraits<_GUID>::Hash (guid=...) at /root/Concord/src/unix/inc/atl/atlcoll.h:208
208	/root/Concord/src/unix/inc/atl/atlcoll.h: No such file or directory.
[Current thread is 1 (LWP 2401)]
(gdb) bt
#0  ATL::CElementTraits<_GUID>::Hash (guid=...) at /root/Concord/src/unix/inc/atl/atlcoll.h:208
#1  ATL::CAtlMap<_GUID, int (*)(_GUID const&, void**), ATL::CElementTraits<_GUID>, ATL::CElementTraits<int (*)(_GUID const&, void**)> >::GetNode (
    this=<optimized out>, key=..., iBin=<optimized out>, nHash=<optimized out>, pPrev=<optimized out>) at /root/Concord/src/unix/inc/atl/atlcoll.h:2649
#2  ATL::CAtlMap<_GUID, int (*)(_GUID const&, void**), ATL::CElementTraits<_GUID>, ATL::CElementTraits<int (*)(_GUID const&, void**)> >::Lookup (
    this=<optimized out>, key=...) at /root/Concord/src/unix/inc/atl/atlcoll.h:2716
#3  CConcordClassMap::GetClassObject (this=0xf7712bd0 <s_VSDebugEngImplClassMap>, riid=..., ppvObject=<optimized out>, rclsid=...)
    at /root/Concord/src/inc/concord/ConcordClassMap.h:49
#4  VSDebugEngImplGetClassObject (rclsid=..., riid=..., ppv=<optimized out>) at /root/Concord/src/inc/concord/VSDebugEng.Impl.ComponentHost.h:219
#5  DllGetClassObject (rclsid=..., riid=..., ppv=0xffb1ebf0) at /root/Concord/src/impl/dll/vsdebugeng.impl.cpp:127
#6  0xf73c9b8c in dispatcher::XapiClassInfo::LoadNativeClass (this=0x1ff878) at /root/Concord/src/Dispatcher/XapiClassInfo.cpp:343
#7  0xf73c9ab8 in dispatcher::XapiClassInfo::LoadNativeImplementation (this=0x1ff878, pOperation=0xffb1ee70, pInterfaceImpl=0x1ff894)
    at /root/Concord/src/Dispatcher/XapiClassInfo.cpp:234
#8  0xf73bf312 in dispatcher::XapiRuntime::LoadImplementation (pOperation=0xffb1ee70, pInterfaceImpl=0x1ff894)
    at /root/Concord/src/Dispatcher/XapiRuntime.cpp:2160
#9  0xf73bef06 in dispatcher::XapiRuntime::InitUnicastOperation (pOperation=0xffb1ee70,
    methodLocationConstraint=dispatcher::XapiMethodLocationConstraint::NoWP) at /root/Concord/src/Dispatcher/XapiRuntime.cpp:891
#10 0xf7358be2 in dispatcher::DefaultPort::DkmTransportConnection::GetSystemInformation (this=0x20b628,
    NativeSystemInfo=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ppSystemInfo=0xffb1ef10)
    at /root/Concord/src/api/CompilerBaseline/VSDebugEng.ImplDefinition.h:115485
#11 0xf767f7e8 in Microsoft::VisualStudio::Debugger::DefaultPort::DkmTransportConnection::GetSystemInformation (this=0xf7712bd0 <s_VSDebugEngImplClassMap>,
    NativeSystemInfo=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ppSystemInfo=0xffb1ef10)
    at /root/Concord/src/api/CompilerBaseline/VSDebugEng.Internal.h:104167
#12 ManagedDM::CV2Process::GetSystemInformation (pConnection=0xf7712bd0 <s_VSDebugEngImplClassMap>, ppSystemInformation=0xffb1ef10)
    at /root/Concord/src/impl/ManagedDM/V2/V2Process.cpp:3310
#13 ManagedDM::CV2Process::CreateEventThread (this=0x254038, pRequest=0x2a88b0, ppDkmProcess=0xffb1efdc)
    at /root/Concord/src/impl/ManagedDM/V2/V2Process.cpp:527
#14 0xf7687726 in ManagedDM::CV2EntryPoint::AttachToProcess (this=<optimized out>, pRequest=0x2a88b0, ppDkmProcess=0xffb1efdc)
    at /root/Concord/src/impl/ManagedDM/V2/_V2EntryPoint.cpp:62
#15 0xf733e98c in dispatcher::Start::DkmProcessAttachRequest::AttachToProcess (this=0x2a88b0, ppProcess=0xffb1efdc)
    at /root/Concord/src/api/CompilerBaseline/VSDebugEng.ImplDefinition.h:89035
#16 0xf77314f0 in Microsoft::VisualStudio::Debugger::Start::DkmProcessAttachRequest::AttachToProcess (this=0xf7712bd0 <s_VSDebugEngImplClassMap>,
    ppProcess=0xf73e6074 <_GUID_00000001_0000_0000_c000_000000000046>) at /root/Concord/src/api/CompilerBaseline/VSDebugEng.Internal.h:100807
#17 CVsDbg::AttachToProcessInternal (this=<optimized out>, processId=2409, startMethod=<optimized out>, pDkmConnection=0x20b628, pPath=<optimized out>,
    ppDkmProcess=0xffb1f06c) at /root/Concord/src/vsdbg/lib/vsdbg.cpp:253
#18 0xf77320bc in CVsDbg::LaunchProcess (this=0x20afe0, noDebug=<optimized out>, processName=...) at /root/Concord/src/vsdbg/lib/vsdbg.cpp:587
#19 0xf7772166 in VsCode::CVsCodeProtocol::HandleConfigurationDoneRequest (this=0x290d88, doc=..., errString=..., errCode=@0xffb1fac4: 0,
    telemetryAlreadyReported=<optimized out>) at /root/Concord/src/vsdbg/lib/vscode/VsCodeProtocol.cpp:773
#20 0xf776ed82 in VsCode::CVsCodeProtocol::HandleRequest (this=<optimized out>, szRequest=<optimized out>)
    at /root/Concord/src/vsdbg/lib/vscode/VsCodeProtocol.cpp:300
#21 0xf7763cd0 in std::function<int ()>::operator()() const (this=0xf7712bd0 <s_VSDebugEngImplClassMap>)
    at /crossrootfs/arm/usr/lib/gcc/arm-linux-gnueabihf/4.8/../../../../include/c++/4.8/functional:2471
#22 VsCode::CommandQueue::SyncCommand(std::function<int ()> const&)::$_0::operator()() const (this=<optimized out>)
    at /root/Concord/src/vsdbg/lib/vscode/CommandQueue.cpp:59
#23 std::_Function_handler<int (), VsCode::CommandQueue::SyncCommand(std::function<int ()> const&)::$_0>::_M_invoke(std::_Any_data const&) (__functor=...)
    at /crossrootfs/arm/usr/lib/gcc/arm-linux-gnueabihf/4.8/../../../../include/c++/4.8/functional:2056
#24 0xf7763aae in std::function<int ()>::operator()() const (this=<optimized out>)
    at /crossrootfs/arm/usr/lib/gcc/arm-linux-gnueabihf/4.8/../../../../include/c++/4.8/functional:2471
#25 VsCode::CommandQueue::CommandLoop (this=<optimized out>) at /root/Concord/src/vsdbg/lib/vscode/CommandQueue.cpp:33
#26 0xf776d6e8 in CVsCodeMainLoop::Run (this=0x25e398) at /root/Concord/src/vsdbg/lib/vscode/VsCodeMainLoop.cpp:39
#27 0xf7726584 in RunMainLoop () at /root/Concord/src/vsdbg/lib/EntryPoint.cpp:202
#28 0x00008f3c in main (argc=<optimized out>, argv=<optimized out>) at /root/Concord/src/vsdbg/exe/main.cpp:37
(gdb)

Thanks! I am not sure why this doesn't cause an alignment fault on our ARM devices, but I think I see the problem.

I did some more reading, and I think I might know what is going on. ARMv6 now has hardware support for unaligned memory access. It is an optional feature, but it sounds like most Linux kernel distributions enable it. I will see if I can find a way to disable unaligned fixups on one of our test devices.
In the mean time, it sounds like you can hopefully work around this using:
echo 1 > /proc/cpu/alignment
Useful references that I found:

https://www.kernel.org/doc/Documentation/arm/mem_alignment
https://stackoverflow.com/questions/25909691/arm-doesnt-generate-unaligned-exception

Hmmm, I don't seem to have /proc/cpu/alignment .  Seems like that may be only valid on 32-bit ARM, but I haven't found a definitive answer on that yet.... I'll definitely turn that on if I can find way.

Haven't had much chance to look for a way to prevent that signal yet.  However I do have an extra one of these boards you could borrow if it would help:
https://www.avnet.com/shop/us/products/avnet-engineering-services/aes-ultra96-g-3074457345634920668/

I took a look at your coredump, and the faulting instruction is a multi-register read instruction (ldmia.w r6, {r0, r1, r2, r3}). I am hoping that this is just a difference between ARM implementations in their unaligned fixups with these multi-register operations, and therefore this should be fairly rare (as it doesn't affect the normal memory reads). I will try and get you out a private today or Monday. If this proves to be the primary problem spot - great! If not, we can reassess at that point.

Sounds good.  I appreciate all the effort.

Okay, here is an updated libvsdebugeng.so that will hopefully fix this. Can give try replacing yours on your ARM device and let me know how it goes.

It's Alive!  ALIVE!!!!!! Nice work.

Excellent! I will put out an official release when I have the chance.

https://aka.ms/getvsdbgsh now points at a version of vsdbg with the fix. Thanks for reporting this!


Add supported backend information in Sig (signature for a expression)

Description
Sig is a mapping from spark expression to substrait name, used to convert spark functions to substrait and offload the funciton.
Currently, it includes below information:
case class Sig(expClass: Class[_], name: String)

IIUC all the registered functions are maintained in ExpressionMappings and serve for both velox and clickhouse backend. But some of them are supported by only one of the backend. For example: Expm1 and StringLocate are only supported by clickhouse.
In the mapping, it's hard to tell which function is supported by which backend. Adding supported backend information here might be helpful for the maintenance/track the status of the functions.

cc @PHILO-HE does this make sense? Please let me know if I missed anything. Thanks.

Hi @ivoson, thanks for your proposal!
Actually, a kind of validate-by-trying-offloading (covering Java/Scala code & native code) is introduced in Gluten to reduce code maintenance efforts. At least for Velox backend (CH backend may still have some gaps to support validate-by-trying-offloading), if validation process fails (e.g., function signature is not found in velox), we expect fallback happens.
Maybe, there are some cases that the expected fallback doesn't happen when validation fails. Do you have any findings that need us to fix? Thanks!

Maybe, there are some cases that the expected fallback doesn't happen when validation fails. Do you have any findings that need us to fix?

@PHILO-HE thanks for the response. I didn't find any cases need to be fixed. I created this issue because that I supposed the functions added in ExpressionMappings would be all supported by velox, but it was not. So it's not clear from the code to identify that how many of the expressions supported in velox/clickhouse. I am just wondering whether it's worth doing it. If not, we can rely on the docs up-to-date I think.

Maybe, there are some cases that the expected fallback doesn't happen when validation fails. Do you have any findings that need us to fix?

@PHILO-HE thanks for the response. I didn't find any cases need to be fixed. I created this issue because that I supposed the functions added in ExpressionMappings would be all supported by velox, but it was not. So it's not clear from the code to identify that how many of the expressions supported in velox/clickhouse. I am just wondering whether it's worth doing it. If not, we can rely on the docs up-to-date I think.

@ivoson, I think it is not worth doing that. cc @FelixYBW, @zhouyuan

In our plan, the validation should all happens in native library. native library should have all the info it supports the expression/operators or not. But Velox itself doesn't have this feature due to its low priority, we still put all the logic in Gluten c++ now. It's where the mismatch comes from. It should be tracked when we nightly rebase to Velox but currently we didn't. How the sig() function can solve the mismatch issue

In our plan, the validation should all happens in native library. native library should have all the info it supports the expression/operators or not. But Velox itself doesn't have this feature due to its low priority, we still put all the logic in Gluten c++ now. It's where the mismatch comes from. It should be tracked when we nightly rebase to Velox but currently we didn't. How the sig() function can solve the mismatch issue

Thanks @PHILO-HE @FelixYBW for your explanation. The proposal is not targeting for plan validation, it's more like a metadata to point out the expression is supported by which backend.
For example, adding the metadata in Sig like Sig(Add, 'add', Set(Velox, CH)) can tell that the expression is supported by both Velox and CH. There could be 2 scenarios we can leverage this:

When do plan validation, it can be eagerly detected if the function is not supported by the backend;
Track the function support status;

When do plan validation, it can be eagerly detected if the function is not supported by the backend;

We had this API https://github.com/apache/incubator-gluten/blob/182a0299993b38e04ee053158f71b2500c59eb09/gluten-core/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala#L556 to customize the supported expression list from backend implementations. Did you have a check on that code?

When do plan validation, it can be eagerly detected if the function is not supported by the backend;

We had this API
https://github.com/apache/incubator-gluten/blob/182a0299993b38e04ee053158f71b2500c59eb09/gluten-core/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala#L556
to customize the supported expression list from backend implementations. Did you have a check on that code?

Thanks @zhztheplayer. I noticed this, but I am not very clear about what's is the design purpose of this API.
In the very beginning, I supposed that the expressions defined in ExpressionMappings should be supported by both engines(velox and ch), and the ones supported by only one of the engines should be defined with in above API. But it's not the case in current code implementation.

@ivoson, some functions have multiple signatures. And it is possible that not all signatures are supported by native engine. If we add a fag in ExpressionMappings, we have to consider such partially supported state, which makes the code a bit complex. As I said, the native side validation can reduce the efforts for code maintenance, though it is not clear to developers to know which functions are not supported.

@ivoson, some functions have multiple signatures. And it is possible that not all signatures are supported by native engine. If we add a fag in ExpressionMappings, we have to consider such partially supported state, which makes the code a bit complex. As I said, the native side validation can reduce the efforts for code maintenance, though it is not clear to developers to know which functions are not supported.

Understood. As the potential changes could introduce maintenance burden, closing the issue.


Calculate goodness of fit (r2_score) based on posterior expectation rather than posterior prediction

Previously, the r2_score was calculated by comparing y_true and y_pred, where y_pred was the likelihood-influenced predictions i.e. with the observation noise added. Now we set y_pred to be based on the model's expected outcomes.
Closes #428

📚 Documentation preview 📚: https://causalpy--429.org.readthedocs.build/en/429/

Oh, I think this is because of rng being used multiple times (in the docstring there was a call to rng.normal before the model.score) which changed its state. Simple fix will be to rearrange order of calculations in the docstring. Though I'm not totally on top of why doctests were passing locally and not remotely.

Why are these f*king doctests passing locally but not remotely. Does rng have different behaviour on different systems or numpy versions?


feat(cdc): support transaction for shared cdc source

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
close https://github.com/risingwavelabs/risingwave/issues/14348

Add transaction metadata parsing logic to PlainParser for the CDC source job, which fixed to plain json format.
Add more solid tests for emitting transactional chunk
Fix transaction id parsing logic for specific database

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.
User can specify transactional = true parameter to the WITH options to preserve upstream transaction semantic  when create a CDC source.

Add transaction metadata parsing logic to PlainParser for the CDC source job, which fixed to plain json format.

I am a little worried that we are mixing too much logic into plain parser. shall we make a new format dedicated to CDC source?

Hmm, the modification to plain parser is small, only add a new branch to parse transaction metadata if it needs to.
https://github.com/risingwavelabs/risingwave/pull/14375/files#diff-8c7e5305d8e1796a4ccc76a9522ccb881ea31abeeb6d8010427679c5d9fa801fR91-R95

Add transaction metadata parsing logic to PlainParser for the CDC source job, which fixed to plain json format.

I am a little worried that we are mixing too much logic into plain parser. shall we make a new format dedicated to CDC source?

Hmm, the modification to plain parser is small, only add a new branch to parse transaction metadata if it needs to.
https://github.com/risingwavelabs/risingwave/pull/14375/files#diff-8c7e5305d8e1796a4ccc76a9522ccb881ea31abeeb6d8010427679c5d9fa801fR91-R95

Yes, I am ok with the current impl, just wonder if you have such plan.

Add transaction metadata parsing logic to PlainParser for the CDC source job, which fixed to plain json format.

I am a little worried that we are mixing too much logic into plain parser. shall we make a new format dedicated to CDC source?

Hmm, the modification to plain parser is small, only add a new branch to parse transaction metadata if it needs to.
https://github.com/risingwavelabs/risingwave/pull/14375/files#diff-8c7e5305d8e1796a4ccc76a9522ccb881ea31abeeb6d8010427679c5d9fa801fR91-R95

Yes, I am ok with the current impl, just wonder if you have such plan.

Right now the modification is small, so I think it is ok to just extend the plain parser. If there are more features need to add to the plain parser (e.g. handle schema change events), we can define a new parser and format for it at that time.

May I ask what's PlainParser for? Why aren't we using JsonParser here?

It is for parsing message for source in FORMAT PLAIN format, and actually we don't have JsonParser, json is an encoding not format in our terminology.
https://github.com/risingwavelabs/risingwave/blob/414c6ec1c05e3d13aa4842d30624c13b83b9692c/src/connector/src/parser/plain_parser.rs#L58-L63

User can specify transactional = true parameter to the WITH options to preserve upstream transaction semantic when create a CDC source.

Is it possible to enable this feature by default? Will this involve additional cost?

User can specify transactional = true parameter to the WITH options to preserve upstream transaction semantic when create a CDC source.

Is it possible to enable this feature by default? Will this involve additional cost?

The overhead should be small. But I think we should conduct a performance test to make this decision.  @lmatz Does our ch-benchmark use BEGIN and COMMIT on upstream database to generate workload? If it is we can create a transactional source and eval its performance.

User can specify transactional = true parameter to the WITH options to preserve upstream transaction semantic when create a CDC source.

Is it possible to enable this feature by default? Will this involve additional cost?

The overhead should be small. But I think we should conduct a performance test to make this decision. @lmatz Does our ch-benchmark/sysbench use BEGIN and COMMIT on upstream database to generate workload? If it is we can create a transactional source and eval its performance.

Any follow-ups? @StrikeW

User can specify transactional = true parameter to the WITH options to preserve upstream transaction semantic when create a CDC source.

Is it possible to enable this feature by default? Will this involve additional cost?

The overhead should be small. But I think we should conduct a performance test to make this decision. @lmatz Does our ch-benchmark/sysbench use BEGIN and COMMIT on upstream database to generate workload? If it is we can create a transactional source and eval its performance.

Any follow-ups? @StrikeW

@cyliu0 is working on the test, the ETA is next week. https://github.com/risingwavelabs/risingwave-test/issues/577

Update the evaluation result (nightly-20240125)
The overhead is about 5% for mysql and 4% for pg roughly and it is increased inside the debezium connector.

I think the overhead is not trivial, it is better to remain the transactional to false. @fuyufjh

I think the overhead is not trivial, it is better to remain the transactional to false by default. @fuyufjh

Thanks for the testing work!
I prefer to enable this by default so that we can claim the CDC connector is transactional. Otherwise, telling people "Yes-but" (Yes it supports transaction but needs to be enabled manually) sounds bad to me.

I got it. Let's do the change in v1.7.0.


add "optimise selectors" rule

Should we rephrase this into "Optimise selectors for performance"?  There's a lot of things you can optimise for, e.g. "flexibility".
Also I think we typically find component root elements ($('[data-my-component]')) and then .find() descendants within ($component.find('[data-my-component-item]) or bind (.on()) on those ($component.on('click', '[data-my-component-button], function() { /* ... */ })`). I'm not sure how we align that with this rule.

I've updated the examples to not suggest that using $component.find(...) is a bad thing. Maybe we should add a section explaining why we commonly use this approach?

Can we not make it more specific and say you should aim to only have a single selector?
Should we say something about using class or id selectors?
Regarding targeting selectors inside components: this guide is not about module based development, but you could explain that $('[data-table]').find('[data-table-amount]') is better than just doing $('[data-table-amount]');

@Ex-iT @jpsc @reinoute tackle this one in person tomorrow? I'm not convinced yet :)
And @reinoute regarding "What's our recommendation around using data() for reading data-attribute values?", move that to a separate issue?


Search Service V2V3 Should support gzip results

With this change to the client, we can now support properly gzip-ed results from the search service.
We should update the search service to minimize the payload for search results by both minifiying the json response and gzip-ing (and whatever else we can do to minimize the payload size without breaking the client).
The work is to build, test performance impact, test customer impact on gallery, V3.3 and V3.4 clients

CC @joyhui @maartenba @johnataylor

merhaba who started to made that in map box project, about istanbul to couple american city...

V2V3 search is now gzipped + minified.
V2V3 registrations blobs are now minified.
What's left here?
/cc @xavierdecoster

Test against 3.3 and 3.4, registration blobs are going to get further changes including gzipping

I believe John has tested these. If not, please reopen.


False resource leak because of freopen

I ran infer on the redis project and found a RESOURCE_LEAK. Here is a small part of the report:
   resource of type `__sFILE` acquired by call to `fopen()` at line 564, column 12 is not released after line 574, column 5
  574. >     lf.f = freopen(filename, "rb", lf.f);  /* reopen in binary mode */

It's saying that the lf.f you pass to the function is not freed. This is a false alarm, because freopen actually closes the stream you pass it (that's how I understand the man page). Currently, the model of freopen can nondeterministically return NULL. I think that a free in that branch should do the job.

Thanks for the report!
In the future, it always helps to provide standalone repro to make it easier for us to understand the issue.
Minimal repro in this case:
#include <stdio.h>

int test() {
  FILE* f = fopen("blah", "r");
  f = freopen("blah2","r", f); // FP here
  fclose(f);
}

Sorry that I did not do that. And thanks for creating it :-).
I am not completely sure about what the specification of freopen should be. I believe it will always close the stream, even when opening the new file fails. This is different from realloc.


[7.0] Rename flow AST Type NumericLiteralTypeAnnotation to NumberLiteralTypeAnnotation

Flow and babylon6 use a different node type for the following typeAnnotation:

var a : 1 = 1;

babylon uses NumericLiteralTypeAnnotation, whereas flow uses NumberLiteralTypeAnnotation
We should change babylon and rename the node to use the same name as flow.

We also need to do this change in babel itself. See https://github.com/babel/babel/pull/5199 as a reference.

Done in https://github.com/babel/babel/issues/5229


Margaret Mead Bug Fix

Bug: Margaret Wouldn't work on Observatory/Seowon as intended
Fixed by FlashyFeeds, suggested by ZhenjaMax.

This is not a bug fix since it change the way the great people.
This will be considered as a proposal, and not a bugFix.


Release 0.2.0

@Vici37 Anything else you'd like to include in 0.2.0?

Nope, enough has changed that a 0.2.0 seems like a good version cut :+1:


authors: stub message for the old module

Signed-off-by: Mateusz Susik<EMAIL_ADDRESS>

when you fix my comments you can add Reviewed-by: me.

Fixed, tested, ready.

Fixed, tested, ready.


Extend page view template

I need help and try to simplify the problem. I'm doing a small version of cms to manage some sections of my frontend part application.
The pages are saved in the db and through a unique alias are recalled in the following way
$app->get('/:alias', ['\App\Controllers\Pages', 'getPage']);
and the controller is
class Pages {
	public static function getPage(App $app, $alias = null) {
        echo $app->view('/frontend/page.html.twig', [
	....
	]);
}

page.html.twig has the classic structure
<html>
<head></head>
<body>
{% block content %}{% endblock %}
</body>
</html>

Ok everything work good. Now start my problem
Each page has contents that are connected with the page by id
MYSQL
TABLE PAGES :: ID -> 1 - ALIAS -> alias
TABLE CONTENTS :: ID -> 1 - PAGE -> 1 - TEMPLATE -> content.html.twig
The result I would like to achieve is to dynamically include contents template from the controller in page view (page.html.twig)
My Idea is to use a select in controller on Table Contents by id and pass the template as extend
for example
content.html.twig is
{% extends '/frontend/page.html.twig' %}

{% block content %}
	{{ parent() }}
	.. my code ..
{% endblock %}

But this is wrong because the view template should be the content.html.twig and not page.html.twig
Hope to receive a valid and elegant solution :)
Thanks

Change the name of page.html.twig to layout.html.twig create a new file with page.html.twig with this code:
{% extends  'frontend/layout.html.twig' %}

Sorry for the long delay!
Did you solve it by extending from the new layout.html.twig?
Otherwise, wouldn’t something like this have worked?
public static function getPage(App $app, $alias = null) {
    $template = $app->db()->selectValue(
        "SELECT template FROM pages AS p JOIN contents AS c ON c.page = p.id WHERE p.alias = ?",
        [ $alias ]
    );

    if (!empty($template)) {
        echo $app->view($template);
    }
    else {
        echo $app->view('/frontend/page.html.twig');
    }
}

By the way, if you let the user choose paths at the root of your site, you always have to be careful, of course. You don’t want any user to choose robots.txt, for example. But if only the administrator has access, that might not even be a problem.
Finally, for untrusted content in templates, you may be interested in Twig’s sandbox:

https://twig.symfony.com/doc/1.x/tags/sandbox.html
https://twig.symfony.com/doc/1.x/api.html#sandbox-extension

Even if in delay your answer are always very welcome. :) Anyway select the template for each alias would be a solution but my structure is different layout, module, contents,  articles, elements are created from be and stored in db so I think is better to render the view to a dynamic page.
I don't understand what do you mean can you explain with an example ?

By the way, if you let the user choose paths at the root of your site, you always have to be careful, of course. You don’t want any user to choose robots.txt

Working with your framework I start to learn TWIG so not so familiar yet. Sandbox is for include templates and restrict the value available in that template ? I am using macro with custom filter for pass only the value available for that template.
Thanks.

I don't understand what do you mean can you explain with an example ?

So you’re building a CMS, right? If you let your users choose paths at the root of your domain for their content, they could choose “/robots.txt” for a post or something like “/contact” that you’d like to use for other things, etc. So you may not want to allow for that. You could either prevent this by having a common prefix for all user-generated content, e.g. “/page/...”, or by creating a blacklist of paths or names that cannot be used.

Sandbox is for include templates and restrict the value available in that template ?

You can read about Twig’s sandbox feature in their official documentation. It’s basically so that you can allow untrusted users to write pages in Twig. For example, when building a CMS, you may want to allow for any employee of a company, e.g. their web designer or an editor, to modify parts of their site that are written in Twig. In your example, you said you wanted to allow for dynamic templates stored in the database. If those templates can contain Twig code, you may want to improve security by not allowing every single of Twig to be used in there.