MySQL source parsing NULL value error

import polars as pl
import pandas as pd
from sqlalchemy import create_engine
import pyarrow
#
print(pl.__version__)
# 0.8.20
print(pd.__version__)
# 1.3.0
pip list | grep connector*
# connectorx                    0.2.0
pyarrow.__version__
# '4.0.1'

# pandas first
sql = '''select ORDER_ID from tables     '''
engine = create_engine('mysql+pymysql://root:***@*.*.*.*:*')
df = pd.read_sql_query(sql, engine)
df.dtypes
# ORDER_ID                      int64

# polars second
conn = "mysql://root:***@*.*.*.*:*"
pdf = pl.read_sql(sql, conn)
`
---------------------------------------------------------------------------
PanicException                            Traceback (most recent call last)
<timed exec> in <module>

~/miniconda3/envs/test/lib/python3.8/site-packages/polars/io.py in read_sql(sql, connection_uri, partition_on, partition_range, partition_num)
    556     """
    557     if _WITH_CX:
--> 558         tbl = cx.read_sql(
    559             conn=connection_uri,
    560             query=sql,

~/miniconda3/envs/test/lib/python3.8/site-packages/connectorx/__init__.py in read_sql(conn, query, return_type, protocol, partition_on, partition_range, partition_num)
    126             raise ValueError("You need to install pyarrow first")
    127
--> 128         result = _read_sql(
    129             conn,
    130             "arrow",

PanicException: Could not retrieve i64 from Value

`

Hi @ztsweet, in order to narrow down the problem, can you let us know the database source and dataframe destination that you are using? And also what data types are included in your query result?

@wangxiaoying I have edited the question again

@ztsweet Thanks for the details. Can you also check what is the data type of ORDER_ID in MySQL?

@wangxiaoying in mysql the type is ``ORDER_ID bigint(19) DEFAULT NULL

@ztsweet It seems like we have a but when parsing NULL values in MySQL source. Will fix this issue it in our next release (0.2.1).

@wangxiaoying thanks

Hi @ztsweet, we just released the connectorx-0.2.1a1 which has fixed the NULL value bug in mysql. You can try it by using pip install -U connectorx==0.2.1a1.
If all tests go well, we plan to release the 0.2.1 version in early October. Please feel free to have a try and let us know if you found any issue.

Are you using Python or Rust?
Python.
What version of polars are you using?
polars==0.10.27
connectorx==0.2.2
What operating system are you using polars on?
CentOS Linux release 7.9.2009 (Core)
Describe your bug.
read_sql returns following exceptions if any column contains missing values, i.e. no value.
Note: No error occurs if all rows have values
Errors
import polars as pl
pl.read_sql("select column1 from Table", <url_to_database>)

Replicated the issue using cx.read_sql. Same errors
If column1 completely empty
RuntimeError: Invalid argument error: column types must match schema types, expected Time64(Nanosecond) but found LargeBinary at column index 0
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-67-2867bd04669a> in <module>
----> 1 df_pl = pl.read_sql("""
      2 select Replaced_Query_Bill
      3 from Event_Master limit 10""", cart)#.to_pandas()
      4 # Follows_Waterfall, Condition, Event_Limit, Slug, Conversation, Campaign_Key
      5 # Arms, Is_MAB_Driven, Is_Good_Time, Is_On_Demand

~/ipynbs/.venv/lib/python3.9/site-packages/polars/io.py in read_sql(sql, connection_uri, partition_on, partition_range, partition_num)
    802     """
    803     if _WITH_CX:
--> 804         tbl = cx.read_sql(
    805             conn=connection_uri,
    806             query=sql,

~/ipynbs/.venv/lib/python3.9/site-packages/connectorx/__init__.py in read_sql(conn, query, return_type, protocol, partition_on, partition_range, partition_num, index_col)
    133             raise ValueError("You need to install pyarrow first")
    134
--> 135         result = _read_sql(
    136             conn,
    137             "arrow",

RuntimeError: Invalid argument error: column types must match schema types, expected Time64(Nanosecond) but found LargeBinary at column index 0

If column1 has datetime values but if even one row has no value
PanicException: Could not retrieve chrono::naive::datetime::NaiveDateTime from Value
---------------------------------------------------------------------------
PanicException                            Traceback (most recent call last)
<ipython-input-101-17df85ddf35d> in <module>
----> 1 cart.read_sql('select * from Event_Master limit 10')

<ipython-input-97-7fb8f6c351ab> in read_sql(self, query)
     25     def read_sql(self, query):
     26         try:
---> 27             result = pl.read_sql(query, self.__url)
     28         except Exception as e:
     29             print('Polar failed trying pandas')

~/ipynbs/.venv/lib/python3.9/site-packages/polars/io.py in read_sql(sql, connection_uri, partition_on, partition_range, partition_num)
    802     """
    803     if _WITH_CX:
--> 804         tbl = cx.read_sql(
    805             conn=connection_uri,
    806             query=sql,

~/ipynbs/.venv/lib/python3.9/site-packages/connectorx/__init__.py in read_sql(conn, query, return_type, protocol, partition_on, partition_range, partition_num, index_col)
    133             raise ValueError("You need to install pyarrow first")
    134
--> 135         result = _read_sql(
    136             conn,
    137             "arrow",

PanicException: Could not retrieve chrono::naive::datetime::NaiveDateTime from Value

What is the expected behavior?
Read empty values in dataframe just like pandas

Hi @hamzamohdzubair  thanks for reporting the issue. It looks like the null data does not match with the type we fetched. May I ask which database you are using? Also what is the type for column1 defined in your database?

@wangxiaoying updated my previous comment
The first error is TYPE = MEDIUMTEXT
second error is TYPE = DATETIME

@wangxiaoying Also, why am i unable to catch the PanicException in a try except clause. I am trying to do this:
read a dataframe using polars, if it fails read it using pandas, but i am unable to catch the PanicException

@hamzamohdzubair , thanks for the info. It is weird since we have the mysql test case when DATETIME is null. I also test on MEDIUMTEXT and it also passes. We use the official docker image of mysql in our environment. I will try to set up and test for mariadb to see whether I can reproduce the error.
As for the PanicException, I think the panic happens in the underlying mysql driver. I will also take a look at it if I can reproduce this error.

Hi @hamzamohdzubair, thanks for waiting.
For the first error on MEDIUMTEXT, turns out it is a bug in the type mapping in arrow. It has been fixed it here: https://github.com/sfu-db/connector-x/commit/46bf99afe632ad151ea3cf6e599b2b0336b2cb8f. We will release an alpha version to this bug fix. Will let you know once it's done.
For the second error onDATETIME, I cannot reproduce the error using mysql / mariadb docker image when output pandas / arrow.  Is it possible if you could provide an example SQL script for constructing the table (create and insert values to the table) that can cause the error?


Allow User-Configurable Port for MSP

Currently, the Mutation Server Protocol (MSP) reports the selected port via STDOUT, but there’s no mention in the protocol specification about using a user-configurable port. This works fine in local development but becomes challenging in containerized environments.
For example, in the Stryker Cloud Orchestrator setup, which distributes the mutation testing process across multiple workers, the tool implementing MSP will run in a separate container. Which makes it hard to retrieve the port from STDOUT.
In my use case, I want to control the mutation testing tool via MSP from a separate worker container. Without a way to set a fixed port, it’s challenging to establish reliable communication between the worker and the MSP server since the port may change dynamically.
Proposed Update
I suggest extending the base protocol documentation with the following note:

The mutation server may use a user-configurable port specified through a command-line option or configuration setting.

This small addition to the documentation would clarify that implementations can support a user-defined port, making it clear for users and tool developers that this option is available when needed.

@nicojs @ any objections?

I agree with the proposed addition to the documentation, good idea! Allowing a user-configurable port would provide the flexibility needed for containerized environments.


Raspberry Pi: proj executable not found. Please set the PROJ_DIR variable

I'm trying to use pyproj on my Raspberry Pi, but I can't get it to install properly using pip.
Installation method/steps

Installation method: pip 21
How did you install PROJ? Where is it installed? It isn't. From what I read PROJ isn't required with pip >= 19.
Please provide all commands/steps you used to install pyproj and PROJ.
python3 -m pip install --no-cache-dir --upgrade pyproj==3.1.0

Environment Information

pyproj version you are attempting to install: 3.1.0
PROJ version (Execute proj command and give version here.) N/A (see above)
Python version 2.7.16 (default, Oct 10 2019, 22:02:15)  [GCC 8.3.0]
Operation System Information Linux-5.10.17+-armv6l-with-debian-10.10

I get the following error:
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pyproj==3.1.0
  Downloading pyproj-3.1.0.tar.gz (182 kB)
     |████████████████████████████████| 182 kB 793 kB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /home/pi/.local/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpfoar8m2w
       cwd: /tmp/pip-install-uevnkrfy/pyproj_6f80f7ea3dbe4490a839aed785c21d4d
  Complete output (1 lines):
  proj executable not found. Please set the PROJ_DIR variable. For more information see: https://pyproj4.github.io/pyproj/stable/installation.html
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/7c/1d/20ea3b603db61ccc60f45064a9e00ba2e6263f1de560e33306f6f3d42fcb/pyproj-3.1.0.tar.gz#sha256=67b94f4e694ae33fc90dfb7da0e6b5ed5f671dd0acc2f6cf46e9c39d56e16e1a (from https://pypi.org/simple/pyproj/) (requires-python:>=3.7). Command errored out with exit status 1: /usr/bin/python3 /home/pi/.local/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpfoar8m2w Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pyproj==3.1.0 (from versions: 1.8.0, 1.8.1a0, 1.8.1, 1.9.3, 1.9.4, 1.9.5, <IP_ADDRESS>, 1.9.6, 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.4.2, 2.4.2.post1, 2.5.0, 2.6.0, 2.6.1, 2.6.1.post1, 3.0rc2, 3.0.0, 3.0.0.post1, 3.0.1, 3.1.0)
ERROR: No matching distribution found for pyproj==3.1.0

Building PROJ myself isn't an option on the Pi, because it just doesn't have enough memory.

See: https://github.com/piwheels/packages/issues/60. I recommend following up there.

Closing for the upstream issue.


HDFS-16464. Create only libhdfspp static libraries for Windows

Description of PR
While building dynamic libraries (.dll) on Windows, there's a constraint that all the dependent libraries be linked dynamically. This poses an issue since Protobuf (which is an HDFS native client dependency) runs into build issues when linked dynamically. There are a few warning notes on the Protobuf repository's build instructions page as well.
Thus, to keep things simple, we can resort to do only static linking and thereby only produce statically linked libraries on Windows. In summary, we'll be providing only Hadoop .lib files initially. We can aim to produce Hadoop DLL on Windows eventually once we're able to resolve Protobuf's DLL issues.
How was this patch tested?
Hadoop Jenkins CI validation ensures that the Linux components aren't affected. Verified that the static library is produced as a result of successful build on my local Windows system.
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
37m 55s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
38m 49s

trunk passed

+1 :green_heart:
compile
3m 56s

trunk passed

+1 :green_heart:
mvnsite
0m 49s

trunk passed

+1 :green_heart:
shadedclient
62m 41s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 24s

the patch passed

-1 :x:
compile
0m 49s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
hadoop-hdfs-native-client in the patch failed.

-1 :x:
cc
0m 49s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
hadoop-hdfs-native-client in the patch failed.

-1 :x:
golang
0m 49s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
hadoop-hdfs-native-client in the patch failed.

-1 :x:
javac
0m 49s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
hadoop-hdfs-native-client in the patch failed.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 27s

the patch passed

+1 :green_heart:
shadedclient
18m 54s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
0m 59s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
hadoop-hdfs-native-client in the patch failed.

+1 :green_heart:
asflicense
0m 54s

The patch does not generate ASF License warnings.

125m 33s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4571

Optional Tests
dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang

uname
Linux a922b0575d14 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / a6c1a45f303502d7d8b178aa7f342e25f50f7dc6

Default Java
Red Hat, Inc.-1.8.0_332-b09

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/1/testReport/

Max. process+thread count
717 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/1/console

versions
git=2.9.5 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 44s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
40m  3s

trunk passed

+1 :green_heart:
compile
4m 11s

trunk passed

+1 :green_heart:
mvnsite
0m 45s

trunk passed

+1 :green_heart:
shadedclient
65m 33s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 22s

the patch passed

+1 :green_heart:
compile
3m 31s

the patch passed

+1 :green_heart:
cc
3m 31s

the patch passed

+1 :green_heart:
golang
3m 31s

the patch passed

+1 :green_heart:
javac
3m 31s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 27s

the patch passed

+1 :green_heart:
shadedclient
19m  6s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
36m 19s

hadoop-hdfs-native-client in the patch passed.

+1 :green_heart:
asflicense
0m 53s

The patch does not generate ASF License warnings.

129m 21s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4571

Optional Tests
dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang

uname
Linux e4ee9e2b0bc6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 9dc5cd0d060ed87bdbc9b20e7bf6d4a90578bf68

Default Java
Red Hat, Inc.-1.8.0_332-b09

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/

Max. process+thread count
704 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console

versions
git=2.9.5 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
21m 27s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
22m 36s

trunk passed

+1 :green_heart:
compile
4m  6s

trunk passed

+1 :green_heart:
mvnsite
1m  4s

trunk passed

+1 :green_heart:
shadedclient
47m 33s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 31s

the patch passed

+1 :green_heart:
compile
3m 43s

the patch passed

+1 :green_heart:
cc
3m 43s

the patch passed

+1 :green_heart:
golang
3m 43s

the patch passed

+1 :green_heart:
javac
3m 43s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 34s

the patch passed

+1 :green_heart:
shadedclient
19m 10s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
33m  5s

hadoop-hdfs-native-client in the patch passed.

+1 :green_heart:
asflicense
1m  2s

The patch does not generate ASF License warnings.

129m 44s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4571

Optional Tests
dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang

uname
Linux 5dd21a11e1e5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 9dc5cd0d060ed87bdbc9b20e7bf6d4a90578bf68

Default Java
Red Hat, Inc.-1.8.0_312-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/

Max. process+thread count
680 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console

versions
git=2.27.0 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
21m  2s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
24m 53s

trunk passed

+1 :green_heart:
compile
3m 34s

trunk passed

+1 :green_heart:
mvnsite
0m 44s

trunk passed

+1 :green_heart:
shadedclient
56m 15s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 25s

the patch passed

+1 :green_heart:
compile
3m 17s

the patch passed

+1 :green_heart:
cc
3m 17s

the patch passed

+1 :green_heart:
golang
3m 17s

the patch passed

+1 :green_heart:
javac
3m 17s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 27s

the patch passed

+1 :green_heart:
shadedclient
26m 18s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
32m 22s

hadoop-hdfs-native-client in the patch passed.

+1 :green_heart:
asflicense
0m 47s

The patch does not generate ASF License warnings.

143m 19s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4571

Optional Tests
dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang

uname
Linux 1237e12f1dd5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 9dc5cd0d060ed87bdbc9b20e7bf6d4a90578bf68

Default Java
Debian-11.0.15+10-post-Debian-1deb10u1

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/

modules
C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console

versions
git=2.20.1 maven=3.6.0

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 44s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
21m 34s

trunk passed

+1 :green_heart:
compile
3m 58s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
4m  4s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
mvnsite
0m 44s

trunk passed

+1 :green_heart:
shadedclient
50m 32s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 24s

the patch passed

+1 :green_heart:
compile
3m 43s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
cc
3m 43s

the patch passed

+1 :green_heart:
golang
3m 43s

the patch passed

+1 :green_heart:
javac
3m 43s

the patch passed

+1 :green_heart:
compile
3m 43s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
cc
3m 43s

the patch passed

+1 :green_heart:
golang
3m 43s

the patch passed

+1 :green_heart:
javac
3m 43s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 28s

the patch passed

+1 :green_heart:
shadedclient
19m 51s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
33m  9s

hadoop-hdfs-native-client in the patch passed.

+1 :green_heart:
asflicense
0m 47s

The patch does not generate ASF License warnings.

115m 43s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4571

Optional Tests
dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang

uname
Linux 06912fdbae28 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 9dc5cd0d060ed87bdbc9b20e7bf6d4a90578bf68

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/

Max. process+thread count
683 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


Highlight redirections

Allow to define a style for redirection tokens. The doc is here.

Merged in 4ec821b.


{Mysql} az mysql flexible-server import create: Change sku to default for external migration

Related command

az mysql flexible-server create
Description
The defaults for SKU and tier is changed back to Burstable and Standard_B1ms.

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

Mysql


[VL] Track all the failed unit test in Spark 3.4.

Description
Our plan is to ensure that Spark 3.4 passes all unit tests. This issue will track all the failed unit tests in Spark 3.4, and we will work on fixing them one by one.
GlutenSQLQueryTestSuite

[ ] - describe-query.sql *** FAILED ***
[ ] - sql-compatibility-functions.sql *** FAILED ***
[ ] - cast.sql *** FAILED ***
[ ] - show-tables.sql *** FAILED ***
[ ] - intersect-all.sql *** FAILED ***
[ ] - cte-nested.sql *** FAILED ***
[ ] - random.sql *** FAILED ***
[ ] - datetime-formatting-invalid.sql *** FAILED ***
[ ] - group-by.sql *** FAILED ***
[ ] - show-create-table.sql *** FAILED ***
[ ] - cte-nonlegacy.sql *** FAILED ***
[ ] - inline-table.sql *** FAILED ***
[ ] - describe-table-column.sql *** FAILED ***
[ ] - map.sql *** FAILED ***
[ ] - describe.sql *** FAILED ***
[ ] - datetime-parsing-invalid.sql *** FAILED ***
[ ] - show_columns.sql *** FAILED ***
[ ] - comments.sql *** FAILED ***
[ ] - udaf.sql *** FAILED ***
[ ] - json-functions.sql *** FAILED ***
[ ] - describe-table-after-alter-table.sql *** FAILED ***
[ ] - show-tblproperties.sql *** FAILED ***
[ ] - grouping_set.sql *** FAILED ***
[ ] - charvarchar.sql *** FAILED ***
[ ] - table-valued-functions.sql *** FAILED ***
[ ] - regexp-functions.sql *** FAILED ***
[ ] - ilike-any.sql *** FAILED ***
[ ] - change-column.sql *** FAILED ***
[ ] - string-functions.sql *** FAILED ***
[ ] - columnresolution-negative.sql *** FAILED ***
[ ] - group-by-ordinal.sql *** FAILED ***
[ ] - timestamp-ntz.sql *** FAILED ***
[ ] - interval.sql *** FAILED ***
[ ] - date.sql *** FAILED ***
[ ] - bitwise.sql *** FAILED ***
[ ] - natural-join.sql *** FAILED ***
[ ] - timezone.sql *** FAILED ***
[ ] - ilike-all.sql *** FAILED ***
[ ] - datetime-legacy.sql *** FAILED ***
[ ] - csv-functions.sql *** FAILED ***
[ ] - pivot.sql *** FAILED ***
[ ] - group-by-filter.sql *** FAILED ***
[ ] - except-all.sql *** FAILED ***
[ ] - join-lateral.sql *** FAILED ***
[ ] - like-any.sql *** FAILED ***
[ ] - table-aliases.sql *** FAILED ***
[ ] - transform.sql *** FAILED ***
[ ] - query_regex_column.sql *** FAILED ***
[ ] - count.sql *** FAILED ***
[ ] - show-views.sql *** FAILED ***
[ ] - timestamp.sql *** FAILED ***
[ ] - try_arithmetic.sql *** FAILED ***
[ ] - extract.sql *** FAILED ***
[ ] - cte.sql *** FAILED ***
[ ] - cte-legacy.sql *** FAILED ***
[ ] - order-by-ordinal.sql *** FAILED ***
[ ] - literals.sql *** FAILED ***
[ ] - limit.sql *** FAILED ***
[ ] - like-all.sql *** FAILED ***
[ ] - tablesample-negative.sql *** FAILED ***
[ ] - ansi/cast.sql *** FAILED ***
[ ] - ansi/decimalArithmeticOperations.sql *** FAILED ***
[ ] - ansi/map.sql *** FAILED ***
[ ] - ansi/datetime-parsing-invalid.sql *** FAILED ***
[ ] - ansi/string-functions.sql *** FAILED ***
[ ] - ansi/interval.sql *** FAILED ***
[ ] - ansi/date.sql *** FAILED ***
[ ] - ansi/timestamp.sql *** FAILED ***
[ ] - ansi/try_arithmetic.sql *** FAILED ***
[ ] - ansi/literals.sql *** FAILED ***
[ ] - timestampNTZ/timestamp-ansi.sql *** FAILED ***
[ ] - timestampNTZ/timestamp.sql *** FAILED ***
[ ] - udf/udf-intersect-all.sql - Scala UDF *** FAILED ***
[ ] - udf/udf-except-all.sql - Scala UDF *** FAILED ***
[ ] - udf/udf-udaf.sql - Scala UDF *** FAILED ***
[ ] - udf/udf-group-by.sql - Scala UDF *** FAILED ***
[ ] - udf/udf-except.sql - Scala UDF *** FAILED ***
[ ] - udf/udf-pivot.sql - Scala UDF *** FAILED ***
[ ] - udf/udf-inline-table.sql - Scala UDF *** FAILED ***
[ ] - udf/postgreSQL/udf-select_having.sql - Scala UDF *** FAILED ***
[ ] - typeCoercion/native/windowFrameCoercion.sql *** FAILED ***
[ ] - typeCoercion/native/decimalPrecision.sql *** FAILED ***
[ ] - typeCoercion/native/ifCoercion.sql *** FAILED ***
[ ] - typeCoercion/native/dateTimeOperations.sql *** FAILED ***
[ ] - typeCoercion/native/booleanEquality.sql *** FAILED ***
[ ] - typeCoercion/native/mapZipWith.sql *** FAILED ***
[ ] - typeCoercion/native/caseWhenCoercion.sql *** FAILED ***
[ ] - typeCoercion/native/widenSetOperationTypes.sql *** FAILED ***
[ ] - typeCoercion/native/promoteStrings.sql *** FAILED ***
[ ] - typeCoercion/native/stringCastAndExpressions.sql *** FAILED ***
[ ] - typeCoercion/native/inConversion.sql *** FAILED ***
[ ] - typeCoercion/native/division.sql *** FAILED ***
[ ] - typeCoercion/native/mapconcat.sql *** FAILED ***
[ ] - subquery/in-subquery/in-basic.sql *** FAILED ***
[ ] - subquery/negative-cases/subq-input-typecheck.sql *** FAILED ***
[ ] - subquery/negative-cases/invalid-correlation.sql *** FAILED ***
[ ] - postgreSQL/create_view.sql *** FAILED ***
[ ] - postgreSQL/boolean.sql *** FAILED ***
[ ] - postgreSQL/aggregates_part1.sql *** FAILED ***
[ ] - postgreSQL/window_part3.sql *** FAILED ***
[ ] - postgreSQL/strings.sql *** FAILED ***
[ ] - postgreSQL/window_part4.sql *** FAILED ***
[ ] - postgreSQL/int8.sql *** FAILED ***
[ ] - postgreSQL/union.sql *** FAILED ***
[ ] - postgreSQL/aggregates_part4.sql *** FAILED ***
[ ] - postgreSQL/int4.sql *** FAILED ***
[ ] - postgreSQL/numeric.sql *** FAILED ***
[ ] - postgreSQL/window_part2.sql *** FAILED ***
[ ] - postgreSQL/with.sql *** FAILED ***
[ ] - postgreSQL/interval.sql *** FAILED ***
[ ] - postgreSQL/date.sql *** FAILED ***
[ ] - postgreSQL/join.sql *** FAILED ***
[ ] - postgreSQL/select_having.sql *** FAILED ***
[ ] - postgreSQL/text.sql *** FAILED ***
[ ] - postgreSQL/limit.sql *** FAILED ***
[ ] - postgreSQL/float4.sql *** FAILED ***
[ ] - postgreSQL/select_implicit.sql *** FAILED ***
[ ] - postgreSQL/aggregates_part3.sql *** FAILED ***

GlutenBroadcastJoinSuite will hang.
VeloxDataTypeValidationSuite

[ ]  Velox Parquet Write

VeloxParquetWriteForHiveSuite

[ ] test hive static partition write table
[ ] test hive write table

VeloxParquetWriteSuite

[ ] test write parquet with compression codec
[ ] test ctas
[ ] test parquet dynamic partition write

TPCDSPlan

[ ] check simplified (tpcds-v1.4/$q)

GlutenParquetV1FilterSuite

[ ] filter pushdown - StringPredicate

GlutenParquetV2FilterSuite

[ ] filter pushdown - StringPredicate
[ ] Gluten - filter pushdown - date

GlutenDynamicPartitionPruningV1SuiteAEOff

[ ] - SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec
[ ] - static scan metrics
[ ] - Gluten - static scan metrics

GlutenDynamicPartitionPruningV1SuiteAEOn

[ ] SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan

[ ] SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan

[ ] SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV2SuiteAEOff

[ ] SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV2SuiteAEOn

[ ] SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV2SuiteAEOnDisableScan

[ ] SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan

[ ] SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenCoalesceShufflePartitionsSuite

[ ] - Gluten - SPARK-24705 adaptive query execution works correctly when exchange reuse enabled

GlutenAdaptiveQueryExecSuite

[ ] - SPARK-30291: AQE should catch the exceptions when doing materialize
[ ] - SPARK-33933: Materialize BroadcastQueryStage first in AQE
[ ] - gluten Logging plan changes for AQE
[ ] - gluten test log level

GlutenEnsureRequirementsSuite

[ ] - SPARK-41986: Introduce shuffle on SinglePartition

GlutenKeyGroupedPartitioningSuite

[ ] - SPARK-41413: partitioned join: partition values from one side are subset of those from the other side
[ ] - SPARK-41413: partitioned join: partition values from both sides overlaps
[ ] - SPARK-41413: partitioned join: non-overlapping partition values from both sides
[ ] - SPARK-41413: partitioned join: partition values from one side are subset of those from the other side
[ ] - SPARK-41413: partitioned join: partition values from both sides overlaps
[ ] - SPARK-41413: partitioned join: non-overlapping partition values from both sides
[ ] - SPARK-42038: partially clustered: with different partition keys and both sides partially clustered
[ ] - SPARK-42038: partially clustered: with different partition keys and missing keys on left-hand side
[ ] - SPARK-42038: partially clustered: with different partition keys and missing keys on right-hand side
[ ] - SPARK-42038: partially clustered: left outer join
[ ] - SPARK-42038: partially clustered: right outer join
[ ] - SPARK-42038: partially clustered: full outer join is not applicable
[ ] - SPARK-42038: partially clustered: with dynamic partition filtering

GlutenParquetSchemaSuite

[ ] - SPARK-40819: parquet file with TIMESTAMP(NANOS, true) (with nanosAsLong=true)
[ ] - SPARK-40819: parquet file with TIMESTAMP(NANOS, true) (with default nanosAsLong=false)

GlutenWriteDistributionAndOrderingSuite

[ ] - ordered distribution and sort with same exprs: append
[ ] - ordered distribution and sort with same exprs: overwrite
[ ] - ordered distribution and sort with same exprs: overwriteDynamic
[ ] - clustered distribution and sort with same exprs: append *** FAILED ***
[ ] - clustered distribution and sort with same exprs: overwrite *** FAILED ***
[ ] - clustered distribution and sort with same exprs: overwriteDynamic *** FAILED ***
[ ] - clustered distribution and sort with extended exprs: append *** FAILED ***
[ ] - clustered distribution and sort with extended exprs: overwrite *** FAILED ***
[ ] - clustered distribution and sort with extended exprs: overwriteDynamic *** FAILED ***
[ ] - ordered distribution and sort with manual global sort: append *** FAILED ***
[ ] - ordered distribution and sort with manual global sort: overwrite *** FAILED ***
[ ] - ordered distribution and sort with manual global sort: overwriteDynamic *** FAILED ***
[ ] - ordered distribution and sort with incompatible global sort: append *** FAILED ***
[ ] - ordered distribution and sort with incompatible global sort: overwrite *** FAILED ***
[ ] - ordered distribution and sort with incompatible global sort: overwriteDynamic *** FAILED ***
[ ] - ordered distribution and sort with manual local sort: append *** FAILED ***
[ ] - ordered distribution and sort with manual local sort: overwrite *** FAILED ***
[ ] - ordered distribution and sort with manual local sort: overwriteDynamic *** FAILED ***
[ ] - clustered distribution and local sort with manual global sort: append *** FAILED ***
[ ] - clustered distribution and local sort with manual global sort: overwrite *** FAILED ***
[ ] - clustered distribution and local sort with manual global sort: overwriteDynamic *** FAILED ***
[ ] - clustered distribution and local sort with manual local sort: append *** FAILED ***
[ ] - clustered distribution and local sort with manual local sort: overwrite *** FAILED ***
[ ] - clustered distribution and local sort with manual local sort: overwriteDynamic *** FAILED ***

GlutenInsertSuite

[ ] - INSERT rows, ALTER TABLE ADD COLUMNS with DEFAULTs, then SELECT them
[ ] - SPARK-39557 INSERT INTO statements with tables with array defaults
[ ] - SPARK-39557 INSERT INTO statements with tables with struct defaults
[ ] - SPARK-39557 INSERT INTO statements with tables with map defaults
[ ] - SPARK-39844 Restrict adding DEFAULT columns for existing tables to certain sources

GlutenJoinSuite

[ ] - SPARK-43113: Full outer join with duplicate stream-side references in condition (SMJ)

GlutenFallbackSuite

[ ] - test fallback logging
[ ] - test fallback event

GlutenJsonV2Suite

[ ] - SPARK-39731: Correctly parse dates and timestamps with yyyyMMdd pattern

GlutenParquetIOSuite

[ ] - Read TimestampNTZ and TimestampLTZ for various logical TIMESTAMP types
[ ] - SPARK-40128 read DELTA_LENGTH_BYTE_ARRAY encoded strings
[ ] - SPARK-41096: FIXED_LEN_BYTE_ARRAY support

GlutenCTEHintSuite

[ ] - Resolve join hint in CTE

GlutenExpressionsSchemaSuite

[ ] - Check schemas for expression examples

GlutenDataSourceV2Suite

[ ] - ordering and partitioning reporting

GlutenSQLQuerySuite

[ ] - the escape character is not allowed to end with
[ ] - SPARK-40245: Fix FileScan canonicalization when partition or data filter columns are not read
[ ]  SPARK-38548: try_sum should return null if overflow happens before merging

GlutenQueryExecutionErrorsSuite

[ ] - FAILED_EXECUTE_UDF: execute user defined function
[ ] - UNRECOGNIZED_SQL_TYPE: unrecognized SQL type -100
[ ] - INVALID_BUCKET_FILE: error if there exists any malformed bucket files
[ ] - SCALAR_SUBQUERY_TOO_MANY_ROWS: More than one row returned by a subquery used as an expression
[ ] - UNSUPPORTED_FEATURE.MULTI_ACTION_ALTER: The target JDBC server hosting table does not support ALTER TABLE with multiple actions.

GlutenDataFrameSuite

[ ] - SPARK-41048: Improve output partitioning and ordering with AQE cache

GlutenFileMetadataStructSuite

[ ] - SPARK-41896: Filter on row_index and a stored column at the same time
[ ] - SPARK-43450: Filter on aliased _metadata.row_index

GlutenCSVv1Suite

[ ] - SPARK-23786: warning should be printed if CSV header doesn't conform to schema
[ ] - lineSep with 2 chars when multiLine set to

GlutenCSVv2Suite

[ ] - SPARK-23786: warning should be printed if CSV header doesn't conform to schema
[ ] - lineSep with 2 chars when multiLine set to
[ ] - test for FAILFAST parsing mode
[ ] - SPARK-39731: Correctly parse dates and timestamps with yyyyMMdd pattern

GlutenCSVLegacyTimeParserSuite

[ ] SPARK-23786: warning should be printed if CSV header doesn't conform to schema
[ ] lineSep with 2 chars when multiLine set to

GlutenFileSourceSQLInsertTestSuite

[ ] - SPARK-41982: treat the partition field as string literal when keepPartitionSpecAsStringLiteral is enabled

GlutenDSV2SQLInsertTestSuite

[ ] -SPARK-41982: treat the partition field as string literal when keepPartitionSpecAsStringLiteral is enabled

CC: @zhli1142015

I would like to contribute to this issue and can start by picking few of the suites.

@ayushi-agarwal  @vibhaska
Thank you very much for your contributions. I have two small suggestions that may expedite the PR merge process.

The tests in Spark 34 have all been copied from Spark 33. So, for the failed PRs, if they indeed need to be excluded, please explain why they passed in Spark 33 but are failing in Spark 34. If it is indeed a bug in Spark 34, we may need to open a separate PR to fix it, ensuring the robustness of Spark 34.
Once the PRs are merged into the gluten branch, kindly update this issue list with the updated information. You can put a checkmark in front of the relevant options. And can directly update in this comment .

@ayushi-agarwal @vibhaska Thank you very much for your contributions. Once the PRs are merged into the gluten branch, kindly update this issue list with the updated information. You can put a checkmark in front of the relevant options. And can directly update in this comment .

@JkSelf Is there any way to reopen https://github.com/oap-project/gluten/issues/3559?
As of now @ayushi-agarwal and me do not have privilege to either re-open, edit or  put a checkmark against selected option.

@ayushi-agarwal @vibhaska Thank you very much for your contributions. Once the PRs are merged into the gluten branch, kindly update this issue list with the updated information. You can put a checkmark in front of the relevant options. And can directly update in this comment .

@JkSelf Is there any way to reopen #3559?
As of now @ayushi-agarwal and me do not have privilege to either re-open, edit or put a checkmark against selected option.

You can either update the comment , or let's sync the progress offline, I can update the list for you.
Thanks.

With merged PR https://github.com/oap-project/gluten/pull/4143 fixed following unit tests
GlutenDynamicPartitionPruningV1SuiteAEOff
GlutenDynamicPartitionPruningV1SuiteAEOn
GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
GlutenDynamicPartitionPruningV2SuiteAEOff
GlutenDynamicPartitionPruningV2SuiteAEOn
GlutenDynamicPartitionPruningV2SuiteAEOnDisableScan
GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan

@zhli1142015 Following suites can also be marked as fixed:
GlutenCoalesceShufflePartitionsSuite
GlutenAdaptiveQueryExecSuite
GlutenEnsureRequirementsSuite
GlutenKeyGroupedPartitioningSuite
GlutenParquetSchemaSuite
GlutenWriteDistributionAndOrderingSuite
GlutenJoinSuite
GlutenFallbackSuite
GlutenJsonV2Suite
GlutenCTEHintSuite
GlutenExpressionsSchemaSuite
GlutenDataSourceV2Suite
GlutenSQLQuerySuite:

the escape character is not allowed to end with
SPARK-40245: Fix FileScan canonicalization when partition or data filter columns are not read
GlutenQueryExecutionErrorsSuite

@ayushi-agarwal @vibhaska We have upgraded Spark from version 3.4.1 to 3.4.2, and as a result, four additional unit tests have failed. Here is the list of the failed tests:
1. GlutenKeyGroupedPartitioningSuite # SPARK-44641: duplicated records when SPJ is not triggered
2. GlutenArithmeticExpressionSuite # SPARK-45786: Decimal multiply, divide, remainder, quot
3. GlutenJoinSuite # SPARK-45882: BroadcastHashJoinExec propagate partitioning should respect CoalescedHashPartitioning
4. GlutenMathFunctionsSuite # SPARK-44973: conv must allocate enough space for all digits plus negative sign

I have recorded here https://github.com/oap-project/gluten/issues/3559#issue-1967407941. Please help to enable the additional failed unit tests. Thanks for your help.

With Merged PR https://github.com/oap-project/gluten/pull/4254 following suites got fixed:
GlutenCSVv1Suite
GlutenCSVv2Suite
GlutenCSVLegacyTimeParserSuite

With merged PR https://github.com/oap-project/gluten/pull/4255 following suites got fixed
GlutenFileSourceSQLInsertTestSuite

SPARK-41982: treat the partition field as string literal when keepPartitionSpecAsStringLiteral is enabled

GlutenDSV2SQLInsertTestSuite
-SPARK-41982: treat the partition field as string literal when keepPartitionSpecAsStringLiteral is enabled

@zhli1142015 These 3 can also be marked as done:
GlutenQueryExecutionErrorsSuite
GlutenDataFrameSuite
GlutenFileMetadataStructSuite

@JkSelf just noted the issue is closed? let's keep it open until all UT fixed.

@JkSelf just noted the issue is closed? let's keep it open until all UT fixed.

@FelixYBW  Sure. When the pull request is merged, this issue will be automatically closed if the commit message includes [GLUTEN-3559], which corresponds to the specific issue number. As a result, manual reopening of the issue will be required.

@zhli1142015 @JkSelf Can we mark all tests under GlutenSQLQueryTestSuite as done, except following:

typeCoercion/native/mapconcat.sql *** FAILED ***
-- mapconcat.sql is disabled for 3.4
udaf.sql *** FAILED ***
-- but udaf/udaf.sql passed so just need to verify if same test
udf/postgreSQL/udf-select_having.sql - Scala UDF *** FAILED ***
-- select_having.sql is disabled for 3.4
describe-table-column.sql *** FAILED ***
-- missing test file?
postgreSQL/select_having.sql *** FAILED ***
-- select_having.sql is disabled for 3.4
subquery/scalar-subquery/scalar-subquery-select.sql *** FAILED *** (Gluten throws exception: more than one row returned by a subquery used as an expression)
-- disabled for 3.4
group-by.sql *** FAILED ***
-- PR raised
group-by-ordinal.sql *** FAILED ***
-- PR raised

@zhli1142015 @JkSelf Can we mark followng test as done:
GlutenParquetV1FilterSuite
filter pushdown - StringPredicate
GlutenParquetV2FilterSuite
filter pushdown - StringPredicate
Gluten - filter pushdown - date
Ref: https://github.com/oap-project/gluten/pull/4381 and https://github.com/oap-project/gluten/pull/4363

can this issue be closed ? @JkSelf / @weiting-chen

@gaoyangxiaozhu Closed.


in noise sim, t_ker should be a function of tau_r and tau_d

We should algorithmically define t_ker as a function of tau_r and tau_d, rather than requiring users to define it manually.

How about just 5*tau_r + 5*tau_d?


[WIP] Spanner - Use gax for runWithRetries()

Instead of using its own retry logic, SpannerImpl should use the retry helper and options from gax.
This PR is based on #4749 Spanner refactor spannerimpl, which should be processed first.
Issues / discussion:
SpannerImpl already uses io.grpc.Context to enable setting timeouts and cancelling database operations. This will interfere with using the functionality from gax. The implementation in this PR supports both, but the behavior is different from the previous version if the user for example sets a short timeout on Context in combination with a longer timeout on RetrySettings. If the timeout of RetrySettings is longer than the timeout set on Context, the context timeout will only occur after the timeout of RetrySettings has occurred (i.e. the time before the timeout exception is thrown is longer than the user would have expected).
Possible solutions as I see them at this moment:

Remove support for using io.grpc.Context for setting timeouts and cancelling database operations, and only rely on the functionality from gax (which use the RetrySettings). This would change the behavior for clients who currently use this to set a timeout or to cancel calls.
Use the deadline from io.grpc.Context if one is set (or if the current context is a CancellableContext), and use gax in all other cases. This basically means 'use the current functionality if the user has set a deadline on the Context or indicated that it should be cancellable, and use RetrySettings in all other cases'.
Do not migrate this functionality to use gax.

@olavloite ApiFuture returned by callable do support cancellation of calls. Also with every call to callable, we can provide call context with deadline. I think whole of retrying logic can be migrated to gax and configured in handwritten client.

@olavloite ApiFuture returned by callable do support cancellation of calls. Also with every call to callable, we can provide call context with deadline. I think whole of retrying logic can be migrated to gax and configured in handwritten client.

@ajaaym Yes, I agree with you on that, but as far as I can see we will be changing the behavior for users that currently set a deadline or cancellation themselves. Consider the code in SpannerImplRetryTest.contextDeadlineExceeded() and assume that this was not a test case, but code written by a user. Moving the retry logic of Spanner from the current implementation to gax will lead to the question: Which deadline should we apply to a database operation. The one specified in the RetrySettings of the SpannerOptions that was used to create the Spanner instance, or the one the user has set on the call context? Gax will by default use the one specified in RetrySettings.
Or am I missing something here?

Replaced by #4940


Link to the Facebook case study is broken

There is a broken link that should be pointing towards case study from Facebook. It links to facebook.com/unsupportedbrowser instead.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d1612f19-f267-493a-c230-2ca18e4eb50b
Version Independent ID: d312db0f-6cf1-87ca-09b0-604a66e18a07
Content: Facebook, Google, and external provider authentication in ASP.NET Core
Content Source: aspnetcore/security/authentication/social/index.md
Product: aspnet-core
Technology: aspnetcore-security
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Can you copy/paste the link?

https://www.facebook.com/unsupportedbrowser

@Rick-Anderson  I just had a look into the history and the old link was https://developers.facebook.com/docs/apps/register and replaced with this https://www.facebook.com/unsupportedbrowser in commit id

Code commit https://github.com/dotnet/AspNetCore.Docs/pull/24371


Add a Custom 404 Page

Make it so, when configured to do so, the system generates a custom 404 page.
The default behavior should be to NOT generate this page, to maintain backward compatibility. It should be enabled using a build OR configuration switch.
Compatibility with GitHub custom 404 pages is recommended: https://docs.github.com/en/pages/getting-started-with-github-pages/creating-a-custom-404-page-for-your-github-pages-site

Should the source come from a template page, or just be generic based on the content settings (Site title, description, etc)?


Follow up tasks from i18n PR #24

I left a number of comments unresolved in PR navapbc/template-infra#24. We should follow up that PR and address them!

[x] How do the i18n translation files in public/locale impact JS bundles and compression? comment link
[x] Refactor some of the duplicated code in .storybook/i18next.js and tests/jest-i18n.ts comment link
[x] Consider refactoring the i18n props in getServerSideProps() comment link

Closing:

Configuration has been DRY'ed in https://github.com/navapbc/template-application-nextjs/pull/59
Could not modularize getServerSideProps due to the way Next.js removes server-side imports, and documented that nuance here https://github.com/navapbc/template-application-nextjs/pull/60
Bundle isn't impacted by the locales being in the public/ directory


Update env file for oracles

Description
Updated oracles to 1.0.0-RC2 and (TODO) removed cLabs-run oracles.
Other changes
Describe any minor or "drive-by" changes here.
Tested
An explanation of how the changes were tested or an explanation as to why they don't need to be.
Related issues

Fixes #[issue number here]

Backwards compatibility
Brief explanation of why these changes are/are not backwards compatible.
Documentation
The set of community facing docs that have been added/modified because of this change

dupe


Fix JNI type casts in AndroidCallbacks.cpp

Fix JNI type casts in AndroidCallbacks.cpp
Changes:

Audit method lookups and use the correct types (generally J for long instead of I for int as methods evolved)
Apply static_cast to JNI-specific types. Without this we found that auto-promotion may break on architectures where sizeof(jlong) > sizeof(jint), in which case we were passing a 32-bit integer but java was taking out 64-bit integers via varargs, resulting in illegal memory access.

Do need merge, debugging Java test CI. Thanks

The Android TV App seems to be giving build errors.


CartoDB -> CARTO - cartodb.com -> carto.com

Changes CartoDB by CARTO
Changes cartodb.com by carto.com
Changes these things (:point_up:) in default attribution

cc: @matallo @csobier @xavijam @javitonino

I think http://docs.carto.com should be https://carto.com/docs. But the redirect works, so it is not very important. LGTM :+1:

LGTM, tests failing?

Retest this please.

👍


ISSUE-301 Update Keycloak version to 17.0.1

Fixes #301.
Signed-off-by: Thomas Darimont<EMAIL_ADDRESS>

@thomasdarimont Thank you for your PR!. I will run conformance tests as regression for doublecheck.

@thomasdarimont To work on updating keycloak version to 18, I would like to merge this PR and send a follow-up PR from me to add "--proxy=passthrough". Thank you.


[map tile] consider rendering a single svg instead of separate legend and map svgs

Both the node server and chart embed need to take the map svg and legend svg and combine it into one svg. Consider just having the tile do the merging instead.

not planned


Add support for S3 URI in custom_chef_cookbook.

Issue #, if available: Fixes #1640
Description of changes:
I added support for using S3 URI in the custom_chef_cookbook section. I modified the CFN template to create a presigned URL when the custom_cookbook variable starts with s3://. This minimizes the changed needed to the UserData fields.
I have also added a custom validator that checks that the S3 bucket in the URI is also in the s3_read_resource field. The assumption is that the primary use of S3 URIs for custom_chef_cookbook is for private use, so it would make sense to list the bucket there. More advanced validation based on policies could be performed, if deemed necessary.
I ran tox -e autoformat and tox. All tests were successful.
I also tried to create a cluster with
[cluster default]
s3_read_resource = {*, arn:aws:s3:::dumontj-cfn-test2, arn:aws:s3:::dumontj-cfn-test23}
s3_read_write_resource = {*}
custom_chef_cookbook = {s3://dumontj-cfn-test2/cookbooks/aws-parallelcluster-cookbook-2.6.1.tgz}

where {} indicate a selection. The validation phase emitted warnings as expected.
Note that the names of the bucket cannot be quoted, as quotes are parsed literally by configparser.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Some minor comments:

Travis is failing with the message: pcluster/config/validators.py:592:1: C901 'url_validator' is too complex (11).
To fix this you could create a new function to be called in the url_validator one in the s3 case (or eventually add # noqa: C901 at the end of the row with the function definition)
Then I'd ask you to fixup all the commits into a single one.

Thanks!

Codecov Report

Merging #1749 into develop will increase coverage by 0.22%.
The diff coverage is 80.64%.

@@             Coverage Diff             @@
##           develop    #1749      +/-   ##
===========================================
+ Coverage    53.01%   53.24%   +0.22%
===========================================
  Files           24       24
  Lines         3984     4010      +26
===========================================
+ Hits          2112     2135      +23
- Misses        1872     1875       +3

Impacted Files
Coverage Δ

cli/pcluster/config/validators.py
82.63% <80.64%> (+0.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1fab116...0391bf8. Read the comment docs.

Hi Enrico,
Sorry about the double rebase,  I figured there was a simpler way of doing things than my original solution. I seperated S3 URI validation in its own function to fix the C901 linting error, and rebased on the latest develop.

@enrico-usai, for context, the clusters I spin up are joined to Active Directory. I use additional recipes to setup/manage /etc/security/access.conf and sudoers.d configuration files (among others). I also setup/manage custom profile.d files. I also modify the default OpenSSH configuration to allow for X forwarding, and for password authentication to be allowed except for the cluster user.
Also, I wanted to setup X11 forwarding with slurm, but slurm printed warnings when PrologFlags=X11 was set, which broke nodewatcher. So I patched slurm and modified the slurm recipe to apply the patch before compiling and installing.
I say manage above because I set up cfn-hup with a custom update configSet to be able to manage all of the configuration items named above (and more) with pcluster update.
Hope this helps.


Any thoughts on supporting <dialog />?

It seems that the dialog element is widely supported and it brings benefits being a native element. Any thoughts if this could be supported within the project?
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/dialog

@flexchar
Thank you to bring this up! I've been thinking about supporting composable like useDialog() based on <dialog /> for a while.
Basically, useModal() and VueFinalModal.vue kind of implementing the similar behaviors to <dialog /> element.
Recently I will start implementing v5 and put useDialog() and VueFinalDialog.vue into the new feature list.

To be completely honest, after I played with both I figured that <dialog /> is essentially an alternative rather than a base. The only problem is that one cannot control where the models will be based - that's what <ModelsContainer /> does now.
However other than that, I'd say that it may completely replace the need for libraries in many cases and will allow reclaim maintainers their time. :)


psalm_clan.init() missing 2 required positional arguments: 'input_dim' and 'output_dim'

Traceback (most recent call last):
  File "/disk1/user/nong/source/Protein/annotate/PSALM.py", line 36, in <module>
    fire.Fire(Main)
  File "/disk1/user/nong/miniconda3/envs/PSALM/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/disk1/user/nong/miniconda3/envs/PSALM/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/disk1/user/nong/miniconda3/envs/PSALM/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/disk1/user/nong/source/Protein/annotate/PSALM.py", line 19, in annotate
    PSALM = psalm(clan_model_name="ProteinSequenceAnnotation/PSALM-1-clan",
  File "/disk1/user/nong/miniconda3/envs/PSALM/lib/python3.10/site-packages/psalm/psalm.py", line 195, in __init__
    self.clan_model = psalm_clan.from_pretrained(clan_model_name)
  File "/disk1/user/nong/miniconda3/envs/PSALM/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/disk1/user/nong/miniconda3/envs/PSALM/lib/python3.10/site-packages/huggingface_hub/hub_mixin.py", line 569, in from_pretrained
    instance = cls._from_pretrained(
  File "/disk1/user/nong/miniconda3/envs/PSALM/lib/python3.10/site-packages/huggingface_hub/hub_mixin.py", line 789, in _from_pretrained
    model = cls(**model_kwargs)
TypeError: psalm_clan.__init__() missing 2 required positional arguments: 'input_dim' and 'output_dim'

Hi! Would you be able to provide a little more context (maybe a code snippet)? The psalm_clan class inherits from the HuggingFace PyTorchModelHubMixin class, which has the from_pretrained method. That allows us to get the PSALM clan weights + other configs like input_dim and output_dim from the HF model hub. It looks like maybe the config.json file is not being recognized for some reason.

sorry for missing the explanation. I follow the Usage as your provided and running the code as below , but error comes out
import os, sys, re, gzip
from pathlib import Path
sys.path.append('/disk1/user/nong/source/commonFunctions')
import commonFunctions
import torch
from psalm import psalm

class Main:
    def annotate(self, ):
        # Load PSALM clan and fam models
        PSALM = psalm(clan_model_name="ProteinSequenceAnnotation/PSALM-1-clan",
                    fam_model_name="ProteinSequenceAnnotation/PSALM-1-family",
                    device = 'cuda') #cpu by default, replace with 'cuda' or 'mps' as needed

        # Prepare data (use PSALM.read_fasta(fasta_file_path) to get data directly from a FASTA file)
        data = [
            ("Human Beta Globin", "MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWTQRFFESFGDLSTPDAVMGNPKVKAHGKKVLGAFSDGLAHLDNLKGTFATLSELHCDKLHVDPENFRLLGNVLVCVLAHHFGKEFTPPVQAAYQKVVAGVANALAHKYH"),
            ("Flavohemoprotein", "MLDAQTIATVKATIPLLVETGPKLTAHFYDRMFTHNPELKEIFNMSNQRNGDQREALFNAIAAYASNIENLPALLPAVEKIAQKHTSFQIKPEQYNIVGEHLLATLDEMFSPGQEVLDAWGKAYGVLANVFINREAEIYNENASKAGGWEGTRDFRIVAKTPRSALITSFELEPVDGGAVAEYRPGQYLGVWLKPEGFPHQEIRQYSLTRKPDGKGYRIAVKREEGGQVSNWLHNHANVGDVVKLVAPAGDFFMAVADDTPVTLISAGVGQTPMLAMLDTLAKAGHTAQVNWFHAAENGDVHAFADEVKELGQSLPRFTAHTWYRQPSEADRAKGQFDSEGLMDLSKLEGAFSDPTMQFYLCGPVGFMQFTAKQLVDLGVKQENIHYECFGPHKVL")
        ]

        # Visualize PSALM annotations (add optional save_path argument: PSALM.annotate(data,save_path="save_folder")
        PSALM.annotate(data)

if __name__ == '__main__':
    import fire
    print('Beginnng...')
    fire.Fire(Main)

I just tested it again on my end, and I'm not getting this error. Which version of the package are you using? It should just pull the config and weights from the PSALM HuggingFace model hub. If it doesn't work still, you can try a manual workaround by importing psalm_clan and psalm_fam individually from psalm:

Initialize both the clan and fam models (i.e. specifying input_dim and output_dim). These will be specified in the config.json files for fam and clan
Download the weights for each model (fam and clan) from the HF model hub. For fam, you will also need the maps.pkl file that contains all of the relevant mappings between fam and clan as well as their respective indices for parsing the output of the models.
Load the weights into the model (guide)


[Backport 1.x] flaky transform test fix attempt

Issue #, if available:
Description of changes:
CheckList:

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.
Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>

Codecov Report

Merging #550 (6292dad) into 1.x (b39bd64) will increase coverage by 0.88%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                1.x     #550      +/-   ##
============================================
+ Coverage     75.58%   76.46%   +0.88%
- Complexity     2020     2068      +48
============================================
  Files           254      254
  Lines         11521    11770     +249
  Branches       1823     1830       +7
============================================
+ Hits           8708     9000     +292
+ Misses         1779     1775       -4
+ Partials       1034      995      -39

Impacted Files
Coverage Δ

...xmanagement/rollup/actionfilter/FieldCapsFilter.kt
48.10% <0.00%> (-26.01%)
:arrow_down:

...ndexmanagement/settings/IndexManagementSettings.kt
71.42% <0.00%> (-11.91%)
:arrow_down:

...nt/indexstatemanagement/model/destination/Chime.kt
43.47% <0.00%> (-11.07%)
:arrow_down:

...t/refreshanalyzer/RefreshSearchAnalyzerResponse.kt
65.78% <0.00%> (-9.22%)
:arrow_down:

...earch/indexmanagement/transform/TransformRunner.kt
78.91% <0.00%> (-8.36%)
:arrow_down:

...indexmanagement/rollup/model/RollupFieldMapping.kt
75.00% <0.00%> (-8.34%)
:arrow_down:

.../opensearch/indexmanagement/rollup/RollupRunner.kt
59.29% <0.00%> (-6.20%)
:arrow_down:

...anagement/indexstatemanagement/model/Transition.kt
61.64% <0.00%> (-5.97%)
:arrow_down:

...pensearch/indexmanagement/IndexManagementRunner.kt
50.00% <0.00%> (-5.56%)
:arrow_down:

...shanalyzer/TransportRefreshSearchAnalyzerAction.kt
86.95% <0.00%> (-4.72%)
:arrow_down:

... and 176 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


[LogisticRegression] Match Spark CPU behaviors when dataset has one label

build

build

Any update on this?

You will probably need to patch the ci docker image in this pr to get tests to pass as rapidsai-nightly no longer has cuml 23.12.   switch to rapidsai channel.

build

You will probably need to patch the ci docker image in this pr to get tests to pass as rapidsai-nightly no longer has cuml 23.12. switch to rapidsai channel.

Added the caplog, and a test case to check invalid label. Just updated ci docker image and yes seems ci can run.

build


explicitly set Swift Version for tvOS

otherwise, the carthage build command for tvOS (/usr/bin/xcrun xcodebuild -project Example.xcodeproj -scheme StatefulViewController-tvOS -configuration Release -sdk appletvos ONLY_ACTIVE_ARCH=NO BITCODE_GENERATION_MODE=bitcode CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES clean build) fails

This should be fixed on tag 2.0, the swift2.3 branch is not up to date and should probably be deleted.

thx

It is still broken for tvOS in the 2.0 release, only fixed for iOS.

Running carthage build --no-skip-current --toolchain com.apple.dt.toolchain.Swift_2_3 actually builds fine for me. Have you tried adding the toolchain explicitly?

I did not try that. This surely is one way to fix it.
The targets differ in this point nonetheless, so it could be a good addition to the Readme.

cleaning up my pull requests:
carthage update --platform tvOS,iOS works fine for Release 3.0.1.
Closing this


Fix compilation errors in Swift 1.2 (Xcode 6.3 beta).

AppDelegate:
The type of the launchOptions parameter of the didFinishLaunchingWithOptions function was changed in Swift 1.2.

"launchOptions: NSDictionary?"
has become: "launchOptions: [NSObject: AnyObject]?"

GameScene:
Swift 1.2 has introduced a native Set type that bridges with NSSet.

override func touchesBegan(touches: NSSet, withEvent event: UIEvent) {
has become: override func touchesBegan(touches: Set, withEvent event: UIEvent) {

GameViewController:
Forced conversions in Swift 1.2 are represented with as!. This makes it clear that the conversion may fail if you attempt to downcast to a type that doesn’t represent the value’s type.

let scene = archiver.decodeObjectForKey(NSKeyedArchiveRootObjectKey) as GameScene
has become: let scene = archiver.decodeObjectForKey(NSKeyedArchiveRootObjectKey) as! GameScene
let skView = self.view as SKView
has become: let skView = self.view as! SKView

NSString objects need to be converted to a Swift String value by casting.

let path = NSBundle.mainBundle().pathForResource(file, ofType: "sks")
has become: let path = NSBundle.mainBundle().pathForResource((file as String), ofType: "sks")

It works!  Think you!

Thanks! But fixed in a separate commit. Feel free to resubmit with conflicts resolved if you feel there are other material changes that need merged.


[8pt] Clean up levee data preprocessing .py code, check dependencies, and push to repo

Move current NLD preprocessing code from local to repo

@RyanSpies-NOAA What is the status of this? Can we close?

@BradfordBates-NOAA Let's leave this open. We are currently using a manually downloaded version of the USACE NLD. I think it would be worthwhile to build this into the automated download and preprocessing workflow sometime soon.


Add a maximum queued height metric to the finalized state

Motivation
To diagnose pipeline bugs like #1259, we need more information about Zebra's internal queues.
Solution

Add a maximum queued height metric to the finalized state
Rename all the finalized state metrics to contain "finalized".

Distinguishes finalized state metrics from the non-finalized state metrics in N+1

The code in this pull request has:

[x] ~Documentation~ Comments

adding another metric to existing metrics

~Unit Tests and Property Tests~

I ran the code, and the metrics work as expected

Review
@yaahc is working on the state right now.
This change can happen at any time.
@hdevalence should be aware of this change, because it breaks existing finalized state metrics dashboards.
Related Issues
#1259 duplicate error in checkpoint verifier
#1261 checkpoint verifier metrics fix
#1263 non-finalized state metrics
Follow Up Work
None

The macOS failure is bug #1259.


feat: Extension inference for conditional nodes

This builds on from #463 (merged) and #461.
Generate constraints saying that Case nodes' interfaces should match those of their Conditional and add a test

I've refactored the test to avoid using builder methods, since we want the parent Conditional to have the empty resource set and in the near future the builder will default to making it open. Unfortunately, this means losing access to the CaseBuilder too


User model

Merging blocked on #536, #551, #555, #560, #562 and #563 because this PR contains commits from those PRs.
To use, rebuild containers:
docker-compose down
rm build/api build/api-db build/ssh build/auth-server
make build/api
make build/api-db
make build/ssh
make build/auth-server

Maybe also need to rm the build/yarn-workspace-builder file?
TODO:

[x] Add new tables to database
[x] Modify existing tables in database
[x] Fix database views
[x] Change JWT to include user id, not SSH key
[x] Update ssh token command to verify SSH key against users
[x] Migrations for database tables
[x] Update stored procedures (ssh_key, project_ssh_key, customer_ssh_key)
[x] Update all consumers of database tables and views (ssh_key, project_ssh_key, customer_ssh_key, permissions)
[x] Make Drush automatic tokens compatible with new JWTs (do we still need to do this?)
[x] Look for any code with potential to rely on old structure (ssh.+key)
[x] Automatic migration script for existing data in old format
[x] New additions to the DAO
[x] Modifications to the DAO
[x] New additions to GraphQL schema
[x] Modifications to GraphQL schema
[x] Update all queries and mutations from GraphQL consumers (including api-data.gql)
[x] Comment in commented-out migrations
[x] Fix tests
[x] Update documentation

Later:

Tests for database (queries in DAO, also tables and views?)
Tests for GraphQL
Validation for user emails
CLI integration? (lagu users, lagu user)
Move DAO methods to more idiomatic GraphQL folder / file structure

Planning:
Database Tables
User fields partly inspired by Django User model
user (new)
userid email          firstname lastname
1<EMAIL_ADDRESS>karl      horky

ssh_key (no changes)
sshid key
1     foo
2     bar

user_ssh_key (new)
userid sshid
1      1
1      2  // multiple SSH Keys per user

customer_user (previously customer_ssh_key)
customerid userid
3          1

project_user (previously project_ssh_key)
projectid userid
9         1

GraphQL
User {
  SshKeys: [
    {}
    {}
    {}
  ]
}

Project {
  Customer {
    Users [
      {}
      {}
      {}
    ]
  }
  Users [
    {}
    {}
    {}
  ]
}

Hm, lots of merge conflicts in the new changes from master. Since we have these commits also in #632, I'm going to close this PR and let #632 supersede this.


phpstan-shim vs phpstan

Why is the plugin skeleton using the shim version of phpstan instead of just phpstan/phpstan?

Hi,
I think it's historic.
Since version 0.12 it is directly integrated into phpstan/phpstan :smile:

Indeed, however, we still have in composer.json - we should probably delete it 💃

Looksa a bit outdated, fixed by #182 and v1.7.5 for sure does not include phpstan-shim 🎉


Overview screen

[ ] Show recent, unseen operations
[ ] Show possible new duplicates
[ ] Balance overview over the last month, last 3 or 6 months

I would maybe allow to validate an operation (instead of unseen operations).
Use case :

An operation is done on date X
The operation is registered by the bank at X+1week (happened to me the date is X+10 days).
Beteween date X and date X+1week, many operations occured.
The operation is imported by kresus with date X on day X+1 week.
The operation is "lost" between all imported operations between X and X+1 week.

The seen/unseen mechanism could help to solve this use case. But having a human action on this could be nice also.
This could be as the validate option of Homebank.


[Feature Request] First Libra onlineIDE https://libraide.com

🚀 Feature Request
We build the first Libra IDE which can help developers from the world to have a quick try in 3 seconds. If you are interesting in Libra and Move, our onlineIDE will save hours to do initial setup and environment. It is free and we hope to bring more developers know Libra quickly!
Motivation
We create a free onlineIDE which allow developers to code & compile without initial setting! If we help 1 dev save 1hr to setup, we can save years around the world!
Pitch
Our website is https://libraide.com, please have a try. We want to help more people get into Libra!
Are you willing to open a pull request? (See CONTRIBUTING)
Additional context

Cool, keep it up.
The "Compiled File" is also confusing as it first looks like that is the compile button. And I'd maybe turn the compile button into one that recognizes the currently open file and avoid the dropdown menu - but not sure about this.
Can one inspect the bytecode itself? (Among other things, I'd like to know how the typing on bytecode level, as mentioned in the whitepaper, comes into play).

Good start. Thank you.  How about some sample/example files and a HELP resource.

Aha~ Examples are in the case templates on the left~ the resources are in the setting you can find extra infomation~ We will keep working on the IDE and make it better~

Love it. Is your project OpenSource, if so a link for its Github Repo would be helpful.

Thank you ! We will continue work on more features to help the Libra developers! We are not OpenSource now because we finish ChainIDE in a hurry! I will discuss with team and at least clean the code before we make it OpenSource XD We will make ChainIDE free for everyone to use for sure! Feel free to contact me at<EMAIL_ADDRESS>if you have any comments!

looks cool !

So ,can you remove the limit about the chain. If that you can use your own chain to upload contract

so cool!