[SPARK-15243][ML][SQL][PYTHON] Add missing support for unicode in Param methods & functions in dataframe

What changes were proposed in this pull request?
This PR proposes to support unicodes in Param methods in ML, other missed functions in DataFrame.
For example, this causes a ValueError in Python 2.x when param is a unicode string:
>>> from pyspark.ml.classification import LogisticRegression
>>> lr = LogisticRegression()
>>> lr.hasParam("threshold")
True
>>> lr.hasParam(u"threshold")
Traceback (most recent call last):
 ...
    raise TypeError("hasParam(): paramName must be a string")
TypeError: hasParam(): paramName must be a string

This PR is based on https://github.com/apache/spark/pull/13036
How was this patch tested?
Unit tests in python/pyspark/ml/tests.py and python/pyspark/sql/tests.py.

cc @sethah, @holdenk, @viirya and @k-yokoshi

Test build #73579 has started for PR 17096 at commit d421a82.

Test build #73579 has finished for PR 17096 at commit d421a82.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73579/
Test PASSed.

Thank you for taking this over @HyukjinKwon :)

Let's double check with @viirya to make sure his comment was addressed, but I really appreciate the improved test coverage :)

Few minor comments. Overall looks good.

Thank you @viirya. I think it is ready now.

Test build #73688 has started for PR 17096 at commit 9ac773c.

Test build #73688 has finished for PR 17096 at commit 9ac773c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73688/
Test PASSed.

In StructType.add, if the given field name is a basestring, we will directly use it as key in names property.
If we pass in a StructField, we take StructField.name as key in names. But StructField will encode field name as utf-8 if it is not a str.
And StructType.__getitem__ will find matched field by comparing each StructField.name with given search key. So it is unable to find the field back with the unicode field name.
    from pyspark.sql.types import StructType, StringType, StructField

    struct1 = StructType().add(u"a", "string", True)
    struct2 = StructType([StructField(u"a", StringType(), True)])
    self.assertTrue(struct1 == struct2)  # pass
    self.assertTrue(struct1[u"a"] == struct2[u"a"]) #pass

    struct1 = StructType().add(u"測試", "string", True)
    struct2 = StructType([StructField(u"測試", StringType(), True)])
    self.assertTrue(struct1 == struct2)  # fail
    self.assertTrue(struct1[u"測試"] == struct2[u"測試"]) # fail, you can't find the field with key u"測試"

@viirya, thank you so much for taking a look and your time.
So, basically, the second case it compares str to unicode as below:
>>> u"測試" == u"測試".encode("utf-8")
False

Apparently, it seems we could pass unicode as is? Let me raise another issue for this after testing and looking into this. Actually, the support in StructType.add seems not the problem specified in the JIRA.

Let me check if each is fine for sure.

Test build #73822 has started for PR 17096 at commit cd235a7.

@holdenk and @viirya, I got rid of the changes in types.py and only left that I am pretty sure.
There are two kind of changes here that look used in the only local scope.
One seems for used getattr I guess it is fine as below:
>>> getattr("a", u"__str__")
<method-wrapper '__str__' of str object at 0x10a24e580>
>>> getattr("a", "__str__")
<method-wrapper '__str__' of str object at 0x10a24e580>

and other one seems used for setting an parameter to JVM which seems already used in the code base much more.

Test build #73822 has finished for PR 17096 at commit cd235a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73822/
Test PASSed.

Remaining changes LGTM. cc @holdenk

Thank you @viirya for your sign-off.

Thanks for taking the time to review this @viirya :)

Hey @holdenk. I am willing to close this for now if you are not confident enough for merging it for now. I can re-open this later.

@HyukjinKwon @holdenk Hi, are you still working on this?

Yea, I was waiting for the feedback. It is also about the unicode vs byte string.

I'd really like to see that further test I was talking about, @HyukjinKwon -- it shouldn't be too hard to do in this pr right? Just add a unicode string which doesn't make sense in down converted ascii.

Do you mean a test case such as self.assertEqual(testParams._resolveParam(u"아"), testParams.아)  ?

@HyukjinKwon Pretty much, yes.

Test build #79038 has started for PR 17096 at commit 830b4fe.

Test build #79038 has finished for PR 17096 at commit 830b4fe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79038/
Test PASSed.

Merged build finished. Test PASSed.

gentle ping ...

Sorry for the delay. Lets get Jenkins to retest this and make sure everything is ok but it looks like a good change :)
LGTM pending jenkins/merge issues (if they show up during jenkins). Jenkins retest this please.

Jenkins retest this please.

Test build #81488 has started for PR 17096 at commit 830b4fe.

Test build #81488 has finished for PR 17096 at commit 830b4fe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81488/
Test PASSed.

Thanks, merged to master!

Thank you @holdenk, @viirya and @ueshin.


Use consistent aliases for namespaces in tests and examples

Most of the tests use btproto as a namespace alias for google::bigtable::v2 and btadmin for google::bigtable::admin::v2.  But some use admin_proto, or other names.  Not very important, but a pass cleaning these up would be nice.

Why is this part of the milestone?  It does not affect what users see.


⚠ Samsung has degraded performance

In 3a078b4, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2745 ms

Resolved: Samsung performance has improved in aa65a65.


Add more buildings to the New Zealand dataset

Some cities in New Zealand are growing rapidly and therefor some cities lack some buildings. Would it be possible to request just certain areas to scan again and suggest more buildings?
It would be awesome if it could be possible right in the editor to select an area and request a scan.
Example in Cambridge: https://rapideditor.org/rapid#background=LINZ_NZ_Aerial_Imagery&datasets=ef9136b079e24bbfb432be9bbdd246ff&disable_features=boundaries&map=16.92/-37.92282/175.48239
Another example: https://rapideditor.org/edit#background=LINZ_NZ_Aerial_Imagery&datasets=ef9136b079e24bbfb432be9bbdd246ff&disable_features=boundaries&map=19.20/-38.03469/175.35401

Great idea and thanks for the request @simonschaufi. Right now I don't think we have a very good story for 'continually updating' datasets from ESRI. Till now they've been mostly datasets that represent a snapshot in time. The features get added to OSM, and once done, the feature set is retired or taken offline.
I'll be attending ESRI UC next week and will bring up the idea of 'tasking' and/or continually refreshing a dataset such as this with our partners there.

Great!

chatted with ESRI about this today, expect a response from them soon in this space. :)

Yes, we talked and we are planning to prepare an update of the LINZ buildings shortly (next couple weeks) and will update the layer that is hosted in ArcGIS Online and accessible in Rapid.  We'll post an update here when that update is done!


esm file must be of extension .mjs

tldr: https://github.com/vuejs/vitepress/issues/1399#issuecomment-1258364648
hi, when component builds inside vitepress, it's fails with error:
✖ rendering pages...
build error:
 file:///Users/path-to-project/docs/.vitepress/.temp/SomeComponent.c214d190.js:4
import { Sortable } from "sortablejs-vue3";
         ^^^^^^^^
SyntaxError: Named export 'Sortable' not found. The requested module 'sortablejs-vue3' is a CommonJS module, which may not support all module.exports as named exports.
CommonJS modules can always be imported via the default export, for example using:

import pkg from 'sortablejs-vue3';
const { Sortable } = pkg;

    at ModuleJob._instantiate (node:internal/modules/esm/module_job:124:21)
    at async ModuleJob.run (node:internal/modules/esm/module_job:190:5)

to handle such cases with different environments, need to change the extension of the file from .es.js to .es.mjs in package.json and vite.config of this project.
checked locally with the this changes, project builds without errors and everything works

@said-m can you create a PR for this change? I am using vitesse and get the same error


rocprof -i input.txt with several pmc lines crash

When running
./rocprofiler/bin/rocprof -i input.txt --hsa-trace -d traces ./roctracer/test/MatrixTranspose/MatrixTranspose

with several pmc lines in the input.txt file, the program crash with a segmentation fault error after the second run of the program
./rocprofiler/bin/rocprof: line 271: 861977 Segmentation fault      (core dumped) "./roctracer/test/MatrixTranspose/MatrixTranspose"

I use rocm-4.0.0 and I cloned and built rocprofiler and roctracer from the github repositories. I used both amd-master and rocm-4.0.x branches and the error occurs in both cases.
input_error.txt
input.txt

This issue is still happening on ROCm 5.0.0.

Hi @yoann-heitz and @arfio, I was not able to reproduce this issue on the latest ROCm 6.2.0 release. I used the input.txt file provided with the MatrixTranspose example found here. I will close this issue, if you are still encountering the segmentation fault please open a new ticket, thanks!


Add basic observer controls

New function, Room.getRandomRoomInRange(roomname, range), which returns the name of a random room.

StructureObserver.monitor(room, ticks) tells the system to attempt to monitor the specific room with observers for a specific number of ticks.

the new EmpireObservers program goes through each observer and either assigns it a target (from the monitor function) or a random room.

Right now a random room is choosen when no targets are available. It's possible another algorithm would be better- for instance, once power mining is enabled we'll need to do a better job looking at hallways- but this method is super low CPU, doesn't depend on memory, and should cover all rooms in range in a reasonable number of ticks.

Oh, as a secondary feature this PR also fixes a bug in Room.getRoomsInRange where it would previously return rooms that were not on the map.

This Pull Request has been merged by GitConsensus.
Vote Totals

Yes
No
Abstain
Voters

3
0
0
3

Vote Breakdown

User
Yes
No
Abstain

tedivm
✔

dragoonreas
✔

starwar15432
✔

Vote Results

Criteria
Result

Has Quorum
True

Has Votes
True


User id not mapping in container

I'm on a 2010 Mac Pro on Big Sur.  I've installed docker via homebrew.
Docker version 20.10.7, build f0df35096d
ddev version v1.17.7
When I'm creating new projects with ddev, the images are not mapping my system id (501) to the docker user (1000).
Consequently I run into a lot of permissions issues.  Just wondering if there's a way to get this resolved.
Thank you

Hi @jpea72 - The docker user is not important, it's your user that's important.
Could you please (on the host) report the results of
id -a
ddev exec id -a
ddev exec ls -l

I note that Big Sur doesn't support 2010 macs AFAIK, I thought 2013 was the latest year supported. Do you have something else interesting going on with your mac? (https://support.apple.com/kb/sp833?locale=en_US)

$ id -a
uid=501(jkay) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),101(com.apple.access_screensharing-disabled),399(com.apple.access_ssh),400(com.apple.access_remote_ae),701(com.apple.sharepoint.group.1)

$ ddev exec id -a
uid=501(jkay) gid=20(dialout) groups=20(dialout)

$ ddev exec ls -l
total 0
drwxr-xr-x 1 1000 staff 160 Jul 28 16:48 web

I'm running open core, everything else seems to be fine.  I'm using docker to build executables for elixir and it seems to work just fine.  Just having this issue with the permissions on ddev projects.

As you see, ddev is certainly mapping your uid correctly. The only problem I see offhand is that the filesystem doesn't have the correct ownership (1000 instead of 501).
Are you perhaps using Docker Toolbox, which is completely unsupported by ddev?
Please run the script https://github.com/drud/ddev/blob/master/scripts/test_ddev.sh and provide the results as a gist (gist.github.com). Thanks.

Here's the gist:
https://gist.github.com/jpea72/0fd05f6f53e8d13c5fc791397ec906e8

"Docker Desktop for Mac File Doesn't Exist"
So you're using something other than Docker Desktop.
I don't know if there might be a way for you to get Docker Toolbox working, I've never tried. DDEV has never supported it.
But you can probably do a sudo chown -R "$(id -u):$(id -g)" /var/www/html in a ddev post-start hook.

No response from OP, and docker toolbox is not supported on any OS. But I hope you'll write up your techniques if you get it to work! People sometimes want to do interesting things with old macs!
Closing for now, but happy to continue the conversation here or elsewhere.


Added paper "Principal Neighbourhood Aggregation for Graph Nets"

https://github.com/lukecavabarrett/pna

Thank You Gabriele, super cool stuff!

Thank you!


config-options: Add 'initialvaluedesc' option field needed for "Project features" section

We will need a new key title for "Initial value", see here

Thanks @gabrielmougard !
Can you also increase the version number in setup.cfg, then I can publish.

@ru-fu updated

@gabrielmougard published :)


Not working anymore

Although the BWT Perla is accessable in my browser, the integration does not work anymore. It fails and in the integration page it says "Retrying setup: None"
I tried disableing and enabling. restating HA. nothing works.
In the logs I can only see: 2022-07-24 18:19:30 ERROR (MainThread) [custom_components.bwt_perla] Error fetching bwt_perla data:
When I delete the integration and try to set it up again. I get "wrong host / code. But: when I open the host in the browser and enter the code I can login

Think this is because they changed their software to "VNC Only"
https://community.home-assistant.io/t/bwt-best-water-tech-nology-support/270745/27?u=weidi
I´m in the same boat but @fuatakgun seems having a solution already.

Mhh that's strange. not sure if I got the update already but my Perla is still available via web UI. Or what is meant by "VNC"?

They implemented a web VNC control instead of a real website
bernji @.***> schrieb am So., 31. Juli 2022, 08:30:

Mhh that's strange. not sure if I got the update already but my Perla is
still available via web UI. Or what is meant by "VNC"?
—
Reply to this email directly, view it on GitHub
https://github.com/fuatakgun/bwt_perla/issues/7#issuecomment-1200359106,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAQWGHSAQLQO23NN3NUF3S3VWYMRLANCNFSM54RHHMAQ
.
You are receiving this because you commented.Message ID:
@.***>

I see. By looking at the source it seems that there is some fallback implemented. Maybe both VNC and non VNC works?

Hello
@fuatakgun Is there any update on this? In the HA community you wrote that it seems to be working for you a couple of months ago.
It would be really great to have this great integration working again as the BWT app and website is just crap.
thanks a lot!

There is a new API from BWT: https://bwt-real-smarthome-backend.azurewebsites.net/

I am working on real time extraction using remote connection to VNC server and ocr (image to text)

There is a new API from BWT: https://bwt-real-smarthome-backend.azurewebsites.net/

This API exists since a while, but unfortunately is limited and completely cloud-based.
There seems to be a new (yet undocumented) local REST API built in into newer firmware versions. Unfortunately up to now just a read interface without any possibility to change values or enable/disable the holiday mode.
See https://www.loxforum.com/forum/projektforen/loxberry/plugins/278330-plugin-bwt-aqua-anbindung-wasserenthärter/ for latest details (in german)

Are you sure this is the new version? It looks like (source code) accessing my previous version of the machine at home? But after the upgrade, it is not working anymore locally for me.

Are you sure this is the new version? It looks like (source code) accessing my previous version of the machine at home? But after the upgrade, it is not working anymore locally for me.
Especially look here: https://github.com/paolobazzi/LoxBerry-Plugin-BWT-Aqua/blob/87f03dbdb736f30ec666a9bf71b6dda2087d8d99/webfrontend/html/index.cgi#L126
This is what has been implemented inside this repository but it won't work with new devices.

Sorry my answer below seems to have been missleading. The code you quoted is my BWT plugin for loxone which - as you wrote - accesses the old local API which was removed when BWT switched to the VNC-based UI.
But if you follow the whole thread linked above the latest posts indicate a new local REST API which is available but still undocumented (maybe you have to activate the local API in your BWT UI on the machine).
Check out the following local URLs:  [using simple Basic Auth]

http://user:<your-pass>@192.168.x.y:8080/api/GetCurrentData
http://user:<your-pass>@192.168.x.y:8080/api/GetDailyData
http://user:<your-pass>@192.168.x.y:8080/api/GetMonthlyData
http://user:<your-pass>@192.168.x.y:8080/api/GetYearlyData

@paolobazzi I just got a reply from BWT, who mentions that Firmware 2.0203, which I understand you have for BWT Aqua equipment, is not available for BWT Perla, which this integration is created for.


Weird display of GUID (RAW(16)) type (Oracle)

Description
as shown...
DBeaver Version
22.2.0 CE
Operating System
win 10
Database and driver
Oracle

Steps to reproduce
No response
Additional context
No response

See:
SELECT sys_guid() FROM dual
for testing

You can set Value display format to Database native. Maybe it's similar to requested representation.

You can set Value display format to Database native. Maybe it's similar to the requested representation.

Yes, It helps. But it is very non-obvious.... I dont know, maybe the issue is not needed anymore, what do you think?

We discussed that we could try improving guid representation by default. Let's keep this issue open. Thank you for report.

This also applicable to BLOB(16) in SQLite.

More recent form for the same problem (DBeaver 23.1.2)
Attributes

Data grid output

I encountered the same problem until I stumbled upon this problem. The fix isn't immediately apparent.


ci: Preparing next development version

Codecov Report

Merging #74 (d9dc17f) into main (27f9b21) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #74   +/-   ##
=======================================
  Coverage   91.32%   91.32%
=======================================
  Files           8        8
  Lines        1878     1878
  Branches      278      278
=======================================
  Hits         1715     1715
  Misses        161      161
  Partials        2        2

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Frax and FXS

bumping, please ser @michaelkaplan13

Hi there, thanks for your interest in Avalanche. We have recently updated the review process for tokens on the Avalanche Bridge. I'm closing all of the pending PRs in this repo now, and someone from our business development team should reach out to you soon if they have not already.


OStorageException: Invalid value of TX counter (Exception during commit of active transaction)

OrientDB Version, operating system, or hardware.

v2.2.12

Operating System

Linux

Expected behavior and actual behavior
Steps to reproduce the problem
I'm not sure if it's a bug because  java.lang.OutOfMemoryError: GC overhead limit exceeded raised.
It's a chat server. When I send many messages in short time (600~1000 in about 10minutes). This exception raised. After that, any time I save a record, ODatabaseException will raise.
orientdb-0_0.log.txt
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Error in data flush background thread, please restart database and send full stack trace inside of bug report
DB name="main"
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkLowDiskSpaceFullCheckpointRequestsAndBackgroundDataFlushExceptions(OAbstractPaginatedStorage.java:4185)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.createRecord(OAbstractPaginatedStorage.java:1036)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:2095)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:246)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:179)
... 28 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Hi @vtliem that is definitively caused by out of memory exception. Could you raise amount of memory, but before that could you make import/export procedure to fix possible issues with database ?

I changed from
-Xmx1000m -Dstorage.diskCache.bufferSize=500
to -Xmx2000m -Dstorage.diskCache.bufferSize=500 and can send 1000 message in 10minutes.
I also did import/export to clean up database.
I read OrientDB document and found

So rather than:
java -Xmx8g ...
You could instead try this:
java -Xmx800m -Dstorage.diskCache.bufferSize=7200 ...

but if limit heap too low, java.lang.OutOfMemoryError: GC overhead limit exceeded raises if do a big transaction or many write in a short time (db is not too big, ~ 30k records + 500k index entries). In this case, is 500MB of bufferSize enough or I need more?

@vtliem if your db is about 1-2GB that is OK for disk cache size.


Better sidecar support

When testing mm-less druid with istio-proxy, I noticed that when you have a controller that automatically injects the sidecar for all pods in a namespace.  The istio-proxy sidecar gets injected as the first container.  The problem here was the assumption that the first container was always the druid container.
I added a configuration option which should assist with this.
if you are running with druid.indexer.runner.sidecarSupport=true
you can specify druid.indexer.runner.primaryContainerName=<overlord_container_name> to be the container name for your overlord.
For example if we have this for your overlord podSpec:
spec:
  containers:
      - image: istio:latest
        name: istio-proxy
      - image: druid:latest
        name: druid-overlord
      - image: splunk:latest
        name: splunk

If your istio is automatically injected by a controller then you would specify this in your overlord config:
druid.indexer.runner.primaryContainerName=druid-overlord

I encountered this problem exactly, patched our druid which fixed the problem on our clusters.

@a2l007 - This has been approved but not merged yet. Are you expecting more changes?

@abhishekagarwal87 No, this is good to go. I probably forgot to merge it. Did you have any comments on this PR?

No. I am good. Just merged it. Thank you @churromorales for your contribution.


Twitter Logo Missing

While going through the website I discovered that the the twitter logo image file  located at the footer along with the animation was corrupted so I replaced the file in the source of the image.

Please assign me this issue


ERROR:Uncaught exception occured. RPCError 406: UPDATE_APP_TO_LOGIN (caused by SendCodeRequest)

When running against any public channel the script (installed with pip install telegram-messages-dump) can't dump anything:
Try to load exporter 'text.py'...  OK!
INFO:Initializing session...
Connecting to Telegram servers...
INFO:Connecting to <IP_ADDRESS>:443/TcpFull...
INFO:Connection to <IP_ADDRESS>:443/TcpFull complete!
Initial connection failed.
First run. Sending code request...
ERROR:Uncaught exception occured. RPCError 406: UPDATE_APP_TO_LOGIN (caused by SendCodeRequest)
0 messages were successfully written in the resulting file. Done!

To solve this issue you must install an updated library for authorization (as suggested in https://github.com/LonamiWebs/Telethon/issues/3220): pip install tulir-telethon
The installer of this script should be updated

see pr here: https://github.com/Kosat/telegram-messages-dump/pull/42


Slash Command - p!lock (misc.py)

Create a slash command for the channel lock command.
Also ensure that the command shows up in discord's slash command menu.
You are free to update the discord-py-slash-command library from 3.0.3, provided you explain why.

Hey, can I be assigned to this :)


run sac_n.py, wandb  init came to down

It seems to be the same as before, it died when wandb was initialized

Is this a problem with my network environment?

Looks like a local problem with an installation, have you tried avoiding unicode symbols (i.e. chinese symbols in the path)? Not one hundred perecents sure this is the problem, but you could give it a try.

It also might be a problem of wandb version. You can try to install the newest one if you don't have it.

Thank you very much, I checked it, it is indeed the reason that the path contains Chinese


scheduler: Fix error when recycling offer-stash.

offer-stash was initialized to nil and was only being reset! to its own
value, @offer-stash - meaning it could never acquire a value other than
nil.
Thus, attempting to put its value back on the channel in the exception
handling block would in turn raise an exception (can't put nil on
channel).  Since this exception was not handled, the thread would die,
and handle-resource-offers! would stop happening, resulting in no offers
to match until Cook was restarted.
I believe the var offers-scheduled contains what @offer-stash USED to
be reset! to, prior to commit a134d1f8776eb1b0627eb74a073bae125bf6d52a.

Yup, you are spot on. Good find!


Pop in testing && improvements

Changes
fixes https://github.com/mintlify/mint/issues/269 but hydration warnings for grouped code blocks that describe custom components will appear.
I think it would make sense to address these in a different Issue.
Tests
Tested locally with a starter template and also the mintlify docs:
For example: http://localhost:3000/development in the mintlify docs is loading without any hydration errors or pop-in, even the code blocks,
but if you add a Component to the code block like so:
<Accordion title="I am an Accordion.">
  You can put any content in here.
</Accordion>

the hydration will fail again. As far as I am aware this is the only Issue left regarding pop-in/hydration of the code blocks.
So far I observerd that any code block, nested in p , grouped and highlighted works, except the ones with custom components.
I will try to narrow it down more, could also be related to the kind of highlighting that is done on the code.

I left this as draft so we can futher investigate the hydration errors without having to use any hacks or refactoring on the code blocks first.
If its okay with you, we can keep it open until we solve the remaining hydration errors.

I tried to reproduce it but could not manage to get this error. I was using yarn preconfigure ../docs
Except for the hydration errors of highlighted code it was rendering fine:

Are you getting a usable call stack with the error?

I was going to review/fix in this PR some more. There was one issue with multiple Tabs in the CodeGroups, I fixed it now on a new branch.


MQTT sensor values from esphomelib sensor are not displayed

Home Assistant release with the issue:
0.81.6
Last working Home Assistant release (if known):
Unknown
Operating environment (Hass.io/Docker/Windows/etc.):
Docker
Component/platform:
https://www.home-assistant.io/components/mqtt
Description of problem:
MQTT sensor values from esphomelib sensor are not displayed
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
mqtt:
  broker: <PRIVATE_IP>
  discovery: true

Traceback (if applicable):
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on homeassistant/sensor/bedroom_environment/bedroom_gas/config: b'{"unit_of_measurement":"\xe2\x84\xa6","expire_after":180,"icon":"mdi:gas-cylinder","name":"Bedroom Gas","state_topic":"bedroom_environment/sensor/bedroom_gas/state","availability_topic":"bedroom_environment/status"}'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on bedroom_environment/sensor/bedroom_gas/state: b'nan'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on homeassistant/sensor/bedroom_environment/bedroom_humidity/config: b'{"unit_of_measurement":"%","expire_after":180,"icon":"mdi:water-percent","name":"Bedroom Humidity","state_topic":"bedroom_environment/sensor/bedroom_humidity/state","availability_topic":"bedroom_environment/status"}'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on bedroom_environment/sensor/bedroom_humidity/state: b'nan'
2018-11-05 18:27:47 INFO (MainThread) [homeassistant.components.mqtt.discovery] Component has already been discovered: sensor bedroom_environment_bedroom_gas, sending update
2018-11-05 18:27:47 INFO (MainThread) [homeassistant.components.mqtt.discovery] Component has already been discovered: sensor bedroom_environment_bedroom_humidity, sending update
2018-11-05 18:27:47 INFO (MainThread) [homeassistant.components.mqtt] Got update for entity with hash: ('sensor', 'bedroom_environment_bedroom_gas') '{'unit_of_measurement': 'Ω', 'expire_after': 180, 'icon': 'mdi:gas-cylinder', 'name': 'Bedroom Gas', 'state_topic': 'bedroom_environment/sensor/bedroom_gas/state', 'availability_topic': 'bedroom_environment/status'}'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on homeassistant/sensor/bedroom_environment/bedroom_pressure/config: b'{"unit_of_measurement":"hPa","expire_after":180,"icon":"mdi:gauge","name":"Bedroom Pressure","state_topic":"bedroom_environment/sensor/bedroom_pressure/state","availability_topic":"bedroom_environment/status"}'
2018-11-05 18:27:47 INFO (MainThread) [homeassistant.components.mqtt] Got update for entity with hash: ('sensor', 'bedroom_environment_bedroom_humidity') '{'unit_of_measurement': '%', 'expire_after': 180, 'icon': 'mdi:water-percent', 'name': 'Bedroom Humidity', 'state_topic': 'bedroom_environment/sensor/bedroom_humidity/state', 'availability_topic': 'bedroom_environment/status'}'
2018-11-05 18:27:47 INFO (MainThread) [homeassistant.components.mqtt.discovery] Component has already been discovered: sensor bedroom_environment_bedroom_pressure, sending update
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on bedroom_environment/sensor/bedroom_pressure/state: b'nan'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on homeassistant/sensor/bedroom_environment/bedroom_temperature/config: b'{"unit_of_measurement":"\xc2\xb0C","expire_after":180,"name":"Bedroom Temperature","state_topic":"bedroom_environment/sensor/bedroom_temperature/state","availability_topic":"bedroom_environment/status"}'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on bedroom_environment/sensor/bedroom_temperature/state: b'nan'
2018-11-05 18:27:47 INFO (MainThread) [homeassistant.components.mqtt] Got update for entity with hash: ('sensor', 'bedroom_environment_bedroom_pressure') '{'unit_of_measurement': 'hPa', 'expire_after': 180, 'icon': 'mdi:gauge', 'name': 'Bedroom Pressure', 'state_topic': 'bedroom_environment/sensor/bedroom_pressure/state', 'availability_topic': 'bedroom_environment/status'}'
2018-11-05 18:27:47 INFO (MainThread) [homeassistant.components.mqtt.discovery] Component has already been discovered: sensor bedroom_environment_bedroom_temperature, sending update
2018-11-05 18:27:47 INFO (MainThread) [homeassistant.components.mqtt] Got update for entity with hash: ('sensor', 'bedroom_environment_bedroom_temperature') '{'unit_of_measurement': '°C', 'expire_after': 180, 'name': 'Bedroom Temperature', 'state_topic': 'bedroom_environment/sensor/bedroom_temperature/state', 'availability_topic': 'bedroom_environment/status'}'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on bedroom_environment/sensor/bedroom_temperature/state: b'21.2'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on bedroom_environment/sensor/bedroom_pressure/state: b'671.4'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on bedroom_environment/sensor/bedroom_humidity/state: b'54.2'
2018-11-05 18:27:47 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on bedroom_environment/sensor/bedroom_gas/state: b'138087.0'

Additional information:
I configured a ESP32 with an BME680 to send all sensor values via MQTT to HA. The auto discovery seems to work, but HA don't display any of the values in the ui. In the log you can see that all values are gathered, but nothing more!

Edit: I'm using an external mosquitto MQTT broker.

Same issue here

What do you see under the MQTT topic bedroom_environment/status ?
unavailable means the ESP is not connected to the MQTT broker and the availability reporting fails.

Same issue here. The "status" topic for the sensor show it's "online" but after 30" the sensor on Home Assistant is shown as "unknown"

Is this still an issue you are experiencing? Can you please try upgrading to the latest version of Home Assistant (0.90) and report back if this is still a problem? Thanks!


Enhance IPAddress Model to use get_or_create method with namespace attribute

As ...
Austin - Network Automation Engineer
I want ...
to be able to use the IPAddress.objects.get_or_create() method with specifying a namespace,
So that ...
I can allow the ORM to use the IPAddress._get_closest_parent() method for a namespace other than the default.
I know this is done when...
I can use the following syntax: IPAddress.objects.get_or_create(host="<IP_ADDRESS>", mask_length=24, namespace=, status=)
Optional - Feature groups this request pertains to.

[ ] Automation
[ ] Circuits
[ ] DCIM
[X] IPAM
[ ] Misc (including Data Sources)
[ ] Organization
[ ] Apps (and other Extensibility)
[ ] Security (Secrets, etc)
[ ] Image Management
[ ] UI/UX
[ ] Documentation
[ ] Other (not directly a platform feature)

Database Changes
This would be possible outside of making changes to the database
External Dependencies
none

A current workaround is:
closest_parent = Prefix.objects.get_closest_parent(cidr=f"{ip_address}/{subnet_mask}", include_self=True)
ip_obj, created = IPAddress.objects.get_or_create(host=ip_address, mask_length=cidr, parent=closest_parent, defaults=defaults)


Fixed deeply dynamic model support

This fixed the case when the model dynamically changes in the middle of the chain - for example, you were using deep Fieldset hierarchy, and any of them had dynamic model
The problem is caused by the fact that resolveModel decorator, that is used internally to allow partial models, was caching model value upon mounting, thus preventing it from updating properly.
I've also removed several performance optimizations, as they were also preventing context updates. I'm open to any way this could be done better.

I'll have to see how performance is affected with this. Can you elaborate more in how exactly the performance optimizations were preventing context updates? Maybe we can work around it without sacrifices.

The intermediate component you've created, DefaultConnectedControl, does not know anything about contexts that are used by resolveModel. So we can't check if it is updated, and restrictive shouldComponentUpdate implementation would block the context propagation, leaving Controls with an old model value.
Some kind of related discussion can be found here: https://github.com/facebook/react/issues/2517.
I can declare contextTypes on DefaultConnectedContol though, and then check if they've updated in shouldComponentUpdate. But it seems hacky for thic component to know anything about context props used in resolveModel.

I can declare contextTypes on DefaultConnectedContol though, and then check if they've updated in shouldComponentUpdate. But it seems hacky for thic component to know anything about context props used in resolveModel.

That might be an idea. The reason for resolveModel is that React-Redux's connect() doesn't know anything about context, unfortunately. If it did, that would fix the issue, as we can do resolved model-checking logic in there.

Looks good, thanks for adding the sCU logic back in!


Format + Clippy!

Finally! Closes #13.

I think we should leave this until the next development -> master merge. It'll cause a lot of merge conflicts for the active branches that aren't really worth it. What do you think?

I mean... I want this tbh.

This is great though, been needed for a while. :P
Definitely try merge ASAP.


Update flow-plugin-base, flow-server to 24.3.5

About this PR
📦 Updates

com.vaadin:flow-plugin-base
com.vaadin:flow-server

from 23.3.29 to 24.3.5 ⚠
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.vaadin" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.vaadin" }
}]

labels: library-update, early-semver-major, semver-spec-major, commit-count:1

Superseded by #104.


Add solution notebooks for day2 lessons

Thank you so much Juan for an amazing session! I learned a lot and it looks like most of the participants did, too!


[$500] Search - "Please enter a valid phone number..." while still entering phone number

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.3.92-0
**Reproducible in staging?:**Y
**Reproducible in production?:**Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Open New Dot app
Click on the green + button
Click on the "Send message" option
Start typing valid phone number

Expected Result:
There should be no error message while adding numbers
Actual Result:
"Please enter a valid phone number..." message appear while still adding numbers, but when total numbers are added - number is displayed and message is dismissed
Please enter a valid phone number without brackets or dashes. If you're outside the US please include your country code (e.g. +15005550006)
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

https://github.com/Expensify/App/assets/78819774/3465898f-c6fa-456c-8e71-151e0a3f735b

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01cd3cad0bf6f078d5
Upwork Job ID:<PHONE_NUMBER>179428352
Last Price Increase: 2023-10-27

Hello! I saw this add on Upwork.
The issue is that the input box is performing a validation check as soon as it has been interacted with (most likely an onClick handler or using the pattern HTML attribute).
I would fix this by moving whatever functionality that checks for phone number validation to be performed only after 1) the input form is considered "dirty" - meaning it has had content added to it, and 2) when the user has navigated away.
Thank you for your consideration!

Proposal
Please re-state the problem that we are trying to solve in this issue.
Invalid phone number messaging is shown prematurely - almost immediately after a single digit is entered.
What is the root cause of that problem?
The search field has two validation modes, one for values that may be a phone number, and one for values that are clearly not (eg., they contain letters). As long as a value contains only numbers and certain characters such as - and +, the value is immediately tested for phone number validation. Side note: this seems to be by design.
What changes do you think we should make in order to solve the problem?
You'll need to define what qualifies as an invalid phone number. I would limit phone number validation to:

strings with 6 - 12 numbers (within reason for typos)
optionally including any number of the characters -, ., or /
optional leading +

Strings that don't match won't be validated as phone numbers and will lead to "no result" if there's no search match, just as other values currently do.
What alternative solutions did you explore? (Optional)
Honestly? Leaving it as is. We're only guessing the user's intent when we validate for a phone number, validating any numbers-only(ish) string as we currently do doesn't feel completely invalid. But the proposed solution will help reduce the false negatives.
Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.

Proposal
from: @cooldev900
Please re-state the problem that we are trying to solve in this issue.
"Please enter a valid phone number..." while still entering phone number.
What is the root cause of that problem?
The root cause is because we didn't set displayName in participant so that displayName is set to payerEmail in optimisticPersonalDetailListAction.
https://github.com/Expensify/App/blob/cb5aeb14f6dfbdcd4b4e831d55fb17dc678e093f/src/libs/OptionsListUtils.js#L1543-L1545
https://github.com/Expensify/App/blob/cb5aeb14f6dfbdcd4b4e831d55fb17dc678e093f/src/CONST.ts#L1289
What changes do you think we should make in order to solve the problem?
You  need more validation including phoneNumer length with this condition as well.
What alternative solutions did you explore? (Optional)
N/A

Proposal
Please re-state the problem that we are trying to solve in this issue.
When searching for a phone number, the UI will display a message saying the telephone number is invalid while user is still typing.
What is the root cause of that problem?
    const isValidPhone = parsePhoneNumber(LoginUtils.appendCountryCode(searchValue)).possible;

    const isValidEmail = Str.isValidEmail(searchValue);

    if (searchValue && CONST.REGEX.DIGITS_AND_PLUS.test(searchValue) && !isValidPhone && !hasSelectableOptions) {
        return Localize.translate(preferredLocale, 'messages.errorMessageInvalidPhone');
    }

On src/libs/OptionsListUtils.js:1619 we check if the input contains only numbers and if so, and its not a valid phone number and no results returned, we assume its an invalid phone number and display the error message. This doesn't account for users that are still typing, resulting in a bad experience.
What changes do you think we should make in order to solve the problem?
The search field can take either a Name, Email or Telephone and it's really tricky to determine the intent. Also, it's not a great experience to display an error while users are still typing it. For example, a partially typed email will result in Invalid email which is the same issue as with telephone numbers. Because this is a real time search, my proposal is to get rid of the validation error messages as I don't believe they work well in this context. We'll search by anything the user types, and if we get no result, we show no results found. We don't try to validate what user is typing.
Additionally, we could make improvements to the error message. We can make certain assumptions based on the input and give suggestions, not errors. For example:
A search string with only numbers and + can result in:
No results found. Please ensure that you entered a valid phone number. Use only numbers without brackets or dashes. If you're outside the US please include your country code (eg: +15005550006).
A search string with @ sign can result in:
No results found. Please ensure that you entered a valid email address.
A search string that doesn't match any of the above result in:
No results found.
The messages are just an example and I'm open to suggestions here.
The idea is to ALWAYS say No results found as to tell the user We did search, but could not find anything and then give an educated suggestions based on what they entered. Sometimes user might miss something and the suggestion will guide them to fix, but without saying straight up Your input is invalid.

What alternative solutions did you explore? (Optional)
Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.

Proposal
Please re-state the problem that we are trying to solve in this issue.
The UI immediately displays a validation error message when the user starts entering a phone number in the "Start Chat" sidebar on the /new page.
What is the root cause of that problem?
The root cause is not in how the field is validated, but when it gets validated. We are validating immediately as we type because we update searchTerm on change.
// When search term updates we will fetch any reports
const setSearchTermAndSearchInServer = useCallback((text = '') => {
  if (text.length) {
    Report.searchInServer(text);
  }
  setSearchTerm(text);
}, []);

What changes do you think we should make in order to solve the problem?
Add a debounce function to avoid altering searchTerm while the user is typing.
The new onTextChange handler will do two things:

Update an internal state that is connected to the text input (so that users see what they type)
Call the debounced version of setSearchTermAndSearchInServer

This will ensure that the validation will not start until the user stops typing for a preset amount of time (maybe 300ms). It will also delay the search till the user stops typing, which makes sense and is a classic use-case for debounce (fewer network calls).
What alternative solutions did you explore? (Optional)
I considered altering the validation logic to wait until a minimum number of digits is provided, but that doesn't work because the textfield accepts emails as well as phone numbers.
For example, the email<EMAIL_ADDRESS>is a valid email format. So if we alter the validation logic to wait until we have 5 characters or so, we risk having the "Invalid phone number" message showing up even though the user is trying to provide an email.

Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.

Contributor details
Your Expensify account email<EMAIL_ADDRESS>Upwork Profile Link: https://www.upwork.com/freelancers/~018158153edee56310

@rushatgabhane let's get these proposals reviewed! 🙏

I agree let's just close


Resolve deploy:local not connecting to local anvil

Resolves #58

updates .env in each project to point to the Anvil instance running in Docker on port 8546 to match mprocs behaviour.
adds a Foundry profile in foundry.toml for local which includes the anvil RPC on port 8546.
amends deploy:local to use the local profile from foundry.toml.

Additionally:

rationalises .gitignore for each project so it is self contained, i.e. you can copy the folder out and the ignores still work in a different git repository.
removes files that should be ignored to keep commits clean.

Testing
Tested all three of the projects (smart-gate, smart-storage-unit and smart-turret) with both pnpm@latest-8 as per the docs and pnpm@latest which CCP seem to use internally.

For clarity, while the initial problem has been solved by the above there is a separate issue with pnpm mock-data:
Error: Failure on receiving a receipt for 0xf2ec7ad2db25931708d5818c6d98b69dea6a81316137c4cd20e30d3b98d1b839:
deserialization error: duplicate field `status` at line 1 column 5885
Failure on receiving a receipt for 0x34157a2760599a8139caee6fb9be1529006ee1c49e61c34edf8a0157e0f02e5d:
deserialization error: duplicate field `status` at line 1 column 5693
Failure on receiving a receipt for 0x993be6c3f9892c62facef19144425f0cd817af30c7b6cfdf2e3e44871219de5c:
deserialization error: duplicate field `status` at line 1 column 4861
Failure on receiving a receipt for 0x840d8c11ccea8c64e0a537731c78e649fff51cb2eb45110977d7b3df01bcf981:
deserialization error: duplicate field `status` at line 1 column 8641
Failure on receiving a receipt for 0xdd4f50bf5f7c5e305b911e99386f91bc06b86146f1ee00521a608826fe982683:
deserialization error: duplicate field `status` at line 1 column 11362
Failure on receiving a receipt for 0x30125ae0b84928de2aa9a6eb371d1cf0542b3684777b1147c667c877a47d88d4:
deserialization error: duplicate field `status` at line 1 column 20641

Great work! @Scetrov. All looks good and have approved the PR.


PHOENIX-6518 Implement SHOW CREATE STATEMENT SQL command

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 28s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
5m 21s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
17m 39s
master passed

+0
hbaserecompile
29m 52s
HBase recompiled.

+1 :green_heart:
compile
1m 35s
master passed

+1 :green_heart:
checkstyle
2m  3s
master passed

+1 :green_heart:
javadoc
1m  9s
master passed

+0 :ok:
spotbugs
3m  3s
phoenix-core in master has 964 extant spotbugs warnings.

+0 :ok:
spotbugs
0m 43s
phoenix-tools in master has 5 extant spotbugs warnings.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 11s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
13m 38s
the patch passed

+0
hbaserecompile
19m 59s
HBase recompiled.

+1 :green_heart:
compile
1m 34s
the patch passed

+1 :green_heart:
javac
1m 34s
the patch passed

-1 :x:
checkstyle
1m 49s
phoenix-core: The patch generated 428 new + 2618 unchanged - 20 fixed = 3046 total (was 2638)

+1 :green_heart:
checkstyle
0m 16s
phoenix-tools: The patch generated 0 new + 60 unchanged - 361 fixed = 60 total (was 421)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
javadoc
1m  9s
the patch passed

-1 :x:
spotbugs
3m 19s
phoenix-core generated 6 new + 964 unchanged - 0 fixed = 970 total (was 964)

+1 :green_heart:
spotbugs
0m 54s
phoenix-tools generated 0 new + 3 unchanged - 2 fixed = 3 total (was 5)

_ Other Tests _

-1 :x:
unit
111m 54s
phoenix-core in the patch failed.

-1 :x:
unit
3m 41s
phoenix-tools in the patch failed.

+1 :green_heart:
asflicense
1m 24s
The patch does not generate ASF License warnings.

188m 43s

Reason
Tests

FindBugs
module:phoenix-core

Found reliance on default encoding in org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String):in org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String): new java.io.FileReader(File)  At SchemaSynthesisProcessor.java:[line 211]

Null passed for non-null parameter of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:[line 80]

Null passed for non-null parameter of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:[line 81]

org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String) may fail to close stream  At SchemaSynthesisProcessor.java:stream  At SchemaSynthesisProcessor.java:[line 211]

Invocation of toString on Throwable.getStackTrace() in org.apache.phoenix.util.QueryUtil.getShowCreateStatementStmt(PhoenixConnection, String, TableName)  At QueryUtil.java:in org.apache.phoenix.util.QueryUtil.getShowCreateStatementStmt(PhoenixConnection, String, TableName)  At QueryUtil.java:[line 788]

org.apache.phoenix.util.QueryUtil.getShowCreateStatementStmt(PhoenixConnection, String, TableName) may fail to clean up java.sql.Statement on checked exception  Obligation to clean up resource created at QueryUtil.java:to clean up java.sql.Statement on checked exception  Obligation to clean up resource created at QueryUtil.java:[line 804] is not discharged

Failed junit tests
phoenix.end2end.BackwardCompatibilityIT

phoenix.schema.SchemaToolExtractionIT

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/phoenix/pull/1271

Optional Tests
dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile xml

uname
Linux 36dc51b6ef0f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev/phoenix-personality.sh

git revision
master / 6842c1d

Default Java
Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08

checkstyle
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt

spotbugs
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/1/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html

unit
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt

unit
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/1/artifact/yetus-general-check/output/patch-unit-phoenix-tools.txt

Test Results
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/1/testReport/

Max. process+thread count
14695 (vs. ulimit of 30000)

modules
C: phoenix-core phoenix-tools U: .

Console output
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/1/console

versions
git=2.7.4 maven=3.3.9 spotbugs=4.1.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 28s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
5m  9s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
17m 19s
master passed

+0
hbaserecompile
30m 16s
HBase recompiled.

+1 :green_heart:
compile
1m 32s
master passed

+1 :green_heart:
checkstyle
2m  3s
master passed

+1 :green_heart:
javadoc
1m 12s
master passed

+0 :ok:
spotbugs
3m  6s
phoenix-core in master has 964 extant spotbugs warnings.

+0 :ok:
spotbugs
0m 44s
phoenix-tools in master has 5 extant spotbugs warnings.

-0 :warning:
patch
0m 59s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 11s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
13m 43s
the patch passed

+0
hbaserecompile
20m 41s
HBase recompiled.

+1 :green_heart:
compile
1m 33s
the patch passed

+1 :green_heart:
javac
1m 33s
the patch passed

-1 :x:
checkstyle
1m 49s
phoenix-core: The patch generated 428 new + 2618 unchanged - 20 fixed = 3046 total (was 2638)

+1 :green_heart:
checkstyle
0m 16s
phoenix-tools: The patch generated 0 new + 60 unchanged - 361 fixed = 60 total (was 421)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
javadoc
1m 10s
the patch passed

-1 :x:
spotbugs
3m 22s
phoenix-core generated 6 new + 964 unchanged - 0 fixed = 970 total (was 964)

+1 :green_heart:
spotbugs
0m 54s
phoenix-tools generated 0 new + 3 unchanged - 2 fixed = 3 total (was 5)

_ Other Tests _

-1 :x:
unit
113m 11s
phoenix-core in the patch failed.

-1 :x:
unit
3m 36s
phoenix-tools in the patch failed.

+1 :green_heart:
asflicense
1m 22s
The patch does not generate ASF License warnings.

191m  3s

Reason
Tests

FindBugs
module:phoenix-core

Found reliance on default encoding in org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String):in org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String): new java.io.FileReader(File)  At SchemaSynthesisProcessor.java:[line 211]

Null passed for non-null parameter of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:[line 80]

Null passed for non-null parameter of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:[line 81]

org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String) may fail to close stream  At SchemaSynthesisProcessor.java:stream  At SchemaSynthesisProcessor.java:[line 211]

Invocation of toString on Throwable.getStackTrace() in org.apache.phoenix.util.QueryUtil.getShowCreateStatementStmt(PhoenixConnection, String, TableName)  At QueryUtil.java:in org.apache.phoenix.util.QueryUtil.getShowCreateStatementStmt(PhoenixConnection, String, TableName)  At QueryUtil.java:[line 788]

org.apache.phoenix.util.QueryUtil.getShowCreateStatementStmt(PhoenixConnection, String, TableName) may fail to clean up java.sql.Statement on checked exception  Obligation to clean up resource created at QueryUtil.java:to clean up java.sql.Statement on checked exception  Obligation to clean up resource created at QueryUtil.java:[line 794] is not discharged

Failed junit tests
phoenix.end2end.PermissionNSEnabledWithCustomAccessControllerIT

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/phoenix/pull/1271

Optional Tests
dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile xml

uname
Linux c9b4b2252172 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev/phoenix-personality.sh

git revision
master / 6842c1d

Default Java
Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08

checkstyle
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt

spotbugs
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/2/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html

unit
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt

unit
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/2/artifact/yetus-general-check/output/patch-unit-phoenix-tools.txt

Test Results
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/2/testReport/

Max. process+thread count
15739 (vs. ulimit of 30000)

modules
C: phoenix-core phoenix-tools U: .

Console output
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/2/console

versions
git=2.7.4 maven=3.3.9 spotbugs=4.1.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 58s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
4m 58s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
21m 55s
master passed

+0
hbaserecompile
35m 21s
HBase recompiled.

+1 :green_heart:
compile
1m 56s
master passed

+1 :green_heart:
checkstyle
1m 50s
master passed

+1 :green_heart:
javadoc
1m 20s
master passed

+0 :ok:
spotbugs
3m 56s
phoenix-core in master has 964 extant spotbugs warnings.

+0 :ok:
spotbugs
0m 50s
phoenix-tools in master has 5 extant spotbugs warnings.

-0 :warning:
patch
1m  7s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
17m 27s
the patch passed

+0
hbaserecompile
24m 17s
HBase recompiled.

+1 :green_heart:
compile
1m 38s
the patch passed

+1 :green_heart:
javac
1m 38s
the patch passed

-1 :x:
checkstyle
1m 34s
phoenix-core: The patch generated 408 new + 2638 unchanged - 0 fixed = 3046 total (was 2638)

-1 :x:
checkstyle
0m 16s
phoenix-tools: The patch generated 3 new + 57 unchanged - 364 fixed = 60 total (was 421)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
javadoc
1m  9s
the patch passed

-1 :x:
spotbugs
3m 23s
phoenix-core generated 6 new + 964 unchanged - 0 fixed = 970 total (was 964)

+1 :green_heart:
spotbugs
0m 52s
phoenix-tools generated 0 new + 3 unchanged - 2 fixed = 3 total (was 5)

_ Other Tests _

+1 :green_heart:
unit
116m 53s
phoenix-core in the patch passed.

-1 :x:
unit
3m 23s
phoenix-tools in the patch failed.

+1 :green_heart:
asflicense
0m 28s
The patch does not generate ASF License warnings.

202m 12s

Reason
Tests

FindBugs
module:phoenix-core

Found reliance on default encoding in org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String):in org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String): new java.io.FileReader(File)  At SchemaSynthesisProcessor.java:[line 211]

Null passed for non-null parameter of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:[line 80]

Null passed for non-null parameter of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:[line 81]

org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String) may fail to close stream  At SchemaSynthesisProcessor.java:stream  At SchemaSynthesisProcessor.java:[line 211]

Invocation of toString on Throwable.getStackTrace() in org.apache.phoenix.util.QueryUtil.getShowCreateStatementStmt(PhoenixConnection, String, TableName)  At QueryUtil.java:in org.apache.phoenix.util.QueryUtil.getShowCreateStatementStmt(PhoenixConnection, String, TableName)  At QueryUtil.java:[line 788]

org.apache.phoenix.util.QueryUtil.getShowCreateStatementStmt(PhoenixConnection, String, TableName) may fail to clean up java.sql.Statement on checked exception  Obligation to clean up resource created at QueryUtil.java:to clean up java.sql.Statement on checked exception  Obligation to clean up resource created at QueryUtil.java:[line 794] is not discharged

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/3/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/phoenix/pull/1271

Optional Tests
dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile xml

uname
Linux 4a5cb0890046 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev/phoenix-personality.sh

git revision
master / 9b76bd5

Default Java
Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08

checkstyle
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt

checkstyle
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix-tools.txt

spotbugs
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/3/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html

unit
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/3/artifact/yetus-general-check/output/patch-unit-phoenix-tools.txt

Test Results
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/3/testReport/

Max. process+thread count
14392 (vs. ulimit of 30000)

modules
C: phoenix-core phoenix-tools U: .

Console output
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/3/console

versions
git=2.7.4 maven=3.3.9 spotbugs=4.1.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 32s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
4m 54s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
19m  4s
master passed

+0
hbaserecompile
32m  4s
HBase recompiled.

+1 :green_heart:
compile
1m 44s
master passed

+1 :green_heart:
checkstyle
2m  4s
master passed

+1 :green_heart:
javadoc
1m 12s
master passed

+0 :ok:
spotbugs
3m 23s
phoenix-core in master has 964 extant spotbugs warnings.

+0 :ok:
spotbugs
0m 47s
phoenix-tools in master has 5 extant spotbugs warnings.

-0 :warning:
patch
1m  3s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 13s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
15m 10s
the patch passed

+0
hbaserecompile
21m 52s
HBase recompiled.

+1 :green_heart:
compile
1m 41s
the patch passed

+1 :green_heart:
javac
1m 41s
the patch passed

-1 :x:
checkstyle
1m 49s
phoenix-core: The patch generated 427 new + 2618 unchanged - 20 fixed = 3045 total (was 2638)

-1 :x:
checkstyle
0m 18s
phoenix-tools: The patch generated 3 new + 57 unchanged - 364 fixed = 60 total (was 421)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
javadoc
1m 11s
the patch passed

-1 :x:
spotbugs
3m 32s
phoenix-core generated 6 new + 964 unchanged - 0 fixed = 970 total (was 964)

+1 :green_heart:
spotbugs
0m 56s
phoenix-tools generated 0 new + 3 unchanged - 2 fixed = 3 total (was 5)

_ Other Tests _

-1 :x:
unit
115m 14s
phoenix-core in the patch failed.

-1 :x:
unit
3m 49s
phoenix-tools in the patch failed.

+1 :green_heart:
asflicense
1m 20s
The patch does not generate ASF License warnings.

198m 27s

Reason
Tests

FindBugs
module:phoenix-core

Found reliance on default encoding in org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String):in org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String): new java.io.FileReader(File)  At SchemaSynthesisProcessor.java:[line 211]

Null passed for non-null parameter of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:[line 80]

Null passed for non-null parameter of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:of new org.apache.phoenix.parse.SQLParser(String) in org.apache.phoenix.schematool.SchemaSynthesisProcessor.synthesize(String, String)  Method invoked at SchemaSynthesisProcessor.java:[line 81]

org.apache.phoenix.schematool.SchemaSynthesisProcessor.getQueriesFromFile(String) may fail to close stream  At SchemaSynthesisProcessor.java:stream  At SchemaSynthesisProcessor.java:[line 211]

Invocation of toString on Throwable.getStackTrace() in org.apache.phoenix.util.QueryUtil.getShowCreateTableStmt(PhoenixConnection, String, TableName)  At QueryUtil.java:in org.apache.phoenix.util.QueryUtil.getShowCreateTableStmt(PhoenixConnection, String, TableName)  At QueryUtil.java:[line 788]

org.apache.phoenix.util.QueryUtil.getShowCreateTableStmt(PhoenixConnection, String, TableName) may fail to clean up java.sql.Statement on checked exception  Obligation to clean up resource created at QueryUtil.java:to clean up java.sql.Statement on checked exception  Obligation to clean up resource created at QueryUtil.java:[line 794] is not discharged

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/4/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/phoenix/pull/1271

Optional Tests
dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile xml

uname
Linux 85ac32b8f57d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev/phoenix-personality.sh

git revision
master / 9b76bd5

Default Java
Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08

checkstyle
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt

checkstyle
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix-tools.txt

spotbugs
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/4/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html

unit
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/4/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt

unit
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/4/artifact/yetus-general-check/output/patch-unit-phoenix-tools.txt

Test Results
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/4/testReport/

Max. process+thread count
10509 (vs. ulimit of 30000)

modules
C: phoenix-core phoenix-tools U: .

Console output
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1271/4/console

versions
git=2.7.4 maven=3.3.9 spotbugs=4.1.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Don't forget to change the statement name in the ticket and commit message as well.


Personal Preferences

Move this to V3

Personal Preferences are stored in our V3 database

Same as conversation history ie you need to sync from another device that has the personal preferences.
You also need to sync all changes from that point forward. You need to relay that change from that point forward. A persistant identity group chat. Label it as group metadata for those that sync later.

Potentially breaking: Mutable metadata on the sync group? The way we do sync groups currently is not compatible with this. Which might require us to support external joins. State snapshot as part of the message history sync.

~~ The same as V2 ~~
Anytime a user signs in on a new device we ask them for a wallet signature. We give them the same payload to sign every time. Derive an encryption key from those bytes. A secret key that everyone of your devices can get. As long as they have access to your wallet. Wallet apis dont give us anything else but the ability to sign. Smart wallets don't have signatures over time and you might not have access to all wallets on each device.
UX burden to request another signature from a user
Deriving a encryption key from signature bytes is not standard

Make the user transport a secret between devices.
They set a passcode or we give them a passcode. They have to remember this passcode on each login. When the sign in on a new device it pulls that file. A lot of UX burden.  But not enough to get the full secret. Requires the backends to collide together. We would need to re implement the algorithm. We would need to store the keys.

Edge case
If you loose your device create a new account entirely.
This is only a PPPP issue since conversations will still get messages and would show up eventually.
We could let a split brain happen and they would merge eventually. Once you sign in and sync your consent the broadcast that to the new group. Whenever you log into another device with an inconstant state it can broadcast. Consent state is written in group metadate in one group just for consent.
We store the consent state and do a join table for them.
We have a group that is only your self and it consists of all of your installation. There needs to be mechanism to catch up a device that is behind. You can tell the device is behind because it sent a message history request that nobody responded to. Still requires a device online to catch up the new device.
We don't wipe we let you continue without the existing consent state.
If you don't sync everything will be in pending. When you come online from an old device then it will fix everything. Conflict resolution.
In order to get your history you need to have your other device.
Let the converse app do backups for you to the iCloud/GoogleCloud.
Work to be done

A database of personal preference records stored in V3 with a join table on groups
Back up and restore of the full state when the snapshot was requested. Upload a file to a server and have the other device download the file
Broadcasting updates find the most recent sync group find the most recent sync group and send a message on that group communicating that action
Local state machine of reading those updates and updating your local state
Catching up the state if time has passed between device online
Duel sending of V2 and V3 consent for a period of time
Migration to port V2 consent into V3 database

[ ] Write an XIP
[ ] A new table for personal preference records in the local db
[ ] Use sync group for sending preference snapshot (confirm server is not needed)

[ ] Broadcasting updates find the most recent sync group find the most recent sync group and send a message on that group communicating that action
[ ] Local state machine of reading those updates and updating your local state

[ ] Duel sending of V2 and V3 consent for a period of time

[ ] Migration to port V2 consent into V3 database

[ ] Update the sync to handle receiving a request to get this message sent.
[ ] Proto for this new message type so that when received we know what to do with it.

[ ] process_external_message new type

[ ] Add the code to receive this message and update the local database with it.

[ ] Serialize and Deserialize from the database file so we can send it in a message

Client 1
Sending the request to send the consentMessage (sends a new consent message type proto to the sync group) -- should happen anytime a new installation is created by default but can be disabled.
Receiving the consentMessage (via a stream versus via sync NEED TO VALIDATE)
Deserializing the consentMessage
Client 2
Receiving the request to send the consentMessage (in the sync group)
Serializing and sending the consentMessage (sends to the sync group)
There should only be 1 sync group. Look for the sync group if we can't find it create it.
Manually post the current state of the group sync. Embed it into a message. 1mb is the max size of a message. Compress the string. 600,000 wallets. Set an option to not have implicit consent. Allowing people to be able to download a file that's encrypted. Also add a button to export and import. Deletes backup every week. No history url because it's just a message.


Learning rate & its scheduling

I cannot find the specific value of learning rate and how the author schedule to change the learning rate over epochs.
How do you implement and reproduce the results in the paper?

To be honest, I was also a bit disappointed that there were no implementation details given by the authors. So for, I did not do any hyperparameter tuning and just set the learning rate to be the same as used in the original VQGAN paper for the transformer part. That also means that I didn't experiment with any lr-schedules so feel free to experiment with it I would say. And for the last question, I did not reproduce the results yet and they are still sitting on the TODO list. For now all I did was implementing the Bidirectional Transformer into the VQGAN code. Furthermore, the authors refered to two papers when talking about the Bidirectional Transformer which might lead to some serious performance improvement. I simply took my own standard implementation of BERT. So again, feel free to do some optimization there.

Thanks for the reply. I agree with you that the paper does not include the implementation details.