Connection error and Unable to change port for external postgres connection using helm

My actions before raising this issue

[x] Read/searched the docs
[x] Searched past issues

Expected Behaviour
Connect to enterprise postgres db by enabling external flag in helm chart.
Current Behaviour
Running into connection error  while creating super user. And also the port is not the default one : 5432 but 2108.
System check identified some issues:

WARNINGS:
?: (urls.W005) URL namespace 'v1' isn't unique. You may not be able to reverse all URLs in this namespace
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/opt/venv/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name "pogresdb.xyc.com" to address: Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/venv/lib/python3.8/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 79, in execute
    return super().execute(*args, **options)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 370, in execute
    self.check_migrations()
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 459, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "/opt/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/opt/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 53, in __init__
    self.build_graph()
  File "/opt/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 216, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/opt/venv/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 77, in applied_migrations
    if self.has_table():
  File "/opt/venv/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 55, in has_table
    with self.connection.cursor() as cursor:
  File "/opt/venv/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/opt/venv/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/opt/venv/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not translate host name "pogresdb.xyc.com" to address: Name or service not known

Possible Solution

Steps to Reproduce (for bugs)
updated below values in Values.yaml
postgresql:
  #See https://github.com/bitnami/charts/blob/master/bitnami/postgresql/ for more info
  enabled: false # false for external db
  external:
    host: pogresdb.xyc.com
    port: 2108
    user: postgres
    password: postgres
    dbname: cvat

Context

Your Environment

Git hash commit (git log -1):
Docker version docker version (e.g. Docker 17.0.05):
Are you using Docker Swarm or Kubernetes?
Operating System and version (e.g. Linux, Windows, MacOS):
Code example or link to GitHub repo or gist to reproduce problem:
Other diagnostic information / logs:
Logs from `cvat` container

Next steps
You may join our Gitter channel for community support.

could not translate host name "pogresdb.xyc.com" to address: Name or service not known
problem on k8s side


I get

I used to get a white activity bar foreground.

@kieferrm you are in a light theme and as such the icons of the activity bar flip to the light theme variant. You will get the other icons when you use a dark theme to begin with.
I think this issue is more generally asking to control the color of icons when theming is applied, so I have renamed it.

@bpasero You can use the css fillproperty to set the svg color. So, if you set fill: currentColor svg icons will have the same color of the foreground (if is set to the parent and icons are inside it). I don't know how is handled, but in css is:
.ActivityBar {
 color: #FFFF;
}

.ActivityBarText {
  color: currentColor;
}

.ActivityBarIcons {
  fill: currentColor;
}

So by changing one property you can change all color/fill attribute.

@equinusocio as far as I know this trick is only possible if SVG icons are part of the document. In our case they are coming in via css background-image property where your trick does not work.

@bpasero Ok, you can use the mask property, check my pen here
https://codepen.io/equinusocio/pen/XRjdzO?editors=1100

@equinusocio yup, we are using mask already in the status bar, thanks.

Nice!

I added activityIconForeground as color for activity bar icons.


Lame is not defined

While integrating inside the angular Application, it is showing like this,
index.js:15500 Uncaught ReferenceError: Lame is not defined
at index.js:15500
at index.js:2
at Object../node_modules/mic-recorder-to-mp3/dist/index.js (index.js:5)
at webpack_require (bootstrap:79)
at Module../src/app/robotic/robotic.component.ts (main.js:592)
at webpack_require (bootstrap:79)
at Module../src/app/app-routing.module.ts (main.js:106)
at webpack_require (bootstrap:79)
at Module../src/app/app.module.ts (app.component.ts:8)
at webpack_require (bootstrap:79)
(anonymous) @ index.js:15500
(anonymous) @ index.js:2
./node_modules/mic-recorder-to-mp3/dist/index.js @ index.js:5
webpack_require @ bootstrap:79
./src/app/robotic/robotic.component.ts @ main.js:592
webpack_require @ bootstrap:79
./src/app/app-routing.module.ts @ main.js:106
webpack_require @ bootstrap:79
./src/app/app.module.ts @ app.component.ts:8
webpack_require @ bootstrap:79
./src/main.ts @ main.ts:1
webpack_require @ bootstrap:79
0 @ main.ts:12
webpack_require @ bootstrap:79
checkDeferredModules @ bootstrap:45
webpackJsonpCallback @ bootstrap:32
(anonymous) @ main.js:1
client:52 [WDS] Live Reloading enabled.

I have the same problem. Did you find a solution to it ?

If you share a reproducible issue, I may try to fix it.

I have the same problem. Did you find a solution to it ?

I just added source for lame directly to the html file. i know its not a good fix.

@manu-ust Can you share a minimal reproducible CodeSandbox?

I'll close this issue for inactivity – will reopen in case someone shares a minimal reproducible repo/sandbox etc.

#32 fixes this bug

Untill admin approves changes, you can use
npm i mic-recorder-to-mp3-fixed
and can be used as:
import MicRecorder from 'mic-recorder-to-mp3-fixed';
I have merged the 32 in this package.
Thanks


Add options for disabling S3/DynamoDB access

There are multiple ways in which #3 could be solved but they all introduced some kind of extra template that should be read or complicated options or prompt. I believe this PR is good enough for most use cases, if more granularity is needed in permissions, policies can be added to the role in AWS giving access to only specific buckets.
Adds command line options for disabling the default permissions on S3 or DynamoDB.
The default option is still full rights to all resources.
Closes #3

I've tested the changes against a fresh example, looks good to me.
Thanks!


QifDom.Import() erases

I wonder if you could provide a parameter for the Import method to switch off the erasing of all data, so I may import several files into one QifDom?

Give the latest a go.


copyright corrections

Copyright corrections for iotfunctions now

Nothing (or only small things) has been changed in those files in 2021 yet. Therefore no 2021 in the copy-right statement. Once we change the code we have to add 2021, of course.


File not opening up in VS Code when set to default

VSCode Version: Code 1.10.2 (8076a19fdcab7e1fc1707952d652f0bb6c6db331, 2017-03-08T14:02:52.799Z)
OS Version: Windows_NT ia32 10.0.14393
Extensions:

Extension
Author
Version

html-snippets
abusaidm
0.1.0

python
donjayamanne
0.6.0

vscode-npm-script
eg2
0.1.8

Angular2
johnpapa
2.2.3

mssql
ms-mssql
0.3.0

csharp
ms-vscode
1.8.1

Steps to Reproduce:

Set VS Code as default application for opening a file (json, sql)
Double click the file to open that file in visual studio
Notice that vs code opens with new untitled tab, but the intended file is not opened

@vinu11 can you completely uninstall and then reinstall VS Code to see if that fixes it?

reg_keys.zip

@bpasero issue resolved by uninstalling vs code and then reinstalling latest version.


Executing V2 issues

Since the new version came out, I have been trying to get things working. Here are a couple issues that I ran into, and resolved:
needed s5cmd so had to install conda then s5cmd
installed docker rootless although networking is unavailable, so for now, running docker as root
default.conf is missing lines for the AWS Secret and ID. Added them no problem.
when running the
bash scripts/run_prep_artifacts.sh \
  --config configs/rp_v2.0.conf \
  --listings /path/to/listings/file.txt\
  --max_workers 32

I modified to run in my environment (Ubuntu 22.04 WSL2):
sudo bash scripts/run_prep_artifacts.sh \
  --config configs/default.conf \
  --listings ../data/listings/listing.txt \
  --max_workers 32

I get the following error:
Created run id: 7f15c068
Writing run id to file /nfs/slow/data/artifacts-7f15c068/_RUN_ID
copied listings file from ../data/listings/listing.txt to /nfs/slow/data/artifacts-7f15c068/listings/listings.txt
__SNAPSHOT_LISTINGS_SUCCESS__ 2014-15
__SNAPSHOT_LISTINGS_SUCCESS__ 2014-23
__SNAPSHOT_LISTINGS_SUCCESS__ 2014-35
__SNAPSHOT_LISTINGS_SUCCESS__ 2014-41
__SNAPSHOT_LISTINGS_SUCCESS__ 2014-42
__SNAPSHOT_LISTINGS_SUCCESS__ 2014-49
__SNAPSHOT_LISTINGS_SUCCESS__ 2014-52
__SNAPSHOT_LISTINGS_SUCCESS__ 2015-14
__SNAPSHOT_LISTINGS_SUCCESS__ 2015-22
__SNAPSHOT_LISTINGS_SUCCESS__ 2015-27
__SNAPSHOT_LISTINGS_SUCCESS__ 2015-32
__SNAPSHOT_LISTINGS_SUCCESS__ 2015-35
__SNAPSHOT_LISTINGS_SUCCESS__ 2015-40
__SNAPSHOT_LISTINGS_SUCCESS__ 2015-48
__SNAPSHOT_LISTINGS_SUCCESS__ 2016-07
__SNAPSHOT_LISTINGS_SUCCESS__ 2016-18
__SNAPSHOT_LISTINGS_SUCCESS__ 2016-22
__SNAPSHOT_LISTINGS_SUCCESS__ 2016-26
__SNAPSHOT_LISTINGS_SUCCESS__ 2016-30
__SNAPSHOT_LISTINGS_SUCCESS__ 2016-36
__SNAPSHOT_LISTINGS_SUCCESS__ 2016-40
__SNAPSHOT_LISTINGS_SUCCESS__ 2016-44
__SNAPSHOT_LISTINGS_SUCCESS__ 2016-50
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-04
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-09
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-17
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-22
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-26
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-30
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-34
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-39
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-43
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-47
__SNAPSHOT_LISTINGS_SUCCESS__ 2017-51
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-05
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-09
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-13
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-17
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-22
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-26
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-30
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-34
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-39
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-43
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-47
__SNAPSHOT_LISTINGS_SUCCESS__ 2018-51
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-04
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-09
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-13
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-18
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-22
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-26
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-30
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-35
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-39
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-43
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-47
__SNAPSHOT_LISTINGS_SUCCESS__ 2019-51
__SNAPSHOT_LISTINGS_SUCCESS__ 2020-05
__SNAPSHOT_LISTINGS_SUCCESS__ 2020-10
__SNAPSHOT_LISTINGS_SUCCESS__ 2020-16
__SNAPSHOT_LISTINGS_SUCCESS__ 2020-24
__SNAPSHOT_LISTINGS_SUCCESS__ 2020-29
__SNAPSHOT_LISTINGS_SUCCESS__ 2020-34
__SNAPSHOT_LISTINGS_SUCCESS__ 2020-40
__SNAPSHOT_LISTINGS_SUCCESS__ 2020-45
__SNAPSHOT_LISTINGS_SUCCESS__ 2020-50
__SNAPSHOT_LISTINGS_SUCCESS__ 2021-04
__SNAPSHOT_LISTINGS_SUCCESS__ 2021-10
__SNAPSHOT_LISTINGS_SUCCESS__ 2021-17
__SNAPSHOT_LISTINGS_SUCCESS__ 2021-21
__SNAPSHOT_LISTINGS_SUCCESS__ 2021-25
__SNAPSHOT_LISTINGS_SUCCESS__ 2021-31
__SNAPSHOT_LISTINGS_SUCCESS__ 2021-39
__SNAPSHOT_LISTINGS_SUCCESS__ 2021-43
__SNAPSHOT_LISTINGS_SUCCESS__ 2021-49
__SNAPSHOT_LISTINGS_SUCCESS__ 2022-05
__SNAPSHOT_LISTINGS_SUCCESS__ 2022-21
__SNAPSHOT_LISTINGS_SUCCESS__ 2022-27
__SNAPSHOT_LISTINGS_SUCCESS__ 2022-33
__SNAPSHOT_LISTINGS_SUCCESS__ 2022-40
__SNAPSHOT_LISTINGS_SUCCESS__ 2022-49
__SNAPSHOT_LISTINGS_SUCCESS__ 2023-06
__SNAPSHOT_LISTINGS_SUCCESS__ 2023-14
Toal number of listings: 83
__LANG_PREP_START__ en @ Wed Nov  1 12:50:35 MDT 2023
[sudo] password for theskaz:
[2023-11-01 18:50:41,592]::(PID 1)::INFO::Start preparing artifacts for en
[2023-11-01 18:50:41,592]::(PID 1)::INFO::num_samples: 500000
[2023-11-01 18:50:41,592]::(PID 1)::INFO::PYTHONHASHSEED: 42
[2023-11-01 18:50:41,596]::(PID 1)::INFO::CCNetDownloader(en) Start loading input listings...
[2023-11-01 18:50:41,597]::(PID 1)::INFO::CCNetDownloader(en) Partitioning inputs by snapshot...
Traceback (most recent call last):
  File "/usr/app/src/prep_artifacts.py", line 186, in <module>
    main(artifacts_dir=args.artifacts_dir,
  File "/usr/app/src/prep_artifacts.py", line 114, in main
    ccnet.run(logger=logger)
  File "/usr/app/src/artifacts/downloaders/ccnet_downloader.py", line 95, in run
    1, self._num_samples // len(inputs_by_snapsh)
       ~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: integer division or modulo by zero
Error: scripts/run_prep_artifacts.sh:7: command `sudo docker run --env AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" --env AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -v "${DATA_ROOT%/}":"${DOCKER_MNT_DIR%/}" -t "${DOCKER_REPO}" python3 src/prep_artifacts.py --artifacts_dir "${ARTIFACTS_DIR%/}" --cc_input "${ARTIFACTS_DIR%/}/listings/listings.txt" --cc_input_base_uri "${S3_BUCKET%/}${S3_CCNET_PREFIX%/}" --cache_dir "${DOCKER_MNT_DIR%/}/.hf_cache" --lang "${lang}" --max_workers "${MAX_WORKERS}" --endpoint_url "$DOCKER_S3_ENDPOINT_URL" --dsir_num_samples "${DSIR_NUM_SAMPLES}" --dsir_feature_dim "${DSIR_FEATURE_DIM}" --classifiers_num_samples "${CLASSIFIERS_NUM_SAMPLES}" --max_paragraphs_per_book_sample "${MAX_PARAGRAPHS_PER_BOOK_SAMPLE}" --max_samples_per_book "${MAX_SAMPLES_PER_BOOK}"` failed with exit code 1

is my listing parameter correct? or is there some other issue

I noticed an error where it was looking for "ls /rs_cc_net/*" could someone explain the usage of these parameters?
S3_ENDPOINT_URL=""
S3_BUCKET=""
S3_CCNET_PREFIX="/rs_cc_net"
S3_PROFILE=""
I do not wish to upload anything to S3. I am using AWS creds similar to V1 because I assume Im doing the same download.

I see that line 88 in ccnet_downloader.py shows
inputs_by_snapsh = defaultdict(list)
I am unable to see where list is defined.

I can verify that len(inputs_by_snapsh) is 0

Hi @hicotton02 , thanks for your question!

default.conf is missing lines for the AWS Secret and ID
We deliberately left these out since so that users specify these via export AWS_SECRET... (reduces the risk of uploading access keys to github).

Once the environment variables are specified, you can get the listings via
s5cmd --profile "$S3_PROFILE" --endpoint-url "$S3_ENDPOINT_URL" \
    ls "${S3_BUCKET%/}${S3_CCNET_PREFIX%/}/*" | grep "\.json\.gz$" | awk '{print $NF}' >"${LISTINGS_FILE}"

what should produce a file with contents of the form:
2014-15/0000/en_head.json.gz
2014-15/0000/en_middle.json.gz
2014-15/0001/en_head.json.gz
2014-15/0001/en_middle.json.gz

Let me know if this helps!:)

Thank you so much for the response.
Is the s5cmd command supposed to point to my own s3 bucket or someone else's?
I created a bucket but it is blank at this time. I remember in V1 we were downloading info from I think Arxiv's bucket.

There is no data that needs to be pulled from an external S3 bucket, only your own where you have the ccnet output stored -- it is also only required to create the artifacts. Are you creating your own artifacts for a custom dataset or are you trying to reproduce the quality signals we have provided?
You can download the ccnet out from the public urls (https://data.together.xyz/redpajama-data-v2/v1.0.0/) and then upload it to your own S3 bucket. Also check out the huggingface repo here which contains instructions on how to download the data.

There is no data that needs to be pulled from an external S3 bucket, only your own where you have the ccnet output stored -- it is also only required to create the artifacts. Are you creating your own artifacts for a custom dataset or are you trying to reproduce the quality signals we have provided?
You can download the ccnet out from the public urls (https://data.together.xyz/redpajama-data-v2/v1.0.0/) and then upload it to your own S3 bucket. Also check out the huggingface repo here which contains instructions on how to download the data.

I am going to end up doing both. Right now I am just learning how you guys did all this. Once I have that done, and have some sort of understanding on what is going on, I want to add and remove data to see how that affects everything. I am in my master's for AI/ML and using this to learn in addition to what I am learning in school.

Wrote a python script to download all the ccnet data based on your links above. it does this in parallel and is basic. saturated my connection and server to get the most efficient process going.
import os
import subprocess
import multiprocessing as mp

CC_SNAPSHOT_IDS = [
  "2014-15",
  "2014-23",
  "2014-35",
  "2014-41",
  "2014-42",
  "2014-49",
  "2014-52",
  "2015-14",
  "2015-22",
  "2015-27",
  "2015-32",
  "2015-35",
  "2015-40",
  "2015-48",
  "2016-07",
  "2016-18",
  "2016-22",
  "2016-26",
  "2016-30",
  "2016-36",
  "2016-40",
  "2016-44",
  "2016-50",
  "2017-04",
  "2017-09",
  "2017-17",
  "2017-22",
  "2017-26",
  "2017-30",
  "2017-34",
  "2017-39",
  "2017-43",
  "2017-47",
  "2017-51",
  "2018-05",
  "2018-09",
  "2018-13",
  "2018-17",
  "2018-22",
  "2018-26",
  "2018-30",
  "2018-34",
  "2018-39",
  "2018-43",
  "2018-47",
  "2018-51",
  "2019-04",
  "2019-09",
  "2019-13",
  "2019-18",
  "2019-22",
  "2019-26",
  "2019-30",
  "2019-35",
  "2019-39",
  "2019-43",
  "2019-47",
  "2019-51",
  "2020-05",
  "2020-10",
  "2020-16",
  "2020-24",
  "2020-29",
  "2020-34",
  "2020-40",
  "2020-45",
  "2020-50",
  "2021-04",
  "2021-10",
  "2021-17",
  "2021-21",
  "2021-25",
  "2021-31",
  "2021-39",
  "2021-43",
  "2021-49",
  "2022-05",
  "2022-21",
  "2022-27",
  "2022-33",
  "2022-40",
  "2022-49",
  "2023-06",
  "2023-14"
]

def download_snapshot(snapshot_id, semaphore):
    with semaphore:
        LANG = "en"
        BASE_URL = "https://data.together.xyz/redpajama-data-v2/v1.0.0"
        PARTITION = "head_middle"
        listings_tag = f"{LANG}-{snapshot_id}-{PARTITION}"
        os.makedirs("listings", exist_ok=True)
        subprocess.run(["wget", f"{BASE_URL}/listings/{listings_tag}.txt", "-O", f"listings/{listings_tag}.txt"])

        with open(f"listings/{listings_tag}.txt", "r") as listings_file:
            for line in listings_file:
                line = line.strip()
                url = f"{BASE_URL}/documents/{line}.json.gz"
                dest = f"documents/{line}.json.gz"
                os.makedirs(os.path.dirname(dest), exist_ok=True)
                subprocess.run(["wget", url, "-O", dest])

                url = f"{BASE_URL}/quality_signals/{line}.signals.json.gz"
                dest = f"quality_signals/{line}.signals.json.gz"
                os.makedirs(os.path.dirname(dest), exist_ok=True)
                subprocess.run(["wget", url, "-O", dest])

            COMPS = ["minhash", "duplicates"]
            for comp in COMPS:
                listings_file.seek(0)
                for line in listings_file:
                    line = line.strip()
                    url = f"{BASE_URL}/{comp}/{line}.{comp}.parquet"
                    dest = f"{comp}/{line}.{comp}.parquet"
                    os.makedirs(os.path.dirname(dest), exist_ok=True)
                    subprocess.run(["wget", url, "-O", dest])

if __name__ == "__main__":
    os.chdir("/nfs/slow/data/ccnet")
    semaphore = mp.Semaphore(mp.cpu_count())
    processes = [mp.Process(target=download_snapshot, args=(i, semaphore)) for i in CC_SNAPSHOT_IDS]
    for process in processes:
        process.start()
    for process in processes:
        process.join()
    print("All Threads Completed", flush=True)


move from master -> main

As github started moving from naming default branch master -> main , I've done that now.
@prabshr  You would need to adapt this within your fork/local repo.

Done !

OK, since this has been the only fork so far, this can be closed now.


Fix rest params order

Description
REST layer is calling *ByParams core api providing ExternalID and ParentResourceID in wrong order.
How Has This Been Tested?

Create registered model and model version
Run curl -X 'GET' \ 'https://localhost:8080/api/model_registry/v1alpha1/model_version?name={VERSION_NA<E}&parentResouceID={REG_MODEL_ID}
You'll get error: No model versions found
Run curl -X 'GET' \ 'https://localhost:8080/api/model_registry/v1alpha1/model_version?name={VERSION_NA<E}&externalID={REG_MODEL_ID}
You'll get the correct result but providing the reg model id as external id.

Merge criteria:

[x] The commits and have meaningful messages; the author will squash them after approval or will ask to merge with squash.
[x] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[ ] The developer has manually tested the changes and verified that the changes work

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (e56fc36) 74.52% compared to head (303b3c3) 83.92%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #245      +/-   ##
==========================================
+ Coverage   74.52%   83.92%   +9.40%
==========================================
  Files          17       12       -5
  Lines        2296      591    -1705
  Branches       73       73
==========================================
- Hits         1711      496    -1215
+ Misses        420       75     -345
+ Partials      165       20     -145

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Generic union type does not seem to properly handle recursivity and arrays

Bug Report
🔎 Search Terms
Generic union type, recursivity in unions, generic union with arrays, TS 2349
🕗 Version & Regression Information

This is the behavior in every version I tried (nightly included), and I reviewed the FAQ for entries about it

⏯ Playground Link
Playground link with relevant code
💻 Code

interface A {
    id: string;
    items: A[];
    things: number[];
}
interface B {
    id: string;
    items: B[];
    things: string[];
}

const foo = <T extends A | B>(root: T, id: string): T | undefined => {
    return root.items.find(v => v.id === id); // not working
}

const foo2 = <T extends A | B>(root: {id: string, items: T[]}, id: string): T | undefined => {
    return root.items.find(element => element.id === id); // working
}

const bar = <T extends A | B>(root: T): number => {
    let count = root.things.length;
    root.items.forEach(element => { // working
        count += bar(element);
    });
    return count;
}

🙁 Actual behavior
TypeScript raises an error when using the first .find, saying "This expression is not callable. Each member of the union type has signatures, but none of those signatures are compatible with each other."
🙂 Expected behavior
I would expect the .find method to properly work on the items list, even with an union type, like the two other functions work with no error.
There seems to be an issue with type unions that I can't understand.

Duplicate of #44373. Used search terms: union find in:title


I gave it a quick look and that's the timeline:

2015/06/03 21:44:54 I enable the rancher agent
2015/06/04 18:00:21 the docker deamon is out of fd
A few days later I noticed the issue and killed the agent

I grab the whole docker and rancher agent logs if you want me to search for something in them.

@Nicofuma Was the docker daemon out of FDs or the rancher agent?  Also, is this RancherOS?  In RancherOS we need to increase the default ulimits given to Docker.

It is rancherOS yes. The user docker daemon was out of FDs and when I killer the rancher agent it was back to normal with around 100 FDS instead of 1024.

@Nicofuma please reopen if you see this issue again.  There has been tons of fixes in Docker since this issue was files that address memory and FD leaks.


upward binding inside repeat template - read-only and read-write

Originally posted on SO here.
Suppose I have a custom x-child element with the following properties:
properties: {
    prop_read_write: {type: Number, value: 1, notify: true},
    prop_read_only: {type: Number, value: 101, notify: true, readOnly: true}
}

If I put that inside a custom x-parent in the following way...
<x-child prop_read_write="{{parent_a}}" prop_read_only="{{parent_b}}"></x-child>

...I get what I expect, which is that the default values are bound upward.  However if I do something similar within a repeat I can't work out how to get that same behavior.
<template is="dom-repeat" items="{{kids}}">
  <x-child prop_read_write="{{item.a}}" prop_read_only="{{item.b}}"></x-child>
</template>

I've fleshed this out a bit further in a proper test page - see gist test.html.
The output looks like this:

There's two reasons why this behavior happens:

Default values in the properties object are only applied if there is no incoming state from property bindings or attributes
When a property is set to an object, the any bindings to the subproperties of that object are set at that time.

There is no good workaround for the behavior right now, other than using the readOnly metadata property, which is why prop_read_only isn't affected.
We are considering ways of improving this behavior.

Sorry for the long delay on this. I believe this behavior is working as intended so we're going to close this issue.
Polymer's data binding system works on a "push" model. When data is pushed to elements, they react. Any data bound into an element overrides an element's default property value.
By design a path binding (e.g. {{item.a}}) receives data when the root item property changes.  This means a binding to {{parent_a}} behaves differently than a binding to {{item.a}}.
What you can do instead in this case... We recommend avoiding a 2-way binding when you need to populate repeated data via a default property generated within an element. Instead, add an event listener to the element and set the data yourself (e.g. something like on-prop_read_write-changed="_popuplateItemA"). Hope that helps.


hide-navigation doesn't work

A hide-navigation prop does not work
Steps to reproduce the behavior:

Go to https://vue3datepicker.com/props/general-configuration/#hide-navigation
Click on a datepicker
See the datepicker has a year navigation for example

Expected behavior
Datepicker shouldn't have a year navigation
Screenshots

Desktop Chrome 114.0.5735.199

This is more of a documentation/prop naming, it actually hides the back button from the overlay, intended for usage with flow prop.


"Multiple errors are preventing progress" reported during the cluster deployment process

Describe the bug
Inspected several errors during the cluster deployment (virt env). The deployment was able to recover and ended successfully eventually.
To Reproduce
$ git rev-parse HEAD
93a8fc0af1e44df6c270e3b4b042f82c4359307f
Deploy a new cluster with "make"
Expected/observed behavior
Several "Multiple errors are preventing progress" appears in the log, but installation was able to recover and deploys successfully eventually.

/home/kni/go/src/github.com/openshift-metalkube/kni-installer/bin/kni-install --dir ocp --log-level=debug wait-for install-complete
level=debug msg="OpenShift Installer 3e17e2e5b4cfcd2b357eb7ef866e7b57b028fce8"
level=debug msg="Built from commit 3e17e2e5b4cfcd2b357eb7ef866e7b57b028fce8"
level=info msg="Waiting up to 30m0s for the cluster at https://api.ostest.test.metalkube.org:6443 to initialize..."
level=debug msg="Still waiting for the cluster to initialize: Working towards 4.0.0-0.ci-2019-04-17-133604: 24% complete"
level=debug msg="Still waiting for the cluster to initialize: Working towards 4.0.0-0.ci-2019-04-17-133604: 67% complete"
level=debug msg="Still waiting for the cluster to initialize: Multiple errors are preventing progress:\n* Cluster operator authentication is still updating: missing version information for integrated-oauth-server\n* Cluster operator openshift-apiserver has not yet reported success\n* Cluster operator openshift-samples is still updating\n* Cluster operator storage has not yet reported success\n* Could not update oauthclient "console" (267 of 331): the server does not recognize this resource, check extension API servers"
level=debug msg="Still waiting for the cluster to initialize: Multiple errors are preventing progress:\n* Cluster operator authentication is still updating: missing version information for integrated-oauth-server\n* Cluster operator openshift-apiserver has not yet reported success\n* Cluster operator openshift-samples is still updating\n* Cluster operator storage has not yet reported success\n* Could not update oauthclient "console" (267 of 331): the server does not recognize this resource, check extension API servers"
level=debug msg="Still waiting for the cluster to initialize: Working towards 4.0.0-0.ci-2019-04-17-133604"
level=debug msg="Still waiting for the cluster to initialize: Working towards 4.0.0-0.ci-2019-04-17-133604: downloading update"

Note that all of these messages are "debug", and would not normally be visible.  If it completed successfully in the end, you can safely ignore all of this.


team and model names across season

Some teams changed their name from year to year, and thus a single team-model can have multiple folders in the model-output folder. If this change in team-model differs only trivially (for example, spelling name change, or writing our the model name instead of acronym), rather than a substantive change in the model itself, then perhaps we should work towards consolidating these multiple folders

My .02 (and I raised an issue over in the hubverse-transform repo): any cloud-based data transforms should handle all incoming file/folder names gracefully.
To date, that transform module has assumed that incoming data has gone through the hubverse model-output validation function, but if we want to use it with converted archived data, it should handle these types of edge cases.


Update rustdocs

In this PR, the rustdocs are updated to show the features which enable the relevant APIs.

Hello, I am a code review bot on flows.network.
It could take a few minutes for me to analyze this PR. Relax, grab a cup of coffee and check back later. Thanks!

@L-jasmine Thanks for the review!


[Infra] Refactors small areas of the ZipBuilder

Context

I made some unrelated refactors while working on #9829 and wanted to send them in their own PR

#no-changelog

Confirmed generated zip directory structure matches 9.2.0


[GOBBLIN-860] Process flow-level events for setting/retrieving flow status

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-860

Description

[x] Here are some details about my PR, including screenshots (if applicable):

Process flow-level events for flow status instead of computing it based on job-level events. In more detail:

Modify FS and Mysql job retriever to return all job states (including the flow level one) instead of filtering that one out
Modify flow status resource to read flow state instead of computing from job states
Add FLOW_RUNNING event sent by DagManager
Modify job status monitor to accept these flow level events

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@arjun4084346 please review

+1

Codecov Report

Merging #2715 into master will increase coverage by <.01%.
The diff coverage is 37.5%.

@@             Coverage Diff              @@
##             master    #2715      +/-   ##
============================================
+ Coverage     45.08%   45.08%   +<.01%
- Complexity     8739     8743       +4
============================================
  Files          1879     1880       +1
  Lines         70158    70169      +11
  Branches       7703     7700       -3
============================================
+ Hits          31628    31635       +7
- Misses        35609    35612       +3
- Partials       2921     2922       +1

Impacted Files
Coverage Δ
Complexity Δ

.../org/apache/gobblin/metrics/event/TimingEvent.java
70% <ø> (ø)
15 <0> (ø)
:arrow_down:

...bblin/service/monitoring/FsJobStatusRetriever.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...in/service/monitoring/MysqlJobStatusRetriever.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

.../service/monitoring/KafkaAvroJobStatusMonitor.java
51.66% <0%> (ø)
12 <0> (ø)
:arrow_down:

...org/apache/gobblin/service/FlowStatusResource.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...blin/service/modules/orchestration/DagManager.java
78.99% <100%> (-0.06%)
12 <0> (ø)

...ervice/modules/orchestration/TimingEventUtils.java
95.45% <100%> (+9.09%)
4 <1> (+2)
:arrow_up:

...service/modules/orchestration/DagManagerUtils.java
84% <75%> (-1.14%)
32 <2> (ø)

...lin/restli/throttling/ZookeeperLeaderElection.java
70% <0%> (-2.23%)
13% <0%> (ø)

.../org/apache/gobblin/metrics/RootMetricContext.java
78.12% <0%> (-1.57%)
15% <0%> (-1%)

... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb0c72d...561604b. Read the comment docs.

@sv2000 please merge


Typo in text in exercise "Condense arrays with reduce"

Challenge Condense arrays with reduce has an issue.
User Agent is: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
In the 3rd paragraph it is written "iterating" instead of "iteraCting". With the C. Hope it helps! Thank you for the creator of this amazing tool and all of its amazing community for everything achieved so far!

Closed it because the bot message told me that "The issue tracker is for reporting bugs only." It got me confused whether this a bug or not. It is definitely a typo in the text. But not sure it is a bug. Can you give me a hand?

@victorwestmann, oh okay.   dont worry about bot message.  you could report for bug or typo.   Someone will check on your issue.  thanks

@victorwestmann I think the spelling is correct. Note that 'iterating' means to perform repeatedly so in this case iterating through an array means cycling through all the items in an array. Reopen this if I misunderstood.
Happy coding.


Small caps broken in LuaLaTeX-dev when using Renderer=Harfbuzz

Description
Small caps in LuaLaTeX-dev are OK when using RawFeature={mode=harf} but broken when using Renderer=Harfbuzz
Check/indicate

[ ] Relevant for XeTeX
[*] Relevant for LuaTeX
[*] Issue tracker has been searched for similar issues?
[ ] Links to <tex.stackexchange.com> discussion if appropriate

Minimal example demonstrating the issue
\documentclass{article}
\usepackage{fontspec}
\setlength{\parindent}{0pt}
\setlength{\parskip}{\normalbaselineskip}
\begin{document}
Standard text. \textsc{Small caps text}.
\par
\fontspec{texgyretermes-regular.otf}%
Switch to TeX Gyre Termes \\
Standard text. \textsc{Small caps text}.
\par
\fontspec{texgyretermes-regular.otf}[Renderer=Harfbuzz]%
Switch to TeX Gyre Termes with \verb+Renderer=Harfbuzz+ \\
Standard text. \textsc{Small caps text}.
\par
\fontspec{texgyretermes-regular.otf}[RawFeature={mode=harf}]%
Switch to TeX Gyre Termes with \verb+RawFeature={mode=harf}+ \\
Standard text. \textsc{Small caps text}.
\end{document}

Further details
Might be linked to https://github.com/wspr/fontspec/issues/396. I'm not quite sure how to test for that. The bug was noticed today with everything updated via tlmgr (actually TeX Live Utility on the Mac, with MacTeX 2019 installed). Fontspec v2.7e from 2020/01/09.
SmallCapsTest.pdf

closing as resolved by luaotfload update.


Add Download Button For Generated Audio

Hey, sorry for the late response. Can all of the formatting changes be excluded? Thanks.

No download button is needed, the player that plays the audio has the download option

No download button is needed, the player that plays the audio has the download option

That behaviour is browser-controlled and not standard


Provisioning complex Assets

Category

[X] Question
[ ] Typo
[ ] Bug
[ ] Additional article idea

Hi Guys,
Currently in SPFX the assets that i can provision are:
Fields
Content types
List instances
List instances with custom schema
In PnP we can provision Pages, MasterPages, Pages Layouts, complex Content Types, etc.
In SPFX is possible? If not, it´s in Roadmap?
Best Regards

Technically you could provision other assets also from SPFx solutions, but it does get a bit complicated. Given the direction with SharePoint Online, master pages or page layouts are also no longer used as such. We are not looking into investing more in here at least through SPFx solutions, rather through other areas.

a


fix: git required by kitchen-salt

:tada: This PR is included in version 1.0.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Configure PORT and update README

Context
Updates README to use foreman to run the server
Changes proposed in this pull request

configure PORT in .env.example and .env.development (required by puma to run server at port 3000 and not default to puma configured PORT 5000)

Things to check

[x] This code does not rely on migrations in the same Pull Request
[x] If this code includes a migration adding or changing columns, it also backfills existing records for consistency
[x] API release notes have been updated if necessary
[x] New environment variables have been added to the Azure config

run yarn installation with --ignore-engines property to resolve incompatible engine error

The reason for the error is because we recently started specifying a Node version in our package.json.
Rather than using yarn with --ignore-engines, could you install and run the version of node LTS that we use in production? :+1:
We also removed foreman from the Gemfile because that's what the author recommends.

@tvararu Thanks for the feedback. I wrongly assumed that any missing dependencies would have been install by yarn as my attempt to use brew to setup that specific version was not working. Sorted now.

Reopened as #3410 @duncanjbrown


Performance issue on TPCH comparing with sparksql

Description
I'm trying to run TPC-H Q3 and compare the performance between Wayang and SparkSQL under the following setup:

Running both Spark (3.5.1) and Wayang on a local VM with 32 CPU cores and 128GB memory
Running a postgres instance that maintains all the TPC-H tables (sf=10) on a remote VM

I try to keep the spark setting the same on both runs. And for Q3 wayang took around 3 minutes while spark took only 40 seconds.
To reproduce
To run Wayang, I compile the project locally (using tag 1.0.0) and use the benchmark code under wayang-benchmark directly: ./wayang-1.0.0-SNAPSHOT/bin/wayang-submit org.apache.wayang.apps.tpch.TpcH exp\(123\) spark,postgres file:///path/to/wayang.properties Q3
The wayang.properties file is like the following:
wayang.postgres.jdbc.url = jdbc:postgresql://{POSTGRES_IP}:{POSTGRES_PORT}/{TPCH_DB}
wayang.postgres.jdbc.user = {POSTGRES_USER}
wayang.postgres.jdbc.password = {POSTGRES_PASSWORD}

spark.master = local[32]
spark.driver.memory = 110G
spark.executor.memory = 110G
spark.executor.cores = 32
wayang.giraph.hdfs.tempdir = file:///tmp/result/

spark.rdd.compress = true
spark.log.level = INFO

To run Spark, I use the following code:
import sys
import time
from pyspark.sql import SparkSession
from contexttimer import Timer

SPARK_JARS = "path/to/jar/postgresql-42.3.8.jar"
POSTGRES_URL = "jdbc:postgresql://{POSTGRES_IP}:{POSTGRES_PORT}/{TPCH_DB}"
POSTGRES_USER = "{POSTGRES_USER}"
POSTGRES_PASSWORD = "{POSTGRES_PASSWORD}"

TPCH_Q3 = """SELECT
    l_orderkey,
    sum(l_extendedprice * (1 - l_discount)) AS revenue,
    o_orderdate,
    o_shippriority
FROM
    customer,
    orders,
    lineitem
WHERE
    c_mktsegment = 'BUILDING'
    AND c_custkey = o_custkey
    AND l_orderkey = o_orderkey
    AND o_orderdate < CAST('1995-03-15' AS date)
    AND l_shipdate > CAST('1995-03-15' AS date)
GROUP BY
    l_orderkey,
    o_orderdate,
    o_shippriority
ORDER BY
    revenue DESC,
    o_orderdate"""

def registerPostgres(spark, tables, url):
    for name in tables:
        spark.sql(f"""
            CREATE TEMPORARY VIEW {name}
            USING org.apache.spark.sql.jdbc
            OPTIONS (
              driver "org.postgresql.Driver",
              url "{url}",
              dbtable "public.{name}",
              user '{POSTGRES_USER}',
              password '{POSTGRES_PASSWORD}',
              pushDownAggregate 'true'
            )
            """)

def registerViews(spark):
    registerPostgres(spark, ["lineitem", "customer", "orders", "nation", "region", "supplier", "part", "partsupp"], POSTGRES_URL)

def run_query(spark, query):
    with Timer() as timer:
        df = spark.sql(query)
        df.collect()
    print(f"get {df.count()} rows, {len(df.columns)} cols")
    print(f"plan: {df.explain()}")
    print(f"took {timer.elapsed:.2f} in total")
    # print(df)
    print()
    sys.stdout.flush()

if __name__ == '__main__':

    spark = (
        SparkSession.builder.master("local[32]")
        .appName("test-spark")
        .config("spark.jars", SPARK_JARS)
        .config("spark.executor.memory", "110g")
        .config("spark.driver.memory", "110g")
        .config("spark.log.level", "INFO")
        .config("spark.ui.port", "4040")
        .getOrCreate()
    )

    print(spark.sparkContext.getConf().getAll())
    registerViews(spark)

    run_query(spark, TPCH_Q3)
    time.sleep(2)
    spark.stop()

Some investigation
The queries that are used to fetch data from postgres using both platforms, which are basically the same (filter and projection pushdown are enabled).
I try to print the logs of spark execution as much as I can to see the difference between the two.  One significant overhead I found is that wayang produces much larger ShuffleMapTask for join than spark does (~46500000 bytes v.s. 8000 bytes), which causes ~2 seconds to serialize each task (64 tasks in total) one by one and result in a 1 minutes overhead. On the other hand, the serialization time on spark is negligible.
I'm not very familiar with spark execution, so I'm not sure why it is the case. Can anyone give me a pointer? Is there anything I'm missing such as the way I run the query or something in configuration? Thank you!

Here is a log snippet related to the serialization overhead of spark execution with wayang I mentioned above:
24/03/26 18:18:26 WARN TaskSetManager: Stage 2 contains a task of very large size (45404 KiB). The maximum recommended task size is 1000 KiB.
24/03/26 18:18:26 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 0) (<IP_ADDRESS>, executor driver, partition 0, PROCESS_LOCAL, 46494022 bytes)
24/03/26 18:18:26 INFO TaskSetManager: start serialize task 1...
24/03/26 18:18:26 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 1)
24/03/26 18:18:26 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 1)
24/03/26 18:18:28 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 1)
24/03/26 18:18:28 INFO TaskSetManager: finish serialize task 1...
24/03/26 18:18:28 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 1) (<IP_ADDRESS>, executor driver, partition 1, PROCESS_LOCAL, 46490225 bytes)
24/03/26 18:18:28 INFO TaskSetManager: start serialize task 2...
24/03/26 18:18:28 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 2)
24/03/26 18:18:28 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 2)
24/03/26 18:18:29 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 2)
24/03/26 18:18:29 INFO TaskSetManager: finish serialize task 2...
24/03/26 18:18:29 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 2) (<IP_ADDRESS>, executor driver, partition 2, PROCESS_LOCAL, 46488713 bytes)
24/03/26 18:18:29 INFO TaskSetManager: start serialize task 3...
24/03/26 18:18:29 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 3)
24/03/26 18:18:29 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 3)
24/03/26 18:18:31 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 3)
24/03/26 18:18:31 INFO TaskSetManager: finish serialize task 3...
24/03/26 18:18:31 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 3) (<IP_ADDRESS>, executor driver, partition 3, PROCESS_LOCAL, 46488685 bytes)
24/03/26 18:18:31 INFO TaskSetManager: start serialize task 4...
24/03/26 18:18:31 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 4)
24/03/26 18:18:31 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 4)
24/03/26 18:18:32 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 4)
24/03/26 18:18:32 INFO TaskSetManager: finish serialize task 4...
24/03/26 18:18:32 INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 4) (<IP_ADDRESS>, executor driver, partition 4, PROCESS_LOCAL, 46486155 bytes)
24/03/26 18:18:32 INFO TaskSetManager: start serialize task 5...
24/03/26 18:18:32 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 5)
24/03/26 18:18:32 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 5)
24/03/26 18:18:34 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 5)
24/03/26 18:18:34 INFO TaskSetManager: finish serialize task 5...
24/03/26 18:18:34 INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 5) (<IP_ADDRESS>, executor driver, partition 5, PROCESS_LOCAL, 46490264 bytes)
24/03/26 18:18:34 INFO TaskSetManager: start serialize task 6...
24/03/26 18:18:34 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 6)
24/03/26 18:18:34 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 6)
24/03/26 18:18:36 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 6)
24/03/26 18:18:36 INFO TaskSetManager: finish serialize task 6...
24/03/26 18:18:36 INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 6) (<IP_ADDRESS>, executor driver, partition 6, PROCESS_LOCAL, 46488245 bytes)
24/03/26 18:18:36 INFO TaskSetManager: start serialize task 7...
24/03/26 18:18:36 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 7)
24/03/26 18:18:36 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 7)
24/03/26 18:18:38 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 7)
24/03/26 18:18:38 INFO TaskSetManager: finish serialize task 7...
24/03/26 18:18:38 INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 7) (<IP_ADDRESS>, executor driver, partition 7, PROCESS_LOCAL, 46487068 bytes)
24/03/26 18:18:38 INFO TaskSetManager: start serialize task 8...
24/03/26 18:18:38 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 8)
24/03/26 18:18:38 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 8)
24/03/26 18:18:40 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 8)
24/03/26 18:18:40 INFO TaskSetManager: finish serialize task 8...
24/03/26 18:18:40 INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 8) (<IP_ADDRESS>, executor driver, partition 8, PROCESS_LOCAL, 46489857 bytes)
24/03/26 18:18:40 INFO TaskSetManager: start serialize task 9...
24/03/26 18:18:40 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 9)
24/03/26 18:18:40 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 9)
24/03/26 18:18:41 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 9)
24/03/26 18:18:41 INFO TaskSetManager: finish serialize task 9...
24/03/26 18:18:41 INFO TaskSetManager: Starting task 9.0 in stage 2.0 (TID 9) (<IP_ADDRESS>, executor driver, partition 9, PROCESS_LOCAL, 46489664 bytes)
24/03/26 18:18:41 INFO TaskSetManager: start serialize task 10...
24/03/26 18:18:41 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 10)
24/03/26 18:18:41 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 10)
24/03/26 18:18:43 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 10)
24/03/26 18:18:43 INFO TaskSetManager: finish serialize task 10...
24/03/26 18:18:43 INFO TaskSetManager: Starting task 10.0 in stage 2.0 (TID 10) (<IP_ADDRESS>, executor driver, partition 10, PROCESS_LOCAL, 46488355 bytes)
24/03/26 18:18:43 INFO TaskSetManager: start serialize task 11...
24/03/26 18:18:43 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 11)
24/03/26 18:18:43 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 11)
24/03/26 18:18:44 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 11)
24/03/26 18:18:44 INFO TaskSetManager: finish serialize task 11...
24/03/26 18:18:44 INFO TaskSetManager: Starting task 11.0 in stage 2.0 (TID 11) (<IP_ADDRESS>, executor driver, partition 11, PROCESS_LOCAL, 46488438 bytes)
24/03/26 18:18:44 INFO TaskSetManager: start serialize task 12...
24/03/26 18:18:44 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 12)
24/03/26 18:18:44 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 12)
24/03/26 18:18:46 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 12)
24/03/26 18:18:46 INFO TaskSetManager: finish serialize task 12...
24/03/26 18:18:46 INFO TaskSetManager: Starting task 12.0 in stage 2.0 (TID 12) (<IP_ADDRESS>, executor driver, partition 12, PROCESS_LOCAL, 46488729 bytes)
24/03/26 18:18:46 INFO TaskSetManager: start serialize task 13...
24/03/26 18:18:46 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 13)
24/03/26 18:18:46 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 13)
24/03/26 18:18:48 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 13)
24/03/26 18:18:48 INFO TaskSetManager: finish serialize task 13...
24/03/26 18:18:48 INFO TaskSetManager: Starting task 13.0 in stage 2.0 (TID 13) (<IP_ADDRESS>, executor driver, partition 13, PROCESS_LOCAL, 46486309 bytes)
24/03/26 18:18:48 INFO TaskSetManager: start serialize task 14...
24/03/26 18:18:48 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 14)
24/03/26 18:18:48 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 14)
24/03/26 18:18:50 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 14)
24/03/26 18:18:50 INFO TaskSetManager: finish serialize task 14...
24/03/26 18:18:50 INFO TaskSetManager: Starting task 14.0 in stage 2.0 (TID 14) (<IP_ADDRESS>, executor driver, partition 14, PROCESS_LOCAL, 46489736 bytes)
24/03/26 18:18:50 INFO TaskSetManager: start serialize task 15...
24/03/26 18:18:50 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 15)
24/03/26 18:18:50 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 15)
24/03/26 18:18:51 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 15)
24/03/26 18:18:51 INFO TaskSetManager: finish serialize task 15...
24/03/26 18:18:51 INFO TaskSetManager: Starting task 15.0 in stage 2.0 (TID 15) (<IP_ADDRESS>, executor driver, partition 15, PROCESS_LOCAL, 46489059 bytes)
24/03/26 18:18:51 INFO TaskSetManager: start serialize task 16...
24/03/26 18:18:51 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 16)
24/03/26 18:18:51 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 16)
24/03/26 18:18:53 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 16)
24/03/26 18:18:53 INFO TaskSetManager: finish serialize task 16...
24/03/26 18:18:53 INFO TaskSetManager: Starting task 16.0 in stage 2.0 (TID 16) (<IP_ADDRESS>, executor driver, partition 16, PROCESS_LOCAL, 46488938 bytes)
24/03/26 18:18:53 INFO TaskSetManager: start serialize task 17...
24/03/26 18:18:53 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 17)
24/03/26 18:18:53 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 17)
24/03/26 18:18:54 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 17)
24/03/26 18:18:54 INFO TaskSetManager: finish serialize task 17...
24/03/26 18:18:54 INFO TaskSetManager: Starting task 17.0 in stage 2.0 (TID 17) (<IP_ADDRESS>, executor driver, partition 17, PROCESS_LOCAL, 46485930 bytes)
24/03/26 18:18:54 INFO TaskSetManager: start serialize task 18...
24/03/26 18:18:54 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 18)
24/03/26 18:18:54 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 18)
24/03/26 18:18:56 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 18)
24/03/26 18:18:56 INFO TaskSetManager: finish serialize task 18...
24/03/26 18:18:56 INFO TaskSetManager: Starting task 18.0 in stage 2.0 (TID 18) (<IP_ADDRESS>, executor driver, partition 18, PROCESS_LOCAL, 46486144 bytes)
24/03/26 18:18:56 INFO TaskSetManager: start serialize task 19...
24/03/26 18:18:56 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 19)
24/03/26 18:18:56 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 19)
24/03/26 18:18:58 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 19)
24/03/26 18:18:58 INFO TaskSetManager: finish serialize task 19...
24/03/26 18:18:58 INFO TaskSetManager: Starting task 19.0 in stage 2.0 (TID 19) (<IP_ADDRESS>, executor driver, partition 19, PROCESS_LOCAL, 46491094 bytes)
24/03/26 18:18:58 INFO TaskSetManager: start serialize task 20...
24/03/26 18:18:58 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 20)
24/03/26 18:18:58 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 20)
24/03/26 18:18:59 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 20)
24/03/26 18:18:59 INFO TaskSetManager: finish serialize task 20...
24/03/26 18:18:59 INFO TaskSetManager: Starting task 20.0 in stage 2.0 (TID 20) (<IP_ADDRESS>, executor driver, partition 20, PROCESS_LOCAL, 46492332 bytes)
24/03/26 18:18:59 INFO TaskSetManager: start serialize task 21...
24/03/26 18:18:59 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 21)
24/03/26 18:18:59 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 21)
24/03/26 18:19:01 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 21)
24/03/26 18:19:01 INFO TaskSetManager: finish serialize task 21...
24/03/26 18:19:01 INFO TaskSetManager: Starting task 21.0 in stage 2.0 (TID 21) (<IP_ADDRESS>, executor driver, partition 21, PROCESS_LOCAL, 46485583 bytes)
24/03/26 18:19:01 INFO TaskSetManager: start serialize task 22...
24/03/26 18:19:01 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 22)
24/03/26 18:19:01 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 22)
24/03/26 18:19:03 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 22)
24/03/26 18:19:03 INFO TaskSetManager: finish serialize task 22...
24/03/26 18:19:03 INFO TaskSetManager: Starting task 22.0 in stage 2.0 (TID 22) (<IP_ADDRESS>, executor driver, partition 22, PROCESS_LOCAL, 46488872 bytes)
24/03/26 18:19:03 INFO TaskSetManager: start serialize task 23...
24/03/26 18:19:03 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 23)
24/03/26 18:19:03 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 23)
24/03/26 18:19:04 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 23)
24/03/26 18:19:04 INFO TaskSetManager: finish serialize task 23...
24/03/26 18:19:04 INFO TaskSetManager: Starting task 23.0 in stage 2.0 (TID 23) (<IP_ADDRESS>, executor driver, partition 23, PROCESS_LOCAL, 46489351 bytes)
24/03/26 18:19:04 INFO TaskSetManager: start serialize task 24...
24/03/26 18:19:04 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 24)
24/03/26 18:19:04 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 24)
24/03/26 18:19:06 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 24)
24/03/26 18:19:06 INFO TaskSetManager: finish serialize task 24...
24/03/26 18:19:06 INFO TaskSetManager: Starting task 24.0 in stage 2.0 (TID 24) (<IP_ADDRESS>, executor driver, partition 24, PROCESS_LOCAL, 46486199 bytes)
24/03/26 18:19:06 INFO TaskSetManager: start serialize task 25...
24/03/26 18:19:06 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 25)
24/03/26 18:19:06 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 25)
24/03/26 18:19:08 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 25)
24/03/26 18:19:08 INFO TaskSetManager: finish serialize task 25...
24/03/26 18:19:08 INFO TaskSetManager: Starting task 25.0 in stage 2.0 (TID 25) (<IP_ADDRESS>, executor driver, partition 25, PROCESS_LOCAL, 46485572 bytes)
24/03/26 18:19:08 INFO TaskSetManager: start serialize task 26...
24/03/26 18:19:08 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 26)
24/03/26 18:19:08 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 26)
24/03/26 18:19:09 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 26)
24/03/26 18:19:09 INFO TaskSetManager: finish serialize task 26...
24/03/26 18:19:09 INFO TaskSetManager: Starting task 26.0 in stage 2.0 (TID 26) (<IP_ADDRESS>, executor driver, partition 26, PROCESS_LOCAL, 46486711 bytes)
24/03/26 18:19:09 INFO TaskSetManager: start serialize task 27...
24/03/26 18:19:09 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 27)
24/03/26 18:19:09 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 27)
24/03/26 18:19:11 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 27)
24/03/26 18:19:11 INFO TaskSetManager: finish serialize task 27...
24/03/26 18:19:11 INFO TaskSetManager: Starting task 27.0 in stage 2.0 (TID 27) (<IP_ADDRESS>, executor driver, partition 27, PROCESS_LOCAL, 46491886 bytes)
24/03/26 18:19:11 INFO TaskSetManager: start serialize task 28...
24/03/26 18:19:11 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 28)
24/03/26 18:19:11 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 28)
24/03/26 18:19:13 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 28)
24/03/26 18:19:13 INFO TaskSetManager: finish serialize task 28...
24/03/26 18:19:13 INFO TaskSetManager: Starting task 28.0 in stage 2.0 (TID 28) (<IP_ADDRESS>, executor driver, partition 28, PROCESS_LOCAL, 46490368 bytes)
24/03/26 18:19:13 INFO TaskSetManager: start serialize task 29...
24/03/26 18:19:13 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 29)
24/03/26 18:19:13 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 29)
24/03/26 18:19:14 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 29)
24/03/26 18:19:14 INFO TaskSetManager: finish serialize task 29...
24/03/26 18:19:14 INFO TaskSetManager: Starting task 29.0 in stage 2.0 (TID 29) (<IP_ADDRESS>, executor driver, partition 29, PROCESS_LOCAL, 46489395 bytes)
24/03/26 18:19:14 INFO TaskSetManager: start serialize task 30...
24/03/26 18:19:14 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 30)
24/03/26 18:19:14 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 30)
24/03/26 18:19:16 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 30)
24/03/26 18:19:16 INFO TaskSetManager: finish serialize task 30...
24/03/26 18:19:16 INFO TaskSetManager: Starting task 30.0 in stage 2.0 (TID 30) (<IP_ADDRESS>, executor driver, partition 30, PROCESS_LOCAL, 46488916 bytes)
24/03/26 18:19:16 INFO TaskSetManager: start serialize task 31...
24/03/26 18:19:16 INFO JavaSerializerInstance: before serialize stream ShuffleMapTask(2, 31)
24/03/26 18:19:16 INFO JavaSerializerInstance: after serialize stream ShuffleMapTask(2, 31)
24/03/26 18:19:17 INFO JavaSerializerInstance: after write object ShuffleMapTask(2, 31)

Thanks for all the info @wangxiaoying. I can take a look at it next week.
In the meantime, can you confirm that the operations executed in postgres and in Spark with SparkSQL are the same when executed in Wayang? In Wayang you can actually force each operator where to be executed with the .withPlatform() method so that you make sure that the plans are the same.

Thank you @zkaoudi for the quick response!

In the meantime, can you confirm that the operations executed in postgres and in Spark with SparkSQL are the same when executed in Wayang?

Yes. I checked the log at postgres side.  It seems like sparksql would generate additional filters with "IS NOT NULL", but it won't really filter our any data since the TPC-H dataset does not contain NULL value. In addition it didn't pushdown the LIKE 'BUILDING%' predicate as wayang does, which may cause more data to transfer for spark (although the table is not that big comparing to lineitem).
P.S.
Another thing I found when I check the log is that spark would issue the three sql queries in parallel, while wayang issue them one by one. I tried to enable parallelism in wayang by setting wayang.core.optimizer.enumeration.parallel-tasks = true, however it gives me an exception:
        Exception in thread "Thread-0" java.util.ConcurrentModificationException
	at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1134)
	at org.apache.wayang.core.platform.CrossPlatformExecutor.getOrCreateExecutorFor(CrossPlatformExecutor.java:391)
	at org.apache.wayang.core.platform.CrossPlatformExecutor$ParallelExecutionThread.run(CrossPlatformExecutor.java:1104)
	at java.base/java.lang.Thread.run(Thread.java:834)

I think it is due to the racing on the executors member inside the CrossPlatformExecutor.java.
This can be one of the reason for the performance difference, but I think the later execution difference inside spark platform is more significant in terms of the whole query.

Hi @wangxiaoying,
before digging into the details and just to make sure we are comparing same installations, I was wondering whether you are using the same Spark version for both runs.

The performance difference also stems from the current implementation of the Postgres To Spark channel conversion; https://github.com/apache/incubator-wayang/blob/e89191332dc9357526c8342cf4706652b3ebaa74/wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/operators/SqlToRddOperator.java#L46

Hi @wangxiaoying,
before digging into the details and just to make sure we are comparing same installations, I was wondering whether you are using the same Spark version for both runs.

Sorry for the late reply, I was out for last week. Yes, I can confirm this.

The type of join that Spark SQL uses. Wayang's current join operator maps to the corresponding join in RDDs, which if I'm not mistaken is implemented as as hash join. Maybe Spark SQL uses a broadcast join and thus, the difference in the data transferred?

Yes, I think the executed join algorithms are different from the two approaches. Below is the default physical plan generated by spark:
+- == Final Plan ==
   *(8) Sort [revenue#138 DESC NULLS LAST, o_orderdate#58 ASC NULLS FIRST], true, 0
   +- AQEShuffleRead coalesced
      +- ShuffleQueryStage 5
         +- Exchange rangepartitioning(revenue#138 DESC NULLS LAST, o_orderdate#58 ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [plan_id=448]
            +- *(7) HashAggregate(keys=[l_orderkey#0, o_orderdate#58, o_shippriority#61], functions=[sum((l_extendedprice#5 * (1 - l_discount#6)))])
               +- *(7) HashAggregate(keys=[l_orderkey#0, o_orderdate#58, o_shippriority#61], functions=[partial_sum((l_extendedprice#5 * (1 - l_discount#6)))])
                  +- *(7) Project [o_orderdate#58, o_shippriority#61, l_orderkey#0, l_extendedprice#5, l_discount#6]
                     +- *(7) SortMergeJoin [o_orderkey#54], [l_orderkey#0], Inner
                        :- *(5) Sort [o_orderkey#54 ASC NULLS FIRST], false, 0
                        :  +- AQEShuffleRead coalesced
                        :     +- ShuffleQueryStage 4
                        :        +- Exchange hashpartitioning(o_orderkey#54, 200), ENSURE_REQUIREMENTS, [plan_id=341]
                        :           +- *(4) Project [o_orderkey#54, o_orderdate#58, o_shippriority#61]
                        :              +- *(4) BroadcastHashJoin [c_custkey#36], [o_custkey#55], Inner, BuildLeft, false
                        :                 :- BroadcastQueryStage 3
                        :                 :  +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=225]
                        :                 :     +- AQEShuffleRead local
                        :                 :        +- ShuffleQueryStage 0
                        :                 :           +- Exchange hashpartitioning(c_custkey#36, 200), ENSURE_REQUIREMENTS, [plan_id=132]
                        :                 :              +- *(1) Project [c_custkey#36]
                        :                 :                 +- *(1) Filter (staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, c_mktsegment#42, 10, true, false, true) = BUILDING  )
                        :                 :                    +- *(1) Scan JDBCRelation(public.customer) [numPartitions=1] [c_custkey#36,c_mktsegment#42] PushedFilters: [*IsNotNull(c_custkey)], ReadSchema: struct<c_custkey:int,c_mktsegment:string>
                        :                 +- AQEShuffleRead local
                        :                    +- ShuffleQueryStage 1
                        :                       +- Exchange hashpartitioning(o_custkey#55, 200), ENSURE_REQUIREMENTS, [plan_id=139]
                        :                          +- *(2) Scan JDBCRelation(public.orders) [numPartitions=1] [o_orderkey#54,o_custkey#55,o_orderdate#58,o_shippriority#61] PushedFilters: [*IsNotNull(o_orderdate), *LessThan(o_orderdate,1995-03-15), *IsNotNull(o_custkey), *IsNotNull(o_..., ReadSchema: struct<o_orderkey:int,o_custkey:int,o_orderdate:date,o_shippriority:int>
                        +- *(6) Sort [l_orderkey#0 ASC NULLS FIRST], false, 0
                           +- AQEShuffleRead coalesced
                              +- ShuffleQueryStage 2
                                 +- Exchange hashpartitioning(l_orderkey#0, 200), ENSURE_REQUIREMENTS, [plan_id=150]
                                    +- *(3) Scan JDBCRelation(public.lineitem) [numPartitions=1] [l_orderkey#0,l_extendedprice#5,l_discount#6] PushedFilters: [*IsNotNull(l_shipdate), *GreaterThan(l_shipdate,1995-03-15), *IsNotNull(l_orderkey)], ReadSchema: struct<l_orderkey:int,l_extendedprice:decimal(15,2),l_discount:decimal(15,2)>

I tried to add config: .config("spark.sql.join.preferSortMergeJoin", "false") when building the spark session so the SortMergeJoin above will become a HashJoin, but the performance does not changes much. BoradcastJoin is still used though.

I'm not very familiar with the views in Spark, but when one registers the temporary views are they materialized in memory? If so, the timer you have would measure data accessed via memory. But again not sure how the temp views in Spark work. Maybe you could time the registerviews method to check this out.

Spark uses lazy evaluation so the view creation does not take much time (only some metadata will be fetched). And as I have shown above in the postgres log, spark does fetch the three tables (with projection and filter pushdown) during runtime like wayang does.

Thanks @wangxiaoying. I guess the broadcast join reduces the amount of data shuffled for this specific dataset/query. Could you disable the broadcast join in Spark to make sure if the difference comes from the join only?
Sth like: spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1) That would at least make equal the amount of data shuffled to better understand the performance difference.

To go around the potential performance issue of the Postgres To Spark channel conversion you could add the Java platform and see what you get. So sth like: ./wayang-1.0.0-SNAPSHOT/bin/wayang-submit org.apache.wayang.apps.tpch.TpcH exp(123) spark,postgres,java file:///path/to/wayang.properties Q3

Thanks @zkaoudi for the suggestion. Adding java improves the overall performance (since the final plan won't involve the usage of spark). It results in 44 seconds (similar but still a little bit inferior comparing to sparksql (40 seconds).


search filters

update search bar to filter courses, the filter will allow the lower and upper case,  will be responsive while typing a text.

The filter may need to be a dropdown menu of categories, that way we can just add each item to a specific category.

kind of is gonna be a block-inline on top of the search bar
Java  | JS  | C++ | React | etc
search bar here


restart to update issue

VSCode Version: 1.9.1
OS Version:MacOS

Steps to Reproduce:

Click on code->restart to update and it closes all windows of vscode but doesn't updates and not even the application opens automatically

This is a duplicate of #7426.  @shaddu the issue lists how to fix the issue.


Initial Setup: Remove "Add Bot" step and move intents setup

Actually I can’t merge so granny or vankka will need to


PDB Plugin Change Users Assets to Auto Assets

Duplicate symbol support now exists.
Fixed in 2.3.2731-dev.


MINOR: update collaborator (v3.4.1 release)

update collaborator while releasing v3.4.1.
 git shortlog --email --numbered --summary --since=2022-06-01

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

I wonder if we should wait until the next major release (3.6) before updating it given that we put this list together 2-3 weeks ago. It is a bit weird to already remove folks now.

We could just do the reordering, to see if the invites are sent again (for the 2 that didn't accept in time)

I would also opt for major releases. Minor releases are usually done around major releases and sometimes we also do multiple minor releases around a major release. So we would update the list quite often.
I am also wondering what happens to the people that are dropped from the list.

I wonder if we should wait until the next major release (3.6) before updating it given that we put this list together 2-3 weeks ago. It is a bit weird to already remove folks now.

Hmm... yes, it's only less than one month since we updated the list. Agree! Let's close this PR now and wait for v3.5.1 or v3.6.0 to update it.
cc @satishd  who is the release manager of v3.6.0.

@showuon we could still do the reordering to see if the invites are sent again

@showuon we could still do the reordering to see if the invites are sent again

Let's give it a try!


Screens for Carts and CartItems (READ, UPDATE, DELETE)

Cart

[x] Read
[x] Checkout cart (POST to ~/Orders)
[ ] Update (postponed)
[ ] Delete (postponed)

CartItems

[x] Read
[ ] Update (postponed)
[ ] Delete (postponed)

Check out card is implemented and works, except that there is the same problem like with the approval: API return a job and sends a wrong contect type (appplication/xml instead of application/atom+xml)


Settings window not responding

Environment
Windows build number: 10.0.20161.1000
PowerToys version: v0.19.1
PowerToy module for which you are reporting the bug (if applicable):

Steps to reproduce

Right click on the PowerToys icon in the taskbar.
Click on Settings.
Try clicking on anything inside the window.

Expected behavior
The window will respond to taps or clicks on buttons or any other links in the Settings window.
Actual behavior
The window does not respond and freezes. Sometimes, it even closes without warning. (Tested on Surface Book 2)
Screenshots

I have similar problem. PowerRun doesnt work and I went to settings and settings open. On click of PowerRun menu, it crashes and also Key Board Manager actions buttons don't respond either - Surface Laptop 3. I'm not sure when this issue started of sudden.

@nitishvijai / @karthik20  does anything show up in event viewer?
Are you running PT under admin?

@nitishvijai / @karthik20  does anything show up in event viewer?
Are you running PT under admin?

Yes. I had it uninstall and just now, I reinstalled from Winget. It crashes the same way on click on Power Toys Run. I'm not sure if this has anything to do specific to 'Surface' after recent firmware updates etc.,

@nitishvijai with 0.20 and our pending 0.21 release, we've done a lot of improvements and updated some nuget packages that may have been affecting this.  Can you verify this still happen?

@crutkas Hello Clint, I'm on v0.20.1 and I haven't noticed any issues with the Settings window yet. I'll post more on here if I come across anything.
So far, so good!

@crutkas , I still have the same issue. I updated yesterday and it crashes the same way. Surface Laptop 3.
I also get an error trace on open:
Version: 1.0.0 OS Version: Microsoft Windows NT 10.0.19041.0 IntPtr Length: 8 x64: True Date: 08/29/2020 10:39:52 Exception: System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.    at _CxxThrowException(Void* , _s__ThrowInfo* )    at interop.KeyboardHook.Start()    at interop.HotkeyManager..ctor()    at PowerLauncher.ViewModel.MainViewModel..ctor(Settings settings)    at PowerLauncher.App.<>c__DisplayClass16_0.<OnStartup>b__1()    at Wox.Infrastructure.Stopwatch.Normal(String message, Action action)    at PowerLauncher.App.OnStartup(Object sender, StartupEventArgs e)    at System.Windows.Application.OnStartup(StartupEventArgs e)    at System.Windows.Application.<.ctor>b__1_0(Object unused)    at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)    at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

@karthik20 this trace is different than what is being reported here.

@karthik20  #5414 would be the tracking issue for that.
@nitishvijai since you're not having issues, i'm going to close this and we are tracking the other failure


FlutterActivity transparent mode throw UnsupportedOperationException

.backgroundMode(transparent) throw UnsupportedOperationException
var intent = FlutterActivity
                    .withNewEngine()
                    .backgroundMode(FlutterActivityLaunchConfigs.BackgroundMode.transparent)
                    .build(context!!)

java.lang.RuntimeException: Unable to start activity ComponentInfo{com.freewheelin.pulley.debug/io.flutter.embedding.android.FlutterActivity}: java.lang.UnsupportedOperationException: TextureView doesn't support displaying a background drawable
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2974)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3059)
        at android.app.ActivityThread.-wrap11(Unknown Source:0)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1724)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:164)
        at android.app.ActivityThread.main(ActivityThread.java:7000)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:441)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1408)
     Caused by: java.lang.UnsupportedOperationException: TextureView doesn't support displaying a background drawable
        at android.view.TextureView.setBackgroundDrawable(TextureView.java:316)
        at android.view.View.setBackground(View.java:21731)
        at android.view.View.<init>(View.java:5591)
        at android.view.View.<init>(View.java:5017)
        at android.view.View.<init>(View.java:4996)
        at android.view.TextureView.<init>(TextureView.java:146)
        at io.flutter.embedding.android.FlutterTextureView.<init>(FlutterTextureView.java:101)
        at io.flutter.embedding.android.FlutterTextureView.<init>(FlutterTextureView.java:94)
        at io.flutter.embedding.android.FlutterView.init(FlutterView.java:214)
        at io.flutter.embedding.android.FlutterView.<init>(FlutterView.java:199)
        at io.flutter.embedding.android.FlutterView.<init>(FlutterView.java:179)
        at io.flutter.embedding.android.FlutterActivityAndFragmentDelegate.onCreateView(FlutterActivityAndFragmentDelegate.java:245)
        at io.flutter.embedding.android.FlutterActivity.createFlutterView(FlutterActivity.java:532)
        at io.flutter.embedding.android.FlutterActivity.onCreate(FlutterActivity.java:425)
        at android.app.Activity.performCreate(Activity.java:7258)
        at android.app.Activity.performCreate(Activity.java:7249)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1222)

[✓] Flutter (Channel beta, v1.14.6, on Mac OS X 10.14.6 18G103, locale ko-KR)
    • Flutter version 1.14.6 at /Users/ht/project/flutter
    • Framework revision fabeb2a16f (4 weeks ago), 2020-01-28 07:56:51 -0800
    • Engine revision c4229bfbba
    • Dart version 2.8.0 (build 2.8.0-dev.5.0 fc3af737c7)

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/ht/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling
      support)
    • Platform android-29, build-tools 29.0.3
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_202-release-1483-b49-5587405)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.3.1, Build version 11C504
    • CocoaPods version 1.8.4

[✓] Android Studio (version 3.5)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 43.0.1
    • Dart plugin version 191.8593
    • Java version OpenJDK Runtime Environment (build
      1.8.0_202-release-1483-b49-5587405)

[!] IntelliJ IDEA Ultimate Edition (version 2019.3.3)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • For information about installing plugins, see
      https://flutter.dev/intellij-setup/#installing-the-plugins

[!] Connected device
    ! No devices available

Hi @hyuntae0117
Are you still experiencing this issue?
Thank you

Yes i am
Please help me
-----Original Message-----
From: Taha Tesser<EMAIL_ADDRESS>To: flutter/flutter<EMAIL_ADDRESS>Cc: Chewdy<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Sent: 2020-02-28 (금) 19:08:55 (GMT+09:00)
Subject: Re: [flutter/flutter] FlutterActivity transparent mode throw UnsupportedOperationException (#51626)
Hi @hyuntae0117
Are you still experiencing this issue?
Thank you
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Hi @hyuntae0117
Please refer to this comment for possible solution
Doc: Step 2: Start FlutterActivity with transparency
"FlutterActivityLaunchConfigs is not public" that's probably the reason you can't access this
Related to 45747

Thank you!
But I already read it!
I could not find FlutterActivity.BackgroundMode.transparent,
so I used FlutterActivityLaunchConfigs.BackgroundMode.transparent
Is there something missing?

any changes..?

I found that renderSurface is null in this constructor.
  private FlutterView(
      @NonNull Context context,
      @Nullable AttributeSet attrs,
      @NonNull FlutterTextureView flutterTextureView) {
    super(context, attrs);

    this.flutterTextureView = flutterTextureView;
    this.renderSurface = flutterSurfaceView;

    init();
  }```

   `this.renderSurface = flutterSurfaceView;` should be `    this.renderSurface = flutterTextureView;`

@hyuntae0117 : You can switch to Stable channel, intent.putExtra("background_mode","transparent"); activity.startActivity(intent) will work fine.
But I have another trouble:

Transparent Mode: If you call showModalBottomSheet that has a child is TextField, the keyboard will overlaps ModalBottomSheet.
Opaque Mode: work fine.

@lts1610 Thank you!
I will check it later

Is there any way to make a flutter application transparent (aka see the background through the application) without using Flutter Fragments? @lts1610

any fix found for it?

This issue is not actionable as there was no repro provided.


Add CDR technologies to final-energy reporting

@gidden @jayfuhrman, are you sure that these additional technologies should be added to the final-energy reporting? In other words, are their energy-requirements so large as to be relevant?

Thanks for the answer - will merge this PR if @gidden approves.

LGTM, thanks!


WIP net/frr: Add BGP password support

replacement for https://github.com/opnsense/plugins/pull/2645, adding setkey rafactor in https://github.com/opnsense/plugins/commit/782cedc94945bc058ad1e56de8349e5fbdb544ab

@AdSchellevis this doesn't look right from GitHub

@fichtner hmm, this is odd, let me try to rebase again and see if this is fixable. GitHub was also down from my end yesterday evening, odd

if @mimugmail agrees with the content, we should squash merge it and rewrite the commit message, the trail is a bit messy now.

Sure, love it. Thx for your help :)

@mimugmail you're welcome, it's merged

@AdSchellevis works great on my end. Let's let it sleep in devel until 22.1.1 so ppl will have time to test a devel pkg :)


Update CouchDB package deployed in central services

Impact of the new feature
CouchDB
Is your feature request related to a problem? Please describe.
As we move central services to python3, we also have to move to the python3 couchapp library, which is shipped with the couchdb16.spec spec file.
I just happened to notice that we are actually deploying both couchdb specs (1.5 brings py2-cmscouchapp; 1.6 brings py3-cmscouchapp), as can be seen here [1], both in our private and in CMSWEB VMs. However, the spec that we are running is couchdb15, which honestly is exactly the same service as couchdb16, where the only difference is the python cmscouchapp library.
Some background can be seen in this GH issue (and the respective PR): https://github.com/dmwm/WMCore/issues/10489
Describe the solution you'd like
Update this deployment couchdb line:
https://github.com/dmwm/deployment/blob/master/couchdb/deploy#L22
such that it starts deploying and running the couchdb16 package instead.
AFAIK, it's going to be transparent to wmagent and t0, since the package dependency is properly defined in its own spec file.
Describe alternatives you've considered
Only make this switch when all the CouchDB-based services have migrated to python3.
Additional context
[1]
[cmsweb@vocms0731 current]$ ls sw/slc7_amd64_gcc630/external/couchdb1*
sw/slc7_amd64_gcc630/external/couchdb15:
1.6.1-comp10

sw/slc7_amd64_gcc630/external/couchdb16:
1.6.1-comp

Fixed by https://github.com/dmwm/deployment/pull/1089 and everything looks fine in testbed. Closing


IOS FCM.getInitialNotification() is always return null

null or undefined?

can you provide details?

I have fixed it. I was not using the id key in my notification payload. I think it should be written somewhere in docs.

@pankaj-ag I will include this in the next revision of the docs, where did you include your id key in the FCM payload, in the notification object or in the data object?

a pull is welcome

@evollu will do in a couple days


Work on serialization

Add lots of tests, deserialize the CrossSentenceEventMention, output groupModifier

The serialization has just a little ways to go.  It has small discrepencies in ordering on a round trip test.  That's why it isn't quite ready for master.  It is, however, being used for the DART reading.  There we're not deserializing yet.  It's getting really close.


fix(plugin-optimize): read remote js file throws an error

Changes

When remote script is included in html file, the remote url will be scanned and an error will be thrown. So, if it is a remote URL, return the result, do not read the file by fs.

Testing

Docs

Thank you!

Thank you!


docs: Fix pipe_grant doc

https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/pipe_grant
This PR fix https://github.com/Snowflake-Labs/terraform-provider-snowflake/issues/985
current pipe_grant document contains sequence_name = "sequence" but it must be like `pipe_name = "pipe"

Test Plan

[x] acceptance tests

[ ] …

References

https://github.com/Snowflake-Labs/terraform-provider-snowflake/issues/985

@sfc-gh-swinkler done

@sfc-gh-swinkler would you check this PR?

/ok-to-test sha=88397e2

@sfc-gh-swinkler done

Sorry @yohei1126, could you rebase one more time? I was working on some other changes and merged them in before checking back on this PR

done

/ok-to-test sha=066cb34


Arangosh, how do I bring it up in Terminal? Can't find it.

My Environment

ArangoDB Version:      3.6.5
Storage Engine:             RocksDB
Deployment Mode:       Single Server
Deployment Strategy:   Manual Start
Configuration:               Normal
Infrastructure:               Own
Operating System:        MacOS 10.15.6
Total RAM in your machine:        16GB
Disks in use:        SSD
Used Package:              --

Component, Query & Data
Affected feature:
arangsosh
AQL query (if applicable):
AQL explain (if applicable):

Dataset:

Size of your Dataset on disk:

Replication Factor & Number of Shards (Cluster only):

Steps to reproduce

Problem:
Accessing arangosh prompt
Expected result:

if you use the MacOS DMG-Installer, you should have a wrapper script called arangosh  under the mountpoint of the DMG-image /Arangodb3-CLI.app/Contents/Resources . Other files will be kept in~/opt/arangodb/....

Thank you. I believe I used the MacOS DMG installer. Even when I show hidden files, the only thing that appears in Finder under Applications is Arangodb3-CLI.app. There are no files under that that I can see. I was expecting to see /Contents/Resources there, but don't see them. When I do a search for them in Finder, even with the hidden files shown, nothing appears.

Hi,
DMG stands for DiskiMaGe; hence when you click it, and move the icon over, your mac will mount it.
if you open the terminal and type mount it should enlist the mountpoint.

Thanks. I located the "Contents" folder when I hovered the Arango icon. Since the Contents folder disappears when I move the icon into the Applications icon, I decided to save the Contents folder in Applications so I could get to the arangosh. Will that cause any problems if I copy it anew anytime I upgrade ArangoDB app? I am not able to open the arangosh by opening Terminal and typing in "mount". It does appear I can open arangosh by going to it in the Contents folder I copied and opening arangosh with the Terminal app. The only problem is that it asks for a password and when I enter it, I get error message. That is the same password that gives me access to my databases via my browser. Given the latter, I am going to back burner trying to do anything in arangosh for the time being. Thank you again for your help.

do you have utf-8 signs or umlauts in your password?
I think we fixed the shell-script wrappers to properly handle blanks in passwords - you need to have them wrapped in quotes.

Thanks. I will check password.

My password contains one * and one #, neither of which are at beginning of password.

hm, # is the starting of a comment in the shell - unless you quote it or escape it. That could be the problem when specifying it at the commandline - but should work if you use the dialog for it.

@rusty53 Did you have a chance to try a password without # or escape it?

Thanks for the follow up, Simran. Not yet, will let you know.

Hi Simran.
I changed the arangod.conf to “false” for authentication.
I noticed I have three files title “arangod.conf” (that doesn’t seem right).
I changed all to “false”.
Now when I run, I go right into my root without the opportunity to set a password.
Rusty.

On Sep 9, 2020, at 06:52, Simran<EMAIL_ADDRESS>wrote:
@rusty53 https://github.com/rusty53 Did you have a chance to try a password without # or escape it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/arangodb/arangodb/issues/12435#issuecomment-689484204, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPBUCKCG2YI6LVMQJUJ3HTSE5M6NANCNFSM4P73OIIA.

I changed the arangod.conf to “true” but I get a “Login failed” when I try to insert new password.
Rusty

On Sep 9, 2020, at 06:52, Simran<EMAIL_ADDRESS>wrote:
@rusty53 https://github.com/rusty53 Did you have a chance to try a password without # or escape it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/arangodb/arangodb/issues/12435#issuecomment-689484204, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPBUCKCG2YI6LVMQJUJ3HTSE5M6NANCNFSM4P73OIIA.

you should be able to stop the installation, and use arangodb-secure-installation to set a new password.

Thank you. Will try it.

On Sep 10, 2020, at 03:46, Wilfried Goesgens<EMAIL_ADDRESS>wrote:
you should be able to stop the installation, and use arangodb-secure-installation to set a new password.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/arangodb/arangodb/issues/12435#issuecomment-690056609, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPBUCJZU2PXLKI622K3G3DSFB75TANCNFSM4P73OIIA.

I downloaded arangodb3-3.6.5.x86_64.dmg, clicked it to mount it, accepted the license and dragged ArangoDB3-CLI.app over to Applications to install it. Then I launched the app, which prompted for the storage engine (auto, rocksdb, mmfiles). After that it printed the following in a terminal:
ArangoDB server has been started

The database directory is located at
   '/Users/<username>/Library/ArangoDB/opt/arangodb/var/lib/arangodb3'

The log file is located at
   '/Users/<username>/Library/ArangoDB/opt/arangodb/var/log/arangodb3/arangod.log'

You can access the server using a browser at 'http://<IP_ADDRESS>:8529/'
or start the ArangoDB shell
   '/Applications/ArangoDB3-CLI.app/Contents/Resources/arangosh'

Switching to log-file now, closing this window will NOT stop the server.

Running /Applications/ArangoDB3-CLI.app/Contents/Resources/arangosh gets me into the ArangoShell. Even a password like ␣␣#*//?! (␣ = space) that I set via the Web UI works in the shell.
ps x | grep "arangod" reveals the command that the app uses to start the server:
/Applications/ArangoDB3-CLI.app/Contents/Resources/opt/arangodb/sbin/arangod -c /Users/<username>/Library/ArangoDB-etc/arangod.conf -c /Users/<username>/Library/ArangoDB-etc//arangod.conf --daemon --pid-file /var/tmp/arangod.pid
So the relevant server configuration file is ~/Library/ArangoDB-etc/arangod.conf (hard link, i.e. same file: /System/Volumes/Data/Users/<username>/Library/ArangoDB-etc/arangodb3/arangod.conf)
I'm not sure what the matter is with the other configuration files:

/Users/<username>/Library/ArangoDB-etc/arangodb3/arangod.conf
/Users/<username>/Library/ArangoDB-etc/arangod.conf-e

I suppose the correct way to restart the server is to determine the PID (ps x | grep "arangod"), kill <PID> and clicking the app to start the server again. Configuration changes should then take effect.
My attempt to reset the root password using arango-secure-installation was also successful:

Stop server
/Applications/ArangoDB3-CLI.app/Contents/Resources/arango-secure-installation
Enter new password twice
Launch app to start server

Note that if you change the server endpoint in the config, the app will still open http://<IP_ADDRESS>:8529 in the browser. You will have to adjust e.g. the port number to access the Web UI. Should there be a secondary server instance, e.g. managed via homebrew, then you might access the wrong instance by accident.

In case of a homebrew installation, the binaries are installed to:

/usr/local/opt/arangodb/sbin (server, init and secure installation binaries)
/usr/local/opt/arangodb/bin (client tools, including the starter)

brew install arangodb reported:
An empty password has been set. Please change it by executing
  /usr/local/opt/arangodb/sbin/arango-secure-installation

To have launchd start arangodb now and restart at login:
  brew services start arangodb
Or, if you don't want/need a background service you can just run:
  /usr/local/opt/arangodb/sbin/arangod

About certificates (openssl dependency)
A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
<EMAIL_ADDRESS>and run
<EMAIL_ADDRESS><EMAIL_ADDRESS>is keg-only, which means it was not symlinked into /usr/local,
because macOS provides LibreSSL.
If you need to have<EMAIL_ADDRESS>first in your PATH run:
echo 'export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"' >> ~/.profile
For compilers to find<EMAIL_ADDRESS>you may need to set:
export<EMAIL_ADDRESS>export<EMAIL_ADDRESS>I was able to set the password with spaces etc. using /usr/local/opt/arangodb/sbin/arango-secure-installation without problems.
After brew services start arangodb, brew services list reports:
Name     Status  User   Plist
arangodb started Simran /Users/<username>/Library/LaunchAgents/homebrew.mxcl.arangodb.plist

The following paths all reference the same server config file of the homebrew installation:

/usr/local/etc/arangodb3/arangod.conf
/usr/local/Cellar/arangodb/3.7.2-1/etc/arangodb3/arangod.conf
/System/Volumes/Data/usr/local/etc/arangodb3/arangod.conf
/System/Volumes/Data/usr/local/Cellar/arangodb/3.7.2-1/etc/arangodb3/arangod.conf


RxJava 2 Support

Are there any plans to upgrade to RxJava 2?
If I wanted to upgrade to project to RxJava 2 are you open to a pull request, or is it better to create a separate fork?

We are open for an rxjava 2 pr. We have been talking about an upgrade in Meltwater anyway but haven't gotten down to it yet. Rxjava 1 and 2 can coexist in downstream projects also, which is a good thing when it comes to migration paths.

Hey @karlney !
I will be more than happy to help you out with the RxJava 2 migration path. I will check the current wrappers this weekend and submit a first preview (shy) preview PR next week or so, sounds good?

@georgerb sounds perfect :-)