Split big requests to fit server-side limits

Is the dockerfile stuff really about Split big requests to fit server-side limits? Or is this PR combining two separate things?

Coverage increased (+0.06%) to 83.7% when pulling 2fca2daeef74847cc656cfb5739012eaf5bbe14b on PRED-750 into 05937de8f257ce8c98c34a90752d03dee09a893f on master.

Really it's 3 separate things:
splitting of requests
increasing auto-sampler target size
dockerizing of thing
but last two are really small ones...

I find that code review is easier (and higher quality) when we keep separate things separate... and it's nice for thing 1 to not be blocked by concerns about thing 2, etc.


Up/down keys cause un-handled errors to be thrown, version 2.1.3

Hello,
I installed newest version on this element,
If there is nothing in the textbox, when you press the up or down keys it seems to be attempting to scroll through the suggestions list,
It seems to me that is is due to the fact that there are no suggestions present that it is throwing errors
For some reason I am getting different errors on my test project..
Uncaught TypeError: Cannot read property '2' of undefined
    at HTMLElement._moveHighlighted (paper-autocomplete-suggestions.html:680)
    at HTMLElement._onKeypress (paper-autocomplete-suggestions.html:628)

I pulled down your demo file and I am getting different errors, but that was on master so the code may be different
paper-autocomplete-suggestions.html:685 Uncaught TypeError: Cannot read property 'classList' of undefined
    at HTMLElement._moveHighlighted (paper-autocomplete-suggestions.html:685)
    at HTMLElement._onKeypress (paper-autocomplete-suggestions.html:631)

Ups, what a stupid mistake on my side... Apologies. It is an easy fix. I have something already working, I just need to test it in all browsers and write a unit test for it.
I will be delivering the fix before dinner time :) (if you happen to be in central europe time zone)
About the demo site... I really need to update that. The demo site is showing the component in a much older version. I am afraid to demo the latest version you need to clone and run the component locally.

Fixed in v2.1.4. Again, many thanks for the feedback. I will try to get the demo page updated soon.

Thank you!


⚠ Bofrost.at has degraded performance

In ac8cbba, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 975 ms

Resolved: Bofrost.at performance has improved in 3225fd4.


Add the @discardableResult attribute to OrderedSet.insert(_:at:)

Fixes #19.
It does not seem appropriate to write a test for this change. However, I am open to other points of view and will gladly own further work.

Checklist

[x] I've read the Contribution Guidelines
[x] My contributions are licensed under the Swift license.
[x] I've followed the coding style of the rest of the project.
[ ] I've added tests covering all new code paths my change adds to the project (if appropriate).
[ ] I've added benchmarks covering new functionality (if appropriate).
[x] I've verified that my change does not break any existing tests or introduce unexplained benchmark regressions.
[x] I've updated the documentation if necessary.

@swift-ci test

I did wonder if the following functions should have this attribute, too:
updateOrAppend(_ item: Element) -> Element?
updateOrInsert(_ item: Element, at index: Index) -> (originalMember: Element?, index: Index)

@kielgillard I think so, if only by analogue to SetAlgebra's update(with:) requirement. Can you add those to this PR? 😊
(I wonder how often these would be used without looking at the return value, but there is nothing wrong with discarding it, either -- and I don't think we want to force people to do the _ = foo.updateOrAppend(...) dance to achieve that.)

@swift-ci test

@lorentey done.
My intuition is people will probably be using these "update or" APIs specifically for the return value but, like you said, there's no need for the API to be a jerk about it in case you don't want it!

Looks good, thanks! 👍
@swift-ci test


Metronome 0.6.11 bump on DCOS 1.11

High-level description
What features does this change enable? What bugs does this change fix?
Corresponding DC/OS tickets (obligatory)
These DC/OS JIRA ticket(s) must be updated (ideally closed) in the moment this PR lands:

DCOS-46578 Fixing test_metronome.test_metronome flake

Related tickets (optional)
Other tickets related to this change:

MARATHON-8555 Parameter --leader_proxy_max_open_connections does not actually allow more concurrent connections
MARATHON-8566 Fix race condition causing deployment info to not be available
MARATHON-8555 Include app-id in illegal-state-exception

Checklist for all PRs

[ ] Added a comprehensible changelog entry to CHANGES.md or explain why this is not a user-facing change:
[x] Included a test which will fail if code is reverted but test is not. If there is no test please explain here:
[x] Read the DC/OS contributing guidelines
[x] Followed relevant code rules Rules for Packages and Systemd

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[x] Change log from the last version integrated (this should be a link to commits for easy verification and review): diff
[x] Test Results: CI
[ ] Code Coverage (if available): N/A

@mesosphere-mergebot label ready for review


Added ability for additional, custom fingerprints.

As title, added the ability to specify a custom fingerprints file as an argument (optional). This will append the custom fingerprints to the main, original list of fingerprints.

Thanks so much for your contribution @SecGus ! :)
We also have a Discord server, which you’re more than welcome to join. It's a great place to connect with fellow contributors and stay updated with the latest developments!


asyncio error in quart/wrappers/request.py

service     | Traceback (most recent call last):
service     |   File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
service     |     return await fut
service     |            ^^^^^^^^^
service     |   File "/usr/local/lib/python3.12/site-packages/quart/wrappers/request.py", line 89, in __await__
service     |     yield from self._complete.wait().__await__()
service     |   File "/usr/local/lib/python3.12/asyncio/locks.py", line 212, in wait
service     |     await fut
service     | asyncio.exceptions.CancelledError
service     |
service     | The above exception was the direct cause of the following exception:
service     |
service     | Traceback (most recent call last):
service     |   File "/usr/local/lib/python3.12/site-packages/quart/wrappers/request.py", line 221, in get_data
service     |     raw_data = await asyncio.wait_for(self.body, timeout=self.body_timeout)
service     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
service     |   File "/usr/local/lib/python3.12/asyncio/tasks.py", line 519, in wait_for
service     |     async with timeouts.timeout(timeout):
service     |   File "/usr/local/lib/python3.12/asyncio/timeouts.py", line 115, in __aexit__
service     |     raise TimeoutError from exc_val
service     | TimeoutError
service     |
service     | During handling of the above exception, another exception occurred:
service     |
service     | Traceback (most recent call last):
service     |   File "/home/cloudsdk/service/src/login.py", line 27, in login
service     |     data = await request.get_json()
service     |            ^^^^^^^^^^^^^^^^^^^^^^^^
service     |   File "/usr/local/lib/python3.12/site-packages/quart/wrappers/request.py", line 315, in get_json
service     |     data = await self.get_data(cache=cache, as_text=True)
service     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
service     |   File "/usr/local/lib/python3.12/site-packages/quart/wrappers/request.py", line 223, in get_data
service     |     raise RequestTimeout()
service     | werkzeug.exceptions.RequestTimeout: 408 Request Timeout: The server closed the network connection because the browser didn't finish the request within the specified time.

I'm deploying this in a Docker container based upon python:3.12-slim-bookworm. The application startup but it can't receive any request by the client.
Environment:

Python version: 3.12.4
Quart version: 0.19.6 (I tried also with 0.19.2 and I have the same result).
I'm using Gunicorn with Uvicorn workers with timeout setted at 0. From gunicorn docs this mean that timeout is disabled.

Udpate:

Same error with python:3.11-slim-bookworm and python:3.10-slim-bookworm
I launched gunicorn with timeout set to 120 and the problem still persists.

Please provide a minimal reproducible example.

I will present here a minimal example.
login.py
from quart import Blueprint, request, session, jsonify
from quart_cors import cors, route_cors

app_route = Blueprint("login", __name__)
cors(app_route, **my settings**)

app_route.route("/api/login", methods=["POST"])
@route_cors()
async def login_route():
    try:
        data = await request.get_json() // crash here - check end of the log in first post.
    ....

requirement.txt
quart==0.19.6
quart_cors==0.7.0
Werkzeug==3.0.3

Dockerfile
FROM python:3.12-slim-bookworm

RUN apt-get update

RUN pip install --upgrade setuptools

RUN pip install --upgrade pip

RUN pip install uvicorn gunicorn

RUN pip install -r ./nanoserver/requirements.txt

Gunicorn is called in a docker-compose environment and launched with this configuration:
gunicorn service.src.app:app -b <IP_ADDRESS>:**PORT** -w 1 -k uvicorn.workers.UvicornWorker --timeout 0

I will present here a minimal example.
login.py
from quart import Blueprint, request, session, jsonify
from quart_cors import cors, route_cors

app_route = Blueprint("login", __name__)
cors(app_route, **my settings**)

app_route.route("/api/login", methods=["POST"])
@route_cors()
async def login_route():
    try:
        data = await request.get_json() // crash here - check end of the log in first post.
    ....

requirement.txt
quart==0.19.6
quart_cors==0.7.0
Werkzeug==3.0.3

Dockerfile
FROM python:3.12-slim-bookworm

COPY . /home/cloudsdk/

RUN apt-get update

RUN pip install --upgrade setuptools

RUN pip install --upgrade pip

RUN pip install uvicorn gunicorn

RUN pip install -r ./service/requirements.txt

Gunicorn is called in a docker-compose environment and launched with this configuration:
gunicorn service.src.app:app -b <IP_ADDRESS>:**PORT** -w 1 -k uvicorn.workers.UvicornWorker --timeout 0

Update:
Now the app in running normally.
Probably some error with my Docker or the distro?


React & Redux group project - Space Travelers' Hub

Hi, this week *no coding partner* has been assigned to me in my dashboard. Therefore, I am the only one have been working on this project.
Space Travelers' Hub: Live Demo
Highlights

Upon the first render fetch data from the SpaceX API endpoints.
Rendered UI lists accordingly
Written actions for booking rockets and joining missions
Written actions for canceling rockets and leaving missions
Rendered My Profile Section with booked rockets and missions list.

@Olamarx  Hi, thanks for the review, however, as per the instruction All tasks should be based on the cards from your Kanban board.
Writing tests are not mentioned as mandatory, So, it would be great if you could shed some light on that if it could be a blocker on PR approval, please?
Thanks


[release-4.16] CFE-1087: API Bump for capacity Reservation

Mannual backport for https://github.com/openshift/cluster-control-plane-machine-set-operator/pull/313

/retest

/retest

/retest

/approve
/lgtm
/label backport-risk-assessed

/retest

/retest

/label qe-approved

/label cherry-pick-approved

/retest

@sdodson Can you add jira/valid-bug label on the PR


use ArrayListMultimap instead of new java.util.HashMap[InternalRow, j…

ArrayListMultimap is a HashMap associates each key with an ArrayList of values.

do you have some performance comparison to show there is a improvement.

LGTM then.

@linhong-intel I took a look at the implementation of ArrayListMultimap, it is similar as the way our code is. So I think the performance would be similar too, but code looks simplified.
I'm merging this. Thanks @LuciferYang


Thread pool wait

Hi,
How to check if all tasks are finished in a pool? I am not explicitly creating tasks  so I don't have variable to use task.finished?
pool.wait seem to return after last task consumed, not after it finish.

If you want to wait for all tasks to finish you have to use pool.wait(:done), if you want to check if the pool is done use pool.done?.

If abort_on_exception is true and I run pool.wait(:done) it appears to hang forever if there's an exception. (I don't want open an issue if this is merely a misunderstanding of the interaction between abort_on_exception and pool.wait.)


ABC-178: polyfill URLSearchParams

Summary
IE11 and the current Edge don't support URLSearchParams, newly
introduced as part of the react-router upgrades. This completely
prevents Mattermost from working. The newest Edge in preview does
support URLSearchParams and renders without issue.
Ticket Link
https://mattermost.atlassian.net/browse/ABC-178
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] Ran make check-style to check for style errors (required for all pull requests)
[x] Ran make test to ensure unit and component tests passed

Added to NOTICE.txt https://github.com/mattermost/mattermost-webapp/pull/688/files#diff-d8b7f7a4958ed5af485670e07cf80cac


me again ^^"

hi ehr me again. so 1.0.4 doesn't work for me :/
tried to compile the master on windows 11 myself, there i had to adjust package.json -> index.js instead of ts or i would throw errors
so in theory build did work,
opening localhost:5522 after running install.bat or just run.bat actually never did the trick for me, even with firewall deactived. a dummy nodejs hello world app on port 1337 does open in browser.
so my biggest issue right now :  i got another black screen of death under load as the fan curve aborts at some point, again ^^"
could you maybe add a few other points and recompile it ^^"? like [[40, 15], [50, 30], [70, 50], [80, 80], [88, 90]],

okay deeply confused over here (also not really into nodejs)
wenn i use node14.18.2 to compile the stuff i get that error of the other day again
Error: The module '\?\C:\Users\brdei\Documents\alfc-master\dist\build\Release\service.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 93. This version of Node.js requires
NODE_MODULE_VERSION 83. Please try re-compiling or re-installing
the module (for instance, using npm rebuild or npm install).
at Object.Module._extensions..node (internal/modules/cjs/loader.js:1144:18)
at Module.load (internal/modules/cjs/loader.js:950:32)
at Function.Module._load (internal/modules/cjs/loader.js:790:12)
at Module.require (internal/modules/cjs/loader.js:974:19)
at require (internal/modules/cjs/helpers.js:93:18)
at Object.253 (C:\Users\brdei\Documents\alfc-master\dist\index.js:812:18)
at webpack_require (C:\Users\brdei\Documents\alfc-master\dist\index.js:24:31)
at getServiceWrap (C:\Users\brdei\Documents\alfc-master\dist\index.js:2443:17)
at Object.run (C:\Users\brdei\Documents\alfc-master\dist\index.js:2735:3)
at Object.932 (C:\Users\brdei\Documents\alfc-master\dist\index.js:4432:13) {
code: 'ERR_DLOPEN_FAILED'
when i use 16.13.1
i get "You must build a custom version of edge.node. Please refer to https://github.com/tjanczuk/edge"
also i figured in server/index.js, that it's only going up to version 14 in "versionMap"
:/

Please use the v1.0.4 release to run uninstall, then install again and paste its output here.


Don't set the storage account name for now

Can you provide some details on why you're removing this?  We'll need it eventually


"Constructor" should be RSMP Nordic

At the top for the SXL (or in the Version sheet in Excel) there is some meta data defined, such as Constructor, Reviewed, Approved and so forth.
Constructor should be changed to RSMP Nordic

We should think of how to map this into JSON Schema $schema, $id, $title and $description attributes.

With commit d33f9c4ecceb5c12d5c23a5297afccfd9c775041, the version information has been removed from the PDF and HTML versions of the SXL. The following fields were removed: Plant Name, Constructor, Reviewed, Approved, Created date, SXL revision, Revision date, RSMP version. They are still part of the Excel version of the SXL, but they don't really serve any purpose in this context. I'm closing this issue


chore: open PR [skip ci]

@all-contributors
please add @aiji42 for test.
please add @ismailajizou for translation and test.
please add @maherapp for translation and test.
please add @maherapp for translation and test.
please add @montedonioluiz for translation and test.
please add @izayoi-hibiki for translation and test.

I have added you all to the contributors list.
I am sure you will get numerous notifications, but don't worry about it.

@all-contributors
please add @maherapp for translation and test.
please add @montedonioluiz for translation and test.

@all-contributors please add @izayoi-hibiki for translation and test.

https://github.com/all-contributors please add @izayoi-hibiki for translation and test.

@all-contributors please add @izayoi-hibiki for translation and test.

@all-contributors please add @izayoi-hibiki for translation and test.

https://github.com/all-contributors please add @aiji42 for ideas.

@all-contributors please add @aiji42 for ideas.

@all-contributors please add @izayoi-hibiki for translation and test.


export data DAG running into errors

the export data DAG keeps failing at the def export_active_dags() step. the pause_dags() task completes and all the active DAGs are paused. and then the DAG fails with the error FileNotFoundError: [Errno 2] No such file or directory: 's3://mys3-bucket/mwaa-migration-test/active_dags.csv'.
As a result of this, when the DAG fails, all my active DAGs remain in a paused state and I have to unpause all of them. Am I missing something?

Thank you for opening the issue. Please make sure you have smart_open in your requirements.txt and using from smart_open import open in the imports. Let me know if you still run into issues

thanks @umaragu. do you know if i can update the requirements.txt file for an existing mwaa env or can I only do it before spinning up a new mwaa env? coz i am getting a ModuleNotFoundError: No module named 'smart_open' error when I try to update the requirements file and upload it to the same s3 location.

Please check the Step 5 of the documentation here. If you have a requirements.txt, You will need to add the two additional modules in this Github repo to it.

yes, i added those modules to my existing requirements.txt file and then ran into that issue i mentioned above.

This issue is resolved from a separate conversation. Closing the issue.


Future/FutureOr require returns?

The following triggers an analysis warning in 2.0.0-dev.32.0:
import 'dart:async';

void returnNothing() {
  return; // OK
}

Future<void> returnNothingInTheFuture() async {
  return; // ERROR: Expecting return.
}

FutureOr<void> returnNothingMaybeInTheFuture() async {
  return; // ERROR: Expecting return.
}

I can silence it with return null; ... but can't that be automatic?

Agreed, when return expressionOfTypeT; is allowed in a function Future<T> foo() async .., it is very natural to allow return; in a function Future<void> foo() async ...
We would need a special exception for this, of course, and there may be other cases. We have had some discussions in #32177, and I've added a reference to this issue there.

These are tricky cases, and I don't think we agree entirely on what to say about it.
My personal guiding rules are that:

return; and return voidExpression; must be valid in the same cases (if we allow the latter at all, which we have historically done) ...
... and then return; is really just equivalent to return null as void; (if we allow ... as void) ...
... and the typing rules for return voidExpression; says that it's allowed in an async function if Future<void> is assignable to the return type of the function.

The third entry is the general rule for return e;. A return e; is type-correct if Future<flatten(typeOf(e))> is assignable to the return type of the surrounding function. We know that the actual return value is a Future<flatten(returnType)>, so in the async cases here, the actual return object is a Future<void>, and the Future<flatten(typeOf(e))> thing is really a way of saying that the value of e can be used to complete the actually returned future.
So, I would want the second and third examples to be valid, because they would be for return voidFunction();.
We don't currently have a Dart 2 compatible specification for the return; case, so the law is still open.

The current situation is clearly just wrong:
import 'dart:async';

void foo1() {
  return;
}

Null foo2() {
  return;
}

Future<void> foo3() async {
  return; // warning • Missing return value after 'return' at test.dart:12:3 • return_without_value
}

Future<Null> foo4() async {
  return;
}

void main() { }

I think you can use like this:
import 'dart:async';

void foo1() {
  return;
}

Null foo2() {
  return;
}

Future<void> foo3() async {
  return Future<void>(() {});
}

Future<Null> foo4() async {
  return Future<Null>(() {});
}

With recent updates to the rules about returning (esp. https://github.com/dart-lang/language/pull/941), there are no errors in the following:
import 'dart:async';

void foo1() {
  return;
}

Null foo2() {
  return;
}

Future<void> foo3() async {
  return;
}

Future<Null> foo4() async {
  return;
}

am getting this error on slider OnChange() function
The return type 'Future' isn't a 'void', as required by the closure's context.dart return_of_invalid_type_from_closure
writting custom function that returns null this the error am getting
This expression has a type of 'void' so its value can't be used.
Flutter 2.5.0 stable


add edit functionality

allow the user to edit a db entry

how do we have the user indicate which entry should be updated? as designed so far, it would require the following:

User searches for an entry that should be updated
Results show ID# of entries found (new functionality)
User sends a command to update specific field using that unique ID (new functionality)

It seems clunky from a UX perspective. Though, how often will that need to happen?

with export functionality, there is a way to implement this with no code change:

export data
change data in entries desired (delete any not changed - but works fine if you leave it)
import file

Would be a good idea to add a flag to import to explicitly have user note if they want import to overwrite existing entries...


Status checks are still required (and waiting) when the 'paths-ignored' is in effect.

Describe the bug
When making a PR to edit/add docs, the status checks are still required, even though the workflow is skipped.
Steps to reproduce

Edit docs.
Create PR.
See that checks are pending even though the workflow skips them.

Expected behavior

Expect skipped workflows to pass/not be required as status checks.

https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks


[BUG] Blocking I/O across requests

What is the bug?
I am using opensearch-py client version 2.2.0 to do search on the a cluster. I started using the multi-threading to push more and more request to the cluster. But I see that after 4 threads we are not able to increase the throughput. I see that no throttling happening from Server, but client is not able to push the request.
How can one reproduce the bug?
Create an OpenSearch cluster, and use OpenSearch-py client with multi-threading to start doing then queries. Thread count we tested with 1,2,4,8,16,32,48,96 threads.
What is the expected behavior?
Expectation is OpenSearch client should be able to push through more and more request and if OpenSearch cluster is not able to take the request we should get a throttling exception.
What is your host/environment?
Linux.
Do you have any screenshots?
NA
Do you have any additional context?
I tried increasing the worker pool size from 10 to 20, but that didn't help.

Hey, what are your clusters index settings? I would expect this with default cluster settings as shards in an index are thread bound as well. A default index will have 3 shards and that's the max number of threads that can be used for ingestion.

Are you spawning threads and creating instances of the client? Can you please post a repro? Is your actual usage of the client sync or async?

@dblock
It's the same client which is getting used across the threads.
@dtaivpp , I checked the cluster and the CPU util was not going above 60% even at the maximum thread. That is where I think its the problem where client is not able to push the requests.
I also tried changing the pool_max_size from 10 to 20 even then there is no increase in the requests that can be sent to the cluster.

Are you using a sync client or an async client? My guess is that you're using a sync client.
I wrote a benchmark that compares synchronous, threaded and asynchronous I/O in opensearch-py. I was not able to push the synchronous nor the threaded version of the client past a certain limit, no-matter the amount of pool_maxsize or other settings I tried. The client does use a connection pool, but it's probably just too busy doing other work on a single thread blocked by the GIL, so spawning more threads doesn't help.
Sync vs. threaded inserts make no difference. For 5x50 inserts async outperforms sync 1.4x.
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃          Benchmark ┃ Min     ┃ Max     ┃ Mean    ┃ Min (+)         ┃ Max (+)         ┃ Mean (+)        ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ Sync vs. threaded. │ 1.387   │ 1.604   │ 1.450   │ 1.377 (1.0x)    │ 1.424 (1.1x)    │ 1.399 (1.0x)    │
│    Sync vs. async. │ 1.312   │ 1.479   │ 1.400   │ 0.960 (1.4x)    │ 1.047 (1.4x)    │ 0.994 (1.4x)    │
└────────────────────┴─────────┴─────────┴─────────┴─────────────────┴─────────────────┴─────────────────┘

5x250 items, 2.5x improvement
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ Sync vs. async. │ 4.426   │ 4.994   │ 4.694   │ 1.798 (2.5x)    │ 1.992 (2.5x)    │ 1.894 (2.5x)    │
└─────────────────┴─────────┴─────────┴─────────┴─────────────────┴─────────────────┴─────────────────┘
5x1000 items, 3.5x improvement
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ Sync vs. async. │ 16.629  │ 17.493  │ 16.999  │ 4.863 (3.4x)    │ 5.025 (3.5x)    │ 4.980 (3.4x)    │
└─────────────────┴─────────┴─────────┴─────────┴─────────────────┴─────────────────┴─────────────────┘
We may be able to improve the synchronous client performance on insert, but I am not sure how.

I have a (not very scientific) flame chart I recorded of a synchronous client doing some search requests with SigV4. Obviously these numbers are very dependent on the specific request etc but was done with large machines that hardware shouldn't be a bottleneck.

Here's the sync/async benchmark profile, it's actually stuck doing synchronous I/O.

Here's the sync/async benchmark profile, it's actually stuck doing synchronous I/O.

This is strange, why it is struck in synchronous I/O. My understanding was I/O should be done in async fashion, even when it is a sync client.

This is strange, why it is struck in synchronous I/O. My understanding was I/O should be done in async fashion, even when it is a sync client.

Unfortunately not, and it's by design. The default Urllib3HttpConnection uses urllib3 and the requests are synchronous. This is why aiohttp was introduced, but it requires your code to become async to take advantage of.

@dblock can we use some other implementation of HttpConnection. Generally we use RequestsHttpConnection , does that also have the same problem?

@dblock here is my thinking, as a user of the OpenSearch client I feel that OpenSearch sync client is easy to use. Most of the services have the same thing. But if we cannot use threading with simple sync clients that a bumber. Because async client is not easy.

@dblock can we use some other implementation of HttpConnection. Generally we use RequestsHttpConnection , does that also have the same problem?

Yes it does. That one does allow changing pool_maxsize, but that has little effect on throughput.

@dblock here is my thinking, as a user of the OpenSearch client I feel that OpenSearch sync client is easy to use. Most of the services have the same thing. But if we cannot use threading with simple sync clients that a bumber. Because async client is not easy.

What are other open-source client examples that talk HTTP that don't exhibit this kind of problem?

@dblock can we check milvus.

@dblock can we check milvus.

Seems to be using grpc, whole other ball game.

Let me check if there are any other clients I can find out which are not grpc. This seems to be a standard thing and a fix should be there.

Can you run multiple pythons with multiple clients? And there is also bulk, which helps max out the connection.

@wbeckler No, we don't want to run multiple clients, as that defeats the purpose of what we are trying to benchmark.
Also, the request which we are trying to send is for query and not indexing. Hence bulk is not an option.

Thanks @navneet1v. I do think the problem is clear, we want the client to not block on I/O across requests.

Thanks @navneet1v. I do think the problem is clear, we want the client to not block on I/O across requests.

Yes that is correct.

@dblock @wbeckler is anything required from my side to prioritize this issue?

Is it okay to use python multiprocessing instead of multithreading in your use case?
https://docs.python.org/3/library/multiprocessing.html

@wbeckler we can use that, but given that sync client is easy to use and is already integrated in various ann benchmarks, this is the reason why we want to keep things simple.

I'm suggesting that the benchmarking or high-volume query application rely on multiprocessing, and that way the python client would not be blocking on IO. So instead of multithread: {do python client stuff}, you would have multiprocess: {do python client stuff}
I am not a python expert, so I might be way off here, but that's how it seems it should work from my naive perspective.

I am going to take another look at this, assigning to myself.

Updated benchmarks in https://github.com/dblock/opensearch-py/tree/benchmarks/samples/benchmarks.

Benchmark
Min
Max
Mean
Min (+)
Max (+)
Mean (+)

1 client vs. more clients (async)
1.076
1.076
1.076
0.787 (1.4x)
0.787 (1.4x)
0.787 (1.4x)

1 thread vs. 2 threads (sync)
6.013
6.013
6.013
3.629 (1.7x)
3.629 (1.7x)
3.629 (1.7x)

1 thread vs. 8 threads (sync)
6.031
6.031
6.031
1.186 (5.1x)
1.186 (5.1x)
1.186 (5.1x)

1 thread vs. 32 threads (sync)
5.777
5.777
5.777
1.058 (5.5x)
1.058 (5.5x)
1.058 (5.5x)

We are getting an improvement by using threads (1.7x with 2, 5.1x with 8, with diminishing returns further, 5.5x with 32).
If we set the pool_maxsize to the number of threads this gets improved up to a certain amount.

Benchmark
Min
Max
Mean
Min (+)
Max (+)
Mean (+)

1 thread vs. 2 threads (sync)
7.355
7.355
7.355
2.835 (2.6x)
2.835 (2.6x)
2.835 (2.6x)

1 thread vs. 4 threads (sync)
6.616
6.616
6.616
1.777 (3.7x)
1.777 (3.7x)
1.777 (3.7x)

1 thread vs. 8 threads (sync)
6.340
6.340
6.340
1.272 (5.0x)
1.272 (5.0x)
1.272 (5.0x)

1 thread vs. 32 threads (sync)
6.608
6.608
6.608
1.076 (6.1x)
1.076 (6.1x)
1.076 (6.1x)

@dblock Thanks for doing the deep-dive. Most of the time as a user we go by seeing the examples. If changing the connection client is the only problem we should just fix the examples and our documentation.

In #535 I documented the connection pools and made sure pool_maxsize works for both implementations (it doesn't improve benchmarks by much).
I don't think we should deprecate RequestsHttpConnection, even if it's slower. If an application fully depends on that, you will not want to take a dependency on another HTTP library. Similarly, using async in this client is much faster than Urllib3HttpConnection, but we aren't depredating the synchronous implementation.

Make sense. My whole idea was we should provide our recommendation for best performance and also the challenges if the other Connection libs doesn't perform well, per our benchmarks.

I've spent a lot of time on the synchronous client, having written multiple benchmarks and digging through the code to understand how pooling works. I've varied number of threads, number of clients, search and ingestion, data sizes, switching between the default Urllib3HttpConnection, or the available RequestsHttpConnection connection classes.
The simplest benchmark is to do info() in a loop 100 times. The client improves throughput 5x and 3x with different connection classes on more threads and setting the max pool size to the number of threads.
Benchmark	Min	Max	Mean	Min (+)	Max (+)	Mean (+)
1 thread vs. 32 threads (sync), RequestsHttpConnection	30.104	30.104	30.104	5.697 (5.3x)	5.697 (5.3x)	5.697 (5.3x)
1 thread vs. 32 threads (sync), Urllib3HttpConnection	9.646	9.646	9.646	2.748 (3.5x)	2.748 (3.5x)	2.748 (3.5x)
This does show that RequestsHttpConnection is a lot slower. @navneet1v mentions we should remove it from being the default recommended in the documentation, which seems right.
Overall, these benchmarks are pretty conclusive that the client does not have the limitation as described and we need to dig deeper into how it's being used to see where the bottleneck is.

Its great to see the benchmarks for this client! @dblock Anyway we can make these benchmarks part of this repo or part of opensearch-benchmark?

Its great to see the benchmarks for this client! @dblock Anyway we can make these benchmarks part of this repo or part of opensearch-benchmark?

https://github.com/opensearch-project/opensearch-py/pull/537

I renamed the issue to "degraded performance in sync client".
@navneet1v I know you've been running some anecdotal benchmarks with k-nn. What are your most recent numbers between  RequestsHttpConnection, Urllib3HttpConnection, multiple clients and async?

@dblock Here is results.

So what I can see is for Multi Threading change connection class from  Request HTTP Connection to Urllib3 helped improve the latency if you do more work than just doing the search in same thread(aka parsing and doing some more calculation).
In case you just do search, then there is no impact in latency with and without Request HTTP Connection.
Multi processing code works better with Request HTTP Connection class as compared to Urllib3.

With Request HTTP Connection Client
Multi Threading
With Parsing
average latency:<PHONE_NUMBER>8659936
p50 Latency(sec):<PHONE_NUMBER>0003633
p90 Latency(sec):<PHONE_NUMBER>9999113
p99 Latency(sec):<PHONE_NUMBER>4000638
average took time latency: 12.9771
p50 took Latency(ms): 11.0
p90 took Latency(ms): 18.0
p99 took Latency(ms): 42.0

Without parsing logic

average latency: 55.61106889247895
p50 Latency(sec): 52.75726318359375
p90 Latency(sec): 76.60882472991945
p99 Latency(sec):<PHONE_NUMBER>4025274
average took time latency: 25721.5
p50 took Latency(ms): 23.0
p90 took Latency(ms): 42.0
p99 took Latency(ms): 73.0

Multi Processing
With Parsing
average latency: 41.32260456085205
p50 Latency(ms): 39.499521255493164
p90 Latency(ms): 54.305076599121094
p99 Latency(ms): 85.85322380065924
p50 took Latency(ms): 16.0
p90 took Latency(ms): 28.0
p99 took Latency(ms): 49.0

Without parsing logic
average latency: 52.904314446449284
p50 Latency(ms): 50.8497953414917
p90 Latency(ms): 71.18616104125977
p99 Latency(ms):<PHONE_NUMBER>5036012
p50 took Latency(ms): 22.0
p90 took Latency(ms): 39.0
p99 took Latency(ms): 63.0

With UrlLib3
Multi Threading
With Parsing
average latency: 79.73963953840034
p50 Latency(sec): 77.58072550001316
p90 Latency(sec):<PHONE_NUMBER>0003997
p99 Latency(sec):<PHONE_NUMBER>700498
average took time latency: 11.8679
p50 took Latency(ms): 11.0
p90 took Latency(ms): 14.0
p99 took Latency(ms): 29.0

Without parsing logic
average latency: 54.77132937908173
p50 Latency(sec): 51.97501182556152
p90 Latency(sec): 75.43172836303712
p99 Latency(sec):<PHONE_NUMBER>891602
average took time latency: 25799.7
p50 took Latency(ms): 23.0
p90 took Latency(ms): 41.0
p99 took Latency(ms): 74.0

Multi Processing
With Parsing
average latency: 42.12491910457611
p50 Latency(ms): 40.169358253479004
p90 Latency(ms): 56.438875198364265
p99 Latency(ms): 84.30349588394165
p50 took Latency(ms): 16.0
p90 took Latency(ms): 30.0
p99 took Latency(ms): 51.01000000000022

Without parsing logic
average latency: 53.499785208702086
p50 Latency(ms): 50.940632820129395
p90 Latency(ms): 72.27611541748048
p99 Latency(ms):<PHONE_NUMBER>0466309
p50 took Latency(ms): 23.0
p90 took Latency(ms): 40.0
p99 took Latency(ms): 66.01000000000022

Thanks, this is helpful as a comparison of multithreading vs. multiprocessing and thank you for the conclusions, I agree with them. Here's a more readable version.

requests

urllib3

threads
processes
threads
processes

with parsing

average
108.16
41.32
79.74
42.12

p50
104.17
39.50
77.58
40.17

p90
156.74
54.31
114.84
56.44

p99
216.00
85.85
156.30
84.30

average took
12.98

11.87

p50 took
11.00
16.00
11.00
16.00

p90 took
18.00
28.00
14.00
30.00

p99 took
42.00
49.00
29.00
51.01

without parsing

average latency
55.61
52.90
54.77
53.50

p50
52.76
50.85
51.98
50.94

p90
76.61
71.19
75.43
72.28

p99
114.45
100.53
111.95
106.08

average latency
25.72

25.80

p50 took
23.00
22.00
23.00
23.00

p90 took
42.00
39.00
41.00
40.00

p99 took
73.00
63.00
74.00
66.01

We have 3 kinds of requests possible:

GET without content (e.g. info())
POST with a large payload sent, a small payload returned, e.g. bulk()
POST with a small payload sent, a large payload returned, e.g. search()

I am not sure what "parsing" is in your case, which of the above cases is it?


Fail the GH build/test steps if the swift build/test actually failed.

Currently, if the swift build or swift test commands fail, the GH action step will still be marked as succeeded.
The current behaviour might be what you want, but on the assumption that it isn't, this change should cause a failed step to actually fail.

Update: swift test seems to return 0 even when there's a failure... which is weird.

I wonder if we can use the --xunit-output then run a report over the results?

Eg with something like

https://github.com/marketplace/actions/junit-report-action
https://github.com/marketplace/actions/test-reporter

Thank you so much!


Broken / German Lobbying Meetings Dashboard

German Lobbying Meetings Dashboard seems broken
http://keshif.me/demo/german_lobbying
It does not work in loading data. Log report: Cannot load data(Request timed out)

Thanks @nelsonmau.
It might be a temporary problem on loading the data from the Google Sheet. It's not a small data. On my computer, the data could be loaded now.
Can you try again and let me know if the issue is still there?

@nelsonmau the same for me

Got it!
The sheet 1B0vjide7zhHRjI2VVc_RF1-2rfoF_8AXl-7h7BvBkv8 seems in "not published" status, authorization is required.  @adilyalcin could you be so kind to check it on GDrive?
Tnx a lot!

The Google Sheet was private. Sorry, my mistake!

Yep, just worked after refresh! thank you very much! :)

Closing the issue now. Thanks for letting me know!


cant disable

Help the PySilon key is invalid now it tracks all of my activities(also it stole passwords, makes voice recordings 24/7 and tracks everything i type on the keyboard without me even wanting any of it

Can’t you just end the task and stop disable the startup function also remove it from registry

works. thank you
why didnt i think of that

also what python version do you have


Allow trader to propose collaborative settlement to the maker

This should be the default user interaction.

@klochowicz closing an issue will move it to "Done" but not the other way round :)


fix: Assign ClientOptions if provided

Currently, the function AttachPolicyClientOptions doesn't do anything because the condition is wrong. Only nil options are assigned, which is not correct.
This PR solves the problem

FYI @AsafMah

Wow, great fix.
I'll release a patch soon.

I'll release a patch soon.

Do you have an ETA? We need this fix in KEDa

Probably tommorow, You can refer directly to the master branch in the meantime though

Fixed in 0.13.0


Add ASF header and incubator disclaimer to Dockerfile

Description
As indicated in this doc, we need to add the ASF header and incubator disclaimer to Dockerfiles.

fixed by #2124


Implement the taq instantiate-account and taq fund tasks to instantiate and fund accounts

🌮🌮🌮 Taqueria PR 🌮🌮🌮
Fixes #1306
🪁 Description
Introduced the taq instantiate-account task to instantiate declared accounts and the taq fund task to fund instantiated accounts. Both tasks work under a specific environment.
🪂 Pre-Merge Checklist (Definition of Done)
🚦 Required to merge:

[x] ⛱ I have completed this PR template in full and updated the title appropriately
[x] ⛵ My code builds cleanly, and I have manually tested the changes
[ ] 🏄♂ Another team member has built this branch and done manual testing on the change
[ ] 🏖 New and existing unit tests pass locally and in CI
[ ] 🔱 The test plan has been implemented and verified by an SDET
[ ] 🦀 Automated tests have been written and added to this PR
[x] 🐬 I have commented my code, particularly in hard-to-understand areas
[ ] 🤿 Corresponding changes have been made to all documentation
[ ] 🐚 Required changes to the VScE have been made
[ ] 🪸 Required updates to scaffolds have been made
[ ] 🚢 The release checklist has been completed

🛩 Summary of Changes
Please include a summary of the changes to the codebase. Please also include relevant motivation and context for the change
(e.g. what was the existing behaviour, why did it break, etc.)
🎢 Test Plan
Please describe the testing strategy and plan for this PR
🛸 Type of Change

[ ] 🧽 Chore ➾
[ ] 🛠 Fix ➾
[x] ✨ Feature ➾
[x] 👷 Refactor ➾
[ ] 🧪 Pre-Release ➾
[ ] 🚀 Release ➾

Let's get the --account flag added to this pr


use modal in components-layout-demo-fixed-sider

Reproduction link
https://codesandbox.io/s/ng-zorro-antd-modal-yucot6?file=/src/app/app.module.ts
Steps to reproduce
click show modal button
What is expected?
When the modal appears, the left fixed sider behaves normally
What is actually happening?
The sidebar layout is abnormal. When the modal appears

Environment
Info

ng-zorro-antd
14.0.0

Browser
Chrome

I guess it is because the position: fixed attribute is used to achieve the fixed sidebar, but if this attribute is not used, the sidebar cannot be fixed.

It seems to be caused by this inline style. After removing it, everything is normal, but will this cause other problems?


Remove extraneous quote

Oops.

Hey! this Pr has a commit that shouldn't be there. do you mind creating a breach branch with the latest master, and adding the commit ff1668d only?

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
258

Relevant Lines:
258

💛  - Coveralls


Change Password should verify password every time

Hey, I was just looking through your code. I noticed that you have a verify_password function and a change_password function.
It would probably be more ideal from a security perspective if you called the verify_password function inside the change_password function. This will allow you to reuse your change_password function (when you start to implement multiple ways to access the system. I.e. Website, terminal, etc.) It'll prevent you from forgetting to add a verification into every implementation.
I noticed  when you create a new user, you use the change_password function as well here:
https://github.com/MTRNord/erooster/blob/b7c1da4c3fba816b3fc25e94a92d2b7826b283ac/src/cmds/eroosterctl.rs#L270
If you wanted to refactor it you could add the password field to your add_user SQL query/function. then add in a current_password parameter to your change_password function, and then call verify_password inside change_password.
After all it makes more sense, since you aren't 'changing' a password that hasn't been set.
I hope that makes sense. This is an interesting project and I'd be happy to contribute if you're accepting PR's.

Hi thanks for noting that :) This makes sense for sure.
I am certainly happy to accept PR's 👍
But I think while I am at it I probably can combine this with #9 at least in parts. Aka moving the current auth to some trait based abstraction which then calls db stuff in it :)


[Epic] Align Vavr 1.0 collections to the completely re-designed Scala 2.13 collections

See Scala 2.13 Collections Rework

[ ] Conversion

Bite the bullet, go for simulated higher kinded types and unite *.transform(...) with Iterable.to(...)? Because they are basically the same, except for the lack of HTKs, right?

@nfekete I know you don't like a generic to(...) conversion method. But explicitly enumerating all possible conversion methods has drawbacks and does not scale well. The solution described above does not simulate HKT's. It is just a compositional approach using functions. It aligns to the Scala 2.13 collection rework. Vavr's .ofAll factory method is called fromIterable in Scala. I only we could express on the type level that a type Class<C> has a specific static method...

I'm not for the enumerating all the conversion methods at all. I think a single transform(...) would be enough, if we could only express it in Java without simulated HKTs and other workarounds. @jbgi's work on derive4j/hkt seems very interesting. There is also KindedJ which seems to be adopted by Arrow for Kotlin and AOL's Cyclops
But I also recognize your aim to simplify things as much as possible and staying away of external dependencies.

Yeah, I know. I was involved when the KindedJ Github org was founded. (In fact I'm an admin of that org.)
However, that's not the right direction for Vavr. HKTs are needed by the projects you mentioned in order to lift functions to a level that operates on Monads.  Such algebraic abstractions are completely out of scope for Vavr. We strive for a dead-simple library for 'Avarage Java Joe' that hides away explicit abstractions like Functor, Monad, Monoid, Applicative, Lense, etc.

I'm a little curious though, about to versus transform. am I missing something, or every call to to could be replaced by a call to transform?
If that's the case, maybe cave could have only the latter, to reduce the API surface area?

@emmanueltouzery to is shorter and also present in Scala. transform will disappear. We align to Scala in the first place (and when in doubt). We could introduce the 0.9.2 methods for backcompat and deprecate them. But I would prefer to remove them.

@emmanueltouzery you (and @nfekete) are right! We can widen the codomain of to:
default <C> C to(Function<Iterable<T>, C> fromIterable) {
    return fromIterable.apply(this);
}

But they are not the same.
Vector:
<U> U transform(Function<? super Vector<T>, ? extends U> f)

Map:
<U> U transform(Function<? super Map<K, V>, ? extends U> f)

Iterable:
<U, C extends Iterable<U>> C to(Function<Iterable<T>, C> fromIterable)

There is value is the specificity of the transform method as opposed to the genericity of the to method. Transform takes a function that takes as input the concrete type on which transform is defined. Hence, you can efficiently transform a Vector by directly indexing one of the values in it, or directly taking out a value from a Map. You can't do that with the current version of to because it's too generic, it's takes a function that takes an Iterable of values. You can't take advantage of the concrete type when doing the transformation.
The two methods could be united, but only if you could somehow declare at the top level (at Iterable) that you're gonna take a function that takes as input the current self type. But you cannot express this without HKT's I think.
<U> U transform(Function<? super *SELF*, ? extends U> f)

Because we can't express the above, I think it's better to keep both. transform methods on the concrete implementation level, to on Iterable.

@nfekete That's right. A transformation is just a function. When removing transform we loose the fluent API.
In a first step I will take a look at the existing Scala collections API. It is pretty mature. Afterwards I think about what to do additionally...

Just a data point: I like the fluency of transform and would regret seeing it go. On the other hand, I can see why just removing the to* without adding a to() would leave people confused at how to port these calls.
In the end, if it was me, I think today in vavr, the to* calls and the transform calls serve different functions, and would keep both options.

@nfekete, @emmanueltouzery I also think both, to() and transform(), are helpful. We will implement both in Vavr 1.0.
Also I've thought much about modularity. The first module was a test-balloon. I care about you and existing code-bases. The more I think about modularization and the benefits of the current Vavr 0.9.x monolith, the more I think I need to keep the main functionality of Vavr in one module.
I will prepare a suggestion that preserves most of the existing functionality.

Type Hierarchy

We will have the following implementations:

HashSet and HashMap (probably based on CHAMP as in Scala 2.13, not on HAMT)
TreeSet and TreeMap
Queue, List and LazyList (LazyList has also a lazy head and will replace Stream)
Array, Vector

All other collections will go to an 'extended collections' module

Serializability

interfaces will not extend Serializable
collection classes will extends Serializable (and are probably final)

We need to take care of casting Comparators of sorted collections to (Comparator & Serializable).


Full throughput limit is not being used

Mountpoint for Amazon S3 version
mount-s3 1.5.0
AWS Region
us-east-1
Describe the running environment
Running on Amazon EC2 g5.12xlarge (24 cores)
operating system: Amazon linux2
Mountpoint options
mount-s3 <BUCKET> <DIR>

also tried the following options
1. maximum-throughput-gbps: 50/100
2. max-threads: 16/24/32/48
3. part-size: 8MB / 16MB / 32MB / 80MB

What happened?
The said machine g5.12xlarge has a bandwidth of 40Gbps. While reading a file of size 73.6Gb, the max throughput I observed is around 6Gbps.
PS: I didn't see any server side errors in syslogs
Relevant log output
No response

Hey @karthik1900, that throughput seems really low. Can you share more about how you read data from the mounted directory? Based on our benchmark results (https://awslabs.github.io/mountpoint-s3/dev/bench/), increasing number of concurrent reads and opening a file in O_DIRECT mode might help.

Hey @monthonk, thanks for the quick response
I am using a simple python open file
with open("myfile", "rb") as f:
    byte = f.read()

Will try with the O_DIRECT mode and update here

When I use fio to to read the file, I am getting about 22 Gb/s, I guess the issue is with my read
fio --name=read --rw=read --bs=4m --direct=1 --filename="myfile"

Thanks for the updates. For your workload, you can try spawning threads to read multiple files at the same time. That should get you close to the instance bandwidth. We will also update documentation on our side.

Hey @monthonk
Is there a way where we can set a configuration which will allow multiple threads to read one large file (as s3 allows reading in chunks)

Hi! We merged that particular documentation change, so we now have a section in troubleshooting slower throughput than expected: https://github.com/awslabs/mountpoint-s3/blob/main/doc/TROUBLESHOOTING.md#slower-throughput-than-expected
If there's any more questions here, please let us know - or we can close the issue. Thanks!


Getting rid of unmaintained dependency, core-js@2

Do you think it's possible to upgrade swagger-parser dependency to get rid of this warning?
warning api-schema-builder > swagger-parser > z-schema ><EMAIL_ADDRESS>core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
z-schema upgraded core-js to v3 (3.2.1) in 4.1.1 and swagger-parser uses this since 8.0.1.
z-schema dropped core-js since 4.2.0 and swagger-parser upgraded to it in 8.0.4.

@sergburn Try updating the version of the library that you are using. I believe this has already happened in https://github.com/PayU/api-schema-builder/pull/45

Of course! Thank you!


No GC

A new functional language with a GC is not a earth shattering. If you can do static analysis to figure out the lifetimes without having to specify them at all, this would be a breakthrough. Also see: https://github.com/carp-lang/Carp

Doing a safe functional language with currying and nicely composable functions is super challenging to do without a GC. There's only so much ground you can break in a single language!

It is not easy but perhaps if you track memory usage effects it might be possible.

What do you mean by 'track memory usage effects'?

See https://github.com/Microsoft/dafny and perhaps https://www.fstar-lang.org/

My Bad. It is Koka - http://rise4fun.com/Koka/tutorial/guide through the above languages also has effects.

Also http://rise4fun.com/SLAyer and other reasearch projects at http://rise4fun.com/ has some relavence. Also Nim has effect system which track memory related items: http://nim-lang.org/docs/manual.html#effect-system

Having no gc would be interesting and very fitting for an embeddable language as the gc pauses can be a big detractor for applications such as games which need predictable throughput. I do however hope that having one heap for each executing thread I  gluon can help out by letting the pause times be shorter
I will have to brush up on the linked languages but from what I remember they all come with both additional restrictions and they also are quite complex.

Also this might be of interest for static analysis: http://flix.github.io/
I feel static whole program analysis you might be able to get rid of GC. Starting point would be escape analysis with GCed heap and a non GCed heap where part of the variable's lifetimes are infereded. Iteratively improve it until you are able to infer all usages. Also when each new reference is created the number of references can be counted as part of static analysis. Based on this you can collace which and allocate memory in regions.

I definitely want escape analysis at some point (though that is still some ways of). Even a coarse analysis could go a long for native functions if they could mark whether their arguments escape or not (native_set_position { x = 1.0, y = 0.0 }, the allocation of the record is very likely not to be necessary since for such a small type it could be directly marshalled to a Vec2<f64> or similiar in Rust).
Static whole program analysis may be rather tricky as I hope to eventually have some sort of hot-swapping of code (ala Erlang) and gluon is currently quite close to dynamic languages when it comes to module loading for that reason. This it means that the concept of "the whole program" may be quite malleable which means additional complications when it comes to static analysis (not saying it is impossible though, just more complicated).

Having no gc would be interesting and very fitting for an embeddable language as the gc pauses can be a big detractor for applications such as games which need predictable throughput.

There’s recently been a good conversation about that regarding Mun: https://github.com/mun-lang/mun/issues/206


[WIP][HIVEMALL-108-2] Support -iter option in generic predictors

What changes were proposed in this pull request?
Support -iter option in both generic classifier and regression by buffering input samples, with an auxiliary -tol option.
(Ref. #87 #79 )
This PR is based on #87 .
What type of PR is it?
Improvement
What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-108
How was this patch tested?
unit tests

Coverage increased (+0.3%) to 40.585% when pulling 7bd60a3f1da934721f390f86873d08bc5393b199 on myui:HIVEMALL-108-2 into e9c66f0a1f3789cabd06e3b9fcaa56d52f05c2a0 on apache:master.

Coverage increased (+0.3%) to 40.585% when pulling 7bd60a3f1da934721f390f86873d08bc5393b199 on myui:HIVEMALL-108-2 into e9c66f0a1f3789cabd06e3b9fcaa56d52f05c2a0 on apache:master.

still work in progress. https://github.com/apache/incubator-hivemall/pull/98/commits/5b1af8fea4659ec853b9eceebde90899e734071a is completely based on https://github.com/apache/incubator-hivemall/pull/87

Need to fix a CI error.
Running hivemall.model.FeatureValueTest
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.122 sec <<< FAILURE!
testParseExpectingNumberFormatException(hivemall.model.FeatureValueTest)  Time elapsed: 0.017 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<java.lang.NumberFormatException> but was<java.lang.IllegalArgumentException>
    at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250)
    at java.lang.Double.parseDouble(Double.java:540)
    at hivemall.model.FeatureValue.parse(FeatureValue.java:112)
    at hivemall.model.FeatureValue.parse(FeatureValue.java:94)
    at hivemall.model.FeatureValueTest.testParseExpectingNumberFormatException(FeatureValueTest.java:59)

Coverage increased (+0.6%) to 40.905% when pulling 1997c39652a55a7fde7874ff137e2d311f3677f3 on myui:HIVEMALL-108-2 into e9c66f0a1f3789cabd06e3b9fcaa56d52f05c2a0 on apache:master.

Coverage increased (+0.6%) to 40.905% when pulling 1997c39652a55a7fde7874ff137e2d311f3677f3 on myui:HIVEMALL-108-2 into e9c66f0a1f3789cabd06e3b9fcaa56d52f05c2a0 on apache:master.

Need to fix output OI.
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.hadoop.io.Text
        at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.copyObject(WritableStringObjectInspector.java:36)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:380)
        at org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$TextKeyWrapper.copyKey(KeyWrapperFactory.java:221)
        at org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:786)
        at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:700)
        at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:768)

Also, validation is required for Feature input OI when -densemodel option is specified (only INT or BIGINT is expected).

Need to revise about this CI error.
test(hivemall.regression.GeneralRegressionUDTFTest)  Time elapsed: 0.1 sec  <<< ERROR!
org.apache.hadoop.hive.ql.exec.UDFArgumentException: Only INT or BIGINT is allowed for the element of feature vector when -densemodel option is specified: STRING
	at hivemall.GeneralLearnerBaseUDTF.getFeatureOutputOI(GeneralLearnerBaseUDTF.java:249)
	at hivemall.GeneralLearnerBaseUDTF.initialize(GeneralLearnerBaseUDTF.java:162)
	at hivemall.regression.GeneralRegressionUDTFTest.run(GeneralRegressionUDTFTest.java:254)
	at hivemall.regression.GeneralRegressionUDTFTest.test(GeneralRegressionUDTFTest.java:320)

Confirmed that the following system test passes successfully on EMR.
https://github.com/apache/incubator-hivemall/pull/79
set hivevar:n_samples=16281;
set hivevar:total_steps=32562;

drop table if exists logress_model;
create table logress_model as
select
 feature,
 avg(weight) as weight
from
 (
  select
     train_classifier(add_bias(features), label, '-loss logloss -opt SGD -reg no -eta simple -total_steps ${total_steps}') as (feature, weight)
     -- logress(add_bias(features), label, '-total_steps ${total_steps}') as (feature, weight)
  from
     train_x3
 ) t
group by feature;

WITH test_exploded as (
  select
    rowid,
    label,
    extract_feature(feature) as feature,
    extract_weight(feature) as value
  from
    test LATERAL VIEW explode(add_bias(features)) t AS feature
),
predict as (
  select
    t.rowid,
    sigmoid(sum(m.weight * t.value)) as prob,
    CAST((case when sigmoid(sum(m.weight * t.value)) >= 0.5 then 1.0 else 0.0 end) as FLOAT) as label
  from
    test_exploded t LEFT OUTER JOIN
    logress_model m ON (t.feature = m.feature)
  group by
    t.rowid
),
submit as (
  select
    t.label as actual,
    pd.label as predicted,
    pd.prob as probability
  from
    test t JOIN predict pd
      on (t.rowid = pd.rowid)
)
select count(1) / ${n_samples} from submit
where actual = predicted;

drop table if exists adagrad_model;
create table adagrad_model as
select
 feature,
 avg(weight) as weight
from
 (
  select
     -- train_adagrad_regr(features, label) as (feature, weight)
     train_regression(features, label, '-loss squaredloss -opt AdaGrad -reg no') as (feature, weight)
  from
     train_x3
 ) t
group by feature;

WITH test_exploded as (
  select
    rowid,
    label,
    extract_feature(feature) as feature,
    extract_weight(feature) as value
  from
    test LATERAL VIEW explode(add_bias(features)) t AS feature
),
predict as (
  select
    t.rowid,
    sigmoid(sum(m.weight * t.value)) as prob
  from
    test_exploded t LEFT OUTER JOIN
    adagrad_model m ON (t.feature = m.feature)
  group by
    t.rowid
),
submit as (
  select
    t.label as actual,
    pd.prob as probability
  from
    test t JOIN predict pd
      on (t.rowid = pd.rowid)
)
select rmse(probability, actual) from submit;

Coverage increased (+0.5%) to 40.72% when pulling 902ca08efda5310ffd14be8cbffc3dccb7e6f4b6 on myui:HIVEMALL-108-2 into e9c66f0a1f3789cabd06e3b9fcaa56d52f05c2a0 on apache:master.

Coverage increased (+0.5%) to 40.72% when pulling 902ca08efda5310ffd14be8cbffc3dccb7e6f4b6 on myui:HIVEMALL-108-2 into e9c66f0a1f3789cabd06e3b9fcaa56d52f05c2a0 on apache:master.

I double-checked the code with -iter 1 option :)


refactor: remove _openapi.schema_generation.utils.sort_schemas_and_refrences()

Pull Request Checklist

[ ] New code has 100% test coverage
[ ] (If applicable) The prose documentation has been updated to reflect the changes introduced by this PR
[ ] (If applicable) The reference documentation has been updated to reflect the changes introduced by this PR
[ ] Pre-Commit Checks were ran and passed
[ ] Tests were ran and passed

Description

This method has been at the root of a couple of issues recently:

#2460
#2563

We pass the Schema instances that would go into a oneOf schema property into this function, which attempts to sort the contents by the Schema.type property. However, not all Schema instances define that property, and we get errors due to trying to do a value comparison against None. E.g., where the values of a schema can be other oneOf schemas.
Unfortunately, git blame for the commit that adds sort_schemas_and_references() points to a 61 file commit that adds attrs support to Partial, so hard to draw conclusions as to why it was added from that.
The reason I've opted to remove, is that it is unclear how memebers of the schema should be sorted when they don't have a type, and it is not clear to me whether they even should be sorted, e.g., if a handler returns a tuple[str, int], should those types be sorted alphabetically.
So, this PR removes use of the utility, and modifies the tests that were relying on the schema ordering it provided.
Close Issue(s)

Closes #2563

I'm trying to get to the bottom of this flaky test.
With Union[str, int] passed as an argument to conlist() every now and then we are seeing the return of that function give us Annotated[List[Union[int, str]]]. To identify this, I added an inspection of the test annotation so that it would print when the test failed:
    for pydantic_fn in [conset, conlist]:
        if pydantic_version == "1":
            annotation = pydantic_fn(Union[str, int], min_items=1, max_items=10)
        else:
            annotation = pydantic_fn(Union[str, int], min_length=1, max_length=10)
        print(f"{annotation=}")

When the test fails, it shows the flipping of the order of the annotations:
annotation=typing.Annotated[typing.Set[typing.Union[str, int]], Len(min_length=1, max_length=10)]
annotation=typing.Annotated[typing.List[typing.Union[int, str]], Len(min_length=1, max_length=10)]

Which shows that the union args are somehow being flipped around between creation of the union and return from the pydantic con fn.
Pydantic isn't doing anything crazy with the annotation inside their conlist() func (ref:
def conlist(
    item_type: type[AnyItemType],
    *,
    min_length: int | None = None,
    max_length: int | None = None,
    unique_items: bool | None = None,
) -> type[list[AnyItemType]]:
    ...  # no use of `item_type` in any of redacted func
    return Annotated[List[item_type], annotated_types.Len(min_length or 0, max_length)]

Which is making me wonder if there is something happening inside typing, maybe to do with caching or flattening that causes this every now and then. Unfortunately I've not been able to reproduce it consistently, other than by running the full test suite.
I tried defining the union annotation for the test outside of the loop:
    union_annotation = Union[str, int]
    for pydantic_fn in [conset, conlist]:
        if pydantic_version == "1":
            annotation = pydantic_fn(union_annotation, min_items=1, max_items=10)
        else:
            annotation = pydantic_fn(union_annotation, min_length=1, max_length=10)
        print(f"{annotation=}")

but it still occurred.
I suppose it doesn't really matter what order the annotations of a union type are, practically speaking - so I think I will just modify this test to check that the members of the union in the schema are present, and not rely on their ordering.


[BUG] registry value for autostart not cleaned for windows

During the crc cleanup command it is expected any configuration on the system will be cleaned.
At the moment to control the autostart of the tray a registry value( KEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Run\redhat.codereadycontainers.tray) is added to the registry.
Then during the crc cleanup that value it is not deleted.

This registry value is included as part of install so should it be part of uninstall script instead cleanup side? As of now clean up try to remove configurations which are part of setup. Also moving forward we are trying to go away with tray so this shouldn't be crc responsibility to check the registry changes which made by crc.

This registry value is included as part of install so should it be part of uninstall script instead cleanup side?

yes, this should be done from the msi during uninstall

btw, since this way of  "autostarting" tray is configured from electron, in windows it creates a registry entry to do it, and in macOS it adds the app to Login Items, but there is no easy way to remove Login Items in macOS (as far as i've searched) you could use an applescript to automate the GUI interactions to remove it.


Feature Request: Custom FQDN for cluster

It would be interesting to have docs on how to deploy with a custom domain.

This would also benefit issues, such as rate-limiting for commonly used azure cloudapps domains for let's encrypt. See #59 for example.

In general, it doesn't take much to utilize a custom domain.

Have a custom domain with a CNAME towards the built-in domain of the loadbalancer IP (xxxx.{region}.cloudapp.azure.com)
During deployment of the chart, provide the custom domain for the FQDN. (which is the parameter hostName I believe)
Done?

How to describe a Setup for multiple DNS providers hundreds of dns providers have quoite a lot of ways to setup DNS entrys. This makes it quite difficult for starters, either they need the same DNS provider as in an example or they already need to know how to do the setup.

@1fabi0 I don't think we need to explain how to setup the CNAME record. But the steps 1-3 I described could be part of the docs. It just needs to be verified, that these steps are all that is necessary to get a custom domain to work. Maybe I forgot something.

@osamabinsaleem if you go this route, please let is know if it works and if there were any steps missing. Then we would include these in the docs in the future :)


set up timer page

ability to set a timer after clicking apply redirects you to the countdown page

@ChandlerCabrera any idea on how you're going to encode the URL to specify what machines time you are updating? And where are we storing the time info?

@JesseyH I tried exploring the different methods of sending data between pages. I don't think we should use local storage or GET, so I've been trying to use POST but having trouble implementing it


Error functions

I think we should consolidate all of the error functions (The ones that only return an Error type with some arguments) into a separate module, like error_funcs.rs
Thoughts?

Agreed. I've been thinking about it, but it doesn't seem like a big deal
either way. If we're both for it, sounds good.
On Tue, Jun 7, 2016 at 5:31 PM Ryan Fredlund<EMAIL_ADDRESS>wrote:

I think we should consolidate all of the error functions (The ones that
only return an Error type with some arguments) into a separate module, like
error_funcs.rs
Thoughts?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ProtonLights/cli/issues/20, or mute the thread
https://github.com/notifications/unsubscribe/AFT_gmhT7R7PCTbe6m1Jb5NUob5CMISkks5qJfEkgaJpZM4IwbQM
.

Closed with #23.


Fix: Improve compatibility with Xcode 7.3 and fix a NSRange bug

It seems that Xcode 7.3 change the displayType from enumName to
enum enumName, this commit extract the enumName to make the plug-in
to work again.
This commit also try to fix a potential bug related to NSRange that make
Xcode raise a NSRangeException and crash.
Note: In Xcode 7.3 only the enums defined by NS_ENUM macro are
recognized as Xcode.SourceCodeSymbolKind.Enum, so the plug-in
can only work with the NS_ENUMs consequently.

Well Done! Thanks  😃


Access raw single backing slice?

In circumstances where a user knows they are not going to pop_back/front passed a particular item, but may otherwise be doing various pushes and pops, it would be nice to be able to hold on to an index/iterator that would keep working as long as the specific item being referred to hasn't been removed. The most straightforward way to do this is to give the user the index of an item in the array and an interface for accessing it (where the index is used directly on the raw array backing the container, without taking into account the current location of head or tail).
You can kind of get this effect now by calling as_slices or as_mut_slices right when the container is created and only using the first slice, but this relies on knowledge of implementation details for the initial container (e.g. since the buffer is circular in principle there is no reason that tail couldn't start at N/2). Is there a better way?

It's possible to have an external indexer to remember the raw index on the slice and it can be used to access the ArrayDeque while hiding the details. Dfs of petgragh pretty much does the same thing.
The interface may be:
let mut queue = ArrayDeque::new();
let slot: Slot = queue.slot_of_nth(5);
let item: Option<_> = queue[slot];


Add an abstract Path class for both static asset and SD card pure pursuit

Summary
Add an abstract Path class for both static asset and SD card pure pursuit.
There is now a Path class that can be directly passed to chassis.follow.
The class has two implementations: AssetPath and FilePath. This allows pure pursuit to read from both static asset and the SD card.
Motivation
It was discussed in the server, as being able to have the flexibility of either SD card or static asset is useful.
Test Plan
This still needs to be tested, I don't have access to a chassis to test this with.

Download the template for this pull request:

[!NOTE]
This is auto generated from Add Template to Pull Request

via manual download<EMAIL_ADDRESS>via PROS Integrated Terminal:

curl -o<EMAIL_ADDRESS>https://nightly.link/LemLib/LemLib/actions/artifacts/1070001070.zip;
pros c fetch<EMAIL_ADDRESS>pros c apply<EMAIL_ADDRESS>rm<EMAIL_ADDRESS>

I'm not sure about the method by which this PR implements the functionality. While I do agree that the feature that this PR implements is required, I think that simply passing a vector of waypoints to pure pursuit will be better, and having functions which return a vector of waypoints given the asset name or file path.

I'll get it done tonight

is this still active?


iOS UI Settings Refresh

Early re-work notes:

A lot of UIViewControllerRepresentable shims over existing UIKit view controllers. These misbehave a little. They should be rewritten in SwiftUI.
Settings no longer uses the Vibrant Table View Cell
Add account button moved to the header instead of being a separate cell
Changes to AppDefaults now trigger objectWillSend notifications.

This is ready for testing.

What's changed:

Settings is now SwiftUI (as a result, NNW's vibrant cells are not used)
List rows are in SettingsViewRows
Iconography
AppDefaults is now an ObservableObject
Help/Info section of Settings has far less in it.
Many of the existing UIKit controllers have SwiftUI Representable wrappers, so they can't be retired just yet until they are re-written.

Why are all the vars in SettingsViewRows capitalized?
The Manage Accounts view should have a title of Manage Accounts instead of Accounts. We should also be able to delete Accounts by swiping left on the table cells.
There is no obvious way to exit the Add Account view. After successfully adding an Account we should return to the Manage Accounts view without having to manually exit the Add Account view.
Manage Accounts and Manage Extensions are inconsistent with how we add items. Manage Extensions should also have the plus button and present a sheet to select which extension to add like it is in Manage Accounts. We should also be able to deactivate an Extension by swiping left.
Import/Export Subscriptions should be split into two separate table cells and we should use an Action Sheet to select which Account to use for each option. From the HIG, Use an action sheet — not a menu — to provide choices related to an action.

@vincode-io All addressed in 07d9c3d and ad66d70

@stuartbreckenridge Did you delete all the now obsolete and unused classes?

Not yet. Figured it should be tested then we can get rid.

@stuartbreckenridge Why did you close this pull request? Have you abandoned work on it?

The work is in #3778


Run scraper error.

ENVIROMENT: production
COMPANY: Aptiv
URL: https://scrapers.peviitor.ro/src/aptiv/index.html
Browser: Mozilla firefox
Device: laptop
OS: Windows 10
Precondition:
The site must be opened in Mozilla browser.
Steps:
Click on "Run scraper".

Expected result:
The page is updated with active jobs.
Actual result:
Data.country is undefined message error.

There are no jobs available


[BUG] Analytical engine launching failed

Describe the bug
graphscope cluster starts failed
To Reproduce
Steps to reproduce the behavior:
it is failing to run below command:
sess = graphscope.session(num_workers=2)
Expected behavior
graphscope runs successfully
Screenshots

error logs
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.ABORTED
        details = "$Failed to connect to engine in 60s, deployment may failed. Please check coordinator log for details. The traceback is: Traceback (most recent call last):
  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/coordinator.py", line 413, in CreateAnalyticalInstance
    engine_config = self._operation_executor.get_analytical_engine_config()
  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 369, in get_analytical_engine_config
    response_head, _ = self.run_on_analytical_engine(dag_def, [], {})
  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/monitor.py", line 193, in runOnAnalyticalEngineWarp
    res = func(instance, dag_def, dag_bodies, loader_op_bodies)
  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 169, in run_on_analytical_engine
    response_head, response_bodies = self.run_step(dag_def, dag_bodies)
  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 102, in run_step
    responses = self.analytical_grpc_stub.RunStep(requests)
  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 364, in analytical_grpc_stub
    self._analytical_grpc_stub = self._create_analytical_grpc_stub()
  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 355, in _create_analytical_grpc_stub
    raise RuntimeError(
RuntimeError: Failed to connect to engine in 60s, deployment may failed. Please check coordinator log for details
"
        debug_error_string = "{"created":"@1680008010.548298283","description":"Error received from peer
ipv4:<IP_ADDRESS>:31525","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"$Failed to connect to engine in 60s, deployment may failed. Please check coordinator log for
details. The traceback is: Traceback (most recent call last):\n  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/coordinator.py", line 413, in
CreateAnalyticalInstance\n    engine_config = self._operation_executor.get_analytical_engine_config()\n  File
"/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 369, in get_analytical_engine_config\n    response_head, _ =
self.run_on_analytical_engine(dag_def, [], {})\n  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/monitor.py", line 193, in runOnAnalyticalEngineWarp\n
res = func(instance, dag_def, dag_bodies, loader_op_bodies)\n  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 169, in
run_on_analytical_engine\n    response_head, response_bodies = self.run_step(dag_def, dag_bodies)\n  File
"/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 102, in run_step\n    responses = self.analytical_grpc_stub.RunStep(requests)\n  File
"/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 364, in analytical_grpc_stub\n    self._analytical_grpc_stub =
self._create_analytical_grpc_stub()\n  File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 355, in _create_analytical_grpc_stub\n    raise
RuntimeError(\nRuntimeError: Failed to connect to engine in 60s, deployment may failed. Please check coordinator log for details\n","grpc_status":10}"

error log from coordinator
2023-03-28 13:05:42,942 [INFO][kubernetes_launcher:591]: Starting GAE rpc service on <IP_ADDRESS>:56426 ...
2023-03-28 13:05:44,016 [INFO][kubernetes_launcher:619]: Analytical engine launching command: /opt/openmpi/bin/mpirun --allow-run-as-root --bind-to none -n 2 -host gs-engine-fobolb-0:1.0,gs-engine-fobolb-1:1.0 /opt/graphscope/bin/grape_engine --host <IP_ADDRESS> --port 56426 -v 1 --vineyard_socket /tmp/vineyard_workspace/vineyard.sock
2023-03-28 13:05:44,039 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 0 time
2023-03-28 13:05:44,039 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:05:47,107 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 1 time
2023-03-28 13:05:47,107 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:05:50,111 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 2 time
2023-03-28 13:05:50,112 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:05:53,116 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 3 time
2023-03-28 13:05:53,116 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:05:56,120 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 4 time
2023-03-28 13:05:56,120 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:05:59,124 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 5 time
2023-03-28 13:05:59,124 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:02,128 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 6 time
2023-03-28 13:06:02,128 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:05,208 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 7 time
2023-03-28 13:06:05,208 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:08,212 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 8 time
2023-03-28 13:06:08,212 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:11,217 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 9 time
2023-03-28 13:06:11,217 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:14,222 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 10 time
2023-03-28 13:06:14,222 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:17,226 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 11 time
2023-03-28 13:06:17,226 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:20,231 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 12 time
2023-03-28 13:06:20,231 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:23,307 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 13 time
2023-03-28 13:06:23,307 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:26,312 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 14 time
2023-03-28 13:06:26,312 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:29,317 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 15 time
2023-03-28 13:06:29,318 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:32,322 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 16 time
2023-03-28 13:06:32,323 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:35,327 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 17 time
2023-03-28 13:06:35,328 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:38,332 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 18 time
2023-03-28 13:06:38,333 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:41,408 [WARNING][op_executor:349]: Connecting to analytical engine... retrying 19 time
2023-03-28 13:06:41,409 [WARNING][op_executor:352]: Error code: StatusCode.UNAVAILABLE, details failed to connect to all addresses
2023-03-28 13:06:44,416 [INFO][coordinator:544]: Clean up resources, cleanup_instance: True, is_dangling: False
2023-03-28 13:06:44,416 [INFO][kubernetes_launcher:714]: Cleaning up kubernetes resources

run below command from pods getting error:
 kubectl exec  -n gs-vmgsyz  coordinator-lwywpd-6dc56d567d-tm85f -ti --  /opt/openmpi/bin/mpirun --allow-run-as-root --bind-to none -n 2 -host gs-engine-lwywpd-0:1.0,gs-engine-lwywpd-1:1.0 /opt/graphscope/bin/grape_engine --host <IP_ADDRESS> --port 56774 -v 1 --vineyard_socket /tmp/vineyard_workspace/vineyard.sock

Environment (please complete the following information):

GraphScope version:  0.20.0
OS: openeuler
Version [20.03-LTS-SP3]
Kubernetes Version [1.25.5]

Additional context
build k8s cluster by kubeadm, following below steps
https://www.cnblogs.com/fenghua001/p/16849875.html

run below command from pods getting error:

It's because we set some environment variables when running the mpi. in here and here
Seems the mpi is not running, but I couldn't see any errors.
It's also could be that the coordinator cannot connect to the engine service.
Could you set the env export OMPI_MCA_plm_rsh_agent=/usr/local/bin/kube_ssh and rerun that mpirun command in coordinator pod?

Running on openeuler is a new scenario for us, I'm interested in seeing if it works!

I tried to run below commands:
 kubectl exec  -n  gs-ippmcw coordinator-hwyspf-5dd778df76-ql2vw -ti -- /bin/bash -c "export OMPI_MCA_plm_rsh_agent=/usr/local/bin/kube_ssh"

and then
kubectl exec  -n gs-ippmcw coordinator-hwyspf-5dd778df76-ql2vw  -ti -- /opt/openmpi/bin/mpirun --allow-run-as-root --bind-to none -n 2 -host gs-engine-hwyspf-0:1.0,gs-engine-hwyspf-1:1.0 /opt/graphscope/bin/grape_engine --host <IP_ADDRESS> --port 56496 -v 1 --vineyard_socket /tmp/vineyard_workspace/vineyard.sock

After that I tried below command:
kubectl exec  -n gs-ippmcw coordinator-hwyspf-5dd778df76-ql2vw  -ti -- /bin/bash -c "export OMPI_MCA_plm_rsh_agent=/usr/local/bin/kube_ssh" /opt/openmpi/bin/mpirun --allow-run-as-root --bind-to none -n 2 -host gs-engine-hwyspf-0:1.0,gs-engine-hwyspf-1:1.0 /opt/graphscope/bin/grape_engine --host <IP_ADDRESS> --port 56496 -v 1 --vineyard_socket /tmp/vineyard_workspace/vineyard.sock

it returns nothing:

How should I continue ?

More information for you reference, I tried to debug to the pod , and then export below path
 export OMPI_MCA_plm_rsh_agent=/usr/local/bin/kube_ssh
then I run below command in the container:
 /opt/openmpi/bin/mpirun --allow-run-as-root --bind-to none -n 2 -host gs-engine-eqxyma-0:1.0,gs-engine-eqxyma-1:1.0 /opt/graphscope/bin/grape_engine --host <IP_ADDRESS> --port 56443 -v 1 --vineyard_socket /tmp/vineyard_workspace/vineyard.sock
its running long time and seems like stuck their but no return until the k8s resource clean up by python client

I suspect that mpi cannot connect to workers.
Replace the /opt/graphscope/bin/grape_engine -xxx ... vineyard.sock to a simple command like hostname
Try this.
mpirun <xxx> hostname

And please paste your ifconfig output. Maybe we need to pick another network

See if bond0 or eth0 is in the config.
And try add configurations and run again.
mpirun -mca btl_tcp_if_include eth0 or -mca btl_tcp_if_include bond0

Also you should check the firewall settings and if it is enabled (like it was for me), then try disabling it

Also you should check the firewall settings and if it is enabled (like it was for me), then try disabling it

From Ifconfig we don't have eth0 or bond0, how can we change the command to run mpi, I mean from python client

BTW, I would like to know how does gs trigger mpirun?
It is running  by k8s service ?

I suspect that mpi cannot connect to workers. Replace the /opt/graphscope/bin/grape_engine -xxx ... vineyard.sock to a simple command like hostname Try this.
mpirun <xxx> hostname

-- use hostname
[envbuilder@openeuler40 ~]$ kubectl exec  -n gs-bmjyaf coordinator-fegixv-75bf9ccd4c-q449z -ti -- /bin/bash
[graphscope@coordinator-fegixv-75bf9ccd4c-q449z ~]$ export OMPI_MCA_plm_rsh_agent=/usr/local/bin/kube_ssh
[graphscope@coordinator-fegixv-75bf9ccd4c-q449z ~]$ /opt/openmpi/bin/mpirun --allow-run-as-root --bind-to none -n 2 -host gs-engine-fegixv-0:1.0,gs-engine-fegixv-1:1.0 hostname
command terminated with exit code 137
-- set mca btl_tcp_if_include eth0
[envbuilder@openeuler40 ~]$ kubectl exec  -n gs-jscjip coordinator-flfzwp-68464559d5-pmb69 -ti -- /bin/bash
[graphscope@coordinator-flfzwp-68464559d5-pmb69 ~]$ export OMPI_MCA_plm_rsh_agent=/usr/local/bin/kube_ssh
[graphscope@coordinator-flfzwp-68464559d5-pmb69 ~]$ /opt/openmpi/bin/mpirun --allow-run-as-root --bind-to none -n 2 -host gs-engine-flfzwp-0:1.0,gs-engine-flfzwp-1:1.0 -mca btl_tcp_if_include eth0
--------------------------------------------------------------------------
No executable was specified on the mpirun command line.

Aborting.
--------------------------------------------------------------------------
-- set  mca btl_tcp_if_include and execute grape_engine, it is running until coordinator reach max retry times and clean up resource
[graphscope@coordinator-flfzwp-68464559d5-pmb69 ~]$ /opt/openmpi/bin/mpirun --allow-run-as-root --bind-to none -n 2 -host gs-engine-flfzwp-0:1.0,gs-engine-flfzwp-1:1.0 -mca btl_tcp_if_include eth0 /opt/graphscope/bin/grape_engine --host <IP_ADDRESS> --port 56690 -v 1 --vineyard_socket /tmp/vineyard_workspace/vineyard.sock
command terminated with exit code 137

From Ifconfig we don't have eth0 or bond0, but we have multiple NICs

Could you share your ifconfig?

how can we change the command to run mpi（I mean from python client）

MPI command is not exposed to client. If we confirmed that it's could be resolve by configuring the command, I would submit a patch to enable it to be configured. Right now it's in here
I suspect it's a compatible issue with openeuler. If you could share how you setup your cluster, I would like to get a ECS with openeuler and reproduce it. Is running minikube on openeuler is sufficient?

Sorry, we couldn't share the ifconfig as per company's limitations, we configured two network cards, one is for external network, another is for local network

I'm afraid that you have to tell mpi to use which ip interface.

From Ifconfig we don't have eth0 or bond0, but we have multiple NICs ， how can we change the command to run mpi（I mean from python client）

Could you please find what ip interface on your machine, and include that interface in the -mca btl_tcp_if_include eth0 (replace eth0 to your ip interface accordingly)。 If there are multiple of them. We have to make sure the raw command succeed (the mpirun xxx by hand).

I couldn't got some ECS with openeuler by some internal restrictions. So I'm afraid I couldn't reproduce your cluster setup 😢

And this may help: https://www.open-mpi.org/faq/?category=tcp
I'm not an expert of MPI and not able to see the actual causes if I couldn't have the real scenarios and try myself 😂

Sorry, we couldn't share the ifconfig as per company's limitations, we configured two network cards, one is for external network, another is for local network

I'm afraid that you have to tell mpi to use which ip interface.

From Ifconfig we don't have eth0 or bond0, but we have multiple NICs ， how can we change the command to run mpi（I mean from python client）

Could you please find what ip interface on your machine, and include that interface in the -mca btl_tcp_if_include eth0 (replace eth0 to your ip interface accordingly)。 If there are multiple of them. We have to make sure the raw command succeed (the mpirun xxx by hand).
I couldn't get some ECS with openeuler by some internal restrictions. So I'm afraid I couldn't reproduce your cluster setup 😢

Thanks man, will try

Feel free to reopen this if you have further problems or questions


I want to get the values of other cells with formulas after updateCell(), What should I do?

I want to get the values of other cells with formulas after updateCell(), What should I do?

This library only reads data from static file.
If you update a cell that is used within formulas, you have to

Open the file with LibreOffice or Excel to compute the formulas
Read again the file

There is no roadmap about integrating business logic of LibreOffice/Excel in this library.