Conversation - Reaction tooltip does not update when other user has removed the emoji

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Go to staging.new.expensify.com
Go to any chat as user A.
Add the same reaction to the message that has been reacted by User B.
Long press on the reacted emojj.
As User B, remove the reacted emoji.

Expected Result:
The tooltip will update the reaction list.
Actual Result:
The tooltip does not update the reaction list.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android / native
[x] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: v1.3.46-1
Reproducible in staging?: y
Reproducible in production?: n/a
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/93399543/5e2d6497-67f1-4a57-be5c-cf2e8cc0b8ca
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
View all open jobs on GitHub

:wave: Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open StagingDeployCash deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

Identify the pull request that introduced this issue and revert it.
Find someone who can quickly fix the issue.
Fix the issue yourself.

Adding Deploy Blocker label, because  it shows undefined on emoji reaction tooltip

Does not looks like a Deploy blocker to me. @puneetlath

Regression from: #23371

@hungvu193 Can you please explain how this is a regression from the linked PR?

I don't think this needs to be a deploy blocker given that on production it shows 0:undefined. So we're not regressing at least. So I'm going to remove the deploy blocker label.

@puneetlath can you please check that i have already reported that bug and @StevenKKC mentioned it will be fixed in this PR but it is still reproducible
https://github.com/Expensify/App/pull/23371

Hey @puneetlath , I think I originally have reported this issue here

I reported root case of this bug 7 days ago https://expensify.slack.com/archives/C049HHMV9SM/p1689857009202199?thread_ts=1689857009.202199&cid=C049HHMV9SM

I agree that this is a duplicate of https://github.com/Expensify/App/issues/23308 that @ayazhussain79 reported 8 days ago. Going to close this one in favor of that one 👍


Please provide an exit mechanism

On windows the only way to exit is by using process explorer to kill the process - even control-C does not exit. Tested on Windows 10 Python 2.7.12

Super sorry about this!  I'm hoping to do some issue triage / patching this weekend -- will let you know once this is remedied.


Enable custom buttons for CloudTenant

Adds Cloud Tenant option in Automate > Customization > Buttons, and activates
those custom buttons in the cloud tenant controller.

@tzumainn I am assuming that this should be labeled darga/no.  Please confirm.

@chessbyte Ah, sorry, that's correct!

@miq-bot add_label darga/no


Update Meeting Time

Before merging - do we want to pin this to the daylight savings cycle of Europe (as most maintainers are based there)?
Not urgent to decide right now but winter will come ;)

I think cest would be a good idea, so we don't have to change our schedule when winter comes. I still have to remind myself that the current meeting is 14:30 cet 😆

I've updated the wording on this to clarify the timezone. Let me know if this makes sense :)


[stable/k8s-resources]: support ClusterRoles and ClusterRoleBindings

Description
Adding ClusterRole and ClusterRoleBinding resources support.
Checklist

[x] Title of the PR starts with chart name (e.g. [stable/mychartname])
[x] I have read the contribution instructions, bumped chart version and regenerated the docs
[ ] Github actions are passing

Hello
Although tests are passing locally - to run on GH - the approval is needed, so I haven't ticked a checkbox in PR decsription yet

@max-rocket-internet shouldn't this chart be deprecated in favour of k8s-resources ?

shouldn't this chart be deprecated in favour of k8s-resources ?

In theory I guess but I think they can coexist 🙂


add url replacement for openshift

Description
This PR includes the following proposed change(s):

Add url parsing for the reports endpoints

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Refactoring / Documentation

if your change is a breaking change, please add breaking change label to this PR
How Has This Been Tested?
Please describe the tests that you ran to verify your changes.
Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[ ] New and existing unit tests pass locally with my changes

Stale an partially fix in release


PostCSS 7 conflicts

Hello,
are you sure that "postcss": "^7.0.2" is absolutely needed? It causes other PostCSS plugins to fail that still need PostCSS 6 - therefore I had to downgrade your plugin.
Best regards
Henry

PostCSS 7 plugins will work with PostCSS 6 runtime and other PostCSS 6 other plugins.
PostCSS 7 had no AST changes.

Well, I got this error message - maybe this is not even related to the postcss-custom-media plugin then and just some side effect as it on of the last plugins upgraded to PostCSS 7!?
Working PostCSS setup:
		"postcss": "7.0.2",
		"postcss-color-function": "4.0.1",
		"postcss-color-gray": "4.1.0",
		"postcss-custom-media": "6.0.0",
		"postcss-custom-properties": "7.0.0",
		"postcss-custom-selectors": "5.1.0",
		"postcss-extend": "1.0.5",
		"postcss-import": "12.0.0",
		"postcss-nesting": "6.0.0",
		"postcss-reporter": "6.0.0",
		"postcss-rtl": "1.3.2",
		"stylefmt": "6.0.2",

Upgrading the postcss-custom-media to 7.0.1 plugin is causing the error:
Unknown error from PostCSS plugin. Your current PostCSS version is 6.0.23, but autoprefixer uses 7.0.2. Perhaps this is the source of the error below.

I think it is just the error. Open an issue and post error message with a full stack trace and input CSS.

Andrey, I hope it is okay that you clone into my project and give it a try?
Master is working: https://travis-ci.org/redaxscript/redaxscript/builds/429001005
Debug branch is broken: https://travis-ci.org/redaxscript/redaxscript/builds/429001658

Please install the project: https://github.com/redaxscript/redaxscript#installation
Run grunt build and you will see it is working on the master
Update to latest postcss-custom-media and run grunt build again
You can find the PostCSS setup in tasks/postcss.js

Thank you, I hope it this workflow is okay for you...

Thanks for reporting this error. I believe this is resolved in v7.0.2 https://github.com/postcss/postcss-custom-media/commit/407c33cd8821a869db9dc6fac3d99681ee74a1ef.

Thank you, seems to be working now!


Proper Way To Get Current State With Immer ?

🙋♂ Proper Way To Get Current State
What is the proper way of getting the current state with immer. Do you reassign the output of produce to the state or use something else like "current" which can be imported from the immer package?
Does reassigning not negate the immutability practice of Immer, especially if using Typescript and the state was defined as read-only to enforce immutability. This has not been clear to me from the documentation, hence the question :
import { produce } from "immer";

type AppState = {
    readonly photos: string[];
};

// should state not be read-only and declared with const?
let state: AppState = {
    cards: []
};

// get photos straight from the state at any time.
// what's the right way??
export const getPhotos = () => state.photos;

export const addPhotos = (...recents: string[]) => {
   // re-assigning state here so we can use its latest update ??
    state = produce(state, (draft) => {
        draft.photos.push(...recents);
    });
    return state.photos;
};

This doesn't seem like an Immer question per se.
Is this being used with React? With Redux? Some other library?
Normally Immer is used to create a new state value, but the actual value is then stored and managed in whatever state management tool is appropriate (useState / useReducer for React, createSlice for Redux, etc).

This doesn't seem like an Immer question per se.
Is this being used with React? With Redux? Some other library?
Normally Immer is used to create a new state value, but the actual value is then stored and managed in whatever state management tool is appropriate (useState / useReducer for React, createSlice for Redux, etc).

It is being used in a vanilla HTML/CSS/Typescript frontend application that wants to centralize and better manage state like in modern frontend apps.

Immer doesn't know or care how you're "storing" the state. It's a function that calculates a new state, immutably.  It's up to you to save that result somewhere, just like you would any other value you've calculated.

Immer doesn't know or care how you're "storing" the state. It's a function that calculates a new state, immutably. It's up to you to save that result somewhere, just like you would any other value you've calculated.

This clears things for me. Thanks


RichCommands: SearchUnindexedItems

Resolved / Related Issues

[x] Were these changes approved in an issue or discussion with the project maintainers? In order to prevent extra work, feature requests and changes to the codebase must be approved before the pull request will be reviewed. This prevents extra work for the contributors and maintainers.
Related #11481

Validation
How did you test these changes?

[x] Did you build the app and test your changes?
[x] Did you check for accessibility? You can use Accessibility Insights for this.
[ ] Did you remove any strings from the en-us resource file?

[ ] Did you search the solution to see if the string is still being used?

[x] Did you implement any design changes to an existing feature?

[x] Was this change approved?

[x] Are there any other steps that were used to validate these changes?

Open app
Search something that is not indexed
Click on Search unindexed items

@ferrariofilippo can you split the PR to help have a clearer commit history?

@yaira2 I saw you told me to split this PR, but now I no longer see the comment (did you delete it?), should I still open a separate PR for PermanentDeletion?

It's preferred but I don't want to be too picky.

I'm not convinced this should have an action, it's likely to be confused between the setting and the button on the search page.

I'm not convinced this should have an action, it's likely to be confused between the setting and the button on the search page.

I think we could rename it, but we definitely need to add this action. Otherwise we would need to keep IBaseLayoutCommandImplementation just for this action.

@ferrariofilippo after giving this some thought, I decided that we should remap the button on the search to open the settings page where users can toggle the setting for themselves.
The settings dialog doesn't have support for deep linking to specific pages (not yet) so we're going to have to hold off on this.


TryLoadByJNI crash

BNM::Loading::TryLoadByJNI(env);

You set UNITY_VER to 213 in GlobalSettings.hpp?


fetchmany returns only a single row

Hi,
I am trying to use your driver and it works. Thanks!
Why does fetch_manyreturns a single row albeit it should return several.
I need this for faster dumping into a file (rather than line by line)

If you don't pass any parameter to fetchmany(), the default number of rows to be fetched is cursor.arraysize = 1. You can specify the size parameter to the number of rows to fetch per call.
Here are some solutions:

Use fetchall() to fetch all rows
Call fetchmany(size=N) to fetch N rows
Change cursor.arraysize attribute to M, then every time calling fetchmany() would fetch M rows

Note: The fetchmany method try to fetch as many rows as indicated by the size parameter. If this is not possible due to the specified number of rows not being available, fewer rows may be returned.


token flow resource fix

We didn't properly close the client class during the token flow – an issue caused by the client refactor.
Fixed it by making it hard to use the client without using a context manager
Also refactored the token flow a bit so that all output management sits in the CLI

This caused spurious warnings like this:
2023-01-05T01:28:16+0000 Channel pool not properly closed
2023-01-05T01:28:16+0000 Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fe51340c220>
transport: <_SelectorSocketTransport closing fd=8>
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 910, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 685, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 916, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 711, in _fatal_error
    self._force_close(exc)
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 723, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 719, in call_soon
    self._check_closed()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 508, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

They were harmless but a bit scary looking!


Domain/ValidationStatus: Fix possible enum values

Supported values don't match the ones on Resend documentation.

Resolved.


Ensure the docs site builds

We use docusaurus. We should ensure that it builds in the CI before merging to main.

Related question:
are we interested in a gh-pages deployment? http://polymerdao.github.io/monomer could serve as a temporary destination.

Related question:
are we interested in a gh-pages deployment? http://polymerdao.github.io/monomer could serve as a temporary destination.

Long-term, yes. I'm hesitant to pull the trigger now just in case we move the repo, which may require redirects and other additional complexity.


Better documentation

Probably need to set up a /docs/ folder to go in to detail with everything. Also need inline documentation.

Completely rewrote contributing, much nicer now.


Can not login by blank password

I'd reset my macvm password to empty pwd, after restart vm, it can not login by simply press enter button.
But it can pass the recovery choose disk asking permission password dialog.

Solved.
https://superuser.com/questions/1648047/how-to-set-up-user-account-from-terminal-in-m1-mac-big-sur
Open vm by recovery mode and open terminal, then check your folder, and use script like:
dscl -f "/Volumes/Macintosh HD/private/var/db/dslocal/nodes/Default" localonly -passwd /Local/Target/Users/myusername PASSWORD

it will need you to insert your old password (the blank password), if it didn't work, try again.


spherical polygons set build fails when two edges are aligned but in reverse direction

SphericalPolygonsSet instances can be built from a list of vertices, which are points on the 2D unit sphere. When these vertices form a zigzag or star shaped boundary and two distant edges happen
to be on the same circle (according to the hyperplaneThickness parameter setting) and these edges
are in opposite orientation, then the polygon built is completely wrong.
This test case is an example of this behavior. If the hyperplane thickness (first constructor parameter) is set to 1.0e-10, then all edges are considered to belong to separate circles and the zone is properly built.
If the hyperplane thickness is set to 1.0e-6, then the edge built from vertices at indices 6 and 7 (counting from 0) and the edge built from vertices at indices 10 and 11 are considered to belong to the same circle and the polygons built is different.
@Test
public void testZigZagBoundary() {
    SphericalPolygonsSet zone = new SphericalPolygonsSet(1.0e-6,
                                                         new S2Point(-0.12630940610562444, 0.8998192093789258),
                                                         new S2Point(-0.12731320182988207, 0.8963735568774486),
                                                         new S2Point(-0.1351107624622557,  0.8978258663483273),
                                                         new S2Point(-0.13545331405131725, 0.8966781238246179),
                                                         new S2Point(-0.14324883017454967, 0.8981309629283796),
                                                         new S2Point(-0.14359875625524995, 0.896983965573036),
                                                         new S2Point(-0.14749650541159384, 0.8977109994666864),
                                                         new S2Point(-0.14785037758231825, 0.8965644005442432),
                                                         new S2Point(-0.15369807257448784, 0.8976550608135502),
                                                         new S2Point(-0.1526225554339386,  0.9010934265410458),
                                                         new S2Point(-0.14679028466684121, 0.9000043396997698),
                                                         new S2Point(-0.14643807494172612, 0.9011511073761742),
                                                         new S2Point(-0.1386609051963748,  0.8996991539048602),
                                                         new S2Point(-0.13831601655974668, 0.9008466623902937),
                                                         new S2Point(-0.1305365419828323,  0.8993961857946309),
                                                         new S2Point(-0.1301989630405964,  0.9005444294061787));
    Assert.assertEquals(Region.Location.INSIDE, zone.checkPoint(new S2Point(-0.145, 0.898)));
    Assert.assertEquals(6.463e-5, zone.getSize(),         1.0e-7);
    Assert.assertEquals(5.487e-2, zone.getBoundarySize(), 1.0e-4);
}

A similar problem occurs in Euclidean 2D. The following test triggers a NullPointerException.
@Test
public void testOppositeEdges() {
    PolygonsSet polygon = new PolygonsSet(1.0e-6,
                                          new Vector2D(+1, -2),
                                          new Vector2D(+1,  0),
                                          new Vector2D(+2,  0),
                                          new Vector2D(-1, +2),
                                          new Vector2D(-1,  0),
                                          new Vector2D(-2,  0));
    Assert.assertEquals(6.0, polygon.getSize(), 1.0e-10);
    Assert.assertEquals(2.0 * FastMath.sqrt(13.0) + 6.0, polygon.getBoundarySize(), 1.0e-10);
}


Cron running completely off time

I have the this configuration:
config :my_app, MyApp.Scheduler,
  jobs: [
    {"0 5 * * *",  {MyApp.Scheduler, :restart_monitoring, []}},
    {"0 13 * * *", {MyApp.Scheduler, :restart_monitoring, []}},
    {"0 21 * * *", {MyApp.Scheduler, :restart_monitoring, []}},
    {"0 0 * * *",  {MyApp.Scheduler, :persist_csv_data, []}}
  ]

however looking at the logs generated by the restart_monitoring function, seems an execution has been started at 23:21:07 which is not even on 0 minutes, so shouldn't be a timezone issue, also the system clock (checked with date) seems correct, but even if it wasn't, syslog should report the wrong time, how should I debug this?

 I'll close the issue for now and reopen if someone else has a similar problem.


⚠ Cioos National CKAN Sitemap has degraded performance

In 72fe3ec, Cioos National CKAN Sitemap (https://catalogue.cioos.ca/sitemap/sitemap-1.xml) experienced degraded performance:

HTTP code: 200
Response time: 27511 ms

Resolved: Cioos National CKAN Sitemap performance has improved in 0d8635b after 10 minutes.


Broken pipe Exception

I post a large file to server, and it raise ConnectionError exception.
Traceback (most recent call last):
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/rq/worker.py", line 479, in perform_job
    rv = job.perform()
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/rq/job.py", line 466, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "./jobs/office.py", line 97, in notice_file
    open('test.pdf', 'rb')))
  File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/enterprise/client/api/media.py", line 15, in upload
    'media': media_file
  File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/api/base.py", line 20, in _post
    return self._client._post(url, **kwargs)
  File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/base.py", line 74, in _post
    **kwargs
  File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/base.py", line 39, in _request
    **kwargs
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/api.py", line 49, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))

The test.pdf is 4.2M, when I post it as a file, I get ConnectionError Exception.
But I use curl to post it to server, it can be uploaded successful.

Can you reproduce it every time? What curl command are you using? Does the
server have any useful log messages?
--
Kevin Burke
phone:<PHONE_NUMBER> | twentymilliseconds.com
On Fri, Jan 23, 2015 at 6:03 AM, Cloverstd<EMAIL_ADDRESS>wrote:

I post a large file to server, and it raise ConnectionError exception.
Traceback (most recent call last):
File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/rq/worker.py", line 479, in perform_job
rv = job.perform()
File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/rq/job.py", line 466, in perform
self._result = self.func(*self.args, **self.kwargs)
File "./jobs/office.py", line 97, in notice_file
open('test.pdf', 'rb')))
File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/enterprise/client/api/media.py", line 15, in upload
'media': media_file
File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/api/base.py", line 20, in _post
return self._client._post(url, **kwargs)
File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/base.py", line 74, in _post
**kwargs
File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/base.py", line 39, in _request
**kwargs
File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/api.py", line 49, in request
response = session.request(method=method, url=url, **kwargs)
File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))
The test.pdf is 4.2M, when I post it as a file, I get ConnectionError
Exception.
But I use curl to post it to server, it can be uploaded successful.
—
Reply to this email directly or view it on GitHub
https://github.com/kennethreitz/requests/issues/2422.

Have you tried without using rq? Are you using curl within rq?

Same here. I'm using requests to post large files (about 8 Mb in total) to a Django endpoint. Getting same error. It does work fine with smaller files.

@barseghyanartur perhaps you can give us more details than @cloverstd did, for example, what code will reproduce this?
I have a suspicion as to the cause but I haven't been able to reproduce this yet.

@sigmavirus24
I have gone a bit further in research and this is what I have found.
Basically, it's this:
requests.post(some_url, data=data_dict, files=files_dict)

Environment details:

Python==2.7.3
requests==2.5.1

I managed to reproduce the problem exactly. In my case, the endpoint was a multi-lingual Django site. Requests to URL without the language prefix (like "http://example.com/foo/endpoint/") redirected to same URL with the language prefix (like "http://example.com/en/foo/endpoint/").
If some_url is "http://example.com/foo/endpoint/" I get an error:
('Connection aborted.', error(32, 'Broken pipe'))
If some_url is "http://example.com/en/foo/endpoint/" it all goes fine. Likely, requests doesn't handle large post requests to endpoints which do redirect them to some other endpoints.
I have just tried it with requests==2.6.0 and I do get the same problem.

@barseghyanartur can you give us the status code from the redirect?

I wonder if the remote end isn't expecting a body but we're sending it anyway, so we get our connection torn down.

@Lukasa or if we change the method (i.e., in a case like 7231 Section 6.6.4) without stripping the body/content-length. (or strip the body and not the content-length).

Really any information @barseghyanartur can give us about the redirect response and the subsequent request will be helpful

@sigmavirus24
Hey, thanks for getting back on this. I'll post it here quite soon after I get home. :) That's gonna be in about 5 hours.

@barseghyanartur no rush. I'd like to fix this once and for all. That can be today or next week. As long as it's fixed, I'll be happy.

@sigmavirus24
I don't get any response back, since it fails hard.
Traceback (most recent call last):
File "requests_test.py", line 29, in
response = requests.post(url, data=data, files=files)
File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/api.py", line 108, in post
return request('post', url, data=data, json=json, **kwargs)
File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
resp = self.send(prep, **send_kwargs)
File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))
And this is the code:
import os

import requests

url = 'http://localhost:8001/foo/endpoint/'

data = dict(['input_{0}'.format(index), 'some value {0}'.format(index)] for index in range(10))

FILES = (
    '/home/me/bbrepos/requests_test/media/1.gif',
    '/home/me/bbrepos/requests_test/media/2.gif',
)

files = dict(['file_{0}'.format(index), (os.path.basename(file_path), open(file_path, 'rb'))] for index, file_path in enumerate(FILES))

response = requests.post(url, data=data, files=files)

@barseghyanartur you can determine what the redirect codes, etc. are with requests.post(url, data=data, files=files, allow_redirects=False)

@sigmavirus24
It was 301, as far as I remember.

Given that this is all plaintext, you should be able to get a packet capture of the whole transaction. Running tcpdump -nnvvXSs 1514 port 8001 and host localhost should do the trick.

I'm having the same problem:  small (1MB) data works, but large (50MB) data breaks ('Connection aborted.', error(32, 'Broken pipe')).
The complications in my case are that (1) I don't control the Web service -- it's the property of another company; (2) the protocol is HTTPS, but perhaps the packet capture would still be informative, even if the payload data is encrypted from tcpdump's point of view.
I will get a packet capture of a broken large transaction and reply back.

I created the tcpdump file https://gist.github.com/JazzFan/fc55e28c525518ada71f with this command:
sudo tcpdump -nnvvXSs 1514 port 443 and host <IP_ADDRESS>
My program ran under Python 2.7.8, with requests 2.5.3 (but I feel pretty confident that I can get it to break under other versions of Python or requests, if you would prefer others).
The relevant lines of the Python program:
    import requests
    ...
    headers = {'Content-Type':  'application/json'}
    response = requests.post(URL, data=wrapped_chunk, headers=headers)

The idea of the program is to split a large file into 50MB chunks, wrap each chunk with a bit of JSON, then POST it to the Web service API that a vendor company provides to my company.
The stack trace (home dir redacted with ~)
Traceback (most recent call last):
  File "~/bin/progname.py", line 399, in <module>
    main()
  File "~/bin/single_process.py", line 33, in _
    return f(*a, **kw)
  File "~/bin/progname.py", line 392, in main
    max_return_code = send_file_chunks(fn, api_key, args.parallelism)
  File "~/bin/progname.py", line 347, in send_file_chunks
    chunk_filenames))
  File "~/anaconda/envs/py278/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "~/anaconda/envs/py278/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
requests.exceptions.ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))

I realize the stack trace has everything but requests in it; let me know if that's a problem and I'll write a simpler program to show the problem with a clean stack trace.

@jazzfan thanks for doing that. I'll look tonight and I suspect that @Lukasa will look when he wakes up

@sigmavirus24 You forget that I am in NYC at the minute, and so am (relatively) awake!
As I've said elsewhere, EPIPE almost always comes when we attempt to send on a connection that has been remote-closed.
I think it would be most interesting if you could run a patch in requests' vendored copy of urllib3 that would throw the socket object up with the ConnectionError, and then read from it. The exact patch to apply here is a bit unclear and overlaps with some other stuff we've been thinking about when it comes to EPIPE.
Alternatively, a non-SSL repro would work!

@Lukasa, does this mean that the SSL tcpdump file is essentially useless?  Also, I'm afraid I'm too much of a newbie to be able to figure this patch you're describing, especially if it's unclear to you.
I guess I'll rig up some kind of little test HTTP server on my localhost and see if I can reproduce the error using that.

@JazzFan Sadly, it's not very easy to tell from it whether or not a response was sent. That said, we can definitely see that the remote end tore the connection down at 15:17:24.141410 (sending a TCP FIN). Shortly thereafter we emitted another packet, which caused the remote end to send a barrage of TCP RSTs (though most of these appear to be retransmits, and in practice appear to be in response to the other TCP packets that were in flight when the FIN was emitted (as shown by the sequence/acknowledgement numbers).
What we really want to understand is what triggered the connection teardown.

@Lukasa
Well, one thing is clear - it happens only when large amounts of data (files) and transferred.

@barseghyanartur Yup, but that doesn't necessarily mean much. You can only get an EPIPE if the remote end forcibly closes the connection, which is only likely to happen for large data files.

@Lukasa
Are you interested to know what happens if 20 Mb plain text (textarea) is posted?

I'm really interested in seeing something happening here in plaintext. =D

Well, for whatever reason I just can't get tcpdump to actually capture the packets.  I get output like this:
0 packets captured
303 packets received by filter
0 packets dropped by kernel

But this example breaks for me, again with Python 2.7.8 and requests 2.5.3:
#!/usr/bin/env python

import requests

line = '.' * 79 + '\n'
data = line * 10000000
response = requests.post('http://<IP_ADDRESS>:8000', data=data)
print(response)

Here is the stack trace (again with home dir redacted as ~):
$ ./requests_test.py
Traceback (most recent call last):
  File "./requests_test.py", line 7, in <module>
    response = requests.post('http://<IP_ADDRESS>:8000', data=data)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/api.py", line 99, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/api.py", line 49, in request
    response = session.request(method=method, url=url, **kwargs)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))

I can capture the packets with Wireshark.  I know that you're looking for a particular kind of plaintext output from tcpdump, but would you happen to know how to get the equivalent plaintext output from Wireshark?
On the off chance that it would work for you, I've created a gist, wireshark.txt, https://gist.github.com/JazzFan/8517983ff5eed2d95b10.

@JazzFan In addition to what you sent me I need the actual packet data. There's a 40-byte TCP payload coming from your server in frame 60 that Wireshark is claiming is HTTP data, but it hasn't broken out the data itself so I can't see it.
What tcpdump command are you running?

I'm running
sudo tcpdump -nnvvXSs 1514 port 8000 and host <IP_ADDRESS> > tcpdump.txt

and variations on that (e.g., omitting the and host <IP_ADDRESS>).  After triggering the bug in another window and hitting Ctrl-C on the tcpdump command, I get something like
0 packets captured
492 packets received by filter
0 packets dropped by kernel

and an empty tcpdump.txt file.  I interpret the output to mean that the filter saw the packets but didn't capture them, but I don't know why it didn't capture them.

Also, FWIW, the 40 bytes of TCP data payload on packet 60 are

POST!

This is just the standard response that my dummy HTTP server makes to any post request.   FWIW, here is my HTTP server:
#!/usr/bin/env python
"""

Very simple HTTP server in python.

Usage::
    ./dummy-web-server.py [<port>]

Send a GET request::
    curl http://localhost

Send a HEAD request::
    curl -I http://localhost

Send a POST request::
    curl -d "foo=bar&bin=baz" http://localhost

"""
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import SocketServer

class S(BaseHTTPRequestHandler):
    def _set_headers(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()

    def do_GET(self):
        self._set_headers()
        self.wfile.write("
hi!
")

    def do_HEAD(self):
        self._set_headers()

    def do_POST(self):
        # Doesn't do anything with posted data
        self._set_headers()
        self.wfile.write("
POST!
")

def run(server_class=HTTPServer, handler_class=S, port=8000):
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    print 'Starting httpd...'
    httpd.serve_forever()

if __name__ == "__main__":
    from sys import argv

    if len(argv) == 2:
        run(port=int(argv[1]))
    else:
        run()

Oops, I forgot to include @Lukasa in my previous comments.

@JazzFan I haven't tested this yet (I'm at PyCon so I'm short on time), but my suspicion is that your test server causes this by immediately writing a response and then going out of scope without draining the POST data. This means that the server will attempt to close the socket, sending a TCP FIN, and then causing the Broken Pipe error.

@Lukasa, please don't let me take you away from PyCon!
I think you are right about needing to drain the POST data.  When I change my test server's do_POST method from this:
    def do_POST(self):
        self._set_headers()
        self.wfile.write("
POST!
")

to this, by adding code to read all of the input stream for this post request:
    def do_POST(self):
        length = int(self.headers["Content-Length"])
        self.rfile.read(length)   # make sure to drain the POST data
        self._set_headers()
        self.wfile.write("
POST!
")

then the error goes away; requests does not terminate in a stack trace.
So based on my evidence, when requests abnormally terminates, it is due to a problem in the server implementation, not in requests itself.  Maybe other people have evidence that it's a problem in requests, but on the face of it, I don't.   Sorry to have troubled you!

No problem @JazzFan! I think we still would like to surface responses in this case, but it's another issue.

Hi, I realise this is closed, but I'm having a similar issue, also with a Flask server. I'm sending a JSON GET request to a flask route that only accepts POST requests. Flask returns immediately with a 403 (Bad Method), but if the request data is large (7Mb in my case), requests keeps trying to push data until the request is fully sent. I'm not doing a file upload.

Wouldn't it be better to return the 405 in this case, since it implies the request was finished, even if the pipe closed prematurely?

Yes. What we don't know is whether we can. We haven't had anyone sit down to work out if httplib will let us do the right thing here.

Okay, I'm happy to do this, just let me know how. I at least have control of the server and the client in this case, but if you need tcp dumps, I'll need no anonymise the test data (It's health stuff)

You'll need to familiarise yourself with httplib in order to make the troublesome request, and then see if you can still meaningfully receive the response.

What sort of level am I looking at here? Am I using httplib.socket.create_connection, or httplib.HTTPConnection?

HTTPConnection is fine. =)

Hmm, this doesn't look positive. It looks like the httplib.HTTPConnection object raises the exception, and doesn't return a response object.
Client code:
$ python
Python 2.7.10 (default, May 30 2015, 12:53:33)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib
>>> conn = httplib.HTTPConnection("localhost", 8080)
>>> R = conn.request("GET", "http://localhost:8080/MyService/", open("example-input.json", "r"), {"Content-Type": "application/json", "Accept": "application/json"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/httplib.py", line 1053, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1093, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 897, in _send_output
    self.send(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 866, in send
    self.sock.sendall(datablock)
  File "/usr/lib64/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 32] Broken pipe
>>>

Server output:
<IP_ADDRESS> - - [21/Jul/2015 15:36:25] "GET http://localhost:8080/SleepModel/ HTTP/1.1" 405 -

I know that. What I want you to do is follow-up from that with conn.getresponse() and see what happens.

Oh right, yup it works,
>>> new_R = conn.getresponse()
>>> new_R.status
405

Ok, so in principle there are certain kinds of exceptions we can still get responses from. I think I floated the idea that we could attach them to the throw exception for BrokenPipe.

@Lukasa I am seeing a similar issue, but the cause is from nginx immediately returning a 413 Request Entiry Too Large. RFC2616 says this about 413:

The server MAY close the connection to prevent the client from continuing the request.

The server closing the connection is the behaviour I am seeing, and requests (actually, I think the issue is in Python's httplib) is not handling that case.
I have a server on EC2 that can be used for testing against. nginx is configured with a max_client_body_size of 1k, and the Python server behind nginx is the example posted above in this issue (with the modification to consume the whole body). The catch is that nginx doesn't proxy the request to the Python server due to the 413 error. You can POST to http://<IP_ADDRESS>/
Using wireshark, I see the below behaviour, where nginx responds upfront with a 413, but Python httplib doesn't handle the response until the very end of the request (happily uploading the whole file while the server has already rejected the request). If the file is large enough, nginx will terminate the connection before httplib/urllib3 handles the response, resulting in either a Broken Pipe, or a Connection reset by peer.
No.	Time			Source		Dest.			Proto.	Size	Info.
41	4.584020000	<IP_ADDRESS>		<IP_ADDRESS>	HTTP	1359	POST / HTTP/1.1
42	4.601824000	<IP_ADDRESS>	<IP_ADDRESS>		TCP		66		80→34354 [ACK] Seq=1 Ack=194 Win=28032 Len=0 TSval=26477364 TSecr=102696248
43	4.602120000	<IP_ADDRESS>	<IP_ADDRESS>		HTTP	446		HTTP/1.1 413 Request Entity Too Large  (text/html)
44	4.602191000	<IP_ADDRESS>		<IP_ADDRESS>	TCP		66		34354→80 [ACK] Seq=1487 Ack=381 Win=30336 Len=0 TSval=102696269 TSecr=26477364
45	4.602875000	<IP_ADDRESS>		<IP_ADDRESS>	TCP		66		34354→80 [FIN, ACK] Seq=1487 Ack=381 Win=30336 Len=0 TSval=102696269 TSecr=26477364
...
continues

cc @cournape

Yeah, as I've said, I think we can do the right thing here in some situations, but I'd want to test a lot of edge cases to confirm we don't get stuck in situations that aren't like this one.

I think that (in my case, at least) there is an underlying issue in httplib where it is just not possible to end a request at the time when the server has sent an error response. The client always tries to send the whole body. So I think while you may be able to give the user a better error (i.e. what the server actually sent), the overall user experience will still be worse than expected unless you override the httplib.HTTPConnection.send() method to handle the server response as soon as it is received. Although that seems to have pretty severe consequences for the whole design of httplib.

Yeah, you're absolutely right on that front. This will always be sub-par, but I think that's ok: raising an exception in this case is probably the right thing to do.

@sjagoe Thanks for this info!

Using wireshark, I see the below behaviour, where nginx responds upfront with a 413, but Python httplib doesn't handle the response until the very end of the request (happily uploading the whole file while the server has already rejected the request). If the file is large enough, nginx will terminate the connection before httplib/urllib3 handles the response, resulting in either a Broken Pipe, or a Connection reset by peer.

This indeed seem to be the case.
I have the same exact trouble for a completely different server (VirusTotal). The easy way to check this behavior is by creating a file slightly bigger than 64Mib and another slightly smaller with:
# Create a 64 MiB file:
$ dd if=/dev/urandom of=zzzz.txt bs=1048576 count=64
...
67108864 bytes (67 MB) copied, 0.182048 s, 369 MB/s

Submitting this gives the (104, 'ECONNRESET') exception, indicating that the foreign server dropped the connection. (This happens fast.)
The smaller file was created with:
$ dd if=/dev/urandom of=zzzz.txt bs=1048576 count=63
...
66060288 bytes (66 MB) copied, 0.184529 s, 358 MB/s

Indeed results in the correct server response of 413, exceeding the 32 MB API limit. (This process is slow.)
What is weird is the speed of the returned result, which is opposite of what is expected.

May i ask if there is a resolution for this issue ?

@chenziliang There is not at this time.

Thanks @Lukasa ! May i ask if there is workaround ? I didn't run into broken pipe issue by using httplib2 by issuing the same request with the same payload.

@chenziliang Are you sure they were the same? httplib2 and requests handle connections differently and set different headers, so it would be worth using something like Wireshark to confirm that the behaviours really are the same.

From application level, probably they are the same, since same headers, payload etc. But will use wireshark to confirm. Thanks !

I see that this issue is closed (https://github.com/kennethreitz/requests/issues/2422#issuecomment-91747721), but is it? Subsequent comments indicate that there's a problem that remains.
I've also just experienced the issue where a server doesn't consume the POST request, but replies with an error (the request was invalid) and then closes down the socket. The server is permitted to do this as it has also indicated a Connection: close in its reply.
As consequence my user receives a BrokenPipeError in place of the HTTP error returned by the server.

@huntc The ultimate problem is that Requests currently has no way to detect that a response has arrived and abort processing of the request. httplib just doesn't let us in at that low a level. The only fix to your problem is to remove httplib from the equation, and while that's a thing we're working on it represents a really pretty major change to urllib3 and to Requests. It'll take a while to happen.

Thanks for the explanation. Isn't the issue still technically an issue that remains though, and remain so until it is fixed? Had I first seen that the issue was open then it would've prevented me from wading through the comments only to discover that it is still an issue? Seeing that the issue was closed initially distracted me from treating it as a requests one, and caused me to spend time in the server, which ends up doing the right thing. I appreciate that you might not be able to fix this for a while though. Thanks.

It depends on what your definition of this issue is. We can keep a ticket open for the general case of "requests doesn't handle early 4XX errors during requests with large bodies", but that's not what this issue was really about, and the original poster of this issue is totally AWOL. That's fundamentally why it was closed: this issue is a confused mess of different discussions, none of which is actually the issue you're discussing and encountering.
Usually, when people encounter this issue, the actual problem they have is fixable: they're doing something to trigger a 4XX response from a server, and they should un-do that. That's why we closed the issue.

Fair call!
I think in general the issue is that the request lib cannot handle a reply from the server while in the midst of sending its request payload.
Our server will consume the first part of a multipart request and validate it before continuing to read the remainder. If it finds that the part is invalid then it will reply without consuming the remainder of the request.
This is a valid 1.1 scenario of course. If you'd like me to open another issue then I'd be happy to. Otherwise I'll leave you alone. Thanks!

So I agree that the issue we're discussing here is a real Requests problem. We can open a ticket for that, but the reality is that such a ticket is only of value in the sense that it says "Yes, we know there is a problem. No, we have no timeline for fixing it." It has been my experience that issues like this are mostly a magnet for a hoard of "is this fixed yet" questions, which repeatedly require the answer "no".
If you want to open such a ticket, I won't close it. =) I will, however, almost certainly lock the conversation because any conversation on that ticket is unlikely to be of much value. In the meantime, there is work underway to try to get to a place where we can resolve this problem, but we're simply not there yet and unlikely to be any time soon.
Ultimately it's up to you. =)

This is a fairly serious issue when using Kerberos (with requests-kerberos), since it returns a 401 challenge as part of the normal negotiation process.

@DavidMcLaughlin We know it's a problem, we know it affects lots of people, and we know that the amount of work that is required to fix it is really very substantial. There is no amount of reprioritisation that will get this moved up the list because the work required is O(months) of time. I am currently working on it in and amongst my many other maintenance duties.
As I noted in my above comment, I warned that tickets like this are unlikely to be of much value because of a hoard of "me too" comments. Now that they have begun, I am immediately locking this issue to further discussion.


Support multiple examples specified in Open API schema

As mentioned in #582 we can generate multiple examples via the examples keyword:
parameters:
  - in: query
    name: limit
    schema:
      type: integer
      maximum: 50
    examples:       # Multiple examples
      zero:         # Distinct name
        value: 0    # Example value
        summary: A sample limit value  # Optional description
      max: # Distinct name
        value: 50   # Example value
        summary: A sample limit value

And I am very happy to hear that you'd like to implement it :) Let me know if I can provide support on this

@Stranger6667 sounds great! I will start exploring the approaches you mentioned and will see your approach to GraphQL when you open a PR. Just to clarify, I plan to implement the version where we pair examples arbitrarily:
request(limit=0, id=1), request(limit=50, id=2), request(limit=0, id=3)

Also, do we need to consider the possibility of property-level examples in request bodies as discussed in #467, or will #467 be implemented later?

I think that we can start with limited scope for now and expand it later :) But it will be good if we'll arrange the code for this feature in the way so it will be easier to extend it later with #467

@Stranger6667 it looks like the jsonschema package does not support the examples keyword.

I looked at the JSON Schema specification and found that it does have an examples keyword, but they treat the value of examples as an array.
"examples" : [
    "Anything",
    4035
]

I'm going to open an issue against hypothesis-jsonschema, and I hope they will consider supporting examples without validating the value of examples`.

Hm, it doesn't seem related to the jsonschema package, since we have JSON Schema definitions for Open API copied from the official sources, and there are examples keywords in the relevant places. What schema are you validating?
I am also not sure how hypothesis-jsonschema is related to it since it generates valid data for the passed schema. Could you, please, provide more info?

Based on these definitions we have the following
Open API 2:
examples is an object with an unlimited amount of properties
Only examples inside response:
"200":
    description: Test
    examples:
      foo: ...

Open API 3:
There are more places where it is possible to put examples.

inside Components which is probably not relevant
inside MediaType, basically in the response or parameter serialization
in Parameter

Example in Open API 3 has some more validation, but it is an object still:
        {
            "type": "object",
            "properties": {
                "summary": {"type": "string"},
                "description": {"type": "string"},
                "value": {},
                "externalValue": {"type": "string", "format": "uri-reference"},
            },
            "patternProperties": {"^x-": {}},
            "additionalProperties": False,
        }

What version of the spec you were working with?

You're right. I thought the examples were defined under schema the same way an example is defined. This is what I had that produces the exception above.
components:
    schemas:
        SimpleSchema:
            type: object
            description: an object with a string field
            required:
                - foo
            properties:
                foo:
                    type: string
            examples:
                example1:
                    value:
                        foo: this is the first example!
                example2:
                    value:
                        foo: this is the other example!

I did not notice the difference when reviewing the OpenAPI spec, but I moved the examples "up" one level, and the schema was accepted.

@Stranger6667 I made a bit of progress on this issue, but I am stuck on trying to understand where the code should live and how to handle a requestBody with multiple mediaTypes.
My strategy is to use the EndpointDefinition to collect all of the examples from parameters and collect all of the examples from requestBody and then merge the examples to form a list of static_parameter dictionaries.
So far, I have written code for this in _hypothesis.py. This is code is incomplete. I am just showing the code to retrieve all of the examples from parameters:
def get_static_params_from_examples(endpoint: Endpoint)-> Dict[str, Any]:
    endpoint_def = endpoint.definition.raw
    all_parameter_examples = []
    for param in endpoint_def.get("parameters", []):
        if "examples" in param:
            # add value to list of params with examples
            param_type = {"header": "headers", "cookie": "cookies", "path": "path_parameters", "query": "query"}.get(param["in"])
            assert param_type in PARAMETERS, f"param type: {param_type} not in {PARAMETERS}"
            all_parameter_examples.append({
                "type": param_type,
                "name": param["name"],
                "examples": [example["value"] for example in param["examples"]],
            })
...

Questions:

Currently, this code is in _hypothesis.py. Do we need to implement this in specs.openapi.schemas to distinguish between OpenAPI 3 and Swagger 2 implementations? Should we implement this in Swagger 2 using x-examples because examples are only supported for responses? This is what I'm thinking:
class Endpoint:
...
def static_params_from_examples(self)-> Dict[str, Any]:
return self.schema.static_params_from_examples(self)

How should we retrieve examples from requestBody with multiple mediaTypes? Should we collect examples from all mediaTypes? Or does Schemathesis only use one mediaType for testing (if, so how do we know which mediaType is being used)?
 requestBody:
     content:
         application/json:
             examples: ...
             schema: ...
         multipart/form-data:
             examples: ...
             schema: ...
 ...

This question assumes we should collect examples from all mediaTypes. Should  body and form_databe mutually exclusive in the static_parameters object? As in, we should not have this:
static_parameters = {
...
"body": {
...
},
"form_data": {
...
}

Is there a well-defined mapping Schemathesis that shows which mediaTypes map to the body parameter and which mediaTypes map to the form_data parameter?

Hi @barrett-schonefeld
I'll reply later today or tomorrow

@Stranger6667 hey, just want to check in. I plan to continue with this work tomorrow.

Hi @barrett-schonefeld
Sorry for not getting back to you.

Yep, I think we can put it to specs/openapi/examples.py and then connect it in the way you mentioned.
Currently, we don't handle multiple media types - only the first one defined in content or application/json in openapi/serialization.py. Probably we can leave it for later and support only the first media type - supporting multiple media types in all areas will require much more work.
I think we can have only one media type for now
Oh, no, I don't think we have it right now

Let me know if I can provide more info

Implemented in #620


Use Nullable.GetValueOrDefault()

We have over 270 uses where Nullable.HasValue is before Nullable.Value.
Most of these use cases could be replaced with Nullable.GetValueOrDefault() in some way.
(It is not LINQ!)
https://github.com/dotnet/runtime/issues/33792

can I give this a go?

@georgettica Yes, feel free to pull PR. No need to fix all at one PR - please keep PR as small as possible. I suggest you make only changes of the same type in one PR - there are many different patterns and I suggest to fix only one pattern per PR.

wow! good job yall! I am gonna back off from the other PR's i've made and let you do your thing

thanks for all your support @iSazonov
and sry if I made you do double work

@iSazonov Please see my questions left in #13804 and #13793. It's not clear why changing in thsoe two places.

Quote my comment from https://github.com/PowerShell/PowerShell/pull/13804#discussion_r524868056:

I think the comment https://github.com/dotnet/runtime/issues/33792#issuecomment-649665286 in that issue has a point. .Value is semantically more correct given that HasValue is already checked. There is a duplicate check on the boolean hasvalue field, but I don't think that affects the perf in practice.
For places where we can use GetValueOrDefault to replace both .HasValue and .Value, it makes more sense to make the change, like in #13805 and #13808. But if we need to keep the .HasValue check, then I think using .Value afterwards is better.

I agree in common. Only there is another side - using different patterns decreases readability. I would prefer to use one pattern even though it looks unusual, especially since there are only a few of them in our code base. There is another reason why I prefer this. Now we often look at a code in .Net Runtime for a better understanding of how an application works and we consider this code as the best practice and high quality. Switching to a different code with a different style immediately causes discomfort. I believe that following .Net in this sense is the right direction - many of those who learn PowerShell code will be grateful to us.

If we're going to use GetValueOrDefault() there, we shouldn't also be checking HasValue. It's fine to use one or the other, but using both is duplicating the same checks unnecessarily.

Only there is another side - using different patterns decreases readability.

This may be true to some syntax, but to Nullable<T>.Value and Nullable<T>.GetValueOrDefault(), I don't think readability would be affected given that the property/method names are so readable.

@vexx32 This could change execution logic - no value and default value can have different semantics in a code path.

This may be true to some syntax

Yes, I say about common rule we could/should follow but there are always exceptions to rule. :-)

When the .HasValue check is necesary, I think we should keep using .Value afterwards. It's semantically more clear that "I want the current value, not the default value". When the .HasValue is unnecessary, it makes sense to use .GetValueOrDefault(), like in #13805 and #13808.
@iSazonov If you agree, I think we should revert  #13804 and #13793

I have no objection.

In .NET 6.0, Nullable<T>.Value following Nullable<T>.HasValue has now been optimized in the JIT (dotnet/runtime#13699).
So we can close this issue and revert changes to use GetValueOrDefault() to the more natural form.

@iSazonov Actually, there may still exist some patterns that could be optimised like in #13808, for example:
bool M(bool? b) => b.HasValue && b.Value;

.NET 6.0.0 (<IP_ADDRESS>719), X64 RyuJIT
; Program.M(System.Nullable`1<Boolean>)
       mov       [rsp+10],rdx
       cmp       byte ptr [rsp+10],0
       je        short M01_L00
       movzx     eax,byte ptr [rsp+11]
       ret
M01_L00:
       xor       eax,eax
       ret
; Total bytes of code 21

bool N(bool? b) => b.GetValueOrDefault();

.NET 6.0.0 (<IP_ADDRESS>719), X64 RyuJIT
; Program.N(System.Nullable`1<Boolean>)
       mov       [rsp+10],rdx
       movzx     eax,byte ptr [rsp+11]
       ret
; Total bytes of code 11

@xtqqczze The issue was more for cleanup and using right pattern than performance. For performance it is better to look and measure specific scenario.

@iSazonov Unfortunately, most of @bcwood's PRs linked to this issue seem to have issues with change of logic. Could you please consider closing them and look at reverting #13849 as explained there: https://github.com/PowerShell/PowerShell/pull/13849#issuecomment-903717301.

My apologies. I see what you're getting at with the change of logic. I'll close them for now until I have a chance to re-evaluate.


Default behavior of the react plugin (jsx is invalid in js files)

Describe the bug
First things first, I agree people shouldn't put jsx in js files. Second, I opened this as an issue since the react plugin actually logs a solution in the console when that happens. But that solution doesn't work.
✘ [ERROR] The JSX syntax extension is not currently enabled

    main.js:6:11:
      6 │     return
Hello world!
;
        ╵            ^

  The esbuild loader for this file is currently set to "js" but it must be set to "jsx" to be able to parse JSX syntax. You can use "loader: { '.js': 'jsx' }" to do that.

After you specify the loader, you receive this error:
[vite] Internal server error: Failed to parse source for import analysis because the content contains invalid JS syntax. If you are using JSX, make sure to name the file with the .jsx or .tsx extension.
  Plugin: vite:import-analysis
  File: /home/projects/vitejs-vite-xundyr/main.js:6:32
  3  |  class Main extends React.Component {
  4  |    render() {
  5  |      return
Hello world!
;
     |                                   ^
  6  |    }
  7  |  }

Use case
In scrimba, we teach people to code (front-end for now), and a part of the learning experience is for students to download a project they're working on and run it locally.
We've been using webpack till now, and we want to move to a simpler setup by using Vite instead.
Teachers in scrimba agree to follow the convention, but I still want to provide a good experience for our students and make projects "just work" even if js files contain jsx.
Solution?
How do I make it work? The config file in the repro didn't fix the issue and I didn't find any solution so far in the internet
Reproduction
https://stackblitz.com/edit/vitejs-vite-xundyr?file=vite.config.js,main.js,index.html&terminal=dev
Steps to reproduce
No response
System Info
*

Used Package Manager
npm
Logs
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

JSX in .js files won't be officially supported by Vite. https://github.com/vitejs/vite/issues/2727#issuecomment-808679467
This config seems to work though. (You won't get HMR support)
import { defineConfig } from 'vite';

export default defineConfig({
  esbuild: {
    loader: 'jsx',
    include: /.*\.jsx?$/,
    exclude: []
  },
  optimizeDeps: {
    esbuildOptions: {
      loader: {
        '.js': 'jsx',
      },
    },
  },
});

The esbuild loader for this file is currently set to "js" but it must be set to "jsx" to be able to parse JSX syntax. You can use "loader: { '.js': 'jsx' }" to do that.

This error is coming from esbuild and should be changed by Vite that tell you to change the file extension but was passed through.

Thanks @sapphi-red that did the trick.


How to quit Android App？I used BackHandler

Version
Tell us which versions you are using:

react-native-router-flux ^4.0.0-beta.21 (v3 is not supported)
react-native v0.48.0

Expected behaviour
Exit App
Actual behaviour
Use BackHandler after crashing
Steps to reproduce

const reducerCreate = params => {
    const defaultReducer = new Reducer(params);
    return (state, action) => {
        console.log('ACTION:',action)

        if(action.type === 'Navigation/BACK'){
            console.log('退出APP');
            BackHandler.exitApp();
        }
}

2.Error: Cannot read property 'startsWith' of undefined
handleException @ index.android.bundle:11796
handleError @ index.android.bundle:12611
reportFatalError @ index.android.bundle:685
__guard @ index.android.bundle:2064
callFunctionReturnFlushedQueue @ index.android.bundle:1947
(anonymous) @ debuggerWorker.js:71

You seem to call startsWith method on something that is undefined. I don't think backHandler is responsible for this error. Could you check if error is not occuring elsewhere ?

You  can try to override BackHandler from react native in root component

I'm adding the code in the Example @vhoen

I was in the Example Launch  to add your code, click the Android return button can not be used

this library has solved my issue
https://github.com/wumke/react-native-exit-app


Teams notification won't go away

There's a notification on the "teams" side tab, and it won't go away.
I think it's because a member of my team was locked out of his account, and reset it. Ever since then, there's this "ghost notif" that I can't shake.
Keybase GUI Version: 5.8.0-20210920184218+ac2177aec7

We're now running into this as well.  Same root cause; same annoying orange circled number.
If there's a GUI or CLI way to reset that count, we'd surely appreciate it.

Same here.  UI has no option to remove notification dialog once it's present.

Same here. On top of what was mentioned above there is no indication in the UI what that notification is about.

same.. still present

seems related: #13822

This has no solution. Unfortunately, Keybase is scrapped. There are some updates, but no support. Issues are often ignored here.
The only solution that worked for me was to leave the group. I couldn't even delete it, so I gave ownership of the group to an account that wasn't being used and left it.


icon-sdk-js randomly throwing Unexpected token < in JSON at position 0

Bug: This is a very weird bug that I have faced in production. Have occurred twice in the last month. I couldn't exactly pinpoint the place which caused this because occurs rarely in the codebase.
I suspect that this might have been triggered through a getter call. Please look into it.

Here is the implementation:
  public async invokeCallBuilder(score: string, method: string, params: any): Promise<unknown> {
    try {
      const call = new CallBuilder().to(score).method(method).params(params).build();
      const response = await this.iconService.call(call).execute();
      return response;
    } catch (error) {
      throw new Error(error);
    }
  }

@inwonkim Do you have any comments on this?

Need more information.
Can you inform what data is passed to params?

@0xrpj Can I get the endpoint, SCORE address information?
The response value seems strange.

According to the discord comments, it uses https://ctz.solidwallet.io/api/v3 for the endpoint.
In node implementation, it doesn't send any HTML response. So, I think that it may come from HTTPS reverse proxy which is applied to the citizens. Nowadays, some nodes crash from JSON RPC Bug(which is fixed in v1.3.7).
So, it may cause HTML response from the reverse proxy and an exception in parsing JSON.


Execution order with before and describe

Hello,
If you follow this example :
before(function() { console.log('before exterior'); });
    after(function() { console.log('after exterior'); });
    describe('test', function() {
        describe('testUnder', function() {

            var declaration = "Declaration";
            console.log(declaration);

            before(function() { console.log('before'); });
            after(function() { console.log('after'); });

            beforeEach(function() { console.log('beforeEach'); })
            afterEach(function() { console.log('afterEach'); });

            it('fails sync', function(done) {
                console.log("Func 1");
                done();
                //throw new Error('failed');
            });

            it('fails async', function(done) {
                console.log("Func 2");
                process.nextTick(function() {
                    done();
                    //throw new Error('failed');
                });
            });
        });
    });

The result is  :
LOG: 'Declaration'
LOG: 'before exterior'
LOG: 'before'
LOG: 'beforeEach'
LOG: 'Func 1'

Start:
  test
    testUnder
      ✔ fails sync
LOG: 'afterEach'
LOG: 'beforeEach'
LOG: 'Func 2'
      ✔ fails async
LOG: 'afterEach'
LOG: 'after'
LOG: 'after exterior'

I notice that the variable assignment of declaration is done before the execution of the before statement. It looks really counterintuitive.
Should not it be the opposite ?

console.log(test); should be console.log(declaration);
It couldn't work differently. When the describe block is executed in order to register descending hooks, suites and tests, console.log(declaration); is found before anything else inside that function/block.
Furthermore, until all suites have been registered, no tests will be ran:
➜  temp  cat test/lol.js
describe('suite 1', function () {
  console.log('a log inside suite 1')
  it('should pass', function () {})
})

describe('suite 2', function () {
  console.log('a log inside suite 2')
  it('should pass', function () {})
})
➜  temp  mocha
a log inside suite 1
a log inside suite 2

  suite 1
    ✓ should pass

  suite 2
    ✓ should pass

  2 passing (3ms)

If it's not clear, please let us know. I hope this helped!

Thank you.
Sorry for my mistake :

console.log(test); should be console.log(declaration);

I just corrected it.


runTsc breaks in Typescript v5.7 when providing extraSupportedExtensions

When using the runTsc function from the @volar/typescript npm package, the code will break if the user has Typescript v5.7 or higher and they have a non-empty array provided on the extraSupportedExtensions option (see this issue for an example).
This happens because in Typescript v5.7, they start making use of a shim in order enable Node.js/v8 compile caching. This means that the tscPath provided to runTsc may be a shim that pulls in the main code. It's no longer safe to assume like we do in transformTscContent that the tsc argument includes any of the code we want to update.
I spent quite a while digging through to code to figure out the exact issue. I'm happy to submit a PR to fix the issue, but I thought it might be best to submit an issue first in case we wanted to discuss what the ideal solution would look before I start messing with code.
The easiest way I can see to solve it off the top of my head would be to add a check for the file that gets required at the end of the file at the tscPath if it's missing the content that we expect, but that could be finicky. I'm open to suggestions.

Since Volar relies on parsing Typescript's source code, it's not immune to updates, but I put up a PR that makes the safest assumptions about the structure of the code that I can think to make. I'm happy to change things if someone has a better idea on how to handle it, but the changes do fix the problem while maintaining backwards compatibility.

parsing Typescript's source code

This seems insane. Why not contribute the necessary APIs that you need to Typescript and have volar function as a plugin to typescript, enabling users to run the official tsc? Would be much cleaner and not break every new Typescript release.


testing pull request to trigger ci-int

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update
Refactor (refactoring code, removing useless files)
Unit tests (no changes in the code)
Bump-up dependent library (no changes in the code)
Behavioral tests (no changes in the code)
REST API tests
Benchmarks (no changes in the code)
Documentation update

Testing steps
Please describe how the change was tested locally. If, for some reason, the testing was not done or not done fully, please describe what are the testing steps.
Checklist

[ ] make before_commit passes
[ ] updated documentation wherever necessary
[ ] added or modified tests if necessary
[ ] updated schemas and validators in insights-data-schemas in case of input/output change

Codecov Report

Merging #131 (a90d26a) into stable (0107808) will increase coverage by 18.45%.
The diff coverage is 5.68%.

@@             Coverage Diff             @@
##           stable     #131       +/-   ##
===========================================
+ Coverage    0.00%   18.45%   +18.45%
===========================================
  Files           3        3
  Lines         374      439       +65
===========================================
+ Hits            0       81       +81
+ Misses        374      347       -27
- Partials        0       11       +11

Impacted Files
Coverage Δ

cleaner.go
0.00% <0.00%> (ø)

storage.go
20.09% <ø> (ø)

config.go
68.51% <27.77%> (+68.51%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0107808...a90d26a. Read the comment docs.


No uri change till client. fs and store will have uri of config.

The uri in fs and store is changed from the one given in config to something else if enbaleBlob is set true. for ex:
<EMAIL_ADDRESS>to<EMAIL_ADDRESS>This is not required to flow in correct uri to client. Reason being a given method in AbfsClient will either call dfs endpoint or blob endpoint. No method will be such that it can call both endpoint and need hook from fs to call the required endpoint. If a method has to call dfs endpoint, it will always call that. Similarly, any method calling blob endpoint, will always call blob endpoint.
Change in pr:

the uri that flows throught fs and store to client will be what is given in config.
If a method is not compatible with the uri, it will change that via changePrefixFromBlobtoDfs or changePrefixFromDfsToBlob method.
The prefixMode is set in the fs. Store uses prefixMode is decide if it has to carry out an operation for dfs or blob. And as per the flow it calls endpoint relevant methods in client.

What will a be major change:

The path in the fileStatus object will have the endpoint as given in uri of config. If its<EMAIL_ADDRESS>
fs.getFileStatus("/path) or<EMAIL_ADDRESS>or<EMAIL_ADDRESS>will give filestatus obj having path<EMAIL_ADDRESS>similar for listStatus, remoteListStatusIterator.

:::: AGGREGATED TEST RESULT ::::
HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 9 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=119).
[ERROR] Errors:
[ERROR]   ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 792, Failures: 1, Errors: 1, Skipped: 197
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 49
HNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=29).
[ERROR] Errors:
[ERROR]   ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 792, Failures: 1, Errors: 1, Skipped: 197
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 49
NonHNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 792, Failures: 0, Errors: 0, Skipped: 291
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 45
NonHNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 792, Failures: 0, Errors: 0, Skipped: 291
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 45
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 9 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ITestAzureBlobFileSystemLease.testAcquireRetry:366 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 792, Failures: 0, Errors: 1, Skipped: 197
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 49
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit 7fad0940a85f033387d170078e2fa91456e944c7 (HEAD -> noUriChange, origin/noUriChange)
Author: Pranav Saxena<EMAIL_ADDRESS>Date:   Tue Aug 1 04:11:38 2023 -0700

    indentation in ITestBlobOperation

:::: AGGREGATED TEST RESULT ::::
HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ITestAzureBlobFileSystemLease.testAcquireRetry:368 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 792, Failures: 0, Errors: 1, Skipped: 197
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 49
HNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 792, Failures: 0, Errors: 1, Skipped: 197
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 49
NonHNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=309).
[INFO]
[ERROR] Tests run: 792, Failures: 1, Errors: 0, Skipped: 275
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 45
NonHNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=64).
[INFO]
[ERROR] Tests run: 792, Failures: 1, Errors: 0, Skipped: 275
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 45
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ITestAzureBlobFileSystemLease.testAcquireRetry:376->Object.clone:-2 » TestTimedOut
[INFO]
[ERROR] Tests run: 792, Failures: 0, Errors: 1, Skipped: 197
[INFO] Results:
[INFO]
[WARNING] Tests run: 284, Failures: 0, Errors: 0, Skipped: 49
Time taken: 46 mins 15 secs.
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit dc880b6fe0e2713ad2bccc6e98ac304d40ba9873 (HEAD -> noUriChange, origin/noUriChange)
Author: Pranav Saxena<EMAIL_ADDRESS>Date:   Thu Aug 3 23:09:37 2023 -0700
code refactor: move methods to one place


enhancementremove constraint that root node must maintain output order

Proposed changes
Before this PR, we must let ROOT plan of Nereids maintain its output order to ensure the result output's order is same with query.
In this PR, we record the output order after analyze and inject it into fragment after translate. So we could remove the constraint of ROOT plan.
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.21 seconds
load time: 481 seconds
storage size:<PHONE_NUMBER>0 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221116135227_clickbench_pr_47295.html

use another way to implement it


MIC-784: Enable Kirkham (KMI) in dev to test bug

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.46%. Comparing base (1c97bee) to head (ecf8fea).

Additional details and impacted files
@@            Coverage Diff            @@
##               main     #200   +/-   ##
=========================================
  Coverage     91.46%   91.46%
  Complexity      712      712
=========================================
  Files           131      131
  Lines          2132     2132
  Branches        155      155
=========================================
  Hits           1950     1950
  Misses          121      121
  Partials         61       61

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Delete stemcell.png

Uploaded image with new version(2.4.4)

Created request in master repo by mistake. Closing it.
Sorry for the inconvenience.


BestResponsePolicy requires concrete policy probability values, fails with JAX abstract tracer values

best_response.BestResponsePolicy (which is being called by my code via exploitability.nash_conv) does not work when policy probabilities are JAX abstract tracer values:
Traceback (most recent call last):
[...]
  File "/usr/local/lib/python3.10/site-packages/open_spiel/python/algorithms/best_response.py", line 181, in <genexpr>
    return sum(p * self.q_value(state, a)
jax._src.errors.ConcretizationTypeError: Abstract tracer value encountered where concrete value is expected: Traced<ShapedArray(bool[])>with<BatchTrace(level=1/1)> with
  val = Traced<ShapedArray(bool[11])>with<DynamicJaxprTrace(level=0/1)>
  batch_dim = 0
The problem arose with the `bool` function.
This Tracer was created on line /usr/local/lib/python3.10/site-packages/open_spiel/python/algorithms/best_response.py:182 (<genexpr>)

See https://jax.readthedocs.io/en/latest/errors.html#jax.errors.ConcretizationTypeError

This is due to the if p > self._cut_threshold clauses, which require p to be concrete. Is there a recommended workaround? If not, perhaps a threshold flag could be added to disable the threshold check completely.

Hmm, not sure I understand. I don't know what an abstract tracer value is... But don't all the probabilities need to be concrete for the best to compute a best response in the first place?
Does removing the threshold logic locally work for you?

Is it because there is a jitted function that calls BestResponsePolicy? If it is then I think it is normally not solvable because the execution flow of a jitted JAX function cannot be dependent on the concrete values of the input (e.g., if conditions. see this link).

@lanctot A tracer value is JAX's internal representation of intermediate values during transformations (jit, vmap, etc.). It keeps track of all operations (addition, multiplication, etc.) that created it from the inputs so as to compile the resulting computation, which can then be run quickly many times, used for autograd, etc. See here for more details.
I tried removing the threshold logic, but it hit another snag here in BestResponsePolicy.best_response_action's call to max, which uses Python conditionals internally. I think the algorithm's logic would have to be rewritten to be tracer-friendly (see below).
@rezunli96 Yes. Technically, JAX can handle control flow via these functions, but transforming plain Python control flow to that form requires care and is not always trivial.
I wrote my own implementation below (alongside OpenSpiel's, for comparison):

def get_best_response(root, policy, i):
    reach_prob = dict(get_cf_reach_prob(root, policy, i))
    states = get_infoset_states(root, i)

    value_cache = {}
    def get_value(state):
        state_str = state.serialize()
        if state_str not in value_cache:
            if state.is_terminal():
                value_cache[state_str] = state.returns()[i]
            elif state.is_chance_node():
                value_cache[state_str] = sum(
                    prob * get_value(state.child(action))
                    for action, prob in state.chance_outcomes()
                )
            else:
                if state.current_player() == i:
                    move = get_move(state.information_state_string())
                    # value_cache[state_str] = get_value(state.child(move))
                    value_cache[state_str] = sum(
                        (action == move) * get_value(state.child(action))
                        for action in state.legal_actions()
                    )
                else:
                    value_cache[state_str] = sum(
                        prob * get_value(state.child(action))
                        for action, prob in zip(state.legal_actions(), policy(state))
                    )
        return value_cache[state_str]

    move_cache = {}
    def get_move(infoset):
        if infoset not in move_cache:
            move_cache[infoset] = sum(
                jnp.stack([
                    get_value(state.child(action))
                    for action in state.legal_actions()
                ]) * reach_prob[state.serialize()]
                for state in states[infoset]
            ).argmax()
        return move_cache[infoset]

    get_value(root)
    return {
        'root_value': value_cache[root.serialize()],
        'value_cache': value_cache,
        'move_cache': move_cache,
    }

def get_best_response_spiel(root, policy, i):
    def action_probabilities(state):
        return dict(zip(state.legal_actions(), policy(state)))
    policy_obj = SimpleNamespace(action_probabilities=action_probabilities)
    br = BestResponsePolicy(root.get_game(), i, policy_obj)
    return {
        'root_value': br.value(root)
    }

Note the commented line, which I had to rewrite as a sum with an indicator value.

@lanctot Is this wontfix? Just wondering.

Hi @carlosgmartin, yeah sorry-- this code was not made with your use case in mind. We can updates the docs to make that more clear, but I am surprised that you expected it to work, so you would be the best person to add clarification to the docs?
I would be happy if you or anyone else volunteered to fix this but I don't see any easy solution that would keep this code clean.. do you? So mu advice is fork it, make it work for your use case, and maybe contribute it if you are so inclined?

@lanctot No problem. I wrote a self-contained module of JAX-compatible helper functions for OpenSpiel games.


License of the pwiz/utility/misc/pinned_gcroot.h file

Greetings,
currently packaging libpwiz for Debian, I would like to know what license governs the redistribution of the pwiz/utility/misc/pinned_gcroot.h file, apparently copyrighted to Microsoft. I need to list the various copyright holders of the files in the distribution for libpwiz to enter the Debian servers.
I wonder, however, if this file is needed at all in the GNU/Linux build environment ?
Thank you for your kind attention,
Sincerely
Filippo

Right, it's not needed for native Linux/OSX builds.

Great, thanks, I'll close the issue.
Cheers,
Filippo


fusion: AYON renaming menu

Changelog Description
Fusion is having Ayon menu.
Testing notes:

open Fusion in Ayon and see the difference

AYON menu shows up, but Openpype is still in header of the menu.


Modify Agent Data on the Host (Init/Step)

As of #178 Agent data can be transparently accessed on the host in Init, Step and Exit functions.
It may be useful in some cases to modify this data on the host, which would then be transparently transferred on the device at the end of an init or step function.
This would be similar to host based agent creation, but for modification of individuals.
i.e.
set_Agent_default_variable_time_alive(agentIndex, value);
set_Agent_default_variable_example_array(agentIndex, arrayIndex, value);
set_Agent_default_variable_example_array(agentIindex, arrayIndex, value);

Calling these functions would required that:

The copy of the data on the host is up to date
When set is called a flag is updated indicating that data needs transferring to the device

This could be a single flag for the array
or two unsigned integers: the first agent to update and the last agent to update.

Additionally, an optional function to explicitly causes the data movement could be provided - which would enable use within custom visualisation code (non init / step functions).

This will be implemented in FLAME GPU 2


Is it any way to inject bean to rest controller?

Got out of memory error as it is now

I assume you are talking about Spring. This should work normally. Java (and Spring) are pretty memory hungry, if you look at the Lambda logs it should tell you how much memory it consumed, what do you see there? Have you tried to increase the memory available to the Lambda function?

Did increasing the amount of memory available to the function help? Can I close this issue?

I'm closing this issue since we haven't heard back in 27 days


handle DataFailureException

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

5 of 7 (71.43%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.1%) to 60.047%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

catalyst_utils/models.py
5
7
71.43%

Totals

Change from base Build<PHONE_NUMBER>:
-0.1%

Covered Lines:
508

Relevant Lines:
846

💛  - Coveralls


Add openlab ha command

Add the openlab ha command support. This patch add the commands for
openlab ha node CRUD.

maybe it is better to split openlabcmd and deployment part, as these functionalities mostly need to run in the environment(this change need to access zk), but deployment don't not need. and there isn't close relationship between these funcionalities and deployment

Addressed all the comments, Please review again. Thanks.


feat: Revamp matching service

Change the logic of matching service:

Users get added into a matchmaking queue
At the same time, we add their username into topic Sets in Redis and store their matching request information in a HashSet in Redis
Peek at front of the queue --> retrieve user info from hash --> Loop through topics set (based on user's specified topics) one by one and try to find the first match --> return match (if any) together with the common topic and difficulty --> trigger a notifyMatch function to send result to both websockets --> attempt to close both websockets context

Added new type match_rejected when a user within an existing queue tries to matchmake again @chiaryan @solomonng2001 can take note of this for frontend. Refer to README.md for more details.
Current Situation:

There is a bug where after the matched user gets informed, the matchCtx fails to cancel even though the cancelFunc() is called, resulting in retrieving of match_found then subsequently timeout caused by the timeoutCtx.
@tituschewxj need your help to checkout my branch and apps/matching-service/handlers/websocket.go

After resolving the problem, we should be able to merge 👍

Newest Update on PR

Fix error with matchmaking service

Detailed Explanation of Matchmaking
I utilise 3 Redis data structure in the matchmaking process:

A Redis Queue (to store users who wants to match, basically a matchmaking queue)
Multiple Redis Sets (these sets are identified by topics, for example if user1 has topics: ["Arrays", "Trees"], user1's username will be added into the Arrays set and Trees set. This is to allow us to identify potential matches and also guarantees uniqueness (properties of a set)
A Redis Hashset (to store users matching request, for example the json request sent via client side for easy retrieval in the future)

I also attempted to utilise mutex to ensure that each time there is only one process editing/accessing the Redis Data Structures. Since there can be multiple connections, I created several global maps for matchContext, matchChannel, websocket connections.
General Flow

User client initiate websocket connection via a matchrequest -->
Checks if username already in the live websocket connections, if it exist reject the connection, else continue -->
User added into the matchmaking queue -->
Loop through the topics specified and add user into the respective Redis Sets -->
Store user MatchRequest in the Redis HashSet -->
Only one MatchMaking algorithm runs in the background -->
This repeats for all the users that initiate a connection

Matching Algorithm

Peek at the front of the queue
Retrieve user (at the front of the queue) info from Hashset
Loop through its topics and find for first matching in the Topic Sets
Find highest common difficulty else return min difficulty of the 2 matched users' difficulties array
Notify both user of the match and result
Cleanup, cancel relevant contexts and close connections


Practice Exam 11 Q32

Right Answer is C. For enviroment better issue will be Regions.

@PyFaNNy
The correct answer is B. Availability Zones.
Replicating infrastructure across multiple Availability Zones (AZs) within the same AWS Region is a recommended practice for achieving fault tolerance and business continuity. An Availability Zone is a distinct physical location within a Region that is engineered to be isolated from failures in other Availability Zones, with independent power, cooling, and networking.
Edge locations are a part of Amazon CloudFront's content delivery network (CDN) and are used to cache and deliver content to end-users with low latency. They are not directly related to fault tolerance and business continuity.
Regions are a geographic area that consists of multiple Availability Zones. Replicating infrastructure across multiple regions can provide additional redundancy and protect against region-wide disruptions, but it may also increase costs and complexity.
Amazon Route 53 is a DNS web service that can be used for routing traffic to various AWS resources, but it is not directly related to achieving fault tolerance and business continuity.


fix: remove default HelpPopup ofset

Add css rule left: 4 by default look like not so good solution.
What if i want render this Popover exactly in place? What if i want add padding with padding or margin rules?
And another important point is the evidence of behavior. How i should to know that this component is shifted on 4 pixels?
Summing up the above i suggest remove this default offset at all and add the ability for users to manage the margins themselves

Looks like a breaking change

Oh, I guess it would be breaking change:) why do you keep 4px in scss file?


Is halite a dependency or a dev_dependency?

I'm currently working in a pretty locked down organization, so the fewer gems I need to ask to be allowed the simpler it is to get approved.
I noticed that poise only has one dependency, halite, and then halite has several more. However, it looks like halite is just used for developing and releasing gems (including poise) as a cookbook and isn't actually part of implementing the poise functionality itself. Could halite be changed from a dependency to a developement_dependency (like poise-boiler)?

No, having halite as a runtime dependency is what marks a cookbook as a "halite-style cookbook gem". That said, you don't need it your environment, it's all baked to actual Chef cookbooks before upload to Supermarket so when you consume it from there you don't need to know anything about my weirdo development style. You would not generally consume Poise and my other libs as actual gems.