refactor: Look at the big picture and reprogram the catalog

@intyouss Hi, is necessary to create Dockerfile and Makefile in every service directory?

@intyouss Hi, is necessary to create Dockerfile and Makefile in every service directory?

Sorry, the app folder has not been rectified and needs to be cleaned🙂


integration: don't ignore exit codes of test processes

Signed-off-by: Andrey Vagin<EMAIL_ADDRESS>

Whoa, weird fail.

LGTM


Delete s3 cache upon DELETE request

This is useful in case you want to clear a bunch of pages, but recomputing all at once would overload your server.

Thanks for this! Sorry I'm just now getting to it.
I don't think the delete function should be a part of the server. You should have a separate script or API to delete pages in this way. Also, someone could delete your cached pages which would be weird. It seems like this should live somewhere else though.
Thanks for the PR though! I appreciate it.

Weird... you expose this server to the public? I have mine behind a proxy. Seems like if you're exposing directly to the public you would have more issues than just delete. (Set is even worse!)

Well, we host it as a service. We require the use of a token for any S3 actions on our public server so we don't have any issues. Even with your server being behind a proxy, I still don't think delete should be a part of the server there. You should just use the S3 api (or aws-sdk modules) to delete the URL you need.


Arel errors in latest commit

I'm working off of chewy master. Letting you know that commit https://github.com/toptal/chewy/commit/6b02cef1493ea25bc545b80462c30b38f5ed5715
breaks at least the rake tasks. It works in the previous commit: https://github.com/toptal/chewy/commit/02422f0188a4d27daa805723ddcb7057681c6726
Running rails 4.2.0.
Exerpt from rake chewy:reset in https://github.com/toptal/chewy/commit/6b02cef1493ea25bc545b80462c30b38f5ed5715
** Execute chewy:reset
Resetting SchoolsIndex
  Imported SchoolsIndex::School for 0.01s, documents total: 0
rake aborted!
ActiveRecord::StatementInvalid: PG::SyntaxError: ERROR:  syntax error at or near "::"
LINE 1: SELECT  #<struct Arel::Attributes::Attribute relation=#<Arel...
                             ^
: SELECT  #<struct Arel::Attributes::Attribute relation=#<Arel::Table:0x007fee98bb52b8 @name="schools", @engine=School(id: integer, name: string, web_url: string, created_at: datetime, updated_at: datetime, trust_level: integer), @columns=nil, @aliases=[], @table_alias=nil, @primary_key=nil>, name="id"> FROM "schools"  ORDER BY "schools"."id" ASC LIMIT 1000

Is was force-pushed, so you can check it out again, currently it should be just fine

Thanks for the report, man!

Excellent. Master works for me now. Thanks for this great gem!


CMake not working

On MacBigSur 11.7.10
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
GENERICCODEGEN
    linked by target "mgl" in directory /Users/quentin/MGL
    linked by target "test_mgl" in directory /Users/quentin/MGL/test_mgl
GLSLANG
    linked by target "mgl" in directory /Users/quentin/MGL
    linked by target "test_mgl" in directory /Users/quentin/MGL/test_mgl
GLSLANGDEFAULT
    linked by target "mgl" in directory /Users/quentin/MGL
    linked by target "test_mgl" in directory /Users/quentin/MGL/test_mgl
MACHINEINDEPENDENT
    linked by target "mgl" in directory /Users/quentin/MGL
    linked by target "test_mgl" in directory /Users/quentin/MGL/test_mgl
OGLCOMPILER
    linked by target "mgl" in directory /Users/quentin/MGL
    linked by target "test_mgl" in directory /Users/quentin/MGL/test_mgl
OSDEPENDENT
    linked by target "mgl" in directory /Users/quentin/MGL
    linked by target "test_mgl" in directory /Users/quentin/MGL/test_mgl
SPIRVLIB
    linked by target "mgl" in directory /Users/quentin/MGL
    linked by target "test_mgl" in directory /Users/quentin/MGL/test_mgl
SPVREMAPPER
    linked by target "mgl" in directory /Users/quentin/MGL
    linked by target "test_mgl" in directory /Users/quentin/MGL/test_mgl

-- Generating done (0.1s)
CMake Generate step failed.  Build files cannot be regenerated correctly.

I got it semi-working using CMake see #78: https://github.com/julcst/mgl


Cook API attempts to log Kerberos user even if configured auth isn't Kerberos

This causes 500 errors on API requests when not using Kerberos.
This was introduced here:
bb2b63228566400c423925e4ac30f838ebddf052
Should leave the same access logging behavior in,  just do it only when it's actually configured to use Kerberos.

Ring_ Stacktrace.pdf
Attached the error message when this occurs. It appears that this stackoverflow discusses the issue: http://stackoverflow.com/questions/15029148/tomcat-fails-to-authenticate-using-spnego-cannot-locate-default-realm
Perhaps the best thing to do is change to use a different auth method since it is only used as a shim  to get a user associated with the request.

This is no longer an issue following the merging of PR #125


Test 10.0.0

Is migrating to this version easier than v11?
⚛👋 Hello there! Welcome. Please follow the steps below to tell us about your contribution.

Copy the correct template for your contribution

🐛 Are you fixing a bug? Copy the template from https://bit.ly/atom-bugfix-pr
📈 Are you improving performance? Copy the template from https://bit.ly/atom-perf-pr
📝 Are you updating documentation? Copy the template from https://bit.ly/atom-docs-pr
💻 Are you changing functionality? Copy the template from https://bit.ly/atom-behavior-pr

Replace this text with the contents of the template
Fill in all sections of the template
Click "Create pull request"

@icecream17 I don't think it is. I have updated the description of Electron 11.4.7 upgrade PR to include blockers. I am closing this to encourage us to consolidate all the efforts in getting electron 11.4.7 working.


Apache Conf do not work for Apache 2.4.18

Hello,
I followed the configuration instructions for Apache and HTTPS. Unfortunately the websockets cannot be connected. Anybody have any idea what might be causing this?
Apache v. 2.4.18
Settings
ProxyPreserveHost On
ProxyRequests Off
LogLevel debug

<Proxy *>
	Require all granted
</Proxy>

RewriteEngine On
RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* wss://<IP_ADDRESS>:8065%{REQUEST_URI} [P,QSA,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://<IP_ADDRESS>:8065%{REQUEST_URI} [P,QSA,L]

<LocationMatch "^/api/v(?<apiversion>[0-9]+)/(?<apiusers>users/)?websocket">
	Require all granted
	ProxyPass wss://<IP_ADDRESS>:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
	ProxyPassReverse wss://<IP_ADDRESS>:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
	ProxyPassReverseCookieDomain <IP_ADDRESS> my-domain.com
</LocationMatch>

<Location />
	Require all granted
	ProxyPass http://<IP_ADDRESS>:8065/
	ProxyPassReverse http://<IP_ADDRESS>:8065/
	ProxyPassReverseCookieDomain <IP_ADDRESS> my-domain.com
</Location>

Loaded Apache Modules
core_module (static)
so_module (static)
watchdog_module (static)
http_module (static)
log_config_module (static)
logio_module (static)
version_module (static)
unixd_module (static)
access_compat_module (shared)
aclr_module (shared)
actions_module (shared)
alias_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_core_module (shared)
authn_file_module (shared)
authz_core_module (shared)
authz_host_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cgi_module (shared)
data_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
dbd_module (shared)
deflate_module (shared)
dir_module (shared)
env_module (shared)
expires_module (shared)
filter_module (shared)
headers_module (shared)
heartmonitor_module (shared)
include_module (shared)
ldap_module (shared)
mime_module (shared)
mpm_event_module (shared)
negotiation_module (shared)
passenger_module (shared)
perl_module (shared)
proxy_module (shared)
proxy_express_module (shared)
proxy_fcgi_module (shared)
proxy_http_module (shared)
proxy_wstunnel_module (shared)
python_module (shared)
remoteip_module (shared)
rewrite_module (shared)
security2_module (shared)
session_module (shared)
setenvif_module (shared)
slotmem_shm_module (shared)
socache_shmcb_module (shared)
ssl_module (shared)
status_module (shared)
suexec_module (shared)
unique_id_module (shared)
userdir_module (shared)
Error Messages
GET /api/v4/websocket HTTP/1.0	- SSL/TLS-Access for Apache
[ssl:debug] ssl_engine_kernel.c(354): AH02034: Initial (No.1) HTTPS request received for child 87 (server my-domain.com:443) - Apache-Error
[authz_core:debug] mod_authz_core.c(809): AH01626: authorization result of Require all granted: granted - Apache-Error
[authz_core:debug] mod_authz_core.c(809): AH01626: authorization result of : granted - Apache-Error
[proxy:debug] mod_proxy.c(1160): AH01143: Running scheme wss handler (attempt 0) - Apache-Error
[proxy_fcgi:debug] mod_proxy_fcgi.c(879): AH01076: url: wss://<IP_ADDRESS>:8065/api/v4/websocket proxyname: (null) proxyport: 0 - Apache-Error
[proxy_fcgi:debug] mod_proxy_fcgi.c(882): AH01077: declining URL wss://<IP_ADDRESS>:8065/api/v4/websocket - Apache-Error
[proxy_http:debug] mod_proxy_http.c(1943): AH01113: HTTP: declining URL wss://<IP_ADDRESS>:8065/api/v4/websocket - Apache-Error
[proxy_wstunnel:debug] mod_proxy_wstunnel.c(341): AH02900: declining URL wss://<IP_ADDRESS>:8065/api/v4/websocket (not WebSocket) - Apache-Error
AH01144: No protocol handler was valid for the URL /api/v4/websocket. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule. - Apache-Error
[authz_core:debug] mod_authz_core.c(809): AH01626: authorization result of Require all granted: granted - Apache-Error
[authz_core:debug] mod_authz_core.c(809): AH01626: authorization result of : granted - Apache-Error
[proxy:debug] mod_proxy.c(1160): AH01143: Running scheme http handler (attempt 0) - Apache-Error
[proxy_fcgi:debug] mod_proxy_fcgi.c(879): AH01076: url: http://<IP_ADDRESS>:8065/error_docs/internal_server_error.html proxyname: (null) proxyport: 0 - Apache-Error
[proxy_fcgi:debug] mod_proxy_fcgi.c(882): AH01077: declining URL http://<IP_ADDRESS>:8065/error_docs/internal_server_error.html - Apache-Error
[proxy:debug] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (<IP_ADDRESS>) - Apache-Error
[proxy:debug] proxy_util.c(2213): AH00944: connecting http://<IP_ADDRESS>:8065/error_docs/internal_server_error.html to <IP_ADDRESS>:8065 - Apache-Error
[proxy:debug] proxy_util.c(2422): AH00947: connected /error_docs/internal_server_error.html to <IP_ADDRESS>:8065 - Apache-Error
[proxy:debug] proxy_util.c(2175): AH00943: : has released connection for (<IP_ADDRESS>) - Apache-Error
[deflate:debug] mod_deflate.c(853): AH01384: Zlib: Compressed 3169 to 1366 : URL /error_docs/internal_server_error.html - Apache-Error

Hi @indikatordesign! Apologies for late reply. This is the Docs repo, so we don't follow the Issues here as regularly for troubleshooting issues - would you like to reopen this in the Server repo or the Forums so that we can help troubleshoot this? Thank you!


two-line title on iphone SE

there is a problem with the two-line title (and a high font, example 37) on iphone SE: the second line does not appear

Hey @furiosFast
Thanks for your bug report.
Have you tried to reduce the font size for smaller screen sizes like the iPhone SE?
Maybe you calculate the font size based on the screen scale:
https://developer.apple.com/documentation/uikit/uiscreen/1617836-scale
Currently the TitleView height is set to 20% of the screen height and does not adjust its content based on the Label text to avoid reducing the yOrigin of the ItemsView.


MLDB-2043 tabular big int

Fixes problems with Bit_Extractor<uint64_t> causing data corruption in tabular datasets.  Drags along part of a refactoring of the frozen columns.

(Test failure is spurious)


Feature/Setup knowledge graph component

Pretty rough and early state, but is viewable in the /testGraph page with hardcoded data.

I'll close the PR for now and open again when ready.


librealsense: 2.55.1 -> 2.56.1

Diff: https://github.com/IntelRealSense/librealsense/compare/v2.55.1...v2.56.1

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 341406 run on x86_64-linux 1

17 packages built:

envision
librealsense (librealsenseWithoutCuda)
librealsense-gui
librealsense-gui.dev
librealsense.dev (librealsenseWithoutCuda.dev)
librealsenseWithCuda
librealsenseWithCuda.dev
monado
opencomposite-helper
python311Packages.pyrealsense2 (python311Packages.pyrealsense2WithoutCuda)
python311Packages.pyrealsense2.dev (python311Packages.pyrealsense2WithoutCuda.dev)
python311Packages.pyrealsense2WithCuda
python311Packages.pyrealsense2WithCuda.dev
python312Packages.pyrealsense2 (python312Packages.pyrealsense2WithoutCuda)
python312Packages.pyrealsense2.dev (python312Packages.pyrealsense2WithoutCuda.dev)
python312Packages.pyrealsense2WithCuda
python312Packages.pyrealsense2WithCuda.dev


14102: disable dedupe detection parameter for the UP

This PR disables the dedupe detection code for the UP as it's broken and can cause no items to get routed
Test Steps:

Send a HL7 message with the allowDuplicate header set and verify that the report gets routed

Changes

Disables the duplicate detection in the Universal pipeline receiver

Checklist
Testing

[x] Tested locally?
[ ] Ran ./prime test or ./gradlew testSmoke against local Docker ReportStream container?
[ ] (For Changes to /frontend-react/...) Ran npm run lint:write?
[ ] Added tests?

Process

[ ] Are there licensing issues with any new dependencies introduced?
[ ] Includes a summary of what a code reviewer should test/verify?
[ ] Updated the release notes?
[ ] Database changes are submitted as a separate PR?
[ ] DevOps team has been notified if PR requires ops support?

Linked Issues

#14102

To Be Done
Create GitHub issues to track the work remaining, if any

#14103

Specific Security-related subjects a reviewer should pay specific attention to

@JFisk42 Yep, you are correct those are not part of the UP


[Feature] Output sshd_config parameters

Very neat tool, thanks for making it!
I think it would be neat to have a flag which would output 'good' sshd_config config line. Something that crossed my mind while I was testing my ssh servers.
Thanks,
G

Yeah, something like https://mozilla.github.io/server-side-tls/ssl-config-generator (cli only, of course) would be very handy

+1

This is a semi hard problem.
It pertains to various vendors who make their own SSH server and their constant renaming of various aspect of SSH KeX, ciphers, MAC, and cipher suite that are being sent/received over the SSH control channel plane.
I should know, I audit some of the vendors.
And this tool isn't the place to start versioning of SSH control channel across several SSH vendors, just to get the proper setting.


[Android] Correct logging

Summary
Fixes incorrect logging that I found whilst debugging
Changelog
[Android] [FIXED] - Fixed incorrect logging of isCatalystInstanceAlive in exception handler
Test Plan
N/A as change is related to logging

Can you merge it for me please @robhogan, I do not have access rights

Hi @jonnycaley - I’ve imported it, it’ll be merged once someone else at Meta stamps it, probably next week.


Add function parseHomepagePath with more logic

It checks if the string starts with @ and if does, if removes the @ and goes along without parsing the url.
Just a POC for the fix I suggested at https://github.com/facebookincubator/create-react-app/issues/1279

Is it related to https://github.com/facebookincubator/create-react-app/pull/937?

It is. I'm sorry but I didn't check the other PRs before submiting mine.
What I proposed might break legitimate use cases for the homepage package field, and I guess this will be a problem, but for simple use cases it's simple and does the job.

Thank you for your contribution, but this was superseded by #937 and #1504.
I look forward to more contributions from you in the future!


Fix upload test-helper

This should fix #144 and #70
Inspired by #89

:heart: I’ll get this in and released soon!


Support GitHub caution and important via custom admonition

Thanks for this great extension!
I was wondering if it would be possible to support GitHub caution and important via custom admonitions.
I did a quick test and was able to get something similar working using custom admonitions with the https://oprypin.github.io/markdown-callouts/ extension and css from https://github.com/squidfunk/mkdocs-material/issues/6667#issuecomment-2165055127
Just was curious if it would be possible to have these admonitions provided by this extension.

I don't think that the mkdocs plugin should be adding CSS. However, maybe it can be psuedo supported by:

Allowing users to configure the plugin with a custom mapping that specifies how to convert admonition names from GH to mkdocs.
Example code for adding the CSS, defining the custom admonitions. and configuring the plugin.

I don't think that the mkdocs plugin should be adding CSS

Makes sense. I was originally thinking that it might be nice to just have this work "out of the box" without having to provide any custom CSS. That said, this is probably something that people will probably want to customize (different colors, icons, etc), especially if they aren't using the Material theme.
When I originally filed this issue, I was just looking for a way to customize the appearance of the Caution and Important alerts. However, I was actually able to get this working with the markdown-gfm-admonition extension and some custom CSS (see https://github.com/squidfunk/mkdocs-material/issues/6667#issuecomment-2308612453), so if you want to close this out, I'm totally happy to just use that solution instead.
That said, if you still want to implement this, I think any solution that allows using custom CSS (as described in https://squidfunk.github.io/mkdocs-material/reference/admonitions/?h=admoni#custom-admonitions) would solve my original request.

I'll leave it open for now and re-evaluate what to do when I would actually have time to work on it. Thanks again for the suggestion :)


MM-21153 Count mentions in DM channels using mention_count

Submitted for release-1.26
See https://github.com/mattermost/mattermost-mobile/pull/3700
mattermost-redux - https://github.com/mattermost/mattermost-redux/commit/3fac9150fac5f8157817215173db827334a94c9b

Thanks @saturninoabril


Move /webhook to its own service

In order to investigate https://github.com/huggingface/dataset-viewer/issues/2395, we can separate /webhook from the rest of the endpoints served by the API service. Indeed, it's by far the slowest one to respond (15% of the requests take > 700ms, 5% > 7s !) compared to the other ones (the worst one, /info, has 15% > 60ms, 5% > 600ms).
Moving /webhook to its own service is similar to what we did for /search or /rows. Note that we might want to have its own set of nodes, in which case we need to add a node groups here: https://github.com/huggingface/infra/blob/main/projects/hub/00-aws/variables/hub-aws-prod-us-east-1.tfvars

Implemented by https://github.com/huggingface/dataset-viewer/pull/2849


fix: bump default build task to 6gb

Despite all the performance WG testing in PR #551, once the switch to memory for the build task /var/lib/containers landed; various builds started to fail because of memory issues.  Before we revert, we are bumping the default some more.
@mmorhun @arewm PTAL
if this does not suffice, we'll revert the change

PaC is ailing and best as I can tell our e2e's never got start in RH prod ... I'm going to try and close / reopen these to see if I can get CI going

e2e's grep @arewm @mmorhun
do either of you have merge button authority, or should I do it?


generic-armel-iproc: override GO linuxloader

GO on ARM links against the wrong linuxloader, so manually override it until the issue is fixed upstream.


ASTMiner COBOL output

I have added ANTLR grammer of COBOL to ASTMiner and implemented the wrapper. When I run the ASTMiner, it generates the below files:
node_types.csv
path_contexts.csv
paths.csv
tokens.csv
I know this ASTMiner output is compatible with code2vec. I need some help/pointers on how to use this output in code2vec to get predictions.

I think you need to convert the path_contexts.csv to the format like tuples(source,path,target,mask) in order to be used in the code2vec.
You can refer to preprocess.sh which I think will be helpful:)

Please, check out the "code2vec" option of the CLI, it may better suit your needs.

I did look at preprocess.sh, it is actually preprocess new dataset, extracting paths from the source code, etc., which is already done in our case, we are using trained model, preprocessed data already provided by code2vec and paths are already extracted by ASTMiner.
Now the question is how to used ASTMiner output directly in code2vec.


function equivalent of "do" notation

Let's consider a problem which a hypothetical do function might help us to solve:

Given an object, return the square root of the number represented by the string at path a.b.c. Do not assume, however, that the path a.b.c will exist.

Here's the obvious naive solution:
// naive :: Object -> Number
const naive = o => Math.sqrt(parseFloat(o.a.b.c));

This won't handle all the following inputs:

{a: {b: {c: '2.25'}}}
{a: {b: {c: 'blah'}}}
{a: {b: {}}}
{a: {}}
{}

This is how I would solve this problem today:
// safe1 :: Object -> Maybe Number
const safe1 = R.pipe(
  S.get('a'),
  R.chain(S.get('b')),
  R.chain(S.get('c')),
  R.chain(S.parseFloat),
  R.map(Math.sqrt)
);

With two small changes we can make this more uniform:
// safe2 :: Object -> Maybe Number
const safe2 = R.pipe(
  S.Just,
  R.chain(S.get('a')),
  R.chain(S.get('b')),
  R.chain(S.get('c')),
  R.chain(S.parseFloat),
  R.chain(R.compose(S.Just, Math.sqrt))
);

If we then changed the type from Object -> Maybe Number to Maybe Object -> Maybe Number we could remove the first step of the pipeline:
// safe3 :: Maybe Object -> Maybe Number
const safe3 = R.pipe(
  R.chain(S.get('a')),
  R.chain(S.get('b')),
  R.chain(S.get('c')),
  R.chain(S.parseFloat),
  R.chain(R.compose(S.Just, Math.sqrt))
);

Now we have a form which can surely be abstracted! Given a list of functions we want to apply R.chain to each, then apply R.pipe to the list as positional arguments:
// $do :: Monad m => [(a -> m b), (b -> m c), ..., (y -> m z)] -> m a -> m z
const $do = fs => R.apply(R.pipe, R.map(R.chain, fs));

This can be written point-free:
// $do :: Monad m => [(a -> m b), (b -> m c), ..., (y -> m z)] -> m a -> m z
const $do = R.pipe(R.map(R.chain), R.apply(R.pipe));

Let's rewrite safe3 in terms of $do:
// safe4 :: Maybe Object -> Maybe Number
const safe4 = $do([
  S.get('a'),
  S.get('b'),
  S.get('c'),
  S.parseFloat,
  R.compose(S.Just, Math.sqrt),
]);

$do works for any Monad, and should reduce demand for "composed-chain" functions such as S.gets.
What do you think? Shall we add this? If so, what should it be named? S.do is problematic since do is not a valid identifier. S.$do is pretty ugly. Does do go by other names?

wow, that is nice sugar. FWIW, I think $do is fine

wow, that is nice sugar.

I think so too! Shall I open this for discussion on the Ramda issue tracker? The function could be defined there instead. :)

your call. looks pretty interesting to me.

It also seems like there is a way to write this with `ap`, but don't have time to prove that right now

The types don't line up in your definition, @buzzdecafe. You're fully applying (ahem) R.apply (which takes two arguments).

Why isn't do a valid identifier?

do is a keyword: do...while.

yep, i was reading apply like compose for some reason

Oh, duh. :goat:
The only limitation I see here is that this form of do notation only really allows you to feed through a single monadic value. You lose the ability to bind different variables to different monadic values in the same do expression like the following example:
foo :: Maybe String
foo = do
    x <- Just 3
    y <- Just "!"
    Just (show x ++ y)
``

The only limitation I see here is that this form of do notation only really allows you to feed through a single monadic value.

True. R.compose and R.pipe share this limitation, of course.

R.compose and R.pipe share this limitation, of course.

I don't really see how that relates. I think you might be able to come up with a way to bind different variables but I'm guessing it would be pretty hacky.
I think this is cool for the single pipeline case though!

I think you might be able to come up with a way to bind different variables but I'm guessing it would be pretty hacky.

indeed. I was trying to come up with some way to do that, but all i could come up with could only generously be called "hacky"

Does do go by other names?

The name you're looking for is Kleisli: https://en.wikipedia.org/wiki/Kleisli_category
What you've Greenspun'd is exactly Kleisli composition. :)
See also: >=> and <=<

The name you're looking for is Kleisli

It sounds as though Ramda needs more composition functions. R.composeK and R.pipeK, anyone?

:scream:

Sorry, I didn't actually look close enough. $do isn't entirely kleisli composition.
Also, note that you don't actually need the full power of Monad for this.

It sounds as though Ramda needs more composition functions. R.composeK and R.pipeK, anyone?

No, it definitely needs less (ideally one). They're all specific cases of a more general Semigroupoid idea.

They're all specific cases of a more general Semigroupoid idea.

I recall you and @scott-christopher discussing that in the gitter room. i may take a whack at unifying our many composers into one.
Which raises the thought: A good name for a javascript composition lib would be Bach.js

@joneshf, do you think such a function should return a -> m z rather than m a -> m z?
I'm having a bit of trouble seeing how something of the shape
R.pipe(
  S.Just,
  R.chain(f),
  R.chain(g),
  R.chain(h),
  R.chain(i),
  R.chain(j)
);

can be rewritten as a pipeline where every function in the pipeline is a partially applied R.chain.

@joneshf, do you think such a function should return a -> m z rather than m a -> m z?

I was just nitpicking, and looking more closely at the haskell stuff.
They're equivalent, so either way is fine, since (f >=> g) x = f x >>= g and x >>= f = (id >=> f) x. So choose either one and you can provide the other easily.

I'm having a bit of trouble seeing how something of the shape
...
can be rewritten as a pipeline where every function in the pipeline is a partially applied R.chain.

Hmm, I think I'm having a brain fart here, or I'm not understanding the question.

They're equivalent, so […] choose either one and you can provide the other easily.

That makes sense. :)

I think I'm having a brain fart here, or I'm not understanding the question.

I didn't see at the time how to return a -> m z rather than m a -> m z. I do now. The implementation is a bit more involved, since one must apply R.chain to all the arguments but the first:
// $do :: Chain m => [(a -> m b), (b -> m c), ..., (y -> m z)] -> a -> m z
const $do = R.pipe(R.converge(R.prepend,
                              R.head,
                              R.pipe(R.tail, R.map(R.chain))),
                   R.apply(R.pipe));

[Ramda's various composition functions and the proposed Kleisli composition functions are] all specific cases of a more general Semigroupoid idea.

I'm intrigued by the possibility of a composition function which can perform Kleisli composition as well as "regular" function composition. I've no idea how this is possible. :)

Thinking about it now, I think the only way to make it happen is to wrap the monadic function in a Kleisli type.

Also, if it wasn't clear from the signature Kleisli == ReaderT, and a -> b == ReaderT Identity a b. So that's why I feel these can be unified. But I think without a compiler implicitly passing around things and wrapping/unwrapping, it'd end up being a less than optimal api.

But I think without a compiler implicitly passing around things and wrapping/unwrapping, it'd end up being a less than optimal api.

That's often where the rubber meets the road with JS libs.  There a lot of 'almost's and '_if only+'s floating around, usually having to do with its type system.

Thinking about it more, you'd run into the same api issues in haskell. And the ways around it are just as suboptimal, but for different reasons. I guess I just want too much.

ramda/ramda#1212 added a variadic version of the proposed $do function as R.pipeK.

@davidchambers, any interest in revisiting this now that Sanctuary is no longer dependent on Ramda?

Absolutely! Let's open a new issue: pipeK is more limited than Haskell's do notation because it doesn't provide access to the results of all previous computations. Would you mind opening an issue for pipeK, @gabejohnson?

I can do that.
I've also been looking at arrows and categories. Seems like something that would be useful to add to the spec.

Looks like there's already an issue for it https://github.com/fantasyland/fantasy-land/issues/208

See #380


easytier : 2.0.0 -> 2.0.3

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Please squash commits to a single commit here to make tree clean. See https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#commit-conventions

Reopen at https://github.com/NixOS/nixpkgs/pull/348268


Vector data

We will need to support vector tiled data for #3152 anyway, but there are other vector sources (e.g. QA layers) that would be great to bring into iD.

We will need to support vector tiled data for #3152 anyway, but there are other vector sources (e.g. QA layers) that would be great to bring into iD.

There's lots of types of vector layers, what are we looking at supporting?

There's lots of types of vector layers, what are we looking at supporting?

Not sure yet.  It might be worth having something like (or built into) editor-layer-index but for vector layers that are useful for editing.  This ticket is just to kick around ideas, and also to track that we need something like vector-tile-js.

#4164 provides a starting point for bringing in vector QA layers from ArcGIS-based services, if that is of interest. Not vector tiles, but along the same lines and I think the implementation (in the Map Data expando-tab) could be similar. #3452 also mentions this and might be one way that #4164 could be helpful to start (read-only QA layers from ArcGIS sources).

CC @kriscarle for visibility - I proposed this as a GSoC project, and would be great to get this support into iD sometime soon.

Just cross linking: https://www.openstreetmap.org/user/Princi Vershwal/diary/43802

The first step towards supporting vector tiled data is that a user can add a URL and have the vector tiles show up on the screen. It will look something like this :

For this, I have added another layer 'mvt'. Currently, it will support .pbf(Protocolbuffer Binary Format) and .mvt(Mapbox Vector Tiles) format. These files contain data in Protocolbuffer binary format which is converted to geojson using mapbox's vector-tile-js and this data is passed directly to the D3 draw functions to show up the tiles on the screen.
All the code related to this will be here.

For some details: https://www.openstreetmap.org/user/Princi Vershwal/diary/44151

This happened 🎉

I have tried the first step towards supporting vector tiled data is that a user can add a URL and have the vector tiles show up on the screen. It worked and it displayed as :

but the problem am facing is it is not loading on specific lat and long where it has to be. can you help me to fix this problem

but the problem am facing is it is not loading on specific lat and long where it has to be.

@sahithiduddukuri Hmm I'm guessing that your data does not match the expected projection. Make sure it's in WGS84.

@quincylvania I tried working with mvt/pbf drag and drop using github code(https://github.com/openstreetmap/iD/issues/3742#issuecomment-396352744). Follwed the same and added few UI features to the code. Can you tell me will mvt file contains coordinates(lat,long) which specifies/instructs it to display on specific geo location. When i grag and drop an mvt file it is displaying in a specific location as shown in below picture [9.72/39.7075/-74.0209].
In mvt.js how can i change [ var feature = layer.feature(i).toGeoJSON(150, 194, 4);] to my specific location


Storing the user's nametag in the DB

Implemented storing of the user's nametag in the DB.

Added some more comments.

Hi @BrennanJones, can you run npm run build and fix all the build errors?

Hi @BrennanJones, can you run npm run build and fix all the build errors?

Done


mostly fix #72 (still need EventListener)

introduce sortLUrev(), which sorts the bars in reverse in fig 3.
@jread-usgs and @ldecicco-USGS - i think this solves issue #72.
@jiwalker-usgs and @mwernimont - could one of you please work some ScrollMagic to trigger sortLUrev() in the right place?

👍


Issue with 5.2.2: "SQLite upgrade required: SQLite 3.0 is outdated and no longer supported by Flyway. Flyway currently supports SQLite 3.7.2 and newer."

Which version and edition of Flyway are you using?
I am trying to migrate from 5.2.1 to 5.2.2 or 5.2.3
If this is not the latest version, can you reproduce the issue with the latest one as well?
(Many bugs are fixed in newer releases and upgrading will often resolve the issue)
Yes with 5.2.3
Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin)
Gradle plugin
Which database are you using (type & version)?
Sqlite 3.16.1
Which operating system are you using?
Windows 7
What did you do?
(Please include the content causing the issue, any relevant configuration settings, the SQL statement that failed (if relevant) and the command you ran.)
I changed the dependency in my build.gradle from 5.2.1 to 5.2.2, then I relaunched my project.
I already have a dependency to org.xerial:sqlite-jdbc:3.16.1 that I left untouched
What did you expect to see?
I expect Flyway to continue accepting my version of Sqlite as it's a minor release
What did you see instead?
I got the following error: "SQLite upgrade required: SQLite 3.0 is outdated and no longer supported by Flyway. Flyway currently supports SQLite 3.7.2 and newer."

Thanks for reporting. Definitely a regression. Will be fixed in the next point release.


Mark pwm module as unproven

Similar to #145 the Pwm types are unproven in embedded-hal, mark the modules as unproven so they aren't compiled in without the feature.

Looks like I just beat you to this in the other PR. Already merged.

For the Pwm types? They’re still unconditional in master.

Oh yeah nm, thought this was the watchdogs


Fix du on macOS

Split out from https://github.com/conda-incubator/conda-store/pull/559.

Nevermind, I see why it was necessary there. There's a new test that runs du across Linux, macOS, and Windows. Without this, macOS runner fails.


Inserting a block above the first one

We have buttons to insert new blocks below existing blocks. But none to insert a block above the first block.
Suggestion: Show buttons below the scene title when the mouse hovers over the space.

Fixed 🥳


Failure of fetching the location by clicking Google Maps

In the footer , the Google Map option is provided but on clicking that link it is not opening the actual location on Google Maps, which should be the bad user experience.
Screencast from 2024-10-10 18-56-41.webm
I can fix this issue , please assign this issue to me, Thanks!

@Jils31 solve it then


Adding CI steps

Current dependencies on/for this PR:

main

PR #2   👈

This comment was auto-generated by Graphite.


[CP Staging] Fix the NewWorkspace button when there is no workspace yet

Details

There was a typo in the param of the components
Fixed Issues

$ https://github.com/Expensify/App/issues/35820
PROPOSAL:
Tests

Sign up to an account which does not have any workspace yet
Click the wrench to go to Settings
Go to workspaces
Click the New Workspace button in the big section block
Verify the workspace got created

[ ] Verify that no errors appear in the JS console

Offline tests

Same as tests
QA Steps

Same as tests

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos
Simple logic change, including the web, desktop and ios mweb tests

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

https://github.com/Expensify/App/assets/36083550/7f496747-080d-415b-81e8-b442b81fc74c

MacOS: Chrome / Safari

https://github.com/Expensify/App/assets/36083550/835fdb2a-ea1c-40f0-a6d4-2500a256e4e9

MacOS: Desktop

https://github.com/Expensify/App/assets/36083550/8ba50803-5f95-4da9-a79d-c80e9b1d60ce

Closing in favour of https://github.com/Expensify/App/pull/35622/files


[WIP] Upgrade to Spark 3.3.0

Why is this PR needed?
What changes were proposed in this PR?
Does this PR introduce any user interface change?

No
Yes. (please explain the change and update document)

Is any new testcase added?

No
Yes

Build Failed  with Spark 3.1, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbon_PR_Builder_3.1/751/

Build Failed  with Spark 3.1, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbon_PR_Builder_3.3/1/

Build Failed  with Spark 3.1, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbon_PR_Builder_3.1/752/

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbon_PR_Builder_2.4.5/4621/

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbonPRBuilder2.3/6376/

Build Failed  with Spark 3.1, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbon_PR_Builder_3.1/762/

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbonPRBuilder2.3/6378/

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbon_PR_Builder_2.4.5/4633/

Build Failed  with Spark 3.1, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbon_PR_Builder_3.1/764/

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12602/job/ApacheCarbon_PR_Builder_2.4.5/4662/


How to use the transform of MultiBSplineTransformWithNormal

Hi, could anyone help me about the use of (Transform "MultiBSplineTransformWithNormal")?
I try to use transform of (Transform "MultiBSplineTransformWithNormal"), to fix a sliding motion registration. At elastix's parameter database, the param0016 example used this transform, which should provide a label image to figure out the two different sliding object in fix image. But the author's label files are unable to get.
I used ITK-snap to segment a 0/1 label image of a sliding object, and the command I use is:
elastix -f fixImage.nii -fMask fixMask.nii -labels fixBladder.nii -m movingImage.nii -out c:/Users/yc/elastix/output -p param-affine.txt -p param-bspline-multi.txt
But I got an error when running:

itk::ERROR: IsoContourDistanceImageFilter(000002004EBFB380): Gradient norm is lower than pixel precision

Someone said that's because you should use float pixel type, not short. But what I used is excatly float. So, what's the problem? I also remove the fixMask image to see if it is because the fix mask and fix label are conflicting, but this error still occur. Could anyone give some advice? Thanks a lot!

It seems to be a version problem. The implement from paper's author about sliding motion registration use a 4.6 version, but I use 5.0 version. However, when I try to download a 4.6 binaries from offical website, it just show a thanks page, but download nothing. Could anyone help me to get a 4.6 version? Thanks a lot.

it should be downloadable at the new website

In addition, perhaps the original author may be able to help you on how to use this component: http://dx.doi.org/10.1088/0031-9155/58/5/1303


Question on how to build envoy with prebuild dependencies

I have been trying to run the steps for the Production environment build from the documentation.
If I understand this correctly and the ci/WORKSPACE is a working example I should be able to:
% cd /home/projects/
% git clone https://github.com/envoyproxy/envoy.git
% cd envoy/ci
   #  place dependencies in /home/projects/ci/prebuild/thirdparty_build
% bazel build --package_path /home/projects/envoy //source/exe:envoy-static

When I run these steps I get the following error:
ERROR: error loading package '': Encountered error while reading extension file 'bazel/repositories.bzl': no such package '@envoy_api//bazel': Traceback (most recent call last):
        File "/home/projects/envoy/bazel/repositories.bzl", line 116
                ctx.symlink(ctx.path(ctx.attr.api).dirname.g...), ...)
        File "/home/projects/envoy/bazel/repositories.bzl", line 116, in ctx.symlink
                ctx.path(ctx.attr.api)
Unable to load package for @envoy//api:BUILD: not found.
ERROR: error loading package '': Encountered error while reading extension file 'bazel/repositories.bzl': no such package '@envoy_api//bazel': Traceback (most recent call last):
        File "/home/projects/envoy/bazel/repositories.bzl", line 116
                ctx.symlink(ctx.path(ctx.attr.api).dirname.g...), ...)
        File "/home/projects/envoy/bazel/repositories.bzl", line 116, in ctx.symlink
                ctx.path(ctx.attr.api)
Unable to load package for @envoy//api:BUILD: not found.
INFO: Elapsed time: 0.566s
FAILED: Build did NOT complete successfully (0 packages loaded)

Could you let me know what am I missing?
Thanks

Can I get an update on this question?
Thanks

@htuch  Could you answer this question?
Thanks

@georgi-d you have to have literally bazel build --package_path %workspace%:<path to Envoy source tree> //source/exe:envoy-static, i.e. the %workspace%: needs to be there.

@htuch Thanks for the reply.
I tried adding %workspace% and now it seems to start working but still fails:
% bazel build --package_path %workspace%:/home/projects/envoy //source/exe:envoy-static
Extracting Bazel installation...
..........................
Loading:
Loading: 0 packages loaded
DEBUG: /home/.cache/bazel/_bazel_username/82e6adcfecaf6358cfa23f1a6d8e8c04/external/io_bazel_rules_go/proto/def.bzl:133:3: You no longer need to call proto_register_toolchains(), it does nothing
Loading: 0 packages loaded
    currently loading: source/exe
Loading: 0 packages loaded
    currently loading: source/exe
Loading: 0 packages loaded
    currently loading: source/exe
Analyzing: target //source/exe:envoy-static (1 packages loaded)
ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': no such package '@envoy//bazel': /home/.cache/bazel/_bazel_username/82e6adcfecaf6358cfa23f1a6d8e8c04/external/envoy must be an existing directory
INFO: Elapsed time: 17.824s
FAILED: Build did NOT complete successfully (2 packages loaded)

@georgi-d the workspace in ci/WORKSPACE is setup for the CI environment and its paths (see the absolute path /source in the local_repository for Envoy). You'll need to modify to use this outside Docker or in your environment.

@htuch Thanks again,
I was able to proceed further by overriding the envoy repository location
bazel build --package_path %workspace%:/home/projects/envoy --override_repository=envoy=/home/projects/envoy //source/exe:envoy-static

Now the bazel finds the packages and starts the compilation.
I think it would be very helpful if a more detailed description was present in the documentation. At least explaining that the local_repository location needs to be updated or overridden on the command line.

@georgi-d PRs welcome; since you have direct experience doing this you might be best positioned to add the bridging text from the user perspective here.

@htuch  I will prepare a pull request.


Removes deprecated functions to test pull request analysis.

All is well.

retest this please

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.

retest this please

All is well.


Destination rules only apply to inbound

We have a single client replica, multiple server replica setup. If I set up a destination rule, seems that it will affect both client side and server side:

The client side won't be able to make more than 2 requests at the same time;
The server side replica won't be able to handle more than 2 requests at the same time;

Is it possible to config that we only want the server side rejection?
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: http-max-requests
spec:
  host: ...
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests: 2

How did you confirm that the server cannot handle to more than 2 requests at the same time?

How did you confirm that the server cannot handle to more than 2 requests at the same time?

I sent 10 requests from client to server, and only 2 succeed and 8 are rejected.  This is the behavior I was expecting.

If I have one client replica and two server replica, if I sent 10 requests from server to client, still, only 2 succeed. This is a bit annoying, since I was expecting 4 succeeds instead of 2 succeeds, since the server replica doubles.

Also, I tried to use EnvoyFilter to customize the circuit breaker thresholds, but it doesn't work, since Istio has the default configuration, and the existing threshold always trumps the new added threshold. Is there a way to remove the default Istio thresholds so that my customized threshold could take effect?

outbound is executed on the client side, not the server side.

If you want to confirm the impact on the server side. You should use one server and multiple clients.

@wulianglongrd

You should use one server and multiple clients.
Yes, I have tried this and server side is working correctly.

But what I want is 1) keeping the server side impact; 2) dropping the client side impact; Is it possible?

TBH, I'm not sure, it sounds like server-side rate limit.
https://istio.io/latest/docs/tasks/policy-enforcement/rate-limit/

Rate-limit and destination rule is different though.

Rate limit configures number of requests per time unit;
Destination rule configures number of concurrent request at a given time;

Can you describe your usage scenario in detail?

@wulianglongrd
Yes, basically we have N client replica and M server replica and we want each of the M server replica be able to reject requests if the server itself is handling K requests at the moment.

Support local rate limit. If that does't solve your problem, maybe... we can wait for other's insights. :)

@wulianglongrd Yes, basically we have N client replica and M server replica and we want each of the M server replica be able to reject requests if the server itself is handling K requests at the moment.

you should try with global rate limit.

@zirain but we want to limit the number of outstanding request instead of requests/per-minute.

Only for server side, please wait for this implementation https://github.com/istio/api/pull/2961

fixed by https://github.com/istio/istio/pull/47313


rename repository to Level/mem

shall we do this?

Are you looking for symmetry? E.g. leveldown is wrapped by level so memdown should be wrapped by mem?

I was thinking more about consistency, we have level-packager at level/packager and level-iterator-stream at level/iterator-stream. There are other repos following the old naming scheme still, but those should be moved over as well in my opinion.

Do it :)


Update Jupyter notebook

This PR updates the Jupyter notebook to reflect changes in #63 and clears the output of each cell.
I presume we do not want to add track the generated data and execution counts in Git. Furthermore, we currently do not track the generated SVG files in Git, so the cell outputs will not be shown correctly in the browser anyway.

so can we merge this PR?


Validation

How do we achieve validation?

let c: FormControl = new FormControl('', Validators.required);
this.userForm.addControl('rolesSelected', c);
When do the above validation works but the error message is there on form loading itself.

You could use the touched property for showing only after the select component has been touched. See also the "Use the select component with a reactive form" example.

Hi basvandenberg,
Thank you for your quick reply. I am using the form.valid to deactivate the submit button to achieve the validation at the moment.
If I submit my empty form, all the text fields with Validators.required are prompting message (default message I guess ) 'Please fill out this field' one after another. But that is not the case with ng-select.
Please advise.


Restore legacy content. Fixes #27

@jenlindner I think that's because this depends on #50 being merged first.

I commented in the other PR, but should have here -- we can't kill this content until we've replaced it. It's not great and probably should go somewhere else, maybe into a legacy tutorials section, but the WG committed to keeping it alive in some form when we took over its url.

@jenlindner this PR isn't killing the content, it's restoring it to it's former glory.

My bad! Heh, thanks. I'm going to link it from the side nav.

I moved the deeper dive and touring index files to page directory and assets to asset directories, works with side navigation.

I think it now looks bad again. The javascript isn't working.  I'm also disappointed you didn't put your changes up for review as described in the contributing guide: https://github.com/projecthydra/projecthydra.github.io/blob/master/CONTRIBUTING.md#submitting-changes


Add documentation on autoreloading of file.properties

Documentation for https://github.com/wso2/micro-integrator/issues/2332

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.sdkot seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

@sdkottegoda, please make sure to sign the Contributor License Agreement (CLA)

@sdkottegoda , pls send this fix to the 4.0.0 branch as well

@sdkottegoda, please make sure to sign the Contributor License Agreement (CLA)

I have signed, but it shows me as not signed

@sdkottegoda , pls send this fix to the 4.0.0 branch as well

This is related to the public fix which was shipped as a wum update for MI-1.2.0. And the fix is not ported to 4.0.0 yet, hence not applicable.


Expose more methods for UndoRedo

set_max_steps
get_max_steps
redo
undo

LGTM, would there be any reason why exposing these could be dangerous?

@hpvb I'm making tools for designer to edit data of our game project. The UndoRedo is very convient for that.
BTW. What is reason for the class is exposed but the key functions are hiden ?

no.. probably just forgot to bind them
On Thu, Sep 14, 2017 at 8:28 AM, Geequlim<EMAIL_ADDRESS>wrote:

@hpvb https://github.com/hpvb I'm making tools for designer to edit
data of our game project. The UndoRedo is very convient for that.
BTW. What is reason for the class is exposed but the key functions are
hiden ?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/11223#issuecomment-329453924,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF-Z24MdifmJfn1ebf6SHubXUxUxvF_cks5siQ3bgaJpZM4PVihp
.


Updated to Swift 4.2

Updates repo to support Swift 4.2 and Xcode 10.

Now supports Swift 5, Dark Mode, Swift Linted, and other tweaks


Handle empty values for task and datasource conditions in pod template selector

Description
The condition based selector that evaluates tasks based on specified criteria failed to handle empty values for task type or datasource. If the condition is empty or null it can be matched to any task, but previous logic checked for task type or task datasource in the empty set resulting in skipping of that selector. Expected behavior is to treat empty sets the same as null.
Fixed the evaluate method in Selector to check if the task type or datasource conditions are null or empty instead of just a null check.
Release note
Fixed: You can now pass empty arrays to type and dataSource keys in selector based pod template selection strategy

This PR has:

[x] been self-reviewed.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

@suneet-s The evaluate method has 100% test coverage. Passing null was already being tested by the existing tests. So we should be good to go from the testing perspective


legion_spy issue

Config:

commit d4f807d4
2 ranks, 1 rank per node
command line: mpiexec -np 2 mbind.x -cs -t40 -v ../builds/base_1d/exec/solver -logfile logs/log_%.log -hdf5:forcerw -dm:memoize -ll:cpu 0 -ll:util 2 -ll:csize 20000 -ll:rsize 0 -ll:onuma 0 -ll:ht_sharing 1 -ll:ocpu 1 -ll:othr 18 -lg:spy

The followings log files with legion_spy.py -d log* produces the following output:
Performing logical analysis...
Performing logical dependence analysis for top_level_task (Shard 0)...
ERROR: iface_residual_task (UID=736) failed to generate a close operation that we normally would have expected. This is likely a runtime bug. Re-run with logical checks to confirm.
FAIL

With legion_spy.py -e log*, it produces the following output:
Traceback (most recent call last):
  File "/nobackupp17/kbando/Softwares/legion/tools/legion_spy.py", line 13734, in <module>
    main(temp_dir)
  File "/nobackupp17/kbando/Softwares/legion/tools/legion_spy.py", line 13689, in main
    state.perform_physical_analysis(physical_checks)
  File "/nobackupp17/kbando/Softwares/legion/tools/legion_spy.py", line 12810, in perform_physical_analysis
    if not top_task.perform_task_physical_verification(perform_checks):
  File "/nobackupp17/kbando/Softwares/legion/tools/legion_spy.py", line 8724, in perform_task_physical_verification
    if not op.perform_op_physical_verification(perform_checks):
  File "/nobackupp17/kbando/Softwares/legion/tools/legion_spy.py", line 7918, in perform_op_physical_verification
    if not self.remove_restriction(index, req, perform_checks):
  File "/nobackupp17/kbando/Softwares/legion/tools/legion_spy.py", line 7723, in remove_restriction
    assert index in self.mappings
TypeError: argument of type 'NoneType' is not iterable

log_0.log
log_1.log
With a build with -DLEGION_SPY, legion_spy.py -l log* produces a bunch of lines like
Performing logical analysis...
Performing logical dependence analysis for top_level_task (Shard 0)...
INFO: Falling back to transitive mapping dependences for Mapping Op 68 due to either imprecision in the runtime analysis or a bug.
INFO: Falling back to transitive mapping dependences for Fill Op 72 due to either imprecision in the runtime analysis or a bug.
INFO: Falling back to transitive mapping dependences for Fill Op 74 due to either imprecision in the runtime analysis or a bug.
INFO: Falling back to transitive mapping dependences for Fill Op 78 due to either imprecision in the runtime analysis or a bug.
INFO: Falling back to transitive mapping dependences for Mapping Op 96 due to either imprecision in the runtime analysis or a bug.
INFO: Falling back to transitive mapping dependences for Mapping Op 114 due to either imprecision in the runtime analysis or a bug.
INFO: Falling back to transitive mapping dependences for Mapping Op 160 due to either imprecision in the runtime analysis or a bug.

log_0.log
log_1.log

Without -dm:memoize, the following line
ERROR: iface_residual_task (UID=736) failed to generate a close operation that we normally would have expected. This is likely a runtime bug. Re-run with logical checks to confirm.

goes away. I have another bug that seems to go away without -dm:memoize and was hoping to use legion spy to debug this. Maybe the two are somehow related.

I can't reproduce it with a single node. I originally had a different issue and Sean suggested to try Legion spy to debug it, hence the most recent github issues I posted. The original issue has to do with checkpoint files. I dump the content of a region to an HDF5 file using a CopyLauncher. If I run 10,000 iterations and dump every 1,000 iterations, the last checkpoint file systematically has missing copies (the shared elements on both ranks). We checked that using Realm logging. This doesn't happen without -dm:memoize. In practice, I use these checkpoint files for an error analysis and the corrupted elements obviously mess this up.

Both Legion and Legion Spy should be good for non-control-replication runs (even with multiple nodes). Both of them are buggy though when it comes to attach/detach operations with control replication, which I suspect is most of the problem in your case. If you actually use tasks to read/write to your files directly that might be a work-around for now.
Are you using index attach operations or just normal attach operations in the control replicated context?

I use a normal attach operation. When you say that they are buggy, do you mean I shouldn't be using attach operations? Checkponting and restarting are things that I very often do and I am not sure if this means that I shouldn't trust any of that. As a matter of fact, even post-processing files to visualize the solution involve an attach operation.
I ran with the master branch, the bug seems to go away. This was about 2x slower on two nodes which was surprising.  I also have weird values for some things that I log which doesn't make me very confident about running with master. I basically always work with control_replication so there might be several things that don't work correctly.

I use a normal attach operation. When you say that they are buggy, do you mean I shouldn't be using attach operations?

I would not trust any results from attach/detach/inline mapping operations launched in a control replicated parent task right now. There are known races in the physical analyses for those operations that can result in non-deterministic runtime behavior. (Index attach/detach operations should be fine, but Legion Spy will not understand them.)

I ran with the master branch, the bug seems to go away.

If you're referring to the Legion Spy issue, then that is expected. Legion Spy understands everything that is implemented in the master branch. It does not understand everything that is implemented for control replication yet.

This was about 2x slower on two nodes which was surprising.

We would need to look at a profile, but it's not terribly surprising to me. Control replication is an important execution strategy for distributing runtime meta-analysis work.

I also have weird values for some things that I log which doesn't make me very confident about running with master

The version of Legion in the master branch is very stable. It's effectively running code that is all at least 3 years old at this point and has been thoroughly debugged. Legion Spy should also validate everything you do on the master branch if you really want to give yourself confidence that the runtime is executing correctly.

Running 10 iterations with some outputs at the end with the mater branch of Legion with the following command-line
mpiexec -np 2 mbind.x -cs -t40 -v ../builds/base_1d_master/exec/solver -logfile logs/log_%.log -hdf5:forcerw -dm:memoize -ll:cpu 0 -ll:util 2 -ll:csize 20000 -ll:rsize 0 -ll:onuma 0 -ll:ht_sharing 1 -ll:ocpu 1 -ll:othr 18 -lg:warn -lg:partcheck -lg:safe_mapper

then processing the logs with the following command line
legion_spy.py -lpcsw log*

gave the followings
Performing logical analysis...
Performing logical dependence analysis for top_level_task...
INFO: Falling back to transitive mapping dependences for checkpoint_helper_task due to either imprecision in the runtime analysis or a bug.
INFO: Falling back to transitive mapping dependences for attach_and_write_solution_file_task due to either imprecision in the runtime analysis or a bug.
INFO: Falling back to transitive mapping dependences for compute_diagnosis_task due to either imprecision in the runtime analysis or a bug.
INFO: Falling back to transitive mapping dependences for compute_max_CFL_task due to either imprecision in the runtime analysis or a bug.
Pass
Performing logical sanity analysis for top_level_task...
ERROR: Failed logical sanity check. No mapping dependence between previous 0 and later Fence Op 218

Performing user event leak checks...
WARNING: ev(0x800080000420001f) is an untriggered runtime user event
  Incoming:
  Outgoing:
WARNING: ev(0x8000800005e00008) is an untriggered runtime user event
  Incoming:
  Outgoing:
Legion Spy analysis complete.  Exiting...

log_0.log
log_1.log
Is there an application bug here?

Should I close this?

Up to you. It's known and documented (#765) that Legion Spy doesn't fully work yet for verifying control replication programs. If you want to leave it open until the merge is complete to remind us to check before we do the merge that is fine.

@bandokihiro Please try again now the the collective branch has merged into the control replication branch. Legion Spy should be able to validate these kinds of programs. You'll need to regenerate the logs.

Initial attempts hit the following error
#7  0x00007ff922434729 in __assert_fail_base (fmt=0x7ff9225ca588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ff926d11048 "(collective_mapping == NULL) || !collective_mapping->contains(remote_inst)", file=0x7ff926d10c70 "/home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/garbage_collection.cc", line=593, function=<optimized out>)
    at assert.c:92
#8  0x00007ff922445fd6 in __GI___assert_fail (assertion=0x7ff926d11048 "(collective_mapping == NULL) || !collective_mapping->contains(remote_inst)", file=0x7ff926d10c70 "/home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/garbage_collection.cc", line=593,
    function=0x7ff926d10fe8 "void Legion::Internal::DistributedCollectable::update_remote_instances(Legion::AddressSpaceID)") at assert.c:101
#9  0x00007ff925f2e6bb in Legion::Internal::DistributedCollectable::update_remote_instances (this=0x7ff8fc1e56f0, remote_inst=1) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/garbage_collection.cc:593
#10 0x00007ff9263eb587 in Legion::Internal::ReplAttachOp::create_manager (this=0x7ff8f828aec0, node=0x7ff8f8287ad0, field_set=std::vector of length 1, capacity 1 = {...}, field_sizes=std::vector of length 1, capacity 1 = {...}, mask_index_map=std::vector of length 1, capacity 1 = {...}, serdez=std::vector of length 1, capacity 1 = {...}, external_mask=...)
    at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_replication.cc:7451
#11 0x00007ff92664dad0 in Legion::Internal::FieldSpaceNode::create_external_instance (this=0x7ff8f82823c0, field_set=std::vector of length 1, capacity 1 = {...}, node=0x7ff8f8287ad0, attach_op=0x7ff8f828aec0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/region_tree.cc:14278
#12 0x00007ff92661b7c2 in Legion::Internal::RegionTreeForest::create_external_instance (this=0x55ba72686010, attach_op=0x7ff8f828aec0, req=..., field_set=std::vector of length 1, capacity 1 = {...}) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/region_tree.cc:3104
#13 0x00007ff9262d215d in Legion::Internal::AttachOp::create_external_instance (this=0x7ff8f828aec0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_ops.cc:20886
#14 0x00007ff9262d2099 in Legion::Internal::AttachOp::trigger_prepipeline_stage (this=0x7ff8f828aec0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_ops.cc:20870
#15 0x00007ff92627da4e in Legion::Internal::Operation::execute_prepipeline_stage (this=0x7ff8f828aec0, generation=0, from_logical_analysis=false) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_ops.cc:1613
#16 0x00007ff92612dcca in Legion::Internal::InnerContext::process_prepipeline_stage (this=0x7ff8fc495e30) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_context.cc:7909
#17 0x00007ff92613e35a in Legion::Internal::InnerContext::handle_prepipeline_stage (args=0x7ff8f8284240) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_context.cc:11710
#18 0x00007ff9267527b0 in Legion::Internal::Runtime::legion_runtime_task (args=0x7ff8f8284240, arglen=12, userdata=0x55ba75ac91a0, userlen=8, p=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:31694
#19 0x00007ff9237fbbd2 in Realm::LocalTaskProcessor::execute_task (this=0x55ba7219dd60, func_id=4, task_args=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/proc_impl.cc:1135
#20 0x00007ff923874521 in Realm::Task::execute_on_processor (this=0x7ff8f82840c0, p=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:302
#21 0x00007ff923879724 in Realm::UserThreadTaskScheduler::execute_task (this=0x55ba72629470, task=0x7ff8f82840c0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:1632
#22 0x00007ff9238774c0 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x55ba72629470) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:1105
#23 0x00007ff92387ffa0 in Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop> (obj=0x55ba72629470) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/threads.inl:97
#24 0x00007ff92388edd2 in Realm::UserThread::uthread_entry () at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/threads.cc:1355

Pull and try again.

New failure mode
#8  0x00007fedc55ccfd6 in __GI___assert_fail (assertion=0x7fedc717c9ea "0 && \"invalid memory handle\"", file=0x7fedc717b350 "/home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/runtime_impl.cc", line=2667,
    function=0x7fedc717c9a8 "Realm::MemoryImpl* Realm::RuntimeImpl::get_memory_impl(Realm::ID)") at assert.c:101
#9  0x00007fedc69b79e2 in Realm::RuntimeImpl::get_memory_impl (this=0x564d66389760, id=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/runtime_impl.cc:2667
#10 0x00007fedc694a1b0 in Realm::Memory::kind (this=0x7feda0a99610) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/mem_impl.cc:61
#11 0x00007fedc9570839 in Legion::Internal::ReplAttachOp::create_manager (this=0x7fed98289b80, node=0x7fed9827d000, field_set=std::vector of length 1, capacity 1 = {...}, field_sizes=std::vector of length 1, capacity 1 = {...},
    mask_index_map=std::vector of length 1, capacity 1 = {...}, serdez=std::vector of length 1, capacity 1 = {...}, external_mask=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_replication.cc:7298
#12 0x00007fedc97d32d6 in Legion::Internal::FieldSpaceNode::create_external_instance (this=0x7fed98278460, field_set=std::vector of length 1, capacity 1 = {...}, node=0x7fed9827d000, attach_op=0x7fed98289b80)
    at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/region_tree.cc:14278
#13 0x00007fedc97a0fc8 in Legion::Internal::RegionTreeForest::create_external_instance (this=0x564d668a08b0, attach_op=0x7fed98289b80, req=..., field_set=std::vector of length 1, capacity 1 = {...})
    at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/region_tree.cc:3104
#14 0x00007fedc9458049 in Legion::Internal::AttachOp::create_external_instance (this=0x7fed98289b80) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_ops.cc:20886
#15 0x00007fedc9457f85 in Legion::Internal::AttachOp::trigger_prepipeline_stage (this=0x7fed98289b80) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_ops.cc:20870
#16 0x00007fedc940393a in Legion::Internal::Operation::execute_prepipeline_stage (this=0x7fed98289b80, generation=0, from_logical_analysis=true) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_ops.cc:1613
#17 0x00007fedc9403a99 in Legion::Internal::Operation::execute_dependence_analysis (this=0x7fed98289b80) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_ops.cc:1634
#18 0x00007fedc92b4370 in Legion::Internal::InnerContext::process_dependence_stage (this=0x7feda048e090) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_context.cc:8030
#19 0x00007fedc92c42d0 in Legion::Internal::InnerContext::handle_dependence_stage (args=0x7fed9826e030) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_context.cc:11720
#20 0x00007fedc98d7fca in Legion::Internal::Runtime::legion_runtime_task (args=0x7fed9826e030, arglen=12, userdata=0x564d69cf3cc0, userlen=8, p=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:31699
#21 0x00007fedc6982bd2 in Realm::LocalTaskProcessor::execute_task (this=0x564d663c8d60, func_id=4, task_args=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/proc_impl.cc:1135
#22 0x00007fedc69fb521 in Realm::Task::execute_on_processor (this=0x7fed9826deb0, p=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:302
#23 0x00007fedc6a00724 in Realm::UserThreadTaskScheduler::execute_task (this=0x564d66854170, task=0x7fed9826deb0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:1632
#24 0x00007fedc69fe4c0 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x564d66854170) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:1105
#25 0x00007fedc6a06fa0 in Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop> (obj=0x564d66854170) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/threads.inl:97
#26 0x00007fedc6a15dd2 in Realm::UserThread::uthread_entry () at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/threads.cc:1355

Pushed a fix, try again. This code is just poorly tested at the moment and I don't have enough time to test all the configurations of different file attaches in control replicated settings.

New backtrace
#8  0x00007f16400d7fd6 in __GI___assert_fail (assertion=0x7f1644a8dc40 "tree_finder != current_instances.end()", file=0x7f1644a856e8 "/home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc", line=10520,
    function=0x7f1644a8ea50 "Legion::Internal::RtEvent Legion::Internal::MemoryManager::detach_external_instance(Legion::Internal::PhysicalManager*)") at assert.c:101
#9  0x00007f1644391085 in Legion::Internal::MemoryManager::detach_external_instance (this=0x7f161c18eee0, manager=0x7f161c49d390) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:10520
#10 0x00007f1643eeb9a9 in Legion::Internal::PhysicalManager::detach_external_instance (this=0x7f161c49d390) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_instances.cc:3302
#11 0x00007f1643f6a595 in Legion::Internal::DetachOp::trigger_complete (this=0x7f161028c060) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_ops.cc:22503
#12 0x00007f164407d0a9 in Legion::Internal::ReplDetachOp::trigger_complete (this=0x7f161028c060) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_replication.cc:7641
#13 0x00007f1643dc0598 in Legion::Internal::InnerContext::process_trigger_completion_queue (this=0x7f161c001270) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_context.cc:8370
#14 0x00007f1643dcf6c2 in Legion::Internal::InnerContext::handle_trigger_completion_queue (args=0x7f1618013430) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/legion_context.cc:11804
#15 0x00007f16443e30aa in Legion::Internal::Runtime::legion_runtime_task (args=0x7f1618013430, arglen=12, userdata=0x56464d78c110, userlen=8, p=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:31704
#16 0x00007f164148dbd2 in Realm::LocalTaskProcessor::execute_task (this=0x564649e60d60, func_id=4, task_args=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/proc_impl.cc:1135
#17 0x00007f1641506521 in Realm::Task::execute_on_processor (this=0x7f16180132b0, p=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:302
#18 0x00007f164150b724 in Realm::UserThreadTaskScheduler::execute_task (this=0x56464a2ec5c0, task=0x7f16180132b0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:1632
#19 0x00007f16415094c0 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56464a2ec5c0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:1105
#20 0x00007f1641511fa0 in Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop> (obj=0x56464a2ec5c0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/threads.inl:97
#21 0x00007f1641520dd2 in Realm::UserThread::uthread_entry () at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/threads.cc:1355

Did you by any chance delete the region tree for this external instance before you issued the detach operation? If not I will need a reproducer to understand what is going wrong here.

I have a reproducer at /scratch2/bandokihiro/Issue1327 that you can run with sbatch runslurm.sh.

Pull and try again. There's also a fix for another failure mode that you hadn't observed before.

New backtrace
#6  0x00007fdaa8e32859 in __GI_abort () at abort.c:79
#7  0x00007fdaad14d2db in Legion::Internal::Runtime::report_error_message (id=482, file_name=0x7fdaad7d2fd0 "/home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/region_tree.cc", line=4605, message=0x7fda8c415020 "Unable to find entry for region tree ID 9")
    at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:31424
#8  0x00007fdaad01d9b7 in Legion::Internal::RegionTreeForest::get_tree (this=0x55d8896de0e0, tid=9, first=true) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/region_tree.cc:4605
#9  0x00007fdaad063f9e in Legion::Internal::RegionNode::handle_top_level_request (forest=0x55d8896de0e0, derez=..., source=0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/region_tree.cc:20380
#10 0x00007fdaad13085b in Legion::Internal::Runtime::handle_top_level_region_request (this=0x55d88a713860, derez=..., source=0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:23806
#11 0x00007fdaad0fee5f in Legion::Internal::VirtualChannel::handle_messages (this=0x7fda8c416f60, num_messages=1, runtime=0x55d88a713860, remote_address_space=0, args=0x7fda90008f90 "\t", arglen=12)
    at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:11972
#12 0x00007fdaad0fe1c3 in Legion::Internal::VirtualChannel::process_message (this=0x7fda8c416f60, args=0x7fda90008f74, arglen=32, runtime=0x55d88a713860, remote_address_space=0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:11546
#13 0x00007fdaad100755 in Legion::Internal::MessageManager::receive_message (this=0x7fda8c213da0, args=0x7fda90008f70, arglen=40) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:13213
#14 0x00007fdaad135f1c in Legion::Internal::Runtime::process_message_task (this=0x55d88a713860, args=0x7fda90008f6c, arglen=44) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:25938
#15 0x00007fdaad14da10 in Legion::Internal::Runtime::legion_runtime_task (args=0x7fda90008f60, arglen=48, userdata=0x55d88cb21570, userlen=8, p=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/legion/runtime.cc:31651
#16 0x00007fdaaa1f5bd2 in Realm::LocalTaskProcessor::execute_task (this=0x55d8891cb7c0, func_id=4, task_args=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/proc_impl.cc:1135
#17 0x00007fdaaa26e521 in Realm::Task::execute_on_processor (this=0x7fda90008de0, p=...) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:302
#18 0x00007fdaaa273724 in Realm::UserThreadTaskScheduler::execute_task (this=0x55d889681950, task=0x7fda90008de0) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:1632
#19 0x00007fdaaa2714c0 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x55d889681950) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/tasks.cc:1105
#20 0x00007fdaaa279fa0 in Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop> (obj=0x55d889681950) at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/threads.inl:97
#21 0x00007fdaaa288dd2 in Realm::UserThread::uthread_entry () at /home/bandokihiro/scratch/Builds_CPU/legion/runtime/realm/threads.cc:1355

The same reproducer should work if needed.

There was an unrelated bug that was introduced last night. Please pull and try again.

So I ran the case that initiated this issue with -DLEGION_SPY and processed the logs using
legion_spy.py -lpcwg log_*.log

There seems to be only one error during the physical analysis. Please look at /home/bandokihiro/Temp/log_21647.out and let me know if this is good enough.

Same log file should validate now.

Yes, it looks good. Thanks!


[TASK][EASY] Create PySpark batch jobs tests for RESTful API

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before creating

[X] I have searched in the task list and found no similar tasks.

Mentor

[ ] I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.

Skill requirements

Knowledge about Spark submit
Basic Knowledge of Scala testing and PySpark programming

Background and Goals
Restful API supports creating Batch jobs by posting /batches requests with or without resource file uploading. Currently, it is mainly used for submitting Spark Jars. As PySpark jobs become popular approach for data exploring and processing, we need to create tests for creating PySpark jobs.

Rest API docs for creating batches

https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches
https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches-with-uploading-resource

Rest API test caes:

POST without uploading
https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/BatchRestApiSuite.scala#L141
POST with uploading https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/BatchRestApiSuite.scala#L90

Implementation steps
Add tests for creating PySpark jobs via Rest API

create tests for creating PySpark jobs using POST /batches with file on path of storage , e.g.s local filesystem, HDFS
create tests for creating PySpark jobs with resource file uploading (https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches)
create tests with Rest API client
zip PySpark scripts and related files before if necessary

Additional context
Original reporter is @bowenliang123

@pan3793 Could you please assign it to me ?  I suppose to complete this task on 27th October

Thanks. Assigned to @weixi62961 . Looking forward to your PRs.

@bowenliang123  cc @pan3793
I want to talk about current progress.  If you have any question, please tell me.
Before writing  test cases,  I first used curl to test some PySpark jobs on Kyuubi 1.7.1. They were mainly tested in the local machine.  All these curl test cases were passed.

PySpark job file is hosted in HDFS

# hdfs pi.py
curl -H "Content-Type: application/json" \
-X POST \
-d '{"batchType": "PYSPARK", "resource":"hdfs:/tmp/upload/pyspark_submit_sample/pi.py",  "name": "PySpark PI", "conf": {"spark.master": "local"}, "args": [10]}' \
http://localhost:10099/api/v1/batches

PySpark job file is uploaded through uploading resource

# upload resource: pi.py
curl --location --request POST 'http://localhost:10099/api/v1/batches' \
--form 'batchRequest="{\"batchType\":\"PYSPARK\",\"name\":\"PySpark Pi\",\"args\":[10]}";type=application/json' \
--form<EMAIL_ADDRESS>

PySpark job file depends on other modules

# module dependency
curl -H "Content-Type: application/json" \
-X POST \
-d '{"batchType":"PYSPARK","resource":"hdfs:/tmp/upload/pyspark_submit_sample/test_module_dependency.py","name":"PySpark Module Dependency","conf":{"spark.master":"local","spark.submit.pyFiles":"hdfs:/tmp/upload/pyspark_submit_sample/my_module.zip"}}' \
http://localhost:10099/api/v1/batches

Refer to the existting 2 test cases for Spark jars, I'll wirte two test case for PySpark for "POST without uploading" and "POST with uploading".
Is it OK?
In addition, due to the complexity of PySpark's package management, it is not within the scope of this topic. For more information, refer to spark offical site: Python Package Management
BTW, I have created a small project for test [pyspark submit sample] (https://github.com/weixi62961/pyspark_submit_sample)

Thanks for updaing the detailed progress. Small PRs for subtasks are welcomed.
Raise the first PR for case 2, which sounds like the easiest case for minimum testing.
POST with uploading + test py file in resource folder for testing + local standalone for Spark master.
WDYT?

WDYT?

OK， Thanks

@weixi62961 May I have contact with you? You can send your private wechat account number to my email address<EMAIL_ADDRESS>

@weixi62961 May I have contact with you? You can send your private wechat account number to my email address<EMAIL_ADDRESS>
Already sent

Summary
Let‘s summarize the status of PySpark Batch Job. In my opinion, For PySpark jobs, it is recommended to submit jobs by post batches existing resources instead of uploading resources.
Problem

up to Kyuubi 1.8.0，Rest API for creating batches in two ways：

https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches
https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches-with-uploading-resource

Currently, uploading-resource can only upload one resource file, which is not enough for normal PySpark jobs.

https://github.com/apache/kyuubi/blob/ceb84537835b61776629d3f1f23dade862059e1e/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala#L177

PySpark jobs are different from JarSpark jobs, which are usually an uber-jar/fat-jar with all dependencies. Normal PySpark jobs require not only resource file, but also dependency zip packages(--py-files).  spark-submt examples as follows:

# JarSpark job
$SPARK_HOME/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
fat-jar-with-dependencies.jar

# PySpark job
$SPARK_HOME/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--py-files dependency.zip \
main.py

Workaround

For PySpark jobs, it is recommended to submit jobs by post batches existing resources instead of uploading resources.
First upload the resource file and dependency zip packages to a location that is accessible to all nodes, such as HDFS or NFS. Then submit the job through the post-batches API. Here is an example, using HDFS.
json parameters in pretty format

{
	"batchType": "PYSPARK",
	"resource": "hdfs:/tmp/upload/main.py",
	"conf": {
		"spark.master": "yarn",
		"spark.submit.deployMode": "cluster",
		"spark.submit.pyFiles": "hdfs:/tmp/upload/dependency.zip"
	}
}

curl command ( it works!)

curl -H "Content-Type: application/json" \
-X POST \
-d '{"batchType":"PYSPARK","resource":"hdfs:/tmp/upload/main.py","conf":{"spark.master":"yarn","spark.submit.deployMode":"cluster","spark.submit.pyFiles":"hdfs:/tmp/upload/dependency.zip"}}' \
http://localhost:10099/api/v1/batches

WDYT？ @bowenliang123  cc @pan3793
For more PySpark job submit cases, please refer to my small project for  [pyspark submit sample(https://github.com/weixi62961/pyspark_submit_sample)


[SPARK-12421][SQL] Prevent Internal/External row from exposing state.

It is currently possible to change the values of the supposedly immutable GenericRow and GenericInternalRow classes. This is caused by the fact that scala's ArrayOps toArray (returned by calling toSeq) will return the backing array instead of a copy. This PR fixes this problem.
This PR was inspired by https://github.com/apache/spark/pull/10374 by @apo1.
cc @apo1 @sarutak @marmbrus @cloud-fan @nongli (everyone in the previous conversation).

Test build #48581 has started for PR 10553 at commit 632b5dc.

Test build #48582 has started for PR 10553 at commit daa123b.

Test build #48581 has finished for PR 10553 at commit 632b5dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48581/
Test PASSed.

Test build #48582 has finished for PR 10553 at commit daa123b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48582/
Test FAILed.

Test build #48584 has started for PR 10553 at commit 4d5d162.

Test build #48584 has finished for PR 10553 at commit 4d5d162.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48584/
Test PASSed.

Merged build finished. Test PASSed.

Thanks, merging to master.


Migrate to simple reference resolution and selection

Description of your changes

This adds a reference selector for all existing references, and uses the new managed.APISimpleReferenceResolver to select and resolve references. See https://github.com/crossplane/crossplane-runtime/pull/151 for more detail.
Checklist

I have:

[x] Run make reviewable to ensure this PR is ready for review.
[x] Ensured this PR contains a neat, self documenting set of commits.
[ ] Updated any relevant documentation, examples, or release notes.
[ ] Updated the dependencies in app.yaml to include any new role permissions.

I've tested this by applying examples/cache, examples/compute, and examples/database. All resources successfully resolved their references and created. I also tested that error messages appeared as expected if I attempted to create a managed resource before the resource it depended on existed, or before the resource it depended on had actually set the referenced value.
This PR decreases test coverage by a little over 1%, presumably because each referencer function under apis/ previously had tests. I'm not planning on adding tests for the new ResolveReferences methods, as they're all effectively boilerplate calls to reference.NewAPIResolver, which has its own tests.

has that behavior been tested as well?

@hasheddan I didn't test 100% of them, but I have validated that the code generally works as expected by using a selector to pick a resource group reference for a MySQLServer.


Prefer healthy containers when scaling down services

Rancher Version: 1.0.1
Docker Version: 1.10.3
OS: Ubuntu 14.04
Steps to Reproduce:

Make a service with a scale of 5
Power off one host during the service initialization so that one container is in the "starting" state
Scale the service down

Results:
The healthy containers are not necessarily the ones that are preserved. It is possible that the container in the "starting" state will be chosen to survive the scaled down.
Expected:
Rancher should prefer to save containers that are healthy when scaling down services.

Closing due to inactivity, please reopen if you think this is still relevant to current releases & important.


chore: bump all version and update code

Update version
yarn audit

Old:
38 vulnerabilities found - Packages audited: 632
Severity: 2 Low | 15 Moderate | 20 High | 1 Critical
Done in 1.86s.

This commit:
12 vulnerabilities found - Packages audited: 676
Severity: 7 Moderate | 4 High | 1 Critical
Done in 1.86s.

Update marked:
var marked = require('marked');

// =>

var { marked } = require('marked');

Update underscore:
  return _.template(str, params, templateSettings);
  return _.template(str, templateSettings)(params);

preview looks good!
https://deploy-preview-935--xenodochial-hugle-b9ec84.netlify.app/

Pushed to prod, looks good this time :)