[Blueprint] Compiling error: Function '__tact_dict_get_int_int' does not exist in imported FunC sources

Problem
Trying to compile big project with many contracts dependencies, and getting next error:
Function '__tact_dict_get_int_int' does not exist in imported FunC sources
Trait, where error is being thrown:
trait LockableAction{

    lockableAction: map<Int as uint64, Bool>;

    inline fun lock(userId: Int){
        self.lockableAction.set(userId, true);
    }
    inline fun unlock(userId: Int){
        self.lockableAction.del(userId);
    }

    inline fun requireNotLocked(userId: Int){
        require(self.lockableAction.get(userId) == null, "Please wait untill previous transaction finish!");
    }
}

Generated func code:
((int, slice, int, slice, cell, int, int), ()) $UserPost$_fun_requireNotLocked((int, slice, int, slice, cell, int, int) $self, int $userId) impure inline {
    var (($self'postId, $self'text, $self'ownerUserId, $self'master, $self'lockableAction, $self'likesCount, $self'dislikesCount)) = $self;
    ////////////////////////////////          |> error here
    throw_unless(1151, null?(__tact_dict_get_int_int($self'lockableAction, 64, $userId, 1)));
    return (($self'postId, $self'text, $self'ownerUserId, $self'master, $self'lockableAction, $self'likesCount, $self'dislikesCount), ());
}

Steps to reproduce
git clone https://github.com/imartemy1524/ton-social -b tact-error
cd ton-social
npm i
npx blueprint build # error here

Other comments
It seems, that this is the problem with Bool value type in map, because when one replaces map<Int as uint64, Bool> with map<Int as uint64, Int as uint8>, the problem fixes.
My temporary solution:
trait LockableAction{

    lockableAction: map<Int as uint64, Int as uint8>;

    inline fun lock(userId: Int){
        self.lockableAction.set(userId, 0);
    }
    inline fun unlock(userId: Int){
        self.lockableAction.del(userId);
    }

    inline fun requireNotLocked(userId: Int){
        require(self.lockableAction.get(userId) == null, "Please wait untill previous transaction finish!");
    }
}

@imartemy1524 Let me thank you for such a well-written report! We will look into this asap. Incidentally, blueprint build should report a failure but it does not, I filed an issue here: https://github.com/ton-org/blueprint/issues/144


Efficient skip in TarInputStream

I certify that I own, and have sufficient rights to contribute, all source code and related material intended to be compiled or integrated with the source code for the SharpZipLib open source product (the "Contribution"). My Contribution is licensed under the MIT License.
Added a small bit of code that allows a TarInputStream to efficiently
skip portions of its underlying InputStream without screwing up the
bookkeeping in TarBuffer.

I realize that this is pretty old and cannot be merged as is anyway, but this really needs tests to "prove" that it can safely skip the parts. If it gets updated and some tests are added I can reopen this.

Thanks for the response.  I no longer work for the company that needed this fix and don't have the surrounding code or good example data, but if I ever get access to all that again, I'll try to submit something with tests.  Incidentally, this is 1/2 of the equation for being able to seek quickly around TAR archives; the other half is fixing TarInputStream.Position, which does not accurately report the true position in the stream.  I submitted a pull request for that as well, but it was closed as wontfix.

Yeah, sorry about that. But both of them had been stale for several years without anyone touching them, and as I wrote then, I didn't see how that was supposed to be used. Going through and verifying changes like this really needs to motivated by an outstanding issue etc, otherwise it's just adding risk and maintenance burden to a project that already has limited resources as is.


first workspace vanishes when doing a workplace swipe guesture

Hyprland Version
Hyprland, built from branch  at commit 3bb9c7c5cf4f2ee30bf821501499f2308d616f94  (). Tag:   flags: (if any) debug
Bug or Regression?
Bug
Description
This only happens in the first workspace, no matter what window is on it. other workspace are completely unaffected
if you only partially swipe do it it still stays gone until switching away and back again.
using the workspace switch ie, mod 1,2,3. works fine.
it seems to be specifically transitioning away from using gestures, to with gestures works fine.
How to reproduce
https://github.com/knoc-off/nixos/tree/d4d2405a47e742a56d15553704cf32a4cdfdc966
my nixos configs. please let me know if i can send any info
using intel integrated graphics.
Crash reports, logs, images, videos
hyprland.log

#3437


Add track to playlist prepend (position 0) instead of append to end

The Spotify API supports the optional position integer. I'd like to use it to add a new track as the first track of the playlist.
I'm not an PHP expert, so I'm wondering how I can reach this. I tried:

$api->addUserPlaylistTracks($me->id, $playlistId, '6sLY2CNublHBA0nsjnXvGR', ['position=0']);
$api->addUserPlaylistTracks($me->id, $playlistId, '6sLY2CNublHBA0nsjnXvGR', ['\"position\": 0']);
$api->addUserPlaylistTracks($me->id, $playlistId, ['59FjL5y7X8P8PmR8o9lmpi'],['\"position\": 0']);

But none of it works. It seems that I can use the option array in your php code but probably I don't form the data well?  There is no error thrown, the track just keeps being appended to the end of the list.
I hope you can help. Thank you in advance for your time.

The syntax should be:
$api->addUserPlaylistTracks($me->id, $playlistId, ['someTrackId'], ['position' => 0]);
Whether it has the intended effect of prepending the track I don't know.


[Android] [SampleApp] Some brushes not applied correctly when switching themes

Current behavior
Some colors don't seem to be applied correctly when switching from Light Theme to Dark Theme and viceversa. This might be present in a number of places in the app, but it's easily reproducible in the Sample App in the Android head.
Expected behavior
When switching themes in Android in the sample app, the colors should be consistent across all platforms.
How to reproduce it (as minimally and precisely as possible)

Clone the project and switch to the master branch.
Run the Uno.Themes.Samples.Droid head and deploy to an Android device / simulator.
Using the burger menu, click the "FAB" option. Notice the FAB colors.
Using the burger menu, click on the Theme toggle to switch themes
Observe the label colors.
Repeat the process above with a different platform (any). Notice the colors on the labels there.

Please notice this might be present in other controls, but the reproduction steps may vary.
Environment

Nuget Package: N/A
Package Version(s): N/A
Affected platform(s):

[ ] iOS
[X] Android
[ ] WebAssembly
[ ] UWP
[ ] MacOS

Anything else we need to know?

Same for Button in addition to FAB
Plus the main NavigationView is also not changing theme correctly

Similar issue on WASM -> https://github.com/unoplatform/Uno.Gallery/issues/974

https://github.com/unoplatform/Uno.Gallery/pull/1022/files is now merged with a workaround, need to test it again on Android


Wheels slowly drifting away from the robot

I don't know what has been causing this, it might be a bug related to rotation since I tweaked those values. I'll report any findings related to this bug if I find any.

found minimal reproducible test case:
# rcj_soccer_player controller - ROBOT Y3

###### REQUIRED in order to import files from Y1 controller
import sys
from pathlib import Path

sys.path.append(str(Path(".").absolute().parent))
# You can now import scripts that you put into the folder with your
# robot B1 controller
from rcj_soccer_player_y1 import rcj_soccer_robot, utils

######

# Feel free to import built-in libraries
import math

class MyRobot(rcj_soccer_robot.RCJSoccerRobot):
    def run(self):
        while self.robot.step(rcj_soccer_robot.TIME_STEP) != -1:
            if self.is_new_data():
                data = self.get_new_data()

                # Get the position of our robot
                robot_pos = data[self.name]
                # Get the position of the ball
                ball_pos = data["ball"]

                # Get angle between the robot and the ball
                # and between the robot and the north
                ball_angle, robot_angle = self.get_angles(ball_pos, robot_pos)

                # Compute the speed for motors
-                 direction = utils.get_direction(ball_angle)
+                direction = 1

                # If the robot has the ball right in front of it, go forward,
                # rotate otherwise
                if direction == 0:
                    left_speed = -5
                    right_speed = -5
                else:
                    left_speed = direction * 4
                    right_speed = direction * -4

                # Set the speed to motors
                self.left_motor.setVelocity(left_speed)
                self.right_motor.setVelocity(right_speed)

my_robot = MyRobot()
my_robot.run()

found minimal reproducible test case:
# rcj_soccer_player controller - ROBOT Y3

###### REQUIRED in order to import files from Y1 controller
import sys
from pathlib import Path

sys.path.append(str(Path(".").absolute().parent))
# You can now import scripts that you put into the folder with your
# robot B1 controller
from rcj_soccer_player_y1 import rcj_soccer_robot, utils

######

# Feel free to import built-in libraries
import math

class MyRobot(rcj_soccer_robot.RCJSoccerRobot):
    def run(self):
        while self.robot.step(rcj_soccer_robot.TIME_STEP) != -1:
            if self.is_new_data():
                data = self.get_new_data()

                # Get the position of our robot
                robot_pos = data[self.name]
                # Get the position of the ball
                ball_pos = data["ball"]

                # Get angle between the robot and the ball
                # and between the robot and the north
                ball_angle, robot_angle = self.get_angles(ball_pos, robot_pos)

                # Compute the speed for motors
-                 direction = utils.get_direction(ball_angle)
+                direction = 1

                # If the robot has the ball right in front of it, go forward,
                # rotate otherwise
                if direction == 0:
                    left_speed = -5
                    right_speed = -5
                else:
                    left_speed = direction * 4
                    right_speed = direction * -4

                # Set the speed to motors
                self.left_motor.setVelocity(left_speed)
                self.right_motor.setVelocity(right_speed)

my_robot = MyRobot()
my_robot.run()

Hello @Bluskript. Could you please test if this problem occurs with the newest software?
I'll close this issue for now. If you ran into similar problem like you reported, feel free to reopen this issue.


Identify and triage test and release details for all product projects

SnarkyJS
zkapp in Mina/SnarkyJS connective tissue
zkapp-cli
Proof of Alpha + any full zkapp projects
Faucet
Docs Website
Eslint SnarkyJS Plugin
zkApps cross-product e2e testing
zkApps Holistic Release Checklist

done


feat: Support disabling aztec vm in non-wasm builds

As in the title.

Benchmark results
Metrics with a significant change:

proof_construction_time_sha256_30_ms (16): 1,372 (-21%)
protocol_circuit_witness_generation_time_in_ms (private-kernel-tail): 2,174 (+20%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Proof generation
Each column represents the number of threads used in proof generation.

Metric
1 threads
4 threads
16 threads
32 threads
64 threads

proof_construction_time_sha256_ms
5,727 (-1%)
1,545 (-1%)
695 (-1%)
741 (-3%)
769

proof_construction_time_sha256_30_ms
11,447 (-1%)
3,066 (-1%)
:warning: 1,372 (-21%)
1,410 (-11%)
1,461 (-4%)

proof_construction_time_sha256_100_ms
44,001 (-1%)
11,763
5,436
5,446 (+1%)
5,350

proof_construction_time_poseidon_hash_ms
78.0
34.0
34.0
58.0
88.0

proof_construction_time_poseidon_hash_30_ms
1,521 (-1%)
419 (-1%)
201 (-1%)
227
263 (-3%)

proof_construction_time_poseidon_hash_100_ms
5,751 (-1%)
1,578 (-2%)
725 (-1%)
782 (-1%)
793

L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
4 txs
8 txs
16 txs

l1_rollup_calldata_size_in_bytes
1,412
1,412
1,412

l1_rollup_calldata_gas
9,464
9,450
9,464

l1_rollup_execution_gas
610,285
610,271
610,285

l2_block_processing_time_in_ms
702 (-1%)
1,285
2,486

l2_block_building_time_in_ms
22,666 (+1%)
46,228 (+1%)
88,617 (+1%)

l2_block_rollup_simulation_time_in_ms
22,558 (+1%)
45,873 (+1%)
88,224 (+1%)

l2_block_public_tx_process_time_in_ms
19,411 (+1%)
42,427 (+1%)
84,927 (+1%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 8 txs.

Metric
3 blocks
5 blocks

node_history_sync_time_in_ms
5,885
8,436

node_database_size_in_bytes
9,461,840
12,755,024

pxe_database_size_in_bytes
9,242
15,127

Circuits stats
Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
256 leaves
512 leaves
1024 leaves
32 leaves

batch_insert_into_append_only_tree_16_depth_ms
10.4 (-1%)
17.3 (+2%)
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.7
31.8
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.603 (-1%)
0.531 (+2%)
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
49.9 (+3%)
76.3
134 (-1%)
245 (-1%)
476
N/A

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
95.9
159
288
543
1,055
N/A

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.510 (+3%)
0.470
0.458 (-1%)
0.445
0.445
N/A

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
61.1 (+2%)
113
185
354
703
N/A

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
109
208
355
691
1,363
N/A

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.517 (+2%)
0.505
0.490
0.479
0.483
N/A

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
60.8

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
N/A
N/A
N/A
N/A
N/A
103

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.556 (+1%)

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
79,300
673,633

Transaction size based on fee payment method
| Metric |  |
| - |  |


Clean up the readme file so it makes sense.

The readme was created in a hurry by number of people. I think it's time we apply some structure to the actual file. Let's keep it basic and move the other development details into a top-level docs directory. We can then add a table of contents to the bottom of the readme file linking to required development details.
High-level thoughts:

Create a /docs directory
Create new readme files for development, testing, deployments
Move required development, testing information out of the current readme.md and into the required /docs/<topic>/readme.md.
Include a table of contents into the root readme.md that links to other /docs files.

.... Do you want to take this a step further and setup readthedocs? Go right ahead, I will help support that effort!

I would like do contribute !!

@divyankkarolia97 awesome! Please let me know if you need any help.

I'd like to help as well. Spinning up readthedocs right now. :)

@khorne3 please see if you can add webhook as a contributor..

PR #51 moved the documentation into the /docs directory.

@divyankkarolia97 could you fix the docs to be relative path instead of absolute. This would allow the docs to load locally.

@BondAnthony just modified the paths, take a look at the PR.

@BondAnthony No dice.
Instructions are here.
Payload URL: readthedocs.org/api/v2/webhook/status-service/15358/

@khorne3 try again. I added the hook so you will probably need to create an event by opening a pull request and updating content that would be sent to readthedocs.

Something strange is going on, @BondAnthony. The build will succeed but not generate docs. I've forked a version so I can check it out locally. Will report back when I have more.


Not able to navigate from within redux action creators.

Hello, I am one of those people who has just switched from using react-router-flux.  We use redux for our app. Everything seems to be going smoothly with the react-navigation implementation except I am not able to use navigation outside the screens.
The home screen includes a sign up button which when pressed loads a redux action creator which dispatches an action.  In our actions file I have imported import { NavigationActions } from 'react-navigation'; at the top and then tried to navigate the user to the sign up screen with an action dispatch:
  export const signupClicked = () => {
   return (dispatch) => {
    dispatch({ type: SIGNUP_CLICKED });
    dispatch(NavigationActions.navigate({ routeName: 'SignupForm' }));
  };
};

This does not seem to be working.  While using react-router-flux instead of     dispatch(NavigationActions.navigate({ routeName: 'SignupForm' })); I would have had something like Actions.Signup(); and that would navigate to the sign up page.  Any help would be appreciated.
Thanks

StackNavigator props navigation={addNavigationHelpers({redux.dispatch, redux.state})}

Thanks for the reply, I am really new to react-native, so can you tell me where I would put the code you provided above?  How should I utilize it exactly?

@yiluzhang  can you elaborate on your answer.

@yiluzhang can you please elaborate?

https://reactnavigation.org/docs/guides/redux
Using Redux to manage navigation.
Your AppNavigator needs navigation props.
AppNavigator props is created by the addNavigationHelpers method.
In Redux, routing is changed through the AppNavigator method of router.getStateForAction.
if (action.type === 'Navigation/NAVIGATE') {
    switch (action.routeName) {
        case 'TAB_NAVIGATOR_MYSELF':
        case 'STACK_NAVIGATOR_USER_XXX':
        case 'STACK_NAVIGATOR_USER_XXXX':
            if (state.notlogin) { action.routeName = 'STACK_NAVIGATOR_LOGIN'; }
            break;
      }
}
nav = AppNavigator.router.getStateForAction(action, state.nav);

<AppNavigator navigation={addNavigationHelpers({redux.dispatch, redux.state})}/>

Can I help you with my answer? @jasan-s @roderickfung

@yiluzhang Do you have to use Redux to manage navigation, in order for components/screens to use redux ?

Pinging OP @Moebats since this issue is quite old and related to an old version of the lib. If no answer I'll close in 7 days.

Hi there @Moebats ,
In an effort to cleanup this project and prioritize a bit, since this issue had no follow up since my last comment I'm going to close it.
If you are still having the issue (especially if it's a bug report) please check the current open issues, and if you can't find a duplicate, open a new one that uses the new Issue Template to provide some more details to help us solve it.


feat: frequency annotation helper

Issue
partially resolves https://github.com/validator-labs/validator/issues/358
Description

A constant was added to define the annotation key validation.validator.labs/reconciliation-frequency.
Function FrequencyFromAnnotations which accepts annotation map and returns ctrl.Result.
Tests for FrequencyFromAnnotations function.

No worries @TylerGillson
I have updated the logs to reflect the changes.

@ADorigi thanks for making those changes. We just need a make reviewable and it'll be good to go!

@TylerGillson
Running make reviewable made a few changes in chart/validator/values.yaml and chart/validator/README.md, which may not be as expected. Let me know if these need to be reverted.

@ADorigi you needed to do a git submodule update --remote then make reviewable.


fix(react): fix build typescript declaration file

General summary
WIP fix the generation of the typescript declaration file (that should be located under dist/lumx.react.d.ts)
Currently in error:
https://github.com/unimonkiez/ts-declaration-webpack-plugin/issues/4

Fixed in https://github.com/lumapps/design-system/pull/322


DOC: add a little note about the Normal survival function (Q-function)

As prompted in the mailing list, I'm adding this note which hopefully will be helpful for anyone looking for an implementation of the Q-function.
Is this place ok or should this note go elsewhere (too)?

The other way around: "The survival function, norm.sf, is also referred to as a Q-function in some contexts (see, e.g., [some reference])".
Rationale: The term "Q-function" is ambiguous, http://mathworld.wolfram.com/Q-Function.html

@@            master   #5864   diff @@
======================================
  Files          237     237
  Stmts        43510   43510
  Branches      8159    8159
  Methods          0       0
======================================
- Hit          33940   33938     -2
- Partial       2600    2601     +1
- Missed        6970    6971     +1

Review entire Coverage Diff as of 41b11b1

Powered by Codecov. Updated on successful CI builds.

@ev-br :+1:

Could you squash the two commits please. $ git rebase -i HEAD~2, then change the second "pick" to "squash" and edit the commit message to start with "DOC".

@ev-br I'm not fluent in git, so I think I messed things up here, sorry.

No problem, a first rebase is always a mess... Two issues here: do not go too far into the history (this picks up earlier commits which should not be here), and a merge -- better rebase, don't merge. To fix it up: $git fetch upstream, then git rebase upstream/master. Then git rebase -i upstream/master, and edit the commits. If in doubt, delete the hashes in the editor, this will abort the rebase, also git rebase --abort.

@mirca Something like this:
$ git fetch upstream                # <<<<------
remote: Counting objects: 256, done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 256 (delta 206), reused 187 (delta 187), pack-reused 20
Receiving objects: 100% (256/256), 230.70 KiB, done.
Resolving deltas: 100% (212/212), completed with 114 local objects.
From https://github.com/scipy/scipy
   94ee2c4..340ada7  master     -> upstream/master
 <... some more... stuff specific to my setup...>
$
$ git rebase upstream/master                           # <<<<<<----
First, rewinding head to replay your work on top of it...
Applying: add a little note about the Q-function
Applying: add a little note about the normal survival function
Applying: BUG: fix stats.ttest_ind_from_stats to handle arrays.
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
Applying: add a little note about the Q-function
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
$
$ # now git log only shows your two commits
$
$ git rebase -i HEAD~2       # <<<<<<-------
$  # .... edit the commit messages: pick --> squash etc
$ git push origin HEAD -f     # <<<<<<< -----   Note the -f switch
$

The result is the top commit at https://github.com/ev-br/scipy/commits/pr/5864 --- you can either take it over or repeat the steps above, whichever you prefer.

@ev-br I redo your steps, still no success. An empty vim window shows up after git rebase -i HEAD~2. I think I messed up something in my first tentative for squashing. Anyways, how do I take that top commit over?

FTR, the canonic way to grab somebody else's tree is https://github.com/scipy/scipy/pull/5859#issuecomment-186613589
In this case though, it's probably easier to merge this directly than to keep wrangling with git.
Which I did: https://github.com/scipy/scipy/commit/426619468ee712bbeffdb20cb2baf570273f541c is the merge commit and https://github.com/scipy/scipy/commit/41054d1f12d09fae681fbe5de3c92ff617953508 lists the actual author for posterity. Thank you @mirca and keep them coming!

@ev-br Thanks!


Update ModelTransformer.php

resolves further problems within issue #9
So really only requested fields will be given in response

@swen100
It works!!

Thanks.


lib: add toposort, nixos: use toposort for fileSystems to properly support bind and move mounts

I hope it's okay now.

Cool, an automatic dependency discovery for mountpoints. So how do you solve this:
mount somedev /foo
mount /foo/image /bar
mount --bind /bar /foo

Actual dependencies:
/foo -> /bar -> /foo/image -> /foo -> foodev

There's a cycle there, I don't think your toposort can solve it.
The problem is: mounting is a problem that cannot be solved simply by automatic dependency discovery. It's just as not expressive as the mounting problem.

I don't. My primary use case is mount binding simple sequences in initrd
(complicated netboot).
But notice that your example also can't be expressed with fileSystems:
fileSystems = {
  "/" = { fsType = "tmpfs"; };
  "/foo" = { device = "/dev/some";
             fsType = "xfs"; };
  "/bar" = { device = "/foo/image";
             fsType = "xfs"; };
  "/foo" = { device = "/bar";
             fstype = "none";
             options = "bind";
           };
};

repeats "/foo".
I can, however, imagine a way to implement this by a kind of explicit
"version" field in fileSystems.* submodule, or, more NixOS-sy,
describing actual mountpoint explicitly and making newer "/foo" receive
higher "version" than old "/foo", e.g.:
fileSystems = {
  "/" = { fsType = "tmpfs"; };
  "/foo-v1" = { mountpoint = "/foo";
                device = "/dev/some";
                fsType = "xfs"; };
  "/bar"    = { device = "/foo/image";
                fsType = "xfs"; };
  "/foo-v2" = { mountpoint = "/foo";
                needs  = [ "/foo-v1" ];
                device = "/bar";
                fstype = "none";
                options = "bind";
              };
};

What you've reminded me is that I didn't think about other types of
cycles which might hang nix-instantiate with current implementation. I
need to check this.

It's just easier to define after and before systemd options in the mount options, instead of trying to be clever.

In initrd?

So what? Both ways you'll have to give the same information to either
this toposort or systemd which will do its own toposort inside. Which
means changing fileSystems in some way or another.

I won't oppose if someone adds proper after and befores to systemd
units in NixOS, it just won't solve my problem in initrd, so I did this
(and applied the same thing to /etc/fstab to solve it for free in the
rest of the system, but I can remove that line if you think that no
solution at all is better).

It's still a broken implementation. Systemd should go in initrd. In the meanwhile you can as well mount stuff yourself with initrd hooks.

No your toposort is an improvement but it's not a solution, and might as well lead to other problems in ordering.

(Edit's don't get mailed, can you please not use them except for typos,
not whole paragraphs?)
My point is that this PR is not broken, because

it solves a problem that can not be solved in any other way except
putting the whole systemd into initrd (I would be happy if someone
does this, I waited long enough, but, AFAIK, that effort is dead),

any other solution is bound to be as "broken" since it's gonna use
the same inputs and algorithms.

Clever discovery of dependencies in this context just doesn't work.

Yes, that's why I don't do it.
Let's be constructive. Show me an example that current fileSystems can
express and toposorting it breaks something.

What you've reminded me is that I didn't think about other types of
cycles which might hang nix-instantiate with current implementation.
I need to check this.

Okay, I had been thinking better than I though I was. listDfs ignores
cycles, so current implementation toposorts cycles as if they start
from arbitrary node.
AFAIK, it doesn't break anything that is not already broken in user's
configuration.nix, but I can make it into an evaluation error. Should
I?

@oxij tnx for the examples, that make it easier to understand.
i'd wait for @lethalman opinion wheather to raise an evaluation error.

I'm in favor of a partial solution that can accurately detect when it doesn't know what to do and throw an evaluation error. Silently doing the wrong thing (i.e., what we have today) always makes me uncomfortable.

I'm still not in favor of this sorry. I'd rather put some manual ordering in fileSystems, before and after, which could apply both to initrd fstab and systemd mounts as well.

I agree with @oxij and @copumpkin. This solution saves our "lives" sometimes. Now we don't have any solution at all. So it's better to have this than to have nothing.

(triage) Any update on systemd in initrd? As far as I understand, we still don't have it.
I think we should decide if this PR is ok to merge (after rebasing) or should be rejected.

I want to have toposort in lib to use set instead of a list in #17822 and to nicely integrate hidepid there. Also I don't think that it would hurt having it for fileSystems, too -- it won't make things worse and may help somebody. At least it's some determined and logical order that we can document and expect people to understand its limitations.
@oxij, do you want to rebase it by yourself?  My point for having it is that it allows to cleanup things more (see #17822) and gives some benefits -- I don't expect systemd in initrd on any short term (I actually have tried to add it and dracut, but got enormous initrd size increase and lots of broken FHS assumptions and dropped the idea).

Thanks for showing some interest! I thought this was dead.
Problems with the implementation presented here:

it doesn't generate errors on encountered cycles (I have a local
version that does that),

fsBefore is, strictly speaking, incorrect (algebraically, UNIX paths
are lists separated via "/", not strings, e.g. "/dev/ab" is not a
parent of "/dev/abc"). I'm pretty sure I can fix that.

Pretty much no interest till now.

Anyway, do you need anything else for sets in #17822 (which is pretty
nice, btw) or the current state rebased onto master suffices?

Problems you list can't really do anything worse that it is now, and it covers my usecase still, so you can just rebase for now. Thanks!

Well, this function uses ++ so it probably has O(n^2) performance. IMHO this kind of processing is better done in the builder or at runtime.

It's probably possible to do this efficiently using builtins.genericClosure. Still, it's better to deal with ordering at runtime (as systemd already does - this is only an issue because we don't have systemd in the initrd).

I agree that doing this in systemd sounds nicer (and would abolish need in #17822, too, because mount options would be used right from systemd), but I don't think we will have it anytime soon ~_^.
Doing this in builder may be an option if we want to reduce computations in Nix, but I think it's a good idea to try and optimize this first (we already have the code, after all). I'm not sure how can genericClosure help but it looks interesting! Unfortunately it's not documented but the source code helps.

Well, this function uses ++ so it probably has O(n^2) performance.

In the worst case, assuming I (++) is lazy.

It's probably possible to do this efficiently using
builtins.genericClosure.

I'll look into that.
In any case, I didn't notice the evaluation time of this implementation
in my local branch for n=4..10. Not to mention the fact that NixOS
already has strings-with-deps.nix which does some crazy (++) stuff (I
can't really understand what it does exactly) that gets evaluated all
the time too.

Still, it's better to deal with ordering at runtime (as systemd
already does - this is only an issue because we don't have systemd in
the initrd).

Personally, I fail to see why one would want to push for
systemd-specific NixOS instead of making NixOS more generic when
possible. E.g. lately I dream of having an an option to use something
simpler in place of systemd, e.g. OpenRC, because every second systemd
update breaks something on at least one of my machines, and policykit
and journald drive my crazy. (And in an ideal world, I'd want to have an
initd that evaluates nix-expressions.)

Fun fact while rebasing: 3d16af70bf894ce15ec9bdcad3c9ac736dc43630 sorts
fileSystems for completely irrelevant reasons so that "/" gets to be the
first one.
I say this shows that they should be topologically sorted in
nix-expressions anyway.

Pushed the reworked branch.
Now it generates pretty assertions, e.g.:
{lib, pkgs, config, ...}: with lib;
{

    boot.loader.grub.device = "/dev/sda";

    fileSystems = {
      "/" = { fsType = "tmpfs"; };
      "/foo" = { device = "/bar";
                 fsType = "none";
                 options = "bind";
               };
      "/bar" = { device = "/foo";
                 fsType = "none";
                 options = "bind";
               };
    };

}

produces:
Failed assertions:
- The ‘fileSystems’ option can't be topologically sorted: mountpoint dependency path /bar -> /foo loops to /bar

Also, I think it should actually speed evaluation up a bit on average,
since I fixed all the things to toposort and extract fileSystems attrset
to list only once, instead of like ten times.
Also cleaned up some copy-paste.
Evaluates ok here.
I hope I didn't break #17784, would be nice if @obadz could check that.

I've revieved an updated patchset and it looks okay to me. I haven't run any tests yet.
@edolstra, do you think we need to optimize this further before a merge?
I'm going to merge in few days if everyone is okay with it, but before I'll conduct several tests with updated #17822 on top of that to ensure things are okay. Also would be nice to get more reviews -- I haven't got a close look at toposort and listDfs, assuming their correctness for now (I'll do that later).

Fun fact while rebasing: 3d16af7 sorts fileSystems for completely irrelevant reasons so that "/" gets to be the first one.

What makes you think the reasons are irrelevant?

@obadz I think what @oxij means is that we already did ad-hoc toposort for #17784 (which is irrelevant to #17822 that also needs it).

AFAI can prove, toposorting should push / to the top of the list in
all the situations I can think of, since everything should be mounted
somewhere under / and fsBefore compares on mountPoints.
Yep, I meant "irrelevant [to the current discussion] reasons" and didn't
even think that out of context that could be read as derogatory. Sorry
about that.

Tested with my own system, verified that / is first and that ordering is correct and also did another review.
@oxij Thanks for your work!

You're welcome. Thanks for making the sane approach prevail :)


Fix result based erros

Things left to implement:-

[ ] Mutate balances accordingly.
[ ] Remove orders from the orderbook mirror.

@ksr30 @haerdib There are many changes required in asset_exchange functions and consume order. I have reviewed it partially. I will try to complete by night


Setup a common Jekyll Author library

that could be used for a common experience across gems like jekyll-feed and jekyll-seo-tag.
I made a working version here. @benbalter already mentioned that it being in ruby and not just liquid it could have some security issues that would need to be reviewed.
I could go through the work of doing it in liquid, but I'd love to get some feedback if possible.

There are no security issues with your library.  Though it would be nice if you did add a few Liquid helpers.

@envygeeks Good idea. I could see it returning the author name as a link to the user's url, and a ul of links for multiple authors. Anything I'm missing there?
Also, I published the gem as jekyll-author.


Create a organization level website

Includes:

Org level website documentation
Points to project level website documentation

Closing. This is a duplicate of https://github.com/oscal-compass/oscal-compass.github.io/issues/1.


Git normalizes end-of-line codes

Git normalizes end-of-line encoding to the operating system that the repo was cloned too.  HSS code is specifically meant to run on Ubuntu Linux, so Windows based developers receive files with incompatible encoding.  Windows = 0x0D 0x0A.  a.k.a. \r\n or ^M^J.  Unix/Linux/Mac OS X = 0x0A (\n or ^J) only.  This prevents Windows devs from provisioning a Vagrant VM server for development.  However, the issue is actually caused by Git, not Vagrant.

End line normalization can be desigated in .gitattributes file.  This is currently missing in the repo, so I will create one.

Problem persists after Merge #291.  I am continuing to investigate.

Merge #295 open,

Merge #295 has corrected the EOL normalization issue with Windows development.  However, Vagrant still failed to provision a development VM.  Here is a snippet examples from the extensive list of errors:
==> default: sed: can't read __CONFIGURE__FILLIN__HSS_INSTALL_DIR__/bin/untrusted_execute.c: No such file or directory

This issue has technically been fixed.  Closing this issue.  Opened #297 to address the additional problem(s) explained in the comment above.


Free AggregatorFactory implementors from boilerplate in getMergingFactory()

Currently AggregatorFactory implementors should write code like the following themselves:
if (other.getName().equals(this.getName()) && other.getClass() == getClass()) {
  ...
} else {
  throw new AggregatorFactoryNotMergeableException(...);
}

or
if (other.getName().equals(this.getName()) && other instanceof MyBaseAggregatorFactory) {
  ...
} else {
  throw new AggregatorFactoryNotMergeableException(...);
}

if MyBaseAggregatorFactory is extended by other aggregator factory classes.
It could be replaced by e. g. by adding a generic parameter BaseMergeAggregatorFactory to AggregatorFactory class; adding a method Class<BaseMergeAggregatorFactory> getBaseMergeAggregatorFactory() method; and changing the signature of getMergeFactory() to accept BaseMergeAggregatorFactory as the parameter.

@leventov getBaseMergeAggregatorFactory() will be an abstract method so it needs to be implemented in all the classes with an empty definition?

@leventov sounds good. I will do the changes.


Standard naming of parameters

We should standardize on a convention, e.g., PSRNAME_SIGNALNAME_PARNAME_SELECTION; or should it be (more logical, but perhaps less readable) PSRNAME_SELECTION_SIGNALNAME_PARNAME?

I think we do have a standard naming convention, PSRNAME_SELECTION_SIGNALNAME_PARNAME. At least that is the case for all the ones I can think of right now but I may be wrong for the common signals.


Upgrade OpenLayers to version 9.1

This upgrades the OpenLayers library to the latest 9.1.0 version. According to the upgrade notes of OL no further code changes seem to be necessary in the Wegue code.
But be aware that your custom application code maybe needs some adaption due to the breaking changes in OpenLayers 9.0.0. Check the link above for details.

Thanks for your review @sronveaux. Since it has been a couple of weeks I also did another test with a bigger Wegue app. Works as expected, so I am going to merge now.


Can OCR api return the text and the language that it detected?

How can we know the language of the image we sent to the OCR api?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: afd2df6e-4c2e-eaf8-1fe2-9bae807f5680
Version Independent ID: d1fb5a5b-a555-4cd0-0330-855681654a38
Content: Quickstart: Read client library or REST API - Azure Cognitive Services
Content Source: articles/cognitive-services/Computer-vision/quickstarts-sdk/client-library.md
Service: cognitive-services
Sub-service: computer-vision
GitHub Login: @PatrickFarley
Microsoft Alias: pafarley

Thanks for the feedback! We are currently investigating and will update you shortly.

Hello,
OCR can not detect the language in your image automatically, you need to use Text Analytics - Language detection to recognize the language : https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/how-tos/text-analytics-how-to-language-detection
So the step should be:
Use OCR/ Read API to recognize the content in your image.
Input your result in step 1 to Text Analytics API for language detection. Then you can get the result of the language and also content.
Regards,
Yutong

We will now proceed to close this thread. If there are further questions regarding this matter, please respond here and @YutongTie-MSFT and we will gladly continue the discussion.


[eks] Upcoming 1.24 release - WIP

Not yet released, but mentioned on the upstream table.
WIP - Let's not merge till 1.24 is released on EKS List: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#available-versions

@captn3m0 lets we also add latest tag incase if we forget when we will merge

@captn3m0 looks like they forget to tag all 1.24 release tags on github because they even release
1.24-eks-3 ( https://distro.eks.amazonaws.com/#releases )

@captn3m0 ping?


Bulk is timing out for more than 50000 Request

Hi,
I am trying to load more than six millions documents in Elastic. I am using NEST bulk method to insert the records. If my batch size (number of documents per request) is greater than 50K, i am http time out exception.  I am using four core processor and using "parallel.for" to load four batch at a given time. Here is the code for bulk insert:
BulkInsertSymbol:
protected bool BulkInsertSymbol(List symList)
{
bool valid = false;
try
{
ElasticClient client = GetClient();
if (client != null)
{
IBulkRequest request = new BulkRequest()
{
Refresh = true,
Consistency = Consistency.One
                };
                request.Operations = new List<IBulkOperation>();
                foreach (Symbol symObj in symList)
                {
                    IBulkCreateOperation<Symbol> symRequest = new BulkCreateOperation<Symbol>(symObj);
                    request.Operations.Add(symRequest);
                }

                IBulkResponse response = client.Bulk(request);

                valid = response.IsValid;
            }

        }
        catch (Exception ex)
        {
            _logger.Error("Exception occured in BulkInsertSymbol method.",ex);

        }
        return valid;
    }

Right now, to load six million documents it would take close to 10 minutes, I would like to improve the performance. Please suggest ways to improve performance.
Mahesh Babu

@maheshbabu7 you can try increasing the request timeout via ConnectionSettings:
new ConnectionSettings().SetTimeout(60000);

In terms of performance, check out: https://www.elastic.co/guide/en/elasticsearch/guide/master/indexing-performance.html
Hope that helps.

Honestly, I think a better solution to this is to decrease the number of documents in each bulk request. It'll be friendlier to memory too.

its working for 50000 as batch. I would like to improve performance, it there any other way to load huge data in Elastic Search.

@maheshbabu7 Hi Mahesh.. Even I'm doing something like this. But i have to get data from SQL db and index them. I tried doing using IndexMany which is working for 30000 documents as of now. I dont know how it behaves when I try to index 6million documents.  I tried using Bulk also..but not sure if im using the right approach of code or not.
                        List<Class1> feedlist = new List<Class1>();
                        while (reader.Read())
                        {
                            client.Bulk(b => b.Index<Class1>(i => i
                            .Document(new Class1
                            {
                                Id = Int32.Parse(reader[0].ToString()),
                                Title = reader[1].ToString(),
                                BodyContent = reader[2].ToString(),
                                ModifiedDt = Convert.ToDateTime(reader[3].ToString()),
                            }
                            )));
                            //feedlist.Add(fd);
                            count++;
                        }

Now I'm trying to index using json file. But not sure how to do it using NEST. I serialized the feedlist to JSON object and i have to index it. All i can find is using curl. But nothing related to NEST.
Can you please help me solving this using BULK method which you have mentioned above.
TIA


bash-env-json: init at 0.9.1

Things done
Added a package for bash-env-json a bash script that interprets bash environment files and exports results to json so that shells like nushell and elvish can import them.
This will allow the creation of a nixos module like programs.fish but for nushell

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Issue Tracker:
https://github.com/tesujimath/bash-env-json/issues/12


I need integrate cryptomus -  Could you helP?

cryptomus - is more convenient for users, who don`t use stripe and bymeacoffee. Almost zero commisions.

hey @joker096, I don't have the bandwidth to help with a cryptomus integration. PRs are welcome but I don't know if st-paywall is a great spot for crypto payments


Up-down scaling behavior configuration is ignored in Kubernetes version >= 18.0

Scaling up-down behavior configuration https://github.com/kedacore/keda/pull/805 is ignored in Kubernetes 19.0 and 18.8 (only 2 versions I checked on).
Here is info from logs
2020-09-21T07:17:45.828Z        INFO    controllers.ScaledObject        Creating a new HPA      {"ScaledObject.Namespace": "default", "ScaledObject.Name": "kafka-receiver-scaledobject", "HPA.Namespace": "default", "HPA.Name": "keda-hpa-kafka-receiver-scaledobject"}
2020-09-21T07:17:45.968Z        INFO    controllers.ScaledObject        Warning: Ignoring scaledObject.spec.behavior, it is only supported on kubernetes version >= 1.18 {"ScaledObject.Namespace": "default", "ScaledObject.Name": "kafka-receiver-scaledobject", "kubernetes.version": "1.19"}
2020-09-21T07:17:45.968Z        INFO    controllers.ScaledObject        Initializing Scaling logic according to ScaledObject Specification      {"ScaledObject.Namespace": "default", "ScaledObject.Name": "kafka-receiver-scaledobject"}

Expected Behavior
Scaling behavior should be functioning
Actual Behavior
It's not. Logs confirm that.
Steps to Reproduce the Problem

Install keda, create ScaledObject with scaledDown, scaleUp advanced config.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-receiver-scaledobject
spec:
  scaleTargetRef:
    name: test
  pollingInterval: 5
  cooldownPeriod:  5
  minReplicaCount: 1
  maxReplicaCount: 4
  advanced:                                          # Optional. Section to specify advanced options
    horizontalPodAutoscalerConfig:                   # Optional. Section to specify HPA related options
      behavior:                                      # Optional. Use to modify HPA's scaling behavior
        scaleDown:
          stabilizationWindowSeconds: 1000
          policies:
          - type: Percent
            value: 1
            periodSeconds: 1000
          - type: Pods
            value: 1
            periodSeconds: 1000
          selectPolicy: Disabled
        scaleUp:
          stabilizationWindowSeconds: 1000
          policies:
          - type: Pods
            value: 1
            periodSeconds: 1000
          policies:
          - type: Percent
            value: 1
            periodSeconds: 1000
          selectPolicy: Disabled
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.default:9090
      metricName: sender-metric-test
      threshold: '10'
      query: sum(irate(spring_integration_send_seconds_count{application_name="kafka-receiver"}[1m])) * 10

Test it, see that it's not functioning, see logs.

Specifications

KEDA Version: 2.0.0 beta
Platform & Version: tested in Minikube
Kubernetes Version: 19.0
Scaler(s): Prometheus

@dkuzmianok-sc Hm, it must be something with Minikube version string, would you mind sharing the beginning of the Operator log? There should be a log message like this:
{"level":"info","ts":1600678755.6548228,"logger":"controllers.ScaledObject","msg":"Running on Kubernetes 1.18+"}

@dkuzmianok-sc there was a small bug in the version resolving, failing on some corner cases. Thanks for finding this! Will send a PR in a moment.

@zroubalik
2020-09-21T07:12:49.093Z INFO controller-runtime.metrics   metrics server is starting to listen    {"addr": ":8080"}
2020-09-21T07:12:49.095Z INFO controllers.ScaledObject     Running on Kubernetes 1.19
2020-09-21T07:12:49.095Z INFO setup   Starting manager
2020-09-21T07:12:49.095Z INFO setup   KEDA Version: v2
2020-09-21T07:12:49.095Z INFO setup   KEDA Commit: 717aab81817f0d9b6d02b0e6d1499367c751faf2
2020-09-21T07:12:49.095Z INFO setup   Go Version: go1.15.1
2020-09-21T07:12:49.095Z INFO  setup   Go OS/Arch: linux/amd64

Here are logs if still needed.
Thank you for quick response.


ViGEmBus Silent install switch not working?

Describe the bug
Attempting to perform silent install does not appear to work.
To Reproduce
Steps to reproduce the behavior (example):

Launching .MSI with:  ViGEmBusSetup_x64.msi /qn
Nothing happens (no prompts, error, etc. - app doesn't install)

Expected behavior
Expecting to have the app silently install but it doesn't.
Screenshots
N/A
System details (please complete the following information):

OS: [e.g. Windows 10 20H2 64bit]
Feeder software: [e.g. ViGEmBusSetup]
Driver Version: [e.g. 1.17.333]

Additional context
Manually installing via .MSI seems to work, just quiet install doesn't.

Turns out it wouldn't install because it needed elevated privileges.   I used this method from elevated command prompt and it installed silently.
msiexec /i C:\Temp\vgpemu\ViGEmBusSetup_x64.msi /qn


Text overlap

When I pop up, there is text overlap

@dennisreimann

Hi guys. Any more details? Screenshot etc.?

I have a same issue~~

same issue

i changed UILabel backgroundColor to fixed issue

我也遇到了这个问题
I also encountered this problem.


Add DNS support for external name resolution

This PR equips the nex node with an embedded nameserver, running on an ephemeral port, which is provided to sandboxed agent VMs via Firecracker MMDS. The agent uses iptables to setup NAT OUTPUT chain rules for sending local DNS packets destined for port 53 (UDP or TCP) to the configured nameserver.

So we weren't able to do this with just configuration? :sob:

Is it possible to check the node configuration and only activate the DNS stuff is sandbox is on? If the node isn't using firecracker then running the embedded DNS is unnecessary

Is it possible to check the node configuration and only activate the DNS stuff is sandbox is on? If the node isn't using firecracker then running the embedded DNS is unnecessary

Yes, and definitely something I wanted to bring up. Will only activate this in sandbox mode.


AIP-6339: Zodiac service per nb

Support zodiac service and team env variables and adds correct cost labels to kfp pods. Please refer to comment in code as to why we add labels here.  This work is for supporting zodiac service per notebooks and the workflows launched from them.
This is not tied to CICD, only individual profile notebooks. CICD does not contain zodiac metadata but may be something we visit in the future with further alignment to zkd.

Looks like some integration tests are failing - you can do a rebase onto feature/kfp to fix them.


tensorflow-1.11.0-cp36-cp36m-linux_aarch64.whl failed.

Successfully installed tensorboard-1.11.0 tensorflow-1.11.0
[toybrick@localhost work]$ python3
Python 3.6.8 (default, Jan 31 2019, 09:06:51)
[GCC 8.2.1 20181215 (Red Hat 8.2.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import tensorflow as tf
Traceback (most recent call last):
File "/home/toybrick/.local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/toybrick/.local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/toybrick/.local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/usr/lib64/python3.6/imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib64/python3.6/imp.py", line 343, in load_dynamic
return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

Make sure your environment is the same as the one below.
Python3.x + TX2 aarch64 (JetPack-L4T-3.3-linux-x64_b39)

L4T R28.2.1（TX2 / TX2i）
L4T R28.2（TX1）
CUDA 9.0
cuDNN 7.1.5
TensorRT 4.0
VisionWorks 1.6

sorry. I misunderstood it.


OwnedElement builder method names are inconsistent

Most of the builder method names describe the Ion element they're constructing:
https://github.com/amzn/ion-rust/blob/76ceb3b0edab4760af78455e10f11dee21932de2/src/value/owned.rs#L194-L214
However, some of the numeric methods are named after the Rust types that they accept:
https://github.com/amzn/ion-rust/blob/76ceb3b0edab4760af78455e10f11dee21932de2/src/value/owned.rs#L174-L180
https://github.com/amzn/ion-rust/blob/76ceb3b0edab4760af78455e10f11dee21932de2/src/value/owned.rs#L190-L192
We should be consistent; the above would be new_float. I suggest we distinguish between input parameter types via a method name suffix: new_int_from_i64, new_int_from_bigint.

Fixed in #479


Added dark mode for the Web UI

Description
Added Dark mode for the web interface.

Added colors variables
Added colors set for dark mode
Added settings toggle "Dark mode"

Related issue https://github.com/mitmproxy/mitmproxy/issues/3886
How it looks you can check in this video:
https://user-images.githubusercontent.com/2677592/196033872-74873f19-ed17-4a88-8968-5191d15c581a.mp4
Checklist

[x] I have updated tests where applicable.
[x] I have added an entry to the CHANGELOG.

Thank you for the PR! I very much appreciate the work you've put into this and the intent to improve mitmproxy. Nonetheless, I'm relatively strongly against merging this. Dark mode adds a substantial amount of maintenance overhead, which is something I don't want to take on. As the primary maintainer of mitmweb, I don't want the additional burden of testing all changes with multiple color schemes and adding dark mode styles for all new UI elements. We have some long-overdue modernization to do on the CSS front (upgrading from Bootstrap 3), and dark modes just adds more work on top of this.
Long story short, I'm happy to have mitmweb dark mode stay around as a user style, and I'd be more than happy to link it prominently in examples/contrib and refer folks to it. However, unless the rest of @mitmproxy/devs comes out as being strongly in favor of this (with some maintenance commitments), I don't think we'll merge this PR. Of course, I'd be more than happy to chat about other potential mitmweb improvements before you start hacking on stuff!

Thank you for your comment! I agree with your point of view - the additional burden in maintaining or testing it is not good, however, in this PR I added an approach with color variables instead of using different color values in the styles. This approach should reduce efforts of maintaining UI in the future, I mean that having a list of the colors is easier than having infinity variants of the colors in the styles. If new UI elements will use colors from the existing set it shouldn't have any impact on maintenance. In case if some new elements will need new colors it is easy to add them to the colors set.

I made a half-hearted stab at implementing dark mode after being put off by lack of it earlier on, but soon abandoned after finding Bootstrap 3 unpleasant to override.
Just want to say, I find your implementation looking good and code clean. Great job on hunting down colours throughout these stylesheets! :+1:

Certainly I agree that Bootstrap 3 needs upgrading from. Though for now going forward with having extracted all colours used in the UI to CSS variables seems more maintainable than before. :thinking:

This approach should reduce efforts of maintaining UI in the future, I mean that having a list of the colors is easier than having infinity variants of the colors in the styles. If new UI elements will use colors from the existing set it shouldn't have any impact on maintenance. In case if some new elements will need new colors it is easy to add them to the colors set.

Thanks - I've looked at the implementation and I'm very familiar with the pattern.^1 First, I don't think you get around doing additional testing in dark mode, so it definitely comes with a constant maintenance cost. Second, dark mode often is not as easy as "adding new colors to the color set". For example, you can't simply change icon colors unless you construct SVG icons in a very specific way or use icon fonts. Finally, upgrading from Bootstrap 3 will be a major rewrite, and an already-existing, conflicting set of color variables just gets in the way.

Just want to say, I find your implementation looking good and code clean.

FWIW I absolutely agree that this PR looks great, and I probably should have pointed that out more prominently in my first response. I like the code, but I don't like the long-term maintenance implications. If anyone with a track record of making constant contributions to the project would want this as their pet feature in I'd be the last one to complain, but I simply don't want to spend my time maintaining on dark mode.
Again, I have nothing against a mitmweb dark mode, but it does not have to live in our main repository, I'd much rather prefer it as a user style maintained somewhere else. :)

Ok, I can move styles to another repository with custom styles, but could you help me with instructions on how custom styles can be added to the existing installation of the mitmproxy?
For example, if I created a repository with custom styles, how other people can use them?
During working on this PR, I thought maybe will be better to have a separate repository for web UI? Probably in this case more people will be involved in working on Ui part. (It is just my thoughts)

Ok, I can move styles to another repository with custom styles, but could you help me with instructions of how custom styles can be applied to the existing installation of the mitmproxy?

I'm afraid we don't have a really good API for this. I was mostly thinking of something like https://userstyles.org/, which works via a browser extension and not a mitmproxy addon.

During working on this PR, I thought maybe will be better to have a separate repository for web UI? Probably in this case more people will be involved in working on Ui part. (It is just my thoughts)

Can you explain how you think that would help attract contributors? I'm genuinely curious. :)

https://github.com/darkreader/darkreader
This extension works fine for me

Can you explain how you think that would help attract contributors? I'm genuinely curious. :)

I mean that a separate repository with only the frontend part will be easier to maintain by people who have experience in frontend development only and it prevents breaking the main project. Maybe in this case more people will add more improvements to UI which is important to make the whole project more popular.

I created a new PR


Cleanup trinity log level setting to include defaults and make them overridable

From: https://github.com/ethereum/py-evm/issues/806#issuecomment-417987842
What is wrong?
We have some defaults that are used to quiet down some noisy loggers:
https://github.com/ethereum/py-evm/blob/v0.2.0-alpha.31/trinity/utils/logging.py#L111-L114
Were the use to supply overrides to these via the command line they would not be respected
How can it be fixed
This lines should be relocated such that they are merged with the user supplied logging levels before being passed into setup_log_levels.
https://github.com/ethereum/py-evm/blob/v0.2.0-alpha.31/trinity/main.py#L111-L112

@pipermerriam just want to align. Do we want [1] to keep those noisy logs at INFO level (i.e as current, less noisy) then overrides it when log level is defined from the command line?
OR
[2] We want to take out the INFO level config (i.e will become more noisy), then override when log level defined in command line?


revise sim interface - use terminal-kit

Please check if the PR fulfills these requirements

[ ] Passes e2e testing

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
Feature

What is the current behavior? (You can also link to an open issue here)
#1324

What is the new behavior (if this is a feature change)?
Changes the sim interface to use 'terminal-kit' instead of inquirer

bump

@rssk bump


[DRAFT] Persistence for background ddl / recoverable backfill

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Closes #8051 .

[x] Commit table to meta store on start_stream_job.
[x] Commit table fragments to meta store on start_stream_job. Should already be done (create_streaming_job_impl calls FragmentManager::start_create_table_fragments).
[x] ~Do the same for sink, index.~ Separate PR.
[x] ~In backfill, when we receive initial barrier, we need to indicate the backfill status, along with the row count.~ https://github.com/risingwavelabs/risingwave/pull/12415
[ ] In meta service, after barriers collected, we should update the row count from the barriers. Currently row count is maintained incrementally in meta store. Instead we should maintain the ENTIRE row count in backfill.
[ ] https://github.com/risingwavelabs/risingwave/pull/12355#pullrequestreview-1630600913
[ ] We need to clean up foreground ddl.
[ ] Remove table from meta store on cancel
[x] Remove table from meta store drop (should already work as per normal?).
[ ] Perhaps creating_streaming_job needs to be partitioned, currently we return create materialized view immediately. We should maybe only return once the actors have been scheduled?
[ ] We also need someway to distinguish between background ddl and foreground ddl. Our Table perhaps needs to store this info, instead of CreateMaterializedViewRequest. We should only persist background ddl, and discard foreground ddl.

In order for hummock to be aware of the table, HummockManager::register_table_fragments should be called. HummockManager::unregister_table_fragments_vec and HummockManager::unregister_table_ids should not be called.
https://github.com/risingwavelabs/risingwave/blob/48d7d08c65ef892b131b9e7dde962ebab6df80e6/src/meta/src/stream/stream_manager.rs#L416
https://github.com/risingwavelabs/risingwave/blob/48d7d08c65ef892b131b9e7dde962ebab6df80e6/src/meta/src/stream/stream_manager.rs#L427
https://github.com/risingwavelabs/risingwave/blob/48d7d08c65ef892b131b9e7dde962ebab6df80e6/src/meta/src/stream/stream_manager.rs#L536
In the current main, HummockManager::unregister_table_ids will be put into revert_funcs and when CREATE MV fails these revert_funs will be called. I think we need to refactor this logic in order to support persistent backfill.

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Able to run the following script and recovery the mview at this point.
./risedev k
./risedev clean-data
RUST_LOG="risingwave_stream=trace,info" ./risedev d full
./risedev psql -f test.sql.log
cp .risingwave/log/compute-node-*.log .risingwave/log/cn-old.log
./risedev k
RUST_LOG="risingwave_stream=trace,info" ./risedev d full
./risedev psql -a

Still needs further work to cleanup failing jobs.

After #12493 we have fixed some bugs for no_shuffle_backfill:

Recovery of progress.
Recovery of state.

So this test works now (the difference is the 3s sleep) where it was previously failing:
./risedev k
./risedev clean-data
RUST_LOG="risingwave_stream=trace,info" ./risedev d full
./risedev psql -c "
CREATE TABLE t1(v1 int);

INSERT INTO t1 SELECT * FROM generate_series(1, 1000000);

SET BACKGROUND_DDL TO TRUE;

CREATE MATERIALIZED VIEW m1 as SELECT * FROM t1;
"
sleep 3
cp .risingwave/log/compute-node-*.log .risingwave/log/cn-old.log
./risedev k
RUST_LOG="risingwave_stream=trace,info" ./risedev d full
./risedev psql -a
# At this point you can use SHOW JOBS
# and once materialized view is finished, you can also query it.

Still needs further work to cleanup failing jobs.

At commit https://github.com/risingwavelabs/risingwave/pull/12167/commits/4a93f0a7d83a6e0053275a0483edfe808f13cba1 we passed ddl as well. Streaming failed however.

https://github.com/risingwavelabs/risingwave/pull/12167/commits/9df0bb89c7707e4463012f959f73e6a8eef1a02c
Failed in streaming tests.

Passed recovery test.

We now have the following matrix the handle cleanup:

cancel_type -> job_type ↓
CN Killed
Meta Killed
CANCEL JOBS
Before job build (e.g. cordoned nodes)

foreground
B->A
D
C->A
A

background (before recovery)
B->A
D
C -> A
A

background (after recovery)
B->A
D
C
A

State cleanup methods
A: Stream Manager cleanup (only tables for the stream job).
B: Barrier Recovery (all dirty fragments).
C: CancelStreamJob cleanup (specific table and fragments).
D: Bootstrap Recovery (all dirty tables and fragments)
We should simplify the cases with overlap in state cleaning, otherwise it's really complicated to maintain.
We also need to verify that in the case of CN killed, what is the expected behaviour of the cluster:

Resume jobs which are already creating and discard hanging table fragments (why will there be hanging table fragments though?).
Or discard ALL jobs which were in creating process.

Can probably write a deterministic recovery integration test for it, and see what's the expected behaviour.

Only codecov is failing.

Good job, I will take a look later.

I'd like to confirm the behavior of today:

If BACKGROUND_DDL = false (default), create MV command will fail once recovery happens before it finishes.
If BACKGROUND_DDL = true, create MV command will also be recovered and continued to run when recovery happens.

Is it the case? cc. @kwannoel

I'd like to confirm the behavior of today:

If BACKGROUND_DDL = false (default), create MV command will fail once recovery happens before it finishes.
If BACKGROUND_DDL = true, create MV command will also be recovered and continued to run when recovery happens.

Is it the case? cc. @kwannoel

Yes. There's a caveat for the second case. If create mv runs, and the recovery happens before first barrier is collected, we will also lose the mv. But this case rarely happens. Besides that the description is accurate.


Corrected the excessive garbage generated by new Axios()

Describe what you would like to know or do
Corrected the excessive garbage generated by new Axios()
Describe the solution you'd considered
	public static Axios post(String url) throws Exception{
		return post(url,null,null);
	}

yep, your solution solves this problem very well, thank you for your support to Axios


Mjml validation

When building mjml templates errors sometime sneak in. The templates are often large/complex, and debugging the root cause can be difficult.
I'd like to pass a template string to a MRML validate method, and have it returns a collection of the errors it has encountered. An output similar to what mjml-cli -v produces would be amazing:
$ mjml -v malformed.mjml
Line 8 of test.mjml (b) — Element b doesn't exist or is not registered

Command line error:
Validation failed

Is this something you'd be open to?

This should work now by doing mrml-cli validate malformed.mjml.


File gap_texture.png is missing.

When training gap task, there is an error showing that file assets/mujoco_textures/gap_texture.png is missing.

just pushed a fix. Could you try again?

Thanks, it works.


⚠ Bofrost.de has degraded performance

In 97fb584, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 2078 ms

Resolved: Bofrost.de performance has improved in 5985439 after 6 minutes.


Here's a new site I'd like you to consider blocking

Here's a new site I'd like you to consider blocking:
www.facebook.com
(please include any other relevant details)

I've tried many times, but the names of the elements change too much.


ci: Configure automated deploys to staging

:tada: This PR is included in version 1.1.0-staging.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


pikvm-rest-api: update revision of submodule

e.g. for UEFI Secure Boot tests we need a fix for UploadImagePiKVM keyword that was introduced in 2de063f810aced5fa90546481e5c68e93e2aed3b revision of pikvm-rest-api repository

Already added here https://github.com/Dasharo/open-source-firmware-validation/pull/118/commits so I will close this


net/http: Dial I/O Timeout even when request context is not canceled

Seems related to https://github.com/golang/go/commit/869e576517f825aecdc8730b0d22f8d6b59bd749 (c.f. https://github.com/kubernetes-sigs/aws-encryption-provider/issues/61).
Maybe this is an expected behavior... Is it possible to see dial timeouts even when the request context has not been canceled? I am using https://golang.org/pkg/net/http/#RoundTripper with dial timeout 30 seconds and request context is 300µs. If this is possible, can you help understand why this happens?
What version of Go are you using (go version)?

$ go version

go version go1.13.6 darwin/amd64

Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/leegyuho/Library/Caches/go-build"
GOENV="/Users/leegyuho/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/leegyuho/go"
GOPRIVATE="*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/y_/_dn293xd5kn7xlg6jvp7jpmxs99pm9/T/go-build128444364=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"strings"
	"time"
)

func main() {
	// this happens about 50% of the time, so try it a few times.
	for i := 0; i < 10; i++ {
		fmt.Println("trying", i)
		try()
	}
}

func try() {
	mux := http.NewServeMux()
	mux.HandleFunc("/test", func(w http.ResponseWriter, req *http.Request) {
		switch req.Method {
		case "GET":
			fmt.Fprint(w, `test`)
		default:
			http.Error(w, "Method Not Allowed", 405)
		}
	})

	ts := httptest.NewServer(mux)
	defer ts.Close()

	u := ts.URL + "/test"

        // default dial timeout is 30-second
        // https://golang.org/pkg/net/http/#RoundTripper
	cli := http.DefaultClient

	timeout := 300 * time.Microsecond
	reqs := 20
	errc := make(chan error, reqs)
	for i := 0; i < reqs; i++ {
		go func() {
			ctx, cancel := context.WithTimeout(context.TODO(), timeout)
			defer cancel()

			req, err := http.NewRequest(http.MethodGet, u, nil)
			if err != nil {
				errc <- err
				return
			}
			_, err = cli.Do(req.WithContext(ctx))
			if err != nil {
				// can be: ctx.Err() == nil && err == "i/o timeout"
				// Q. but how is that possible?
				fmt.Println("Do failed with", err, "/ context error:", ctx.Err())
			}
			errc <- err
		}()
	}

	// "context deadline exceeded" for requests after timeout
	exp := `context deadline`
	for i := 0; i < reqs; i++ {
		err := <-errc
		if err == nil {
			continue
		}
		fmt.Println("error:", err)
		if !strings.Contains(err.Error(), exp) {
			panic(fmt.Sprintf("#%d: expected %q, got %v", i, exp, err))
		}
	}
}

What did you expect to see?
No panic.
What did you see instead?
trying 0
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
error: Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
Do failed with Get http://<IP_ADDRESS>:54098/test: context deadline exceeded / context error: context deadline exceeded
Do failed with Get http://<IP_ADDRESS>:54098/test: context deadline exceeded / context error: context deadline exceeded
Do failed with Get http://<IP_ADDRESS>:54098/test: context deadline exceeded / context error: context deadline exceeded
Do failed with Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout / context error: <nil>
panic: #0: expected "context deadline", got Get http://<IP_ADDRESS>:54098/test: dial tcp <IP_ADDRESS>:54098: i/o timeout

@odeke-em I am aware 300µs is very short, even for local test server. Your code doesn't reproduce my issue.
My confusion is:

whether 300µs timeout context is being used for http client dial call (it seems so)
if that's the case, how is it possible to receive dial tcp i/o timeout even when the context has not been canceled (see where my code panics above)

Btw, thanks for helping on this!

@gyuho you may get more/quicker help by posting to golang-nuts.

whether 300µs timeout context is being used for http client dial call (it seems so)

Yes, it is being passed in for the dial and is documented as controlling the lifetime of the request. If the dial happens fast enough, you'll get a result back, otherwise you'll get an I/O timeout, which I tried to illustrate with the net dialling snippet above, but that happens non-deterministically.

if that's the case, how is it possible to receive dial tcp i/o timeout even when the context has not been canceled (see where my code panics above)

For a context, as I had mentioned there are 2 ways that it can be rendered moot: 1. If the deadline is exceeded, 2. If cancel is invoked. In your case the deadline is what controls how long the operation is expected to be alive for, and in that short duration the dial wasn't completed hence the "dial TCP I/O timeout", when passed into the net.Dialer as per

That context is being passed directly to net.Dialer.DialContext when using http.DefaultClient.

@gyuho you may get more/quicker help by posting to golang-nuts.

@networkimprov while this is true, it rubs off wrong and sounds a bit dismissive of others' efforts. Please try to assume best/good intent and also try to be more productive when replying to answers, as we are working on providing clarity to their answer, with my own time.

Btw, thanks for helping on this!

Glad to be of help as much as I can :)
Thanks!

@odeke-em I think you misread my comment. This appears to be a question, not a bug report, and usually those are referred to the forums so as not to overburden folks like you!

Thank you for the clarification with clearer wording Liam, and my apologies
for taking it the wrong way!
On Mon, Feb 3, 2020 at 3:08 PM Liam<EMAIL_ADDRESS>wrote:

@odeke-em https://github.com/odeke-em I think you misread my comment.
This appears to be a question, not a bug report, and usually those are
referred to the forums so as not to overburden folks like you!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/36848?email_source=notifications&email_token=ABFL3V572MSD2W57UETYXPDRBCPYBA5CNFSM4KMXB7WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKVXVFY#issuecomment-581663383,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABFL3V6MLK5Q6FIL4XSS3MDRBCPYBANCNFSM4KMXB7WA
.

Sorry for jumping on this one here but it seems to me this could highlight some kind of inconsistency.
I have a similar issue when performing multiple GET request, each from a separate go routine, each with a context derived from a parent context with a deadline of 2.5 seconds. If the machine is put under heavy load, some of the requests start failing with a deadline exceeded error, and some with i/o timeout which I find misleading since the Timeout set for the dial is 30 seconds (i am using http.DefaultClient and http.DefaultTransport).
I think it would be more consistent to have a deadline exceeded error in this case since the Dial was interrupted because of the deadline set instead of a timeout on a dial as the error would suggest. Does this make sense?
(Sorry if it doesn't, first time commenting on an issue, please be kind :D)

@giannimassi could you post a working code example for the scenario you describe?

@networkimprov Yes, I should've done it in the first place, thanks for asking.
The following code prints errors that are not deadline exceeded for each request done concurrently as explained in the previous comment.
package main

import (
	"context"
	"fmt"
	"net"
	"net/http"
	"strconv"
	"strings"
	"sync"
	"time"
)

const (
	n       = 1000
	timeout = 10000 * time.Microsecond
)

func main() {
	ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(timeout))
	defer cancel()
	doConcurrentRequests(ctx, n)
}

func doConcurrentRequests(ctx context.Context, n int) {
	// Explicitly configure Dial Timeout to 30 seconds
	// (this is not necessary since the default client already has this configuration)
	transport := http.DefaultTransport
	transport.(*http.Transport).DialContext = (&net.Dialer{
		Timeout:   30 * time.Second,
		KeepAlive: 30 * time.Second,
		DualStack: true,
	}).DialContext
	client := http.DefaultClient
	client.Transport = transport

	wg := sync.WaitGroup{}
	for i := 0; i < n; i++ {
		wg.Add(1)
		go func(id int) {
			defer wg.Done()
			now := time.Now()
			err := doRequest(client, ctx)
			if err != nil && !strings.Contains(err.Error(), "deadline") {
				fmt.Println(strconv.Itoa(id) + " - Request took:" + time.Since(now).String() + " " + err.Error())
			}
		}(i)
	}
	wg.Wait()
}

func doRequest(client *http.Client, ctx context.Context) error {
	req, err := http.NewRequest("GET", "http://example.com", nil)
	if err != nil {
		return fmt.Errorf("while making new request: %w", err)
	}

	_, err = client.Do(req.WithContext(ctx))
	if err != nil {
		return fmt.Errorf("while executing request: %w", err)
	}
	return nil
}

(Btw I know this is far from efficient code 😄 )

@arl makes sense. Might be some syscall return value interpretation issue. It could also be the fact that the behaviour requires different parameters, with the same parameters the behaviour changes a lot on my machine.
For the record this is an example output with go version go1.13.8 darwin/amd64:
706 - Request took:8.420581ms while executing request: Get http://example.com: dial tcp [2606:2800:220:1:248:1893:25c8:1946]:80: i/o timeout
76 - Request took:8.604937ms while executing request: Get http://example.com: dial tcp [2606:2800:220:1:248:1893:25c8:1946]:80: i/o timeout

I'm hoping to find some time to investigate this a bit more this weekend, and maybe make my first attempt at a fix in the standard library, although it doesn't look like the simplest piece of code (hints/help/feedback appreciated @networkimprov @cagedmantis).

Can this be selinux related? I see "i/o timeout" instead of expected context deadline when running on selinux enabled centos 8. On my local machine i get deadline exceeded.

Can this be selinux related? I see "i/o timeout" instead of expected context deadline when running on selinux enabled centos 8. On my local machine i get deadline exceeded.

cc @bradfitz @neild @minux

cc @bradfitz @neild @minux

any update?


Dialog backdrop is persisted when initialized as open but then immediately closed

This should not be a duplicate of the transition issue as I'm not using a Transition, this is a default Dialog component (reproduced below using the Component API Demo page).
Essentially I think flipping the open prop too quickly results in the backdrop being persisted. I know ideally I should get it to initialize with the right value, but the prop is driven off of Redux and initially I won't have the updated state until some other components that handle stuff are run (this is auth related, so validating tokens and stuff all happens outside this component).

[x] This is a v1.x issue.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Expected Behavior
When dialog is mounted initially with open true and then immediately reset back to false, backdrop should not be shown or block UI.
Current Behavior
When Dialog is initially mounted as open={true} but then immediately set to false (in my case, due to getDerivedStateFromProps that flips it immediately due to business logic), the UI backdrop is still being persisted and blocks the UI.
Steps to Reproduce
I changed the demo file to start with open set to true and then on mount, set it to false to emulate the situation I'm in.
Link: https://codesandbox.io/s/14w19w1px4

Start app
Try to click somewhere like the button

Context

Your Environment

Tech
Version

Material-UI
v3.0.2

React
16.3.2

Browser
Chrome 68

TypeScript
Yes (no in demo)

etc.

After you set open to false it does not set exited property of modal to true. So hidden class does not applied and dialog covers all area. Am I right? @kamranayub
I think that handleExit could not be called by a reason that i don't know after getDerivedStateFromProps.
I could be related with getHasTransition method.
Now i have a bigger problem: My daughter wants play with me:). So maybe later i will debug it.
@oliviertassinari: Please verify us , are we in right way?

As far as I could look into the issue, it's not related to the backdrop. You can remove it and still experience the problem. The onExited hook of react-transition-group isn't called.

Setting disablePortal on the Portal solves the problem
Using a div over a Portal sovles the problem
Using NoSsr over the Portal maintaine the issue.

So, we have it, it's a rendering order issue. We need the transition to render once wit open true to get the onExited called when open is set to false.

@mbrn are you working on this?

@nathanmarks i will.

Thanks for your help!
On Mon, Sep 10, 2018, 22:54 Mehmet Baran<EMAIL_ADDRESS>wrote:

@nathanmarks https://github.com/nathanmarks i will.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mui-org/material-ui/issues/12831#issuecomment-420138067,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAiaa_VdQUCG2r2rI72pRSFOHBhS03x2ks5uZzPhgaJpZM4WhzCx
.

@oliviertassinari You are right. When i change disablePortal props of Modal to true, the problem is being solved. Should i send disablePortal:true to Modal from Dialog or make Modal's default disablePortal value true?
What do you offer? @nathanmarks @oliviertassinari

Should i send disablePortal:true to Modal from Dialog or make Modal's default disablePortal value true?

Please don't. It was just to identify what's going on. If few people are affected by this issue. @kamranayub and others can stick to this option. But if we can find a simple fix, not something that requires 50 LOCs, then yes, why not fixing it :). We could explore the onEnter hook or thedialogRef existence.

ok @oliviertassinari. @kamranayub you can set disablePortal={true} to solve your problem.

I'll try it out today.
On Tue, Sep 11, 2018, 01:26 Mehmet Baran<EMAIL_ADDRESS>wrote:

ok @oliviertassinari https://github.com/oliviertassinari. @kamranayub
https://github.com/kamranayub you can set disablePortal={true} to solve
your problem.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mui-org/material-ui/issues/12831#issuecomment-420161349,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAiaa3cEfgSzHmwM5wEXiYUJQ5aP0Re1ks5uZ1emgaJpZM4WhzCx
.

I actually managed to remove the need for a dialog in my use case so I'm no longer affected by this. I can close and if someone else finds a need for a fix, can re-open perhaps.

Re-opening this since it's a legitimate bug

Sure, no problem. disablePortal={true} does indeed work.

I noticed this as well when using a transition. It works fine without, but as soon as I add a slide down transition, if I open, then close, the backdrop isn't removed.

Just adding that I am still seeing this issue. The behavior is very unexpected...thereby cost me many minutes to debug. My problem was only solved when I deleted the transition entrirely. Setting disablePortal={true}, did not solve the issue - DOM still blocked by invisible modal.
When I removed the transition my problem was solved. Not too crazy about the transition so no problem for now, but I copied it from docs for Full Screen Dialog and that is the transition which I had to delete to get the expected behavior. So maybe update the full screen dialog example at least so it works.

I have just encountered this issue after upgrading from v5.10.0 to v5.11.10.
My issue is similar to @kamranayub, I am not using transitions, but have a situation where the dialog open state is immediately changing from true to false after the app launches.
I agree with @megphillips91 that this issue should perhaps be reopened. Although, I was able to solve my issue using the suggested disablePortal={true} fix, this is not described in the MUI documentation for Dialog components, and the reason this issue is happening is not clear/obvious.


Fix CI failures

CI seems to be failing due to QCPortal issues. This PR attempts to fix this.

Codecov Report

Merging #99 (fccbae6) into master (504c167) will not change coverage.
The diff coverage is n/a.

Closing, since no longer relevant!


diverging state of master and devel branches

I am somewhat confused why master and devel branch have diverged so much. As someone new to the codebase I can't even be sure to what exactly the differences are.
From what I've seen the master branch takes hermit-sys from crates.io, and this makes it hard to keep devel and master in sync. What are the benefits in downloading hermit-sys from crates.io versus using hermit-sys with the lib-hermit submodule at a fixed commit?

Hm, it is confusing. For external user is the crate.io version easier to use. They have not to add libhermit as submodule. In addition, if they use libhermit as submodule, they are not able to publish an application at crates.io.

I think it might be better to point an external user to a specific repository that provides a demo app using crates.io. That's basically what https://github.com/hermitcore/rusty-demo/ was meant for, wasn't it?
Regarding your second point, isn't Multiple locations for dependencies a solution for that problem?

Today, the branch devel isn't longer used. See #28


[SPARK-14363] Fix executor OOM due to memory leak in the Sorter

What changes were proposed in this pull request?
Fix memory leak in the Sorter. When the UnsafeExternalSorter spills the data to disk, it does not free up the underlying pointer array. As a result, we see a lot of executor OOM and also memory under utilization.
This is a regression partially introduced in PR https://github.com/apache/spark/pull/9241
How was this patch tested?
Tested by running a job and observed around 30% speedup after this change.

Can one of the admins verify this patch?

cc @davies

ok to test

Test build #55544 has started for PR 12285 at commit 707c9bc.

Test build #55544 has finished for PR 12285 at commit 707c9bc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55544/
Test FAILed.

Merged build finished. Test FAILed.

Test build #55561 has started for PR 12285 at commit c318a35.

@andrewor14 - Thanks for taking a look. Handled the test case failures.

Test build #55561 has finished for PR 12285 at commit c318a35.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55561/
Test FAILed.

Merged build finished. Test FAILed.

@andrewor14 - Seems like some transient Jenkins failure. Can we rerun the test?

@davies - Thanks for looking into it. I agree with you that its not a memory leak because that memory may be used later. However, not reducing the pointer array size to the initial size in case of spill is causing heavy memory underutilization because the tasks are not able to get sufficient memory for the storing the records and this situation often lead to the executor OOM.  Also, I don't see any reason why would we want to keep the bloated pointer array if we are spilling all data to disk and not have anything to store in the pointer array. This change is restoring the behavior of the sorter before the PR #9241 in https://github.com/apache/spark/pull/9241/files#diff-3eedc75de4787b842477138d8cc7f150L321.
The physical plan looks something like this -
== Physical Plan ==
SortBasedAggregate(key=[shard_id#7L,id#11L,target#9,target_id#12L], functions=[(hiveudaffunction(HiveFunctionWrapper(UDAFCollectMap@270df931),feature_id#10,feature_value#13,false,0,0),mode=Complete,isDistinct=false)], output=[shard_id#7L,id#11L,target#9,target_id#12L,feature_map#14])
+- ConvertToSafe
   +- Sort [shard_id#7L ASC,id#11L ASC,target#9 ASC,target_id#12L ASC], false, 0
      +- TungstenExchange hashpartitioning(shard_id#7L,321), None
         +- Project [(id#11L % 321) AS shard_id#7L,id#11L,target#9,target_id#12L,feature_id#10,feature_value#13]
            +- Filter ((((id#11L > 0) && (target_id#12L > 0)) && NOT (id#11L = target_id#12L)) && (cast(feature_value#13 as double) > 0.001))
               +- HiveTableScan [id#11L,feature_value#13,feature_id#10,target#9,target_id#12L], MetastoreRelation x, y, None, [(ds#8 = 2016-03-20),target#9 IN (1,2,3),feature_id#10 INSET (1, 2)]

In your case, inside sorting, the key has 4 columns, the row has 6 columns, so each pair will need about 90 bytes, the array used by sort needs 16 bytes, so the memory used by array should be 15% of all memory used by execution. In worse case, free the array finally could waste about 15% of the memory, how can it make that big difference?
If your data set is huge, which requires spill multiple times, the size of spilled data should be similar each time, so the required array should be similar. If we free that finally, we don't need to grow the array in the middle or two spills (the grow require 50% more memory for array), that's the reason I changed to free the array finally.
The reason your job will OOM is that  the memory used by Hive UDAF UDAFCollectMap is not managed by Spark, the better solution could be reduce the memory faction for Spark to leave more memory for UDAFCollectMap. After this patch, you may still see OOM, if UDAFCollectMap use even more memory.
I agreed that the current patch is good (try to free memory as much as it can). Just try to understand more, please correct me if something is wrong.

@davies Thanks for the explanation, your calculation makes sense. You are right that freeing the array can only make a difference of 15% in ideal case. But what we are experiencing is something different.
Consider the following scenario - We have total shuffle memory of 10G available for 5 tasks. So in ideal situation, each tasks should be assigned 2G of shuffle memory each. And out of those 2G, 300MB should be allocated to the pointer array and rest for storing the records. Now lets say 3 of the tasks finish at the same time and before the driver could run additional tasks on the executor, rest 2 running task aggressively expend their memory and take upto 5G of shuffle memory on the executor, resulting in the pointer array size of around 750MB. Now when the driver runs additional 3 tasks on the executor, previous 2 tasks will be forced to spill, but the pointer array of size 750MB is never freed. This will result in heavy underutilization of memory for the task and in cases where the pointer array actually grew more than the fair share memory of the task, it will result in executor OOM and causing all other tasks on the executor to die.
The job we are running is processing a huge data set of size more than 50TB and we were seeing more than 5% task failure due to OOM. After this fix, we are the failure rate has come down to less than 0.01% and we gain a massive 30% cpu speedup.

Test build #2777 has started for PR 12285 at commit c318a35.

That make sense, thanks for the explanation.

Test build #2777 has finished for PR 12285 at commit c318a35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #55626 has started for PR 12285 at commit 75a44f9.

@davies - Thanks for the review. I have addressed all the comments, please let me know how it looks.

Test build #55626 has finished for PR 12285 at commit 75a44f9.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55626/
Test FAILed.

Test build #55627 has started for PR 12285 at commit b102c25.

Test build #55627 has finished for PR 12285 at commit b102c25.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55627/
Test PASSed.

@sitalkedia Sorry for the trouble.

@davies - no issues, I will change it back.

Test build #55646 has started for PR 12285 at commit d89adf8.

LGTM, will merging once passing the tests. Thanks for working on it.

Thanks a lot for your quick review and response :).

Test build #55646 has finished for PR 12285 at commit d89adf8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55646/
Test PASSed.

Merged build finished. Test PASSed.

Merged into master and 1.6 branch (fixed the conflicts)

Sorry to camp this old issue.
With spark 1.6.1, I've a lot of executor OOM when I do a rdd.saveAsHadoopFile, with parameter classOf[GzipCodec], to gzip the outpout files.
Without the GzipCodec parameter it works fine.
Do you think the root cause of my issue is also this memory leak in the Sorter?
Thanks a lot for your help.