Increase automation of release job

What problem are you facing?
At the moment, the release process for metal-cli is fairly manual:

Look at the <previous> release version
Look at the commits to main since the previous release
Decide on the <next> release version
Create a tag on main in the format v<next>
Push the tag to GitHub if it was created locally
Tag creation triggers a GitHub workflow that runs goreleaser

Now that we are ensuring that the commit log has Conventional Commits tags, we could move all of these steps into the release job so that releasing this CLI is just a matter of pushing a button and letting GitHub Actions handle all of the details.

Similar work has been done for other projects where we use Conventional Commits:

metal-go
metal-java

In both of those cases, we used https://github.com/cycjimmy/semantic-release-action to drive the release process; that requires adding a .releaserc.json to the project root that configures the necessary release steps.
In this repo, we use goreleaser, which builds the CLI and creates the GitHub release.  We may need some investigation to see how to integrate semantic-release-action with goreleaser.


Codetidy/nick/optional parameters should not be used

Replaced optional parameters where possible with overloading methods. Please check for functional changes

I'm checking with Dan whether he would like to review this one before I dive in. There are some API packages that I consider to be owned by Dan - Meraki.Api being the most obvious example - and I'm not sure if this one might possibly fall into the same category.

Dan does not feel a need to do the code review, so I am able to crack on and do this myself.
I should liaise with Ryan, who is the primary user of this package. We should probably create a new version of the package and then have Ryan test that his consuming app builds and works as expected.

All done, thanks. We'll come back to the nullability warnings at a later date


1001 raster unitest

This pull request needs to be merged after PR #989 is merged.

Can one of the admins verify this patch?

@pkdash this is ready for merge.

@gantian127 I have couple of additional comments. Once you address those I can +1.

@pkdash please review the code update and let me know if I need to make more edits. Thanks~

+1


NIFI-12725 Update json-schema-validator to 1.3.0

Summary
NIFI-12725
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[X] Apache NiFi Jira issue created

Pull Request Tracking

[X] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[X] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[X] Pull Request based on current revision of the main branch
[X] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

The changelog for version 1.2.0 indicates breaking changes, same for 1.3.0. The json-schema-validator does not follow semantic versioning, so this upgrade will require code changes. I am closing this pull request for now, pending further changes necessary to adapt to API changes.


Updated to zend-http 2.7

Coverage remained the same at 87.933% when pulling 3871d0ecc0508f18d66599a6e314ed60de47cf6e on webimpress:zend-http into 62fa66fe5403d1d085975a76f94e9b49bdfa2dce on zendframework:develop.

I've bumped it to 2.8.1 during merge, and it will go into version 3.2.0.


Add spec covering case where no user email returned from OAuth

Originally created by Tom Smyth at 2019-05-01 20:53, migrated from redmine-#9653

Version changed from 6.6 to 6.5

By Tom Smyth at 2019-05-02 01:02

Status changed from Resolved to Closed

By Tom Smyth at 2019-05-02 13:11

Tracker changed from Training to Testing

Status changed from New to Resolved

By Tom Smyth at 2019-05-02 01:09

Tracker changed from Testing to Feature

By Tom Smyth at 2019-06-07 03:16


Loosen L-PDFT testing for water

They were a bit too tight testing requirements for water.
CI/CD is still occurring on this branch on my fork, will comment once it passes.

Passes

Sorry I didn't see this before my PR was merged in origin. I loosened some of these but not all of them. The CI/CD passes on origin so I'm closing this for now, but maybe you can push this to origin yourself if it's still necessary?


Add a centralized Save Button that can save multiple things (eg forms) in a SubmissionBoundary

This introduces a SubmissionBoundary (context) that can contain multiple areas that get saved together using a SubmissionBoundarySaveButton.
Any component can register itself in the SubmissionBoundary by providing a hasChanges boolean and a doSave method. This also adds a default implementation for FinalForm:

if FinalForm is rendered inside a SubmissionBoundary, it registers
else the FinalForm renders (like previously) a router Prompt

Prompts are not needed for the inner components, instead SubmissionBoundary renders a single prompt containing all hasChanges from registered components.
See products admin for an implementation example.
Also changed in Demo Products: ProductsPage now contains Toolbar and Save Button for Detail Page. That way the form is now completely independent of where it is used and could also be put into an EditDialog.
Alternative:
A poosible alternative would be to re-use the router Prompt component that also has a save action. But I decided for an additional implementation because:

PromptHandler stores dirty in a ref instead of state, so enabling/disabling save button is not possible (could be changed to state though)
Prompt has no clear edges, as a new SubmissionBoundary has

see #1449 for additional usage of SubmissionBoundary

I've had an idea about naming: How about SaveBoundary or SubmissionBoundary? Would be in line with ErrorBoundary

@nsams please fix the lint workflow.


Remove the patched nixpkgs-vte-gtk4 flake input in favour of nixpkgs when its vte gets gtk4 support

This flake currently uses a fork of nixpkgs nixos-unstable branch that adds gtk4 support to vte by pointing it at GNOME's master gitlab branch.
Once GNOME puts out a new release for vte, this fork should switch to the new release and the gtk4 flag should get upstreamed into nixpkgs' vte package.
See the fork diff here:
https://github.com/NixOS/nixpkgs/compare/nixos-unstable...mitchmindtree:vte-master-gtk4?expand=1

This commit removes the nixpkgs fork in favour of copy-pasting the vte module in as a vte-gtk4 module. At least for now this ensures all packages share the same set of nixpkgs dependencies. Leaving this open though until we can switch away from this custom module to the upstream vte instance.


Run zone guarded?

Would it be right to run my app in a zone and track exception stack trace like so?
void main() async {
  runZonedGuarded<Future<void>>(() async {
    final binding = WidgetsFlutterBinding.ensureInitialized();
    AppCenter.start(secret: "");
    FlutterError.onError = (final details) async {
      await AppCenterCrashes.trackException(
        message: details.exception.toString(),
        type: details.exception.runtimeType,
        stackTrace: details.stack,
      );
    };
    runApp(App());
  }, (Object error, StackTrace stack) {
    if (kReleaseMode) {
      AppCenterCrashes.trackException(message: error.toString(), stackTrace: stack);
    }
  });
}

Thank you, @markst, for your question,
I don't know.
You can try with generateTestCrash(), if you get the error in AppCenter (it may take some time).
When I look at the example code in very_good_core, FlutterError.onError is not in runZonedGuarded.

From this post, it seems to be different. One is for synchronous and one for asynchronous exceptions.
https://github.com/flutter/flutter/issues/97516#issuecomment-1025754820

Thank you, @frederikstonge, for the details.
I hope his answer helps you @markst. If it doesn't, you can reopen the issue.


	Ruby installation found @
	/usr/bin/ruby

	Checking Home Brew installation
	Home Brew installation found @
	/usr/local/bin/brew

	Checking dependencies.
	Checking automake ... 		 automake found
	Checking Libtool ... 		 Libtool found
	Checking OpenSSL ... 		 OpenSSL found
	Checking llvm ... 		 llvm found
	Checking wget ... 		 wget found
	Checking CMake ... 		 CMake found
	Checking GMP ... 		 GMP found
	Checking gettext ... 		 gettext found
	Checking MongoDB ... 		 MongoDB found
	Checking Doxygen ... 		 Doxygen found
	Checking Graphviz ... 		 Graphviz found
	Checking LCOV ... 		 LCOV found
	Checking Python3 ... 		 Python3 found

	No required Home Brew dependencies to install.

	Checking boost library installation.
	Boost 1.67.0 found at /usr/local.

	Checking MongoDB C++ driver installation.
	Mongo C++ driver found at /usr/local/lib/libmongocxx-static.a.

	Checking secp256k1-zkp installation.
	secp256k1 found at /usr/local/lib/.

	Checking LLVM with WASM support.
	WASM found at /usr/local/wasm/bin/.

>>>>>>>> ALL dependencies sucessfully found or installed . Installing EOSIO

>>>>>>>> CMAKE_BUILD_TYPE=Release
>>>>>>>> ENABLE_COVERAGE_TESTING=false
>>>>>>>> DOXYGEN=false

-- The C compiler identification is Clang 4.0.1
-- The CXX compiler identification is Clang 4.0.1
-- Check for working C compiler: /usr/local/bin/clang
-- Check for working C compiler: /usr/local/bin/clang -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/local/bin/clang++
-- Check for working CXX compiler: /usr/local/bin/clang++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Secp256k1: /usr/local/lib/libsecp256k1.a
-- Using custom FindBoost.cmake
-- Boost version: 1.67.0
-- Found the following Boost libraries:
--   thread
--   date_time
--   filesystem
--   system
--   program_options
--   signals
--   serialization
--   chrono
--   unit_test_framework
--   context
--   locale
--   iostreams
-- Configuring EOSIO on OS X
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
-- binaryen building with -std=c++11
-- binaryen building for platform x86-64
-- binaryen building with -Wall
-- binaryen building with -Werror
-- binaryen building with -Wextra
-- binaryen building with -Wno-unused-parameter
-- binaryen building with -fno-omit-frame-pointer
-- binaryen building with -fPIC
-- binaryen building with -O2
-- binaryen building with -UNDEBUG
-- Using 'SYS' as CORE symbol name
-- Using 'EOS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV' as public key for 'eosio' account
-- Found WASM: /usr/local/wasm/bin/clang
-- Using WASM clang => /usr/local/wasm/bin/clang
-- Using WASM llc => /usr/local/wasm/bin/llc
-- Using WASM llvm-link => /usr/local/wasm/bin/llvm-link
-- Found Secp256k1: /usr/local/lib/libsecp256k1.a
-- Found GMP: /usr/local/lib/libgmp.a
-- Configuring fc to build on Unix/Apple
-- Found OpenSSL: /usr/local/opt/openssl/lib/libcrypto.a (found version "1.0.2o")
-- Found ZLIB: /usr/lib/libz.dylib (found version "1.2.11")
-- zlib found
-- Configuring Builtins
-- Configuring SoftFloat
-- Using custom FindBoost.cmake
-- Boost version: 1.67.0
-- Found the following Boost libraries:
--   thread
--   date_time
--   filesystem
--   system
--   chrono
--   unit_test_framework
--   locale
-- Configuring ChainBase on OS X
-- Using custom FindBoost.cmake
-- Boost version: 1.67.0
-- Found the following Boost libraries:
--   thread
--   date_time
--   filesystem
--   system
--   chrono
--   program_options
--   unit_test_framework
--   locale
-- Configuring AppBase on OS X
-- Found Git: /usr/bin/git (found version "2.15.2 (Apple Git-101.1)")
-- Found bsoncxx headers: /usr/local/include/bsoncxx/v_noabi
-- Found bsoncxx library: /usr/local/lib/libbsoncxx-static.a;/usr/local/lib/libbson-static-1.0.a;/usr/lib/libm.dylib
-- Found mongocxx headers: /usr/local/include/mongocxx/v_noabi;/usr/local/include/bsoncxx/v_noabi
-- Found mongocxx library: /usr/local/lib/libmongocxx-static.a;/usr/local/lib/libbsoncxx-static.a;/usr/local/lib/libbson-static-1.0.a;/usr/lib/libm.dylib;/usr/local/lib/libmongoc-static-1.0.a;/usr/local/lib/libbson-static-1.0.a;/usr/lib/libm.dylib;/usr/lib/libsasl2.dylib;-framework CoreFoundation;-framework Security;/usr/lib/libz.dylib;resolv
-- Git commit revision: bf28f8bb
-- Git commit revision: bf28f8bb
-- Found Intl: /usr/local/include
-- Git commit revision: bf28f8bb
-- Found Doxygen: /usr/local/bin/doxygen (found version "1.8.14") found components:  doxygen dot
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
cocoa_framework
    linked by target "wallet_plugin" in directory /Users/info/_sites/eos/plugins/wallet_plugin
corefoundation_framework
    linked by target "wallet_plugin" in directory /Users/info/_sites/eos/plugins/wallet_plugin
localauthentication_framework
    linked by target "wallet_plugin" in directory /Users/info/_sites/eos/plugins/wallet_plugin
security_framework
    linked by target "wallet_plugin" in directory /Users/info/_sites/eos/plugins/wallet_plugin

-- Configuring incomplete, errors occurred!
See also "/Users/info/_sites/eos/build/CMakeFiles/CMakeOutput.log".

	>>>>>>>>>>>>>>>>>>>> CMAKE building EOSIO has exited with the above error.

xcode-select can be found.
xcode installed
all frameworks can be found here: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/
i've even tried to add this path to $PATH
but build script still failing.

@jgiszczak , this seems related to https://github.com/EOSIO/eos/pull/4244#commitcomment-30068178
@Vasiliy-Bondarenko , just a guess: can you try to cleanup everything (rm -rf build, tmp files etc.) or to go for a fresh clone/build?

rm -rf build - done.
tmp files - did not find any.
it is a fresh clone/build. i did not build it on this machine before.
same result.

Are you manually installing or using the .sh script? Our configs are identical according to your output. Here are my brew versions if that helps

 norsegaud@norsegaudnet  ~  ll /Users/norsegaud/Library/Caches/Homebrew                                                                                 1 ↵  2012  17:09:14
total 1388568
drwxr-xr-x  4 norsegaud  staff   128B Jun 11 22:26 Cask
-rw-r--r--  1 norsegaud  staff   871K Sep 14  2017 autoconf-2.69.high_sierra.bottle.4.tar.gz
-rw-r--r--  1 norsegaud  staff   943K Mar 11 19:32 automake-1.16.1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff    87M Apr 18 11:30 boost-1.67.0_1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff    12M Jul 17 11:56 cmake-3.12.0.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff    75K Sep 18  2017 cpptest-1.1.2.high_sierra.bottle.1.tar.gz
-rw-r--r--  1 norsegaud  staff   4.0M Dec 28  2017 doxygen-1.8.14.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   1.2M May  3 06:33 fontconfig-2.13.0.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   877K May  2 07:45 freetype-2.9.1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   290K Sep 15  2017 gd-2.2.5.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   190K Jun 28 06:54 gdbm-1.16.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   7.8M Sep 14  2017 gettext-<IP_ADDRESS>.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   1.0M Mar 19 14:02 gmp-6.1.2_2.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   6.9M Sep 15  2017 graphviz-2.40.1.high_sierra.bottle.1.tar.gz
-rw-r--r--  1 norsegaud  staff    25M Jun 21 22:15 icu4c-62.1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   298K Feb 26 04:11 jpeg-9c.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   144K Sep 23  2017 lcov-1.13.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff    97K Sep 14  2017 libffi-3.2.1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   217K May 18 06:25 libidn2-2.0.5.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   445K Jul 25 08:20 libpng-1.6.35.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   1.0M Jul  2 03:33 libtiff-4.0.9_4.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   1.0M Sep 14  2017 libtool-2.4.6_1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   1.4M May 25 13:21 libunistring-0.9.10.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   100K Sep 14  2017 libyaml-0.1.7.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   204K Jul 31 07:46 linkage.db
-rw-r--r--  1 norsegaud  staff   319M Sep 17  2017<EMAIL_ADDRESS>-rw-r--r--  1 norsegaud  staff   5.7M Mar 27 09:07 media-info-18.03.1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff    85M Jun 28 12:21 mongodb-4.0.0.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff    16M Jul  4 23:08 node-10.6.0.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   3.7M Mar 28 07:30 openssl-1.0.2o_1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   3.7M Jun 17 04:02 openssl-1.0.2o_2.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   236K Sep 14  2017 pkg-config-0.29.2.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff    11M Mar  2 08:14 postgresql-10.3.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff    22M Jul 23 18:22 python-3.7.0.high_sierra.bottle.1.tar.gz
-rw-r--r--  1 norsegaud  staff    18M Jul 23 18:22<EMAIL_ADDRESS>-rw-r--r--  1 norsegaud  staff   495K Sep 14  2017 readline-7.0.3_1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   494K Jul  5 12:21 readline-7.0.5.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   8.0M Mar 28 20:54 ruby-2.5.1.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   1.7M Jun  4 21:17 sqlite-3.24.0.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff    94K Dec 19  2017 telnet-54.50.1.high_sierra.bottle.1.tar.gz
-rw-r--r--  1 norsegaud  staff    75K Feb  7  2018 uriparser-0.8.5.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   835K Apr 21 03:04 webp-1.0.0.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   1.3M May  6 23:52 wget-1.19.5.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   373K May  1 07:07 xz-5.2.4.high_sierra.bottle.tar.gz
-rw-r--r--  1 norsegaud  staff   923K May 23 17:02 yarn-1.7.0.tar.gz

Are you using eosio_build.sh and then the install.sh?
I’m almost 100% certain this is a problem with something you have installed separate from EOS and it’s requirements. Create a new user account on your Mac and try the build/install from scratch

latest versions include an eosio_install.sh script, so you go with:
./eosio_build.sh
sudo ./eosio_install.sh

@NorseGaud created new user, same story. what kind of requirements are you talking about except the one, that script installs itself? is there some list i need to go through?
i'm just doing
git clone https://github.com/EOSIO/eos --recursive
cd eos
./eosio_build.sh

am i missing something?
@lazaridiscom i don't have problems installing, i can't compile.

@Vasiliy-Bondarenko , my last comment complemented @NorseGaud "build/install" comment.

@lazaridiscom Of course the comment on #4244 seems related—it's the same person with the same problem report.
For once, Homebrew is not at fault.  There is something wrong with the XCode and framework installation.  CMake's find_library() function is used by the wallet plugin to determine the linking flags required for its build.  Check permissions, check that the specific required frameworks really are present, check the filesystem for corruption.  If none of those things help, file a problem report with CMake.  The find_library() function works on any normal, healthy system.
Failing all of that, VirtualBox with a Linux installation has been reported to work by another Mac user with a permanently sick system.

@jgiszczak

Of course the comment on #4244 seems related—it's the same person with the same problem report.

Yes, an its standard-procedure to interlink related issues/PR's (which I've done by mentioning the PR).

Failing all of that, VirtualBox with a Linux installation has been reported to work by another Mac user with a permanently sick system.

This does not solve this issue.
This issue should stay open, until the team has find a way to help this user, and to avoid similar incidents in the future. Actions would be:

Issue: provide a MAC-OS sanity-check
Issue: integrate a sanity-check within the MAC-OS build-scripts

@lazaridiscom Mac OS X and XCode technical support is provided by Apple Corporation.  This forum is for EOS.IO software problems.  CMake's functions constitute a sanity check.  If CMake fails to locate XCode frameworks, the problem lies with the system, not the EOS.IO software.  Systems damaged or deviant enough to prevent CMake from functioning are not supported.  The issue is closed.

Ran into the same problem, but since I've run into this before with a case-sensitive file system, I figured it'd be a simple enough fix. Pull request with the solution has been submitted to develop.

@jgiszczak

[...] The issue is closed.

You (and the project-lead) should really urgently revise your attitude and policies.
@aaroncox
Great!
Related: https://github.com/EOSIO/eos/issues/2719#issuecomment-386611988

@aaroncox thanx man.
obviously it was no issues with my system, apple, xcode or cmake :)

@Vasiliy-Bondarenko Oh it's definitely an issue with Apple.

@Vasiliy-Bondarenko , we, the simple users, must do our best to keep things around EOS evolving.
Thank you for filing this issue and for your patience.

@jgiszczak :))) yep, you are right, apple did a lot of nonsense staff, especially in the last two years :)
@lazaridiscom appreciate it. looks like you know how the team is doing software testing - can you elaborate a little how do you test for performance bottlenecks? our team is going to investigate hardware optimisation for different kind of nodes. thanks.

@Vasiliy-Bondarenko , could swear I answered you already - but my comment is gone... anyway, again:

looks like you know how the team is doing software testing

I'm just processing/evaluating the build/install/test system in an abstract manner, in the role of a simple user (or: contract developer).

can you elaborate a little how do you test for performance bottlenecks

I would go the usual way:

Specify Requirements
Select a Standard-Tool (Profiler)

our team is going to investigate hardware optimisation for different kind of nodes

Sounds like a challenging task. Such functionality could be standardized and included into an EOS SDK.


feat: send transaction waku object wip

TODO:

[ ] overview view
[ ] detect who is the sender and who is recipient
[x] object schema validation
[ ] await on transaction finish
[x] make the chats/[id]/object/[object_id]/new/[[view]] page universal
[ ] implement fiat pricing
[ ] validate user has enough money

Not part of this PR:

waku & blockchain implementation

I get an error after the transaction seemingly successfully executed:

So the reason for the error is that message.data may not be populated yet. Not sure really why, but for now it is fixed with https://github.com/logos-innovation-lab/waku-objects-playground/pull/100/commits/4614ab6c3829946f34378cd04a69670090a2c682#diff-682f0f3dcd82eab92c405c874027d2e250f3330f2d37d752e81444f3f533d3aaR34-R36


add: map support

It is kinda nice to be able to simply say "run this function on these inputs using parallel resources".

Ok, what would the syntax for this look like?
result = ezq.map(f, items)

But what about additional args? Also, blowing away the builtin map function seems like a bad idea.

That second step implies we have a few more args:

number of workers
thread or process

mypy is yelling about the return type for this function. How do you annotate a generator?

Ah, Iterator (StackOverflow). Generator is probably more correct except that I'm still supporting python 3.8, so until that reaches end of life (did it already?), I want to be maximally compatible.

Me:  I'm debating renaming ezq.map to ezq.apply. What are reasons for and against this name change?
ChatGPT: Reasons for renaming ezq.map to ezq.apply:

Familiarity: The name "apply" may be more familiar to some users, as it is used in other popular libraries like Pandas, where it is used to apply a function along an axis of a DataFrame. This familiarity can make the library more intuitive to use for those already familiar with other similar libraries.

Semantics: In functional programming, "apply" is a term used for applying a function to its arguments, which is essentially what the ezq.map function is doing in a parallel manner. The term "apply" might be more descriptive of this process than "map," which is often associated with applying a function element-wise to a single iterable.

Reasons against renaming ezq.map to ezq.apply:

Consistency with Python built-ins: The map function is a built-in Python function that applies a function to all items in an input list. By naming the function ezq.map, you maintain consistency with the built-in Python function, which might make it easier for users to understand its purpose.

Familiarity with other parallel processing libraries: Libraries like concurrent.futures and multiprocessing use the term "map" for their parallelized version of the built-in map function (e.g., Executor.map() and Pool.map()). Keeping the name ezq.map would maintain consistency with other parallel processing libraries in Python, making it more intuitive for users who have experience with these libraries.

In conclusion, the decision to rename ezq.map to ezq.apply depends on the goals and target audience of your library. If you want to maintain consistency with built-in Python functions and other parallel processing libraries, keeping the name ezq.map might be better. On the other hand, if you believe that the term "apply" is more familiar and better conveys the purpose of the function, renaming it to ezq.apply could be a good choice.


Update for scheduled-events.md

Update for scheduled-events.md, adding EventSource and Description for Windows and Linux.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: knightx7 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

@knightx7 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@knightx7 is this the same as #52999 ? Or should I close the other one and work on this one?

This one has both Windows and Linux docs, you can only review this and close #52999.
Please hold off the merging as we checking on some settings internally. Thanks!

@knightx7 does this PR need to me merged?

#sign-off


Swift 3 Conversion Errors

Conversion to Swift 3 generated the following error. Please advise if there is a workaround or release date of a Swift 3 version. Thanks heaps; the DDMathParser is simply brilliant.
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/objdump: '/Users/mazenkilani/Library/Developer/Xcode/DerivedData/8Calc-fstijmwatjfdevfmpimywmndifyk/Build/Intermediates/SwiftMigration/8Calc/Products/Debug-iphonesimulator/8Calc.app/Frameworks/MathParser.framework/MathParser': No such file or directory.
fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/otool: internal objdump command failed

I just pushed version 3.0, which is fully Swift 3 compatible. It's on the master branch, or is the 3.0.0 tag.

What do I write in the podfile if I want to install the swift 3 version?

pod ‘DDMathParser’, :git => 'https://github.com/davedelong/DDMathParser.git’, :tag => ‘3.0.0’

Thank you!


Fix "Unexpected package naming" error

Before:
Previously, when an object existed in the repo and locally but was not included in the package hierarchy of the repository, you would get the following error (when opening the repo):

Test case in https://github.com/abapGit/abapGit/issues/5728
After:
With this PR, log messages will list all objects that are not included in the package hierarchy.

When pulling, the popup will show these objects and ask for confirmation to overwrite. I added an icon and description to the "package overwrite" popup similar to the "object selection" popup.

After confirming, the existing object is overwritten with the remote version and assigned to the correct package:

Closes #5728

@vonglan can you help testing this change?

currently on holiday (2 weeks), will look into it when I am back at work

@vonglan did you have time to test? 🍹

Tested successfully!


Github url link is dependent on the terminal width, breaks out the url

When the terminal is not wide enough, then the PR url gets chopped off. Here the PR number is 14, but on clicking it takes you to PR 1.
So of the width of the terminal where we are running this is less than the URL width, we will be redirected somewhere else.
one more eg:

That's an interesting one I'll have to check how we're doing line breaking because I would expect that'd work fine

This will work fine if we resize the terminal after we run the command in a larger width screen.

Yeah I'll get that logic fixed up (I wanted to do this already)


feat: add new provider api and tests

closes: https://github.com/FuelLabs/fuels-rs/issues/605 and https://github.com/FuelLabs/fuels-rs/issues/350
I have updated provider with the new APIs and added respective unit tests.

This is really cool @segfault-magnet, I love it. But, I believe we should implement and experiment with this on the events querying issue you're working on; it's a lot more likely the amount logs will be huge, and users will likely want to query it in near real-time, which justifies using streams and having this wonderful interface you came up with.
For the case of the current Provider APIs, I think we should keep it simple, for now, and avoid premature optimization. I'd say if the client's API takes in a PaginationRequest, the SDK wrapper API should also take it as well. cc @hal3e

In the last commit, I have added a POC for the new provider API. The code is a bit more complicated than before but the UX is IMO much better. The idea was to give the user the possibility to do the pagination but still make the new API clean and easy to use.
As this is a POC I "converted" only two functions to get your opinion. If  you like it I will convert the rest. Let's compare the old and new get_coins API. Previously we would return all coins associated to an address and there was no way to change it..
let coins = provider.get_coins(wallet.address(), asset_id).await?;

with the new API the call looks like this
let num_results = 2;
let coins = provider
    .get_coins_new(wallet.address(), asset_id, num_results)
    .call()
    .await?
    .results;

we use a builder pattern to create a ProviderPaginationCaller (I desperately need a better name :smile: ) and then we make and await the acctual call. We can also set the cursor, if we have one from a previous call, and we can also set the direction. Here is an example.
let paginated_result = provider
    .get_coins_new(wallet.address(), asset_id, 4)
    .call()
    .await?;
dbg!(paginated_result.results.len());

let paginated_result = provider
    .get_coins_new(wallet.address(), asset_id, 2)
    .with_cursor(paginated_result.cursor)
    .with_direction(PageDirection::Backward)
    .call()
    .await?;
dbg!(paginated_result.results.len());

Now about the code. If you look at the ProviderPaginationCaller struct you will see some complex types. The tricky part was to make the struct accept generic types and save the async function so that we can call it later with the call() method. As we do not know the size of the returned Futures we had to use dyn Future trait and use a Box. If anybody could make this code cleaner I would highly appreciate it :smile: . Here are the crazy types:
type BoxFnFuture<'a, T, U> = Box<dyn Fn(PaginationRequest<T>) -> BoxFutureResult<'a, U> + 'a>;
type BoxFutureResult<'a, U> = Box<dyn Future<Output = Result<U, ProviderError>> + 'a + Unpin>;

pub struct ProviderPaginationCaller<'a, T, U>
where
    T: Debug,
{
    /// The cursor returned from a previous query to indicate an offset
    pub cursor: Option<T>,
    /// The number of results to take
    pub results: usize,
    /// The direction of the query (e.g. asc, desc order).
    pub direction: PageDirection,
    // The function to call
    pub function: BoxFnFuture<'a, T, U>,
}

Talking strictly related to the UX of this new API (and not related to the actual implementation), I believe the API has a nicer UX this way.
An important context to remember is how this would look like without @hal3e's proposal. We would need to always create a PaginationRequest; even if you're feeling lazy and don't want to paginate at all, which is most use cases. So it would look like this:
let pagination_request = PaginationRequest { // I have to import this
                        cursor: None, // Always passing None until I'm actually paginating
                        results: 100,
                        direction: PageDirection::Forward, // And also import this one;
                    };

let coins = provider
    .get_coins_new(wallet.address(), asset_id, pagination_request) // Always need to pass a pagination request.
    .await?
    .results;

So, as we can see, we gain something by not having to .call(), but then we always have to create a PaginationRequest (or pass None if we were to change to accept an Option<PaginationRequest>).
With @hal3e's proposal, we drop the mandatory PaginationRequest, and whenever we need to paginate, we chain the methods with_cursor() and with_direction(). The downside is that we add a .call() to the workflow.
IMO, the .call() isn't too bad as it aligns with contract calls; all contract calls do have a .call() (or .simulate()). Kinda sounds like whenever you're calling something on-chain, you append the .call() to make it explicit, which I think is kinda cool :).
Let me know your thoughts @FuelLabs/sdk-rust.

@MujkicA and @leviathanbeak were able to reproduce the inconsistent behavior when getting paginated results from the client. @leviathanbeak will investigate it from the client side.


Clean working directory when running smelt

For issue #31
Added CleanupDirectory to utils and run before smelting. Ignoring ./.gitkeep, but allowing ./pre/.gitkeep removal as ./pre gets recreated automatically.

We just need to update the latest so we can merge.


NcOpen exceptions do not include the file name

If an ncOpen exception is thrown it is difficult to know which file was not found, as the only file name is the netCDF source file from which the exception was thrown.

PR #481 was merged


Learn More markdown link in inline chat not being rendered

Issue marked as unreleased but unable to locate closing commit in issue timeline. You can manually reference a commit by commenting \closedWith someCommitSha, or directly add the insiders-released label if you know this has already been releaased

@lramos15 how can I trigger this?

@meganrogge A blank python file and asking it to 'Write tim sort in python' is a good way to repro this

That is expected, right now we do not support markdown in the inline widget.


Update index.adoc

trimmed duplicate information.

we had a PR for that already open https://github.com/apache/netbeans-antora-site/pull/27

thanks for the PR but bad timing unfortunately


ui: Show update branch item in merge box when it's necessary

As title, should show it without care about whether this pr can be merged.
example view:

fix #10959

Button should be vertically centered with the text. Maybe try compact or tiny classes. I'd flexbox-center it but it'd require a few wrapper elements.

After fixing styling please ensure it is not broken on other views (when merge button is present, etc)

@silverwind @CirnoT done. Thanks for your help.

Unfortunately the merge button appears to still have floated out of line from the out of date comment.

Maybe split it into it's own "section":

diff --git a/templates/repo/issue/view_content/pull.tmpl b/templates/repo/issue/view_content/pull.tmpl
index 110b80ea2..fcdaf9372 100644
--- a/templates/repo/issue/view_content/pull.tmpl
+++ b/templates/repo/issue/view_content/pull.tmpl
@@ -192,19 +192,24 @@
 				{{end}}

 				{{$canAutoMerge = true}}
 				{{if (gt .Issue.PullRequest.CommitsBehind 0)}}
-

-						{{svg "octicon-alert" 16}}
-						{{$.i18n.Tr "repo.pulls.outdated_with_base_branch"}}
-						{{if .UpdateAllowed}}
-							<form action="{{.Link}}/update" method="post" class="ui floating right">
-								{{.CsrfTokenHtml}}
-								<button class="ui compact small button" data-do="update">
-									{{$.i18n.Tr "repo.pulls.update_branch"}}
-								</button>
-							</form>
-						{{end}}
+

+

+

+							{{svg "octicon-alert" 16}}
+							{{$.i18n.Tr "repo.pulls.outdated_with_base_branch"}}
+

+

+							{{if .UpdateAllowed}}
+								<form action="{{.Link}}/update" method="post" class="ui update-branch-form">
+									{{.CsrfTokenHtml}}
+									<button class="ui compact small button" data-do="update">
+										{{$.i18n.Tr "repo.pulls.update_branch"}}
+									</button>
+								</form>
+							{{end}}
+

 				{{end}}

 				{{if and (or $.IsRepoAdmin (not $notAllOverridableChecksOk)) (or (not .RequireSigned) .WillSign)}}
diff --git a/web_src/less/_repository.less b/web_src/less/_repository.less
index 756ad6d6a..c0bb56e36 100644
--- a/web_src/less/_repository.less
+++ b/web_src/less/_repository.less
@@ -994,10 +994,19 @@

                     > .merge-section {
                         background-color: #f7f7f7;

-                        .item + .item {
-                            padding-top: .5rem;
+                        .item {
+                            padding: .25rem 0;
+                        }
+
+                        .item-section {
+                            display: flex;
+                            align-items: center;
+                            justify-content: space-between;
+                            padding: 0;
+                            margin-top: -.25rem;
+                            margin-bottom: -.25rem;
                         }

                         .divider {
                             margin-left: -1rem;

Maybe we want to apply the style with the divider between each item for all of them, I think it looks cleaner that way.

Maybe we want to apply the style with the divider between each item for all of them

BTW, I tried that and came to the conclusion that it's not possible to render all dividers correctly in all cases because of the complex template logic involved so it'd need to be done via CSS .item + .item, but certainly stuff for another PR.

ping LG-TM


How does System.import() work?

Premise
System.import() allows webpack to define split points for code splitting using es6 modules. It is a replacement for require.ensure()
Findings

Multiple dependencies cannot be imported using System.import() alone. We need to Promise.all[System.import(), System.import()]. But as per https://github.com/webpack/webpack/issues/3141, that does not work as expected as well.
Named chunks are not possible using System.import(). require.ensure() has a parameter that allows us to name the chunks, hence separating or combining code from other split points in the application. This does not seem to be available with System.import()

Ask
System.import() does not seem to be a perfect replacement and seems to create more problems. Are there workarounds for these issues?

The workaround is to create a new module that contains the modules you want to lazy load.

Maybe this can be merged with the discussion in #589? (or at least tackled in the same PR)

Sure!


Custom Date Range

Is there any way to allow for custom date ranges? Sometimes I only want to see the data for the last week/month/quarter/year etc.

This is an issue with GoAcccess but it seems like they are working on it. I do have an issue open on this project to provide an option to not load the archived logs but I'm guessing your looking at this via a GUI perspective. Filtering your date ranges.
Please close this if that answers your question. Thanks!
https://github.com/allinurl/goaccess/issues/478
https://github.com/allinurl/goaccess/issues/117


Enable PEFT models evaluation

This script allows one to run MTEB evaluation against a PEFT model. This enables one to inference and evaluate models with significant memory consumption.
In order to load PEFT models from HuggingFace and use 4 bit quantization to save GRAM, Transformer and SentenceTransfomrer object from sentence-transformer is modified.

@Muennighoff

@NouamaneTazi , thanks for self assigning as a reviewer.
Look forward to seeing your comments. :)

cc @NouamaneTazi

Hi @yutsai84, I am sorry that this code hasn't been reviewed yet. As it is a script I will close it, but feel free to re-open the PR  (if you I will make sure it is reviewed).


make test fails on x86

$ make test
Running tests...
Test project /p/project/cjzam11/kitayama1/build/core
      Start  1: ring_TEST
 1/12 Test  #1: ring_TEST ........................***Failed    0.55 sec
      Start  2: ring_binqueue_TEST
 2/12 Test  #2: ring_binqueue_TEST ...............***Failed    0.06 sec
      Start  3: ring_multisend_TEST
 3/12 Test  #3: ring_multisend_TEST ..............***Failed    0.07 sec
      Start  4: ring_spike_buffer_TEST
 4/12 Test  #4: ring_spike_buffer_TEST ...........***Failed    0.06 sec
      Start  5: ring_duplicate_TEST
 5/12 Test  #5: ring_duplicate_TEST ..............***Failed    0.05 sec
      Start  6: ring_permute1_TEST
 6/12 Test  #6: ring_permute1_TEST ...............***Failed    0.06 sec
      Start  7: ring_permute2_TEST
 7/12 Test  #7: ring_permute2_TEST ...............***Failed    0.06 sec
      Start  8: ring_gap_TEST
 8/12 Test  #8: ring_gap_TEST ....................***Failed    0.06 sec
      Start  9: ring_gap_binqueue_TEST
 9/12 Test  #9: ring_gap_binqueue_TEST ...........***Failed    0.05 sec
      Start 10: ring_gap_multisend_TEST
10/12 Test #10: ring_gap_multisend_TEST ..........***Failed    2.02 sec
      Start 11: ring_gap_permute1_TEST
11/12 Test #11: ring_gap_permute1_TEST ...........***Failed    0.06 sec
      Start 12: ring_gap_permute2_TEST
12/12 Test #12: ring_gap_permute2_TEST ...........***Failed    0.06 sec

0% tests passed, 12 tests failed out of 12

Total Test time (real) =   3.17 sec

The following tests FAILED:
	  1 - ring_TEST (Failed)
	  2 - ring_binqueue_TEST (Failed)
	  3 - ring_multisend_TEST (Failed)
	  4 - ring_spike_buffer_TEST (Failed)
	  5 - ring_duplicate_TEST (Failed)
	  6 - ring_permute1_TEST (Failed)
	  7 - ring_permute2_TEST (Failed)
	  8 - ring_gap_TEST (Failed)
	  9 - ring_gap_binqueue_TEST (Failed)
	 10 - ring_gap_multisend_TEST (Failed)
	 11 - ring_gap_permute1_TEST (Failed)
	 12 - ring_gap_permute2_TEST (Failed)
Errors while running CTest
make: *** [test] Error 8

I'm on JURECA at JSC.

Hello @ikitayama ,
I am sorry for the late reply but I somehow missed this ticket. Could you please provide the modules you loaded, the cmake command you used to build CoreNEURON and run the tests with
ctest -V

from the same directory, to get a more verbose output? I will also try to reproduce the issue in JURECA.
Thank you very much.
Ioannis

$ cmake -DCMAKE_INSTALL_PREFIX=$PROJECT/kitayama1/opt/coreneuron ../../projects/CoreNeuron/

$ ml

Currently Loaded Modules:
  1) GCCcore/.8.3.0 (H)   9) bzip2/.1.0.6       (H)  17) X11/20190311
  2) binutils/.2.32 (H)  10) expat/.2.2.6       (H)  18) hwloc/2.1.0
  3) StdEnv         (H)  11) libpng/.1.6.36     (H)  19) UCX/1.7.0
  4) GCC/8.3.0           12) freetype/.2.10.0   (H)  20) libevent/.2.1.8 (H)
  5) zlib/.1.2.11   (H)  13) gperf/.3.1         (H)  21) OpenMPI/4.0.2
  6) numactl/2.0.12      14) ncurses/.6.1       (H)  22) JUBE/2.2.2
  7) XZ/.5.2.4      (H)  15) util-linux/.2.33.1 (H)  23) CMake/3.14.0
  8) libxml2/.2.9.9 (H)  16) fontconfig/.2.13.1 (H)

  Where:
   H:  Hidden Module

$ ctest -v
Test project /p/project/cjzam11/kitayama1/build/core
      Start  1: ring_TEST
 1/12 Test  #1: ring_TEST ........................***Failed    0.06 sec
      Start  2: ring_binqueue_TEST
 2/12 Test  #2: ring_binqueue_TEST ...............***Failed    0.05 sec
      Start  3: ring_multisend_TEST
 3/12 Test  #3: ring_multisend_TEST ..............***Failed    0.04 sec
      Start  4: ring_spike_buffer_TEST
 4/12 Test  #4: ring_spike_buffer_TEST ...........***Failed    0.04 sec
      Start  5: ring_duplicate_TEST
 5/12 Test  #5: ring_duplicate_TEST ..............***Failed    0.06 sec
      Start  6: ring_permute1_TEST
 6/12 Test  #6: ring_permute1_TEST ...............***Failed    0.06 sec
      Start  7: ring_permute2_TEST
 7/12 Test  #7: ring_permute2_TEST ...............***Failed    0.06 sec
      Start  8: ring_gap_TEST
 8/12 Test  #8: ring_gap_TEST ....................***Failed    0.06 sec
      Start  9: ring_gap_binqueue_TEST
 9/12 Test  #9: ring_gap_binqueue_TEST ...........***Failed    0.05 sec
      Start 10: ring_gap_multisend_TEST
10/12 Test #10: ring_gap_multisend_TEST ..........***Failed    0.04 sec
      Start 11: ring_gap_permute1_TEST
11/12 Test #11: ring_gap_permute1_TEST ...........***Failed    0.04 sec
      Start 12: ring_gap_permute2_TEST
12/12 Test #12: ring_gap_permute2_TEST ...........***Failed    0.07 sec

0% tests passed, 12 tests failed out of 12

Total Test time (real) =   0.64 sec

The following tests FAILED:
	  1 - ring_TEST (Failed)
	  2 - ring_binqueue_TEST (Failed)
	  3 - ring_multisend_TEST (Failed)
	  4 - ring_spike_buffer_TEST (Failed)
	  5 - ring_duplicate_TEST (Failed)
	  6 - ring_permute1_TEST (Failed)
	  7 - ring_permute2_TEST (Failed)
	  8 - ring_gap_TEST (Failed)
	  9 - ring_gap_binqueue_TEST (Failed)
	 10 - ring_gap_multisend_TEST (Failed)
	 11 - ring_gap_permute1_TEST (Failed)
	 12 - ring_gap_permute2_TEST (Failed)
Errors while running CTest

Hello @ikitayama ,
The tests that are running with make test need MPI, so to run them in a cluster like Jureca you need to have an allocation to run them on. You can pass the allocation parameters through the CMake command like that:
cmake \
    .. \
    -DCMAKE_INSTALL_PREFIX=./install \
    -DTEST_MPI_EXEC_BIN="srun;--account=<ACCOUNT_NAME>;--partition=<PARTITION>;-t;00:05:00" \
    -DTEST_EXEC_PREFIX="srun;-n;6;--account=<ACCOUNT_NAME>;--partition=<PARTITION>;-t;00:05:00" \
    -DAUTO_TEST_WITH_SLURM=OFF \
    -DAUTO_TEST_WITH_MPIEXEC=OFF/

Just replace <ACCOUNT_NAME> and <PARTITION> with the account name and partition you would like to run the tests.
I managed to compile and run the tests on Jureca with the above CMake command using GCC 8.3.0 and IntelMPI.
Please feel free to contact us if you see any other issue.
Ioannis

Test project /p/project/cjzam11/kitayama1/build/core
      Start  1: ring_TEST
 1/12 Test  #1: ring_TEST ........................***Failed   12.84 sec
      Start  2: ring_binqueue_TEST
 2/12 Test  #2: ring_binqueue_TEST ...............***Failed   70.28 sec
      Start  3: ring_multisend_TEST
 3/12 Test  #3: ring_multisend_TEST ..............***Failed   51.60 sec
      Start  4: ring_spike_buffer_TEST
 4/12 Test  #4: ring_spike_buffer_TEST ...........***Failed   68.13 sec
      Start  5: ring_duplicate_TEST
 5/12 Test  #5: ring_duplicate_TEST ..............***Failed   60.45 sec
      Start  6: ring_permute1_TEST
 6/12 Test  #6: ring_permute1_TEST ...............***Failed   83.32 sec
      Start  7: ring_permute2_TEST
 7/12 Test  #7: ring_permute2_TEST ...............***Failed   36.43 sec
      Start  8: ring_gap_TEST
 8/12 Test  #8: ring_gap_TEST ....................***Failed   52.92 sec
      Start  9: ring_gap_binqueue_TEST
 9/12 Test  #9: ring_gap_binqueue_TEST ...........***Failed   67.07 sec
      Start 10: ring_gap_multisend_TEST
10/12 Test #10: ring_gap_multisend_TEST ..........***Failed   60.32 sec
      Start 11: ring_gap_permute1_TEST
11/12 Test #11: ring_gap_permute1_TEST ...........***Failed   60.29 sec
      Start 12: ring_gap_permute2_TEST
12/12 Test #12: ring_gap_permute2_TEST ...........***Failed   47.37 sec

0% tests passed, 12 tests failed out of 12

Total Test time (real) = 671.07 sec

The following tests FAILED:
	  1 - ring_TEST (Failed)
	  2 - ring_binqueue_TEST (Failed)
	  3 - ring_multisend_TEST (Failed)
	  4 - ring_spike_buffer_TEST (Failed)
	  5 - ring_duplicate_TEST (Failed)
	  6 - ring_permute1_TEST (Failed)
	  7 - ring_permute2_TEST (Failed)
	  8 - ring_gap_TEST (Failed)
	  9 - ring_gap_binqueue_TEST (Failed)
	 10 - ring_gap_multisend_TEST (Failed)
	 11 - ring_gap_permute1_TEST (Failed)
	 12 - ring_gap_permute2_TEST (Failed)
Errors while running CTest
make: *** [test] Error 8

My loaded modules are:
$ ml

Currently Loaded Modules:
  1) GCCcore/.8.3.0 (H)   9) bzip2/.1.0.6       (H)  17) X11/20190311
  2) binutils/.2.32 (H)  10) expat/.2.2.6       (H)  18) hwloc/2.1.0
  3) StdEnv         (H)  11) libpng/.1.6.36     (H)  19) UCX/1.7.0
  4) GCC/8.3.0           12) freetype/.2.10.0   (H)  20) libevent/.2.1.8 (H)
  5) zlib/.1.2.11   (H)  13) gperf/.3.1         (H)  21) OpenMPI/4.0.2
  6) numactl/2.0.12      14) ncurses/.6.1       (H)  22) JUBE/2.2.2
  7) XZ/.5.2.4      (H)  15) util-linux/.2.33.1 (H)  23) CMake/3.14.0
  8) libxml2/.2.9.9 (H)  16) fontconfig/.2.13.1 (H)

  Where:
   H:  Hidden Module

Hello @ikitayama ,
It seems now that the tests have started executing but failed at some point.
Could you please provide the output of ctest -V?
Also in which system are you trying to run the tests? CPU (Jureca/Juwels), KNLs (Jureca Booster) or GPUs (Jureca/Juwels)?
Thank you very much again.

Did the tests run ok on JURECA, by the way?
If so, the issue probably due to my configuration.
On Fri, Jul 3, 2020 at 19:52 Ioannis Magkanaris<EMAIL_ADDRESS>wrote:

Hello @ikitayama https://github.com/ikitayama ,
It seems now that the tests have started executing but failed at some
point.
Could you please provide the output of ctest -V?
Also in which system are you trying to run the tests? CPU (Jureca/Juwels),
KNLs (Jureca Booster) or GPUs (Jureca/Juwels)?
Thank you very much again.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BlueBrain/CoreNeuron/issues/321#issuecomment-653485600,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHLBZ2A7EFRHV3DUWP52ADRZWZ5BANCNFSM4OI4IPZA
.

Hello @ikitayama ,
I used this script to run install and run the tests of CoreNEURON in Jureca Booster (KNLs)
############################################################
# Edit following env vars according to your `ACCOUNT_NAME` and `PARTITION`
export ACCOUNT_NAME=<ACCOUNT_NAME>
export PARTITION=<PARTITION>
############################################################
git clone --recursive https://github.com/BlueBrain/CoreNeuron.git
cd CoreNeuron
mkdir build
cd build
module load Architecture/KNL
module load CMake Intel/2019.5.281-GCC-8.3.0 IntelMPI/2019.6.154
cmake \
    ..
Please try if this works for you and let me know if you need any further help.
We are also preparing documentation on how to run CoreNEURON with NEURON, so we will also update you on that as soon as it's ready.
Thank you very much,
Ioannis

Hi @iomaganaris
Loading modules you recommend for NEURON and Core users, see below:

$ ml

Currently Loaded Modules:
  1) StdEnv             (H)  15) hwloc/2.1.0
  2) zlib/.1.2.11       (H)  16) libevent/.2.1.8             (H)
  3) numactl/2.0.12          17) JUBE/2.2.2
  4) XZ/.5.2.4          (H)  18) CMake/3.14.0
  5) libxml2/.2.9.9     (H)  19) Architecture/Haswell        (S)
  6) bzip2/.1.0.6       (H)  20) GCCcore/.8.3.0              (H)
  7) expat/.2.2.6       (H)  21) binutils/.2.32              (H)
  8) libpng/.1.6.36     (H)  22) icc/.2019.5.281-GCC-8.3.0   (H)
  9) freetype/.2.10.0   (H)  23) ifort/.2019.5.281-GCC-8.3.0 (H)
 10) gperf/.3.1         (H)  24) Intel/2019.5.281-GCC-8.3.0
 11) ncurses/.6.1       (H)  25) mpi-settings/default
 12) util-linux/.2.33.1 (H)  26) UCX/1.6.1
 13) fontconfig/.2.13.1 (H)  27) IntelMPI/2019.6.154
 14) X11/20190311

  Where:
   S:  Module is Sticky, requires --force to unload or purge
   H:             Hidden Module

I was able to run a set of CoreNeuron tests with Intel MPI successfully!

$ make test
Running tests...
Test project /p/project/cjzam11/kitayama1/build/core
      Start  1: ring_TEST
 1/12 Test  #1: ring_TEST ........................   Passed   12.17 sec
      Start  2: ring_binqueue_TEST
 2/12 Test  #2: ring_binqueue_TEST ...............   Passed   49.37 sec
      Start  3: ring_multisend_TEST
 3/12 Test  #3: ring_multisend_TEST ..............   Passed   60.46 sec
      Start  4: ring_spike_buffer_TEST
 4/12 Test  #4: ring_spike_buffer_TEST ...........   Passed   59.48 sec
      Start  5: ring_duplicate_TEST
 5/12 Test  #5: ring_duplicate_TEST ..............   Passed   61.67 sec
      Start  6: ring_permute1_TEST
 6/12 Test  #6: ring_permute1_TEST ...............   Passed   58.28 sec
      Start  7: ring_permute2_TEST
 7/12 Test  #7: ring_permute2_TEST ...............   Passed   54.97 sec
      Start  8: ring_gap_TEST
 8/12 Test  #8: ring_gap_TEST ....................   Passed   65.13 sec
      Start  9: ring_gap_binqueue_TEST
 9/12 Test  #9: ring_gap_binqueue_TEST ...........   Passed   23.03 sec
      Start 10: ring_gap_multisend_TEST
10/12 Test #10: ring_gap_multisend_TEST ..........   Passed   23.62 sec
      Start 11: ring_gap_permute1_TEST
11/12 Test #11: ring_gap_permute1_TEST ...........   Passed   73.51 sec
      Start 12: ring_gap_permute2_TEST
12/12 Test #12: ring_gap_permute2_TEST ...........   Passed   59.93 sec

100% tests passed, 0 tests failed out of 12

Total Test time (real) = 601.64 sec

The only question remains is this: why GCC and OpenMPI did not work?

Hello @ikitayama ,
I am not sure why there is an issue with OpenMPI. In the beginning however I tried to run with OpenMPI on KNLs and had issues running the tests.
Did you use a KNL partition or a CPU one?

I'm using CPU partitions only.
On Mon, Jul 6, 2020 at 4:34 PM Ioannis Magkanaris<EMAIL_ADDRESS>wrote:

Hello @ikitayama https://github.com/ikitayama ,
I am not sure why there is an issue with OpenMPI. In the beginning
however I tried to run with OpenMPI on KNLs and had issues running the
tests.
Did you use a KNL partition or a CPU one?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BlueBrain/CoreNeuron/issues/321#issuecomment-654067467,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHLBZ7CZIBVAHA6OBPBRSLR2F47VANCNFSM4OI4IPZA
.

Off topic though isn’t loading Architeture/Haswell a bit problematic when save/ restore?
I’m contacting Support but I appreciate you tell your experience.

Hello @ikitayama ,
I used this script to run install and run the tests of CoreNEURON in Jureca Booster (KNLs)

############################################################

# Edit following env vars according to your `ACCOUNT_NAME` and `PARTITION`

export ACCOUNT_NAME=<ACCOUNT_NAME>

export PARTITION=<PARTITION>

############################################################

git clone --recursive https://github.com/BlueBrain/CoreNeuron.git

cd CoreNeuron

mkdir build

cd build

module load Architecture/KNL

module load CMake Intel/2019.5.281-GCC-8.3.0 IntelMPI/2019.6.154

cmake \

    ..
Please try if this works for you and let me know if you need any further help.
We are also preparing documentation on how to run CoreNEURON with NEURON, so we will also update you on that as soon as it's ready.
Thank you very much,
Ioannis


move trait method: from RaftLogStorage to RaftLogReader::read_vote()

To Ensuring Consecutive Raft Log Replication
Problem Description
In a Raft-based system, there is no guarantee that sequential read-log operations, such as read log[i] and read log[i+1], will yield consecutive log entries. The reason for this is that Raft logs can be truncated, which disrupts the continuity of the log sequence. Consider the following scenario:

Node-1, as a leader, reads log[4].
Subsequently, Node-1 receives a higher term vote and demotes itself.
In the interim, a new leader emerges and truncates the logs on Node-1.
Node-1 regains leadership, appends new logs, and attempts to read log[5].

Now, the read operation for log[5] becomes invalid because the log at index 6, term 5 (6-5) is no longer a direct successor to the log at index 4, term 4 (4-4). This violates Raft's log continuity requirement for replication.
To illustrate:
i-j: Raft log at term i and index j

Node-1: | vote(term=4)
        | 2-3  2-4
               ^
               '--- Read log[4]

Node-1: | vote(term=5) // Higher term vote received
        | 2-3  2-4

Node-1: | vote(term=5) // Logs are truncated, 4-4 is replicated
        | 2-3  4-4
               ^
               '--- Truncation and replication by another leader

Node-1: | vote(term=6) // Node-1 becomes leader again
        | 2-3  4-4  6-5 // Append noop log

Node-1: | vote(term=6)
        | 2-3  4-4  6-5
                    ^
                    '--- Attempt to read log[5]

Proposed Solution
To ensure consecutive log entries for safe replication, it is necessary to verify that the term has not changed between log reads. The following updated operations are proposed:

read vote (returns v1)
read log[i]
read log[i+1]
read vote (returns v2)

If v1 is equal to v2, we can confidently say that log[i] and log[i+1] are consecutive and the replication is safe. Therefore, ReplicationCore must check the term (vote) as part of its replication process to ensure log consecutivity.
Conclusion
The current release mitigates this issue by immediately halting communication with ReplicationCore to prevent any new replication commands from being processed. However, future iterations of ReplicationCore may operate more proactively. To maintain the integrity of replication, ReplicationCore will be required to independently ensure the consecutivity of logs before proceeding. This is an important consideration for the ongoing development of the system.
TODO
Because RaftLogStorage implements RaftLogReader, read_vote() should be move from RaftLogStorage to RaftLogReader.

/assignme


impose stricter connection requirements for peers

Recently, the metrics for uploads and downloads have been poor so the network needs to have a stricter requirements imposed on peers.
Some ideas:

Peers that do not respond to ping requests or respond correctly to chainsync request should be immediately blocklisted.
Request timeouts should be more aggressive.

Blocklist peer if there is a discrepancy of storage radius.
If the peer's storage radius is different from ours and both ours and the peer's reserve size is stable (eg > 40% full), blocklist the peer for some time.

A side effect of agresive blocklisting is it can cause bad nodes (eg wrong storage radius, resource constrained, low network bandwidth) to lose even more peers, potentially causing the storage radius to decrease to an even further worse value.
Using the status protocol, we can impose a minimum storage radius a node can reach.

Here's a snapshot of some broken nodes in the network to consider while devising the requirements:
{"peer":"****","proximity":8,"reserveSize":2261871,"pullsyncRate":0,"storageRadius":9,"connectedPeers":145,"neighborhoodSize":7}
bee-0-0-debug.bee-gateway.mainnet.internal
"Estonia"
{"peer":"****","proximity":8,"reserveSize":2261871,"pullsyncRate":0,"storageRadius":9,"connectedPeers":145,"neighborhoodSize":7}
bee-0-debug.bee-gateway.mainnet.internal
"Estonia"
{"peer":"****","proximity":1,"reserveSize":3815665,"pullsyncRate":0,"storageRadius":7,"connectedPeers":80,"neighborhoodSize":2}
bee-1-0-debug.bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":7,"reserveSize":3591216,"pullsyncRate":0,"storageRadius":7,"connectedPeers":71,"neighborhoodSize":1}
bee-1-0-debug.bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":1,"reserveSize":3815665,"pullsyncRate":0,"storageRadius":7,"connectedPeers":80,"neighborhoodSize":2}
bee-1-debug.bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":7,"reserveSize":3591216,"pullsyncRate":0,"storageRadius":7,"connectedPeers":71,"neighborhoodSize":1}
bee-1-debug.bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":3,"reserveSize":3319433,"pullsyncRate":0,"storageRadius":7,"connectedPeers":73,"neighborhoodSize":5}
bee-3-0-debug.bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":3,"reserveSize":3319433,"pullsyncRate":0,"storageRadius":7,"connectedPeers":73,"neighborhoodSize":5}
bee-3-debug.bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":0,"reserveSize":0,"pullsyncRate":0,"storageRadius":0,"connectedPeers":0,"neighborhoodSize":0,"requestFailed":true}
bee-1-0-debug.bee-storage.mainnet.internal
"Colombia"
{"peer":"****","proximity":0,"reserveSize":3417918,"pullsyncRate":0,"storageRadius":7,"connectedPeers":76,"neighborhoodSize":1}
bee-1-0-debug.bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":0,"reserveSize":3417918,"pullsyncRate":0,"storageRadius":7,"connectedPeers":76,"neighborhoodSize":1}
bee-1-debug.bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":0,"reserveSize":3359271,"pullsyncRate":0,"storageRadius":7,"connectedPeers":47,"neighborhoodSize":0}
bee-2-0-debug.bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":0,"reserveSize":3359271,"pullsyncRate":0,"storageRadius":7,"connectedPeers":47,"neighborhoodSize":0}
bee-2-debug.bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":6,"reserveSize":4092959,"pullsyncRate":0,"storageRadius":7,"connectedPeers":72,"neighborhoodSize":0}
bee-1-0-debug.dev-bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":6,"reserveSize":4092959,"pullsyncRate":0,"storageRadius":7,"connectedPeers":72,"neighborhoodSize":0}
bee-1-debug.dev-bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":4,"reserveSize":3864185,"pullsyncRate":0,"storageRadius":7,"connectedPeers":79,"neighborhoodSize":2}
bee-2-0-debug.dev-bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":5,"reserveSize":3890077,"pullsyncRate":0,"storageRadius":7,"connectedPeers":59,"neighborhoodSize":0}
bee-2-0-debug.dev-bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":4,"reserveSize":3864185,"pullsyncRate":0,"storageRadius":7,"connectedPeers":79,"neighborhoodSize":2}
bee-2-debug.dev-bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":5,"reserveSize":3890077,"pullsyncRate":0,"storageRadius":7,"connectedPeers":59,"neighborhoodSize":0}
bee-2-debug.dev-bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":1,"reserveSize":3936080,"pullsyncRate":0,"storageRadius":4,"connectedPeers":109,"neighborhoodSize":34}
bee-3-0-debug.dev-bee-gateway.mainnet.internal
"France"
{"peer":"****","proximity":4,"reserveSize":3848149,"pullsyncRate":11.754392888888889,"storageRadius":6,"connectedPeers":82,"neighborhoodSize":3}
bee-3-0-debug.dev-bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":1,"reserveSize":3936080,"pullsyncRate":0,"storageRadius":4,"connectedPeers":109,"neighborhoodSize":34}
bee-3-debug.dev-bee-gateway.mainnet.internal
"France"
{"peer":"****","proximity":4,"reserveSize":3848149,"pullsyncRate":11.75397688888889,"storageRadius":6,"connectedPeers":82,"neighborhoodSize":3}
bee-3-debug.dev-bee-gateway.mainnet.internal
"China"
{"peer":"****","proximity":1,"reserveSize":3319433,"pullsyncRate":0,"storageRadius":7,"connectedPeers":73,"neighborhoodSize":5}
bee-0-0-debug.dev-bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":2,"reserveSize":4170437,"pullsyncRate":0,"storageRadius":6,"connectedPeers":69,"neighborhoodSize":1}
bee-1-0-debug.dev-bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":3,"reserveSize":0,"pullsyncRate":0,"storageRadius":0,"connectedPeers":144,"neighborhoodSize":144}
bee-1-0-debug.dev-bee-storage.mainnet.internal
"Germany"
{"peer":"****","proximity":5,"reserveSize":3890077,"pullsyncRate":0,"storageRadius":7,"connectedPeers":59,"neighborhoodSize":0}
bee-1-0-debug.dev-bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":3,"reserveSize":3489430,"pullsyncRate":0,"storageRadius":7,"connectedPeers":75,"neighborhoodSize":0}
bee-2-0-debug.dev-bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":3,"reserveSize":3991412,"pullsyncRate":10.880430666666667,"storageRadius":6,"connectedPeers":69,"neighborhoodSize":2}
bee-2-0-debug.dev-bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":4,"reserveSize":3886460,"pullsyncRate":0,"storageRadius":7,"connectedPeers":80,"neighborhoodSize":0}
bee-2-0-debug.dev-bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":2,"reserveSize":2874346,"pullsyncRate":0,"storageRadius":7,"connectedPeers":87,"neighborhoodSize":0}
bee-3-0-debug.dev-bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":5,"reserveSize":3422745,"pullsyncRate":0,"storageRadius":6,"connectedPeers":64,"neighborhoodSize":0}
bee-3-0-debug.dev-bee-storage.mainnet.internal
"China"
{"peer":"****","proximity":8,"reserveSize":2161396,"pullsyncRate":0,"storageRadius":7,"connectedPeers":81,"neighborhoodSize":1}
bee-0-0-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":8,"reserveSize":2161396,"pullsyncRate":0,"storageRadius":7,"connectedPeers":81,"neighborhoodSize":1}
bee-0-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":3,"reserveSize":4132925,"pullsyncRate":0,"storageRadius":1,"connectedPeers":1,"neighborhoodSize":1}
bee-2-0-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":3,"reserveSize":1900073,"pullsyncRate":114.70086414444444,"storageRadius":3,"connectedPeers":34,"neighborhoodSize":2}
bee-2-0-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":3,"reserveSize":4132925,"pullsyncRate":0,"storageRadius":1,"connectedPeers":1,"neighborhoodSize":1}
bee-2-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":3,"reserveSize":1900073,"pullsyncRate":115.61152147777779,"storageRadius":3,"connectedPeers":34,"neighborhoodSize":2}
bee-2-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":1,"reserveSize":0,"pullsyncRate":0,"storageRadius":0,"connectedPeers":0,"neighborhoodSize":0,"requestFailed":true}
bee-3-debug.bee-bootnode.mainnet.internal
"Finland"
{"peer":"****","proximity":2,"reserveSize":3898823,"pullsyncRate":293.25959911111113,"storageRadius":5,"connectedPeers":33,"neighborhoodSize":2}
bee-4-0-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":2,"reserveSize":3972396,"pullsyncRate":133.74654027777777,"storageRadius":0,"connectedPeers":1,"neighborhoodSize":1}
bee-4-0-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":6,"reserveSize":3319527,"pullsyncRate":0,"storageRadius":7,"connectedPeers":73,"neighborhoodSize":5}
bee-4-0-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":2,"reserveSize":3898823,"pullsyncRate":293.0834728888889,"storageRadius":5,"connectedPeers":33,"neighborhoodSize":2}
bee-4-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":2,"reserveSize":3972396,"pullsyncRate":132.8692861111111,"storageRadius":0,"connectedPeers":1,"neighborhoodSize":1}
bee-4-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":6,"reserveSize":3319527,"pullsyncRate":0,"storageRadius":7,"connectedPeers":73,"neighborhoodSize":5}
bee-4-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":1,"reserveSize":3943582,"pullsyncRate":0,"storageRadius":0,"connectedPeers":1,"neighborhoodSize":1}
bee-5-0-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":1,"reserveSize":3943582,"pullsyncRate":0,"storageRadius":0,"connectedPeers":1,"neighborhoodSize":1}
bee-5-debug.bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":0,"reserveSize":3263641,"pullsyncRate":0,"storageRadius":0,"connectedPeers":1,"neighborhoodSize":1}
bee-0-0-debug.dev-bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":0,"reserveSize":4090759,"pullsyncRate":0,"storageRadius":3,"connectedPeers":1,"neighborhoodSize":0}
bee-0-0-debug.dev-bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":0,"reserveSize":4064838,"pullsyncRate":0,"storageRadius":5,"connectedPeers":1,"neighborhoodSize":0}
bee-0-0-debug.dev-bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":2,"reserveSize":0,"pullsyncRate":0,"storageRadius":0,"connectedPeers":0,"neighborhoodSize":0,"requestFailed":true}
bee-0-0-debug.dev-bee-bootnode.mainnet.internal
"United States"
{"peer":"****","proximity":8,"reserveSize":4150439,"pullsyncRate":24.560575872222223,"storageRadius":6,"connectedPeers":67,"neighborhoodSize":2}
bee-0-0-debug.dev-bee-bootnode.mainnet.internal
"China"
{"peer":"****","proximity":9,"reserveSize":3512404,"pullsyncRate":0,"storageRadius":6,"connectedPeers":69,"neighborhoodSize":2}
bee-0-0-debug.dev-bee-bootnode.mainnet.internal
"China"

Personally, I wouldn't consider "broken" anything with a non-zero pullsyncRate regardless of the storageRadius.  Isn't that the whole point of pullsync?  They're trying to get their reserve populated?

@ldeffenb notice that the storage radius and connected peers count is all over the place.
A "healthy" node needs to have > 130 peers and a storage radius of 8 based on the current status of the network,

ideas:
remove kademlia ping
add new status worker that blocklist based on certain criteria:

compute moving average of status reponse duration from all peers, and blocklist any peer that exceeds the duration with some extra threshold
compute the storage depth of the network (through the peers) and bloclist any peer whose radius is below network radius -1

2. compute the storage depth of the network (through the peers) and bloclist any peer whose radius is below network radius -1

But if you blocklist lower-radius peers, they'll be disconnected and never actually have a chance for pullsync to fix their radius?

compute the storage depth of the network (through the peers) and bloclist any peer whose radius is below network radius -1

But if you blocklist lower-radius peers, they'll be disconnected and never actually have a chance for pullsync to fix their radius?

if a node's radius is significantly lower than the radius of the network, the node is in a really bad shape and provides no value to the network. The node also negatively affects syncing and retrieval protocol requests across the network as well.

@ldeffenb it doesn't. It's a bad node that gets filtered out.


⚠ GitLab Server has degraded performance

In 79e4ec7, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 11176 ms

Resolved: GitLab Server performance has improved in 8d4b094.


[SPARK-23291][SQL][R] R's substr should not reduce starting position by 1 when calling Scala API

What changes were proposed in this pull request?
Seems R's substr API treats Scala substr API as zero based and so subtracts the given starting position by 1.
Because Scala's substr API also accepts zero-based starting position (treated as the first element), so the current R's substr test results are correct as they all use 1 as starting positions.
How was this patch tested?
Modified tests.

Test build #86908 has started for PR 20464 at commit a2ffdc1.

One more thing to notice is that the two parameters (starting and ending positions) of R's substr API is also unaligned with Scala's substr which takes starting position and substring length.

One thing to keep in mind is what the user's perception of the API is. If R users are going to use 1-based indexing then this might not be the right fix ? http://stat.ethz.ch/R-manual/R-devel/library/base/html/substr.html is the base R function FWIW

@shivaram This fix is to make it correctly 1-based. Previously SparkR substr API substracts starting position by 1, so it becomes zero-based.
This fix matches R's substr in above link as I test:
> substr("Michael", 4, 6)
[1] "hae"

Before this fix, SparkR's substr returns "cha".

Thanks for clarifying @viirya. Is the PR description accurate ? I read it as ..SQL's substr also accepts zero-based starting position while R uses a 1-based starting position.

Test build #86908 has finished for PR 20464 at commit a2ffdc1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86908/
Test PASSed.

Merged build finished. Test PASSed.

@shivaram Thanks for pointing out it. I made change to the description. Hopefully it is clearer now. Basically I just want to clarify why R's substr tests are correct previously.

I was just manually double checking both substr in R and this. It seems correct; however, I think we should add a note in the doc and release note ...
One followup question is though, would it be difficult to match the behaviour with substr in R when the index is 0 or minus? If i understood https://github.com/apache/spark/pull/20464#issuecomment-362150632 correctly, it sounds better to match it to substr's behaviour in R. Took a quick look/test and seems we can just set start to 1 for both cases.
If this followup question is something we are not sure yet, I think we might be okay as is.

Just in case, I am testing with:
df <- createDataFrame(list(list(a="abcdef")))
collect(select(df, substr(df$a, 4, 5)))
substr("abcdef", 4, 5)

just in case it helps to check and reproduce.

One followup question is though, would it be difficult to match the behaviour with substr in R when the index is 0 or minus? If i understood #20464 (comment) correctly, it sounds better to match it to substr's behaviour in R. Took a quick look/test and seems we can just set start to 1 for both cases.

If we both consider the indices at starting and ending, setting them to 1 seems not enough. E.g.,
> substr("abcdef", -2, -3)
[1] ""
> substr("abcdef", 1, 1)
[1] "a"

For the cases when only ending is zero/negative, no matter what starting is, the result is empty string.
For the cases when only starting is zero/negative, we can set it to 1.
For the cases they are both zero/negative, the result is empty string.
We can address this in another PR.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/473/
Test PASSed.

Test build #86924 has started for PR 20464 at commit 95c8a4e.

Test build #86924 has finished for PR 20464 at commit 95c8a4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86924/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/525/
Test PASSed.

Test build #86985 has started for PR 20464 at commit d994d76.

Test build #86985 has finished for PR 20464 at commit d994d76.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86985/
Test PASSed.

I think @felixcheung has the most context here, so I'd suggest we wait for his comments.

ping @felixcheung

Sorry, I'm a bit occupied with testing 2.3 RC, will get back to this after.

@felixcheung Thanks!

Because 2.3 is released, ping @felixcheung again

Test build #87993 has started for PR 20464 at commit 0ebdf74.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1305/
Test PASSed.

Test build #87993 has finished for PR 20464 at commit 0ebdf74.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87993/
Test PASSed.

appveyor tests failed, could you close and reopen this PR to trigger it.
strange, I haven't seen anything like this on appveyor a long time.
1. Error: create DataFrame with complex types (@test_sparkSQL.R#535) -----------
8712org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 114.0 failed 1 times, most recent failure: Lost task 0.0 in stage 114.0 (TID 116, localhost, executor driver): java.net.SocketTimeoutException: Accept timed out

Test build #88039 has started for PR 20464 at commit 8c1a8ec.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1347/
Test PASSed.

Test build #88039 has finished for PR 20464 at commit 8c1a8ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88039/
Test PASSed.

merged to master, thanks!


JAVA-3143: Extend driver vector support to arbitrary subtypes and fix handling of variable length types (OSS C* 5.0)

This branch is on top of https://github.com/apache/cassandra-java-driver/pull/1931


Dont hardcode route name in test

Please review
@Ladicek
@spisiakm

This isn't necessary.

Closing as @Ladicek is right, this change is not necessary and could possibly cause some future issues.


Infinite scroll 'blank block'

When using the infinite scroll at http://ui-grid.info/docs/#/tutorial/212_infinite_scroll
If you scroll down, content will load but not be shown until the 'up' scrollbar is pressed.

+1 to this.  I am seeing the same problem with infinite scroll.  I also see a similar problem without infinite scroll on a grid with a large data set.

I just noticed this as well, and looks like this was broken between 1/2 and 1/15.

@Arkh1 is it still breaking for you on the large data set without infinite scrolling? I couldn't get it to happen.

Actually after switching over to the unstable version of ui-grid I no longer experience the issues I was having with large datasets.

OK, excellent, closing.

This issue is still reproducable, I can see it on the demos and when moving over to unstable on my project. I have reproduced it on both chrome and firefox, all you need to do is go to http://ui-grid.info/docs/#/tutorial/212_infinite_scroll and scroll down via scrollbar or the mouse.

When this happens, the canvas has a height of 6000px, the first element has a margin-top of 4350px in the style. If I remove the margin on both, the scroll bar is very jumpy and initially at the wrong postion, but I can see content 'normally' for a while until it loads more content and the height changes again.

My group actually has a fix for this issue, we are working through it and will post a fix asap.
We're also working on code to enable infinite scroll in the UP direction.  Just FYI.

Cool  Ryszard !
Yes ,currently the infinite scrolling is not working . Should be taking out of the feature implemented until a basic sample works such as the sample from the tutorial !
Not working or missing features .
1-blank block after scrolling until a request is triggered to the backend to load more records.
2-keeping in memory only a subset of the records loaded to minimize footprint
3-scrolling up and down should trigger a call to the backend if the records previously loaded were reclaimed for memory management.
4-currently ,the implementation is pushing records to the end of the array . May need more sophisticated to push ,insert in specific positions. (see slickgrid for example)
Thank you
Claude Hussenet

The blank lines issue seems better but before (totally subjective), but is still there.
Considering this is also in the 'vanilla' add rows demo, I presume that there are some other related core issues.


Pagination: Move aria-label inside element

Who is this for?
Best practices
What is it doing for them?
Pagination is usually used more than once in most pages where it is found (before and after results). As one of its child element uses an id, it was flagged as an issue as it was repeated throughout the same page. I move the label inside the element itself, removing the need for the id altogether.

Neat solution. I'm just questioning the text in the label (which I know you didn't change).
I'm not sure it describes what the input is for very well, but it does give useful information about what page of results is being viewed.
I'd also be tempted to replace the '/' with 'of' so it reads better.


Detect timezone changes

Fixes #479, because upstream does not intend to.
It may be less expensive to do that only once in a while, or to inotify-watch on /etc/timezone, but this is good enough.

Thanks! :)


fix to make tests run with 'python setup.py test'

This is the result of "python setup.py test" before applying this fix.
$ python setup.py test
running test
Traceback (most recent call last):
  File "setup.py", line 57, in <module>
    'Topic :: Utilities',
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 141, in run
    cmd = ' '.join(self._argv)
TypeError: sequence item 2: expected string, bool found
$

Hey, I think this is actually related to setuptools 18 rather than any version of Python.. I ran into this with pywb as well.. I think removing the test_suite though breaks it on setuptools<18
I fixed it by adding:
    def initialize_options(self):
        TestCommand.initialize_options(self)
        self._argv = []

Thanks for fixing it.

Have a better fix now, that didn't work in setuptools 18.5, as that was now a property.. Just setting test_suite = ' ' so its a string and not-false for previous versions...

Thanks for figuring this out.


Warning banner on dev website

Saying it's not the official one, so data can disappear at any time, the website can be done, be buggy, ...
location: red thin band on top of navbar?

Done: mechanism in place + config adjusted on the dev-alert website.


Use AsMut/AsRef-like deref instead of unions

Currently, svd2rust supports overlapping registers via the cli flag --nightly. This will generate an union of all registers that overlap each other.
Since it is not expected that non-Copy fields in unions will be stabilized soon (see tracking issue rust-lang/rust#32836), another system for access would prove beneficial for library maintainers and users wanting to access overlapping registers using stable rust.
cc @wez @japaric
related #16

Yeah, makes sense.  Referencing this example from atsamd21:
https://docs.rs/atsamd21g18a/0.2.1/atsamd21g18a/tc3/union.RegisterBlock.html
instead of the union generate something like this:
pub struct RegisterBlock {}

impl RegisterBlock {
    fn as_count8_mut(&mut self) -> &mut COUNT8 {
        unsafe {
            &mut *((self as *mut Self) as *mut COUNT8)
        }
    }

    fn as_count8_ref(&self) -> &COUNT8 {
        unsafe {
            &*((self as *const Self) as *const COUNT8)
        }
    }

    fn as_count16_mut(&mut self) -> &mut COUNT16 {
        unsafe {
            &mut *((self as *mut Self) as *mut COUNT16)
        }
    }

   // as_count16_ref(), as_count32_mut(), as_count32_ref() here
}

I've found that in atsamd21-hal I'd often make a safe accessor for the union anyway, so this removes some boilerplate for me.

I can play around with this, since I think I'm the only person currently with a crate that requires and is using this :)

I have some WIP in https://github.com/wez/svd2rust/tree/no_unions
It happens to look like the suggestion in https://github.com/rust-embedded/svd2rust/issues/230

I believe this is now solved by #318.


Removal / De-Provisioning of APPs not functional

1.) 1507-1709:
Get-AppxProvisionedPackage -Online | Where-Object {$.PackageName -Like "Microsoft.BingNews"} | ForEach-Object { Remove-AppxProvisionedPackage -Online -PackageName $.PackageName}
This command is wrong, should be corrected to: [...] | Where-Object {$.Displayname [...] or shorten it completly to: Get-AppxProvisionedPackage -Online | where {$.DisplayName -eq "Name.of.The.App"} |
Remove-AppxProvisionedPackage -online

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e8228962-7158-6be1-e791-e6535b88f343
Version Independent ID: 46e4c7f3-e8b0-ebc9-da58-b69149803937
Content: Manage connections from Windows operating system components to Microsoft services (Windows 10)
Content Source: windows/configuration/manage-connections-from-windows-operating-system-components-to-microsoft-services.md
Service: w10
GitHub Login: @brianlic-msft
Microsoft Alias: brianlic-msft

Duplicate of issue 874


Fix start position of intersection/union node

Fixes #30995

I am not sure how to write a baseline test for startPos, do we have  docs or examples?
parseOptionalToken use scanner.getStartPos() as start position. What about use scanner.getTokenPos() instead?

I am not sure how to write a baseline test for startPos, do we have  docs or examples?

Have a look at my attempt: #31265
For the test @rbuckton requested:

add a file to tests/cases/compiler with a comment // @declaration: true at the top and the desired content after that.
run gulp runtests-parallel (will fail)
run gulp baseline-accept (and verify the diff) now all tests should pass

parseOptionalToken use scanner.getStartPos() as start position. What about use scanner.getTokenPos() instead?

In my PR (linked above) I used scanner.getStartPos() to avoid creating a Token object that is never really used.

Hi. There is a discussion about how this should be fixed on the original issue #30995. Especially, this PR leaves the leading operator as floating if the second operator didn't exist.


Does World Locking Tools work with Scene Understanding?

Just a quick question as mentioned in the title, I was wondering if anyone has tried to use World Locking Tools and Scene Understanding, I couldn't find anything online about this topic, cheers.

Yes: World locking tools works by moving the player camera's parent node the same way you might implement teleporting the player to a different part of the map.
The player camera and scene understanding will both still get transform updates from the same coordinate system or XrSpace.


Can't call OpenScreen

OpenScreen can't be called in Unity's button components because of unrecognizable params.

See 04bfd393fbc5b9983ab42aa931c99f8935af8734.
Resolves method that can't be called in Unity's UI Engine.


MSIX version tearing causes instability and crashes

This bug represents a class of issues with the MSIX platform running mixed versions of our package at the same time. This violates fundamental assumptions of our code and causes instability and crashing in a variety of ways.
We are pursuing this issue internally with the MSIX platform, but this main bug will represent any other issues that get root-caused to this situation.

Now internal at MSFT:33501832

Terminal - unable to find the selected font error at startup - MSFT:32337657 was also sorta related to this. That was the font bug that I think we tried working around. The workaround is now causing #10211. That's less about a torn state, and more about a botched upgrade.

OOP COM Server object activation tearing - MSFT:33501832

This is being investigated internally, but this question was proposed internally. If anyone is seeing this and willing to capture this data, please let us know so we can connect you with the platform team investigators:
"If you encounter this can you export the contents of HKCU\Software\Classes\PackagedCom? I want to check the pattern against what I see on my machine."

I am seeing a crash on startup with this stack trace:
(3134.229c): Security check failure or stack buffer overrun - code c0000409 (!!! second chance !!!)
Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT
ucrtbase!abort+0x4e:
00007ffa`78fa286e cd29            int     29h
0:000> kn
 # Child-SP          RetAddr               Call Site
00 0000006f`cb0fcbe0 00007ffa`78fa1f9f     ucrtbase!abort+0x4e
01 0000006f`cb0fcc10 00007ffa`4dce1aab     ucrtbase!terminate+0x1f
02 0000006f`cb0fcc40 00007ffa`4dce2317     VCRUNTIME140_1!FindHandler<__FrameHandler4>+0x45b [D:\a01\_work\26\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 693]
03 0000006f`cb0fce10 00007ffa`4dce40d9     VCRUNTIME140_1!__InternalCxxFrameHandler<__FrameHandler4>+0x267 [D:\a01\_work\26\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 357]
04 0000006f`cb0fceb0 00007ff6`7384627c     VCRUNTIME140_1!__CxxFrameHandler4+0xa9 [D:\a01\_work\26\s\src\vctools\crt\vcruntime\src\eh\risctrnsctrl.cpp @ 306]
05 0000006f`cb0fcf20 00007ffa`7b3d217f     WindowsTerminal+0x1627c
06 0000006f`cb0fcf50 00007ffa`7b381454     ntdll!RtlpExecuteHandlerForException+0xf
07 0000006f`cb0fcf80 00007ffa`7b3811a5     ntdll!RtlDispatchException+0x244
08 0000006f`cb0fd690 00007ffa`78c94b89     ntdll!RtlRaiseException+0x185
09 0000006f`cb0fe4c0 00007ffa`4dcc6480     KERNELBASE!RaiseException+0x69
0a 0000006f`cb0fe5a0 00007ff6`7383639f     VCRUNTIME140!_CxxThrowException+0x90 [D:\a01\_work\26\s\src\vctools\crt\vcruntime\src\eh\throw.cpp @ 75]
0b 0000006f`cb0fe600 00007ff6`73839154     WindowsTerminal+0x639f
0c 0000006f`cb0fe660 00007ff6`7383b203     WindowsTerminal+0x9154
0d 0000006f`cb0fe7b0 00007ff6`7383f055     WindowsTerminal+0xb203
0e 0000006f`cb0fe800 00007ff6`73842b18     WindowsTerminal+0xf055
0f 0000006f`cb0fe8d0 00007ff6`738404e6     WindowsTerminal+0x12b18
10 0000006f`cb0fe9f0 00007ffa`7af4e858     WindowsTerminal+0x104e6
11 0000006f`cb0fea70 00007ffa`7af4e3dc     user32!UserCallWinProcCheckWow+0x2f8
12 0000006f`cb0fec00 00007ffa`7af62bf7     user32!DispatchClientMessage+0x9c
13 0000006f`cb0fec60 00007ffa`7b3d0c54     user32!__fnINLPCREATESTRUCT+0x87
14 0000006f`cb0fecc0 00007ffa`791b1ec4     ntdll!KiUserCallbackDispatcherContinue
15 0000006f`cb0fee38 00007ffa`7af47d8b     win32u!NtUserCreateWindowEx+0x14
16 0000006f`cb0fee40 00007ffa`7af47958     user32!VerNtUserCreateWindowEx+0x20f
17 0000006f`cb0ff1d0 00007ffa`7af477a2     user32!CreateWindowInternal+0x1a4
18 0000006f`cb0ff330 00007ff6`7384108c     user32!CreateWindowExW+0x82
19 0000006f`cb0ff3c0 00007ff6`738376a5     WindowsTerminal+0x1108c
1a 0000006f`cb0ff4a0 00007ff6`7383563d     WindowsTerminal+0x76a5
1b 0000006f`cb0ff690 00007ff6`738448f2     WindowsTerminal+0x563d
1c 0000006f`cb0ff840 00007ffa`799c7034     WindowsTerminal+0x148f2
1d 0000006f`cb0ff880 00007ffa`7b382651     KERNEL32!BaseThreadInitThunk+0x14
1e 0000006f`cb0ff8b0 00000000`00000000     ntdll!RtlUserThreadStart+0x21

From 1.8.1444.0. I think this is the same issue? Reg dump for the requested key is attached. I'm also happy to keep it in a broken state and continue to run/trace/dump more things. If there are instructions on how to trace an app package with ttd I can also do that -- just tracing wt.exe didn't seem to work.
wt-crash.txt


Gcs url reader

Hey, I just made a Pull Request!
This Pull request adds a new UrlReader for Google Cloud Storage. This can be used for registering components that are defined in a yaml blob stored in GCS.

What would be a good place to add documentation for this?
I'm not sure what would be the best way to test the actual integration with GCS in the context of this project. Should I write an integration test?

:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[x] Added or updated documentation
[x] Tests for new functionality and regression tests for bug fixes
[x] ~Screenshots attached (for UI changes)~
[x] All your commits have a Signed-off-by line in the message. (more info)

Thanks @Rugvip , I think I fixed the last comments now.


Extended vCardArray parser

Added telephone parsing and removed parser coupling with vCardArray data

Thanks for the PR. Looks good, the only issue I can see is that using TypedDict and the typing will mandate at least Py3.8. This is probably fine, but worth mentioning.

We can require typing_extension for 3.6-3.7

That would be an acceptable option. Do you want to add a patch to include typing_extension? If not I can add it before the next release.

Excellent, thanks! I'll bundle this into the next release in a day or two.