Change bsearch's / 2 to >> 1 for faster performance

Using the benchmark:
require "benchmark"

RANGE = (0..100_000_000)
ARRAY = RANGE.to_a

Benchmark.ips do |x|
  x.report("Array#bsearch") { ARRAY.bsearch { |number| number > 77_777_777 } }
  x.report("Range#bsearch") { RANGE.bsearch { |number| number > 77_777_777 } }
end

Before:
Array#bsearch   6.83M (146.38ns) (± 2.92%)  0 B/op        fastest
Range#bsearch   6.11M ( 163.7ns) (± 2.31%)  0 B/op   1.12× slower

After:
Array#bsearch   13.1M ( 76.35ns) (± 3.12%)  0 B/op        fastest
Range#bsearch   9.88M (101.23ns) (± 3.00%)  0 B/op   1.33× slower

I was surprised to see how much slower bsearch was than find (originally noticed in @konung's fast-crystal), so I'm looking into better algorithms.  Until I find one that works well, I figured I'd submit this simple win.  It uses the less expensive shift-right operation to do integer division by 2 as opposed to the division operation.  This change provides a 48% speedup in Array#bsearch and a 38% speedup in Range#bsearch

The reason is that division checks for zero and might raise. That makes it slower. I think there's also unsafe_div. But this is fine.

I had originally tried unsafe_div and it was slower than >> 1.  I had also tried unsafe_shr, but there was no change over >> 1

Actually /2 shouldn't be slower because it's a hardcoded value. This is strange...

I was actually surprised that LLVM didn't have a built-in optimization for / 2

FWIW:
/ 2
Array#bsearch   6.83M (146.38ns) (± 2.92%)  0 B/op        fastest
Range#bsearch   6.11M ( 163.7ns) (± 2.31%)  0 B/op   1.12× slower

>> 1
Array#bsearch   13.1M ( 76.35ns) (± 3.12%)  0 B/op        fastest
Range#bsearch   9.88M (101.23ns) (± 3.00%)  0 B/op   1.33× slower

unsafe_div(2)
Array#bsearch   9.09M (109.97ns) (± 2.65%)  0 B/op        fastest
Range#bsearch   7.35M (136.13ns) (± 3.28%)  0 B/op   1.24× slower

unsafe_shr(1)
Array#bsearch  13.06M ( 76.54ns) (± 3.66%)  0 B/op        fastest
Range#bsearch   9.92M (100.84ns) (± 3.46%)  0 B/op   1.32× slower

Signed division could be slower because it rounds up to zero (at least as LLVM docs say), and shift rounds to negative infinity. Benchmarking confirms that unsigned division has same speed and signed is 1.23 times slower.
On the other hand looks like signed division in Crystal always rounds to negative infinity - i have no idea how to explain it.

Just tried being more explicit by doing an unsigned div with / 2_u32 and /2_u8.  No change from regular / 2.  Updated my post above.

The left side is still signed. Can't currently check myself, but I'm sure that if you will convert left side to UInt32 (unsafely) and then do unsafe_div, performance of bsearch will match the one with >> 1.

@konovod Doesn't seem to have worked, but it's possible I did it wrong:
Array#bsearch   6.91M (144.62ns) (± 2.82%)  0 B/op        fastest
Range#bsearch   6.16M (162.37ns) (± 3.25%)  0 B/op   1.12× slower

Is this the "right" way to do it?
-      mid = from + ((to - from) >> 1)
+      mid = from + ((to - from).unsafe_as(Int32) / 2_u32)

unsafe_as(UInt32)

LOL oops..of course :smile:
So, yes, using .unsafe_as(Int32).unsafe_div(2_u32) has similar performance to >> 1
Array#bsearch  13.19M ( 75.81ns) (± 3.50%)  0 B/op        fastest
Range#bsearch   9.92M (100.76ns) (± 3.81%)  0 B/op   1.33× slower

I think that in this (and similar) places, >> 1 is easier to understand and looks better than converting to unsigned.
For a general case (unsigned division by, say, 3 is faster then signed too), maybe there should be something like unsafe_positive_div but this looks like a low priority and debatable thing.

Thank you @Fryguy


use tpu client next in validator

Problem
This is draft PR which based on https://github.com/anza-xyz/agave/pull/3515, it will be split into separate smaller PRs.
Summary of Changes

closed in favour of https://github.com/anza-xyz/agave/pull/3893


theme:create Command writes removed function getThemeConfigPath

Description
The command theme:create creates a base plugin class with the function getThemeConfigPath. The call to this function was removed, when the path to theme.json was hardcoded in https://github.com/shopware/platform/commit/a19789ce6add5a285e30db1555cda3835a34b8fd.
Possible Solution
Remove the following lines https://github.com/shopware/platform/blob/aeda071c35c334a2b3a6db5dc48ade8b7911f2bb/src/Storefront/Theme/Command/ThemeCreateCommand.php#L141-L144

Hello,
thank you for creating this issue.
I have opened a ticket on our Issue Tracker for you. See the ticket link: https://issues.shopware.com/issues/NEXT-7779
Please use this ticket to track the state of your issue.

Hi @s-diez
I have imported this issue to our internal ticket system.
If you don't want to wait for us to fix this an pull request is very welcome :)
If this is okay for you I'd like to close this issue here.
Sunny Reagrds
Pantrtxp

Okay thanks


feat: checked if controller enabled or not

Codecov Report

Merging #5343 (790778c) into main (5146942) will decrease coverage by 0.02%.
Report is 4 commits behind head on main.
The diff coverage is 0.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5343      +/-   ##
==========================================
- Coverage   80.06%   80.04%   -0.02%
==========================================
  Files         189      189
  Lines       13098    13101       +3
==========================================
  Hits        10487    10487
- Misses       2199     2201       +2
- Partials      412      413       +1

Files
Coverage Δ

.../27-interchain-accounts/controller/keeper/relay.go
66.66% <0.00%> (-8.34%)
:arrow_down:


Certificate pinning crash with prelease GMS

Since we shipped 2.7.4 in a hotfix release, we're receiving many reports of this crash, confined completely to the Nexus 5X, Nexus 6, and Nexus 6P on Android 6.0.1.
Any pointers would be appreciated.
Fatal Exception: java.lang.NullPointerException: Attempt to invoke virtual method 'java.security.cert.X509Certificate java.security.cert.TrustAnchor.getTrustedCert()' on a null object reference
       at com.squareup.okhttp.internal.tls.AndroidTrustRootIndex.findByIssuerAndSignature(AndroidTrustRootIndex.java:43)
       at com.squareup.okhttp.internal.tls.CertificateChainCleaner.clean(CertificateChainCleaner.java:68)
       at com.squareup.okhttp.internal.io.RealConnection.connectTls(RealConnection.java:207)
       at com.squareup.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:149)
       at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:112)
       at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184)
       at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
       at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
       at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
       at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
       at com.squareup.okhttp.Call.getResponse(Call.java:286)
       at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243)
       at com.banksimple.http.ApiInterceptor.intercept(ApiInterceptor.java:50)
       at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:232)
       at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205)
       at com.squareup.okhttp.Call.execute(Call.java:80)
       at com.banksimple.http.BaseApiClient.execute(BaseApiClient.java:94)
       at com.banksimple.http.BaseApiClient.execute(BaseApiClient.java:106)
       at com.banksimple.http.ClientSettingsApiClient.getAndroidSettings(ClientSettingsApiClient.java:28)
       at com.banksimple.service.ClientSettingsManagerIntentService.getClientSettings(ClientSettingsManagerIntentService.java:83)
       at com.banksimple.service.ClientSettingsManagerIntentService.onHandleIntent(ClientSettingsManagerIntentService.java:69)
       at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:66)
       at android.os.Handler.dispatchMessage(Handler.java:102)
       at android.os.Looper.loop(Looper.java:148)
       at android.os.HandlerThread.run(HandlerThread.java:61)

Info about our configuration:

We install the GMS security provider.
We pin a public key and not an entire cert like so:

    /**
     * Returns the {@code CertificatePinner} to be used with this client or null.
     */
    public CertificatePinner getCertificatePinner() {
        String path = getPinnedCertsPath();
        if (TextUtils.isEmpty(path)) {
            return null;
        }

        PublicKey publicKey;
        try {
            publicKey = getPubKey(mContext, path);
        } catch (Exception e) {
            SFLog.e("Failed to initialize certificate pinning!", e);
            return null;
        }

        CertificatePinner.Builder builder = new CertificatePinner.Builder();
        builder.add(getHostname(), sha1(publicKey));
        return builder.build();
    }

    /**
     * Generate a SHA-1 hash of a public key in the format expected by {@code CertificatePinner}
     */
    private static String sha1(PublicKey key) {
        return "sha1/" + Util.sha1(ByteString.of(key.getEncoded())).base64();
    }

    /**
     * Returns a public key loaded from the given path relative to the assets directory.
     * <p/>
     * The key must be stored in the {@code DER} format.
     */
    private static PublicKey getPubKey(Context context, String path) throws IOException,
            NoSuchAlgorithmException, InvalidKeySpecException {
        InputStream in = context.getAssets().open(path);

        // Read the key file
        byte[] bytes = new byte[in.available()];
        in.read(bytes);
        in.close();

        // Load the public key
        X509EncodedKeySpec spec = new X509EncodedKeySpec(bytes);
        KeyFactory keyFactory = KeyFactory.getInstance("RSA");
        return keyFactory.generatePublic(spec);
    }

Any chance you can reproduce this? It’s weird and bad that it can’t find the trusted cert that just succeeded the handshake.

@swankjesse: Not able to reproduce, but we have found that the affected users are running an Android security patch level of "March 1, 2016", and we have confirmed at least in two cases that the users are Googlers. Assuming the rest are not time travelers, I believe this is an issue with an upcoming change to the system JSSE provider.

We've had a Googler install a test build with a synchronous GMS SecurityProvider install, and that does not appear to solve the issue.

It appears the OS version reported may not be correct; we've only been able to confirm this crash occurs on "unreleased" OS builds, and the engineers we've been in contact with have declined to state their OS builds or versions :)

We've been seeing this issue as well. It increased very sharply on Friday Feb 12th, even though we hadn't released any new code for 2 weeks at that point. That build of our app was running okhttp v3.0.1. We've also seen this crash since in a couple of patch updates that we've pushed containing okhttp v3.1.1 and v3.1.2.
It seems likely that this crash is being caused by an incompatibility in a new version of Google Play Services's Security Provider that was pushed around Feb 12th. We were able to confirm with an affected user that his GPS version is 8.7.01, our devices running 8.4.89 are unaffected.
We are also doing sha1 cert pinning, but are not manually installing the GMS Provider. Please let me know if I can provide any additional info!

I've got a bit more info:

I shipped an affected user a build without cert pinning, there is no crash.
He also had a device running the new Android version, but without the latest GPS. this runs our current prod app without the crash.
Interestingly, he also ran the app on a Pixel C tablet with the latest GPS and our prod app, and still no crash. I'm really curious what's different about the Pixel C in this scenario.

Sorry if this is beating a dead horse at this point -- all signs point to the new GPS and using cert pinning.

Here from Android,
AndroidTrustRootIndex is using reflection to call conscrypt's TrustManagerImpl.findTrustAnchorBySubjectAndPublicKey, which will return null if the cert is not in the trusted set, which can be pretty common and totally normal, you should be checking that the return value isn't null before calling getTrustedCert().

Hey Chad, what's the best way to discover the trusted CA in this case?

The logic in CertificateChainCleaner.clean already looks like it correctly handles the case where the current cert is not the CA, but the current code will never have trustRootIndex.findByIssuerAndSignature actually return null since findTrustAnchorBySubjectAndPublicKey will return null, leading to a NPE when you call getTrustedCert.
Your current code should work so long as you return null if findTrustAnchorBySubjectAndPublicKey returns null.

How does it work without a trust root?

(We expect clients to be able to pin the root certificate)


aptly prevent any operation when no space left on device

Hi Andey,
after our normal mirror update operation got interrupted by a: "no space left on device" I tried to access to  the list of snapshot and cleanup the database but for some reason aptly cannot perform any operation when the disk is full:
[repo@repo ~]$ df -h | grep aptly
/dev/mapper/vg_aptly-lv_aptly
                      492G  492G   17M 100% /aptly
[repo@repo ~]$ aptly mirror list
ERROR: can't open database: write /aptly/ubuntu/.aptly/db/179528.ldb: no space left on device
[repo@repo ~]$ aptly db cleanup
Loading mirrors, local repos, snapshots and published repos...
ERROR: can't open database: write /aptly/ubuntu/.aptly/db/179528.ldb: no space left on device

This is kind of a dead end since I cannot even try to free some space, and it's probably not a good idea to wipe some packages from the pool.
So the only possibility in this case would be to expend the volume group...
any thoughts?
Best Regards
Eric

What File System is /aptly?
You could try this:
$ tune2fs -m 0 /dev/mapper/vg_aptly-lv_aptly

Well, aptly requires some free space to create log for the database (it happens while opening DB). You can move DB to some other partition and symink DB directory back.

@richard-scott /dev/mapper/vg_aptly-lv_aptly on /aptly type ext4 (rw)
@smira: the symbolic link did the trick.


Unable to use flatten + unflatten when original dict key names contain seperator char

Hi, I like this library but it has 1 flaw preventing me from using it. I am unable to flatten and unflatten dictionaries that include the separating character. I only noticed this because I use underscores in my key names a lot and that is the default separating char for this library.
Example:
from flatten_json import flatten, unflatten_list
import pprint

starter_dict = {
    'normal': 'kskdaskad',
    'nested': {'dict': 'asdasd'},
    'array': [1, 2, 3],
    'deeparray': [
        'string',
        {'key': 'val'},
        ['yet', 'another', 'list', ['moar']]
    ],
    'MY_KEY_WHICH_INCLUDES_UNDERSCORES': 'a single value'
}
print('-------------- original dict --------------')
pprint.pprint(starter_dict)
flat = flatten(starter_dict)
print('-------------- flat packed --------------')
pprint.pprint(flat)
unflat = unflatten_list(flat)
print('-------------- unpacked --------------')
pprint.pprint(unflat)

Output:
-------------- original dict --------------
{'MY_KEY_WHICH_INCLUDES_UNDERSCORES': 'a single value',
 'array': [1, 2, 3],
 'deeparray': ['string', {'key': 'val'}, ['yet', 'another', 'list', ['moar']]],
 'nested': {'dict': 'asdasd'},
 'normal': 'kskdaskad'}
-------------- flat packed --------------
{'MY_KEY_WHICH_INCLUDES_UNDERSCORES': 'a single value',
 'array_0': 1,
 'array_1': 2,
 'array_2': 3,
 'deeparray_0': 'string',
 'deeparray_1_key': 'val',
 'deeparray_2_0': 'yet',
 'deeparray_2_1': 'another',
 'deeparray_2_2': 'list',
 'deeparray_2_3_0': 'moar',
 'nested_dict': 'asdasd',
 'normal': 'kskdaskad'}
-------------- unpacked --------------
{'MY': {'KEY': {'WHICH': {'INCLUDES': {'UNDERSCORES': 'a single value'}}}},
 'array': [1, 2, 3],
 'deeparray': ['string', {'key': 'val'}, ['yet', 'another', 'list', ['moar']]],
 'nested': {'dict': 'asdasd'},
 'normal': 'kskdaskad'}

I understand I could use the separator param and make it something really weird, but I am always running the risk of ruining my original dict structure if (heaven forbid) any of my key names include that same string pattern. Maybe quote encapsulation could help here?
Fixing this issue would make the library more plug-and-play since I am sure I am not the only one that likes to use underscores in their JSON key names.

I've created a PR for allowing automatic replacing of separators if they are found in any keys.
This might help with your issue (depending on what you need)
https://github.com/amirziai/flatten/pull/61


Add goreleaser supply chain template

https://github.com/goreleaser/goreleaser-example-supply-chain

@riteshnoronha , are you working on this ?


Reorganize MoE experts distribution logic to support multiple strategies.

Describe your changes
Reorganize MoE experts distribution logic to support multiple strategies.
Also, generate ranked models parallelly to improve performance.
Checklist before requesting a review

[ ] Add unit tests for this change.
[ ] Make sure all tests can pass.
[ ] Update documents if necessary.
[x] Format your code by running pre-commit run --all-files
[ ] Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

(Optional) Issue link

/azp run


Update script for composer.

Largely based on the work @ https://github.com/docker-solr/docker-solr/blob/master/update.sh

Tested so far by deleting the following directories then re-generating:
rm -rf 1.0.1
./update 1.0.1
git diff

rm -rf 1.0.0-beta2
./update 1.0.0-beta2
git diff

Looks good! Anything else that we need? Thinking it's merge time!  :+1:

I ran a few tests and seems to be working.
a) Scenario 1 we specify 7.0 so it will look for latest of that branch so finds 7.0.0
b) Scenario 2 we specify 7 so it will look for latest of that branch so find 7.3.0
c) Scenario 3 we specify 7.0.0-rc2 it will look for exactly that branch and use it
Scenario 1
$ ./update 1.1.0

diff --git a/1.1.0/Dockerfile b/1.1.0/Dockerfile
new file mode 100644
index 0000000..c906da9
--- /dev/null
+++ b/1.1.0/Dockerfile
@@ -0,0 +1,11 @@
+# Composer Docker Container
+FROM composer/composer:base
+MAINTAINER Rob Loach<EMAIL_ADDRESS>+
+ENV COMPOSER_VERSION 1.1.0
+
+# Install Composer
+RUN php /tmp/composer-setup.php --no-ansi --install-dir=/usr/local/bin --filename=composer --version=${COMPOSER_VERSION} && rm -rf /tmp/composer-setup.php
+
+# Display version information.
+RUN composer --version
diff --git a/1.1.0/alpine/Dockerfile b/1.1.0/alpine/Dockerfile
new file mode 100644
index 0000000..02d8d6d
--- /dev/null
+++ b/1.1.0/alpine/Dockerfile
@@ -0,0 +1,11 @@
+# Composer Docker Container
+FROM composer/composer:base-alpine
+MAINTAINER Rob Loach<EMAIL_ADDRESS>+
+ENV COMPOSER_VERSION 1.1.0
+
+# Install Composer
+RUN php /tmp/composer-setup.php --no-ansi --install-dir=/usr/local/bin --filename=composer --version=${COMPOSER_VERSION} && rm -rf /tmp/composer-setup.php
+
+# Display version information.
+RUN composer --version

Scenario 2
$ ./update 1.0.1

no changes

Scenario 3
$ ./update 1.1.0-RC

diff --git a/1.1.0-RC/Dockerfile b/1.1.0-RC/Dockerfile
new file mode 100644
index 0000000..aa5c8a3
--- /dev/null
+++ b/1.1.0-RC/Dockerfile
@@ -0,0 +1,11 @@
+# Composer Docker Container
+FROM composer/composer:base
+MAINTAINER Rob Loach<EMAIL_ADDRESS>+
+ENV COMPOSER_VERSION 1.1.0-RC
+
+# Install Composer
+RUN php /tmp/composer-setup.php --no-ansi --install-dir=/usr/local/bin --filename=composer --version=${COMPOSER_VERSION} && rm -rf /tmp/composer-setup.php
+
+# Display version information.
+RUN composer --version
diff --git a/1.1.0-RC/alpine/Dockerfile b/1.1.0-RC/alpine/Dockerfile
new file mode 100644
index 0000000..34d8969
--- /dev/null
+++ b/1.1.0-RC/alpine/Dockerfile
@@ -0,0 +1,11 @@
+# Composer Docker Container
+FROM composer/composer:base-alpine
+MAINTAINER Rob Loach<EMAIL_ADDRESS>+
+ENV COMPOSER_VERSION 1.1.0-RC
+
+# Install Composer
+RUN php /tmp/composer-setup.php --no-ansi --install-dir=/usr/local/bin --filename=composer --version=${COMPOSER_VERSION} && rm -rf /tmp/composer-setup.php
+
+# Display version inform

Nicely done, @sylus! I've added you as a contributor on the project, feel free to issue PRs and such, or merge PRs as needed. :+1:

That is awesome! Love to help however I can :) Thx so much!


--shell tests failing for specific commands

Most mysterious bug ever.
Expected:

dockerized --shell <command> -c env should print the current environment

Actual:

In some specific conditions, there is no output from the container at all, and dockerized just exits.
Run bin/dockerized -v --shell perl -c env | tee ~/shell.log
Dockerized root: /home/runner/work/dockerized/dockerized
Loading: '/home/runner/work/dockerized/dockerized/.env'
Compose files: /home/runner/work/dockerized/dockerized/docker-compose.yml
Setting up shell in container for perl...
Entrypoint: [/bin/bash]
Command:    [-c env]
Preparing compose environment...
Network dockerized_default  Creating
Network dockerized_default  Created
Volume "dockerized_node_modules"  Creating
Volume "dockerized_node_modules"  Created
Volume "dockerized_python_modules"  Creating
Volume "dockerized_python_modules"  Created
Volume "dockerized_pip_cache"  Creating
Volume "dockerized_pip_cache"  Created
Volume "dockerized_go_cache"  Creating
Volume "dockerized_go_cache"  Created
Volume "dockerized_home"  Creating
Volume "dockerized_home"  Created
Running one-off container...
Container exited with code 0.

Observations:

These shell tests worked perfectly in march, and suddenly broke.
Many official docker images were updated under the same tags. Alpine and busybox were changed.
Affected commands:

go, python, perl, rustc

Still works perfectly locally on windows
It only breaks the first time. Running the command twice makes it work somehow.
Running docker-compose before dockerized also causes the --shell test to work.
For some commands, the issue is with bash. Switching to sh for --shell fixes the bug.
sh is not a universal fix. rustc uses sh and still doesn't work.
Was able to reproduce once on local Ubuntu VM. Cannot reproduce after. Even docker system prune -af won't help to reproduce it, running without a tty. (true | dockerized --shell perl -c env | cat)
docker-compose run --rm <command> always seems to work.

Hypothese (and motivation):

Something related to setting up the networks / compose project

There could be small implementation differences between docker-compose run and dockerized
Since docker-compose seems to work without failure, there must be something different causing the bug.

Something TTY related

Github Actions have no TTY whereas local running from a shell will provide a TTY, explaining reproduction difficulty
:x: Then why does it work the second time?

Related to updates in Bash and Alpine in official docker images

This did work a month ago and suddenly broke. Re-ran an old successful action and it didn't work anymore.
Specifying the exact alpine version in the docker image tag has fixed the issue for some commands. Unfortunately, the old digests are not published by dockerhub, so it's not possible to restore all commands. Also, this is not a good long term solution.
:x: Then why does it work the second time?

Something specifically related to the env command

Excluded causes:

Related to pulling the images the first time.

:x: Added dockerized --pull. Running this before the shell test doesn't make a difference.

Related to a race condition in creating the network.

:x: The issue occurs consistently with a specific subset of commands and is not unpredictable

Another weird observation:

When running the --shell option 2x, all commands succeed (except zip, for different reasons).
When running first --shell -c 'echo hello world' before the env test, all commands succeed, except 'perl'.


The training speed of pervasive attention model

Hello,
I am trying to run the pervasive attention model recipes as recommended in the README:
https://github.com/elbayadm/attn2d/blob/master/examples/pervasive/README.md
Based on my observation, the model is too slow with almost 30 words per second on a single GPU 1080! May you please give me a rough idea of what should be the speed expectations based on your experiments?
Thanks for your insights
Parnia

Hi Parnia,
When training on a single GPU (a Titan XP), the speed is approximately 880 wps.
In this log excerpt I'm training with a batch size of 348 tokens and I accumulate the gradients from 23 passes.
| epoch 016 | valid on 'valid' subset | loss 4.701 | nll_loss 3.093 | ppl 8.53 | num_updates 9760 | best_loss 4.70069 | epoch 017 | loss 4.686 | nll_loss 3.066 | ppl 8.38 | wps 883 | ups 0 | wpb 5895.925 | bsz 261.884 | num_updates 10370 | lr 0.000310535 | gnorm 0.585 | clip 0.000 | oom 0.000 | wall 60939 | train_wall 68947 | WARNING: 21 samples have invalid sizes and will be skipped, max_positions=(100, 100), first few sample ids=[5046, 4961, 1344, 1190, 6924, 371, 4174, 4044, 6636, 6746]
I do aim for 8K tokens per update (max-tokens * update-freq * distributed-world-size) and try to adapt the batch size to my resources.
The max-source-positions and max-target-positions parameters can also help reduce the memory load and speed up the training. In the recommended recipe, those are set to 100 tokens and that works well with the previously described batching settings. You can lower them down and that would help at the cost of a worse accuracy (NLL or BLEU).
In any ways, 30 wps is extremely slow. Try these tweaks and let me know if they help.
Best,
Maha


Selecting an answer : verify and feedback

@Mones-Hamd When I don't answer a question currently, and just click 'next question' it shows an error that I should give an answer, but it also show a red background that I gave a false answer, could you review the nextQuestion function to see why both of these things happen? I should only see the error message.


Faker localization 'en_ZA' not rendering $firstNameFemale for names with salutation

The faker localization 'en_ZA' not rendering $firstNameFemale for names with salutation. I am getting the following outputs in my database.
Miss {{$firstNameFemale}} Munyai
Dr. {{$firstNameFemale}} Watson
Mrs. {{$firstNameFemale}} Hattingh

Bug spotted and fixed. :)


Fix for dvorak layout detection

Layout detection works perfect if replace
if (!(d = XOpenDisplay(NULL))) {
with
if (!(d = XkbOpenDisplay(getenv("DISPLAY"), NULL, NULL, NULL, NULL, NULL))) {
on line 200 in dvorak.c.

Well, that's strange. Layout detecting works if i run the script from cli after login, but if i install and reboot it doesn't. Solution for me is to autostart script in .xprofile

Awesome, just implemented it and for me it works. I added ":0" as a default when env is not set

Same for me after reboot, layout detection not working on startup, when running from shell, all ok:
dvorak[1001]: No protocol specified
dvorak[1001]: cannot open display (default)

Improved now layout detection, using gdbus to get current layout from gnome-shell


[Docs-Bug]: sessionExpiresIn configuration docs - 'active period' and 'idle period' links dead

Describe the bug
Here is the page with the issue. Clicking on the active period and idle period links lead to a 404.

Are you aware if they have dedicated pages yet?

Doesn't look like it. This is the closest I found.

The definitions there don't seem to be too extensive, might not even be worth having a separate section for them each. Could potentially just include the description inside the sessionExpiresIn section or link it to the session states section in a nice way somehow?
Unless it could be expanded upon, ofcourse.

Yes, I could open a pull request for that. I think the links originally pointed to a similar section, but the page was moved from /start-here/concepts to /basics/sessions. Should be an easy fix.


When user turns off peer, peer getting doubled.

Steps to reproduce:
Just turn off registered peer.

Test1 and Libvirt are in my machine.

Failed in master

Here is another example:

Brouht two peers up. Turned them off.
Result:
Peer Disapp get doubled.

Stopped and started again the peers with different providers
Fixed in master

Could not reproduce on prod.
Peers did not get doubled.
Tested in prod.


Prepare composer, assign type of wordpress-plugin

This way when installing this in a composer-based WP project (eg. Bedrock) - it will know it's a plugin and install the folder correctly under the wp-content/plugins directory.
[ref. daggerhartlab.com/how-to-setup-wordpress-plugin-composer-packagist]

This PR seems to have a lot of depencies that I don't need...

This PR seems to have a lot of depencies that I don't need...

This PR seems to have a lot of depencies that I don't need...

Whoah, hmm. I was too quick.

This PR seems to have a lot of depencies that I don't need...

Whoah, hmm. I was too quick.

Thanks!
Meanwhile I fixed some things myself 👍

Thanks!
Meanwhile I fixed some things myself 👍

O, and the WordPress repo is at spatie/wordpress-ray

O, and the WordPress repo is at spatie/wordpress-ray

spatie/wordpress-ray

You see. Clouded. LMFAO - I'm going to close Github - sorry :x

spatie/wordpress-ray

You see. Clouded. LMFAO - I'm going to close Github - sorry :x


add frame time to newFrameReadyAtTime block argugment

Hi Brad, I think I want to have newFrame and frameTime in GPUImageRawDataOutput class.
Please review.
Regards,
takecian

I were looking for the same thing 👍


[BUG] Upgrade: stuck at pre-drained state

Describe the bug

We have spotted some situations instance-manager pod can't be drained out of a node:

Longhorn is being upgraded, and there are 2 instance-manager-r pods on a node.
A node is just drained and comes back; if we drain the next node, there is a high probability the next node is stuck.

To Reproduce
Steps to reproduce the behavior:

Upgrade to master ISO from v1.0.3. (which bumps Longhorn to 1.3.1).
Many users report upgrades being stuck in pre-drain problems.

Expected behavior

The upgrade should not be stuck.
Support bundle

Environment

Harvester ISO version: v1.0.3
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630):

Additional context
Add any other context about the problem here.

We can test this when other upgrade works are done.

[x] single node cluster upgrade

Tested in 3 node ipxe-examples vagrant upgrade scenario with Harvester v1.0.3 -> v1.1.0-rc2. I did this with a Harvester v1.1.0-rc2 iso stored on the ipxe server.
supportbundle_63f35a4d-17ba-4f4c-bb61-9e58685a5305_2022-10-11T01-41-18Z.zip
apiVersion: harvesterhci.io/v1beta1
kind: Version
metadata:
  name: v1.1.0-rc2
  namespace: harvester-system
spec:
  isoChecksum: 1d95b8808412e06916c48bed7939b002608f8c1d343bfadfc06acfd9d226e3272f961d63566a5f31d770e0d30dfdd4e5ef4edd1877e4d45e9a6efaa787ee4ac4
  isoURL: http://<IP_ADDRESS>/harvester/harvester.iso  # change to local ISO URL
  releaseDate: '20221007'

@noahgildersleeve Thanks. I apologize I didn't put a test plan (updated), and the description doesn't describe the fix well. The issue tracks the logic to wait for all volumes to become healthy before draining a node. So from the log, the code does its job well:
2022-10-11T01:41:28.374457669Z ++ kubectl get volumes.longhorn.io/pvc-6819374b-6d44-4225-9ffb-3e15a605130a -n longhorn-system -o 'jsonpath={.status.robustness}'
2022-10-11T01:41:29.570537863Z + robustness=degraded
2022-10-11T01:41:29.570579244Z + '[' degraded = healthy ']'
2022-10-11T01:41:29.570608024Z + '[' -f /tmp/skip-pvc-6819374b-6d44-4225-9ffb-3e15a605130a ']'
2022-10-11T01:41:29.570653114Z + echo 'Waiting for volume pvc-6819374b-6d44-4225-9ffb-3e15a605130a to be healthy...'
2022-10-11T01:41:29.570663824Z + sleep 10
2022-10-11T01:41:29.570677074Z Waiting for volume pvc-6819374b-6d44-4225-9ffb-3e15a605130a to be healthy...
2022-10-11T01:41:39.571498800Z + '[' true ']'
2022-10-11T01:41:39.571566571Z + '[' 3 -gt 2 ']'
2022-10-11T01:41:39.572013754Z ++ kubectl get volumes.longhorn.io/pvc-6819374b-6d44-4225-9ffb-3e15a605130a -n longhorn-system -o 'jsonpath={.status.robustness}'
2022-10-11T01:41:40.671983742Z + robustness=degraded
2022-10-11T01:41:40.672040863Z + '[' degraded = healthy ']'
2022-10-11T01:41:40.672066313Z + '[' -f /tmp/skip-pvc-6819374b-6d44-4225-9ffb-3e15a605130a ']'
2022-10-11T01:41:40.672073273Z + echo 'Waiting for volume pvc-6819374b-6d44-4225-9ffb-3e15a605130a to be healthy...'
2022-10-11T01:41:40.672101833Z Waiting for volume pvc-6819374b-6d44-4225-9ffb-3e15a605130a to be healthy...
2022-10-11T01:41:40.672110813Z + sleep 10

As for the stuck, there might be some reasons:

https://github.com/harvester/harvester/issues/2920 (should be fixed in v1.1.0-rc3)
Lack of CPU resources. This PR increases CPU amount: https://github.com/harvester/ipxe-examples/pull/54/files

Could you try applying the patch and see if you can see it in v1.1.0-rc3? Thanks.

@noahgildersleeve Could you give this a try again? I believe the fix is already available in v1.1.0.

As discussed in https://github.com/harvester/harvester/issues/2833#issuecomment-1306095354
The question is:  we are targeting v1.0.3 to v1.1.0/v1.1.1 upgrade, how can such check be effective in v1.0.3 version ? should the ugprade script itself checks such condition in the very beginning of upgrade, fail asap, to avoid the upgrade stucking in middle state ?
As v1.0.3 has no check per https://github.com/harvester/harvester/issues/2833, the possible way is let the upgrade script itself to do such check befeore applying any change.  Fail at beginning.
From v1.1.0/v1.1.1, the webhook will block when upgrade condition is not met.  But still if some new conditions are added in  e.g. v1.2.0, the check should still be done by the upgrade script itself at beginning, v1.1.1 webhook has no knowledge of those new conditions.


[DOC]: Is there documentation on how to create hostfiles

📚 The doc issue
According to the distributed training startup instructions on the documentation,
colossalai run --nproc_per_node 4 --hostfile ./hostfile --master_addr host1  test.py

I need to create a hostfile file where each row is the parameter configuration for a node,
host1
host2

but is there any documentation or example that explains what parameters are for each row and column of hostfile? I don't saw any instructions for that,  such as ip, port, gpu id, etc

thank you for your reply. what I also want to ask is how to write hosts.txt? for example, each line in hosts.txt contains the following information:
ip gpu_ids port
xx.xxx.xx.22 0,1,2,3 9999
xx.xxx.xx.23 0,1,2,3 8888

but I'm not sure what the canonical way is, it doesn't seem to be documented

You should only put in the IPs. I may update the docs when available, or you can submit a PR


get_solarpostion in modelchain.prepare_inputs does not use weather data.

Problem
get_solarpostion method in modelchain.ModelChain.prepare_inputs does not use weather data.
To Reproduce
change line 745 - 746
755:       self.solar_position = self.location.get_solarposition(
756:            self.times, method=self.solar_position_method)

to
755:       self.solar_position = self.location.get_solarposition(
756:            self.times, method=self.solar_position_method,
757:            pressure=self.weather.pressure,
758:            temperature=self.weather.temp_air)

Versions:

pvlib.__version__:  0.5.2
pandas.__version__: 0.23.3
python: 3.5.2

Good idea to pass this data on to get_solarposition if available. What about the case where one or more is not available? Maybe use pvlib.tools._build_kwargs?
Pull request welcome. Might not happen otherwise.

thank you for response.
ok, I consider how to do that.
If I found any idea, I share this.

@kenkuwae thanks for opening the issue.  I am re-opening so that we can keep this enhancement on the list.

@kenkuwae  @cwhanse  I have taken up this issue, and fixed the changes and applied for a PR.

closed by #936, but possibly a bug.


OS X + gcc (was: Node-gyp rebuild error)

Hi @lovell was wondering if you could help me, trying to make relax support node 4, and I'm stuck on sharp build, getting the following:
CXX(target) Release/obj.target/sharp/src/common.o
c++: error: unrecognized command line option '-stdlib=libc++'
make: *** [Release/obj.target/sharp/src/common.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/brunomota/.nvm/versions/node/v4.1.2/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:270:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Darwin 15.0.0
gyp ERR! command "/Users/brunomota/.nvm/versions/node/v4.1.2/bin/node" "/Users/brunomota/.nvm/versions/node/v4.1.2/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/brunomota/Sites/relax/node_modules/sharp
gyp ERR! node -v v4.1.2
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok
npm ERR! Darwin 15.0.0
npm ERR! argv "/Users/brunomota/.nvm/versions/node/v4.1.2/bin/node" "/Users/brunomota/.nvm/versions/node/v4.1.2/bin/npm" "i" "sharp"
npm ERR! node v4.1.2
npm ERR! npm  v3.3.6
npm ERR! code ELIFECYCLE

I'm using gcc version 4.8.4, am I missing something? Not my strong this area :disappointed:
Thanks!

Hi Bruno, "Darwin 15.0.0" tells me you're using OS X 10.11, for which sharp assumes xcode/clang is being used rather than gcc.
Are you able to test the following change to binding.gyp locally to see if this works with gcc?
     'xcode_settings': {
       'OTHER_CPLUSPLUSFLAGS': [
-        '-std=c++11',
-        '-stdlib=libc++',
         '-fexceptions',
         '-Wall',
         '-O3'
       ],
+      'CLANG_CXX_LIBRARY': 'libc++',
+      'CLANG_CXX_LANGUAGE_STANDARD':'c++11',
       'MACOSX_DEPLOYMENT_TARGET': '10.7'
     },

Just tried, added as a git dependency from this branch https://github.com/bruno12mota/sharp, the error is the same though. Should I look into updating clang instead? looking for a solution which would work the best to give as simple instructions for the users

By the way my clang version, if needed:
Apple LLVM version 7.0.0 (clang-700.0.72)
Target: x86_64-apple-darwin15.0.0
Thread model: posix

Did you recently upgrade to OS X 10.11? I've heard from a couple of sources that xcode might need to be reinstalled via xcode-select --install.

Will try it out now, thanks so much for your help! Will let you know if it works

Still no luck, updated xcode:
Apple LLVM version 7.0.0 (clang-700.1.76)
Target: x86_64-apple-darwin15.0.0
Thread model: posix

But getting the exact same error, tried with the changed fork, and with the current sharp version. Will try a few things out, will let you know if I can find a solution

Tried removing the gcc version i had installed, so using the default xcode clang version, still errors out but it is a different one, any idea?
 System Darwin 15.0.0
gyp ERR! command "/Users/brunomota/.nvm/versions/node/v4.1.2/bin/node" "/Users/brunomota/.nvm/versions/node/v4.1.2/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/brunomota/Sites/relax/node_modules/sharp
gyp ERR! node -v v4.1.2
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok

Please can you provide the output npm install --verbose, perhaps as a gist.
https://github.com/node-inspector/node-inspector/issues/760#issuecomment-154361714 might be relevant too.

https://gist.github.com/bruno12mota/efa055e9f02553fcb2d5 here it is, the sharp build part only. I'll take a look to the node-inspector issue

OH, got it! made a npm install touch as someone said in the node-inspector issue and it compiled perfectly :smile:
Cheers, thanks so much for your help!

Glad to hear you got everything working again after the upgrade.
I'm not sure how many people use the "proper" gcc instead of the default clang's gcc-like wrapper on OS X. Supporting both might be more of a gyp responsibility anyway.
There is a (very) basic OS X 10.9 + Node v0.10 test environment for sharp e.g. https://travis-ci.org/lovell/sharp-osx-ci/builds/90407217 and I hope to improve that situation via #248.
Closing this for now - feel free to re-open if this problem occurs again.

@bruno12mota can you explain what issue was caused regarding npm install touch and how you fixed it?


Additional language support for mnemonics

I have no experience thus I am asking directly: Is it possible to add different languages (like Turkish or German) for mnemonic words?  If so, how?

@DoDzilla-ai,
Yes, It's possible to use different languages but the language it needs confirmation. On this library, the languages are dependent on python-mnemonic library. You can also add languages by creating pull request on python-mnemonic wordlists repo.


I can not import "Wolfy87EventEmitter" as "EventEmitter" into my project

Hello @evmyann
I am try to your project for a little typescript project.  I have a problem  when importing "Wolfy87EventEmitter" as dependency.
Error is here:
[ts] 'EventEmitter' refers to a UMD global, but the current file is a module. Consider adding an import instead.
My 'vendor.ts':
// Application Dependencies

import 'lodash';
import "wolfy87-eventemitter";

My 'vendor.d.ts':
// Application Dependencies

import 'lodash';
import "wolfy87-eventemitter";

My manager.test.ts

Hi @muratsplat ! Thanks for using the CLI and reporting this issue 👍🏽
In fact you can solve your issue in 2 ways and this stackoverflow post brings a good highlight on how to do it: https://stackoverflow.com/questions/39415661/what-does-resolves-to-a-non-module-entity-and-cannot-be-imported-using-this
1- (Quick) Use the CommonJS-style import syntax to use this module
import CustomEventEmitter = require('wolfy87-eventemitter');
const ee = new CustomEventEmitter();

2- (Preferred) Update your tsconfig.json with the property allowSyntheticDefaultImports set to true
import CustomEventEmitter from 'wolfy87-eventemitter';
const ee = new CustomEventEmitter();

I will amend the generated tsconfig.json with this property so we'll be able to import UMD package in the future.
Thanks!


Run internal modules and scripts in isolated manner.

Use runpy to run script or module without adding CWD to sys path.
add method to PythonExecutionService, etc. to use that script
update calls for internals scripts to use the new method

This is done now!


Update C# Examples on functions-bindings-event-hubs-trigger.md to Make Functions Instances

Removed the static qualifier from the C# examples to make the functions so they are instance methods.
The idea being that it will be easier to add dependency injection later on.

@ssfcultra : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#label:"aq-pr-triaged"

Bumping this. What are the next steps to getting this completed?

@ggailey777
Could you review this proposed update to your article and enter #sign-off in a comment if it's approved for merge?
Thanks!

Greetings all,
This PR is super simple and has been sitting around for over a month. It's a bit demotivating to contribute to MSFT docs if it's going to take this long. Especially as I am interested in helping maintain docs as a contribution to the community.
Thanks!

@mijacobs
Can you review the proposed changes? IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.

I'm checking with the product team if this change is best in all cases. If so, we should also update the template. I'll ping them again about this.

I checked with the product team, and they agree with your assertion and want to find a way to update all the C# in-process templates to support instance methods to make it easier for DI. In the meantime, I'll accept this PR, but will still be many, many static examples in our docs that would also need to be updated to use instance methods.  #sign-off


Form Components - Typing Error

Error:
Binding element 'field' implicitly has an 'any' type
Code
"use client"

import { zodResolver } from "@hookform/resolvers/zod"
import { useForm } from "react-hook-form"
import * as z from "zod"

import { Button } from "@/components/ui/button"
import {
  Form,
  FormControl,
  FormDescription,
  FormField,
  FormItem,
  FormLabel,
  FormMessage,
} from "@/components/ui/form"
import { Input } from "@/components/ui/input"

const formSchema = z.object({
  username: z.string().min(2, {
    message: "Username must be at least 2 characters.",
  }),
})

export function ProfileForm() {
  // ...

  return (
    <Form {...form}>
      <form onSubmit={form.handleSubmit(onSubmit)} className="space-y-8">
        <FormField
          control={form.control}
          name="username"
          render={({ field }) => (
            <FormItem>
              <FormLabel>Username</FormLabel>
              <FormControl>
                <Input placeholder="shadcn" {...field} />
              </FormControl>
              <FormDescription>
                This is your public display name.
              </FormDescription>
              <FormMessage />
            </FormItem>
          )}
        />
        <Button type="submit">Submit</Button>
      </form>
    </Form>
  )
}

Hi @im-py-dev, is this the entire code you are trying to use in your project? Currently what you have written is an incomplete part of an example from the shadcn docs. From the rest of the guide, you at least need to add in the following (replace the line // ...):
  // 1. Define your form.
  const form = useForm<z.infer<typeof formSchema>>({
    resolver: zodResolver(formSchema),
    defaultValues: {
      username: "",
    },
  })

  // 2. Define a submit handler.
  function onSubmit(values: z.infer<typeof formSchema>) {
    // Do something with the form values.
    // ✅ This will be type-safe and validated.
    console.log(values)
  }
}

field should be fully typed by the resolver schema passed to useForm.

Thank you @ChrisCoastal , I'm actually added them but in VSCode I see that error, in build works

Anyone found a solution for this btw?


Render the favicon

I added <link rel="icon" href="favicon.png"> yet it didn't render in the Live Preview.

The embedded browser doesn't support displaying favicons yet, but the external browser should correctly do so.
Embedded browser:

External browser:

Are you talking about rendering in the embedded browser or external browser?

embedded @andreamah

Yup, that's expected. I'll add this to the feature backlog.


Support for ARM in openblas

Motivation for this change
Attempting to build openblas on ARM resulted in the error "unsupported system: armv7l_linux". This PR resolves that issue and some other issues that pop up when trying to build openblas for ARM.
This shouldn't change the package for other architectures.

Could this also be cherry picked on release-16.09 please? I would like to install NixOS-16.09 with openblas on my Raspberry Pi.

Done!

Great, thanks!


2024-12-11_01-36-07_master-next_aws-lc

Automatically created.

Backport failed for kirkstone-next, because it was unable to cherry-pick the commit(s).
Please cherry-pick the changes locally and resolve any conflicts.
git fetch origin kirkstone-next
git worktree add -d .worktree/backport-10708-to-kirkstone-next origin/kirkstone-next
cd .worktree/backport-10708-to-kirkstone-next
git switch --create backport-10708-to-kirkstone-next
git cherry-pick -x 9353989a267527ca32d0d93cd632d9f2d9d23028

Successfully created backport PR for scarthgap-next:

#10714

Successfully created backport PR for styhead-next:

#10715


Setup single sign-on for github, aws, dockerhub, etc

https://github.com/jupyterhub/oauthenticator

complete


Can't run main after migrating to the v0.10

After upgrading to the vite-plugin-electron to the version v0.10.3 I have got a following error.

I was unable to track down dependecy/module which may cause this behaviour, but I tested various versions of vite and this plugin, and only changing version of this plugin to v0.9 fixes the issue.

Packages that i use

Options

Can you provide a minimal reproduction Demo?

Can you provide a minimal reproduction Demo?

I will try provide one ASAP

Fixed by v0.11


Login by name? Not slug?

Hi,
I've had a lot of trouble logging in to Journal after setting it up and playing with the database. It took me some time to figure out, but I wonder if you shouldn't think through the login process again?
It currently checks the user's name and password, but the name field is a blob in the database which I think is a bit strange for a login value... And even more, on the Admin page it seems pretty clear that Name can be changed to something different, for instance the full name of the user, instead of just a short username.
So after I had changed the Name field to a long, proper name, I suddenly found myself unable to log back in. :-)
Would you consider modifying the login procedure to check the slug or email fields instead of the name field, or possibly allow a match on either of the three fields? I'd be happy to do the change for you, but I'm not sure if the current process is by design or not. In case you'd like to keep checking against the name field, I think it should be specified a bit clearer on the login page which data to input.
Also, I think this is an awesome project. I hope to start contributing a bit within the next few days.
Best regards,
Tore M. Havn

Hi Tore,
I agree - that is confusing.
I plan on revamping the login system in the near future, possibly using something like https://github.com/go-authboss/authboss that can handle password reset via email.
I was planning on adding authentication via email in addition to name then, but you are more than welcome to change it right now :)
I think I'll even make email the official login method like Ghost does. That way there is no confusion when the user changes his name.
I'm not so sure about slug. The slug is something the typical user won't have any knowledge of (even though it is basically just his user name in lower case). I don't think it's necessary once email is implemented.
Regarding the blob field in the database: That was a (debatable) design choice. All DB fields that might be output by a ghost helper (in a theme) are stored like this because they are handled as []byte in Journey (string concatenation is done via bytes.Buffer).
It may have been better to convert the values into strings again before storing them in the database, but that wasn't on my radar at the time. But since all of those DB fields are read into []bytes by Journey, it might even be faster this way. I didn't do benchmarks on that.
Thanks for your input. I would love to have you contribute :)
Best,
Kai


Add max_width option for all heuristics.

This is useful when working with very small max_widths like 79 chars. I agree with this comment but rustfmt's heuristics generate a lot of vertical space in https://github.com/xiph/rav1e project, which uses  a max_width of 79 chars.

Thanks!


[NEW ITEM]  Bottle of Sand

Item Name

Bottle of Sand

Author

iceyou

Description

I don't like sand, it's annoying and gets everywhere

Item

minecraft:splash_potion

NBT Data
,tag:{CustomPotionColor:15384463,CustomPotionEffects:[{Ambient:0b,Amplifier:127b,Duration:2147483647,Id:11b,ShowIcon:1b,ShowParticles:1b},{Ambient:0b,Amplifier:0b,Duration:2147483647,Id:7b,ShowIcon:1b,ShowParticles:1b},{Ambient:0b,Amplifier:18b,Duration:2147483647,Id:7b,ShowIcon:1b,ShowParticles:1b}],HideFlags:32,Potion:"minecraft:empty",display:{Lore:['{"text":"Sand II","color":"red","italic":false}'],Name:'{"text":"Bottle of Sand","color":"reset","italic":false,"hoverEvent":{"action":"show_text","contents":[{"text":"Bottle of Sand\\n","color":"reset","italic":false},{"text":"Sand II\\n","color":"red","italic":false},{"text":"minecraft:bottle_of_sand\\nNBT: 1 tag(s)","color":"dark_gray","italic":false}]}}'}} 61

I'll add this, but this is likely going to be the last item added via GitHub Issues. The switch to .nbt files already made the Issues process even more convoluted, plus GitHub won't let me attach .nbt files in comments so yeah.
More information coming soon, likely via #96 or a new issue.


Make arena_allocator usable in std containers like unordered_map and others

Description
The current arena_allocator cannot be used in c++ standard containers which are more complex than std::vector as for example std::unordered_map. There is a missing constructor for rebinding (see Additional info below):
template<typename U>
arena_allocator(const arena_alocator<U>& other);

Example
The following does currently not compile:
#include <unordered_map>
#include "stan/math.hpp"

int main()
{
    std::unordered_map<int, int, std::hash<int>, std::equal_to<int>, stan::math::arena_allocator<std::pair<const int, int>>> m;
    m.insert(std::make_pair(1,1));
}

The compilation error is very lengthy because of missing concepts in c++17. But it basically tells that there is missing the respective constructor for arena_allocator.
Expected Output
Compilation should be successful.
Possible fix:
Add this to arena_allocator:
  arena_allocator() = default;

  arena_allocator(const arena_allocator& rhs) = default;

  template <class U>
  arena_allocator(const arena_allocator<U>& rhs) {}

Additional info
See this SO discussion on custom allocators for std containers:
https://stackoverflow.com/questions/64790174/stdunordered-map-with-custom-allocator-local-allocator-does-not-compile
Current Version:
v4.3.1

Yes this would be nice! I don't have the bandwidth to do this atm but if you want to make a PR I'm happy to review. I think we just need to define all of the members and constructors like std::allocator

Most of them should be simple since that allocator is just a wrapper around the stan's global stack allocator

Yes it is sinple because the class is just a wrapper. The only constructor which is missing is the one described above. With this added, the class can be used as a replacement for std::allocator. I would like to submit a PR and I already have a fork with the change, however so far I did not look into the test suite of stan. But I can do und then run the tests before submitting the PR. It is probably also worth to add a test for the std containers with the arena_allocator.

Yes our tests in rev/core should help you see what to do here

@rok-cesnovar should we close this?

Indeed, thanks!


new release 1.4.1

based on PR #17 a new release was tagged, and is visisble in releases tab. But wasn't uploaded to luarocks.

the tag is not on the latest rockspec updated release

retagged on the last commit.
not entirely sure what that does to the releases?

I botched some things about this release, but it doesn't help that you're messing with it at the same time. Please wait.

sorry about that. Hadn't noticed. The last CI now fails on pushing because it exists on LR already.
I'll wait for now 🙏

I think things should be straightened out now. Sorry for the mess, definitely my mistakes while publishing combined with the catch-22 in the CI/publishing workflow.
The underlying issue is https://github.com/lunarmodules/.github/issues/4 — in this case I leaked a rockpec format 3 field into the tagged rockspec which is supposed to be compatible with older formats. It wasn't caught be CI because our test builds don't work until after tagging. To top matters off it isn't even fixable at that point because adding a rockrel bump doesn't trigger a publish even if CI passes.
I'll mess with the workflow and see if I can come up with something that doesn't leave projects in such a pickle.

thx for the hard work!


TypeError: Cannot read properties of undefined (reading 'body')

I'm trying to use the lock action headless to enable locks to make sure an environment can only be used for one PR at a time.
But when I try to lock the environment I get this error:
/home/runner/work/_actions/github/lock/v2/src/main.js:63
    const body = context.payload.comment.body.trim()
^
TypeError: Cannot read properties of undefined (reading 'body')
    at run (/home/runner/work/_actions/github/lock/v2/src/main.js:63:1)
    at /home/runner/work/_actions/github/lock/v2/src/main.js:289:1
    at /home/runner/work/_actions/github/lock/v2/dist/index.js:326[82]([REPO_URL]/actions/runs/11520936978/job/32073423092#step:5:83):3
    at Object.<anonymous> (/home/runner/work/_actions/github/lock/v2/dist/index.js:32685:12)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1024:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:174:12)
    at node:internal/main/run_main_module:28:49

The step I am trying to run that fails
...
- name: Lock environment
  uses: github/lock@v2
  id: lock-environment
  if: steps.lock-check.outputs.locked == 'false'
  with:
    environment: ${{ needs.get-slot-from-label.outputs.slot }}
    github_token: ${{ secrets.GITHUB_TOKEN }}
...

It looks like it assumes there's a body from an issue comment to read. This is not the case when using the action purely headless.

I was missing the "mode" input. Works now.


:ConjureSchool lessons should append text to the buffer only once

Hello! I've just started using this plugin and I just want to give some feedback on the documentation, specifically on the :ConjureSchool document.
The interactive tutorial is very nice but I feel the lessons should only evaluate once. This lets the user to experiment more with the plugin and test their intuitions for a little while. My experience with the plugin is me testing out the keybindings more with the previous lessons and it resulted in a document with duplicate lessons in the buffer. To be fair, a part of it is my fault for forgetting that the lessons append text to the buffer but if a user has a similar way of testing, it could get messy. I feel this experience could be better.
What do you think? (Also, thank you for creating this plugin making writing code in Lispy languages a nicer place in Neovim.)

Hmm you make a good point although it's a little tricky. We wouldn't want to gate it so each function can ONLY be run once I think. If the user executes one lesson then deletes, damages or undoes the print (which does work! You can undo the buffer changes as if you wrote them) then they won't be able to re-run the code to get it back.
So this means we'd have to scan the buffer for the expected output and not print it if we found the text there already. Not impossible or terrible but could be flawed. It relies on finding an exact match, so if the user has changed the last printed result even by one character it'll still print the next text again because it doesn't match.
I considered more complex things like inserting some hidden markers to tell Aniseed that a given form is somewhere in the buffer through metadata (I think that's possible?) but that seems overly complex. I think I like the idea of scanning the file for the lines we're about to insert and not doing it if we find them best. It seems relatively easy to implement and less prone to weird issues that are hard to debug.

Done, implemented as described in my last paragraph.


Background image, icon positions, etc. does not work in Catalina

Hi,
Thanks for the nice script !
I am using it in Catalina.
I failed to set the background image and locate the icons in the specified positions.
I could, finally, make it work properly by running the apple script again right before unmounting (hdutil detach).
Hope this helps

I have no trouble in 10.15.7 (19H114).

I also have this situation

Having a similar issue https://github.com/jamulussoftware/jamulus/blob/23bb83c1f2e9495ac4867fc27c309ed6bbc2393a/mac/deploy_mac.sh
it seems as if the .DS_Store file doesn’t get created

Quick & dirty fix for macos 11+
$ diff -b template.applescript.orig template.applescript
24,31d23
< 			set opts to the icon view options of container window
< 			tell opts
< 				set icon size to ICON_SIZE
< 				set text size to TEXT_SIZE
< 				set arrangement to not arranged
< 			end tell
< 			BACKGROUND_CLAUSE
<
42a35,41
> 			set opts to the icon view options of container window
> 			tell opts
> 				set icon size to ICON_SIZE
> 				set text size to TEXT_SIZE
> 				set arrangement to not arranged
> 			end tell
> 			BACKGROUND_CLAUSE

Seem to have overthought the script in the repo.

@serge-ivamov just moving the code fixed the issue for you?
@ann0see @yuanmeibin @lacasta can you check if this fixes the issue for you?
I’ve never replicated this issue so I’m blind here.

yes, looks like old problem with close/open.
https://stackoverflow.com/questions/96882/how-do-i-create-a-nice-looking-dmg-for-mac-os-x-using-command-line-tools
...
On Snow Leopard, the above applescript will not set the icon position correctly - it seems to be a Snow Leopard bug. One workaround is to simply call close/open after setting the icons, i.e.:

I think I succeeded now. It was just that I overthought the cli arguments (and added --skip-jenkins without thinking). BTW if --skip-jenkins is added as last argument, the script gives an error since it probably thinks the path following --skip-jenkins belongs to this parameter and is not the .dmg file. But that's another issue.


extensions\apidoc\helper\ApiIndexer.php - add Unicode support FileInfo(jsSearch) for bootstrap-guide mode.

extensions\apidoc\helper\ApiIndexer.php - add Unicode support FileInfo(jsSearch) for bootstrap-guide mode.
preg_match(/(.*)/s)

replaced
preg_match(/(.|\r|\n*)/u)

What are you trying to solve with this change? .* should match any unicdeo character even without u...

preg_match don't match utf8 without /u modifier (I was generating russian(utf8) guide with apidoc - and having empty fileInfo in jssearch index).
But /u and /s modifiers don't work together (/su or /us) - so I added /s modifier functionality using .|\r|\n and added /u modifier (tested ok)

In the process of Guide generating is created file jssearch.index.js - it should contain jssearch.files variable, but after generating of Russian Guide(utf8) jssearch.files remains empty (jssearch.files="";). In English Guide jssearch.files normaly contains info about files and it used for output listing search result on a page.

I have reproduced the issue, if you enable PHP warnings in your php config which you definitively should, you get an exception about invalid UTF8.


Feature request - hide url from application

Hi, would it be possible to add an option to hide the URL from the Applications on the dashboard? On all or even better by application?

Duplicate of #105


mvn package failure complaining libnetty_tcnative.jnilib is missing

Expected behavior
mvn package success
Actual behavior
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (unpack-jar-features) on project netty-testsuite-shading: An Ant BuildException has occured: Warning: Could not find file /Users/delegacy/code/netty/testsuite-shading/target/classes-shaded/META-INF/native/libnetty_tcnative.jnilib to copy.
[ERROR] around Ant part ...<copy file="/Users/delegacy/code/netty/testsuite-shading/target/classes-shaded/META-INF/native/lib${nativeTcnativeLib}" tofile="/Users/delegacy/code/netty/testsuite-shading/target/classes-shaded/META-INF/native/libshaded_${nativeTcnativeLib}"/>... @ 15:247 in /Users/delegacy/code/netty/testsuite-shading/target/antrun/build-main.xml
[ERROR] -> [Help 1]

Steps to reproduce
mvn package with 3e8e52725b (HEAD at this moment) on mac
Minimal yet complete reproducer code (or URL to code)
Netty version
JVM version (e.g. java -version)
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)

OS version (e.g. uname -a)
Darwin Young-MBPII.local 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec  2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64 x86_64

Firstly, I guessed it is caused by the wrong condition introduced by this commit.
(I thought it should be like the below, considering that it should copy one with the classifier when a classifier is given.)
<condition property="nativeTcnativeLib" value="netty_tcnative.jnilib" else="netty_tcnative_osx_${os.detected.arch}.jnilib">
  <equals arg1="${tcnative.classifier}" arg2="" />
</condition>

However, I realized the commit was added two years ago. I might miss something, then.
I can't find libnetty_tcnative.jnilib from some related artifacts like netty-tcnative, netty-tcnative-osx-x86_64, and netty-tcnative-boringssl-static, though I found ones with the classifier from them.

can you try mvn clean install ?

mvn clean install  does not work either. (the same error)
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (unpack-jar-features) on project netty-testsuite-shading: An Ant BuildException has occured: Warning: Could not find file /Users/delegacy/code/netty/testsuite-shading/target/classes-shaded/META-INF/native/libnetty_tcnative.jnilib to copy.
[ERROR] around Ant part ...<copy file="/Users/delegacy/code/netty/testsuite-shading/target/classes-shaded/META-INF/native/lib${nativeTcnativeLib}" tofile="/Users/delegacy/code/netty/testsuite-shading/target/classes-shaded/META-INF/native/libshaded_${nativeTcnativeLib}"/>... @ 14:247 in /Users/delegacy/code/netty/testsuite-shading/target/antrun/build-main.xml

I just wanted to run mvn test at the beginning. 😭 (following https://netty.io/wiki/developer-guide.html)
Then, I met MDEP-98 issue and tried mvn package as the recommended workaround here.
These are all files in the directory.
~/code/netty 4.1* ⇣
❯ ll testsuite-shading/target/classes-shaded/META-INF/native/
total 1008
drwxr-xr-x  10 delegacy  staff   320B Dec 23 20:52 include
drwxr-xr-x   3 delegacy  staff    96B Dec 23 20:52 lib
-rw-r--r--   1 delegacy  staff   138K Dec  8 10:30 libnetty_tcnative_osx_x86_64.jnilib
-rw-r--r--   1 delegacy  staff    92K Dec 23 20:51 libnetty_transport_native_kqueue_x86_64.jnilib
-rw-r--r--   1 delegacy  staff    74K Dec 23 20:51 libnetty_transport_native_unix_osx_x86_64.jnilib
-rw-r--r--   1 delegacy  staff    92K Dec 23 20:52 libshaded2_netty_transport_native_kqueue_x86_64.jnilib
-rw-r--r--   1 delegacy  staff    92K Dec 23 20:52 libshaded_netty_transport_native_kqueue_x86_64.jnilib

This is 4.1 branch ?

Yes, I'm trying to run it on 4.1 branch.

I can't reproduce this when running ./mvnw clean package :/

I can't reproduce this when running ./mvnw clean package :/

I see. It's strange. 🤔
This mvn.log might be helpful for further investigation. (Retried with OpenJDK 1.8.0_275 to follow How to build section in README.md)
If I knew where libnetty_tcnative.jnilib should come from, I could debug more for myself.

I see. It's strange. 🤔
This mvn.log might be helpful for further investigation. (Retried with OpenJDK 1.8.0_275 to follow How to build section in README.md)
If I knew where libnetty_tcnative.jnilib should come from, I could debug more for myself.

it should be included in the netty-tcnative artefact.

it should be included in the netty-tcnative artefact.

libnetty_tcnative.jnilib seems renamed with libnetty_tcnative_osx_x86_64.jnilib.
➜ jar xvf netty-tcnative-2.0.31.Final-osx-x86_64.jar
...
 inflated: META-INF/io.netty.versions.properties
 inflated: META-INF/maven/io.netty/netty-tcnative/pom.properties
 inflated: META-INF/maven/io.netty/netty-tcnative/pom.xml
 inflated: META-INF/native/libnetty_tcnative.jnilib
...

 ➜ jar xvf netty-tcnative-2.0.32.Final-osx_x86_64.jar
...
 inflated: META-INF/io.netty.versions.properties
 inflated: META-INF/maven/io.netty/netty-tcnative/pom.properties
 inflated: META-INF/maven/io.netty/netty-tcnative/pom.xml
 inflated: META-INF/native/libnetty_tcnative_osx_x86_64.jnilib
...

https://github.com/netty/netty-tcnative/compare/netty-tcnative-parent-2.0.31.Final...netty-tcnative-parent-2.0.32.Final
https://github.com/netty/netty-tcnative/pull/557
Would it be a regression since then?


OAK-10727 : log revisionDetailedGcType in VersionGarbageCollector con…

…structor

@rishabhdaim  wdyt?

I am wondering if I shouldn't add this to VersionGCStats so that it gets logged along with other stats.

+1 to adding it also to VersionGCStats - we can still see which one turns out more/how useful and adjust along the way in case


Modify API to support date-based vector requests.

To include:

Basic ISO-8601 support?
LoC EDTF support? http://www.loc.gov/standards/datetime/pre-submission.html

This seems moot at this point, doesn't it? We have start_date, end_date working, we have Overpass. And we have other issues specifically about handling approximate date formats and other alt formats


IDVA6-1324: ACSP manage users api to lookup acsp memberships

Short description outlining key changes/additions

Implement POST /acsps/{acspNumber}/memberships/lookup endpoint
Add fetchAcspMemberships method to AcspMembersService
Create new repository methods for fetching memberships by user ID and ACSP number
Add integration tests for new functionality
Update existing tests to accommodate changes

JIRA Ticket Number

https://companieshouse.atlassian.net/browse/IDVA6-1324

Type of change

[ ] Bug fix
[x] New feature
[ ] Breaking change
[ ] Infrastructure change

Pull request checklist

[x] I have added unit tests for new code that I have added
[x] I have added/updated functional tests where appropriate
[x] The code follows our coding standards

An exhaustive list of peer review checks can be read here

CI: Security warnings found!

CI: Security warnings found!

CI: Security warnings found!

CI: Security warnings found!

CI: Security warnings found!

CI: Security warnings found!


About the results on DOTA-v1.5

Hi,
Thanks for the great work, I wonder if you have done experiments on the DOTA-v1.5 which is a more challenging dataset.
Because our results with your Orient RCNN code on DOTA-v1.5 is only about 71% mAP.

We don't test our model on DOTA-1.5. But the baseline results (Faster RCNN + RT) achieves 65.03% mAP as we refer in AerialDetection model zoo. So I think 71% mAP is a pretty high performance if without augments.

@jbwang1997 But I get the results with "configs/obb/oriented_rcnn/faster_rcnn_orpn_r50_fpn_1x_ms_rr_dota15.py" , which is multi-scale training with flip and rotation augments.

The multi-scale training needs to split images under scales 1.5 and 0.5. This operation is completed by BboxToolkit. I wonder if your skip this operation.
Besides, The results in AerialDetection model zoo are trained on the DOTA train and val set. Check if you only train the model on train set.

We have tested our method on DOTA-2.0 and the results are higher than Faster R-CNN+RT. So I think it's the same on DOTA-1.5.

@jbwang1997 Can you tell us the results on DOTA-2.0 and the corresponding training and test config, Thanks.


CDPD-25578:Set "hive.metastore.try.direct.sql.ddl" to true

screenshot of the property added to hms safety valve:

@satyabollineni The cluster where you have tried this, pls execute "set hive.metastore.try.direct.sql.ddl;" on beeline prompt to confirm new value of the config is "true".

@satyabollineni Also AFAIK this JIRA is being targeted for 7.2.10, so you should be updating 7.2.10 templates too.

@satyabollineni You should also change the screenshot in this PR to show correct safety valve.


Jenkins: Debug failures

Signed-off-by: Eloy Coto<EMAIL_ADDRESS>

This change is

test-me-please

https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/4008/flowGraphTable/

test-me-please

test-me-please

test-me-please

test-me-please

test-me-please

test-me-please


Switching between Call App and Opt-in keeps argument 0

If i set a method name in the Call App state on a loaded contract, and THEN switch to opt-in because i have, say, forgotten to opt in, arg 0 should be my first arg in the opt-in.  It currently stays set to whatever i have set in the method name field and my arguments to opt-in start at arg 1 like they would if i were calling an API.
Opt in should just ignore whatever is in the Method Name field.

huh... it seems this code should handle this case:
https://github.com/thencc/Irulan/blob/a89db5658274930e90d513ad4aad3dd03fc0f242/src/components/ContractTool.vue#L132-L134
i ran thru what you said and it worked as desired

@youraerials
i believe this is actually a bug in your contract code.
perhaps you are re-using the name subroutine for opting in and noop call app?
in noop app calls, the first arg (arg 0) is the method name but when in opting in you'll want to use arg 0 as the first thing you local put.
glancing at the teal, it seems you're starting localPuts at app arg 1 instead of app arg 0.


turbine_temperature error with fractional multistart time lags

In co-opting turbine_temperature for my own code, I found a problem that likely also needs to be addressed there. I think turbine_temperature needs to account for the system's time interval to avoid indexing failures. For the TAMU day-ahead system, here's an error based on lines 768-770 in thermal_generation.jl:
g = "FAYETTE_POWER_U2_J1"
"FAYETTE_POWER_U2_J1"

julia> g_startup = get_start_time_limits(get_component(ThermalMultiStart, system, g))
(hot = 1.25, warm = 4.0, cold = 10.0)

julia> ix = 1
1

julia>             UnitRange{Int}(Int(g_startup[ix]), Int(g_startup[ix + 1] - 1))
ERROR: InexactError: Int64(1.25)
Stacktrace:
 [1] Int64(x::Float64)
   @ Base .\float.jl:723
 [2] top-level scope
   @ REPL[209]:1

For my own code, which has hourly resolution, I just threw a ceil around each Int argument.

Hi Kate, The code does convert the startup data from hours to time periods using this function. https://github.com/NREL-SIIP/PowerSimulations.jl/blob/eae59fab1184a53354502c72af263144cc048138/src/devices_models/devices/thermal_generation.jl#L688-L698
https://github.com/NREL-SIIP/PowerSimulations.jl/blob/eae59fab1184a53354502c72af263144cc048138/src/devices_models/devices/thermal_generation.jl#L939
https://github.com/NREL-SIIP/PowerSimulations.jl/blob/eae59fab1184a53354502c72af263144cc048138/src/devices_models/devices/thermal_generation.jl#L1007
I also tried to replicate your error using this small example but I can't get it to replicate the same error.
using PowerSystemCaseBuilder
const PSB = PowerSystemCaseBuilder
sys = PSB.build_system(PSITestSystems, "c_sys5_pglib_sim")
set_start_time_limits!(get_component(ThermalGen, sys, "115_STEAM_1"), (hot = 2.125, warm = 4.0, cold = 12.0))

template = OperationsProblemTemplate(CopperPlatePowerModel)
set_device_model!(template, ThermalMultiStart, ThermalMultiStartUnitCommitment)
set_device_model!(template, RenewableDispatch, RenewableFullDispatch)
set_device_model!(template, PowerLoad, StaticPowerLoad)
set_device_model!(template, HydroDispatch, HydroDispatchRunOfRiver)

UC = OperationsProblem(
    template,
    sys,
    optimizer = solver,
    optimizer_log_print = true,
)
build!(UC; serialize = false, output_dir = pwd())
solve!(UC)

@sourabhdalvi and @kdayday can you confirm the PSI versions you are using?

@sourabhdalvi and @kdayday can you confirm the PSI versions you are using?

PowerSimulations v0.14.1

Looks like it's already taken care of, I didn't have that pre-processing in my copy/paste version.


emailreport: fix golint warnings

golint warns about some var naming issues, fix them up.
Fixes: #150
Signed-off-by: Graham Whaley<EMAIL_ADDRESS>

lgtm

lgtm


[WIP] Metrics collection

Crude implementation of metrics collection in swarmkit.
This is a proof of concept
Output:
# HELP swarmkit_nodes_total The number of nodes
# TYPE swarmkit_nodes_total gauge
swarmkit_nodes_total{state="disconnected"} 0
swarmkit_nodes_total{state="down"} 0
swarmkit_nodes_total{state="ready"} 2
swarmkit_nodes_total{state="unknown"} 0

/cc @rogaha

Open questions while hacking this together:

Do we want a proper collector like in this PR or should we just instrument the dispatcher? My thought here was that we'll probably want to do the same for services and tasks which makes the collector more convenient
Same for managers (and leader status) - we could either instrument the leadership code or stash that code here
What about leadership? With this current approach, every single manager will export metrics based on their local Raft state (which may lag behind)
Are we expecting operators to scrape all managers? The sum of metrics doesn't make sense (every manager will export the totality of nodes/services/etc). (@binman-docker: thoughts on this one?)

@aluzzardi - POC of using the prometheus.Collector interface in https://github.com/moby/moby/pull/32945 . As I state, the metrics collector there at the moment aren't right; I copied what @rogaha as I was trying to illustrate the technique. Prometheus metrics are usually scraped on the order of ten's of seconds and above, so this is a way to only require extra processing/network resources on demand.
my thoughts on questions above:

I think the collector angle makes more sense, given prevalence of prometheus metrics.
I think it's appropriate for managers to export their own state, even if it lags. Since metrics are only gathered sporadically, they're not expected to be realtime views on the state of the system.
scraping all managers: I believe so: more than likely, would expect every engine to be scraped.

On the two last points:

Yes, we would want to see each node (manager and worker) export their own state as well as their view of the swarm as a whole, including the ID of the Swarm they are attached to, on their own endpoint or API. This will let us monitor individual state as well as split-brain. You're right that a sum of all collected values would add up to N^2, but that's ok - we wouldn't sum that metric, we would look for an individual value less than the number we expect (one node sees managers=2, etc).

Counters are most useful for events (join/left/etc). We can then calculate the increase over time to find how many events happened. These are more reliable because collection of the metric does not happen at a guaranteed time, so a gauge for "joined in last X seconds" could miss events depending on collection time. Gauges are however useful for current state (there are 3 managers right now, X services running, etc).

@alfred-landrum err, some confusion by the naming - what I referred to as collector is the Collector struct created in this PR, not prometheus.Collector. Not sure if that changes your answer

By this:

Do we want a proper collector like in this PR or should we just instrument the dispatcher

I meant: Do we want to have a "component" like I did in this PR that sets all relevant metrics for the Swarm. OR, do we want to just add metrics code a little bit everywhere (e.g. export node metrics in the Dispatcher, which handles node connections and such).

Do we want a proper collector like in this PR or should we just instrument the dispatcher? My thought here was that we'll probably want to do the same for services and tasks which makes the collector more convenient

My only concern about having a separate collector is that it makes it harder to expose internal state from the various components. For example, maybe we want to expose metrics on the scheduler that show how long it takes to schedule a task, and which nodes are being given low priority by the scheduler because of failures. With the collector as a separate component, we can't really expose this without storing all this information in Raft.
If there's a good solution to this, I don't see any other problems with this design.

What about leadership? With this current approach, every single manager will export metrics based on their local Raft state (which may lag behind)

It looks like in the current approach, the collector only runs on the leader. There may be a few seconds of overlap where two nodes think they are the leader, but we recently turned on CheckQuorum, which makes nodes step down from leadership if they aren't still in touch with a quorum.

The usual approach recommended by Prometheus people is to have instrumentation live directly in the code paths which it is meant to instrument - similar to how you would have logging lines in there. For example, increment a counter in the same code path where the event it is counting is handled, etc. You'd normally only use a separate collector if you don't directly control the code you want to get Prometheus metrics from and/or you are thus just proxying third-party metrics (like from a database). By instrumenting code directly, you can easily get whitebox metrics from wherever you need them.
In this case, it looks like you are indeed only proxying third-party metrics state though (for now). For just proxying whatever is in the Raft store, an independent collector seems ok. But if you then want to add more other whitebox metrics to the manager (like what is the manager itself doing, what latencies is it seeing, what do queue lengths look like, etc.), those should probably live in the manager code itself.

@aluzzardi - thanks, but no, doesn't change the suggestion: Instead of having a polling goroutine that needs to watch internal stage changes & update counters, you can have code that only runs when an external users asks for your metrics. Prometheus's name for this is a "Collector", which agree is confusing. It's useful if gathering up your metrics is something that is comparatively expensive, or needs to issue remote calls.
by coincidence, the prometheus documentation example for this is to provide stats about a cluster, where you need to make one or more network requests to get the current state to report:
https://godoc.org/github.com/prometheus/client_golang/prometheus#Collector

@alfred-landrum Yes, I was wondering about that. As long as the computation of metrics is cheap enough to do on every scrape, we recommend computing it during scrape time. If it gets too expensive though, caching/precomputing (as the PR currently does) can become necessary. I don't know enough about the cost of pulling together all the Raft data here though, and how frequently people might want to scrape them.

Thanks for the review! This is helpful.
@aaronlehmann @juliusv

My only concern about having a separate collector is that it makes it harder to expose internal state from the various components.

The usual approach recommended by Prometheus people is to have instrumentation live directly in the code paths which it is meant to instrument

Here are my thoughts:
I think internal state of various components would be exposed directly by the component itself - the Collector controller doesn't exclude that.
However, for things such as nodes and tasks by state (and basically everything that's readily available in raft), there's no particular component that "owns" that.
The dispatcher gets updates from nodes on task states, but it feels awkward to me to have the dispatcher handle task metrics. That component's job is not to handle tasks but rather get updates from nodes regarding their tasks and treats them as an opaque blob. Once they are stored, other components are triggered (e.g. orchestrator), but none of them truly owns all tasks.
Also, in order to have more "advanced" metrics (such as gauges labeled by state), the dispatcher would need to keep a bunch of state it doesn't have today (e.g. query all tasks).
But now let's say we want to have a counter to track nodes that disconnect from the dispatcher. I think it feels right to have that counter live in the dispatcher itself and expose its internal state (rather than trying to infer that from the central collector).
As a convention, perhaps those internal metrics should use a subsystem (https://github.com/docker/go-metrics/blob/master/namespace.go#L17) e.g. swarmkit_dispatcher, or swarmkit_scheduler while "top-level" metrics would not have a subsystem (e.g. swarmkit_nodes)
@alfred-landrum

@aluzzardi - thanks, but no, doesn't change the suggestion: Instead of having a polling goroutine that needs to watch internal stage changes & update counters, you can have code that only runs when an external users asks for your metrics. Prometheus's name for this is a "Collector", which agree is confusing. It's useful if gathering up your metrics is something that is comparatively expensive, or needs to issue remote calls.

Querying for all objects is very expensive. With the approach in this PR we would do it only once at startup then just watch for changes. We have tens of similar watches throughout the code and they are extremely cheap.
But maybe I'm wrong. @aaronlehmann @juliusv second opinion?

@aluzzardi Agreed on all points you mentioned in https://github.com/docker/swarmkit/pull/2157#issuecomment-298425714.
As for the state computation: yes, if it's that expensive to query the entire state, then doing it once at startup and maintaining the in-memory state via watches is the way to go. This is then naturally decoupled from scrape time. For good measure, you might of course still want to refresh all state once in a while, depending on how confident you feel that incremental updates will be reliable over a long period of time :)

What about leadership? With this current approach, every single manager will export metrics based on their local Raft state (which may lag behind)

I think only leader should export cluster level counters. It'll provide a consistent view of cluster. The only exception could be at leader switch. If we collect counters from every manager, we may see inconsistent data, without a good way to fix them. Each manager may still export some local counters, like how many nodes connected to it, which is not cluster level counters.

@dongluochen Typically you will want to expose all those metrics from all manager processes (not just the leader), but also export a metric which tells you whether a process is a leader or not (0 / 1 value), then join against that metric in queries as necessary. That still allows you to debug what the other managers are seeing.

See also https://prometheus.io/docs/instrumenting/writing_exporters/#deployment, last paragraph of that section:
"This doesn’t apply when you’re monitoring a system with master-election, in that case you should monitor each instance individually and deal with the masterness in Prometheus. This is as there isn’t always exactly one master, and changing what a target is underneath Prometheus’s feet will cause oddities."
(that doc is about writing Prometheus exporters as separate processes, but the same applies here)

Pushed a few changes - it's now using the previous node given by the watch interface rather than keeping track of old states. Also avoiding making unnecessary changes (if the state didn't change) and always publishing (on all managers)

always publishing (on all managers)

Add a leader counter to distinguish them?

Add a leader counter to distinguish them?

A gauge would be the thing to have here. Something like swarmkit_is_leader -> 0 / 1

Yeah, I think we need more info: version, node id, swarm id and leadership. Among leadership, there are other bits of useful data: number of managers, etc etc.
So yeah, I agree we need something like swarmkit_is_leader although I don't know if that information should be grouped together something else.

version, node id, swarm id

These are not numeric counters. I don't think they belong here.

Metadata like that can be exported as label values on a single info metric, like the build info metric we have in Prometheus itself:
# HELP prometheus_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which prometheus was built.
# TYPE prometheus_build_info gauge
prometheus_build_info{branch="HEAD",goversion="go1.8",revision="8b8ba2612968cd192d8dba7ab745480a1774dcc3",version="1.6.1"} 1

The leadership state should be a separate metric though, so it can be easily used in queries.

I see. It's multi-dimension counter.

It's not a counter, but a gauge (doesn't matter much because it's only 1 anyways). See https://prometheus.io/docs/concepts/metric_types/

I don't think they belong here.

Here where?

My mistake. I didn't realize you are define these with labels.
I think it good to list/group the counters. We can review the data model before implementation.

hey @juliusv! Thanks for sharing with us the prometheus knowledge! :) I really appreciate it!
@aluzzardi do you think we could include the swarmkit_info metrics into this PR as suggested by @juliusv (https://github.com/docker/swarmkit/pull/2157#issuecomment-298446753)? Having the version, node id, swarm id and leadership would be very helpful, it would accomplish what we were trying to do here https://github.com/moby/moby/issues/32945.

@aluzzardi @rogaha This needs to be balanced with instrumentation on the agent. That can get things like per node task assignment/state. swarmkit_info would be good in general.

Makes sense - however, info doesn't belong in the Collector (this only runs on managers, we need that information on all nodes).
@aaronlehmann @stevvooe Any idea where we could stash the metric for info? It needs access to the node ID and swarm ID. Maybe node/node.go?

Yes, node/node.go already has n.nodeID populated from securityConfig.ClientTLSCreds.NodeID(). You could also add n.clusterID and populate it with securityConfig.ClientTLSCreds.Organization().

@aaronlehmann however, we have no idea whether it's a leader or not. It might be tricky to expose all this info as a single gauge since that information is spread apart.

Okay, tentatively added info in node.go.
Few problems/comments:

I used a gauge and exported the information as labels like we do in the engine. This doesn't allow updates though? I end up with two metrics:

swarmkit_info_{node_id="k6q5x5wal3w478h4d8ze78kdv",role="manager",swarm_id="9dmuhxt69kzrasmqn2t5sjw97"} 1
swarmkit_info_{node_id="k6q5x5wal3w478h4d8ze78kdv",role="worker",swarm_id="9dmuhxt69kzrasmqn2t5sjw97"} 1

What's the best practice to export general information? /cc @juliusv

I'm watching Role changes in node.go. However this doesn't catch Swarm ID / Node ID changes. I'm just going to assume this is not going to happen, but that's a limitation (/cc @aaronlehmann)

I have no way of exposing manager leadership in here. This should be done elsewhere and therefore must be exported by a different metric. @aaronlehmann maybe this should go in manager.go? or raft itself? manager already has a clean structure for election

I've tweaked the collector NOT to expose metrics at init() time anymore. Instead, it will publish them when started and unpublish them when stopped. The idea is if a node is a worker, it won't expose node metrics (which are going to be at zero anyway since it's not a manager), and if it's demoted to worker, it will unpublish metrics (so we don't keep stale data out). Is this an okay practice, @juliusv?

swarmkit_info... bit of a lousy name. Maybe swarmkit_node? but we have a nodes already

Should we change swarmkit_nodes to swarmkit_manager_nodes (same for all manager metrics)? at least it would be obvious that the metric would not be there if the node is not a manager.

/cc @rogaha @stevvooe

General comment: Do we want to prefix these with swarmkit instead of swarm? I didn't think swarmkit was a user-facing name; it's kind of an implementation detail that the swarm code lives in swarmkit.

However this doesn't catch Swarm ID / Node ID changes. I'm just going to assume this is not going to happen, but that's a limitation

That's a safe assumption. These values can't change.

I have no way of exposing manager leadership in here. This should be done elsewhere and therefore must be exported by a different metric. @aaronlehmann maybe this should go in manager.go? or raft itself? manager already has a clean structure for election

I would say manager.go.

General comment: Do we want to prefix these with swarmkit instead of swarm? I didn't think swarmkit was a user-facing name; it's kind of an implementation detail that the swarm code lives in swarmkit.

Yeah, I was wondering the same. The metrics are going to be low-level so implementation details do matter, but agreed.
Changed to swarm

@aluzzardi Stateful things, like leadership, should probably be a separate metric.
Do you have an example of the current output?

That's a safe assumption. These values can't change.

That's good, it'd make things more complicated if they could change.

swarmkit_info... bit of a lousy name. Maybe swarmkit_node? but we have a nodes already

swarmkit_node_info would be my first thought.

I have no way of exposing manager leadership in here. This should be done elsewhere and therefore must be exported by a different metric.

As this is a boolean which changes over time, it should be its own metric with value 0/1 and no labels.

The idea is if a node is a worker, it won't expose node metrics (which are going to be at zero anyway since it's not a manager), and if it's demoted to worker, it will unpublish metrics (so we don't keep stale data out). Is this an okay practice

Once a time series is exposed it should be exposed until the end of the process, preferably since the start of the process too. Time series appearing and disappearing is difficult to deal with semantically on the PromQL end, and very likely to result in users missing the associated subtleties. This is not considered stale data, in fact unpublishing metrics will cause staleness problems.
Generally if code is inside the binary, its associated metrics should be registered and exposed, preferably during init(). They might end up all 0s, but that's very cheap to deal with.

Should we change swarmkit_nodes to swarmkit_manager_nodes (same for all manager metrics)?

Having manager in there makes sense for any manager-related metrics.
Think of it as naming the "library" the metric is coming from, this is the swarmkit manager library. There might be many many libraries inside a single binary, so you want them to have distinct prefixes so that they're less likely to clash and that the end user will have a better idea of what a given metric name means.

swarmkit_info... bit of a lousy name. Maybe swarmkit_node? but we have a nodes already

I would use swarm_nodes_info as suggested by @brian-brazil.

I have no way of exposing manager leadership in here. This should be done elsewhere and therefore must be exported by a different metric.

+1 on exposing the leadership metrics separately.
example
clusterLeader = prometheus.NewGaugeVec(prometheus.GaugeOpts{
		Namespace: Namespace,
		Subsystem: subsystem,
		Name:      "cluster_leader",
		Help:      "Whether or not this node is a leader (yes=1, no=0)",
	}, []string{"node_id"})

@rogaha @brian-brazil How would this aggregate if you are monitoring multiple clusters?

Each cluster would have some distinguishing label as a target label on the Prometheus end, as it's ultimately up to the person configuring the Prometheus as to how they model such a setup. Similarly any aggregation would be up to the user.
The only role of swarm itself is to try to avoid using instrumentation labels that have a good chance of clashing with such a target label (e.g. cluster).

@brian-brazil

As this is a boolean which changes over time, it should be its own metric with value 0/1 and no labels.

Oh, thanks, I didn't know that. Right now I'm exposing a role label in info - which does change over time. So it should go into its own metric?
Maybe something like:
swarm_node_info node_id=... swarm_id=... 1
swarm_node_manager 0|1
swarm_node_leader 0|1

Yes, that's the way to approach it. swarm_node_info is using what we call the "machine roles" approach, and the others are normal booleans.
https://www.robustperception.io/how-to-have-labels-for-machine-roles/
https://www.robustperception.io/exposing-the-software-version-to-prometheus/
swarm_node_manager may better belong under swarm_manager, to go with all the other manager metrics.

hi @aluzzardi is there any design questions pending here? Let me know please if you need any help. We merged today the extra prometheus metrics on the engine side.

@rogaha My bad. I don't think there are design questions pending here (the last comments from @brian-brazil were enough), just lack of time

Np, thanks for the update @aluzzardi. I know you guys are quite busy. Is there anything I can help with? What is missing here?

@rogaha Yeah there's a bunch of unaddressed comments here, mostly the label names

@aluzzardi would be possible to have at least the basic metrics for now? e.g. swarm_id & node_id? These labels seem to be very straightforward.

Final update, PTAL @rogaha @brian-brazil @juliusv @stevvooe @aaronlehmann
There are two "namespaces": swarm_node and swarm_manager:

swarm_node contains node related metrics (e.g. shared between managers and workers)

swarm_node_info: labeled gauge with node_id and swarm_id (never changes)
swarm_node_is_manager: gauge (0 or 1) specifying whether the node is a manager or not

swarm_manager contains manager specific metrics

swarm_manager_nodes: Labeled gauge of nodes by state

Manager metrics are always present in the output (e.g. at init time).
Manager metrics get reset to 0 on demotion (so we show all gauges to 0 rather than stale data)
Example:
# HELP swarm_node_info Information related to the swarm
# TYPE swarm_node_info gauge
swarm_node_info{node_id="k6q5x5wal3w478h4d8ze78kdv",swarm_id="9dmuhxt69kzrasmqn2t5sjw97"} 1

# HELP swarm_node_is_manager Whether this node is a manager or not
# TYPE swarm_node_is_manager gauge
swarm_node_is_manager 1

# HELP swarm_manager_nodes The number of nodes
# TYPE swarm_manager_nodes gauge
swarm_manager_nodes{state="disconnected"} 0
swarm_manager_nodes{state="down"} 2
swarm_manager_nodes{state="ready"} 0
swarm_manager_nodes{state="unknown"} 0

@aluzzardi LGTM

Seems generally fine to me. I'm not fully sure about setting the metrics to 0 on demotion, but I guess we'll see how that work out operationally.

Why not swarm_node_manager and swarm_node_worker?

I'm ok with swarm_node_manager as well.

LGTM

LGTM


HDDS-6074. Remove flaky tag from TestOzoneManagerHAWithData

What changes were proposed in this pull request?
TestOzoneManagerHAWithData passed in 200/200 runs, I think we can remove the flaky annotation.
https://issues.apache.org/jira/browse/HDDS-6074
How was this patch tested?
https://github.com/adoroszlai/hadoop-ozone/actions/runs/6197108608
https://github.com/adoroszlai/hadoop-ozone/actions/runs/6197659648

Thanks @errose28 for reviewing and merging this.


⚠ DDoS-Relay US-E-ash-2 has degraded performance

In 26a24e7, DDoS-Relay US-E-ash-2 (http://<IP_ADDRESS>:42069/;csv) experienced degraded performance:

HTTP code: 200
Response time: 12 ms

Resolved: DDoS-Relay US-E-ash-2 performance has improved in eb9bb3a after 1 hour, 6 minutes.


Need to clean up modem's IMSI string

Right now, we're capturing the whole response from AT+CIMI.  It's profoundly ugly, carriage returns and all sorts of junk.  Need to use a regex to grab just the digits.

Good in test!  Closing...