Feeds with many items are slow to open and navigate

Newsboat version 2.12.0:
Steps to reproduce the issue:

Add an RSS URL to .newsboat/urls

Open Newsboat and reload RSS feed (over some time) until it has about 15000 items

In the Your feeds section, select the feed and press enter. <- Newsboat will take about a minute to respond to this

Press down to select the next item within the feed. <- Newsboat will take a few seconds to respond to this

It looks to me like Newsboat loads all the entries in a feed when that feed is opened. Perhaps it should only load the ones that it will actually display?

It looks to me like Newsboat loads all the entries in a feed when that feed is opened. Perhaps it should only load the ones that it will actually display?

Correct. There are a few reasons for that:

the obvious one: this wasn't prematurely optimized, but now we have the rest of Newsboat on top of this, and optimization is complicated;
you can't do sorting unless you load the whole feed;
you can't apply ignores (when in ignore-mode "display") without loading the whole feed.

There are two possible workarounds:

limit the size of your feeds with max-items;
upgrade to faster storage (will only help with 3).

As for 4: do you have mark-as-read-on-hover enabled? Does Newsboat use a lot of CPU time after you press the key?

max-items works great. Thanks.

I still consider this a problem—Newsboat shouldn't slow down no matter how big the cache file. So I'll keep this issue open.
I will be grateful if you could answer the questions I asked earlier, @hjek:

As for 4: do you have mark-as-read-on-hover enabled? Does Newsboat use a lot of CPU time after you press the key?

I still consider this a problem

Cool that you consider it within the scope of this program to handle feeds with >10000 entries. I like that.

I will be grateful if you could answer the questions I asked earlier

No problem. Sorry I forgot.

do you have mark-as-read-on-hover enabled?

No, I don't have mark-as-read-on-hover enabled.

Does Newsboat use a lot of CPU time after you press the key?

Yes, Newsboat maxes out the CPU for a few seconds after pressing up or down key in the feed entry list.

Something new here ;-)?

@Eucyon, nope. Blocked on #232 right now; need to move to a UI library that doesn't require us to format the whole list at once.

@dennisschagt has proven me wrong by fixing this within the constraints of our current UI library: #2357.
Query feeds might still be slow to open though; this is tracked in #1730. The rest should be fast now.

(Had to re-open this in order to link the PR to this issue.)


perf(@ngtools/webpack): remove usage of native private class fields

TypeScript will downlevel native private class fields in form a global variable which can cause memory retention.
var _fileEmitter, _registrations;
Object.defineProperty(exports, "__esModule", { value: true });
exports.FileEmitterCollection = exports.FileEmitterRegistration = exports.AngularPluginSymbol = void 0;
exports.AngularPluginSymbol = Symbol.for('@angular-devkit/build-angular[angular-compiler]');
class FileEmitterRegistration {
    constructor() {
        _fileEmitter.set(this, void 0);
    }
    update(emitter) {
        __classPrivateFieldSet(this, _fileEmitter, emitter);
    }
    emit(file) {
        if (!__classPrivateFieldGet(this, _fileEmitter)) {
            throw new Error('Emit attempted before Angular Webpack plugin initialization.');
        }
        return __classPrivateFieldGet(this, _fileEmitter).call(this, file);
    }
}

which can cause memory retention.

Do we know this is the case? I'd expect normal fields to behave the same; as long as the instance is reachable it will retain the field and that's the same for the retention in WeakMap.


Added a StorageQualifier parameter to the Material methods where it is r...

Summary:
This PR adds support for attribute parameters in shaders/materials. This is achieved by adding a storage qualifier parameter to the Material interface methods.
As a consequence of this, all Material.set*(...) calls have been updated. Previously, these calls could be assumed to set uniforms. This assumption now has to be made explicit using the new parameter.
Motivation:
Attributes are required for instanced rendering of particles.
Attributes are useful in general where similar objects are rendered (in batch) using the same shader, but with different input values for each object.

Can one of the admins please verify this patch?

Closed; per vertex data belongs in Meshes, not Materials


add cli 2.0 sample instead of xplat cli

in CLI section- "azure" (not just "az)" in a command it's a pointer that used- old xplat cli. to be aligned with latest updates- it will be right to have azure cli 2.0 sample.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 57944e9d-0efa-3503-9fe8-46c91119ecc9
Version Independent ID: 32caa5c0-aac4-75f8-42f8-b483d55a8392
Content: Azure SQL database metrics and diagnostics logging
Content Source: articles/sql-database/sql-database-metrics-diag-logging.md
Service: sql-database
GitHub Login: @veljko-msft
Microsoft Alias: vvasic

Azure CLI

@SychevIgor Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

@Mike-Ubezzi-MSFT I'm the owner of this content now, please reassign to me.
@SychevIgor I would need a more precise description of what is it exactly that you ask. I'm not familiar with azure cli 2.0 - could you point me in the right direction and/or provide examples. Thanks.

#please-close in light of no communication from customer since requested on 5/4

I expect that find an azure cli 2.0 command will be easy and you don't need me for that
I expect, that we can set diagnostic using
https://docs.microsoft.com/en-us/cli/azure/monitor/diagnostic-settings?view=azure-cli-latest#az-monitor-diagnostic-settings-create
I didn't find command for log analytics, but we don't have it in xplat cli either.

@SychevIgor We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.

Hi there.
I couldn't find the new AZ command anywhere.
Has this been updated?

@A144546 No, CLI 2.0 is not supported at this time. You can use other means available to you (CLI 1.0 and PowerShell). Thank you.


Turbo-Native copy in Micronaut using Thymeleaf

Original Node/Express version here: https://github.com/hotwired/turbo-native-demo
Todo

[x] Kotlin version
[x] Java version
[x] Groovy version
[ ] Tests
[x] Guide

@sdelamo Awesome changes 👍


feat: batching/filter banks support of lfilter

Resolves #1476, resolves #1528.
This implementation doensn't rely on multithreading library or custom cuda kernel, but use only python/C++ API of PyTorch.
After this changese, a_coeffs and b_coeffs can be 1D or 2D Tensor, where the optional dimension means number of filters and should be broadcastable to waveform.shape[:-1].
Will do some benchmarks later.

Hi @yoyololicon
Thanks for the PR. This is another wonderful addition. Can you add a batch consistency test?
Questions about batch behavior

What happens to samples that are padded for the sake of batching? Will the padding value have an impact on the result?
Is it easy to compute the valid length of padded signal?

Hi @yoyololicon
Thanks for the PR. This is another wonderful addition. Can you add a batch consistency test?

Sure.

Questions about batch behavior

What happens to samples that are padded for the sake of batching? Will the padding value have an impact on the result?
Is it easy to compute the valid length of padded signal?

I'm not sure whether the padding operation you are talking about is the behavior I call broadcasting.  As what I know, no extra values are padded to the signal.
I can give some examples of the behavior of filter from this merge:

two filters, one signal

>>> x = torch.randn(44100)
>>> a_coeffs = torch.rand(2, 3)
>>> b_coeffs = torch.rand(2, 3)
>>> F.lfilter(x, a_coeffs, b_coeffs).shape
(2, 44100)

The signal has been filtered by a set of filters, corresponding to the feature proposed in #1528 .

two filters, two signals

>>> x = torch.randn(2, 44100)
>>> a_coeffs = torch.rand(2, 3)
>>> b_coeffs = torch.rand(2, 3)
>>> F.lfilter(x, a_coeffs, b_coeffs).shape
(2, 44100)

In this example, each signal is filtered by its own set of filter coefficients, corresponding to the batching feature proposed in #1476.

two filters, multiple signals

>>> x = torch.randn(10, 1, 44100)
>>> a_coeffs = torch.rand(2, 3)
>>> b_coeffs = torch.rand(2, 3)
>>> F.lfilter(x, a_coeffs, b_coeffs).shape
(10, 2, 44100)

Batches of signals are filtered by a set of filters.

two filters, batches of stereo signals

>>> x = torch.randn(10, 2, 44100)
>>> a_coeffs = torch.rand(2, 3)
>>> b_coeffs = torch.rand(2, 3)
>>> F.lfilter(x, a_coeffs, b_coeffs).shape
(10, 2, 44100)

In this example, each channel of the signals is filtered by different coefficients.
I might benchmark the filter speed later this weekend cuz my machine is running some training these days.
Because the generic_lfilter_core_loop has also been modified to support batching, we also need to benchmark on CUDA as well.

@yoyololicon
By padding, I mean when I make a batch from signals with different number of samples.
Say, I make a batch from N and N/2 frame signals, I have to pad the end of the second signal.
Batch Input
- Signal 1: |xxxxxxxxxxxxxxxx|
- Signal 2: |xxxxxxxx--------|

Batch Result
- Signal 1: |yyyyyyyyyyyy|
- Signal 2: |yyyyyy------| <-- What is the valid length of resulting signal2?
                 ^^  <- Is the padding at the end have some effect on the result?

In this case, to get the resulting filtered signal2, I need to do the slicing and to do that I need to know the length of the resulting signal. i.e. batch[1][:valid_length]. This padding then masking operation happens commonly in RNNs and Attention mechanism, so I wanted to learn if something similar has to be in consideration.
However, looking at the examples you showed, lfilter seems to always return the same length of signals, so if user code knows the valid length, then it's easy to get the filtered signal. Is that correct?

@yoyololicon
By padding, I mean when I make a batch from signals with different number of samples.
Say, I make a batch from N and N/2 frame signals, I have to pad the end of the second signal.
In this case, to get the resulting filtered signal2, I need to do the slicing and to do that I need to know the length of the resulting signal. i.e. batch[1][:valid_length]. This padding then masking operation happens commonly in RNNs and Attention mechanism, so I wanted to learn if something similar has to be in consideration.

@mthrok
I see, thanks for the thorough explanation. Because lfilter is a causal system along time, you can just take the first N / 2 output values of signal 2. The remaining values have no impact on the result, in the above case.

However, looking at the examples you showed, lfilter seems to always return the same length of signals, so if user code knows the valid length, then it's easy to get the filtered signal. Is that correct?

According to above explaination, you are right.

Benchmarks using the same script from https://github.com/pytorch/audio/pull/1441#issuecomment-820148957, running on the same cpu.
I'm currently having problems with building CUDA enabled environment, so if someone can help compiled the changes, run this script on GPU and report the result here I'll be appreciated.
Before
[-------------- IIR filter --------------]
                   |  forward  |  backward
1 threads: -------------------------------
      [32, 256]    |    336.5  |   1089.4
      [32, 1024]   |    709.1  |   2537.1
      [32, 4096]   |   2447.2  |   9652.5
      [64, 256]    |    484.0  |   1587.9
      [64, 1024]   |   1214.6  |   4503.8
      [64, 4096]   |   8046.5  |  22566.8
      [128, 256]   |    715.6  |   2521.7
      [128, 1024]  |   2306.6  |   9286.6
      [128, 4096]  |  22508.9  |  52563.9
2 threads: -------------------------------
      [32, 256]    |    292.3  |    909.0
      [32, 1024]   |    581.4  |   1882.7
      [32, 4096]   |   1661.6  |   6729.1
      [64, 256]    |    409.6  |   1269.9
      [64, 1024]   |    909.0  |   3018.8
      [64, 4096]   |   8329.7  |  17457.2
      [128, 256]   |    594.3  |   1880.2
      [128, 1024]  |   1629.7  |   6745.9
      [128, 4096]  |  22276.9  |  44076.9
4 threads: -------------------------------
      [32, 256]    |    274.6  |    841.4
      [32, 1024]   |    513.3  |   1584.7
      [32, 4096]   |   1411.8  |   5429.1
      [64, 256]    |    361.4  |   1062.1
      [64, 1024]   |    786.3  |   2363.2
      [64, 4096]   |   7515.3  |  16583.1
      [128, 256]   |    514.5  |   1556.7
      [128, 1024]  |   1378.2  |   5592.4
      [128, 4096]  |  24044.3  |  42045.6

Times are in microseconds (us).

After
[-------------- IIR filter --------------]
                   |  forward  |  backward
1 threads: -------------------------------
      [32, 256]    |    356.1  |   1222.8
      [32, 1024]   |    731.5  |   3058.8
      [32, 4096]   |   2558.1  |  12071.8
      [64, 256]    |    502.4  |   1869.7
      [64, 1024]   |   1278.5  |   5729.2
      [64, 4096]   |   9484.0  |  27662.5
      [128, 256]   |    777.6  |   3126.6
      [128, 1024]  |   2330.3  |  11658.8
      [128, 4096]  |  22594.8  |  60734.4
2 threads: -------------------------------
      [32, 256]    |    309.1  |    982.0
      [32, 1024]   |    607.0  |   2186.2
      [32, 4096]   |   2005.4  |   8002.2
      [64, 256]    |    425.7  |   1414.8
      [64, 1024]   |    934.9  |   3624.9
      [64, 4096]   |   9420.4  |  20646.1
      [128, 256]   |    622.6  |   2195.2
      [128, 1024]  |   1697.0  |   7563.6
      [128, 4096]  |  22331.7  |  48910.4
4 threads: -------------------------------
      [32, 256]    |    294.0  |    910.8
      [32, 1024]   |    530.9  |   1781.8
      [32, 4096]   |   1391.8  |   5934.0
      [64, 256]    |    378.9  |   1188.6
      [64, 1024]   |    816.5  |   2699.3
      [64, 4096]   |   9485.3  |  17686.3
      [128, 256]   |    533.5  |   1707.9
      [128, 1024]  |   1410.3  |   6197.0
      [128, 4096]  |  22828.6  |  43533.4

Times are in microseconds (us).

Hi @yoyololicon
Sorry for the delayed response. I was busy with release-related work. Now torchaudio 0.9 is out and I put your work around lfilter autograd support and performance improvement as one of the major update. Thanks again for all the contribution.
For this PR, I am finding that filter bank support is more complicated than I had imagined. This is natural and I see it analogous to how convolution and broadcasting there is complicated as well.
So can we split this PR into one for batch support and the other for filter bank support? I think we can tackle batch support first as the goal is simple, it is just about applying the same operation to a batch of sample without for-loop. Once it's done we can look at the filter bank support to discuss the broadcasting semantics.

@mthrok
It's nice to see that my contribution is included in the new release, thanks again for your kind help and support. 🎉

I think we can tackle batch support first as the goal is simple, it is just about applying the same operation to a batch of sample without for-loop.

I have a different understanding of the batch support feature.
Based on the example from #1561, the shape of waveform is (batch, time), and the shape of *_coeffs are (batch, n_order). What it was trying to do is filtering different signal with different coefficients in parallel, so I think it's not the same operation for each sample in batch.
Current version of lfilter has actually already support the type of feature you are talking about, where the shape of waveform is (..., time), and ... can be interpreted as batch dimensions.
I think the new feature we are going to support, is batch of parallel lfilters.
I'm ok to split this PR into two, or even three where one for batch, one for filter bank, and the other for broadcasting semantics.
We can leave this PR as dicussion place or the last PR of three.
I actually think filter bank support is easier than batch support so I might open it first.

Current version of lfilter has actually already support the type of feature you are talking about, where the shape of waveform is (..., time), and ... can be interpreted as batch dimensions.
I think the new feature we are going to support, is batch of lfilters in parallel.

Oh sorry, yes that's what was in my mind. It is not about adding a batch support, but making the batch support efficient (presumably by moving the for loop into convolution ops)

I'm ok to split this PR into two, or even three where one for batch, one for filter bank, and the other for broadcasting semantics.
We can leave this PR as dicussion place or the last PR of three.
I actually think filter bank support is easier than batch support so I suggest to open it first.

Sure. That works. Please proceed with the approach most comfortable for you. I will try my best to understand the underlying logic.

Since #1587 and #1638 has now merged, I will close this draft MR.


Not very clear on how to filter query via dataFilter prop in IndexAction

What steps will reproduce the problem?
I have the following scenario:

I am using REST API by Bearer type of token. User is logged in, all ok.
Then I am trying to query certain data from db and would like to modify query to have current identitie's ID in it, so we can see only his own items

I tried to use dataFilter but no result, by the debugger I can see it is now passing validations phase.
What is the expected result?
Understand on how to filter query by dataFilter property in IndexAction.
This example is not very clear.
What do you get instead?
Not filtered response.
Additional info

Q
A

Yii version
2.0.16

PHP version
7.2.x

Operating system
Ubuntu

Could you give example and possibly add it to the here and official docs if possilble.
Thank you.

Your controller actions method needs to look something like this:
	public function actions() {
		$actions = parent::actions();

		$actions['index']['dataFilter'] = [
			'class' => ActiveDataFilter::class,
			// these seem to get put into attributes of the above specified class
			'queryOperatorMap' => ['LIKE' => 'ILIKE'],
			'searchModel' => function () {
				return (new DynamicModel(['name' => null, 'description' => null]))
					->addRule(['name', 'description'], 'trim')
					->addRule(['name', 'description'], 'string');
			}
		];

		return $actions;
	}

But that's not enough. You also need to override prepareDataProvider:
	protected function prepareDataProvider() {
		$request_params = Yii::$app->getRequest()->getQueryParams();

		/**
		 * shouldn't need to do this (might be fixed in newer versions of Yii2?) but the filter argument doesn't
		 * seem to get properly converted before being passed to vendor/yiisoft/yii2/data/DataFilter.php
		 */
		if (array_key_exists('filter', $request_params)) {
			$request_params['filter'] = json_decode(stripcslashes($request_params['filter']), true);
			// put the fixed filter query parameter back into Yii2's query parameter store
			Yii::$app->getRequest()->setQueryParams($request_params);
		}

		$data_provider = parent::prepareDataProvider();
		// when $data_provider is filtering, it becomes a DataFilter for some reason
		if ($data_provider->hasProperty('sort')) {
			$data_provider->sort->enableMultiSort = true;
		}

		return $data_provider;
	}

This is using Yii <IP_ADDRESS> so maybe the last bit is no longer necessary.

Filtering info got updated in https://github.com/yiisoft/yii2/pull/19357 does that address this issue?

It should. Thanks for the info!


Overview Timeline?

I've seen other people post screenshots in the issues like this:

On my own setup I never see this timeline widget. I've pulled to make sure everything is up to date.

How do you display the timeline, or should it do so by default?

That's daily charts and was introduced in 0.2.1

You need to add these to your settings file.
DAILY_REPORTS_CHART_ENABLED = True
DAILY_REPORTS_CHART_DAYS = 8

@mterzo great it's working now. Looks awesome 👍

@manuq Added it props to him.


Allow number and boolean as JSON schema options

See https://github.com/hyperledger/firefly-signer/pull/11

Allows number for non-integer types (fixed/ufixed) that can be specified as non-string
Update boolean support to include true/false strings as an option (all elementary types allow string basically)

This is necessary, but the changes in this PR allow several combinations that should not be allowed. For example, this allows a schema that accepts both a boolean and a number.  I've implemented it slightly differently in:
https://github.com/hyperledger/firefly-common/pull/22


The plugin mobile_scanner requires your app to be migrated to the Android embedding v2.

This error happens if I try to build  app on github actions or bitbacket pipiline.
Im using mobile_scanner 3.5.5 version and
Flutter 3.16.2
After I run dart pub get  action I get this error:
The plugin mobile_scanner requires your app to be migrated to the Android embedding v2. Follow the steps on the migration doc above and re-run this command.

The mobile_scanner plugin uses the Android embedding V2 setup, as the Android embedding V1 is deprecated.
Your should be using the migration guide to update your app to Android Embedding V2
See https://docs.flutter.dev/release/breaking-changes/android-v1-embedding-create-deprecation

👍 Thnx


[BUG] Different answers between SumF and SumSP

I was using SumSP and was getting an answer of 18 but on one machine (Lower spec) it consistently got 17 ???
So I had a look at the code and on the machine that get 18, and I get the following answers
(Sorry cannot give you the data set as it is embedded in the datastores)
        float lengthSPM = lengths.SumSP(MIN_STEP_FILTER_POINTS); // get 18
        float lengthSP = lengths.SumSP(); // get 13
        float lengthS = lengths.SumS(); // get 17
        float lengthF = lengths.SumF(); // get 17

floating point math is not commutative, so if you add things in different orders, you can get different answers.  a+b+c+d != (a+b)+(c+d) for instance.
the order in which things get added up when you go parallel is not deterministic (depends on core count for instance) so you can get different answers.  For the same reason a SIMD sum may not equal a non SIMD sum (though the SIMD will be more accurate, actually)


Saves try crouch to avoid issues on save game load

Save the variable 'try_crouch' in CogitoPlayer and ensures crouching doesn't interfere with other movement actions.

Tested saving and loading from beneath the stairs with "Toggle Crouching" set to both off and on
Tested behavior with keypad interaction, while attempting to sprint and sit

I was in the process of writing up a bug report but decided to fix the issue myself as it wasn't too problematic. I found that the player could get stuck if saving the game while crouched beneath the stairs and then loading from main menu.
In this video "Toggle Crouching" is set to OFF in options and I'm holding the crouch button down while entering the pause menu and pressing save. Then on save game load the player character automatically stands up and gets locked in place.
https://github.com/user-attachments/assets/1b568ee9-096a-47ac-9324-f0f35de14259

Hey, thanks for this. I've had a family emergency, so for the next couple weeks, I won't be able to work on Cogito as much as I used to. I'll trust all Cogito collaborators to review and approve any PRs and merges.

No problem @Phazorknight, hope all will be well with your family. @niefia has been quite active here so perhaps they can review this minor commit before merging.


Setup emotion and convert Button

Description
Configures emotion for use within the site and jest. Also converts the <Button /> component to use emotion rather than CSS modules.

Would we be able to do a walk through of this just so I know some best practices for using emotion?

Totally!


Fix ssl deprecate warn

ssl.wrap_socket() is deprecated from python 3.7 (see https://docs.python.org/3/library/ssl.html#ssl.SSLSocket). Based on documentation SSLContext.wrap_socket() should be used instead.

is it possible to add / update relevant tests as well?

All the existing tests are passing.

This needs to be released. It's not only deprecated, as of Python 3.12 the feature has been removed.

@auvipy Hey! Can we by chance bump the version to release this change?


Push Notifications setting lost again

Description
It appears I have again lost the setting where I enabled push notifications. I think this was an issue in the past that was fixed at one point.
Steps to Reproduce
Open PWA and find Push Notifications disabled on the notifications page when it was enabled previously.
Expected behavior
Push notifications remains enabled if enabled.
Screenshots
N/A
Logs
N/A
Environment:
iOS PWA
Additional context
N/A
cc @ekzyis

No, I am not sure it was fixed. I thought it was reported as fixed, but perhaps that was just optimistic :)
Let me know if there's any additional information I can provide to help debug further.

No, I am not sure it was fixed. I thought it was reported as fixed, but perhaps that was just optimistic :)

Ah, I see, haha, yes, we were hoping in this release it would be fixed now with these changes.

Let me know if there's any additional information I can provide to help debug further.

Thanks for the offer, now that you ask, there might indeed be something: Can you confirm if anything is saved as subscription in your IndexedDB? Don't share the full URL / the keys of the subscription since anyone could use that to send push subscriptions to your device. If anything is stored there, it should be the current subscription and thus fine.

Other than that, I don't think there is any more information we can request at the moment. I also just noticed we don't log user ids during web push errors so we can't associate errors with users with specific subscriptions at the moment. Will push a fix soon.
Opt-in client-side logging would be useful at this point for debugging now. I have only worked with Sentry so far and Sentry seems overkill for us at this stage. This website mentions some simpler alternatives including just rolling our own code.
@huumn, what do you think about client-side logging?

I'm for it. Two requirements:

doesn't in any way violate user privacy
without a doubt helps us help stackers

Thanks for the offer, now that you ask, there might indeed be something: Can you confirm if anything is saved as subscription in your IndexedDB? Don't share the full URL / the keys of the subscription since anyone could use that to send push subscriptions to your device. If anything is stored there, it should be the current subscription and thus fine.

On my desktop, yes. I don't know how to check this on my iOS device though, which is where I noticed the problem.

Can confirm this isn't fixed on iOS at least

Same, my setting gets lost every few days it seems.

#463 worked and I think we found the culprit thanks to a HN post:

7-Day Cap on All Script-Writeable Storage

-- https://webkit.org/blog/10218/full-third-party-cookie-blocking-and-more/
So this means your Apple devices delete the existing subscription in IndexedDB and then, when the service worker checks if the subscription needs an update, it does not find a previous one to check if it should sent the updated subscription to the server.

I haven't fully confirmed this since there might have been some changes (the article is from 2020) but I think this is it.

Mhh, but reading the resource thoroughly:

deleting all of a website’s script-writable storage after seven days of Safari use without user interaction on the site

What counts as a "user interaction"? Anything? Then IndexedDB should not have been cleared.

A Note On Web Applications Added to the Home Screen
As mentioned, the seven-day cap on script-writable storage is gated on “after seven days of Safari use without user interaction on the site.” That is the case in Safari. Web applications added to the home screen are not part of Safari and thus have their own counter of days of use. Their days of use will match actual use of the web application which resets the timer. We do not expect the first-party in such a web application to have its website data deleted.
If your web application does experience website data deletion, please let us know since we would consider it a serious bug. It is not the intention of Intelligent Tracking Prevention to delete website data for first parties in web applications.

Web apps added to home screen = PWAs? Also, actual use resets the timer? The timer does not seem to have been reset here.
So either:

this is a bug
this behavior changed and it gets deleted even with user interaction
i don't really understand this text

For sake of completeness, here the related log lines:
2023-09-19T04:02:13.342Z |  51 [info] ███ | sent SYNC_SUBSCRIPTION to service worker | - | os=iOS
2023-09-19T04:02:13.463Z |  54 [info] ███ | [sw:handlePushSubscriptionChange] old subscription matches existing subscription | - | os=iOS
2023-09-19T15:08:32.235Z | 205 [info] ███ | sent SYNC_SUBSCRIPTION to service worker | - | os=iOS
2023-09-19T15:08:32.321Z | 208 [info] ███ | [sw:handlePushSubscriptionChange] no existing subscription found | - | os=iOS
2023-09-20T04:24:48.269Z | 441 [info] ███ | subscribed to push notifications | endpoint=███ | os=iOS

log 51 and 54 show that there was still a push subscription.
log 205 and 208 then show there is suddenly no push subscription anymore ... but wait.
I just checked the code and this line is printed if the service worker has no active subscription anymore:
  newSubscription ??= await self.registration.pushManager.getSubscription()
  if (!newSubscription) {
    // no subscription exists at the moment
    messageChannelPort?.postMessage({ message: '[sw:handlePushSubscriptionChange] no existing subscription found' })
    return
  }

So "existing" does not mean there is no subscription stored in IndexedDB at the moment. It means the service worker itself has no push subscription (=> checkbox unchecked). Therefore, the issue is because the service worker just lost the push subscription. However, the resource also mentions that registrations are deleted:

Service Worker registrations and cache

So this may still be the reason, I just missed that detail that it's not IndexedDB that we see to be suddenly empty, it's the actual registration of the service worker. And with that, push subscriptions are probably also gone.

This might be a red herring but it looks like we initially think service workers aren't supported, then we register one.
2023-09-20T12:42:05.725Z | 747 [info] serene-voyager-4004 | device does not support service worker | - | os=iOS
2023-09-20T12:42:05.818Z | 748 [info] serene-voyager-4004 | sent SYNC_SUBSCRIPTION to service worker | - | os=iOS
2023-09-20T12:42:05.864Z | 749 [info] serene-voyager-4004 | [sw:handlePushSubscriptionChange] old subscription matches existing subscription | - | os=iOS
2023-09-20T12:42:05.866Z | 750 [info] serene-voyager-4004 | [sw:message] received message | action=MESSAGE_PORT | os=iOS
2023-09-20T12:42:06.034Z | 752 [info] serene-voyager-4004 | service worker registration successful | - | os=iOS

Yes, it looks weird in the logs but this happens for every device. IIRC this is because of render order / useEffect. It first logs no support, then in the next render, the state is updated and the service worker is registered

I've created a 500k bounty for this: https://stacker.news/items/277155
Also, we've learned this isn't strictly iOS. It also affects android devices

A push subscription is a message delivery context established between the user agent and the push service on behalf of a web application. Each push subscription is associated with a service worker registration and a service worker registration has at most one push subscription. ^1

This makes me wonder if we are replacing service worker registrations and that is causing the existing PushSubscription to be cleared?
I verified that the subscription is still stored in IndexedDB, but this line indicates that there is no PushSubscription found.
I also tried re-subscribing and verified that there is no expiration time set on the subscription, so that eliminates expiration as the source of this issue.
I don't know enough about service workers, but these lines feel like we're replacing the service worker registration, which could lead to removal of the push subscription, though if that was the case, I'd expect this to be a more persistent issue.

I just realized that I've been using the wrong function signature for pushsubscriptionchange:
https://github.com/stackernews/stacker.news/blob/e24e2a6b20f174dd856f1dbf26fbcf2998551f01/sw/index.js#L74
https://github.com/stackernews/stacker.news/blob/e24e2a6b20f174dd856f1dbf26fbcf2998551f01/sw/eventListener.js#L100-L102
But I think that's unrelated to this bug since most browsers don't support this event anyway and it's the right function signature for the workaround code:
https://github.com/stackernews/stacker.news/blob/e24e2a6b20f174dd856f1dbf26fbcf2998551f01/sw/eventListener.js#L155-L157

this is probably the bug and we can register a new push subscription in the background and send that one to the server.

My only concern is if this is "OK" to do since it wouldn't be driven by an explicit user interaction? To register a new push subscription automatically under certain conditions. Just food for thought. I like the workaround in general, should improve the experience!

My only concern is if this is "OK" to do since it wouldn't be driven by an explicit user interaction? To register a new push subscription automatically under certain conditions. Just food for thought. I like the workaround in general, should improve the experience!

Good point but I think it's okay since this only works if permissions were already granted. We're acting in the interest of our users here since they told us they want push notifications.
The user can always opt-out via browser settings. The "game theory" of permissions is that applications should behave with the permissions they were granted else the user will remove permission via the browser which is a "nuclear option" since the application won't even be able to ask for permission again then:

Your site should explain to your users how they can disable push. If you don't, users are likely to take the nuclear option and block permission permanently.

-- https://web.dev/articles/push-notifications-permissions-ux#the_bad_ux

Just lost mine again today ... after what felt like longer than normal at least!

Mhh, I think this was the related log message right:

2023-11-14T23:46:33.717Z | 53568 [info] serene-voyager-4004 | [sw:handlePushSubscriptionChange] no existing subscription found | - | os=iOS

Since it says no existing subscription found instead of service worker lost subscription as in this line think your old subscription in IndexedDB wasn't created  with swVersion: 2 set yet
Maybe the resubscribe code will work now for you

Okay cool, will keep commenting here when they're lost

I lose my notifications daily now and receive the "Stacker News notifications are active now" nearly every time I open the app, and it's a coinflip if my checkbox is checked. It's super super broken.
I suspect we're misunderstanding something about how these work on phones.

I lose my notifications daily now

What does "daily now" mean? How many days already?

and receive the "Stacker News notifications are active now" nearly every time I open the app, and it's a coinflip if my checkbox is checked. It's super super broken.

Yeah I wonder if I just never noticed that my checkbox might be unchecked ... i haven't tested if I might still receive push notifications even if it "seems" unchecked
But weird, I think we haven't changed anything in the service worker?
$ git log -5 --oneline -- sw/*
0b872cb Only resubscribe on compatible old subscriptions (#616)
522c821 Notification badges (#595)
5dfeb70 Resubscribe if service worker lost push subscription (#597)
7040dbf Fix missing merge of FOLLOW push notifications (#596)
eeaf2d5 Fix FORWARDEDTIP change lost in serviceworker (#582)

It's been broken for awhile not just recently.

I wonder if this might be related to storage getting cleared when a device is running out of space. I have an old iphone that's constantly evicting apps and is probably also evicting pwa storage.
Perhaps we could use persistent storage? https://developer.mozilla.org/en-US/docs/Web/API/StorageManager/persist

I wonder if this might be related to storage getting cleared when a device is running out of space.

Ohh, my phone is also constantly running out of space. This might be indeed the reason that I also started to lose my push subscription recently.

Perhaps we could use persistent storage? https://developer.mozilla.org/en-US/docs/Web/API/StorageManager/persist

Sounds like a good idea!

I don't have them enabled on desktop. I always have a tab open

:eyes:

You can ask the browser for persistent storage on compatible platforms to avoid automatic data eviction after inactivity or on storage pressure. If granted, the browser will never evict data from storage. This protection includes the service worker registration, IndexedDB databases, and files in cache storage. Note that users are always in charge, and they can delete the storage at any time, even if the browser has granted persistent storage.

-- https://web.dev/learn/pwa/offline-data#data_persistence
Will create PR soon. If I understand this right, it's really just one simple API call.

Are we maybe accidentally doing silent pushes? https://dev.to/progressier/how-to-fix-ios-push-subscriptions-being-terminated-after-3-notifications-39a7

Are we maybe accidentally doing silent pushes? https://dev.to/progressier/how-to-fix-ios-push-subscriptions-being-terminated-after-3-notifications-39a7
We're using event.waitUntil in onPush but we aren't awaiting all the showNotification calls, ie we're returning a promise that returns a promise in several instances.

I think returning a promise inside a promise is fine. As far as my JS knowledge goes, there is no difference for the calling context if you do return <Promise> or return await <Promise> except if it's wrapped inside try/catch. Then you need to use await to trigger catch on errors. And afaict, we're always at least returning them. There are no "loose promises" when it comes to showNotification.
But we might be doing silent pushes elsewhere. I vaguely remember that I was wondering if X could cause a silent push but I no longer know what X was :thinking:

I think returning a promise inside a promise is fine.

I don't think it is. What happens here afaict is by returning a promise the async function resolves to a potentially unresolved showNotification promise. Meaning, waitUntil is not waiting until showNotification resolves and the notification is not shown by the end of the event deterministically and these pushes are sometimes silent depending on how fast showNotification resolves.

I think returning a promise inside a promise is fine.

I don't think it is. What happens here afaict is by returning a promise the async function resolves to a potentially unresolved showNotification promise. Meaning, waitUntil is not waiting until showNotification resolves and the notification is not shown by the end of the event deterministically and these pushes are sometimes silent depending on how fast showNotification resolves.

Mhh, I see. I also see how my argument might have been too hand-wavy. I'll create a PR with return await on all these promises in a few minutes.
I'll also think about skipNotification more.

Basically this function can only return a promise if that promise resolves, and only resolves, if showNotification resolves.

For bookkeeping purposes, I'll write down here what we've just seen in our logs and discussed internally on Slack:
We've noticed that this if block gets triggered in iOS:
if (notifications.length > 1) {
  const message = `[sw:push] more than one notification with tag ${tag} found`
  messageChannelPort?.postMessage({ level: 'error', message })
  console.error(message)
  return null
}

because we've found a lot of

2023-12-29T04:25:28.208Z | 170071 [error] serene-voyager-4004 | [sw:push] more than one notification with tag REPLY found | - | os=iOS

in our logs.
This makes sense since when we released push notifications, we realized pretty quick that tag is not properly supported by Safari on iOS. I created a WebKit bug report because of this back in July.
This is another silent push because of which Safari might cancel push subscriptions.
However, reading the last response from someone working at Apple or WebKit (?) got me confused:

An additional data point: this appears to be partially working on macOS Safari. While a second notification with the same tag appears as a new entry in the system notification center, it does replace an existing one in the array returned by ServiceWorkerRegistration.getNotifications(), and a notificationclose event is fired for the old/replaced notification. (For comparison, macOS Chrome replaces the notification in both places, and does not fire a notificationclose event for the replaced notification.)
On iOS Safari, ServiceWorkerRegistration.getNotifications() appears to always return an empty array, so there's no way to compare that behaviour.

Afaict, this doesn't add up with what we're seeing on iOS. According to the log message, getNotifications({ tag }) does return an array with more than one element.
tl;dr: Next attempt to fix this issue: Create PR which does not rely on tag anymore in showNotification. And I'll call getNotifications({ tag }) but manually make sure that the array indeed only contains notifications with the same tag. As mentioned in the bug report, the tag property is correctly set on the notifications themselves:

To be clear: the tag attribute on Notification is exposed. You can set it via the Notification constructor/ServiceWorkerRegistration.showNotification and access its value later on. But the browser currently doesn't use the property to coalesce notifications. That will require additional work.

Regarding UX on Android since #719: It still works but I can definitely tell that it no longer uses the native tag property. The notifications appear but no longer smoothly replace each other. Especially on desktop, I can see how the notification is closed and then a new notification pops up since there are enough milliseconds between these two calls that it's noticeable.
So I think it makes sense to add only run the code which does not depend on tag on iOS devices. This means we would need to detect the OS in the service worker, too. We currently only do that in the logger.

Agreed. We don't want to degrade android UX because iOS is behind.
Also, I've maintained my notifications longer than usual but I did lose them once yesterday. It might've just been a fluke.
I also spent sometime looking around for any clues about potential causes. It looks like many other devs have the same issue: https://developer.apple.com/forums/thread/728796

I have access to an iPhone with Safari on iOS 17.1 now. I was able to gather some information about what works and doesn't work.
The columns of the log messages are:

log id in database (shows log insertion order),
[sw:push]
unix timestamp recorded on device using Date.now(). This is supposed to show the definitive execution order on the device since the log messages are sent to the server and logged there so network conditions will scramble the order. However, the lacking precision (millisecond) of Date.now() prevents to know a definitive execution order of all log messages since some have the same timestamp. Will check tomorrow if I can get more accurate timings inside a service worker to make super sure we know the order in which our service worker code is executed. (But not hugely relevant imo)
randomly generated UUID per incoming notification (for easier tracing)
-
log message

"slow zaps": zapping the same post three times and waiting until notification shows up on device between zaps:
78515 [sw:push]<PHONE_NUMBER>114 52b1d2bf-de9a-4f3f-9cc7-8d39942ffac2 - received notification with tag TIP-328086
78516 [sw:push]<PHONE_NUMBER>118 52b1d2bf-de9a-4f3f-9cc7-8d39942ffac2 - built-in tag filter: []
78517 [sw:push]<PHONE_NUMBER>118 52b1d2bf-de9a-4f3f-9cc7-8d39942ffac2 - found 0 TIP-328086 notifications
78518 [sw:push]<PHONE_NUMBER>118 52b1d2bf-de9a-4f3f-9cc7-8d39942ffac2 - found 0 TIP-328086 notifications after manual tag filter
78519 [sw:push]<PHONE_NUMBER>118 52b1d2bf-de9a-4f3f-9cc7-8d39942ffac2 - manual tag filter: []
78520 [sw:push]<PHONE_NUMBER>118 52b1d2bf-de9a-4f3f-9cc7-8d39942ffac2 - TIP-328086 notifications replace previous notifications
78521 [sw:push]<PHONE_NUMBER>126 52b1d2bf-de9a-4f3f-9cc7-8d39942ffac2 - closing existing notifications
78522 [sw:push]<PHONE_NUMBER>126 52b1d2bf-de9a-4f3f-9cc7-8d39942ffac2 - show notification with title "your post stacked 4200 sats"

78523 [sw:push]<PHONE_NUMBER>854 09ae56c0-15e6-4385-9414-dda7b5df93c1 - received notification with tag TIP-328086
78524 [sw:push]<PHONE_NUMBER>859 09ae56c0-15e6-4385-9414-dda7b5df93c1 - found 1 TIP-328086 notifications
78525 [sw:push]<PHONE_NUMBER>859 09ae56c0-15e6-4385-9414-dda7b5df93c1 - manual tag filter: ["TIP-328086"]
78526 [sw:push]<PHONE_NUMBER>859 09ae56c0-15e6-4385-9414-dda7b5df93c1 - found 1 TIP-328086 notifications after manual tag filter
78527 [sw:push]<PHONE_NUMBER>859 09ae56c0-15e6-4385-9414-dda7b5df93c1 - built-in tag filter: ["TIP-328086"]
78528 [sw:push]<PHONE_NUMBER>859 09ae56c0-15e6-4385-9414-dda7b5df93c1 - TIP-328086 notifications replace previous notifications
78529 [sw:push]<PHONE_NUMBER>866 09ae56c0-15e6-4385-9414-dda7b5df93c1 - closing existing notifications
78530 [sw:push]<PHONE_NUMBER>866 09ae56c0-15e6-4385-9414-dda7b5df93c1 - show notification with title "your post stacked 4300 sats"

78531 [sw:push]<PHONE_NUMBER>520 be2ede18-cb9e-4e12-bb8e-1ae58c482145 - received notification with tag TIP-328086
78532 [sw:push]<PHONE_NUMBER>526 be2ede18-cb9e-4e12-bb8e-1ae58c482145 - built-in tag filter: ["TIP-328086","TIP-328086"]
78533 [sw:push]<PHONE_NUMBER>526 be2ede18-cb9e-4e12-bb8e-1ae58c482145 - found 2 TIP-328086 notifications after manual tag filter
78534 [sw:push]<PHONE_NUMBER>526 be2ede18-cb9e-4e12-bb8e-1ae58c482145 - found 2 TIP-328086 notifications
78535 [sw:push]<PHONE_NUMBER>526 be2ede18-cb9e-4e12-bb8e-1ae58c482145 - manual tag filter: ["TIP-328086","TIP-328086"]
78536 [sw:push]<PHONE_NUMBER>527 be2ede18-cb9e-4e12-bb8e-1ae58c482145 - TIP-328086 notifications replace previous notifications
78537 [sw:push]<PHONE_NUMBER>533 be2ede18-cb9e-4e12-bb8e-1ae58c482145 - closing existing notifications
78538 [sw:push]<PHONE_NUMBER>533 be2ede18-cb9e-4e12-bb8e-1ae58c482145 - show notification with title "your post stacked 4400 sats"

Conclusions:

tag property is exposed but not implemented. Notifications are not replacing each other even though they use same tag. This is already known since this is exactly the bug that was reported in July 2022.
Notification.close() does not seem to work, contrary to what is shown on MDN.

Conclusions:
"fast zaps": multiple zaps in quick succession
78707 [sw:push]<PHONE_NUMBER>301 c9213c71-2b6f-42f0-a354-1d13187c7443 - received notification with tag TIP-328086
78708 [sw:push]<PHONE_NUMBER>302 5da86de2-a182-4094-87c9-d31f024e54c8 - received notification with tag TIP-328086
78709 [sw:push]<PHONE_NUMBER>325 c9213c71-2b6f-42f0-a354-1d13187c7443 - found 0 TIP-328086 notifications
78710 [sw:push]<PHONE_NUMBER>325 c9213c71-2b6f-42f0-a354-1d13187c7443 - built-in tag filter: []
78711 [sw:push]<PHONE_NUMBER>325 c9213c71-2b6f-42f0-a354-1d13187c7443 - found 0 TIP-328086 notifications after manual tag filter
78712 [sw:push]<PHONE_NUMBER>325 c9213c71-2b6f-42f0-a354-1d13187c7443 - manual tag filter: []
78713 [sw:push]<PHONE_NUMBER>326 c9213c71-2b6f-42f0-a354-1d13187c7443 - TIP-328086 notifications replace previous notifications
78714 [sw:push]<PHONE_NUMBER>327 5da86de2-a182-4094-87c9-d31f024e54c8 - found 0 TIP-328086 notifications
78715 [sw:push]<PHONE_NUMBER>327 5da86de2-a182-4094-87c9-d31f024e54c8 - found 0 TIP-328086 notifications after manual tag filter
78716 [sw:push]<PHONE_NUMBER>327 5da86de2-a182-4094-87c9-d31f024e54c8 - built-in tag filter: []
78717 [sw:push]<PHONE_NUMBER>327 5da86de2-a182-4094-87c9-d31f024e54c8 - manual tag filter: []
78718 [sw:push]<PHONE_NUMBER>327 5da86de2-a182-4094-87c9-d31f024e54c8 - TIP-328086 notifications replace previous notifications
78719 [sw:push]<PHONE_NUMBER>460 c9213c71-2b6f-42f0-a354-1d13187c7443 - closing existing notifications
78720 [sw:push]<PHONE_NUMBER>460 c9213c71-2b6f-42f0-a354-1d13187c7443 - show notification with title "your post stacked 6600 sats"
78721 [sw:push]<PHONE_NUMBER>460 5da86de2-a182-4094-87c9-d31f024e54c8 - closing existing notifications
78722 [sw:push]<PHONE_NUMBER>460 5da86de2-a182-4094-87c9-d31f024e54c8 - show notification with title "your post stacked 6700 sats"
78723 [sw:push]<PHONE_NUMBER>666 a6a9b495-fd34-410d-9244-d74dfeb9d82c - received notification with tag TIP-328086
78726 [sw:push]<PHONE_NUMBER>557 a857a2d7-067d-406d-82ef-b8d8cf66bd0b - received notification with tag TIP-328086
78725 [sw:push]<PHONE_NUMBER>558 c77d7b27-d99b-4ebf-a13a-a977e9572fce - received notification with tag TIP-328086
78724 [sw:push]<PHONE_NUMBER>557 a3578425-94a8-4c16-96c0-7dfd085e2fe1 - received notification with tag TIP-328086
78727 [sw:push]<PHONE_NUMBER>650 a6a9b495-fd34-410d-9244-d74dfeb9d82c - found 2 TIP-328086 notifications
78728 [sw:push]<PHONE_NUMBER>650 a6a9b495-fd34-410d-9244-d74dfeb9d82c - built-in tag filter: ["TIP-328086","TIP-328086"]
78729 [sw:push]<PHONE_NUMBER>683 a3578425-94a8-4c16-96c0-7dfd085e2fe1 - found 2 TIP-328086 notifications
78732 [sw:push]<PHONE_NUMBER>685 a3578425-94a8-4c16-96c0-7dfd085e2fe1 - built-in tag filter: ["TIP-328086","TIP-328086"]
78730 [sw:push]<PHONE_NUMBER>651 a6a9b495-fd34-410d-9244-d74dfeb9d82c - found 2 TIP-328086 notifications after manual tag filter
78731 [sw:push]<PHONE_NUMBER>651 a6a9b495-fd34-410d-9244-d74dfeb9d82c - manual tag filter: ["TIP-328086","TIP-328086"]
78733 [sw:push]<PHONE_NUMBER>653 a6a9b495-fd34-410d-9244-d74dfeb9d82c - TIP-328086 notifications replace previous notifications
78734 [sw:push]<PHONE_NUMBER>685 a3578425-94a8-4c16-96c0-7dfd085e2fe1 - manual tag filter: ["TIP-328086","TIP-328086"]
78735 [sw:push]<PHONE_NUMBER>686 a857a2d7-067d-406d-82ef-b8d8cf66bd0b - found 2 TIP-328086 notifications
78736 [sw:push]<PHONE_NUMBER>685 a3578425-94a8-4c16-96c0-7dfd085e2fe1 - TIP-328086 notifications replace previous notifications
78737 [sw:push]<PHONE_NUMBER>686 a857a2d7-067d-406d-82ef-b8d8cf66bd0b - built-in tag filter: ["TIP-328086","TIP-328086"]
78738 [sw:push]<PHONE_NUMBER>686 a857a2d7-067d-406d-82ef-b8d8cf66bd0b - found 2 TIP-328086 notifications after manual tag filter
78739 [sw:push]<PHONE_NUMBER>685 a3578425-94a8-4c16-96c0-7dfd085e2fe1 - found 2 TIP-328086 notifications after manual tag filter
78740 [sw:push]<PHONE_NUMBER>686 c77d7b27-d99b-4ebf-a13a-a977e9572fce - found 2 TIP-328086 notifications
78741 [sw:push]<PHONE_NUMBER>686 a857a2d7-067d-406d-82ef-b8d8cf66bd0b - manual tag filter: ["TIP-328086","TIP-328086"]
78742 [sw:push]<PHONE_NUMBER>686 a857a2d7-067d-406d-82ef-b8d8cf66bd0b - TIP-328086 notifications replace previous notifications
78743 [sw:push]<PHONE_NUMBER>686 c77d7b27-d99b-4ebf-a13a-a977e9572fce - built-in tag filter: ["TIP-328086","TIP-328086"]
78744 [sw:push]<PHONE_NUMBER>687 c77d7b27-d99b-4ebf-a13a-a977e9572fce - manual tag filter: ["TIP-328086","TIP-328086"]
78745 [sw:push]<PHONE_NUMBER>687 c77d7b27-d99b-4ebf-a13a-a977e9572fce - found 2 TIP-328086 notifications after manual tag filter
78746 [sw:push]<PHONE_NUMBER>687 c77d7b27-d99b-4ebf-a13a-a977e9572fce - TIP-328086 notifications replace previous notifications
78747 [sw:push]<PHONE_NUMBER>726 a6a9b495-fd34-410d-9244-d74dfeb9d82c - closing existing notifications
78748 [sw:push]<PHONE_NUMBER>759 a6a9b495-fd34-410d-9244-d74dfeb9d82c - show notification with title "your post stacked 6800 sats"
78749 [sw:push]<PHONE_NUMBER>764 a3578425-94a8-4c16-96c0-7dfd085e2fe1 - closing existing notifications
78750 [sw:push]<PHONE_NUMBER>770 a3578425-94a8-4c16-96c0-7dfd085e2fe1 - show notification with title "your post stacked 6900 sats"
78751 [sw:push]<PHONE_NUMBER>771 a857a2d7-067d-406d-82ef-b8d8cf66bd0b - closing existing notifications
78752 [sw:push]<PHONE_NUMBER>771 a857a2d7-067d-406d-82ef-b8d8cf66bd0b - show notification with title "your post stacked 7000 sats"
78753 [sw:push]<PHONE_NUMBER>774 c77d7b27-d99b-4ebf-a13a-a977e9572fce - closing existing notifications
78754 [sw:push]<PHONE_NUMBER>784 c77d7b27-d99b-4ebf-a13a-a977e9572fce - show notification with title "your post stacked 7100 sats"

Conclusions:

(most likely) due to bad support of tag, there is a race condition when it comes to getNotifications({ tag }). The first two notifications fetch 0 notifications of this tag, while the remaining notifications all fetch 2 of this tag. This is no problem for zaps since we don't rely on the amount of existing notifications there. We simply want to replace the previous notification. However, this gets more important for notification like replies.

"slow" replies
78755 [sw:push]<PHONE_NUMBER>668 854cf725-a817-4b36-82b5-61f2d2f8d955 - received notification with tag REPLY
78756 [sw:push]<PHONE_NUMBER>721 854cf725-a817-4b36-82b5-61f2d2f8d955 - found 0 REPLY notifications
78757 [sw:push]<PHONE_NUMBER>721 854cf725-a817-4b36-82b5-61f2d2f8d955 - built-in tag filter: []
78758 [sw:push]<PHONE_NUMBER>722 854cf725-a817-4b36-82b5-61f2d2f8d955 - found 0 REPLY notifications after manual tag filter
78759 [sw:push]<PHONE_NUMBER>753 854cf725-a817-4b36-82b5-61f2d2f8d955 - manual tag filter: []
78760 [sw:push]<PHONE_NUMBER>754 854cf725-a817-4b36-82b5-61f2d2f8d955 - no existing REPLY notifications found
78761 [sw:push]<PHONE_NUMBER>755 854cf725-a817-4b36-82b5-61f2d2f8d955 - show notification with title "@036d6006e2 replied to you"

78762 [sw:push]<PHONE_NUMBER>673 1391ffc8-55c3-4170-858b-add2cc723eca - received notification with tag REPLY
78763 [sw:push]<PHONE_NUMBER>726 1391ffc8-55c3-4170-858b-add2cc723eca - incoming payload.options.data: {"url":"/items/328086?commentId=328102","itemId":328102}
78764 [sw:push]<PHONE_NUMBER>724 1391ffc8-55c3-4170-858b-add2cc723eca - found 1 REPLY notifications
78765 [sw:push]<PHONE_NUMBER>724 1391ffc8-55c3-4170-858b-add2cc723eca - built-in tag filter: ["REPLY"]
78766 [sw:push]<PHONE_NUMBER>724 1391ffc8-55c3-4170-858b-add2cc723eca - found 1 REPLY notifications after manual tag filter
78767 [sw:push]<PHONE_NUMBER>724 1391ffc8-55c3-4170-858b-add2cc723eca - manual tag filter: ["REPLY"]
78768 [sw:push]<PHONE_NUMBER>726 1391ffc8-55c3-4170-858b-add2cc723eca - using REPLY for control flow
78769 [sw:push]<PHONE_NUMBER>726 1391ffc8-55c3-4170-858b-add2cc723eca - notifications: ["{\"url\":\"/items/328086?commentId=328101\",\"itemId\":328101}"]
78770 [sw:push]<PHONE_NUMBER>726 1391ffc8-55c3-4170-858b-add2cc723eca - initial amount: 1
78771 [sw:push]<PHONE_NUMBER>727 1391ffc8-55c3-4170-858b-add2cc723eca - merged payload: {"url":"/items/328086?commentId=328101","itemId":328101,"amount":2}
78772 [sw:push]<PHONE_NUMBER>727 1391ffc8-55c3-4170-858b-add2cc723eca - calculated title: you have 2 new replies
78773 [sw:push]<PHONE_NUMBER>727 1391ffc8-55c3-4170-858b-add2cc723eca - closing existing notifications
78774 [sw:push]<PHONE_NUMBER>753 1391ffc8-55c3-4170-858b-add2cc723eca - show notification with title "you have 2 new replies"

78775 [sw:push]<PHONE_NUMBER>506 3168f30d-a4ff-4dba-b316-a6d885dbe656 - received notification with tag REPLY
78776 [sw:push]<PHONE_NUMBER>644 3168f30d-a4ff-4dba-b316-a6d885dbe656 - found 2 REPLY notifications
78777 [sw:push]<PHONE_NUMBER>647 3168f30d-a4ff-4dba-b316-a6d885dbe656 - incoming payload.options.data: {"url":"/items/328086?commentId=328103","itemId":328103}
78778 [sw:push]<PHONE_NUMBER>645 3168f30d-a4ff-4dba-b316-a6d885dbe656 - more than one notification with tag REPLY found
78779 [sw:push]<PHONE_NUMBER>644 3168f30d-a4ff-4dba-b316-a6d885dbe656 - built-in tag filter: ["REPLY","REPLY"]
78780 [sw:push]<PHONE_NUMBER>645 3168f30d-a4ff-4dba-b316-a6d885dbe656 - found 2 REPLY notifications after manual tag filter
78781 [sw:push]<PHONE_NUMBER>647 3168f30d-a4ff-4dba-b316-a6d885dbe656 - using REPLY for control flow
78782 [sw:push]<PHONE_NUMBER>645 3168f30d-a4ff-4dba-b316-a6d885dbe656 - manual tag filter: ["REPLY","REPLY"]
78783 [sw:push]<PHONE_NUMBER>647 3168f30d-a4ff-4dba-b316-a6d885dbe656 - notifications: ["{\"url\":\"/items/328086?commentId=328101\",\"itemId\":328101,\"amount\":2}","{\"url\":\"/items/328086?commentId=328101\",\"itemId\":328101}"]
78784 [sw:push]<PHONE_NUMBER>647 3168f30d-a4ff-4dba-b316-a6d885dbe656 - initial amount: 2
78785 [sw:push]<PHONE_NUMBER>645 3168f30d-a4ff-4dba-b316-a6d885dbe656 - skip bail -- merging notifications with tag REPLY manually
78786 [sw:push]<PHONE_NUMBER>649 3168f30d-a4ff-4dba-b316-a6d885dbe656 - merged payload: {"url":"/items/328086?commentId=328101","itemId":328101,"amount":4}
78787 [sw:push]<PHONE_NUMBER>649 3168f30d-a4ff-4dba-b316-a6d885dbe656 - calculated title: you have 4 new replies
78788 [sw:push]<PHONE_NUMBER>649 3168f30d-a4ff-4dba-b316-a6d885dbe656 - closing existing notifications
78789 [sw:push]<PHONE_NUMBER>676 3168f30d-a4ff-4dba-b316-a6d885dbe656 - show notification with title "you have 4 new replies"

78790 [sw:push]<PHONE_NUMBER>214 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - received notification with tag REPLY
78791 [sw:push]<PHONE_NUMBER>349 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - found 3 REPLY notifications
78792 [sw:push]<PHONE_NUMBER>352 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - incoming payload.options.data: {"url":"/items/328086?commentId=328104","itemId":328104}
78793 [sw:push]<PHONE_NUMBER>352 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - using REPLY for control flow
78794 [sw:push]<PHONE_NUMBER>351 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - skip bail -- merging notifications with tag REPLY manually
78795 [sw:push]<PHONE_NUMBER>349 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - built-in tag filter: ["REPLY","REPLY","REPLY"]
78796 [sw:push]<PHONE_NUMBER>350 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - found 3 REPLY notifications after manual tag filter
78797 [sw:push]<PHONE_NUMBER>352 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - notifications: ["{\"url\":\"/items/328086?commentId=328101\",\"itemId\":328101,\"amount\":4}","{\"url\":\"/items/328086?commentId=328101\",\"itemId\":328101,\"amount\":2}","{\"url\":\"/items/328086?commentId=328101\",\"itemId\":328101}"]
78798 [sw:push]<PHONE_NUMBER>350 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - manual tag filter: ["REPLY","REPLY","REPLY"]
78799 [sw:push]<PHONE_NUMBER>352 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - initial amount: 4
78800 [sw:push]<PHONE_NUMBER>351 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - more than one notification with tag REPLY found
78801 [sw:push]<PHONE_NUMBER>354 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - merged payload: {"url":"/items/328086?commentId=328101","itemId":328101,"amount":7}
78802 [sw:push]<PHONE_NUMBER>354 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - calculated title: you have 7 new replies
78803 [sw:push]<PHONE_NUMBER>354 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - closing existing notifications
78804 [sw:push]<PHONE_NUMBER>371 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - show notification with title "you have 7 new replies"

Conclusions:

We're showing the wrong amount of replies because we're adding the amount of existing notifications basically twice. Relevant log messages (they should make clear that this is what is happening):

78796 [sw:push]<PHONE_NUMBER>350 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - found 3 REPLY notifications after manual tag filter
78797 [sw:push]<PHONE_NUMBER>352 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - notifications: ["{\"url\":\"/items/328086?commentId=328101\",\"itemId\":328101,\"amount\":4}","{\"url\":\"/items/328086?commentId=328101\",\"itemId\":328101,\"amount\":2}","{\"url\":\"/items/328086?commentId=328101\",\"itemId\":328101}"]
78798 [sw:push]<PHONE_NUMBER>350 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - manual tag filter: ["REPLY","REPLY","REPLY"]
78799 [sw:push]<PHONE_NUMBER>352 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - initial amount: 4
78800 [sw:push]<PHONE_NUMBER>351 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - more than one notification with tag REPLY found
78801 [sw:push]<PHONE_NUMBER>354 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - merged payload: {"url":"/items/328086?commentId=328101","itemId":328101,"amount":7}
78802 [sw:push]<PHONE_NUMBER>354 2e9b5457-02d1-4db1-a07c-a35ebf6e473a - calculated title: you have 7 new replies

Or in other words: The data of the last notification has amount set to the amount of existing notifications that were seen when this notification was created (this works well on Android due to proper support of tag and close). However, since close doesn't work, the previous notifications still exist and we just add them again in our reducer for merging notifications:
// this should reflect the amount of notifications that were already merged before
log(`[sw:push] ${Date.now()} ${nid} - notifications: ${JSON.stringify(currentNotifications.map(({ data }) => JSON.stringify(data)))}`)
const initialAmount = currentNotifications[0].data?.amount || 1
log(`[sw:push] ${Date.now()} ${nid} - initial amount: ${initialAmount}`)
const mergedPayload = currentNotifications.reduce((acc, { data }) => {
  let newAmount, newSats
  if (AMOUNT_TAGS.includes(compareTag)) {
    newAmount = acc.amount + 1
  }
  if (SUM_SATS_TAGS.includes(compareTag)) {
    newSats = acc.sats + data.sats
  }
  const newPayload = { ...data, amount: newAmount, sats: newSats }
  return newPayload
}, { ...incomingData, amount: initialAmount })

I think that we show the wrong amount of replies is fixable on our side by adding more iOS-specific logic. Will do this tomorrow.

replying and closing notifications manually (swiping them away)
78865 [sw:push]<PHONE_NUMBER>022 aedf1d79-2945-4db5-9936-ecaa8531b9a1 - found 0 REPLY notifications after manual tag filter
78866 [sw:push]<PHONE_NUMBER>931 aedf1d79-2945-4db5-9936-ecaa8531b9a1 - received notification with tag REPLY
78867 [sw:push]<PHONE_NUMBER>020 aedf1d79-2945-4db5-9936-ecaa8531b9a1 - found 0 REPLY notifications
78868 [sw:push]<PHONE_NUMBER>020 aedf1d79-2945-4db5-9936-ecaa8531b9a1 - built-in tag filter: []
78869 [sw:push]<PHONE_NUMBER>022 aedf1d79-2945-4db5-9936-ecaa8531b9a1 - manual tag filter: []
78870 [sw:push]<PHONE_NUMBER>025 aedf1d79-2945-4db5-9936-ecaa8531b9a1 - show notification with title "@036d6006e2 replied to you"
78871 [sw:push]<PHONE_NUMBER>024 aedf1d79-2945-4db5-9936-ecaa8531b9a1 - no existing REPLY notifications found

78872 [sw:push]<PHONE_NUMBER>947 d9b05a59-7f0e-454b-be62-e5faeb141a64 - received notification with tag REPLY
78873 [sw:push]<PHONE_NUMBER>955 d9b05a59-7f0e-454b-be62-e5faeb141a64 - found 1 REPLY notifications
78874 [sw:push]<PHONE_NUMBER>955 d9b05a59-7f0e-454b-be62-e5faeb141a64 - built-in tag filter: ["REPLY"]
78875 [sw:push]<PHONE_NUMBER>958 d9b05a59-7f0e-454b-be62-e5faeb141a64 - incoming payload.options.data: {"url":"/items/328086?commentId=328117","itemId":328117}
78876 [sw:push]<PHONE_NUMBER>956 d9b05a59-7f0e-454b-be62-e5faeb141a64 - found 1 REPLY notifications after manual tag filter
78877 [sw:push]<PHONE_NUMBER>958 d9b05a59-7f0e-454b-be62-e5faeb141a64 - using REPLY for control flow
78878 [sw:push]<PHONE_NUMBER>956 d9b05a59-7f0e-454b-be62-e5faeb141a64 - manual tag filter: ["REPLY"]
78879 [sw:push]<PHONE_NUMBER>958 d9b05a59-7f0e-454b-be62-e5faeb141a64 - notifications: ["{\"url\":\"/items/328086?commentId=328116\",\"itemId\":328116}"]
78880 [sw:push]<PHONE_NUMBER>958 d9b05a59-7f0e-454b-be62-e5faeb141a64 - initial amount: 1
78881 [sw:push]<PHONE_NUMBER>959 d9b05a59-7f0e-454b-be62-e5faeb141a64 - merged payload: {"url":"/items/328086?commentId=328116","itemId":328116,"amount":2}
78882 [sw:push]<PHONE_NUMBER>959 d9b05a59-7f0e-454b-be62-e5faeb141a64 - calculated title: you have 2 new replies
78883 [sw:push]<PHONE_NUMBER>959 d9b05a59-7f0e-454b-be62-e5faeb141a64 - closing existing notifications
78884 [sw:push]<PHONE_NUMBER>965 d9b05a59-7f0e-454b-be62-e5faeb141a64 - show notification with title "you have 2 new replies"

78885 [sw:push]<PHONE_NUMBER>146 967f421d-be9c-4440-9da0-12d67dc775f2 - received notification with tag REPLY
78889 [sw:push]<PHONE_NUMBER>152 967f421d-be9c-4440-9da0-12d67dc775f2 - built-in tag filter: ["REPLY","REPLY"]
78888 [sw:push]<PHONE_NUMBER>152 967f421d-be9c-4440-9da0-12d67dc775f2 - manual tag filter: ["REPLY","REPLY"]
78886 [sw:push]<PHONE_NUMBER>152 967f421d-be9c-4440-9da0-12d67dc775f2 - found 2 REPLY notifications after manual tag filter
78887 [sw:push]<PHONE_NUMBER>153 967f421d-be9c-4440-9da0-12d67dc775f2 - more than one notification with tag REPLY found
78890 [sw:push]<PHONE_NUMBER>152 967f421d-be9c-4440-9da0-12d67dc775f2 - found 2 REPLY notifications
78891 [sw:push]<PHONE_NUMBER>155 967f421d-be9c-4440-9da0-12d67dc775f2 - incoming payload.options.data: {"url":"/items/328086?commentId=328118","itemId":328118}
78893 [sw:push]<PHONE_NUMBER>153 967f421d-be9c-4440-9da0-12d67dc775f2 - skip bail -- merging notifications with tag REPLY manually
78892 [sw:push]<PHONE_NUMBER>155 967f421d-be9c-4440-9da0-12d67dc775f2 - using REPLY for control flow
78894 [sw:push]<PHONE_NUMBER>155 967f421d-be9c-4440-9da0-12d67dc775f2 - notifications: ["{\"url\":\"/items/328086?commentId=328116\",\"itemId\":328116,\"amount\":2}","{\"url\":\"/items/328086?commentId=328116\",\"itemId\":328116}"]
78895 [sw:push]<PHONE_NUMBER>156 967f421d-be9c-4440-9da0-12d67dc775f2 - merged payload: {"url":"/items/328086?commentId=328116","itemId":328116,"amount":4}
78896 [sw:push]<PHONE_NUMBER>155 967f421d-be9c-4440-9da0-12d67dc775f2 - initial amount: 2
78897 [sw:push]<PHONE_NUMBER>156 967f421d-be9c-4440-9da0-12d67dc775f2 - calculated title: you have 4 new replies

// closing second notification

78898 [sw:push]<PHONE_NUMBER>156 967f421d-be9c-4440-9da0-12d67dc775f2 - closing existing notifications
78899 [sw:push]<PHONE_NUMBER>161 967f421d-be9c-4440-9da0-12d67dc775f2 - show notification with title "you have 4 new replies"
78900 [sw:push]<PHONE_NUMBER>007 7a729557-6bba-4b28-abfb-489cbe211f58 - received notification with tag REPLY
78901 [sw:push]<PHONE_NUMBER>078 7a729557-6bba-4b28-abfb-489cbe211f58 - found 2 REPLY notifications
78902 [sw:push]<PHONE_NUMBER>081 7a729557-6bba-4b28-abfb-489cbe211f58 - more than one notification with tag REPLY found
78903 [sw:push]<PHONE_NUMBER>081 7a729557-6bba-4b28-abfb-489cbe211f58 - skip bail -- merging notifications with tag REPLY manually
78904 [sw:push]<PHONE_NUMBER>079 7a729557-6bba-4b28-abfb-489cbe211f58 - found 2 REPLY notifications after manual tag filter
78905 [sw:push]<PHONE_NUMBER>078 7a729557-6bba-4b28-abfb-489cbe211f58 - built-in tag filter: ["REPLY","REPLY"]
78906 [sw:push]<PHONE_NUMBER>079 7a729557-6bba-4b28-abfb-489cbe211f58 - manual tag filter: ["REPLY","REPLY"]
78907 [sw:push]<PHONE_NUMBER>083 7a729557-6bba-4b28-abfb-489cbe211f58 - incoming payload.options.data: {"url":"/items/328086?commentId=328119","itemId":328119}
78908 [sw:push]<PHONE_NUMBER>083 7a729557-6bba-4b28-abfb-489cbe211f58 - using REPLY for control flow
78909 [sw:push]<PHONE_NUMBER>083 7a729557-6bba-4b28-abfb-489cbe211f58 - notifications: ["{\"url\":\"/items/328086?commentId=328116\",\"itemId\":328116,\"amount\":4}","{\"url\":\"/items/328086?commentId=328116\",\"itemId\":328116}"]
78910 [sw:push]<PHONE_NUMBER>083 7a729557-6bba-4b28-abfb-489cbe211f58 - initial amount: 4
78911 [sw:push]<PHONE_NUMBER>086 7a729557-6bba-4b28-abfb-489cbe211f58 - merged payload: {"url":"/items/328086?commentId=328116","itemId":328116,"amount":6}
78912 [sw:push]<PHONE_NUMBER>086 7a729557-6bba-4b28-abfb-489cbe211f58 - calculated title: you have 6 new replies
78913 [sw:push]<PHONE_NUMBER>086 7a729557-6bba-4b28-abfb-489cbe211f58 - closing existing notifications
78914 [sw:push]<PHONE_NUMBER>087 7a729557-6bba-4b28-abfb-489cbe211f58 - show notification with title "you have 6 new replies"

Conclusion:

closing notification manually is correctly registered: getNotifications({ tag }) does no longer show this notification. This is the behavior we would like to also see when we use the Notification API (Notification.close()).

Just leaving a note that this seems to have been happening a lot more than usual lately (past few days).


can download single songs but i can't download playlists

got this error:
typed this into my terminal, "spotdl https://open.spotify.com/playlist/2CWn8zek9TYbiwtEsWBQFB"
and this is what happened:
Fetching Playlist...
Traceback (most recent call last):
File "/Users/macbookpro2016/opt/anaconda3/bin/spotdl", line 8, in
sys.exit(console_entry_point())
File "/Users/macbookpro2016/opt/anaconda3/lib/python3.8/site-packages/spotdl/main.py", line 114, in console_entry_point
songObjList = get_playlist_tracks(request)
File "/Users/macbookpro2016/opt/anaconda3/lib/python3.8/site-packages/spotdl/search/utils.py", line 83, in get_playlist_tracks
'https://open.spotify.com/track/' + songEntry['track']['id'])
TypeError: can only concatenate str (not "NoneType") to str
0%|
please assist!
Yan

Are you using vanilla python or anaconda?

Hi there Mikhail,
Actually i'm not sure, I seem to have both. i would actually like to reinstall all my python/ anaconda things. how do i do that?
Appreciate your help!
Yan

I fiddled around with my python and anaconda, uninstalled and reinstalled python. now i faced this error:
$ pip install https://github.com/spotDL/spotify-downloader/archive/next-rel-dev.zip
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Defaulting to user installation because normal site-packages is not writeable
Collecting https://github.com/spotDL/spotify-downloader/archive/next-rel-dev.zip
Using cached https://github.com/spotDL/spotify-downloader/archive/next-rel-dev.zip
ERROR: Package 'spotdl' requires a different Python: 2.7.16 not in '>=3.6'
Any tips?

Newest update:
I tried pip install spotdl, and then I added the path to the search path below:
macbookpro2016. ~/Downloads/Mum (13:41) ❌
$ export PATH=$PATH:/Users/macbookpro2016/Library/Python/2.7/bin
macbookpro2016. ~/Downloads/Mum (13:43)
$ which spotdl
/Users/macbookpro2016/Library/Python/2.7/bin/spotdl
after which i got this error:
macbookpro2016. ~/Downloads/Mum (13:43)
$ spotdl https://open.spotify.com/track/4IdiGMOzEYXOh2897XOV8i
Traceback (most recent call last):
File "/Users/macbookpro2016/Library/Python/2.7/bin/spotdl", line 5, in
from spotdl import main
File "/Users/macbookpro2016/Library/Python/2.7/lib/python/site-packages/spotdl.py", line 5, in
from core import metadata
File "/Users/macbookpro2016/Library/Python/2.7/lib/python/site-packages/core/metadata.py", line 7, in
import urllib.request
ImportError: No module named request
feels like i'm getting somewhere?

Well, i just realized the problem. I should have noticed earlier. spotdl v2 and v3 run only on python3.5+


Refactor Map Operator Factory

The map operator factory scheme is not scalable as more operators are added - it becomes very hard to maintain. In addition, it does not follow the open-closed principle as users can not inject their own implementations into the factory. We need to assess our construction scheme and use an abstract builder/factory pattern.

also invalid parameter list will not throw but will return a null ptr. is that really what we want?

Enhancement for version 2 so will not fix.


we need to end copiedArgs

we need to end copiedArgs. In fact, why do we need copyargs?

@swift-ci please test


Adopt AIND-Schemas for standardization of data

rig
subject
session
task
water calibration
laser calibration

rig jsons should be in repo
SLIMS has database of instruments

#303 partially overlapping

https://github.com/AllenNeuralDynamics/aind-behavior-blog/issues/68


"complete suicide" is inconsiderate

Thanks, @rugk!

Strange merging technique, but okay.

I am unsure what you mean?

You seem to have merged the commit directly and not this PR so this PR is shown as closed (unmerged) on GitHub.

Yes, because I had to fix some stuff. For example, the tests started failing, and the built patterns were not included in your PR.
 I did not know there were also tests for this...

No problem


buitin filters round and int do not work

I'm using Jinja2==2.7.1
In [2]: t = jinja2.Template("{{ x/2 | round }}") #does not work?

In [3]: t
Out[3]: <Template memory:161e910>

In [4]: t.render(x=7)
Out[4]: u'3.5'

In [5]: t2 = jinja2.Template("{{ x | float }}") # float works

In [6]: t2.render(x=3)
Out[6]: u'3.0'

In [12]: t3 = Template("{{ x / 2 | int }}") # does not work

In [13]: t3.render(x=7)
Out[13]: u'3.5'

Just ran into the exact problem!
Thanks @hvnsweeting and @Naddiseo

Same issue, same solution. In my case, it was in Home Assistant where I ended up with:
{{ ((states.input_number.guestroom_setpoint_shift.state | float) * 2) | int }}
Thanks!


Response headers?

Hey,
I can't retrieve all the response headers, it gives me only:
headers: {
  cache-control: "max-age=0, private, must-revalidate"
  content-type: "application/json; charset=utf-8"
}

My bad, you need to set Access-Control-Expose-Headers if you use CORS.

@iam4x Did just adding the Access-Control-Expose-Header header solve this for you?
From this response headers I only get the Content-Type from response.headers (with axios 0.7.0):
Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,X-XS-Test
Access-Control-Allow-Methods: GET,PUT,PATCH,POST,DELETE
Access-Control-Allow-Origin: *
Access-Control-Expose-Header: PROTOCOL,X-Powered-By,Etag
Connection: keep-alive
Content-Length: 3
Content-Type: text/plain; charset=utf-8
Date: Mon, 05 Oct 2015 07:52:51 GMT
Etag: W/"3-dSgDWpPuac7bHb3bLwv8yA"
X-Powered-By: Express

Gives me just this:
{"content-type":"text/plain; charset=utf-8"}

Thanks for your help!

I get the same issue. Using axios 0.7.0 I only get the Content-Type header in the response.headers object.

In my case it was just a simple typo, I was missing the "s" at the end of Access-Control-Expose-Headers :disappointed:

Yes in my case, just put in .htaccess file:

Header set Access-Control-Expose-Headers "X-Pagination-Current-Page, X-Pagination-Total-Count"

This for my custom pagination in yii2.


Remove limitation on the location of the images dir, #1172

Rather than allowing only subdirs of /home as default image dir, this change only disallows the file system root itself.
A user may want to use mountpoints in /mnt, for instance, as the images dir.

So it is of course risky, since piscsi has root access, to configure it to use /etc or /bin etc. as images dir. You could potentially delete critical system files with the web ui. How much do we want to trust users here?

@rdmark I disagree with this change because of the root access issues you mentioned. I intentially did not allow any folder/path to be the default folder.
If a user wants to use a directory not in /home she/he can simply create a softlink in order for this folder to appear in the /home folder, e.g.:
ln -s /some_folder /home/some_folder

This is less dangerous and does not require any change in the PiSCSI sources. Softlinks for images folders are explicitly supported.

@uweseimet I see your point, however using a subdir of /home seems to me arbitrarily tied to a particular user on the system, whereas piscsi could potentially be a shared concern by multiple users on the system. I did a bit of reading  on people's thoughts on Linux file system practices, and the idea that a service that is strictly a local concern on a system is encouraged to use arbitrary new mountpoints in the root. I would argue that something like /piscsi/images would be a neat and safe default location. What do you think?

@rdmark Some time ago I pointed out that in my opinion easyinstall is too intrusive because it
installs all kinds of packages without user control. It effectively reconfigures/takes control over the Pi on a level that IMHO is not justified and may cause issues when you want to install/run other software than just piscsi. The response I received was that a user would not want to run anything else on a Pi with piscsi installed anyway. I also mentioned that not using /opt as the default package installation path is against standards and created a related ticket. There was no interest in adressing that.
Now you are talking about the Pi as a multi-user system, which implicitly means that multiple users independently install/run various pieces of software on it and standards become even more important. I see a contradiction here.
Anyway, I don't see that defining the Pi as a multi-user system means that piscsi must support any path for the images folder. On the contrary: On a multi-user system topics related to security/data integrity are particularly important, so it would even be more of a problem if you lose data by giving a process running as root unrestricted access to the whole filesystem. If something goes wrong (during an image file operation) not only your own data but also those of others may be gone. Therefore the multi-user scenario is rather a reason of not allowing any path for the images folder, isn't it?
Regarding new mount points in root: This sounds like a bad practice to me. I do not know any system that encourages this. Or maybe I misunderstand your point?

Using a fixed /var/lib/piscsi location sounds like a reasonable solution to me. However as you alluded to, I think such a change should be made as part of a proper packaging scheme (deb or otherwise) rather than a narrow install location fix. Some discussion about deb packaging is in https://github.com/PiSCSI/piscsi/issues/740 so I will make a note over there.
Discarding this PR


Device group attribute registery is wrong

It needs to be changed to tm:cm:device-group:device-groupstate

It needs to be changed to tm:cm:device-group:device-groupstate

@szakeri is it like that on 11.x and 12.x?

That's a good question. I am testing against 11.6. @pjbreaux do you see this issue on 12 as well?

Yes:

from f5.bigip import ManagementRoot as mr
b = mr('', '', '')
b.tm.cm.device_groups.get_collection()
Traceback (most recent call last):
File "", line 1, in
File "/root/common/f5/bigip/resource.py", line 550, in get_collection
raise UnregisteredKind(error_message)
f5.bigip.resource.UnregisteredKind: u'tm:cm:device-group:device-groupstate' is not registered!

k. just wanted to make sure it wasn't one of those commands that received a name change during 11->12

Setting this for the next release.

@szakeri what's the status of this issue?

This is fixed, see #579 .


how pipenv know Pipfile.lock is outdated

$ pipenv install
-- creates a virtualenv and created Pipfile and Pipfile.lock
$ pipenv install django
Q: It places the django in the Pipfile and also regenerated the Pipfile.lock with all the dependecies.
So whenever i install any package using pipenv, Pipfile.lock is also updated.
HOw does the hash comparision works so that form the hash of Pipfile it comes to know whether Pipfile.lock is outdated or not.

@sant527 The hash is calculated from the Pipfile content, and stored in Pipfile.lock. When the Pipfile content is changed, Pipenv will calculate the hash again and compare it with that stored in Pipfile.lock to see if it needs to be updated.

@sant527 Thanks for your curiosity about the implementing details, you can always refer to the source code to find the answer.

is there a way using pipenv command to know whether my Pipenv.lock is outdated or not

pipenv update --outdated
why not go through all commands and find out what they do?
you should find answer yourself that are included in docs

Hopefully this is cleared up. Closing this out, thanks for submitting the issue

pipenv update --outdated
why not go through all commands and find out what they do?
you should find answer yourself that are included in docs

Sorry for being 3 years late to the party, but I have to point out that above reply doesn't actually answer the question.
what the OP meant, I think, is a way to tell, in advance, whether .lock is older than the Pipfile itself (for example, to fail the build as this would silently make the build non-determinstic).
The closest solution I found is to use pipenv install --deploy, this will fail if the "Pipfile.lock out of date" situation is encountered.


Update to 6.5.2

Elasticsearch 6.5.2 Release Notes
Logstash 6.5.2 Release Notes
Kibana 6.5.2 Release Notes

@mw-jko done, sorry for the delay.


Fix pyspark tests

Description

Updates to pyspark notebook

set seed value for ALS
update displayed values for new seed

Updates to smoke test for pyspark notebook

using updated values from above

Updates to integration test for pyspark notebook

dropping 10m, and 20m tests, updating values based on seed used above

Motivation and Context
Fix failing smoke / integration tests and nightly pyspark staging builds
This removes variability introduced during ALS training to make tests work better
Additionally it reduces the size of tests to ensure integration tests can complete in a timely fashion
Addresses #369
Releated Issues
Should replace #370 and #371
Checklist:

[x] My code follows the code style of this project, as detailed in our contribution guidelines.
[x] I have added tests.
[x] I have updated the documentation accordingly.

I checked the history of the integration test and TOL = 0.5 goes back to November 1. So, there have been many changes in the notebook since, and the integration tests were meaningless. since you are at it, it is a good opportunity to change also the other test files that have TOL = 0.5. I found the following:
integration/test_notebooks_python.py
smoke/test_notebooks_pyspark.py (it is 0.1)
smoke/test_notebooks_python.py

@anargyri probably we do another PR so that this one can resolve the immediate problem. What do you think?

It looks like @gramhagen changed all the occurrences of TOL = 0.5. The unit tests passed; Scott could you check too that smoke and integration tests work (and if not to change the assert statements as appropriate)?

@anargyri smoke and integration tests pass for normal and spark setups, so I think we're good to go
https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/results?buildId=2053258
https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/results?buildId=2053344


There is no access fn for neighbor distances

Added in commit ad82742e6df81624e9fcf54cb7caa49b730b568f . Access function is atom.neighbor_distance


Develope broken  by camera fork

What
  error • 'SmoothCameraController.startImageStream' ('Future<void> Function(dynamic Function(dynamic))') isn't a valid override of 'CameraController.startImageStream' ('Future<void> Function(dynamic Function(dynamic), {bool persistToFile})') • packages/smooth_app/lib/pages/scan/camera_controller.dart:94:16 • invalid_override
  error • The argument type 'void Function(CameraImage)' can't be assigned to the parameter type 'dynamic Function(dynamic)' • packages/smooth_app/lib/pages/scan/ml_kit_scan_page.dart:304:22 • argument_type_not_assignable

Yes I know, but the easiest way would be to review this PR in priority.

#2953 or which one

Yes it was #2953


Apply to AT&T Aspire Accelerator

Website: http://about.att.com/csr/aspireaccelerator
Haven't had a chance to look at the application yet, but this was sent my way and I think we should apply if the application isn't too crazy – lots of funding and I'd imagine connections to potential industry sponsors.

Application has been submitted!


Truncated PR description

Describe the bug
I'm using Generate release notes (XPlat).
The PR description is being truncated at around ~385 chars. I tested with 2 long descriptions and they were both trimmed, one at 377 and another at 393. I'm guessing the difference is related to the fact I'm using markdown formatting in the description. I couldn't find an option on the task for that, that's why I'm opening this issue.
To Reproduce
Steps to reproduce the behavior:

Create a Generate Release Notes (XPlat) task
Use the PR description

{{#forEach pullRequests}}
{{#if isFirst}}
### Pull Requests
{{/if}}
**PR {{this.pullRequestId}}** {{this.title}}
**By**: {{this.createdBy.displayName}}
**Description:**

{{{this.description}}}
{{/forEach}}

Create a "long" PR with over 400 characters
See generated file for truncated description

Expected behavior
The full description displayed.
What is the issue see:

Azure DevOps Services (VSTS)

Please note I'm using {{{ to sanitize.

Sorry for not getting back sooner, I have been on annual leave. I don't have an immediate answer,  I will need to investigate this issue and get back to you

@rfennell thank you, I appreciate it :)

I can't repo this issue, I used the template
# Associated Pull Requests ({{pullRequests.length}})
{{#forEach pullRequests}}
{{#if isFirst}}### Associated Pull Requests (only shown if  PR) {{/if}}
*  **PR {{this.id}}**  {{this.title}} - {{this.description}}
{{/forEach}}

and got approx 500 characters back

when the PR was set as follows

Not sure were to go now with this one. Best I can come up with is that there is a terminating character in your description.
You could look at one of the dump options (as shown below, or one of the newer dump parameters)  to see of the data is present (and just not rendering). Maybe try
{{json this.pullRequests}}

This will show all the object array in the output

@rfennell thank you, I'll do the dump test, but do you know which terminating character that would be?

So, I've just tested with and without sanitize and it breaks at the same place. Evaluating the dump showed me the description field is being returned truncated, but there's another field completionOptions.mergeCommitMessage that has the full text with some extras. It has the following structure:
Merged PR 13607: [PR title]\n\n[Full description]\n\nRelated work items: #107981
I thought it could be some limitation on my rules, but I checked the completed PRs and they all have the full description there.
Any ideas? :/

Could you send over the output of the PR dump, might be easier to understand

Here it is https://codebeautify.org/jsonviewer/cb3ca9c1

The completionOptions.mergeCommitMessage are there because either

your PR is merged and that was the commit message
you have set automated completion and that is to be the message

I tried dropping your description text into my test PR, still seems to work.
What do you get if you look at the PR via the API, something like
https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/pullrequests/<id>
e.g.
`https://dev.azure.com/richardfennell/GitHub/_apis/git/repositories/VSTSBuildTaskValidation/pullrequests/28'
Does this show the field truncated too?

No, if I use the URL you provided, it returns the complete description field.

Very strange, that would suggest it is the Node SDK or my code doing the trim. Strange thing is I have now done an and to end using test with a release pipeline and your PR descriptin and I get no truncation.
I will keep digging

Thank you, I agree, that makes perfect sense. And here it is the result from the API request, so you can safely focus on that: https://codebeautify.org/jsonviewer/cb19c235
Thank you, again :)

Sorry I can't repeat the problem you are having and can't see any code that could be causing the problem.
All the task does is get all the PRs in the project, it then selects the PRs that are linked to any commits associated with build/release. At no point is there any processing done to the PR objects returned from API, so I am just returning the description as returned by the API
As I cannot get a repo, all I can suggest is you do some diagnostics on your local system. You might find the command line tester could help. This will allow you to rerun the build/release that is showing the issue.
I will mark this as a known issue for now

@rfennell thanks Richard, I'm thinking here... which URL do you use to get that data? The one you shared or you get one that is returned with some metadata request on the project (I don't know).
The reason I ask is because when I tried the URL you shared (new pattern dev.azure...) I got the full description, but when I run the task I get it truncated. So I'm thinking that if you use some sort of metadata to get the PR URL it could be that I'm using the old pattern on this project (xxx.visualstudio.com) and that the endpoint behaves differently (like truncating). What are your thoughts on that? Because honestly, other than that and the task having a bug (which you already discarded) I'm out of ideas as well.

I use the Azure DevOps Node API which is just a one for one wrapper for REST API calls.
As I mentioned I first get all the completed PRs for the project
export async function getPullRequests(gitApi: GitApi, projectName: string): Promise<GitPullRequest[]> {
    return new Promise<GitPullRequest[]>(async (resolve, reject) => {
        let prList: GitPullRequest[];
        try {
            var filter: GitPullRequestSearchCriteria = {
                creatorId: "",
                includeLinks: true,
                repositoryId: "",
                reviewerId: "",
                sourceRefName: "",
                sourceRepositoryId: "",
                status: PullRequestStatus.Completed,
                targetRefName: ""
            };
            prList = await gitApi.getPullRequestsByProject( projectName, filter);
            resolve(prList);
        } catch (err) {
            reject(err);
        }
    });
}

So this should get the PR with a full description as this is just 'https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/pullrequests'
I then loop array this list to match merge commits, I never touch the objects, I push select the ones that match and add them to an array
  var allPullRequests: GitPullRequest[] = await getPullRequests(gitApi, prProjectFilter);
                if (allPullRequests && (allPullRequests.length > 0)) {
                    agentApi.logInfo(`Found ${allPullRequests.length} Azure DevOps for PRs`);
                    globalCommits.forEach(commit => {
                        if (commit.type === "TfsGit") {
                            agentApi.logInfo(`Checking for PRs associated with the commit ${commit.id}`);

                            allPullRequests.forEach(pr => {
                                if (pr.lastMergeCommit) {
                                    if (pr.lastMergeCommit.commitId === commit.id) {
                                        agentApi.logInfo(`- PR ${pr.pullRequestId} matches the commit ${commit.id}`);
                                        globalPullRequests.push(pr);
                                    }
                                } else {
                                    console.log(`- PR ${pr.pullRequestId} does not have a lastMergeCommit`);
                                }
                            });

                        } else {
                            agentApi.logDebug(`Cannot check for associated PR as the commit ${commit.id} is not in an Azure DevOps repo`);
                        }
                    });
                } else {
                    agentApi.logDebug(`No completed Azure DevOps PRs found`);
                }

The bit I don't understand is why I cannot repo the problem. I seem to be able to get any length of description.
As I said all I can suggest is using my local test harness (that I use to develop templates and Handlebars extensions and adding extra code to dump out the object as each point to see what is occurring or monitor the network with fiddler, but I can't do that as I cannot repeat the issue on my system

I understand and appreciate your help.
I'll continue to use completionOptions until I can test it properly.
Thank you very much!

Do you want to leave this issue open, or close it and re-open if more information comes to light?

Well, for maintenance purposes, I think it might be better to close it. If anything else changes, I'll reopen. Thanks.


Added reference to branch creation

Status
Ready
Description
Added reference to branch creation

@fvigo Until we'll make the changes we talked about can you review these small editions?


Tapeline 2.0.3 error

When continuing a saved game the following appears with Tapeline 2.0.3:
Error while running event Tapeline::on_configuration_changed
No character.
stack callback:
[C]: in function '__index'
Tapeline/scripts/migration.lua:20: in function 'func'
flib/migration.lua:65: in function 'run'
flib/migration.lua:87: in function 'on_config_changed'
Tapeline/control.lua:82: in function <Tapeline/control.lua:81>
I always run in God mode without a character.
The save file size is 76 MB. Even smaller ones that I have are more then 40 MB. If you really need a save file I have to create a new one which needs some time...
Trying to upload the log file results in error: Something went really wrong, and we can't process that file. Try again
Therefore here it is inline:
edit by maintainer: removed log file since I can reproduce this easily

Same for me. Happens when loading saves with god mode on.

Oops, fixed for the next release.


Migration of logging handlers to log4net (#69)

DONE:

Integrate log4net into WinSW, migrate wrapper.log engine

TODOs:

Migrate EventLogs to log4net
Delegate exceptions handling to log4net
Allows setting up log levels from ServiceDescriptor
(?) Migrate STDERR/STDOUT logs to log4net
(?) Support XML configurations of log4net loggers

Signed-off-by: Oleg Nenashev<EMAIL_ADDRESS>

Merging the basic logging to allow the usage of ServiceWrapper log in other dependencies.

@oleg-nenashev We would like to enable sending all logs to Windows Event Viewer. Would Adding support for the above features Migrate STDERR/STDOUT logs to log4net and Support XML configurations of log4net loggers enable us to do that?

@aminjam I doubt Support XML configurations of log4net loggers will be implemented in such way. #158 just addresses the minimal config.
log4net already allows configuring everything in exe.config file, hence you will be able to get this feature once this issue is integrated (likely via the logging=log4net mode).


Update how to run tests

The links about running unit tests and static analysis in the CONTRIBUTING.md are broken. This PR is to fix it.
Also added the how to run CATs.

[x] I have reviewed the contributing guide

[x] I have viewed, signed, and submitted the Contributor License Agreement

[x] I have made this pull request to the master branch

[ ] I have run all the unit tests using bundle exec rake

[ ] I have run CF Acceptance Tests

No need to run unit tests and CATs since the code changes are only for docs.

Thanks!


Typo while creating an appservice plan

[Enter feedback here]
https://docs.microsoft.com/en-us/azure/frontdoor/quickstart-create-front-door-cli
az appservice plan create
--name myAppServicePlanSouthCentralUS
-resource-groupg myRGFDSouthCentral
should be --resource-group

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9790f3ea-1c30-b926-46c3-f04939ca2170
Version Independent ID: a94ce6cf-7c60-aca4-eabf-145033c7c9aa
Content: Quickstart: Set up high availability with Azure Front Door - Azure CLI
Content Source: articles/frontdoor/quickstart-create-front-door-cli.md
Service: frontdoor
GitHub Login: @duongau
Microsoft Alias: duau

@kumarallamraju, appreciate your feedback. The doc has been edited accordingly and it would take some time for the changes to get reflect over the doc.
we are closing this issue for now. If there are further questions regarding this matter, please reply and we will gladly continue the discussion.


lnd sends announcement_signatures before reestablish_channel

Background
You're supposed to send reestablish first, and also wait for the reply.
https://github.com/ElementsProject/lightning/issues/2559
Your environment
The source was Pierre Rochard's node, which he tells me is 0.5.2-99-beta commit=queue/v1.0.1-109-g9f6a1403e9773ba97ec0e9fdee3b654789970d3a
Steps to reproduce
Take your c-lightning node (minus above fix) up and down several times while lnd connects with a relatively fresh channel.
Expected behaviour
Delay other channel msgs until REESTABLISH sent and received.
Actual behaviour
Rusty writes nasty code in c-lightning to defer unexpected packets.  We had an earlier issue with funding_locked, but I had special cased that.  Now I just defer all weird stuff.

Hmm, BOLT 2 says that BOLT 7 messages are not required to wait for ChannelReestablish:

Note that messages described in BOLT #7 are independent of particular channels; their transmission requirements are covered there, and besides being transmitted after init (as all messages are), they are independent of requirements here.

and BOLT 7 does not include anything about ChannelReestablish in the retransmission requirements for AnnouncementSignatures.
I assume then, that this the issue in regards to delaying BOLT 7 messages only if:

They pertain to a channel with the current peer, and
We have not sent ChannelReestablish for that channel?

If I understand correctly, what is the benefit of being able to send info about everyone else's channels immediately, but withhold relevant gossip info from your channel partner until after ChannelReestablish?

Closing as a duplicate of #2555.


disable spam protection for custom translator

can I config the delay of spam protection?
I write a simple server to do zh-CN to zh-TW , since it's a directory mapping(sort of) so it's very fast ,
the full http request to local server is below 10ms and can processed multiple thread,
so I really hope it can spam my own local server.

try to write dll version , still bit slow