Add team receiving page

https://github.com/gratipay/gratipay.com/issues/3705, https://github.com/gratipay/gratipay.com/issues/3779

Screenshots:

Payment distribution:

I'm made the team pages visible to only admins and the team's owner. My reasoning here is that the values we are exposing aren't polished enough to show to the general public. For example, the below graph would make sense to the owner of a team who is aware of charging in arrears - but to others, it'd just look like the number of payments reduced drastically over a week.

Here are a few pages generated from our DB (team names hidden):

The estimated next week's payment for Gratipay looks out of line though...

Probably a bug somewhere, I'm going to write a few simple test cases to see if something pops up

The values of unfunded dues are pretty high for most teams, that's due to https://github.com/gratipay/gratipay.com/issues/3818

Nice!

Rebased on master after https://github.com/gratipay/gratipay.com/pull/3787

The values of unfunded dues are pretty high for most teams, that's due to #3818

Fixed that in #3876, lets look if these values look better after those changes...

Before:

After:

The estimated payment is definitely closer, but still pretty high. The last bar in "Dollars received per week" is ~$275.

Everything looks right from my side, I guess we'll find out how good this 'estimated' value is tomorrow :)

I'm going to run a script to generate the estimated values for all teams - And I'll write another one to compare them once we run Payday.

@rohitpaulk My main concern after once read-through of this PR is that hiding the receiving amount sets a precedent. If we would rather have team receiving be public then we should make it that way from the outset.

If we would rather have team receiving be public then we should make it that way from the outset.

There's a reason behind this, mentioned at https://github.com/gratipay/gratipay.com/pull/3873#issuecomment-164165342

I've made the team pages visible to only admins and the team's owner. My reasoning here is that the values we are exposing aren't polished enough to show to the general public. For example, the below graph would make sense to the owner of a team who is aware of charging in arrears - but to others, it'd just look like the number of payments reduced drastically over a week.

Once I've worked on #3851, I'll work on https://github.com/gratipay/gratipay.com/issues/3705#issuecomment-164167705 and then this will be ready to be shown to the public.
(said the same over at https://github.com/gratipay/gratipay.com/issues/3705#issuecomment-164168024)

I'm not confident that this is a good enough user experience for the public - but it certainly is for a team owner. And we know from our own experience that huge PRs don't make it till production, hence the attempt to make incremental improvements :)

I'm not confident that this is a good enough user experience for the public - but it certainly is for a team owner.

Right, but it doesn't seem right to roll this out as a private page and then automatically make it a public page at some point in the future. If we roll it out as private that'll become the default by default, and we'll have to add a switch to opt in to sharing it publicly once that's ready. I don't think we want that, because openness should be the default for Teams on Gratipay. I'm disinclined to even offer private receiving as an option, actually.

And we know from our own experience that huge PRs don't make it to production,

Well, that's true. :disappointed:

hence the attempt to make incremental improvements :)

In the past, of course, we've simply "released early and often" even if the UX is half-baked (at best). If we want to start raising the bar on what we're willing to release, then one pattern we could try is to make PRs against PRs—PR inception! <:-) That keeps our batch size small for the primary review queue so we can process it efficiently, while giving us a chance to aggregate work units for release based on user-centric concerns. See https://github.com/saxifrage/cityasacampus/pull/364 for an example.
The three options I'm seeing are:

Deploy this as-is. Team receiving pages are private by default.
Make Team receiving pages public, and then deploy. They're confusing for the general public.
Block this on #3851 and https://github.com/gratipay/gratipay.com/issues/3705#issuecomment-164167705, using PRs against this PR as necessary.

How do you think we should proceed, @rohitpaulk?

I'm against option 3, PRs against PRs don't really solve the core problem, which is PRs going stale. Everything works fine as long as we're moving fast - but if someone decides to create the second PR (on top of one) 3 months later, it's still going to be difficult to rebase and update the base PR. What PRs against PRs does solve, is huge PRs that are difficult to review. Since we make our commits oh-so-atomic, I don't think that's a big problem for us :)
Although I think that option 1 is the way to go, both option 1 and 2 are fine with me. Thoughts, @mattbk @webmaven @techtonik @rorepo @kzisme?

I'm against option 3, PRs against PRs don't really solve the core problem, which is PRs going stale. Everything works fine as long as we're moving fast - but if someone decides to create the second PR (on top of one) 3 months later, it's still going to be difficult to rebase and update the base PR.

Agreed.

both option 1 and 2 are fine with me.

I'm fine with option 2 if no-one else objects. To soften the blow we could add a note to the chart like we used to have about "minimum charge upped to $10 in week 8" or whatever. There may still be a note CSS class hanging around for this occasion.
This PR is good work and we should get it out there! :-)

I see option 1 as a "beta" to allow team owners to give feedback on how accurate it seems ("truthiness," not actual math). If it's deployed privately with a statement that it will be made public in the future (when we decide it's out of beta), it should end up being opt-out, rather than opt-in.

it should end up being opt-out, rather than opt-in.

Why opt-anything? Options are confusing.

I see option 1 as a "beta" to allow team owners to give feedback on how accurate it seems ("truthiness," not actual math).

I could maybe see that if we turned it off entirely for a while to rework it based on feedback before releasing publicly.

If we roll it out as private that'll become the default by default, and we'll have to add a switch to opt in to sharing it publicly once that's ready.

@mattbk @rohitpaulk I haven't seen you directly address this. Do you admit this consequence of option 1 yet accept it? Or do you not admit this consequence? Or ... ?

I haven't seen you directly address this

I think this depends on how we present it... As long as we make it clear (by adding a note, as @mattbk suggested) that we intend to make it public in the near future, we won't be deceiving anyone. If we face significant pushback on the same, it's probably a sign that we should reconsider providing anonymous receiving options.

I see option 1 as a "beta" to allow team owners to give feedback on how accurate it seems ("truthiness," not actual math). If it's deployed privately with a statement that it will be made public in the future (when we decide it's out of beta), it should end up being opt-out, rather than opt-in.

Why opt-anything? Options are confusing.

Shouldn't we consider deciding this based on the feedback that we receive from team owners? Rather than forcing this upon them - why not open this to them and see what they have to say?

This is blocked on https://github.com/gratipay/gratipay.com/pull/3876

Rebased on master at @ec2c732d822cb446dd8c43540c5a7c3a9ea8522a. Previous head was @673475357000a7f0bf8e5fd4a6fff58843734a0f

#3980 and #3981 are out, this is no longer blocked.

Rebased on master.

Did a bit of browsing through on a recent backup, the charts look okay to me.

Alright, this is super-rough but we know that. I'm merging in order to keep our velocity up. I've reticketed #3992 for further iteration, still on Finish Team Basics.
!m @rohitpaulk

(P.S. Also this is read-only so it's low-risk.)


Weird dependency

Is this line correct? I feel like you meant to inject the internal github package? Its currently breaking my build.
https://github.com/neelance/graphql-go/blob/master/internal/exec/packer.go#L11

errors is a pretty well known Go package.
https://godoc.org/github.com/pkg/errors

It's currently breaking my build.

What's your error? Perhaps you need to go get it?

@shurcooL yes, the problem was that I needed to go get it but @neelance is using it in the exact same way that he uses his internal errors package. It just feels like go imports choose the wrong package in this case.

It just feels like goimports choose the wrong package in this case.

Ah, that might've been the case then.


PO Product Item Incorrectly Displayed as Pending Despite Full Receipt

Current behavior
When a user receives all of the ordered quantity for a purchase order (PO) product item, the item remains displayed in a pending state in the user interface (UI), despite moving it to completed category.
Expected behavior
Upon receiving all of the ordered quantity for a PO product item, the item should transition to the completed state in the UI. Also 'Receive all' button should be disabled in this case.
Steps to reproduce the issue
Receive a purchase order (PO) product item for the full ordered quantity.
Verify that the received quantity matches the ordered quantity.
Observe that the PO product item remains displayed in the pending state in the UI, despite all items being received.
Environment details

App version: v2.23.1-
Environment: UAT

Additional information
https://jam.dev/c/2d08c1d1-b17b-4c95-a016-a5efda2d85fd

@Adameddy26 you have to explicitly close the items to move the items to the completed section. Please retest this ticket with "receive and close" function and share your findings

@dt2patel Retested the ticket with "receive and close" function and found that item is still in pending state. On refreshing the URL, it is getting marked as completed .

When we close a item in purchase order then the router redirect us to purchase orders page where all the purchase orders are fetched again using solr query.
It seems like solr query is taking time to update purchase orders hence the item status not getting updated post closing PO items. Need to find a solution for that.

Issue verified and working fine in v2.25.0.
https://jam.dev/c/27dbd808-2764-40c3-a52c-2403b75ccd9f


Drop distutils support

I feel it's about time to drop distutils support as we rely on extras_require of setuptools to handle yaml dependencies..
What's your opinion on this matter?

Also see #605

:+1:

Yeah, good point


Use Nanoleaf A19 Matter Bulbs for HomeKit Bulbs

Checklist

[X] I have filled out the template to the best of my ability.
[X] This only contains 1 feature request (if you have multiple feature requests, open one feature request for each feature request).
[X] This issue is not a duplicate feature request of previous feature requests.

Is your feature request related to a problem? Please describe.
I would like powercalcs for the fleet of Nanoleaf A19 bulbs that I have in my house. They are the original release which communicated through Thread over WiFi exclusively to HomeKit or the Nanoleaf App. Aside from that they are identical to the newer Matter successors.
Describe the solution you'd like
Can we use the stats added in #2194 for HomeKit bulbs?
Describe alternatives you've considered
I could duplicate the work done by @DawidPietrykowski to confirm, but based on all documentation they are identical aside from the communication protocol.
Additional context
Here's what the bulb looks like in the HomeKit Device Integration

Communication protocol can "matter" in power consumption. As Wifi tend to use a tidy bit more.
Are you able to take a few samples at different settings, and see if power consumption is similar?


[Decode]Remove AVC level 6.0 check, Do Not Merge For Now!

Change can not be merged - there are some conflicts. Please, rebase the change, resolve conflicts and squash all changes into a single commit.

Rebased, please try again.

Change can not be merged - there are some conflicts. Please, rebase the change, resolve conflicts and squash all changes into a single commit.

Make the code change in another pr [Decode]Remove AVC level 6.0 check,  removed the line "(void)(ignore_level_constrain);" to make code cleaner.


Signature certificate expired

The signature certificate is expired since January, 2023.
Is it possible to renew it?

Can you elaborate on what signature is expired?
We sign the .exe's with timestamping, so no expiration should matter in that aspect. If it's an individual profile from eduroam CAT, that's an entirely different thing we need to look at.

Can you check if this is any different with the newly released binaries,
https://github.com/geteduroam/windows-app/releases/tag/v4.1.3
I tried to sign them in a slightly different way. Maybe it makes no difference.

I do hope that the issue is resolved, possibly with the 4.1.3 binaries. Otherwise, please re-open/let us know.


Position of labels in notebooks

Which area is this feature request for?
Website User Interface, Other
Describe the feature you'd like to request
https://myst-tools.org/docs/mystjs/cross-references#targeting-cells says:

You can label notebook cells using a comment at the top of the cell, using a #| label: syntax, or have this added directly in the notebook metadata for the cell.

Suggestion:
I think it would be nice to have also the option to place the label not only at the first line, but also at the last line of a cell.
That would have two advantages:

For cells with long examples, label and image would stay close together and are easier to associate towards each other.
Currently, when there's a lot of other cell content in between label and image, the connection between these two might be more difficult to grasp with one glance.
Another benefit would be that one could search for "#| label: and would directly have the option to jump to all labelled images together with their tags.

The first line of a jupyter cell is also the place of jupyter cell magic:

%%time
import matplotlib.pyplot as plt
plt.plot([1,2], [10,20], c = "orange")

and cell magics can not be placed in a second line.
When there is a comment above the cell magic, then jupyter thinks it's not a cell magic but a linemagic (these are used in any lines of a notebook) and throws a UsageError.

but
%%time
import matplotlib.pyplot as plt
plt.plot([1,2], [10,20], c = "orange")
#| label: my-plot

will run without problems.
Having these two point in mind:
What do you think about having the label ONLY in the last line?
Describe the solution you'd like
No response
Describe alternatives you've considered
No response

And another thought:
myst-nb uses cell tags to hide-input, hide-output and hide-cell.
See here -> https://myst-nb.readthedocs.io/en/latest/render/hiding.html
maybe these cell tags can be uses for mystjs for labels as well.

I will tackle this now -- very similar to #542!

I have done a partial improvement in #543. This allows the the #| label: comment to come after an ipython magic command, but it still requires it at the top of the cell.
Will leave this issue open and would love to hear from @fwkoch and others if we want that syntax to be supported anywhere.

that's great to hear, thanks for implementing this @rowanc1!
One more thought on label position:
Cells with plots can have many lines on code, and it can happen that cell label and cell output don't fit on the screen simultaneously:

I think in that particular case, a label at the button would make it easier to associate the connection between label and output.

Yep! I would love to hear from @fwkoch on if there are any downsides to opening this up to be allowed anywhere. :)
The potential downsides I see are (1) that it might be tempting to think that the labels are associated with the output, when they are associated with the cell. For example
#| label: fig1
plot([1,2,3])
#| label: fig2
plot([5,6,7])

And (2): Should we support indented labels? e.g. inside of a function?
Curious on your thoughts on those potential issues?


Update jquery.creditCardValidator.js

credit card validation ELO

This is incomplete. jCCV is written in CoffeeScript so you should modify the .coffee file, compile, then push both .coffee and .js files.
I also need a source that says these prefixes indicate ELO card.


Scoreboard page has an UI bug

When scorlled down, the scoreboard grid goes over the main menu:
https://www.dropbox.com/s/wqqm2sodat93xqe/Screenshot 2015-09-28 15.48.21.png?dl=0
Might be as simple as upping the z-index of the menu/header.
/cc @joseihf

I belive my pull request #420 will fix this issue.

I can confirm this is now fixed. Thanks @ercchy!


Updated flink 1.8 -> 1.17. Added smoke tests for Flink

Updated  the Apache Flink - related  code to the most recent version. It can be considered a first step towards https://github.com/dmlc/xgboost/issues/8650 as most recent Flink versions support scala 2.13. We can, certainly, follow Flink approach and deprecate scala-related flink logic, but I am not a person to make this decision.

Thank you for the work on flink! It will take a few days for a review since it has been a long time there was no contributor to the flink interface.

@trivialfis no worries. I've mostly converted existing code and partially translated some of scala code to java as flink moved significant part of their code base that way. I am happy to hear any suggestions/comments.

@CodingCat Hi, do you have some bandwidth for looking into this?

rebased to include latest changes.

I've noticed that flink does not publish flink-scala for 2.13. Perhaps, it is wise to keep just java API in place.

@trivialfis my former team was using flink extensively, but they relied on jpmml. I have noticed that the flink xgboost interface was out of date and wanted to fix that.
My current job relies on spark and we want to upgrade our scala version as well. Hence I am in the "two birds with one stone" situation.
I am fine with the "minimum amount of changes that can get things going" and, perhaps, deprecating the flink integration. I do not mind supporting the interface for a while, but I can not devote myself for the flink support unless there is a community interest.

Actually, I'm quite excited that you work on the flink interface! We recently learned about the flink project and its strength in streaming data, then an expert come and bring it up to date! What are the odds?

but I can not devote myself for the flink support unless there is a community interest.

No worries, we can see how it goes. I will try to catch up on this area as well.

@trivialfis Great to hear. Well, a while ago I have done some work on flink ml, but I can not call myself an expert. Xbgoost can consider collaborating with the flink-ml project though. But this is, perhaps, a future topic :)

I am not familiar with Flink, but the code looks neat, and the training pipeline is quite similar to spark. LGTM.

rebased on top of the latest master branch

Please help resolve the conflict. Thank you again for the nice work on bringing up the flink interface!

@trivialfis I see that the XGBoost-R-Tests / Test R on OS windows-latest, R release is failing. I am not familiar with R and I do not use windows at the moment. Is this a known issue?

Please ignore it for the moment, we are aware of the issue https://github.com/dmlc/xgboost/issues/9091 .


Cannot find module '../../package' from 'node.js'

Not sure if this is a bug related to this package or typescript, but I isolated it and was able to get it to work in plain JS.
Here's the bug:
Cannot find module '../../package' from 'node.js'

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:191:17)
      at Object.<anonymous> (node_modules/babel-core/lib/api/node.js:60:16)

For reproducibility, I've built out an example that includes both a success (using plain js) and a failure test (using the typescript test)
https://github.com/pyramation/typescript-starter-example

Seems to be an issue in babel-core. I've opened a PR, so you can keep track in there.


Use fallback version when no Git tags are found?

Currently, it looks like the fallback version is only used when pdm-backend completely fails reading anything from the SCM, i.e. the project is not a Git repository. When the project is a Git repository, but doesn't have any tags, pdm-backend will infer a version like 0.1.devN+gHASH.dDATE.
I'm in a situation where uv refuses to solve my dependencies, because my current project dev-depends on a library that depends on back the current project, and since the current project version is computed as 0.1.etc (because it doesn't have Git tags[^1]) , it's not compatible with what the library requires, and uv fails. See https://github.com/astral-sh/uv/issues/8148. I think uv should not fail, or have a config flag to ignore the current project version.
[^1]: It doesn't have Git tags for various reasons. For example it could be a contributor fork, and GitHub doesn't copy tags when creating a fork.
In the meantime, I'm trying to use the fallback version, but since my project is a Git repo, the fallback version doesn't seem to be used. That's also what I gather from reading the code:
            scm_version = get_version_from_scm(
                context.root, tag_regex=tag_regex, tag_filter=tag_filter
            )
            if scm_version is None:  # <-- probably false, and branch is not entered
                if fallback_version is not None:
                    version = fallback_version

Do you think it would be feasible to use the fallback version if no tags were found, even though the project is a proper Git repository?

OK I found a uv-based solution (see https://github.com/astral-sh/uv/issues/8148#issuecomment-2476395325), this issue can be closed :slightly_smiling_face:

Actually that led to other issues. So I used my brain and found a pdm-backend solution:
[tool.pdm.version]
source = "call"
getter = "scripts.get_version:get_version"

# scripts/get_version.py
import re
from pathlib import Path

from pdm.backend.hooks.version import Version, SCMVersion, get_version_from_scm, default_version_formatter

_root = Path(__file__).parent.parent
_changelog = _root / "CHANGELOG.md"
_changelog_version_re = re.compile(r"^## \[(\d+\.\d+\.\d+)\].*$")
_default_scm_version = SCMVersion(Version("0.0.0"), None, False, None, None)

def get_version():
    scm_version = get_version_from_scm(_root) or _default_scm_version
    if scm_version.version <= Version("0.1"):  # Missing Git tags?
        if _changelog.exists():
            with _changelog.open() as file:
                for line in file:
                    if match := _changelog_version_re.match(line):
                        scm_version = scm_version._replace(version=Version(match.group(1)))
                        break
    return default_version_formatter(scm_version)


[Azure Stack] Rename folder to correct version

Latest improvements:
I renamed the folder to match the API version, also updated readme.md to use correct folder/package values.
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[X] I have reviewed the documentation for the workflow.
[X] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[X] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

Automation for azure-sdk-for-js

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-python

Automation for azure-sdk-for-java

Automation for azure-sdk-for-node

Automation for azure-sdk-for-go

Can one of the admins verify this patch?


Bug: nvarchar length on MSSQL Server doubles or triples in table metadata

To whom it may concern,
On DBeaver, the length of all nvarchar fields on my table read 1020.  Looking on SSMS, the actual length is 255.  Earlier today, those same fields were reading 510 as a length.  This looks to be a bug

Thanks for the bug report

It looks now there is a different issue:
In looking at the column length under table properties in MSSQL, the length shows a length of zero.  When I try to alter this, it changes the length to a length of one.  Thank you for your help with this and for your work on DBeaver

Thanks Serge!

Still reproduced in 6.0.0201903101945:
sqlcmd> create table dbo.dbeaver_str( id int, vc varchar(10), nvc nvarchar(10))
But after DDL generation in DBeaver:
-- Drop table

-- DROP TABLE test_vc.dbo.dbeaver_str GO

CREATE TABLE dbeaver_str (
	id int NULL,
	vc varchar(10) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
	nvc nvarchar(20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL
) GO;

I have both navicat & dbeaver, and on dbeaver the nvarchar is doubled in length
dbeaver version:

navicat:

dbeaver:

This is still an issue.

I think this issue should be re-opened. I can reproduce it on version 23.0.0.

@dr-1 please create a new issue with steps to reproduce


Add a basic schema generator app

Description
start of https://github.com/creek-service/creek-json-schema/issues/3
Add an application that echos its arguments. This app will become the schema generator.
Reviewer checklist

[ ] Ensure relevant issues are linked (description should include text like "Fixes #")
[ ] Ensure any appropriate documentation has been added or amended

Coverage decreased (-40.5%) to 59.459% when pulling 1ebe2bf5e8995253de8a46482df7a1587ef0ac80 on empty_app into b9c15980b29ef126c5973e9dc677769fe1b8b11c on main.


docs: Document GraphQL mutations.

Document the 3 types of GraphQL mutations: delete, insert and update.

edb.errors.EdgeQLSyntaxError: Unexpected '}'
The above exception was the direct cause of the following exception:
AssertionError: unable to parse edgeql code block in
/home/travis/build/edgedb/edgedb/docs/clients/99_graphql/mutations.rst, around line 82


AmbassadorEventWatcher OOM with 0.75.0

Describe the bug
AmbassadorEventWatcher OOM and seem to stop watching new service changes.
Some relevant log entries
2019-09-30 22:19:50 diagd 0.75.0 [P687TAmbassadorEventWatcher] ERROR: could not reconfigure: [Errno 12] Out of memory
2019-09-30 22:19:50 diagd 0.75.0 [P687TAmbassadorEventWatcher] ERROR: [Errno 12] Out of memory
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador_diag/diagd.py", line 645, in run
    self.load_config_watt(rqueue, url)
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador_diag/diagd.py", line 820, in load_config_watt
    self._load_ir(rqueue, aconf, fetcher, scc, snapshot)
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador_diag/diagd.py", line 827, in _load_ir
    open(aconf_path, "w").write(aconf.as_json())
OSError: [Errno 12] Out of memory
2019/09/30 22:19:50 aggregator: watch hook failed: signal: killed
2019/09/30 22:19:50 aggregator: found 0 kubernetes watches
2019/09/30 22:19:50 aggregator: found 0 consul watches
2019/09/30 22:19:50 kubewatchman: processing 0 kubernetes watch specs
2019/09/30 22:19:50 consulwatchman: processing 0 consul watches
2019/09/30 22:19:51 kubebootstrap: found 2036 "service" in namespace "*"

output from top inside the pod
Mem: 120882248K used, 471977472K free, 1148544K shrd, 739936K buff, 11741152K cached
CPU:   1% usr   0% sys   0% nic  98% idle   0% io   0% irq   0% sirq
Load average: 1.87 1.25 1.03 2/3713 13034
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  782     1 8888     S    1060m   0%  15   0% envoy -c /ambassador/bootstrap-ads.json
<PHONE_NUMBER>     S     194m   0%  53   0% {diagd} /usr/bin/python3 /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --kic
  696     1 8888     S    5174m   0%  41   0% /ambassador/watt --port 8002 --notify sh /ambassador/post_watt.sh -s service -s AuthService -s Mapping -s Module -s RateLimitService -s TCPMapping -s TLSContext -
  668     1 8888     S     122m   0%  20   0% ambex -ads 8003 /ambassador/envoy
  675     1 8888     S    42080   0%  43   0% {diagd} /usr/bin/python3 /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --kic
13020 13017 8888     S    19600   0%  36   0% python3 /ambassador/post_update.py http://localhost:8002/snapshots/997
    1     0 8888     S     2324   0%  18   0% {entrypoint.sh} /bin/bash ./entrypoint.sh
13022     0 8888     S     2316   0%   0   0% bash
13017   696 8888     S     1588   0%  32   0% sh /ambassador/post_watt.sh http://localhost:8002/snapshots/997
13034 13022 8888     R     1528   0%  51   0% top

To Reproduce
N/A
Expected behavior
Should recover from OOM and not become stale.
Versions (please complete the following information):

Ambassador: 0.75.0
Kubernetes environment: bare metal
Version 1.13.6

Additional context
We have three replicas running at the same time, two ran into this issue, one was ok.  Not sure how to reproduce it.  I delete one of the problem one and it recovers after restart.

I also encounter the exact same issue for all my pods. I think this is an important feature because there is no notification at all. The issue is only discovered when I tried to deploy a new service.

I also faced a similar recently where "EventWatcher" was not running since 5days.
So, iPs started getting stale in envoy.json.
Ambassador wasn't showing errors or crashes.


feat: DataStore and API Lazy Loading with Custom Selection Set

Issue #, if available:
Table of Contents

Custom Selection Set -> https://github.com/aws-amplify/amplify-swift/pull/2526
LazyReference -> https://github.com/aws-amplify/amplify-swift/pull/2583 (You are here)
Codegen -> https://github.com/aws-amplify/amplify-codegen/pull/504
CLI feature flag -> https://github.com/aws-amplify/amplify-cli/pull/11566
Docs -> https://github.com/aws-amplify/docs/pull/4934

Instructions for installing this PR

See codegen PR (https://github.com/aws-amplify/amplify-codegen/pull/504) to set up amplify-dev to generate the models used by this PR
To install this PR in your iOS App, Add Package and use

repo: https://github.com/aws-amplify/amplify-swift.git
branch: data-dev-preview

Description of changes
This PR introduces the LazyReference type and plugin implementations to extend scenarios in API and DataStore Model decoding. Without this PR and related codegen changes, a belongs-to/has-one association of a model must exist when it is a required association and must be in the data payload for it to be decoded successfully into the Model with that association. For example, a Comment belongs-to a Post, when retrieving the Comment, the data payload being decoded must include the Post because the model type defintion of the Comment includes the Post as a reference:
struct Comment: Model {
  let id: String
  var content: String?
  var post: Post
}

The data payload must include post for it to decode, (shown as json below):
data: {
   id: "commentId",
   content: "content",
   post: {
      postId: "postId123",
      title: "postTitle"
   }
}

This is known as eager loading the associated data. All retrievals of the Comment requires that the post data is retrieved at the same time as well- in DataStore, this is done using a SQL join statement. In API plugin, this is specified in a nested selection set containing the post fields.
This PR introduces the LazyReference type. The post reference in the Comment type is replaced with an instance of  LazyReference<Post>.
struct Comment: Model {
  let _post: LazyReference<Post>
  public let post: Post {
     get async throws {
         try await _post.require()
     }
  }
}

The computed property post is backed by the internal _post of type LazyReference.  Alternative solutions to having the computed property backed by the internal property is to have a property wrapper instead. The main disadvantage of the property wrapper is the inability to perform an async operations, like how we are achieving it with the async throws getter of the computed property.
With the computed property and internal property, the developer call pattern becomes:
let post = try await comment.post

When the association is required, the underlying API used is _post.require(), if the association is optional, then _post.get(). _post.require() will throw if the model cannot be loaded, while get() will return nil. The LazyReference type allows the data payloads to include all fields (eager loaded) or include only the primary keys of post (lazy loaded), used for decoding to the type into one of the two states.
When a LazyReference instance is created during data decoding, a model provider is created using the registered plugin's model provider registry.
When using API plugin:
flowchart TB
  A["Amplify.addPlugin(plugin)"] --> B["Amplify.configure()"]
  B --> C["APIPlugin.configure()"]
  C --> D["ModelProviderRegistry.registry(AppSyncModelDecoder.self)"]
  E["Amplify.API.query(.get(Comment.self, byId: 'commentId'))"] -- response payload --> F["LazyReference decoder"]
  F --> G["ModelProviderRegistry.decode()"]
  G --> I["LazyReference's modelProvider = AppSyncModelProvider()"]

When using DataStore plugin:
flowchart TB
  A["Amplify.addPlugin(plugin)"] --> B["Amplify.configure()"]
  B --> C["DataStorePlugin.configure()"]
  C --> D["ModelProviderRegistry.registry(DataStoreModelDecoder.self)"]
  E["Amplify.DataStore.query(Comment.self, byId: 'commentId')"] -- response payload --> F["LazyReference decoder"]
  F --> G["ModelProviderRegistry.decode()"]
  G --> I["LazyReference's modelProvider = DataStoreModelProvider()"]

The LazyReference instance delegates its lazy loading capabilities to the underlying plugin's model provider, which performs requests to its data source (either SQLite or AppSync directly)
classDiagram
    class LazyReference
    LazyReference : +ModelProvider modelProvider
    LazyReference : +LazyReferenceValueState _state
    LazyReference : +get()
    LazyReference : +required()

For example, API plugin will make a GraphQL get request using the model schema and identifiers of the model to retrieve the entire model and transition the LazyReference to a loaded state.
flowchart TB
  A["let post = try await comment.post"] --> B["try await _post.require()"]
  B --> C["try await modelProvider.load()"]
  C --> D["try await Amplify.API.query(.get(Post.self, byId: 'postId'))"]

The selection set created above includes only the primary keys of the post when using the latest codegen that includes the rootPath, to control the selection set customizations from  @drochetti's PR  https://github.com/aws-amplify/amplify-swift/pull/2526. This is a pre-requiste PR to allow selection set customizations to create "not loaded" LazyReference instances.
DataStore will perform a SQL select operation.
flowchart TB
  A["let post = try await comment.post"] --> B["try await _post.require()"]
  B --> C["try await modelProvider.load()"]
  C --> D["try await Amplify.DataStore.query(Post.self, byId: 'postId')"]

The plugin will manipulate the data payload to always add the model's identifier as metadata to the last level of the model graph. When dealing with has-many associations, it will add the target's associated identifier (comments hold the post identifier as metadata). When dealing with the belongs-to, it will add ithe target's identifier (post will hold the post's identifier). The added metadata gets decoded to a LazyReference and List types to hold the metadata in its model provider. This allows the developer to tranverse the model graph indefinitely using the existing lazy List implementation and the new LazyReference implementation, regardless of the initial loaded state of the model.
Blog-Post-Comment example
flowchart TB
  A["let post = try await comment.post"] --> B["let blog = try await post.blog"]
  B -->  C["try await blog.posts?.fetch()"]
  C --> D["let posts = blog.posts"]
  D --> E["try await post.comments?.fetch()"]
  E --> F["let comments = post.comments"]

AppSync LazyReference metadata
AppSyncModelProvider's idenifier data is an ordered dictionary of its identifiers, the ordering matters to specify the ID! type and String! types when making the AppSync get request. For default identifier, this is ["id":"idValue"]. For a composite key, with a renamed primary key for example, will be ["postId":"postIdValue", "title":"titleValue"]
DataStore LazyReference metadata
DataStoreModelProvider's identifier is the parent object's foreign key value. The column name is either the single column name like "id" or "postId" or for composite keys, it is called @@primaryKey.
Codegen Changes
PR with related codegen changes https://github.com/aws-amplify/amplify-codegen/pull/504
Selection Set changes

DataStore establishes subscriptions with a reduced selection set, the nested selection set only contains primary keys.
The codegen models have to be the ones with the LazyReference type since it can decode the reduced selection set
DataStore's mutations do not change, since older iOS clients or iOS clients using the previously codegenerated models will continue to decode the response payload successfully.
(iOS DataStore imeplementation detail) The __typename is injected at runtime from the request to the response that allows response payloads missing the __typename to be decoded successfully.

Developer Experience / Migration Guide
https://github.com/aws-amplify/docs/pull/4934
Issues
Use case: Has-One Bidirectional belongs to
See https://github.com/aws-amplify/amplify-swift/issues/504#issuecomment-1368132314
See https://github.com/aws-amplify/amplify-swift/issues/505#issuecomment-1368127124
Use case: Cross platform DataStore events (Android)
DataStore on Android syncs data to AppSync using a selection set in the mutation to AppSync differently from iOS. For the existing issue, see x. The problem is that there is not enough data in the subscription event, due to Android's selection set missing fields that are required for iOS DataStore. For example, in the Comment Post example, Android may sync a comment with the selection set
comment {
  id
  post {
    id
  }
}

This will fail to decode to iOS without the LazyReference codegen changes because the swift model type Post that is a field on the Comment may have required fields. With the LazyReference encapsulation, the post field on the Comment will be decoded as a "not loaded" Post with the identifiers. In DataStore's reconciliation logic, it only needs to extract the identifiers of the Post from the comment and store that as the FK values in the Comment table. It doesn't need the entire Post object.
See https://github.com/aws-amplify/amplify-swift/issues/1753#issuecomment-1363104512
Use case: Cross platform DataStore events (Studio/JS)
Studio/JS sends a mutation events with a selection set of this structure:
mutation MyMutation2 {
  createBookAuthor(input: {authorId: "b87ebfd4-04e7-45aa-b54a-6916928bb5c8", bookId: "532078b3-376a-451d-a470-608cc7e462a5"}) {
    _deleted
    _lastChangedAt
    _version
    authorId
    bookId
    createdAt
    id
    updatedAt
    author {
      id
      _deleted
    }
    book {
      id
      _deleted
    }
  }
}

no __typename
version metadata fields at the top level, _deleted, _lastChangedAt, _version
nested models only containing its primary keys, in this case id
_deleted

These events are successfully reconciled by the iOS client using this implementation because __typename is injected at runtime by DataStore to decode successfully to the AnyModel type used by DataStore. The author and book in this example is codegenerated to be encapsulated by the LazyReference type and decodes to the "not loaded" states. Their identifiers are retrieved as the FK for saving the bookAuthor join model
See https://github.com/aws-amplify/amplify-swift/issues/1601#issuecomment-1366818315
Check points: (check or cross out if not relevant)

[x] Added new tests to cover change, if needed
[x] Build succeeds with all target using Swift Package Manager
[x] All unit tests pass
[x] All integration tests pass
[x] Security oriented best practices and standards are followed (e.g. using input sanitization, principle of least privilege, etc)
[x] Documentation update for the change if required
[x] PR title conforms to conventional commit style
[x] If breaking change, documentation/changelog update with migration instructions

DataStore checkpoints (check when completed)

[x] Ran AWSDataStorePluginIntegrationTests
[x] Ran AWSDataStorePluginV2Tests
[x] Ran AWSDataStorePluginMultiAuthTests
[x] Ran AWSDataStorePluginCPKTests
[x] Ran AWSDataStorePluginAuthCognitoTests
[x] Ran AWSDataStorePluginAuthIAMTests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Error Handling
Lazy List APIs were launched in Amplify v1 with a return type that is shared across DataStore and API users. Because, they pivot on behavior based on the underlying plugin being used, error scenarios are the plugin's list/model provider's responsiblity to return the correct error case. However, in this architecture, there's also a default list/model provider which is used when neither plugin specific providers can be instantaited. The explicit return type on the APIs off the List type cannot be either DataStoreError or APIError. The default providers also should not return either DataStoreError or APIError since it is a plugin agnostic core type. Because of these reasons we introduced the CoreError, with the API signature like this:
public func fetch(_ completion: @escaping (Result<Void, CoreError>) -> Void)

With the launch of Amplify v2, the API signatures have been updated to the Async APIs here:
public func fetch() async throws

The Async API removed the requirement to explicitly define the CoreError type on the API, except that we still need to return something in the implementation of the core types, such as in the default providers or the code within the Lazy types that manage creating plugin specific providers. This leaves us a path forward in this PR for the minor version bump:

In this PR, the new plugin providers will return their respective errors, APIError and DataStoreError. The existing list providers will continue CoreError.listOperation
CoreError is still needed for the default providers and some other places in the implementation of Amplify core types. We shouldn't remove CoreError, nor can we extend CoreError to new cases that we may want to communicate. For now, we'll return clientValidation error until this changes.

For a major version bump, we can change the behavior of the plugin providers

list providers to return APIError or DataStore instead of CoreError. Extend APIError/DataStore (breaking change) to what is required to accomplish this.
CoreError can be extended to improve the cases it can communicate. There are two cases are
dataUnavailable indicates some data marked as required was unavailable
dataIntegrity indicates expected data could not be loaded due to the lack of integrity.

I'm not sure if I can respond on this thread but is there any update to this in regards to the main. Currently this branch is the only one that I can get @auth owner to work on both physical device and ios simulator (M1). When trying to use amplify 2.0.0, updates do not work.

Sorry it worked for a few times then stopped. It does work when I put @auth private for both the physical device and ios simulator (M1).

Hi @lostdawg3, thanks for trying out DataStore! Feel free to provide us more information in a GitHub Issue . Your schema will be helpful and steps you took to reproduce the issue. That way we'll have a dedicated thread to dig deeper into issues you are experiencing. If there's something that's working on this branch but not on the current main (v2), we can pinpoint the changes necessary and apply them outside of this PR to get you unblocked faster.


[DOC] Evaluate automatic generation of storybook knobs from components.json

Description:
As a developer, I do not want to manually configure each property for a component in Storybook. Therefore, a utility is needed that makes it possible to create default props for stories.
Rationale:
This is needed so save time and can be seen in various frameworks (e.g. when using Stencil or Lit).

Only relevant for FAST. Will be solved with https://github.com/SickDesignSystem/sds/tree/feat/integrate-shoelace-cli


After decreasing render distance server doesn't send some cubes to client

After decreasing render distance, all cubes that are outside of the new render distance but inside the old render distance are not sent to the client.

This may have been fixed in b80c386646c7f871688e8e41bacc3b1f7d3a45e2


update: added something in READme.md

LGTM! Great stuff you've added! The resource was placed in the wrong section. I've committed the changes.
Keep contributing ^^


gulp rather than grunt ?

I see that you do have gulp file , so why don tyou have it on the readme page ?

Hi @jpca999, thanks for noticing! The gulp tooling is in alpha state at the moment as part of our move towards 1.0 - http://support.ghost.org/v1-0-alpha/
I'm going to close this as documentation will be added in the run-up to 1.0 coming out of alpha.


_this3.$refs.match is undefined

Hi,
All works fine, but i have this kind of error at console(_this3.$refs.match is undefined). Do you know why?
Thanks

That’s not good. what happens when you console.log(this.$refs)?
Can you share your calling code?
Rob Rogers
415.615.2859
<EMAIL_ADDRESS>
On Dec 10, 2018, at 11:41 PM, villybob<EMAIL_ADDRESS>wrote:
Hi,
All works fine, but i have this kind of error at console(_this3.$refs.match is undefined). Do you know why?
Thanks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/robrogers3/vue-single-select/issues/10, or mute the thread https://github.com/notifications/unsubscribe-auth/ACpX2j6u3klUe_rfnqvXSIDY-T0G4FUoks5u32GjgaJpZM4ZM1Dp.

I just know that error is only when I use "watch" for hidden the select. If I delete watch I don't have error.
This is the code:
                             <vue-single-select                                  option-label="name"                                 :options="api"                                 v-model="coinSelected">                                     <template slot="option" slot-scope="{option}">

                                                 {{ option.symbol }}

                                                 -

                                                 {{ option.name }}

                                     </template>                             </vue-single-select>

And JS watch:  watch: {         coinSelected(){             if(this.coinSelected.hasOwnProperty('name')){                 this.dropdown=!this.dropdown;                 this.logo=this.coinSelected.icon;                 this.symbol1=this.coinSelected.symbol;                 this.name1=this.coinSelected.name;                 this.coin=this.coinSelected.buy;             }         }     },

ok. makes sense. so $refs.match doesn’t always exist. you can check it via typeof this.$refs.match !== ‘undefined’ before you try to reference it.
Rob Rogers
415.615.2859
<EMAIL_ADDRESS>
On Dec 10, 2018, at 11:51 PM, villybob<EMAIL_ADDRESS>wrote:
I just know that error is only when I use "watch" for hidden the select. If I delete watch I don't have error.
This is the code:
And JS watch: watch: { coinSelected(){ if(this.coinSelected.hasOwnProperty('name')){ this.dropdown=!this.dropdown; this.logo=this.coinSelected.icon; this.symbol1=this.coinSelected.symbol; this.name1=this.coinSelected.name; this.coin=this.coinSelected.buy; } } },
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/robrogers3/vue-single-select/issues/10#issuecomment-446104753, or mute the thread https://github.com/notifications/unsubscribe-auth/ACpX2iEggm03bLpmJwX0EiPUv_9XOwYZks5u32QCgaJpZM4ZM1Dp.


Sync charts with istio/installer

Two way sync with istio/installer charts. After this it should be possible to reference the istio/installer charts directly without duplicating in this repo.
See https://github.com/istio/installer/pull/284 for sync in the other direction.

@ostromart: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

operator-lint-master
c3394b4d19332655b092bd170da7aedf70eb2f13
link
/test operator-lint-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ostromart: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

operator-lint-master
c3394b4d19332655b092bd170da7aedf70eb2f13
link
/test operator-lint-master

operator-test-master
c3394b4d19332655b092bd170da7aedf70eb2f13
link
/test operator-test-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ostromart: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

operator-test-master
c3394b4d19332655b092bd170da7aedf70eb2f13
link
/test operator-test-master

operator-lint-master
0cbc54d38d117fcb80c2f1c7122b507dafb2e0c7
link
/test operator-lint-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test operator-lint-master

@morvencao no, the plan is to start pointing to release branches directly in the istio/installer repo.


Update version to 1.47.1 for hotfix

Bump version for hotfix

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.004%) to 41.574%

Totals

Change from base Build<PHONE_NUMBER>:
0.004%

Covered Lines:
30787

Relevant Lines:
69355

💛  - Coveralls


Can't install azure-cli in Azure Linux 3

Describe the bug
I have trouble during azure-cli installation. I'm not able to install it on Azure Linux 3
The error raised when I run install command.
Related command
tdnf install azure-cli
Errors
azure-cli package not found or not installed
Error(1011) : No matching packages

Issue script & Debug output
N/A
Expected behavior
azure-cli installed without error.
Environment Summary
N/A
Additional context

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."


Fix eq and ne for classes implementing them

Main issue, which concerns Vector, Location, and ShapeList:
Comparison with an object of a different type should not cause an exception - they are simply not equal. Raising an exception in  __eq__ can (and will*) break unrelated code that expects __eq__ to be well-behaved.
(* I noticed this bug when cq-editor choked on it while trying to find a name for an object in a dictionary of local variables)
There's a second more minor issue, which concerns the rest of the classes:
When the other type in __eq__ is not supported, one should technically return NotImplemented instead of False, to allow the other type to take part in the comparison, in case they know about our type.
(__ne__ should also not generally be implemented as just the negation of __eq__ because of this, but the one for Plane can just be removed - Python will automatically do the right thing based on __eq__ there -- ShapeList, on the other hand, needs to also have its __ne__ overridden, because it directly subclasses list, which already implements it)
Technically, the __eq__ for Vector and Plane is also broken in another way: It's not transitive.
>>> a, b, c = Vector(0), Vector(9e-6), Vector(18e-6)
>>> a == b == c
True
>>> a == c
False

They should really eg. have a separate is_close() or something for approximate comparison, but this isn't fixed here, since I have no idea how many places it'd break, for one.
I also noticed that Shape.__eq__ calls Shape.is_same instead of Shape.is_equal which just seems... wrong.

Thanks!  A lot of effort has been put into making the code fully align with Python (the original CadQuery code which this is based off didn't follow Python conventions) but this was area that needed work.
You are correct that Shape.__eq__ isn't really doing what an == operator should do but there is no way to compare CAD models to create the == operator otherwise.  Too Tall Toby uses a volume (actually weight) comparison but that can easily be shown to be incomplete.
With respect to the transitive property, I don't know of a way to solve this problem given some tolerance must be provided. The Python isclose function works with tolerance values too.  If you know of a robust solution to this please let me know.


Enable referencing plugin components by ID (internal-registry only)

What does this PR do?
Implements the internal registry for components that specify a plugin by ID. This re-enables the normal web terminal flow (including defaulting the tooling container component)
What issues does this PR fix or reference?
Part of https://github.com/devfile/devworkspace-operator/issues/249
Is it tested? How?
make docker install
oc apply -f samples/web-terminal.yaml

Note: opening the web terminal from the openshift UI will not work, due to https://github.com/devfile/devworkspace-operator/issues/258

/test v5-devworkspaces-operator-e2e

/retest

/test v5-devworkspaces-operator-e2e

/retest

/test v5-devworkspaces-operator-e2e

The full WTO flow works with changes from https://github.com/eclipse/che-machine-exec/pull/129 applied \o/


Reverse CDDL testing

Generate a mini protocol message with QuickCheck
and test that the message satisfies the CDDL spec.

@karknu Thx for the review. Please check my cleanups

bors r+

bors r+


launch vThunders via nova

We need a module that can launch a vthunder via nova. there should be some nova client code that looks in the neutron config for the nova creds in neutron or octavia to look at.

[ ] add options to config file to specify vthunder image, flavor size, etc.
[ ] launch vthunder (mgmt_network, data_network), check vthunder, cleanup vthunder.

This will be part of a10-neutron-lbaas for now

[ ] Instance CRD

[ ] Unit tests
[ ] Integration tests
[ ] Need to know the image name so we can get the ID (passed from UI?) Look at existing instance creation UI - do we allow selection of vThunder image?
[ ] Need to know the flavor name so we can get the ID (passed from UI? Look at existing instance creation UI - it constricts flavors to image requirements)


Performance metatask

Just making this metatask to track performance issues and notes.

The HTML for the Reference page is quite large. The text compression on GH Pages server handles this for the most part but flagging here in case the platform is ever switched.
CodeEmbeds appear to be loading their p5 scripts in parallel and in my browser the cache isn't catching
As expected there is a significant loading state on the search results page. Layout placeholders should be used to mitigate dramatic shift

It is possible that the font file loading can be shortened and masked with this plugin. Should only matter for initial load but is noticeable on very slow connections

Potentially worthwhile to migrate the Icon to an astro component


Show account with both followers/following if they are both followers and being followed

Currently the map displays account info as only followers or following even if they are both. Also there is no account info.

Twitter does display that info, e.g. see below. Just follow the twitter design

[x] Show both followers/following
[x] Offer "un/follow" button
[x] Show account info text
[x] Show location info of account
[x] Keep the info of number of tweets, followers, following in the view (as currently implemented)

Very nice work. Is it possible to make the effect like a slide in/slide out effect of the map (just like the activity sidebar) and instead of loading the side expanding the central activity bar to become the main area?


stability: improve recovery after a crash of the majority of servers

This relates to the scenario mentioned at https://github.com/dotnet/dotNext/issues/89#issuecomment-1008726787.
Scenario: running the modified example on 3 nodes, kill 2 nodes (ctrl - c)  and then restart one of them.
Expected: consistently resumes normal operation when at least 2 out of 3 nodes are running again
Actual: cluster often does not recover consistenly
Notes:

it is usually possible to recover the cluster by retrying to restart the different nodes multiple times
we also saw a failure where a cluster that had been running without error during the weekend, failed a while after we connected to the 3 nodes. In this case, 2 of the nodes logged request timeouts, while another one logged an error from the data modifier without any further output until restarted (even after restarting the other nodes first). Given the busy nature of the run, some crazy ideas:   what if the extra resource usage or any timing difference writing to the console made it easier to hit some limit and hit a race condition
we unintentially made a different test where we pulled the power of one of the nodes. When recovering it, its log had broken, which looked clearly different to the falures above. Unfortunately I did not keep the message, but it was something an index being out of range or not found in the log.

but it was something an index being out of range or not found in the log.

That may be an issue with filesystem setup. For instance, when SSD/NVMe is in place, Linux allows you to configure buffer caches for disk I/O. It means that even if you write some bytes to FileStream/SafeFileHandle they can be not yet written to the disk. Persistent WAL has WriteThrough flag to skip any intermediate OS-level buffers when writing to the disk  However, it's always up to OS I/O layer.

@freddyrios , look at https://github.com/dotnet/dotNext/discussions/94#discussioncomment-1946231. Looks like the more accurate calculation of heartbeat timeout on the leader node (now it includes subtraction of replication duration) leads to more stable cluster. Anyway, this is a step forward to our goal.

Here is a new branch where I'm working on new protocol on top of TCP: https://github.com/dotnet/dotNext/tree/feature/new-raft-tcp

The deployment had default rasbian, so it makes sense there could be
something like that at play for that log issue.
On Tue, Jan 11, 2022, 18:44 SRV @.***> wrote:

but it was something an index being out of range or not found in the log.
That may be an issue with filesystem setup. For instance, when SSD/NVMe is
in place, Linux allows you to configure buffer caches for disk I/O. It
means that even if you write some bytes to FileStream/SafeFileHandle they
can be not yet written to the disk. Persistent WAL has WriteThrough flag
to skip any intermediate OS-level buffers when writing to the disk However,
it's always up to OS I/O layer.
—
Reply to this email directly, view it on GitHub
https://github.com/copenhagenatomics/dotNext/issues/1#issuecomment-1010208144,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQ42ZDIKG3QFDMM3HSOHIU3UVRUAJANCNFSM5LTMQR6Q
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

I am on a phone/train, so had trouble fully following.
However, what I understood is that the response time trying to send
heartbeats/entries to one follower prevented the leader from sending
heartbeats/entries at the expected rate to a different follower.
I could see addressing that helping the stability for some of the scenarios
we have hit.
One thing I noticed when playing with the raft simulator, is that
heartbeats do not always align across different nodes.so depending on what
was going on, it looked a lot like the leader could keep the heartbeats
frequency independently for each follower. I suspect that decoupling is an
alternate way one deals with it. However, it does make sense that it would
keep trying at the right frequency, even if one of the appends happened to
be a bit heavy due yo the entries being sent.
For healthy followers the time to append entries is supposed to be an order
of magnitude less than the election timeouts. If these are slow enough to
matter for healthy nodes, then it could make some of the raft paper
assumptions problematic I guess.
Another independent but related stability area I have thought about is:
what happens when appending entries to a followers takes long (due to the
size and amount of entries for example), can this result in the follower
becoming a candidate even though the leader is actively communicating to
it? (not sure what the right behavior of such case, as the problem
appending those entries could be in either side).
On Tue, Jan 11, 2022, 19:16 SRV @.***> wrote:

@freddyrios https://github.com/freddyrios , look at dotnet#94 (comment)
https://github.com/dotnet/dotNext/discussions/94#discussioncomment-1946231.
Looks like the more accurate calculation of heartbeat timeout on the leader
node (now it includes subtraction of replication duration) leads to more
stable cluster. Anyway, this is a step forward to our goal.
—
Reply to this email directly, view it on GitHub
https://github.com/copenhagenatomics/dotNext/issues/1#issuecomment-1010233843,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQ42ZDJX6VJKP3EAQI2NNLTUVRXWDANCNFSM5LTMQR6Q
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

what happens when appending entries to a followers takes long (due to the
size and amount of entries for example), can this result in the follower
becoming a candidate even though the leader is actively communicating to
it?

Yes, it is. We can mitigate this by stopping the timer counting on the follower node during execution of AppendEntriesAsync.

@freddyrios , done: https://github.com/dotnet/dotNext/commit/a6ab90237433ae4504947dfe4193f742cfcc8f1f

@freddyrios , I'm still working on new TCP transport. I expect to finish it this holidays. After that, I'll start testing and stabilizing the release using use cases you mentioned previously. All patches and new transport will be released as 4.2.0 version.

Alright, TCP transport is re-implemented now. Of course, I'll do the necessary tests using your examples. Now I think it's reasonable to implement recovery procedure for persistent Write-Ahead Log. Here is my plan:

Implement checksum for node's state file: node.state
Basic functionality for recovery procedure:
1.1. If node's state file is corrupted then the entire WAL can be considered as invalid. We can cleanup the WAL.
1.2. Add ClearAsync method to PersistentState
1.2. Add ability to override InitializeAsync method of persistent WAL. For each log entry, we can embed a checksum or even duplicate the data. Therefore, at startup we will able to validate all committed log entries. To be more precise, we will able to do that with existing functionality through ReplayAsync that allows to warmup underlying state machine. During initialization, we can easily verify the checkum and throw exception. In very bad case we still be able to execute the node even with empty WAL.

P.S.: Regarding our protection from bit flips: I think SHA is very heavyweight for our purposes. It's crypto-strong hash code typically not considered for data consistency check. FNV1a or CRC is much better and more performant.

Verification of node.state could prevent from accidental power off and incorrect indexes within the log.

My investigation shows that storing checksum for node.state makes a little sense. In general, we can't handle accidental power off normally:

Disk writes must be transactional and underlying file system must support copy-on-write semantics
Or WAL must support reliable writes via insertion of checkpoints and checksum for each checkpoint

Anyway, fault-tolerant WAL is another field of theoretical and practical research. I didn't have a goal to write WAL for such cases. The current implementation just works, at least under normal circumstances.
However, we can have workaround to simulate simple checkpoint. Before every append operation, we can save WriteInProgress indicator to the external file. The indicator can have a size of 1 byte. When operation is finished, just reverse this indicator. In case of accidental shutdown, the file will keep WriteInProgress indicator. It's enough to decide that the entire WAL probably broken and skip all the data in it to have a fresh setup. All these things out of scope at the moment, so I'll focus on stability of the cluster itself.

When recovering it, its log had broken, which looked clearly different to the falures above.

Found the root cause of this issue even in case of graceful shutdown of the node. The problem is in snapshot when it is installed for the index that is greater than the first index in the partition. In that case, the partition becomes completely invalid because the binary offset calculated incorrectly within partition file.

https://github.com/dotnet/dotNext/commit/ebd3b96c010050fea5bd60baf312bb83f28c2142:

Fixed aligned pointer access on ARM devices (on ARM devices, dereference of unaligned pointer can cause SEGFAULT)
Fixed bug that I mentioned previously: incorrect calculation of write position within the partition file when the caller trying to insert the log entry in the mid of the partition after snapshot installation.

Fixed aligned pointer access on ARM devices (on ARM devices, dereference of unaligned pointer can cause SEGFAULT

Nice catch. One I hit on our side in the past with ARM was trying to use https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.memorymarshal.cast?view=net-6.0.

Fortunately, there is a small amount of such code. It is used on the hot path of program execution where the serialization/deserialization of log entry metadata required.

Test branches have been rebased to latest (if updating local copies, might want to delete them locally and pull them fresh):

https://github.com/copenhagenatomics/dotNext/tree/feature/bigstruct-example-rebased
https://github.com/copenhagenatomics/dotNext/tree/feature/bigstruct-slow

The first example uses only default settings, except for BufferSize that is now 4096*2, which is a bit larger than one of the big entries (8000). Some short testing showed better stability, including in the scenario where 2 nodes crash.
The second example adds a 1 second delay after writting 16 entries. Note this is equivalent to the old slow + frequent snapshots, as the 50 records per partition was not changed (based on input on that being bad on its own). Short testing ran into 2 of the below crashes during re-elections. Restarting the crashed node recovered succesfully.
Process terminated. Assertion failed.
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState.LogEntry..ctor(MemoryOwner`1& cachedContent, LogEntryMetadata& metadata, Int64 index) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.LogEntry.cs:line 44
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Partition.Read(Int32 sessionId, Int64 absoluteIndex, LogEntryReadOptimizationHint hint) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.Partition.cs:line 218
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ApplyAsync(Int32 sessionId, Int64 startIndex, CancellationToken token) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\MemoryBasedStateMachine.cs:line
455
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ApplyAsync(Int32 sessionId, Int64 startIndex, CancellationToken token)
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ApplyAsync(Int32 sessionId, CancellationToken token) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\MemoryBasedStateMachine.cs:line 479
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass20_0.<<CommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\MemoryBasedStateMachine.cs:line 311
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass20_0.<CommitAsync>g__CommitAndCompactSequentiallyAsync|0()
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.CommitAsync(Nullable`1 endIndex, CancellationToken token) i[40/1934]s\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\MemoryBasedStateMachine.cs:line 293
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState.CommitAsync(Int64 endIndex, CancellationToken token) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 718
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftCluster.cs:line 465
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token)
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster.DotNext.Net.Cluster.Consensus.Raft.TransportServices.ILocalMember.Appen[28/1934]sync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftCluster.DefaultImpl.cs:line 272
   at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\Tcp\TcpServer.cs:line 246
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token)
   at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.ProcessRequestAsync(MessageType type, ProtocolStream protocol, CancellationToken token) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\Tcp\TcpServer.cs:line 177   at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\Tcp\TcpServer.cs:line 134
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)     [14/1934]   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
   at DotNext.Net.Cluster.Consensus.Raft.TransportServices.ConnectionOriented.ProtocolStream.ReadMessageTypeAsync(CancellationToken token) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\TransportServices\ConnectionOriented\ProtocolStream.ReadOperations.cs:line 93
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()                             [0/1934]   at System.Threading.ThreadPool.<>c.<.cctor>b__86_0(Object state)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _)
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags)
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Byte[] socketAddress, Int32 socketAddressSize, SocketFlags receivedFlags, SocketError socketError)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
Aborted

I caught another exception:
System.IO.InternalBufferOverflowException: System error.
         at DotNext.Buffers.SpanReader`1.Read(Int32 count) in /home/roman/projects/dotnext/src/DotNext/Buffers/SpanReader.cs:line 201
         at DotNext.IO.FileReader.Read[T]() in /home/roman/projects/dotnext/src/DotNext.IO/IO/FileReader.Binary.cs:line 22
         at DotNext.IO.FileReader.ReadAsync[T](CancellationToken token) in /home/roman/projects/dotnext/src/DotNext.IO/IO/FileReader.Binary.cs:line 42
         at RaftNode.SimplePersistentState.UpdateValue(LogEntry entry) in /home/roman/projects/dotnext/src/examples/RaftNode/SimplePersistentState.cs:line 75
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ApplyAsync(Int32 sessionId, Int64 startIndex, CancellationToken token) in /home/roman/projects/dotnext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 456
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass20_0.<<CommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /home/roman/projects/dotnext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 311
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /home/roman/projects/dotnext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 465
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /home/roman/projects/dotnext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 246
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /home/roman/projects/dotnext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /home/roman/projects/dotnext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 134

I'll fix that shortly.

The previous issue has been fixed.

Found a new one:
DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from <IP_ADDRESS>:42824
      System.InvalidOperationException: Unknown Raft message type 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /home/roman/projects/dotnext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 134

I'm working on it.

The root cause was incorrect skip of the last log entry when it is not consumed by the follower. Fixed and pushed to develop branch.

Found the root cause of failed assertion. Here is the steps to reproduce:

Append non-empty entry at index X with addToCache=true argument, but not commit it
Rewrite log entry at index X with the entry of size 0
Request the entry from WAL

Raft allows to rewrite uncommitted log entries only. This is by-design and described in Raft paper. In the described scenario, WAL uses internal cache when read operation is called. However, the actual content is stored on the disk with different length.
How to fix: invalidate cache at the specific index when rewrite happened.

All the previously shared scenarios are working now.
Additional testing shows these new scenarios:

(low priority) node starting again seems to always cause an election, which makes me wonder if prevoting is happening or some other unintended consequence of the start of that node
(critical) cutting power to 1 out of 3 nodes: the 2 alive nodes did not resume for 13+ minutes. Even then, it only recovered after the device was powered again, even though the raft node program was not started on the device (OS seems to explicitly reset the connection in this case)
broken log when taking out power, regardless of WriteThrough setting. Even if some alternatives might appear, the ease to reproduce could hint at some less unavoidable issue to be at play, so its at least worth exploration.

2 is fixed.

3rd issue. I've investigated WAL dumps from ARM devices. Fs (I think it was ext4) was unable to restore some of the partition files. That's why the index of the last log entry stored in the node.state file larger than the partition file. There is no silver bullet to solve this issue. However, there are few practices that we can use:

Persistent WAL supports live backups, see PersistentState.CreateBackupAsync and PersistentState.RestoreFromBackupAsync methods
On initialization, if the WAL is broken, we can just call ClearAsync and join as the follower. The latest cluster state will be replicated as soon as possible.

In worst case, the app can implement incremental backups. But I think the option 2 is enough for our purposes.

@freddyrios , you mentioned performance or read/write operations in WAL. Unfortunately, Linux historically did not have a good syscall for async disk I/O. It was many attempts to do that without any great success. However, in the last kernel version, io_uring syscall has been added. However, it is not supported by .NET: https://github.com/dotnet/runtime/issues/51985
It means that all asynchronous calls transformed to synchronous calls scheduled via ThreadPool.

@sakno closing this issue as done, as all the raised scenarios have been addressed. The only thing remaining is around node restarts usually triggering an election, but this is not a priority at the moment.


Add C# examples #70

I added lookup_blocks.cs and mute_a_user.cs. Let me know your thoughts. I can do many more 👍

recheck

recheck

Is anyone available to let me know what I'm missing here? Thank you in advance!


Add Go 1.11rc2

Add Go 1.11rc2
https://groups.google.com/d/topic/golang-announce/zb6p0aFMBJM/discussion
https://golang.org/dl/#go1.11rc2

https://github.com/syndbg/goenv/releases/tag/1.20.0 released :tada:


SQL error after creating new data type

Laravel Version: 5.3.30
Voyager Version: 0.10.10
PHP Version: 5.6.24
Database Driver & Version:
mysql

Description:
I create a new simple data type called Widgets. I am using 3 fields, ID, name and description. Saved table. then created BREAD and added the permission to my admin role. also added it to menu using menu builder. when I go to page with the correct slug admin/widgets I am getting this SQL error:
SQLSTATE[42S22]: Column not found: 1054 Unknown column 'created_at' in 'order clause' (SQL: select * from widgets order by created_at desc)
I did not chose TIMESTAMPS this time as I was having problems with timestamps on my other ticket with the Products table/BREAD I created.
Steps To Reproduce:

Doing some research today, Looks like at VoyagerBreadController line 46 it calling the latest. But the data type I created does not have timestamps. So I did a temporary fix changes this line:
$dataTypeContent = call_user_func([$model->with($relationships)->latest(), $getter]);
to this since there are no TIMESTAMPS
$dataTypeContent = call_user_func([$model->with($relationships), $getter]);
now my widgets main browser view works.
Of course this is a temporary fix, looks like their needs to be another else type statement with data types without any Timestamps????

You must add the following to the Widget model.
protected $timestamps = false;

thanks Mark, will do!
thanks for all of your help!

Always

Mark,
I just placed the code you suggested in the Widgest model and got an error stating that is has to be public.

So I just replaced protected to public and now it works?
Is this okay to keep it public??
protected $timestamps = false;
class Widget extends Model {     public $timestamps = false; }

Yea, of cause. My bad, should be public.


Fix minimum amount logic for BTC wallet

Found during testing:

getting these errors in the logs

WARN  Sending onAnimatedValueUpdate with no listeners registered.
WARN  Sending onAnimatedValueUpdate with no listeners registered.

BTC wallet test: shows 2501 as minimum amount, good. when I change the amount to lower than 2501 sats, then I get an error where the QR code should be, and a log error. Better error handling needed, maybe prevent the user from leaving the enter amount screen if they do not have >=  2501 sats

After receiving the 2501 sats, I see a balance of 1 sat. (this is expected, breez might take all the setup fee) so maybe we should put the minimum higher so they are not left with nothing as the first experience on non-custodial lightning.

When I go back to the receive button, it is still telling me I need to do a minimum of 2501 sats. Only when i close the app and open it again (or do a dev reload), do I see my transaction history, and only then I can press the receive button and get an invoice for 1 sat.

New error:

when trying to use the USD wallet, I'm getting an error saying "Amount must be at least $1.18". This should only apply to the BTC wallet when the BTC wallet transaction history is zero

@islandbitcoin Fixed the issue. Please check one more time

@islandbitcoin Fixed the issue. Please check one more time

getting closer. but when the default wallet is USD, it

still shows an error when getting a QR code for the BTC wallet.

@islandbitcoin

last thing:

Label change: Initial deposit of $5 recommended ...

Ship it!


error running asciidoctor-pdf-cjk-kai_gen_gothic-install

[1/20] Downloading KaiGenGothicCN-Bold-Italic.ttf
Traceback (most recent call last):
17: from /home/lzhou/bin/asciidoctor-pdf-cjk-kai_gen_gothic-install:23:in <main>' 	16: from /home/lzhou/bin/asciidoctor-pdf-cjk-kai_gen_gothic-install:23:in load'
15: from /home/lzhou/.gem/ruby/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:30:in <top (required)>' 	14: from /home/lzhou/.gem/ruby/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:30:in chdir'
13: from /home/lzhou/.gem/ruby/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:31:in block in <top (required)>' 	12: from /home/lzhou/.gem/ruby/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:31:in each_with_index'
11: from /home/lzhou/.gem/ruby/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:31:in each' 	10: from /home/lzhou/.gem/ruby/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:33:in block (2 levels) in <top (required)>'
9: from /home/lzhou/.gem/ruby/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:33:in open' 	 8: from /home/lzhou/.gem/ruby/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:34:in block (3 levels) in <top (required)>'
7: from /usr/share/ruby/open-uri.rb:35:in open' 	 6: from /usr/share/ruby/open-uri.rb:735:in open'
5: from /usr/share/ruby/open-uri.rb:165:in open_uri' 	 4: from /usr/share/ruby/open-uri.rb:224:in open_loop'
3: from /usr/share/ruby/open-uri.rb:224:in catch' 	 2: from /usr/share/ruby/open-uri.rb:226:in block in open_loop'
1: from /usr/share/ruby/open-uri.rb:755:in buffer_open' /usr/share/ruby/open-uri.rb:377:in open_http': 403 Forbidden (OpenURI::HTTPError)

same here.
Checking the script:
https://github.com/chloerei/asciidoctor-pdf-cjk-kai_gen_gothic/blob/master/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install
It downloads fonts from
https://github.com/chloerei/asciidoctor-pdf-cjk-kai_gen_gothic/releases/download/v0.1.0-fonts/
and put them into FONT_DIR.
I tried to download manually, but above page returns 404.

Sorry, This gem is no longer maintained, I suggest follow the asciidoctor-pdf theme guides to use your local font.
https://github.com/asciidoctor/asciidoctor-pdf/blob/v1.5.0.beta.7/docs/theming-guide.adoc#custom-fonts
If you want to manually complete the install script, your can found fonts here: https://github.com/chloerei/asciidoctor-pdf-cjk-kai_gen_gothic/releases/tag/v0.1.0-fonts

https://github.com/life888888/asciidoctor-pdf-cjk-ext/releases/tag/v0.1.0
Fonts :

Noto Sans CJK
Noto Sans Mono CJK
Noto Serif CJK

Font Style:

Regular
Bold
Bold Italic
Italic

Support language:

Simplified Chinese   sc
Traditional Chinese  tc
Hong Kong hk
Japanese jp
Korean kr

If you want Noto Serif CJK, and JP (Japanese) , download notoserif-cjk-jp.zip
If you want Noto Sans Mono CJK, and TC (Traditional Chinese)  , download notosansmono-cjk-tc.zip
If you want Noto Sans CJK, and SC (Simplified Chinese) , download notosans-cjk-sc.zip
Support Theme list is here:  https://github.com/life888888/asciidoctor-pdf-cjk-ext/blob/main/ThemeList.adoc
Every zip include 6 theme (For Asciidoctor PDF 2.x Theme , For Asciidoctor PDF 1.x Theme)
Naming Rule is:

default-ext-notosans-cjk-xx
default-notosans-cjk-xx
notosans-cjk-xx-theme

For Asciidoctor PDF 2.x Theme:

default-ext-notoFONTXXX-cjk-xx
default-notoFONTXXX-cjk-xx
notoFONTXXX-cjk-xx

For Asciidoctor PDF 1.x Theme

default-ext-notoFONTXXX-cjk-xx-pdf-1
default-notoFONTXXX-cjk-xx-pdf-1
notoFONTXXX-cjk-xx-pdf-1

Font List

Theme List

https://github.com/life888888/asciidoctor-pdf-cjk-ext/blob/main/install-zip-to-downloaded-resources-dir.sh
provide download shell script but only for linux, bash.
execute ./install-zip-to-downloaded-resources-dir.sh sc
will auto download SC (Simplified Chinese) - ( sans, snasmono, serif )

Download Noto CJK Fonts TTF
https://github.com/life888888/cjk-fonts-ttf#download-fonts


Update password rules (2023-06-23)

Updating password rules from remote source, pulled on 2023-06-23

Closing this off and aiming to land https://github.com/duckduckgo/duckduckgo-autofill/pull/338 today so we'll get a green build tomorrow


fix(JSON): a panic could occur when using All+JSON+embeddedOp+$1

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-2.0e**-05%**) to 99.852%

Totals

Change from base Build<PHONE_NUMBER>:
-2.0e-05%

Covered Lines:
9437

Relevant Lines:
9451

💛  - Coveralls


Fixing CI test failures

Added a torch wrapper for trace (torch.trace doesn't support the offset argument or stacking)
Wrap numpy, cupy, and torch nonzero to raise an error for zero-dimensional input arrays.

I don't believe it, but the CI is actually green.

Nice work!


Moff cooking. Part one.

About the PR

Add baking for moth.
Cotton dough = 5 flour + 10 water + 10 fabric
Bread = Cotton dough + 10 sec microwave
Pizza = Flat cotton dough + 4 cotton bolls + 30 sec microwave
Why / Balance

Moff rejoice!
Technical details

no
Media

https://github.com/user-attachments/assets/a09d5644-abb0-4a84-b475-5eeaa8c793d0
Requirements

[X] I have read and I am following the Pull Request Guidelines. I understand that not doing so may get my pr closed at maintainer’s discretion
[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Changelog

:cl:

add: Cotton dough added to the game! Check the guidebook for new recipes.

Could alternatively make it from fiber as a sort of mothy flour replacer, and add fiber bags to the chef's vendor/lockers.

Could alternatively make it from fiber as a sort of mothy flour replacer, and add fiber bags to the chef's vendor/lockers.

nah, chef can butch some clothing or request cotton from botany

This is nice. I like this.

if it possible, can you make it flammable?

possible, but i don't want it

moth rejoice!


<IP_ADDRESS> Database listener not 1521

In the <IP_ADDRESS> VM the database listener is not listening on port 1521.

Fixed.


Track state changes with a has many association

Previously, I added named timestamps for each state.  After discussion
with Paul, it seemed more sensible to change this to use a has many
association where each change is tracked indiviudally. We can use the
created_at timestamp from VisitStateChange to calculate processing
times and this approach allows for new states to be added.

Looks fine to me.


Fix theoretical corruption after fsync failure

See #445 and #299
Summary of my understanding of the issue:

read() and reads from mmap()'ed memory may return any value that was written since the last successful fsync()/msync() and cannot be assumed to return the latest value. The granularity is an OS page, so values that cross pages may be torn
if fsync() reports a failure (even in a different process), all pages written since the last successful fsync() must be assumed to have become poisoned and they must be explicitly re-initialized by writing to them.

Corollaries:

constructing a slice from an mmap'ed pointer is only safe if that memory is clean (successfully fsync()'ed). Otherwise, it is immediately UB
reads of dirty data (not yet fsync'ed) must be served from a buffer in userspace.

Using O_DIRECT might get around some of these issues, because it avoids the page cache. However, using it with mmap likely had undefined behavior.

From #445:

For example, all three file systems mark pages clean after fsync fails, rendering techniques such as application-level retry ineffective.

This is quite weird to me. If the pages were still marked dirty, the application could just retry the fsync and if it succeeded all would be well. This makes me wonder if marking the pages clean somehow simplifies the implementation, makes it faster, etc.

However, the content in said clean pages varies depending on the file system; ext4 and XFS contain the latest copy in memory, while btrfs reverts to the previous consistent state.

I suppose that on-disk state is totally undefined, so you have to assume it's some mix of pre and post-write blocks. For 2PC, can't you just return an error from the commit after a failed fsync? But also, if there's an fsync in the middle of a transaction (not just on commit) then any outstanding references in rust code would be invalid, so that would be insta UB.

I'm pretty sure this means the 1PC+C strategy won't work, because if fsync can flush some blocks but rollback others it could flush the transaction block but rollback some/all of the data blocks :/

Yah, yikes.

This is quite weird to me. If the pages were still marked dirty, the application could just retry the fsync and if it succeeded all would be well. This makes me wonder if marking the pages clean somehow simplifies the implementation, makes it faster, etc.
Ya, the Postgres devs thought that too :) I watched this talk about it, which is pretty good. Ya, I found the answer to this, which is that the most common case is people pulling out USB drives, and so keeping the dirty pages around would lead to a permanent memory leak since the pages can't be flushed without the drive there.

The thing that doesn't make sense to me though is why the pages are marked clean and kept in the page cache. It seems like they should be evicted from the page cache to avoid poisoning it.

But also, if there's an fsync in the middle of a transaction (not just on commit) then any outstanding references in rust code would be invalid, so that would be insta UB
redb only fsync's at the end of a transaction, but ya I'm starting to wonder whether mmap'ed memory can be arbitrarily rolled back if there's memory pressure and flushing to disk fails. From that LWN article it looks like the Postgres devs were encouraged to use O_DIRECT and never use the page cache.

For 2PC, can't you just return an error from the commit after a failed fsync?
I don't think that's sufficient. If the on-disk data and page cache are silently out of sync, you have to assume all written pages are poisoned and need to be explicitly zero'ed so that the page cache is back in sync with the on-disk data. Even worse, there's discussion about how you can't even assume the page cache is working correctly when you first open() a file, because some other program could have had a failure and poisoned it. For example, using cp to copy a redb database file, having a failure, and then opening it with redb could be UB :exploding_head:

page_cache_poisoning.zip
Here's a test showing that it's trivial to poison the page cache. At a minimum Database::open()'s SAFETY section needs to document that the page cache for the database file must be in a good state

Ah, it looks like posix_fadvise(DONTNEED) can be used to clear out the page cache. The Linux implementation calls into invalidate_mapping_pages here: https://elixir.bootlin.com/linux/v6.0.10/source/mm/fadvise.c#L168

but ya I'm starting to wonder whether mmap'ed memory can be arbitrarily rolled back if there's memory pressure and flushing to disk fails.

That seems like it just can't be true. If it was, how could using mmapped memory ever be safe? (On the other hand, so much is broken and weird so who knows.)

fuse doesn't seem to support mmap (or maybe the fuser crate just doesn't), so this doesn't 100% prove it, but this shows that you can definitely poison the page cache so that read() and read(O_DIRECT) return different values, and that you can change the value returned by read() by flushing the page cache. Since mmap uses the page cache, it seems highly likely that mmap'ed memory can mutate during a page flush.
page_cache_poisoning.zip


[PERF] Parallelize reduction for CPU

Can significantly reduce the latency for ops like mean and variance, but also increase latency for small reduction ops such as sum, max, etc. However these small ops are fast anyway. So I think it's worth to parallelize the reduction by default.
Will add some benchmark result soon...
@anijain2305 @Huyuwei @kevinthesun

Just to add, this is essential for QNN. QNN adds many sum operators on large tensors. Using this, I can cut down e2e latency by 40-50%.

@icemelon9 Did you get a chance to work on the comments?

ping @icemelon9

Ping @icemelon9 :)

ci is green now. @anijain2305 @kevinthesun @soiferj Could you review this again?

Thanks @icemelon9 @anijain2305 @soiferj @kevinthesun


doc(Data/Fin/Tuple/Basic): Compare start/end/middle addition in module doc

Definitions around adding an entry to a Fin-indexed tuple are a mess. This PR documents the mess so that it's easier to fix in a later PR.

Do you think I should update the docstrings of the corresponding declarations to include information similar to what I'm putting in the module doc?

This is helpful, thanks!
bors merge