feat(cdc): in-memory backfill for single mysql cdc table

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
This PR is the first part of the https://github.com/risingwavelabs/rfcs/pull/63. It implements a CdcBackfillExecutor to take care of the backfill logic for a CDC table. Right now, we will instantiate a CdcBackfillExecutor to wrap a source executor so that we can use the changelog events to backfill the table.
Main changes are as follows:

Add a session variable cdc_backfill to gate the this new feature
Add a StreamChunkMeta field to the StreamChunk for storing offsets of change log events, because the backfill algorithm needs to filter events based on offset (snapshot chunk doesn't have offsets)
ExternalTableReader component for reading snapshot and binlog from upstream table, this PR implements a table reader for MySQL
Frontend will store external table information  as a field in StreamSourceInfo in the process of CREATE TABLE

TODO:

This PR supports backfill for the MySQL table, Postgres support will be done in another PR
Records backfill offset so that we can resume the backfill progress upon recovery
~- Rewrite primary key comparison conditions for MySQL~
Support more MySQL data types

tracked: https://github.com/risingwavelabs/risingwave/issues/11079
Checklist

[x] I have written necessary rustdoc comments
[x] I have added necessary unit tests and integration tests
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[x] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

FYI: The performance of ingestion has been greatly improved, 6M rows lineitem table can be ingested into RW within 1 minute.

Would you please help to update the PR body based on recent changes? 🥰

Add a session variable cdc_backfill to gate the this new feature

Seems like it is set to true by default https://github.com/risingwavelabs/risingwave/pull/11707/files#diff-4e19861816f5cf913890db3abd8088d01fa135d85f43080d5e6ee5c6c8ac40aaR337
Is that expected?

Add a session variable cdc_backfill to gate the this new feature

Seems like it is set to true by default https://github.com/risingwavelabs/risingwave/pull/11707/files#diff-4e19861816f5cf913890db3abd8088d01fa135d85f43080d5e6ee5c6c8ac40aaR337
Is that expected?

Yes.


Tonel

Migration to Tonel (as a requirement for processing updates from VisualWorks 8.2)

This conversion happened elsewhere.  Appreciate the contribution though.


Correct Swift Decimal.divide to call NSDecimalDivide.

What's in this pull request?

Correct Swift Decimal.divide to call NSDecimalDivide.
Test case for Decimal.divide and Decimal.multiply.

I'm not sure that this is the best place or way to perform this test but I'm also not entirely clear on what the existing tests are testing.
Resolved bug number: (SR-)
None

Before merging this pull request to apple/swift repository:

[ ] Test pull request on Swift continuous integration.

Triggering Swift CI
The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:
Smoke Testing
    Platform     | Comment
    ------------ | -------------
    All supported platforms     | @swift-ci Please smoke test
    All supported platforms     | @swift-ci Please smoke test and merge
    OS X platform               | @swift-ci Please smoke test OS X platform
    Linux platform              | @swift-ci Please smoke test Linux platform

A smoke test on macOS does the following:

Builds the compiler incrementally.
Builds the standard library only for macOS. Simulator standard libraries and
device standard libraries are not built.
lldb is not built.
The test and validation-test targets are run only for macOS. The optimized
version of these tests are not run.

A smoke test on Linux does the following:

Builds the compiler incrementally.
Builds the standard library incrementally.
lldb is built incrementally.
The swift test and validation-test targets are run. The optimized version of these
tests are not run.
lldb is tested.

Validation Testing
    Platform     | Comment
    ------------ | -------------
    All supported platforms     | @swift-ci Please test
    All supported platforms     | @swift-ci Please test and merge
    OS X platform               | @swift-ci Please test OS X platform
    OS X platform               | @swift-ci Please benchmark
    Linux platform              | @swift-ci Please test Linux platform

Lint Testing
    Language     | Comment
    ------------ | -------------
    Python       | @swift-ci Please Python lint

Note: Only members of the Apple organization can trigger swift-ci.

Probably worth mentioning that I came across this bug in XCode Version 8.0 beta 3 (8S174q). I thought that I might have lost it for a second there but... I'm still apparently mostly sane 😝 .

Thanks @mjmdavis !

@swift-ci Please test and merge

The OS X test passed and this code is not present on Linux, so I'm going to merge.


GLUE Script for Tensorflow

Codecov Report

Merging #1329 into tf2 will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              tf2    #1329      +/-   ##
==========================================
- Coverage   86.01%   85.95%   -0.06%
==========================================
  Files          79       79
  Lines       12041    12028      -13
==========================================
- Hits        10357    10339      -18
- Misses       1684     1689       +5

Impacted Files
Coverage Δ

pytorch_transformers/modeling_utils.py
91.44% <0%> (-1.48%)
:arrow_down:

pytorch_transformers/tests/modeling_common_test.py
73.61% <0%> (-0.41%)
:arrow_down:

...orch_transformers/tests/modeling_tf_common_test.py
94.73% <0%> (-0.27%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8e956d...cc73950. Read the comment docs.


HNS Endpoint DNSSuffix not taking effect

Environment:
PS C:\> (Get-WmiObject Win32_OperatingSystem).Version
10.0.17763

My team is working on kubernetes CNI support for windows container, and we need to add a DNS suffix search list to container network. By passing the DNS search list to DNSSuffix, seems we configured the HNS endpoint correctly:
PS C:\Users\bosh_e6eb4f2e0bd3484> get-hnsendpoint

ActivityId                : 03C31E90-A258-4807-B41B-73DA844A4490
AdditionalParams          :
CreateProcessingStartTime :<PHONE_NUMBER>15722566
EncapOverhead             : 0
GatewayAddress            : <IP_ADDRESS>
Health                    : @{LastErrorCode=0; LastUpdateTime=132457439315554069}
ID                        : 72A0776A-BB47-4F98-8953-2A738F4BAF4D
IPAddress                 : <IP_ADDRESS>
MacAddress                : 00-50-56-96-00-0E
Name                      : agent0
Policies                  : {}
PrefixLength              : 30
Resources                 : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; Health=;
                            ID=03C31E90-A258-4807-B41B-73DA844A4490; PortOperationTime=0; State=1;
                            SwitchOperationTime=0; VfpOperationTime=0; parentId=7B252054-1D2D-40A7-941B-3030776A9E28}
SharedContainers          : {}
StartTime                 :<PHONE_NUMBER>28228959
State                     : 2
Type                      : Transparent
Version                   :<PHONE_NUMBER>7
VirtualNetwork            : a1ae87fc-5ecf-414a-b464-fd638455a075
VirtualNetworkName        : transit

ActivityId                : A5308748-62DA-4746-AA67-D65CFF1F3CE9
AdditionalParams          :
CreateProcessingStartTime :<PHONE_NUMBER>37743593
DNSServerList             : <IP_ADDRESS>
DNSSuffix                 : svc.cluster.local,cluster.local
EncapOverhead             : 0
GatewayAddress            : <IP_ADDRESS>
Health                    : @{LastErrorCode=0; LastUpdateTime=132457502837743593}
ID                        : CAE5A21C-B068-45F2-9446-72A9D307F5E2
IPAddress                 : <IP_ADDRESS>
MacAddress                : 04-50-56-00-e8-3d
Name                      : 9a72f1aa15fa0bd06795406772d2bd090c3cf469be3e1df4a4106312106a8d3b_eth0
Policies                  : {}
PrefixLength              : 24
Resources                 : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; Health=;
                            ID=A5308748-62DA-4746-AA67-D65CFF1F3CE9; PortOperationTime=0; State=1;
                            SwitchOperationTime=0; VfpOperationTime=0; parentId=7B252054-1D2D-40A7-941B-3030776A9E28}
SharedContainers          : {9a72f1aa15fa0bd06795406772d2bd090c3cf469be3e1df4a4106312106a8d3b,
                            7a095a174a5b5ff47bf2cb8dd4bc7816edfee195bd578ef2d74f
...

As we can see above, DNSSuffix is set to svc.cluster.local,cluster.local. The in-cluster DNS service (kube-dns with virtual IP <IP_ADDRESS>) can resolve FQDN like kubernetes.default.svc.cluster.local as following:
PS C:\> nslookup kubernetes.default.svc.cluster.local
Server:  kube-dns.kube-system.svc.cluster.local
Address:  <IP_ADDRESS>

Name:    kubernetes.default.svc.cluster.local
Address:  <IP_ADDRESS>

We expect that we can perform DNS lookup for kubernetes.default within container. Unfortunately it didn't work as expected:
PS C:\> nslookup kubernetes.default
Server:  kube-dns.kube-system.svc.cluster.local
Address:  <IP_ADDRESS>

*** kube-dns.kube-system.svc.cluster.local can't find kubernetes.default: Non-existent dom
ain

From the error message, the windows container made attempt to directly resolve kubernetes.default where we expect it to also make attempts for resolving kubernetes.default.svc.cluster.local.
From the container, we also saw that SearchList property in registry key HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters is not configured:
PS C:\> Get-itemproperty -path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SearchList
Get-itemproperty : Property SearchList does not exist at path
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.
At line:1 char:1
+ Get-itemproperty -path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (SearchList:String) [Get-ItemProperty], PSArgumentException
    + FullyQualifiedErrorId : System.Management.Automation.PSArgumentException,Microsoft.PowerShell.Commands.GetItem
   PropertyCommand

We manually ran following commands in the container powershell:
PS C:\> $arguments = @('nicconfig','call','SetDNSSuffixSearchOrder',"('default.svc.cluster.local','svc.cluster.local','
cluster.local')")
PS C:\> wmic $arguments

Which effectively set the registry value as mentioned above:
PS C:\> Get-itemproperty -path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SearchList
SearchList   : default.svc.cluster.local,svc.cluster.local,cluster.local
PSPath       : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Paramet
               ers
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip
PSChildName  : Parameters
PSDrive      : HKLM
PSProvider   : Microsoft.PowerShell.Core\Registry

And then, we can resolve kubernetes.default as expected:
ubuntu@opsmgr-06-slot-50-pez-vmware-com:~$ kubectl exec test-win-64f858c7f9-zntbj -- nslookup kubernetes
Server:  kube-dns.kube-system.svc.cluster.local
Address:  <IP_ADDRESS>

Name:    kubernetes.default.svc.cluster.local
Address:  <IP_ADDRESS>

Our questions are:

Given the nature of k8s CNI design, we cannot perform intrusive modifications within windows containers. All what we can do is to pass configuration during the creation of HNS endpoint. Is DNSSuffix the parameter to pass the DNS search list or we have the wrong expectation?
Is the format (comma separated string) of DNSSuffix correct? From the code we're under impression that it should be a comma separated string.

Also please let us know if you got a confirmation that this is an implementation issue on HNS endpoint, thanks!

@daschott, can you please take a look at this issue
cc: @marosset

@daschott is out of the office right now. I will take care of it.

We are investigating the issue right now. Will update as soon as we have news

@hanlins
DNS Resolver on Windows treats any domain name with a . in it as if it was FQDN:
https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#dns-limitations
This is a platform difference that may be possible to investigate. Can you elaborate more details explaining why resolution for kubernetes.default is needed?
As for setting domain, can you try setting "--cluster-domain" flag in kubelet? On top of this, depending on CNI plugin you'd need to add the suffices to the CNI configuration file as well (or pod spec if you are passing DNS information that way). By default it will only have "svc.cluster.local" inside it
E.g. in win-bridge or win-overlay you can add field "dns" in the CNI configuration:
"dns":  {
"Nameservers":  [
"<IP_ADDRESS>"
],
"Search":  [
"svc.cluster.local"
]
},
or if you enabled DNS in the CNI plugin by setting DNS capabilities:true, you should be able to pass DNS configuration via pod spec as well.

Hi @daschott thanks for getting back! I see, that's interesting, we haven't checked the windows specific doc you mentioned, so I think this issue is prevalent for all current CNI solutions. We found the issue in both our proprietary implementation as well as Antrea windows support that the clusters failed the sig-windows conformance tests because of this limitation. I can check with my team and let them provide the details on the failed tests. Also maybe @ydp can help on retrieving the detailed CNI spec we passed to the cluster.
Besides that, you mentioned there's a inconsistency between windows and linux that windows treats anything contains a '.' as a FQDN, is this intentional or this is something to be fixed? Is it possible to expose an option to change this behavior(so windows will also make attempts for appending the suffixes to FQDNs) during the creation of HNS endpoint?

Hi @daschott , I will try to setup a new env and get more info here. In the mean time, can you help explain that windows only have 1 DNS suffix with pod namespace? Does this suffix need we pass in or the container automatically find this suffix? Since from our tests for antrea, we already passed  $namespace.svc.cluster.local in, but the query never try to append any suffix at all, which is why our CNI plugin cannot pass conformance test.

      : No
   Autoconfiguration Enabled . . . .  : Disabled
   Connection-specific DNS Suffix Search List :
                                       cluster.local
                                       default.svc.cluster.local
                                       svc.cluster.local

You could see we have default.svc.cluster.local passed in.
Lookup FQDN success:
kubectl exec iis-site-windows-5769b547f5-6fwc2 -- nslookup kubernetes.default.svc.cluster.local
Server:  kube-dns.kube-system.svc.cluster.local
Address:  <IP_ADDRESS>

Name:    kubernetes.default.svc.cluster.local
Address:  <IP_ADDRESS>

Lookup PQDN failed:
win2$ kubectl exec iis-site-windows-5769b547f5-6fwc2 -- nslookup kubernetes
Server:  kube-dns.kube-system.svc.cluster.local
Address:  <IP_ADDRESS>

*** kube-dns.kube-system.svc.cluster.local can't find kubernetes: Non-existent domain

It's looking for kubernetes, not kubernetes.default.svc.cluster.local.
Could you please help point where is wrong in our configuration?

@ydp thanks for following up - have you tried using resolve-dnsname cmdlet?
The thing about nslookup is that it doesn’t use the Windows DNS client resolver. It should not be used for testing DNS connectivity on Windows.
https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#dns-limitations

Hi @daschott  thanks for pointing out the cmdlet, I have tried, seems it works fine with our latest NCP fix, but nslookup not. Here is my result:
kubo@jumper:~/win2$ kubectl exec iis-site-windows-5769b547f5-6fwc2 -- powershell.exe Resolve-DnsName -Name kubernetes

Name                                           Type   TTL   Section    IPAddres
                                                                       s
----                                           ----   ---   -------    --------
kubernetes.default.svc.cluster.local           A      5     Answer     10.100.2
                                                                       00.1

kubo@jumper:~/win2$ kubectl exec iis-site-windows-5769b547f5-6fwc2 -- powershell.exe Resolve-DnsName -Name kubernetes.default.svc.cluster.local

Name                                           Type   TTL   Section    IPAddres
                                                                       s
----                                           ----   ---   -------    --------
kubernetes.default.svc.cluster.local           A      5     Answer     10.100.2
                                                                       00.1

kubo@jumper:~/win2$ kubectl exec iis-site-windows-5769b547f5-6fwc2 -- powershell.exe Resolve-DnsName -Name my-nginx

Name                                           Type   TTL   Section    IPAddres
                                                                       s
----                                           ----   ---   -------    --------
my-nginx.default.svc.cluster.local             A      5     Answer     10.100.2
                                                                       00.30

kubo@jumper:~/win2$ kubectl exec iis-site-windows-5769b547f5-6fwc2 -- powershell.exe Resolve-DnsName -Name my-nginx.default.svc.cluster.local

Name                                           Type   TTL   Section    IPAddres
                                                                       s
----                                           ----   ---   -------    --------
my-nginx.default.svc.cluster.local             A      5     Answer     10.100.2
                                                                       00.30

Can you share some insights about the difference between nslookup and Resolve-DnsName?

resolve-dnsname is representative of DNS resolution used by vast majority of applications and other tools. Just don't use nslookup and you should see consistent results everywhere else, including Invoke-Webrequest, curl, .NET apps, IIS apps, resolve-dnsname, etc.

As outlined by @daschott nslookup might fail where resolve-dnsname will succeed. I am closing this issue as there is a workaround.


How does Spatial MLP implement muti-head?

How does Spatial MLP implement muti-head? I don't understand this, isn't this just a conv1d? Why does it say: use group convolution to implement multi-head MLP?

How does Spatial MLP implement muti-head? I don't understand this, isn't this just a conv1d? Why does it say: use group convolution to implement multi-head MLP?

We find that using a linear mixer layer performs better than a 2-layer MLP. It is also faster.


(tiny): Fix edge overlay issue

Fixes edge overlay issue mentioned in #49 by simply bringing the finder container up in the vertical axis.
Before

After

Nice, was about to open this PR. 😄

It would be nice but unfortunately this package is practically abandoned. There hasn't been a update in over 2 years. @shahnawaz please approve this!

Nice, was about to open this PR. 😄

It would be nice but unfortunately this package is practically abandoned. There hasn't been a update in over 2 years. @shahnawaz please approve this!

I'm starting to think this won't go anywhere.
On another note, I've noticed that on android the problem is still there, solved it by adding elevation: 1 in the same place


This PR fixes the error, by providing placeholder fileNames to attachments when they do not have theirs.

thanks for this @mykeels! Sorry it took me this long to look at this :-/

thanks for this @mykeels! Sorry it took me this long to look at this :-/


Use go 1.18 to build artifacts

This should not merge until we are ready for all armada consumers to use go 1.18.
Issue #1142

Nice catch; yes I do. Probably should run go mod tidy, too.
Gonna be honest; I didn't expect this change to pass tests so I didn't jump through all the needed hoops yet :laughing:

Once https://github.com/G-Research/armada/pull/1276 merges, this rebased on top of it should pass tests.

This should not merge without explicit approval and timing from GR.

LGTM- let's coordinate with @JamesMurkin to switch the ci over to requiring 1.18 to pass

The work has already been done on the public side; if there's anything I can do to help with other CI systems please just ask.


fix: RewardBallotWinners determinism

Description
Fixes RC1 Canon consensus failure.

Author Checklist
All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.
I have...

[ ] included the correct type prefix in the PR title
[ ] added ! to the type prefix if API or client breaking change
[ ] added appropriate labels to the PR
[ ] targeted the correct branch (see PR Targeting)
[ ] provided a link to the relevant issue or specification
[ ] added a changelog entry to CHANGELOG.md
[ ] included comments for documenting Go code
[ ] updated the relevant documentation or specification
[ ] reviewed "Files changed" and left comments if necessary
[ ] confirmed all CI checks have passed

Reviewers Checklist
All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.
I have...

[ ] confirmed the correct type prefix in the PR title
[ ] confirmed all author checklist items have been addressed
[ ] reviewed state machine logic
[ ] reviewed API design and naming
[ ] reviewed documentation is accurate
[ ] reviewed tests and test coverage
[ ] manually tested (if applicable)

approved +1 was this introduced in #1391 or was this in the oracle previously / should we have seen that in the latest beta?

No. #1381 didn't touch RewardBallotWinners. Basically, the bug is that we iterate over a map for k.distrKeeper.AllocateTokensToValidator  to send distributions in an AVL tree. I will make it more clear in few secs

Map bugs are elusive - they'll only reveal themselves if the underlying slices were allocated differently, and the conditions when that will appear in practice are undefined.
Probably didn't appear in betas due to the small number of validators, operating under similar conditions

Codecov Report

Merging #1433 (19e1e01) into main (1f8bc6f) will increase coverage by 0.00%.
The diff coverage is 66.66%.

@@           Coverage Diff           @@
##             main    #1433   +/-   ##
=======================================
  Coverage   52.80%   52.81%
=======================================
  Files          70       70
  Lines        6751     6752    +1
=======================================
+ Hits         3565     3566    +1
  Misses       2916     2916
  Partials      270      270

Impacted Files
Coverage Δ

ante/fee.go
78.35% <33.33%> (ø)

app/upgrades/v3/min_commission.go
85.71% <100.00%> (ø)

x/oracle/client/tests/suite.go
98.38% <100.00%> (ø)

x/oracle/keeper/reward.go
86.36% <100.00%> (+0.31%)
:arrow_up:

ohhh, so in the sum part it didn't matter that much, because sum the whole amount in map or slice is the same thing,
but when we iterate again here to send rewards, it matters

@RafilxTenfen correct.


tools/docker: add various fsck-like commands to the environment

As part of #5518, I'm adding fsck logs as annotation to the mounted file system assets. For this, I need a variety of fsck-like commands in the ci environment as well as eventually in the production environment.

$ docker build --no-cache -t gcr.io/syzkaller/env tools/docker/env
< ... >
E: Package 'hfsprogs' has no installation candidate

Does it work on your side?

Let's skip hfs for now.

Done

Deployed gcr.io/syzkaller/env -- you can now try to check whether CI tests began to work for the other PR.

Yes all tests on that PR pass now :)


EPD test suite

are the EPDs taken randomly or serially from ogpt_chess_startpos.epd? (ie same positions for each budget or random?)
are the games back to back (eg one side plays black, then reverse colours same epd?)

are the EPDs taken randomly or serially from ogpt_chess_startpos.epd? (ie same positions for each budget or random?)

It is random per budget. order=random
https://github.com/fsmosca/Lakas/blob/de04653e09ab40fc9087b0a06201fc68d7c508f3/lakas.py#L346
It can happen that a position in one budget may appear again in another budget.

are the games back to back (eg one side plays black, then reverse colours same epd?)

Yes, repeat 2
https://github.com/fsmosca/Lakas/blob/de04653e09ab40fc9087b0a06201fc68d7c508f3/lakas.py#L343

This change is now in master.


Deprecated SetListInterface in WorksomeSetList.php

Hey, how’s everything going? I wanted to leave this issue here: in the latest Rector update, they removed an interface that’s currently in use in WorksomeSetList.php
https://github.com/rectorphp/rector-src/pull/6590/files

Regars! Happy New Year's Eve!

Hi @sojeda, I will fix this on the 6th when I'm back to work (currently away from my laptop). Thanks for the heads up! If you need it sooner, feel free to PR a fix.


PR - Run Go tests

Added unit tests as part of the PR build checks

I'll wait for the working tests from the other branches to be merged before going forward with this PR


Add ECS CMIP6 Pangeo Example

Add an example of using funnel with CMIP6 ECS calculations

Closes #55

I addressed the different comments/suggestions, adding quite a bit more documentation, docstrings, etc.

A suggestion for cleaning up get_collection_values() and get_collection_keys(): can you replace them with
@task
def get_task_attr(obj, attr):
	return getattr(obj, attr)

and then
values = get_task_attr(collection_of_datasets, "values")
keys = get_task_attr(collection_of_datasets, "keys")

This generalizes the functions so that you can specify an arbitrary attribute rather than only being available for values and keys, and also removes the assumption that the object is an xcollection

A suggestion for cleaning up get_collection_values() and get_collection_keys()...

This may already exist in prefect -- it's hard to tell based on the documentation, and I haven't found any examples of other people using this... the wording "Helper task that retrieves a specific attribute of an upstream task's result" makes me think that it might not be.
I think that would be the cleanest implementation, though, and might even be worth a PR to prefect if the existing function isn't right for us.

Alrighty - this should be good to go! Feel free to have a look @andersy005 @mnlevy1981


feat: support native PostgreSQL logging format

Someone may need to parse PostgreSQL logs and do some processing. For example, cnpg reads PostgreSQL logs in csv format and pass them to users in JSON format.
Background worker logs do not follow PostgreSQL configuration. We need to implement a logger that follows PostgreSQL configuration.
Reference for contributors:

https://cloudnative-pg.io/documentation/1.22/postgresql_conf/#log-control-settings
https://www.postgresql.org/docs/current/runtime-config-logging.html

Although I'm new to both rust and pgvecto.rs, I wanted to try the following to complete this issue. By referring to the documentation and issue #376, as well as my own experiments, I temporarily learned:

At this stage, bgwork ignores all errors in the session function.
https://github.com/tensorchord/pgvecto.rs/blob/a1e79bba571a79c574996f0e5d8b98b8dfdf8763/src/bgworker/normal.rs#L18
pgrx supports collecting painc and outputting the corresponding format according to the configuration of log_destination.
Therefore, errors in the bgwork session function should be thrown with painc, so that pgrx can output to the target file in json/csv/.. format.

So I think that the error collection requirements in issue #376 can be met first by throwing the fn session errors. However, it should be noted here that the session is in a child thread.
I don’t know if there is any deviation in my understanding, or I would like to know more about implement a logger that follows PostgreSQL configuration. Is there a more detailed description to help me understand the requirements?

We use log to print logs to stderr in the background process. However, this crate does not control the output format of logs. env_logger implements trait.Log to do actually work (format and print these logs). What we need is to write another implementation of Log to respect the config about logging format in postgresql.conf.

I have done some testing recently and have the following thoughts on the implementation of logs.
Current situation
The current log implementation is to redirect the logs collected by env_logger to stderr. When log_destination is set to stderr and logging_collector is on. if an error occurs in bgworker, it can be successfully collected. But when log_destination is jsonlog, the current solution is still to redirect the log to the .log log file. And even if the log::error! function string is in json format, it still cannot be redirected to the .json log file.
Implementation plan
plan 1:
If I directly use the macros provided by pgrx, such as log! and error!, these functions can directly accept error messages, and pgrx will automatically output the corresponding format according to the postgres configuration.
plan 2:

Read the log configuration in postgres conf;

Format msg according to configuration, but all fields may need to be defined by myself. For example, the following is the postgres json log format I obtained.

{"timestamp":"2024-03-24 21:43:58.486 CST","pid":37650,"session_id":"66002cf0.9312","line_num":4,"session_start":"2024-03-24 21:38:56 CST","txid":0,"error_severity":"LOG","message":"checkpoint complete: wrote 18 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.745 s, sync=0.003 s, total=1.765 s; sync files=15, longest=0.002 s, average=0.001 s; distance=41 kB, estimate=41 kB","backend_type":"checkpointer","query_id":0}

Output the log information. If it is stderr, I can still use the current method to write the log; if it is json/csv, I need to manually read the current target log file path and write log msg to it. But I have not verified the feasibility of reading the file log path and writing.

I hope you can give me some more suggestions on solutions.

If I directly use the macros provided by pgrx, such as log! and error!

You cannot do it. pgrx macros call PostgreSQL functions, which are not thread-safe^1. However, the background process takes advantages of multithreading: logs come from all threads. If you try it, I think pgrx will print an error and abort the process.

the current solution is still to redirect the log to the .log log file

It's because we do not follow the protocol requested by PostgreSQL. You can follow the comments in the code.

Format msg according to configuration, but all fields may need to be defined by myself.

Yes, we must do it.

Currently completed

Read the log_destination configuration
Support writing log messages(only message) to stderr
Support writing log messages(only timestamp & message) to jsonlog
Implemented a preliminary logger
WIP

Some Idea
I plan to implement the writing of stderr, jsonlog and csvlog in the first stage, and then the fields will first support timestamps, pid, and message. At the same time, let VectorLogger support filter to be compatible with the existing writing methods in this repo.
In the second stage, all supported fields in pg log are gradually added.
Question
pipe protocol need pid, No idea how to get this pid. I also hope you can give me some suggestions on implementation ideas.

No idea how to get this pid.

https://doc.rust-lang.org/std/process/fn.id.html should help.


Images are served via http

What I'm trying to achieve
Store runs on https but images are loaded via http
What I expected to happen
Images load via https

Hi, same issue here. Did you fix it?

A workaround is to store the images on S3. https://docs.saleor.io/docs/developer/running-saleor/s3/

Images should be hosted on a dedicated cloud platform (like s3 from AWS) which delivers https connection.
More info: https://docs.saleor.io/docs/developer/running-saleor/s3


Modify config service http request url at proto file

Category

[ ] New feature
[ ] Bug fix
[ ] Improvement
[ ] Refactor
[x] etc

Description

modify config service http request url at proto file

format is {service}/{version}/{resource}/{verb}
all http method is post
declare request body for all request

add option go_package

It's about where Go package's import
In order to generate GO code this is must be declared in .proto file. Here is official documentation for this

Known issue

That was my bad, thanks for the check.
I changed it.


Not working in Rails 6.1 with Ruby 3.0

Bundler could not find compatible versions for gem "rails":
In Gemfile:
rails (~> 6.1)
devise_token_auth (~> 1.1.4) was resolved to 1.1.4, which depends on
  rails (>= 4.2.0, < 6.1)

You can fix it by using:
gem 'devise_token_auth', github: 'lynndylanhurley/devise_token_auth'

You can fix it by using:
gem 'devise_token_auth', github: 'lynndylanhurley/devise_token_auth'

Not a great solution though if you depend on gems. Are there any plans to publish the new version of the gem?

Can't do that because I don't have git installed in my docker containers and I am not about to install it either :)
Can we have a new release? Obviously, master is working (this I already confirmed). Come on maintainers, cut us a new version!

@pnghai @luisalima @enomotodev @melnik0v @theblang @croatech @dirkjonker who's in charge of this project? Who can cut a new release? I verified it is working on rails 6.1 but I can't install git repositories in our docker containers.

@mhenrixon there is a v1.1.5 which works fine for me (rails v<IP_ADDRESS>, ruby v3.0.1). See here it was committed just before release: https://github.com/lynndylanhurley/devise_token_auth/compare/v1.1.4...v1.1.5

@KiwiKilian I am on version 1.1.5 and I get errors on connection_config being used instead of connection_db_config

@mhenrixon which rails version are you on? See here: https://github.com/lynndylanhurley/devise_token_auth/commit/5b2ef6c2d394894d10ffd87a85b633570ee6c4c8

Different issue here, rails 6.1 and ruby 3.0.0:
devise_token_auth: 1.2.2
FrozenError (can't modify frozen Hash: {"Location"=>"http://<IP_ADDRESS>:3000/", "Content-Type"=>"application/json; charset=utf-8", "Cache-Control"=>"no-cache"}):
.rvm/gems/ruby-3.0.0/gems/devise_token_auth-1.2.2/app/controllers/devise_token_auth/concerns/set_user_by_token.rb:117:in `update_auth_header'

Is no one encountering this issue  after upgrading from ruby 2.7.x to 3.0.0 ??


Update vendored dependencies for terraform 0.7.13

Updates for terraform 0.7.13.

retest this please

:+1: :)

👍

@adamdecaf I modified the Makefile as well. Was fixing the jenkins build but it was still using old big and it's own command to test instead of what was in the repo.

retest this please

retest this please

Cool. Looks good

retest this please

retest this please

@adamdecaf I did one more change to the Makefile, and that was using a local docker-compose yaml file instead of big to run tests. It was getting hard to debug around big related things when it does a bunch of things this project doesn't need (like automatically adding the LB stuff). This is sort of in a direction to remove dependencies on anything Banno specific as well. Another step would be to run these tests from a container linked to the kafka scheduler so DNS isn't something to worry about either (also makes the nasty if statement go away), but leaving that for another PR.


Strange looking sections

After last Update i've got this
Atom 1.5.4
Material-UI 1.2.5

I'm sorry… what's strange about it?

You mean this?

The blue(ish) Field right.
On other PC with Material-UI 1.2.0 it looks correct.
Same as in your Screenshot

Thanks! Should be fixed in 1.2.6 :+1:

Thx too for ur great work...


width (-w) handling depending on resolution

Hi,
I'm playing around with catimg (the c app, not the script) and don't fully understand the width (-w) parameter.
Especially I don't see why setting the resolution to 1 or 2 has an impact on the image width. The terminal width (tput cols) is the same in both scenarios, no? Is it supposed to behave like that?
/Thomas

the resolution makes one pixel span over two characters or not. then the width uses that count, so -r 1 will produce a larger image. In other words, the width isn't in terms of columns but rather in term of rendered pixels

Ok, I see. Thanks for clarifying it ... it's quite confusing but nevermind


Add: video_device clap arg to monitor particular video devices on linux

I tried to add the mac code but the data coming back for com.apple.cmio is limited.

I tried to add the mac code but the data coming back for com.apple.cmio is limited.

@mbwilding I'm not sure there is a sane way to do this on macOS, to be honest!

Can you rebase this too, please? I tweaked the CI workflows in https://github.com/timrogers/litra-autotoggle/pull/9, and I am hoping that will make the full build run for your PR!

Ok done!

Just noticed the comments, will sort it.

That look good?


updated plex-media-server (<IP_ADDRESS>.1620-e0b7243)

Thank you for the contribution. It was merged directly as 7195b01e7d43fd55670eeb7d92635581692f566c to keep commit history cleaner. Your contribution is still credited to you.

@vitorgalvao I suppose this pullrequest has not been merge by error

Yep, same reason. Thank you for the notice.

Merged as cdd5764eb7057deaad550295e86754473567d6c5.


Configure travis/sauce setup for this

Please document this so other folks can do this as well, going forward.

where? is there a repo that I should add a README? or does one of the repos have a wiki?

Good question, maybe wiki on /ampersand??

yea, I was thinking that also - would allow us to document other publishing scripts and meta org requirements

travis is configured and the steps are documented in https://github.com/AmpersandJS/ampersand/wiki/Travis-CI


False positive keysMissing detection

Long story short
Josh & Pacu experienced this error state to be prompted even though the next start of Zashi resolved the issue. Unfortunately, there are no steps to reproduce.
Sorry, the long story
I put a lot of thoughts into this and here is my assumption of what might be going on and what the implications are.
There are 2 scenarios:

The keys are truly missing, typically after iPhone restore
The keys are present but Zashi ends up in a conclusion the keys are missing (the next of this ticket explains why it might be the case)

First of all, this is a serious issue that has direct implication on #1024. @true-jared, I'm really worried about this because whenever this happens and it's a false positive we prompt them with message that is not true and even worse, we try to pull them into a flow of seed validation.
So the way I see it we must resolve this issue first before we can focus on a valid case when the keys are truly missing.
The seed rules Zashi
With seed, Zashi works and can work. Where is no seed, there is no working Zashi. This is very simple fact. Zashi tries to get the seed from the Keychain as soon as possible, ideally right after the start of Zashi.
The keychain
A few facts:

The seed is stored in the keychain of an iPhone, nowhere else. The keychain API is probably the oldest API in iOS SDK, it doesn't support async await syntax whatsoever, it's very low level old way of coding.
The keychain is a system "storage", so it lives outside of Zashi. We can't control it, we can't demand a priority read or write, anything like it. So if we try to read the values from it, there are no guarantees it'll work at that very exact moment.

Especially the last point hit me and basically created a space for speculation(s). What if keychain is locked at the moment we want to read it? With async syntax, the system would wait and provide the value(s) asap. The API is not async one though so it either can provide the value or not. If not, it fails, simple as that.
The assumption is the keychain uses either a lock inside or operates with an operation queue so it should always return value. Yet there are cases when the retrival of the seed fails. We need to understand why.
The options we have
Since the API is something we can't fix or improve, we need to do some precautions. I implemented some retry simple logic that is apparently not enough and hasn't solved the issue.
We could try to be more aggressive, increase the number of retry attempts, wait more time. This is something we will probably need to do anyway.
Another option is to implement a prompt with a spinner that says something like syncing with the keychain..., that will block users from accessing Zashi and periodically tries to read the values from the keychain until it gets the seed. The assumption here is that it must work and at some point it'll pass and unblocks users from accessing Zashi. We should try this as a prototype and internal solution so we can get some experience and answers.
My biggest worry now is that there might be some bug on iOS side and only new start of Zashi gets job done. This is also and still an option. Maybe Josh's and Pacu's run of Zashi simply failed because the API or system failed and fresh start was basically a restart of a computer (you know those cases when a restart is the only option). What I mean by this is maybe even a periodical try won't work until Zashi is restarted. We don't know that yet until we try to implement proposed prototype.

I dug deep to understand the lowest levels of iOS, there are 56! possible error codes. My goal is to get the error code from the keychain and propagate it to the UI so we know which one exactly it is.

I stress tested the keychain API, 1M reads/writes with no error, I think the API is pretty solid so I started to look somewhere else and actually there's a theory that would work so I'll focus on implementing this scenario to prove or deny it.
The theory:
The keychain access flag is set to kSecAttrAccessibleWhenUnlockedThisDeviceOnly. We have background task for bcg syncing every 3-4am. Even though our experience says the bcg sync works, it's at least confusing. The fresh start of the app during bcg syncing should fail to read see according to the flag. Because the device was locked.
Possible steps to reproduce the issue:

Start Zashi to make it work, go to the background to schedule bcg syncing for the next morning.
Kill the app from the background.
Lock the device, leave it on power and wifi.
Open the app next morning.

Expected behaviour:

The bcg syncing will awake Zashi at scheduled time.
Zashi will go through fresh start because it was killed.
Initialization pipeline will try to read seed from the keychain. kSecAttrAccessibleWhenUnlockedThisDeviceOnly flag should prevent from reading data.
Zashi fails and triggers alert view.
Zashi opened next morning starts from the background because it was awaken so it's not a fresh start.
Users will see alert view with keysMissing.

It is expected that background processes started with the background task APIs fail to get the keychain elements under the policies used by Zashi.
I think that it can be beneficial to actually split the logic between background start ups of the app and foreground. This will make Zashi start up without the keychain access that is not needed. An app started from the background can be assumed to have a db file that sync can work with.
Or make it simpler: not worrying about the spending capabilities until spending happens.
The old ECC wallet gave up on background sync because of its clunkyness of the time. The buffer start screen worked fine for us.
Allowing Background sync changes that game and the start up cycle of the app will have to accommodate the fact that the seed should not be accessed from the background with the device locked
I hope this context and walk through memory lane helps!

@pacu I resolved everything on Saturday, a few PRs will be created in coming days.

I finally found steps to reproduce the issue + I know the solution.
Why it’s happening?
Background syncing is the root cause but the conditions are very specific. Zashi was killed by the system from the background meanwhile, system was overloaded and running low on resources. Background task awakes the app and the SDK initalization starts, that process needs to get the seed from keychain but keychain is set to flag kSecAttrAccessibleWhenUnlockedThisDeviceOnly which doesn’t allow to read keychain when the device is locked.
And that is very likely true when you are sleeping, iPhone is connected to the wifi + power. So you wake up and find the alert view.
Solution(s)
I’m listing 2 and only 2 possible solutions just so you have full list but it’s clear that we will use only 1 of them

say bye to background syncing and stay as restrictive as possible when it comes to the keychain setup
change the flag of keychain, I propose kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly which means keychain can be accessed even when iPhone is locked (less secure btw) but must be unlocked after any restart of the device (all possible flags here)

Fix is a one liner, we simply change the flag but I wanted you all to know the implications
What to expect after fix?
You will never see this error again as a false positive, you will see this error when it’s true, the seed is really not stored in the keychain, which is a task we still need to solve.
NOTE: you see now the status code in the alert view thanks to changes yesterday, here is the specification of this specific status code errSecInteractionNotAllowed       = -25308,  /* User interaction is not allowed. */
PS: none of the possible flags is actually best for this error, all leave the door open for a very specific and very rare edge case the error is thrown again, so I’ll also implement an internal flag that will prevent from throwing this error and throwing a different one with message like Something went wrong, please restart Zashi. I think this PS is post-1.0 task though


[FIX] - ADD VARIABLE LIST FOR ADDING AS MUCH TAGS AS A USER WANTS BUT REMOVE HARDCODED

remove this ..
tags = {
Name        = "SERVER01"
Environment = "DEV"
OS          = "UBUNTU"
Managed     = "IAC"
}

I have added the below: using map(string) in variables.tf
variable "instance_tags" {
type = map(string)
default = {
Name = "SERVER01"
Environment = "DEV"
OS = "UBUNTU"
Managed = "IAC"
}
}
Was able to call the module and can see the tags getting associated with the created EC2.

Now checking the possibility of overriding these tags from the calling module (may check the option to define the same in the cloud-init yaml as well)

User Defined Tags as variables from calling module

# Calling Module
module "ec2-vm" {
  source          = "/home/shq1kor/aws-ec2-vm"
  region          = "eu-central-1"
  vpc             = "vpc-ec6e8e86"
  ami             = "ami-023adaba598e661ac"
  itype           = "t2.micro"
  subnet          = "subnet-19213454"
  publicip        = true
  key_name        = "id_rsa.pub" ### name of public key ###
  secgroupname    = "IAC-Sec-Group"
  ssh_path        = "/home/shq1kor/.ssh/id_rsa.pub" ### path of public key ###
  package_upgrade = "true"
  package_update  = "true"
  instance_tags   = { Name = "SERVER02", Environment = "QA", OS = "LINUX", MANAGED = "IAC" }    ### User defined ###
  packages        = ["wget", "whoami", "chmod", "mv", "update-ca-certificates"]
  users           = [{ name = "ankit-1", gecos = "ankit-1 user", groups = "admin, ubuntu", lock_passwd = "true", sudo = "ALL=(ALL) NOPASSWD:ALL", ssh_authorized_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVPOX0dQu5b+c0K4XM+JyELPGp3b529gLqDYtVflMtnc+kGGfPNeC/8LkaVKvtPBeLCQ7NqEaiHBmn4UpADw827XTBVK+JLsDVUJOIdpx/Ef0K12zkkt2EAMVZqBkV1Xzg48ZE9wZ4UrzKY3eCg5o2frUrEs9K4zKKBgfzv3O9xKChKqqmWJ4x6H46C0d/y87C8LEwtiNwPWjDfc6QzB925FsVaooPkpinpflZ4H7Z6fJmVa3nh2Ui+f6hPs1V+NdIgf3+JbgnouQYvNG5EXPFgllzV+skx+SY+27mIVAmeKGPHLHHzVv8cjV5FsBv/ILVrFUH8YskySpr0OYS9x8+JD6IdQn1OEPngGlSAY47fCWHGEHd5bRFcWXn69pCm7cUV9D0JzT8rxXhXEiu16v+7jl1X0gPQJtRHjhAOfc24SKKdUkYScf6im86KlvC3fdOXp4HntlhLQq8f/PX9JR7VMRSsLgCiCbjz9mAVJxChfV6uAz1Km41D6f7NalpUKs= shq1kor@BMH2-C-00134" }]
}

output "ec2-vm" {
  value = [module.ec2-vm.ec2instance]
}

same has been pushed to our feature branch as well

looks good! BUUUT var instance_tags should have no default value.. can you remove/male it empty?

Yes Sir,
Its done and now looks like this:
variable "instance_tags" {
type = map(string)
default = {}
description = "To define the tags for resources"
}
and with this, the user can define the variables from the calling module to associate with the EC2 he wants to provision

works sir!


Support Installation Expiration

This PR adds the ability for the client to add the expirationTime property, and adds the necessary infrastructure to automatically refresh the installation at a known cadence when an app is regularly being used.

In case an english explanation of when the expiration will be updated would help:
Everytime a new Installation is built-up, the new class ExpirationAdapter checks to see if any of the following criteria are applicable:

Is the expiration field of this installation empty?
Is at least one property of this Installation different than the last Installation seen by the ExpirationAdapter?
Is this Installation already expired, or within seven days of expiring?

If any of those criteria are true, it will set the expiration field for 30 days from now. If none of them are true, it leaves the installation alone.
Additionally, if it can tell a different component has changed the expiration since the last time this set it, it leaves it alone. This leaves the door open for a customer to write their own visitor that sets an expiration date.


New version: martinrotter.RSSGuard version 4.7.4

Created by 🥟 Dumplings in workflow run #4508.
Logs
Updated: 4.7.3 → 4.7.4
Cannot validate argument on parameter 'ChatID'. The argument is null, empty, or consists of only white-space characters. Provide an argument that contains non white-space characters, and then try the command again.
Submitting WinGet manifests
Cannot validate argument on parameter 'ChatID'. The argument is null, empty, or consists of only white-space characters. Provide an argument that contains non white-space characters, and then try the command again.

Microsoft Reviewers: Open in CodeFlow

Automatic Validation ended with:
(Automated response - build 896.)


added saliency maps for deep learning

Adds a tutorial I wrote on saliency maps.
https://medium.com/@thelastalias/saliency-maps-for-deep-learning-part-1-vanilla-gradient-1d0665de3284

Thanks for the PR @andrewschreiber !


Add LICENSE

I haven't gone through the process of putting a license on this yet, and I'm not sure which I'd recommend.
I know I've used the MIT License before, but the following may help:

 comparative list
guide/documentation
simplified clickable decision tree

Some conversations have to happen before we decide which Open Source license (if any) get added to this project. For now, assume nothing has changed and continue contributing to the project without adding a license. Thanks!

It seems to me that the MIT and Apache 2.0 licenses make the most sense.
both permit

commercial use
distribution
modification
private use

...without warranty (hold user/distributor liable) under the condition that the user includes a copy of the license.
However, the Apache 2.0 license allows for patent use (while holding onto the trademark). The MIT license does not.
Given this, and as the creator of the project, I'd recommend MIT.

Thank you for your recommendation. As I stated before, there are some conversations that need to happen internally before any further decision is made.


How to get consumer metrics ?

Versions
Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
Sarama Version:
Top of the tree - master
Kafka Version:
./bin/kafka-topics.sh --version
2.0.0 (Commit:3402a8361b734732)
Go Version:
go version go1.11 darwin/amd64
Configuration
What configuration values are you using for Sarama and Kafka?
Logs
When filing an issue please provide logs from Sarama and Kafka if at all
possible. You can set sarama.Logger to a log.Logger to capture Sarama debug
output.
Problem Description
How can I obtain kafka consumer specific metrics ? How the following metrics related to any kafka consumer ?
incoming-byte-rate
incoming-byte-rate-for-broker-

Sarama only exposes Broker and Producer related metrics.
Can you please elaborate on your second question.

Hi,
If you consider the following two metrics -
incoming-byte-rate . (Bytes/second read off all brokers)
incoming-byte-rate-for-broker- . (Bytes/second read off a given broker )
Is there any relationship with any kafka consumer which is consumer data off the cluster or a particular broker ?

@dhbarman Currently it is not plausible with Sarama to detect the io byte rate on a Kafka cluster. Please take a look at the documentation provided by confluent on monitoring. You have to setup a JMX metrics exporter that collects the io byte rate of Kafka.

kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
Aggregate incoming byte rate.
kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec
Aggregate outgoing byte rate.


fix(client): optimize tx submission sm retries

Re #6530
I was a bit worried about not starting await before first succesful submit, but it looks like it's OK - the consensus has latency as well, so client has time to make a connection and it doesn't seem to matter.
old:
### LATENCY FM PAY: min: 1.0s, avg: 1.1s, median: 1.1s, p90: 1.4s, max: 1.4s, sum: 11.2s
### LATENCY FM PAY: min: 1.0s, avg: 1.6s, median: 1.1s, p90: 6.1s, max: 6.1s, sum: 16.2s

new:
### LATENCY FM PAY: min: 1.0s, avg: 1.1s, median: 1.1s, p90: 1.2s, max: 1.2s, sum: 11.0s
### LATENCY FM PAY: min: 1.0s, avg: 1.1s, median: 1.1s, p90: 1.2s, max: 1.2s, sum: 11.2s
### LATENCY FM PAY: min: 1.0s, avg: 1.1s, median: 1.1s, p90: 1.7s, max: 1.7s, sum: 11.4s

@elsirion Seems like a likely cause of executor "getting stuck" on bad internet connections?

Backcompat tests failing with:
00:00:20     2024-12-10T23:39:18.451651Z DEBUG fedimint_cli: Command failed err=Reissue failed: Transaction not accepted "The transaction had an invalid input: 1-The note is already spent"

for fedimintd 0.2.1.
I vaguely remember that we had an API race condition like that in the past, and possibly the timing changes in this PR are triggering it.

I was a bit worried about not starting await before first succesful submit, but it looks like it's OK

Why make this optimization? Why not already start the subscription?

Why make this optimization? Why not already start the subscription?

It cuts the number of calls running in parallel by two, at least initially.


Proposal on extending the mechanism of imports

Tact should support importing external libraries like most programming languages.
Your ideas are very welcome!
Tagging @talkol, since you expressed interest in this :)

External imports is a tradeoff - you make it easier to bring dependencies, but you're more prone for dependency hell and hiding from users what's actually being imported. In some extremes like NPM, this creates node_modules with hidden dependencies of 2GB and minor versions updating in sub dependencies without anyone noticing.
I think that for smart contract development, this tradeoff is more negative than positive, so I'm personally against this feature.
When a user decides to use a contract, it should be very clear exactly what code they're dealing with because there's money at stake. I think it's more prudent for the developer to manually bring the dependencies inside their codebase. This makes the developer take full responsibility and be aware of exactly what code goes into the contract.

One idea I had is to extend the capabilities of an import statement to allow cherry-picking of what's being imported and what is not. But it seemed unnecessary complex to implement for the little value it provides, as it's way easier with contract code (compared to conventional programming languages) to simply re-organize contents of the imported .tact or .fc files and cut away at any unnecessary things that way.
Which also makes me lean towards @talkol's conclusion: it's probably better for the developer to manually bring the dependencies into their codebase. It may be an UX drawback, but it's a long-term win for security and asset management of the contracts.

Closing this for now. We might return to this issue in the future


[Qt] Static builds depending on qt5-imageformat will fail under some circumstances

see #8205
@JackBoosY For your information
Reason:
tiff or webp installed
->
qt5-imageformat autodetects system libraries:
without tiff installed (or debug build with tiff-> not found due to extra debug suffix ):
Running configuration tests...
Checking for jasper... no
Checking for mng... no
Checking for tiff... no
Checking for webp... no
Done running configuration tests.

Configure summary:

Further Image Formats:
  JasPer ................................. no
  MNG .................................... no
  TIFF ................................... yes
    Using system libtiff ................. no
  WEBP ................................... yes
    Using system libwebp ................. no

with tiff installed (release build):
Running configuration tests...
Checking for jasper... no
Checking for mng... no
Checking for tiff... yes
Checking for webp... no
Done running configuration tests.

Configure summary:

Further Image Formats:
  JasPer ................................. no
  MNG .................................... no
  TIFF ................................... yes
    Using system libtiff ................. yes
  WEBP ................................... yes
    Using system libwebp ................. no

since tiff is also dependent on lzma qt5-tools will fail as seen in #8205
How to fix:
Pass -system-tiff and TIFF_LIBS with the correct libraries to the submodule.
same for webp

NOTE: the variables might be LIBTIFF_LIBS and LIBWEBP_LIBS instead of TIFF_LIBS and WEBP_LIBS. I do not 100% understand how qt uses/discovers those variables in their buildscripts.


Filters - default to Cited by Derrida in DG

Cited by Derrida in DG should be selected and disabled by defualt, in both library page, and reference list page, if we're not doing chapters.
currently on reference list page, "Cited by Derrida in DG" in selected by defualt, but when any other filter is selected, this will become unselected. and it can't be selected mannualy because it's disablesd

@kac3 I had a possible aha moment about this, wanted to check with you: according to Xinyi's design, the library page should be restricted only to books that are cited in DG - I had that turned off because it was causing problems, it seemed to be restricting the list too much.  My possible aha - do we want to restrict to instances that are cited, or do we want to show all copies of a work that is cited?

@rlskoeser We definitely want to show all instances, not just cited instances.
My preference for the library page, in other words, would be to show all cited instances -- i.e., all instances that have a reference associated with them -- AND all other instances of the same work.
We should also display any extant instances of the Derrida work itself (in this case DG).
In principle that should include everything we have in the system at this point.

@kac3 I did some querying on production to get some counts to help us figure this out.  The numbers in production look much better than the data I have, so I can tell that the team has been hard at work!

159 work instances that are books (excludes journal articles and book sections)
71 of those are extant
67 of those are cited directly (I'm using the cited_in property in the database to check this)
when I expand my filter to include any instances of a work that is cited, the count goes up to 70; the one that is not included is Essai sur l'origine des langues (1817)

When I switch to querying based on related references in the database, I only get 59 extant instances that have references directly associated and 61 for instances where any copy of the work is cited.  I'll stick to that is_cited flag.
I also did a preliminary check on digital editions matching up with instances.

54 extant books are currently associated with digital editions; there is one journal instance, Les anagrammes de Ferdinand de Saussure (1964) that is associated with a digital edition, looks like a mis-association to me
12 manifests (digital editions) in production are currently not associated with an instance; I'll re-run the digital edition import script and report on which ones aren't automatically matching

Forgot to say in my last comment: I don't think  DG is currently included as a library work / instance, so it won't be displayed on the library browse page.  But you could add records to document the copies that he owned.

@kac3 here are the decisions I think we need from you:

what filters do you want included by default on the library browse? it is currently restricted to extant books
do you want the "cited in" filter pre-populated to DG? currently the design calls for cited in preset to DG and disabled (i.e., you can't turn off that filter), but I think doing that loses some content we should display
please confirm: the "cited in" filter should select all copies of a work that has any instance cited (I think this will make it harder to identify actual cited copies)

Correction: I see now that Xinyi's latest design has the extant filter turned on by default, but it is something people can turn off.  That seems valuable - I'll update the site so you can see that.

@rlskoeser

Having the extant filter turned on by default but able to be turned off (as it currently is on the test site) is perfect.
I see what you mean about losing content by having the "cited in" filter preset to DG and disabled.  Let's not have it be preset, then.
You're totally right that this (i.e., having "cited in" filter select all copies of a work that has any instance cited) makes it harder to identify the actual cited copies.  As long as we don't have the "cited in" pre-selected -- so that the non-cited instances are visible by default on the Library page -- I think we should revert to having only cited instances appear when "cited in DG" is selected.  I was just worried about other editions/translations getting buried if they didn't show up on the Library page, but this seems like a very good solution.

@kac3 please provide updated help text for the library page, as it's no longer accurate with these changes.  Current text:

Only books extant in the Library of Jacques Derrida collection, held at the Princeton University Library’s Department of Rare Books and Special Collections, are displayed below.

@rlskoeser I think we can just eliminate this text and leave what's above ("Browse Derrida's copies..."). Does that make sense?

@kac3 ah, I see how it's kind of redundant now...  It still seems nice to mention PUL/RBSC, though; does it make sense to mention digitized editions are from RBSC?

@rlskoeser What if we make sure to mention PUL/RBSC on the home page (as in current version of text) but not mention it here?  I say this partly because the Library page text has always seemed a bit cluttered to me and I'm relishing the possibility of being able to unclutter it a bit.  (I'm also open to including something, though, if we can come up with the wording.)

@kac3 either way sounds fine to me; @xinyil any design implications?

@xinyil @rlskoeser Another little wording issue -- not filters, but search results:

"Annotations" should now read "Interventions," yes?

removing this sentence sounds good to me. mentioning PUL/RBSC on the home page and credit page would be enough.
it reminds me that we have a related help text in the image gallery, which doesn't seem to be implemented yet?


[24469] Implement CsvIO.Write and supporting classes

This PR closes #24469 and addresses #20312 by implementing CsvIO.Write and supporting classes.  Please see #24469 for more details.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
~[ ] Update CHANGES.md with noteworthy changes.~
~[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.~

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Note that WriteFiles supports a lot of additional properties like dynamic destinations, no spilling, ... that users typically want. They are pretty easy to add since they are just boilerplate builder stuff. Your call if you want to add them now or later.

@lukecwik I'm glad you suggested this as it inspired me to remove fields shared between CsvIO.Write and FileIO.Write and instead just maintain an inner instance of FileIO.Write.  For example, CsvIO.Write's withCompression calls the inner instance of FileIO.Write's withCompression instead of maintaining a redundant compression property.

Run Python PreCommit

Run Python PreCommit

Run Python_Runners PreCommit

Run Java PreCommit

Run Java PreCommit


Fixed typo in vpr-main.cpp

Motivate of the pull request

[X] To address an existing issue. If so, please provide a link to the issue: <#1821>
[ ] Breaking new feature. If so, please describe details in the description part.

Describe the technical details
What is currently done? (Issue link: https://github.com/lnis-uofu/OpenFPGA/issues/1821)
Currently there is a typo in the completion message for vpr

What does this pull request change?
Fixed typo

Which part of the code base require a change

[ ] VPR
[ ] Tileable routing architecture generator
[ ] OpenFPGA libraries
[ ] FPGA-Verilog
[ ] FPGA-Bitstream
[ ] FPGA-SDC
[ ] FPGA-SPICE
[X] Flow scripts
[ ] Architecture library
[ ] Cell library
[ ] Documentation
[ ] Regression tests
[ ] Continous Integration (CI) scripts

Impact of the pull request

[X] Require a change on Quality of Results (QoR)
[ ] Break back-compatibility. If so, please list who may be influenced.

Once CI is green.  I will merge. Thank you for the contribution.


Middle mouse click on tab to close it?

This is the behaviour for general tabs.

Fixed in https://github.com/xcv58/Tab-Manager-v2/pull/2477

1.25.1 published https://chromewebstore.google.com/detail/tab-manager-v2/nimllkpgmmbdglnjneeakdgcpkbgbfbp


MHR CTD

with the reFramework alone installed with no other mods, my game crashes as soon as I touch the canteen. I get this crash
[2022-10-01 23:37:13.932] [REFramework] [error] Exception occurred: c0000005
[2022-10-01 23:37:13.932] [REFramework] [error] RIP: 1411084db
[2022-10-01 23:37:13.932] [REFramework] [error] RSP: 13635fc28
[2022-10-01 23:37:13.932] [REFramework] [error] RCX: 29f3b3e0
[2022-10-01 23:37:13.932] [REFramework] [error] RDX: ffffffffc03303ba
[2022-10-01 23:37:13.932] [REFramework] [error] R8: 2d22e9f8
[2022-10-01 23:37:13.932] [REFramework] [error] R9: 2ddd49b8
[2022-10-01 23:37:13.932] [REFramework] [error] R10: 207a8130
[2022-10-01 23:37:13.932] [REFramework] [error] R11: 13635f588
[2022-10-01 23:37:13.932] [REFramework] [error] R12: 38dede30
[2022-10-01 23:37:13.932] [REFramework] [error] R13: 21cccd90
[2022-10-01 23:37:13.932] [REFramework] [error] R14: 20
[2022-10-01 23:37:13.932] [REFramework] [error] R15: 270d9740
[2022-10-01 23:37:13.932] [REFramework] [error] RAX: 28a76ad0
[2022-10-01 23:37:13.932] [REFramework] [error] RBX: 38dede30
[2022-10-01 23:37:13.932] [REFramework] [error] RBP: 0
[2022-10-01 23:37:13.932] [REFramework] [error] RSI: c03303ba
[2022-10-01 23:37:13.932] [REFramework] [error] RDI: 207a8110
[2022-10-01 23:37:13.932] [REFramework] [error] EFLAGS: 10246
[2022-10-01 23:37:13.932] [REFramework] [error] CS: 33
[2022-10-01 23:37:13.932] [REFramework] [error] DS: 2b
[2022-10-01 23:37:13.932] [REFramework] [error] ES: 2b
[2022-10-01 23:37:13.932] [REFramework] [error] FS: 53
[2022-10-01 23:37:13.932] [REFramework] [error] GS: 2b
[2022-10-01 23:37:13.932] [REFramework] [error] SS: 2b
[2022-10-01 23:37:13.932] [REFramework] [error] Module: 140000000 R:\SteamLibrary\steamapps\common\MonsterHunterRise\MonsterHunterRise.exe
Someone mentionted that this is an issue related to AMD GPUs. I wanna know what the issue is so I can fix. Thanks in advance.
reframework_crash.zip
re2_framework_log.txt
Dxdiag error is:
+++ WER7 +++:
Fault bucket<PHONE_NUMBER>286959915, type 5
Event Name: BEX64
Response: Not available
Cab Id: 0
Problem signature:
P1: MonsterHunterRise.exe
P2: <IP_ADDRESS>
P3: 6311801c
P4: dinput8.dll
P5: <IP_ADDRESS>
P6: 633114cc
P7:<PHONE_NUMBER>405115
P8: c0000409
P9:<PHONE_NUMBER>000007
P10:
After I removed Reframework, it worked fine. I installed Hunterpie and I got this error instead:
+++ WER8 +++:
Fault bucket<PHONE_NUMBER>726546582, type 5
Event Name: AppHangB1
Response: Not available
Cab Id: 0
Problem signature:
P1: MonsterHunterRise.exe
P2: <IP_ADDRESS>
P3: 6311801c
P4: 7c7d
P5: 67246080
P6:
P7:
P8:
P9:
P10:
Capcom CrashReport:
SteamLibrary\steamapps\common\MonsterHunterRise\MonsterHunterRise.exe
ExceptionPointer: 000000014F684AA0
ExceptionRecord:
ExceptionCode   : C0000005
ExceptionFlags  : 00000000
ExceptionAddress:<PHONE_NUMBER>1084DB

FIXED: Deleted all files in MHR folder except for the Chunk files. Verified Integrity and it worked now with HP, haven't tried ReFramework yet, but anyway, tip for people having the same issue.


Add "default" context and/or namespace

Would there be any way to add a "default" namespace and cluster context that kubectx would switch to either after a timeout or in each new bash window?
Primarily to make it harder to accidentally apply something to a production cluster.

This is tough to design. Most users don't expect something like a timeout and the timeout-based approach makes things harder.
I have this proposal for a while https://github.com/ahmetb/kubectx/issues/12 it might work for you. Please read and let me know if that existed, would that solve your problem?

I think it'd help, if I'm understanding it correctly. I've got 6 clusters now with another 6 that I could add and just wanted to make it harder to accidentally hit the production clusters with something unexpected.
I suppose I could also just add something to my bashrc to just reset on every new window.

I'll close this as I see no way of implementing this in a good way.
I recommend you add this to your crontab or bashrc. Thanks for bringing it up.


Remove logo and animations

When I demo Algolia I want to showcase the speed and animations don't really help with that. I suggest we remove them.

Suggest we follow Airbnb pattern as seen on https://www.airbnb.com/

We need to make the skeleton dynamic based on the shape of a hit

Tech spec

Build a skeleton loader instead of a loading window to smoothify the loading of pages in the Boilerplate.
This will increase the speed of the demo, as well as increasing positive customer UX
The requires a rebuild of the current skeleton loaders in use on the Search Results Page, followed by implementing it across the Boilerplate as a whole.
This will solve the issue by removing the need for a loading window
I expect this improvement to be put into the V3 production

Let's maybe separate this into three tasks:

[X] Remove homepage loader
[ ] Remove animations
[ ] Implement skeleton

I don't know if it's all push and done but there is always the Logo of the Loader @BenDanielGreep
https://share.getcloudapp.com/v1uObmrL


[FEATURE] Specify SQLEngine inside Fugue SQL

We can change SQLEngine programming interface when running a SQL. But this is not implemented inside Fugue SQL
This should work
[SqliteEngine] SELECT * FROM some_table

[<engine> (params)] should be the expression to specify sql engine inside Fugue SQL

CHANGE:
The syntax will be
CONNECT  (params) SELECT ...


Add Ikea ICPSHC24-10EU-IL-1 (2x IKEA Model L1305 URSHULT 2W)

Add Ikea ICPSHC24-10EU-IL-1 in combination with 2x IKEA Model L1305 URSHULT 2W

Need to reject this PR as this is a LED driver, any bulb can be added to it, which have other power characteristics.
You can however add the directories and files to config/powercalc-custom-models directory to use them in your installation.
Also see:
https://homeassistant-powercalc.readthedocs.io/en/latest/library/protocol.html


Merge latest Guardian changes as of 18-04-2024

What does this change?
This change pulls in the latest Guardian changes into the BBC's version of the Grid, so as to bring it up to date and to pull in our changes that have been approved by the Guardian.
How should a reviewer test this change?
The changes in this PR have already been tested and approved by the Guardian. A quick sense check on the commits that this PR brings in would be good.
Tested? Documented?

[ ] sense checked by reviewer

looks fine to me


Add support for OpenTelemetry (OTEL)

What does this PR do?
Creates various OpenTelemetry instruments like counters, gauges, histograms, etc. that expose metrics relevant to retraced functioning.

Should switch to using the metrics lib before we merge this


No forwarding possible

Unfortunately, newly created subdomains do not work. no forwarding possible. Old entries created work.

What happens if you drop the proxies and create them in a different order?
So first create the subdomain proxy, and then the domain proxy.
I seems to me that the domain matches before the subdomain, because of the order they are created from.


Audio development

New fancy audio system
Asset Manager
???
PROFIT!

Sound priority chart:
PRIORITY_LOWEST - ambient sounds (wind, birds, etc)
PRIORITY_LOW - NPC/Mob sounds
PRIORITY_NORMAL - events sound (block break, explosion, etc)
PRIORITY_HIGH - player sounds (footsteps, etc)
PRIORITY_HIGHEST - ??? (futher chat notifications, etc)

Heyo - reviewed this one again and everything looks to be working great now! :-)
It has fallen behind main a bit tho and there are a few conflicts to resolve. Nothing crazy, the three merge conflicts were easy to solve, but at one point new audio usage was introduced in BulletPhysicsRenderer.java and I'm not sure what to update that to :-)
Could you take a look at it, pull from develop, and tweak the little bit of stuff needed so we can merge this to develop? Here's a little more detail about the conflict:
[ant:groovyc] E:\Dev\GitProjects\t3\Terasology\src\org\terasology\rendering\physics\BulletPhysicsRenderer.java:335: cannot find symbol
[ant:groovyc] symbol  : method playVaryingSound(java.lang.String,float,float)
[ant:groovyc] location: class org.terasology.logic.manager.AudioManager
[ant:groovyc]                     AudioManager.getInstance().playVaryingSound("Loot", 1.0f, 0.1f);

Seems like this one is ready for a merge attack!

Yep, doing last prep!

Only took a good 4 hours longer than I thought, woo! Finally done :D


Create SECURITY.md

Intentionally generic

Thanks, but I don't really think this file makes sense in this repository.

Was supposed to be a draft for now my apologies

@Coins2payroll: Presumably you already know this but the security file is in the main Bitcoin Core repo. If there was a vulnerability in this repo that needed reporting you can use the guidance there to report it.


does this work on VL53L0X?

first thing i want to thank you from the buttom of my heart you really dont know what did this meant to me
anyway i am trying to work with VL53L0X on esp32-S3.
i tried to make it work but all i get on terminal: VL53L5CX not detected at requested address.
i searched and changer the address (i know it's the same) just in case and i get the same error.
is there anything i can do?

Hi @na3eddin
VL53L0X and VL53L5CX are two distinct sensors from ST. They do share the same technology, but they are rather different. The VL53L0X can measure a single point up to 2m whereas the VL53L5CX is able to measure 64 points up to 4m. As a result, the firmware is very different and this library is not compatible with the VL53L0X so make to sure to check out dedicated libraries like this one ESP32-VL53L0X (i have not tested it).


How to export elements from a JSON array and process them batch wise

Hi
Thank you for first for this cool project.
I have the requirement to process a list of items in batch of 1000 records.
I have started loading the json file:
data = json_stream.load(f)

after that I have iterated of ther rows.
for row in data.persistent():
    print(json.dumps(row, default=default))

And this works.
But now I wanted exit after 1000 rows from the loop and continue later:
index = 0
batch=""
for row in data.persistent():
    index += 1
    batch += json.dumps(row, default=default))
    if index == 1000:
      break
#do something with the batch => that works

I tough now as I quit the for loop before I could renter it and proceed where I was:
index = 0
batch=""
for row in data.persistent():

=> but this gives me the error that the element is already used up and can't be restarted. So I assume my understanding that "data" can be used like a generator is wrong.
I have also tried the
for row in data:
without the persistent() part but in that case, I don't know how to export the whole JSON element out of the row as it looks json.dumps can't be applied in that situation.
What would be the correct way to read x elements from a json array
Input:
[{ "key":"value"},{"key":"value1", "anotherkey":"anothervalue"},{"key":"value2"}]
lets say if I want to process the first 2 entries in a first batch and then continue and process the next batch?
By the way: I don't know the name of the key + I don't know how many keys are in there. I'm interested in the full element including all keys and values.
batch one a string containing: [{ "key":"value"},{"key":"value1", "anotherkey":"anothervalue"}]
batch two a string containing: [{"key":"value2"}]
Thank you for any idea what the right solution can be.

Hey!
Your issue is that you when you start the second batch, your trying to restart iteration on the top level transient list:
f = StringIO('[{ "key":"value"},{"key":"value1", "anotherkey":"anothervalue"},{"key":"value2"}]')

# data is a transient (non restartable) list
data = json_stream.load(f)

# start iterating  - this is your first "chunk" (a chunk of size 1 is still a chunk right?)
for item in data.persistent():
    print(item)  # item is a persistent dict
    break

# now try iterating again for the second "chunk"
for item in data.persistent():
    print(item)  # you'll get a TransientAccessException here
    break

The above approach wouldn't work for a regular list either, as you'd just always get the first chunk over and over.
Instead you need to keep a reference to an iterator (by the way, a for loop automatically creates a new iterator when it starts).
data = json_stream.load(f)

# it is an iterator that will produce the next list item when asked
it = iter(data.persistent())

# use the iterator to get the first "chunk"
for item in it:
    print(item)
    break

# use the same iterator to get the next "chunk"
for item in it:
    print(item)
    break

Finally here's a nice neat function to achieve what you wanted:
def chunker(iterable, n):
    it = iter(iterable)
    done = False

    def _chunker():
        nonlocal done
        for _ in range(n):
            try:
                yield next(it)
            except StopIteration:
                done = True

    while not done:
        yield _chunker()

data = json_stream.load(f)

for chunk in chunker(data.persistent(), 2):
    print('-'*10)
    for item in chunk:
        print(dict(item))

Thank you so much for the very detailed explanation. That makes it very clear and works! Is there any way to support you and/or the project with a financial contribution?

Sure! That's very generous!
You can send something through PayPal here .


Fullscreen Border & Padding Fix

Fixes border AND padding which are displaying in fullscreen mode.
See: #389

@Invariant-Change we're splitting the code base so that this repo will house the server code, while the app and components will be in separate repos.  Currently the css is in https://github.com/lbryio/www.spee.ch; if you could make this pr on that repo, that would be great.


Index of C:\Users\BEHNAMKHANI\

Remote Access Diagnostic Report
Report Version:	1.0
Date and Time:	2/3/2022 4:49:31 AM
Username:	BEHNAMKHANI-PC\BEHNAMKHANI
Computer Name:	BEHNAMKHANI-PC
Windows Directory

Why:
Closes [issue link]

What's being changed:

Check off the following:

[ ] I have reviewed my changes in staging (look for "Automatically generated comment" and click Modified to view your latest changes).
[ ] For content changes, I have completed the self-review checklist.

Writer impact (This section is for GitHub staff members only):

[ ] This pull request impacts the contribution experience

[ ] I have added the 'writer impact' label
[ ] I have added a description and/or a video demo of the changes below (e.g. a "before and after video")

@behnamkhanioffical
This seems to be opened by accident, so I am going to close it now :yellow_heart: Take a look at the contributing.md to learn how to contribute in this repo. You can open a discussion if you have any questions :sparkles:


Build pipeline test

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: doyle-kfd sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Update podspec infos using package.json infos

Overview
Avoid inconsistency of informations with podspec and package.json
Test Plan
Go to example/ios run pod install and run example App and everything is working as expected

nice @luancurti - we do the same in device-info, works well. You specified a different homepage here but I checked and it's valid and this way is better for this repo - +1

Yes the homepage in package.json has #readme. But I think is more clear to redirect to readme :)


Change default password

The warning section in the README describing how we need to delete the pvc to change the default password is not correct.
We can indeed reset that password without having to delete the PVC by:

Deleting the file /var/lib/neo4j/data/dbms/auth
Running neo4j-admin set-initial-password <new password>
Deleting the pod / restart the service

I will create a PR with this section updated.

Fixed by: https://github.com/equinor/helm-charts/pull/7


Naming Critique

As I'm refactoring the library in my separate branch (also thinking about renaming it to avoid the confusion), I'm thinking more and more about renaming the internal macros, ending with I. Prefixing with _ is a lot better than suffixing with I, because those macros actually clutter the global completion namespace. So when you try to auto-complete something like TUPLE, a lot of internal "garbage" macros appear in the list, which is not really useful.
Another thing is that certain macros of the library are actually missing the M_ prefix, which in turn clutter the global namespace, which should be system- or user-defined.
I hope you don't mind some critique from the actual user of your tech ;)

The new _ct suffix for type aliases is also confusing. Does that mean "constant type"? Well, that is even more vague, since the constant may actually mean const. You could also prefix the internally used type alias with an underscore, IMO.

I think this is also a typo (m-shared.h):

I'm thinking more and more about renaming the internal macros, ending with I. Prefixing with _ is a lot better than suffixing with I, because those macros actually clutter the global completion namespace. So when you try to auto-complete something like TUPLE, a lot of internal "garbage" macros appear in the list, which is not really useful.

Prefixing with _  is illegal as per the C standard. It is a reserved prefix:

All identifiers that begin with an underscore and either an uppercase letter or another underscore are always reserved for any use (§7.1.3)
All identifiers that begin with an underscore are always reserved for use as identifiers with file scope in both the ordinary and tag name spaces (§7.1.3)

Future version of the norm lists more exactly the reserved pattern in §J.6.1:

[a-zA-Z][a-zA-Z0-9_]*

Prefixing with something else than may be better indeed for auto-completion (even if TUPLE_ should only propose valid macros).

certain macros of the library are actually missing the M_ prefix, which in turn clutter the global namespace, which should be system- or user-defined.

Which one for example?

Does that mean "constant type"?
No. The suffix _ct indicates a confidential type (not exported)

I think this is also a typo (m-shared.h):

Yes, indeed. There is two '_' in the suffix. However, this confidential type is not used in the oplist, so there is no impact.
So, I'll remove theses useless types instead. Thanks.

One more consideration from me is about C-string methods, string_get_str for example. This is opinionated really, but I would personally call those methods with something like cstr instead of str, as str is really confusing - your own string type is also starting with str.

Prefixing with _ is illegal as per the C standard. It is a reserved prefix:

Are you talking about a single underscore or double underscore? I think the former is fine, as I'm using it currently without any warnings while using the strictest level of warning generation.

this is even more confusing with general-purpose methods like _in_str where no C-string is present....

Are you talking about a single underscore or double underscore? I think the former is fine, as I'm using it currently without any warnings while using the strictest level of warning generation.

Yes. More exactly, as per the standard (see the extract of the standard in my previous comment):

underscore + underscore : reserved for anything.
underscore + uppercase letter : reserved for anything
underscore + lowercase letter: reserved for anything except macro, label of goto and members of  structures  or  unions

underscore + digit is not reserved however.

this is even more confusing with general-purpose methods like _in_str where no C-string is present....

Yes, naming is not perfect. :)
In particular string_get_str which is not related to cstr at all.

I suggest you ditch the _str methods for FILE and just rename them to _stream or _file. that's what I'm going to to in my branch, at least. =)

So when you try to auto-complete something like TUPLE, a lot of internal "garbage" macros appear in the list, which is not really useful.

I try changing the prefix of the internal macros (so that it start with something). But the auto completion I used (VS code) is "too smart" and still presents then even if they start with another prefix. So the renaming was useless (and not committed).

I'd suggest you could use something simpler like read\write serialize\deserialize. those are also pretty standard these days.

For internal macro naming, I think I'll use the following rule:

M_ + F(container_name) + _ + macro name

with F transform a little bit the container name (preventing the automatic completion to works). Like this for list:

#define M_L1ST_DEF_P4(....)

For generated internal functions, I think I will change the naming as per:

M_C3(m_ + F(comtainer_name) + _, name, _function_name)

Well, it would be better to auto-complete starting at M_. The list should include "public" entities only then, with no internal "garbage" terms.
M_L1ST_DEF_P4 looks rather scary, man)) Better use some hexadecimal representation of the name's symbols (joking). Why not choose some kind of uncommon prefix. I'm currently using a small i letter for my prefixes and it does just fine.
M_ ... public stuff for the end user.
iM_ ... lib's development area
Isn't the m_ prefix historically relates to members?

Yes, it is a little bit strange.
Unfortunately, the analyzer of the auto-complete function of some IDE (example Visual Code) don't just take the prefix to complete but tries to find a match even if the prefix doesn't match. So even if I start a macro with M_i_LIST_something, when I write LIST_, I will still have M_i_LIST_something as a potential match. The only thing I found which works is to rewrite completely the prefix LIST into something else (here L1ST). I don't know how to inform the auto-complete function that a symbol shall not be auto-completed, or to inform it of a kind of "priority" in the symbols.
The m_ prefix is indeed used by a lot of C++ code to identify the members of a class.

My personal opinion is that changing the prefix is "good enough" as changing the name to l33dspeak is "too much".

All the public API have been put in the M_ namespace. All the internal API have been put in their respective namespace.
A wrapper to the old names is provided (which is enabled by default and can be disabled by defining M_USE_SMALL_NAME to 0 before including any header file).
Planning:

for this version, keep the wrapper to the old name and enable it.
for the next version, keep the wrapper to the old names and disable it.
the the next next version, remove the wrapper.

The renaming of the _str suffix has been put in its own ticket (#99)


thoughts: GautamMandsorwale

Hey @sambhav2612
Request you to please check and approve this PR.
Thanks Again!

done @GautamMandsorwale


PAGE XML contains TextEquiv with empty Unicode

The PAGE XML files contain lots of text regions without text in their TextEquiv and a few text files without text in their TextEquiv:
# Text regions without text.
% git grep "^                <Unicode></Unicode>" | wc -l
   17097
# Text lines without text:
% git grep "^                    <Unicode></Unicode>" | wc -l
       6

Text from regions with text in lines but without text in the region gets lost when the PAGE XML file is converted to pure text using ocr-transform.

https://github.com/UB-Mannheim/Fibeln also has 11 files which contain text regions without text. Those files where also created using Transkribus. This indicates that it might be a general problem of that software.

Solved: a65f5ba91c1ea1be13f9443a96cedbcb249b346d


Consider using Shadow Collection?

I love this package but it had a few issues and think I have build a solution to them with: https://github.com/Meteor-Reaction/shadow-collection.
What you would get to using this:

non-conflict - other packages that want to proved helpers can do so without conflicting with this package.
it lets Mongo.Collection transform work again
It removes the console reference to Mongo.Collection.helpers.Document.(anonymous function)

If this interests you I would be happy to do a PR, or leave it to you.

Also worth noting that I have not, yet, published shadow-collection as I am willing to make adjustments as needed.

Hey @Kestanous interesting idea, I'm interested to know more about your use-case for this. I can see how it might be a nice idea to extend helpers from an existing transform, but on the other hand I worry that this could easily lead to very difficult bugs to trace. For example if you were to unknowingly override an existing method.

@dburles My end goal is full models: https://github.com/Meteor-Reaction/gravity-models. But @rclai work inspired this code https://github.com/rclai/meteor-document-methods.
Maybe we could add a warning or throw an exception if an existing method were to be overwritten? I am not sure just yet but I think the idea is worth investigating. Happy for feedback, actually if you are willing to switch to this I can just add you to the "shadow" teem (shadow-collection & shadow-describer). That way you can you can maintain collection-helpers without delays.

Perhaps an idea might be to provide a configuration option that will allow this behaviour. By default it'll throw an error.

@dburles thats also what I was thinking. It would be neat if we could add an exception rule for each method, rather then all or nothing.
exception = {
  helpers: false,  //throw error
  sillyName: true //allow override
}

:+1:

I'm not totally sold on the idea of the shadow collection package, I feel there's not a real-world requirement for that functionality. I'm interested to hear though if it was built within the context of a real-world application and how having that functionality was required.

@dburles so there is a real-world need but it is very abstract at this level.
The issue I often have is I need my model layer to be self describing and self encapsulating. Your packages (helpers & instances) does a really go job on the encapsulating side. I am building shadow-describer to cover the description side.
Consider the following (obviously the code could be cleaned up)
Here is how it is now:
dynamicAddToFoo = function (data, collection, type) {
  if (type === 'cursor') {
    data.foreach(function (obj) {
      Mongo.Collection.get(collection).update(obj._id, { foo: ++obj.foo });
    })
  } else {
    Mongo.Collection.get(collection).update(data._id, { foo: ++data.foo });
  }
}

vs
dynamicAddToFoo = function (data) {
  if (data.$shadow.type === 'cursor') {
    data.foreach(function (obj) {
      Mongo.Collection.get(data.$shadow.collectionName).update(obj._id, { foo: ++obj.foo });
    })
  } else {
    Mongo.Collection.get(data.$shadow.collectionName).update(data._id, { foo: ++obj.foo });
  }
}

Now imagine the above code is in a package, say a form package, where the collection name may not be known. This adds a whole different level of abstraction to our data. In order to get shadow-describer, validity, and document-methods to play nice with collection-helpers, I need it to use shadow-collections.
tl;dr
I think it makes sense to add this but if you still disagree then no worries. I will likely just end up creating a shadow-helpers package that suits my needs. Mostly I think this can help the community as a whole and not just my apps.

Actually I forgot I was adding a pointer to the collection itself in shadow-describer so I would not actually need to use your collection-instances package.

I hope you can understand that I'm a bit hesitant to make changes such as making use of shadow-helpers, as it's doing some funky stuff, considering the number of people already using this package (+ production apps). I'll have a bit more of a think about the idea of extending from an existing transform.

Just as a side note, it's possible for a package to depend on collection-helpers and apply helpers just as normal. The Collection.helpers function can be called as many times as you like. So you can possibly achieve things that way.

@dburles I fully understand and appreciate your hesitation, this needs to be done right or not at all. Here are my thoughts.
1: Can you speak to the part that bothers you the most? I don't see how this is much worse then what is already happening and its mutates much less then Collection2 or PeerDB. Do you see a way we could make this safe enough to make it work. I am willing to do what it takes to get us there. If you could elaborate on the risks you see I will do my best to nullify them.
2: Unfortunately your solution would not solve the following issues I am facing, but maybe you see an alternative to those as well.
I need each instance, output from a cursor, to have a hidden object for state management. I am looking at using something like observe-js to watch changes on an object, auto run validations live, and keep a change log. To do that I need to store that state, for each instance, outside of the instance itself.
validity is checking if a schema exists on the collection before it adds the $validate helper. This prevents polluting collections that done need validations and could also let apps define schemas after meteor startup. I don't see a real world need for the second option but there you have it.
@rclai's document-methods is using a nasty hack to play nice with collection-helpers. I would like to remove that need. (@rclai it was very well done though)

I am willing to look into alternatives for all of that but shadow-* packages solve so many of my issue. Both in production and in my head -_^

There was a point when my package depended on collection-instances and looked really nice: https://github.com/rclai/meteor-document-methods/blob/471f583ddf2037751a38acf1fad8528a5f10ddc6/document-methods.js
The reason I moved into the monkey patching technique was because collections are usually instantiated outside of a Meteor.startup in a lib folder and some people might end up doing some collection operations outside of Meteor.startup and if they do that, the methods are undefined because the runtime hasn't reached the point where I inject the methods. So essentially I wanted the methods to be available at the earliest time possible and not just after Meteor.startup.
Should I should go back to what I had?

Hey @rclai why were you calling Mongo.Collection.getAll() within startup?

@dburles would getting more opinions on this help? We could to a community feedback request on crater be of value? We could link to this issue or you could lock this and we could move this to crater.

I think I'll park on it for a while.

@dburles just so you know I have a temp package over at https://github.com/Meteor-Reaction/shadow-helpers.

@Kestanous :+1: This is a good fit for my needs too

When adding transforms on top of another, isn't it going to cause the documents to no longer share a prototype function but rather all have a copy of the function at the top level?

@rclai no because it defines the functions for each doc, see https://github.com/Meteor-Reaction/shadow-collections/blob/master/lib/shadow-collection.js#L35.