How we can reduce latency between Primary Postgres cluster and Replica Cluster in CNPG for Kubernetes environment?

Currently, we have a setup running with Primary and multiple Replica clusters. We are also using S3 object storage to store WAL archives and base backups. Our Replica clusters are pointing to this storage to replicate the data but we have a data latency of approximately ~5 minutes and data is not getting replicated immediately.
Is there any way to reduce the latency or could you please suggest parameters to tune it?

Those 5 minutes sound like it's the archive_timeout closing and shipping the WALs every 5 minutes because your instance generates little WAL traffic (there's a caveat explaining that in the WAL Archiving section of the doc). Could that be the case?
CNPG sets a default of 5 minutes which you can tune, but as explained also in the PostgreSQL doc a value in the range of minutes is usually recommended. It's a tradeoff in the end, you don't want to set it too high because that way you'll have unarchived data for a long time if your server has a low workload, but you also don't want to set it too low otherwise you'll end up generating a lot of WAL files (which will also impact the recovery time).
If you want the replica clusters to replicate the data immediately you should rather use streaming replication, which you can setup via the externalClusters section (assuming you can grant the access between your primary cluster and your replica cluster)

Those 5 minutes sound like it's the archive_timeout closing and shipping the WALs every 5 minutes because your instance generates little WAL traffic (there's a caveat explaining that in the WAL Archiving section of the doc). Could that be the case?
CNPG sets a default of 5 minutes which you can tune, but as explained also in the PostgreSQL doc a value in the range of minutes is usually recommended. It's a tradeoff in the end, you don't want to set it too high because that way you'll have unarchived data for a long time if your server has a low workload, but you also don't want to set it too low otherwise you'll end up generating a lot of WAL files (which will also impact the recovery time).
If you want the replica clusters to replicate the data immediately you should rather use streaming replication, which you can setup via the externalClusters section (assuming you can grant the access between your primary cluster and your replica cluster)

@NiccoloFei  Thanks for the Reply. it is not exactly 5 minutes for each transaction but sometimes data gets replicated within 5 minutes or even faster. This is basically randomly taking the time to replicate the data to Replica clusters. But still, I will try to tweak this parameter.
We can not use streaming replication because we want to use multiple Kubernetes clusters(i.e. minimum 4 Kubernetes clusters). On a single Kubernetes Cluster, we would set up a Primary Postgres Cluster, and the other three Kubernetes clusters would be running the replica clusters.  We will also use the DB failover operation across the sites.
e.g we will promote Replica to Primary and Primary to Replica after failover. That's why we are using object storage as "externalClusters" for this solution. We will not use this solution for disaster recovery but we are trying to use this solution for GeoRedundancy.

Hi @oberai07,

@NiccoloFei Thanks for the Reply. it is not exactly 5 minutes for each transaction but sometimes data gets replicated within 5 minutes or even faster. This is basically randomly taking the time to replicate the data to Replica clusters. But still, I will try to tweak this parameter.

Mhh ... it is not randomly. It is actually deterministic. It is called RPO and we are capping it to 5 minutes. So RPO <= 5 minutes by default for backup/disaster recovery purposes.

We can not use streaming replication because we want to use multiple Kubernetes clusters(i.e. minimum 4 Kubernetes clusters).

Can you please help me understand the correlation between streaming replication and the number of clusters? Postgres has cascading replication, if needed, so you could have a serialized architecture:
K8s1 -> K8s2 -> K8s3 -> K8s4
You could have a replica cluster in K8s2 streamin from K8s1, another replica cluster in K8s3 streaming from K8s2, and K8s4 streaming from K8s3.
I assume each Kubernetes cluster is with a single availability zone. Is that correct?

On a single Kubernetes Cluster, we would set up a Primary Postgres Cluster, and the other three Kubernetes clusters
would be running the replica clusters and eventually, DB failover operation would happen across the sites.

Just to be clear: failover across Kubernetes cluster needs to be done manually at the moment.

e.g we will promote Replica to Primary and Primary to Replica after failover.

Another clarification: at the moment, CloudNativePG does not allow you to resync a former primary as a standby. You need to destroy it and recreate it. We are planning to improve replica clusters' failover and switchover management in 2024.

We will not use this solution for disaster recovery but we are trying to use this solution for GeoRedundancy.

Interested in knowing the differences as to me geo redundancy is still part of the broader disaster recovery strategies.
Keep us posted. Let us know if we can close this ticket.
Thanks,
Gabriele

There will also be a role for Network latency and hardware performance because the Primary Postgres cluster and Replica cluster are in different clusters or regions.

Mhh ... it is not randomly. It is actually deterministic. It is called RPO and we are capping it to 5 minutes. So RPO <= 5 minutes by default for backup/disaster recovery purposes.

I tried to tweak this value in the Postgres configuration and set it to 2 minutes. But it still did not reduce the latency.

Can you please help me understand the correlation between streaming replication and the number of clusters? Postgres has cascading replication, if needed, so you could have a serialized architecture:
K8s1 -> K8s2 -> K8s3 -> K8s4
You could have a replica cluster in K8s2 streamin from K8s1, another replica cluster in K8s3 streaming from K8s2, and K8s4 streaming from K8s3.

I assume each Kubernetes cluster is with a single availability zone. Is that correct?

We have four Kubernetes clusters running on four different availability zones and for each K8s cluster we have a Postgres cluster running.  (i.e. on k8s1 - Primary Postgres cluster deployed and the other three k8s clusters (i.e. k8s2, k8s3, and k8s3) are running with Replica clusters (Designated Postgres clusters). All Postgres clusters are pointing to BarmanObjectStore for backup and restoring the backup purposes. This design was proposed by the Enterprise DB Postgres(EDB) team to fulfill our requirements.
My other concern is how the Replica cluster which is running on different k8s clusters in different availability zones will make the connection to the Primary Postgres cluster. Because the Postgres cluster always exposes the K8s cluster-ip service, then we cannot communicate externally with the Cluster-ip service.

Another clarification: at the moment, CloudNativePG does not allow you to resync a former primary as a standby. You need to destroy it and recreate it. We are planning to improve replica clusters' failover and switchover management in 2024.

We are exactly doing the same i.e. destroying the  Primary cluster and creating the replica cluster after failover.

Hi oberai07,
I lead EDB solution engineer team in EMEA, can we have a quick chat to understand what replication solution you are looking for, and how is your configuration looks like.
You have mentioned also specific design on the architecture, based on the limitation.   Curious to know more about that too. Perhaps we have the solution allows you to implement replica data to different AZ or regions.  Since you are already working with the EnterpriseDB team, I am interested to understand the proposed solution .  You can reach out to me<EMAIL_ADDRESS>I look forward to connecting with you.
Cheers, Kevin

Hi oberai07,
I lead EDB solution engineer team in EMEA, can we have a quick chat to understand what replication solution you are looking for, and how is your configuration looks like. You have mentioned also specific design on the architecture, based on the limitation. Curious to know more about that too. Perhaps we have the solution allows you to implement replica data to different AZ or regions. Since you are already working with the EnterpriseDB team, I am interested to understand the proposed solution . You can reach out to me<EMAIL_ADDRESS>I look forward to connecting with you.
Cheers, Kevin

Sure @Kevinckli, I will send you an email with all details and design document.


Can't find CCM cipher, once installing openSSL 1.1.1

And also using AES_128_GCM_SHA_256 cipher from openSSL1.1.1 not much improve in latencies in comparision with AES_128_GCM_SHA_256 with openSSL 1.1.0g
Please Help!

Hi Team,
is there is any way to enable CCM ciphers with oepnssl version 1.0.2k


[Sessions] Log events to console command line flag

==========================================================
Session Event:
  event_type: SESSION_START
  session_data
    session_id: <EMPTY>
    previous_session_id: <NULL>
    event_timestamp_us:<PHONE_NUMBER>000000
    data_collection_status
      crashlytics: UNKNOWN
      performance: UNKNOWN
  application_info
    app_id: 1:1080744134505:ios:bbb8009438b93cc673dff8
    device_model: <NULL>
    development_platform_name: <NULL>
    development_platform_version: <NULL>
    session_sdk_version: 10.1.0
    apple_app_info
      bundle_short_version: <NULL>
      network_connection_info
        network_type: 0
        mobile_subtype: 0
      os_name: UNKNOWN_OSNAME
      mcc_mnc: <NULL>
==========================================================

#no-changelog

Tests are failing after decoding:


[release/v1.4] Add changelog for v1.4.13

What this PR does / why we need it:
Add changelog for v1.4.13.
What type of PR is this?
/kind documentation
Does this PR introduce a user-facing change? Then add your Release Note here:
NONE

Documentation:
NONE

/lgtm
/approve


Tidb, pd and tikv are not in a same network segment for servers with mutiple network cards

If localhost is used to configure smartserver, the hostname in os will be used to set the address of agent master. If this hostname is not binded to the desired ip, the user just needs to configure the right hostname for smart server.


CPU RAM memory leak when using the beam search decoder

Hi,
I used the ctc beam decoder from this link https://github.com/joshemorris/pytorch-ctc. However, I found that after I finished decoding one utterance, the decoder does not release RAM memory. After decoding more and more sentences, the RAM was full. This is especially true if I use a large beam width such as 100, in which case RAM usage quickly blow up.
My code looks like this:
import pytorch_ctc
from pytorch_ctc import Scorer
decoder = pytorch_ctc.CTCBeamDecoder(Scorer(), labels, top_paths = 1, beam_width = 100, blank_index = 0, space_index = -1, merge_repeated=False)
for i in range(total_num_utterances):
decoded, _, out_seq_len = decoder.decode(prob_tensor_i,  seq_len_i)
Anyone has any ideas how to fix this issue?
Thank you very much.

Are you creating a new scorer each time per chance? We don't do a whole lot of dynamic memory allocation.


rinalli/larval

Expected behavior
Actual behavior
Information about the Issue
Steps to reproduce the behavior

...
...

first


Add new env var AUTH_OIDC_REDIRECT_LOGOUT for OIDC logout

Description
Current behaviour:
I have setup homarr and oidc keycloak following the SSO section from the doc. It works well, I can log in with keycloak realm accounts, with or without admin right.
However when logging out from homarr, it doesn't logout the session from keycloak. Trying to login again it will login directly with previous user while not asking for user credentials.
Suggestion:
Adding an env var AUTH_OIDC_REDIRECT_LOGOUT that user can set up specific oidc logout url with docker compose envs.
Please tick the boxes

[X] You've read the docs
[X] You've checked for duplicate issues

@yulinhuang I believe you might be able to do this yourself ! You can probably ask ChatGPT / GitHub Copilot for help.
Take a look at this pr to see how it's done. Then you can submit yours, having a PR on the docs as well (https://github.com/homarr-labs/documentation/pull/65) would be extra nice. Good luck with it 😉

@ajnart
I may need some enlightment as I'm not familiar with front-end and js devlopment.
Here is my thought process:

Add an env var
modification in AvatarMenu.tsx

Access env var
modify logout on click to redirect to env var

It appears that AvatarMenu is executed on client side so I can not access directly the env var from the server side. So I modify the env var with NEXT_PUBLIC prefix, declare it in .env, build the project from source and it works locally with yarn start. Not sure this workaround is appropriate since other OIDC env are on server side.
Besides, when I try to build the docker image and use the docker container while declaring the env var, it doesn't seem to work properly.

docker image build from project Dockerfile

It seems the server is always taking the value define in local .env file, and not able to read from docker env.

I also tried to use the image build by github action

It seems the env var value is always undefined

Any suggestion?

The GitHub action will be undefined because these values need to be set in the settings of the project

Try yarn build then docker build to create the image, but it's normal that the local .env takes priority, it's not there on GitHub actions

You're right the logout in the Avatar is client-side. I think you might want to look at our middleware file / next auth files to see how the login process in handled and plugging a new fonction that would execute when the user logs out (on the server)

This would make it so that every logout button applies your intended fix.
Other than that I think your thought process it good 😉

@yulinhuang I'm currently working on adding a few env variables for session expiry. I can work this in if you want.
If you want to do it now though, I would understand and appreciate having a new contributor :)

@SeDemal Thx for the proposition. Yes please work on it if you can.
I feel like I'm doning something nasty when modifying in middleware files.


Ignore all theme color option sometimes uses theme color anyway while the page loads

I probably should have made an issue about this earlier, but I've noticed that while the page loads (or just for a few seconds), the theme color is used for a few or split seconds, then ignores that color afterwards
With sites with brighter theme colors, it can be kind of jarring having it flash like that

Hello! @Freeplayg. I actually know it happens and why it happens (see below, optional).
_A content script is used to extract a webpage’s theme colour. But, the script is newly loaded when a new page is opened, which means every time a new page is opened, the script has to read the user’s preference to decide if the theme-color tag should be ignored.
I was concerned about that’ll cause a slight delay between a new page being opened and tab bar colour being applied, so the content script reads and sends the colour of the page to the background script using default settings (using theme-color tag) as soon as the page is loaded. At the same time, the background script detects that a new page is opened, and sends the actual user’s preferences to the content script.
That’s why you’re noticing the flash. In that split moment of flash, after the content script has sent the colour to the background script, the content script hasn’t received the user’s preferences yet. And when the content script receives the user’s preferences, it reads the colour of the page again and new tab bar colour is applied._
I think I can change the content script a bit. Immediacy might be forfeited a little, but stability can be improved.

Moreover, to avoid flashing, you can also turn off “Allow light/dark tab bar”, which is closer to Safari’s behaviour on macOS. Safari prevents its tab bar from showing jarring colour by giving the tab bar its default colour during the webpage is being loaded.

@Freeplayg I changed how content script works. Settings for ignoring theme-color should be able to load before new tab bar colour being applied.


Adding Rancher Turtles Overview - Integrations with Rancher Section

Adding the CAPI integrations dropdown and overview page to Rancher latest, v2.8, and v2.7. Adjusting the sidebars for the respective versions as well. The CAPI documentation site was used for reference. The overview page consists of installing/uninstalling Rancher Turtles, an architecture diagram, and extension security.

Thanks for the review @martyav, just applied the changes across versions.

@btat Updated after applying your suggestions, thanks for the review Billy!

Sorry about that, I did intend to serve at the URL /integrations-in-rancher/cluster-api. Adjusted the .md titles and sidebars respectively.

Just updated, thanks for catching that Billy!


project can't run

hi
i am using your 'develop' branch, and flutter  2.2.3, and i get this while trying to run the imessage sample:
Execution failed for task ':app:processDebugMainManifest'.

Unable to make field private final java.lang.String java.io.File.path accessible: module java.base does not "opens java.io" to unnamed module @75a20156

can you please fix?

hey @noChamp what app are you running?

Imessage
On Fri, Aug 20, 2021, 10:34 Salvatore Giordano @.***>
wrote:

hey @noChamp https://github.com/noChamp what app are you running?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GetStream/flutter-samples/issues/47#issuecomment-902496994,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC7O3ZGB7TASQ66AHK32YRDT5YAR7ANCNFSM5CPX6NYQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

hi Guys its been a while
any help?

I'm trying to fix it today, I'll post an update


Remove all support for Python 3.6

AWS Lambda is ending support for Python 3.6, so all references to Python 3.6 should be removed from MIE. This follows Python 3.6 End-Of-Life (EOL) reached on December 23, 2021 [1].
[1] https://peps.python.org/pep-0494/

Completed in v5.1.0


Typed events

Events should be defined in the yml and potentially have data associated with them
Then you can use it like this:
ctx.sendEvent(Event.Foo, {foo: "Bar"}, userId)

for both sendEvent and Broadcast Events

any update on this?  i ran into a usage for it today

No updates yet @jyoung4242. Let me know if you'd be interested in taking a stab at contributing this feature, I'd be happy to guide you through what needs to be done in hathora to make it happen.

actually, i can try looking into it, sounds like an interesting problem to solve

@jyoung4242 did you end up looking into this? if not i may take a stab at it as this would make one of my current projects much easier

I'm okay with that, I'm super busy at moment... hpx7 and I have had some brief chats about what it entails

after talking with @hpx7 this is the proposal ive come to
hathora.yml:
SomeType:
  foo: string
events:
  myEvent: SomeType
  myOtherEvent: SomeOtherType

api types:
enum HathoraEvents{
  myEvent,
  myOtherEvent,
}

server:
ctx.sendEvent(HathoraEvents.myEvent, {foo: "bar"}, userId)

client:
export type HathoraEvent = {type: HathoraEvent.myEvent; val: SomeType} | {type: HathoraEvent.myEvent; val: SomeOtherType}

@jyoung4242 if you see any issue with this let us know :) I can't see any issues but thats not to say there arnt any

I think it's perfect but would we consider adding this to broadcastEvents as well?
That was I don't have to loop through each client and to a sendEvent

@jyoung4242, I'm hoping @hpx7 has an opinion on that, I'm open to looking into that as well, I got slightly busy the last day or two and haven't been able to actually start implementing it yet.

Yeah for broadcast it would just be
ctx.broadcastEvent(HathoraEvents.myEvent, {foo: "bar"})

@IanSSenne , yup, i'm on board... lemme know if you need any assistance... i will be traveling most of next week

@jyoung4242 just wanted to bring you up to speed on what I have managed so far, the types for Event Type, Event Payload Type, and the Context have been updated or created using hathora gen, the next thing to do is client side type updates (just need to change a function signature) and the server side implementation. I may be missing something but so far its going ok.

@IanSSenne , that's fantastic! Looking forward to testing it out

@jyoung4242 I have a working implementation at this point, just has a couple of quarks I'm waiting to hear back from hpx7, feel free to take a look at the implementation here https://github.com/IanSSenne/hathora-builder-events and let me know if you can see any glaring issues :)

Fixed in https://github.com/hathora/builder/pull/418


PDF Generation

PDF Generation will be used for contract and certificate for donation when a user donates.

@PepiPetrov Let's discuss the possible PDF lib options here prior the development


Capstone project PR

Project requirements

Link to the loom live video:
https://www.loom.com/share/9a656150e0a54ea6ad80fbcc8da810db
To follow the figma template design guides including:

Colors.✅
Typography.✅
Layout: composition and space between elements.✅
The home page.✅
The about page.✅

Each of these pages should have versions for 2 different screen sizes:
-Mobile: up to 768px wide.✅
-Desktop: 768px or wider.✅
-Interactions✅
Links:

The home page should have a link in the menu to the about page.✅
The logo in the header links to the home page.✅
Mobile menu✅
When the user clicks (or taps) the hamburger button on the header, the mobile menu appears over the page.✅
The mobile menu should have a close (X) button that closes the menu.✅

The section "Featured speakers" should be created dynamically in JavaScript.
-You should use a JavaScript variable with the data about the speakers and use it when the page loads to create the HTML for this section dynamically.✅
Hope you like it! 😎👍

@DamianDib9
Mr. Damian
Its fantastic project, that's ownsome for me.
Please always try to keep grow up your good skills,
🎉congratulations for approval 🎡


Safe shutdown script is working but POWER LED does not work.

I have the US Case version. The one with the purple switches. The thing is even after installing the safe shutdown script, the power led of the case doesn't work.
Does anyone have a solution for that? Thank you.

Did you flip the switch inside the case?

My case has the same issue @foxtochop Did you ever find a solution? As soon as I turn safe shutdown switch to on, the power LED never works.

I’m also having the same problem. As soon as I flip the switch, the front LED powers off and vice versa. This behavior occurs with and without the safe script installed.

Seems someone had a similar issue and resolved it themselves in #122. Unsure how they edited the script - if they uninstalled, edited the script, and reinstalled OR if they edited a file or files on their pi after installing from Github. Any help is appreciated, I'm having the same issue and would like to try the solution in #122.


[Question]: how can infinity support full_text index for other language like chinese?

Describe your problem
It performed very well when I use full text index on columns with english text. However, it returns always empty result when used on chinese text  column.

Hi, you could specify the parameter of  analyzer to "chinese", and put the analyzer dictionary to correct path.  The dictionary lies in another repo resource, and should be put according to suitable configuration.

@yingfeng many thx for your quick reply, it works well.


Fix code example to match text body in the documentation

Since the text reads "Or dynamically calculate the color" I assume the example is meant to show this.
I confirmed the updated example works.

Thanks!


How to get playbackPosition out of a generator?

Hi
How to get playbackPosition out of a generator? I tried
final val = generator.playbackPosition;
And
final val = generator.playback_position;
However, None of this works, Did it get renamed to something else?
Regards.

Hi,
You can't find it, because apparently I forgot to add it.
I'll do that later.
From the way you're writing your issues, it sounds like you might not have
found the docs https://pub.dev/documentation/dart_synthizer/latest/.
Thank you for reporting, keep 'em coming until there's none left!
Take care,
Chris Norman
On Wed, 29 Sept 2021 at 16:07, mohamedSulaimanAlmarzooqi <
@.***> wrote:

Hi
How to get playbackPosition out of a generator? I tried
final val = generator.playbackPosition;
And
final val = generator.playback_position;
However, None of this works, Did it get renamed to something else?
Regards.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/chrisnorman7/dart_synthizer/issues/2, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABBUZ2KRSHFFWEQVMMNYR3LUEMTT7ANCNFSM5FADZJCA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Ah, Thanks, Useful.

Fixed with commit 44e0574ae0c4ac9b9c1a9fa3ec73975b3e21d90d.


Indexing: handle cpu & single-gpu without using multiprocessing & dist. data parallel

Looking good to me at first glance! Thank you so much

This fixed a problem I was having downstream in RAGatouille, always running in distributed mode on wsl2 with a single gpu. Thank you for the PR!
https://github.com/bclavie/RAGatouille/issues/30#issuecomment-1889669431


Implement OpenTracing across all Dispatch components

[ ] Bug
[x] Feature
[x] Enhancement

Detailed Description
Add OpenTracing support to other Dispatch components (function manager, secret store, image manager, API manager, identity manager).
Context
OpenTracing is an open interface for implementing distributed tracing. It enables request tracking by showing the path of the request as it passes all the services that take part in handling the request.
It should be useful in debugging, performance measurements, finding bottlenecks and security issues.
Possible Implementation
#244 introduced partial support for OpenTracing in event manager. To complete it, other services need to inject tracing spans. Also, an opentracing backend (like Jaeger or Zipkin) must be added, and services must be configured to use it.
Complexity

[ ] Low - Simple enhancement or bug fix, no architectural changes or refactoring
[ ] Medium - Change requires some thought, but is relatively isolated
[x] High - Significant architectural change or large refactor

Impact

[ ] Low - Annoyance, but does not impact business or functionality
[x] Medium - Issue can be worked around, but is causing pain
[ ] High - Blocker

#410 added backend installation, still needs tracing between services

This is completed with #410 and #461


How to generate example file?

rasa NLU version : 0.7.3
Used backend / pipeline (mitie, spacy_sklearn, ...): [“nlp_mitie”, “tokenizer_jieba”, “ner_mitie”, “ner_synonyms”, “intent_featurizer_mitie”, “intent_classifier_sklearn”]
Operating system:  Centos
Issue:
Should I have to generate example json file(e.g. "data/examples/rasa/demo-rasa_zh.json") by manual?
Content of configuration file (if used & relevant):

Yes, this json file is for training your own NLU model, so you have to generate as much data as possible in it.

I see, thanks.


bad sample for CString

you wrote:
CStringW str1;
CStringW str2;
str1.Format(L"%s", static_cast<LPCWSTR>(str2));
but in this way You throw away all the beauty of CString portability for multibyte/unicode:
do in THIS way:
CString str1;
CString str2 = _T("hello!");
str1.Format(_T("%s"), static_cast<LPCTSTR>(str2));

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e52d20ec-2725-57ac-55ad-6584017059ff
Version Independent ID: 38807675-ee62-3472-653b-d2d5d2433d45
Content: C++ conformance improvements
Content Source: docs/cpp-conformance-improvements-2017.md
Product: visual-cpp
GitHub Login: @mikeblome
Microsoft Alias: mblome

@ingconti Suggestion accepted! will appear by 6-19. Thanks.


Editor - need to implement errors that come back from the meta service

This is a...

[ ] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report
[ ] Documentation issue or request

Description
When invalid input is received the backend throws an error and I believe sets some message in the _meta attribute in the response somewhere.  The old code caught this and then showed the message like this using this template and we should too.

To reproduce:

And hit next:

So we need to catch this and instead show an alert with info from the meta service.


Extend Travis CI build matrix to cover ARM

Make sure the build does not break on different architectures to avoid problems as in pull request #419.

I did some investigation towards armh builds.
currently softethervpn is not friendly for cross-builds at all.
so, we have to choose from

do nothing
modernize build system to support cross builds (and use travis-ci)
upload sources to launchpad (it builds natively as far as I seen)

armhf builds for all supported Ubuntu version (https://code.launchpad.net/~paskal-07/+archive/ubuntu/softethervpn/+packages) on Launchpad.
I think this issue should be reviewed, a lot changed since March.

Is ARM support still something to be considered? Since macOS moved cmpletely to ARM and Windows also supports it now I think I should be considered.

I enabled macos-14 builds recently
https://github.com/SoftEtherVPN/SoftEtherVPN/actions/runs/8995375739
I think, we can close this issue. Travis-CI is not an option anymore (unless someone is going to invest into it)

as for Windows on ARM, currently it is not implemented in any public CI/CD unfortunately

I enabled macos-14 builds recently

Any issue I can subscribe to, to receive a notification when a ARM build for macOS is available, or will it be on the GH release page?

as for Windows on ARM, currently it is not implemented in any public CI/CD unfortunately

Oh ok, so this is something somebody needs to compile locally from source.

not sure what you mean by subscribing an issue. macos-14 is enabled

Sometimes authors make issues for thinks like that to provide feedback about the process until a release build is made.
Yes I saw that already but I cannot download the build from there, so I was asking if it will be available on the release page here or somewhere else. Because I can subscribe to release here on GH to be notified :)

scope of this particular issue was adding CI on ARM. it was done.
I didn't mean ARM releases (at least within this issue)
currently, with the help of @barracuda156 , releases are done on MacPorts https://github.com/SoftEtherVPN/SoftEtherVPN/pull/1884#issuecomment-1679671617
(we still did not add any link to them)
maybe it makes sense to extend macos-14 build to releases, if you have an appepitite for that, you are welcome to contribute


CC 1.0.0 - Updating quick start and supported features [4.0.0]

Purpose

Updated the quick start guide
Updated the logos of docker and K8s
Updated the supported features

Outputs:

Please apply the same fixes to the PR going into the master branch.

Changes are done with https://github.com/wso2/docs-apim/pull/5012/commits/877891c33aed33fd26768308653306d191562895

I have done the requested changes in the master branch PR as well.


Remove the extends operator from the language

The extends operator allowed an arbitrary function to inherit from
another arbitrary function. As the same logic is no longer being used to
implement class inheritance the value of maintaining the extends
operator is up for debate.

It may not add anything now that we have proper classes, but removing it would seem to break backward compatibility for no real gain.

I don't feel strongly either way. It just feels like quite a fringe feature when it's not also used for classes (e.g. proposals for things like merge operators, mixins, etc. tend to be closed with "this would be better fulfilled by a library").
As it doesn't impact https://github.com/jashkenas/coffeescript/pull/4354 I'll close this for now and leave the branch in case we want to revisit it later.

I guess the question is, how often is it used in the wild? To extend a regular function, I mean, not a class. If it’s undocumented and almost-never-used, then sure, let’s kill it. But if it could break lots of people’s code, that seems a high cost to pay for cleanup.

It seems it is documented:

The extends operator helps with proper prototype setup, and can be used to create an inheritance chain between any pair of constructor functions.

The only examples containing extends are class examples, though.


Add docker image and k8s configurations for testing on Travis

Signed-off-by: terrytangyuan<EMAIL_ADDRESS>
What this PR does / why we need it:
Part of #17. This PR adds necessary Dockerfile, k8s configurations. Note that the result of  kubectl apply -f deploy/ftlib-test.yaml won't be verified here yet. We'll add follow up PR to check the deployment status.
Which issue(s) this PR is related to (optional, link to 3rd issue(s)):
Fixes #
Reference to #

Special notes for your reviewer:
/cc @your-reviewer

Release note:

NONE

/lgtm
On Tue, Dec 10, 2019 at 21:29 Yuan Tang<EMAIL_ADDRESS>wrote:

@terrytangyuan commented on this pull request.
In deploy/ftlib-test.yaml
https://github.com/caicloud/ftlib/pull/24#discussion_r356035741:

 app: ftlib-test-app

template:
metadata:

 labels:

   app: ftlib-test-app

spec:

 containers:

   - image: ftlib:deploy

     imagePullPolicy: Never

     name: ftlib-test-c

     ports:

     - containerPort: 3000

       name: http

     - containerPort: 22

       name: ssh

     command: ["python3", "/ftlib/test/generic_test.py"]

Good call. I updated this to pytorch test. We haven't added the validation
logic around this yet. Testing the fault-tolerant part can be a bit tricky
that requires some thoughts.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/caicloud/ftlib/pull/24?email_source=notifications&email_token=AK7V6ITGI6LJQBBWXLT72HLQX6KU3A5CNFSM4JYLSLT2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCOUNSLY#discussion_r356035741,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AK7V6IR2UPAEN7XLMUA2PDTQX6KU3ANCNFSM4JYLSLTQ
.


Allow querying with atoms in Ecto.Enum

Solves #3412
I think I've got the basic scenarios working as expected.
I'm still getting familiar with the codebase though, so there are probably missing pieces and edge cases I haven't covered.
Thank you for letting me try it!
PS: atoms are a bit tricky because they are used for built-in like booleans and nil as well, so I'm a bit afraid of a potential unintended side effect.

This is a great first step! I have added two comments, if for some reason doing the changes I mentioned are too complex, let me know and I will take a look at it!

So it was a simply fix after error: the errors are raised on normalize, no plan. Changed that fixed it. I have pushed it to master, thank you!

Thanks @josevalim ! :purple_heart:
I totally missed normalize, this is a much more satisfying answer indeed.


使用kubernetes部署时 pg 数据库和 mongo 数据库没有容器化

    使用kubernetes 部署时，pg数据库和mongo数据库没有容器化安装方法，只给了各自的官方指导文档，不方便用户的安装，增加了安装难度

The service of database is a very complex issue. How to ensure its HA, containerization is not necessarily the best choice. It is not recommended to give containerization as a standardized installation.

Database installation is a standard process, please refer to the corresponding official documentation.


What do  and  mean in Program?

I assume that <b> splits functions, and <c> splits arguments of functions, is it true?
Thanks for your reply.

Exactly~

I haven't actually experimented to the stage of generating results, so there's a question.
The transformer tokenizer, i.e., BART tokenizer here, generally separates symbols like < > from words.
Is there any post-process for seq2seq results to eliminate the side effect, or deal with the possible ill-format SPARQL or Program?

For SPARQL，there is  post-process as shown in https://github.com/shijx12/KQAPro_Baselines/blob/2c5049900ba11947daf32cffcdfbdb4e985f1108/Bart_SPARQL/predict.py#L90.


Problem Button on Smart Lock 4 break

Problem Button on Smart Lock 4 Pro ( when Button used to Unlock/Lock its work but notification not work ( notification are not sent to Nuki App or apple home or Home assistan  ) To update the lock status.

I can reproduce this too on the Smark Lock 4 (not the Pro version).


[BEAM-13491] [Playground] Examples' catalog doesn't close after selecting example

Now example catalog is closing when the user select an example from it.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

LGTM Thanks!


Without @ComponentScan - how?

Quick question: can I use springfox swagger without @ComponentScan annotation?
If so - how? (2.5.0 springfox & springboot 1.3+)

Could you clarify? without @ComponentScan where? If you're talking about not using @EnableSwagger2 annotation then its possible Only you'd have to do all of the bean registration yourself. I believe someone has done it already

Here is the issue where @SergiiShapoval has an example with direct bean registration

@EnableSwagger2 is ok, but we create beans "manually" (in @Configuration components, with constructor arguments) and hence does not use @ComponentScan (Less magic - more control)
I looked at the linked issue but cant understand how to do the "direct bean registration"?

@achoice Check out this fork especially how he's done the direct bean registration

@dilipkrish in that example I still see
@ComponentScan(basePackageClasses = PetController.class)I am OK enabling Componentscan for Swagger/springfox-stuff if needed. But I do want to create the actual rest-controller myself. I expected something like;docket.addController(restController) // manually add a controller for Swagger to document`

Ah ok. You can register the controller yourself as a bean. No need to add it specifically to the docket. It will automatically be picked up.
// In your config

@Bean
public PetController pets() {
    return new PetContoller(...);
}

thats it. Springfox will pick it up as long as its a controller.

Thanks again for prompt reply. server-error-controller is picked up. I now see we create our controllers in separate sub-web-application-contexts. Perhaps that confuses/ makes them invisible for Springfox. (This complexity is to separate context for internal/ external use / security stuff.)
`
/* ProviderNotificationConfiguration creates the Controller as a bean /
@Bean
public ServletRegistrationBean providerCallbackApiServletRegistrationBean() {
return createChildContext(ProviderNotificationConfiguration.class,
"providerCallbackApiServletRegistrationBean",
"/entercash/provider-api/");
}
private ServletRegistrationBean createChildContext(Class context, String name, String... urlMappings) {
/* create the context */
AnnotationConfigWebApplicationContext annotationConfigWebApplicationContext = new AnnotationConfigWebApplicationContext();
annotationConfigWebApplicationContext.register(context);
annotationConfigWebApplicationContext.setParent(this.applicationContext);
/* create the dispatcher servlet */
DispatcherServlet dispatcherServlet = new DispatcherServlet();
dispatcherServlet.setDetectAllHandlerExceptionResolvers(false);
dispatcherServlet.setApplicationContext(annotationConfigWebApplicationContext);

/* create the servlet registration bean */
ServletRegistrationBean servletRegistrationBean = new ServletRegistrationBean(dispatcherServlet, urlMappings);
servletRegistrationBean.setName(name);
servletRegistrationBean.setLoadOnStartup(1);
return servletRegistrationBean;

}
//...
`

@achoice yeah it is complex. Perhaps if you can isolate your problem to a small working sample repo and share I could take a look


CSGO Timer Dont Work

Hello, i just fixed my bot and 2 in my jackpot went in and the timer dont start and the skins is stuck how do you change it?

add me on Steam: http://steamcommunity.com/id/B0LT3R/ i might can help you


Update klipper-lb image version

Proposed Changes

Update klipper-lb image version to v0.4.6
Types of Changes

Version bump
Verification

Verify that klipper-lb image works by deploying a service of type external
Testing

Linked Issues

https://github.com/k3s-io/k3s/issues/9487
User-Facing Change

Further Comments

Appears there may be something wrong with the new image:
  svclb-traefik-4fadc231-4f5fh
  Expected
      <string>: CrashLoopBackOff
  to equal
      <string>: Running

Appears there may be something wrong with the new image:
  svclb-traefik-4fadc231-4f5fh
  Expected
      <string>: CrashLoopBackOff
  to equal
      <string>: Running

It works OK for me locally, but is consistently failing the e2e tests; I'm not sure why.

I'll investigate when I finish the multus integration

Error identified and valid. Great work CI! It can't find iptables in OS using iptables-legacy as default (e.g. what CI uses, Ubuntu 20). Probably the reason is that Alpine 3.19 switched to iptables-nft by default. Need more investigation

I suspect we need to add iptables-legacy to https://github.com/k3s-io/klipper-lb/blob/master/package/Dockerfile#L5

I suspect we need to add iptables-legacy to https://github.com/k3s-io/klipper-lb/blob/master/package/Dockerfile#L5

Yes, that''s probably the best solution. x-tables-legacy-multi does not exist in 3.19 (after installing iptables), that's why it fails. The entry script requires it. I'm going to create a PR and once merged, I'll create v0.4.7 and declare v0.4.6 broken

https://github.com/k3s-io/klipper-lb/pull/64


850/rest filter boxes on reset map

I edited the location-list-map.js so we can reference the Selector objects for the selected filters any time we select/deselect them. This allows us to go through these objects and deselect them upon resetting the map.

Needs review, currently storing reference to Selector objects in the DOM but according to the second comment on this stack overflow page (https://stackoverflow.com/questions/1402693/is-there-a-good-way-to-attach-javascript-objects-to-html-elements), it can lead to memory leakage.

Let's go with it for now. The comment indicates it may leak. I would guess this happens in a refcount cycle if the javascript Object or closure holds the element and the element now refers to the Object.
However this is only a problem if we're removing the element from the dom. Since that's never happening, then when the Javascript "World" in Chrome is torn down along with the Document, all of it is dropped.
The real problem design wise is the code doesn't allow for us to register a map (eg ES6's WeakMap) without making it a module global.
If this annoys someone enough, they can refactor it later.


ReferenceError: Cannot access 'WEBPACK_DEFAULT_EXPORT' before initialization

Getting this while using https://giraphql.com/plugins/complexity
Happening while getting introspection query

/cc @hayes

https://gitlab.com/yo/devparty/-/merge_requests/862/diffs

this should be fixed now


BleConnectionCompat ignores autoConnect value passed to it in some situations

Summary
On BleConnectionCompat.java the boolean autoConnect passed to it's connectGatt method is only taken into account if calling native implementation if SDK_INT >= Nougat
Further down that class there's on lines 60, 67, 70, and 85 calls to connectGatt implementations that simply pass true instead.
Maybe that's a specific hack for some broken Android version, but there's no code or commit comment about it. Hence I'm opening a issue, instead of Pull Request.
If it's a real issue, I'll be happy to open a PR

Thanks for analyzing the code :)
The algorithm is following:

For Android < N or if autoconnect == true => go with a reflection-based setup, this fixes race condition related to a connection setup. Therefore true is forcefully passed.

Otherwise, use connectGattCompat and
2.1. For Android M+, use new API to force connection over Bluetooth LE (the device may support LE and classic BT at the same time)
2.2. For older devices use TRANSPORT_AUTO

Feel free to reopen if you find anything else that is related.

Thanks for the quick reply, I guess I missed the || !autoConnect part of the if statement.
So that means that for autoconnect = false it's just a matter of using or not the TRANSPORT_LE flag.
Thaks

100% correct, the issue was related to autoConnect = true.


vertical menu with folder metisFolder as child

I am trying to combine the vertical menu and folder together.
The result is not good because folder does not get the styles needed. i.e. all the folder nodes are shifted to the right.
What is the way to resolve this?

Home

				Ajax Pte Ltd

                    Facility 1

                        Collateral 1

                            Doc 1

@iramtay see this.
https://jsfiddle.net/orwzdawz/

Hi,
maybe my post is not accurate enough.
please find the screenshots that will be easier for you to understand.
01: this is the existing CSS that i have, and i wanted to migrate to use metismenu
02: this is the result of the migration. the normal menu is okay except for the folder tree.      folder tree is affected by the style from menubar and menubar-nav      the nodes of the folder tree should be on the left side
02a: to show the different metimenu CSS used
Regards.
Chee Chong
  From: Osman Nuri Okumuş<EMAIL_ADDRESS>
To: onokumus/metisMenu<EMAIL_ADDRESS>Cc: iramtay<EMAIL_ADDRESS>Sent: Saturday, February 6, 2016 8:05 PM
Subject: Re: [metisMenu] vertical menu with folder metisFolder as child (#83)
@iramtay see this.
https://jsfiddle.net/orwzdawz/—
Reply to this email directly or view it on GitHub.


Support parsing GUID ids from rawLogParser

Request Ids can be either strings (GUIDs) or numbers. The raw log parser fails if it encounters GUIDs.
This PR modifies the regex to allow that, and adds a test to make sure it is parsed correctly.
For the API in question:
https://github.com/Microsoft/vscode-languageserver-node/blob/master/jsonrpc/src/messages.ts

/**
 * Request message
 */
export interface RequestMessage extends Message {
	/**
	 * The request id.
	 */
	id: number | string;

        // other members.....
}

/**
 * A response message.
 */
export interface ResponseMessage extends Message {
	/**
	 * The request id.
	 */
	id: number | string | null;

        // other members.....
}

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: OmarTawfik sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

cc @octref for review.

@octref any chance that gets deployed soon?

Sorry for the late response.
@dbaeumer I don't see it on the spec. Can request id be GUID?

The request id can be a string or a number according to JSON RPC. So if the GUID is passed as a string the answer is yes.


 At some point hundreds of these messages per hour. Note that this application is handling multiple multiple livekit rooms at the same time, each with 1-5 video streams & 2 audio streams.

The error seems to originate from the pion turn project, which is used by Pion Webrtc. From what I read in the the code of pion turn, it seems this error happens when the turn server does not respond to the turn createpermission request. Since were not seeing any failed to refresh allocation messages, my guess is that somehow the TURN connection is still alive? It's weird.
I cannot see anything weird in the actual video streams. But I'm worried that these errors are a sign of a leftover connection or something that's not cleaned up properly.

This is still an issue. At this time we are seeing this error continuously, about 13000 times per day.

sorry we have not had a chance to look into this one. thanks for the heads up.

We are still seeing about 19k of these errors every day.
Could anyone maybe point me into the direction of where to start debugging? I'm happy to add extra logs or do some experimentation. But I have no idea where to even start looking. Guidance would be much appreciated.

We are now at ~43k of these error messages every day. The growth may be related to our grows in number of rooms & tracks.

1, we facing the same error


Unresolve icon

Resolve/unresolve is now a curved arrow to imply "unresolve comment"

Wrong target branch

Wrong target branch


Test testing logic

If we retry the URL only, seems to fix the tests in their original state.

@dcousens Ideally I'd like to merge this, but I already got it working in the current main branch so it doesn't make much sense, then again I should just review the tests in general and improve them, we should discuss at some point.


#145 immersion UI scaffold page layout

I should probably just keep banging out the keyboard  stuff here, but wanted to share progress @alexfarrill

scaffolds compact and client view
adds tailwind css

TODO:

keyboard controls
styling
make index redirect to first preview

@alexfarrill want to merge this into v0.8.0 so we can review smaller PR's?
I would consider #145 done here :)

Yeah, good idea


Release v2.1.8

Codecov Report
Attention: 192 lines in your changes are missing coverage. Please review.

Comparison is base (dedeb3b) 50.75% compared to head (16a3a84) 38.56%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@              Coverage Diff              @@
##             master     #188       +/-   ##
=============================================
- Coverage     50.75%   38.56%   -12.20%
- Complexity     1080     1100       +20
=============================================
  Files           172      237       +65
  Lines          7083     9437     +2354
  Branches        715      910      +195
=============================================
+ Hits           3595     3639       +44
- Misses         3235     5545     +2310
  Partials        253      253

Files
Coverage Δ

...ain/java/tech/ydb/table/impl/pool/SessionPool.java
97.14% <100.00%> (ø)

...va/tech/ydb/topic/read/DecompressionException.java
0.00% <ø> (ø)

...ava/tech/ydb/topic/read/impl/OffsetsRangeImpl.java
100.00% <100.00%> (ø)

...ad/impl/events/StartPartitionSessionEventImpl.java
0.00% <ø> (ø)

...in/java/tech/ydb/topic/read/DeferredCommitter.java
0.00% <0.00%> (ø)

...db/topic/read/events/AbstractReadEventHandler.java
0.00% <0.00%> (ø)

...ch/ydb/topic/read/impl/DisjointOffsetRangeSet.java
93.75% <93.75%> (ø)

.../topic/settings/StartPartitionSessionSettings.java
0.00% <0.00%> (ø)

.../java/tech/ydb/topic/read/impl/SyncReaderImpl.java
0.00% <0.00%> (ø)

...ain/java/tech/ydb/topic/read/impl/MessageImpl.java
0.00% <0.00%> (ø)

... and 9 more

... and 47 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Table reload behaviour

OpenUI5 version: 1.26.7
Browser/version (+device/version):
Google Chrome Version 41.0.2272.101 m
Any other tested browsers/devices(OK/FAIL):
Internet Explorer, Firefox
URL (minimal example if possible):
User/password (if required and possible - do not post any confidential information here):
Steps to reproduce the problem:

Load table with no data
No data label is not shown

What is the expected result?
No ldata label should be shown
What happens instead?
Just a blank table row is shown
Any other information? (attach screenshot if possible)

Hi,
i tried to put a Table with no data
http://jsbin.com/vixiricabi/1/edit?html,output
the label shows, its 1.26.9 though
Could you try reproducing your issue in the jsbin?
Best regards,
Tobias

We are loading a JSONModel with data from our webservice (using AJAX). After the AJAX request is complete, we set the data to the model. But the table seems to not update correctly. This is appearing in version > 1.24.5
It is working in versions <= 1.24.5 though.
I try to call a webservice to fill data with jsbin.

Hi Adrian,
I tried Tobias' jsbin with 1.24.6 and it also works...
Should this jsbin exhibit the issue? Do you also see it with 1.24.6?
Your second comment sounds quite different than the original one (it mentions data, while the first one was about no data), please clarify.
Regards
Andreas

No response in several months, not reproducible -> closing.


BulletList formatted incorrectly on insertContent

Describe the bug
When calling insertContent with bullet point markdown (for both numbering and bullet points), an extra bullet point/number is added at the beginning of the list.
To Reproduce
Pass the following input string into editor.commands.insertContent:
Title: The Importance of A/B Testing for SEO Success: Boosting Your Brand's Ranking on Google

Introduction:
- Briefly introduce the concept of A/B testing and its significance in the world of SEO.
- Highlight the potential impact of A/B testing on improving a brand's visibility and ranking on Google.

1. Understanding A/B Testing in SEO:
- Explain what A/B testing entails in the context of SEO.
- Discuss its purpose, which is to compare and contrast different versions of website elements to identify the one that performs better in terms of SEO metrics.
- Highlight the two variants involved in A/B testing: the control (original) version and the variant (newly implemented) version.

Expected behavior
Extra bullet/number should not appear before each list. Should work the same way as it does with setContent
Screenshots

Desktop (please complete the following information):

Browser: Chrome
Version: 0.8.3

I'm having the same issue

Fixed in v0.8.7


Make dev rpms update

Developer Environment: Added dependencies to make dev-env and documented.
This PR has added openssl required by make generate_local_pg_tls_cert.
This PR has added a known issue in python-ldap using OpenLDAP 2.5+ to build.

@costaconrado as discussed via chat, the commits should use the imperative form before it can be merged


pyecmd: Use correct relative import for python3 in base.py

@thejta any issue with this?

Sorry - didn't catch I was an approver.  No issues come to mind, I assume it had basic tests done.


add QuerySolution()

Codecov Report

Merging #132 (1eb3197) into main (1a5f9b7) will increase coverage by 0.05%.
The diff coverage is 88.00%.

@@            Coverage Diff             @@
##             main     #132      +/-   ##
==========================================
+ Coverage   88.95%   89.00%   +0.05%
==========================================
  Files          18       18
  Lines        5050     5075      +25
==========================================
+ Hits         4492     4517      +25
  Misses        484      484
  Partials       74       74

Impacted Files
Coverage Δ

solutions.go
97.08% <75.00%> (-2.92%)
:arrow_down:

interpreter.go
94.30% <100.00%> (+2.07%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a5f9b7...1eb3197. Read the comment docs.


Send MP for Validation

As a user I want to be able to send a MP for validation so that I can ensure that the data is compliant
AC
Given all formatting checks are resolved
When the user sends for validation
Then the MP is stored for check engine validation

Duplicate of #171

Duplicate of #171


Table alignment problems, depending on font

I'm trying to create a Node based on a table, with left-aligned lines. However, even with .align(Labeljust.LEFT), I can't seem to create consistently aligned lines. The results vary depending on the used font.
Creation of the node:
String fontName = "...";
Html.Table table = table()
   .border( 1 )
   .cellBorder( 0 )
   .cellSpacing( 0 )
   .cellPadding( 4 )
   .tr( td().text( "Test test test" ).fontName( fontName ).align( Labeljust.CENTER ) )
   .tr( td().text( "Another test test" ).fontName( fontName ).align( Labeljust.CENTER ) )
   .tr( td().cellPadding( 1 ).bgColor( Color.BLACK ).height( 1 ) )
   .tr( td().text( "someProperty: Test test123" ).fontName( fontName ).align( Labeljust.LEFT ) )
   .tr( td().text( "description: This is a test description" ).fontName( fontName ).align( Labeljust.LEFT ) )
   .tr( td().text( "anotherProperty: Another12" ).fontName( fontName ).align( Labeljust.LEFT ) )
   .tr( td().text( "see: http://example.com/" ).fontName( fontName ).align( Labeljust.LEFT ) )
   .tr( td().text( "yetAnother: yetAnotherTest" ).fontName( fontName ).align( Labeljust.LEFT ) );
Node node = Node.builder().color( Color.BLACK ).table( table ).build();

The results, for various fonts tried:
Roboto Condensed:

Arial:

Noto Sans:

DejaVu Sans:

Is there something I can do to force multiple lines to be properly left aligned?

This is a known issue in the schedule task. The font from svg needs to be mapped with the font from java.awt, otherwise there will be some errors when measuring the text length.

I tried the following:
// java.io.File
// java.awt.Font
// java.awt.GraphicsEnvironment
File fontFile = new File("RobotoCondensed.ttf");
Font font = Font.createFont(Font.TRUETYPE_FONT, fontFile);
GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment();
ge.registerFont(font);

to make the font known to AWT before generating the SVG, but unfortunately it made no difference.

Please take it from the font name below:

String[] fonts = GraphicsEnvironment.getLocalGraphicsEnvironment().getAvailableFontFamilyNames();
for (String f : fonts) {
  System.out.println(f);
}

2.Please confirm whether the machine that browses the svg is the same as the machine that generated the svg. The fonts of different machines may not be the same, and the results of the svg generated on the server to the svg renderer of different machines may be different.

I have tried that as well (before explictly registering the font in the AWT GraphicsEnvironment). getAvailableFontFamilyNames() contains all the fonts all the font names I've tried in the generation. I'm doing all tests locally, i.e, the machine generating the SVG is the same machine that is viewing the SVG.

I can not reproduce on my local env, would you mind provide env info and font output?

I'm running Ubuntu Linux 20.04.5, with Xfce 4.14. Adoptium/Temurin JDK 17.0.8.
The output of GraphicsEnvironment.getLocalGraphicsEnvironment().getAvailableFontFamilyNames() has over 2000 entries on my machine, but it contains the fonts:

Please let me know if there is other specific info that would help you.

This problem is due to the inaccurate measurement of FontMetrics in some fonts. At present, I have not found a better way to measure the length of the string. If you have any local more accurate test methods, you can register the implementation of MeasureText locally to override someone's. method to measure the length of the string, or you can try to use the font with the most accurate measurement.

We supported export image directly in latest version and the more accuracy font name detect because render and measure at the same way.

1.3.1 most flexible font family select policy


Revert "Re-enable JVMTI test WaitNotifySuspendedVThreadTest"

Reverts adoptium/aqa-tests#4558

The FIX for https://github.com/eclipse-openj9/openj9/issues/16689 is being reverted since it caused side-effects (unexpected failures). Thus, adding back the exclude for https://github.com/eclipse-openj9/openj9/issues/16689.
fyi @llxia @karianna


housekeeping: Add Basic GitHub Actions Configuration

This PR adds a basic GitHub Actions configuration which builds the project and runs the tests. Closes https://github.com/worldbeater/XamlNameReferenceGenerator/issues/1

Looks like we are unable to build the project via dotnet build because the sources aren't generated by the generator.

The release process now works like this: we open a PR, we add a release label, and once we merge the PR, a new version of the XamlNameReferenceGenerator ships to NuGet. This is how the flow should work at least, probably worth trying to ship the first version after this PR is merged. Also with this PR, we are using Nerdbank.GitVersioning for incrementing assembly file versions.


ERROR: fseek() stream does not support seeking while not using BrainMonkey

I wanted to create a test suit for Wordpress which includes lucatume/WP-Browser, BrainMonkey and Robo. I put them in the same composer vendor. Robo is a tool for executing commands, when I tried to test a tiny command in Robo, this error occured. And I tested some other commands but this error appeared after each execution. The strange thing is, I just didn't use any fonction of BrainMonkey. Here is the screenshot for the error :

Then I tried to remove dependencies, when I removed BrainMonkey 2.0.2, everything worked fine. I also tried BrainMonkey 1.5.0 and this one works fine.
Here are the dependencies of my composer.json file:

I really want to use BrainMonkey 2.0, please help me out, thank you.

@zengheng3 I have been reported of some issues by loading Patchwork from autoloader.
Your case seems another possible issue of this kind.
I just pushed a branch named patchwork-load-env whic allow to disable the autoload of Patchwork by setting an environment variable: BRAINMONKEY_PATCHWORK_NO_AUTOLOAD.
If you require BrainMonkey from dev-patchwork-load-env than you could try to run your Robo commands after the env var is set. e.g. something like:
export BRAINMONKEY_PATCHWORK_NO_AUTOLOAD=1 && robo some-command

If it works for you, this could be a good step to solve the issue.

I tried this solution and it works well, but I think this should be a default behavior, setting such a long variable before every execution is really annoying. I saw that lucatume/WP-Browser uses antecedent/patchwork too but this error didn't occured.

Sure. I am concerned that now removing the autoload could be a breaking change. I have to do some tests.

This is old. Brain Monkey v2 does not load Patchwork from autoload anymore.


Adjust test framework to handle single line result

Adjust the test framework to deal with single-line results that include new lines. This is necessary to test the results from EXPLAIN statements.

Discussion is still open here: https://github.com/kuzudb/kuzu/issues/1755


Update dependencies

Hi @faymanns, while working on  https://github.com/conda-forge/staged-recipes/pull/24434
We noticed the following:
The following dependencies are directly imported but not listed as a dependency
- napari
- pandas
- scikit-image
- tifffile
The following are listed as dependencies but are not imported in the source code
-  scikit-learn
Could you update this for the following release?
Thanks! 🚀

Hi @goanpeca,
I updated the dependencies in 94c867207b7bf01caea8b4999d47caa10aed1fe1.
Let me know if there is anything else that you need me to do.


unused variables

Unused variables listed. The variables memberServerName and memberDatabaseName never used any where in the script.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 856c3148-4309-2d97-d093-b29a4481cb8f
Version Independent ID: 0d057151-804f-3c7e-fc1f-be792b7e59f2
Content: PowerShell: Sync between multiple databases in Azure SQL Database - Azure SQL Database
Content Source: articles/azure-sql/database/scripts/sql-data-sync-sync-data-between-sql-databases.md
Service: sql-database
Sub-service: data-movement
GitHub Login: @stevestein
Microsoft Alias: sstein

@pbakhil Thank you for the feedback. We are actively investigating and will get back to you soon.

@pbakhil Thank you for the feedback. We are actively investigating and will get back to you soon.

@pbakhil Thanks for pointing this out. If we look at code piece (New-AzSqlSyncMember), the same command can be used for both Azure SQL Database as well as on-premise SQL Server Database. The code which is provided in article is for on-premise version and hence both those variable are not used. If we use it for Azure SQL Database then both the variables will used. The document is kind of generic code implementation hence included these.

You can refer to below article to check how can we use both the command differently:
https://docs.microsoft.com/en-us/powershell/module/az.sql/New-azSqlSyncMember?view=azps-5.4.0
Please let us know if this helps.

@pbakhil Awaiting your response on above comment.

@AnuragSharma-MSFT Thanks for the clarification.

@pbakhil We will now proceed to close this thread. If there are further questions regarding this matter, please comment and we will gladly continue the discussion.


2.0 API

<EMAIL_ADDRESS>is coming to you very soon. The signature of patch needs to change, and I need your feedback to get it right.
Closes

https://github.com/picodom/picodom/issues/80: I don't understand this issue.
https://github.com/picodom/picodom/issues/77: I will add new tests.
https://github.com/picodom/picodom/issues/92: Either one of below proposals will enable you to target a DOM element to be replaced instead of always pre-pending to a container.
https://github.com/picodom/picodom/issues/89: The "Double Espresso Steamed Soy Milk Foam" proposal below fixes this issue, but I'll likely proceed with "Bitter Tea" and close that issue as wontfix. There is no way to please everyone.
https://github.com/picodom/picodom/issues/65: I'll give a shot at adding types for VDOM events and close there.

Bitter Tea
Bring back the old pre-1.0 signature. This API is more flexible and easier to implement at the expense of a slightly worse developer experience (is it?). The justification is that Picodom is intended to be used to create tiny Hyperapp-like view frameworks instead of directly consumed. Still, this is IMO much better than similar popular alternatives.
import { h, patch } from "picodom"

// First and every other patch.
let element = patch(
  container,
  element,
  oldNode,
  newNode
)

Double Espresso Steamed Soy Milk Foam
This proposal arguably improves the surface API providing at the expense of "slightly" less flexibility. This also has out of the box SSR DOM rehydration. This API makes consuming picodom directly more fun, but a bit awkward to integrate into your own library because you need to be aware there would be essentially two ways to call patch, the first time you call it, and every other time.
SSR recycling is enabled by providing a recycleElement to patch that works the same way as https://github.com/hyperapp/hyperapp/pull/590.
import { h, patch } from "picodom"

// First patch.
let element = patch(vnode, containerElement)

// or first patch with server side rendered rehydration

let element = patch(vnode, containerElement, recycleElement)

// Every other patch.
patch(vnode, element)

/cc  @mindplay-dk @JSteunou  @estrattonbailey @pspeter3 @maxholman @dancespiele

@mindplay-dk @JSteunou  @estrattonbailey @pspeter3 @maxholman @dancespiele

I wonder if it is worth making initial patch and update patch two different methods. Right now the name could be confusing.

@pspeter3 That might be a way to make "Double Espresso Steamed Soy Milk Foam" easier to document. But I am strongly swaying in favor of "Bitter Tea".

Yeah, I personally agree: the API doesn't need to be overly concerned with developer experience.
When I was poking around a month or so ago, the main thing I wanted was a way to patch sub-trees so that state updates could be localized and not have to walk the entire tree. Would this be possible given the Bitter Tea approach?

Did you ever look at this? I've given up and have settled for Preact, but I'm positive this rewrite was headed in the right direction - I just can't dedicate the time to complete it. For one, this is fully type-hinted, so it should be much easier to work with - the test-suite is also much faster, fewer dependencies, type-hinted test-suite, inline comments, and so on.

Bring back the old pre-1.0 signature. This API is more flexible and easier to implement at the expense of a slightly worse developer experience (is it?). The justification is that Picodom is intended to be used to create tiny Preact/Hyperapp-like view frameworks instead of directly consumed.

IMO, you're bringing back at least two major problems.
First, I don't believe it's actually possible to build something Preact-like - e.g. something that supports constructors and stateful components, because the only way that happens, is if the framework is in charge of creating/destroying component instances, the same way it manages the life-cycle of DOM elements. It has to be deeply integrated. At least that's the conclusion I came to.
Secondly, the long-debated problem of deferred removals - the signature, where someone has to pass in the old state and new state, makes it very difficult "cheat" on this part, e.g. somehow "faking" a prior VDom node state of something that has already been removed. Since you need the past information to make this happen, it's much easier and simpler to start with a design where you manage the VDom state internally: don't require someone to pass in the old and new state. There are only two things you can do with that feature of the API: either correctly or incorrectly pass in the prior state - you can limit that to "correctly" by simply maintaining that state internally, and I wouldn't recommend injecting it into the VDom model (like e.g. Snabbdom) so that leaves either the DOM, or a state object of some sort, perhaps one that gets created by a mount() function; or something like jQuery's .data()... either way, putting that on the consumer of the API is asking for trouble.
Perhaps most importantly, look at the internal signature of my patch-function:
function patchTarget(
    target: TargetNode | undefined,
    parent: Element | undefined,
    nextSibling: Node | undefined,
    vNode: VChildNode,
    callbacks: Function[]): Node)

Yes, that's a lot of arguments - but they're all extremely necessary: you want to be able to either patch an existing DOM node, or have the framework create the correct node for you and position that node correctly, so there's no way around it, you need at least the target element (which may be undefined), as well as it's parent and it's next sibling for positioning. You can't have fewer arguments than that, or it will be impossible to position a created element.
It might even make sense to make the public function very similar to this, which would provide the most flexibility - e.g. same signature minus the callbacks which are an internal detail.
I can answer any question you might have about that code, I just can't dedicate the time to complete it myself. But I think, if you just start over, you're going to end up wasting a lot of time on issues I already spent wayyyy to much time on. Totally up to you. Just trying to help :-)

My diff/patch rewrite will fix the deferred problem, this proposal is just about API. And you are right, I am not trying to build something like Preact here, this is just Hyperapp's VDOM stripped so that people can play with it, customize or depend on to build their own tiny "functional" view frameworks. I think you always wanted to build something else.

@mindplay-dk You can't have fewer arguments than that, or it will be impossible to position a created element.

Haha, sure I can.

My diff/patch rewrite will fix the deferred problem

I concluded it wasn't possible, so I really look forward to seeing your solution ;-)

I think you always wanted to build something else, unfortunately.

I don't think so? You just said yourself above that you wanted people to be able to build something like Preact with this - it isn't possible without some means of maintaining state.

Yes, but I definitely had something more like Hyperapp in mind. I updated the original post to reflect this.

Replacing/updating elements instead of prepending/appending to a container?

Yep! I just wanted to contain state to a component and update the component and it's children starting from its position in the DOM, instead of having to traverse the entire tree from the top. I made a demo using my fork a while back.
I'm not sure if it would be compatible with your rewrite, but that's okay! AFAICT that's the main blocker to stateful components.

I'm not sure why some of you think a return to the older call signature would make Preact-style stateful components impossible. I used the old call signature in version 0.2.0 to do just that. That call signature was:
function(oldNode, node, element, parent, cb)

Inside my component this becomes:
// Patch DOM with component update.
this.element = patch(
  this.oldNode,
  (this.oldNode = this.render(__data)),
  this.element,
  this.container
)

My component class keeps track of oldNode, element and parent (container). node is created when the component render function is called, passing in the new state. Because the component instance keeps track of oldNode, element and parent, it passes these to patch on each update. This means, like Preact, you can create multiple stateful components in the same parent (container). Picodom updates the correct tree for that component because each component has a reference to that tree thru the element property that it gets from patch.  This is not the case when patch assumes the tree to update is always parent.children[0]. Here's a Codepen example of what I'm talking about: https://codepen.io/rbiggs/pen/ppZRWE?editors=0010
After the call signature was changed, I was never able to replicate this behavior. I tried many times. The newer call signature works fine for functional components, as long as you don't need to change the scope in which they're rendered or have only one in its own parent. With the old call signature, the component instance always has a reference to its oldNode, element and parent. This means you can define and instantiate them in one file, export them and import them into another file, where, when state changes, they will update as expected. With the simplified call signature there's no proper reference to these and so Picodom with the newer call signature winds up either recreating the entire tree from scratch, or creates a totally new tree prepended to the parent.
By the way, here's the Preact version of my Picodom multiple list example, same markup, data, etc.: https://codepen.io/rbiggs/pen/BYdQGa?editors=1111
So, I think the call signature change proposal for 2.0 is good.
By the way, I don't think anyone should try to recreate Preact with Picodom other than for fun and learning. After all, Preact is only 3KB. By the time you add everything to make Picodom work exactly like Preact, you'll probably have something much larger. In my example I made something that was Preactish/Reactish without completely replicating the paradigm. My component classes are real classes that you have to, gasp!, instantiate with the new keyword. But that's what I wanted, so I would have a reference to the component instance that I could pass around, export, import or whatever.
Maybe I'm completely misunderstanding what everyone is saying here and I need to down a couple of cups of double espresso soy milk foams. ☕☕

This proposal only affects how we mount stuff to the DOM, it doesn't (shouldn't) affect whether you
can make stateful components using Picodom or not, or am I wrong?

@estrattonbailey I just wanted to contain state to a component and update the component and it's children starting from its position in the DOM...

Hmm, and where did you want to contain the state?

I don't think anyone should try to recreate Preact with Picodom other than for fun and learning. After all, Preact is only 3KB. By the time you add everything to make Picodom work exactly like Preact, you'll probably have something much larger.

For the record, neither am I.
The main reason I took an interest in Picodom, was because of element-level life-cycle hooks, which seems much simpler and more elegant than having to write classes to achieve this control.
For this to work out in practice though, I concluded, you'd need some kind of "transparent" element - something like DocumentFragment, which represents a subtree with no common ancestor, for scenarios when you need to control the life-cycle of multiple elements that don't have a common parent. Preact has Component to address the need for life-cycle events for subtrees. JSX recently added the <> empty element, which might be a possible approach, though I'm not sure if it's possible to add attributes to an empty element? < oncreate={...}>. Hmm. Probably not.
The other major problem is the lacking ability to abort rendering of a subtree - if you have 10 date-pickers with hundreds of elements, you'll need a hook that lets you check if the value has changed, and skip rendering the entire sub-tree, without losing that subtree. Preact has shouldComponentUpdate. If this were a thing in Picodom, likely it should be implemented as another element-level life-cycle hook, in order to stay the course and maximize consistency.
These are the two hard problems that a library needs to solve, in my opinion: rendering (with full control, esp. the ability to abort) and reuse (some kind of stateful component life-cycle.)

In the case of the calendar example, it feels to me that the calendar should be its own VDOM as it is re-rendered at a different time to its surroundings.
If you can (or could?) include a raw DOM element in your VDOM, this would be quite easy to do from the API consumer end.

If this were a thing in Picodom, likely it should be implemented as another element-level life-cycle hook, in order to stay the course and maximize consistency.

We don't need one in Hyperapp as this can be solved in userland. Same for Picodom. We don't need shouldComponentUpdate, because we don't have stateful components.

We don't need one in Hyperapp as this can be solved in userland

I have seen too many wonky patterns to rely on userland to get this right.
I'm of course fine with a library not handling this - that's a matter of sticking to a defined scope, but if you expect a framework to be able to do this, the library must make that possible.
If it doesn't, then, yeah, you were probably right, maybe what I wanted was always something different, I just didn't know it yet ;-)

Just to clarify, the new JSX syntax <></> is not an empty element. It's just a shorthand convenience for using <React.Fragment></React.Fragment>, which in turn creates a documentFragment. Since documentFragments only exist in memory and are thrown away when they are inserted into the DOM, they don't have properties. They do have a few attributes, such as childNodes, etc., and methods such as appendChild.
You can't have a documentFragment that would fire an oncreate since they never get created in the DOM. Same problem with onupdate, etc. In React, lifecycle hooks are defined at the component level, not on the elements, so this isn't a problem for them.
Introducing support for documentFragment in React was not trivial. It took a lot of work from their team. Both Preact and Inferno are looking at supporting it but have been delaying it due to the amount of rewriting of the patch and render algos. Mithril does now currently support this. Personally I don't think this is worth implementing in Hyperapp and Picodom since it would require quite a bit of extra code for a trivial feature that we can live without. My opinion. The React crowd got along fine without it for years. I understand the support for it in Babel and Buble is still kind of flaky.
Maybe after other micro-libraries have managed to implement support for it you might want to take a look at how they did it.

@rbiggs You nailed it! 🙇 👍

The more a look at it, the more I like the Double Espresso Steamed Soy Milk Foam proposal. Funny how that just rolls off the tongue. I like the support for SSR too. It basically gives those who want them a mount and a render/update. They can easily write a wrapper function for each. Besides, I've never been a big fan of bitter tea. A double espresso, no problem.

Just to clarify, the new JSX syntax <></> is not an empty element. It's just a shorthand convenience for using <React.Fragment></React.Fragment>, which in turn creates a documentFragment

You're describing React - not JSX, which merely defines it as a zero-or-more list of children. The JSX spec itself defines the syntax only, not the meaning.

You can't have a documentFragment that would fire an oncreate since they never get created in the DOM. Same problem with onupdate, etc. In React, lifecycle hooks are defined at the component level, not on the elements, so this isn't a problem for them.

That's what I was trying to get at. React provides a component concept, which provides a means of controlling the life-cycle of zero-or-more children. We currently don't have something like that in Picodom, and I personally found this to be quite problematic, as there are common use-cases both for entire components that toggle on and off, which requires support for zero elements; as well as for components that require more than one element, where, presently, you need to introduce a useless wrapper
 to get the life-cycle events.
Clearly JSX fragements are not the way to solve this problem - not if we want to consistently have element-level life-cycle-events and avoid the introduction of a component concept.
The other option then is something like <React.Fragment>, a document fragment with a life-cycle.
It requires a different design for sure, but it's not necessarily substantially more complex - it's a matter of coming up with a design that uses a homogeneous model, which of course isn't easy. React wasn't able to - components and elements are something very different. That's what attracted me to Picodom - the idea of a single concept that is general enough to support all use-cases.
We didn't get there in the first attempt. In order to truly map from the JSX model to the HTML model, we have to support zero and multiple children, consistently, everywhere - because HTML does. I think that means at least some kind of fragment element with life-cycle events is required, since that's the only way to group more than one node with JSX syntax. (other than e.g. inline arrays, which look terrible.)
A concrete use-case I ran into on friday, was a color sample swatch that optionally displays a darker and lighter version of the color sample - the only way to accomplish that, was to duplicate the test:
const Swatch = (color, variations) => (
    <div class={`swatch--${color}`}>

        {variations && }
        {variations && }

);

Here, a color swatch has one sample unless variations={true}, otherwise it has three.
An extra wrapper element around the optional two darker/lighter samples is useless - it's generally unacceptable that the framework or syntax should dictate the markup you use; you should be able to use the markup you need.
With document fragments, you can do this:
const Swatch = (color, variations) => (
    <div class={`swatch--${color}`}>

        {variations && <>

        </>}

);

If the color swatches all had to float left, you might not even want a wrapper div around that component at all:
const Swatch = (color, variations) => (
    <>
        <span class={`swatch--sample color-${color}`}>
        {variations && <>

        </>}
    </>
);

If this component had a life-cycle, you could use Component and it would provide the life-cycle events for that fragment.
Now, React/Preact is far from ideal for me, on account of having two completely different concepts of elements and components, but it works - Picodom, presently, dictates the markup I have to use, like adding extra wrappers, etc.
In my opinion, a good HTML library must work equally well for all types of markup, which is why I've resigned myself to using Preact for the time being.
I'd love to see Picodom provide a markup-agnostic, homogeneous model with element-level life-cycle events, but I don't see any way of doing that without some kind of fragment concept to deal with cases of zero or multiple elements with no common ancestor?

You can return an array from components, what's your point?
BTW I am closing here since I am ready to merge #96.

@mindplay-dk IIRC Preact components can return arrays in the same way Hyperapp components can, but there's no fragment support like in React.


How to change the tab name?

under which file that i could change the words?

be more precise please


Update for v13

Discord.js now requires you to specify your intents so could you please add that to the template?

Completed


[BEAM-1763] Verify PAssert execution in runners which support metrics.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Run Spark ValidatesRunner

Run Flink ValidatesRunner

Run Dataflow ValidatesRunner

Run Apex ValidatesRunner

R: @tgroh / @kennknowles
CC: @aljoscha @amitsela @pabloem
@aljoscha Flink runner has several failures that are surfaced by these validations that need fixing. We can probably reuse fixes implemented in https://github.com/apache/beam/pull/2263.

Coverage decreased (-0.04%) to 69.789% when pulling 6ca854c9cbe3e8239945dcb1b277d89fadf5dfdd on aviemzur:testpipeline-passert-verification into b82cd2446fa681576f83bae0fc2bdd1f03be5e7e on apache:master.

Run Spark ValidatesRunner

Run Flink ValidatesRunner

Run Dataflow ValidatesRunner

Run Apex ValidatesRunner

Retest this please

Coverage decreased (-0.09%) to 69.744% when pulling e44bfec44b63ad3bc18ef114413555d286e35654 on aviemzur:testpipeline-passert-verification into b82cd2446fa681576f83bae0fc2bdd1f03be5e7e on apache:master.

@jkff SplittableDoFnTest.testOutputAfterCheckpoint() fails on the Flink Runner. If you instrument the test (with a little sysout printing in a ParDo) you see that the SDF only emits elements up to 12344. If, in BoundedSourceWrapper (which executes the Create.of("foo") on Flink), you replace the end of the run() method by
    // emit final Long.MAX_VALUE watermark, just to be sure
    // ctx.emitWatermark(new Watermark(Long.MAX_VALUE));

    while (isRunning) {
      Thread.sleep(500);
    }

you see all values correctly emitted but the test never stops because the source is now unbounded. (Interestingly, if you replace that one line by ctx.emitWatermark(new Watermark(BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis())); you get values up to 34567)
The problem seems to be that the code running the SDF sees the high watermarks and then doesn't emit stuff anymore. Do you have an idea what could be going on there? If not we'll probably have to delve deeper into the code.

Run Flink ValidatesRunner

@aljoscha SDF code does not inspect watermarks, it only sets watermark holds. For further debugging I'd recommend to add some instrumentation in OutputAndTimeBoundedSplittableProcessElementInvoker, as well as in the fn under test itself.
It is expected, in this test, with default settings of that invoker (which I don't think Flink changes), that it will output up to 12344 in one invocation of ProcessElement, and output the rest in subsequent invocations. If it's getting stalled, that means the continuation is not getting scheduled - e.g. perhaps because the watermark hold didn't get set or got lost, and Flink simply dropped the continuation as "late data".

See also ProcessFn.processElement which sets the watermark holds.

Run Dataflow ValidatesRunner

Run Dataflow ValidatesRunner

Run Flink ValidatesRunner

Run Apex ValidatesRunner

Run Spark ValidatesRunner

Coverage decreased (-0.05%) to 69.921% when pulling 0facb9e12de68d1a0515692a9355cd2467a0c131 on aviemzur:testpipeline-passert-verification into dc0fdcb7e62067fdc629b8558659fce7e37e4856 on apache:master.

What's the status of this, now that the Flink test got "fixed"?

Looks like there are failures in other tests now, perhaps these are flaky? @aljoscha can you take a look at the Flink ValidatesRunner that failed in the checks section?

Dataflow failure is unrelated as the build has been broken there for a while.
What's left to do:

Figure out why Flink tests fail (flaky?) @aljoscha
Since we moved this from Pipeline to TestPipeline as discussed with @kennknowles some of the Spark unit tests don't go through this verification because they do not use TestPipeline, I have a solution for this locally and will discuss with @amitsela

Run Flink ValidatesRunner

It's hard to know why the tests fail because it just says that only 2 out of 3 expected assertions succeeded. I think we have to add what @kennknowles once suggested: to give each assertion a unique ID and to listen on all assertions succeeding instead of just counting them.

I'm a little bit lost about the unique ID bit. I meant to have a separate downstream transform listening to each one. I wonder if your idea is even better?

@kennknowles I'm basically talking about something like this: https://github.com/apache/flink/blob/9ed98f2e5a32fb14de03b9a8ea1dd45851cc3a7e/flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/StatefulJobSavepointFrom12MigrationITCase.java#L153-L153
Each operator does one verification and updates a unique accumulator. The restoreAndExecute() listens on the accumulators and waits until each one of them is reported.
In Beam, the operators that update the Accumulator (or Metric) would be PAssert operators that each have a unique ID for their assert. The TestPipeline would then listen on the IDs that it can get using pipeline traversal.

I consider the unique-id-thing future work, by the way. I would even be in favor of merging it as is and trying to figure out what the Flink failures are as we go. The tests are not broken by these additions, we just get a better signal and that’s telling us that there seems to be some flakiness.
What do you think?

Makes sense to do it as future work. Can you open a ticket for what you were thinking of?

Run Spark ValidatesRunner

R: @amitsela PTAL at the latest commit, removed Spark runner's PipelineRule in favor of TestPipeline (since that is where the PAssert validation happens) and added a Surefire plugin execution for streaming tests (similarly to what is done in Flink runner) . Wanted to make sure there was no reason to keep PipelineRule.
@kennknowles made some minor changes to TestPipeline PTAL.

Last time I checked, Dataflow ValidatesRunner was the only one that failed - was it just AvroCoderTest or was it more than that?

Yeah it was AvroCoderTest

Retest this please.

run flink validatesrunner

run dataflow validatesrunner

Coverage increased (+0.6%) to 70.536% when pulling 43e8f32f57a1a17ad010bfa3af2657ea8906aa21 on aviemzur:testpipeline-passert-verification into 0d19d3cb210ef179e1831b971f7cf6d3d6bb1dde on apache:master.

Coverage increased (+0.6%) to 70.511% when pulling be900bac3bde0130bac8e28cf1dfba4714d79b96 on aviemzur:testpipeline-passert-verification into 0d19d3cb210ef179e1831b971f7cf6d3d6bb1dde on apache:master.


Add minimum supported Rust version

Currently it's 1.18.

@RazrFalcon would you like to submit a PR that adds 1.18 to the .travis.yml file?

Only travis? What about readme?

We could put the min version in the readme too. If we did I think we should add a comment to the travis config with a reminder to update the readme too, maybe like:
- 1.18.0 # don't forget to update the readme too

so they're less likely to get out of sync.


COEUS grant loader fails  in demo env. due to duplication of some records

Troubleshoot and address this issue.

There's no need to troubleshoot, we already know the reason that it fails.  There are duplicate records in Elasticsearch for some Users and so far, one Funder.
I am working with Aaron to clear up these issues.  I am working on a way to force close the ssh tunnel until we can split these two functions (harvesting and loading) into two functions that can be run on separate servers.  This way, the harvesting can be done inside the University network as required and the loader can be run inside the AWS network, as required.  This will eliminate the need for the fragile ssh tunnel.

I believe we don't know yet why/how these duplicated records are created.
That said, this issue can be considered resolved when the issue is addressed and the grant loader in both demo and production are back to running daily.

That's correct, we don't know why.  All signs point to this ssh tunnel but we may never know.  This seems to be reaching back all the way to early august.
We, most definitely, do not keep logs of coeus runs that long.  It's not integrated into the AWS logging infrastructure.

I have redesigned the way that we run this at night.  It has been running two nights in a row without issue.  At this point, I believe it's safe to close this.
It'll be nice to be able to break this out in the future.


Fixed #6021 -- Render inherited content even when the parent is cached

Fixed #6021

Coverage remained the same at 82.688% when pulling d09a61015e767938ab9a5a7adb072915a6231d69 on czpython:fixes/3.4.x/6021 into 9074e583d045409b560a8cb07e1c2bca496e4c4f on divio:release/3.4.x.